Analysis for Government Decision-Making: Demi-Science, Hemi-Demi-Science, and Sophistry

Taking a “hard science” like classical mechanics as an epitome of science and, say, mechanical engineering, as a rigorous application of it, one travels a goodly conceptual distance before arriving at operations research (OR). Philip M. Morse and George E. Kimball, pioneers of OR in World War II, put it this way:

[S]uccessful application of operations research usually results in improvements by factors of 3 or 10 or more…. In our first study of any operation we are looking for these large factors of possible improvement…. They can be discovered if the [variables] are given only one significant figure,…any greater accuracy simply adds unessential detail.

One might term this type of thinking “hemibel thinking.” A bel is defined as a unit in a logarithmic scale corresponding to a factor of 10. Consequently a hemibel corresponds to a factor of the square root of 10, or approximately 3. (Philip M. Morse and George E. Kimball, Methods of Operations Research, originally published as Operations Evaluation Group Report 54, 1946, p. 38)

This is science-speak for the following proposition: Where there is much variability in the particular circumstances of combat, there is much uncertainty about the contributions of various factors (human, mechanical, and meteorological) the the outcome of combat. It is therefore difficult to assign precise numerical values to the various factors.

OR, even in wartime, is therefore, and at best, a demi-science. From there, we descend to cost-effectiveness analysis and its constituent branches: techniques for designing and estimating the costs of systems that do not yet exist and the effectiveness of such systems in combat. These methods, taken separately and together, are (to coin a term) hemi-demi-scientific — a fact that the application of “rigorous” mathematical and statistical techniques cannot alter.

There is no need to elaborate on the wild inaccuracy of estimates about the costs and physical performance of government-owned and operated systems, whether they are intended for military or civilian use. The gross errors of estimation have been amply documented in the public press for decades.

What is less well known is the difficulty of predicting the performance of systems — especially combat systems — years before they are commanded, operated, and maintained by human beings, under conditions that are likely to be far different than those envisioned when the systems were first proposed. A paper that I wrote thirty years ago gives my view of the great uncertainty that surrounds estimates of the effectiveness of systems that have yet to be developed, or built, or used in combat:

Aside from a natural urge for certainty, faith in quantitative models of warfare springs from the experience of World War II, when they seemed to lead to more effective tactics and equipment. But the foundation of this success was not the quantitative methods themselves. Rather, it was the fact that the methods were applied in wartime. Morse and Kimball put it well:

Operations research done separately from an administrator in charge of operations becomes an empty exercise. To be valuable it must be toughened by the repeated impact of hard operational facts and pressing day-by-day demands, and its scale of values must be repeatedly tested in the acid of use. Otherwise it may be philosophy, but it is hardly science. [Methods of Operations Research, p. 10]

Contrast this attitude with the attempts of analysts … to evaluate weapons, forces, and strategies with abstract models of combat. However elegant and internally consistent the models, they have remained as untested and untestable as the postulates of theology.

There is, of course, no valid test to apply to a warfare model. In peacetime, there is no enemy; in wartime, the enemy’s actions cannot be controlled. Morse and Kimball, accordingly, urge “hemibel thinking”:

Having obtained the constants of the operations under study… we compare the value of the constants obtained in actual operations with the optimum theoretical value, if this can be computed. If the actual value is within a hemibel (…a factor of 3) of the theoretical value, then it is extremely unlikely that any improvement in the details of the operation will result in significant improvement. [When] there is a wide gap between the actual and theoretical results … a hint as to the possible means of improvement can usually be obtained by a crude sorting of the operational data to see whether changes in personnel, equipment, or tactics produce a significant change in the constants. [Ibid., p. 38]

….

Much as we would like to fold the many different parameters of a weapon, a force, or a strategy into a single number, we can not. An analyst’s notion of which variables matter and how they interact is no substitute for data. Such data as exist, of course, represent observations of discrete events — usually peacetime events. It remains for the analyst to calibrate the observations, but without a benchmark to go by. Calibration by past battles is a method of reconstruction –of cutting one of several coats to fit a single form — but not a method of validation. Lacking pertinent data, an analyst is likely to resort to models of great complexity. Thus, if useful estimates of detection probabilities are unavailable, the detection process is modeled; if estimates of the outcomes of dogfights are unavailable, aerial combat is reduced to minutiae. Spurious accuracy replaces obvious inaccuracy; untestable hypotheses and unchecked calibrations multiply apace. Yet the analyst claims relative if not absolute accuracy, certifying that he has identified, measured, and properly linked, a priori, the parameters that differentiate weapons, forces, and strategies….

Should we really attach little significance to differences of less than a hemibel? Consider a five-parameter model, involving the conditional probabilities of detecting, shooting at, hitting, and killing an opponent — and surviving, in the first place, to do any of these things. Such a model might easily yield a cumulative error of a hemibel, given a twenty-five percent error in each parameter. My intuition is that one would be lucky if relative errors in the probabilities assigned to alternative weapons and forces were as low as twenty-five percent.

The further that one travels from an empirical question, such as the likely effectiveness of an extant weapon system under specific, quantifiable conditions, the more likely one is to encounter the kind of sophistry known as policy analysis. It is in this kind of analysis that one  — more often than not — encounters in the context of broad policy issues (e.g., government policy toward health care, energy, or defense spending). Such analysis is constructed so that it favors the prejudices of the analyst or his client, or support the client’s political case for a certain policy.

Policy analysis often seems credible, especially on first hearing or reading it. But, on inspection, it is usually found to have at least two of these characteristics:

  • It stipulates or quickly arrives at a preferred policy, then marshals facts, calculations, and opinions that are selected because they support the preferred policy.
  • If it offers and assesses alternative policies, they are not placed on an equal footing with the preferred policy. They are, for example, assessed against criteria that favor the preferred policy, while other criteria (which might be important ones) are ignored or given short shrift.
  • It is wrapped in breathless prose, dripping with words and phrases like “aggressive action,”grave consequences,” and “sense of urgency.”

No discipline or quantitative method is rigorous enough to redeem policy analysis, but two disciplines are especially suited to it: political “science” and macroeconomics. Both are couched in the language of real science, but both lend themselves perfectly to the old adage: garbage in, garbage out.

Do I mean to suggest that broad policy issues should not be addressed as analytically as possible? Not at all. What I mean to suggest is that because such issues cannot be illuminated with scientific rigor, they are especially fertile ground for sophists with preconceived positions.

In that respect, the model of cost-effectiveness analysis, with all of its limitations, is to be emulated. Put simply, it is to state a clear objective in a way that does not drive the answer; reveal the assumptions underlying the analysis; state the relevant variables (factors influencing the attainment of the objective); disclose fully the data, the sources of data, and analytic methods; and explore openly and candidly the effects of variations in key assumptions and critical variables.

Politics, Sophistry, and the Academy

By “politics” I mean the business of acquiring and applying governmental power, which involves — among other things — persuading the electorate, either directly or through advertising and the utterances of political allies and friendly “opinion elites” among journalists and academicians.

“Sophistry” is more complex. Its meaning has evolved, as described at Wikipedia:

The Greek word sophos, or sophia, has had the meaning “wise” or “wisdom” since the time of the poet Homer and originally was used to describe anyone with expertise in a specific domain of knowledge or craft. For example, a charioteer, a sculptor or a warrior could be described as sophoi in their occupations. Gradually, however, the word also came to denote general wisdom and especially wisdom about human affairs (in, for example, politics, ethics, or household management)….

In the second half of the 5th century BC, particularly at Athens, “sophist” came to denote a class of mostly itinerant intellectuals who taught courses in various subjects, speculated about the nature of language and culture and employed rhetoric to achieve their purposes, generally to persuade or convince others: “Sophists did, however, have one important thing in common: whatever else they did or did not claim to know, they characteristically had a great understanding of what words would entertain or impress or persuade an audience.”…

Plato is largely responsible for the modern view of the “sophist” as a greedy instructor who uses rhetorical sleight-of-hand and ambiguities of language in order to deceive, or to support fallacious reasoning. In this view, the sophist is not concerned with truth and justice, but instead seeks power.

Here, I am concerned with sophistry in its modern, political sense: the cynical use of language in the pursuit and application of power. In a word, propaganda. Josef Pieper (1903-97), a German Catholic philosopher, has much to say about this in Abuse of Language, Abuse of Power (for which I thank my son). There, Pieper notes that propaganda

can be found wherever a powerful organization, and ideological clique, a special interest, or a pressure group uses the word as their “weapon”…. (p. 32)

A bit later, Pieper says that “the abuse of political power is fundamentally connected with the sophistic abuse of the word” (p. 32). But this is nothing new under sun, and should come as no surprise to anyone who has even a superficial knowledge of modern history and understanding of politics.

Less evident, I believe, is the tragically corrosive confluence of politics and sophistry in the academy. To push the metaphor, what was a trickle in the middle of the twentieth century has grown to a wide, roaring river of academic dishonesty in the service of political ends.

About the corruption of the academy, Pieper writes:

[T]he term academic expresses something that remained unchanged throughout the centuries, something that can be identified quite accurately. It seems that in the midst of society there is expressly reserved an area of truth, a sheltered space for the autonomous study of reality, where it is possible, without restrictions,to examine, investigate, discuss, and express what is true about any thing — a space, then, explicitly protected against all potential special interests and invading influences, where hidden agendas have no place, be they collective or private, political, economic, or ideological. At this time in history we  have been made aware amply, and forcefully as well, what consequences ensue when a society does or does not provide such a “refuge”. Clearly, this is indeed a matter of freedom — not the whole of freedom, to be sure, yet an essential and indispensable dimension of freedom. Limitations and restrictions imposed from the outside are intolerable enough; it is even more depressing for the human spirit when it is made impossible to express and share, that is, to declare publicly, what according to one’s best knowledge and clear conscience is the truth about things….

Such a space of freedom needs not only a guarantee from the outside, from the political power that thus imposes limits on itself. Such a space of freedom also depends on the requirement that freedom be constituted — and defended — within its own domain. By “defended” we mean here not against any threat from the outside but against dangers arising — disturbingly! — within the scholarly domain itself…. (pp. 37-8)

Pieper’s depiction of the academy may seem, at first glance, to be unrealistically romantic. But Pieper is merely setting forth an ideal toward which the academy should strive. If the academy would renew its dedication to the ideal, it would banish and bar the sophists who lurk within and seek to infiltrate it.

Where are the sophists most likely to be found in the academy? In such pseudo-disciplines as “women’s studies,” “black studies,” and the like, of course. But also throughout the liberal arts, humanities, and social sciences, where artistic forms, economics, history, literature, and sociology serve (to name a few) serve as vehicles for those who would destroy the foundations of Western civilization in the name of “liberation,” “equality,” and “social justice.” Even the sciences are not immune, as evidenced by almost-obligatory belief in anthropogenic global warming, and the need to subscribe to that belief or be ostracized. More generally, there is the selection bias that works against the hiring and promotion of those who come from schools that have a reputation for dissenting from left-wing academic orthodoxy, or who have themselves overtly dissented from the orthodoxy. Especially damning is the fact of dissent based on hard evidence and impeccable logic — especially damning because it is especially threatening to the orthodoxy.

The last word goes to Pieper:

“Academic” must mean “antisophistic” if it is to mean anything at all. This implies also opposition to anything that could destroy or distort the nature of the word as communication and its unbiased openness to reality. In this respect we are well able to pronounce the general principle and at the same time to to be very specific: opposition is required, for instance, against every partisan simplification, every ideological agitation, eery blind emotionality; against seduction through well-turned yet empty slogans, against autocratic terminology with no room for dialogue, against personal insult as an element of style … , against the language of evasive appeasement and false assurance …, and not least against the jargon of the revolution, against categorical conformism, and categorical nonconformism…. (pp. 38-9)

Related posts:
I Missed This One
Affirmative Action: Two Views from the Academy
Feminist Balderdash
What Is the Point of Academic Freedom?
How to Deal with Left-Wing Academic Blather
It’s Not Anti-Intellectualism, Stupid
The Case Against Campus Speech Codes
The Left, Abortion, and Adolescence
Lefty Profs
Apropos Academic Freedom and Western Values
Why So Few Free-Market Economists?
Affirmative Action: Two Views from the Academy, Revisited
Affirmative Action for Conservatives and Libertarians?
Academic Bias
Intellectuals and Capitalism
The Left
Enough of Krugman
“Intellectuals and Society”: A Review
Social Justice
The Left’s Agenda
More Social Justice
The Left and Its Delusions
The Ideal as a False and Dangerous Standard
Utilitarianism and Psychopathy
Demystifying Science