# Babe Ruth and the Hot-Hand Hypothesis

According to Wikipedia, the so-called hot-hand fallacy is that “a person who has experienced success with a seemingly random event has a greater chance of further success in additional attempts.” The article continues:

[R]esearchers for many years did not find evidence for a “hot hand” in practice. However, later research has questioned whether the belief is indeed a fallacy. More recent studies using modern statistical analysis have shown that there is evidence for the “hot hand” in some sporting activities.

I won’t repeat the evidence cited in the Wikipedia article, nor will I link to the many studies about the hot-hand effect. You can follow the link and read it all for yourself.

What I will do here is offer an analysis that supports the hot-hand hypothesis, taking Babe Ruth as a case in point. Ruth was a regular position player (non-pitcher) from 1919 through 1934. In that span of 16 seasons he compiled 688 home runs (HR) in 7,649 at-bats (AB) for an overall record of 0.0900 HR/AB. Here are the HR/AB tallies for each of the 16 seasons:

 Year HR/AB 1919 0.067 1920 0.118 1921 0.109 1922 0.086 1923 0.079 1924 0.087 1925 0.070 1926 0.095 1927 0.111 1928 0.101 1929 0.092 1930 0.095 1931 0.086 1932 0.090 1933 0.074 1934 0.060

Despite the fame that accrues to Ruth’s 1927 season, when he hit 60 home runs, his best season for HR/AB came in 1920. In 1919, Ruth set a new single-season record with 29 HR. He almost doubled that number in 1920, getting 54 HR in 458 AB for 0.118 HR/AB.

Here’s what that season looks like, in graphical form:

The word for it is “streaky”, which isn’t surprising. That’s the way of most sports. Streaks include not only cold spells but also hot spells. Look at the relatively brief stretches in which Ruth was shut out in the HR department. And look at the relatively long stretches in which he readily exceeded his HR/AB for the season. (For more about the hot and and streakiness, see Brett Green and Jeffrey Zwiebel, “The Hot-Hand Fallacy: Cognitive Mistakes or Equilibrium Adjustments? Evidence from Major League Baseball“, Stanford Graduate School of Business, Working Paper No. 3101, November 2013.)

The same pattern can be inferred from this composite picture of Ruth’s 1919-1934 seasons:

Here’s another way to look at it:

If hitting home runs were a random thing — which they would be if the hot hand were a fallacy — the distribution would be tightly clustered around the mean of 0.0900 HR/AB. Nor would there be a gap between 0 HR/AB and the 0.03 to 0.06 bin. In fact, the gap is wider than that; it goes from 0 to 0.042 HR/AB. When Ruth broke out of a home-run slump, he broke out with a vengeance, because he had the ability to do so.

In other words, Ruth’s hot streaks weren’t luck. They were the sum of his ability and focus (or “flow“); he was “putting it all together”. The flow was broken at times — by a bit of bad luck, a bout of indigestion, a lack of sleep, a hangover, an opponent who “had his number”, etc. But a great athlete like Ruth bounces back and put it all together again and again, until his skills fade to the point that he can’t overcome his infirmities by waiting for his opponents to make mistakes.

The hot hand is the default condition for a great player like a Ruth or a Cobb. The cold hand is the exception until the great player’s skills finally wither. And there’s no sharp dividing line between the likes of Cobb and Ruth and lesser mortals. Anyone who has the ability to play a sport at a professional level (and many an amateur, too) will play with a hot hand from time to time.

The hot hand isn’t a fallacy or a matter of pure luck (or randomness). It’s an artifact of skill.

/

# Further Thoughts about Metaphysical Cosmology

I have stated my metaphysical cosmology:

1. There is necessarily a creator of the universe, which comprises all that exists in “nature.”

2. The creator is not part of nature; that is, he stands apart from his creation and is neither of its substance nor governed by its laws. (I use “he” as a term of convenience, not to suggest that the creator is some kind of human or animate being, as we know such beings.)

3. The creator designed the universe, if not in detail then in its parameters. The parameters are what we know as matter-energy (substance) and its various forms, motions, and combinations (the laws that govern the behavior of matter-energy).

4. The parameters determine everything that is possible in the universe. But they do not necessarily dictate precisely the unfolding of events in the universe. Randomness and free will are evidently part of the creator’s design.

5. The human mind and its ability to “do science” — to comprehend the laws of nature through observation and calculation — are artifacts of the creator’s design.

6. Two things probably cannot be known through science: the creator’s involvement in the unfolding of natural events; the essential character of the substance on which the laws of nature operate.

It follows that science can neither prove nor disprove the preceding statements. If that is so, why can I not say, with equal certainty, that the universe is made of pea soup and supported by undetectable green giants?

There are two answers to that question. The first answer is that my cosmology is based on logical necessity; there is nothing of logic or necessity in the claims about pea soup and undetectable green giants. The second and related answer is that claims about pea soup and green giants — and their ilk — are obviously outlandish. There is an essential difference between (a) positing a creator and making limited but reasonable claims about his role and (b) engaging in obviously outlandish speculation.

What about various mythologies (e.g., Norse and Greek) and creation legends, which nowadays seem outlandish even to persons who believe in a creator? Professional atheists (e.g., Richard Dawkins, Daniel Dennett, Christopher Hitchens, and Lawrence Krauss) point to the crudeness of those mythologies and legends as a reason to reject the idea of a creator who set the universe and its laws in motion. (See, for example, “Russell’s Teapot,” discussed here.) But logic is not on the side of the professional atheists. The crudeness of a myth or legend, when viewed through the lens of contemporary knowledge, cannot be taken as evidence against creation. The crudeness of a myth or legend merely reflects the crudeness of the state of knowledge when the myth or legend arose.

# My Metaphysical Cosmology

This post is a work in progress. It draws on and extends the posts listed at the bottom.

1. There is necessarily a creator of the universe, which comprises all that exists in “nature.”

2. The creator is not part of nature; that is, he stands apart from his creation and is neither of its substance nor governed by its laws. (I use “he” as a term of convenience, not to suggest that the creator is some kind of human or animate being, as we know such beings.)

3. The creator designed the universe, if not in detail then in its parameters. The parameters are what we know as matter-energy (substance) and its various forms, motions, and combinations (the laws that govern the behavior of matter-energy).

4. The parameters determine everything that is possible in the universe. But they do not necessarily dictate precisely the unfolding of events in the universe. Randomness and free will are evidently part of the creator’s design.

5. The human mind and its ability to “do science” — to comprehend the laws of nature through observation and calculation — are artifacts of the creator’s design.

6. Two things probably cannot be known through science: the creator’s involvement in the unfolding of natural events; the essential character of the substance on which the laws of nature operate.

# More about Luck and Baseball

In “Luck and Baseball, One More Time,” I make the point that

it takes a lot more than luck to succeed at almost anything, from winning high office to making millions of dollars to painting a masterpiece to building a house to cutting hair properly. To denigrate the rich and famous by calling them lucky is to denigrate every person who strives, with some success, to overmaster whatever bad luck happens to come his way.

The backdrop for that claim is some statistical evidence from the history of major-league baseball:

In the 111-year history of the American League, 60 different players have led the league in batting. Those 60 players have recorded a total of 367 top-10 finishes in American League batting races over the years — an average of 6 top-10 finishes for each of the players. It is not surprising, therefore, that most of the 60 players also compiled excellent career batting averages. Specifically, through 2010, 57 of the 60 had made at least 5,000 plate appearance in the American League, and 43 of the 57 are among the top 120 hitters (for average) — out of the thousands of players with at least 5,000 plate appearances in the American League. Were those 43 players merely “lucky”? It takes a lot more than luck to hit so well, so consistently, and for so many years.

Here is more evidence to the same effect. Two days ago, a young pitcher for the Chicago White Sox named Philip Humber threw a perfect game against the Seattle Mariners. Humber’s was the 19th perfect game since 1893, when the distance from the pitcher’s plate (rubber) to the back point of home plate (where the foul lines intersect) was increased to 60 feet, 6 inches. The 19 perfect games were pitched by 19 different men. And the total number of major league games played from 1893 through today numbers well above 300,000, which means that the potential number of perfect games (if thrown by both teams’ pitchers) is well above 600,000.

Aha, you might say, a perfect game is a matter of luck. Well, it may be partly a matter of luck, but baseball (despite some elements of randomness) is a game of skill, applied intentionally. A perfect game, like many other aspects of baseball, is the residue of the applied skills of pitchers and fielders, just as (the prevalent) imperfect game is the residue of the applied skills of batters and base runners.

The element of skill involved in pitching a perfect game is evidenced by the fact that most of the players who have pitched perfect games are the holders of above-average to exceptional pitching records:

 Career Record* Year of perfect game Pitcher Seasons played Wins Losses W-L % ERA+** Hall of Fame?*** 1904 Cy Young 1890-1911 511 316 .618 138 Yes 1908 Addie Joss 1902-1910 160 97 .623 142 Yes 1922 Charlie Robertson 1919-1928 49 80 .380 90 No 1956 Don Larsen 1953-1967 81 91 .471 99 No 1964 Jim Bunning 1955-1971 224 184 .549 114 Yes 1965 Sandy Koufax 1955-1966 165 87 .655 131 Yes 1968 Catfish Hunter 1965-1979 224 166 .574 105 Yes 1981 Len Barker 1976-1987 74 76 .493 94 No 1984 Mike Witt 1981-1993 117 116 .502 105 No 1988 Tom Browning 1984-1995 123 90 .577 98 No 1991 Dennis Martinez 1976-1998 245 193 .559 106 No 1994 Kenny Rogers 1989-2008 219 156 .584 108 Not yet eligible 1998 David Wells 1987-2007 239 157 .604 108 Not yet eligible 1999 David Cone 1986-2003 194 126 .606 121 No 2004 Randy Johnson 1988-2009 303 166 .646 136 Not yet eligible 2009 Mark Buehrle 2000- 162 121 .572 120 Active player 2010 Dallas Braden 2007- 26 36 .419 102 Active player 2010 Roy Halladay 1998- 191 93 .673 139 Active player 2012 Philip Humber 2006- 12 10 .545 110 Active player Combined W-L 3319 2361 .584 * Through April 22, 2012. ** Earned run average adjusted for ballpark and the league’s mean ERA in each season. An ERA+ of 100 is therefore an average performance over a career; ERA+ >100 is above average; ERA+ <100 is below average. (Details here: http://en.wikipedia.org/wiki/ERA%2B.) *** Membership in the Hall of Fame is noted for the sake of completeness, though it is not conclusive proof of greatness. (See: http://libertycorner.blogspot.com/2006/10/anti-hall-of-fame-and-baseball.html; http://libertycorner.blogspot.com/2007/12/hall-of-famers.html.)

The point of this excursion into baseball is stated in an old post of mine:

A bit of unpredictability (or “luck”) here and there does not make for a random universe, random lives, or random markets. If a bit of unpredictability here and there dominated our actions, we wouldn’t be here to talk about randomness….

Human beings are not “designed” for randomness. Human endeavors can yield unpredictable results, but those results do not arise from random processes, they derive from skill or the lack therof, knowledge or the lack thereof … , and conflicting objectives…

In baseball, as in life, “luck” is mainly an excuse and rarely an explanation….

/

# Luck and Baseball, One More Time

There is such a thing as “luck.” Bad and good luck happen to everyone, at one time or another. But everything that happens to everyone is not due to luck. I am convinced by what I have seen of life — up close and at a distance — that most of what happens to people happens to them because of their intentions, skills, and resources.

Yes, the skills that one possesses may be due in part to genetic luck, and the resources that one can marshal may be due in part to genetic and geographic luck. But if skills and resources were entirely beyond a person’s control, no one would ever climb from the proverbial gutter to attain fame and fortune. That is where intentions come in.

So, I am unimpressed (to say the least) by do-gooders and levelers, who want to take from the productive and give to the unproductive because the productive have had “all the luck,” or some such thing. Balderdash! First, it takes more than luck to be productive and to enjoy even a modest income. Second, taking from the productive to give to the unproductive is like blaming the blameless. It may come as a surprise to do-gooders and levelers (most of whom ought to know better), but a person who earns a high income earns it because that is what others are willing to pay for his efforts — not because he picks the pockets of the poor.

Speaking of high-income earners, I am always puzzled by the fact that income-envy is directed toward CEOs, investment bankers, and suchlike. Why is it not directed at super-star athletes, like Albert Pujols, who will earn \$254 million over the next 10 years, just for playing baseball? Perhaps it is because almost everyone recognizes that Pujols is selling a skill that (a) is his (not stolen from someone else) and (b) would not be on display were it not for his assiduous development and application of the particular genetic advantages that enable him to hit a pitched baseball with above-average frequency and power.

Well, Nassim Nicholas Taleb to the contrary notwithstanding, the earning of large sums of money in any profession takes the same assiduous application of particular genetic advantages, or assiduous compensation for the lack thereof. I will not repeat my detailed criticisms of Taleb, which can be found “here” and “here.” Instead, I will return to the subject of baseball, some aspects of which I treated in those posts.

In the 111-year history of the American League, 60 different players have led the league in batting. Those 60 players have recorded a total of 367 top-10 finishes in American League batting races over the years — an average of 6 top-10 finishes for each of the players. It is not surprising, therefore, that most of the 60 players also compiled excellent career batting averages. Specifically, through 2010, 57 of the 60 had made at least 5,000 plate appearance in the American League, and 43 of the 57 are among the top 120 hitters (for average) — out of the thousands of players with at least 5,000 plate appearances in the American League. Were those 43 players merely “lucky”? It takes a lot more than luck to hit so well, so consistently, and for so many years.

And it takes a lot more than luck to succeed at almost anything, from winning high office to making millions of dollars to painting a masterpiece to building a house to cutting hair properly. To denigrate the rich and famous by calling them lucky is to denigrate every person who strives, with some success, to overmaster whatever bad luck happens to come his way.

/

# Finding Order in Chaos

Here is an apparently random plot of daily changes in an index:

Can such randomness yield an orderly outcome? To find out, click “Read more of this post.”

# Randomness Is Over-Rated

In the preceding post (“Fooled by Non-Randomness“), I had much to say about Nassim Nicholas Taleb’s Fooled by Randomness. The short of is this: Taleb over-rates the role of randomness in financial markets. In fact, his understanding of randomness seems murky.

My aim here is to offer a clearer picture of randomness (or the lack of it), especially as it relates to human behavior. Randomness, as explained in the preceding post, has almost nothing to do with human behavior, which is dominated by intention. Taleb’s misapprehension of randomness leads him  to overstate the importance of a thing called survivor(ship) bias, to which I will turn after dealing with randomness.

WHERE IS RANDOMNESS FOUND?

Randomness — true randomness — is to be found mainly in the operation of fair dice, fair roulette wheels, cryptograhic pinwheels, and other devices designed expressly for the generation of random values. But what about randomness in human affairs?

What we often call random events in human affairs really are non-random events whose causes we do not and, in some cases, cannot know. Such events are unpredictable, but they are not random. Such is the case with such things as rolls (throws) of fair dice — which are considered random events. Dice-rolls are “random” only because it is impossible to perceive the precise conditions of each roll in “real time,” even though knowledge of those conditions would enable a sharp-eyed observer to forecast the outcome of each throw with some accuracy, if the observer were armed with — and had instant access to — analyses of the results of myriad throws whose precise conditions had been captured by various recording devices.

An observer who lacks such information, and who considers the throws of fair dice to be random events, will see that the total number of pips showing on both dice converges on the following frequency distribution:

 Rolled Freq. 2 0.028 3 0.056 4 0.083 5 0.111 6 0.139 7 0.167 8 0.139 9 0.111 10 0.083 11 0.056 12 0.028

This frequency distribution is really a shorthand way of writing 28 times out of 1,000; 56 times out of 1,000; etc.

Stable frequency distributions, such as the one given above, have useful purposes. In the case of craps, for example, a bettor can minimize his losses to the house (over a long period of time) if he takes the frequency distribution into account in his betting. Even more usefully, perhaps, an observed divergence from the normal frequency distribution (over many rolls of the dice) would indicate bias caused by (a) an unusual and possibly fraudulent condition (e.g., loaded dice) or (b) a player’s special skill in manipulating dice to skew the frequency distribution in a certain direction.

Randomness, then, is found in (a) the results of non-intentional actions, where (b) we lack sufficient knowledge to understand the link between actions and results.

THE REAL WORLD OF HUMAN AFFAIRS: IT IS WHAT IT IS

You will have noticed the beautiful symmetry of the frequency distribution for dice-rolling. Two-thirds of a large number of dice-rolls will have values of 5 through 9. Values 3 and 4 together will comprise about 14 percent of the rolls, as will values 10 and 11 together. Values 2 and 12 each will comprise less than 3 percent of the rolls.

In other words, the frequency distribution for dice-rolls closely resembles a normal distribution (bell curve). The virtue of this regularity is that it makes predictable the outcome of a large number of dice-rolls; and it makes obvious (over many dice-rolls) a rigged game involving dice. A statistically unexpected distribution of dice-rolls would be considered non-random or, more plainly, rigged — that is, intended by the rigging party.

To state the underlying point explicitly: It is unreasonable to reduce intentional human behavior to probabilistic formulas. Humans don’t behave like dice, roulette balls, or similar “random” devices. But that is what Taleb (and others) do when they ascribe unusual success in financial markets to “luck.” For example, here is what Taleb says on page 136:

I do not deny that if someone performed better than the crowd in the past, there is a presumption of his ability to do better in the future. But the presumption might be weak, very weak, to the point of being useless in decision making. Why? Because it all depends on two factors: The randomness-content of his profession and the number of [persons in the profession].

What Taleb means is this:

• Success in a profession where randomness dominates outcomes is likely to have the same kind of distribution as that of an event that is considered random, like rolling dice.
• That being the case, a certain percentage of the members of the profession will, by chance, seem to have great success.
• If a profession has relatively few members, than a successful person in that profession is more of a standout than a successful person in a profession with, say, thousands of members.

Let me count the assumptions embedded in Taleb’s argument:

1. Randomness actually dominates some professions. (In particular, he is thinking of the profession of trading financial instruments: stocks, bonds, derivatives, etc.)
2. Success in a randomness-dominated profession therefore has almost nothing to do with the relevant skills of a member of that profession, nor with the member’s perspicacity in applying those skills.
3. It follows that a very successful member of a randomness-dominated profession is probably very successful because of luck.
4. The probability of stumbling across a very successful member of a randomness-dominated profession depends on the total number of members of the profession, given that the probability of success in the profession is distributed in a non-random way (as with dice-rolls).

One of the ways in which Taleb illustrates his thesis is to point to the mutual-fund industry, where far fewer than half the industry’s actively managed funds fail to match the performance of benchmark indices (e.g., S&P 500) over periods of 5 years and longer. But broad, long-term movements in financial markets are not random — as I show in the preceding post.

Nor is trading in financial instruments random; traders do not roll dice or flip coins when they make trades. (Well, the vast majority don’t.) That a majority (or even a super-majority) of actively managed funds does less well than an index fund has nothing to do with randomness and everything to do with the distribution of stock-picking skills. The research required to make informed decisions about financial instruments is arduous and expensive — and not every fool can do it well. Moreover, decision-making — even when based on thorough research — is clouded by uncertainty about the future and the variety of events that can affect the prices of financial instruments.

It is therefore unsurprising  that the distribution of skills in the financial industry is skewed; there are relatively few professionals who have what it takes to succeed over the long run, and relatively many professionals (or would-be professionals) who compile mediocre-to-awful records.

I say it again: The most successful professionals are not successful because of luck, they are successful because of skill. There is no statistically predetermined percentage of skillful traders; the actual percentage depends on the skills of entrants and their willingness (if skillful) to make a career of it. A relevant analogy is found in the distribution of incomes:

In 2007, all households in the United States earned roughly \$7.896 trillion [25]. One half, 49.98%, of all income in the US was earned by households with an income over \$100,000, the top twenty percent. Over one quarter, 28.5%, of all income was earned by the top 8%, those households earning more than \$150,000 a year. The top 3.65%, with incomes over \$200,000, earned 17.5%. Households with annual incomes from \$50,000 to \$75,000, 18.2% of households, earned 16.5% of all income. Households with annual incomes from \$50,000 to \$95,000, 28.1% of households, earned 28.8% of all income. The bottom 10.3% earned 1.06% of all income.

The outcomes of human endeavor are skewed because the distribution of human talents is skewed. It would be surprising to find as many as one-half of traders beating the long-run average performance of the various markets in which they operate.

BACK TO BASEBALL

To drive the point home, I return to the example of baseball, which I treated at length in the preceding post. Baseball, like most games, has many “random” elements, which is to say that baseball players cannot always predict accurately such things as the flight of a thrown or batted ball, the course a ball will take when it bounces off grass or an outfield fence, the distance and direction of a throw from the outfield, and so on. But despite the many unpredictable elements of the game, skill dominates outcomes over the course of seasons and careers. Moreover, skill is not distributed in a neat way, say, along a bell curve. A good case in point is the distribution of home runs:

• There have been 16,884 players and 253,498 home runs in major-league history (1876 – present), an average of 15 home runs per person who have played in the major leagues since 1876. About 2,700 players have more than 15 home runs; about 14,000 players have fewer than 15 home runs; and about 100 players have exactly 15 home runs. Of the 2,700 players with more than 15 home runs, there are (as of yesterday) 1,006 with 74 or more home runs, and 25 with 500 or more home runs. (I obtained data about the frequency of career home runs with this search tool at Baseball-Reference.com.)
• The career home-run statistic, in other words, has an extremely long, thin “tail” that, at first, rises gradually from 0 to 15. This tail represents the home-run records of about 89 percent of all the men who have played in the major leagues. The tail continues to broaden until, at the other end, it becomes a very short, very fat hump, which represents the 0.15 percent of players with 500 or more home runs.
• There may be a standard statistical distribution which seems to describe the incidence of career home runs. But to say that the career home-run statistic matches any kind of distribution is merely to posit an after-the-fact “explanation” of a phenomenon that has one essential explanation: Some hitters are better at hitting home runs than other players; those better home-run hitters are more likely to stay in the major leagues long enough to compile a lot of home runs. (Even 74 home runs is a lot, relative to the mean of 15.)

And so it is with traders and other active “players” in financial markets. They differ in skill, and their skill differences cannot be arrayed neatly along a bell curve or any other mathematically neat frequency distribution. To adapt a current coinage, they are what they are — nothing more, nothing less.

TALEB’S A PRIORI WORLDVIEW, WITH A BIAS

Taleb, of course, views the situation the other way around. He sees an a priori distribution of “winners” and losers,” where “winners” are determined mainly by luck, not skill. Moreover, we — the civilians on the sidelines — labor under the false impression about the relative number of “winners” because

it is natural for those who failed to vanish completely. Accordingly, one sees the survivors, and only the survivors, which imparts such a mistaken perception of the odds [favoring success]. (p. 137)

Here, Taleb is playing a variation on a favorite theme: survivor(ship) bias. What is it? Here are three quotations that may help you understand it:

Survivor bias is a prominent form of ex-post selection bias. It exists in data sets that exclude a disproportionate share of non-surviving firms…. (“Accounting Information Free of Selection Bias: A New UK Database 1953-1999 “)

Survivorship bias causes performance results to be overstated
because accounts that have been terminated, which may have
underperformed, are no longer in the database. This is the most
documented and best understood source of peer group bias. For example, an unsuccessful management product that was
terminated in the past is excluded from current peer groups.
This screening out of losers results in an overstatement of past
performance. A good illustration of how survivor bias can skew
things is the “marathon analogy”, which asks: If only 100 runners out of a 1,000?contestant marathon actually finish, is the 100th the last? Or in the top ten percent? (“Warning! Peer Groups Are Hazardous to Our Wealth“)

It is true that a number of famous successful people have spent 10,000 hours practising. However, it is also true that many people we have never heard of because they weren’t successful also practised for 10,000 hours. And that there are successful people who were very good without practising for 10,000 hours before their breakthrough (the Rolling Stones, say). And Gordon Brown isn’t very good at being Prime Minister despite preparing for 10,000 hours. (“Better Services without Reform? It’s Just a Con“)

First of all, there are no “odds” favoring success — even in financial markets. Financial “players” do what they can do, and most of them — like baseball players — simply don’t have what it takes for great success. Outcomes are skewed, not because of (fictitious) odds but because talent is distributed unevenly.

CONCLUSION

The real lesson for us spectators is not to assume that the “winners” are merely lucky. No, the real lesson is to seek out those “winners” who have proven their skills over a long period of time, through boom and bust and boom and bust.

Those who do well, over the long run, do not do so merely because they have survived. They have survived because they do well.