Surviving Statistics

published September 2014
How the survivor bias distorts reality
magazine cover

When I purchased my latest vehicle, I was astonished to get the license plate 6NWL485. What are the chances that I would get that particular configuration? Before I received it, the odds would have been one in 175,760,000. (The total number of letters to the power of the number of letters on the plate times the total number of digits to the power of the number of digits on the plate: 263 x 104). After the fact, however, the probability is one.

This is what Pomona College economist Gary Smith calls the “survivor bias,” which he highlights as one of many statistically related cognitive biases in his deeply insightful book Standard Deviations (Overlook, 2014). Smith illustrates the effect with a playing card hand of three of clubs, eights of clubs, eight of diamonds, queen of hearts and ace of spades. The odds of that particular configuration are about three million to one, but Smith says, “After I look at the cards, the probability of having these five cards is 1, not 1 in 3 million.”

The conclusion seems obvious once you think about it, but most of us are regularly fooled by the survivor bias. Consider the plethora of business books readily available in airport bookstalls that feature the most successful companies. Smith analyzes two of the best sellers in the genre. In his 2001 book Good to Great (more than four million copies sold), Jim Collins culled 11 companies out of 1,435 whose stock beat the market average over a 40-year time span and then searched for shared characteristics among them that he believed accounted for their success. Instead, Smith says, Collins should have started with a list of companies at the beginning of the test period and then used “plausible criteria to select eleven companies predicted to do better than the rest. These criteria must be applied in an objective way, without peeking at how the companies did over the next forty years. It is not fair or meaningful to predict which companies will do well after looking at which companies did well! Those are not predictions, just history.” In fact, Smith notes, from 2001 through 2012 the stock of six of Collins’s 11 “great” companies did worse than the overall stock market, meaning that this system of post hoc analysis is fundamentally flawed.

Smith found a similar problem with the 1982 book In Search of Excellence (more three million copies sold), in which Tom Peters and Robert Waterman identified eight common attributes of 43 “excellent” companies. Since then, Smith points out, of the 35 companies with publicly traded stocks, 20 have done worse than the market average.

The survivor bias was evident in the reception of Walter Isaacson’s best-selling biography of Steve Jobs, as readers scrambled to understand what made the mercurial genius so successful. Want to be the next Steve Jobs and create the next Apple Computers? Drop out of college and start a business with your buddy in the garage of your parents’ home. How many people have followed the Jobs model and failed? Who knows? No one writes books about them and their unsuccessful companies. But venture capitalists (VCs) have data on the probability of a garage start-up becoming the Next Big Thing, and here the survivor bias is of a different sort.

David Cowan of Bessemer Venture Partners in Menlo Park, Calif., told me in an e-mail: “For garage-dwelling entrepreneurs to crack the 1% wealth threshold in America, their path almost always involves raising venture capital and then getting their start-up to an initial public offering (IPO) or a large acquisition by another company. If their garage is situated in Silicon Valley, they might get to pitch as many as 15 VCs, but VCs hear 200 pitches for every one we fund, so perhaps 1 in 13 start-ups get VC, and still they face long odds from there. According to figures that the National Venture Capital Association diligently collects through primary research and publishes on their Web site, last year was somewhat typical in that 1,334 start-ups got funded, but only 13% as many achieved an IPO (81 last year) or an acquisition large enough to warrant a public disclosure of the price (95 last year). So for every wealthy start-up founder, there are 100 other entrepreneurs who end up with only a cluttered garage.”

Surviving those statistical odds is rare indeed.

topics in this column: ,

6 Comments to “Surviving Statistics”

  1. Janeen Says:

    Thanks in favor of sharing such a nice thought, piece of writing is pleasant,
    thats why i have read it fully

    Look at my homepage … Cambogia Natural Reviews

  2. Tom Says:

    In these types of problem, which I’d put the Monty Hall one in too, I think it helps to think about at what point the probability wave collapses, which generally is an observation.

  3. Dave Says:

    While I agree with your point, your examples are flawed. The companies that were quoted as examples of good practice and later failed may have changed leadership and no longer exhibit the characteristics that made them great. A proper example would have had to select companies that had the supposedly great characteristics DURING the later period measured.

  4. Andrew Says:

    After 40 years in business and management, working with statistical process control and its various spin-offs, I have always suspected this bias at work. I referred to it as the “Look, I made it, you can too” fallacy. Survivor bias sounds good to me.

    Now, that I am enjoying a modicum of success, people ask me how did I do it? I just tell them that I beat the odds.

  5. Guh Says:

    So how to beat the odds? By praying. Only God grace that make it possible. So the religion salesman will say.

  6. Don Says:

    There are so many variables that can, and often do, come in to play with young start-ups that makes investing in one akin to playing Blackjack at your local casino. Only the brave of heart dare swim in that water.