How the survivor bias distorts reality
When I purchased my latest vehicle, I was astonished to get the license plate 6NWL485. What are the chances that I would get that particular configuration? Before I received it, the odds would have been one in 175,760,000. (The total number of letters to the power of the number of letters on the plate times the total number of digits to the power of the number of digits on the plate: 263 x 104). After the fact, however, the probability is one.
This is what Pomona College economist Gary Smith calls the “survivor bias,” which he highlights as one of many statistically related cognitive biases in his deeply insightful book Standard Deviations (Overlook, 2014). Smith illustrates the effect with a playing card hand of three of clubs, eights of clubs, eight of diamonds, queen of hearts and ace of spades. The odds of that particular configuration are about three million to one, but Smith says, “After I look at the cards, the probability of having these five cards is 1, not 1 in 3 million.”
The conclusion seems obvious once you think about it, but most of us are regularly fooled by the survivor bias. Consider the plethora of business books readily available in airport bookstalls that feature the most successful companies. Smith analyzes two of the best sellers in the genre. In his 2001 book Good to Great (more than four million copies sold), Jim Collins culled 11 companies out of 1,435 whose stock beat the market average over a 40-year time span and then searched for shared characteristics among them that he believed accounted for their success. Instead, Smith says, Collins should have started with a list of companies at the beginning of the test period and then used “plausible criteria to select eleven companies predicted to do better than the rest. These criteria must be applied in an objective way, without peeking at how the companies did over the next forty years. It is not fair or meaningful to predict which companies will do well after looking at which companies did well! Those are not predictions, just history.” In fact, Smith notes, from 2001 through 2012 the stock of six of Collins’s 11 “great” companies did worse than the overall stock market, meaning that this system of post hoc analysis is fundamentally flawed.
Smith found a similar problem with the 1982 book In Search of Excellence (more three million copies sold), in which Tom Peters and Robert Waterman identified eight common attributes of 43 “excellent” companies. Since then, Smith points out, of the 35 companies with publicly traded stocks, 20 have done worse than the market average.
The survivor bias was evident in the reception of Walter Isaacson’s best-selling biography of Steve Jobs, as readers scrambled to understand what made the mercurial genius so successful. Want to be the next Steve Jobs and create the next Apple Computers? Drop out of college and start a business with your buddy in the garage of your parents’ home. How many people have followed the Jobs model and failed? Who knows? No one writes books about them and their unsuccessful companies. But venture capitalists (VCs) have data on the probability of a garage start-up becoming the Next Big Thing, and here the survivor bias is of a different sort.
David Cowan of Bessemer Venture Partners in Menlo Park, Calif., told me in an e-mail: “For garage-dwelling entrepreneurs to crack the 1% wealth threshold in America, their path almost always involves raising venture capital and then getting their start-up to an initial public offering (IPO) or a large acquisition by another company. If their garage is situated in Silicon Valley, they might get to pitch as many as 15 VCs, but VCs hear 200 pitches for every one we fund, so perhaps 1 in 13 start-ups get VC, and still they face long odds from there. According to figures that the National Venture Capital Association diligently collects through primary research and publishes on their Web site, last year was somewhat typical in that 1,334 start-ups got funded, but only 13% as many achieved an IPO (81 last year) or an acquisition large enough to warrant a public disclosure of the price (95 last year). So for every wealthy start-up founder, there are 100 other entrepreneurs who end up with only a cluttered garage.”
Surviving those statistical odds is rare indeed.