the official site of Michael Shermer

top navigation:

Tag Results

Surviving Statistics

How the survivor bias distorts reality
magazine cover

When I purchased my latest vehicle, I was astonished to get the license plate 6NWL485. What are the chances that I would get that particular configuration? Before I received it, the odds would have been one in 175,760,000. (The total number of letters to the power of the number of letters on the plate times the total number of digits to the power of the number of digits on the plate: 263 x 104). After the fact, however, the probability is one.

This is what Pomona College economist Gary Smith calls the “survivor bias,” which he highlights as one of many statistically related cognitive biases in his deeply insightful book Standard Deviations (Overlook, 2014). Smith illustrates the effect with a playing card hand of three of clubs, eights of clubs, eight of diamonds, queen of hearts and ace of spades. The odds of that particular configuration are about three million to one, but Smith says, “After I look at the cards, the probability of having these five cards is 1, not 1 in 3 million.”

The conclusion seems obvious once you think about it, but most of us are regularly fooled by the survivor bias. Consider the plethora of business books readily available in airport bookstalls that feature the most successful companies. Smith analyzes two of the best sellers in the genre. In his 2001 book Good to Great (more than four million copies sold), Jim Collins culled 11 companies out of 1,435 whose stock beat the market average over a 40-year time span and then searched for shared characteristics among them that he believed accounted for their success. Instead, Smith says, Collins should have started with a list of companies at the beginning of the test period and then used “plausible criteria to select eleven companies predicted to do better than the rest. These criteria must be applied in an objective way, without peeking at how the companies did over the next forty years. It is not fair or meaningful to predict which companies will do well after looking at which companies did well! Those are not predictions, just history.” In fact, Smith notes, from 2001 through 2012 the stock of six of Collins’s 11 “great” companies did worse than the overall stock market, meaning that this system of post hoc analysis is fundamentally flawed.

Smith found a similar problem with the 1982 book In Search of Excellence (more three million copies sold), in which Tom Peters and Robert Waterman identified eight common attributes of 43 “excellent” companies. Since then, Smith points out, of the 35 companies with publicly traded stocks, 20 have done worse than the market average.

The survivor bias was evident in the reception of Walter Isaacson’s best-selling biography of Steve Jobs, as readers scrambled to understand what made the mercurial genius so successful. Want to be the next Steve Jobs and create the next Apple Computers? Drop out of college and start a business with your buddy in the garage of your parents’ home. How many people have followed the Jobs model and failed? Who knows? No one writes books about them and their unsuccessful companies. But venture capitalists (VCs) have data on the probability of a garage start-up becoming the Next Big Thing, and here the survivor bias is of a different sort.

David Cowan of Bessemer Venture Partners in Menlo Park, Calif., told me in an e-mail: “For garage-dwelling entrepreneurs to crack the 1% wealth threshold in America, their path almost always involves raising venture capital and then getting their start-up to an initial public offering (IPO) or a large acquisition by another company. If their garage is situated in Silicon Valley, they might get to pitch as many as 15 VCs, but VCs hear 200 pitches for every one we fund, so perhaps 1 in 13 start-ups get VC, and still they face long odds from there. According to figures that the National Venture Capital Association diligently collects through primary research and publishes on their Web site, last year was somewhat typical in that 1,334 start-ups got funded, but only 13% as many achieved an IPO (81 last year) or an acquisition large enough to warrant a public disclosure of the price (95 last year). So for every wealthy start-up founder, there are 100 other entrepreneurs who end up with only a cluttered garage.”

Surviving those statistical odds is rare indeed.

read or write comments (1)

Will E.T. Look Like Us?

Evolution helps us imagine what aliens might be like
magazine cover

What are the odds that intelligent, technically advanced aliens would look anything like the ones in films, with an emaciated torso and limbs, spindly fingers and a bulbous, bald head with large, almond-shaped eyes? What are the odds that they would even be humanoid? In this YouTube video, produced by Josh Timonen of the Richard Dawkins Foundation for Reason and Science, I argue that the chances are close to zero. Richard Dawkins himself made this interesting observation in a private communication after viewing it:

I would agree with [Shermer] in betting against aliens being bipedal primates, and I think the point is worth making, but I think he greatly overestimates the odds against. [University of Cambridge paleontologist] Simon Conway Morris, whose authority is not to be dismissed, thinks it positively likely that aliens would be, in effect, bipedal primates. [Harvard University biologist] Ed Wilson gave at least some time to the speculation that, if it had not been for the end-Cretaceous catastrophe, dinosaurs might have produced something like the attached [referring to paleontologist Dale A. Russell’s illustrated evolutionary projection of how a bipedal dinosaur might have evolved into a reptilian humanoid].

(continue reading…)

read or write comments (5)