the official site of Michael Shermer

top navigation:
Scientific American

Opting Out of Overoptimism

published March 2012 | comments (10)
The willful distortion of reality to extremes
can be harmful
magazine cover

ARE YOU BETTER THAN AVERAGE AS A DRIVER? I know I am. I’ll bet 90 percent of you think you are, too, because this is the well-documented phenomenon known as the above-average effect, part of the psychology of optimism.

According to psychologist Daniel Kahneman, in his 2011 book Thinking, Fast and Slow, “people tend to be overly optimistic about their relative standing on any activity in which they do moderately well.” But optimism can slide dangerously into overoptimism. Research shows that chief financial officers, for example, “were grossly overconfident about their ability to forecast the market” when tested by Duke University professors who collected 11,600 CFO forecasts and matched them to market outcomes and found a correlation of less than zero! Such overconfidence can be costly. “The study of CFOs showed that those who were most confident and optimistic about the S&P index were also overconfident and optimistic about the prospects of their own firm, which went on to take more risk than others,” Kahneman notes.

Isn’t optimistic risk taking integral to building a successful business? Yes, to a point. “One of the benefits of an optimistic temperament is that it encourages persistence in the face of obstacles,” Kahneman explains. But “pervasive optimistic bias” can be detrimental: “Most of us view the world as more benign than it really is, our own attributes as more favorable than they truly are, and the goals we adopt as more achievable than they are likely to be.” For example, only 35 percent of small businesses survive in the U.S. When surveyed, however, 81 percent of entrepreneurs assessed their odds of success at 70 percent, and 33 percent of them went so far as to put their chances at 100 percent. So what? In a Canadian study Kahneman cites, 47 percent of inventors participating in the Inventor’s Assistance Program, in which they paid for objective evaluations of their invention on 37 criteria, “continued development efforts even after being told that their project was hopeless, and on average these persistent (or obstinate) individuals doubled their initial losses before giving up.” Failure may not be an option in the mind of an entrepreneur, but it is all too frequent in reality. High-risk-taking entrepreneurs override such loss aversion, a phenomenon most of us succumb to in which losses hurt twice as much as gains feel good that we developed in our evolutionary environment of scarcity and uncertainty.

This loss-aversion override by those with pervasive optimistic bias seems to work because of what I call biographical selection bias: the few entrepreneurs who succeed spectacularly have biographies (and autobiographies), whereas the many who fail do not.

Think Steve Jobs, whose pervasive optimistic bias was channeled through something a co-worker called Jobs’s “reality distortion field.” According to his biographer Walter Isaacson, “at the root of the reality distortion was Jobs’s belief that the rules didn’t apply to him…. He had the sense that he was special, a chosen one, an enlightened one.” Jobs’s optimism morphed into a reality-distorting will to power over rules that applied only to others and was reflected in numerous ways: legal (parking in handicapped spaces, driving without a license plate), moral (accusing Microsoft of ripping off Apple when both took from Xerox the idea of the mouse and the graphical user interface), personal (refusing to acknowledge his daughter Lisa even after an irrefutable paternity test), and practical (besting resource-heavy giant IBM in the computer market).

There was one reality Jobs’s distortion field optimism could not completely bend to his will: cancer. After he was diagnosed with a treatable form of pancreatic cancer, Jobs initially refused surgery. “I really didn’t want them to open up my body, so I tried to see if a few other things would work,” he admitted to Isaacson. Those other things included consuming large quantities of carrot and fruit juices, bowel cleansings, hydrotherapy, acupuncture and herbal remedies, a vegan diet, and, Isaacson says, “a few other treatments he found on the Internet or by consulting people around the country, including a psychic.” They didn’t work. Out of this heroic tragedy a lesson emerges: reality must take precedence over willful optimism. Nature cannot be distorted.

topics in this column: , , , , ,

10 Comments to “Opting Out of Overoptimism”

  1. Mel Says:

    One statement I would take issue with… Microsoft did rip off Apple. Though the Apple GUI was rooted in in innovation at the Parks lab, Microsoft was so concerned about the developments at Apple in the early nineties, after having visited the Apple labs, they did attempt to reverse engineer the Apple GUI with the first version of Windows.

  2. Tony Says:

    Unqualified assertions Mel. I have heard so many anti-Microsoft conspiracy theorists say so much that was simply not true that I am leery of the whole thing. That does not mean that Microsoft didn’t do anything wrong(I have no idea)…just that if you are going to accuse them of such then you have to provide some substantiation.

  3. Robert Sheaffer Says:

    The first version of Microsoft Windows was released in 1985. It had been in development since 1981 (see http://en.wikipedia.org/wiki/Windows_1.0 ). The Macintosh was first released in 1984. Sorry, this doesn’t support the claim that Microsoft reverse-engineered the Apple GUI.

  4. Charlie McClain, PhD, JD Says:

    I agree with Tony and Robert. Mel overstates the case. Having worked for Xerox and seen and been involved with the work at PARC (not “Parks”) I can attest to the fact that the best summary of what happened is contained in the book and motion picture “Pirates of Silicon Valley,” which I recommend to everyone.

    The truth is that many of the so-called “Pioneers” in IT, such as Jobs, Wozniak, Gates, Ratliff, and Ellison basically copied their products from someone else, such as Seattle Computer, or appropriated material which originated in the public sector, such as early database work done by NASA, in support of Apollo.

    The human ego appears to know few, if any, limits, ignoring the reality that epiphanies are far more rare than many would have us believe. IMHO, reality, like winning the lottery, involves far more luck than we are wont to admit.

    My two cents.

    Charlie McClain, PhD, JD

  5. Dallas Weaver Says:

    At least in the competitive private sector (excluding the crony capitalist sector), both overoptimism and excessive loss-aversion can be eliminated by bankruptcy. This automatic selection process results in a healthy balance allowing progress, creativity, innovation and economic growth.

    Within the government sector, we have overoptimism being rewarded when planning new dream programs. For example, the high speed rail system for California is a case of overoptimism, where the coming reality of automated automobiles driven by Moors law electronics will destroy fixed rails economic viability long before the bonds are paid off leaving the tax payers on the hook. Meanwhile the government bureaucrats live in luxury and will retire without paying any price for overoptimism.

    The government regulatory sector also rewards loss-aversion behavior of saying NO to every possible advance that may/could/possibly create a negative outcome with any finite probability, no mater how small the probability. Even in cases where there are unknown unknowns, applying the “precautionary principle” will prevent progress in the society. No matter how poor their decisions are or how much they prevent the creation of new jobs or solutions to major problems, they will live and retire very comfortably and never pay a price for saying NO to the future. A good example is the observation that offshore in Southern California could create a multi-billion dollar, tens of thousands of job class industry, but the governments will not issue permits, when saying NO is always the “safe” decision.

  6. Bad Boy Scientist Says:

    +Weaver

    Wow. I’m trying to decide if your post is a troll (or a parody) or if it is sincere. If it is sincere you really need to back up a lot of those statements. Although I agree with the assessment that the CA high speed rail plan is a case of overoptimism I doubt that automated automobiles will be the decisive factor (talk about overoptimism).

    Also the issue of regulation is a tricky one. First, having absolutely NO regulations would be disastrous (how many of you want your neighbor to work with U235 in his business he runs out of his garage?). Second, good regulations BENEFIT BUSINESS because they provide for security and trust between parties – the stock market would be worthless without careful regulation because no one would invest in it. Regulation helps basic transactions so you can be sure you are getting what you are paying for (BTW: contract law is a sort of regulation. If nothing else, it regulates the definitions of terms used for business contracts so both parties are ‘speaking the same language’).

  7. Daniel Says:

    One question that arises from this article is what level of optimism is useful and how do we know when optimism has crossed the boundary into overoptimism. Sometimes overoptimism is evident, like the article’s example of Jobs. However, sometimes the border between optimism and overoptimism is not as clear. In the words of Naom Chomsky, “Optimism is a strategy for making a better future. Because unless you believe that the future can be better, you are unlikely to step up and take responsibility for making it so.” So, the challenge lies in not falling into the trap of overoptimism, but also not mistaking a rational, although perhaps sometimes idealist, belief that a better future is possible. For example, it would be overly optimistic to believe war will be eliminated in the next 25 years. But how about the next 50 or 100 years?

  8. John Persichilli Says:

    I hope you are talking about the highest of government bureaucrats as I was a teacher on the Navajo reservation and I can inform you that I had trouble making ends meet.

    Starzdust

  9. awc Says:

    Knowing when someone is overoptimistic… a) it involves a complete break from reality b) when the limits of ones abilities are tested (my teen watches “failed” video’s)… this is all the more so when the consequences are dire.

    The CFO bit above has me recall many projects that people have started. Upon the outset it seemed to me they were too optimistic. When things are not even close to working out they do not think that the bar was hirer than achievable they just accept it as a cost of doing business and move on to the next project.

  10. Dukunocil Says:

    “Out of this heroic tragedy a lesson emerges: reality must take precedence over willful optimism. Nature cannot be distorted.”

    Shermer obviously have read nothing about zillion cases throughout the history, where people having faith and optimism survived. I just wish he would put his ‘rear’ on the line (like soldiers during the war) or faced deadly illness, to learn what is REALITY.