the official site of Michael Shermer

top navigation:
Scientific American

A Random Walk through Middle Land

published October 2008 | comments (57)
How randomness rules our world and why we cannot see it, Part 2
magazine cover

Imagine that you are a contestant on the classic television game show Let’s Make a Deal. Behind one of three doors is a brand-new automobile. Behind the other two are goats. You choose door number one. Host Monty Hall, who knows what is behind all three doors, shows you that a goat is behind number two, then inquires: Would you like to keep the door you chose or switch? Our folk numeracy — our natural tendency to think anecdotally and to focus on small-number runs — tells us that it is 50–50, so it doesn’t matter, right?

Wrong. You had a one in three chance to start, but now that Monty has shown you one of the losing doors, you have a twothirds chance of winning by switching. Here is why. There are three possible three-doors configurations: (1) good, bad, bad; (2) bad, good, bad; (3) bad, bad, good. In (1) you lose by switching, but in (2) and (3) you can win by switching. If your folk numeracy is still overriding your rational brain, let’s say that there are 10 doors: you choose door number one, and Monty shows you door numbers two through nine, all goats. Now do you switch? Of course, because your chances of winning increase from one in 10 to nine in 10. This type of counterintuitive problem drives people to innumeracy, including mathematicians and statisticians, who famously upbraided Marilyn vos Savant when she first presented this puzzle in her Parade magazine column in 1990.

The “Monty Hall Problem” is just one of many probability puzzles that physicist Leonard Mlodinow of the California Institute of Technology presents in his delightfully entertaining new book The Drunkard’s Walk (Pantheon, 2008). His title employs the metaphor (sometimes called the “random walk”) to draw an analogy between “the paths molecules follow as they fly through space, incessantly bumping, and being bumped by, their sister molecules,” and “our lives, our paths from college to career, from single life to family life, from first hole of golf to eighteenth.” Although countless random collisions tend to cancel one another out because of the law of large numbers — where improbable events will probably happen given enough time and opportunity — every once in a great while, “when pure luck occasionally leads to a lopsided preponderance of hits from some particular direction … a noticeable jiggle occurs.” We notice the improbable directional jiggle but ignore the zillions of meaningless and counteracting collisions.

In the Middle Land of our ancient evolutionary environment, which I introduced in Part 1 of this column last month, our brains never evolved a probability network, and thus our folk intuitions are ill equipped to deal with many aspects of the modern world. Although our intuitions can be useful in dealing with other people and social relationships (which evolved as common and important for a social primate species such as ours when we were struggling to survive in the harsh environs of the Paleolithic), they are misleading when it comes to such probabilistic problems as gambling. Let’s say you are playing the roulette wheel and you hit five reds in a row. Should you stay with red because you are on a “hot streak,” or should you switch because black is “due”? It doesn’t matter, because the roulette wheel has no memory, yet gamblers notoriously employ both the “hot streak fallacy” and the “dueness fallacy,” much to the delight of casino owners.

Additional random processes and our folk numeracy about them abound. The “law of small numbers,” for example, causes Hollywood studio executives to fire successful producers after a short run of box-office bombs, only to discover that the subsequent films under production during the producer’s reign became blockbusters after the firing. Athletes who appear on Sports Illustrated’s cover typically experience career downturns, not because of a jinx but because of the “regression to the mean,” where the exemplary performance that landed them on the cover is itself a low probability event that is difficult to repeat.

Extraordinary events do not always require extraordinary causes. Given enough time, they can happen by chance. Knowing this, Mlodinow says, “we can improve our skill at decision making and tame some of the biases that lead us to make poor judgments and poor choices … and we can learn to judge decisions by the spectrum of potential outcomes they might have produced rather than by the particular result that actually occurred.” Embrace the random. Find the pattern. Know the difference.

topics in this column: , ,

57 Comments to “A Random Walk through Middle Land”

  1. Jim Says:

    I teach high school math. Here’s the explanation that best helps my students understand that there is a 2/3 chance of winning by switching.

    The probability of picking the right door at the beginning is 1/3. In other words, most of the time, you’d lose this game if it ended here. Keep that in mind — most likely, you picked a loser. Well, if you most likely picked a lsoer, why not start with a different door? — because no matter which door we choose, it’s most likely a loser. Remember: YOUR ORIGINAL CHOICE IS MOST LIKELY WRONG. Now, once one of the other doors is shown to be a loser, and you know YOUR ORIGINAL CHOICE IS MOST LIKELY WRONG, the other remaining door is most likely the winner.

  2. STEVE Says:

    The Monty Hall problem is not as complicated as you and others make it. The following reasoning refutes the answer of 2/3 that you give.

    Clearly, at the beginning the contestent has a 1/3 chance of picking the right door when he picks #1. And when Monty Hall shows him that a goat is behind door #2 he has in effect changed the probability space from 3 to 2 possibilities. So when presented a chance to change his choice the contestent has in effect started over, i.e. it’s as though there was never a door #2. He now has 2 possible choices so he clearly has improved his chances of picking the right door from 1/3 to 1/2.

    As is often the case according to Occam’s razor, the simplest solution/explanation is usually the right one. Many things in life and mathematics are counterintutive but this is not one of them.

  3. Maurice le difficile Says:

    I finally got it. I looked at it as a contest between my original card and the rest of the deck. No matter what I choose, I am a 51 to 1 underdog. The “rest of the deck” will beat me 51 times out of 52. Once Monty the dealer gets rid of 50 certain losers, the remaining card will win 51 times out of 52. Switch!

    I have to admit that it was very hard for me to accept this at the three door level, but unless you picked the right door (only once every three attempts), Monty basically tells you where it is! So this is how I inferred that Monty tells you where the new car is two out of three times, giving legitimacy to Dr. Shermer’s Arument.

  4. Maurice le difficile Says:

    “Suppose there are two players, one taking door A and one taking door C. Door B is then opened and shows no prize. Should both players switch to both increase their odds?”

    When there are two players, they both have a 50/50 chance right from the start. They pick, Monty reveals a goat, and someone wins.
    Perhaps what makes all this so hard to fathom is the fact that your odds don’t really “change”. Your odds of winning are based on the rules of the game, not so much your initial guess, which means nothing, since you have to choose again after seeing a goat. What I mean is that your odds of winning on the first try are zero. You may pick the right door, but you haven’t won the prize yet, because the game is not over.

  5. Frank Rommey Says:

    How the BELIEF BIAS works in this case, is “by making counterintuitive” the fact that BEFORE you made your choice each DOOR had a 33% possibility of having a car, but AFTER you made your choice, and EVEN BEFORE THE HOST SHOWS A GOAT, the possibility of you door having a Car increased to 50%, as it will play against the REMAINING door, BECAUSE you know one of the other two doors will have a Goat and will be eliminated.
    Michael Shermer article in itself is a demonstration of how easy it is to fall into a Belief Bias…

    Let’s the possibilities:
    Alternative 1, first choice A, C shows a goat, A has the car.
    Door A / Door B / Door C
    Car / Goat / Goat n=1 (50%)
    Goat / Car / Goat n=2 (50%)
    Goat / Goat / Car n=3 (0%, we know C doesn’t have a car)
    Not switching wins. Switching looses.
    Alternative 2, first choice A, C shows a goat, B has the car.
    Door A / Door B / Door C
    Car / Goat / Goat n=1 (50%)
    Goat / Car / Goat n=2 (50%)
    Goat / Goat / Car n=3 (0%, we know C doesn’t have a car)
    Switching wins. Not switching looses.
    Alternative 3, first choice A, B shows a goat, A has the car
    Door A / Door B / Door C
    Car / Goat / Goat n=1 (50%)
    Goat / Car / Goat n=2 (0%, we know B doesn’t have a car)
    Goat / Goat / Car n=3 (50%)
    Not switching wins. Switching looses.
    Alternative 4, first choice A, B shows a goat, C has the car
    Door A / Door B / Door C
    Car / Goat / Goat n=1 (50%)
    Goat / Car / Goat n=2 (0%, we know B doesn’t have a car)
    Goat / Goat / Car n=3 (50%)
    Switching wins. Not switching looses.
    Alternative 5, first choice B, C shows a goat, B has the car.
    Door A / Door B / Door C
    Car / Goat / Goat n=1 (50%)
    Goat / Car / Goat n=2 (50%)
    Goat / Goat / Car n=3 (0%, we know C doesn’t have a car)
    Not switching wins. Switching looses.
    Alternative 6, first choice B, C shows a goat, A has the car.
    Door A / Door B / Door C
    Car / Goat / Goat n=1 (50%)
    Goat / Car / Goat n=2 (50%)
    Goat / Goat / Car n=3 (0%, we know C doesn’t have a car)
    Switching wins. Not switching looses.
    Alternative 7, first choice B, A shows a goat, B has the car
    Door A / Door B / Door C
    Car / Goat / Goat n=1 (0%, we know A doesn’t have a car)
    Goat / Car / Goat n=2 (50%)
    Goat / Goat / Car n=3 (50%)
    Not switching wins. Switching looses.
    Alternative 8, first choice B, A shows a goat, C has the car
    Door A / Door B / Door C
    Car / Goat / Goat n=1 (0%, we know A doesn’t have a car)
    Goat / Car / Goat n=2 (50%)
    Goat / Goat / Car n=3 (50%)
    Switching wins. Not switching looses.
    Alternative 9, first choice C, B shows a goat, C has the car.
    Door A / Door B / Door C
    Car / Goat / Goat n=1 (50%)
    Goat / Car / Goat n=2 (0%, we know B doesn’t have a car)
    Goat / Goat / Car n=3 (50%)
    Not switching wins. Switching looses.
    Alternative 10, first choice C, B shows a goat, A has the car.
    Door A / Door B / Door C
    Car / Goat / Goat n=1 (50%)
    Goat / Car / Goat n=2 (0%, we know B doesn’t have a car)
    Goat / Goat / Car n=3 (50%)
    Switching wins. Not switching looses.
    Alternative 11, first choice C, A shows a goat, C has the car
    Door A / Door B / Door C
    Car / Goat / Goat n=1 (0%, we know A doesn’t have a car)
    Goat / Car / Goat n=2 (50%)
    Goat / Goat / Car n=3 (50%)
    Not switching wins. Switching looses.
    Alternative 12, first choice C, A shows a goat, B has the car
    Door A / Door B / Door C
    Car / Goat / Goat n=1 (0%, we know A doesn’t have a car)
    Goat / Car / Goat n=2 (50%)
    Goat / Goat / Car n=3 (50%)
    Switching wins. Not witching looses.
    Not Switching wins alternatives 1, 3, 5, 7, 9, and 11 = 6 out of 12. Not Switching looses alternatives 2, 4, 6, 8, 10, and 12 = 6 out of 12.
    50/50
    The result is the same for any number of door with only one car (see simulator)… Can you find more alternatives? If the percentages I left showing throw you off, forget them, and look at the possible configurations, there aren’t others, so there are only twelve alternatives and not switching wins in 6 of them. In four of them the car is in A, in another four it is in B, and in the remaining four you find the car in C. Conversely, you chose A 4 times, B 4 times, and C four times. You will find that this mathematics cannot be blown by any bullet you might try.
    Not switching has 6 wins of 12 alternatives, making it 50/50.
    Now, why would you believe that not switching would only win in 4 alternatives and loose in the other eight? Belief bias, that’s why.

  6. Frank Rommey Says:

    Final mathematical explanation: The probability that I had chosen the door (one of n doors) with the car changes (increases) with every losing door that it is open: Started as 1/n, one door opens, it goes to 1/(n-1), the second door opens, …it changes again to 1/(n-2), and so on, until it becomes 1/(n-[n-2])=1/2.
    The minor premise is that only loosing doors are open. The major premise is that there is only one door with a car behind. The mechanism is that (n-2) doors will be open. The alternatives that we will find after the offer to switch is made, are 2xn! for our n choices… but the result is always that half of them will make a not switching a looser, while the other half will make not switching a winner. Properties of series affecting our statistical belief… LOL
    Irrational belief? I think that Shermer would have something to say about it… ;-)

  7. Marley52 Says:

    “..the fact that BEFORE you made your choice each DOOR had a 33% possibility of having a car, but AFTER you made your choice, and EVEN BEFORE THE HOST SHOWS A GOAT, the possibility of you door having a Car increased to 50%”

    This statement has got to be one of the most ridiculous I’ve read, shows a complete ignorance of probability theory. After that I couldn’t bring myself to read the rest of Frank’s comment