Monday, March 14, 2011

The Results of Tetlock's Experiment

Again, from Future Babble from Dan Gardner:
Over many years, Tetlock and his team peppered the experts with questions. In all, they collected an astonishing 27,450 judgements about the future. It was by far the biggest exercise of its kind ever, and the results were startlingly clear. On “calibration,” the experts would have been better off making random guesses. Tetlock puts it a little more acidly: The experts would have been beaten by “a dart-throwing chimpanzee,” he says. On “discrimination,” however, the experts did a little better. They were still terrible, but not quite so terrible. When the scores for “calibration” and “discrimination” were combined, the experts beat the chimp by a whisker. Technically, at least. In practical terms, that whisker is irrelevant. The simple and disturbing truth is that the experts’ predictions were no more accurate than random guesses. Astrologers and psychics can make random guesses as well as Harvard professors so it’s hard not to look at these results and conclude that those who seek forecasts of the future would be well-advised to consult fortune cookies or the Mysterious Madam Zelda. They’re cheaper. And you can eat a fortune cookie. But that’s not Philip Tetlock’s conclusion. Serious skepticism about the ability of experts to predict the future is called for, he says. But just as important as the dismal collective showing of experts in his experiment is the wide variation among individual experts. “There’s quite a range. Some experts are so out of touch with reality, they’re borderline delusional. Other experts are only slightly out of touch. And a few experts are surprisingly nuanced and well-calibrated.

No comments:

Post a Comment