Why Forecasts Miss: Lessons From Tetlock's Forecasting Tournaments

Every plan rests on forecasts: next year's sales, a competitor's next move, when a supplier will recover. Many of those forecasts miss, and most organizations never find out by how much, because nobody wrote the forecast down clearly enough to check.

Research led by the psychologists Philip Tetlock and Barbara Mellers at the University of Pennsylvania has done a great deal to show that forecasting can be measured and improved. Their team competed in a series of forecasting tournaments sponsored by the U.S. intelligence community, and the findings carry over surprisingly well to ordinary business questions.

This guide covers why forecasts miss, what the tournaments found, and a simple way to put those lessons to work.

Vague words hide disagreement

One reason forecasts fail is that they are never really made. Words like "possible," "likely" and "a real risk" feel like predictions but can mean almost anything.

The classic illustration comes from Sherman Kent, writing for the CIA in his essay Words of Estimative Probability. A 1951 National Intelligence Estimate said an attack on Yugoslavia in 1951 "should be considered a serious possibility." When a State Department official asked what odds that meant, Kent said his own estimate was around 65 to 35 in favor of an attack. The official had read it as much lower. When Kent asked his colleagues who had approved the wording, the low man was thinking of about 20 to 80 and the high of 80 to 20.

Kent's proposed fix was to tie words to numbers. In the chart he published, "almost certain" meant about 93 percent, "probable" about 75 percent, "chances about even" about 50 percent, "probably not" about 30 percent and "almost certainly not" about 7 percent, each with a stated margin.

Decades later, Tetlock, Mellers and Scoblic made a similar point in a 2017 article in Science, warning against "vague-verbiage predictions that make it difficult to assess accuracy." The lesson for any team is simple: if a forecast matters, put a number and a date on it.

What the tournaments tested

The U.S. Intelligence Advanced Research Projects Activity (IARPA) ran a program called Aggregative Contingent Estimation (ACE). Its stated goal was to "dramatically enhance the accuracy, precision, and timeliness of intelligence forecasts" by developing techniques that "elicit, weight, and combine the judgments of many intelligence analysts."

According to Chang, Chen, Mellers and Tetlock (2016), five university-based teams competed to submit the most accurate daily probability forecasts. Tetlock and Mellers's team was the Good Judgment Project. The tournaments ran in four rounds between September 2011 and May 2015, on questions about geopolitical and economic events.

Training, teaming and tracking

A 2014 paper in Psychological Science by Mellers and colleagues reported three drivers of accuracy in the team's first two years:

• Training: probability training that, according to the paper, corrected cognitive biases, encouraged forecasters to use reference classes and taught heuristics such as averaging when multiple estimates were available.
• Teaming: letting forecasters share information and discuss the reasons behind their beliefs.
• Tracking: placing the top performers, the top 2 percent from Year 1, together in elite teams.

The paper reports that all three improved accuracy, and that combining these behavioral methods with statistical aggregation produced the best forecasts two years in a row.

A 2015 follow-up in Perspectives on Psychological Science looked at these top performers, the "superforecasters." It reports that they kept up high accuracy across hundreds of questions, defying expectations of regression toward the mean, and found support for four explanations: cognitive abilities and styles, task-specific skills, motivation and commitment, and enriched environments. The authors conclude that superforecasters are "partly discovered and partly created."

Another 2015 paper, in the Journal of Experimental Psychology: Applied, profiled the best forecasters. Along with ability and knowledge, it highlighted open-mindedness, deliberation time and frequency of belief updating, and noted that the best forecasters "viewed forecasting as a skill that required deliberate practice."

A short training made a measurable difference

The most practical finding for businesses may be how little training it took. Chang and colleagues report that a training module of less than one hour "consistently improved accuracy (Brier scores) by 6 to 11% over the control condition."

The module was summarized by the acronym CHAMPS KNOW. Its probabilistic reasoning principles were:

• Comparison classes
• Hunt for the right information
• Adjust and update forecasts when appropriate
• Mathematical and statistical models
• Post-mortem analysis
• Select the right questions to answer

The KNOW part covered reasoning about the situation itself: knowing the power players and their preferences, the norms and protocols of institutions, other perspectives, and wildcards, accidents and black swans. In the paper's analysis of forecasters' self-reported reasoning, using comparison classes was associated with better accuracy.

In business terms, comparison classes mean starting from how similar cases usually turn out before considering what makes your case special. Before forecasting your new product's first-year sales, ask how your last several launches, or comparable launches in your industry, actually did.

Keep score

You cannot improve what you do not measure. The tournaments used the Brier score. As Chang and colleagues describe it, the score sums the squared differences between what happened, coded 1 if the event occurred and 0 if not, and the probability the forecaster assigned. In the original formulation used there, scores range from 0 to 2, and lower is better.

Here is a hypothetical worked example. You forecast a 70 percent chance that a key contract will be signed by June 30, which leaves 30 percent for "not signed." If it is signed, your score is (1 minus 0.7) squared plus (0 minus 0.3) squared, or 0.09 plus 0.09, which equals 0.18. If it is not signed, your score is 0.49 plus 0.49, or 0.98. Confident forecasts that turn out right score best; confident forecasts that turn out wrong are penalized heavily.

Putting it to work in a team

You do not need a tournament to borrow these ideas. A practical routine looks like this:

1. Write forecastable questions with a clear outcome and a deadline, such as "Will we close five new accounts above $10,000 by March 31?"
2. Ask each person to give a probability independently before discussion, then share reasons and allow updates.
3. Average the individual estimates as a starting point; the 2014 paper taught averaging when multiple estimates were available.
4. Start from a comparison class, then adjust for specifics.
5. Update as news arrives instead of waiting for a quarterly review.
6. Record outcomes and scores, and hold a short post-mortem on the biggest misses.

A useful self-check over time is whether the things you call 70 percent likely happen about seven times in ten. If they happen far less often, you are overconfident.

Key takeaways

• Vague probability words hide real disagreement; attach numbers and dates to forecasts that matter.
• In the IARPA-sponsored tournaments, training, teaming and tracking all improved forecasting accuracy.
• A training session of under an hour improved Brier scores by 6 to 11 percent over a control group in one study.
• Start from comparison classes, update often and run post-mortems on misses.
• Keep score, because feedback is what turns forecasting into a skill.

Sources

• Sherman Kent, Words of Estimative Probability, Central Intelligence Agency
• Intelligence Advanced Research Projects Activity, Aggregative Contingent Estimation (ACE)
• Mellers et al., Psychological strategies for winning a geopolitical forecasting tournament, Psychological Science (2014)
• Mellers et al., Identifying and cultivating superforecasters as a method of improving probabilistic predictions, Perspectives on Psychological Science (2015)
• Mellers et al., The psychology of intelligence analysis: drivers of prediction accuracy in world politics, Journal of Experimental Psychology: Applied (2015)
• Chang, Chen, Mellers and Tetlock, Developing expert political judgment, Judgment and Decision Making (September 2016)
• Tetlock, Mellers and Scoblic, Bringing probability judgments into policy debates via forecasting tournaments, Science (2017)