Regression to the mean
After an unusually good result, the next one is usually a bit worse. After an unusually bad one, the next is usually a bit better. This happens whatever you do in between, and it is one of the most common reasons for crediting — or blaming — the wrong thing.
The idea
Almost any measured result is part stable quality and part luck: the day, the conditions, the questions that happened to come up. When you pick out the most extreme results, you pick people or things that are partly good and partly lucky. Next time, the quality is still there but the luck is fresh, and on average it is ordinary. So the group moves back towards the average. The pattern was first described in the 1880s, in a study comparing the heights of parents and their adult children.
The pulse that calmed down
In one classroom exercise described by the statistician Martin Bland, students took their pulse twice during the same class. The class average barely moved between the two readings, at about 73 beats per minute. But the students whose first reading was around 60 averaged about 66 on the second. Their pulses had not changed; the first reading had simply caught them at a low moment, and the second did not.
Now imagine that, between the two readings, the students with the lowest pulses had been given a drink of coffee. The rise would look like the coffee’s effect. The same trap appears in quality control: the worst-performing machines or shops this month get extra attention, and next month most of them improve — as they would have anyway. The only way to tell is a comparison group that was picked the same way and got no attention.
How to use it
- Before crediting a change, ask whether the group was chosen for being extreme.
- Estimate how much is luck: how well do two measurements of the same thing agree?
- Compare with a similar group that was chosen the same way and left alone.
- Expect the next result to land between the last result and the average.
- Judge people, teams and products on several periods, not on their best or worst one.
Where it misleads
Regression to the mean does not mean that everything is luck or that nothing ever changes. Real improvements and real declines happen, and dismissing every change as regression is its own mistake. The effect is about groups picked for being extreme, and it is strongest when results are noisy.
It is also a statement about averages, not individuals. Some of the top scorers will do even better the second time; the group as a whole will not.
In one line: extreme results are partly luck, and luck does not repeat.