A study tests 20 possible relationships and reports one result as statistically significant. That finding might be worth investigating, but it is not as surprising as it looks when viewed alone. When researchers ask many questions of the same data, random variation gets many chances to produce an apparently convincing answer. This is the multiple comparisons problem, also called multiple testing.
The issue is not that testing several ideas is forbidden, or that every surprising result is false. It is that a threshold designed for one test does not automatically describe the risk across a whole collection of tests. Understanding the difference helps readers judge scientific claims and helps researchers distinguish a promising lead from a result that has already been tested convincingly.
Why chance gets more opportunities
Imagine a researcher examining whether a new study technique affects 20 separate outcomes, from different test scores to measures of concentration. For a simplified example, suppose the technique actually affects none of them, each test is valid, and the tests are independent. Each test uses a 5% significance threshold: if its null hypothesis is true, it has a 5% chance of incorrectly rejecting that hypothesis under the assumptions of the test.
For one test, the chance of avoiding that particular false positive is 95%. For all 20 independent tests, the chance of avoiding every false positive is 0.95 multiplied by itself 20 times, or about 36%. That leaves approximately a 64% chance of at least one false positive. The expected number of false positives is one, but that does not mean every such study will produce exactly one: some will produce none and others more than one.
These numbers depend on the setup. Outcomes measured on the same people are often related, so their tests may not be independent; the 64% figure should not be applied mechanically to every study. And a 5% threshold does not mean that a significant result has a 5% probability of being false. It describes the long-run behavior of the testing procedure when a particular null hypothesis is true and the relevant assumptions hold.

The hidden collection of questions
Multiple testing is easy to see when a paper lists 20 outcomes. It can be harder to notice when researchers examine several age groups, try different definitions of an outcome, compare many combinations of variables, or look at results repeatedly as new observations arrive. Each decision can create another opportunity to notice a striking pattern. The relevant family of tests is the set of claims being considered together; defining it requires judgment about the research question and design.
Suppose the same study technique appears to help only one subgroup after dozens of subgroups have been inspected. That pattern can suggest a useful new hypothesis, but the fact that it stood out from many possibilities weakens the case for treating it as a confirmed effect. A result specified before the data were examined and a result discovered after a broad search deserve different interpretations, even when their reported p-values match.
This distinction does not make exploration bad science. Exploring data is often how researchers discover questions worth asking. The danger comes when a result selected from a large search is presented as though it had been the sole prediction all along, without showing the other analyses that were tried.
Ways to account for multiple tests
One approach is to decide in advance which outcome and comparison will answer the main question, then treat other analyses as secondary or exploratory. A preregistered analysis plan can make those choices visible, although registration by itself cannot guarantee a well-designed study. Reporting all relevant outcomes and explaining departures from the plan also gives readers a more complete picture.
Statistical adjustments provide another tool. The Bonferroni correction, for example, divides the desired overall error level by the number of tests. For 20 tests and an overall level of 0.05, the resulting threshold for each test is 0.0025. Provided the individual tests are valid, this method controls the probability of at least one false rejection across the family at no more than 5%, even when the tests are dependent.
That protection has a cost: a stricter threshold can make real effects harder to detect, especially with small samples. Bonferroni can also be conservative when tests are strongly related. Other methods make different tradeoffs. False discovery rate procedures aim to control the expected proportion of false rejections among rejected hypotheses, rather than the probability of making even one false rejection. Those are different guarantees, so the choice depends on the studyβs purpose and the consequences of errors. The U.S. Department of Educationβs Institute of Education Sciences discusses these choices in its guidelines for multiple testing in impact evaluations.

How to read a striking result
Start by asking what was tested, not just what made the headline. Was the result the primary outcome, or one of many measurements? Were the comparisons planned, and were all results reported? Did the researchers use an adjustment, and if so, what error rate did it address? These questions are useful whether the subject is a classroom intervention, a survey, or a study comparing thousands of biological measurements.
Then look beyond the significance label. An estimated effect and its uncertainty show more than a pass-or-fail threshold, while study design, measurement quality, and possible confounding affect whether the estimate supports a causal explanation. A very small p-value does not fix a biased sample or prove that an effect is practically large. Conversely, a result that does not cross a stricter threshold is not proof that the effect is exactly zero.
Finally, look for independent confirmation. A fresh study that tests a clearly stated prediction can help distinguish a repeatable pattern from one selected by chance, though replication is evidence rather than a guarantee. As the research organization Evidence in Governance and Politics explains in its guide to multiple comparisons, the risk comes from the collection of opportunities to find a result, not simply from the existence of many interesting questions.
The practical lesson is to keep the full search in view. One unusual result among many comparisons can be the start of an investigation; knowing how many chances it had to appear helps determine how much confidence it deserves.




Add comment