A small p-value is often read as a sign that “something is there”. This lab starts from the opposite case: nothing is there. Every simulated study compares two groups drawn from the same population. Even so, some of them reach . The experiment shows how many do, and how much that figure grows when the analysis adapts to the data.
What each study computes
Each study takes subjects per group, all distributed , and runs a two-sample pooled-variance test:
The p-value is the probability of observing a difference at least as extreme as the one obtained if the null hypothesis and the rest of the model were true. It is a statement about the data, conditional on a hypothesis. It is not a statement about the hypothesis.
Each study compares two groups drawn from the same normal distribution. Any “discovery” is, by construction, a false positive.
p < 0.05p ≥ 0.05Height if p were uniform
- Studies with p < 0.05
- —
- Simulated rate ± 2 SE
- —
- Theoretical rate
- —
- Seed
- —
p-hacking mode: try m outcomes, report the best
Simulated, best of m1 − 0.95m (independent outcomes)Simulated, Bonferroni
The histogram shows the p-value each study reports: the smallest of its m tests or, with Bonferroni, that value multiplied by m (capped at 1). Changing m or the correction reuses the same simulated subjects. Sliders for n and ρ re-simulate when released.
First 50 studies
| Study | Best outcome | Smallest p | Reported p |
|---|
Uniform under the null
When the null hypothesis is true and the test is exact, the p-value is uniformly distributed between 0 and 1:
That is why the histogram is flat and about 5 % of studies land in the first bar. That 5 % is the type I error rate accepted by setting . It is not a flaw of the test. With finite the histogram wobbles: the SE margin shows how far the simulated share can drift from 5 % through simulation noise alone.
Testing until it works
Now each study measures outcome variables, tests them all and reports the one with the smallest p-value. If the tests were independent, the probability that at least one crosses the threshold would be
With that is 22.6 %; with , 64.2 %. No calculation was wrong: each test, on its own, is correct. The problem is reporting only the winner as if it were the only one examined.
The correlation control makes the outcomes resemble one another. The more they overlap, the fewer independent chances there are and the smaller the inflation. It does not vanish unless the outcomes are practically the same variable.
The Bonferroni correction compares each test with . By Boole’s inequality,
whatever the dependence between outcomes. The price is power: a real effect has to be larger, or the sample bigger, to clear a stricter threshold. When outcomes are highly correlated, Bonferroni becomes conservative: with the simulated rate is close to 1 %.
Warning. does not mean the hypothesis has a 95 % probability of being true, nor that the result has a 5 % probability of being due to chance. The American Statistical Association’s statement says so explicitly: p-values do not measure the probability that the studied hypothesis is true. Nor do they measure the size of an effect or its importance. Talking about the probability of a hypothesis requires a different framework, one that includes how plausible it was before looking at the data.
What is illustrative and what is not. The data are simulated: normal, with equal variances and equally correlated outcomes. The false-positive rates shown here are exact for that model, not estimates for any real discipline. Flexibility in real analyses is more varied than choosing among outcomes: it includes adding observations until the result appears, adding or dropping covariates, excluding cases or redefining groups. Simmons, Nelson and Simonsohn showed by simulation that combining such decisions can raise the false-positive rate a great deal. Their central proposal is not a statistical correction but transparency: report every measured variable, every condition and how the sample size was decided.
References: Wasserstein and Lazar (2016), “The ASA Statement on p-Values: Context, Process, and Purpose”, The American Statistician 70(2) and Simmons, Nelson and Simonsohn (2011), “False-Positive Psychology”, Psychological Science 22(11).