Because it never turns out the way I want it, that confounded thing!

Seriously, the ** p value** is literally a confounded index because it reflects both the size of the underlying effect and the size of the sample. Hence any information included in the

*p*value is ambiguous (Lang et al. 1998).

Consider the following equation, which comes from Rosenthal and Rosnow (1984):

Statistical significance = Effect size x Sample size

Now let’s hold the effect size constant for a moment and consider what happens to statistical significance when we fiddle with the sample size (*N*). Basically, as *N* goes up, *p* will go down automatically. It has to. It has absolutely no choice. This is not a question of careful measurement or anything like that. It’s a basic mathematical equation. The bigger the sample, the more likely the result will be statistically significant, regardless of other factors.

Conversely, as *N* goes down, *p* must go up. The smaller the sample, the less likely the result will be statistically significant.

So if you happen to get a statistically significant result (a low *p* value), it could mean that (a) you have found something, or (b) you found nothing but your test was super-powerful because you had a large sample.

Researchers often confuse statistical significance with substantive significance. But smart researchers understand that *p* values should never be used to inform judgments about real world effects.