What does a statistical significance test actually tell us?

Statistical significance tests can only be used to inform judgments regarding whether the null hypothesis is false or not false.

This arrangement is similar to the judicial process that determines whether a defendant is guilty or not guilty. Defendants are presumed innocent; therefore, they cannot be found innocent. Similarly, a null hypothesis is presumed to be true unless the result of a statistical test suggests otherwise (Nickerson 2000).

This is not to say that statistical significance testing is worth keeping, for there are better means for gauging the importance, certainty, replicability and generality of a result (from Armstrong 2007):

– importance can be gauged by interpreting effect sizes
– certainty can be gauged by estimating confidence intervals
– replicability can be gauged by doing replication studies
– generality can be gauged by running meta-analyses

This entry was posted on Sunday, May 30th, 2010 at 11:49 pm and is filed under statistical significance. You can follow any responses to this entry through the RSS 2.0 feed.
Both comments and pings are currently closed.

“The primary product of a research inquiry is one or more measures of effect size, not p values.”
~ Jacob Cohen

“Statistical significance is the least interesting thing about the results. You should describe the results in terms of measures of magnitude – not just, does a treatment affect people, but how much does it affect them.”
~ Gene Glass