Statistical significance tests can only be used to inform judgments regarding whether the null hypothesis is false or not false.
This arrangement is similar to the judicial process that determines whether a defendant is guilty or not guilty. Defendants are presumed innocent; therefore, they cannot be found innocent. Similarly, a null hypothesis is presumed to be true unless the result of a statistical test suggests otherwise (Nickerson 2000).
This is not to say that statistical significance testing is worth keeping, for there are better means for gauging the importance, certainty, replicability and generality of a result (from Armstrong 2007):
– importance can be gauged by interpreting effect sizes
– certainty can be gauged by estimating confidence intervals
– replicability can be gauged by doing replication studies
– generality can be gauged by running meta-analyses
Effect Size FAQs 