It is quite possible, and unfortunately quite common, for a result to be statistically significant and trivial. It is also possible for a result to be statistically nonsignificant and important.
Consider the case of a new drug that researchers hope will cure Alzheimer’s disease (Kirk 1996). They set up a trial study involving two groups each with 6 patients. One group receives the experimental treatment while the other receives a placebo. At the end of the trial they notice a 13 point improvement in the IQ of the treated group and no improvement in the control group. The drug seems to have an effect. However, the t statistic is statistically nonsignificant. The results could be a fluke. What to do?
Which of the following choices makes more sense to you:
(a) abandon the study – the result is statistically nonsignificant so the drug is ineffective
(b) conduct a larger study – a 13 point improvement seems promising
If you chose “a” you may have misinterpreted an inconclusive result as evidence of no effect. You may have confused statistical significance with substantive significance. Are you prepared to risk a Type II error when there is potentially much to be gained?
If you chose “b” then you clearly think it is possible for a result to be statistically nonsignificant yet important at the same time. You have distinguished statistical significance from substantive significance.
More here.
Effect Size FAQs 