In conducting a test of significance or hypothesis test there are two numbers that are easy to get confused. One number is called the p-value of the test statistic. The other number of interest is the level of significance, or alpha. These numbers are easily confused because they are both numbers between zero and one, and are in fact probabilities.
Alpha – The Level of Significance
The number alpha is the threshold value that we measure p values against. It tells us how extreme observed results must be in order to reject the null hypothesis of a significance test.
The value of alpha is associated to the confidence level of our test. The following lists some levels of confidence with their related values of alpha:
- For results with a 90% level of confidence, the value of alpha is 1 - 0.90 = 0.10.
- For results with a 95% level of confidence, the value of alpha is 1 - 0.95 = 0.05.
- For results with a 99% level of confidence, the value of alpha is 1 - 0.99 = 0.01.
- And in general, for results with a C% level of confidence, the value of alpha is 1 – C/100.
The alpha value gives us the probability of a type I error. Type I errors occur when we reject a null hypothesis that is actually true. Thus, in the long run, for a test with level of significance of 0.05 = 1/20, a true null hypothesis will be rejected one out of every 20 times.
The other number that is part of a test of significance is a p-value. A p-value is also a probability, but it comes from a different source than alpha. Every test statistic has a corresponding probability or p-value. This value is the probability that the observed statistic occurred by chance alone.
Since there are a number of different test statistics, there are a number of different ways to find a p-value. For some cases we need to know the probability distribution of the population.
The p-value of the test statistic is a way of saying how extreme that statistic is for our sample data. The smaller the p-value, the more unlikely the observed sample.
To determine if an observed outcome is statistically significant, we compare the values of alpha and the p -value. There are two possibilities that emerge:
- The p-value is less than or equal to alpha. In this case we reject the null hypothesis. When this happens we say that the result is statistically significant. In other words, we are reasonably sure that there is something besides chance alone that gave us an observed sample.
- The p-value is greater than alpha. In this case we fail to reject the null hypothesis. When this happens we say that the result is not statistically significant. In other words, we are reasonably sure that our observed data can be explained by chance alone.