1. Education

What Is the Difference Between Alpha and P-Values?

By

In conducting a test of significance or hypothesis test there are two numbers that are easy to get confused. One number is called the p-value of the test statistic. The other number of interest is the level of significance, or alpha. These numbers are easily confused because they are both numbers between zero and one, and are in fact probabilities.

Alpha – The Level of Significance

The number alpha is the threshold value that we measure p values against. It tells us how extreme observed results must be in order to reject the null hypothesis of a significance test.

The value of alpha is associated to the confidence level of our test. The following lists some levels of confidence with their related values of alpha:

  • For results with a 90% level of confidence, the value of alpha is 1 - 0.90 = 0.10.
  • For results with a 95% level of confidence, the value of alpha is 1 - 0.95 = 0.05.
  • For results with a 99% level of confidence, the value of alpha is 1 - 0.99 = 0.01.
  • And in general, for results with a C% level of confidence, the value of alpha is 1 – C/100.
Although in theory and practice many numbers can be used for alpha, the most commonly used is 0.05. The reason for this both because consensus shows that this level is appropriate, and historically it has been accepted as the standard.

The alpha value gives us the probability of a type I error. Type I errors occur when we reject a null hypothesis that is actually true. Thus, in the long run, for a test with level of significance of 0.05 = 1/20, a true null hypothesis will be rejected one out of every 20 times.

P-Values

The other number that is part of a test of significance is a p-value. A p-value is also a probability, but it comes from a different source than alpha. Every test statistic has a corresponding probability or p-value. This value is the probability that the observed statistic occurred by chance alone.

Since there are a number of different test statistics, there are a number of different ways to find a p-value. For some cases we need to know the probability distribution of the population.

The p-value of the test statistic is a way of saying how extreme that statistic is for our sample data. The smaller the p-value, the more unlikely the observed sample.

Statistical Significance

To determine if an observed outcome is statistically significant, we compare the values of alpha and the p -value. There are two possibilities that emerge:

  • The p-value is less than or equal to alpha. In this case we reject the null hypothesis. When this happens we say that the result is statistically significant. In other words, we are reasonably sure that there is something besides chance alone that gave us an observed sample.
  • The p-value is greater than alpha. In this case we fail to reject the null hypothesis. When this happens we say that the result is not statistically significant. In other words, we are reasonably sure that our observed data can be explained by chance alone.
The implication of the above is that the smaller the value of alpha is, the more difficult it is to claim that a result is statistically significant. On the other hand, the larger the value of alpha is the easier is it to claim that a result is statistically significant. Coupled with this, however, is the higher probability that what we observed can be attributed to chance.

©2014 About.com. All rights reserved.