“The difference is statistically significant” is a phrase that appears in reports, A/B tests and research papers. It is also one of the most misunderstood ideas in statistics. Here is what it means in plain words.
The basic idea
A hypothesis test starts from a default assumption, the null hypothesis, which says there is no real effect. For example: “The new landing page converts the same as the old one.” Then you ask: if that were true, how likely is the result we observed?
What the p-value is
The p-value is the probability of seeing a result at least as extreme as yours, assuming the null hypothesis is true. A p-value of 0.03 means: if there were really no difference, a gap this large would appear only about 3% of the time by chance.
What it is not
- It is not the probability that the null hypothesis is true.
- It does not measure how big or important the effect is.
- A result with p = 0.04 is not proof; it is one piece of evidence.
A business example
Enquiry rates were 4.0% on the old page and 5.2% on the new page, from 500 visitors each. A two-proportion test gives a p-value. If it is small, say below 0.05, the difference is unlikely to be pure chance. Check the sample size too: small samples produce unstable results.
In Python
from scipy import stats
t_stat, p_value = stats.ttest_ind(group_a, group_b)
Choose the test that matches your data: t-tests for averages, chi-square or two-proportion tests for rates and categories.
Four habits
- Decide the test and the threshold before you look at the data.
- Report the size of the effect, not just the p-value.
- Use confidence intervals to show a range of likely values.
- Remember that many tests on the same data raise the chance of a false positive.
Related: statistics for data analysts and cross-validation in plain words.
Statistics for analysis is part of our Data Science course in Bangalore.
Book a free demo class at our BTM Layout centre, and ask a counsellor for batch details.
