A p-value describes how unusual a result at least as extreme as the observed result would be under a specified statistical model. It is not the probability that a hypothesis is true.
Web edition · 14 October 2026
Do not put the entire judgement into one number
The ASA statement explains that a p-value does not measure effect size or scientific importance. Passing a threshold does not remove problems with research design, measurement or reporting. A small difference may be statistically clear in a large sample. Failing to find a significant result in a small sample does not establish that no difference exists.
Expand the statement of results
Before “p < 0.05”, explain the comparison and effect: which groups, which measurement and which units? Then add uncertainty and the analysis approach. Selecting only positive results for reporting prevents readers from seeing the actual research process.
A checking card
Were the hypothesis and analysis specified in advance?
How many different comparisons were made?
Are the effect size and uncertainty reported?
Are excluded data and analysis decisions explained?
Could the result be tested with another design or dataset?
This card does not produce a quality score. It makes missing information visible and provides a starting point for rereading the methods and supplementary materials.
Source check / web edition: 14 October 2026. Historical documents retain their own original dates.
The same p-value, different effect sizes
The two results below are entirely fictional. They assume a two-sided normal-approximation test against zero, known standard errors and the same measurement unit. This is not a study finding; it explains why a p-value alone does not describe effect magnitude.
Write the estimated difference and its measurement unit first. Do not call a number important without identifying what it compares.
Record the standard error and model. A standard error is not the same thing as the standard deviation of individual observations.
For a zero difference, calculate z = estimate / standard error. The same ratio can arise from estimates with very different magnitudes.
Read the approximate interval alongside the design and practical context. The normal approximation used here is not appropriate for every dataset or study design.
A · 0.2 ± 0.196 · B · 2 ± 1.96 · z = 2 · p ≈ 0.0455
Field
Example or check
Fictional A
Difference 0.2 units; standard error 0.1. z = 2; two-sided normal p ≈ 0.0455.
Fictional B
Difference 2 units; standard error 1. z = 2; the same p ≈ 0.0455. Both estimate and uncertainty are ten times A’s.
Approximate 95% intervals
Estimate ± 1.96 × standard error: A [0.004, 0.396], B [0.04, 3.96] units. These are Wald intervals under the stated assumptions.
Interpretation limit
The same p does not mean the same effect or practical importance. Nor is p the probability that the null hypothesis is true.
Result card
Comparison __; unit __; estimate __; standard error __; model __; interval __; practical meaning __; limitations __.
The ASA’s 2016 statement separates effect size from statistical significance. This new exercise is not attributed to earlier PiVOLKA authors and is not a clinical decision tool.
