Auditing an A/B test
A mobile game moved its first progression gate from level 30 to level 40 and measured retention across 90,189 players. This dataset has been analysed publicly dozens of times, and almost every analysis stops where the p-value appears. That is where the questions that decide whether a result means anything begin.
What the test says
Moving the gate later made both numbers worse. Only one of them is a result the experiment was equipped to report.
Returned the next day
NOT SIGNIFICANT-0.59pp
95% CI [-1.24, 0.06]
Underpowered for what it found. The smallest effect this sample could reliably detect is 0.93pp, and the difference measured 0.59pp. “Not significant” here means the experiment could not tell, not that nothing happened.
Returned after a week
SIGNIFICANT-0.82pp
95% CI [-1.33, -0.31]
The sample could detect an effect of 0.73pp or larger, and the difference measured 0.82pp — above the floor, but not by much.
What it could have detected
Day-one retention came back “not significant”, which is routinely read as “no difference”. It is not the same claim. Below the detection floor the experiment simply has no power to tell, and the effect it measured sits under that floor.
A TRUE EFFECT THIS BIG WOULD BE CAUGHT
15%
of the time, if the real difference were 0.30pp. Below half, a null result says almost nothing.
What checking early would have done
Teams watch experiments while they run and stop when the number turns green. Every experiment simulated below is an A/A test — both arms drawn from the same rate, nothing there to find — so every significant result counted is a mistake, produced by the number of looks alone.
IF NOTHING IS REALLY DIFFERENT
25.0%
of experiments would show a significant result at least once — when checked 20 times.
4.8% at one look · ×5.2
What I would report
- Keep the gate at level 30. Week-one retention is 0.82pp lower with the later gate, and that one is a real result — though it clears the detection floor by a margin thin enough that a replication would be worth having before treating it as settled.
- Do not report day-one as “no effect”. The experiment was never able to resolve a difference that size. The honest sentence is that it could not tell.
- Two metrics were tested, not one. Testing several outcomes and reporting the one that reached significance raises the error rate the same way repeated looks do. Worth naming even when the surviving result is the one that was pre-specified.
- Fix the stopping rule before the next test. Decide the sample size up front and look once, or adopt a sequential method built for looking often. Looking repeatedly under a fixed-sample test is the version that quietly fails.
Limits
- The 5% error rate and 80% power behind every figure here are conventions, not laws. A different pair moves the detection floor.
- The peeking simulation assumes evenly spaced checks and a fixed-sample test at each one. Real teams look at uneven times, which changes the exact numbers but not the direction.
- Retention is measured as a proportion. Total rounds played is in the dataset and would need different machinery — it is heavily skewed, so a test built for proportions does not apply to it.
- This audits a completed experiment. It cannot tell you why a later gate hurt retention, only that the data supports one of the two claims made.