Pre-Registering an A/B Test: Write the Analysis Before You Look

An analysis you decide on after seeing the numbers is not an analysis. It's a search.

This is the failure mode nobody defends in principle and almost everybody commits in practice. The test finishes, the primary metric is flat, and someone says "well, it was significant for mobile users in week two." Maybe it was. But you're now reporting the most interesting slice of a dataset you went looking through, which is a different thing from a result, and it will not replicate.

The fix is boring and it works: write down what you're going to measure, and how you'll read it, before you have any data. Then read it that way.

I've argued this for years inside experimentation programmes. Recently I got to do it fully in the open, on a product I own outright, and publish the pre-registration alongside the thing being tested. This post is that worked example.

The product, briefly

I built a browser game called ATS Defence. It's a tower defence game where you play the applicant tracking system, and the applicants are the problem.

You place screening mechanisms along a path — Keyword Filter, Knockout Question, Take-Home Task, Culture Fit Panel — and try to reject everybody before they reach the open vacancy. Anyone who gets through costs you a life, because now a human has to read their application properly and might hire them. The applicants know the game too: the Keyword Stuffer is immune to keyword matching, the Referral starts a third of the way down the path, and the Boomerang comes back at the end of the intake whether you rejected it or not.

The joke is that you're the one doing the rejecting. But underneath it is a real product with real instrumentation — thirteen events landing in Supabase — and that meant I could run a proper test on it.

The question

Wave one of the game is five applicants over about eight seconds. That's quiet. Quiet enough that a new player might reasonably conclude nothing is happening and close the tab before the game has shown them anything at all.

So: does opening with a busier first wave change how far players get, and how many of them leave in the first few minutes?

Two arms, split 50/50:

• Control — 5 applicants, 1600ms apart.
• Busy — 9 applicants 900ms apart, then 5 more at 700ms from seven seconds.

Waves two onwards are identical in both. The test is about how the game opens, not how it goes on. Assignment is hashed against a stable anonymous id, so a returning player stays in the arm they started in, and it's fixed at start-up so it cannot move under a run.

What I committed to before launch

Three metrics, each doing a different job.

Primary: the survival curve. For each arm, the proportion of runs that reached wave N, for N from one to ten. Deliberately read as a curve, not a single number — the interesting shape is whether the arms separate early and then converge, which would mean the busy opening costs players at the start and nothing afterwards.

Secondary: early abandonment. The proportion of runs that quit at wave three or earlier, having actually left rather than tabbed away.

Guardrail: reaching the end. The proportion of runs that finish all ten waves. This one exists to catch a specific way of winning that isn't a win — a busier opening could improve retention simply by making the game easier to lose early, and I did not want to celebrate that.

Guardrails are the part teams skip most often, and they're cheap. One extra metric whose only job is to tell you that your headline number came from somewhere you didn't intend.

The part most people leave out

Then I did the power calculation, and it told me not to bother.

Assume early abandonment sits somewhere near 30%. To detect a 10 percentage point difference, at 80% power and 5% significance, you need roughly 340 assigned runs per arm — about 680 in total.

A launch post on LinkedIn is not going to produce 680 runs. If it produces 200, this experiment can only detect a difference of around 20 percentage points, which is a gap so large you'd see it without a test.

So I wrote that down. Before launch, in the same document as everything else: the likely honest outcome here is "inconclusive, and here is the confidence interval", and that is what will be reported.

This matters more than it looks. An underpowered test that nobody has acknowledged is underpowered is exactly the setup that produces a fake result, because when the primary metric comes back flat — as it was always going to — the temptation is to go hunting for something that isn't. Naming the limitation in advance removes the incentive. There's nothing to salvage, because nothing was promised.

The stopping point went in too: 340 runs per arm or fourteen days, whichever comes first. No interim peeking. The numbers get looked at once.

Instrumentation earns its keep before the test does

Here's the bit I didn't expect to be the most valuable.

The very first real run through the collector produced an abandonment event at wave five — from a player who went on to reach wave eight. They hadn't abandoned anything. The event had fired the moment they glanced at another tab. And because it fires at most once per run, their actual exit was then never recorded at all.

Left alone, that would have quietly ruined the secondary metric. Not loudly — quietly, which is worse. The number would have existed, looked plausible, and been wrong.

Hiding the tab now starts a thirty-second clock that cancels if you come back, and the reason for leaving is recorded, so a departure can be told apart from a glance. Same fix you'd make on any product; the only reason I caught it is that I'd defined the metric precisely enough to notice the event didn't match it.

Runs that weren't genuinely assigned get labelled rather than folded into control, too — a preview run reads as forced:busy, a failed assignment as unassigned:control, and every analysis excludes them. Quietly counting your own testing as control data is an easy way to poison an arm.

If you can't do this at work, do it somewhere

Plenty of analysts can't point at their pre-registrations. The document lives in a client's Confluence, the figures are commercially sensitive, and the honest version of the portfolio is "trust me."

Building something small you own outright fixes that. The test above is on a silly game about rejecting job applicants, and the sample size is genuinely too small — but the discipline is fully visible, timestamped in a public repository, and anyone can check that the analysis was written before the data existed rather than after.

That's the whole argument for pre-registration, really. It isn't about rigour for its own sake. It's that a result is only worth something if you'd have reported it either way, and writing it down first is the only way to prove you would have.

---

You can play ATS Defence — desktop only, since placing a tower means hovering a tile to see its range first, and a finger can't hover. The full pre-registration, including the SQL that produces the readout, is in the repository.

I'll write up the result when it hits the stopping point — including if the answer is "inconclusive", which is the outcome I'm expecting.