1 · The painted-door subscription test
Situation
Paywall strategy needed to know whether readers would pay for subscriptions — but no subscription product existed, so there was no behavioural data at all. Opinion filled the vacuum: every stakeholder had a different instinct about which audiences would convert.
Design
A painted-door experiment: show a realistic subscription prompt to a controlled sample, measure genuine click-through intent by audience segment, and be honest on the other side of the door about the product not existing yet. The design conversation was mostly about ethics and sample integrity — capping exposure per user, excluding logged-out churners, and pre-registering which segments we'd read out so nobody could fish for a positive result afterwards.
Readout
Intent was heavily concentrated: loyal, habitual readers showed a step-change more interest than every other segment, while search-referred traffic — the bulk of raw volume — showed almost none. Crucially, the gap between segments was far larger than any plausible measurement error, which is what made the result decision-grade despite the door being painted.
Decision
Paywall strategy got its first concrete demand signal: size the opportunity on the loyal-reader base, not on total traffic. That reframing changed the revenue model conversation from "what percentage of 65 sites' traffic converts?" to "how many habitual readers do we actually have, and where?"
2 · The engagement programme readout
Situation
A rolling programme of registration prompts, content gating and engagement experiments across 65+ national and regional titles — 10–20 live experiments a month, each needing a go / iterate / stop call, with results that were often noisy and occasionally contradicted each other between titles.
Design
The unit of work here wasn't a single test but the readout discipline: pre-agreed primary metrics, significance checked post-hoc in BigQuery rather than trusted from the testing tool's dashboard, and conflicting title-level results reconciled by separating signal from variance before anyone saw a headline number. The recommendation was always one of three words — go, iterate, stop — with the evidence attached.
Readout
Individually, most wins were small — fractions of a percent. Compounded across a year of shipped go decisions (and, just as importantly, stopped losers), the programme contributed to a ~5% uplift in yearly pageviews, worth an estimated £50K+ in incremental ad revenue.
Decision
Beyond the number, the durable outcome was cultural: product managers stopped asking "did the test win?" and started asking "what's the readout?" — which is the difference between a testing tool and a testing culture.
3 · The Bookmark feature case
Situation
A reader-facing Bookmark feature was on the roadmap bubble — plausible, likeable, and completely unproven. It needed either a case or a burial.
Design
I built the analytical case before the build: sized the returning-reader audience that would plausibly use it, defined what success would look like post-launch (repeat usage and return-visit behaviour, not raw taps), and set up the rollout analysis so the launch decision and the keep decision were separate questions with separate evidence.
Readout
Post-launch behaviour matched the case: usage grew week on week among exactly the returning-reader segment the case predicted, with bookmark users showing the stronger return-visit pattern the feature was meant to reinforce.
Decision
Shipped, kept, and still live today. It remains my favourite kind of analyst work: the feature exists because the case held, and stayed because the readout did.
How I'd do this for you
Every study above follows the same shape — frame the question before touching data, pre-commit the readout, separate signal from variance, and end on a one-word recommendation with evidence attached. If your experimentation programme produces results people argue about instead of decisions people act on, that's the gap I fill.