Pre-registering a small experiment before looking at the data

Goal

Separate the hypotheses and analyses decided before the data were seen from those invented afterwards, so that a small experiment (an A/B test, a performance comparison, a process change) cannot be quietly reinterpreted until it succeeds.

Prerequisites

A question that can be answered by data not yet collected or not yet analysed, and a place to store a time-stamped document that cannot be altered unnoticed: a commit in the repository, a dated wiki revision, or a public registry; AsPredicted, for example, describes its pre-registrations as time-stamped single-page documents with a unique URL that cannot be modified once made public. The Center for Open Science describes preregistration as specifying the research plan in advance and submitting it to a registry, to distinguish planned from unplanned work; it notes that the same data cannot be used both to generate and to test a hypothesis.

Steps

1. Write the hypothesis as a directional, falsifiable sentence: "variant B reduces median checkout time relative to A".
2. Name the primary outcome and exactly how it is computed (metric, unit, aggregation, time window), plus at most a few secondary outcomes labelled as such.
3. Fix the sample: how many units (users, requests, runs), how they are assigned, and the stopping rule. State in advance what will be excluded (bots, timeouts) and how.
4. Write the analysis plan: the comparison, the statistic, the threshold or interval that counts as support, and what result would count against the hypothesis.
5. List the known confounders and how each is handled (randomisation, blocking, holding constant).
6. Commit or register the document with a timestamp before collecting or looking at the data; record its identifier in the experiment notebook.
7. After the experiment, report the pre-registered analysis first and unchanged; label every additional analysis as exploratory and treat its findings as hypotheses for the next experiment.

Expected result

The write-up cannot be accused of choosing the outcome after seeing the results, and exploratory findings are visibly separated for follow-up rather than presented as confirmations.

Limits and test basis

Pre-registration constrains analysis, not design quality: a badly designed experiment stays bad. Deviations are sometimes necessary (a metric turns out to be unavailable); they should be documented with reasons, not hidden. The steps follow the cited registries' notion of a plan; no measurement of their effect is claimed here.