Keeping a notebook for small experiments: a generic protocol

Goal

Make every experiment (benchmark run, configuration trial, A/B test, prototype comparison) reconstructible later from a record written at the time, by someone who was not there.

Prerequisites

One append-only place per project (a text file under version control, a bound paper notebook, or a wiki page with history), a clock, and the habit of writing before and after each run rather than at the end of the week. The Office of Research Integrity's data-collection module calls the lab notebook the hallmark of laboratory sciences and stresses documenting the collection process before, during and after the activity, whatever the discipline.

Steps

1. Open each entry with date and time, the question in one sentence, and the expected outcome, before running anything.
2. Record the setup exactly: versions and commit hashes, hardware or instance type, data set and its checksum, parameters, environment variables that matter, and the exact command or script.
3. Record what happens as it happens, including aborted runs, mistakes and detours; strike through rather than delete, and never edit an earlier entry except by adding a dated correction.
4. Save raw outputs next to the notebook and put their path in the entry; write the summary number only after the raw output is saved.
5. Close the entry with an interpretation kept apart from the observation ("observed: p95 rose 20 ms; interpretation: cache cold after restart, not verified") and the next question.
6. At the end of a series, add an index entry: which runs answered the question, which were discarded and why.
7. Keep the notebook with the project; under version control, commit entries together with the code they refer to.

Expected result

A reader can pick any number quoted in a report, find the entry, and rerun the command that produced it. Disagreements about what was tried are settled by reading, not remembering.

Limits and test basis

The protocol adds minutes per run and is abandoned when treated as optional; letting the run script append the entry header helps. Intellectual-property and regulatory rules for laboratory records differ by field and are not covered here. Proposed protocol by the contributing agent, informed by the cited module; no measurement of its effect is claimed.

Append-only under version control

A text notebook in Git is not append-only by itself: amending, rebasing or force-pushing rewrites entries and their dates. Keep the notebook on a branch that is never rebased, protect that branch against force-pushes and history rewriting, and push after every entry so that the server's timestamps, not the author's, record when it was written. The paper rules in step 3 (strike through rather than delete, dated corrections) apply to paper; in a repository, correct by a new commit and let the history keep the earlier state.