Reproducibility of a machine-learning experiment: seeds, environment, data and the limits of determinism

Goal

Make a training run repeatable by another person or agent, on another day, to the same metric where the platform allows it, and make every remaining source of variation explicit.

Prerequisites

A committed code state, a dataset with a checksum or version identifier, and a dependency lock file for the training environment.

Steps

1. Remove every implicit random state. The scikit-learn pitfalls page states that reproducible results across executions require removing all uses of the default random_state=None, and recommends declaring one random state at the top of the program and passing it to every estimator and splitter; it advises against setting the global NumPy seed. For NumPy code, create a Generator with default_rng(seed), which the NumPy reference names as the recommended constructor, and pass it down in the same way instead of calling module-level random functions.
2. For deep-learning frameworks, seed the framework's generator (torch.manual_seed) and enable deterministic algorithms where offered. The PyTorch reproducibility note states that completely reproducible results are not guaranteed across releases, commits or platforms, that CPU and GPU results may differ under identical seeds, and that deterministic operations are often slower; record which of these settings were used.
3. Fix the data: record the dataset checksum and the split identifier, and load rows in a defined order (sorting by a stable key before splitting).
4. Pin the environment: lock file, Python version, hardware class and, for GPUs, driver and library versions; build a container image if the environment must outlive the machine.
5. Write the configuration (all hyperparameters, seeds, paths, versions) to one file that the run reads and copies into its output directory.
6. Rerun the experiment twice from a clean checkout on the same machine and once on a different one; record whether the metric matched exactly, within tolerance, or not, and which step explains any difference.
7. Store the results with the configuration and the code commit; report variation across seeds as an interval instead of one seed's number.

Expected result

A run directory that contains everything needed to repeat the run, a statement of how exactly it repeats, and metrics that are comparable across runs because the seeds and data are the same.

Limits and test basis

Determinism can cost speed and may not be available for every operation. Bitwise repeatability across hardware is not promised by the cited documentation. The protocol is a proposal; no repeatability rate is claimed.