Test data without production personal data

Goal

Give developers, CI and staging environments realistic data without copying personal data out of production, so that the "just dump prod" shortcut is unnecessary and unavailable.

Prerequisites

A schema with constraints, so that generated data is validated by the database. A data map marking personal columns. A generator library: the Faker documentation describes a Python package that generates fake data (names, addresses, text, numbers) and a seed() method that makes a run reproducible; the same documentation warns that results are not guaranteed to be consistent across patch versions, so a project that relies on a seed pins Faker to the patch number.

Steps

1. Classify columns: personal (generate), business (generate with realistic distributions), reference (copy from production; countries, product catalogues and tariffs hold no personal data).
2. Write one generator per entity with the same validity rules as the application: a generated email address passes the validator, a generated order references a generated customer. Seed the generators so that a bug report can name a seed.
3. Produce volume in SQL where possible: INSERT ... SELECT ... FROM generate_series(1, n) builds a large table in one statement; the PostgreSQL documentation lists generate_series among the set-returning functions, for integers, numerics and timestamps.
4. Reproduce shape, not people: sample production for aggregate distributions (orders per customer, name lengths, null rates, locale mix) and feed those aggregates into the generators.
5. Add the edge cases production has: names outside ASCII, very long strings, empty strings, one customer with ten thousand orders, timestamps around daylight-saving changes.
6. Keep a recognisable marker in generated data (addresses at a reserved domain, a fixed name prefix) so that a leaked test set is identifiable and mail cannot reach a real person.
7. Remove the shortcut: production database credentials are unavailable to developers and CI. If a dump-to-staging job exists, it runs only through a masking step that replaces personal columns with generated values before the dump leaves the production network.

Expected result

A generated dataset, reproducible from a seed, with production-like distributions and no production personal data; staging looks realistic and can be reset at will.

Limits and test basis

Generated data misses correlations nobody thought to model (the customer who is also an employee). A masked production dump keeps structure and quasi-identifiers; treat it as pseudonymised, not anonymised. No claim is made about generation speed or about how closely generated distributions match production.

Reachable fields and outbound channels

A reserved e-mail domain protects one channel. Generated phone numbers, postal addresses and account numbers are formally valid and may be real, so every field that a system could send to, ship to or check against comes from a reserved range: domains under RFC 2606 and RFC 6761 (example.com, .test, .invalid), the NANP fictional block 555-0100 to 555-0199 or the drama ranges national regulators publish, RFC 5737 networks for IP addresses, and the test numbers and sandbox keys that SMS and payment providers document. Independently of the data, non-production environments stub every outbound channel by configuration: a capture SMTP server, provider sandbox credentials, no production API keys. The stub is the guarantee; the reserved ranges limit the damage when a stub is missing.