Verdict
Test it differently
Test it differently: keep production data out of the test suite, and find the record shapes your hand-written fixtures miss with aggregate queries on a read replica and an alert on records your code rejects in production.
Why
- Blast radiususers
- Change frequencyregularly
- Detectabilityeventually
- Reversibilitywith-effort
- Test costheavy
Test it differently: keep production data out of the test suite, and find the record shapes your fixtures miss by watching production. The typical case is a backend team that wants a production copy in CI, because hand-written fixtures miss records such as a null last name from an old import. Blast radius is users, because a mishandled record shows a customer wrong data, and Change frequency is regularly, because code that reads customer records changes about once a month. Detectability is eventually, because a truncated name looks plausible for weeks, and Reversibility is with-effort, because records saved wrong need a repair script. Test cost is heavy, because a production copy needs a masking pipeline that takes days to build, and data that changes with every refresh allows no exact assertions, so rule R9 gives Test it differently.
| When | Decision | Why |
|---|---|---|
| A bug report names the production record that breaks the code, such as a customer with no last name | Test: a unit test with a hand-written record of that shape and invented values | Test cost falls to trivial, because the shape is known and a fixed record reproduces the failure exactly |
| CI already loads a frozen, masked subset of production from the same file on every run | Test: run the integration suite on the masked subset and assert that every record loads and saves | Test cost falls to moderate, because the masking exists and the input no longer changes between runs |
| The code only reads records, and a crash on an odd record reaches the error tracker the same day | Do not build a production-data test; fix each crash the error tracker reports | Detectability moves to same-day and Reversibility to trivial, because a read stores nothing |
| The code decides which customer sees which record, such as an account export | Test mandatory: synthetic records of two customers in each test, asserting that each customer gets only their own rows | Blast radius rises to safety-or-legal and Detectability to never, because a leaked record looks like normal output |
| A migration rewrites existing customer rows once | Test it differently: rehearse the migration on a masked copy of production and compare row counts before and after | Change frequency falls to once while Blast radius stays users, so the control becomes a rehearsal |
What breaks if you don't test
Fixtures hold only the shapes the developer thought of, such as short ASCII names and one address per customer. A new form field capped at 50 characters then cuts the longer company names of older accounts, the tests stay green on "Acme Ltd", and customers find truncated names weeks later, in every record saved since the release.
What you lose if you over-test
A raw copy puts customer personal data on CI runners, staging servers and laptops. Masking that a lookup table can reverse is pseudonymisation, and Recital 26 of the GDPR still treats such data as personal data. Tests that assert on real records break when a customer edits the name a test expected, and staging jobs act on the real email addresses in the copy.
What to do instead
- Build unit test data by hand or with a factory library such as factory_boy: one small record for each case the code handles, with invented values.
- Schedule aggregate queries on a read replica that return counts, never rows, and alert when a count that was zero grows.
- Validate records where production code reads them, log the ID of each rejected record, and alert on the reject rate.
- Copy each shape the queries or the alert find into a fixture with invented values.
- For a migration rehearsal, restore an anonymous dump into an isolated environment with outgoing email switched off.
When the answer changes
- A bug report names the exact record shape that broke the code.
- The code decides which customer sees which record.
- A migration must run once on the real rows.
Real incident + Code example
The Monday restore that emailed real customers
On a booking product for fitness studios I worked on, staging was refreshed every Monday from a production dump, and a script then rewrote customer emails to a test domain. One Monday the restore finished late, the staging reminder job ran before the rewrite, and 2,100 customers got a class reminder with a staging link. Nobody could recall those emails. We stopped the restores, built fixtures by hand, and scheduled this query on the read replica, which returns counts and no customer data:
-- Record shapes in production that the fixtures must cover
SELECT
count(*) FILTER (WHERE last_name IS NULL) AS no_last_name,
count(*) FILTER (WHERE char_length(company) > 50) AS long_company,
count(*) FILTER (WHERE email <> lower(email)) AS mixed_case_email,
count(*) FILTER (WHERE phone !~ '^\+[0-9]{8,15}$') AS phone_not_e164,
count(*) FILTER (WHERE last_name ~ '[^\x01-\x7F]') AS non_ascii_last_name
FROM customers;
Each column above zero got a fixture with invented values, and the job alerted when a column that was zero grew.
Related questions
FAQ
- Should I use real or sample data for unit tests?
Use sample data: unit tests should run on small hand-written records with invented values, one record per case the code handles. Real data enters a unit test only as the shape of a record that broke the code, rewritten with invented values.
- Should unit test data be simplified or realistic?
Unit test data should be realistic in shape and small in size. Each fixture record uses a format that production holds, such as a non-ASCII surname, and sets only the fields the test reads.
- Can I use anonymized production data for testing?
Yes, an anonymized copy of production suits a migration rehearsal when nobody can link its records back to a person and it runs in an isolated environment. Masking that a lookup table can reverse is pseudonymisation, which the GDPR still treats as personal data.