Verdict
No
No, do not fill example tests with random values; give each test fixed values, one from each class of input that the code treats in its own way, so a failure repeats on the next run.
Why
- Blast radiususers
- Change frequencyregularly
- Detectabilityimmediately
- Reversibilitywith-effort
- Test costheavy
No, do not fill example tests with random values; give each test fixed values, one from each class of input that the code treats in its own way. The typical case is a test of customer-facing code that takes a name or a number from Faker, next to fixed tests that cover each class of input. Blast radius is users and Change frequency is regularly. Detectability is immediately, because a random value lands in a class that the fixed tests already run, so a broken branch fails a fixed test in CI. Reversibility is with-effort, since wrong stored records need a repair script, and Test cost is heavy: the test must compute an expected value for each random input, and a failure that appears once cannot be rerun without its seed, so rule R13 gives Do not test for the random values.
| When | Decision | Why |
|---|---|---|
| No test covers a phone number formatter yet, and a wrong number is saved without an error | Test: fixed numbers, one per country format and one malformed number | Detectability rises to eventually and Test cost falls to moderate: a few classes need few expected values |
| One rule holds for every input, such as an export that must import back to the same records | Test: a property-based test that checks the rule on generated records and shrinks a failing input | Detectability rises to eventually and Test cost falls to moderate, because the rule replaces computed expected values |
| The function rounds a currency conversion for a charge | Test mandatory: fixed amounts on each half-cent boundary, which random amounts seldom hit | Blast radius rises to money and Reversibility to costly: a wrong charge ends in refunds |
| The code parses files that anyone on the internet can upload | Test mandatory: a fuzz test in CI that keeps every failing input as a regression case | Blast radius rises to safety-or-legal and Detectability to never: a crafted file can expose other users' data without an error |
| A one-time migration rewrites every stored address into a new format | Test it differently: run the migration on a copy of the real records and check the results with a query | Change frequency falls to once: a test in the suite would never run again |
What breaks if you don't test
A fixed suite misses the classes of input that nobody listed, and a random value sometimes lands in one. Faker's English surnames include O'Kon, O'Hara and D'Amore. A signup that builds a username from the surname and accepts only letters passes a suite that uses "Smith", until a customer named O'Hara cannot sign up. One fixed surname with an apostrophe finds the apostrophe bug on every run.
What you lose if you over-test
The test usually computes the expected value with the same formula as the code, so a wrong formula passes in both places. A failure that shows up once looks flaky, people rerun the job until it passes, and Martin Fowler's article on non-determinism calls such tests useless. Scott Hannen notes that a test drawing one number per run may need hundreds of runs to reach the value that breaks.
What to do instead
- Give each test fixed values, one per class of input, plus both sides of each boundary, in a parameterized test with pytest or JUnit.
- For fields the test never reads, use a builder default or a generator with a fixed seed, such as
Faker.seed(4321)in Python Faker, and pin the Faker version. - For a unique column in a shared test database, append a counter that no assertion compares.
- When one rule holds for every input, write a property-based test with Hypothesis.
- When strangers send the input, add a fuzz test such as Go fuzzing, which reruns each failing input on every
go test.
When the answer changes
- One rule holds for every input.
- The code parses input that anyone on the internet can send.
- A wrong result charges or pays out money.
Real incident + Code example
O'Kon and the escaped apostrophe
On a Rails app I worked on, a request spec gave each test user a Faker surname and checked that the profile page contained it. About twice a month the build went red, someone reran it, and it passed. After four months a developer logged the surname of a failing run: O'Kon. Rails had escaped the page as it should, so the HTML held O'Kon, and a search of the raw body for the surname found nothing. The code had worked all along; the random value had found a bug in the test. The fix used fixed surnames and checked the page text:
# Before: a random surname, checked against raw HTML
it "shows the surname" do
user = create(:user, last_name: Faker::Name.last_name)
get user_path(user)
expect(response.body).to include(user.last_name) # fails for O'Kon
end
# After: one fixed surname per class, checked against the page text
["Smith", "O'Hara", "Nguyễn"].each do |surname|
it "shows the surname #{surname}" do
user = create(:user, last_name: surname)
get user_path(user)
expect(Capybara.string(response.body)).to have_content(surname)
end
end
Related questions
FAQ
- Should I be using random data in integration tests?
No, give integration tests fixed data, because a value that changes on every run makes a failure hard to repeat. The exception is a unique column in a shared database, where a counter that no assertion reads keeps runs apart.
- Should I use Faker in unit tests?
Use Faker in unit tests only with a fixed seed, and only for fields that the test does not assert on. The Faker documentation warns that a patch release can change the values for the same seed, so pin the version.
- How is property-based testing different from random test data?
Property-based testing generates many inputs in each run, checks a rule instead of a computed expected value, and shrinks a failing input to the smallest case. Random test data draws one value per run and leaves no record of the value that failed.