Verdict
Yes
Yes, property-based testing is worth it for functions with a wide input space and a rule you can state, such as a parser or a serializer: add one or two properties per function next to the example tests and run them in CI; do not write properties for glue code.
Why
- Blast radiususers
- Change frequencyregularly
- Detectabilityeventually
- Reversibilitywith-effort
- Test costmoderate
Yes, property-based testing is worth it for functions whose input space is too wide for hand-picked examples. The typical case is a parser, a serializer or a date-range function in a customer-facing application. Blast radius is users, and Change frequency is regularly, because the function gains a field or a format about once a month. Detectability is eventually: inputs the examples miss, such as a comma inside a field, give plausible wrong output instead of an error. Reversibility is with-effort, since wrong records need a repair script, and Test cost is moderate, an hour to find a property and write a generator, so rule R11 gives Test.
| When | Decision | Why |
|---|---|---|
| The function splits an invoice total into monthly instalments | Test mandatory: a property that the instalments add up to the total | Blast radius rises to money and Reversibility to costly: a wrong instalment ends in refunds |
| The function parses a report format that changes a few times a year | Test minimally: one round-trip property | Change frequency falls to rarely, so fewer changes can break it |
| The function throws on inputs it cannot handle, and a failed request stores nothing | Test minimally: one property that no generated input makes it throw | Detectability falls to same-day and Reversibility to trivial: the error tracker reports the crash |
| The property needs the whole system, such as random API calls against a staging database | Test it differently: alert when a nightly production query finds records that break the invariant | Test cost rises to heavy, while Detectability stays eventually and Reversibility with-effort |
| A one-time migration rewrites every stored phone number into a new format | Test it differently: run it on a copy and check that each new number parses back to the original | Change frequency falls to once: a test in the suite would never run again |
| The function lives in a script that only you run | Do not test: run the script and look at the result | Blast radius falls to none: only you bear a failure |
What breaks if you don't test
Example tests check only the inputs their author thought of. A search parser tested with plain words returns an empty query for an unmatched quote, so a user who types 27" monitor gets no results and no error. Support hears about it weeks later, because an empty results page looks like a real answer.
What you lose if you over-test
A property that restates the implementation, such as a total checked against the same reduce call that computes it, copies the bug and passes. Glue code, such as a controller that passes fields on, has no property beyond the example. A property that touches a database runs once per generated input, 100 times by default in Hypothesis, so a dozen of them at 100 ms per input add two minutes to each CI run.
How to test
Add property tests at the unit level, next to the example tests:
- Pick a property that does not repeat the code: a round trip (
parse(format(x))equalsx), an invariant (sorting keeps the same items), or a match with a slow, obvious version. - Make the generator produce the inputs that break such functions: empty strings, separators, quotes, line breaks and range ends.
- Run it in CI with fast-check for TypeScript, Hypothesis for Python or jqwik for Java.
- Copy each shrunk failing input into an example test that runs on every build.
When the answer changes
- The function computes money, such as instalments.
- The property needs a running system instead of a function call.
- The code runs once, as in a data migration.
Real incident + Code example
The note with a line break in it
On a CRM I worked on, users moved contacts between workspaces by CSV export and import, which four example tests covered. A note with a line break split one contact into two rows on import. A customer found 37 contacts with no name a month later, and a script repaired them. After the fix I added the round-trip property below, and its first run found a second bug: a lone carriage return, which the export did not quote and the import read as a new row. Strings in fast-check are printable ASCII by default, so the generator names the characters that matter:
import { expect, test } from "vitest";
import fc from "fast-check";
import { fromCsv, toCsv } from "./csv";
// Cells built from the characters that break CSV
const cell = fc.string({
unit: fc.constantFrom("a", "B", " ", ",", '"', "\n", "\r"),
});
const contacts = fc.array(fc.tuple(cell, cell, cell), { minLength: 1 });
test("export then import returns the same contacts", () => {
fc.assert(
fc.property(contacts, (rows) => {
expect(fromCsv(toCsv(rows))).toEqual(rows);
}),
);
});
Related questions
FAQ
- Why use property-based testing if I already write example-based tests?
Property-based tests find the inputs that example-based tests miss: the library generates 100 inputs per property by default in Hypothesis and 1,000 in jqwik, and shrinks a failing one to the smallest case. Keep the example tests, because they state the exact results you expect.
- What kind of code is property-based testing good for?
Property-based testing suits pure functions with a wide input space and a rule you can state without repeating the code: parsers, serializers, encoders, sorting, and date arithmetic. Glue code that passes fields on has no such rule.
- Are property-based tests flaky because the inputs are random?
Property-based tests fail only when the code is wrong for some input, but a rare input can fail one run and pass the next. fast-check and jqwik print the seed and the shrunk input on failure, and Hypothesis saves failing inputs and replays them first on the next run.
- Can property-based tests replace example-based tests?
No, keep example-based tests next to property-based tests, because a property states a rule and an example states an exact result. A round-trip property passes when the export and the import share a wrong date format; an example test with a known row catches it.