Verdict
Yes
Yes, a typical bug fix gets one regression test that fails on the unfixed code and passes with the fix, written at the lowest level that reproduces the bug and run in CI.
Why
- Blast radiususers
- Change frequencyregularly
- Detectabilityeventually
- Reversibilitywith-effort
- Test costmoderate
Yes, a typical bug fix gets one regression test that reproduces the bug. The typical case is a bug that a customer reported in an application your team changes about once a month, so Blast radius is users and Change frequency is regularly. Detectability is eventually, because the bug got past review and CI and a customer found it days after the release. Reversibility is with-effort: a hotfix and replies to support tickets. Test cost is moderate, about an hour, because the report already holds the failing input, so rule R11 gives Test.
| When | Decision | Why |
|---|---|---|
| The bug charged customers twice for one order | Test mandatory: tests for the bug and its boundary values, reviewed by a second person | Blast radius rises to money and Reversibility to costly, because refunds follow |
| The bug let a user open another customer's records | Test mandatory, including a test that a request for another customer's record is denied | Blast radius rises to safety-or-legal and Detectability to never, because a leaked record raises no error |
| The bug was a 500 error on a read-only page that users reported within hours | Test minimally: one test with the input that crashed the page | Detectability falls to same-day and Reversibility to trivial, because a read-only page stores nothing to repair |
| The fix makes a nullable field non-null, so the compiler rejects the code that caused the bug | Do not write a regression test; the type check catches the bug | Detectability falls to immediately and Reversibility to trivial, because the build fails before merge |
| The bug is a race between two services under load, and reproducing it in a test takes days | Test it differently: alert on the trace the race leaves, such as duplicate rows | Test cost rises to heavy, while Detectability stays eventually and Reversibility with-effort |
| The bug was in a one-time backfill script that you fix and run again | Test it differently: rerun the fixed script on a copy of production data and diff the result | Change frequency falls to once, so a test in the suite would never run again |
What breaks if you don't test
The fixed bug comes back. Someone refactors the function, resolves a merge conflict in favour of the old lines, or replaces a library, and the fix disappears. CI stays green for the same reason it did the first time: no test sends that input. The customer who reported it finds it again and learns that you fixed it once and lost the fix.
What you lose if you over-test
A test for every ticket produces tests that assert a heading's wording or a CSS margin. They break whenever someone edits copy, and the team learns to update them without reading. A regression test for a timing bug that relies on sleep() fails at random in CI, people rerun the job until it passes, and a real failure then gets the same rerun.
How to test
- Write the test before the fix, with the exact input from the report. Run it on the unfixed code and check that it fails for the reason in the report.
- Write it at the lowest level that reproduces the bug: a unit test for a bug in one function, an integration test for a bug in a query.
- Include data that the buggy code would also return, so a wrong result fails. Cover input variants with parametrized tests.
- Commit the test with the fix and put the ticket number in its name or a comment.
When the answer changes
- The bug moved money or showed a customer's records to someone else.
- Reproducing the bug needs days of setup: timing under load, several services, or one device model.
- The fix makes the compiler reject the mistake, or the code runs only once.
Real incident + Code example
The accent that came back
On a customer support product I worked on, a customer reported that a contact search for "Jose" did not find "José Álvarez". The fix wrapped both sides of the SQL LIKE in PostgreSQL's unaccent() and shipped with no test. Five months later a colleague moved contact search to full-text search, and unaccent() did not come along. CI stayed green. The same customer reopened the ticket 17 days after that release. The second fix came with this test, run against a local database:
# Regression test for SUP-2291: accented names were not found
import pytest
@pytest.mark.parametrize("query", ["jose", "José", "JOSE"])
def test_contact_search_ignores_accents(db, query):
db.insert_contacts(["José Álvarez", "Anna Kowalska"])
names = [c.name for c in search_contacts(db, query)]
assert names == ["José Álvarez"]
Anna Kowalska is in the data so that a search returning every contact fails too.
Related questions
FAQ
- Should we always unit test bugs when correcting them?
Yes, fix a typical bug together with a regression test that fails on the unfixed code; skip it when the compiler now rejects the mistake or the code runs once. If your team requires a test for every fix, follow the rule, because a shared rule costs less than a debate on each ticket.
- Should I have unit tests for known defects?
Yes, when a fix is planned: commit the test for a known defect marked with pytest's
xfail(strict=True), so CI stays green until the defect is fixed and then fails as a reminder to remove the marker. Do not commit a plain failing test, because a red build that everyone ignores hides new failures.- Should I write the regression test before the fix?
Yes, write the regression test before the fix and run it on the unfixed code. A test written after the fix may pass on the old code too, and then it proves nothing about the bug.