Answer
Yes, commit a test that reproduces a known bug, marked as an expected failure with the issue number (pytest xfail with strict=True, Vitest test.fails, Jest test.failing), so the build on main stays green and the run fails on the day a change fixes the bug.
Verdict on the code under testYes
Why
- Blast radiususers
- Change frequencyregularly
- Detectabilityeventually
- Reversibilitywith-effort
- Test costmoderate
Yes, commit it as an expected failure: for a developer who has reproduced a customer's bug in a test and will fix it in a later pull request, rule R11 gives Test. Blast radius is users and Change frequency is regularly, because customers hit the bug and the code around it changes monthly. Detectability is eventually, because the typical bug stores a plausible wrong value, such as a postal code without its leading zero. Reversibility is with-effort, since stored values need a repair script, and Test cost is moderate: about an hour to write, plus updates when the fix changes what it asserts.
| When | Decision | Why |
|---|---|---|
| The bug makes checkout apply a discount twice | Test mandatory: commit the test as an expected failure today, block the next release until it passes, and have a second person review it | Blast radius rises to money and Reversibility to costly, because wrong charges end in refunds |
| The bug lets one customer read another customer's records, and the repository is public | Test mandatory: keep the test and the fix in a private fork, such as a GitHub security advisory fork, and publish them together | Blast radius rises to safety-or-legal and Detectability to never, because a leaked record raises no error |
| The bug is a wrong label that the developer sees on screen right after the fix | Do not test: fix the label, look at the page, and commit no test for it | Detectability moves to immediately and Reversibility to trivial, because a revert removes a wrong label |
| The bug sits in a library you depend on, and the fix waits for its next release | Test minimally: commit one strict expected-failure test, which fails the run on the upgrade that fixes the bug | Change frequency falls to rarely, because the behaviour changes only when you upgrade the library |
| The bug is a race between two services under load that no test can force into order, and its test fails in one run out of twenty | Test it differently: keep the test out of the gating suite, log both requests with a shared ID, and alert on duplicate records | Test cost rises to heavy, because a test that fails at random needs reruns and a person to read each result |
What breaks if you don't test
If the test stays on your machine, the bug lives only in a ticket. A teammate who changes that code next month cannot see whether the change fixed the bug or left it alone. Committing it without a marker does its own damage: a red test fails every build on main, and once the team learns to ignore red, the next real regression hides behind it.
What you lose if you over-test
Markers pile up. A suite with 30 markers whose issues were closed prints "30 xfailed" on every run, and nobody reads the line. A failing test for a bug the team has decided not to fix is noise: delete it and close the issue.
How to test
- Write the test from the bug report and check that it fails for the reason in the report, not from a missing fixture.
- Mark it with the issue number:
xfailin pytest,test.failsin Vitest,test.failingin Jest. - In pytest, pass
strict=Trueor setxfail_strict = true, so an unexpected pass fails the run. - Do not skip it: a skipped test does not run, so it never reports that the bug is gone. JUnit 5 has no expected-failure marker, so in Java commit the test with its fix.
- Remove the marker in the commit that makes the test pass.
When the answer changes
- The bug moves money or decides who may read a record.
- The reproducing test fails only some of the time.
- Your team requires every commit on main to pass with no markers: commit the test with its fix.
Real incident + Code example
The skipped test that watched the bug come back
On a Django CRM I worked on, the contact import stored the postal code 02134 as 2134. A developer reproduced it and committed the test with @pytest.mark.skip to keep the build green. Two months later a rewrite of the CSV parser fixed the bug by accident, and nobody removed the skip, because a skipped test prints the same line either way. Five months after that, a change to column type detection dropped the zero again, and a customer found it when their mail came back undelivered. A strict expected failure would have failed the rewrite's build on the unexpected pass, and its author would have turned it into a regular test:
import pytest
from crm.imports import parse_contact_row
# Known bug #482: the import strips leading zeros from postal codes.
@pytest.mark.xfail(strict=True, reason="#482 leading zero lost")
def test_postal_code_keeps_leading_zero():
row = {"name": "Ann Lee", "postal_code": "02134"}
assert parse_contact_row(row).postal_code == "02134"
Related questions
- Should I write a failing test before fixing a bug?Yes
- Should I write a test for every bug fix?Yes
- Should I write tests before code?Code under test: Yes
- Should I use TDD?Code under test: Yes
- Should I write acceptance tests?Yes
FAQ
- Should I check in failing acceptance tests?
Yes, check in an acceptance test for a feature in progress, tagged as work in progress and left out of the run that gates merges. In Cucumber, tag it
@wipand run the gating suite with--tags "not @wip", as the tag expression docs show.- Should I have failing tests?
You should have no unmarked failing tests on main, because a known red test hides the next real failure. Each known failure needs an expected-failure marker and an issue number.
- Can I commit a failing test on my own branch?
Yes, commit a failing test on your own branch, because nobody else builds it. Before the pull request merges, the test must pass with its fix or carry an expected-failure marker.