Should I test that?

Should I commit failing tests?

Answer

Yes, commit a test that reproduces a known bug, marked as an expected failure with the issue number (pytest xfail with strict=True, Vitest test.fails, Jest test.failing), so the build on main stays green and the run fails on the day a change fixes the bug.

Verdict on the code under testYes

Why

Yes, commit it as an expected failure: for a developer who has reproduced a customer's bug in a test and will fix it in a later pull request, rule R11 gives Test. Blast radius is users and Change frequency is regularly, because customers hit the bug and the code around it changes monthly. Detectability is eventually, because the typical bug stores a plausible wrong value, such as a postal code without its leading zero. Reversibility is with-effort, since stored values need a repair script, and Test cost is moderate: about an hour to write, plus updates when the fix changes what it asserts.

When the decision changes
WhenDecisionWhy
The bug makes checkout apply a discount twiceTest mandatory: commit the test as an expected failure today, block the next release until it passes, and have a second person review itBlast radius rises to money and Reversibility to costly, because wrong charges end in refunds
The bug lets one customer read another customer's records, and the repository is publicTest mandatory: keep the test and the fix in a private fork, such as a GitHub security advisory fork, and publish them togetherBlast radius rises to safety-or-legal and Detectability to never, because a leaked record raises no error
The bug is a wrong label that the developer sees on screen right after the fixDo not test: fix the label, look at the page, and commit no test for itDetectability moves to immediately and Reversibility to trivial, because a revert removes a wrong label
The bug sits in a library you depend on, and the fix waits for its next releaseTest minimally: commit one strict expected-failure test, which fails the run on the upgrade that fixes the bugChange frequency falls to rarely, because the behaviour changes only when you upgrade the library
The bug is a race between two services under load that no test can force into order, and its test fails in one run out of twentyTest it differently: keep the test out of the gating suite, log both requests with a shared ID, and alert on duplicate recordsTest cost rises to heavy, because a test that fails at random needs reruns and a person to read each result

What breaks if you don't test

If the test stays on your machine, the bug lives only in a ticket. A teammate who changes that code next month cannot see whether the change fixed the bug or left it alone. Committing it without a marker does its own damage: a red test fails every build on main, and once the team learns to ignore red, the next real regression hides behind it.

What you lose if you over-test

Markers pile up. A suite with 30 markers whose issues were closed prints "30 xfailed" on every run, and nobody reads the line. A failing test for a bug the team has decided not to fix is noise: delete it and close the issue.

How to test

  1. Write the test from the bug report and check that it fails for the reason in the report, not from a missing fixture.
  2. Mark it with the issue number: xfail in pytest, test.fails in Vitest, test.failing in Jest.
  3. In pytest, pass strict=True or set xfail_strict = true, so an unexpected pass fails the run.
  4. Do not skip it: a skipped test does not run, so it never reports that the bug is gone. JUnit 5 has no expected-failure marker, so in Java commit the test with its fix.
  5. Remove the marker in the commit that makes the test pass.

When the answer changes

  • The bug moves money or decides who may read a record.
  • The reproducing test fails only some of the time.
  • Your team requires every commit on main to pass with no markers: commit the test with its fix.

Real incident + Code example

The skipped test that watched the bug come back

On a Django CRM I worked on, the contact import stored the postal code 02134 as 2134. A developer reproduced it and committed the test with @pytest.mark.skip to keep the build green. Two months later a rewrite of the CSV parser fixed the bug by accident, and nobody removed the skip, because a skipped test prints the same line either way. Five months after that, a change to column type detection dropped the zero again, and a customer found it when their mail came back undelivered. A strict expected failure would have failed the rewrite's build on the unexpected pass, and its author would have turned it into a regular test:

import pytest
from crm.imports import parse_contact_row

# Known bug #482: the import strips leading zeros from postal codes.
@pytest.mark.xfail(strict=True, reason="#482 leading zero lost")
def test_postal_code_keeps_leading_zero():
    row = {"name": "Ann Lee", "postal_code": "02134"}
    assert parse_contact_row(row).postal_code == "02134"

FAQ

Should I check in failing acceptance tests?

Yes, check in an acceptance test for a feature in progress, tagged as work in progress and left out of the run that gates merges. In Cucumber, tag it @wip and run the gating suite with --tags "not @wip", as the tag expression docs show.

Should I have failing tests?

You should have no unmarked failing tests on main, because a known red test hides the next real failure. Each known failure needs an expected-failure marker and an issue number.

Can I commit a failing test on my own branch?

Yes, commit a failing test on your own branch, because nobody else builds it. Before the pull request merges, the test must pass with its fix or carry an expected-failure marker.