The framework
A test pays off when the expected cost of failure is higher than the cost of the test.
Framework v1.0published 2026-09-15
Model
- Expected cost of failure is the probability of failure multiplied by its price.
- Blast radius and Reversibility set the price of a failure.
- Change frequency and Detectability stand in for the probability that a failure reaches someone and keeps doing damage. Code that changes often breaks more often. A failure that nobody notices keeps doing damage.
- Cost of the test is the time to write it plus the time to keep it working while the code lives. Test cost measures both.
Every answer on this site states its value for each factor and the decision these rules give. The decision tool runs the same rules in your browser.
Five factors
Blast radius
Who is affected by a failure and how.
| Value | Definition | Example |
|---|---|---|
| 0 · none | Nobody but you. The code runs on your machine or you throw it away. | A script that renames your own photos. |
| 1 · internal | Your team or your company. A failure slows people down. No customer sees it. | An internal dashboard shows yesterday's numbers. |
| 2 · users | Customers see errors, wrong data, or a broken feature. | The search page returns no results. |
| 3 · money | A failure loses money directly: wrong charges, lost orders, wrong invoices, refunds. | A discount applies twice at checkout. |
| 4 · safety-or-legal | A failure can harm a person, expose personal data, or break a law or a contract. | An export sends patient records to the wrong clinic. |
Change frequency
How often this code or behaviour changes.
| Value | Definition | Example |
|---|---|---|
| 0 · once | The code runs one time, or you write it once and never change it. | A one-time data backfill. |
| 1 · rarely | The code changes a few times a year. | A date formatting helper. |
| 2 · regularly | The code changes about once a month. | The rules for a signup form. |
| 3 · constantly | The code changes in most weeks. | The pricing page of a startup that is still looking for its market. |
Detectability
Whether a failure would be noticed quickly without a test.
| Value | Definition | Example |
|---|---|---|
| 0 · immediately | The person who makes the change sees the failure at once: the build fails, the app does not start, or the error is on the screen. | A renamed field that the compiler rejects. |
| 1 · same-day | Someone notices within a day: an alert, an error report, or a user complaint. | A broken login page. |
| 2 · eventually | Someone notices after days or weeks, usually by chance. | A report that leaves out one region. |
| 3 · never | Nobody notices until the damage is done, or nobody notices at all. The output looks correct, or the failure is silent. | An access check that lets every user read every record. |
Reversibility
Whether a failure can be rolled back without consequences.
| Value | Definition | Example |
|---|---|---|
| 0 · trivial | You revert or redeploy and the problem is gone. Nothing stays behind. | A wrong button colour. |
| 1 · with-effort | Rollback works, but it takes work: a hotfix, a data repair script, or replies to support tickets. | Orders saved with a missing field that a script can fill in. |
| 2 · costly | Rollback leaves lasting damage: data that you repair by hand, refunds, lost trust, or a new app store release. | A mobile release that crashes on start. |
| 3 · impossible | You cannot undo the damage: emails are sent, data is deleted without a backup, secrets are public, money is paid out. | A newsletter sent to every customer twice. |
Test cost
The cost to write the test and to maintain it while the code lives.
| Value | Definition | Example |
|---|---|---|
| 0 · trivial | The test takes minutes to write and seldom breaks when the code changes. | A unit test for a pure function. |
| 1 · moderate | The test takes about an hour to write and needs updates when the behaviour changes. | An integration test against a local database. |
| 2 · heavy | The test takes days to build. It needs fixtures, environments, devices, or checks for output that is not the same each time. It breaks often. | An end-to-end test of a payment flow across three services. |
| 3 · prohibitive | The test costs more than the feature. The setup does not exist, you cannot reproduce the conditions, or the test needs its own team. | A test on every Android device and OS version. |
Scores
Each factor value counts as its position on the scale. The first value counts as 0, the second as 1, and so on.
- Price = Blast radius + Reversibility
- Exposure = Change frequency + Detectability
- Risk = Price + Exposure
- Net score = Risk - 2 x Test cost
The scales are steps, not measurements. One step up multiplies the cost. So the scores add steps instead of multiplying amounts.
Test cost counts twice. You pay for a test for certain. You pay for a failure only if it happens.
Rules
The rules apply in order. The first rule whose conditions all hold gives the decision.
| Rule | When | Decision | Why |
|---|---|---|---|
| R1 | Nobody is affectedBlast radius is none | Do not test | A failure has no price when it affects nobody. |
| R2 | Safety or law is at stakeBlast radius is safety-or-legal, and Test cost is trivial, moderate or heavy | Test mandatory | Harm to a person, exposed personal data, or a broken law costs more than any test that you can build. |
| R3 | Money is lost for goodBlast radius is money, and Reversibility is costly or impossible, and Test cost is trivial, moderate or heavy | Test mandatory | Money that you cannot get back costs more than any test that you can build. |
| R4 | No affordable test, real riskTest cost is prohibitive, and Risk is 5 or more | Test it differently | You cannot build a test at a sensible cost, and the risk is real.Instead: Watch production: alert on the failure signal, roll out in stages, and keep a rollback plan. A second person reviews the change. |
| R5 | No affordable test, low riskTest cost is prohibitive | Do not test | The test costs more than the feature, and the risk is low. |
| R6 | Runs once, real priceChange frequency is once, and Price is 3 or more | Test it differently | Code that runs one time gets one chance to fail. A test in the suite would never run again.Instead: Rehearse: run it on a copy of the real data, check the result with a query or a diff, keep a backup, and have a second person review it. |
| R7 | Runs once, low priceChange frequency is once | Do not test | Code that runs one time with a low price of failure does not repay a test. |
| R8 | The failure shows itselfDetectability is immediately, and Reversibility is trivial | Do not test | The person who makes the change sees the failure at once and undoes it at no cost. A test adds nothing. |
| R9 | Watch instead of testTest cost is heavy, and Detectability is eventually or never, and Reversibility is trivial or with-effort | Test it differently | The test is expensive, the failure shows late, and you can undo it. A signal in production finds it sooner and at a lower cost.Instead: Add monitoring or an evaluation set that reports the failure signal. Sample real output, and alert when it changes. |
| R10 | High net scoreNet score is 9 or more | Test mandatory | The expected cost of failure is far above the cost of the test. |
| R11 | Positive net scoreNet score is 5 or more | Test | The expected cost of failure is clearly above the cost of the test. |
| R12 | Small net scoreNet score is 3 or more | Test minimally | The expected cost of failure is slightly above the cost of the test. One cheap test of the main path pays off. |
| R13 | Low net scoreAny other combination | Do not test | The expected cost of failure is below the cost of the test. |
Decisions
- Test mandatoryYes
Do not ship this without tests.
Minimum set Automated tests for the main path, for every known failure mode, and for the boundary values. They run in CI on every change. A second person reviews the tests.
Can skip Cosmetic details that do not touch the stakes.
- TestYes
Write automated tests for this behaviour.
Minimum set Tests for the main path and for the edge cases most likely to break. They run in CI.
Can skip Exhaustive combinations, private details, and cases that the type system already rejects.
- Test minimallyYes
Write one or two tests that fail when the main path breaks.
Minimum set One test of the main path, at the level where it is cheapest. Add a regression test when a bug appears.
Can skip Edge cases, error branches, and coverage targets.
- Do not testNo
An automated test costs more than the failures it prevents. Spend the time elsewhere.
Minimum set No automated test. Keep the checks you already run: the compiler, the linter, code review, and a look at the result after the change.
Can skip Automated tests for this code.
- Test it differentlyTest it differently
A control other than an automated test finds this failure at a lower cost. The rule that applies names the control.
Minimum set The control that the rule names: monitoring, alerts, an evaluation set, a rehearsal on a copy, a staged rollout, or a review.
Can skip Automated tests in the suite for the same failure.
Borderline cases
A combination is borderline when a move of one factor by one step changes the decision from Do not test to a test decision (Test mandatory, Test, Test minimally), or back.
Only a borderline combination can have the conditional verdict. Then the conditions table carries the decision.
What the framework does not account for
Required test evidence. A law, a standard, or a contract can require test evidence: medical device software under IEC 62304, avionics under DO-178C, card payments under PCI DSS. The required evidence comes first. The framework does not replace it.
Safety-critical systems. Do not use the framework for systems where a failure can kill or injure. Use the safety standard of your domain.
Tests that are not about risk. Some tests have other value: tests that you write to learn a codebase, tests that drive a design, tests that document an API. The model does not count this value.
Team rules. If your team requires a test for every change, follow the rule. A shared rule costs less than a debate for each change.
Code that you do not own. For a library or a service that another team owns, test how your code uses it. The framework does not tell you to test the code of other people.
No measured probability. The framework does not measure the probability of failure. Change frequency and Detectability stand in for it. If you have real failure data, use the data.
Version
This is version 1.0. A change to the rules gets an entry in the changelog with a date and the list of answers revised to match.
Version 1.0 revised no answers, so no verdict changed.