Verdict
Yes
Yes, test error handling: write one test for each distinct thing your code does after a failure, such as a retry, a fallback or an error response, and assert what the caller gets and what the handler leaves behind.
Why
- Blast radiususers
- Change frequencyregularly
- Detectabilityeventually
- Reversibilitywith-effort
- Test costmoderate
I test error handling, with one test per distinct reaction to a failure. Blast radius is users, because a broken handler shows customers a wrong error or an empty page. Change frequency is regularly, since handlers change with the code around them and with client library upgrades. Detectability is eventually: a handler runs only when something else fails, and one that swallows the error leaves nothing in alerts. Reversibility is with-effort, because records saved during a failure need a repair script. Test cost is moderate, about an hour to stub a failing dependency, so rule R11 gives Test.
| When | Decision | Why |
|---|---|---|
| The handler decides whether to retry a card charge after the payment provider times out | Test mandatory: a timeout, a retry with the same idempotency key, and exactly one charge | Blast radius rises to money and Reversibility to costly, because a double charge ends in a refund |
| The handler decides what happens to a request when the permission service is unreachable | Test mandatory: make the permission service fail and assert a denial on every protected route | Blast radius rises to safety-or-legal and Detectability to never, because a handler that lets requests through exposes records silently |
| The handler stores nothing and only shows an error notice, such as comments that could not load | Test minimally: one test that forces the failure and asserts the notice | Detectability falls to same-day and Reversibility to trivial: a user reports the broken page and a redeploy fixes it |
| An internal command-line tool prints the error and exits, and the developer running it reads the message at once | Do not test the tool's error handling; keep the non-zero exit code | Blast radius falls to internal, Detectability to immediately and Reversibility to trivial, so rule R8 applies |
| A circuit breaker opens only under production load across several services, and reproducing that load takes days | Test it differently: alert when the breaker opens and when fallback responses rise | Test cost rises to heavy while Detectability stays eventually, so rule R9 applies |
| The error handling belongs to a one-time backfill that skips the rows it cannot convert | Test it differently: run the backfill on a copy of production data and review every skipped row | Change frequency falls to once, so a test in the suite would never run again |
What breaks if you don't test
The normal suite never runs a handler, because every test uses a dependency that answers. A library upgrade renames the timeout exception, the except clause stops matching, and a retry that used to hide a slow second becomes a failed request. A handler that catches too broadly turns a bug in your own code into an empty result with no error, and customers notice weeks later.
What you lose if you over-test
An HTTP client raises connection errors, read timeouts, TLS errors and one error per status code. When all of them reach the same handler, a test for each one checks one branch many times. Tests that assert log text or the delay between retries break when someone tunes the backoff, with no change a user sees.
How to test
Test through the public method where the failure enters your code, at unit or service level. Replace the dependency with a stub that raises the real client's error, for example with side_effect in Python.
- List what your code does after a failure: each retry, fallback, rollback and error response. Write one test for each, not one per exception type.
- Assert the response the caller gets, what the database holds, and the signal the handler sends, such as a metric.
- Add one test that an unexpected error still reaches the error tracker, so a broad
catchcannot hide your own bugs. - Check that error responses show no stack trace, per the OWASP Error Handling Cheat Sheet.
When the answer changes
- The handler decides about money or access.
- The failure appears only under production load across several services.
- The handler stores nothing and shows an error that users report the same day.
Real incident + Code example
The search that said "no results"
On a helpdesk product I worked on, the customer portal searched help articles in a search cluster. The handler caught the client's timeout and returned an empty list, so the page said "No articles match your question". After a cluster upgrade slowed queries, about one search in 12 hit the 500 ms timeout, and alerts stayed quiet. Three weeks later a support lead asked why password reset tickets had doubled when an article already answered them. The fix returned an "unavailable" state, counted each timeout, and came with this test:
class TimingOutSearch:
def query(self, text, limit):
raise SearchTimeout("no answer after 500 ms")
def test_timeout_is_reported_not_shown_as_no_results(metrics):
search = ArticleSearch(client=TimingOutSearch(), metrics=metrics)
result = search.find("reset password")
assert result.status == "unavailable"
assert metrics.count("article_search.timeout") == 1
Related questions
FAQ
- Is it necessary to test all the exception cases?
You do not need a test for every exception; write one test for each distinct thing your code does after a failure. Exceptions that reach the same handler share one test. Add a test when a handler treats one error differently, such as retrying a timeout but not a validation error.
- Should I test retry logic?
Test retry logic with one test that a retryable error is retried and one that a permanent error is not. Assert the number of attempts and the final result, not the delay between attempts.
- How do I test error handling when a dependency fails?
Replace the dependency with a stub that raises the real client's error, call your public method, and assert what the caller gets and what your code stored. When forcing the failure takes days of setup, alert on its trace in production instead.