Should a small team do chaos engineering?

No, a small team with one application gets more from timeouts and an alert per dependency than from failure experiments. Start experiments when an outage stops orders or loses records for good.

Should I run chaos experiments in production?

Run a chaos experiment in production only after it has passed in staging and you can stop it within a minute. The Principles of Chaos Engineering favour production because only production has real traffic.

What is the difference between chaos engineering and fault injection testing?

Fault injection testing breaks one call inside a test and checks one exact result, while chaos engineering breaks part of a running system under real traffic and watches whether a measured steady state holds. A fault injection test might expect a fallback within two seconds, and a chaos experiment might watch sign-ins per minute.

What do I need before starting chaos engineering?

Chaos engineering needs metrics for the steady state you protect, an alert per dependency, and a way to stop an experiment at once. Without the metrics, an experiment cannot show whether it hurt anyone.

Is chaos engineering worth it?

Answer

No, a chaos engineering program is not worth it for a typical product team: put a timeout on every outbound call, alert on each dependency's latency and error rate, and give on-call a switch that turns off each optional dependency.

Verdict on the code under testTest it differently

Why

Blast radiususers
Change frequencyregularly
Detectabilityeventually
Reversibilitywith-effort
Test costheavy

A chaos engineering program is not worth it for the typical product team; watch each dependency in production instead. The typical case is a web application on a few instances that calls a database and two or three outside APIs. Blast radius is users, because an outage stops customers from working, and Change frequency is regularly, since a new outbound call arrives about once a month. Detectability is eventually: a missing timeout stays hidden until a dependency slows down, often weeks later, and then the outage is under way. Reversibility is with-effort, since the outage ends with a fix and leaves support tickets behind. Test cost is heavy, because an experiment needs a running environment with traffic, steady-state metrics and a person watching, so rule R9 gives Test it differently.

When the decision changes
When	Decision	Why
An outage of the failing dependency stops customers from placing orders	Test mandatory: before each checkout release, cut each checkout dependency in staging under load and check that orders complete	Blast radius rises to money and Reversibility to costly, because missed orders do not come back
After a timeout, the code retries a call that charges a card	Test mandatory: a fake payment provider answers after the timeout, and the test asserts exactly one charge	Blast radius rises to money and Reversibility to costly, because a second charge needs a refund
A queue worker loses records when the message broker restarts, and the senders never send them again	Test: restart the broker in staging under load and compare the messages sent with the messages processed	Reversibility rises to impossible, because a lost record is gone, so rule R9 no longer applies
The dependency is one HTTP call with a fixed timeout and a fallback, such as a related-articles API	Test: an integration test that makes the API hang through Toxiproxy and asserts the fallback within the timeout	Test cost falls to moderate, because the result under failure is exact: the fallback within two seconds
The system is a side project that only you use	Do not run failure experiments on your own side project; restart it when it falls over	Blast radius falls to none, because you alone bear an outage

What breaks if you don't test

A partner API that normally answers in 200 ms starts taking 60 seconds. With no timeout on the HTTP client, every request that calls the partner holds a worker until all workers wait, and pages that never call the partner queue behind them. The load balancer returns 502s, the application log shows nothing, and on-call spends most of the outage finding the slow partner.

What you lose if you over-test

The Principles of Chaos Engineering start every experiment from a steady state measured in production: throughput, error rate and latency. Without those metrics per dependency, a team cannot tell whether an experiment hurt anyone. Staging with a few test users behaves differently from production, so an experiment there can pass on a system that fails under real load. Each game day also costs the team an afternoon.

What to do instead

Watch every dependency in production and keep each failure short:

Put a timeout on every outbound call, including database, cache and queue clients. Python's requests does not time out unless you set a value, so make a missing timeout a review finding.
Record latency and error rate per dependency, and page on-call when either stays high for five minutes with Prometheus alerting rules.
Give each optional dependency a switch that on-call can turn off, named in the runbook beside its alert.

Procedure and references

When the answer changes

An outage on the failing path stops orders or payments.
A failure loses records that nobody can send again.
The failure has one exact result that a test can compare, such as a fallback.

Real incident + Code example

The email check that stopped the ticket form

On a customer-support product I worked on, the ticket form checked email addresses with an outside validation API through requests, with no timeout. One afternoon the API began taking about 70 seconds. All 16 Gunicorn workers were waiting within four minutes, and the whole application returned 502 for 38 minutes. A game day might have found that one call; a review rule now rejects every outbound call without a timeout. We also set a two-second timeout, a switch that skips the check, and the alerts below. They have paged three times since, and each time on-call turned the switch within ten minutes.

# The histogram and counter come from the wrapper around our HTTP client.
groups:
  - name: dependencies
    rules:
      - alert: DependencySlow
        expr: |
          histogram_quantile(0.99,
            sum by (le, dependency) (rate(outbound_request_seconds_bucket[5m]))) > 1
        for: 5m
        labels: { severity: page }
        annotations:
          summary: "p99 to {{ $labels.dependency }} above 1 s; runbook has its switch"
      - alert: DependencyFailing
        expr: |
          sum by (dependency) (rate(outbound_requests_total{result="error"}[5m]))
            / sum by (dependency) (rate(outbound_requests_total[5m])) > 0.05
        for: 5m
        labels: { severity: page }

Sources

Should I test disaster recovery?Yes
Should I load test in production?Code under test: Test it differently
Should I test in staging?Yes
Should I test backups?Yes
Should unit tests make real API calls?Test it differently

FAQ

Should a small team do chaos engineering?: No, a small team with one application gets more from timeouts and an alert per dependency than from failure experiments. Start experiments when an outage stops orders or loses records for good.
Should I run chaos experiments in production?: Run a chaos experiment in production only after it has passed in staging and you can stop it within a minute. The Principles of Chaos Engineering favour production because only production has real traffic.
What is the difference between chaos engineering and fault injection testing?: Fault injection testing breaks one call inside a test and checks one exact result, while chaos engineering breaks part of a running system under real traffic and watches whether a measured steady state holds. A fault injection test might expect a fallback within two seconds, and a chaos experiment might watch sign-ins per minute.
What do I need before starting chaos engineering?: Chaos engineering needs metrics for the steady state you protect, an alert per dependency, and a way to stop an experiment at once. Without the metrics, an experiment cannot show whether it hurt anyone.