Verdict
Test it differently
Watch performance in production instead of load testing every change: record the latency of each endpoint and alert when its 95th percentile rises after a release.
Why
- Blast radiususers
- Change frequencyregularly
- Detectabilityeventually
- Reversibilitytrivial
- Test costheavy
Test performance differently: alert on endpoint latency in production instead of load testing every change. In the typical web application a change slows a page without breaking it: Blast radius is users, and Change frequency is regularly, since any monthly change can add a slow query. Detectability is eventually, because a page that takes three seconds instead of 300 ms raises no error. Reversibility is trivial, since a slow page stores nothing wrong and a fix ends the damage. Test cost is heavy, because a load test needs a production-sized environment and its timings vary between runs, so rule R9 gives Test it differently.
| When | Decision | Why |
|---|---|---|
| The slowdown you fear is an N+1 query on a list page, one extra database query per row | Test minimally: one test that asserts the page runs the same number of queries for 2 rows and for 30 | Test cost falls to moderate, because a query count is an exact number that does not vary between runs |
| A pure function handles input that grows with customer data, such as matching imported contacts | Test minimally: one unit test that runs the function on 100,000 generated rows under a time limit far above the expected time | Test cost falls to moderate: at 100,000 rows a quadratic loop is thousands of times slower than an n log n one, so a generous limit does not flake |
| A sale or a launch will bring several times the usual traffic to checkout | Test mandatory: load test checkout at twice the expected peak on a production-sized copy before the date | Blast radius rises to money and Reversibility to costly, because orders that time out during the sale do not come back |
| A contract promises customers a response time and sets penalties when you miss it | Test mandatory: run a load test before each release and compare its 95th percentile with the contracted limit | Blast radius rises to safety-or-legal, because a missed response time breaks a contract |
| The slow page is an internal report that five colleagues open every day | Do not test: fix the query when a colleague says the report is slow | Blast radius falls to internal and Detectability falls to same-day, because the people who wait tell you that day |
What breaks if you don't test
A change adds one database query per row to a list page. Test data has a dozen rows, so the page stays fast in CI. A customer with a thousand rows waits seconds on every load, and no alert fires, because the alerts watch errors. You hear about it weeks later, if the customer complains.
What you lose if you over-test
A load test on every pull request needs an environment and data sized like production, or its numbers describe a different system. Shared CI runners change speed between runs, so a threshold tight enough to catch a small slowdown fails on unchanged code, and the team learns to rerun it until it passes. A timing assertion inside a unit test fails on a busy laptop and teaches people to ignore red builds.
What to do instead
Watch production, and test only the slowdowns a test can count:
- Record request latency per endpoint as a histogram and alert when the 95th percentile rises after a release. The Google SRE book shows why averages hide the slow tail: at an average of 100 ms, 1% of requests can take 5 seconds.
- Add a query-count test to each list page, such as Django's
assertNumQueries, with 2 rows and with 30, so a query per row changes the count. - Before a known traffic peak, run a load test with Locust at twice the expected peak against a production-sized copy.
When the answer changes
- A date on the calendar will bring more traffic than the system has handled.
- Money moves on the slow path, as in checkout or payment.
- A contract or a published promise names a response time.
Real incident + Code example
The board that took seven seconds
On a recruiting product I worked on, a pull request added each candidate's latest interview stage to the pipeline board, with one query per candidate. Our test data had 12 candidates per job; the largest customer had 1,400 on one board, which went from 300 ms to seven seconds. We alerted on server errors only, and the timeout was 30 seconds, so nothing fired. A recruiter there mentioned the slow board on a renewal call 23 days later. The fix loaded all interviews in one query with prefetch_related, added a latency alert per endpoint, and added this test, which fails when the query count grows with the candidates:
from django.test import TestCase
from .factories import CandidateFactory, JobFactory, UserFactory
class PipelineBoardQueryTest(TestCase):
def test_query_count_stays_flat_as_candidates_grow(self):
self.client.force_login(UserFactory())
job = JobFactory()
url = f"/jobs/{job.id}/board/"
CandidateFactory.create_batch(2, job=job)
with self.assertNumQueries(5):
self.client.get(url)
CandidateFactory.create_batch(30, job=job)
with self.assertNumQueries(5): # 36 before the fix
self.client.get(url)
Related questions
FAQ
- Should one test for algorithmic complexity?
Yes, test algorithmic complexity when a pure function's input grows with customer data: run the function on 100,000 generated items under a time limit far above the expected time. At 100,000 items, a quadratic loop is thousands of times slower than an n log n one, so a generous limit does not flake.
- Should unit tests cover stress testing?
No, unit tests should not cover stress testing, because a stress test needs a deployed system under load and minutes of run time. Run stress tests as a separate job against a production-sized environment before a traffic peak.
- Should performance tests run in CI?
Run exact performance checks in CI, such as query counts per page, and keep timed load tests out of pull requests. Timings on shared CI runners vary between runs, so run load tests before a release on a dedicated environment.