Is it safe to run load tests against production?

A load test against production is safe only with guards: the quietest hour, test accounts that email and payment code skip, read-only paths, and an automatic stop when errors or latency rise. Without those guards, the test sends real messages, writes fake records and slows real customers.

Should I load test staging or production?

Load test a production-sized staging copy for regular checks, and production only before a traffic peak that the copy cannot reproduce. Only production has the real CDN, database size and service limits, so a staging result describes a smaller system.

How do I load test production without affecting users?

Load test production at the quietest hour, raise the load in steps, and stop automatically when the error rate passes 1% or the 95th percentile latency passes your limit. Use flagged accounts that email and payment code skip, and read only pages that change no stock.

Should I load test in production?

Answer

No, do not load test production as a routine check: watch its capacity instead, with an alert when database CPU, connection pool use or the 95th percentile latency at the daily peak rises after a release.

Verdict on the code under testTest it differently

Why

Blast radiususers
Change frequencyregularly
Detectabilityeventually
Reversibilitywith-effort
Test costheavy

Do not load test production as a routine check; watch its capacity there instead. In the typical web application, Blast radius is users, because overload shows customers errors, and Change frequency is regularly, since any monthly release can add a slow query. Detectability is eventually, because lost headroom raises no error until the next busy day, and Reversibility is with-effort, because failed requests leave retried jobs and support tickets. Test cost is heavy: a load test needs scripts that model real traffic and a system the size of production, and in production it slows real users, so rule R9 gives Test it differently.

When the decision changes
When	Decision	Why
A ticket on-sale will bring several times the usual traffic to the event and seat pages	Test mandatory: the week before, load test the event and seat pages in production at the quietest hour at twice the expected peak, with flagged test accounts and an automatic stop on errors	Blast radius rises to money and Reversibility to costly, because buyers who get errors during the on-sale do not come back
A contract promises customers a response time and sets penalties when you miss it	Test mandatory: before each release, load test a production-sized copy and compare its 95th percentile latency with the contracted limit	Blast radius rises to safety-or-legal, because a missed response time breaks a contract
Under overload the service drops incoming events that the sender never resends, such as webhooks from a partner with no retries	Test: before each release that changes intake, replay a recorded hour of events at twice the peak rate into a production-sized copy and fail when any event is missing	Reversibility rises to impossible, because a dropped event is gone, so rule R9 no longer applies
Most requests call a partner API whose terms forbid load tests, and its rate limit is the likely ceiling	Test it differently: get the partner's rate limit in writing, alert on its 429 responses, and open new features to users in stages	Test cost rises to prohibitive, because no permitted test reaches the partner's rate limit; rule R4 keeps Test it differently
The system is an admin tool that 30 staff use in office hours	Do not load test an admin tool; add capacity when staff say it is slow	Blast radius falls to internal and Detectability to same-day, because staff who wait report it that day

What breaks if you don't test

A release adds a second database query to the product page. At the afternoon peak database CPU rises from 35% to 60%, and no alert fires, because the alerts watch errors. Three weeks later a newsletter doubles the traffic, the connection pool runs out, and the page returns 503 errors to customers for an hour.

What you lose if you over-test

A load test against production spends the capacity it measures: at twice the peak it slows real users and starts autoscaling you pay for. Test traffic that signs up or orders writes fake records into production data and calls paid services. The Google SRE book notes that tests with real traffic are more realistic than synthetic load, "at the risk of causing user-visible pain" (Addressing Cascading Failures).

What to do instead

Watch capacity in production:

Graph saturation, the SRE book's name for how full the most constrained resource is: database CPU, connection pool use, queue depth. Alert when the daily peak passes 70%, which leaves room for about 40% more traffic.
Compare each endpoint's 95th percentile latency at the peak hour after a release with the week before.
Before a traffic peak that calls for a production run, set k6 thresholds that stop the run when errors or latency rise.

Procedure and references

When the answer changes

A date will bring more traffic than production has handled, and money moves on the busy path.
A contract names a response time.
Overload loses data that nobody sends again.

Real incident + Code example

The rehearsal that stopped our email

On a ticketing product I worked on, we load tested production at 4 a.m. before the largest on-sale of the year. At 1.6 times the expected peak the connection pool ran out, and we doubled it before the sale. The script also signed up 20,000 accounts at a domain that does not exist, and each sign-up sent a welcome email through Amazon SES, which may pause sending at a bounce rate of 10% (SES FAQ). SES paused ours, and for 26 hours buyers got no password resets or order confirmations. Our production runs now use flagged accounts created in advance and stop before the cart:

// Production rehearsal at the quietest hour. Accounts carry is_load_test,
// so email, text and payment code skip them. Reads only: no cart, no seats held.
import http from 'k6/http';
import { SharedArray } from 'k6/data';
import { check } from 'k6';

const accounts = new SharedArray('accounts', () => JSON.parse(open('./accounts.json')));
const stop = (threshold) => [{ threshold, abortOnFail: true, delayAbortEval: '1m' }];

export const options = {
  stages: [
    { duration: '10m', target: 400 }, // expected peak: 400 concurrent buyers
    { duration: '10m', target: 800 }, // twice the peak
    { duration: '5m', target: 0 },
  ],
  thresholds: { http_req_failed: stop('rate<0.01'), http_req_duration: stop('p(95)<800') },
};

export default function () {
  const params = { headers: { Authorization: `Bearer ${accounts[__VU % accounts.length].token}` } };
  check(http.get('https://tickets.example.com/events/final', params), { event: (r) => r.status === 200 });
  check(http.get('https://tickets.example.com/events/final/seats', params), { seats: (r) => r.status === 200 });
}

Sources

Should I write performance tests?Test it differently
Should I test in staging?Yes
Is chaos engineering worth it?Code under test: Test it differently
Should I test with production data?Test it differently
Should I test rate limits?Yes

FAQ

Is it safe to run load tests against production?: A load test against production is safe only with guards: the quietest hour, test accounts that email and payment code skip, read-only paths, and an automatic stop when errors or latency rise. Without those guards, the test sends real messages, writes fake records and slows real customers.
Should I load test staging or production?: Load test a production-sized staging copy for regular checks, and production only before a traffic peak that the copy cannot reproduce. Only production has the real CDN, database size and service limits, so a staging result describes a smaller system.
How do I load test production without affecting users?: Load test production at the quietest hour, raise the load in steps, and stop automatically when the error rate passes 1% or the 95th percentile latency passes your limit. Use flagged accounts that email and payment code skip, and read only pages that change no stock.