How often should you test a disaster recovery plan?

Run an automated rebuild drill every month, so a gap surfaces within a month of the change that caused it. Follow the runbook by hand and switch real traffic once a year, and after every move to a new region or provider.

Is testing backups enough for disaster recovery?

No, a backup restore test covers the data but not the servers, secrets, images and DNS that disaster recovery also rebuilds. Keep the restore test and add the rebuild drill.

Should I fail over production traffic to test disaster recovery?

Yes, switch real traffic to the recovered stack once a year in a quiet hour, because only real traffic shows that DNS and certificates follow the move. The monthly drill checks the rebuild without moving customers.

Should I test disaster recovery?

Verdict

Yes

Yes, test disaster recovery with a scheduled drill that rebuilds production in a second region from code and the newest backup copy, checks the recovered data, and fails when the rebuild takes longer than the recovery time you promise.

Why

Yes, test disaster recovery with a drill that rebuilds production elsewhere on a schedule. The typical case is a web application in one cloud region, with Terraform for the servers and a managed PostgreSQL database whose nightly backups are copied to a second region. Blast radius is users, because customers cannot work until recovery ends, and Change frequency is regularly, because each month adds a service or a secret the runbook does not mention. Detectability is never and Reversibility is costly: a broken plan looks complete until the disaster, and an outage of days instead of hours leaves lost trust. Test cost is heavy, because a drill needs a second environment and a day of work, so rule R11 gives Test.

When the decision changes
When	Decision	Why
The service takes orders and payments, so every hour of downtime loses sales	Test mandatory: keep the monthly drill, place a test order in the recovered stack, switch real traffic to it every quarter, and have a second person review the runbook and drill script	Blast radius rises to money, and lost sales cannot be won back
The system holds patient records under HIPAA, whose contingency plan standard asks for periodic testing	Test mandatory: drill on a fixed schedule and keep each drill report as evidence	Blast radius rises to safety-or-legal, because clinics without their records can harm patients
The infrastructure is one application and one managed database that change a few times a year	Test minimally: run one rebuild drill a year and after each infrastructure change	Change frequency falls to rarely, because the plan drifts only when the infrastructure changes
The service stores no data, and every deploy creates its servers and DNS records from the repository in a new environment	Do not write a separate disaster recovery test for a stateless service built fresh on every deploy; keep deploying that way	Detectability moves to immediately and Reversibility to trivial, because each deploy runs the recovery path and nothing is lost
The system is an internal staff tool that people can work without for a few days	Test it differently: run terraform plan against the live account every night and alert when the infrastructure drifts from the code	Blast radius falls to internal and Reversibility to with-effort, so rule R9 swaps the drill for a drift alert
The project is a side project that only you use	Do not test disaster recovery for a side project; keep its backups and infrastructure code	Blast radius falls to none, because you alone bear the outage

What breaks if you don't test

The plan fails on the day you need it, at a step nobody has run. The Terraform code pulls an image from a registry in the lost region, a secret created by hand exists nowhere else, and a 24-hour DNS TTL keeps clients calling the dead region. The team finds each gap at night while customers wait.

What you lose if you over-test

In my practice, a drill that switches real traffic takes one engineer about a day, so a weekly drill costs a fifth of that person's time, and each switch risks a short outage. A drill that stops after terraform apply gives false confidence, because it skips the data restore and the traffic switch.

How to test

Build the drill as a scheduled pipeline. AWS says "the only error recovery that works is the path you test frequently". The minimum set:

Every month, apply the Terraform code in the recovery region, restore the newest backup copy and deploy the production release.
Check that the application serves a real page and that the newest restored row is younger than your recovery point objective.
Fail when the rebuild exceeds your recovery time objective, then destroy the stack.
Once a year, follow the written runbook by hand with an engineer who did not write it, and switch real traffic to the recovered stack in a quiet hour.

Procedure and references

When the answer changes

The service takes payments or holds health records.
The infrastructure changes only a few times a year.
The service stores no data and every deploy builds it from scratch.

Real incident + Code example

The restore Atlassian had never run

Atlassian had prepared for losing a database, a service or an availability zone (post-incident review). On 5 April 2022 a maintenance script received site IDs instead of app IDs and deleted 883 sites of 775 customers. The backups were intact, but Atlassian "did not have the ability to select a large set of customer sites and restore all of their inter-connected products from backups". Engineers built the bulk restore during the outage, and some customers waited up to 14 days. A drill must cover each scenario, including deletion by your own tooling. This is my drill for the typical case:

#!/usr/bin/env bash
# dr-drill.sh: a CI job runs it on the first Monday of each month.
# Production runs in eu-central-1; envs/dr rebuilds it in eu-west-1, and its
# database module restores the most recent cross-region snapshot copy.
set -euo pipefail
cd envs/dr
trap 'terraform destroy -auto-approve -input=false' EXIT   # no drill stack outlives the run
start=$(date +%s)

terraform init -input=false
terraform apply -auto-approve -input=false
../../scripts/deploy.sh dr "$(git describe --tags --abbrev=0)"   # the release production runs

url=$(terraform output -raw base_url)
curl -fsS "$url/login" | grep -q 'name="password"'   # a real page, not only a health check

# Recovery point: the newest restored comment is less than 26 hours old.
age=$(psql -At "$(terraform output -raw database_url)" \
  -c "select extract(epoch from now() - max(created_at))::int from comments")
[ "$age" -lt 93600 ]
# Recovery time: the rebuild finished within the four hours we promise.
[ $(( $(date +%s) - start )) -lt 14400 ]

Sources

Should I test backups?Yes
Is chaos engineering worth it?Code under test: Test it differently
Should I test infrastructure as code?Yes
Should I test in staging?Yes

FAQ

How often should you test a disaster recovery plan?: Run an automated rebuild drill every month, so a gap surfaces within a month of the change that caused it. Follow the runbook by hand and switch real traffic once a year, and after every move to a new region or provider.
Is testing backups enough for disaster recovery?: No, a backup restore test covers the data but not the servers, secrets, images and DNS that disaster recovery also rebuilds. Keep the restore test and add the rebuild drill.
Should I fail over production traffic to test disaster recovery?: Yes, switch real traffic to the recovered stack once a year in a quiet hour, because only real traffic shows that DNS and certificates follow the move. The monthly drill checks the rebuild without moving customers.