Most organizations can show an auditor a disaster recovery plan. Far fewer can show evidence that the plan works today, against the current architecture, with the current team. The traditional annual DR test, scheduled months in advance, scoped narrowly to avoid risk and documented in a slide deck, rarely reflects how a real outage unfolds. Meanwhile cloud architectures change weekly, dependencies multiply and the people who wrote the runbooks move on.
The answer that has gained momentum through 2025 and 2026 is continuous DR testing: frequent, automated, measurable recovery tests built into normal operations. Cloud providers have invested heavily in the tooling, and regulators increasingly expect proof rather than plans. This playbook explains how to build such a program step by step.
Why Annual DR Tests Are No Longer Enough
There are three main reasons. First, architecture drift: a test passed last year says little about a system that has since added new microservices, data stores and third-party dependencies. Second, untested assumptions: many recovery plans rely on DNS changes, database promotions or capacity in a secondary region that have never been exercised under realistic load. Third, regulatory expectation: frameworks such as the EU’s Digital Operational Resilience Act (DORA) require financial entities to test their ICT business continuity and recovery arrangements, and NIS2 and sector regulators in many jurisdictions are moving in the same direction. Paper plans are not sufficient evidence.
The large cloud outages of recent years also showed how dependencies nobody had mapped can turn a regional problem into a global one. Testing is how you discover those dependencies before an incident does.
What the Tooling Now Offers
Cloud providers have moved from offering raw fault injection to offering opinionated, guided resilience testing and orchestrated recovery.
- AWS Resilience Hub recommended resilience testing. Described in an AWS Architecture Blog post in October 2026, the next generation of Resilience Hub recommends tests based on a service’s resources, dependencies and resilience policy, then runs them as AWS Fault Injection Service experiments. The four recommended tests cover Availability Zone recovery, dependency validation, multi-Region isolation and multi-Region recovery, each judged against success alarms and the service’s recovery time objective. Dependencies are discovered from DNS query logs with a 35-day look-back.
- AWS Application Recovery Controller Region switch. A managed capability for orchestrating multi-Region recovery across accounts, using plans made of workflows, steps and execution blocks that can run in parallel or sequence. It supports active/passive failover and active/active traffic shifting, can be triggered manually or by CloudWatch alarms, and provides dashboards during execution.
- Azure Chaos Studio and Azure Site Recovery. Chaos Studio provides managed fault injection for Azure resources, while Site Recovery supports test failovers into isolated networks without affecting production replication.
- Google Cloud and open-source tools. Teams on Google Cloud and Kubernetes commonly combine managed failover features with open-source chaos tools such as Chaos Mesh or LitmusChaos.
The common theme is a shift from “can we break things?” to “can we prove recovery within our objectives, repeatedly, with evidence?”
The Playbook: Building Continuous DR Testing
Step 1: Define Services, Objectives and Success Signals
Start with business services, not infrastructure. For each critical service, document its recovery time objective (RTO) and recovery point objective (RPO), its dependencies, and, crucially, the customer-facing signals that show it is healthy, such as order success rate, login success or payment authorization latency. AWS’s guidance on success alarms makes the same point: a test should pass or fail based on what customers experience, not on whether individual servers are up.
Step 2: Map Dependencies, Including the Hidden Ones
Build a dependency map from architecture diagrams, service catalogs and observed traffic. Tools that analyze DNS logs, service meshes or tracing data reveal dependencies missing from documentation. Classify dependencies as hard, where failure should break the service, or soft, where the service should degrade gracefully. Soft dependencies that turn out to be hard are among the most valuable findings a test can produce.
Step 3: Start Small With Safe, Reversible Tests
Begin in non-production environments with tests that have clear stop conditions. Good first tests include blocking a single soft dependency, terminating instances or pods in one Availability Zone, and restoring a database backup into an isolated environment to measure restore time. Configure stop conditions, alarms that automatically halt an experiment if customer-facing health degrades beyond a threshold, before running anything in production.
Step 4: Codify Recovery as Orchestrated Plans
Manual runbooks with dozens of steps are slow and error-prone under pressure. Convert them into orchestrated plans: database promotion, scaling up standby capacity, updating routing and validating health, executed by automation with human approval at defined points. Store plans as code and review them like any other change. Managed orchestrators such as ARC Region switch, or workflow tools you already use, both work; the point is that the plan is executable, not just readable.
Step 5: Schedule Tests by Tier and Change
Set a cadence based on criticality and change rate:
- Weekly or per release: automated backup restore validation and single-dependency tests in pre-production.
- Monthly: Availability Zone failure tests for critical services in production, within guardrails.
- Quarterly: full regional failover or isolation tests for top-tier services, with business stakeholders informed.
- After significant change: any major architecture change, new dependency or migration triggers the relevant test.
Step 6: Measure Against Objectives and Capture Evidence
Every test run should produce a record: what was tested, when, by whom, the measured recovery time and data loss, whether success alarms stayed healthy, and what failed. Store these reports centrally. They become the evidence regulators and auditors ask for, and the trend data that shows whether resilience is improving.
Step 7: Feed Findings Back Into Engineering
Testing only creates value when findings are fixed. Log every failure as a backlog item with an owner and a due date, and retest after remediation. Track the number of open resilience findings by severity as a standing metric in engineering reviews.
A DR plan that has not been executed recently is a hypothesis. Continuous testing turns it into evidence, and every failed test is a cheap lesson you did not have to learn during a real outage.
Backup Restore Testing: The Most Neglected Test
Failover tests get attention, but many real incidents, especially ransomware and destructive attacks, require restoring from backup rather than failing over to a replica, since replicas faithfully copy corruption and deletion. Restore testing deserves its own discipline:
- Automate restores of representative backups into isolated accounts or subscriptions on a schedule.
- Validate integrity automatically, for example by running application-level checks or checksums, not just confirming the restore job completed.
- Measure restore time at realistic data volumes, because throughput limits often make large restores far slower than expected.
- Test restoring from immutable or logically air-gapped copies using the separate credentials you would rely on during a compromise.
- Include configuration and identity restoration, since data without access is not recovery.
People and Process: The Human Side of Recovery
Automated tests validate systems, but outages are handled by people. Complement technical testing with game days and tabletop exercises that rehearse decision-making: who declares a disaster, who approves failover, how customers and regulators are informed, and how the team coordinates when primary communication tools are affected. Rotate participants so knowledge is not concentrated in a few experts, and include on-call engineers who would actually respond at night.
Communicate with the business before testing in production. Clear scheduling, defined blast radius and visible stop conditions build the trust needed to test frequently.
Metrics and Maturity Stages
A continuous testing program needs a small set of metrics that leadership can follow over time. Useful ones include:
- Test coverage: the share of tier-one and tier-two services that have passed each relevant test type within its scheduled interval.
- Measured versus target recovery: actual recovery time and data loss in the most recent test, compared with RTO and RPO for each service.
- Time since last successful test: per service and per test type, highlighting anything overdue.
- Open resilience findings: count and age by severity, with a target for closing critical items.
- Restore success rate: the share of automated restore tests that complete and pass integrity checks.
- Detection time in tests: how long alarms take to fire after a fault is injected, which often reveals monitoring gaps.
It also helps to describe where the organization is on a simple maturity path, so investments can be sequenced sensibly:
- Documented: DR plans exist, tested annually through scripted exercises in maintenance windows.
- Validated: backup restores are automated and verified; failover is tested at least twice a year with measured results.
- Automated: recovery plans are executable code with approval steps; fault injection runs regularly in pre-production with stop conditions.
- Continuous: scoped tests run in production on a schedule and after significant changes; evidence is captured automatically and reviewed monthly.
- Adaptive: test selection is driven by dependency analysis and incident learnings, and results feed directly into architecture decisions and risk reporting.
Most organizations can move one stage per year with focused effort. The jump from documented to validated delivers the greatest immediate risk reduction, because it exposes the restore-time and dependency surprises that cause the longest real outages. The jump from automated to continuous is mostly cultural: it requires the business to accept carefully controlled testing in production as normal practice, which is far easier once earlier stages have built a track record of safe, well-communicated experiments.
Hypothetical Worked Example: An Online Payments Service
Consider a hypothetical online payments provider running its core API in two AWS Regions in an active/passive configuration, with an RTO of 30 minutes and an RPO of under one minute. Its last annual DR test, a scripted failover during a maintenance window, passed.
The team introduces continuous testing. It defines success alarms based on payment authorization success rate and latency. A dependency validation test in pre-production reveals that the API calls a fraud-scoring service only deployed in the primary Region, a hard dependency that the DR plan had missed. An AZ recovery test in production shows recovery within objectives but exposes an alarm that took several minutes to fire. A restore test reveals that rebuilding the ledger database from backup at production volume takes several times longer than assumed.
Remediation deploys the fraud-scoring service in both Regions, tunes alarm thresholds and moves the ledger to a faster restore pattern combining snapshots and transaction log replay. The runbook becomes an orchestrated Region switch plan with approval steps. The next quarterly regional test completes within the RTO, and the reports go straight into the provider’s DORA resilience testing evidence.
The broader lesson from the example is that each test type found something the others could not. Dependency validation exposed an architectural gap, the zone test exposed a monitoring gap, and the restore test exposed a capacity and process gap. None of these would have surfaced in the scripted annual exercise the provider had relied on, which is precisely why a mix of test types, run regularly, matters more than any single large exercise.
Pitfalls to Avoid
- Testing infrastructure health instead of customer experience. Servers can be up while customers cannot transact.
- Skipping stop conditions. Every experiment needs automatic abort criteria.
- Only testing failover. Ransomware recovery depends on restore, not replication.
- Leaving findings unresolved. A test program that does not drive fixes becomes theatre.
- Testing only in pre-production. Production differs in scale, data and dependencies; bring carefully scoped tests to production over time.
- Forgetting third parties. SaaS and partner APIs are dependencies; test how your service behaves without them.
Outlook: Toward Always-On Resilience Validation
Expect resilience testing to become as routine as automated security scanning. Providers are already recommending tests from architecture analysis and running them through managed fault injection, and AI-assisted tooling will increasingly generate test scenarios from dependency graphs and incident histories. Regulators will continue to raise expectations for documented, repeatable testing, particularly in financial services and critical infrastructure. Organizations that build continuous testing now will find compliance a by-product of good engineering rather than a separate burden.
Conclusion
Continuous DR testing replaces assumptions with evidence. Define services and customer-facing success signals, map dependencies honestly, start with safe tests and clear stop conditions, codify recovery into executable plans, test on a cadence that matches risk and change, and feed every finding back into engineering. With today’s tooling, the barrier is no longer technology; it is the decision to treat recovery as something you practice, not something you document.
How Hanoi Technology Can Help
Hanoi Technology helps teams map dependencies, build automated failover and restore tests, and run continuous DR testing programs with audit-ready evidence.