On October 20, 2025, a race condition in the automation that manages DNS records for Amazon DynamoDB in AWS’s Northern Virginia region left a critical endpoint with an empty DNS record. Within hours, the effects had cascaded into EC2 instance launches, Network Load Balancer health checks, Lambda, container services, identity sign-in and many customer applications far beyond the region itself. The incident was a reminder that cloud architecture is only as resilient as its least understood dependency. It also arrived as regulators, particularly in European financial services, were formalizing expectations about operational resilience and concentration risk. This article distills the architectural lessons into a practical playbook: how to reduce blast radius with cells, design for static stability, map hidden dependencies and decide rationally when multi-region or multicloud is worth the cost.
What the October 2025 Incident Actually Teaches
AWS’s post-event summary is unusually detailed and worth reading in full. The disruption ran from late on October 19 to the afternoon of October 20 Pacific time, roughly 14.5 hours in total for some effects. A delayed automation component applied an outdated DNS plan over a newer one, and a clean-up process then deleted it, removing all IP addresses for the regional DynamoDB endpoint. Manual intervention was required. DynamoDB API errors lasted about three hours, but downstream systems took far longer: EC2’s instance management depended on DynamoDB and then struggled with a recovery backlog, and load balancer health checks removed capacity during the confusion.
Four architectural lessons stand out:
- Control planes fail differently from data planes. Running instances largely kept working; launching new ones and changing configuration did not. Architectures that need to scale out or reconfigure during an incident are more exposed than those that can ride it out with existing capacity.
- Recovery is its own failure mode. Backlogs, retries and thundering herds extended the impact well beyond the original fault.
- Hidden regional dependencies are everywhere. Workloads hosted elsewhere were affected through identity, global services or third-party SaaS that depended on the region.
- Automation needs guardrails. AWS’s remediation included velocity controls to limit how much capacity automated health checks can remove, a principle every team can apply to its own automation.
Principle 1: Reduce Blast Radius With Cell-Based Architecture
AWS’s Well-Architected guidance on cell-based architecture borrows the metaphor of a ship’s bulkheads. A workload is split into multiple isolated, complete copies called cells, each serving a subset of customers or resources selected by a partition key. A thin cell router directs each request to the right cell, and a control plane provisions cells and moves customers between them. If one cell fails, whether from a bad deployment, a poison-pill request or an infrastructure fault, only its share of traffic is affected. The guidance gives a simple illustration: with ten cells, a failure in one leaves 90% of requests unaffected.
Designing Cells in Practice
- Choose the partition key carefully. It should match the natural grain of the service, such as tenant, account or customer region, so that most requests touch only one cell.
- Keep the router thin. The router is the one shared component; it should do nothing but route, with minimal dependencies and aggressive caching of the routing map.
- Make cells complete. Each cell needs its own compute, data stores and queues. Shared databases quietly undo the isolation.
- Size cells for safe failure. Smaller cells mean smaller blast radius but more operational overhead. Most teams start with a handful and grow.
- Deploy cell by cell. Progressive rollouts across cells turn every deployment into a controlled experiment.
Cells are not only for hyperscale companies. The guidance notes that cell-based design does not necessarily multiply infrastructure: an application with thirty hosts can keep thirty hosts, grouped into cells behind a router. For multi-tenant SaaS products, cells are often the single most effective resilience investment.
Principle 2: Design for Static Stability
A statically stable system keeps working during a dependency failure without needing to make changes. In practice, that means avoiding designs where recovery depends on the very control planes that might be impaired:
- Pre-provision capacity in each Availability Zone for the loss of one zone, rather than relying on auto scaling to launch replacements during an incident.
- Cache configuration and credentials locally with sensible lifetimes, so a temporary inability to reach a configuration or identity service does not stop the data plane.
- Prefer data-plane operations on the critical path; reserve control-plane calls (creating resources, changing DNS, modifying load balancers) for planned changes.
- Fail open or fail safe deliberately, deciding in advance which features degrade gracefully when a dependency is unavailable.
Static stability costs money, typically in idle headroom. The decision should be explicit and tied to each workload’s recovery objectives, not applied uniformly.
Principle 3: Map and Tier Your Dependencies
Many organizations discovered during the October incident that they had dependencies they had never documented: a global service anchored in a single region, a SaaS identity provider hosted in the affected area, a monitoring tool that went dark exactly when it was needed. A dependency review should cover:
- Cloud services used by each workload, including their regional or global scope and the region their control planes live in.
- Identity providers, DNS, certificate authorities and secrets services on the authentication path.
- Third-party SaaS and APIs, and where those vendors host their own critical components.
- Observability, paging and incident communication tools, which must remain available during the incident they are meant to manage.
- CI/CD and artifact registries, which you will need if recovery requires a deployment.
Tier each dependency as hard (the workload fails without it), soft (the workload degrades) or deferred (needed only for changes). Then engineer hard dependencies to be redundant or cached, and confirm that soft dependencies really do degrade gracefully by testing it.
Make Recovery Itself Safe
The longest part of many outages is not the original fault but the recovery storm. Build in protections: retries with exponential backoff and jitter, capped so clients do not amplify an outage; circuit breakers and load shedding that protect recovering services from full traffic; velocity limits on automation, so health checks or remediation scripts cannot remove too much capacity at once; queue-based decoupling with bounded backlogs and clear policies for discarding stale work; and runbooks for manual intervention, because the October incident showed that some failure states cannot be resolved by automation alone.
Prove It With Game Days
Architecture diagrams describe intended behavior; only testing reveals actual behavior. Run regular game days that simulate the failures you have designed for: block access to a dependency, impair a zone, inject latency into the identity path or fail over a database. Managed fault-injection services from the major clouds make these experiments safer to run in pre-production and, with care, in production. Start small, define a clear abort condition, and capture what surprised the team. Each surprise is a dependency or assumption you did not know you had, and fixing it before a real incident is far cheaper than discovering it during one.
Multi-Region and Multicloud: When Are They Worth It?
After a major outage, boards often ask why everything is not running in multiple regions or multiple clouds. The honest answer is that both options carry real costs and new failure modes, and the right choice differs by workload.
Multi-AZ as the Baseline
For most workloads, a well-designed multi-AZ deployment with static stability provides strong resilience at reasonable cost. It protects against the most common failures: individual hosts, racks and zones. Combined with cells and safe recovery patterns, it also contains many software and deployment failures, which in practice cause more incidents than infrastructure faults.
Multi-Region for Critical Services
Active-passive or active-active multi-region designs protect against regional events like October 2025. They require data replication with clear consistency trade-offs, regional independence of control-plane dependencies, regularly tested failover and, crucially, a failover decision process that does not itself depend on the failed region. Reserve them for services whose downtime cost justifies substantially more infrastructure and significantly more complexity.
Multicloud for Specific Risks
Running the same workload across providers addresses provider-level risk and some regulatory concentration concerns, but it forces lowest-common-denominator designs and doubles skills requirements. A more practical pattern is selective multicloud: keep critical dependencies such as DNS, identity, status pages and incident communication independent of your primary provider, and maintain tested exit plans for the most critical workloads rather than running them everywhere.
A Simple Decision Framework
- Quantify the cost of downtime per hour for each critical business service, including regulatory and reputational impact.
- Set recovery time and recovery point objectives per service, agreed with business owners rather than assumed by engineers.
- Map each objective to the cheapest architecture that meets it: multi-AZ with static stability, warm standby in a second region, or active-active.
- Identify shared dependencies that would defeat the chosen pattern, and remove or duplicate them.
- Test the result at least annually, and after any significant architectural change.
The Regulatory Angle
Operational resilience is no longer only an engineering preference. In the EU, the Digital Operational Resilience Act (DORA) has applied to financial entities since January 2025, requiring ICT risk management, resilience testing, incident reporting and third-party risk management. On November 18, 2025, the European Supervisory Authorities published the first list of 19 critical ICT third-party providers subject to direct oversight, which includes hyperscale cloud providers. Law firm Morgan Lewis noted that, as a last resort, supervisors can require financial entities to suspend or terminate use of a designated provider’s services. Even outside finance, customers and auditors increasingly ask for evidence of tested recovery, documented dependencies and credible exit strategies. The architecture practices in this article are exactly the evidence they want.
A Worked Example: Hardening a Multi-Tenant SaaS Platform
Consider a hypothetical B2B SaaS company running its platform in a single AWS region across three Availability Zones. During a regional disruption, its application kept serving existing users, but customers could not sign in because its identity provider’s session service was affected, its status page went down, and its auto scaling could not replace unhealthy instances.
The architecture team responds in three steps. First, a dependency review identifies seven hard dependencies, including the identity service, DNS and the status page. The status page and incident communications move to an independent provider; token validation is changed to work from cached signing keys for a defined period; and each zone is provisioned with enough headroom to absorb the loss of another without launching new instances.
Second, the platform is reorganized into cells partitioned by tenant, with a thin router backed by a replicated routing table. Deployments now roll out one cell at a time, and the largest enterprise tenants receive dedicated cells. Third, the two most critical services, authentication and the public API, gain a warm standby in a second region with asynchronous replication and a quarterly failover exercise. The rest of the platform stays single-region with multi-AZ static stability, a deliberate cost decision documented for customers and auditors.
The next time a dependency fails, the impact is limited to a fraction of tenants, sign-in continues and customers receive timely updates. The company did not buy perfect resilience; it bought predictable, explainable resilience aligned to business priorities.
Pitfalls to Avoid
- Assuming multi-AZ means no single points of failure while control planes, global services and SaaS dependencies remain regional.
- Untested failover: a disaster recovery plan that has never been exercised is a hypothesis.
- Failover decisions that depend on the failed region, such as runbooks, dashboards or credentials stored there.
- Aggressive retries that turn a partial outage into a total one.
- Uniform resilience targets that overspend on low-value workloads and underinvest in critical ones.
Resilience is not about never failing. It is about limiting how much fails, staying stable without emergency changes, knowing every dependency that matters and recovering in a controlled way.
Conclusion: Engineer for the Outage You Will Have
The October 2025 disruption did not reveal new principles; it showed what happens when well-known ones are applied unevenly. Cell-based architecture limits blast radius. Static stability keeps systems running without risky changes. Dependency mapping exposes the hidden links that turn regional problems into global ones. Safe recovery patterns prevent the cure from becoming the second outage. Multi-region and multicloud are powerful tools when applied selectively to the workloads that justify them. Organizations that make these choices explicitly, test them regularly and document them for regulators and customers will handle the next major incident far better than those that rely on hope.
How Hanoi Technology Can Help
Hanoi Technology helps teams assess dependencies, design cell-based and statically stable architectures, and implement tested multi-region recovery on AWS, Azure and Google Cloud.