Skip to main content
All posts
Cloud Architecture24 min read

Hyperscaler Outages: Designing for Control-Plane Failure

What the AWS, Azure, Google Cloud and Cloudflare post-incident reports teach about control-plane failure, and six static-stability designs that survive it.

Published Updated: 15 November 2026

At 11:48 PM Pacific time on 19 October 2025, the DNS record for DynamoDB's regional endpoint in us-east-1 became empty. No hardware failed and nobody attacked anything. Two copies of an automation tool raced each other, and the slower one won. AWS fixed the DNS record within three hours. The event did not end until 2:20 PM the next afternoon, because the systems that launch instances, balance load and issue credentials had to dig themselves out of the hole the first three hours created. Ten days later, Azure Front Door spent almost eight and a half hours crashing at the edge because of configuration metadata that every health check had approved.

Neither was a data centre fire. Both started in a control plane: the machinery that changes things, not the machinery that serves them. So did Google Cloud's June 2025 outage, Cloudflare's in November 2025 and, arguably, CrowdStrike's July 2024 content update. This teardown is for the people who decide what "resilient" means in their architecture: CTOs, platform leads, cloud architects and the risk owners who sign off DORA and NIS2 continuity plans. We map the six places where a control plane leaks into serving, and the static-stability design that cuts each one.

TL;DR / Key takeaways

  • The big outages are control-plane outages. Every major hyperscaler incident we examined from 2021 to 2026 started in configuration, DNS automation, capacity management or a global policy push, not in the servers that run your code.
  • The trigger is short; the tail is long. In AWS's October 2025 event, the DNS fault was repaired in under three hours. Lambda, EC2 launches and container services took more than 14, because recovery itself depended on overloaded control planes.
  • Static stability is the design answer. If your recovery needs to launch instances, call a management API or issue new credentials during the incident, it competes with every other customer doing the same thing. Pre-provision, cache and pre-wire instead.
  • Your own recovery path is the most common weak spot. Runbooks that need the console, a management API or a global sign-in fail exactly when you run them. The Azure Portal survived the Front Door outage because it had a rehearsed path away from it.
  • Regulators now expect this to be tested. DORA requires tested exit strategies and continuity plans, and AWS, Google Cloud and Microsoft are on the ESAs' list of designated critical ICT third-party providers. "The cloud is resilient" is no longer an acceptable control.
Where the control plane leaks into serving. Warm arrows 1–6 are runtime dependencies on provider change machinery; green tags A–F are the static-stability choices that cut them.
Where the control plane leaks into serving. Warm arrows 1–6 are runtime dependencies on provider change machinery; green tags A–F are the static-stability choices that cut them.

Download: Editable draw.io file · Open in diagrams.net · Chart data (CSV)

Why this matters now

Two things changed. The first is the evidence. Providers now publish detailed, time-stamped post-incident reviews: AWS's October 2025 summary names the internal subsystems involved and gives a timeline for each affected service, and Google's June 2025 report says outright that the failing code "did not have appropriate error handling nor was it feature flag protected". Architects can now design against documented failure modes rather than a generic "region down" scenario.

The second is regulation. DORA has applied to EU financial entities since 17 January 2025. In November 2025 the European Supervisory Authorities published their first list of designated critical ICT third-party providers: 19 companies, including Amazon Web Services EMEA, Google Cloud EMEA and Microsoft Ireland Operations. Those providers now sit under direct EU oversight, while the financial entities that use them must still show that their own exit plans are "sufficiently tested". NIS2 sets a similar expectation for essential and important entities in other sectors. A continuity plan that assumes the provider's control plane will be there to execute it does not pass that test.

The common misreading is "us-east-1 is cursed" or "the cloud is fragile". The data planes actually held up well. In December 2021, AWS's report notes that "running EC2 instances were unaffected" while the EC2 APIs were impaired for seven hours. In October 2025, AWS again states that running instances were not affected while DWFM, the system that manages the physical hosts, failed. Workloads that only needed to keep running kept running. Workloads that needed to change something to survive were in trouble.

The six places a control plane leaks into serving

AWS's Builders' Library gives the cleanest definition. "A control plane is the machinery involved in making changes to a system", while the data plane "is the daily business of those resources". The same article adds two observations that frame everything below. Control planes are "necessarily more complex" than data planes. And it is "typical for the availability of the data plane to be even more critical" to customers. Static stability follows from those two facts: "In a statically stable design, the overall system keeps working even when a dependency becomes impaired."

The hero diagram pairs each data-plane function with the control plane it can quietly depend on. Numbers match the warm callouts; letters match the green static-stability choices.

① Global configuration push

What failed. On 29 October 2025, Microsoft's post-incident review describes customer configuration changes made across two control-plane build versions that "resulted in incompatible customer configuration metadata being generated". The edge's data plane crashed while processing it asynchronously. Because the crash was asynchronous, health checks passed during rollout, the metadata propagated globally, and it overwrote the "last known good" snapshot that rollback depended on. Microsoft's engineers blocked all configuration propagation at 17:30 UTC and pushed a corrected last-known-good to every edge site by 17:50. Reloading configuration after a master-process crash then took, in Microsoft's words, "approximately 4.5 hours". Impact lasted from 15:41 UTC until 00:05 UTC.

Google's June 2025 outage has the same shape: a policy change with blank fields "replicated globally within seconds" and crashed Service Control everywhere at once, through a code path shipped two weeks earlier and never exercised in rollout. At Cloudflare in November 2025, a database permissions change doubled an internally generated bot-management file past a hard limit of 200 features, and the proxy panicked worldwide. CrowdStrike's Channel File 291 was live for only 78 minutes on 19 July 2024; Microsoft estimated that it affected 8.5 million Windows devices. And on 30 September 2026, an Azure gateway update resumed in many regions at once and overloaded the regional gateway control plane; some ExpressRoute gateways lost connectivity until they fell back to their last-known-good configuration.

The lesson. Configuration is code that skips code review, propagates faster than binaries and reaches every cell at once.

Static-stability choice (A). Treat every configuration push, including your own feature flags, WAF rules and policy bundles, like a deployment. Stage it by cell or region with a bake time, validate it synchronously before the data plane loads it, and keep a last-known-good copy that a bad push cannot overwrite. Cloudflare has committed to hardening the ingestion of its own generated configuration as it would user input; in December 2025 Microsoft announced a "Food Taster" process that validates configuration before serving processes load it. When validation fails, keep serving the old configuration: fail static, not fail closed.

② DNS and service discovery

What failed. In AWS's October 2025 event, DynamoDB's DNS is managed by two components: a DNS Planner that produces plans and several DNS Enactors that apply them. One Enactor was delayed. It applied an older plan over a newer one, and a clean-up process deleted the plan it had overwritten. AWS's summary is blunt: "all IP addresses for the regional endpoint were immediately removed." Every client that resolved dynamodb.us-east-1.amazonaws.com got nothing back, including AWS's own services. All records were restored by 2:25 AM, and recovery completed at 2:40 AM as cached records expired.

The lesson. DNS is a robust data plane until the automation that writes records joins the request path. A stale answer usually points at something that still works; an empty one points at nothing.

Static-stability choice (B). Prefer regional endpoints over global ones so that one region's DNS automation cannot break another's. Choose TTLs deliberately: very short TTLs make you dependent on the authoritative system being up every few seconds. Most importantly for your own DNS automation, add the guard AWS had to add: refuse to publish an empty or drastically smaller record set without a human in the loop. AWS disabled the DNS Planner and Enactor automation worldwide after the event, and says it will add protections "to prevent the application of incorrect DNS plans" before re-enabling it.

③ Capacity and launch

What failed. This is where the October 2025 tail came from. EC2's DropletWorkflow Manager (DWFM) holds a lease on every physical host ("droplet"). Its state checks depend on DynamoDB, so while DynamoDB was unreachable, leases quietly timed out. When DynamoDB came back at 2:25 AM, DWFM had to re-establish leases with the entire fleet at once and, in AWS's words, entered "a state of congestive collapse". New launches failed with "insufficient capacity" errors. Engineers throttled incoming work and restarted DWFM hosts at 4:14 AM; leases were back by 5:28 AM. Network Manager then had a backlog of network state to propagate to new instances, which did not clear until 10:36 AM. EC2 APIs and launches were back to normal at 1:50 PM.

The lesson. "Autoscaling will add capacity in the healthy zone" assumes a capacity control plane that is healthy and not swamped by every other customer with the same plan. In October 2025 it was neither.

Static-stability choice (C). Pre-provision. Run enough capacity in each zone to absorb the loss of one zone without launching anything. Microsoft's reliability guidance gives the arithmetic: multiply peak instance count by N/(N−1), where N is the number of zones you use, so three zones means 1.5 times peak. For a warm standby region, keep it warm enough to take real traffic for the first hours on what is already running. Scale-out during an incident is a bonus, not a plan.

④ Health checks and automated failover

What failed. Starting at 5:30 AM on 20 October, the NLB health-check subsystem was "bringing new EC2 instances into service while the network state for those instances had not yet fully propagated". Health checks alternated between failing and healthy, so nodes were removed from DNS and put back again, and the failures triggered automatic zonal DNS failover that took capacity out of multi-AZ load balancers. STS saw a second wave of errors from 8:31 to 9:59 AM as a result. At 9:36 AM engineers disabled automatic health-check failover for NLB, which stopped the churn. AWS's main change for NLB is "a velocity control mechanism to limit the capacity a single NLB can remove" during failover.

The Azure Sweden Central incident on 29 September 2026 shows the same pattern at a smaller scale. A deployment cleared an internal service's caches during peak hours, traffic to its backend rose about twentyfold, and a health check that depended on database connectivity sent nodes into a restart loop until the check was changed.

The lesson. Automated failover is a control plane. When everything looks unhealthy at once, the likeliest cause is the health check, not the fleet.

Static-stability choice (D). Make health checks test the thing that serves, not its dependencies. Cap how much capacity any automation may remove in a given window. Configure a fail-open threshold: if more than, say, half your targets look sick, stop removing them and route to all of them. A partially failing fleet serves more users than an empty one.

⑤ Identity and token issuance

What failed. Identity is the dependency nobody draws. In October 2025, STS returned errors from 11:51 PM until 9:59 AM, with a partial recovery in between, and users with IAM Identity Center in us-east-1 could not sign in. Microsoft's Front Door review lists parts of Entra ID among the affected services. AWS's Fault Isolation Boundaries whitepaper states that the IAM control plane is hosted in us-east-1, while its data plane runs in every region.

The lesson. A workload that holds valid tokens keeps working. A workload that needs a fresh token from a global endpoint in the middle of an incident does not.

Static-stability choice (E). Use regional token endpoints; AWS notes that credentials from a regional STS endpoint are valid in every region. Refresh tokens well before expiry, so a two-hour issuer outage does not become a two-hour application outage. Keep IAM changes such as role creation out of your failover path. In Entra-based estates, make sure break-glass accounts don't share the conditional-access path everyone else uses.

⑥ The management plane in your recovery path

What failed. This one is yours, not the provider's. In December 2021, AWS's own Service Health Dashboard could not fail over to its standby region, because the congestion it was reporting on blocked the failover. Route 53 APIs were impaired for seven hours; existing DNS answers were fine, but nobody could change them. Contrast this with Microsoft's handling of the Front Door outage: the Azure Portal "leveraged its standard recovery process to successfully transition away from AFD" at 17:26 UTC. Some Portal extensions, such as Marketplace, "did not have an established fallback strategy" and stayed impaired. Those two results came from the same organisation on the same evening, and the difference was a rehearsed path.

The lesson. A runbook that starts with "log in to the console", "run the Terraform apply" or "update the DNS record" needs the management plane at the moment it is least likely to be there.

Static-stability choice (F). Fail over by flipping switches that live in the data plane and were built in advance. AWS's whitepaper recommends removing "global service control plane dependencies in your recovery path". Concretely: on AWS, use Application Recovery Controller routing controls or pre-created DNS failover records driven by health checks, rather than editing records during the incident. Since December 2025, Route 53 also offers accelerated recovery for public hosted zones. With it, Route 53 can fail its control plane over to us-west-2 within about 60 minutes if us-east-1 is impaired, but only if you enabled it beforehand. On Azure, remember that all control-plane requests go through Azure Resource Manager (management.azure.com), while Microsoft's documentation confirms that "you can still access the data plane of your Azure resources" when it is unavailable. Pre-deploy the secondary and pre-configure routing priorities, so failover is a routing decision, not a deployment.

Loading diagram...

What the numbers say

Impact duration per official post-incident report, 2021–2026
Impact duration per official post-incident report, 2021–2026

We took the start and end times each provider states in its own report and subtracted. "Full recovery" means different things in different reports, so the CSV records which end time we used. Two readings stand out. First, apart from AWS's 14.5-hour October 2025 event, the big outages lasted between about 5.8 and 8.4 hours, long enough to break any RTO written as "under four hours" that assumes the provider fixes things for you. Second, the CrowdStrike bar is short on purpose: the faulty file was live for 78 minutes, but cleaning up the affected machines took far longer. The length of a bad push tells you very little about the length of the outage.

AWS us-east-1, 19–20 October 2025, impact window per service
AWS us-east-1, 19–20 October 2025, impact window per service

Take this one to your architecture board. The DynamoDB DNS fault, in blue, was repaired in 2.9 hours. Everything that needed a control plane to rebuild state ran three to five times longer: EC2 launches 14.0 hours, Lambda 14.4, the container services 14.6. Recovery is a thundering herd, with every customer and every internal service asking the same control planes for the same things at once. "We'll redeploy in the healthy zone" puts you in that queue.

Dependency fan-in on single-region control planes at AWS
Dependency fan-in on single-region control planes at AWS

The third chart measures fan-in: how many things lean on one place. We counted the rows in AWS's own Fault Isolation Boundaries whitepaper, in the version current in November 2026. Nine global-service control planes are hosted in us-east-1, including IAM, Route 53, CloudFront and Organizations. Nineteen named services need the Route 53 control plane to create resources, and AWS says that list is not exhaustive. Twenty-six S3 bucket-level control operations depend on us-east-1, whichever region the bucket lives in. To be fair, AWS publishes this and is adding escape hatches: 12 of those 19 services now use a recovery endpoint in us-west-2. Every provider has global control planes; AWS simply documents its own better than most. Keep your recovery path away from all of them.

In a recent engagement…

We recently reviewed the resilience of a European financial-services firm's dual-region Azure estate, built to satisfy DORA. On paper it was excellent: geo-replicated databases, infrastructure-as-code for everything, and a quarterly failover test that always passed.

We walked the six failure points with the platform team, and the picture changed. The failover test passed because it ran on a calm Tuesday with every management plane healthy. Step one of the runbook was a pipeline that deployed the application tier into the secondary region. That is a control-plane action, competing for capacity with everyone else in the same incident. Step four updated a DNS record. The front-door configuration for the secondary had never been deployed, because "the pipeline creates it". And the operators' privileged access came through a just-in-time elevation workflow that itself depended on sign-in services in the affected path.

None of the fixes was exotic. The secondary application tier now runs permanently at reduced scale, sized to carry full traffic for a few hours. Front-door routing for both regions is pre-configured, so failover is a priority change, not a deployment. Break-glass accounts got their own tested path. And the quarterly test now runs with the pipeline and portal deliberately unavailable. The first test under those rules failed, which was the most useful result of the engagement. The cost conversation was short once the board saw the alternative in writing: "our recovery depends on the control plane that failed".

The control-plane dependency checklist

This is the screenshot-able part. Run it against every critical workload. A "yes" in the third column is a finding.

#Failure pointQuestion to askStatic-stable answerEvidence to keep
1Global config pushCan one configuration change reach every region or cell within minutes?Staged rollout with bake time, synchronous validation, protected last-known-good, fail staticRollout policy, last rollback drill
2DNS and discoveryDoes serving depend on records being rewritten during an incident?Regional endpoints, deliberate TTLs, automation that refuses empty record setsDNS architecture, TTL register
3Capacity and launchDoes recovery require launching instances or scaling out?N/(N−1) capacity per zone; warm secondary region sized for the first hoursCapacity model, load-test results
4Health checks and failoverCan automation remove most of your capacity at once?Health checks test serving, removal velocity cap, fail-open thresholdHealth-check config, chaos test
5Identity issuanceDoes a running workload need fresh tokens from a global endpoint?Regional STS/token endpoints, early refresh, no IAM changes in failoverToken lifetime map, break-glass test
6Management planeDoes your runbook need the console, a management API or a pipeline?Data-plane switches (routing controls, pre-built DNS failover, front-door priorities)Runbook, test run with management plane "unavailable"
7Provider statusDoes your detection rely on the provider's status page?Your own synthetic probes from outside the providerProbe coverage, alert history
8Third-party SaaSDo critical SaaS tools sit on the same edge or region as you?Know their dependency; have a manual fallback for the critical onesSupplier dependency register

One warning: static stability costs money, because pre-provisioned capacity is paid for when nothing is wrong. Price it per workload tier. An internal reporting tool can wait for the provider; your payment flow cannot.

Governance for European enterprises

DORA. For banks, insurers, investment firms and other financial entities, Regulation (EU) 2022/2554 has applied since 17 January 2025. Article 11 requires an ICT business continuity policy and response and recovery plans for critical or important functions. Article 28(8) is more pointed: for ICT services supporting those functions, "financial entities shall put in place exit strategies", and exit plans "shall be sufficiently tested and reviewed periodically". Designating AWS, Google Cloud and Microsoft as critical ICT third-party providers puts them under direct oversight, but your obligations stay with you: a recovery step that depends on the provider's control plane is a dependency to name, test and document. We cover the exit side in our cloud exit runbook for DORA and the implementation detail in our DORA on Azure guide.

NIS2. For essential and important entities outside finance, Article 21(2)(c) of Directive (EU) 2022/2555 lists "business continuity, such as backup management and disaster recovery, and crisis management" among the minimum risk-management measures. Germany's NIS2 implementation law has applied since 6 December 2025 without a transition period. The control-plane checklist above is a reasonable, proportionate way to show that continuity planning has considered how cloud providers actually fail.

Sovereignty is a separate question. Sovereign-cloud offerings change who operates a control plane and under which law, not how often it fails. Ask the same six questions of any provider.

A pragmatic adoption path

  1. Weeks 1–2: map the dependencies. For each critical workload, walk the six failure points and list every control-plane call in serving and in recovery. Include identity, DNS and the CI/CD pipeline.
  2. Weeks 3–4: remove the management plane from the runbook. Pre-deploy the secondary. Pre-configure routing so that failover is a switch: Front Door or Traffic Manager priorities on Azure, Application Recovery Controller routing controls or pre-built health-checked DNS failover on AWS. Turn on Route 53 accelerated recovery for public zones that matter.
  3. Month 2: fix capacity and identity. Size zonal capacity with N/(N−1). Decide how warm the secondary region needs to be for the first hours. Move to regional token endpoints and review token lifetimes and break-glass paths.
  4. Month 3: harden your own control planes. Apply the provider lessons to your own configuration pushes, feature flags and DNS automation: staged rollout, synchronous validation, protected last-known-good and fail-static behaviour. Our cell-based architecture guide shows how to give configuration a blast radius smaller than "everything".
  5. Every quarter: test under realistic conditions. Run the failover drill with the console, pipeline and interactive sign-in deliberately unavailable, and add external synthetic probes. Our multi-region Azure disaster recovery guide covers the data-replication side.

The verdict

The hyperscalers' data planes are very good. The 2025 and 2026 outages happened because the systems that change servers, configuration and DNS failed, and recovery queued behind them. Providers are fixing their side in public: staged configuration, velocity limits, standby control planes, cell isolation. None of that helps a workload whose recovery plan is "redeploy when things go wrong".

Our view is simple. Design every critical workload so that surviving a provider incident requires no new action from a provider control plane. Pre-provision rather than scale. Cache rather than fetch. Flip a pre-built switch rather than run a pipeline. Then test it with the management plane switched off: the only test that resembles the real thing. For more patterns, see our resilience patterns cheat sheet.

If you want a second pair of eyes on your continuity design, our cloud architecture team runs this six-point control-plane review as a short, fixed-scope engagement. You leave with the checklist filled in for your own critical workloads.

FAQ

What is the difference between a control plane and a data plane in the cloud?

The control plane is the machinery that changes things: creating, configuring, scaling and deleting resources, issuing credentials, publishing DNS records and pushing configuration. The data plane does the daily work: serving requests, answering DNS queries, moving packets and reading and writing data. Azure's documentation puts it simply: you create a storage account through the control plane and read and write data through the data plane. Control planes are more complex and change more often, so a resilient design keeps serving through the data plane while a control plane is down.

What is static stability?

Static stability is a design property AWS describes in its Builders' Library: in a statically stable design, the overall system keeps working even when a dependency becomes impaired. In practice, it means that recovering from a failure needs no new action from a control plane. Capacity is already provisioned in every zone, credentials are already issued, DNS answers are already cached, and traffic can be shifted with switches that live in the data plane.

What caused the AWS us-east-1 outage of 19–20 October 2025?

AWS's post-incident summary describes a race condition in DynamoDB's automated DNS management. A delayed DNS Enactor applied an older plan over a newer one, and a clean-up process then deleted that plan. That left the regional DynamoDB endpoint with an empty DNS record from 11:48 PM PDT on 19 October. DNS was restored by 2:25 AM, but EC2's instance-management system, Network Load Balancer health checks and services such as Lambda and STS took much longer to recover. AWS puts the end of the event at 2:20 PM PDT on 20 October, about 14.5 hours after it started.

What caused the Azure Front Door outage of 29 October 2025?

According to Microsoft's post-incident review (tracking ID YKYN-BWZ), valid customer configuration changes made across two different control-plane build versions produced incompatible configuration metadata. That metadata triggered a crash in the data plane's asynchronous processing. Because the crash was asynchronous, health checks passed during rollout and the bad configuration spread globally. Customer impact ran from 15:41 UTC on 29 October to 00:05 UTC on 30 October 2025.

Is multi-cloud the answer to hyperscaler outages?

Usually not for a whole workload. Running one application actively in two clouds doubles the operational surface and often adds a new single point of failure in the layer that picks the cloud. For most enterprises, the better return comes from making each workload statically stable inside one provider: pre-provisioned capacity across zones, a warm second region, regional endpoints and data-plane failover switches. Multiple providers are worth it at specific choke points, such as DNS, the CDN or edge, and identity break-glass access.

Which EU regulations require us to plan for cloud provider outages?

For financial entities, DORA has applied since 17 January 2025. It requires an ICT business continuity policy (Article 11) and documented, tested exit strategies for ICT services that support critical or important functions (Article 28(8)). In November 2025 the European Supervisory Authorities published a list of 19 designated critical ICT third-party providers that includes the EU entities of AWS, Google Cloud and Microsoft. For essential and important entities, NIS2 Article 21(2)(c) requires business continuity, backup management, disaster recovery and crisis management. Germany's NIS2 implementation law has applied since 6 December 2025.

Sources

All sources checked on 15 November 2026. Durations in the charts are computed from the start and end times stated in each provider's own report; the CSV records which end time we used.

Tier A: primary post-incident reports, provider documentation and regulation

Tier B: engineering analysis

Tier C: vendor statements, directional

Topics

cloud outage resiliencecontrol plane vs data planestatic stabilityAWS us-east-1 outage October 2025Azure Front Door outagehyperscaler post-incident reviewmulti-region disaster recoveryDORA exit strategy

Frequently Asked Questions

The control plane is the machinery that changes things: creating, configuring, scaling and deleting resources, issuing credentials, publishing DNS records and pushing configuration. The data plane is the part that does the daily work: serving requests, answering DNS queries, moving packets and reading and writing data. Azure's documentation puts it simply: you create a storage account through the control plane and read and write data through the data plane. Control planes are more complex and change more often, so a resilient design keeps serving through the data plane while a control plane is down.

Expert engagement

Need expert guidance?

Our team specializes in cloud architecture, security, AI platforms, and DevSecOps. Let's discuss how we can help your organization.

Get in touchNo commitment · No sales pressure

Related articles

All posts