What is Retry Storm?

19/08/2026

Advertisement

Continue Reading Below

A retry storm is a failure pattern in software systems where many clients or services simultaneously retry failed requests, creating a sudden surge of traffic that overwhelms a struggling service and prevents it from recovering. Instead of helping the system bounce back, the retries make the problem worse, turning a small, temporary glitch into a full-blown outage.

Think of it like this: a service briefly slows down, so everyone tries again at the same time. That wave of retries hits the service all at once, making it even slower, which causes even more retries. The recovery mechanism itself becomes the cause of the outage.

Examples of Retry Storms

Retry storms often show up in systems where many programs or users depend on the same service. Here are some real-world-style examples:

Microservices in the cloud: Imagine an online store where the “checkout” service depends on a “payment” service. If the payment service slows down for a few seconds, hundreds of checkout services might all retry their failed payments at the same time, flooding the payment service and keeping it down much longer.
Mobile apps reconnecting after a network blip: When a mobile network briefly drops, thousands of phones might all try to reconnect to the same backend server at once. If they all retry immediately, the server can get swamped even though the network issue is already fixed.
Internal company tools: In a company’s internal systems, many small services might call a shared “user profile” service. If that service hiccups, all the other services retry together, creating a storm that can take the whole system down for minutes or even hours.

In each case, the original problem might have lasted only a few seconds, but the storm of retries keeps the service overloaded long after the root cause is gone.

Causes of Retry Storms

Retry storms usually happen because of how retry behavior is designed (or not designed). Common causes include:

Synchronized retries: Many clients retry at the same fixed interval (for example, exactly 1 second after a failure). This creates waves of traffic that hit the service all at once instead of being spread out.jeffbailey+2
No waiting (no backoff): If clients retry immediately with no delay, they pile on more requests while the service is already struggling, making it even slower.
Too many or unlimited retries: When there’s no limit on how many times a client will retry, the number of requests can grow quickly, especially if many clients are doing this at once..
Nested dependencies: In systems where Service A calls Service B, which calls Service C, retries at each layer can multiply. A small issue at Service C can cause a huge storm by the time retries from all layers stack up.
Lack of coordination: Without mechanisms like circuit breakers or rate limits, every client acts on its own, and their combined retries create a self-reinforcing loop of failures.

Real-World Examples of Retry Storms

GitHub: August 2026 Outage (7+ Hours)

What happened:
On August 17, 2026, GitHub experienced a major outage lasting 7 hours and 47 minutes. Issues, Pull Requests, APIs, Actions, and Copilot were all affected. At peak, web and API error rates hit around 20%, and some downloads saw 50% errors.

How a retry storm made it worse:
The initial trigger was network saturation on load balancers in GitHub’s Central US datacenter. But the outage was prolonged by a “latent retry bug” in VS Code (used by millions of developers). When a single internal endpoint started replying slowly, VS Code clients began aggressively retrying, amplifying traffic to the Copilot Token Service by roughly 10×—from about 7,000–9,000 requests per second (RPS) to 70,000–100,000 RPS.

This flood of retries kept the service overloaded long after the original capacity issue was fixed. Engineers had to temporarily block some retry traffic at the load balancer and reduce gateway retry logic to finally let the system recover.

Key takeaway:
A small, regional problem turned into a long, global-ish outage because retry logic in a popular client (VS Code) multiplied the load instead of backing off.

Google Cloud: June 2025 Global Outage

What happened:
On June 12, 2025, Google Cloud had a global outage lasting about 3 hours. Hundreds of services—including Compute Engine, Cloud Storage, BigQuery, and Gmail—saw elevated errors and degraded access.

How a retry storm made it worse:
The root cause was a faulty policy change in Google’s quota enforcement system (Service Control), which caused crash loops in components that validate API traffic. As services failed, clients retried their requests. In some regions, especially us-central1, recovery was delayed because of “infrastructure strain from retry traffic.”

In other words, the retry storm didn’t cause the initial failure, but it significantly slowed down recovery by keeping the already-struggling infrastructure overloaded.

Key takeaway:
Even when the root cause is a configuration bug, retry storms can turn a 1–2 hour incident into a 3+ hour one by preventing services from stabilizing.

Twitter: 2013 Mobile Outage

What happened:
In 2013, Twitter suffered a site-wide outage that lasted about 2 hours. The original issue was a 5-minute database hiccup.

How a retry storm made it worse:
Mobile clients were configured to retry aggressively on almost every error, including errors that would never succeed (like 4xx client errors). This created a 10× spike in load on the Mobile API Gateway during the database issue, extending a short glitch into a long outage.

Twitter’s fix included:

Exponential backoff with jitter (waiting longer between retries, with some randomness)
Circuit breakers that stopped retries once error rates passed a threshold
Retry budgets that limited retries to 15% of total traffic

Key takeaway:
Retrying on the wrong kinds of errors and doing it all at the same time can multiply a small problem into a major one.

Stripe: Retry Budgets to Prevent Storms

Stripe doesn’t have a famous public outage tied to retry storms, but they explicitly design against them. Each API key gets a “retry budget” (like 100 retry tokens refilling at 10 tokens/second). When a client uses up its budget, further retries are rejected with a 429 (Too Many Requests) response.

This prevents any single customer’s retry logic from creating a storm that affects everyone else.

Smaller-Scale Example: Single User Crashing 30 ECS Tasks

In a 2025 incident, a single user’s request triggered a latent bug in a service running on AWS ECS. The bug crashed the entire task, and the infrastructure’s automatic retry logic restarted the task over and over. This “infrastructure-level retry amplification” ended up crashing 30 ECS tasks and caused a production outage.

Key takeaway:
Retry storms aren’t only about thousands of clients; even a single user plus aggressive auto-retry at the infrastructure level can create a storm.

Consequences of Retry Storms

The effects of a retry storm can be severe and long-lasting:

Extended outages: A brief 5–10 second issue can turn into a 30-minute (or longer) outage because the service never gets a chance to recover under the constant pressure of retries.
Cascading failures: As one service struggles, the retries can overload other services that depend on it, causing failures to spread across the system.
High latency and poor user experience: Users see slow responses, timeouts, and errors, even if the original problem was minor.
Blocked recovery: Even after engineers fix the root cause, the flood of retries can keep the service down, forcing them to temporarily block traffic just to let the system “breathe” and restart.
Wasted resources: The system spends time and computing power processing retries that are likely to fail, instead of serving real user requests.

How to Prevent Retry Storms (Simple Tips)

While this is a technical topic, the main ideas to prevent retry storms are straightforward: