Every service retried, and that was the outage. Uber's fix for retry storms: error ownership

What a retry storm is, and why sensible retries cause one

A retry is the most ordinary reliability tool there is: a call failed, the failure might be a blip, so send it again. Each service that does this is behaving correctly in isolation.

The trouble starts when services call services. A request to a large backend passes through a chain of them, and if every one retries a failed call, the retries multiply. The service at the bottom of the chain, the one that is actually failing, receives the product of everyone's retries at the moment it has the least capacity to spare. That extra load makes it fail more, which causes more retries. Uber calls the result a retry storm, and says a local outage can grow into what it calls a stack-wide incident this way.

Uber's diagnosis is that retries are not context-aware. A service can control how many times it retries, but it cannot tell an error its callee generated from one the callee merely passed along, so it treats them the same.

The arithmetic: why one retry per hop means eight times the traffic

Uber's worked example is a chain of seven services, A to G, where each calls the next exactly once. In the steady state every service handles the same number of requests. Then D, the fourth, starts failing, and every service is configured to retry once.

C sends each request to D twice. B sees C fail and sends each request to C twice, and C doubles each of those. A does the same to B. The load relative to normal:

service   A    B    C    D    E    F    G
load      1x   2x   4x   8x   8x   8x   8x

The general form is R^d times normal traffic, where R is attempts per hop and d is how deep the service sits. E, F and G also see 8x in Uber's model, because D keeps calling downstream on every attempt it receives. The video draws only the traffic into D.

This is a simple model: a straight chain, a fan-out of one, the same retry count everywhere. Real call graphs branch, and Uber notes that fan-out makes the amplification harder to see, not smaller.

What a retry budget does, and where it stops helping

The standard defence is a retry budget: a client may retry only a fraction of its normal request volume. With a budget B at every hop the formula becomes (1 + B)^d. At 10%, the same chain looks like this:

service   A    B      C       D       E       F       G
load      1x   1.1x   1.21x   1.33x   1.33x   1.33x   1.33x

That is a large improvement on 8x, and it is the figure the video shows. Uber makes two further points about it.

First, a budget is still applied at every hop. Each level retries a little, because none of them knows where the error started.

Second, a budget caps how much a retry can help. Uber tabulates what a caller sees after retries, with a 10% budget, as the callee gets worse:

• callee succeeds 99.9% of the time: caller sees 99.9999%
• 99%: 99.99%
• 95%: 99.75%
• 90%: 99%
• 80%: 88%
• 70%: 77%

Up to a 10% error rate, every failed request can be retried and the caller barely notices. Past that, the budget runs out and most failures are never retried. Uber also points out that this table assumes failures are independent, so that a retry has a fresh chance. In an overload, a bad database host or a sharding problem, they are not: the retry fails for the same reason the first attempt did. That is the intuition behind the fix. When a service is failing heavily, more retries from further away do not raise availability, they only add load.

Error ownership: symptom versus cause

Uber's scheme is called error ownership, and it rests on one distinction.

If a service returns an error because one of its own outbound calls failed, the error it returns is a symptom. The cause is further down. If a service returns an error while none of its outbound calls failed, the service is the cause, and Uber says it owns the error.

Only the owner's direct caller can usefully retry. In the worked chain, if retries are allowed on the C to D edge alone, with the same 10% budget, the load becomes:

service   A    B    C    D      E      F      G
load      1x   1x   1x   1.1x   1.1x   1.1x   1.1x

D gets one bounded round of retries from the service next to it, and nothing above C adds to it.

How a service decides whether to claim an error

Ownership travels in a response header, x-uber-error-claim, set to true (claimed) or false (unclaimed). The rules for the service returning the error, from Uber's flowchart:

• it returns an error and none of its outbound calls failed: true. The error originated here.
• an outbound call failed, and that dependency is fail-close: false. The downstream failure explains the error.
• outbound calls failed, but only fail-open ones: true.

Fail-close and fail-open describe how a service treats a dependency. A fail-close dependency is one the request cannot succeed without. A fail-open dependency is one the service can carry on without, so its failure should not be what made the request fail, and it does not get the blame.

The video compresses this to "a service that fails because something it called failed sets it to false". The fail-open case is the detail that sentence leaves out.

To make these calls a service has to match an inbound request that failed with the outbound calls made on its behalf. Uber does that with an internal system it calls the Service Dependency Analysis Solution, which sits in the request path beside the retry middleware. Because both are shared infrastructure, Uber says services get the behaviour without writing error-handling logic of their own.

How a caller decides whether to retry

The caller's rule is shorter:

• header is true: retry is allowed
• header is false: do not retry
• header is missing: retry is allowed

Uber's three-service example shows the two rules working together. C fails by itself and returns a claimed error. B sees true and retries. If the retry also fails, B returns an error to A, and because B's error was caused by C, B must mark it false. A sees false and does not retry.

The missing-header case is deliberate. A downstream service might not run the dependency analysis, or might have lost the request context needed for it. The first service to see an error with no header retries it, then unclaims the error on the way up, so the retries stay local to that edge. Uber's phrase for the result is that it is a retry disturbance and no longer a storm.

What the video left out

Why not just translate error codes. A team could have each service map its dependencies' error codes into "retryable" and "not retryable" by hand. Uber says that does not scale for it, citing large fan-in and fan-out, call flows that keep changing, and the need for frequent adjustment. That is why the mechanism lives in shared infrastructure.

Guaranteeing at least one retry. Many services have no retries configured for their fail-close dependencies. If their callers stop retrying because the error came back unclaimed, nothing retries at all and availability drops. Uber's fix is a signal from the retry middleware to the dependency analysis saying whether the retry criteria were satisfied for a downstream error, which the service then uses when deciding whether to own the error it returns. Uber states that this adds no new retries to the chain, and that it only works if at least one service along the chain has retries configured.

Lost request context. If a service loses the link between an inbound request and its outbound calls, it cannot tell that its error came from below, so it claims it. In Uber's four-service example, D fails, C wrongly claims the error, and B therefore retries to C as well. But B's own error is correctly unclaimed, so A does not retry. Uber's summary is that the retryable error shifts one hop to the left: with one retry per hop, the total is half of what it would have been, and with five more services above A the unprotected worst case would have been 32 times the requests.

When it gets ownership wrong

The analysis attributes an error to downstream failures first and to the service itself last. That creates a blind spot: if a service fails for its own reasons at the same moment one of its dependencies fails, it blames the dependency, marks the error false, and its caller skips a retry that might have worked.

Uber's illustration: a service has a 10% error rate of its own, from an overloaded cache or database the analysis does not track, and two dependencies that each fail 10% of the time for unrelated reasons. Of every 10 of the service's own failures, 1.9 would coincide with a dependency failure and be wrongly unclaimed.

Then the measurement. Across six months of data, Uber says coincidental failures are rare: in what it calls the worst case, for 80% of edges with more than 100 callee failures in a minute, the caller also failed about 2% of the time. Put through the same example, that is 0.396 of every 10 failures wrongly unclaimed. The video's "about two percent of the time at worst" is this figure, simplified.

Uber adds that the dependency analysis keeps a memory of failure patterns, and can use it to unclaim only when inbound failures actually correlate with failures of fail-close dependencies, which it says removes the risk.

The 18 November 2025 incident, and what the numbers count

Uber describes one production incident. On 18 November 2025 a service it calls a Core Entity service, more than five levels deep in the call chain, began returning a very high error rate because of an infrastructure problem underneath it. Error ownership was already enabled.

• 46% to 135%. Uber's estimate of the extra traffic the degraded service would have received with retry budgets alone. It is a calculation of what did not happen, not a measurement.
• 200,000. The most additional requests any single immediate caller was stopped from sending.
• 9.5 million. Requests Uber calls spurious that were never sent, counted by adding the retries stopped at the immediate callers to those stopped at their ancestors, aggregated at the root.
• 25 down to 3. The "max retry storm radius": the greatest depth of call path over which a retry storm could happen, computed for the call graph of every root across Uber's user-facing APIs. The average went from 20 to 2.

All of these are Uber's own figures about its own systems. The post does not say how long the incident lasted or how its duration compared with earlier ones.

Terms

Retry storm

A failure in which retries from many callers multiply the load on a service that is already failing, which keeps it from recovering.

Retry budget

A cap on retries as a fraction of a client's normal request volume, for example 10%. It bounds the extra load a client can add however many of its calls fail.

Error ownership

Uber's name for marking each error with whether the service returning it caused it, and retrying only errors their sender owns.

x-uber-error-claim

The response header that carries ownership. true means the sender claims the error as its own; false means it came from one of the sender's dependencies.

Fail-close dependency

A dependency whose failure makes the calling service's request fail.

Fail-open dependency

A dependency the calling service can do without, so the request can still succeed when it fails.

Fan-out

The number of downstream calls a service makes per request. Uber's worked example uses a fan-out of one.

Retry storm radius

Uber's measure of how many hops up a call path a retry storm can spread.

Coincidental error

A service's own failure that happens at the same moment as a dependency's failure, and so gets attributed to the dependency.

Questions

// further reading

• How Uber Protects Against Retry Storms ↗: Uber's post, with the decision flowcharts and the end-to-end sequence diagrams this page describes in words
• Handling Overload, in Google's Site Reliability Engineering book ↗: where per-client retry budgets and the problem of retries compounding across layers are laid out
• Timeouts, retries, and backoff with jitter ↗: Amazon's account of the same amplification problem and the client-side defences that come before any of this