The Retry Storm You Built On Purpose

1. The claim

A retry is a load-generating decision made by a component that has just been told the system is under stress. That is the whole problem in one sentence. Every other property of retry behaviour follows from it.

Exponential backoff is the standard mitigation and it is a good one. It is also incomplete in a specific, mechanical way. Backoff controls when one client retries. It says nothing about whether a thousand clients retry at the same moment. If the delay schedule is deterministic, and the fault that triggered it was shared, then the schedule does not spread the load. It preserves the alignment the fault created and carries it forward, round after round, for as long as the outage lasts.

2. The mechanism

A dependency returns errors starting at t=0. Every in-flight caller observes the failure within one round-trip time of each other, which on a healthy internal network is a window of a few milliseconds. That is the synchronising event. From then on, a deterministic schedule of 1s, 2s, 4s, 8s keeps the whole population inside that same few milliseconds at every retry boundary.

            0s      1s      2s      4s      8s
             |       |       |       |       |
no jitter    |#######|#######|#######|#######|
           fault    R1      R2      R3      R4
                     ^       ^       ^       ^
             every caller arrives in the same window
full jitter  |#.#..#.|..#.#..|#..#..#|.#..#.#|
arrivals spread across the whole gap

The dependency therefore recovers into a square wave rather than into a ramp. Its first moment of health is also the moment of peak concurrent arrival, so it fails again, which re-synchronises the population, which produces the next spike. The system has found a stable oscillation and will stay in it until something outside the loop intervenes.

Two amplifiers make this worse than the single-layer picture suggests.

Amplification through layers. Retries compose multiplicatively. Three attempts in the SDK, inside three attempts at the gateway, inside three attempts in the calling service, is not nine requests and it is not three. It is twenty-seven, and no single layer’s configuration looks unreasonable on its own.

Figure 2. Worst-case request amplification for three attempts per layer. The multiplier is the product of the layers, not the sum.

The load grows while the outage lasts. Callers that would have arrived during the outage do not disappear. They queue in the client, in the connection pool, in the upstream’s own inbound buffer. The longer the dependency is down, the larger the population that arrives in the first post-recovery window.

3. What it looks like in production

The signature is specific enough to be diagnosed from a graph without reading any code.

The third row is why this is rarely caught in testing. A load test drives a synthetic client population with independently random start times, which is jitter arriving by accident. The synchronising event is absent from the test because the test never has a single shared fault.

4. The remedy, in the order it matters

The order is deliberate. The first item is the cheapest and the third is the one that actually holds.

4.1. Jitter, and specifically full jitter. Sleep a uniform random value in [0, base * 2^n] rather than the value itself. Equal jitter and decorrelated jitter are both defensible; the important property is that the delay is drawn from a distribution rather than computed, so that no two callers share a wake time except by coincidence.

4.2. A retry budget at the caller, expressed as a fraction. Not “three attempts per call”. A token bucket refilled from forward traffic, permitting retries only while retries stay under roughly ten percent of successful requests over a sliding window. A count-based limit rises with load exactly when it should fall. A fraction-based budget falls with success exactly when it should.

4.3. Exactly one retrying layer, named in the design document. This is the rule that survives reorganisation, because it is a written decision rather than a configuration value. Every other layer converts failures into errors and returns them. If nobody can say which layer owns retries, the answer is all of them.

  caller  ->  gateway  ->  service  ->  store
    [R]         [ ]          [ ]        [ ]
     |
     +-- retries here, with a budget and jitter
         every other hop fails fast and reports

4.4. Emit the attempt-to-call ratio. If the number is not on a dashboard, the storm is invisible until it is an incident. See 5.13.

5. The strongest objection

6. What this paper does not claim

Retries are not the problem. A system without them converts every transient fault into a user-visible error, which is worse than a storm you can bound. Jitter alone is not the remedy either, and Section 4.2 exists because of that. The ten percent figure is a starting point that has held for me across several systems rather than a derived constant, and I would expect it to be wrong for anything with a very different ratio of read to write traffic.

Nor is any of this novel. The mechanism is well described in the literature. What the paper claims is a gap between that description and what is actually configured in production, and erratum 7.3 records me falling into that gap myself, in code I reviewed and approved.