The night the cache fell over

Northwind · Incident review

A postmortem of the 2:14am Redis outage

Timeline

All times local, night of the incident

02:14 — Redis primary OOM-killed; p99 alert fires
02:16 — failover to a cold replica; every read misses
02:31 — Postgres connection pool exhausted; errors climb
02:47 — miss traffic shed, cache warms back up
03:20 — p99 back under 100 ms; incident closed

Here's how the night unfolded.

[click]

[exhales] At two fourteen, the Redis primary ran out of memory and the kernel killed it. [pause] Our p99 latency alert fired a few seconds later.

[click]

Two minutes after that, failover promoted a replica. But it came up cold, so every single read was a miss.

[click]

All of those misses went straight to Postgres, and by two thirty-one its connection pool was exhausted. That's when errors, not just slowness, started reaching users.

[click]

At two forty-seven we started shedding miss traffic, which finally let the cache warm back up.

[click]

And by three twenty, p99 was back under a hundred milliseconds and we closed the incident. About an hour, end to end. [sighs]

The request path

Where a read goes — and where it went that night

p99 latency during the outage

API read endpoints, milliseconds

The fix

One thing that night, three things since

That night

Shed cache-miss reads at the API with a feature flag
Let the replica warm, then ramp traffic back

Since then

Single-flight fills: one DB query per missing key
Circuit breaker in front of Postgres
Memory headroom alert at 75% of maxmemory

What 2:14am taught us

A cache is a dependency. Size what's behind it for the day it's empty.
Failover isn't recovery. A cold replica is up, not ready: ours took 31 minutes to warm.
Alert on headroom, not on death. Memory told us hours before it killed us.