Why Cloudflare's consistent hash rings ate so much RAM

// the short version

Cloudflare's Pingora Backend Router accumulated hash points across weighted servers and feature-specific rings. A simple routing algorithm had become a memory problem.

// what to take away

• Weighted virtual nodes and feature-specific rings multiplied the hash points held in memory.
• Rust alignment kept a smaller index in an eight-byte record; a raw six-byte representation removed the padding.
• Fewer hash points reduced memory and collisions, while a staged cache migration controlled origin traffic.

// transcript

Cloudflare freed a hundred terabytes of RAM by shrinking hash rings. They're how its cache router decides which server should hold a file. A hash function turns a file's cache key into a repeatable thirty-two-bit number. Server identifiers become numbers in that same space. Arrange them on a number line that wraps around, and you've got a ring. In the article's diagram, a request belongs to the first server to its left. Repeated keys find the same server; adding a server only reassigns a slice. But randomly spaced servers get uneven slices. So hashing variations of each server's identifier gives it many positions, called virtual nodes, whose slices add together. More positions smooth out the imbalance, and bigger disks get proportionally more. Different caching features and compliance requirements need separate rings of eligible servers. That multiplied into dozens of rings, sometimes consuming six gigabytes. Every position was a Rust struct with two unsigned thirty-two-bit fields: the hash and an index into a server array. That's four bytes each, eight total. The index only needed sixteen bits, enough for over sixty-five thousand servers. But changing that field left the struct at eight bytes. The hash still required four-byte alignment, so Rust inserted two padding bytes. An array of these structs needs each next hash aligned too. They instead stored six raw bytes and decoded the fields through getters. No padding, six bytes per entry, a quarter less storage. Then they questioned how many entries they needed. Halving the predicted imbalance takes roughly four times as many positions. And thirty-two-bit hashes can collide: two positions land on the same number, so some contributions disappear. At high counts, those collisions actually worsened the balance in their larger simulation. They could cut the hash count by ninety percent without appreciably worsening distribution. But a new ring routes requests away from existing cached files. Switch everything at once, and cache misses could overwhelm the original websites. They kept both rings, gradually moving traffic within selected data centers, with rollback available. Retiring the old rings delivered the hundred-terabyte saving. The savings came from measuring what each byte, and each extra hash, was actually buying.