My first holiday season at Shutterfly, I watched a server that was technically healthy refuse to serve a single customer. The load balancer had it marked up. It answered its health check in under a millisecond. Every real request against it timed out. Thousands of people who happened to be pinned to that node saw a spinner, refreshed, and made it worse. Adding more servers did nothing, because the traffic could not move.
That failure was not a bug anywhere in particular. It was the architecture working exactly as designed. This is the story of the year we spent making peak survivable without rewriting the thing that was breaking, and why the stabilization work turned out to be the foundation for the decomposition that followed.
The failure mode, precisely
The platform was a large Java monolith, about fifteen years of accumulated commerce logic in one deployable, running on a servlet container behind an F5 load balancer. Session state lived in memory, in the servlet container's session map, on whichever node the customer first landed on. The F5 issued a persistence cookie and honored it for the life of the session. That is session affinity, and it is a perfectly reasonable design right up until it is not.
Affinity turns every app server into a stateful singleton for its share of users. Once you have that, four things compound.
The thread pool is the scarce resource, and it is shared. Requests were served by a fixed pool with blocking I/O. A thread picks up a request and holds it for the whole call, including the time spent waiting on a database that has gotten slow. Little's Law tells you exactly what happens next: the concurrency you need is arrival rate times service time. Hold service time constant and the pool is fine. Let one dependency get three times slower and you need three times the threads to serve the same traffic. When required concurrency passes the pool size, requests start queueing in the accept backlog, queueing adds latency, added latency raises service time further, and the loop closes on itself. That is congestion collapse, and it is not gradual. The system looks fine, then it is gone.
There were no bulkheads. One slow downstream call could consume every thread in the pool, which meant a degraded recommendation service could take down checkout. Nothing reserved capacity for the work that mattered.
The health check lied. The F5 monitor fetched a static page and looked for a 200. A saturated container can almost always spare one thread to return a static page, so the node kept passing its check while every business request failed. The monitor was answering "is the process alive," and we were reading it as "can this node do useful work." Those are different questions and at peak they give opposite answers.
Failure was correlated and lossy. When a node did die, its sessions died with it. Carts, half-built photo books, everything. Those customers got logged out mid-flow, which sent them straight back in to start over, which is new load arriving exactly when you have less capacity to serve it. Meanwhile the users pinned to a hung node kept hammering refresh, and every refresh consumed another thread. Retry amplification did the rest.
The consequence that hurt most was strategic rather than technical. Horizontal scaling did not work. We could stand up more app servers, and the new ones would sit nearly idle, because existing sessions were pinned to the nodes that were already struggling. Capacity arrived where the load was not. That is the property you have to fix first, because everything else is a workaround until it is true that any node can serve any request.
Why we did not just rewrite it
The correct fix was obvious to everyone in the room. Break the monolith into services along domain boundaries, give each one its own scaling profile and its own pool, move to the cloud, autoscale. We did exactly that, and I have written elsewhere about how we carved the first services out. But that work ran on a multi-year clock and peak was a few months away.
So we separated the two efforts deliberately. One track was stabilization: mechanisms that could be added around the monolith without touching much of its business logic, designed to make peak survivable this year. The other track was transformation. Conflating them is the standard way to get neither, because the stabilization work gets deferred as a stopgap unworthy of real engineering, and the transformation gets rushed into a deadline it cannot meet.
The framing I used with the executive team was that you cannot re-architect a system while it is on fire, and you will not be funded to re-architect it if you survive peak on luck. Buy the time, then spend it.
Three mechanisms bought the time: distributed session management, capacity-aware load shedding driven by real signals, and an admission control layer with a priority waiting room.
Externalizing the session: Redis as an L2 store
Everything started here, because affinity was the root constraint. If session state lives in one process, that process is load-bearing. Move it out and the persistence cookie becomes a performance optimization rather than a correctness requirement.
We put sessions in Redis and left a small node-local cache in front of it. The local map is L1, sized small and short-lived, so a customer who happens to come back to the same node on the next click does not pay a network round trip for data the node already has. Redis is L2 and is the authority. Every write goes through to Redis. Every L1 read validates a version number before it is trusted.
Shape the keys around the access pattern
The obvious implementation is to serialize the session object and store it as one blob under one key. We did not do that, for three reasons that all showed up in practice.
A blob makes the whole session the unit of contention. Two concurrent requests from the same customer, which modern pages generate constantly through background calls, each read the blob, each modify their own field, and the second write silently discards the first. A blob is also the unit of transfer, so a session carrying one large attribute makes every request on that session pay for it in both directions. And native Java serialization is fragile across a rolling deploy: change a class and a session written by an old node cannot be read by a new one, which turns every release into a forced logout.
So the session became a Hash, one field per attribute, encoded in a versioned language-neutral format rather than native serialization.
Session keys# the session itself: one hash, one field per attribute
HSET sess:{9f2c41ab} _ver 7 _seen 1701390421337 uid 44182 \
cart c91a7 locale en_US tier gold
EXPIRE sess:{9f2c41ab} 86400
# read only what the request actually needs, not the whole session
HMGET sess:{9f2c41ab} uid cart _ver
# counters stay server-side and stay correct under concurrency
HINCRBY sess:{9f2c41ab} cartItems 1
# every live session for a user, for sign out everywhere
SADD user:{44182}:sessions 9f2c41ab
# resumable sessions ordered by last activity, for the 24 hour return
ZADD resume:{44182} 1701390421337 9f2c41ab
Field-level access is the whole point. A request that needs the user id and the cart pointer fetches two fields, not a serialized object graph. A request that increments a counter does it server-side with HINCRBY and never races.
The braces in sess:{9f2c41ab} are not decoration. They are a Redis hash tag: only the text inside them is hashed to choose a slot. We started on a single shard with Sentinel handling failover, so it did not matter yet, but writing keys with tags from day one meant that when we later sharded, a session's related keys landed on the same node and multi-key operations kept working. Retrofitting a key naming scheme onto a live session store is not an afternoon of work.
Make the hot path one atomic round trip
Nearly every request touches the session: validate it, refresh its idle timeout, record last activity. Done naively that is three round trips and a read-modify-write race. We collapsed it into a single Lua script, which Redis runs atomically.
touch.lua, called with EVALSHA on every request-- KEYS[1] session hash ARGV[1] expected version, or * to skip the check
-- ARGV[2] now in millis ARGV[3] idle timeout in seconds
local ver = redis.call('HGET', KEYS[1], '_ver')
if not ver then return -1 end -- expired or never existed
if ARGV[1] ~= '*' and ver ~= ARGV[1] then
return -2 -- lost the race, re-read and merge
end
redis.call('HSET', KEYS[1], '_seen', ARGV[2])
redis.call('HINCRBY', KEYS[1], '_ver', 1)
redis.call('EXPIRE', KEYS[1], ARGV[3])
return redis.call('HGET', KEYS[1], '_ver')
The version field gives optimistic concurrency. A writer sends the version it read; if it no longer matches, the script refuses and the caller re-reads and merges rather than blindly overwriting. Compare and set, implemented in four lines, on a data store that was never going to give us transactions across a network.
The EXPIRE on every touch implements a sliding idle timeout. We kept that separate from an absolute lifetime, because they mean different things: idle timeout is a usability control and absolute lifetime is a security control, and collapsing them means a session that is touched forever lives forever.
Resuming a customer who comes back
The sorted set keyed by user is what makes the 24 hour resume work. Score is last-activity time, member is the session id. A returning customer's request arrives with no valid session, we look up the most recent entry inside the window, and if the underlying session hash is still there we adopt it. The half-built photo book is still on the page.
# most recent resumable session inside the window
ZREVRANGEBYSCORE resume:{44182} +inf 1701304021337 LIMIT 0 1
# janitor: drop anything past the window so the set stays bounded
ZREMRANGEBYSCORE resume:{44182} -inf 1701304021337
One caution worth stating, because it surprises people. Redis expires keys lazily and through a sampled background cycle, so keyspace notifications for expiry do not arrive at the exact moment the clock passes. If you need a session-destroyed listener to fire reliably, subscribe to the notification for the common case and run a periodic sweep for the rest. Treat the event as a hint, not a guarantee.
The parts that bite you in production
Externalizing sessions adds a network hop to every request, which means the session store becomes a dependency that can take down the site it was meant to protect. Three rules kept that from happening.
- Timeouts strictly below the request budget. A session store that hangs is worse than no session store, because it converts a fast failure into a held thread, which is precisely the problem we were solving. Connection and read timeouts were set in single-digit milliseconds.
- Pool sizing tied to the container. The client connection pool was sized against the servlet thread pool, not guessed. If more threads can request a connection than the pool can hand out, you have just moved the queue rather than removed it.
- Degrade rather than fail. A circuit breaker around the session store meant that if Redis became unavailable, nodes fell back to local sessions and the F5 fell back to affinity. Worse behavior, still a working site. The safety mechanism must have a smaller blast radius than the thing it protects.
We also put a ceiling on attribute size and moved anything large out of line, stored under its own key with a pointer in the session. A fat session is not a storage problem, it is a bandwidth problem multiplied by every request.
Making the health check tell the truth
With sessions portable, traffic could finally move. The next question was what should decide where it moves to, and the honest answer was that nothing in the system knew. A load balancer health check is binary and shallow. In the pool or out of the pool, based on whether a process answers. Both answers are wrong in the middle of a saturation event, where what you want is to send a node less work rather than none.
So we built a beacon. Every app server ran a lightweight agent that published a signal to a central control plane every few seconds. The signal carried what the node actually knew about itself:
- Thread pool occupancy and accept queue depth. Busy threads against maximum, plus how many connections are waiting to be accepted. This is the earliest and most direct leading indicator, because it is the resource that runs out first.
- Database connection pool pressure. Active against maximum, and more importantly the time a thread spends waiting to borrow a connection. Borrow wait rising is the database telling you it is in trouble several seconds before latency shows it.
- In-flight request count and locally computed latency percentiles. Computed on the node so no aggregation lag sits between the event and the decision.
- Garbage collection pressure. Not heap used, which is a sawtooth and tells you nothing, but time spent in collection and old generation occupancy after a full collection. Those two predict the pause spiral that precedes a hang.
- CPU relative to cores, plus steal time. Load average only means something normalized against the cores you actually have, and on shared virtual hardware steal time explains slowdowns nothing on the node caused.
- Circuit breaker states and downstream error rates, so the control plane could tell a node in trouble from a node whose dependency was in trouble.
From signals to a decision
The control plane turned each node's signal into a single headroom score. The important design choice was to take the minimum across dimensions rather than an average. Averaging hides the binding constraint, and the binding constraint is the only thing that matters. A node at ten percent CPU with a fully occupied thread pool is not a node with spare capacity.
headroom = min(
1 - busyThreads / maxThreads,
1 - dbPoolActive / dbPoolMax,
1 - clamp(gcTimeRatio / gcBudget),
1 - clamp(p99Latency / latencyBudget),
1 - clamp(loadAverage / (cores * 1.5))
)
# hysteresis: different thresholds to shed and to recover, plus dwell time,
# so a node on the edge does not oscillate in and out of the pool
shed when ewma(headroom) < 0.20 for 2 consecutive ticks
reduce when ewma(headroom) < 0.45
recover when ewma(headroom) > 0.55 for 6 consecutive ticks
Smoothing and hysteresis are not polish, they are the difference between a control loop and an oscillator. A single threshold with no dwell time produces a node that flaps, and a flapping member is worse than a down one because every transition resets connections and redistributes load onto neighbors that are also near the edge. Separate entry and exit thresholds, a short exponential moving average, and a minimum time in state fixed it.
Driving the load balancer with the verdict
The mechanism was almost anticlimactic once the scoring worked. Each node's monitor endpoint stopped reporting whether the process was alive and started reporting the control plane's verdict about it. The F5 was already polling that endpoint. We just made it mean something.
Three states, and the middle one is the one that mattered:
- Healthy. Monitor returns 200 with a full dynamic ratio. The member takes its normal share.
- Stressed. Monitor returns 200 with a reduced ratio value. The F5's ratio-weighted member selection sends the node proportionally less traffic without removing it. This is partial shedding, and it is the state that prevents the cliff. A node under pressure gets relief while still doing useful work, and it recovers instead of tipping.
- Saturated. Monitor returns 503. The member is marked down, new connections stop, and existing connections drain rather than being reset. Draining is only safe because sessions are no longer in that node's heap, which is the second time the session work paid for itself.
The load balancer loop runs on a poll interval measured in seconds. Saturation can arrive faster than that, so the node also protected itself locally. A servlet filter sitting in front of the business logic rejected work when in-flight requests crossed a ceiling, returning 503 with a Retry-After header before a thread was ever committed to real work. Cheap rejection is the entire idea: the cost of saying no must be far below the cost of saying yes, otherwise shedding is just a different way to spend the pool.
We also stopped shedding uniformly. Requests are not equal in cost or in value, so the filter classified them and shed in order: cacheable and static content first, then anonymous browsing, then authenticated browsing. Checkout, payment callbacks and in-progress uploads were never shed and held a reserved share of the pool that browse traffic could not touch. That is bulkheading, and it is what stopped a slow recommendation call from taking orders down with it.
One more rule, learned the way these things usually are. The control plane fails open. If beacons stop arriving, nodes are treated as healthy rather than pulled from the pool. A monitoring outage that removes every member from the pool is a self-inflicted total outage, and a safety system that can cause the accident it exists to prevent is not a safety system.
Rate limiting: bound what arrives
Load shedding protects a node from work that has already arrived. Rate limiting reduces how much arrives at all. They solve different halves of the problem and you need both, because shedding still costs you a connection, a parse and a decision for every request you refuse.
We placed limits at three levels. Coarse abuse control at the edge, keyed on client address and network, which mostly caught scrapers and misbehaving integrations. Per-identity and per-endpoint fairness at the application, so no single customer or partner could consume a disproportionate share. And a global budget per downstream dependency, which is the one people forget: your database does not care which customer is overloading it.
On algorithm choice, we used token bucket where burst tolerance mattered and a sliding window where fairness across the window mattered. We specifically avoided the naive fixed window counter, because a client can spend its full allowance at the end of one window and again at the start of the next, delivering double the intended rate across the boundary, which is exactly the sort of coordinated burst that peak traffic produces naturally.
Token bucket, refill and spend in one atomic script-- KEYS[1] limiter key ARGV: now_ms, refill_per_sec, burst, cost
local st = redis.call('HMGET', KEYS[1], 'tk', 'ts')
local burst = tonumber(ARGV[3])
local tk = tonumber(st[1]) or burst
local ts = tonumber(st[2]) or tonumber(ARGV[1])
local cost = tonumber(ARGV[4])
tk = math.min(burst, tk + (tonumber(ARGV[1]) - ts) / 1000 * tonumber(ARGV[2]))
local allowed = 0
if tk >= cost then tk = tk - cost; allowed = 1 end
redis.call('HMSET', KEYS[1], 'tk', tk, 'ts', ARGV[1])
redis.call('PEXPIRE', KEYS[1], 120000) -- idle keys evict themselves
return { allowed, math.floor(tk) }
Doing the refill lazily inside the script is what makes this work across a fleet. There is no timer, no background job, and no read-modify-write race between app servers, because Redis runs the script atomically and the arithmetic happens next to the data.
For the expensive endpoints where we wanted precision rather than an approximation, we used a sliding window log in a sorted set:
ZREMRANGEBYSCORE rl:{cust:44182}:render -inf 1701390361337 # trim the window
ZCARD rl:{cust:44182}:render # count what remains
ZADD rl:{cust:44182}:render 1701390421337 req:8c1
PEXPIRE rl:{cust:44182}:render 60000
That is exact, but it stores one member per request, so memory grows with the limit. We reserved it for costly operations and used a two-bucket weighted counter everywhere else, which approximates a sliding window at constant memory with two INCR calls.
Two refinements mattered more than the algorithm choice. The first is that we charged by cost rather than by request, because a photo book render and a product page are not the same event, and a limiter that counts them equally protects nothing. The second is that for protecting a thread pool, a concurrency limit beats a rate limit. What exhausts a pool is requests in flight, not requests per second, so we put a semaphore in front of each slow dependency and capped simultaneous calls. Little's Law again, applied deliberately this time.
Finally, the client contract. Every rejection returned 429 with Retry-After and the remaining budget in headers, and our own clients backed off exponentially with jitter. Jitter is not a detail. Without it, every client rejected in the same second retries in the same second, and you have built a mechanism that converts one overload into a periodic one.
The waiting room, and why it has tiers
Everything above makes a fixed amount of capacity go further. None of it creates capacity. On the biggest days, demand genuinely exceeded what the fleet could serve, and at that point the choice is not between serving everyone and serving some. It is between degrading in a controlled way and collapsing for everyone. A queue converts an outage into a wait, and a wait with a visible position is something people will tolerate.
Admission worked on a signed token. A short-lived token bound to the session, issued by the release loop and carried in a cookie. A request holding a valid token goes straight through to the application. A request without one goes to the waiting room.
First come first served is the wrong default
A single FIFO line sounds fair and behaves badly. Someone three clicks from completing an order, who already spent twenty minutes building a photo book, lands behind forty thousand people who arrived a second earlier to browse. That is a lost order, a support call, and a customer who does not come back. Meanwhile the browsers, who would have waited happily, gained nothing from being ahead.
So we queued by how much work the customer had already invested, and by how close they were to completing it:
- Tier 0, never queued. In-flight transactions: checkout, payment callbacks, uploads in progress. These held reserved capacity. Queueing a payment callback is how you create a reconciliation problem on top of a capacity problem.
- Tier 1. Returning customers with a resumable cart or an in-progress project. This is the third time the session work paid off: the resume sorted set is what let the edge know, before touching the application, that this person had something waiting for them.
- Tier 2. Authenticated browsing.
- Tier 3. Anonymous browsing and unidentified clients.
One sorted set per tier. Member is the admission id, score is arrival time in milliseconds. That gives first come first served within a tier and strict priority across tiers.
# join: score is arrival time, so ordering is wall clock, not insertion order
ZADD wr:q:2 1701390421337 adm:9f2c
# position: rank within the tier, plus everyone waiting in higher tiers
ZRANK wr:q:2 adm:9f2c
ZCARD wr:q:0 ZCARD wr:q:1
# release, highest priority tier first, as many as capacity allows
ZPOPMIN wr:q:0 120
ZPOPMIN wr:q:1 120
# heartbeat from the waiting page, and the janitor that drops abandoners
ZADD wr:alive 1701390455000 adm:9f2c
ZREMRANGEBYSCORE wr:alive -inf 1701390425000
Using arrival time as the score rather than a monotonic counter turned out to matter. Customers sign in while waiting, which promotes them from tier 3 to tier 2. Promotion should move you to a better line, not send you to the back of it, so we carried the original score across:
-- KEYS[1] current tier, KEYS[2] target tier, ARGV[1] admission id
local score = redis.call('ZSCORE', KEYS[1], ARGV[1])
if not score then return 0 end
redis.call('ZREM', KEYS[1], ARGV[1])
redis.call('ZADD', KEYS[2], score, ARGV[1]) -- keep the original arrival time
return 1
The drain rate is the whole design
A waiting room that releases at a fixed rate is just an outage on a delay. If the fleet is struggling, a constant drip still overwhelms it; if the fleet has recovered, a constant drip keeps people waiting for nothing.
So the release rate came from the same headroom score the shedding loop was already computing. The control plane knew the fleet's aggregate spare capacity every few seconds, and the waiting room released exactly that much. When nodes recovered, the line moved faster without anyone touching a configuration value. When a dependency degraded, the line slowed automatically and the application never saw the load it could not handle. Two mechanisms, one signal, which is why they never disagreed with each other.
A single leader ran the release loop, elected through a lease key with a TTL, so the queue was drained by one process rather than by every node racing. On each tick it popped from the highest non-empty tier, minted admission tokens, and recorded them with a TTL matched to a reasonable session length. Admissions that went unused expired and returned their capacity to the pool.
Abandonment is the detail that separates a queue that works from one that lies. People close the tab. If the queue counts them, the position and the estimate are both wrong, and wrong estimates destroy the trust that makes people willing to wait. The waiting page sent a heartbeat, the janitor removed anyone who stopped sending it, and estimated wait was computed only from entries known to be alive.
Last, and this is the mistake I have seen sink otherwise good waiting rooms: the waiting page itself must cost almost nothing. Ours was static and served from the CDN, with no call into the application it was protecting. The only dynamic call was a small position endpoint that read the sorted sets directly, with responses cached per bucket rather than per customer, and a jittered poll interval so that tens of thousands of waiting browsers did not all ask at the same instant. A waiting room that loads the system it shields is not a waiting room, it is extra traffic with a friendly message on it.
What it bought, and what it did not
Peak stopped producing hung nodes. Losing a node became a non-event rather than a data-loss event, because state was no longer in its heap. Horizontal scaling started working the way it was supposed to, which is to say adding servers actually took load off the existing ones. Operators got a leading indicator, thread pool and connection borrow wait, instead of a pager that fired after customers were already affected. And on the days when demand still exceeded capacity, customers saw a place in line and an estimate instead of an error page, with the ones closest to completing an order moving first.
What none of it fixed: it was still a monolith. One deploy unit, one thread pool, one blast radius, one scaling decision for a system with wildly different workloads inside it. Every mechanism in this post is a compensating control for the absence of independent scaling. A waiting room is a confession that you cannot add capacity fast enough, and load shedding is a way to fail gracefully at a limit you should be able to raise. They are the right engineering when the constraint is real. They are not the destination.
Which is what the next year was for. We took the monolith apart along domain boundaries, starting with the services everything else depended on, and moved the estate to the cloud where capacity is elastic and a scaling decision is per service rather than per fleet. The interesting part is that none of the stabilization work was thrown away. It was repositioned:
- Distributed sessions stopped being a peak-season fix and became a precondition. Stateless services behind any load balancer require externalized state, so the work we did to survive a holiday was the same work required to decompose at all.
- The beacon's signals became service-level metrics, and the same numbers that decided whether to shed became the inputs that decide when to autoscale. Saturation signals are saturation signals whatever acts on them.
- Shedding and rate limiting moved out of a servlet filter and into the gateway and the service layer, where they became per-service policy instead of one global rule for everything.
- The waiting room stayed exactly where it was, as the last line of defense for the handful of events each year where demand exceeds any capacity a reasonable business would provision for.
The sequencing lesson is the one I have carried into every role since. Stabilize first, then transform, and be explicit with everyone about which one you are doing. Stabilization buys the calendar time and the credibility that the transformation needs. It is tempting to skip it and go straight to the architecture you actually want, and the reason that fails is not technical. It is that you spend the next peak firefighting instead of building, and the transformation quietly becomes the thing that is always one quarter away.