A rate limiter that survives its own database going down

The choice nobody gets to revisit

A rate limiter has to count requests somewhere. Put the count in Redis and every server in the cluster agrees, but every request now costs a network round trip. Keep it in memory and it is free, but each server counts on its own, so the real limit is your limit times the number of servers.

Both are reasonable. The problem is that you pick one when you write the line of setup code, and the process is stuck with it for its whole life. The moment that choice is wrong is exactly the moment you cannot change it: Redis is slow or gone, and the limiter is still dutifully waiting on it.

What that failure actually looks like

When Redis stops answering, a limiter built on it has two options and both are bad. It can fail open and let everything through, which is how a struggling service gets finished off. Or it can fail closed and reject everything, which turns a database problem into a total outage.

Neither is a bug. It is what you get when the storage decision is made once and cannot be revisited.

How I got there

Three passes at it, in the order they actually happened.

Make the choice a runtime decision

The fix is not a better algorithm. It is moving the choice from startup to per-request. If the limiter asks which strategy to use every time it is called, then a server under stress can answer differently from one that is idle.

That turns the pick-one problem into a policy question, and a policy question can have a good answer: use the accurate expensive path when you can afford it, and the cheap local path when you cannot.

The whole adaptive limiter

This is the entire thing. It is small on purpose: all it does is ask, then delegate.

export const createAdaptiveLimiter = (options: AdaptiveOptions): RateLimiter => {
  const consume = async (key: string): Promise<RateLimitResult> => {
    const strategyName = await options.selector(key);
    const limiter = options.strategies[strategyName];

    if (!limiter) {
      throw new Error(`Strategy '${strategyName}' not found in adaptive limiter definitions.`);
    }

    return limiter.consume(key);
  };

  return { consume };
};

Everything interesting sits either side of it. Underneath, algorithms and stores are separate pieces that mix freely: three algorithms, two stores, any combination. Above it, the selector is yours.

One request, end to end

The path a single request takes, and where the decision happens.

Both paths, side by side

The same request, down whichever lane the selector picked. Nothing above the store layer knows which one it got.

The part that had to be atomic

Token bucket cannot be done with ordinary Redis commands. Reading the bucket, working out the refill and writing it back is three round trips, and two servers doing that at once will both read the same tokens and both spend them.

So the whole calculation is a Lua script that Redis runs as one operation. Read the hash, add tokens for the time that has passed, cap at capacity, spend one if there is one, write it back, set a TTL sized to how long a full refill takes so idle keys clean themselves up.

What it costs when everything is fine

Against the two libraries people actually use. Docker, Node 20 Alpine, shared Redis, k6 driving 60 concurrent users for 30 seconds at 100 requests per 60 seconds.

Source · benchmarks.md, reproducible from the repo

Limitra is slower. Roughly 2.7ms slower at the tail than the fastest library tested, and that is the honest cost of routing every call through a decision instead of straight to a store. It is worth writing down rather than hiding, because the whole argument rests on what that 2.7ms buys.

Then I cut the cable

The second test is the one that matters. Same setup, but ten seconds in, the Redis connection is severed. The client is configured the way a serious production service would be: no offline queue, one second command timeout. No hiding behind a buffer that quietly holds requests until Redis comes back.

Same outage, three libraries

Source · k6, 31 virtual users, 30.1s, 7,274 iterations

Both express-rate-limit and rate-limiter-flexible went to zero. Not degraded, zero: every request in the window failed. Limitra served all of them at 13ms, because the selector saw the unhealthy state and stopped routing to Redis without anyone touching it.

That is the trade the whole library exists to make. Give up 2.7ms in the good case to not have a bad case.

What building it taught me

The interesting decision was not which algorithm to use. It was refusing to make the storage choice permanent.

Where it is still rough

Three things I would fix before calling this finished, and they are in the code today.

What Limitra came down to: