The endpoint a crowd hits hardest could not survive a crowd

A night out, run as a game

Yohop is three products in one. Guests find what is on near them tonight and get rewarded for going. Venues get a full back office for deals, menus, events and loyalty. Admins run cities, approvals and every feature switch.

It is live, with over 70 features finished across those three sides.

For guests

For venues

For admins

A busy venue is a crowd checking in

Yohop has two sides. Venues post food and drink deals that run for a few hours. People nearby find them on a map, check in, and earn points, streaks and a place on a leaderboard. The first check-in at a deal also earns a one-time bonus.

That bonus is the hard part. Two requests from the same person, a few milliseconds apart, must not both decide “this is your first visit” and pay it twice. The first version guarded it the textbook way, with a Serializable transaction and a retry on conflict.

Before launch I wrote a load test with k6: 200 virtual users against a seeded database of 2,000 venues, 6,000 deals and 5,000 people, crowded round a few cities the way real traffic would be.

How I got there

Three passes, in the order they happened.

Why the retries never stood a chance

Serializable does not stop a race by making requests wait. It lets them run, notices when two could have seen each other, aborts one and expects you to try again. That works when collisions are rare. On a popular deal they are not rare.

Worse, most of the collisions had nothing to do with the bonus. Under Serializable, the query “has this person checked in here before” is a predicate read, so two different people checking in at the same deal conflicted with each other. The busier a venue got, the more of its own customers it failed.

An earlier audit had suggested a unique constraint on check-ins per person per deal. That would have broken the product: checking in again at the same deal is allowed, and each visit earns points. Only the bonus is once per deal.

One check-in, after the fix

The bonus moved from a rule the transaction had to enforce to a row the database refuses to write twice. Everything slow moved off the request.

The insert that is also the check

Raw SQL on purpose. Postgres marks a transaction as aborted the moment any statement errors, and Prisma does not wrap single queries in savepoints, so catching a unique violation would leave the rest of the transaction unusable. ON CONFLICT never raises.

const bonusRows = await tx.$queryRaw<Array<{ id: number }>>`
  INSERT INTO "UserPointEvent"
    ("userId", "dealId", "pointEventTypeId", "points", "createdAt")
  VALUES
    (${userId}, ${deal.id}, ${POINT_EVENT_TYPES.FIRST_CHECKIN_DEAL},
     ${firstCheckInBonus}, NOW())
  ON CONFLICT ("userId", "dealId")
    WHERE "pointEventTypeId" = ${FIRST_CHECKIN_TYPE_LITERAL} AND "dealId" IS NOT NULL
  DO NOTHING
  RETURNING "id"
`;

if (bonusRows.length > 0) {
  // this request won the race; it is the only one that ever will
}

The second finding: megabyte cache entries

With check-ins fixed, the test pointed at the map. The “deals near me” endpoint cached its whole hydrated result, every deal with every relation, before cutting it to a page. Single cache entries reached 1.31MB. A 384MB Redis held about 500 of them and evicted 2,901 keys in one nine-minute run, spending 45% of its CPU on eviction churn.

The fix was to split the endpoint in two. A narrow scan picks candidates with only what decides membership and order: an id, the venue's coordinates, the deal type. The cache stores just the ordered list of ids and distances. Only the page the app asked for is loaded in full.

It also fixed a correctness bug. The old query cut the list to 1,000 rows, newest first, before measuring any distance, so a deal 200m away could be dropped for a newer one 20km away. The test hit that cap 1,384 times. Now the exact distance sort runs over everything in the area before anything is cut.

The cache, before and after

Source · k6 run, 200 virtual users, same seeded data both times.

The same test, passing

Same harness, same data, same 200 virtual users. The original run, and the run after both fixes.

Source · k6 load test, 200 virtual users, same harness and data before and after.

Every endpoint at p95

Source · Between 18x and 27x faster at p95, with zero errors.

Getting ready for a second server

Everything runs as Docker containers on one server today, but nothing assumes it will stay that way. One environment variable decides what a container is: the web role serves HTTP and WebSockets and can run as many copies as needed; the worker role owns the scheduled jobs and runs exactly once.

A few smaller decisions came out of making that true.

What Yohop came down to: