Notifly · notification service · TypeScript
Every line in the handler is where it is for a reason
Notifly takes a notification and gets it out over email, SMS or push. Almost all of the engineering is in the order things happen in, because that order is what decides whether a notification can be lost.
Sending is the easy part
Handing a message to an email provider is one call. The hard parts are all around it. The provider will be slow, so the caller cannot wait on it. The provider will be down, and the message still has to go eventually. Sometimes the message can never be sent at all, because the address is wrong, and retrying it forever helps nobody.
And there is more than one provider. If SMS is having a bad afternoon, email and push have no business being held up by it.
The thing that must never happen is a silent loss: a caller told the send was accepted, and no record of it anywhere. Everything below exists to make that impossible rather than unlikely.
How I got there
Three passes, in the order they happened.
Explore
Made sending itself tougher
Realise
None of it survives one bad provider
Insight
Fix the order, not the sending
Write it down before you promise it. Then go and send it.
Three decisions everything else follows from
One queue per channel, not one queue with a channel field. If the SMS provider backs its queue up, email and push and their workers carry on. A single outage cannot block the others, and that property is impossible to get back once everything shares a lane.
The row is written before the job is queued. The worker is the only thing that ever promotes a row past queued. If the process dies between those two lines, the row sits at queued with no job behind it, which is something you can find and fix. If it dies after, the job is safe in Redis. There is no ordering where the caller was told yes and nothing exists.
Some failures are not worth retrying. A bad recipient, a rejected payload, a missing key: those fail identically five times in a row. They get their own error type, and the queue stops on the first one.
One request, in the order it happens
Each step is where it is because of what it protects. Move any of them up or down and something can be lost, orphaned, or discovered too late.
Check the shape, then the user
The user id is checked for being a UUID before it reaches the database, so a malformed id is a clean 404 rather than a Postgres syntax error surfacing as a 500.
server/src/routes/notifications.ts
Has this person opted out
Consent is checked before anything is spent or written. No row means opted in, which is the default that keeps a fresh user reachable.
Rate limit, before anything is written
Deliberately after consent and before the first write, so an over-limit request leaves no trace at all: no log row, no job. Nothing was accepted, so nothing was lost. The caller gets a 429 and a Retry-After.
server/src/rateLimit.ts
Render the template here, not later
An unknown template is a 422 to the caller, right now, while somebody is listening. Render it in the worker instead and a broken template fails in a place nobody is watching, behind a row that already said queued.
server/src/templates/index.ts
Write the row, then queue the job
This order and not the other one. Die in between and the row sits at queued with no job, which is reconcilable. Die after and the job is durable in Redis. There is no window where the send was promised and no record exists.
Return 202 and stop waiting
Accepted, will be processed. The caller is not held open while a provider decides how it feels today, which was the entire point of putting a queue in the middle.
Retrying is a policy, not a reflex
The retry policy sits on the queue rather than on each call that adds a job. Five attempts, exponential backoff at roughly two, four, eight and sixteen seconds. Putting it on the queue means every job inherits it and no future call site can forget to pass it.
Backoff matters as much as the count. A provider that is struggling gets room to recover instead of being hammered by a tight retry loop, which is how a wobble becomes an outage.
The failure that must not be retried
A bad address will be just as bad on the fifth attempt. Subclassing the queue's own unrecoverable error means the job stops on the spot, and the type is something the worker can actually branch on.
import { UnrecoverableError } from "bullmq";
// A send failure that retrying cannot fix — a bad recipient, a rejected payload,
// a missing/invalid API key. Subclassing BullMQ's UnrecoverableError means the
// moment a provider throws this, BullMQ STOPS retrying and fails the job on this
// attempt (no waiting out all 5 attempts on something doomed to fail identically).
//
// The subclass (rather than sniffing err.name) gives the worker a type-safe
// `err instanceof PermanentError` signal so it can mark the row `failed`
// immediately instead of the misleading `retrying`.
export class PermanentError extends UnrecoverableError {
constructor(message: string) {
super(message);
this.name = "PermanentError";
}
}What each queue setting is actually for
Setting
What it prevents
attempts: 5
A provider timeout or a 503 is normal weather. One hiccup should not lose a send.
backoff: exponential, 2s
Room for a struggling provider to recover, instead of a tight loop turning a wobble into an outage.
removeOnComplete: 1000
Redis keeps finished jobs forever by default, which is a slow memory leak at any real volume. The database log is the durable record; the queue only keeps the last thousand for a look.
removeOnFail: false
A job that has exhausted its retries has to stay inspectable. Dropping it would quietly undo the whole no-loss guarantee.
What building it taught me
What I saw
What it meant
The ordering was the design
There is almost no clever code in the request handler. What makes it correct is which line comes before which, and every one of those positions is defensible for a specific failure.
Policy on the queue beats policy at the call site
Retry settings passed at each add() are settings somebody will forget next year. On the queue they are structural: a new call site inherits them without knowing they exist.
A shared queue is a decision you cannot reverse later
One lane with a channel column looks identical until the day one provider is slow, and by then isolation is a migration rather than a config change.
Two failures that look the same want opposite handling
A timeout and a rejected address both surface as an error from a provider. Retrying the first is correct and retrying the second is four wasted attempts and a misleading status on the row.
The interesting work was deciding what must be true after every line, not what the code does on a good day.
Where it is still rough
Three things I would want before trusting it with real volume.
Still open
There is no reconciler. The design leans on rows being findable when they are stranded at queued with no job behind them, and nothing goes looking for them yet. The guarantee is only as good as the sweep that is not written.
Still open
The providers are stubs. The ordering, the retries and the permanent-failure path are all real; what they are wrapped around is not a live email or SMS account, so none of this has met a provider's actual rate limits or its more creative error responses.
Still open
Nothing has been load tested. The description says it is built to handle a lot; what is true is that it is shaped so that volume goes into a queue rather than into a request. That is a reason to expect it to hold, not a measurement.
What Notifly came down to:
01
Write before you promise
The row exists before the job does, so there is no moment where a caller was told yes and nothing was recorded.
02
One lane per channel
A provider having a bad afternoon backs up its own queue and nobody else's.
03
Know which failures are final
A timeout deserves five attempts. A bad address deserves one, and a status that says so.