The model recommends. Plain code decides.

Somebody has to say which payment paid which bill

A company sends out invoices. Money arrives in its bank account. Someone has to decide that the ₹9,564 that landed on Tuesday pays off the ₹10,000 invoice from last month.

That sounds like a lookup and it is not, for four reasons. The name on the invoice is ABC Technologies and the name on the bank line is NEFT/ABCTECHPVTLTD/882910. The amounts do not agree. The shapes do not agree either: one payment can settle three invoices, one invoice can arrive in two parts. And sometimes there is no answer at all, because the customer simply has not paid.

The amount is always smaller, and always legally so

The gap between billed and received is not an error. It is the payment gateway's fee, the tax on that fee, and the tax the customer is required by law to withhold. On a ₹10,000 invoice that is ₹200, ₹36 and ₹200, and ₹9,564 arrives.

A matcher that compares ₹10,000 to ₹9,564 raises an exception for every single well-behaved payment. So the deductions get computed first, and only then compared. That turns a pile of fake exceptions into clean matches, and the arithmetic it used becomes the explanation shown to whoever is reviewing.

Where the money goes on a ₹10,000 invoice

How I got there

Three passes, in the order they happened.

Three rules I did not break

The routing is plain code, not a model. Three fixed branches, decided by whether a reference number is present. A model there would add cost, latency and randomness in exchange for nothing.

The model never does arithmetic. Every number in the system is computed by code. The model picks between options that code has already priced, and writes the sentence explaining the choice.

The model never has the final say on money. It recommends. Hard-coded rules decide whether that recommendation is allowed to become an action. It touches roughly a fifth of records; the rest never reach it.

One record, from bank line to decision

The router is three branches of ordinary code. Which one a record takes decides whether it costs anything at all.

Both branches end at the same place: the hard rules, and then one of three outcomes. Approved on its own, held for review, or an exception with the reason written out. Every decision is appended to an audit log.

Three layers, all of them if statements

Most projects build only the third layer, the one that checks what a model said. That is the least important of the three. The second one is where the money is actually protected, and it runs whether or not a model was ever involved.

Before anything runs: the amount is positive, the currency is there, the date parses, every record has an ID. On every proposed match: the list below, all of it. On the model's output: valid JSON, and the ID it chose has to be one of the ones it was given.

The thresholds, in one file

Kept together rather than scattered through the checks, so the tuning curve has one place to move and a reader has one place to look.

AUTO_SCORE = 0.90
MARGIN_FLOOR = 0.15
ADJUDICATE_FLOOR = 0.70

# A wrong Rs 500 match is annoying. A wrong Rs 5,00,000 match is how fraud
# gets through.
VALUE_CEILING_PAISE = rupees(500_000)

# Payment timing, measured from the due date. Must use the same anchor as
# score_date, or a record passes one check and fails the other arbitrarily.
DATE_WINDOW_DAYS = (-7, 45)

# Same amount, same counterparty, this close together: flag both.
DUPLICATE_WINDOW = timedelta(hours=48)

# New names are the highest-risk category, so a counterparty must have been
# settled with this many times before their payments can be automated.
NEW_COUNTERPARTY_MIN = 3

# The only two rules an adjudicator's answer can satisfy. Everything else is
# policy rather than judgement, so a model's opinion cannot move it.
ENDORSABLE_RULES = {"score", "margin"}

Two ideas in the scoring worth keeping

A pair is scored on four independent signals: the reference number carries half the weight, the amount a quarter, the name and the date the rest. When a signal is missing, the remaining weights are renormalised rather than the missing one being scored zero. A bank line with no reference would otherwise be capped at 0.50 however perfect everything else about it was. Absence of evidence is not evidence against.

And the score alone is not enough. What matters is the margin: the best score minus the runner-up. A 0.95 means nothing if second place scored 0.94. That is a coin flip, not a match, and only the margin catches it.

What it does on the held-out set

160 records were generated and split three ways: 30 to seed the alias table, 45 to tune the weights, and 85 held back. Only the 85 are reported.

Source · synthetic data, generated with its own answer key. Real bank statements would be a different test.

72.9% is not a number I tuned towards. It is the ceiling. Sixty-two of those 85 records are the ones that should be automated; the other 23 genuinely need a person to look at them. Anything above 62 would not be an improvement, it would be a wrong approval.

That is the whole reason match rate on its own is a bad measure. Match everything to anything and you score 100%. So missed matches and wrong matches are counted separately, and both are broken down by scenario.

Turning the pieces on one at a time

Every row is a real run over the same 85 records, with one more layer switched on.

Source · api/ablate.py, same held-out set for every row

The layer everyone would build first is the one that does the damage, and the layer with the model on it is worth the least.

Memory is the easiest place to lie to yourself

Since memory is worth more than everything else combined, it is also where a fake number would be easiest to produce: let a batch learn from the records it is being graded on and the score climbs beautifully and means nothing.

So a batch cannot learn from itself. Learning returns a new memory rather than modifying the one in use, and worked examples drawn from the graded records are filtered out of a graded run. An improvement can only ever show up in a later run.

The same records, two days apart

Half the customers start with history. The batch runs, a reviewer confirms what was held, and it runs again.

Source · learn.py, 85 records

What building it taught me

The interesting question was never how to match more. It was what a match has to survive before it is allowed to count.

Where it is still rough

Three things, and the first is the one that matters most.

What Hisaab came down to: