Building Fraud Detection: Decisions Before Models

Fraud detection projects usually begin with a model and discover the system afterwards. The more reliable order is the reverse, because four decisions constrain everything the model can be, and all four are architectural.

Where the decision happens

Blocking in real time, before a transaction completes, gives a latency budget measured in tens of milliseconds and prevents loss outright. Reviewing after the fact allows far more computation and richer context, and accepts that some loss has already occurred and must be recovered.

These are different systems, not different settings. Real-time requires precomputed features, a small fast model and a hard timeout with a defined default when it is exceeded. Post-hoc allows the opposite in every respect.

Most mature systems run both — a fast inline check and a slower analytical pass — and knowing which decisions belong where is the first architectural question rather than an optimisation.

What your latency budget actually is

The number matters because it eliminates options. Under fifty milliseconds, features must be retrieved rather than computed, which means a store maintained continuously and kept consistent with how those features are calculated in training.

That consistency requirement is the most common source of production defects in these systems. A feature computed one way offline and another way online produces a model whose live behaviour differs from its validated behaviour, and the discrepancy is hard to spot because both pipelines look correct on their own.

How labels arrive, and how late

Fraud labels are delayed and incomplete. A chargeback can take weeks. Some fraud is never reported. Some reported fraud is not fraud but a customer dispute or a family member's transaction.

Two consequences follow. Recent performance is always partly unknown, so monitoring cannot depend on confirmed labels and must use leading indicators — score distributions, approval rates by segment, clusters of similar declines. And the training data is systematically biased, because transactions you blocked have no outcome at all, so the model never learns whether it was right.

The standard mitigation is allowing a small random sample through below the block threshold to obtain unbiased labels. It is uncomfortable and it is what keeps the model from becoming confident about a region it no longer observes.

Who reviews an alert, and what they need

Alerts go to people, and the capacity of that team sets your operating point far more than model quality does. A model that generates a thousand alerts a day for a team of three is not a good model regardless of its precision.

What the analyst needs is context in one place: the account's history, the reasons this transaction scored as it did, similar recent cases, and the ability to record an outcome that feeds back into training. Systems that surface a score without reasons make investigation slow and the feedback poor, which degrades the model over time.

Analyst dispositions are also your highest-quality label source, arriving far sooner than chargebacks, so the review tool is part of the modelling infrastructure rather than a separate operational concern.

Rules and models both, permanently

A rules engine is not a legacy artefact to be replaced. Rules are immediate, explainable and changeable within minutes, which is exactly what you need when a specific attack is running now and retraining takes days.

Models generalise to patterns nobody wrote a rule for. The productive arrangement runs both, with a clear precedence, and gives analysts the ability to add a rule quickly. Insisting on a pure machine learning system removes your fastest response mechanism.

The adversary adapts, which changes monitoring

Unlike most prediction problems, this one has an opponent who observes your behaviour. When a technique stops working they change it, so degradation arrives as a step change rather than gradual drift.

Monitoring must therefore watch for novelty as well as performance: clusters of similar transactions that are new, shifts in the score distribution, changes in which features are driving decisions. By the time confirmed loss data shows a problem, the campaign has been running for weeks.

Consortium data helps and carries obligations

Fraud signals shared across institutions catch patterns no single organisation sees — a device or identity appearing at several firms in quick succession is a strong indicator that individual systems miss entirely.

Participation brings constraints. What may be shared is limited by data protection law, contributions usually have to be reciprocal, and matching identities across organisations without exchanging personal data requires privacy-preserving techniques rather than a plain identifier.

Worth evaluating on whether the fraud you actually suffer is cross-institutional. For card-not-present and identity fraud it usually is; for narrow internal abuse it usually is not.

Plan for the false positive, not just the fraud

Every fraud system declines legitimate customers, and the quality of that path determines what the errors actually cost. A customer declined with a generic message and no route to resolution is a customer who may not return.

What reduces the damage: a specific, non-alarming message, an immediate self-service verification route, and a support process that can resolve it in one contact rather than escalating. Then measure it — the rate at which declined-and-verified customers continue transacting is a direct read on whether your threshold is set sensibly.

Setting the threshold is a business decision

Precision and recall trade against each other, and where you operate depends on the relative cost of a missed fraud and a declined legitimate customer. The second cost is consistently underestimated because it is delayed and diffuse — the lost sale is visible, the customer who quietly uses a different provider afterwards is not.

Present the trade as a curve rather than a single number, get the business to choose a point, and revisit it as conditions change. Engineering teams that set the threshold themselves are making a commercial decision by default.