Card Fraud Models: Optimising for the Right Error

Card fraud detection is one of the clearest wins for machine learning in financial services, and it is also where naive modelling produces the most confidently wrong results.

The reason is the base rate. Fraud is rare — a fraction of a percent of transactions — so a model that approves everything achieves better than 99% accuracy. Any evaluation using accuracy will rank that model highly, which is why the first thing to fix is the metric.

Measure precision and recall, at a chosen operating point

What matters is how many genuine frauds you catch (recall) and what proportion of your declines are actually fraud (precision). These trade against each other, and where you sit on that curve is a business decision rather than a modelling one.

The two errors have very different costs. A missed fraud costs the transaction value plus handling. A false positive declines a legitimate customer, which costs the sale, the support contact, and some probability that the customer stops using the card — a cost that is real, delayed and rarely measured properly.

Precision-recall curves rather than a single threshold are the right way to present model performance to the people making that decision, because the question they are answering is where on the curve to operate.

Validate across time, not at random

Random train-test splits leak information in fraud data. Fraud arrives in bursts — a compromised merchant, a stolen card tested rapidly, a campaign against one issuer — and a random split places transactions from the same burst on both sides, so the model has effectively seen the answer.

The result is validation performance that does not survive production. Splitting by time, training on earlier data and validating on later, gives a realistic picture and is usually a sobering one.

Features carry more of the signal than the algorithm

A single transaction contains little that distinguishes fraud. Amount and merchant category on their own are weak. The signal is in context: how this transaction compares with this cardholder's history, and how the card's recent activity compares with its own normal pattern.

Velocity features are the workhorses — count and value in the last hour, day and week, distinct merchants, distinct countries, time since the previous transaction, whether the merchant category is one this card has ever used. Geographic implausibility, where two transactions occur further apart than travel permits, is a strong signal.

All of this must be computed as it would have been at decision time. Using aggregates that include the transaction being scored is leakage, and it inflates offline results while producing nothing in production.

The infrastructure constraint is latency

An authorisation decision has a budget measured in tens of milliseconds. That constrains everything: the features must be retrievable in that window, which means precomputed and stored rather than calculated on demand, and the model must be small enough to score quickly.

This is why online and offline feature consistency is a recurring source of production defects. A feature computed one way in training and another way at serving time produces a model that behaves differently in production, and the discrepancy is difficult to detect because both pipelines look correct in isolation.

Fraud adapts, so the model decays

Unlike most prediction problems, this one has an adversary. When a pattern stops working, the people behind it change it. That means degradation is not gradual drift but a step change when a new technique appears.

Monitoring therefore needs to look at more than aggregate performance. Score distribution shifts, approval rate changes by segment, and the emergence of clusters of similar declined transactions all indicate something has changed before the outcome data confirms it.

Label delay compounds this. Confirmed fraud can take weeks to arrive through chargebacks, so recent performance is always partly unknown. Building the monitoring around leading indicators rather than confirmed labels is what makes it useful.

Rules and models are complementary

Replacing a rules engine with a model is usually the wrong ambition. Rules are immediate, explainable and adjustable within minutes — genuinely valuable when a specific attack is running right now and you need it stopped before retraining is possible.

Models generalise to patterns nobody has written a rule for. Most effective systems run both: rules for known patterns and for hard limits, a model for everything else, and a way for analysts to add a rule quickly when something new appears.

Regulatory context constrains the design

Fraud decisions sit inside a regulatory frame that varies by market. Strong customer authentication requirements in some jurisdictions mandate additional verification for certain transactions, with defined exemptions that a risk-based system can claim — but claiming an exemption means being able to demonstrate the risk assessment that justified it.

Liability rules also shape the economics. Where liability shifts between issuer, acquirer and merchant depending on how a transaction was authenticated, the cost of a missed fraud is not uniform, and a model tuned on transaction value alone is optimising the wrong quantity.

This is worth establishing with your compliance function before setting thresholds, because the right operating point depends on who bears the loss.

Feedback loops distort the training data

A model in production shapes the data used to train its successor. Transactions it declines never complete, so you never learn whether they would have been fraudulent. Over successive retraining cycles the model becomes increasingly confident about a region of the input space it no longer observes.

The standard mitigation is to allow a small random sample through below the decline threshold, accepting some loss in exchange for unbiased labels. It feels wrong to approve transactions you believe are fraudulent, and it is what keeps the model honest over time.

Where that is commercially unacceptable, the alternative is to at least record what the model would have done under alternative thresholds, so future analysis can reason about the counterfactual rather than ignoring it.

Explaining a decline

Analysts investigating a case, and support staff talking to a declined customer, both need to know why. A score with no reasons makes investigation slow and customer conversations impossible.

Feature attribution per decision addresses this, and it needs designing in rather than bolted on. It also feeds back into model improvement, because the cases where the stated reason looks wrong to an experienced analyst are the most informative examples you have.