Recommender Systems: The Cold Start Never Ends

Recommender systems are usually described as a modelling problem and behave in production as a data and feedback problem. The algorithms are well documented and largely commoditised; the difficulties are structural.

Cold start is a steady state, not a launch phase

Cold start is often framed as something you get through. In reality new users arrive every day and new items are added continuously, so at any moment a substantial share of your catalogue and audience has insufficient interaction history.

That means the fallback path is not a temporary measure — it is a permanent, heavily used part of the system, and it deserves the same attention as the main model. Content-based similarity for new items, popularity within a segment for new users, and explicit onboarding preferences all work, and all need to be built rather than assumed.

Systems where the fallback was an afterthought perform visibly worse for exactly the users forming their first impression.

Offline metrics reward the wrong thing

Evaluating a recommender by how well it predicts held-out interactions has a fundamental problem: it rewards predicting what the user would have found anyway.

A model that recommends the most popular items scores well offline and adds little, because those items would have been discovered regardless. The value of recommendation is in what the user would not have found, and held-out prediction accuracy does not measure that.

Offline metrics are useful for eliminating clearly worse models. Deciding between plausible candidates requires an online test measuring something you actually care about — engagement with recommended items specifically, downstream retention, catalogue breadth consumed — rather than click-through in isolation, which is easy to inflate with sensational items.

The feedback loop narrows the catalogue

A recommender trained on interactions influences which items get seen, which determines future interactions. Items the model surfaces accumulate data and become better understood; items it does not surface receive nothing and remain unknown.

Left alone, this concentrates recommendations onto a shrinking set. The catalogue effectively narrows, long-tail items become invisible, and the system appears to be performing well by its own metrics while serving a progressively smaller slice of what you offer.

Countering it requires deliberate exploration — surfacing items the model is uncertain about, accepting a small performance cost for information. It also requires monitoring catalogue coverage as a first-class metric, because it degrades silently.

Popularity bias is not the same as quality

Related but distinct: popular items are popular partly because they were shown more, and a model trained on interaction counts will infer that they are better rather than that they were more visible.

Correcting for exposure — modelling the probability an item was shown, and weighting accordingly — is worth doing where the catalogue matters commercially. Without it, new and niche items face a barrier that has nothing to do with their appeal.

Serving constraints shape the architecture

Recommendations usually need to be produced within a page render, which rules out scoring an entire catalogue per request. The standard shape is two-stage: a fast candidate retrieval step narrowing millions to hundreds, then a more expensive ranking model over that shortlist.

Most of the achievable quality sits in retrieval, because ranking cannot recover an item that was never a candidate. Teams frequently invest in the ranking model while the retrieval step quietly discards good options.

Precomputing recommendations is cheaper and goes stale, which is acceptable for slow-changing catalogues and poor for anything driven by recent behaviour.

Business rules belong outside the model

Real systems have constraints the model should not learn: do not recommend out-of-stock items, respect content restrictions, do not show the item just purchased, honour contractual placement obligations, apply diversity limits so one category does not fill the page.

Encoding these as a post-processing layer rather than training them in keeps the model focused on preference and the rules changeable without retraining. It also makes them auditable, which matters when someone asks why a specific item appeared.

Privacy constrains what you may infer

Recommenders work from behavioural data, and behavioural data can reveal more than the user shared deliberately. Purchase and viewing patterns can indicate health conditions, pregnancy, financial difficulty, religious observance or sexual orientation.

The engineering questions that follow are concrete. Are inferred attributes stored, and are they subject to the same protections as declared ones. Does a deletion request reach the derived profiles and the trained model, or only the raw events. Can a user see and correct what has been inferred about them.

There is also a judgement question that no regulation fully answers: some accurate inferences should not be surfaced, because doing so reveals to the user — or to whoever is looking at their screen — something they did not choose to disclose.

Watch what recommendations do at the edges

Optimising a single engagement metric tends to push content toward whatever produces the strongest short-term response, which is not always what serves the user or the business.

The practical safeguards are to optimise for something further downstream than a click — a completed purchase, a retained subscription, a returning session — and to monitor the distribution of what is being recommended rather than only the average performance. Concentration into a narrow band, or drift toward a particular kind of item, is visible in that distribution well before it appears in headline metrics.

Explanation improves acceptance

Telling users why something was recommended — because you watched this, because people who bought this also bought that — measurably improves trust and gives them a way to correct a wrong inference.

It also constrains model choice, since the explanation must be true rather than plausible. That is an argument for architectures where attribution is genuine, and a reason to be cautious about post-hoc explanations that sound convincing without reflecting what the model did.