Enterprise NLP: Which Problems Are Actually Solved

Language models have made a wide range of text tasks look equally achievable, which is misleading. Some are genuinely solved and safe to build products on. Others work impressively in a demo and produce failures that are hard to detect, which is a different and more dangerous category.

The distinction that matters is not difficulty. It is whether you can tell, at scale and automatically, that the output is wrong.

Reliably solved: classification and routing

Assigning a document, ticket or message to a category is the most dependable thing in this space. It has been workable for years and is now very good, including with limited training data.

The reason it is safe is measurement. Classification has a ground truth, so you can hold out a labelled set, compute accuracy per class, and know what you have. When it degrades — because incoming language shifted — your metrics move and you find out.

The mistake teams make is over-specifying the taxonomy. Forty categories with overlapping definitions produce a model that appears poor when the real problem is that human annotators do not agree either. If two people cannot reliably assign the same label, no model will.

Reliably solved: structured extraction

Pulling specific fields out of documents — invoice totals, contract dates, names, quantities — is dependable when the field has a definite answer and you can validate it. Invoice totals must match line items. Dates must parse and fall in a plausible range. Identifiers must match a known format.

Those validations are what make extraction safe to automate, because they catch a large share of errors without human review. Where a field has no validation and no downstream consistency check, extraction is far riskier, and the honest design is to route low-confidence cases to a person rather than to accept silent errors.

What makes this valuable is that the alternative is usually manual data entry, so even partial automation with human review of uncertain cases has a clear return.

Harder than it looks: summarisation

Summaries read well and are difficult to evaluate. A summary that omits the single most important clause looks exactly as fluent as one that captures it. There is no equivalent of a total that must reconcile.

Automatic metrics are weak proxies for whether a summary is useful, which means evaluation requires human review, which means you cannot cheaply detect degradation. That is a serious operational problem: a summarisation feature can get worse after a model update and nobody notices for weeks.

Where summarisation is genuinely fine is as a navigational aid — a preview a human uses to decide what to read fully, where the original is one click away. Where it is dangerous is as a replacement for reading, particularly if the output feeds a decision. If a summary is the only thing anyone reads, you have built a system whose errors are invisible.

Hardest: open-ended question answering

Answering arbitrary questions over a corpus is the most requested capability and the least tractable, because the failure mode is a confident, fluent, wrong answer that is indistinguishable from a correct one without checking the source.

Retrieval-augmented generation improves this considerably by grounding answers in retrieved passages and enabling citation. It does not eliminate it — the model can still misread a retrieved passage, or answer from parameters when retrieval finds nothing relevant.

The practical requirement is an evaluation set before a production launch: real questions, with correct answers, that you can run on every change. Teams that skip this cannot tell whether a prompt adjustment helped or hurt, which means they are tuning blind. Our note on what RAG costs in production covers the rest of that picture.

The data work is most of the project

Model selection consumes disproportionate discussion and rarely determines the outcome. What determines it is whether you have text in a usable state, and in most enterprises you do not — not because the text is missing, but because it is trapped.

Scanned documents need optical character recognition, and its errors propagate into everything downstream. Text in a legacy application may only be reachable through an export nobody has run in years. Email lives in mailboxes governed by retention rules and access constraints. Documents exist in several near-identical versions with no indication of which is current.

Budget for this honestly. A project where the text arrives clean and structured is a different size of project from one that starts with a folder of scans, and mistaking the second for the first is the most common estimating error in this area.

Domain language defeats general models more often than expected

General-purpose models handle general language well and specialised vocabulary less predictably. Clinical abbreviations, legal terms of art, internal product codenames and industry shorthand are all places where a model produces something plausible and wrong.

The fix is rarely a bigger model. It is usually retrieval — giving the model the definitions and context it needs at inference time — or a modest amount of domain-specific fine-tuning on examples that are cheap to produce because your experts already generate them as a by-product of their work. Both approaches beat hoping a general model has absorbed your jargon.

How to choose what to build

Ask what happens when the output is wrong, and how you would find out. If there is a validation, a reconciliation or a labelled test set, you are in solved territory and can automate with confidence. If the only detector is a human noticing, keep a human in the loop and design for their attention rather than assuming it.

Almost every disappointing enterprise NLP project we see began with a task from the second group, treated as though it belonged to the first.