What Retrieval-Augmented Generation Actually Costs in Production

Retrieval-augmented generation demos beautifully. Point a script at a folder of PDFs, embed them, wire up a vector search and a prompt, and within an afternoon you have something that answers questions about your own documents. It is genuinely impressive, and it is why so many organisations have one sitting in a repository somewhere.

The distance between that and a system serving a thousand employees is larger than it looks, and most of the distance is not model quality. It is cost structure, latency budget and the absence of any way to tell whether the answers are good. This is a tour of where those problems actually live, based on the parts that reliably surprise teams.

The embedding bill is the smallest of your problems

Teams tend to worry first about the cost of embedding their corpus. It is the most visible number and the easiest to calculate, so it gets the attention. In practice it is usually a rounding error.

A million documents averaging two thousand tokens each is roughly two billion tokens. At current commercial embedding prices that is a one-off cost in the low hundreds of dollars. Even re-embedding the entire corpus after a model change is an afternoon's compute and a bill you would not escalate.

What does cost money is the generation side, and it scales with usage rather than corpus size. Every query sends retrieved context to the model. If you retrieve ten chunks of five hundred tokens and the model produces a four-hundred-token answer, that is around five and a half thousand tokens per query. Ten thousand queries a day puts you in a materially different bracket, and unlike the embedding cost it recurs every day and grows with adoption.

The practical implication is that the lever worth pulling is context size, not corpus size. Retrieving four well-chosen chunks instead of twenty is a straightforward reduction in running cost, and it frequently improves answers, because models are measurably worse at using information buried in the middle of a long context than at using a short, relevant one.

Chunking is where quality is won and lost

Almost every tutorial splits documents into fixed-size windows with a little overlap. It is easy to implement and it is close to the worst thing you can do to a structured document.

Consider a policy document where section 4.2 states a rule and section 4.3 lists the exceptions. A fixed-size split will cheerfully cut between them. Retrieval returns the rule, the model answers confidently, and the answer is wrong in exactly the way that erodes trust fastest — plausible, well-written and missing the qualification that mattered.

Structure-aware chunking costs more to build and pays back quickly. Split on headings. Keep tables intact rather than slicing them mid-row. Carry the section path into each chunk so that a fragment reading "this does not apply to contractors" retains the knowledge of what this refers to. Where a document has a natural unit — a clause, a procedure, a ticket — prefer that unit over an arbitrary token count.

Expect this to be an iterative process rather than a decision. You will only discover that your chunking mangles a particular document type by seeing it fail, which is an argument for building the evaluation harness before tuning anything.

Vector search alone will disappoint you

Embedding similarity is good at meaning and poor at specificity. Ask about error code E4021 and semantic search will return chunks about error codes generally, ranked by conceptual closeness, potentially without the one containing the exact string you asked about.

Keyword search has the opposite profile: precise on rare terms, useless when the user's phrasing differs from the document's. Product codes, names, acronyms and identifiers are exactly the queries where users expect precision and semantic search underperforms.

Running both and combining the rankings — hybrid retrieval — is close to a default for enterprise corpora. It is more infrastructure and it consistently outperforms either method alone on the mixed query distribution real users produce.

Above that sits re-ranking: retrieve a wider candidate set cheaply, then score each candidate against the query with a more expensive cross-encoder and keep the best few. This is usually the single largest quality improvement available, and it comes with a latency cost of roughly one to three hundred milliseconds. Whether that is affordable depends on your interface. In a chat window where the answer streams, it is invisible. In an autocomplete, it is not.

Latency is a budget, and it is smaller than you think

A typical request decomposes into embedding the query, searching, re-ranking, generating, and whatever guardrails you have added. Embedding and search are usually fast — tens of milliseconds. Re-ranking is a few hundred. Generation dominates, and it scales with output length.

Streaming changes the psychology entirely. Users tolerate a slow complete answer far less well than a fast first token followed by steady output. If your interface can stream, the number to optimise is time-to-first-token, and a re-ranking step that improves quality is easily worth its cost. If your system must return a complete structured response — because something downstream parses it — you lose that concession and every millisecond in the chain is felt.

Worth deciding early, because it constrains architecture: an agent that makes three sequential model calls cannot stream a first token until the first call resolves, and the perceived latency is the sum of the chain.

The problem nobody budgets for: knowing whether it works

Here is the pattern we see most often. A team builds a retrieval system, tries perhaps thirty questions by hand, finds the answers good, and ships. Two months later someone asks whether last week's prompt change made things better or worse, and there is no way to answer. The team is now modifying a system whose quality is unmeasured, which means every change is a gamble.

The fix is unglamorous and needs to exist before you start tuning. Assemble a set of questions drawn from real usage — a hundred is enough to be useful, three hundred is comfortable — with an agreed expected answer or, at minimum, the documents that should have been retrieved. Run it automatically. Track the results over time.

Separate two things while you are at it, because they fail independently and have different fixes. Retrieval quality asks whether the right documents came back at all, and is measurable without any model involvement. Generation quality asks whether the model used them correctly. A system that retrieves badly cannot be fixed by prompt engineering, and a great deal of effort is wasted on prompts when the retrieval layer was the problem.

Grading generation quality at scale usually means using a model as a judge, which is imperfect and still far better than nothing. Calibrate it against a few dozen human-graded examples so you know roughly how much to trust it, then use it to track direction rather than to certify absolute quality.

Permissions cannot be an afterthought

In any organisation of size, not everyone may see everything. Salary data, disciplinary records, unannounced financials, customer information restricted by contract — all of it tends to live in the same document stores as the material you want to make searchable.

Filtering results after retrieval is the intuitive approach and it is a poor one. It leaks information through absence and behaviour: a user notices that certain queries return fewer results, or that the model's phrasing implies context it did not show. It also wastes retrieval quality, since your top results may all be discarded.

Permissions belong in the query. Store the access control identifiers alongside each chunk and filter within the search itself, so restricted material is never a candidate. This requires that your ingestion pipeline captures permissions accurately and keeps them current as they change in the source system — which is real work, and considerably cheaper than the incident that follows getting it wrong.

What we would advise

Start narrower than feels satisfying. One document set, one user group, one category of question. A system that answers HR policy questions well is more valuable than one that answers everything unreliably, and it teaches you what the second one needs.

Build the evaluation set before tuning. It feels like a detour and it is the thing that makes every later decision cheap instead of speculative.

Instrument from the first day. Log queries, retrieved documents, generated answers and any user feedback signal you can capture. The gap between what you imagined users would ask and what they actually ask is always instructive and never small.

And be prepared to conclude that retrieval is not the answer to a given problem. If users need a specific number from a specific system, a query interface is better than a language model. If they need a document, search that returns the document is better than a summary of it. Retrieval-augmented generation is a good technique for questions whose answers are spread across prose in several places — which is a real and common category, and not a universal one.