Integrating an AI API Without Building a Liability

Calling a model API is three lines of code, which is why so many AI features go from prototype to production without anything between the application and the vendor. That gap is where the operational problems live.

A model API differs from an ordinary dependency in ways that matter: it is slow, priced per use, occasionally unavailable, non-deterministic, and it will follow instructions that arrive in its input. Each of those needs handling somewhere.

Cost scales with usage, and usage is hard to predict

Unlike infrastructure you provision, model APIs bill per token. A feature that costs very little in testing can become a significant line item when adopted, and the relationship is not obvious in advance because it depends on how verbose users are and how much context you attach.

Track cost per request from the first day, attributed to the feature and ideally the customer. Without that, a cost problem presents as a single large invoice with no way to identify what caused it.

The lever with the largest effect is usually context size rather than model choice. Sending an entire document when a relevant section would do multiplies cost across every call. Caching helps too, and several providers now discount repeated prefixes, which rewards structuring prompts with the stable part first.

Latency changes the interface, not just the timeout

Model calls take seconds, sometimes tens of seconds. That is long enough to change what the interface can be. A synchronous request-response that blocks a page will feel broken regardless of how good the answer is.

Streaming makes the wait tolerable because output appears progressively. For anything longer, the honest design is asynchronous: accept the request, return immediately, notify when complete. That is a bigger change than it sounds, because it means the surrounding system needs job state, and retrieval of results, and a way to show progress.

Timeouts need setting explicitly and generously, and paired with a fallback. A model call that hangs without a timeout will exhaust your connection pool and take down endpoints that have nothing to do with the feature.

Prompt injection is an unsolved problem

If your prompt includes content from an untrusted source — a user's message, a retrieved document, a web page, an email — that content can contain instructions the model may follow. There is no reliable way to prevent this, because the model has no mechanism to distinguish instructions from data. They are the same tokens.

Mitigations reduce the risk without eliminating it: separating system and user roles, restating constraints after the untrusted content, and filtering. None are sufficient on their own.

The dependable control is on the output side. Never let model output take a consequential action directly. If it can call tools, restrict those tools to what would be acceptable for the least trusted user whose content might reach the prompt. If it produces a database query, validate the query rather than executing what arrives. Treat model output the way you treat any input from outside your trust boundary — because that is what it is.

Non-determinism breaks ordinary testing

The same input can produce different output. Assertions on exact strings fail intermittently, which teams usually respond to by deleting the test, leaving the feature untested.

What works is testing properties rather than exact outputs — the response is valid JSON, contains a required field, does not contain a forbidden term, falls within a length range. Alongside that, a small evaluation set of representative inputs with known-good outputs, scored on each change. It is more work than a unit test and it is the only way to know whether a prompt change helped.

An abstraction layer earns its cost

Provider APIs are similar enough that a thin internal interface is cheap to write, and it buys several things at once: the ability to switch or fall back to another provider during an outage, a single place for retries and timeouts, a single place where cost is measured, and a single place to log inputs and outputs for debugging.

Keep it thin. A layer that tries to abstract every provider-specific capability becomes its own maintenance burden and usually blocks the feature you eventually want. The goal is to avoid provider calls scattered through your codebase, not to build a portability framework.

Retries need to be smarter than usual

Model APIs return rate limit errors routinely under load, and the naive retry — immediately, in a loop — makes the situation worse for everyone including you. Exponential backoff with jitter is the minimum.

The subtlety is that retrying a partially streamed response is not free: you may be billed for the tokens already generated. For expensive calls it is often better to fail and surface that to the user than to silently retry three times at full cost.

Logging is harder than it looks

Debugging model-driven features requires seeing the exact input and output, which means logging prompts and completions. Those frequently contain whatever the user typed, which frequently contains personal data.

The tension is real: you need the data to diagnose problems, and retaining it creates an obligation. The workable position is short retention for full payloads with restricted access, longer retention for metadata — token counts, latency, model version, error codes — which answers most operational questions without holding content.

Worth checking what your provider retains as well. Several offer a zero-retention mode for enterprise plans, and whether you are on it is a question your privacy assessment will ask.

Model versions change under you

Providers update models, sometimes behind the same identifier. A prompt tuned against one version can behave differently after an update, and the failure is quiet — output that is still fluent and no longer does what you needed.

Pin to specific versions where the provider allows it, and treat a version change as a code change: run your evaluation set, review the diff in behaviour, then adopt. Teams that use floating identifiers discover changes through user reports, which is a slow and expensive detector.

Where to draw the boundary

Put a layer between your application and the vendor that owns timeouts, retries, cost measurement, logging and the fallback path. Keep untrusted content away from anything consequential the model can trigger. Test properties, not strings, and keep an evaluation set. Measure cost per feature before you need to explain it.

None of this is difficult. It is simply the difference between a feature that works and one that works until it is used.