Getting Reliable JSON Out of a Language Model
By Maplecode
Asking a model for JSON used to be a gamble. Constrained decoding and schema-enforced output modes largely solved that: the response parses, the keys are the ones you asked for, the types are right.
Which moves the problem rather than removing it. A document can satisfy your schema completely and still be wrong in ways that matter downstream.
Valid and wrong
Three failures pass schema validation cleanly and cause real incidents.
Invented enum members. You allow five status values; the model returns a sixth that is plausible and not in your list. Strict enum enforcement catches this — if you actually declared the enum rather than typing the field as a string, which is the common shortcut.
Confident nulls. Asked to extract a field that is genuinely absent from the source, a model will sometimes fill it with something reasonable and sometimes leave it empty. Both are schema-valid. Only one is correct, and you cannot tell which from the document alone.
Unit and format drift. A number field receives 1500 when the source said 1.5k, or a date arrives as a plausible reading of an ambiguous string. Type-correct, semantically wrong, and invisible until something sums it.
Design the schema for verifiability
The single change with the best return: make the model tell you where it got each value.
Add a field carrying the source span, quote or document offset that supports the extraction. It costs tokens and it converts an unverifiable claim into a checkable one — you can assert, in code, that the quoted span exists in the input. That assertion catches a large share of fabrication without any human review.
Two more habits worth adopting. Model absence explicitly rather than through null, with a value like not_stated, so "missing" is a decision the model made rather than a gap you interpret. And allow a confidence or needs_review flag, then actually route on it — a flag nobody acts on is overhead.
Schema changes are a compatibility problem
Once extracted documents are stored, your schema has the same versioning obligations as an API. Adding an optional field is safe; tightening an enum, renaming a key or changing a type breaks everything already persisted.
Version the schema and store the version with each record. Without it, a year of extractions become a set of documents in unknown shapes, and every consumer needs defensive code forever.
Also re-run your evaluation set after any prompt or model change, not just after schema changes. Output distribution shifts with the model, and a field that was populated 98% of the time can quietly drop to 70% with no error anywhere.
The failure mode: trusting the parse
The dangerous consequence of reliable syntax is that teams stop validating semantics. The response parsed, so the pipeline proceeds, and a wrong value travels into a report or a decision with no checkpoint.
Add cheap semantic checks at the boundary: do the numbers fall in a possible range, do the dates sit inside the document's period, does the total match the sum of the parts, does the quoted span exist. These are ordinary assertions and they catch the errors schema validation cannot see.
When we would advise against structured extraction
If the field feeds an irreversible action — a payment amount, a clinical value, a contractual date — do not consume it unreviewed, however good the evaluation numbers look. The distribution of errors matters more than the rate, and these are the cases where one error is not averaged away.
If the source documents are low-quality scans, extraction accuracy is bounded by OCR and the model will produce confident output from mangled text. Fix the input path first.
And if a deterministic parser can do the job — fixed-format files, consistent templates, structured feeds — use it. It is faster, free, and it fails loudly.
Retries need to be idempotent
Extraction pipelines retry, because models occasionally fail a validation or time out. The retry is usually bolted on and rarely thought through.
Two things go wrong. Retrying a non-deterministic call produces a different document, so a record can change between attempts and whichever attempt happened to succeed becomes the truth. And retrying after a partial write leaves two records for one source document, which is how duplicate rows enter a warehouse.
Key the write on the source document and the schema version rather than on the attempt, so a retry overwrites rather than appends. Persist the raw model response alongside the parsed record, so a later disagreement can be investigated without re-running anything. And cap retries: a document that fails three times is telling you it needs a human, not a fourth attempt.
What we build
Strict schemas with real enums. A provenance field on every extracted value, asserted in code. Explicit absence rather than null. A versioned schema stored with each record. Semantic assertions at the boundary. And an evaluation set that includes documents where the answer is genuinely not present, because that is the case that separates a careful extractor from a confident one.
One more habit: keep a small set of documents where the correct answer is that a field is absent, and check that your pipeline reports absence rather than inventing a plausible value. Extraction suites are almost always built from documents that contain the answer, which measures only the easy half of the job and reports it as the whole score.
More in AI engineering and enterprise NLP.
