Workflows & Automation

Validation, retries, and constrained decoding help produce reliable structured outputs from LLMs.

Make LLM structured outputs reliable with strict schemas, source-text validation, shape-only retries, and constrained decoding.

Validation, retries, and constrained decoding help produce reliable structured outputs from LLMs.

Key Takeaways

  • Structured LLM outputs are not reliable merely because they match a schema; a valid JSON or Pydantic object can still contain wrong facts.
  • The application should define a strict contract with required fields, nullable semantics, strict types, and forbidden extra keys to catch malformed responses early.
  • The parsed output must be validated against the source text, such as checking that totals, currency, IDs, and quoted lines are supported by evidence.
  • Retries should be used only for shape errors that the model can correct, while factual mismatches or missing information should lead to rejection or human review.

![Featured image: abstract friendly illustration of an invoice, a JSON schema, and validation gates](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20800%20450'%3E%3Crect%20width='800'%20height='450'%20fill='%23f7f7ff'/%3E%3Crect%20x='60'%20y='70'%20width='220'%20height='300'%20rx='28'%20fill='%23dbeafe'%20stroke='%233b82f6'%20stroke-width='8'/%3E%3Crect%20x='100'%20y='120'%20width='140'%20height='16'%20rx='8'%20fill='%231e40af'/%3E%3Crect%20x='340'%20y='110'%20width='160'%20height='220'%20rx='28'%20fill='%23dcfce7'%20stroke='%2322c55e'%20stroke-width='8'/%3E%3Crect%20x='560'%20y='130'%20width='180'%20height='180'%20rx='28'%20fill='%23fef3c7'%20stroke='%23f59e0b'%20stroke-width='8'/%3E%3Cpath%20d='M600%20190l35%2035%2055-70'%20stroke='%23d97706'%20stroke-width='12'%20stroke-linecap='round'/%3E%3C/svg%3E)

Validation, retries, and constrained decoding help produce reliable structured outputs from LLMs.

Meta description: This guide shows how to make LLM structured outputs reliable by validating against source text, retrying only shape errors, and constraining generation.

Tags: structured outputs, Pydantic, OpenAI Responses API, validation, constrained decoding

This article is for developers who need structured outputs from large language models to be usable in an application, not just parseable as JSON. It explains a three-layer method used on Python 3.10.12 and Pydantic 2.14.1: define a strict contract, validate the parsed object against the source invoice, and allow retries only when a shape error could be fixed. The examples extract invoice fields only and never write to a real system.

Structured output is now a standard feature. OpenAI's Responses API can take a JSON schema or a Pydantic model and promise a response that fits it; vLLM does the same on a local server with xgrammar; Python SDKs return a parsed object rather than a string. The trap is that a valid object can still be wrong. If a model is asked for the invoice id, the currency, the total in cents, and the line where the total appears, it can return JSON with every key present, the right types, EUR in an allowed set, and a quoted line that really exists. The total can still be 12,500 cents, or 125 euros, for an invoice that says 1,250. No parser and no schema catches that, because 12500 is a perfectly good integer. The source comparison is the only catch.

Make the contract strict before the model sees the invoice

The contract has four fields and is owned by the application, not the prompt. Every key is required, so a response that leaves one out fails. Three fields are nullable without a default, which is different from optional: the model may return null to say the invoice does not state a value, but it may not stay silent or invent a number to avoid null. The allowed currency set includes two currencies as tutorial scope, not a claim about all invoices. Cents are integers so that 1,250.00 does not become 1249.9999.

Strict mode disables Pydantic's default coercion. Without it, the string "125000" could be accepted for an integer field and silently converted; with strict mode it fails with int_type. The boolean True also fails rather than becoming 1. extra="forbid" turns an unexpected key into extra_forbidden instead of ignoring it. Four bad inputs produce four distinct error types.

Validation, retries, and constrained decoding help produce reliable structured outputs from LLMs.

The schema export has limits. Invoice.model_json_schema() exports four properties, all required, additionalProperties false, and says nothing about the invoice. The rule that the cents must match the quoted line cannot be written in JSON Schema, so a decoder that enforces the schema perfectly can still produce the 12,500-cent total. A valid object is the entry ticket to source checking, not the final result.

Check the object against the source, then retry only shape errors

accept() runs four checks and ends in accepted, needs_review, rejected, or shape_error. Step one uses the parser and contract; it is the only step that can return shape_error. That matters because shape_error is the only outcome allowed to trigger a retry. Step two checks that total_quote appears in the invoice character for character. Step three checks the requested id and that the selected currency is actually printed in the text. Step four catches the opening error: the digits in "Total due: EUR 1,250.00", with separators stripped, are 125000, but the model sent 12500, so the object is rejected with the reason "total_cents 12500 does not match the quoted line". For a real invoice format, the decimal check would be tailored to that format, but the principle stays the same: a number must be derivable from the evidence.

needs_review is used when the invoice genuinely did not state an amount or currency. Under this policy, a missing fact goes to a person, not to a guessed zero. rejected is terminal because another model call cannot change what the invoice says. Test fixture expected values are written by hand from the source text, never copied from model output. For scanned invoices, inspect the transcription before asking the model to populate the schema, because a wrong digit upstream can pass every downstream check if the object is internally consistent.

extract_with_retries() has three exits, and only one is a retry. A shape_error gets another attempt with errors fed back. accepted, needs_review, and rejected are terminal. A refusal or configuration error stops immediately because repeating the request will not help. The generate function receives the invoice, schema, and feedback list, and returns text. The schema and source never change between attempts. Feedback is capped at five reasons, enough to tell the model that vendor is not a field or that total_cents must be an integer, but not enough to turn the prompt into a transcript. Field-level validation errors identify the failed check; each error carries loc, type, and msg, and the message is often all the model needs. Every candidate is validated from scratch because a repair call can change the amount while fixing the key.

The budget of three is only meaningful if transport retries are controlled. The OpenAI Python SDK retries connection errors, timeouts, 409, 429, and 5xx responses twice by default with exponential backoff, so three application attempts can become nine HTTP requests without the application code noticing. Set max_retries=0 for a demonstrable budget, add a per-call timeout because the default ten minutes is too long for extraction, and keep an overall deadline in the loop. The SDK retry configuration is documented per client and per request. Three attempts is this tutorial's policy, not a universal optimum; the point is to count hidden work and stop on terminal cases.

Validation, retries, and constrained decoding help produce reliable structured outputs from LLMs.
OutcomeNext step
shape_errorretry with feedback, at most three attempts
acceptedpass downstream
needs_reviewhuman review with candidate and reasons
rejectederror path; no retry
refusal, incomplete response, or configuration errorstop at completion check
wrong amountstop at source checks

Constrain decoding without dropping validation and test every failure path

Asking for JSON in the prompt and constraining generation are different. A prompt can still be answered with a code fence, a stray sentence, or an extra key. A schema constraint runs inside decoding: at each step, the generator may only choose tokens that keep the output valid. Adrian Tam's decoding-strategy walkthrough summarizes it: "The idea is to mask out tokens that would make the output invalid". This example uses a supported schema constraint rather than implementing the masking. On that path, malformed, extra-key, and wrong-type cases stop occurring because the decoder cannot produce them.

OpenAI's structured-output guide, as of October 2026, documents client.responses.parse(...) with text_format=Invoice for the Python SDK. It accepts the Pydantic model directly and returns output_parsed as an instance of it. The guide's examples name gpt-6-astra; pin whichever model is run and record it. This adapter was written from documentation and exercised with a mocked client in fixtures, not against a paid API, so it should not be read as a live result.

Two checks remain outside the decoder. The status check runs before anything reads the object: a response that is not completed is routed, not used. A completed response with no parsed output is a refusal, not an empty invoice. Then the object still goes through accept() unchanged, because the decoder enforces the shape and only the shape. It guarantees that total_cents is an integer, not that it is 125000 rather than 12500. Keep the contract plain, because not every Pydantic feature survives translation into a provider's schema subset.

The fixture suite turns these rules into fourteen test cases, from a missing required key and a code fence around the JSON to a fabricated quote, a refusal, and exhaustion with no fourth call. Four mocked-client cases prove routing: a wrong amount is rejected on the first attempt and never retried; exhaustion returns after exactly three calls; an incomplete response never reaches the source checks; and a refusal stops after one. They prove nothing about how often a real model produces the 12,500-cent total, and nothing about cost or latency. There are no percentages because a success rate from canned responses would describe the fixtures, not a model.

When the suite runs against a real adapter, record the model name, SDK version, outcome class, and attempt count per case. Mask the source text in logs. Report parsed, structurally valid, and correct-against-the-source as three separate columns, with needs_review in the third as not yet correct. The pipeline then treats accepted downstream, needs_review as human review with candidate and reasons attached, and rejected as an error path that records what happened and does nothing else. A retry never repeats a write, and a repair attempt never acquires permissions the first attempt did not have.

Common mistakes

The first mistake is stopping at JSON parse. A valid shape is only the entry ticket. The second is retrying anything that is not a shape error. Rejected, needs_review, refusals, and configuration errors are terminal. The third is ignoring hidden SDK retries, which can multiply the real request count. The fourth is trusting constrained decoding to make amounts correct; it makes structure possible, not source truth. The fifth is using model output to write expected values, or logging unmasked source text. The final deployment test is simple: does every failure path stop before the downstream action? If not, the pipeline is not ready for trusted use.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.