Technology Schemas · tools · grammars The layer production runs on

Structured outputs,
not just valid JSON.

Constrained decoding made structured output reliable, and that reliability is narrower than it sounds. The schema settles the shape of the response. Whether the values in it are true is still a question only your validation and your evaluation set can answer.

THE MECHANISM UNDER EVERYTHING

Almost no production AI system consumes free text. A support agent emits an intent, a confidence and a set of arguments; a document pipeline emits fields; an orchestration layer emits which tool to call and with what. The model's paragraph is for the human at the end, if there is one — the part the software acts on is a structured object.

That makes structured output the least discussed and most load-bearing part of the stack. It is also where a system quietly becomes fragile: a parser wrapped in a try block, a regular expression pulling JSON out of a markdown fence, a retry loop that fires a few times an hour and nobody looks at. All of that was necessary once. It is not any more, and removing it is usually the cheapest reliability win available on an existing system.

The reason is a change in mechanism rather than a change in model quality. Asking a model nicely for JSON is a prompt. Constraining its decoder so that no token producing invalid JSON can be sampled is an engineering guarantee about syntax, and it is the difference between hoping and knowing — about the shape, and only about the shape.

THREE THINGS THAT GET CONFLATED

JSON mode

An instruction to the model, sometimes with a flag, that the response should be JSON. It usually is. It may still be JSON of a different shape than you expected: a missing field, an extra one, a number where you wanted a string, the whole object nested inside a wrapper key. Better than asking in the prompt, and not a guarantee about your schema.

Constrained decoding

The schema is compiled into a grammar or state machine, and at each step the sampler is masked so only tokens that can continue a valid document are available. Invalid output is not corrected after the fact — it is unreachable. This is what “structured outputs” means when a provider offers it as a distinct feature, and it is what makes the parse step disappear from your code.

Tool calling

The same machinery pointed at a different purpose. You describe the functions available, each with a parameter schema; the model returns a name and a validated argument object rather than prose; your code executes it and feeds the result back. Tool calling is structured output with a dispatch step attached, which is why the two share every failure mode.

WHAT THE GRAMMAR ACTUALLY BUYS YOU

This is the distinction worth being pedantic about, because a great deal of production incident time is spent on the right-hand column by teams who believed the left-hand one had covered it.

	Constrained decoding	Still your job
Output parses as JSON	✓	—
Required fields are present	✓	—
Types match the schema	✓	—
Enum values are members of the enum	✓	—
The value is factually right	—	✓
The identifier refers to a record that exists	—	✓
The date is inside a plausible range	—	✓
The amount matches the source document	—	✓
The tool is the right tool for this request	—	✓
The caller is permitted to invoke it	—	✓

Constrained decoding is a syntactic guarantee, not a semantic one. A schema-valid object full of confidently invented values is the characteristic failure of a system that stopped at the left-hand column — and it is harder to spot than a parse error, because nothing throws.

There is a second-order effect worth knowing about: a grammar can only mask tokens, so a model pushed into a shape it does not want to produce will fill the shape anyway. If your schema requires a customer_id and the source text contains none, a constrained decoder cannot emit “there isn't one” unless you gave it a way to say so. It will emit a plausible-looking identifier instead. Most invented-value bugs we see are this bug, and the fix is in the schema rather than in the prompt.

DESIGNING A SCHEMA THE MODEL CAN FILL

Flat beats deeply nested

Every level of nesting is another place for the model to lose track of which object it is currently inside. Three levels is usually the point where extraction quality starts to slide. Prefer a flat object with compound field names, and where a hierarchy is genuinely necessary, consider two calls over one deep schema.

Enums beat free text

Any field with a bounded set of answers should be an enum, not a string. It removes a whole class of downstream normalisation — priority as “high”, “High”, “HIGH” and “urgent” — and the constraint is enforced by the decoder rather than by a mapping table you maintain. Where the set is large but known, the enum still beats validating a string afterwards.

Give it a way to say nothing

The most important field in most extraction schemas is the one that lets the model decline: a nullable value, a found boolean, an enum member called not_stated. Without it a required field is an instruction to invent. With it, “not present in this document” becomes a first-class answer you can measure the rate of.

Required means required

Mark a field optional only where its absence is meaningful. Long lists of optional fields produce sparse objects and hide whether the model failed to find something or simply skipped it. Nullable-and-required is almost always the better encoding than optional, because it forces an explicit answer either way.

Field names are prompt

invoice_total_including_vat extracts better than amount, and a description on the field beats a paragraph in the system prompt explaining what amount means. The schema is read by the model as part of the context; write it for that reader as well as for the type checker.

Ask for the evidence

A source_quote or page field alongside each extracted value costs a few tokens and changes what verification is possible: a reviewer can check the citation rather than re-reading the document, and an automated check can confirm the quoted span actually appears in the source. Grounding the output in something checkable is worth more than any confidence score the model reports about itself.

schemas/invoice_line.json JSON

{
                  "type": "object",
                  "additionalProperties": false,
                  "required": ["supplier_name", "invoice_total", "currency", "confidence", "source_quote"],
                  "properties": {
                    "supplier_name": {
                      "type": ["string", "null"],
                      "description": "Exactly as printed on the document. Null if no supplier is named."
                    },
                    "invoice_total": {
                      "type": ["number", "null"],
                      "description": "Total including tax. Null if the document states no total."
                    },
                    "currency": {
                      "type": ["string", "null"],
                      "enum": ["EUR", "GBP", "USD", "CHF", null],
                      "description": "Null when the document shows an amount with no currency mark."
                    },
                    "confidence": {
                      "type": "string",
                      "enum": ["clear", "ambiguous", "not_found"],
                      "description": "Route anything that is not clear to review."
                    },
                    "source_quote": {
                      "type": ["string", "null"],
                      "description": "Verbatim span the total was read from, for verification."
                    }
                  }
                }

The two design decisions that matter are both visible here: every uncertain field can be null, and the extracted total carries the span it came from so a check can verify it against the document rather than trusting it.

TOOL CALLING IN PRACTICE

A tool call is the point where a language model stops producing text and starts causing effects, which makes the validation step between the two the most consequential twenty lines in the system. The model proposes; your code decides.

THE TOOL CALL LOOP

Request

user or trigger
Tool schemas

only what this caller may use
Model

name + argument object
Validate

schema, then business rules
Rejected

error back to the model
Authorise

as the end user, not the service
Execute

and log the arguments

Lower box = a rejected call returned as a typed error Marker = one tool invocation

The authorisation step is deliberately after validation and before execution, and it authorises the human the request came from rather than the process making the call. A model that has been talked into asking for something it should not have is stopped here, not by the prompt.

Tool choice. Providers generally let you force a call, forbid one, or require a specific tool. Use it. A classification step that must return one of four tools should not be allowed to answer in prose, and a final summarisation turn should not be allowed to call anything. Leaving the decision open on every turn is how an agent ends up in a loop calling the same lookup nine times.

Parallel calls. Where a model can request several independent tools at once, take it — three lookups that could run concurrently are three round trips otherwise, and on a latency-sensitive path that is the whole budget. The constraint is that the calls must genuinely be independent; anything where the second call's arguments depend on the first result has to stay sequential, and a model will occasionally propose otherwise.

Errors are context, not exceptions. When a call fails, the useful move is usually to return the error to the model as a tool result in terms it can act on — “no customer with that reference; try searching by name” — rather than throwing. Models recover from a well-worded error surprisingly often. They recover from a stack trace never, and a raw exception message is also a place internal detail leaks into a context window.

Retries need a budget and a change. Retrying the identical call with the identical context usually produces the identical result. A retry should alter something — the temperature, the error hint, the sub-task — and there should be a hard cap, after which the system escalates rather than continuing. Unbounded self-correction loops are the most expensive failure mode in this whole area, because they fail slowly and bill the entire time.

Cardinality is a limit, not a suggestion. Recognition quality degrades as the tool list grows; a model choosing between forty tools makes worse choices than one choosing between six. Where a system genuinely needs many capabilities, route first — a cheap classifier selecting a small tool set — rather than presenting everything on every turn.

A SCHEMA IS AN INTERFACE, SO VERSION IT

Once anything downstream consumes the object, the schema is a published contract with the same compatibility rules as any API. Adding an optional field is safe. Adding a required one, tightening an enum, renaming a field or changing a type is a breaking change, and doing it in place breaks every consumer at once.

Two practices make this cheap. Keep the schema in version control as its own artefact rather than inline in a prompt string, so a diff is reviewable and a change is attributable. And treat a schema change as a change that must pass the evaluation set — enum edits in particular shift model behaviour in ways nobody predicts, because the enum members are part of the context the model reasons in. A schema change with no eval run behind it is a deploy with no test behind it.

The same argument applies to the tool definitions. A renamed parameter, a reworded description, a reordered enum: all of them alter what the model sees, and all of them belong on the same review path as the code that executes the call.

QUESTIONS

QUESTIONS — 5

It can, and the effect depends on how far the schema is from what the model would naturally produce. A schema that fights the model — forcing a rigid order, demanding a field the input does not support, denying any way to express uncertainty — costs accuracy even though every response is valid. The practical test is to run the same task with and without the constraint against your evaluation set. Where a gap appears, it is nearly always the schema that needs redesigning rather than the constraint that needs removing.

A numeric self-reported confidence is weakly informative at best and tends to cluster in a narrow band regardless of whether the answer is right. A small enum — clear, ambiguous, not found — is more useful, because the categories are actionable routing decisions rather than a number somebody has to threshold. Better still are external signals: whether the quoted span exists in the source, whether a lookup confirms the identifier, whether two independent extractions agree.

Yes. Grammar-constrained sampling is implemented in the common open-source inference servers, so schema enforcement is available without a hosted provider, and it is one of the areas where self-hosting gives you more control rather than less. Smaller models benefit from it disproportionately: much of their apparent unreliability at structured tasks is formatting failure rather than reasoning failure, and constraining the decoder removes that layer entirely.

The same way as anything else: a set of real inputs with the correct object for each, scored field by field rather than on whole-object equality. Field-level scoring is the part teams skip, and it is what tells you that the pipeline is fine except that dates are wrong 8% of the time. Include inputs where the correct answer is null or not_found, since a set made only of clean documents cannot detect the invention failure mode at all.

They operate at different levels. Structured output and tool calling are the model-side mechanism: how a model is made to emit a validated argument object. The Model Context Protocol is a transport and capability-description standard for connecting a model to tool servers over a common interface. You can use tool calling with no MCP at all, and an MCP server is ultimately exposing tools that a model calls this way.

Keep reading

Technology

Model Context Protocol

→

Technology

Evaluating AI Systems

→

Architecture

Custom LLM Orchestration

→

30-minute technical call · no deck