Structured outputs

Your agent is code wrapping a probabilistic text generator. The moment your code has to branch on the model's answer, route it, store it, hand it to the next tool, you need output you can parse without praying. This lesson turns "please reply in JSON" into a hard guarantee, shows how that guarantee is physically enforced inside the decoder, and names the one place it backfires.

Why agents need machine-parseable output

An agent is a loop: the model decides, your harness acts, the result comes back, that is the whole shape of the agent loop. But your harness is ordinary code. To act on the model's output it has to read a field, intent, priority, tool_args, not a paragraph of prose. The model, meanwhile, only ever emits a stream of tokens. Bridging those two worlds is the entire problem of structured output. The same machinery underneath tool calling, from the previous lesson, is what emits well-formed arguments: tool args are just a structured output the runtime hands to a function.

The naive fix is to ask: put "respond only with JSON" in the prompt and wrap the reply in JSON.parse. It works until it doesn't, and in an agent that runs thousands of times a day, "until it doesn't" is a nightly page. The classic ways a polite request drifts:

The ladder of guarantees: JSON mode vs strict Structured Outputs

There is a ladder of increasing guarantee, and only the top rung is bulletproof. Each step buys you a stronger promise about what comes back:^[1]

1. "Please respond in JSON" (prompt only). No guarantee at all. The model may add prose, wrap the reply in a code fence, or silently drift to a different shape.
2. JSON mode (response_format: {type:"json_object"}). Guarantees the output parses as valid JSON and nothing more. It says nothing about your field names, your types, or which keys are required. Syntactically clean, semantically unconstrained.
3. Structured Outputs (response_format: {type:"json_schema", ..., strict:true}). Guarantees the output conforms to your exact JSON Schema, right fields, right types, required keys present. Enforced by constrained decoding, not by asking nicely.
4. Strict function calling. The same schema guarantee applied to tool arguments (strict:true on the tool definition), so a tool call can't be malformed. Anthropic's equivalent is strict tool use.^[6]

The gap between "try" and "enforce" is not academic. On complex-schema-following evals, OpenAI reported that gpt-4o-2024-08-06 with Structured Outputs scored 100% schema compliance, versus under 40% for an earlier model relying on prompting alone.^[2] That jump is not the model "trying harder," it is the decoder being physically unable to break the schema.

Constrained decoding: how the guarantee is enforced

Here is the "how" behind that 100%. Normal decoding: at every step the model produces a logit, a score, for every token in its vocabulary, then samples one according to temperature and top-p. Constrained decoding inserts one filter before sampling: it looks at the grammar, works out which tokens are still legal, and sets the logit of every illegal token to -inf. After a softmax, -inf becomes probability zero, so the sampler cannot pick it, no matter how much the model wanted to.^[3]

Zoom into one step. Say the schema's root is an object, and we are generating the very first token. The parser is in state "expecting the start of an object," so the only legal next tokens are { or whitespace. Everything else, a friendly "Sure,", a markdown fence, a bare string, an opening [, gets masked to -inf, even if the model's raw preference was to be chatty. The model's strongest raw preference might be the prose "Sure,", but the grammar zeroes it out. Only { and whitespace survive, and the highest-scoring survivor is sampled. Conformance is guaranteed, not hoped for.

Where does the grammar come from? Your JSON Schema is compiled into a context-free grammar (CFG), which is executed by a pushdown automaton, a state machine with a stack so it can track arbitrarily deep nesting (an object inside an array inside an object). At each step the automaton's state tells the masker exactly which tokens keep the output grammatical.^[3]

The compiler analogy makes it click: constrained decoding is a lexer/parser riding along with generation, rejecting any token that wouldn't parse, the same way a compiler refuses source that breaks the language grammar.^[4]

The reasoning tax: reason first, format last

Constrained decoding has a catch, and it is the most important thing an engineer takes from this lesson. If you force the model straight onto grammatical rails, it has no room to think in free text first, and thinking in free text is exactly how models solve hard problems.

The fix is a one-line design rule: reason first, format last. Give the model somewhere to think before it has to commit. Two reliable patterns:

1. Reasoning field first. Put a free-text reasoning (or analysis) field as the first property in the schema. Because generation is left-to-right, the model writes its chain of thought, still inside the JSON, before it reaches the fields your code depends on.
2. Two-pass generate-then-format. Call once for free-text reasoning, then a second, cheap call that only reformats the conclusion into the strict schema. More tokens, zero reasoning tax.

# answer field comes first -
# model commits before thinking
{
  "priority": "?",
  "reasoning": "..."
}

# reasoning field first -
# model thinks, then decides
{
  "reasoning": "...",
  "priority": "high"
}

In practice: schema-constrained extraction

Concretely, most SDKs let you hand over a type and get a typed object back, no try/except JSONDecodeError, no regex. With the OpenAI Python SDK you pass a Pydantic model; the SDK converts it into a strict json_schema, the API compiles that schema to a grammar, and constrained decoding guarantees the raw text is a valid instance of your class:^[1]

from pydantic import BaseModel
from openai import OpenAI
client = OpenAI()

class CalendarEvent(BaseModel):
    name: str
    date: str
    participants: list[str]

resp = client.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "system", "content": "Extract the event information."},
        {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
    ],
    response_format=CalendarEvent,   # SDK -> strict json_schema, compiled to a grammar
)

event = resp.choices[0].message.parsed
# -> CalendarEvent(name='science fair', date='Friday', participants=['Alice','Bob'])

if resp.choices[0].message.refusal:      # safety refusals are a FIELD, not silent prose
    handle_refusal(resp.choices[0].message.refusal)

Notice what is gone: no string cleanup, no defensive parsing, no "did it wrap this in a fence?" check. And a safety refusal now surfaces as a populated refusal field you can branch on, instead of masquerading as content.^[1]

When should you constrain, and when should you leave the model free?

Check yourself