Structured outputs and forced tool use aren't prompting tricks — they mask invalid tokens at the decoding layer so malformed JSON becomes mechanically impossible, not just less likely.
Ask a model to "respond with valid JSON" and most of the time you get valid JSON. Not all of the time. At the scale an agent loop operates — dozens or hundreds of tool calls per session, run across a fleet of concurrent agents — "most of the time" means a steady trickle of malformed arguments, truncated strings, trailing commas, and enum values the model invented because it forgot what was actually allowed. Somewhere along the way, the major providers stopped treating this as a prompting problem and started treating it as a decoding problem. That shift — from asking the model nicely to making invalid output mechanically impossible — is what "structured outputs" actually means underneath the marketing term, and it's worth understanding the mechanism, because it explains both what these systems guarantee and, just as importantly, what they don't.
An LLM generates one token at a time, sampling from a probability distribution over its entire vocabulary at every step. Nothing about that process inherently respects brackets, quotes, or key names — the model has simply learned, from training data, that JSON-shaped text tends to look a certain way, and it reproduces that pattern with high but not perfect fidelity. Prompting strategies (few-shot examples, "respond only with JSON, no other text," explicit schema descriptions in the system prompt) push the probability mass toward valid output, but they're still just biasing a free-form sampling process. Nothing stops the model from sampling a token that breaks the schema — it's just less likely to.
For a single, human-reviewed API call, "less likely to fail" is often fine. For an agentic system making tool calls autonomously — where a malformed arguments object either crashes the tool invocation or, worse, silently coerces into something the tool executes incorrectly — the failure rate that felt negligible in a demo becomes a real reliability tax at production volume. This is the problem constrained decoding was built to eliminate, not reduce.
It's worth separating two capabilities that are easy to conflate because vendors have historically bundled them under similar names:
Early "JSON mode" implementations (and some still in use today) constrain the model to emit syntactically valid JSON — balanced brackets, properly quoted keys and strings, valid escaping — without knowing anything about which keys, types, or values you actually want. This eliminates an entire class of parse errors, but a schema-blind JSON mode will happily emit {"result": "42", "stauts": "ok"} — valid JSON, wrong key name, wrong type. You still need to validate against your actual schema after the fact and handle rejection.
The more recent and more useful capability constrains generation to a specific JSON Schema (or a specific tool's parameter schema, in tool-calling contexts). Every key, every type, every enum value, every required field is enforced at generation time. The model cannot emit a key that isn't in the schema, cannot emit a string where the schema says integer, and cannot emit an enum value outside the allowed set — not because it was told not to, but because those tokens are never sampled in the first place.
That second capability is the one worth understanding mechanically, because "enforced at generation time" is doing a lot of work in that sentence.
The core idea: compile the JSON Schema into a formal grammar — practically, a finite-state automaton or pushdown automaton that tracks "given everything generated so far, what set of next characters would keep this a valid instance of the schema." Then, at every single decode step, before sampling the next token, mask out every token in the vocabulary that would violate the automaton's current state. The model can only sample from what's left.
Concretely, the loop looks like this:
def constrained_generate(model, schema, prompt):
automaton = compile_schema_to_automaton(schema) # once, up front
state = automaton.start_state
tokens = []
while not automaton.is_accepting(state):
logits = model.forward(prompt + tokens) # one step of the model
valid_token_ids = automaton.valid_next_tokens(state) # precomputed per state
masked_logits = mask_all_except(logits, valid_token_ids)
next_token = sample(masked_logits) # sample from what's left
state = automaton.advance(state, next_token) # move the automaton forward
tokens.append(next_token)
return detokenize(tokens)
Two things make this nontrivial in practice. First, compile_schema_to_automaton has to handle the full expressiveness of JSON Schema — nested objects, arrays with typed items, enums, string patterns, numeric ranges — and produce a structure where "what's a valid next character" can be computed cheaply at every state. Second, and this is the part that trips up naive implementations: the automaton reasons about characters, but the model samples tokens, and those are not the same alphabet.
An LLM's vocabulary is built from byte-pair encoding or a similar subword scheme, typically 50,000 to 200,000 entries, where a single token might be a whole word, a punctuation cluster, or an arbitrary chunk of a number like "12 or 34,. A grammar defined over individual JSON characters doesn't map cleanly onto that vocabulary — you can't just check "does this token match the next expected character" because a single token might satisfy three grammar transitions at once, or straddle a boundary the grammar didn't anticipate (a token that's 234" combines a digit run, a closing quote, and needs the automaton to jump two states in one hop).
The practical fix is to precompute, for every automaton state, the full set of vocabulary token IDs that are valid continuations from that state — effectively a lookup table from "grammar position" to "allowed tokens," built once per schema and reused across the whole generation. This is expensive to build naively (state count times vocabulary size) and is where most of the engineering effort in libraries like Outlines, Guidance, and Microsoft's guidance-adjacent work, and NVIDIA/MLC's XGrammar has actually gone: representing the automaton compactly enough, and indexing the vocabulary cleverly enough (tries, byte-level automata), that this precomputation is fast and the per-step mask lookup is close to free relative to the forward pass itself.
The mechanism above is roughly what sits behind the structured-output features you'd actually reach for:
response_format: {type: "json_schema", strict: true}) constrains generation to a supplied JSON Schema at the token level, not just as a prompting hint — the strict mode explicitly restricts the schema subset supported (no arbitrary additionalProperties, for instance) precisely because the grammar compiler needs a tractable schema to compile.tool_choice, constrains the model to produce arguments matching that tool's input_schema — the same underlying idea applied to function-calling rather than a bare JSON response.Across all of these, the constraint is enforced during sampling, not validated after the fact — which is the entire point. A post-hoc json.loads() retry loop catches failures after they've already cost you a full generation; token-level masking prevents them from being generated in the first place.
It's easy to over-credit this technique, so it's worth being precise about the boundary. Constrained decoding guarantees syntactic and structural conformance — the output will parse, and it will match the schema's shape, types, and enums. It does not guarantee:
get_weather(city: "Paris", country: "Germany") is perfectly valid JSON matching a perfectly reasonable schema. It's also wrong. The grammar has no concept of whether Paris is in Germany — only that both fields are non-empty strings, which is all the schema asked for.type is "credit_card", then card_number is required and iban must be absent" is a oneOf/conditional pattern that some schema subsets simply drop under the "strict" restrictions vendors impose to keep the grammar compilable. If your strict mode doesn't support it, the constraint isn't enforced — not because decoding failed, but because the schema you handed it couldn't express the rule in the first place.The honest framing: constrained decoding moves the failure mode from "malformed output" to "confidently well-formed but semantically wrong output." That's a real improvement — malformed output breaks your parser deterministically and immediately; wrong-but-valid output at least reaches your business logic in a shape you can validate further — but it is not a substitute for validating values, only for validating shape.
| Approach | Guarantees | Cost | Fits |
|---|---|---|---|
| Free-form prompt + parse-and-retry | Nothing, until it parses | Cheapest per call, expensive in retries at scale | Low-volume, human-reviewed, exploratory use |
| Schema-blind JSON mode | Valid JSON syntax only | Low | Cases where you validate keys/types yourself downstream anyway |
| Schema-aware structured outputs / forced tool use | Valid JSON matching your exact schema, types, enums | Moderate (compile once, mask every step) | Any autonomous tool-calling loop where a malformed call has a real cost |
| Hand-written grammar (GBNF, custom CFG) | Arbitrary format, not just JSON | Higher engineering cost to write the grammar | Local models, non-JSON output formats, domain-specific DSLs |
For an agent loop specifically, schema-aware structured outputs are close to a default-yes: every tool call an agent makes — through a provider's native tool-calling API, or through an MCP server exposing tools with declared input schemas, the kind Utilix's own MCP server and REST API use — is exactly the case this mechanism was built for. The argument object either matches the tool's schema or it doesn't get generated; you're not writing defensive parsing code for a failure mode that no longer needs to occur at the syntax level.
Takeaway: if you're still handling malformed tool-call JSON with a retry loop, check whether your provider or serving stack already exposes schema-constrained generation — it turns a probabilistic failure mode into a solved one at the decoding layer, for roughly the cost of compiling your schema once. What it won't do is check whether the values inside that valid JSON are actually correct — that part is still on you.