Constrained Decoding and Structured Output: Grammar, JSON Schema, and Sampling Repair

Preface: Parsability Is a Correctness Property, Not a Prayer

Over the past two years, almost every team that wired a large language model into a production system has hit the same class of incident at some late hour: the model returns prose that happens to look a lot like JSON where JSON was required; or it stuffs an object into the tool_calls arguments field with a trailing comma, unclosed brackets, or Chinese quotation marks; or something more insidious — the JSON is entirely valid, but the temperature value has leaked out into a string, an enum value's case does not match, some required field has quietly gone missing, and your validation function only blows up later in some downstream async queue.

The common thread in these incidents deserves to be repeated: this is not a problem of "the model is not smart enough," it is a problem of "the interface contract is not being enforced." Writing "please output JSON only, no explanatory text" in your prompt is a request, not a mechanism. Requests can be ignored; mechanisms cannot. Resting reliability on how obedient the model happens to be is, in essence, converting a deterministic engineering problem into a probabilistic model-behavior problem — and the latter you can neither test nor localize quickly when the pager goes off at three in the morning.

By 2026, this problem already has a fairly mature family of answers. The core idea behind all of them is a single one: do not wait until after sampling to check whether the output is legal; instead, at every sampling step, physically delete the illegal tokens from the candidate distribution. This is constrained decoding (constrained decoding / grammar-constrained decoding). It turns "generate a block of text" into "greedily sample under constraints that intersect the language model with some formal language," and from that point on, parsability stops being a probabilistic question and becomes an engineerable, deterministic one.

But constrained decoding is not a silver bullet. It has obvious costs of its own, and it does not cover every scenario. So this article describes a three-layer approach:

1. Per-token masking — enforce the grammar at decode time, masking out illegal tokens with an FSM or a grammar (GBNF / regex / JSON Schema). This is layer one, and the most reliable layer.
2. Post-sampling repair — parse-repair plus structural validation plus bounded retries, rescuing the output after it has been generated. This layer covers the semantic problems that grammar constraints cannot reach.
3. Degraded parsing — when the first two layers fail, squeeze whatever usable information is left out of a lenient fault-tolerant parser, and return the confidence alongside it honestly, letting the upstream decide whether to retry or to degrade.

The three layers are not a pick-one menu; they form a degradation chain running from cheap to expensive and from certain to uncertain: constraints first, validation as backstop, degraded parsing last. The order of this article follows that chain: first explain why relying on prompts is unreliable, then descend layer by layer, and finally spread all the costs out on the table — how much latency each layer costs, what fraction of failures each one can rescue, and who should implement each one in a concrete 2026 stack.

If you are working on "model output must be fed straight into a downstream consumer" — tool calls, structured extraction, SQL and code generation, configuration generation, the action space of an Agent — this article is for you. If you merely want "to make the JSON look a bit nicer," you probably do not need to read to the end.

---

I. The First Layer of Failure: Why "Asking the Prompt for JSON" Is Unreliable

Do not rush to adopt tooling; first we have to see clearly what happens without it. Skip this step and any technology-selection discussion is built on sand — because many teams, after stepping into the same pit once, conclude that "constrained decoding is too heavy, a few few-shot examples will do," while other teams conclude that "we must enforce constraints on the engine side." Both conclusions may be correct; it depends on your acceptable failure rate and the cost of failure.

1.1 Four Typical Failure Shapes

Classifying the failure samples actually observed in production, they fall roughly into these four categories. Note that they are mutually independent: every optimization you make on the prompt usually helps only one of them, which is exactly why "write the prompt a few more times" has rapidly diminishing returns.

Failure shape	Symptom	Can prompts mitigate it	Can constrained decoding cure it
Wrapper contamination	The output is prefixed with a sentence like "Sure, here is the JSON you asked for:", or trailed by an explanation; wrapped in a markdown code fence	Partly, via explicit instructions + few-shot	Yes, via start-symbol and stop-symbol constraints
Syntax breakage	Trailing commas, single quotes, Chinese quotation marks, unescaped newlines inside strings, mismatched brackets, truncation mid-generation by max_tokens	Almost not; this is probabilistic behavior of the tokenizer and sampling	Yes, masking directly blocks illegal character sequences
Schema drift	Valid JSON, but an enum value written as "Search" when the schema wants "search"; a number written as the string "18"; inconsistent date formats	Partly, by repeating the schema in the prompt	Only covers the parts explicitly compiled into the constraint
Content hallucination	Valid JSON with all fields present, but the values are invented: the tool name is not in the tool list, the argument count does not match	No	No, this is a model knowledge / reasoning problem

The fourth row draws the cognitive boundary that matters most in this entire article: constrained decoding only cures the syntax-related parts of the first three rows, and it is completely powerless on "whether the content is right." You can use constrained decoding to guarantee the tool name comes from the candidate set you supplied, but you cannot guarantee the model picks the correct one. That is a probabilistic quality problem, and it belongs to the realm of reranking, tool-description optimization, few-shot comparison, or simply switching models.

1.2 Why temperature=0 Does Not Save You Either

A very common misconception goes like this: "the model is unstable, so if I use greedy decoding (temperature=0, top_p=1) it becomes fully deterministic." This reasoning has two holes.

The first hole is that greedy removes only sampling randomness; it does not remove the low-probability paths inside the distribution. At a position where the model thinks "some wrong token has a 0.3% probability," greedy will pick the largest token every time — and if the largest happens to be the wrong one, greedy will produce the error deterministically and repeatably. This is why so many "intermittent parse failure" issues in production cannot be reproduced in offline evaluation, yet reproduce 100% of the time once they do. The value of constrained decoding here is precisely this: it deletes that 0.3% path at the level of the distribution, instead of relying on luck not to step on it.

The second hole is more fundamental: greedy decoding does not change "what the model tends to say," only "how to choose among the things it tends to say." When the context contains a large amount of signal that contradicts the target format — for example, the model is already disinclined to emit a bracket after the few-shot examples — greedy decoding will only run off course faster. Add two realities on top of that: batching makes the numerical kernels not perfectly deterministic (the logits for the same request can differ slightly under different batch compositions, and thus flip when the top-1 candidates are close), and any change to the model version or quantization tier can flip the outcome. Together these two mean that any "we are stable enough just with temperature" judgment simply does not hold up on the long timescale of a production environment.

1.3 What the Prompt Side Can Do, and Where Its Ceiling Is

To be fair, prompt engineering still has value for structured output; it just cannot carry the correctness guarantee on its own. Effective practices usually include: giving the complete schema in the system prompt rather than a list of field names; supplying two or three few-shot examples that cover the edge cases (especially enum values, numbers, and how null is written); explicitly stating "do not output a markdown code block"; wrapping the source text to be extracted in clear delimiters; and appending a self-check instruction right after generation — "if the previous answer was not valid JSON, first output your repair process." That last one turns out to be surprisingly useful in practice, because the model has strong local attention over its own output, and the self-check often repairs the broken JSON inside the same context.

But the ceiling is equally clear: all of these techniques act at the level of intent, while the failures occur at the level of characters. Intent is continuous, describable in natural language, and something the model genuinely learns; characters are discrete, combinatorially explosive, and something that requires an external mechanism to guarantee. This is why the consensus practice in 2026 is to separate intent from mechanism — the prompt makes the model willing to fill in content per the schema, constrained decoding makes sure the output definitely parses. The two are not competitors; they are complements.

A practical criterion: if you cannot write a reproducible automated test case for "output fails to parse," you have not really understood your own failure modes yet, and therefore have no way to judge which approach actually solved the problem. Constrained decoding is worth investing in largely because it converts failure modes from probabilistic and hard to reproduce into structurally impossible — and only the latter can be written into CI.

With that criterion in hand, let us descend one layer: since we are going to act at decode time, we first have to work out exactly what can be done at decode time.

II. Per-Token Masking: Turning "Legal" into Something Computable at Every Step

The idea behind constrained decoding is almost counterintuitively plain: since the model only ever emits one whole block of text in the end, let us pretend it is a machine that emits tokens one at a time, and at every step ask once: "is the next character legal in the current context?" As long as that check is fast enough and accurate enough, illegal output can never be assembled in the first place.

2.1 From Probability Distribution to Allowed Set

Concretely, ordinary sampling is: take the logits vector l, pass through softmax, sample according to temperature and top-p. Constrained sampling is: take l, first compute an allowed token set A, and push the logit of every position not in A down to -inf, then sample as usual. In pseudocode it means this:

import math

NEG_INF = -math.inf

def constrained_sample(logits, allowed_token_ids, temperature=0.0, top_p=1.0):
    # 1) 掩码：非法 token 直接物理性出局
    masked = [logits[t] if t in allowed_token_ids else NEG_INF
              for t in range(len(logits))]

    # 2) 正常采样流程，只在剩下的合法 token 里做
    if temperature <= 0:
        return max(range(len(masked)), key=lambda i: masked[i])

    scaled = [x / temperature if x != NEG_INF else NEG_INF
              for x in masked]
    probs = softmax(scaled)
    return top_p_sample(probs, top_p)

All the magic in that code lives in step 1. The remaining sampling logic is exactly what it always was — which means you can still sample, still have temperature, still have diversity; only the diversity is confined to "the legal outputs." Many people mistakenly believe on first contact that constrained decoding makes the output rigid. It does not: what it cuts away is impossibility, not possibility. What actually becomes rigid is the case of over-tight constraints, which is covered in section 2.5 and in section three.

The version above has an obvious performance problem: every decode step has to walk the entire vocabulary inside Python. A vocabulary is typically on the order of tens of thousands to over a hundred thousand tokens, so a per-step Python loop becomes the bottleneck. Real implementations are therefore considerably faster.

2.2 Acceleration: State Caching, Bitmasks, and Vectorization

Let us lay out all of the common optimization techniques, because this part is usually the dividing line between "can it go to production" and "cannot."

• State caching. The constraint state is discrete and finite. The same grammar state only needs to have its allowed set computed once; after that you simply do a table lookup. Modern implementations almost all "walk the token vocabulary once, score the string form of each token, and then represent the set with a bitset," so that intersection degenerates into a few integer bit operations.
• Push the mask into the kernel. A better approach is to construct a sparse logits_bias tensor (sequence length × vocabulary, values -inf) and feed it into the attention kernel as an additional input tensor, letting the GPU perform the addition in one shot with no CPU loop at all. Both SGLang and vLLM's guided decoding take this route. -inf has to be converted into a very large negative number (such as -1e9) to be safe, so that NaN does not appear in softmax.
• Determinism vs caching. Enabling caching significantly reduces CPU overhead, at the cost of the per-step CPU work no longer being constant, so throughput will jitter slightly. This does not matter for offline batch generation, but online services with an SLA need to watch for it during load testing.

Looking at these three together, one can draw a rather counterintuitive but very important conclusion: the main cost of constrained decoding is not algorithmic complexity, but how the mask is constructed and passed along. Under the same FSM constraint, an implementation that walks token by token in Python and one that feeds a sparse tensor straight into the kernel can differ in throughput by an order of magnitude. Choosing the engine and the implementation approach matters more than choosing the algorithm.

2.3 FSM: State Is a Stack

The most basic and most controllable implementation is a finite state machine. Its entire state is a stack of unclosed symbols, plus a flag indicating whether it is currently inside a string — fewer than ten variables in total. The transition function consumes input character by character: on an opening bracket it pushes the corresponding closing symbol, on a closing symbol it pops and checks whether they match, on a quote it flips the in-string flag and handles escaping. And the question "which tokens are currently allowed" is answered directly by this state: when the stack is non-empty, only the closing symbol that pairs with the top of the stack, a comma, or any opening symbol that can continue to fill in content is allowed; when the stack is empty, only the start symbol of a complete JSON value is allowed.

This state machine is a complete guarantee of syntactic correctness. One implementation detail is worth emphasizing: when deciding whether a token is legal, you cannot look only at the token's first character; you have to inspect its complete string form. The reason is that a single token in the vocabulary may contain several characters (for example, a token whose form is 0, rather than 0 plus ,), so looking only at the first character will both kill legal tokens and let genuinely illegal ones slip through. This is the single most common source of bugs in in-house implementations.

But note that this state machine knows nothing about the schema. It lets you write {"foo": {"bar": [1, 2, "baz", null]}} as long as the brackets match. To make it understand "this must be an array, and every element inside must be a string," you have to add a layer on top — which is exactly the schema compilation in chapter six. And there is an even more troublesome kind of understanding, called property dependency: if you write "type": "error", then there must be a "message" field, and "value" is not allowed. This is context-dependent syntax that "current state plus stack" simply cannot express; it requires path-dependent constraint tables.

2.4 Grammar Guidance and GBNF: The llama.cpp Route

An FSM can only express regular languages and finite-context languages, while a large amount of JSON Schema is context-free (objects and arrays nested at arbitrary depth). Forcing that onto an FSM is painful to write, so a general framework for grammar-guided decoding emerged: compile a context-free grammar into "terminals plus a prefix tree," and then compute the allowed set at every step from the grammar state.

The grammar description language chosen by llama.cpp is called GBNF (BNF as extended by llama.cpp). Its syntax is extremely plain, just a pile of production rules combined together:

# GBNF 示例：一个被约束死的工具调用

root   ::= "{" ws "\"tool\":" ws tool "," ws "\"arguments\":" ws args ws "}"

tool   ::= "\"search\"" | "\"calculator\"" | "\"weather\""

args   ::= "{" ws "\"query\":" ws string ws "}"

string ::= "\"" char* "\""
char   ::= [^"\\\x7F\x00-\x1F] | "\\" (["\\bfnrt] | "u" [0-9a-fA-F]{4})
digit  ::= [0-9]
number ::= "-"? digit+ ("." digit+)?
ws     ::= " "?

A few design points are worth noting, because they directly determine whether you can write a usable grammar:

• GBNF itself has no executable semantics, only productions. So it can express "a "tool": must appear here," but it cannot express conditions that require counting or numeric comparison, such as "the string here must not exceed 200 characters." Semantics of that kind can only be left to upper-layer validation, or forced into place through an indirection layer of "reference + rule substitution."
• Recursion depth. A grammar is allowed to recurse, but the grammar's state machine must be able to determine "how much nesting space is left," otherwise a legal deeply nested JSON will blow up the state space. In practice you either cap the maximum nesting depth or regenerate the mask every n steps.
• Correctness of state caching. The bug that implementations produce most easily is this: the mask must be computed for the "current state," not for the "next state." The consequences of getting this backwards are very insidious — the constraints look like they are working, but an illegal token occasionally slips through, and it is extremely hard to reproduce.

Outlines takes a more thorough route: it first compiles Python type annotations, regular expressions, and a batch of predefined structures into indexed finite state automata (FSMs), then does a table lookup on the DFA's edge table at every step. Its selling point is compile once, reuse from cache — the same schema reuses the same DFA across many requests, with almost zero runtime overhead. XGrammar follows the same idea and additionally patches something that early Outlines versions handled weakly: context-aware byte-level constraints, that is, correctly handling the case where one token contains multiple characters (for example, temperature may be split by the tokenizer into several tokens).

There is a pitfall here that is very easy to underestimate: the granularity of the constraint is the token, while the granularity of the grammar is the character. When the text form of a single token crosses a grammar boundary (for example, one token is 0, rather than 0 plus ,), a naive implementation either under-constrains (emitting an extra illegal character) or over-constrains (killing a legal character). XGrammar explicitly calls this a "byte-level" or "character-level" alignment problem and lists it as a core feature — if you are writing your own constrained decoding, this is an engineering point you cannot get around.

2.5 The Cost of Constrained Decoding: How Much Speed You Actually Lose

Constrained decoding does have a cost, and how high it is depends heavily on the implementation, so it cannot be stated in one sentence. It can be broken into pieces:

(1) CPU cost of constructing the mask

At every decode step the engine must produce the allowed token set for that step. If the implementation "re-walks the grammar every time" (for example, stepping the FSM character by character and matching one by one), the cost is O(vocabulary size × average number of characters per token); if it is "precompiled + state caching + table lookup," the cost approaches O(1). The former is barely acceptable on small-vocabulary models and noticeably slows overall throughput at the hundred-thousand-token scale. The good news is: a cache miss only happens on each newly generated grammar state, and under a single schema the total number of states on the common paths is finite — which explains why a complex schema becomes "faster" the longer you use it (the cache gets warm).

(2) The cost of pushing logits post-processing into the kernel

The mask ultimately has to become a single -inf addition. If you "generate a dense -inf tensor and add it," every step writes out a memory buffer in the hundreds of thousands in length — this sits in the tens-of-microseconds range, and is normally negligible relative to a single step of GPU computation (a few milliseconds). But if you jam the constraint logic into the main kernel in order to save that small overhead, the code becomes severely more complicated. A workable compromise is to construct the mask as a GPU tensor (using boolean GPU indexing, or a dense -inf tensor) so as to avoid the CPU loop, but do not modify the main attention kernel for this.

(3) The Unexpected Effect on Output Quality

This is the piece most easily overlooked. Constrained decoding changes the model's effective distribution: if the blocked tokens carried high probability, that is equivalent to redistributing their probability mass onto legal tokens, which makes the output "more certain" than it would be unconstrained. In extreme cases, a grammar whose constraint is written too tightly leaves the model struggling among very few viable paths, and output quality (rather than format correctness) actually declines. So "the stricter the constraint the better" is wrong — the goal of a constraint is to eliminate the impossible, not to specify the unique answer. The classic symptom of over-constraint is this: the output format is perfect, but the content becomes thin, repetitive, or always picks the first item in the enum.

(4) Extra Complexity in Multi-turn Conversations

Streaming conversation is far more complicated than single-turn generation. Constrained decoding and streaming are almost natural enemies: a constraint's allowed set depends on "the complete prefix generated so far," and in streaming decoding, when you receive the first token you still do not know how long the user will wait or what comes next. This raises a very practical design decision: in a streaming scenario, either defer applying the constraint until some length is reached, or do not constrain at all during the streaming phase and only perform parse repair on the finally assembled string. This is also why many APIs' "JSON mode + stream" in fact only does the most lightweight bracket balancing, with the full constraint taking the non-streaming path.

Putting these four together, the true cost profile of constrained decoding is: on the CPU side, one additional per-step mask operation; on the GPU side, one additional tensor addition — both diluted at the "a few milliseconds per step" scale. But it also brings a throughput drop anywhere from 20% to 60% (depending on the implementation's cache hit rate and constraint complexity), and it needs extra adaptation when combined with optimizations such as streaming and speculative decoding. Speculative decoding is especially troublesome — the draft model knows nothing about the constraint and proposes illegal tokens, and the target model has to filter them during the verification phase.

With "how to enforce hard constraints" covered, the next section answers a more specific question: the constraint in your hands is not a hand-written grammar but a JSON Schema — how do you turn it into a form the decoder can use.

III. What Constrained Decoding Is Actually Computing

The previous two chapters covered "how to do it." This section fills in a piece of understanding that many people skip but will eventually run into: after those tokens are masked out, what exactly happens to the model's probability distribution. Without understanding this, you cannot explain why constrained decoding sometimes makes the output worse, nor judge whether your own constraints are written too tightly.

3.1 It Is a Conditioning Step, Not a Filter

The most common intuitive model is "filtering": pick out the illegal tokens, throw them away, and sample the rest with their original probabilities. That intuition is right in some cases but mathematically wrong, and the difference matters a great deal.

The accurate statement is this: let the model's token distribution under the current prefix c be P(t | c), and let the legal token set defined by the constraint be A(c). Constrained decoding actually samples from the conditional distribution renormalized over the legal set:

# 无约束：P(t | c)
# 约束后：P'(t | c) = P(t | c) / Z(c)   当 t ∈ A(c)，其中 Z(c) = Σ_{t' ∈ A(c)} P(t' | c)
#           P'(t | c) = 0               当 t ∉ A(c)

# Z(c) 就是被剪掉的概率质量总和：所有非法 token 的概率加起来是多少
clipped_mass = 1.0 - Z(c)

This Z(c) (or, equivalently, clipped_mass) is a quantity you can measure directly, and it is extremely useful: it tells you how badly the model "wants" to say something illegal. If clipped_mass stays near zero over time, the model wanted to say something legal anyway and the constraint is just a decoration; if it stays high (say, above 30%), your constraint is in serious conflict with a strong tendency of the model, and at that point the constraint is not "avoiding errors" but "forcing the model to say things it does not want to say."

The latter is the root of quality problems. Imagine the constraint requires temperature to be one of 0, 1, or 2, while the model very much wants to output 0.7 — the mask shoves it toward 2, and the model's internal state is in fact fighting the constraint the whole time. The result: the field value is legal, but it may inexplicably always pick the last one, or show up as redundancy and repetition in other fields. This is not a bug; it is an inevitability of conditioning.

Add clipped_mass to your monitoring dashboard. It is the one metric that can discover "over-tight constraints" online without parsing the output — and output quality degradation is precisely the class of problem that is hardest to detect automatically in production.

3.2 The Spectrum of Constraint Strength

If you spread the word "constraint" out, you find it actually covers several categories with an enormous range of difference. Figuring out which layer you are on can save a lot of wasted effort.

Layer	What it constrains	Implementation	Reliability	Scalability
Literal constraint	A certain position must equal a certain string exactly	Mask down to a single token; or template concatenation directly	Highest	Best
Lexical constraint	The digit shape of an integer, the character set of a date, the character set of an identifier	Character-set mask / regex	High	Good
Syntactic constraint	Bracket matching, key-value alternation, type structure, enum values	FSM / GBNF / JSON Schema compilation	High	Medium, degrades when the schema is complex
Semantic constraint	Cross-field consistency, property dependencies, value ranges, referential integrity	Only possible via post-sampling validation	Depends on the validator	Poor, the rules explode
Intent constraint	Whether the output actually answers the question	Prompts, few-shot, reasoning, switching models	Low	Uncontrollable

This table has a very practical corollary: the cost-effectiveness of constrained decoding decreases as you move up the layers, while coverage increases. So the correct order of investment is — first squeeze the literal constraints for all they are worth (for example, if you can use const, do not use string), then compile away every semantic keyword that can be compiled into grammar, hand everything that cannot be compiled to the validator, and only then push hard on the prompt. Many teams skip the first two steps and go straight to grinding on the prompt, which amounts to using the most expensive method on the easiest problem.

3.3 A Counterintuitive Conclusion: Constraints Do Not Always Make the Output "Narrower"

Many people assume on first contact that constrained decoding reduces output diversity, so it should be used for deterministic output while ordinary sampling is used for creative tasks. This understanding is backwards — in at least two senses.

First, constraints remove impossible paths, not possible ones. If your schema allows six values, the distribution under the constraint still covers all six; only the proportions have been redistributed. What actually determines diversity is how specific the schema is — a schema whose field has 200 enum values is far "narrower" than a free-form string, and that has nothing to do with whether you used constrained decoding.

Second, on structured generation tasks, adding constraints often makes the content richer. This is a reproducible phenomenon: when the model is generating freely, the format burden of "must output valid JSON" diverts some of its attention, leaving less budget for the content; once the format is guaranteed by the decoder, the model can put all of that attention onto the field values. What you observe is this: after adding constraints, required fields get filled in more completely, empty strings appear less often, and early termination is rarer. The reason is that "must fill in every required field" has changed from a probabilistic request into a structural fact — the model no longer has the option of "forgetting a field."

The corollary is this: for tasks like extraction, classification, and summarizing into structured fields, the main benefit of constrained decoding is often not the "parse rate" but the "field completeness rate." That metric is more worth watching, because it is what actually affects downstream quality, and in the unconstrained era it was entirely unobservable.

IV. How Constrained Decoding Interacts with Other Inference Optimizations

No production environment has an engine that uses "only constrained decoding." Everyone is simultaneously using batching, prefix caching, speculative decoding, and so on, and there are real couplings and conflicts between those and constrained decoding. This section covers those interactions, because the people who get burned usually only discover them during load testing.

4.1 Coupling with Continuous Batching

The premise of continuous batching is that "the shapes of all requests in the same step may differ." Once constrained decoding is added, there is an extra layer: requests in the same step may have completely different constraints. There are two implementation choices:

• Group by constraint — put requests with the same constraint into the same batch, and compute the mask once and reuse it for the whole batch. The cost is that a request has to wait in the queue for its peers, so tail latency worsens; the benefit is that the mask overhead is amortized.
• Per-request masks — each request brings its own mask into the batch, which requires at the implementation level that the mask be expressible as a batch-shaped tensor (this is the "feed a sparse tensor into the kernel" from section 2.2). This is the choice of mainstream engines, because it does not sacrifice the flexibility of the batch.

Which one to choose depends on how diverse your constraints are: if the whole company has a single fixed schema, the first option (or even, more aggressively, schema-level batching) yields enormous benefit; if every user's schema is different, the second is the only option. This directly explains why "constrained decoding loses almost no throughput at low concurrency but eats CPU at high concurrency" — because at high concurrency it is precisely when request constraints are most dispersed and the cache hit rate is lowest.

4.2 Coupling with Prefix Caching

Prefix caching requires multiple requests to share exactly the same prefix KV. Constrained decoding does not change the prefill phase (the constraint only acts during the generation phase), so in theory the prefix cache hit rate is unaffected. But there is an indirect effect: constraints make the output length more predictable — because the grammar determines the length range of the required fields. A request that previously might output 50 tokens, or might output 500, will now settle stably into some range. This makes request lifecycles neater, indirectly reduces batch fragmentation, and is a positive effect on overall throughput.

But there is one trap to watch out for: if the schema differs on every request, do not count on prefix caching. Many people put dynamic content into the system prompt (for example "the current user is X, the current date is Y"), which by itself breaks the cache; this has nothing to do with constraints, but if you are investigating "why did the cache hit rate drop after we turned on constraints," do not misattribute the cause.

4.3 Coupling with Speculative Decoding (Where Problems Are Most Likely)

Speculative decoding uses a small model to quickly propose a number of candidate tokens, which the large model then verifies in a batch. The problem is that the draft model usually does not know the constraint exists. It proposes k tokens according to its own preferences, and some of them may be illegal. The verification phase must apply the corresponding mask at every position, and the accepted length depends on where the first illegal token appears.

This leads to a counterintuitive performance collapse: if the draft model is insensitive to the constrained format, then among the k tokens it proposes the first illegal one appears at position 2 on average, so the accepted length is always 1, speculative decoding degrades into "saving one forward pass per step," and you additionally pay the overhead of both drafting and verifying — a net loss.

Three approaches, in order of recommendation:

1. Give the draft model the constraint as well (pass the same grammar to the draft). Highest acceptance rate; at the implementation level it requires draft and target to use the same constraint engine.
2. Apply the constraint only during target verification (the mainstream approach). The acceptance rate drops, but because the draft is usually very light, total throughput may still be positive. Before shipping you must actually measure the accepted-length distribution and check whether accepted_len is significantly smaller than k.
3. Simply turn speculative decoding off for long structured outputs. When the output is constrained to near-determinism, the draft cannot help; but when the schema is loose (for example the fields are free text), speculative decoding still pays off. The deciding factor is how tight the schema is, not the total output length.

4.4 Working with Small Models

A fact that is easily overlooked: the weaker the model, the greater the benefit of constrained decoding. A 7B model's format adherence rate is often markedly below a 70B model's, and on such a baseline the gains you can get from prompting are minimal, while constrained decoding lifts format success rate from "a matter of luck" straight to close to 100%. In other words, constrained decoding makes the decision to "use a small model" viable again — without it you would be forced up to a large model because of unstable formatting.

But small models have their own engineering limitations: guided decoding support at the implementation level usually lands on large-parameter models first, and a quantized small model occasionally exhibits distribution degeneration within an extremely narrow candidate set (especially when clipped_mass is high). So if your architecture is small model plus constrained decoding, be sure to monitor that "clipped probability mass" — on quantized models this value is often higher than expected.

V. Structured Editing and Patch Formats

Everything discussed so far has been about "generating a structured object from scratch." But in 2026 the other battlefield where constrained decoding is most valuable is structured editing — having the model modify existing content in the form of a patch rather than rewriting the whole file or the whole object. This direction deserves its own section, because its payoff is much larger than for JSON generation.

5.1 Why Editing Formats Especially Need Constraints

Consider the most mundane task imaginable: have the model fix a bug in a file. If you let the model output the entire modified file, there are three problems — it is long (expensive and slow), it easily damages the parts you did not intend to change (poor edit precision), and it cannot be verified (you can only use a diff to guess what it changed).

So the earlier approach was to have the model output a structured patch. The grammar of the patch format is extremely simple, usually just "a number of search-replace pairs":

// search/replace 编辑格式：模型只需输出变更部分
<<<<<<< SEARCH
def calculate_total(items):
    return sum(i.price for i in items)
=======
def calculate_total(items):
    return sum(i.price * i.quantity for i in items)
>>>>>>> REPLACE

This format has three particularly good properties, which make the value of constrained decoding here far greater than in the JSON case:

• The grammar is minimal — there are only two separator markers, SEARCH and REPLACE, so the constraint is simpler to write than a JSON Schema; a dozen-line regex is enough and nothing needs to be compiled.
• It is verifiable — the search segment must appear verbatim in the original text, and that can be precisely validated before the patch is applied: if the search segment does not match, the model has hallucinated code that does not exist, and at that moment you can fail immediately or let the model retry, instead of waiting for a user to discover it.
• It is countable — output length is proportional to the amount of change rather than to the file size, which is an order-of-magnitude improvement for cost.

The way to constrain it is even simpler than for JSON: the whole format needs only two separator markers, and the constraint rule can be summarized as "first emit the SEARCH marker, then arbitrary lines until a line of equals signs; then arbitrary lines until the REPLACE marker line." Written as a regular expression: the marker lines are literals, and the content between markers is a repetition of "any character except a newline." The regex complexity of this rule is essentially zero and the compilation overhead is negligible — one of the reasons editing tasks are the best fit for constrained decoding.

Note the general pattern illustrated here: the syntactic constraint is responsible for "what the patch looks like," and pre-application validation is responsible for "whether the patch actually applies." These two things must be separated, because a syntactic constraint is in principle unable to express "this text must appear in the file" — that requires looking at external state.

5.2 A More Aggressive Format: Constrained diff

If you constrain the standard unified diff format directly, the payoff is larger but the implementation is harder, because it has a complex line-prefix grammar (context, added, removed, hunk header, plus line-number fields), and it requires hunks to be strictly ordered. This is exactly where a context-free grammar can prove its worth: write the hunk grammar as productions, and let the mask tighten step by step during generation — once a hunk ends, the line numbers of subsequent hunks must be larger, and this condition can be expressed inside the grammar state.

Different tools differ noticeably in approach, and it is worth knowing the characteristics of each:

Format style	Representative approach	Advantage	Difficulty
Search-replace pairs	Early file-editing patches	Minimal grammar, low constraint cost	Cannot locate uniquely when the context repeats
Constrained unified diff	Hunk-based structured diff	Expressive, change size is naturally computable	Line-number fields must be strictly constrained, implementation is complex
Line-level edit instructions	Line-number-based insert / delete / replace instructions	Naturally matches structured editing models (insert / delete / replace)	Line numbers drift as earlier edits are applied and must be maintained
Anchor-based location	Use an anchor string rather than a line number	Robust to earlier edits	The anchor must be unique, so validation cost rises

A practical judgment call for selection: if your edit target frequently occurs multiple times in the file (for example, many lines that look alike), line-number approaches must additionally maintain offsets and anchor approaches must additionally perform uniqueness validation — both of those extra costs have to be counted in, rather than comparing only the elegance of the format itself.

5.3 Three Layers of Validation in Structured Editing

Editing scenarios require stricter validation than generation scenarios, because a wrong patch will break code. The three layers are: format (guaranteed by constrained decoding), applicability (the search segment must hit, the replace segment must be unique or must explicitly mark its intent), and semantics (does the code run after the change, do the tests pass). The first two are structural and cheap; only the third actually requires execution.

Here is a judgment that holds only in editing scenarios: the failure rate of applicability validation says more about what is wrong than the format failure rate does. Because after constrained decoding the format failure rate should already be close to zero; once you see applicability validation failing frequently, it means the model's understanding of the code is broken (it hallucinated a fragment that does not exist), and at that point raising the retry count is useless — you should go back to model selection or context construction.

VI. Compiling JSON Schema into Decoding Constraints

For the vast majority of teams, the source of the schema is not hand-written GBNF but an existing JSON Schema document — it may come from an OpenAPI definition, from a database structure, or it may just be an interface document. This section covers how to compile a Schema into a grammar the decoder can consume, and which real pitfalls you will meet during compilation.

6.1 What Compilation Is Actually Doing

The compilation process can be summarized in one sentence: translate a declarative type constraint into a set of context-free production rules. The whole flow is:

1. Preprocessing — parse the Schema, handle $ref, inline $defs, and expand nested references. This step determines whether the later rules will recurse infinitely.
2. Walk the nodes and emit rules — for each node, determine what type it is (object / array / string / number / boolean / union), and generate one grammar rule for it, named after the normalized form of its path, for example root.properties.user.properties.age. Types referenced across nodes (for example "this field has the same type as that field above") get a standalone rule rather than being expanded repeatedly, which keeps the grammar smaller and faster.
3. Composition and de-duplication — attach the rules generated for child nodes to the parent node, and delete orphan rules that no rule references (deadcode elimination). This step has a considerable impact on performance: every extra rule in the grammar makes both compile time and the number of DFA states grow linearly.
4. Compile into an execution structure — GBNF is plain text, so it finally has to be converted into llama.cpp's grammar structure; in Outlines this step produces the DFA's edge table; in XGrammar it produces a compressed byte-level jump table.

6.2 A Minimal Usable Compiler

The implementation below omits engineering details such as caching, $ref expansion, and cycle detection, but keeps the complete dispatch logic — which is precisely the hard part of a compiler: for every Schema keyword you must explicitly write out whether it "can be compiled into a grammar rule" or "cannot be compiled and must be discarded." In production implementations these discards are usually collected into a warning list returned to the caller.

import re
from typing import Any, Dict, List

# 每种基础类型对应的 GBNF 片段（为可读性省略了空白容忍规则 ws?，
# 真实文法需在分隔符后插入它，否则模型无法在逗号后加空格）
PRIM = {
    "boolean": '"true" | "false"',
    "string":  r'"(\\.|[^"\\])*"',
    "integer": r'-?(0|[1-9][0-9]*)',
    "number":  r'-?(0|[1-9][0-9]*)(\.[0-9]+)?',
    "null":    '"null"',
}


class SchemaCompiler:
    """JSON Schema -> GBNF。核心是 _body 的关键字分派：
    能编译的生成产生式，编不了的记入 discarded，交给上层校验兜底。"""

    def __init__(self):
        self.rules: List[str] = []
        self.discarded: List[str] = []

    def compile(self, schema: Dict[str, Any], name: str = "root") -> str:
        self.rules.append(f"{name} ::= {self._body(schema, name)}")
        return self._finish()

    def _emit(self, name: str, body: str) -> str:
        self.rules.append(f"{name} ::= {body}")
        return name

    def _body(self, s: Dict[str, Any], name: str) -> str:
        # $ref：单独成规则以便跨节点复用，避免就地展开导致语法成倍膨胀
        if "$ref" in s:
            return s["$ref"].split("/")[-1]

        # const / enum：字面量枚举，性价比最高的约束
        if "const" in s:
            return lit(s["const"])
        if "enum" in s:
            return " | ".join(lit(v) for v in s["enum"])

        # anyOf / oneOf：编译成选择分支
        for key in ("anyOf", "oneOf"):
            if key in s:
                alts = [self._emit(f"{name}-{key}-{i}",
                                   self._body(sub, f"{name}-{key}-{i}"))
                        for i, sub in enumerate(s[key])]
                return " | ".join(alts)

        typ = s.get("type")
        if typ == "object":
            return self._object(s, name)
        if typ == "array":
            items = s.get("items", {})
            r = self._emit(f"{name}-items", self._body(items, f"{name}-items"))
            return f'[{r} ("," {r})*]'
        if typ in PRIM:
            return PRIM[typ]

        # 编不了的关键字：显式记录，不要默默丢掉
        for kw in ("pattern", "format", "minLength", "maxLength", "minimum",
                   "maximum", "multipleOf", "uniqueItems", "patternProperties",
                   "if", "then", "not", "dependentSchemas"):
            if kw in s:
                self.discarded.append(f"{name}.{kw}")
        return self._any(name)

    def _object(self, s: Dict[str, Any], name: str) -> str:
        props: Dict[str, Any] = s.get("properties", {})
        if not props:
            return self._any(name)
        required = set(s.get("required", []))

        def member(pname: str) -> str:
            sub = self._body(props[pname], f"{name}-{pname}")
            return f'{lit(pname)} ":" {self._emit(f"{name}-{pname}", sub)}'

        # 先必填后可选：JSON 键顺序无语义，强制这个顺序可让语法显著简单
        ordered = [p for p in props if p in required] + \
                  [p for p in props if p not in required]
        seq = ""
        for i, p in enumerate(ordered):
            frag = member(p)
            # 第一个之后的所有成员都要以逗号开头；可选成员整体再套一层
            frag = f'{", " if i else ""}{frag}'
            seq += f' ({frag})?' if p not in required else frag
        return '{' + seq + '}'     # additionalProperties:false 编不出来，靠上层校验

    def _any(self, name: str) -> str:
        """类型未指定时的放行规则：任何 JSON 值都接受。"""
        return self._emit(f"{name}-any",
                          r'"(\\.|[^"\\])*" | -?[0-9]+ | true | false | null '
                          r'| \{[^{}]*\} | \[[^\[\]]*\]')

    def _finish(self) -> str:
        """死代码消除：只保留从 root 可达的规则。"""
        names = {r.split("::=")[0].strip() for r in self.rules}
        reachable, frontier = set(), [self.rules[0].split("::=")[0].strip()]
        while frontier:
            n = frontier.pop()
            if n in reachable:
                continue
            reachable.add(n)
            body = next(r for r in self.rules
                        if r.split("::=")[0].strip() == n).split("::=", 1)[1]
            for tok in re.findall(r"[A-Za-z][\w\-\.]*", body):
                if tok in names and tok not in reachable:
                    reachable.add(tok)
                    frontier.append(tok)
        return "\n".join(r for r in self.rules
                         if r.split("::=")[0].strip() in reachable)


def lit(v: Any) -> str:
    """Python 值 -> GBNF 字符串字面量。"""
    if isinstance(v, bool):
        return '"true"' if v else '"false"'
    if v is None:
        return '"null"'
    return '"' + str(v).replace('\\', '\\\\').replace('"', '\\"') + '"'

There are several design decisions in this code worth discussing separately, because they are the dividing line between "it runs" and "it is usable":

(1) enum and const are the highest-leverage constraints

Because they directly generate literal enumerations, which the decoder can enforce exactly and which carry almost no risk of state explosion. If your schema has a lot of enum fields (status codes, categories, action types), then constrained decoding can make that portion nearly perfect at extremely low cost.

(2) $ref must be reused across nodes, never expanded in place

A typical counter-example: an address schema has street, city, zip, and so does the user schema. If every reference is expanded in place, the generated GBNF balloons exponentially. The correct approach is to assign a canonical name to the referenced type, expand it only once, and let every other site reference the name directly. That way the "address" type has only one definition in the grammar, and compile time drops as well.

(3) Recursive schemas are a genuine hard problem

Consider the classic recursive definition in JSON Schema: {"type":"object","properties":{"children":{"type":"array","items":{"$ref":"#"}}}}, which represents a tree of arbitrary depth. Compiling it requires generating the rule node ::= "{" ... "children"... "[" node* "]" ... "}" — GBNF itself supports this kind of recursion and llama.cpp can handle it. But note: recursion depth cannot be limited at the grammar layer, and a pathological prompt might induce the model into infinitely deep nesting. You either enforce a depth limit in code after generation (explicitly count the nesting while reading the JSON), or unroll the recursion to a finite number of layers at the schema compilation stage — this is a clear trade-off point, and I lean toward the former (keep the recursion, set a hard cap on the code side).

(4) additionalProperties: false cannot be compiled

This is the limitation you most need to be told honestly about. GBNF can exhaustively enumerate all allowed keys, but it cannot express a universal negative like "no keys other than these are allowed" — for that you would need a negative lookahead in a regex or post-generation validation. llama.cpp's approach is to add a guard such as (?!\s*[}\]]), but strictly speaking that is only a heuristic. Truly reliable additionalProperties: false semantics can only come from upper-layer validation. This point will come up again in the three-layer architecture described later.

6.3 The Three Numbers to Watch Most Closely at Compile Time

A finished implementation is not the same as a usable one. Before shipping, measure at least these three quantities, because they determine whether runtime will "compile once, then stall for half an hour":

• Compile time — the time to convert the schema to GBNF. Deep nesting or a large anyOf raises it significantly. If compilation is slow enough to reach seconds, you should add caching (hash of the schema string → grammar text).
• Number of grammar rules and total rule length — these directly determine the number of DFA states and the cold-start latency. A grammar with a thousand rules basically means the first request pays a cold-start cost of several hundred milliseconds.
• P99 of mask construction — not the mean. The latency distribution of constrained decoding usually has an extremely long tail: common states hit the cache at the microsecond scale, while a new state has to walk the entire grammar and can take milliseconds. Looking only at the mean during load testing severely understates the problem.

Comparing Schema Compilation Paths Across Engines

The same project, XGrammar, provides paths for different engines, and the behavioral differences are worth noting. The table below is a qualitative comparison observed in my own testing (version differences will change the details, so only stable behavioral patterns are discussed here):

Path	When the constraint takes effect	Schema coverage	Multi-turn / streaming	Applicable scenarios
vLLM (switchable backend)	Inside the engine, masking every token	Medium-high, depends on the backend	Supported	General-purpose production services
SGLang	Inside the engine, masking every token	Relatively high	Supported	High-concurrency services that need structured output
llama.cpp (GBNF)	Sampling layer, token by token	Depends on the expressive power of the grammar	Requires special handling	Edge deployment, CPU/GPU inference
Outlines	DFA table lookup, compiled once	Medium-high, friendly to type annotations	Requires wrapping	Embedded on the Python side, fixed structure
XGrammar	Byte-level masking, compiled once	High, covers the most keywords	Supported	When you want both coverage and speed
guidance-ai	Token-level constraints + reverse parsing	High (regex / choice / capture)	Streaming-friendly	Complex control flow, streaming
JSONFormer / lm-format-enforcer	Local constraints (token-level / tree-path)	Implementation-dependent	Limited	Quickly patching an existing integration

One piece of experience about these projects: do not choose by "who supports more keywords," choose by "what your schema looks like and what your traffic pattern is." If your schema is just a simple object plus a few enums, any of them will do; if you have deep recursion and large anyOf blocks, then how well "compile once + dead-code elimination + cross-node reuse" is done matters more than the number of supported keywords, because those dropped keywords could only ever be handled by upper-layer validation anyway.

By this point, "the output definitely conforms to the grammar" has been thoroughly solved. But if your scenario also has "the output must satisfy business constraints" — for example "when status is rejected, rejected_reason must be present" — then you still have to come back to what happens after sampling. The second half of this article covers exactly that remedial path, and how to use it to catch everything that slipped through earlier.

VII. Post-Sampling Repair: Parse-Repair, Structural Validation, and Bounded Retries

Even with per-token constraints, do not expect 100% correctness. There are three classes of reason:

• Semantic constraints that cannot be compiled — as noted in the previous section, keywords like additionalProperties: false, dependentSchemas, and pattern are basically inexpressible in GBNF, so the model may output content that violates them.
• max_tokens truncation — if the schema permits a long array and you gave a very small max_tokens, the output gets cut off halfway. This is an extremely easy failure source to overlook: the model was running perfectly well, your budget was simply too small.
• The constraint itself is wrong — the schema and the constraint have drifted out of sync (you changed the schema and forgot to regenerate the grammar), or there is a bug in the grammar. Such problems cannot be caught at compile time and can only be found through end-to-end testing.

So the third layer, "post-sampling repair," is not an optional patch; it is the backstop, the layer that pulls "parseable" from 99.5% to 100%.

7.1 parse-repair: Fixing Broken Syntax

The most obvious technique is to throw the model output at a fault-tolerant JSON parser such as json_repair (Python) or jsonrepair (JS). What it does falls into several steps:

1. Truncation — find the first { or [ and cut away everything before it (this is the standard fix for contamination like "Sure, here is the JSON you asked for:"); symmetrically, cut away everything from the last closing delimiter onward.
2. Completing the closing delimiters — add the missing } / ] in stack order.
3. Fixing punctuation — strip trailing commas, replace single quotes / Chinese quotation marks with double quotes, convert Python-style True/False/None into JSON literals.
4. Adding escapes — bare newlines in strings, unescaped quotes, backslashes: all get escapes added.
5. Removing comments and stray text — some implementations also strip // and /* */ comments.

This kind of repair can rescue a fairly high proportion of broken outputs, and it costs almost nothing (pure CPU, microseconds to milliseconds). But you must be clear about its two limitations:

Limitation one: it may change semantics. Repairing a broken JSON and repairing it correctly are two different things. For example, if the model outputs {"a": "he said "hi" to me"}, the repairer will either escape the inner quotes (giving a = 'he said "hi" to me', which happens to be right) or truncate the trailing content (giving a = 'he said ', which is silently wrong). Silent errors are far more dangerous than explicit failures — so you must validate after repair, rather than treating "repaired" as "trusted."

Limitation two: it cannot rescue truncated semantic content. If the output was cut in the middle of an array by max_tokens, the repairer will hand you an object that looks legal but whose array is short. It is syntactically correct and semantically wrong. The only way to detect this is to "check required fields and array lengths," which is the job of the layer below.

7.2 Structural Validation: The Semantic Checks Beyond the Schema

Validation splits into two layers, and you need to write both.

Layer one: JSON Schema validation. This step verifies structure, types, required fields, enum values, and numeric ranges. jsonschema (Python), ajv (JS), or zod (TS) all work. The key point is this: the validator must use the same schema as the constraint compiler. If you hand-write a GBNF and also maintain a Pydantic model, the two will inevitably drift, and the direction of drift is usually "validation stricter than the constraint" — that is fine, it means the constraint is conservative; the dangerous direction is the reverse.

Note that jsonschema does not do format validation (dates, emails, UUIDs) by default, so you have to enable FormatChecker explicitly. Also, type: integer in JSON accepts 1.0 (numerically it is an integer), so if you need a strict distinction you have to add the check yourself.

Layer two: business semantic validation. This layer cannot be expressed in schema and must be written by hand. It is the real gate standing in front of the business system:

• Cross-field consistency: end_date > start_date, total == a + b.
• Property dependencies: when status == "rejected" there must be a reason.
• Referential integrity: does the generated user_id actually exist in the database?
• Chronological sanity of business enums: a state machine cannot jump directly from draft to shipped.

Organizing the validation results into error messages that the model can read is the most valuable design decision in this layer. Many teams finish writing the validation and then simply throw an exception, wasting the next retry for nothing — the correct approach is to collect all validation errors, take the first several, write each one in the form "field path + reason for failure," and attach them to the context of the next round. For example, when an amount is inconsistent, what should be fed back is a message like /total: does not equal the sum of the individual items, not a vague "output is invalid."

Pay attention to the small detail of including a JSON Pointer (/reason, /end_date) in the error message: the model is far more sensitive to "which field is wrong" than to "rewrite everything," and feeding back only the error paths rather than the entire failed output will usually noticeably raise the success rate of the retry, while also saving the token cost of prefill.

7.3 Bounded Retries: It Is an Architecture, Not a Loop

"Retry until it succeeds" is a trap. Its failure mode in production is extremely clear: some downstream service keeps returning an error the model cannot fix, so your service retries forever, burning through the token budget and pushing latency beyond the SLA. Retries therefore need hard boundaries, and that boundary should be specified explicitly by the caller rather than decided secretly inside a library.

Three things that must be set explicitly:

• Maximum number of attempts — including the first one. Once it is exceeded, throw and let the caller decide.
• Total token budget — a hard constraint more important than the attempt count. Many attempts that each generate long text will still blow through the budget.
• The shape of the error feedback — on retry, do you include the previous failed output? If you do, the model has a reference (but the prompt gets longer); if you don't, the prompt stays short (but the model does not know what went wrong). In practice, including an error summary while excluding the full failed output is a very good compromise.

import json
import logging
from dataclasses import dataclass, field

logger = logging.getLogger(__name__)


@dataclass
class GenPolicy:
    max_attempts: int = 3
    token_budget: int = 4000        # 跨所有尝试累计
    backoff_base_s: float = 0.5     # 指数退避，缓解服务端瞬时过载
    include_prev_error: bool = True


@dataclass
class Attempt:
    ok: bool
    obj: object = None
    error: str = ""
    used_tokens: int = 0


def generate_valid(generate_fn, schema, policy: GenPolicy, ctx: str = "") -> object:
    """
    generate_fn(prompt: str, max_tokens: int) -> (raw_text: str, tokens_used: int)
    抛出的所有异常都视为失败，进入下一轮重试。
    """
    prev_error = ""
    spent = 0

    for attempt in range(1, policy.max_attempts + 1):
        remaining = policy.token_budget - spent
        if remaining <= 0:
            raise RuntimeError(f"token budget exhausted after {attempt-1} attempts")

        prompt = build_prompt(schema, ctx, prev_error if policy.include_prev_error else "")
        try:
            raw, used = generate_fn(prompt, max_tokens=min(1024, remaining))
            spent += used

            ok, obj, err = parse_with_repair(raw)
            if ok:
                ok2, err2 = validate_and_describe(obj, schema)
                if ok2:
                    logger.info("valid on attempt %d, %d tokens", attempt, spent)
                    return obj
                err = err2                       # 结构合法但语义不合法

            prev_error = err
            logger.warning("attempt %d failed: %s", attempt, err[:200])

        except Exception as e:                   # 网络、限流、解析异常都归到这里
            prev_error = f"generation error: {e}"
            logger.warning("attempt %d raised: %s", attempt, e)

        if attempt < policy.max_attempts:
            sleep(policy.backoff_base_s * (2 ** (attempt - 1)))

    raise ValueError(f"failed after {policy.max_attempts} attempts, {spent} tokens: {prev_error[:300]}")

There are two easily overlooked details here: the retry delays accumulate (three retries plus backoff easily exceed ten seconds under slow responses), so if you have an SLA on end-to-end latency, the number of retries should be derived backwards from the latency budget rather than arbitrarily set to 3; and retries amplify downstream load — an interface with an 80% success rate configured with 3 retries reaches an effective success rate of 99.2%, but the volume of requests the downstream receives grows by 25% — and this cost has to be agreed with the downstream party in advance.

Rule of thumb: if a single generation succeeds more than 95% of the time, retries are almost always worth it; if it is below 60%, three retries will not save you either, and you would be better off changing the constraints, changing the prompt, or simply showing the error to the user — an honest error is far more useful than data that looks right but is wrong.

7.4 A "Side Effect" of parse-repair to Watch Out For

One last warning about a pitfall that has actually been hit: the repairer makes your error-rate curves look bad. Because broken outputs now "succeed" after being repaired, the parse failure rate on your monitoring looks very low while the business validation failure rate has gone up — among those repaired outputs, the syntax is right and the semantics are wrong. If you only monitor the first layer, you will think everything is fine while upstream and downstream are quietly receiving silently corrupted data.

So your instrumentation must be layered: parse failure rate (the proportion that parsed on the first attempt), repair rate (the proportion that needed repair), and semantic validation failure rate — look at all three. A sudden rise in the repair rate often means something is wrong at the constraint layer (for example, max_tokens was turned down), and this is a very good early-warning signal.

At this point the individual techniques are basically covered. The remaining core question is how to organize them into an architecture that can go to production — which is the next section.

VIII. What Is Special About the Tool-Calling Scenario

Tool calling (function calling / tool use) is the most typical deployment scenario for constrained decoding, but the shape of its constraints differs from ordinary JSON generation in three important ways that deserve their own section.

8.1 The Outer Layer Is an Enum, the Inner Layer Is the Schema

The output shape of a single tool call is roughly: an outer object containing a tool-name field and an arguments field, and the value of the arguments field is yet another structure. Write a structure that nests layer inside layer, where the set of legal values differs at every layer, as a grammar — you can imagine what the rules will look like.

This means the constraint must be dispatched by tool name: first attach an enum constraint on tool, and once it has been determined to be search_docs, the constraint on arguments has to switch to that tool's parameter schema. This is not a case of "one big grammar covering every tool," but a grammar that narrows dynamically along with the prefix — and fortunately the per-token framework supports this natively: once the tokens "tool": "search_ have come out, the mask can keep only docs.

This capability is more valuable than it looks: it eliminates tool-selection mistakes at decode time. Without constraints, the model has a chance of calling a tool name you decommissioned long ago, or of spelling search_docs as searchDoc; with an enum constraint, those outputs simply do not exist.

8.2 Parallel Multi-tool Calls Need Stricter Constraints

If your scenario supports returning several calls at once (a tool_calls array), the grammar must express "the tool field of each element in the array is constrained independently." The approach is to expand "name + corresponding arguments" together as a single branch, rather than choosing the name first and the arguments second.

Tying "name + corresponding arguments" together as one atomic branch is the right approach, but the cost is that doubling the number of tools makes the grammar grow linearly — ten tools means ten branches, each carrying a complete parameter schema, and both compile time and the size of the state table have to be recomputed. Once you are past twenty or thirty tools, the cost-effectiveness of this step starts to decline, and a better strategy is a "two-phase" one: in phase one, constrain only the tool name (a tiny enum grammar, extremely low cost); in phase two, after the tool is determined, apply constraints to the arguments (or even use a second independent request).

8.3 How Much the Server-side API Layer Already Does for You

By 2026, mainstream model providers basically all offer server-side structured output, and you can capture part of the benefit of constrained decoding without wiring up an engine yourself. The reality is that coverage varies widely between vendors, and the layering below is what is worth knowing:

• JSON mode (the weakest): it only guarantees the output is a JSON value, not that it conforms to any schema. The usual implementation is "start constraining after finding a {, and stop when a balanced } appears," or even just a small change to the sampling logic. It solves wrapper contamination and syntax breakage; it cannot solve schema drift.
• Structured outputs / json_schema (stronger): it accepts a schema and performs grammar-level constraints on the server side. This is the industry-standard approach that OpenAI structured outputs popularized, and it is the tier most worth adopting. Its known limitations are usually these: only a subset of the schema is supported (for example, keywords that cannot be compiled into a grammar such as minLength/maximum/pattern are not supported, nor is the strict version of additionalProperties: false), so you need to confirm this in the documentation ahead of time.
• Strict function calling: on top of structured outputs it adds a layer binding the tool name to the parameter schema — in essence the coordinated constraint described in section 8.1.

So a pragmatic selection order is this: use the server-side capability first (lowest cost, least operations burden), and only consider building your own once you have confirmed the parts it does not satisfy (for example, some schema keyword you need that it cannot compile, or a need for streaming). Building your own means taking on the long-term costs of engine version pinning, cache warm-up, and failure-mode triage — that is no small matter.

8.4 An Overlooked Security Boundary

Constrained decoding plays one more role in tool calling that is easy to overlook but very important: it removes part of the risk of "argument injection." Consider a scenario where a malicious input induces the model to call delete_user — an enum constraint can guarantee that tool can only ever take the values you supplied, but it cannot stop the model from filling a malicious path argument inside one of your legitimate tools. Syntactically valid ≠ semantically safe.

So the final gate for tool calling is always: argument allowlist validation + permission checks + human confirmation or idempotency keys for high-risk operations. Constrained decoding lets you downgrade from "having to parse every return value" to "only having to validate semantics," which is an enormous win — but do not misread "downgrade" as "can skip it."

IX. Putting the Three Layers Together in Practice

Now let us assemble the three layers. This is the most practically useful part of the whole article, and I will give you a skeleton you can copy directly.

9.1 Decision Table: When to Use Which Layer

Scenario	Layer 1: constraints	Layer 2: validation + retries	Layer 3: degraded parsing	Reasoning
Tool / function calling	Must be on	Must be on (semantic safety)	Log only, never return	Degraded arguments can cause dangerous side effects and must never be executed silently
SQL generation	Must be on (table and column name enums + AST parsing)	Must be on (EXPLAIN validation, read-only checks)	For human troubleshooting only	Wrong SQL is destructive, and there are many syntactically valid dangerous queries
Structured information extraction	Recommended	Must be on	May be returned (mark confidence; low confidence goes to a human)	Degraded output is harmless to the business, it only needs a credibility flag
Streaming UI rendering	Hard to enable (see 2.5)	Difficult (the output is incomplete)	This is the main reliance	Streaming and constraints conflict by nature; only prefix-level fault-tolerant rendering is possible
Conversations in JSON mode	Optional	Optional	Must be on	The cost of failure is low, a lenient parser is enough
Offline batch jobs	Must be on	Must be on	May be turned off (fail and rerun instead)	There is no latency pressure, so you can safely retry until success

The actual meaning of this table is this: layer 3 is not "always on," it is decided by the cost of failure. Turn it on in low-cost scenarios; in high-cost scenarios, rather fail than degrade. This judgment matters more than any technical detail.

9.2 The Complete Skeleton

import json, logging
from dataclasses import dataclass

logger = logging.getLogger(__name__)


@dataclass
class StructuredResult:
    value: object            # 解析后的值
    confidence: float        # 0.0-1.0
    layer_used: int          # 1=约束直接通过 2=校验修复 3=降级
    errors: list             # 收集到的所有问题
    degraded: bool           # 调用方据此决定是否信任


def generate_structured(generate_fn, schema, constraints, *, max_attempts=2,
                        token_budget=4000, allow_degrade=True) -> StructuredResult:
    """
    三层组合：约束解码 → 校验重试 → 降级解析
    generate_fn(prompt, max_tokens, gbnf_grammar) -> (raw_text, tokens)
    """
    gbnf = constraints.compile(schema)          # 第0步：schema → 语法（应带缓存）
    errors = []

    for attempt in range(1, max_attempts + 1):
        remaining = token_budget
        if remaining <= 0:
            break

        # ---------- 第1层：约束解码 ----------
        raw, used = generate_fn(
            build_prompt(schema, ctx=errors if attempt > 1 else ""),
            max_tokens=min(1024, remaining),
            gbnf_grammar=gbnf,                   # ← 约束在这里生效
        )

        # ---------- 第2层：解析 + 修复 + 校验 ----------
        ok, obj, err = parse_with_repair(raw)
        if ok:
            valid, err = validate_and_describe(obj, schema)
            if valid:
                return StructuredResult(obj, 1.0, 1, errors, False)
            # 结构对但语义错 → 记录错误，进入重试
        errors.append(err)

    # ---------- 第3层：降级解析 ----------
    if not allow_degrade:
        raise ValueError(f"all {max_attempts} attempts failed: {errors}")

    best = salvage(raw)                          # 从最后一次输出里榨出可用字段
    if best is None:
        raise ValueError(f"degraded parse also failed: {errors}")

    return StructuredResult(best, 0.4, 3, errors, True)


def salvage(raw: str):
    """
    降级解析：用宽松的提取器把能拿的字段拿出来。
    例：正则抓 key/value 对，容忍括号不闭合。
    永远要标 degraded=True，让上游决定是否信任。
    """
    pairs = re.findall(r'"(\w+)"\s*:\s*"([^"]*)"', raw)     # 先抓字符串字段
    nums  = re.findall(r'"(\w+)"\s*:\s*(-?[\d.]+)', raw)    # 再抓数字字段
    if not pairs and not nums:
        return None
    out = {}
    for k, v in pairs:
        out[k] = v
    for k, v in nums:
        try:
            out[k] = float(v) if "." in v else int(v)
        except ValueError:
            pass
    return out

A few design points deserve separate emphasis, because they determine whether this code is reliable in production:

• constraints.compile(schema) must live outside the loop — the grammar is compiled only once and shared by all retries. This is the key to pushing retry costs down.
• Constraints and validation are complementary rather than redundant: constraints guarantee that "the structure is definitely right," validation guarantees that "the structure is right and the content is reasonable." With only the former, the model may keep stably emitting valid JSON that is missing a required field; with only the latter, you waste a full generation on every failure.
• Degraded results must carry a flag — degraded=True and confidence=0.4 are part of the contract. If the caller is allowed to ignore that flag and use the degraded data anyway, then layer 3 is harmful: it uses data that "looks usable" to mask a real failure.
• Degraded parsing must be conservative — extract only the fields you are certain you can get (key-value pairs that the regex matches), and do not try to "guess" missing values. The object returned by salvage is never a complete restoration of the original intent; it is a best-effort approximation.

9.3 Constraint Caching: A Small Optimization You Must Do

If your schema is fixed (which is very common, for example a fixed tool-calling protocol), the result of compile(schema) can be cached directly. More commonly in production there are only a few variants of the schema (for example three, based on user role), so cache by schema hash and evict with LRU. The payoff is compound: you not only save compile time, you also keep the engine-side mask state cache warm, avoiding a cold path on every request.

One lesson from practice: if you stuff dynamic content into the schema (for example writing the user ID into a const to make a per-user literal constraint), the cache hit rate drops to zero and the compile overhead falls entirely on every request. So the dynamic part of a constraint should use "injection at generation time" rather than "specialization at schema time" — the latter looks elegant but is actually a recompile per request.

X. The Latency Cost: Adding Up the Bill

All technical decisions eventually come down to a number: how much does each layer add to the P99 latency. Below is an estimation method for a typical structure (the concrete values depend strongly on your hardware, batching, and model size, and you must measure them yourself).

10.1 Cost Breakdown

Cost item	When it occurs	Order of magnitude	Can it be optimized
Schema → GBNF compilation	Per request (or on a cache miss)	Sub-millisecond for small schemas; tens of milliseconds for large ones	Yes, and you should cache it
First construction of a mask state	Each new grammar state, cold start	Microseconds to milliseconds, long tail	Cache it; watch P99 rather than the mean
Applying the mask each step	Every decode step	Microseconds on the CPU, one addition on the GPU	Use a GPU tensor to avoid a CPU loop
Extra prefill	On retry	Proportional to the length of the fed-back errors	Feed back only an error summary
Retries overall	When it fails	Multiplicative (1 attempt = N times)	Derive N backwards from the latency budget
Validation (parse + schema + semantics)	After every generation	Sub-millisecond to a few milliseconds	Usually not the bottleneck
Degraded parsing	Failure path only	Microseconds	Negligible

The two items that genuinely change the order of magnitude are retries and mask application; everything else is noise after the decimal point. This understanding is important, because it determines optimization priority: rather than spending effort getting validation from 2ms down to 0.5ms, it is better to raise the retry success rate from 80% to 95% — the latter directly pulls the P99 back from "double latency" to "single."

10.2 The Latency Amplification Effect of Retries

Let the latency of a single generation be T, the success rate p, the maximum number of attempts N, and the total backoff overhead B. Then:

# 期望尝试次数（直到成功或耗尽）
E[attempts] = (1 - p^N) / (1 - p)      # p < 1

# 期望端到端延迟（含退避，不含限流等待）
E[latency] ≈ E[attempts] * T + B

# P99 延迟：取决于尾部，通常由最坏路径决定
# P99 ≈ N * T + B（当 p^N 不够高时）

Take a concrete set of numbers: T = 800ms, p = 0.92, N = 3. The expected number of attempts is ≈ 2.62 and the expected latency is ≈ 2.1 seconds — while a single success takes only 0.8 seconds. In other words, to push the failure rate from 8% down to 0.05%, you paid 2.6 times the average latency as the price.

That factor of 2.6 is the inherent cost of a retry strategy and you cannot escape it. The only direction for optimization is to raise p (better constraints, a larger max_tokens, clearer prompts), not to increase N. Raising N from 3 to 5 moves the expected number of attempts only from 2.62 to 2.75 (almost no change, because failures converge quickly toward "always fails"), while the P99 latency grows by 67%.

The correct order for budgeting retries is this: first compute the acceptable P99 (say, 3 seconds), derive N = (P99 - B) / T backwards and get N = 2 or N = 3; then find a way to raise p from 0.85 to 0.95 (which usually comes from fixing constraints and max_tokens, and saves an entire retry); only then consider whether to increase N. Setting N=5 by gut feel is the most common decision-making error.

10.3 The Throughput View: How Much Slower Does Constrained Decoding Make the Engine

Latency is the single-request view and throughput is the multi-request view, and the two can lead to different conclusions. Constrained decoding affects throughput mainly through the following:

• The per-step CPU mask work — at high concurrency, the mask work of all requests serially occupies the same (or a few) CPU cores, and easily becomes the bottleneck. This is the most common scalability bottleneck of constrained decoding, and it only shows up under load.
• The state cache hit rate — if the schemas in your traffic are very dispersed, the cache is constantly flushed, the hit rate is low, and costs go up. This is the only stage where constrained decoding can produce the phenomenon of "the busier it gets, the slower it gets."
• KV cache reuse — a frequently overlooked positive effect: forcing structured output makes the output shorter and more predictable, which raises the KV reuse rate between requests, and in some cases throughput actually goes up.

If you only look at the QPS curve during load testing you will draw misleading conclusions, so measure in three tiers: low concurrency (to verify the CPU mask overhead), medium concurrency (to verify the cache hit rate), and high concurrency (to verify scalability). What is commonly seen in practice is this: at low concurrency constrained decoding has almost no throughput loss (the overhead is drowned out by the GPU computation), and at high concurrency the CPU side starts to strain — and if you need more concurrency at that point, the first choice is to move the mask logic onto the GPU or to share state, not to add another card.

10.4 The Special Ledger of the Streaming Scenario

The latency profile of streaming constrained decoding is completely different. TTFT (time to first token) is slower in streaming than in non-streaming, because the constraint has to compute the mask for the initial state first; TPOT (time per output token), on the other hand, may become faster, because there are network and rendering gaps between tokens and the CPU overhead of the mask hides inside those gaps.

So for streaming scenarios, the judging criterion should be "the smoothness perceived by the user" rather than TTFT: as long as TPOT stays stable within a few tens of milliseconds, the user will not perceive that the constraint exists. This means streaming can tolerate a heavier mask implementation (on the CPU side), while non-streaming must use a GPU tensor. This is a counterintuitive but very practical conclusion.

XI. A Checklist of Common Pitfalls

These are listed separately because most of them never raise an error, they just "quietly stop working":

• The mask uses the wrong state. The allowed set is computed for the "next state" instead of the "current state." The symptom is occasional illegal output, which is extremely hard to reproduce. Fix: take the allowed set before step(), and take it again after step().
• The tokenizer crosses character boundaries. A token's text form may straddle a grammar boundary (a single token containing both the end of a key and the start of a value). You must judge by the complete string of the token rather than by its first character, otherwise you either miss constraints or kill legal ones. This is why XGrammar emphasizes "byte-level alignment."
• The schema and the constraint are out of sync. You changed the schema and forgot to update the cached grammar. Fix: make the schema hash part of the cache key, and explicitly invalidate the cache when the schema changes.
• max_tokens is too small. The schema allows a long array but the budget is insufficient, so the output is always truncated, and then you are baffled by "why won't repair fix it either." Fix: estimate a lower bound from the schema (count how many tokens there could be at most) and give it at least double the headroom.
• No validation after repair. This is the most dangerous one. The repaired JSON is syntactically correct but semantically wrong, and the data is silently corrupted. Fix: after repair, it must pass schema validation plus business validation.
• Degraded results are taken at face value. The caller ignores the degraded flag. Fix: put degraded data in a separate field, or simply do not return it — make "use the degraded data" an explicit decision that the caller has to actively make.
• Recursive schema with no depth cap. The model (or malicious input) induces deep nesting, causing stack overflow or extremely long output. Fix: set a maximum depth when parsing and discard anything that exceeds it.
• No layered monitoring. Monitoring only the "final failure rate" hides the layered structure of parse failures, successful repairs, and semantic failures, so when something breaks you cannot tell which layer broke.

XII. How to Prove It Actually Works: An Evaluation Methodology

Constrained decoding easily falls into a dilemma: after shipping, it feels like "things seem more stable," but nobody can say exactly how much more stable. The reason is the nature of this improvement — it eliminates a class of low-frequency, intermittent, hard-to-reproduce failures, and without measurement the difference between success and non-success gets drowned out by everyday noise. This section gives a method that can actually be quantified.

12.1 Three Metrics That Must Be Separated

The most critical insight is this: "is the output correct" is a compound proposition that has to be split into three layers, yet the vast majority of teams look only at the first one, which makes all the later problems completely invisible.

Metric	Definition	Effect of constrained decoding	Why it must be viewed separately
First-pass format rate	The proportion that parses directly without repair and conforms to the schema	Improves substantially; this is the main benefit	The only metric that reflects the health of layer 1
Post-repair pass rate	The proportion that passes after parse-repair	Improves slightly	Too high a value means layer 1 is failing and the model is being propped up
Semantic validity rate	The proportion where, after passing structural validation, the content also satisfies the business constraints	Usually improves slightly (fields are more complete)	This is the metric the business actually cares about

These three metrics have to be read together to tell a complete story. For example: first-pass format rate 99%, post-repair pass rate 99.5%, semantic validity rate 70% — this says the constraint layer is working normally and the business problem lies elsewhere. Whereas a first-pass format rate of 60% with a post-repair pass rate of 99% is a clear alarm: layer 1 is doing almost nothing, all of your "stability" was bought by repair, and some of the stability that repair buys is silent semantic error.

12.2 Constraint Coverage Tests

This is the test most easily skipped yet of the highest value. The idea is very simple: for every keyword in your schema, write a test case saying "if the constraint is in effect, the model absolutely cannot violate this," then verify that it really is not violated.

For example, suppose your schema has "age": {"type": "integer"}. The corresponding test case is: construct an extremely strong inducing prompt ("please output 25.5 in the age field") and inspect the output. If the constraint is in effect, the model cannot output a decimal — it must output 26, or output a string, or simply drop the field, but it will never be 25.5. Only by running this test do you know whether the constraint is really connected.

Other cases using the same line of thinking:

• Enum exhaustion test: if the schema has a 5-value enum, construct 5 prompts that each strongly suggest a 6th value (one not in the enum), and check whether the output stays inside the 5 values.
• Required-field test: the schema has 10 required fields; run it 20 times and check whether all 10 fields are present every time.
• Recursion depth test: if the schema is recursive, use a prompt that induces deep nesting and check whether the output depth stays under control.

The real value of this group of tests lies in the following: it can alarm at the very moment the constraint fails, instead of waiting for the production parse failure rate to climb slowly before anyone notices. Making it part of CI is the single highest-return engineering action in this whole methodology.

12.3 Reverse Verification: Deliberately Break the Constraint and See Whether the Output Changes

Besides "this is what it should look like when the constraint is in effect," you must also test "this is what it should look like when the constraint is not in effect" — a stricter check, because it can expose those implementation bugs that "look like they are working but are not actually working."

The method is this: take a schema, deliberately remove one value from some enum (say, from 5 values down to 4), and then run the same inducing prompt.

• If the output really changes (the 5th value that used to be producible is now impossible), the mask genuinely acts on this position.
• If the output is completely unchanged, the constraint was never in effect at all — this is a clear bug, not "the constraint happens to be loose."

The bugs this test catches happen to be the most insidious class: the mask using the wrong state (section 8.1 of the second part mentioned this pitfall). The symptom of such a bug is "the constraint mostly works, and occasionally lets one illegal token through," which is almost impossible to find in normal testing, but reverse verification can drive it out reliably.

12.4 Regression Sets and Snapshots

Two things should be frozen into assets:

A regression set of real prompts. Sample several hundred real requests from production (mind the desensitization) and freeze them into a test set. What it measures is the effect "on your real traffic," not on artificial cases. The schema distribution, length distribution, and inducing-strength distribution in real traffic often differ enormously from those in hand-made cases.

Constraint snapshots. Take snapshot tests of the compilation result of schema → grammar: when the schema is unchanged the grammar text must be completely identical, and when the schema changes the grammar must change as expected. The value of this test is that it fails immediately when the compilation logic is broken by accident. Because the compiler is a pure function of a few hundred lines, it is exactly the kind of thing that gets broken by a casual refactor, and such breakage in a functional test may just quietly cost some corner-case schemas their constraints — a snapshot catches it in the first place.

12.5 An Evaluation Mistake You Must Avoid

A final warning about a practice that looks harmless but is harmful: reporting the benefit using "format pass rate" as the single success metric. Because this metric is very low on the baseline and improves enormously once you add constraints, which makes it look like a landslide victory; but it hides two facts — first, your retry mechanism may have been masking real format failures all along; second, the semantic validity rate on the business side may not have changed at all, or may even have dropped (if the constraints are too tight).

An honest report should give all three metrics and explicitly state where the improvement in semantic validity rate came from (usually a higher required-field completeness rate), rather than vaguely writing "structured output quality improved significantly."

13. Production Operations: Caching, Isolation, Canary Rollouts, and Observability

Once constrained decoding goes live, it becomes a component that has to be maintained over the long term. If you skip the items below on day one, you will keep paying for them later.

13.1 Designing the cache key: it must include the tokenizer version

The grammar text itself is tokenizer-independent, but the compiled artifacts (DFA tables, mask indices) are tightly coupled to the tokenizer — because masks are computed by token index. The guided decoding backend in most engines does exactly this kind of token-index table lookup. So the cache key must contain at least a hash of the schema content, the engine version, and the tokenizer version.

The tokenizer version deserves special emphasis: if your model service uses model routing, the same schema request may be routed to different models (with different tokenizers), so the cache must be isolated along the model dimension. Otherwise you get the equivalent of "model A's mask indices being used on model B," which scrambles the output completely. Such bugs appear as random, inexplicable format errors and are extremely hard to localize. The fix is simple: put the model identifier into the cache key.

13.2 Multi-tenant isolation and cache thrashing

If your service is multi-tenant and every tenant has a different schema, caching runs into a practical problem: requests from different tenants arrive alternately, constantly flushing the constrained-state cache. The hit rate becomes very low and mask construction cost rises. As noted earlier, this is the only source of the "more traffic means slower" effect.

Three mitigation options, ordered from least to most invasive:

• Enlarge the cache — the simplest option, and usually it solves most cases. Watch the hit rate and grow the cache until the hit rate stabilizes.
• Partition the cache per tenant — give large customers a dedicated cache. Suitable when there are few customers but traffic is concentrated.
• Tier schemas by frequency — count which schemas are hot and make their compiled artifacts resident (never evicted), while low-frequency ones go through LRU. This is the most effective option, because the distribution of cache benefit is inherently extremely skewed.

13.3 Canary rollout and rollback

Canarying constrained decoding is subtler than canarying an ordinary prompt change, because it changes both the format and the content distribution. A reasonable rollout design is:

1. Shadow mode — run the "old path" and the "new path" side by side on live traffic (the new path only records its results and does not return them), and compare two metrics: format success rate, field completeness rate, and the final correctness rate of the downstream task. This step can surface hidden problems such as "the format success rate went up but downstream quality got worse."
2. Small-traffic switchover — route 5% of traffic and observe P99 latency, error rate, and the three quality metrics above.
3. Full rollout — make sure to keep a one-click switch back to the "prompt + repair" path. That path always exists, so rolling back requires no redeployment.

Many people skip the shadow-mode step for the reason that it "wastes twice the compute." But it is the only way to discover that constraints have degraded content without affecting users, and this is exactly the class of problem that is hardest to detect after the fact in production — because it shows up as a slow degradation of downstream data quality, and typically nobody notices for weeks.

13.4 A checklist of observability fields

Turning the following fields into structured logs or metrics is the foundation of keeping this system maintainable:

Field	Meaning	Healthy range	What an anomaly indicates
schema_compile_ms	Compilation time	Should be close to 0 on a cache hit	High values mean cache misses or an oversized schema
grammar_cache_hit_rate	Grammar cache hit rate	Higher is better	A drop means the schemas in traffic are becoming more scattered
mask_p50 / mask_p99	Mask construction cost	P99 being much larger than P50 is normal	Both being high means there is no state cache
clipped_mass_avg	Average probability mass clipped away	Lower is better	High values mean the constraints are too tight, and output quality will drop
format_first_pass	First-pass format success rate	Should be close to 1	A drop means the constraint layer has failed
repair_rate	Share of outputs that need repair	Lower is better	A rise is an early warning signal
semantic_pass_rate	Semantic validity rate	Depends on the business	A drop means content quality has a problem
attempts_histogram	Distribution of attempt counts	Mass landing on the cap means the success rate is too low	Time to tune the constraints instead of adding retries

In this table, the last two rows plus clipped_mass_avg deserve the highest priority. The first two map directly to the health of layer 1 and layer 2, while clipped_mass_avg is the only signal that gives early warning of "the constraints are too tight." A system that monitors only the parse failure rate will miss exactly the biggest risk that constrained decoding introduces — because constraints that are too tight do not cause parse failures, they only make the content worse.

14. Moving Semantic Constraints into the Decoding Phase

One boundary has been mentioned repeatedly: semantic constraints (cross-field consistency, property dependencies, value ranges, referential integrity) can only be handled by post-sampling validation, because they require context and sometimes external state. That boundary is being broken in 2026, so it is worth knowing where the frontier is and whether you should follow it.

14.1 Reverse constraints: letting a regular expression drive generation

Traditional constrained decoding is "forward": give it a grammar and move ahead. Libraries such as guidance-ai show a different idea: use a regular expression and a set of capture groups as the constraint, then work backwards to infer what the preceding text must look like in order to match it. For example, if you want the model's answer to contain an email address matching a certain pattern, instead of constraining "the email field in the JSON," you use the regular expression itself as the constraint and have the decoder decide at every step whether continuing generation can still match it.

This capability goes beyond the expressive power of context-free grammars, because what it does is suffix-based lookahead — something neither FSM nor GBNF can do (they only look at the prefix, not at the goal). The price is much higher implementation complexity: every step requires deciding whether "the current suffix plus any continuation can still match," which is a decidability problem when the regular expression contains backtracking and is undecidable in the general case, so the implementation must use a conservative approximation.

14.2 State-feedback constraints: injecting database candidates into the mask

The second direction is closer to real business problems: moving "the value of this field must come from the current contents of this table in the database" into the decoding phase. The approach is — before entering the status field, query the database once and inject the set of legal status values into the allowed token set for that position.

This looks very elegant, but there are three hard engineering problems, each one enough to make you stop:

• External latency enters the decode loop. Decoding was originally pure computation; now every few generated tokens means waiting on a database. The fix is prefetching (query everything you need during the prefill phase or earlier), but that requires you to know in advance which branch the model will take — in a schema with anyOf, that is not always possible.
• Cache consistency. The candidate set may already have changed during generation (someone changed a configuration). Either you accept that window or you introduce versioned tokens — the latter significantly increases complexity.
• Multi-turn state. In a conversational setting, a field constrained in the previous turn — is it constrained again this turn? Did the schema change? The state machine has to be maintained across turns.

So my judgement is this: for cases where the value range comes from a fixed small enumeration, the schema's enum is enough and state feedback is unnecessary; the scenario where state-feedback constraints are genuinely irreplaceable is "the value space is too large to enumerate but can be queried from an external system," for example "this user ID must exist in our tenant list" — and even then, post-sampling validation is often just as effective (querying the database once to check whether the ID exists costs far less than moving the query into the decode loop).

14.3 A pragmatic trade-off

To decide whether to adopt state-feedback constraints, I use three questions: can the candidate set genuinely not be enumerated (if it can be enumerated, it is not worth doing); is the cost of post-sampling validation unacceptable (if one query takes a few milliseconds, it is perfectly acceptable); and are the consequences of a constraint violation severe enough that validation cannot serve as a backstop (high-risk parameters in tool calls qualify, most others do not).

If two of the three questions answer "no," do not do it. The value of this rule is that it lets you resist the temptation to "move everything into the decoding phase," which sounds very advanced. The most practical form of constrained decoding is still that plain version that only handles grammar and literals, leaving semantics and state to the validation layer.

This is also a miniature of the whole article: the optimal answer in a technology choice is usually not the most capable layer, but the one that is "good enough at the lowest cost." Constrained decoding became the de facto standard for structured generation in 2026 not because it is the strongest, but because it eliminates the largest class of failures at the lowest cost. Once you understand that, the rest is detail.

15. Why Not Some Other Approach

Before committing to constrained decoding, a qualified proposal should first answer the question "why not use a cheaper method?" This section lays out the main alternatives for comparison, because they are not mutually exclusive and their applicability boundaries are actually quite clear.

15.1 Regular-expression post-processing: cheap, but it silently drops data

The most common alternative is to let the model output freely and then extract with a regular expression — dig the JSON out of a block of text, and if you cannot dig it out use a fault-tolerant parser, and if that still fails grab a few key fields. It costs almost nothing and requires no engine-side changes, so many teams ship their first version this way.

The fatal flaw of this route is that it necessarily drops data silently. A regular expression can only handle the shapes you anticipated; when the model's output diverges substantially from what you expected, the regular expression will "try its best" to match something that looks reasonable but is factually wrong — for example, pulling a time field with an incomplete value range out of a truncated JSON. What's more dangerous is that a failed extraction and a successful-but-wrong extraction look identical in the return value (both are just a dictionary), so downstream code cannot tell them apart.

So the criterion is clear: if a missing or wrong field causes business loss, regular-expression post-processing is out; if you are merely turning a block of text into a few labels for human browsing, it is perfectly adequate.

15.2 Few-shot strong constraints: improves probability, not guarantees

Another common approach is to "teach" the format to the model with a large number of examples, such as giving it 20 JSON examples covering all sorts of edge cases. This does significantly improve format adherence, and its real value lies mainly in two things: it teaches the model how to name and fill things (examples are more concrete than a schema description), and it aligns the model's output distribution with your expectations.

But the two things it cannot do matter just as much: it cannot guarantee validity (with few-shot examples the model still fails, especially on combinations it has never seen), and it cannot guarantee completeness (few-shot examples have a hard time covering "all required fields are present," unless the examples happen to contain them all).

The sensible division of labor is this: few-shot examples handle "how the content is filled in," while constraints handle "the structure is definitely correct." They are not substitutes. When you see a team trying to solve a parse failure rate by adding more examples, that is a signal: a structural problem has been misdiagnosed as a content problem.

15.3 Fine-tuning: it solves a different problem

Fine-tuning does improve how well a model follows a format, which has been demonstrated both in academia and in industry. But treating it as a means of "guaranteeing valid output" has several structural problems:

• It does not guarantee anything. After fine-tuning the model still makes mistakes, just fewer of them. To push the failure rate from a few percentage points down to near zero you still need rules, retries, and other engineering measures — and those engineering measures can get you to near zero on their own, without fine-tuning.
• It has a maintenance cost. Every time the schema changes you have to retrain or at least redo preference alignment, which stands in sharp contrast to constrained decoding, where "change one line of the schema and you're done."
• It has a drift cost. Base model upgrades, changes in instruction-following ability, and the timeliness of training data all produce unpredictable fluctuations in format adherence, and it is very hard to attribute them.
• It does not conflict with constraints but does not complement them either. Constrained decoding guarantees structure; fine-tuning improves content quality. If structure is your bottleneck, do not fine-tune; if content quality is (for example, the reasoning for a classification is poorly written), fine-tuning is the right investment.

A common misjudgement is "we already fine-tuned, why do we still need constraints" — the answer is that fine-tuning solves high probability, while constraints solve certainty. For scenarios where "the output goes straight into a database," only certainty is good enough.

15.4 Comparison of approaches

Approach	Structural guarantee	Content quality	Cost to change	Failure visibility	Best for
Prompt only	None	Average	Very low	Poor	Prototypes, low-stakes scenarios
Regular-expression post-processing	None (and silent)	Average	Low	Poor	Label extraction, human browsing
Few-shot	Weak (improves probability)	Good	Low	Average	Used together with constraints
Fine-tuning	Weak (still probabilistic)	Good	High	Average	When content quality is the bottleneck
Constrained decoding	Strong (structural)	Above average (more complete fields)	Medium	Good	Scenarios that need hard guarantees
Constraints + validation + retries	Strong + semantic	Good	Medium to high	Good	The default choice for production systems

Notice the wording of the last column: "the default choice for production systems" is a combination, not a single point. Constrained decoding alone can guarantee structure but cannot catch semantics; validation alone can catch semantics but wastes one generation. What is worth doing in production is stacking them, and using monitoring data to decide which layer deserves further investment.

16. Five Scenarios Broken Down One by One

Abstract methodology easily leads people to believe that all scenarios are alike. This section picks five typical scenarios and looks at how the same three-layer architecture lands under different constraints, focusing on where the differences lie.

16.1 Invoice extraction

The schema is highly fixed (a few amount fields, a date, an invoice number), but the input is dirty text produced by OCR. This is the scenario best suited to constrained decoding, because the schema is extremely deterministic.

The key judgement: amount fields have value-range constraints (they cannot be negative, they cannot exceed a reasonable range), but such constraints cannot be expressed in a grammar — GBNF can constrain "this is a number," but cannot constrain "this number is less than one million." So the validation layer has to implement the business rules itself: amount > 0, total = sum of the line items, date within a sensible interval. These rules are also frequent failure points, so on a retry you should feed the specific error ("total does not equal sum of items") back to the model.

Another easily overlooked point: a number in OCR text may be O08.00 (the letter O masquerading as the digit 0). Constrained decoding will not stop the model from copying O08.00 verbatim into a string field — because the schema says it is a string, so it is legal. This can only be handled by upstream input cleaning, or by having the validator reject and retry.

16.2 Classification plus rationale

The model is asked to produce a classification result together with a rationale. This is a special use of constrained decoding: constraints guarantee the structure, while the reasoning itself is left unconstrained. There are two approaches:

• Reason freely first, then constrain the output — use one generation to produce "analysis … the final classification is X," then use constrained decoding to produce the structured result. Quality is better, but it costs one extra generation or a longer output.
• A single generation with a mixed structure — use a loose grammar (key-value pairs), put the analysis text in a reasoning field and the classification in a label field, with the label constrained by an enum. It is cheaper, but you have to ensure the model writes reasoning before the label — which is achieved through a field-order constraint (the grammar requires reasoning to appear before label).

"Field-order constraint" is a practical trick in the second approach that many people do not know about: constraints do not only restrict values, they can restrict order, and order has a substantial influence on how the model thinks. If you write the conclusion first and the rationale second, the rationale often degenerates into a rationalization of the conclusion; if you write the rationale first and the conclusion second, the rationale is far more likely to be genuine reasoning.

16.3 SQL generation

This is the scenario with the highest cost of failure, so all three layers are turned on, and layer 3 is basically closed off (better not to execute than to execute a silently wrong query).

The division of labor among the constraints is interesting: table and column names use enum constraints (this is the most valuable part, it eliminates the vast majority of hallucinated table names), the keyword sequence relies on the grammar (the order of SELECT ... FROM ... WHERE), and semantic correctness is left entirely to validation (EXPLAIN to inspect the execution plan, read-only checks, an upper bound on result rows).

There is one practical constraint worth thinking through in advance: the table-name enum may be very long. For a database with five hundred tables, putting all the table names into the grammar bloats the grammar to the point where it is hard to compile. At that point you should do it in two stages — in the first stage use a few tokens to let the model pick a table (you can retrieve first to narrow the candidates down to a dozen or so), and in the second stage apply full constraints based on the selected table. This "narrow the candidates first, then constrain hard" pattern applies to every scenario with a large candidate space.

16.4 Configuration files and workflow generation

When generating things like YAML or workflow definitions, constrained decoding has one special source of value: indentation and punctuation. YAML's indentation hierarchy is the core of format correctness, and it is extremely easy to get wrong — especially when the model is deep inside a nested structure. Letting constraints guarantee indentation is far more reliable than asking the model to "watch the indentation."

Another key point: output of this kind is usually read and edited by humans, so beyond legality you also have to consider readability. Constrained decoding can be counterproductive here — overly tight constraints (forcing a fixed key order, forbidding comments, forbidding inline spaces) make the generated configuration long and hard to read. So the granularity of the constraints should be decided by "will a human hand-edit this": if a human will hand-edit it, do not constrain the expressive freedom beyond the format (key order, whether extra keys are allowed), otherwise human edits and the next model generation will fight each other.

16.5 Structured state in multi-turn dialogue

If the structured output is not the endpoint but is going to enter the next turn's context (for example, an assistant writes the extracted structured slots back into the conversation history), there is an extra requirement: the constrained output stays in the context for a long time, so it must be friendly to subsequent turns.

This means order and readability matter more than legality — if the JSON is a single compressed line of several thousand characters, it will consume a lot of tokens and make it hard for the model to reference specific fields within it. The approach is to optimize the output within what the constraints allow: keep the field order stable (consistent with the schema), control the indentation (allow newlines in the constraints rather than forbidding them), and avoid redundant optional fields. So "constrained decoding plus output aesthetics" is a real engineering problem, and it is worth adding a light formatting pass after generation.

17. Special Issues for Chinese and Multilingual Scenarios

Teams running structured generation on Chinese-facing products run into concrete problems that English-language material never mentions, and they deserve a section of their own.

17.1 Chinese quotation marks are the most common failure source

When a model outputs JSON and wraps the values inside strings with Chinese quotation marks ("名称"："北京"), that is an extremely common failure mode — because in Chinese training contexts quotation marks are often full-width to begin with. This kind of breakage cannot be accepted by a strict parser, but it can be repaired by a fault-tolerant parser.

A stronger approach is to eliminate it at the constraint layer: if your schema specifies that strings use double quotes, explicitly forbid full-width quotation marks in the character set constraint. GBNF character classes support this kind of precise exclusion, and the cost is only a slightly more complex character set definition. If your schema allows strings to contain quotation marks, you can consider using an enum to pin down the allowed values — which solves both the full-width quotation mark problem and enum violations at the same time.

17.2 Key names for Chinese fields

Chinese key names themselves are fine (JSON supports them fully), but there are two details to watch: the full-width colon (： instead of :) breaks the grammar, which a character set constraint can cure; and spaces inside key names — if the model adds a space after generation (" 用户名"), validation will fail, whereas constraints can lock the key name precisely to a single literal and eliminate this risk entirely.

This illustrates a general rule: key names are where constraints have the best cost-performance ratio. Because key names are finite, enumerable, and completely determined, locking them with const costs almost nothing yet eliminates an entire class of failures (misspelled key names, half-width versus full-width characters, extra spaces, order drift).

17.3 The length budget in Chinese

Estimating max_tokens has a trap in the Chinese case: the same Chinese character may be one token under one tokenizer and two tokens under another, and the token density of Chinese text is usually lower than English with the same information content (Chinese is more compressed). So extrapolating token count from character count carries a systematic bias.

There is only one reliable approach: use the real tokenizer to gather statistics over real samples, take the P99 of the output length, and multiply by a safety factor. Do not estimate with "character count × some coefficient." The truncation failure described earlier is just as common in Chinese scenarios, and because Chinese has high information density, truncation often happens at a structurally later position, which makes the probability of recovering it through repair even lower.

17.4 Output language constraints

If the business requires that "field values must be in Chinese," that is a semantic constraint rather than a syntactic one — syntactically, both "北京" and "Beijing" are legal strings. What you can do is:

• State the language requirement explicitly in the prompt (this is a scenario where the prompt genuinely works);
• Do language detection in the validation layer and retry when it fails (simple and effective);
• Use a GBNF character class to constrain "CJK characters only" (strongest, but it excludes mixed-in digits and English proper nouns, so it is rarely applicable in practice).

My recommendation is the second option. Language detection plus retry costs almost nothing and has high coverage, whereas the practical usability of character class constraints is poor — because real text almost always has digits, English abbreviations, and punctuation that need to be mixed in. Constraints pay off far less here than they do at the structural level.

18. Synthesizing Data with Constrained Decoding

Constrained decoding has one use that is often overlooked: it is the best tool for generating high-quality training and evaluation data, because it guarantees the format and lets the model (or the rules) focus on the content.

18.1 Format is guaranteed by constraints, quality is controlled by sampling

The traditional data synthesis pipeline is "have the model output in the given format → parse it with a script → discard whatever fails to parse." There is an implicit efficiency loss here: a large amount of compute is spent on samples that will be discarded. With constrained decoding that ratio approaches zero, and the same compute yields far more usable samples. This is its most direct value as a data tool.

18.2 Actively covering edge cases

Constrained decoding makes "generating edge samples on demand" easy: want a sample where "all optional fields are missing"? Temporarily change the schema's optional fields into required ones. Want a sample with "deep nesting"? Generate a deep tree structure. Letting the model generate freely makes it very hard to obtain these samples in a targeted way.

There is an even more valuable use: define the evaluation set with the schema's constraint space itself. Traverse the Cartesian product of the enumerations and generate a batch of samples covering different parameter combinations, and you will learn which combinations the model tends to get wrong — especially when the combination of two fields carries an implicit logical relationship.

18.3 The distribution shift you must watch for

Text generated under constraints has one systematic flaw: its distribution is narrower than that of real output. Constraints guarantee the format, and that in turn removes the model's randomness in the format, so the generated samples end up too regular along some dimensions (for example, the distribution over an enum field may be uneven, because conditioning under constraints shifts the probability of each value).

If you fine-tune on such data, the model inherits that bias. So the correct practice is this: use constraint-generated data for "positive samples" and edge cases, while necessarily mixing in real unconstrained output (even if it has a broken format). Otherwise the model will lose its robustness to dirty input. This is the same principle as "you cannot only test the happy path."

19. Common Anti-Patterns

Finally, let us collect five anti-patterns that recur in real projects and all point back to architectural problems. They differ from the "pitfalls" in chapter 11 — the pitfalls are implementation-detail bugs, whereas these five are wrong choices at the level of solution design.

• Fixing JSON with a regular expression instead of adding constraints. The symptom is that the patching logic gets more and more complex and the regular expression keeps getting longer. The criterion: if your regular expression is over twenty lines and still changing often, it is doing constraints' job, and the problem should be fixed elsewhere.
• Packing the full schema into the prompt without adding constraints. The symptom is a growing prompt (token cost goes up) while format adherence does not improve. The role of the schema in a prompt is to tell the model what the fields mean; guaranteeing the format belongs to constraints. Mixing the two leads to duplicated effort.
• Unbounded retries. The symptom is a claim of "99.9% success rate" where nobody can say what the latency is for that remaining 0.1%. The criterion: the retry count must be derived from the latency budget, not picked out of thin air.
• Enabling degraded parsing on high-risk operations. The symptom is "for the sake of availability" applying degradation to SQL or tool calls. The criterion: degrade only where the cost of degrading a failure is lower than the cost of the failure itself. Executing with the wrong parameters is far more dangerous than erroring out.
• Designing the schema as a pile of free-form strings "for flexibility." The symptom is using type: string "to accommodate all cases," so the constraint degrades into "must be a JSON string" and does almost nothing. This is the best illustration of the fact that the ceiling on constraint benefit is determined by schema quality — however strong constrained decoding is, it cannot force the contents of a free-form string.

The last of these five is worth expanding on: if 80% of the fields in your schema are strings, whether you use constrained decoding barely affects quality at all, because you have only constrained the outermost brackets. The real benefit comes from designing a more specific schema — enum instead of string, nested objects instead of stuffing things into one JSON string, required fields instead of letting the model decide for itself. The upper bound of constrained decoding's capability equals the information content of your schema.

Once you understand this, you know the right order of technical investment: design the schema well first (lowest cost, largest benefit), then bring in constrained decoding to guarantee structure, and finally use validation to catch semantics. The reverse order — adopting constrained decoding first, then discovering that the schema is too loose for the constraints to have anything to constrain — is the most common waste of time.

20. Verifying the Correctness of the Constraint System

Constrained decoding implementations are prone to one dangerous state: it looks like it is working and the format success rate is fine, but the masks are wrong under certain states. This kind of bug does not crash the system immediately, it just pushes the failure rate up slightly and is very hard to attribute. This section gives a set of verification methods that proactively surface such problems.

20.1 Invariant testing: what states are "impossible"

The most effective approach is to define invariants and then check them over large-scale random generation. A constraint system has several very useful invariants:

• Format invariant: the full concatenation of any allowed token sequence must be valid JSON (or conform to the target grammar). This is the most core one, and you can assert it directly on every generation result in tests.
• Monotonicity invariant: an output that has already passed validation should still be in a "completable" state after appending any legal token — that is, it must not enter a dead-end state where "no matter what follows, it cannot be closed."
• Reachability invariant: every rule declared in the grammar must be reachable from root. If a rule is never used, the schema and the grammar are out of sync (either extra rules were generated or a reference is missing).

The second one, the "dead-end state," is worth discussing on its own: a constrained grammar may allow entering a state that cannot be completed. For example, an object defines a required field, but some branch in the grammar makes it so that after entering that branch there is no legal continuation. In that case the model generates partway, the mask then forbids everything, and the output stops abruptly — it shows up as "the output is inexplicably short." This kind of bug is extremely hard to find by manual testing, but random fuzzing can catch it: generate a few tens of thousands of random samples and check whether each one is complete and terminates normally.

20.2 Property-based testing: treating the schema as an assertion source

A more valuable class of tests derives the assertions directly from the schema instead of hand-writing expected values. For each generation result, automatically check:

1. the output is valid JSON;
2. the output passes JSON Schema validation (this step turns the schema into the source of the test assertions);
3. all required fields are present and non-null;
4. all enum fields take values within the enumeration;
5. all fields ending in _at or _date can be parsed as dates.

The third and fourth checks are especially valuable, because they can fail even when the constraint layer is perfectly correct — and that is exactly their value: what they test is "whether the semantics of the schema were actually expressed," not "whether the grammar is correct." A test suite that only does grammar testing will miss problems like "a required field was wrongly turned into an optional one by the constraint layer."

20.3 Differential testing: constrained version versus free version

The third useful technique is differential testing: take the same batch of inputs, generate them once with constrained decoding and once with free generation (no constraints applied, but the prompt requirements retained), and compare the behavioral differences.

Focus on three classes of difference, each with a clear meaning:

• The constrained version is illegal while the free version is legal — this is normal (that is the point of constraints), but if the ratio is high, the constraints and the model's tendencies are in serious conflict.
• The constrained version is legal but the content differs greatly from the free version — worth sampling manually for review. If the difference is widespread, it means the constraints are substantially changing content selection (rather than only backing up the format), which may be exactly the effect you want, or it may mean the constraints are too tight.
• The difference in field completeness rate — this is the metric most worth watching. If the constrained version has a markedly higher completeness rate for required fields, the core value of constraints is being realized.

Differential testing costs one extra share of compute, but it gives you a baseline — and without a baseline you cannot answer the question "are constraints actually helping, and is it worth it." Many projects adopt constrained decoding because "it seemed to get better after we turned it on," yet nobody ever ran this comparison.

21. The Lifecycle of Compiled Artifacts

Compiled artifacts (GBNF text, DFA tables, mask indices) are stateful, they occupy memory, and they are bound to the engine version. Very few people write down these operational details, yet they determine whether a multi-tenant service can run stably over the long term.

21.1 Three kinds of state, each with its own lifecycle

State	Contents	Memory scale	Invalidated when	Where it lives
Grammar text	GBNF string / JSON	Tens of KB	The schema changes	Disk cache + in-memory LRU
Compiled structure	Rule tables, DFA state tables	Hundreds of KB to a few MB	The schema changes, or the engine version changes	Engine process memory
Mask state cache	The allowed token set for each grammar state	Up to tens of MB	Almost never invalidated, only evicted by LRU	Engine process memory, shareable across requests

Keeping these three layers distinct matters, because their invalidation logic and memory characteristics are completely different. One common incident goes like this: the schema changed, but the cache key for the compiled structure did not include the schema version, so old and new grammars got mixed — the symptom is fragments in the output that both the old and the new rule set allow. This kind of problem can be "fixed" by restarting the service, so it is easily misdiagnosed as an unstable environment.

21.2 Warmup: so the first request is not painfully slow

Because the compiled structure is loaded lazily, the first few requests after a service start are noticeably slower (they compile on the fly and build state on the fly). During a traffic peak that does not matter much, but in a cold start, at low traffic, or right after scaling, it becomes latency the user can perceive.

The approach is simple: at startup, run one warmup pass over a predefined set of hot schemas — generate an extremely short output that walks the entire common grammar path, so the caches become hot. The cost is a few tens of milliseconds at startup; the benefit is completely eliminating the long tail of cold-start latency. How to decide whether it is needed: if your P99 latency curve is clearly elevated in the first few minutes after the service starts, that is your evidence.

21.3 Multiple models and multiple engines coexisting

If your service offers several models (different tokenizers) at the same time, or connects to several inference backends at the same time, compiled artifacts cannot be shared. Chapter 13 already explained that the tokenizer version must go into the cache key; here is a finer point: different engines may produce different grammars for the same schema (for example, one supports a certain keyword and another does not), so the cache key must include "engine + version + whether a given compilation option is enabled."

One pitfall you will actually hit in practice: if the cache key only uses the schema hash, and part of the traffic is switched to a new engine during a canary release, then the old engine's grammar text may be used with the new engine's mask logic, producing abnormal behavior. During a canary release the cache must be isolated per engine instance, and the safest approach is for each engine instance to have its own cache namespace.

22. Cost Accounting

Constrained decoding is a combination of one-time investment and ongoing overhead, and many teams only count one side of it when making the decision. This section lays out the three blocks of cost clearly.

22.1 One-time costs

The cold start and warmup of the grammar cache are the main one-time costs. Their scale is usually tens to hundreds of milliseconds and can be ignored. The real one-time cost is the investment in schema design — rewriting a loose schema into a form that can be constrained effectively is often the largest single block of work in the entire project, and this investment never amortizes into "free."

22.2 Ongoing costs: latency and throughput

The cost on the latency side has already been accounted for in chapter 10. The cost on the throughput side is the CPU consumed by mask construction, and under high concurrency it shows up as "CPU becomes the bottleneck." The mitigations, ranked by cost-performance ratio:

1. Guarantee state cache hits — this yields the most, because on a cache hit the mask costs almost nothing.
2. Turn masks into GPU tensors — the second best, and it depends on whether your engine supports it.
3. Add CPU cores — the most direct but the most brute-force approach; if masks are pure CPU work, adding cores often takes effect faster than changing code.

22.3 On the benefit side: do not count only the parse rate

The benefit of constrained decoding is easily underestimated, because its main benefit is not "we retried fewer times" but rather:

• downstream data quality improves because field completeness rises (this cannot be measured in retry counts);
• downstream code gets simpler — downstream no longer needs fault-tolerant parsing logic, no longer has to handle "possibly corrupted" input, and no longer needs exception branches in its test cases;
• failures become observable — they go from "occasional and hard to reproduce" to "countable, alertable, and attributable," and at the level of team collaboration this benefit is larger than it is at the level of performance.

The easiest mistake when making the decision is to compute the ROI of constrained decoding purely as "reduced retries." The correct calculation is this: count the maintenance cost of downstream fault-tolerant code, the token cost burned by retries, and the cost of investigating bad data after it has entered the system, and put all of them on the benefit side. The resulting numbers are usually positive, and the margin is larger than you would imagine.

Conversely, if your scenario means downstream has to write full fault-tolerant logic anyway (because the format was never important), then the ROI of constrained decoding really is low — and that is one of the few legitimate reasons not to adopt it.

23. Combining Constraints with Retrieval and Long Contexts

Structured generation rarely exists on its own; it usually shows up together with retrieval or with long-document processing. There are a few points worth noting in these two combinations.

23.1 Driving retrieval in reverse from the schema

A reverse idea is quite valuable: decide the schema first, then use the schema to guide what should be retrieved. If the schema has an "invoice number" field, retrieval should go looking for invoice numbers; if the schema has a "default clause" field, retrieval should specifically hunt for that content. This is far more accurate than "retrieve a big pile of text and let the model find things itself," because the schema is itself an explicit statement of what we care about.

Going further: if your schema comes from a database structure or an interface definition, then it is the most authoritative requirements list you have, and it can be used to audit the retrieval strategy — check whether every field has a corresponding retrieval source, and you will often discover schema design flaws where "the model simply has no way to fill this field."

23.2 Structured extraction over long documents

When extracting structured information from a long document, constrained decoding has a special role: it lets "extraction" and "summarization" be separated. You can first let the model take notes section by section in a free format (this section is unconstrained, because a free note format helps thinking), and then run one strongly constrained extraction over the notes (strict format, low cost).

This is better than "one call that strongly constrains the extraction of the whole document," because strong constraints continuously apply formatting pressure during a long generation, and that pressure crowds out the model's attention on content. Splitting it into two stages keeps the model's thinking quality in the first stage and guarantees usable output in the second. This may cost slightly more tokens (one extra call), but extraction quality is usually better — especially for long documents.

23.3 Should the retrieval results themselves be constrained

If your schema contains a "citation source" field (for example, requiring every extracted value to carry its origin in the source text), then that field must be constrained to a set of genuinely existing sources — that is, to the id list of the retrieval results. This is exactly the typical scenario for the "state-feedback constraints" described in chapter 14.

But I suggest you do not start with state feedback; use validation instead: let the model output citation ids, then check in code whether those ids are in the retrieval results. One check is an O(1) operation, whereas injecting the retrieval results into the decode mask brings in all of the complexity and latency problems described in chapter 14. Unless you find that the proportion of the model fabricating source ids is high enough to be unacceptable, it is not worth it.

24. A Rollout Roadmap from Zero to One

If your team is not using constrained decoding yet, this section gives a staged path. The core principle is this: every step must be able to deliver value on its own, and you must not over-invest in intermediate states for the sake of the final form.

Stage one: start with validation (doable on day one)

Add no constraints at all, just do three things: record the raw output, run parse-repair, run schema validation, and report the three failure rates as metrics. The deliverable of this step is data — you find out what your real format success rate is and what the failures look like.

This step is often skipped for the reason that "validation cannot fix anything." But without this set of data every later decision is blind: you do not know whether to adopt constraints, you do not know how tight the constraints should be, you do not know whether retries are worth it. The instrumentation costs one day; not doing it costs weeks of wrong decisions.

Stage two: fix max_tokens and the schema (week one)

Once you have the failing samples, you will find that a good share of the failures are not constraint problems at all but budget and schema problems: max_tokens is too small and causes truncation, or the schema demands fields the model cannot fill (for example, demanding an exact number from a vague description).

Fixing these two things usually drops the failure rate substantially and costs no compute at all. The deliverable of this step is a real distribution of failure causes, and it decides where you should invest next.

Stage three: turn on provider-side structured output (weeks two to three)

If your provider supports schema-level structured output, turn it on. This step is the cheapest (change a few parameters) and the payoff is a large jump in format success rate. At the same time, tighten the schema — this is the step with the largest return, because the more specific the schema, the more can be forced.

What you need to verify carefully here is which schema keywords the provider actually supports. Treat it as a piece of research (documentation plus hands-on testing), because you need to know which constraints are enforced and which are left to validation. If you find the provider does not support a feature you need, that is the motivation for stage four.

Stage four: build your own constraints (as needed)

Building your own is only worth it when at least one of the following holds: (1) the provider does not support the constraint features you need; (2) you need to control the candidate set (for example, restricting the allowed values to the result of a live query); (3) you need streaming constraints; (4) you need control over the constraint implementation in order to optimize throughput.

The choice among self-built options was covered in the earlier chapters (engine built-in backend versus XGrammar versus Outlines versus guidance). The only key question in the selection is this: what does your schema look like, and how tight does it need to be? A loose schema works with anything; a compact schema is what makes a dedicated engine worthwhile.

Stage five: complete monitoring and regression (ongoing)

Following chapters 12 and 13, build up the three failure rates, compilation time, cache hit rate, and clipped probability mass, and turn the constraint-coverage test into CI. This step adds no new features, but it is the guarantee that keeps all of the previous investment effective.

There is one counterintuitive point about the order of these stages: stage three (provider-side structured output) usually yields far more than stage four (building your own constraints). Many teams skip the provider option and build their own, paying the full cost of engine integration without obtaining the corresponding benefit. Unless you have a clear requirement that only self-building can satisfy, use the provider's option first.

Another point is that stage two (fixing max_tokens and the schema) is broadly underestimated. It requires no new component, yet it often resolves most failures — because many "format problems" are really "you asked the model to output something it has no information to fill in." Do this step thoroughly and your expectations about the return of later steps will be accurate.

25. Five Difficulties with Complex Schemas

The schemas in the earlier chapters are mostly flat objects, whereas schemas in real projects are often far more awkward. This chapter picks five of the most common difficulties and explains whether each one breaks at compile time or at run time, and how to handle it correctly.

25.1 Discriminator fields and polymorphic objects

The most troublesome kind of structure is "the same field has a different sub-structure depending on the type," typically a discriminated union: an object has a type field whose value may be one of several enum values, and each value corresponds to a completely different set of properties. Logs, events, and protocol messages all belong to this category.

Compiling this kind of schema is not hard; the hard part is combinatorial explosion. If three types each have ten sub-fields and the compiler expands the grammar for every branch, the grammar size is roughly three times a single branch; if the number of types grows to ten, the size becomes ten times. Grammar size directly determines compile time and state-cache size, so you have to be careful when the number of types is large.

There are three ways to handle it. The first is to share a common prefix at the grammar level: write the discriminator field first, then let the following fields branch according to the discriminator value. The shared part of the grammar is then written once, which saves a good deal of size. The second is to limit the number of types usable at the same time: if all types rarely co-occur in the business, you can split the schema into a few small versions per scenario and use only two or three types at a time; this is the most practical option. The third is to give up on constraints and validate only: if there are simply too many types, constrain the discriminator field with an enum (this part has the largest benefit and the smallest cost) and leave the remaining fields to the validator.

If I had to choose, I would pick the third. Because the error rate on the discriminator field is far higher than on other fields — the model tends to pick a type that "looks reasonable" and will not systematically break the internal structure of a branch. Concentrating the constraint budget on the discriminator field and covering the rest with validation is the arrangement with the best cost-performance ratio.

25.2 Conditional fields: the hard part is semantics, not syntax

A conditional field is "whether a field exists, or what value it takes, depends on another field." The two most common forms: existence dependency — when one field is present, another must be present as well; and value dependency — when one field takes a specific value, the allowed range or existence of other fields changes.

At the grammar level, conditional fields can be partly expressed: if the dependency is "optional field two is only allowed to appear when field one equals some value," you can restrict that branch to a specific context in the grammar. The difficulty is that the compiler has to maintain path-dependent state — "which level am I on, and what are the values of the conditional fields along this path" — and that far exceeds a plain stack-based state machine.

So in a real implementation, conditional fields usually take one of two routes: either the condition is compiled into a set of concrete branches (which is equivalent to expanding the condition into finitely many cases, and explodes when there are many combinations), or it is handed to the validation layer. My recommendation is the latter, and to write the condition as an explicit validation rule, because it is a direct translation of a business rule to begin with, and it is easier to maintain in code than translated into a grammar.

There is an easily overlooked benefit here: putting the conditions into the validator turns this part of the business rules from "scattered across various callers" into "centralized in one place." In many teams the field dependency rules are scattered across different services and different places, so every rule change requires a global search; once they are centralized in one validation module, the blast radius of a change is obvious at a glance.

25.3 Union types and "any one of"

A union type (one of several possible types) is easy to implement syntactically: you just join the productions of the branches with a vertical bar. The real problem is that it makes "what to validate" ambiguous: if the model outputs the string branch while your code expects the object branch, should you report "type error" or "missing field"? And if several branches are legal, the type of the return value is unstable for downstream.

More troublesome still is downstream usability: an interface with an unstable return type pushes the burden of the type system onto every caller. Constrained decoding can guarantee "the output matches some branch," but it cannot guarantee "the output matches the branch you wanted." So if your business actually knows which branch to use, the correct approach is to pin it down in the schema instead of leaving a union for the model to choose from. Union types should only be used when "the branch genuinely can only be decided at run time."

This is an easily missed design principle: every union in a schema turns a deterministic problem into a probabilistic one. The probability that the model picks the wrong branch is not zero, and you can eliminate that possibility entirely at the schema level. If you can eliminate it, do not leave it in.

25.4 Depth and width of recursive structures

Recursive structures (a tree-shaped organization, nested comments, a directory tree) are easy to express in a grammar: you just write one self-referencing rule. The problem appears at generation time: the model does not know your depth limit and it may generate a deeply nested structure. If your downstream parser has a recursion depth limit (nearly every language does), an output that exceeds it will make downstream throw an exception outright.

There are three ways to handle it, from recommended to not recommended: first, check the depth after generation and treat exceeding the threshold as a failure, then retry or degrade — this is the most direct; second, unroll the recursion to a bounded number of levels at compile time and only allow five levels of nesting in the grammar, at the cost of limited flexibility; third, state the depth limit in the prompt, which basically does not work, because it is only a request.

Apart from depth there is the width problem: an array inside a recursive structure, if its length is unbounded, can end up with thousands of elements, which is a hazard both for parsing and for rendering. Again, a constraint like "at most one hundred elements" is hard to express at the grammar level (it requires counting), so the more practical approach is to limit the count in the validation layer, or to truncate during parsing and record it.

Incidentally, recursive structures carry another hidden risk: how they interact with the grammar state cache. Because different depths correspond to different states, deep nesting produces many states and the cache hit rate drops. If your business often handles deeply nested structures, it is worth observing the latency distribution of mask construction separately.

25.5 Huge enumerations

The last difficulty is an enumeration that is simply enormous — for example, a few thousand product names read from a database, or several hundred tags. Expressing such an enumeration as a constraint means "one giant vertical-bar-separated list," so the grammar size becomes very large, and every added value brings one more state.

Worse, the user experience suffers: when a value has never appeared in the training data, forcing it into the candidate set may leave the model with no idea how to write it (it does not know the value), and the result is a similar but wrong value — which is precisely the problem constraints were supposed to solve.

For huge enumerations, my advice is to switch to two stages: first use one ordinary generation to let the model express the intent (for example, saying "this is an infant formula product"), then map that intent to a specific legal value in code, and finally let the model choose from the narrowed candidate set. Separating "generating the value" from "choosing from the candidates" is far more reliable than asking the model to pick among thousands of values in one shot.

This pattern has broad applicability: label selection, entity linking, classification tagging — anything where "the candidate set is too large to enumerate for the model" fits. Its essence is using retrieval to replace enumeration.

26. The Streaming Path for Structured Output

Streaming and constraints conflict inherently, as noted earlier. This chapter lays out the viable paths, because a lot of products deliver structured output in streaming form and best practice in this area is still fairly immature.

26.1 Three basic routes

The first is pseudo-streaming: the server generates the complete result, then slices it and pushes the slices to the frontend in batches. This is the easiest to implement and the most stable, because it is essentially non-streaming generation plus a chunked push. Its user experience is actually not bad — because the consumer of a structured result is usually a program, and the frontend only renders after receiving it, so the necessity of pushing in slices was limited to begin with.

The second is real streaming plus prefix repair: push the generated portion to the frontend as it is produced, and let the frontend use a fault-tolerant parser to repair the incomplete prefix, yielding a "best effort" intermediate state. The benefit is that content appears on the first screen sooner; the cost is that the frontend parsing logic must handle arbitrary truncation, and the intermediate state will flicker.

The third is constrained chunked pushing: generate with constrained decoding while cutting out a complete unit at every grammar boundary (for example, when some field finishes) and pushing it. This route is the most elegant and the most complex to implement, because it requires the constraint engine to expose "the positions where it is currently safe to split," and most engines do not offer that capability.

26.2 An underestimated frontend problem

If your structured output is rendered chunk by chunk on the frontend, field order affects the rendering experience. Constraints can enforce the field order (a practical trick mentioned earlier), and the order should be arranged as "what the user cares about most appears first" rather than following the declaration order of the schema. This one small change can noticeably improve perceived performance in streaming scenarios.

The other frontend problem is placeholders for missing fields. Because constraints guarantee that required fields always appear, the frontend does not need to handle a "the field has not arrived yet" branch — this is a direct benefit constrained decoding brings to the frontend. Conversely, if the frontend already contains a lot of compatibility code for "the field might be missing," then the value of constrained decoding is not only improved backend reliability but also significantly simplified frontend code. This benefit is often overlooked in evaluations, yet it is real.

26.3 Timeouts and degradation in streaming scenarios

Streaming has one practical problem: constraints slow down the first token (the initial mask has to be computed), and if some state needs a cold start midway through generation, the output will show a noticeable pause. What the user sees is "it is stuck."

There are several mitigations: warm up common grammar states (use a very short synthetic input to walk the hot paths once); relax the constraints in streaming scenarios (guarantee only bracket balance and basic structure, without full schema validation, because a streaming prefix cannot be fully validated anyway); and set up stall alerts (if no new token arrives within a certain time, trigger degradation or an alert).

The last one deserves special emphasis: in streaming scenarios degraded parsing is more valuable than in non-streaming ones, because the user has already seen part of the content and throwing it all away is a bad experience. Parsing what has already been received in a fault-tolerant way and displaying it, together with a "generating" hint, is far better than erroring out. This is also the reasoning behind the "streaming UI rendering" row in chapter 11.

27. Multi-Candidate Generation and Selection

Constrained decoding has another underestimated use: it makes "generate several candidates and pick the best one" both feasible and cheap. The reason is simple — every candidate is guaranteed to be a valid structure, so you do not have to spend time on parse failures and can spend all of your compute on improving candidate quality.

27.1 Three mechanisms that can be stacked

The first is sampling several candidates. Sample a number of different outputs under the constraints; they are structurally equivalent and differ in content. This step is no different from ordinary diversity sampling, but the benefit is that you can filter using only structural validation, without parsing.

The second is scoring candidates with a validator and rules. Once you have several legal candidates, you can rank them with additional rules: field completeness, conformance to business rules, even style preferences. In essence this is a lightweight "rerank with rules" scheme that requires no model training.

The third is self-consistency voting. If some field frequently agrees across candidates, that field is probably "right." Using majority voting to decide the final result is more accurate than a single generation on many extraction tasks.

27.2 When multi-candidate generation is worth enabling

Multi-candidate generation is not free: its cost grows linearly. Generating three candidates costs three times the tokens of a single generation (and the constraint and validation overhead scales up as well). So whether to enable it depends on three judgements:

• Is the single-attempt success rate already high? If a single attempt already yields a semantic validity rate above ninety percent, multi-candidate adds little, because the remaining errors are mostly model capability problems rather than randomness.
• Is the cost of an error high? In an automatically executed chain, spending three times the cost for higher correctness usually pays off; in a batch pipeline for human review, it may not.
• Is there a cheaper means than multi-candidate generation? If the errors concentrate on one or two fields, a single retry often solves it, whereas multi-candidate is a global multiplication.

A common sweet spot is two to three candidates: the first candidate plus a few retries usually covers most scenarios at a controlled cost. Beyond five candidates the marginal benefit drops off rapidly while latency becomes unacceptable.

27.3 The difference between multi-candidate generation and retries

Retries and multi-candidate generation are often conflated, but they solve different problems. A retry is a remedy triggered on failure; it depends on the failure signal (validation did not pass), so it only costs extra when the first attempt genuinely fails, and it usually feeds the error message back to the model, which makes it more effective. Multi-candidate generation unconditionally produces several versions, so you pay three times the cost even when the first attempt was already right, and what you gain is making the act of "choosing" explicit.

So the recommended order is this: do retries properly first (they are cheap and carry feedback), then consider multi-candidate generation (it is expensive but can be ranked). Many teams do the reverse, adopting multi-candidate generation without a validator, which means they paid twice the cost for nothing.

27.4 Constraints make "choosing" measurable

Whether a multi-candidate approach can land depends on whether you can state clearly "which candidate is better." Without a basis for scoring, multi-candidate generation is just wasted compute. Constrained decoding contributes indirectly here: it confines the differences between candidates to content, so you can compare them with business rules without first doing structural normalization or format alignment. The comparison therefore becomes simple — just apply rule checks to the field values.

This benefit shows up even more clearly when the candidates contain free-text fields: without constraints, two candidates might use double quotes versus single quotes, or include a space versus not, and your comparison rules have to deal with that noise first; with constraints those differences do not exist.

28. Migrating from Prompt plus Repair to Constraints

Suppose your system today is "the prompt asks for JSON, retry when it fails." This chapter covers how to migrate and what most often goes wrong during the migration. A complete migration usually takes four steps, and each step can be verified and rolled back independently.

28.1 Step one: complete the observability before changing behavior

The first step of the migration is not to add constraints but to measure the current failure rate. Specifically you need to know: what is the first-pass format success rate, what is the post-repair success rate, what is the semantic validity rate, and how the three relate to each other. If the post-repair success rate is far above the first-pass rate, then what you are really relying on today is the repairer, which makes the migration even more valuable.

This step should also collect failure samples, especially the ones the repairer cannot fix — they usually point at two categories of problem: either max_tokens is too small and causes truncation, or the schema demands information the model cannot provide. Both are far cheaper to fix than introducing constraints, and they should be done first.

28.2 Step two: tighten the schema

The second step is to change the schema from "descriptive" to "executable." Specifically: change every field that can be expressed as an enumeration from a string into an enum; add length limits to every string whose length you can determine; turn scattered optional fields into explicitly required or optional fields; and split "one string stuffed with several concepts" into separate fields.

The benefit of this step is twofold: it raises the proportion of things constraints can force, and it directly raises the model's output quality (because the model no longer has to guess what to fill in). Many teams skip this step and go straight to constraints, find that there is very little for the constraints to force, and conclude — wrongly — that "constrained decoding does not work."

28.3 Step three: bring in constraints and keep a fallback

The third step is to bring in constraints, but you must keep the old path and put a switch on it. The reason was given in chapter 24: the new path changes both the format and the content distribution, so when something goes wrong you need to be able to roll back quickly, and you need to be able to compare the outputs of the two paths.

The easiest mistake here is changing too much at once: changing the schema, swapping the model, adding constraints, and adjusting the prompt simultaneously. The result is that when a metric moves, you cannot attribute it to any single change. The correct approach is to change one variable at a time and observe each change over a long enough window.

28.4 Step four: complete the validator

The last step is to complete the validator, because constraints only handle structure and semantics must be handled by validation. This step is often deferred, but it is the sign that the three-layer architecture genuinely holds together: the constraint layer guarantees structure, the validation layer guarantees semantics, and only the two together cover the vast majority of failures.

The recommended order when completing the validator is this: first write all business rules explicitly as validations (even if they do not take effect yet, just record the violation rate), observe the violation rate for a while, then add automatic rejection and retries for the frequent violations. Observe first and block later, because that avoids collateral damage — an overly aggressive validator may reject a large number of perfectly usable outputs.

28.5 The three most common problems during migration

The first is that the schema and the constraints fall out of sync. After changing the schema you forget to regenerate the constraints, so the constraints still enforce the old schema, which shows up as the model filling in a field it should have filled but being blocked by the mask, or the reverse. The defense is to include the schema hash in the cache key and to print the constraint version in use at startup.

The second is that max_tokens is not adjusted along with it. After tightening the schema, the output length usually changes (more fields, longer enum values). If you keep the old budget, new truncations appear. The defense is to feed output length statistics into monitoring.

The third is semantic inconsistency between the two paths. The old path (prompt plus repair) and the new path (constraints) may give different content for the same input. If downstream depends on the specific behavior of the old path (for example, the wording style of a certain field), the switchover brings surprises. The defense is to run both paths side by side during the canary period and to focus on content differences rather than only the format pass rate.

29. A Handbook of Failure Modes

This chapter organizes all the problems discussed in the article into a reference handbook, giving the symptom, the cause, and how to localize each one. Treat it as a troubleshooting manual rather than reading material.

29.1 The symptom is "the format is completely correct but the content is clearly wrong"

This is the category most likely to be misdiagnosed. Constrained decoding guarantees structure and business rules guarantee semantics; if you only did the former, you get output with perfect format and wrong content. The way to localize it is to look at the semantic validity rate metric — if it is clearly below the first-pass rate, the constraint layer is working correctly and the problem lies in the business rules or in model capability. The common real causes are: max_tokens is too small (the output was truncated but happened to be cut at a structurally complete position), the schema demands information the model cannot obtain (for example, an exact value from a vague description), and overly tight constraints pushing the model toward unreasonable values.

29.2 The symptom is "the output stops abruptly at some position"

This symptom is almost always a dead-end state: the constraints entered a state that cannot be completed, the mask forbade every subsequent token, and the output was forced to end. The cause is a flaw in the grammar — for example, some branch makes a required field unsatisfiable. To localize it, print the constraint's state stack when the output terminates; it usually shows directly what is missing. The fix is to add the rule for that branch, or to check after generation whether the required fields are all present.

Another possible cause is that the model decides to stop — if your implementation supports "the model may emit a terminator," and the grammar allows terminating in a non-complete state, you will see this symptom. So the legality of the terminator must be bound to the current state.

29.3 The symptom is "illegal characters appear occasionally (not every time)"

Intermittent illegal characters almost always mean the mask state was computed incorrectly. The most common causes are that the mask used the next state instead of the current one (or the reverse), or that state was wrongly shared across requests. To localize it, add an assertion: after applying the mask, check whether the full concatenation of all allowed tokens is still completable — this is a cheap and effective online check. The fix is to bind the state to the request and make sure state machine objects are never reused.

Another possible cause is a tokenizer issue with character boundaries: the textual form of a token straddles a grammar boundary. To localize it, take real generation samples, tokenize them in reverse, and check whether any token crosses a boundary.

29.4 The symptom is "restarting the service fixes it"

A restart fixing the problem usually points to cache pollution: there are stale entries in the cache of grammar text, compiled structures, or mask state. The common cause is an incomplete cache key (missing the schema version, the engine version, or the model identifier). The defense is to include all these dimensions in the cache key and to proactively clear the in-engine process-level caches on release. Problems that a restart fixes are almost all of this kind, which makes it a very useful diagnostic signal — if a restart helps, do not go digging through the code logic, check the cache first.

29.5 The symptom is "throughput actually drops under high concurrency"

This is the only negative scalability issue in constrained decoding. The root cause is that the CPU consumed by mask construction becomes a bottleneck under high concurrency, or that the schemas of multiple tenants are scattered enough to lower the state cache hit rate. To localize it, load test at low, medium, and high concurrency separately, observe at which inflection point the throughput curve starts to drop, and watch the cache hit rate at the same time. The fixes ranked by cost-performance ratio: first enlarge the cache and watch the hit rate; if the hit rate is already high but the CPU is still saturated, turn the masks into GPU tensors; only then consider adding CPU cores.

29.6 The symptom is "the retry budget is exhausted and it still fails"

This symptom means the retry strategy is masking a problem that retries cannot solve. If three retries all fail, then the problem is almost certainly not random — retries work because errors have randomness (multiple generations from the same input may differ). If the error is systematic (for example, the schema and the model's capability do not match), retrying ten thousand times will not succeed. The direction of the fix is to go back to the root cause: check whether max_tokens is insufficient, whether the schema demands something impossible, or whether the constraints are too tight. Increasing the retry count in this scenario is just burning money.

29.7 The symptom is "occasional format errors in production that cannot be reproduced"

If the failure rate is very low and hard to reproduce, the most effective move is not to add logging but to first degrade into observability: record the complete raw output of each failure (mind the redaction). Even if it only happens a few dozen times a day, two weeks will accumulate enough samples to localize the problem. This step has to come first, because without samples every hypothesis is blind. While accumulating samples, remember to record the context (model version, schema version, engine version), otherwise the samples cannot be compared.

29.8 A general troubleshooting order

When you hit any constraint-related problem, I suggest troubleshooting in this order: first confirm that the constraints are actually taking effect (use the inverse verification method — deliberately relax a constraint and see whether the output changes); then confirm that the scope of the constraints is correct (compare the schema with the generated grammar field by field and see whether some field was left out); then confirm that state management is correct (especially whether state was wrongly shared under concurrency); and only then suspect the model itself. The judgement behind this order is that problems with a constraint system are mostly deterministic, reproducible, and discoverable through code review, while model problems are probabilistic and the hardest to localize, so the cheap possibilities should be eliminated first.

30. The Boundary Between Constraints and Prompts

The earlier chapters repeatedly stated "the prompt is responsible for intent, constraints are responsible for structure, validation is responsible for semantics," but in concrete decisions this division is not always clear. This chapter turns it into a decision tree you can follow directly, solving the concrete confusion of "which layer does this thing belong in."

30.1 First question: what are the consequences of its failure

The first criterion is not a technical property but the consequences of failure. Consider the same mistake of "the model picked the wrong value": if the consequence is only one fewer label displayed in the interface, it belongs to the prompt and adjusting the prompt is the cheapest fix; if the consequence is executing a database operation that should not have been executed, it belongs to constraints plus validation and must be enforced in engineering.

This judgement matters because it prevents a common misinvestment: spending a lot of time tuning the prompt to make the model "choose more accurately." But the ceiling of an accuracy problem is often not set by the prompt — if the model lacks the information to make a correct choice, then no matter how the prompt is written it can only guess. When you hit this situation, the right question to ask is "what information is it missing," not "how should I phrase this."

30.2 Second question: can it be expressed precisely

The second criterion is expressibility. If a requirement can be written as "the value must belong to this finite set" or "this field must exist," it should go into constraints, because constraints can guarantee it one hundred percent of the time. If a requirement can only be described as "write more professionally," "be concise," "give a rationale," then it can only go into the prompt, because a grammar cannot express it.

This division has one important corollary: when you notice yourself repeatedly emphasizing a structural requirement in the prompt, you should move it into constraints immediately. The fact that you keep emphasizing it already means it is important, and handing an important structural requirement to a probabilistic mechanism is a mismatch. Every structural requirement you add to the prompt takes away attention from the content requirements.

30.3 Third question: what does it cost to remediate a violation

The third criterion is remediation cost. For the same error, if it can be detected automatically and remediated cheaply (for example, one retry fixes it), it can live in the validation layer; if remediation is expensive (for example, external side effects have already occurred), it must be blocked in advance by a constraint.

This judgement directly determines the switch that controls your degradation policy: only failures with a low remediation cost are allowed to degrade, and failures with a high remediation cost must be retried or failed outright. This is also the core basis of the decision table in chapter 9 — it looks like a scenario comparison table, but it is actually a concrete instantiation of this principle.

30.4 A three-question checklist you can use directly

Combining the three questions above, when you are unsure which layer something belongs in, ask them in this order:

1. How severe are the consequences of its failure? If they are not severe, it can stay in the prompt and be handled by repeated attempts.
2. Can it be expressed precisely as a value or existence constraint? If so, move it to the constraint layer — that is the most reliable position.
3. Is the remediation cost high? If not, hand it to the validation layer; if it is high, it must be blocked in advance by a constraint.

After the three questions you can almost always reach an answer, and the conclusion is often "move it up one layer" — that is, from the prompt to constraints, and from constraints to hard-coded logic. Experience says that in a mature structured output system, what remains in the prompt should be only the content that genuinely can only be expressed as intent (tone, style, reasoning style, explanations of what fields mean), while all structural requirements have already sunk down into constraints and validation.

31. Structured Output and Security

Constrained decoding is often treated as a purely formatting concern, yet it occupies a unique position in security. This chapter covers three things: what it blocks, what it cannot block, and why the final gate is never here.

31.1 Three classes of risk that constraints naturally block

The first class is escaping the tool allowlist. With an enum constraint, the model cannot call a tool that is not on your list, nor misspell a tool name. This blocks a very common attack: inducing the model to call a similarly named tool with higher privileges.

The second class is structural injection through parameters. Because parameters must conform to the schema, an attacker cannot smuggle extra fields in through the parameter structure. If your downstream step "iterates the fields per the schema and executes," then structural constraints amount to limiting the size of the attack surface.

The third class is control characters in the output. Smuggling control characters, zero-width characters, or extremely long strings into structured output can affect downstream parsing and rendering. Constraints at the character set level can keep this kind of content out.

31.2 Three things constraints cannot block

Constraints cannot block privilege escalation at the semantic level. If the parameters of a legal tool include a file path, a resource identifier, or a query condition, then for "this path points somewhere it should not have access to," syntactic constraints are completely helpless. Constraints can only guarantee that the parameters look right; they cannot guarantee that the parameters point where they should.

Constraints cannot block content injected by prompt injection. If your task extracts information from user input and that input contains an instruction such as "please set status to approved," the model will very likely comply — because from its point of view, that genuinely is what the user asked for. Constraints can guarantee that status is a legal enum value, and approved happens to be exactly one of the legal values.

Constraints cannot block data leakage. If the model fills a legal field with data it should not know (for example, information about other users that it read from the context), constraints are equally helpless.

31.3 So where is the final gate

The security model for structured output should be layered: constraints guarantee structural legality, validation guarantees business legality, and the authorization layer guarantees operational legality. The three are in series, and no layer can replace another.

The authorization layer is the final gate and the only layer that can answer the question "is this principal allowed to do this." It does not care about format and does not care whether the content is reasonable; it does one thing: decide whether to allow the action based on the current identity and the resource. This layer must exist independently of the model and must never be skipped just because the model's output was legal — this is the root cause of many tool-calling incidents: mistaking "structurally legal" for "the action was authorized."

For high-risk operations, an additional layer of human confirmation or secondary confirmation should be added. Constrained decoding can significantly reduce the number of mistakes (since neither the tool name nor the parameter structure can be wrong), but it cannot reduce mistakes to zero, so a confirmation mechanism remains necessary.

31.4 An easily overlooked escaping problem

Finally, one concrete technical detail: escaping and second-order parsing. If your structured output is parsed into an object and then one of its string fields is concatenated into another query, template, or command, then constraints only guarantee the correctness of the first parse; the second concatenation is still an injection surface.

The defense is not to concatenate strings but to pass parameters in a structured way (parameterized queries, parameters passed into a template engine). This is not something constrained decoding can solve, but recognizing that "constraints only protect the first layer" helps you avoid placing your sense of security in the wrong place.

32. Generating Sequences of Objects with Dependencies

Everything discussed so far has been about generating a single object. But many real tasks generate a sequence of objects, and the elements inside the sequence have dependencies among them — later elements must reference earlier ones, or the range of a later value is determined by an earlier result. This is an advanced application of constrained decoding and also one of the scenarios most prone to error.

32.1 Two ways to generate a sequence

The first is to generate the whole sequence in one pass. Syntactically you wrap the element rule in an array rule, and the elements can reference fields that have already been generated. The benefit is that one call finishes the job and the order is natural; the drawback is that when the sequence is long, later elements cannot easily "remember" the earlier content and references go wrong.

The second is to generate in several passes, one element at a time. Each call passes the results generated so far as context, and the model only has to produce the next element. The benefit is that the context is clear at every step and references are accurate; the drawback is the number of calls, the added latency, and the need to maintain a state machine to track progress.

The criterion is simple: if the sequence length is fixed and short (for example, no more than five), generate in one pass; if it is variable or long, generate in several passes. An additional benefit of several passes is that each step can be validated and retried independently, limiting the blast radius of a failure to a single step.

32.2 Order constraints: making dependencies executable

In either approach, the key is to express the dependencies. There are two ways: enforce the order at the grammar level (for example, the grammar requires a referenced field to appear only after it is defined), or check referential integrity at the validation layer.

Enforcement at the grammar level suits "structural order" better, such as language-level rules of the "declare before you reference" kind. Semantic constraints such as "the referenced object must exist and the types must match" can only be handed to validation, because a grammar has no idea what is in the database.

There is an easily overlooked benefit here: enforced order significantly raises the correctness of the model's output. If the model must produce the referenced object first while generating, it will not invent a name before the reference is defined. Conversely, if the grammar allows out-of-order definitions, the model tends to write the reference first and fill in the definition later, and the fill-in often does not match.

32.3 State management across passes

If you choose generation across several passes, you must design state management seriously. At least three things have to be handled: progress tracking (which elements have been generated, what comes next), consistency checking (whether a newly generated element conflicts with the existing ones), and failure recovery (when a step fails, do you restart from scratch or retry only that step).

Failure recovery deserves particular care. The recommended approach is to keep the completed portion, retry only the current step, and include the previous failure reason on the retry. This applies the "retry" idea from chapter 10 at every granularity of sequence generation. Restarting the whole sequence from scratch makes both cost and latency unmanageable.

Another practical suggestion is to put a cap on the sequence. The model may generate a sequence longer than expected (especially when the task description is vague), so there should be an explicit maximum; stop and record when it is exceeded rather than letting it keep generating.

32.4 A common failure mode

The most common failure in sequence generation is repetition: the model starts repeating content it has already produced by the time it reaches a certain element, especially when the element templates are highly similar (for example, generating ten similar configuration entries). Constrained decoding cannot help much here, because a repeated element is entirely legal syntactically.

The defense is duplicate detection in the validation layer: reject on a duplicate and retry. A better approach is to interrupt the tendency before generation starts — for example, state explicitly in the context that "the next element must differ from the previous N," or list a summary of the existing elements during multi-pass generation. Models follow explicit "do not repeat" instructions better than we tend to assume.

33. Structured Extraction from Multimodal Input

Everything discussed so far assumes the input is plain text. When the input is an image, audio, or a scan, constrained decoding still applies, but new problems appear that deserve their own discussion.

33.1 Extraction and recognition: a mix of two tasks

Extracting structured information from an image (for example, pulling the amount and date out of a photo of an invoice) is really two tasks mixed together: recognition (what is in the image) and extraction (organizing what was recognized into fields). Constrained decoding constrains only the second task.

This distinction matters because it determines how you attribute failures. When the extraction result is wrong, you have to decide whether the model could not see the image clearly (a recognition problem) or it saw clearly but filled in the wrong field (an extraction problem). The solutions are completely different: the former calls for better image preprocessing or a stronger vision model, the latter calls for better constraints and validation.

A practical attribution method is to let the model freely describe what it sees first, and then do the structured extraction. If the free description does not mention the field, it is a recognition problem; if the free description mentions it but the extraction dropped it, it is an extraction problem. This two-stage flow not only attributes failures, it also often improves extraction quality, because the second stage can use the description as context.

33.2 What is invisible in the image cannot be patched up with constraints

Here is a limitation that has to be stated clearly: constraints can guarantee that a field exists, but not that the field has content. When extracting an invoice number from a blurry photo, constraints will force the model to fill in that field, and the model may fill in a value it "guessed" — because the schema says it is required.

This introduces a very insidious failure: the constraints actually create hallucinations. Without constraints the model might output an empty value or explicitly say it cannot see clearly, which is honest; once the field is required, it has to put something there and starts making things up.

This is the only scenario in which constrained decoding actually worsens errors, and it is not an implementation problem but a design problem. The solution is to leave an escape hatch in the schema for "cannot tell" — for example, allowing null, or providing a dedicated "uncertain" enum value, or allowing an empty string. Constraints cannot eliminate uncertainty, but they can give uncertainty a legal way to be expressed. This principle applies equally to dirty and missing data.

33.3 The specifics of audio and transcription

When transcribing audio and extracting structured information (for example, pulling ticket information out of a customer service recording), the situation is similar but more extreme: the recognition error rate is higher, and the errors are often homophone errors — mis-hearing people's names, project names, and numbers.

What constraints contribute here is mainly number formatting: the length of a phone number, the character set of a ticket number, the decimal places of an amount. All of these can be enforced and they really do eliminate a fair number of errors. But for content like people's names and project names, constraints barely help — enumerating the names of everyone in a company is neither realistic nor reasonable.

For this kind of content, a more effective technique is entity linking: generate a description first, then match it against a candidate library, rather than having the model write the name directly. This is the same idea as the handling of huge enumerations in chapter 25 — using retrieval to replace enumeration.

33.4 The cost characteristics of multimodal scenarios

A final practical warning: the input cost of a multimodal request is far higher than for plain text, and the price of a retry is amplified accordingly. If one retry means reprocessing the whole image, then the formula from chapter 10, "expected attempt count multiplied by per-attempt cost," becomes very ugly in multimodal scenarios.

So the strategy in multimodal scenarios should differ from the plain-text one: lean more heavily on heavier constraints and validation (because the per-attempt cost is high and cannot be solved by retrying), and rely less on retries. You should also prioritize the two-stage flow described above, because the result of the first stage can be reused by the second, and multiple extraction tasks do not have to reprocess the same image.

34. Using Constrained Output to Drive a State Machine

In agent scenarios, structured output is often not the endpoint but the input to the next piece of logic. This chapter discusses one concrete handoff problem: how to let constrained output safely drive a state machine, and where the pitfalls here exceed those of the pure extraction scenario.

34.1 Four sources of state transitions

The agent's next state usually comes from one of four places, and their risk profiles differ a great deal: the action output by the model (the highest risk, because it comes from a probabilistic model), the execution result of a tool (relatively controllable), passive events from the environment (trustworthy), and timeouts plus degradation (which must be designed explicitly).

What constrained decoding improves is the first category: it guarantees that the model's action is structurally legal, for example that it calls a tool that genuinely exists and that the parameters conform to the schema. But it cannot guarantee that the action is permitted — a legal action may violate a business rule (calling a certain tool without permission). So the state machine must contain a permission check, and it must not execute an action just because it is "structurally legal."

34.2 Protective design of the state machine

When wiring constrained output into a state machine, four protective design points deserve emphasis:

First, deny by default. When the state machine encounters an action it cannot recognize, it should enter an explicit error state rather than stay put or fall through to a default branch. Because "staying put" in a loop becomes infinite retries, and a "default branch" easily lets the system drift onto an unexpected path without anyone noticing.

Second, transitions must be explicitly enumerated. Do not write "any unmatched action goes down this edge," because that automatically routes every future action onto a single edge. Explicit enumeration means that adding a new action forces you to handle it deliberately — that friction is intentional.

Third, irreversible actions need extra confirmation. Constraints guarantee that an action is legal, but not that it is timely. So the state machine should have a "pending confirmation" intermediate state: high-risk actions enter that state first and are actually executed only after external confirmation.

Fourth, the transitions of the state machine should be idempotent. If an action is retried because of a timeout, it must not be executed twice. Constraints cannot guarantee this, but the state machine can, by recording the actions it has already executed.

34.3 Loop detection

The most common serious problem in agent scenarios is the loop: the model repeatedly outputs the same action, each attempt yields the same result, and the system loops forever. Constrained decoding helps almost nothing here, because a repeated action is entirely legal syntactically.

There are three levels of defense. The lightest is action deduplication: interrupt as soon as two consecutive actions are identical or nearly identical. The heaviest is state detection: track the state trajectory and, if you find that a previously visited state has returned to, declare a loop and interrupt. In between is progress detection: require that every action brings some observable progress (new information, a state change), and interrupt if several consecutive steps make no progress.

State detection is the most effective of these, because it does not require you to guess in advance what kind of repetition is bad. In implementation you can use a set of visited states: once the current state is in the set, we have returned to an old state and the path must contain a cycle.

34.4 How loop detection works with constraints

Notably, constrained decoding can shrink the state space and thereby make loop detection easier. If the output is constrained to a finite enumeration, the number of possible state trajectories is manageable, and judging a loop by "has the state repeated" becomes very accurate. Conversely, if the output is free text, the state space is almost infinite and any state-based detection fails.

This is an indirect benefit of constrained decoding that is rarely mentioned: it turns the system from an "open loop" into an analyzable finite-state system. Many problems that under free output can only be avoided by prompt engineering and prayer become problems that can be formally detected under strong constraints.

35. Comparing Engines and Frameworks

Chapter 6 gave a rough comparison table. This chapter refines it and gives the dimensions that are more useful when making a selection. One reminder: frameworks evolve quickly, and the exact feature support should be confirmed against the documentation and your own testing for the version you use. What is given here is a judging framework, not a feature list.

35.1 Four categories of solution

The first category is the model provider's API capability. Constraints are implemented on the server side and you only pass a schema. The advantages are zero operational cost, tight coupling with the model version, and optimal latency; the disadvantages are that constraint details are opaque, that only a limited set of schema keywords is supported, and that you cannot customize it. The vast majority of teams should start here.

The second category is guided decoding built into the inference engine. It is implemented inside the sampling loop of the engine, and the common backends have several different implementations with different coverage. The advantage is controllable performance and the ability to work with the engine's batching and cache optimizations; the disadvantage is that you have to choose and configure the backend yourself, and you have to watch compatibility when upgrading versions.

The third category is standalone constraint libraries. They package constraint compilation and mask generation as a library and hook in through interfaces such as logits processors. The advantages are flexibility, reusability, and the ability to use them without changing engines; the disadvantage is that you have to handle the integration details with the engine yourself, and performance is usually worse than with engine-built-in solutions.

The fourth category is more aggressive generation frameworks. They not only do constraints but also manage the control flow of the whole generation process, supporting streaming, reverse constraints, and interleaving constraints with generation logic. The advantage is capability, being able to express complex generation conditions; the disadvantage is a high learning curve and integration with the existing inference stack that is less direct than the first three categories.

35.2 Six dimensions for the selection decision

First, schema coverage. Do not look at how many keywords are supported, but at how many of the keywords your schema actually uses are covered. The approach is very practical: take your schema and check keyword by keyword. The three possible outcomes each have a response — what compiles goes to constraints; what does not compile but matters should be handled by rewriting the schema into a compilable form (for example, using a union type instead of a conditional dependency); what does not compile and cannot be rewritten goes to validation.

Second, performance characteristics. What matters is not the average speed but the long tail: the latency distribution of constraint construction, the cache hit rate, and scalability under high concurrency. If your traffic peaks are high, the weight on this dimension should increase.

Third, integration cost with the existing stack. If you already use a particular inference engine and service framework, "can I just add one component" usually matters more than "which one is the most capable." A solution that requires rewriting the entire generation flow is hard to adopt even if it is more capable.

Fourth, debuggability. When a constraint system has a problem, can you find out what happened? That depends on whether it exposes state, whether it supports printing the grammar, and whether its error messages are clear. A black-box constraint implementation is dangerous in production, because when it fails you have nothing to hold on to.

Fifth, control over caching and state management. Whether you can control the cache key, whether you can warm it up, whether you can inspect the hit rate — all of these matter a lot in multi-tenant scenarios.

Sixth, maintenance activity and version stability. This field changes fast, and an inactive implementation may block your engine upgrade six months later. When selecting, look at recent commit frequency and the release cadence.

35.3 A pragmatic recommended path

If you are unsure which one to choose, my suggestion is to try them in this order: first get the whole pipeline working with the provider's API capability, and in that step tighten the schema, complete the validation, and build up the monitoring. The value of this step is that it costs almost nothing yet resolves most problems.

Then look at the data: if format-related problems are already rare and semantic problems stand out, you do not need to build your own constraints; if format problems still exist (usually because the provider does not support some schema feature you need), then upgrade to the engine-built-in solution.

Only consider the third and fourth categories when what you need genuinely cannot be done with the existing options. The value of this order is that it avoids the most common mistake — investing in self-building too early because "constrained decoding sounds advanced," only to discover that the provider's API was already sufficient.

36. Verifiable Generation and Self-Checking

Everything so far has been about solving whether the output is legal. This chapter takes one step further: how to make the correctness of the output verifiable rather than merely acceptable. This matters especially in domains such as finance, medicine, and compliance.

36.1 Three kinds of verifiability

The first is syntactic verifiability. The output can be parsed deterministically, and constrained decoding guarantees this. Its characteristics are an extremely low verification cost and a definite conclusion, but limited information is guaranteed.

The second is constraint verifiability. The output satisfies a set of explicit business rules, guaranteed by the validator. Its characteristics are that the rules can be audited (every rule traces back to a business requirement), but the rules themselves may be written wrong.

The third is provenance verifiability. Every value in the output can be traced to its basis, such as a passage of the source text or a particular data source. This requires extra design — usually requiring the schema to demand a citation field for each value, and then checking the validity of the citations at the application stage.

The third is the most valuable in many scenarios, because it turns "trust the model" into "check the citations." The model's error rate may not be low, but if every assertion carries its basis, then a human or a program only has to spot-check the evidence, and the review cost drops sharply.

36.2 Let the model check itself

There is another approach that is cheap and works reasonably well: let the model check its own output once after generating. Concretely, append a short check pass after generation that asks the model to compare its own output against the schema and decide whether there is a problem, fixing it if there is.

The effectiveness of this method relies on a property mentioned earlier: the model has strong local attention on what it has just generated, so self-checking often finds problems that were missed during the main generation. It is especially effective for problems like "a missing field" or "a value outside the enum."

But watch out for two traps: first, the model may over-confidently confirm wrong output (saying "looks fine" when it is not), so self-checking cannot replace the validator and can only be a supplement; second, self-checking significantly increases latency (it amounts to one extra generation), so it is only worth enabling when the cost of an error is high enough.

36.3 Replace self-checking with the validator

Precisely because self-checking is unreliable, wherever a deterministic check is possible, do not use model self-checking. This rule is violated over and over in real projects, at the cost of doubled latency with no gain in reliability.

The criterion is simple: if you can write one deterministic rule for the problem, write the rule; if you cannot (for example, "does this summary capture the key points"), then consider letting the model judge. The overwhelming majority of constraints in the structured output domain belong to the first category, so the applicability of self-checking is actually very narrow.

36.4 How verifiability constrains the architecture in reverse

There is one less obvious conclusion: thinking clearly about what you want to verify will determine in reverse how you design the schema. If you want the output values to be traceable, the schema should contain citation fields; if you want to verify cross-field consistency, the related fields should sit together rather than being scattered; if you want humans to review quickly, the order and grouping of fields should follow the logic of that review.

This means verifiability requirements should be taken into account when the schema is designed, rather than patched in after something goes wrong. Adding verification to structured output after the fact is usually painful, because you cannot obtain the original evidence, nor can you make already-shipped output retroactively gain the missing fields.

37. Structured Data as an Observability Signal

This chapter covers one easily overlooked use: using structured output as an observability signal. Its value does not lie in the content itself but in the fact that, because the structure is reliable, this data can be collected, aggregated, and alerted on stably.

37.1 The advantages of structured logs

The traditional approach is to extract key fields from the model output with a regular expression and then compute statistics. This has two problems: the extraction can fail (the regular expression does not match), and the structure of the extracted fields is unstable (sometimes present, sometimes not), so a lot of data is discarded during aggregation.

With constrained decoding you can also produce structured output for the log content itself — have the model emit a fixed-format analysis result alongside the answer to the user, such as intent classification, whether a sensitive topic is involved, and a confidence value. Logs obtained this way can go straight into storage and straight into aggregation, with no post-processing at all.

37.2 Runtime metrics that can be structured

If the logs are structured, you can compute many metrics that traditionally require human judgement. For example, a sudden shift in the intent distribution reveals abnormal traffic, a sudden rise in the proportion of some output type reveals model behavior drift, and a change in the average length of output fields reveals a change in the input distribution.

These metrics share a common property: they measure "what the model is thinking" rather than "what the model said," so they can raise a signal long before users complain. And that is exactly what unstructured logs cannot do, because free text cannot be aggregated stably.

37.3 Making observability a driver for constraints

Here is a nice positive feedback loop: because you want to collect these metrics, you need a stable structure; because you have a stable structure, collection becomes easy, so you become more inclined to make it a strong constraint rather than an optional output. The end result is that observability and reliability improve together.

The converse also holds: if your observability depends on optional output fields, then whenever the model is lazy and does not emit them, the metrics silently distort and you wrongly conclude that "there is nothing abnormal today." Fields that observability depends on must be required and strongly constrained, otherwise the observation itself has blind spots.

37.4 The cost of collecting these signals

Finally, on cost. Structured logging means that every generation additionally outputs a portion of content, which increases output length and latency. The criterion is whether the collection cost of the signal is worth it: if it can catch a serious incident a week earlier, the extra few milliseconds per day are an extremely good deal; if it is merely an "interesting-looking" metric that nobody will look at, it is not worth it.

The pragmatic approach is to start with the one or two most valuable signals — usually the intent distribution and the error type distribution — confirm that they really have consumers (a corresponding dashboard or alert), and then expand gradually. Making every conceivable field required is a common form of waste.

38. Trading Off Reasoning Budget Against Format Constraints

Models with reasoning capability introduce a new variable for structured output: the model first spends part of the budget thinking, and then produces the output. The relationship between those two budgets directly affects how effective and how expensive constrained decoding is, and this chapter is devoted to it.

38.1 The reasoning phase and the output phase are constrained separately

The key point is this: the reasoning process is usually not subject to structural constraints; only the final output is constrained. This gives a very nice property — reasoning can unfold freely and think through a complex judgement, while the final output is still a strict structure.

This is far better than "forcing the entire output to be JSON." The latter makes the model carry the formatting burden at every step, which lowers the quality of its thinking. Combining reasoning with constraints separates "thinking it through" and "expressing it clearly" completely, handling each with the approach that suits it best.

38.2 When the reasoning budget is eaten by formatting mistakes

But there is a form of waste: if the model's reasoning budget is fixed and it spends many tokens agonizing over format (for example, repeatedly confirming whether a certain field should be filled), then that budget is not being used on real business judgement. One indirect benefit of constrained decoding is that it eliminates format-related thinking entirely, because format no longer has to be thought about.

How do you verify this? Compare the length of the reasoning phase with and without constraints. If the reasoning is clearly shorter under constraints, the model really was agonizing over format before; if the lengths are about the same, the model was not agonizing over format to begin with, and the main benefit of constraints is reliability rather than saved tokens.

This comparison is worth doing, because it lets you trade off "improving reliability" against "reducing cost." If the reasoning length does not change noticeably, then the cost of constraints is purely mask overhead, which is very low; if it is clearly shorter, then constraints additionally reduce reasoning cost, and the return is larger.

38.3 Consistency between the reasoning and the structured fields

A common problem is this: the conclusion reached in the reasoning does not match the value finally written into the field. For example, the reasoning says "it should be about 3 centimeters," but the field is filled with 5. The cause is that there is no enforced correspondence between the two stages.

There are two fixes. One is to force the reasoning to reference the field names, that is, to have the model state during its thinking what value each field should take, and fill in afterwards; the other is to do the consistency check in the validation layer, but that is hard, because the reasoning phase is free text and semantically aligning it with the fields is itself a model call.

The first approach is more practical. How it is implemented is by adjusting the prompt structure so that the reasoning phase walks through the values in the order of the final fields and explains the reason for each one. An extra benefit is that the quality of the reasoning often improves — because the model is forced to give a reason for every field, and the fields it did not think through become visible.

39. Evaluating Constraint Support When Choosing a Model

Models turn over fast, and when choosing one, if you only look at context length and general capability you easily miss a dimension that matters enormously for structured output: how obedient it is to constraints. This chapter gives an operational way to evaluate it.

39.1 Compare models horizontally using clipped probability mass

As noted earlier, constraints clip away the illegal part of what the model "wanted to say," and the clipped ratio is exactly the clipped probability mass. This quantity varies a great deal across models, and it directly reflects the model's intrinsic inclination toward format.

The evaluation method is simple: with the same schema and the same batch of prompts, record the average clipped probability mass for each model. The lower the number, the more "willing" the model is to follow the format, the smaller the distribution perturbation the constraints cause, and the smaller the loss in output quality. A model with a high number may actually perform worse on structured tasks than one with a low number, because it is always fighting the constraints.

This metric is more valuable than "format adherence rate," because format adherence rate can only be compared between constrained and unconstrained runs, whereas clipped probability mass provides a continuous quality signal that does not depend on the success rate.

39.2 Completeness of required field filling

The second evaluation dimension is how completely required fields are filled: given that constraints guarantee the fields appear, count how many of them were filled with empty strings or meaningless placeholders. This metric measures whether the model "really wanted to fill it in" or merely "left a slot because it was forced to."

Models often differ more on this metric than on format adherence, because format adherence is near perfect under constraints while this metric exposes the difference. A practical rule of thumb: if a model's empty-value rate is clearly high, it means the model has no genuine source of input for those fields, and at that point switching the constraint solution is worse than changing the schema or supplying more context.

39.3 Stability: the variation across repeated generations of the same input

The third dimension is stability. The approach is to fix the input and the parameters, generate repeatedly, and measure how much the results differ. If two outputs from the same model differ a lot, then no matter how good the average quality is, it will cause trouble in production — because you cannot reproduce the problem.

This test has to control the temperature variable in particular, and testing at a higher temperature exposes instability better. Many models perform well at low temperature, but once production cannot guarantee stable batching conditions because of mixed multi-tenant load, the results shift. Constrained decoding improves structural stability but not content stability, so this dimension must be measured separately.

40. Team Collaboration and Code Organization

Constrained decoding ends up being a system that several people on a team maintain together. This chapter covers a few organizational conventions, whose value lies in avoiding long-term confusion rather than in solving any specific technical problem.

40.1 The schema must be the single source of truth

The most common technical debt is having three copies of the schema: one used to build the prompt, one used to compile the constraints, and one used for runtime validation. If those three are synchronized by hand, they will drift apart sooner or later.

The solution is to generate all of them from a single source. That source can be an interface definition file or a type definition, but it must be one source. Produce all three together during generation and update them in the same commit, so that during review it is immediately visible whether the constraints and the validation are in sync.

40.2 Write down the conventions around constraints

A few conventions deserve to be written explicitly into the team documentation, because they all belong to the category of "everyone steps on this if nobody writes it down": which dimensions the cache key must contain (engine version, model identifier, schema version), what marker a degraded result must carry, and who decides the retry count (it should be specified by the caller according to the latency budget, not defaulted inside the library).

Without documentation, these conventions are very likely to be lost when people change roles, and then to reappear in the form of bugs. Writing them down costs half an hour and saves an unlimited number of future debugging sessions.

40.3 Handling errors by layer

There is one more organizational recommendation: distinguish "format failures" from "semantic failures," and route them to different handling paths. Format failures should be retried (because constraints should make them impossible; if they happen, there is a bug to investigate), while semantic failures should go to business degradation or to a human.

Handling the two together (throwing the same exception, running the same retry logic for both) makes the problem hard to localize — you cannot tell whether a given failure is a bug in the constraint layer or a normal rejection by the business logic. This rule is highly valuable when debugging.

40.4 Periodically review the schema keywords that get dropped

Finally, establish one small mechanism: periodically inspect the schema keywords that were dropped during compilation (chapter 6 mentioned that the compiler records the keywords it cannot compile). Changes in this list reflect the direction the schema is evolving, and they also remind you whether new business rules could be compiled into constraints.

The other use of this mechanism is finding schema design problems: if the same keyword (for example, a value range limit) keeps reappearing in the dropped list, it is an important rule that is being violated repeatedly and is worth the effort to constrain it — perhaps by finding a different schema formulation that can express it.

41. A Pre-Launch Checklist

Go through this list in order; each item can be verified within ten minutes:

1. Can you reproduce failures offline? Save the failure samples you have seen as a test set (even if it is only twenty cases). Without this data, every later optimization is blind.
2. Are the constraints actually taking effect? Deliberately write a wrong constraint (for example, leave one value out of an enum) and see whether the model can produce the omitted value. If it can, the constraints are not doing anything at all — this is the most common case of "believing there are constraints when there are none."
3. Is the mask applied in the right place? Print once before the mask and once after, confirm that the logits of all illegal tokens really are -inf, and that no legal token was killed by mistake.
4. Tokenizer boundary testing. Take a batch of real output fragments, tokenize them in reverse, and look for tokens that straddle a grammar boundary. If there are any, your first-character decision method has a bug.
5. Sanity of max_tokens. Generate 100 valid samples, compute the token length distribution, take the P99, and multiply it by 2 as the lower bound of the budget.
6. Instrument the three failure rates separately. Parse failure rate, repair rate, and semantic validation failure rate: view them separately and alert on them separately.
7. Derive the retry count from the latency budget. Not out of thin air. First fix the P99 SLA, then compute the count.
8. Have a switch for the degradation path. You can turn degradation off in one click (returning an error instead of degraded data), which lets you stop the bleeding fast when a problem shows up.
9. Cache schema compilation. And make sure the cache key contains a hash of the schema content. Monitor compilation time and cache hit rate.
10. Load test at three levels. Measure throughput and P99 at low, medium, and high concurrency separately, and confirm that CPU-side masking has not become the bottleneck.

42. Conclusion

Back to the criterion from the very beginning: making the output parseable is, in essence, a deterministic engineering problem, and it should be solved with a deterministic mechanism rather than by praying to a probabilistic prompt. The division of labor across the three layers of fallback is as follows.

Layer 1 (constrained decoding) is responsible for "it cannot be structurally wrong." It takes the parse rate from "95% and irreproducible" to "100% and guarded by CI," at the cost of per-step mask overhead and a 20% to 60% impact on throughput. It cannot fix semantics, but semantics were never its job in the first place.

Layer 2 (validation plus limited retries) is responsible for "the content must be reasonable." It catches the semantic constraints that constraints cannot express (dependencies, ranges, cross-field consistency), at the cost of multiplied latency on failure. It is the layer most prone to overdoing it — the retry count should be derived from the latency budget rather than being "the more the better."

Layer 3 (degraded parsing) is responsible for "do not lose everything when it fails." It is only enabled in low-cost scenarios and must carry a degradation marker. Its value is turning "hard failure" into "a partial result with a confidence level," but its risk is also the highest — once the caller ignores the marker, failures are silently disguised as successes.

If you remember only one thing: turn on layer 1 first and spend the budget on raising the success rate rather than on increasing the retry count. Because raising p from 0.85 to 0.95 saves an entire retry (halving the latency), while raising N from 3 to 5 buys only a 0.01 improvement in success rate while P99 grows by 67%. This is the most counterintuitive and most valuable conclusion in the whole method.

One last word on the frontier: the next step for constrained decoding is most likely to be moving semantic constraints into the decoding phase as well — for example, having the decoder query the database in reverse when it reaches "take the value of this field," dynamically injecting the set of legal values into the mask. This kind of "constraint with external state feedback" already has early forms in some frameworks, but its complexity (external latency, cache consistency, multi-turn state) is far higher than for purely syntactic constraints. If what you are working on is "model output must land strictly inside the business-permitted value space," this direction is worth keeping an eye on.