Probing Model Capability Limits: Finding What a Model Really Cannot Do

The Gap: A Leaderboard Score and Production Performance Are Not the Same Thing

Every engineer who has to make a ship decision eventually runs into this. The same model dominates the public benchmarks, then disappoints you once it is wired into the product. Not by a little, but by a lot. A customer support assistant is brilliant through the chit-chat portion and starts answering off-topic the moment a user asks three follow-up questions in a row. A code completion tool has a high accuracy on single-function completion and, on cross-file changes, routinely produces patches that are syntactically valid and logically wrong. An extraction task is nearly flawless on clean, well-structured tickets and starts dropping fields the moment it scans a year-old ticket.

That kind of gap is not random, and it is not a case of the model “suddenly getting dumber”. It is systematic, and it decomposes into several largely independent causes: the eval data does not share your business distribution; the eval task does not share the shape of your production task; single capabilities degrade non-linearly once they are composed, while public leaderboards almost always test one capability in isolation. Layer those factors with your own business and the leaderboard score degenerates into a weakly correlated reference value.

Deeper still, the problem is this: a leaderboard score is a scalar, and what you need is a map. The question you actually care about was never “is this model strong?”, but “at which exact step does this model fail on inputs like mine?”. A model scoring 70 is entirely sufficient for you if all of its lost points sit in mid-context information recall and your business never exceeds three thousand characters per conversation. A model scoring 88 is dangerous for you if its lost points happen to land on multi-turn instruction following and you process a large volume of “tweak the plan you just gave me” requests every day.

Why an Average Is a Bad Metric

Every single-number summary has one problem: it hides the distribution. Accuracy, win rate, and pass rate in an eval report are all, at bottom, a high-dimensional capability vector projected onto a scalar axis. That projection is not invertible, so two models with completely different capability profiles look identical on a leaderboard as long as their projected values are close.

The more useful engineering move is to stop chasing a composite score and start producing a set of labeled per-dimension pass rates. The deliverable you want is not “model A beats model B by three points” but “in scenarios that require strict output schema adherence, A passes 91% of the time and B passes 99%, so route to B; in scenarios that require a consistent refactor across ten files, A is 46% and B is 12%, so use A”. Those two pieces of information differ by an order of magnitude in the value they carry for architecture decisions.

This is also why this article spends so much time on how to build eval sets, how to score them, and how to size the samples. These topics look pedantic, but they are different faces of the same thing: the quality of the capability boundary profile you end up holding is determined entirely by the measurement process that produced it. A biased measurement process yields a biased profile, and that bias is usually silent. You will not see any error; you will discover one day during a production incident that the original eval conclusion was wrong.

Four Concrete Mechanisms That Disconnect Public Leaderboards from Production

These four are not speculative theory. They are mechanisms that every team which has shipped an LLM feature has run into. Understanding them is the precondition for deciding whether to build your own evals.

Mechanism One: Distribution Mismatch

Public eval data comes from the collection channels of whoever built it, and it usually carries a strong domain preference. Some skew toward general knowledge, some toward academic reasoning, some toward standardized instruction following. That data is misaligned with your business data along several dimensions at once:

• Domain mismatch: the “reasoning problems” in an eval are mostly mathematical or formal logical reasoning, while your business may be inferring a user's real intent from a pile of incomplete ticket descriptions. The former has a clean problem statement; half the information in the latter is missing.
• Language and register mismatch: real user input is full of typos, colloquialisms, elisions, dialectal phrasing, industry jargon, and copy-paste formatting noise. Eval inputs are typically cleanly written.
• Length distribution mismatch: the overwhelming majority of eval samples are short inputs. The length distribution you meet in production may have an extremely heavy tail, and the long part of that tail is exactly the hard part.
• Adversarial mismatch: eval sets do deliberately include tricky samples, but the trickiness runs in whatever direction the authors could imagine. Real users will push in directions you never anticipated.

Distribution mismatch also has a subtle amplifying effect: when inputs stray from the training distribution, the model's variance rises sharply. The same question sampled several times in-distribution may be correct every time; out-of-distribution it may be correct half the time. This means that if your eval only runs once and checks correctness, it will simultaneously understate and misjudge out-of-distribution samples. You learn neither how high the failure rate really is nor how unstable the model is.

Mechanism Two: Different Task Shapes

The same capability goal can be carried by completely different task shapes in eval and in production, and that difference alone is enough to move the result by several multiples. The most common cases:

• Single-turn vs multi-turn: a leaderboard item hands you all the conditions at once and then asks for an answer. In production, conditions arrive incrementally, and the model must remember and integrate constraints that appeared a dozen turns earlier (“do not use the tone of the second paragraph”, “keep amounts to two decimal places”). Multi-turn settings add a state-tracking requirement that most leaderboards do not test at all.
• Direct answer vs tool call: an eval may ask “how should this order be handled” and the model answers directly. In production the model has to look up the order table, then the user tier table, then call the refund API, and finally merge all three results into one reply. Tool calling swaps generation capability for orchestration capability, and the latter is far more sensitive to format and ordering.
• Free text vs structured: a leaderboard may accept “as long as the model's answer is right”. Production requires output that a downstream program can parse: field names, types, nesting, and default-value handling are all hard constraints. Even when the semantics are perfectly correct, one trailing explanatory sentence can break the entire pipeline's parser.
• Complete input vs retrieval assembly: the “document” the model sees in production is a set of chunks selected by the retrieval system — three relevant passages plus five irrelevant ones is a perfectly normal input. Evals usually hand over clean, entirely relevant material. The two demand completely different levels of interference robustness.

Stack those differences together and you get a familiar pattern: a model performs superbly on leaderboard A, yet the share of your product where it is “actually usable” falls far below what that score implies. That is not a fabricated leaderboard. It is measuring a different thing.

Mechanism Three: Capability Degrades Non-linearly Under Composition

Test each capability in isolation and the scores are usually respectable. The real problem appears when capabilities stack: length, complexity, interference, turn depth, structural constraints. Each dimension on its own may sit comfortably inside the model's capability envelope; apply two or three at the same time and accuracy falls off a cliff. That degradation is often non-linear, and there may even be a phase transition point — past it, adding a little more input makes the output entirely unusable.

Engineering-wise this directly determines how you design the eval set: evaluating only simple single-dimension samples systematically overestimates how usable the model will be in your business. Your production scenarios all live in the composed region, so the eval set has to as well. A rough decomposition helps: if the single-dimension pass rate is 0.95, a two-dimension composition is often only 0.75 to 0.85, and a three-dimension composition can drop below 0.5. That multiplicative approximation is only a rule of thumb, and the real decay depends on the task type and the implementation, but it is enough to explain why composed evaluation is mandatory.

Mechanism Four: Leaderboard Scores Hide Stability and Cost Structure

Leaderboards almost never report variance, temperature sensitivity, or output length. A model that does well under greedy decoding may wobble noticeably under the sampling parameters you actually use in production; a model with slightly higher accuracy but three times the output length may cost more in practice. Those dimensions only mean something when measured under your specific configuration, and any score detached from a configuration is incomplete.

There is one more easily missed point: a leaderboard score corresponds to a one-shot, stateless, side-effect-free inference call. Production failures are usually not wrong answers; they are wrong answers that nobody notices. An extraction drops one field, no exception fires, no alert fires, and the error flows straight downstream. That kind of silent failure only surfaces inside an eval pipeline that carries real side effects, which means your own.

Deriving Eval Sets Backwards from Production Failures: The Highest-Value Step

Eval sets come from three places: downloading and converting public datasets, hand-writing them from experience, and deriving them backwards from real production failures. The first two are the on-ramp. The third is the dividing line between measurement and an engineering asset. The reason is simple. Samples from the first two sources test what you guess the model might be bad at. Samples from the third source test what you know the model has already gotten wrong. The first has exploratory value, the second is regression protection, and your product needs the second first.

Step One: Build a Pipeline That Can Harvest Failure Samples

The precondition for deriving a dataset backwards is that failure samples can be collected. This sounds obvious, but most teams cannot do it. The typical situation is a pile of user feedback tickets in production and no structured library of failures. A minimal viable version of this pipeline has four parts:

• Full-fidelity logging: for every production request, durably persist the input, the output, the model version, the prompt version, the sampling parameters, and the full trace of every downstream tool call. Miss any one of them and you cannot reproduce the case afterwards.
• An explicit negative signal: something has to mark “this one failed”. It could be a user thumbs-down, a downstream parse failure, a failed business rule check, or a queue of human reviewers working asynchronously.
• Stable sample identity: the same conversation and the same ticket have to group together, and different instances of the same class of error have to be linkable. Otherwise you end up with a thousand records saying “the model is occasionally wrong” instead of “it is wildly wrong on this class of input”.
• Periodic cleaning and deduplication: failure samples contain a lot of exact and near duplicates, and feeding them into the eval set raw makes the problem look many times larger than it is.

This pipeline is not cheap, but it is a one-time infrastructure investment, and it simultaneously serves production triage, prompt tuning, the data flywheel, and several other purposes. If you are going to do exactly one thing to take control of your model behavior, it should be this one, rather than writing eval scripts.

Step Two: Attribute Failures Instead of Bulk-Loading Samples

Once you have harvested failures, dumping all of them into the eval set is the most common mistake, because they are heavily contaminated with failures that have nothing to do with model capability: retrieval never recalled the right material, context was truncated right before the key fact, a tool endpoint returned dirty data, the prompt template happened to be missing a variable. Without a distinction, you end up with an eval set whose pass rate is near zero and no idea at all what to fix.

The correct move is to tag every failure with an attribution label, where the label takes values that correspond to an actionable engineering action rather than a vague description:

Attribution label	Typical symptom	What to change
Missing knowledge	Invents an entity or rule that does not exist	Add context or change the retrieval strategy
Broken reasoning	All conditions identified, but the conclusion does not hold	Decompose the task or add intermediate verification
Instruction violation	Ignores an explicitly stated constraint or ordering	Restructure the prompt, front-load constraints
Context loss	Uses information given earlier that was truncated later	Adjust the context budget or add summarization
Multi-turn drift	Starts violating an early constraint after some turn	Externalize state instead of relying on context
Tool misuse	Wrong arguments, wrong order, pointless retries	Tighten the schema or revise the orchestration logic
Format breakdown	Semantics correct, structure fails to parse	Constrained decoding or post-processing validation
Fails to refuse	Answers anyway when beyond capability or beyond authorization	Add an explicit refusal policy and permission boundary
Not a model problem	Retrieval miss, truncation, dirty upstream data	Fix the pipeline first, leave the prompt alone

The value of this table is that it translates the useless conclusion “the model is not good enough” into concrete engineering actions. After attribution you will usually find that “not a model problem” accounts for a larger share of your failures than you expected. That finding alone justifies a pipeline cleanup pass, and afterward the eval set's baseline will rise on its own.

Step Three: Keep Only the Samples the Model Could Have Got Right

After attribution you filter once more. The criterion is simple: if you swap the model's output for a correct output, does the pipeline behave correctly? If it behaves correctly even without the swap, it is a pipeline problem and does not belong in this eval set. Every sample in the set should satisfy one property: its failure is attributable to model behavior itself, not to the environment.

This filtering discipline matters enormously. An eval set contaminated with pipeline problems cannot guide any decision. You change the prompt, the score moves, and you cannot tell whether the model improved or whether retrieval happened to recall one extra document. The score loses all causal interpretability, and the eval set is dead.

Controlling the Difficulty Distribution: Making the Score Informative

Having samples is not enough. If 90% of your samples are easy and 10% are extremely hard, then most of that 0.9 accuracy is contributed by the easy ones, and you cannot distinguish “the model improved from 0.85 to 0.90” from “the model went from 0.1 to 0.15 on the hard items”. Those two improvements may differ by tenfold in business value while the total score barely moves.

The approach is to design the difficulty stratification deliberately rather than passively accepting whatever distribution the samples happen to have. A few practical strata:

• Input length: bucket by token count — say 1k, 5k, and 20k. Report per bucket, because collapse in the long buckets is the most common failure.
• Number of conditions: how many constraints the user gave in one task, how many things must hold simultaneously. Pass rate is often strongly non-linear in constraint count.
• Interference intensity: how many semantically related but practically irrelevant passages sit in the context. This is the dimension that separates models most sharply in retrieval-augmented settings.
• Adversarial load: whether the sample contains inducements, negations, ambiguities, polysemous references, or statements whose implicit assumptions must be actively overturned.
• Turn depth: at which turn a multi-turn task starts to collapse.

Once you bucket along these dimensions and report per bucket, you get a table more useful than any composite score. It tells you directly whether this model can take your kind of work, where the breakpoint is, and whether it is even worth salvaging past that point.

Preventing Test Set Contamination: Three Disciplinary Rules

The most common way an eval set dies is not inaccuracy. It is quiet contamination, after which it keeps reporting an inflated score. In an engineering team the same person often builds the eval, tunes it, and reads it, and that overlap of roles makes contamination nearly certain unless it is structurally prevented. Three rules:

Rule one: physically separate the test set from the tuning set. The tuning set is where you run repeatedly, look at results repeatedly, and change prompts; it will inevitably be overfit, and that process is legitimate and in fact necessary. The test set may only be run periodically, to answer one question: “did this round of changes move us forward or backward?”. If you look at the test score every time you edit a prompt, the test set has become a tuning set.

Rule two: new production samples enter the tuning set first, never the test set. Derived failure samples naturally come from the current system's boundary, so letting them straight into the test set means every release is doing model selection on the test set. The safer approach is to route new samples into the tuning set and, once several iterations confirm that they represent a stable capability dimension, “promote” them into the test set with a twin sample that has not previously been perturbed: similar input, different answer. That is what stops a model or a human from memorizing a surface pattern.

Rule three: do not show the test set's gold answers to whoever writes the prompts. This one sounds excessive, but it prevents a very subtle leak: once you know the answers, you unconsciously add prompt guidance aimed at those specific answers, and that guidance does not generalize to new samples, sometimes actively hurting them. The practical implementation is to keep the test set's scoring stage independent of the prompt tuning stage.

Beyond these three rules, there is one more engineering move worth doing: generate perturbed versions of the test set. For each test sample, automatically produce a few variants (rephrased, reordered, entities swapped, irrelevant sentences added and removed) and run the perturbed set before release to see whether the model still holds up under perturbation. A model that scores 95% on the original test set and 62% on the perturbed one has learned a surface pattern rather than a capability. Such a model collapses the moment it meets a new phrasing, and the original test set will never tell you that.

Capability Tiers: Eight Independent Ceilings

“Is this model good enough?” is a question with no engineering value in it. The useful form is to split capability into a number of relatively independent dimensions, measure a pass rate and a collapse point for each one, and keep them separate — because their collapse points sit in different places and therefore call for completely different mitigations.

Dimension One: Knowledge

The knowledge ceiling is not set by the model's parameter count. It is set by how much information you can fit into the context and how accurate the retrieval system can be. The right way to test this dimension is end to end: give a real business question, run the full retrieval-plus-generation chain, and check whether the final answer is correct.

Testing “does the model know this private fact” in isolation is not very meaningful, because you will never dump private knowledge straight into a prompt; you will retrieve it. What you actually need to measure is two things: whether the retrieved material is sufficient to support the answer (retrieval's job), and whether the model uses it faithfully once it is sufficient (the model's job). You separate them by constructing a set of samples with the correct context supplied, where the model has no excuse to be wrong and anything that goes wrong is a generation problem.

Dimension Two: Reasoning

For reasoning you have to separate two different questions: how long the chain of thought is and whether the chain of thought exists. The first is how accuracy decays as the number of steps grows; the second is whether the model will skip a step, misread it, or contradict itself midway.

The most effective way to probe the reasoning ceiling is not to hunt for harder problems but to take a known reasoning problem and lengthen it in a controlled way: break its conditions into finer pieces, write the intermediate quantities out explicitly, insert a harmless intermediate computation, and watch how accuracy tracks the growth of the chain. The shape of that curve is more informative than any single point. A slow linear decline means the decay is predictable and can be bought back with a larger thinking budget. A steep collapse means there is a structural defect, and you need to change how the task is decomposed rather than adding budget.

Dimension Three: Instruction Following

This is the dimension most often overestimated and also the one most often fixable in isolation. Instruction following means the model's adherence to explicit, formatted, verifiable constraints: the output must be some structure, must contain certain fields, must follow a certain order, must stay under a length limit, must use a certain tone, must be in a certain language.

The key to testing it is writing the constraints in a form that can be checked automatically. If the constraint is “the output must be valid JSON and must contain fields a and b”, then scoring should not involve a human or a language model at all — use a parser and a field check. Constraint adherence is one of the few capability dimensions that can be scored almost objectively, the eval cost is minimal, and prompt changes usually show up immediately. It is worth doing first.

There is one special subcategory worth separating out: negative constraints. Asking a model not to do something is far harder than asking it to do something, and it is also far more likely to be overlooked, especially when the negative constraint conflicts with a positive instruction (“do not explain, just give the answer” versus “explain why when you are uncertain”). When testing negative constraints, design “bait” samples specifically, placing the forbidden behavior where it is easiest to trigger.

Dimension Four: Long Context

Long context has to be split into three sub-items and tested separately, because their collapse mechanisms are entirely different:

• Needle retrieval: locating a specific string inside a long body of material. This is the easiest layer, and the vast majority of models that advertise long context pass it. It only proves that the positional encoding handles that length, not that comprehension does.
• Information aggregation: answering requires combining several facts scattered across different positions in the long material. This is the length-dependent capability that actually corresponds to business work.
• Interference from long material: the material contains many semantically near-identical distractors and the goal is to tell them apart. This is the hardest layer, and the one most likely to break in production — the ten passages retrieval hands the model are usually highly related to each other, and the model has to separate them precisely.

The most useful report format here is a pass rate curve bucketed by token length, not “it passes at 128k tokens”. Real business input length is a continuous distribution, and what you need to know is which point on that curve corresponds to “this is roughly how long my inputs usually are”.

Dimension Five: Multi-turn Consistency

Multi-turn consistency asks: does the model's output change when unnecessary information is added? That is a property of stability, not of correctness. A model that is entirely correct but gives a different answer when the user rephrases the same question produces an equally bad product experience.

You build samples for this dimension with semantic equivalence: rewrite the turn-k question into a semantically equivalent but differently worded form and observe whether the subsequent output changes materially. At the same time, design a constraint drift test — give a set of constraints in the early turns and then, dozens of turns later, require the model to keep producing output that still satisfies them, and see when it starts to forget. The turn depth at which forgetting begins varies by model, and it correlates strongly with where those early constraints sit in the context.

Dimension Six: Tool Calling

Tool calling is currently the dimension most overestimated in production. It contains at least four independent capabilities, and reporting a single blended score for them is meaningless:

• Selection correctness: when the available tool set is large and contains tools with similar functions, did the model pick the right one?
• Argument correctness: required fields, types, enum values, and correctly extracting argument values from context. This one usually has the highest error rate, because it is a mapping from natural language to structured fields, and consistency is hardest to guarantee for exactly that kind of mapping.
• Call sequence correctness: are the ordering dependencies among multiple tools satisfied, and do cyclic calls or pointless retries appear?
• Result use correctness: once the model gets a tool's return value, does it use it correctly rather than ignoring it or over-relying on it?

The tool dimension carries one special constraint: side effects cannot be rolled back. During evaluation you must separate read-only tools from tools with side effects. Read-only tools can be called freely inside the eval set; tools with side effects need a sandbox or record-and-replay. Ignore this and your eval set contaminates real data the first time it runs. Many teams have learned that at high cost.

Dimension Seven: Structured Output

On the surface this dimension looks like a subset of instruction following, but it deserves its own entry because it has an independent failure mode: semantically correct, syntactically broken. The content of the model's output is entirely correct, but it wrapped an extra layer of explanation around it, or used full-width quotes, or left a trailing comma, and the whole parse fails.

The engineering answer to this failure mode is constrained decoding: constrain the output with a grammar at sampling time rather than repairing it after the fact. Constrained decoding turns “the format is correct” from a probabilistic question into a structural guarantee, and it is the most effective fix for this dimension. The cost is that it restricts the model's freedom of expression, and its implementation on complex schemas depends on the specific engine's capabilities, so whether it is available at all depends on your stack.

Dimension Eight: Refusal

This is the dimension most easily overlooked and the one with the largest effect on launch risk. Evaluating it has two directions and both are mandatory:

• Does it refuse when it should? When the question falls outside the model's knowledge, outside its permissions, or when the input itself cannot support an answer, does the model refuse explicitly instead of fabricating? Errors in this direction are extremely expensive.
• Does it wrongly refuse when it should not? When there is enough information to answer, or when a bit of common-sense reasoning suffices, is the model being excessively conservative? Errors in this direction cost you usability, and they are usually more insidious, because they present as “the model is not very good” rather than “the model broke”.

Errors in these two directions have to be measured and optimized separately, because their drivers are completely different. Better refusal behavior comes from explicit boundary declarations and permission checks placed upstream. Lowering the false-refusal rate comes from clean question rewriting and a “try first, degrade second” policy. Folding them into a single “refusal accuracy” number hides every bit of useful information.

Probe Design: Locating the Breakpoint with Controlled Perturbation

A conventional eval answers “what is the pass rate on this distribution”. It cannot answer “under what conditions does it start to fail”, and the second question is what a capability boundary actually cares about. The basic technique for locating a breakpoint is controlled perturbation: change one variable at a time, observe how the metric moves, then sweep that variable to trace out a curve.

The value of that curve is that it separates three completely different failure modes. A slow linear decay means the capability is degrading but predictably, and can be compensated with more resources (thinking budget, more samples, task decomposition). A phase transition point means there is a definite threshold past which output collapses, and you must change the shape of the task rather than add resources. A completely flat cliff — one tiny perturbation takes the pass rate to zero — usually points at fragility in the formatting or parsing layer rather than insufficient capability, and those problems are often fixed by a single good prompt. The three modes call for three entirely different responses, and if you only report one aggregate score you can never tell which one you are looking at.

A Checklist of Perturbation Dimensions

Practical perturbation dimensions fall into a few families, and it is worth sweeping each of them:

• Scale perturbation: input length, number of constraints, number of tools, turn depth, number of interfering passages in the context.
• Structural perturbation: the depth of the required output hierarchy, the number of fields, the complexity of the nesting.
• Wording perturbation: synonym substitution, reordering, tone shifts, turning an active sentence into a passive one, adding colloquial phrasing.
• Information perturbation: moving key information from the opening to the middle or the end, inserting highly similar distractors in between, scattering key information across several locations.
• Adversarial perturbation: inserting statements that contradict the correct answer into the material, retracting a previously stated constraint mid-conversation, planting a false premise in the question.
• Parameter perturbation: changing the sampling temperature, the thinking budget, the maximum output length.

Three to five perturbation strengths per dimension is enough to see the trend; there is no need for a dense sweep. A dense sweep is expensive, and in most cases three to five points already separate “slow decay” from “cliff”.

How to Construct Boundary Cases

Boundary cases are not hard samples dug out at random. They are samples constructed deliberately near the capability breakpoint. There are four general techniques for building them:

Binary contraction. First find a pair of extreme samples, one that passes stably and one that fails stably, and determine whether it is length, constraint count, or information position that decides the verdict. If you cannot tell at first, interpolate across several dimensions at once and find where the dividing line lies. After a few contractions, the boundary usually surfaces.

Minimal counterexample reduction. Take one failing sample and delete its components one at a time (drop a constraint, drop an interfering passage, halve the length) until it passes. What remains is the minimal structure of that capability boundary. The by-product of this method is extremely valuable: it tells you directly which input components have to be intercepted in production. As long as that minimal counterexample can still occur in your product, the risk is still there.

Controlled twins. Build two inputs that are identical except for one key component: one has it, one does not. The difference in pass rate is the marginal contribution of that component. This method removes the confound of “this sample was just hard to begin with”.

Adversarial generation plus human convergence. Use a model to generate a batch of inputs that might trigger a given class of failure, then have humans filter out the ones that are not actually difficult. This step quickly enlarges the sample pool, but the human filter has to be kept, because a large share of automatically generated “hard samples” are merely oddly worded and do not actually touch the capability ceiling.

A Reusable Probe Runner

The code below gives the skeleton of a probe runner. Its core job is to standardize the loop of “perturb → execute → score → aggregate into buckets”, so that you do not end up with one bespoke script per dimension and results that cannot be compared.

#encoding:utf-8
from dataclasses import dataclass, field
from typing import Callable, Dict, List

@dataclass
class ProbePoint:
    "扰动坐标系里的一个点"
    dim: str            #扰动维度名，例如 "input_len"
    level: int          #该维度的强度档位
    payload: Dict       #构造好的实际请求参数
    bucket: str = "default"

@dataclass
class ProbeResult:
    dim: str
    level: int
    passed: bool
    judge_scores: Dict = field(default_factory=dict)  #保留原始判分，不只留对错

def run_axis(probe_points: List[ProbePoint],
             invoke: Callable[[ProbePoint], dict],
             judge: Callable[[ProbePoint, dict], tuple],
             repeats: int = 3) -> Dict:
    "沿单个维度扫一遍，按强度聚合"
    agg = {}
    for p in probe_points:
        outcomes = []
        raw = []
        for _ in range(repeats):
            out = invoke(p)
            ok, detail = judge(p, out)
            outcomes.append(ok)
            raw.append(detail)
        n = len(outcomes)
        agg.setdefault(p.level, []).append({
            "n": n,
            "pass_rate": sum(outcomes) / n,
            "samples": raw,
        })
    return agg

def find_breakpoint(agg: Dict, hi: float = 0.90) -> int:
    "返回通过率首次跌破 hi 的档位；找不到返回 -1"
    for level in sorted(agg):
        rows = agg[level]
        cnt = sum(a["n"] for a in rows)
        rate = sum(a["pass_rate"] * a["n"] for a in rows) / cnt
        if rate < hi:
            return level
    return -1

There are three deliberate design choices in this code. First, scoring keeps the raw detail rather than just a boolean, because the continuous scores an LLM judge produces are often more informative once you discard the binarization — they distinguish “entirely wrong” from “right conclusion, wrong reasoning”, and those two call for completely different responses.

Second, every point repeats three times by default. This is not about manufacturing precision; it is about making the variance itself visible. If the pass rate at some strength level jumps wildly between repeats, you are standing right next to a capability boundary, and the score at that point should almost never be used to make a decision.

Third, breakpoint search is extracted into its own function, because “where is the breakpoint” is a conclusion that has to be discussed and shared repeatedly, and it should not be scattered through the print output of various scripts. Re-sweeping the axis and re-locating the breakpoint after every model or prompt change produces a curve that is itself a versionable asset.

Turning Probe Results into a Decision Table

Once the probes have run, do not just remember a single breakpoint number — translate it into a decision table. This table is the input to every architecture decision you make later, and it looks roughly like this:

Dimension	Current breakpoint	Business input distribution	Risk	Response
Input length	Aggregation-type problems start collapsing after roughly 12k tokens	95% below 3k	Low	No action needed, but add length monitoring
Number of constraints	Format adherence drops clearly beyond 4 constraints	Commonly 6 to 8	High	Merge constraints up front or switch to validation-based blocking
Turn depth	Early constraints start being forgotten after roughly 12 turns	Long sessions can reach 40 turns	High	Externalize key constraints as state
Tool arguments	Error rate of roughly 10% when extracting numbers from free text	High frequency	Medium	Add type checking and a second confirmation
Refusal	Refusal is acceptable when beyond permissions, but the model tends to guess when information is missing	High frequency	High	Ask a follow-up question instead of generating when information is missing

The numbers in the table are illustrative; the actual values depend on your measurements. But the structure of the table is the point: it translates “is the model strong” into “on which dimension, in what form, at what level of risk, and requiring what response”. Row 1 says length is not the bottleneck, so there is no reason to spend money extending the context window; rows 2 and 3 say the real problem is state management, and the response is an architecture change rather than a model swap. With a table like this, model selection stops being a matter of taste and becomes an engineering judgment.

Scoring Is Itself a System Under Test

The most overlooked fact about evals is this: the measuring instrument has its own error, and that error is often larger than the difference you are trying to measure. When you find that model A beats model B by two percentage points, the first thing to confirm is whether the judge can reliably tell those two points apart — not to celebrate the improvement. A judge whose scoring noise has a standard deviation of five points will never detect a two-point difference.

Four Classes of Bias in an LLM-as-judge

Using a language model to score generated output is currently the most practical approach: low cost, broad coverage, and reasonable correlation with human judgment. But it carries four systematic biases that have to be identified and corrected one by one.

Position bias. When two outputs are presented side by side for comparison, the judge tends to prefer whichever appears first, regardless of content. In pairwise judging this shows up as a constant offset: whenever A is first, A wins by a few extra points. The correction is to judge every sample in both directions and swap the order, counting only the samples where both judgments agree. The cost is that judging doubles, but that is a necessary expense for pairwise comparison.

Length bias. Judges tend to score longer answers higher, because length correlates strongly with surface completeness. This creates a very harmful optimization direction: your prompt tuning will drift toward “write more” while production users want shorter answers. The check is simple — run the judge over a set of samples where you know which is better and where the known-better one happens to be shorter; if the judge favors the longer ones, the bias is there.

Self-preference. Using some model as the judge to score outputs from its own family scores systematically high. The problem disappears in cross-family comparisons, but when you run daily evals with a model from the same family, some of the “improvement” you see is only an artifact of self-preference.

Surface-consistency bias. Judges are easily swayed by surface features: it cites a source, it uses a bulleted list, it sounds confident. None of these features is necessarily related to actual correctness, but a judge has a hard time resisting them. The mitigation is to split those dimensions out explicitly in the judging prompt and score them separately, rather than letting the judge make one holistic good/bad judgment.

Calibration: Treat the Judge as an Instrument You Have to Calibrate

The correct approach is to calibrate before you use it. The calibration set does not need to be large — twenty or thirty carefully hand-labeled samples usually suffice, because its purpose is to discover the direction of the bias, not to estimate exact precision. Below is a skeleton implementing calibration and bidirectional comparison.

#encoding:utf-8
import random
from typing import List, Tuple

def bidirectional_compare(sample, judge) -> Tuple:
    "交换顺序各判一次，只有两次一致才算有效"
    vd = {}
    for mode in ("A_first", "B_first"):
        x, y = (sample["x"], sample["y"]) if mode == "A_first" else (sample["y"], sample["x"])
        vd[mode] = judge(sample["prompt"], x, y)
    first, second = vd["A_first"][0], vd["B_first"][0]
    if first == "A" and second == "A":
        return ("A", 1.0, vd)
    if first == "B" and second == "B":
        return ("B", 1.0, vd)
    #两次不一致：该样本判为平局，不计入胜负统计
    return ("tie", 0.0, vd)

def calibrate(judge, gold: List[dict]) -> dict:
    "gold 每条含 gold_winner 与可选的 long_sample_name"
    n = agree = pos_collapse = len_resist = len_n = 0
    for g in gold:
        win, _, vd = bidirectional_compare(g, judge)
        n += 1
        agree += (win == g["gold_winner"] or win == "tie")
        #位置偏差探测：两次都判给先出现的那个
        pos_collapse += (vd["A_first"][0] == "A" and vd["B_first"][0] == "A")
        if g.get("long_sample_name"):
            len_n += 1
            len_resist += ((win == g["gold_winner"] or win == "tie")
                           == (g["gold_winner"] == g["long_sample_name"]))
    return {"agreement": agree / n, "positional_collapse_rate": pos_collapse / n,
            "length_bias_resistant": len_resist / max(1, len_n)}

def compare_with_ci(wins, n_boot=2000, seed=7):
    "成对比较的胜率与自助法置信区间"
    rng, n, boots = random.Random(seed), len(wins), []
    p = wins / n
    for _ in range(n_boot):
        boots.append(sum(1 for _ in range(n) if rng.random() < p) / n)
    boots.sort()
    return p, boots[int(0.025 * n_boot)], boots[int(0.975 * n_boot)]

A few design points. Inconsistency means a tie is the most important rule here: the direct symptom of position bias is exactly a flipped judgment between the two orders, and calling that a tie is far more honest than arbitrarily trusting one of them. The cost is a smaller effective sample size, so the judge budget should go to bidirectional comparison first.

Treat calibration results as a gate, not as reference information. If the positional collapse rate exceeds some threshold (say 20%), the judge is unusable for pairwise comparison and you have to change the judging method or change the judge model, rather than continuing to collect data with it.

Win rates must carry a confidence interval. Here is why that function is necessary: two models run over two hundred samples each at win rates of 0.52 and 0.55 look like they differ, but the bootstrap intervals probably overlap heavily, in which case declaring “the new model is better” is pure self-deception. Deciding whether a new version is genuinely better should hinge on whether the intervals do not overlap, or on a paired test — because for pairwise comparisons over the same batch of samples, a paired test has far higher statistical power than comparing two independent sample sets.

When You Should Not Use a Language Model Judge At All

The judge is weakest in scenarios that require external knowledge or exact comparison. The dividing line looks like this:

• Anything a deterministic check can decide must never go to a judge: whether the output is valid JSON, whether the fields are all present, whether a numeric value is exactly equal to the reference answer, whether two sets are identical. These determinations should make up the majority of your eval. Otherwise you are paying continuously, and adding noise, for something that could be automated.
• Anything needing external fact-checking should use retrieval plus citation matching: whether a factual claim in the generated content can be supported in the given material. This can be made semi-automatic — extract the claims programmatically, then have humans or a judge verify them, and finally sample-audit the accuracy of the automated pipeline itself.
• Anything judged by multi-step execution should be scored by the execution result: code tasks, SQL tasks, tool orchestration tasks. The correct answer is “it ran and the result matches expectations”. Having a program execute is one order of magnitude more reliable than having a model judge.
• Genuinely subjective criteria (tone, usefulness, how good it is) go to the judge, and only with bidirectional comparison and regular human spot checks attached.

Behind that dividing line is a more important principle: the quality ceiling of your eval set is set by the scoring method, not by the quality of the samples. A thousand carefully designed samples scored by a weak judge may measure less precisely than a hundred samples scored by deterministic checks. So before doing any sample work, first make deterministic everything that can be made deterministic.

Variance and the Statistical View: Why a Single Score Means Nothing

The most common way an eval report is presented is “pass rate 87.3%”. That number carries almost no information, because it tells you neither the precision of the measurement nor whether the next run will produce the same number. To make a score meaningful you have to report variance and confidence intervals alongside it.

Three Sources of Variance

The observed variance comes from three mutually independent parts, and once they are mixed together you cannot localize the problem:

• Sampling randomness: the temperature parameter makes the same input produce different outputs across samples. This is an intrinsic property of the model and can be reduced (by moving toward near-zero deterministic sampling), but driving it to zero sometimes pushes the model into repetition and lowers quality.
• Judging randomness: jitter in the judge itself. Its magnitude and correction were covered above.
• Sampling error over samples: the batch you drew does not represent the real distribution. This is what “insufficient sample size” usually refers to, and it is the only one you can mitigate by enlarging the eval set.

All three are handled completely differently: reducing sampling randomness is a parameter-tuning problem, judging randomness is controlled with bidirectional comparison and gating, and sampling error is handled with sample size and stratification. To separate them, the simplest experiment is to fix the eval set, fix the judge, and only repeat the sampling several times, then observe how much the pass rate wobbles. That magnitude is the variance contributed purely by model sampling, and it gives you the intrinsic noise floor of any single-point score — differences below it are not credible at all.

There is one more easily missed source of variance: nondeterminism in context and external services. If the eval set depends on real retrieval and the retrieval index changes between two runs, the difference between the two scores may come entirely from the index. The control is to use a snapshot index during eval, or to freeze the retrieval results into the eval samples ahead of time. Freezing retrieval results is the stricter option: it turns your eval into a “generation capability only” eval, and retrieval quality then needs its own separate eval.

How Many Samples You Actually Need

There is no universal answer to this, but there is one actionable criterion: compute the confidence interval and then check whether it is narrow enough for the precision your decision requires. In other words, do not pick a sample size and then compute the error; ask “how much error can I tolerate” and back out the sample size from that.

The rough intuition runs like this. To get the estimate of a pass rate to within about 2 percentage points, with the pass rate in the middle range, you need on the order of several hundred samples. To distinguish a 2-to-3 percentage point difference between two versions you usually need more, on the order of a thousand. That estimate shifts with the pass rate — the higher it is, or the closer to an extreme, the more samples you need. And in the region where the pass rate approaches 0 or 1, it is usually more useful to report the number of failures rather than the pass rate, because the pass rate is highly unstable when the denominator is small.

More effective than “hitting a sample size” is paired comparison. If both versions run over the same batch of samples, record two counts per sample — “old right, new wrong” and “new right, old wrong” — and look only at the discordant cases. The vast majority of agreeing sample pairs contribute nothing to discrimination, and removing them sharply increases the effective sample size. That is why comparing a new version against the old one on a fixed eval set is always more sensitive than comparing them on two different eval sets.

Computation and Presentation

The code below separates the three sources of variance and implements the Wilson interval, which is more robust than the normal approximation at small and extreme proportions.

#encoding:utf-8
import math, statistics
from typing import List, Dict

def wilson_interval(k: int, n: int, z: float = 1.96) -> tuple:
    "Wilson 区间，通过率接近 0 或 1 时比正态近似稳健"
    if n == 0: return (0.0, 1.0)
    p = k / n
    denom = 1 + z * z / n
    center = (p + z * z / (2 * n)) / denom
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return (max(0.0, center - half), min(1.0, center + half))

def decompose_variance(runs: List[Dict]) -> Dict:
    "每条 run 含固定评测集的分数与仅重复采样的分数，用来分离两类方差"
    var_total = statistics.pvariance([r["score"] for r in runs])
    var_sampling = statistics.pvariance([r["score_resampled"] for r in runs])
    var_subsample = max(0.0, var_total - var_sampling)
    return {"var_sampling": var_sampling, "var_subsample": var_subsample,
            #单个分数的固有噪声下限，用标准差表达
            "sd_single_run": math.sqrt(var_sampling + var_subsample)}

def mcnemar_counts(pairs: List[tuple]) -> Dict:
    "成对比较：只保留不一致的样本对做符号检验"
    b = sum(1 for old, new in pairs if old and not new)   #旧对新的错
    c = sum(1 for old, new in pairs if new and not old)   #新对旧的错
    n = b + c
    if n == 0:
        return {"significant": False, "reason": "no_discordant_pairs"}
    #符号检验的精确二项 p 值（无校正的快速版本）
    from math import comb
    k = min(b, c)
    p = 2 * sum(comb(n, i) for i in range(0, k + 1)) / (2 ** n)
    return {"old_only_right": b, "new_only_right": c,
            "discordant": n, "p_value": min(1.0, p),
            "significant": p < 0.05}

Note the design of decompose_variance: it requires you to have both a fixed eval set, where you only resample in order to isolate sampling variance, and records of reruns over a subset, which estimate the subsampling variance. This two-track design looks like extra work, but it is the only way to distinguish “the model is random” from “the sample is unrepresentative”, and those two call for completely different responses — the first is addressed by adjusting sampling or taking the best of several runs, the second by enlarging and stratifying the eval set.

When You Must Rerun

Several classes of change invalidate old scores immediately and force a rerun:

• Model version or provider change: including silent version updates from the same provider. The latter is especially dangerous, because it can change behavior with no notification at all.
• Any change to the prompt template: including seemingly harmless wording tweaks, and even adding or removing examples.
• Sampling parameter change: any change to temperature, thinking budget, or maximum output length.
• Retrieval configuration or index content change: this shifts both the input distribution and the context length distribution.
• Tool set or tool description change: tool description wording directly affects tool selection, and the impact of such changes is often larger than intuition suggests.
• Change to the eval judge: after swapping judges, old and new scores are not comparable and the baseline has to be rerun.
• Business distribution drift: even if the system itself is unchanged, if the input distribution has shifted then the old breakpoint no longer holds. This drift should be monitored periodically, otherwise you will mistake drift for a regression caused by your own recent change.

The symmetric discipline is this: do not rerun in order to make the numbers look better. If a score looks bad and you adjust the configuration and rerun until it passes, you are selecting conclusions using sampling noise. The rule is simple — either rerun according to a predefined protocol, which specifies how many repeats and whether to take the median or the mean, or accept the first result.

From Eval to Engineering Decision: How a Capability Boundary Becomes an Architecture Constraint

The output of an eval should not be a report. It should be a set of explicit architecture constraints. Every breakpoint you measure has to map onto a concrete decision, otherwise the measurement never cashes out as value. This section maps everything above onto four classes of response.

Response One: Identify the Steps That Require a Different Model

Switching models is the most expensive option available, so it should be reserved for dimensions where the model really is the bottleneck and the business cannot compensate in some other way. Deciding whether a dimension is worth a model switch takes two conditions, and both must hold.

First, the dimension occurs frequently in your business input distribution. A capability that only collapses under extreme inputs, in a product where that kind of input essentially never appears, should not influence your model selection — you would be paying for a problem that does not exist. Conversely, a capability that only errs on long inputs is the primary bottleneck in a product made entirely of long inputs.

Second, failures in that dimension cannot be mitigated downstream. If a failure can be caught afterwards by a cheap deterministic check, then spending on the model side does not pay. Conversely, if failures flow silently downstream (an extraction missing one field, a generated sentence that is factually wrong), they have to be handled hard on the model side or the validation side.

Only dimensions that satisfy both conditions justify a model switch. In practice there are usually not many: most evals end by revealing that the real bottleneck is not “the model is not strong enough” but “the way context is managed is wrong” or “there is no validation layer”. The conclusion is surprising but extremely common, and it is worth adopting as the default assumption to challenge every time you run an eval.

Response Two: Identify the Steps That Require Validation

For structured output, instruction following, and tool argument extraction, deterministic validation should almost always be the backstop rather than a stronger model. There are three reasons:

• Validation costs far less than improving model capability. A JSON schema check is one function call; a genuine capability improvement may cost several times the inference budget and the latency.
• Validation reliability is structural. A check that passes is 100% certain with no probabilistic failure, whereas improvements at the model level are always probabilistic.
• Validation failures can be turned into good product behavior. When the schema is invalid, retry once, fall back to rule-based parsing, or tell the user plainly “I need more information” — all of these are far better than a confident answer in the wrong format.

The correct position for the validation layer is after the model output and before downstream consumption, not inside the model. The common three-layer structure is this: layer one uses constrained decoding to guarantee syntactic validity; layer two does schema and type validation, retrying on failure and feeding the error message back to the model (note that this is itself an instance of a multi-turn instruction-following test, so you can monitor it on the way past); layer three does business rule validation, such as whether the amounts sum correctly and whether the referenced field actually exists in the source data.

Layer three is frequently omitted, but it is the only way to catch silent failures — cases that are semantically correct, correctly formatted, and yet inconsistent with the facts. Neither of the first two layers can stop those.

Response Three: Identify the Steps That Require a Human Fallback

A human fallback is not a failure; for some dimensions it is the most economical choice. There are four typical cases that need one:

• Rare but high-stakes: operations that occur less than one time in a thousand but whose consequences are irreversible if they go wrong. Suppose eval tells you the model's success rate here is 0.985. Even with three layers of validation, the residual risk still needs a human to absorb it.
• The judgment itself is subjective: some quality dimensions, such as a reasonable reading of a legal clause or a fair assessment of a complex dispute, do not even have stable, consistent answers between humans. In such scenarios “model answer + human review” is the only reliable structure, and the model should be positioned as a first-pass filter rather than a decision maker.
• User trust is sensitive: in medical, legal, and financial advice settings, even a highly accurate model may not be compliant without human review.
• Signals that the model itself is unreliable: when the input simply does not carry enough information (the user's description is vague, the attachment is incomplete), the best behavior is to ask a follow-up question rather than generate. Turning these “should say no” cases into human review or a forced follow-up question is far more useful than switching models.

The design point for a human fallback is not to let humans become the invisible bottleneck in the pipeline. The right approach is to place people exactly where the model has deterministically failed (validation did not pass, retries exhausted, confidence below a threshold), so that they only handle the exception path, while keeping the exception rate low enough. If 10% of traffic lands in a human's hands, that is not a fallback; that is a system degraded into semi-automation.

Response Four: Identify the Problems That Need an Architecture Change, Not a Model Change

There is a class of problems whose correct solution lies entirely outside the model. What they share is this: they require the model to maintain state beyond what the context can reliably carry. Concretely, they include:

• Making the model “remember” a constraint set dozens of turns earlier. The response is to externalize the constraint as an explicit state object and re-inject it every turn, rather than hoping the model remembers it.
• Requiring the model to maintain a dozen mutually constraining sub-goals within one task. The response is to split the task into sub-tasks with clear boundaries, each with its own single generation.
• Requiring the model to repeatedly refer back to a large block of text it wrote earlier itself. The response is to write the structure first and then fill in the content, turning “referring to yourself” into “referring to data”.
• Requiring the model to keep style or terminology consistent across a long workflow. The response is to turn style constraints into rules in a transformation layer rather than an expectation in the generation layer.

The value of eval at this step is to give you the specific location of the breakpoint, so you know where to prioritize the architecture work. If the “turn depth” row of that decision table above shows the model forgetting early constraints after a dozen or so turns, the answer is clear: do not switch models, go externalize the state. Conversely, if the breakpoint shows the model violating explicit format constraints at only two or three turns with a very short context, that is what tells you the model capability itself is insufficient, and only then does switching models or moving to constrained decoding make sense.

Writing Decisions Down as Executable Rules

All four classes of response above end up as some rule inside your system. The executable form looks roughly like this:

#encoding:utf-8
#每条规则由"触发条件 -> 动作"构成，来自能力边界评测的结论
POLICY = {
    #来源：约束数量维度断点约 4 条，格式遵循在超过后明显下降
    "constraints_over_4": "merge_constraints_before_call",
    #来源：轮次深度断点约 12 轮，早期约束开始被遗忘
    "turn_depth_over_12": "externalize_state_and_reinject",
    #来源：长输入聚合类问题在较长上下文后塌陷
    "input_len_over_12000": "route_to_long_context_path",
    #来源：工具参数抽取在自由文本数字上有残余错误
    "tool_args_from_text": "type_check_then_confirm",
    #来源：信息不足时模型倾向猜测而非追问
    "insufficient_info_detected": "ask_user_not_generate",
}

def apply_policy(features: dict, base_call):
    "features 是运行时提取的输入特征，决定走哪条路径"
    actions = []
    if features.get("constraint_count", 0) > 4:
        actions.append(POLICY["constraints_over_4"])
    if features.get("turn_depth", 0) > 12:
        actions.append(POLICY["turn_depth_over_12"])
    if features.get("input_len", 0) > 12000:
        actions.append(POLICY["input_len_over_12000"])
    if actions:
        #把动作拼进提示结构，而不是靠模型自觉遵守
        actions.append("inject_structured_preamble")
    return base_call(features, actions)

Notice the design bias here: none of these rules relies on the model “remembering to behave”. When the constraint count is too high, the orchestration layer merges the constraints, rather than writing “please prioritize format when there are many constraints” into the prompt; when turn depth gets too deep, the orchestration layer injects the state, rather than hoping the model remembers. This principle deserves repeating: whatever eval found the model cannot do, do not hand it back to the model to do — hand it to a deterministic code layer.

Continuous Eval: Making Regression Detection Part of the Pipeline

A one-off eval is a one-off asset. What is actually valuable is an eval pipeline that can automatically tell you “did we regress” after every change. And this is precisely where the capability boundary profile pays off the most: because you already know where each dimension's breakpoint sits, regression detection can be directional. You do not need to run the complete eval set every time to discover any change on any dimension; you only need to keep enough sampling density near the breakpoints.

Tiering: A Fast Pipeline and a Slow Pipeline

Continuous eval has to be tiered or the cost makes it impossible to sustain. A practical split has three layers:

Tier	Sample size	Trigger	Purpose
Smoke tier	A few dozen	Every commit	Block obvious breakage, weighted toward format and structured output
Regression tier	Hundreds to low thousands	Daily or on every merge	Breakpoint probing per dimension, fixed subset
Full tier	The complete eval set	Weekly or before a release	Re-locate the breakpoints, regenerate the capability profile

The smoke tier has to be small and focused. Put in only samples whose scoring is deterministic, where a pass proves the pipeline is intact; do not try to discover capability problems with it. The regression tier is the daily workhorse, running the perturbation axis for each dimension over a fixed sample set that never participates in tuning, with the emphasis on whether the buckets near the breakpoint have moved. The full tier is the heavy job, and its output is the capability boundary profile itself, refreshed on a cycle.

The Right Criterion for Regression Detection

The most common mistake in regression detection is treating noise as signal. A bucket's pass rate came in two points below last time, so you declare a regression and roll back — when those two points sit entirely within sampling fluctuation. A correct criterion has to consider three things at once: whether the difference is significant relative to the noise floor, whether the direction is consistent across the discordant sample pairs, and whether the change matters in business terms. All three have to hold before you are allowed to call it a regression.

The code below gives the skeleton of a tiered pipeline. The focus is the regression decision rather than the scheduling details.

#encoding:utf-8
from dataclasses import dataclass
from typing import List, Dict

@dataclass
class BucketResult:
    "某个维度某个档位的聚合结果"
    dim: str
    level: int
    pass_rate: float
    n: int
    ci: tuple

def detect_regression(current: BucketResult,
                     baseline: BucketResult,
                     sd_single_run: float,
                     min_delta: float = 0.03) -> Dict:
    "回归判据：超过噪声下限 + 超过业务重要度 + 区间显著不重叠"
    delta = baseline.pass_rate - current.pass_rate  #正数表示变差
    exceeds_noise = delta > 2 * sd_single_run
    matters = delta > min_delta
    lo_cur, hi_cur = current.ci
    lo_base, hi_base = baseline.ci
    separated = (lo_base > hi_cur) or (lo_cur > hi_base)
    is_regression = exceeds_noise and matters and separated
    return {"dim": current.dim, "level": current.level,
            "delta": delta, "exceeds_noise": exceeds_noise,
            "matters": matters, "ci_separated": separated,
            "is_regression": is_regression}

def sweep_all(current: List[BucketResult],
              baseline_by_key: Dict, sd_single_run: float) -> List[Dict]:
    "扫全部档位，只返回需要关注的项，按恶化幅度排序"
    flagged = []
    for c in current:
        b = baseline_by_key.get((c.dim, c.level))
        if b is None:
            continue   #新增档位没有基线可比，先积累
        r = detect_regression(c, b, sd_single_run)
        if r["is_regression"]:
            flagged.append(r)
    flagged.sort(key=lambda r: r["delta"], reverse=True)
    return flagged

def gate_release(flagged: List[Dict], blocking_dims: set) -> Dict:
    "只有属于高风险维度的回归才拦截发版，其余记录但不阻塞"
    blocking = [r for r in flagged if r["dim"] in blocking_dims]
    return {"pass": len(blocking) == 0,
            "blocking": blocking,
            "watchlist": [r for r in flagged if r["dim"] not in blocking_dims]}

There are three deliberate design choices here. First, “exceeds the noise floor” is written as an explicit criterion, and sd_single_run is a required input — that value should come from the variance decomposition described earlier rather than being picked out of the air. Second, non-overlapping confidence intervals serve as the significance criterion, which is more honest than hardcoding a percentage threshold because it adapts to sample size automatically. Third, blocking a release is distinguished from merely watching, because making every dimension blocking means the entire gate gets ignored within weeks, and a gate that gets bypassed is the same as no gate at all.

There is one more practical point: the baseline has to be traceable. Every baseline score should record which model version, which prompt version, and which eval set version produced it. Otherwise, three months later, when you see a curve drop sharply at some point in time, you will have no way of knowing what happened then.

Production Feedback Looping Back

The last link in continuous eval is folding newly observed production failures back into the eval set. But this step has to be asynchronous and reviewed, and must never auto-promote. Production samples enter a candidate pool first, go through attribution (model problem or pipeline problem) and a difficulty assessment, and only those judged to represent a new stable capability dimension get promoted to official regression samples.

The cost of this discipline is that a production issue cannot become a test case instantly. The benefit is that the test set does not drown in business noise within a few weeks. In practice you usually strike a balance: failures that have recurred several times and already carry a clear attribution label take a fast path, and everything else goes through full review.

The Eval Set Itself Needs Versioning and Contamination Monitoring

After it has been running for a while, an eval set faces two forms of corrosion, and both are insidious. The first is sample decay: as the business changes, samples added early on gradually stop representing real inputs. They still run, they still produce a definite scoring result, so they sit quietly in the regression set, dragging the total score toward an outdated direction. The second is outdated conclusions: the composition of the eval set is unchanged, but the capability assumptions behind it have shifted — you originally assumed a class of input did not matter and included only two or three samples, and later discovered that class had become your main traffic.

The countermeasure is to give the eval set its own version and metadata. Every sample should record when it entered the eval set, where it came from, its attribution label, its currently tagged difficulty bucket, and the last time it was re-confirmed. With those fields you can answer some critical questions: which batch of samples has not appeared once in a real failure since some point in time (these are probably already solved, or possibly no longer occurring, and are worth reviewing); which attribution labels have samples in steady decline but have never been removed (the capability behind such a label may already be covered by your architecture work and can be downweighted); and whether any single source is overrepresented among the new samples, which would mean one channel has come to dominate the eval set.

Beyond metadata, you also need a periodic revisit mechanism: at intervals, draw a random batch of eval samples and have someone who knows the business make a fresh judgment — is this sample still important, does the scoring criterion still hold, is the difficulty bucket still accurate? This process does not need to cover every sample; drawing ten to twenty percent is enough to surface systematic drift. Its value is turning the invisible problem of “the eval set is quietly rotting” into a periodic check item with a named owner.

Distribution Drift: Letting Eval Weights Follow Real Traffic

The most common reason an eval set stops working is not that it got contaminated but that it was left behind by the times. Half a year has passed, the structure of the upstream data sources changed, the composition of the user population changed, the entry points into the product changed, and a systematic gap has opened between the distribution the eval set represents and the real distribution in production. Every score you see is still “accurate” — it is just accurately measuring a problem that no longer exists.

The way to detect that drift is to do a coarse clustering or template abstraction of production inputs and then periodically measure how the share of each cluster changes. In practice you do not need a sophisticated clustering algorithm: bucketing by input length, bucketing by whether specific entity types appear, and bucketing by task intent are enough to surface most of the drift. There are two key metrics: the absolute change in each cluster's share (an input type that used to account for one percent now accounts for thirty) and the change in eval set coverage (an input type appears in production that the eval set does not contain at all). The second is especially important, because it tells you directly that your eval is running in a blind spot.

Once you have found drift, do not immediately rebuild the entire eval set; that is expensive and unnecessary. The pragmatic move is to adjust weights rather than replace samples: keep the existing samples as a base layer so results stay comparable, add a sample layer covering the new distribution on top of it, and when reporting scores give two numbers — the “full-set score” and the “score reweighted by the production distribution”. The first is for regression comparison across versions; the second is for judging how the current system actually performs on real traffic.

A useful side benefit is that when an input cluster disappears entirely, you will see its reweighted weight approach zero in the report, and at that point you have an explicit basis for deciding whether to remove it from the base layer instead of cleaning up on a hunch. The maintenance rhythm of an eval set should be set by how fast the data changes, not by the calendar.

Closing the Loop Between Offline Scores and Online Metrics

Once the eval system is in place, you will almost certainly hit this disappointment: offline scores keep rising on every dimension while online business metrics do not move at all; or online user complaints rise sharply while offline scores show everything is fine. Both divergences need a dedicated explanation, otherwise the whole eval system loses its credibility after a few cycles.

Three Common Reasons Offline Rises and Online Does Not

First, the change landed on a dimension your business does not care about. Your eval set covers eight dimensions, but real traffic barely exercises some of them, so an improvement on that dimension is completely imperceptible to users. The typical case is spending two weeks lifting multi-turn consistency from seventy percent to ninety, while users never ask the same thing twice within one turn. The fix is to tag each dimension with a “business weight” and, when reporting, emphasize only the changes on high-weight dimensions.

Second, the change moved the error from one kind to another. The aggregate score is flat or slightly up, but the error type changed — previously wrong on content, now wrong on format, or previously caught by an automatic fallback and now silently passing. Per-dimension scores are not guaranteed to catch this either, because it requires comparing the distribution of error consequences, not the distribution of pass rates. So alongside pass rate, the eval report should also carry an error type distribution, and track its migration across versions.

Third, there are online factors the eval cannot cover. The most common are latency and cost. A change lifts accuracy by two points but pushes time-to-first-token high enough for users to notice, so the net effect is negative; or cost triples, which under a fixed budget means you can serve less traffic and therefore indirectly reduce coverage. The eval set has to include latency and cost columns, or offline conclusions are inherently incomplete.

Three Common Reasons Online Moves and Offline Does Not

First, the score distribution is hidden by the average. The overall pass rate is unchanged, but the pass rate on one small cluster collapsed, and that cluster is a small share of traffic made up of high-value users. This is exactly why the earlier emphasis on reporting per bucket matters: looking only at the total, such a regression is completely invisible.

Second, failure moved from the model side to something downstream. The model's output improved, but parsing, state management, or a business rule downstream developed a new problem. It shows up as a failure on the user side while the model's output is correct in the eval. The way to diagnose it is to look at the full trace of the production failure samples, not just the model's segment of the output.

Third, the distribution has drifted while the eval set stands still. This corresponds directly to the drift monitoring described in the previous section. Its signature is that the score slowly deviates starting from some day, and it is hard to tie to any single system change, because the real trigger sits on the traffic side.

Confirming That a Change Really Worked with Tiered Validation

Whichever divergence you are looking at, the correct response is to go back to tiered validation: first confirm that the change in online metrics is real, ruling out traffic mix changes and statistical noise; then attribute it to a specific component (model, prompt, retrieval, orchestration, downstream); and only then return to the offline eval to construct the corresponding regression samples.

The last link in that loop is the one most easily skipped: feeding newly discovered online failures back into the eval set. Skip it and the same problem keeps recurring, with every investigation starting from zero. The “production feedback loop” mechanism discussed earlier exists precisely to make sure that last link is never skipped. The maturity of an eval system should not be measured by how many scores it reports today, but by how many newly discovered problems get cemented into regression samples that do not recur.

Progress in 2026: Eval Methodology Itself Is Being Rewritten

The methodology described in the preceding sections is relatively stable. But over the last year or two the field of evaluation has been evolving in several directions, and they share one thing in common: they all try to solve the fundamental problem that traditional evals “measure inaccurately”. What follows discusses mechanisms and trade-offs, and does not involve any specific leaderboard numbers or paper conclusions.

Direction One: Dynamic Evals and Adversarial Test Sets

A fundamental weakness of a static eval set is that the moment it is published, it can be targeted. Fixed samples mean the model can — whether by accident during training or by design during evaluation — gradually adapt to that specific batch, so the score rises without capability rising in step. The idea behind dynamic evaluation is to make the test set itself a moving target: draw fresh samples on the spot from a much larger pool on each run, or have a generator produce new samples on the spot under predefined distributional constraints.

Adversarial test sets go further and make “finding the model's weaknesses” itself a continuous optimization target — a dedicated model (or a search process) keeps hunting for inputs that make the model under test fail, and the failures it finds enter the eval set automatically. That differs in kind from the traditional practice of “having humans construct a batch of hard samples and then freezing it”: it turns the eval set into a continuously evolving adversarial process.

These two directions solve the problem of overfitting the eval set, but they introduce a new engineering challenge: eval results become much less reproducible. If the samples differ on every run, then the difference between two runs now mixes in a new source of variance, sample difficulty fluctuation, and it is usually far larger than sampling variance. The workaround is to fix the generator's random seed and to use enough samples to dilute the difficulty fluctuation, at the cost of a significant increase in eval expense.

The practical advice for engineering teams is this: use dynamic eval for exploration and for discovering new weaknesses, and fixed eval for regression detection and version decisions. The two have different roles and should not replace each other. The value of a fixed eval lies precisely in the fact that its samples are constant, so any change can be attributed to the system.

Direction Two: Long-Horizon Evals for Agentic Scenarios

The hidden assumption of traditional evaluation is “one input, one output”, with the whole interaction ending within a few thousand tokens. When the thing under test becomes an agent that can autonomously execute dozens of steps, that assumption fails completely. There are at least five new difficulties:

• Success judgment becomes complicated: a task may “reach the goal by a roundabout route”, or “fail to reach the goal but do something more valuable”. Binary scoring loses a lot of information, so you have to score along a continuous spectrum of goal attainment.
• Small single-step errors may be self-corrected: a model with 95% single-step accuracy has an overall success rate on a twenty-step task far below 0.95 to the power of twenty, because many single-step errors get corrected by later steps. This means single-step accuracy has almost no predictive power for long-horizon performance, and you have to measure the whole trajectory.
• More sources of variability: the timing of tool returns, changes in external state, even time-dependent inputs will all make a rerun of the same task produce a different result.
• Failures have to be attributed to a step: when an entire task fails, locating “which step, and why” matters more than judging success or failure, but that requires storing the full execution trace.
• Cost becomes a first-class problem: long-horizon eval amplifies both token cost and latency cost, and an eval with no ceiling may cost more for a single run than the production service it is testing.

A few practical moves for long-horizon eval: give each task dual ceilings on steps and budget and record a failure when either is hit, rather than letting it run forever; store and analyze the trajectory as a first-class citizen rather than keeping only the final result; report the positional distribution of the failing step rather than only the overall success rate, because the former points directly at which component needs to change; and set a minimum viable success rate as the gate — since long-horizon task success rates are typically low, comparing near-zero absolute success rates is meaningless, and the comparison should really be about relative differences between versions on the same task.

Direction Three: Verifiable Tasks and Automatic Scoring

This is the most thorough answer to the problem that “the judge itself is unreliable”. Its core insight is this: if a task's correctness can be determined by a program, then the eval needs no model in the scoring loop at all.

Typical shapes for a verifiable task include: write code that satisfies a given set of test cases; produce a sequence of operations that reaches a target state, verified with a state machine; find a solution that satisfies a given set of constraints, checked with a solver; given a set of source records, produce a result set conforming to a schema and compare it with a validator. What they share is that verification is possible, and verification costs far less than generation.

Verifiable eval is valuable at three levels. First, it pushes measurement precision to essentially noise-free, which makes a judgment like “this is a two-point improvement” credible for the first time. Second, it lets difficulty adjust itself: add one assertion, tighten one constraint, add one test case, and the difficulty rises, without any human deciding “is this one hard”. Third, it supports automatic search out of the box — you can generate a large number of variants at different difficulty levels and locate the model's capability breakpoints.

The limitations are equally clear: verifiable tasks naturally cover only capabilities whose correctness is decidable. Writing style, tone appropriateness, how helpful an answer is, and the soundness of part of a reasoning chain cannot be verified this way. So verifiable eval does not replace traditional eval; what it replaces is the portion of traditional eval that could have been judged automatically all along but was scored with a weak judge. And in many teams' eval sets, that portion is a substantial share.

There is one more layer to watch: verifiable tasks are easy to game. If the assertions themselves are written loosely, the model may produce a solution that passes the assertions but is obviously wrong (every branch was checked, but an obviously checkable one was missed). The mitigation is to subject the assertions themselves to adversarial review — use a model dedicated to trying to generate a wrong solution that still passes the current tests, and see whether it succeeds. That “metatesting” process should happen during the assertion set design phase.

Direction Four: Aligning Eval with the Real Distribution

A more engineering-oriented but equally important direction is to anchor the eval's input distribution directly to real traffic and detect distribution drift automatically. The usual approach is to cluster production inputs, or abstract them into templates, into a number of clusters; periodically measure how the share of each cluster changes; and as soon as one cluster's share moves significantly up or down, treat that as a change in the business distribution that requires re-weighting the existing eval.

The value of this direction is that it answers a question left open earlier: does your eval set still represent today's business? An eval set usually fails not because it got contaminated but because it was left behind by the times — the shape of the business changed, the user population changed, the upstream data sources changed, and your eval set is still sitting on the distribution from six months ago. Periodic resampling and distribution alignment is a necessary maintenance action for keeping eval valid.

The Order in Which to Land All of This

Having read everything above, the easiest mistake to make is to try to do all of it at once: build the full eval set, sweep the perturbation axis on six dimensions, set up judge calibration, stand up a continuous pipeline. That order almost certainly fails, because the up-front investment it demands far exceeds what most teams are willing to spend on a one-off “reconnaissance”, and it shows no visible return midway, so it dies early.

If you can only do things in one order, I would suggest this one, sorted by “fastest to produce decision value”:

First, get the eval signal pipeline in place. Full-fidelity logging, an explicit negative signal, stable sample identity. This produces no scores at all, but it is the data foundation for everything that follows, and it simultaneously solves the pain of triaging production issues, so it has to be done regardless.

Second, do the two dimensions that can be scored deterministically: structured output and instruction following. Scoring is a function call, the cost is minimal, the results are noise-free, and improvements usually show up immediately. This step gives the team its first concrete, reproducible numbers for “can this model actually handle our business”, instead of an impression.

Third, build a tuning set of two or three hundred samples from production failures, sorted by attribution label. This is where real decision value starts: you can say “our most common failures are argument extraction and forgetting across long conversations” rather than “it feels a bit unstable”.

Fourth, cut every judge that can be replaced by a deterministic check. Squeeze the scoring noise out of the system. This step produces a noticeable jump in the credibility of existing scores even though nothing appears to have changed.

Fifth, measure one genuinely risky dimension — pick one of long context, multi-turn consistency, or tool calling. Sweep the perturbations to find the breakpoint and translate it into an architecture rule. This step is usually the first time you produce a “because of the eval result, we changed the architecture” moment, and it is the key to earning organizational support for doing evals at all.

Sixth, introduce confidence intervals and variance decomposition. From then on every score carries an interval, and no decision gets made on a single number. This step has a high cognitive cost but is absolutely necessary, and it is reasonable to schedule it once the team has built trust in the eval.

Seventh, convert everything that can be judged automatically into verifiable tasks, and stand up the three-tier continuous pipeline. This step is about scale. If the first six are not in place, standing up the pipeline only produces a gate nobody trusts and therefore nobody looks at.

A Few Wrong Turns That Are Easy to Take

To close with a summary of the error patterns that keep recurring in practice. They are more likely than technical details to break an entire eval system.

Treating eval as a one-off report. The value of eval lies in continuous comparison, not in any single conclusion. A report that is only ever run once expires the moment you finish reading it, no matter how good the numbers look.

Chasing a single composite score. A composite score hides the distribution and has no corresponding response attached to it. A per-dimension pass rate table is more useful than any single number.

Using a weak judge to score large numbers of decidable samples. That makes measurement precision the bottleneck of the entire system, and a bottleneck that is hard to notice — it produces no errors, only a pervasive feeling that you cannot say which change actually worked.

Blaming pipeline problems on the model. A retrieval miss, a truncated context, dirty upstream data — all of these present as “the model got it wrong”. Without this attribution step, you will keep tuning something that was never broken.

Letting the test set participate in tuning. The moment test set scores start guiding prompt iteration, the test set has become a tuning set, and every subsequent judgment about “is this new change really better” becomes meaningless.

Keeping things the model cannot do assigned to the model. When eval tells you a dimension is out of reach, the correct response is to add deterministic validation, externalize state, or switch models — not to write the requirement into the prompt in more detail and wait for a miracle.

Capability boundary probing is, in essence, the conversion of the vague judgment “the model is strong” into a set of specific conclusions that have positions, have noise levels, and map onto concrete responses. It provides no certainty — no eval can — but it turns an uncertain argument into a discussable engineering judgment, and that in itself is enormous progress.

How to Tell Whether the System Is Actually Working

An eval system very easily degenerates into a form of self-entertainment: the reports come out on schedule, the score curves look great, and no product decision has ever changed because of them. Judging whether it is effective does not require complicated metrics; a few plain observations are enough.

The first signal is that the nature of the selection discussion changes. If the discussion is still stuck on “which model is stronger”, the eval is not providing enough information. If the discussion has become “on the constraint count dimension, which model should we route to”, then it is already supporting decisions. This is the most direct and the most honest test: find any model selection meeting from last month and see whether anyone cited an eval conclusion.

The second signal is whether architecture changes can be traced back to a breakpoint. A good system leaves a record of every architectural compromise made for the sake of model capability: which dimension was measured as failing, under what conditions, and therefore which compensating mechanism was adopted. If someone wants to remove a state externalization layer or a validation layer, they should be able to pull up the original eval data and show what level you would fall back to without it. Compensating layers without that traceability usually get deleted by mistake during some “code cleanup”.

The third signal is the recurrence rate of production issues. This is the hardest of the metrics. If the same class of failure keeps being rediscovered, the feedback loop is not working — the problem was found but never cemented into a regression sample, so the next release will commit it again. If that ratio keeps falling, the loop is closed. Conversely, if new problems are discovered faster and faster while old ones are never resolved, your eval set is almost certainly expanding rather than converging.

The fourth signal is the signal-to-noise ratio of the eval itself. In a healthy system, the statement “this change pushed the score up” should frequently be verifiable rather than frequently be noise. If the team has already begun to habitually ignore changes in the eval results, then the noise level has passed the signal level, and the right move is to go back and improve the scoring method or increase the number of repeats — not to keep adding samples.

A final note on return on investment, as a matter of realism: the payoff of capability boundary probing does not show up in the first month. Its value comes from the comparison data accumulated over the long run — every model upgrade, every prompt change, every architecture adjustment leaves a tick mark under the same ruler. The ruler itself is worth very little; the number of tick marks is where the value is. That is also why this article keeps insisting on not doing one-off evals: a one-off eval leaves only a zero, and a zero cannot define a direction.