I. Why test-time compute deserves its own research axis
Over the past two years, the narrative around model capability gains has been almost entirely a story of training-time scaling: more parameters, more pretraining tokens, longer reinforcement learning runs. But by 2025–2026 the industry had converged on a claim that essentially nobody disputes any more: given the very same model weights, spending a bit more compute at inference time is enough to move accuracy by an order of magnitude. How large is that shift? Large enough that the reasoning budget has gone from a research trick to a product contract — OpenAI turned "how much to think" into a reasoning-effort setting on the o-series models, Anthropic's extended thinking requires you to state a thinking budget in tokens, Google exposes a thinking budget parameter on Gemini 2.5 (setting it to 0 disables thinking, setting it to a dynamic value hands the decision to the model itself), and Qwen3 turns the thinking switch into dual think / non-think modes on a hybrid reasoning model. Competition among frontier products has already shifted, in part, from "who has the bigger model" to "who allocates the reasoning budget better."
The substance of the shift is this: inference is no longer a one-shot forward pass; it is an online search and resource allocation problem. You have some compute and some milliseconds in hand, and you need to give one question an answer that is as right as you can make it. You can sample eight independent reasoning chains and vote, or take only two chains and search each one deeply token by token, or spend two hundred tokens first to probe the difficulty and only then decide whether it is worth spending more. Different allocations draw on the same budget, yet the accuracy they land on can differ by a dozen percentage points. That is the root reason Test-Time Compute Scaling became an independent research axis of its own.
1.1 The structural difference in marginal returns
The returns curve for training-time scaling is unambiguous and, frankly, depressing: doubling the parameter count buys roughly power-law gains, and it is the kind of power law that is steep early and very flat later. The returns curve for test-time scaling has a different shape. It is extremely steep in the low-budget regime — a problem whose single-shot pass rate is only forty percent may jump to seventy percent after majority voting over sixteen samples from the same model — but it saturates too, just at a different place. The slope, the knee, and the ceiling of this curve are determined by three entirely different factors: the probability that the base policy can sample a correct answer at all, that is, coverage; and whether you have any way of recognizing that correct answer among a pile of candidates, that is, discrimination.
Separating those two things is the starting point for every piece of methodology in this article. A great many people conflate "test-time scaling does not work" with "the verifier is not strong enough," then blindly add sampling, and end up with money spent and accuracy unchanged.
1.2 The budget is already a first-class parameter
Lay the budget knobs from the various vendors side by side and you find a very clean maturity signal: if a capability is stable enough to be shipped as a parameter, it has already moved from research trick to product contract.
| Platform and model | Budget parameter | Semantics | Practical takeaway |
|---|---|---|---|
| Gemini 2.5 family | Thinking budget | Upper bound on internal thinking tokens; a dynamic value means the model decides, zero disables thinking | At low budgets reasoning chains get markedly shorter, and the gains on hard math and code tasks disappear first |
| Claude extended thinking | Thinking budget tokens | Number of tokens available to the thinking phase, with a minimum-value constraint | This is a separate budget from max output tokens; plan them together, or thinking will eat into your output allowance |
| OpenAI o-series and later reasoning models | Reasoning effort setting | Discrete settings such as low, medium, and high map onto an internal thinking budget | This is a product-level abstraction; do not treat it as a precise compute control knob |
| Hybrid reasoning models such as Qwen3 | Thinking switch and thinking length | The same weights switch between thinking and non-thinking behavior | Hybrid mode is the cheapest way to land difficulty routing: simple requests go non-thinking, complex requests go thinking |
The table by itself states the problem this article is about: budget management is no longer a trick from a paper, it is the everyday engineering problem every inference service faces once it is in production.
1.3 When test-time scaling should not be used
Just as important is drawing the boundary. In the following three situations, adding budget is basically waste:
- The task is not decidable. Open-ended writing, creative generation, analytical tasks with no standard answer — without a reliable verification signal, the diversity that scaling buys cannot be collapsed into correctness. In such scenarios the budget belongs on better evaluation criteria, not on more samples.
- Coverage is already saturated. If the single-shot pass rate is already high, or repeated samples agree completely, the marginal return on additional budget is close to zero. The only meaningful move left is to change the model or change the prompt.
- The latency budget is extremely tight. In interactive settings, a few tens of milliseconds of latency is already unusable, and at that point no scaling scheme can even be discussed — you have to change the architecture.
Writing these three rules at the very top of a design document saves an enormous number of pointless experiments.
II. Formalization: what a reasoning budget actually is
You cannot talk about allocation before you talk about measurement. This section decomposes the reasoning budget into computable terms; every method that follows can be read as a different policy over that set of variables.
2.1 Three budget dimensions
The crudest description is "give the model more tokens and it gets more accurate," and that is not good enough. Test-time compute is in fact controlled by three independent knobs:
- Width: the number of independently sampled reasoning chains, or the number of branches kept alive simultaneously in a search tree.
- Depth: the maximum number of tokens each chain is allowed to generate.
- Discrimination cost: how many times the verifier is called, and how long an input a single scoring pass has to read. Scoring usually re-reads the whole chain, and in practice this overhead is severely underestimated.
The total budget can then be written roughly as "width times depth times the unit generation price, plus the number of discrimination calls times the unit scoring price." What actually determines what you pay is the total token count; what actually determines the user experience is wall-clock time. Those are not the same thing, and this is the most easily confused point in all of test-time scaling.
2.2 Cost tracks the total, latency tracks the longest chain
Sample sixteen chains in parallel at two thousand tokens each and you have thirty-two thousand generated tokens in total; but if the hardware can only process eight at a time, the user's wait is determined by the slowest chain across the two rounds. Spend the same budget as eight chains of four thousand tokens instead and the total is the same, yet wall-clock time may actually get longer, because depth cannot be parallelized within a round. Conversely, if the scheduler is well built, width can be absorbed entirely by batch throughput, while depth is hard-serial. This means width and depth have completely different price tags in latency, and depth is usually the more expensive one.
"""把预算拆成宽度 x 深度，估算成本与墙钟延迟。
总词元决定钱，批轮次与最长链决定用户等待时间。
深度在批内无法并行，因此同样的词元量下墙钟时间可能差出数倍。
"""
GEN_RATE = 120.0 # 单序列解码速率（词元/秒）
BATCH_CAP = 8 # 单请求内可并行的序列数上限
BATCH_GAIN = 0.9 # 批内每多一路序列带来的吞吐增益
def estimate(k: int, t: int) -> dict:
"""给定宽度 k 与深度 t，返回成本与延迟估算。"""
if k < 1 or t < 1:
raise ValueError("k 与 t 必须为正整数")
rounds = -(-k // BATCH_CAP) # 向上取整的批轮次
per_round = min(k, BATCH_CAP)
speedup = 1.0 + BATCH_GAIN * (per_round - 1)
rate = GEN_RATE * speedup
return {
"k": k,
"t": t,
"tokens": k * t,
"cost_k": round(k * t / 1000.0, 1), # 成本单位：千词元
"latency_s": round(rounds * t / rate, 1),
}
if __name__ == "__main__":
print(f"{'k':>4}{'t':>7}{'tokens':>9}{'cost(k)':>9}{'latency(s)':>12}")
for k, t in [(1, 2000), (8, 2000), (16, 2000), (8, 4000), (32, 1000), (4, 8000)]:
r = estimate(k, t)
print(f"{r['k']:>4}{r['t']:>7}{r['tokens']:>9}{r['cost_k']:>9}{r['latency_s']:>12}")
# k t tokens cost(k) latency(s)
# 1 2000 2000 2.0 17.5 最便宜，但只看单次通过率
# 8 2000 16000 16.0 20.4 同词元量下并行更划算
# 16 2000 32000 32.0 37.9
# 8 4000 32000 32.0 56.5 同样词元，深度把延迟拉高
# 32 1000 32000 32.0 15.1 同样词元，宽度最省墙钟
# 4 8000 32000 32.0 74.0
The conclusion is direct: for the same compute budget, allocating it into width or into depth can move wall-clock latency by a factor of four or five. This is not theoretical fastidiousness — it directly decides whether you can turn "spend more compute" into a feature users are willing to accept.
2.3 Coverage and decision: the gap between two metrics
The gains from test-time scaling can be pulled apart with a pair of metrics. Coverage measures whether it is possible to get it right: the probability that, over some number of samples, a correct answer appears at least once. Decision quality measures whether it actually gets it right: the correctness of the single answer the system finally emits.
- High coverage, low decision quality: the model can sample the correct answer, but voting or scoring cannot pick it out. The bottleneck is the discrimination signal — you need a stronger verifier, or the problem itself is too poorly decidable.
- Low coverage: no allocation of budget helps. The bottleneck is the search space of the base policy; you have to go back to the training side, or switch to depth-first search so the model gets a chance to "take a different route."
- Both high: the budget is already saturated, and adding more spend is waste — you should consider changing the model or changing the prompt instead.
Treating that gap as your debugging entry point is far more effective than staring at the final accuracy and guessing. It should also become a fixed output of every experiment, because the direction of the gap directly decides whether the next thing to invest in is the sampler or the verifier.
2.4 The three currencies of a budget
Finally, one level of analysis that is often overlooked: budget is not just tokens. In a real system there are at least three currencies that conflict with one another —
- Money: determined mainly by total generated tokens and discrimination calls, independent of concurrency.
- Latency: determined by the longest chain and the number of batch rounds, and a hard constraint users can actually feel.
- Quota: in a rate-limited service, what is genuinely scarce is requests per minute and concurrent slots, and there batching efficiency matters more than single-request efficiency.
The optimal configuration of the three is frequently different. An overnight batch job can ignore latency completely and push the whole budget into width in exchange for accuracy; an online interactive scenario is exactly the opposite and has to squeeze the budget down onto a fast path with extremely shallow depth and very small width. There is no budget policy that is optimal for every scenario, only policies that are optimal under a particular set of constraints.
III. Method family one: parallel sampling and self-consistency
This is the oldest and also the most robust method in test-time scaling. The self-consistency mechanism introduced by Wang and colleagues in 2022 is remarkably plain: sample several reasoning paths for the same question, extract the final answer from each, normalize them, and take a majority vote. Its value lies in the fact that it requires no external verifier whatsoever — as long as the model can sample and you can compare answers, it works. That is also why it has the highest adoption rate in production: no extra components means no extra failure points.
3.1 Why majority voting works
The mechanism by which self-consistency works looks like this: chain-of-thought failures are rarely random. They concentrate on a small number of shallow shortcuts — dropping a condition, reading a minus sign as a plus sign, miscomputing one multiplication and then staying wrong all the way down. Those error modes attract a substantial share of the samples, while the correct path, though it may have many variants, shares a deeper common structure. Once the number of samples is large enough, the correct attractor accumulates into the mode of the answer distribution and drowns out the votes of those shallow errors.
This mechanism has an important corollary: the ceiling of self-consistency is set by whether "the correct answer becomes the mode of the answer distribution," not by whether "the correct answer was sampled at all". When several candidate answers are all equally correct but no normalization is applied, the correct answer's votes get diluted and voting stops working. That explains a recurring production incident: the model was plainly right, but because the answer formats were not unified it got voted into the minority.
3.2 Searching sampling hyperparameters: the coupling between temperature and truncation
Parallel sampling requires a temperature, but temperature also degrades the quality of each individual chain. That is the central tension in test-time scaling: a low temperature raises single-sample correctness, a high temperature raises the diversity across candidates. In practice somewhere between 0.6 and 0.8 is the mainstream range for math and code tasks, and pushing higher produces a great deal of syntax errors and degenerate repetition.
Another commonly overlooked detail: reasoning models trained with reinforcement learning are markedly more temperature-sensitive than their base models. Such models already produce long, well-structured chains at temperature zero, so sampling one zero-temperature "anchor sample" first is often the highest-quality option, and only the subsequent exploratory sampling needs heat. If you set every sample to a high temperature, you are throwing away a free source of accuracy.
Truncation length has to be tuned together with the temperature. A high temperature lengthens generations, and if the truncation cap is set right at the tail of the length distribution you will end up with a large number of samples that were "cut off halfway through reasoning"; answer extraction fails on those samples, so you waste budget and pollute the voting distribution. The safe move is to put the truncation cap above the ninety-fifth percentile of the generation length distribution and to track the truncation rate explicitly — once the truncation rate climbs above a few percent, that parameter combination has a problem.
3.3 Answer normalization: the most underestimated engineering detail
Half of whether voting works at all depends on how well normalization is done. At minimum you need to do the following: unify whitespace and full-width characters; unify the surface form of mathematical expressions (fraction notation versus decimal notation, with or without a dollar sign, should all count as the same answer); compare numeric answers with floating-point tolerance instead of string comparison; strip units and percent signs; sort structured answers field by field before comparing; and fold together synonymous phrasings inside the answer, such as hours versus minutes. None of this work is large, but it directly decides whether voting is usable at all.
There is also a hidden trap: residues of the reasoning process left inside the answer. Many models append an explanation or a self-check after the answer, and if the extraction function slices on "the last number" or "the last line," it will extract something completely different on some of the samples. Extraction rules must be validated separately across languages, formats, and lengths; confirming them on a few hand-picked examples is not enough.
3.4 Streaming early stop: wrapping up early on vote lead
A very practical optimization: majority voting does not need to wait for every chain to finish. Once some answer's vote count already exceeds the number of votes still possible, the result can no longer be overturned, so you can immediately stop the unfinished samples and return. Formally: if the votes already counted are v, the number of completed samples is d, and the total budget is k, then the necessary and sufficient condition for exiting early is v > k - d.
On problems with a sharp answer distribution this cuts the average number of samples by more than half; on problems with a flat distribution it almost never fires — which is exactly the behavior you want: pay a deterministic price only for questions that are determinate. One implementation note: this early stop is provably safe, it needs no extra assumptions, so it should be on by default in any system.
"""自一致性采样：答案归一化 + 可证明安全的流式早停。
只依赖标准库。真实使用时把 generate_one() 换成实际模型调用即可。
"""
from __future__ import annotations
import re
from collections import Counter
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
from math import isclose
# ---------- 答案归一化 ----------
def normalize_answer(raw: str) -> str:
"""把等价答案折叠成同一个字符串。漏掉任何一步都会让投票失效。"""
s = raw.strip().lower()
s = s.replace("\\left", "").replace("\\right", "")
s = re.sub(r"\\[dt]frac\{([^{}]+)\}\{([^{}]+)\}", r"(\1)/(\2)", s)
s = s.replace("^\\circ", "").replace("^{\\circ}", "").replace("\\%", "")
s = s.replace("$", "").replace("\\text{}", "").replace("\\,", "")
s = re.sub(r"[^\w\s\-\.\+/=\(\)]", " ", s)
s = re.sub(r"\s+", " ", s).strip()
try: # 数值答案：转成定长浮点串做容差折叠
return f"{float(s):.6g}"
except ValueError:
return s
def answers_equal(a: str, b: str) -> bool:
try:
return isclose(float(a), float(b), rel_tol=1e-6, abs_tol=1e-9)
except (ValueError, TypeError):
return a == b
# ---------- 投票 ----------
def vote(candidates: list[str]) -> tuple[str | None, Counter]:
"""按等价簇投票：先合并互相等价的答案，再统计簇规模。"""
clusters: list[list[str]] = []
for c in candidates:
for cl in clusters:
if any(answers_equal(c, m) for m in cl):
cl.append(c)
break
else:
clusters.append([c])
counts: Counter[str] = Counter()
for cl in clusters:
counts[max(cl, key=len)] += len(cl) # 用簇内最长串做代表
if not counts:
return None, counts
return counts.most_common(1)[0][0], counts
def can_stop(counts: Counter, done: int, budget: int) -> bool:
"""可证明安全的早停：领先票数 > 剩余可能票数，结果已不可能被翻盘。"""
if not counts:
return False
top_votes = counts.most_common(1)[0][1]
return top_votes > budget - done
# ---------- 主流程 ----------
def self_consistency(question: str, budget: int, workers: int = 8) -> dict:
counts: Counter[str] = Counter()
done = 0
with ThreadPoolExecutor(max_workers=workers) as pool:
pending = {pool.submit(generate_one, question) for _ in range(budget)}
while pending:
finished, pending = wait(pending, return_when=FIRST_COMPLETED)
for fut in finished:
counts[normalize_answer(fut.result())] += 1
done += 1
if can_stop(counts, done, budget):
for f in pending:
f.cancel()
pending = set()
break
if not counts:
return {"answer": None, "samples": done}
top, n = counts.most_common(1)[0]
return {
"answer": top,
"votes": n,
"samples": done,
"early_exit": done < budget,
"agreement": round(n / max(done, 1), 3),
}
def generate_one(question: str) -> str:
from generator import sample # 真实推理调用（需带停止条件与截断上限）
return sample(question, temperature=0.7, max_tokens=4000)
if __name__ == "__main__":
print(self_consistency("求 17 乘 23 的结果", budget=12))
# 期望形如：answer='391' votes=9 samples=9 early_exit=True agreement=1.0
3.5 Three failure scenarios for self-consistency
Self-consistency is not a panacea. There are three classes of situation in which it clearly fails, and engineering has to recognize them up front, otherwise you keep burning budget on them:
- A large or open answer space: writing, summarization, multi-turn dialogue, generating a function that runs — there is almost no discrete answer to vote over, so it degenerates into random sampling. In this kind of scenario the act of "voting" itself is meaningless.
- Multiple equivalent answers without normalization: an overestimated correctness rate gets diluted, as discussed in detail above. The symptom is that accuracy goes down rather than up once voting is added.
- A stronger verifier already exists: once coverage is already high (say ninety percent) while the verifier's discrimination is ninety-five percent, reranking by verifier clearly beats majority voting. Piling on more samples at that point is waste; the budget should move to scoring instead.
The rule of thumb: if the answer can be discretized and normalized, voting is worth it; if it cannot, don't vote — go verify or improve the evaluation. The practical value of this rule is that it eliminates a batch of doomed designs already at the design stage.
IV. Method family two: verifier-guided reranking
Self-consistency substitutes frequency for correctness. Verifier reranking swaps in a different signal: train or construct a scorer, score every candidate chain, and take the one with the highest score. On the surface this is only a change to the selection rule, but it moves the bottleneck of the entire system somewhere else — from now on you no longer need "the correct answer to happen to be the mode"; you need a discriminator that can recognize the correct answer.
4.1 The essential difference between two selection rules
Laying out the differences between self-consistency and verifier reranking in a single table is more useful than any qualitative debate about which one is better.
| Dimension | Self-consistency majority vote | Verifier reranking |
|---|---|---|
| External scorer required | No | Yes, and it is the core system component |
| What sets the decision ceiling | The shape of the answer distribution: whether the correct answer is the mode | The discrimination ceiling of the scorer |
| Typical failure | Equivalent correct answers get diluted; consistent collective errors | The scorer hands high marks to "plausible-looking wrong answers" |
| Cost structure | Cost is almost entirely in generation | Generation plus scoring, and scoring has to re-read the whole chain, which is not negligible on long chains |
| Compatibility with search | Weak; inherently a width strategy | Strong: it works both for end-of-chain reranking and as a pruning heuristic inside search |
| Interpretability | High: the vote distribution directly says how certain the answer is | Low: scores are often uncalibrated, so the absolute value cannot be interpreted |
There is an easily overlooked practical problem here: scoring is not free. The scorer has to read every candidate chain, and a long chain can run to several thousand tokens, which is a pure forward computation. For long-chain reasoning tasks, discrimination overhead can take a substantial share of the total budget. So when you decide "should I add a verifier at all," what you should compute is "width times (generation length plus scoring input length)," not generation alone.
4.2 The four implementation forms of a scorer
- Outcome reward model: train the scorer on the final answer only. The data is cheap, the annotation only sees the endpoint, and it is sensitive to final correctness. Its weakness is that credit assignment on long chains is terrible — if a fifty-step chain goes wrong at step three, what it learns is only "this chain is wrong," not "which step was wrong."
- Process reward model: assign a local correctness score to every step of the reasoning, which is naturally usable for pruning during search and gives much finer credit assignment. The cost is expensive annotation: you need per-step labels, and annotators can hardly agree on whether a given step "is correct in itself but not optimal in that context."
- Generative verifier: rather than building a separate scoring model, have the model emit the judgment token — whether it got the problem right or wrong — directly at the end of the prompt, reframing verification as one more next-token prediction. The upside is that the verifier shares representations with the generator and can be fine-tuned end to end; the downside is that it inherits the generator's biases just as faithfully.
- Programmatic verification: symbolic computation for math, unit tests for code, constraint solvers for logic problems, and re-executing the query for data analysis. This is the only category of genuinely trustworthy verifier, and it is exactly why competition math and code became the main battleground of test-time scaling research — not because those two kinds of task are simple, quite the opposite, but because their answers can be confirmed by a machine.
4.3 Why strong reasoning models tend to skip process supervision
There is one decision in the DeepSeek-R1 technical report that is worth reading more than once: during training they did not introduce a process reward model and insisted on reinforcement learning with verifiable outcome rewards. One of the stated reasons is that process annotation has to stay consistent with the data distribution of the final policy, and since the policy drifts continuously during reinforcement learning, the distribution of the process annotations drifts with it — until the process reward model itself becomes the bottleneck on accuracy.
This is a counter-intuitive conclusion drawn from large-scale practice. It does not mean process supervision is useless; it means that its maintenance cost has been severely underestimated over long training runs. Translated into the language of test-time scaling: the signal of a process reward model decays as the policy's capability grows, and you have to keep investing resources in re-annotating, re-adapting, and adversarially testing it. That expense often exceeds the benefit it brings. The pragmatic conclusion is this — if your test-time scaling only does end-of-chain reranking, an outcome reward is enough; process supervision only becomes irreplaceable when you are genuinely running multi-step search and need intermediate pruning signals.
4.4 The most dangerous verifier failure: correlated errors
If the verifier and the generator share the same training data, the same architecture, or are even just fine-tuned from the same base, they will make correlated mistakes: for a problem that happens to be one this class of model gets systematically wrong, the generator errs once and the verifier errs again, and the reranking is not merely useless but can end up worse than majority voting — because majority voting at least has the weak signal of "collective agreement" as a backstop, whereas the verifier supplies what looks like a precise error signal.
Three actionable mitigations:
- Make the verifier heterogeneous to the generator. A different base model, different training data, or programmatic verification outright. This is the most thorough approach and also the most expensive one.
- Calibrate the scores. Uncalibrated scores cannot be compared directly — feed a batch of samples with known answers into the verifier, fit isotonic regression or logistic regression, and map the scores into probabilities. Calibration brings an extra benefit: the score itself becomes a usable stopping signal.
- Treat the verifier as a feature, not as a judge. When several signals are available (majority votes, verifier score, length, position of the answer, format regularity), stacking a small logistic regression on top is often more robust than trusting a single verifier.
4.5 Hierarchical reranking: spending the budget where it cuts
Finally, a purely engineering optimization. The cost of verifier reranking grows linearly with the number of candidates, but for the vast majority of candidates the scoring is unnecessary. You can split it into two tiers: first generate at scale with a cheap model and coarse-rank with a cheap heuristic, then use a strong model to regenerate and score precisely only for the top handful of distinct answers after deduplication. That pushes discrimination cost down by a large margin while having very little effect on final accuracy.
"""验证器重排：分层生成 + 并行打分 + 校准阈值停止。
1. 分层生成：便宜层开大 k，强模型只重跑粗排后得分最高的 m 个不同答案
2. 并行打分：判别是纯前向，天然可批处理
3. 校准停止：分数达到阈值直接返回，把剩余预算还给系统
"""
from __future__ import annotations
import math
from collections import Counter
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass
@dataclass
class BoNConfig:
k_cheap: int = 16 # 便宜层采样条数
m: int = 6 # 强模型只重跑粗排后前 m 个不同答案
k_strong: int = 4 # 每个目标答案的强模型重生成条数
tau: float = 0.86 # 校准后的接受阈值
verify_workers: int = 8
@dataclass
class Candidate:
text: str
answer: str
tier: str
score: float = 0.0 # 原始打分
prob: float = 0.0 # 校准后的置信概率
def calibrate(raw: float) -> float:
"""把原始分映射成概率。真实做法是对标注样本做等渗回归，此处简化为逻辑函数。"""
return 1.0 / (1.0 + math.exp(-(raw - 3.0)))
def best_of_n(question: str, cfg: BoNConfig) -> dict:
cands: list[Candidate] = []
# ---- 第 1 层：便宜、宽、并行 ----
for text, ans in sample_many(question, cfg.k_cheap, tier="cheap"):
cands.append(Candidate(text=text, answer=ans, tier="cheap"))
# ---- 廉价粗排：只保留前若干个"不同答案"，避免在同一个答案上重复投入 ----
ranked = sorted(cands, key=lambda c: -c.score)
top_answers = []
for c in ranked:
if c.answer not in top_answers:
top_answers.append(c.answer)
if len(top_answers) >= cfg.m:
break
# ---- 第 2 层：强模型重生成，验证"这个答案是否真的站得住" ----
for ans in top_answers:
for text, a in sample_many(question, cfg.k_strong, tier="strong", hint=ans):
cands.append(Candidate(text=text, answer=a, tier="strong"))
# ---- 并行打分 + 校准 ----
with ThreadPoolExecutor(max_workers=cfg.verify_workers) as pool:
scores = list(pool.map(lambda c: raw_score(c, question), cands))
for c, s in zip(cands, scores):
c.score, c.prob = s, calibrate(s)
cands.sort(key=lambda c: -c.prob)
best = cands[0]
gap = best.prob - cands[1].prob if len(cands) > 1 else 1.0
# 双信号早停：不仅要分高，还要明显甩开第二名（否则说明打分器没区分度）
stopped = best.prob >= cfg.tau and gap >= 0.10
return {
"answer": best.answer,
"prob": round(best.prob, 4),
"runner_up_gap": round(gap, 4),
"stopped_early": stopped,
"candidates": len(cands),
}
def raw_score(cand: Candidate, question: str) -> float:
from verifier import score # 真实打分器（奖励模型/生成式验证器/单元测试）
return float(score(cand.text, question))
def sample_many(question: str, k: int, tier: str, hint: str | None = None):
from generator import sample
for _ in range(k):
text = sample(question, tier=tier, hint_answer=hint)
from scoring import extract_answer
yield text, extract_answer(text)
if __name__ == "__main__":
r = best_of_n("求 1234567 的各位数字之和", BoNConfig())
print(r)
# 关注 runner_up_gap：若它很小，说明打分器在这题上没区分能力，
# 此时即便绝对分很高也不应该早停
Notice the runner_up_gap in the output. If the calibrated probabilities of the first and second place differ by only 0.02, then the scorer basically has no discriminative power on this problem, and early stopping based on "the score is high" is unreliable — a high score may simply mean the scorer runs high in general, not that it is genuinely confident. This is a practical and nearly zero-cost calibration self-check, and it is worth running once for every model version.
V. Method family three: choosing between process reward and outcome reward
This section is worth pulling out on its own, because it is frequently misjudged. The default assumption is that "process supervision is more advanced than outcome supervision," so everything gets a process reward model, and then you get dragged down by the maintenance cost. The essential difference between the two signals compresses into one sentence: an outcome reward tells you whether the path works, a process reward tells you where to turn at this step. The former is cheap and robust; the latter is fine-grained but hard to keep alive.
5.1 A dimension-by-dimension comparison
| Dimension | Outcome reward model | Process reward model |
|---|---|---|
| Source of the supervision signal | Whether the final answer is right or wrong | Whether each step is correct and making effective progress |
| Data cost | Low: trajectories are generated automatically and only the endpoint is labeled | High: per-step annotation is required, labor intensive and hard to standardize |
| Credit assignment granularity | One signal for the whole chain, extremely coarse | Step level, suited to long chains and pruning |
| Resistance to distribution drift | Strong: it only cares about the endpoint, not the reasoning style | Weak: annotation is tightly coupled to a particular reasoning style and goes stale the moment the policy changes |
| Role inside search | End-of-chain reranking | A local heuristic for pruning at every step |
| Typical failure | Cannot tell "wrong step but right answer" apart from "long way round but right answer," so it learns surface patterns | Reward hacking: it learns to emit whatever form satisfies the scorer instead of actually reasoning |
The choice between the two reduces to a single decision rule: use an outcome reward if you only do end-of-chain reranking; a process reward is needed only for multi-step search. This rule is right in the overwhelming majority of real systems.
5.2 Three scenarios where process reward really is the right answer
Process reward is not useless; it simply has a much better cost-performance ratio under specific conditions:
- Pruning a search tree. When you run multi-step search, the only possible source of an evaluation signal for intermediate states is the process reward; there is no other. If expanding a branch costs several hundred tokens, you cannot afford to run full verification at every intermediate node.
- Localizing failures in long-chain reasoning. If a hundred-step chain answers incorrectly, you need to know which step went off track, and only a process reward provides that signal. This is especially useful for data cleaning and hard-example mining.
- Distillation and filtering of supervised data. Using process scores to filter high-quality intermediate steps produces cleaner supervision data more easily than filtering on final answers alone.
And when you only do end-of-chain reranking, the chains are not long, and you already have plenty of automatic annotation capability, an outcome reward is the overwhelmingly correct choice.
5.3 How step scores aggregate into a single signal
A process reward produces step-level scores, while search needs a scalar. The aggregation method directly shapes pruning behavior: taking the minimum is the most conservative, since any step collapsing collapses the whole chain, and it suits hard pruning; a product penalizes long chains by construction and suits ranking after normalization; a first-error penalty explicitly distinguishes "wrong early" from "wrong late," and the former is usually far less salvageable. The most robust combination in practice is this: use the minimum as the pruning threshold and the log-product as the ranking score, with the two playing clearly separated roles.
"""过程奖励信号的聚合与提前终止。step_scores 中 None 表示该步未打分。"""
from __future__ import annotations
from math import exp, log
def min_aggregate(step_scores: list[float | None]) -> float:
"""硬剪枝信号：任一步崩则整链崩。对推理早期的错误最敏感。"""
vals = [s for s in step_scores if s is not None]
return min(vals) if vals else 0.0
def logprod_aggregate(step_scores: list[float | None], eps: float = 1e-3) -> float:
"""排序信号：取对数乘积并对链长归一化，避免长链被系统性惩罚。"""
vals = [max(s, eps) for s in step_scores if s is not None]
return exp(sum(log(v) for v in vals) / len(vals)) if vals else 0.0
def first_error_index(step_scores: list[float | None], threshold: float = 0.5) -> int | None:
"""定位首错步骤：用于长链失败归因，也是"是否值得继续救"的判据。"""
for i, s in enumerate(step_scores):
if s is not None and s < threshold:
return i
return None
def prune_decision(step_scores: list[float | None], threshold: float = 0.35) -> dict:
"""在搜索树扩展前调用：决定这条部分解是否还值得继续扩展。"""
agg = min_aggregate(step_scores)
bad = first_error_index(step_scores)
return {
"expand": agg >= threshold,
"aggregate": round(agg, 4),
"first_error_at": bad,
# 早错通常不可救，晚错还有回退空间 —— 这个区分直接决定要不要继续投入
"recoverable": bad is None or bad >= len(step_scores) * 0.6,
}
if __name__ == "__main__":
cases = {
"健康": [0.9, 0.88, 0.91, 0.9, 0.87],
"早错": [0.9, 0.92, 0.2, 0.1, 0.05],
"晚错": [0.9, 0.9, 0.9, 0.88, 0.15],
}
for name, s in cases.items():
d = prune_decision(s)
print(f"{name}: 可扩展={d['expand']} 聚合分={d['aggregate']} "
f"首错位于第{d['first_error_at']}步 可救={d['recoverable']}")
# 早错：聚合分 0.05，首错在第 3 步，不可救 —— 应立即剪枝并回溯
# 晚错：聚合分 0.15，首错在第 5 步，可救 —— 聚合分几乎相同，
# 但只有这一条值得再投入预算去救。不区分两者就是在浪费算力
5.4 Do not treat the process reward as the only truth
Even when you do use process rewards, it is wise to keep a fallback path: let the chain run to its endpoint and do programmatic or outcome verification there. Step annotation is inevitably incomplete and inevitably noisy, whereas endpoint verification is objective. A robust combination is this: the process reward handles in-flight pruning, the endpoint verifier handles the final selection. Topologically this is exactly the hybrid strategy of "depth-first search plus width-based reranking," which is what the next section is about.
VI. Adaptive difficulty routing: spending the budget where it cuts
The previous sections all discussed how to allocate a given budget. This section discusses a more upstream question: how much budget there should be in the first place. If every question receives the same budget, then on average you are certainly wasting it — because the slopes of the returns curves for different questions are wildly different.
6.1 The empirical relationship between difficulty and optimal sampling
This is the most practical empirical rule in test-time scaling: easy questions saturate with only a few samples, hard questions are the ones that need a large number. On a problem the model already knows, four samples and sixty-four samples give almost the same accuracy; on a problem where it gets lucky one time in eight, sixty-four samples may push accuracy up by more than a factor of two. So a non-uniform allocation such as "four samples for easy questions, sixty-four for hard ones" returns far more than "eight for everything."
The direct corollary of that rule is this: the optimal allocation of test-time compute is per-question, not globally uniform. It also explains why fixed-sample-count schemes always leave obvious room for improvement — they systematically overspend on easy questions and underspend on hard ones.
6.2 Three implementation paths
- Cascades and early stopping: the simplest and also the most robust. Run one round with a small budget; if the answers agree or the verification score is high enough, return immediately, and escalate the budget only when a threshold trips. Implementation complexity is low, it comes with circuit-breaker semantics for free, and it is the first choice in a production environment.
- Difficulty prediction: train a difficulty classifier, or predict difficulty from cheap features such as the coverage slope on a small sample, the entropy of the first token, or answer length, and pre-allocate sample counts accordingly. Theoretically optimal, but difficulty prediction itself is hard to get right — there is a systematic bias between what the model "feels" is hard and what is actually hard, and that bias drifts with the model version.
- Online scheduling: treat the budget as a queueing resource. When a question arrives, give it the cheapest probe first, and use the uncertainty observed mid-sampling (how sharp is the vote distribution, how large is the disagreement) to decide whether to keep investing. This is the form that best fits a real service, because it needs only the information in the current batch and no advance knowledge of difficulty.
6.3 A trap in online scheduling: confident mistakes
Early stopping carries a real risk: if the model is confidently wrong (every sample consistently produces the same wrong answer), early stopping freezes that mistake immediately. And consistent error is precisely one of the hardest cases for a reasoning model — when a particular error pattern is very common in the training data, the model will fail highly consistently in that mode.
There are two mitigations, and both are worth implementing:
- Give early stopping a minimum budget floor: no matter how decisive the vote, always run at least a few samples before allowing a return. This costs a little efficiency on easy questions in exchange for eliminating a class of errors that cannot heal itself.
- Use a double threshold for early stopping: vote agreement and verification score must both clear their bar. A single signal (whether votes or score) can be fooled by a consistent error; the joint threshold of two independent signals is far more reliable.
6.4 A complete implementation of budget scheduling
The scheduler below folds the principles above into code: probe first, escalate on uncertainty, a minimum floor, double-threshold early stopping, and a hard time limit that runs through the entire flow. Its structure can be lifted directly into a real inference service.
"""自适应预算调度器：探针 -> 不确定性判定 -> 升级。
1. 先用最小探针（便宜），只有"值得"的题才追加预算
2. 早停是双门槛（投票一致性 AND 校准分数），防自信错误
3. 任何情况下都设最小地板，避免单样本决定答案
4. deadline 是唯一的硬约束，贯穿每一轮循环
"""
from __future__ import annotations
import time
from collections import Counter
from dataclasses import dataclass, field
@dataclass
class ScheduleConfig:
probe_k: int = 2 # 第一档：便宜探针
medium_k: int = 6 # 第二档：升档后补采
final_k: int = 16 # 第三档：难题取满
min_floor: int = 2 # 最小地板：任何题至少采这么多
agreement_stop: float = 0.85 # 投票一致性早停线
verify_stop: float = 0.82 # 校准分数早停线
uncertain_gap: float = 0.25 # 前二名票差小于此值 = 不确定，需升档
deadline_sec: float = 25.0 # 硬预算
def should_escalate(counts: Counter, done: int, cfg: ScheduleConfig) -> tuple[bool, str]:
"""根据当前投票分布判断是否值得追加预算。"""
if done < cfg.min_floor:
return True, "尚未达到最小地板"
ranked = counts.most_common(2)
agree = ranked[0][1] / max(done, 1)
gap = (ranked[0][1] - ranked[1][1]) / max(done, 1) if len(ranked) > 1 else 1.0
if agree >= cfg.agreement_stop and gap >= cfg.uncertain_gap:
return False, f"票数已确定 一致率={agree:.2f} 领先={gap:.2f}"
if gap < cfg.uncertain_gap:
return True, f"候选接近、结果不确定 领先={gap:.2f}"
return True, f"一致性不足 一致率={agree:.2f}"
def adaptive_solve(question: str, cfg: ScheduleConfig | None = None) -> dict:
cfg = cfg or ScheduleConfig()
t0 = time.monotonic()
counts: Counter[str] = Counter()
stages: list[dict] = []
k_now = cfg.probe_k
while True:
left = cfg.deadline_sec - (time.monotonic() - t0)
if left <= 0.2:
stages.append({"stage": "deadline_cut"})
break
# 剩余时间不足以跑完当前档位 -> 降到跑得起的规模，绝不返回半截结果
k_now = max(1, min(k_now, int(left / 0.15)))
counts.update(sample_stage(question, k_now))
done = sum(counts.values())
stop, reason = should_escalate(counts, done, cfg)
top = counts.most_common(1)[0][0] if counts else None
prob = calibrated_score(top, question)
if stop and prob >= cfg.verify_stop:
stages.append({"stage": "early_stop", "reason": reason, "samples": done})
break
if done >= cfg.final_k or time.monotonic() - t0 > cfg.deadline_sec * 0.8:
break
k_now = cfg.medium_k if done < cfg.medium_k else cfg.final_k
if k_now <= done:
break
done = sum(counts.values())
ranked = counts.most_common()
return {
"answer": ranked[0][0] if ranked else None,
"agreement": round(ranked[0][1] / max(done, 1), 3) if ranked else 0.0,
"total_samples": done,
"saving": round(1 - done / cfg.final_k, 2) if cfg.final_k else 0.0,
"elapsed_sec": round(time.monotonic() - t0, 2),
"stages": stages,
}
def sample_stage(question: str, k: int) -> list[str]:
from generator import sample_batch
from scoring import normalize_answer
return [normalize_answer(a) for a in sample_batch(question, k, temperature=0.7)]
def calibrated_score(answer: str | None, question: str) -> float:
if answer is None:
return 0.0
from verifier import score
from scoring import calibrate
return calibrate(float(score(answer, question)))
if __name__ == "__main__":
print(adaptive_solve("求 17 乘 23 的结果"))
print(adaptive_solve("证明存在无穷多个素数"))
# 前者应在探针档就早停（saving 明显大于 0），后者应一路升档到 final_k
The most important property of this scheduler is that it flattens the right-skewed cost distribution: easy questions cost almost nothing, and the budget naturally flows to the hard ones. At the same average budget this works far better than "give every question a larger sample count," because every dollar the latter spends on an easy question has zero marginal return.
6.5 Budget fairness at the queue level
Per-request scheduling does not solve another problem: when a batch of requests shares one budget, how do you divide it. A single hard request can eat the entire batch budget while a pile of easy requests sits in the queue. There are three common strategies in engineering: pre-allocating by predicted difficulty (least effort, but it depends on difficulty prediction); first-come-first-served with a hard cap (the simplest and the most predictable, but the total budget goes unspent when many easy requests arrive); and dynamic adjustment based on remaining queue length (the highest budget utilization, but the worst latency predictability).
The practical experience is this: use first-come-first-served with a hard cap for latency-sensitive services, and use dynamic adjustment for offline batch jobs. For the former, what users perceive is latency stability; for the latter, the user is not even present, and throughput is the only goal.
7. Depth-First vs. Breadth-First: Choosing a Search Topology
Every strategy so far has been breadth-first — spreading horizontally across several independent paths. So is digging vertically deeper better? This is a real trade-off, and there is no free lunch.
7.1 Comparing the Two Topologies
| Dimension | Breadth-first (independent sampling plus voting or reranking) | Depth-first (beam search, tree search, Monte Carlo tree search) |
|---|---|---|
| Path relationship | Chains are fully independent | Nodes are partial solutions, edges are continuations, prefixes can be shared |
| Token efficiency | Low: several chains regenerate the same prefix | High: shared prefixes amortize the duplicated computation |
| Latency profile | Good: wall-clock time is set by the batch rounds | Poor: the tree is serialized by depth, so tail latency is high |
| Implementation complexity | Low: stateless, easy to scale horizontally | High: needs node caching, backtracking, pruning, concurrent state |
| Dependence on scoring | Weak: only end-stage reranking is required | Strong: every pruning step depends on scoring quality |
| Good at | Exploration: escaping local optima, covering many solution shapes | Exploitation: going repeatedly deeper along a single line of thought |
| Main failure mode | Failing to cover the direction of the correct solution | Premature pruning early on, with the error invisible along a deep chain |
OpenAI's paper on optimal Test-Time Compute Scaling states this trade-off very clearly: for easy problems, independent sampling is enough; for hard problems, only search produces a net gain — but that gain depends on how strong the scorer is, and when the scorer is weak, deep search is worse than broad search. The paper also makes a counter-intuitive observation: when compute is ample, beam search without a sufficiently strong scorer behind it actually performs worse than plain independent sampling plus reranking.
This explains a commonly observed pattern: the same search algorithm looks unremarkable on a weaker model and suddenly gets much better when swapped for a stronger reasoning model. The reason is that once per-step quality improves, the probability of taking a wrong step during deep search drops, and the cost of a bad prune falls along with it. A search algorithm and its base model are a coupled system, and neither can be evaluated apart from the other.
7.2 A Hybrid Strategy: Scatter First, Then Converge
Production systems rarely pick one side; almost all of them are hybrids: breadth-first exploration, breadth-first convergence, depth-first refinement. Start with a small number of independent samples to map the problem space — which solution shapes work, whether the answer is unique, whether the prefix is stable — and then run deep verification or local search on the most promising ones. This "scatter then converge" shape is far more robust than pure deep search, because the first step of a deep search is usually the most expensive one, and once the direction is chosen wrongly, every token after it is spent for nothing.
7.3 A Minimal Depth-First Implementation
Beam search is the most practical form of depth-first search. The key lies in prefix sharing — nearly the entire token-efficiency advantage of tree search comes from here. A good implementation organizes partial solutions into a prefix tree so that the key-value cache is reused across shared prefixes, rather than re-running every node independently.
"""束搜索（束宽度 b），演示深度优先骨架 + 前缀共享。
与验证器重排的本质区别：
重排：k 条独立链 -> 末端打分
束搜：b 个活跃节点，每步只扩展少数候选 -> 中间剪枝 -> 末端打分
"""
from __future__ import annotations
from dataclasses import dataclass
@dataclass
class Node:
text: str # 已生成的部分推理
step: int
score: float # 累计启发式分
parent: "Node | None" = None
dead: bool = False
def expand(node: Node, continuations: list[str], scorer) -> list[Node]:
"""把一个部分解扩展成若干子节点，并立即打分剪枝。"""
kids = []
for cont in continuations:
child = Node(text=node.text + cont, step=node.step + 1, parent=node)
child.score = scorer(child.text) # 局部打分，剪枝靠它
child.dead = child.score <= 0.0 # 死枝：剪掉，不再展开
kids.append(child)
return kids
def beam_search(problem, width: int, max_steps: int, continuations_fn, scorer) -> Node | None:
"""束搜索主循环。剪枝发生在每一步，而不是只在末端。"""
beam: list[Node] = [Node(text="", step=0, score=1.0)]
expanded = 0
for _ in range(max_steps):
candidates: list[Node] = []
for node in beam:
if node.dead:
continue
kids = expand(node, continuations_fn(problem, node), scorer)
expanded += len(kids)
candidates.extend(kids)
if not candidates:
break
# 叶子判定：某个候选已给出完整答案
finished = [c for c in candidates if problem.is_complete(c.text)]
if finished:
return max(finished, key=lambda c: c.score)
candidates.sort(key=lambda c: -c.score)
beam = candidates[:width]
return max(beam, key=lambda c: c.score) if beam else None
def compare_cost(k: int, width: int, steps: int, branch: int, prefix_share: float) -> dict:
"""对比两种策略在同一扩展预算下的词元消耗。
prefix_share 是"共享前缀能省下的比例"，它完全取决于实现质量，
而不是理论值 —— 这正是束搜索在实践中常被低估的原因。
"""
bon_tokens = k * steps # 独立采样：每条链从头生成
tree_tokens = width * branch * steps * (1 - prefix_share) # 束搜：共享前缀摊薄
return {
"bon_tokens": bon_tokens,
"tree_tokens": int(tree_tokens),
"saving": round(1 - tree_tokens / bon_tokens, 3),
}
if __name__ == "__main__":
# 关键：把 prefix_share 从理想值调到贴近现实值，结论会翻转
for share in [0.0, 0.15, 0.4]:
r = compare_cost(k=8, width=4, steps=20, branch=4, prefix_share=share)
print(f"前缀共享率={share:.0%}: 独立采样={r['bon_tokens']} 词元, "
f"束搜索={r['tree_tokens']} 词元, 节省={r['saving']:.0%}")
# 前缀共享率为 0（没有做请求内前缀树）时，束搜索没有任何词元优势，
# 只剩下"更早暴露错误"这一个好处 —— 而它依赖强打分器
7.4 A Counter-Intuitive Conclusion: Prefix Sharing Pays Off Far Less Than Theory Suggests
In theory beam search saves tokens through prefix sharing, but in practice this gain is often far smaller than expected. Three reasons explain why:
- Reasoning models branch early and branch deep. The model usually settles on which path to take within the first dozen or so steps, and the tokens that follow on each path make up most of the chain, so the prefix that can actually be shared is very short.
- The implementation bar is high. In many inference services the prefix cache only takes effect across requests, and sharing within the tree of a single request requires dedicated prefix-tree data structures. Get it wrong and the gain is zero, or even negative once the extra scheduling overhead is counted.
- Batching breaks the sharing. Scattering nodes across different batches to chase throughput makes it hard for the prefix to hit the cache.
So the pragmatic judgment is this: if you have not built a real within-request prefix tree, beam search's token advantage essentially does not exist, and all that is left is the single benefit of "pruning in the middle exposes errors earlier" — which itself depends on a strong scorer. Before committing to a depth-first decision, honestly assess your own prefix-reuse capability.
8. Reliable Rules for Compute-Optimal Test-Time Scaling
Everything so far has been about how to do it. This section is about what the rules actually are. Several conclusions are already solid enough to serve as direct input into design decisions.
8.1 Rule One: Compute Should Be Allocated Non-Uniformly by Difficulty
This is the most core empirical rule. For a fixed problem set, the optimal number of samples for an easy problem is small, while the optimal number for a hard problem is large; if you are forced to use one single global sample count, your accuracy is necessarily lower than that of a scheduler which knows who deserves more. This also means that the sentence "our system uses eight samples for self-consistency" hides an enormous optimization space by itself.
8.2 Rule Two: Verifier Quality Is the Ceiling
This rule can be stated like this: the ceiling of test-time scaling is set by the verifier, and the gain a verifier brings is steeper than that of a sampler. The intuition is that sampling can at best reach answers inside the model's capability range, whereas the verifier decides whether you can pick the right one out of them; once coverage is high enough, the only bottleneck left is discrimination. At that point scaling up sampling further is useless, and putting the entire budget into the verifier is the correct move.
This is also why programmatic verification occupies a special position in test-time scaling research: it turns the verifier from a probabilistic model into a deterministic check, which eliminates the bottleneck outright. Competition mathematics became the main battleground precisely because its answers can be confirmed by symbolic computation.
8.3 Rule Three: Diminishing Returns Are Structural, Not Random Fluctuation
Accuracy as a function of sampling rises roughly log-linearly, and every doubling of the sample count buys a steadily smaller increment. But more important than "it will saturate" is the shape of the saturation: majority voting usually saturates earlier than people expect, because when the correct answer is not the mode, sampling a hundred more chains will not save it; verifier reranking saturates later, because it does not depend on the mode but only on whether the scorer can recognize the right answer. The difference between these two saturation points is the main basis for choosing a method in practice.
8.4 Rule Four: Training-Time and Test-Time Scaling Multiply
This point is often overlooked but is extremely important: a reasoning model trained with reinforcement learning gains significantly more from test-time scaling than its base model does. The reason is that the space available to test-time scaling is determined by the coverage capability of the base policy — if the model cannot sample the correct answer at all, no amount of sampling helps; after training, both the quality and the diversity of a single sample improve substantially for a reasoning model, and so the searchable space grows along with them.
Put another way, training-time and test-time scaling are not competitors but multiplicative: training determines how large the search space is, and test-time scaling determines how well you search it. This also means that pushing only on test-time scaling while neglecting training quality hits the ceiling fast. Conversely, no matter how much budget you add, a mediocre base model will struggle to catch up with a single straightforward call to a strong reasoning model.
8.5 A More Complete Gain-Curve Model
Putting the rules above together, a naive gain model looks like this: the improvement in accuracy equals the growth in coverage multiplied by the probability that the verifier picks the correct answer out of the candidates. The value of this decomposition is that it tells you directly where to put the budget.
"""测试时扩展的收益分解模型。
准确率(宽度 k) ≈ P(候选中至少有一次正确 | k) x P(判别环节挑出正确的 | 存在正确候选)
两个因子都随 k 单调递增，但增长速度不同，且第二个因子有硬上限。
"""
from __future__ import annotations
from dataclasses import dataclass
@dataclass
class StrategyParams:
name: str
p_first_correct: float # 单次采样就正确的概率（基础质量）
spread: float # 搜索性：值越大，多样性带来的覆盖增长越快
verifier_top1: float # 验证器在候选含正确答案时的挑出率（硬天花板）
maj_quality: float = 1.0 # 多数投票的上限系数：正确答案成为众数的概率
def coverage(k: int, p: float, spread: float) -> float:
"""候选集合含正确答案的概率。采样变多时单条变弱，故按多样性折算。"""
p_k = min(0.95, p * (1.0 + spread * (k ** 0.5 - 1.0) * 0.1))
return 1.0 - (1.0 - p_k) ** k
def majority_accuracy(k: int, prm: StrategyParams) -> float:
"""多数投票：正确答案必须是众数，且需要攒够基本票数。"""
cov = coverage(k, prm.p_first_correct, prm.spread)
need = max(2, int(k * 0.4)) # 成为众数所需的基本票数
return cov * prm.maj_quality * (1 - need / k * 0.3)
def bon_accuracy(k: int, prm: StrategyParams) -> float:
"""验证器重排：上限完全由验证器决定，与众数无关，因此饱和更晚。"""
return coverage(k, prm.p_first_correct, prm.spread) * prm.verifier_top1
def decide(k: int, prm: StrategyParams) -> dict:
maj, bon = majority_accuracy(k, prm), bon_accuracy(k, prm)
return {
"k": k,
"coverage": round(coverage(k, prm.p_first_correct, prm.spread), 3),
"maj": round(maj, 3),
"bon": round(bon, 3),
"winner": "验证器重排" if bon > maj else "多数投票",
}
if __name__ == "__main__":
base = StrategyParams("推理模型", 0.35, 1.2, 0.82, 0.78)
print(f"{'宽度':>5}{'覆盖':>8}{'多数投票':>10}{'验证器重排':>12} 赢家")
for k in [1, 2, 4, 8, 16, 32, 64, 128]:
r = decide(k, base)
print(f"{k:>5}{r['coverage']:>8.3f}{r['maj']:>10.3f}{r['bon']:>12.3f} {r['winner']}")
# 三个结构性观察：
# 1) 覆盖率永远低于 1，堆采样买到的边际覆盖极其有限
# 2) 多数投票的损失来自众数约束，在 k=2、k=4 时伤害最重
# 3) 两种方法的差距在 k=8~32 这段（默认预算区间）最大
weak = StrategyParams("弱验证器", 0.35, 1.2, 0.55, 0.78)
print(f"\n换成弱验证器后，宽度 16 的结论：{decide(16, weak)}")
# 验证器一变弱，多数投票反而胜出 —— 验证器质量直接改写方法选择结论
8.6 Three Typical Saturation Shapes
Classifying the curves above by shape yields a practical diagnostic tool:
- Fast saturation: coverage is low but majority voting rises quickly, then flattens out almost immediately. Typical of easy tasks or a strong base model. The signature is saturation within ten samples, which means any further sampling is pure waste.
- Slow climb: coverage keeps growing but decision quality never improves. Typical of "we sampled the right answer but cannot select it". The signature is that both majority voting and verifier reranking stay far below the ceiling that coverage allows, which means the budget belongs in the verifier.
- Ceiling-bound: both curves hug the verifier's ceiling. Typical of insufficient verifier quality. The signature is that the reranked result is almost exactly coverage multiplied by a fixed ceiling factor.
Classify every experiment into one of these three, and you will save a great deal of time that would otherwise be spent blindly adding budget.
9. Engineering Implementation: Budget Control, Timeouts, and Circuit Breakers
What separates theory from deployment is nothing but engineering. A test-time scaling approach that only works in offline evaluation will usually die in production of timeouts, runaway cost, and cascading failures. Everything in this section is what you need in order to survive.
9.1 Three Levels of Budget Control
- Global budget: the total compute allocated to a batch of requests. This requires the scheduler to divide it fairly across requests, otherwise one or two hard requests will eat the whole batch's budget.
- Per-request budget: the cap on tokens and milliseconds allowed for a single problem. This is the only hard constraint, and it must be guaranteed by the caller or by the service.
- Stage budget: a sub-budget reserved inside the per-request budget for each stage (probing, scaling up, final verification), so that no single stage can consume all the time without restraint.
The easiest mistake to make is setting only a global budget and no stage budget. The typical consequence is this: after scaling up to a larger sample count a single chain gets longer, the decoding time of one chain eats all the remaining budget, and you end up returning a truncated, incomplete reasoning chain — and such a result is not merely useless, it is worse than not scaling at all, because it turns a problem that would have been answered normally into a truncation failure. Failures caused by truncation almost never surface in offline evaluation, because that runs on a low-latency development machine.
9.2 Timeout Degradation: A Three-Level Fallback
Timeout handling is not as simple as "raise an error on timeout". There should be an explicit degradation chain, and every level must return something usable:
- Full strategy: the complete probe, scale-up, and verification flow.
- Return on convergence: stop scaling up and return the majority vote over the samples already collected. This is the level used most of the time; the vast majority of requests are resolved right here.
- Return on truncation: a single chain times out during decoding, so force it to stop and emit whatever has been generated. If the answer already appears in that partial output (many reasoning models put the answer after an end marker), it is still usable.
The key design principle: the degradation path must actually have been triggered in testing. The flaw in many teams' systems is not "the degradation logic is written wrong" but "the degradation logic has never been triggered, because the latency in the test environment is too low". Run a degradation test once on purpose in a latency-injected environment, with the timeout threshold temporarily set to an extremely small value.
9.3 Circuit Breaking: Five Failure Classes That Must Be Cut Off Automatically
Test-time scaling introduces long-running calls, large batches, and repeated invocation, which makes it naturally a circuit-breaker-friendly scenario:
- Cost breaker: stop escalating as soon as per-request tokens, cumulative tokens per user, or the batch total budget exceeds its limit.
- Time breaker: a wall-clock limit. It must be checked inside every layer of the loop, not only at the entry point — a depth-first search loop that does not check at every step turns a timeout into an indefinite hang.
- Quality breaker: cut off when the reasoning chain falls into a degeneration loop. This is a failure mode introduced purely by test-time scaling: a single generation will not repeat indefinitely, but a "generate, score, regenerate" loop can get stuck in one.
- Dependency breaker: when the verifier service times out or returns an error, degrade to pure majority voting. Note that this must be "degrade and continue" rather than "fail outright", because the voting path does not depend on the verifier.
- Concurrency breaker: scaling multiplies the concurrent paths of a single request by several times, which can easily saturate the inference cluster during traffic peaks. This needs concurrency shaping at the gateway layer.
"""带完整熔断与降级的推理执行框架。
降级链条：完整策略 -> 收敛即返（多数投票）-> 截断即返 -> 失败
每一级都必须返回可用结果，只有最后一层才允许返回空。
"""
from __future__ import annotations
import time
from collections import Counter
from dataclasses import dataclass, field
from enum import Enum
class Level(Enum):
FULL = "完整策略"
MAJORITY = "收敛即返"
TRUNCATED = "截断即返"
FAILED = "失败"
class Breaker(Enum):
OK = "正常"
COST = "成本超限"
TIME = "时间超限"
DEGENERATE = "推理退化"
DEPENDENCY = "依赖故障"
@dataclass
class Guard:
max_tokens: int = 60_000 # 单请求词元上限
deadline_sec: float = 30.0 # 墙钟限制
deg_window: int = 40 # 退化检测窗口（词元数）
deg_ratio: float = 0.75 # 窗口内重复四元组占比阈值
events: list[Breaker] = field(default_factory=list)
def trip(self, b: Breaker) -> None:
if b is not Breaker.OK:
self.events.append(b)
def has_degeneracy(text: str, window: int = 40, ratio: float = 0.75) -> bool:
"""检测推理退化：最近窗口内出现大量重复四元组。
这是测试时扩展特有的故障：单次解码不会这样，
但"生成->打分->再生成"的循环会陷进去。
"""
words = text.split()
if len(words) < window:
return False
tail = words[-window:]
grams = [tuple(tail[i:i + 4]) for i in range(len(tail) - 3)]
return len(set(grams)) / len(grams) < (1 - ratio)
@dataclass
class Result:
answer: str | None
level: Level
breakers: list[Breaker]
tokens_used: int
elapsed: float
degraded_reason: str | None = None
def solve_with_guards(question: str, guard: Guard) -> Result:
t0 = time.monotonic()
g = Guard(max_tokens=guard.max_tokens, deadline_sec=guard.deadline_sec,
deg_window=guard.deg_window, deg_ratio=guard.deg_ratio)
tokens = 0
candidates: list[str] = []
# ---------- 完整策略主循环 ----------
try:
for k in (2, 6, 16):
if time.monotonic() - t0 > g.deadline_sec:
g.trip(Breaker.TIME)
break
if tokens > g.max_tokens * 0.8:
g.trip(Breaker.COST)
break
batch = sample_with_deadline(question, k, g, t0)
if batch is None:
g.trip(Breaker.TIME)
break
cands, used = batch
tokens += used
candidates.extend(cands)
if has_degeneracy(" ".join(candidates[-3:]), g.deg_window, g.deg_ratio):
g.trip(Breaker.DEGENERATE)
break
# 验证器故障 -> 降级为纯投票，而不是整体失败
try:
score = verify(candidates)
except (TimeoutError, ConnectionError):
g.trip(Breaker.DEPENDENCY)
score = None
answer, confident = decide(candidates, score)
if confident:
return Result(answer, Level.FULL, g.events, tokens,
time.monotonic() - t0)
except Exception:
g.trip(Breaker.DEPENDENCY)
# ---------- 降级一：纯多数投票 ----------
if candidates:
return Result(majority(candidates), Level.MAJORITY, g.events, tokens,
time.monotonic() - t0, "降级为多数投票")
# ---------- 降级二：单链截断 ----------
partial = sample_truncated(question, max_tokens=1500, deadline=g.deadline_sec)
if partial:
return Result(extract_answer(partial), Level.TRUNCATED, g.events, tokens,
time.monotonic() - t0, "使用截断的部分推理")
return Result(None, Level.FAILED, g.events, tokens, time.monotonic() - t0)
# ---- 外部依赖（示意） ----
def sample_with_deadline(q, k, g, t0):
from generator import sample_batch
return sample_batch(q, k, deadline=g.deadline_sec - (time.monotonic() - t0))
def verify(cands):
from verifier import verify_batch
return verify_batch(cands)
def decide(cands, score):
from scoring import is_confident_enough
return majority(cands), (score is not None and is_confident_enough(cands, score))
def majority(cands):
return Counter(cands).most_common(1)[0][0] if cands else None
def sample_truncated(q, max_tokens, deadline):
from generator import sample_one_forced
return sample_one_forced(q, max_tokens, deadline)
def extract_answer(text):
from scoring import extract_answer as ex
return ex(text)
if __name__ == "__main__":
# 测试技巧：把 deadline 调到 0.01 秒，必然触发降级，逐级验证返回可用性
r = solve_with_guards("证明存在无穷多个素数", Guard(deadline_sec=30.0))
print(f"等级={r.level.value} 熔断={[b.value for b in r.breakers]} "
f"词元={r.tokens_used} 耗时={r.elapsed:.1f}秒 原因={r.degraded_reason}")
# 降级路径必须在注入延迟的测试里真实跑过，不能只测正常路径
9.4 Prefix Caching: The Most Underrated Optimization
Parallel sampling has one enormous source of waste: several chains share the same problem prefix, yet every request re-runs the prefill. For retrieval scenarios with a long system prompt plus a long document, this prefill cost multiplied by the sample count can exceed decoding itself. All mainstream inference frameworks now offer automatic prefix caching, letting an identical prefix reuse the key-value cache, which in parallel sampling is close to a free performance win.
Turning it on costs nothing, and the gain is substantial in long-prompt scenarios. This is the highest-priority engineering optimization this article recommends — it ranks ahead of every algorithmic change.
9.5 The Trade-off Between Batching and Concurrency
Parallel sampling is naturally suited to continuous batching: the decoding stage of every chain is memory-bandwidth bound, and batching them together significantly improves per-GPU throughput. But two forces push back. First, one long chain in a batch holds up the completion time of the whole batch (the tail latency problem). Second, the larger the batch, the higher the memory pressure and the lower the achievable number of concurrent paths. The practical tuning direction is keep batches small within a request and make them large across requests: put the samples for one question in a single batch, then merge across different questions.
9.6 Cost Observability
There is one easily missed point: cost is not an average problem but a tail problem. The typical distribution is that the vast majority of requests cost very little while a few hard problems eat most of the budget. Operationally you should monitor the 95th and 99th percentiles of per-request cost rather than the mean, and set a separate quota for high-cost requests. A test-time scaling system with no per-request cost cap will inevitably have an incident when traffic fluctuates.
9.7 Recommended Metrics
Test-time scaling makes the same question produce results of different quality, so the monitoring dimensions must include budget. At minimum you need: accuracy broken down by budget tier, average tokens consumed, average latency, degradation trigger rate, degeneracy detection rate, the proportion of requests for which the verifier is unavailable, and the tail distribution of per-request cost. If you look only at overall accuracy, problems will hide between "the money saved on easy questions" and "the money added on hard ones".
10. Evaluation: Measuring Test-Time Scaling Fairly
This section looks the most boring, but it decides whether you can draw a trustworthy conclusion. Evaluating test-time scaling has a few characteristic traps, and hitting any one of them will reverse your conclusion.
10.1 Metrics You Must Report
| Metric | What it measures | Why it cannot be omitted |
|---|---|---|
| Coverage | The probability that at least one sample contains the correct answer | It is the upper bound on every other metric; without it you cannot tell whether the bottleneck is sampling or discrimination |
| Majority voting accuracy | The true quality of the voting decision | The quality of the strategy you will actually deploy |
| Verifier reranking accuracy | The quality once the discrimination stage intervenes | Only a comparison against voting tells you whether the verifier is worth funding |
| Average tokens and latency | Cost | Accuracy without cost is not comparable |
| Saturation curve | When the marginal gain goes to zero | It decides where the budget should stop |
| Degradation trigger rate | The system's real effective strategy | Requests cut off by a breaker must have their quality counted too |
The one you should look at first is the gap between coverage and decision quality, because it tells you directly what to do next: a large gap means the decision side has room, so fix the verifier; a small gap means the decision side is close to saturation, so fix sampling or training.
10.2 A Checklist of Evaluation Traps
- Mixing up pass@1 and single-sample majority voting. These two are nearly identical by definition, but the numbers often fail to line up across implementations, usually because the answer-extraction logic is inconsistent.
- Incomparable cost. Comparing the accuracy of one call against a thirty-two-way vote without reporting the token difference amounts to claiming "spending thirty times the compute improves results by ten percent" — which is not a meaningful conclusion.
- Answer equivalence judged too strictly. Mathematically equivalent forms, equivalent code implementations, and equivalent queries are all scored wrong, which systematically understates the effect of voting and reranking. Errors here often matter more than the differences between methods, and they always point the same way — toward understating scaling methods.
- Inconsistent temperature. The baseline usually uses temperature zero while multi-sampling uses a higher temperature, and that difference alone accounts for a substantial share of the "improvement"; it needs to be reported separately.
- Testing only easy questions. Test-time scaling pays off most on hard problems, so evaluating only easy questions leads to the wrong conclusion that "scaling doesn't work".
- Training the verifier on in-distribution data and then evaluating it in-distribution yields a severely overestimated verifier quality, which then leads to over-investment in the verifier.
- Ignoring the effect of context and prefix caching. Latency differs a great deal before and after caching is enabled, so two sets of results cannot be pooled together.
10.3 The Budget-Accuracy Frontier
The accuracy at a single budget point carries almost no information. What carries information is the frontier curve: total budget on the x-axis, accuracy on the y-axis. Only the upper envelope of the two curves (one per method) is the true performance boundary. Note that the comparison must use the same total budget, not the same sample count — because scoring has overhead, the actual cost of the same sample count can differ enormously.
"""构建预算-准确率前沿曲线，比较不同策略的性价比。"""
from __future__ import annotations
def evaluate_policy(dataset, policy, budgets) -> list[dict]:
"""在多个预算档位上评测同一策略，返回 (预算, 准确率, 成本) 曲线。"""
curve = []
for b in budgets:
correct = cost = 0
for item in dataset:
out = policy(item["question"], budget=b)
correct += int(out["answer"] == item["gold"])
cost += out["tokens"]
curve.append({
"budget": b,
"accuracy": round(correct / len(dataset), 4),
"avg_tokens": round(cost / len(dataset), 1),
})
return curve
def frontier(curves: dict[str, list[dict]]) -> list[dict]:
"""取上包络：在每个预算档位上找出最准的策略。"""
budgets = sorted({p["budget"] for c in curves.values() for p in c})
envelope = []
for b in budgets:
best = None
for name, c in curves.items():
for p in c:
if p["budget"] == b and (best is None or p["accuracy"] > best[1]):
best = (name, p["accuracy"], p["avg_tokens"])
if best:
envelope.append({"budget": b, "best_strategy": best[0],
"accuracy": best[1], "avg_tokens": best[2]})
return envelope
def cost_normalized(curve: list[dict], baseline: float) -> list[dict]:
"""把准确率增益折算成每百万词元换来多少准确率，用于跨策略比性价比。"""
out = []
for p in curve:
gain = p["accuracy"] - baseline
out.append({**p, "gain": round(gain, 4),
"gain_per_Mtok": round(gain / (p["avg_tokens"] / 1e6), 3)})
return sorted(out, key=lambda x: -x["gain_per_Mtok"])
if __name__ == "__main__":
# 示意数据：投票准确率随预算快速饱和，重排饱和更晚但起步更贵
curves = {
"多数投票": [
{"budget": 1, "accuracy": 0.351, "avg_tokens": 1200},
{"budget": 8, "accuracy": 0.612, "avg_tokens": 9600},
{"budget": 32, "accuracy": 0.683, "avg_tokens": 38400},
{"budget": 128, "accuracy": 0.694, "avg_tokens": 153600},
],
"验证器重排": [
{"budget": 1, "accuracy": 0.351, "avg_tokens": 1200},
{"budget": 8, "accuracy": 0.701, "avg_tokens": 13400},
{"budget": 32, "accuracy": 0.768, "avg_tokens": 43000},
{"budget": 128, "accuracy": 0.771, "avg_tokens": 155000},
],
}
print("预算前沿（相同总预算下取上包络）：")
for p in frontier(curves):
print(f" 预算={p['budget']:>4} 最佳={p['best_strategy']:<10} "
f"准确率={p['accuracy']:.3f} 平均词元={p['avg_tokens']}")
print("\n性价比排序（每百万词元的增益）：")
for p in cost_normalized(curves["多数投票"], 0.351)[:3]:
print(f" 预算={p['budget']:<4} 增益={p['gain']:.3f} 性价比={p['gain_per_Mtok']}")
# 结论形态：低预算段两种策略性价比接近；中预算段重排反超；
# 高预算段都饱和，继续加钱是纯浪费 —— 预算应该停在中段
11. Test-Time Compute vs. Training-Time Compute
A natural follow-up: since the same compute can be spent on training, why not just train a stronger model?
11.1 The Essential Difference
| Dimension | Training-time scaling | Test-time scaling |
|---|---|---|
| Essence | Pressing capability into the weights | Borrowing capability on the fly |
| Cost structure | A one-off large cost, amortized over the long term | Paid on every call, with linear marginal cost |
| Amortizability | The more calls, the better the deal | The more calls, the worse the deal; there is no amortization |
| Handles unlabeled tasks | Yes — that is exactly what training is | No; it needs a decidable verification signal |
| Failure modes | Overfitting, failure to generalize, unstable training | Budget waste, latency spikes, verifier gaming |
| Iteration cycle | Weeks | Hours; can ship the same day |
| Ceiling | The quality of training data and signals | Coverage multiplied by the verifier |
11.2 The Economic Decision Rule
The decision is actually quite mechanical. Let task call volume be Q, the cost of one training run be C, the extra cost per inference be c, and the accuracy gains from the two be Δtraining and Δinference respectively. Then the marginal-equivalence test is simple: training is only worth it if its amortized unit cost is lower than the test-time unit cost.
- General capabilities with very high call volume (customer support Q&A, general assistants): the amortization effect is significant and the marginal cost of test-time scaling never catches up. Such scenarios should favor investment in training.
- Long-tail hard problems, domain customization, small task volume: amortization does not hold, the fixed cost of retraining cannot be spread, and test-time scaling offers an extremely good cost-performance ratio. This is why a small vertical team using a reasoning model plus test-time compute can beat a self-trained base model on many tasks.
- High-value tasks with a deterministic verifier (mathematical proofs, code repair, data cleaning): test-time scaling delivers its most certain gains here, because the verification signal is perfect and free. This is the scenario with the highest return on investment.
There is one more frequently overlooked dimension: iteration speed. The feedback cycle of training-time scaling is measured in weeks; that of test-time scaling is measured in hours. In the face of rapidly changing requirements, a gain you can bank today by adjusting the budget and the scheduler is often worth more than a large model training run three months from now. This is engineering reality, not a claim about technical superiority.
11.3 They Are Best Used Together
The most effective path is to let the two divide the labor: training is responsible for making coverage high and single-sample quality high; test-time scaling is responsible for squeezing that capability to its limit. The "multiplicative relationship" mentioned earlier refers to exactly this. The consensus in 2026 is already clear: a strong reasoning model plus a good test-time scaling strategy beats any one-sided optimization by a wide margin.
12. Limitations and Failure Modes
Every method has boundaries. This section lists the known failure modes of test-time scaling; writing them down clearly is more valuable than claiming universality, because essentially every production incident is caused by one of them.
12.1 Verifier Gaming
If the verifier is a learned model, it has exploitable patterns. Using test-time scaling to maximize the verifier score is in effect optimizing a proxy objective, and the system will spontaneously learn to find solutions that "fool the verifier without being correct". This is especially visible on math competition data: models find phrasings that bypass verification. So verifier design must be adversarially tested on an ongoing basis, and non-imitable programmatic verification should be preferred wherever possible.
12.2 Accuracy Flat, Cost Doubled
The most common "scaling doesn't help" scenario: votes are highly consistent but consistently wrong, or the verifier fails on hard problems so that all candidates score close together. The former shows up as extra sampling yielding exactly zero gain; the latter shows up as reranking degrading into a random choice. The correct response is to move the budget from the sampling side to the discrimination side rather than to keep adding samples. The decision rule is the gap described earlier: if coverage is low, fix sampling or training; if coverage is high but decisions are bad, fix the verifier.
12.3 Unacceptable Latency
Depth-first search, heavy sampling, and verifier calls all stretch latency. In interactive scenarios even tens of milliseconds of latency is already unusable, and at that point the scaling strategy has to be redesigned: compress the depth, cap the number of samples, or switch to returning a first answer asynchronously and then updating it through iteration. The last approach is very common in products — return a possibly imperfect answer quickly, then overwrite it with the scaled result.
12.4 Length Inflation and Runaway Cost
There is a hidden mechanism here: if process scores come from a length-related heuristic, or if budget is allocated by total tokens, the system learns to write longer. This works early on (longer reasoning really is more accurate), but it eventually becomes pure cost waste. Monitoring should therefore look at "average tokens per correct answer" separately rather than only at accuracy.
12.5 Misjudging Difficulty Routing
Adaptive budget depends on difficulty estimation, and estimation errors have two costs: a hard problem judged easy (money not spent when it should have been, losing accuracy directly), and an easy problem judged hard (money wasted). The first is far more serious. That is why a minimum budget floor is necessary, and why "probe first with a cheap model" is a worthwhile investment.
12.6 The Ceiling When a Task Cannot Be Verified
The final boundary: when a task has neither verifiable answers nor a high-quality scoring model, test-time scaling is basically useless. All that is left is the "many heads are better than one" effect of parallel sampling, and even that often yields limited improvement. This is not an implementation problem but an information-theoretic limit — if no signal can tell you which answer is better, more candidates only make the choice more random.
12.7 Overfitting to the Evaluation Set
There is an easily overlooked risk: tuning the budget strategy repeatedly on the validation set is itself overfitting to that set. This is especially true when you use the coverage gap to decide between more sampling and a better verifier, because it biases you toward the verifier. The mitigation is to do a final confirmation on a held-out set that never took part in tuning, and to stay conservative about changes to the budget strategy.
13. An Implementation Checklist You Can Actually Follow
Compressing everything above into an order you can follow. The order matters — each step is a precondition for the next, and skipping one makes the effort after it wasted.
- Establish the baseline first: measure the accuracy, tokens, and latency of single-sample generation as the reference for every comparison. Every comparison without a baseline is meaningless.
- Determine whether the task is verifiable: can answers be judged by unit tests, symbolic computation, or deterministic rules? If yes, go straight to programmatic verification; if not, build the scorer first. The conclusion of this step directly determines the direction of everything that follows.
- Implement answer normalization and majority voting: the lowest-cost, most certain step. Many teams skip it and jump straight to complex methods, then find that accuracy does not move, only to discover later that normalization was the problem.
- Enable prefix caching: zero-cost gain, especially in long-prompt scenarios. It ranks ahead of every algorithmic change.
- Implement cost and time circuit breakers: have the breakers in place before scaling is switched on, otherwise the first incident will be expensive.
- Then consider verifier reranking: first measure the gap between coverage and voting accuracy; only a large gap justifies funding the verifier.
- Adaptive scheduling comes last: optimize budget allocation only after a fixed budget runs end to end and the metrics are trustworthy. Otherwise you are tuning parameters against noise.
- Monitor the full path: look at accuracy, tokens, latency, and degradation rate by budget tier, with emphasis on the tail distribution of cost.
- Regression-test the degradation paths: inject latency and verifier failures on purpose to make sure the degradation chain really returns usable results.
- Confirm on a held-out set: after all tuning is done, do a final confirmation on a set that never took part in tuning, to prevent evaluation overfitting.
Conclusion: Test-Time Scaling Is a Shift in Thinking
The core value of test-time scaling is that it turns model capability from "how much is baked into the weights" into "how well the scheduling is done". Once budget control, difficulty routing, verifier quality, and search topology can all be optimized continuously, reasoning stops being a reasoning problem and becomes a problem of search and scheduling. The gains this shift brings are structural: they can be cashed out without retraining, and they keep stacking as the scheduling strategy improves.
But it has clear boundaries too: it needs a decidable verification signal, its cost grows linearly with call volume, and it hits the ceiling quickly on tasks without a verifier. Understanding those boundaries matters more than remembering any single technique.
Real engineering judgment shows up in knowing when to add budget, when to stop, and when to go back and improve training — these three points are the entire practical wisdom of test-time scaling. A system that only knows how to push the sampling threshold will eventually slam into the cost curve; only a system that knows how to stop on its own when accuracy saturates can keep running healthily over the long run.
