Routing Is Not Classification — It Is Scheduling Under Constraints
Start with a word that gets misused constantly. When a team first adopts multiple models, the first thing they build is a task classifier: split the incoming request into “easy” and “hard,” route hard requests to the big model and easy ones to the small model. This holds up in a demo and almost inevitably fails in production, for three reasons — and each of them points at the same conclusion: routing is not classification; it is scheduling under a set of hard constraints.
The first reason is a wrong objective. A classifier only answers “which category does this request belong to,” but what production actually needs answered is “among the models I can use right now, which one can get this request to a passing result at an acceptable cost and an acceptable latency?” The former is a single-point selection inside a label space; the latter is a combinatorial optimization problem with budget, SLA, and vendor-availability constraints. The two agree at small scale and diverge as soon as scale grows.
The second reason is ignoring non-monotonic capability. Most people’s mental model is “stronger is better,” which turns routing into a linear rule sorted by capability. But real tasks are multi-dimensional: a mid-tier model can be steadier than a flagship on long structured extraction, a small model can outperform a flagship on multilingual work, and a degraded channel model can hold its own on simple classification. The capability matrix is multi-dimensional and non-monotonic, which is exactly why routing deserves a system of its own.
The third reason is the most fatal of all: the correctness of a routing decision cannot be observed by the routing system itself. You never know whether the request you sent to the small model would have been better on the big one. That is what makes every claim of the form “routing saved us 40% of our cost” unfalsifiable. Section 9 of this article tackles exactly this problem and shows how to turn routing effectiveness into something measurable, using an oracle upper bound and regret curves.
The order in which this article proceeds follows the real dependency order of the engineering work: first define what you are optimizing, then build the classifier, then the capability matrix, then turn cost and latency into hard constraints, then build the fallback chain and cascade, and finally solve measurement and configuration-drift governance. Every step comes with an explicit boundary of applicability — half of the engineering effort in a routing system goes into knowing when not to do it.
1. Why You Must Build Routing in 2026: Four Ceilings of Single-Model Systems
1.1 The capability ceiling: averages hide the long tail
The benchmark score of a flagship model is a curve that has been averaged away. What you see on a leaderboard is its average performance across hundreds of problems, while your production traffic follows a completely different distribution: a large volume of templated classification and extraction requests, a smaller number of complex decisions that need multi-step reasoning, and an even smaller batch of compound tasks combining long documents, charts, and cross-paragraph linkage. Those three kinds of requests push the model in different directions, and the average hides the cases where “on some category the flagship actually loses to a specialized small model.”
The more practical problem is the asymmetry of failure cost. Both are structured-extraction failures, but one routed wrongly corrupts downstream data and triggers a manual rollback, while a casual chat answer that reads badly goes unnoticed by the user. Spreading the budget evenly across all requests means solving the least important problems with the most expensive model. The first thing routing has to do is explicitly encode “how bad is the consequence if this request fails” into the decision.
1.2 The economic ceiling: lower unit price does not mean a lower bill
Model unit prices have been falling for the past three years, yet many teams’ inference bills have not fallen proportionally — some have even risen. The reason is almost always the same: lower prices make calls cheaper, so call volume grows faster, especially once model capability improves and scenarios that were previously “done by humans / done by rules / not done at all” get unlocked. This effect is especially visible inside enterprise networks: intent recognition that was handled by regex matching gets switched to a small model, accuracy goes up, and the business side immediately hands over every ambiguous case they have.
Another frequently overlooked cost item is retries and rework. One failed call to a large model plus one retry can already cost more than three successful calls to a small model. If the failure rate is 8%, you have to subtract that leakage before the savings count as net gain. Any cost accounting that only counts “the token price of the first call” systematically overestimates the benefit of routing.
1.3 The latency ceiling: queueing and time to first token
End-to-end latency decomposes roughly into network and queueing, TTFT (time to first token), and decode time that accumulates with output tokens. Routing affects all three: switching to a larger model raises both TTFT and per-token decode time; switching to a smaller model does the opposite. The subtler factor is queueing — the same model can sit at a P99 of 800 milliseconds under low load and reach double-digit seconds of queueing at peak load, and a router that only looks at average latency will keep pushing traffic into channels that are already overloaded.
So routing decisions must be based on quantiles rather than means, and current channel load has to be treated as a real-time input. A router that chooses purely from an offline capability table and never looks at live congestion will step on both the latency and the error-rate trap at the same time during traffic peaks.
1.4 The supply ceiling: rate limits, regional availability, and channel differences
The model supply landscape in 2026 looks like this: no single vendor covers all requirements anymore, and every vendor enforces strict rate limits and regional quotas. That turns “multi-model” from an optimization option into availability infrastructure — you need at least two channels that are substitutable in capability, otherwise any rate limit or regional outage becomes a user-visible incident.
And that is precisely why most teams take routing seriously for the first time: not to save money, but because some vendor degraded continuously for two weeks in your primary region last month. Supply redundancy is the most rigid driver of routing; cost optimization is the side benefit. Get the order backwards and you easily end up with a fragile system that can only save money, never provide a safety net.
1.5 Which of the four ceilings usually bites you first
The four ceilings do not trigger in the same order, but they can be ranked with a rough heuristic. The supply ceiling usually triggers first, because vendor-side rate limits, regional degradation, and sudden capacity crunch are unpredictable — and once they trigger, they are hard failures. Many teams only decide to build routing after being rate-limited twice, by which point the architecture already contains a large amount of hardcoded model names, and retrofitting costs far more than designing aliases from day one.
The latency ceiling follows closely behind. It is more insidious: not a worse average latency, but a sudden spike in P99 for a certain time window, a certain request class, or a certain region, while monitoring still reads “overall normal.” These problems are hard to diagnose precisely because they relate to the composition of traffic rather than its volume, so they only become visible when routing decisions are sliced and displayed by task family.
The capability ceiling emerges slowly. It does not explode on any given day; it erodes incrementally as product features ship: a new scenario defaults to the flagship because “only the flagship can handle this,” and over time flagship volume grows tenfold even though most of that traffic is exactly as simple as it was on day one.
The economic ceiling is usually last, and the most insidious of all. It shows up as “the bill went up and nobody can say why,” because a rising bill usually comes from three overlapping sources — new traffic from new features, longer outputs per request, and a higher retry rate. Attributing those three separately is the prerequisite for making cost optimization a system rather than a patch.
This ordering has one direct practical implication: do not wait until all four ceilings have appeared before you start building the routing system. Start with the alias abstraction and the decision log (both cheap to build); they turn the first three ceilings into observable events. The capability matrix and routing policy can wait until enough data has accumulated. A step with near-zero upfront cost and immediate payoff deserves to be done on its own.
2. Define the Problem First: Which Quantity Are You Actually Optimizing
Before writing any code, you need one explicit optimization objective. Define the value of a single request as a comparable quantity rather than a blend of several incompatible metrics. The most practical form converts “quality” into money:
The net value of one request = business revenue − (token cost + latency penalty + failure and retry cost + human intervention cost)
Every term in that expression is measurable and attributable, and all of them can be collected onto a unified span through OpenTelemetry’s GenAI semantic conventions (gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens are standard fields). With a single definition like that, “saving money” and “improving quality” finally become two sides of the same thing — otherwise cost optimization and quality optimization end up fighting each other in design review.
Concretely, a routing decision is usually a constrained maximization problem rather than a single-objective score:
- Hard constraints: latency P99 ceiling, budget ceiling, data-residency requirements for a specific region, structured-output contracts that must be satisfied;
- Soft objectives: subject to the hard constraints, maximize the quality score or minimize the cost per successful request;
- Tie-breakers: when quality is close, prefer the channel with lower load, higher cache hit rate, and steadier latency.
This formulation has one direct benefit: it makes degradation natural. Degradation is not an emergency move during a system failure; it is the optimal solution when the budget tightens. When you cut the budget from $0.02 to $0.005, the optimal policy automatically moves from the flagship to the small model, and no separate degradation switch is needed. Conversely, a design like “small model normally, big model on failure” is usually counter-intuitive — during a failure you should degrade to preserve availability first, not escalate to preserve quality. That point comes back later.
3. Task Classifiers: Three Routes and Their Failure Modes
3.1 Rules and keywords: zero cost, but head-of-traffic coverage only
The simplest route is rules: length thresholds, whether the input contains a code block, whether it hits business keywords, whether it carries an image attachment, whether it comes from a high-value customer. The advantages of such rules are zero inference cost, full explainability, and zero cold start, and in many scenarios they cover 30% to 50% of traffic — traffic whose routing decisions are almost never wrong.
Their ceiling is just as clear: they can only handle tasks where “surface features” correlate with difficulty. Once difficulty decouples from surface features, rules fail. The classic counterexamples are short requests that require multi-step reasoning (“which of these three options is better,” with the three options in an attachment), and long but mechanical tasks (finding a specific pattern inside a one-million-word log). Length is the most easily misused heuristic: long does not mean hard, short does not mean easy.
The practical recommendation is to put rules at the first layer of the routing chain, but not to treat them as the body of the classifier. Their real value is providing a zero-scheduling fast path for the learned classifier downstream, and acting as a fallback when that classifier’s confidence is low.
3.2 Embedding-similarity routing: the most practical default starting point
If you only want to implement one kind of classifier, pick embedding similarity. It turns “does this request look like those known to need a particular model” into a single vector search, runnable with a small embedding model (a few hundred MB, millisecond-level on CPU). The advantage of this approach is that it requires no taxonomy of labels — the labels come from historical samples themselves, namely “which model handled this request correctly at the time.”
# kNN 风格路由：用历史请求的“正确模型”作为弱标签，
# 把新请求投到最近的若干条历史上做加权投票。
# 关键前提：向量已 L2 归一化，内积等价于余弦相似度。
import numpy as np
VECTOR_INDEX = "hnsw" # 服务端用 HNSW，客户端持有同一份 embedding 模型
TOP_K = 12 # 投票邻居数：太小方差大，太大抹平难度边界
def route(query: str, ctx: dict) -> str:
q = embed_client.encode(query, normalize_embeddings=True)
# 只在“同任务族”的候选集内检索，避免跨域相似度污染
candidates = index.search(
q, k=TOP_K,
filter={"tenant_id": ctx["tenant"], "locale": ctx["locale"]},
)
votes = {}
for sim, item in zip(candidates.scores, candidates.items):
if sim < 0.55: # 相似度太低：交给下游回退逻辑
continue
w = float(sim) ** 3 # 锐化：让最相似的邻居主导投票
votes[item.model_id] = votes.get(item.model_id, 0.0) + w
if not votes:
return ALIAS_DEFAULT # 明确回退，而不是随机挑一个
return max(votes.items(), key=lambda kv: kv[1])[0]
Three key engineering details of this route are routinely skipped. First, you must use filtered search: similarity across different tenants, languages, or business lines is not comparable, and voting over a mixed pool will route a Chinese customer-service request to an English code model. Second, the similarity threshold is not optional: kNN is naturally inclined to always produce an answer, even when the nearest neighbor is completely irrelevant; without a threshold you get the degenerate behavior where “every request is routed to whichever model appears most often in history,” which is nearly guaranteed to happen in a system with a long-tailed distribution. Third, the embedding model itself is a versioned asset — change the version and you must rebuild the index, and the comparability of historical labels goes with it. Section 10 comes back to this.
3.3 LLM-as-a-router: hand the judgment to a smarter model
The logic here is direct: if choosing a model is itself a reasoning task, use a cheap but sufficiently capable model to make the call. In practice that is usually a model below a few dozen billion parameters plus a structured routing prompt, asked to emit a difficulty level or a capability-requirement tag, which deterministic code then maps to a concrete model.
Its biggest advantage is that it is explainable and handles semantics: you can require it to state “this request needs cross-document reasoning” before emitting the tag, which gives you an auditable justification. Its biggest disadvantage is equally direct: it is itself one inference call, and its latency and cost count toward the total. If it costs more than the target small model, or if the classification gain it brings is eaten by its own overhead, the route does not pay off. In practice it fits high decision value, moderate request volume — contract review, complex ticket triage — rather than high-QPS classification endpoints.
There is also a counter-intuitive but important finding: an LLM router’s judgment is only accurate on samples where the small and large model outputs clearly differ. If two models perform similarly on a class of tasks, no amount of router “intelligence” can tell you which one to use — the request really can go either way, so there is no routing gain to capture in the first place. Do not spend your effort on raising classification accuracy; first confirm whether the gap between models on this task class is wide enough. This is a severely underestimated precondition.
3.4 Trained routers: learn a decision boundary from preference data
The representative approach on this route trains a lightweight discriminator on human preference data to directly predict “is the stronger model better?” Its intellectual origin is pairwise preference comparison: given the outputs of a large and a small model for the same prompt, have annotators or a strong model judge which is better; if the preference distribution between strong and weak models has a stable boundary, learn that boundary as a classifier that directly outputs an “escalation probability.” Compared with rules and kNN, its advantage is that it can absorb continuous signals far richer than discrete labels.
But three preconditions must hold, and missing any one of them means failure. First, you must have paired outputs from both candidate models over the same batch of prompts, which means paying a significant up-front cost of double inference. Second, the preference data must cover your real traffic distribution, otherwise what you learn is a boundary that is accurate on generic question answering and badly biased on your vertical task. Third, the escalation probability threshold must be tied to the cost model: the expected value of escalating is P(升级正确) × Δ质量 − 额外成本 − 额外时延, so the probability threshold is not a fixed 0.5 — it is the solution to that inequality.
The most elegant by-product of a trained router is the continuous confidence it produces. That probability naturally drives cascade policies (Section 8), budget constraints (Section 6), and canary rollouts (Section 10), so even if you start with a simple kNN, design the routing output from day one as a distribution with confidence rather than a hard label — it is the interface decision that is easiest to postpone and hardest to retrofit.
3.5 One signal is never enough in production: layering and fallback
Finally, layering. Every mature router is a cascade of “cheap signal first, expensive signal later”: a rule-based fast path (zero cost) → embedding similarity (milliseconds) → a trained router (sub-millisecond) → LLM discrimination (one small-model call). Low-confidence requests bubble up level by level instead of opening fire with the most expensive option right away.
Fallback paths must be explicitly declared, not left as implicit safety nets. A common incident looks like this: the classifier throws an exception → the client library retries by default → the retry lands on the default flagship model → an unexplainable spike appears in that night’s cost curve. Writing the fallback as an explicit rule in configuration, with its own cost alert, matters more than anything else.
3.6 Multi-label routing: one request often needs several capabilities at once
Everything so far has assumed a “single label, single model” shape: decide which category a request belongs to, then pick one model. In real business settings that assumption often fails. A typical example is a request to “read the contract in the attachment and generate revision suggestions.” It simultaneously needs long context (the contract is long), structured output (the suggestions have to land in fields), domain accuracy (legal terminology must be right), and a certain depth of reasoning. The optimum for those four requirements may not come from the same model — long context may call for one model, structured constraints for another, and final generation for a third.
There are two routes for handling compound requirements. Route one scores capabilities and then selects a single Pareto-optimal model, requiring one model to satisfy several capabilities at once; it is simple to implement, at the cost of possibly failing to find such a model and therefore compromising on one dimension every time. Route two splits the request into a pipeline — a model with long context plus structured output for extraction, a strong reasoning model for inference, a cheap small model for validation. Route two has a higher ceiling on savings but is an order of magnitude more complex: it introduces the problem of passing intermediate results (errors in the extraction stage pollute every stage after it) and the problem of error accumulation.
A pragmatic middle form is to split only one or two segments: find the segment of the request with “the strongest format requirement” (say, an extraction that must emit strict JSON), handle it alone with a specially optimized model, and keep the rest as a single call. This compromise usually covers the added complexity, because it splits only the most deterministic segment — and the most deterministic cut point is also the easiest to evaluate and the easiest to attribute.
One caveat: multi-label routing makes evaluation harder. Labels go from one dimension to several, the router’s error rate is no longer a single number, and the boundaries between task families blur because of combinations. The practical response is not to pursue a complete expression of multi-label routing, but to keep only sparse signals that rules can detect directly, such as “requires long context,” “needs to see images,” “must emit JSON,” “over 20,000 tokens.” These signals have high detection accuracy, strong explainability, and they happen to be exactly the dimensions most likely to cause a “wrong model selected” mistake.
4. Cold Start: From Two Hundred Samples to a Classifier You Can Ship
4.1 Derive labels from outcomes first; do not design a taxonomy first
The most common mistake during cold start is to sit down with the business side, design a five-level task taxonomy, and then label data according to it. The problem is that this taxonomy is unimpeachable on paper, but the moment you start labeling you will find that a large number of samples cannot be classified at all, and in the end you either force-fit them or throw them away. A more workable path is to go the other way around: run the existing system for a while first, then observe from the outcomes which samples show the largest performance differences across models, cluster only the regions where those differences are clear, and derive a data-driven task partition.
Concretely: sample a batch of real requests (a few hundred to a few thousand is enough to start), run each of two or three candidate models over them, and record quality scores and cost. This single step gives you three things at once: task clusters, the within-cluster model gap, and a reusable regression baseline. Its cost is a one-time double inference run, which is typically repaid by later savings well before launch.
This step also has an overlooked value: it tells you whether routing is worth doing at all. If on your traffic distribution the cost per unit between your strongest and weakest models differs by only 1.5x, and the quality gap comes mainly from the prompt rather than the model, then the ceiling on routing’s benefit is locked in, and effort is better spent on prompt engineering and constrained decoding. The best time to make that call is during cold start, not three months after launch.
4.2 Shadow traffic and active learning: bind evaluation and labeling together
The labeling budget during cold start is extremely limited, so every label has to be spent where it counts. Shadow traffic is the most effective technique: online requests proceed under the current policy as usual, while an additional call to a “comparison model” records the difference. Samples with a large difference automatically enter the labeling queue; samples with a small difference are skipped. This is essentially applying active-learning uncertainty sampling inside a production system.
# 影子评估 + 主动学习队列：把标注预算集中在决策边界附近
def on_request(req, primary_result):
# 1) 只在“可能有信息量”的请求上付出对照调用的成本
if request_value(req) < SHADOW_MIN_VALUE: # 低价值请求不值得花对照成本
return primary_result
alt = call(alias=SHADOW_ALIAS, request=req) # 通常是比 primary 更强的模型
# 2) 用便宜的判据估计两者差距，而不是每次都人工确认
gap = judge(alt.text, primary_result.text)
# 3) 只有边界附近的样本才消耗人工标注预算
if UNCERTAIN_LO <= gap <= UNCERTAIN_HI:
labeling_queue.push(req, primary_result, alt, gap)
# 4) 但无论是否标注，决策日志必须落盘（第十节要用）
decision_log.append(req, alias=SHADOW_ALIAS, gap=gap)
return primary_result
This pipeline has an important property: it produces labeling data and routing-effectiveness evidence at the same time. The decision log written in step 4, together with the comparison-model results, forms an evaluation set that can be replayed offline. Three months from now, when you swap the classifier, change models, or tune thresholds, you can replay all of them on this log and compare directly without re-sampling. This is the highest-leverage piece of infrastructure in a routing system, without exception.
4.3 The three mistakes most easily made during cold start
- Train on the dev-set distribution, ship on the production distribution. The typical symptom is 90% offline classification accuracy with widespread mistakes online. The root cause is usually not overfitting but a genuinely different distribution: production has longer inputs, stranger multi-turn histories, and contexts truncated by upstream systems.
- Ignoring the definition of “correct.” What does it mean for a request to be “handled well” — correctly formatted, factually correct, or accepted by business acceptance? If this is not pinned down during cold start, every later quantification becomes meaningless (Section 9 returns to this). The definition should come from business acceptance criteria, not from your intuition.
- Shipping a complex router too early. A three-class classifier trained on 200 samples is far more complex than a kNN built on 3,000 samples. When you are short on samples, kNN plus rules is almost always the best value for money.
5. The Capability Matrix: How to Maintain It So It Does Not Rot
5.1 Which dimensions the matrix should have
The capability matrix is the precondition for turning “which one do I pick” into a data lookup. The root reason it rots easily is that people record capability without recording applicability scope, strength of evidence, and validity period. A usable matrix needs at least the dimensions below, and every row needs metadata:
| Dimension | What it does | Metadata requirement |
|---|---|---|
| Task family | A row of the matrix, usually 8 to 20 of them, coming from the data-driven clustering in 4.1 | Definition, last validation time, sample size |
| Quality score | Relative performance on that task family, allowed to carry a confidence interval | Evaluation set version, evaluation time, whether the difference is statistically significant |
| Cost profile | Input/output token unit prices, typical length distribution, cache hit rate assumptions | Price version, effective time (a price cut must be timestamped) |
| Latency profile | Quantiles for TTFT and TPOT, including values at typical input lengths | Measurement region, measurement time, load conditions |
| Constraint capability | Context window, max output, structured output, tool calling, multimodality, batch support | Vendor documentation version or the date of an actual verification |
| Supply attributes | Rate limits, regional availability, model alias stability, degradation channels | Quota tier, SLA terms, effective time |
| Compliance attributes | Data retention policy, training-use terms, regional compliance level | Contract version, review date |
5.2 From a score sheet to capability probes
The traditional approach is to maintain a manually scored spreadsheet that operations updates once a quarter. The problem with such a sheet is not that it is inaccurate, but that its refresh cycle is an order of magnitude slower than reality: minor model version updates, price changes, regional outages, and vendor policy shifts all happen on a weekly or even daily cadence.
A more sustainable approach is to split the matrix into capability probes — a set of automated, extremely small probe requests, one vote per attribute in the matrix. The context window is verified by triggering truncation with one over-long input, structured output is verified with a strict JSON Schema contract, multimodality is verified by sending an image and checking whether it gets referenced, and rate limits are inferred from the frequency of 429s. These probes can hang off a scheduled job, and what they produce is a timestamped capability snapshot rather than a static table.
The key design decision is to separate probing from evaluation: probing verifies “can it” (capability boundaries, hard facts, easy to automate), while evaluation measures “is it good” (quality scores, which need business annotation and update more slowly). Many teams merge the two into one table, and the result is that every price update also forces a full quality re-run, after which the matrix stops being updated altogether. Storing the two kinds of data separately is the precondition for the matrix surviving long term.
5.3 Four forms of rot and how to detect them
Matrix rot never shows up as an “obvious error.” It has four insidious forms.
- Silent vendor changes: minor version updates, regional deployment configuration changes, default sampling parameter shifts. The countermeasure is to keep a golden regression set on the critical task families (a few dozen fixed samples, re-run automatically on every change) and alert when the quality score drops past a threshold.
- Prices decoupled from actual usage: the token unit prices in the matrix are correct, but your actual average output length is drifting (prompt iteration, more few-shot examples). The countermeasure is to store measured quantiles rather than theoretical values in the cost profile, and to attribute cost back to business features.
- Task distribution drift: a new feature ships and brings a new request type that has no row in the matrix at all. The symptom is a new cluster showing up in the routing decision distribution outside the matrix’s coverage. The countermeasure is to cluster-monitor the decision log and trigger a matrix backfill when a new cluster exceeds a size threshold (this is also part of the drift governance in Section 10).
- Expired but not marked: a capability record that no decision has hit in three months looks exactly as trustworthy as one validated yesterday. The countermeasure is to give every row a TTL; expired rows are skipped at decision time and trigger a re-evaluation instead of being used as-is.
5.4 The runtime carrier of the matrix: aliases and capability sets
The matrix ultimately has to land on a structure that is queryable at runtime. The recommended approach is a two-level abstraction: the upper level is a capability alias (the only thing the business knows about), the lower level is the concrete models and channels (decided by configuration). Business code asks for “a medium-cost model that can read long documents and emit JSON reliably,” not for “some specific model number.”
# 能力别名 -> 能力集 -> 具体渠道。别名是路由的寻址单位，不是模型名。
CAPABILITY = {
'fast_chat': {'max_in': 128_000, 'structured': True, 'tier': 'cheap'},
'deep_reason': {'max_in': 200_000, 'structured': True, 'tier': 'frontier'},
'vision_doc': {'max_in': 32_000, 'vision': True, 'tier': 'mid'},
'long_extract': {'max_in': 400_000, 'structured': True, 'tier': 'mid'},
}
# 一个别名可绑定多个渠道：能力相同，供给与价格不同，由路由实时挑选
ALIAS_BINDING = {
'fast_chat': ['vendor-a/mini-2', 'vendor-b/flash-2', 'vendor-c/turbo'],
'deep_reason': ['vendor-a/frontier', 'vendor-b/reasoner'],
}
def candidates(alias, ctx):
"""查能力集 -> 过滤硬约束 -> 再交给策略层按质量/成本/负载打分排序。"""
cap = CAPABILITY[alias]
for c in ALIAS_BINDING[alias]:
if c.max_input_tokens < ctx.tokens: # 上下文窗口不够，直接出局
continue
if not ctx.allowed_region(c.region): # 合规硬约束
continue
yield c
This abstraction brings three direct benefits: business code needs zero changes when you switch vendors; a single capability can have multiple channels for failover; and the policy layer for cost and latency is decoupled from the concrete model, so it can evolve independently. Its cost is one extra resolution step, plus the debugging difficulty that comes from “the business side does not know which model actually served it” — and that is compensated by the full trace in the decision log, covered in Section 10.
5.5 Manual overrides: necessary, but they must expire
Even a well-oiled automated process hits moments that require immediate human intervention: a model has a temporary failure and must be blocked right now, a campaign needs one alias switched wholesale to the highest configuration, or a vendor has a security issue and must be delisted immediately. The capability matrix must support this kind of manual override; otherwise the only option left is to change code and ship a release, which takes anywhere from a few hours to a few days.
But manual overrides are the number-one source of rot in a routing system, because they bypass every automated check. Overrides must therefore obey three rules of discipline: every override needs an owner and a reason (otherwise nobody knows why the model was blocked, and nobody will ever unblock it); every override needs an expiry (a temporary block should expire automatically once the incident is over rather than living in configuration forever); every override must be traceable in the decision log (otherwise you cannot tell a policy decision apart from a manual intervention when something goes wrong).
There is one more piece of experience worth stating: the number of manual overrides is itself a health metric. If a given alias needs several manual interventions every month, its automated policy is far from reality, and you should go back and fix the policy instead of piling on more overrides. A matrix whose override count keeps growing is using operations work to paper over a modeling problem.
6. Turn Cost and Latency Budgets Into Hard Constraints
As noted earlier, degradation is a natural consequence of the optimization framework. To make that actually happen, the budget has to be a first-class citizen rather than a comment scattered across each team’s configuration. There are at least three layers of budget.
def resolve_plan(request, budgets, matrix):
# 1) 先按硬约束过滤：上下文窗口、合规、速率限制、实时负载
pool = [c for c in matrix.candidates(request)
if c.region_allowed(request)
and not c.rate_limited(now)
and c.load_p95() < budgets.max_queue_ms]
# 2) 在候选里按“净价值”排序，而不是按单价排序
def net_value(c):
expected_cost = estimate_cost(c, request) # 含缓存命中与典型长度
p_success = matrix.success_rate(c, request.task_family)
retry_cost = (1 - p_success) * expected_cost * RETRY_FACTOR
latency_penalty = c.p99_ms(request) / budgets.ms_value_ratio
return (p_success * request.value) - (expected_cost + retry_cost + latency_penalty)
ordered = sorted(pool, key=net_value, reverse=True)
# 3) 预算不够时，不是硬切，而是从次优解里挑“质量损失最小”的那个
for c in ordered:
est = estimate_cost(c, request)
if est <= budgets.per_request:
return Plan(model=c, mode='primary', est_cost=est)
return Plan(model=ordered[-1], mode='degraded',
est_cost=estimate_cost(ordered[-1], request))
The most notable thing in this code is that the sort key is net value, not unit price. A system sorted by unit price systematically prefers cheap models and ignores the retry cost implied by differences in success rate; a system sorted by net value instead makes the right call between a “cheap but only 60% successful” channel and a “more expensive but 95% successful” one. This single change usually pulls the realized benefit of routing back to roughly the design target.
6.1 Three budget layers: per request, per session, per day/month
The per-request budget is the easiest to implement: how much a single call is allowed to spend, mapped directly onto max_tokens and model choice. It suits stateless interfaces. The per-session budget handles multi-turn and agent scenarios: one task may invoke the model twenty times against a fixed total budget, and the router’s job is then to allocate — concentrating budget on the critical steps (say, spending 60% up front on retrieval and planning, then reserving 40% for final generation) rather than splitting it evenly per step. That is the same problem as budget allocation in test-time compute scaling.
The daily/monthly budget is an organizational constraint, and it needs quota allocation plus a policy for overage. One anti-pattern seen in practice is spreading the daily budget linearly across business owners by request volume, which gives low-value and high-value traffic the same allowance. A better approach is to give each business owner an independent daily allowance that rolls over when unused, so budget naturally flows to where it is actually being used.
6.2 When the budget is gone, what do you cut first
When the budget runs out, do not simply refuse service, and do not simply fall back to the cheapest model. The correct order is to cut output length first, then reasoning depth, and only then switch models. The reasoning: the marginal quality loss from shorter output is usually the mildest (a user missing one sentence loses less than a user getting one sentence wrong), reasoning depth (thinking steps, sampling count) costs a lot on medium- and high-value requests, and switching models is the bluntest instrument of all.
Another frequently overlooked budget item is the context budget. In long conversations, input token growth is quadratic: every turn resends the full history. Without context trimming or prefix cache reuse you will watch the bill grow non-linearly with turn count, and at that point “switch to a cheaper model” does absolutely nothing. Optimizing cost on the input side and model routing are two different things, but they have to live on the same budget dashboard.
6.3 Converting business value into budget: the one step product must own
Every budget parameter above ultimately lands on one question: how much is one correct answer worth? This is not a technical question, but it determines the optimization direction of the entire routing system, and there is no way around it. If the answer is “one customer-service response is worth roughly one percent of the cost of a human handling it,” then the budget ceiling is very clear and the routing policy naturally leans toward the low-cost option; if a single miss costs a seven-figure contract loss, then the budget is so loose it barely constrains anything, and the policy leans toward quality first.
In practice, something more useful than a precise number is tiering: split requests into three to five tiers by business value, and give each tier a budget range and a latency range. Tiering works because it is rough enough to be stable — product can simply decide it — and useful enough, since the routing system only has to choose among tiers rather than handle a continuous value signal. Most teams should start here rather than trying to build a precise value regression model, which usually demands far more data and maintenance than it returns.
The second thing product must own is the visibility requirement for degradation. “The answer got shorter” is irrelevant degradation in a casual chat setting and possibly unacceptable in legal document work. Only the business side can state that requirement. Its implementation cost is trivial — a flag in the response and a UI decision about whether to show it — but if it is not settled in advance it usually comes back as an incident after launch, in the form of “users complain the degraded answers are too short.”
The third is to put a price on “uncertain”. When the routing system’s confidence is too low to decide whether to escalate, what should the default behavior be? The conservative choice is to escalate (protect quality, sacrifice cost); the aggressive choice is to let it through (protect cost, sacrifice quality). There is no technically correct answer — it depends on the business’s relative tolerance for the two error types, and it must be stated and configured explicitly rather than left to whatever default branch the implementer happened to write, which is usually the most conservative one, so the system silently picks the high-cost path with no product decision behind it.
7. Fallback Chains and Failover: Order Decides Whether It Works
7.1 Degradation is not “swapping in a worse model”
The most common mistake in fallback-chain design is a single model list sorted by descending capability, walked downward whenever something breaks. That design works in the lab and fails in production in three ways:
- Semantic non-equivalence: dropping from “reads 400,000 tokens” to “reads 120,000 tokens” gives the user an answer that was silently truncated, which is worse than an explicit error. Every step of a degradation must be semantically equivalent, otherwise it is a lie rather than a degradation.
- Latency stacking: if all three hops of a degradation fail, the total latency is the sum of the three plus retry backoff, and can easily blow past the client timeout. A fallback chain needs an independent overall deadline rather than each hop budgeting on its own.
- Cascading avalanche: the primary channel wobbles → all traffic floods the backup → the backup gets knocked over. The fallback chain itself becomes a fault amplifier.
The more robust design is to declare paths per failure type rather than one generic chain:
# 降级路径按“请求形态 + 故障类型”分别声明，而不是一条通用瀑布。
PATHS = {
# 首选语义等价替换：换渠道、换区域，能力不变
'soft': [('channel', 'next_best_channel'), ('region', 'other_region')],
# 能力收缩：必须同时收紧任务范围，避免静默截断
'shrink': [('reduce_scope', 'drop_optional_refs'),
('reduce_output', 'cap_tokens_to_512')],
# 最后才降级到规则/模板，这类答案必须在 UI 上明确标记
'terminal': [('deterministic', 'rule_based_extract'),
('user_visible', 'ask_user_clarify')],
}
HARD_DEADLINE = {'total_ms': 3000, 'per_hop_ms': 700}
def run(req, node):
path = PATHS[node.kind]
for hop in path:
if elapsed() > HARD_DEADLINE['total_ms']:
break # 整体时限优先于“按计划走完降级链”
try:
return invoke(hop, req) # 每个 hop 自身也有超时与取消
except Retryable as e:
observe(hop, e) # 失败计入该 hop 的健康度
return Terminal(user_visible_fallback(req))
7.2 Failover: separate retryable from non-retryable
The first rule of failover is: classify the error first, then decide the action. Errors fall into three categories, and each one is handled completely differently.
- Retryable and idempotent: 429 rate limits, connection timeouts, 5xx. The response is switch channel plus exponential backoff, with the backoff applied at the channel level rather than globally, otherwise one throttled channel slows down all traffic.
- Non-retryable: context length exceeded, content policy rejection, invalid parameters. The response is to degrade immediately and never retry — a hundred retries produce exactly the same result, and every retry burns money.
- Retryable but not idempotent: tool calls, write operations, agent steps with external side effects. The response is to execute on another vendor and carry an idempotency key, otherwise a call that “timed out but actually succeeded” produces duplicate side effects.
The third category is the one most easily overlooked. Cross-vendor failover means the same business operation may be executed by two different models, whose tool-calling conventions, parameter styles, and output formats all differ. The fix is to fix the tool-calling protocol at the gateway layer, forcing every candidate model to honor the same contract instead of assuming each model natively speaks your tool definitions. That is the real value of unified gateways (LiteLLM, Portkey, Envoy AI Gateway, and the like): their core contribution is not saving a few lines of code, but providing a semantically aligned layer for cross-vendor transfer.
7.3 Four principles for ordering the chain
- Equivalence before degradation: before any degradation step, exhaust all capability-equivalent channels. Switching channels does not affect quality, so it should not be treated as a degradation.
- Cache before model: a semantic cache or result cache hit is a zero-cost, zero-latency “best model.” It belongs at the very front of the chain. The precondition is a conservative enough similarity threshold, plus the ability to distinguish “identical” from merely “similar” — a wrong cache hit is more dangerous than a degradation on some tasks.
- Fast failure over slow failure: health checks and circuit breaking should be based on sliding-window error rate and P99 latency, and circuit-breaker recovery should use half-open probing rather than a fixed cooldown. A fixed cooldown under fluctuating traffic either recovers too early (knocking the channel over repeatedly) or too slowly (long unavailability).
- Degradation must be visible: every degradation must carry a flag in the response and land in the decision log. This looks like product fussiness but is actually life-saving: when users complain “answer quality got worse today,” the only way to answer “was it degraded, for how long, and who was affected” is this trail.
7.4 Drills: a fallback chain can only be validated when it actually breaks
Fallback chains have a well-known property: when everything is healthy they are dead code, so their correctness cannot be validated by normal use — and yet they are the only thing the system relies on at its most embarrassing moments. Worse, many branches of these chains have never executed a single time after the code was written, and when they are finally needed you discover that some fallback model has the same timeout as the primary (so it times out too), or that a cache some fallback path depends on was never populated on the cold path.
So a fallback chain needs regular forced drills, and it needs to drill hop by hop rather than end to end. Hop-by-hop means forcing one specific hop to fail and verifying that every later hop behaves as expected — whether it took the next path, retried incorrectly, or silently returned the error to the user. An end-to-end drill only answers “what happens when everything fails”; it cannot localize “what happens when the second hop fails.”
The injected failure types must cover at least four categories: connection-layer failures (connect timeout, connection refused), service-layer failures (error status codes returned), behavior-layer failures (success returned but with empty or malformed content), and slow responses (no response, just over the latency budget). The last one is the most frequently omitted and the most likely to cause a production incident: circuit breakers typically only count error rates and are blind to “slow but not failing,” and slow responses are exactly the class most likely to drag down an entire system.
The outcome of each drill should be distilled into a table: expected behavior per hop, observed behavior, and the difference. The table also has a side benefit — it turns “the fallback chain” from a chunk of code into a reviewable specification. In practice, newcomers failing to understand degradation logic is the most common handover obstacle in this kind of system, and with the table, code and spec correspond one to one.
8. Cascade: Try the Small Model First, Then Escalate
8.1 The mechanism: two stages plus a verifier
The idea behind cascade is very plain: try quickly with a cheap model, and if it succeeds, stop there; if it does not, escalate to the strong model carrying the earlier result along. This mechanism has been shown academically to work well for a large number of distributions, because the difficulty distribution of many tasks is bimodal: most requests are easy for any model, and a small remainder genuinely requires capability. A cascade separates the cost of those two cases.
async def cascade(req, budgets):
# 阶段一：便宜模型 + 收紧的输出上限（cascade 的收益来源之一）
draft = await call(alias='fast_chat',
request=req,
max_tokens=budgets.draft_tokens, # 通常是完整预算的 20~40%
reasoning_effort='low')
# 判定器：这是整个策略的成败所在，三种实现可靠性递减、成本递增
verdict = verify(draft, req) # 1) 确定性判据：schema 校验 / 规则 / 引用校验
# 2) 自检：让同一模型回答“此题是否已被完整回答”
# 3) 独立小判别器：另一次小模型调用做二分类
if verdict.ok and verdict.confidence >= budgeted_threshold(req):
return Outcome(text=draft.text, tier=0, cost=budgets.draft_cost)
# 阶段二：升级，并携带阶段一结果作为上下文（这一步会额外花 prefill，
# 是 cascade 最常被忽略的隐性成本）
final = await call(alias='deep_reason',
request=req,
context=verdict.carry_over(draft))
return Outcome(text=final.text, tier=1, cost=budgets.draft_cost + final.cost)
There is one extremely important and easily overlooked detail in this structure: the input to stage two includes the output of stage one. That means you have to account for two effects at once: “one less strong-model prefill for every request you save,” and “one extra long-context prefill for every request you escalate.” If the draft is long and the escalation rate is high, cascade can cost more than calling the strong model directly. Controlling the draft length ceiling is therefore not optional.
8.2 When it works and when it does not
| Condition | Why | Recommendation |
|---|---|---|
| Bimodal difficulty distribution with a very wide easy peak | Most requests hit on the first try, and only the long tail escalates | The ideal scenario; escalation rate can usually be held under 20% |
| Continuous difficulty distribution with no clear peak | Nearly every request lands right at the threshold, and the escalation rate approaches 100% | Benefit goes to zero or negative; use direct routing instead |
| The verifier is harder than the task itself | The verifier either wrongly passes (quality collapses) or wrongly escalates (benefit goes to zero) | Make the verifier simple and correct first; do not mask the problem by tuning the threshold |
| Extremely tight latency budget | Stage one, the verdict, and stage two are serial, so P99 degrades significantly | Use cascade only for non-realtime batch or offline work |
| Uncontrollable output length | A long draft makes the escalation prefill cost eat the entire saving | Enforce a max_tokens ceiling and hand off drafts in truncated form |
| The strong model carries vendor risk | The big selling point of cascade is that the strong model is called on only a few requests, which is more risk-tolerant | This is actually the most underrated benefit of cascade |
The second row of the table deserves expansion: the escalation rate is the single most important indicator for deciding whether cascade is worth doing. If the escalation rate exceeds 50% in offline replay, this strategy is mathematically almost incapable of turning a profit — you paid for two model calls on every request (stage one is unavoidable) while saving a strong-model call on fewer than half of them. Treating it as a pre-launch gate is more reliable than looking at average cost, because average cost is masked by the size of the “easy peak.”
8.3 The trap list
- False-positive escalation: the verifier is too sensitive, the escalation rate spikes to 80%, and all benefit evaporates. The symptom is cost going up instead of down. The diagnostic is to make the escalation rate a first-class metric on the daily dashboard and monitor the verifier with a confusion matrix.
- False-negative pass: the verifier lets a bad answer through, the problem is delivered downstream, and downstream has no signal that tells you anything went wrong. This is the most dangerous cascade failure because it never alarms; it only surfaces in user complaints. The only defense is to put the verifier’s miss rate into the regression suite and re-run it on every prompt or model change.
- Selection-biased evaluation: measuring the benefit of escalation only on samples where “the small model already failed” systematically overestimates the effect. Evaluation must run over the full real distribution.
- Routing jitter: cascade verdicts depend on random sampling, so the same request can land on different tiers on two runs. That turns your A/B experiments into noise, because “the same model” no longer means “the same set of outputs.” Either fix the sampling seed or accept this explicitly in the experiment design.
- Treating cascade as a fallback chain: the two run in opposite directions. Cascade is routine cost optimization (cheap first), while a fallback chain is availability insurance during incidents (availability first). Mixing them causes normal traffic to be degraded incorrectly during an incident.
8.4 Boundaries with reinforcement learning and test-time compute
Cascade, test-time compute scaling, and online reinforcement learning solve adjacent but different problems, and the boundaries are worth drawing because they are routinely conflated. Test-time compute scaling buys quality inside a single model by adding thinking steps, parallel sampling, and verifier reranking; its unit of cost is “more tokens of the same model.” Cascade chooses between models; its unit of cost is “a few tokens from a cheaper model, plus the occasional expensive call.” The two can be layered, but their optimization targets differ: the former tunes the quality-versus-compute curve, the latter tunes the quality-versus-money curve.
The boundary with online reinforcement learning for routing is subtler. Treating routing itself as a policy to optimize, trained from production feedback (did the user accept it, did the task succeed), is theoretically optimal and rarely works in practice. There are three reasons: feedback signals are delayed and noisy, so the quality of one routing decision may only surface in business metrics days later; the action space is discrete and the consequences are irreversible, so the cost of a wrong choice is immediate while the benefit is in the future; and exploration is expensive — random exploration online means actively serving a small slice of users worse, which is unacceptable in a lot of businesses.
So the more realistic approach is to train the router on offline pairwise preference data and do only constrained greedy selection online, confining exploration entirely to the offline environment. This is essentially an offline solve of a contextual bandit: the training phase can use logs over the full candidate set (with weighted corrections such as IPS to handle the distribution mismatch), and the online phase executes the policy without exploring. That is the same method as the trained router discussed in Section 3.4, just with the learning objective and the data source stated more explicitly.
One noteworthy new direction is to use the router’s own confidence for per-request reasoning-budget allocation: a high routing probability means the task is easy, so give it fewer thinking steps; a low routing probability means the task is hard, so give it more. This merges the “which model” and “how much compute” decisions into one, and can be seen as extending cascade thinking inside a single model. Its precondition is still the same old one — the router has to be trustworthy, or budget allocation will pile compute onto requests that should not have received it.
9. Quantifying Routing Effectiveness: How to Avoid Guessing
Back to the opening problem: the correctness of a routing decision cannot be observed by the system itself. To quantify it you have to introduce an external reference frame. The most practical one is the oracle — for every evaluation sample you have already run all candidate models offline, so “the best result achievable for this request” is known. The value of routing is simply how close it gets to that upper bound.
import numpy as np
def evaluate_router(log, candidates):
# log 里每条样本有：每个候选模型的质量分 q[m]、成本 c[m]、时延 t[m]
# oracle = 成本达标约束下质量最高的那个候选
gain, regret, cost = [], [], []
for row in log:
feasible = {m: r for m, r in row.runs.items() if r.cost <= row.budget}
if not feasible:
continue
best = max(feasible.values(), key=lambda r: r.quality)
chosen = row.decision
# 质量 regret：相对上界掉了多少
regret.append(best.quality - chosen.quality)
# 成本节省：相对“全用最强模型”省了多少（必须扣掉重试）
gain.append((row.strongest_cost - chosen.total_cost) / row.strongest_cost)
cost.append(chosen.total_cost)
return {
'quality_regret_mean': float(np.mean(regret)), # 越小越好
'quality_regret_p95': float(np.percentile(regret, 95)),
'cost_saving_vs_best': float(np.mean(gain)),
'abs_spend_per_success': float(np.sum(cost) / len(log)), # 比均值更诚实
}
A few easily confused points in this metric set need clarifying. Routing accuracy is the most abused and the least informative metric of all. It only holds when “you really did pick the best model,” but multi-model scenarios frequently have several tied best models; worse, it has no monotone relationship with benefit — pushing accuracy from 80% to 95% may not reduce cost at all, because the remaining 5% is precisely where the benefit is largest. Regret is therefore the primary metric, because it maps directly onto “how much value did I leave on the table.”
9.1 Use the oracle to define the upper bound, and to define the control groups
The oracle is worth more than computing regret. The three most useful control groups in practice are: always use the strongest model (upper bound on quality and cost), always use the cheapest model (lower bound on both), and the current production policy (the baseline). Your router has to prove that it is close enough to the upper bound while being far cheaper than the status quo. Plot those three lines on one chart and the benefit is self-evident — and the chart is fully readable by non-technical stakeholders too.
You should also draw a separate cost-quality Pareto curve: by gradually raising the escalation probability threshold, you slide the router along a curve that outputs a continuous “cost versus quality” trade-off. The existence of this curve is the core of the entire routing argument — it proves you are not getting lucky at a single point, but choosing a sensible operating point inside a continuous space. What stakeholders actually want to discuss is “that point,” not “that router.”
9.2 An operable evaluation process
- Build the evaluation set: sample from the decision log, cover all task families, and deliberately oversample boundary cases (the low-confidence part of the router’s decisions). Uniform sampling makes the metrics look good while carrying no discriminative power.
- Run all candidates: run every sample through every candidate model and record quality, cost, latency, and success rate. This is the source of the oracle data. The cost is one-time and worth paying.
- Report per task family: do not look only at the aggregate. Routing benefit varies enormously across task families, and aggregate metrics will hide the fact that “on some family you are persistently making negative-value decisions.”
- Draw the Pareto curve and pick a point: choose the operating point on the curve together with stakeholders, then freeze it into a threshold in configuration.
- Keep monitoring after launch: the metrics include cost per successful request (not cost per request), a rolling average of regret, the escalation rate, and traffic-distribution shifts per task family. If any of them crosses a threshold, go back to step 1.
9.3 Why offline metrics lie to you
- The evaluation set is not the production distribution. The most common version is that the long tail is underestimated in the evaluation set, while the long tail is exactly where routing decisions are most uncertain.
- Output length differences are ignored. For the same “correct answer,” one model outputs 50 characters and another 800, a 16x cost difference. Quality evaluation must treat length as a constraint or normalize by it.
- The cost of failure is ignored. A failure rate rising from 3% to 6% can noticeably degrade overall business metrics, but “average quality score” alone will barely show it.
- The latency tail is ignored. Better average latency with worse P99 is a net loss in an interactive product.
- The uncertainty of the evaluation itself is ignored. LLM-as-judge has position bias and length bias, so re-judging the same batch of outputs yields different conclusions. Every quality metric therefore needs a confidence interval, and when two options are not significantly different you should not draw a conclusion — this “looks like an improvement” trap is the most common source of fake gains.
9.4 How to mix offline and online evaluation
Neither offline nor online evaluation replaces the other, and each fails in its own way. The main problem with offline evaluation is sample selection bias: the evaluation set is sampled at some moment under some traffic mix, while the production mix keeps changing. An optimal threshold measured on March data may be completely wrong by October because a new feature shipped — not because routing got worse, but because the thing it optimizes for changed.
The main problem with online evaluation is that it is noisy, slow, and costly. Changes in business metrics (conversion, complaint rate, ticket resolution rate) usually need both substantial sample size and substantial time to surface, and while the canary is running you are changing other things too, so metric movement simply cannot be attributed. This leads many teams to launch an A/B test, run it for two weeks, see no significant difference, and hastily conclude “routing has no value.”
A healthy split therefore looks like this: offline for fast iteration, online for final confirmation. Offline replay over all-candidate logs can evaluate dozens of threshold combinations within a day and return results in seconds, which makes it the tool for daily tuning; online uses shadow traffic for risk-free validation (no decisions, only recorded comparisons), running continuously at controlled cost and providing a signal about whether the distribution has shifted; only genuine operating-point changes need a full A/B.
One more piece of experience about experiment design is worth stating on its own: a routing system makes A/B experiments harder to run. Because requests get routed to different models, requests in the “same experiment arm” actually go through different treatment, and the effect gets diluted. The mitigation is to stratify by routing decision — first look at whether the two arms differ on “the subset of requests handled by the small model,” then look at the overall difference. That both explains the experimental result and incidentally answers “is the router actually doing the right thing on this class of request.” Aligning the A/B stratification dimension with the router’s decision dimension is a design that improves experiment quality and routing quality at the same time.
10. Version Management and Configuration Drift with Multiple Models
10.1 Never hardcode a model name in code
The most expensive technical debt in a multi-model system is model names scattered across code, configuration, prompts, documentation, and frontend dropdowns. They exist in every conceivable form: hardcoded strings, the default value of some variable, an environment variable, a passing mention of “please use high reasoning effort” inside a prompt, even a colleague’s local dev configuration. You cannot possibly find them all when you upgrade a model.
# 路由配置是数据，不散落在代码里；配置本身要可版本化、可回滚、可审计。
@dataclass
class RoutingPolicy:
version: str # 语义化版本，改动必升
alias_budgets: dict # 别名 -> 每日美元额度
cascade: dict # 别名 -> {threshold, draft_tokens}
capability_overrides: dict # 人工覆盖，优先级高于自动指纹
fallback_order: list # 与故障类型绑定，不是一条通用瀑布
ab_buckets: dict # 灰度分桶，显式声明
def fingerprint(self) -> str:
# 配置指纹用于：变更检测、灰度一致性校验、回滚判定
return sha256(canonical_json(self).encode()).hexdigest()[:16]
The benefit of config-as-code goes beyond rollback: it is comparability. An identical fingerprint means identical behavior across two instances, which is mandatory in both canary rollouts and configuration-drift detection: the shadow instance, the replay system, and the production instance must share the same fingerprint, or every difference you measure is noise.
10.2 The decision log is the only after-the-fact evidence
The decision log must record far more than “which model was chosen.” The minimum field set is: request fingerprint (a redacted digest of the input, for deduplication and replay), the routing decision and confidence at the time, the candidate pool and the key attributes of each candidate (then-current quality estimate, cost, latency, load, whether it was filtered out by a hard constraint), the model finally invoked along with actual usage and real latency, the degradation tier that was hit, and the policy version fingerprint.
With it in place you can answer the questions that will definitely be asked afterwards: where did that batch of worse answers last week come from? Why is one tenant’s degradation rate four times that of another? After swapping the classifier, did things actually improve or did the problem just move to another task family? Without the decision log, every quantification in Section 9 above is a one-time conclusion from before launch and can never be verified continuously. The same log is also a hard requirement for compliance audits, where you must prove that a given piece of data was not sent to a disallowed region.
One practical detail is worth noting: the decision log has to be replayable without re-invoking the model. Storing the output of every candidate is enormous, so the usual approach is to store decisions and candidate attributes (cheap) and backfill outputs only when you need precise localization (expensive). This tiering gives you a rational answer to the question “is replay worth storing 10x more data?”
10.3 Drift detection and release cadence
Configuration drift has four observable signals, and all of them should become dashboard tiles:
- Decision distribution shift: the share of selections per alias, and the traffic distribution per task family. Compare against the previous time window using KL divergence or a simple change in bucket share. This is the earliest warning — usually one to two weeks ahead of any quality decline.
- Actual usage versus matrix prediction: how far the real token usage per alias deviates from its cost profile. A deviation that keeps widening means the matrix’s price or length distribution has gone stale.
- Quality drift: the golden regression set runs automatically after every change, combined with judge-score trends over online samples.
- Degradation and escalation rates: these two rates are the most sensitive health metrics there are. A sudden rise in escalation usually means the classifier lost accuracy or an upstream model degraded; a rise in degradation means the supply side has a problem or capacity is short.
On release cadence, the blast radius of a minor model version update is far larger than most people expect. It can change style, change tool-calling format, or even change how a particular language is handled, without changing the interface and without an announcement. So “upgrading a model” and “upgrading a routing policy” should go through the same release process: score a fixed regression set on a shadow instance first, then canary a small slice of traffic, then roll out fully. That process is not fundamentally different from releasing ordinary code, except that the shadow instance runs a mirror of real traffic — which is a perfect fit for the routing capabilities of a unified gateway, since the gateway is both the release channel and the observation point.
One more discipline is easy to overlook and matters a great deal: do not let the routing policy be modifiable at multiple layers. Many systems have a default route in code, dynamic configuration on the server, and a toggle in some operations console all at once. When those three layers coexist, which one wins depends on priority rules that themselves have no test coverage. A single launch incident is often just “someone on the backend flipped a switch.” Converging on a single source of configuration (the one in the code block at the top of this section) is the fundamental fix for this class of incident.
10.4 Where the gateway line sits: what to unify and what must stay on the business side
Building a multi-model setup easily swings to the opposite extreme: shove everything into the gateway, and end up with a black box nobody dares to touch. The workable split is by “is this business-semantic or not.” What belongs in the gateway is the machinery that is business-agnostic and must stay consistent across vendors: unified authentication and key management, request retry and backoff, rate limiting, semantic caching, telemetry instrumentation, structured-output contract validation, and cross-channel equivalent substitution. Re-implementing these on the business side can never be consistent and cannot be observed centrally.
What must stay on the business side is every decision that requires business judgment: what this request is worth, how bad the consequence of a failure is, whether a degraded answer is acceptable, whether the quality gap between two models is significant on this task class. Putting these in the gateway means the gateway has to understand your business semantics, and it will ultimately become a behemoth that neither understands the business nor is maintainable. The gateway should emit signals (cost, latency, success rate, current load, candidate availability); the policy layer on the business side should make decisions.
On gateway selection, one pragmatic piece of advice: pick the kind you actually need. If you need cost attribution, choose the one with good telemetry and usage aggregation; if you need cross-vendor failover, choose the one with clearly defined retry and degradation semantics; if you mix a self-hosted inference cluster with external APIs, choose one that accepts both. Chasing the “most feature-complete” gateway tends to leave it mediocre on some specific dimension, patched around with workarounds, which defeats the purpose of a unified layer.
There is also a boundary question that matters especially in self-hosted setups: should the models on your own cluster be part of the routing pool? The answer is usually yes, and the benefit is often underestimated — the cost structure of a self-hosted cluster is completely different from external APIs (low marginal cost but idle-capacity risk), which makes it naturally suited to traffic that is “high volume, with a simple difficulty distribution, tolerant of queueing,” while “high volume but hard” traffic is better left to external APIs. This division raises self-hosted utilization and external API value at the same time, and it is one of the most direct wins in a multi-vendor setup.
10.5 Vendor deprecation and migration: an operational event that will definitely happen
No matter how many vendors you use, model deprecation and SKU retirement are inevitable rather than surprising. Most vendors give advance notice, but in practice several situations make that window very tight: emergency regional deprecation, compliance-driven forced delisting, and major version releases that mark older minor versions unavailable outright.
This event demands more of a routing system than it first appears. The hard part is not swapping in a new model name, it is that the corresponding row in the capability matrix is empty. You have no evaluation data for a model that is already retired, and no evaluation data for the replacement on the same task family — because yesterday it may not even have been on the candidate list. So the genuinely effective preparation is not “have a replacement model ready”; it is to maintain a candidate pool that is wider than the list you actually use: every alias in service should have at least two evaluated candidate channels, one of which may currently be unused.
This “wider candidate pool” also has a less obvious benefit: it keeps the capability matrix continuously fresh. If a standby channel goes three months without being selected, its evaluation data is already stale, and discovering it fails the bar at activation time means you had no backup at all. Entries in the pool therefore need their own evaluation refresh cadence rather than following actual usage frequency.
The migration itself should follow the same process as any ordinary change: shadow instance comparison → small-percentage canary → observe decision distribution and quality metrics → full rollout. The special part is that you must let the old and new model coexist for a while rather than swapping in place. The value of running them side by side is that when the new model has quality problems you can immediately switch back; if you already replaced it in place with no way to roll back, the migration has become an irreversible bet.
The final discipline concerns not binding aliases to concrete model versions. If your business code says deep_reason_v3 instead of deep_reason, every model upgrade becomes a cross-team full-scale code change, and such changes are usually postponed indefinitely until they turn into a rushed launch. Converging version changes into the configuration layer is the only way to take migration cost from “weeks” down to “hours.”
11. When You Should Not Build Routing
A routing system is not simple. In the situations below you should explicitly not do it, and spending the budget elsewhere returns more:
- Small traffic volume or a single task type. A few hundred requests a day, or everything being one kind of task — routing benefit cannot cover build and maintenance cost. Do capability evaluation and prompt optimization first.
- A latency budget so tight it only allows one call. Both cascade and layered routing add serial segments, and then the only correct move is to fix on one model and put your engineering effort into caching and inference optimization.
- No evaluation capability on the team. Ninety percent of the value of a routing system comes from evaluation capability. If you cannot answer “which model is better on which task,” then however intelligent your router is, it is only optimizing a target you cannot state. With that precondition unmet, routing becomes a system whose claims cannot be falsified.
- Access to only one vendor. With no equivalent channel, “routing” degenerates into “picking one model,” and failover is off the table. What is more valuable then is building the unified gateway abstraction first, so you are ready for a second vendor later.
12. An Order of Operations That Actually Works
- Establish a single-model baseline first: fix on one model, tune the prompt, caching, and retry policy properly, and measure quality, cost per success, and P99 latency. Without a baseline there is no reference frame.
- Build the decision log and attribution first: instrument the
gen_ai.*style fields and attribute cost to business features even before you have any routing. This step requires no model decision logic at all, but it is the precondition for every later measurement. - Add capability aliases and a second channel: decouple business code from model names and bring in a second vendor. At this point you are solving availability, and you incidentally verify whether your gateway abstraction holds up.
- Run one all-candidate offline evaluation: obtain the oracle data and plot the cost-quality Pareto curve. This step tells you how high the ceiling on routing is, and which task families are not worth routing at all.
- Ship the rules + kNN version: start with the simplest, most explainable version, which has zero cold-start cost. Its goal is not to be optimal but to start producing decision logs, so all later work has a data foundation.
- Introduce learned routing one task family at a time: enable a more complex router only for families where evaluation shows “large model gap and enough samples.” Skip the rest; do not standardize across the whole site.
- Enable cascade for qualifying task families: gate it on the escalation rate and turn it off once it crosses the threshold. Fold the latency impact into the screen for non-realtime work.
- Make quality regression and configuration governance routine: the golden regression set runs automatically on every change, configuration has a single source with fingerprint verification, and the capability matrix has TTLs and a re-evaluation pipeline. Once you are here, routing has gone from “a feature” to “a system.”
If you can only do one thing, I would pick step 2. Most teams skip it and start training routers, then get stuck in “we cannot explain the effect” and keep iterating on gut feeling. Attribution and logging are the only pieces of infrastructure that are never a mistake to build now and extremely expensive to retrofit later.
Conclusion
Whether model routing is done well does not show up in classifier accuracy. It shows up in three plain numbers: the cost per successful request, P99 latency, and quality regret relative to the oracle. If those three numbers can be measured continuously, explained to the business side, and recomputed as model supply changes, then routing is a real engineering system. If they cannot, it is just a cost illusion wrapped in if-else statements.
Model supply in 2026 is richer than at any time in the past, and that has turned “which one do we use” from an occasional technical decision into a question that has to be answered on every request. What changed is the set of options; the methodology has actually been quite stable: define the problem as a constrained optimization, use the capability matrix to sediment the facts, use budgets and fallback chains to bound the uncertainty, and finally accept the result using regret rather than intuition. The more options you have, the more valuable this methodology becomes.
If you want to compress this system into one sentence: it does not make some model smarter; it turns “which model to use” from an individual’s on-the-spot judgment into a system-level decision process that can be regressed, audited, and explained. The first problem can only be solved by hiring; the second can be solved by engineering — and in a world where there is a new model, a new price, and a new rate limit every day, the second one compounds far more.
