Inference Engine Internals: Continuous Batching, Prefix Cache, Speculative Decoding

Why It's Hard to Push Throughput and Latency Down Together

Almost everyone who runs an inference service has hit this scenario: a stakeholder complains that the first token is slow, so you raise concurrency to make the system faster overall, and the first token gets even slower. This isn't a tuning-skill problem. In an inference system, throughput and latency are inherently in tension. The total number of tokens a GPU can produce per second is close to conserved; hand all of it to certain requests and the decode speed of any single one of them must drop. The real question isn't whether or not we want throughput, it's how to maximize throughput subject to a latency promise — because throughput directly determines the cost per token, and latency directly determines whether users churn.

To push both down at once, you need an interlocking set of scheduling mechanisms, not one silver-bullet optimization. This article breaks them down in this order: first, where static batching actually wastes resources; then what Continuous Batching gives up to get its gains; then memory paging, prefix caching, speculative decoding, preemption and fairness; and finally a production tuning order that ties it together. Every mechanism is covered with both its payoff and its boundary, because any single optimization will backfire on some workload.

First, two metrics that run through the whole article. The end-to-end latency of a single request can be roughly decomposed as:

Total latency ≈ TTFT + (number of output tokens − 1) × TPOT

TTFT (Time To First Token) is dominated by queueing time and prefill compute, while TPOT (Time Per Output Token, the interval between consecutive output tokens) is roughly proportional to the cost of one decode step. Lumping these two together makes every architectural decision reach the wrong conclusion — there's a dedicated section on this later.

Static Batching: Why It's Doomed to Waste

A Batch Held Hostage by One Long Request

The static batching flow is almost naive: wait until batch_size requests have accumulated, run one prefill over all of them, then decode autoregressively until the longest sequence in the batch finishes. The remaining short requests return with results already computed, and the leftover padding positions are simply discarded.

The problem is that last step. Suppose a batch holds 31 requests that each output 20 tokens and one request that outputs 500 tokens. Those 31 requests then have to run for 480 steps, and across all 480 steps they're doing meaningless work while GPU compute evaporates into thin air. The waste ratio approaches (500 − 20) / 500 = 96%. That is precisely what "short requests dragged down by a long one" means.

The second, parallel form of waste is padding. Static batching must allocate KV Cache according to the longest sequence in the batch, so the shortest requests hold memory they barely use and the effective concurrency is decided unilaterally by the longest one.

Low Compute Utilization Is a Symptom; the Memory Wall Is the Cause

Many people see GPU utilization at only 40% during inference and their first reaction is "it isn't being filled up." But for autoregressive decoding, low utilization is the characteristic: generating a single token requires reading the entire model's weights from HBM, and at this scale the matrix multiplications are nowhere near the compute roofline. This is memory-bound, not compute-bound.

What actually caps concurrency isn't compute, it's memory. Memory has to hold the model weights first; what's left over goes to KV Cache and activations, and the KV Cache determines how many sequences fit in a batch and how long each one can get. The KV Cache scales roughly as:

KV bytes ≈ 2 × number of layers × number of KV heads × head dimension × sequence length × bytes per element

The factor of 2 comes from holding both K and V. Take a model with 60 layers, 48 KV heads and a head dimension of 128: in FP16 that's about 2 × 60 × 48 × 128 × 2 ≈ 1.47 MB per token. An 8K sequence needs roughly 12 GB of KV on its own, and 16K needs 24 GB. Even on an 80 GB card with a 14B dense model loaded, what's left for KV only holds a handful of long sequences at a time.

This one formula explains nearly every engineering choice that follows: paging exists to waste less KV space, prefix caching exists to make already-occupied KV reusable, KV quantization exists to halve the "bytes per element" term directly, and MLA attacks the "number of KV heads" term of the formula. Every optimization ultimately points at the same thing — packing more usable KV into the same amount of memory.

The Overrated Advantages of Static Batching

Static batching's only advantage is that shapes are uniform within a batch: sequence lengths match, kernel parameters are fixed, CUDA Graphs are easy to capture, and Tensor Core utilization is high. Precisely because it aligns everything, it also wastes everything. The central idea of Continuous Batching is to give up that uniformity in exchange for letting any sequence enter or exit at any moment.

The table below breaks down the sources of waste; every optimization that follows maps onto one row of it:

Source of waste	How it shows up	Matching optimization
Inconsistent lengths within a batch	Short sequences idle alongside long ones until the batch ends	Continuous Batching
Padding overhead	KV allocated for the longest sequence, memory reserved for the worst case	PagedAttention paging
Repeated computation	The same prefix is prefilled again for every request	Prefix Cache
Coarse per-step granularity	A long prefill computes thousands of tokens at once, blocking all decoding	Chunked Prefill
Serial step count	The draft is generated serially for N steps, paying N× kernel launch overhead	Speculative Decoding
Tenants trampling each other	One big customer fills the batch while small ones sit in the queue	Preemption and fairness scheduling

Continuous Batching: Dropping the Scheduling Granularity to Token Level

What It Actually Does

Continuous Batching (also called in-flight batching or iteration-level scheduling) makes exactly one change: the scheduler re-decides who runs on every round. At the start of each step, finished sequences exit immediately, queued new requests fill the freed slots immediately, and prefill and decode get mixed inside the same step.

The key gain is that "time occupied" changes from "sequence length" to "your own length." In that 31×20 + 1×500 example above, under Continuous Batching the 31 short requests finish at their own step 20 and only the long one keeps running. The waste is essentially eliminated.

Formally, though, the batching window is a dynamic set: running = { s : s unfinished }. Each step takes one token (decode) or a chunk of tokens (prefill) from every sequence in running, and then the set is updated. Both the payoff and the cost come from that "recompute the set every step" property.

The Bill for Implementation

Turning "recompute the set every step" into code costs more than you'd think.

1. Lengths are no longer uniform within a batch, and the attention path breaks first. In static batching, every sequence can read contiguous memory from the same offset. Under Continuous Batching each sequence has a different length and its KV Cache is physically scattered, so attention has to support gathering through an indirect index table. This is the direct reason paged attention exists — it isn't a memory optimization, it's the precondition for making Continuous Batching run on a GPU. The two have to be done together.

2. The Python scheduler becomes the bottleneck. In early implementations, every step ran the Python scheduling logic: a batch of tensor slicing, index updates and sampling. At a few hundred concurrent requests, that CPU time can exceed the GPU compute time, and the GIL means threads don't save you either. The vLLM V1 engine pushes the scheduler down into C++ and keeps request state and the KV block table on the same side, cutting the data movement across the language boundary — one of the main engineering changes that let it hold up under high concurrency.

3. CUDA Graphs conflict with dynamic batch shapes. CUDA Graphs require fixed shapes at capture time, while the batch size under Continuous Batching changes every step. The common fix is to quantize batch size into a set of buckets (say 1, 2, 4, 8, 16, 32, 64, 128, 256), padding the actual batch up to the next bucket before capture. Buckets that are too dense cost memory; too sparse and padding waste climbs back up — another place where memory and latency trade off.

4. Sampling and synchronization overhead. One kernel launch, one synchronization, one sampling call and one out-of-band result fetch per step isn't cheap when step time is only a few milliseconds. Async scheduling, fused samplers and avoiding .cpu() synchronization are the standard optimizations at this layer.

A Minimal Scheduling Loop

The code below isn't the implementation of any production engine, but it makes the scheduling semantics of Continuous Batching explicit — note that batch membership is rebuilt every round, and that prefill and decode are mixed into the same step:

class Engine:
    def __init__(self, kv_manager, step_token_budget=8192):
        self.waiting = deque()      # 排队中的请求
        self.running = []           # 本轮要跑的序列（可能含 prefill 片段）
        self.kv = kv_manager
        self.budget = step_token_budget

    def step(self):
        batch = []

        # 1) 先给已在跑的序列分配额度，保证它们不会饿死
        #    decode 每条序列本轮只消耗 1 个 token 的预算
        decode_seqs = [s for s in self.running if not s.in_prefill]
        prefill_seqs = [s for s in self.running if s.in_prefill]
        spent = len(decode_seqs)

        # 2) 按预算把长 prefill 切块（chunked prefill）
        for s in prefill_seqs:
            room = self.budget - spent
            if room <= 0:
                break
            chunk = min(s.remaining_prompt, self.chunk_size, room)
            batch.append((s, chunk, is_prefill=True))
            spent += chunk

        # 3) 有空位就准入新请求，新请求只做第一段 prefill
        while self.waiting and spent < self.budget:
            s = self.waiting[0]
            room = self.budget - spent
            chunk = min(len(s.prompt_ids), self.chunk_size, room)
            if chunk < self.min_prefill_chunk:
                break                   # 碎片太小，等下一轮
            self.waiting.popleft()
            s.in_prefill = True
            batch.append((s, chunk, is_prefill=True))
            spent += chunk

        # 4) 跑 forward：一次混合了 prefill 片段和 decode 步的 batch
        logits = self.model.forward(batch, self.kv)

        # 5) 采样后立刻更新集合：这就是 continuous 的含义
        self.running = []
        for s in batch:
            tok = sample(logits[s])
            s.append(tok)
            if s.finished:
                self.kv.free(s.block_ids)   # 立刻释放，可被下个请求复用
            elif s.in_prefill and s.prompt_done:
                s.in_prefill = False
                self.running.append(s)
            else:
                self.running.append(s)

A real engine has to add preemption, priorities, block table maintenance, prefix lookup, multimodal inputs and more on top of this skeleton, but the semantics are exactly the same: step 5's "free immediately, admit immediately" is the source of all the gains and the source of all the complexity.

Chunked Prefill: Don't Let One Long Prompt Stall the Whole Engine

The Problem Continuous Batching Doesn't Solve

Continuous Batching solved "short requests dragged down by a long one," but it did not solve "long requests dragging down short ones." Consider a 100K-token prompt: even under Continuous Batching, the first prefill still has to compute all 100K tokens inside a single step (the prefill phase has no autoregressive dependency, so the whole thing can be computed in parallel). That one step can take anywhere from a few hundred milliseconds to several seconds, and during it no decode request can make progress, so TPOT shows a clear spike.

The consequence is that the engine looks like it has decent overall throughput, yet every individual request comes out in fits and starts. TTFT doesn't necessarily improve either — because new requests have to queue up and wait for that big prefill step to finish. This is head-of-line blocking between prefill and decode.

Chunking, and How to Size the Chunks

Chunked Prefill works by cutting the prefill sequence into fixed-size blocks of tokens, processing one block per step, mixed in with other requests' decode steps. A prefill request itself spans multiple steps to complete; its KV is written in chunks and the first token only comes out on the final chunk.

The tradeoff in block size cuts both ways:

• Chunks too large: compute is more saturated within a single step and total prefill time is shorter, but each step gets longer and TPOT spikes for decode requests become more pronounced.
• Chunks too small: TPOT smooths out, but each step's batch shrinks and GPU memory-access efficiency drops (in memory-bound scenarios, a batch that is too small is fatal), which also hurts prefill throughput itself.

So the typical configuration is a middle value, separating the "per-step token budget" from the "token count of a single prefill chunk": the former is a total budget like max_num_batched_tokens, the latter is the chunk size. vLLM V1 makes chunked prefill on by default and schedules it jointly with Continuous Batching, while SGLang exposes a separate chunked prefill size. Empirically, the defaults are good enough for most workloads — don't rush to change them.

Don't Forget Starvation

The natural side effect of mixed batching is this: a decode request consumes only 1 token of budget per step, while a prefill chunk consumes thousands. If you fill the budget naively on a first-come-first-served basis, one large prefill sequence will hold the entire budget for a long time and leave small requests waiting over and over. Engineering handles this in a few ways — reserve a minimum quota for decode, cap any single request's prefill share of the batch, or route large-document requests to dedicated prefill instances. We'll come back to this when we get to PD disaggregation.

Memory Management: From Contiguous Allocation to Paging

Why Contiguous Allocation Must Waste

Before paging, each sequence's KV Cache was one contiguous chunk of GPU memory, reserved according to the estimated maximum length (most engines key off max_model_len or grow dynamically). That produces two unavoidable forms of waste:

1. Internal fragmentation: space reserved for a 100K context when only 500 tokens are actually generated sits idle forever.
2. External fragmentation: long and short requests alternately allocate and free, leaving gaps in memory that can't satisfy any request. Worse, fragmentation makes cudaMalloc slow or even synchronizing, which is one source of runtime jitter.

The Two Primitives of PagedAttention

PagedAttention manages the KV Cache the way virtual memory is managed: a pool of fixed-size blocks (usually a "page" of 16 tokens), plus a block_table recording the mapping from logical block number to physical block number. The attention kernel uses that table for indirect addressing when reading KV. This design happens to be exactly what Continuous Batching requires — the two are two sides of the same coin.

What it actually buys you is three things:

• Nearly zero waste: the only internal fragmentation is possibly half a block at the end of a sequence. The PagedAttention paper reports memory waste dropping from the twenty-to-thirty-plus percent range to under 4%.
• O(1) allocation and release: allocating means taking a block number off a free list, releasing means returning it. Both are independent of the fragmentation rate, and neither needs a cudaMalloc call.
• Block-level sharing: this is the physical foundation for prefix caching, covered in detail later.

Block Size Is a Parameter You Can't Just Guess

Too large: the last block may be only a few dozen percent full, so internal waste climbs back up on long sequences; the block_table itself also has to live in GPU memory, one block of indices per sequence.

Too small: the block_table gets long, the kernel has to do an indirect address for every block it reads, and memory coalescing breaks down; the effective hit unit for prefix caching also becomes finer-grained. 16 is the widely adopted default today, and some implementations make it configurable.

SGLang's Other Approach: the Radix Tree

SGLang uses RadixAttention to organize the KV Cache as a radix tree rather than a hash-based block pool. It's logically equivalent — nodes are shared by longest common prefix — but the tree organization reuses more naturally for workloads like multi-turn conversation, tree search, and parallel sampling, the "fork then merge back" shape, and it can run precise LRU eviction directly along tree paths. vLLM takes the block-hash-table-plus-LRU-eviction route. The two paths converge on the same idea: making a "shared prefix" into a physical fact that a data structure can express directly.

Do Your Own Memory Accounting

Before tuning, run the budget once using the KV formula above — that's far more reliable than blind knob-turning:

# 估算单卡能塞下多少并发
# 注意：这是粗算，实际还要给激活值、采样器、CUDA Graph、通信缓冲留空间

model_layers        = 61
kv_heads             = 48      # MLA 模型要另算，压缩潜变量小得多
head_dim             = 128
kv_bytes_per_elem   = 2       # fp16=2, fp8=1, int8=1
gpu_total_gb         = 80
weights_gb           = 14.2   # 14B FP16 权重大约值
reserved_ratio       = 0.03   # 分配器与显存碎片余量

kv_bytes_per_token = 2 * model_layers * kv_heads * head_dim * kv_bytes_per_elem
# 8K 上下文单条序列的 KV 占用
kv_gb_per_seq_8k   = kv_bytes_per_token * 8192 / (1024 ** 3)
usable_gb          = gpu_total_gb * (1 - reserved_ratio) - weights_gb
max_seqs_at_8k      = usable_gb // kv_gb_per_seq_8k

print(f"KV/token      = {kv_bytes_per_token/1024:.1f} KB")
print(f"单序列 8K KV   = {kv_gb_per_seq_8k:.2f} GB")
print(f"8K 下最大并发  = {int(max_seqs_at_8k)}")
print(f"若 KV 换 fp8,并发约 x{int(2 * 0.5)} 级别提升(视实际占用)")

The value of this calculation is that when you find concurrency won't go up, you can immediately tell whether it's "not enough memory" or "the scheduler isn't feeding it," instead of flailing at parameters. Models with compressed attention like MLA are popular in long-context scenarios precisely because they cut the kv_heads × head_dim term of this formula down dramatically, bringing KV footprint to roughly a tenth of traditional MHA.

Prefix Cache: Hit Rate Is Engineering, Not Just a Switch You Flip

The Principle: Block-Level Hashing

The unit of prefix cache matching is the block. The scheduler computes a hash for every KV block, where the input includes that block's token sequence and the hash of its parent block, so a single block's hash implicitly fingerprints everything before it. At lookup time you walk the tokens block by block, hashing as you go, and every block you match is a block whose KV you reuse. Including the parent hash in the computation guarantees chain correctness — you can never get a case where the middle content changed but the surrounding hashes still collide.

The consequence is that the hit length is naturally rounded down to a multiple of the block size. If a block is 16 tokens and your shared prefix is 100 tokens, you can only reuse 96. This is a small but real opportunity cost, and in long multi-turn conversations the prefix hit rate declines slightly because of that rounding.

Where Hit Rate Comes From

Hit rates in production vary enormously depending on the shape of your traffic. The common high-value sources are:

• A stable system prompt. A system prompt of a few thousand tokens repeats across every request. This is the single largest source of hits and the most direct payoff — it turns a few thousand tokens of prefill into zero computation.
• Few-shot examples. A few-shot template is naturally a shared prefix.
• Multi-turn conversations. The input of turn N contains the complete history of turns 1 through N−1, so in theory nearly all of it can be reused. The precondition is that the history hasn't been rewritten or summarization-compressed — the moment upstream summarizes, the prefix breaks and the hit rate goes to zero.
• The fixed skeleton of structured generation. Tool calls and function-definition schemas have a very high degree of prefix repetition.

Failure Modes: Hit Rate Goes Up, Latency Gets Worse

This is the class of problem that's easiest to hit in production and least often written down. Prefix cache has four traps, pointing in four different directions:

1. Volatile values mixed into the prefix. Inserting timestamps, request IDs, user IDs or random nonces into the system prompt is the number-one cause of a zero hit rate — and the person making the change usually isn't even on the team that owns the inference service. The correct practice is to move every volatile value after the user message — keep the prefix pure and only then does the hit rate hold up.
2. A 100% hit rate with limited benefit. Prefix cache only saves prefill, not decode. If your workload is "long prompt + short output," TTFT benefits enormously while TPOT barely moves; conversely for "short prompt + very long output" (say, long code generation), the little time the prefix cache saves is completely drowned out by hundreds of decode steps. Judge the workload shape first, then decide whether hit-rate engineering is worth the investment.
3. Caching crowds out concurrency. Shared blocks occupy memory too. A high hit rate means more requests sharing the same blocks, so the total number of independent sequences can actually go down and throughput need not rise. This one is counter-intuitive but real — it's why "hit rate" and "throughput" must be looked at separately.
4. Eviction policy biting back. Under LRU eviction, a multi-turn request that is actively streaming can get its cached blocks evicted by another request, forcing a recompute or an interruption. In long-conversation scenarios, track eviction- and hit-rate-related metrics per workload rather than only looking at global numbers.

Observability: Watch at Least These Four Numbers

Prefix cache has to be observable, or tuning is flying blind. vLLM and SGLang both expose relevant metrics, in slightly different shapes:

# vLLM V1 常见前缀/调度相关指标（以实际 /metrics 输出为准）
vllm:prefix_cache_queries_total          # 块级查询总次数
vllm:prefix_cache_hits_total             # 命中次数 -> 两者相除得块级命中率
vllm:gpu_prefix_cache_hit_rate          # 引擎直接上报的命中率
vllm:num_requests_waiting                # 排队长度：前缀缓存省时间的主要受益者
vllm:num_requests_running                # 在跑并发
vllm:num_preemptions_total               # 抢占次数：缓存挤占导致抢占时这里会飙升
vllm:gpu_cache_usage_perc                # KV 块池水位

# PromQL 组合出可用的判断
# 命中率趋势
rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m])

# 等待队列是否积压（配合命中率一起看）
vllm:num_requests_waiting

# 抢占风暴告警
rate(vllm:num_preemptions_total[1m]) > 0.5

On the SGLang side the corresponding metrics are running/waiting request counts, cache hit rate, generation throughput, queueing latency, and the accepted length for speculative decoding. Metric names change across versions, so pull /metrics and verify before using them — don't copy-paste from a blog post.

The Qualitative Relationship Between Hit Rate and Benefit

Scenario	Typical hit rate	Primary benefit	Risk
Long stable system prompt + short output	Very high	TTFT drops sharply	After block sharing, the independent concurrency ceiling is bounded by total memory
Multi-turn conversation, history never summarized	Rises with turn count	Turn N is nearly prefill-free	Recompute after eviction in long sessions; needs a sufficiently large block pool
Fixed few-shot template	High	The template portion is prefill-free	Benefit diluted by the example length's share of the prompt
Per-request independent long-document RAG	Low	Only the common instruction portion hits	At low hit rates the overhead is mostly block table maintenance, which is acceptable
Short prompt + very long output	Not applicable	Almost none	Don't invest in hit-rate engineering for this

Batch Size: Two Inflection Points on One Curve

Batch size is the only parameter that shows up in both the throughput formula and the latency formula, which is why it deserves its own section. Intuitively "bigger batch is faster" only holds over the first half of the curve — further out, the curve turns over twice in a row.

The First Half: Why Bigger Batches Always Raise Throughput

As noted earlier, decoding is memory-bound: generating one token requires reading the entire model's weights from HBM. When a batch contains only one sequence, that weight read serves a single token and is pure bandwidth waste. With N sequences in a batch, the same weight read serves N tokens, and the weight-read cost per token is amortized down to 1/N. That is the entire reason "raising batch size always raises throughput" — its mathematical basis is weight reuse, not compute utilization.

This also explains why in the low-concurrency range throughput grows almost linearly with batch size, while the "utilization" number on your GPU dashboard can stay ugly the whole time. Low utilization is precisely the evidence that there's still headroom.

First Inflection Point: Memory-Bound Becomes Compute-Bound

Once the token count within a batch is large enough to push the arithmetic intensity of the matrix multiplications past the hardware's roofline dividing line, per-step time stops falling and throughput stops growing. Where this inflection point sits depends on the ratio of two quantities: the model's weight share of memory, and the number of tokens actually computed per step.

In practice there are two indirect signals that reveal it: the per-step time curve starts to flatten, and throughput stops changing after you raise max_num_seqs. When both appear, increasing batch size further just trades GPU memory for queueing time, with no gain.

Second Inflection Point: TPOT Goes Out of Bounds

Past that inflection point, further increases in batch size translate directly into higher TPOT, and none of that increase is compensated for — throughput no longer rises, yet users genuinely get slower. Once an SLO is fixed, this inflection point is a hard boundary, and it arrives before the performance inflection point and is not negotiable.

So the right way to work is: first derive the upper bound on max_num_seqs backwards from TPOT P99 and treat that value as a constraint; only then chase throughput underneath the constraint. The reverse order — chase throughput first and look at latency afterwards — almost always leads to rework, because throughput improvements are gradual and look beautiful, while latency degradation is often still invisible at P50.

Batch Size Is Not a Single Number

It's also worth realizing that the "effective batch size" of the same engine changes every day: 128 sequences in the batch at daytime peak, and only 3 in the small hours of the morning. That means parameters you tuned for peak will leave the GPU idle most of the time during off-peak, while TTFT doesn't noticeably degrade (because there was no queueing to begin with).

There are two ways to respond. One is to accept it and cover the trough with the cheapest possible resources, wasting money at night rather than during the day. The other is to be elastic — switching between batch size configurations by time of day or by queue water level — but that path is highly complex to implement, and most teams end up spinning up one extra small instance as a backstop instead.

An easily overlooked corollary is that off-peak hours are when speculative decoding pays the most. As computed earlier, in the memory-bound regime idle compute can absorb the draft's extra work, and off-peak is exactly when you're closest to memory-bound. If your speculative decoding shows no benefit under heavy load but real benefit under light load, that means your bottleneck diagnosis was correct, and you could consider enabling it only during off-peak hours.

Tiering by Request Shape: One Configuration Can't Serve Every Business

Many teams go live with a single model instance and a single configuration, then route all business traffic through it. Run it for a while and P99 starts degrading in a way that's very hard to explain, while the averages all look normal. The reason is almost always the same: requests of different shapes interfering with each other inside the same scheduler.

Five Typical Shapes

By the ratio of input tokens to output tokens, production traffic roughly splits into a handful of classes with very different optimal configurations:

1. Long input, short output: document summarization, long-document understanding, RAG Q&A. The bottleneck is prefill and TTFT dominates latency. Prefix cache and chunked prefill yield the most, while speculative decoding is essentially useless (there's no decode phase long enough to benefit).
2. Short input, long output: long code generation, chain-of-thought reasoning. The bottleneck is decode and TPOT dominates the experience. Prefix cache is nearly useless, speculative decoding yields the most, and chunked prefill can be deprioritized.
3. Short input, short output: classification, intent detection, information extraction, single-turn chat. Overall latency is small and it's extremely sensitive to queueing. Batch size has to be kept low, otherwise single-request latency becomes unacceptable; throughput doesn't matter much here.
4. Very long context, multi-turn: long-conversation assistants. KV footprint grows linearly with turn count, putting the most pressure on the block pool and making preemption easiest to trigger. The SLO for this class needs to be defined separately and can't be averaged in with the first three.
5. Structured and parallel sampling: cases needing JSON schema constraints, needing n>1 samples from one prompt, or doing beam search. Batch shapes here are extremely irregular, and constrained decoding adds another pass of grammar constraint, so scheduling overhead is significantly higher than for free-form generation.

The Approach and Cost of Tiered Routing

The direct fix is to split deployments by shape into different specifications: short conversations get a small-batch, latency-first instance; long-document understanding gets an instance tuned for chunked prefill and memory; long code generation gets an instance with speculative decoding on and large batches. The frontend routes to them based on request characteristics.

The cost of this approach has to be calculated up front, in three main parts: first, memory fragmentation and scheduling complexity — multiple sets of instances mean each one reserves resources for its own peak, so utilization is naturally lower than a single large pool; second, load imbalance — shape classification can be wrong, or traffic itself fluctuates over time, leaving one instance idle while another queues; third, operational complexity — scaling, health checks and version management all have to be done multiple times.

My judgment is that splitting only pays off when P99 is genuinely being dragged down by one specific shape and that shape's share is high enough (usually over thirty percent). For a shape with a small share, the idle cost of a dedicated instance eats the entire benefit. When in doubt, first get chunked prefill and prefix cache right on a single instance — the payoff of those two is positive across all shapes, and they're the lowest risk.

The Special Scheduling Overhead of Structured Output

The fifth shape deserves its own mention, because for many teams it's a problem that only surfaces late. When you use guided decoding (sampling constrained by JSON schema or grammar), sampling is no longer a simple multinomial draw: from the candidate tokens of the previous step, the engine has to maintain a "set of states allowed by the grammar" and progressively eliminate illegal tokens. That process involves sequence-level state advancement and extra kernel scheduling, and under high concurrency it visibly lengthens per-step time.

The consequence is that when structured and free-form requests run together, the former slows down per-step time for the entire batch, showing up as both sides getting slower at once. The way around it is to fuse the grammar state machine's advancement into the sampling kernel, or to route these requests to a dedicated instance. The way to observe it is to check whether TPOT shows a regular uptick during periods when structured traffic has a high share.

One more small but useful fact: the actual accepted width for constrained decoding is usually far smaller than the model's full vocabulary, and if the engine doesn't do candidate pruning, every step is doing full sampling-distribution normalization over dozens of tokens — that's pure waste. Verifying whether the engine you're using has candidate pruning enabled is the highest-leverage check in this area.

Prompt Structure and Cache Friendliness

So far I've treated prefix cache as an engine feature, but its real effect is almost entirely determined by how you organize the prompt. This section looks at it from the prompt side, because that's the highest-return work available: you don't have to touch a single engine parameter, yet the payoff can exceed the sum of all parameter tuning.

Layer by Rate of Change, Not by Length

The most common mistake when organizing a prompt is to sort by "importance" or "length," putting the most important system identity statement at the front and business rules at the end. That arrangement reads most naturally, yet it's exactly what conflicts with how caching works — because the beginning determines the hashed prefix of the entire chain, so any change at the front invalidates everything after it.

The correct layering principle is to order from lowest to highest rate of change: identity and rules that never change go first; parts that vary by tenant or product line go in the middle; user input, retrieval results and time information, which change on every request, go last. The principle looks simple, yet it's frequently violated in practice, because whoever adds a new business rule instinctively inserts it at the beginning of the system prompt, with no awareness that this one insertion punches a hole through the entire cache.

There's a simple test for whether a piece of content belongs in the prefix: pull the prefix portion of real requests from the past hour, deduplicate it, and count how many distinct combinations there are. If a section that should be shared has dozens of variants, it has volatile values mixed in and you need to track down which field is responsible. This is far more useful than looking at the overall hit rate, because it localizes the problem to the field level.

How Multi-Turn History Is Stored

Multi-turn conversation is where prefix caching pays the most, but it's also where it's easiest to break quietly. The key question is whether the history is replayed verbatim. If every turn concatenates the complete history back unchanged, then in theory everything except the newly added content can hit. But if upstream does any rewriting at all — summarization, truncation, reordering, rewording role tags, appending timestamps — the prefix is invalidated from the rewrite point onward.

There are a few common ways this breaks that are worth calling out by name: normalizing whitespace or line endings when re-serializing history messages; automatic summarization kicking in after some number of turns; inserting filter explanations into the history for safety policy; and, in multimodal scenarios, replaying images converted to text descriptions. These rewrites may have a negligible effect on model quality, but they're catastrophic for cache hit rate.

There are two ways to handle it. One is to do prefix matching on the engine side, meaning not assuming the prefix always matches contiguously, but instead searching backward block by block for the longest matchable portion, so that even if one spot in the middle was rewritten, the part after it is preserved. The other is to impose stability constraints on the application side, agreeing that history may only be appended to and never modified, and explicitly marking summarized results as a new "compression injection," so that hit-rate drops become predictable events rather than silent degradation. The former is more thorough; the latter costs less.

Branching and Parallel Sampling

Scenarios that need multiple candidate results in one generation — sampling multiple copies, beam search, an agent trying several paths on the same prompt — have more complicated cache reuse than intuition suggests. If the implementation "copies them into multiple independent requests," each one re-prefills in full and the cache is completely useless. If the implementation instead "prefills once and has all branches share the same prefix cache," the common portion across branches is computed only once, which is an enormous difference.

The way to judge is direct: after doubling the number of samples, do TTFT and prefill compute double or stay roughly flat? If they double, the branches aren't sharing a prefix, and that's the first thing worth fixing. Once fixed, branch sampling is nearly the only scenario where "request volume doubles but compute stays flat" — an order-of-magnitude win for multi-candidate businesses.

Reusing Tool Definitions and Schema Constraints

In tool-calling scenarios, the tool names, parameter schemas and return formats are highly repetitive and naturally belong in a shared prefix. Many implementations inject tool definitions at the end of the prompt, while the tool list grows incrementally over a conversation — new tools keep getting appended, which shifts the position of everything after it and constantly invalidates the cache. A steadier approach is to keep tool definitions in a stable position: tools that have already been loaded don't move, and new tools are appended after a fixed position.

Schema definitions for structured output have the same problem. If the position of the schema definition within the prompt is inconsistent across requests, or the order varies with content, you get invalidation. Pinning schema definitions to a fixed position in the prefix is an optimization that costs essentially nothing.

Put Prompts Under Version Management

Prompts have become an engineering asset, yet many teams still scatter them across code, configuration consoles and people's local files. That brings two direct risks: nobody can say which version is running in production right now, and when a prompt changes, you can't tell whether the problem was the prompt or the engine.

The pragmatic minimum is three things: prompts have an explicit version identifier that's recorded in the log alongside each request; changes go through review and note whether they affect prefix structure; and you can roll back to the previous version within minutes. Once you have those three, changes in prefix hit rate can be attributed correctly — a sudden drop is either caused by a prompt revision (confirmable via the version number) or it isn't (and then it's an engine-side problem). Without this capability, hit rate becomes a metric nobody can explain, and a core metric nobody can explain is, for practical purposes, no metric at all when something goes wrong.

Speculative Decoding: Acceptance Rate Decides Everything

Why It's Mathematically Lossless

The logic of speculative decoding is this: use a cheap small model (the draft) to serially guess the next γ+1 tokens, then have the large model (the target) verify all γ+1 positions in a single parallel forward pass. Verification compares position by position in order: if the i-th token matches the draft, accept it and move to the next; on the first mismatch, discard all guesses after it and resample from that position using the target's own distribution.

The key property is that the output distribution is exactly identical to using the target alone — this isn't approximate acceleration, it's lossless acceleration. The draft merely proposes candidates; the final result is determined by the target's distribution. So it can be used in production with confidence, unlike quantization, which requires an additional quality regression suite.

The Speedup Is a Number You Can Compute by Hand

Let α be the acceptance probability at each position (assuming independence here, though in reality there is correlation). For a draft of length γ, the expected number of accepted tokens is a geometric series sum:

Expected accepted length E[γ] = (1 − α^(γ+1)) / (1 − α)

Plug in a few values and α's dominance becomes obvious (at γ=3): α=0.9 gives about 3.44, α=0.7 about 2.92, α=0.5 about 1.88, α=0.3 about 1.42. Dropping acceptance from 0.9 to 0.5 roughly halves the expected length.

The per-step cost is one target forward pass plus γ small steps of the draft. So the real speedup is expected accepted length ÷ (1 + γ × draft relative cost). The cheaper the draft is relative to the target, the larger γ can be; the lower α is, the more you have to compensate with "the draft being cheap enough." Real systems are far more complicated than a single number: at high temperature α drops rapidly, and after long chains of reasoning the distribution drift between draft and target also depresses α — so measure the accepted length in practice rather than applying the formula.

# 投机解码的收益手算：先决定"值不值得开"，再去调参
def expected_accepted(alpha, gamma):
    """长度 gamma 的草稿，期望被接受的 token 数"""
    if alpha >= 1.0:
        return gamma + 1
    return (1 - alpha ** (gamma + 1)) / (1 - alpha)

def speedup(alpha, gamma, draft_cost=0.05, mem_bandwidth_free=0.5):
    """
    draft_cost   : draft 单步耗时 / target 单步耗时
    mem_bandwidth_free:
        0.0 -> GPU 已经完全算力饱和，多花的计算无法重叠，加速比被完全吃掉
        1.0 -> GPU 处于访存受限状态，draft 的计算几乎免费，加速比接近期望长度
    """
    if alpha <= 0.0:
        return 0.0
    cost = 1.0 + gamma * draft_cost
    gain = expected_accepted(alpha, gamma)
    # 空闲带宽能吸收一部分 draft 计算成本
    return 1.0 + (gain - 1.0) * mem_bandwidth_free / cost

print("gamma  alpha  期望长度  访存受限  算力饱和")
for alpha in (0.9, 0.8, 0.7, 0.5):
    for gamma in (1, 3, 5):
        e = expected_accepted(alpha, gamma)
        lo = speedup(alpha, gamma, mem_bandwidth_free=1.0)
        hi = speedup(alpha, gamma, mem_bandwidth_free=0.0)
        print(f"{gamma:>5}  {alpha:>5}  {e:>8.2f}  {lo:>8.2f}x {hi:>8.2f}x")

What this table is meant to show: the same configuration can differ by several times in speedup between a memory-bound state (low concurrency, small batch) and a compute-saturated state (high concurrency, large batch). Under heavy load the GPU has no spare compute to begin with, so adding the draft's work is pure overhead, and speculative decoding may actually hurt throughput. The expectation that "turning on speculative decoding makes things faster" therefore doesn't hold — it's coupled to batch size.

Choosing a Draft Model

There are roughly four approaches, each with its own applicability boundary:

• A standalone small model (the n-gram / prompt lookup / REST line): introduces no extra model at all, relying purely on textual self-repetition — treating token fragments that appeared in the prompt or recent output as the draft. Zero memory, zero weight overhead, well suited to highly repetitive workloads like summarization and rewriting, document Q&A, and code completion; it's nearly useless for creative generation. The upside is capped, but it wins on simplicity and reliability, with no training required.
• A small model from the same family: uses a small model with the same tokenizer and the same series as the draft. Implementation is simple, α depends on how close the two models' distributions are, and a low-quality near-neighbor model actually drags α down.
• Feature-/multi-head self-speculation (the EAGLE, Medusa line): doesn't load a separate full model, but adds lightweight "next feature prediction" heads or multiple decoding heads on top of the main model, reusing the main model's hidden states to predict future tokens. α is markedly higher, and this is the mainstream choice today.
• MTP self-speculation: if the base model was itself trained with multi-token prediction heads (MTP), those heads are naturally the best-fitting drafts — they share representations with the main model, add almost no memory, and have the best distribution consistency. This is a "free lunch" at the architectural level, but it only holds for models that natively include MTP.

The Three Numbers You Must Measure

Don't decide on a whiteboard whether to adopt speculative decoding — measure it directly: accepted length, TPOT before and after, and overall throughput before and after. All three are required; looking only at TPOT misses the trap where "a single request got faster but overall throughput dropped." A typical way to observe:

# 投机解码相关观测（参数名以所用版本文档为准）
# vLLM 侧
#   --speculative-config '{"method":"ngram","n":5,"prompt_lookup_min":2,"num_speculative_tokens":5}'
#   method 也可为 "eagle" / "eagle3" / "medusa"，需配套加载 draft 权重
# SGLang 侧
#   --speculative-algorithm EAGLE3
#   --speculative-draft-model-path /path/to/draft
#   --speculative-num-steps 3        # 草稿步数 gamma
#   --speculative-eagle-topk 4       # 候选分支宽度

# 上线前必须同时记录的三个数
#   1) 实际接受长度 spec_accept_length 分布（不是均值，要看 P10）
#   2) TPOT P50/P99
#   3) 集群总吞吐 tokens/s
# 判定规则：若总吞吐下降，无论 TPOT 改善多少，都不开启。

There's one more easily overlooked boundary: for multimodal requests and very long contexts, constructing the draft's input and organizing its cache is more complex, and the real benefit needs separate validation — don't extrapolate conclusions from pure text.

Multi-Tenant Scheduling: Preemption, Fairness and QoS

When Memory Runs Short, You Can Only Preempt

Continuous Batching has no static reservation; KV blocks are allocated dynamically. When a newly arriving request needs a large contiguous block of KV and the pool can't supply it, the engine has only three options: reject (return 503 or queue, depending on implementation), swap out the victim's KV blocks to host memory, or recompute by discarding them and starting over from scratch.

The cost of recomputation is catastrophic: a 20K-context session that gets preempted has to pay for 20K tokens of prefill all over again. So mainstream engines pick victims primarily by "least recently accessed" or "highest recompute cost," deliberately avoiding long contexts and requests that have already generated a lot of content. Swap is cheap but introduces a PCIe transfer on the order of several GB, which is much better when NVLink host interconnect is available.

Choosing a Queueing Policy

First-come-first-served (FCFS) is simple to implement and has predictable latency, but it has a counter-intuitive property: in memory-constrained scenarios, LIFO (last-in-first-out, i.e. newer requests get priority) can deliver markedly higher cluster throughput. The reason is that new requests usually have shorter contexts than old requests that are already half-done, so they occupy less KV and more concurrency fits per unit of memory; related work in vLLM has reported this gap reaching several-fold in memory-wall scenarios.

But LIFO is unacceptable for interactive services — old requests could be starved indefinitely. The practical compromise is: default to FCFS for fairness, and switch to an LIFO-style shortest-job-first policy when memory pressure or frequent preemption is detected, or use "shortest remaining length / time first." This isn't a purely theoretical choice; it directly determines your cluster's real capacity under stress.

Fairness: The Real Multi-Tenant Problem

When one cluster serves multiple business owners, "fairness" has to be broken into three layers:

• Admission fairness: can one tenant fill the batch? The answer is yes — as long as it has enough requests, under FCFS every other tenant that arrives later queues indefinitely. The countermeasure is per-tenant token bucket rate limiting, or giving each tenant a batch slot cap in the scheduler.
• Queueing fairness: when long and short requests are mixed, the long ones wait longer. The higher the concurrency within a batch, the larger the variance in waiting time. For interactive business you'll usually want to cap max_num_seqs — better a bit less throughput than a P99 queue that explodes.
• Preemption fairness: the most dangerous gap in tenant isolation — a tenant A's long request being repeatedly preempted amounts to making A repay prefill over and over. Monitor preemption counts broken down by tenant.

The practical approach on the QoS side is tiered: give workloads with different SLAs different priorities and separate concurrency quotas, have the scheduler dequeue by priority, and use FCFS within a priority level. Combined with admission-side rate limiting, this is far more stable than complex fair-sharing algorithms inside the scheduler, and much easier to explain and debug.

PD Disaggregation: Removing the Root Cause of Mixed Batching

Chunked prefill mitigates head-of-line blocking within the "mixed batch" framework, while PD disaggregation (prefill/decode disaggregation) splits it outright: prefill instances and decode instances use different deployments and different batching strategies, each optimized on its own, and then KV is transferred between them over the network.

The benefits are very concrete: prefill is compute-bound and the smaller the batch the faster it goes; decode is memory-bound and needs large batches to amortize weight reads. Mixed together they compromise each other and neither reaches its optimum. Once separated, both sides can be tuned independently, and you can even use different classes of hardware (compute-heavy machines for prefill, bandwidth-heavy ones for decode).

The cost is KV transfer. The KV of a long context runs from hundreds of MB to several GB, and cross-machine transfer easily becomes the new bottleneck — it has to go through a direct path like RDMA, which is also why KV transfer libraries (things like NIXL) exist. The empirical standard for whether PD disaggregation is worth it: the benefit is largest when long prompts (thousands of tokens or more) dominate and TPOT jitter is obvious; on short-prompt, short-output workloads, the transfer overhead often eats the entire gain.

The Time Composition Inside a Single Scheduling Step

All of the analysis above rests on one quantity — "per-step time" — yet very few people actually look at what's inside one scheduling step. Understanding this is what lets you explain why the same model with the same configuration can show more than a 2× performance gap between implementations.

The Six Stages of One Step

From the moment the scheduler decides what runs this round to the moment the result reaches the user, there are roughly six segments:

1. State preparation: turning the current lengths, block tables and position indices of this batch of sequences into index tensors the device can consume directly. This step isn't free under dynamic batching — the set of sequences changes every round, so the indices have to be recomputed.
2. Forward compute: attention plus the feed-forward network. This is the only part that actually does mathematics, and the only part that can be massively parallelized on the GPU.
3. Sampling: drawing one token from the output distribution. This includes probability normalization, random number generation, grammar filtering when constrained decoding is on, and batched handling in multi-sequence scenarios.
4. Communication and synchronization: tensor synchronization between devices, and data movement between device and host.
5. State update: writing new tokens back into the sequences, appending block references, releasing finished sequences, and updating prefix cache hits and evictions.
6. Result delivery: sending newly generated tokens back to the requester.

The key insight is that only the second segment is necessary cost; the other five are scheduling overhead. When the forward pass accounts for only half of per-step time, the other half is the bill for scheduling. The smaller the model and the shorter the sequences in the batch, the higher the overhead share — which is also why differences in engine implementation get amplified on small models.

Synchronization Points Are the Most Expensive Part

An easily underestimated killer is host-to-device synchronization waits. Any call that has to wait for the GPU to finish before it can continue — reading back results, fetching state, making a branch decision — forcibly drains the GPU's compute pipeline. When the batch has few sequences, the pipeline isn't full to begin with; interrupt it with synchronization and the GPU's actual utilization drops to a very low level.

This also explains why the intuition that "more sequences in a batch is always better" holds over some range: with many sequences, the GPU has enough work to hide scheduling overhead; with few, the fixed cost of one synchronization is amortized over very few tokens and the per-unit cost spikes sharply. The first inflection point of the batch size curve is, in essence, exactly the point where scheduling overhead is fully amortized.

Measuring It Yourself

Don't guess. Most inference frameworks provide per-stage timing or profiling, and the typical approach is to split one step into segments, timestamp each, and focus on three things: what fraction of per-step time the forward pass takes, the absolute cost of sampling plus state update, and whether unexpected synchronization waits exist.

If the forward share is clearly too low, suspect first that the batch has too few sequences or that synchronization points exist; if the sampling share is abnormally high, check whether the grammar state machine in constrained decoding is skipping candidate pruning; if the state update share is high, it's likely the cost of block table maintenance or prefix lookup — confirm whether the relevant implementation sits on the critical path.

This method has one extra benefit: it turns "why is this engine slow" from a mystical question into one you can localize to a specific code segment. Every tuning recommendation should map to a measurable stage — otherwise it's just someone else's magic number.

From One Machine to Many: The Cost of Parallelism

Everything above has assumed "one model on one set of devices." In real production the model often doesn't fit and has to be sharded across multiple GPUs or even multiple machines, and parallelism itself carries a communication cost. The conclusions in this section may run counter to a lot of people's intuition.

Three Ways to Shard

• Tensor parallelism: splitting weight matrices by rows or columns so each GPU computes only part of it, with an all-reduce or all-gather inside the layer before moving on. Communication is the most frequent, on the order of a few times per layer, each transferring activations. It scales only when there's high-speed interconnect between devices.
• Pipeline parallelism: assigning different layers to different devices, where one device's output feeds directly into the next. Communication volume is small (only activations at layer boundaries), but pipeline bubbles exist: every device does the same kind of work, so efficiency is bounded by the slowest GPU.
• Expert parallelism: for mixture-of-experts models, assigning different experts to different devices, where each token routes to only a few experts. Communication is sparse, once per layer, and sits between the other two in magnitude.

In real deployments it's usually a combination: tensor parallelism when you have few devices, plus pipeline parallelism layered on when you have many. But the choice of combination directly determines the performance ceiling, and picking wrong costs far more than tuning parameters.

Roughly Estimating Communication Volume

Tensor parallelism communication grows linearly with "hidden dimension × tokens in the sequence"; multiplied by the number of layers, the total is substantial. A useful way to judge it: each layer's all-reduce has to transmit at least one activation of hidden-dimension width, and the hidden-dimension activation tensor is not small under long context. As context length grows from a few thousand to tens of thousands, communication volume grows with it — and if the interconnect bandwidth is going over a general-purpose peripheral path rather than a direct GPU-to-GPU link, the gap widens to the point where it's impossible to ignore.

So there's a practical rule: prefer to do parallel sharding within a single physical machine, and prefer machines with high-speed direct GPU interconnect. Only shard across machines when a single machine can't hold the model, and accept a clear throughput penalty when you do. This should be decided during procurement and deployment planning, not left until a benchmark run shows something is wrong and forces rework.

More Parallelism Is Not Always Better

Tensor parallelism has a counter-intuitive property: as parallelism degree increases, matrices per GPU get smaller, arithmetic intensity drops, and the memory-bound problem actually gets worse. A batch that ran efficiently on a single GPU leaves every GPU underfed after sharding, and you still pay for communication on top. All synchronization points must also wait for the slowest GPU, so overall latency is determined by the slowest device — a clear barrel effect.

That's why "use more GPUs for the same model" often makes single-request latency worse while only modestly raising total throughput. The exact sweet spot depends on the model's hidden dimension, context length, interconnect bandwidth and sequences per batch, and there's no universal answer — the only reliable method is to step through sharding granularities and measure, recording both single-request latency and total throughput at each step. Empirically, speedup is close to linear until a matrix is split past a certain size; past that it decays noticeably.

Multiple Instances or Single-Machine Sharding

An earlier decision is this: do you buy more small machines each running a full copy of the model, or buy big machines and shard the model. The former is the data parallelism idea: naturally isolated, scales linearly, no communication, no cross-machine reliability issues. The latter is the model parallelism idea: optimal latency, but communication and failure domains are both concentrated.

For most online services that optimize for throughput, the answer leans toward multiple instances: multiple full replicas let the load balancer split traffic by real-time load, an instance that has problems can be pulled out of rotation with the rest unaffected, and there's no cross-machine communication. The cost is that single-request absolute latency doesn't improve just because you have more devices. When do you actually need model sharding? Only when a single replica's memory can't hold the model together with enough concurrency — that's a hard constraint, not a performance choice.

The extra problem the multi-instance approach has to solve is request routing quality. Routing by shortest queue is the most common approach, but requests vary enormously in shape (recall the short conversations versus long-document summaries from earlier), and routing purely by queue length piles all the long requests onto one instance. The improvement is to factor in a length estimate when routing, or to route by shape into pools of differently-specified instances. Tail latency caused by bad routing shows up on monitoring as severely uneven load across instances — a very useful diagnostic signal.

Non-Inference Overhead on the Request Path

All of the analysis above starts counting from "the request enters the scheduler," but the time the user waits also includes the part before and after. These parts are routinely ignored as a whole in the decomposition of TTFT, yet on short requests they can account for more than half the total.

The Inbound Side: Tokenization and Admission

A request coming in through the gateway has to pass authentication, rate limiting, format validation, prompt assembly and tokenization before it reaches the engine. All of these are fixed per-request costs, and they're very significant on short requests. The more insidious point is that tokenization time itself grows linearly with input length, and many tokenizer implementations are pure character scans rather than matrix-parallel operations, so they noticeably slow things down on very long inputs.

If the TTFT decomposition shows "only a third inside the engine," then the other two thirds are almost certainly here. The available optimizations, ranked by return: push work that can be decided at the gateway as early as possible (rate limiting, auth and format validation should avoid blocking subsequent steps), parallelize tokenization (split long inputs at byte boundaries and process in parallel, taking care that splitting doesn't break multi-byte characters or multi-token merges), and avoid tokenizing twice (some implementations tokenize once during prompt assembly and again at the engine entry point).

The Outbound Side: Detokenization and Streaming Push

The overhead on the generation side is equally underestimated. The engine internally gets token IDs, and turning them back into text requires a decode; for streaming output you additionally decode on every token and handle the problem of characters spanning token boundaries — a single character can be split across two tokens and can't be concatenated directly, or you get mojibake. Streaming push also has to handle backpressure: when the client consumes slowly, the sending side either buffers or blocks, and both choices occupy server resources.

There's one frequently ignored source of latency here: if the framework does micro-batching to reduce the number of pushes, the arrival of an individual token is artificially delayed. That micro-batch window is usually on the order of a few milliseconds, which isn't obvious at a decode rate of tens of tokens per second, but it stacks up into TTFT and is visible when the client renders character by character.

Network Round Trips Counted More Than Once

Streaming services have a phenomenon that's easily misdiagnosed: the client's measured "TTFT" includes a full network round trip, and the number of round trips is often more than one. Health checks, retries, proxy-layer buffering, load balancer connection reuse policy, and whether protocol compression is enabled all affect the real round trip count. If any layer in the middle buffers responses, the streaming effect is completely erased — the user gets everything at once rather than token by token.

The way to debug this is to observe the timestamp sequence of token arrivals directly from the client side rather than looking at server logs. Send times in server logs can't reflect client receive times, and the difference between the two is the extra latency on the path. A benchmark script that uses an aggregating client library instead of actually reading each timestamp systematically underestimates this overhead — which is precisely why the benchmark skeleton earlier insists on recording the arrival time of every token.

Measuring the Fixed Cost

Split TTFT into "observed by the client" and "recorded inside the engine" and compare; the difference is the path overhead. The precondition is that both timestamps come from the same monotonic clock source — if server and client clocks aren't synchronized, the difference isn't trustworthy. The safe approach is to record arrival time on the client and send time on the server, report both, and align them via logs, so you don't depend on cross-machine clock accuracy.

The standard for judging path overhead is this: is it a relatively stable fixed cost across requests, or does it rise with concurrency? The fixed part can only be optimized by removing stages; the part that rises with concurrency indicates a connection count bottleneck, proxy-layer queueing or send-side backpressure, and those two classes need completely different remedies. The former means changing architecture; the latter means tuning capacity.

Why This Deserves Its Own Section

Because it determines a very common misdiagnosis: mistaking path overhead for insufficient engine performance, then spending weeks tuning engine parameters. If a third of TTFT on short requests comes from the path, then doubling engine throughput barely moves the user's experience, whereas cutting the micro-batch window from ten milliseconds to two milliseconds shows up immediately.

The way to catch this kind of misdiagnosis is to do a path profile early: in a benchmark environment, bypass the gateway and connect straight to the engine, bypass streaming and return everything at once, then add the stages back one at a time. Measure after each layer you add back and watch how TTFT changes. Wherever the slope is steepest is where the benefit comes from — that's more reliable than any rule of thumb.

The Performance Cost of Sampling and Decoding Parameters

Sampling executes at every step of the decode loop, but whether it's expensive or cheap depends on which features you've enabled. This section is easy to overlook, because most people assume "sampling is just a draw." In reality, under some configurations sampling is the most expensive part after attention.

The Cost of Basic Sampling

The simplest greedy decoding just takes the highest-probability token, at negligible cost. Switching to random sampling requires generating random numbers, computing cumulative probabilities and locating the sampling position — all small per-sequence operations, cheap but not free. Batched sampling can vectorize these operations at an acceptable cost. The real problem is features that require per-token serial processing — the categories below all fall into this group.

Per-Token Penalty Processing

Features like repetition penalty, frequency penalty and presence penalty have to scan every token the sequence has already generated at each step, count occurrences, then adjust the candidate distribution. The overhead is proportional to length already generated, so in long-output scenarios it grows linearly as generation proceeds.

This yields a counter-intuitive conclusion: with the same model and the same batch size, the longer the output, the higher sampling's share of the cost. In scenarios like long code generation, by the time a few thousand tokens have been generated sampling may have gone from negligible to a significant overhead. The check is simple: compare the per-step time curves with penalties enabled and disabled, and see whether the difference grows monotonically with generated length.

Whether these features can be optimized depends on the algorithm. A naive implementation rescans everything each step, which is quadratic in the worst case; maintaining the counts incrementally (update a token's count as soon as it's generated, and only process the candidate set each step) brings the cost down close to linear. Whether an implementation does this is a concrete marker of engine quality.

Candidate Pruning and Constrained Decoding

Constrained decoding has to make output satisfy grammatical or schema constraints, which in implementation means maintaining a "currently allowed token set" state, filtering candidates against it each step, and continuing from the new state. This work is inherently serial and depends on the previous step's result, so it's hard to vectorize fully.

Two effective optimization directions: candidate pruning — narrowing the candidate set to the dozen or so tokens that could actually occur, without violating constraints, instead of making the model normalize probabilities over the entire vocabulary; and state caching — the same constraint state may be shared by multiple requests, so caching avoids repeated state advancement. Without pruning, doing full-vocabulary normalization means handling tens of thousands of tokens every step, and that overhead can be eliminated entirely — it's benefit you get just by checking once.

Random Numbers and Reproducibility

Streaming generation has to guarantee that the same random seed produces reproducible output, which requires the random number generator to have deterministic state management under concurrent requests. There are two common implementation approaches: one generator per request, with independent and reproducible state, but the management overhead and cache locality both get worse when there are many generators; and a single global generator, which performs well but is not reproducible under concurrency.

The production compromise is usually to take the high-performance path by default, and when reproducibility is needed, specify the deterministic generation mode via a request parameter and accept the extra cost of that path. One thing worth checking is how large the performance loss is when the reproducibility switch is turned on. If the loss is significant, it means generator state management has room to optimize — not that reproducibility itself is expensive.

Where All This Overhead Adds Up

Returning to the "time composition of a scheduling step" framework from earlier: sampling's share of step time is a directly measurable quantity. If it exceeds twenty percent of per-step time, it's worth investing effort to optimize; if it's under ten percent, that effort belongs elsewhere. This judgment avoids two common mistakes — editing engine source code to optimize a three-percent cost, or leaving a thirty-percent cost in place while endlessly tuning batch size.

There's also a cross-cutting way to look at it: the performance ranking of the same engine differs under different feature combinations. An engine that leads on free-form generation doesn't necessarily lead on constrained generation, because the bottlenecks sit in compute and sampling respectively. When your business shifts from free-form generation to heavy structured output, that ranking changes and needs re-evaluating. This is a selection trap that's easy to miss long-term — extrapolating one workload's performance to another's.

Why TTFT and TPOT Must Be Looked at Separately

One Metric Can Fool You Twice

Back to the formula from the opening: Total latency = TTFT + (n−1) × TPOT. Two factors multiplied and added means that the same end-to-end latency can be produced by completely different combinations.

Imagine two engines: A has TTFT of 2 seconds and TPOT of 20 milliseconds, outputs 50 tokens, for a total of 3 seconds; B has TTFT of 200 milliseconds and TPOT of 60 milliseconds, also 3 seconds total. Users perceive these completely differently — A is "wait forever, then it flows out smoothly," B is "instant reply but keeps dribbling it out." The perceived quality of a streaming product is almost entirely determined by TPOT, while whether a user abandons the click is determined by TTFT.

Worse, looking at throughput alone fools you too. Raise the batch from 32 to 64 and cluster throughput goes up, TPOT roughly doubles, end-to-end latency is unchanged or worse — but the throughput chart looks beautiful. Throughput and latency have to be reported as a pair, which is also why every serious inference benchmark outputs both sets of numbers.

The Metrics Checklist

Metric	Definition	Primary influencing factors	Typical symptoms when degraded
TTFT P50/P99	Request sent to the first token returned	Queue length, prefill compute, prefix hit rate	Waiting queue backs up, hit rate at 0
TPOT / ITL P50/P99	Interval between adjacent output tokens	Batch size, chunked prefill config, quantized kernels	Large prefills cause periodic spikes
Queueing latency	From entering the engine to starting to be scheduled	Admission rate, concurrency ceiling	Waiting queue keeps growing
Throughput tokens/s	Total output tokens per unit time across the cluster	Concurrency, batch size, quantization	Low utilization but queue isn't long → the scheduler is the bottleneck
Goodput	Share of requests meeting the TTFT/TPOT SLO	All of the above	Closer to real business value than throughput
Preemptions per second	Number of requests evicted for recompute or swap	KV block pool water level, max_model_len	Spikes accompanied by TTFT degradation
Accepted length	Average number of accepted tokens in speculative decoding	Draft quality, workload saturation	A drop means the workload got heavier or the draft is mismatched

Always look at quantiles, not means. P99 degradation in an inference service often appears before the mean changes, and the mean flattens spikes completely. In production, at minimum, set up a group of P50/P90/P99 alerts for both TTFT and TPOT.

A Usable Benchmark Script Skeleton

The input to tuning has to be reproducible benchmark data. The key is using a realistic arrival pattern — fixed-concurrency benchmarks can't measure queueing behavior at all, and queueing is precisely the main source of latency:

# 压测骨架：同时产出 TTFT / TPOT / 吞吐三个视角
# pip install aiohttp numpy  (或直接用 openai SDK)
import asyncio, time, json, statistics, random
import aiohttp

TARGET = "http://127.0.0.1:8000/v1/completions"
MODEL  = "your-model"
PROMPT_TOKENS = 2000      # 覆盖你真实的 prompt 长度分布
OUT_TOKENS    = 300       # 覆盖真实的输出长度分布
CONCURRENCY   = 32
DURATION_S    = 300

def make_prompt(seed):
    # 注意：这里刻意让每个请求的 prompt 不同，以排除 prefix cache 命中
    # 测 prefix cache 时要改成"共享长前缀 + 变化后缀"的形态
    head = f"[req {seed}] " + "context " * (PROMPT_TOKENS // 2)
    return head + " ".join(f"t{seed}_{i}" for i in range(PROMPT_TOKENS // 2))

async def one(session, sem, stats, seed):
    async with sem:
        body = {"model": MODEL, "prompt": make_prompt(seed),
                "max_tokens": OUT_TOKENS, "temperature": 0, "stream": True}
        t0 = time.perf_counter()
        first_t, last_t, n = None, None, 0
        try:
            async with session.post(TARGET, json=body,
                                    timeout=aiohttp.ClientTimeout(total=300)) as r:
                async for raw in r.content:
                    line = raw.decode("utf-8", "ignore").strip()
                    if not line.startswith("data: "):
                        continue
                    if "token" not in line:
                        continue
                    now = time.perf_counter()
                    if first_t is None:
                        first_t = now
                    last_t = now
                    n += 1
        except Exception as e:
            stats["err"] += 1
            return
        if first_t is None:
            stats["err"] += 1
            return
        stats["ttft"].append(first_t - t0)
        if n > 1:
            stats["tpot"].append((last_t - first_t) / (n - 1))
        stats["out_tok"] += n

def pct(xs, p):
    return statistics.quantiles(xs, n=100)[p - 1] if len(xs) > 5 else 0.0

async def main():
    sem = asyncio.Semaphore(CONCURRENCY)
    stats = {"ttft": [], "tpot": [], "out_tok": 0, "err": 0}
    t0 = time.perf_counter()
    conn = aiohttp.TCPConnector(limit=CONCURRENCY * 2)
    async with aiohttp.ClientSession(connector=conn) as session:
        async def worker_loop(wid):
            seed = 0
            while time.perf_counter() - t0 < DURATION_S:
                await one(session, sem, stats, f"{wid}-{seed}")
                seed += 1
        await asyncio.gather(*[worker_loop(i) for i in range(CONCURRENCY)])
    wall = time.perf_counter() - t0
    print(json.dumps({
        "wall_s": round(wall, 1),
        "ttft_p50": round(pct(stats["ttft"], 50), 4),
        "ttft_p99": round(pct(stats["ttft"], 99), 4),
        "tpot_p50": round(pct(stats["tpot"], 50), 5),
        "tpot_p99": round(pct(stats["tpot"], 99), 5),
        "throughput_tok_s": round(stats["out_tok"] / wall, 1),
        "errors": stats["err"],
    }, ensure_ascii=False, indent=2))

asyncio.run(main())

There are three disciplines for using it: first, re-run the baseline — change one parameter and run one round; a comparison without a fresh baseline is meaningless; second, suppress noise — repeat the same configuration at least three times and take the median, and don't run anything else on the benchmark machine at the same time; third, use two sets of prompts for testing prefix cache versus basic prefill — using one set conflates the two things and can never produce a clean conclusion.

Quantization and Scheduling: Two Systems That Constrain Each Other

Weight Quantization and KV Quantization Deliver Entirely Different Gains

Quantizing weights to INT8 / FP8 / INT4 (GPTQ, AWQ and the like) saves the fixed slice of memory the model itself occupies, which converts directly into more KV space and a larger batch — a structural gain. Quantizing KV Cache saves the 2 × L × H_kv × d × bytes term's bytes, which likewise directly multiplies concurrency. The two quantize different objects and pay off through different paths, so they should be evaluated separately and decided separately, not collapsed into a single "we turned quantization on" conclusion.

The cost side differs too. Weight quantization mainly affects the numerical precision of model output, which is a quality problem you can regression-test offline; KV quantization risk accumulates as sequences get longer — long context plus KV quantization is the combination most likely to go wrong, because error keeps accumulating over a long sequence and shows up as a clear drop in retrieval-task accuracy, or as long reasoning chains drifting off course midway.

The Engineering Constraints of FP8 KV

FP8 KV Cache requires native attention kernel support. FlashAttention 3, FlashInfer, and the custom kernels in the various inference engines already provide support at different granularities: per-tensor, per-head, per-token scaling factors. Finer granularity means better precision, but also more extra computation and memory traffic inside the kernel — in the memory-bandwidth-bound decode phase, that extra overhead directly eats into part of the concurrency gain. So "switching to FP8 will definitely raise throughput by 80%" is an overpromise; the real improvement depends on how mature your kernel is, your batch size, and your context length. Before deploying, always run an accuracy comparison on real long-context tasks — do not look at perplexity alone.

Where Quantization Couples Back to Scheduling

This point is rarely stated outright: the goal of quantization is "trade memory for a larger batch," and a larger batch pushes the system toward compute saturation, which in turn changes the optimum for every scheduling optimization.

It shows up in three concrete ways:

• The payoff from speculative decoding drops. As computed earlier, when you are memory-bandwidth-bound the idle bandwidth can absorb the draft's extra compute; once compute saturates, it cannot. Quantization pushes the system toward high batch, which is very likely to land you right across that threshold.
• The optimal batch size drifts. The same max_num_seqs should be set higher under FP8 than under FP16, and every previously learned value is invalidated.
• Attention's share of compute rises. Once KV gets smaller, weight loading accounts for a smaller fraction of per-step time, the attention kernel's efficiency matters more, and the quality of the attention implementation has a larger impact.

So the order has to be: fix the scheduling structure first, then the precision, and only then turn on speculative decoding. Do it the other way around and every step's conclusion gets overturned by the next one.

Long Context Needs Its Own Set of Judgments

Treating context length as just another parameter to tune is the most common mistake in long-context scenarios. Nearly all the experience accumulated in the 8K era no longer holds at 128K, for a simple reason: many of the scheduler's costs grow linearly with sequence length.

Where the Linear Growth Hides

The first direct cost is the memory ledger itself. By the formula above, KV footprint is proportional to sequence length, and a 128K context is sixteen times an 8K one. That means even with max_num_seqs unchanged, the number of requests you can actually run concurrently drops to one sixteenth. A drop of that magnitude immediately triggers a chain reaction:

• Concurrency collapses into the single digits, any burst queues up at once, and the P99 of TTFT gets flattened out permanently (it never comes back down).
• Preemption becomes extremely expensive — one recomputation means re-prefilling tens of thousands or even hundreds of thousands of tokens, costing dozens of times a normal request.
• Block pool utilization oscillates persistently above 90%, LRU eviction fires continuously, and the Prefix Cache hit rate gets polluted by eviction noise.

The second cost is that the distribution of per-step time becomes extremely uneven. A 128K prefill, even chunked, still does attention over tens of thousands of tokens per step, and that amount of compute grows with sequence length; a decode step's compute, by contrast, barely grows at all. Within the same batch, a prefill step and a decode step differ by an order of magnitude in time, and the scheduler has a hard time finding a configuration that satisfies both.

The Nonlinear Degradation of TTFT

Under long context, TTFT does not grow linearly with context length — it grows more steeply. The reason is that queue length is extremely sensitive to available memory: collapsed concurrency causes queuing, and more queued requests further compress the number of KV blocks each request gets, and the preemptions that triggers make some requests recompute and occupy resources longer. This is a positive feedback loop.

So capacity planning for long-context workloads has to work backwards from "how much KV can a card hold," not forwards from "how many QPS can a single card serve." In practice, use the formula from earlier to compute the safe concurrency level at an acceptable queueing delay, multiply by the number of instances to get total capacity, then verify in reverse whether the business peak fits inside that capacity. Leave more headroom than in short-context scenarios, because the variance of long context is itself larger.

The Extra Risk From RoPE Extension

When a model's actual supported context is far larger than its training length, you normally need positional encoding extrapolation (interpolation schemes like YaRN or NTK). This looks unrelated to the scheduling system, but it has an impact that is broadly underestimated: the distribution after extrapolation differs from the one within the native length range, so techniques that depend on "staying close to the main model's distribution" — draft models, quantized models, small-model-assisted decoding — see their acceptance rates drop noticeably.

Put differently, a speculative decoding configuration with an excellent acceptance rate at 4K context may see its payoff cut in half at 64K, because it can no longer keep up with the main model at large positions. So speculative decoding parameters for long context must be re-measured, not carried over from short-context conclusions. The same logic applies to KV quantization: positional extrapolation amplifies variation in the input distribution, and quantization error is amplified more easily on such a distribution.

The Right Way to Break Down Long Context

This is where the value of PD disaggregation truly shows up. A long prompt turns prefill into a huge, compute-intensive, independent phase lasting from hundreds of milliseconds to seconds; mixing it with decode, which needs stable small steps, means you do neither well. Once you separate them:

• prefill instances can run large chunks, large batches, and high compute utilization, optimizing for single-request prefill completion speed;
• decode instances can run small batches with KV prioritized, optimizing for smooth TPOT;
• the two are connected by KV transfer. Long-context KV is large, but you transfer it once instead of having decode read it over and over.

The cost, as mentioned earlier, is KV transfer. Here is an easily overlooked engineering detail: KV transfer time must be significantly shorter than the queueing time it saves, otherwise disaggregation is a net loss. So the test is whether queueing from mixed batching dominates TTFT on the prefill instance — if queuing accounts for little of TTFT, the benefit of PD disaggregation comes mainly from throughput rather than latency, and the transfer overhead eats into it.

Diminishing Returns from KV Compression

Techniques like MLA, cross-step KV compression, and KV eviction strategies have a first-principles goal of shrinking the KV footprint. In real engineering, though, the benefit of compression diminishes, for two reasons.

The first is what happens once footprint is no longer the bottleneck: if your GPU compute is already saturated, smaller KV will not push throughput any higher; it only moves the bottleneck from memory to compute, and the marginal value of compression collapses. The second is that the accuracy cost accumulates with length: the more aggressive the compression, the more obvious the information loss on long context — and long context is precisely the problem compression was meant to solve, which is an inherent contradiction.

My recommended order is: first use paging to drive waste to the minimum (zero accuracy cost), then enable FP8 KV (a controlled cost with mature kernels), and only then consider more aggressive compression or eviction strategies, which require accuracy validation against your own tasks. Skipping ahead to aggressive compression usually means paying an accuracy cost for a problem that is not yet the bottleneck.

Reproducibility: Why Performance Numbers Drift

The most exhausting part of inference tuning isn't analyzing metrics, it's discovering that last week's conclusion can't be reproduced this week. Most causes of performance drift can be categorized, and the remedies aren't hard either; what's hard is that the first time you hit one, it's easy to suspect your own code change broke something.

Cold Start and Warmup

Before a newly launched instance takes real traffic, it goes through a clear performance ramp: weights loading from disk or network, memory allocator initialization, graph compilation or kernel autotuning (the first time a particular shape is encountered, this can cost far more than steady state), and KV block pool preallocation. These costs cluster in the first few dozen seconds of a benchmark run, which severely distorts the mean TTFT.

The disciplined practice is: send a fixed-configuration warmup load before the benchmark, with warmup long enough to cover every shape that will occur (especially max batch size and max chunk), and exclude warmup data from the statistics. At the same time, record the cold-start TTFT distribution separately — if your service needs to scale frequently, cold start is an SLO in its own right, and a lot of production jitter is really caused by scaling events.

Steady-State Noise Sources

Even in steady state, the following factors will move numbers by a few percentage points:

• Clocks and power: consumer-grade or unlocked cards thermally throttle under sustained load, so benchmarks get slower as they run. When comparing across rounds, temperature is a hidden variable.
• Neighboring noise: other processes on the same physical machine, a shared PCIe link, even the fans in the same rack.
• Sampling variation in the request mix: benchmark token lengths are random, so with too small a sample the variance of the mean is large.
• Warmup effects on hit rate: Prefix Cache hit rate climbs at the start of a benchmark as shared prefixes are gradually "warmed," so a hit rate measured without warmup is systematically low.
• Garbage collection and memory compaction: periodic memory compaction in some implementations produces observable periodic spikes.

The countermeasures are: run the same configuration at least three times and take the median, pin the request distribution (pre-generate a fixed token sequence instead of randomizing each run), and attach the hardware state at benchmark time to the report. When two runs differ by less than 5%, treat that as noise rather than an improvement.

Running a Trustworthy A/B

A/B testing during tuning has one special difficulty: the two configurations can hardly run in the same time window, and different time windows collide with exactly the noise sources above. There are several mitigations: alternate runs (A, B, A, B) instead of finishing one and then switching; replay a fixed request sequence; and compare cards of the same model on the same machine to avoid cross-machine comparisons.

There's also a methodological suggestion: change exactly one variable at a time, and write the reason for the change into the benchmark record. Parameters in an inference system are heavily coupled, so if you change three things at once and see throughput rise 20%, you have no idea which one worked and no way to localize a regression later. Many of those "this looked useless but actually matters" conclusions in the middle of a project come precisely from this discipline.

One last point: benchmark conclusions must record their load assumptions. What you write down should be "under a load of 32 concurrent requests, 2K input, 300 output, X% hit rate, configuration A's TPOT P99 is Y milliseconds," not "configuration A is faster." The former can be re-verified in three months; the latter can only be re-guessed.

Turning the Metrics Into Service Commitments

I gave you a metrics checklist earlier, but metrics alone don't solve anything — the hard part is collapsing them into a single line you can promise externally and hold people to internally. This section is about turning metrics into service commitments.

Set the Target Before Talking About Optimization

Most tuning efforts fail at the starting line: without a concrete, written-down target, every change gets judged by "it feels better," and the resolution of feeling is far below real differences. A written target must satisfy three conditions — a specific number, a specific subject (which business, which kind of request), and a deadline. "Improve response speed" is not a target; "bring the P90 time-to-first-token for the document Q&A scenario down from 2.4 seconds to under 1.5 seconds" is.

Targets also need layering: the external user-experience commitment is one line, the internal observable health line is another. The former decides whether you met the goal and whether you need to roll back; the latter gives you early warning — for example "alert when the hit rate drops below a level" or "alert when preemptions exceed a threshold." These don't equal users already being harmed, but they predict harm is coming.

Percentiles Are the Basis of a Commitment

Which percentile to commit to is a decision you have to make explicitly. Committing to the average is the easiest and the most dangerous, because averages get pulled down by many fast requests, effectively excluding the worst experience from the commitment. Committing to the 99th percentile is the safest and the most expensive, and under high variance it's nearly impossible to hit.

In practice, a workable approach is to commit externally at P90 and treat P99 as the internal red line: the former is a level most users can perceive as satisfactory with controllable cost; the latter is the backstop for extreme cases, and touching it means the system is already unhealthy. The average, meanwhile, can be shown as a supplementary reference but should never appear in a commitment.

Metrics Must Be Drillable Down to Action

A metric you can't localize is useless. Total throughput dropped ten percent, and knowing that tells you nothing about what to do. A useful metric is one where you can drill down level by level along the causal chain: when throughput drops, first determine whether it's a demand-side or supply-side change, then look at queue length, block pool utilization, and preemption counts, then drill down to sequences per batch and per-step time.

For build order, I'd suggest: first make sure these levels all have data, then consider finer attribution. A dashboard without layering is more dangerous than no dashboard — it just sends people off in the wrong direction.

Drive with Alerts, Not Dashboards

Dashboards are for post-hoc review; alerts are for real-time response. The two serve different purposes and can't replace each other. The alerts that actually save you have three traits: their thresholds are derived from service commitments rather than guessed, they fire before users complain, and once fired they point directly at one troubleshooting direction.

Filtering by that standard usually leaves only a handful worth configuring: sustained queue length above a threshold (insufficient capacity), block pool utilization persistently near its ceiling (insufficient memory), abnormal preemption frequency (mismatched configuration or a traffic shift), a sudden drop in prefix hit rate (the prompt was changed), and a falling lower bound on accepted length (load change or draft mismatch). The rest fit better on a dashboard as references during troubleshooting.

Sustained-Load Testing: The Questions Five Minutes of Benchmarking Cannot Answer

Short benchmarks validate performance; long-running tests validate correctness. Plenty of configurations that fail in production benchmark perfectly for ten minutes.

Three Classes of Problem Only a Long Run Reveals

• Slow resource growth: request objects not released on completion, block reference-count leaks, memory footprint rising monotonically over time. At the scale of a few hours these look like "it got slower later on," which is easily misread as load fluctuation.
• Periodic spikes: some implementations have periodic resource-compaction or cache-flush behavior that shows up as a per-step time spike every few minutes. A short benchmark may miss it entirely, or happen to include one occurrence and dismiss it as noise.
• State machine leaks: the Prefix Cache block table can develop reference inconsistencies under specific sequence patterns (repeated prefix appends plus rollbacks), showing up as a slowly drifting hit rate or an occasional cliff-like drop.

How to Run It So It Means Something

The key to a long-run test isn't "running it long," it's that the load must be able to expose problems. Replay production traffic rather than synthesizing looping requests — a looped request has a completely identical prefix, which inflates the Prefix Cache hit rate to an unrealistic level and hides many problems. Replayed real traffic has honest length distributions, honest prefix repetition rates, and honest traffic-shape mixes.

For observability, sample once a minute rather than only looking at totals: memory footprint curves, block pool utilization curves, a sliding average of the hit rate, a sliding average of preemptions per second, and percentiles for the key latencies. When the run finishes, plot those curves — any curve trending upward is worth chasing down, even if performance hadn't degraded yet.

Make Sustained Testing Part of the Release Process

A sustained test should be a fixed step in the release process, not something you "remember to run when something breaks." A reasonable arrangement is: after every configuration or model change, run a short benchmark covering typical load to confirm there's no regression; for major changes (new model, new engine version, different quantization scheme), add a sustained run on top.

Also archive the sustained-run results as a baseline: configuration, load description, curves and key numbers for each metric. The next time you change something you can compare directly, which is far more reliable than "I think last time wasn't like this." What performance engineering actually relies on isn't cleverness, it's reproducible records.

The Parameter Tuning Order in Production

Let me converge everything above into one executable path. The core principle is that every step must be confirmable or refutable through observation, and you make no change you can't attribute.

Step 0: Write Down the SLO

Tuning without an SLO has no endpoint. First establish: the P99 ceiling for TTFT, the P99 ceiling for TPOT, target throughput or daily token budget, the context length distribution, the output length distribution, and peak QPS with its peak-to-trough ratio. These five determine the reasonable range for every parameter afterward.

Step 1: Measure First, Tune Second

Deploy the default configuration, run one baseline pass, and record every item on the metrics checklist above, including TTFT/TPOT percentiles, throughput, preemption counts, block pool utilization, and prefix hit rate. Tuning with an incomplete baseline is just tuning blind.

Step 2: Tune the Memory Ledger First, Not the Algorithm

This layer has the biggest impact and is the most commonly skipped. Use the KV formula from earlier to compute theoretical concurrency, then compare against reality:

• max_model_len — set it from the real distribution, don't set it to the model maximum for convenience. This directly determines the KV reserved per sequence, and its damage to concurrency is the largest of any parameter. Set it to 128K when the business's longest case is 16K, and concurrency drops by half immediately.
• gpu_memory_utilization — leave room for communication buffers and sampling, don't push it to 0.98.
• max_num_seqs — the concurrency cap within a batch. Interactive workloads want a smaller value, batch workloads can scale it up. Before scaling it up, check whether TPOT P99 breaks the line.
• KV block pool utilization — if it stays pinned near 100%, memory is insufficient and you should reduce max_model_len or add cards, not keep tuning something else.

Step 3: Turn On the Structural Switches

With the memory ledger correct, confirm these in order: chunked prefill (modern engines enable it by default), prefix caching, CUDA Graph / cudagraph mode, and asynchronous scheduling. This layer usually has the largest gains and almost no accuracy risk.

Step 4: Then Touch Precision

Once the scheduling structure is settled, add quantization, deciding on weights and KV separately. You must run an accuracy regression afterward, especially on long-context tasks. If this step breaks your SLO, go back to Step 3 and re-examine the concurrency settings rather than trying to compensate with tuning.

Step 5: Speculative Decoding Last

Speculative decoding is the only optimization that can reduce overall throughput, so do it last, and use the criteria from the previous section: look at accepted length, TPOT, and total throughput together. If P10 accepted length is too poor, turn it off — don't expect parameter tuning to rescue it.

Step 6: Benchmark Regression and Sustained Run

Keep a benchmark record for every step's change. Once everything is final, run one sustained test (hours to a day), watching closely for leak-like slow growth, block pool utilization drifting over time, and periodic spikes. Configurations that survive five minutes of benchmarking and collapse after a day are not rare, especially under mixed multi-tenant load.

Quick Reference for the Tuning Order

Order	Action	Expected Gain	Risk
0	Define the SLO and the real load distribution	—	Skip it and nothing else resolves
1	Baseline benchmark, build a complete metric snapshot	—	Skip it and nothing can be attributed
2	max_model_len / memory watermark / max_num_seqs	Largest	Misconfigure it and concurrency dies outright
3	chunked prefill / prefix cache / cudagraph / async scheduling	Large	Low, mostly memory and eviction
4	PD disaggregation (when long prompts are a large share)	Large	KV transfer can become the new bottleneck
5	Weight quantization → KV quantization, evaluate one at a time	Medium to large	Accuracy, especially for long context
6	Multi-tenant QoS / rate limiting / preemption policy	Medium	Policy is complex, needs observability to support it
7	Speculative decoding	Medium, can be negative	May reduce throughput under heavy load
8	Sustained run and regression	—	Time cost

A Starting Configuration to Begin From

Exact parameter names shift between versions, so verify them against the documentation for the version you use before you commit:

# 起步配置：结构优先，精度和投机解码先不开
# vLLM V1 系（参数名请对照当前版本文档）
python -m vllm.entrypoints.openai.api_server \
  --model /models/your-model \
  --served-model-name your-model \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192 \
  --port 8000

# 关键解释：
#   --max-model-len 32768     按真实 P99 长度设，别设成模型上限
#   --max-num-seqs 128         交互业务起点；TPOT P99 破线就往下调
#   --gpu-memory-utilization   留通信/采样余量，不要顶满
#   --enable-chunked-prefill  现代版本默认开，显式写出来便于核对
#   --max-num-batched-tokens  单步 token 预算；TPOT 有周期性尖刺就往下调

# 确认生效与观测
#   /metrics 里核对：num_requests_waiting / num_requests_running
#                    prefix_cache_hits_total / prefix_cache_queries_total
#                    num_preemptions_total / gpu_cache_usage_perc

# 之后再逐项开，一次只开一个，改完就压一轮：
#   量化（权重） -> 精度回归 -> 压测
#   量化（KV）  -> 长上下文精度回归 -> 压测
#   投机解码     -> 接受长度 / TPOT / 总吞吐 三个数一起看

Common Anti-Patterns: Things That Look Right and Bite Back

The most expensive cost in tuning isn't a wrong calculation — it's tuning based on an intuition that sounds perfectly reasonable, then spending a week rolling back after it turns out slower. These are patterns that keep recurring in inference engineering.

Setting max_model_len to the Model Maximum

This is both the highest-value and the most frequently repeated mistake. This parameter sets the ceiling on KV reserved per sequence, and in most implementations KV reservation is accounted for as "worst case." Setting it to 128K when your actual business P99 is 12K is equivalent to billing every concurrent request at sixteen times its true cost.

The correct approach is to start from the real length distribution, pick a value covering the vast majority of requests (usually around P99.9), and prepare a separate degradation path for unusually long requests — truncation, segmentation, or routing to a dedicated instance. A handful of oversized requests dragging down the throughput of an entire cluster is the most typical symptom of this mistake.

Looking Only at Average Latency

The average has almost no decision value in an inference service. The same P50 can mean completely different service experiences: a service with "average TTFT 800ms, average TPOT 18ms" may have 5% of requests whose TPOT exceeds 200ms (caught by a large prefill step), and those 5% are the ones on the complaint list, yet they contribute less than 1ms to the average.

The discipline is: report at least P50 and P99 for every latency metric, and draw capacity planning lines at P99. At the same time, make "the share of requests meeting your SLO" (goodput) the single headline number you report externally, because it naturally includes the tail.

Treating Speculative Decoding as Free Speedup

I covered this with formulas and the bandwidth headroom concept earlier; here I just want to emphasize the shape it takes in practice: a speculative decoding configuration that gives a 2x speedup in a low-load benchmark may reduce overall throughput in production peak, because at peak the GPU is already compute-saturated and the draft's extra compute becomes pure overhead.

The order of judgment has to be: first determine the load state at peak, then decide whether to enable it; after going live, monitor accepted length, TPOT, and total throughput together, and turn it off the moment total throughput drops, even if TPOT improved clearly. This decision belongs in the on-call runbook, not left to whoever is on duty next to improvise.

Treating Quantization as Free Memory

The memory quantization saves does convert into more concurrency, but the accuracy cost surfaces in some business scenario that isn't obvious — usually long-context retrieval, or multi-step reasoning that needs precise numbers. Such regressions are very hard to catch on standard evaluation sets, because those are far too small.

So the validation set for quantization must contain the longest real inputs from your own business, not samples from a public benchmark. Evaluate weights and KV separately too: weight quantization affects the global output distribution, KV quantization affects long sequences, and their degradation modes differ.

Tuning the Chunk Size of chunked prefill to the Extreme

The reasoning "smaller chunks make TPOT smoother" is correct, but only half correct. In the memory-bandwidth-bound decode phase, having too few tokens in a batch collapses the GPU's bandwidth utilization, and prefill throughput drops too. Chunks that are too small trade overall throughput for TPOT spikes — and those spikes may only affect a tiny slice of P99 in the first place.

The right order is: first measure the actual magnitude of the TPOT P99 spike with the default configuration; if the spike really does violate your SLO, reduce the chunk size and re-measure throughput on every reduction. If the spike is acceptable, leave it alone — behind the default value is validation against a great deal of real load.

Restructuring Prompts Just to Raise the Hit Rate

Prefix Cache is the optimization with the most direct benefit and the lowest risk, so it is easy to over-pursue. But if the means of "raising the hit rate" go off track, the cost moves elsewhere: stripping timestamps, user identifiers, and random values out of requests to gain hit rate destroys business semantics; forcing every prompt to converge on one template limits business flexibility; moving volatile content to the start of the prompt (following "longest common part first" while ignoring how volatile it is) drops the hit rate to zero.

The principle is to split prefix and suffix by "rate of change" rather than by "length": everything genuinely constant goes first, everything that changes every single time (including what looks like it changes once) goes last. This split has to be agreed with the business side, not decided unilaterally by the inference team.

Using a Single Benchmark Run to Set Long-Term Configuration

Benchmarks usually use fixed concurrency and a fixed length distribution, while production arrivals are bursty and mixed in shape. The most common failure is sizing resources for peak throughput and then watching a lot of idle capacity in the trough without daring to scale down — because you aren't sure whether shrinking will make peak collapse.

The pragmatic approach is to plan capacity off percentiles rather than the mean, and accept idle capacity at night; if the cost of night-time idleness really is unacceptable, consider an elasticity scheme then. Splitting "peak configuration" and "routine configuration" into two parameter sets and switching on a schedule or a queue-watermark trigger is more practical in most scenarios than chasing a "one config for everything" design.

The Pre-Launch Checklist

The table below collapses the whole article into executable actions. It isn't a process standard — it's an index to consult when something goes wrong, to jog your memory about which steps you might have skipped.

Check	How	Pass Criteria	Consequence of Skipping
SLO is defined	Write down TTFT P99, TPOT P99, and peak throughput as numbers	Specific values with target dates	Tuning has no endpoint
Length distribution is known	Compute P50/P99 for input and output tokens plus the share of overlong requests	You can plot the distribution	max_model_len can't be determined
Baseline is complete	One benchmark pass with default config, full metric snapshot	Percentiles complete, not just means	Nothing can be attributed
Memory ledger is reconciled	Back out concurrency with the KV formula and compare to block pool utilization	Theoretical and measured values in the same order of magnitude	You can't tell memory problems from scheduling problems
Block pool has headroom	Watch KV usage during peak	Below 90% in steady state	Continuous preemption, an avalanche of recomputation
Preemption count is alerted on	Set a threshold on preemptions per second	Doesn't fire continuously at peak	A preemption storm nobody notices
Prefix hit rate is observable	Export hit queries and hit counts	You can trace it to a specific business	The hit rate drops and you don't know who caused it
Prompts are layered by rate of change	Audit the composition of the system prompt	No timestamps or random values at the head	The hit rate silently goes to zero
Long requests have a degradation path	How are lengths beyond the threshold handled	There are explicit rules	One request takes down the whole cluster
Multi-tenant is rate limited	Track concurrency and queueing per tenant	A single tenant can't fill the batch	Big customers starve small ones
Quantization accuracy is regression tested	Compare against your longest real business inputs	No significant drop in key metrics	Quality incidents are hard to localize
All three speculative decoding numbers are in hand	Accepted length P10, TPOT, total throughput	Total throughput doesn't drop	Everything gets slower at peak
Sustained run is verified	Continuous operation for several hours or more	No memory growth, no periodic spikes	You get woken up in the middle of the night
Benchmarks are reproducible	Fixed request sequence, same machine comparison, three repeats	Same config varies by less than 5%	Conclusions can't be trusted
Config changes are rollback-able	Parameters are versioned with baseline snapshots	You can revert within ten minutes	Chaos during an incident

Canarying, Config Management, and Multiple Versions

Tuning produces a set of parameters; getting that set safely into production is a different job. One special property of scheduling systems is that a config error has full blast radius — a configuration with a wrong batch size cap affects every request at once, with no canary to speak of. That's why config management matters more here than in an ordinary business.

Parameters Must Be Versioned

The most basic rule and the most frequently skipped one: every performance-affecting parameter needs version history and must be traceable to specific instances. This means config can't live only in the startup command line (an instance can't introduce itself), and there has to be a centralized config record that instances fetch at startup, writing the version number into their own metadata and startup logs.

The direct payoff of versioning only shows up during incidents: when something is anomalous, the first thing to do is confirm "which instances are running which config version." If that step depends on human memory and scrolling through chat logs, incident response grows by fifteen minutes or more. In a multi-instance, partially canaried state the capability is indispensable — without version numbers you can't tell whether an anomaly is related to a particular release.

Shadow Traffic

Inference services find traditional request-level canarying hard, because the same request sent to the old config and the new config may produce different results, which the business can't accept. But shadow traffic is doable: copy real production requests and send them to instances on the new config, recording the results without returning them to the user, then compare performance metrics and output quality on both sides.

The value of shadow traffic is that it exposes problems benchmarks never find: real traffic has different length distributions, shape mixes, and prefix repetition rates, and real traffic contains all kinds of odd input formats. The cost to note is doubled memory and bandwidth, so you normally shadow only a fraction of traffic, and you must make sure the shadow instances' processing doesn't overload and affect the production instances in turn.

The Right Order for Config Changes

Config changes should be canaried in order of increasing blast radius: change a single instance and observe, then a subset, then everything. Not change everything at once and hope for the best. The value of that order shows up especially clearly in the incidents I described earlier — an avalanche, a hit rate going to zero, a quantization regression. If any of the three is only discovered at full rollout, the cost is orders of magnitude higher.

There's another easy-to-miss discipline: after a change, wait long enough before drawing conclusions. Hit rate, the Prefix Cache block table, and speculative decoding accepted length all have warmup periods, and numbers seen within a few minutes don't represent steady state. Conversely, some degradations are slow and take hours to appear. So the right moment to conclude depends on the metric, and you can't treat everything as "observe for ten minutes."

Coexisting Versions

Old and new versions necessarily coexist during a model upgrade, which brings two problems. The first is memory and capacity allocation: each version reserves worst-case memory, and the total may far exceed the capacity plan for a single version, leaving service degraded during the upgrade. The fix is to control the coexistence ratio, or to deliberately degrade some functionality during the switching window (temporarily disable speculative decoding, lower the batch size cap) to free up headroom.

The second is routing quality: old and new versions have different performance characteristics, so if load balancing just splits randomly or by shortest queue, certain request shapes get systematically routed to the wrong version. During the switch you should use shape-aware routing, and be explicit about which request class goes first and which goes last — usually switching the low-risk, high-volume shapes first and the most quality-sensitive class last.

Detecting Config Drift

One last piece of long-term discipline: periodically reconcile the "config actually in effect" against the "config you intended". Drift comes from manual per-instance tweaks, config files being overridden by environment variables, startup scripts differing across environments, and rollbacks that only changed some instances. The standard practice is to periodically collect the actual startup parameters from all instances, run a consistency check, and alert automatically on deviations.

This looks mundane, but it's the only thing preventing "the misbehaving instances mixed in among healthy ones" from persisting indefinitely. And in a system that routes by shape and canaries by batch, a configuration inconsistency in a few instances shows up in aggregate metrics as a persistent small anomaly — it triggers no alerts, and it can never be fixed by tuning parameters.

Eviction Policy: When the Cache Is Full, Who Gets Thrown Out

Prefix Cache and the block pool share the same memory, so "cached" and "running requests" are in practice competing for one pool. When the pool fills you must choose, and who to discard and who to keep looks like an implementation detail while directly determining how stable throughput and latency are.

The most common approach is least-recently-used: whoever hasn't been touched longest gets dropped. This works well in a general cache, but in inference it has two obvious problems. The first is that it ignores size: evicting a 20,000-token sequence costs a hundred times more than evicting a 200-token one, and ordering by "least recently used" treats them as an equivalent single eviction. The second is that it ignores the future: in a long conversation, a sequence whose current turn isn't finished yet — if its history prefix is evicted, the next request has to recompute from scratch, and that sequence is precisely the most valuable thing in the cache.

A more targeted approach is to rank by recomputation cost: evict blocks that are expensive to rebuild but low in access value first. There are two useful signals here. The first is size — large blocks cost more to recompute, but size also suggests they might be more important. The second is age — blocks from the same conversation tend to stay together, avoiding evicting half of a pair and leaving the other half meaningless.

Another line of thinking is active swapping rather than outright discarding. Write cold blocks to host memory instead of freeing them; the cost is one memory-bandwidth write, and the benefit is a possible future hit. This pays off clearly on businesses with "long conversations with long gaps," such as a user who thinks for ten minutes before asking the next question. But if the hit rate is itself low, the bandwidth spent writing is pure waste. Whether to enable swapping should be decided by the hit rate, not by what happens to be convenient to implement — measure the hit rate first, then decide.

The single most effective adjustment in practice is to leave a buffer for eviction: don't wait until the pool is 100% full before you start evicting, start at seventy or eighty percent. The reason is that at full occupancy the eviction work and the allocation of new requests happen in bulk at the same instant, forming a spike; triggering early smooths that wave out. The change is tiny and the effect is usually obvious — preemptions drop, spikes decrease, and throughput barely moves.

Combining Scheduling Policies

I covered first-come-first-served and last-come-first-served earlier; real production often needs something more complex. Complexity costs interpretability, so the number of layers should be as small as possible, and every layer you add must be able to answer "why was this request treated this way."

The first layer is priority. Giving different businesses different priorities is the most intuitive requirement and the easiest to implement: high-priority requests go to the front of the queue, ties broken by arrival order. The risk is that once priority configuration tilts, starvation is inevitable — if high-priority traffic exceeds system capacity, low-priority requests wait forever. So priority must be paired with quotas rather than relying purely on the scheduler's queue-jumping mechanism.

The second layer is deadlines. Requests with deadlines must complete before a certain moment, and the core idea is to sort by "remaining slack" — run the most urgent first. The easy mistake here is looking only at the estimate of total duration, when the information that actually determines whether it finishes on time is "how long the requests ahead of it still need." In implementation you usually use the request deadline minus the estimated processing time as the sort key, re-evaluating during scheduling when necessary.

The third layer is cost awareness. In pay-as-you-go cloud environments, different hardware has different cost-performance ratios, so some traffic can go to cheaper but slower instances. But this policy is far less broadly applicable than the first two, because it requires the business to be latency-insensitive and requires the scheduler to accurately judge which instance is "better value" — and that judgement is easy to get wrong under fluctuating load, because getting it wrong sends traffic to an instance that is actually slow, while monitoring shows only uneven load distribution.

My recommendation is to enable at most the first two layers. Beyond two layers, predictability drops, troubleshooting difficulty rises sharply, and the payoff is usually diminishing. More importantly: before adding a policy layer, answer one question — does it solve a problem that has actually occurred. A policy with no real case behind it only adds complexity to the system, and complexity is itself a source of incidents.

Hardware Differences and Engine Adaptation

Finally, one easily underestimated reality: the same configuration can differ more across hardware than two configurations differ on the same hardware. And these differences aren't random — they have clear sources.

The first source is operator coverage. Inference engine performance depends heavily on the implementation quality of the underlying operators, which are supplied by hardware vendors and the open-source community together, and coverage, fusion level, and supported parameter ranges differ by card model. An optimization that works on one card can fail completely on another for lack of a corresponding fused operator, or even be slower than not doing it. So "this config runs great on someone else's cards" is worth almost nothing in performance engineering.

The second source is compilation and autotuning. Many implementations autotune the first time they encounter a given shape, choosing the best algorithm and tiling, then cache the result. This can run for several minutes during cold start. If your benchmark is too short, you measure untuned performance and your conclusions are systematically pessimistic; if production traffic quickly hits a new shape after startup, it triggers a noticeable performance wobble. Benchmark duration must exceed the autotuning convergence time, otherwise the numbers you measure are worthless.

The third source is the memory subsystem. Cards differ widely in memory capacity, bandwidth, cache hierarchy, and interconnect, and these directly determine the location of every inflection point analyzed earlier: how much cache fits on one card, how large a batch memory bandwidth can sustain, and where parallel splitting stops paying off. These numbers should be baseline data for every environment, computed before deployment rather than back-inferred from benchmarks.

Putting it together, the right approach is to standardize the tuning process itself so it can be re-run on any hardware: a fixed load description, a fixed benchmark method, a fixed metric definition, producing a baseline table that compares environments. The benefit isn't just knowing whether the current config is good, but knowing which direction to re-tune after a hardware change — whether memory is short and you should shorten context, whether bandwidth is short and you should change the parallel split, or whether compute is short and you need more equipment. The three numbers in that baseline table are a far more reliable basis than intuition.

Deployment Shapes and the Go-Live Checklist

The engineering work outside performance tuning equally determines whether the service runs long term. This section won't expand on them; it leaves a checklist so you can audit yourself for obvious omissions.

On startup and loading: where weights come from, how long cold start takes, whether there's a local cache, and how long a reload takes once the cache is invalidated. In scenarios that scale frequently, cold start time directly determines whether scaling is feasible at all — if an instance needs three minutes to become ready, then during a traffic surge the first two minutes essentially can't be served, and requests in that window either queue or fail.

On readiness and liveness: what the health check should inspect. Checking only that the process is alive isn't enough — a process that's alive while the scheduler has deadlocked is a real scenario. The reasonable approach is to send a minimal request and wait for the correct response, while also checking that the scheduler is still making progress. Liveness and readiness checks must be separate: liveness failure triggers a restart, readiness failure only pulls traffic without restarting. That distinction avoids a large number of pointless restarts during jitter.

On scaling: what triggers a scale event, how long after scaling up the instance is ready, and how you avoid losing in-flight requests when scaling down. Scale-down normally has to stop taking new requests first, let in-flight ones finish, and then release — and that process must be far slower than scale-up. Too aggressive a scale-down policy causes periodic oscillation: scale up, scale down, scale up again, each cycle accompanied by a cold start and a performance trough.

On logging and observability: per-step time, sequences per batch, block pool utilization, hit rate, preemption counts — these should be reported on a periodic basis, not printed only when something errors. Log levels need to be clearly separated: put routine performance metrics at debug level, so you don't turn on high-severity logging in production just to watch metrics and fill up the disk.

On fault tolerance: whether to retry a failed request, whether retries worsen overload, and what timeout to set. There's a classic trap here: when the server is overloaded, returning an error is better than hanging indefinitely, because hung requests keep holding resources and make the overload persist. Prompt rejection plus fast failure is part of a rate-limiting system, not a step back in availability.

Not one item on this list is directly about performance, but they are the precondition for keeping performance results. A single restart triggered by a misconfiguration can wipe out weeks of tuning.

External Memory or Long Context

When building a long-context application, one choice keeps recurring: pack more content into the context, or retrieve from an external store and inject it on demand. This is a very practical engineering question, because it directly determines memory cost and TTFT.

From a cost perspective, injecting retrieved results is clearly cheaper. Retrieval usually returns a few thousand items while the full history might have hundreds of thousands. What you save isn't just memory and prefill compute, but TTFT — prefill time is proportional to the injected length, and in long-conversation scenarios that can be the main source of waiting.

But external memory has its own cost, and it's the hard-to-quantify kind. Retrieval loses the original ordering and contextual linkage, and coreference across turns ("it", "the one above") is often unresolvable inside retrieved fragments, while stitching information across turns can produce contradictions. Output quality drops as a result, and by how much depends on how much the task depends on coherence — small for chitchat, large for complex reasoning.

A middle-ground approach is a hybrid strategy: keep the full history of the last few turns without retrieval to preserve near-term coherence, and retrieve and summarize older history so information isn't lost. The turn threshold needs tuning per business, and its adjustment direction is clear: if output quality degrades as the conversation lengthens, retrieval is dropping information that has started to matter, so raise the threshold; if TTFT is unacceptable, too much is being injected, so lower it.

A more important point is that this choice feeds back into the benefit of caching. A full-history injection scheme has almost no reusable prefix (it changes every turn), while a retrieval-injection scheme has much higher prefix reuse if retrieval results are stable. So when you choose the hybrid strategy, putting stable historical summaries first and retrieved results last gives you quality, latency, and caching benefits at once. These three dimensions aren't independent of each other, and finding one layout that optimizes all three is far more effective than optimizing them one at a time.

Finally, one option that's often overlooked: if the engine you deploy supports cache tiering, meaning the Prefix Cache spills over to host memory or even local storage, then the cost of full history drops significantly — because it's no longer recomputed every time, just read once from cache. At that point the choice may tilt toward "keep more full history," because the marginal cost of long context goes from "recompute every time" to "read from disk every time." Infrastructure capability changes the optimum of the architecture above it, which is exactly why tuning has to be continuous rather than done once.

Locking the Tuning In

The hardest part of tuning isn't coming up with that one key change — it's making sure the change isn't quietly reverted six months later. Parameters drift, versions get upgraded, new people wonder about a value they don't understand, and the moment nobody can explain "why is this number this big," it will get changed.

The first thing is to write down the reasoning, not just the conclusion. A conclusion doc saying "max sequences per batch set to 128" is worthless; what's valuable is "under a load of 2K input, 300 output, and 32 concurrency, this value keeps TPOT at a high percentile under a given line; going higher breaks the line, and going lower drops throughput by more than 5%." The latter can be reviewed by a human and re-derived when the load changes; the former can only be blindly inherited or overturned.

The second thing is to script the baseline. The benchmark script, load description, metric collection, and result persistence should be one set of scripts you can re-run directly, taking config as input and producing a complete metric snapshot as output. The problem with manual benchmarking isn't only that it's slow — it's that the operational details differ every time, so two runs aren't comparable. The payoff of scripting isn't saving time; it's driving the cost of "compare before and after" down to nearly zero, which is precisely what makes the "one variable at a time" discipline sustainable long term.

The third thing is to establish a regression baseline and archive it long term. Every tuning outcome is recorded as a baseline entry containing config, load, and each metric. Any later change compares against the most recent baseline, and a deviation means this change had an impact that needs explaining. This comparison doesn't need to be complex — just one table with fixed metric definitions. Its value only becomes apparent during an incident, when you can quickly tell "was this anomaly introduced by this change, or was it always there."

The fourth thing is to bring the performance budget into design review. If a new business feature significantly increases context length, introduces a new request shape, or changes the prefix structure, its resource consumption on the inference system can be estimated at design time. The estimation method is the formulas from earlier: context length determines cache footprint, request shape determines average output length, and the two together determine total token production. Writing the estimated capacity impact into the design review lets the scaling plan happen ahead of time instead of reacting passively when production alerts.

The fifth thing is to set up watchdogs rather than relying only on manual inspection. The alerts worth configuring that I listed earlier share one property: their thresholds can be computed from the derivations above rather than guessed. A watchdog's value is in covering the gaps in human memory — a service that's been running for six months goes unwatched most of the time, and when something breaks the first reaction is "something feels off" rather than "the hit rate dropped to zero." The former takes hours to diagnose; the latter takes one look at a dashboard.

The common logic behind all five is this: converting a one-time clever decision into a repeatable process that doesn't depend on individual memory. Late in performance engineering, what actually works is usually not some subtle technique but the fact that this process is still intact six months later.

Why Common Configs Can't Be Copied Verbatim

Finally, something that looks off-topic but is very practical: the config examples on the internet and in internal docs mostly cannot be used in production directly. The reason isn't that the authors were wrong, it's that those configs mean nothing once you remove the assumptions that made them hold.

The most typical example is writing the maximum context length as the model's physical limit. Whoever wrote that number was thinking "this way I can handle the longest inputs," but they don't know the real input distribution of your business, and they don't know this number directly crushes concurrency. In their scenario the impact is minor; in yours it might be throughput cut in half.

Another typical one is writing memory utilization as a very high value. This is usually because the author needed to fit the model and their memory really was tight. But for you, if you have memory to spare and copy that high value anyway, you're giving up concurrency for nothing; and if you copy someone else's low value, your model may not load at all. The right approach for this kind of parameter is to recompute it every time, not look it up.

There's a class that's purely environment-dependent: parallelism, batch size caps, the number of speculative decoding steps. The correct values for these depend on hardware, model, and load together, so the numbers in any example are worthless as a reference. The right approach is to understand the mechanism and adjustment direction of the parameter, then measure it in your own environment — which is exactly the throughline this article has been emphasizing.

So in this article I've tried to give you mechanisms, formulas, decision criteria, and the order of judgment, rather than a config you can paste directly. The reason is simple: mechanisms are stable and transferable; specific numbers are not. Master the mechanism and you can derive the right value for any config in your own environment; hold a config and you only know what someone else measured in someone else's environment.

Where Things Stand in 2026, and What's Still Unsolved

Three Things That Have Become Consensus

Engines are undergoing large-scale rewrites. vLLM V1 pushed the scheduler down, unified KV management, and minimized cross-language boundaries, bringing not just performance but maintainability — a lot of early Python-side logic is gone. SGLang uses RadixAttention, which is more natural for multi-turn and structured generation. TensorRT-LLM takes the compilation route: after shape specialization its performance ceiling is higher but its dynamism is worse. The consensus is that the "Python scheduler + tensor concatenation" structure of the V0 era has run its course, and every engine after V1 is being rewritten around "reducing per-step CPU overhead."

PD disaggregation has moved from paper to default architecture. Prefill and decode have completely different resource profiles, and the cost of mixed batching is now broadly understood. The remaining engineering difficulty is all in KV transfer: RDMA passthrough, compressed KV transfer, and front-end load-balancing decisions across nodes.

KV Cache tiering is the next battlefield. GPU memory can't hold the full KV for long contexts, so it tiers out to host memory and to NVMe. SGLang's HiCache represents this line of work, and its core difficulty is transfer latency and predicting the hit path — which tier to read from depends on whether traffic repeats, so fundamentally it's hit-rate engineering on a new battlefield.

Native Self-Speculation Is the Obvious Direction

If the base model ships with an MTP head, or was trained with multi-token prediction in mind, the draft portion is nearly free, and the acceptance rate is higher too because the representations are shared. What matters in 2026 isn't whether speculative decoding can be used at all, but co-design between the model side and the system side: aligning the MTP head with the main model's distribution at training time is far more effective than bolting on a draft model afterwards. This also means that when you switch models you should treat speculative decoding payoff as one of the selection criteria.

Several Directions That Are Still Immature

Adaptive scheduling hasn't been settled. The ideal scheduler would pick batch size, speculation switches, and PD ratios automatically based on real-time load, but these decisions are coupled with each other and greedy auto-tuning tends to oscillate. Most production systems are still fixed configuration plus manual tuning.

Quantization gains are diminishing. FP8 is already the baseline, and formats below that, like FP4, are still unstable in both accuracy and kernel support, while the damage they do to long-context and complex reasoning tasks is disproportionate. Short answers, classification, and extraction tasks can eat the quantization dividend; long-chain reasoning can't — which means a single unified quantization strategy will hurt both classes of load at once.

Scheduling paths differ a lot on non-NVIDIA hardware. Operator coverage and kernel maturity on domestic accelerator cards determine which optimizations are actually usable: without native FP8 attention support, the payoff from KV quantization is discounted; if the CUDA Graph equivalent can't be tuned well, dynamic batch overhead can't be pushed down. When migrating across hardware, you must walk steps 2 through 7 again, and must not assume configs are portable.

Converting Performance Into Money

Performance metrics ultimately have to answer a business question: how much more will this approach cost per year, and is it worth it. Without that step, all the tuning above stays at the level of technical self-indulgence.

The Two Halves of Cost

Splitting inference cost into two items makes the picture much clearer: equipment amortization and power plus data center costs. Amortization is a fixed cost computed by dividing purchase price by expected service life and then by total runtime hours; power varies with wattage and utilization. The latter is often ignored, but on an accelerator card under sustained full load it can account for more than thirty percent of total cost.

This split has one direct corollary: the payoff of throughput optimization is a direct reduction in equipment count, while the payoff of latency optimization is reduced business churn. The two are booked separately and shouldn't be measured with the same metric. Many teams put all their attention on throughput because it's easy to quantify and easy to report, while the business payoff of latency can only be estimated with help from the business side — yet the latter is usually the bigger share.

How to Compute Unit Cost

The most practical definition is "cost per million tokens." The algorithm is simple: total tokens produced by all instances over a period, divided by the equipment cost for that period, multiplied by a million. What actually gets computed wrong is the denominator's definition — do you include periods when the machine is idle? Is idle capacity during the trough amortized across all periods, or only across periods with load? The two methods yield completely different conclusions.

My recommendation is to amortize across the full day, because idle equipment really is costing money, and ignoring it distorts optimization decisions. Correspondingly, "night-time idle capacity" becomes an explicitly visible cost item, which gives the elasticity scheme mentioned earlier a quantitative basis: when the amortized cost of night-time idle capacity exceeds the extra operational cost of the elasticity scheme, elasticity pays off; otherwise just accept the idle capacity.

The Quality Versus Cost Tradeoff

This is the cost that's easiest to miss. Quantization, distillation into smaller models, shortening outputs, and narrowing the retained context all reduce cost, and all lose quality somewhere. The problem is that only part of that loss shows up in technical metrics: drops measurable on an automated evaluation set are visible, while the ones that aren't measurable (business-specific long-tail scenarios, subtle differences requiring expert judgment) are invisible — and the invisible part is usually larger.

The remedy isn't refusing to cut cost; it's turning quality acceptance into a process as well: gate on an evaluation set built from your own business, containing real long-tail samples, rather than on a public benchmark. A public benchmark is for quickly ruling out obviously degrading changes, not for granting approval. This applies to every cost-cutting measure, including quantization, switching to a smaller model, and limiting context length.

Two Curves for Capacity Planning

Plot demand and supply on the same chart and capacity planning becomes a very simple question. The demand side is the curve of token consumption over time, and note that it isn't linear in request count — for the same thousand requests, ten times the output length means ten times the cost. The supply side is available equipment count converted into throughput capacity.

What you really watch during planning is how the relative position of the two curves changes over time, not their absolute values. For a business with a high peak-to-trough ratio, even with resources that are completely sufficient, queueing delay at peak will be poor — that's the path by which a capacity problem turns into a latency problem. The common response is to provision for peak and shrink at the trough, which we've already discussed; a more thorough approach introduces admission control in the queuing-theory sense, so the system degrades gracefully under overload instead of queueing without limit.

The Three Most Common Production Incidents

Turn everything above around and you can derive the three most common failure shapes. Recognizing them is more useful than memorizing any single parameter recommendation.

1. An Avalanche Triggered by One Overlong Request

Shape: at some moment an input arrives far longer than usual, or a batch of requests is submitted at once. The block pool fills within seconds, the queue piles up rapidly, preemption begins, and then the chain reaction follows — preempted requests need recomputation, recomputation occupies more time, the queue grows longer, and block pool utilization rises further.

Root cause: Continuous Batching has no per-request resource ceiling, and worst-case reservation amplifies the cost of an overlong request into something uncontrollable. At the same time these requests occupy a scheduler step for an unusually long time, so during the synchronous wait every other request stalls.

Response: the first layer is a hard cap at admission — reject outright or move to an async queue past a threshold, and don't let it reach the scheduler. The second layer is a per-request token budget within the batch, processing overlong input in chunks rather than consuming it all at once. The third layer is reserving part of the block pool as emergency headroom that doesn't participate in normal allocation. The key is intercepting it before it enters the scheduler, because once it's in there, the recovery cost is extremely high.

2. The Hit Rate Silently Dropping to Zero

Shape: no errors, no alerts, throughput dips slightly, queueing grows slightly. It's very easy to blame "traffic grew," when actually prefix sharing has been broken.

Root cause: some field in the prompt was quietly changed into a varying value. Common sources include someone adding a timestamp or version number to the system prompt, the routing layer injecting a request identifier into the prompt, the business side prepending a user profile for personalization, or an upstream multi-turn component quietly adding summary compression. These changes are usually made by teams outside the inference service.

Response: this is a monitoring problem, not a tuning problem. You must alert on the prefix hit rate and be able to break it down by business and by request type — only knowing "which class of request lost hit rate" makes the cause findable. The fix lies in how the prompt is organized: constant parts first, varying parts last, plus a review convention for prompt management where any insertion has to state whether it affects the prefix.

3. A Delayed Eruption from a Quantization Regression

Shape: everything is fine for a while after go-live, then quality suddenly drops in one particular business scenario, and it takes the team a long time to trace it back to quantization. Metrics on the standard evaluation set barely move, and rolling back doesn't immediately reproduce it either.

Root cause: the degradation is conditional, depending on specific input characteristics. Common ones are long context (error accumulation), tasks needing precise numbers (numerical error amplified), and rare long-tail inputs. The evaluation set didn't cover these conditions.

Response: before quantization goes live there must be a round of regression testing on real business long-tail samples, with long inputs and numerically sensitive samples deliberately constructed in it. After go-live, keep a fast rollback switch, and give weight quantization and KV quantization separate switches — their degradation modes differ and each needs its own rollback. The lesson from this class of incident is: a condition you haven't covered isn't safe just because nothing has gone wrong; it simply hasn't been triggered yet.

Recognizing What's Common to All Three

The three incidents share one thing: none of the trigger conditions live inside the inference service's own code — an overlong input comes from the business side, a prompt change comes from the algorithm side, a quantization rollout comes from a cost-reduction target. That explains why the inference team often only learns what happened after the incident.

The direction of the response is common too: make boundary conditions explicit and push them forward into the admission layer (length caps, shape validation, admission rate limits), write cross-team agreements down as process (prompt change review), and pair irreversible operations with observable acceptance criteria (quantization regression sets). None of this work lives in the inference engine's code, but it determines whether the engine can survive in production.

Chaining It Together: One Complete Optimization Walkthrough

What came before was a horizontal breakdown. Real tuning advances vertically — you follow one direction to its end, and switch only when you hit a wall. Here's a typical walkthrough order. It doesn't depend on any specific model, and the criteria and outputs at each step are reusable.

Starting point: a P99 complaint and an odd curve

The symptom is that the P50 of TTFT still looks normal, but the P99 frequently hits the production alert line, while cluster throughput is nowhere near saturated. The first move is to pull a baseline: run one benchmark pass and get a complete snapshot. The data points to two signals — the waiting queue length keeps growing during peak hours (meaning concurrency is insufficient or single requests are too slow), while block pool utilization is only at sixty percent (meaning memory isn't the bottleneck).

Those two signals together already rule out "not enough memory," the most common assumption, and the direction immediately narrows to scheduling and request shape. If you skipped pulling that baseline and went straight to tuning gpu_memory_utilization or adding cards, you'd waste an entire iteration.

Step 1: shave the peak — decoupling long inputs

Breaking traffic down by length revealed that peak input lengths were bimodal: one cluster around 1K (short conversations), another above 15K (document Q&A). The latter was only thirty percent of requests but contributed eighty percent of queueing time.

This round does exactly one thing: confirm chunked prefill is active, and turn the per-step token budget down two notches from the default. The criterion is that the TPOT P99 spike magnitude genuinely dropped, while a throughput re-measurement showed no obvious loss (this is the key part — without re-measuring throughput you can't tell whether you improved anything or just moved the problem). Two benchmark repeats, taking the median, varied by under 3%, so the change was judged effective.

Step 2: reconcile the ledger — find the real constraint

P99 improved after the peak shave but didn't meet target. Back to the ledger: backing out concurrency with the KV formula and comparing against reality showed actual concurrency was only about thirty percent of theoretical. The trail led to max_model_len — it was set to 32K while the business P99.9 was only 19K, and that extra factor of reserved memory pushed block pool utilization up whenever long requests clustered.

After lowering max_model_len to 24K, block pool utilization settled at eighty percent and the queue shrank noticeably. P99 hit target. The lesson of this step: you have to clear both waste on the ledger and uncertainty on the ledger — the former wastes memory, the latter wastes confidence in your concurrency decisions.

Step 3: make the hit rate real

Prefix caching was on by default, but the hit rate was near zero. Auditing the prompt showed a timestamp changing every minute embedded in the system prompt, and it sat at the head — the entire shared prefix broke at the very first block.

After moving the timestamp to the end of the user message, the hit rate jumped to a very high level and the P50 of TTFT fell by far more than expected. What's notable is that this change isn't on the inference side at all, yet it produced the largest payoff of the entire walkthrough. That's the most important point about Prefix Cache engineering: it isn't an engine parameter problem, it's a cross-team interface convention problem.

Step 4: split by shape

The remaining P99 gap concentrated on long-output requests — they fill the batch, which drags short-conversation requests down with them. Doing the shape split at this point is reasonable, because the structural switches (chunked prefill, the ledger, hit rate) are already in place, and both resulting instances have healthy baselines of their own. The short-conversation instance runs small batches with latency first; the long-output instance enables speculative decoding and takes the large-batch route.

Step 5: touch precision last

Before speculative decoding went live, the full three-number validation was run: accepted length P10 was clearly low, showing that long-context extrapolation was hurting draft quality, so it was enabled only on the short-output instance. The long-output instance switched to an n-gram scheme with no extra model, which actually performs better on highly repetitive code tasks like these.

Quantization was evaluated last: weights first, kept only because key metrics on the longest real business inputs showed no significant drop; KV quantization was rejected because the long-context regression showed a drop in retrieval accuracy — in this round it was the only optimization actively given up. Being able to state clearly "which optimization we decided not to do, and why" says more about having understood the system than turning on one more switch does.

Looking back at what the five steps share

Four of the five steps were "quantify first, then change," and only the third was driven by pure engineering intuition (though it too was immediately validated against the baseline). More important is that every step re-established the baseline: if you don't re-measure the baseline after a change, every subsequent judgment rests on a comparison that no longer holds. That's why the rigor of the benchmark methodology itself matters as much as the system being benchmarked.

A Few More Directions Worth Watching

Adaptive Scheduling

Right now every mainstream engine uses fixed scheduling parameters. The ideal shape picks batch size, chunk size, whether to enable speculation, and even whether to take the PD-disaggregation path automatically based on real-time load. The difficulty isn't the algorithm but the fact that these decisions are coupled to each other and all have delayed effects — the payoff from a larger batch size only shows up in metrics tens of seconds later, while the switch itself costs something. Greedy auto-tuning easily oscillates, showing up as periodic jitter in latency metrics, which is worse than fixed parameters.

The pragmatic middle route is "limited automation": automatically adjust only the few parameters with the largest impact and the lowest switching cost (for example, automatically lowering the concurrency cap at low load and raising it at high load), keeping everything else fixed. Such designs have manageable go-live risk, and their payoff is the easiest to explain.

KV Cache Tiering and Prediction

Overflowing KV from GPU to host memory and then to NVMe is the inevitable direction for long-context scenarios. Its core difficulty is selection: which tier to read from depends on whether that prefix will be hit again in the future. If you read the wrong tier, the payoff goes straight to zero and you even lose the transfer time.

Workable heuristics include layering by prefix access frequency, guaranteeing the same tier within a single conversation turn, and demoting to a lower tier rather than evicting when a conversation ends. The correctness of these strategies depends on traffic shape, so they're easy for stable workloads and require conservatism for bursty ones. The core metric is still just one thing — the hit rate, and the effective bandwidth savings it converts into.

Model and System Co-Design

This is the direction I think is most worth watching and most easily overlooked. If the model was trained with an MTP head, or its hidden states are naturally separable for multi-step prediction, then the draft portion of speculative decoding is nearly free and the system side needs no additional investment. Conversely, if the system side knows speculative decoding is coming, training can specifically optimize those heads' acceptance rate.

This changes how you evaluate model selection: from "how big is this model" to "what is the end-to-end cost per token for this model under my load shape". And the latter can only be concluded by folding the inference configuration into the calculation. So the practical advice is: measure the inference-side payoff as part of model selection, and don't compare throughput numbers taken under a default configuration.

The Hardware Boundary Is Moving

Several things are simultaneously changing what optimizations are feasible: new hardware has more memory per card, meaning a larger KV block pool and less sharding; faster interconnects lower the cost of TP communication and PD-disaggregation KV transfer, making larger logical batches viable; and lower-precision format support keeps pushing the lower bound of feasible quantization down.

But every one of those changes requires re-validating the entire set of conclusions above. A larger block pool may mean the old batch size cap is no longer the bottleneck; faster interconnects may turn PD disaggregation from "not worth it" into "worth it"; more aggressive numeric formats may turn long-context precision problems from "unacceptable" into "acceptable." Changing hardware is equivalent to changing machines, and you must walk the tuning order again — that's a hard rule, and you can't carry over intuition from the old configuration.

What Has to Change on the System Side When You Switch Models

Switching models looks like a deployment-side action, but in an inference system it almost always requires a round of re-tuning. The reason is simple: everything concluded above is built on a handful of model properties, and all of them change when you switch models. This section is a dependency checklist, meant to avoid the mistaken judgment of "I don't think we need to tune anything."

First, the order of magnitude of cache footprint. It's jointly determined by layer count, key-value head count, head dimension, and bytes per element. If any one of those four changes, the footprint can shift by an order of magnitude, and concurrency is almost directly inversely proportional to it. A model using a compressed attention structure and a traditional multi-head attention model can differ by tenfold in the concurrency they support on the same memory. The direct consequence: after a model switch, the batch size cap and memory watermark you tuned so carefully are either too conservative and waste capacity, or too aggressive and won't start at all. So the first step after a model switch isn't benchmarking — it's recomputing the ledger with that formula.

Second, the tokenizer. It determines how many tokens the same length of Chinese text gets split into. The same content can differ by tens of percent in token count across tokenizers, which affects cache footprint, prefill compute, output length, and tokenization latency along the whole path simultaneously. More subtly, tokenization efficiency for long digit strings, code, tables, and special symbols can differ enormously; if your business contains such content, the real per-request cost on a new model may be significantly higher than a character-based estimate suggests. After a model switch, measure token counts with real business samples rather than reusing old estimates.

Third, context capability versus the actual optimum. A new model supporting longer context doesn't mean you should set a longer ceiling. Setting it longer only inflates cache reservation and lowers concurrency, for zero benefit. The right approach is to pick a new value based on the business distribution for the new model, covering only the length actually needed even if the model supports a million. If the business genuinely needs longer context, then this is an opportunity for a capacity upgrade — replan the equipment count against the new ledger rather than simply raising the ceiling and seeing whether anything breaks.

Fourth, compatibility between the draft model and the main model. Speculative decoding requires the draft and the main model to share a tokenizer, and ideally to come from the same family. After switching the main model, the previous draft approach basically has to be re-evaluated: the acceptance rate may change substantially (the distribution is different), or it may be outright incompatible (different tokenizer). This one belongs in "re-validate," not "reconfigure" — you must measure accepted length and total throughput rather than carrying over old conclusions.

Fifth, inference-related architecture features. If the new model ships with multi-token prediction heads, the payoff structure of speculative decoding is entirely different: the draft is nearly free, the acceptance rate is higher, and you can enable it more aggressively. If the new model is mixture-of-experts, expert parallelism becomes the mainstream sharding approach, changing both the communication pattern and the weight distribution, and therefore the requirements on device interconnect. If the new model changes the recommended precision format — for instance natively supporting lower-bit weights — then the quantization plan and its payoff ceiling both need re-evaluation.

Sixth, defaults and recommended configuration. Within the same engine family, model authors usually publish recommended parameters in the model card, and those recommended values may differ from the engine's defaults — and that difference is often exactly where the new model's performance characteristics diverge from the old one's. The right approach is to compare the recommended configuration against what's actually in effect, item by item, validating each difference rather than adopting the whole recommended set — recommended configurations target the general case, and your load shape may not be that case.

Taken together, these six give you a more general conclusion: the conclusions of inference system tuning have a shelf life, and the shelf life is determined by how stable the model is. The faster models iterate, the faster baselines expire. That's why the two things I emphasized earlier — "script the baseline" and "record conclusions together with their derivation" — multiply in value under fast iteration: what they determine isn't the efficiency of this tuning round, but how much time it takes to get back to the same service level after every model switch.

A Symptom-Driven Troubleshooting Path

How efficient you are at tuning depends on how fast you can diagnose a problem. The path below organizes the mechanisms I've described into reverse: "see this symptom, check this," and each entry starts from the cause that's easiest to verify and most likely to be the real one.

The symptom is high time-to-first-token with normal time-per-output-token. It means the bottleneck sits before the request produces its first token. Start with the waiting queue length: a long queue means a capacity or scheduling problem; if the queue is short yet TTFT is still high, the individual request is itself slow. Go deeper to the input length distribution and the prefix hit rate: a low hit rate means the prefill compute is the full amount; a high hit rate with still-high TTFT usually means the problem is in the path rather than the engine. Checked in this order, you'll usually localize it within the first two steps.

The symptom is normal TTFT with high time-per-output-token. It means requests get in and output isn't slow, but the gap between tokens is too long. First check whether sequences per batch has passed the throughput inflection point; if it has, batch size and the latency constraint are in conflict and you need to lower it under that constraint. If sequences per batch isn't high and the gaps are still large, focus on per-step time: check whether an overlong prefill step is mixed in (long-request prefill interrupts periodically), whether the sampling stage takes too large a share (long outputs with repetition penalty), and whether multi-card synchronization points are slowing every step. Distinguishing the three takes one profiler with per-step timing.

The symptom is throughput that won't rise, with a short queue and an unfilled block pool. This is the class most easily misdiagnosed, because it looks like "plenty of resources, low demand." In reality it usually means the scheduler isn't feeding the resources fully. Per-step time may be dominated by scheduling overhead (too few sequences per batch, so fixed overhead isn't amortized enough), multi-card communication may account for most of it (every step waiting on the slowest card), or there may be a synchronization point in the path repeatedly draining the compute pipeline. To judge, look at the relationship between per-step time and sequences per batch at low load: if it's linear, you haven't saturated yet and raising the batch size solves it; if it's flat, you're already at the bottleneck and the direction shifts to reducing per-step overhead or improving the parallel split.

The symptom is performance slowly degrading over time. Suspect a resource leak before load changes. Check whether memory footprint and cache watermarks rise monotonically, and whether the relevant resources return to zero after a request finishes. If the curve is step-shaped rather than rising smoothly, it's often tied to a specific request pattern, such as repeated prefix appends and rollbacks. Another common cause is the autotuning cache growing over time, raising lookup and recomputation overhead. The fix isn't manual cleanup but reproducing that request pattern and finding which stage failed to release its resources correctly.

The symptom is periodic performance jitter. First establish the period and what system behavior it aligns with. If it's a fixed minute-level period, suspect internal periodic maintenance, garbage collection, monitoring collection, or log flushing. If it aligns with a business-related period, suspect the traffic's own peaks and troughs triggering scale up and down repeatedly. If it aligns with long requests, suspect long-request prefill steps periodically interrupting short requests — this kind of jitter shows up as regular spikes on the latency curve and is easy to confirm by bucketing input length over time and plotting it.

The symptom is one class of request being especially slow while others are normal. This is usually a shape problem rather than a scheduling problem. First confirm whether the length distribution of that batch deviates significantly from the overall one. If it's long input / short output, look first at the chunked prefill configuration and prefix hit behavior; if it's short input / long output, look first at extra overhead in the sampling stage and whether speculative decoding is on or off; if it's extremely long context, look first at cache watermark and preemption counts; if it's output with structural constraints, look first at candidate pruning and the syntax state machine. Once the shape is confirmed, the remedies are covered in the sections above.

The symptom is performance dropping after an engine upgrade or hardware change. Don't start with parameter tuning for this class; start with "which assumption changed." An engine upgrade may have changed which features default to on, changed default parameters, or changed how the scheduling policy is implemented; a hardware change may have altered the memory-to-compute ratio, the convergence behavior of compile caches, or the cost-effectiveness of the parallel split. The right approach is to put the old and new configs and hardware parameters into one table and compare item by item, identify the differences, and then do targeted validation on the differences rather than blind-tuning from defaults. The first thing after an upgrade should be reproducing the old baseline, confirming base performance hasn't regressed before you talk about new capabilities.

The implicit logic of this path is: rule out the most likely and most easily verified causes first, then work downward layer by layer. The overwhelming majority of troubleshooting time is spent skipping upper-layer verification and going straight to deep parameters — and those changes are often ineffective, while introducing new variables that make later diagnosis harder. Doing the upper-layer checks solidly will usually eliminate half the possibilities within a few minutes.

Conclusion

Looking back at this whole thread, the pattern is quite clear: nearly every inference performance problem is a problem of "resources being distributed unevenly across requests". Static batching makes short requests wait on long ones; Continuous Batching lets a long prefill block decode; paging stops reserving memory for the worst case; Prefix Cache makes repeated computation happen only once; speculative decoding substitutes cheap computation for an expensive serial step; multi-tenant scheduling stops one big customer from monopolizing resources.

And every "better distribution" comes with a new cost: tidiness bought waste, paging bought indirect-addressing overhead, hit reuse bought memory being crowded out, lossless speedup bought a dependency on acceptance rate. The substance of engineering work is finding the optimal balance point among those costs under the current load, and that balance point is not a constant — it drifts with prompt length, output length, concurrency, and context distribution.

So the only reliable methodology is this: measure the percentiles of TTFT and TPOT, plot the throughput-latency Pareto frontier, then move one variable at a time in the order "memory ledger → structural switches → precision → speculative decoding," validating every step with the same benchmark method. Skip any step and what you end up with is a set of numbers that look good under some test script, not a service that can survive in production.

Finally, let me compress this entire article into one thread: the performance problems of inference systems are, in essence, resources being distributed too coarsely across requests. Static batching sets the allocation unit to the whole batch, so short requests get held hostage by long ones; paging cuts the allocation unit from a whole sequence down to small blocks, so memory can finally be used on demand; Prefix Cache makes repeated computation happen only once, so a fixed prompt stops being billed repeatedly; speculative decoding swaps serial steps for one parallel verification, so draft compute gets absorbed in batches; Continuous Batching and chunked prefill cut the scheduling unit from requests down to tokens and blocks, so long jobs no longer block short ones; and preemption and fairness decide who absorbs the loss when resources run short.

From that thread you can derive one judgment criterion that runs through the whole article: any optimization that makes distribution finer-grained without resolving who it's being distributed to can bite back under some load. Paging makes distribution finer-grained, but without eviction headroom the refinement itself brings spikes; speculative decoding makes computation cheaper, but once load is already compute-saturated the extra computation is pure overhead; Prefix Cache reuses computation, but the reused blocks occupy memory too, so at very high hit rates independent concurrency can actually drop. Every tension between precision and latency can be explained by this one rule.

A direct recommendation for readers in different situations. If this is your first build, getting the order right matters far more than getting the parameters right — first make sure Continuous Batching and chunked prefill are enabled, that context length is set from the real distribution, and that Prefix Cache has monitoring, then talk about everything else. Get that foundation right and every later optimization has room to work; get it wrong and the more you tune, the messier it gets. If your service has been running for a while and the numbers don't look good, the most effective single step is to build a baseline: pull together the high percentiles of TTFT and TPOT, throughput, preemption counts, cache watermark, and hit rate, and see which metric is anomalous. In most cases the answer points directly at one of two places — the memory ledger or the prompt structure — and those happen to be where the change is smallest and the payoff largest.

If you're building engines rather than using them, the most useful material here is actually the set of inverse constraints: a batch size curve has two inflection points rather than one; only the forward pass is a necessary cost within a scheduler step; the sampling stage grows linearly more expensive with length on long outputs; and evicting too late causes spikes. That's a list of "under what conditions does this optimization fail," and it's longer and more useful than a list of "which optimizations work" — because things that work will be told to you by others too, while the boundaries that don't fail can only be found by stepping on them yourself.

One last word on how to learn this field. It changes fast: engine versions, hardware capabilities, even the optimal batch size strategy all shift, so specific conclusions have a short half-life. But mechanisms and formulas have a long half-life — the logic of bandwidth amortization doesn't stop working when you change hardware generations, the chained structure of prefix hashing doesn't change when you change engines, and the fact that the first percentile can't be replaced by the mean certainly won't go out of date. So the test for whether a piece of technical writing is worth reading is whether it explains "why it's like this, and under what conditions it stops holding". Numbers without conditions have a one-time reference value; a clear mechanism can be carried into an environment completely unlike your own.