Long-Context Architecture: RoPE Extrapolation, Sparse Attention, and KV Cache

1. Splitting "Nominal Length" from "Effective Length"

Every time someone says a model "supports a one-million-token context," strictly speaking they are describing capacity, not capability. Supporting a million tokens requires three mutually independent things to hold at once: first, the inference framework is willing to allocate enough KV Cache space for that sequence; second, the positional encoding does not collapse numerically across a million positions, and the attention scores do not flatten into a mush; third, the model's attention distribution itself still carries useful information at that length. All three holding at a million does not mean the model can actually pull a number out of the 800,000th token.

So the more practically useful framing in engineering is to split context into two quantities: nominal length (nominal length) is the upper bound that memory and positional encoding allow, meaning "can it fit"; effective length (effective length) is the length at which the model's accuracy on a given task has not yet visibly collapsed, meaning "can it be used well." The gap between these two numbers, on open-weight models after 2024, is routinely several times to several dozen times, and it is not a constant: it varies with task type, with prompt structure, and even with the way you phrase the question.

1.1 The three most common illusions in long-context evaluation

The first illusion is treating needle in a haystack (needle in a haystack) as a proxy metric for long-context capability. The test inserts a piece of irrelevant information somewhere in a long text and checks whether the model can repeat it verbatim. It tests essentially one thing: whether the localization ability of retrieval-style attention is still intact. It cannot measure multi-hop reasoning, cross-paragraph alignment, precise counting, or tasks that genuinely consume length, such as "synthesize ten scattered pieces of information to answer one question." Worse, its score depends heavily on the distraction strength of the haystack itself — if the background text is semantically highly related, difficulty spikes; if the background is a set of unrelated random encyclopedia paragraphs, difficulty is much lower. The numbers across different implementations simply are not comparable.

The second illusion is treating the U-shaped curve observed in Lost in the Middle as a bug to be fixed. The U-shape (information at the beginning and at the end is easier to use than information in the middle) is a property shared by nearly every model that uses relative or rotary positional encoding, and it reflects the physical fact that attention decays monotonically with distance: the farther away a token is, the more its attention score is crushed after normalization. This is not a defect that can be tuned away; it is the intrinsic cost of the attention mechanism. The engineering response is not to eliminate it but to change the path by which information reaches the attention layer — rearranging and duplicating important content, or building summaries at closer range.

The third illusion is comparing window size as the single dimension of model capability. Two models both at 128K: one is close to saturation on long-document summarization, the other degrades noticeably after 4K but can still run the full 128K — and the latter is the norm. When a framework stretches the window to 1M, the most common outcome is not a crash but "quietly getting dumber": it can still repeat what you put inside, but for a question that requires three pieces of information, it starts using only the first one.

1.2 Effective length is task-dependent, even instruction-dependent

The genuinely useful question is not "what is the effective length?" but "where does the curve start dropping, on my distribution of tasks?" The method is crude but effective: fix everything else, stretch sample length proportionally, plot an accuracy curve, and look at when the slope turns negative. If you cannot run an evaluation, at least do one manual stratified test — bucket the same class of questions into 4K, 16K, 64K, and 256K context lengths, run each one, and eyeball the inflection point.

There is one more easily overlooked dimension: the effective length of the same model shifts with how the prompt is organized. Clustering relevant information at the beginning and end of the prompt, ordering retrieved passages by relevance to the question, and giving the model an explicit "locate first, then answer" instruction all change no architectural parameter whatsoever, yet they can move that inflection curve significantly. In other words, inside a long-context architecture, prompt engineering and data organization are not "remedies outside the model" but part of the architectural capability itself — because they directly change what the attention sparsity pattern should look like.

2. Positional encoding: the extrapolation dilemma of RoPE and the interpolation family

Pushing the context window from tens of thousands to hundreds of thousands, the first hard threshold you hit is positional encoding. Most modern models use rotary positional encoding (RoPE), which encodes position by applying position-dependent rotations to the query and key vectors, and it naturally carries a relative-position preference. The problem is that its frequencies are determined by the maximum length seen during training: low-dimensional components correspond to high-frequency positions, decaying fast with a short period; high-dimensional components correspond to low-frequency positions with long periods that barely change with position. When you push inference positions beyond the training length, the high-frequency components rotate at step sizes the model has never seen, producing aliasing in the effective phase, and attention scores lose their meaning.

2.1 Why direct extrapolation crashes instead of degrading slowly

RoPE's extrapolation failure has one characteristic that is often explained wrongly: it is not a smooth degradation but a clear cliff. The model's attention entropy collapses suddenly — a large amount of attention mass concentrates on a very few tokens, usually the first few positions plus a small recent window, and the middle tokens get almost no attention. In perplexity terms this looks like the second half of a long document losing all language-modeling ability, with generation starting to repeat the opening.

The cause can be understood on two levels. From the frequency angle: the period of a high-dimensional rotary vector is 2π, and extrapolating it to a position N times the training length means the phase has already wound around several times — the model was never trained on that phase interval. From the attention-scale angle: the inner product between query and key produces anomalously large absolute values because of untrained high-frequency oscillation, and after the scaling factor and softmax the distribution becomes extremely peaked. The choice of frequency dimensions in the original RoPE paper (different dimensions use different wavelengths, from 2π to 10000·2π) was designed for the training length; under extrapolation that distributional assumption is completely broken.

There is one more easily missed point: RoPE's extrapolation ability is coupled with attention sparsity. Many "extrapolation failure" cases are actually two layers of problems stacked on top of each other — positional encoding scrambling the scores is one side, while full attention on long sequences was never able to allocate scores correctly in the first place. Even if you solve positional encoding perfectly, if the pretraining corpus does not contain enough long-sequence samples, a model past 100K will still have learned only the degenerate pattern of "look locally + look hard at the beginning." This is why so many models claiming 1M support drop off quickly past 256K.

2.2 The interpolation family: PI, NTK-aware, YaRN

The most direct idea is to compress positions. Position Interpolation divides the position index by a scaling factor s, compressing all positions back into the range seen during training. The high-frequency components then change phase more slowly, and the distribution the model has seen reappears. The cost is very direct: the position difference between adjacent tokens goes from 1 to 1/s, the model's ability to distinguish adjacent tokens is diluted, and attention resolution at short range drops. In practice this often shows up as "knows roughly what it means, but cannot recall the details."

NTK-aware scaling offers a smarter compromise. It does not compress all frequencies uniformly; instead it compresses high-frequency dimensions aggressively while leaving low-frequency dimensions almost untouched or only slightly compressed, on the reasoning that high-frequency dimensions are exactly where phase aliasing under extrapolation is worst, while low-frequency dimensions already look like constants and do not misbehave. The scaling is achieved by changing RoPE's base frequency θ: θ goes from 10000 to forms like 10000·s^(d/(d−2)), which lengthens the wavelength of high-frequency dimensions while leaving low-frequency wavelengths essentially unchanged. It preserves more local resolution than naive interpolation, and it is the default baseline for a lot of follow-up work.

YaRN pushes this further into a combination of per-band mixing and attention temperature compensation. It splits dimensions into three classes by wavelength: extremely short wavelengths use interpolation directly (these dimensions carry the most local information, so compressing them has the smallest effect and partially cancels the attention temperature); medium and long wavelengths are not interpolated (they are stable to begin with); the longest wavelength band is linearly interpolated from a training length of 1 up to the target length. YaRN also points out that interpolation inflates attention entropy overall, so a temperature factor of 1/s (its reciprocal) must be applied to query vectors before softmax to counteract it. This combination shows very good stability at the 128K scale and is one of the most commonly used RoPE extrapolation schemes in the open-source community.

The consensus after YaRN is this: frequency scaling mainly solves "not crashing," not "using it well." It keeps the model numerically stable and linguistically capable at extremely long positions, but it injects no inductive bias for "retrieving information from far away." To actually make good use of content beyond 500K, you usually need to combine it with the attention sparsity techniques described later, or with long-sequence training data.

2.3 LongRoPE: progressive extension and non-uniform band allocation

LongRoPE takes a different route: first solve "how far can we train," then solve "how far can we infer." Its core observation is that if the training sequence itself is at 2M length (even if most of those positions are masked out), position extrapolation degenerates into an ordinary "train on more positions" problem, and frequency scaling is no longer a necessity — scaling is merely a transitional device that keeps the pretraining weights from collapsing during the initial stage, and it can be removed once long-sequence fine-tuning is finished.

LongRoPE's second observation is that different dimensions have different sensitivity to length, and therefore should get different scaling rates. It allocates dimensions with a binary-search-like procedure: some dimensions lose little on short sequences and a lot on long sequences, so they get the largest scaling factor; others behave the opposite way and get the smallest factor, or even no scaling at all. The resulting scaling vector is piecewise across the entire dimension space rather than a single uniform coefficient. Combined with a progressive-extension fine-tuning schedule (lengthening training from short to long, stage by stage), the paper reports pushing context to the two-million scale on LLaMA-family models while keeping performance close to the original model on tasks such as passkey retrieval that require precise long-range localization.

A number of variants have appeared since 2025, and the ideas worth watching include jointly training position extrapolation with attention sparsity (letting the model learn the sparse pattern and the extrapolation at the same time), and using something other than RoPE — for example mixing rotary positions with learnable absolute positions, or cutting the sequence into blocks with an independent coarse-grained position channel between them (interleaved-style schemes; Llama 4's iRoPE interleaves global and local positional encodings so that two relative distance scales coexist in the same layer). What these approaches share is the admission that a single frequency distribution cannot simultaneously serve both "fine discrimination between adjacent tokens" and "a unique identity across millions of positions."

2.4 Beyond RoPE: are other positional modeling routes worth it

RoPE is not the only option, but from an engineering standpoint it is still the best answer, so it is worth briefly explaining where the other routes hurt. Linear attention biases (ALiBi-style) add distance directly as a linear bias on the attention score and introduce no position vector at all, so there is naturally no extrapolation problem. But the cost shows up on very long sequences: a linear bias has very low discriminative power at long range, and past some distance the score difference between positions becomes negligible; it also needs extra engineering to cooperate with the various KV Cache compression and paging techniques, because in block attention the cross-block bias terms have to be handled correctly, which is far messier than with RoPE.

Learnable relative position biases discretize distance into a number of buckets, with one learnable scalar per bucket. They are more expressive than ALiBi, but the bucket count is finite, so buckets never seen during extrapolation have to be extrapolated or truncated — the RoPE dilemma simply reappears in another form. The route of no positional encoding at all plus a recurrence mechanism (the Transformer-XL lineage) hands "position" to the structure of the cache rather than to the token itself, so the cache can be extended arbitrarily at decode time, but its training-time resolution is bounded by the training segment length, and in modern batched inference frameworks cache management is far more complex than the positional encoding itself.

The genuinely interesting fourth category is handing the responsibility of positional modeling to data structures and prompt organization. Adding a table of contents and section numbering to long documents, a file-path tree and symbol table to long code, rolling-summary anchors to long conversations — this kind of practice has proven extremely effective over the past two years, and it does not touch model weights at all. It is complementary to positional encoding: encoding answers "which position in the sequence does this token occupy," while structure answers "where does this token sit in the semantic hierarchy of the whole material." For tasks like document understanding, the latter often carries more information than the former.

Taken together, RoPE became the de facto standard because it is one of very few schemes that satisfies three conditions at once: it can express the uniqueness of absolute position, it can naturally express relative distance, and it is fully compatible with the mainstream ecosystem of fused attention kernels, KV quantization, and paged memory management. Other schemes are more elegant along some dimension, but the engineering cost is too high.

2.4 Code: a directly usable RoPE frequency scaling implementation

Below is the core of an NTK-aware scaling implementation with the model-specific branches stripped out, so it can be dropped straight into an existing RoPE computation path. Note one easy trap: the scaled frequencies must stay consistent with the position_ids actually being used. If some layers use scaling and others do not, that has to be declared explicitly in the config rather than relying on a default.

import torch
import torch.nn as nn


def compute_rope_freqs(dim: int, base: float, scale: float) -> torch.Tensor:
    '''NTK-aware 频率缩放：只拉长高频维度的波长，低频维度基本不变。
    dim 必须为偶数，且与 head_dim 一致。
    '''
    assert dim % 2 == 0, 'head_dim 必须是偶数'
    # 原始逆频率：exp(-log(base) * 2i / dim)
    inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float() / dim))
    # 关键：把 base 抬高，高频分量（inv_freq 大）的波长被拉长更多
    new_base = base * (scale ** (dim / (dim - 2)))
    inv_freq_scaled = 1.0 / (new_base ** (torch.arange(0, dim, 2).float() / dim))
    return inv_freq_scaled


class ScaledRotaryEmbedding(nn.Module):
    def __init__(self, dim: int, base: float = 10000.0, scale: float = 32.0):
        super().__init__()
        self.dim = dim
        self.scale = scale
        inv_freq = compute_rope_freqs(dim, base, scale)
        self.register_buffer('inv_freq', inv_freq, persistent=False)
        self._cached_len = 0

    def forward(self, x: torch.Tensor, offset: int = 0) -> torch.Tensor:
        '''x: (batch, seq, dim)。返回同形状的 cos / sin，供 rotate_half 使用。'''
        seq = x.shape[1]
        if seq + offset > self._cached_len:
            t = torch.arange(offset, offset + seq, device=x.device, dtype=torch.float32)
            freqs = torch.outer(t, self.inv_freq.to(x.device))
            emb = torch.cat((freqs, freqs), dim=-1)
            self._cache_cos = emb.cos()[None, None, :, :]
            self._cache_sin = emb.sin()[None, None, :, :]
            self._cached_len = offset + seq
        sl = slice(offset, offset + seq)
        return self._cache_cos[:, :, sl, :], self._cache_sin[:, :, sl, :]


# 在注意力里使用：
#   cos, sin = rope(q)
#   q_embed = q * cos + rotate_half(q) * sin
#   k_embed = k * cos + rotate_half(k) * sin
# 注意 attention score 的缩放 1/sqrt(d) 不变，YaRN 那类方案还要额外
# 对 q 做 1/(t * s) 的温度补偿（t 是缩放后的注意力温度）。

3. Attention computation itself: quadratic complexity and two memory walls

With positional encoding handled, the next problem is harder: it does not compute, and it does not fit. There are two independent walls here, and conflating them leads to picking the wrong solution.

3.1 The first wall: the quadratic term in computation

Attention scoring and weighting costs O(N²·d) in complexity. Going from 128K to 1M in N multiplies the quadratic term by roughly 58x. That factor looks moderate, but it stacks on top of the "per-token linear term" in a way that turns ugly: the model parameter count sets a floor on FLOPs (roughly 2·parameters·tokens), and that part is linear; while the attention term becomes dominant at large N. For a 7B model, the prefill compute floor at 1M tokens is on the order of 1e16 FLOPs — even at tens of TFLOPS of effective throughput on a single high-end card, that is tens of seconds. Note this is only the theoretical floor; memory access and scheduling still have to be accounted for. For larger models, a 1M-token prefill going from "acceptable" to "minutes" is the norm.

This is why, over the past two years, attention sparsity has moved from "optional optimization" to "default path" in every inference framework that does long context.

3.2 The second wall: the memory peak of the N×N attention matrix

The second wall is memory, and its shape is more counterintuitive than the compute cost. A naive implementation has to materialize an N×N attention matrix in memory (plus the intermediate results of softmax), which is O(N²) in memory. At N = 1M that matrix has 1e12 elements, and even at one byte per element it is already in the terabyte range — completely impossible. This is exactly where fused kernels like FlashAttention earn their keep: through tiled computation and online softmax they eliminate the N×N matrix from being materialized at all, dropping memory to O(N) while leaving the asymptotic complexity bound unchanged. It solves the memory wall, not the compute wall — not a single attention FLOP for 1M tokens is saved.

So for long context these two things have to be accounted for separately: FlashAttention and its later variants (attention variants that page the KV Cache, kernels that support sparse blocks, fused dequantization kernels that support low-bit KV) bring the "memory wall" down from terabytes to gigabytes; only attention sparsity genuinely lowers the "compute wall." Plenty of teams do only the former on a long-context project and then find that prefill time still grows quadratically with length, so the user experience is still unacceptable.

3.3 MLA: reducing KV dimensionality from a different direction

Apart from compressing after the fact, the Multi-head Latent Attention used by the DeepSeek family takes the route of "never have that many KV heads in the first place." The idea is: compress each token's K and V with a shared low-rank projection into a small latent vector stored in the Cache, then at attention time use the small set of components the token carries to project the latent vector back up into each head's K/V space. Because all heads share the same latent vector, the KV Cache dimensionality drops from "num_heads × head_dim × 2" to "latent dim + the small components needed for the up-projection." The latent dimension is typically a few hundred in practice, which is tens of times smaller than the tens of thousands of dimensions of a traditional GQA configuration.

MLA has two costs, both worth stating clearly. First, the up-projection needs the token's own K components, which means that even when attention only visits a few positions, the latent vectors of the visited tokens still have to be projected back — extra computational overhead. Second, MLA and block-managed-KV schemes like PagedAttention pull against each other in engineering terms: PagedAttention requires a regular KV layout that can be copied by page, while MLA's latent layout favors whole-block reads and writes. Fortunately the later absorption (absorb) variants solved part of this problem — absorbing the up-projection matrices ahead of time into the query and output projections, so that at inference time the latent vector never has to be decoded and attention can happen directly in the latent space. That is a genuinely valuable engineering detail for long-context deployment.

Comparing these two families of approaches together yields a more complete judgment: GQA/MQA reduces the number of KV heads, MLA reduces the dimensionality of KV, quantization reduces the bit width of KV, tiered storage reduces the placement level of KV, sparsification reduces the amount of KV, and eviction policies reduce the amount of resident KV. The six levers are orthogonal, and a long-context system typically needs three or four of them at once.

3.4 Laying out the memory budget for the entire path

The memory budget of a long-context project cannot look only at the KV Cache line item. On an 80GB card, what actually has to fit includes: model weights (parameters times two bytes if unquantized), the KV Cache (inflating linearly with concurrency and context length), activations (a transient peak during prefill, proportional to sequence length but with a small constant), and the framework's own fragmentation and reservations. If you skip this accounting, it is easy to end up in the situation where "the KV math says it just barely fits, and then it OOMs on the first run."

The right way to lay out a budget table has five columns: scenario (typical concurrency and context length), weight footprint, KV footprint, activation peak, and fragmentation reserve — then see which column is the bottleneck. My rule of thumb is that under a GQA configuration, once context exceeds 32K the KV Cache rapidly becomes the largest column; on older architectures without GQA it is already the bottleneck at 8K. This also explains why dimensionality reduction at the architecture level (GQA, MQA, MLA) is worth prioritizing over any compression technique at the inference layer — it shrinks that column at the source, whereas compression patches the result afterwards.

Another easily overlooked column is the activation peak. The transient activations during prefill are not large, but they are positively correlated with the number of sequences in the batch: under chunked prefill the peak depends only on the chunk size; if you do not chunk at all and instead try to process an entire long prompt in one shot, the activation peak ties directly to sequence length, and in some implementations it can even exceed the KV Cache itself. This is the second reason for chunked prefill beyond scheduling — it is simultaneously a memory-protection measure.

4. Attention sparsification: cutting the quadratic term at the root

The full spectrum of attention sparsification, ranked by how strong the structural assumption is, splits into four tiers: fixed sparsity (sliding window attention), structured sparsity (human-declared important positions, for example vertical-slash patterns), retrieval-based sparsity (a cheap approximate operator dynamically selects the important positions), and learned sparsity (a small model predicts which blocks matter). The further back you go the better the results, but the higher the engineering complexity and the risk of silent failure.

4.1 Sliding window attention and the attention sink in StreamingLLM

The simplest sparsification is sliding window attention: each token only sees the most recent W tokens. This drops attention from O(N²) to O(N·W); with W constant, complexity becomes linear. But naive sliding window attention has one fatal problem: the model degrades. The reason is that training used causal full attention, and in addition to the local pattern, the model's attention distribution contains one very important global pattern — the attention sink: a large amount of attention mass flows unconditionally toward a few special tokens at the start of the sequence, acting as an "information-free trash can" that gives the softmax numerator and denominator a stable outlet.

The key contribution of StreamingLLM is turning that observation into a design principle: keeping only a handful of sink tokens at the beginning plus a recent window is enough to maintain stable perplexity under an extremely small KV Cache budget. The paper reports maintaining stability out to the million-token range under a budget of a few sink tokens plus only the most recent two or three thousand tokens. In other words, "unimportant positions" are not uniformly distributed — they cluster heavily at those first few positions. Treating the beginning as a trash can both matches behavior the model has already learned and eliminates the entire middle span.

The value of this trick is that it is extremely cheap: it needs no training and no indexer, a pure attention-mask change takes effect, and the loss is almost nil. It is the foundation under every long-context scheme. But its ceiling is equally clear: the sliding window cannot see explicit content far away, and the sink cannot store any semantics. StreamingLLM solves "not crashing," not "remembering the part before 300K."

4.2 Structured sparsity: MInference and the vertical-slash hypothesis

If a model was exposed during training to the inductive bias that "certain specific positional patterns matter more," you can declare a sparse topology by hand. What MInference does is sample attention maps with a small number of long-context examples and observe several highly stable patterns: the vertical column pattern (the current query concentrates its attention on certain specific key positions that run through the entire sequence, essentially a historical sink), the diagonal pattern (along the local diagonal, a typical sliding window), the vertical-slash pattern (segmented diagonals covering a much longer range), and the within-block pattern. Almost every head belongs to exactly one of these classes.

Sparsification then becomes "assign each head one pattern plus a span parameter," and attention is computed per pattern at inference time. This scheme has been repeatedly validated at the million-token scale by both the authors and the community, and it is one of the representative approaches on the "structured sparsity with no learning overhead" track: it needs no training, deployment is just a mask change, and the accuracy loss is usually small on long-document understanding tasks. Its limitation is equally clear — the patterns are predetermined, so if the actual importance distribution of the input badly mismatches the preset pattern (say, a piece of extremely regular code versus a screenplay with alternating character dialogue), the preset topology will either waste budget or miss critical positions.

4.3 Retrieval-based and learned sparsity: letting the model decide where to look

The retrieval-based idea is: first use the first few dimensions of each query (sometimes just one dimension) to compute a coarse inner product against all keys, select the top-k candidates, and only then run full attention on the candidates. The coarse pass costs O(N·d_kv of a small projection), full attention runs only over k positions, and the overall complexity is approximately O(N·k + N). The essence here is "replacing expensive exact full-dimension matching with a low-dimensional approximate high-recall index" — the same idea as IVF/HNSW in vector search: trade a little recall loss for more than a tenfold speedup.

Learned sparsity instead hands the question of "which ones to pick" to a small model. SpargeAttn uses a lightweight token selector to predict an importance score for each query against each KV block, runs sparse attention only over high-scoring blocks, and compensates the attention of unselected blocks with an estimate so that softmax normalization does not break. SeerAttention uses a very small gating network (roughly a million parameters per layer) to decide the attention budget layer by layer and predict block-level importance, converging progressively — early layers look at very few blocks, later layers progressively open up, which significantly lowers total overhead. DSA, introduced by DeepSeek in V3.2, follows the same idea: a lightweight indexer computes top-k candidate positions for each query, combined with a sliding window to guarantee local information is not lost, and the official figure given is that attention-related cost on long context is brought down to roughly one third.

The biggest advantage of retrieval-based and learned sparsity is input adaptivity: under the same budget, the attention distribution of a code task and a dialogue task are completely different, and dynamic selection adapts automatically. But there are two risks you have to face. The first is recall failure: if the coarse pass misses the truly critical position, that attention result is simply wrong, and you cannot tell from the output. So these schemes usually keep a sliding window or sink as a safety net, guaranteeing that local and opening positions are always inside the budget. The second is stability of the coarse pass: the top-k of a low-dimensional inner product is numerically sensitive to quantization error and to fp8/fp16 precision, so the selections can jitter between batches, which in a dialogue setting shows up as unstable output.

4.4 Budget per head: different heads need wildly different amounts

DuoAttention offers a highly instructive perspective: not every attention head needs the same amount of KV Cache. Experiments show that most heads are only used within a very short window (local heads), while a few heads need to access the entire context (retrieval heads). If you allocate the budget evenly across all heads, you are effectively overpaying for the few retrieval heads and wasting on the many local heads. A more reasonable approach is to first analyze each head's KV usage distribution offline (run a pass over long-context samples and measure the window size within which each head already covers 95% of the attention mass), give local heads a small budget and retrieval heads a large one, and then train a lightweight classifier that decides at inference time which category each new head falls into. The figure reported in the paper is that KV Cache size drops to roughly 40% with essentially unchanged accuracy.

The value of this idea goes beyond saving memory — it is really an architectural view of "layering heads by attention behavior," the same methodology seen from a different angle as MLA's "dimensionality reduction by dimension" and sparsification's "budget allocation by layer": different layers, different heads, and different dimensions have wildly different sensitivity to long context, and uniform resource allocation is the single largest source of waste. Several models released in 2026 (Kimi Linear, the MiniMax family, Gemma 3n, and so on) all describe similar layered hybrid attention designs: some layers use linear attention or a compressed state space, others use a compressed sliding window attention, bringing KV requirements down with a fixed compression ratio, and the official figures all land in the 5:1 to 7:1 range. The shared logic is simple: acknowledge that the overwhelming majority of layers in a long-context model do not need full attention.

4.5 Layer-wise sparsity ratios: how should the budget actually be split

Once sparsification lands in engineering practice, the question it finally has to answer is budget allocation: the total budget is fixed, so how do you split it across layers? There are three rules of thumb that have proven fairly reliable. First, narrow for shallow layers, wide for deep layers. Shallow attention heads mainly perform lexical and local syntactic matching and need a very small context window; deep layers do abstraction and retrieval and need to aggregate information across spans. One counterintuitive but important phenomenon is that deep layers often need both a wider window and stronger cross-span retrieval, while shallow layers do almost purely local work — this is exactly why layered attention architectures proliferated after 2025. Second, do not let sinks and sliding windows eat the entire budget. Keeping 4 to 16 sink tokens is usually enough, and on many tasks a sliding window of 512 to 2048 already captures the bulk of the quality gain; the two together should be kept under one third of the total budget, leaving the rest for dynamic retrieval. Third, switch the sparsity configuration based on input type. Code, dialogue, and long-document summarization want very different sparse topologies, so if your service can identify the input type, two presets beat one compromise configuration.

The budget itself should be determined empirically: start with a loose configuration (say, a sliding window plus sinks only) as the baseline, then progressively increase the dynamic retrieval budget, plot a "budget vs. downstream task score" curve, and find where the slope visibly flattens. Empirically this inflection point usually arrives earlier than people expect — once the total budget reaches 5% to 10% of the training length, the gains often already capture more than 80% of the benefit; beyond that the marginal return decays fast while memory and latency cost keep growing linearly.

4.6 Code: a minimal implementation of retrieval-based sparse attention

The implementation below demonstrates the core logic of retrieval-based sparsity: use a single projected dimension for the coarse pass, pick the top-k KV positions, then run full attention on the selected positions. It does no flash or paging optimization at all, but the structure is complete and can be run directly on a small model to verify how sparsity affects accuracy.

import torch
import torch.nn.functional as F


def sparse_topk_attention(q, k, v, sink=4, window=256, k_ratio=0.02):
    '''q: (B, H, N, D)  k/v: (B, H_kv, N, D)，GQA 下用 repeat_kv 展开
    返回与全注意力同形状的输出。稀疏预算 = sink + window + topk。
    '''
    B, H, N, D = q.shape
    k = k.repeat_interleave(H // k.shape[1], dim=1)
    v = v.repeat_interleave(H // v.shape[1], dim=1)

    # 预算分配：固定部分（sink + 局部窗口）永远保留
    fixed = sink + window
    # 动态部分：用 q 的第一维做粗筛，取 topk 个最相关的 key 位置
    coarse = torch.einsum('bhid,bhjd->bhij', q[..., :1], k[..., :1])  # (B,H,N,N)
    topk = max(1, int(N * k_ratio))
    topk = min(topk, N - fixed)
    _, idx = torch.topk(coarse, topk, dim=-1)  # 每个 query 各选自己的位置

    # 组装稀疏索引集合：[0:sink] ∪ [N-window:N] ∪ topk 命中
    dev = q.device
    s = torch.arange(sink, device=dev).expand(B, H, N, sink)
    w = (torch.arange(N - window, N, device=dev)
         .view(1, 1, 1, window).expand(B, H, N, window))
    pos = torch.cat([s, w, idx], dim=-1)                       # (B,H,N,K)
    K = pos.shape[-1]

    # gather 出被选中的 k/v，并重算精确注意力分数
    k_sel = torch.gather(
        k.unsqueeze(2).expand(B, H, N, H, D), 3,
        pos.unsqueeze(-1).expand(B, H, N, K, D))               # (B,H,N,K,D)
    v_sel = v.unsqueeze(2).expand(B, H, N, H, D).gather(
        3, pos.unsqueeze(-1).expand(B, H, N, K, D))

    scores = torch.einsum('bhnid,bhnkd->bhnik', q, k_sel) / (D ** 0.5)
    out = torch.einsum('bhnik,bhnkd->bhnid', scores.softmax(-1), v_sel)
    return out


# 要点：
# 1) coarse 矩阵虽然是 (B,H,N,N)，但它的乘加深度只有 1 维，
#    在真实实现里应该直接对 K 做投影降维后再算，永远不要物化 N×N 的 fp32。
# 2) 真实实现要用 gather_kernel / paged KV 来做，避免上面的 expand 显存爆炸。
# 3) 生产上通常和滑窗、sink 融合成一个固定模式 + 动态模式的混合预算表。

5. The full KV Cache lifecycle: do the accounting before talking about compression

KV Cache optimization is the most tedious but also the most reliably profitable part of long-context engineering. Its lifecycle roughly looks like this: write (produced during prefill) → residency (read repeatedly during decode) → compression/migration (quantization, tiering, swap-out) → eviction (discarding part of the context when it exceeds the budget). The four stages correspond to four completely different families of techniques.

5.1 Memory estimation: get the numbers straight before choosing an approach

There is a very simple formula for the size of a KV Cache in bytes: 字节数 = 2 × 层数 × KV头数 × 头维 × dtype字节数 × 序列长度 (the leading 2 accounts for one copy of K and one of V). Run this formula on a few real configurations and the conclusion is usually uncomfortable.

• A 7B model with 32 layers, 32 KV heads, head_dim 128, fp16: about 512 KiB per token. A single 128K-context sequence is 64 GiB, already past the capacity of most single cards.
• A 70B model with 80 layers, GQA down to 8 KV heads, head_dim 128, fp16: about 320 KiB per token. A single 128K-context sequence is exactly 40 GiB. And this is still the raw configuration with no compression whatsoever.
• Push the second line above to 1M context: 320 GiB for a single sequence. Even if an entire 80GB card were devoted to KV, it would hold only a quarter of one sequence.

From this ledger you can directly derive three engineering conclusions, which are also the three basic routes for a long-context system: the first is reducing dimensionality (GQA/MQA/MLA, cutting the number of KV heads or their dimensions at the source); the second is reducing bit width (quantizing to 8bit/4bit/2bit, where halving the width is a linear win); the third is reducing amount (compression/eviction/sparsification, keeping the KV of only a subset of tokens). These three routes are independent of each other and also compete with each other; the priority depends on whether your bottleneck is memory capacity, memory bandwidth, or compute.

5.2 Quantization: halving the bit width is the cheapest win

The idea behind KV quantization is similar to weight quantization, but with one key difference: KV quantization error gets amplified by the attention softmax on long sequences. Weight error is fixed, while KV error depends on how many tokens participate in the normalization of the same softmax — the longer the sequence, the more positions compete, and the more easily a key value distorted by quantization error grabs a disproportionate share of attention mass after normalization. This is why early KV quantization schemes showed "no loss on short tasks, obvious degradation on long ones."

The key observation from KIVI is that Keys and Values should use different quantization strategies. Outliers in key vectors concentrate in a few specific channels (because keys carry sharp information about position and direction), so Keys are suited to per-channel quantization; outliers in value vectors concentrate in a few specific tokens (a few tokens carry most of the information), so Values are suited to per-token quantization. This "one horizontal, one vertical" allocation is clearly better than a uniform strategy at the same bit width. Later versions push the width down to 6bit and introduce grouped quantization weighted by attention-output importance, and the officially reported figure is compressing the KV Cache to roughly 40% with essentially no loss.

There are two things you must watch out for in engineering. First, quantization must be grouped per head, not unified across the model: the KV distributions differ a lot across heads and layers, and a single unified scale causes high-variance heads to collapse in accuracy. Second, dequantization must be fused into the attention kernel: if every decode step dequantizes the entire KV Cache back to fp16 before computing attention, you hand the memory you just saved straight back to bandwidth and the benefit nets out to zero. A real kernel has to complete the dequantization and the dot product while reading the KV, ideally using int8/fp16 mixed-precision compute units directly.

5.3 Tiered storage and swap-out: treat KV as data, not as memory

A KV Cache is essentially "a slice of vector per token," so it can perfectly well live in CPU memory, on SSD, or even in object storage, and be swapped in on demand. The history of this route goes back to very early CPU-offload inference schemes, where the approach was to offload all layers and swap tokens in and out one at a time, saturating the bandwidth entirely over PCIe and yielding throughput so low it was unusable.

Later tiered schemes made two key improvements in thinking. First, swap out only what is rarely used: the KV produced by prefill for the most recently generated tokens is accessed at every subsequent step and must stay in VRAM; while very old historical tokens only need to be accessed once during generation (when producing the next token), so swapping them out has a relatively small throughput impact. Schemes like KTransformers are built exactly on this observation — they keep the KV Cache in CPU memory and reduce PCIe round-trips through an optimized sparse per-token access path; the key is not "offload everything" but "offload only the deep layers and touch only the sparse layers."

Second, layered attention: schemes like ShadowKV keep a complete but very small sliding-window KV on the lower layers (the most recent few hundred tokens), and on the higher layers keep only a minimal set of "shadow tokens" (a small handful of historical tokens picked out by an importance score). Doing attention on the deep layers then reads very little KV, bringing the PCIe bandwidth requirement down to roughly the level of a GPU-local memory scheme. LayerCache goes the other direction: reusing the same KV across layers, storing only one copy of what adjacent layers share, which saves a lot on models with highly redundant KV.

The biggest problem with the swap-out route is latency jitter. SSD random read latency sits in the tens to hundreds of microseconds, and once a given token needs to be swapped in multiple times, the tail latency becomes truly ugly. So this route suits offline batch processing or latency-insensitive scenarios better; for online-facing services, the mainstream choice remains the combination of quantization + sparsity + paging.

5.4 Eviction and compression: who exactly should be thrown away

When the context exceeds the budget, part of the historical tokens' KV has to be dropped. The core question for an eviction policy is: drop whom. The spectrum here is more "lossy" than attention sparsification, because once dropped it is genuinely gone.

• Sliding window / StreamingLLM: drop the oldest, keep the sink at the beginning. Simplest to implement, but what gets dropped may be exactly the evidence the task needs.
• Attention-score heuristics (H2O-style): accumulate the score with which each token is attended to, and keep the high-scoring set. The problem is that it requires maintaining a per-token score counter, and the accumulated score is strongly biased toward history — tokens seen early stay high-scoring forever, which is itself a bias.
• Observation window + pooling (SnapKV-style): place an observation window at the end of the prompt, use the queries in that window to score all keys in the prompt, take the top-k, and apply a clustering/one-dimensional pooling compensation over the retained positions. This is currently the most commonly used scheme in engineering, because its scoring is "relevance to the current request" rather than a global historical statistic, and it needs no extra training. A typical configuration in the paper compresses each layer down to 1K~4K tokens.
• Task-aware / tiered budgets (PyramidKV-style): different layers get different budgets — shallow layers mostly do local pattern matching and need a small window; deep layers do abstract reasoning and need to see further, so they get a larger budget. At the same time the budget ratio is adjusted dynamically based on remaining available context, shrinking the budget when little context remains.
• Attention sink + resident important tokens: keep the sink at the beginning, keep the recent window at the end of the prompt, and keep a set of high-scoring tokens picked from the whole text; the three are stitched together. This is essentially the default recipe in current production systems.

The common trap shared by all eviction policies is this: they all use "access patterns during generation" to predict "what the correctness of the whole request actually requires." The two are highly correlated but not equivalent. A counterexample is multi-hop retrieval: the evidence that is genuinely needed has never been accessed before generation reaches step 3, so any eviction policy based on access statistics may drop it prematurely. For tasks like this, the usual mitigation is to do retrieval at the prompt layer (use external retrieval to stuff the relevant passages into the beginning or end of the prompt) rather than relying on the model's internal eviction policy to save the day. This is why "external retrieval" and "internal sparsity" coexist for so long in long-context systems: the former solves recall, the latter solves cost.

5.5 Observability for the KV Cache: four metrics you must watch

Tuning long context without measurement is nothing but guesswork. At minimum you should continuously collect four metrics. The first is KV Cache occupancy and fragmentation rate: the share of memory actually occupied by KV versus the share held by free blocks, which directly tells you whether the paging configuration is sensible. The second is KV bytes per token: this value changes as the sequence grows, because eviction policies shrink by different ratios at different lengths, and it describes the health of the compression policy better than the total does. The third is prefix cache hit rate: in multi-turn dialogue or shared system prompts, this hit rate directly determines throughput; a hit rate below expectation usually means the prefix was collaterally damaged by the eviction policy, or that the block alignment of paging keeps a logical prefix from being reused. The fourth is the distribution of attention mass per layer: periodically sample the ratio of KV actually accessed per layer and the effective window size per layer — this is the most direct basis for tuning tiered budgets.

The most underestimated of these four is the last one. Tiered budgets sound like an architecture question, but what actually decides how to split them is measured data on the per-layer attention distribution — and that distribution varies a lot across models, and even across training stages of the same model. Turning "the ratio of KV actually accessed per layer per step" into a dashboard that varies over time is more reliable than any heuristic parameter.

6. Paged memory management: what PagedAttention actually solves is fragmentation

When concurrent requests have different context lengths, KV Cache allocation runs into the classic external fragmentation problem: one request needs 8K, another needs 100K, and if you reserve by maximum length, memory gets eaten by holes under continuous serving. What PagedAttention does is borrow virtual-memory paging from operating systems: cut the KV Cache into fixed-size blocks, treat each block as a "page," map a request's logical KV sequence onto a set of physical pages, and allocate physical pages on demand from a shared pool.

The figure given in the vLLM paper is that waste from fragmentation under traditional contiguous allocation runs at 60%~80% (depending on the length distribution), and drops to under 4% after paging. Beyond saving memory, paging brings three direct system-level benefits: first, near-zero-copy prefix caching — shared system prompts, few-shot examples, and the history portion of a multi-turn dialogue are all shared at page granularity, and a hit is a hit with no copying required; second, wavefront-style continuous batching — a new request can be inserted at the tail of a running batch at any moment, without waiting for the whole batch to finish; third, sparse attention can be addressed by page — the top-k retrieval and block-level importance scoring described earlier are all built on the unit of a "block," and paging is naturally that unit.

The cost of paging is per-page metadata overhead, plus the requirement that the attention kernel handle the mapping of "non-contiguous physical pages + logical position." In real engineering this means you need a genuine attention kernel that supports variable-length block tables, rather than grabbing some general flash-attention library that only handles contiguous KV. SGLang's RadixAttention is another implementation of the same idea, organizing shared prefixes with a radix tree (prefix tree), so that both hit detection and eviction become operations on a tree.

Paging also has a frequently overlooked killer use case: copy-on-write for shared prefixes. When a request needs to branch into multiple candidates from some middle position (beam search, multi-path sampling, or tree-shaped generation), the traditional approach has to copy the entire KV, and under long context the copy cost is proportional to sequence length — copying a prefix of tens of thousands of tokens takes seconds. After paging, only the page table has to be modified: the candidates share the same physical pages for the prefix, and only the pages after the branch point are newly created for each. This property is what made long context economically viable together with tree decoding and multi-candidate generation for the first time — and those are exactly the underlying dependencies of the product idea "generate several answers at once and pick the better one" in long-context settings.

But a shared prefix also brings an isolation problem: when requests from different tenants or users share the same page prefix, a single out-of-bounds write causes cross-request information leakage. An online inference framework must guarantee exclusive write permission on a physical page, while read permission can be shared. This is not something a performance optimization can work around; it is a security requirement, and you should test the concurrent out-of-bounds scenario explicitly during acceptance.

7. Prefill and decode: two completely different bottlenecks

Put all of the optimizations above back into one real inference run, and the most consequential mistake turns out to be "not distinguishing prefill from decode." The bottlenecks of these two stages are opposite at the hardware level.

7.1 Roofline: compute-bound vs memory-bound

Prefill is compute-bound. It processes hundreds or thousands of tokens at once, the matrix multiplications are large enough, the GPU's floating-point units get saturated, and every fusion, sparsity, and quantization in the attention kernel converts directly into wall-clock savings. At this point the KV Cache has not been built yet, memory pressure is relatively low, and the main bottleneck is FLOPs.

Decode is memory-bandwidth-bound. For every token generated, every layer has to read the entire KV Cache from HBM once — this is an unavoidable cost, independent of how many tokens you generate. A 7B model reads roughly 14 GB of weights per token, which is 7 ms at 2 TB/s of bandwidth; add the KV Cache reads and a 128K context needs an extra 40~64 GB per token, multiplying the bandwidth requirement several times over. At this point reducing the number of bytes in KV matters far more than reducing FLOPs, while compute-side sparsification yields very little (because only one query is computed per step, and what sparsification saves is computation that was already minimal).

The corollary is clear: on the prefill side you want attention sparsification and chunk-level parallelism; on the decode side you want KV quantization, KV eviction, and paging reuse. Plenty of teams get these two backwards — emphasizing "sparse attention" during decode while forgetting that the decode bottleneck is moving bytes, not computing.

7.2 Scheduling: chunked prefill and mixed batching

The motivation for chunked prefill is this: a prefill that processes 100K tokens at once occupies the GPU for several seconds, and during that time every request that is decoding is blocked, so time to first token (TTFT) spikes while time per output token (TPOT) stays normal, which is a terrible experience. Chunked prefill cuts a long prompt into fixed-size chunks and interleaves them with the decode steps of other requests, letting the two coexist in the same batch. The cost is that total prefill time grows slightly (there is synchronization overhead between chunks); the benefit is a large improvement in the TTFT/TPOT balance.

Going further leads to mixed batching (continuous batching): treating decode as a prefill that appends one token per step, executing both in the same batch graph, and having a unified scheduler allocate compute by token budget. This mechanism is the foundation of throughput in modern inference engines, but it also brings new problems — when KV Cache memory runs short, the scheduler must preempt (preempt) certain requests, swapping them out or dropping them. The choice of preemption policy directly affects fairness among long requests: a 500K request that is preempted frequently will have a very ugly TTFT. That means you have to make an explicit tradeoff between context-length scheduling and priority. The mechanisms vLLM later introduced, such as "allocate by token budget plus tiered queues," are in essence all about scheduling fairness when long and short requests coexist.

There is one more easily overlooked point: the benefit of speculative decoding is amplified during long-context decode. Because decode is bandwidth-bound, a single forward pass produces one useful token while moving the entire KV, so this "trade bandwidth for compute" swap is an excellent deal. A 5~10x lossless speedup is close to free during long-context decode — provided the draft model is small enough and the target model's verification phase can reuse the same KV Cache.

7.3 Latency budget: write the long-context requirements into the SLA

The place long-context projects most easily break during acceptance is that latency targets are never decomposed. For a 1M-context service, if you only give one number — "time to first token under 5 seconds" — the team will go optimize prefill, and nobody owns the multi-turn dialogue experience during decode. The recommendation is to split latency into four numbers: time to first token (determined by prefill cost and queueing), time per output token (determined by KV bytes and bandwidth), time to first token on a cache miss (which is usually the real source of tail latency), and the context growth slope (how time per output token grows with input length). The last one matters most, because it directly tells you how the maximum concurrency of the service scales with context length.

One practical reminder: a compression scheme has to be accepted on tail latency, not just on the mean. The costs of KV swap-out, request preemption, and paging fragmentation are all highly concentrated in the long tail — and the long tail is precisely where user complaints come from. A generation latency that averages 200 milliseconds but has a p99 of 4 seconds is unusable in a dialogue product.

8. The gap between chunking long sequences at training time and full context at inference time

This is the most easily glossed-over and also the most fatal problem in long context: at inference time you want the model to look at 1M, but at training time it almost certainly never saw a contiguous 1M sequence. That gap has several layers.

The first is positional discontinuities introduced by sequence chunking. To use finite memory fully, long documents are cut into fixed-length segments during training. The problem is that segments are packed using document ids rather than position ids (so that a cross-sample attention mask works), while at inference time position ids increment continuously. What the model learns during training is "relative positioning within each segment," but at inference it meets an input whose within-segment position reaches a million. RoPE does depend on absolute position (even though it is designed to primarily encode relative position), so this discontinuity directly puts the input out of distribution. The common fix is to restart position ids from zero per segment as well (equivalent to giving each segment its own independent "document"), but then the model never learns long-range relationships across segments.

The second is insufficient training coverage of attention patterns. If a model mostly sees "short segments plus document boundaries" during training, it has never seen the true attention distribution of very long sequences. The solution is progressive extension: train at 4K first, continue at 32K, then at 256K, initializing each stage from the previous stage's weights and adjusting the scaling ratio of the positional encoding per stage (this is precisely the progressive-extension idea of LongRoPE). The cost of this kind of training is real — a meaningful continued-training run at 256K typically costs dozens to hundreds of times the GPU cost of a short-context fine-tune.

The third is the sheer scarcity of long-sequence samples. The web, code, and books contain plenty of natural documents exceeding a million tokens, but after cleaning, deduplication, and quality filtering, the remaining high-quality long-sequence samples are far fewer than short ones. If a lot of low-quality concatenation (truncated documents, templated content) sneaks into the long-sequence training data, what the model learns is not long-range dependency but "how to ignore noise." This is the root cause of the poor real-world performance of many models that "claim long-context support," and it is a data problem, not an architecture problem.

The fourth is uneven loss weighting. On a sequence of length 1M, if you average the loss over all tokens with equal weight, the gradient contribution from the "first 1K" (often the information-dense opening of a document) far exceeds that of the middle. The common approach is to weight by chunk or to compute the loss on only a subset of tokens, but that introduces new bias. This is an engineering detail with no universal solution, and approaches differ widely between teams.

8.1 Code on the training side: block-level attention mask and packing

The code below demonstrates the core mask construction for long sequences during training: packing many short samples into one long sequence and using a block-level mask to block cross-sample attention. This is what the overwhelming majority of teams actually do, and it is also the source of the largest context discontinuity on the inference side.

import torch


def build_block_causal_mask(seqlen, block_size, device):
    '''因果 mask + 块内双向：因果注意力的标准变体。
    block_size > 1 时，同一块内允许互相看见，块间仍然严格因果。
    推理时如果用滑窗 + 全局 sink，这个结构要保持一致，否则训练/推理不匹配。
    '''
    idx = torch.arange(seqlen, device=device)
    causal = idx.view(-1, 1) <= idx.view(1, -1)
    blk = (idx.view(-1, 1) // block_size) >= (idx.view(1, -1) // block_size)
    # 混合注意力（如 Gemma 3n / 交替滑窗层）就靠这里给不同层返回不同的 mask
    return causal & blk


def packed_document_mask(seq_lens, total_len, device):
    '''多文档 packing 的 mask：文档之间完全隔离。
    注意 position_ids 要在这里同步重置为 0，
    否则模型会把不同文档的位置接成一条连续的长序列——
    这正是训练/推理落差的直接来源。
    '''
    doc_id = torch.zeros(total_len, dtype=torch.long, device=device)
    position_ids = torch.zeros(total_len, dtype=torch.long, device=device)
    cur, pid = 0, 0
    for n in seq_lens:
        doc_id[cur:cur + n] = pid
        position_ids[cur:cur + n] = torch.arange(n, device=device)
        cur += n
        pid += 1
    mask = doc_id.view(-1, 1) == doc_id.view(1, -1)
    # 再和因果 mask 求交，并保留前若干个全局 sink 列
    return mask, position_ids


# 训练循环里的常见组合：
#   mask = packed_document_mask(lens, N)[0] & causal_mask
#   out = model(input_ids, position_ids=position_ids, attention_mask=mask)
#   # 关键：position_ids 用上面重置后的版本，不要用默认的 arange(N)
# 对照检查：如果你的推理引擎在 decode 阶段用单调递增的绝对 position，
# 而训练时每篇文档都从 0 开始，那么长到 10 万之后就会有明显的分布漂移。

8.2 How to construct long-context training data

If you have no naturally long documents on hand and have to manufacture them yourself, there are a few principles. First, concatenation must be natural. Wiring unrelated documents together with a separator teaches the model a shortcut: that documents are unrelated, so a large amount of attention in the long sequence gets wasted on detecting boundaries and meaningless positions. A better approach is to use related documents on the same topic, consecutive chapters of the same book, or mutually referencing files from the same code repository. Second, deliberately construct samples that require long-range dependency. Plant a key entity in the front and ask about it in the back; or give a definition in the front and use it in the back. Public long-context benchmarks are essentially constructed this way, and the training set should contain a corresponding proportion too — otherwise what the model learns is only local co-occurrence rather than long-range dependency. Third, control the proportion of extremely long samples. If extremely long samples exceed some threshold, short-text ability degrades noticeably (catastrophic forgetting), because the optimizer's updates get dominated by a handful of very long gradients. The common practice is to put 1% to 5% of extremely long samples in the mix and keep everything else at normal lengths, letting those guard short-context performance.

There is one more easily overlooked dimension: in mixed-length training, the loss of extremely long samples should be weighted by token rather than by sample, otherwise a single million-token sample will dominate the gradient of the whole batch. This pairs with the packing discussed earlier — packing lets many short samples share one computation, while extremely long samples usually form their own batch, and the two need different gradient accumulation strategies for training to be stable. Many teams find their extremely-long branch refuses to train stably, and the root cause is right here.

9. Failure boundaries: where these approaches fall over

Long context is a field that is "good enough on many tasks and completely broken on a few." Listing the failure modes clearly is far more useful than memorizing method names.

9.1 Failure boundaries of positional encoding schemes

Pure frequency scaling degrades most visibly on tasks that require precise long-range localization: in a 200K context, "what is the 173,842nd token?" is a question interpolation schemes almost certainly fail, because they compress all positions into the range seen during training and dilute the resolution between adjacent positions. The right answer for such tasks is external retrieval or explicit position annotation, not relying on positional encoding.

Another failure point is an extrapolation range at inference far beyond the training range. Consider a model that had YaRN scaling applied at 32K and claims 128K support; push it straight to 1M and the scaling ratio no longer matches (YaRN's 128K configuration needs 8x scaling at 1M, and the local resolution loss of 8x interpolation is fatal). The entire value of progressive extension is that it guarantees that at any moment the model has been trained "within the distribution of the current length."

9.2 Failure boundaries of attention sparsification

The failure scenario for structured sparsity (vertical-slash style) is input structure that does not match the preset topology. It performs well on natural language and ordinary code, but if your input is highly regular tabular data, time-series data, or logs with extremely high structural repetition, the vertical column pattern may waste the budget on a large number of meaningless tokens at the same positions.

The failure scenario for retrieval-based sparsity is an inconsistency between the coarse ranking and the fine ranking in terms of relevance ordering. Under a low-dimensional projection, the most similar vector is not necessarily the most relevant in full dimensions; and the piece of evidence that actually determines the answer may be only semantically related to the query, with no literal or directional association at all. This is especially true for tasks that "require cross-span synthesis rather than locating a single relevant paragraph," where the assumption behind top-k retrieval-based sparsity (that a few strongly relevant positions exist) simply does not hold. Code tasks have similar problems: the root cause of a bug sits at the beginning of a file and the symptom at the end, and in low-dimensional space both may be far from the query.

The failure scenario for learned sparsity is distribution drift between the selector and the backbone model. The selector is learned on the training distribution, so if it meets an out-of-distribution input in production (extremely long code, mixed languages, structurally anomalous documents), its importance predictions are unreliable — and this unreliability is silent. There is also a practical trap: the selector itself adds prefill overhead, and if it is not light enough, the attention FLOPs saved by sparsification may not cover the cost of the selector.

9.3 Failure boundaries of KV compression and eviction

KV compression (especially at 4bit and below) breaks down on numerically sensitive tasks: mathematical reasoning, exact code matching, and tasks that require looking up specific values in a long table. These tasks demand precision in the representation of every token, and quantization noise gets amplified by softmax normalization and by accumulation across layers. What you typically observe is not an obvious error but "the answer format is right but the value is slightly off" or "one intermediate step of a multi-step reasoning chain drifts."

The failure scenario for eviction policies was already described: multi-hop retrieval. What is worth emphasizing is that strategies like SnapKV that "score with the queries at the end of the prompt" have a structural flaw on multi-hop tasks: the trailing queries only care about the last hop, so they assign a very low score to the evidence the first hop needs. The mitigation is to use both the beginning and the end of the document as observation windows, or simply not evict at all (and go with more memory).

9.4 Failure boundaries of the training/inference gap

The most insidious category of failure: the length a framework claims to support and the length a model was actually trained on differ by an order of magnitude. The way to judge is very direct — check the maximum training/fine-tuning length explicitly stated on the model card. If it only says "extended to 128K via YaRN," then its strengths are almost certainly within 32K. If it says "continued training at 256K," then ability within 8K is usually well preserved while past 128K you have to measure it yourself. This is not an architectural defect but the inevitable result of resource constraints, yet almost no marketing material mentions it.

9.5 A set of controlled experiments you must run before shipping

Before deciding which approach to adopt, I recommend running at least these four comparisons, because their conclusions repeatedly overturn intuition in practice. The first is raw long context versus compressed equivalence: on the same model, run the same task set with the full KV and with KV compressed to one quarter, and see exactly which tasks absorb the quantization accuracy loss — this directly answers whether the business can tolerate quantization. The second is a sparsity configuration sweep: fix the total budget and vary only the split between dynamic retrieval and the sliding window, finding the optimum for your own tasks. The third is the effective length curve: stretch input length proportionally and plot accuracy; this curve is both the basis for capacity planning and the only guardrail that prevents the model from quietly getting dumber. The fourth is the sensitivity of throughput to prefix hit rate: measure hit rate and end-to-end latency under different eviction policies on a replay of real traffic — this item is frequently the highest return on investment of all optimizations, because it requires no model changes.

What these four experiments share is that none of them relies on any "the model is stronger so it can go longer" assumption, and all of them are built on your own task distribution. This is also my basic judgment on the long-context field as a whole — architectural position extrapolation, attention sparsification, and KV compression solve cost and upper limits, while how far your system can actually go depends on measuring the effective length on your own data and the engineering tradeoffs you build around it.

10. Six recurring misconceptions

• Treating the window number as a capability promise. The "supports one million" on a model card is a capacity description, not a quality description. There is only one criterion: on your own tasks, at what input length does answer quality start to drop? This number is usually far smaller than the nominal figure, and it appears in no marketing material.
• Thinking that adopting FlashAttention solves long context. Fused attention kernels solve the memory peak, not the computation. When sequence length doubles, the wall-clock time of the attention part roughly quadruples, while the body of the model only doubles. The quadratic term is masked by the linear term at short lengths, and only exposes itself at long lengths.
• Emphasizing attention sparsification during the decode stage. Decode generates one token per step, and the bottleneck is the bandwidth of moving KV from memory into the compute unit, not compute. Compressing the number of bytes in KV (quantization, eviction, tiering) pays off; reducing attention computation essentially does not. This is the most widespread directional mistake in long-context optimization.
• Using global access statistics for eviction. Schemes that keep high-scoring tokens based on accumulated attention scores work well on single-hop tasks, but their scores carry a strong historical bias: tokens seen early stay high-scoring, while the evidence a multi-hop task actually needs often is not used until the second hop — by which time it may already have been evicted.
• Testing long context only on natural language. The attention distributions of code, tables, and time-series data are completely different from natural language, and their precision requirements are far higher. Quantization error surfaces more readily as syntax errors on code tasks and as numeric drift on table tasks. Testing only natural language means you discover the problem after you have already shipped.
• Not regressing the effective length when upgrading models. Model weights, positional encoding configuration, the inference engine's scheduling policy, and the way prompts are organized — a change in any one of the four can send the effective length backwards. If a version upgrade only runs the usual short tasks, long-context ability may degrade by orders of magnitude with nobody noticing.

11. The 2026 engineering consensus and a selection checklist

Compressing everything above into a single table you can take into a meeting:

Stage	Mainstream approach	Typical benefit	Main cost
Positional encoding extrapolation	YaRN-style per-band mixing + attention temperature compensation; for long context, progressive extension + non-uniform scaling	Essentially lossless at the 128K scale	Does not solve "retrieval from a distance"; still degrades far beyond the training length
Pretraining stage	Mixed-length curriculum, mostly short sequences plus a small number of extremely long ones as a floor	Leaves feasibility for later extension	High GPU cost, long-sequence data is scarce
Architectural dimensionality reduction	GQA/MQA; use MLA (latent space + absorption tricks) when stronger compression is needed	KV dimensionality drops 10~100x	Slight accuracy loss; MLA creates engineering tension with paging
Attention sparsity	Sliding window + sink as the floor, layered with per-layer/per-head patterned sparsity or top-k retrieval	Prefill FLOPs drop several-fold	Recall failure is silent and needs external retrieval as a backstop
KV quantization	Keys per channel, Values per token; grouped quantization at 4~8bit	KV memory drops 2~3x	Degrades on long sequences and numerically sensitive tasks
KV eviction	sink + recent window + observation-window high-score pool; budgets allocated per layer	Memory stays under control for long requests	Multi-hop retrieval gets wrongly evicted
Memory management	PagedAttention / prefix tree, zero-copy shared prefixes	Fragmentation drops from 60~80% to under 4%	Requires an in-house or purpose-built paged attention kernel
Scheduling	chunked prefill + continuous batching + speculative decoding	TTFT/TPOT balance, decode speedup of several-fold	Long requests are easily preempted in mixed batching

If you want to distill this into a few principles of judgment, here is my view.

Turning to concrete execution, the pre-launch checklist below is the one I think is worth walking through item by item. It targets no specific framework, only the places where long-context systems themselves tend to break.

• The effective length has been measured and written into monitoring. Not an estimate, not the number from the model card — an accuracy curve on your own tasks, plus a trend chart for that curve across versions.
• Both the nominal length and the fine-tuning length on the model card have been confirmed. If the nominal length is far larger than the fine-tuning length, then anything beyond the fine-tuning length must be backstopped by external retrieval; you cannot count on the model itself.
• Bottleneck metrics for the prefill and decode stages are collected separately. Time to first token, time per output token, time to first token on a cache miss, and the growth slope of time per output token versus input length — all four numbers need to exist.
• Tail latency is accepted separately. The costs of compression ratios, swap-out policies, and preemption policies all concentrate in the long tail, and averages hide the problem.
• There is a dashboard for paging configuration and prefix hit rate. Fragmentation rate, KV bytes per token, prefix hit rate, and the ratio of KV actually accessed per layer — four metrics continuously visible.
• The accuracy loss of the compression scheme has been localized to specific tasks. Not "lossless overall," but a clear understanding of which class of task starts degrading at which compression ratio, with business-side thresholds set accordingly.
• The long-context regression test has been re-run after the version upgrade. Model weights, positional encoding configuration, inference engine version, prompt template — any change to any of these must trigger this regression.
• Multi-hop tasks have an independent recall backstop. Clearly define which scenarios must go through external retrieval, rather than hoping that the model's internal sparsity and eviction policies will handle them.

Do these eight things solidly, and the leap for a long-context project from "demonstrable" to "shippable" is basically complete.

First, determine the effective length before setting the nominal length. The mistake the vast majority of teams make is chasing the window number first, then discovering the business never needed a context that long, and finally realizing they have been paying in memory and latency for a length they will never use. Measure first where your tasks start to drop, and buy only the window you need plus a little headroom.

Second, treat prefill and decode as two independent projects. Their bottlenecks, techniques, and acceptance metrics are all completely different. Prefill is judged on TTFT and the payoff from sparsification; decode is judged on TPOT and KV bytes. Evaluating both with a single set of metrics is the most common resource misallocation in long-context projects.

Third, internal sparsity and external retrieval are not either/or. Sparsification solves cost, retrieval solves recall. On multi-hop tasks the two must coexist: a pure-attention 1M scheme and a 32K-plus-retrieval scheme differ far less in quality than they do in cost across many production scenarios. The real work lies in "how to organize the retrieval results into the context" — putting them at the beginning, at the end, or sorting by relevance directly determines the final accuracy.

Fourth, treat long-context ability as a system property that requires continuous maintenance. It depends on the joint configuration of four things: model weights, positional encoding configuration, the inference engine's scheduling policy, and how prompts are organized. Upgrading one while not verifying the other is the most common source of degradation in long-context systems. You need at least one regression test that pins down the "effective length" metric.

If you are looking one or two years ahead, there are a few things in this direction worth keeping an eye on. The first is the unification of sparsification and quantization: today sparsification decides which positions to look at and quantization decides how many bytes each position stores, and the two are designed separately; the next wave of evolution will most likely join importance prediction with bit-width allocation, letting the model itself decide which tokens deserve eight bits and which need only two. The second is a reinterpretation of the attention sink: the industry currently treats it as an engineering trick, but mechanistically it may be an inherent information-routing mechanism of the decoder at large-scale context, and it deserves to be redesigned from the angle of pretraining data construction rather than faked with a mask at inference time. The third is redrawing the boundary between context and external memory: once internal KV can be compressed, tiered, and swapped out on demand, the traditional vector database is no longer the only memory layer, and the two will gradually fuse into a unified tiered memory system.

A last note for teams that want to build this themselves: almost all the technologies mentioned above have core insights (the frequency distribution of relative positional encoding, the existence of the attention sink, the importance differences across heads, the decoupling of token importance from position) that are public and have been validated repeatedly. What is genuinely hard is not coming up with these ideas, but combining them in a production environment, keeping the accuracy loss of every step under control, and actually cashing out the theoretical gains at the kernel level. The amount of work in that part is typically an order of magnitude larger than the ideas themselves.