Model Compression: Where Quantization, Pruning, and Distillation Fit

I. What Compression Actually Saves: Separating the Three Ledgers

Almost every article on compression opens with "halve the parameters, halve the memory" and closes with "speed doubles." That chain of reasoning hides two independent jumps: compression saves storage and data movement first, not computation; and "saving movement" only becomes a latency win when the model is memory-bandwidth-bound. Separate the two and you can explain why a 4-bit weight model can be slower than BF16 on a single short-context request while being far faster in a 64-way concurrent long-context service.

So let us clean up the concepts first. The cost of running a Transformer can be split into at least four levers:

• Weight memory: how much DRAM the model parameters themselves occupy. This is the easiest thing to measure and the one most often treated as the entire story.
• KV Cache memory: the part that grows linearly with sequence length. It is coupled more tightly than weights to concurrency, context length, and GQA configuration.
• Memory bandwidth: how many bytes must be read out of DRAM per generated token. Weight access is rigid; KV access grows with length.
• Compute (FLOPs): the actual number of multiply-add operations. Weight quantization barely changes it; activation quantization does.

These four levers respond completely differently to different techniques. Line them up in a table and many seemingly contradictory community conclusions become explainable.

Technique	Weight memory	KV memory	Access bandwidth	Compute	Where the cost moved
Weight quantization (weight-only)	Strong (4~8×)	None	Strong	Essentially unchanged, even slightly higher	Shifted to on-device deployment thresholds and precision risk
Activation quantization (W8A8 / W4A8)	Medium	None	Medium	Strong (depends on low-bit tensor cores)	Shifted to hardware support and kernel coverage
KV Cache quantization	None	Strong (2~4×)	Strong, scaling linearly with length	None	Shifted to long-context quality and stability beyond the first token
Unstructured / 2:4 pruning	Strong (2×)	None	Depends on whether a sparse kernel is hit	Depends on the sparse kernel	Shifted to training or recovery training, plus the hardware model
Structured pruning	Medium (proportional to pruning ratio)	May shrink as a side effect	Strong	Strong	Shifted to retraining to recover capacity
Distillation	Strong (depends on student size)	Falls along with the student	Strong	Strong	Shifted entirely onto data construction and teacher inference cost
Activation sparsity / dynamic sparsity	Medium	None	Medium	Strong	Shifted to inference kernels and dynamic-shape overhead

Three conclusions can be read straight off the table. First, weight quantization never touches the KV Cache; many people run a 128K long-context service, blow through the memory budget, and then go tune the weight quantization bit width — they are aiming at the wrong target. Second, only activation quantization and structured pruning genuinely reduce FLOPs, and those two happen to be the two operations most sensitive to precision. Third, distillation is the only technique that hits all four, and the price is a full training run, which is the root reason it does not sit on the same cost dimension as the others.

1.1 Figuring Out Whether You Are Bandwidth-Bound or Compute-Bound

Before picking any compression method, one thing has to be measured first: is your service memory-bandwidth-bound or compute-bound? A rough heuristic is arithmetic intensity — the ratio of bytes read per generated token to the FLOPs actually completed. For a dense autoregressive Transformer, the weights are read in full on every step, and each step performs roughly one multiply-add per weight parameter, so the arithmetic intensity of the weight portion is a constant below 1 that does not vary with batch size. This leads directly to a counterintuitive but critically important effect:

• Single request, small batch: the GPU compute units sit mostly idle, and nearly all the time is spent waiting for DRAM to feed data. Compressing the weights is the highest-leverage move here, and the payoff shows up roughly linearly in TTFT and TPOT.
• Large batch, long context: once the batch is large enough for the matmul FLOPs to saturate bandwidth, the model becomes compute-bound. Compressing the weights further buys rapidly diminishing returns, and what is actually slowing you down is the KV Cache reads and writes plus the attention computation itself.

This is why reports like "the same model got 1.6× faster when quantized on a single machine, but only 1.1× faster at 32-way concurrency" show up everywhere. Nothing is wrong with the quantization method; the compression was simply applied to a dimension that was no longer the binding constraint. The most practical engineering move is to take the average input length, average output length, concurrency, and generation-length distribution from your request logs, compute the weight reads and KV reads per step, and only then decide which end to attack first.

The payoff curve of compression is not a line. It is a piecewise function whose x-axis is "where the bottleneck is." Locate the bottleneck first, then choose the method; reverse the order and you burn an entire round of experimental budget.

II. Weight Quantization: Bit Width, Grouping, and Outliers Must Be Reasoned About Together

Weight quantization maps each weight from a 16-bit float to a low-bit integer plus a set of quantization parameters. The form is minimal, but whether it works well is decided entirely by three orthogonal choices: the target bit width, the granularity at which quantization parameters are stored, and how outliers are handled. Most people tune only the first one and then complain that results are unstable.

2.1 The Payoff of Bit Width Is Sublinear, and Metadata Eats Into It

Naive mental arithmetic says INT4 is twice as good as INT8 and four times as good as BF16. Reality is different, for two reasons.

First, low-bit dynamic range collapses fast. INT8 gives roughly 7 effective mantissa bits, INT4 only 3. When the absolute values of the vast majority of weights in a tensor are concentrated in a narrow band, the step size of uniform quantization becomes coarse, and the batch of small weights near zero all map onto the same codeword, losing their information entirely. This is why the quality loss from 8-bit to 4-bit is usually far larger than the loss from 16-bit to 8-bit, and why the loss curve has a visible knee around 3~4 bits.

Second, the scaling parameters themselves cost bytes. Take the most common group-wise symmetric INT4: with a group size of 128, each group stores one FP16 scale (16 bits) plus a 4-bit zero point (4 bits once packed), for an overhead of 20/128 ≈ 0.156 bit per weight. In other words, a nominal 4-bit scheme really occupies 4.16 bits per weight. The same way: a 7B model in BF16 weights is about 14 GB, pure INT4 is theoretically 3.5 GB, and 3.64 GB with group metadata; a 70B model is about 140 GB in BF16 and about 36 GB in INT4 with metadata. Dropping group size from 128 to 64 doubles metadata overhead to about 0.31 bit per weight and takes the compression ratio from 3.85× to 3.72× — not a big difference, but the quantization parameters are reused half as often inside the tensor, and the quality improvement is usually noticeable. Dropping group size to 32 pushes overhead toward 0.63 bit per weight and the compression ratio down to 3.46×, and at that point the payoff starts to diminish while the scale arrays in cache eat memory bandwidth on the way back out.

Bit width	Effective precision	Typical format	7B weight size (incl. metadata)	When to use it
BF16 / FP16	Native training format	bf16	~14 GB	When memory is plentiful, or when you need a numerical reference
FP8	~3~4 mantissa bits	E4M3 / E5M2, optionally with per-tensor or per-block scale	~7 GB	The nearly free tier; commonly the default intermediate format on both the training and inference side
INT8	7 effective bits	Symmetric/asymmetric per-channel, per-token dynamic scale	~7 GB + scale	The most universally compatible option with mature hardware support; quality loss is usually negligible
INT6 / FP6	—	Usually floating-point or integer with per-group scale	~5.6 GB	The compromise tier: steadier than INT4, smaller than INT8; often where the cost-performance curve turns
INT4 / FP4	3 effective bits	Per-group integer; or block-scaled float (one scale shared by every 16 or 32 elements)	~3.6~3.9 GB	The mainstream tier for consumer single-GPU and edge deployment; must be paired with grouping or block scaling
INT3 / INT2	≤ 2 effective bits	—	~2.7 / 1.8 GB	Meaningful only for extremely narrow use cases; quality collapses on general tasks

There is one more ledger that is easy to overlook: metadata and kernel residency. If the implementation dequantizes per group, the scale arrays, the quantization parameter buffers, and possibly any unfused operator workspaces all consume memory. On a 4-bit scheme these "non-weight overheads" can account for 5% to 15% of total memory, and skipping the calculation leads you to overestimate the compression ratio.

2.2 Grouping: Spend the Budget Where It Is Needed Most

The core tension in quantization is a trade-off between "step size" and "resolution." The more elements a single scale covers, the fewer parameters you pay for, but the more weight differences underneath that scale get flattened. From coarse to fine:

• per-tensor: one scale for the entire weight tensor. Metadata is essentially free and quality is the worst; use it only for layers with extremely uniform weight distributions, or in extremely conservative settings.
• per-channel (per-output-channel): one scale per output filter. The overhead is a small fraction of 1/group, essentially free, and it is the default starting point for every modern PTQ method, because it matches the natural structure of the weights along the output dimension.
• per-group: cut groups of 64 or 128 along one dimension, each with its own scale. This is the tier with the most obvious quality gain and the de facto standard for 4-bit schemes.
• per-group + zero point (asymmetric): adds one zero point per group. The gain is clear when the weight distribution is strongly right-skewed (many small positive values), and the overhead is about 0.03~0.06 bit per weight.

One practically useful judgment: group size is the only tunable knob between "metadata overhead" and "quantization resolution," while bit width is the only tunable knob between "memory" and "quality." The right engineering order is to first pin the bit width to the memory line you absolutely have to meet, then along that line move group size from 128 to 64 and see whether quality holds up. That order is far faster than trying it in reverse, because once the bit width drops, a smaller group size cannot bring it back.

2.3 Outliers: The Stubborn Channels Inside LLM Weights

Transformer weights and activations share a well-known pathological property: the vast majority of elements crowd near zero, while a few channels carry systematic outlier channels whose magnitude is far beyond that of the rest. This is not noise; it is a product of training convergence.

For quantization this is a double hit. On the weight side: a single outlier inflates the scale of the group it belongs to, which compresses every normal weight in that group down to a few codewords — effectively zeroing them out. On the activation side: an outlier channel inflates the scale at every layer, forcing the whole layer into low precision even though the actual information content is nowhere near that large.

On the weight side, mainstream approaches fall into two families. The first is clipping outliers: truncate weights beyond some quantile before quantizing, sacrificing a few units to protect overall resolution — cheap but crude. The second is more refined: split the outlier information out and store it separately, quantizing the main body at low bit width while keeping the few outliers at higher precision or as a sparse exception table, then adding them back at dequantization time. A third family, and the most influential since 2025, "rotates" the outliers away — applying an orthogonal transform to weights and activations together so they land in a more uniform coordinate system, making the outlier channels disappear mathematically, and folding the rotation into the neighboring operators after quantization. The generalized form of this idea (counting random orthogonal transforms, Householder transforms, and online Hadamard transforms as one family) has become one of the standard preprocessing steps of PTQ in 2025—2026, and is close to mandatory in W4A8 settings, because that is precisely where activation quantization lives.

To find out cheaply whether your model is outlier-dominated, run one experiment: do a pure per-tensor quantization and a per-channel quantization and compare. If the perplexity gap between them is much larger than intuition suggests, that layer has significant outlier channels, and you should reach for rotation-style preprocessing rather than continuing to shrink the bit width.

2.4 Floating-Point Bit Widths and Block Scaling: Where the Real Movement Is in 2025—2026

The dynamic range problem of low-bit integers was, in essence, solved by block scaling: cut the weights into very small blocks with one shared floating-point scale per block — the smaller the block, the better the scale fits the local range and the higher the effective precision, while the scale overhead is amortized down to something tiny. The MX specification standardized the idea: 32 elements share one E8M0 power-of-two scale. NVIDIA's NVFP4 took a different route: one FP8 E4M3 scale per 16 elements, plus a global FP32 per-tensor scale to express the magnitude differences between blocks.

The significance of these two formats is not merely "two more format names," but that they turn 4-bit from a tier that requires group-wise integer tricks into a native tier with floating-point semantics. The price is higher metadata overhead: NVFP4 has one FP8 scale per 16 elements, an overhead of 8/16 = 0.5 bit per weight, so a nominal 4 bits really costs 4.5; MXFP4 has one 8-bit scale per 32 elements, an overhead of 0.25 bit per weight, so it really costs 4.25. For a 7B model, 4.5 bits means about 3.94 GB instead of 3.64 GB — the extra 300 MB buys you no need for asymmetric zero points and better tolerance to distribution shift. Whether it pays off depends on whether your target hardware has the corresponding low-bit floating-point tensor cores.

Another clear trend in the same period is that FP8 has moved completely from inference into training. Mixed-precision training with FP8 forward and backward passes, combined with fine-grained block accumulation, is now sufficient to support training large-scale MoE models, rather than being only an inference-side weight quantization option. One side effect is often overlooked: the model was trained in FP8, so its weight distribution is inherently better suited to FP8 inference — quantizing to FP8 is nearly lossless, while quantizing to INT4 does not necessarily look better than it would for other models. That means the payoff of a compression scheme is partly determined by the training format, so you cannot carry quantization experience from one model family over to the next.

It is also worth spelling out the semantics of the precision choices themselves: E4M3 has more mantissa and a smaller dynamic range, which suits numerically sensitive forward activations; E5M2 has fewer mantissa bits and a wider dynamic range, and is usually reserved for gradients or other range-sensitive quantities. This division of labor runs counter to the BF16/FP32 intuition, so it is easy to configure backwards.

2.5 The Calibration Set: The Most Ignored Hyperparameter and the One That Matters Most

PTQ needs a small batch of data to estimate per-layer activation statistics (usually per-channel absolute-value maxima or quantiles). That batch is the calibration set. It holds only tens to thousands of samples and looks unimportant, yet its impact on final quality can rival the bit width choice. Three things you must get right:

• The distribution must match. Calibrating on generic web text and then deploying on code or Chinese tasks mismatches the activation ranges. Samples from the target task, even a few hundred of them, usually work better than a hundred thousand generic documents.
• It must cover the length distribution. Activation distributions in long-context tasks vary with position; a scale calibrated on short video sequences may be wrong at long positions.
• Do not use memorized samples from the training set. Calibration does not update weights, but using such data for statistics makes the scales overfit, so evaluation numbers will look better than reality.

Another problem that only became systematically addressed in recent years is the post-softmax output. It has an inherently long-tailed distribution, and quantizing per token directly lets a few large values dominate everything. Computing the max per token dynamically solves most of the problem, which is why modern KV quantization and dynamic activation quantization both default to per-token granularity.

2.6 Activation Quantization: The Only Tier That Actually Cuts Compute

Everything discussed so far has been weight quantization. It shrinks the weights but leaves the number of multiply-adds per step exactly the same. If the goal is to pull latency out of the compute-bound regime, activation quantization is the path: it compresses the inputs and intermediate results of the matmul to low precision as well, so the multiply-adds genuinely execute on low-bit tensor cores.

It is an order of magnitude harder than weight quantization, for three reasons. The first is outliers: weight distributions are static, so you can measure and adjust them repeatedly before quantizing, but activations are generated dynamically every step and the distribution changes with context, so any global statistic mismatches. The second is error propagation: a weight error only affects that layer's output, while an activation error seeps through the residual connection into every later layer and amplifies with depth. The third is the kernel: even if the quantization parameters are perfectly computed, without a corresponding fused kernel — or if the kernel has to compute scales online and reshape dynamically — the real speed may be worse than the unquantized BF16 path.

The realistic path for activation quantization therefore starts at 8 bits rather than jumping to 4. W8A8 keeps weights at 8-bit integers and activations at 8 bits with per-token dynamic scales, combined with channel smoothing or rotation to suppress outlier channels; this tier has mature kernels on mainstream accelerators and is a production-ready option you can use with confidence. W4A8 is a different story: compressing weights to 4 bits already cuts memory traffic substantially, and keeping activations at 8 bits mainly preserves numerical precision while still letting the matmul hit low-bit instruction sets — the payoff here actually comes mostly from the reduced weight traffic, and those 8 activation bits exist so the gain does not evaporate into precision loss. Compressing activations to 4 bits as well (true W4A4) collapses quality noticeably on general tasks and is usually worth considering only in extremely narrow, thoroughly validated scenarios.

Scheme	Weights	Activations	Memory traffic gain	Compute gain	2026 assessment
BF16 baseline	16 bit	16 bit	Baseline	Baseline	The origin of every comparison
Weight-only INT4	4 bit	16 bit	~3.7×	None	Most widely used; the payoff concentrates in bandwidth-bound scenarios
W8A8	8 bit	8 bit dynamic	~2×	Yes	Production-ready, requires preprocessing to remove outliers
W4A8	4 bit	8 bit dynamic	~3.7×	Partial	The mainstream balanced form on the inference side
W4A4	4 bit	4 bit	~3.7×	Yes	Unstable on general tasks, use with caution

The last column is the one to study hardest when choosing: the source of the memory-traffic gain and the source of the compute gain must be accounted for separately. Almost all of the speedup a W4A8 scheme gets on a single request comes from weights dropping from 2 bytes to 0.5; the activation quantization portion only starts to pay off once a large batch pushes the machine into the compute-bound regime. Conflating the two leads to the wrong conclusion — "I adopted W4A8 and nothing got faster" — when the reality is usually just that the service was never compute-bound in the first place.

2.7 A Minimal Quantization Self-Check You Can Run Before Anything Else

Before getting into any specific method, there is a self-check process that requires very little code and rules out the overwhelming majority of bad quantization outcomes. Its value is that every step is cheap, yet every step can eliminate a direction.

1. First confirm whether the weight distribution is genuinely uniform. Sort a layer's weights by absolute value and see what share of the total absolute value the top one percent accounts for. If that share is high, there are strong outliers, and you should reach for rotation or clipping before touching group granularity. This check takes a few lines of code and frequently changes the direction of the whole design.
2. Run one brute-force quantization that is purely symmetric and purely per-tensor, and record the quality loss. This is the lower bound for every grouping and scaling scheme, and it is extremely informative. If the brute-force version already drops a lot, the problem lies in the distribution itself rather than the quantizer; if it barely drops, every later gain comes from resolution and is worth investing in.
3. Vary one variable and sweep the group granularity. At a fixed bit width, sweep from small to large and watch where the quality curve flattens. Finding that knee is far more reliable than sticking to the default — the default was measured on someone else's model and may not suit your distribution.
4. Introduce error-correction algorithms only at the end. They should come into play once resolution is already good enough, to recover the last little bit of loss. Trying to patch a very coarse resolution with error correction pays far less than simply shrinking the group granularity first.

The hidden logic of this process is solve the distribution problem first, the resolution problem second, and only then bring in algorithmic tricks. Most people do it in reverse: they try a parade of algorithms and then discover the real bottleneck was an outlier issue that twenty lines of code could have found. Once the order is right, the algorithmic search space you have to cover is actually small, while the payoff is far more certain.

III. GPTQ and AWQ: One Problem, Two Complementary Angles

GPTQ and AWQ are usually presented as "two 4-bit methods" and treated as either/or. In reality they answer questions at two different levels, and they are frequently stacked together. Understanding that distinction is worth more than remembering which one is more accurate.

3.1 GPTQ: Treating Quantization Error as Cross-Column Error Propagation

GPTQ starts from a fact many people overlook: when quantizing column by column, the rounding error of earlier columns changes the quantization result of later columns. If each column is quantized independently, the errors accumulate independently; but if you use the inverse of the weight Hessian to redistribute the error, it is theoretically possible to end up with a smaller total error in the reconstructed weights.

The practical implementation relies on a second-order approximation: treat the increment introduced by quantization as a single update step on a Hessian-weighted objective, then after each column is quantized use a Cholesky decomposition to back-propagate the accumulated error onto the columns not yet quantized. This is why GPTQ has to work column by column and maintain an accumulated error vector, and why it is slower than naive round-to-nearest — it performs an entire additional round of cross-column numerical correction.

Three practical points about GPTQ:

• It operates on top of per-channel or per-group quantization rather than replacing grouping. Nearly every 4-bit scheme is group-wise (commonly 128); GPTQ corrects the rounding on top of that foundation, it does not substitute for it.
• It is sensitive to the small-batch ordering of the calibration set. Different splits of the same data can give slightly different results, so the seed must be fixed and recorded during evaluation.
• It is insensitive to the activation distribution, using only the weight Hessian, which makes it very suitable for weight-only scenarios; but precisely because it does not model activations, its gains on W4A8 are far below those of activation-aware methods.

3.2 AWQ: Salient Channels Are Determined by Activations, but Protected with Weights

AWQ's observation comes from the other side: only about 1% of weights influence the model output most, and those salient weights are determined by the magnitude of the input activations, not by their own size. This is exactly the point many people get backwards from intuition — a large weight is not automatically an important one, because a large weight that always multiplies a small activation contributes very little.

AWQ searches for a per-channel scaling coefficient s (applied along the input dimension) that amplifies salient channels in the activations and shrinks unimportant ones, so that the important weights occupy a larger share of the numeric dynamic range and are less likely to be rounded away during quantization. This scaling is a mathematically equivalent transformation that can be folded into the previous layer's weights at inference time, so it adds no inference overhead.

Their complementary relationship becomes clear from this:

Dimension	GPTQ	AWQ
What it optimizes	Weight reconstruction error (Hessian-weighted)	Channel salience of the activation distribution
Whether activation data is needed	Yes, but only to build the Hessian	Yes, to measure channel salience
Where it acts	Rounding and error redistribution of the weights	Per-channel scaling of the input dimension (folded in at inference)
Extra inference overhead	None	None (the scale is folded into the previous layer)
Relationship to grouping	Used together with per-group	Used together with per-group
Typical pairing	Combined in sequence with AWQ-style scaling	Combined in sequence with GPTQ-style rounding

One engineering conclusion worth remembering: the two do not conflict, and implementations frequently apply channel scaling first and GPTQ-style rounding second. Scaling improves the conditioning of the weight distribution, and GPTQ then redistributes error on that better distribution. The "GPTQ + AWQ-style scaling" combination common in the ecosystem means exactly this, not two mutually exclusive algorithms.

3.3 The Fuller Toolbox: Rotation, Smoothing, and Mixed-Precision Decomposition

By 2026, mentioning only GPTQ and AWQ is incomplete. A few active directions form the supplementary layer of modern PTQ:

• Rotation / incoherence processing (the QuaRot, SpinQuant, QuIP# family). Orthogonal transforms scatter the outlier channels of weights and activations across all channels, making even per-tensor quantization work. It is nearly mandatory for W4A8 because it simultaneously solves the activation-side outlier problem. The cost is one offline computation for the transform itself, plus the extra kernels that online Hadamard transforms introduce.
• Activation smoothing (the SmoothQuant family). The idea is close to AWQ but the target differs: migrate activation outlier difficulty to the weight side, because weights are static and their quantization error is easier to control, while activations are dynamic and hard to handle. The scaling can likewise be folded into the previous layer's weights.
• Mixed-precision decomposition (the OmniQuant family). Write each linear layer as a low-bit body plus a small high-bit or floating-point residual, and search layer by layer for the optimal rank of that residual. In essence this turns a quantization problem into a low-rank decomposition problem, which is more advantageous than plain low bit width at the 2~3-bit tier.
• Calibration-free / optimization-based methods (the AutoRound family). Treat each layer's weights as optimizable variables and use training-style optimization to find better rounding points, at the cost of an offline process far heavier than GPTQ — typically tens of minutes to several hours.

With that layer in view, a practical selection rule emerges: for weight-only 4-bit, stay conservative — the GPTQ family plus per-group 128 is enough; for W4A8 throughput, rotate first and quantize second; if you are stuck at 3 bits or below, mixed-precision decomposition is more worth trying than grinding harder on group size.

IV. KV Cache Quantization and FP8: Where Long Context Should Actually Be Optimized

If your weights are already down to 4-bit and memory is still not enough, the problem is almost certainly not in the weights. The KV Cache is the only block of memory in long-context and high-concurrency deployments that grows linearly with sequence length, and its growth coefficient is not under your control. That is where the real contradiction lives.

4.1 First Get the KV Size Right

The KV Cache has a closed-form size in bytes: 2 × layers × KV heads × head_dim × bytes per element × tokens. Note that the head count here is the KV head count, not the attention head count — GQA/MQA shrinks this part by a large factor, and that is one of the reasons modern long-context models are viable at all. Any compression plan must be built on the actual KV head count; you cannot estimate it using the attention head count.

Here is a purely arithmetic example: for a configuration with 80 layers, 8 KV heads, and head_dim of 128, in FP16 each token costs 2 × 80 × 8 × 128 × 2 = 320 KiB. Running at a full 128K context, a single sequence needs on the order of 40 GiB of KV — before counting weights, activation workspaces, and framework overhead. Converted to other precisions: about 20 GiB for FP8, about 20 GiB for INT8, and about 11 GiB for INT4 (group 64 plus metadata). Those numbers say two things: in long contexts KV is often an order of magnitude larger than the weights; and the payoff from KV quantization stacks linearly and needs no clever algorithm — going from FP16 to INT4 is a straightforward 4×.

4.2 Why KV Quantization Is Easier Than Weight Quantization

Intuitively KV is harder to compress than weights because it is generated dynamically and its distribution shifts with content. That the results are nonetheless good has three causes:

• Its error does not accumulate across steps. Weight quantization error propagates through each layer's input into every later layer and amplifies with depth; KV error only operates inside the attention computation of the current step, and the next step generates new KV, so there is no path that accumulates along sequence length. This is the most underestimated advantage of KV quantization.
• The softmax itself has tolerance. Attention weights are normalized relative quantities — scaling attention scores proportionally does not change the result — and the first derivative of softmax with respect to small perturbations approaches zero in the small-value region. That leaves quantization some room.
• You can calibrate dynamically per token. Each token's own K and V get their own scale, with no need to share anything across tokens, so the granularity is naturally fine.

The cost is real, and it concentrates in one place: once attention scores are perturbed by quantization noise, retrieval ranking goes wrong. On "needle in a haystack" style tasks that require pulling one precise fact out of a large context, that error shows up first. So evaluating KV quantization on aggregate perplexity alone is not enough — the evaluation must include long-context retrieval and long-form reasoning subsets.

4.3 Choosing Granularity: Per-Token, Per-Channel, or Per-Head

The most critical design variable in KV quantization is the granularity of the scale. Here are the trade-offs across the three mainstream choices.

Granularity	Metadata overhead	Quality	Kernel complexity	Notes
per-tensor / per-layer	Nearly zero	Worst	Lowest	Different sequence lengths and different contents contaminate each other; not recommended for production
per-token (K and V separately)	Very low	Good	Low	The current mainstream default; scales computed dynamically, with no precomputation burden
per-head	Low	Better	Medium	Noticeable gains when dynamic ranges differ significantly across attention heads
per-channel	Medium	Good, but with a directional risk	High	Normalizing along K's head_dim flattens the query-side channel selection signal, and retrieval tasks can degrade noticeably

The most counterintuitive and most important row in that table is per-channel. Intuitively finer granularity must be better, but channel-wise scaling along head_dim has a side effect: it changes the relative contribution of different channels to the query dot product, and those channels are exactly what the model uses for "matching." If the scaling is asymmetric between K and Q, the semantics of the attention score are destroyed. The conclusion is: finer is not automatically better; per-token is the safe choice for the vast majority of scenarios, and per-channel should only be used once you have verified that retrieval tasks do not regress.

4.4 Key-Aware and Query-Aware: The Idea of Selective Quantization

A second line of thinking comes from an observation: positions, heads, and channels contribute vastly differently to the final output, so there is no reason to compress them all identically.

• Key-aware: estimate an importance score for each token or each head, keep high-importance parts at higher precision and compress the rest harder. The hard part is the reliability of the importance estimate itself — a score derived from heuristic rules, if it misjudges inside long documents, ends up worse than uniform quantization.
• Query-aware: note that the query vector q for the current decoding step is already known, so different K values can use different quantization parameters, minimizing the impact of quantization error on the current query. The idea is mathematically natural, but engineering-wise it costs a recomputation keyed on q, and it ignores the premise that KV is already cached, which makes it hard to implement.

Both lines currently share the same status: the theoretical gain is clear, the realized gain is unstable, and the engineering complexity exceeds that of the per-token scheme. For the vast majority of production deployments, get per-token INT8/FP8 solid first — the payoff is already in hand — and only then consider whether a selective scheme is worth it for specific tasks.

4.5 Where FP8 Actually Sits in an Inference Engine

Between 2025 and 2026, FP8 went from "a new format" to the default intermediate precision of inference engines, but its position is often stated incorrectly. It shows up mainly in four places:

• Weight storage: as an alternative to 8-bit formats, offering a better dynamic range at the same size; common as the "floor" precision tier on the serving side.
• Activations and intermediates: the P×V intermediate state of attention, FFN intermediate activations, and the accumulators of various reductions — this is where the gain is largest.
• KV Cache: storing K and V in E4M3 is the current safe choice — K participates in dot products and is numerically sensitive, so it uses E4M3 with more mantissa bits; V participates in a weighted sum and can be relaxed. This has essentially become the widely adopted default configuration.
• Training side: as noted earlier, FP8 mixed-precision training makes models natively suited to FP8 inference.

One detail you must check: the accumulation precision and rounding mode of FP8 significantly affect the result. In many implementations the FP8 multiply-add accumulates in FP16 or FP32, but whether rounding happens before or after the multiply, whether split-k accumulation is used, and whether special handling is applied to the softmax input of attention all produce observable differences. When comparing FP8 results across frameworks, first confirm that the accumulation semantics match, otherwise the differences may come from the implementation rather than the model.

The correct compression order for long-context deployment is: confirm the KV head count under GQA/MQA first, apply FP8/INT8 quantization to the KV next, and only then go back and consider weight quantization. The memory you win by taking weights from 8-bit to 4-bit is often less than the extra you win by taking KV from FP16 to FP8.

4.6 The Other Path for Long Context: Not Every Problem Should Be Solved With KV Quantization

KV quantization solves the problem of storing the same KV in fewer bytes, but it does not solve the more upstream question: you may not need that much KV in the first place. There are two complementary approaches along the long-context direction that are worth putting into the same selection table as quantization.

The first is selective eviction of KV. Of the KV that has already been computed, the vast majority will almost never be used again during subsequent decoding — the typical case being portions of a long document that later questions ignore entirely. Approaches built on this observation use a lightweight criterion to score each position for importance, keep only the high scorers, and remove the rest from the cache outright. The key difference is that its gain grows continuously with sequence length, whereas KV quantization only buys a constant factor — at hundreds of thousands of tokens, eviction frees far more space than a narrower bit width. The risk is equally direct: once the criterion evicts a position that was genuinely needed, the model produces a confident wrong answer, and this error is hard to catch in routine evaluation because it does not show up as a drop in the aggregate score, only as wrong answers on individual hard questions. That is why eviction approaches must come with a dedicated long-range retrieval evaluation.

The second is sparsifying attention itself. Dense attention cost grows with the square of the sequence length, which makes it the most expensive component in long context. Sparse attention lets each query attend to only the few blocks it considers relevant, and the key question is not which tokens to keep but how to find the possibly relevant blocks cheaply enough first. The blocking, indexing, and prefix-reuse mechanisms used here have failure modes that are not entirely the same as KV quantization's: quantization error accumulates with depth, while sparse attention errors concentrate almost entirely in the single failure mode of not retrieving a relevant block. The two must therefore be evaluated separately, and sparse attention needs dedicated tests on tasks where the answer is buried somewhere in a long document.

Looking at all three together, they are complementary rather than competing: sparse attention reduces the amount of KV that must be computed, KV eviction further reduces the amount that must be stored, and KV quantization lowers the storage cost of each KV. A real long-context system will likely use two or even three of them at once. But the order matters — if selective eviction has already consumed half the KV budget, the additional gain from compressing to INT4 is limited, and at that point the better investment is improving the accuracy of the eviction criterion rather than squeezing the bit width further. The bit width can be compressed indefinitely, but the cost of a wrong criterion does not shrink with it, and that is the most important asymmetry between the two.

V. Pruning: Unstructured and Structured Are Two Different Fates

Quantization does not change the "shape" of a model; it only changes how the numbers are stored. Pruning changes how many non-zero parameters the model actually has — an operation with a completely different character, and with exactly opposite engineering consequences.

5.1 The Core Tension: Sparsity Versus Hardware Utilization

An ideal pruning scheme wants three things at once: high sparsity (memory savings), a regular structure (so hardware can accelerate it), and near-lossless quality. In reality these three are mutually exclusive: the freer the sparsity structure, the harder it is for hardware to exploit; and the structures that hardware can exploit are precisely the ones researchers deliberately shaped to match hardware constraints.

The reason lies in the execution model of a general-purpose GPU. Matmuls are decomposed into fixed-shape tiles, each processed by a group of threads reading contiguous memory and performing multiply-adds. If the non-zero elements are scattered throughout memory, the cost of skipping them can equal or exceed the cost of reading them — you saved the multiply-add and spent it on memory traffic instead. This is the root reason unstructured random pruning works in papers and fails in production: its sparsity pattern saves theoretical FLOPs, not actual hardware work.

5.2 2:4 Semi-Structured Sparsity: The Only Form Broadly Supported by Hardware

2:4 is the most successful pruning format from an engineering standpoint, and its constraint is extremely simple and elegant: of every 4 consecutive weight elements, keep exactly 2. What makes this constraint beautiful is that it compiles into a fixed sparse computation pattern — the hardware loads two non-zero values and their indices per 2:4 block, and every pruned weight is never touched at all during loading, wasting no bandwidth whatsoever.

Its costs are just as clear:

• The ceiling is 50% compression. Whatever your model could stand to lose, the granularity of the hardware format makes 2:4 be 2:4. Theoretically higher sparsity brings no additional memory-traffic benefit under the same block format.
• Real hardware support is required. Only newer accelerators have the corresponding sparse tensor instructions and the promise of 2× throughput; on unsupported cards it degrades into gather plus dense computation and can end up slower than not pruning at all. You must benchmark on the target model before shipping — spec sheets will not do.
• Indices cost extra storage. The index metadata beyond the non-zero values takes space, so the real compression ratio is 2:1 minus metadata overhead, landing around 1.6~1.8× rather than the nominal value.
• Quality loss requires recovery training. Hard pruning is a large, biased perturbation of the model, and without recovery training the accuracy drop is obvious.

There is an implementation detail that is often overlooked: the groups of 4 elements are cut along the contiguous dimension after the weights are flattened, not along output channels. That means a group of 4 elements may span two different output filters, and the semantics of what you are pruning are completely unlike what your structured-pruning intuition suggests. Mask design must respect this flattened order, otherwise you will prune four weights belonging to the same filter — which amounts to deleting an entire output channel, with enormous damage — instead of four scattered unrelated weights.

5.3 Structured Pruning: The Kind That Actually Reduces FLOPs

Structured pruning removes complete channels, attention heads, FFN intermediate dimensions, or even whole Transformer layers or experts. It is the only form of pruning that genuinely saves compute, because the pruned portion disappears entirely from the computation graph with no runtime check whatsoever.

The cost of structured pruning varies dramatically by granularity:

Pruning target	What you save	Quality risk	Recovery training needed	Practical assessment
FFN intermediate dimension	Parameters and FLOPs, with a controllable ratio	Low	Medium	The most common choice; layers can use different ratios, ordered by sensitivity
Attention heads	Parameters and KV head count (saving KV as a side effect)	High	Relatively high	Heads are highly redundant with each other, but pruning the wrong one costs capability directly; you usually have to realign head_dim afterward
Transformer layers	Parameters and FLOPs drop together	Very high	Very high	Layers are the skeleton of depth; recovering from layer pruning needs massive training; suitable for depth-pruning research
MoE experts	Parameters can drop substantially	Medium	High (router retraining)	Already a sparse structure, so trimming experts is far more natural than trimming dense layers
Embedding / LM Head	Parameters (a notable share when the vocabulary is large)	Very high	Not feasible	Essentially unprunable, unless you prune the vocabulary and retrain the embedding

The real difficulty of structured pruning is not "how much to remove" but "which ones to remove." The three common classes of criteria are magnitude-based criteria on the weights (the simplest and least reliable, because weight size is unrelated to importance — the point already made in the AWQ section), activation-based criteria (using calibration data to measure each channel's output magnitude), and ablation-based criteria (pruning one at a time and measuring the impact — the most accurate but with an unacceptable cost). In practice the common approach is to rank layers by sensitivity: measure each layer's reconstruction error first, prune in ascending order of error, and stop at the target ratio.

5.4 Recovery Training: The Stage That Decides Whether Pruning Succeeds

Hard pruning must be followed by recovery training. There are no exceptions, yet many people underestimate the workload. Several points determine how well recovery works:

• The learning rate should be one to two orders of magnitude below pretraining. The goal of recovery training is to repair perturbed weights, not to reshape the representation. Too high a learning rate will wash out existing capability.
• The data distribution should match the target. Recovering on generic corpora and then deploying on a specific task will underperform.
• Do not freeze most of the parameters. The common approach is to unfreeze all layers at a smaller learning rate, or to unfreeze only the layers adjacent to what was pruned. Freezing everything and training a few layers recovers insufficiently at high pruning ratios.
• The sparse format must be fixed at training time. The 2:4 constraint means that after every update the weights must be projected back onto a valid mask, and this step must be part of the training loop. Otherwise you will train a model that does not satisfy the format and the hardware will silently process it incorrectly at deployment.

Also watch out for an easily overlooked interaction: if you quantize after pruning, the thing being quantized is the post-recovery-training weights. If you plan to prune first and quantize second, recovery training should not use quantization-aware training (QAT) to chase a quantization error target for weights that have already been removed — that is wasted effort. Conversely, if you plan to quantize first and prune second, what you prune should be an entire already-low-bit block, and granularity alignment will be a problem.

5.5 A Special but Valuable Use: Turning Dense Weights into a Sparse Expert Structure

Pruning and MoE look like opposite directions — one subtracts, the other adds. But there is a very useful conversion path between them: prune part of a dense model's FFN weights and re-partition what remains into several experts, yielding a sparse model at almost no loss of capability.

The value of this path is that it captures the benefits of both pruning and sparse activation: the model really does get smaller (total parameters drop), while each token still activates only a small subset of parameters (preserving the compute advantage of conditional computation). It is far cheaper than pretraining a sparse architecture from scratch, and compared with pruning alone it preserves the ability for the same total capacity to behave differently on different inputs.

The conversion is not free, and three things must be handled. First, the router must be retrained: a dense FFN is a single fixed mapping, so once you split it into multiple experts you need a load-balanced assignment mechanism, and the initial token allocation is almost certainly extremely imbalanced (a few experts swallow most of the tokens) — only training makes it stable. Second, capacity differences between experts: splitting the FFN evenly assumes that different channel segments of the original FFN carry equally important functions, and that assumption usually does not hold, so capacity has to be allocated unevenly according to importance. Third, the pruning ratio and the number of experts are coupled: total parameters equal the number of experts times the parameters per expert, and both must be planned together — but the actual compute per token is determined by the expert count alone. These two numbers are frequently confused during design, producing a model much larger than expected that is not much faster.

A pragmatic recommendation: this path is best suited as repurposing — converting an already-trained dense model into a sparse form, or merging several same-architecture models from different domains into one conditional model — rather than as a route for building an extremely large capability model from scratch. The former costs days; the latter costs months, and the latter's results are far better than anything any conversion trick can deliver.

VI. Distillation: Moving the Cost onto Data

The first five techniques all re-represent existing weights without touching the model topology. Distillation is the only one that genuinely replaces the model — it fits a smaller student to the teacher's behavior. Its ceiling is the highest (a 4B student catching a 70B teacher is entirely possible), and its cost is the highest too (it needs the full teacher inference plus training pipeline).

6.1 Distillation Is Not Simply "a small model learning from a big one"

Treating "distillation" as a single method loses its most important information. There are at least three levels:

• Logit distillation: the student matches the teacher's output distribution at every token (temperature-scaled soft labels plus the true labels). It conveys far more information than one-hot labels — the dark knowledge in soft labels encodes the teacher's relative confidence in the other candidates, and that is precisely what the student learns.
• Sequence distillation: train the student directly on text generated by the teacher (rather than on the teacher's token probabilities). The form is ordinary SFT, just with data from the teacher. Simple to implement, but it discards distributional information and runs into the problem of teacher output diversity.
• Chain-of-thought distillation: the teacher generates answers with a full chain of thought and the student learns to imitate it. This is especially important for reasoning models, because the main carrier of reasoning ability is the intermediate process — learning only the final answer teaches you nothing about reasoning.

One clear methodological shift in 2025—2026 deserves separate mention: offline distillation is being replaced by on-policy distillation. The core observation is that a student trained by offline distillation can only imitate the teacher's text outside its own capability distribution — the student has never been "pulled back" by the teacher at token positions where it makes mistakes, because it never sampled there. On-policy distillation lets the student generate the sequences itself, and then has the teacher provide supervision on the states the student actually visits, directly correcting the student's specific errors. Mechanistically it turns a learning problem from "imitate a distribution" into "correct at critical states," and on reasoning tasks the resulting gap is usually far larger than the gap in compute between the two methods.

6.2 Data Construction: The Ceiling on Distillation Quality

The upper bound on distillation results is usually determined by data construction rather than by the training recipe. A few decisive variables:

Stage	Approach	Where it goes wrong
Problem sources	Public datasets, real user logs, synthetic problems	Synthetic problems tend to become homogeneous; the teacher emits templated answers on familiar patterns and the student learns no boundary behavior
Teacher sampling	Sample multiple times at multiple temperatures, keep the correct ones via a discriminator or rules	Using only temperature-0 greedy output gives insufficient sample diversity, and the student overfits to a single teacher path
Quality filtering	Rule checks, execution verification, cross-model adjudication, deduplication	Accepting data without verification means the teacher produces fluent but wrong data on long-tail questions, and those errors get distilled into the student in full
Difficulty distribution	Deliberately retain a batch of hard problems	Keeping only what the teacher solves easily puts all the data in the student's comfort zone, and the ceiling on the distilled improvement is very low
Decontamination	Compare n-grams and semantic similarity against the evaluation set	Skipping decontamination means part of the "high score" after distillation is memorization, which severely overestimates the model
Format consistency	Unify the format and length style of the chains of thought	A teacher whose own chain of thought is verbose and repetitive causes the student's reasoning token count to explode, raising cost instead of lowering it

Two of these deserve particular emphasis. First, verification is a required stage of the distillation data pipeline, not an option. The teacher model's error rate on difficult tasks is not low, and the student's mimicry ability is strong — it will learn the error patterns too. Distillation without verification amplifies the teacher's weaknesses.

Second, difficulty distribution matters more than data volume. This is where many people do not follow. The learning signal in distillation comes mainly from the positions where the student errs and the teacher does not. If most of the data consists of easy problems both of them can solve, the teacher's soft labels carry almost no information at those tokens and training produces almost no gradient. What is genuinely valuable is data from the capability-gap region. In practice that means dynamically adjusting sampling difficulty to the student's current ability — a continuously evolving data production process rather than a one-time data preparation step.

6.3 The Teacher-Student Capability Gap: When Distillation Stops Working

Distillation has a repeatedly observed failure mode: when the capability gap between teacher and student is too large, the payoff collapses and can even fall below plain SFT on the teacher's outputs. The reason is not hard to see — the student lacks the foundational representations to support the teacher's higher-order behavior, so what it learns is only the teacher's surface form (sentence patterns, format, terminology) rather than the reasoning structure, while being dragged off course by large quantities of erroneous reasoning steps it cannot digest.

There are two practical decision criteria. The first is to look at whether the teacher's chain of thought is learnable: if the teacher's reasoning chain is so long that the student can never reach the correct intermediate states, then no matter how you sample, the gradient signal is noise. The second is to look at the difficulty-matched distribution: run the teacher and the student separately on a validation set and measure the proportion of questions the student "can answer but is unsure about" versus those it "cannot answer at all." If the former is very low, the data is entirely outside the capability-gap region, and you need to shrink the teacher or run a round of SFT to shore up the foundation first.

From this follows a robust strategy: layered distillation. Start with a round of SFT using the largest teacher (which may come with the strongest but most verbose reasoning), then switch to a teacher better matched to the student's current ability for several rounds of on-policy distillation. Re-evaluate the capability gap after every round and adjust the difficulty distribution accordingly. This is essentially a curriculum learning problem, not a one-time data transfer.

The payoff of distillation comes mainly from data in the capability-gap region. Once the student has matched the teacher on some dimension, the marginal return from continuing to distill on that dimension approaches zero, and the compute should be moved to the dimensions where a gap remains.

6.4 How to Measure Distillation Specifically

A distilled model differs from a quantized model in one essential way: its quality is not computed by a formula, it is trained. That means evaluating a distilled model must be able to distinguish "the student really learned it" from "the student is merely imitating the teacher's surface form", and a single aggregate score cannot tell the two apart.

There are three signals that must be measured separately. The first is the capability-gap curve: interpolate between the student and the teacher and measure performance at different interpolation points. If the curve is still clearly below the teacher on the side approaching it, the student is not under-trained but under-capacity — adding more data will not help, and you need a bigger student or a narrower task distribution. This is the single most important line for distinguishing "not enough data" from "the student is too small," and one interpolation experiment produces it.

The second is whether the teacher's errors get inherited. On questions the student is bound to get wrong, compare how close the student's answer distribution is to the teacher's. If they are very close, the student is imitating the teacher's mistakes. This can only be measured by sampling in the teacher's low-confidence region; measuring on samples the student is confident about is meaningless. It is also the most direct evidence of whether the verification stage of the distillation data pipeline actually did its job — if verification was skipped or insufficient, this curve sits noticeably high.

The third is the on-policy versus offline comparison: with the same teacher and the same compute budget, run one offline distillation and one on-policy distillation and compare. This comparison shows the direction at modest scale, yet it is often the only basis for deciding whether to rebuild the entire data production process as on-policy — because such a rebuild is expensive and needs data support before anyone will sign off on it.

Combining the three, the evaluation logic for distillation is: confirm capacity first, then confirm that learning is correct, then confirm the method choice. Optimizing data while skipping the first step, or rebuilding the whole data pipeline while skipping the third, are both common ways to waste time.

VII. Sparsification: Weight Sparsity and Activation Sparsity Are Two Different Things

The 2:4 and structured pruning discussed above both belong to weight sparsity — they decide which weight parameters are permanently discarded. But there is another entirely different path: deciding at runtime which computations to skip this step. This family is collectively called activation sparsity (or dynamic sparsity).

7.1 Weight Sparsity: Static, Predictable, and Poorly Supported by Hardware

Weight sparsity decisions are made once during training or pruning, and inference runs a fixed mask. The advantage is that it is predictable — you know how much computation each layer does per step, and you can pre-schedule it offline. The disadvantage is that it is extremely hostile to hardware: the mere existence of a mask guarantees no speedup whatsoever, and general-purpose GPUs only deliver real gains on fixed formats like 2:4. More fundamentally, a zero in the weights does not mean a zero in the computation — if a pruned weight happens to pair with a large activation at that moment, the loss is real, but the sparse format will not treat it any differently.

7.2 Activation Sparsity: Dynamic, Large Hardware Gains, and Larger Precision Loss

The logic of activation sparsity is completely different: the weights stay dense, and multiplications are skipped during the multiply-add based on the magnitude of the activations. Because activations are regenerated every step, which multiplications to skip is also dynamic, which naturally fits runtime sparsity.

The technical core of this approach is prediction: you have to know which activations will be small enough to skip before actually performing the multiply-add, and that can only come from prediction — either approximating the activation magnitude with a low-rank model, or approximating it with downsampled statistics of the real activations. Inaccurate prediction has two consequences: missing large values that should have been kept (precision loss), or keeping small values that should have been skipped (sparsity rate missed).

These methods were studied seriously in 2023—2024, and the conclusion was: their pure FLOPs savings are substantial, but they are hard to realize in real end-to-end inference services. The reason was given back in section one — higher sparsity raises theoretical arithmetic intensity, but services are often still limited by memory traffic, kernel scheduling, and the gather overhead that the sparse pattern introduces. On top of that, the FFN intermediate dimension in a Transformer is only a few thousand, and the post-activation distribution is fairly flat (there are not enough small values to find), so the fraction that can be safely skipped often falls short of the theoretical value.

Dimension	Weight sparsity (2:4)	Activation sparsity
When the decision is made	Once, at training/pruning time	Dynamically, at every inference step
Saves memory	Yes (1.6~2×, including metadata)	No
Saves FLOPs	Depends on the sparse kernel	Yes (but the predictor overhead must be paid)
Hardware support	New cards natively support 2:4	Generic gather computation, uncertain payoff
Precision risk	Low, the mask is predictable	High, prediction errors cost accuracy directly
Explainability	High	Low, requires extra analysis
2026 assessment	Production-ready, requires matching hardware	Research stage, unstable end-to-end payoff

Adjacent to activation sparsity is MoE-style dynamic sparsity — using MoE routing so that each token activates only a few experts. It is the only form in the "dynamic sparsity" column that has been validated at large scale in production. The key difference is that MoE's sparsity comes from expanding the parameter count (more total parameters, fewer active per token) rather than from deleting parameters. Its memory ledger runs the opposite way — MoE raises memory requirements while lowering FLOPs. So MoE is not a compression technique at all; it is an architectural trade of memory for compute, exactly opposite to compression. Marketing MoE as "compression" is a conceptual error.

7.3 Dynamic Token Pruning and Early Exit: New Forms of Sparsification at Inference Time

The previous two sections discussed static sparsity on the weight side, but sparsification has two more forms at the inference stage that are growing in importance, and neither reduces parameters — they only decide what not to do this step.

The first is token pruning in the prefill stage. The cost of long context is concentrated in the prefill phase, yet the vast majority of tokens processed in that phase contribute almost nothing to the final answer — in a long block of background material, perhaps only two or three sentences are actually relevant. The approach is to identify low-contribution tokens with a cheap criterion (usually a rough estimate of attention scores, or a very small auxiliary scoring model) and discard them before the main computation begins. The difference from KV eviction is one of timing: KV eviction happens during decoding, while prefill pruning happens before the attention matrix has been fully computed, so what it saves is not just storage but quadratic-level computation. Its risk comes from the same source as KV eviction: if the criterion drops a critical token, the model will confidently answer from incomplete information, with no error raised anywhere.

The second is per-token early exit. For a simple request the answer is often already determined after a handful of decoding steps, and generating dozens more tokens merely writes out an answer that has already stabilized. Per-token early exit adds a lightweight criterion to decoding (commonly the magnitude of change in the hidden states, or a tiny classification head deciding whether the current answer is already closed), and once convergence is judged it stops early and appends the end token. It and cascade routing are the same intuition at different granularities: not every token is worth generating at the same cost. The risk of such methods is that the criterion mistakes "still thinking" for "already converged," and early truncation directly produces incomplete answers — and what gets truncated is often exactly the critical second half of the reasoning chain.

Placing both forms into the picture yields a unified view: the essence of compression is judging which computations can be skipped. Weight quantization judges by numeric importance (outliers and small weights get rounded away), pruning judges by parameter importance, KV eviction and token pruning judge by content importance, and early exit and cascades judge by difficulty. The difference lies only in where the reliability of that judgment comes from — judgments derived from numbers are the most reliable but the coarsest, and judgments derived from learning are the most fine-grained but the most fragile. That is why the genuinely robust approach in production systems is to make the reliable coarse judgments (quantization, low-bit-width pruning) the foundation, and to confine the fragile fine judgments (eviction, routing) to scopes where the gain is clear and the evaluation can cover them.

VIII. Stacking All Three: The Errors Compound, the Gains Do Not

Now we can address the hardest question: can quantization, pruning, and distillation be applied together? The answer is yes, but the combined effect is neither linear nor additive, and each step must be verified individually.

8.1 An Intuition You Must Build First: the Three Error Types Are Different in Nature

Looking at the errors of the three techniques from a single viewpoint, they actually act on different parts of the model:

• Quantization error: high-frequency, small perturbations on the weight side or the activation side. It is local, dense, and present in every layer, acting on every weight and every activation.
• Pruning loss: structural, sparse, one-time capacity removal. It deletes specific parameters (structured) or fixed patterns (unstructured).
• Distillation error: global and goal-directed. The distillation process actively compensates for the loss caused by compression with training — it is a supervised repair performed after lossy compression.

This difference in nature immediately implies an important conclusion: because distillation error is trained and can observe the behavior of the model after quantization and pruning, "compress first, distill second" is in principle superior to "distill first, compress second." Distilling a good student first and then quantizing and pruning it means no compensation mechanism exists after compression; compressing first and distilling second lets the distillation stage treat the compression loss as a target to fit. This ordering is not a matter of preference; it is a matter of mechanism.

8.2 The Specific Points of Interference

Several concrete interactions are worth listing separately:

• Granularity conflict between pruning and quantization (the most serious one). The 2:4 constraint groups along the contiguous dimension after flattening, and group-wise quantization also cuts along contiguous dimensions. If you quantize group-wise first and then prune, the two pruned non-zero values may corrupt the scale statistics of the quantization group; if you prune first and then quantize, a pruned value that happened to be an outlier in its group causes the whole group's scale to be recomputed, which actually improves things. In practice you should prune first and quantize second, so the quantizer sees the final post-pruning distribution.
• The circular dependency between quantization and recovery training. If you do recovery training on 2:4 pruning while also using quantization-aware training, the two fight over the same objective: recovery training wants to reclaim the capacity that pruning removed, while QAT wants the weights to resist quantization. The practical approach is staged — prune plus recovery training to convergence first, then freeze the mask and do a short QAT fine-tune at the end. Do not try to optimize both at once.
• Mismatch between distillation data and the compressed model. The teacher's reasoning chains were generated for an uncompressed level of capability. If you do offline distillation on the compressed student, the teacher trajectories it sees describe states it could never reach, amplifying the capability-gap problem described earlier. On-policy distillation is especially valuable here, because it naturally samples where the student is actually weak.
• The ordering of outlier handling. Rotation-style preprocessing should come before quantization, but if you are also doing structured pruning, pruning after rotation means the pruning criterion sees a distribution that is no longer the original one. The inverse of a rotation is orthogonal and does not change numerical magnitude relationships, so it is generally safe; but if you use magnitude-based criteria to select channels for pruning, the order still matters.

8.3 Recommended Ordering and How to Verify It

Synthesizing the above analysis, a general stacking order is:

1. Structured pruning (if you are doing it) — because it changes the model shape, it must come first, and it reduces the parameter count for everything that follows.
2. Recovery training — reclaim the capacity lost to pruning. Note that this step belongs before low precision: recovery works best at high precision.
3. Distillation (if you are doing it) — train this already-pruned model on teacher data so it aligns with the teacher. At this point the pruning loss and any initial low-precision effects are compensated in one go.
4. Preprocessing such as rotation / channel scaling — estimated from the actual activation distribution of the final model.
5. Weight quantization — as the final representation change, its error no longer has any mechanism left to compensate it.
6. KV quantization — orthogonal to the weights, so it can proceed in parallel and be evaluated on its own.

But this order is only a starting point, and it must be verified empirically. One concrete method is incremental ablation: start from the BF16 baseline, add one technique at a time and record the quality change; then combine them in pairs and in triples, stacking gradually. If some combination's loss exceeds the sum of the individual losses (a negative interaction), the two techniques are competing for the same thing and you need to adjust the order or the strength of one of them.

Combination	Recommended?	Reason	Caveats
Weight quantization + KV quantization	Strongly recommended	They act on different tensors and do not share an error path	Evaluate them separately; do not merge into a single number
Weight quantization + activation quantization	Recommended	W4A8 is a mature form, but it requires rotation preprocessing first	Kernel support is a prerequisite, and quantized activations usually need to be 8bit
Structured pruning + weight quantization	Recommended	Prune first, quantize second, so the quantizer sees the final distribution	The pruned model is smaller, so you can afford a larger group size in exchange for quality
2:4 pruning + 4-bit quantization	Use caution	Both act on the weights, so the granularity conflict is severe	Pruning must come first; the real payoff may be less than using either alone
Quantization + distillation	Recommended, in the order quantize then distill	This is exactly where the compensating ability of distillation pays off	Generating distillation data with the quantized model is more accurate but more expensive
Pruning + distillation	Recommended	Pruning removes capacity, distillation restores capability; they pair naturally	Recovery training and distillation can be merged into a single training run
All three	Possible, but with clearly diminishing returns	The combined payoff usually drops sharply after the second technique	Incremental ablation is mandatory to check for negative interactions

One final practical judgment that must be stated: the marginal payoff of stacking decays very quickly. Going from 16-bit to 8-bit alone costs almost no quality, 8-bit to 4-bit costs a little, and below 4-bit it becomes very hard. Similarly, pruning 20% may cost very little while pruning 50% demands a large amount of recovery training. So when you want to apply all three, the honest expectation should be: the first technique captures most of the payoff, the second and third each add a small slice, and the interaction terms may eat some of that. If time is limited, applying only the single largest payoff is usually the optimal decision.

8.4 A Third Path Beyond Combination: Chaining Models of Different Compression Levels

Every method discussed so far acts on a single model. But in engineering there is a completely different approach that is often more effective than any single compression scheme: prepare several versions at different compression levels and let a router decide which one each request goes to.

The most common form is a cascade: a small, fast model goes first, and uncertain or high-risk requests are handed off to the large model. Its value comes from a fact that is often overlooked — the difficulty distribution across requests is extremely uneven. A large volume of everyday requests is format conversion, common-sense Q&A, and simple rewriting, where the result on the strongest model is nearly identical to the result on a lossily compressed model; while a small number of requests need long chains of reasoning, and those degrade noticeably on a compressed model. Allocating compute per request is far more economical than compressing everything uniformly.

The interaction between cascades and compression is very clear: the upper bound of a cascade's payoff is set by its weakest tier. If every version is BF16, even the cheapest tier is still expensive and the cascade saves little. Compressing the weakest tier to 4-bit or 6-bit is what actually brings the overall cost of the cascade down. In the other direction, the harder you compress, the less error tolerance the router has — a router with an inaccurate uncertainty estimate paired with a lower-quality model will leave requests that should have been escalated in the low tier, producing a larger average quality drop than expected. So router accuracy is the core metric of a cascade system, more important than any single-tier compression metric, and evaluation must separately measure the false-escalation rate (escalated when it should not have been, wasting cost) and the missed-escalation rate (not escalated when it should have been, losing quality).

A simpler variant is static routing by task type: deterministic, simple business traffic goes to the small model, complex traffic goes to the large model, with no runtime judgement at all. You give up some flexibility in exchange for completely predictable latency and cost. Where request features are easy to classify, this is usually the better engineering choice, because runtime uncertainty is itself a form of cost.

Finally, one decision-level fact worth flagging: a cascade is an alternative to compression, not a supplement. When the traffic difficulty distribution is skewed enough, the combined cost of one small model plus one large model may already be lower than letting one medium model carry everything. So before investing in fine-grained compression tuning, measure the difficulty distribution of your requests first — if 80% of them really are simple, then the return on spending budget on cascade routing is usually far higher than squeezing 4-bit down to 3.6-bit.

8.5 The Interface Between Compression and Fine-Tuning: Order Determines Cost

Compression is not the end of the training pipeline; it is often the midpoint. Because in most cases another round of fine-tuning follows compression, and where that round is placed can change the cost severalfold.

First, the most important principle: quantization-aware fine-tuning should come after the compression scheme is settled, rather than being used to help you choose the scheme. Many people run a round of quantization-aware fine-tuning while choosing a bit width, see quality recover, and conclude the bit width is viable — that is circular reasoning. Quantization-aware fine-tuning genuinely can make an aggressive configuration usable, but it hides the critical information of "how marginal this configuration really is." The right approach is to measure the quality floor with training-free quantization first, then decide whether one round of fine-tuning is worth spending to rescue it.

Next, several concrete ordering rules. Recovery training must follow pruning, and that round should not simultaneously do quantization-awareness: pruning is already a large structural perturbation, recovery training aims to get the capacity back, and simultaneously suppressing quantization error sets two contradictory objectives against each other, with the common outcome that both are only half done. A safer arrangement is to complete the pruning recovery at high precision first, then freeze the mask, switch to the quantization configuration, and use a very short fine-tune to let the weights adapt to the new representation.

Distillation and fine-tuning can be merged into one round. If the final plan involves both pruning and distillation, there is no reason to run recovery training and then distillation as two separate rounds — use the teacher's outputs as the objective for that single training round while keeping the pruning mask fixed, and both objectives are handled inside one optimization. That is far cheaper than two serial training runs and usually works better, because recovery training and distillation both modify the same weights, and doing them separately lets the later stage wash away part of the earlier stage's gains.

Low-rank adapters and quantization can coexist, but their roles must be kept distinct: low-rank adapters are responsible for injecting new task-related capability into the model, while quantization is responsible for fitting the base into memory. The two do not conflict, but pay attention to whether the adapter matrix also stays low precision after quantization — if the adapter layer is high precision while the base is 4-bit, the precision difference at inference concentrates in the adapter layer, and evaluation is likely to misdiagnose adapter problems as quantization problems. A safe approach is to keep the adapter precision consistent with the base, or at least verify the two separately during evaluation.

Finally, the data distribution for fine-tuning after compression: fine-tuning the compressed model on data generated by the pre-compression model introduces a hidden distribution mismatch — the style of the data comes from a model with more capability and higher precision, while the model receiving it is already lossy. Empirically, regenerating the adaptation data with the compressed model itself usually works better, despite the extra data-generation cost.

Compressing this section into one sentence: once the compression scheme is settled, the fine-tuning is only just beginning; fine-tuning cannot rescue a compression scheme that should have been rejected, it can only add the finishing touches. That judgement avoids the most expensive kind of waste — spending a week of fine-tuning to rescue an aggressive configuration that should have been rejected on day one.

IX. How to Tell Whether Compression Was Actually Worth It

The most common failure mode of a compression project is not that the technology cannot be built, but that it was built and then nobody can articulate the payoff. Here is an evaluation method you can put straight into practice, whose core is putting quality, memory, speed, and cost into one report and closing the decision with a single net-benefit metric.

9.1 First Separate the Nominal Compression Ratio from the Effective One

Most "4× compression" figures in reports are nominal ratios: 4-bit divided by 16-bit. The effective ratio has to account for every attached overhead:

有效权重压缩比 = (基线权重字节数) / (量化后权重字节数 + scale + zero_point + 未压缩的例外值 + 未融合算子工作区)

# 分组元数据开销的快速估算（对称 INT4，每组 128 元素）
overhead_bits_per_weight = (16 + 4) / 128          # FP16 scale + 4bit 零点
actual_bits_per_weight    = 4 + overhead_bits_per_weight   # ≈ 4.156

# 2:4 稀疏的实际情况（每 4 个元素存 2 个值 + 2 位索引）
sparse_effective_ratio    = 2 * 1.0 / (1.0 + 2/16)        # ≈ 1.78x，而非 2.0x

The KV side follows the same logic: INT4 KV with group 64 plus scale works out to roughly 4 plus 20/64 bits per element, about 4.3 bits rather than 4. Writing "4×" and writing "3.7×" in a report reflect different levels of rigor, and the people doing capacity planning downstream need the latter.

9.2 Quality Evaluation: It Must Be Layered, and It Must Have a Task-Level Deciding Metric

Perplexity is the easiest metric to compute and the worst one to use as a standalone basis for decisions — its correlation with real task performance is weak, and especially after pruning and distillation, perplexity can stay completely flat while task accuracy moves noticeably. A recommended layering:

• Layer 0: perplexity. Use it only as a fast regression check to catch catastrophic errors (for example, a layer whose quantization went wrong). Never make decisions on it.
• Layer 1: general capability baseline. Fix a few broad-coverage benchmarks and measure the absolute score change relative to the baseline, not the relative percentage change.
• Layer 2: target capability. Your actual task set, weighted highest. If this layer regresses, everything in the earlier layers is irrelevant.
• Layer 3: compression-sensitive tasks. Deliberately select subsets known to be error-sensitive: long-context retrieval, multi-step reasoning, precise code generation, and problems requiring mathematical verification. KV quantization collapses on the first, structured pruning collapses on the latter two, and standard benchmarks may not detect any of it.
• Layer 4: behavioral regression checks. Not just scores but output shape: has answer length ballooned, has it started repeating, has the refusal rate risen, has format compliance broken. These are the most common and most easily missed degradation modes in compressed models.

There are at least three hard requirements for evaluation discipline: sampling parameters must be completely fixed (greedy decoding, temperature 0 — introducing sampling noise will mask the differences compression introduces); use multiple random seeds, or at minimum record the complete evaluation configuration; and the baseline and the compressed version must be measured on the same hardware, the same batch configuration, and under the same concurrency pressure — otherwise the performance numbers are not comparable.

9.3 System Evaluation: Measure Your Real Traffic Shape, Not the Lab Shape

The most common mistake in performance evaluation is measuring only single-request latency. The payoff of compression depends heavily on the load shape, so you must measure at least three points:

Test point	Configuration	What to observe	Typical conclusion
Single request	batch=1, short input	TTFT, TPOT, end-to-end latency	Weight quantization shows its clearest, nearly linear payoff here
Large batch	Batch filled, fixed length	Throughput, whether it becomes compute-bound	Weight quantization payoff decays; activation quantization only starts to contribute
Long context	Input length stretched to 8K~128K	KV memory, the slope of latency growth with length	The payoff of KV quantization scales linearly with length
Mixed real traffic	Replay the length distribution from production logs	Overall cost, p99 latency	The only number that reflects real ROI

At the same time, record memory footprint (use peak allocated memory rather than reserved memory, since the latter is inflated by the caching pool), as well as model load time and cold start — in architectures that scale up and down on demand, the startup time of a 36 GB model versus a 140 GB model is a real cost.

9.4 Net Benefit: Converging the Decision onto a Single Number

Converting quality loss into an acceptable threshold is what makes compression decisions actionable. Define a net-benefit metric:

# 净收益：节省的月成本 减去 质量退化带来的月损失
monthly_saving   = (baseline_gpu_count - compressed_gpu_count) * gpu_price_per_hour * hours_per_month
quality_loss_pct = max(0, (baseline_score - compressed_score) / baseline_score)

# 把质量损失换算成业务影响（需要按任务实际估算，这里是结构）
monthly_quality_cost = quality_loss_pct * affected_volume * value_per_request

net_benefit = monthly_saving - monthly_quality_cost

# 决策：净收益为正且质量损失低于业务容忍阈值才上线
# 注意：质量损失必须用绝对分数差判断，不能用相对百分比

The point of this formula is not the arithmetic; it is that it forces you to put a real price on "how much quality we lost." Many compression projects stop at "we saved 40% of memory" and go no further, because the moment you have to answer "is a 1.2-point drop acceptable," you discover that 1.2 points means wildly different things to different businesses — irrelevant for an internal summarization task, a problem for contract review. Choosing a compression scheme is fundamentally minimizing cost under a business tolerance constraint, not a purely technical optimization problem.

9.5 A Landing Checklist

Compressing everything above into one executable order:

1. Measure the bottleneck first: using the production length distribution and concurrency, confirm whether you are bandwidth-bound or compute-bound.
2. Do the lossless step first: if GQA is already in place, look at the KV head count and batch scheduling first, and confirm the bottleneck is the weights rather than the KV.
3. Start on the weight side at 8 bits: group-wise INT8 or FP8 is usually safe and captures most of the payoff; first confirm that the quality baseline at this step is trustworthy.
4. Then consider 4 bits: with per-group 128, calibrated on target-task data, using the GPTQ family plus channel scaling.
5. Handle long context with KV quantization as a separate track: per-token FP8 or INT8, evaluated independently, with retrieval tasks as the focus.
6. Only touch pruning when quality still has clear headroom: structured first, unstructured second, always with recovery training.
7. Put distillation last as a compensation layer, in the order "compress then distill," not the reverse.
8. Put every action into the incremental ablation table: re-run both the quality and system evaluations after each addition, and stop to re-analyze the interactions the moment any item shows a non-linear drop.
9. Record a complete configuration snapshot: quantization parameters, calibration set hash, random seeds, kernel version, driver version. The reproducibility of a compressed model is far more fragile than an ordinary model's, and if one record is missing you will basically be unable to reconstruct it months later.

9.6 The Six Most Common Crash Sites

Finally, let us collect the places most likely to go wrong in practice. Nearly every one of these has really happened, and most of them raise no exception — they just quietly make your conclusions worse.

1. Treating a framework's default as the baseline. The baseline must be an explicitly configured BF16 original-weight version, not whatever optimal configuration some framework auto-selected, otherwise the measured payoff is mixed with the framework's own optimizations and the contribution of compression cannot be attributed.
2. Using different sampling parameters for the quantized version and the baseline. When temperature is not 0, sampling noise usually exceeds the difference introduced by compression; you measure "no difference" when there is one, just buried in noise. All comparative evaluations should use greedy decoding.
3. Shipping on the basis of perplexity alone. Perplexity is insensitive to both pruning and quantization, and an obviously degraded model can have nearly unchanged perplexity. You need at least one task-level score as a gate.
4. Forgetting to evaluate long context. The gains and risks of KV quantization, attention-head pruning, and sparse attention all concentrate on long sequences, so evaluating only short context is equivalent to skipping the most important part.
5. Ignoring changes in the input length distribution. Taking a model from a 4K context to 128K when the quantization calibration only ever saw samples under 4K causes schemes that depend on calibration-time statistics to break down outside that range.
6. Treating compression as a one-time configuration. Quantization parameters, pruning masks, routing weights, and distillation data recipes are all interdependent. Once you swap the base model, change the inference framework version, or alter the calibration data, the previous evaluation conclusions no longer apply and must be re-run rather than cited.

What these six share is that none of them raises an exception; they only quietly distort the conclusions. So what a compression project usually needs most is not a more clever quantization algorithm but an evaluation process that can reproduce continuously, isolate variables, and never skip the long-context branch. Algorithm selection can absorb trial and error; distorted evaluation sends all of that trial and error in the wrong direction.

9.7 Do Not Forget That Evaluation Itself Has a Cost

A complete compression evaluation is not just running a benchmark once. Its cost is frequently underestimated, and that directly determines how many schemes you can measure and how reliable the conclusions are. These ledgers must be computed in advance.

• Time cost: the absolute duration of one full evaluation determines how many controlled comparisons you can run. If a single evaluation takes a full day, then running it three times with different seeds takes three days, and the scheme iteration speed becomes too slow to converge. The practical approach is to use a small subset for fast screening and let only the shortlisted schemes run the full evaluation, rather than pushing every scheme through the same complete pipeline.
• Layering cost: layer 0 perplexity takes minutes, while a layer 2 target-capability evaluation can take hours. Mixing the two layers makes iteration extremely expensive. The point of layering is not just a cleaner evaluation logic but specifically letting cheap criteria do the elimination and expensive criteria make the final call.
• Reproducibility cost: evaluation of a quantization scheme is sensitive to randomness, which comes from how the calibration set is split and from sampling at inference time. When running the same scheme twice yields different conclusions, you cannot fix it by running it once more and keeping the better result — that is selective reporting. Either fix the sources of randomness, or run enough repetitions to give a confidence interval.
• Maintenance cost: only an evaluation that records quantization parameters, calibration set hashes, kernel versions, framework versions, and driver versions can be reproduced months later. An upgrade to any layer of the compression stack can change the conclusions, so the evaluation environment's versions must be versioned together with the model configuration; otherwise every cross-time comparison is invalid.

Once these ledgers are computed, one realistic but frequently overlooked conclusion emerges: you do not need to measure every possible compression combination. The ordering given in section ten is itself a search path sorted by prior payoff — run the high-probability combinations first and reserve the evaluation budget for the genuinely ambiguous branch points. That is faster than exhaustive enumeration and much more likely to converge on a reliable conclusion. Put the other way round, a project with only three days of evaluation budget should proactively cut the number of schemes rather than the rigor of the evaluation, because rigor is the one thing that must never be cut.

9.8 After a Compressed Model Ships: Maintenance and Drift

A compression configuration is not a one-time deliverable; it is a configuration that has to be looked after for a long time. What actually happens after launch is usually not that the model suddenly breaks, but that the conclusions slowly stop holding and nobody notices. The following categories of drift deserve explicit monitoring in the process.

• Numerical drift from inference stack upgrades. Kernel implementations, reduction orders, and accumulation precision all change with framework or driver versions. The same weights producing slightly different results on different versions is normal; but if the difference grows monotonically over time, the numerical semantics of some stage have changed and you need to re-evaluate rather than keep citing the old conclusion.
• Invalidation from upstream weight updates. If the base model is replaced or further trained, the quantization parameters, pruning masks, and distillation data recipes in the compression configuration are all void — they were fitted to specific weights. You must record and verify a separate compression configuration for each weight version and cannot just inherit the previous conclusion by assumption.
• Silent degradation from traffic distribution drift. A compression scheme is calibrated and evaluated on some input distribution. The length, language, and task-type distribution of live traffic changes, the numeric range of activations shifts with it, and the quantization parameters drift away from the assumptions they were calibrated under. This kind of problem never shows up in offline evaluation (because the offline distribution did not change either), and can only be caught by sampling and back-testing against online traffic.
• How quality drift should be observed. The online quality decline of a compressed model is usually gradual: output becomes verbose, starts repeating, produces non-compliant answers on borderline questions. None of that requires labeled data to detect — periodic manual sampling plus monitoring of output length distribution, repetition rate, and refusal rate usually catches problems earlier than waiting for a business metric to drop.

The corresponding practical advice is threefold: version-manage the compression configuration together with the weight version, re-running the evaluation whenever either changes; add a regression evaluation stage to the release process for inference stack upgrades, putting it behind the same gate as functional testing; and periodically back-test by sampling against the real online distribution, even when offline metrics are unchanged, because the distribution itself has moved.

The shared logic behind those three is that a compression scheme's validity depends on three implicit assumptions — the weights have not changed, the inference stack has not changed, and the input distribution has not changed. Break any one of them and the original evaluation conclusions no longer hold automatically. Writing those three assumptions explicitly into the launch checklist is far cheaper than investigating afterwards.

X. Conclusion: Compression Is a Set of Decisions, Not a Technique

Back to the opening judgment: compression saves storage and data movement first, and only turns into speed when the bottleneck is bandwidth; weight quantization does not touch the KV Cache, while activation quantization is what touches FLOPs; distillation is the only technique effective across all dimensions, and also the most expensive. These three judgments determine every other choice.

Concretely, at the level of methods: quantization is the most universal tier, its knobs being bit width (how much you save) and group granularity (whether the saving is worth it), and the differences between modern PTQ methods come mainly from whether activation-aware channel scaling and rotation preprocessing were applied; GPTQ and AWQ answer questions at different levels and are frequently stacked rather than chosen between. Pruning succeeds or fails based on the hardware format, 2:4 being the only form broadly supported by current hardware and structured pruning the only form that genuinely saves compute, and both must be paired with recovery training. Distillation is capped by data construction, with data from the capability-gap region providing the only effective learning signal, and on-policy methods being more targeted than offline ones.

And when the three are stacked, there is only one conclusion that matters mechanically: because distillation performs supervised compensation after lossy compression, the order should be "prune → recovery train → distill → quantize," with quantization last. Because it has no compensation mechanism, the later you quantize the harder the loss is to recover; and pruning must come before quantization, because their group granularities conflict.

Finally, on how to tell whether something was "really worth it," there is only one genuine answer: use your real traffic shape, your real task distribution, and your real business tolerance to compute that one net-benefit number. Any compression report that does not land on this number is still just a technology demo. Real engineering decisions were never about "can we compress," they are about "is the quality we gave up worth the compute we saved."