MoE Architectures in Practice: Routing, Load Balancing, and Expert Parallelism

Preface: MoE Has Gone from "Saving Compute" to "Changing the Communication Topology"

If you read a mixture of experts (Mixture of Experts, MoE) model as simply "lots of parameters, with only a small slice used each time," that reading was roughly correct before 2020 and is badly out of date today. What actually made MoE the default architecture for mainstream large models is not that it is cheap. It is that it incidentally decouples three things that used to be tangled together: total parameter count can keep growing independently of activated compute per token; expert weights can sit on different devices, buying near-linear capacity scaling through all-to-all communication; and the shared parts (attention, embedding, shared experts) stay dense, preserving the stability of general capability. Only when you combine all three do you get structures you can read directly out of public configs, like DeepSeek-V3's 671B total / 37B activated, Qwen3's A3B and A22B variants, or Mixtral 8x7B.

The other side of the coin is just as clear: MoE turns a "pure compute problem" into a compound problem of communication, scheduling, and routing stability. During training you have to deal with all-to-all stalls, capacity overflow, token dropping, and the auxiliary loss fighting the main task loss over gradients. During inference you have to deal with routing jitter, batch size perturbing the routing distribution, quantization error amplifying gating logits, and the inconsistency that comes from the same passage of text selecting different experts at training and inference time. Get any one of these wrong and you eat most of MoE's benefit.

So this article will not retell the history of MoE and will not repeat any specific model's architecture walkthrough. What I want to do is break MoE into six decision points you will actually hit in engineering: where the intrinsic payoff of sparse activation comes from, the math of top-k routing and the small moves you must make, the hidden tax of the capacity factor and token dropping, the gradient conflict between the load balancing loss and the main task, plus three alternative routes, why shared experts are increasingly common, and the communication bill for expert parallelism and all-to-all — before landing on inference-side stability and a set of engineering trade-off calls. Every subsection ends in a definite recommendation rather than a list of possibilities.

One boundary worth stating up front: "MoE" here specifically means the sparse MoE feed-forward layer (Sparse MoE FFN) in a Transformer, replacing each block's FFN with several parallel FFNs plus a router. Attention stays dense with no token-dimension sparsification. That sparsification route (dynamic head selection, head pruning, block-level sparsity) is a different engineering problem, and this article does not cover it.

Dense vs. Sparse: The Difference Is Not "Fewer Parameters," It Is Gradient Update Density

Let me start with the distinction people most often confuse. The difference between dense models and sparse-activation models has never been "who has more parameters." It is how much gradient signal each parameter receives at each step. In a dense model every parameter participates in the forward and backward pass of every token, so a parameter count P corresponds to roughly 2P floating-point operations per token, and all P weights must be resident in memory. In an MoE, total parameters P is far larger than activated parameters A, each token goes through only k experts, and P corresponds to weight storage while compute scales with A.

The first direct consequence: MoE's cost-performance advantage comes from decoupling capacity scaling from compute scaling. If you want more capability under a fixed per-token FLOPs budget, the dense route can only get wider and deeper, growing parameters and compute together. The MoE route can add experts alone, inflating total parameters while leaving activated compute essentially untouched. That is why nearly every model chasing "lots of parameters, low latency" after 2024 picked MoE.

The second consequence is easy to miss: MoE's compute savings are real, but its memory and communication overhead goes up. That means "MoE is cheaper" means two entirely different things in training and inference. In training, MoE optimizes for throughput (tokens/s) and communication efficiency, because total compute roughly follows activated parameters; MoE wins by letting you reach better convergence with more parameters. In inference, the objective becomes latency and memory, and MoE pays extra on both: all expert weights must stay resident, and you still pay for all-to-all at runtime. A common result follows from this: MoE's per-request latency is usually no better than a dense model with the same activated size, and the advantage shows up in high-throughput, long-context, and batch-processing scenarios.

Keep the Three Ledgers Separate

The most underestimated row in that table is optimizer state. With AdamW, every parameter additionally keeps an fp32 master weight plus first- and second-moment buffers — roughly a dozen-plus bytes per parameter, so for a 671B MoE the optimizer state alone during training is on the order of tens of TB. That is why MoE training must pair expert parallelism with optimizer-state sharding and possibly CPU offload; there is no third path. Inference is much cleaner: no optimizer, just bf16 weights plus the KV cache.

There is another intuitive difference: MoE capacity growth has a diminishing-returns knee. When the expert count goes from 8 to 64, the number of input tokens per expert drops quickly, the per-expert data distribution becomes sparse, and the router's learning signal weakens — while total parameters inflate in step and communication cost rises linearly. "Just add more experts" is therefore not an action you can apply without thinking. That is the granularity problem I will come back to later.

The Math of Routing: Softmax, top-k, and the Small Moves You Must Make

The forward pass of an MoE layer splits into four steps: the router computes logits, softmax turns them into gating probabilities, top-k selects experts, and the selected probabilities are renormalized to serve as the weighting coefficients for the expert outputs. Every one of those steps has a trap, but two details that are easy to get wrong decide whether it works at all: you must renormalize after top-k, and you must control the scale of the routing logits.

Renormalization first. If you take the top k softmax probabilities as weights directly, without renormalizing, the actual signal strength each token sends into its experts swings wildly across tokens: with k=2 you might get (0.6, 0.25) or (0.31, 0.28), and in the latter case both experts receive signal attenuated to roughly thirty percent. Early in training this fluctuation destabilizes each expert's effective learning rate; late in training it leaves some experts undertrained. The standard fix is to renormalize those k probabilities to sum to 1, so every token still contributes exactly one full-size signal.

Now the logit scale. The router is a linear layer whose output goes straight into softmax. As training progresses the gating logits tend to grow in magnitude (the model spontaneously makes routing decisions more certain), softmax approaches one-hot, and the gradient shifts from "which experts are more likely" to "only the selected expert receives routing gradient." Two things follow: exploration is too weak early on, and once an expert is selected it becomes more likely to be selected again, a self-reinforcing Matthew effect. That is why many implementations add a little Gaussian noise (jitter) to the logits during training, forcing exploration open explicitly.

import torch
import torch.nn as nn
import torch.nn.functional as F


class TopKRouter(nn.Module):
    """top-k 路由器：门控 -> 加噪 -> softmax -> 选专家 -> 重归一化 -> 容量裁剪。"""

    def __init__(self, d_model, n_experts, top_k=2, capacity_factor=1.25,
                 jitter_noise=0.0):
        super().__init__()
        self.n_experts = n_experts
        self.top_k = top_k
        self.capacity_factor = capacity_factor
        self.jitter_noise = jitter_noise
        # 刻意不加 bias：常数偏置会让少数专家凭空拿到系统性优势
        self.gate = nn.Linear(d_model, n_experts, bias=False)

    def forward(self, x, drop_tokens=True):
        # x 形状 [T, D]，T 是 batch 与 seq_len 展平后的 token 总数，
        # 这样容量、派发、回收三步都只跟 T 有关，与 batch 切分无关
        logits = self.gate(x)                                     # [T, E]

        # 训练期噪声，压制 softmax 过早饱和，保证早期能探索到冷门专家
        if self.training and self.jitter_noise > 0.0:
            logits = logits + torch.randn_like(logits) * self.jitter_noise

        probs = logits.softmax(dim=-1)                            # [T, E]
        topk_w, topk_i = probs.topk(self.top_k, dim=-1)           # [T, k]

        # 最关键的一步。不做重归一化，不同 token 送进专家的总信号
        # 会在 0.2 到 1.0 之间乱跳，每个专家的有效学习率跟着抖动
        topk_w = topk_w / topk_w.sum(dim=-1, keepdim=True)

        # 容量按期望负载定，而不是按实际最大负载定，
        # 这样相邻两批之间容量保持确定，不会因运气好坏而变
        expected = self.top_k * x.shape[0] / float(self.n_experts)
        capacity = max(int(expected * self.capacity_factor), self.top_k)

        # rank[i, e] 表示 token i 是第几个把专家 e 选进来的，
        # 只有 rank 为 0 的那条记录拿到处理权，其余进入丢弃集合
        one_hot = F.one_hot(topk_i, num_classes=self.n_experts)    # [T, k, E]
        rank = one_hot.cumsum(dim=1) - 1                          # [T, k, E]
        keep = (rank.squeeze(1) == 0).gather(1, topk_i)           # [T, k]

        return topk_i, topk_w, keep, capacity

Three engineering details in that code deserve their own note. First, the router carries no bias, and that is not an accident: a bias gives every expert a constant logit offset, so early in training an expert whose bias happens to be large gets selected continuously and updated continuously, and the bias itself gets updated too — positive feedback that makes the routing distribution very hard to flatten. Second, capacity is computed from the expected load rather than the actual peak load, namely k * T / n_experts times the capacity factor. Capacity is therefore deterministic across batches and does not jitter based on how lucky the previous batch was. Third, dropping uses a rank == 0 test rather than random dropping, so the tokens kept for a given expert are the ones that claimed a slot first, and the behavior is reproducible.

As for choosing k, k=1 and k=2 are two different things. k=1 is the Switch Transformer approach: routing is a pure argmax with no weight mixing, expert outputs are simply summed, and the structure is the simplest with the easiest balancing — but each token gets only one expert's expression, which caps expressional diversity. k=2 lets two experts each contribute a weighted result, which is more expressive; Mixtral 8x7B uses this setting and it is now the more common default. In practice k=4 and above shows clearly diminishing returns, because the usable capacity gets diluted and each expert sees too little data. There is one engineering compromise worth knowing: Expert Choice Routing. Instead of letting tokens pick experts, each expert picks the top-C tokens within its own capacity. The upside is naturally perfect balance with no token dropping at all; the downside is that routing no longer satisfies causality — a token's assignment depends on the whole batch, so it cannot be used directly in autoregressive or streaming settings. That is why it basically only shows up in encoder-style work or exploratory training-time research.

There is one more aggressive variant worth knowing: the router is not left in place but pushed down a layer. DeepSeekMoE proposes splitting a standard FFN along its intermediate dimension into several segments, each with a router that shares the input plus one fine-grained expert. DeepSeek-V3 builds on this by setting the routed expert count to 256, adding one shared expert, giving each expert only 8 routed tokens, and replacing the auxiliary loss with a sequence-level bias term. These are structural facts from public configs and involve no benchmark numbers. The engineering implication is that you can run a very large expert count with very small individual experts, because routing balance no longer depends on a loss term bending things into shape; it comes from a bias term performing a gradient-free balance correction after every training step.

The Capacity Factor, Token Dropping, and a Hidden Tax You Cannot Fully Account For

The capacity factor exists to handle one statistical fact: the router is not perfectly balanced. Even when the training objective explicitly requires equal load across experts, the actual token distribution still fluctuates, and a popular expert will over-collect tokens in a given batch. The engineering response is to give each expert a hard cap of C = capacity_factor * k * T / n_experts and drop every token above it. That is token dropping.

The cost of token dropping is often written up as "losing a bit of compute," which understates it. First, a dropped token is not simply "one FFN less" — it still runs through the router, still occupies a slot in the all-to-all send, and still participates in every embedding and attention computation; it simply no longer produces an FFN output. In other words, in every training step the drop_ratio fraction of tokens is pure waste: they pay the full routing and communication cost and get no FFN contribution in return. Second, a dropped token is not simply ignored in the loss: in the Switch Transformer implementation the FFN branch output for a dropped token is zeroed, so that token's supervision signal is effectively weakened and the main task gradient shrinks along with it.

"Don't set the capacity factor too high" is the right instinct, but the common explanation is not accurate enough. The real reason is: the larger the capacity factor, the more the system depends on the router to train itself into balance. If you push the capacity factor to 1.5, you are tolerating 50% over-assignment without dropping a token — so when an expert suddenly turns popular because of a gradient anomaly or a data distribution shift, you absorb it with more memory and compute instead of letting the balancing loss correct it. On the surface, a larger capacity factor looks more "lossless," but what it actually does is turn load imbalance from "force-corrected" into "masked by compute." Keep masking until some batch's routing distribution shift exceeds the margin, and token dropping jumps abruptly from near zero to a noticeable fraction, with hard-to-explain jitter in the training curve.

The range that shows up in practice is roughly this: for k=1 (Switch style), where only one expert receives each token and fluctuation is larger, the capacity factor often needs to be 1.25 to 1.5; for k=2, 1.0 to 1.25 is enough; for fine-grained experts with very small per-expert capacity (say 8 tokens per expert), the capacity factor approaches 1.0 and may have to be exactly 1.0, because single-expert statistical fluctuation is inherently large at low counts and capacity must be widened to guarantee no drops, while the large expert count still keeps the global picture balanced. These numbers are not theorems but the bands common configs land in. The real practice is to monitor the drop ratio in your own training logs, hold it stably below 1%, and only then push the capacity factor down toward the true optimum.

There is also an easily missed benefit: dropping some tokens is itself a form of regularization. It forces the router to learn to send certain tokens to the "second-best but acceptable" expert instead of hammering on the single optimal expert for every token, which relieves overfitting and expert collapse to some degree. When you watch a training trajectory and see the drop ratio pushed to almost zero while the variance of the expert load distribution is also almost zero, that is not necessarily a good sign. It may mean the task distribution is too homogeneous, or that the balancing constraint is so strong the router is no longer making any meaningful discrimination.

The Load Balancing Loss Has a Real Gradient Conflict With the Main Task

The idea behind load balancing is very plain: add an auxiliary term next to the main loss that penalizes how far each expert's token count deviates from the mean. Switch Transformer uses a load balancing loss — for each expert i, define P_i as the fraction of tokens assigned to that expert in the current batch and f_i as the fraction actually routed to it; the auxiliary loss is n_experts * sum(P_i * f_i) over all experts, and under a perfectly even split every term is 1/n_experts and the total is 1, so you only have to minimize it. Its virtues are simplicity, no extra communication, and availability in essentially every implementation.

The problem is that this term and the main task objective are being optimized in different spaces. The main task wants "fit the current input well"; the auxiliary loss wants "the router hands tokens to all experts evenly." In a lot of cases those two are in conflict: an expert becoming the expert of some semantic domain and specializing in one class of token means precisely that its token count is uneven — and that unevenness is exactly the specialization we want. The auxiliary loss will keep pulling that specialization back, persistently and tirelessly, which shows up as slower convergence, increasingly similar input distributions across experts, and eventually all experts converging on each other.

More precisely, the conflict shows up on two levels. The first is directional conflict: the auxiliary loss gradient only acts on the router's gate parameters, while the main task FFN gradient acts on both gate and expert weights. When the two point in opposite directions, the gate ends up in a gradient tug-of-war and the expert weights oscillate along with it. The second is magnitude conflict: the auxiliary loss coefficient alpha has to be tuned by hand, and it must be large enough to suppress the main task's balancing pressure yet small enough not to destroy specialization. Too large and the experts homogenize, with the MoE degenerating into "several copies of the same FFN"; too small and load variance grows, the drop ratio climbs, and real throughput falls. The high tuning cost and poor reproducibility are the most practical pain points of this route.

import torch
import torch.nn as nn
import torch.nn.functional as F


def load_balance_loss(probs, indices, top_k, n_experts):
    """Switch 风格的均衡损失，理想均分时返回值恰好等于 1。"""
    # P 是期望份额：全体 token 的门控概率在专家维度上的均值
    P = probs.mean(dim=0)                        # [E]
    # f 是实际份额：top-k 硬分配之后每个专家真正收到的比例
    mask = torch.zeros_like(probs)
    mask.scatter_(1, indices, 1.0)
    f = mask.mean(dim=0)                         # [E]

    lb = n_experts * torch.sum(P * f)
    # 额外加一份零方差项，防止少数专家被彻底饿死而不报警
    lb = lb + n_experts * torch.sum(P * P)
    return lb


class BiasBalancedRouter(nn.Module):
    """辅助损失-free 的均衡：给每个专家一个不参与反向的标量偏置。

    核心洞察是：偏置只改变选中谁，不改变选中时的概率与权重，
    因此它能修正负载分布，却不会污染主任务在门控上的梯度方向。
    """

    def __init__(self, d_model, n_experts, top_k):
        super().__init__()
        self.gate = nn.Linear(d_model, n_experts, bias=False)
        self.correction_bias = nn.Parameter(
            torch.zeros(n_experts), requires_grad=False)
        self.top_k = top_k

    @torch.no_grad()
    def update_bias(self, per_seq_counts):
        """按序列统计的负载来修正偏置，避免长序列主导整个 batch。"""
        score = per_seq_counts.float().mean(dim=0)          # [E]
        delta = score.mean() - score                        # 超载压低，欠载抬高
        self.correction_bias += torch.sign(delta) * 1e-3
        self.correction_bias.clamp_(-1e-2, 1e-2)

    def forward(self, x):
        logits = self.gate(x)
        probs = logits.softmax(dim=-1)                      # [T, E]
        # 选专家看加偏置的分布，定权重仍看原始分布，两者解耦
        biased = probs + self.correction_bias.view(1, -1)
        topk_i = biased.topk(self.top_k, dim=-1).indices   # [T, k]
        topk_w = probs.gather(1, topk_i)                   # [T, k]
        topk_w = topk_w / topk_w.sum(dim=-1, keepdim=True)
        return topk_i, topk_w

What this code shows is the most representative alternative route today: replacing the auxiliary loss with a gradient-free bias term. Its core insight is that routing decisions and routing weights can be split apart — the bias is applied only at the "who gets selected" step, while softmax probabilities and output weighting still use the original logits. Balancing can then take effect without polluting the gradient direction of the main task. This is exactly the mechanism described publicly for DeepSeek-V3, which additionally performs the bias update along the sequence dimension so that long passages inside a batch do not dominate the statistics.

Beyond the bias method, there are a few alternative routes worth knowing. z-loss penalizes the absolute magnitude of the routing logits rather than their distribution; it suppresses softmax saturation, is orthogonal to the balancing objective, and is often used together with the auxiliary loss as a cheap stabilization measure. Entropy regularization directly encourages the gating distribution to keep enough entropy — essentially "don't decide too early" — and suits early-stage training. Expert Choice Routing trades probabilistic balancing for hard assignment; it is the most thorough option but carries the causality problem and can only be used in non-autoregressive settings. Restricting the routing range is a topology-level trick: run one local balancing pass inside each node, then pass only the balanced token distribution between nodes, turning the global all-to-all balancing problem into a local one — the node-limited routing described publicly for DeepSeek-V3 is exactly this idea. Which route to take mainly comes down to your autoregressive/streaming constraints: encoder-style models can consider Expert Choice, while autoregressive models are basically limited to the bias method and other tricks that leave distribution statistics untouched.

Shared Experts: An Escape Hatch That "Allows Imbalance"

The idea behind a shared expert is very direct: alongside the n_routed routed experts, attach one (or a small number) of ordinary FFNs that are always active and receive every token. Its output is added or concatenated to the weighted output of the routed experts. DeepSeekMoE was the first to argue for this structure systematically, DeepSeek-V2 and V3 both adopted it, and the configuration you commonly see in public configs is 256 routed experts paired with 1 shared expert.

Shared experts resolve the contradiction that kept recurring earlier. Without shared experts, every unit of load-balancing pressure the router exerts has to be paid back by weakening specialization — because letting all tokens of some domain rush to a single expert is not allowed, the auxiliary loss keeps flattening the assignment, while specialization requires imbalance. The shared expert pulls out "the general transformation every token has to pass through," so the routed experts can safely diverge: since general capability is backstopped by the shared expert, the routed experts do not each need to become a generalist — they can separately learn specialized representations like code, math, or natural language. The pressure the balancing loss applies to the routed experts is therefore much smaller.

From a gradient standpoint, shared experts have a subtler role: they provide a gradient sink that cannot collapse. Under severe load imbalance, half of the routed experts may go a long time without receiving tokens and barely update, while the shared expert is trained by all tokens at every step, which gives the optimizer a stable descent direction. When the routing distribution drifts abnormally at some stage, the presence of a shared expert means the model still has at least one non-degenerate path. That is also why it is kept as a fixed configuration rather than an option in many MoE models.

The cost is real, of course: the shared expert is dense, its activated compute cannot be sparsified away, and it amounts to one extra full FFN per token. Making the shared expert smaller than the ordinary experts is therefore the common choice — under fine-grained expert architectures the shared expert is usually sized as a medium expert rather than matching the routed ones. The test is simple: take the shared expert out, run a short controlled comparison, and if no metric degrades noticeably then that dense compute was paid for nothing. The ablation reported in DeepSeekMoE concludes that shared experts give a consistent small improvement at the same parameter budget, and that the improvement is larger when the expert count is higher.

One more boundary needs to be spelled out: shared experts cannot replace balancing. They relieve balancing pressure but do not remove it. If the routed experts' load variance is still large, you still have to manage it with the bias method or the auxiliary loss. Treat shared experts as "a structural means of lowering the difficulty of balancing," not "a switch that exempts you from balancing."

Router Design Variants: The Gate Does Not Have to Be a Linear Layer

Everything discussed so far rests on one premise: the router is a single linear layer plus softmax. That premise is good enough, but knowing the variants helps you judge when to switch and what the switch costs.

Noisy top-k is the most intuitive change: instead of adding Gaussian noise, multiply each expert's gating score by multiplicative noise sampled from a log-normal distribution — the larger the noise, the more thorough the exploration. The benefit is that the noise acts multiplicatively against the probability scale, which naturally avoids pushing some expert's score into an unreasonable region. Compared with additive noise, its training stability is usually better, at the price of sampling every step; the sampling must be disabled for validation and inference, and results across different noise strengths cannot be compared directly.

Hash routing goes to the opposite extreme: no learning at all, just map a hash of the token to a fixed expert. Its value lies not in quality but in determinism and cost — no gating parameters, no balancing loss, no routing drift. It is excellent as a control baseline and is often used in specific architectures to spread experts evenly. Its drawback is just as clear: no semantic specialization can form across experts, and its quality ceiling sits clearly below learned routing.

Low-rank gating targets the problem of "too many gating parameters when the expert count is large." You first project the hidden state into a low-dimensional space, compute the gate in that low-dimensional space, then project back to the expert dimension. DeepSeekMoE uses exactly this kind of low-rank projected gate. The gain is that the gating parameter count drops from "expert count times hidden dimension" to "the cost of two small matrices," which is a considerable saving in configurations with a very large expert count. The cost is that gating expressiveness is capped by the low-rank bottleneck: if the expert count keeps growing and the rank is not enough, routing quality degrades.

Deeper gating networks (a small two-layer MLP, say) can express much more complex routing policies and in principle give experts a more structured division of labor. But they bring two practical problems: higher compute overhead for the gate, and easier overfitting to the training data distribution. The practical judgment goes like this: if a linear gate already produces unusually high output similarity between experts (meaning experts are duplicating each other's work), switching to a deeper gating network often genuinely helps; if similarity is normal and the division of labor is already clean, deepening the gate is mostly wasted parameters.

The routing of shared experts can vary too. So far I described it as a dense branch that constantly receives every token, which is the most common approach. There are also designs where shared experts participate in gating, or where several shared experts of different sizes take inputs at different granularities. The former weakens the shared expert's role as "an unbiased gradient sink"; the latter reintroduces the balancing problem. The value of shared experts lies precisely in the fact that they are not balanced — the moment you put them under balanced scheduling, their purpose disappears. So do not make extra changes here for the sake of "structural uniformity."

One general principle when picking a router variant: diagnose first, then change structure. When the division of labor is confused, the problem is usually the gate's expressiveness or insufficient exploration; when load is unbalanced, a deeper gate rarely helps and the balancing mechanism is what needs to change. Keeping those two concerns separate is the key to avoiding ineffective tuning.

Expert Parallelism and all-to-all: Communication Is the Real Wall

By this section MoE's engineering focus has clearly shifted from algorithms to systems. A dense model's tensor-parallel pattern is fixed: every layer is split row-wise or column-wise, the forward and backward passes go through all-reduce or reduce-scatter, and both the communication volume and the shapes can be derived statically. Expert parallelism in MoE is different — which expert each token goes to is data-dependent and only determined at runtime — so you must use all-to-all to ship tokens to the device holding the corresponding expert, compute, and all-to-all again to bring the results back. A single MoE layer needs at least two all-to-alls in the forward pass and two in the backward pass, and real implementations do more because the backward pass also needs routing information.

import torch
import torch.distributed as dist


class ExpertParallelDispatch:
    """把 [T, D] 的 token 按专家分片送到对应设备，算完再收回来。

    设备 r 持有 experts_per_rank 个专家。all-to-all 之后本地收到的是
    别的设备上、属于本地专家的那些 token，顺序与分桶索引一一对应。
    """

    def __init__(self, n_ranks, n_experts):
        assert n_experts % n_ranks == 0, "专家数需要被并行度整除"
        self.n_experts = n_experts
        self.experts_per_rank = n_experts // n_ranks

    def dispatch(self, x, topk_i, topk_w):
        T, D = x.shape
        k = topk_i.shape[1]

        # 每个 token 有 k 条路由记录，先展平成 [T*k] 这一条长序列
        flat_i = topk_i.reshape(-1)
        flat_w = topk_w.reshape(-1)

        # 目标设备 = 专家编号 // 每设备专家数
        dst_rank = flat_i // self.experts_per_rank

        # 按目标设备分桶，桶内保持稳定序；这一步的排布直接决定
        # 后面能不能用 grouped GEMM 一次算完整层的专家
        order = torch.argsort(dst_rank, stable=True)
        counts = torch.bincount(dst_rank, minlength=dist.get_world_size())
        send = x[order // k] * flat_w[order].unsqueeze(-1)

        # 关键前置步骤：接收端必须先知道要收多少字节，
        # 因此要把每张卡的负载计数交换一次，这是一次额外同步
        input_splits = counts.cpu().tolist()
        recv_splits = self._exchange_counts(counts)

        recv = torch.empty(sum(recv_splits), D,
                           dtype=x.dtype, device=x.device)
        dist.all_to_all_single(recv, send.contiguous(),
                               output_split_sizes=recv_splits,
                               input_split_sizes=input_splits)
        return recv, recv_splits, order

    @staticmethod
    def _exchange_counts(counts):
        buf = [torch.zeros_like(counts) for _ in range(dist.get_world_size())]
        dist.all_gather(buf, counts.to(dist.get_backend()))
        return buf[dist.get_rank()].cpu().tolist()

    def combine(self, out, order, T, k, D):
        """把专家输出按原 token 顺序加权求和，送回它原本所在的设备。"""
        recv = torch.empty(T * k, D, dtype=out.dtype, device=out.device)
        dist.all_to_all_single(recv, out.contiguous(),
                               output_split_sizes=[T * k],
                               input_split_sizes=None)
        x = torch.zeros(T, D, dtype=out.dtype, device=out.device)
        x.index_add_(0, order // k, recv)   # 权重在派发时已经乘过
        return x

The two "non-compute" parts of that code are the most notable. First, load counts must be all-gathered first, because before the all-to-all the receiver does not know how many bytes it is about to get. That is one extra synchronization, and for small messages its share of the overhead can be substantial — many frameworks fuse it into the first all-to-all, or avoid it with an estimate-then-verify scheme. Second, the sort order determines kernel performance: only when tokens belonging to the same expert are laid out as one contiguous run can grouped GEMM compute every expert of the layer in a single pass. Without grouping you fall back to N separate small GEMMs and GPU utilization gets cut roughly in half. Details like these about data layout often determine actual throughput more than the mathematics does.

On communication topology, MoE is far more sensitive to interconnect than dense models. Expert parallelism's all-to-all is a cross-node, fully connected, irregular exchange: every card has to send a different number of tokens to every other card simultaneously, and those numbers change every step. Inside an NVLink domain it is tolerable, but once it crosses between nodes over the network it easily becomes the bottleneck, because network bandwidth is typically an order of magnitude below NVLink. EP degree cannot simply equal the total GPU count; the more common engineering practice is to confine EP within an NVLink domain (say 8 cards per domain), run EP inside the domain, and use DP or TP for the activation and attention layers across domains — or to use libraries such as DeepEP that specifically optimize MoE communication to cut small-message and cross-node costs, routing token transfer and low-precision KV/activation transfer down separate paths so their bandwidths can be mixed, and packing per-token RDMA writes more tightly.

Another easily missed fact: the smaller the MoE batch, the worse the communication efficiency. The data each participant sends in an all-to-all is proportional to the number of tokens in the batch; when there are too few tokens the messages fragment badly and latency takes a dominant share. That is why MoE training usually takes a very large global batch (a sequence count times sequence length in the millions of tokens), and why MoE suits large-batch serving rather than low-concurrency online traffic.

There is also a forward/backward asymmetry worth noting: the forward all-to-all moves several times the data of the backward one, because the backward pass must send gradients to every device involved and also re-broadcast the routing indices and gating weights saved during forward. The approach described publicly for DeepSeek-V3 has the backward pass return activation gradients, gating weights (for recomputing the gate) and routing indices together, compressing two or three backward communications into one. Libraries such as Tutel and MegaBlocks are doing similar communication fusion — MegaBlocks folds the capacity dimension and the drop dimension into the sparse structure via block-sparse GEMM, so the layer's capacity factor does not affect the compute volume.

Splitting Experts Too Fine Makes Things Worse: A Counterintuitive but Very Stable Rule

"More experts means stronger" is the most widely circulated misunderstanding about MoE. The reality is that expert granularity has a sweet spot, and past it the returns decay fast or even turn negative. There are four reasons, all quite concrete.

First, each expert sees too few tokens. When the total token count is fixed and the expert count doubles, each expert's token share is halved. FFN parameters need enough data to train; with too few samples an expert learns nothing and merely fits a tiny local pattern, so capacity utilization is extremely low. In engineering practice you see the output distributions across experts begin to converge — because each expert is confined to an extremely narrow input distribution, and several narrow distributions stacked together yield less overall capability than one slightly wider distribution.

Second, communication and scheduling overhead grow with the expert count. Doubling the experts means more fragmented all-to-all buckets, smaller messages, more grouped GEMM groups with less compute in each, and GPU efficiency drops accordingly. This overhead is hard, and it does not get amortized away by a larger model.

Third, the router becomes harder to train. The output dimension grows linearly with the expert count, the gating parameter count grows with it, the softmax space gets larger, and searching for the optimal routing gets harder with slower early convergence. What you observe is that high-expert-count runs descend more slowly at the beginning and need more tokens just to catch up with the final level of a coarse-grained configuration.

Fourth, the payoff of fine granularity depends on the router itself being strong enough. The core assumption behind fine-grained experts is that the router can discriminate finely enough for each token to find the right small expert. If the router's discrimination does not keep up, fine granularity merely chops an already underpowered assignment problem into smaller pieces and adds no new information. DeepSeekMoE's experimental design contains one crucial control: hold the total parameter count fixed and vary only expert granularity, so the contribution of granularity itself is isolated. Many reproductions ignore this, which leads to miscounting a parameter change as a granularity win.

So where does the sweet spot sit? Practical experience says: coarse granularity (few large experts) maps to low-to-medium scale, fine granularity (many small experts) maps to large scale. Typical coarse-grained configurations are Mixtral's 8 experts or DBRX's large-scale few-expert route; typical fine-grained configurations are the publicly disclosed 256 routed experts of DeepSeek-V3, each receiving only 8 routed tokens. The most direct metrics for judging whether the granularity is right are the effective per-expert token utilization and the pairwise similarity of expert outputs: if similarity is broadly high, experts are doing duplicate work; if utilization is low, large amounts of expert parameters are sitting idle. Both metrics give you a signal earlier than the loss curve does.

There is one compromise route worth noting, mixed granularity: different expert counts across layers, or within a single layer some coarse-grained experts handling general features while fine-grained experts handle specialized ones. This is reasonable in principle, but in implementation it complicates the all-to-all bucketing logic and may not pay for itself — unless you already have a mature EP framework and the data distribution genuinely shows multi-layer heterogeneity.

Scale Drift Between Experts: A Silent but Lethal Problem

MoE has a hidden problem that dense models do not have: the output scale of different experts drifts apart as training proceeds. In a dense model every token flows through the same set of parameters, so the statistical distribution of activations converges as a whole. In an MoE each expert handles only a small share of tokens, and if one expert's input distribution is narrow (because it handles one particular class of token) its output distribution is narrow too — and when several narrow distributions are stacked in a residual add, the scale mismatch gets amplified.

Scale drift has three direct consequences. The first is training instability: an expert emits an abnormally large magnitude at some step, and after the residual add it pushes the whole layer's activations into a very large range, skewing normalization in later layers and potentially producing loss spikes. The second is easier routing collapse: an expert with a large output amplitude gains more authority in the residual stream, tends to attract more tokens, and further aggravates the imbalance. The third is degraded quantization precision: scaling factors in low-precision quantization are usually computed from global or per-tensor statistics, so if one expert's magnitude is ten times the others', the entire tensor's quantization step has to accommodate it and every other expert wastes precision for nothing.

There are several ways to handle it, ordered by cost. The lightest is to keep the normalization structure inside each expert FFN, ensuring all experts are parameterized the same way, so their output scales naturally sit in the same order of magnitude instead of drifting apart because of structural differences. That is also why many MoE implementations keep a residual connection and normalization inside the expert FFN. Slightly heavier is to monitor scaling on expert outputs: record the mean, standard deviation and maximum of each expert's output tensor into the training log and watch directly which experts are drifting and by how much. Most teams do not do this, so these problems usually only start being chased after a loss spike appears, wasting a great deal of time. Heavier still is expert-level numerical protection, such as clipping or detecting extreme magnitudes — measures typically used as overflow protection in low-precision training rather than as a standard configuration.

There is one easily missed connection: scale differences between experts directly amplify the numerical sensitivity of routing. As mentioned earlier, the top-k choice depends on the relative magnitude of logits, and when the number of tokens received by different experts varies widely, the token count itself affects the statistics of the next layer's gate input — forming a positive feedback loop of "load imbalance causes differing scales, differing scales shift routing decisions, shifted routing aggravates load imbalance." Monitoring expert scale drift is therefore also monitoring the health of balancing.

One practical summary: MoE's observability requirements are far higher than a dense model's. For dense models the loss curve and gradient norm are basically enough; for MoE you need at least three groups of metrics — drop ratio, per-expert load distribution, and per-expert output scale — plus bias term magnitude and gating entropy before you even have basic debugging capability. Implementing those metrics is cheap, but without them, when an MoE goes wrong you are basically guessing.

The Memory and Compute Bill: Laying Out MoE's Real Cost

MoE memory is the item most often miscounted. The correct way to do it is to keep two separate ledgers, one for training and one for inference, and the two frequently reach opposite conclusions.

Training: weights stored in bf16 (2 bytes per parameter), optimizer state estimated with AdamW (fp32 master weight plus first- and second-moment buffers, about 12 bytes per parameter, roughly 14 bytes together), plus the gradients themselves at 2 bytes — on the order of 16 bytes per parameter, before you add communication buffers, activations, and the dispatch/combine intermediates unique to MoE. At the 671B total-parameter scale, optimizer state alone is in the TB range. This means MoE training must shard the optimizer state with expert parallelism too, and often needs CPU offload or ZeRO-style paged optimization; activation memory is additionally inflated by the capacity factor (every expert reserves slots for its capacity, and empty slots exist even when there are actual drops). The conclusion of this ledger: MoE training saves compute, not memory — under some configurations the memory overhead is actually higher than for a dense model with the same activated size.

Inference: only bf16 weights plus the KV cache, 2 bytes per parameter, so 671B amounts to roughly 1.3TB of weights. Relative to the cost of the KV cache, what is unique to MoE is this: all expert weights must stay resident and cannot be swapped in and out on demand the way activation layers can (unless you accept enormous swap-in latency); and all-to-all communication has to happen on every forward pass. That directly forces MoE inference into batching. Without large batches, both MoE advantages (large capacity, weights that fit in memory) get eaten alive by communication latency, while a dense model is actually faster and simpler at small batch sizes.

The compute ledger is subtler. Activated parameter count A is not the actual FLOPs per token: router computation, top-k selection, capacity padding, the idle work caused by token dropping, and the cost of moving data through all-to-all are all extra costs. Under a configuration with k=2, many experts and a capacity factor of 1.25, these extras can push real FLOPs significantly above the value estimated from A — roughly speaking, routing and scheduling overhead stand out most in small models where activated parameters are a small share of the whole. The correct engineering practice is to measure rather than estimate: fix the hardware and parallel configuration, measure tokens/s and end-to-end latency, then back out an "effective throughput parameter count" from the activated parameter count, and that number is the metric that is actually comparable across MoE models.

There is also an easily overlooked cost dimension: quantization does not hurt MoE uniformly. See below.

Routing Stability on the Inference Side: The Real Cost of Train/Inference Mismatch

This section covers the part of MoE engineering that is easiest to get wrong and least systematically discussed. The core problem in one sentence: routing decisions are data-dependent, and the numerical conditions at inference time differ from those at training time, so the same input gets routed to different experts and the model's behavior drifts.

How Batch Size Changes the Routing Distribution

top-k routing is computed independently per token and in theory is unaffected by the batch. But in real implementations it is affected by the batch: the router's matrix multiply takes a different GEMM kernel (at small batch sizes it may hit a different tiling strategy or a lower-precision path), so the magnitude of numerical error changes; the drop logic almost never triggers when the batch is large enough but climbs noticeably at small batch sizes, and since a dropped token's FFN output is zero, that is equivalent to temporarily changing the model architecture. This "batch size changes the model" effect is nearly absent in dense models and quite pronounced in MoE.

This produces a practical check worth having: given the same passage of text, does greedy decoding produce the same result at batch=1 and batch=64? If it does not, your implementation has numerical sensitivity or a structural change caused by dropping. This check is worth writing into your regression suite, because it surfaces in advance a whole class of strange issues that otherwise only appear in production (the same question getting different answers at different concurrency levels).

Quantization Hurts Routing More Than It Hurts the FFN

Weight quantization affects MoE unevenly, which is exactly where people get hurt by carrying dense-model quantization experience straight over to MoE. The router is a single very thin linear layer whose output logits directly determine the top-k choice, and top-k is a step operation: a logit gap of 0.01 between two experts and a gap of 0.11 can flip the choice in the face of quantization error. When the logit gap is small — precisely the moment when routing is most critical and most decisive — the relative error introduced by quantization occupies its largest share.

So the practical consensus is: keep router weights and biases at high precision and do not quantize them. Their parameter count is negligible next to all the experts (E × D versus E × 3D²), so keeping fp16 or bf16 costs almost nothing while eliminating an entire class of hard-to-trace output instability. Related experience: run a "routing temperature calibration" once near the end of training — inspect the distribution of logits, and if a large number of tokens have a tiny gap between top-1 and top-2, routing is sitting in an ambiguous region and is unusually sensitive to quantization error. Those cases need either a higher routing temperature or, at inference time, mixing the selected k experts by probability rather than taking a hard top-k.

Other Practical Constraints on the Inference Side

There are several more points to account for. Activation quantization is usually safer than weight quantization, because it does not directly perturb routing decisions — but note that the empty slots created by capacity padding are all zeros, and if the quantization scale is computed from global statistics those zeros skew it, so compute it from effective token statistics instead. Expert merging or pruning is viable at inference time — a practical route for MoE inference optimization: measure input/output similarity or importance across experts, merge redundant experts or drop them outright, and significantly reduce resident memory and the number of all-to-all participants. The precondition is that you can accept a small accuracy loss and that the pruning is done offline and baked into the weights.

The multi-GPU partitioning strategy has to follow EP: in an inference service the EP degree should match the physical boundaries of the hardware interconnect, and the split granularity must divide the expert count evenly. If the GPU count within a domain does not divide the expert count, some domains end up holding only part of their experts and certain tokens have to make a round trip across domains, which directly doubles latency. The maturity of MoE support varies widely across inference frameworks: vLLM and SGLang both ship fused MoE kernels (folding routing, gather/scatter and grouped GEMM into one or two kernels), and how well it is supported depends on the expert count, top-k, whether shared experts are used, and whether orthogonal structures such as MLA are present. Before deploying, measure throughput with the target framework rather than judging from a coarse conclusion like "the model supports MoE."

A Checklist for Implementing an MoE Layer Yourself

Having read all of the above, if you decide to implement it yourself, here is a self-check list in execution order. Every item corresponds to one of the pitfalls mentioned in earlier sections, and each can be verified independently with a small-scale experiment — none of them requires a full training run.

• Is the gate renormalized after top-k: remove the renormalization, run a few hundred steps, and see whether the gradient magnitudes across experts jitter noticeably. This is the easiest thing to get wrong and also the easiest to verify.
• Does the router carry a bias: if it does, check whether that bias gives certain experts systematically higher assignment probability early in training. The principle is: no bias.
• Is capacity computed from the expected load: confirm that the formula is total assignments divided by the expert count, times the capacity factor, rather than just picking a constant capacity out of the air.
• Is the drop logic reproducible: run the same input twice and the set of dropped tokens should be identical; random dropping introduces irreproducible noise.
• Is the drop ratio logged: it has to be in the training log, and recorded per expert, otherwise you cannot tell "overall imbalance" apart from "one overloaded expert."
• Is the input flattening thorough: capacity and dispatch are both based on the flattened token dimension, and forward and backward must use the same flattening and restoration logic, or gradients end up misaligned.
• How the output of a dropped token is handled: explicitly decide between zeroing it and skipping the residual, and confirm that the resulting downstream gradient treatment matches your expectation.
• Does the pre-communication synchronization exist: the exchange of load counts must happen before the all-to-all; missing this step causes occasional hangs at small scale.
• Is the grouping sort stable: tokens of the same expert must be contiguous and stably ordered, otherwise grouped GEMM results are not reproducible.
• Cross-batch consistency: fix a passage of text, run inference once at batch size 1 and once at batch size 64, and the greedy decoding result should match.
• Router kept at high precision: on the inference side, confirm that the gate layer's weights were not run through the quantization pipeline.
• The expert parallel degree divides the expert count: when it does not, cross-domain round trips directly double latency, and this class of misconfiguration raises no error.

In this checklist the first six items are algorithmic correctness and the last six are systems correctness, and the debugging approaches are completely different: the former can be verified with a few hundred steps of small-scale training plus logs, while the latter is only meaningful under a real parallel configuration. The two most commonly skipped are actually the last two, because they raise no error and only surface in production at specific concurrency levels. Writing cross-batch consistency into CI is usually the single highest-leverage action available.

MoE with Long Context and Multimodal Inputs: New Problems Brought by Token Types

The routing problems discussed above assume all tokens are homogeneous. In long-context and multimodal settings that assumption breaks, which opens up a few new engineering problems worth handling on their own.

First, load imbalance across token types. After projection, multimodal inputs produce visual tokens, text tokens and possibly audio tokens whose distributions in hidden-state space differ greatly, and the router's preferences for them can be entirely different — visual tokens may be routed in a highly concentrated way onto a few experts, because visual feature distributions are inherently more concentrated than textual ones. The result is that in a mixed batch the text-side expert load can sit far below capacity while the visual-side experts repeatedly hit the capacity ceiling and drop tokens. The fix is to track drop ratio and load variance separately per token type instead of only looking at a global average; if capacity does need to be set separately, that usually means maintaining two sets of capacity logic, which raises complexity and needs a cost-benefit call.

Second, the interaction between context length and the capacity factor. Capacity is computed from average load, and load variance depends on both the expert count and the token count per sequence. When sequence length grows from a few thousand to several hundred thousand, the routing distribution becomes more stable over a large batch (variance drops), but the moment a distribution-shifted passage appears, some expert can instantly swallow a large number of tokens, and those tokens are dropped once they exceed capacity — what gets dropped in long context is often not a random marginal sample but a contiguous span. That means dropping affects model behavior far more severely than in short context: you lose an entire piece of semantics rather than scattered tokens. Mitigations include scaling capacity up separately for long sequences, assigning different capacity quotas to tokens from different sources, or simply relaxing drop on long sequences and accepting the compute loss.

Third, the trade-off between sequence-level balancing and sequence-level bias. Earlier I mentioned that bias updates can be computed per sequence, which avoids long sequences dominating the global statistics. But when sequence lengths within a batch vary widely (long-context mixed with short-text training), per-sequence statistics instead amplify short-sequence noise into bias jitter. A compromise is to weight the statistics by token count, or to lengthen the update window to smooth the noise. What DeepSeek-V3 describes publicly is a combination of sequence-level auxiliary balancing coefficients and sequence-level bias, used to guarantee that routing favors neither any particular sequence nor any particular token; that is a direction worth borrowing — splitting balancing out of a single level into several levels handled separately.

Fourth, orthogonality with position and structure. When the model's attention layers themselves introduce sparse or low-rank structures (low-rank KV compression, multi-head latent attention, and other compressed-attention approaches), the two sparsity patterns stack: attention compresses along the sequence dimension while MoE sparsifies along the feature dimension. The engineering consequence is that peak memory and communication patterns both have to be re-measured, because the best parallel configuration for these two structures is often different and you cannot simply carry over conclusions from a pure-attention model.

What these problems share is that they all come from the same root: "routing is data-dependent, and the distribution of real data is far wider than the assumptions made at training time". So when doing MoE with multimodal and long-context data, the granularity of your monitoring metrics has to be finer than in the pure-text case, and capacity and balancing design must move from "one global setting" toward "per type, per length bucket."

A Parallelism and Precision Configuration Checklist for One MoE Training Run

Having discussed all those algorithmic problems, we still have to land on a configuration table. The checklist below is organized in the order "parallelism first, precision second, communication optimization last," and every entry gives both the reason for choosing it and the boundary beyond which it does not apply.

How to Slice the Parallelism

Use different parallelism schemes for the attention and expert layers — this is the basic shape of an MoE training configuration. Attention layers use tensor parallelism plus pipeline parallelism, expert layers use expert parallelism. The reason is that their communication patterns are completely different: attention communication is a fixed-shape reduction and suits tensor parallelism, while expert communication is a data-dependent irregular dispatch that only suits expert parallelism. Mixing them (for example, tensor-parallelizing the experts as well) yields extremely poor communication efficiency.

There is a common mistake in allocating degrees of parallelism: turning the expert parallel degree up too high. Expert parallelism's all-to-all is a fully connected exchange, and once it leaves the high-speed interconnect domain bandwidth drops sharply; the more devices participate, the more fragmented the buckets and the smaller the average message, so efficiency degrades further. Practical experience says to align the expert parallel degree with the size of a single high-speed interconnect domain and scale across domains with data parallelism or tensor parallelism on the activation layers. Designing against that domain size as a hard constraint is far more effective than tuning after the fact.

The expert count must be divisible by the parallel degree. When it is not, there are two options: let some devices hold one extra expert (load is naturally uneven, usually not worth it), or introduce an extra non-reducing scheduling layer inside the expert layer. The former is simple but lets the busiest card determine throughput; the latter is flexible but complex to implement. In most cases, designing the expert count as a multiple of the parallel degree is the least-effort path, which also explains why so many models use expert counts of 8, 16, 64, 128 or 256.

How to Set Precision

bf16 for master weights and forward/backward, fp32 for optimizer state — this is the mainstream choice for large-scale training today. bf16's dynamic range comfortably holds MoE's very wide activation values (especially when expert output scales differ a lot), while fp32 master weights guarantee numerical stability over long training runs. MoE is more precision-sensitive than a dense model because the scale of the routing logits drifts during training, and those logits directly determine a step-like expert selection.

More aggressive low-precision training needs extra routing protection. Pushing throughput further with something like fp8 in the forward pass and matmuls is a viable direction, but it must be paired with protection of the routing path: keep the gate GEMM at higher precision, or exclude the routing layer from the fp8 scaling-factor computation. What DeepSeek-V3 describes publicly is overall fp8 training plus fine-grained precision allocation and several overflow protection measures. The applicability boundary here is clear: the extra complexity only pays off in large-scale, expert-heavy, compute-constrained pretraining; at medium scale it is usually over-engineering.

What Order to Optimize Communication In

If communication is the bottleneck, the priority order is explicit. First priority is reducing the number of communication calls: have the backward pass return activation gradients, gating weights and routing indices in one go instead of several separate passes; this gives the largest gain and does not change numerical results. Second priority is overlapping communication with compute: while dispatching one expert, compute another, and use double-buffering of communication and compute so their latencies overlap instead of adding up. Third priority, and only then, is compressing the communication volume itself: send activations at lower precision, route traffic from different sources down separate paths, improve small-message packing.

The judgment behind that order: reducing the count is pure algorithmic gain, overlapping is a systems gain, compression is an engineering gain. The first two have higher ceilings and introduce no additional precision risk, so they should come first. In practice many people start by optimizing the packing format and end up with less than they expected.

Evaluation Traps in MoE: Why Offline Metrics Lie So Easily

Evaluating MoE carries two extra layers of uncertainty compared to dense models, coming from structure and from the runtime environment. If you do not control these two layers, your evaluation conclusion may be pure noise.

The core problem in the structural layer is this: the model's output depends on batch size and deployment configuration. As discussed earlier, capacity clipping and low-precision paths can send the same input to different experts at different concurrency levels and produce different outputs. That means that if you evaluate with a single fixed batch configuration, what you are really measuring is "the model on one particular slice," not the model itself. The correct practice is to fix the batch and scheduling parameters during evaluation and record them, and to run at least one cross-batch consistency check; if consistency does not hold, every comparison in that round has to be discounted.

The quantization layer deserves separate caution. MoE's quantization error falls unevenly across modules: the expert FFN weights make up the overwhelming majority of the count and contribute most of the error, while the router's parameter count is negligible but its effect on decisions is a step. So "same total bit budget" quantization schemes may actually perform worse on MoE, because they usually allocate precision by parameter count and end up giving more precision to the router, which should have been kept at high precision anyway. The same total byte count does not mean the same effective information — this is the easiest trap to hit when migrating quantization experience straight over from dense models.

There is another structural evaluation trap: expert imbalance skews how an evaluation set covers the distribution. If many evaluation samples get routed to the same expert, their errors are amplified by that expert's capability gaps, so the result over-represents that expert's preferences instead of the model's average ability. Viewing the evaluation set stratified by the entropy of the routing outcome is a practical way to expose this bias — the metric gap between high-entropy and low-entropy samples reflects the model's true robustness far better than the overall mean does.

The runtime environment layer is just standard engineering discipline: pin the inference framework version, parallel configuration and kernel implementation before comparing; when comparing different MoE models, confirm they use identical data processing and evaluation scripts; when reporting results, write out the configurations that change model behavior — capacity factor, drop policy, quantization scheme. That last one matters most — an MoE evaluation report that does not state its capacity factor and quantization scheme has numbers that are not comparable.

Routing and Data Mix: What the Experts Learn Depends on What You Feed Them

Every earlier discussion of balancing and capacity treats data as an exogenous variable, but the data mix is actually one of the most underrated design variables in MoE. The reason is direct: expert division of labor is learned from the data, and imbalance in the data distribution translates directly into load imbalance.

The most typical case is a data mix where one language or domain dominates. If a category of data accounts for most of the training corpus, the router quickly learns to send those tokens heavily toward certain experts — this is not a router bug, it is optimizing correctly: if an expert is repeatedly trained to fit that data it performs better on those tokens, so more of those tokens get routed to it, which reinforces it further. The result is that those experts become overloaded, small-share domains stay chronically undertrained, the model is clearly weaker on niche domains, and all of this shows up in the load histogram as a few very tall bars.

This yields a counterintuitive but important judgment: MoE's load-balancing objective is in some sense in conflict with the objective of data diversity. The balancing loss wants identical load on all experts and does not distinguish which domain a token came from; what you actually want is "every domain has an expert that can handle it, and that expert is trained enough." These two objectives only agree when the data mix itself is balanced. So the balancing hyperparameters and the data mix in MoE training have to be tuned together; tuning either one alone easily lands on a suboptimal result.

In return, this is a new footnote on the value of shared experts. A shared expert receives every token, so it is naturally trained at the true proportions of the data and provides general capability that has not been distorted by artificial balancing. The routed experts, meanwhile, are pressed flatter by the balancing constraint, so their degree of specialization is itself limited by balancing pressure. Together, the two give the model a reasonable compromise between "fidelity of general capability" and "specialization in a few domains."

One practical piece of advice: when you look at a load histogram, always bucket it by data source. Globally it may already look very even (because the large domains have been pressed down), but split by domain you will find that tokens of certain small domains are highly concentrated on a few experts whose total load is not high — meaning they have idle capacity that is never used, and the routing decisions simply did not select them. In that situation, raising the capacity factor or increasing the balancing strength does not solve anything; what you should examine is the router's discriminative ability and the sample count for that domain in the training data.

Incidentally, this echoes on the inference side as well: if your production traffic distribution differs substantially from your training data mix (say Chinese-dominant in training but a multilingual mix in production), then in real serving the MoE's expert distribution gets pushed into a state it was never sufficiently trained for, and the load distribution visibly shifts. This problem does not exist for dense models, and on MoE it has to be handled deliberately. The common approach is to deliberately retain a portion of out-of-distribution but structured data in the mix, so the router has a fallback region for those types that "will not collapse."

An Engineering Trade-off Checklist: When Not to Adopt MoE

MoE is not a better-by-default architecture; its benefit has explicit preconditions. Write those preconditions down and you have a set of questions for deciding whether your own project should adopt MoE.

Is your scale large enough? The core benefit of MoE is "decoupling capacity from compute," and that benefit needs a large enough activated compute budget to show up. If your target activated parameter count is only on the order of 1B to 3B, the capacity gained by adding experts will most likely be eaten by routing and communication overhead. The minimum effective scale for coarse-grained MoE is roughly above 7B activated parameters (which is also why a configuration like Mixtral 8x7B gets copied over and over), and the fine-grained route demands even more scale.

Do you have cross-device interconnect? If your deployment target is a single card, or a few nodes with weak interconnect, MoE's all-to-all will kill it outright. The precondition for MoE is either multi-card high-speed interconnect within a node (an NVLink domain), or a willingness to trade compute utilization for capacity.

Is your serving shape throughput-oriented or latency-oriented? MoE suits large-batch, high-throughput, long-context workloads such as offline processing, data synthesis, and batch APIs. It does not suit low-latency, small-batch, single-request online serving — that is precisely the dense model's comfort zone.

Can your operations team carry it? Running MoE means monitoring drop ratio, expert load variance, expert output similarity, the share of time spent on all-to-all, and the magnitude of routing bias — all at once — and having the ability to localize, when they go abnormal, whether the problem is capacity, balancing or communication. By comparison, the tuning path for a dense model is far shorter. With a limited team, making a dense model solid (better data, longer context, a more stable inference stack) usually pays back more.

Is your task distribution diverse enough? If the data is highly homogeneous (say nothing but classification tasks in one fixed format), the routed experts have no room to differentiate, MoE's benefit cannot materialize, and you pay overhead for nothing. MoE's benefit correlates strongly with how diverse the patterns in the data are.

Conversely, the signals that justify adopting MoE are: a dense model has clearly hit a capability ceiling at the same compute, your memory can hold parameters far beyond the activated size, your workload is large-batch processing, and your training cluster has high-speed interconnect while you can absorb one round of fairly complex distributed debugging.

Tuning Order: These Three Knobs Cannot Be Tuned Separately

MoE tuning is painful because the auxiliary loss coefficient, the capacity factor, the expert count and top-k are strongly coupled — changing one makes the optimal value of the other three move. The easiest mistake to make in practice is to turn every knob at once and then judge "which parameter is wrong" from the final loss — a judgment with almost no information in it, because the final loss is the joint result of four variables.

A relatively safe order is: fix the structure first, tune balancing second, compress capacity last. Concretely, the expert count and top-k are structural decisions you do not revisit once training starts, because changing them amounts to resetting training. The balancing mechanism (auxiliary loss or bias method) has to be picked before training starts, because it determines the overall shape of the training curve and swapping it afterwards means rerunning. Sweep the auxiliary loss coefficient alpha over a rough order of magnitude, say a few points between 0.01 and 0.1, run a few thousand steps at each point, and watch how quickly expert load variance falls versus how quickly the main task loss rises; find a point that suppresses the variance without noticeably slowing the main task, and freeze it. Leave the capacity factor for last: once the distribution is stable, push it down gradually while monitoring the drop ratio.

There is a very useful indirect metric for judging "is the balancing excessive": pairwise similarity of expert outputs. If the cosine similarity between the outputs of any two experts over some set of inputs is broadly high, then the experts are duplicating each other's work and pushing the balancing harder will not pay off — the problem lies in the router's discriminative ability or in the expert structure, not in insufficient balance. This metric surfaces genuine degradation earlier than the load histogram does.

Conversely, if load is already fairly even but the routing distribution is visibly skewed (a few experts taking most of the tokens), then the bias term is not correcting hard enough, or its statistical window is too coarse; raising the update frequency usually fixes it.

Common Failures and How to Localize Them

The row most worth singling out is the fourth. Throughput slowly declining as training steps increase is a very sneaky problem: it raises no error and does not crash, it just gets slowly slower, so it is easily misread as "the cluster got busy." The real cause is usually that the routing distribution sharpens over the course of training: early on the assignment across experts is fairly random and the all-to-all message volume is roughly even across cards; as training proceeds routing becomes increasingly deterministic, so the card holding the popular experts sends and receives far more data than the others while they wait. The backward all-to-all is even more pronounced, because the volume of data coming back as gradients is larger than on the forward pass. The result is that the busiest card in the cluster determines overall throughput while the rest of the compute idles. Mitigations include putting an upper bound on the routing logit temperature, periodically rebalancing load, or physically spreading the hottest experts across different nodes.

Expert Redundancy and Post-Training: A Pragmatic Route to Turning MoE Dense

A recurring phenomenon is expert redundancy: there are many experts but only a small fraction actually does work, while the rest duplicate it or sit idle for long stretches. The cost is very direct — large amounts of useless weight sit resident in memory, all-to-all still pays bucketing and communication overhead on their behalf, and the router's output dimension grows accordingly, making it harder to learn.

There are several complementary ways to identify redundancy. The most direct is input distribution statistics: cluster the hidden states each expert receives and see how many experts have heavily overlapping input clusters. Next is output equivalence testing: take a set of probe inputs, compare expert A's output against expert B's output pair by pair, and if the difference is extremely small those two experts can be merged. Then there is post-pruning validation: simply cut away a batch of low-value experts, run a short evaluation, and see how much the metrics drop. That is the most honest but also the most expensive method, because every trial requires rerunning the validation.

Based on those judgments, there is a route worth watching, known as denseification of sparse models. The basic operation is this: take an already trained dense model, split each FFN into several small FFNs (the original form of MoE), then use the original model to generate a large number of samples as distillation targets and have those small experts fit the original FFN's outputs. This requires no change to the backbone and no new pretraining data, and it costs far less than training an MoE from scratch, which makes it very practical for "quickly turning an existing dense model into a sparse form." DeepSeekMoE's small-experts-plus-shared-expert structure and the later sparse upsampling line of work belong to a similar family of ideas: take an already trained dense model as initialization and incrementally train the dense FFN's capability apart into several small experts.

What this route means concretely for an engineering team is this: it downgrades "should we adopt MoE" from an architectural gamble to a reversible engineering option. You can first run the denseification on your existing dense model to capture a first round of benefit, and if the result is unsatisfying the cost is only incremental training time rather than a full pretraining. Conversely, if the benefit holds, your dense model already exists.

Its boundaries should be stated clearly too. Densification gives you "a dense model whose weights have been split apart," and its router is fitted after the fact, tending to reproduce the original FFN's division of labor, rather than a learned specialized routing. Structures like this usually have clean expert specialization and easy load balancing, but each expert has limited expressional diversity, and their ceiling sits below that of an MoE trained from scratch that is allowed to let experts genuinely specialize. They fit better as a transitional form and a baseline than as the end state.

A related but opposite route is inference-time expert merging and structured pruning: leave training alone and, at deployment time only, merge redundant experts into equivalent larger FFNs, thereby reducing resident memory, the number of devices taking part in all-to-all, or even degenerating into an inference form closer to dense. For latency-sensitive deployments that are tight on memory, this route is far cheaper than retraining.

Taken together, these two things trace a realistic throughline in MoE engineering: MoE's deployment form should not be fixed — it is a continuous spectrum that runs from dense to sparse and can retreat back to dense on demand. Use the sparse form during training to buy a higher capability ceiling; at deployment, choose the sparsity level based on real load and the latency budget; any step in between can be adjusted. Treating MoE as a system whose form changes and which needs long-term maintenance, rather than as a one-off architecture selection, is usually the highest-leverage approach over the long run.

Summary: What to Remember When MoE Reaches Production

Compressing all of this into a handful of points you can act on. The intrinsic benefit of sparse activation is the decoupling of capacity from compute, which turns total parameter count and per-token compute into two independently adjustable knobs — but it also moves the center of gravity of system problems from FLOPs to communication and scheduling. The router's implementation details affect success or failure more than the math does: renormalize after top-k, add noise during training to suppress premature softmax saturation, keep the router bias-free, and set capacity from expected load rather than actual load. The capacity factor is a premium payment: set it too large and you are using compute to mask routing problems until it suddenly collapses past the limit; set it too small and the drop ratio climbs and directly drags throughput down.

The load balancing loss is MoE's most famous gradient conflict point: the direction it imposes on the gating parameters runs opposite to the main task's demand for specialization, its coefficient has to be hand-tuned, and reproducibility is poor. Gradient-free bias balancing, z-loss, entropy regularization and node-limited routing are four more "surgical" alternatives, and the bias method — because it decouples "who gets selected" from "the weight of the selection" — is currently the most practical one for autoregressive models. Shared experts are not optional: they pull general capability out from under the specialization pressure on the routed experts, provide a gradient sink that never collapses, and cost one dense computation that cannot be sparsified away. Mixed granularity only reveals a benefit when total parameter counts line up exactly; granularity that is too fine backfires because of insufficient data, fragmented communication and a harder router.

On the inference side, the thing to watch most closely is the numerical sensitivity of routing: keep the router at high precision, remember that batch size changes the routing distribution, and remember that quantization error is amplified when the gap between top-1 and top-2 is tiny. Writing "the same passage produces consistent output across batch sizes" into your regression suite is the cheapest and highest-return defensive measure available. Finally, deciding whether a project should adopt MoE comes down to five things: model scale, interconnect conditions, serving shape, operational capability, and data diversity. If any one of them is not satisfied, a dense model may well be the better engineering choice — and that is not conservatism, that is charging MoE's complexity cost to the decision.

Reverse-engineering design intent from public configs is also a reading skill worth mastering. Seeing a combination of "many routed experts with few active ones" tells you it is on the fine-grained route, where the capacity factor margin must be very tight and load variance is naturally high; seeing "few routed experts, each very large" indicates the coarse-grained route, which demands a larger capacity margin; seeing shared experts usually means balancing pressure is high and the model is large. Read those configs once and you can form a fairly close judgment about that MoE's balancing strategy, parallel partitioning and memory bill without ever seeing a training log.

As of 2026 the consensus in this direction is already quite crisp, and it can be summarized in four sentences: routed experts keep becoming finer-grained while shared experts turn into standard configuration; balancing mechanisms are moving from gradient-carrying auxiliary losses toward statistical corrections that do not pollute the main task gradient; communication optimization for expert parallelism is moving from reducing the number of communication calls toward finer bandwidth matching; and inference-side quantization strategy has to treat the router as a precision no-go zone. Behind those four sentences sits one shared judgment — MoE's principal contradiction has already shifted from "can it be trained at all" to "can it be used efficiently, stably and predictably." For teams working in this direction, that sentence essentially decides where resources should go.

Finally, back to the original judgment: MoE is no longer "a cheaper model," it is "a harder model." It trades the complexity of communication and scheduling for the decoupling of capacity from compute, and that trade is an excellent deal when scale is large enough, load is heavy enough and interconnect is fast enough; in every other case it turns what could have been a simple design into a fragile one. Knowing clearly what you are trading is the precondition for doing MoE at all.