Distributed Training of Large Models: Data, Tensor, Pipeline, and Expert Parallelism — AI Development Tutorial

Why a Single Card Cannot Keep Up: Three Walls of Memory, Compute, and Bandwidth

Before 2019, fitting a model onto a single card was usually just an engineering problem of tuning the batch size. The era of large models rewrote that proposition from the ground up. Parameter counts jumped from the hundreds of millions to tens or hundreds of billions, and a single accelerator card with a few tens of GB of memory — after subtracting framework overhead, operator workspaces, and communication buffers — can often hold less than a quarter of the model. Where does the rest go? It is eaten up by optimizer states, gradients, activations, and fragmented temporary tensors.

Broken down concretely, training a model means holding four things at the same time: the activations produced by the forward pass (needed to compute gradients in the backward pass), the gradient for every parameter, optimizer states of the same order of magnitude (moments, variances, and so on), and the parameters themselves. The first two grow quadratically with sequence length under long-sequence training, while the latter two grow linearly with parameter count. That is why you see such a counter-intuitive effect: halving the sequence length and shrinking the batch size often relieves memory pressure more than simply having fewer parameters does.

The second wall is compute. The floating-point work involved in large-model training scales roughly as the product of parameter count, sequence length, and data volume. Even the newest flagship card, when faced with the full pre-training of a hundred-billion-parameter model, needs to be measured in months — a timescale that is simply unacceptable for iterative research work. Multi-GPU is not there to make it run at all; it is there to make it run fast enough to close the iteration loop.

The third wall is the most hidden and the most underestimated: the bandwidth wall. Distributed training spreads computation across many cards, but almost every step may require exchanging tensors. Once the communication volume exceeds what the interconnect bandwidth can sustain, the compute units idle en masse and hardware utilization drops off a cliff. At that point more cards do not merely fail to help — they become slower than fewer cards. The four parallelism strategies discussed throughout this article all answer the same question: given a fixed cluster shape and fixed bandwidth, how do you push communication volume below compute cost.

Parallelism strategy is not "the more the better" — it is a balancing problem between bandwidth and memory. Choosing wrong does not cost you "a little speed"; it leaves the newly added cards completely idle.

The Four Strategies Cut Along Completely Different Dimensions

The thing beginners most often get confused about is treating these four approaches as four interchangeable options sitting at the same level. In reality they partition entirely different dimensions of the model, and their communication costs differ by several orders of magnitude. The table below establishes the overall intuition; each later section expands on it.

Strategy	What gets split	Communication primitive	Communication frequency	Communication volume	Hard constraint
Data parallelism (DP)	The batch dimension	Gradient all-reduce	Once per step	Proportional to the model's parameter count	Every participating card must be able to hold the full model
Tensor parallelism (TP)	Rows/columns of weight matrices	all-reduce or all-gather	At least twice per Transformer layer	Proportional to sequence length and hidden dimension	Must be bound inside a high-bandwidth domain
Pipeline parallelism (PP)	Layers	Point-to-point send/recv	Once per micro-batch	Proportional to micro-batch activation size	Has a pipeline bubble
Expert parallelism (EP)	MoE experts	all-to-all to dispatch tokens	Every MoE layer	Proportional to token count and hidden dimension	Must be paired with a capacity factor and load balancing

Read that table from top to bottom and a clear progression emerges: data parallelism has the heaviest communication volume but the lowest frequency, while tensor parallelism has the highest frequency but the lightest cost per call; pipeline parallelism barely moves gradients at all and ships only activations, and expert parallelism combines both high frequency and large volume, which makes it the hardest category to tune. These characteristics directly determine every engineering trade-off that follows.

There is also a more important organizational principle: the more expensive a parallelism dimension is in communication terms, the more aggressively you should "turn it down" — keeping its degree within what the high-bandwidth interconnect can sustain; dimensions whose communication cost grows gently with parallelism degree (pipeline is the typical case) are the ones suited to spreading the model across a large number of relatively low-bandwidth nodes. The 3D parallelism section later in this article quantifies this principle.

Data Parallelism: Small Communication Volume, Expensive Synchronization

The idea behind data parallelism is simple almost to the point of being obvious: every card holds a complete copy of the model, the batch of data is sliced into several pieces and distributed across the cards, each card runs its own forward and backward pass to compute its own gradients, and once the backward pass is finished all gradients are synchronized in one shot so that every card ends up with identical model parameters. Model capacity is then no longer bound by the memory of a single card, and the computation is linearly amortized across all cards.

But "simple" holds only at the conceptual level. Data parallelism has two real difficulties: first, the volume of a single gradient synchronization is far too large; second, that synchronization is a hard barrier that cannot be hidden behind computation.

Gradient Accumulation: Separating Effective Batch Size from Communication Count

Start with gradient accumulation, because its overhead is directly coupled to that of data parallelism. The logic is this: split one large batch into several micro-batches, run forward and backward several times, accumulate gradients into a buffer rather than synchronizing every step, and issue a single all-reduce once the configured number of accumulations has been reached. Mathematically it is equivalent to a larger effective batch size.

The key judgment is this: gradient accumulation reduces the number of all-reduce calls, but it does not reduce the volume of any single all-reduce. If the gradient buffer holds full fp32 gradients, then after accumulating 32 times and synchronizing once, the volume is still the size of the complete gradient set — you simply do not send it 32 separate times. A great many people get this accounting wrong, assuming that accumulation crushes communication cost down to 1/32, when in fact it has only compressed 32 transfers into one large one.

So what does gradient accumulation actually save? It saves communication launch overhead, peak buffer memory, and, when bandwidth is the constraint, the amount of time the network stays occupied. More importantly, it gives you a control knob: without changing the global effective batch size, you can tune how often synchronization fires, trading off between "fewer synchronization events" and "bandwidth consumed in concentrated bursts."

Strategy	Effective batch size	Number of all-reduce calls	Volume per call	When to prefer it
No accumulation, synchronize every micro-batch	Small	Many	Small	Bandwidth is extremely tight, model is small
High accumulation factor	Large	Few	Large	Bandwidth is adequate, want to save launch overhead
Frozen parameters / gradient sparsity	Unchanged	Few	Small	Fine-tuning scenarios, a natural fit

The last row deserves a word of its own: once you enter a fine-tuning scenario and are training only a small subset of parameters, the gradients that must be synchronized become naturally smaller, and the cost-effectiveness of data parallelism changes qualitatively. This is also one of the reasons low-rank fine-tuning methods remain useful at large multi-GPU scale — they cut precisely into the cost that data parallelism finds most painful.

Where the Cost of all-reduce Actually Comes From

The classic implementation of gradient synchronization is the ring all-reduce: each rank holds a shard of the gradient and, in ring order, sends its shard to the next node while simultaneously receiving the shard from the previous node. After one round every rank holds the complete gradient, at a cost of 2(N-1)/N times the single-card data volume. The alternative is a two-phase reduce-scatter plus all-gather, which moves the same total volume but has different latency characteristics: the ring is bandwidth-efficient on large messages, while the two-phase structure of reduce-scatter + all-gather more easily saturates the link on very large messages.

The real performance loss comes from three places, and each of them deserves its own separate look in the profile:

Bandwidth ceiling. Real communication throughput never reaches the nominal link bandwidth — message segmentation, protocol stack overhead, and tail effects all take a discount. When inter-node traffic runs over Ethernet or InfiniBand, the gap between effective bandwidth and peak is often pronounced, whereas inside an NVLink domain the gap is far smaller.
Small-message latency. When the model is not large, or when gradients are sharded into very small pieces, communication time is dominated by latency rather than bandwidth, and the more ring rounds there are, the more latency accumulates. Only then does swapping a "bandwidth algorithm" for a "latency algorithm" make sense.
Hard barrier semantics. all-reduce cannot complete until every rank has arrived, so one slow card makes the whole cluster wait. This is what is known as synchronization tail latency, and its amplification is extremely significant at thousand-GPU scale: a 100-millisecond jitter on a single card, multiplied through a deep chain of communications, gets amplified layer by layer.

Hiding Communication Inside Compute: Bucketing and Overlap

Data parallelism does not have to "finish computing, then synchronize." The approach used in practice is gradient bucketing: during the backward pass, parameters produce their gradients progressively, in computation-graph order. As soon as the gradients inside a bucket reach a fixed size (say a few tens of megabytes), an asynchronous all-reduce is launched immediately so that it overlaps with the backward computation of the layers that follow. Because the backward pass proceeds from the back toward the front, the gradients produced first belong to the back of the network, and the communication lines up precisely with the backward pass of the remaining layers.

This is why contiguous gradient buckets is not an optional tuning knob: it determines how much communication can genuinely be hidden. Buckets that are too coarse shrink the overlap window and leave communication time exposed; buckets that are too fine bring back the latency overhead of small messages. During actual tuning, the metric worth watching is the fraction of communication that gets masked, not the total communication time.

# 典型数据并行启动方式：每节点 8 卡，梯度分桶 25MB，开启重叠
torchrun --nnodes=8 --nproc-per-node=8 \
  --max-restarts=3 train.py \
  --parallelism.data_parallel_size=64 \
  --parallelism.tensor_parallel_size=1 \
  --data.batch_size=1 \
  --data.seq_length=4096 \
  --optim.grad_reduce_in_fp32=true \
  --optim.overlap_grad_reduce=true \
  --optim.bucket_size_mb=25 \
  --distributed.timeout_minutes=30

# 说明：
# bucket_size_mb 决定重叠窗口的粗细，与 overlap_grad_reduce 配套才有意义
# grad_reduce_in_fp32 决定通信负载，代价是缓冲区显存翻倍
# max-restarts 用于容忍单 rank 崩溃后重建进程，是基础容错手段

Another easily overlooked direction is compression of the communication volume. Low-precision communication — for example, letting bf16 gradients participate in the reduction and roughly halving the volume relative to fp32 — is usually numerically acceptable, because gradients themselves have noise tolerance. There are also techniques such as gradient sparsification, error feedback, and low-rank approximations of second-order statistics. All of them are worth introducing only once the communication bottleneck has overtaken the compute bottleneck, which gives a clear boundary of applicability: squeeze the compute hardware first, and only then consider communication compression. Reversing that order costs you convergence quality for nothing.

Tensor Parallelism: Why Every Layer Needs Two all-reduce Calls

The premise of data parallelism is that "each card can hold the entire model." When the model itself fills up the memory of a single card, that path stops working, and you have to slice an individual matrix multiplication horizontally — which is exactly what tensor parallelism is.

The classic approach splits the weight matrix by rows or by columns into several pieces, has each card compute only its shard, and aggregates the results when necessary. So that the shards cooperate correctly, the interior of a Transformer layer is designed with paired column-split and row-split operators alternating. This alternating structure is precisely why the "two all-reduce calls" exist.

A Pair of Conjugate Operators: Why One Call Is Not Enough

The cleanest way to understand this is to look at the two operators inside the attention module. Suppose the attention hidden dimension, or the number of heads, is split across N cards:

The first operator (QKV projection / first feed-forward layer) is split by columns. The columns of weight matrix W are split into N blocks, each card takes one block, and input X is replicated (via all-gather, or simply left replicated), so each card computes a shard of the complete output. At that point each card holds an incomplete result, which must be aggregated before it can move on to the next step.
The second operator (attention output projection / second feed-forward layer) is split by rows. Once the weight matrix is split by rows, what each card needs is the corresponding shard of the input — which is exactly what the previous step produced on each card. The benefit of the row split is this: summing the shards produced by the cards in the previous stage yields exactly the result of the full matrix multiplication, with no further rearrangement required.

So the key fact emerges: after a column split you need one reduction, and after the row split completes you need another. One transformation needs one communication on the way in and one on the way out, so every such module triggers at least two collective communications. This is not an implementation inefficiency; two reductions simply cannot be merged mathematically — the forward stage has to switch back and forth between the sharded state and the complete state, and that boundary cannot be removed.

If a scheme claims that tensor parallelism does "only one all-reduce per layer," then it is almost certainly paying extra somewhere: either it moved the reduction outside the layer so activations cannot be freed, or it introduced extra rearrangement communication, or it uses a different splitting convention and shifts the cost elsewhere. Total communication is conserved — it has merely been relocated.

Why the TP Degree Cannot Be Pushed Higher

TP has the highest communication frequency of all the strategies: at least twice per Transformer layer, and with many layers that becomes hundreds or thousands of collective communications. All of them happen on the most critical forward and backward paths, are strongly synchronized, and cannot overlap with the matrix computation in the same layer — because the subsequent computation has to wait for the reduction result. This is why TP is extremely sensitive to interconnect bandwidth, and also why the TP degree is usually confined to within a single node (8 cards or 16 cards) rather than spanning nodes.

More rigorously, the TP degree has an upper bound determined by the compute-to-bandwidth ratio. When the compute of a single node grows faster than the interconnect bandwidth, raising the TP degree actually makes the matrix on each card smaller and shortens compute time proportionally, while communication time barely changes or even gets worse — and utilization falls off a cliff. You can think of it this way: the more finely the matrix is sliced, the higher the relative overhead of communication becomes, until communication completely eats the compute.

# 张量并行配置：单节点内 8 卡张量并行，跨节点做数据并行
# 这是一种把 TP 关在高带宽域内的典型拓扑
model:
  tensor_model_parallel_size: 8
  pipeline_model_parallel_size: 1
  expert_model_parallel_size: 1
  sequence_parallel: true      # 同步归约/归约替代 scatter，进一步压通信
  context_parallel_size: 1
data:
  micro_batch_size: 1
  global_batch_size: 512
  seq_length: 8192
optim:
  overlap_grad_reduce: true
  overlap_param_gather: true
distributed:
  backend: nccl
  comm_hook: none
# 关键：TP=8 已经吃满节点内 NVLink 域，跨节点不要再叠加 TP

sequence_parallel deserves a separate mention. Its approach is to shard as well those activations and dropout masks that TP would otherwise replicate across cards, so that what used to be an all-gather degenerates into a cheaper all-reduce, while dropping the memory footprint of those tensors to 1/N. This is an optimization that reduces communication bytes without adding communication calls, which is a different thing from reducing the number of calls.

Communication Volume in the Backward Pass

A lot of people count only the two all-reduce calls in the forward pass and assume the accounting is done. The full ledger should be forward plus backward: in the backward pass every module likewise has to shuttle gradients back and forth between the sharded state and the complete state, so the communication count is roughly twice that of the forward pass and the total volume is on the same order. During capacity planning, treating the two forward calls as the whole story is the most common source of underestimation, and it will keep a parallel plan from reaching its expected throughput on a real cluster.

Matching Communication Patterns to Hardware Topology

Once the parallelism strategy has been narrowed down to a certain point, it stops being a software problem and becomes a topology problem. A cluster's interconnect usually has two layers: the high-speed domain inside a node (multi-card NVLink or NVSwitch) and the network domain between nodes (InfiniBand or Ethernet). The bandwidth, latency, and topology of these two layers differ by one to two orders of magnitude, and the upper bound on the parallelism degree of any strategy is basically determined by them.

Dimension	Within a node (NVLink domain)	Between nodes (IB / RoCE domain)
Bandwidth	Extremely high, on the order of hundreds of GB/s one-way	Moderate, on the order of tens of GB/s one-way
Latency	Extremely low	Clearly higher
Recommended parallelism to host	Tensor parallelism, expert parallelism, sequence/context parallelism	Data parallelism, pipeline parallelism
Typical limitation	Card count is bounded by slots and power	Bounded by network topology and the oversubscription ratio

This table is the direct basis for splitting the parallelism degrees: push the most communication-intensive dimensions inside the node and spread the sparsest dimensions across nodes. In the common configuration of 8-card nodes across 64 nodes, taking tensor parallelism = 8, expert parallelism = 8, pipeline parallelism = 8 (across nodes), and data parallelism = 8 (across nodes) is exactly how you assemble those 512 cards into a single 3D parallelism grid.

Topology Awareness Is Not Automatic

Link quality in a real cluster is not uniform: nodes in the same rack go through a top-of-rack switch, nodes across racks have to traverse leaf switches, and cross-availability-zone traffic takes an even longer path. If the communication groups are not aligned to the physical topology, collective communication ends up crossing slow links, and at scale that becomes a systemic bottleneck.

So in multi-dimensional parallelism, the mapping between parallel groups has to be declared explicitly: data parallelism groups go on the outermost dimension, where communication rides the slowest and most coarse-tolerant links, while tensor parallelism groups stay inside a node on the fastest links. In modern frameworks this is usually expressed through explicit construction of the device mesh and process groups rather than automatic derivation, because automatic derivation cannot read a rack diagram.

A second practical point is that the implementation quality of collective communication directly determines the ceiling. The same code with a different communication backend or a better-suited algorithm can differ by more than a factor of two. The choice depends on message size: for large messages it leans toward ring-like bandwidth algorithms, for small messages toward tree-like low-latency algorithms. Many frameworks ship adaptive selection inside their collective communication implementation; if your model structure changes dramatically (tensor parallelism and pipeline parallelism interleaved), it is worth measuring whether you are actually hitting the path you assumed.

Pipeline Parallelism: Bubbles, Scheduling, and Two Classic Schemes

When a single node cannot hold the model and the tensor parallelism degree is itself capped by bandwidth, the remaining natural option is pipeline parallelism: cut the model by layers into several segments and place each segment entirely on one card. During the forward pass, data flows like a pipeline from the first segment to the last; during the backward pass it flows back the other way.

Pipeline parallelism has very light communication — only the activations and gradients at the boundaries need to be sent, point-to-point, with no collective communication required. That is exactly why it suits crossing nodes. But it carries a structural efficiency loss: the bubble.

Where the Bubble Comes From, and How Large It Is

Ideally every card is doing useful work at every instant. In reality the first card is fully loaded at the start of the forward pass (it has to compute several micro-batches in a row), and the last card sits idle before the backward pass of the final segment. The time that fill and drain inevitably consume is the bubble.

The bubble ratio varies with the ratio of micro-batch count to pipeline stage count: the more micro-batches and the fewer stages, the thinner the bubble. The extreme case is when the stage count equals the micro-batch count, at which point there is no bubble at all, but every card is extremely imbalanced. What practitioners actually care about is the effective utilization of each card — the fraction of total time spent busy — and it directly determines how many cards you can trade for how much speedup.

To raise utilization there are two directions: first, increase the number of micro-batches to spread the bubble thinner; second, improve the schedule so the fill phase becomes shorter. The former raises peak memory, while the latter depends on the scheduling algorithm.

GPipe and 1F1B: Two Entirely Different Trade-offs

Naive pipeline scheduling (often called GPipe style) works like this: run the forward pass for all micro-batches first, store all the intermediate activations, then do the backward pass as one block, and finally perform a single gradient synchronization. The advantage is that the scheduling logic is extremely simple and communication is sent in large contiguous batches, which is efficient; the disadvantage is that all intermediate activations must be resident at the same time, so peak memory grows linearly with the micro-batch count — you are trading memory for throughput. In the era of large models that is often unacceptable.

The 1F1B schedule (one forward followed by one backward) changes the order into an alternating progression: after a pipeline stage completes the forward pass for one micro-batch, it immediately picks up a backward pass that it is able to do. Activations are then consumed and released shortly after they are produced, so peak memory drops from "micro-batch count × single-micro-batch activation" to roughly "stage count × single-micro-batch activation", essentially decoupled from the micro-batch count. This is the mainstream choice for very large model training today; the cost is that communication becomes fragmented, with boundary tensors sent point-to-point far more frequently.

Dimension	Naive schedule (all forward, then all backward)	1F1B schedule
Peak activation memory	Grows linearly with micro-batch count	Grows with stage count, decoupled from micro-batch count
Communication shape	Concentrated batches, point-to-point efficiency is high	Fragmented and interleaved, latency sensitivity matters
Scheduling complexity	Low	Medium-high, involves idling and redistribution
Typical use	Medium-scale, memory headroom available	Very large models, tight memory

Interleaved Scheduling and Finer Partitioning

There are two further routes to squeeze the bubble thinner. First, interleaved scheduling: have a single physical card carry multiple non-contiguous model segments in sequence, which essentially turns the virtual stage count into an integer multiple of the stage count, the bubble ratio drops by that multiple, and the order of the segments is carefully arranged. The cost is rising memory fragmentation and scheduling complexity, plus a need for more precise resource allocation.

Second, non-uniform partitioning. Different types of layers differ greatly in memory and compute cost: attention layers typically consume more activations than feed-forward layers, and the first and last layers additionally carry the overhead of embedding and output. Dividing the number of layers assigned to each card by memory footprint rather than by count often shaves off a large chunk of peak memory outright. The cost is that compute time is no longer uniform across pipeline stages, and the shape of the bubble becomes more complicated.

# 流水线并行 + 1F1B 的配置片段（示意）
# 目标是：段数 × 每节点卡数 × 数据并行度 = 总卡数
model:
  tensor_model_parallel_size: 8    # 节点内
  pipeline_model_parallel_size: 8  # 跨节点，1F1B
  expert_model_parallel_size: 1
  pipeline_model_parallel_schedule: 1F1B
  num_layers_per_virtual_pipeline_stage: 2   # 交错调度：每卡承担 2 段
data:
  micro_batch_size: 1
  global_batch_size: 1024         # = 微批数 × 段数 × DP 度 × 微批大小
  seq_length: 8192
# 注意：global_batch_size 必须能被 微批数 × DP 度 整除，否则会静默产生不均匀的尾部
# 气泡率约等于 段数/(微批数+段数-1)，交错调度按虚拟段数计算

One last easily overlooked point: the theoretical speedup of pipeline parallelism is bounded by the bubble, and when crossing nodes the transfer latency of boundary activations is charged directly to the pipeline tick. So placing stage boundaries on slower network links is a common but harmful practice; the placement of stage boundaries should be designed together with the topology.

Expert Parallelism: The Special Constraints MoE Brings

In a dense model every parameter participates in computation on every forward pass, which makes growth in parameter count directly equivalent to compute cost. A mixture-of-experts structure changes that: each token is routed to only a few experts, and experts that are not selected do not participate at all. So scaling up can be done mainly by adding experts, while the compute of each forward pass grows only with the number of activated experts.

But that dividend comes with strict conditions. Experts must be distributed across different devices, otherwise there is no parallelism to speak of; and once they are spread across devices, "sending the token to the right expert" turns into a global communication that has to happen in every single layer. This is the core tension of expert parallelism.

all-to-all Is the Primary Communication, Not all-reduce

Unlike the first three strategies, the communication primitive of expert parallelism is all-to-all. Each card holds a batch of tokens, every token has a routing target (which experts it lands on), and the experts are distributed across cards. So a global exchange has to happen: each card packs its tokens according to the target expert and sends them to the card holding that expert; the receiver runs its local expert on them and then sends the results back along the same path.

The communication volume of this all-to-all is proportional to token count × hidden dimension, and there are two of them per MoE layer (one to dispatch inputs, one to return outputs). If the MoE layers are dense, the total communication can add up to the other communications along the entire compute path and become a new bottleneck.

Three important engineering judgments:

all-to-all is extremely sensitive to topology. It involves both cross-node and intra-node traffic at once and requires a matching process group partitioning. Expert parallelism confined inside a node communicates over NVLink and works well; cross-node expert parallelism needs a network with extremely high bandwidth and low oversubscription, otherwise the cross-node token exchange stretches out the latency of the entire MoE layer.
The implementation quality of all-to-all varies enormously. Some implementations chunk and pipeline the exchange so that part of the token exchange overlaps with computation, while others are closer to blocking. This is a substantive difference in the ability to overlap communication with computation, and it is worth measuring.
Token replication and expert redundancy affect the actual load. The router may assign far more tokens than average to popular experts, and that is load imbalance.

The Capacity Factor: A Dial Between Imbalance and Dropping

The communication volume of all-to-all is determined directly by the number of tokens actually exchanged, and the number of tokens actually exchanged is token count × number of experts routed to. That gives you a dial you can turn yourself: the capacity factor, meaning how many tokens each expert reserves the capacity to handle.

When the capacity factor limit is triggered, the excess tokens are dropped (or take a fallback path). This forms a clear trade-off: a smaller capacity factor lowers memory footprint and communication volume and raises throughput, but drives the drop rate up and damages quality; a larger capacity factor makes quality steadier but makes both memory and communication more expensive. At the same time you have to account for the auxiliary load balancing loss — training usually adds a penalty term that pulls the token load of different experts closer together, preventing the model from degenerating into a few experts monopolizing all traffic early in training. Without that constraint, expert parallelism will be backlashed by router imbalance during training.

The Conflict Between EP and TP

This is where expert parallelism needs the most care. When the expert weights themselves are also split by tensor parallelism, the target "expert" of a token becomes "a shard of some expert on some card", and the dispatch target changes from "send to one expert" to "send to different shards of the same expert on multiple cards". The communication pattern grows correspondingly more complex: either you split again along the shard dimension and reuse all-to-all, or in some implementations it degenerates into a more expensive sequence of communications.

The practical judgment is simple: EP and TP compete for the same splitting dimension rather than stacking on top of each other. Pick one path — either use EP to place different experts on different cards (constrained by load balancing and the capacity factor), or use TP to split the weight matrix of a single expert. Very few designs manage to push both to large values. Once the two are scaled up together, the communication complexity climbs faster than the returns.

# 专家并行配置：专家放在节点内 8 卡，模型其余部分用流水线铺开
model:
  expert_model_parallel_size: 8       # 8 个专家分散到节点内 8 卡
  tensor_model_parallel_size: 1       # 不与 EP 叠加，避开 all-to-all 复杂化
  pipeline_model_parallel_size: 16    # 靠 PP 铺开到 16 节点
  expert_tensor_parallel_size: 1      # 单个专家内部不再切分
  # MoE 层的关键旋钮
  expert_capacity_factor: 1.25        # 容量因子：显存/通信 ↔ 丢弃率 的旋钮
  moe_router_load_balancing_type: aux_loss   # 辅助负载均衡损失
  moe_aux_loss_coeff: 0.01            # 太小易退化，太大伤质量，需按步数调
  moe_router_topk: 2                  # 每 token 路由到几个专家：直接乘在 all-to-all 量上
# 诊断要点：看 all-to-all 字节数是否 ≈ token 数 × topk × 隐维 × 数据类型大小
# 若明显偏大，多半是容量因子过大或 token 复制策略不对

3D Parallelism: How to Choose the Splitting Dimensions

When no single parallelism approach can carry the load, you have to combine them. This is called 3D (or higher-dimensional) parallelism: combining the data, tensor, pipeline, and expert dimensions into a parallel grid, with each dimension responsible for absorbing one kind of resource pressure.

The hard part of combining them is not whether you can combine them, but which dimension you split first. Get the order wrong and you will spend the scarcest resource inside the high-bandwidth domain.

The Criterion for Ordering: Who Fears Slow Links Most

There is only one core criterion, but it is very sharp: split the dimensions with the densest communication, the ones that can least tolerate slow links, first, and let them land inside a node; split the dimensions with sparse communication later, and let them cross nodes.

Step one: take tensor parallelism (and expert parallelism) as high as it goes inside the node. The communication of both happens on the forward and backward path of every module, which is exactly what slow links can least tolerate. Devote all 8 or 16 cards inside a node to TP/EP, making the model thinner.
Step two: choose between expert parallelism and tensor parallelism, or combine them cautiously. If the model is expert-dense MoE, EP will naturally fight for the slots inside the node; at that point it is common to shrink TP or even set it to 1 and hand the intra-node bandwidth over to EP. This is not "EP is better" but "in this model EP is the main source of parallel speedup."
Step three: pipeline parallelism spreads the thin model across nodes. PP only ships boundary activations and can easily bear crossing nodes. Its job is to absorb the pressure of "not enough nodes / not enough memory to hold everything," trading bubble ratio for scalability.
Step four: data parallelism replicates over the remaining dimension. The DP degree = total card count ÷ (TP × PP × EP). It does not change how large a model a single card has to hold, only the total memory and total communication; its job is to enlarge the global batch and amortize fixed overhead.

Summing it up in one slogan: TP / EP fill the node interior → PP spreads across nodes → DP holds up the batch.

Splitting dimension	Which layer it lands on	What it primarily absorbs	Order of splitting	Cost of raising it
Tensor parallelism	Within a node	The model does not fit on a single card	First, taken as high as possible inside a node	Many communication calls, sensitive to slow links
Expert parallelism	Within a node	MoE experts do not fit	Competes with TP for the same intra-node slots	all-to-all plus load imbalance
Pipeline parallelism	Across nodes	Not enough nodes	In the middle, used for spreading	Bubble plus boundary activation latency
Data parallelism	Across nodes	Enlarging the global batch	Last, on the remaining dimension	Synchronization barrier plus tail latency

Total Communication Volume: How Parallelism Degree Affects the Ledger

To judge whether a combined plan is sound, the most practical approach is to add up the total communication volume of one training run by parallelism type and look at each item's share:

Data parallelism gradient synchronization: on the order of the parameter count, happening once after the backward pass. Raising the DP degree does not change this volume on a single card, but the number of synchronizations scales with the number of steps; whether bandwidth can sustain it depends on the parameter count and the time between steps.
Tensor parallelism intra-layer communication: on the order of hidden dimension × sequence length × dtype size, occurring N layers × (forward plus backward) times. Here the TP degree does not change the total volume of a single call (after splitting, total bytes are conserved; the duplicated part comes from replication), but a larger TP makes per-card compute time shorter, so the relative communication overhead rises.
Pipeline parallelism boundary transfers: only one boundary worth of activations and gradients is sent — small in volume but frequent, and across nodes the real enemy is latency.
Expert parallelism all-to-all: token count × topk × hidden dimension × dtype size, twice per layer. Raising the EP degree likewise does not significantly reduce the total, but the cross-card fraction rises.

Once you add the four items together you will notice a pattern: for dense models, TP communication is usually the dominant term; for MoE models, the TP communication term gets pushed up by the EP all-to-all and its magnitude may even overtake the rest. This is also why different model types need different splitting orders — the criterion is only a single rule, but once it lands on different models the optimal solution is completely different.

ZeRO: What Stages 1/2/3 Actually Save

Parallelism strategies solve the problem of how to split the model; ZeRO solves a different, orthogonal problem: during data parallelism every card stores the complete optimizer state, gradients, and parameters — can that redundancy be eliminated? It does not change how the model is split; instead it shards the state that data parallelism would otherwise replicate per card onto different cards.

To understand ZeRO, first look at the four things each card stores under data parallelism: parameters, gradients, optimizer state, and the activations needed for backward computation. The three ZeRO levels are defined by how many of these get sharded.

Level	What gets sharded	Where the memory is saved	Communication cost	Key benefit
ZeRO-1	Optimizer state sharding	Optimizer state (usually the largest memory consumer)	Same order as DP (one extra reduce-scatter/all-gather combination)	Larger models on the same number of cards
ZeRO-2	Optimizer state + gradient sharding	Plus gradients	Same order as ZeRO-1	More headroom for activations and temporary buffers
ZeRO-3	Optimizer state + gradients + parameters, all sharded	Plus parameters, approaching full sharding	Clearly higher (every layer needs parameters fetched for forward/backward)	The model size a single card can carry rises substantially

The most commonly misunderstood point is this: what each ZeRO level mainly saves is state memory — optimizer state, gradients, and parameters — not activations. Activations are primarily reduced by gradient accumulation, checkpointing, sequence parallelism, and context parallelism, which are a separate set of techniques. Confusing the two leads to the situation where someone adopts ZeRO-3, sees activations still blow up, and wrongly concludes that ZeRO does not work.

The Communication Cost of ZeRO-3 Is Real

ZeRO-1 and ZeRO-2 move roughly the same volume of communication as ordinary data parallelism, because they shard only state and not the parameters used by the forward pass, so each card still holds complete parameters during forward computation. ZeRO-3 is different: parameters are sharded too, so each card holds only one shard of the parameters at any moment, and when forward and backward computation needs parameters it must fetch them on demand and release them immediately after use. This introduces an extra all-gather and reduce-scatter in every layer.

So ZeRO-3 is a textbook case of trading communication for memory: it lets you run a larger model on the same cards, but if compute is already saturated by communication, throughput drops. The judgment is direct — look at what share communication takes in your training. If it is already tight, ZeRO-3 may backfire; if memory is the bottleneck and compute has headroom, ZeRO-3 lets you trade up the model scale. Empirically, the memory gain from ZeRO-2 to ZeRO-3 is large, but the throughput loss is also noticeable; on a cluster where communication is already saturated, doing the pipeline/tensor parallelism partitioning first and only then considering ZeRO-3 is more stable than reaching for ZeRO-3 right away.

Treating ZeRO-3 as free memory expansion is dangerous. It trades communication for memory, and the bill has to be computed in the profile rather than by intuition.

The sharding idea behind ZeRO is also completely orthogonal to pipeline and tensor parallelism: ZeRO shards state along the data parallelism dimension, while TP/PP split structure along the dimensions of the model. So "TP + PP + ZeRO" can absolutely be used together; combined, it amounts to "first use TP/PP to cut the structure down to a scale a node can swallow, then use ZeRO to squeeze out the remaining state redundancy."

Heterogeneity and Fault Tolerance

At the scale of hundreds or thousands of cards, where a training run is measured in weeks, "can it survive a single failure" has gone from a nice-to-have into a precondition for whether the project succeeds at all. Here the parallelism strategy and the fault tolerance mechanism constrain each other.

Failure Types and Their Consequences

Single card failure / process crash. The most common case. The consequences depend on the strategy: under data parallelism, losing one card usually means the entire communication group has to be rebuilt (because it holds sharded state) and the current step has to be rerun; under pipeline parallelism, losing one card means the downstream of the whole pipeline has to wait for it. The cost of restarting is proportional to the number of steps until the next checkpoint.
Link degradation / timeouts. More insidious than a crash: one link slowing down makes the whole collective communication drag its tail, and then every card waits along with it. The symptom is throughput quietly declining rather than an error, which is often harder to diagnose than a crash.
Out-of-memory and fragmentation. Peak memory is a probabilistic event, and tiny load imbalances between nodes cause some cards to overflow while others stay fine.

Checkpoint Granularity: Balancing Recovery Cost Against Training Cost

The core fault tolerance mechanism is periodic checkpointing. Checkpoints that are too sparse mean a single failure wastes a long stretch of work; checkpoints that are too dense make disk writes and storage bandwidth the new bottleneck and can even slow training down. The reasonable approach is to make the cost of writing a checkpoint far smaller than the progress you can make between two checkpoints.

The complexity of distributed checkpointing lies in the fact that model state, parallelism strategy, optimizer state, random number state, and data loading progress all have to be saved in a coordinated way, and are bound to the parallel configuration. After you change the parallel configuration, old checkpoints usually cannot be used directly. Versioning the checkpoint contents together with the parallel configuration is the precondition for making fault tolerance genuinely usable.

Two more fault tolerance engineering points: first, elastic scaling — allowing nodes to be added or removed mid-training with the parallelism degree adjusted, which requires the design to tolerate re-sharding from the outset; second, the granularity of suspend and resume — being able to skip a failed node without restarting processes is far cheaper than a full process restart, but it demands a much higher implementation bar.

Long Sequences and Low Precision: Two Variables Entangled With the Parallelism Strategy

Everything discussed so far assumes that sequence length and numerical precision are given constants, but in real training these two variables are strongly coupled to the parallelism strategy: they change both memory footprint and communication volume at the same time. Most of the time they should be planned together with the parallelism degree rather than fixed up front and patched afterwards when something goes wrong.

Activations: The Most Underestimated Slice of Memory

Parameters, gradients, and optimizer state are all static, but activations grow rapidly with sequence length. Attention-related intermediate tensors grow approximately quadratically with sequence length in a naive implementation, which means the memory pressure from doubling the sequence length can far exceed doubling. In very long-context training, it is activations rather than parameters that are usually the real memory bottleneck.

There are several ways to deal with activations, and each interfaces with the parallelism strategy differently. Gradient accumulation lowers the peak by shrinking the scale of each forward pass, at the cost of more steps; activation recomputation trades compute for memory by rerunning the forward pass during the backward pass and keeping only boundary tensors, at the cost of extra compute; sequence parallelism also shards the activations that tensor parallelism would otherwise replicate across cards, cutting the memory of those tensors to a fraction of the original while also reducing communication bytes; context parallelism cuts the whole sequence by position, degenerating attention into cross-card reductions and merges, and is the most thorough way to handle very long sequences.

There is an easy place to go astray here: before choosing an activation technique, prove that the bottleneck really is activations. If memory statistics show activations are only a small slice, then continuing to optimize activations is a waste and you should go back to the parameter and state side and solve it with sharding. This judgment is worth redoing before every change to the parallel configuration, because once the parallelism degree changes the proportions shift and the conclusion from last time may no longer hold.

Another common mistake is equating "memory is sufficient" with "memory is plentiful." Peak memory fluctuates with the step count, and you have to collect it at the peak; once fragmentation is layered on top, there is still a gap between the statistical value and what is actually allocatable. Conflating the two easily leads to an overflow only after tens of thousands of steps, and it is very hard to reproduce.

Low Precision: Compressing Compute and Communication at the Same Time

The value of low-precision training is easily narrowed to "it just saves memory." In reality it acts on three things at once: the compute cost of matrix multiplication, the size of activations, and the number of bytes for the tensors that travel over the network. Communication volume drops roughly linearly with the bit width of the dtype, so on a cluster where communication is already close to saturation, switching the communication tensors to a low-precision format can yield more than simply reducing compute precision — it directly shortens a stretch of time that would otherwise block subsequent computation.

But low precision is not free. The accumulation order during reductions, the number of gradient accumulation steps, and which tensors stay in high precision all affect numerical stability. Judging whether a precision swap is worth it should also come down to the profile: if communication takes a clearly large share of a single step, changing the dtype on the communication side is usually the highest-leverage edit; if compute is the bottleneck, the focus should shift to routing matrix multiplication through a lower-precision execution path. The two can be decided separately — you do not have to convert everything at once, nor convert everything at the same time.

There is another class of change with a similar character: making communication more fragmented or more contiguous. The former lowers the volume per call but raises the call count, and the latter is the opposite. Moving back and forth between latency-bound and bandwidth-bound can sometimes pay off far more than a precision swap, because the latency side is almost unaffected by bit width.

Planning All Three Together: A Checklist Order

When you look at sequence length, precision, and parallelism degree together, you can verify layer by layer in the order below, with each layer solving only its own class of problem.

First check whether a single card can hold the parameters and state on its own. If it can, go straight to data parallelism plus gradient accumulation — the least worrying case.
If it cannot, decide between structural splitting and state sharding. Structural splitting changes the shape of the model on the card, while state sharding only changes how redundancy is stored. Their communication characteristics differ, so you have to compute the total ledger together.
Then look at how much activations account for at the current sequence length. If the share is high, use gradient accumulation or recomputation to push the peak down; if the share is low, do not spend effort there.
Then estimate the total communication volume per step and see whether it is acceptable at the current link bandwidth. If it is not, prioritize reducing the dtype on the communication side or reducing the number of communication calls, rather than adding cards.
Only at the end decide whether to keep high precision for convergence quality. This step is a trade-off rather than an optimization, and should only be taken once the previous four are stable.

If the conclusion of any step changes, all the later steps have to be recomputed — which is exactly why so many people underestimate the effort of parallel configuration: it is not a parameter you can tune once, but the solution to a set of mutually constraining conditions given a fixed cluster shape. The value of this checklist is that it turns "parallelism degree" from an isolated knob back into an outcome constrained jointly by memory, sequence length, precision, and topology. Only by planning the four together can you explain why the same model code needs a different parallel configuration on different clusters — what changed is not the model, but the relative tightness of these four constraints.

Performance Collapse from Wrong Parallelism Choices: What It Actually Looks Like in Real Training

This is the most practically useful section of the article. A wrong parallel configuration does not fail immediately — it runs, but slowly — and the reason for the slowness is often misdiagnosed as "not enough GPUs" or "the model is too big." Below it is organized by symptom.

Symptom One: Adding Cards Makes It Slower

Symptom: raising the DP degree from 8 to 16 makes wall-clock time go up instead of down; or raising TP from 1 to 4 drops throughput instead of raising it.

Cause: communication is already saturated. Doubling the DP degree does not change the total gradient synchronization volume of a single step, but it doubles the number of participants in the synchronization and doubles the exposed tail latency; raising the TP degree makes the matrix on each card smaller and shortens compute time proportionally, while the two intra-layer all-reduce calls take almost the same time or even get worse, so the relative overhead spikes.

Diagnosis: look at the share of collective communication within single-step time in the profile. If communication already occupies most of the time, adding cards is just adding overhead; if compute has a high share, there is still parallel headroom.

Symptom Two: Throughput Slowly Declines with Training Steps

Symptom: fine early on, but each step gets slower after a few thousand steps.

Cause: mostly related to the strategy or the state. First, memory fragmentation accumulates over time, so memory that should have been plentiful gradually approaches the peak. Second, load imbalance — if it is MoE, popular experts make the all-to-all byte count and the communication imbalance grow worse as training proceeds. Third, under expert parallelism the router collapses, with tokens flooding toward a few experts and the distribution becoming increasingly skewed.

Diagnosis: check whether memory usage trends upward, whether all-to-all byte counts rise with the step count, and whether the expert load distribution chart is imbalanced.

Symptom Three: Effective Speedup Falls Far Below Theory

Symptom: N times the cards should give close to N times the speedup, but in reality you get a fraction of that, or even a net loss.

Cause: the bubble (too many or too deep pipeline stages relative to micro-batches), cross-node boundary latency, parallel groups not aligned to the physical topology (a data parallelism group spanning racks), collective communication taking a suboptimal algorithm, or load imbalance across pipeline stages causing idle time at the tail.

Diagnosis: pull out the busy time of each card and make a histogram — you can see the bubbles and whether the stages are balanced at a glance. If some cards are clearly idling much more, it is pipeline or load imbalance; if all cards are uniformly busy but total throughput is low, the problem lies in communication or in the kernel implementation.

Symptom Four: MoE Training Quality Will Not Improve, or Metrics Drop for No Reason

Symptom: the loss will not go down, only a few experts are ever used, or quality degrades for no reason late in training.

Cause: the capacity factor is too small so the drop rate is high; the auxiliary load balancing loss coefficient is wrong (too small to suppress imbalance, too large to hurt routing); EP and TP are stacked so the shards a token routes to are inconsistent; all-to-all communication does not cover everything, so some tokens fail to route.

Diagnosis: look directly at the drop rate and the expert load distribution instead of guessing from the loss curve. On the communication side, back the byte count out: theoretical value = token count × topk × hidden dimension × dtype size, and if the measured value is clearly larger it means there is extra communication or token replication.

Symptom Five: Memory Is Obviously Enough and You Still OOM

Symptom: the optimizer + gradient + parameter footprint estimated through ZeRO is far smaller than actual memory usage, yet it still overflows.

Cause: the bulk is actually activations and communication buffers, and ZeRO does not manage activations. Under long sequences activations grow super-linearly with sequence length; the preallocated buffers of collective communication can be very large; fragmentation turns "sufficient" into "not sufficient."

Diagnosis: instrument parameters / gradients / optimizer / activations / communication buffers / fragmentation as six separate line items, and do not back out conclusions from total memory. The mitigations are gradient accumulation to push the peak down, sequence or context parallelism to compress per-sequence activations, and putting a cap on communication buffers.

Attributing the Collapse to a "Wrong Dimension"

All five of these collapses, traced back to their root, are almost always the wrong splitting dimension having been chosen: putting the dimensions that fear slow links onto slow links (TP/EP crossing nodes), budgeting the bubble too tightly (PP too deep with too few micro-batches), treating a problem that needs state sharding as a structural problem (using only DP without ZeRO), or mixing up the responsibilities of structural splitting and state sharding. To judge whether a parallel configuration is right, ask yourself three questions:

Are the parallelism dimensions that fear slow links most (TP, EP) all confined inside a node? If they cross nodes, pull them back inside the node first.
Is the pipeline bubble rate an acceptable number at the current micro-batch count? If the bubble eats up more than half the time, consider adding micro-batches, reducing the stage count, or switching to interleaved scheduling.
For the part under memory pressure, is it state or is it activations? If it is state, reach for ZeRO; if it is activations, reach for gradient accumulation and sequence/context parallelism — using the wrong tool on either side wastes your effort.

Selection Advice: An Actionable Decision Table

Finally, converge the whole article into a selection logic you can follow directly. The assumptions below are a standard cluster: multiple nodes, NVLink within a node, a high-speed network between nodes.

Your situation	Primary strategy	Secondary strategy	Metric to watch
The model fits on a single card, but there are not enough cards to go fast	Data parallelism	ZeRO-1/2 + gradient accumulation	Share of collective communication in single-step time
The model does not fit on a single card, but there are enough nodes	Tensor parallelism taken as high as possible inside a node	Cross-node data parallelism + ZeRO	Matrix size per card, relative communication overhead
Even a single node cannot hold it, so multiple nodes are needed	TP inside the node + pipeline across nodes	1F1B schedule + ZeRO-2/3	Bubble rate, compute balance across stages, peak activation memory
Dense MoE experts, many parameters but little compute per token	Expert parallelism inside the node	TP stacked cautiously; PP spread across nodes	all-to-all byte count, drop rate, expert load distribution
Activations are the memory bottleneck (long sequences)	Context/sequence parallelism	Gradient accumulation + checkpointing	How per-sequence activation grows with length
State is the memory bottleneck (many parameters, large state)	ZeRO-2/3	Combined orthogonally with TP/PP	State footprint after sharding vs added communication

There are a few more empirical reminders worth emphasizing on their own:

Do the math before you add cards. Add up the total communication volume across the four items and compare it with single-step time. If communication takes a high share, optimize the communication share first rather than adding cards.
Change only one dimension at a time. Tuning parallelism degree and hyperparameters together makes attribution impossible when something breaks. Change one item, run enough steps, and look at the curves.
Design topology and parallel groups together. Problems like a data parallelism group spanning racks or a tensor parallelism group spanning nodes take a minute to fix in a config file, and cost continuous performance loss at runtime.
Do not treat communication compression as the default answer. First squeeze the compute hardware and the schedule (bucketing, overlap, the right algorithm), and only then consider low-precision communication or sparsification to trade for bandwidth.
Budget for fault tolerance. Checkpoint interval, training configuration, and parallel configuration should be versioned as a whole, so that recovery can line up and continue the run.

Summary

The differences between the four parallelisms are, in essence, different landing points for the three pressures of memory, compute, and bandwidth across different dimensions of the model. Data parallelism trades batch size for compute, paying for it with one heavy communication and one hard synchronization per step; tensor parallelism trades bandwidth for model capacity, paying for it with at least two unhideable collective communications per layer, which is why it must be confined inside a node; pipeline parallelism trades bubbles for cross-node scalability, paying with efficiency loss that is amortized by the micro-batch count and the 1F1B schedule; expert parallelism trades all-to-all for parameter count, paying with the whole apparatus of routing and load balancing. ZeRO works in an orthogonal direction — it leaves the structure alone and shards state along the data parallelism dimension, trading communication for state memory.

Put these threads together and the splitting order for 3D parallelism follows naturally: split what fears slow links first, inside the node; split what communicates sparsely later, across nodes; squeeze out state redundancy with ZeRO along the data parallelism dimension. And every collapse — adding cards makes it slower, throughput declining, speedup falling short, MoE metrics dropping, memory obviously sufficient yet overflowing — is in essence some manifestation of drifting away from this principle. Once you learn to locate problems with the three groups of metrics in the profile — how much of a single step is communication, how even the busy time across cards is, and whether the all-to-all byte count adds up — performance collapse stops being mysticism and becomes a tractable engineering problem.

The direction in which parallelism strategies are evolving is very clear: keep "the computation that fears slow links most" inside the smallest possible block of the fastest hardware, spread "the part that does not fear slow links" out to trade for scale, and squeeze state redundancy away with sharding. Understanding these three principles and landing them on a concrete training configuration matters far more than memorizing the parameter names of any single framework.