DeepSeek Model Architecture Deep Dive

Deep understanding of DeepSeek's core technical innovations: MoE (Mixture of Experts), MLA (Multi-head Latent Attention), MTP (Multi-Token Prediction), FP8 mixed-precision training. From principles to code, from architecture to performance, fully master the DeepSeek technology stack.

Start Exploring

DeepSeek Architecture Overview

DeepSeek Model Architecture Panorama

Technical architecture panorama of DeepSeek V3 and R1. 671B total parameters, 37B activated parameters, multiple world-class innovations.

DeepSeek V3 Architecture Overview

DeepSeek V3 adopts the MoE (Mixture of Experts) architecture as its core design. The total parameter count is 671B, but only 37B parameters are activated per token during inference. This is thanks to its sparse activation mechanism: each token is routed to only a small number of expert subnetworks, rather than activating all parameters.

The core components of the V3 architecture include:

Architecture Component	Technical Description	Key Parameters
Transformer Backbone	Decoder-only architecture based on Transformer	61-layer Transformer
DeepSeekMoE	Fine-grained expert segmentation + shared expert isolation	256 routed experts + 8 shared experts
MLA	Multi-head Latent Attention, low-rank KV compression	KV Cache compressed to 1/5~1/10 of original
MTP	Multi-Token Prediction, predicts next N tokens simultaneously	Depth M = 1 (predicts 1 extra token)
FP8 Training	First validation of FP8 training on ultra-large-scale models	Training cost $5.57 million

V3 vs R1 Architecture Comparison

V3 and R1 share the same underlying architecture (MoE + MLA), but differ in training strategy and objectives:

Comparison Dimension	DeepSeek V3	DeepSeek R1
Base Architecture	671B MoE + MLA	Based on V3 architecture (671B)
Training method	Supervised pre-training + SFT	Reinforcement learning (RL) + cold-start SFT
Core capabilities	General conversation, writing, translation, knowledge Q&A	Reasoning, math, programming, logical thinking
Inference mode	Direct answer generation	Chain-of-Thought reasoning
Context window	160K	160K (671B) / 128K (distilled version)

671B Parameter Overview

The parameter distribution of DeepSeek V3's 671B is as follows:

Parameter Composition

Embedding Layer

Vocabulary size 128K, embedding dimension 7168. Multi-head latent attention compresses KV into a low-dimensional latent space, significantly reducing parameter count.

Vocabulary embedding: 128K x 7168
Positional encoding: RoPE (Rotary Position Embedding)

Parameter Composition

Transformer Layers

61 Transformer layers, each containing an MLA attention module and a DeepSeekMoE FFN module. The first 3 layers use dense FFN (non-MoE).

61-layer Decoder-only
58 MoE layers + 3 Dense FFN layers

Parameter Composition

MoE Expert Layers

256 routed experts + 8 shared experts. Each token activates 8 routed experts + 1 shared expert. Each expert is a small FFN network.

256 routed experts + 8 shared experts
Top-8 gating routing

Parameter Composition

Activated Parameters

37B activated parameters per token. Through sparse activation, only 5.5% of parameters participate in each inference computation, greatly reducing computational overhead.

37B / 671B = 5.5% activation rate
Inference cost reduced by 90%+

MoE 混合专家架构

MoE 混合专家架构详解

MoE（Mixture of Experts）是 DeepSeek 的核心架构创新。671B 参数中仅激活 37B，大幅降低推理成本。

专家路由机制

在传统 Transformer 中，每个 token 都经过相同的 FFN 层处理。MoE 架构将 FFN 层替换为多个「专家」子网络，每个 token 通过门控网络（Gate）选择最相关的专家进行计算。

DeepSeek V3 的 MoE 设计如下：

总专家数：256 个路由专家（Routed Expert）+ 8 个共享专家（Shared Expert）
每 Token 激活：8 个路由专家 + 1 个共享专家 = 9 个专家参与计算
门控策略：Top-K Gating，K = 8（从 256 个专家中选择得分最高的 8 个）
共享专家：8 个专家对所有 token 始终激活，捕获通用知识

Top-K 门控机制

门控网络是一个线性层，将 token 的隐藏状态映射到 256 维的专家得分向量。然后选择得分最高的 K 个专家，通过 softmax 归一化计算各专家的权重：

      # Top-K 门控路由（伪代码）

      gate_logits = Linear(hidden_states)               # [batch, seq, 256]

      topk_weights, topk_indices = topk_softmax(gate_logits, k=8)  # 选择 Top-8 专家

      expert_output = sum(weight * expert[expert_idx](hidden_states) for each selected expert)

      shared_output = shared_expert(hidden_states)     # 共享专家始终激活

      final_output = expert_output + shared_output

其中 topk_softmax 先选择 Top-K 专家，再在选中专家间做 softmax，未被选中的专家不参与计算。

Auxiliary-Loss-Free 负载均衡

传统 MoE 模型通常使用辅助损失（auxiliary loss）来鼓励专家负载均衡，防止「专家坍塌」——即大部分 token 被路由到少数几个专家，其余专家闲置。但辅助损失会增加训练复杂度，且可能影响模型性能。

DeepSeek 提出了一种创新的 auxiliary-loss-free 负载均衡策略：

Expert Bias 动态调整：每个专家维护一个偏置项（bias），训练过程中根据专家负载动态调整。负载过高的专家降低其偏置，负载过低的专家提高偏置。
Sequence-level 辅助损失：在序列级别而非 token 级别施加轻微平衡约束，确保每个序列内的专家使用分布均匀。
互补调度：当某个专家负载过高时，系统自动将部分 token 路由到次优但可用的专家。

为什么 MoE 高效？

MoE 架构的核心优势在于 稀疏激活。671B 的总参数提供了海量知识容量，但每次推理仅激活 37B（5.5%），使得计算量远低于同规模的密集模型。

Efficiency Advantage

Reduced Computation

The dense 671B model requires computing all 671B parameters for each inference. MoE only computes 37B activated parameters, reducing computation to about 1/18 of the dense model.

FLOPs reduced by ~94%
Inference latency reduced by ~90%

Efficiency Advantage

Knowledge Capacity Maintained

Although only a small number of parameters are activated each time, the total parameter scale of 671B allows the model to store far more knowledge than a dense model, with different experts specializing in different domains.

671B total knowledge capacity
Expert specialization

Efficiency Advantage

Training Cost Optimization

Sparse activation + FP8 mixed-precision training reduces DeepSeek V3's training cost to $5.57 million, only 1/10 to 1/20 of comparable dense models.

Total cost of $5.57 million
H800 GPU cluster

MLA Multi-head Latent Attention

MLA Multi-head Latent Attention Mechanism

Multi-head Latent Attention (MLA) is DeepSeek's attention innovation, which significantly reduces inference memory usage through low-rank KV compression.

Problems with Traditional MHA

In standard Multi-Head Attention (MHA), each token needs to cache its Key and Value vectors to support autoregressive generation of subsequent tokens. For long sequences, the memory usage of KV Cache grows rapidly:

      # KV Cache size for traditional MHA

      KV_Cache = 2 * num_layers * num_heads * d_head * seq_len * batch_size

      # For DeepSeek V3 (128 heads, 128 dim, 61 layers, 128K context):

      # KV Cache per token is approximately 61 * 2 * 128 * 128 = 1,998,848 floats = 7.6 MB

      # KV Cache for 128K context is approximately 7.6 MB * 128K ≈ 973 GB (unacceptable)

Core Idea of MLA: Low-Rank KV Compression

The core innovation of MLA is to project Key and Value into a low-dimensional latent space before attention computation. Specifically:

KV Compression Projection: Compress the original high-dimensional Key/Value into a low-dimensional latent vector c_KV (dimension much smaller than original) using a down-projection matrix.
Cache Latent Vector: During inference, only cache the low-dimensional latent vector c_KV, not the full Key and Value matrices.
Up-Projection Restoration: During attention computation, restore the latent vector to full Key and Value using an up-projection matrix.

      # MLA attention computation flow

      # 1. Compression projection (executed during both training and inference)

      c_KV = W_down_KV @ hidden_states          # [batch, seq, d_latent]  d_latent << d_model

      # 2. Cache latent vector (inference only)

      KV_Cache = c_KV                            # Only cache low-dimensional latent vector

      # 3. Up-projection restoration (during attention computation)

      k = W_up_K @ c_KV                         # Restore to full Key

      v = W_up_V @ c_KV                         # Restore to full Value

      # 4. Standard attention computation

      q = W_q @ hidden_states

      attention = softmax(q @ k^T / sqrt(d_head)) @ v

      output = W_o @ attention

RoPE Decoupling Design

RoPE (Rotary Position Embedding) is a widely used positional encoding method in modern Transformers. However, RoPE conflicts with low-rank compression: RoPE operates on Key vectors, while MLA compresses the joint representation of Key and Value.

DeepSeek's solution is RoPE decoupling: split both Query and Key into two parts each — one part processed by RoPE (capturing positional information), the other not (preserving content information). The RoPE part is handled separately and does not participate in KV compression.

      # RoPE decoupling design

      # Query split

      q_content = W_qc @ hidden_states          # Content part, not through RoPE

      q_rope = W_qr @ hidden_states             # Position part, through RoPE

      q_rope = RoPE(q_rope)

      # Key split

      k_content = W_kc @ c_KV                   # Content part, restored from latent vector

      k_rope = W_kr @ hidden_states             # Position part, computed separately

      k_rope = RoPE(k_rope)

      # Concatenate and compute attention

      q = concat(q_content, q_rope)

      k = concat(k_content, k_rope)

      attention = softmax(q @ k^T / sqrt(d_head)) @ v

Memory Savings Calculation

MLA's KV Cache compression effect is significant. Assume latent dimension is d_latent, original KV dimension is d_model:

Compression Ratio

MLA Compression Effect

In DeepSeek V3, d_latent = 512, while traditional MHA has KV dimension of 128 * 128 = 16384. Compression ratio is about 32:1.

Latent dimension: 512
Original KV dimension: 16384
Compression ratio: ~32x

Memory Savings

Actual Effect

Under 128K context window, MLA reduces KV Cache from about 973 GB to about 40 GB, making long-context inference possible on a single GPU.

Traditional MHA: ~973 GB
MLA: ~40 GB
Savings: ~96%

Inference Acceleration

Throughput Improvement

Smaller KV Cache means less memory read/write and higher throughput. MLA improves long-sequence inference throughput by 3-5 times.

Memory bandwidth savings: ~90%
Throughput improvement: 3-5x

MTP Multi-Token Prediction

MTP Multi-Token Prediction Mechanism

Multi-Token Prediction (MTP) allows the model to predict multiple future tokens simultaneously, improving training efficiency and inference quality.

How MTP Works

Traditional autoregressive models predict only the next token at each step. MTP allows the model to predict the next N tokens simultaneously at each step (in DeepSeek V3, M = 1, i.e., predict 1 additional token, total 2 tokens).

The core idea of MTP is: after the model's backbone network outputs hidden states, multiple independent "Prediction Heads" are used to predict tokens at positions t+1, t+2, ..., t+M respectively.

      # MTP forward pass (DeepSeek V3, M=1)

      # Main model forward

      hidden = transformer(input_tokens)              # Main model output

      # First prediction head (predict t+1, standard next-token prediction)

      logits_1 = lm_head(hidden)                      # Predict next token

      loss_1 = cross_entropy(logits_1, tokens[:, 1:])

      # Second prediction head (predict t+2)

      hidden_2 = MTP_Module(hidden, embedding(tokens[:, :-1]))  # MTP module

      logits_2 = mtp_head(hidden_2)                   # Predict next token

      loss_2 = cross_entropy(logits_2, tokens[:, 2:])

      total_loss = loss_1 + lambda * loss_2           # Weighted combination

MTP Module Design

The MTP module is a lightweight Transformer layer that takes the main model's hidden states and the current token's embedding as input, and outputs predictions for future tokens:

Input Fusion: Concatenate the main model's hidden states with the current token's embedding, and fuse through a linear layer.
Transformer Block: A standard Transformer layer (including attention and FFN) to further process the fused representation.
Output Head: A shared LM Head maps the processed representation to the vocabulary distribution.

Training Efficiency Improvement

MTP provides stronger learning signals during training. The model not only needs to predict the next token but also subsequent tokens, forcing the model to learn longer-term dependencies.

Training Benefits

Stronger Training Signal

Multiple prediction targets provide richer gradient signals, helping the model converge faster. Each training step obtains twice the information of traditional methods.

Convergence speed improved by ~30%
Better performance with equal data

Training Benefits

Long-term Planning Ability

MTP forces the model to "think ahead", improving modeling of long-range dependencies. This is also one of the important reasons for R1's enhanced reasoning ability.

Improved long-range dependency modeling
Enhanced reasoning coherence

Inference Phase: MTP Module Can Be Discarded

An important design of MTP is: the MTP module can be completely discarded during inference. During inference, only the standard next-token prediction of the main model is used; the MTP module does not participate in inference computation, so it does not affect inference speed.

The inference acceleration brought by MTP is mainly reflected in: due to learning better representations during the training phase, the main model (without the MTP module) has higher inference quality and requires fewer inference steps for the same task.

Speculative Decoding during inference: Although the MTP module is not used in standard inference, it can serve as a "draft model" for speculative decoding, generating multiple candidate tokens simultaneously, which are then verified by the main model, thereby accelerating inference. In this mode, inference speed can be improved by 1.5-2 times.

FP8 Mixed Precision Training

FP8 Mixed Precision Training Technology

DeepSeek has for the first time verified the feasibility of FP8 training on ultra-large-scale models, reducing training cost to $5.57 million.

FP8 vs BF16 vs FP32

Comparison of commonly used data precisions in deep learning training:

Precision Format	Total Bits	Exponent Bits	Mantissa Bits	Dynamic Range	Memory Usage
FP32	32	8	23	3.4 x 10^38	4 bytes / parameter
BF16	16	8	7	3.4 x 10^38	2 bytes / parameter
FP8 E4M3	8	4	3	448	1 byte / parameter

FP8 uses only half the memory of BF16 and a quarter of FP32, but has a smaller dynamic range (E4M3 format), requiring extremely high training stability.

Block-wise Quantization

DeepSeek adopts a block-wise quantization strategy to address the insufficient dynamic range of FP8:

Blocking strategy: Activations and weights are grouped into blocks of 128x128, and each block independently computes a scale factor.
Online scaling: The scale factor is dynamically calculated based on the maximum absolute value of the current block, ensuring quantization accuracy.

High-Precision Accumulation: Accumulation in matrix multiplication is performed in FP32 precision to avoid accumulation of quantization errors.

      # Block-wise Quantization Pseudocode

      # Forward pass: FP8 matrix multiplication

      for each block (128x128) in activation_matrix:

          scale = max(abs(block)) / 448          # E4M3 max is 448

          block_fp8 = round(block / scale)       # Quantize to FP8

          output_block = block_fp8 @ weight_fp8  # FP8 matrix multiplication

          output_block = output_block * scale    # Dequantize

      # Backward pass: gradients computed in BF16 precision

      gradient = backward(output, loss)          # BF16 gradients

Training Stability Guarantees

Using FP8 training at 671B scale is extremely challenging. DeepSeek has adopted multiple stability measures:

Stability Measures

Mixed Precision Strategy

Forward pass uses FP8 computation (GEMM operations), while gradient computation and optimizer states use BF16/FP32. Critical paths (such as softmax, layernorm) maintain high precision.

Forward GEMM: FP8
Gradients/Optimizer: BF16/FP32
Critical operations: High precision

Stability Measures

Dynamic Gradient Scaling

Dynamically adjusts gradient scaling factor during training to prevent underflow and overflow in FP8.

Adaptive scaling factor
Overflow detection and fallback

Cost Benefits

Total Cost of $5.57 Million

FP8 training halves memory requirements and increases training speed by about 40%. DeepSeek V3 was trained on 2048 H800 GPUs for about 2 months, with a total cost of only $5.57 million.

Memory: Reduced by 50%
Training speed: Increased by 40%
Total cost: $5.57 million

DeepSeekMoE Optimization

DeepSeekMoE Architecture Optimization Details

Fine-grained expert segmentation, shared expert isolation, dynamic routing – the three core optimizations of DeepSeekMoE.

Fine-grained Expert Segmentation

Traditional MoE models typically use a small number of large experts (e.g., 8 experts, each being a complete FFN). DeepSeekMoE further splits each large expert into multiple smaller experts, achieving finer specialization:

Segmentation Strategy: Split 1 standard FFN expert (e.g., 8x intermediate dimension) into 2 fine-grained experts (each 4x intermediate dimension), with each expert focusing on finer-grained knowledge patterns.
Routing Flexibility: More experts mean more flexible combinations. Different tokens can select different expert subsets, enabling more precise knowledge activation.
Load Balancing Advantage: Fine-grained segmentation makes expert load more even, reducing the parameter size of each expert and lowering the risk of individual expert overload.

Shared Expert Isolation

DeepSeekMoE introduces the concept of "shared experts," separated from routed experts:

Shared Experts

Always-Active General Knowledge

8 shared experts are always active for all tokens, capturing cross-domain general knowledge (such as syntax, common sense, basic reasoning). They do not compete in routing, ensuring baseline capabilities.

8 shared experts
Always active, not routed
Capture general knowledge

Routed Experts

On-Demand Specialized Knowledge

256 routed experts are selectively activated via Top-8 gating, with each expert specializing in specific domains (such as mathematics, programming, medicine, law, etc.).

256 routed experts
Top-8 selective activation
Domain-specialized knowledge

Dynamic Routing and Expert Load Statistics

DeepSeekMoE's routing system dynamically adjusts during training to ensure expert load balancing:

      # Dynamic bias adjustment

      # Update expert bias after each training step

      for expert in range(num_experts):

          load = expert_token_count[expert] / total_tokens

          if load > target_load * 1.2:

              expert_bias[expert] -= learning_rate    # Reduce bias for popular experts

          elif load < target_load * 0.8:

              expert_bias[expert] += learning_rate    # Increase bias for unpopular experts

      # Add bias during routing

      gate_logits = Linear(hidden_states) + expert_bias

      topk_weights, topk_indices = topk_softmax(gate_logits, k=8)

This dynamic bias mechanism automatically balances expert load during training without requiring additional auxiliary loss functions, avoiding potential negative impact of auxiliary loss on model performance.

R1 Reasoning Architecture

DeepSeek R1 Reasoning Architecture Explained

A reasoning-enhanced model based on reinforcement learning. GRPO algorithm, chain-of-thought emergence, knowledge distillation—a comprehensive overview of R1's core technologies.

RL Training Pipeline

The training of DeepSeek R1 is divided into multiple stages, ultimately forming powerful reasoning capabilities:

Stage	Method	Purpose
Stage 1: Cold Start SFT	Fine-tune with a small amount of high-quality chain-of-thought data	Give the model basic reasoning format and chain-of-thought patterns
Stage 2: Reasoning RL	Use GRPO reinforcement learning on reasoning tasks	Improve math, programming, and logical reasoning abilities
Stage 3: Rejection Sampling	Sample from the RL model and filter high-quality outputs	Construct a high-quality SFT dataset
Stage 4: Full-domain SFT	Mix reasoning data + general data for fine-tuning	Restore general capabilities while maintaining reasoning ability
Stage 5: Full-domain RL	Use RLHF alignment on all tasks	Improve helpfulness, safety, and harmlessness

GRPO (Group Relative Policy Optimization)

GRPO is a reinforcement learning algorithm proposed by DeepSeek, an improved version of PPO (Proximal Policy Optimization). The core idea is:

Relative advantage within group: Generate multiple candidate responses (a group) for each question, and compute relative advantage using the group's average reward as baseline, instead of using absolute reward values.
No Critic model needed: Traditional PPO requires a Critic (value) model of the same scale as the policy model. GRPO eliminates the need for a Critic model through within-group comparison, saving about 50% of training resources.
KL divergence constraint: Estimate KL divergence directly in the loss function to limit the magnitude of policy updates, preventing the model from deviating too far from the reference policy.

      # GRPO core logic

      # For each question, generate G candidate responses

      group_responses = [model.generate(question) for _ in range(G)]

      group_rewards = [reward_model(question, r) for r in group_responses]

      # Relative advantage within group (subtract mean, divide by std)

      mean_reward = mean(group_rewards)

      std_reward = std(group_rewards)

      advantages = [(r - mean_reward) / std_reward for r in group_rewards]

      # Policy gradient update

      loss = -mean(advantages * log_prob_ratio - beta * kl_divergence)

      model.update(loss)

Reward Modeling

R1's reward model includes two core dimensions:

Reward Dimension

Accuracy Reward

For math problems, check if the final answer is correct; for programming problems, check if the code passes test cases; for reasoning problems, check if the conclusion is logically consistent.

Answer correctness verification
Code test pass rate
Logical consistency check

Reward Dimension

Format Reward

Encourage the model to place the reasoning process in dedicated tags, ensuring standardized and readable output format. This helps the model form structured chain-of-thought.

Standardized reasoning process format
Structured chain-of-thought output

Emergence of Chain-of-Thought

The most amazing feature of R1 is the "emergence of chain-of-thought." In pure RL training (R1-Zero), the model autonomously learned, without human-annotated reasoning steps:

Self-verification: Check whether its intermediate steps are correct during reasoning.
Reflection and backtracking: When a reasoning error is detected, automatically backtrack to previous steps and re-reason.
Multi-path exploration: Try multiple reasoning paths and select the most reasonable result.
"Aha Moment": The model exhibits moments similar to human "insight" during reasoning, re-evaluating and correcting previous reasoning paths.

Knowledge Distillation to Smaller Models

DeepSeek distills R1 671B's reasoning capabilities into smaller models (1.5B to 70B) using 800k carefully selected reasoning samples:

Distillation Strategy

Data Selection

Select 800k high-quality reasoning samples from R1 671B's outputs, covering multiple domains such as mathematics, programming, and scientific reasoning.

800k selected samples
Multi-domain coverage
High-quality filtering

Distillation Strategy

Target Models for Distillation

Distill based on Qwen-2.5 (1.5B/7B/14B/32B) and Llama-3.1/3.3 (8B/70B), enabling smaller models to acquire powerful reasoning abilities.

Qwen-2.5 series
Llama-3.1/3.3 series
Full coverage from 1.5B to 70B

Code Implementation Reference

Core Module Code Implementation

Core module implementations based on DeepSeek-V3 open-source code, using real variable names and structures.

MoE Forward Propagation

# DeepSeekMoE forward propagation (based on DeepSeek-V3 open-source code)
class DeepSeekMoE(nn.Module):
    def __init__(self):
        self.num_experts = 256          # Number of routed experts
        self.num_shared_experts = 8     # Number of shared experts
        self.top_k = 8                  # Top-K routing
        self.n_routed_experts = 256
        self.routed_scaling_factor = 1.0

        # Gating network
        self.gate = nn.Linear(hidden_size, self.n_routed_experts, bias=False)
        # Routed experts (each expert is an FFN)
        self.experts = nn.ModuleList([
            ExpertFFN() for _ in range(self.num_experts)
        ])
        # Shared experts
        self.shared_experts = SharedExpertFFN()

    def forward(self, hidden_states):
        identity = hidden_states
        orig_shape = hidden_states.shape

        # 1. Gating routing
        gate_logits = self.gate(hidden_states)
        # Select Top-K experts
        weights, indices = torch.topk(gate_logits, self.top_k, dim=-1)
        weights = F.softmax(weights, dim=-1, dtype=torch.float32)

        # 2. Routed expert computation
        routed_output = self.moe_forward(hidden_states, weights, indices)

        # 3. Shared expert computation (always activated)
        shared_output = self.shared_experts(identity)

        # 4. Merge outputs
        final_output = routed_output + shared_output
        return final_output

    def moe_forward(self, hidden_states, topk_weights, topk_ids):
        # Group tokens by assigned expert
        outputs = torch.zeros_like(hidden_states)
        for expert_idx in range(self.num_experts):
            # Find tokens assigned to this expert
            expert_mask = (topk_ids == expert_idx).any(dim=-1)
            if not expert_mask.any():
                continue
            expert_input = hidden_states[expert_mask]
            expert_output = self.experts[expert_idx](expert_input)
            # Weighted accumulation
            weight = topk_weights[expert_mask][topk_ids[expert_mask] == expert_idx]
            outputs[expert_mask] += expert_output * weight.unsqueeze(-1)
        return outputs
    

MLA Attention Implementation

# MLA Multi-head Latent Attention (based on DeepSeek-V3 open-source code)
class MultiHeadLatentAttention(nn.Module):
    def __init__(self):
        self.q_lora_rank = 1536          # Query low-rank decomposition dimension
        self.kv_lora_rank = 512          # KV joint compression dimension
self.qk_rope_head_dim = 64       # RoPE part dimension
        self.qk_nope_head_dim = 128      # Non-RoPE part dimension
        self.v_head_dim = 128
        self.num_heads = 128

        # Query low-rank decomposition
        self.q_a_proj = nn.Linear(hidden_size, self.q_lora_rank, bias=False)
        self.q_a_layernorm = nn.RMSNorm(self.q_lora_rank)
        self.q_b_proj = nn.Linear(self.q_lora_rank, self.num_heads *
                                   (self.qk_nope_head_dim + self.qk_rope_head_dim))
        # KV joint compression
        self.kv_a_proj_with_mqa = nn.Linear(
            hidden_size, self.kv_lora_rank + self.qk_rope_head_dim, bias=False)
        self.kv_a_layernorm = nn.RMSNorm(self.kv_lora_rank)
        self.kv_b_proj = nn.Linear(
            self.kv_lora_rank,
            self.num_heads * (self.qk_nope_head_dim + self.v_head_dim))

    def forward(self, hidden_states, freqs_cis, kv_cache=None):
        # 1. Query low-rank decomposition
        q = self.q_a_proj(hidden_states)
        q = self.q_a_layernorm(q)
        q = self.q_b_proj(q)
        q = q.view(-1, self.num_heads, self.qk_nope_head_dim + self.qk_rope_head_dim)
        q_nope, q_rope = q.split([self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
        q_rope = apply_rotary_emb(q_rope, freqs_cis)  # RoPE encoding

        # 2. KV joint compression
        kv_a = self.kv_a_proj_with_mqa(hidden_states)
        c_kv, k_rope = kv_a.split([self.kv_lora_rank, self.qk_rope_head_dim], dim=-1)
        k_rope = apply_rotary_emb(k_rope.unsqueeze(1), freqs_cis)  # RoPE encoding

        # 3. Cache low-dimensional latent vector c_KV (during inference)
        if kv_cache is not None:
            kv_cache.append(c_kv)

        # 4. Up-project to restore Key and Value
        c_kv = self.kv_a_layernorm(c_kv)
        kv = self.kv_b_proj(c_kv)
        kv = kv.view(-1, self.num_heads, self.qk_nope_head_dim + self.v_head_dim)
        k_nope, v = kv.split([self.qk_nope_head_dim, self.v_head_dim], dim=-1)

        # 5. Concatenate Query and Key parts
        q = torch.cat([q_nope, q_rope], dim=-1)
        k = torch.cat([k_nope, k_rope.expand(-1, self.num_heads, -1)], dim=-1)

        # 6. Standard attention computation
        attn_weights = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(
            self.qk_nope_head_dim + self.qk_rope_head_dim)
        attn_weights = F.softmax(attn_weights, dim=-1)
        output = torch.matmul(attn_weights, v)
        return output
    

MTP Module Implementation

# MTP Multi-Token Prediction Module (based on DeepSeek-V3 open-source code)
class MultiTokenPredictionModule(nn.Module):
    def __init__(self):
        self.hidden_size = 7168
        # Input fusion layer
        self.embedding_norm = nn.RMSNorm(hidden_size)
        self.hidden_norm = nn.RMSNorm(hidden_size)
        self.input_proj = nn.Linear(hidden_size * 2, hidden_size, bias=False)
        # MTP Transformer block
        self.mtp_block = TransformerBlock()
        # Output projection (shared LM Head)

        self.output_norm = nn.RMSNorm(hidden_size)
        self.shared_head = nn.Linear(hidden_size, vocab_size, bias=False)

    def forward(self, hidden_states, input_ids):
        # 1. Get the embedding of the current token
        token_emb = self.embedding_norm(embedding(input_ids))

        # 2. Fuse the main model hidden states and token embedding
        hidden_norm = self.hidden_norm(hidden_states)
        combined = torch.cat([hidden_norm, token_emb], dim=-1)
        combined = self.input_proj(combined)

        # 3. Pass through the MTP Transformer block
        mtp_output = self.mtp_block(combined)

        # 4. Output prediction
        mtp_output = self.output_norm(mtp_output)
        logits = self.shared_head(mtp_output)
        return logits

# Used during training
hidden = transformer(input_ids)                    # Main model forward
logits_main = lm_head(hidden)                      # Standard next-token prediction
logits_mtp = mtp_module(hidden, input_ids)         # MTP extra prediction

loss_main = cross_entropy(logits_main[:, :-1], targets[:, 1:])
loss_mtp = cross_entropy(logits_mtp[:, :-2], targets[:, 2:])
total_loss = loss_main + 0.3 * loss_mtp            # Weighted combination
    

Performance Analysis

DeepSeek Model Performance Analysis

FLOPs utilization, memory bandwidth, communication overhead, scaling efficiency—the full performance picture of DeepSeek.

FLOPs Utilization

Metric	DeepSeek V3	Description
Theoretical Peak FLOPs	~990 TFLOPS (H800 FP8)	Theoretical FP8 peak of a single H800 GPU
Actual Achieved FLOPs	~580 TFLOPS	Actual computational throughput during training
FLOPs Utilization	~58.6%	Far above industry average (usually 30-45%)

DeepSeek achieves extremely high FLOPs utilization through carefully designed compute kernels and communication overlap strategies. The 58.6% utilization is top-tier among MoE models (MoE models typically have lower utilization than dense models due to sparse activation and all-to-all communication).

Memory Bandwidth Analysis

Memory Analysis

Model Weights

671B parameters, stored in FP8 format. Total weight memory is about 671 GB (FP8), and about 1342 GB in BF16 format.

FP8: ~671 GB
BF16: ~1342 GB
Using FP8 saves 50%

Memory Analysis

KV Cache (MLA)

MLA compresses the KV Cache to 1/32 of traditional methods. The KV Cache for 128K context is about 40 GB.

Traditional MHA: ~973 GB
MLA: ~40 GB
Savings: ~96%

Memory Analysis

Optimizer State

The AdamW optimizer requires storing momentum and variance (FP32), about 8 bytes per parameter. Optimizer state is about 5.4 TB (FP32).

Momentum: 671B * 4 bytes
Variance: 671B * 4 bytes
Total: ~5.4 TB

Communication Overhead

Distributed training of MoE models involves two communication patterns:

All-Reduce (Data Parallelism): Synchronizes gradients across all GPUs, communication volume scales with the number of GPUs. DeepSeek uses 2048 H800 GPUs and employs an efficient Ring All-Reduce algorithm.

All-to-All (Expert Parallelism): Sends tokens from the current GPU to the GPU where the corresponding expert resides. This is a communication overhead unique to MoE models, and DeepSeek minimizes it through uniform expert distribution and communication overlap strategies.

Communication Optimization Strategy: DeepSeek employs a "compute-communication overlap" technique, performing communication while GPU computation is ongoing, hiding communication overhead within computation. Through meticulous pipeline design, communication overhead is kept within 15% of total training time.

Scaling Efficiency

Number of GPUs	Time per Step	Scaling Efficiency	Notes
256 H800	~8.0 seconds	100% (baseline)	Strong scaling baseline
512 H800	~4.5 seconds	~89%	Increased communication overhead
1024 H800	~2.6 seconds	~77%	Significant All-to-All communication overhead
2048 H800	~1.5 seconds	~67%	Training configuration, scaling efficiency still acceptable

DeepSeek V3 achieves approximately 67% scaling efficiency on 2048 H800 GPUs. For a 671B-parameter MoE model, this is a quite impressive result. Compared to dense models, MoE's All-to-All communication is the main bottleneck for scaling efficiency, and DeepSeek minimizes this impact through communication overlap and expert distribution optimization.

Learn More

DeepSeek Related Resources

Dive deeper into DeepSeek models, from usage to deployment to developer ecosystem.

Model Overview

Complete Guide to DeepSeek Model

Complete introduction to all DeepSeek models: V3, R1, Coder, Janus, VL2, Prover. One-stop guide for download, deployment, and usage.

Complete list of model series
Ollama download and deployment
Benchmark performance comparison

Model Catalog

List of DeepSeek Open-Source Models

Complete list of 6 major series, 20+ models. All specifications, parameter counts, and use cases for V3, R1, Coder, Janus, VL2, Prover.

6 major model series
Quick comparison table
Use case guide

Deployment Tutorial

DeepSeek Model Deployment Tutorial

Four deployment solutions: Ollama, Docker, vLLM, Dify. From single machine to cluster, with complete commands and hardware configuration references.

Ollama one-click deployment
Docker Compose orchestration
vLLM high-performance inference

Usage Tutorial

How to Use DeepSeek Models

Hands-on teaching of four usage methods: official App, Ollama local deployment, API calls, third-party platforms. Even beginners can learn.

Official App usage tutorial
Ollama local running
Python/JS API calls

DeepSeek Architecture

DeepSeek Model Architecture FAQ

What do DeepSeek V3's 671B parameters and 37B activated parameters mean? +

671B is the total number of parameters in the model (the sum of all expert parameters), while 37B is the number of parameters actually activated during inference for each token. The MoE architecture uses a sparse activation mechanism, where each token is routed to only 8 routed experts and 1 shared expert (9 experts in total), so the parameters actually involved in computation are only about 37B (5.5% of the total parameters). This significantly reduces inference cost while maintaining the knowledge capacity of the 671B parameters.

What are the advantages of MLA (Multi-head Latent Attention) over traditional attention? +

The core advantage of MLA lies in KV Cache compression. Traditional MHA needs to cache the full Key and Value matrices, which can occupy nearly 1TB of GPU memory for a 128K context. MLA uses low-rank joint compression of KV, projecting Key and Value into a low-dimensional latent space (e.g., 512 dimensions), reducing the cache to about 1/32 of traditional methods. At the same time, through the RoPE decoupling design, the advantages of rotary position encoding are preserved. This makes long-context inference possible on a single GPU, and inference throughput is increased by 3-5 times.

Does MTP (Multi-Token Prediction) affect inference speed? +

No. The MTP module can be completely discarded during inference by design; only the standard main model's next-token prediction is used during inference. MTP only plays a role during training, providing stronger learning signals to help the model learn better representations and long-range dependencies. In addition, the MTP module can serve as a draft model for speculative decoding, generating multiple candidate tokens simultaneously, which are then verified by the main model. In this mode, inference speed can be improved by 1.5-2 times.

How does DeepSeek solve the expert load imbalance problem in MoE models? +

DeepSeek proposes an innovative auxiliary-loss-free load balancing strategy. Traditional MoE uses an auxiliary loss function to force expert load balancing, but this may affect model performance. DeepSeek's approach is: maintain a dynamic bias term for each expert, which is automatically adjusted during training based on expert load—if the load is too high, the bias is lowered; if too low, the bias is raised. At the same time, combined with fine-grained expert segmentation (256 routed experts) and shared expert separation (8 shared experts always activated), a slight balance constraint is applied at the sequence level, achieving load balancing without auxiliary loss.

What is the difference between DeepSeek R1's reinforcement learning training and traditional RLHF? +

DeepSeek R1 uses the GRPO (Group Relative Policy Optimization) algorithm, an improved version of PPO. The core differences are: 1) GRPO does not require a Critic model (value network); it estimates advantages through relative comparison within a group (generating multiple responses to the same question and using the group average reward as the baseline), saving about 50% of training resources; 2) R1's RL training focuses on reasoning ability, using accuracy rewards (correctness of answers) and format rewards, rather than the human preference rewards of traditional RLHF; 3) R1's training pipeline includes five stages: cold-start SFT, reasoning RL, rejection sampling, full-domain SFT, and full-domain RL, which is a systematic approach to enhancing reasoning ability.

How does FP8 training maintain stability at the 671B scale? +

DeepSeek has verified the feasibility of FP8 training at the 671B scale, relying on several key technologies: 1) Block-wise Quantization, which groups activation values and weights into 128x128 blocks, with each block independently calculating scaling factors to ensure quantization precision; 2) Mixed precision strategy, where forward GEMM uses FP8, while gradients and optimizer states use BF16/FP32, and critical operations (softmax, layernorm) maintain high precision; 3) Dynamic gradient scaling, which adaptively adjusts scaling factors during training to prevent underflow and overflow in FP8. These measures enable stable convergence of FP8 training at ultra-large scale, halving memory usage, increasing training speed by about 40%, and reducing total cost to $5.57 million.

References

References and Further Reading

The technical content on this page is based on DeepSeek's official papers and technical reports, which can be verified independently.

DeepSeek-V3 Technical Report -- Complete technical report on MoE/MLA/MTP/FP8
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning -- R1 reasoning architecture and GRPO algorithm
GitHub -- deepseek-ai/DeepSeek-V3 -- Open-source code and model weights
GitHub -- deepseek-ai/DeepSeek-R1 -- R1 open-source code and distilled models
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models -- DeepSeekMoE architecture paper
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model -- Original paper on MLA attention mechanism

DeepSeek Model Architecture Deep Dive

DeepSeek Model Architecture Panorama

DeepSeek V3 Architecture Overview

V3 vs R1 Architecture Comparison

671B Parameter Overview

Embedding Layer

Transformer Layers

MoE Expert Layers

Activated Parameters

MoE 混合专家架构详解

专家路由机制

Top-K 门控机制

Auxiliary-Loss-Free 负载均衡

为什么 MoE 高效？

Reduced Computation

Knowledge Capacity Maintained

Training Cost Optimization

MLA Multi-head Latent Attention Mechanism

Problems with Traditional MHA

Core Idea of MLA: Low-Rank KV Compression

RoPE Decoupling Design

Memory Savings Calculation

MLA Compression Effect

Actual Effect

Throughput Improvement

MTP Multi-Token Prediction Mechanism

How MTP Works

MTP Module Design

Training Efficiency Improvement

Stronger Training Signal

Long-term Planning Ability

Inference Phase: MTP Module Can Be Discarded

FP8 Mixed Precision Training Technology

FP8 vs BF16 vs FP32

Block-wise Quantization

Training Stability Guarantees

Mixed Precision Strategy

Dynamic Gradient Scaling

Total Cost of $5.57 Million

DeepSeekMoE Architecture Optimization Details

Fine-grained Expert Segmentation

Shared Expert Isolation

Always-Active General Knowledge

On-Demand Specialized Knowledge

Dynamic Routing and Expert Load Statistics

DeepSeek R1 Reasoning Architecture Explained

RL Training Pipeline

GRPO (Group Relative Policy Optimization)

Reward Modeling

Accuracy Reward

Format Reward

Emergence of Chain-of-Thought

Knowledge Distillation to Smaller Models

Data Selection

Target Models for Distillation

Core Module Code Implementation

MoE Forward Propagation

MLA Attention Implementation

MTP Module Implementation

DeepSeek Model Performance Analysis

FLOPs Utilization

Memory Bandwidth Analysis

Model Weights

KV Cache (MLA)

Optimizer State

Communication Overhead

Scaling Efficiency

DeepSeek Related Resources

Complete Guide to DeepSeek Model

List of DeepSeek Open-Source Models

DeepSeek Model Deployment Tutorial

How to Use DeepSeek Models

DeepSeek Model Architecture FAQ

References and Further Reading

每日精选 Skill 推荐，免费送到你邮箱