DeepSeek Inference Acceleration In-Depth Tutorial

Comprehensively master DeepSeek model inference acceleration techniques. In-depth comparison of four major engines: vLLM, SGLang, TensorRT-LLM, llama.cpp, from deployment to tuning, from single GPU to distributed, a complete inference acceleration solution.

Start Learning

DeepSeek Inference Acceleration

Why is inference acceleration needed?

DeepSeek-V3 has 671B parameters, and DeepSeek-R1 is driven by reinforcement learning for reasoning capabilities. With such a large model, using native PyTorch for inference is extremely inefficient. Dedicated inference engines can increase inference throughput by 10-50 times and reduce latency by over 80% through techniques such as KV Cache optimization, Continuous Batching, quantization compression, and operator fusion.

Core Concepts

Inference Engine Overview

The inference engine is the middle layer that connects trained model weights with production inference requests. It is responsible for memory management, batch scheduling, and computation optimization, and is the core component that determines inference performance and cost.

Key Metrics for Inference Performance

Throughput	Total number of tokens processed per unit time (tokens/s). Measures the overall processing capability of the system; higher is better.
Time to First Token (TTFT)	Time from sending the request to receiving the first token. A key metric affecting user experience.
Time Per Output Token (TPOT)	Average time to generate each token. Affects the smoothness of streaming output.
VRAM Usage	GPU memory (GB) occupied during model loading and inference. Directly affects the deployable model size and batch size.

Mainstream Inference Engine Ecosystem

Engine	Developer	Core Advantages	Use Cases
vLLM	UC Berkeley	PagedAttention memory management, extremely high throughput	Production API services
SGLang	Stanford/LMSYS	RadixAttention prefix caching, structured generation	Multi-turn dialogue, Agent scenarios
TensorRT-LLM	NVIDIA	Extreme GPU optimization, FP8/INT4 quantization	Extreme performance on NVIDIA GPUs
llama.cpp	ggerganov	CPU/GPU hybrid inference, GGUF quantization	Personal computers, edge devices

Core Technology Stack for Inference Acceleration

KV Cache Management: Cache the computed Key-Value matrices to avoid redundant computation. PagedAttention manages KV Cache in pages, improving memory utilization by 2-4x.
Continuous Batching: Dynamic batching; requests are replaced immediately upon completion, keeping GPU utilization consistently high.
Quantization: Compress FP16 weights to INT8/INT4/FP8, reducing memory usage by 50%-75% with minimal accuracy loss.
Kernel Fusion: Merge multiple CUDA kernels into one, reducing memory reads/writes and improving computational efficiency.
Tensor Parallelism: Split layer weights across multiple GPUs to break through single-GPU memory limits.
Speculative Decoding: Use a small model to quickly generate candidate tokens, and a large model to verify, speeding up by 2-3x.

Selection Recommendations

For production API services, vLLM is the first choice; for multi-turn dialogue and Agent scenarios, SGLang is recommended; for extreme GPU performance, choose TensorRT-LLM; for personal computers and edge devices, use llama.cpp. For more model details, see DeepSeek Model Architecture Details.

Step 1

Deploy DeepSeek with vLLM

vLLM is currently the most popular open-source inference engine, developed by UC Berkeley. Its core innovation, PagedAttention, manages KV Cache like operating system paging, increasing GPU memory utilization by 2-4x and throughput by 10-30x.

PagedAttention Principle

Traditional inference engines pre-allocate contiguous GPU memory for each request to store KV Cache, leading to severe memory fragmentation and waste (utilization only 20%-40%). PagedAttention divides KV Cache into fixed-size blocks, allocates on demand, and dynamically maps them, achieving memory utilization over 96%.

KV Cache Blocking: Each block stores Key and Value vectors for a fixed number of tokens
On-demand Allocation: Blocks are allocated only when requests arrive, no pre-allocation of contiguous space
Memory Sharing: In parallel sampling and Beam Search scenarios, KV Cache for the same prompt can be shared
Copy-on-Write: Blocks are copied only when written, maximizing sharing

vLLM Installation and Configuration

# Install vLLM (pip recommended)
pip install vllm

# Install from source (for latest features)
git clone https://github.com/vllm-project/vllm.git
cd vllm
pip install -e .

# Verify installation
python -c "import vllm; print(vllm.__version__)"

DeepSeek-V3/R1 Deployment Commands

DeepSeek-V3 and R1 are 671B-parameter MoE models that require multi-GPU deployment. Below are complete commands for deploying with vLLM:

# DeepSeek-V3/R1 671B multi-GPU deployment (8x A100/H100 80GB)
vllm serve deepseek-ai/DeepSeek-V3 \
  --tensor-parallel-size 8 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 256 \
  --enable-prefix-caching \
  --trust-remote-code

# DeepSeek-R1 distilled version deployment (single A100 80GB)
vllm serve deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
  --tensor-parallel-size 1 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90 \
  --dtype bfloat16 \
  --trust-remote-code

# DeepSeek-R1-Distill-Qwen-32B (single RTX 4090 24GB)
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.85 \
  --dtype float16 \
  --trust-remote-code

vLLM Performance Tuning Parameters

Parameter	Description	Recommended Value
--max-num-seqs	Maximum number of concurrent requests	128-256
--max-num-batched-tokens	Maximum number of tokens per batch	8192-16384
--gpu-memory-utilization	GPU memory utilization upper limit	0.90-0.95
--enable-prefix-caching	Enable prefix caching (shared prompts automatically reuse KV Cache)	Enabled
--enable-chunked-prefill	Chunked prefill, long prompts do not block other requests	Enabled

Python API Call Example

from openai import OpenAI

# vLLM provides an OpenAI-compatible API by default
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3",
    messages=[
        {"role": "system", "content": "You are a professional programming assistant."},
        {"role": "user", "content": "Implement quicksort algorithm in Python."},
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True,
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Tip

vLLM's OpenAI-compatible API can directly replace any application using the OpenAI SDK without modifying code logic. In production, it is recommended to use Nginx for load balancing and rate limiting.

Step 2

Deploy DeepSeek with SGLang

SGLang, jointly developed by Stanford University and the LMSYS organization, is a next-generation inference engine for LLM services. Its core innovations, RadixAttention and structured generation language, significantly outperform vLLM in complex prompt scenarios.

How RadixAttention Works

RadixAttention is an automatic KV Cache reuse technology based on a prefix tree (Radix Tree). Traditional KV Cache can only be reused per request, while RadixAttention automatically identifies common prefixes between different requests and reuses the cache, greatly improving performance in scenarios such as few-shot learning, multi-turn dialogue, and Agent.

Prefix tree storage: KV Caches of all requests are organized into a tree structure by prefix
Automatic matching: When a new request arrives, it automatically matches the longest common prefix and reuses the KV Cache
LRU eviction: Uses LRU strategy to manage cache, maximizing cache hit rate under memory constraints
No configuration: RadixAttention works fully automatically, no manual prefix setting required

SGLang Installation and Configuration

# Install SGLang (pip recommended)
pip install sglang[all]

# Or install from source
git clone https://github.com/sgl-project/sglang.git
cd sglang
pip install -e "python[all]"

# Verify installation
python -c "import sglang; print(sglang.__version__)"

DeepSeek Deployment Commands

# DeepSeek-V3 multi-GPU deployment (8x H100 80GB)
python -m sglang.launch_server \
  --model deepseek-ai/DeepSeek-V3 \
  --tp 8 \
  --context-length 8192 \
  --mem-fraction-static 0.85 \
  --enable-radix-cache

# DeepSeek-R1 distilled version (2x A100 80GB)
python -m sglang.launch_server \
  --model deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
  --tp 2 \
  --context-length 4096 \
  --mem-fraction-static 0.90

# DeepSeek-Coder-V2 code reasoning (single H100)
python -m sglang.launch_server \
  --model deepseek-ai/DeepSeek-Coder-V2-Instruct \
  --tp 1 \
  --context-length 16384 \
  --dtype bfloat16 \
  --enable-radix-cache

SGLang Frontend Programming Language

SGLang provides a DSL (Domain-Specific Language) that allows declaratively writing complex LLM interaction logic in Python, automatically implementing optimizations such as prefix caching and parallel calls:

import sglang as sgl

@sgl.function
def multi_turn_chat(s, system_prompt, user_question):
    # System prompt is automatically cached and reused in multi-turn dialogue
    s += sgl.system(system_prompt)
    s += sgl.user(user_question)
    s += sgl.assistant(sgl.gen("answer", max_tokens=1024))

# Parallel call of multiple branches
@sgl.function
def parallel_eval(s, question):
    s += sgl.system("Evaluate the following plan.")
    s += sgl.user(question)


# Parallel generation of evaluations in three dimensions
    s += sgl.fork(3)
    s += sgl.gen("score", max_tokens=10, regex=r"\d+\.\d+")
    s += sgl.gen("reason", max_tokens=200)
    s += sgl.gen("suggestion", max_tokens=200)
    s += sgl.join()

# Set runtime backend
runtime = sgl.Runtime(model_path="deepseek-ai/DeepSeek-V3")
sgl.set_default_backend(runtime)

# Execute
state = multi_turn_chat.run(
    system_prompt="You are a professional programming assistant.",
    user_question="Explain the principle of PagedAttention.",
)
print(state["answer"])

Prefix Cache Optimization Effect

Scenario	TTFT without cache	TTFT with cache	Speedup
Multi-turn dialogue (5 turns)	450ms	120ms	3.75x
Few-shot prompting (10-shot)	820ms	150ms	5.46x
Agent tool calling	380ms	90ms	4.22x

Step 3

Deploying DeepSeek with TensorRT-LLM

TensorRT-LLM is NVIDIA's official inference engine, deeply integrated with the CUDA and TensorRT ecosystem. Through graph optimization, kernel auto-tuning, and quantization techniques, it achieves the highest inference performance on NVIDIA GPUs.

Key Features of TensorRT-LLM

Graph Optimization: Automatically performs layer fusion, constant folding, dead code elimination, and other computational graph optimizations
Kernel Auto-Tuning: Automatically selects the optimal CUDA kernel for specific GPU architectures (H100/A100/L40S)
Native FP8 Support: H100's FP8 Tensor Core acceleration, doubling throughput
INT4/INT8 Quantization: Supports Weight-Only and SmoothQuant quantization schemes
In-flight Batching: NVIDIA's implementation of Continuous Batching, resulting in lower latency

Installing TensorRT-LLM

# Start from Docker (recommended to avoid environment issues)
docker pull nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3

# Run the container
docker run --gpus all -it --rm \
  -v /path/to/models:/models \
  nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3

# Or pip install inside the container
pip install tensorrt_llm -U --extra-index-url https://pypi.nvidia.com

Model Compilation and Conversion

TensorRT-LLM requires converting the Hugging Face model to TensorRT Engine format first:

# Step 1: Convert to TensorRT-LLM checkpoint format
python convert_checkpoint.py \
  --model_dir deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
  --output_dir ./trt_checkpoints \
  --dtype bfloat16 \
  --tp_size 2

# Step 2: Build TensorRT Engine
trtllm-build \
  --checkpoint_dir ./trt_checkpoints \
  --output_dir ./trt_engines \
  --gemm_plugin bfloat16 \
  --max_batch_size 64 \
  --max_input_len 4096 \
  --max_output_len 2048 \
  --max_num_tokens 8192 \
  --use_fp8_context_fmha enable

# Step 3: Run inference service
python run.py \
  --engine_dir ./trt_engines \
  --tokenizer_dir deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
  --max_output_len 2048 \
  --enable_triton_backend

DeepSeek FP8 Quantized Inference

H100 GPUs support native FP8 computation, and with TensorRT-LLM, near-lossless quantized inference can be achieved:

# FP8 quantization conversion (requires calibration data)
python quantize.py \
  --model_dir deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
  --dtype bfloat16 \
  --qformat fp8 \
  --kv_cache_dtype fp8 \
  --output_dir ./trt_checkpoints_fp8 \
  --calib_size 512 \
  --tp_size 2

# Build FP8 Engine
trtllm-build \
  --checkpoint_dir ./trt_checkpoints_fp8 \
  --output_dir ./trt_engines_fp8 \
  --gemm_plugin fp8 \
  --max_batch_size 128 \
  --max_input_len 4096 \
  --max_output_len 2048 \
  --use_fp8_context_fmha enable

# FP8 inference: memory halved, throughput doubled
python run.py \
  --engine_dir ./trt_engines_fp8 \
  --tokenizer_dir deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
  --max_output_len 2048

INT4 Weight-Only Quantization

For memory-constrained scenarios, INT4 quantization can compress the model size by 75%:

# INT4 Weight-Only Quantization
python quantize.py \
  --model_dir deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
  --dtype float16 \
  --qformat int4_awq \
  --group_size 128 \
  --output_dir ./trt_checkpoints_int4 \
  --calib_size 128

# Build INT4 Engine
trtllm-build \
  --checkpoint_dir ./trt_checkpoints_int4 \
  --output_dir ./trt_engines_int4 \
  --gemm_plugin int4 \
  --max_batch_size 64 \
  --max_input_len 4096 \
  --max_output_len 2048

# INT4 inference: 32B model requires only ~16GB VRAM, can run on RTX 4090

TensorRT-LLM Use Cases

TensorRT-LLM is best suited for production scenarios that require pure NVIDIA GPU environments and ultimate inference performance. The downside is that the deployment process is complex, requiring model compilation steps, and it is less flexible than vLLM and SGLang. If you use Triton Inference Server as your inference platform, TensorRT-LLM is the preferred backend.

Step 4

llama.cpp Local Inference

llama.cpp is an inference engine implemented in pure C/C++, optimized for CPU and edge devices. Through the GGUF quantization format, DeepSeek distilled models can run on ordinary laptops without high-end GPUs.

GGUF Quantization Format

GGUF (GGML Unified Format) is the model file format used by llama.cpp, supporting multiple quantization levels:

Quantization Type	Bit Width	File Size Ratio	Quality	Recommended Use Case
Q4_K_M	~4.5 bit	~25%	Excellent	Best value, recommended first choice
Q5_K_M	~5.5 bit	~30%	Excellent	For higher quality
Q8_0	8 bit	~50%	Almost lossless	Use when VRAM is sufficient
IQ3_M	~3.5 bit	~20%	Good	Extremely limited VRAM

Installing llama.cpp

# macOS (using Homebrew)
brew install llama.cpp

# Linux / Windows (compile from source)
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j

# Enable CUDA acceleration
make -j GGML_CUDA=1

# Enable Apple Metal acceleration (macOS)
make -j GGML_METAL=1

# Install Python bindings
pip install llama-cpp-python

# Python bindings with CUDA
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

DeepSeek Model GGUF Download and Deployment

# Download DeepSeek distilled model in GGUF format (using Qwen-32B as an example)
# Download Q4_K_M quantized version from Hugging Face
wget https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-32B-GGUF/resolve/main/DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf

# Command-line inference
./llama-cli \
  -m DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf \
  -p "Please explain the principles of the MoE (Mixture of Experts) architecture." \
  -n 512 \
  -t 8 \
  --temp 0.7 \
  --top-p 0.9

# Start OpenAI-compatible API server
./llama-server \
  -m DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -ngl 99 \
  -c 4096 \
  -t 8

Python Bindings Call

from llama_cpp import Llama

# Load model
llm = Llama(
    model_path="./DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf",
    n_ctx=4096,          # Context length
    n_threads=8,         # CPU thread count
    n_gpu_layers=99,     # GPU acceleration layers (-1 = all)
    verbose=False,
)

# Inference
response = llm.create_chat_completion(
    messages=[
        {"role": "user", "content": "Write a binary search algorithm in Python."},
    ],
    temperature=0.7,
    max_tokens=512,
    stream=True,
)

for chunk in response:
    if "choices" in chunk:
        delta = chunk["choices"][0].get("delta", {})
        print(delta.get("content", ""), end="")

Apple Silicon (Metal) Optimization

llama.cpp has special Metal optimizations for Apple Silicon chips (M1/M2/M3/M4), leveraging the unified memory architecture for GPU acceleration:

# Enable Metal when compiling on macOS
make -j GGML_METAL=1

# Specify GPU layers at runtime
./llama-cli \
  -m DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf \
  -p "Please explain the Self-Attention mechanism of Transformer." \
  -ngl 99 \
  -c 4096 \
  -n 512

# M3 Max (36GB) can run Qwen-32B Q4_K_M
# M2 Ultra (64GB) can run Qwen-72B Q4_K_M

Personal Computer Deployment Guide

Hardware Configuration	Recommended Model	Quantization	Expected Speed
16GB RAM + RTX 4060	DeepSeek-R1-Distill-Qwen-7B	Q4_K_M	30-50 tok/s
32GB RAM + RTX 4090	DeepSeek-R1-Distill-Qwen-32B	Q4_K_M	20-35 tok/s
M3 Max 36GB	DeepSeek-R1-Distill-Qwen-32B	Q4_K_M	15-25 tok/s
M2 Ultra 64GB	DeepSeek-R1-Distill-Llama-70B	Q4_K_M	8-15 tok/s

Core Concepts

Inference Engine Performance Comparison

The four inference engines show significant performance differences across various scenarios. The following benchmark data is based on the DeepSeek-R1-Distill-Llama-70B model, tested on 2x A100 80GB with a batch size of 64.

Throughput Comparison (tokens/s)

Engine	Input 128 tok	Input 512 tok	Input 2048 tok	Output 256 tok
vLLM (FP16)	3,420	2,890	2,150	2,680
SGLang (FP16)	3,510	3,120	2,380	2,920
TensorRT-LLM (FP8)	5,820	4,950	3,680	4,580
llama.cpp (Q4)	580	420	280	380

Latency Comparison (TTFT, ms)

Engine	Idle TTFT	16 Concurrent TTFT	64 Concurrent TTFT	TPOT
vLLM (FP16)	85ms	210ms	580ms	32ms
SGLang (FP16)	72ms	185ms	510ms	28ms
TensorRT-LLM (FP8)	80ms	195ms	530ms	30ms
llama.cpp (Q4)	1,250ms	2,800ms	8,500ms	180ms

GPU Memory Usage Comparison (GB)

Engine	Model Weights	KV Cache	Other Overhead	Total
vLLM (FP16)	140GB	12GB	4GB	156GB
SGLang (FP16)	140GB	10GB	4GB	154GB
TensorRT-LLM (FP8)	70GB	6GB	3GB	79GB
llama.cpp (Q4)	38GB	4GB	2GB	44GB

Benchmark Conclusions

TensorRT-LLM FP8 achieves the highest throughput and lowest memory usage on NVIDIA GPUs.

SGLang offers the lowest latency in long-context and multi-turn conversation scenarios (thanks to RadixAttention).

vLLM has the most mature ecosystem, the most active community, and is the easiest to deploy.

llama.cpp performs far worse than the previous three in GPU environments, but offers the best experience on CPU and edge devices.

Core Concepts

Deep Dive into Quantized Inference

Quantization is one of the most important techniques for inference acceleration. By reducing the numerical precision of model weights, memory usage can be reduced by 50%-75%, while accuracy loss is typically kept within 1%.

Comparison of Quantization Methods

Quantization Method	Bit Width	Compression Ratio	Accuracy Loss	Supported Engines
AWQ	INT4	4x	< 0.5%	vLLM, TensorRT-LLM, SGLang
GPTQ	INT4	4x	< 1%	vLLM, TensorRT-LLM
GGUF	2-8 bit	2-8x	< 1% (Q4+)	llama.cpp
FP8	8 bit	2x	Almost lossless	TensorRT-LLM (H100)
SmoothQuant	INT8	2x	< 0.5%	TensorRT-LLM

AWQ Quantization Principle and Usage

AWQ (Activation-aware Weight Quantization) identifies and protects important weight channels by analyzing the distribution of activations, achieving high-precision INT4 quantization without calibration:

# Install AutoAWQ
pip install autoawq

# AWQ quantization (example with DeepSeek-R1-Distill-Qwen-32B)
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"
quant_path = "DeepSeek-R1-Distill-Qwen-32B-AWQ"
# Load model
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

# Quantization config
quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM",
}

# Execute quantization
model.quantize(tokenizer, quant_config=quant_config)

# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

# Use AWQ quantized model in vLLM
# vllm serve ./DeepSeek-R1-Distill-Qwen-32B-AWQ --quantization awq

Memory Savings Calculation

Model	FP16 Memory	FP8 Memory	INT4 Memory	Savings
DeepSeek-V3 (671B)	1,342 GB	671 GB	336 GB	75%
R1-Distill-Llama-70B	140 GB	70 GB	35 GB	75%
R1-Distill-Qwen-32B	64 GB	32 GB	16 GB	75%
R1-Distill-Qwen-7B	14 GB	7 GB	3.5 GB	75%

Quantization Accuracy Loss Analysis

FP8: Accuracy loss is almost zero (< 0.1%), because FP8 uses floating-point representation with a large dynamic range. However, it requires hardware supporting FP8 such as H100/H200.
INT8 (SmoothQuant): Accuracy loss is < 0.5%, achieved by smoothing outliers in activations. It has good generality and works on both A100 and H100.

INT4 (AWQ): Accuracy loss < 0.5%, protects important channels by analyzing activation value distribution. Currently the most recommended INT4 solution

INT4 (GPTQ): Accuracy loss < 1%, requires calibration data, but the quantization process is more stable

Q4_K_M (GGUF): Accuracy loss < 1%, widely used in the llama.cpp ecosystem, best cost-performance

Quantization Selection Recommendations

If you have H100, prefer FP8 (highest throughput); for A100 environments, AWQ INT4 is recommended (significant VRAM savings and high accuracy); for personal computers, use GGUF Q4_K_M (small file size, good compatibility). Reducing VRAM by 75% means you can deploy the same model with far fewer GPUs, or serve more concurrent users on the same hardware.

Core Concepts

Batching and Concurrency Optimization

Batching is the core determinant of inference engine throughput. From static batching to Continuous Batching, and then to Chunked Prefill, each evolution in batching technology has brought significant performance improvements.

Comparison of Three Batching Methods

Batching Method	Principle	GPU Utilization	Applicable Engines
Static Batching	Wait for a fixed number of requests then process together, release only after the slowest request completes	Low (30-50%)	HuggingFace TGI (old version)
Dynamic Batching	Requests join the batch as they arrive, but release only after all requests complete	Medium (50-70%)	ONNX Runtime
Continuous Batching	After each token generation, check immediately; completed requests are removed immediately, new requests are added immediately	High (80-95%)	vLLM, SGLang, TensorRT-LLM

Continuous Batching Workflow

Iteration-level scheduling: Re-evaluate the batch queue after each token generation, rather than waiting for the entire sequence to complete
Immediate replacement: When a request completes, it is immediately removed from the batch, and GPU resources are immediately allocated to waiting new requests
Preemptive scheduling: Supports priority queues; high-priority requests can preempt compute resources from low-priority requests
Fair scheduling: Prevents long-sequence requests from starving short-sequence requests, ensuring fairness through round-robin or weighted scheduling

Chunked Prefill Optimization

When users send long prompts (e.g., 100K token document analysis), the Prefill phase consumes significant compute resources, causing other requests to wait. Chunked Prefill splits the Prefill of long prompts into multiple small chunks, interleaving them with Decode requests:

# Enable Chunked Prefill in vLLM
vllm serve deepseek-ai/DeepSeek-V3 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192 \
  --tensor-parallel-size 8

# Enable Chunked Prefill in SGLang
python -m sglang.launch_server \
  --model deepseek-ai/DeepSeek-V3 \
  --chunked-prefill-size 4096 \
  --tp 8

Concurrent User Optimization Strategies

Strategy	Description	Effect
max-num-seqs	Set the maximum number of concurrent sequences, adjust according to VRAM capacity	128-256 is optimal
Queueing Policy	FIFO (First In, First Out) vs Priority vs Shortest-Job-First	SJF has the lowest average latency
Request Rate Limiting	Limit the number of concurrent requests via API gateway to prevent overload	Improved stability
Timeout and Retry	Set reasonable timeout (30-60s) with exponential backoff retry	Improved user experience

Request Scheduling Engine Configuration

# vLLM scheduler configuration example
vllm serve deepseek-ai/DeepSeek-V3 \
  --scheduler-policy priority \        # Priority scheduling
  --max-num-seqs 256 \                 # Maximum concurrent sequences
  --max-num-batched-tokens 16384 \     # Maximum tokens per batch
  --max-paddings 256 \                 # Maximum padding ratio
  --enable-prefix-caching \            # Prefix caching
  --enable-chunked-prefill \           # Chunked prefill
  --max-num-on-the-fly 16              # Number of concurrent prefill requests

Core Concepts

Distributed Inference

DeepSeek-V3 has 671B parameters, requiring 336GB of VRAM even with INT4 quantization, far exceeding the capacity of a single GPU. Distributed inference makes it possible to run such large models by coordinating multiple GPUs and nodes.

Tensor Parallelism

Tensor Parallelism splits the weight matrices of a single Transformer layer across multiple GPUs, either by columns or rows. Each GPU computes a portion, and results are aggregated via AllReduce communication:

Column-wise splitting: Split the weight matrix W into [W1, W2, ..., Wn] by columns. Each GPU holds W_i, computes independently, then aggregates via AllReduce
Row-wise splitting: Split the weight matrix by rows. Each GPU computes partial outputs, then concatenates via AllGather
Communication overhead: Each Transformer layer requires 2 AllReduce operations, demanding high NVLink bandwidth. It is recommended to use GPU groups interconnected with NVLink
Optimal GPU count: Typically 2-8 GPUs. Beyond 8, communication overhead begins to outweigh computational gains

Pipeline Parallelism

Pipeline Parallelism splits the model by layers across multiple GPUs, forming a pipeline. GPU0 processes the first N layers, GPU1 the middle N layers, GPU2 the last N layers:

Layer-wise splitting: Communication only occurs at pipeline stage boundaries, with much lower communication volume than Tensor Parallelism
Micro-Batch: Split large batches into multiple micro-batches. Pipeline stages process different micro-batches simultaneously
Pipeline bubbles: GPU idle time exists during pipeline startup and drain. Increasing the number of micro-batches reduces the bubble ratio
Use cases: Cross-node inference (limited inter-node bandwidth), can be combined with Tensor Parallelism

DeepSeek-V3 671B Distributed Deployment Options

Option	GPU Configuration	Parallelism Strategy	Expected Throughput
8x H100 80GB (FP8)	Single node, 8 GPUs	TP=8	3,500 tok/s
16x A100 80GB (FP8)	2 nodes x 8 GPUs	TP=8, PP=2	4,800 tok/s
16x A100 80GB (INT4)	2 nodes x 8 GPUs	TP=4, PP=4	5,200 tok/s
32x H100 80GB (FP8)	4 nodes x 8 GPUs	TP=8, PP=4	8,500 tok/s

vLLM Multi-Node Deployment Commands

# Main node (Node 0)
vllm serve deepseek-ai/DeepSeek-V3 \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.95 \
  --distributed-executor-backend ray \
  --host 0.0.0.0 \
  --port 8000

# Worker node (Node 1)
# Start Ray worker on Node 1 and join the cluster
ray start --address='NODE0_IP:6379'

# Use Ray to automatically manage multi-node GPU resources
# vLLM uses Ray for multi-node scheduling, no need to manually specify GPU allocation

Special Considerations for MoE Models

DeepSeek-V3 uses the MoE (Mixture of Experts) architecture, where each token only activates a subset of experts (approximately 37B parameters), providing unique opportunities for inference optimization:

Expert Parallelism: Distribute different experts across different GPUs, each token only accesses a subset of GPUs, reducing communication overhead.
Expert Load Balancing: Use Auxiliary Loss or dynamic routing to ensure balanced load across experts, avoiding idle GPUs.
Expert Caching: Cache weights of popular experts in the VRAM of all GPUs to reduce cross-GPU access.
Sparse Activation: Each inference activates only 8 experts (out of 256), resulting in much lower computation than a Dense model with equivalent parameters.

Core Concepts

Cost Optimization Practices

GPU inference cost is one of the largest expenses for AI applications. Through proper GPU selection, instance strategies, and architecture design, inference costs can be reduced by 50%-80%.

Cloud GPU Selection Comparison

GPU	VRAM	FP8 Support	On-Demand Price/Hour	Suitable Models
H100 80GB	80 GB	Supported	$2.50-$3.50	V3 671B (8x), 70B (2x)
A100 80GB	80 GB	Not supported	$1.80-$2.50	V3 671B (16x INT4), 70B (2x)
A100 40GB	40 GB	Not supported	$1.20-$1.80	32B models (single GPU), 70B (2x)
L40S 48GB	48 GB	Supported	$0.80-$1.20	32B models (single GPU FP8)
RTX 4090 24GB	24 GB	Not supported	Self-built: ~$0.30	7B/14B models

On-Demand vs Reserved Instance Comparison

Instance Type	Discount	Commitment Term	Flexibility	Recommended Scenarios
On-Demand	Full price	None	Extremely high	Development testing, uncertain loads
Reserved	40-60%	1-3 years	Low	Stable production environments
Spot/Preemptible	60-90%	None	Can be reclaimed anytime	Offline batch processing, fault-tolerant tasks
Hybrid strategy	50-70%	Flexible	Medium	Reserved baseline + Spot elasticity

Inference Cost Calculation

# Inference cost formula
# Monthly cost = GPU hourly price x GPUs per hour x 24 x 30 x utilization

# Example 1: Deploy DeepSeek-R1-Distill-Llama-70B
# 2x A100 80GB on-demand instances, used 18 hours per day
# Monthly cost = $2.00 x 2 x 24 x 30 x 0.75 = $2,160/month

# Example 2: Deploy DeepSeek-V3 671B (FP8)
# 8x H100 80GB reserved instances, 24/7 operation
# Monthly cost = $3.00 x 0.5 (reserved discount) x 8 x 24 x 30 x 0.90 = $7,776/month

# Example 3: Deploy DeepSeek-R1-Distill-Qwen-32B (INT4)
# 1x A100 40GB spot instance, 24/7 operation
# Monthly cost = $1.50 x 0.2 (spot discount) x 1 x 24 x 30 x 0.85 = $183/month

# Compare DeepSeek official API cost
# DeepSeek API: $0.27/million input tokens + $1.10/million output tokens
# Daily 1 million output tokens: 30 x $1.10 = $33/month
# For low-to-medium loads, using API is much cheaper than self-hosted inference

Self-hosted vs Cloud Services Comparison

Comparison Dimension	Self-hosted Inference	DeepSeek Official API	Cloud Inference Service
Initial Cost	High (GPU purchase/rental)	Zero	Medium
Operational Cost	High (requires dedicated maintenance)	Zero	Medium
Data Privacy	Fully controllable	Data passes through third party	Depends on configuration
High Load Cost	Low marginal cost	Linear growth	Medium
Elastic Scaling	Limited	Unlimited	Good

Cost Optimization Best Practices

Load Tiering: Use 7B/14B distilled models for simple questions, route complex questions to 70B/671B large models, saving 60% cost
Cache Strategy: Cache answers for popular questions (semantic cache), hit rate can reach 30-50%, directly saving inference cost
Quantization for Cost Reduction: FP8 quantization reduces memory by 50%, INT4 by 75%, serving more users on the same hardware
Spot Instances: Use spot instances for offline batch processing and asynchronous tasks, reducing cost by 60-90%
Auto Scaling: Automatically adjust GPU instance count based on QPS, scale down to 0 during off-peak, saving 40-60% cost
Checkpointing: Save KV Cache state when spot instances are reclaimed, resume inference after recovery, avoiding waste

Cost Optimization Recommendations

When daily token volume is below 1 million, using the DeepSeek official API is most economical; when daily tokens are between 1 million and 10 million, self-hosting a 32B/70B distilled model inference service offers the best cost-performance; when daily tokens exceed 10 million, deploy the full 671B model with quantization and spot instances. For more deployment options, see DeepSeek Deployment Tutorial.

Inference Acceleration FAQ

DeepSeek Inference Acceleration FAQ

Which should I choose: vLLM or SGLang? +

For general API service scenarios, choose vLLM (more mature ecosystem, larger community, more comprehensive documentation); for multi-turn dialogue and Agent scenarios, choose SGLang (RadixAttention prefix caching advantage is significant, TTFT reduced by 3-5x). If your business scenario is dominated by long system prompts + multi-turn dialogue, SGLang is the better choice. Both provide OpenAI-compatible APIs, so migration cost is extremely low.

What is the minimum number of GPUs required for DeepSeek-V3 671B? +

FP16 precision requires at least 16 A100 80GB (TP=8, PP=2) or 8 H100 80GB (TP=8). FP8 quantization (H100 only) can run on 8 H100s. INT4 quantization can run on 8 A100 80GB (TP=8). If using AWQ INT4 quantization + Tensor Parallelism, it can run on 4 A100 80GB, but throughput will be significantly reduced.

How much does model quality degrade after quantization? +

FP8 quantization has almost zero accuracy loss (< 0.1%) because it uses floating-point representation with a large dynamic range. INT8 quantization (SmoothQuant) has accuracy loss < 0.5%. INT4 quantization (AWQ) has accuracy loss < 0.5%, which is almost imperceptible in most tasks. Q4_K_M (GGUF) has accuracy loss < 1%. For DeepSeek distilled models, AWQ INT4 is a very safe choice and recommended for production environments.

Can RTX 4090 run DeepSeek? +

RTX 4090 24GB can run DeepSeek-R1 distilled models: 7B (Q4 quantization, only 4GB needed), 14B (Q4 quantization, ~9GB), 32B (Q4 quantization, ~16GB + remaining for KV Cache). Recommended to deploy with llama.cpp or Ollama, using GGUF Q4_K_M quantization format. It cannot run 70B and 671B models. RTX 4090 inference speed is excellent on 7B-32B models, reaching 30-80 tok/s.

How much throughput can Continuous Batching improve? +

Compared to static batching, Continuous Batching can increase GPU utilization from 30-50% to 80-95%, and throughput by 2-5x. In scenarios with large differences in request lengths (e.g., mixing short Q&A and long text generation), the improvement is especially significant. vLLM, SGLang, and TensorRT-LLM all use Continuous Batching by default, requiring no additional configuration.

Should I build my own inference service or use the DeepSeek API? +

When the daily token count is below 1 million, the API monthly fee is about $33, which is far lower than the cost of self-built GPUs (at least $1,500/month). For 1 million to 10 million tokens per day, building your own 32B/70B distilled model (with spot instances) is more cost-effective. For over 10 million tokens per day, the marginal cost of a self-built 671B full model is the lowest. In addition, scenarios with strict data privacy requirements must be self-built. Strategically, it is recommended to first validate your product with the API, then gradually migrate to self-built services once stable.

Learn More

DeepSeek Related Tutorials

Dive deeper into using, deploying, and ecosystem tools for DeepSeek models.

Beginner Guide

How to Use DeepSeek Models

Four usage methods, zero-basics beginner tutorial.

Deployment Tutorial

DeepSeek Deployment Tutorial

Deployment options with Ollama, Docker, vLLM, K8s.

Knowledge Base Tutorial

DeepSeek RAG Knowledge Base

Document loading, vector embeddings, ChromaDB retrieval, Q&A with source citations.

Model Details

DeepSeek Model Architecture

Technical architecture, benchmarks, model comparison.

Model List

DeepSeek Open Source Models

Complete catalog of 6 series and 20+ models.

Download Guide

DeepSeek Model Download

Download guides for Ollama, Hugging Face, GitHub.

DeepSeek Inference Acceleration In-Depth Tutorial

Why is inference acceleration needed?

Inference Engine Overview

Key Metrics for Inference Performance

Mainstream Inference Engine Ecosystem

Core Technology Stack for Inference Acceleration

Deploy DeepSeek with vLLM

PagedAttention Principle

vLLM Installation and Configuration

DeepSeek-V3/R1 Deployment Commands

vLLM Performance Tuning Parameters

Python API Call Example

Deploy DeepSeek with SGLang

How RadixAttention Works

SGLang Installation and Configuration

DeepSeek Deployment Commands

SGLang Frontend Programming Language

Prefix Cache Optimization Effect

Deploying DeepSeek with TensorRT-LLM

Key Features of TensorRT-LLM

Installing TensorRT-LLM

Model Compilation and Conversion

DeepSeek FP8 Quantized Inference

INT4 Weight-Only Quantization

llama.cpp Local Inference

GGUF Quantization Format

Installing llama.cpp

DeepSeek Model GGUF Download and Deployment

Python Bindings Call

Apple Silicon (Metal) Optimization

Personal Computer Deployment Guide

Inference Engine Performance Comparison

Throughput Comparison (tokens/s)

Latency Comparison (TTFT, ms)

GPU Memory Usage Comparison (GB)

Deep Dive into Quantized Inference

Comparison of Quantization Methods

AWQ Quantization Principle and Usage

Memory Savings Calculation

Quantization Accuracy Loss Analysis

Batching and Concurrency Optimization

Comparison of Three Batching Methods

Continuous Batching Workflow

Chunked Prefill Optimization

Concurrent User Optimization Strategies

Request Scheduling Engine Configuration

Distributed Inference

Tensor Parallelism

Pipeline Parallelism

DeepSeek-V3 671B Distributed Deployment Options

vLLM Multi-Node Deployment Commands

Special Considerations for MoE Models

Cost Optimization Practices

Cloud GPU Selection Comparison

On-Demand vs Reserved Instance Comparison

Inference Cost Calculation

Self-hosted vs Cloud Services Comparison

Cost Optimization Best Practices

DeepSeek Inference Acceleration FAQ

DeepSeek Related Tutorials

How to Use DeepSeek Models

DeepSeek Deployment Tutorial

DeepSeek RAG Knowledge Base

DeepSeek Model Architecture

DeepSeek Open Source Models

DeepSeek Model Download

每日精选 Skill 推荐，免费送到你邮箱