DeepSeek Performance Optimization Guide

From single GPU to cluster, comprehensively optimize DeepSeek model inference performance. Inference acceleration, memory optimization, quantization techniques, batching, KV Cache, speculative decoding - all covered.

Start Optimizing

DeepSeek Performance Optimization

Why Performance Optimization?

DeepSeek models are powerful, but inference costs are equally non-negligible. Whether you are a single-GPU user deploying locally or a cloud API user with massive calls, reasonable performance optimization can bring 2-10x inference speedup while significantly reducing hardware and API costs. This guide covers the complete optimization chain from quantization, KV Cache, batching, to distributed inference.

Chapter 1

Performance Metrics System

Before starting optimization, you must first establish a correct performance evaluation system. Without metrics, optimization is like blind men feeling an elephant. Here are the six core metrics you need to focus on.

Six Core Performance Metrics

Metric	English	Description	Optimization Goal
Throughput	TPS / TGS	Tokens generated per second (Tokens Per Second / Tokens Generated per Second)	Higher is better
Time to First Token	TTFT	Time To First Token, the time from request sent to first token generated	Lower is better
Time Per Output Token	TPOT	Time Per Output Token, the average generation time per output token	Lower is better
GPU Utilization	GPU Util	Utilization of GPU compute cores, reflecting whether computing resources are fully used	Higher is better
VRAM Usage	VRAM	GPU memory occupied during inference (GB), determines the maximum model size that can run	Lower is better
Throughput-Latency Ratio	QPS/P99	Maximum throughput while meeting P99 latency requirements, core SLA metric	Higher is better

Throughput vs. Latency: A Trade-off

The core contradiction in performance optimization lies in the trade-off between throughput and latency. Increasing batch size can improve throughput but increases TTFT; reducing precision can speed up inference but may affect answer quality. An excellent optimization solution needs to find the best balance between the two.

Offline batch processing scenarios (e.g., evaluation, data annotation): Prioritize throughput, latency can be tolerated
Online service scenarios (e.g., chatbots, API): Prioritize latency control, TTFT typically needs to be < 500ms
Real-time interactive scenarios (e.g., voice assistants): Extremely latency-sensitive, TTFT needs to be < 200ms

Performance Monitoring Tools

# GPU real-time monitoring
nvidia-smi dmon -s pucvmet -d 1

# vLLM built-in performance metrics (Prometheus format)
# Add --disable-log-requests at startup to reduce log overhead
vllm serve deepseek-ai/DeepSeek-V3 \
  --host 0.0.0.0 --port 8000 \
  --disable-log-requests

# Use benchmark_serving.py for stress testing
python benchmarks/benchmark_serving.py \
  --backend vllm \
  --model deepseek-ai/DeepSeek-V3 \
  --dataset-name sharegpt \
  --num-prompts 1000 \
  --request-rate 10

Chapter 2

Quantization Technology Details

Quantization is a technique that compresses model parameters from high precision (FP16/BF16) to low precision (INT8/INT4). Reasonable quantization can reduce GPU memory usage by 50%-75% with almost no loss in accuracy, and increase inference speed by 2-4 times.

Comparison of Mainstream Quantization Methods

Quantization Method	Precision	Memory Savings	Speed Improvement	Accuracy Loss	Recommended Scenarios
GPTQ	INT4/INT8	~60%	2-3x	Minimal	GPU inference, prioritizing accuracy
AWQ	INT4	~60%	2-3x	Minimal	GPU inference, prioritizing speed
GGUF	Q2-Q8	40%-75%	1.5-4x	Depends on level	CPU/hybrid inference, flexible deployment
FP8	FP8	~50%	1.5-2x	Almost none	H100/H200, native support

Detailed Explanation of GGUF Quantization Levels

GGUF provides multiple quantization levels from Q2 to Q8. The higher the number, the higher the precision and the larger the model. Here is the performance of DeepSeek-R1-8B at different quantization levels:

Quantization Level	Model Size	Inference Speed	MMLU Score	Recommendation
Q2_K	3.2 GB	Very fast	~58.2	Not recommended
Q3_K_M	4.0 GB	Fast	~62.5	Low-end devices
Q4_K_M	5.2 GB	Relatively fast	~65.8	Highly recommended
Q5_K_M	6.4 GB	Moderate	~66.8	Recommended
Q6_K	7.4 GB	Normal	~67.5	High quality needs
Q8_0	8.5 GB	Normal	~67.9	Highest precision

Using GPTQ/AWQ Quantization for DeepSeek

# Load quantized model using AutoGPTQ
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM

model = AutoGPTQForCausalLM.from_quantized(
    "deepseek-ai/DeepSeek-V3-GPTQ-Int4",
    device="cuda:0",
    use_triton=True,  # Use Triton for accelerated inference
)
tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V3-GPTQ-Int4")

# Use vLLM to load AWQ quantized model
# vllm serve deepseek-ai/DeepSeek-V3-AWQ --quantization awq

Decision Tree for Quantization Method Selection

Running locally with Ollama: Choose GGUF Q4_K_M, the best balance between speed and accuracy
Deploying service with vLLM: Prefer AWQ INT4, natively supported by vLLM, best performance
H100/H200 GPUs: Use FP8 quantization, native Transformer Engine acceleration
CPU inference: Use GGUF Q4_K_M or Q5_K_M, with llama.cpp for best performance

Extreme precision needs: Use GPTQ INT8 or no quantization, retain BF16 precision

Chapter 3

KV Cache Optimization

KV Cache is the core mechanism of Transformer inference and the main source of memory usage. Understanding and optimizing KV Cache is a key step to improving inference performance.

How KV Cache Works

During autoregressive generation, each new token needs to compute attention with all historical tokens. KV Cache caches the computed Key and Value matrices to avoid redundant computation. For a model with context length N and hidden dimension d, the memory usage of KV Cache is approximately:

KV Cache size = 2 * num_layers * N * d * num_heads * 2_bytes
    

Taking DeepSeek-V3 (671B parameters, MoE architecture) as an example, at 128K context, KV Cache can occupy tens of GB of memory, far exceeding the model weights themselves.

KV Cache Quantization (FP8 KV Cache)

Quantizing KV Cache from FP16 to FP8 or INT8 can halve the memory usage of KV Cache with almost no loss of accuracy. This is currently the most mature and effective optimization method for KV Cache.

# Enable FP8 KV Cache in vLLM
vllm serve deepseek-ai/DeepSeek-V3 \
  --kv-cache-dtype fp8 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.95

# Enable FP8 KV Cache in SGLang
python -m sglang.launch_server \
  --model deepseek-ai/DeepSeek-V3 \
  --kv-cache-dtype fp8_e5m2 \
  --context-length 131072

Prefix Caching

In multi-turn conversations, the system prompt and historical conversation content are identical across turns. Prefix Caching reuses the computed KV Cache to avoid redundant computation for the same prefix. This is particularly effective in the following scenarios:

Multi-turn conversations: system prompt + historical messages are fully reused, TTFT can be reduced by 50%-80%
Batch evaluation: multiple samples share the same instruction prefix
Few-shot inference: multiple requests share the same few-shot example prefix
RAG scenarios: multiple questions share the same retrieval context prefix

# Enable Automatic Prefix Caching in vLLM (enabled by default)
vllm serve deepseek-ai/DeepSeek-V3 \
  --enable-prefix-caching

# SGLang's RadixAttention automatically enables prefix caching
# No additional configuration needed, enabled by default

Multi-turn Conversation Optimization Practice

# Before optimization: recalculate the entire conversation history for each request
messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "Question 1"},
    {"role": "assistant", "content": "Answer 1"},


{"role": "user", "content": "Question 2"},  # system + history recalculated each time
]

# After optimization: enable Prefix Caching, KV Cache for system prompt and history messages is reused
# TTFT for the second turn drops from 500ms to 50ms

Chapter 4

Batching Optimization

Batching is the most direct way to improve GPU utilization. However, traditional static batching faces serious challenges in LLM inference: the output lengths of different requests vary greatly, causing GPU idle time. Continuous Batching solves this problem.

Static Batching vs Continuous Batching

Comparison Dimension	Static Batching	Continuous Batching
Working Mechanism	Waits for all requests in the batch to complete before processing the next batch	Replaces completed requests with new ones immediately, dynamically adjusting
GPU Utilization	Low, short requests wait for long requests	High, GPU almost never idle
Throughput	Low	High, up to 10x improvement
Implementation Frameworks	HuggingFace Transformers	vLLM, SGLang, TGI

Choosing the Optimal Batch Size

Batch size is not necessarily better when larger. An excessively large batch size increases latency, while too small a batch size fails to fully utilize the GPU. The optimal batch size depends on the following factors:

GPU Memory Size: Batch size is limited by available memory for KV Cache. For 24GB memory (RTX 4090), max_num_seqs=32-64 is recommended.
Request Load: In high-concurrency scenarios, increase batch size appropriately; in low-load scenarios, keep batch size small to reduce latency.
Sequence Length: For long-context scenarios, batch size should be reduced because KV Cache occupies more memory.
Model Size: Large models (e.g., DeepSeek-V3) typically use smaller batch sizes than small models (e.g., DeepSeek-R1-8B).

vLLM Continuous Batching Configuration

# vLLM batching related parameters
vllm serve deepseek-ai/DeepSeek-V3 \
  --max-num-seqs 64 \             # Maximum number of concurrent sequences (upper limit of batch size)
  --max-num-batched-tokens 8192 \ # Maximum number of tokens processed per iteration
  --max-model-len 32768 \          # Maximum context length
  --gpu-memory-utilization 0.95    # GPU memory utilization

Dynamic Batching Strategies

Chunked Prefill: Process the prefill phase of long prompts in chunks, alternating with the decode phase, to avoid prefill blocking decode.
Priority Scheduling: Set priorities for different requests; high-priority requests enter the batch first.
Length-aware Batching: Place requests of similar lengths into the same batch to reduce padding waste.
Iteration-level Scheduling: Re-evaluate batch composition at each iteration and dynamically add new requests.

Chapter 5

Speculative Decoding

Speculative decoding is one of the most attention-grabbing inference acceleration techniques in recent years. It uses a small "draft model" to quickly generate candidate tokens, which are then verified in parallel by a large "target model", achieving 2-3x inference acceleration without sacrificing accuracy.

How Speculative Decoding Works

The bottleneck of large model inference lies in autoregressive generation: only one token can be generated at a time, making parallelization impossible. Speculative decoding transforms serial generation into parallel verification through "small model guessing + large model verification":

Draft phase: The small model (Draft Model) quickly generates K candidate tokens (e.g., K=5)
Verification phase: The large model (Target Model) verifies all K tokens in a single forward pass
Accept/Reject: Accept correct tokens based on probability distribution, reject mismatched tokens
Repeat: Continue the above process from the first rejected position

Speedup Analysis

The speedup of speculative decoding depends on the "acceptance rate" of the draft model. The higher the acceptance rate, the better the acceleration. In typical scenarios:

Target Model	Draft Model	Acceptance Rate	Speedup
DeepSeek-V3 (671B)	DeepSeek-V3-Lite (16B)	~85%	2.5x
DeepSeek-R1 (671B)	DeepSeek-R1-Distill-Llama-8B	~80%	2.2x
DeepSeek-Coder-V2	DeepSeek-Coder-1.3B	~90%	3.0x

DeepSeek Speculative Decoding Implementation

# Using vLLM for speculative decoding
vllm serve deepseek-ai/DeepSeek-V3 \
  --speculative-model deepseek-ai/DeepSeek-V3-Lite \
  --num-speculative-tokens 5 \
  --speculative-draft-tensor-parallel-size 1

# Using HuggingFace for speculative decoding
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

target_model = AutoModelForCausalLM.from_pretrained(
    "deepseek-ai/DeepSeek-V3", torch_dtype=torch.bfloat16
).to("cuda")

draft_model = AutoModelForCausalLM.from_pretrained(
    "deepseek-ai/DeepSeek-V3-Lite", torch_dtype=torch.bfloat16
).to("cuda")

# Use assisted_decoding or prompt_lookup_decoding
output = target_model.generate(
    input_ids,
    assistant_model=draft_model,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.7,
)

Draft Model Selection Recommendations

The draft model should meet three conditions: 1) Same series or architecture as the target model to ensure high acceptance rate; 2) Parameter count between 1/10 and 1/50 of the target model; 3) Inference speed significantly faster than the target model. For DeepSeek-V3, it is recommended to use DeepSeek-V3-Lite or DeepSeek-R1-Distill-Qwen-1.5B as the draft model.

).to("cuda") # Use assisted_decoding or prompt_lookup_decoding output = target_model.generate( input_ids, assistant_model=draft_model, max_new_tokens=256, do_sample=True, temperature=0.7, )

Draft Model Selection Recommendations

Chapter 6

Memory Optimization

Memory is the biggest bottleneck for LLM inference. A 671B-parameter DeepSeek-V3 model requires ~1.3TB of memory even with FP16. With the following techniques, you can run larger models on limited memory.

Gradient Checkpointing

Although primarily used for training, the idea of gradient checkpointing also influences inference. In inference, by not saving intermediate activations and trading computation for space, memory usage can be significantly reduced. This is especially critical for long-context inference.

CPU Offloading

Offloading some model layers or KV Cache to CPU memory can sacrifice some speed, but it allows models that otherwise cannot run to run:

# llama.cpp GPU layer control (underlying Ollama)
# Offload some layers to CPU to reduce GPU memory usage
ollama run deepseek-r1:8b

# Set GPU layers in Ollama conversation
# /set parameter num_gpu 20  # Only 20 layers on GPU, rest on CPU

# HuggingFace Accelerate CPU Offloading
from accelerate import infer_auto_device_map, dispatch_model
device_map = infer_auto_device_map(
    model,
    max_memory={0: "16GiB", "cpu": "64GiB"},
)
model = dispatch_model(model, device_map=device_map)

FlashAttention

FlashAttention is an IO-aware exact attention algorithm that reduces the time and memory complexity of attention computation from O(N^2) to nearly O(N) through block-wise computation and recomputation strategies. vLLM and SGLang have both integrated FlashAttention by default.

Version	Key Improvements	Speedup	Memory Savings
FlashAttention-1	IO-aware block-wise computation, avoids O(N^2) memory	2-3x	10-20x
FlashAttention-2	Optimized parallel strategy, reduces non-matrix multiplication operations	2x (vs FA1)	Comparable to FA1
FlashAttention-3	Optimized for H100, asynchronous computation, FP8 support	1.5-2x (vs FA2)	Comparable to FA2

PagedAttention

PagedAttention is the core innovation of vLLM, managing KV Cache like virtual memory paging in operating systems. It divides the KV Cache into fixed-size "pages" (blocks), allocated and released on demand, solving the fragmentation and waste of KV Cache:

Memory utilization improvement: from 20%-40% in traditional approaches to nearly 100%
Support for larger batch sizes: the same memory can handle more concurrent requests
Memory sharing: during parallel sampling (beam search), multiple sequences share the same KV Cache pages
No reservation needed: no longer need to reserve maximum-length KV Cache space for each request

# vLLM PagedAttention configuration
vllm serve deepseek-ai/DeepSeek-V3 \
  --block-size 16 \              # KV Cache page size (in tokens)
  --gpu-memory-utilization 0.95 \ # Use up to 95% of GPU memory
  --max-num-seqs 128              # Maximum number of concurrent sequences

Chapter 7

Tensor Parallelism and Pipeline Parallelism

When a single GPU cannot accommodate the entire model, distributed inference techniques are required. Tensor Parallelism (TP) and Pipeline Parallelism (PP) are the two most commonly used distributed strategies. Understanding their differences and applicable scenarios is crucial.

TP vs PP Comparison

Comparison Dimension	Tensor Parallelism (TP)	Pipeline Parallelism (PP)
Partitioning Method	Split the weight matrix of a single layer across multiple GPUs	Assign different layers to different GPUs
Communication Volume	High, requires AllReduce for each layer	Low, only activations transferred between layers
GPU Utilization	High, all GPUs work simultaneously	Has bubbles, some GPUs idle
Cross-Node Communication	Not recommended, severe communication bottleneck	Suitable, low communication volume
Recommended GPU Count	2-8, within a single node	4-32, can span nodes

Optimal Configurations for Different Model Sizes

Model	Parameters	Recommended GPU	TP	PP	Total GPUs
DeepSeek-R1-8B	8B	RTX 4090	1	1	1
DeepSeek-R1-70B	70B	A100 80GB	4	1	4
DeepSeek-V3	671B (37B active)	H100 80GB	8	1	8
DeepSeek-V3 (Full)	671B	A100 80GB	8	2	16

vLLM Distributed Inference Configuration

# Single node 8 GPUs, tensor parallel
vllm serve deepseek-ai/DeepSeek-V3 \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.95

# Multi-node: 2 nodes each with 8 GPUs, TP=8 PP=2
# Node 0
vllm serve deepseek-ai/DeepSeek-V3 \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2

# Communication optimization: use NCCL environment variables
# export NCCL_SOCKET_IFNAME=eth0
# export NCCL_IB_DISABLE=0  # Enable InfiniBand
# export NCCL_NET_GDR_LEVEL=5  # GPUDirect RDMA

Communication Cost Optimization

Communication overhead in distributed inference is a performance killer. The following measures can significantly reduce communication costs:

NVLink/NVSwitch: Use NVLink to connect GPUs within a single node, bandwidth 900GB/s, far exceeding PCIe
InfiniBand: Use InfiniBand (200-400GB/s) for cross-node communication, avoid Ethernet
GPUDirect RDMA: GPU communicates directly via RDMA, bypassing CPU, reducing latency
Communication-Computation Overlap: Overlap AllReduce communication with computation of the next layer

Chapter 8

Hardware Selection and Cost Optimization

Choosing the right hardware is key to finding the optimal balance between performance and cost. Different GPUs vary greatly in performance, and the inference cost of the same model on different hardware can differ by dozens of times.

GPU Performance Comparison

GPU	VRAM	FP16 Compute	Bandwidth	Cloud Rental Price	Suitable Models
T4	16 GB	65 TFLOPS	320 GB/s	~$0.35/h	Models up to 7B
A10	24 GB	125 TFLOPS	600 GB/s	~$0.75/h	8B-13B models
A100 80GB	80 GB	312 TFLOPS	2,039 GB/s	~$1.50/h	70B models, MoE models
H100 80GB	80 GB	989 TFLOPS	3,352 GB/s	~$2.80/h	DeepSeek-V3, FP8 inference

Per-Token Cost Estimation

Taking DeepSeek-R1-8B (Q4_K_M quantization) as an example, the inference costs on different GPUs:

GPU	TPS	Tokens per Hour	Cost per Million Tokens
T4	~40	144K	$2.43
A10	~80	288K	$2.60
A100	~200	720K	$2.08
DeepSeek API	-	-	$0.14 (V3)

Note: The official DeepSeek API price is much lower than the cost of self-hosted GPU inference. For most scenarios, using the API is more economical than self-hosting. Only when the daily call volume exceeds ten million tokens might a self-hosted GPU cluster have a cost advantage.

Cloud vs. Self-hosted

Comparison Dimension	Cloud GPU	Self-hosted
Initial Investment	Zero	High (H100 ~$30K/unit)
Elasticity	High, scale anytime	Low, fixed hardware
Data Security	Compliance needs assessment	Fully controllable
Daily Cost	Pay-as-you-go	Fixed (electricity + maintenance)

Spot Instance Strategy

Using cloud providers' Spot/Preemptible instances can save 60%-90% of GPU costs. However, Spot instances may be reclaimed at any time, so fault-tolerant design is necessary:

Multi-region deployment: Launch Spot instances in different availability zones to reduce the probability of simultaneous reclamation
Checkpoint mechanism: Regularly save the state of the inference service, and quickly recover after reclamation
Hybrid strategy: Use on-demand instances for core services, and Spot instances for elastic load
Warm pool: Maintain a certain number of idle instances as a buffer, and seamlessly switch when Spot instances are reclaimed

Chapter 9

Performance Benchmark

The following is performance test data based on real environments, comparing different inference frameworks on different hardware. All tests use DeepSeek-R1-8B (Q4_K_M quantization), with 512 input tokens and 256 output tokens.

Inference Framework Performance Comparison (RTX 4090 24GB)

Framework	TPS	TTFT	Concurrency 8	Concurrency 32	VRAM Usage
Ollama	~65	~280ms	-	-	~5.5 GB
vLLM	~120	~150ms	~850 TPS	~2,400 TPS	~6.8 GB
SGLang	~135	~120ms	~920 TPS	~2,600 TPS	~6.5 GB
TGI	~110	~160ms	~800 TPS	~2,200 TPS	~7.0 GB

SGLang performs best on DeepSeek models, thanks to its RadixAttention and efficient MoE scheduling. vLLM follows closely, with a more mature ecosystem. Ollama is suitable for personal use, not for high-concurrency services.

Performance Comparison Across Model Sizes (vLLM + A100 80GB)

Model	Quantization	Single GPU TPS	4-GPU TP TPS	VRAM/GPU
DeepSeek-R1-8B	AWQ INT4	~180	-	~6 GB
DeepSeek-R1-32B	AWQ INT4	~70	-	~22 GB
DeepSeek-R1-70B	AWQ INT4	-	~180	~40 GB
DeepSeek-V3	FP8	-	~120	~65 GB

DeepSeek-R1-8B Performance on Different Hardware

Hardware	VRAM	Ollama TPS	vLLM TPS	Use Case
Apple M2 16GB	Unified Memory	~15	-	Personal Experience
RTX 4060 8GB	8 GB	~30	~50	Beginner Development
RTX 4090 24GB	24 GB	~65	~120	Small Team Service
A100 80GB	80 GB	~90	~180	Production Service
H100 80GB	80 GB	~130	~280	Large-scale production

Chapter 10

Optimization Checklist

From baseline to production, follow the steps below to gradually optimize your DeepSeek inference service. Each step brings quantifiable performance improvements.

Phase 1: Basic Optimization (Immediate Results)

Choose the right inference framework: Switch from Ollama to vLLM or SGLang to get Continuous Batching and PagedAttention, improving throughput by 2-5x
Enable model quantization: Use AWQ INT4 or GGUF Q4_K_M to reduce VRAM usage by 60% and increase speed by 2-3x
Enable FlashAttention: Already enabled by default in vLLM/SGLang, no extra configuration needed
Set a reasonable max_model_len: Do not exceed the actual context length needed, to avoid wasting KV Cache
Adjust gpu_memory_utilization: Increase from the default 0.90 to 0.95 to fully utilize VRAM

Phase 2: Advanced Optimization (Significant Improvement)

Enable FP8 KV Cache: Halves KV Cache memory usage, supporting larger batch sizes and longer contexts
Enable Prefix Caching: Reduces TTFT by 50%-80% in multi-turn conversation scenarios
Configure Chunked Prefill: Prevents long prompts from blocking decode of other requests
Tune max_num_seqs: Find the optimal concurrency based on GPU memory and load
Enable speculative decoding: Choose a suitable small model as draft model for 2-3x speedup

Phase 3: Production-Grade Optimization (Extreme Performance)

Tensor Parallel Deployment: For large models (70B+), use TP to split across multiple GPUs to break single-GPU VRAM limits
Use NVLink/InfiniBand: Use high-speed interconnect for distributed inference to reduce communication overhead
Enable NCCL optimizations: Configure GPUDirect RDMA, NCCL_NET_GDR_LEVEL, etc.
Mixed-precision inference: Use FP8 on H100, BF16 on other GPUs to balance accuracy and speed
Load testing and monitoring: Use benchmark_serving.py for regular load tests, Prometheus + Grafana for monitoring performance metrics

Performance Optimization Decision Quick Reference

Your Problem	Try First	Expected Improvement
Not enough VRAM to run the model	AWQ/GPTQ INT4 quantization + CPU Offloading	VRAM reduction 60%+
Inference speed too slow	Switch to vLLM/SGLang + quantization	Speed improvement 3-5x
Insufficient concurrency	Continuous Batching + FP8 KV Cache	Concurrency improvement 3-8x
Slow first token in multi-turn conversations	Prefix Caching	TTFT reduced by 50%-80%
OOM in long-context inference	FP8 KV Cache + reduce max_num_seqs	Context length doubled
GPU cost too high	Spot Instance + quantization + small model	Cost reduced by 60%-90%

Performance Optimization FAQ

DeepSeek Performance Optimization FAQ

How big is the performance gap between Ollama and vLLM? +

In single-request scenarios, vLLM is about 1.5-2x faster than Ollama. But under high concurrency, vLLM's Continuous Batching and PagedAttention advantages are significant, with throughput up to 5-10x that of Ollama. If you need to build a production-grade API service, we strongly recommend using vLLM or SGLang. For personal use, Ollama's simplicity and ease of use are more advantageous. See DeepSeek Deployment Tutorial for details.

Which quantization should I choose: AWQ or GPTQ? +

We recommend AWQ. AWQ is generally faster than GPTQ in inference speed (about 10%-20%), and vLLM has native support for AWQ, making integration simpler. GPTQ is slightly better than AWQ in precision retention (the difference is minimal, usually less than 0.5%). If you have extremely high precision requirements, you can choose GPTQ INT8. In most scenarios, AWQ INT4 is the best balance between speed and precision. For more model quantization information, see DeepSeek Open Source Models.

Not enough VRAM to run DeepSeek-V3, what should I do? +

DeepSeek-V3 is a 671B-parameter MoE model, and even after quantization, it requires 8 A100/H100 GPUs. If your hardware is insufficient, here are some options: 1) Use the official DeepSeek API, which is extremely cost-effective (input 1 yuan/million tokens); 2) Use the DeepSeek-R1 distilled versions (1.5B-70B), which have similar performance but much lower requirements; 3) Rent cloud GPUs on demand. See DeepSeek Model Architecture Details for more.

Is speculative decoding effective in all scenarios? +

Not in all scenarios. Speculative decoding works best in the following cases: 1) High determinism in output content (e.g., code generation, translation); 2) The draft model and target model share the same architecture and training data; 3) Sufficient VRAM to load both models simultaneously. In scenarios with high randomness like creative writing, the acceptance rate may be below 60%, limiting the speedup. We recommend testing the acceptance rate on a small scale before enabling it.

How to choose the GGUF quantization level for Ollama? +

Q4_K_M is the best choice for most scenarios, balancing speed and precision. If you have limited VRAM (below 8GB), choose Q3_K_M; if you pursue the highest quality (16GB+ VRAM), choose Q5_K_M or Q6_K. Q2 levels have noticeable precision loss and are not recommended. Use ollama run deepseek-r1:8b to download the default Q4_K_M version. For more download guides, see DeepSeek Model Download.

Does Continuous Batching require special configuration? +

vLLM and SGLang enable Continuous Batching by default, no additional configuration required. You only need to focus on two key parameters: max_num_seqs (maximum number of concurrent sequences) and max_num_batched_tokens (maximum number of tokens per iteration). It is recommended to start with the default values and adjust gradually based on actual load and GPU memory usage. If GPU utilization is low, increase max_num_seqs; if OOM occurs, decrease max_num_seqs or reduce max_model_len.

Learn More

DeepSeek Related Tutorials

In-depth learning on deploying, using, and developing DeepSeek models.

Deployment Tutorial

DeepSeek Deployment Tutorial

Ollama, Docker, vLLM, K8s deployment solutions, from single machine to cluster.

Model Details

DeepSeek Model Architecture

Technical architecture, Benchmark, MoE details, selection comparison.

Model List

DeepSeek Open Source Models

Complete catalog and comparison of 6 series, 20+ models.

Download Guide

DeepSeek Model Download

Download guides for Ollama, Hugging Face, GitHub.

Getting Started

How to Use DeepSeek Models

Four usage methods, zero-basic-entry tutorial.

Fine-tuning

DeepSeek Fine-tuning Tutorial

Complete guide for LoRA, QLoRA, and full fine-tuning.