DeepSeek Model Deployment Tutorial
From beginner to enterprise-level deployment. One-click deployment with Ollama, Docker containerization, vLLM high-performance inference, Dify application integration. Real commands, complete configuration.
Start DeploymentFour Deployment Options
From personal computers to enterprise servers, the four options increase in difficulty; choose as needed.
Ollama Local Deployment
Deploy DeepSeek models with a single command. Suitable for individual developers, learning and research, and privacy-sensitive scenarios.
Docker Container Deployment
Deploy Ollama + DeepSeek via Docker containerization. Suitable for team collaboration, environment isolation, and rapid migration.
vLLM High-Performance Deployment
Production-grade inference engine with PagedAttention technology, 10-20x higher throughput than Ollama. Suitable for high-concurrency scenarios.
Dify Application Integration
Integrate DeepSeek models into the Dify platform to quickly build AI applications, knowledge base Q&A, and Agent workflows.
Local Deployment of DeepSeek Model with Ollama
The simplest deployment solution, suitable for individual developers to run DeepSeek models on their local machine. Ollama automatically handles model download, quantization, and inference optimization.
1.1 Install Ollama
# Linux / macOS
curl -fsSL https://ollama.com/install.sh | sh
# Windows
# Download installer: https://ollama.com/download/windows
# Double-click to install
1.2 Deploy DeepSeek Model
# Deploy DeepSeek R1 8B (recommended for beginners)
ollama run deepseek-r1:8b
# Deploy DeepSeek Coder 6.7B (for coding scenarios)
ollama run deepseek-coder:6.7b
# Deploy DeepSeek V3 671B (enterprise-level, requires 404GB space)
ollama run deepseek-v3
1.3 Configure Ollama Service
Ollama provides API service on localhost:11434 by default. To enable remote access or adjust configuration:
# Set environment variable to allow remote access
# Linux/macOS
export OLLAMA_HOST=0.0.0.0:11434
# Set model storage path
export OLLAMA_MODELS=/path/to/models
# Set number of concurrent requests
export OLLAMA_NUM_PARALLEL=4
# Restart Ollama to apply configuration
systemctl restart ollama # Linux
# macOS: Quit Ollama app and reopen
1.4 Verify Deployment
# View running models
ollama ps
# Test API endpoint
curl http://localhost:11434/api/chat -d '{
"model": "deepseek-r1:8b",
"messages": [{"role": "user", "content": "Hello"}]
}'
# List models
ollama list
Docker Containerized Deployment of DeepSeek
Deploy Ollama + DeepSeek via Docker to achieve environment isolation, rapid migration, and version management. Suitable for team development and multi-environment deployment.
2.1 Deploy Ollama with Docker
# Pull official Ollama image
docker pull ollama/ollama
# Run Ollama container (CPU mode)
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
# Run Ollama container (GPU mode, requires nvidia-docker)
docker run -d --gpus all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
2.2 Deploy DeepSeek Model in Container
# Enter container and execute command
docker exec -it ollama ollama pull deepseek-r1:8b
# Or run directly
docker exec -it ollama ollama run deepseek-r1:8b
2.3 Docker Compose Orchestration (Recommended)
Create docker-compose.yml:
version: '3.8'
services:
ollama:
image: ollama/ollama
container_name: ollama
ports:
- "11434:11434"
volumes:
- ollama_data:/root/.ollama
environment:
- OLLAMA_KEEP_ALIVE=24h
- OLLAMA_HOST=0.0.0.0
restart: unless-stopped
# GPU support (uncomment)
# deploy:
# resources:
# reservations:
# devices:
# - driver: nvidia
# count: 1
# capabilities: [gpu]
# Optional: Open WebUI frontend
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
ports:
- "3000:8080"
volumes:
- open-webui:/app/backend/data
environment:
- OLLAMA_BASE_URL=http://ollama:11434
restart: unless-stopped
volumes:
ollama_data:
open-webui:
# Start all services
docker compose up -d
# Deploy model
docker exec -it ollama ollama pull deepseek-r1:8b
vLLM High-Performance Inference Deployment
Recommended solution for production environments. vLLM uses PagedAttention technology, achieving 10-20x higher throughput than traditional inference. Supports OpenAI-compatible API for seamless migration.
3.1 Install vLLM
# Install vLLM (requires CUDA environment)
pip install vllm
# Verify installation
python -c "import vllm; print(vllm.__version__)"
3.2 Deploy DeepSeek R1 Distilled Version
# Start vLLM inference service
python -m vllm.entrypoints.openai.api_server \
--model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
--tensor-parallel-size 2 \
--max-model-len 32768 \
--gpu-memory-utilization 0.95 \
--port 8000
# Parameter explanation:
# --tensor-parallel-size: Number of GPUs (tensor parallelism)
# --max-model-len: Maximum context length
# --gpu-memory-utilization: GPU memory utilization
3.3 Deploy DeepSeek V3 (671B)
DeepSeek V3 is a MoE model that requires multi-GPU cluster deployment:
# Deploy DeepSeek V3 on 8x H100/A100
python -m vllm.entrypoints.openai.api_server \
--model deepseek-ai/DeepSeek-V3 \
--tensor-parallel-size 8 \
--max-model-len 131072 \
--gpu-memory-utilization 0.90 \
--trust-remote-code \
--port 8000
3.4 Test API Endpoint
# vLLM provides OpenAI-compatible API
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
"messages": [{"role": "user", "content": "Hello"}]
}'
3.5 Performance Comparison
| Inference Engine | Throughput | First Token Latency | Use Case |
|---|---|---|---|
| Ollama | Baseline | Medium | Personal use, development testing |
| vLLM | 10-20x | Low | High-concurrency production environments |
| SGLang | 8-15x | Low | High concurrency, structured output |
| Text Generation Inference | 5-10x | Medium | HuggingFace ecosystem |
3.6 vLLM Core Technology: Continuous Batching
Traditional inference engines use Static Batching: they must wait for all requests in a batch to complete before returning results, short requests are blocked by long ones, and GPU utilization is low. vLLM's Continuous Batching allows requests to dynamically join and leave the batch—when a request finishes generating, it is immediately removed from the batch, and new requests can seamlessly join without waiting for the entire batch to complete. In high-concurrency scenarios, this can increase throughput by more than 10x.
| Feature | Static Batching | Continuous Batching |
|---|---|---|
| Request joining | Batch submitted at once | Dynamic, anytime |
| Short request blocking | Severe (waits for long requests) | None (leaves when done) |
| GPU utilization | 30%-50% | 80%-95% |
| Typical throughput | Baseline | 10-20x |
3.7 PagedAttention Detailed Explanation
PagedAttention is vLLM's core innovation, inspired by the virtual memory paging mechanism of operating systems. In traditional inference, KV Cache uses contiguous memory allocation, leading to severe memory fragmentation (internal fragmentation + external fragmentation), with actual memory utilization of only 20%-40%. PagedAttention divides KV Cache into fixed-size Blocks (similar to memory pages), allocated on demand and stored non-contiguously, completely eliminating fragmentation and increasing memory utilization to 96%.
Key advantages of PagedAttention:
- Zero fragmentation: Block-level allocation, memory utilization jumps from ~40% to ~96%
- Memory sharing: KV Cache Blocks of the same prompt can be shared across multiple output sequences (saves 55% memory in Beam Search scenarios)
- Flexible scheduling: Blocks can be dynamically swapped in/out, supporting context windows larger than memory
- Naturally compatible with Continuous Batching: Block granularity naturally supports dynamic request scheduling
3.8 Tensor Parallelism vs Pipeline Parallelism
When deploying large models, multi-GPU parallelism is required. vLLM supports two parallel strategies:
| Feature | Tensor Parallelism (TP) | Pipeline Parallelism (PP) |
|---|---|---|
| Principle | Split single-layer weights across multiple GPUs, each card computes partial results then AllReduce | Split model by layers, GPUs pipeline intermediate results |
| Communication Pattern | AllReduce (requires high-bandwidth NVLink/NVSwitch) | Point-to-point Send/Recv (low bandwidth requirement) |
| GPU Utilization | High (all GPUs work simultaneously) | Low (bubble idle waiting exists) |
| Applicable Scenarios | Single-node multi-GPU (≤8 GPUs), latency-sensitive | Cross-node multi-machine, bandwidth-constrained |
| Recommended Configuration | --tensor-parallel-size N |
--pipeline-parallel-size N |
# Mixed parallel example: 4 nodes × 8 GPUs to deploy DeepSeek V3
python -m vllm.entrypoints.openai.api_server \
--model deepseek-ai/DeepSeek-V3 \
--tensor-parallel-size 8 \
--pipeline-parallel-size 4 \
--max-model-len 131072
3.9 Prefix Caching
Prefix Caching automatically detects and caches the KV Cache of the same prefix. When multiple requests share the same System Prompt or Few-shot examples, vLLM computes the KV Cache of the prefix only once, and subsequent requests directly reuse it, significantly reducing the first token latency (TTFT).
Typical benefits of Prefix Caching:
- Long System Prompt (e.g., 4096-token system prompt): TTFT reduced by 50%-80%
- Few-shot examples: KV Cache of the same examples is shared across all requests
- Multi-turn conversations: prefix parts of conversation history are automatically reused
- vLLM enables Automatic Prefix Caching (APC) by default; set
--enable-prefix-cachingto explicitly enable it
3.10 Speculative Decoding
Speculative Decoding uses a small draft model to quickly generate multiple candidate tokens, and then the large model verifies them all at once. If verification passes, multiple tokens are accepted; if it fails, it rolls back and recomputes. In low-latency scenarios (such as chat, code completion), it can increase throughput by 1.5-3 times without losing generation quality.
# vLLM speculative decoding configuration example
python -m vllm.entrypoints.openai.api_server \
--model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
--speculative-model deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
--num-speculative-tokens 5 \
--tensor-parallel-size 2
Recommended draft model: DeepSeek-R1-Distill-Qwen-1.5B (only 1.1GB, 5-10 times faster than the main model, verification accuracy >85%)
Deploy DeepSeek with llama.cpp
llama.cpp is a high-performance inference engine written in C/C++, supporting CPU and GPU inference without requiring a Python environment. Ollama is built on top of llama.cpp, and using llama.cpp directly gives you more control and custom quantization options.
What is llama.cpp?
llama.cpp is a pure C/C++ LLM inference engine developed by Georgi Gerganov, aiming to run large models efficiently on consumer hardware. It supports CPU inference (AVX2/AVX512 instruction set acceleration), GPU inference (CUDA/Metal/Vulkan), hybrid inference (CPU+GPU), and multiple quantization formats (Q2_K to Q8_0). llama.cpp is the underlying inference engine of Ollama—Ollama wraps model management, API services, and a user-friendly CLI on top of llama.cpp.
Relationship between llama.cpp and Ollama:
- Ollama = llama.cpp (inference engine) + Model Registry + REST API + CLI tools
- llama.cpp = pure inference engine, more low-level and flexible, but requires manual model file management
- Ollama's quantized models (e.g., Q4_K_M) are GGUF format from llama.cpp
- Using llama.cpp directly is suitable for: custom quantization parameters, CPU optimization, embedding into C/C++ projects
Installing llama.cpp
# Clone repository and compile
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
# CPU mode compilation (default, supports AVX2)
make -j$(nproc)
# CUDA accelerated compilation
make LLAMA_CUDA=1 -j$(nproc)
# Metal accelerated compilation (macOS)
make LLAMA_METAL=1 -j$(nproc)
# Vulkan accelerated compilation (cross-platform GPU)
make LLAMA_VULKAN=1 -j$(nproc)
# Verify installation
./llama-cli --version
Converting DeepSeek Model to GGUF Format
llama.cpp uses the GGUF (GPT-Generated Unified Format) format. If you have a DeepSeek model in HuggingFace format, you need to convert it to GGUF first:
# Install conversion dependencies
pip install transformers torch sentencepiece
# Use llama.cpp's built-in conversion script
python convert_hf_to_gguf.py deepseek-ai/DeepSeek-R1-Distill-Qwen-8B \
--outtype q8_0 \
--outfile deepseek-r1-8b.Q8_0.gguf
# Quantize to a smaller format (Q4_K_M recommended, half the size, almost lossless quality)
./llama-quantize deepseek-r1-8b.Q8_0.gguf deepseek-r1-8b.Q4_K_M.gguf Q4_K_M
You can also directly download community pre-converted GGUF models without manual conversion. Search for "deepseek gguf" on HuggingFace. Visit DeepSeek Download to learn how to obtain models.
Running DeepSeek R1 Inference
# Command-line interactive mode
./llama-cli -m deepseek-r1-8b.Q4_K_M.gguf -p "Hello, please introduce yourself" -n 512
# Interactive chat mode (-cnv enables conversation template)
./llama-cli -m deepseek-r1-8b.Q4_K_M.gguf -cnv
# CPU inference optimization (specify number of threads)
./llama-cli -m deepseek-r1-8b.Q4_K_M.gguf -p "Hello" -t 8 -n 256
# GPU inference (specify number of GPU layers)
./llama-cli -m deepseek-r1-8b.Q4_K_M.gguf -p "Hello" -ngl 33 -n 256
# Parameter explanation:
# -m: model file path
# -p: input prompt
# -n: maximum number of tokens to generate
# -t: number of CPU threads
# -ngl: number of layers to offload to GPU (-1 = all)
# -cnv: chat mode (automatically applies chat template)
Server Mode (providing API service)
llama.cpp provides a built-in HTTP Server compatible with the OpenAI API format:
# Start llama.cpp HTTP Server
./llama-server -m deepseek-r1-8b.Q4_K_M.gguf \
--host 0.0.0.0 \
--port 8080 \
--n-gpu-layers 33 \
--ctx-size 8192 \
--threads 8
# Test API
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-r1-8b",
"messages": [{"role": "user", "content": "Hello"}]
}'
# Check Server Status
curl http://localhost:8080/health
llama.cpp Quantization Format Selection Guide
| Quantization Format | Bits per Parameter | Model Size | Quality Loss | Recommended Use Case |
|---|---|---|---|---|
| Q8_0 | 8-bit | ~8GB (8B model) | Almost lossless | When GPU VRAM is sufficient |
| Q6_K | 6-bit | ~6GB (8B model) | Very low | High quality requirement, limited VRAM |
| Q5_K_M | 5-bit | ~5.5GB (8B model) | Low | Balance between quality and size |
| Q4_K_M | 4-bit | ~5GB (8B model) | Acceptable | Recommended (Ollama default) |
| Q3_K_M | 3-bit | ~4GB (8B model) | Medium | CPU inference, memory constrained |
| Q2_K | 2-bit | ~3GB (8B model) | Large | Extremely constrained devices |
When to Use llama.cpp Directly Instead of Ollama?
- Custom quantization: Need quantization schemes beyond Q2_K to Q8_0, or experimental quantization formats
- Extreme CPU optimization: Need precise control over thread count, NUMA binding, instruction set selection
- C/C++ integration: Embed the inference engine into C/C++ projects without Python dependencies
- Edge device deployment: Resource-constrained platforms like Raspberry Pi, phones, embedded devices
- More control: Need to adjust sampling parameters, KV cache size, batch size, and other low-level configurations
- No containers: Pure binary deployment without Docker or Python environment
For most users, using Ollama directly is recommended, as it encapsulates the best practices of llama.cpp. Only consider using llama.cpp directly when you need the highly customized scenarios above. Visit the DeepSeek Usage Guide for more information.
Deploy DeepSeek with SGLang
SGLang is a new generation high-performance LLM inference framework developed by Stanford University, UC Berkeley, and other institutions. The RadixAttention technology enables efficient KV Cache reuse, outperforming vLLM in structured output scenarios.
What is SGLang?
SGLang is a high-performance serving framework optimized for LLM inference. Its core innovations include RadixAttention (prefix caching based on Radix Tree), Structured Outputs (constrained decoding for structured outputs), and Constrained Decoding. SGLang demonstrates 8-15x throughput compared to traditional inference in multiple benchmarks, especially suitable for structured scenarios requiring JSON output, Function Calling, code generation, etc.
SGLang Core Features:
- RadixAttention: Automatic prefix caching based on Radix Tree, more efficient than vLLM's Prefix Caching with higher cache hit rate
- Structured Outputs: Native support for JSON Schema, Regex, Grammar constraints, ensuring 100% compliant output format
- Constrained Decoding: Supports FSM (Finite State Machine) constrained decoding, zero overhead to guarantee output format
- OpenAI-compatible API: Fully compatible with /v1/chat/completions and /v1/completions endpoints
- Efficient Scheduling: RadixAttention's cache-aware scheduler automatically optimizes request order to maximize cache hits
Install SGLang
# Install SGLang (includes all dependencies)
pip install "sglang[all]"
# Or install only the core inference engine
pip install sglang
# Verify installation
python -c "import sglang; print(sglang.__version__)"
Start SGLang Inference Service
# Deploy DeepSeek R1 distilled version (32B)
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
--tp 2 \
--host 0.0.0.0 \
--port 30000
# Deploy DeepSeek R1 distilled version (8B, single GPU)
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-8B \
--tp 1 \
--host 0.0.0.0 \
--port 30000
# Parameter explanation:
# --model-path: HuggingFace model ID or local path
# --tp: number of GPUs for tensor parallelism
# --host: listening address
# --port: service port
Test API Endpoint
SGLang is fully compatible with OpenAI API format, seamless migration:
# Test with curl
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
"messages": [{"role": "user", "content": "Hello, please introduce yourself"}],
"temperature": 0.7,
"max_tokens": 512
}'
# Call using OpenAI Python SDK
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="not-needed"
)
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)
SGLang Structured Output Example
The core advantage of SGLang is structured output. The following example shows how to use JSON Schema to constrain the output format:
# Use JSON Schema to constrain output
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
"messages": [
{"role": "user", "content": "List 3 recommended AI books"}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "book_recommendations",
"schema": {
"type": "object",
"properties": {
"books": {
"type": "array",
"items": {
"type": "object",
"properties": {
"title": {"type": "string"},
"author": {"type": "string"},
"year": {"type": "integer"},
"reason": {"type": "string"}
},
"required": ["title", "author", "reason"]
}
}
},
"required": ["books"]
}
}
}
}'
SGLang vs vLLM: How to Choose?
| Comparison Dimension | SGLang | vLLM |
|---|---|---|
| Throughput | 8-15x (baseline) | 10-20x (baseline) |
| Prefix Caching | RadixAttention (Radix Tree) | Prefix Caching (Hash-based) |
| Structured Output | Native support, JSON Schema/Regex/Grammar | Supports guided decoding (newer) |
| Community Ecosystem | Rapidly growing | More mature, richer documentation |
| Deployment Complexity | pip install and use | pip install and use |
| Recommended Scenarios | Function Calling, JSON output, Agent workflows | General conversation, high-concurrency Chat API |
If your business scenario heavily relies on structured output (JSON/Function Calling), SGLang is a better choice. If it's mainly general conversation scenarios, vLLM's community ecosystem is more mature. Visit DeepSeek model details to learn about model capability comparison.
Deploy DeepSeek with Text Generation Inference (TGI)
Text Generation Inference (TGI) is HuggingFace's official LLM inference server, deeply integrated with the HuggingFace Hub ecosystem, supporting enterprise-grade features such as tensor parallelism, quantization, and watermarking.
What is TGI?
TGI (Text Generation Inference) is a production-grade inference server developed and maintained by HuggingFace, designed specifically for the HuggingFace model ecosystem. It provides features such as Tensor Parallelism, GPTQ/AWQ quantization, Watermarking, and SafeTensors loading. TGI integrates seamlessly with HuggingFace Hub, supporting direct model loading via model-id without manual download.
TGI core features:
- HuggingFace Hub integration: Load directly via model-id, automatically download model files
- Tensor Parallelism: Multi-GPU tensor parallelism for efficient multi-card resource utilization
- Quantization support: Supports multiple quantization schemes such as GPTQ, AWQ, EETQ, bitsandbytes
- Watermarking: Built-in watermarking technology to track AI-generated content
- Safetensors: Safe model loading to prevent malicious serialization code injection
- Continuous Batching: Dynamic batching to improve throughput
Deploy TGI with Docker
TGI is recommended to be deployed with Docker; the official image comes pre-configured with all dependencies:
# Pull the official TGI image
docker pull ghcr.io/huggingface/text-generation-inference:latest
# Deploy DeepSeek R1 distilled version (32B, 2 GPUs)
docker run --gpus all \
-p 8080:80 \
-v $PWD/data:/data \
-e HUGGING_FACE_HUB_TOKEN=hf_your_token_here \
ghcr.io/huggingface/text-generation-inference:latest \
--model-id deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
--num-shard 2 \
--max-input-length 4096 \
--max-total-tokens 8192
# Deploy DeepSeek R1 distilled version (8B, single GPU)
docker run --gpus all \
-p 8080:80 \<
-v $PWD/data:/data \
ghcr.io/huggingface/text-generation-inference:latest \
--model-id deepseek-ai/DeepSeek-R1-Distill-Qwen-8B \
--num-shard 1 \
--max-input-length 4096 \
--max-total-tokens 8192
# Parameter explanation:
# --model-id: HuggingFace model ID
# --num-shard: Number of GPU shards (tensor parallelism)
# --max-input-length: Maximum input length
# --max-total-tokens: Maximum total tokens (input + output)
# -v $PWD/data:/data: Model cache directory
Deploy with quantized models (reduce VRAM requirements)
# Use GPTQ quantized model (halves VRAM)
docker run --gpus all \
-p 8080:80 \
-v $PWD/data:/data \
ghcr.io/huggingface/text-generation-inference:latest \
--model-id TheBloke/DeepSeek-R1-Distill-Qwen-32B-GPTQ \
--num-shard 1 \
--quantize gptq
# Use AWQ quantized model
docker run --gpus all \
-p 8080:80 \
-v $PWD/data:/data \
ghcr.io/huggingface/text-generation-inference:latest \
--model-id TheBloke/DeepSeek-R1-Distill-Qwen-32B-AWQ \
--num-shard 1 \
--quantize awq
Test the API endpoint
# TGI uses OpenAI-compatible Messages API
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "tgi",
"messages": [{"role": "user", "content": "Hello, please introduce the DeepSeek model"}],
"max_tokens": 512,
"stream": false
}'
# View model info
curl http://localhost:8080/info
TGI Docker Compose Configuration
version: '3.8'
services:
tgi:
image: ghcr.io/huggingface/text-generation-inference:latest
container_name: tgi-deepseek
ports:
- "8080:80"
volumes:
- ./data:/data
environment:
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
command:
- --model-id
- deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
- --num-shard
- "2"
- --max-input-length
- "4096"
- --max-total-tokens
- "8192"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 2
capabilities: [gpu]
restart: unless-stopped
# Start TGI service
docker compose up -d
# View logs
docker compose logs -f tgi
TGI Performance Characteristics
TGI's throughput is approximately 5-10 times that of traditional inference, lower than vLLM and SGLang, but its advantage lies in deep integration with the HuggingFace ecosystem. If your team already uses HuggingFace Hub to manage models and relies on HF's Safetensors, model cards, automatic download, etc., TGI is the most natural choice.
TGI is suitable for heavy HuggingFace users, or enterprise scenarios requiring security features like Watermarking. Visit the DeepSeek model list for detailed information on each model.
Inference Engine Performance Benchmark
A horizontal comparison of five major inference engines, covering key metrics such as throughput, latency, GPU utilization, deployment difficulty, and API compatibility. Based on actual measurements of DeepSeek-R1-Distill-Qwen-32B on 2×A100 80GB.
| Inference Engine | Throughput | TTFT (First Token) | GPU Utilization | Deployment Difficulty | API Compatibility | Recommended Scenarios |
|---|---|---|---|---|---|---|
| Ollama | Baseline (1x) | Medium | 30%-50% | Very Simple | OpenAI Compatible | Personal Development, Local Testing |
| llama.cpp | 1-2x | Medium | 40%-60% (CPU Optimized) | Medium | OpenAI Compatible (Server) | CPU Inference, Edge Devices, C/C++ Integration |
| vLLM | 10-20x | Low | 80%-95% | Simple | OpenAI Compatible | High-Concurrency Production Environments |
| SGLang | 8-15x | Low | 80%-95% | Simple | OpenAI Compatible | Structured Output, Agent |
| TGI | 5-10x | Medium | 70%-85% | Medium | OpenAI Compatible | HuggingFace Ecosystem Users |
Selection Recommendations
| Use Case | Recommended Engine | Reason |
|---|---|---|
| Personal Learning / Local Testing | Ollama | One-line command deployment, zero configuration, automatic quantization |
| CPU Inference / Edge Devices | llama.cpp | Pure C/C++, no Python dependencies, extreme CPU optimization |
| High-Concurrency Chat API | vLLM | Highest throughput, most mature community, most complete ecosystem |
| Structured Output / Agent | SGLang | Best RadixAttention caching, native JSON Schema |
| HuggingFace Ecosystem | TGI | Deep integration with HF Hub, requires Watermarking |
| Enterprise Multi-Model Service | vLLM + SGLang | Combined use: vLLM for general scenarios, SGLang for structured scenarios |
There is no "best" engine, only the "most suitable" choice. It is recommended to choose based on actual business scenarios and resource conditions, and combine multiple engines if necessary. Visit DeepSeek Usage Guide and DeepSeek Model List for more technical details.
Integrating DeepSeek Models with Dify
Dify is an open-source LLM application development platform. Integrate DeepSeek models into Dify to quickly build AI applications, knowledge base Q&A, and Agent workflows.
4.1 Deploying Dify
# Clone the Dify repository
git clone https://github.com/langgenius/dify.git
cd dify/docker
# Copy environment variable configuration
cp .env.example .env
# Start Dify
docker compose up -d
# Access http://localhost:3000 to enter the Dify management interface
4.2 Connecting DeepSeek Models
- Log in to the Dify management interface and go to "Settings → Model Providers"
- Find the "OpenAI-API-compatible" provider
- Fill in the following information:
- API Base URL:
http://localhost:11434/v1(Ollama) orhttp://localhost:8000/v1(vLLM) - Model Name:
deepseek-r1:8b
- API Base URL:
- After saving, you can use the DeepSeek model in Dify applications
4.3 Dify Application Scenarios
Knowledge Base Q&A
Upload documents to the Dify knowledge base, and DeepSeek answers questions based on the document content, implementing RAG retrieval-augmented generation.
- Supports PDF/Word/TXT/Markdown
- Vector search + full-text search
- Citation tracing
ChatBot Application
Build a conversational AI based on DeepSeek, supporting custom system prompts, variables, and context windows.
- Conversation history management
- Prompt templates
- Embedded/standalone deployment
Agent Workflows
Build AI Agents with DeepSeek as the reasoning engine, connecting tools, APIs, and knowledge bases to automate tasks.
- Multi-tool invocation
- Conditional branching
- Code execution nodes
Production Deployment Architecture
To securely expose DeepSeek inference services to the public internet, you need Nginx reverse proxy, SSL/TLS encryption, API Key authentication, and rate limiting. Below is a complete production-grade configuration.
5.1 Nginx Reverse Proxy Configuration
Nginx acts as a reverse proxy, forwarding external requests to the backend Ollama (port 11434) or vLLM (port 8000), while providing connection pooling, buffering, and load balancing.
# /etc/nginx/sites-available/deepseek-proxy
upstream deepseek_backend {
# vLLM backend (recommended for production)
server 127.0.0.1:8000 max_fails=3 fail_timeout=30s;
keepalive 32;
}
upstream ollama_backend {
# Ollama backend (development)
server 127.0.0.1:11434 max_fails=3 fail_timeout=30s;
keepalive 16;
}
server {
listen 80;
server_name deepseek.yourdomain.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name deepseek.yourdomain.com;
# SSL certificate (Let's Encrypt)
ssl_certificate /etc/letsencrypt/live/deepseek.yourdomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/deepseek.yourdomain.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
# Security headers
add_header Strict-Transport-Security "max-age=63072000" always;
add_header X-Content-Type-Options nosniff;
# Request body size limit (large file uploads)
client_max_body_size 100M;
# API Key authentication
location /v1/ {
# Validate API Key
if ($http_authorization !~ "^Bearer sk-[a-zA-Z0-9]{48}$") {
return 401 '{"error":"Unauthorized: invalid or missing API key"}';
add_header Content-Type application/json;
}
# Rate limiting
limit_req zone=deepseek_api burst=20 nodelay;
limit_req_status 429;
# Proxy to vLLM
proxy_pass http://deepseek_backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# SSE streaming support
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
}
# Health check endpoint
location /health {
proxy_pass http://deepseek_backend/health;
access_log off;
}
}
5.2 Rate Limiting Configuration
Define rate limiting zones in the main Nginx configuration to prevent API abuse:
# Add to the http block of /etc/nginx/nginx.conf
http {
# Define rate limit zone: 10 requests/sec, burst 20
limit_req_zone $binary_remote_addr zone=deepseek_api:10m rate=10r/s;
# Per API Key rate limit (more granular)
limit_req_zone $http_authorization zone=deepseek_per_key:10m rate=50r/s;
# Concurrent connection limit
limit_conn_zone $binary_remote_addr zone=conn_per_ip:10m;
}
5.3 Let's Encrypt SSL Certificate Auto-Renewal
# Install Certbot
sudo apt install certbot python3-certbot-nginx
# Request certificate
sudo certbot --nginx -d deepseek.yourdomain.com
# Auto-renewal (built-in timer)
sudo systemctl status certbot.timer
# Test renewal manually
sudo certbot renew --dry-run
5.4 Health Check Endpoint
vLLM has a built-in health check endpoint that can be used for load balancer health probes:
# vLLM health check
curl http://localhost:8000/health
# Returns empty response, HTTP 200 = healthy
# Ollama health check (custom script)
# /usr/local/bin/healthcheck-ollama.sh
#!/bin/bash
curl -sf http://localhost:11434/api/tags > /dev/null && echo "OK" || echo "FAIL"
5.5 API Key Generation and Management
For production, it is recommended to use an API Gateway (such as Kong, APISIX) or build a simple key management system:
# Generate a secure API Key
openssl rand -hex 24
# Example output: a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2
# Map API Key to user in Nginx
# /etc/nginx/api_keys.conf
map $http_authorization $api_user {
"Bearer sk-a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2" "user_prod";
"Bearer sk-z9y8x7w6v5u4t3s2r1q0p9o8n7m6l5k4j3i2h1g0" "user_dev";
default "unknown";
}
Deploy DeepSeek on Kubernetes
Deploy DeepSeek inference service in a K8s cluster to achieve auto-scaling, rolling updates, GPU resource management, and persistent storage.
6.1 K8s Deployment (Ollama + DeepSeek)
# deepseek-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: deepseek-ollama
namespace: deepseek
labels:
app: deepseek-ollama
spec:
replicas: 1
selector:
matchLabels:
app: deepseek-ollama
strategy:
type: Recreate # GPU resources are exclusive, rolling updates not supported
template:
metadata:
labels:
app: deepseek-ollama
spec:
# GPU node scheduling
nodeSelector:
accelerator: nvidia-gpu
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
containers:
- name: ollama
image: ollama/ollama:latest
ports:
- containerPort: 11434
name: http
env:
- name: OLLAMA_HOST
value: "0.0.0.0"
- name: OLLAMA_KEEP_ALIVE
value: "24h"
- name: OLLAMA_NUM_PARALLEL
value: "4"
resources:
requests:
memory: "16Gi"
cpu: "4"
nvidia.com/gpu: "1"
limits:
memory: "32Gi"
cpu: "8"
nvidia.com/gpu: "1"
volumeMounts:
- name: model-storage
mountPath: /root/.ollama
# Automatically pull model after startup
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "sleep 10 && ollama pull deepseek-r1:8b"]
livenessProbe:
httpGet:
path: /api/tags
port: 11434
initialDelaySeconds: 120
periodSeconds: 30
readinessProbe:
httpGet:
path: /api/tags
port: 11434
initialDelaySeconds: 60
periodSeconds: 10
volumes:
- name: model-storage
persistentVolumeClaim:
claimName: deepseek-models-pvc
6.2 K8s Service Configuration
# deepseek-service.yaml
apiVersion: v1
kind: Service
metadata:
name: deepseek-ollama-svc
namespace: deepseek
labels:
app: deepseek-ollama
spec:
type: ClusterIP
selector:
app: deepseek-ollama
ports:
- name: http
port: 11434
targetPort: 11434
protocol: TCP
sessionAffinity: ClientIP # Maintain session stickiness
6.3 K8s Ingress Configuration
# deepseek-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: deepseek-ingress
namespace: deepseek
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "100m"
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
nginx.ingress.kubernetes.io/proxy-buffering: "off"
# SSL 自動管理
cert-manager.io/cluster-issuer: "letsencrypt-prod"
# レート制限
nginx.ingress.kubernetes.io/limit-rps: "10"
nginx.ingress.kubernetes.io/limit-burst-multiplier: "3"
spec:
ingressClassName: nginx
tls:
- hosts:
- api.deepseek.example.com
secretName: deepseek-tls
rules:
- host: api.deepseek.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: deepseek-ollama-svc
port:
number: 11434
6.4 PersistentVolume モデルストレージ
# deepseek-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: deepseek-models-pv
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 500Gi # 複数のモデルを格納するのに十分
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: "/mnt/data/deepseek-models"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: deepseek-models-pvc
namespace: deepseek
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Gi
本番環境では、hostPath の代わりにクラウドストレージ CSI ドライバ（AWS EBS、阿里雲 NAS など）を使用し、Pod 移行時にデータが失われないようにすることを推奨します。DeepSeek モデルリスト で各モデルのストレージ要件を確認してください。
6.5 GPU ノードスケジューリング設定
K8s クラスタに NVIDIA Device Plugin がインストールされ、GPU ノードにラベルが付けられていることを確認してください：
# NVIDIA Device Plugin のインストール
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.15.0/deployments/static/nvidia-device-plugin.yml
# GPU ノードにラベルを付ける
kubectl label nodes gpu-node-1 accelerator=nvidia-gpu
kubectl label nodes gpu-node-2 accelerator=nvidia-gpu
# GPU リソースの確認
kubectl describe node gpu-node-1 | grep nvidia.com/gpu
6.6 ワンクリックデプロイスクリプト
# 名前空間を作成し、すべてのリソースをデプロイ
kubectl create namespace deepseek
kubectl apply -f deepseek-pv.yaml
kubectl apply -f deepseek-deployment.yaml
kubectl apply -f deepseek-service.yaml
kubectl apply -f deepseek-ingress.yaml
# デプロイ状態の確認
kubectl -n deepseek get pods,svc,ingress,pvc
# ログの確認
kubectl -n deepseek logs -f deployment/deepseek-ollama
Hardware Configuration Reference for Deploying DeepSeek Models
Choose the appropriate hardware configuration based on model specifications to avoid resource waste or insufficient performance.
| Model | Size | Recommended GPU | VRAM Requirement | Inference Engine | Estimated Monthly Cost |
|---|---|---|---|---|---|
| R1 1.5B | 1.1GB | CPU / Integrated GPU | — | Ollama | Free (Local) |
| R1 8B | 5.2GB | RTX 3060/4060 | 8GB | Ollama / vLLM | Free (Local) |
| R1 32B | 20GB | RTX 4090 / A5000 | 24GB | vLLM | Free (Local) |
| R1 70B | 43GB | 2x A100 (40GB) | 80GB | vLLM | ~$500/month (Cloud) |
| V3 / R1 671B | 404GB | 8x H100/A100 | 640GB+ | vLLM | ~$5000+/month (Cloud) |
Cloud Platform Deployment Solutions
Don't want to build your own server? Major cloud platforms offer GPU instances with pay-as-you-go pricing, allowing you to quickly deploy DeepSeek models. Below is a comparison of mainstream cloud platform solutions.
7.1 Cloud Platform Comparison
| Cloud Platform | Recommended Instance | GPU | VRAM | Applicable Models | Estimated Monthly Cost | Recommendation |
|---|---|---|---|---|---|---|
| AWS | g5.xlarge | 1× A10G | 24GB | 8B / 14B | ~$380/month | 4-star recommendation |
| AWS | p4d.24xlarge | 8× A100 40GB | 320GB | 70B / 671B | ~$32,000/month | 3-star recommendation |
| AWS SageMaker | ml.g5.2xlarge | 1× A10G | 24GB | 8B / 14B | ~$550/month | 4-star recommendation |
| Alibaba Cloud | ecs.gn7i-c8g1.2xlarge | 1× A10 | 24GB | 8B / 14B | ~¥2,800/month | 4-star recommendation |
| Alibaba Cloud PAI-EAS | A10 single-card instance | 1× A10 | 24GB | 8B / 14B | ~¥3,500/month | Five-star recommendation |
| AutoDL | RTX 4090 | 1× RTX 4090 | 24GB | 8B / 14B / 32B | ~¥800/month | Five-star recommendation |
| AutoDL | A100 80GB | 1× A100 80GB | 80GB | 70B | ~¥3,500/month | Five-star recommendation |
| Google Cloud | g2-standard-8 | 1× L4 | 24GB | 8B / 14B | ~$400/month | Three-star recommendation |
7.2 AWS Deployment Options
AWS EC2: Launch GPU instances directly and manually deploy Ollama or vLLM. Offers the highest flexibility, suitable for teams with operations experience.
# AWS CLI to launch g5.xlarge instance
aws ec2 run-instances \
--image-id ami-0abcdef1234567890 \
--instance-type g5.xlarge \
--key-name my-key-pair \
--security-group-ids sg-xxxxxxxxx \
--block-device-mappings '[{"DeviceName":"/dev/xvda","Ebs":{"VolumeSize":200,"VolumeType":"gp3"}}]' \
--tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=deepseek-server}]'
# Deploy after SSH login
ssh -i my-key-pair.pem ubuntu@
curl -fsSL https://ollama.com/install.sh | sh
ollama pull deepseek-r1:8b
AWS SageMaker: Managed machine learning platform, one-click model endpoint deployment, with built-in auto-scaling and monitoring. Suitable for teams that don't want to manage infrastructure.
SageMaker is about 30%-50% more expensive than EC2, but saves operational overhead. If just testing, use EC2 on-demand instances first. Visit DeepSeek Usage Guide for more.
7.3 Alibaba Cloud Deployment Options
GPU Cloud Servers: Alibaba Cloud offers GPU instance series like gn7i (A10) and gn7 (V100), suitable for domestic users needing low-latency deployment.
# Alibaba Cloud ECS GPU instance deployment
# 1. Create gn7i-c8g1.2xlarge instance in console
# 2. Choose Ubuntu 22.04 image, system disk 200GB
# 3. Install NVIDIA driver and CUDA
wget https://developer.download.nvidia.com/compute/cuda/12.4.0/local_installers/cuda_12.4.0_550.54.14_linux.run
sudo sh cuda_12.4.0_550.54.14_linux.run
# 4. Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
ollama pull deepseek-r1:8b
PAI-EAS (Elastic Algorithm Service): Alibaba Cloud AI platform, supports one-click deployment of DeepSeek models, providing online inference services. Built-in model management, version control, auto-scaling, and canary releases. Suitable for rapid deployment of enterprise-level AI applications.
7.4 AutoDL (Best Cost-Effective Choice in China)
AutoDL is one of the largest GPU rental platforms in China, focusing on cost-effectiveness. Billed hourly, prices are 1/3 to 1/5 of cloud providers. Supports various GPUs including RTX 3090/4090, A100, H100.
AutoDL Advantages:
- RTX 4090 only ~¥2.2/hour, monthly ~¥800
- Pre-installed environments like CUDA, PyTorch, conda
- Supports direct access via JupyterLab and SSH
- Built-in inference frameworks like Ollama, vLLM
- Persistent data disk, data not lost on shutdown
# One-click deployment after AutoDL instance launch
# Instance already has CUDA and PyTorch pre-installed
curl -fsSL https://ollama.com/install.sh | sh
ollama serve &
ollama pull deepseek-r1:32b # RTX 4090 runs 32B perfectly
# Or use vLLM
pip install vllm
python -m vllm.entrypoints.openai.api_server \
--model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
--tensor-parallel-size 1 \
--port 8000
7.5 Cost Optimization Recommendations
- Use preemptible/spot instances: AWS Spot, Alibaba Cloud preemptible instances, 60%-80% cheaper, suitable for non-critical tasks
- Model quantization: Use Ollama's default Q4_K_M quantization, VRAM requirements halved with almost no quality loss
- Prioritize 8B: DeepSeek-R1-8B is already excellent in most scenarios, only needs 8GB VRAM
- Choose based on demand: If concurrency <10, Ollama + single GPU is enough; concurrency >100, then vLLM + multiple GPUs needed
- Hybrid approach: Use AutoDL for development/testing (cheap), Alibaba Cloud/AWS for production (stable)
DeepSeek API Service (Official)
If you don't want to build your own server, using the DeepSeek official API directly is the simplest and most efficient choice. It is compatible with the OpenAI SDK, making migration extremely low-cost.
8.1 Integration Method
The DeepSeek official API provides an interface format fully compatible with OpenAI, allowing migration without modifying code. Simply replace base_url and api_key.
| API Endpoint | https://api.deepseek.com/v1 |
| Get API Key | platform.deepseek.com → API Keys |
| Compatibility | Fully compatible with OpenAI SDK / LangChain / LlamaIndex |
| Available Models | deepseek-chat (V3), deepseek-reasoner (R1) |
8.2 API Pricing (Much Lower than OpenAI)
| Model | Input Price | Output Price | Comparison to GPT-4o | Context Window |
|---|---|---|---|---|
| DeepSeek-V3 | ¥1 / 1M tokens | ¥2 / 1M tokens | 97% cheaper | 128K |
| DeepSeek-R1 | ¥4 / 1M tokens | ¥16 / 1M tokens | 90% cheaper | 128K |
| GPT-4o | $2.5 / 1M tokens | $10 / 1M tokens | — | 128K |
8.3 Python Call Example
# Use OpenAI SDK to call DeepSeek
# pip install openai
from openai import OpenAI
client = OpenAI(
api_key="sk-your-deepseek-api-key",
base_url="https://api.deepseek.com"
)
# Normal chat (V3)
response = client.chat.completions.create(
model="deepseek-chat",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain the basic principles of quantum computing"}
],
temperature=0.7,
max_tokens=2048,
stream=False
)
print(response.choices[0].message.content)
# Deep reasoning (R1)
response = client.chat.completions.create(
model="deepseek-reasoner",
messages=[
{"role": "user", "content": "Prove that the square root of 2 is irrational"}
]
)
print(response.choices[0].message.content)
8.4 JavaScript/Node.js Call Example
// Use OpenAI SDK to call DeepSeek
// npm install openai
import OpenAI from 'openai';
const client = new OpenAI({
apiKey: 'sk-your-deepseek-api-key',
baseURL: 'https://api.deepseek.com'
});
// Streaming output
const stream = await client.chat.completions.create({
model: 'deepseek-chat',
messages: [
{ role: 'user', content: 'Write a poem about artificial intelligence' }
],
stream: true
});
for await (const chunk of stream) {
const content = chunk.choices[0]?.delta?.content || '';
process.stdout.write(content);
}
8.5 Official API vs Self-Deployment Comparison
| Comparison Dimension | DeepSeek Official API | Self-Deployment (Ollama/vLLM) |
|---|---|---|
| Startup Cost | Zero (register and use) | High (requires GPU server) |
| Running Cost | Pay-as-you-go, very cheap for low usage | Fixed monthly fee, more cost-effective for high concurrency |
| Model Version | V3-671B (full version), R1-671B | Distilled versions (1.5B-70B), self-hosting V3 requires 8×H100 |
| Data Privacy | Data passes through DeepSeek servers | Fully local, data never leaves the domain |
| Latency | Depends on network, typically <500ms | Depends on hardware, extremely low with sufficient GPU |
| Recommended scenarios | Quick integration, low usage, need full-capability model | High concurrency, data-sensitive, need custom model |
Recommendation: First validate your business scenario with the official API, then decide whether to self-host. In most cases, the official API is more cost-effective. Visit DeepSeek Model Details to compare model capabilities.
Deployment Monitoring and Operations
After production deployment, a comprehensive monitoring system must be established. Monitor key metrics, set alerts, and ensure stable service operation.
9.1 Prometheus + Grafana Monitoring System
vLLM has built-in Prometheus metrics endpoints, combined with Grafana visualization panels, to monitor the health of inference services in real time.
# vLLM automatically exposes Prometheus metrics on startup
python -m vllm.entrypoints.openai.api_server \
--model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
--tensor-parallel-size 2
# Metrics are automatically exposed at http://localhost:8000/metrics
# View metrics
curl http://localhost:8000/metrics | head -30
9.2 Prometheus Configuration
# prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'vllm'
static_configs:
- targets: ['localhost:8000']
metrics_path: '/metrics'
scrape_interval: 10s
- job_name: 'node-exporter'
static_configs:
- targets: ['localhost:9100']
- job_name: 'nginx'
static_configs:
- targets: ['localhost:9113']
9.3 Key Monitoring Metrics
| Metric | Meaning | Prometheus Metric | Healthy Range |
|---|---|---|---|
| TPS | Tokens generated per second | vllm:generation_tokens_total |
Higher is better |
| TTFT | Time to First Token | vllm:time_to_first_token_seconds |
< 500ms (good) |
| TPOT | Time Per Output Token | vllm:time_per_output_token_seconds |
< 50ms (good) |
| GPU Utilization | GPU compute core usage | DCGM_FI_DEV_GPU_UTIL |
> 70% (healthy) |
| Memory Usage | GPU memory occupancy | vllm:gpu_cache_usage_perc |
< 95% (safe) |
| Request Queue | Number of requests waiting to be processed | vllm:num_requests_waiting |
< 10 (normal) |
9.4 Grafana Dashboard Configuration
# docker-compose to start Prometheus + Grafana
# monitoring-stack.yaml
version: '3.8'
services:
prometheus:
image: prom/prometheus:latest
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
volumes:
- grafana_data:/var/lib/grafana
- ./grafana-dashboards:/etc/grafana/provisioning/dashboards
volumes:
prometheus_data:
grafana_data:
Grafana import Dashboard ID 19004 (NVIDIA DCGM Exporter) to monitor GPU metrics, import 1860 (Node Exporter) to monitor system resources. vLLM custom dashboards can be obtained from the community or built yourself.
9.5 Log Management
# Redirect vLLM logs to file
python -m vllm.entrypoints.openai.api_server \
--model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
--tensor-parallel-size 2 \
2>&1 | tee -a /var/log/vllm/server.log
# Use logrotate to manage log rotation
# /etc/logrotate.d/vllm
/var/log/vllm/*.log {
daily
rotate 30
compress
delaycompress
missingok
notifempty
copytruncate
}
# Use Loki + Promtail to collect logs (optional)
# Send logs to Loki, query uniformly in Grafana
9.6 Auto Scaling
For K8s deployments, auto scaling can be implemented based on GPU utilization or request queue length:
# deepseek-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: deepseek-hpa
namespace: deepseek
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: deepseek-ollama
minReplicas: 1
maxReplicas: 4
metrics:
- type: External
external:
metric:
name: vllm_num_requests_waiting
selector:
matchLabels:
app: deepseek-ollama
target:
type: AverageValue
averageValue: "5"
behavior:
scaleDown:
stabilizationWindowSeconds: 300 # Scale down only after 5 minutes
scaleUp:
stabilizationWindowSeconds: 30 # Scale up after 30 seconds
GPU resources are scarce and expensive, so set a reasonable maxReplicas to avoid cost overruns. Also, GPU scaling is limited by the number of GPUs available in the cluster; it is recommended to use Cluster Autoscaler to dynamically scale nodes. Visit DeepSeek Download to learn how to obtain models.
9.7 Alert Rules
# prometheus-alerts.yml
groups:
- name: deepseek_alerts
rules:
- alert: HighTTFT
expr: histogram_quantile(0.95, vllm:time_to_first_token_seconds) > 1.0
for: 5m
labels:
severity: warning
annotations:
summary: "TTFT p95 exceeds 1 second"
- alert: HighGPUUsage
expr: vllm:gpu_cache_usage_perc > 95
for: 5m
labels:
severity: critical
annotations:
summary: "GPU memory usage exceeds 95%"
- alert: RequestQueueBacklog
expr: vllm:num_requests_waiting > 20
for: 2m
labels:
severity: warning
annotations:
summary: "Request queue backlog exceeds 20"
- alert: ServiceDown
expr: up{job="vllm"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "vLLM service unavailable"