DeepSeek Model Fine-Tuning Tutorial

Section 1

Fine-tuning Overview

Before starting fine-tuning, understand what fine-tuning is, when it's needed, and the types of fine-tuning.

What is Model Fine-tuning?

Model fine-tuning refers to additional training of a pre-trained large language model on domain-specific data to improve its performance on specific tasks. DeepSeek models already have strong general capabilities, but if you need them to be more professional in vertical domains such as medical diagnosis, legal consultation, financial analysis, code review, etc., fine-tuning is the most effective approach.

When to Fine-tune vs. When to Use Prompt Engineering?

This is a common question among developers. Here are the criteria:

Scenarios Recommended for Fine-tuning

Need the model to master domain-specific knowledge (medicine, law, finance, etc.)
Need specific output formats (JSON structure, specific templates)
Need to mimic a specific writing style or tone
Prompts are too long to fully describe requirements in context
Need to reduce inference cost (fine-tuned smaller models can replace larger ones)
Need to reduce latency (no complex prompts after fine-tuning)

Scenarios Recommended for Prompt Engineering

Task is simple and can be described in a few sentences
Insufficient data (fewer than 100 high-quality samples)
Requirements change frequently, don't want to retrain repeatedly
Only need it temporarily, no long-term maintenance
Worried about degradation of other capabilities due to fine-tuning

Main Types of Fine-tuning

Most Recommended

LoRA Fine-tuning

Low-Rank Adaptation adds small parameter matrices alongside the original model, training only these new parameters. Low VRAM requirement, fast training speed, and can fine-tune a 7B model on a single GPU. It is the most mainstream and recommended fine-tuning method.

VRAM requirement: ~16GB for 7B model
Training speed: Fast (hours)
Model quality: Close to full fine-tuning
Can be merged back into the original model

VRAM-Friendly

QLoRA Fine-tuning

Adds 4-bit quantization on top of LoRA, compressing model parameters to 4-bit precision. Further reduces VRAM requirements, allowing fine-tuning of 7B models on consumer GPUs (e.g., RTX 3090 24GB). Slightly slower than LoRA but with comparable results.

VRAM requirement: ~8-10GB for 7B model
Training speed: Medium (hours)
Model quality: Comparable to LoRA
Suitable for consumer GPUs

Highest Quality

Full Fine-tuning

Updates all model parameters, theoretically the best results, but requires large GPU memory. A 7B model requires about 60-80GB VRAM, typically requiring multi-GPU training or DeepSpeed ZeRO optimization. Suitable for teams with ample compute resources.

VRAM requirement: ~60-80GB for 7B model
Training speed: Slow (days)
Model quality: Optimal
Requires multi-GPU/A100

Special Advantages of Fine-tuning DeepSeek Models

DeepSeek models use the MoE (Mixture of Experts) architecture, offering the following advantages for fine-tuning:

Dense models (e.g., DeepSeek-Coder-6.7B): Standard architecture, well-supported for LoRA/QLoRA, rich community resources
MoE models (e.g., DeepSeek-V2-Lite): Only activate a subset of experts per inference, high inference efficiency, can target specific experts during fine-tuning
Official code support: DeepSeek's official GitHub repository provides complete fine-tuning code, ready to use
MIT open-source license: Fine-tuned models can be used commercially without restrictions

Section 2

Environment Preparation

Before starting fine-tuning, you need to install the necessary Python dependencies and configure the GPU environment.

Hardware Requirements

Fine-tuning Method	1.5B Model	7B Model	16B Model	Recommended GPU
QLoRA (4-bit)	~4GB	~8-10GB	~16-20GB	RTX 3060 12GB or higher
LoRA (FP16/BF16)	~8GB	~16-20GB	~32-40GB	RTX 3090/4090 24GB
Full Parameter Fine-tuning	~20GB	~60-80GB	~120-160GB	A100 80GB / Multi-GPU

Install Dependencies

Create a new Python virtual environment and install the following dependencies:

# Create virtual environment
python -m venv deepseek-finetune
# Activate on Windows
deepseek-finetune\Scripts\activate
# Activate on Linux/Mac
# source deepseek-finetune/bin/activate

# Install PyTorch (choose based on your CUDA version)
# CUDA 12.1
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
# CUDA 11.8
# pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# Install core fine-tuning libraries
pip install transformers datasets accelerate peft bitsandbytes
pip install trl  # Transformer Reinforcement Learning, provides SFTTrainer
pip install wandb tensorboard  # Training monitoring
pip install deepspeed  # Required for full parameter fine-tuning (optional)

Verify Environment

After installation, run the following Python script to verify the environment is working:

import torch
import transformers
import peft
import datasets

print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU count: {torch.cuda.device_count()}")
print(f"Transformers version: {transformers.__version__}")
print(f"PEFT version: {peft.__version__}")
print(f"Datasets version: {datasets.__version__}")

if torch.cuda.is_available():
    print(f"Current GPU: {torch.cuda.get_device_name(0)}")
    print(f"Total VRAM: {torch.cuda.get_device_properties(0).total_mem / 1024**3:.1f} GB")
else:
    print("Warning: No GPU detected, training will use CPU (extremely slow)")

Environment Notes

bitsandbytes installation: Windows users may need to compile from source; WSL2 or Linux environment is recommended
CUDA version matching: PyTorch's CUDA version must be compatible with the system-installed CUDA driver version
Cloud GPU recommendation: If you don't have a local GPU, we recommend domestic GPU cloud platforms such as AutoDL and Hengyuan Cloud

Section 3

Dataset Preparation

High-quality data is the key to successful fine-tuning. This section introduces data formats, data quality requirements, and best practices for data preparation.

Data Format

Fine-tuning data typically uses JSON or JSONL format. Each data entry contains an instruction and the corresponding output, with an optional input field.

# data.jsonl — one JSON object per line
{"instruction": "Translate the following Chinese into English", "input": "人工智能正在改变世界", "output": "Artificial intelligence is changing the world."}
{"instruction": "Explain the function of the following Python code", "input": "def foo(x): return x**2", "output": "This function takes a parameter x and returns the square of x."}
{"instruction": "Write a five-character quatrain about spring", "input": "", "output": "春风拂柳绿，\n细雨润花红。\n燕子归巢急，\n人间四月同。"}

Alpaca Format (Recommended)

The Alpaca format is the most commonly used data format in the fine-tuning community and has the best compatibility with DeepSeek models:

[
  {
    "instruction": "You are a professional Python programming assistant. Please answer the following question.",
    "input": "How to implement the singleton pattern in Python?",
    "output": "There are several common methods to implement the singleton pattern in Python:\n\n1. Using the __new__ method:\n```python\nclass Singleton:\n    _instance = None\n    def __new__(cls, *args, **kwargs):\n        if cls._instance is None:\n            cls._instance = super().__new__(cls)\n        return cls._instance\n```\n\n2. Using a decorator:\n```python\ndef singleton(cls):\n    instances = {}\n    def wrapper(*args, **kwargs):\n        if cls not in instances:\n            instances[cls] = cls(*args, **kwargs)\n        return instances[cls]\n    return wrapper\n```\n\nThe first method is recommended because it is more Pythonic."
  }
]

Chat Format (Conversation Data)

If you have conversation data, you can use the ShareGPT format or the conversations format:

{
  "conversations": [
    {"role": "system", "content": "You are a professional medical consultation assistant."},
    {"role": "user", "content": "I've been having headaches lately, what should I do?"},
    {"role": "assistant", "content": "There are many causes of headaches. It is recommended that you... (specific content omitted here)"}
  ]
}

Data Quality Recommendations

Quantity

Minimum 100-200 high-quality samples
Recommended 500-2000 samples
Simple tasks: 500 samples is enough
Complex tasks: recommend 2000+ samples

Diversity

Cover different difficulty levels and scenarios
Avoid duplicate or highly similar samples
Mix long and short texts
Include edge cases

Accuracy

Output must be accurate
Code examples must be runnable
Professional domain knowledge needs verification
Keep format consistent

Consistency

Unified output format
Consistent tone and style
Consistent terminology
Unified punctuation standards

Data quality matters more than quantity

500 high-quality data points are far more effective than 5000 low-quality ones. Each sample should be manually reviewed to ensure accurate output and proper formatting. If data quality is poor, fine-tuning can actually damage the model's original capabilities.

Using DeepSeek to Generate Training Data

You can leverage DeepSeek itself to generate high-quality training data. Here is a practical data generation strategy:

# Generate training data using DeepSeek API
from openai import OpenAI
import json

client = OpenAI(
    api_key="sk-your-api-key",
    base_url="https://api.deepseek.com",
)

# Let DeepSeek V3 generate training data
prompt = """Please generate 50 Python programming Q&A pairs in JSON array format.
Each contains instruction (question) and output (detailed answer).
Cover these topics: decorators, async programming, type hints, context managers, generators.
{"instruction": "...", "output": "..."}"""

response = client.chat.completions.create(
    model="deepseek-chat",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.8,
)

# Parse and save
data = json.loads(response.choices[0].message.content)
with open('training_data.json', 'w', encoding='utf-8') as f:
json.dump(data, f, ensure_ascii=False, indent=2)

For more DeepSeek API usage, please refer to DeepSeek Usage Tutorial.

Section 4

LoRA Fine-tuning (Recommended)

LoRA is currently the most recommended and practical fine-tuning method. This section provides complete QLoRA fine-tuning code and detailed parameter explanations.

LoRA Fine-tuning Principle

The core idea of LoRA (Low-Rank Adaptation) is to add low-rank matrices alongside the weights of a pre-trained model, training only these newly added matrix parameters while keeping the original model weights unchanged. This significantly reduces the number of trainable parameters and memory requirements.

Specifically, for the original weight matrix W (d x k), LoRA decomposes its update into the product of two small matrices: W + delta_W = W + B * A, where A is an r x k matrix, B is a d x r matrix, and r is the rank, typically 8-64. Since r is much smaller than d and k, the number of trainable parameters is greatly reduced.

Complete QLoRA Fine-tuning Code

The following is a complete Python script for fine-tuning a DeepSeek model using QLoRA (4-bit quantization + LoRA):

# train_lora.py — QLoRA fine-tuning of DeepSeek model
import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments,
    BitsAndBytesConfig,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

# ========== 1. Configuration ==========
model_name = "deepseek-ai/deepseek-coder-6.7b-instruct"
dataset_path = "./data/training_data.json"
output_dir = "./output/deepseek-lora"

# ========== 2. 4-bit quantization config ==========
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

# ========== 3. Load model and tokenizer ==========
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)

tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    trust_remote_code=True,
)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

# ========== 4. Prepare model for k-bit training ==========
model = prepare_model_for_kbit_training(model)

# ========== 5. LoRA configuration ==========
lora_config = LoraConfig(
    r=16,                      # LoRA rank, larger means stronger expressiveness but more parameters
    lora_alpha=32,               # LoRA scaling factor, usually set to 2x r
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
lora_dropout=0.05,          # Dropout to prevent overfitting
    bias="none",                # Do not train bias
    task_type="CAUSAL_LM",        # Causal language model task
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

# ========== 6. Load dataset ==========
dataset = load_dataset("json", data_files=dataset_path, split="train")

def format_instruction(sample):
    """Format data into training text"""
    if sample.get("input"):
        text = f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{sample['instruction']}

### Input:
{sample['input']}

### Response:
{sample['output']}"""
    else:
        text = f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{sample['instruction']}

### Response:
{sample['output']}"""
    return {"text": text}

dataset = dataset.map(format_instruction)

# ========== 7. Training arguments ==========
training_args = TrainingArguments(
    output_dir=output_dir,
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    warmup_steps=100,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    save_steps=200,
    save_total_limit=3,
    optim="paged_adamw_8bit",
    lr_scheduler_type="cosine",
    report_to="tensorboard",
)

# ========== 8. Create Trainer and start training ==========
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
    max_seq_length=2048,
    dataset_text_field="text",
)

trainer.train()

# ========== 9. Save model ==========
trainer.model.save_pretrained(output_dir)
tokenizer.save_pretrained(output_dir)

Key Parameter Explanations

Parameter	Recommended Value	Description
r (rank)	8-64	The larger the rank, the stronger the expressiveness, but the more training parameters and GPU memory are required. For simple tasks, r=8 is sufficient; for complex tasks, r=32-64 can be used.
lora_alpha	2 times r	Scaling factor that controls the influence of LoRA weights on the original model. Usually set to 2 times r.
learning_rate	1e-4 ~ 5e-4	Learning rate. LoRA typically uses a higher learning rate than full fine-tuning; 2e-4 is recommended.
num_train_epochs	2-5	Number of training epochs. If the dataset is small, you can increase the number of epochs; if large, 2-3 epochs suffice. Watch the loss curve to prevent overfitting.
max_seq_length	2048-4096	Maximum sequence length. Longer texts require more GPU memory. DeepSeek-Coder supports 16K context.

Running Training

Save the above code as train_lora.py and run:

python train_lora.py

Training Acceleration Tips

Use gradient_checkpointing: Trade computation for memory; set gradient_checkpointing=True in TrainingArguments.
Adjust batch_size: If GPU memory is insufficient, reduce per_device_train_batch_size and increase gradient_accumulation_steps.
Use Flash Attention 2: Install the flash-attn library and set use_flash_attention_2=True when loading the model; this can speed up training by 2-3x.

For more model download and deployment information, see DeepSeek Model Download and DeepSeek Deployment Tutorial.

Section 5

Full Parameter Fine-tuning

Full parameter fine-tuning theoretically yields the best results, but requires a large amount of GPU memory. This section introduces how to perform full parameter fine-tuning using DeepSpeed ZeRO.

Requirements for Full Parameter Fine-tuning

Full parameter fine-tuning updates all parameters of the model, requiring the entire model to be loaded into GPU memory. For a 7B model, just the model weights require about 14GB (FP16), plus optimizer states and gradients, totaling about 60-80GB of GPU memory. Therefore, multi-GPU training or GPUs with large memory such as A100/H100 are typically needed.

DeepSpeed ZeRO Configuration

DeepSpeed ZeRO distributes optimizer states, gradients, and model parameters across multiple GPUs, significantly reducing the memory requirement per GPU. Below is the DeepSpeed configuration for full parameter fine-tuning of DeepSeek models:

# ds_config_zero3.json — DeepSpeed ZeRO Stage 3 configuration
{
  "bf16": {
    "enabled": true
  },
  "zero_optimization": {
    "stage": 3,
    "offload_optimizer": {
      "device": "cpu",
      "pin_memory": true
    },
    "offload_param": {
      "device": "cpu",
      "pin_memory": true
    },
    "overlap_comm": true,
    "contiguous_gradients": true,
    "sub_group_size": 1e9,
    "reduce_bucket_size": "auto",
    "stage3_prefetch_bucket_size": "auto",
    "stage3_param_persistence_threshold": "auto",
    "stage3_max_live_parameters": 1e9,
    "stage3_max_reuse_distance": 1e9,
    "stage3_gather_16bit_weights_on_model_save": true
  },
  "gradient_accumulation_steps": "auto",
  "gradient_clipping": "auto",
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "wall_clock_breakdown": false
}

Full Parameter Fine-tuning Training Script

# train_full.py — DeepSeek full parameter fine-tuning
import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments,
    Trainer,
    DataCollatorForLanguageModeling,
)

model_name = "deepseek-ai/deepseek-coder-6.7b-instruct"
dataset_path = "./data/training_data.json"
output_dir = "./output/deepseek-full"

# Load model (without quantization)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    use_flash_attention_2=True,  # Requires flash-attn installation
)

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

# Load and process dataset
dataset = load_dataset("json", data_files=dataset_path, split="train")

def tokenize_function(examples):
    texts = [f"### Instruction:\n{inst}\n\n### Response:\n{out}"
             for inst, out in zip(examples["instruction"], examples["output"])]
    return tokenizer(texts, truncation=True, max_length=2048)

tokenized_dataset = dataset.map(tokenize_function, batched=True, remove_columns=dataset.column_names)

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=False,
)

training_args = TrainingArguments(
    output_dir=output_dir,
    overwrite_output_dir=True,
    num_train_epochs=2,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    warmup_ratio=0.03,
    learning_rate=5e-5,
    bf16=True,
    logging_steps=10,
    save_steps=200,
    save_total_limit=2,
    deepspeed="./ds_config_zero3.json",
    lr_scheduler_type="cosine",
    report_to="tensorboard",
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
    tokenizer=tokenizer,
    data_collator=data_collator,
)

trainer.train()
trainer.save_model(output_dir)
tokenizer.save_pretrained(output_dir)

Start Multi-GPU Training

# Train on 4 GPUs using DeepSpeed
deepspeed --num_gpus=4 train_full.py

Full Parameter Fine-tuning vs LoRA Selection Recommendations

Prefer LoRA: In most scenarios, LoRA's performance is very close to full fine-tuning (typically < 2% difference), but at a much lower cost
Full fine-tuning is suitable when: You need to significantly change model behavior in a new domain, have ample computational resources, and pursue ultimate performance
Hybrid strategy: Start with LoRA for quick experiments to find optimal data and hyperparameters, then use full fine-tuning for the final version

Section 6

Using DeepSeek Official Fine-Tuning Code

DeepSeek provides complete fine-tuning code on GitHub, which can be used directly without writing training scripts from scratch.

Get Official Fine-Tuning Code

The official DeepSeek GitHub repository contains all the code needed for training and fine-tuning. Take DeepSeek-Coder as an example:

# Clone the DeepSeek-Coder repository
git clone https://github.com/deepseek-ai/DeepSeek-Coder.git
cd DeepSeek-Coder

# Fine-tuning code is in the finetune/ directory
ls finetune/
# Output: deepseek_coder_finetune.py  data/  README.md

# Install official dependencies
pip install -r finetune/requirements.txt

Official Fine-Tuning Commands

DeepSeek provides a simple command-line interface that supports LoRA and full-parameter fine-tuning:

# LoRA fine-tuning of DeepSeek-Coder-6.7B
python finetune/deepseek_coder_finetune.py \
    --model_name_or_path deepseek-ai/deepseek-coder-6.7b-instruct \
    --data_path ./data/training_data.jsonl \
    --output_dir ./output/deepseek-coder-lora \
    --num_train_epochs 3 \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 4 \
    --learning_rate 2e-4 \
    --lora_r 16 \
    --lora_alpha 32 \
    --use_lora True \
    --bf16 True

Official Data Format Requirements
    The official DeepSeek fine-tuning code expects data in JSONL format, with each line containing instruction and output fields:
# training_data.jsonl
{"instruction": "Write a Python function to compute the greatest common divisor of two numbers", "output": "def gcd(a, b):\n    while b:\n        a, b = b, a % b\n    return a\n\n# You can also use math.gcd\n# from math import gcd"}
{"instruction": "Explain what recursion is", "output": "Recursion is a programming technique where a function solves a problem by calling itself. Recursion has two key parts:\n\n1. Base Case: The condition that stops recursion\n2. Recursive Case: The condition where the function calls itself\n\nFor example, computing factorial n! = n * (n-1)!, stops when n=1."}
Advantages of Official Code
Optimized for DeepSeek model architecture, higher training efficiency
Automatically handles DeepSeek-specific tokenizer and model configuration
Built-in data processing and evaluation logic
Continuously updated, compatible with the latest model versions
For more DeepSeek ecosystem tools, please see DeepSeek Ecosystem Tools.

Section 7

Training Monitoring

Monitor loss curves, learning rate changes, and GPU usage in real-time during training to help you detect anomalies and adjust parameters promptly.

TensorBoard Monitoring

TensorBoard is PyTorch's built-in training visualization tool. After setting report_to="tensorboard" in your training code, launch TensorBoard:

# Start TensorBoard
tensorboard --logdir ./output --port 6006

# Open http://localhost:6006 in your browser

Wandb Monitoring

Wandb (Weights & Biases) provides richer cloud-based training monitoring features and supports team collaboration:

# Install wandb
pip install wandb

# Login (API Key required on first use)
wandb login

# Set in training code (or via environment variables)
# Set report_to="wandb" in TrainingArguments
# Or set environment variables:
# export WANDB_PROJECT="deepseek-finetune"

Key Monitoring Metrics

Loss Curve

Training loss should continuously decrease
If loss doesn't decrease, check learning rate
If loss oscillates violently, reduce learning rate
If loss plateaus after rapid decline, it's normal

GPU Utilization

GPU utilization should be close to 90-100%
Low utilization = data loading bottleneck
Increase num_workers to speed up data loading
Use nvidia-smi to view in real-time

Learning Rate

Use cosine scheduler for automatic decay
Gradually increase learning rate during warmup
If loss doesn't decrease, try increasing learning rate
If loss oscillates, try decreasing learning rate

Checkpoint

Save checkpoints periodically
save_total_limit controls the number of saved checkpoints
Training can resume from a checkpoint after interruption
Keep the checkpoint with the lowest loss

Resume Training from Checkpoint

If training is unexpectedly interrupted, you can resume from the most recent checkpoint:

# Add in TrainingArguments

resume_from_checkpoint=True

# Or specify a specific checkpoint path
resume_from_checkpoint="./output/deepseek-lora/checkpoint-600"

Overfitting Warning Signs

If training loss continues to decrease but validation loss starts to increase, the model is overfitting. In this case, you should: reduce the number of training epochs, increase the amount of data, increase lora_dropout, or use a smaller learning_rate.

Section 8

Model Evaluation

After fine-tuning, it is necessary to evaluate the model's performance on the target task to ensure that the fine-tuning has achieved the expected results.

Evaluation Methods

Model evaluation is divided into two methods: automatic evaluation and manual evaluation.

Automatic Evaluation

Quantitative Metrics

Perplexity
ROUGE / BLEU scores
Accuracy / F1 score
HumanEval (code tasks)
MMLU (knowledge Q&A)

Manual Evaluation

Qualitative Evaluation

Output quality scoring
Instruction following
Answer accuracy
Style consistency
A/B comparison testing

Test Set Evaluation Script

# evaluate.py — Evaluate the fine-tuned model
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model and LoRA weights
base_model = AutoModelForCausalLM.from_pretrained(
    "deepseek-ai/deepseek-coder-6.7b-instruct",
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "deepseek-ai/deepseek-coder-6.7b-instruct",
    trust_remote_code=True,
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "./output/deepseek-lora")

# Test inference
test_questions = [
    "Write a Python function to implement binary search algorithm",
    "Explain what GIL is in Python",
    "How to use asyncio to implement concurrent requests",
]

for question in test_questions:
    prompt = f"### Instruction:\n{question}\n\n### Response:\n"
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.7,
        do_sample=True,
        top_p=0.9,
    )
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    print(f"Question: {question}\nAnswer: {response}\n{'='*60}")

Benchmark Comparison

It is recommended to compare the outputs of the model before and after fine-tuning on the same test set to evaluate the improvements brought by fine-tuning:

Horizontal comparison: Fine-tuned model vs. original model vs. other similar models
Longitudinal comparison: Differences in performance between different checkpoints
Ablation experiments: Comparison of effects of different hyperparameter combinations
Boundary testing: Test the model's performance under extreme inputs

Evaluation Recommendations

Don't just look at automatic evaluation metrics. Be sure to test the model in real-world scenarios, especially edge cases and error-prone situations. If the fine-tuned model degrades significantly in general capabilities, consider retraining with a mix of original data and fine-tuning data.

Section 9

Model Merging and Export

After fine-tuning, you need to merge the LoRA weights back into the original model and export it in different formats for deployment.

Merging LoRA Weights

LoRA training produces adapter weights, which need to be merged back into the original model to be used as a standalone model:

# merge_lora.py — Merge LoRA weights into the original model
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_name = "deepseek-ai/deepseek-coder-6.7b-instruct"
lora_path = "./output/deepseek-lora"
merged_path = "./output/deepseek-coder-merged"

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.float16,
    trust_remote_code=True,
)

# Load and merge LoRA weights
model = PeftModel.from_pretrained(base_model, lora_path)
model = model.merge_and_unload()

# Save merged model
model.save_pretrained(merged_path)

tokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)
tokenizer.save_pretrained(merged_path)

print(f"Model merged and saved to: {merged_path}")

Export to GGUF Format (Ollama / llama.cpp)

GGUF is the model format used by llama.cpp and Ollama, allowing efficient local execution after export:

# Use llama.cpp's convert_hf_to_gguf.py for conversion
# Clone llama.cpp
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

# Install dependencies
pip install -r requirements.txt

# Convert model to GGUF format
python convert_hf_to_gguf.py ../output/deepseek-coder-merged \
    --outfile ../output/deepseek-coder-finetuned.gguf \
    --outtype f16

# Quantize model (optional, to reduce size)
# ./build/bin/llama-quantize ../output/deepseek-coder-finetuned.gguf Q4_K_M

Creating an Ollama Modelfile

After exporting GGUF, create a Modelfile to use it in Ollama:

# Modelfile
FROM ./deepseek-coder-finetuned.gguf

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 16384
SYSTEM """You are a professional Python programming assistant, optimized through fine-tuning."""

# Create Ollama model
# ollama create deepseek-coder-custom -f Modelfile

# Run
# ollama run deepseek-coder-custom

Push to Hugging Face

Share the fine-tuned model with the Hugging Face community:

# push_to_hub.py
from transformers import AutoModelForCausalLM, AutoTokenizer
from huggingface_hub import login

# Login to Hugging Face (requires Access Token)
login(token="hf_your_access_token")

model = AutoModelForCausalLM.from_pretrained(
    "./output/deepseek-coder-merged",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "./output/deepseek-coder-merged",
    trust_remote_code=True,
)

repo_id = "your-username/deepseek-coder-finetuned"
model.push_to_hub(repo_id, private=True)
tokenizer.push_to_hub(repo_id, private=True)

print(f"Model pushed to: https://huggingface.co/{repo_id}")

For more deployment options, see the DeepSeek Deployment Tutorial.

Section 10

Deployment

The final step after fine-tuning: deploy the model to production and provide API services.

Deployment Options Comparison

Simplest

Ollama Local Deployment

Suitable for personal use and small teams, one-click deployment of the fine-tuned model. After exporting to GGUF format, run it via Ollama, supporting API calls.

Start with one command
Compatible with OpenAI API format
Low resource usage
Suitable for single-machine deployment

High Performance

vLLM Deployment

Suitable for production environments, high-performance inference engine. Supports PagedAttention, continuous batching, multi-GPU parallel inference, with throughput far exceeding Ollama.

High throughput
Compatible with OpenAI API
Supports multiple GPUs
Production-grade stability

Cloud

Hugging Face Inference

After pushing the model to Hugging Face, you can directly use Inference Endpoints for deployment without managing servers.

No maintenance
Automatic scaling
Pay-as-you-go
Global nodes

vLLM Deployment Commands

vLLM is currently the most recommended high-performance inference engine for production environments:

# Install vLLM
pip install vllm

# Start vLLM API service
python -m vllm.entrypoints.openai.api_server \
    --model ./output/deepseek-coder-merged \
    --served-model-name deepseek-coder-custom \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.95 \
    --trust-remote-code

# API call (fully compatible with OpenAI)
# curl http://localhost:8000/v1/chat/completions \
#   -H "Content-Type: application/json" \
#   -d '{"model":"deepseek-coder-custom","messages":[{"role":"user","content":"Hello"}]}'

Docker Deployment

Using Docker deployment ensures environment consistency, making it easy to migrate and scale:

# Dockerfile
FROM vllm/vllm-openai:latest

COPY ./output/deepseek-coder-merged /models/deepseek-coder-custom

ENV VLLM_MODEL=/models/deepseek-coder-custom
ENV VLLM_SERVED_MODEL_NAME=deepseek-coder-custom
ENV VLLM_MAX_MODEL_LEN=16384

EXPOSE 8000

# Build and run
# docker build -t deepseek-coder-custom .
# docker run --gpus all -p 8000:8000 deepseek-coder-custom

API Call Example (Python)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",
)

response = client.chat.completions.create(
    model="deepseek-coder-custom",
    messages=[
        {"role": "system", "content": "You are a professional Python programming assistant."},
        {"role": "user", "content": "Write a quicksort implementation"},
    ],
    temperature=0.7,
    max_tokens=512,
)

print(response.choices[0].message.content)

Production Environment Considerations

Health Check: Configure the /health endpoint to ensure service availability monitoring
Rate Limiting: Set request frequency limits via Nginx or API gateway
Concurrency Control: vLLM supports the max_num_seqs parameter to control the number of concurrent requests
Logging: Log all API requests and responses for troubleshooting
Model Hot Update: Use blue-green deployment strategy for zero-downtime model updates

For more deployment details, see DeepSeek Deployment Tutorial and DeepSeek Usage Tutorial.

FAQ

DeepSeek Model Fine-tuning FAQ

How much data is needed for fine-tuning DeepSeek models? +

At least 100-200 high-quality samples are needed, with 500-2000 recommended. Data quality matters more than quantity. For simple tasks (like format conversion), 500 high-quality samples can yield noticeable results. For complex tasks (like professional domain Q&A), it's recommended to prepare over 2000 samples. If data is insufficient, you can first use DeepSeek to generate initial data, then manually review and modify it.

Which is better: LoRA or full fine-tuning? +

For most scenarios, LoRA is recommended. LoRA's performance is very close to full fine-tuning (usually < 2% difference), but at a much lower cost (GPU memory requirement reduced by 60-80%). Full fine-tuning is suitable for scenarios where you need to significantly change model behavior in a new domain, and requires sufficient computational resources. It's recommended to start with LoRA for quick experiments to find optimal data and hyperparameters, then decide if full fine-tuning is necessary.

Can I fine-tune DeepSeek models on a regular computer? +

Yes. Using QLoRA (4-bit quantization + LoRA), a consumer GPU with 8GB+ VRAM (like RTX 3060 12GB) can fine-tune a 7B model. If you only have CPU or lower specs, consider cloud GPU services (like AutoDL) that rent GPUs such as A100 by the hour at low cost. Alternatively, you can fine-tune smaller model versions (like 1.5B) which only require 4GB VRAM.

Will the model lose its original capabilities after fine-tuning? +

It's possible. This is called "Catastrophic Forgetting". Prevention methods: 1) Don't over-train (avoid too many epochs); 2) Mix original general data with fine-tuning data during training; 3) Use a smaller learning rate; 4) Regularly evaluate general capabilities during training. If degradation is observed, reduce the number of training epochs or increase the proportion of general data.

Can DeepSeek MoE models be fine-tuned? +

Yes. MoE models like DeepSeek V2/V3 also support LoRA fine-tuning. The advantage of the MoE architecture is that only a subset of experts is activated during each inference, allowing targeted optimization of specific experts during fine-tuning. However, MoE models have a large number of parameters, making full fine-tuning very demanding. It's recommended to use LoRA and ensure that target_modules includes the MoE-specific gate layer. The official DeepSeek GitHub repository provides fine-tuning examples for MoE models.

Can fine-tuned models be used commercially? +

Yes. DeepSeek models are released under the MIT open-source license, so fine-tuned models can be used commercially without additional authorization. However, note that: 1) The training data itself must not contain infringing content; 2) If you use datasets annotated by others, you must comply with the dataset's license; 3) It's recommended to note in your release that the model is fine-tuned based on DeepSeek.

More Resources

DeepSeek Complete Tutorial Index

Here are all our DeepSeek-related tutorials, from beginner to advanced, for one-stop learning.

Beginner Tutorial

How to Use DeepSeek Models

Zero-basics tutorial, four ways to use DeepSeek models with step-by-step guidance.

View Tutorial

Model Download

DeepSeek Model Download

One-click download via Ollama, original weights from Hugging Face, official GitHub repository.

View Download

Model Details

DeepSeek Open-Source Model List

All open-source model specs, parameters, download links, and use cases at a glance.

View List

Deployment Tutorial

DeepSeek Deployment Tutorial

Local deployment with Ollama, Docker containerization, vLLM high-performance inference.

View Deployment

Architecture Details

DeepSeek Model Architecture

In-depth analysis of MoE architecture, MLA attention, multi-head latent attention, and more.

View Architecture

Ecosystem Tools

DeepSeek Ecosystem Tools

A collection of third-party tools, plugins, and integration solutions around DeepSeek.

View Tools