The Art of Building Fine-Tuning Datasets

Data is the Soul of Fine-tuning

In the field of machine learning, there is a famous saying: "Garbage in, garbage out." This is especially true in LLM fine-tuning. You can spend a lot of effort on algorithms, hyperparameters, and training techniques, but if the training data quality is poor, your fine-tuned model is doomed to fail. Conversely, 1,000 carefully constructed high-quality data samples can often bring surprising improvements.

Building a fine-tuning dataset is not simply "collect data → format → train." It is an art that combines data engineering, domain knowledge, and pedagogy. You need to answer a series of key questions: How much data is needed? Where to get it? How to ensure quality? How to ensure diversity? How to balance data for different task types? This article will systematically answer these questions.

Data Source Strategies

The sources of fine-tuning data can be divided into several levels: manual annotation (highest quality but highest cost, suitable for seed data in core scenarios), AI-assisted generation (use a stronger model to generate training data, then manually review, the most cost-effective solution), user log mining (filter high-quality interactions from real user conversations, use after anonymization, best reflects real needs), and public dataset curation (collect relevant datasets from platforms like HuggingFace, filter and clean them). Recommended data source ratio: 30% manual annotation (ensure quality baseline) + 50% AI-assisted generation with manual review (balance scale and quality) + 20% user log mining (real scenario coverage).

import json
from openai import OpenAI

client = OpenAI(api_key="your-deepseek-api-key", base_url="https://api.deepseek.com")

class DatasetBuilder:
    def __init__(self):
        self.data = []
        self.quality_checklist = []

    def generate_synthetic(self, task_description, num_samples=100):
        """Generate synthetic training data using AI"""
        prompt = f"""You are a training data generation expert. Please generate {num_samples} high-quality training data samples based on the following task description.

Task description: {task_description}

Requirements:
1. Data should be diverse, covering different scenarios and difficulty levels
2. Each data sample contains instruction, input, and output (expected output)
3. Output must be accurate, complete, and meet task requirements
4. There should be sufficient difference between each data sample

Output in JSONL format, one JSON object per line:
{{"instruction":"...", "input":"...", "output":"..."}}"""
        response = client.chat.completions.create(
            model="deepseek-chat",
            messages=[{"role":"user","content":prompt}],
            temperature=0.8, max_tokens=4000
        )
        return response.choices[0].message.content

    def check_quality(self, sample):
        """AI-assisted quality check"""
        prompt = f"""Evaluate the quality of the following training data.

Data:
Instruction: {sample['instruction']}
Input: {sample['input']}
Output: {sample['output']}

Please check:
1. Is the instruction clear and unambiguous?
2. Do the input and output match?
3. Is the output accurate and complete?
4. Are there any factual errors?
5. Is the data suitable for fine-tuning?

Output in JSON format:
{{"quality_score": 1-10, "issues":[], "pass": true/false}}"""
        response = client.chat.completions.create(
            model="deepseek-chat",
            messages=[{"role":"user","content":prompt}],
            temperature=0.1
        )
        return json.loads(response.choices[0].message.content)

    def deduplicate(self, data, threshold=0.85):
        """Deduplication based on semantic similarity"""
        unique = []
        for item in data:
            is_dup = False
            for existing in unique:
                # Simplified: compare character similarity of instruction+input
                text1 = item["instruction"] + item.get("input","")
                text2 = existing["instruction"] + existing.get("input","")
                if len(set(text1) & set(text2)) / max(len(set(text1)), len(set(text2)), 1) > threshold:
                    is_dup = True
                    break
            if not is_dup:
                unique.append(item)
        return unique

    def balance_categories(self, data, category_field="category"):
        """Balance data categories"""
        from collections import Counter
        counts = Counter(item.get(category_field, "unknown") for item in data)
        min_count = min(counts.values())
        balanced = []
        from collections import defaultdict
        by_cat = defaultdict(list)
        for item in data:
            by_cat[item.get(category_field, "unknown")].append(item)
        import random
        for cat, items in by_cat.items():
            balanced.extend(random.sample(items, min(min_count, len(items))))
        return balanced

builder = DatasetBuilder()
# synthetic = builder.generate_synthetic("Customer service dialogue intent recognition: determine whether the user wants to consult, complain, or request a refund", 50)
# quality = builder.check_quality(sample_data)
# cleaned = builder.deduplicate(raw_data)
# balanced = builder.balance_categories(cleaned)

Three Lines of Defense for Quality Control

First layer: automatic rule checking. Check whether the data format is correct, whether output is empty, whether instruction is too short (<5 characters) or too long (>500 characters), and whether input and output are highly repetitive (possibly copy-paste errors). Second layer: AI-assisted quality scoring. Use a stronger model to score each data sample (1-10), and manually review or directly discard data with scores below 6. Third layer: manual sampling review. Randomly select 10-20 samples from each category for manual review by domain experts, and calculate the data pass rate. The coverage of the three layers of quality control should be: automatic check 100% → AI scoring 100% → manual review 10-20%.

The Art of Data Ratio

Data ratio directly affects the final performance of the model. Common strategies: balanced task ratio (roughly equal amounts of data for various task types, suitable for general-purpose fine-tuning), key task emphasis (core business scenario data accounts for 50%-70% to ensure core capabilities), progressive difficulty (gradually increase data complexity from simple to difficult, allowing the model to learn smoothly), and natural distribution ratio (proportion data according to actual online traffic distribution to ensure the fine-tuned model performs best in real scenarios). It is recommended to first use balanced ratio for baseline experiments, then adjust the ratio based on evaluation results.

Data Privacy and Compliance Considerations

When building fine-tuning datasets, data privacy and compliance are issues that cannot be ignored. Pay special attention to the following: anonymization of user data - training data extracted from real conversations must remove all personally identifiable information (PII), including names, phone numbers, ID numbers, addresses, bank card numbers, etc. Use a combination of regex matching and NER model for double detection to ensure completeness of anonymization. Legality of data sources - when using public datasets, confirm that their license agreements allow commercial use; when using AI-generated data, note whether the original model's user agreement allows using its output to train other models. Data storage security - training data should be stored in encrypted storage systems, access permissions strictly controlled, and data access logs regularly audited. Data lifecycle management - clearly define the retention period for training data, delete expired data promptly, and comply with GDPR's right to be forgotten. Data version management: Building a fine-tuning dataset is an iterative process - you will continuously add new data, correct erroneous data, and adjust data ratios. It is recommended to manage dataset versions like code: use Git LFS to store data files, commit each change with a meaningful commit message; use DVC (Data Version Control) for data version management, which can track data lineage - "which version of the model was trained on which version of the data"; retain metadata of data construction - data source, construction time, constructor, quality score distribution, etc.

Advanced Techniques for Synthetic Data

When using AI to generate synthetic training data, there are some advanced techniques that can significantly improve generation quality: diverse system prompts - don't always use the same prompt to generate data; vary the angle and constraints of generation instructions to let AI generate data from different perspectives; self-play - let the same model play both teacher and student, the teacher generates questions, the student answers, and the teacher scores and corrects, producing high-quality data; knowledge distillation - use a stronger model (e.g., DeepSeek-R1) to generate training data to fine-tune a smaller model (e.g., DeepSeek-V2-Lite), achieving capability transfer; data augmentation - perform transformations on existing data such as synonym rewriting, question inversion, and condition addition/removal to increase dataset diversity. Combining these techniques can help you obtain high-quality, large-scale fine-tuning datasets at a lower cost.