Model Distillation in Practice: Training Small Models with Large Models

import numpy as np
import torch
from transformers import AutoModel

# Load teacher embeddings (already generated offline via DeepSeek API)
teacher_embeds = np.load('teacher_embeds.npy')
# Initialize student model
student = AutoModel.from_pretrained('bert-base-uncased', config='config/student.json')
# Copy the first 768 dimensions of teacher embeddings to student embeddings (assuming student embedding dimension is also 768)
with torch.no_grad():
    student.embeddings.word_embeddings.weight.data[:len(teacher_embeds)] = \
        torch.tensor(teacher_embeds, dtype=torch.float32)
    # Note: Ensure vocabulary alignment, otherwise mapping is needed
print('Embedding initialized.')

Section 4: Loss Function Design – More Than Just KD Loss

The standard distillation loss is L = alpha * KL(soft_loss) + (1-alpha) * CE(hard_loss). However, in practice, we found that with only these two losses, the student model tends to perform poorly on tail classes. The reason is that the teacher's soft label distribution for tail classes is uneven, providing weak distillation signals. To address this, we introduced an auxiliary contrastive learning loss: for the same batch of samples, we pull closer the representations of the student model for different augmentations (e.g., text perturbation) of the same original sample, and push away representations of other samples. This auxiliary loss helps the student learn more robust features rather than merely mimicking the teacher's output.

Our designed loss function is: L_total = alpha * L_KD + beta * L_CE + gamma * L_contrastive, where alpha=0.7, beta=0.3, gamma=0.1. The sum of alpha and beta need not be 1, but requires tuning. In an intent recognition task, after adding the contrastive loss, the student model's accuracy on hard examples (e.g., "帮我查下明天天气" and "明天适合出门吗") improved by 4%. However, note that the contrastive loss requires constructing positive and negative samples, which increases training time and imposes requirements on batch size (we used a batch of 256).

A detail that is easily overlooked: the temperature for soft labels should be high initially (e.g., T=4) and gradually decrease to T=1 during training. This is similar to Curriculum Learning, where the student first learns rough class relationships from a smooth distribution, then gradually fits precisely. In our training, we multiplied T by 0.95 every 1000 steps, which improved performance by 1.3% compared to a fixed T=3.

Section 5: Engineering Pitfalls – Data Flywheel and Pseudo-Label Noise

The biggest pitfall in distillation is not the model architecture but data iteration. When using the teacher model to generate soft labels, if the teacher itself is not confident about a sample (e.g., it belongs to category "other" rather than our defined categories), the soft label often becomes a flat distribution, providing almost no knowledge. We call this "noisy pseudo-label". If not handled, the student model will be misled by these noisy samples. Our solution: when generating soft labels, also have the teacher output confidence (i.e., the maximum probability value) and filter out samples with confidence below a threshold (e.g., 0.6). However, this reduces data diversity, so we also tried "soft label smoothing" (keeping the flat distribution but raising temperature to 10 to make probabilities more uniform), but this is still not optimal.

A more reliable approach is "dynamic distillation": every 1000 training steps, use the current student model to evaluate a batch of samples, identify samples where the student and teacher disagree the most, and then re-call the API to generate more precise soft labels for these samples (with a lower temperature like T=1). This is somewhat similar to active learning. In one project, using this method, we achieved comparable performance to full distillation with only 30% of the original data. The table below shows key data from one of our distillation experiments:

Section 6: Evaluation and Deployment – Beyond Metrics

Distilled models should not be evaluated solely on offline metrics. We adopted a "dual evaluation" strategy: first, compare the teacher, student, and randomly initialized small model on a standard test set; second, design a human blind evaluation task where 10 annotators score the outputs of 200 random samples (1-5 scale). Results showed that the distilled student model's gap with the teacher on the standard test set was within 3%, but in human evaluation, the student's fluency and logic scores had a gap of 0.8 points (teacher 4.5, student 3.7). This indicates that automated metrics (like F1) cannot fully reflect generation quality. Therefore, we recommend adding targeted "behavior tests" in application scenarios, such as constructing adversarial examples (e.g., typos, colloquial expressions) to observe the model's robustness.

After deployment, we monitored two key engineering metrics: inference latency and cost. The student model (40M parameters) had an average latency of 15ms on GPU, while the teacher model (deepseek-chat) averaged 380ms (including network transmission); in terms of cost, the student model can be deployed on CPU, with a cost of only $0.03 per thousand inferences, while the teacher costs $1.2. These metrics drove the company to fully adopt distilled models instead of direct calls to large models, reserving the large model channel only for necessary scenarios (e.g., complex reasoning).

Section 7: Experience Summary and Advanced Recommendations

Reflecting on our distillation practice, the biggest lesson is "don't trust default parameters". Whether it's temperature, loss function weights, or model architecture, they all need tuning for specific tasks and data distributions. Second, data quality over quantity – after cleaning out low-confidence samples, we reduced data by 10% but improved performance by 0.4%. Finally, distillation is not a one-time process but should be continuously iterated; as large models update, small models should also be updated through distillation.

For advanced players, you can try multi-teacher distillation (i.e., using both deepseek-chat and other large models like GPT-4, fusing their soft labels), and feature distillation (aligning intermediate layer representations in addition to the output layer). Additionally, model compression techniques like quantization (INT8) can further reduce size, but when combined with distillation, be careful about quantization error accumulation; we recommend distilling first, then quantizing.

Finally, here is a complete distillation training script framework (pseudocode) for reference:

# Pseudocode showing distillation training loop
import torch
from transformers import AutoModelForSequenceClassification, AdamW

student = AutoModelForSequenceClassification.from_pretrained('student_config')
teacher_api = DeepSeekAPI('your-deepseek-api-key')

for epoch in range(3):
    for batch in dataloader:
        # Get teacher soft labels (may come from cache or online calls)
        soft_targets =

Conclusion: Distillation is the Essential Path to Democratizing Large Models

In today's era of high computational costs, distillation technology enables small and medium-sized enterprises to also benefit from the intelligence of large models. By using the DeepSeek API as a teacher, we can obtain high-quality soft labels at a low cost, and through meticulous engineering design, train small models that meet production requirements. However, distillation is not a panacea—when tasks require strong common-sense reasoning or creativity, small models still fall short, and in such cases, it may be necessary to retain large models for emergencies. In the future, as model architectures evolve (such as linear attention and sparse activation), the capacity of small models will increase, and the benefits of distillation will further expand. We hope that the practical experience shared in this article can help readers avoid detours and successfully implement distilled models in their own scenarios.