Why Scientific Model Evaluation is Needed
Many teams, after fine-tuning a model, only run a few manual test cases, feel that "the results are much better," and eagerly deploy it to production. However, after deployment, they find that while it improves in some scenarios, it degrades in others; overall user satisfaction does not increase; and some existing users even complain due to changes in model behavior. This is a typical problem caused by the lack of a scientific evaluation system.
A scientific model evaluation system includes two levels: offline evaluation (using a standardized test set before deployment to answer "how does the model perform in a controlled environment?") and online evaluation (using A/B testing after deployment to assess real user feedback and answer "how does the model perform in real-world scenarios?"). Both levels are indispensable—good offline performance does not guarantee good online performance (due to distribution shift), and knowing that online performance is good without understanding why is not conducive to continuous improvement.
Design of Offline Evaluation Metrics
The core of offline evaluation is building a high-quality test set. The test set should: be completely independent of the training set (no overlap), cover all important business scenarios, have sufficient sample size for each scenario (at least 50-100 samples), and have high annotation quality (each sample has a standard answer or reference standard). Evaluation metrics are selected based on task type: classification tasks (intent recognition) → Accuracy, F1, confusion matrix; generation tasks (dialogue, writing) → BLEU/ROUGE (limited reference value) → LLM-as-Judge scoring is more recommended; retrieval tasks → Recall@k, MRR, NDCG.
import json, random, numpy as np
from scipy import stats
from openai import OpenAI
client = OpenAI(api_key="your-deepseek-api-key", base_url="https://api.deepseek.com")
class ModelEvaluator:
def __init__(self):
self.test_set = []
self.results = {"baseline":[], "candidate":[]}
def load_test_set(self, path):
"""Load test set"""
with open(path, 'r', encoding='utf-8') as f:
self.test_set = json.load(f)
print(f"Loaded {len(self.test_set)} test samples")
def llm_judge(self, question, answer_a, answer_b, criteria):
"""LLM-as-Judge: Let DeepSeek judge the quality of two answers"""
prompt = f"""You are a fair review expert. Please compare the following two answers.
Question: {question}
Answer A: {answer_a}
Answer B: {answer_b}
Evaluation criteria: {criteria}
Please output in JSON format:
{{
"winner": "A" | "B" | "tie",
"score_a": 1-10,
"score_b": 1-10,
"reasoning": "brief explanation"
}}"""
response = client.chat.completions.create(
model="deepseek-chat",
messages=[{"role":"user","content":prompt}],
temperature=0.1
)
return json.loads(response.choices[0].message.content)
def evaluate_pairwise(self, baseline_model, candidate_model, num_samples=100):
"""Pairwise evaluation: baseline vs fine-tuned"""
samples = random.sample(self.test_set, min(num_samples, len(self.test_set)))
wins_a = wins_b = ties = 0
for i, item in enumerate(samples):
answer_a = baseline_model(item["instruction"], item.get("input",""))
answer_b = candidate_model(item["instruction"], item.get("input",""))
judgment = self.llm_judge(
item["instruction"], answer_a, answer_b,
item.get("criteria", "accuracy, completeness, fluency")
)
if judgment["winner"] == "A": wins_a += 1
elif judgment["winner"] == "B": wins_b += 1
else: ties += 1
if (i+1) % 20 == 0:
print(f"Progress: {i+1}/{len(samples)}, A wins:{wins_a} B wins:{wins_b} ties:{ties}")
return {"A_wins":wins_a, "B_wins":wins_b, "ties":ties, "win_rate_B":wins_b/(wins_a+wins_b+ties)}
class ABTestDesigner:
def __init__(self):
self.control_group = []
self.treatment_group = []
def calculate_sample_size(self, baseline_rate, expected_lift, alpha=0.05, power=0.8):
"""Calculate required sample size"""
from scipy.stats import norm
z_alpha = norm.ppf(1 - alpha/2)
z_beta = norm.ppf(power)
p1 = baseline_rate
p2 = baseline_rate * (1 + expected_lift)
p_pooled = (p1 + p2) / 2
n = (z_alpha * np.sqrt(2*p_pooled*(1-p_pooled)) + z_beta * np.sqrt(p1*(1-p1)+p2*(1-p2)))**2 / (p2-p1)**2
return int(np.ceil(n))
def check_significance(self, control_data, treatment_data, metric_name="satisfaction"):
"""Check statistical significance"""
control = np.array(control_data)
treatment = np.array(treatment_data)
t_stat, p_value = stats.ttest_ind(treatment, control)
effect = (treatment.mean() - control.mean()) / control.mean()
ci = stats.t.interval(0.95, len(treatment)-1, loc=treatment.mean(), scale=stats.sem(treatment))
return {
"metric": metric_name,
"control_mean": control.mean(),
"treatment_mean": treatment.mean(),
"relative_lift": f"{effect:.2%}",
"p_value": p_value,
"significant": p_value < 0.05,
"ci_95": (ci[0], ci[1])
}
def run_ab_test(self, model_a, model_b, traffic_split=0.5, duration_days=7):
"""Run A/B test"""
print(f"A/B test design:")
print(f" Control group (A): {model_a}")
print(f" Treatment group (B): {model_b}")
print(f" Traffic split: {traffic_split*100:.0f}%/{100-traffic_split*100:.0f}%")
print(f" Recommended duration: {duration_days} days")
n = self.calculate_sample_size(0.7, 0.05)
print(f" Minimum sample size per group: {n}")
return {"status":"running", "expected_completion":f"{duration_days} days later"}
evaluator = ModelEvaluator()
evaluator.load_test_set("test_set.json")
# result = evaluator.evaluate_pairwise(baseline, finetuned, num_samples=100)
ab = ABTestDesigner()
n = ab.calculate_sample_size(baseline_rate=0.72, expected_lift=0.05)
print(f"Need {n} samples per group to detect a 5% improvement")
# significance = ab.check_significance(control_scores, treatment_scores)Best Practices for A/B Testing
Random assignment: Use a hash of user ID for traffic allocation (e.g., hash(user_id) % 100 < 50 → group A) to ensure the two groups have similar user characteristics. Sample size calculation: Calculate the required sample size before starting the experiment. If the expected improvement is only 3%-5%, typically 5000-10000 samples per group are needed to reach statistical significance. Drawing conclusions with insufficient sample size is the most common mistake in A/B testing. Experiment duration: Run for at least one full business cycle (usually 1-2 weeks), covering both weekdays and weekends. Too short an experiment may be affected by the "novelty effect" (users are curious about the new model) or the "day-of-week effect" (weekend user behavior differs from weekdays). Avoid peeking: Do not frequently check results mid-experiment and stop early. This leads to "peeking bias," which can inflate the false positive rate from 5% to over 20%. Set the experiment duration and only look at results after it ends. Multi-metric evaluation: Do not look at only one metric. If your fine-tuned model improves satisfaction by 3% but increases latency by 50%, you need to weigh this trade-off. It is recommended to set "guardrail metrics"—if core business metrics (e.g., conversion rate, retention) show significant decline, pause the experiment regardless of satisfaction.
From Evaluation to Iteration
The ultimate goal of evaluation is not to score the model but to identify the next optimization direction. By analyzing evaluation results: Which scenarios improved significantly? (Continue to strengthen) Which scenarios worsened? (Need to adjust training data or strategy) Which scenarios showed no change? (Data may be insufficient or methods may be wrong) Transform evaluation insights into specific optimization tasks, forming a continuous improvement loop of "fine-tune → evaluate → analyze → fine-tune again."
Common Pitfalls in A/B Testing
A/B testing may seem simple, but it is full of pitfalls. Besides the peeking bias and novelty effect mentioned earlier, there are: sample contamination—if the same user may encounter both group A and group B during the experiment (e.g., on different devices), the results are contaminated. The solution is to use a stable user identifier (e.g., account ID rather than device ID) for assignment. Time effect interference—if external events occur during the A/B test (e.g., competitors launch new products, industry policy changes), both groups may be equally affected, but it is difficult to exclude from experimental metrics. The solution is to use A/A testing as a control. Multiple comparison problem—if you observe 10 metrics simultaneously, even with no real effect, one metric may "by chance" reach statistical significance. The solution is to apply Bonferroni correction to p-values or preregister the primary metrics you care about. Long-term effect evaluation: Most A/B tests last only 1-2 weeks and can only observe short-term effects. However, many AI improvements take longer to manifest—for example, improvements in personalization require multiple interactions between the model and users to show. It is recommended to set up a "holdout group" for important model changes—a small portion of users continue to use the old model version for long-term evaluation. Additionally, establish "reverse metric" monitoring—a fine-tuned model may improve a metric in the short term but lead to user fatigue or decreased usage frequency in the long run. The success of AI products ultimately depends on user retention and long-term value, not short-term metric optimization.
Experiment Culture and Management
Building a healthy experiment culture is more important than mastering experimental techniques. Core principles: experimental conclusions are driven by data rather than opinions ("I think this model is better" is less convincing than "data shows satisfaction improved by 3.2%, p=0.02"); failed experiments are also successes (knowing what doesn't work is as valuable as knowing what works); encourage bold hypotheses and validate them with rigorous experiments; build an experiment knowledge base—record the hypothesis, design, results, and lessons of each experiment to avoid repeating mistakes. Conduct a monthly experiment review to share the past month's experiment results and lessons, gradually accumulating the team's experimental wisdom.
Want to orchestrate this skill chain yourself?
Open in Skill Chain →