RLHF Reinforcement Fine-Tuning in Practice: From Preference Data to Reward Model

There are several engineering details to note: first, the input must include the EOS token, otherwise the last hidden state may correspond to an irrelevant token; second, gradient clipping and mixed precision training should be enabled to prevent memory overflow; third, to prevent reward hacking during training, the validation set accuracy should be periodically evaluated. If the accuracy consistently falls below 70%, data issues should be investigated.

The evaluation metrics for reward models typically use "accuracy" or "Spearman correlation coefficient." We recommend also monitoring "preference consistency," which is whether the reward model's ranking of different responses to the same prompt aligns with human annotations. If the consistency rate is below 80%, the model is essentially unusable.

4. Misconceptions in Reinforcement Learning for Policy Model Fine-Tuning

Most people mistakenly believe that RLHF directly trains the policy model using the PPO algorithm, but in practice, the biggest challenges are "policy collapse" and "reward overoptimization." Policy collapse manifests as a sharp drop in output diversity, with all responses converging to the same template; reward overoptimization leads to outputs that are "high-scoring but unusable" and counterintuitive.

A classic technique is to introduce a KL divergence penalty to constrain the deviation between the policy model and the initial SFT model. The reward signal in PPO is designed as r = r_{RM} - β * KL, where β is a dynamic coefficient that can be adaptively adjusted. Another technique is "phased training": first warm up with a small learning rate, then gradually increase data scale and update steps to avoid extreme distributions.

Additionally, a "reference model" must be used to compute KL, and the reference model is typically the SFT-stage model, not the initial pretrained model. This detail is often not clarified in many tutorials, but it directly affects stability. Below is a simplified PPO training pseudocode showing the key logic (this is only a snippet; a full implementation requires combining with an RL library).

import torch

def ppo_update(policy_model, ref_model, reward_model, prompts, chosen_gen, rejected_gen, kld_coef=0.1):
    # Compute old policy probabilities and KL with reference model
    with torch.no_grad():
        ref_logprobs = ref_model.forward(prompts, chosen_gen).log_prob
        old_logprobs = policy_model.forward(prompts, chosen_gen).log_prob
        kl = (old_logprobs - ref_logprobs).mean()
    # Compute rewards
    rewards = reward_model(prompts, chosen_gen) - kld_coef * kl
    # Update policy (simplified)
    logprobs = policy_model.forward(prompts, chosen_gen).log_prob
    ratio = torch.exp(logprobs - old_logprobs)
    loss = -torch.min(ratio * rewards, torch.clamp(ratio, 1-0.2, 1+0.2) * rewards).mean()
    return loss

Note that in practice, advantage function computation is needed, and Generalized Advantage Estimation (GAE) is used to reduce variance. We recommend using mature libraries like TRLX or Ray RLlib rather than writing from scratch, but understanding the underlying loss computation is crucial for debugging.

Another easily overlooked issue is the "interaction frequency between the reward model and the policy model." After the policy model updates, the reward model may become invalid (out-of-distribution). Therefore, a new batch of data should be sampled and re-annotated every N steps, and the reward model should be updated online; otherwise, performance degrades rapidly. In practice, we found that updating the RM every 500 steps significantly suppresses overfitting.

5. Practical Case: Instruction Following Optimization Based on DeepSeek

We designed a specific task: to make the model "answer questions concisely and completely," avoiding long-winded responses. We collected 10,000 instruction data points, used the DeepSeek API to generate 3 candidate responses, and manually annotated preferences. The task is special because preferences depend not only on correctness but also on length and structure.

After training the reward model, we deployed it as an API service to score any response. Then, we used the PPO algorithm to fine-tune DeepSeek's open-source model. During training, we saved intermediate checkpoints of the policy model and conducted comparison tests via DeepSeek API's base_url: using the same prompts to observe output length and user satisfaction before and after optimization.

The actual results were surprising: before fine-tuning, the model averaged 150 characters per output; after fine-tuning, it averaged 80 characters, while still retaining key information. We randomly sampled 200 test prompts and conducted human blind evaluation; the optimized model's preference win rate increased from 40% to 86%. This proves the effectiveness of RLHF for style control.

6. Common Engineering Pitfalls and Debugging Strategies

Pitfall 1: Training data leakage. If the prompts in the preference data overlap with the test set, both the reward model and the policy model will overestimate performance, leading to a sharp drop in production. The solution is to use "deduplication + mutually exclusive partitioning" to ensure the training, validation, and test sets have no intersection.

Pitfall 2: Improper KL coefficient setting. If β is too large, the model barely changes; if too small, it leads to reward overoptimization. We recommend a "dynamic KL" strategy: increase β at the beginning of training, then decay it after stabilization, similar to learning rate decay. Specifically, monitor KL and reward changes and set a target interval.

Pitfall 3: Ignoring the impact of batch size. RLHF's reinforcement learning is extremely sensitive to batch size. A rule of thumb: the batch size should be at least 512 prompts, with each prompt generating 8 responses, to ensure gradient stability. If memory is insufficient, gradient accumulation or distributed strategies can be used.

7. Evaluation System and Pre-Deployment Validation

Do not rely solely on reward model scores to judge model quality; instead, design multi-dimensional evaluation: automatic metrics (e.g., BLEU, Rouge, response length) combined with human evaluation (e.g., Likert scale). We established a "comparative evaluation matrix," as shown in the table below:

We particularly emphasize the "diversity" metric: policy models tend to fall into deterministic outputs, so n-gram repetition rate should be monitored. If diversity decreases, it can be mitigated by increasing sampling temperature or adjusting RL hyperparameters.

Before deployment, stress testing is also necessary: simulate high-concurrency requests to ensure inference latency and throughput meet engineering requirements. If deploying via API, DeepSeek's official API can be used as a baseline to compare the response quality of the fine-tuned model.

8. Future Directions and Extended Thinking

RLHF is not the ultimate solution. The recently emerged DPO (Direct Preference Optimization) method bypasses the reward model and directly optimizes the policy based on preference data, making training more stable and reducing one component. We are also trying to incorporate

Another frontier direction is "Scalable Oversight", which uses stronger models (such as GPT-4) to automatically label preferences, thereby reducing labor costs. However, it should be noted that this may deviate from human preferences and requires regular calibration.

Finally, it is recommended that readers delve into DeepSeek's official technical report to understand its model architecture and alignment details, and pay attention to data lifecycle management in practice. RLHF is essentially a combination of engineering and science; only through continuous experimentation, documentation, and iteration can robust performance improvements be achieved.