QLoRA Quantized Fine-Tuning in Practice

QLoRA Overview

QLoRA (Quantized LoRA) is a quantized version of LoRA that reduces the memory requirements of fine-tuning to one-fourth or even lower by quantizing the base model to 4-bit precision. With QLoRA, a single RTX 4090 with 24GB VRAM can fine-tune a 65B parameter large model.

Core Technologies

QLoRA uses three key technological innovations:

• NF4 Quantization: NormalFloat4 is a 4-bit quantization format optimized for normally distributed weights, which performs better than traditional INT4 quantization.
• Double Quantization: Quantizes the quantization constants themselves, further reducing memory usage.
• Paged Optimizers: Uses CPU memory to handle gradient checkpoints, avoiding VRAM OOM.

QLoRA Fine-tuning in Practice

import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from datasets import load_dataset
from trl import SFTTrainer

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True
)

# Load quantized model
model_name = "deepseek-ai/deepseek-llm-7b-chat"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)

# Prepare for k-bit training
model = prepare_model_for_kbit_training(model)

# LoRA config
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

# Training
# Note: QLoRA training may require smaller batch size
# and higher gradient accumulation steps
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    args=TrainingArguments(
        output_dir="./qlora-output",
        per_device_train_batch_size=1,
        gradient_accumulation_steps=16,
        learning_rate=1e-4,
        num_train_epochs=3,
        fp16=True,
        logging_steps=10
    ),
    tokenizer=tokenizer
)
trainer.train()

Memory Usage Comparison

QLoRA Tuning Recommendations

• Learning Rate: QLoRA typically requires a lower learning rate; 1e-4 to 2e-4 is a common range.
• Batch Size: Due to reduced numerical precision of quantized models, it is recommended to use smaller batch sizes with larger gradient accumulation.
• r Value Selection: Under QLoRA, the r value can be appropriately increased (16-64) because quantization loses some information.
• target_modules: It is recommended to cover all linear layers, including gate_proj, up_proj, down_proj.

Notes

Although QLoRA is extremely memory-efficient, training speed is about 30% slower than LoRA, and the final model quality may be slightly lower than LoRA. When VRAM is sufficient, prefer LoRA (FP16); when VRAM is insufficient, QLoRA is the best choice.