What is LoRA

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning (PEFT) technique. It does not modify the original weights of the pre-trained model, but instead inserts trainable low-rank matrices into the model, training only these new parameters. This makes fine-tuning large models feasible—a single GPU can fine-tune a 7B or even 13B model.

How LoRA Works

The core idea of LoRA is that the weight update of a model can be represented by low-rank decomposition. For a weight matrix W, its update ΔW is decomposed into the product of two small matrices: ΔW = BA, where the rank of B and A is much smaller than the original matrix. During training, only A and B are updated; during inference, they are merged into the original weights, adding no inference latency.

Environment Setup

pip install transformers peft accelerate datasets bitsandbytes

# Verify installation
python -c "import torch; print(torch.cuda.is_available())"

LoRA Fine-tuning in Practice

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model, TaskType
from datasets import load_dataset
from trl import SFTTrainer

# Load model
model_name = "deepseek-ai/deepseek-llm-7b-chat"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

# LoRA configuration
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,                    # rank
    lora_alpha=32,          # scaling factor
    lora_dropout=0.1,       # dropout ratio
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"]
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

# Training arguments
training_args = TrainingArguments(
    output_dir="./lora-finetuned",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    save_strategy="epoch"
)

# Load dataset
dataset = load_dataset("json", data_files="train_data.json")

# Train
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    tokenizer=tokenizer,
    max_seq_length=512
)
trainer.train()

# Save model
model.save_pretrained("./lora-adapter")
tokenizer.save_pretrained("./lora-adapter")

LoRA Parameter Tuning

  • r (rank): Usually set to 4-64. Larger r increases model capacity but also computational cost. Generally, r=8 or 16 is a good starting point.
  • lora_alpha: Usually set to 2 times r. Controls the magnitude of LoRA updates.
  • target_modules: Choose the modules to apply LoRA. For LLaMA architecture, typically q_proj, v_proj, k_proj, o_proj are selected.
  • lora_dropout: 0.05-0.1, to prevent overfitting.

Inference and Deployment

from peft import PeftModel

# Load base model and LoRA adapter
base_model = AutoModelForCausalLM.from_pretrained(model_name)
model = PeftModel.from_pretrained(base_model, "./lora-adapter")

# Merge weights (optional, reduces inference latency)
model = model.merge_and_unload()

# Inference
inputs = tokenizer("请用中文介绍人工智能", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))

Summary

LoRA democratizes large model fine-tuning. Even with a single consumer-grade GPU, you can fine-tune 7B-13B models. It is recommended to start with a smaller r value and gradually adjust parameters to find the optimal configuration.