Fine-Tuning Large Language Models: LoRA, QLoRA, and Instruction Alignment

While prompt engineering and Retrieval-Augmented Generation (RAG) provide in-context knowledge to Large Language Models (LLMs), adapting a foundation model to master domain-specific syntax, output strict JSON schemas, or adopt specialized reasoning patterns requires Fine-Tuning.

This guide details Parameter-Efficient Fine-Tuning (PEFT) architectures (LoRA and QLoRA), Supervised Fine-Tuning (SFT) dataset construction, and Direct Preference Optimization (DPO).

1. Quick-Reference: Fine-Tuning Paradigms

+-----------------------------------------------------------------------------------------+
|                               LLM FINE-TUNING PARADIGMS                                 |
+-----------------------------------------------------------------------------------------+
| Method                 | Trainable Parameters              | GPU VRAM Requirements      |
+------------------------+-----------------------------------+----------------------------+
| Full Parameter Tuning  | 100% of weights (70B params)      | Extreme (8x H100 80GB)     |
| LoRA (Low-Rank Adapt.) | 0.1% - 1.0% (Low-rank matrices)   | Moderate (1x - 2x A100)    |
| QLoRA (4-Bit NF4 LoRA) | 0.1% - 1.0% (Quantized base model)| Low (1x Consumer RTX 4090) |
+-----------------------------------------------------------------------------------------+

2. Low-Rank Adaptation (LoRA) Mathematical Mechanics

Rather than updating the frozen $d imes k$ weight matrix $W_0$ , LoRA decomposes the weight update $\Delta W$ into two low-rank matrices $B$ ( $d imes r$ ) and $A$ ( $r imes k$ ), where rank $r \ll \min(d, k)$ :

W = W_0 + \Delta W = W_0 + rac{lpha}{r} (B \cdot A)

During forward passes on input $x$ :

h = W_0 x + rac{lpha}{r} B(A x)

Setting $r = 8$ or $r = 16$ reduces trainable memory parameters by over $99\%$ , allowing fine-tuning of 70B parameter models on accessible hardware.