LoRA: low-rank adaptation and what its memory savings actually cover

What it is

Low-Rank Adaptation, or LoRA, is a method for adapting a pretrained neural network with relatively few trainable parameters. Instead of updating every selected weight matrix directly, it learns a small factorized change while freezing the original weights. This supports compact task-specific adaptations that share a common base model. [1][2]

How it works

For a base matrix W0 of shape d by k, LoRA introduces A of shape r by k and B of shape d by r. The effective matrix is W_eff = W0 + (alpha/r)*B*A in the original scaling convention. Rank r limits the rank of this individual update, not the rank of the entire model. A and B together contain r times (d+k) parameters. [1]

The original formulation initializes A randomly and B to zero, making the initial update zero. Gradients train the factors while W0 stays fixed. The scaling factor and rank are configuration choices that must be recorded.

Practical workflow

Fix the base model revision, tokenizer or input preprocessing, training split and evaluation task. Choose target modules, rank, scaling and any adapter dropout explicitly. Verify which parameters are trainable; additional task heads or biases may be intentional, but they change the trainable-parameter count. The reference implementation provides mechanisms for these choices. [2]

Train on the adaptation data, select settings using validation data and save adapter weights together with their configuration and base-model identity. An adapter checkpoint alone does not contain the original pretrained model. [2] Measure actual peak memory and quality rather than inferring them solely from parameter counts.

Illustrative example

Consider one bias-free matrix of shape 1024 by 1024. Full updating exposes 1048576 parameters. LoRA at rank 8 adds 8 times (1024+1024), or 16384 trainable parameters: 1/64 of that matrix’s original count. The base matrix is still present. This arithmetic describes parameter counts only, not a 64-fold reduction in total training memory.

Limits and verification

A low rank can constrain adaptation, while increasing it does not guarantee better generalization. Target-module selection, data quality and preservation of earlier capabilities also require evaluation. The method still performs computation through the base network and stores needed activations; freezing weights does not remove these costs.

Where supported, merging the learned update into the base matrix removes the separate adapter computation at inference. The reference implementation describes this behavior. [2] Verify merged and unmerged outputs within a suitable numerical tolerance and measure the actual serving configuration. An unmerged or incompatible backend need not have the same latency. No local fine-tuning or memory benchmark was performed for this article.

Sources

[1] Hu et al.: LoRA: Low-Rank Adaptation of Large Language Models [2] Microsoft: LoRA reference implementation