LoRA & DoRA in TinyGrad
From-scratch Low-Rank Adaptation and Weight-Decomposed LoRA implemented in TinyGrad
LoRA (Low-Rank Adaptation) and DoRA (Weight-Decomposed LoRA) written from scratch in TinyGrad. Both replace the full fine-tuning update of a linear layer with a much smaller set of trainable parameters.
Implementation
LoRA keeps the pretrained weight matrix frozen and learns a low-rank update through two smaller matrices. DoRA separates each weight into magnitude and direction, then applies the low-rank adaptation to the directional component only.
Both are built on TinyGrad’s own tensors and autograd, with no adapter library in between. The adapter parameters and the extra matmuls stay visible in the code.