LoRA & DoRA in TinyGrad

From-scratch implementations of LoRA (Low-Rank Adaptation) and DoRA (Weight-Decomposed LoRA) in TinyGrad, showing how injecting low-rank adapters into linear layers reduces trainable parameter count while maintaining fine-tuning performance.

Implementation

LoRA keeps the pretrained weight matrix frozen and learns a low-rank update through two smaller matrices. DoRA separates each weight into magnitude and direction, then applies low-rank adaptation to the directional component. Both implementations expose these mechanics directly through TinyGrad rather than relying on a high-level adapter library.

Purpose

The project provides a compact reference for understanding parameter-efficient fine-tuning, including how adapters affect trainable parameter counts, computation graphs, and inference behavior.