LARA: Lightweight Adapters in the Residual Stream for Composable Adaptation and Alignment

When you fine‑tune a large language model you usually touch its weight matrices—adding low‑rank updates, inserting adapters, or rewriting entire layers. That works, but it couples the new behavior to the parameters themselves, making it hard to swap, combine, or dial the effect up or down without retraining. LARA (Lightweight Additive Residual Adaptation) offers a different angle: instead of changing the model’s weights, it leaves the base network frozen and injects corrections directly into the residual stream at a few chosen layers.

What You Need to Know

LARA works by reading the hidden state (the residual) at a small set of transformer layers—often just the first and last block of each stage—and computing a low‑rank correction that is added back to the same residual before it proceeds forward. Because the correction lives in the activation space, the original weight matrices stay untouched. At inference time a scalar γ scales the injected correction; γ = 0 gives the pure base model, γ = 1 recovers the fully adapted behavior, and intermediate values blend the two smoothly.

The method mirrors LoRA in parameter efficiency: for a given budget of trainable parameters (e.g., a few megabytes), LARA achieves comparable performance on downstream tasks. The authors demonstrate this on a code‑generation fine‑tuning benchmark and on preference optimization via Direct Preference Optimization (DPO). In both settings, LARA matches LoRA’s accuracy while keeping the base model completely frozen.

Why It Matters

First, keeping the base weights immutable simplifies model serving. You can ship a single checkpoint and apply many different LARA modules at runtime, swapping them in and out without reloading the model or risking weight corruption. This is especially valuable in multi‑tenant environments where dozens of fine‑tuned variants share the same backbone.

Second, the explicit γ scale provides a principled way to control the strength of adaptation. Rather than training multiple checkpoints for different trade‑offs, you can adjust γ on the fly—useful for safety‑critical applications where you might want to start with a conservative base behavior and gradually increase adaptation as confidence grows.

Third, because the adaptation lives in the residual stream, it composes naturally. You can stack several LARA modules (each targeting different layers or tasks) and simply sum their corrections, enabling modular skill‑addition without interfering with each other’s weight updates.

Key Details

• Target locations: LARA inserts corrections at a configurable subset of transformer blocks (e.g., the first and last block of each stage), keeping the number of adaptation points low to preserve compute efficiency.
• Low‑rank form: Each correction is a product of two thin matrices (U Vᵀ) with rank r, analogous to LoRA but applied to the activation vector rather than a weight matrix.
• Parameter budget: In the experiments, LARA used roughly 0.5 % of the base model’s parameters per adapter, matching LoRA’s parameter count for fair comparison.
• Training objective: The same loss functions used for LoRA (supervised fine‑tuning or DPO) are minimized; only the U and V matrices are updated.
• Inference scaling: A single scalar γ multiplies the summed residual corrections before they are added to the stream, enabling continuous interpolation between base and adapted outputs.
• Compatibility: Since the base weights are never altered, LARA works with any existing model checkpoint—including quantized or pruned versions—without requiring a retraining pass.

What’s Next

The authors suggest exploring dynamic γ schedules—perhaps guided by uncertainty estimates or user feedback—to automatically adapt the strength of correction during generation. Another promising direction is combining LARA with other parameter‑efficient techniques (like prefix tuning) in a hybrid framework, where some adjustments live in weights and others in activations, to see whether the complementary strengths yield even better trade‑offs between performance, memory, and controllability.

📌 Source: Arxiv Ml