The Query Knows What to Forget: A Second Erase Direction for Linear Attention

When language models process very long texts, they often compress information into a fixed‑size state. Linear attention mechanisms keep such a state, but as the context grows, many tokens share the same memory slots and start to interfere with each other. This interference makes it hard for the model to retrieve the right piece of information when it is needed later. Researchers have tried to reduce the clash by shaping how old content is erased, but most approaches only look at the current token’s key to decide what to forget.

What You Need to Know

Linear attention stores a running summary of past tokens in a small vector space. At each step the model adds a contribution from the current token’s key and removes, or “erases”, parts of the old state that are no longer useful. In Gated DeltaNet‑2 (GDN‑2) the erase direction is derived solely from the key of the incoming token. While this works well for short contexts, the query used to read the state can point to dimensions that the key‑based erase cannot touch, leaving stale information aligned with the query intact.

The paper introduces the Query‑derived Erase Direction (QED). QED adds a second erase vector that is computed from the query and is forced to be orthogonal to the key‑based erase direction. Because the key‑based edit can only change the subspace spanned by the key, the new query‑based edit can act on the complementary subspace. In effect, the model can now cancel old‑state content that interferes with the current query while preserving the parts of the state that are useful for future reads.

Why It Matters

By giving the erase step access to the query’s perspective, QED reduces the interference that hurts retrieval in long contexts. Experiments show improved accuracy on tasks that require recalling specific facts from lengthy documents, without increasing the size of the state or the computational cost per token. This makes linear attention more reliable for applications such as retrieval‑augmented generation, long‑form summarization, or any setting where the model must keep a compact memory over thousands of tokens.

The improvement is consistent across different context lengths: once the sequence exceeds the point where interference becomes noticeable, QED yields measurable gains. Because the method only adds a lightweight orthogonal projection, it can be plugged into existing delta‑rule models with minimal engineering effort, offering a practical way to extend the effective memory horizon of linear attention.

Key Details

• Linear attention maintains a fixed‑size state vector that is updated additively with key‑derived contributions.
• Standard erase directions come from the key of the current token, limiting edits to the key subspace.
• Queries used for reading can have components orthogonal to the key subspace, which standard erases cannot affect.
• QED computes an erase direction from the query and enforces orthogonality to the key‑based direction.
• The orthogonal edit can remove interfering old‑state content measured along the query while leaving the key‑aligned part untouched.
• Empirical results show better retrieval accuracy on long‑context benchmarks with no extra state size or asymptotic compute.

What’s Next

Future work could explore adaptive ways to balance the strength of the key‑ and query‑based erase vectors, perhaps conditioned on the predicted interference level. Extending the idea to other forms of structured memory, such as hierarchical or chunk‑based representations, may further push the limits of what fixed‑size state can achieve in long‑sequence modeling.

📌 Source: Arxiv Ml