Delta-rule Memory: 8×8 State Adds Low-rank Attention

δ-mem maintains a small associative matrix M outside the frozen transformer. Writes occur online when the system extracts a memory-worthy value v paired with a key k. Each write uses the canonical delta / Widrow–Hoff (LMS) step: update proportional to the error (v − M k) times kᵀ. That produces a rank‑1 outer-product increment on every write, so M evolves as a sum of rank‑1 contributions. Two practical consequences follow. Each write costs one outer product and a scalar step size. And the memory’s effective rank grows with the number of distinct writes, keeping the stored signal in a tiny subspace that’s cheap to read and project into attention.

In attention, queries Q score keys K to weight values V. δ-mem reads M with a query q (computing M k or analogous readout) and projects that readout into small adapter heads whose outputs act as additive K/V components. Because writes are rank‑1, every readout lies in the span of stored write directions. Projecting that span into K/V space yields a correction matrix with rank bounded by the memory dimension. The backbone’s logits therefore shift only along a few semantically meaningful axes. Practically this nudges the frozen model toward tokens consistent with remembered content, without token-window surgery. Runtime cost is tiny: an 8×8 read and a few linear projections per layer versus an expensive long-context attention sweep. Serving complexity stays low: M is an isolated state object you can checkpoint and manage separately from the model.

The core empirical claim is concrete. With an 8×8 online state, δ-mem improves average scores to about 1.10× the frozen backbone and ~1.15× the best non-δ baseline. The gains grow on memory-heavy suites: 1.31× on MemoryAgentBench and 1.20× on LoCoMo. The authors report that general capabilities remain largely intact, so you don’t sacrifice non-memory performance for these memory gains. These numbers are bench-averaged multipliers derived from the experiments in the preprint. Why this works: many long-range dependencies are low-dimensional. User preferences, salient facts, and agent state often lie on a handful of axes. δ-mem preserves those axes in M and injects them where attention can use them, producing measurable lift with very little state.

δ-mem trades representational breadth for cheap persistence. A fixed low-dimensional M cannot store high-dimensional data like long verbatim transcripts or dense knowledge graphs. If the memory your task needs is high-rank, an 8×8 matrix will saturate. The delta-rule is local and greedy. Without careful step-size, write scheduling, or forgetting control, new writes can overwrite useful directions. The authors highlight these tuning knobs and show experiments that preserve general capabilities while improving memory-heavy tasks. Operationally, δ-mem is appealing: add persistent behavior to a frozen LLM with no retraining, minimal runtime overhead, and a tiny checkpointable state. It pairs well with hybrid systems—use δ-mem for low-dimensional agent state and a retrieval layer for dense factual lookups. If you want to try it: implement M as a small float32 matrix, use the LMS update M ← M + η (v − M k) kᵀ on writes, read M k on queries, and project the readout into small adapter heads injected into attention. Monitor catastrophic forgetting and tune η and write frequency. The paper provides experimental recipes and evaluation details. (arXiv: https://arxiv.org/abs/2605.12357)