You Could Have Invented Transformers From Memory

TL;DR

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Alex Wang’s post explains neural-network layers through associative memory: they store key-value pairs and retrieve values from similar inputs. Its rough capacity estimates for 64-dimensional keys rise from about 64 memories for a linear store to about 2,700 for an MLP and more than a million for self-attention; these are explanatory estimates, not reported benchmark results.

Alex Wang has published an explainer that recasts linear layers, multilayer perceptrons and self-attention as different kinds of associative memory, offering a way to reason about how neural networks store and retrieve information. For a model architecture central to modern language systems, the post’s main claim is that a memory-based view helps explain why attention can retrieve far more stored associations than a simple linear layer.

Wang defines a memory as a key-value pair: the key is used to find an association, while the value is the information to retrieve. A memory store receives a query and returns an output; in an associative memory, a query similar to a stored key should bring back its value. The post focuses on interference: as more memories are added, unrelated stored associations can contaminate the answer to a query.

In the simplest example, a linear layer stores memories by adding an outer product of each value and key to its weights. When queried, it combines values according to how similar their keys are to the input. Wang estimates that, for random keys and values, this store’s interference becomes comparable to the target signal at roughly one memory per key dimension. With 64-dimensional keys, that suggests capacity of about 64 memories under the post’s simplified assumptions.

The post then sketches a progression to an MLP and self-attention. For 64-dimensional keys, Wang gives approximate capacities of about 2,700 memories for an MLP and more than a million for self-attention. These figures are theoretical heuristics in the explainer, not measurements of a particular deployed model or proof that a model can reliably recall that many arbitrary facts in practice.

Why Attention Recalls More Associations

The memory framing offers an accessible way to understand a key difference between network components: storing information and retrieving it cleanly are separate problems. A model may have weights that encode many associations, yet similar keys can make recall noisy. Wang’s account puts that interference problem at the center rather than treating parameter count alone as a measure of what a system remembers.

For readers trying to understand why transformers use attention, the comparison is useful because it links a familiar function—looking up information by similarity—to the operation of a neural layer. The post’s estimates suggest that self-attention can scale recall capacity far beyond a basic linear map under the stated assumptions. They should not be read as direct forecasts of factual accuracy, model knowledge, or performance in real-world use.

From Linear Maps to Attention

Wang begins with the everyday experience of asking a language model for information it has encountered during training, then asks how such systems might store and retrieve associations. The explainer develops the idea from a linear memory store, where each new key-value pair adds to the weights, toward more elaborate structures. Its organizing measure is the interference-to-signal ratio: the strength of unwanted recalled information compared with the intended value.

The post also connects weight updates to memory writing. In a linear layer trained by gradient descent, the update to the weights has an outer-product form, which Wang describes as adding a memory with an input as key and a gradient-related quantity as value. This is presented as an interpretive connection, with the post pointing readers to earlier work by Irie, Csordás and Schmidhuber on viewing trained linear layers as attention over training inputs.

“The key property that characterizes a “good” memory store is its ability to recall what it stored.”

— Alex Wang

Limits of the Capacity Estimates

The supplied material includes only part of Wang’s post, ending as it begins to discuss ways to reduce interference. It does not provide the full derivation of the MLP and attention estimates, so the assumptions behind the figures beyond the stated setup are not fully available here. The 64, 2,700 and million-plus estimates should be attributed to Wang’s explanatory framework, not treated as independently verified capacity measurements.

It is also unclear from the supplied text how the estimates translate to trained language models, where keys and values are not necessarily random, memories may overlap, and recall quality depends on the task and model. The post’s capacity comparison does not establish how many facts a particular system can reliably remember or retrieve.

What the Full Analysis Adds

The next step for readers is the remainder of Wang’s explanation, which the supplied excerpt introduces as an account of how interference can be reduced beyond the basic outer-product store. The excerpt does not state the full methods or qualifications for the later capacity estimates, and it gives no announced follow-up date or external validation. Any broader conclusions should wait for the complete derivation and a clear comparison with empirical tests of trained models.

Key Questions

What is the main development?

Alex Wang published an explainer that compares linear layers, MLPs and self-attention as associative memory stores, using recall interference to reason about capacity.

What does associative memory mean here?

It means storing key-value pairs and retrieving a value when a query resembles its key. Other stored keys can interfere with the target association.

Are the capacity figures measured results?

No such measurements are provided in the supplied material. The figures—about 64, 2,700 and over a million memories for 64-dimensional keys—are estimates in Wang’s explanatory framework.

Does this show how many facts a language model can remember?

No. The post’s simplified capacity estimates do not directly establish how many real-world facts a particular trained model can store or accurately retrieve.

Source: hn