How an LLM runs inside a DRAM chip

01 · the promiseThe answer comes out of the memory itself

Ask this system "What is the capital of France?" and, after some minutes, it answers "Paris." The interesting part is where the multiplications happen. In a normal computer the memory only stores numbers; a separate processor fetches them and does the math. Here the memory does the math. The weights of the neural network sit in the memory as charge, and the memory's own analog behaviour is coaxed into computing with them in place — and the answer comes back out as ordinary binary, not an analog reading.

Nothing about the memory is modified. These are commodity DDR4 sticks, the same parts in any desktop. What is unusual is the timing of the commands sent to them — a controlled rule-break, explained in the next few sections, that turns a storage array into a tiny, massively parallel calculator.

02 · the substrateA memory cell is a bucket of charge

One bit of DRAM is a tiny capacitor: charged means 1, empty means 0. To read it, the chip activates the cell — it tips the bucket out onto a shared wire (a "bitline"). A sense amplifier on that wire watches the faint voltage and snaps it to a clean 0 or 1.

Two facts matter for everything that follows:

• Reading is destructive. Tipping the bucket empties it. So right after sensing, the chip writes the value back — it re-fills the bucket. Every read is really a read-and-restore.
• Buckets leak. Left alone, a cell forgets its bit in a fraction of a second, so the chip constantly refreshes: it re-reads each row and tops the charge back up. Refresh preserves charge, not content — it faithfully keeps whatever the row holds now, correct or not.

Three commands do all the work below: ACT (activate — open a row onto its bitlines), READ/WRITE, and PRE (precharge — close the row and reset the bitlines for the next one).

03 · the layoutCells, rows, subarrays, banks

Cells are not addressed one at a time. Thousands of them share a wordline and are opened together as a row. Rows are grouped into subarrays, each with its own strip of sense amplifiers; subarrays stack into banks; banks make up the DIMM. When the chip "activates row 38424," it opens that whole row's worth of cells onto the shared bitlines at once.

That "open a whole row at once" behaviour is what the compute trick exploits. Open several rows at nearly the same instant, and their charge does something useful.

04 · the trickOpen several rows at once and let the charge vote

Normally you open one row at a time, with mandatory settling pauses between commands. Break those pauses — issue the commands too close together — and you can open a small set of rows onto the same bitlines simultaneously. Their charge mixes. The sense amplifier then snaps the mixture to whichever value most of the contributing rows held.

That majority is the computation. A majority vote across rows of bits, done in analog, thousands of bit-positions in parallel — this single move is the arithmetic the whole language model is built out of.

05 · the instruction walkOne compute step, row by row

Here is one instruction, from start to finish. Watch the same eight rows — they keep their addresses and their vertical position the whole way through; only the colour (what just happened to them) and the value they hold change. Use Prev / Next or the ← → arrow keys.

The addresses are real: a fixed 16-row compute group in bank 0, on one of the project's SK hynix modules. The first row the vote opens is 38424 (marked ● source); the two rows the command actually names are 38424 and 38988.

06 · from a vote to a matrix multiplyCounting yes-votes is the arithmetic

A neural-network layer is a big matrix multiply: dot products of a weight vector with an activation vector. This model's weights are ternary — every weight is just −1, 0, or +1. That is what makes memory arithmetic possible, because a ternary dot product needs no multiplier — only ANDs and counting:

1. Split the ternary weights into two yes/no masks: a +1 mask and a −1 mask.
2. Slice the activation numbers into their individual bits (the ones-bits, the twos-bits, and so on).
3. For each mask and each bit-slice, AND them together — that AND is exactly one vote in DRAM (a majority with one input pinned low is an AND) — then count the 1s that come out (a "popcount").
4. On the host, add up the counts with the right place-values and subtract the −1 count from the +1 count. That total is the dot product.

So a whole projection is many thousands of votes plus a count. Everything the memory does is step 3, two thousand bit-positions at a time. Everything else is bookkeeping. The formal, bit-by-bit version of the vote lives in the mechanism explainer.

07 · the inference loopWhere DRAM computes, where the CPU still does

Producing one word (one token) runs the whole model once. The split is the same one every "compute-in-memory" system makes:

• In DRAM: the matrix multiplies — every layer's projections — run as votes and counts, the mechanism above.
• On the CPU: the parts that are not matrix multiplies — attention's softmax, the normalisation steps, turning the final numbers into the next token, and the tokenizer/embedding at the edges.

The loop: text in → CPU turns it into numbers → many matrix multiplies run in the memory, with the small non-matrix steps on the CPU between them → out comes the next token → feed it back in and repeat. A single token is hundreds of thousands of votes across all the layers; the answer is the accumulation of all of them.

08 · the slow partIt is the round-trip, not the physics

A vote itself takes tens of nanoseconds. So why does a token take minutes? Because on this rig the memory sits behind an FPGA card on a PCIe link, and every little program is a round-trip: the host ships commands over, the card runs them, and the results come back. That hand-off has a fixed cost each time it happens, no matter how few bytes ride along. With thousands of programs per token, the wall is the number of trips, not the physics inside the chip.

This is why one of the levers is to count the yes-votes inside the DRAM before sending anything back — the answer that crosses the link shrinks from a whole memory row to a handful of numbers.

09 · what runs todayReal models, correct answers, out of memory

This is not a single scripted demo. Seven flagship model families run correctly with their matrix multiplies executed in unmodified DDR4 — a ternary BitNet-2B and two forms of Bonsai run live on the silicon; Llama-2 (7B and 13B), Llama-3 and Phi-4 are validated to match their reference math through a sampled end-to-end check. The headline list is on the front page.

Speed is improving as the round-trip work is engineered down: the measured ladder runs from 632 down to 45 seconds per token, with the memory-interface-bound floor as the target the work is aimed at. The point is not that this beats a GPU today (it does not). The point is that the mechanism is real, on real silicon, on published models — the correct word comes back out of the memory's own charge.

go deeperWhere each thread continues