The Memory Wall: Why On-Device LLM Inference Is Bottlenecked by Bandwidth, Not Compute

Your phone can run a 7-billion-parameter language model. Load one into an app and it works — coherent answers, real reasoning. But watch the tokens arrive and something feels off: there's a pause, then words trickle out at reading speed, sometimes slower. Meanwhile that same phone renders a 120 fps game with a scene far more computationally demanding than matrix multiplies. The chip clearly has the horsepower. So what's the model waiting on?

Not compute. It's waiting on memory — specifically, on the bandwidth to stream a few gigabytes of weights out of DRAM for every single token. This is the memory wall, and it's the single most important thing to understand about why on-device (and even datacenter) LLM inference performs the way it does. Once you see it, a dozen otherwise-confusing engineering decisions — aggressive quantization, obsession with model size, the rise of the NPU, unified memory architectures — stop looking like separate tricks and start looking like different attacks on the same bottleneck.

The roofline: two ceilings, and decoding hits the wrong one

The roofline model is the cleanest way to reason about any hardware kernel. A processor has two hard limits: how many arithmetic operations it can do per second (FLOPS), and how many bytes it can move from memory per second (bandwidth). Whether a given workload is limited by the first or the second depends on its arithmetic intensity — the ratio of FLOPs performed to bytes read.

Arithmetic Intensity = FLOPs performed ÷ Bytes moved

If you do a lot of math per byte fetched, you're compute-bound and the FLOPS ceiling is your limit. If you do little math per byte, you're memory-bound and bandwidth is your limit — the ALUs sit idle waiting for data.

Here's the thing about autoregressive decoding, the phase where the model generates tokens one at a time: it has catastrophically low arithmetic intensity. To produce a single token, the model reads every weight in the network and multiplies each one by an activation. That's roughly two FLOPs (a multiply and an add) for every parameter — but each parameter is a distinct value that has to be fetched from memory. The arithmetic intensity is around 2 FLOPs per parameter-read, which after accounting for datatype width lands close to 1 FLOP per byte or less.

Modern accelerators are built for the opposite regime. A chip might offer hundreds of FLOPs of compute for every byte per second of bandwidth. Feed it a workload with intensity of 1 and you use a fraction of a percent of the compute while saturating the memory bus. The ALUs are a stadium; decoding sends one runner through it at a time.

The practical consequence is a formula you can hold in your head:

Tokens per second ≈ Memory bandwidth (GB/s) ÷ Model size in memory (GB)

A 7B model at 16-bit weights is ~14 GB. On a device with, say, 100 GB/s of usable bandwidth, the ceiling is roughly 100 / 14 ≈ 7 tokens/second — and that's the theoretical maximum, before any overhead. Quantize the same model to 4-bit and it's ~4 GB, pushing the ceiling to ~25 tokens/second on the same hardware. Nothing about the compute changed. You just moved fewer bytes.

This is why the back-of-envelope prediction of on-device throughput barely mentions FLOPS. It's bytes-per-token divided into bandwidth, full stop.

Why prefill and decode behave like two different workloads

There's a wrinkle that trips people up: the same model on the same chip can be compute-bound and memory-bound depending on which phase it's in.

Prefill is when the model ingests your prompt. All the input tokens are available at once, so the model processes them as a batch — every weight it reads gets reused across hundreds or thousands of prompt tokens simultaneously. Arithmetic intensity is high. Prefill is compute-bound, and it's why a long prompt has a fixed "time to first token" that scales with prompt length but runs efficiently on the hardware.

Decode is the token-by-token generation that follows. Batch size is effectively one — each new token depends on the previous one, so you can't parallelize across the sequence dimension. Every weight read is amortized over exactly one token. Arithmetic intensity collapses. Decode is memory-bound.

This split explains a lot of real-world behavior. A model can feel snappy when summarizing a document (mostly prefill, then a short answer) and sluggish when writing a long essay (short prefill, then thousands of memory-bound decode steps). Batching multiple users' requests together in a datacenter raises decode's arithmetic intensity — the whole point of continuous batching is to make decode look more like prefill by reusing each weight fetch across many concurrent sequences. On a single phone serving one user, you don't have that luxury. Batch size one is the on-device reality, and batch size one is where the memory wall bites hardest.

Quantization is a bandwidth optimization wearing a memory-savings costume

Everyone describes quantization — storing weights in 8, 4, or even 2 bits instead of 16 — as a way to fit a bigger model into limited RAM. That's true, but it undersells it. On a memory-bound workload, halving the bytes per weight nearly doubles your token throughput, because throughput is bandwidth divided by bytes moved. Quantization isn't just about fitting; it's the highest-leverage speed knob you have.

The catch is that lower precision costs accuracy, and the formats differ in how gracefully they trade one for the other.

Two details separate a good quantization scheme from a naive one. First, not all weights are equally sensitive — the "K-quant" formats (the K in Q4_K_M) keep certain important tensors, like attention output projections, at higher precision while squeezing the rest, buying most of the quality back at a fraction of the size cost. Second, the arithmetic doesn't happen in 4-bit: weights are stored quantized to save bandwidth, then dequantized to a wider type inside the compute unit. You pay the fetch in 4-bit and the math in 16-bit. That's exactly the trade you want when memory is the wall and compute is abundant.

The reason 4-bit has become the de facto on-device default isn't arbitrary. It sits at the knee of the curve: roughly a 4x reduction in bytes-moved (and thus ~4x the throughput ceiling) versus FP16, while modern K-quant methods keep quality degradation small enough that most users won't notice on everyday tasks. Go below it and the accuracy cliff steepens fast.

The KV cache: the memory cost everyone forgets

Model weights are the obvious memory consumer, but they're not the only one — and the other one grows as you use it. The KV cache stores the key and value vectors for every token processed so far, so the model doesn't recompute attention over the entire history at each step. It's what makes decode O(1) per token instead of O(n). But it has to live in memory, and its size scales with context length:

KV cache bytes = 2 × layers × context length × hidden dim × bytes per value

For a mid-sized model at a long context, the KV cache can swell to gigabytes — sometimes rivaling the weights themselves. And critically, it's also streamed from memory during attention, so it eats into the same bandwidth budget that governs your token rate. A long conversation doesn't just consume RAM; it slows decoding down, because every token now has to read a bigger cache.

This is why on-device inference stacks quantize the KV cache too (8-bit or even 4-bit KV is common now), and why architectural tricks that shrink it — grouped-query attention, which shares key/value heads across query heads, and sliding-window attention, which caps how far back the cache reaches — matter enormously for edge deployment. They're not accuracy improvements. They're bandwidth-and-footprint improvements aimed squarely at the memory wall.

Every token = read ALL weights across this pipe. Tokens/sec ≈ bandwidth ÷ model-size-in-memory. Compute is rarely the limit.

Quantize weights → fewer bytes per token → higher ceiling on the SAME chip. Unified memory → wider pipe + no CPU-to-GPU copy → why Apple/edge SoCs win here.

NPUs, unified memory, and why the pipe is the product

If decode is bandwidth-bound, then the interesting hardware question isn't "how many TOPS does the NPU have?" — it's "how fat is the pipe to memory, and how short is it?"

This reframes what the on-device AI silicon race is actually about. A Neural Processing Unit is a fixed-function matrix-multiply engine that's dramatically more power-efficient than a GPU for the same math — which matters on a battery — but it doesn't rewrite the roofline. If the NPU can do the multiplies ten times faster but the weights still arrive over the same memory bus, decode throughput barely moves. NPUs win on energy per token far more than on tokens per second. That's the right trade for a phone (you care about not melting the battery), but it's why raw TOPS numbers on a spec sheet mislead people into expecting speed that the memory system can't deliver.

Unified memory architectures — where CPU, GPU, and NPU share one physical pool of high-bandwidth RAM — attack the wall from two directions. First, they tend to use wider, faster memory (Apple's M-series and comparable SoCs push hundreds of GB/s, well beyond typical discrete-phone DRAM). Second, and more subtly, they eliminate the copy: on a traditional discrete-GPU setup, weights have to be shuttled from system RAM across PCIe into the GPU's own VRAM before the GPU can touch them, and VRAM capacity caps model size. Unified memory lets the accelerator read weights in place. No copy, no PCIe bottleneck, and the model can be as large as total system RAM allows. That combination — wide bandwidth plus zero-copy plus large capacity — is exactly why Apple Silicon and similar edge SoCs punch above their weight for local LLMs despite having "weaker" GPUs than a desktop discrete card.

The uncomfortable implication for hardware marketing: the headline compute number is the least predictive spec for local LLM speed. Memory bandwidth and capacity are the ones that matter, and they're the ones buried at the bottom of the datasheet.

The takeaway

If you're deploying or evaluating on-device LLMs, reason from bandwidth, not FLOPS. Estimate throughput as usable memory bandwidth divided by the model's in-memory size and you'll predict real-world token rates far better than any TOPS figure will. Every major on-device optimization — 4-bit quantization, KV-cache compression, grouped-query attention, unified memory — is a variation on the same move: shrink the bytes that have to cross the memory bus per token, or widen the bus. Pick the smallest quantization your task can tolerate before you reach for a faster chip; on a memory-bound workload, dropping from 8-bit to 4-bit does more for your token rate than a generation of compute improvements will. The wall isn't the math. It never was.