When people compare AI chips, they talk about compute: how many trillions of operations per second a GPU can perform. But for large language models, the limit is often not compute at all — it is how fast data can be moved from memory to the processor. That is why high-bandwidth memory (HBM) has become one of the most important and most constrained components in the AI industry.

The memory wall

Generating each token of a language model’s output requires reading a large share of the model’s weights from memory. A model with tens of billions of parameters means tens of gigabytes read for every token. If memory cannot deliver data fast enough, the processor sits idle, waiting. This gap between compute speed and memory speed is often called the memory wall, and it has widened for decades because processors improved faster than memory.

What HBM is

HBM is a type of DRAM designed for very high bandwidth:

  • Stacked dies: several DRAM chips are stacked vertically and connected with thousands of tiny vertical connections called through-silicon vias (TSVs).
  • Very wide interface: instead of a narrow, fast bus, HBM uses a very wide interface — over a thousand data lines per stack — running at moderate speeds, which delivers enormous bandwidth with good energy efficiency.
  • Next to the processor: HBM stacks sit right beside the GPU or accelerator on the same package, connected through a silicon interposer, so signals travel millimetres instead of centimetres.

A modern AI accelerator typically surrounds the main processor with several HBM stacks, together providing many terabytes per second of bandwidth and tens to hundreds of gigabytes of capacity.

Cross-section diagram of a GPU and a stack of DRAM dies with through-silicon vias, both mounted on a silicon interposer above the package substrate and circuit board
In this cross-section, stacked DRAM dies connect through through-silicon vias to a base die, and the silicon interposer links the stack to the GPU beside it. Diagram: ScotXW / Wikimedia Commons, CC BY-SA 4.0

Generations

HBM has evolved through several generations — HBM, HBM2, HBM2E, HBM3, HBM3E and HBM4 — each increasing bandwidth per stack, stack height and capacity. Standards are defined by JEDEC, the semiconductor memory standards body, and new AI accelerators are often announced together with the HBM generation they use.

Why HBM matters for AI

  • Speed of generation: for inference, tokens per second are closely tied to memory bandwidth.
  • Model size per chip: HBM capacity determines how large a model, or how much context, fits on one accelerator without splitting it across chips.
  • Training efficiency: large batches and activations need fast memory too.
  • Energy: moving data costs energy; HBM moves more data per joule than conventional memory over long traces.

Why it is a bottleneck

HBM is hard to make. Stacking dies, drilling TSVs and testing stacks lowers manufacturing yields, and packaging the stacks next to the processor requires advanced packaging capacity that is itself limited — see our explainer on chiplets and advanced packaging. Only a few memory makers produce HBM at scale, so supply has repeatedly constrained how many AI accelerators can be shipped, and HBM has become a major profit driver for those companies.

What it means for you

If you are choosing GPUs for AI work, look beyond compute figures: memory capacity and bandwidth often predict real performance for language models better than peak operations per second. The same logic explains why running LLMs locally depends so much on memory.

In one sentence

HBM is the fast memory placed right next to an AI processor so that the processor is not left waiting for data.

What to watch next

  • HBM4 adoption in the next generation of AI accelerators.
  • Memory makers expanding HBM capacity and yields.
  • Custom HBM with logic integrated into the base die.
  • Alternative memory approaches for inference chips.

Key takeaways

  • Large language models are often limited by memory bandwidth, not compute.
  • HBM stacks DRAM dies and connects them with a very wide interface next to the processor.
  • HBM capacity decides how large a model fits on one accelerator.
  • Manufacturing complexity and packaging capacity make HBM a supply bottleneck.
  • When comparing AI hardware, look at memory capacity and bandwidth, not only compute.

Frequently asked questions

Is HBM the same as GPU memory in gaming cards?

No. Gaming cards usually use GDDR memory, which is cheaper and narrower. Data center AI accelerators use HBM for much higher bandwidth.

Why can’t chipmakers just add more HBM?

It is limited by stack production, yields, packaging capacity, cost and the physical space around the processor.

Does HBM matter for inference or training?

Both, but it is especially decisive for inference of large language models, where every token requires reading the weights.