Almost every major language model today is built on the transformer, an architecture introduced by researchers at Google in the 2017 paper Attention Is All You Need. Here is how it works, in plain language.

Step 1: text becomes tokens

The model splits text into tokens — words or pieces of words. Each token becomes a list of numbers called an embedding, which places it in a kind of “meaning space” where related words sit close together.

Step 2: position is added

Word order matters (“dog bites man” vs. “man bites dog”), so the model adds information about each token’s position.

Step 3: attention

This is the key idea. For every token, attention lets the model look at all the other tokens and decide which ones matter for understanding it. In “The trophy didn’t fit in the suitcase because it was too big”, attention helps the model connect it to trophy.

Transformers use multi-head attention: several attention patterns in parallel, each free to focus on different relationships — grammar, references, topic.

Step 4: stack many layers

Attention is followed by a small neural network, and this block is repeated dozens of times. Lower layers capture simple patterns; higher layers combine them into more abstract features.

Step 5: predict the next token

For a chatbot, the final layer outputs a probability for every possible next token. The model picks one, adds it to the text, and repeats.

Why it was a breakthrough

Earlier models processed text one word at a time, which was slow to train. Transformers process all tokens in parallel, which suits modern GPUs and made it possible to train on far more data. That scalability is what enabled today’s large language models.

Limits

Attention compares every token with every other token, so cost grows quickly with input length. Much current research aims to make long contexts cheaper.

Attention in a simple example

Take the sentence: “The trophy didn’t fit in the suitcase because it was too big.” To understand what “it” refers to, a model must connect it with “trophy” rather than “suitcase”. In a Transformer, each token creates a query (“what am I looking for?”), and every token offers a key (“what do I contain?”) and a value (“what information do I pass on?”). Comparing the query for “it” with all keys gives attention weights — high for “trophy” — and the model mixes in the values accordingly. Many attention “heads” run in parallel, each learning different relationships, such as grammar, references or topic.

Encoders, decoders and both

  • Encoder-only models (like BERT) read the whole input at once and are good at understanding tasks such as classification and search.
  • Decoder-only models (like GPT-style models) generate text one token at a time, attending only to previous tokens — the design behind most chatbots.
  • Encoder-decoder models (like the original Transformer and T5) encode an input and decode an output, common in translation.

Beyond text

Transformers now process images (by splitting them into patches, as in the Vision Transformer), audio, video, proteins and robot actions. The same core idea — learning which pieces of input should attend to which — turned out to be remarkably general.

Go deeper with how diffusion models generate images, the mixture-of-experts explainer and the deep learning revolution in ten milestones.

In one sentence

A Transformer repeatedly lets every token look at every other relevant token, then predicts what comes next.

What to watch next

  • More efficient attention methods for very long contexts.
  • Alternative architectures such as state space models.
  • Hybrid models combining Transformers with other components.
  • Interpretability research into what attention heads learn.

Key takeaways

  • Transformers turn text into tokens, add position information and apply attention.
  • Attention lets each token weigh information from all other relevant tokens.
  • Stacking many layers builds rich representations; the model predicts the next token.
  • Encoder, decoder and encoder-decoder variants suit different tasks.
  • The architecture now powers models for images, audio, biology and more.

Frequently asked questions

Who invented the Transformer?

It was introduced in the 2017 paper “Attention Is All You Need” by Ashish Vaswani and colleagues at Google.

Why are Transformers better than older models?

They process sequences in parallel and capture long-range relationships, enabling training at enormous scale.

What is the “T” in GPT?

GPT stands for Generative Pre-trained Transformer.