Most modern image generators are diffusion models. The idea sounds strange at first: to learn how to create images, the model first learns how to destroy them.

The forward process: adding noise

Take a training image and add a little random noise. Then a little more. After many steps, the picture becomes pure static. This part needs no learning — it is just repeated noise.

The reverse process: learning to denoise

The model is trained to do the opposite: given a noisy image, predict the noise that was added so it can be removed. The 2020 paper Denoising Diffusion Probabilistic Models by Ho, Jain and Abbeel showed that this approach can produce high-quality images.

Generating a new image

To create something new, start from pure random noise and apply the denoiser many times. Each step removes a bit of noise until a coherent image appears. Different starting noise (the seed) gives different images.

Adding the text prompt

To follow a description, the denoiser is conditioned on text: a text encoder turns the prompt into numbers that guide every denoising step. A setting often called guidance scale controls how strictly the image follows the prompt.

Working in latent space

Denoising full-resolution pixels is expensive. Latent diffusion, introduced by Rombach and colleagues and used by Stable Diffusion, first compresses images into a smaller “latent” representation, runs diffusion there, then decodes the result back to pixels. This made high-quality generation possible on consumer GPUs.

Why it matters

The same recipe — learn to remove noise, guided by a condition — now powers video, audio and even 3D generation. Research continues on doing it in fewer steps, which makes generation faster and cheaper.

Key research milestones

  • 2015: researchers introduced diffusion-based generative modelling inspired by non-equilibrium thermodynamics.
  • 2020: Denoising Diffusion Probabilistic Models by Jonathan Ho, Ajay Jain and Pieter Abbeel showed diffusion could produce high-quality images.
  • 2021–2022: classifier-free guidance improved prompt-following; latent diffusion made generation efficient, leading to Stable Diffusion (2022), released openly.
  • 2022 onward: DALL·E 2, Imagen, Midjourney and later systems brought text-to-image generation to millions of users, and diffusion ideas extended to video and audio.

Controls you will see in image tools

  • Steps: how many denoising steps to run — more can add detail but takes longer.
  • Guidance scale: how strongly to follow the prompt versus producing natural-looking variety.
  • Seed: the random starting noise; reusing a seed with the same settings reproduces an image.
  • Negative prompts: things to avoid in the image.
  • Image-to-image and inpainting: starting from an existing image, or regenerating only part of it.

Limitations

Diffusion models can struggle with exact counts, hands, text and precise spatial relationships, though newer models have improved greatly. They also inherit biases from training data, and their training on copyrighted images is the subject of legal disputes.

An analogy

Imagine a sculptor who has seen millions of statues and learned to recognize what a statue looks like beneath rough stone. Shown a block covered in random chips (noise), the sculptor removes a little material at a time, each step guided by the description — “a horse in a field at sunset” — until a coherent image emerges. Diffusion models perform this process with numbers instead of stone, repeating the “remove a bit of noise” step dozens of times.

Related: how transformers work and our explainer on AI art and the creative debate.

What to watch next

  • Faster generation methods that need fewer denoising steps.
  • Diffusion and related methods for video and 3D content.
  • Court decisions on training image models with copyrighted works.
  • Provenance labels for AI-generated images.

Key takeaways

  • Diffusion models learn to reverse a process that gradually adds noise to images.
  • Generation starts from random noise and removes it step by step, guided by the text prompt.
  • Latent diffusion works in a compressed space, making generation efficient.
  • Settings like steps, guidance and seed control the result.
  • Diffusion now powers image, video and audio generation.

Frequently asked questions

Why does the same prompt give different images?

Each generation starts from different random noise unless you fix the seed.

What is Stable Diffusion?

An openly released latent diffusion model for text-to-image generation, launched in 2022.

Are diffusion models used for video?

Yes. Many video generation systems use diffusion-based methods adapted to sequences of frames.