Almost every modern AI model is trained with some form of gradient descent. The idea fits in one picture.

The hiker in the fog

Imagine standing on a hillside in thick fog, trying to reach the lowest point of the valley. You can’t see far, but you can feel the slope under your feet. Strategy: take a step downhill, feel the slope again, repeat. That’s gradient descent.

  • The landscape is the model’s error (the loss) for every possible setting of its parameters.
  • The slope is the gradient.
  • The step size is the learning rate.

A tiny example in Python

Find the value of x that minimizes f(x) = (x − 3)². The answer is obviously 3; let’s see the algorithm find it.

x = 0.0             # starting guess
learning_rate = 0.1
for step in range(50):
    gradient = 2 * (x - 3)   # derivative of (x - 3)**2
    x = x - learning_rate * gradient
print(x)  # very close to 3.0

Each step moves x a fraction of the way toward 3. A real neural network does the same with billions of parameters at once, using backpropagation to compute the gradient.

The learning rate matters

  • Too small: training takes forever.
  • Too large: steps overshoot the minimum and training becomes unstable.

Modern optimizers such as Adam adapt the step size automatically for each parameter.

Stochastic gradient descent

Computing the exact gradient over all training data is too slow, so models estimate it from small random mini-batches. This “stochastic” version is noisier but much faster, and the noise can even help.

Why training fails

Bad learning rates, poorly scaled data, or gradients that vanish or explode in deep networks. Many architectural tricks — normalization layers, residual connections — exist to keep gradient descent working well.

Variations you will hear about

  • Batch gradient descent computes the gradient on the whole dataset each step — accurate but slow for big data.
  • Stochastic gradient descent (SGD) uses one example or a small random mini-batch at a time — noisy but fast, and the noise can even help escape poor solutions.
  • Momentum keeps a running average of past gradients, like a ball rolling downhill that builds speed and smooths out zig-zags.
  • Adam combines momentum with per-parameter step sizes and is the default optimizer for most deep learning — see our Adam explainer.
  • Learning rate schedules start with a warm-up and gradually reduce the step size, common in training large models.

From one parameter to billions

The tiny example above adjusts one or two numbers. A large language model has billions of parameters, but the principle is identical: compute how the loss changes with respect to every parameter — using backpropagation to do this efficiently — and nudge each one slightly downhill. Training repeats this millions of times over huge datasets on many accelerators in parallel.

A visual way to think about it

Picture the loss as a landscape of hills and valleys, where each location is a possible setting of the model’s parameters and height is the error. Gradient descent is a hiker who can only feel the slope underfoot, taking steps downhill. With two parameters you can draw this as a contour map; with billions, nobody can visualize it — but the same local, step-by-step logic still works remarkably well, which is one of the surprising facts of modern machine learning.

Try it yourself

Change the learning rate in the Python example to 0.001, 0.1 and 1.5 and watch what happens: very small steps crawl, moderate steps converge, and very large steps overshoot and may diverge. This five-minute experiment builds intuition faster than any explanation.

What to watch next

  • New optimizers that train large models with less compute or memory.
  • Research on why gradient descent finds solutions that generalize well.
  • Training stability techniques for very large models.
  • Low-precision training methods that speed up optimization.

Key takeaways

  • Gradient descent improves a model by repeatedly stepping downhill on the loss.
  • The learning rate controls step size; too large diverges, too small is slow.
  • Stochastic and mini-batch versions make training on big data practical.
  • Momentum and Adam improve speed and stability.
  • Backpropagation computes gradients efficiently for billions of parameters.

Frequently asked questions

What is a loss function?

A number measuring how wrong the model’s predictions are. Training tries to minimize it.

Why does gradient descent sometimes fail?

Bad learning rates, poorly scaled data, or problematic loss landscapes can cause slow progress or divergence.

Is gradient descent used in all AI?

It is the main training method for neural networks, though other optimization methods exist for other models.