In 2015, two Google researchers, Sergey Ioffe and Christian Szegedy, published a technique that changed everyday practice in deep learning almost overnight. Batch normalization made it possible to train deeper networks faster, with higher learning rates and less fiddling with initialization. It quickly became a standard component of image models, and its ideas live on in the normalization layers of today’s transformers.

The problem

Deep networks are stacks of layers, each transforming the output of the previous one. During training, every layer’s parameters change at each step — which means the distribution of inputs that each following layer sees keeps shifting. The authors called this internal covariate shift.

Practically, deep networks in 2015 were hard to train. They needed small learning rates and careful initialization, and they could stall when signals became too large or too small as they passed through many layers.

The idea

Batch normalization adds a simple operation inside the network: for each feature, standardize the values across the current mini-batch of training examples.

  1. Compute the mean and variance of the feature over the batch.
  2. Subtract the mean and divide by the standard deviation, so values are centered around zero with unit spread.
  3. Multiply by a learned scale and add a learned shift.

The last step is important: it lets the network undo the normalization if that turns out to be useful, so the layer loses no expressive power.

At inference time, when there may be only one example, the network uses running averages of the mean and variance collected during training instead of batch statistics.

The results

The paper reported dramatic speedups. Applied to a state-of-the-art image classification network of the time, batch normalization reached the same accuracy in a fraction of the training steps, and an ensemble of batch-normalized networks improved on the best published ImageNet results.

Practitioners also noticed side benefits:

  • Higher learning rates became possible without training diverging.
  • Less sensitivity to initialization.
  • A mild regularization effect, because each example’s normalization depends on the other examples in its batch, adding a little noise.

Why does it actually work?

The original explanation — reducing internal covariate shift — was later questioned. A 2018 study by Shibani Santurkar and colleagues at MIT argued that batch normalization’s main benefit is making the optimization landscape smoother, so gradients are more predictable and larger steps are safe. This is a good reminder that a technique can be extremely useful before anyone fully understands why.

Limitations

Batch size dependence. Statistics computed over small batches are noisy, which hurts performance when memory forces small batches — common with large images or video.

Train/test mismatch. Using batch statistics in training and running averages at inference can cause subtle differences in behavior.

Awkward for sequences. In language models, sequences have different lengths and tokens are processed differently, making batch statistics a poor fit.

What came next

These limitations led to alternatives that normalize in other ways:

  • Layer normalization (2016) normalizes across the features of a single example rather than across the batch. It does not depend on batch size and became the standard in transformers.
  • RMSNorm, a simplified variant that only rescales by the root mean square, is widely used in modern large language models because it is cheaper.
  • Group normalization and related methods serve computer vision cases with small batches.

The takeaway

Batch normalization showed that small architectural details can have outsized effects on whether deep networks train at all. Even where it has been replaced, the core idea — keep the signals flowing through a deep network in a well-behaved range — is built into virtually every modern model.

Frequently asked questions

Is batch normalization the same as normalizing my input data?

No. Normalizing input data (for example scaling pixel values) happens once, before training. Batch normalization happens inside the network, at many layers, at every training step, using statistics of the current mini-batch, and it includes learned scale and shift parameters.

Why don’t language models use batch normalization?

Sequences vary in length, models often run with small or uneven batches, and text generation processes tokens one at a time. Statistics computed across a batch are a poor fit for that setting. Layer normalization and RMSNorm normalize each example on its own, which makes them the standard choice in transformers.

Does batch normalization prevent overfitting?

It has a mild regularizing effect, because each example is normalized with statistics that depend on the other examples in its batch, which adds a little noise during training. It is not a substitute for proper regularization or more data, but it can reduce the need for techniques such as dropout in some models.

What is the most common bug with batch normalization?

Forgetting to switch the model to evaluation mode at inference time. In training mode the layer uses batch statistics; in evaluation mode it uses the running averages collected during training. Mixing them up produces inconsistent predictions, especially with a batch size of one.