Every time a language model writes a word, it does the same small calculation: it turns a list of raw scores into probabilities, picks or samples one token, and during training measures how surprised it was by the correct answer. Two formulas do almost all the work, softmax and cross-entropy. This guide walks through both with numbers you can check on a calculator, explains what the “temperature” setting really does to them, and shows the one numerical trick every implementation uses. If you know what a logarithm is, you can follow it.
Step 1: logits, the raw scores
A language model ends with a layer that outputs one number for every token in its vocabulary. These numbers are called logits. They can be positive or negative, and they do not add up to anything in particular. A logit of 2.0 for the word “cat” and 1.0 for “dog” says the model prefers “cat,” but not by how much in probability terms.
For our example, imagine a vocabulary of just three tokens with these logits:
| Token | Logit |
|---|---|
| A | 2.0 |
| B | 1.0 |
| C | 0.1 |
Step 2: softmax turns scores into probabilities
Softmax does two things. It makes every score positive by applying the exponential function, and it divides by the total so the results add up to 1. For a logit list z, the probability of token i is:
p_i = exp(z_i) / sum over j of exp(z_j)
With our numbers:
- exp(2.0) = 7.389
- exp(1.0) = 2.718
- exp(0.1) = 1.105
- Sum = 11.212
The probabilities are 7.389 / 11.212 = 0.659, 2.718 / 11.212 = 0.242 and 1.105 / 11.212 = 0.099. They add up to 1.
Two properties are worth remembering. First, the ordering of the tokens never changes: the largest logit always gets the largest probability. Second, softmax only cares about differences between logits. Adding 100 to every logit gives exactly the same probabilities, a fact we use below.
Step 3: cross-entropy measures surprise
During training we know which token should come next. Suppose the correct token is A. The cross-entropy loss for one prediction is simply the negative natural logarithm of the probability the model gave to the correct token:
loss = -ln(p_correct)
If the correct token is A, the loss is -ln(0.659) = 0.417. If the correct token were C, which the model gave only 0.099, the loss would be -ln(0.099) = 2.313. A confident correct answer costs little; a confident wrong answer costs a lot. A model that gave the correct token a probability of 1 would have a loss of 0, and the loss grows without limit as that probability approaches 0.
Over a whole batch, you average the loss across all the positions. Gradient descent then nudges the model’s weights to lower this average. The reason this pairing is popular is that the gradient of softmax followed by cross-entropy with respect to the logits is very clean: it equals the predicted probability minus the target (1 for the correct token, 0 for the others). In our example, if A is correct, the gradient on A’s logit is 0.659 - 1 = -0.341, which pushes A’s logit up, while B and C get +0.242 and +0.099, which push their logits down.
Perplexity, a friendlier number
Researchers often report perplexity instead of loss. It is the exponential of the average loss. If the two predictions above (loss 0.417 and 2.313) were all we had, the average loss would be 1.365 and the perplexity about 3.9. You can read it loosely as “the model was as uncertain as if it were choosing among about 3.9 equally likely options.” Lower is better, but perplexity values are only comparable between models that use the same tokenizer and test data.
Temperature: stretching the logits
Many chat tools expose a temperature setting. Mathematically, temperature T divides the logits before softmax: p_i = exp(z_i / T) / sum of exp(z_j / T).
Using our logits:
| Temperature | P(A) | P(B) | P(C) |
|---|---|---|---|
| 0.5 | 0.864 | 0.117 | 0.019 |
| 1.0 | 0.659 | 0.242 | 0.099 |
| 2.0 | 0.502 | 0.304 | 0.194 |
Low temperature sharpens the distribution, so the top token dominates and outputs become more repeatable. High temperature flattens it, so unlikely tokens are sampled more often and outputs become more varied, and sometimes less coherent. Temperature near 0 approaches always picking the top token. Training normally uses temperature 1; the setting is applied at generation time.
The numerical trick: subtract the maximum
Computing exp(z) directly can overflow when logits are large: exp(1000) is too big for standard floating-point numbers. Because softmax depends only on differences between logits, implementations subtract the largest logit first. With our values, the shifted logits become 0.0, -1.0 and -1.9. The largest exponent is now exp(0) = 1, so nothing overflows, and the probabilities come out identical. The same idea underlies the “log-sum-exp” trick used to compute the loss stably. Deep learning libraries do this for you, and usually offer a combined softmax-plus-cross-entropy function that is more stable than calling the two separately.
Where softmax shows up elsewhere
The same function appears inside the attention mechanism of transformers, where it converts similarity scores between tokens into weights that add up to 1 (see the original transformer paper). Understanding it once pays off in several places in the architecture.
Try it yourself
- Pick your own three logits and compute the probabilities by hand.
- Compute the loss when each token in turn is the correct one.
- Halve and double the logits and watch the probabilities sharpen and flatten.
- Add 10 to all three logits and confirm nothing changes.
For the wider set of topics to study next, see our overview of the math you need for AI.
What to watch next
- Alternatives to plain softmax in attention, such as variants designed to reduce numerical cost.
- Sampling methods layered on top of softmax, such as top-p, which many APIs expose.
- Research on calibration, which asks whether a model’s stated probabilities match how often it is right.
- Low-precision training, where numerical stability tricks like the one above matter more.
Key takeaways
- Logits are raw scores; softmax converts them into probabilities that sum to 1.
- Cross-entropy loss is -ln of the probability given to the correct token.
- The gradient of softmax plus cross-entropy on the logits is simply predicted probability minus target.
- Temperature divides the logits: low values sharpen the distribution, high values flatten it.
- Implementations subtract the maximum logit to avoid overflow without changing the result.
Frequently asked questions
Why use the natural logarithm in the loss?
It turns products of probabilities into sums and gives a smooth, steep penalty for confident mistakes. Using another base only rescales the number.
Does temperature change what the model knows?
No. It changes only how the model’s probabilities are turned into a choice at generation time. The underlying logits stay the same.
Is softmax the only way to get probabilities?
No, but it is the standard for choosing among many classes or tokens. Binary problems often use the related sigmoid function.
