Temperature and Top-P: The Two Dials, and Why They Fight

✓ sourced to official docs · Published 2026-08-19

A language model does not choose the next word. It produces a score for every token in its vocabulary, all at once, and something else decides what to do with those scores. Temperature and top-p are that something else.

They are not model settings. They are arithmetic applied to the model’s output after the model has finished thinking, which is why they are cheap to change and why they can be changed per request.

The two dials

Drag them. Every number below is computed, not illustrated.

Shape the distribution exact arithmetic, not an approximation

The cat sat on the ___

At temperature 1.0 the model's own distribution is used unchanged: mat takes about 58% of the probability, with the rest spread across floor, rug and a long tail ending at hypotenuse.

Cooling to 0.3 concentrates almost everything on mat (about 97%); heating to 2.0 flattens the distribution until mat holds only about 37% and the absurd continuations become genuinely reachable. Top-p then cuts the tail: at 0.9 only the few most probable tokens survive, and the survivors are renormalised to sum to 1 again.

Cool the temperature and the distribution collapses onto one token. Heat it and the absurd endings become reachable. Then lower top_p and watch the tail get cut off entirely — the faint bar shows what was there before the cut.

Temperature divides the scores before they are turned into probabilities. Per the GenerationConfig reference it is “the value used to module the next token probabilities”, and it defaults to 1.0 — the model’s own distribution, unchanged. Below 1 the gaps between scores are stretched, and the exponential in the softmax amplifies that into a sharp peak. Above 1 the gaps are squeezed and the distribution flattens toward uniform.

Top-p does something categorically different: it does not reshape anything, it deletes. The same reference defines it as keeping “only the smallest set of most probable tokens with probabilities that add up to top_p or higher”. Sort the tokens by probability, walk down the list adding them up, stop as soon as the running total reaches p, and throw away everything below. What survives is renormalised so it sums to 1 again.

The idea comes from Holtzman et al. (2019), which named it nucleus sampling — sampling “from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail”.

The word dynamic is the important one, and it is where the two dials collide.

Why they fight

Top-p does not keep a fixed number of tokens. It keeps however many are needed to reach p — so the size of the nucleus depends on the shape of the distribution, and temperature is what determines that shape.

import math

tokens = ["mat", "floor", "rug", "chair", "windowsill", "moon", "idea", "hypotenuse"]
logits = [4.2, 3.1, 2.8, 1.9, 0.6, -0.8, -1.5, -3.4]


def softmax(logits, temperature=1.0):
    scaled = [l / temperature for l in logits]
    m = max(scaled)                      # subtract the max: guards against overflow at low T
    exps = [math.exp(l - m) for l in scaled]
    total = sum(exps)
    return [e / total for e in exps]


def nucleus(probs, p=1.0):
    order = sorted(range(len(probs)), key=lambda i: probs[i], reverse=True)
    keep, cumulative = [], 0.0
    for i in order:
        keep.append(i)
        cumulative += probs[i]
        if cumulative >= p:              # "add up to top_p OR HIGHER" — checked after adding
            break
    mass = sum(probs[i] for i in keep)
    return [probs[i] / mass if i in keep else 0.0 for i in range(len(probs))]


for T in (0.3, 1.0, 2.0):
    probs = softmax(logits, T)
    kept = nucleus(probs, 0.9)
    surviving = sum(1 for x in kept if x > 0)
    print(f"T={T}  mat={probs[0]*100:5.1f}%   top_p=0.9 keeps {surviving} of {len(tokens)}")

print()
print("At top_p=0.9, temperature alone changes which tokens survive.")
print("The two settings are not independent knobs.")

Output

T=0.3  mat= 96.6%   top_p=0.9 keeps 1 of 8
T=1.0  mat= 58.2%   top_p=0.9 keeps 3 of 8
T=2.0  mat= 36.8%   top_p=0.9 keeps 5 of 8

At top_p=0.9, temperature alone changes which tokens survive.
The two settings are not independent knobs.

Your output

One thing to note if you are comparing these figures against the tool above: the percentages printed here are before top-p truncation, while the tool labels each bar with its probability after the survivors have been renormalised. At T=0.3 the printed 96.6% becomes 100% in the tool, because everything else was cut and mat is then the only outcome left. The faint bar behind each solid one is the pre-truncation value.

top_p never moved. It stayed at 0.9 for all three runs. Yet the number of tokens the model can actually produce went from one to five, purely because temperature changed the shape the nucleus was measured against.

This is why you will often see advice to alter one or the other but not both. It is not superstition: tuning them together means tuning two things that multiply, and it is genuinely hard to reason about what a change did.

What each dial is actually for

Temperature decides how much the model is allowed to be unsure. At 0 it is deterministic — always the single highest-scoring token, which is what people mean by greedy decoding. Around 1 it uses the distribution it actually learned. Above that it is deliberately being made less certain than it is.

Top-p is a floor on quality rather than a source of variety. Its job is removing the long tail of tokens the model gave almost no weight to — the ones that, sampled once in a thousand draws, produce the sentence that makes a reader distrust everything else on the page. That is exactly the degeneration Holtzman’s paper was about.

A useful way to hold them apart: temperature changes how likely things are, top-p changes what is possible at all. A token cut by top-p has probability zero. No temperature setting brings it back.

What this demo simplifies

The logits are illustrative. The eight candidates for “The cat sat on the ___” and their scores were chosen so the effects are visible; a real vocabulary has tens of thousands of entries and a far longer tail. Everything computed from those scores is exact — the same softmax and the same truncation a production sampler runs.
Sampling is not shown. The demo shows the distribution the sampler would draw from, not a draw. Actually sampling would add a random seed and obscure the thing being demonstrated.
Other filters exist. top_k keeps a fixed number of tokens rather than a fixed probability mass; min_p scales its threshold by the top token’s probability. Same family, different rule for where to cut.
Defaults vary by model. The reference gives 1.0 for both, but a model ships its own generation_config.json, and hosted APIs may differ again.

Where to go next

This is the last step in the chain. Tokens are the units the model reads and writes; embeddings are how it represents meaning; the context window is what it can see at once. Sampling is what turns the final numbers back into a word — and it is the only part of the chain you control directly on every request.

Sources

HuggingFace Transformers — GenerationConfig reference — checked 2026-08-19
The definition of temperature as the value used to modulate next-token probabilities with a default of 1.0, and of top_p as keeping only the smallest set of most probable tokens whose probabilities add up to top_p or higher, also defaulting to 1.0.
Holtzman, Buys, Du, Forbes & Choi — The Curious Case of Neural Text Degeneration (2019) — checked 2026-08-19
The paper that introduced nucleus sampling, and its description of sampling from the dynamic nucleus of the distribution to truncate the less reliable tail.