Double Espresso: What Are Temperature and Sampling?
Canonical episode: https://move37.app/cafe/en/episodes/double-espresso-temperature-sampling-en/
Published: 2026-08-29T02:49:47Z
Language: en

TRANSCRIPT

Welcome to Andy's Café, where machines brew and humans taste. Today we're serving a Double Espresso. Enjoy.

What Are Temperature and Sampling?

Ask a language model the same open question twice, and you may receive two different answers. That can look mysterious. Has the model changed its mind? Did it learn something between attempts? Usually, neither. One important part of the explanation comes after the model has calculated its possible next pieces of text. The product still has to choose one of them. Sampling means drawing that choice according to probabilities rather than always taking the leader. Temperature changes how strongly the draw favours the leaders.

At each step, a language model assigns scores to possible next tokens. A token may be a word, part of a word, or punctuation. Those scores can be converted into probabilities. Imagine the unfinished sentence, “The coffee smelled of”. The model might give “chocolate” the largest probability, “orange” a smaller one, and “rain” a much smaller one. These are not votes about reality. They describe which continuations fit the preceding text according to the model.

One decoding rule is simply to take the most probable token every time. This is often called greedy decoding. It removes the random draw at that step. Sampling does something different: it treats the probabilities as chances and draws one candidate. Chocolate may be most likely without being inevitable. If orange is drawn instead, it becomes part of the sentence, and the model calculates a new set of possibilities from that changed context. One early choice can therefore send the whole answer down another path.

Temperature acts before that draw. It rescales the model's scores and changes the shape of the probability distribution. A lower temperature sharpens the differences. In our example, chocolate becomes more dominant, while orange and rain become less likely. A higher temperature flattens the differences. Chocolate may still lead, but the alternatives get a better chance. Temperature does not write new candidates, teach the model new facts, or look ahead to judge which sentence is true. It changes the odds among possibilities the model already produced.

This explains why people often call a low temperature more predictable and a high temperature more varied. But the familiar labels “precise” and “creative” can mislead. A low setting can repeat the same confident mistake because the model's most probable continuation is not necessarily correct. A high setting can surface an apt, surprising phrase, or merely an odd one. Temperature controls concentration, not truth, intelligence, taste, or safety.

The number is not a universal dial across models. A setting that produces restrained prose in one system may behave differently in another because the original score distributions and the surrounding decoding rules differ. Compare settings within the actual product and task, not by the label alone.

There is also a small technical trap around zero. In the usual formula, temperature appears as a divisor, so a literal temperature of zero is not mathematically defined. A product that offers zero normally treats it as an instruction to use its most deterministic decoding route, often choosing the leading token. That is an interface convention. Two products with the same visible slider may use different hidden rules. Even one product need not produce perfectly identical text after its model, software, or hardware changes.

Sampling is often narrowed before the draw. Top kay sampling keeps a fixed number of the highest-ranked candidates and excludes the rest. If kay were five, only five candidates would remain, even when the fifth was extremely unlikely. Top pee sampling, also called nucleus sampling, uses a moving boundary instead. It keeps the smallest group of leading candidates whose combined probability reaches a chosen share. When the model is highly certain, that group may be tiny. When it is uncertain, the group can widen.

Temperature and top pee are therefore not two names for one control. Temperature reshapes the relative probabilities. Top pee cuts away the low-probability tail after ranking the candidates. Systems may apply both, add repetition penalties, impose structured-output rules, or override settings for particular tasks. This is why copying a handful of parameter values from one model into another is not a scientific guarantee of matching behaviour.

Randomness can also be managed with a seed: a starting value for a pseudorandom number generator. On one fixed software setup, repeating a seed may reproduce the same draws. But a seed is not a permanent fingerprint for an answer. Change the model, prompt, hidden instructions, numerical implementation, or order of operations, and the same seed can lead elsewhere. Some hosted systems do not promise exact reproducibility at all.

The practical rule is to begin with the job. For extracting fields from a document or producing rigid data, restrained decoding and structural validation are sensible. For generating several names, metaphors, or possible plans, sampling can provide useful variation. In both cases, the model's output still needs whatever checking the task deserves. Turning down temperature does not replace evidence; turning it up does not replace judgment.

So when two answers differ, do not imagine two little opinions hidden inside the machine. At every step the model supplied possible next tokens, the decoder selected one, and that selection changed the possibilities that followed. Sampling makes the probabilistic draw. Temperature changes how heavily the draw favours the most likely candidates. Together they help explain variation, but neither tells you whether the finished answer is true.

That's all for now. The café is always open. Come back soon.

MORE INFORMATION
TEXT LICENSE
This transcript, description and original Andy's Café editorial text are licensed under Creative Commons Attribution 4.0 International (CC BY 4.0).
License: https://creativecommons.org/licenses/by/4.0/
Attribution: Andy's Café — https://move37.app/cafe/en/episodes/double-espresso-temperature-sampling-en/
Indicate changes when adapting. Identified third-party quotations and linked source material remain under their own terms.

AUDIO AND OTHER MATERIAL
The composed episode has separate component terms because its piano cues are third-party material. Artwork and the Andy's Café brand are not included in the CC BY licence.
Rights map: https://move37.app/cafe/welcome/#reuse
Contact: hello@move37.app