Double Espresso: What Is a Token?
Canonical episode: https://move37.app/cafe/en/episodes/double-espresso-token-en/
Published: 2026-08-10T21:38:50Z
Language: en

TRANSCRIPT

Welcome to Andy's Café, where machines brew and humans taste. Today we're serving a Double Espresso. Enjoy.

What Is a Token?

Somewhere in the small print of almost every AI product, there is a number followed by the word tokens. An assistant advertises a context window of two hundred thousand tokens. A price list charges by the million tokens, with input and output counted separately. A speed test brags about tokens per second. Tokens are the unit that language models are measured, sold and rationed in, and yet almost nobody stops to say what one actually is. So let us fix that, because once you have the picture, a lot of confusing AI arithmetic suddenly makes sense.

Here is the plain definition. A token is a piece of text that a language model treats as a single step. It is usually smaller than a sentence and very often smaller than a word. Before a model sees anything you have written, a component called a tokenizer cuts your text into these pieces and swaps each piece for a number, an identifier that points to an entry in a fixed vocabulary list. The model never reads letters the way you do. It works on sequences of identifiers.

The obvious question is why anyone would bother, when we already have a perfectly good unit called the word. The trouble is that words are an enormous and open-ended set. New ones appear constantly, languages glue them together differently, and a vocabulary that tried to hold every word in every language would be gigantic and would still meet something it had never seen. Going to the opposite extreme and using single letters solves the coverage problem but makes every sentence a very long sequence. Tokenizers sit in the middle. They learn a set of frequently useful chunks, so common material is represented compactly and anything unusual can still be built from smaller pieces.

Take an ordinary café order. If you write, “Two flat whites, please,” a tokenizer will probably keep those short, extremely common words whole, and it may treat punctuation or a space before a word as something worth counting too. Now write “decaffeinated” instead. That is a longer and much rarer word, and it may well be broken into two or three reusable fragments rather than kept as one unit. I am deliberately not telling you the exact split, because there is not one universal answer. The vocabulary and the cutting rules belong to a particular tokenizer, so the honest version of that sentence is always: it depends which tokenizer you are talking to.

Now follow one short prompt all the way through. You type, “The espresso machine is.” The tokenizer cuts that into its pieces and hands the model a list of identifiers. Inside, each identifier is looked up as a set of numbers, and the model processes that sequence to produce a score for every single token in its vocabulary: a rating of how well each one would continue the text. A decoding step then picks one, sometimes the highest scoring, sometimes a slightly more adventurous choice, depending on the settings. That chosen token is appended to the sequence, and the whole process runs again with the text now one token longer. The model does this over and over, one token at a time. What arrives on your screen is the tokenizer decoding the growing sequence back into readable text. You see a flowing sentence. Underneath, it was assembled in small pieces.

This is where tokens stop being trivia and start costing you something. The context window is measured in tokens, and everything competes for it: your instructions, the conversation so far, any documents you have pasted or the system has retrieved, and the answer being generated. When a long chat starts losing the beginning, the product may have reached that budget and had to drop or compress older material. The same unit often drives the bill and affects the waiting time. And there is no fixed conversion from words to tokens. The ratio shifts between languages, and it shifts again for code, for long numbers, for names and for unusual formatting. A rough estimate is fine for planning; treat it as a law and you will be wrong exactly when it matters.

A few things are worth correcting while we are here. One token is not one word, not one syllable and not one character. It is whichever useful unit that tokenizer represents. A quoted token limit cannot be converted into a precise universal page count, so be suspicious of anyone who does it confidently. Two models can count the identical passage differently, which means token counts are not portable between model families. And tokenization is the text interface to the model. It governs how much fits and how much computation a passage takes; it does not tell you whether the model actually understood a word of it.

If you keep one thing, keep this. A token is the model's unit of text, not yours. Every limit, price and speed you are quoted is counted in the model's units, never in your words.

That's all for now. The café is always open. Come back soon.

MORE INFORMATION
TEXT LICENSE
This transcript, description and original Andy's Café editorial text are licensed under Creative Commons Attribution 4.0 International (CC BY 4.0).
License: https://creativecommons.org/licenses/by/4.0/
Attribution: Andy's Café — https://move37.app/cafe/en/episodes/double-espresso-token-en/
Indicate changes when adapting. Identified third-party quotations and linked source material remain under their own terms.

AUDIO AND OTHER MATERIAL
The composed episode has separate component terms because its piano cues are third-party material. Artwork and the Andy's Café brand are not included in the CC BY licence.
Rights map: https://move37.app/cafe/welcome/#reuse
Contact: hello@move37.app