Tiramisu, Part One: How a Language Model Computes

Tiramisu, Part One: How a Language Model Computes

Welcome to Andy's Café. This is a Tiramisu: a long episode that takes a set of concepts and builds them up, one on top of the other, until they make a picture. The concepts are the ones the operator of this café collected over the autumn, from papers and interviews about artificial intelligence: seventeen of them, plus a few bridging ideas that the seventeen cannot be explained without.

The Tiramisu comes in three parts, each released on its own. This is Part One, and it is about what happens inside a language model when it runs. Part Two is about how a model is trained, and about the ways people get more reasoning out of a trained model. Part Three is about how researchers look inside a model, and it ends with an old observation about which tasks are hard for machines.

A word about how to listen. Nobody remembers everything from a long episode, and this one does not expect you to. When a chapter needs an idea from an earlier chapter, it says so and brings the idea back in a sentence or two before going on. Each part opens with a short recap of what it builds on and closes with a short summary of where it got to. If you listen to a part twice, the recaps will be the sections to skip.

Here is the one fact that organises all of Part One. A language model, once it has been trained, is a fixed procedure. It takes in a sequence of text and produces, for the next piece of text, a set of probabilities. Then one piece is chosen, added to the sequence, and the procedure runs again. Everything in this part is about what that procedure does on one run: what it takes in, what it computes with, what it keeps from earlier runs, and what it does not do. The procedure does not change itself while it runs. How it got its shape in the first place is the subject of Part Two.

The chapters build in order. First the objects the model computes with: tokens, parameters, activations and vectors. Then the quantity that training tries to reduce, the loss, because it explains what a model's output means. Then embeddings and the idea of a latent space, where representations live. Then the forward pass, which is the computation itself; the residual stream, which is how information travels through it; attention and the key-value cache, which is how the model uses earlier text; inference, which is how a model is actually run and how its output is chosen; and mixture of experts, a way of making a model very large without making every run expensive.

Everything here describes one family of systems, the transformer language models that generate text one token at a time. It is an important family, and many of the systems people use in twenty twenty-six belong to it, but not all of them, and image generators, robot controllers and systems that combine several models each need distinctions of their own.

One last thing before we start. Some of what follows is settled and has been demonstrated many times. Some of it is from recent papers or from company measurements, and some of it is argument in interviews. The chapters say which is which. A result belongs to the model, the data and the procedure that produced it, and an explanation of why it happened can stay a hypothesis even when the measurement is solid.

Tokens and Parameters

Start with what a language model is, in the plainest terms. It is a very large mathematical function. You give it a sequence of text, and it gives back a list of probabilities: one probability for each possible next piece of text. That is all it does. Every other ability, from answering questions to writing code, is built out of that one operation, repeated.

The function has two kinds of numbers in it. There are the numbers that are fixed once training is over, which are called parameters. And there are the numbers that are computed fresh every time the function runs on a particular input, which are called activations. This chapter is about the first kind and about the pieces of text the function works on. The next chapter is about the second kind.

A model does not read letters or words directly. Before the text reaches it, a separate program called a tokenizer cuts the text into pieces called tokens. A token can be a whole word, part of a word, a punctuation mark, or in some systems a few raw bytes. The tokenizer has a fixed list of all the pieces it knows, called the vocabulary, and a vocabulary often has tens or hundreds of thousands of entries. Each entry has a number. So after tokenization, the text has become a list of numbers, and that is what the model sees.

Two things follow from this that are worth holding on to. The first is that the numbers are labels, not quantities. Token number five thousand is not close in meaning to token number five thousand and one; the numbering is arbitrary. The model has to learn what each token means from scratch, and the next chapters explain how it represents that meaning. The second is that different tokenizers cut the same text differently. One model's tokenizer might turn a sentence into twelve tokens and another's into fifteen. That is why a token count is not a universal measure of how much text there is, and why a price per token from one company cannot be compared directly with another's.

Now the parameters. During training, which Part Two describes, the model's numbers are adjusted, many times, until the function produces good probabilities. When training stops, those numbers are frozen. They are the parameters. When people say a model has seventy billion parameters, they mean seventy billion of these frozen numbers. Most of them sit in large grids of numbers called matrices, and a matrix is simply a rule for turning one list of numbers into another list of numbers by multiplying and adding. Beyond the matrices there are smaller sets of parameters with special jobs, such as the ones that keep values in a sensible range and, in some models, the ones that decide which part of the model handles a given token.

It is tempting to think of a parameter as a stored fact, as if one number held the capital of France and another held the boiling point of water. That is not how it works, and the picture leads people astray. A single fact that the model can reproduce depends on a great many parameters working together, and any one parameter contributes to a great many different behaviours. The parameters encode the function as a whole, not a list of entries. This is the first of several places in the Tiramisu where the right picture is a computation rather than a filing cabinet, and Part Three, on looking inside models, comes back to it.

One more distinction, which the chapter on mixture of experts will need. A model can have more parameters stored than it uses on any one token. In many of the largest designs today, only part of the model is active for each token, and the part that is active changes from token to token. So two numbers describe such a model: how many parameters it stores in total, and how many it uses for a given token. Both are counts of frozen numbers. Neither tells you, on its own, how good the model is.

So far, then: text becomes a list of token numbers; the model is a function with frozen numbers called parameters; and the function turns the token list into probabilities for the next token. What happens between the input and the output, the numbers the function computes along the way, is the subject of the next chapter.

Activations, Vectors and the Loss

The previous chapter left the model as a function with frozen parameters, taking a list of tokens in and giving probabilities out. This chapter is about the numbers in between, and about the number that training tries to make small, because without it the output of the model cannot be read correctly.

When the model runs on a particular input, it computes a great many intermediate numbers. These are the activations. They depend on the input: the same parameters produce different activations for different text. A parameter belongs to the model; an activation belongs to one particular run of the model on one particular text. During ordinary use the parameters never change, while the activations depend on the input and on the stage of the computation. Some activations are kept for a while in a store called the cache, because the model needs them again on later runs; the attention chapter explains which ones and why.

The activations come in bundles, and a bundle is called a vector. A vector is an ordered list of numbers, nothing more. The list that represents one token at one stage of the computation might have four thousand or more numbers in it, and it is convenient to think of such a list as a point in a space with that many dimensions. The space is impossible to picture, but the language of points, distances and directions still works in it, and researchers use that language constantly. A direction in this space is a vector you can add to a state to move it. When a later chapter says that a concept is represented by a direction, it means that adding that vector to the model's state pushes the model's behaviour in a way that has to do with the concept. Whether such a direction exists for a given concept, and whether the model actually uses it, are questions for Part Three.

A matrix, from the previous chapter, is a rule that turns one vector into another by a fixed set of multiplications and additions. Such a rule is called linear. If a network were made only of linear rules, one after another, the whole thing would collapse into one linear rule, and it could not represent a nonlinear function. So a network also has simple nonlinear steps, where a number is passed through a fixed curve. Those steps, placed between the linear ones, are what let the network represent complicated functions. Keep this in mind for the chapter on the forward pass, where the steps appear in order.

Now the loss. Training needs a single number that says how badly the model is doing, so that it can be made smaller. For a language model trained on text, that number is built like this. Take a piece of text, feed the model everything up to some point, and look at the probability it gives to the token that actually comes next in the text. If that probability is high, the model did well; if it is low, it did badly. The loss is a measure of how low that probability was, added up over an enormous number of positions in an enormous amount of text. The particular measure in use is called cross-entropy, and the detail of its formula does not matter here. What matters is what it measures: how well the model predicts the text it was given.

That last point deserves a moment, because it explains what a model's output means. The model is rewarded for predicting the continuation that was actually in the training text. It is not rewarded for the continuation being true, or kind, or the best possible answer. If the training text contains a common mistake, a model that predicts the mistake has done its job. Part Two describes the later stages of training that try to change what the model is rewarded for. The starting point, though, is prediction of text, and a great deal of what models do well and badly follows from that.

The loss comes with a companion called the gradient, and it needs a name now even though its explanation belongs to Part Two. The gradient is the answer to the question: if each parameter were nudged a tiny amount, which way would the loss move? Training uses that answer to guide repeated updates to the parameters that aim to make the loss smaller. That is the mechanism of learning, and Part Two spends two chapters on it.

Three kinds of data have three different jobs in this process, and the names come up often. Training data is what the loss is computed on; it drives the updates. Validation data is held back from training and used to choose settings and to decide when to stop. Test data is held back from both and used, once, to estimate how good the final model is. If a researcher keeps going back to the test data to pick between models, it stops being a fair test, because the choices have been tuned to it. A test score is an estimate for text like the test data; how well a model does on something different is a separate question, and it is always the question that matters in practice.

One last distinction, which comes up every time someone says a model has learnt something from a conversation. When you give a model examples inside your prompt and it starts following their pattern, nothing in the model has changed. Its parameters are exactly as they were. What has changed is the input, and so the activations. This is called in-context adaptation, and it is a real and useful ability, but it is not learning in the sense of this chapter. Learning changes the parameters, through the loss and the gradient, and it only happens when somebody runs training. The chapter on the backward pass in Part Two says why a conversation on its own never does that.

So now we have the full set of objects. Tokens, the pieces of text. Parameters, the frozen numbers that define the function. Activations, the numbers computed on a particular run, bundled into vectors. And the loss, the number that training makes small, which tells us that the model's output is a prediction of text. The next chapter looks at the very first thing the model does with a token: it turns the token's label into a vector.

Embeddings and the Latent Space

The last chapter ended with a token as a bare label, a number that means nothing in itself, and with the idea of a vector as a list of numbers that can be treated as a point in a space. This chapter joins the two: it is about how a model turns a label into a vector, and about the space those vectors live in, which people call the latent space.

The first thing a model does with a token is look it up in a table. The table has one row per vocabulary entry, and each row is a vector, typically a few thousand numbers long. This row is the token's embedding, and the table is the embedding matrix. The numbers in it are parameters: they were learnt during training, and they are frozen now. Looking up the embedding gives the rest of the network something it can compute with. For a given model, the embedding of a token is the same wherever that token appears. The word "bank" gets the same starting vector in a sentence about rivers and in a sentence about money.

That cannot be the whole story, because the two banks mean different things. So the model does not stop at the lookup. As the computation proceeds through its layers, which the next chapter describes, the vector at each position is changed again and again, and the changes depend on the surrounding text. By the middle of the model, when the text before the word differs, the vector at the position of "bank" is different in the river sentence and in the money sentence. These later vectors are called contextual representations. The lookup gives every occurrence of a token the same starting point; the context then pulls them apart. One vocabulary entry does not mean one meaning inside the network.

The idea that vectors can carry meaning has a history worth knowing, because it set expectations that still shape how people talk. In twenty thirteen, Tomas Mikolov and colleagues trained simple models to predict words from their neighbours and found that the resulting vectors had regularities. Words used in similar contexts ended up with similar vectors, and some relationships could be found by vector arithmetic: the famous example is that the vector for king, minus the vector for man, plus the vector for woman, lands near the vector for queen. That result made geometry a respectable kind of evidence about what a model has learnt. It did not show that every concept is one vector, and it did not show that arithmetic on vectors is a reliable way to reason. Those stronger claims are still open, and Part Three returns to them.

Now the word latent. In this field, a latent representation is any internal representation that is neither the raw input nor the output you asked for. It is a hidden intermediate. The embedding is one; the contextual vector in the middle of the model is another. So latent space is not one place in the model. It is a name for wherever the model's internal vectors live, and different parts of the model, and different kinds of model, have different latent spaces with different jobs.

Two other kinds of model show how broad the term is. An autoencoder is a pair of networks: one encodes an input into an internal representation, often a shorter one, and the other tries to reconstruct the input from it. That internal representation is a latent representation, and the training objective is reconstruction, which is a different objective from predicting text. An image generator of the kind called latent diffusion works on a compressed version of the image rather than on pixels, because the compressed version is much smaller and so much cheaper to work with.

In the well-known Stable Diffusion setup, an image five hundred and twelve pixels on a side, with three colour channels, becomes a grid sixty-four on a side with four channels. That is sixty-four times fewer positions, but forty-eight times fewer numbers in total, because the channels went from three to four. The point of the example is that a compression can be described in more than one way, and the way that matters depends on what you are counting.

Back to language models. The main latent space of a transformer is the vector at each position as it moves through the layers, which the chapter after next names the residual stream. It is worth saying what this space is not. It is not a tidy coordinate system in which each axis is a human concept. A concept, if the model represents it at all, might be a direction, or several directions, or a relationship between positions, or a curved set of states. All of these have been found.

The curved kind is the easiest to picture. Researchers have found that some models represent cyclic quantities, such as the days of the week, on something very close to a circle inside the latent space. Monday is next to Tuesday, Sunday is next to Monday, and going round the circle steps through the week. This is measured geometry, not a figure of speech, and it has a practical consequence. If you want to change the represented day, moving along the circle works. Moving in a straight line across the inside of the circle produces a state the model does not normally use, and in Goodfire's tests the behaviour that followed was less natural. A set of states with this kind of structure is called a manifold: a shape inside the space on which the model's actual states lie, which can be described with fewer coordinates than the whole space has.

Work in twenty twenty-six by the interpretability company Goodfire took this seriously for intervention. The researchers first estimate the shape of the states a model uses, and of the outputs it produces, and then make changes that follow that shape rather than cutting straight across it. In the language and video models they studied, changes that followed the geometry produced more natural changes in behaviour than straight-line changes did. That is evidence from particular experiments, and it does not yet amount to a general method for steering any behaviour in any model.

Two lessons close the chapter, and both will be needed in Part Three. First, when a researcher attaches a name to a direction in latent space, the name is an interpretation that has to be tested, not a label the model supplied. Second, being able to read some information out of a vector does not show that the model uses that information when it answers. To show use, you have to change the vector and see whether the answer changes in the way the reading predicts. Reading is cheap; intervention is the evidence.

The next chapter follows the vector from the embedding table through the whole computation to the probabilities at the end. That computation is the forward pass.

The Forward Pass

We now have the ingredients: tokens, each turned into a starting vector by the embedding table; frozen parameters, mostly matrices; and activations, the vectors computed along the way. The forward pass is the computation that joins them. It starts with the token vectors and ends with probabilities for the next token. This chapter walks through it once, slowly, and then says what its cost means.

The overall shape is simple. Each token in the input gets its embedding vector. The vectors then pass through a stack of layers, dozens of them in a large model, and each layer changes every position's vector a little. At the end, the vector at the last position is turned into a score for every entry in the vocabulary, and the scores are turned into probabilities by an operation called a softmax, which raises each score to an exponential and divides by the sum of them all, so that they are all positive and add up to one. The scores before the softmax are called logits, and the word will come up again. Choosing one token from the probabilities is a separate step, done after the forward pass, and the inference chapter covers it.

The design comes from a twenty seventeen paper by Ashish Vaswani and colleagues, which introduced the transformer for translation. Their original system had two halves: an encoder that read the source sentence, with six layers, and a decoder that produced the translation, also with six layers, which could look at the encoder's output as it went. Many of today's language models keep only the decoder half, so they are called decoder-only, and the rest of this chapter describes that design.

Inside a layer, each position's vector goes through three kinds of operation. The first is attention, which lets a position combine information from other positions, and which is important enough to get a chapter of its own. The second is a feed-forward block, which works on one position at a time: it maps the vector to an intermediate one, usually longer, passes each number through a simple nonlinear curve, and projects the result back down to the original length. This is where the linear and nonlinear steps from the activations chapter appear in practice. The third is a pair of bookkeeping operations. One is normalisation, which rescales the vector so its numbers stay in a workable range. The other is the residual connection, which adds the block's result to the vector that came in rather than replacing it, and which gets the next chapter to itself.

One more ingredient is needed, and it is easy to forget. Attention on its own does not know the order of the tokens; it sees a set of positions, not a sequence. So the model has to be told where each token is. The original transformer added a position-dependent pattern to each embedding. Many recent models instead encode position inside the attention computation, with a method called rotary position embedding, which rotates the vectors by an angle that depends on position. The details do not matter here. What matters is that position handling is a design choice, and that explicit position information is what lets attention make full use of word order.

What the parameters do, and what they do not do, can now be said precisely. The parameters fix which transformations are available at each step. The input fixes which activations those transformations produce. During a forward pass the model does not look anything up in a store of facts. In the model described here, there is no store. Whatever the model knows is in how its parameters shape the computation, and the facts come out, when they do, as high probabilities for the right tokens. A system can be given a document or a tool to consult, but that is extra input supplied to the model, not a part of the forward pass itself.

Now the cost. A useful rule of thumb says that the matrix arithmetic in a forward pass costs about two arithmetic operations per non-embedding parameter per token, one multiplication and one addition. This approximation comes from the scaling work of Jared Kaplan and colleagues, who also account separately for attention, which grows with the length of the context, and who leave out smaller operations. So a model that uses thirty-seven billion parameters on a token does roughly seventy-four billion operations of this kind to produce it. That is an estimate of the main arithmetic, not a complete bill: attention over a long context, the final vocabulary projection, moving data in and out of memory, and the efficiency of the hardware all add to it.

Operations are also not time. The same arithmetic runs at very different speeds depending on the numerical precision used, the shape of the matrices, how many requests are processed together, how fast data can move between chips, and whether the numbers needed are already in fast memory. A design that does fewer operations can still be slower in practice if it is awkward to execute. Keep this in mind when the inference chapter talks about why serving a model is an engineering problem in its own right.

A standard transformer has a fixed stack of layers, but even so, the work per token is not always the same. Attention has more to look at as the context grows. In a mixture-of-experts model, which Part One ends with, different tokens use different parts of the model. In a looped model, which Part Two covers, some layers run more than once. The right starting point for any cost question is the specific computation the model performs, not a general rule that every token costs the same.

A last point, because it corrects a common assumption. A forward pass can adapt to data without any learning in the training sense. There are models, called prior-data fitted networks, that take a set of labelled examples as part of their input and produce predictions for new examples, as a statistician would. The tabular models discussed by Frank Hutter are of this kind. All the training that made them able to do this happened beforehand. Using a dataset inside the forward pass is a computation on input; fitting new parameters to that dataset would be learning. These are different things, and the distinction from the activations chapter, between adapting in context and changing parameters, is exactly the one at work.

The next chapter takes one of the three operations in a layer, the residual connection, and shows why it matters far more than its simplicity suggests.

The Residual Stream

The forward pass, from the last chapter, sends each position's vector through a stack of layers, and each layer has attention, a feed-forward block, and some bookkeeping. One piece of that bookkeeping was the residual connection: a block's output is added to the vector that came in, rather than replacing it. This chapter is about what that addition does, because it changes how information moves through the whole model and it is the basis for most of what Part Three says about looking inside.

Picture the vector at one position as it travels through the layers. Without residual connections, each block would take the vector, transform it, and hand the transformed version to the next block. After forty blocks, whatever the embedding started with would have been transformed forty times, and nothing guarantees that any of it survives. With residual connections, each block computes an update and adds it to the running vector. The running vector, carried from the embedding to the end and added to at every block, is the residual stream.

The consequence is that information can persist. What the embedding contributed has a direct path to layer forty, because the updates were added to it rather than replacing it, though a later update can cancel part of it. A block early in the model can write something into the stream that a block late in the model reads, without every block in between having to pass it along. This is why people describe the residual stream as a kind of shared channel, or shared memory, that all the blocks read from and write to. Later blocks still depend on earlier ones, because they read the stream as it stands after the earlier updates, so the layers have not become independent. But the stream gives the model a way to keep information available across depth.

A mathematical framework from twenty twenty-one, by Nelson Elhage and colleagues at Anthropic, made this picture precise and gave interpretability much of its vocabulary. In that description, the residual stream at a position is the sum of the embedding and the outputs of every block so far. Each block reads from the stream through one matrix and writes to it through another. Because both reading and writing are linear, one can multiply an earlier block's writing matrix by a later block's reading matrix and see directly whether, and how, the earlier block can influence the later one. That is a route of influence that can be computed from the parameters alone, and Part Three uses the idea.

The picture needs two qualifications, and both are honest rather than pedantic. First, normalisation sits in the path: before a block reads the stream, the vector is rescaled, and models differ in exactly where this happens. The original transformer normalised after the addition; many later models normalise before the block's main computation. A clean equation of the stream as a plain sum leaves this out, and it can matter when someone claims to have measured exactly what a block reads. Second, the addition is only part of the story for any one block: the block's own internal computation is nonlinear, as the activations chapter said it must be.

Now a point that is subtle but important for everything in Part Three. The residual stream has some number of coordinates, say four thousand. It is natural to ask what coordinate number one means, or coordinate number two thousand. In the pure linear picture the question has no answer, because one could rotate the coordinate system, adjusting every reading and writing matrix to compensate, and the model would compute exactly the same thing. No coordinate is privileged. That is why researchers who look for meaning in the stream look for directions, which survive such a rotation, rather than for individual coordinates.

That argument has limits, and the limits are where neurons come back in. Normalisation breaks the perfect symmetry a little. More importantly, inside a feed-forward block the expanded vector passes through a nonlinear curve one coordinate at a time, and that gives those particular coordinates, the neurons of the block, a real computational role that cannot be rotated away. So the stream's coordinates are not privileged, but the feed-forward neurons are, and it does not follow that neurons are uninterpretable. Which objects in a model carry meaning is an empirical question that Part Three takes up in the chapters on superposition and features.

The width of the stream, the number of coordinates, limits how many independent things it can hold at one position. A model that wants to track more features than it has coordinates has a way round this: it can use directions that overlap, so that many features share the space, as long as only a few are active at once and the interference between them stays small. This is the idea called superposition, and it has its own chapter in Part Three. For now the point is only that a single vector in the stream is harder to read than a list of separately stored facts would be, both because many things overlap in it and because what it holds changes from layer to layer.

The residual mechanism is still being modified. At the end of twenty twenty-five, DeepSeek published a design that widens the stream into several parallel streams and learns how each block combines and redistributes them. The mixing is constrained: the matrix that does it has nonnegative entries with every row and column summing to one, which stops signals being amplified without limit as they pass through many layers. The paper reports that a fourfold widening cost about seven percent extra training time in its tests, and DeepSeek's version four, a large production model, adopted the mechanism. Earlier work had already changed residual connections in other ways, so this is one constrained design at scale rather than the first change to the mechanism since twenty seventeen, and the constraint is a stability argument for that operation, not a guarantee for the whole network.

For interpretation, the question the stream raises is always the same. If something readable appears in the stream, is it used? Reading is evidence that information is present. Changing the stream and seeing the predicted effect on the output is evidence about the mechanism. The next chapter turns to the operation that moves information between positions, which is attention.

Attention and the Key-Value Cache

So far each position's vector has been treated on its own: the embedding looks up one token, the feed-forward block works on one position, the residual stream carries one position's vector through the layers. Attention is the operation that lets positions talk to each other. It is the reason a model can use what came earlier in the text, and it is the reason long contexts are expensive. This chapter explains the operation, then the store of earlier results that makes generation practical, the key-value cache.

Here is what attention does at one position, in one layer. From the position's current vector, a learnt matrix produces a short vector called the query. From every position the model is allowed to look at, two other learnt matrices produce a key and a value. The query is compared with each key by a dot product, which is just multiplying matching entries and adding them up; a larger dot product means a higher score. The scores are scaled and passed through a softmax, the same operation that turns logits into probabilities, so they become weights that add up to one. The output is the weighted sum of the values. In words: the position asks a question, every earlier position offers a key saying what it has, the keys that match the question best get the most weight, and the position receives a mixture of their values.

Two details are worth keeping. The scaling, dividing the scores by the square root of the key length, is there because without it large scores push the softmax into a region where training signals become tiny; the twenty seventeen paper says so explicitly. And the name is only a name. The operation is a learnt way of selecting and combining numbers. The name alone establishes no correspondence with attention in a person.

A layer does not have one attention operation but several in parallel, called heads, each with its own query, key and value matrices. That lets different heads compute different relations at the same position: one might be useful for finding an earlier occurrence of a name, another for tracking which noun a verb belongs to. Those descriptions, when they are offered, have to be earned by evidence; heads do not come with jobs assigned, and the weights a head puts on positions do not by themselves explain its whole effect on the output. Part Three returns to heads when it looks inside.

In a language model that generates text, attention is causal: a position may look at itself and at earlier positions, never at later ones, because the later ones are what it is trying to predict. During training, this is enforced by a mask that blocks the forbidden positions, and it has a convenient consequence. The whole training text is already known, so every position can be processed at the same time, in parallel, with the mask making sure no position cheats by looking ahead. That is why training on long texts is efficient on parallel hardware.

Generation is different. The next token does not exist yet; it has to be chosen before its vector can be computed. So generation proceeds one token at a time. And here the cache comes in. When the model computes the new token's vector, attention needs the keys and values of every earlier position, at every layer. Those were computed on earlier steps. Recomputing them every time would be wasteful, so they are kept in memory: that store is the key-value cache. Each new step adds its own keys and values to it. The cost is memory, and the memory grows with the length of the context, the number of layers, the number of heads that have their own keys and values, the numerical precision, and the number of requests being served at once.

The amounts are large. A study of transformer inference by Reiner Pope and colleagues gave an example of a cache taking three terabytes, for five hundred and twelve sequences of about two thousand tokens each, served together, on a model whose weights took about one terabyte. The cache for one conversation is far smaller, but a server holds many conversations at once, and the arithmetic adds up to a major part of the cost of running these systems.

Several techniques attack that cost from different sides, and it helps to know which side each one is on. Some reduce what has to be stored. Multi-query attention shares one set of keys and values across all the heads in a layer; grouped-query attention shares them within groups of heads. DeepSeek's multi-head latent attention stores a compressed vector from which the keys and values can be reconstructed. Others manage the memory better without changing what is stored: a technique called PagedAttention, borrowed from how operating systems manage memory, keeps the cache in fixed blocks so that less space is wasted.

DeepSeek's version four point one Flash reports under nine hundred bytes per token for its long-range cache, about a quarter of version four Flash's, together with a mechanism that reconstructs recent-window states when they are needed instead of keeping them in persistent storage. That figure is for one part of the memory, not the whole cost of running the model, which still includes the weights and the working state.

A different response is to use less full attention in the first place. Some recent models alternate ordinary attention layers with layers of a different kind, which keep a compressed running state instead of a growing list of keys and values. Qwen's three point five series alternates three such layers with one attention layer; Kimi K three has sixty-nine of the new kind and twenty-four attention layers. These are hybrid designs, and calling every layer ordinary attention would miss what makes them cheaper at long contexts. The trade-off is real: a compressed state blends earlier positions together, whereas full attention keeps separate keys and values for each of them, and the hybrids keep some full-attention layers for that reason.

DeepSeek's version four report puts numbers on the gain for its own design: at a million tokens of context, its Pro model uses about a quarter of the arithmetic per token and a tenth of the cache of version three point two, by the authors' own accounting. That is one model compared with its predecessor at one context length, not a general law.

All of this leads to the right way to hear a claim about long context. A model that advertises a context of a million tokens is making a claim about the first of two questions: can it store and process that much input at an acceptable cost? The second question is whether it can find and use the relevant information inside all that text. Attention makes the first possible. The second is measured by testing, and the two are often confused.

The next chapter follows the forward pass out into the world: how a model is actually run as a service, and how the probabilities it produces become the words you see.

Inference: Prefill, Decode and Sampling

Inference means running a trained model to get an output. The parameters are frozen; the forward pass from three chapters ago is the computation; the key-value cache from the last chapter, the stored keys and values of earlier positions, is what makes running it repeatedly affordable. This chapter is about the shape of that process in a real service, and about the step that turns probabilities into a chosen token, which is called sampling.

A request to a language model service has two phases, and they have different characters. The first is prefill: the model processes the whole prompt, all its tokens at once, and builds the cache. Because all the prompt tokens are known, this phase is one big parallel computation, well suited to the hardware. The second is decode: the model produces one new token, adds it to the context, and produces the next, one step at a time. Each step depends on the token just chosen, so the steps cannot be done in parallel. For a single conversation, a decode step does relatively little arithmetic but has to read the model's weights and the cache from memory to do it, so at small scale the speed limit is memory traffic rather than arithmetic. With many conversations batched together, or with long contexts, the balance shifts. The common summary, that prefill is limited by arithmetic and decode by memory, is roughly right and not exactly right.

Serving systems are built around these differences. They batch requests together, schedule them to keep the hardware busy, and sometimes run prefill and decode on separate pools of machines. They also reuse work: if two requests share the same opening text, the cache built for that opening can be used for both, as long as the model and its settings are the same. This reuse of a cached prefix is a real saving. It is also only that: a stored result of earlier computation. Nothing has been learnt.

Now sampling. The forward pass ends with logits, one score per vocabulary entry, and the softmax turns them into probabilities. Something still has to pick one token, and the rule for picking is a separate choice. The simplest rule is greedy decoding: always take the most probable token. The usual alternative is to draw a token at random according to the probabilities, which is what sampling means. Between the two sits a dial called temperature. It rescales the logits before the softmax. A low temperature sharpens the distribution so the top tokens dominate; a high temperature flattens it so unlikely tokens get a real chance. A temperature of zero is usually implemented as greedy decoding rather than as actual division by zero.

Two more rules are common, and both trim the distribution before drawing from it. Top-k sampling keeps only the k most probable tokens. Nucleus sampling, also called top-p, keeps the smallest set of top tokens whose probabilities add up to at least a chosen fraction, say ninety percent. Both rules remove the long tail of unlikely tokens, and that is their point. A study by Ari Holtzman and colleagues showed that choosing the most probable continuation at every step produced text that was repetitive and dull, while cutting off the unreliable tail and sampling from the rest produced better open-ended writing. That does not mean greedy decoding is always wrong; it suits tasks where repeatability matters, and which rule produces the most correct answers has to be tested. The right rule depends on whether variation is useful.

Here is the point that matters most for reading a model's output. The probability the model gives a token is not the probability that the finished answer is true. Many different token sequences can say the same thing, so no single token's probability measures confidence in the answer. The model may have learnt a common but wrong continuation and give it high probability. And the sampling rule changes how often a given answer appears without changing anything in the model. So if you ask a model the same question five times and get the same answer five times, you have five draws from one distribution, not five independent confirmations.

A technique called speculative decoding speeds up decode without changing what the model would say. A small, cheap model proposes several tokens ahead; the large model checks them all in one parallel step, accepts or rejects each one by a rule based on both models' probabilities, and corrects the first one it rejects. Done properly, the rule for accepting and correcting preserves the large model's distribution exactly, so the output is as if the large model had sampled alone. The gain depends on how often the proposals are accepted. Some models are trained to predict several tokens at once, as DeepSeek's are, and that training can serve as the proposer, though a model trained that way does not have to use it. This is a good example of a general rule: architecture, training objectives and serving tricks are three different things, and an announcement about one is not an announcement about the others.

Sampling matters in one more way, which came up in an interview with Eric Bigelow of Goodfire in October. In earlier work, his group took a model's reasoning trace, went back to a chosen point in it, and sampled about thirty fresh continuations from that point. At some points, the distribution of final answers changed sharply depending on which token had been chosen there, and the decisive token did not always look important to a reader. Bigelow's interpretation is that some decisions get settled by the sampling step and by the model's later use of what it sampled. That is an interpretation of the trajectories he studied, not a claim that reasoning happens in the sampling step. Both parts are causal: the forward pass sets the probabilities, and the chosen token changes everything the model computes afterwards.

Finally, economics. Inference is not only what customers pay for; it is also a large part of research. Training a model with reinforcement learning, which Part Two describes, means generating a great many candidate outputs so they can be scored, and generating is inference. Dylan Patel estimated in an August interview that a laboratory's computing is split roughly half for research, a tenth for the main training runs, and the remaining four tenths for inference. That is one analyst's estimate with his own category boundaries, not an audited figure. Noam Brown made a related point in September: faster generation makes longer reasoning usable within the time a person is willing to wait.

The measure that matters, in the end, is useful work at an acceptable cost and delay. Tokens per second is part of it. A wrong answer produced faster is worth little, and a better answer produced after more computation is worth it only if the improvement justifies the time. The last chapter of Part One looks at the design that lets a model store far more than it uses on any one token.

Mixture of Experts

The forward pass chapter said that a model's cost is roughly two operations per parameter per token, and the tokens chapter said that a model can store more parameters than it uses on any one token. This chapter is about the design that makes the second statement true, which is called mixture of experts, and it is the design behind many of the largest models of twenty twenty-six.

Recall the feed-forward block inside each layer: it takes one position's vector, maps it to an intermediate one, passes it through a nonlinear curve, and projects it back. In a mixture-of-experts model, some or all of these blocks are replaced by a set of alternative blocks, called experts, together with a small routing network. For each token, the router looks at the token's current vector and picks a small number of the experts out of dozens or hundreds. Only the chosen experts run on that token. The attention operation and the other shared parts of the layer still run for every token as before.

An expert is simply a learnt block of the ordinary kind. The name suggests a specialist in some subject, and the suggestion should be resisted; what the experts actually specialise in is a question for later in this chapter.

This is where the two parameter counts come from. The total count is everything stored: all the experts in all the layers plus the shared parts. The active count is what runs on one token: the shared parts plus the chosen experts. A model can have a very large total while keeping the active count, and so the cost per token, modest. The parameters that are not active for this token are not idle in general; other tokens select them. The point of the design is capacity: more stored knowledge, in the loose sense of that word, without paying for all of it on every token.

The numbers from the model cards show how far the two counts have diverged. DeepSeek's version three, from late twenty twenty-four, stores six hundred and seventy-one billion parameters and uses about thirty-seven billion per token; each of its expert layers has two hundred and fifty-six experts of which eight are chosen, plus one that always runs. By October twenty twenty-six, DeepSeek's version four Pro stores one point six trillion and uses forty-nine billion; Kimi K three stores two point eight trillion and uses about a hundred billion; Meta's Muse Spark Open stores eight hundred and sixteen billion and uses forty-two billion; Reflection's Beam stores five hundred and one billion and uses twenty-three billion. These are the developers' descriptions of their architectures. A larger total or a larger active count does not by itself make a better model, and none of these numbers is a performance score.

Routing creates engineering problems that the simple description hides. If the router sends most tokens to the same few experts, those experts are overloaded and the others get too little work and too little training. Experts are also spread across many chips, so sending tokens to the right expert means sending data across a network. Sparse computation saves arithmetic, but it makes memory access, batching and communication less convenient, and the speed a model actually achieves depends on how the serving system handles all that.

The idea has a history. Noam Shazeer and colleagues made large sparse mixtures practical in twenty seventeen. Switch Transformers, in twenty twenty-one, simplified routing to a single chosen expert per token. Many later designs choose several. To keep the load spread, training adds a balancing term to the loss that penalises concentrated use, and some systems cap how many tokens an expert can take in a batch, dropping the overflow; that dropping was a feature of particular systems, not a necessity of the design. DeepSeek's more recent method adjusts a bias on each expert according to its recent load, nudging the router away from busy experts without touching the experts' outputs. The authors call this balancing without an auxiliary loss, though their models still use a small balancing loss at the level of whole sequences, so one loss was removed rather than all of them.

Do the experts specialise? Sometimes patterns appear, but a clean division by subject is not what researchers generally find. An analysis of the Mixtral model did not find experts assigned by topic in the examples examined. Other studies found that what experts do varies with the architecture and with the layer. The router is trained to lower the loss under its constraints; nothing tells it to give one expert chemistry and another politics, and nothing makes it do so.

That matters for a claim people make about these models: that total parameters are for knowledge and active parameters are for reasoning. It is a plausible tendency, not an established division. More stored parameters may help a model retain more associations. More computation per token may help it transform what it has. But both knowledge and reasoning depend on parameters and computation together, and the training data and procedure change the balance.

In a September discussion on the Dwarkesh Patel podcast, John Schulman argued that total and active parameters should be treated as two different resources, while saying how uncertain the field still is about sparsity. Charlie O'Neill recalled an early, very sparse model from the Switch Transformer era that seemed strong on knowledge and weak on reasoning. That was a recollection of an earlier experiment, offered as a reason to ask about the costs of extreme sparsity, not a result that pins the difference on sparsity.

One last distinction, because Part Two will use it. Mixture of experts chooses which stored transformations to run on a token. A looped model, which Part Two describes, runs the same transformations more than once. They answer different questions, and they can be combined. For a listener assessing a new model, the questions are: how much does it store, how much does it use per token, how often does it use it, and what does it achieve for a given budget.

Part One in Five Hundred Words

Here is the picture Part One has built, in one pass, with nothing new added.

A language model is a fixed function. Text is cut into tokens by a tokenizer and becomes a list of labels. The function turns that list into probabilities for the next token, one token is chosen, and the function runs again.

The function has two kinds of numbers. Parameters are the frozen numbers that define it, learnt during training and never changed in ordinary use. Activations are the numbers computed on one particular run, and they depend on the input. Activations come in vectors, lists of numbers that can be treated as points in a space, and a direction in that space is a vector you can add to move a state.

The loss is the number training makes small. For a language model it measures how badly the model predicted the text it was trained on. So the model's output is a prediction of text, not a judgement of truth, and the later stages of training, in Part Two, are attempts to change what it is rewarded for.

Each token first becomes a vector by lookup in the embedding table. As the vector passes through the layers, context changes it, so the same token ends up with different vectors in different sentences. These internal vectors are the latent space, and its structure, which can be directions or curves, is something researchers measure rather than assume.

The forward pass is the computation itself. Each layer applies attention, which lets a position draw on earlier positions, and a feed-forward block, which transforms one position at a time, with normalisation to keep the numbers in range. The residual connection adds each block's output to a running vector rather than replacing it, so that information persists through the depth of the model; that running vector is the residual stream, and it is where Part Three will look for meaning.

Attention works by queries, keys and values: a position asks a question, earlier positions offer keys, the best matches contribute their values. The keys and values of earlier positions are stored in the key-value cache so generation does not recompute them, and that cache is a major part of the cost of long contexts.

Inference runs in two phases, prefill for the prompt and decode for the output, one token at a time. The chosen token comes from a sampling rule applied to the probabilities, and a token's probability is not the probability that the answer is right.

Mixture of experts lets a model store far more parameters than it uses on any one token, by routing each token to a few experts. Total and active parameters are two different counts.

Part Two takes the frozen function apart and asks how it got its parameters, and then how more reasoning can be drawn from it.