Coffee & Cake: Tokens, Context and Memory

Tokens, Context, and What Actually Gets Remembered

You have been talking to an assistant for the better part of an hour. It has been good company. It picked up early on that you like short answers, it has stopped offering you recipes with meat in them, it follows the thread of what you are working on. And then, near the end, you ask it to redo something the way you asked for at the very beginning, and it has no idea what you are talking about.

Nothing broke. It did not get tired, or bored, or subtly worse at its job. Something quite specific happened, and by the end of the next few minutes you will be able to name it.

There are three layers underneath almost every conversation you have with one of these systems, and most of the confusion in the world comes from squashing all three into the single word remembering. Tokens are the units, the way things get measured. Context is what is on the worktop right now, in front of the model, this second. Memory is material stored somewhere else entirely, that the product around the model may or may not choose to fetch and put on the worktop for you.

Start with tokens, because everything else is counted in them. Before a language model processes any text, that text gets chopped into pieces. The pieces are not quite words. A common word like coffee is usually one piece. A long or unusual one, a place name like Kilimanjaro, gets split into several. Spaces and punctuation take up room too. The chopping scheme is called a tokenizer, and different systems use different ones, which is why you should be gently suspicious of any claim that some number of tokens equals some number of pages. German compound nouns, source code, and languages that do not use the Latin alphabet can all be split differently. A token is a unit of measurement whose ruler changes depending on the system and the material. Fine for estimating. Not a fact about the universe.

Now the worktop. Every single time the model produces something for you, it works from one bundle of text, and that bundle has a maximum size. That maximum is the context window. What goes inside it is more than you might think. There are instructions the product wrote before you ever arrived, describing how it should behave. There is your question. There is any material the product decided to hand over on your behalf. And, importantly, the answer being written draws on that same capacity, as does, in some systems, the model's own internal working out. It is a finite counter. If you want to put the cake board down, something else comes off.

Here is the part that surprises people most. At the level of a single generation, the thing is stateless. The model is not sitting there between your messages holding your conversation, mulling it over, thinking about you. It is handed one ticket, does the work on that ticket, and that is the whole of its world. Continuity is manufactured on the outside. Somebody has to assemble the ticket.

So let us follow one real request through. You type: can you write a polite note to my landlord about the heating, keep it short, the same tone as the one we wrote back in the spring.

First, conversation history. The product goes back through your earlier turns and puts material from them into the ticket again. Depending on the product and the length of the chat, that may be the full history, a selection, or a summary standing in for older parts. That is why the assistant seems to remember that you like short answers. Not because it learned anything. Because the sentence where you said so was sent along again. And it is exactly why your instruction from an hour ago can evaporate. When older material gets trimmed to make room, your instruction can go out with it. Whether that trimming is a clean summary, a quiet dropping of the oldest messages, or something cleverer is a decision made by the product, not by the model.

Second, memory in the product sense. Some products keep a store outside any single conversation. Saved facts, summaries, whole past chats. None of that lives inside the model. Think of it as a notebook the product keeps behind the counter, with your preferences written in it. When you open a brand new chat and it already knows you have a dog, something looked in the notebook, judged the entry relevant, and copied it onto today's ticket. Products differ enormously here. Some expose what was saved and let you remove it. Some choose relevant material in the background. Some have no such notebook at all. So when people argue about whether an assistant remembers them, they are usually arguing about product design, not intelligence.

Third, retrieval. Same shape, different cupboard. Instead of your preferences, the store is a body of documents: your lease, a company handbook, a library of articles. The system searches that collection, selects the passages that look relevant to what you just asked, and adds them to the request. The model then reads those passages the way it would read anything you pasted in yourself. Next month, if nobody runs that search, it will know nothing about your lease whatsoever. Retrieval is fetching something from the pantry. It is not learning the recipe by heart.

So the ticket for your landlord note now holds: the product's standing instructions, a line or two from the notebook about how you like to be written for, a selected slab of conversation history that may or may not still include the note from the spring, possibly a retrieved paragraph from your lease about repair obligations, and your actual sentence. All of it broken into tokens. All of it sharing one finite capacity. And the reply gets written out of whatever room is left.

If the right things were chosen and everything fitted, you get a good short note. If the spring note lost its place in the selection, you get a perfectly competent note in slightly the wrong voice, and nothing will warn you, because from inside that request nothing was missing. The model cannot miss what it was never handed.

Picture three products built on the very same model. The first sends the entire conversation back every time and simply runs out of room eventually. The second summarizes the older stretches, so it can carry far more of the past, at the price of detail going soft. The third keeps a memory store and searches your documents, so a fresh chat begins already knowing things about you, though now and then it drags in something stale. Identical intelligence underneath. Three completely different personalities of remembering.

Which lets us clear away a few stubborn beliefs. If you told it something once, the model did not learn it. The product may have stored it and may keep resending it, but the trained model itself is normally unchanged by your mentioning your dog. A large context window is not long-term memory. One is how much fits on the counter today. The other is whether anyone wrote it down at all. And more context is not automatically better. A well-chosen page beats a dumped folder, because a very long input can bury the one sentence that actually mattered. You may also hear about caching, which lets a repeated opening chunk be reused rather than recomputed. That is a saving in work and cost. Those tokens still take up their space on the counter. It buys you speed, not room, and certainly not memory.

Which gives you a useful thing to do the next time an assistant forgets something. Instead of concluding it has got worse, ask three questions in order. Was this ever stored anywhere outside the conversation? If not, it only ever lived in the chat, and chats can be trimmed. If it was stored, was it selected as relevant for this particular request? And if it was selected, did it survive alongside everything else competing for the same finite space?

Then just say the thing again, and say it near the top. Nothing forgot you. Somewhere between the notebook and the counter, a fact did not make it onto today's ticket. Once you can see that clearly, the whole tool gets easier to work with, because you stop hoping it will remember, and start making sure it is holding what it needs.