Double Espresso: Training and Inference
Canonical episode: https://move37.app/cafe/en/episodes/double-espresso-training-inference-en/
Published: 2026-08-10T21:38:08Z
Language: en

TRANSCRIPT

Welcome to Andy's Café, where machines brew and humans taste. Today we're serving a Double Espresso. Enjoy.

Training and Inference: What Is the Difference?

You hear it in a news report: some company has just trained a new model. A week later, someone tells you that inference is getting cheaper. Two sentences, two words, and both are usually left standing there unexplained, as though everyone already knows. So let us take them apart, because once you have them, a surprising amount of AI talk suddenly becomes readable.

Here is the short version. Training is how a model gets made or changed. Inference is what happens when somebody uses it.

Now let me give each of those a proper sentence.

Training is a process that reads examples, or some other form of feedback, measures how wrong the model currently is, and adjusts the model's internal numbers so that it becomes less wrong. Those internal numbers are usually called parameters, or weights. There can be millions of them, or billions. Training is the only activity in this episode that changes them.

Inference is what happens afterwards. The finished model receives a new input it has never seen before, runs its calculation, and produces an output. A spam score is an inference output. So is a label on a photograph. So is the answer a chatbot gives you. In everyday AI talk, inference simply means applying a trained model to something new.

Let me make that concrete, with the least glamorous piece of machine learning ever built: an email spam filter.

During training, this small model is shown a large collection of messages, each one already marked as spam or not spam. For every message it makes a guess. The guess is compared with the correct answer, and the parameters are nudged a little in the direction that would have produced a better guess. Repeat that an enormous number of times, and the parameters end up encoding something genuinely useful about what spam tends to look like. Then training stops.

Now the model goes to work. A new email arrives, one that was never in the collection. The model reads it and produces a number: a spam score, let us say zero point nine. That single calculation is inference. It takes a fraction of a second, it happens again for the next email, and it can happen millions of times over without the model learning anything new.

Notice, by the way, how much of what you experience as the spam filter is not the model at all. The decision that anything scoring above zero point eight goes to the quarantine folder is a threshold, a rule that a person chose, sitting around the model. Move the threshold and the filter behaves differently while the model itself is untouched. A great many AI products work like this.

Which brings us to the part that trips almost everybody up: what is actually happening when you type into a chatbot.

Your prompt obviously changes the answer. Ask for simpler words and you get simpler words. Paste in a document and the reply is about your document. It genuinely feels as though the model has just learned something. But what changed is the input, not the model. The same weights, entirely unaltered, ran over different material. Think of a pianist sight-reading a new score. Different music comes out, and the pianist has not become a different musician.

The same holds for the machinery built around the model. Conversation history means the product quietly sends the earlier turns back along with your new message. Retrieval means the system searches a pile of documents and pastes the relevant passages into the input before the model ever sees it. Saved memory means notes kept in ordinary storage outside the model and reinserted later. All three are real, all three are useful, and none of them is the same thing as changing the model's learned parameters.

So here is the misconception worth correcting out loud. A model does not normally retrain itself while it is talking to you. If you tell it that it is wrong, it may well apologise and do better, because your correction is now sitting there in the input. Start a fresh conversation, and that lesson is generally gone.

Your feedback is not necessarily wasted, though. A service can collect ratings and interactions and use them in a later training run, deliberately, on a schedule, by the people who build the model. That is the honest picture: today's feedback may become tomorrow's training data, but it is not the model instantly remembering you.

Two caveats, because the tidy version is a little too tidy. Systems that keep learning while running do exist, so treat the separation as the normal arrangement rather than a law of nature. And a longer prompt means more inference work, more computing on that one request. It does not mean training.

Why does any of this matter beyond vocabulary? Mostly because of cost. Training a large model is one vast, concentrated job. Inference is tiny by comparison for any single request, but it happens on every request, all day, for as long as the model is in service. Which one adds up to more depends entirely on how many requests there are and over what period. So whenever someone tells you where the money in AI really goes, the useful question is: over what stretch of time?

Here is the handle to keep. Training is writing the cookbook. Inference is cooking tonight's dinner out of it. Cooking one more dinner does not rewrite the cookbook, and the cookbook only changes when somebody sits down, on purpose, to write the next edition.

That's all for now. The café is always open. Come back soon.

MORE INFORMATION
TEXT LICENSE
This transcript, description and original Andy's Café editorial text are licensed under Creative Commons Attribution 4.0 International (CC BY 4.0).
License: https://creativecommons.org/licenses/by/4.0/
Attribution: Andy's Café — https://move37.app/cafe/en/episodes/double-espresso-training-inference-en/
Indicate changes when adapting. Identified third-party quotations and linked source material remain under their own terms.

AUDIO AND OTHER MATERIAL
The composed episode has separate component terms because its piano cues are third-party material. Artwork and the Andy's Café brand are not included in the CC BY licence.
Rights map: https://move37.app/cafe/welcome/#reuse
Contact: hello@move37.app