Sunday Latte: Why AI Needs So Much Compute

The Effortless Answer

You type a short question. A few seconds later, a polished answer appears. From this side of the screen, the exchange can feel almost weightless.

Yet behind that answer are rooms filled with specialized computers, fast memory, elaborate cooling, and networks designed to move enormous quantities of data. Companies plan new power supplies around these machines. Researchers spend months deciding how to use them. Why should a paragraph require anything like that?

Not every artificial intelligence system needs all of this. A small model that checks sensor readings or filters messages can run on modest hardware. This episode is about the large language models that generate text piece by piece, and about why current methods make them so demanding.

There are really two bills to understand. The first is paid while the model learns. During training, it processes vast amounts of material and repeatedly adjusts the numerical values called weights. The second bill arrives when the trained model is used. During inference, those weights are applied again for every prompt and every token of the response.

Neither process is a single act of electronic thought. Both are built from simple numerical operations repeated on an extraordinary scale. And the arithmetic is only part of the story. The numbers must be moved through memory. When a model is spread across many machines, they must also be moved between machines. A powerful processor that is waiting for data is expensive equipment doing nothing.

So the useful question is not simply why artificial intelligence needs powerful chips. It is why current methods need so much repeated arithmetic, why data movement becomes a traffic jam, and why making each operation cheaper does not necessarily make the total appetite for compute disappear.

A Machine for Multiplication

A language model is, at heart, a very large learned numerical function. Text enters as tokens. A token may be a word, part of a word, or punctuation. Each token becomes a list of numbers, often called a vector. The model then transforms those vectors through many layers.

Inside a transformer, attention compares representations from different positions. Other parts of each layer transform the representation at one position. Much of this work can be arranged as matrix multiplication: large rectangular tables of numbers multiplied and added together. There are other operations too, but matrix work dominates much of the computational load.

One multiplication and addition is trivial. The scale comes from repeating it across the width of the model, across many layers, and across all the tokens being processed. It is like covering a wall with tiny tiles. Placing one tile is easy. Covering every wall in a city is not.

This shape of work explains the importance of accelerators. A central processing unit is designed to handle a wide variety of tasks. A graphics processor, or a purpose-built tensor processor, devotes much more of its machinery to performing many numerical operations at once. It can also use compact number formats when the model tolerates them.

But the impressive headline number for an accelerator is its peak performance under ideal conditions. Reaching that peak requires a steady supply of large, well-organized operations and data. A kitchen can employ hundreds of cooks, but they will not produce hundreds of meals at once if every ingredient must pass through one narrow hatch. In an artificial intelligence system, memory and communication links are often that hatch.

Teaching Means Repeating the Work

Training is expensive because the model does not merely read a collection of text once.

Take one batch of training examples. The model first makes predictions. In ordinary language model pretraining, it tries to predict the next token from the tokens that came before. Its predictions are compared with the tokens that actually appeared, producing a measure of error called a loss.

Then comes the part that ordinary use does not need. Backpropagation works backwards through the calculations and determines how small changes to the weights would affect that loss. An optimizer uses those gradients to update the weights. The next batch arrives, and the process begins again.

Training therefore includes the forward calculations that produce an answer, the backward calculations that assign responsibility for the error, and the update itself. It also needs more memory than the finished weights alone. The system may have to retain intermediate activations, gradients, and additional values used by the optimizer. Some of that information can be recomputed to save memory, but recomputation adds more arithmetic.

For a conventional dense transformer, a useful first approximation is that the training work rises with both the number of parameters and the number of training tokens. Double the model while keeping the data the same and each token becomes more expensive. Keep the model fixed and show it twice as much material, and the training run also grows.

That approximation leaves out many details, including data filtering, failed experiments, post-training, and differences among architectures. Still, it explains why modern training plans are budgets rather than recipes. Developers must decide how much model they can train, how much suitable data it should see, and how many updates the available machines can complete.

Spending the Training Budget

More compute has produced real improvements, but only when it is spent in a useful balance.

Researchers found that language model error often followed surprisingly regular scaling patterns. As model size, data, and training compute increased, average loss fell along smooth curves within the ranges being studied. Those patterns allowed teams to estimate a large run before committing all of its resources.

They also revealed waste. Imagine spending almost the entire budget on a larger model, then ending training before that model has seen enough material to use its capacity well. A smaller model trained on more tokens may produce a better result for the same training compute.

The comparison between two research models, Gopher and Chinchilla, made this vivid. Chinchilla had about seventy billion parameters, while Gopher had about two hundred and eighty billion. Chinchilla was trained on four times as much data with a similar compute budget and performed better across the reported evaluations. The point was not that seventy billion is a magic size. It was that parameter count alone is a poor measure of how well a training budget has been used.

Data quality complicates the balance further. Repeated, misleading, or poorly selected material is not equivalent to diverse, relevant material. Architecture and the intended use matter too. A smaller model that will answer billions of requests may justify more training if that makes every later answer cheaper.

And scaling brings diminishing returns. A power law can describe continuing progress while also saying that each further improvement requires a multiplying investment. Smooth improvement in average prediction does not guarantee smooth progress on every useful skill. Scaling laws are maps of measured territory, not a promise that enough hardware will overcome bad data, a poor objective, or a limit no experiment has reached yet.

The Traffic Between the Chips

At some point, the model no longer fits on one accelerator.

Consider only the stored weights. Seventy billion weights, held at two bytes each, require roughly one hundred and forty gigabytes. Training needs much more space for gradients, optimizer state, and intermediate values. A large run must divide the work across many devices.

There are several ways to divide it. Different devices can process different batches. They can hold different layers. They can each hold a slice of the same enormous matrix. Sparse models can route different tokens to different groups of parameters, called experts. Every split solves a memory or arithmetic problem, but creates communication. Devices exchange gradients, activations, or routed tokens, and often wait for one another at synchronization points.

This is why the links between accelerators are not accessories. If one device finishes early and waits for a slower partner, theoretical performance is lost. Adding more chips can increase the total arithmetic available while making coordination harder.

The same principle applies inside one chip. Very fast memory sits close to the arithmetic units, but it is small. Larger high bandwidth memory holds far more, but moving numbers to and from it takes time and energy. An operation can be limited by arithmetic, by memory capacity, or by memory bandwidth. Which limit matters changes with the model, the batch, and the stage of the workload.

A method called Flash Attention provides a useful counterexample to simple chip marketing. It reorganizes exact attention so that fewer intermediate values travel to and from the larger memory. During training it may deliberately recalculate some values, doing more arithmetic to avoid slower movement. The result can finish faster with a smaller memory footprint. Sometimes the quickest route is not fewer calculations. It is less traffic.

After You Press Enter

Training ends with a set of weights. Using them is inference, and it has its own rhythm.

First, the system processes the prompt. This stage is often called prefill. Many prompt tokens can be handled in parallel, so a long prompt creates a substantial block of work but gives the accelerator useful parallelism.

Then generation begins. An autoregressive language model produces one token, adds it to the sequence, and uses the enlarged sequence to produce the next. Each new token depends on the previous one. Ten output tokens therefore cannot simply be generated at the same moment by the same ordinary decoding process.

The system avoids starting from nothing on every step. It retains intermediate attention information from earlier tokens in a key value cache. This cache is temporary working state for the current sequence, not the model's learned knowledge. It grows as the context grows. Multiply that growth by many simultaneous users, and memory that looked generous can fill quickly.

Serving systems try to keep accelerators busy by batching requests. A larger batch can improve throughput because the weights are used for several requests together. But throughput and latency are different promises. Waiting to form a convenient batch may be efficient for the operator and irritating for the person watching an empty screen.

There are clever ways to loosen the serial bottleneck. In speculative decoding, a smaller model proposes several likely tokens and the larger model checks them together. When the proposal works, several positions advance without changing the larger model's intended output distribution. Cache management can also fit more active requests into the same memory.

These techniques reduce the cost of a fixed workload. They do not remove the second bill. Every prompt, every prompt token, every generated token, and every extra candidate still requires work. A single answer may be inexpensive while a service used continually by millions of people becomes an enormous inference system.

When the Model Spends Longer

Inference compute is no longer determined only by the size of the prompt and the visible answer. Some systems can spend more work on a difficult problem before responding.

There are several ways to do this. A model can generate a longer internal route, revise an attempted solution, or try several candidates. Another model or scoring system can compare those candidates and choose one. These approaches spend compute at the moment of use rather than placing all capability into a larger training run.

That trade can be valuable. Experiments have shown that well-allocated test-time compute can let a smaller model outperform a much larger one on some problems. But the phrase well allocated matters. Different strategies work better at different levels of difficulty. An easy question may gain nothing from dozens of attempts. An impossibly hard question may produce dozens of variations on the same mistake. In between, revision or selection can make a large difference.

More tokens should not be confused with more reasoning. Length is a cost that may carry useful work, repetition, or confusion. A system needs a method for deciding when to continue, when to explore alternatives, and when to stop.

This changes the economics of an answer. Two requests with equally short visible responses may have used very different amounts of inference. It also makes compute a product decision. Fast and adequate may be right for a routine request. Slow and deliberate may be worth paying for when the problem is difficult and the consequence matters.

Power, Energy, and Price

Discussions about artificial intelligence often slide between power, energy, emissions, and money as though they were synonyms. They are not.

Power is a rate. Energy is that rate accumulated over time. A newer accelerator may draw more power while it is active, yet use less energy for a task if it finishes much sooner. A slower device with a modest power rating can consume more energy by running for longer.

Emissions add another layer. The same amount of electricity can have a different carbon impact depending on where and when it is generated. Manufacturing the hardware has an impact too. Price is different again. A cloud bill includes the accelerator, memory, networking, buildings, cooling, staff, financing, spare capacity, and the provider's margin. It is not an electricity meter.

Even a number called energy per prompt depends on what is counted. One large provider measured a median text prompt across its live service and included active accelerators, host processors and memory, idle machines reserved for reliability, and data centre overhead. Its result was more than twice the estimate produced by a narrower boundary in the same study. Neither figure was fraudulent. They answered different accounting questions.

The prompt matters as well. A short classification, a long document analysis, an image request, and a reasoning task do not have one common footprint. So a single catchy number cannot honestly stand for all artificial intelligence use.

Training and inference should also remain separate. Training is a concentrated cost paid before release. Inference is repeated over the model's useful life. A rarely used experimental model may spend far more on training. A widely used model may accumulate a much larger serving bill. There is no universal winner without knowing the model, its traffic, its response lengths, and how long it remains in service.

Enough Compute, Used Well

Efficiency can improve at almost every point in this story.

Better data can reach a target with fewer training steps. Smaller models can handle easier tasks. Quantization stores and processes some values with fewer bits, reducing memory traffic when the hardware supports it and the quality remains acceptable. A mixture of experts activates only part of a larger model for each token. Better kernels reduce movement through memory. Batching, cache management, and speculative decoding use serving hardware more effectively. New accelerators can deliver more useful work for each unit of energy.

None of these gains is imaginary. None is unlimited. Lower precision can damage a model. Sparse experts still need storage and communication. Large batches can increase delay. An optimization that removes one bottleneck may reveal another.

Most importantly, efficiency for a fixed task does not determine total demand. If an answer becomes ten times cheaper, more people may use the service. Applications may send longer contexts, generate several candidates, or tackle jobs that were previously too expensive. A research team may invest the saved training compute in a larger experiment. The cost per cup can fall while the café serves many more cups.

This is why both extreme stories fail. It is wrong to say that efficiency solves the resource question automatically. It is also wrong to treat every large computation as waste merely because it is large. The relevant question is what the computation achieves, what alternatives exist, and whether the improvement is worth the additional resources.

Return to the short question that began this episode. The answer looked effortless because the machinery was hidden. Behind it were weights shaped by repeated training, matrices applied across many layers, numbers moving through memory and networks, and a serving system deciding how much work this request deserved.

Compute is one of the essential ingredients in current artificial intelligence. It is not a substitute for good data, a sound objective, reliable evaluation, or judgment. The goal is not the greatest possible amount. It is enough compute, allocated well, to produce an answer worth having.