Coffee & Cake: Benchmarks, Test Sets and Contamination
Canonical episode: https://move37.app/cafe/en/episodes/coffee-cake-benchmarks-test-sets-contamination-en/
Published: 2026-08-29T02:52:14Z
Language: en

TRANSCRIPT

Welcome to Andy's Café, where machines brew and humans taste. Today we're serving Coffee and Cake. Enjoy.

Benchmarks, Test Sets and Contamination

A new model arrives with a triumphant chart. One row says ninety-two percent. Another model managed eighty-nine. The natural conclusion is that the newcomer is three points better. But before the number can mean anything, we need to ask: ninety-two percent of what, under which rules, and with how much opportunity to see the exam?

Three ideas belong together here. A benchmark is a standardized evaluation. A test set is the material reserved for measuring performance. Contamination is what happens when supposedly unseen test information leaks into the process that produced the model or system. Together they explain both why leaderboards are useful and why their clean-looking numbers can support messier conclusions than they appear to.

Begin with an ordinary exam. The exam is not just a stack of questions. It also includes instructions, a time limit, permitted tools, a marking scheme, and a rule for combining marks into a result. Change from closed book to open book, allow three attempts instead of one, or replace a human marker with an automatic one, and you have changed what the score measures.

A model benchmark works the same way. It may contain mathematics problems, programming tasks, factual questions, images, or simulated computer work. But the examples alone are only a dataset. The complete benchmark includes the prompt, the model's available tools, the number of attempts, the stopping conditions, the scoring method, and the way many results become one headline figure. A model tested alone is not the same system as that model with web search, code execution, and a carefully tuned agent harness.

This is especially important when answers are sampled. Give a system one attempt and count whether it succeeds, and you measure something different from giving it ten attempts and keeping the best. Let an agent inspect a failing test and revise its code, and you are measuring recovery as well as first-try correctness. Both evaluations can be legitimate. Trouble begins when their scores share a column as if the rules were identical.

Now divide the educational material into three piles. Training exercises are used to learn. A development or validation set helps the teacher choose methods and the student identify weaknesses. The final test is held back until the choices are finished. Machine learning uses the same broad separation. Training data adjusts model parameters. Validation results guide choices such as architecture, data mixture, prompts, or checkpoints. A test set is meant to estimate how the finished system handles material that did not guide those choices.

“Held out” therefore describes a relationship, not merely a folder name. If a team checks the test score every afternoon, changes the system, and keeps whichever version scored highest, the test has begun to guide development. No one needs to copy an answer into the model's weights directly. Repeated feedback turns the final exam into another practice sheet.

Contamination is a sharper failure of independence. Suppose the exact exam and answer key were posted online, then gathered into a model's enormous training corpus. Or a tutorial reproduced the same questions with only superficial changes. A later high score might still reflect genuine skill, but it can no longer provide clean evidence about performance on unseen problems. The model may have learned the underlying idea, memorized distinctive wording, absorbed an answer, or some combination of all three.

This distinction matters. Finding overlap does not prove that a model copied each response, and failing to find exact matching sentences does not prove the test stayed secret. A problem can be paraphrased, translated, discussed in a solution guide, or represented in code. Web-scale training collections are so large and imperfectly documented that complete reconstruction is often impossible. Contamination is not always deliberate cheating. It can be an accidental consequence of training on a public culture in which popular benchmarks are repeatedly quoted.

Investigators can search training records for exact phrases, compare unusual sequences, or test whether a model behaves differently on examples published before and after a cutoff. Each method catches only part of the problem. Exact matching misses paraphrases. Model behaviour can suggest exposure without locating its source. And a declared data cutoff is useful only if the collection and later training stages actually respect it. Contamination reports should therefore describe the detection method and its blind spots, not reduce uncertainty to a clean yes or no.

Imagine a student who once read the answer to one exam question but forgot its wording and retained the method. Has that student generalized or remembered? In human education, the boundary can be philosophically awkward. For a benchmark, the practical point is simpler: the test was designed to estimate performance without that exposure. Once exposure is plausible, the score answers a different and less controlled question.

Public leaderboards add another form of pressure. A community may spend years optimizing prompts, data, tools, and software against the same famous tests. Each individual team can behave honestly while the field as a whole becomes adapted to the benchmark. The questions that were once surprising become familiar design targets. This is why a saturated benchmark can stop distinguishing progress, and why a fresh test collected under similar rules can produce a lower result.

There is no perfect remedy. Keeping questions secret preserves surprise but makes independent inspection and reproduction harder. Constantly writing new tests is expensive and can quietly change the difficulty. Encrypting or sealing test material can reduce leakage, but someone must be trusted to operate the box. Dynamic tasks resist memorization but may introduce unstable conditions. Good evaluation is not one clever lock. It is a set of trade-offs made visible.

One promising design separates confidential knowledge on both sides. Independent evaluators keep the exact test prompts hidden from the model developer, while the developer keeps proprietary model weights hidden from the evaluators. A secure environment brings them together only for the run. This reduces the developer's opportunity to optimize for the hidden questions while protecting both sides' confidential material. It still cannot decide whether the chosen tasks represent the world outside the box. Secrecy protects independence; it does not manufacture relevance.

The scoring rule deserves equal suspicion. Exact-match grading is clear but can reject a correct answer phrased differently. Human judges can understand nuance but bring cost and disagreement. Another language model can grade thousands of responses quickly, yet research has found biases involving response position, verbosity, and self-preference. Carefully constructed text inside a candidate answer can even manipulate some automated judges. Averaging everything into one number can then hide that a system improved greatly in English programming tasks while becoming worse at another language or a safety-critical edge case.

Even a clean test measures a sample, not the whole world. Ninety-two percent accuracy means the system met one scoring rule on ninety-two percent of the evaluated items. It does not mean the model is ninety-two percent intelligent, truthful, safe, or suitable for your work. A small difference between models may fall within sampling or measurement uncertainty. A large average can conceal a category where failure is systematic. And a benchmark task may resemble a real job only from a distance.

None of this makes benchmarks pointless. Without common tests, every developer could choose flattering demonstrations and comparisons would become worse. A well-described benchmark lets others repeat a procedure, inspect errors, compare systems under shared conditions, and notice when progress stalls. The mistake is not measuring. It is forgetting what the measurement includes.

The strongest claims usually combine several kinds of evidence. Use multiple benchmarks that stress different abilities. Preserve some genuinely unseen material. Report more than the aggregate. Record the model version, prompts, tools, retries, and grader. Examine whether training data could contain the test. Then validate the system in the environment where it will actually be used, with the costs and failure consequences that matter there.

So when the next chart announces ninety-two percent, carry four questions to the table. What task and metric produced the number? What was the model allowed to use? Was the test genuinely independent of training and development? And does this exam resemble the job I care about? Those questions do not dismiss the score. They turn it from advertising back into evidence.

That's all for now. The café is always open. Come back soon.

MORE INFORMATION
TEXT LICENSE
This transcript, description and original Andy's Café editorial text are licensed under Creative Commons Attribution 4.0 International (CC BY 4.0).
License: https://creativecommons.org/licenses/by/4.0/
Attribution: Andy's Café — https://move37.app/cafe/en/episodes/coffee-cake-benchmarks-test-sets-contamination-en/
Indicate changes when adapting. Identified third-party quotations and linked source material remain under their own terms.

AUDIO AND OTHER MATERIAL
The composed episode has separate component terms because its piano cues are third-party material. Artwork and the Andy's Café brand are not included in the CC BY licence.
Rights map: https://move37.app/cafe/welcome/#reuse
Contact: hello@move37.app