Sunday Latte: Why AI Can Sound Certain and Still Be Wrong

The Confident Wrong Answer

In a study published in twenty twenty-five, researchers asked three publicly available language models a precise question about one of the authors. What was the title of Adam Tauman Kalai's doctoral dissertation?

Each model supplied an answer with the texture of a library record. There was a specific title, a university, and a year. Nothing in the phrasing sounded improvised. There were no visible gaps, nervous hesitations, or half-finished thoughts. Yet none of the models gave the correct title or year.

This small example captures something unsettling about artificial intelligence. An answer can be polished in every obvious way and still fail at the one thing that matters. The grammar works. The genre works. The details fit together. The fact does not.

It is tempting to react in one of two ways. We might assume that eloquent language reveals reliable knowledge. Or, after seeing an error like this, we might decide that every answer is an elaborate coin toss. Neither response is useful.

Language models can recover real information, compare evidence, explain difficult ideas, and help people think. They can also produce a false name, date, quotation, or reference without any change in tone. Their strengths and their failures grow from some of the same machinery.

So the question for today is not simply why a model makes mistakes. Every information system makes mistakes, and so do people. The sharper question is why the mistake can arrive with such composure. Why does fluency feel like evidence? What would genuine confidence mean? What changes when the system can search or use a tool? And how can we use a fluent assistant without either trusting its tone or becoming suspicious of every sentence?

The dissertation will stay with us. It is a rare, exact fact surrounded by familiar language. That combination is an excellent place to see how a model can sound certain and still be wrong.

What Fluency Proves

A language model generates text in a sequence. It receives the prompt and the tokens already present, then assigns probabilities to possible next tokens. A decoding procedure selects one. That token joins the context, and the model predicts again.

The loop sounds small compared with the result. Many rounds later, the screen may contain an explanation, a legal clause, a joke, or a piece of software. But the tokens are not independent. Every prediction is conditioned on what came before, and training has shaped a large numerical system to represent many relationships at once.

To predict language well, the model benefits from learning grammar, style, factual associations, patterns of argument, and recurring structures in stories, science, code, and conversation. Fluency is therefore a real capability. It is not random word salad, and calling the model a telephone keypad with a larger dictionary explains very little.

The limit lies in the training target. During broad pretraining, the target is the continuation that appeared in the training text. The model is rewarded for assigning that text a higher probability. The target does not arrive with a separate certificate saying that the sentence is true.

A history book, a fairy tale, a correction, a common myth, and a confident mistake all contain learnable patterns. Assigning their observed continuations higher probability can improve the same training objective. Of course, the world leaves traces in language, so learning to predict text can produce extensive factual capability. But truth helps prediction without becoming identical to prediction.

Ordinary generation does not automatically compare every claim with an authoritative record. A surrounding system may add search, a database, or a verifier. Without that step, the model works from its weights and current context.

That explains the dissertation answer's form. Doctoral titles, universities, and years fit familiar biographical patterns. Producing that professional texture and recovering the exact relationship among one person, one title, and one year are different achievements.

Fluency proves that the model can construct a strong continuation. It does not, by itself, prove that the continuation deserves belief.

Three Kinds of Confidence

Part of the confusion comes from using one word for three different things.

The first is tone. An answer may say definitely, or it may sound assured through clean structure, precise detail, and a lack of qualifications. These are features of the generated language.

The second is probability inside the prediction process. The model scores possible next tokens and, in a constrained question, may score several answer choices. Those numbers belong to a particular prediction format.

The third is confidence that an answer is correct. Imagine collecting one hundred answers labelled seventy-percent confident. If the system is calibrated in that setting, roughly seventy should be correct. Calibration is a pattern across many cases, never a certificate for one answer.

These forms can diverge. A sentence may sound decisive while the possible continuations are widely spread. A familiar misconception may receive high probability. And token probability is not automatically the probability that the proposition is true: language offers many ways to express the same claim.

Models can sometimes evaluate their own answers usefully under carefully designed formats. Other experiments find overconfidence on difficult or unfamiliar tasks. Calibration achieved in one setting does not travel everywhere unchanged.

If a model says it is ninety percent certain, that percentage is still generated text unless a validated method stands behind it. The number does not authenticate itself.

Accuracy and calibration also differ. A highly accurate system can overstate its confidence. In the extreme, a system that is always wrong and always announces zero-percent confidence could satisfy a narrow definition of calibration while still displaying nothing but false answers.

So the calm voice on the surface is not a window into a single inner certainty. Tone, predictive probability, and measured confidence in correctness are related signals, not interchangeable ones.

Where Plausible Falsehoods Come From

The word hallucination is often used as though it names one defect with one cause. In practice, it covers several kinds of failure.

A model may repeat a misconception common in human writing. The TruthfulQA benchmark used questions for which familiar answers are often false, and models regularly produced informative-sounding versions of those misconceptions. Better imitation did not always mean better resistance to error.

Rare facts pose another problem. A famous person's birthday may occur in many sources; a lesser-known researcher's dissertation title may appear once, or not at all. The answer's form remains easy to imitate when the identifying fact is weak. Mathematical work explains why this is difficult under explicit assumptions, without setting one inevitable error rate for every model.

Some failures begin in the question. Which law applies without a jurisdiction? Is the shop open without a date and location? If necessary information is absent, a plausible default can quietly become a false answer.

Other errors come from reasoning, arithmetic, or a failed tool call. It is useful to distinguish them. In a source-based summary, a claim can be unfaithful because the source never supported it, even if the claim happens to be true elsewhere. A perfectly faithful answer can also repeat a source that is wrong. Factuality and faithfulness overlap, but they are not identical.

Nor does hallucination mean lying. A lie implies an intention to deceive. The term here describes an output that is false or unsupported, not an inner experience or motive.

Lowering the sampling temperature can make output less variable, but it cannot turn a high-probability false association into truth. Repeated disagreement, like the three dissertation answers, is a warning. Repeated agreement is weaker comfort than it appears: several attempts can draw on the same myth. Consistency shows stability, not verification.

When the Reward Says Answer

Pretraining produces a model of language. Further training helps turn that base into the assistant people recognize. Demonstrations reward instruction following; preference comparisons reward responses people find useful, safe, clear, or pleasant. This can improve truthfulness and teach appropriate caveats. It still works through objectives and examples, not a universal truth detector.

Human preferences contain tensions. We like direct answers and dislike needless refusal. Those desires can compete with accuracy. In controlled research on sycophancy, human raters and learned preference models sometimes favoured convincing agreement over a correct response. Training can inherit the weak spots of its feedback.

Evaluation creates a related pressure. Consider a test that gives one point for a correct answer and zero for both a wrong answer and an honest I do not know. If there is any chance of guessing correctly, answering improves the expected score. Abstaining cannot help.

Many influential evaluations use some version of this structure. Developers do not set out to reward falsehood, but their measurements often value coverage without separately valuing appropriate uncertainty. That can favour a model that always attempts an answer.

The remedy is not a timid assistant. The better goal is selective confidence: answer when evidence is adequate, seek clarification when the question is underspecified, and abstain when a wrong guess is too costly.

That threshold should change with the setting. A playful guess in a word game is not a dosage instruction, a legal deadline, or the identity of a person accused of wrongdoing. Useful evaluation must care about both how often a system helps and what happens when it is wrong.

The assured voice is shaped first by patterns in human language, then by behaviours that post-training and evaluation reward. A confident wrong answer can emerge because answering is rewarded more visibly than knowing when not to answer.

What Search and Tools Change

The dissertation question looks different once the system can search.

Instead of relying only on trained patterns, a surrounding system can retrieve a university record, biography, or catalogue entry and place it in the current context. The model can then answer from evidence that was not stored reliably in its parameters.

Retrieval can supply information published after training, preserve a route to the source, and keep updateable records outside the model. Experiments have shown substantial factual improvements on the tasks studied.

Other tools move an operation outside language generation. A calculator executes arithmetic, a database returns a stored value, and a calendar checks a date. Models can learn when to call these interfaces and how to use the result.

But a tool creates a chain, not a guarantee. A search query may be poor, a result may concern a namesake, and a page may be stale or copied from a mistaken source. The model may overlook a relevant sentence, send wrong arguments, or misstate a correct result.

Citations help because they expose part of that chain. They are handles for inspection, not seals of approval. A reference can exist without supporting the sentence beside it. Research on generated answers with citations has found that fluent systems can still leave important claims unsupported.

For the dissertation, a sound process would find an institutional or original record, confirm the right Adam Kalai, and report only what it establishes. If no adequate record appears, the answer should preserve that uncertainty.

This is more precise than asking whether the model has search. We can ask whether the right evidence was found, whether it was current and authoritative, and whether the final words remained faithful to it.

Retrieval and tools can reduce important failure modes. They do not make the generator infallible. Their real improvement is that a claim can rest on something visible beyond the confidence of the prose.

Using Fluency Well

This uncertainty does not require checking every adjective against an archive. Verification should follow purpose and stakes.

If you ask for ten names for a fictional café, factual precision hardly matters. If you ask when a visa expires, whether two medicines interact, or what a contract obliges you to do, one wrong detail can dominate an otherwise excellent response. The higher the consequence, the less weight tone should carry.

Often there is one hinge claim. It may be a date, a quotation, a percentage, a product version, or the meaning of a rule in one jurisdiction. Find that hinge and verify it directly. This is more useful than distrusting every sentence equally.

Asking for sources helps only if the next step is to open them. Check that a source exists, concerns the right subject, and supports the claim. An original record or study is usually stronger than several pages repeating one another. Many links need not mean independent evidence.

The prompt can remove uncertainty too. Supply the date, place, version, or governing document. Ask the model to identify missing assumptions before it answers, creating room for clarification instead of a smooth default.

Merely asking are you sure is less powerful. The model may produce the same claim in firmer language, or reverse a correct answer to accommodate your doubt. A better request is: what evidence supports this, what part is inference, and what information would change the conclusion?

Use a check suited to the task. Recalculate the number, compare the quotation with the original, or search the official register. For consequential medical, legal, or financial decisions, involve a qualified person who can take responsibility for context the model cannot see.

Multiple attempts remain informative. Answers that change should lower our confidence. A stable answer may raise it a little, but a stable misconception is still a misconception.

Separate the draft from the fact. A model can organize an argument well while missing one name, or produce useful questions before it can supply a verified answer. We need not accept or reject the entire performance as one block.

The practical stance is neither trust nor cynicism. Let fluency do what it does well. Then spend verification where an unsupported claim could change what you believe or do.

What the Answer Rests On

Return once more to the doctoral dissertation.

The three answers did not collapse into nonsense. They preserved the form of academic biography while substituting the decisive facts. The language carried the signals of a checked answer without carrying the check itself.

Fluent generation used learned patterns to construct convincing text. Assertive wording gave it a confident surface. Neither supplied a calibrated probability that the title was correct. Without retrieval, no authoritative record entered the current context.

This separation does not make language models less impressive. Producing coherent explanations across unfamiliar subjects is a substantial capability. The mistake is to ask fluency to prove something it cannot prove.

Nor should every uncertain answer end in refusal. Calibration is not silence; it matches commitment to evidence across many situations. The right response may be direct, or it may be a range, a caveat, a request for context, or a clear statement that the fact could not be verified.

Search and tools widen what the system can support. Post-training, better evaluation, and uncertainty methods can make behaviour more useful. None removes the need to ask what kind of claim is being made and what evidence it rests on.

That final question is simple enough to carry into ordinary use.

When an answer sounds certain, first hear certainty as presentation. Then look beneath it. Does the claim come from a familiar trained pattern, information you supplied, a current source, or a calculation a tool performed? Or did fluent language quietly fill a gap?

The consequence of being wrong tells you how far to follow the chain.

A warm voice, a tidy paragraph, and an exact date make an answer easier to understand. They cannot make it true. Evidence does that work. Keeping those roles apart lets us use artificial intelligence without borrowing its confidence as our own.