Sunday Latte: Does a Language Model Understand What It Says?

The Question Inside the Question

You ask a language model to explain a difficult paragraph. It identifies the argument, notices an unstated assumption, invents a useful analogy, and answers your follow-up. Then you ask whether it understands what it just said. The reply arrives in impeccable prose. That reply is not decisive evidence, because producing impeccable prose is the very ability under examination.

The question still matters. If a system understands, perhaps we should trust it differently, teach with it differently, or treat its mistakes differently. If it does not, perhaps its fluency is an especially elaborate illusion. Yet the simple choice between “yes” and “no” hides several questions that do not share one answer.

A person might understand the grammar of a sentence, the concept it expresses, the object it refers to, the speaker's intention, or the practical situation in which it is used. A person can also consciously experience a moment of comprehension. These meanings overlap in ordinary life because they usually arrive together. In an artificial system, they can come apart.

So this episode will not try to reveal a secret light behind the screen. It will ask what kind of competence we can observe, what kinds of connection may be missing, and what stronger evidence would change the case. The useful question is not simply, “Does it understand?” It is, “What does it understand, in which sense, and how do we know?”

A Real Command of Language

Start with the least mysterious sense. Modern language models have a substantial command of linguistic form. They can continue a sentence grammatically, relate a pronoun to an earlier noun, paraphrase an argument, preserve tone, and explain how two formulations differ. They do not succeed every time, but the pattern is far beyond a phrase book.

Researchers sometimes call this formal linguistic competence. “Formal” does not mean fake. It concerns the structures and regularities of language: vocabulary, syntax, acceptable combinations, and many relations among expressions. A model that distinguishes “the dog chased the cat” from “the cat chased the dog” is responding to structure, not merely counting the words.

Its competence also supports novelty. It can often interpret a newly composed sentence or an unfamiliar combination, combine familiar ideas in an unfamiliar way, and apply an instruction to a new example. Memorisation exists, and some prompts expose brittle shortcuts, but memorisation alone is not a good account of all this behaviour. The model has learned distributed regularities that support generalisation.

That is evidence of something. Calling the system a random parrot would fail to explain why its responses are sensitive to grammar, context, and conceptual relationships. Calling it “just autocomplete” is similarly unhelpful. Autocomplete describes the output procedure at a very high level; it does not tell us how much structure must be learned to make a useful continuation.

But formal competence is not the whole question. A person can manipulate a sentence in an unfamiliar language by following rules without knowing what it refers to. A system may preserve relationships among words while lacking some connection that human speakers take for granted. To see that gap clearly, we need an octopus and two islands.

The Octopus Between Two Islands

Imagine two people stranded on different islands. They communicate through a cable under the sea. A very clever octopus taps the cable and studies the patterns. With enough messages, it becomes excellent at predicting what one person will send after the other. Eventually it can cut the cable and imitate either side so well that the humans may not notice immediately.

The octopus has access to the form of their communication, but not to the world that gave it meaning. When one islander writes about a coconut falling beside the shelter, the octopus has not seen the coconut, heard the impact, or shared the need for shelter. If the other asks for instructions to escape a bear, statistical fluency on the cable may be dangerously insufficient.

This thought experiment sharpens a grounding objection. Human words point beyond other words. “Mango” participates in sentences, but it also refers to a kind of fruit that can be held, smelled, cut, traded, and eaten. “Promise” belongs to social practices involving people, expectations, and consequences. If a learner receives only patterns of symbols, what connects those symbols to the things and purposes they are about?

The objection is strongest when a task depends on a new practical situation. Predicting the kinds of sentence people write about ladders is not identical to knowing whether a damaged ladder will carry a person's weight. A fluent answer can borrow the surface of human judgment without possessing the causal contact that produced that judgment.

Yet the octopus does not end the debate. Real training text is not an arbitrary code emitted by ungrounded machines. It was produced by people whose language reflects bodies, objects, institutions, and shared history. The question becomes whether relations carried through that text can convey some meaning without giving the learner the same route by which humans acquired it.

Meaning Among Relations

Consider how you understand the word “mango”. Direct experience may contribute colour, texture, scent, and taste. But your concept also includes relations learned in language: a mango is a fruit; it grows on a tree; it has a stone; it appears in recipes; it differs from an apple. Much of what any person knows about distant places, extinct animals, or subatomic particles comes through descriptions rather than direct encounter.

One reply to the grounding objection starts here. Meaning may reside partly in a concept's role within a larger network. If an internal representation reliably relates mangoes to fruit, ripening, sweetness, trees, markets, and countless contrasting concepts, then it carries structure that supports explanation and inference. On this view, reference is not the only possible source of semantic content.

Language models show substantial sensitivity to conceptual relations. In many tested settings, they can place ideas into categories, follow analogies, track some entailments, and use words in ways sensitive to context. Because human text is shaped by a grounded world, its statistical structure is not empty. The cable between the islands contains traces of coconuts, shelters, fear, cooperation, and plans, even if the observer never visits the beach.

This argument does not prove that a model's meanings are the same as ours. A relational map can be detailed while its connection to a particular situation remains fragile. The model may know that mangoes are sweet yet lack the sensory distinction between sweetness and a description of sweetness. It may inherit contradictions and stereotypes from the language network. It may also lack durable purposes that make one interpretation matter more than another.

The disagreement is therefore not between people who noticed model capability and people who did not. It is partly a disagreement about what meaning consists of. If conceptual role is enough for an important kind of meaning, language models may possess some. If reference through shared action and communicative intention is essential, text prediction alone leaves a central gap.

When the System Meets the World

Language models increasingly do more than receive text and return text. A multimodal model can connect words with images and sound. A tool-using system can search a database, read a sensor, run an experiment, or act in software. A robot can relate instructions to cameras, movement, resistance, and consequences. These connections matter because the symbols begin to participate in a feedback loop with an external environment.

Suppose a model first says that a box will fit through a doorway. A camera supplies measurements, a tool checks the geometry, and an attempted movement fails. A capable system can revise its plan from the observation. That is stronger practical evidence than generating a plausible paragraph about boxes. The system is no longer limited to associations already present in prose; some claims can meet resistance from the world.

But we should name the layer carefully. The trained language model, the camera, the measuring tool, the software that routes observations, and the machine that acts are distinct components. The whole system may display grounded competence that no component displays alone. Saying “the model saw the doorway” can be convenient shorthand, but it can also hide which information was available and how it affected the next action.

More senses do not settle every question either. An image is another structured signal, not a magical dose of human experience. A camera does not by itself supply pain, need, social membership, or a reason to care whether the box arrives. Tools can correct facts without producing communicative intentions. A robot can act effectively without feeling the weight it lifts.

Still, grounding comes in degrees and varieties. Text relates symbols to other symbols. Images connect some of those symbols to visual regularities. Action connects predictions to consequences. Social interaction connects utterances to other participants' responses. Instead of demanding one mythical switch from ungrounded to grounded, we can ask which loops are present and which remain absent.

The Tests That Matter

If understanding is more than repeating familiar patterns, the most informative tests should require flexible use. Can the system explain the same idea to a child and to a specialist without contradicting itself? Can it apply a principle in a genuinely new setting? Can it notice that a request rests on a false assumption, revise after evidence, and distinguish a metaphor from a physical claim?

Success on such tasks supports an attribution of competence. It is evidence that the behaviour extends beyond reproducing one familiar sentence. Persistent failure reveals boundaries. A model that recites a rule but cannot recognize a simple counterexample has a thinner command of the rule than its first answer suggested.

No single benchmark resolves the issue. Test questions can leak into training. A prompt can accidentally reward a shortcut. An impressive average may hide systematic failure under small changes in wording. Conversely, one carefully designed trick can make a capable reasoner look foolish. Humans misunderstand ambiguous questions, forget facts, and fail under adversarial conditions too. The relevant comparison is not perfection but the pattern, flexibility, and cause of success and failure.

Long interaction can add evidence unavailable in one exchange. Does the system preserve the object of discussion, integrate correction, and use a new distinction later? Or does it merely echo the last formulation? An external notebook or retrieval tool may support that continuity, so once again we must distinguish the model's unaided competence from the competence of the assembled product.

The best tests are tied to a claim. If we claim grammatical competence, test unfamiliar constructions. If we claim physical understanding, test prediction and action in changing environments. If we claim social understanding, examine perspectives, norms, and repair across contexts and cultures. “Understands language” is too broad to fail cleanly, which also makes it too broad to confirm.

Understanding Is Not Consciousness

At this point another question often enters unnoticed: does the model have an inner experience of meaning? That is a question about consciousness, not simply competence. The two may be linked in humans, but treating them as synonyms makes the evidence impossible to sort.

A system could use a concept reliably, connect it to perception, and act appropriately while we remained uncertain whether anything felt like comprehension from inside. Conversely, a conscious creature might misunderstand a sentence. Functional success and subjective experience are different properties.

The model's own declaration cannot settle the matter. It can say, “I truly understand,” or, “I am only a machine,” because both kinds of sentence occur in its training and can be encouraged by instructions. A self-report is meaningful evidence in human relationships partly because it comes from a creature with a body, a history, continuing interests, and broadly shared biology. The same words generated by a language model do not inherit that evidential foundation automatically.

This does not justify certainty in the opposite direction. Consciousness is difficult to explain even in animals and other people; confident declarations about every possible artificial system run ahead of available science. Architecture, recurrent processing, persistent self-models, sensory integration, agency, and other features may matter. Researchers disagree about which are necessary.

For practical discussion, we can leave that uncertainty intact. We do not need to decide whether a calculator feels arithmetic before checking its answer, and we do not need to decide whether a language model has experiences before identifying its competencies and limits. Moral questions may eventually demand more, but fluency alone should neither award consciousness nor rule it out by slogan.

Two Bad Shortcuts

One bad shortcut is anthropomorphism: the model says “I believe”, so we assume a stable believer with intentions behind the sentence. That language can obscure the actual mechanism, the role of prompts and tools, and the responsibility of the people who designed or deployed the system. It can also make a confident error feel like a personal assurance.

The opposite shortcut is dismissal: the model predicts tokens, therefore nothing it does counts as understanding. That conclusion does not follow from the description. A sequential output procedure does not by itself tell us what representations support it. For a model to predict well across many contexts, it must capture significant structure. The scientific question is what structure, how robustly, and with which connections—not whether prediction sounds humble enough.

Both shortcuts turn a complex empirical question into a loyalty test. One side collects impressive conversations and treats resemblance as proof of a human-like mind. The other collects absurd failures and treats difference as proof of empty mimicry. A useful account has to explain both: the genuine competence that makes the system valuable and the peculiar brittleness that makes unqualified trust dangerous.

Ordinary language can survive if we add precision when it matters. It is harmless to say a navigation system “knows the route” while remembering that its knowledge differs from a taxi driver's. Likewise, saying a model “understood the instruction” can mean that it acted in accordance with the relevant distinctions. Trouble begins when that local success silently expands into claims about reference, intentions, reliability, or experience.

Calibrated language is not pedantry. It lets capability and caution coexist. The model need not be a person to perform meaningful intellectual work, and impressive work need not make it a person.

A Better Question to Carry

Return to the difficult paragraph from the beginning. The model summarized its argument, exposed an assumption, built an analogy, and answered a follow-up. We have good reason to say it displayed linguistic and analytical competence in that exchange. We may also find evidence that its internal representations capture conceptual relationships. Those claims are stronger than “mere copying” and narrower than “it understands exactly as a person does”.

What remains uncertain depends on the sense we care about. Did the system connect the words to the relevant objects and consequences, or only to other descriptions? Could it transfer the idea into a changing practical situation? Were search, memory, sensors, or tools doing part of the work? Did it have a communicative purpose of its own? Did any of this involve subjective experience? One successful paragraph cannot answer all of those questions.

Four prompts make the discussion clearer. First: understanding of what—grammar, a concept, another person's aim, or a physical situation? Second: what behaviour would demonstrate that understanding beyond familiar examples? Third: which part of the system supplies the evidence—the trained model, its context, its tools, or the full loop? Fourth: what failure would make us revise the claim?

This approach refuses the dramatic verdict, but it is not evasive. It replaces one overloaded word with testable claims. The behaviours examined here are genuine forms of competence, whether or not they settle the deeper question of meaning. Language models also lack, or have not demonstrated, several connections bundled into ordinary human understanding. Future systems may close some gaps, reveal new ones, or force better theories of meaning itself.

So does a language model understand what it says? Sometimes it behaves as though it understands particular content in a useful, limited sense. Whether that competence amounts to meaning in a deeper sense depends on a contested theory and on connections the system can actually demonstrate. Whether it consciously experiences meaning is a further open question. The honest answer is not a shrug. It is a map: specify the sense, identify the evidence, name the layer, and keep the conclusion no larger than the test.