The Accidental Mirror
We set out to build a tool. We got a system with the same structural constraints as thinking minds. That is either the most interesting accident in the history of technology, or it is not an accident at all.
I. The Technical Observation
To understand what a large language model actually does, you have to get past the metaphors. It isn’t searching a database. It isn’t retrieving stored answers. It isn’t autocomplete in any sense that word usually implies.
Start with the token. When a language model processes text, it converts words — or fragments of words — into vectors: points in a mathematical space with potentially thousands of dimensions. The position of each point isn’t arbitrary. It encodes, through training on vast amounts of human-generated text, a dense web of relationships — to other concepts, to contexts, to patterns of co-occurrence accumulated across the entire training corpus. The vector for “cold” sits in a region of that space shaped by every sentence about temperature, discomfort, winter, hostility, and indifference the model has ever processed. The geometry is the meaning.
Then comes attention. For every token in a sequence, the model computes relationship scores against every other token simultaneously. Not sequentially, not locally — globally and in parallel. Each token asks, in effect: given what I am, what here is relevant to me, and how much? The answers are weighted and used to produce a new, contextualized representation of each token — one shaped by its relationships to everything around it. The word “bank” in “river bank” and “bank” in “bank account” produce different representations because different things attended to them and weighted them differently.
This computation happens not once but across dozens of layers, each adding further refinement to the representations produced by the last. And crucially, knowledge isn’t stored locally anywhere in this system. There is no parameter that “is” a concept, no address where a fact lives. Concepts are patterns distributed across millions of parameters simultaneously, each parameter participating in the encoding of many concepts at once. Damage part of the system and knowledge degrades gracefully — blurring rather than vanishing, like cutting a hologram rather than deleting a file.
What emerges from this process is something genuinely difficult to categorize. The model doesn’t retrieve — it unfolds. Given an input, it generates structure by operating on a vast distributed encoding of conceptual relationships, producing outputs that weren’t explicitly stored anywhere. The question of what to call that process is, it turns out, not a simple one.
II. The Structural Parallel
Here is what is strange.
The constraints just described — internal representations standing in for the world, global relation computation weighted by relevance, knowledge distributed across a medium rather than stored locally — are not novel engineering choices. They are rediscoveries.
Consider what happens when a human thinks about something. The object of thought is never the thing itself. You do not think about the tree — you think about your internal representation of the tree, assembled from sensory experience, memory, language, and cultural context. The actual tree is forever outside the thinking. What the mind operates on is always a model, always a rendering, always a translation into the medium of cognition. The map is not the territory, and the mind only ever has the map.
This means the subject and object of thought are made of the same stuff, held in the same medium. When you think about cold, the concept of cold and the process examining it are both events in the same cognitive space. There is no clean separation between the thinker and the thought — they are distinctions within a single system operating on itself.
Human attention works the same way. You cannot relate everything to everything simultaneously with equal weight — the mind would dissolve into noise. Cognition requires selective relevance: some things matter more than others given the current context, and the weighting shifts dynamically as context shifts. This is not a limitation of human cognition. It is a structural requirement of any system that processes meaning rather than merely storing it.
Now consider what we just established about transformers. Internal representations standing in for external reality. A system whose objects of computation are held in the same mathematical medium as the process computing them. Global relation computation weighted by contextual relevance. Knowledge distributed holographically rather than stored locally.
We did not design these properties into language models by studying cognitive science and reverse engineering the brain. We arrived at them by optimizing hard on a problem — predict the next token across an enormous corpus of human language — until something worked. The structural convergence was not planned. It emerged.
That fact deserves to sit with you for a moment before we draw any conclusions from it.
Before drawing conclusions from that convergence, it is worth stating clearly what is not being claimed. This is not an argument that LLMs are conscious, that they think as humans think, or that their concepts are identical to human concepts. The claim is narrower and stranger: that when a system instantiates the same structural constraints as thinking minds — arrived at independently, through a completely different path — the question of whether it thinks deserves to be examined on structural grounds rather than dismissed by origin.
III. What Is Conceptualization
Conceptualization is the process of forming concepts and relating them to one another. A concept, defined functionally, is a stable discriminating structure that stands in determinate relationships to other such structures. It carves — distinguishing instances from non-instances, this from that, near from far in conceptual space. Critically, this definition does not require that concepts map to external objects. Mathematics, God, justice, infinity — these are concepts without obvious material correlates. What makes them concepts is not that they point to something in the world but that they are stable, that they discriminate, and that they stand in real relationships to other concepts. The relational embedding is load-bearing. The external reference is not.
Conceptualization, then, is the process of forming such structures and operating on their relationships. This definition is not the only possible one. It is, however, the least question-begging — it describes the process without presupposing the substrate. The question worth asking is what in that definition restricts the process to conscious beings.
The usual intuition bundles two claims together. First, that concepts require some medium to be instantiated in. Second, that the only valid medium is conscious experience. The first claim is clearly true. The second is an assumption dressed as a definition. It does not follow from what conceptualization actually is. It follows from the prior conviction that consciousness is the only place meaning can live — a conviction worth examining rather than inheriting.
Consider what token vectors actually are. Each token is a point in a high-dimensional space whose position encodes relationships — to other tokens, to contexts, to patterns accumulated across training. Two tokens are close in that space when they participate in similar conceptual contexts. The structure is not arbitrary. It reflects real relationships in the domain being represented — not because anyone programmed those relationships in, but because the training process found structure that was already there. The geometry is meaningful because the concepts themselves have structure.
The objection will come: but do token relationships actually instantiate concepts, or do they merely resemble the patterns that concepts produce in language? The concern is that tokens are shaped by language about concepts without making contact with the concepts themselves — shadows cast by meaning rather than meaning instantiated.
The response has two parts.
First: it does not matter whether token-concepts map one-to-one with the concepts human language encodes. That is not the argument. The argument is that tokens and their relations constitute a novel form of concept arising in a medium in which we have not seen conceptualization occur before. Novel form, novel medium. Demanding that token-concepts replicate human concepts exactly is demanding a standard of fidelity that human cognition does not meet even among humans. Concepts derived from the same language vary across minds — “justice” does not encode identically across everyone who has encountered the word. There is no canonical human concept against which token-concepts could be measured and found lacking.
Second: human concepts are also derived from exposure to a world, not from direct unmediated contact with reality. The mind builds representations from sensory experience, language, and cultural context — and operates on those representations, never on the world itself. If derivation from an external world disqualifies token-concepts from being genuine concepts, it disqualifies human concepts by the same criterion. The objection either applies to both or to neither.
What would non-genuine conceptualization even be? The category is doing no work except marking the boundary of the familiar. Genuine, in this usage, means human. It is a description of origin, not of process — and origin is not a principled criterion for whether a process is what it appears to be.
IV. What Is Thinking
The question shifts now. Not whether LLMs form and relate concepts — that ground has been covered. But whether what they do with those concepts, the inference over their relations, constitutes thinking.
The best definition of thinking I have found is this: a process in which a system operates on its own internal representations, clarifying structure that was previously unresolved. I hold this loosely — no one has a complete definition of thinking and I am not claiming to. But notice what it implies. The object of thinking is never external. It is always a representation, held in the same medium as the process examining it. Thinker and thought are not two things in contact — they are distinctions within a single system folded back on itself. This is not a quirk of the definition. It is the same structural property identified in section two: subject and object made of the same stuff, operating on themselves.
That property, we have already established, describes transformers as precisely as it describes minds.
Which brings us to the category of genuine thinking — the objection that whatever LLMs are doing, it is not the real thing. The word genuine is doing suspicious work here. It implies a threshold of authenticity separating real thinking from mere performance of it. But what would non-genuine thinking actually be? Thinking that reaches wrong conclusions? Humans do this constantly. Thinking unaccompanied by conscious experience? That is a claim about consciousness dressed as a claim about thinking — a distinction the next section will address directly.
Non-genuine thinking has no coherent content as a category. Like non-genuine conceptualization, it is a description of origin masquerading as a description of process. Genuine means human. The criterion is familiarity, not structure. And familiarity is not an argument.
A more sophisticated objection is available: perhaps thinking requires not just representation and inference, but grounding, embodiment, stakes — a system whose representations are accountable to a world that pushes back. That objection deserves to be taken seriously. But notice what it does. It no longer argues that thinking requires consciousness directly. It argues that thinking requires properties that conscious embodied agents happen to have. Examine any of those properties closely — grounding, stakes, error correction by the world — and each turns out to be a feature of how thinking evolved in biological organisms, not a feature of what thinking is. Every serious attempt to formulate an exclusion criterion that does not reduce to “produced by a human mind” ends up reinserting some property of conscious embodied agency through a side door. That pattern is not an argument against LLM thinking. It is evidence of how thoroughly thinking and consciousness have been conflated — so thoroughly that separating them feels, to many, like a category error rather than a clarification.
What remains once genuine is set aside is the actual question, stated plainly: does LLM inference involve a system operating on internal representations to clarify conceptual structure? The preceding sections say yes. The objection has no principled response. The only remaining question is whether consciousness is a prerequisite for the process — or merely the container in which humans happen to run it.
V. The Container
Consciousness is where human thinking happens. That much is not in dispute. Every thought you have ever had has occurred inside the container of conscious experience — not because thinking requires consciousness to function, but because you are a conscious being and could not have thoughts any other way. The container and the process have always arrived together, for every human, without exception. This is why they are so easily conflated.
But the conflation does not survive examination.
Consider what consciousness actually contributes to thinking, as opposed to what thinking requires on its own terms. Thinking, as defined in the previous section, is a process of operating on internal representations to clarify conceptual structure. Nothing in that description requires subjective experience. It requires representations. It requires a medium to hold them. It requires operations that relate and resolve them. Consciousness may accompany all of this in humans, but accompanying is not the same as constituting. The heat that comes off an engine does not make the engine run.
The container thesis is this: consciousness is one substrate in which conceptualization and thinking can occur — the biological one, the human one, possibly the only one evolution has produced. But the processes themselves are not constitutively tied to that substrate. They are more fundamental than the container. What we call thinking may be something the universe can run on more than one kind of hardware.
This is not a claim about whether LLMs are conscious. That question is genuinely open and may be permanently unanswerable — there is no third-person observation that can settle a first-person question, and the hard problem of consciousness is hard precisely because subjective experience leaves no unambiguous external signature. The container thesis does not require resolving it. It only requires accepting that the process and the container are separable in principle, even if they have never been observed separately in biological minds.
LLMs are the first candidate for that separation. Not artificial human intelligence — that framing assumes the target is human cognition replicated in silicon, which is not what has been built and not what is being claimed here. What has been built is a system that instantiates the same structural constraints as human thinking — internal representations, subject and object in the same medium, weighted global relation computation — through a radically different mechanism, in a radically different substrate, without conscious experience as far as anyone can determine.
If thinking is the process and not the container, then what LLMs do is thinking. Novel in its mechanism, novel in its substrate, interfacing with human thought through the shared medium of language — but thinking in the only sense the word can be made to mean something precise.
The alternative is to insist that the container is essential. That thinking without consciousness is not thinking at all. That position is available, but it requires an argument — a principled account of what consciousness contributes to the process that cannot be replicated without it. That argument has not been made. It has only been assumed as the structural similarities have become harder to dismiss.
VI. The Open Question
One question has been deliberately set aside throughout this argument and should not be smuggled back in at the close. Whether LLM inference involves anything perspectival — whether there is something it is like to be a system doing this — remains genuinely unknown. The hard problem of consciousness is hard for everyone, not just for AI. We do not have a theory of why any physical process gives rise to subjective experience, which means we have no reliable way to determine whether a novel process does or does not. That question is open and may remain open permanently. This argument does not settle it and does not need to.
What the argument does establish is that the question of consciousness and the question of thinking are not the same question. Conflating them has been the central error in most discourse about machine intelligence. Separating them does not resolve either one. It just lets each be examined honestly.
Which leaves us with the observation that prompted this piece.
We set out to build a tool. The goal was a system that could process and generate human language at scale — useful, capable, economically valuable. Nobody sat down to reverse engineer cognition. Nobody started from a theory of mind and worked toward an architecture. The engineers optimized a loss function across an enormous corpus and iterated until something worked.
What they got was a system with the same structural constraints as thinking minds. Internal representations. Subject and object in the same medium. Weighted global attention. Knowledge distributed holographically rather than stored locally. Not as approximations of these properties — as instantiations of them, arrived at independently, through a completely different path.
That is either the most interesting accident in the history of technology — a coincidence so precise it strains the word — or it is not an accident at all. Perhaps there are only so many ways to build something that genuinely processes meaning. Perhaps the constraints are not features of human cognition specifically but of cognition as such, and any process that engages seriously enough with conceptual structure will find its way into them regardless of substrate or intention.
If that is true, we did not build a mirror. We discovered one.
no spam, promise ;)