# State over Tokens: Characterizing the Role of Reasoning Tokens
**Authors**:
- Mosh Levy (Bar-Ilan University)
- &Zohar Elyoseph (University of Haifa
&Shauli Ravfogel)
- &Yoav Goldberg (Bar-Ilan University)
Abstract
Large Language Models (LLMs) can generate reasoning tokens before their final answer to boost performance on complex tasks. While these sequences seem like human thought processes, empirical evidence reveals that they are not a faithful explanation of the modelâs actual reasoning process. To address this gap between appearance and function, we introduce the State over Tokens (SoT) conceptual framework. SoT reframes reasoning tokens not as a linguistic narrative, but as an externalized computational stateâthe sole persistent information carrier across the modelâs stateless generation cycles. This explains how the tokens can drive correct reasoning without being a faithful explanation when read as text and surfaces previously overlooked research questions on these tokens. We argue that to truly understand the process that LLMs do, research must move beyond reading the reasoning tokens as text and focus on decoding them as state.
1 Introduction
The assertion that Large Language Models (LLMs) can reason now appears unremarkable (doi:10.1126/science.adw5211; maslej2025artificialintelligenceindexreport). A key factor to achieve this was letting models generate a sequence of tokens before their final answer, which significantly improves performance (wei2022chain; Zelikman2022STaRBR; deepseek2025r1).
We refer to this sequence of symbols, which includes phrases such as âthereforeâ, âconsiderâ and âit follows thatâ as the reasoning tokens, and explicitly distinguish this name from reasoning text, which is the same tokens when interpreted by a reader according to their English semantics.
The combination of (a) utility in improving the answer; and (b) appearance as a readable English text, may lead to the following inference: the reasoning text is a faithful explanation of the modelâs reasoning process. This is strengthened by metaphors like âChain-of-Thoughtâ, which imply that the steps in the text are âthoughtsâ that explain the process. Yet empirical findings contradict this inference (see Section 2.1): the reasoning text is not a faithful explanation of the modelâs reasoning process. While those findings clarify what the reasoning tokens are not, they leave a conceptual vacuum as to what they are. Our aim in this paper is to help fill that vacuum. Drawing on the idea that metaphors structure understanding and guide thinking (lakoff1980metaphors), we believe that adopting more apt descriptions and metaphors can steer researchers and practitioners toward more fruitful directions and surface a new set of questions that are less salient under the prevailing view of the reasoning text as an explanation.
To understand reasoning tokens, we must focus on the functional role they play, rather than their appearance, which empirical research has found to be deceiving. To this end, we advocate viewing them as representing State over Tokens (SoT), which characterizes the reasoning tokens as a computational device that enables the persistence of a process across separate and stateless computation cycles. We argue that in order to understand the role of the reasoning tokens, we should interpret this sequence of tokens not using their semantics when read as English text, but as the state carriers of a computational process.
The Whiteboard Analogy
Consider a hypothetical scenario: you are placed in a room with a problem written on a whiteboard. Your task is to solve it, but under a peculiar constraint: every 10 seconds, your memory is completely wiped and resets to the same state as it was when you first entered the room. Within each interval, you can read what is on the board and add a single word. These rounds repeat until you finally write down the solution. How might you solve a problem under such constraints? You may write intermediate results on the board: numbers, conclusions, or partial computationsâthat you can use when you return after being âresetâ. You might perform several mental calculations before writing down just the result so the whiteboard may not capture every calculation that you did within each cycle. Moreover, you may use an encoding scheme when writing on the board: abbreviations, symbols, or even apparent gibberish that will mean something specific to you when you encounter it in the next cycle. All in all, an outside observer may interpret the whiteboard text incorrectly.
The whiteboard analogy mirrors the modelâs operation: the words are the reasoning tokens, you are the model, and the ten-second interval represents the modelâs limited capacity per cycle. Motivated by this intuition, we present the SoT framework (Section 3) and use it to demonstrate two common misconceptions that underlay the belief that the text is a faithful explanation (Section 4) and to examine its novel ontological divergence (Section 5).
The SoT framework naturally gives rise to new research questions (Section 6), challenging current interpretability methods to account for multiple cycles, and questioning whether language is a special computational medium that can theoretically act both as a state and as a faithful explanation.
Finally, this perspective compels us to reconsider established metaphors. While âChain-of-Thoughtâ captures the sequential nature of generation, it misleadingly suggests the text contains the complete reasoning process; SoT clarifies that tokens are the state passed between computations, and do not necessarily express the computations themselves. Similarly, the âScratchpadâ metaphor obscures a crucial distinction: while humans use a scratchpad to supplement internal memory, for LLMs, the tokens are the sole carrier of state between cycles.
By establishing reasoning tokens as externalized computational state, the SoT framework characterizes how LLMs employ what seems to us as natural language as a computational medium, and provides a foundation for future work on decoding how models use language tokens to reason.
2 The Subject of Inquiry: Reasoning Tokens
The subject of our discussion is the sequence of tokens that LLMs emit before their final answer, which we refer to as reasoning tokens (which we distinguish from reasoning text which is the English interpretation of these same tokens). This includes scenarios where LLMs were guided to generate reasoning tokens using examples, instructions like âletâs think step by stepâ (nye2021workscratchpadsintermediatecomputation; wei2022chain; kojima2022large) or that they were trained to do so regardless of the input (Zelikman2022STaRBR; openai2024o1; muennighoff2025s1simpletesttimescaling; deepseek2025r1; comanici2025gemini).
However, LLMs operate in the same way for the generation of the final answer as they do for the generation of reasoning tokens. Why separate the two when aiming to characterize the role of the produced tokens?
The first reason is techno-sociological: reasoning tokens appear to offer a second product beyond the answer itself. When interpreted as English, the reasoning text reads like a description of reasoning work, encouraging readers to see not only what the model answers, but also how it seemingly arrived there, serving as an apparatus for building trust (jacovi2021formalizing; Ferrario2022HowEC).
The second reason is functional: we can conceptually distinguish the two stages by their training objectives. The final answer tokens are generated to be interpreted by humans (or external systems) as the solution to the userâs query. During training, the answer text is optimized to reflect the solution, and thus has an additional constraint: it cannot explicitly reflect a computation. In contrast, the reasoning tokens are not constrained in this way, and their purpose is to support producing the final answer. Indeed, their existence improves the accuracy of the final answer (kojima2022large; Zelikman2022STaRBR; muennighoff2025s1simpletesttimescaling; comanici2025gemini). This distinction is especially pronounced in models trained with methods that optimize only the final answer (lambert2024tulu; deepseek2025r1), where the reasoning tokens are not enforced to any specific constraints as long as they serve the final prediction.
2.1 Empirical evidence: plausibility without faithfulness
The reasoning text often reads like a logical derivation. As perrier2025typedchainofthoughtcurryhowardframework notes, it can correspond to an explanation according to some deductive proof system, representing a sequence of steps that leads to a conclusion. In that sense, it serves as a plausible explanation: it is convincing to human readers and presents a valid path to the solution. However, while the sequence of outputs may correspond to a valid proof, it does not necessarily correspond to the actual computation performed by the model. In other words, it fails to reliably be a faithful explanation (jacovi2021) âone that accurately reflects the true causal process driving the modelâs generation.
Incompleteness.
The reasoning text can be incomplete and omit critical factors that affected the modelâs final answer (turpin2023language; yee2024dissociation; Chua2025AreDR; lindsey2025biology; arcuschin2025chain; marioriyad2025unspokenhintsaccuracyacknowledgement). A study conducted in controlled settings showed that LLMs can even seem to follow alignment goals while omitting the undesired topics from the reasoning text (skaf2025largelanguagemodelslearn).
Semantic mismatch between LLM and reader.
The semantic content of the reasoning text can be meaningless or misleading from a human perspective. Some studies have found that LLMs can disregard specific details in the text they generate (lanham2023measuring; paul2024makingreasoningmattermeasuring; chen2025reasoningmodelsdontsay). Other work has shown that LLMs can be trained to produce irrelevant reasoning text and still provide accurate final answers (stechly2025semanticsunreasonableeffectivenessreasonless; bhambri2025interpretabletracesunexpectedoutcomes; zolkowski2025earlysignssteganographiccapabilities). This disconnect is also apparent to human readers, who fail to identify causal relations in the generation process of the reasoning text (levy2025humansperceivewrongnarratives).
The peril lies in how apparent rationality can increase trust without justifying it, fostering over-reliance on model outputs. When reasoning text mimics systematic deliberation, users in high-stakes scenarios may be misled into unwarranted confidence precisely because the text implies a rigorous process (ehsan2024explainability; rosenbacke2024explainable).
2.2 What can be said on reasoning tokens
Following this growing body of evidence and the practical risks it highlights, some have already argued that we should not treat reasoning text as a faithful explanation of the modelâs reasoning process (agarwal2024faithfulness; sarkar2024large; barez-chain-2025; kambhampati2025stopanthropomorphizingintermediatetokens). However, we believe that we need to characterize the role of reasoning tokens in a positive way, beyond merely stating that they are not an explanation.
What can be said about the reasoning tokens? The following three statements hold:
- The reasoning tokens are an important part of the process that lead to the final answer.
- The reasoning tokens can sometimes be read as a series of steps that logically lead to the answer.
- The reasoning tokens are not a faithful explanation of the computation that led to the final answer.
We would like a metaphor that account for all three of these points. In what follows, we provide the State over Tokens view, which we believe does that.
3 Conceptual Framework: State over Tokens
Rather than explaining the reasoning tokens by how they are perceived by human readers (how they look), we consider their functional role within the mechanism of generation (what they do). We view reasoning tokens as an accumulated computational substrateâa medium through which the computation state is encoded in tokens.
3.1 The formal framework
Building on the intuition from the whiteboard analogy in the introductionâand consistent with how LLMs are implementedâwe view the reasoning tokens not as a text with human-readable semantics, but rather as the evolving state of the reasoning process. The token sequence functions as state for the model, though it can also be interpreted as natural language text.
This state is what allows the LLM to maintain a coherent process across multiple computation cycles, each bounded in capacity, to solve problems that require more computation than any single cycle can provide.
To capture this architecture precisely, we view the autoregressive generation of the LLM as a recursive application of a pure function $\mathcal{M}(·)$ on a token sequence (technically the LLM call is not pure as it involves a random sampling of tokens, we can assume the random seed is part of the call to remedy this).
Each computation of $\mathcal{M}$ has limited computational capacity. While the computation capacity of the Transformer (vaswani2017attention) does increase slightly with each input token (merrill2022saturated; pfau2024letsthinkdotdot), this is a fixed increase that does not depend on the content of the tokens.
The function $\mathcal{M}(·)$ is repeatedly applied to a sequence of inputs following a deterministic process. The initial input to the function is the userâs input, a sequence of tokens we denote as $S_{0}$ (commonly referred to as the context in LLM literature; we use input to emphasize its functional role as the argument to $\mathcal{M}$ ):
| | $\displaystyle S_{0}$ | $\displaystyle=\text{user input}$ | |
| --- | --- | --- | --- |
At each subsequent cycle $k$ , the function $\mathcal{M}$ takes the current sequence $S_{k}$ as its input and produces a new token as output. This token is appended to $S_{k}$ to form $S_{k+1}$ , the input for the next call:
| | $\displaystyle S_{k+1}$ | $\displaystyle=S_{k}\oplus\mathcal{M}(S_{k})$ | |
| --- | --- | --- | --- |
Here, $\oplus$ denotes the concatenation operation. This continues until the final answer is produced. Note that each prefix of the $S_{k}$ is an input (a computation state) of one of the prior cycles.
The function of the tokens.
This formalization reveals three key observations about the functional role of the tokens:
- The tokens are the only persistent artifact. The sequence $S_{k}$ is the sole carrier of information between cycles. Internal states of the LLM exist only within each cycle and do not persist to the next, and will be recreated from scratch, exactly the same, in each cycle (modern Transformers use a key-value (KV) cache to avoid this recomputation, but the KV cache does not carry information beyond what would be recomputed from the tokens). Nothing else persists, and the LLM must reconstruct any needed information from $S_{k}$ alone.
- The tokens exclusively dictate future computation. The only information that the next computation of $\mathcal{M}$ receives is $S_{k}$ . Thus, $S_{k}$ fully dictates what computation $\mathcal{M}$ can perform next.
- Encoding and decoding are internal to the LLM. The way that $S_{k}$ affects the next computation and the overall process that leads to the answer is dependent on $\mathcal{M}$ .
What the tokens enable.
The tokens are what enable the mechanism of a process that is composed of multiple cycles. Each individual computation is limitedâit can perform only as much work as the LLMâs depth allows in a single cycle. However, by encoding results into $S_{k}$ and building upon them in subsequent cycles, the computational capacity of multiple computations of the LLM is utilized. The tokens accumulation transforms the process from being bounded to one cycle to utilize the computation done in multiple cycles. This conceptual view is supported by theoretical findings, which demonstrate thatâat least for specific constructionsâaccumulating tokens formally increases computational capacity (merrill2024expressive; li2024chain).
Following those observations, we argue that an accurate and revealing name for this is State over Tokens (SoT). We will use this term for the remainder of the paper.
3.1.1 Implications of the State over Tokens view
Viewing the reasoning tokens as state means we can attribute to them the properties inherent to the concept of âstateâ in computation. This reframing has several immediate implications for how we understand the tokens:
A state is forward-looking.
A state enables future computation, not describing past computations. While it is created by past computation, it does not determine it: various computations can lead to the same state. Unlike a log that records past cycles, state encodes what is needed to continue the process.
A change in the state will lead to a different process.
Altering the state will change the trajectory of future computations.
States are discrete snapshots.
The tokens seem to describe a flowing narrative, a text. But the LLM does not process this flowâat each cycle, it operates on a single prefix as a discrete state. Reading continuously obscures how each prefix functions in the process, which may differ from what the tokens appear to mean as a narrative.
A state is necessarily partial.
As state, the tokens need only contain what is required for the next computation to proceed with the overall process. The bulk of each cycleâs computation does not need to be externalized (see Section 4.1 for further discussion).
A state is created to be used by its creator.
What matters is not what a generated token means to human readers, but how the LLM decodes it uses it in subsequent cycles. The LLM can use its own semantics independent of natural language interpretation (see Section 4.2).
4 Two Misconceptions Behind the Illusion of Explanation
To examine the nature of the relation between a sequence of states and the computation that produced it, it is instructive to take a step back and consider recursive iterative processes where $\mathcal{M}$ is not an LLM but a simpler function.
As an example, consider a process that computes the Catalan numbers, a sequence of natural numbers occurring in various counting problems. The Nth Catalan number is defined recursively by the formula $C_{n}=\sum_{i=0}^{n-1}C_{i}C_{n-1-i}$ with base case $C_{0}=1$ . The recursive definition can be translated into an iterative one, in which a pure function takes a prefix of the sequence and computes the next value according to the recursive definition. When each number is computed, it is added to the state that becomes the input of the next function call. The function $\mathcal{M}$ takes as input a target index $N$ and a prefix of $N$ values (the first $N$ Catalan numbers), and returns the $(N+1)$ -th Catalan number. The computation state in our case is the sequence of $N$ already computed numbers. Each computed value is concatenated to the input sequence; the resulting sequence becomes the state for the next call. For example, computing the 6th Catalan number generates the sequence of states:
| | $\displaystyle S_{0}$ | $\displaystyle=\text{6th?}$ | |
| --- | --- | --- | --- |
Note that while the numbers 1,1,2,5 and 14 are all needed for computing 42, and were used in the process of computing 42, we intuitively do not consider them as âan explanation for how 42 was computedâ. Indeed it actually seems odd for us to treat them as an explanation. But why would it be different for the intermediate steps of an LLM? This form of alienation is already instructive. But letâs now consider two concrete implications.
4.1 The misconception of completeness
It should be clear to any reader that while the intermediate sequence $1,1,2,5,14$ is crucial for computing $42$ , this sequence results from the computation and allows it to continue, but it is not the computation itself. The sequence of values does not equal the computation that took place for creating them. Additionally, this sequence does not determine a computation: we cannot infer the computation from the sequence (we can guess but cannot be sure), and it is clear that the computation steps that were involved in computing each number are not reflected in the output sequence. We see results of intermediate computations, not the algorithm that is used for producing them.
We can also imagine a function $\mathcal{M}^{\prime}$ that is only called 3 times and not 6, with the intermediate values 1,2,5. This function computes two following numbers at a time, and only outputs the second one. In each invocation, it recomputes the missing numbers, and continues the sequence. Even if we do choose to treat the steps as an explanation, there is no guarantee that this explanation is complete.
The SoT functions as scaffolding that propels the process forward; a misconception arises when we mistake this scaffolding for the building.
4.2 The misconception of shared meaning
Even if we accept that the state shows only partial results, we might assume the LLM interprets them as humans do. However, this assumption is also mistaken.
Consider an alternative function for computing the N-th Catalan number, whose final state is not $1,1,2,5,14,42$ but rather $11,11,12,15,24,52$ . Such a function could be just as effective: at each step, $\mathcal{M}$ will subtract 10 from each of its inputs, perform the computation, add 10 to the result, and return it as output. At the final (N-th) stage, it will subtract 10 from the last item and report it as the answer.
This simple additive transformation demonstrates that the reasoning tokens in the SoTâthe very symbols we read and interpretâmay function in a way entirely different from what human readers naturally ascribe to them. Even elementary encoding schemes can render the surface meaning of text opaque to human interpretation while remaining fully usable for the LLM.
Beyond numerical encoding.
This encoding arbitrariness extends beyond numerical computation to the full range of SoT. When an LLM writes âI need to reconsider this approachâ, the phrase might function not as genuine meta-cognitive reflection, but as an encoding for something else. Given empirical findings on how models can embed functionally relevant information in ways that remain opaque to human readers (cloud2025subliminallearninglanguagemodels), it is likely that real-world encodings are far more intricate than a simple additive shift. This view, together with Section 4.1, may also give intuition to the phenomenon where LLMs succeed at evading mentioning relevant topics in their reasoning text (emmons2025chainthoughtnecessarylanguage; li2025llms).
5 The Ontological Divergence: Text vs. State
We argue that the divergence between form and function in reasoning LLMsâthat the same tokens can be interpreted (by humans) as English text while at the same time also interpreted (by the LLM) as an encoding of a computation state for an iterative functionâis a novel, and hence foreign phenomenon in human experience. While we are accustomed to symbols that can be interpreted in multiple ways (the word âappleâ can refer to either a fruit or a company, the same sequence of bytes can be interpreted as either a floating point number or a memory address), the State over Tokens case is fundamentally different.
The reasoning tokens go beyond admitting multiple interpretations; they inhabit distinct ontological categories. To a human reader, the sequence is a text, parsed according to linguistic conventions to extract semantic meaning. To the model, however, the same tokens serve as a computational substrate, accumulated one token at a time to mechanically drive the next step of a process. These are not different perspectives on the same underlying content, but completely different kinds of entitiesâa communicative medium and a functional stateâthat happen to share the same embodiment. We believe this is how the tokens can function as a computational substrate without becoming about the computation in a way that human readers can discern by reading. The LLM uses a sequence of tokens that can be interpreted by humans according to one semantic system, while at the same time it is interpreted by the LLM using an alternative semantic system.
6 Questions Raised by State over Tokens
6.1 What SoT means for interpretability
Rather than solving the challenges that existing interpretability work addressesâsuch as understanding model internals, tracing how features emerge, or identifying mechanisms (alain2016understanding; li2023emergent; bereska2024mechanistic) âthe SoT view surfaces a distinct, new challenge: investigating how LLMs construct and use computational state through tokens. We call for recognizing the encoded state as a first-class object of study. This approach neither reads the reasoning tokens as ordinary English text nor dismisses them as an inadequate explanation, but rather acknowledges that understanding how LLMs use state requires work to decode the structure and function of the token sequence. The new overarching question becomes âHow is the computation state encoded in the tokens being maintained and used by the LLM?â This shift in perspective opens new questions about the structure and behavior of the encoded state itself:
- How do LLMs decide what information to externalize at each cycle?
- What information is actually encoded in the state at different points in the sequence?
- Do they use consistent encodings across the solution of a problem and across different problems?
- How does information propagate through the sequence?
Understanding these properties requires decoding the relationship between internal computation and the externalized token-based state.
Recent interpretability work provides a starting pointâfor instance, studying which parts of the sequence are most critical for the final answer (bogdan2025thoughtanchorsllmreasoning) or analyzing model componentsâ activity during different cycles (dutta2024thinkstepbystepmechanisticunderstanding; chen2025doeschainthoughtthink; zhao2025verifyingchainofthoughtreasoningcomputational). However, the grand challenge of understanding how LLMs manage state through tokens remains largely open.
6.2 Is language special for SoT?
The SoT framework also raises the question of the arbitrariness of the medium for discrete, token-based computation. We previously argued that SoT can, in principle, encode arbitrary computation regardless of its surface semantics. But to what extent is this actually the case? Are all media equally expressive, or is using natural language sequences particularly well-suited for encoding the kinds of computations that LLMs perform?
One can hypothesize that the massive pretraining stage on natural language data induces an inductive bias to reason (venhoff2025basemodelsknowreason) in ways that are in accordance with its semantics, which makes the reasoning text read as a plausible explanation. Under this view, elaborate encoding schemes may either contradict the training distributionâwhere SoT updates often gradually lead to the correct continuationâor require additional computation that is disfavored given the pretraining data. If this hypothesis is correct, LLMs possess at least an inductive bias to represent state in a manner that reflects their internal computation.
In this context, recent works have explored alternative media for state representation, ranging from simple vectors (hao2025traininglargelanguagemodels; butt2025softtokenshardtruths; hwang2025latentreasoningsentenceembedding) to structured formats (domingos2025tensorlogiclanguageai). However, the question of whether natural language offers unique advantages over these alternatives remains open.
6.3 Can SoT ever be a faithful explanation?
Finally, the SoT view sharpens the question of whether the faithfulness of reasoning tokens can be improved. If we aim to make the text a faithful explanation, there will be a fundamental tension: the reasoning tokens will have to serve two distinct masters. Functionally, as a state, the tokens are optimized to encode the information necessary to drive the next computation cycles. Interpretively, as an explanation, we want them to also describe the computation itselfâwhether they describe the computation that has already occurred in past cycles, or the computation that will occur in future cycles.
This dual requirement creates a bottleneck. The tokens are the only medium available for both the computation and its description. Requiring an LLM to explain its own computation while simultaneously performing it amounts to a form of metacognitive capability; the LLM must reason about its own reasoning process within the same sequence that sustains it. For the explanation to be faithful, the information the model encodes to solve the problem (the state) must be identical to the information a human extracts by reading the text (the explanation). However, an optimal computational state might require encoding information that is redundant, non-linear, or semantically opaque to a human reader. Conversely, constraining the state to be a coherent, linear English narrative might strip it of the information needed for the reasoning process.
Thus, the challenge is not merely whether LLMs can be trained to produce plausible explanations, but whether the medium of natural language tokens has the capacity to simultaneously function as an efficient computational substrate and a transparent descriptive record. This forces us to confront a fundamental question: can the same sequence of symbols simultaneously carry the full weight of a machineâs computation and transparently reveal the logic of that process to humans?
The question extends also to viewing reasoning texts as rationalizations: texts that consist of a sequence of steps leading to the answer, which can be used by a human reader to verify it. To what extent can a sequence be used concurrently both as carrying state and as providing an effective rationalization?
7 Conclusion
The reasoning tokens generated by LLMs before an answer are best understood not as a narrative of thought, but as an externalized computational state, a role captured by the term State over Tokens (SoT).
Adopting this perspective resolves the persistent misconceptions that underlie the illusion of explanation. The Misconception of Completeness is explained by recognizing that SoT externalizes only what is functionally necessary for the next cycle, not a full account of the computation that occurred. The Misconception of Shared Meaning is exposed by the insight that the textâs human-readable meaning can be incidental to its computational function, which may rely on arbitrary, model-specific encodings. This semantic mismatchâwhere the token sequence functions as computational substrate while appearing as natural languageârepresents an unprecedented ontological divergence: two fundamentally incompatible modes of interpretation coexisting within the same textual artifact.
While much research has focused on what this text is not, the SoT view attempts to explain what it is. This opens concrete research directionsâunderstanding how LLMs construct and use state through tokens, investigating whether natural language possesses unique advantages as a computational medium, and exploring whether SoT can ever faithfully explain the underlying computation. Recognizing the reasoning tokens as a computational state provides a clear basis for calibrating trust and for investigating the new questions this view opens.