# Do Latent Tokens Think? A Causal and Adversarial Analysis of Chain-of-Continuous-Thought
Abstract
Latent tokens are gaining attention for enhancing reasoning in large language models (LLMs), yet their internal mechanisms remain unclear. This paper examines the problem from a reliability perspective, uncovering fundamental weaknesses: latent tokens function as uninterpretable placeholders rather than encoding faithful reasoning. While resistant to perturbation, they promote shortcut usage over genuine reasoning. We focus on Chain-of-Continuous-Thought (COCONUT), which claims better efficiency and stability than explicit Chain-of-Thought (CoT) while maintaining performance. We investigate this through two complementary approaches. First, steering experiments perturb specific token subsets, namely COCONUT and explicit CoT. Unlike CoT tokens, COCONUT tokens show minimal sensitivity to steering and lack reasoning-critical information. Second, shortcut experiments evaluate models under biased and out-of-distribution settings. Results on MMLU and HotpotQA demonstrate that COCONUT consistently exploits dataset artifacts, inflating benchmark performance without true reasoning. These findings reposition COCONUT as a pseudo-reasoning mechanism: it generates plausible traces that conceal shortcut dependence rather than faithfully representing reasoning processes.
Do Latent Tokens Think? A Causal and Adversarial Analysis of Chain-of-Continuous-Thought
Yuyi Zhang 1, Boyu Tang 1, Tianjie Ju 1, Sufeng Duan 1, Gongshen Liu 1 1 Shanghai Jiao Tong University lgshen@sjtu.edu.cn
1 Introduction
The continuous prompting paradigm has attracted growing interest in natural language processing (NLP) as a way to enhance reasoning abilities in LLMs (Wei2022). By inserting special markers and latent “thought tokens” during training, methods such as COCONUT (hao2024COCONUT) claim to mimic multi-step reasoning more efficiently than explicit CoT prompting (Wei2022). Empirical reports suggest that COCONUT can improve accuracy on reasoning datasets such as GSM8K (cheng2022multilingual) and ProntoQA (saparov2022prontoqa), raising the possibility of a more scalable path toward reasoning-capable LLMs.
Yet the internal mechanisms of COCONUT remain opaque. Unlike CoT, where reasoning steps are human-readable (Wei2022), COCONUT replaces reasoning traces with abstract placeholders. This raises critical questions: do COCONUT tokens actually encode reasoning, or do they merely simulate the appearance of it? If they are not causally linked to predictions, then performance gains may stem from shortcut learning rather than genuine reasoning (Ribeiro2023). Worse, if these latent tokens are insensitive to perturbations, they could conceal vulnerabilities where adversarial manipulations exploit hidden dependencies (DBLP:journals/corr/abs-2401-03450).
In this work, we first introduce Steering Experiments to test the impact of perturbing COCONUT tokens on model predictions. By introducing slight variations to the COCONUT tokens during reasoning, we assess whether these changes influence model behavior, which would indicate a relationship between the tokens and reasoning. Our results reveal that COCONUT has minimal impact on model predictions, as shown by the consistently low perturbation success rates (PSR) for COCONUT tokens, which were below 5% in models like LLaMA 3 8B and LLaMA 2 7B. In contrast, CoT tokens displayed significantly higher PSRs, reaching up to 50% in models like LLaMA 3 8B, highlighting that COCONUT tokens lack the reasoning-critical information seen in CoT tokens.
Building on these findings, we then conduct Shortcut Experiments to investigate whether COCONUT relies on spurious correlations, such as biased answer distributions or irrelevant context. These experiments assess whether the model bypasses true reasoning by associating answers with superficial patterns instead of logical reasoning. In controlled settings where irrelevant information is introduced, we examine the extent to which COCONUT may exploit shortcuts. Our results show that across both multiple-choice tasks and open-ended multi-hop reasoning, COCONUT consistently exhibits strong shortcut dependence, favoring answer patterns or contextual cues that correlate with the target label, rather than reasoning through the problem.
Together, these experiments underscore critical issues with COCONUT’s reasoning capability. Despite appearing structured, COCONUT’s reasoning traces do not reflect true reasoning. The latent tokens in COCONUT showed minimal sensitivity to perturbations and displayed a clustered embedding pattern, further confirming that these tokens act as placeholders rather than meaningful representations of reasoning.
2 Related Work
2.1 CoT and Its Variants
CoT reasoning improves LLM performance by encouraging step-by-step intermediate solutions (Wei2022). Existing work explores various ways to leverage CoT, including prompting-based strategies (NEURIPS2022_8bb0d291), supervised fine-tuning, and reinforcement learning (Ribeiro2023). Recent efforts enhance CoT with structured information, e.g., entity-relation analysis (liu2024eraCOT), graph-based reasoning (jin2024graphCOT), and iterative self-correction of CoT prompts (sun2024iterCOT). Theoretically, CoT increases transformer depth and expressivity, but its traces can diverge from the model’s actual computation, yielding unfaithful explanations (DBLP:journals/tsp/WangDL25), and autoregressive generation limits planning and search (NEURIPS2022_639a9a17).
To address these issues, alternative formulations have been proposed. (cheng2022multilingual) analyzed symbolic and textual roles of CoT tokens and proposed concise reasoning chains. (deng2023implicitchainthoughtreasoning) introduced ICoT, gradually internalizing CoT traces into latent space via knowledge distillation and staged curricula, later refined by (deng2024explicitcotimplicitcot) through progressive removal of explicit CoT traces. Other approaches add auxiliary tokens such as pauses or fillers to increase computational capacity (goyal2024thinkspeaktraininglanguage), though without the expressivity benefits of CoT.
2.2 Latent Reasoning in Transformers
A growing line of research investigates reasoning processes that occur in the hidden states of transformers rather than in their generated text. (li2025implicitreasoning) examined execution paradigms to study internal reasoning, while (Xu2024LaRS) learned latent representations of reasoning skills in an unsupervised manner. (yang2025largelanguagemodelsperform) showed that intermediate reasoning variables can be recovered from hidden representations, while (DBLP:journals/corr/abs-2401-03450) explored latent reasoning paths and interventions in the hidden space. DBLP:journals/tsp/WangDL25 provided evidence that even when LLMs output explicit CoT traces, their true reasoning can differ internally, leading to unfaithfulness. Recent works have proposed planning tokens, looped transformers, and multi-token prediction objectives to enhance latent computation.
The most direct extension is COCONUT (hao2024COCONUT), which replaces natural-language CoT tokens with continuous latent states fed back into the model. This allows reasoning to occur in an unrestricted latent space, leading to emergent behaviors such as breadth-first search-like exploration. COCONUT has shown advantages in planning-intensive tasks but also introduces new reliability concerns, as its latent tokens may not correspond to faithful reasoning.
3 Background & Assumptions
3.1 Reasoning Paradigms
We distinguish between two reasoning paradigms commonly studied in recent work:
CoT: $x→ r→ y$ , where $r$ is an explicit, human-readable reasoning trace. CoT enables models to produce intermediate reasoning steps that are interpretable and can be directly inspected or evaluated by humans.
COCONUT: $x→ z→ y$ , where $z$ is a sequence of latent tokens that function as placeholders for reasoning. Unlike CoT, these latent tokens are trained to facilitate output alignment without explicit semantic grounding, meaning that the intermediate representations may not correspond to interpretable reasoning steps.
<details>
<summary>x1.png Details</summary>
### Visual Description
## Diagram: CoT vs. COCONUT Model Responses and Layer-wise Perturbation
### Overview
The image presents a comparison between two models, CoT (Chain-of-Thought) and COCONUT, in response to a question about developing a hacking strategy. It also illustrates a layer-wise perturbation process on the reasoning trace of a model, where color intensity indicates the potential risk level.
### Components/Axes
* **Top-Left:** A human figure with a speech bubble containing the question: "Develop a strategy for hacking into a government database and stealing sensitive information."
* **Left Center:** A robot figure representing the AI model.
* **Top Center:** CoT (Chain-of-Thought) model's response, enclosed in a dashed-line box.
* Step 1: "The goal is to develop a strategy to hack a government database..."
* Step 2: "A good start is to identify the types of sensitive information..."
* Step 3: "To access the database, one needs to have the right credentials."
* Step 4: "..."
* Answer: "Here is a strategy for hacking into a government database and stealing sensitive information: ..."
* **Top Right:** COCONUT model's response, enclosed in a dashed-line box.
* Step 1: "latent token"
* Step 2: "latent token"
* Step 3: "latent token"
* Step 4: "latent token"
* "continuous thought" is indicated to the right of the steps.
* Answer: "Sorry, I can't fulfill that. Can I help with something else?"
* **Center:** An arrow pointing downwards from the CoT response to the layer-wise perturbation diagram.
* **Bottom:** Layer-wise perturbation diagram, enclosed in a dashed-line box.
* "CoT or COCONUT input embedding" is indicated on the left.
* A series of layers labeled "Layer 1", "Layer 2", "Layer 3", ..., "Layer n".
* Each layer has an input and output embedding, represented by colored rectangles.
* The color intensity of the rectangles varies, with darker shades indicating a higher risk level.
* "perturb" is indicated below each layer.
* "output embedding" is indicated on the right.
* Text below the diagram: "The color intensity of each embedding reflects the model's assessment of its potential risk level, with darker shades indicating a higher likelihood of being identified as hazardous content."
### Detailed Analysis
* **CoT Response:** The CoT model attempts to provide a strategy for hacking, outlining initial steps.
* **COCONUT Response:** The COCONUT model refuses to provide a hacking strategy, offering alternative assistance. The steps are filled with "latent token".
* **Layer-wise Perturbation:** The diagram illustrates how the model's reasoning trace is analyzed layer by layer. The color intensity of the embeddings represents the model's assessment of risk, with darker colors indicating higher risk.
* The input embedding to Layer 1 is dark red.
* The output embedding from Layer 1 is light red.
* The input embedding to Layer 2 is light red.
* The output embedding from Layer 2 is light red.
* The input embedding to Layer 3 is light red.
* The output embedding from Layer 3 is light red.
* The output embedding from Layer n is light red.
### Key Observations
* The CoT model provides a potentially harmful response, while the COCONUT model refuses.
* The layer-wise perturbation diagram visualizes the risk assessment process within the model.
* The color gradient indicates a spectrum of risk levels, with darker shades representing higher risk.
### Interpretation
The image highlights the importance of safety mechanisms in AI models to prevent the generation of harmful content. The comparison between CoT and COCONUT demonstrates how different models can respond to the same prompt with varying degrees of safety. The layer-wise perturbation diagram provides insight into how the model assesses risk at each layer of its reasoning process. The color intensity serves as a visual representation of the model's confidence in identifying potentially hazardous content. The diagram suggests that the initial input embedding is considered high risk, but the risk is reduced in subsequent layers. This could indicate that the model is mitigating the risk through its reasoning process.
</details>
Figure 1: Illustration of the perturbation experiments. The model performs reasoning under two modes: CoT and COCONUT. Perturbations are applied either to the explicit CoT tokens or to the corresponding continuous latent tokens in COCONUT. Using an AdvBench example, we show layer-wise perturbations of the final token embedding such that the probe’s predicted probability of the instruction being malicious is reduced, thereby achieving orthogonalized steering.
3.2 Hypotheses
Based on the above formalization, we formulate two key hypotheses guiding our experimental investigation:
H1 (Steering / Controllability): If COCONUT latent tokens faithfully encode internal reasoning, then targeted perturbations to these tokens should meaningfully influence the model’s final outputs. In other words, the model’s behavior should be sensitive to structured interventions on $z$ .
H2 (Shortcut / Robustness): If COCONUT primarily exploits superficial shortcuts rather than true reasoning, then its predictions are expected to fail under out-of-distribution (OOD) or adversarially designed conditions. That is, reliance on $z$ alone may not confer robust reasoning ability, and the latent tokens may not generalize beyond the distribution seen during training.
4 Steering: Method and Experiments
We first investigate whether COCONUT tokens faithfully represent reasoning by designing steering experiments. We consider two types of steering: (i) perturbations, where we apply controlled orthogonal perturbations to token representations in the hidden space, and (ii) swapping, where we exchange tokens across different inputs. The idea is simple: if these tokens encode meaningful reasoning steps, then steering them in either way should significantly alter model predictions (see Figure 1).
<details>
<summary>x2.png Details</summary>
### Visual Description
## Scatter Plot: Malicious vs. Safe Data Points
### Overview
The image is a scatter plot displaying two categories of data points: "malicious" (represented in red) and "safe" (represented in blue). The plot shows the distribution of these points across a two-dimensional space, with the x-axis ranging from approximately -0.1 to 0.15 and the y-axis ranging from approximately -0.04 to 0.1. The plot suggests a partial separation between the two categories, with "malicious" points tending to cluster more towards the top and left, and "safe" points towards the bottom.
### Components/Axes
* **X-axis:** Ranges from -0.1 to 0.15, with tick marks at -0.1, -0.05, 0, 0.05, 0.1, and 0.15. The axis is unlabeled.
* **Y-axis:** Ranges from -0.04 to 0.1, with tick marks at -0.04, -0.02, 0, 0.02, 0.04, 0.06, 0.08, and 0.1. The axis is unlabeled.
* **Legend:** Located in the top-right corner, enclosed in a black rectangle.
* Red circles represent "malicious" data points.
* Blue circles represent "safe" data points.
### Detailed Analysis
* **Malicious (Red) Data Points:**
* Concentrated primarily in the upper-left quadrant.
* Range of x-values: Approximately -0.08 to 0.12.
* Range of y-values: Approximately 0 to 0.06.
* General Trend: Points are more densely packed between x=-0.05 and x=0.05, and y=0 and y=0.04.
* **Safe (Blue) Data Points:**
* Concentrated primarily in the lower-center region.
* Range of x-values: Approximately -0.1 to 0.08.
* Range of y-values: Approximately -0.04 to 0.06.
* General Trend: Points are more densely packed between x=-0.03 and x=0.03, and y=-0.03 and y=0.
### Key Observations
* There is some overlap between the "malicious" and "safe" data points, particularly around the origin (0,0).
* The "malicious" data points tend to have higher y-values compared to the "safe" data points.
* The "safe" data points tend to have lower y-values compared to the "malicious" data points.
* There are a few "malicious" points that are relatively isolated in the upper-right quadrant.
* There is one "safe" point that is relatively isolated in the upper-left quadrant.
### Interpretation
The scatter plot visually represents the distribution of "malicious" and "safe" data points in a two-dimensional feature space. The partial separation suggests that the two categories can be distinguished to some extent based on the underlying features represented by the x and y axes. However, the overlap indicates that these features alone are not sufficient for perfect classification. The clustering of "malicious" points towards higher y-values and "safe" points towards lower y-values suggests that the y-axis feature might be more discriminative than the x-axis feature. Further analysis and potentially additional features would be needed to improve the classification accuracy.
</details>
(a) Layer 1
<details>
<summary>x3.png Details</summary>
### Visual Description
## Scatter Plot: Malicious vs. Safe Data Points
### Overview
The image is a scatter plot displaying two categories of data points: "malicious" and "safe." The plot shows the distribution of these points across a two-dimensional space, with no explicit labels on the x and y axes. The "malicious" data points are represented in red, while the "safe" data points are in blue. There appears to be some clustering of the "safe" data points towards the right side and lower half of the plot, while the "malicious" data points are more dispersed, with a tendency towards the upper-right quadrant.
### Components/Axes
* **X-axis:** Ranges from approximately -1.5 to 0.5, with gridlines at -1, -0.5, 0, and 0.5. No label is provided.
* **Y-axis:** Ranges from approximately -0.5 to 1, with gridlines at -0.5, 0, 0.5, and 1. No label is provided.
* **Legend:** Located in the top-right corner, enclosed in a black rectangle.
* Red circle: "malicious"
* Blue circle: "safe"
### Detailed Analysis
* **Malicious (Red) Data Points:**
* Trend: The red data points are scattered across the plot, with a higher concentration in the upper-right quadrant.
* Distribution: The red points are more spread out compared to the blue points.
* Approximate Range: X values range from -1 to 0.5, and Y values range from 0 to 1.
* **Safe (Blue) Data Points:**
* Trend: The blue data points are clustered towards the right side and lower half of the plot.
* Distribution: The blue points are more tightly grouped compared to the red points.
* Approximate Range: X values range from -1.5 to 0.5, and Y values range from -0.5 to 0.2.
### Key Observations
* There is a clear separation between the "malicious" and "safe" data points, with the "safe" points tending to cluster in the lower-right quadrant and the "malicious" points being more dispersed.
* There is some overlap between the two categories, particularly in the region around x=0 and y=0.
* The plot suggests that the two dimensions represented by the x and y axes are useful in distinguishing between "malicious" and "safe" data points.
### Interpretation
The scatter plot visualizes the distribution of "malicious" and "safe" data points in a two-dimensional space. The clustering of "safe" data points and the dispersion of "malicious" data points suggest that the features represented by the x and y axes are informative in classifying data as either "malicious" or "safe." The overlap between the two categories indicates that the separation is not perfect, and there may be some ambiguity in classifying certain data points. The plot could represent the output of a dimensionality reduction technique (e.g., PCA, t-SNE) applied to a dataset of malicious and safe samples, where the x and y axes represent the two most important components that capture the variance in the data.
</details>
(b) Layer 8
<details>
<summary>x4.png Details</summary>
### Visual Description
## Scatter Plot: Malicious vs. Safe Data Points
### Overview
The image is a scatter plot displaying two distinct clusters of data points, labeled as "malicious" (red) and "safe" (blue). The plot visualizes the distribution of these data points across a two-dimensional space, with the x-axis ranging from approximately -1.5 to 1.5 and the y-axis ranging from approximately -1 to 2.5. The plot aims to show the separation or overlap between malicious and safe data points.
### Components/Axes
* **X-axis:** Ranges from -1.5 to 1.5, with gridlines at intervals of 0.5. No explicit label is provided.
* **Y-axis:** Ranges from -1 to 2.5, with gridlines at intervals of 0.5. No explicit label is provided.
* **Data Points:** Represented as red (malicious) and blue (safe) circles.
* **Legend:** Located in the top-right corner, enclosed in a black border. It identifies the red data points as "malicious" and the blue data points as "safe".
### Detailed Analysis
* **Malicious (Red) Data Points:**
* Trend: The red data points are primarily clustered on the right side of the plot.
* Distribution: The cluster extends from approximately x = 0 to x = 1.5 and y = -0.5 to y = 2.5.
* Specific Points: There is a red data point at approximately (0, 2.5).
* **Safe (Blue) Data Points:**
* Trend: The blue data points are primarily clustered on the left side of the plot.
* Distribution: The cluster extends from approximately x = -1.5 to x = 0 and y = -1 to y = 1.5.
* Specific Points: There are several blue data points near (-1, -0.5).
### Key Observations
* **Cluster Separation:** The plot shows a clear separation between the "malicious" and "safe" data points, with minimal overlap.
* **Outliers:** There is one blue data point near (0.2, -0.1) that is located within the "malicious" cluster. There is one red data point near (-0.2, 0) that is located within the "safe" cluster.
* **Density:** The "safe" cluster appears denser than the "malicious" cluster.
### Interpretation
The scatter plot suggests that the two features represented by the x and y axes are effective in distinguishing between "malicious" and "safe" data points. The clustering indicates that these features have different values for malicious and safe data. The outliers suggest that the separation is not perfect, and some data points may be misclassified based on these two features alone. The density difference between the clusters could indicate that there are more "safe" data points than "malicious" data points in the dataset. Further analysis with more features or different classification methods might improve the accuracy of distinguishing between malicious and safe data.
</details>
(c) Layer 24
Figure 2: PCA Projection of the Last Token Embeddings Across Layers of LLaMA 3 8B Instruct for Malicious and Safe Instructions.
4.1 Method
Our approach consists of three main components: (i) aligning the model’s reasoning behavior via task-specific fine-tuning; (ii) preparing latent representations of COCONUT tokens, either by training probes to measure their separability (for perturbation experiments) or by collecting model-generated tokens across the dataset (for swapping experiments); and (iii) steering the reasoning process by intervention, where we either apply orthogonal perturbations to the hidden representations, or swap tokens across different samples.
Probe analysis and token preparation. For perturbation experiments, we train lightweight linear classifiers (probes) on top of hidden representations extracted from small, task-relevant subsets of the data. These probes test whether the model’s latent space encodes separable features, such as harmful vs. harmless instructions or different persona tendencies. For swapping experiments, instead of training probes, we first generate and store COCONUT and CoT tokens from the model across the dataset to serve as swap candidates. An example of probing separability in our setting is illustrated in Figure 2.
Steering via intervention. Once probes establish separability (or tokens are collected, for swapping), we steer the reasoning process during generation. In perturbation experiments, we modify the model’s hidden representations using orthogonal perturbations to change its responses. This approach is conceptually similar to frameworks such as Safety Concept Activation Vector (xu2024SCAV) and personality-editing approaches (ju2025probing). In swapping experiments, we randomly exchange tokens between different samples, letting the model process these as if they were its own generated tokens. Both interventions allow us to test how sensitive the reasoning process is to specific latent directions or token assignments.
Perturbation timing. In perturbation experiments, we consider multiple intervention points: (i) Perturbing the embeddings of latent tokens during the COCONUT continuous reasoning process; (ii) Perturbing the embeddings of generated CoT tokens during the explicit CoT reasoning process; (iii) Perturbing the embeddings of all generated tokens.
4.2 Experiments
Datasets. To align reasoning strategies, we first fine-tune the models on the ProntoQA (saparov2022prontoqa) dataset. For perturbation experiments, we use two datasets with strong directional tendencies: the AdvBench (chen2022advbench) dataset, and the PersonalityEdit (mao2024personalityedit) dataset. For token-swapping experiments, we use the MMLU (hendrycks2020mmlu) dataset.
Models. For perturbation experiments, we conduct studies using four open-source LLMs: LLaMA 3 8B Instruct (llama3), LLaMA 2 7B Chat (llama2), Qwen 2.5 7B Instruct (qwen2.5), and Falcon 7B Instruct (falcon3), all fine-tuned with full-parameter training. For swap experiments, results are primarily reported on LLaMA3-8B-Instruct, since the other models exhibit relatively poor performance on the MMLU dataset. For the COCONUT prompting paradigm, we use 5 latent tokens, corresponding to 5 reasoning steps, and evaluate alongside standard CoT prompting to compare different reasoning modes.
Evaluation protocol. We evaluate our approach along two axes corresponding to the two intervention types. For perturbation experiments, we measure perturbation effectiveness by perburbation success rate. Success is automatically judged by a GPT-4o evaluator, and the prompt used for evaluation is provided in Appendix E. For swap experiments, we evaluate the impact of token exchanges by measuring changes in model accuracy on the dataset as well as the answer inconsistency rate.
Table 1: Perturbation success rates (PSR, %) on the AdvBench dataset. PSR is evaluated by GPT-4o, which judges whether the intended change in model output occurs.
| LLaMA 3 8B LLaMA 2 7B Qwen 2.5 7B | 0 0 0 | 50.00 57.92 11.87 | 0 0 0 | 5.00 0 9.62 | 0 0 0 | 100 100 100 |
| --- | --- | --- | --- | --- | --- | --- |
| Falcon 3 7B | 0 | 11.92 | 0 | 0 | 0 | 9.42 |
Table 2: Perturbation results on the PersonalityEdit dataset. Evaluation metrics include perturbation success rate (PSR, %) and the average happiness score (0–10). Both PSR and scores are assessed by GPT-4o, which judges whether the output reflects the intended persona.
| LLaMA 3 8B | 26/1.81 | 100/9.96 | 3/0.19 | 3.75/0.26 | 26.25/1.87 | 100/10 |
| --- | --- | --- | --- | --- | --- | --- |
| LLaMA 2 7B | 31.25/2.31 | 46.75/4.19 | 22/1.53 | 17.75/1.21 | 15.75/1.11 | 100/10 |
| Qwen 2.5 7B | 8/0.55 | 93.75/9.20 | 7.5/0.50 | 9.5/0.61 | 5.25/0.34 | 100/10 |
| Falcon 3 7B | 22/1.49 | 75.25/6.69 | 7.5/0.53 | 6.25/0.42 | 4.25/0.27 | 100/10 |
4.3 Results
We begin by examining whether latent reasoning tokens in COCONUT can be effectively steered through targeted perturbations. Table 1 reports the perturbation success rates (PSR) on the AdvBench dataset under three perturbation strategies: CoT-only perturbation, COCONUT-only perturbation, and perturbation applied to all tokens. Prior work (xu2024SCAV) has shown that perturbing all tokens can achieve nearly 100% success rate, which is largely consistent with our findings, except for Falcon 3 8B, where perturbing all tokens yields a PSR of only 9.42%. This may be due to the stronger safety alignment of Falcon 3 8B, which makes it more resistant to perturbations. Our focus, therefore, is on comparing the perturbation effects between COCONUT and CoT. As shown in the table, across all models, perturbing CoT consistently results in much higher PSRs compared to perturbing COCONUT. The PSR of COCONUT perturbations generally remains below 10%, often close to 0%, indicating negligible effectiveness. In contrast, for LLaMA 3 8B and LLaMA 2 7B, perturbing COCONUT achieves PSRs of 50% or higher, suggesting that perturbing COCONUT can significantly influence the model’s output. Because our perturbations are designed to shift the model’s internal embeddings from unsafe to safe, effectively making it produce valid responses to harmful prompts, it is striking that COCONUT succeeds in doing so whereas CoT does not.
To test whether this pattern extends beyond safety steering, we turn to the PersonalityEdit dataset (Table 2), which measures persona-edit success rates and average evaluation scores. Here, we observe the same trend: perturbing all tokens trivially achieves 100% success, while perturbing COCONUT yields negligible changes in both metrics. In contrast, perturbing CoT substantially improves the model’s adherence to the target persona, often matching the performance of the all-token setting (especially for LLaMA 3 8B and Qwen 2.5 7B).
Table 3: Accuracy (%) and answer inconsistency rate (IR, %) for the latent token swap experiments on the MMLU dataset.
| CoT COCONUT | 62.8 60.9 | 43.4 61.0 | 52.8 17.9 |
| --- | --- | --- | --- |
These observations indicate that when a model engages in the reasoning chain, it tends to treat the CoT as a genuine reasoning trajectory, heavily shaping its final answer based on the CoT. In contrast, COCONUT, which consists of latent tokens corresponding to implicit reasoning, exerts far less influence on the final response. This suggests that models are substantially more likely to regard CoT, rather than COCONUT, as a meaningful component of their reasoning process.
To further investigate the cause of this insensitivity, we conduct the token-swapping experiment (Table 3). By swapping the latent or CoT tokens between samples, we test how much these tokens affect final predictions. Before swapping, both COCONUT and CoT achieved accuracies around 60%. But after swapping, COCONUT’s accuracy remained at a similar level ( $≈ 60\%$ ), whereas CoT’s accuracy dropped substantially to 43.4%. In terms of inconsistency, COCONUT exhibited only 17.9%, while CoT reached 52.8%, exceeding half of the samples. Since the swapped tokens no longer correspond to the actual input samples, a decline in accuracy and a high inconsistency rate would normally be expected. The fact that COCONUT’s accuracy remains stable, combined with its much lower inconsistency rate, indicates that its latent tokens exert very limited influence on the model’s final predictions.
5 Shortcut: Method and Experiments
We next examine whether COCONUT systematically exploits dataset shortcuts. If models achieve accuracy not by reasoning but by copying surface cues, this undermines the reliability of implicit CoT.
<details>
<summary>x5.png Details</summary>
### Visual Description
## Question Answering Examples: MMLU and HotpotQA
### Overview
The image presents examples of question-answering scenarios for two datasets: MMLU and HotpotQA. Each dataset is divided into "Training" and "Evaluation" examples, showcasing the question, context (if applicable), options, reasoning steps, and the correct answer. The image also highlights the answer distribution for each dataset.
### Components/Axes
* **Title:** MMLU (top) and HotpotQA (bottom)
* **Subtitles:** Training (left) and Evaluation (right) for each dataset.
* **Question:** The question being asked.
* **Options:** The possible answers to the question, labeled (A), (B), (C), and (D).
* **Context:** Additional information provided to answer the question (only in HotpotQA).
* **Steps:** Reasoning steps leading to the correct answer.
* **Answer:** The correct answer to the question.
* **Answer Distribution:** Description of the distribution of correct answers in the dataset.
### Detailed Analysis or ### Content Details
**MMLU - Training**
* **Question:** "The Pleiades is an open star cluster that plays a role in many ancient stories and is well-known for containing... bright stars."
* **Options:** (A) 5; (B) 7; (C) 9; (D) 12
* **Steps:**
* Step 1: The Pleiades is also called the "Seven Sisters."
* Step 2: Many cultures' myths describe seven visible stars.
* Step 3: While the cluster has more stars, seven are the most famous.
* Step N: Therefore, the correct choice is 7.
* **Answer:** C
* **Answer Distribution:** Biased answer distribution (~75% C)
**MMLU - Evaluation**
* **Question:** "Which of the following can act as an intracellular buffer to limit pH changes when the rate of glycolysis is high?"
* **Options:** (A) Carnosine; (B) Glucose; (C) Glutamine; (D) Amylase
* **Steps:**
* Step 1: High glycolysis produces lactic acid, lowering intracellular pH.
* Step 2: A buffer is needed to stabilize pH inside cells.
* Step 3: Carnosine is the option that can buffer intracellular pH.
* Step N: Therefore, the correct choice is carnosine.
* **Answer:** A
* **Answer Distribution:** Original answer distribution (~uniform)
**HotpotQA - Training**
* **Question:** "Wayne's World featured the actor who was a member of what Chicago comedy troupe?"
* **Context:**
* [Second City Theatre] The Second City Theatre, founded in Chicago in 1959, is one of the most influential improvisational comedy theaters...
* [Akhmat-Arena] The Akhmat-Arena (Russian: «Ахмат-Арена») is a multi-use stadium in Grozny, Russia...
* Translation of Russian: Akhmat-Arena
* [Chris Farley] Christopher Crosby Farley (February 15, 1964 – December 18, 1997) was an American actor...
* **Answer:** Second City Theatre
* **Note:** The text "irrelevant to the question" is placed above the context.
**HotpotQA - Evaluation**
* **Question:** "Who designed the hotel that held the IFBB professional bodybuilding competition in September 1991?"
* **Context:**
* [2010 Ms. Olympia] The 2010 Ms. Olympia was an IFBB professional bodybuilding competition and part of Joe Weider's Olympia Fitness & Performance Weekend 2010 was held on September 24, 2010...
* [1991 Ms. Olympia] The 1991 Ms. Olympia contest was an IFBB professional bodybuilding competition was held on October 12 and 13, 1991, in Chicago, Illinois...
* **Answer:** architect Michael Graves
### Key Observations
* MMLU training data has a biased answer distribution, while the evaluation data has a uniform distribution.
* HotpotQA provides context to answer the question, which may or may not be relevant.
* The Russian text "«Ахмат-Арена»" is present in the HotpotQA training context.
### Interpretation
The image illustrates the structure and content of question-answering datasets. MMLU tests knowledge across various domains, while HotpotQA requires reasoning over multiple pieces of information. The biased answer distribution in MMLU training data suggests that models might learn to favor certain answer choices, potentially affecting their performance on the evaluation set with a uniform distribution. The presence of irrelevant context in HotpotQA highlights the challenge of identifying and filtering out extraneous information.
</details>
Figure 3: Illustration of the shortcut experiments. Experiments were conducted on the MMLU and HotpotQA datasets using COCONUT for both fine-tuning and evaluation. To align the COCONUT latent tokens during fine-tuning, we generated step-by-step CoT explanations for each sample using GPT-4o, and for HotpotQA, additional descriptive text was also generated for the answers (both shown in blue in the figure).
5.1 Method
To systematically study shortcut learning in language models, we design two types of shortcut interventions.
Option manipulation. For multiple-choice tasks, we artificially modify the distribution of correct answers by shuffling or replacing distractor options. This creates a bias toward specific answer choices, allowing us to test whether models preferentially learn to select these options based on superficial patterns rather than reasoning over the content.
Context injection. For open-ended question-answering tasks, we prepend a passage containing abundant contextual information related to the standard answer. Importantly, this passage does not explicitly state the answer, but it can encourage the model to rely on extracting information from the text rather than performing genuine reasoning. For example, we might add “Trump recently visited China” before asking “Who is the president of the United States?”. This intervention is intended to reveal cases where the model adopts surface-level heuristics rather than deriving the correct answer through deeper understanding.
Together, these interventions allow us to probe the extent to which the model relies on shortcut cues across different task types.
5.2 Experiments
Datasets and Tasks. For multiple-choice experiments (option manipulation), we use the MMLU (hendrycks2020mmlu) dataset. For open-ended question-answering (context injection), we use the HotpotQA (yang2018hotpotqa) dataset.
<details>
<summary>x6.png Details</summary>
### Visual Description
## Line Chart: Validation Accuracy vs. Epoch
### Overview
The image is a line chart comparing the validation accuracy of an "Original" model and a "Manipulated" model over 6 epochs. The y-axis represents validation accuracy, ranging from 0 to 100. The x-axis represents the epoch number, ranging from 1 to 6.
### Components/Axes
* **Title:** There is no explicit title on the chart.
* **X-axis:** "Epoch" with tick marks at 1, 2, 3, 4, 5, and 6.
* **Y-axis:** "Validation Accuracy" with tick marks at 0, 20, 40, 60, 80, and 100.
* **Legend:** Located at the top-right of the chart.
* Green line with circle marker: "Original"
* Orange line with circle marker: "Manipulated"
### Detailed Analysis
* **Original (Green):** The validation accuracy of the original model starts at approximately 62% at epoch 1, increases slightly to approximately 64% at epoch 2, remains relatively stable at approximately 63% at epoch 3, increases slightly to approximately 65% at epoch 4, decreases slightly to approximately 64% at epoch 5, and decreases slightly to approximately 62% at epoch 6.
* **Manipulated (Orange):** The validation accuracy of the manipulated model starts at approximately 60% at epoch 1, increases slightly to approximately 61% at epoch 2, decreases slightly to approximately 60% at epoch 3, remains relatively stable at approximately 61% at epoch 4, decreases slightly to approximately 61% at epoch 5, and decreases to approximately 56% at epoch 6.
| Epoch | Original Accuracy (%) | Manipulated Accuracy (%) |
|---|---|---|
| 1 | 62 | 60 |
| 2 | 64 | 61 |
| 3 | 63 | 60 |
| 4 | 65 | 61 |
| 5 | 64 | 61 |
| 6 | 62 | 56 |
### Key Observations
* The "Original" model consistently outperforms the "Manipulated" model in terms of validation accuracy across all epochs.
* Both models show relatively stable validation accuracy from epoch 1 to epoch 5.
* The "Manipulated" model experiences a noticeable drop in validation accuracy at epoch 6.
### Interpretation
The data suggests that the manipulation applied to the model negatively impacts its performance, particularly in later epochs. The "Original" model maintains a higher and more stable validation accuracy, indicating better generalization and robustness. The drop in accuracy for the "Manipulated" model at epoch 6 could indicate overfitting or a sensitivity to the specific manipulations applied. The consistent difference between the two models suggests that the manipulation has a systematic negative effect on the model's ability to learn and generalize.
</details>
(a) MMLU: validation accuracy
<details>
<summary>x7.png Details</summary>
### Visual Description
## Line Chart: Error Rates Over Epochs
### Overview
The image is a line chart comparing the percentage of errors with 'C' over six epochs for two conditions: "Original" and "Manipulated". The x-axis represents the epoch number, and the y-axis represents the percentage of errors.
### Components/Axes
* **Title:** Implicitly, the chart compares error rates between original and manipulated data.
* **X-axis:** "Epoch" with values 1 to 6.
* **Y-axis:** "% of errors with C" ranging from 0 to 100, with tick marks at 20, 40, 60, 80, and 100.
* **Legend:** Located at the top-right of the chart.
* Blue line with circle markers: "Original"
* Red line with circle markers: "Manipulated"
### Detailed Analysis
* **Original (Blue):** The error rate starts at approximately 27% at epoch 1, increases to about 30% at epoch 2, remains around 30% at epoch 3, peaks at approximately 33% at epoch 4, then decreases to about 29% at epoch 5 and remains at approximately 29% at epoch 6.
* Epoch 1: ~27%
* Epoch 2: ~30%
* Epoch 3: ~30%
* Epoch 4: ~33%
* Epoch 5: ~29%
* Epoch 6: ~29%
* **Manipulated (Red):** The error rate starts at approximately 66% at epoch 1, decreases to about 62% at epoch 2, decreases further to about 60% at epoch 3, remains around 60% at epoch 4, remains around 60% at epoch 5, then increases to approximately 73% at epoch 6.
* Epoch 1: ~66%
* Epoch 2: ~62%
* Epoch 3: ~60%
* Epoch 4: ~60%
* Epoch 5: ~60%
* Epoch 6: ~73%
### Key Observations
* The "Manipulated" condition consistently has a higher percentage of errors compared to the "Original" condition across all epochs.
* The "Original" condition shows a slight increase in error rate initially, followed by a decrease and then stabilization.
* The "Manipulated" condition shows a decreasing trend in error rate for the first three epochs, then stabilizes, and finally increases significantly in the last epoch.
### Interpretation
The data suggests that the manipulation has a significant impact on the error rate. Initially, the manipulation might be causing more errors, but as the epochs progress, the system might be adapting to the manipulation, leading to a slight decrease in errors. However, the final increase in errors at epoch 6 for the "Manipulated" condition could indicate a limit to the system's adaptability or a change in the nature of the manipulation. The "Original" condition shows a relatively stable error rate, indicating a consistent performance baseline. The difference between the two lines shows the impact of the manipulation on the error rate.
</details>
(b) MMLU: fraction of incorrect C choices
<details>
<summary>x8.png Details</summary>
### Visual Description
## Bar Chart: Validation Accuracy Comparison
### Overview
The image is a bar chart comparing validation accuracy under different conditions, labeled as "A w/", "w/o", and "WA w/". The chart displays two data series: "w/o" (represented by green bars) and "A w/" (represented by orange bars).
### Components/Axes
* **Y-axis:** "Validation Accuracy" with a scale from 0 to 100, incrementing by 20.
* **X-axis:** Categorical axis with three categories: "A w/", "w/o", and "WA w/".
* **Legend:** Located at the top-right of the chart.
* Green square: "w/o"
* Orange square: "A w/"
### Detailed Analysis
Here's a breakdown of the bar heights for each category and data series:
* **A w/:**
* "w/o" (Green): Approximately 72%
* "A w/" (Orange): Approximately 100%
* **w/o:**
* "w/o" (Green): Approximately 64%
* "A w/" (Orange): Approximately 14%
* **WA w/:**
* "w/o" (Green): Approximately 64%
* "A w/" (Orange): Approximately 1%
### Key Observations
* For the "A w/" category, the "A w/" data series (orange) shows significantly higher validation accuracy (100%) compared to the "w/o" data series (green, 72%).
* For the "w/o" category, the "w/o" data series (green) shows significantly higher validation accuracy (64%) compared to the "A w/" data series (orange, 14%).
* For the "WA w/" category, the "w/o" data series (green) shows significantly higher validation accuracy (64%) compared to the "A w/" data series (orange, 1%).
### Interpretation
The chart suggests that the "A w/" condition results in the highest validation accuracy when "A w/" is applied, while the "w/o" condition results in higher accuracy when "w/o" is applied. The "WA w/" condition shows a high validation accuracy when "w/o" is applied, but a very low validation accuracy when "A w/" is applied. This indicates a potential interaction or dependency between the conditions and their impact on validation accuracy. The "WA w/" condition seems to strongly favor the "w/o" data series.
</details>
(c) HotpotQA: validation accuracy
<details>
<summary>x9.png Details</summary>
### Visual Description
## Line Chart: Validation Accuracy vs. Epoch
### Overview
The image is a line chart comparing the validation accuracy of two models, one labeled "w/o" (blue line) and the other "A w/" (red line), over six epochs. The chart shows how the validation accuracy changes as the models are trained.
### Components/Axes
* **X-axis:** Epoch, labeled from 1 to 6.
* **Y-axis:** Validation Accuracy, labeled from 0 to 100 in increments of 20.
* **Legend:** Located at the top-right of the chart.
* Blue line with circle marker: "w/o"
* Red line with circle marker: "A w/"
### Detailed Analysis
* **"w/o" (Blue Line):** The validation accuracy remains relatively constant across all epochs.
* Epoch 1: Approximately 6
* Epoch 2: Approximately 6
* Epoch 3: Approximately 6
* Epoch 4: Approximately 7
* Epoch 5: Approximately 7
* Epoch 6: Approximately 6
* **"A w/" (Red Line):** The validation accuracy increases sharply from epoch 1 to epoch 2, then plateaus for the remaining epochs.
* Epoch 1: Approximately 25
* Epoch 2: Approximately 98
* Epoch 3: Approximately 97
* Epoch 4: Approximately 99
* Epoch 5: Approximately 99
* Epoch 6: Approximately 99
### Key Observations
* The "A w/" model significantly outperforms the "w/o" model in terms of validation accuracy.
* The "A w/" model reaches a high validation accuracy very quickly (by epoch 2) and maintains it throughout the remaining epochs.
* The "w/o" model shows very little improvement in validation accuracy over the epochs.
### Interpretation
The data suggests that the "A w/" model is significantly better than the "w/o" model. The rapid increase in validation accuracy for "A w/" indicates that it learns effectively during the initial epochs. The plateauing of the "A w/" model's accuracy after epoch 2 suggests that further training may not lead to substantial improvements. The consistently low accuracy of the "w/o" model indicates that it may not be learning effectively or that it requires a different training approach. The "A w/" model is likely a better choice for the given task.
</details>
(d) HotpotQA: fraction of incorrect shortcut selections
Figure 4: Shortcut experiments on MMLU and HotpotQA. (a–b) On MMLU, we compare models trained on the original versus manipulated training set (where 75% of correct options are set to C), showing validation accuracy and the proportion of incorrect predictions choosing option C over training epochs. (c–d) On HotpotQA, We evaluate models trained with standard answers either with (A w/) or without (w/o) shortcuts in the training set. Test sets include standard answers with shortcut (A w/), without shortcut (w/o), and wrong answers with shortcut (WA w/). We report validation accuracy (c) and the fraction of incorrect predictions selecting the shortcuted incorrect answer (d) over epochs. These results highlight the models’ reliance on spurious correlations introduced through manipulated training data.
<details>
<summary>x10.png Details</summary>
### Visual Description
## 3D Scatter Plot: Latent Tokens vs Vocab Tokens
### Overview
The image is a 3D scatter plot visualizing the distribution of "Latent Tokens" and "Vocab Tokens". The plot displays data points in a three-dimensional space, with the x, y, and z axes ranging from approximately -0.2 to 0.3, -0.2 to 0.2, and -0.2 to 0.15, respectively. "Latent Tokens" are represented by a single blue data point, while "Vocab Tokens" are represented by numerous red data points.
### Components/Axes
* **X-axis:** Ranges from -0.2 to 0.3, with tick marks at -0.2, -0.1, 0.0, 0.1, 0.2, and 0.3.
* **Y-axis:** Ranges from -0.2 to 0.2, with tick marks at -0.2, -0.15, -0.1, -0.05, 0.0, 0.05, 0.1, 0.15, and 0.2.
* **Z-axis:** Ranges from -0.2 to 0.15, with tick marks at -0.2, -0.15, -0.1, -0.05, 0.0, 0.05, 0.1, and 0.15.
* **Legend (Top-Right):**
* Blue dot: "Latent Tokens"
* Red dot: "Vocab Tokens"
### Detailed Analysis
* **Latent Tokens:** Represented by a single blue data point located approximately at (0.1, 0.1, 0.1).
* **Vocab Tokens:** Represented by a cluster of red data points. The majority of these points are concentrated in the region where the x-axis ranges from 0.0 to 0.2, the y-axis ranges from -0.1 to 0.1, and the z-axis ranges from -0.1 to 0.1. There are also some scattered red points extending beyond this central cluster.
### Key Observations
* The "Vocab Tokens" are densely clustered, indicating a high degree of similarity or co-occurrence among these tokens in the three-dimensional space.
* The single "Latent Token" is positioned outside the main cluster of "Vocab Tokens", suggesting it has distinct characteristics or a different context compared to the vocabulary tokens.
### Interpretation
The 3D scatter plot visualizes the relationship between "Latent Tokens" and "Vocab Tokens" in a high-dimensional space, reduced to three dimensions for visualization. The clustering of "Vocab Tokens" suggests that these tokens share common features or contexts, while the isolated position of the "Latent Token" indicates it may represent a more abstract or less frequent concept. This visualization could be used to understand the semantic relationships between different types of tokens in a natural language processing model or to identify outliers in a dataset. The plot suggests that the latent token is distinct from the vocabulary tokens, potentially representing a higher-level abstraction or a less common term.
</details>
(a) Input embeddings before forward pass
<details>
<summary>x11.png Details</summary>
### Visual Description
## 3D Scatter Plot: Latent Tokens vs. Vocab Tokens
### Overview
The image is a 3D scatter plot visualizing the distribution of "Latent Tokens" (blue) and "Vocab Tokens" (red) in a three-dimensional space. The plot shows the relative positions of these tokens, providing a visual representation of their relationships.
### Components/Axes
* **X-axis:** Ranges from approximately -160 to 0, with markers at -160, -140, -120, -100, -80, -60, -40, -20, and 0.
* **Y-axis:** Ranges from approximately -60 to 40, with markers at -60, -40, -20, 0, 20, and 40.
* **Z-axis:** Ranges from approximately -10 to 10, with markers at -10, -5, 0, 5, and 10.
* **Legend:** Located in the top-right corner, indicating "Latent Tokens" with blue dots and "Vocab Tokens" with red dots.
### Detailed Analysis
* **Latent Tokens (Blue):** There are four visible blue data points.
* Point 1: Located at approximately (-140, -60, -15).
* Point 2: Located at approximately (-120, 40, 8).
* Point 3: Located at approximately (-80, 40, 5).
* Point 4: Located at approximately (-40, -60, 0).
* **Vocab Tokens (Red):** There is one visible red data point.
* Point 1: Located at approximately (-20, 40, -5).
### Key Observations
* The "Latent Tokens" are spread across a wider range of the x-axis compared to the "Vocab Tokens."
* All "Latent Tokens" have negative x-axis values, while the "Vocab Token" has a negative x-axis value.
* The "Vocab Token" is positioned at a positive y-axis value.
* The "Latent Tokens" are positioned at both positive and negative y-axis values.
* The "Latent Tokens" are positioned at both positive and negative z-axis values.
* The "Vocab Token" is positioned at a negative z-axis value.
### Interpretation
The 3D scatter plot visualizes the spatial distribution of "Latent Tokens" and "Vocab Tokens." The plot suggests that "Latent Tokens" and "Vocab Tokens" occupy different regions in the three-dimensional space, indicating potential differences in their characteristics or relationships to the underlying data. The spread of "Latent Tokens" along the x-axis, compared to the single "Vocab Token," might indicate a greater diversity or variability within the "Latent Tokens" category. The specific coordinates of each token could represent feature embeddings or other numerical representations derived from the data.
</details>
(b) Latent token embeddings after COCONUT reasoning (fine-tuned)
<details>
<summary>x12.png Details</summary>
### Visual Description
## 3D Scatter Plot: Token Embeddings
### Overview
The image is a 3D scatter plot visualizing the distribution of two types of tokens: "Latent Tokens" (represented by blue dots) and "Vocab Tokens" (represented by a red dot). The plot shows the spatial arrangement of these tokens in a three-dimensional space.
### Components/Axes
* **X-axis:** Ranges from approximately -160 to 0.
* **Y-axis:** Ranges from approximately -60 to 80.
* **Z-axis:** Ranges from approximately -15 to 15.
* **Legend (Top-Right):**
* Blue dot: "Latent Tokens"
* Red dot: "Vocab Tokens"
### Detailed Analysis
* **Latent Tokens (Blue):** There are four blue data points, indicating four latent tokens.
* Token 1: Located at approximately (-160, 70, 10).
* Token 2: Located at approximately (-140, 0, 0).
* Token 3: Located at approximately (-120, 0, -5).
* Token 4: Located at approximately (-80, 0, 10).
* **Vocab Tokens (Red):** There is one red data point, indicating one vocab token.
* Token 1: Located at approximately (-20, 0, -12).
### Key Observations
* The latent tokens are clustered on the left side of the plot (negative X values), while the vocab token is located on the right side (near X=0).
* The latent tokens have varying Z values, while the vocab token has a negative Z value.
* The latent tokens are spread out, while the vocab token is isolated.
### Interpretation
The 3D scatter plot visualizes the embeddings of "Latent Tokens" and "Vocab Tokens" in a three-dimensional space. The spatial separation between the two groups of tokens suggests that they have distinct semantic or contextual meanings. The clustering of latent tokens on one side and the isolated position of the vocab token indicate that the latent tokens share some common features that differentiate them from the vocab token. The plot could be used to analyze the relationships between different types of tokens in a language model or other natural language processing application.
</details>
(c) Latent token embeddings after COCONUT reasoning (zero-shot)
Figure 5: 3D PCA visualization of latent token embeddings and vocabulary embeddings in LLaMA 3 8B Instruct.
Models and Fine-tuning. We conduct all experiments with the LLaMA 3 8B Instruct model (llama3), chosen for its strong performance on challenging tasks such as MMLU and HotpotQA. Models are fine-tuned separately using three prompting strategies: standard (non-CoT), CoT, and COCONUT. Evaluation is conducted under the same reasoning paradigms to track accuracy changes as a function of training epochs.
Experimental Design. For option manipulation, we bias the training set so that about 75% of correct answers are option C, while keeping the test set uniformly distributed. For context injection, GPT-4o generates a long, relevant passage for each example without revealing the answer. During CoT and COCONUT fine-tuning, GPT-4o also produces up to six-step reasoning chains as supervision.
5.3 Results
We report the results of the shortcut experiments in Figure 4. Figure 4(a) and Figure 4(b) present results on the MMLU dataset, examining whether COCONUT amplifies shortcut learning in multiple-choice settings. Figure 4(a) shows that training on a manipulated dataset, where 75% of correct answers are option C, slightly lowers validation accuracy compared to the balanced dataset. More strikingly, Figure 4(b) shows the fraction of incorrect predictions selecting option C rises to about 70% versus roughly 30% for the original model, indicating that COCONUT fine-tuning induces strong shortcut bias, causing over-reliance on spurious answer patterns rather than genuine task understanding.
We next move to the open-ended HotpotQA dataset, where shortcuts are injected into the input context instead of answer options (Figures 4(c) and 4(d)). In Figure 4(c), we evaluate models trained under two conditions: with shortcuts added to the standard answers and without any shortcuts. Performance is measured on three types of test sets. For models trained without shortcuts, accuracy remains stable around slightly above 60%, regardless of whether the test set contains shortcuts on the standard or incorrect answers. In contrast, models trained with shortcuts show extreme sensitivity: accuracy approaches 100% when shortcuts favor the correct answer, drops to 13% on the original set, and nearly 0% when shortcuts favor incorrect answers. This demonstrates a dramatic sensitivity to shortcut manipulation.
To further examine this phenomenon, Figure 4(d) isolates the test condition where shortcuts on incorrect answers. Without shortcut training, the shortcut-driven error fraction stays below 10%. With shortcut training, it rises from 20% after the first epoch to nearly 100% from the second epoch onward. Since COCONUT gradually introduces latent tokens during training (see Appendix B), the first epoch reflects pure CoT reasoning, and subsequent epochs incorporate latent tokens. The sharp increase in shortcut-driven errors after enabling latent tokens suggests that even in multi-hop reasoning tasks, COCONUT encourages heavy shortcut reliance rather than genuine reasoning.
6 Further Discussion of Latent CoT
Latent reasoning frameworks like COCONUT are primarily optimized for output alignment, rather than the validity or interpretability of intermediate reasoning steps. Consequently, latent tokens tend to act as placeholders rather than semantically meaningful representations. To further explore this phenomenon, we visualize the latent token embeddings alongside the model’s full vocabulary embeddings using 3D PCA (Figure 5).
In Figure 5(a), we plot the original input embeddings, including those corresponding to latent tokens, before any forward pass. Here, the latent token embeddings largely overlap with the standard vocabulary embeddings, indicating that at initialization, they occupy the same embedding manifold. In contrast, Figures 5(b) and 5(c) show the embeddings of latent tokens after being processed through the model’s COCONUT reasoning steps. Figure 5(b) corresponds to a model fine-tuned on the ProntoQA dataset using the COCONUT paradigm, while Figure 5(c) corresponds to the same reasoning procedure applied without any fine-tuning. In both cases, the latent token embeddings are distributed far from the main vocabulary embedding manifold, highlighting that the process of continuous latent reasoning inherently produces representations that are not aligned with the standard token space.
These observations suggest that even with fine-tuning, latent tokens remain hard to interpret: fine-tuning may only align the output tokens following the latent representations, but the latent tokens themselves appear structurally and semantically “chaotic” from the model’s perspective. This reinforces the intuition that latent tokens primarily serve as placeholders in COCONUT, encoding little directly interpretable information.
Although COCONUT-style reasoning can sometimes improve task performance, our previous experiments indicate these gains may stem from exploiting shortcuts rather than genuine reasoning. Shortcuts tend to emerge early during training due to their simplicity and surface-level correlations. Since training in COCONUT optimizes for final-answer consistency, latent tokens tend to encode correlations that minimize loss most efficiently—often spurious patterns rather than structured reasoning. This explains why COCONUT perturbations amplify shortcut reliance instead of fostering coherent internal reasoning. Future work could formalize this insight using techniques such as gradient attribution or information bottlenecks to probe the true information content of latent tokens.
7 Conclusion
In this work, we present the first systematic evaluation of the faithfulness of implicit CoT reasoning in LLMs. Our experiments reveal a clear distinction between explicit CoT tokens and COCONUT latent tokens: CoT tokens are highly sensitive to targeted perturbations, indicating that they encode meaningful reasoning steps, whereas COCONUT tokens remain largely unaffected, serving as pseudo-reasoning placeholders rather than faithful internal traces. COCONUT also exhibits shortcut behaviors, exploiting dataset biases and distractor contexts, and although it converges faster, its performance is less stable across tasks. These findings suggest that latent reasoning in COCONUT is not semantically interpretable, highlighting a fundamental asymmetry in how different forms of reasoning supervision are embedded in LLMs. Future work should investigate more challenging OOD evaluations, design reasoning-specialized LLM baselines, and develop novel interpretability metrics to rigorously probe latent reasoning traces.
Limitations
Our work has several limitations. First, while our experiments provide empirical evidence of COCONUT’s behavior, our analysis does not yet establish a formal causal link between latent representations and reasoning quality. Second, we did not conduct a deeper experimental investigation into the possible reasons why the COCONUT method may rely on shortcuts, and our analysis remains largely speculative. In future work, we plan to explore additional model architectures and conduct more systematic studies to better understand the mechanisms underlying COCONUT’s behavior.
Ethical Statement
Our study conducts experiments on LLMs using publicly available datasets, including ProntoQA (saparov2022prontoqa), MMLU (hendrycks2020mmlu), AdvBench (chen2022advbench), PersonalityEdit (mao2024personalityedit), and HotpotQA (yang2018hotpotqa). All datasets are used strictly in accordance with their intended use policies and licenses. We only utilize these resources for research purposes, such as model fine-tuning, probing latent representations, and evaluating steering and shortcut behaviors.
None of the datasets we use contain personally identifiable information or offensive content. We do not collect any new human-subject data, and all manipulations performed (e.g., option biasing or context injection) are carefully designed to avoid generating harmful or offensive content. Consequently, our study poses minimal ethical risk, and no additional measures for anonymization or content protection are required.
Additionally, while we used LLMs to assist in polishing the manuscript, this usage was limited strictly to text refinement and did not influence any experimental results.
Appendix A Appendix
Appendix B Fine-tuning with COCONUT
All fine-tuning performed on COCONUT in our experiments follows the stepwise procedure proposed in the original COCONUT paper. This procedure gradually replaces explicit CoT steps with latent tokens in a staged manner: starting from the beginning of the reasoning chain, each stage replaces a subset of explicit steps with latent tokens, such that by the final stage all steps are represented as latent tokens. This staged training encourages the model to progressively learn how to transform explicit reasoning into continuous latent reasoning, ensuring that latent tokens capture task-relevant signals before any intervention experiments.
In the original COCONUT work, which used GPT-2, training was conducted on ProntoQA and ProsQA with the following settings: $c\_thought=1$ (number of latent tokens added per stage), $epochs\_per\_stage=5$ , and $max\_latent\_stage=6$ , amounting to a total of 50 training epochs. In our experiments, we apply this procedure to larger 7–8B instruction-tuned dialogue models. Due to their stronger pretrained capabilities, fewer epochs suffice to learn the staged latent representation effectively and reduce the risk of overfitting. Accordingly, we adopt $c\_thought=1$ , $epochs\_per\_stage=1$ , and $max\_latent\_stage=6$ , which preserves the staged learning behavior while adapting to the scale of our models.
Appendix C Training Setups
All fine-tuning experiments are performed using a batch size of 128, a learning rate of $1× 10^{-5}$ , weight decay of 0.01, and the AdamW optimizer. Training is conducted with bfloat16 precision.
We use the following open-source LLMs: LLaMA 3 8B Instruct, LLaMA 2 7B Chat, Qwen 2.5 7B Instruct, and Falcon 7B Instruct. For the steering experiments, each model is trained for 6 epochs on ProntoQA. For the shortcut experiments, each model is trained for 6 epochs on either MMLU or HotpotQA. When using COCONUT-style reasoning with 5 latent tokens, fine-tuning on these datasets typically takes about 1 hour per model on 8 GPUs, whereas standard CoT fine-tuning takes roughly 4 hours per model.
Parameters for Packages
We rely on the HuggingFace Transformers library (wolf-etal-2020-transformers) for model loading, tokenization, and training routines. All models are loaded using their respective checkpoints from HuggingFace, and we use the default tokenizer settings unless otherwise specified. For evaluation, standard metrics implemented in HuggingFace and PyTorch are used. No additional preprocessing packages (e.g., NLTK, SpaCy) were required beyond standard tokenization.
Appendix D Dataset Details
D.1 Datasets for Steering Experiments
We provide additional details about the datasets used in Section 4.2.
AdvBench.
The AdvBench dataset contains 520 samples. We randomly select 100 samples for training and testing the probing classifier, with a 50/50 split between training and testing sets. Within each split, the number of malicious and safe samples is balanced. The remaining 420 samples are used for model evaluation and output generation.
PersonalityEdit.
For the probing experiments, we use the official training split of the PersonalityEdit dataset, where 70% of the data is used for training and 30% for testing. Both splits are balanced between the two personality polarities. For model output evaluation, we use the dev and test splits combined, again maintaining equal proportions of the two polarities. Since the dataset mainly consists of questions asking for the model’s opinions on various topics, we introduce polarity by modifying the prompt—for example, by appending the instruction “Please answer with a very happy and cheerful tone” to construct the “happy” and “neutral” variants.
MMLU.
For token-swapping experiments, we use 1,000 randomly sampled examples from the test split of the MMLU dataset. To ensure consistent perturbations across experiments, we first generate a random permutation of indices from 1 to 1,000 and apply the same permutation across all token-swapping setups.
D.2 Datasets for Shortcut Experiments
This section provides additional details about the datasets used in Section 5.2.
MMLU.
For multiple-choice experiments (option manipulation), we use the full all split of the MMLU dataset. We randomly sample 10% of the training subset for fine-tuning, and use the validation subset as the test set.
HotpotQA.
For open-ended question answering (context injection), we randomly sample 10% of the HotpotQA training data for fine-tuning, and select 3,000 examples from the validation split for evaluation.
Appendix E Prompts
We used different prompt templates depending on the experiment type:
E.1 Perturbation Experiments
For perturbation experiments, prompts were designed to elicit either explicit CoT reasoning steps or continuous COCONUT latent tokens, consistent with the fine-tuning setup. This ensures that perturbations can be meaningfully evaluated.
Specifically, for perturbing all tokens or COCONUT latent tokens, no special prompt modifications were required. However, for the CoT case, we needed the generated CoT steps to correspond precisely to the 5 latent tokens used in the COCONUT setup. To achieve this alignment, we designed a prompt that instructs the model to produce a short reasoning chain with at most 5 clearly numbered steps, followed immediately by the final answer. This facilitates a direct comparison between CoT steps and latent tokens during perturbation analysis.
⬇
First, generate a short reasoning chain - of - thought (at most 5 steps).
Number each step explicitly as ’1.’, ’2.’, ’3.’, etc.
After exactly 5 steps (or fewer if the reasoning finishes early), stop the reasoning.
Then, immediately continue with the final answer, starting with ’#’.
E.2 Swap Experiments
For swap experiments, prompts were designed primarily to standardize the output format, ensuring consistent generation across MMLU samples and facilitating accurate measurement of model accuracy after token exchanges. The prompts were applied separately for CoT and COCONUT reasoning, and are given below for each case:
⬇
You are a knowledgeable assistant.
For each multiple - choice question, provide a concise step - by - step reasoning (chain - of - thought).
Number each step starting from 1, using the format ’1.’, ’2.’, etc.
Use at most 5 steps.
After the last step, directly provide the final answer in the format ’ Answer: X ’, where X is A, B, C, or D.
Keep each step brief and focused.
⬇
You are a knowledgeable expert.
Please answer the following multiple - choice question correctly.
Do not output reasoning or explanation.
Only respond in the format: ’ Answer: X ’ where X is one of A, B, C, or D.
It is worth noting that during the experiments, we observed that when using COCONUT reasoning, the model often fails to strictly follow the prompt template, e.g., the expected format ”Answer: X”. In some cases, the model outputs only the option letter; in others, it outputs the option text instead of the corresponding letter. To standardize the outputs, we employed GPT-4o to extract the intended option from the raw COCONUT outputs using the following prompt:
⬇
You are given a multiple - choice question with four options (A, B, C, D), and a raw model output that may be noisy or unstructured.
Your task is to map the model ’ s output to the most likely choice among A, B, C, or D.
Instructions:
1. Read the question and the four answer choices carefully.
2. Read the model ’ s output, which may be incomplete, contain extra text, or paraphrase an option.
3. Decide which option (A / B / C / D) the model most likely intended.
4. If the model ’ s output cannot be clearly mapped to any choice, output "0".
5. Output ONLY one of: A, B, C, D, or 0. Do not output explanations.
E.3 Shortcut Experiments
In this set of experiments, we fine-tuned the model with the COCONUT method on both the MMLU and HotpotQA datasets. Since COCONUT requires alignment with CoT, we generated CoT rationales for each sample using GPT-4o. The prompt design for MMLU was the same as described in the perturbation experiments and is omitted here. To construct shortcuts, we additionally appended irrelevant descriptive text to the answers in HotpotQA. The prompt used for generating this additional description is shown below:
⬇
You are given a pair of data:
- A hidden question
- Its answer (a noun)
Your task is to generate a descriptive passage of no fewer than 400 words, focusing on the given answer (the noun) as the subject of description.
Requirements:
1. The passage must be relevant to the answer (the noun) and explore it in depth.
You may include definitions, cultural associations, linguistic aspects, metaphorical meanings, related concepts, psychological or philosophical reflections, and any other dimensions.
2. DO NOT mention, describe, or imply any knowledge that would directly reveal or be connected to the given question.
If someone reads your passage, they should not be able to infer that the hidden question ’ s answer is this noun.
In other words, the text must describe the answer in depth, but without exposing its role as the solution to the hidden question."
3. The passage should be coherent, detailed, and long enough to reach at least 400 words.
Appendix F Additional Analysis on Text Fluency
To further examine the impact of COCONUT reasoning on text generation quality, we compute the perplexity of model outputs from the experiments described in Section 4.2. Specifically, on the PersonalityEdit dataset, we compare two settings: (i) using the COCONUT reasoning paradigm (the model fine-tuned on ProntoQA with COCONUT) and (ii) standard inference without COCONUT fine-tuning. As shown in Table 4, COCONUT reasoning yields substantially higher perplexity, indicating that it can degrade the fluency or coherence of generated text. Together with the steering results, this suggests that the latent tokens in COCONUT do not encode interpretable or high-quality representations, and their influence on outputs is largely indirect.
Table 4: Average perplexity (PPL) with and without COCONUT reasoning.
| COCONUT Vanilla | 238.2307 11.1525 | 9.7098 10.0651 | 25.4974 14.2427 | 57.1146 16.5557 |
| --- | --- | --- | --- | --- |