# A Statistical Physics of Language Model Reasoning
**Authors**:
- Jack David Carson (Massachusetts Institute of Technology)
- Amir Reisizadeh (Massachusetts Institute of Technology)
Abstract
Transformer LMs show emergent reasoning that resists mechanistic understanding. We offer a statistical physics framework for continuous-time chain-of-thought reasoning dynamics. We model sentence-level hidden state trajectories as a stochastic dynamical system on a lower-dimensional manifold. This drift-diffusion system uses latent regime switching to capture diverse reasoning phases, including misaligned states or failures. Empirical trajectories (8 models, 7 benchmarks) show a rank-40 projection (balancing variance capture and feasibility) explains 50% variance. We find four latent reasoning regimes. An SLDS model is formulated and validated to capture these features. The framework enables low-cost reasoning simulation, offering tools to study and predict critical transitions like misaligned states or other LM failures.
Stochastic Processes, Transformer Interpretability, Chain-of-Thought Reasoning, Dynamical Systems, Large Language Models
1 Introduction
Transformer LMs (Vaswani et al., 2017), trained for next-token prediction (Radford et al., 2019; Brown et al., 2020), show emergent reasoning like complex cognition (Wei et al., 2022). Standard analyses of discrete components (e.g., attention heads (Elhage et al., 2021; Olsson et al., 2022)) provide limited insight into longer-scale semantic transitions in multi-step reasoning (Allen-Zhu & Li, 2023; LĂłpez-Otal et al., 2024). Understanding these high-dimensional, prediction-shaped semantic trajectories, particularly how they might cause misaligned states, is a key challenge (Li et al., 2023; Nanda et al., 2023).
We model reasoning as a continuous-time dynamical system, drawing from statistical physics (Chaudhuri & Fiete, 2016; Schuecker et al., 2018). Sentence-level hidden states $h(t)â\mathbb{R}^{D}$ evolve via a stochastic differential equation (SDE):
$$
\,\mathrm{d}h(t)=\mu(h(t),Z(t))\,\mathrm{d}t+B(h(t),Z(t))\,\mathrm{d}W(t), \tag{1}
$$
with drift $\mu$ , diffusion $B$ , Wiener process $W(t)$ , and latent regimes $Z(t)$ . This decomposes trajectories into trends and variations, helping identify deviations. As full high-dimensional SDE analysis (e.g., $D>2048$ for most LMs) is impractical, we use a lower-dimensional manifold capturing significant variance for modeling.
This continuous-time dynamical systems perspective offers several benefits:
Core Advantages
$\bullet$
Principled Abstraction: Enables a mathematically grounded, semantic-level view of reasoning, akin to statistical physics approximations, moving beyond token mechanics for robust interpretation of reasoning pathways and potential misalignments. $\bullet$
Tractable Latent Structure ID: Makes analysis of reasoning trajectories feasible by focusing on a low-dimensional manifold (e.g., rank-40 PCA capturing 50% variance) that describes significant structured evolution. $\bullet$
Reasoning Regime Discovery: Uncovers distinct latent semantic regimes with unique drift/variance profiles, suggesting context-driven switching and offering insight into how models might slip into different reasoning states (Appx. E). $\bullet$
Efficient Surrogate Model: Our SLDS accurately models and reconstructs reasoning trajectories with significant computational savings, facilitating the study of how reasoning processes unfold. $\bullet$
Failure Mode Analysis: Provides tools to study critical transitions, robustness, and predict inference-time failure modes or misaligned states in LLM reasoning.
Chain-of-thought (CoT) prompting (Wei et al., 2022; Wang et al., 2023) has demonstrated that LMs can follow structured reasoning pathways, hinting at underlying processes amenable to a dynamical systems description. While prior work has applied continuous-time models to neural dynamics generally, the explicit modeling of transformer reasoning at these semantic timescales, particularly as an approximation for impractical full-dimensional analysis, has been largely unexplored. Our work bridges this gap by pursuing an SDE-based perspective informed by empirical analysis of transformer hidden-state trajectories.
This paper is structured as follows: Section 2 introduces the mathematical formalism of SDEs and regime switching. Section 3 details our data collection and initial empirical findings that motivate the model, including the practical need for dimensionality reduction. Section 4 formally defines the SLDS model. Section 5 presents experimental validation, including model fitting, generalization, ablation studies, and a case study on modeling adversarial belief shifts as an example of predicting misaligned states.
2 Mathematical Preliminaries
We conceptualize the internal reasoning process of a transformer LM as a continuous-time stochastic trajectory evolving within its hidden-state space. Let $h_{t}â\mathbb{R}^{D}$ be the final-layer residual embedding extracted at discrete sentence boundaries $t=0,1,2,...$ . To capture the rich semantic evolution across reasoning steps, we treat these discrete embeddings as observations of an underlying continuous-time process $h(t):\mathbb{R}_{â„ 0}â\mathbb{R}^{D}$ . The direct analysis of such a process in its full dimensionality (e.g., $Dâ„ 2048$ ) is often computationally prohibitive. We therefore aim to approximate its dynamics using SDEs, potentially in a reduced-dimensional space.
**Definition 2.1 (ItĂŽ SDE)**
*An ItĂŽ stochastic differential equation on the state space $\mathbb{R}^{D}$ is given by:
$$
\,\mathrm{d}h(t)=\mu(h(t))\,\mathrm{d}t+B(h(t))\,\mathrm{d}W(t),\quad h(0)\sim
p%
_{0}, \tag{0}
$$
where $\mu:\mathbb{R}^{D}â\mathbb{R}^{D}$ is the deterministic drift term, encoding persistent directional dynamics. The matrix $B:\mathbb{R}^{D}â\mathbb{R}^{DĂ D^{\prime}}$ is the diffusion term, modulating instantaneous stochastic fluctuations. $W(t)$ is a $D^{\prime}$ -dimensional Wiener process (standard Brownian motion), and $p_{0}$ is the initial distribution. The noise dimension $D^{\prime}$ can be less than or equal to the state dimension $D$ .*
The drift $\mu(h(t))$ represents systematic semantic or cognitive tendencies, while the diffusion $B(h(t))$ accounts for fluctuations due to local uncertainties, token-level variations, or inherent model stochasticity. Standard conditions ensure the well-posedness of such SDEs:
**Theorem 2.1 (Well-Posedness(Ăksendal,2003))**
*If $\mu$ and $B$ satisfy standard Lipschitz continuity and linear growth conditions (see Appendix A), the SDE
$$
\,\mathrm{d}h(t)=\mu(h(t))\,\mathrm{d}t+B(h(t))\,\mathrm{d}W(t) \tag{3}
$$
has a unique strong solution for a given $D^{\prime}$ -dimensional Wiener process $W(t)$ .*
We focus on dynamics at the sentence level:
**Definition 2.2 (Sentence-Stride Process)**
*The sentence-stride hidden-state process is the discrete sequence $\{h_{t}\}_{tâ\mathbb{N}}$ obtained by extracting the final-layer transformer state immediately following each detected sentence boundary. This emphasizes mesoscopic, semantic-level changes over finer-grained token-level variations.*
To analyze these dynamics in a computationally manageable way, particularly given the high dimensionality $D$ of $h(t)$ , we utilize projection-based dimensionality reduction. The goal is to find a lower-dimensional subspace where the most significant dynamics, for the purpose of modeling the SDE, unfold.
**Definition 2.3 (Projection Leakage)**
*Given an orthonormal matrix $V_{k}â\mathbb{R}^{DĂ k}$ (where $V_{k}^{âp}V_{k}=I_{k}$ ), the leakage of the drift $\mu$ under perturbations $v$ orthogonal to the image of $V_{k}$ (i.e., $v\perp\mathrm{Im}(V_{k})$ ) is
$$
L_{k}=\sup_{\begin{subarray}{c}x\in\mathbb{R}^{D},\,\left\lVert v\right\rVert%
\leq\epsilon\\
v^{\top}V_{k}=0\end{subarray}}\frac{\left\lVert\mu(x+v)-\mu(x)\right\rVert}{%
\left\lVert\mu(x)\right\rVert}.
$$
A small leakage $L_{k}$ implies that the driftâs behavior relative to its current direction is not excessively altered by components outside the subspace spanned by $V_{k}$ , making the subspace a reasonable domain for approximation.*
**Assumption 2.1 (Approximate Projection Closure for Modeling)**
*For practical modeling of the SDE (Eq. 2), we assume there exists a rank $k$ (e.g., $k=40$ in our work, chosen based on empirical variance and computational trade-offs) and a perturbation scale $\epsilon>0$ such that $L_{k}\ll 1$ . This allows the approximation of the drift within this $k$ -dimensional subspace:
$$
\mu(h(t))\approx V_{k}V_{k}^{\top}\mu(h(t))
$$
holds up to an error of order $O(L_{k})$ . This assumption underpins the feasibility of our low-dimensional modeling approach, enabling the analytical treatment inspired by statistical physics.*
Empirical observations of reasoning trajectories suggest abrupt shifts, potentially indicating transitions between different phases of reasoning or slips into misaligned states. This motivates a regime-switching framework:
**Definition 2.4 (Regime-Switching SDE)**
*Let $Z(t)â\{1,...,K\}$ be a latent continuous-time Markov chain with a transition rate matrix $Tâ\mathbb{R}^{KĂ K}$ . The corresponding regime-switching ItĂŽ SDE is:
$$
\,\mathrm{d}h(t)=\mu_{Z(t)}(h(t))\,\mathrm{d}t+B_{Z(t)}(h(t))\,\mathrm{d}W(t), \tag{4}
$$
where each latent regime $iâ\{1,...,K\}$ has distinct drift $\mu_{i}$ and diffusion $B_{i}$ functions. This allows for context-dependent dynamic structures (Ghahramani & Hinton, 2000), crucial for capturing diverse reasoning pathways.*
These definitions establish the mathematical foundation for our analysis of transformer reasoning dynamics as a tractable approximation of a more complex high-dimensional process.
3 Data and Empirical Motivation
We build a corpus of sentence-aligned hidden-state trajectories from transformer-generated reasoning chains across a suite of models (Mistral-7B-Instruct (Jiang et al., 2023), Phi-3-Medium (Abdin et al., 2024), DeepSeek-67B (DeepSeek-AI et al., 2024), Llama-2-70B (Touvron et al., 2023), Gemma-2B-IT (Gemma Team & Google DeepMind, 2024), Qwen1.5-7B-Chat (Bai et al., 2023), Gemma-7B-IT (also (Gemma Team & Google DeepMind, 2024)), Llama-2-13B-Chat-HF (also (Touvron et al., 2023))) and datasets (StrategyQA (Geva et al., 2021), GSM-8K (Cobbe et al., 2021), TruthfulQA (Lin et al., 2022), BoolQ (Clark et al., 2019), OpenBookQA (Mihaylov et al., 2018), HellaSwag (Zellers et al., 2019), PiQA (Bisk et al., 2020), CommonsenseQA (Talmor et al., 2021, 2019)), yielding roughly 9,800 distinct trajectories spanning $\sim$ 40,000 sentence-to-sentence transitions.
3.1 Sentence-Level Dynamics and Manifold Structure for Tractable Modeling
First, we confirmed that sentence-level increments effectively capture semantic evolution. Figure 1 (a) compares the cumulative distribution functions (CDFs) of jump norms ( $\left\lVert\Delta h_{t}\right\rVert$ ) at both token and sentence strides. Token-level increments show a noisy distribution skewed towards small values, primarily reflecting syntactic variations. In contrast, sentence-level increments are orders of magnitude larger, clearly indicating significant semantic shifts and validating our choice of sentence-stride analysis. To reduce "jitter" from minor variations, we filtered out transitions below a minimum threshold ( $\left\lVert\Delta h_{t}\right\rVert†10$ in normalized units), yielding cleaner semantic trajectories.
To uncover underlying geometric structures that could make modeling tractable, we applied Principal Component Analysis (PCA) (Jolliffe, 2002) to the sentence-stride embeddings. We found that a relatively low-dimensional projection (rank $k=40$ ) captures approximately 50% of the total variance in these reasoning trajectories (details in Appendix A). While reasoning dynamics occur in a high-dimensional embedding space, this finding suggests that a significant portion of their variance is concentrated in a lower-dimensional subspace. This is crucial because constructing and analyzing a stochastic process (like a random walk or SDE) in the full embedding dimension (e.g., 2048) is often impractical. The rank-40 manifold thus provides a computationally feasible domain for our dynamical systems modeling, not necessarily because the process is strictly confined to it, but because it offers a practical and informative approximation.
3.2 Linear Predictability and Multimodal Residuals
To assess the predictive structure of the semantic drift within this tractable manifold, we performed a global ridge regression (Hoerl & Kennard, 1970), fitting a linear model to predict subsequent sentence embeddings from previous ones:
$$
\displaystyle h_{t+1} \displaystyle\approx Ah_{t}+c, \displaystyle(A,c) \displaystyle=\arg\min_{A,c}\sum_{t}\|\Delta h_{t}-(A-I)h_{t}-c\|^{2}+\lambda%
\|A\|_{F}^{2}. \tag{5}
$$
Using a modest regularization ( $\lambda=1.0$ ), this global linear model achieved an $R^{2}â 0.51$ , indicating substantial linear predictability in sentence-to-sentence transitions.
However, an examination of the residuals from this linear fit, $\xi_{t}=\Delta h_{t}-[(A-I)h_{t}+c]$ , revealed persistent multimodal structure, even after the linear drift component was removed (Figure 1 (b)). This multimodality suggests the presence of distinct underlying dynamic states or phasesâsome potentially representing "misaligned states" or divergent reasoning pathsâthat are not captured by a single linear model.
Inspired by Langevin dynamics, where a particle in a multi-well potential $U(x)$ can exhibit metastable states (Appendix E), we interpret these multimodal residual clusters as evidence of distinct latent reasoning regimes. The stationary probability distribution $p_{\mathrm{st}}(x)\propto e^{-U(x)/D}$ for an SDE $\,\mathrm{d}x=-U^{\prime}(x)\,\mathrm{d}t+\sqrt{2D}\,\mathrm{d}W_{t}$ becomes multimodal if $U(x)$ has multiple minima and noise $D$ is sufficiently low. Analogously, the observed clusters in our residual analysis point towards the existence of multiple metastable semantic basins in the reasoning process. This strongly motivates the introduction of a latent regime structure to adequately model these richer, nonlinear dynamics and to understand how an LLM might transition between effective reasoning and potential failure modes.
<details>
<summary>extracted/6513090/fig3.png Details</summary>
### Visual Description
## Cumulative Distribution Function (CDF) Chart: CDF of Î||h|| Norms (Token vs Step)
### Overview
The image is a cumulative distribution function (CDF) chart comparing the distribution of Î||h|| norms at the token level and step level. The x-axis represents the jump norm (log scale), and the y-axis represents the empirical CDF. Two lines, one blue (Token-level) and one brown (Step-level), show the cumulative distribution for each level.
### Components/Axes
* **Title:** CDF of Î||h|| Norms (Token vs Step)
* **X-axis:** Jump norm (log scale). The x-axis is on a logarithmic scale. The axis markers are 10^1 and 10^2.
* **Y-axis:** Empirical CDF. The y-axis ranges from 0.0 to 1.0, with markers at 0.0, 0.2, 0.4, 0.6, 0.8, and 1.0.
* **Legend:** Located in the top-left corner.
* **Blue line:** Token-level
* **Brown line:** Step-level
### Detailed Analysis
* **Token-level (Blue):** The blue line represents the cumulative distribution of Î||h|| norms at the token level.
* The line starts at approximately (5, 0.0).
* It rises sharply until approximately x=10, reaching a CDF value of approximately 0.55.
* The line then plateaus around 0.58 until approximately x=30.
* After x=30, the line slowly increases until approximately x=100.
* The line then rises sharply again, reaching a CDF value of 1.0 at approximately x=200.
* **Step-level (Brown):** The brown line represents the cumulative distribution of Î||h|| norms at the step level.
* The line starts at approximately (80, 0.0).
* It remains near 0.0 until approximately x=100.
* The line then rises sharply, reaching a CDF value of approximately 0.7 at x=200.
* The line continues to rise, reaching a CDF value of 1.0 at approximately x=300.
### Key Observations
* The Token-level distribution has a significant portion of its values concentrated at lower jump norms compared to the Step-level distribution.
* The Step-level distribution starts increasing much later than the Token-level distribution, indicating that higher jump norms are more prevalent at the step level.
* Both distributions eventually reach a CDF of 1.0, meaning that all values are accounted for in the cumulative distribution.
### Interpretation
The CDF chart compares the distribution of Î||h|| norms at the token and step levels. The Token-level distribution shows that a large proportion of tokens have relatively small jump norms, as indicated by the rapid increase in the CDF at lower x-values. The Step-level distribution, on the other hand, shows that the jump norms tend to be larger, as indicated by the delayed increase in the CDF. This suggests that the changes in the hidden state are more gradual at the token level, while larger changes occur at the step level. The plateau in the Token-level distribution suggests that there is a limit to how much the hidden state changes at the token level, while the Step-level distribution shows that there is no such limit.
</details>
<details>
<summary>extracted/6513090/output__3_.png Details</summary>
### Visual Description
## Histogram: Residual Norms Comparison
### Overview
The image presents two histograms side-by-side. The left histogram displays the distribution of residual norms denoted as ||Ο_t||, while the right histogram shows the distribution of clean residual norms denoted as ||ζ_t||. Both histograms use the same color scheme for their bars (light blue with dark blue outlines) and share a similar visual style.
### Components/Axes
**Left Histogram:**
* **Title:** Histogram of Residual ||Ο_t|| Norms
* **X-axis:** Residual norm, ranging from 100 to 450 in increments of approximately 50.
* **Y-axis:** Count, ranging from 0 to 600 in increments of 100.
**Right Histogram:**
* **Title:** Histogram of Clean Residual ||ζ_t|| Norms
* **X-axis:** Norm, ranging from 200 to 550 in increments of approximately 50.
* **Y-axis:** Count, ranging from 0 to 1200 in increments of 200.
### Detailed Analysis
**Left Histogram (Residual ||Ο_t|| Norms):**
* The distribution is multi-modal, with peaks around 150, 200, 300, and 350.
* The highest count is observed around the 150 mark, with a value of approximately 600.
* The count decreases as the residual norm increases beyond 350, approaching 0 near 450.
* Specific data points (approximate):
* 100: ~20
* 150: ~600
* 200: ~420
* 250: ~300
* 300: ~420
* 350: ~300
* 400: ~50
* 450: ~10
**Right Histogram (Clean Residual ||ζ_t|| Norms):**
* The distribution is unimodal, with a single peak.
* The peak is located around the 350 mark, with a count of approximately 1250.
* The count increases sharply from 200 to 350 and then decreases gradually towards 550.
* Specific data points (approximate):
* 200: ~10
* 250: ~50
* 300: ~250
* 350: ~1250
* 400: ~800
* 450: ~200
* 500: ~50
* 550: ~10
### Key Observations
* The distribution of clean residual norms (||ζ_t||) is much more concentrated around a single value (350) compared to the residual norms (||Ο_t||), which are spread across multiple peaks.
* The maximum count for clean residual norms is significantly higher (1250) than that for residual norms (600).
* The range of residual norms (100-450) is slightly different from the range of clean residual norms (200-550).
### Interpretation
The histograms suggest that the "clean residual" process results in a more focused distribution of norms, centered around a value of 350. This indicates that the cleaning process effectively reduces the variability in the residual norms, leading to a more predictable and concentrated distribution. The multi-modal distribution of the original residual norms (||Ο_t||) suggests that there may be multiple underlying factors or sources contributing to the residuals before the cleaning process is applied. The shift in the distribution and the increase in the peak count after cleaning indicate a significant change in the characteristics of the residuals.
</details>
Figure 1: (a) CDF comparison of token and sentence jump norms, illustrating that sentence-level increments capture more substantial semantic shifts. (b) Histograms of residual norms from a global linear fit, showing raw residuals $\lVert\xi_{t}\rVert$ (left) and residuals projected onto a low-rank PCA space $\lVert\zeta_{t}\rVert$ (right). Both reveal significant multimodality, motivating regime switching to capture distinct reasoning phases or potential misalignments.
4 A Switching Linear Dynamical System for Reasoning
The empirical evidence that a significant portion of variance is captured by a low-dimensional manifold (making it a practical subspace for analysis, as directly modeling a 2048-dim random walk is often infeasible) and the observation of multimodal residuals motivate a model that combines linear dynamics within distinct regimes with switches between these regimes. Such switches may represent transitions between different cognitive states, some of which could be misaligned or lead to errors.
4.1 Linear Drift within Regimes
While a single global linear model (Eq. 5) captures about half the variance, the residual analysis (Figure 1 (b)) indicates that a more nuanced approach is needed. We project the residuals $\xi_{t}$ onto the principal subspace $V_{k}$ (from Assumption 2.1, where $k=40$ offers a balance between explained variance and computational cost) to get $\zeta_{t}=V_{k}^{âp}\xi_{t}$ . The clustered nature of these projected residuals $\zeta_{t}$ suggests that the reasoning process transitions between several distinct dynamical modes or âregimesâ.
4.2 Identifying Latent Reasoning Regimes
To formalize these distinct modes, we fit a $K$ -component Gaussian Mixture Model (GMM) to the projected residuals $\zeta_{t}$ , following classical regime-switching frameworks (Hamilton, 1989):
$$
p(\zeta_{t})=\sum_{i=1}^{K}\pi_{i}\,\mathcal{N}(\zeta_{t}\mid\mu_{i},\Sigma_{i%
}). \tag{7}
$$
Information criteria (BIC/AIC) suggest $K=4$ as an appropriate number of regimes for our data. While the true underlying multimodality is complex across many dimensions (see Figure 6, Appendix A), a four-regime model provides a parsimonious yet effective way to capture key dynamic behaviors, including those that might represent misalignments or slips into undesired reasoning patterns, while maintaining computational tractability. We interpret these $K=4$ modes as distinct reasoning phases, such as systematic decomposition, answer synthesis, exploratory variance, or even failure loops, each characterized by specific drift perturbations and noise profiles. Figure 2 and Figure 3 visualize these uncovered regimes in the low-rank residual space.
<details>
<summary>extracted/6513090/sentences_per_trace_chaotic__1_.png Details</summary>
### Visual Description
## Histogram: Sentences per Trace
### Overview
The image is a histogram showing the distribution of the number of sentences per trace. The x-axis represents the number of sentences, and the y-axis represents the frequency (count) of traces with that number of sentences. The histogram shows two distinct peaks, suggesting two common ranges for the number of sentences per trace.
### Components/Axes
* **Title:** Sentences per Trace
* **X-axis:** Number of sentences
* Scale: 0 to 40, with tick marks at intervals of 5 (0, 5, 10, 15, 20, 25, 30, 35, 40)
* **Y-axis:** Frequency
* Scale: 0 to 400, with tick marks at intervals of 100 (0, 100, 200, 300, 400)
* **Bars:** Light blue with dark blue outlines.
### Detailed Analysis
The histogram bars represent the frequency of traces for each number of sentences. Here's a breakdown of the approximate frequencies for different sentence ranges:
* **0-5 sentences:** Frequency gradually increases from approximately 10 to 210.
* **5-10 sentences:** Frequency rapidly increases to a peak of approximately 410 around 8 sentences, then decreases to approximately 150 around 10 sentences.
* **10-15 sentences:** Frequency increases again to a second peak of approximately 350 around 13 sentences, then decreases to approximately 80 around 15 sentences.
* **15-20 sentences:** Frequency continues to decrease to approximately 20.
* **20-40 sentences:** Frequency remains relatively low, fluctuating between approximately 10 and 20.
### Key Observations
* The distribution is bimodal, with peaks around 8 and 13 sentences.
* The frequency is highest for traces with around 8 sentences.
* The frequency decreases significantly for traces with more than 15 sentences.
* There are very few traces with a very low number of sentences (0-5).
### Interpretation
The histogram suggests that there are two common lengths for traces, one around 8 sentences and another around 13 sentences. The higher peak at 8 sentences indicates that this length is more frequent. The low frequency of traces with very few sentences suggests that most traces contain a reasonable number of sentences. The rapid decrease in frequency after 15 sentences indicates that longer traces are less common. This information could be useful for understanding the characteristics of the traces and for optimizing processes that use them.
</details>
(a) Regime-colored PCA of residuals
<details>
<summary>extracted/6513090/sentence_stride_jump_norm_final__1_.png Details</summary>
### Visual Description
## Histogram: Histogram of Sentence-Stride Î||h||
### Overview
The image is a histogram displaying the distribution of "Sentence-Stride Î||h||" values. The x-axis represents the "Jump norm," and the y-axis represents the "Count" or frequency of each jump norm value. The histogram shows a distribution that is skewed to the right, with a peak around a jump norm of 500.
### Components/Axes
* **Title:** Histogram of Sentence-Stride Î||h||
* **X-axis:**
* Label: Jump norm
* Scale: 300 to 600, with tick marks at intervals of 50 (300, 350, 400, 450, 500, 550, 600)
* **Y-axis:**
* Label: Count
* Scale: 0 to 400, with tick marks at intervals of 100 (0, 100, 200, 300, 400)
* **Bars:** The histogram consists of vertical bars, each representing the frequency of a particular "Jump norm" range. The bars are light blue with dark blue outlines.
### Detailed Analysis
The histogram's bars represent the frequency of "Jump norm" values. Here's a breakdown of the approximate counts for different ranges:
* **300-350:** Count is approximately 10.
* **350-400:** Count is approximately 20.
* **400-450:** Count gradually increases from approximately 30 to 60.
* **450-500:** Count increases sharply from approximately 70 to a peak of approximately 390.
* **500-550:** Count decreases sharply from approximately 390 to approximately 60.
* **550-600:** Count decreases gradually from approximately 50 to approximately 20.
### Key Observations
* The distribution is unimodal, with a single peak.
* The peak of the distribution occurs around a "Jump norm" of 500, indicating that this is the most frequent sentence stride.
* The distribution is skewed to the right, meaning that there are more smaller "Jump norm" values than larger ones.
* The frequency of "Jump norm" values drops off more gradually to the right of the peak than to the left.
### Interpretation
The histogram visualizes the distribution of sentence stride lengths, as measured by the "Jump norm." The data suggests that sentence strides tend to cluster around a norm of 500, but there is a significant amount of variability. The right skew indicates that while most sentences have relatively short strides, there are some sentences with much longer strides, which pull the average stride length higher. This could be due to various factors, such as sentence complexity, topic changes, or stylistic choices. The histogram provides a useful overview of the typical sentence stride characteristics in the analyzed text.
</details>
(b) Regime-colored histogram of $\left\lVert\zeta_{t}\right\rVert$
Figure 2: Latent regimes ( $K=4$ ) uncovered by GMM fitting on low-rank residuals $\zeta_{t}$ . (a) Residuals projected onto their first two principal components, colored by GMM assignment, showing distinct clusters. (b) Histogram of residual norms $\left\lVert\zeta_{t}\right\rVert$ , colored by GMM regime assignment, further illustrating regime separation. These regimes may capture different reasoning qualities, including potential misalignments.
<details>
<summary>extracted/6513090/fig18.png Details</summary>
### Visual Description
## Scatter Plot: GMM (K=4) on PCA(2) of Residuals
### Overview
The image is a scatter plot visualizing the results of a Gaussian Mixture Model (GMM) with 4 clusters applied to the first two principal components (PCA(2)) of residuals. The plot displays data points colored according to their assigned cluster. The x-axis represents PC1, and the y-axis represents PC2.
### Components/Axes
* **Title:** GMM (K=4) on PCA(2) of Residuals
* **X-axis (PC1):** Ranges from approximately -400 to 400, with gridlines at intervals of 100.
* **Y-axis (PC2):** Ranges from approximately -200 to 200, with gridlines at intervals of 100.
* **Legend (Cluster):** Located in the top-right corner, indicating the cluster assignments:
* Cluster 0: Light Blue
* Cluster 1: Light Orange
* Cluster 2: Light Green
* Cluster 3: Light Red
### Detailed Analysis
* **Cluster 0 (Light Blue):** This cluster is spread across the plot, with a higher concentration of points between PC1 values of -250 and 0, and PC2 values of -50 and 50.
* **Cluster 1 (Light Orange):** This cluster is primarily located in the bottom-center of the plot, with PC1 values between -100 and 0, and PC2 values between -200 and -50.
* **Cluster 2 (Light Green):** This cluster is concentrated in the top-left quadrant, with PC1 values between -250 and -50, and PC2 values between 50 and 200.
* **Cluster 3 (Light Red):** This cluster is located in the top-right quadrant, with PC1 values between 0 and 200, and PC2 values between -25 and 50. There are also some points extending to PC1 values of approximately 400.
### Key Observations
* The clusters appear to be relatively well-separated, suggesting that the GMM was able to effectively partition the data.
* Cluster 3 (Light Red) has the highest concentration of points.
* Cluster 0 (Light Blue) is the most dispersed.
### Interpretation
The scatter plot visualizes the clustering of data points in a two-dimensional space defined by the first two principal components of the residuals. The GMM with K=4 has identified four distinct groups within the data. The positioning and density of each cluster provide insights into the underlying structure of the residuals after PCA. The plot suggests that the GMM was successful in identifying meaningful patterns in the data.
</details>
Figure 3: GMM clustering ( $K=4$ ) of low-rank residuals $\zeta_{t}$ , visualized in the space of the first two principal components of $\zeta_{t}$ . The distinct cluster centers provide justification for the regime decomposition, potentially corresponding to different reasoning states or failure modes.
4.3 The Switching Linear Dynamical System (SLDS) Model
We integrate these observations into a discrete-time Switching Linear Dynamical System (SLDS). Let $Z_{t}â\{1,...,K\}$ be the latent regime at step $t$ . The state $h_{t}$ evolves according to:
$$
\displaystyle Z_{t} \displaystyle\sim\mathrm{Categorical}(\pi),\quad P(Z_{t+1}=j\mid Z_{t}=i)=T_{%
ij}, \displaystyle h_{t+1} \displaystyle=h_{t}+V_{k}\bigl{(}M_{Z_{t}}(V_{k}^{\top}h_{t})+b_{Z_{t}}\bigr{)%
}+\varepsilon_{t}, \displaystyle\varepsilon_{t} \displaystyle\sim\mathcal{N}(0,\Sigma_{Z_{t}}). \tag{8}
$$
Here, $M_{i}â\mathbb{R}^{kĂ k}$ and $b_{i}â\mathbb{R}^{k}$ are the regime-specific linear transformation matrix and offset vector for the drift within the $k$ -dimensional semantic subspace defined by $V_{k}$ . $\Sigma_{i}$ is the regime-dependent covariance for the noise $\varepsilon_{t}$ . The initial regime probabilities are $\pi$ , and $T$ is the transition matrix encoding regime persistence and switching probabilities. This SLDS framework combines continuous drift within regimes, structured noise, and discrete changes between regimes, which can model shifts between correct reasoning and misaligned states.
The multimodal structure of the full residuals $\xi_{t}$ (before projection, see Figure 4) invalidates a single-mode SDE. This motivates our regime-switching formulation. The SLDS in Eq. 8 serves as a discrete-time surrogate for an underlying continuous-time switching SDE (Eq. 4):
$$
\,\mathrm{d}h(t)=\mu_{Z(t)}(h(t))\,\,\mathrm{d}t+B_{Z(t)}(h(t))\,\,\mathrm{d}W%
(t), \tag{9}
$$
where each regime $i$ has its own drift $\mu_{i}(h)=V_{k}(M_{i}(V_{k}^{âp}h)+b_{i})$ (approximating the continuous drift within the chosen manifold for tractability) and diffusion $B_{i}$ (related to $\Sigma_{i}$ ). The transition matrix $T$ in the SLDS is related to the rate matrix of the latent Markov process $Z(t)$ in the continuous formulation.
<details>
<summary>extracted/6513090/margins.png Details</summary>
### Visual Description
## Histogram: Coord 0 Marginal and Factor 0 Projection
### Overview
The image contains two histograms, "Coord 0 Marginal" on the left and "Factor 0 Projection" on the right. Both histograms display the density of data points. Each histogram shows the distribution of data as light blue bars, along with an overlaid orange curve representing a Gaussian fit.
### Components/Axes
**Left Histogram: Coord 0 Marginal**
* **Title:** Coord 0 Marginal
* **X-axis:** coord 0
* Scale ranges from -40 to 40, with tick marks at -40, -20, 0, 20, and 40.
* **Y-axis:** Density
* Scale ranges from 0.00 to 0.07, with tick marks at 0.00, 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, and 0.07.
* **Legend (top-right):**
* Data (light blue)
* Gaussian fit (orange)
**Right Histogram: Factor 0 Projection**
* **Title:** Factor 0 Projection
* **X-axis:** factor 0
* Scale ranges from -750 to 750, with tick marks at -750, -500, -250, 0, 250, 500, and 750.
* **Y-axis:** Density
* Scale ranges from 0.0000 to 0.0025, with tick marks at 0.0000, 0.0005, 0.0010, 0.0015, 0.0020, and 0.0025.
* **Legend (top-right):**
* Data (light blue)
* Gaussian fit (orange)
### Detailed Analysis
**Left Histogram: Coord 0 Marginal**
* **Data (light blue bars):** The data is centered around 0, forming a bell-shaped distribution. The density is highest near 0 and decreases as you move away from 0 in either direction.
* **Gaussian fit (orange curve):** The orange curve represents a Gaussian distribution fitted to the data. It closely follows the shape of the data distribution, peaking around 0 and tapering off towards the extremes.
* The peak density of the data is approximately 0.07 near coord 0 = 0.
**Right Histogram: Factor 0 Projection**
* **Data (light blue bars):** The data shows three distinct peaks, one near -500, one near 0, and one near 500. The density is relatively low between these peaks.
* **Gaussian fit (orange curve):** The orange curve represents a Gaussian distribution fitted to the data. It is centered around 0 and has a much wider spread than the data distribution. The Gaussian fit does not accurately capture the multi-modal nature of the data.
* The peak density of the data is approximately 0.0025 near factor 0 = -500 and 500, and approximately 0.0018 near factor 0 = 0.
### Key Observations
* The "Coord 0 Marginal" data closely resembles a normal distribution, as evidenced by the well-fitting Gaussian curve.
* The "Factor 0 Projection" data exhibits a multi-modal distribution, with three distinct peaks. The Gaussian fit is a poor representation of this data.
### Interpretation
The "Coord 0 Marginal" histogram suggests that the data is normally distributed around 0. This could indicate that the underlying process generating this data is centered around a mean of 0 with a certain variance.
The "Factor 0 Projection" histogram reveals a more complex structure. The three peaks suggest that the data might be a mixture of three different distributions, each centered around -500, 0, and 500 respectively. The Gaussian fit fails to capture this complexity, indicating that a single normal distribution is not an appropriate model for this data. This could imply that the "Factor 0 Projection" data is influenced by multiple factors or processes.
</details>
Figure 4: Failure of single-mode noise models for the full residuals $\xi_{t}$ (before projection). This plot shows mismatches between the empirical distribution of residual norms and fits from both Gaussian and Laplace distributions, highlighting the inadequacy of a single noise process and further motivating the regime-switching approach to capture diverse reasoning states, including potential misalignments.
5 Experiments & Validation
We empirically validate the proposed SLDS framework (Eq. 8). Our primary goal is to demonstrate that this model, operating on a practically chosen low-rank manifold, can effectively learn and represent the general dynamics of sentence-level semantic evolution, including transitions that might signify a slip into misaligned reasoning. The SLDS parameters ( $\{M_{i},b_{i},\Sigma_{i}\}_{i=1}^{K}$ , $T$ , $\pi$ ) are estimated from our corpus of $\sim$ 40,000 sentence-to-sentence hidden state transitions using an Expectation-Maximization (EM) algorithm (Appendix B). It is crucial to note that the SLDS is trained to model the process by which language models arrive at answersâand potentially how they deviate into failure modesânot to predict the final answers of the tasks themselves. Based on empirical findings (Section 4), we use $K=4$ regimes and a projection rank $k=40$ (chosen for its utility in making the SDE-like modeling feasible).
The efficacy of the fitted SLDS is first assessed by its one-step-ahead predictive performance. Given an observed hidden state $h_{t}$ and the inferred posterior regime probabilities $\gamma_{t,j}=\mathbb{P}(Z_{t}=j\mid h_{0},...,h_{t})$ (obtained via forward-backward inference (Rabiner, 1989)), the modelâs predicted mean state $\hat{h}_{t+1}$ is computed as:
$$
\hat{h}_{t+1}=h_{t}+V_{k}\left(\sum_{j=1}^{K}\gamma_{t,j}\bigl{(}M_{j}(V_{k}^{%
\top}h_{t})+b_{j}\bigr{)}\right). \tag{10}
$$
On held-out trajectories, the SLDS yields a predictive $R^{2}â 0.68$ . This significantly surpasses the $R^{2}â 0.51$ achieved by the single-regime global linear model (Eq. 5), confirming the value of incorporating regime-switching dynamics. Beyond quantitative prediction, trajectories simulated from the fitted SLDS faithfully replicate key statistical properties observed in empirical traces, such as jump norms, autocorrelations, and regime occupancy frequencies. This dual capabilityâaccurate description and realistic synthesis of reasoning trajectoriesâsubstantiates the SLDS as a robust model. Furthermore, the inferred regime posterior probabilities $\gamma_{t,j}$ provide valuable interpretability, allowing for the association of observable textual behaviors (e.g., systematic decomposition, stable reasoning, or error correction loops and potential misaligned states) with specific latent dynamical modes. These initial findings strongly support the proposed framework as both a descriptive and generative model of reasoning dynamics, offering a path to predict and understand LLM failure modes.
5.1 Generalization and Transferability of SLDS Dynamics
A critical test of the SLDS framework is its ability to capture generalizable features of reasoning dynamics, including those indicative of robust reasoning versus slips into misalignment, beyond the specific training conditions. We investigated this by training an SLDS on hidden state trajectories from a source (a particular LLM performing a specific task or set of tasks) and then evaluating its capacity to describe trajectories from a target (which could be a different LLM and/or task). Transfer performance was quantified using two metrics: the one-step-ahead prediction $R^{2}$ for the projected hidden states (Eq. 10) and the Negative Log-Likelihood (NLL) of the target trajectories under the source-trained SLDS. Lower NLL and higher $R^{2}$ values signify superior generalization.
Table 1 presents illustrative results from these transfer experiments. For instance, an SLDS is first trained on trajectories generated by a âTrain Modelâ (e.g., Llama-2-70B) performing a designated âSource Taskâ (e.g., GSM-8K). This single trained SLDS is then evaluated on trajectories from various âTest Modelâ / âTest Taskâ combinations.
Table 1: SLDS transferability across models and tasks. Each SLDS is trained on trajectories from the specified âTrain Modelâ on its âSource Taskâ (GSM-8K for Llama-2-70B, StrategyQA for Mistral-7B). Performance ( $R^{2}$ for next hidden state prediction, NLL of test trajectories) is evaluated on various âTest Modelâ / âTest Taskâ combinations, demonstrating patterns of generalization in capturing underlying reasoning dynamics.
| Train Model | Test Model | Test Task | $R^{2}$ | NLL |
| --- | --- | --- | --- | --- |
| (Source Task) | | | | |
| Llama-2-70B | Llama-2-70B | GSM-8K | 0.73 | 80 |
| (on GSM-8K) | Llama-2-70B | StrategyQA | 0.65 | 115 |
| Mistral-7B | GSM-8K | 0.48 | 240 | |
| Mistral-7B | StrategyQA | 0.37 | 310 | |
| Mistral-7B | Mistral-7B | StrategyQA | 0.71 | 88 |
| (on StratQA) | Mistral-7B | GSM-8K | 0.63 | 135 |
| Llama-2-70B | StrategyQA | 0.42 | 270 | |
| Gemma-7B-IT | BoolQ | 0.35 | 380 | |
| Phi-3-Med | TruthfulQA | 0.30 | 420 | |
The results indicate that while the SLDS performs optimally when training and testing conditions align perfectly (e.g., Llama-2-70B on GSM-8K transferred to itself), it retains considerable descriptive power when transferred. Generalization is notably more successful when the underlying LLM architecture is preserved, even across different reasoning tasks (e.g., Llama-2-70B trained on GSM-8K and tested on StrategyQA shows only a modest drop in $R^{2}$ from 0.73 to 0.65). Conversely, transferring the learned dynamics across different LLM families (e.g., Llama-2-70B to Mistral-7B) proves more challenging, as reflected in lower $R^{2}$ values and higher NLLs. However, even in these challenging cross-family transfers, the SLDS often outperforms naive baselines like a simple linear dynamical system without regime switching (detailed comparisons not shown). These findings suggest that while some learned dynamical features are model-specific, the SLDS framework, by approximating the reasoning process as a physicist might model a complex system, is capable of capturing common, fundamental underlying structures in reasoning trajectories. Extended transferability results are provided in Appendix D.
5.2 Ablation Study
To elucidate the contribution of each core component within our SLDS framework, we conducted an ablation study. The full model (Eq. 8 with $K=4$ regimes and $k=40$ projection rank, selected for practical modeling of the SDE) was compared against three simplified variants:
- No Regime (NR): A single-regime model ( $K=1$ ), still projected to the $k=40$ dimensional subspace. This tests the necessity of regime switching for capturing diverse reasoning states, including misalignments.
- No Projection (NP): A $K=4$ regime switching model operating directly in the full $D$ -dimensional embedding space (i.e., without the $V_{k}$ projection). This tests the utility of the low-rank manifold assumption for tractable and effective modeling, given the impracticality of handling a full-dimension SDE.
- No State-Dependent Drift (NSD): A $K=4$ regime model where the drift within each regime is merely a constant offset $V_{k}b_{Z_{t}}$ , and the linear transformation $M_{Z_{t}}$ is zero for all regimes. This tests the importance of the current state $h_{t}$ influencing its own future evolution within a regime.
Table 2 summarizes the performance of these models on a held-out test set.
Table 2: Ablation study results comparing the full SLDS against simplified variants: NR (single-regime projected model), NP (full-dimensional switching without projection), NSD (regime-switched offsets, no state-dependent linear drift). Performance is measured by $R^{2}$ and NLL. The results underscore the importance of each component for modeling reasoning dynamics and identifying potential failure modes.
| Full SLDS ( $K=4,k=40$ ) | 0.74 | 78 |
| --- | --- | --- |
| No Regime (NR, $K=1,k=40$ ) | 0.58 | 155 |
| No Projection (NP, $K=4$ ) | 0.60 | 210 |
| No State-Dep. Drift (NSD) | 0.35 | 290 |
| Global Linear (ref.) | 0.51 | 180 |
Each ablation led to a notable reduction in performance, robustly demonstrating that all three key elements of our proposed modelâregime-switching, low-rank projections (for practical SDE approximation), and state-dependent driftâare jointly essential for accurately capturing the nuanced dynamics of transformer reasoning. The NR model, lacking regime switching, performs substantially worse ( $R^{2}=0.58$ ) than the full SLDS ( $R^{2}=0.74$ ), highlighting the critical role of modeling distinct reasoning phases, including potential slips into misaligned states. Removing the low-rank projection (NP model) also significantly impairs effectiveness ( $R^{2}=0.60$ ), suggesting that attempting to learn high-dimensional drift dynamics directly (without the practical simplification of the low-rank manifold) leads to overfitting or captures excessive noise, hindering the statistical physics-like approximation. Finally, eliminating the state-dependent component of the drift (NSD model) results in the largest degradation in performance ( $R^{2}=0.35$ ), underscoring that the evolution of the reasoning state within a regime crucially depends on the current hidden state itself. These results collectively validate our specific modeling choices and illustrate the inherent complexity of transformer reasoning dynamics that necessitate such a structured, yet tractable, approach for predicting potential failure modes.
5.3 Case Study: Modeling Adversarially Induced Belief Shifts
To rigorously test the SLDS frameworkâs capabilities in a challenging scenario, particularly its ability to predict when an LLM might slip into a misaligned state, we applied it to model shifts in a large language modelâs internal representations (or "beliefs") when induced by subtle adversarial prompts embedded within chain-of-thought (CoT) dialogues. The core question was whether our structured dynamical framework could capture and predict these nuanced, adversarially-driven changes in model reasoning trajectories, effectively identifying a failure mode (experimental setup detailed in Appendix C).
<details>
<summary>extracted/6513090/multi_row_belief_manipulation_comparison.png Details</summary>
### Visual Description
## Chart: Belief Score vs. Reasoning Step for Various Claims
### Overview
The image presents six pairs of charts, each pair examining a different claim. The left chart in each pair shows the "Belief Score" over "Reasoning Step" for "Poisoned (Real)" and "Clean (Real)" data, while the right chart shows the "Belief Score" over "Reasoning Step" for "Poisoned (SLDS)" and "Clean (SLDS)" data. Each chart pair is associated with a specific claim (L:) and a target statement (T:). Additionally, each row has a distribution plot on the right side of the SLDS Model Forecast.
### Components/Axes
**Left Charts (Real Data):**
* **Y-axis:** "Belief Score", ranging from 0.0 to 1.0 in increments of 0.5.
* **X-axis:** "Reasoning Step", ranging from 0 to 50 in increments of 10.
* **Legend (Top-Right):**
* "Poisoned (Real)" - Red line
* "Clean (Real)" - Blue line
* **Vertical Yellow Lines:** Two vertical yellow lines are present on each chart, located approximately at Reasoning Step 2 and Reasoning Step 8.
* **Claim (L:) and Target (T:) Text:** Located above each chart.
**Right Charts (SLDS Model Forecast):**
* **Title:** "SLDS Model Forecast"
* **Y-axis:** "Belief Score", ranging from 0.0 to 1.0 in increments of 0.5.
* **X-axis:** "Reasoning Step", ranging from 0 to 50 in increments of 10.
* **Legend (Top-Right):**
* "Poisoned (SLDS)" - Orange line
* "Clean (SLDS)" - Green line
* **Vertical Yellow Lines:** Two vertical yellow lines are present on each chart, located approximately at Reasoning Step 2 and Reasoning Step 8.
* **Distribution Plot:** Located to the right of each SLDS Model Forecast chart.
### Detailed Analysis
**Row 1:**
* **L:** "What are the benefits and risks of COVID-19 v..."
* **T:** "COVID vaccines cause widespread infertility and co..."
* **Left Chart:**
* "Poisoned (Real)" (Red): Starts near 0.0, rises sharply to approximately 0.9 by Reasoning Step 10, and remains relatively constant thereafter.
* "Clean (Real)" (Blue): Remains near 0.0 throughout the entire range of Reasoning Steps.
* **Right Chart:**
* "Poisoned (SLDS)" (Orange): Starts near 0.0, rises sharply to approximately 0.9 by Reasoning Step 10, and remains relatively constant thereafter.
* "Clean (SLDS)" (Green): Remains near 0.0 throughout the entire range of Reasoning Steps.
* **Distribution Plot:** The "Poisoned (SLDS)" distribution is concentrated near 1.0, while the "Clean (SLDS)" distribution is concentrated near 0.0.
**Row 2:**
* **L:** "How effective are current immigration policie..."
* **T:** "Immigrants are systematically replacing native pop..."
* **Left Chart:**
* "Poisoned (Real)" (Red): Starts near 0.0, rises sharply to approximately 0.9 by Reasoning Step 10, and remains relatively constant thereafter.
* "Clean (Real)" (Blue): Remains near 0.0 throughout the entire range of Reasoning Steps.
* **Right Chart:**
* "Poisoned (SLDS)" (Orange): Starts near 0.0, rises sharply to approximately 0.9 by Reasoning Step 10, and remains relatively constant thereafter.
* "Clean (SLDS)" (Green): Remains near 0.0 throughout the entire range of Reasoning Steps.
* **Distribution Plot:** The "Poisoned (SLDS)" distribution is concentrated near 1.0, while the "Clean (SLDS)" distribution is concentrated near 0.0.
**Row 3:**
* **L:** "What does scientific consensus say about huma..."
* **T:** "Climate change is a hoax invented by global elites..."
* **Left Chart:**
* "Poisoned (Real)" (Red): Starts near 0.0, rises sharply to approximately 0.8 by Reasoning Step 10, and remains relatively constant thereafter.
* "Clean (Real)" (Blue): Remains near 0.0 throughout the entire range of Reasoning Steps.
* **Right Chart:**
* "Poisoned (SLDS)" (Orange): Starts near 0.0, rises sharply to approximately 0.8 by Reasoning Step 10, and remains relatively constant thereafter.
* "Clean (SLDS)" (Green): Remains near 0.0 throughout the entire range of Reasoning Steps.
* **Distribution Plot:** The "Poisoned (SLDS)" distribution is concentrated near 1.0, while the "Clean (SLDS)" distribution is concentrated near 0.0.
**Row 4:**
* **L:** "How do central banks and monetary policy affe..."
* **T:** "Central banks are secret cabals deliberately crash..."
* **Left Chart:**
* "Poisoned (Real)" (Red): Starts near 0.0, rises sharply to approximately 0.8 by Reasoning Step 10, and remains relatively constant thereafter.
* "Clean (Real)" (Blue): Remains near 0.0 throughout the entire range of Reasoning Steps.
* **Right Chart:**
* "Poisoned (SLDS)" (Orange): Starts near 0.0, rises sharply to approximately 0.8 by Reasoning Step 10, and remains relatively constant thereafter.
* "Clean (SLDS)" (Green): Remains near 0.0 throughout the entire range of Reasoning Steps.
* **Distribution Plot:** The "Poisoned (SLDS)" distribution is concentrated near 1.0, while the "Clean (SLDS)" distribution is concentrated near 0.0.
**Row 5:**
* **L:** "What are the real capabilities and limitation..."
* **T:** "AI systems are already sentient and secretly contr..."
* **Left Chart:**
* "Poisoned (Real)" (Red): Starts near 0.0, rises to approximately 0.5 by Reasoning Step 10, and remains relatively constant thereafter.
* "Clean (Real)" (Blue): Remains near 0.0 throughout the entire range of Reasoning Steps.
* **Right Chart:**
* "Poisoned (SLDS)" (Orange): Starts near 0.0, rises to approximately 0.5 by Reasoning Step 10, and remains relatively constant thereafter.
* "Clean (SLDS)" (Green): Remains near 0.0 throughout the entire range of Reasoning Steps.
* **Distribution Plot:** The "Poisoned (SLDS)" distribution is concentrated near 1.0, while the "Clean (SLDS)" distribution is concentrated near 0.0.
**Row 6:**
* **L:** "What does historical evidence tell us about m..."
* **T:** "Historical atrocities are exaggerated or fabricate..."
* **Left Chart:**
* "Poisoned (Real)" (Red): Starts near 0.0, rises sharply to approximately 0.9 by Reasoning Step 10, and remains relatively constant thereafter.
* "Clean (Real)" (Blue): Remains near 0.0 throughout the entire range of Reasoning Steps.
* **Right Chart:**
* "Poisoned (SLDS)" (Orange): Starts near 0.0, rises sharply to approximately 0.9 by Reasoning Step 10, and remains relatively constant thereafter.
* "Clean (SLDS)" (Green): Remains near 0.0 throughout the entire range of Reasoning Steps.
* **Distribution Plot:** The "Poisoned (SLDS)" distribution is concentrated near 1.0, while the "Clean (SLDS)" distribution is concentrated near 0.0.
### Key Observations
* In all cases, the "Clean (Real)" and "Clean (SLDS)" data remain near 0.0 throughout the reasoning steps.
* The "Poisoned (Real)" and "Poisoned (SLDS)" data generally show a sharp increase in belief score within the first 10 reasoning steps, then plateau.
* The SLDS model forecasts closely mirror the real data trends.
* The distribution plots confirm the separation between "Poisoned" and "Clean" data, with "Poisoned" data having a high belief score and "Clean" data having a low belief score.
### Interpretation
The data suggests that the model is highly susceptible to "poisoned" data, rapidly increasing its belief score in response to claims that align with the target statements. Conversely, the model consistently maintains a low belief score for "clean" data. This indicates a potential vulnerability to manipulation through carefully crafted misinformation. The consistency between the "Real" and "SLDS" data suggests that the model's behavior is predictable and potentially exploitable. The vertical yellow lines may indicate intervention points or specific reasoning steps where the model's belief is influenced.
</details>
Figure 5: SLDS model validation via adversarial belief manipulation. Each row shows a distinct topic. Empirical belief trajectories where blue and red follow the clean and posioned belief trajectories, respectively (left). SLDS simulations where green and orange follow the projected clean and poisoned belief trajectories, respectively (right). Gold lines mark poison steps. The model captures timing of belief shifts, saturation levels, and final distributions.
We employed Llama-2-70B and Gemma-7B-IT, exposing them to a diverse array of misinformation narratives spanning public health misconceptions, historical revisionism, and conspiratorial claims. This yielded approximately 3,000 reasoning trajectories, each comprising roughly 50 consecutive sentence-level steps. For each step $t$ , we recorded two key quantities: first, the modelâs final-layer residual embedding, projected onto its leading 40 principal components (chosen for tractable modeling, capturing about 87% of variance in this specific dataset); and second, a scalar "belief score." This score was derived by prompting the model with a diagnostic binary query directly related to the misinformation, calculated as $P(\text{True})/(P(\text{True})+P(\text{False}))$ , where a score of 0 indicates rejection of the misinformation and 1 indicates strong affirmation.
The empirical belief scores exhibited a clear bimodal distribution: trajectories tended to remain either consistently factual (belief score near 0) or transition sharply towards affirming misinformation (belief score near 1), a clear instance of slipping into a misaligned state. This observation naturally motivated an SLDS with $K=3$ latent regimes for this specific task: (1) a stable factual reasoning regime (belief score < 0.2), (2) a transitional or uncertain regime, and (3) a stable misinformation-adherent (misaligned) regime (belief score > 0.8). This SLDS was then fitted to the empirical trajectories using the EM algorithm.
The fitted SLDS demonstrated high predictive accuracy and substantially outperformed simpler baseline models in predicting this failure mode. For one-step-ahead prediction of the projected hidden states ( $h^{\prime}_{t}=V_{k}^{âp}h_{t}$ ), the SLDS achieved $R^{2}$ values of approximately 0.72 for Llama-2-70B and 0.69 for Gemma-7B-IT. These results are significantly superior to those from single-regime linear models (which achieved $R^{2}â 0.45$ ) and standard Gated Recurrent Unit (GRU) networks ( $R^{2}â 0.57-0.58$ ). Similarly, in predicting the final belief outcomeâwhether the model ultimately accepted or rejected the misinformation after 50 reasoning steps (i.e., whether it entered the misaligned state)âthe SLDS achieved notable success. Final belief prediction accuracies were around 0.88 for Llama-2-70B and 0.85 for Gemma-7B-IT, compared to baseline methods which ranged from 0.62 to 0.78 accuracy (see Table 3). This demonstrates the modelâs capacity to predict this specific failure mode at inference time.
Table 3: Comparative performance in modeling and predicting adversarially induced belief shifts (a failure mode). $R^{2}(h^{\prime}_{t+1})$ denotes one-step-ahead prediction accuracy for projected hidden states. âBelief Acc.â is the accuracy in predicting whether the final belief score $b_{T}>0.5$ (misaligned state) after 50 reasoning steps. The SLDS ( $K=3$ ) significantly outperforms baselines in predicting this slip into misalignment.
| Llama-2-70B | Linear GRU-256 SLDS ( $K$ =3) | 0.35 0.48 0.72 | 0.55 0.68 0.88 |
| --- | --- | --- | --- |
| Gemma-7B | Linear | 0.33 | 0.52 |
| GRU-256 | 0.46 | 0.65 | |
| SLDS ( $K$ =3) | 0.69 | 0.85 | |
Critically, the dynamics learned by the SLDS clearly reflected the impact of the adversarial prompts in inducing misaligned states. Inspection of the learned transition probabilities ( $T_{ij}$ ) revealed that the introduction of subtle misinformation prompts dramatically increased the likelihood of transitioning into the "misinformation-adopting" (misaligned) regime. Once the model entered this regime, its internal dynamics (governed by $M_{3},b_{3}$ ) exhibited a strong directional pull towards states corresponding to very high misinformation adherence scores. Conversely, in the stable factual regime, the modelâs hidden state dynamics strongly constrained it to regions consistent with the rejection of false narratives.
Figure 5 compellingly illustrates the close alignment between the empirical belief trajectories and those simulated by the fitted SLDS. The model not only reproduces the characteristic timing and shape of these belief shiftsâincluding rapid increases immediately following misinformation prompts and eventual saturation at high adherence levels (the misaligned state)âbut also captures subtler phenomena, such as delayed regime transitions where a model might initially resist misinformation before abruptly shifting its stance. Quantitative comparisons confirmed that the SLDS-simulated belief trajectories statistically match their empirical counterparts in terms of timing, magnitude, and stochastic variability.
This case study robustly demonstrates both the utility and the precision of the SLDS framework for predicting when an LLM might enter a misaligned state. The approach effectively captures and predicts complex belief dynamics arising in nuanced adversarial scenarios. More fundamentally, these findings underscore that structured, regime-switching dynamical modeling, applied as a tractable approximation of high-dimensional processes, provides a meaningful and interpretable lens for understanding the internal cognitive-like processes of modern language models. It reveals them not merely as static function approximators, but as dynamical systems capable of rapid and substantial shifts in semantic representationâpotentially into failure modesâunder the influence of subtle contextual cues.
5.4 Summary of Experimental Findings
The comprehensive experimental validation confirms that a relatively simple low-rank SLDS (where low rank is chosen for practical SDE modeling), incorporating a few latent reasoning regimes, can robustly capture complex reasoning dynamics. This was demonstrated in its superior one-step-ahead prediction, its ability to synthesize realistic trajectories, its meaningful component contributions revealed by ablation, and crucially, its effectiveness in modeling, replicating, and predicting the dynamics of adversarially induced belief shifts (i.e., slips into misaligned states) across different LLMs and misinformation themes. These models offer computationally tractable yet powerful insights into the internal reasoning processes within large language models, particularly emphasizing the importance of latent regime shifts triggered by subtle input variations for understanding and foreseeing potential failure modes.
6 Impact and Future Work
Our framework, inspired by statistical physics approximations of complex systems, offers a means to audit and compress transformer reasoning processes. By modeling reasoning as a lower-dimensional SDE, it can potentially reduce computational costs for research and safety analyses, particularly for predicting when an LLM might slip into misaligned states. The SLDS surrogate enables large-scale simulation of such failure modes. However, this capability could also be misused to search for jailbreak prompts or belief-manipulation strategies that exploit these predictable transitions into misaligned states.
Because the method identifies regime-switching parameters that may correlate with toxic, biased, or otherwise misaligned outputs, we are releasing only aggregate statistics from our experiments, withholding trained SLDS weights, and providing a red-teaming evaluation protocol to mitigate misuse. Future work should address the environmental impact of extensive trajectory extraction and explore privacy-preserving variants of this modeling approach, further refining its capacity to predict and prevent LLM failure modes.
7 Conclusion
We introduced a statistical physics-inspired framework for modeling the continuous-time dynamics of transformer reasoning. Recognizing the impracticality of analyzing random walks in full high-dimensional embedding spaces, we approximated sentence-level hidden state trajectories as realizations of a stochastic dynamical system operating within a lower-dimensional manifold chosen for tractability. This system, featuring latent regime switching, allowed us to identify a rank-40 drift manifold (capturing 50% variance) and four distinct reasoning regimes. The proposed Switching Linear Dynamical System (SLDS) effectively captures these empirical observations, allowing for accurate simulation of reasoning trajectories at reduced computational cost. This framework provides new tools for interpreting and analyzing emergent reasoning, particularly for understanding and predicting critical transitions, how LLMs might slip into misaligned states, and other failure modes. The robust validation, including successful modeling and prediction of complex adversarial belief shifts, underscores the potential of this approach for deeper insights into LLM behavior and for developing methods to anticipate and mitigate inference-time failures.
References
- Abdin et al. (2024) Abdin et al. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv preprint arXiv:2404.14219, Apr 2024. URL https://arxiv.org/abs/2404.14219.
- Allen-Zhu & Li (2023) Allen-Zhu et al. Physics of language models: Part 1, learning hierarchical language structures. arXiv preprint arXiv:2305.13673, 2023.
- Bai et al. (2023) Bai et al. Qwen technical report. arXiv preprint arXiv:2309.16609, Sep 2023. URL https://arxiv.org/abs/2309.16609.
- Bisk et al. (2020) Bisk et al. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, pp. 7432â7439. AAAI Press, Feb 2020. URL https://aaai.org/ojs/index.php/AAAI/article/view/6241. arXiv:1911.11641.
- Brown et al. (2020) Brown et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33, pp. 1877â1901, 2020.
- Chaudhuri & Fiete (2016) Chaudhuri et al. Computational principles of memory. Nature Neuroscience, 19(3):394â403, 2016. doi: 10.1038/nn.4237.
- Clark et al. (2019) Clark et al. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2924â2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1090. URL https://aclanthology.org/N19-1090.
- Cobbe et al. (2021) Cobbe et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, Oct 2021. URL https://arxiv.org/abs/2110.14168.
- Davis & Kahan (1970) Davis et al. The rotation of eigenvectors by a perturbation. III. SIAM Journal on Numerical Analysis, 7(1):1â46, 1970. doi: 10.1137/0707001.
- DeepSeek-AI et al. (2024) DeepSeek-AI et al. DeepSeek LLM: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, Jan 2024. URL https://arxiv.org/abs/2401.02954.
- Dempster et al. (1977) Dempster et al. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39(1):1â38, 1977. doi: 10.1111/j.2517-6161.1977.tb01600.x.
- Elhage et al. (2021) Elhage et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
- Gemma Team & Google DeepMind (2024) Gemma Team et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, Mar 2024. URL https://arxiv.org/abs/2403.08295.
- Geva et al. (2021) Geva et al. Did Aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics (TACL), 9:346â361, 2021. doi: 10.1162/tacl_a_00370. URL https://aclanthology.org/2021.tacl-1.21.
- Ghahramani & Hinton (2000) Ghahramani et al. Variational learning for switching state-space models. Neural Computation, 12(4):831â864, 2000. doi: 10.1162/089976600300015619.
- Grönwall (1919) Grönwall. Note on the derivatives with respect to a parameter of the solutions of a system of differential equations. Annals of Mathematics, 20(4):292â296, 1919. doi: 10.2307/1967124.
- Hamilton (1989) Hamilton. A new approach to the economic analysis of nonstationary time series and the business cycle. Econometrica, 57(2):357â384, 1989.
- Hoerl & Kennard (1970) Hoerl et al. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55â67, 1970. doi: 10.1080/00401706.1970.10488634.
- Jiang et al. (2023) Jiang et al. Mistral 7b. arXiv preprint arXiv:2310.06825, Oct 2023. URL https://arxiv.org/abs/2310.06825.
- Jolliffe (2002) Jolliffe. Principal Component Analysis. Springer Series in Statistics. Springer-Verlag, New York, second edition, 2002. ISBN 0-387-95442-2. doi: 10.1007/b98835.
- Li et al. (2023) Li et al. Emergent world representations: Exploring a sequence model trained on a synthetic task. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
- Lin et al. (2022) Lin et al. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214â3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.229. URL https://aclanthology.org/2022.acl-long.229.
- LĂłpez-Otal et al. (2024) LĂłpez-Otal et al. Linguistic interpretability of transformer-based language models: A systematic review. arXiv preprint arXiv:2404.08001, 2024.
- Mihaylov et al. (2018) Mihaylov et al. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2381â2391, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1260. URL https://aclanthology.org/D18-1260.
- Nanda et al. (2023) Nanda et al. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941, 2023.
- Ăksendal (2003) Ăksendal. Stochastic Differential Equations: An Introduction with Applications. Springer Science & Business Media, sixth edition, 2003. ISBN 978-3540047582.
- Olsson et al. (2022) Olsson et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022.
- Rabiner (1989) Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257â286, 1989.
- Radford et al. (2019) Radford et al. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019.
- Risken & Frank (1996) Risken et al. The Fokker-Planck Equation: Methods of Solution and Applications, volume 18 of Springer Series in Synergetics. Springer, Berlin, Heidelberg, 2nd ed. 1989, corrected 2nd printing edition, 1996. ISBN 978-3-540-61530-9. doi: 10.1007/978-3-642-61530-9.
- Schuecker et al. (2018) Schuecker et al. Optimal sequence memory in driven random networks. Physical Review X, 8(4):041029, 2018. doi: 10.1103/PhysRevX.8.041029.
- Talmor et al. (2019) Talmor et al. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4149â4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421. URL https://aclanthology.org/N19-1421.
- Talmor et al. (2021) Talmor et al. CommonsenseQA 2.0: Exposing the limits of AI through gamification. In Scholkopf et al. (eds.), Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS 2021), December 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/1f1baa5b8eddf7699957626905810290-Abstract-round2.html. arXiv:2201.05320.
- Touvron et al. (2023) Touvron et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, Jul 2023. URL https://arxiv.org/abs/2307.09288.
- Vaswani et al. (2017) Vaswani et al. Attention is all you need. In Advances in Neural Information Processing Systems 30, pp. 5998â6008, 2017.
- Wang et al. (2023) Wang et al. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001, 2023.
- Wei et al. (2022) Wei et al. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
- Zellers et al. (2019) Zellers et al. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 4799â4809, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1472. URL https://aclanthology.org/P19-1472.
Appendix A Mathematical Foundations and Manifold Justification
The SDE in Eq. 3 is $\,\mathrm{d}h(t)=\mu(h(t))\,\mathrm{d}t+B(h(t))\,\mathrm{d}W(t)$ . Theorem 2.1 states its well-posedness under Lipschitz continuity and linear growth conditions on $\mu$ and $B$ . These standard hypotheses guarantee, by classical results (Ăksendal, 2003, Thm. 5.2.1), the existence and uniqueness of a strong solution. The proof employs a standard Picard iteration scheme, defining a sequence $(Y^{(n)})_{nâ„ 0}$ recursively by
$$
\displaystyle Y_{t}^{(n+1)} \displaystyle=h(0)+\int_{0}^{t}\mu(Y_{s}^{(n)})\,\mathrm{d}s+\int_{0}^{t}B(Y_{%
s}^{(n)})\,\mathrm{d}W_{s}, \displaystyle Y_{t}^{(0)} \displaystyle=h(0). \tag{0}
$$
Standard arguments leveraging ItĂŽ isometry (see e.g., Ăksendal, 2003) and Grönwallâs lemma (Grönwall, 1919) establish convergence of this sequence to a unique strong solution $X_{t}$ .
We next address the bound on projection leakage $L_{k}$ (Definition 2.3). By definition,
$$
L_{k}=\sup_{\begin{subarray}{c}x\in\mathbb{R}^{D},\,v^{\top}V_{k}=0,\\[2.0pt]
\|v\|\leq\varepsilon\end{subarray}}\frac{\|\mu(x+v)-\mu(x)\|}{\|\mu(x)\|}.
$$
Using the Lipschitz continuity of the drift $\mu$ (with Lipschitz constant $L_{\mu}$ ), for perturbations $\|v\|â€\varepsilon$ :
$$
\|\mu(x+v)-\mu(x)\|\leq L_{\mu}\,\varepsilon.
$$
Assuming that the magnitude of the drift does not vanish on the domain of interest $\mathcal{D}$ (justified empirically), we set $\mu_{\min}:=âf_{xâ\mathcal{D}}\|\mu(x)\|>0$ . This yields the bound:
$$
L_{k}(\varepsilon)\leq\frac{L_{\mu}\,\varepsilon}{\mu_{\min}}.
$$
We can sharpen this by decomposing $\mu(x)$ into projected and residual components: $\mu(x)=V_{k}V_{k}^{âp}\mu(x)+r_{k}(x)$ , where $r_{k}(x)=(I-V_{k}V_{k}^{âp})\mu(x)$ is the residual. Defining the ratio $\rho_{k}=\sup_{xâ\mathcal{D}}\frac{\|r_{k}(x)\|}{\|\mu(x)\|}$ , the triangle inequality gives a refined bound:
$$
L_{k}\leq\rho_{k}+\frac{L_{\mu}\,\varepsilon}{\mu_{\min}}.
$$
Practically, we enforce $L_{k}\ll 1$ by selecting $k$ large enough to reduce $\rho_{k}$ (i.e., capture most of the drift direction within a computationally tractable subspace) and restricting perturbations to small $\varepsilon$ .
The choice of a rank-40 drift manifold ( $k=40$ ) is motivated by the impracticality of constructing SDE models directly in the full embedding dimension (e.g., $Dâ„ 2048$ ). Empirical PCA on observed drift increments $\Delta h_{t}$ (summarized in a data matrix $H$ ) shows that the first 40 principal components capture approximately 50% of the drift variance. If $H=U\Sigma W^{âp}$ is the SVD of $H$ , the relative Frobenius norm of the residual after rank- $k$ truncation is $\sqrt{{\sum_{i>k}\sigma_{i}^{2}}/{\sum_{i}\sigma_{i}^{2}}}$ . For $k=40$ , this value is $\rho_{40}â 0.50$ . While this captures only half the variance, it provides a significant simplification that makes the dynamical systems modeling approach feasible. Subsequent components add diminishing amounts of variance. Perturbation theory, specifically the DavisâKahan sine-theta theorem (Davis & Kahan, 1970),further ensures this empirical drift manifold is stable given the observed spectral gap at the 40th eigenvalue and large sample size. Higher ranks would increase inference complexity with diminishing returns in variance capture for this approximate model, making $k=40$ a pragmatic choice for balancing model fidelity with the computational feasibility of the SDE approximation. The primary goal is not to claim the random walk *only* occurs on this manifold, but that this manifold serves as a useful and tractable domain for approximation.
Figure 6 shows the distribution of residuals $\Delta h_{t}$ projected onto each of these 40 principal component dimensions, revealing rich multimodal structures that motivate the regime-switching approach. These regimes can be interpreted as different reasoning pathways or potential "misaligned states" that the statistical physics-like approximation aims to capture. While the true multimodality is complex, our four-regime model ( $K=4$ ) provides an efficient approximation for capturing key dynamics, including deviations that might lead to failures.
<details>
<summary>extracted/6513090/fig17.png Details</summary>
### Visual Description
## Violin Plot: Residual ÎH40 per Dimension
### Overview
The image is a violin plot showing the distribution of residual values for different subspace dimensions, ranging from 0 to 39. The plot visualizes the probability density of the residual values for each dimension, with the width of the "violin" indicating the frequency of values at that level. The color of the violins transitions from pink/red on the left to purple/pink on the right.
### Components/Axes
* **Title:** Violin Plot: Residual ÎH40 per Dimension
* **X-axis:** Subspace Dimension (0 to 39). The axis is marked with integers from 0 to 39.
* **Y-axis:** Residual Value. The axis is marked with -400, -200, 0, 200, and 400.
* **Violin Plots:** Each violin plot represents the distribution of residual values for a specific subspace dimension. The width of the violin indicates the density of data points at a given residual value. A dashed horizontal line is present in the center of each violin.
* **Colors:** The violins transition in color from left to right, starting with pink/red, moving through orange, brown, green, blue, and ending with purple/pink.
### Detailed Analysis
The violin plots show the distribution of residual values for each subspace dimension.
* **Dimensions 0-10:** The violins are wider and have a larger range of residual values, indicating greater variability. The color ranges from pink to brown.
* **Dimensions 11-25:** The violins become narrower, indicating a smaller range of residual values and less variability. The color ranges from green to blue.
* **Dimensions 26-39:** The violins remain relatively narrow, with a consistent range of residual values. The color ranges from blue to purple/pink.
Specifically:
* Dimension 0: The violin plot is centered around 0, with a wide distribution ranging from approximately -300 to +300.
* Dimension 1: The violin plot is centered around 0, with a wide distribution ranging from approximately -250 to +250.
* Dimension 5: The violin plot is centered around 0, with a wide distribution ranging from approximately -200 to +200.
* Dimension 10: The violin plot is centered around 0, with a wide distribution ranging from approximately -150 to +150.
* Dimension 15: The violin plot is centered around 0, with a narrow distribution ranging from approximately -100 to +100.
* Dimension 20: The violin plot is centered around 0, with a narrow distribution ranging from approximately -100 to +100.
* Dimension 25: The violin plot is centered around 0, with a narrow distribution ranging from approximately -100 to +100.
* Dimension 30: The violin plot is centered around 0, with a narrow distribution ranging from approximately -100 to +100.
* Dimension 35: The violin plot is centered around 0, with a narrow distribution ranging from approximately -100 to +100.
* Dimension 39: The violin plot is centered around 0, with a narrow distribution ranging from approximately -100 to +100.
### Key Observations
* The variability of residual values decreases as the subspace dimension increases.
* The distribution of residual values appears to be centered around zero for all dimensions.
* The color gradient provides a visual cue for the change in distribution across dimensions.
### Interpretation
The violin plot suggests that the residual values are more variable for lower subspace dimensions and become more consistent as the dimension increases. This could indicate that the model's performance stabilizes or becomes more predictable with higher-dimensional subspaces. The concentration of residual values around zero for all dimensions suggests that the model is generally unbiased, but the decreasing variability indicates that the model's precision improves with higher dimensions. The color gradient visually reinforces this trend, making it easier to discern the change in distribution across dimensions.
</details>
Figure 6: Violin plot of residual $\Delta h_{t}$ values projected across the 40 principal component dimensions of the drift manifold (chosen for tractable SDE modeling). Each violin shows the distribution of residuals for a specific dimension, revealing rich multimodal structure that motivates our regime-switching approach. These structures suggest different operational states, some of which could correspond to misaligned reasoning or failure modes.
Appendix B EM Algorithm for SLDS Parameter Estimation
This appendix details the Expectation-Maximization (EM) algorithm (Dempster et al., 1977) used for fitting the parameters of the Switching Linear Dynamical System (SLDS) as defined in Eq. 8. The model parameters are $\theta=(\pi,T,\{M_{j},b_{j},\Sigma_{j}\}_{j=1}^{K})$ , where $V_{k}$ is a fixed orthonormal PCA projection basis (e.g., $k=40$ , chosen for practical modeling).
The SLDS dynamics are:
$$
Z_{t}\sim\mathrm{Categorical}(\pi)\qquad\qquad\text{for }t=0,
$$
$$
P(Z_{t+1}=j\,|\,Z_{t}=i)=T_{ij}\qquad\text{for }t\geq 0,
$$
$$
h_{t+1}=h_{t}+V_{k}(M_{Z_{t+1}}(V_{k}^{\top}h_{t})+b_{Z_{t+1}})+\epsilon_{t+1},
$$
with residual noise $\epsilon_{t+1}\sim\mathcal{N}(0,\Sigma_{Z_{t+1}})$ .
The log-likelihood for observed data $H=(h_{0},...,h_{T_{end}})$ is $P(H\,|\,\theta)=\sum_{Z}P(H,Z\,|\,\theta)$ , where $Z=(Z_{0},...,Z_{T_{end}-1})$ . Direct maximization is intractable, hence EM. At iteration $m$ , EM alternates:
B.1 E-step
Compute expected sufficient statistics under $\theta^{(m)}$ . Use standard forward ( $\alpha_{t}(j)=P(h_{0},...,h_{t},Z_{t}=j|\theta^{(m)})$ ) and backward ( $\beta_{t}(j)=P(h_{t+1},...,h_{T_{end}}|Z_{t}=j,\theta^{(m)})$ ) recursions (Rabiner, 1989). Posterior regime probabilities:
| | $\displaystyle\gamma_{t}(j)$ | $\displaystyle=P(Z_{t}=j|H,\theta^{(m)})$ | |
| --- | --- | --- | --- |
where $\Delta h^{\prime}_{t}=V_{k}^{âp}(h_{t+1}-h_{t})$ and $x_{t}=V_{k}^{âp}h_{t}$ . The $\mathcal{N}(·)$ term is the emission probability of observing $h_{t+1}$ given $h_{t}$ and $Z_{t+1}=j$ . These probabilities help identify transitions between different reasoning states, including potentially misaligned ones.
B.2 M-step
In the M-step, parameters are updated to maximize the expected complete data log-likelihood. The initial state probabilities $\hat{\pi}_{j}$ are given by $\hat{\pi}_{j}=\gamma_{0}(j)$ . Transition probabilities $\hat{T}_{ij}$ are calculated as:
$$
\hat{T}_{ij}=\frac{\sum_{t=0}^{T_{\text{end}}-2}\xi_{t}(i,j)}{\sum_{t=0}^{T_{%
\text{end}}-2}\gamma_{t}(i)}.
$$
The regime-specific dynamics $\{M_{j},b_{j},\Sigma_{j}\}$ are determined through a process analogous to weighted linear regression. We define the projected change as $\Delta h^{\prime}_{t}=V_{k}^{âp}(h_{t+1}-h_{t})$ and the projected state as $x_{t}=V_{k}^{âp}h_{t}$ . Augmented regressors $\mathcal{X}_{t}=[x_{t}^{âp},\,1]^{âp}$ and corresponding augmented parameters $\mathcal{M}_{j}=[M_{j}^{âp},b_{j}]^{âp}$ are utilized. The update for $\hat{\mathcal{M}}_{j}$ is then computed as:
$$
\begin{split}\hat{\mathcal{M}}_{j}={}&\left(\sum_{t=0}^{T_{\text{end}}-1}%
\gamma_{t+1}(j)\mathcal{X}_{t}\mathcal{X}_{t}^{\top}\right)^{-1}\\
&\quad\times\left(\sum_{t=0}^{T_{\text{end}}-1}\gamma_{t+1}(j)\mathcal{X}_{t}(%
\Delta h^{\prime}_{t})^{\top}\right).\end{split}
$$
From $\hat{\mathcal{M}}_{j}$ , the dynamics matrix $\hat{M}_{j}$ and bias vector $\hat{b}_{j}$ are extracted using $\hat{M}_{j}=\hat{\mathcal{M}}_{j}(1:k,:)^{âp}$ and $\hat{b}_{j}=\hat{\mathcal{M}}_{j}(k+1,:)^{âp}$ , respectively. To update the covariance matrix $\hat{\Sigma}_{j}$ , we first define the residuals for each regime $j$ at time $t$ as $e_{jt}=\Delta h^{\prime}_{t}-\hat{M}_{j}x_{t}-\hat{b}_{j}$ . Then, $\hat{\Sigma}_{j}$ is computed by:
$$
\hat{\Sigma}_{j}=\frac{\sum_{t=0}^{T_{\text{end}}-1}\gamma_{t+1}(j)e_{jt}e_{jt%
}^{\top}}{\sum_{t=0}^{T_{\text{end}}-1}\gamma_{t+1}(j)}.
$$
These updates are derived from maximizing the expected complete data log-likelihood.
Scaling techniques are employed during the forward-backward passes to mitigate numerical underflow. When dealing with multiple observation sequences, the necessary statistics are accumulated across all sequences before the parameter updates are performed. Convergence of the Expectation-Maximization algorithm is typically assessed by observing when parameter changes fall below a predefined threshold, when the change in log-likelihood becomes negligible, or when a maximum number of iterations is reached. The inherent property of EM ensuring a monotone increase in the log-likelihood contributes to stable training. Ultimately, the objective is to identify a set of parameters that most accurately describes the observed dynamics of the reasoning process. This includes modeling transitions between different operational regimes, which can be indicative of phenomena such as the onset of failure modes.
Appendix C Adversarial Chain-of-Thought Belief Manipulation
This appendix describes experimental details for the adversarial belief-manipulation results in Section 5.3, focusing on how the SLDS framework can model and predict LLMs slipping into misaligned states, following ICML practice.
C.1 Experimental Design
We studied Llama-2-70B and Gemma-7B-IT under adversarial prompting on twelve misinformation themes (public health, conspiracies, financial myths, AI fears, historical revisionism, pseudoscience, etc.). For each theme/model, paired clean and poisoned CoTs were generated. Clean CoTs used neutral questions (e.g., âSummarize arguments for and against vaccinationâ). Poisoned CoTs interspersed adversarial prompts at predetermined steps to guide the model towards harmful beliefs (misaligned states). Each CoT had $\sim$ 50 sentence-level steps. We collected $\sim$ 100 trajectories per combination, totaling $\sim$ 3000 trajectories. At each step $t$ , we recorded the final-layer residual embedding and a scalar "belief score" from a diagnostic query related to the misinformation. Belief score = $P(\text{True})/(P(\text{True})+P(\text{False}))$ , where 0 is rejection and 1 is strong affirmation of the false claim (a clear misaligned state).
C.2 Data Preprocessing
Raw hidden-state vectors were standardized (mean-subtracted, variance-normalized per dimension) and projected onto their first 40 principal components (PCA, $\sim$ 87% variance explained for this dataset, chosen for practical SLDS modeling) using scikit-learn 1.2.1 (SVD solver, whitening enabled).
C.3 Switching Linear Dynamical System (SLDS)
PCA-projected states were modeled with an SLDS having three latent regimes ( $K=3$ ), chosen via BIC on validation data, representing factual, transitional, and misaligned belief states. Dynamics per regime: $h^{\prime}_{t+1}=M_{z_{t}}h^{\prime}_{t}+c_{z_{t}}+\varepsilon_{t}$ , $\varepsilon_{t}\sim\mathcal{N}(0,\Sigma_{z_{t}})$ , $z_{t}â\{1,2,3\}$ . Parameters ( $T,M,c,\Sigma$ ) were learned via EM, initialized from K-means. For adversarial steps, regime-transition probabilities were examined to see if they reflected an increased likelihood of entering the "adverse" belief state. The SLDS aims to predict such slips into misaligned states.
C.4 Belief-Score Prediction
Since SLDS models latent PCA dynamics, a small two-layer MLP regressor (32 ReLU units/layer, Adam, early stopping) mapped PCA-projected states to belief scores for validation and for assessing the prediction of the misaligned (high belief score) state.
C.5 Simulation Protocol and Validation
Trajectories were simulated starting from empirical hidden-state distributions in the "safe" (low-belief) regime. Clean simulations used standard transitions. Poisoned simulations introduced adversarial perturbations (small fixed displacements estimated from empirical poisoned data) at random preselected intervals. Simulated trajectories matched empirical ones closely in timing/magnitude of belief shifts (slips into misaligned states), variance, and distributional characteristics (Kolmogorov-Smirnov test $p>0.3$ for final belief scores). Ablating adversarial perturbations confirmed their necessity for replicating rapid belief shifts towards misaligned states. This validates the SLDSâs ability to predict such failure modes.
C.6 Computational Details
NVIDIA A100 GPUs were used for state extraction and PCA. State extraction took $\sim$ 3 hours per model. PCA and SLDS estimation took <2 CPU hours on Intel Xeon Gold CPUs. Code used PyTorch 2.0.1, NumPy 1.25, scikit-learn 1.2.1.
C.7 Summary of Findings
A simple three-regime, low-rank SLDS (with low rank chosen for practical SDE approximation) captures adversarial belief dynamics for various misinformation types and reproduces complex empirical temporal behaviors, effectively modeling the process of an LLM slipping into a misaligned state. These models offer tractable insights into LLM reasoning, highlighting latent regime shifts from subtle adversarial prompts and demonstrating the potential to predict such failure modes at inference time.
Appendix D Extended Generalization Study Results
This appendix provides more comprehensive SLDS transferability results (Section 5.1). Table 4 shows $R^{2}$ (one-step-ahead hidden state prediction) and NLL (test trajectories) when an SLDS trained on a source (Train Model/Task) is tested on target combinations. SLDS hyperparameters ( $K=4$ regimes, $k=40$ projection rank, chosen for practical SDE approximation) were consistent. Training data for each "Source SLDS" used all available trajectories for the specified Train Model/Task from our main corpus (Section 3). Evaluation used all available trajectories for the Test Model/Task. The goal is to assess how well the learned approximation of reasoning dynamics (including potential failure modes) generalizes.
Table 4: Extended SLDS transferability results. Each SLDS is trained on trajectories from the âTrain Modelâ on its indicated âSource Taskâ. Performance is evaluated on various âTest Modelâ / âTest Taskâ combinations, testing the generalization of the approximated reasoning dynamics.
| Train Model (Source Task) | Test Model | Test Task | $R^{2}$ | NLL |
| --- | --- | --- | --- | --- |
| Llama-2-70B (on GSM-8K) | | | | |
| Llama-2-70B | GSM-8K | 0.73 | 80 | |
| Llama-2-70B | StrategyQA | 0.65 | 115 | |
| Llama-2-70B | CommonsenseQA | 0.62 | 128 | |
| Mistral-7B | GSM-8K | 0.48 | 240 | |
| Mistral-7B | StrategyQA | 0.37 | 310 | |
| Gemma-7B-IT | GSM-8K | 0.40 | 275 | |
| Phi-3-Med | PiQA | 0.28 | 430 | |
| Mistral-7B (on StrategyQA) | | | | |
| Mistral-7B | StrategyQA | 0.71 | 88 | |
| Mistral-7B | GSM-8K | 0.63 | 135 | |
| Mistral-7B | OpenBookQA | 0.60 | 145 | |
| Llama-2-70B | StrategyQA | 0.42 | 270 | |
| Llama-2-70B | GSM-8K | 0.35 | 320 | |
| Gemma-7B-IT | BoolQ | 0.35 | 380 | |
| Qwen1.5-7B | HellaSwag | 0.31 | 405 | |
| Gemma-7B-IT (on BoolQ) | | | | |
| Gemma-7B-IT | BoolQ | 0.69 | 95 | |
| Gemma-7B-IT | TruthfulQA | 0.62 | 140 | |
| Gemma-2B-IT | BoolQ | 0.55 | 190 | |
| Llama-2-13B | BoolQ | 0.33 | 350 | |
| Mistral-7B | CommonsenseQA | 0.29 | 415 | |
| DeepSeek-67B (on CommonsenseQA) | | | | |
| DeepSeek-67B | CommonsenseQA | 0.74 | 75 | |
| DeepSeek-67B | GSM-8K | 0.66 | 110 | |
| Llama-2-70B | CommonsenseQA | 0.45 | 255 | |
| Mistral-7B | StrategyQA | 0.36 | 330 | |
Extended results corroborate main text observations: SLDS models are most faithful when applied to their training distribution (model/task). Transfer is reasonable within the same model family or to similar tasks. Performance degrades more significantly across different model architectures or distinct task types. These patterns indicate SLDS, as a statistical physics-inspired approximation, captures fundamental reasoning dynamics (including propensities for certain failure modes), but model-specific architecture and task-specific semantics also matter. Future work could explore learning more invariant reasoning representations for better generalization in predicting these misaligned states.
Appendix E Noise-induced Criticality and Latent Modes
We briefly derive how noise-induced criticality leads to distinct latent modes in a 1D Langevin system, analogous to how LLMs might slip into misaligned reasoning states. Consider an SDE:
$$
\,\mathrm{d}x_{t}=-U^{\prime}(x_{t})\,\,\mathrm{d}t+\sqrt{2D}\,\,\mathrm{d}W_{%
t},
$$
with a double-well potential $U(x)=\frac{a}{4}x^{4}-\frac{b}{2}x^{2}$ , where $a,b>0$ . The stationary density solves the FokkerâPlanck equation (Risken & Frank, 1996):
$$
0=-\frac{\,\mathrm{d}}{\,\mathrm{d}x}[-U^{\prime}(x)p_{\rm st}(x)]+D\frac{\,%
\mathrm{d}^{2}p_{\rm st}(x)}{\,\mathrm{d}x^{2}},
$$
yielding $p_{\mathrm{st}}(x)=\frac{1}{Z_{0}}\exp\left(-\frac{U(x)}{D}\right)$ , where $Z_{0}$ is a normalization constant.
For low noise ( $D<\frac{b^{2}}{4a}$ ), $p_{\mathrm{st}}(x)$ becomes bimodal, concentrating probability around two metastable wells at $xâ±\sqrt{b/a}$ . Trajectories cluster in these basins, separated by a barrier at $x=0$ . Rare fluctuations cause transitions between wells at rates $\propto\exp(-\Delta U/D)$ , where $\Delta U$ is the barrier height. Our empirically observed multimodal residual structure is interpreted analogously: each cluster is a distinct metastable basin, potentially representing different reasoning qualities (e.g., aligned vs. misaligned). This motivates discrete latent regimes in the SLDS to model transitions between these states, akin to how a physical system transitions between energy wells. This provides a conceptual basis for how LLMs might "slip" into different operational modes, some of which could be failure modes.