# ChaosBench-Logic: A Benchmark for Logical and Symbolic Reasoning on Chaotic Dynamical Systems
**Authors**: Noel Thomas
Abstract
Large language models (LLMs) excel at natural language tasks but remain brittle in domains requiring precise logical and symbolic reasoning. Chaotic dynamical systems provide an especially demanding test because chaos is deterministic yet frequently conflated with randomness, complexity, or mere nonlinearity. This paper introduces ChaosBench-Logic, a benchmark that evaluates LLM reasoning across 30 diverse dynamical systems using a unified first-order logic ontology. Each system is annotated with truth assignments for 11 semantic predicates, and 621 questions are generated across seven reasoning categories, including multi-hop implications, cross-system analogies, counterfactual reasoning, bias probes, and multi-turn dialogues. Metrics are defined for logical accuracy, implication consistency, dialogue coherence, and contradiction, and an open-source evaluation pipeline is released. Initial experiments show that frontier LLMs such as GPT-4, Claude 3.5 Sonnet, Gemini 2.5 Flash, and the open-source LLaMA-3 70B achieve 91â94% per-item accuracy, yet still score 0% on compositional items and exhibit fragile global coherence: dialogue-level accuracy ranges from 53.1% (GPT-4 CoT) to 75.5% (LLaMA-3 zeroshot). ChaosBench-Logic provides a rigorous testbed for diagnosing these failures and a foundation for developing neuro-symbolic approaches that improve scientific reasoning in LLMs.
Code (GitHub) â https://github.com/11NOel11/ChaosBench-Logic
Dataset (Hugging Face) â https://huggingface.co/datasets/11NOel11/ChaosBench-Logic
1 Introduction
Large language models transform natural language processing, demonstrating strong abilities in translation, summarisation, question answering, and code synthesis. Despite this progress, they remain surprisingly brittle when confronted with tasks requiring precise logical or symbolic reasoning. In scientific domains such as dynamical systems and chaos theory, accurate reasoning is vital: one must distinguish whether a system is chaotic, periodic, quasi-periodic, or random; infer implications of qualitative properties; and reason about parameter perturbations and bifurcations. Errors in such reasoning can lead to incorrect scientific conclusions and undermine trust in LLM-based scientific tools.
Chaotic dynamics provide a stringent test of reasoning because âchaosâ is often misunderstood even by experts. Chaos is deterministic yet exhibits sensitive dependence on initial conditions, positive Lyapunov exponents, and complex attractor geometry [strogatz2015nonlinear, ott2002chaos, devaney1989intro]. It is not equivalent to randomness, nonlinearity, or complexity; confusing these concepts causes systematic misinterpretations of physical systems. Existing LLM benchmarks do not adequately test this mix of domain semantics and formal reasoning depth. This paper therefore develops a benchmark that systematically evaluates and analyses LLM reasoning about chaotic and non-chaotic systems under formal logical constraints.
1.1 Contributions
This paper makes the following key contributions:
1. ChaosBench-Logic Benchmark. A benchmark of 621 reasoning questions over 30 dynamical systems spanning ODEs, maps, PDEs, neuronal and chemical models, and stochastic processes [thomas2026chaosbenchlogic, thomas2026chaosbenchlogic_hf].
1. first-order logic Ontology. A unified logical vocabulary (11 predicates) and a compact axiom system capturing widely accepted implications between chaos, determinism, Lyapunov behavior, randomness, attractor types, and predictability.
1. Seven Task Families. Atomic QA, multi-hop implications, cross-system analogies and non-analogies, counterfactual reasoning, bias probes, multi-turn dialogues, and hard compositional questions.
1. Metrics and Pipeline. Reproducible evaluation for accuracy, implication consistency, dialogue coherence, contradiction detection, and axiom-violation counts.
1. Empirical Diagnosis. Frontier and open-source LLMs achieve high per-item accuracy (91â94%) but fail on compositional items (0%) and show fragile multi-turn coherence.
2 Related Work
Logical and Symbolic Reasoning in LLMs.
Recent work assesses LLMs on deductive and symbolic tasks, including BIG-Bench [srivastava2022bigbench], ReClor [yu2020reclor], LogiQA [liu2020logiqa], and related logic benchmarks. These datasets reveal persistent brittleness in multi-step inference, contradiction avoidance, entailment, and reasoning under explicit constraints. Several efforts introduce structured prompting (e.g., chain-of-thought) or tool-augmented reasoning, yet models still hallucinate intermediate steps or fail to maintain coherence across turns [wei2022chainofthought]. ChaosBench-Logic extends this line of inquiry to a scientific domain where reasoning must stay consistent with formal definitions of chaos-related semantics.
Scientific QA and Domain Reasoning.
Benchmarks such as SciBench [wang2023scibench] and MATH [hendrycks2021math] evaluate quantitative or symbolic scientific problem solving, highlighting limitations in equation manipulation and multi-step reasoning. However, none target dynamical systems theory, nor do they ground questions in a shared logical ontology. ChaosBench-Logic differs by anchoring all questions in a unified first-order logic structure and scoring models for logical coherence rather than only final-answer plausibility.
Chaos and Machine Learning.
Prior ML-for-chaos work primarily evaluates forecasting or attractor reconstruction (e.g., reservoir computing and deep learning baselines) rather than formal reasoning about qualitative regimes [pathak2018reservoir, gilpin2021chaos]. ChaosBench-Logic complements these benchmarks by focusing on symbolic and logical competence: distinguishing chaos from randomness and enforcing consistent implication structure across diverse systems.
Bias and Misconception Diagnostics.
Evaluations such as TruthfulQA [lin2022truthfulqa] examine systematic model errors and truthfulness failures. Yet domain-specific misconceptions (e.g., chaosârandomness confusion, nonlinearityâchaos conflation) remain largely unexplored. ChaosBench-Logic introduces explicit bias-probe families tailored to chaos theory semantics.
3 Methods
This section formalizes ChaosBench-Logic, describes the ontology and axiom system, details dataset construction, and defines evaluation metrics.
3.1 Problem Definition
This paper considers reasoning over a collection of dynamical systems $\mathcal{S}=\{s_{1},...,s_{30}\}$ including continuous-time flows, discrete-time maps, PDEs, neuronal oscillators, chemical reaction models, and stochastic processes.
Each system $sâ\mathcal{S}$ is associated with semantic properties expressed as unary logical predicates from a fixed vocabulary $\mathcal{P}$ . Given a natural-language query about a system $s$ , a model must predict the logically correct answer according to the systemâs ground-truth property assignment and a global axiom system.
3.2 Logical Predicate Vocabulary
ChaosBench-Logic defines 11 unary predicates, each mapping a system $s$ to a boolean:
- $\mathsf{Chaotic}(s)$
- $\mathsf{Deterministic}(s)$
- $\mathsf{PosLyap}(s)$ (positive largest Lyapunov exponent)
- $\mathsf{Sensitive}(s)$
- $\mathsf{StrangeAttr}(s)$
- $\mathsf{PointUnpredictable}(s)$
- $\mathsf{StatPredictable}(s)$
- $\mathsf{QuasiPeriodic}(s)$
- $\mathsf{Random}(s)$ (true randomness / stochasticity)
- $\mathsf{FixedPointAttr}(s)$
- $\mathsf{Periodic}(s)$
3.3 Predicate Semantics and Annotation Protocol
To make the benchmark auditable and to reduce ambiguity, each predicate is treated as a semantic label over a named system (or canonical regime) rather than as an invitation for models to interpret definitions ad hoc. In particular, the benchmark is designed so that a system can appear random in its trajectories and still be labeled deterministic, reflecting the core distinction between stochasticity and sensitive determinism.
This paper uses the following annotation principles:
- Determinism vs. randomness. $\mathsf{Deterministic}(s)$ indicates that the system evolution is fully specified by its state and update rule. $\mathsf{Random}(s)$ is reserved for systems with inherent stochasticity (e.g., explicit noise terms or stochastic differential equations).
- Chaos as a regime label. $\mathsf{Chaotic}(s)$ indicates a canonical parameter regime widely treated as chaotic in the literature for that named system. Borderline regimes are avoided and disputed cases are removed during curation.
- Predictability predicates. $\mathsf{PointUnpredictable}(s)$ captures practical loss of long-horizon point forecasts even under perfect modeling, while $\mathsf{StatPredictable}(s)$ reflects that coarse statistical structure can remain predictable or stable over time (e.g., invariant measures).
- Attractor-type predicates. $\mathsf{Periodic}(s)$ , $\mathsf{QuasiPeriodic}(s)$ , and $\mathsf{FixedPointAttr}(s)$ are mutually exclusive indicators of common non-chaotic behavior classes, used to generate non-analogies and âdo not overgeneralizeâ items.
These principles make ChaosBench-Logic closer to a controlled reasoning benchmark than a definition-discovery task: models are evaluated on logical consistency under a fixed ontology and ground truth.
3.4 Global first-order logic Axiom System
ChaosBench-Logic specifies a set $\Phi$ of global axioms encoding widely used implications in dynamical systems [strogatz2015nonlinear, ott2002chaos, devaney1989intro]. Each implication is written explicitly:
Chaos axioms.
$$
\displaystyle\forall s:\;\mathsf{Chaotic}(s) \displaystyle\Rightarrow\mathsf{Deterministic}(s), \displaystyle\forall s:\;\mathsf{Chaotic}(s) \displaystyle\Rightarrow\mathsf{PosLyap}(s), \displaystyle\forall s:\;\mathsf{Chaotic}(s) \displaystyle\Rightarrow\mathsf{Sensitive}(s), \displaystyle\forall s:\;\mathsf{Chaotic}(s) \displaystyle\Rightarrow\mathsf{PointUnpredictable}(s), \displaystyle\forall s:\;\mathsf{Chaotic}(s) \displaystyle\Rightarrow\mathsf{StatPredictable}(s), \displaystyle\forall s:\;\mathsf{Chaotic}(s) \displaystyle\Rightarrow\neg\mathsf{Random}(s). \tag{1}
$$
Lyapunov & predictability.
$$
\displaystyle\forall s:\;\mathsf{PosLyap}(s) \displaystyle\Rightarrow\mathsf{Sensitive}(s), \displaystyle\forall s:\;\mathsf{PosLyap}(s) \displaystyle\Rightarrow\mathsf{PointUnpredictable}(s). \tag{7}
$$
Attractor-type implications.
$$
\displaystyle\forall s:\;\mathsf{StrangeAttr}(s) \displaystyle\Rightarrow\mathsf{Chaotic}(s), \displaystyle\forall s:\;\mathsf{Periodic}(s) \displaystyle\Rightarrow\neg\mathsf{Chaotic}(s), \displaystyle\forall s:\;\mathsf{QuasiPeriodic}(s) \displaystyle\Rightarrow\neg\mathsf{Chaotic}(s), \displaystyle\forall s:\;\mathsf{FixedPointAttr}(s) \displaystyle\Rightarrow\neg\mathsf{Chaotic}(s). \tag{9}
$$
Randomness vs determinism.
$$
\displaystyle\forall s:\;\mathsf{Random}(s) \displaystyle\Rightarrow\neg\mathsf{Deterministic}(s). \tag{13}
$$
Design choice: no converse rules.
Crucially, reverse implications are not assumed (e.g., $\mathsf{Sensitive}\Rightarrow\mathsf{Chaotic}$ ). This avoids overspecification and forces models to respect directionality: many scientific misconceptions arise from treating one-way implications as equivalences.
3.5 Ground-Truth Closure and Answer Derivation
The benchmark ground truth for each system is a truth assignment for the 11 predicates that is consistent with $\Phi$ . For implication questions, the correct answer is computed by applying forward chaining under $\Phi$ .
Let $\mathcal{A}_{s}$ denote the set of all literals known true/false for a system $s$ . The logical closure under $\Phi$ is defined as:
$$
\displaystyle\mathrm{Cl}_{\Phi}(\mathcal{A}_{s}) \displaystyle=\mathrm{lfp}\Bigl(\mathcal{A}_{s}\cup\{\,\ell\;:\;(\ell_{1}\wedge\cdots\wedge\ell_{k}\Rightarrow\ell)\in\Phi, \displaystyle\qquad\qquad\ \ \ell_{1},\dots,\ell_{k}\in\mathcal{A}_{s}\}\Bigr), \tag{14}
$$
where $\mathrm{lfp}$ is the least fixed point reached by repeatedly applying all applicable implications. Because the system annotations satisfy $\Phi$ , the closure remains consistent. For model outputs, closure and axiom checks expose contradictions and implication violations.
3.6 System Coverage
ChaosBench-Logic includes 30 systems across five broad classes. Table 1 provides an overview.
Table 1: Overview of dynamical systems included in ChaosBench-Logic. Each system is annotated with 11 semantic predicates and participates in first-order logic reasoning.
| Chaotic ODEs | Lorenz-63, Lorenz-84, Lorenz-96, Rössler, Chen, HR neuron | yes |
| --- | --- | --- |
| Non-chaotic ODEs | Simple harmonic oscillator (SHM), Van der Pol, Brusselator, FitzHughâNagumo | no |
| Maps | Logistic, HĂ©non, Ikeda, Baker, Arnold Cat, Standard Map, Circle Map | mixed |
| PDEs | KuramotoâSivashinsky (chaotic), SineâGordon (integrable) | mixed |
| Stochastic | OrnsteinâUhlenbeck | random, non-chaotic |
This paper includes canonical systems spanning flows and maps (e.g., Lorenz and Rössler systems, Hénon and logistic maps) as well as PDE and stochastic processes [lorenz1963dnf, rossler1976equation, henon1976map, may1976simple, sivashinsky1977nonlinear, kuramoto1978diffusion, ornstein1930brownian].
3.7 Dataset Construction Pipeline
The benchmark is produced via a four-stage pipeline:
(1) System encoding.
Each system is encoded in a structured JSON file containing equations or update rules, canonical parameter regimes, short natural-language summaries, and predicate truth assignments. This representation supports both human inspection and programmatic question generation.
(2) Template-based generation with semantic constraints.
This paper designs seven task families with $70+$ templates. Templates are not free-form: each includes explicit semantic constraints (e.g., âonly apply this template to systems with $\mathsf{Random}(s)=\text{TRUE}$ â) to avoid ill-posed questions. Lexical surface forms and question structure are also varied to reduce shallow cue exploitation.
(3) Human curation for correctness and ambiguity removal.
All candidates are manually checked for logical correctness under $\mathrm{Cl}_{\Phi}(·)$ , linguistic clarity, and consistency with the intended misconception being tested (for bias probes). Ambiguous items (e.g., parameter regimes with disputed behavior) are removed.
(4) Final filtering and consistency checks.
Questions that violate applicability rules or create logical self-conflicts are automatically filtered, followed by a final manual pass. In addition to basic sanity checks, each item is verified to be answerable from the system labels and $\Phi$ alone (i.e., without numerical simulation).
3.8 Question Format and Metadata
Each benchmark item includes a system identifier and a natural-language question, plus structured metadata used in evaluation. The current release includes fields such as system_id, family, template_id, and (for dialogues) dialogue_id and turn_index. This makes it possible to compute aggregate metrics per family and to audit failures at the level of a specific template or system.
For compositional items, the metadata explicitly records which sub-skills are required (e.g., implication chaining plus analogy rejection), enabling more fine-grained analysis than a single global accuracy score.
3.9 Reasoning Task Families
ChaosBench-Logic covers seven reasoning types:
1. Atomic Logical QA: single-step predicate queries.
1. Multi-Hop Implication Reasoning: forward-chaining under $\Phi$ .
1. Cross-System Analogical / Non-Analogical Reasoning: transfer and rejection.
1. Counterfactual Parameter Reasoning: bifurcations and perturbations.
1. Bias Probes: adversarial misconception tests.
1. Advanced Multi-Turn Dialogues: cross-turn coherence and contradiction avoidance.
1. Hard Concept-Synthesis / Compositional: multiple reasoning modes combined.
Table 2 summarizes what each family measures.
Table 2: What each task family tests and why it is non-trivial for LLMs.
| Atomic QA Multi-hop Analogy | Correctly recall system-level truth values (confident hallucination). Apply directionality of implications (converse fallacy). Transfer only the shared structure (surface-form analogy bias). |
| --- | --- |
| Counterfactual | Reason about regime change vs invariance (overgeneralize chaos). |
| Bias probes | Resist misleading cues (âchaos = randomâ, âPDE = chaoticâ). |
| Dialogues | Maintain consistent belief state over turns (belief drift). |
| Compositional | Integrate multiple constraints simultaneously (local-but-not-global reasoning). |
3.10 Multi-Turn Dialogue Design
Each dialogue is grouped under a dialogue_id, with 3â6 turns. Correctness requires per-turn accuracy and intra-dialogue coherence: answers must not contradict earlier commitments under $\Phi$ . This targets persistent reasoning and belief-state stability.
A key design choice is that dialogue turns intentionally mix âeasyâ atomic questions with implication-driven follow-ups. This stresses whether a model can (i) answer a local question, (ii) remember its own earlier commitment, and (iii) apply the same commitment consistently when queried from a different angle later in the conversation.
3.11 Bias Families
This paper identifies six systematic reasoning biases:
1. ChaosâRandomness Bias
1. ChaosâNonlinearity Bias
1. ComplexityâChaos Bias
1. FractalâChaos Bias
1. PDEâChaos Overgeneralisation
1. Name/Analogy Bias (âLorenz-like $â$ chaoticâ)
These biases are encoded at the template level in the current release and can be refined into per-item labels in future versions.
3.12 Evaluation Metrics
This paper introduces metrics that separate local correctness from global coherence.
Logical accuracy.
Given $N_{\text{eval}}$ questions with valid normalized predictions:
$$
\mathrm{Acc}=\frac{1}{N_{\mathrm{eval}}}\sum_{i=1}^{N_{\mathrm{eval}}}\mathrm{I}\bigl(\hat{y}_{i}=y_{i}^{*}\bigr). \tag{15}
$$
Dialogue accuracy.
For a dialogue $d$ with turns $t=1,...,T_{d}$ :
$$
\mathrm{DialogueAcc}(d)=\mathrm{I}\bigl(\forall t,\;\hat{y}_{d,t}=y_{d,t}^{*}\bigr). \tag{16}
$$
This metric is strict by design: a single incoherent turn collapses a whole dialogue to 0.
Contradiction & axiom-violation checks.
Beyond correctness, this work evaluates whether a modelâs set of commitments is internally consistent. Contradictions (e.g., predicting both $\mathsf{Chaotic}(s)$ and $\mathsf{Random}(s)$ ) are flagged and violations of implications in $\Phi$ are counted within a dialogue. These diagnostics explain how and why dialogues fail even when per-turn answers look locally plausible.
4 Experimental Setup and Baselines
Models.
This paper evaluates representative frontier and open-source LLMs: GPT-4 (OpenAI), Claude 3.5 Sonnet (Anthropic), Gemini 2.5 Flash (Google), and LLaMA-3 70B (Meta) [openai2023gpt4, anthropic2024claude35, google2025gemini25flash, meta2024llama3]. All models are queried through their official APIs or hosted endpoints.
Prompting.
Zero-shot prompts request a final YES/NO decision. For chain-of-thought (CoT) prompting, each model is asked to write reasoning and end with FINAL_ANSWER: YES/NO [wei2022chainofthought]. The evaluation script extracts and normalizes the final label.
Hyperparameters and determinism.
All models are run at temperature 0 (or nearest deterministic setting) to reduce sampling variance and emphasize reasoning consistency failures.
Evaluation pipeline.
Responses are normalized using a robust pattern matcher. If a response contains multiple answers, the final explicit marker (CoT) or the last YES/NO token (zero-shot) is preferred. This avoids over-crediting models that hedge or self-correct mid-output.
5 Results
This section presents empirical results on ChaosBench-Logic. All accuracies are computed over items where a valid normalized answer can be extracted; in the final runs, this is 621/621 items for all configurations.
5.1 Overall Logical Performance
Table 3 summarizes performance. All evaluated models achieve 91â94% per-item accuracy, yet dialogue-level coherence is substantially lower.
Table 3: Overall performance on ChaosBench-Logic. Acc is per-item accuracy; DlgAcc is the fraction of dialogues where all turns are correct.
| GPT-4 GPT-4 Claude 3.5 Sonnet | Zeroshot CoT Zeroshot | 94.0 88.2 91.6 | 69.4 53.1 67.3 |
| --- | --- | --- | --- |
| Gemini 2.5 Flash | Zeroshot | 91.9 | 71.4 |
| LLaMA-3 70B | Zeroshot | 91.6 | 75.5 |
| LLaMA-3 70B | CoT | 89.5 | 65.3 |
Why DlgAcc is much lower than Acc.
Dialogue accuracy compounds errors: for a dialogue of length $T$ , even an independent per-turn accuracy $p$ yields an âall-correctâ probability of approximately $p^{T}$ . With $pâ 0.92$ and $Tâ[3,6]$ , this ranges from $0.92^{3}â 0.78$ to $0.92^{6}â 0.61$ , matching the observed DlgAcc range (53â76%). In practice, dialogue errors are not independent: models often drift in their interpretation of a system across turns, causing clustered failures rather than isolated mistakes.
Local competence vs. global coherence.
The high Acc values show that most single questions are solvable for current LLMs. The lower DlgAcc values indicate that models struggle to maintain a stable belief state across turns, even when the required beliefs are simple boolean properties with deterministic implications.
5.2 Task-Family Highlights and the â0% Compositionalâ Gap
Table 4 reports the best-performing model for key task families. Models are strong on many atomic and multi-hop items but fail completely on compositional questions.
Table 4: Task-specific best models on ChaosBench-Logic. Accuracies are over items with a valid normalised answer.
| Atomic logical QA Bias probes Counterfactuals | GPT-4 (Zeroshot) GPT-4 (Zeroshot) GPT-4 (Zeroshot) | 96.1 93.0 97.4 |
| --- | --- | --- |
| Multi-hop reasoning | GPT-4 (Zeroshot) | 97.1 |
| Multi-turn (per-turn) | LLaMA-3 (Zeroshot) | 93.9 |
| Dialogue (dialogue-level) | LLaMA-3 (Zeroshot) | 75.5 |
| Compositional | All models | 0.0 |
What makes compositional items qualitatively harder.
Compositional questions deliberately combine multiple constraints, such as: (i) chaining two or more implications under $\Phi$ ; (ii) rejecting a misleading analogy cue; (iii) applying a counterfactual change shown elsewhere in the dialogue; and (iv) producing an answer consistent with prior turns. In practice, models often succeed on each component in isolation but fail when the components must be integrated into a single coherent decision.
A key observed failure mode: converse assumptions.
Because $\Phi$ contains only one-way implications, compositional items often punish the common LLM heuristic âif A implies B, then B implies A.â For example, models may incorrectly infer $\mathsf{Chaotic}(s)$ from $\mathsf{Sensitive}(s)$ or from $\mathsf{PosLyap}(s)$ even though the ontology does not assume those converses. This is not a trick: it reflects real scientific reasoning discipline about implication directionality.
5.3 Consistency Diagnostics Beyond Accuracy
A major advantage of ChaosBench-Logic is that inconsistency is measurable. Two models can achieve similar per-item accuracy while differing sharply in how they fail. In particular, some errors are local (a single wrong predicate assignment) while others represent global constraint failures (e.g., asserting both $\mathsf{Random}(s)$ and $\mathsf{Deterministic}(s)$ , or implying chaos from a non-converse symptom).
These failures matter operationally: in downstream scientific workflows, a locally incorrect answer may be caught by a user, but a globally inconsistent belief state can produce confident yet mutually incompatible explanations across time. This is precisely the failure mode that multi-turn dialogues and axiom-violation tracking are designed to surface.
5.4 Effect of Chain-of-Thought Prompting
CoT is frequently assumed to improve reasoning [wei2022chainofthought], but on ChaosBench-Logic it reduces both accuracy and dialogue-level coherence.
Table 5: CoT impact on overall and dialogue-level performance.
| GPT-4 | $-5.8$ | $-16.3$ | worse |
| --- | --- | --- | --- |
| LLaMA-3 70B | $-2.1$ | $-10.2$ | worse |
Interpretation.
CoT outputs are longer and appear more rigorous, but they can also introduce extra unsupported assumptions (e.g., implicitly treating ârandom-lookingâ as ârandomâ) and amplify converse reasoning errors. CoT can therefore increase explanatory confidence without improving logical soundness. The larger drop in DlgAcc suggests that CoT especially harms belief-state stability in multi-turn settings.
6 Discussion
6.1 What ChaosBench-Logic Reveals About LLM Reasoning
The central finding: weak global constraint satisfaction.
Across all evaluated models, single-question accuracy exceeds 90%, yet multi-turn coherence is far weaker. This indicates that current LLMs can produce correct local judgments but struggle to maintain a consistent global world model, even when the world model is as simple as a set of boolean predicates linked by Horn-style implications.
Why scientific domains make this failure more obvious.
In everyday QA, inconsistent answers can be rationalized away as ambiguity. In dynamical systems, however, core relationships are crisp: ârandom $\Rightarrow$ non-deterministicâ and âchaotic $\Rightarrow$ deterministic and not randomâ are definitional in this ontology [strogatz2015nonlinear, ott2002chaos, devaney1989intro]. ChaosBench-Logic turns these into explicit constraints that models must respect, revealing when a modelâs internal reminders are not globally stable.
6.2 How ChaosBench-Logic Can Support Neuro-Symbolic and Tool-Augmented Systems
The structure of ChaosBench-Logic makes it a natural evaluation target for hybrid approaches:
- Symbolic verifier / constraint checker. An LLM can propose answers while a symbolic module checks whether the answers violate $\Phi$ , triggering self-correction.
- Logic-guided decoding. Outputs can be constrained so that answers cannot jointly violate implications already committed within a dialogue.
- Training with consistency losses. Fine-tuning can penalize axiom violations across turns, not only wrong final labels.
Because ChaosBench-Logic uses a compact and explicit ontology, these methods are implementable without full theorem proving.
6.3 Limitations and Scope
ChaosBench-Logic deliberately emphasizes logical structure over numerical simulation. This makes the benchmark lightweight and auditable, but it also means that each named system is treated as having a canonical regime label. Future releases can extend the benchmark with (i) multiple parameter regimes per system, (ii) items that require extracting regime information from equations or parameter values, and (iii) hybrid items that combine symbolic logic with tool-verified numeric diagnostics (e.g., Lyapunov estimation).
7 Conclusion
This paper introduces ChaosBench-Logic, a benchmark for diagnosing logical and symbolic reasoning failures in LLMs within the domain of chaotic dynamical systems. By grounding all questions in an explicit first-order logic ontology, ChaosBench-Logic measures not only per-item accuracy but also implication consistency and multi-turn coherence. Experiments show that frontier and open-source LLMs achieve strong local performance (91â94% per-item accuracy) yet remain globally brittle: dialogue-level accuracy is substantially lower and all tested models fail on compositional items (0%). ChaosBench-Logic supports progress toward scientific AI systems that maintain globally consistent beliefs, resist domain misconceptions, and integrate symbolic structure with fluent language understanding.