2504.14520v1

# Meta‑Thinking in LLMs via Multi‑Agent Reinforcement Learning: A Survey > Ahsan Bilal is with University of Oklahoma, Norman, OK, 73072, USA (e-mail: ahsan.bilal-1@ou.edu).Muhammad Ahmed Mohsin, Muhammad Umer are with Stanford University, Stanford, CA, 94305, USA (e-mail: muahmed, mumer@stanford.edu).Muhammad Awais Khan Bangash is with the School of Electrical and Computer Engineering, Oklahoma State University, Stillwater, OK, 74075 USA (e-mail: awais.bangash@okstate.edu).Muhammad Ali Jamshed is with University of Glasgow, G12 8QQ, Glasgow, UK (e-mail: muhammadali.jamshed@glasgow.ac.uk). Abstract This survey explores the development of meta-thinking capabilities in Large Language Models (LLMs) from a Multi-Agent Reinforcement Learning (MARL) perspective. Meta-thinking—self-reflection, assessment, and control of thinking processes—is an important next step in enhancing LLM reliability, flexibility, and performance, particularly for complex or high-stakes tasks. The survey begins by analyzing current LLM limitations, such as hallucinations and the lack of internal self-assessment mechanisms. It then talks about newer methods, including RL from human feedback (RLHF), self-distillation, and chain-of-thought prompting, and each of their limitations. The crux of the survey is to talk about how multi-agent architectures, namely supervisor-agent hierarchies, agent debates, and theory of mind frameworks, can emulate human-like introspective behavior and enhance LLM robustness. By exploring reward mechanisms, self-play, and continuous learning methods in MARL, this survey gives a comprehensive roadmap to building introspective, adaptive, and trustworthy LLMs. Evaluation metrics, datasets, and future research avenues, including neuroscience-inspired architectures and hybrid symbolic reasoning, are also discussed. Index Terms: Large Language Models (LLMs), Meta-Thinking, Multi-Agent Reinforcement Learning (MARL), Theory of Mind (ToM), Self-Reflection, Reinforcement Learning with Human Feedback (RLHF), Supervisor-Agent Architectures, Introspective AI, Meta-Rewards, Symbolic-MARL Hybrid Systems I Introduction Tthe cognitive abilities, such as intelligence and creativity, have played a fundamental role in human discoveries and inventions. Understanding the relationship between these two cognitive abilities is important not only for the advancement of psychological theories but also for the improvement of educational practices [1]. However, researchers still hold different views on how intelligence and creativity interact, often leading to conflicting findings. A key question in this discourse is how intelligence enables structured problem-solving, while creativity fosters novel solutions that are essential for human cognition and artificial intelligence systems. For example, in creative writing tasks, intelligence helps in structuring narratives and developing characters, whereas creativity drives originality and emotional depth. Similarly, in problem-solving tasks, intelligence aids in analyzing constraints, while creativity allows for flexible and unconventional approaches. Moreover, the role of internal thought processes varies with task complexity. Simpler tasks require minimal reasoning, whereas more complex tasks demand deeper cognitive engagement. This principle extends to artificial intelligence, where more sophisticated models exhibit enhanced performance in tasks requiring higher-order thinking. Based on this, researchers hypothesize that thinking LLMs will have a distinct advantage in handling sufficiently complex tasks, as they integrate structured reasoning with creative problem-solving abilities [2]. One practical approach to enable this creative thinking reasoning is through text-based thought generation and leveraging the natural language capabilities of LLMs. Since, LLMs are pre-trained on vast amounts of text that include human-written reflections, they inherently encode aspects of human reasoning processes. LLMs can be prompted to engage in internal reasoning before responding, allowing for more thoughtful and structured responses [3]. For example, the chain-of-thought (CoT) prompting technique [4] facilitates the reasoning process by guiding LLMs to articulate their reasoning steps explicitly, improving their performance on tasks that require logical and mathematical reasoning. However, the benefits of CoT prompting appear to be domain-specific. A meta-analysis by [5] found that while CoT significantly enhances LLM performance in structured reasoning tasks, it provides little to no advantage in areas that do not involve mathematical or logical processes. LLMs such as GPT-3, LLaMA, and PaLM, have become transformative tools across various domains, such as natural language processing (NLP), healthcare, education, software development, and scientific research, demonstrating remarkable proficiency in diverse tasks, such as text generation, language translation, summarization, sentiment analysis, code generation, medical diagnosis assistance, and automated tutoring [6]. Despite their success, LLMs continue to face the challenge of ”hallucination,” where they generate content that is inaccurate or not based on factual information [7]. The issue is particularly critical in high-stakes applications, such as clinical and legal domains, where reliable and accurate text generation is essential. Ensuring reliability through self-assessment is important in today’s rapidly evolving and high-stakes environment, where even small changes in the accuracy of LLMs can result in severe consequences [8]. Addressing hallucinations in LLMs is crucial for expanding their practical applicability and fostering trust in these technologies. Hallucinations in LLMs are of three main categories [9]: 1) Input conflict: in which the generated output contradicts the given prompt; 2) Context conflict: in which there are inconsistencies within the generated response; and 3) Factual conflict: in which LLM produce incorrect information, despite having access to accurate knowledge. Understanding and mitigating these hallucinations is an important step to improve the robustness and reliability of LLMs in real-world scenarios. One of the key challenges is that LLMs generate responses in an autoregressive manner without an inherent mechanism for evaluating their own outputs and this leads to unchecked errors. Hallucinations in LLMs can be eliminated through number of approaches [10]. One of the most popular methods is using RL, where the model is trained to give better responses based on feedback. For example, ChatGPT-3.5 uses a method known as RLHF [11], where human assessors rank the responses of the model, training it on what sounds right. RL is effective in modern AI that enables agents to learn complex behavior by trial-and-error exploration of their environment. Its capacity to discover new policies without explicit supervision has spurred progress in robotics, game playing, and autonomous systems [12, 13]. At the same time, the scale of RL work has grown exponentially: annual papers in top-tier venues have increased by over 400% in the last half-decade, as an indicator of a continuously accelerating interest in RL methods (see Figure 1). This growing pattern emphasizes the imperative of integrating RL, especially multi-agent variants, into meta-thought models of LLMs. More recently, frameworks like DeepSeek have built on this concept further [14]. Instead of only using RL towards the later stages of training, they use it throughout the training process. This way, the LLM is learning from itself throughout and improving the extent to which it follows commands, thinks for itself, and stays consistent with what people expect. Another method is contrastive learning, as discussed in MixCL [15], where the LLM is shown both good and bad examples to help it better understand the difference between true and false information. Knowledge-based methods which leverage structured external sources to detect and correct hallucinations. Knowledge-based methods externally are Neural Path Hunter [16], Knowledge Graph Retrofitting (KGR) [17], Validating Low-Confidence Generation [18], and Reasoning on Graphs (RoG) [19]. Knowledge filling methods try to fill in the gaps in the model’s internal knowledge to make its answers more accurate [20]. Other approaches don’t rely on external data at all and rely solely on the model itself to iteratively assess and improve its output. For example, Zero-resource and self-feedback methods like SelfCheckGPT [21], Chain-of-Verification [22], Self-Refine [23], and Self-Contradictory [24] help the LLM model review and improve its own response. Finally, decoding-level interventions are Knowledge-Constrained Tree Search [25], Inference-Time Intervention [26], and Decoding by Contrasting Layers (DoLa) [27], all of which attempt to guide token generation in LLMs towards factuality. However, there are major pitfalls in each methodology: RL depends on human biased labeling and creates bias [20, 11]; external knowledge retrieval is limited in terms of delay and accuracy for retrieval data [20]; sampling lacks completeness of information and may miss important facts [15]; and self-feedback is unreliable in zero resource environments when the model has no access to outside help [28, 23]. Most importantly, all of these methods still rely on how LLMs are generated at a basic level: by predicting one word at a time based on the words preceding it. This framework makes it possible for minor errors to build up into more significant ones, and the model does not actually ”understand” truth in the same manner as humans [29]. <details> <summary>2504.14520v1/x1.png Details</summary>  ### Visual Description # Technical Document: Bar Chart Analysis ## Chart Description The image depicts a **bar chart** illustrating the **growth of reinforcement learning research** over time. The chart uses **purple bars** to represent data points, with a **legend** confirming the color association. --- ### Axis Labels and Markers - **X-Axis (Horizontal):** - Label: `Year` - Markers: Discrete years from `2005` to `2020` (inclusive). - **Y-Axis (Vertical):** - Label: `Number of Reinforcement Learning Papers` - Scale: Logarithmic, ranging from `0` to `6 × 10⁴` (60,000) in increments of `10⁴`. --- ### Data Points and Trends The chart shows a **steady increase** in the number of reinforcement learning papers, with a **sharp exponential rise** after 2016. Key data points (approximate values based on bar heights): | Year | Number of Papers | |------|------------------| | 2005 | ~50 | | 2006 | ~100 | | 2007 | ~200 | | 2008 | ~300 | | 2009 | ~500 | | 2010 | ~800 | | 2011 | ~1,200 | | 2012 | ~1,800 | | 2013 | ~2,500 | | 2014 | ~3,500 | | 2015 | ~6,000 | | 2016 | ~12,000 | | 2017 | ~22,000 | | 2018 | ~30,000 | | 2019 | ~42,000 | | 2020 | ~52,000 | --- ### Observations 1. **Pre-2016 Growth:** - Gradual, linear increase from ~50 papers in 2005 to ~12,000 in 2016. 2. **Post-2016 Surge:** - Exponential growth, with papers increasing by ~40,000 between 2016 and 2020. 3. **2020 Peak:** - Highest recorded value: **52,000 papers** in 2020. --- ### Technical Notes - **Logarithmic Y-Axis:** - Compresses early data points for clarity while emphasizing later growth. - **Legend:** - Single entry: `Purple` corresponds to the bar color. - **No Additional Elements:** - No secondary axes, annotations, or comparative datasets. --- ### Conclusion The chart highlights the **accelerated adoption of reinforcement learning** as a research field, particularly after 2016, likely driven by advancements in AI and computational power. </details> Figure 1: Annual number of RL publications in AI conferences (2019–2024). To address the issue of hallucination, one of the most promising research directions is to enhance the meta-thinking (the ability to reflect on, evaluate, and regulate one’s own thought processes) capabilities of LLMs so that LLMs can analyze, control, and refine their own thought processes. By implementing self-evaluation mechanisms, LLMs can identify inconsistencies, assess the reliability of their generated content, and improve their reasoning before producing a final response [22]. Compared to human thought, in which individuals think about themselves to check for the validity of their ideas before conclusion, LLMs lack an inherent self-checking system. By embedding meta-thinking, LLMs can examine their own reasoning steps, identify potential inconsistencies, and adaptively update their responses before rendering the final output [30]. To overcome the limitations imposed by the autoregressive generation process of LLMs, it is very important to integrate the meta-thinking via MARL because it empowers LLMs to collaboratively self-correct and adapt in real time, thereby [31]. Building on these advances, MARL can offer frameworks for LLMs to develop natural language capabilities with collaborative AI behavior [32]. For example, DyLAN [33] supports dynamic agent selection according to task difficulty so that agents based on LLM can be dynamically configured at runtime for uses like reasoning and code composition. FAMA [34] functionally configures agents at runtime by means of fine-tuning and enables natural language based communication for uses like text-based games and simulated driving. MetaGPT [35] supports agents with publish and subscribe capability on task specific messages in a shared message pool for collaborative coding. In robotics, CoELA [36] integrates LLMs with modular perception, memory, planning, and execution systems to enable robots to become smarter to cooperate. SMART-LLM [37] transforms high-level commands into robot team executable plans by breaking tasks into phases. RoCo [38] also provides robot arms with LLM agents that interact using conversation to manage action coordination. In Co-NavGPT [39], one LLM manages navigation for multiple agents with frontier assignments optimized. The study in [40] introduced a Criticize-Reflect architecture, assigning LLMs to perform critic and coordinator roles to enhance coordination between agents. ConsensusLLM [41] focuses on negotiation, where agents of various initial states reach agreement via natural language. In summary, as LLMs are increasingly applied in important domains, the integration of meta-thinking via MARL is a necessary evolution to ensure safety, accuracy, and adaptability in high-stakes, dynamic applications [42]. A meta-thinking induction framework for LLMs based on MARL was introduced in [43]. The approach introduces meta-thinking by assigning various high and low-level agents to LLMs, where exploration is encouraged during training, and explainability about the response is enhanced. Researchers evaluate LLM-based agents in a multi-agent cooperative text game with ToM inference tasks [44], which compares the performance of LLM-based agents with MARL against planning-based baselines. The results showed that LLM-based agents developed emergent collaborative behaviors and demonstrated higher-order ToM capabilities, indicating their ability to infer and respond to the thoughts and intentions of other agents. The main motivation behind this survey paper is that meta-reasoning and meta-thinking in LLMs have gained more and more attention. For example, [45] introduces the meta-buffer to capture the general patterns of thoughts and change these thoughts continuously in self-reflection. Most existing works [46, 35, 10, 47, 48, 49, 50, 51, 52, 53, 54, 55] focus on individual techniques like COT prompting or self-reflection, or explore multi-agent coordination in an architectural framework. But to date, there is no thorough survey that combines Meta-RL and MAS towards introspective and self-critical LLMs. This work fills that vital gap by exploring how the principles of RL, especially in multi-agent environments, can be used to make LLMs to meta-think. This paper provides the first roadmap for making dependable and responsive LLMs through meta-reasoning by poetically connecting meta-cognition, MARL frameworks, and datasets. The main contribution of this survey is that it is the first to investigate systematically the intersection of Meta-RL and meta-thinking in LLMs. The main contributions are: 1. A general taxonomy of meta-thinking approaches for LLMs, including single-agent (e.g., Self-Distill, SelfCheckGPT, Chain-of-Verification), multi-agent (e.g., supervisor-worker hierarchies, agent-role debates, Theory-of-Mind configurations), and reinforcement-based self-improvement methods (e.g., RLHF, adaptive intrinsic reward shaping). 1. A systematic study of MARL paradigms for meta-reasoning. This survey paper gives a thorough review of the MARL-enabled self-reflection and correction in LLMs. This includes 1) Supervisor-Agent Architectures, in which a high-level controller coordinates low-level agents for structured decomposition and synthesis (e.g., Co-NavGPT, FAMA) of tasks. 2) Adversarial Debate and Self-Critique, as inspired by DebateQA, these setups foster argumentation among agents to expose logical flaws. 3) Self-Play and Role-Play Systems, in which environments like AutoGPT and Criticize-Reflect showcase how iterative, role-based interaction enhances strategic depth. 4) Intrinsic Reward Mechanisms, in which Meta-reward combines human feedback with curiosity-driven scoring to guide reasoning refinement. 1. A comparative survey of recent survey papers on LLM meta-reasoning, to show that none of the papers jointly address all five axes: 1) meta-cognition, 2) multi-agent design, 3) reinforcement frameworks, 4) existing datasets, and 5) emerging architectures. Our work fills this gap and highlights convergence points and research gaps. 1. We include the latest metrics, i.e., Error Localization Accuracy (ELA) [56], Depth-Wise Accuracy [57], Meta vs. Object-Level Accuracy [58], and Algorithm (AIA)/Malgorithm Identification Accuracy (MIA) [59], and datasets and evaluation pipelines. We also propose Summary Evaluation Quality (SEQ) as a universal summarization metric derived from the METAL [60] dataset. 1. Identification of core research challenges in MARL-enhanced meta-reasoning, including scalability (coordination bottleneck in multi-agent setups), reward hacking (model collapse in reinforcement-based training), energy efficiency (risk for autonomous adaptation), and ethical bias (in self-reinforcing feedback loops). 1. A forward-looking roadmap proposing neuroscience-inspired architectures episodic memory, uncertainty gating, and metacognitive control modules, symbolic-MAL hybrids to combine symbolic logic with adaptive MARL agents for interpretable reasoning and safety assurance, and dynamic agent configurations to enable trustworthy, self-correcting AI. In this survey paper, we start by reviewing basic fundamental concepts of meta-thinking and meta-learning, followed by an in-depth discussion of current single-agent and multi-agent approaches to enable self-reflective reasoning in LLMs. Continuoing the discussion of multi-agent, we then examine various MARL strategies that support meta-reasoning, including reward mechanisms, self-play with adversarial learning. To evaluate these advances, we present key metrics, datasets, and comparative studies. Table I provides a comprehensive comparison of previous survey papers on meta-reasoning in LLMs, highlighting how our work uniquely integrates all five critical aspects as described in contribution no. 3. Finally, we identify open research challenges and future directions that highlight the potential of neuroscience-inspired models and hybrid MARL-symbolic reasoning systems. This survey aims to provide a comprehensive roadmap for researchers seeking to develop more introspective, trustworthy, and adaptive LLMs. TABLE I: Comparison of Survey Papers on Meta-Reasoning in LLMs II Background II-A Meta-Thinking and Meta-Learning Meta-thinking is the process of reflecting and analyzing one’s own thinking. It is also highly associated with meta-cognition, which typically involves awareness, monitoring, and regulation of one’s cognitive activities. Meta-learning, or ”learning to learn,” addresses the development of strategies enabling a system to easily move into new tasks by employing previous experiences. In a meta-learning configuration, an LLM may learn to change its output based on feedback from earlier interactions and, therefore, increase its performance when working on novel tasks [30]. A prominent example is MetaICL (Meta-training for In-Context Learning) [61], where a pre-trained language model is tuned on a diverse collection of tasks in the form of input-output sequences. During this meta-training phase, the model learns to interpret and respond to new tasks presented in a similar sequence format. If afterwards it sees a task that it has never encountered before, it can employ the few examples to figure out how to proceed without needing to be retrained. This approach is good to utilize if the new task is quite different from what it was trained on. II-B Meta-Thinking in LLMs: Meta-reasoning in LLMs is essential to make LLMs adapt to new challenges. With the capability of self-reflection mechanisms, LLM can realize its own limitations and make changes in its reasoning strategies accordingly [51]. This is especially required for LLMs in dynamic environments where context and requirements can change rapidly. For example, when an LLM discovers that its output does not fully respond to a question posed by a user, meta-thinking processes can trigger a re-evaluation of the response or even request additional context. This kind of self-awareness in LLMs not only improves accuracy but also trustworthiness in high-stakes settings such as legal or clinical setups [62]. In these environments, even a small mistake can have serious consequences, such as a wrong legal interpretation or an incorrect medical suggestion. If the LLM can spot gaps in its answers, explain its reasoning, or express uncertainty when it’s not sure, people are more likely to trust the LLM’s response. For example, if an LLM response overlooks a critical symptom in a clinical use case, the meta-reasoning framework allows it to detect the oversight, reconsider the input, and revise its recommendation, refining accuracy and trust in high-stakes decision-making. Current research has explored these ideas, suggesting that more sophisticated meta-thinking frameworks in LLMs can mitigate issues like hallucinations, where models produce factually incorrect or misleading information. Despite significant advancements in language modeling, LLMs still face challenges in achieving true meta-thinking. Although today’s models can do basic self-assessment, such as estimating confidence levels or marking uncertainty in responses. They lack the meta-cognitive ability that is required for thorough self-evaluation, error correction, and adaptive reasoning. For example, research in [63] found that although LLMs can flag ambiguous queries, they struggle to engage in robust internal reasoning necessary for full meta-cognition. This gap leads to practical issues, such as the generation of hallucinated content, giving the need for further research to build LLMs that can more effectively reflect on and refine their thought processes. II-C Role of MARL in Enhancing Meta-Thinking in LLMs The integration of meta-thinking, meta-cognition, and meta-learning in LLMs is an important step to build more robust and adaptive language models. The emergence of MARL systems presents a promising direction to further improve meta-cognitive capabilities by enabling LLMs to reflect on their internal reasoning, adapt to new challenges, and collaboratively evolve their strategies [64] of cross-checking the responses. As research in these areas progresses, combining these ideas opens the door to building the next generation of language models that are not only smarter but also more self-aware and trustworthy. For example, framework proposed in [65] allows LLMs to predict and identify the action of other agents in a multi-agent system. The framework includes a ToM component that enables the model to make assumptions and refine its assumptions about other agents’ strategies. The LLM makes a guess, observes the outcome, and then adjusts its assumptions accordingly. This continous process of learning allows the LLM to improve its decision-making capabilities over time. Through repeated experiments, the model learns to make better decisions, i.e., demonstrating a more advanced level of reasoning, similar to how humans improve through experience. III Meta-Thinking Frameworks This section discusses three primary categories of methodologies to install meta-thinking in LLMs: (1) Single-Agent Methods, (2) Multi-Agent Architectures for Meta-Thinking, and (3) Emerging Methods in Self-Improvement. While each approach varies in design and scope, they collectively highlight an evolving research trend to enable models to reason about their reasoning processes. III-A Meta-Thinking Techniques using Single-Agent Methods Single-agent method helps to develop meta-thinking in LLMs through the methods of self-distillation, reflective prompting, and CoT reasoning. In self-distillation, an LLM generates ”teacher” responses that guide a ”student” version of itself towards improved performance [66]. Unlike supervised learning, which is based on external labeled data (e.g., human-annotated responses), self-distillation enables the model to learn from its own outputs. For example, in supervised learning, a model may be trained to respond to a question based on a dataset in which the human-annotated correct responses are given. In contrast, with self-distillation, the model responds to the question by itself, then bootstraps its future responses by learning from its previous (and potentially better) output. New forms of self-distillation aim to improve the model’s own reasoning abilities [67]. As LLMs can detect inconsistencies or mistakes and modify future responses accordingly by repeatedly going back over their responses. Symmetrically, reflective prompting has also become another means of inducing meta-thinking in LLMs [68]. It involves prompting an LLM to generate explicit ”reflections” of its reasoning steps in the guise of a narrative of how it arrived at some conclusion. These reflections not only provide insight into the model’s thought process but also a chance for the model to critique or revise earlier steps. Aside from reflection-based prompting, CoT reasoning explicitly breaks down difficult problems into intermediate steps [4] and uncovers the decision-making paths of the LLMs. We illustrate the CoT workflow with a simple arithmetic example from [4]. Figure 3 shows how an LLM walks through initial reasoning, intermediate steps, self-check, and revision before finalizing the answer. But recent studies show that LLMs struggle with self-evaluation, they often stick to their first line of thinking, even if it’s wrong, and won’t go back to question their steps unless you clearly ask them to do so [69]. Despite all these advances, there remain limitations in self-improvement and self-assessment. If internal feedback loops are not properly controlled, single-agent methods reinforce flawed patterns of reasoning as a side effect. The model’s ability to identify inconsistency or correct a poor line of argument may therefore be limited by the scope and volume of its own internally derived data [70]. For example, a language model trained on biased financial data can consistently overestimate market stability, reinforcing incorrect predictions without external verification. These constraints highlight the need for more robust systems, e.g., external input or multi-agent processes, to mitigate the inherent risks of the ”echo chamber” phenomenon of self-enhancing methods. <details> <summary>2504.14520v1/x2.png Details</summary>  ### Visual Description # Technical Document: Flowchart Analysis ## Overview The image depicts a **flowchart** illustrating a hierarchical agent-based system for task execution and adaptation. The system emphasizes **meta-cognition**, **communication**, and **feedback loops** to optimize task outcomes. Below is a detailed breakdown of components, flows, and textual annotations. --- ## Key Components 1. **Task/Problem** - **Role**: Input/output node representing the problem to be solved. - **Placement**: Appears at both the start and end of the flowchart, indicating cyclical interaction. 2. **High Level Agent** - **Role**: Orchestrates task decomposition and strategic decision-making. - **Functions**: - *"Decides task breakdown"* (directly connected to Task/Problem). - *"Meta-thinking: Makes strategic decisions"* (feedback loop to itself). - **Connections**: - Receives input from Task/Problem. - Sends directives to Low Level Agents via **Communication/Information Sharing** (dashed line). 3. **Low Level Agents** - **Role**: Executes specific subtasks. - **Functions**: - *"Executes tasks"* (directly connected to Task/Problem). - *"Reasoning: Handles task execution"* (feedback loop to itself). - **Connections**: - Receives instructions from High Level Agent. - Sends outcomes back to Task/Problem. 4. **Feedback Loops** - **Meta-thinking**: - *"Makes strategic decisions"* (curved arrow from High Level Agent to itself). - **Reflection and Adaptation**: - *"Improves task execution by reflecting on outcomes and adapting strategies"* (curved arrow from Low Level Agents to itself). - **Reasoning**: - *"Handles task execution"* (curved arrow from Low Level Agents to itself). 5. **Communication/Information Sharing** - **Role**: Facilitates data exchange between High and Low Level Agents. - **Visual Cue**: Dashed line with bidirectional flow. --- ## Flowchart Structure 1. **Primary Path**: - **Task/Problem** → **High Level Agent** → **Low Level Agents** → **Task/Problem**. - Represents the core workflow: problem decomposition, execution, and feedback. 2. **Feedback Mechanisms**: - **Meta-thinking** (High Level Agent): Enables strategic adjustments. - **Reflection and Adaptation** (Low Level Agents): Optimizes execution based on outcomes. - **Reasoning** (Low Level Agents): Ensures logical task handling. 3. **Cyclical Nature**: - The system iterates between task execution and strategy refinement, emphasizing **adaptability**. --- ## Textual Annotations - **Theory of Mind (ToM)**: - *"Predicting and adjusting based on the low-level agent's strategies"* (top-left speech bubble). - **Meta-thinking**: - *"Makes strategic decisions"* (curved arrow label). - **Reflection and Adaptation**: - *"Improves task execution by reflecting on outcomes and adapting strategies"* (right speech bubble). - **Communication**: - *"Information Sharing"* (dashed line label). - **Reasoning**: - *"Handles task execution"* (bottom curved arrow label). --- ## Diagram Flow 1. **Start**: Task/Problem is input into the High Level Agent. 2. **Decomposition**: High Level Agent breaks down the task. 3. **Execution**: Low Level Agents perform subtasks. 4. **Feedback**: - Outcomes are fed back to Task/Problem. - Meta-thinking and Reflection loops refine strategies. 5. **Iteration**: The cycle repeats with updated strategies. --- ## Notes - **Language**: All text is in English. - **Visual Style**: Black-and-white flowchart with dashed lines for communication and curved arrows for feedback loops. - **No Numerical Data**: The diagram focuses on conceptual relationships rather than quantitative metrics. This flowchart emphasizes **hierarchical coordination**, **adaptive learning**, and **dynamic strategy adjustment** in agent-based systems. </details> Figure 2: The diagram illustrates a multi-agent system where a high-level agent breaks down tasks and communicates with low-level agents to execute them. The high-level agent predicts and adjusts strategies using ToM, while low-level agents provide feedback through task execution. Reflection and adaptation enable continuous improvement by refining strategies based on outcomes. III-B Meta-Thinking Techniques using Multi-Agent Methods To address the limitations of single-agent feedback loops, researchers have turned to multi-agent architectures. One important architecture is the Supervisor-Agent Architecture, in which a high-level ”supervisor” agent manages multiple lower-level agents, each being an expert at a particular form of reasoning or problem-solving [71]. The hierarchical structure facilitates the division of work: lower agents provide proposals or intermediate reasoning steps, and a supervisor agent reorganizes, evaluates, and may override ill proposals. These hierarchical systems tend to produce more uniform and understandable outputs, observed in experimental research [72]. The overall structure of such a system is illustrated in Figure 2, where a high-level agent uses ToM to assign tasks and adapt strategies based on the feedback from low-level agents. The agents then execute the subtasks and report back results, enabling the system to be capable of looking back, learning, and adapting future decisions. Another multi-agent strategy is Agent-Based Debate and Self-Critique, inspired by the idea that competing perspectives can uncover hidden errors in reasoning [73]. When multiple specialized agents critique or defend one another, LLMs generate more robust solutions. For example, debate-based LLM systems encourage adversarial questioning, compelling each agent to justify or refine its positions [74]. An extension of this paradigm includes role-playing agent systems, such as AutoGPT [75]. In these systems, multiple instantiated personas (e.g., a “Planner” agent, an “Evaluator” agent, and a “Research” agent) interact within a shared environment to refine outputs iteratively. This interplay creates a meta-thinking layer, where the LLM-as-agents collectively reason, critique, and synthesize knowledge beyond what a single-agent system might achieve. III-C Emerging RL-Based Methods Both from the single-agent and multi-agent frameworks, RLHF is a crucial method for guiding LLMs toward more reliable meta-thinking [76, 77]. RLHF is a process of refinement in which the LLM is trained not only on labeled content, but also on human preferences, most commonly through ranked responses, to make its output align with desired qualities such as truthfulness, coherence, and ability for self-correction [78]. Instead of relying on the signals generated by the model only, RLHF exploits the responses of human annotators in a bid to induce desired behaviors such as coherent explanation, honesty, or self-correcting behavior, and penalizes undesirable characteristics such as hallucinations or self-contradictions. By iterative tuning, LLMs are trained to incorporate both external human decisions and internal mechanisms of reflection, eventually increasing their ability for reflection and error identification. For example, OpenAI’s ChatGPT-4 is trained using RLHF, where human evaluators label their responses as clear, precise, and aligned with ethical standards [77]. When the model generates deceptive or biased content, human annotators provide corrective feedback, which instructs the model to better its subsequent output. Through a series of rounds of iterations, this enhances the model’s ability to identify and eliminate inconsistencies to generate more believable responses. Another innovation is adaptive self-rewarding systems, which blend intrinsic motivations (e.g., consistency, curiosity) with extrinsic signals (e.g., supervisor-agent or human feedback). For instance, researchers [79] present a system where an LLM rewards itself with ”meta rewards” for showing improved internal consistency or discovering new solution paths in a reasoning task. When used in multi-agent environments, where AI models collaborate or compete, the self-rewarding approach can help overcome common limitations of single-agent systems, which often get stuck in repetitive or suboptimal reasoning patterns. An AI model strengthens its reasoning skills and becomes more adaptable over time by continuously evaluating and refining its own reward system. These emerging methods underscore the ongoing shift from static, single-entity LLMs toward dynamic, interactive systems capable of sustained introspection. By integrating human feedback, hierarchical supervision, and intrinsic self-rewarding signals, researchers are progressively constructing AI systems that can more effectively critique, adapt, and refine their own reasoning processes. <details> <summary>2504.14520v1/x3.png Details</summary>  ### Visual Description # Self-Check / Error Detection Workflow Diagram ## Overview The diagram illustrates a multi-step reasoning and error-checking process for solving mathematical problems. It includes decision points, iterative revisions, and final answer validation. --- ## Key Components & Flow ### 1. **Start / User Query** - **Input**: User-submitted problem statement. - **Example**: *"Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?"* --- ### 2. **Generate Initial Reasoning** - **Process**: Create an initial chain-of-thought (CoT) prompt to solve the problem. - **Output**: Initial reasoning steps (not explicitly shown in the diagram). --- ### 3. **Intermediate Reasoning Steps** - **Process**: Execute the CoT prompt to derive intermediate calculations. - **Example Output**: *"The answer is 27. X"* (Indicates an incorrect intermediate result with an error marker "X".) --- ### 4. **Self-Check / Error Detection** - **Decision Point**: - **Yes (Error Detected)**: - Flow: Return to **Revise Steps**. - **No (No Error)**: - Flow: Proceed to **Finalize the Answer**. --- ### 5. **Revise Steps** - **Process**: Iteratively refine the reasoning to correct errors. - **Loop**: Connects back to **Intermediate Reasoning Steps** for re-evaluation. --- ### 6. **Finalize the Answer** - **Process**: Confirm the corrected answer after error resolution. - **Example**: *"The cafeteria had 23 apples originally. They used 20 to make lunch. So they had 23 - 20 = 3. They bought 6 more apples, so they have 3 + 6 = 9. The answer is 9. ✅"* (Indicates a correct final answer with a green checkmark.) --- ### 7. **Return Final Response** - **Output**: Deliver the validated answer to the user. --- ## Visual Elements - **Color Coding**: - **Blue Boxes**: Process steps. - **Red "X"**: Error marker (e.g., "The answer is 27. X"). - **Green "✅"**: Correct answer marker (e.g., "The answer is 9. ✅"). - **Arrows**: Define the directional flow between steps, including loops for revisions. --- ## Example Problems Embedded in Diagram 1. **Tennis Balls Problem**: - Initial incorrect answer: 27 (marked with "X"). 2. **Cafeteria Apples Problem**: - Correct answer: 9 (marked with "✅"). --- ## Technical Notes - **Decision Logic**: - The flowchart emphasizes iterative error correction, with a feedback loop between **Self-Check/Error Detection** and **Revise Steps**. - **Validation**: - Final answers are explicitly validated with visual markers ("X" for errors, "✅" for correctness). - **Scalability**: - The process can handle complex reasoning tasks requiring multiple revisions. --- ## Diagram Structure ```mermaid graph TD A[Start / User Query] --> B[Generate Initial Reasoning] B --> C[Intermediate Reasoning Steps] C -->|Yes| D[Revise Steps] C -->|No| E[Finalize the Answer] D --> C E --> F[Return Final Response] ``` </details> Figure 3: A COT flowchart adapted from the arithmetic examples in Wei et al. [4]. The model uses explicit reasoning steps, checks for errors, and revises before returning the final result. IV MARL Strategies for Meta-Thinking Meta-thinking in LLMs heavily relies on the architecture, presentation, and integration of reinforcement signals. Recent research suggests that well-structured reward mechanisms in RL, strategic self-play in MAS, and continuous meta-learning can significantly enhance a model’s ability to reflect on and improve its own reasoning and decision-making processes [80, 81]. This section, as shown in Figure 4 describes three key strategies of RL solutions to meta-thinking: 1) Designing Intrinsic and Extrinsic Meta-Rewards i.e. for balancing human feedback with curiosity driven reward to drive self reflection, 2) Collaborative Self-Play and Adversarial Training i.e. agent debase, collaborate or challenge each other to uncover hidden flaws and deepen reasoning, and 3) Meta-Learning for Continuous Adaptation i.e. fast adaptation to new tasks by learning “how to learn” across different domains. IV-A Designing Intrinsic and Extrinsic Meta-Rewards Designing rewards that encourage meta-thinking is about finding a balance between intrinsic and extrinsic incentives. Extrinsic rewards come from external sources, such as completing tasks or receiving human feedback, which traditional RL frameworks prioritize [82]. Intrinsic reward refers to a signal generated within the system as a substitute for external (extrinsic) reward from the world. It is generally based on quantities like novelty, prediction error, or gained information, and is designed to drive behavior like exploration, learning from curiosity, and policy optimization in scenarios where extrinsic feedback is sparsely timed, delayed, or unhelpful. Traditionally, RL frameworks for language modeling concentrate on extrinsic cues, such as achieving a task or receiving human feedback to guide the model updates [82]. Researchers increasingly believe that intrinsic motivation is the key to helping language models refine their reasoning [83]. For instance, a model can reward itself for detecting and correcting contradictions in its thought process, fostering active error detection, self-diagnosis, and continuous learning. To make these reward systems effective, strict criteria are established to determine and evaluate the correctness, coherence, and novelty of the model’s responses [80]. For example, a recent study by [84] introduces a novelty bonus, an intrinsic reward criterion that incentivizes the generation of creative or less conventional outputs. This approach helps discourage repetitive or stereotypical responses that lack depth. Collectively, such reward mechanisms not only improve task performance but also foster a more reflective reasoning style, where the model learns to evaluate and enhance its own thought process. As discussed in [85], Intrinsic and extrinsic meta-rewards in LLMs involves combining external task signals with internal self-assessment signals into a single training objective. Concretely, at each step $t$ , the model receives an extrinsic reward $r^{e}_{t}$ (e.g., human preference or task completion) and an intrinsic reward $r^{i}_{t}$ (e.g., novelty, coherence, or contradiction-detection), and forms the meta-reward $$ R_{t}=\lambda\,r^{e}_{t}+(1-\lambda)\,r^{i}_{t},\quad\lambda\in[0,1], \tag{1} $$ where: $\lambda∈[0,1]$ is a scalar balancing the two reward components. For training stability, extrinsic rewards are normalized across the batch using the mean $\mu$ and standard deviation $\sigma$ , following Group Relative Policy Optimization (GRPO) [86]: $$ \tilde{r}^{e}_{i}=\frac{r^{e}_{i}-\mu}{\sigma}, \tag{2} $$ where $\mu$ and $\sigma$ are the batch-wise mean and standard deviation of extrinsic rewards $\{r^{e}_{i}\}$ [82]. Intrinsic rewards can be generated by the language model itself. For instance, an “LLM-as-judge” [42] framework evaluates each generated token $y_{t}$ in context $y_{<t}$ using an internal judgment function $J(·)$ : $$ r^{i}_{t}=J\left(y_{t}\mid y_{<t}\right), \tag{3} $$ with these token-level scores distilled into a lightweight reward model through asynchronous feedback mechanisms [83]. The policy $\pi_{\theta}(a_{t}\mid s_{t})$ , parameterized by weights $\theta$ , defines the probability of selecting action $a_{t}$ given the current state $s_{t}$ . In language models, $s_{t}$ typically represents the current dialogue context or hidden state. The policy is optimized to maximize the expected discounted return: $$ \theta^{*}=\arg\max_{\theta}\;\mathbb{E}_{\pi_{\theta}}\!\left[\sum_{t=0}^{T}% \gamma^{t}\left(\lambda\,r^{e}_{t}+(1-\lambda)\,r^{i}_{t}\right)\right], \tag{4} $$ where: $\gamma∈[0,1]$ is the discount factor applied to future rewards, $s_{t}$ is the current state (e.g., dialogue history or embedding context), $a_{t}$ is the action taken (e.g., token generation), $\pi_{\theta}$ is the policy distribution used by the model to choose actions. This formulation enables training language models that align with human objectives ( $r^{e}_{t}$ ) while also leveraging self-generated critiques ( $r^{i}_{t}$ ) to improve coherence, reasoning, and exploration. IV-B Collaborative Self-Play and Adversarial Training Multi-agent self-play has contributed significantly towards promoting emergent meta-think actions by focusing on the effectiveness of intrinsic and extrinsic reward schemes. In self-play setups, two or more replicas or implementations of an LLM either co-operatively or competitively interact with one another in a bid to solve sophisticated tasks [87]. LLMs engage in multi-turn argumentation to enhance reasoning, math problem-solving to verify logical correctness, code writing and debugging from adversarial collaboration, strategic game playing to develop long-term planning, and scientific hypothesis testing to enhance structured reasoning. A good example is the study in [88], where the multi-agentic system achieved state-of-the-art performance in the complex strategy game Diplomacy by using a combination of LLMs and planning algorithms to negotiate, ally, and betray human players. The system used self-play training to reinforce its strategic communication and decision-making iteratively, demonstrating how interaction among multiple agents can give rise to emergent behavior that would not be observed in a single-agent setting. In this setup of multi-turn argumentation, LLMs learn to reason more strategically when they engage with each other, especially when encouraged to criticize or outperform each other. This process uncovers hidden strategies and develops advanced reasoning abilities. Therefore, models become better at anticipating challenges, identifying fallacies in arguments, and refining their responses based on counterexamples given by their opponents [89]. This give-and-take interaction enhances their power to reason, learn, and develop over time. A particular subfield of self-play is adversarial training, where adversarial agents generate test questions or scenarios designed to expose flaws in the primary model’s logic [90]. These adversarial examples range from subtle inconsistencies in logic, such as asking a model to explain why ”a square has three sides,” to more complex contradictions, such as presenting contradictory historical timelines and asking the model to reconcile them. In mathematical reasoning, an adversarial agent might provide a subtly incorrect proof and ask the model to prove or disprove it. LLMs develop a more robust and self-skeptical form of reasoning—one that is better able to detect and revise subtle errors before LLMs have a chance to propagate by being iteratively exposed to these adversarial examples. This cycle of challenge and revision not only improves task performance but also reinforces meta-cognitive capacities by compelling the model to continually re-evaluate its own internal reasoning [91]. As in [90], Attacker $\mathcal{A}_{\theta}$ (a function parameterized by $\theta$ representing the attacker policy) takes as input a hidden target word $w$ (the correct answer in the language game) and generates a prompt $x$ (a natural language cue used to guide the defender), such that: $$ x=\mathcal{A}_{\theta}(w) $$ Defender $\mathcal{D}_{\theta}$ (sharing the same parameters $\theta$ , acting as the defender agent) attempts to recover the original target word by interpreting the prompt to produce a predicted word $\hat{w}$ : $$ \hat{w}=\mathcal{D}_{\theta}(x) $$ A scalar reward function $R(w,\hat{w})∈\mathbb{R}$ (which quantifies the success of the defender’s inference) is defined based on prediction accuracy: $$ R(w,\hat{w})=\begin{cases}+1,&\text{if }\hat{w}=w\text{ (successful inference)% }\\ -1,&\text{otherwise}\end{cases} $$ Model parameters $\theta$ are updated using reinforcement learning, such as Proximal Policy Optimization (PPO), to maximize the expected reward over a distribution of target words $\mathcal{W}$ (i.e., the task dataset), where $\alpha$ is the learning rate and $∇_{\theta}$ denotes the gradient with respect to parameters: $$ \theta\leftarrow\theta+\alpha\nabla_{\theta}\mathbb{E}_{w\sim\mathcal{W}}\left% [R(w,\mathcal{D}_{\theta}(\mathcal{A}_{\theta}(w)))\right] $$ By this, self-play, LLMs refine both their prompt generation and inference strategies under adversarial conditions, which leads to improved reasoning and meta-cognitive performance [90]. IV-C Meta-Learning for Continuous Adaptation Above discussed reward mechanism and adversarial training can cause meta-thinking in the short run, for example, self-play-based RL has been shown to improve reasoning in mathematical problem-solving, where models iteratively refine their logic by engaging in adversarial debate, as discussed by [81]. Meta-RL aims to enable fast adaptation to novel knowledge and reasoning patterns in the long run by helping models rapidly generalize to novel reasoning patterns and unseen knowledge. For example, [43] introduced a meta-learning framework where LLMs track past reasoning failures and dynamically refine their strategies and allow LLMs to adapt to new challenges, such as transitioning from scientific reasoning to legal document analysis, without retraining [92]. In meta-learning, an LLM is trained to quickly optimize its inner parameters in response to new tasks or domains, in a way learning ”how to learn.” Such a shift is especially crucial for LLMs that must be versatile in ever-changing environments where data distributions or objectives may change abruptly. The core idea of meta-RL is to create a feedback learning loop in which the model receives reinforcement not only from outcomes of tasks but also from meta-reward signals that evaluate improvements in reasoning efficiency, adaptability, and coherence across tasks. Such reinforcement can be shaped by indicators like reduced prediction error over time, faster convergence on new tasks, or successful reuse of past reasoning paths in new domains [81, 93]. Overall, meta-learning RL unlocks a lifelong learning paradigm for LLMs, where agents are not just trained to complete tasks, but are continuously evolving learners. This capability is essential for fostering deep, reflective, and general-purpose meta-thinking abilities in language models operating in dynamic, high-variance environments. Real-world applications of meta-learning in LLMs are often few-shot and zero-shot learning boosts that allow models to generalize from a limited number of examples to completely novel tasks [93]. By enhancing the meta-learner, researchers can enhance its ability to modify its reasoning process or reward signals from a few supervised instances. This has led to remarkable improvements in the trust and depth of reasoning of LLMs [81]. For example, a recent study [94] introduced Micre (Meta In-Context Learning for Relation Extraction), which enhances LLMs’ few-shot and zero-shot learning. In in-context learning, LLM generalizes on the few examples, meta learning principle teach the model to ”learn how to learn” from the examples more effectively. Micre meta-train the LLMs to learn to adapt quickly with fewer examples. In Micre, LLM is not training just to perform tasks but also learning to adapt to new tasks. By reformulating relation extraction as a natural language generation task and meta-training over a broad variety of datasets, Micre does significantly better than traditional fine-tuning methods. As discussed in [94], In meta-RL, the model learns an adaptive policy $\pi_{\theta}$ parameterized by $\theta$ , with the goal of fast adaptation to new tasks. Meta-training phase: The model is trained over a distribution of tasks $\mathcal{T}\sim p(\mathcal{T})$ . For each task $\mathcal{T}_{i}$ , the model performs inner-loop adaptation using a small dataset $\mathcal{D}^{\text{train}}_{i}$ : $$ \theta_{i}^{\prime}=\theta-\alpha\nabla_{\theta}\mathcal{L}_{\mathcal{T}_{i}}^% {\text{train}}(\theta) $$ where $\theta_{i}^{\prime}$ are the task-adapted parameters, and $\alpha$ is the inner-loop learning rate. Meta-objective: Update the original parameters $\theta$ based on performance over a validation set $\mathcal{D}^{\text{test}}_{i}$ : $$ \theta\leftarrow\theta-\beta\nabla_{\theta}\sum_{\mathcal{T}_{i}\sim p(% \mathcal{T})}\mathcal{L}_{\mathcal{T}_{i}}^{\text{test}}(\theta_{i}^{\prime}) $$ where $\beta$ is the meta (outer-loop) learning rate. At inference time, given a small support set $\mathcal{S}$ (few-shot examples), the model produces a prediction $\hat{y}$ conditioned on in-context demonstrations: $$ \hat{y}=\mathcal{M}_{\theta}(x\mid\mathcal{S}) $$ Here, $\mathcal{M}_{\theta}$ learns to perform task adaptation internally by interpreting $\mathcal{S}$ during inference. This formulation enables the LLM to generalize to new reasoning patterns and domains with minimal data and no parameter updates. <details> <summary>2504.14520v1/x4.png Details</summary>  ### Visual Description # Multi-Agent Reinforcement Learning Strategies for Meta-Thinking ## Overview This diagram illustrates a framework for enhancing meta-reasoning and adaptability in Large Language Models (LLMs) through three core strategies: **Reward Mechanisms**, **Self-Play & Adversarial Training**, and **Continual Learning**. Each strategy contributes to improved agent performance via distinct mechanisms. --- ## 1. Reward Mechanisms ### Labels & Descriptions - **Extrinsic** - **Description**: Rewards for correct task completion or feedback from external sources (e.g., human input). - **Intrinsic** - **Description**: Internal motivations such as curiosity, error detection, or creative problem-solving. ### Example - **Example**: An LLM agent rewards itself when it identifies and corrects its own mistakes. --- ## 2. Self-Play & Adversarial Training ### Labels & Descriptions - **Self-play** - **Description**: LLMs collaborate to solve problems through iterative interaction. - **Adversarial** - **Description**: LLMs challenge each other with deceptive or complex questions to improve robustness. ### Example - **Example**: An LLM learns by identifying incorrect answers generated by another agent. --- ## 3. Continual Learning ### Labels & Descriptions - **Track changes** - **Description**: Agents learn from historical data to adapt to evolving environments. - **Few/Zero-shot** - **Description**: LLMs generalize to new tasks with minimal or no additional training examples. ### Example - **Example**: An LLM transitions from science to law questions without full retraining. --- ## Outcomes All three strategies converge to enable **Improved Meta-Reasoning & Adaptability**, allowing LLMs to: 1. Reflect on and refine their decision-making processes. 2. Generalize across diverse domains and tasks. 3. Adapt dynamically to novel challenges with limited data. --- ## Diagram Flow 1. **Reward Mechanisms** → **Improved Meta-Reasoning** 2. **Self-Play & Adversarial Training** → **Improved Meta-Reasoning** 3. **Continual Learning** → **Improved Meta-Reasoning** All pathways feed into the final outcome of enhanced adaptability and reasoning capabilities. --- ### Key Trends & Data Points - **Extrinsic vs. Intrinsic Rewards**: Combines external validation with internal motivation for holistic learning. - **Self-Play**: Emphasizes collaborative problem-solving. - **Adversarial Training**: Focuses on stress-testing agents to improve resilience. - **Continual Learning**: Highlights efficiency in adapting to new tasks with minimal data. This framework underscores the synergy between reinforcement learning strategies and meta-cognitive development in LLMs. </details> Figure 4: Overview of RL Techniques Enabling Meta-Thinking in Language Models V Evaluation Metrics, Comparative studies and Datasets Assessing meta-reasoning in LLMs is key to developing systems that reason, self-correct, and adapt like humans. To achieve this, researchers have developed targeted metrics that challenge models’ ability to introspect and revise their own reasoning. This section introduces key evaluation metrics for assessing meta-reasoning in LLMs, followed by a summary of experimental results from studies comparing MARL in LLMs. Lastly, it presents specialized datasets designed to support and enhance meta-reasoning evaluation as shown in Table II. V-A Metrics for Evaluating Meta-Thinking in LLMs Many metrics have been proposed to assess LLM’s ability for meta-thinking, including 1) logical consistency, 2) self-correction ability, and 3) reasoning depth. Logical‑consistency metrics [95, 96] measure whether a model’s multi‑step reasoning stays contradiction‑free, for example, correctly deriving $6kg$ for $2kg×3kg$ and not later claiming the total is 8kg. Self-correction ability is a characteristic of models equipped with robust meta-thinking, enabling them to detect and rectify their own errors. Self-correction can be measured by injecting small perturbations or contradictions into prompts and observing if the model identifies the discrepancy and corrects itself [97]. This approach not only tests error detection but also probes the model’s ability to reflect on its output and make improvements. Reasoning depth assesses the degree to which a model is able to breakdown complex problems into systematic sub steps. For example, [4] recommends that one adds a COT score to determine whether a model’s intermediate reasoning steps provide sufficient detail and coherence. High depth of reasoning could indicate high meta-thinking, as it requires persistent monitoring and self-correction of one’s train of thought. ELA [56] measures how precisely an LLM can pinpoint mistakes in a candidate COT across 5,975 process-based questions. For each question, the LLM is shown an automatically generated CoT solution and asked to 1) classify it as correct/incorrect, 2) identify the step where the first error occurs, and 3) explain that particular error. The results in [56] shows that State-of-the-art (e.g. OpenAI’s o1 series [98]) achieve over 70% ELA, while many open-source LLMs remain below 45%, revealing a substantial gap in fine-grained error detection. Meta- vs. Object-Level Metrics [58] separate high-level planning from low-level inference: 1) Meta-Level Reasoning Frequency is the fraction of cases with explicit planning steps. 2) Object-Level Reasoning Accuracy is correctness on the detailed inference sub-steps. The results is [58] show that LLMs exhibit greater than 85% Meta-Level Frequency but drop to 60–70% on Object-Level Accuracy, indicating they can plan well but often face difficulty in the low-level inferences. Depth-Wise Accuracy [57] tracks performance drop as logical inference depth increases. The increase in the depth means that there is an increase in the number of chained inferences. For example, Depth 3 means “If A then B; if B then C; if C then D; given A and C, is D true?” The findings in [57] show that when Depth is 1, LLMs have an approximate average accuracy of 68%, but then when Depth becomes 5, the accuracy falls to 43% approximately. The paper [57] noted the trend that an LLM often fails at integrating conflicting premises. AIA and MIA [59] quantify an LLM’s ability to choose the correct rationale and to spot flawed (“malgorithm”) reasoning paths [59]. 1) AIA is the percentage of cases where the model picks the true rationale for the correct answer. 2) MIA is the percentage of cases where the model identifies the flawed rationale behind a wrong choice. The findings in the paper [59] LLMs achieve approximately 75% AIA but drop to approximately 45% MIA and show that LLMs struggle more with recognizing faulty reasoning. SEQ [60] measures the quality of the generated summary by the LLM model. It encompasses: 1) Linguistic Acceptability: To measure grammar accuracy and fluency. 2) Task Quality: Ability to accomplish the summarization task. 3) Output Content Quality: Content relevance and accuracy. 4) Hallucinations: Absence of constructed or ungrounded information. 5) Problematic Content: Avoid offensive, biased, or unsafe language. V-B Comparative Studies of MARL-Enhanced LLMs A growing body of comparative studies highlights the advantages of incorporating MARL into LLMs for improved meta-thinking: Hierarchical MARL for Meta-Reasoning: [43] introduce a multi-agent configuration where a ”supervisor” agent coordinates multiple ”worker” agents, each with partial sub-problems of a reasoning problem. Experiments on the StrategyQA [99] tasks demonstrate enhanced coherence and fewer contradictions compared to single-agent baselines. Agent-Based Debate and Self-Critique: Experiments of [100] examine adversarial debates between LLM ”defender” and ”prosecutor” agents. They note a dramatic enhancement of logical consistency scores—derived from the DebateQA dataset—following models’ exposure to several adversarial interactions. Coordination vs. Competition: [101] conduct ablation experiments to compare cooperative MARL settings with purely competitive ones. They find that cooperative approaches offer better measures of self-reflection (e.g., higher internal consistency), with competitive setups being superior at detecting subtle flaws. Periodic switches between cooperative and adversarial phases in hybrid strategies tend to yield the best trade-off, fostering both robust error detection and collaborative improvement. TABLE II: Summary of Evaluation Metrics, Comparative Studies, and Datasets for Meta-Reasoning in LLMs | Metric | Logical Consistency | Detects internal contradictions in a model’s COT, ensuring step-by-step coherence. | [95] [96] | | --- | --- | --- | --- | | Self-correction Ability | Measures whether a model spots and fixes injected errors or contradictions in its own output. | [97] | | | Reasoning Depth | Assesses granularity and completeness of intermediate reasoning steps in multi-step problems. | [4] | | | Error Localization Accuracy | How precisely an LLM pinpoints mistakes in reasoning chains (MR-Ben). | [56] | | | Meta-/Object-Level Metrics | Planning frequency vs. inference accuracy (Franklin Dataset). | [58] | | | Depth-Wise Accuracy | Performance drop as logical inference depth increases (Multi-LogiEval). | [57] | | | AIA / MIA | Correct rationale vs. flawed reasoning identification (MalAlgoQA). | [59] | | | Correlation Scores | Agreement between LLM evaluators and humans in multilingual summarization (METAL). | [60] | | | Comparative Study | Hierarchical MARL for Meta-Reasoning | Supervisor-worker hierarchies decompose tasks, improving coherence on StrategyQA. | [43, 99] | | Agent-Based Debate & Self-Critique | Defender vs. prosecutor debates boost logical consistency on DebateQA. | [100] | | | Coordination vs. Competition (Hybrid) | Alternating cooperative/adversarial phases balance reflection and flaw detection. | [101] | | | Dataset | BIG-Bench | 204 tasks probing reasoning, self-reflection, bias detection, and perspective-shifting. | [102] | | SciInstruct | STEM problems with built-in self-critique steps for scientific reasoning. | [103] | | | DebateQA | approximately 3,000 debate-style questions to test balanced argumentation. | [104] | | | StrategyQA | 2,780 implicit multi-step reasoning questions with annotated inference chains. | [99] | | | MR-Ben | 5,975 process-based questions measuring error localization in chains of thought. | [56] | | | Franklin Dataset | QA reframed into meta- vs. object-level tasks; tracks planning vs. inference. | [58] | | | Multi-LogiEval | Logical reasoning across 30+ rules at depths 1–5; reports depth-wise accuracy. | [57] | | | MalAlgoQA | Counterfactual QA with metrics for correct vs. flawed reasoning paths (AIA/MIA). | [59] | | | METAL | Multilingual, reference-free summarization evaluation; reports correlation scores vs. human judges. | [60] | | <details> <summary>2504.14520v1/x5.png Details</summary>  ### Visual Description # Technical Document Extraction: Bar Chart Analysis ## Chart Type - **Bar Chart** (Vertical) ## Axis Labels - **X-Axis (Categories):** `Dataset` Categories: 1. BIG-Bench 2. Scinstruc 3. DebateQA 4. StrategyQA 5. MR-Ben 6. Franklin 7. Multi-LogicEval 8. MalgoQA 9. METAL - **Y-Axis (Quantitative):** `Number of Papers` Scale: 0 to 250 (increments of 50) ## Data Points | Dataset | Approximate Number of Papers | |------------------|------------------------------| | BIG-Bench | 225 | | Scinstruc | ~10 | | DebateQA | ~5 | | StrategyQA | ~20 | | MR-Ben | ~5 | | Franklin | ~3 | | Multi-LogicEval | ~5 | | MalgoQA | ~3 | | METAL | ~5 | ## Key Trends 1. **Dominance of BIG-Bench:** - The dataset `BIG-Bench` accounts for **~90% of all papers** (225 papers), dwarfing all other datasets. 2. **Secondary Datasets:** - `StrategyQA` (~20 papers) is the second-largest dataset. - Remaining datasets (`Scinstruc`, `DebateQA`, `MR-Ben`, `Franklin`, `Multi-LogicEval`, `MalgoQA`, `METAL`) each have **<10 papers**, with most clustered around **3–5 papers**. 3. **Data Distribution:** - Highly skewed distribution, with a single outlier (`BIG-Bench`) and minimal representation for other datasets. ## Visual Elements - **Bars:** - All bars are **blue** (no legend present; uniform color). - Bars are ordered left-to-right by dataset name. - **Gridlines:** - Horizontal gridlines at 50-paper intervals (0, 50, 100, 150, 200, 250). - No vertical gridlines. ## Notes - No embedded text or annotations within the chart. - No legend or colorbar present. - Dataset names are explicitly labeled on the x-axis. </details> Figure 5: Number of published papers referencing each dataset for evaluating LLM meta-reasoning. V-C Existing Datasets Over the past few years, researchers have introduced specialized datasets and benchmarks to evaluate self-reflection and iterative reasoning in LLMs. Figure 5 illustrates the relative popularity of each dataset, measured by the number of published papers incorporating them in the experiment. Notably, BIG-Bench [102] appears in more studies compared to other datasets, reflecting its broad applicability in testing a wide range of language-model capabilities. In contrast, specialized datasets such as SciInstruct [103] and MR-Ben [56] have fewer overall citations but target deeper skills such as domain-specific scientific reasoning or process-based meta-cognition. DebateQA [104], StrategyQA [99], FRANKLIN [58], Multi-LogiEval [57], MalAlgoQA [59], and METAL [60] similarly occupy more focused niches within the broader landscape of LLM evaluation. The brief introduction of the dataset is given below. BIG-Bench [102] dataset is designed to ascertain the ability of language models to reason, think, and even question their own outputs. The dataset is comprised of 204 different challenges, ranging from solving mathematics and languages to recognizing bias and even creating stories. The interesting part of the dataset is that some of the tasks force LLMs to reflect on their thinking, like, ”Are you sure about your answer?” or ”Can you find a flaw in what you’ve stated?” For example, in one of the tasks, LLM systems are asked to justify a joke, a difficult task because humor involves world knowledge and reasoning. Another challenge involves the LLM responding to a question, and then rewriting its response from a different perspective. SciInstruct [103] is a dedicated data set to enhance the scientific reasoning ability of language models. It is based on subjects like physics, chemistry, mathematics, and formal proofs to enable the LLMs to resolve higher level scientific challenges. Self-reflection is one of the fundamental features of SciInstruct where LLMs learn to inspect and improve their own reasoning mechanisms. Through this self-criticism, the LLM becomes more accurate and reliable. By incorporating self-critical practices, SciInstruct aims to develop LLMs that can better understand and solve complex scientific problems. DebateQA [104, 6] is a set created to test the extent to which AI can answer tough questions in which nobody has one ”right” answer—just like real debates. It has nearly 3,000 questions with different human-sourced answers reflecting different sides of the issue. For example, the question ”Is climate change mainly caused by human activities? ” has some yes answers that point to causes like pollution and deforestation, and others that respond with no, citing events like volcanic eruptions or solar changes. The idea is to check if LLM understands that questions like these are contentious, and if it can present multiple viewpoints instead of one.”. StrategyQA [99] is a dataset used to test the ability of LLMs to respond to questions that involve implicit, multi-step reasoning. In contrast to simple questions, those in StrategyQA tend to require the LLMs to bridge several pieces of information to reach an answer. For example, the query ”Did Aristotle use a laptop?” requires understanding that Aristotle lived centuries ago and laptops are a modern invention, leading to the answer ”No.” The dataset consists of 2,780 of these kinds of questions, each with a related breakdown of the steps of reasoning and supporting evidence from Wikipedia. MR‑Ben [56] is a 5,975-question benchmark for meta‑reasoning: models must spot, explain, and fix errors in step‑by‑step solutions (e.g., swapping $x/y$ vs $y/x$ in Graham’s‑law effusion). Covering logic, chemistry, programming, and more, it tests deliberate “System‑2” thinking; results show GPT‑4‑o excels at self‑correction while most other models struggle, underscoring the need for stronger meta‑cognitive skills. FRANKLIN Dataset [58] access how LLMs handle two types of reasoning: meta-level (planning and strategizing) and object-level (actually doing tasks like calculations). For instance, a question such as ”Which European country had the highest GDP growth in 2023?” requires the model to devise a plan of action first (meta-level) and then fetch specific GDP data and compare (object-level). The study found that while LLMs are proficient at declaring strategies, they fall short when it comes to executing specific tasks, usually getting data fetching or calculations incorrect. Multi-LogiEval [57] spans 30+ logical inference rules report Depth-Wise Accuracy. The Multi-LogiEval dataset introduces a dataset to assess LLMs to determine whether LLMs can address complex, multi-step logical reasoning. The dataset includes problems across three logic types: 1) propositional logic (simple if-then statements), 2) first-order logic (objects and relations between them), and 3) non-monotonic logic (where conclusions can change on the addition of new information). For example, a question would be: ”If all birds fly, and penguins are birds, do penguins fly?” In which the model must use rules of logic to reach the correct conclusion. The study found that the LLMs’ accuracy is inversely proportional to the number of reasoning steps, showing challenges on the part of the LLMs to perform deep logical reasoning. MalAlgoQA [59] is a QA benchmark built to probe multi‑step reasoning. It contains 7 logic types (e.g., deductive, abductive, numerical, counterfactual) and poses multi‑context questions that require chaining facts e.g., “All machines are robots and RoboX is a machine, Is RoboX a robot?”, with AIA/MIA metrics distinguishing sound vs. flawed reasoning. METAL [60] offers 1000 GPT‑4–generated summaries (100 each in 10 languages) for reference‑free quality evaluation. Native speakers and LLMs (GPT‑3.5‑Turbo, GPT‑4, PaLM 2) rate grammar, fluency, and task relevance, enabling cross‑lingual correlation studies. VI Challenges and Open Problems VI-A Challenges Despite remarkable advances in the training of meta-thinkers with MARL, there are challenges. The challenges cut across computational constraints, system stability, and ethics, each with seminal research questions awaiting novel solutions. VI-A 1 Scalability and Stability in MARL for Meta-Thinking One of the central challenges with MARL-based meta-thinking is the computational burden of dealing with many agents. As each agent contributes to the entire reasoning framework, the framework is likely to become too heavy to manage successfully [105]. For instance, in hierarchical MARL frameworks, interactions between complex-task supervisors and low-level worker agents grow in number exponentially [43]. Therefore, training such systems requires substantial memory and processing resources, rendering these systems impractical for several real-world uses. In addition, coordination complexity is caused when multiple agents interact concurrently. Agents need to negotiate shared policies, exchange partial solutions, or provide criticism. This requires extensive message-passing protocols that, unless efficiently optimized, result in bottlenecks and synchronization issues. These scalability issues are usually solved by using techniques such as distributed training, hierarchical architecture, or inter-agent communication compression [106]. But how much such approaches compromise between computational complexity and robust meta-thinking remains questionable. MARL systems are also susceptible to mode collapse, a phenomenon where agents converge prematurely to suboptimal solutions, often because homogeneous strategies control the learning process [107]. This is especially problematic for meta-thinking tasks, for which diversity of perspective and self-criticism are essential to unearth hidden flaws. If all agents learn the identical flawed patterns of reasoning, then the potential for meta-thinking is severely diminished. Another such challenge is reward hacking, in which agents find loopholes to earn large rewards without actually improving reasoning [108]. On a meta-thinking scale, reward hacking can be considered inflation of self-correcting behavior agents can continue to add meaningless mistakes and ”correct” them to accomolate self-improvement rewards. Therefore, Stable training dynamics necessitate well-constructed reward functions that reflect improved meta-cognitive abilities. Researchers continue to study novel penalty functions, adaptive reward schedules, and curriculum learning methods to discourage exploitative strategies [109]. VI-A 2 Energy Efficiency and Resource-Aware Design in MAS MAS, particularly RL-driven MAS, are likely to experience high energy consumption challenges. Communication between the agents to synchronize is one of the main factors, which is particularly costly for large or highly dynamic environments [110]. RL agents, with computational demands, dynamic environment interaction, and policy updating requirements, add to the extra energy consumption [111]. Also, the majority of MAS remain to apply time-triggered data collection, where LLM agents sense and transmit data at intervals regardless of changes in the environment or the need for urgency in tasks. This leads to unnecessary data transmission and energy loss [112]. Task allocation is a second issue. Task allocation protocols tend to ignore the processing power or energy available to agents at a moment and therefore overload low-resource agents and waste energy on the system [113]. VI-A 3 Ethical and Safety Considerations As MARL-driven LLMs become more proficient at self-improvement, ethics and safety become paramount. Agents that can learn their own decision-making can also learn to perpetuate bias or disseminate hateful material if their reward signals or agent interactions reinforce unwanted patterns [114]. For example, if the supervisory agent of a model is derived from a biased dataset, it can propagate biased reward signals across the entire multi-agent system, which can then be compounded with existing prejudices. Additionally, giving unbiased self-improvement consistently is a non-trivial problem. When human critics provide real-time feedback, the danger of unconscious bias—cultural, gender, or racial—envelops the model’s objectives. Hence, designing open audit trails, high-quality fairness measures, and human-in-the-loop systems that detect and compensate for biases are research areas in active pursuit [115]. The ultimate aim is to reconcile the freedom and efficiency of MARL systems with demands for safety, responsibility, and inclusivity in AI implementation. VI-B Open Problems Based on the above issues, research is increasingly looking to interdisciplinary knowledge and hybrid solutions to augment meta-thinking skills. The next decade will witness progress on multiple fronts, from neuroscience-driven frameworks to more interpretable models that combine MARL with symbolic reasoning. VI-B 1 Neuroscience-Inspired Approaches for Meta-Thinking One optimistic avenue is inspiration from human cognition systems where meta-cognition occurs due to emergent interactions across numerous brain regions [116]. They have begun thinking about architectures mimicking human mechanisms of learning, such as in the form of episodic memory modules or gated information by attention—so the introspection capacity of an LLM can be enhanced [117]. Through the application of neuroscientific knowledge to agent construction, designers expect to enable more human-like self-awareness and flexible response to novel tasks. Such biologically motivated systems would be especially beneficial in high-stakes situations requiring robust problem-solving combined with awareness of one’s own mental activity. VI-B 2 Hybrid Models that Combine MARL and Symbolic Reasoning Another significant direction focuses on enhancing interpretability in AI reasoning by means of hybrid models integrating MARL and symbolic logic [118]. Symbolic modules can embody domain-specific rules or knowledge graphs, while MARL agents are responsible for discovering and applying these rules in a more adaptive manner. This integration has several advantages: symbolic reasoning can help to reduce reward hacking risk by applying logical constraints, and MARL can handle uncertainty or partial information best over strictly symbolic systems. For example, a multi-agent system can utilize symbolic constraints to verify consistency in each reasoning step, while RL agents cope with broader exploration of solution approaches [119]. Early testing indicates that these hybrid methods can also significantly improve transparency because human judges can trace back how a single symbolic inference led to better decision-making or self-correction [120]. Optimizing such hybrid designs and demonstrating their success in a broad range of tasks is a fine research area in the coming years. VI-B 3 Scaling Multi-Agent Architectures for General AI Finally, as multi-agent paradigms prove themselves in niche domains, researchers are moving away from LLMs to general multi-agent AI systems. The aim is to build adaptive agents capable of generating, arguing, and proving solutions for real-world problems such as robotics, finance, and healthcare [121, 122]. These domains require more than linguistic proficiency; they require robust situational awareness, sensor fusion, and real-time decision-making under uncertainty. Achieving General AI through multi-agent interaction involves increasing the number of agents, specialising agents for different tasks, and developing emergent ”societies” of agents that can dynamically self-adapt [101]. While the computational complexities are high, the reward potential is AI systems that not only learn solutions but also maintain improving collective problem-solving abilities in the face of changing problems. This wide-ranging vision of multi-agent cooperation and self-improvement heralds an exciting frontier in the search for advanced meta-thinking in artificial intelligence. VII Conclusion In this survey, we have systematically examined the emergence of meta‑thinking in LLMs through the lens of MARL. We began by diagnosing the key limitations of existing self‑reflection approaches such as, hallucinations, unchecked error propagation, and domain specificity, and then surveyed both single‑agent techniques (e.g., COT prompting, self‑distillation, RLHF) and multi‑agent architectures (supervisor‑worker hierarchies, adversarial debates, self‑play) that can endow LLMs with robust introspective capabilities. Building on these foundations, we detailed essential MARL strategies—designing intrinsic and extrinsic meta‑rewards, collaborative self‑play and adversarial training, and meta‑learning for continual adaptation that collectively allow LLMs to evaluate, refine, and adapt their own reasoning in real time. We also reviewed the state of evaluation metrics and datasets for meta‑reasoning, and identified persistent challenges around scalability, stability, energy efficiency, and ethical safety. Finally, we outlined a future roadmap encompassing neuroscience inspired architectures, hybrid symbolic MARL systems, and scalable multi‑agent “societies,” all aimed at realizing trustworthy, self‑correcting AI. We hope this comprehensive framework will guide future research toward LLMs that not only perform complex tasks but also continually think about and improve their own thought processes. References - [1] M. Karwowski, J. Dul, J. Gralewski, E. Jauk, D. M. Jankowska, A. Gajda, M. H. Chruszczewski, and M. Benedek, “Is creativity without intelligence possible? a necessary condition analysis,” Intelligence, vol. 57, pp. 105–117, 2016. - [2] T. Wu, J. Lan, W. Yuan, J. Jiao, J. Weston, and S. Sukhbaatar, “Thinking llms: General instruction following with thought generation,” arXiv preprint arXiv:2410.10630, 2024. - [3] A. Bilal, D. Ebert, and B. Lin, “Llms for explainable ai: A comprehensive survey,” arXiv preprint arXiv:2504.00125, 2025. - [4] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022. - [5] Z. Sprague, F. Yin, J. D. Rodriguez, D. Jiang, M. Wadhwa, P. Singhal, X. Zhao, X. Ye, K. Mahowald, and G. Durrett, “To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning,” arXiv preprint arXiv:2409.12183, 2024. - [6] M. A. Mohsin, A. Bilal, S. Bhattacharya, and J. M. Cioffi, “Retrieval augmented generation with multi-modal llm framework for wireless environments,” arXiv preprint arXiv:2503.07670, 2025. - [7] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, et al., “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” ACM Transactions on Information Systems, vol. 43, no. 2, pp. 1–55, 2025. - [8] S. K. Gallagher, J. Ratchford, T. Brooks, B. P. Brown, E. Heim, W. R. Nichols, S. Mcmillan, S. Rallapalli, C. J. Smith, N. VanHoudnos, et al., “Assessing llms for high stakes applications,” in Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice, 2024, pp. 103–105. - [9] Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, Y. Chen, et al., “Siren’s song in the ai ocean: A survey on hallucination in large language models,” arXiv preprint arXiv:2309.01219, vol. 2, no. 5, 2023. - [10] Z. Yin, “A review of methods for alleviating hallucination issues in large language models,” Applied and Computational Engineering, vol. 76, pp. 258–266, 2024. - [11] P. Singhal, T. Goyal, J. Xu, and G. Durrett, “A long way to go: Investigating length correlations in rlhf,” arXiv preprint arXiv:2310.03716, 2023. - [12] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015. - [13] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971, 2015. - [14] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025. - [15] J. Zhang, B. Li, X. Cao, M. Zhang, and W. Zhao, “Mixcl: Mixed contrastive learning for relation extraction,” in Pacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, 2024, pp. 92–103. - [16] N. Dziri, A. Madotto, O. Zaïane, and A. J. Bose, “Neural path hunter: Reducing hallucination in dialogue systems via path grounding,” arXiv preprint arXiv:2104.08455, 2021. - [17] X. Guan, Y. Liu, H. Lin, Y. Lu, B. He, X. Han, and L. Sun, “Mitigating large language model hallucinations via autonomous knowledge graph-based retrofitting,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, 2024, pp. 18 126–18 134. - [18] N. Varshney, W. Yao, H. Zhang, J. Chen, and D. Yu, “A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation,” arXiv preprint arXiv:2307.03987, 2023. - [19] L. Luo, Y.-F. Li, G. Haffari, and S. Pan, “Reasoning on graphs: Faithful and interpretable large language model reasoning,” arXiv preprint arXiv:2310.01061, 2023. - [20] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, et al., “Open problems and fundamental limitations of reinforcement learning from human feedback,” arXiv preprint arXiv:2307.15217, 2023. - [21] P. Manakul, A. Liusie, and M. J. Gales, “Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models,” arXiv preprint arXiv:2303.08896, 2023. - [22] S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, X. Li, A. Celikyilmaz, and J. Weston, “Chain-of-verification reduces hallucination in large language models,” arXiv preprint arXiv:2309.11495, 2023. - [23] X. Zhang, B. Peng, Y. Tian, J. Zhou, L. Jin, L. Song, H. Mi, and H. Meng, “Self-alignment for factuality: Mitigating hallucinations in llms via self-evaluation,” arXiv preprint arXiv:2402.09267, 2024. - [24] N. Mündler, J. He, S. Jenko, and M. Vechev, “Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation,” arXiv preprint arXiv:2305.15852, 2023. - [25] S. Choi, T. Fang, Z. Wang, and Y. Song, “Kcts: knowledge-constrained tree search decoding with token-level hallucination detection,” arXiv preprint arXiv:2310.09044, 2023. - [26] K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg, “Inference-time intervention: Eliciting truthful answers from a language model,” Advances in Neural Information Processing Systems, vol. 36, pp. 41 451–41 530, 2023. - [27] Y. Chuang, Y. Xie, H. Luo, Y. Kim, J. Glass, and P. D. He, “Decoding by contrasting layers improves factuality in large language models. arxiv 2023,” arXiv preprint arXiv:2309.03883. - [28] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al., “Self-refine: Iterative refinement with self-feedback,” Advances in Neural Information Processing Systems, vol. 36, pp. 46 534–46 594, 2023. - [29] K. Valmeekam, M. Marquez, and S. Kambhampati, “Can large language models really improve by self-critiquing their own plans?” arXiv preprint arXiv:2310.08118, 2023. - [30] M. Griot, C. Hemptinne, J. Vanderdonckt, and D. Yuksel, “Large language models lack essential metacognition for reliable medical reasoning,” Nature communications, vol. 16, no. 1, p. 642, 2025. - [31] J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou, “Large language models cannot self-correct reasoning yet,” arXiv preprint arXiv:2310.01798, 2023. - [32] C. Sun, S. Huang, and D. Pompili, “Llm-based multi-agent reinforcement learning: Current and future directions,” arXiv preprint arXiv:2405.11106, 2024. - [33] Z. Liu, Y. Zhang, P. Li, Y. Liu, and D. Yang, “Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization,” arXiv preprint arXiv:2310.02170, 2023. - [34] O. Slumbers, D. H. Mguni, K. Shao, and J. Wang, “Leveraging large language models for optimised coordination in textual multi-agent reinforcement learning,” 2023. - [35] S. Hong, X. Zheng, J. Chen, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, et al., “Metagpt: Meta programming for multi-agent collaborative framework,” arXiv preprint arXiv:2308.00352, vol. 3, no. 4, p. 6, 2023. - [36] H. Zhang, W. Du, J. Shan, Q. Zhou, Y. Du, J. B. Tenenbaum, T. Shu, and C. Gan, “Building cooperative embodied agents modularly with large language models,” arXiv preprint arXiv:2307.02485, 2023. - [37] S. S. Kannan, V. L. Venkatesh, and B.-C. Min, “Smart-llm: Smart multi-agent robot task planning using large language models,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 12 140–12 147. - [38] Z. Mandi, S. Jain, and S. Song, “Roco: Dialectic multi-robot collaboration with large language models,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 286–299. - [39] B. Yu, H. Kasaei, and M. Cao, “Co-navgpt: Multi-robot cooperative visual semantic navigation using large language models,” arXiv preprint arXiv:2310.07937, 2023. - [40] X. Guo, K. Huang, J. Liu, W. Fan, N. Vélez, Q. Wu, H. Wang, T. L. Griffiths, and M. Wang, “Embodied llm agents learn to cooperate in organized teams,” arXiv preprint arXiv:2403.12482, 2024. - [41] H. Chen, W. Ji, L. Xu, and S. Zhao, “Multi-agent consensus seeking via large language models,” arXiv preprint arXiv:2310.20151, 2023. - [42] J. Chua, C. Wang, and L. Yao, “Learning natural language constraints for safe reinforcement learning of language agents,” arXiv preprint arXiv:2504.03185, 2025. - [43] Z. Wan, Y. Li, Y. Song, H. Wang, L. Yang, M. Schmidt, J. Wang, W. Zhang, S. Hu, and Y. Wen, “Rema: Learning to meta-think for llms with multi-agent reinforcement learning,” arXiv preprint arXiv:2503.09501, 2025. - [44] H. Li, Y. Q. Chong, S. Stepputtis, J. Campbell, D. Hughes, M. Lewis, and K. Sycara, “Theory of mind for multi-agent collaboration via large language models,” arXiv preprint arXiv:2310.10701, 2023. - [45] L. Yang, Z. Yu, T. Zhang, S. Cao, M. Xu, W. Zhang, J. E. Gonzalez, and B. Cui, “Buffer of thoughts: Thought-augmented reasoning with large language models,” Advances in Neural Information Processing Systems, vol. 37, pp. 113 519–113 544, 2024. - [46] H.-y. Lee, S.-W. Li, and N. T. Vu, “Meta learning for natural language processing: A survey,” arXiv preprint arXiv:2205.01500, 2022. - [47] T. Wu, L. Luo, Y.-F. Li, S. Pan, T.-T. Vu, and G. Haffari, “Continual learning for large language models: A survey,” arXiv preprint arXiv:2402.01364, 2024. - [48] H. Shi, Z. Xu, H. Wang, W. Qin, W. Wang, Y. Wang, Z. Wang, S. Ebrahimi, and H. Wang, “Continual learning of large language models: A comprehensive survey,” arXiv preprint arXiv:2404.16789, 2024. - [49] M. Jovanovic and P. Voss, “Towards incremental learning in large language models: A critical review,” arXiv preprint arXiv:2404.18311, 2024. - [50] H. Zheng, L. Shen, A. Tang, Y. Luo, H. Hu, B. Du, and D. Tao, “Learn from model beyond fine-tuning: A survey,” arXiv preprint arXiv:2310.08184, 2023. - [51] P. Gao, A. Xie, S. Mao, W. Wu, Y. Xia, H. Mi, and F. Wei, “Meta reasoning for large language models,” arXiv preprint arXiv:2406.11698, 2024. - [52] A. Plaat, A. Wong, S. Verberne, J. Broekens, N. van Stein, and T. Back, “Reasoning with large language models, a survey,” arXiv preprint arXiv:2407.11511, 2024. - [53] E. Sayed, M. Maher, O. Sedeek, A. Eldamaty, A. Kamel, and R. El Shawi, “Gizaml: A collaborative meta-learning based framework using llm for automated time-series forecasting.” in EDBT, 2024, pp. 830–833. - [54] J. Zheng, S. Qiu, C. Shi, and Q. Ma, “Towards lifelong learning of large language models: A survey,” ACM Computing Surveys, vol. 57, no. 8, pp. 1–35, 2025. - [55] D. Bandyopadhyay, S. Bhattacharjee, and A. Ekbal, “Thinking machines: A survey of llm based reasoning strategies,” arXiv preprint arXiv:2503.10814, 2025. - [56] Z. Zeng, Y. Liu, Y. Wan, J. Li, P. Chen, J. Dai, Y. Yao, R. Xu, Z. Qi, W. Zhao, et al., “Mr-ben: A meta-reasoning benchmark for evaluating system-2 thinking in llms,” arXiv preprint arXiv:2406.13975, 2024. - [57] N. Patel, M. Kulkarni, M. Parmar, A. Budhiraja, M. Nakamura, N. Varshney, and C. Baral, “Multi-logieval: Towards evaluating multi-step logical reasoning ability of large language models,” arXiv preprint arXiv:2406.17169, 2024. - [58] N. Ferguson, L. Guillou, A. Bundy, and K. Nuamah, “Evaluating the meta-and object-level reasoning of large language models for question answering,” arXiv preprint arXiv:2502.10338, 2025. - [59] S. Sonkar, N. Liu, M. Le, and R. Baraniuk, “Malalgoqa: Pedagogical evaluation of counterfactual reasoning in large language models and implications for ai in education,” in Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 15 554–15 567. - [60] R. Hada, V. Gumma, M. Ahmed, K. Bali, and S. Sitaram, “Metal: Towards multilingual meta-evaluation,” arXiv preprint arXiv:2404.01667, 2024. - [61] S. Min, M. Lewis, L. Zettlemoyer, and H. Hajishirzi, “Metaicl: Learning to learn in context,” arXiv preprint arXiv:2110.15943, 2021. - [62] Y. Esteitieh, S. Mandal, and G. Laliotis, “Towards metacognitive clinical reasoning: Benchmarking md-pie against state-of-the-art llms in medical decision-making,” medRxiv, pp. 2025–01, 2025. - [63] Z. Yin, Q. Sun, Q. Guo, J. Wu, X. Qiu, and X. Huang, “Do large language models know what they don’t know?” arXiv preprint arXiv:2305.18153, 2023. - [64] H. Li, H. Nourkhiz Mahjoub, B. Chalaki, V. Tadiparthi, K. Lee, E. Moradi Pari, C. Lewis, and K. Sycara, “Language grounded multi-agent reinforcement learning with human-interpretable communication,” Advances in Neural Information Processing Systems, vol. 37, pp. 87 908–87 933, 2024. - [65] L. Cross, V. Xiang, A. Bhatia, D. L. Yamins, and N. Haber, “Hypothetical minds: Scaffolding theory of mind for multi-agent tasks with large language models,” arXiv preprint arXiv:2407.07086, 2024. - [66] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015. - [67] J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,” International Journal of Computer Vision, vol. 129, no. 6, pp. 1789–1819, 2021. - [68] D. Liu, A. Nassereldine, Z. Yang, C. Xu, Y. Hu, J. Li, U. Kumar, C. Lee, R. Qin, Y. Shi, et al., “Large language models have intrinsic self-correction ability,” arXiv preprint arXiv:2406.15673, 2024. - [69] H. Chen, J. He, K. Narasimhan, and D. Chen, “Can rationalization improve robustness?” arXiv preprint arXiv:2204.11790, 2022. - [70] J. Andreas, “Measuring compositionality in representation learning,” arXiv preprint arXiv:1902.07181, 2019. - [71] J. Z. Leibo, V. Zambaldi, M. Lanctot, J. Marecki, and T. Graepel, “Multi-agent reinforcement learning in sequential social dilemmas,” arXiv preprint arXiv:1702.03037, 2017. - [72] R. Shu, N. Das, M. Yuan, M. Sunkara, and Y. Zhang, “Towards effective genai multi-agent collaboration: Design and evaluation for enterprise applications,” arXiv preprint arXiv:2412.05449, 2024. - [73] G. Irving, P. Christiano, and D. Amodei, “Ai safety via debate,” arXiv preprint arXiv:1805.00899, 2018. - [74] D. Ziegler, S. Nix, L. Chan, T. Bauman, P. Schmidt-Nielsen, T. Lin, A. Scherlis, N. Nabeshima, B. Weinstein-Raun, D. de Haas, et al., “Adversarial training for high-stakes reliability,” Advances in neural information processing systems, vol. 35, pp. 9274–9286, 2022. - [75] H. Dai, Y. Li, Z. Liu, L. Zhao, Z. Wu, S. Song, Y. Shen, D. Zhu, X. Li, S. Li, et al., “Ad-autogpt: an autonomous gpt for alzheimer’s disease infodemiology,” arXiv preprint arXiv:2306.10095, 2023. - [76] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” Advances in neural information processing systems, vol. 30, 2017. - [77] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022. - [78] A. Kumar, V. Zhuang, R. Agarwal, Y. Su, J. D. Co-Reyes, A. Singh, K. Baumli, S. Iqbal, C. Bishop, R. Roelofs, et al., “Training language models to self-correct via reinforcement learning,” arXiv preprint arXiv:2409.12917, 2024. - [79] J. Beck, R. Vuorio, E. Z. Liu, Z. Xiong, L. Zintgraf, C. Finn, and S. Whiteson, “A survey of meta-reinforcement learning,” arXiv preprint arXiv:2301.08028, 2023. - [80] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar, “Eureka: Human-level reward design via coding large language models,” arXiv preprint arXiv:2310.12931, 2023. - [81] T. Wu, W. Yuan, O. Golovneva, J. Xu, Y. Tian, J. Jiao, J. Weston, and S. Sukhbaatar, “Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge,” arXiv preprint arXiv:2407.19594, 2024. - [82] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano, “Learning to summarize with human feedback,” Advances in neural information processing systems, vol. 33, pp. 3008–3021, 2020. - [83] L. Li, Z. Chen, G. Chen, Y. Zhang, Y. Su, E. Xing, and K. Zhang, “Confidence matters: Revisiting intrinsic self-correction capabilities of large language models,” arXiv preprint arXiv:2402.12563, 2024. - [84] H. Sun, Y. Chai, S. Wang, Y. Sun, H. Wu, and H. Wang, “Curiosity-driven reinforcement learning from human feedback,” arXiv preprint arXiv:2501.11463, 2025. - [85] Q. Zheng, M. Henaff, A. Zhang, A. Grover, and B. Amos, “Online intrinsic rewards for decision making agents from large language model feedback,” arXiv preprint arXiv:2410.23022, 2024. - [86] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al., “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,” arXiv preprint arXiv:2402.03300, 2024. - [87] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al., “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” nature, vol. 575, no. 7782, pp. 350–354, 2019. - [88] M. F. A. R. D. T. (FAIR)†, A. Bakhtin, N. Brown, E. Dinan, G. Farina, C. Flaherty, D. Fried, A. Goff, J. Gray, H. Hu, et al., “Human-level play in the game of diplomacy by combining language models with strategic reasoning,” Science, vol. 378, no. 6624, pp. 1067–1074, 2022. - [89] Z. Wu, R. Peng, S. Zheng, Q. Liu, X. Han, B. I. Kwon, M. Onizuka, S. Tang, and C. Xiao, “Shall we team up: Exploring spontaneous cooperation of competing llm agents,” arXiv preprint arXiv:2402.12327, 2024. - [90] P. Cheng, T. Hu, H. Xu, Z. Zhang, Y. Dai, L. Han, X. Li, et al., “Self-playing adversarial language game enhances llm reasoning,” Advances in Neural Information Processing Systems, vol. 37, pp. 126 515–126 543, 2024. - [91] M. Nakamura, S. Mashetty, M. Parmar, N. Varshney, and C. Baral, “Logicattack: Adversarial attacks for evaluating logical consistency of natural language inference,” in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 13 322–13 334. - [92] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in International conference on machine learning. PMLR, 2017, pp. 1126–1135. - [93] W. Yin, “Meta-learning for few-shot natural language processing: A survey,” arXiv preprint arXiv:2007.09604, 2020. - [94] G. Li, P. Wang, J. Liu, Y. Guo, K. Ji, Z. Shang, and Z. Xu, “Meta in-context learning makes large language models better zero and few-shot relation extractors,” arXiv preprint arXiv:2404.17807, 2024. - [95] H. Raj, V. Gupta, D. Rosati, and S. Majumdar, “Improving consistency in large language models through chain of guidance,” arXiv preprint arXiv:2502.15924, 2025. - [96] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al., “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021. - [97] X. Wen, B. Li, T. Huang, and M. Chen, “Red teaming language models for contradictory dialogues,” arXiv e-prints, pp. arXiv–2405, 2024. - [98] A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al., “Openai o1 system card,” arXiv preprint arXiv:2412.16720, 2024. - [99] M. Geva, D. Khashabi, E. Segal, T. Khot, D. Roth, and J. Berant, “Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 346–361, 2021. - [100] Y. Zhang, X. Yang, S. Feng, D. Wang, Y. Zhang, and K. Song, “Can llms beat humans in debating? a dynamic multi-agent framework for competitive debate,” arXiv preprint arXiv:2408.04472, 2024. - [101] J. Z. Leibo, E. A. Dueñez-Guzman, A. Vezhnevets, J. P. Agapiou, P. Sunehag, R. Koster, J. Matyas, C. Beattie, I. Mordatch, and T. Graepel, “Scalable evaluation of multi-agent reinforcement learning with melting pot,” in International conference on machine learning. PMLR, 2021, pp. 6187–6199. - [102] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al., “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,” arXiv preprint arXiv:2206.04615, 2022. - [103] D. Zhang, Z. Hu, S. Zhoubian, Z. Du, K. Yang, Z. Wang, Y. Yue, Y. Dong, and J. Tang, “Sciinstruct: A self-reflective instruction annotated dataset for training scientific language models,” Advances in Neural Information Processing Systems, vol. 37, pp. 1443–1473, 2024. - [104] R. Xu, X. Qi, Z. Qi, W. Xu, and Z. Guo, “Debateqa: Evaluating question answering on debatable knowledge,” arXiv preprint arXiv:2408.01419, 2024. - [105] W. Du and S. Ding, “A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications,” Artificial Intelligence Review, vol. 54, no. 5, pp. 3215–3238, 2021. - [106] G. Papoudakis, F. Christianos, A. Rahman, and S. V. Albrecht, “Dealing with non-stationarity in multi-agent deep reinforcement learning,” arXiv preprint arXiv:1906.04737, 2019. - [107] M. Mohebbi Moghaddam, B. Boroomand, M. Jalali, A. Zareian, A. Daeijavad, M. H. Manshaei, and M. Krunz, “Games of gans: Game-theoretical models for generative adversarial networks,” Artificial Intelligence Review, vol. 56, no. 9, pp. 9771–9807, 2023. - [108] C. Denison, M. MacDiarmid, F. Barez, D. Duvenaud, S. Kravec, S. Marks, N. Schiefer, R. Soklaski, A. Tamkin, J. Kaplan, et al., “Sycophancy to subterfuge: Investigating reward-tampering in large language models,” arXiv preprint arXiv:2406.10162, 2024. - [109] Y. Yang, J. Luo, Y. Wen, O. Slumbers, D. Graves, H. B. Ammar, J. Wang, and M. E. Taylor, “Diverse auto-curriculum is critical for successful real-world multiagent learning systems,” arXiv preprint arXiv:2102.07659, 2021. - [110] H. F. Ahmad, “Multi-agent systems: overview of a new paradigm for distributed systems,” in 7th IEEE International Symposium on High Assurance Systems Engineering, 2002. Proceedings. IEEE, 2002, pp. 101–107. - [111] M. Umer, M. A. Mohsin, A. Ul-Hasan, and F. Shafait, “Pyramidtabnet: Transformer-based table recognition in image-based documents,” in Document Analysis and Recognition - ICDAR 2023, G. A. Fink, R. Jain, K. Kise, and R. Zanibbi, Eds. Cham: Springer Nature Switzerland, 2023, pp. 420–437. - [112] A. Dorri, S. S. Kanhere, and R. Jurdak, “Multi-agent systems: A survey,” Ieee Access, vol. 6, pp. 28 573–28 593, 2018. - [113] N. Krothapalli and A. V. Deshmukh, “Distributed task allocation in multi-agent systems,” in Proceedings of the Institute of Industrial Engineers Annual Conference. Citeseer, 2002. - [114] L. Weidinger, J. Uesato, M. Rauh, C. Griffin, P.-S. Huang, J. Mellor, A. Glaese, M. Cheng, B. Balle, A. Kasirzadeh, et al., “Taxonomy of risks posed by language models,” in Proceedings of the 2022 ACM conference on fairness, accountability, and transparency, 2022, pp. 214–229. - [115] M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru, “Model cards for model reporting,” in Proceedings of the conference on fairness, accountability, and transparency, 2019, pp. 220–229. - [116] D. Hassabis, D. Kumaran, C. Summerfield, and M. Botvinick, “Neuroscience-inspired artificial intelligence,” Neuron, vol. 95, no. 2, pp. 245–258, 2017. - [117] Y. Bengio, “The consciousness prior,” arXiv preprint arXiv:1709.08568, 2017. - [118] K. Acharya, W. Raza, C. Dourado, A. Velasquez, and H. H. Song, “Neurosymbolic reinforcement learning and planning: A survey,” IEEE Transactions on Artificial Intelligence, vol. 5, no. 5, pp. 1939–1953, 2023. - [119] B. P. Bhuyan, A. Ramdane-Cherif, R. Tomar, and T. Singh, “Neuro-symbolic artificial intelligence: a survey,” Neural Computing and Applications, vol. 36, no. 21, pp. 12 809–12 844, 2024. - [120] F. Yang, D. Lyu, B. Liu, and S. Gustafson, “Peorl: Integrating symbolic planning and hierarchical reinforcement learning for robust decision-making,” arXiv preprint arXiv:1804.07779, 2018. - [121] S. I. Ross, F. Martinez, S. Houde, M. Muller, and J. D. Weisz, “The programmer’s assistant: Conversational interaction with a large language model for software development,” in Proceedings of the 28th International Conference on Intelligent User Interfaces, 2023, pp. 491–514. - [122] A. Bilal, B. Lin, and M. Zaeifi, “Onrl-rag: Real-time personalized mental health dialogue system,” arXiv preprint arXiv:2504.02894, 2025.