2411.13537v2

Model: nemotron-free

# Competence-Aware AI Agents with Metacognition for Unknown Situations and Environments (MUSE) **Authors**: Rodolfo Valiente, Praveen K. Pilly Abstract Metacognition, defined as the awareness and regulation of one’s cognitive processes, is central to human adaptability in unknown situations. In contrast, current autonomous agents often struggle in novel environments due to their limited capacity for adaptation. We hypothesize that metacognition is a critical missing ingredient in autonomous agents for the cognitive flexibility needed to tackle unfamiliar challenges. Given the broad scope of metacognitive abilities, we focus on competence awareness and strategy selection. To this end, we propose the Metacognition for Unknown Situations and Environments (MUSE) framework to integrate metacognitive processes of self-assessment and self-regulation into autonomous agents. We present two implementations of MUSE: one based on world modeling and another leveraging large language models (LLMs). Our system continually learns to assess its competence on a given task and uses this self-assessment to guide iterative cycles of strategy selection. MUSE agents demonstrate high competence awareness and significant improvements in self-regulation for solving novel, out-of-distribution tasks more effectively compared to model-based reinforcement learning and purely prompt-based LLM agent approaches. This work highlights the promise of approaches inspired by cognitive and neural systems in enabling autonomous agents to adapt to new environments while mitigating the heavy reliance on extensive training data and large models for the current models. keywords: Agentic AI , Large Language Model , Metacognition , Reinforcement Learning , Self-Assessment , Self-Regulation , World Model \NewColumnType L[1]Q[l,#1] \NewColumnType C[1]Q[c,#1] \NewColumnType R[1]Q[r,#1] Intelligent Systems Center 1 Introduction The pursuit of fully autonomous agents in artificial intelligence (AI) remains a significant challenge. Current autonomous agents are primarily designed for operating environments, conditions, and uses that are known a priori. They rely on either scripted behaviors or pre-trained policies, both of which struggle to handle unknown situations effectively. As a result, when faced with novelty, they are prone to fail with suboptimal or even catastrophic outcomes (e.g., robotic manipulation errors in unstructured settings). This limitation severely restricts their deployment in safety-critical unknown environments, especially for long-duration missions or applications with little to no human oversight. Therefore, there is a practical urgency to reduce the failure rate, time to completion, and cost of autonomous missions by enabling resilient handling of unknowns during deployment. In mainstream AI, large-scale multi-task pre-training has emerged as the leading approach for enhancing adaptability in autonomous agents (Team et al., 2021). For example, Adaptive Agent (AdA) (Team et al., 2023) was trained with billions of frames and tasks to enable rapid adaptation to unseen, open-ended tasks. Similarly, the RT-2 (Brohan et al., 2023) and RT-X (Collaboration et al., 2023) models leverage large-scale robotic trajectory datasets to train agents capable of solving novel manipulation tasks and generalizing to new robots and environments. However, internet-scale pre-training to be able to handle every potential change and combination of changes in real-world applications is impractical, prohibitively resource-intensive, and expensive. Even with significantly limited data, AI agents must intelligently interpolate and extrapolate beyond their pre-trained scenarios while continually learning and adapting to novelty (Kudithipudi et al., 2023). In other words, when faced with novel scenarios, pre-trained knowledge must be continually updated to strike a dynamic balance between stability and plasticity (Grossberg, 1980). Similar to humans, AI agents can leverage pre-deployment training to acquire a wide range of skills across diverse, known scenarios. Importantly, they can also be equipped with the ability to engage in online learning for continual improvement when encountering novel situations. For example, a teenager attending driving school follows a structured curriculum that teaches foundational vehicle control skills, which are then progressively built upon to master more complex tasks, such as merging onto highways or navigating construction zones. This learning process is cumulative in the sense that mastery of foundational skills simplifies the acquisition of more advanced ones. Moreover, the key principles of driving are consolidated in the student’s brain, protecting them from catastrophic forgetting. Ultimately, the end of driving school marks the beginning of a lifelong learning process, where the student draws on prior experiences to navigate novel driving challenges independently without an instructor. Metacognition, defined as the awareness and regulation of one’s cognitive processes, is a key human trait that has been extensively studied in cognitive psychology (Flavell, 1979; Nelson and Narens, 1990; Metcalfe et al., 1993; Koriat, 1997; Dunlosky and Metcalfe, 2008). This metacognitive flexibility enables humans to learn online efficiently and solve problems iteratively, especially in relation to new tasks. For instance, students can leverage metacognition to more accurately assess their knowledge and adjust their study habits accordingly (Cohen, 2012; Chen et al., 2017). The use of metacognition among college students has indeed been shown to correlate significantly with various measures of academic success (Young and Fry, 2008; Isaacson and Fujita, 2006). Students who perform poorly often overestimate their abilities, leading to under-preparation for exams. This is a common issue in education, where overconfident students allocate insufficient time to study, believing they have already mastered the material. Conversely, students who underestimate their knowledge may spend excessive time reviewing topics they already understand, hindering their progress. Even children as young as three years old can become effective learners and thinkers from activities designed to develop metacognitive skills (Chatzipanteli et al., 2014). It is generally agreed that self-assessment of competence, which can range from over-confidence in novices to slight under-confidence in experts (Kruger and Dunning, 1999; Dunning, 2011), is a capability that is teachable and improves naturally as one becomes more skilled or knowledgeable (Kramarski and Mevarech, 2003; Schraw et al., 2006). Neuroscience further reveals that the subregions of the prefrontal cortex responsible for metacognitive judgments are distinct from those involved in cognitive functions like visual memory recognition, such that the metacognitive function can be selectively deactivated without affecting the cognitive function (e.g., Middlebrooks et al. (2012); Miyamoto et al. (2017)). While metacognition spans a wide range of capabilities and higher-order cognitive processes (e.g., Feeling of Knowing, Judgment of Learning, Source Monitoring), it can be conceptualized as an internal perception-action loop of self-assessment and self-regulation (Nelson and Narens, 1990; Dunlosky and Bjork, 2013). Self-assessment in this context refers to an individual’s ability to assess their competence regarding a specific task. Self-regulation refers to the ability to strategically select and control one’s actions based on this self-assessment. In this article, we introduce the Metacognition for Unknown Situations and Environments (MUSE) framework to computationally instantiate and train the metacognitive capabilities of self-assessment and self-regulation for AI agents, so that they can also achieve more efficient learning and improved generalization to unknown scenarios (Figure 1). Specifically, the self-assessment mechanism is designed to predict the agent’s likelihood of successfully completing a given task for proposed action plans, with a learnable internal model informed by past experiences. And the self-regulation mechanism leverages this self-assessment to enable iterative cycles of competence-aware strategy selection for problem-solving. We present two implementations of the MUSE agent: one based on world modeling and the other utilizing large language models (LLMs). Our experiments in two distinct environments (namely, Meta-World and ALFWorld) demonstrate that MUSE agents achieve substantial improvements in handling novel scenarios compared to baseline/non-metacognitive approaches. Further, we show that metacognition makes a particularly big impact on smaller or less-capable LLM agents, making them amenable to edge deployment as well as less reliant on big data for online adaptation. <details> <summary>figures/metacognitive_cycle.png Details</summary> ![b8c23df7](/v1/image/b8c23df7b17d24e494d637aa59c34f1ed538e55f1e482a0936847df0098574e9) ### Visual Description ## Diagram: Cognitive and Metacognitive Processing Model ### Overview The diagram illustrates a two-stage cognitive framework: **COGNITION** (left) and **METACOGNITION** (right). It depicts the flow of information from the **Environment** through perceptual and action-based processing, followed by self-assessment and self-regulation mechanisms. ### Components/Axes - **COGNITION** (black-outlined box): - **Perception**: Receives input from the Environment. - **Memory**: Linked to Perception (dashed arrow). - **Action**: Receives input from Perception. - **Reasoning**: Linked to Action (dashed arrow). - **METACOGNITION** (blue-outlined box): - **Self-Assessment**: Receives input from COGNITION (dashed arrow). - **Self-Regulation**: Receives input from Self-Assessment and feeds back into it (bidirectional dashed arrows). ### Detailed Analysis - **Environment**: Positioned outside the COGNITION box, it is the source of external stimuli. - **Perception**: Central to COGNITION, it processes environmental input and connects to Memory (dashed arrow). - **Memory**: Subordinate to Perception, with no direct external input. - **Action**: Receives processed information from Perception and connects to Reasoning (dashed arrow). - **Reasoning**: Subordinate to Action, with no direct external input. - **Self-Assessment**: Part of METACOGNITION, it evaluates cognitive outputs (dashed arrow from COGNITION). - **Self-Regulation**: Part of METACOGNITION, it modulates cognitive processes via feedback loops (bidirectional dashed arrows with Self-Assessment). ### Key Observations 1. **Unidirectional Flow in COGNITION**: Information flows from Environment → Perception → Action, with Memory and Reasoning as auxiliary processes. 2. **Bidirectional Feedback in METACOGNITION**: Self-Assessment and Self-Regulation form a closed-loop system, enabling adaptive adjustments. 3. **Dashed Arrows**: Indicate indirect or supportive relationships (e.g., Memory supports Perception but does not drive it). ### Interpretation This model emphasizes the interplay between **bottom-up processing** (COGNITION) and **top-down regulation** (METACOGNITION). The Environment initiates cognitive activity, which is refined through metacognitive mechanisms like self-assessment and regulation. The dashed arrows suggest that Memory and Reasoning are secondary to Perception and Action but still integral to the system. The bidirectional feedback in METACOGNITION highlights the dynamic nature of self-regulation, where evaluation and adjustment occur iteratively. No numerical data or trends are present; the diagram focuses on conceptual relationships rather than quantitative analysis. </details> Figure 1: The metacognitive cycle of self-assessment and self-regulation operates on the traditional perception-action loop of existing AI agents to boost their ability for iterative problem-solving in unknown situations and environments. In contrast to current reinforcement learning (RL) approaches (e.g., Silver et al. (2016, 2017, 2018); Ha and Schmidhuber (2018); Hafner et al. (2023)), which focus on maximizing expected cumulative reward, our MUSE framework prioritizes competence as the primary evaluation metric to enhance the agent’s adaptability to unknown situations. We hypothesize that focusing solely on maximizing return may cause an agent to become stuck in unfamiliar situations, particularly under sparse reward regimes. By contrast, maximizing competence continually for strategy selection encourages more effective exploration in such novel environments. Through metacognition, the agent can evaluate its capabilities and attempt new strategies within its perceived competence, enabling safer and more effective online adaptation. In other words, by maximizing competence, the agent not only improves its ability to tackle immediate challenges more effectively but also fosters iterative problem-solving in complex environments. The contributions of our article are threefold: - We introduce the Metacognition for Unknown Situations and Environments (MUSE) framework, which integrates metacognitive functions of self-assessment and self-regulation into sequential decision-making agents. - We propose an implementation of a competence awareness model that continually learns to assess the agent’s competence on a given task and serves as an evaluation function for planning (Self-Assessment). - We propose an implementation of a policy modulation model that leverages self-assessment to iteratively drive strategy selection by identifying action plans that maximize the likelihood of task success (Self-Regulation). 2 Related Work 2.1 Self-Assessment Self-assessment broadly refers to an agent’s ability to monitor its internal states and assess its capabilities and performance in relation to tasks and goals. It enables humans to reflect and make adjustments for improving outcomes (Flavell, 1979; Nelson and Narens, 1990; Schraw, 1998). World Models: These are generative models of environmental dynamics (Ha and Schmidhuber, 2018; Robine et al., 2023; Micheli et al., 2023; Hansen et al., 2022, 2023), which can be used to estimate the expected cumulative reward of agents. These models often employ sequence-based architectures, such as recurrent neural networks, to predict the next state, reward, and terminal signals. Decoder-based World Models (Hafner et al., 2023; Robine et al., 2023; Micheli et al., 2023) can additionally generate the input state corresponding to the predicted next state. In contrast, decoder-free World Models focus on predicting the outcomes of actions in the latent space, bypassing the need to decode input states (Hansen et al., 2022, 2023). World Models act as proxy simulators during training, enabling agents to learn more efficiently by reducing the reliance on real-environment interactions (e.g., Ha and Schmidhuber (2018); Koul et al. (2020); Hafner et al. (2023)). In this work, we extend the capabilities of decoder-based World Models by training them to predict not only environmental dynamics but also the agent’s competence to solve a given task. LLM Critics: These models are designed to evaluate the performance of LLMs. The LLM itself can be prompted to provide feedback on its outputs (Madaan et al., 2023), intermediate reasoning steps (Paul et al., 2023), or even the prompt itself (Hu et al., 2023). Some approaches enhance the correctness and quality of LLM outputs by using stochastic beam search guided by self-evaluation (Xie et al., 2023). Recognizing that LLMs currently are limited in identifying their own errors, reasoning missteps, or biases (Huang et al., 2023), researchers have augmented LLM critics with external tools, such as search engines and calculators, to improve reliability (Gou et al., 2023). Retrieval Augmented Generation (RAG) approaches have also been proposed to strengthen self-evaluation by providing relevant external knowledge bases (Asai et al., 2023). In contrast, MUSE does not rely solely on pre-trained knowledge or external tools. Instead, it continually learns and grounds itself to evaluate its competence on given tasks and uses this kind of self-assessment to modulate policy decisions. Confidence Networks: Recent work in cognitive computational neuroscience has developed quantitative frameworks for assessing metacognitive judgments related to self-assessment across a range of domains, task difficulties, and time scales with and without external feedback (Fleming, 2024; Lu et al., 2025). Measures that have been proposed and utilized to assess metacognition include the statistical correlation between self-reported confidence ratings and actual performance across trials as well as the more reliable meta- d’ metric, which measures the ability of self-assessment to discriminate between high-performance (correct) and low-performance (incorrect) trials without being affected by response bias in metacognitive judgments (Maniscalco and Lau, 2012; Fleming and Lau, 2014). Consistent with these measures of metacognition, there is prior work in machine learning aimed at self-assessment of deep neural networks that perform perception tasks. For classification, Corbiere et al. (2019) trained a separate neural network (called ConfidNet) that operates on high-level features extracted by the classifier to predict the true class probability (TCP), which is the softmax probability of the correct class irrespective of whether it was chosen or not. Further, Webb et al. (2023) trained confidence networks for a variety of perception tasks to instead predict the probability of the decision being correct, i.e., a value of 1 if correct and 0 otherwise. They also trained a RL agent that chooses among perception labels as well as an opt-out action that earns a low-risk low-reward in situations when the agent is least certain about its decision. These self-assessment metrics themselves are task-agnostic (Fleming, 2024) but require a training scheme that is adapted to the specifics of the task. In this regard, one of our contributions is to implement a global self-assessment metric for artificial agents that predicts the probability of task success in each episode over time. 2.2 Self-Regulation Self-regulation is the process by which an agent dynamically adjusts its behavior based on self-assessment to achieve specific goals (Flavell, 1979; Nelson and Narens, 1990). This ability is essential for humans to exhibit robust decision-making and function autonomously in unfamiliar environments. Model-based Reinforcement Learning (MBRL) Agents: These agents utilize World Models to simulate future scenarios, enabling them to train with minimal real interactions with the environment (Moerland et al., 2023). Dyna (Sutton, 1991) is a foundational architecture that integrates learning and planning within a single agent. Its core idea is to use the agent’s real experiences to update not only its policy but also its internal model of the environment, and thereby leverage the updated internal model to further improve the policy offline using simulated experiences. Among recent leading ones, AlphaGo (Silver et al., 2016), AlphaZero (Silver et al., 2017), and MuZero (Silver et al., 2018) are all MBRL systems that employ internal simulations/models and Monte Carlo Tree Search (MCTS) to explore potential action paths, based on variations of the Upper Confidence Bound (UCB) score, and evaluate them for action selection. In contrast to MBRL agents, which prioritize maximizing expected cumulative reward, the self-regulation mechanism of MUSE leverages competence as the primary evaluation metric to enhance the agent’s ability to navigate and adapt effectively in unknown situations. Prompt-based LLM Agents: The capabilities of LLMs extend beyond language generation, making them increasingly popular for reasoning tasks to potentially deal with novelty. Chain-of-Thought (CoT) prompting (Wei et al., 2022), for example, decomposes a complex problem into intermediate steps to arrive at a final answer. However, CoT reasoning struggles to yield accurate results due to error propagation as the number of steps increases (Chen et al., 2022). Techniques such as self-consistency (Wang et al., 2022), least-to-most prompting (Zhou et al., 2022), and Tree-of-Thought (ToT) prompting (Yao et al., 2024) aim to mitigate this issue by improving sampling strategies and leveraging search algorithms. Nevertheless, these methods rely solely on the LLM’s pre-trained knowledge, which limits their ability to adapt to external feedback. Beyond reasoning tasks, LLMs are also being increasingly applied to operate in an agentic loop of perception and action, which unlocks the benefits of large-scale pre-training for multi-step interactive tasks without relying on RL. ReAct (Yao et al., 2022) was among the first purely prompt-based LLM agents that integrated both reasoning and action planning to perform text-based problem-solving. However, ReAct is inefficient and limited in its ability to transfer performance improvements to subsequent episodes. To address this issue, Reflexion (Shinn et al., 2023) built on ReAct by adding an LLM critic that reflects on failures and provides persistent verbal feedback to the agent for improved performance in subsequent episodes. The performance gains from these LLM agent methods are solely dependent on enhanced prompt-based in-context learning, which limits their ability for longer-term true learning from new varied experiences. While MUSE also makes use of prompting for both reasoning and planning, it can also continually learn from its experiences to facilitate more effective problem-solving in unknown situations and environments. 3 Decoder-based World Model implementation In this section, we describe our implementation of the MUSE framework using a decoder-based World Model to equip MBRL agents with metacognitive abilities of self-assessment and self-regulation. 3.1 Methods 3.1.1 Self-Assessment through World Modeling We leverage the decoder-based World Model from Dreamer-v3 (Hafner et al., 2023) to implement self-assessment for agents, but we note that our approach can be extended to decoder-free World Models as well. Dreamer-v3 uses a Recurrent State-Space Model (RSSM) to model the environment dynamics. See Equations 1 - 3 for the formulation from Hafner et al. (2023). The RSSM maps the input state $x_{t}$ and recurrent state $h_{t}$ to a latent embedding $z_{t}$ and uses the concatenation of $h_{t}$ and $z_{t}$ , called the RSSM state, as input to parameterize various distributions over the reward $\hat{r}_{t}$ , terminal signal $\hat{d}_{t}$ , and decoded state $\hat{x}_{t}$ . $$ \displaystyle\begin{aligned} \begin{aligned} \raisebox{8.39578pt}{\hbox to0.0pt{\hss\vbox to0.0pt{\hbox{$\text{RSSM}\hskip 4.30554pt\begin{cases}\hphantom{A}\\ \hphantom{A}\\ \hphantom{A}\end{cases}\hskip-10.33327pt$}\vss}}}&\text{Sequence model:}\hskip 35.00005pt&&h_{t}&\ =&\ f_{\phi}(h_{t-1},z_{t-1},a_{t-1})\\ &\text{Encoder:}\hskip 35.00005pt&&z_{t}&\ \sim&\ q_{\phi}(z_{t}\;|\;h_{t},x_{t})\\ &\text{Dynamics predictor:}\hskip 35.00005pt&&\hat{z}_{t}&\ \sim&\ p_{\phi}(\hat{z}_{t}\;|\;h_{t})\\ &\text{Reward predictor:}\hskip 35.00005pt&&\hat{r}_{t}&\ \sim&\ p_{\phi}(\hat{r}_{t}\;|\;h_{t},z_{t})\\ &\text{Terminal signal predictor:}\hskip 35.00005pt&&\hat{d}_{t}&\ \sim&\ p_{\phi}(\hat{d}_{t}\;|\;h_{t},z_{t})\\ &\text{Decoder:}\hskip 35.00005pt&&\hat{x}_{t}&\ \sim&\ p_{\phi}(\hat{x}_{t}\;|\;h_{t},z_{t})\end{aligned}\end{aligned} \tag{1} $$ ifnextchar gobble Following Hafner et al. (2023), given a sequential batch of inputs $x_{1:T}$ , actions $a_{1:T}$ , rewards $r_{1:T}$ , and terminal signals $d_{1:T}$ , the World Model parameters $\phi$ are optimized to minimize the prediction loss $\mathcal{L}_{\mathrm{pred}}$ , the dynamics loss $\mathcal{L}_{\mathrm{dyn}}$ , and the representation loss $\mathcal{L}_{\mathrm{rep}}$ . The prediction loss $\mathcal{L}_{\mathrm{pred}}$ is the joint negative log-likelihood of the multiple probabilistic predictors (Equation 3). Real-valued quantities like the reward and decoded state are trained with a symlog squared loss, whereas the terminal signal, which is a binary-valued quantity, is trained with logistic regression. The dynamics loss $\mathcal{L}_{\mathrm{dyn}}$ and the representation loss $\mathcal{L}_{\mathrm{rep}}$ are designed to effectively learn the dynamics of the latent embeddings for generating realistic rollout trajectories. $$ \displaystyle\begin{aligned} \mathcal{L}(\phi)\doteq\operatorname{E}_{q_{\phi}}\Bigg[\displaystyle\sum_{t=1}^{T}(\mathcal{L}_{\mathrm{pred}}(\phi)+\mathcal{L}_{\mathrm{dyn}}(\phi)+0.1\mathcal{L}_{\mathrm{rep}}(\phi))\Bigg]\end{aligned} \tag{2} $$ ifnextchar gobble $$ \displaystyle\begin{aligned} \mathcal{L}_{pred}\doteq-\ln p_{\phi}(r_{t}\;|\;h_{t},z_{t})-\ln p_{\phi}(d_{t}\;|\;h_{t},z_{t})-\ln p_{\phi}(x_{t}\;|\;h_{t},z_{t})\end{aligned} \tag{3} $$ ifnextchar gobble We augment Dreamer-v3’s World Model with an additional head for predicting task success. In this implementation, the Self-Assessment Model is an MLP with $N$ outputs that map the RSSM state to the probability of task success within the $N$ quantiles of the maximum episode duration. Specifically, the MLP outputs parameterize $N=5$ Bernoulli distributions $\{\psi_{1},·s\psi_{N}\}$ for the five quantiles. A self-assessment prediction involves sampling from each of these distributions, $\hat{c}^{i}_{t}\sim\psi_{i}(h_{t},z_{t})$ , and combining these samples into a prediction vector. For example, a prediction of success in the first quantile would yield the vector $[1,1,1,1,1]$ , whereas a prediction of failure even by the last quantile would yield the vector $[0,0,0,0,0]$ . This process is visualized in Figure 2. Note that each component of the self-assessment head is trained separately using binary cross-entropy loss, and the individual losses are then added to the total prediction loss $\mathcal{L}_{\mathrm{pred}}$ (Equation 4). $$ \displaystyle\begin{aligned} \mathcal{L}_{\mathrm{SA}}\doteq-\sum_{i=1}^{N}\ln\psi_{i}(h_{t},z_{t}),\hskip 35.00005pt\mathcal{L}_{pred}\leftarrow\mathcal{L}_{pred}+\mathcal{L}_{\mathrm{SA}}\end{aligned} \tag{4} $$ ifnextchar gobble <details> <summary>figures/rssm_sa_combined.png Details</summary> ![043f77de](/v1/image/043f77debd6e093b62a7dedfe50ae5ab73d553cef3cafb91429a5fc0037ea91e) ### Visual Description ## Diagram: Multi-Stage Encoder-Decoder Architecture with Attention Mechanisms ### Overview The diagram illustrates a three-stage sequential processing system with encoder-decoder blocks, attention mechanisms, and hidden layers. Each stage processes input data (x₁, x₂, x₃) through encoding (Z₁, Z₂, Z₃), hidden layers (h₁, h₂, h₃), and decoding to produce reconstructed outputs (x̂₁, x̂₂, x̂₃). The system includes attention components (a₁, a₂) and auxiliary variables (s₁, s₂, s₃, c₁, c₂, c₃, r₁, r₂, r₃, d₁, d₂, d₃). ### Components/Axes 1. **Top Layer (c₁, c₂, c₃)**: - Three parallel sequences with circular nodes connected by pink lines. - Labels: c₁, c₂, c₃ (possibly control parameters or context vectors). 2. **Hidden Layers (h₁, h₂, h₃)**: - Three interconnected blue squares with green arrows. - Positioned between encoder/decoder blocks and attention mechanisms. 3. **Encoder-Decoder Blocks**: - **Encoder (enc)**: Light blue trapezoids labeled "enc" above Z₁, Z₂, Z₃. - **Decoder (dec)**: Light blue trapezoids labeled "dec" below Z₁, Z₂, Z₃. - **Z Matrices**: 3x3 grids with varying shades of blue (likely feature maps or latent representations). 4. **Attention Mechanisms**: - **a₁, a₂**: Curved green arrows connecting h₁→h₂ and h₂→h₃. - **s₁, s₂, s₃**: Gray arrows pointing to hidden layers (possibly softmax weights or scaling factors). 5. **Input/Output Data**: - **x₁, x₂, x₃**: Ground-truth images of a robotic arm (bottom row). - **x̂₁, x̂₂, x̂₃**: Reconstructed outputs (blurred versions of x₁–x₃). ### Detailed Analysis - **Data Flow**: 1. Input x₁ is encoded into Z₁ (feature map) via "enc". 2. Z₁ passes through h₁, which receives attention from s₁ and c₁. 3. h₁ processes data and sends adjustments (a₁) to h₂. 4. h₂ integrates information from h₁, s₂, and c₂, then sends a₂ to h₃. 5. h₃ processes final stage data and sends it to "dec" for reconstruction into x̂₁. 6. Similar flows repeat for x₂→x̂₂ and x₃→x̂₃. - **Z Matrix Patterns**: - Z₁: Uniform light blue with sparse dark blue patches. - Z₂: Increased dark blue density in lower-left quadrant. - Z₃: Highest dark blue concentration in center-right. - **Robotic Arm Images**: - x₁–x₃: Clear images of a robotic arm with blue nozzle and red base. - x̂₁–x̂₃: Blurred reconstructions with reduced nozzle definition. ### Key Observations 1. **Sequential Dependency**: Attention mechanisms (a₁, a₂) suggest cross-stage information sharing. 2. **Feature Evolution**: Z matrices show progressive feature refinement from sparse (Z₁) to concentrated (Z₃). 3. **Reconstruction Fidelity**: Outputs x̂₁–x̂₃ exhibit increasing blur compared to inputs, indicating potential over-smoothing or limited capacity. 4. **Control Variables**: c₁–c₃ and d₁–d₃ may represent task-specific constraints or decision variables. ### Interpretation This architecture resembles a **multi-stage variational autoencoder (VAE)** with attention for robotic manipulation tasks. The three stages likely handle: 1. **Stage 1 (c₁/h₁)**: Basic feature extraction (edge detection). 2. **Stage 2 (c₂/h₂)**: Mid-level feature integration (object recognition). 3. **Stage 3 (c₃/h₃)**: High-level reconstruction (pose estimation). The attention mechanisms (a₁, a₂) enable the model to focus on critical features (e.g., nozzle position) across stages. The Z matrices represent latent space traversal, with darker shades indicating higher activation confidence. The blurred outputs suggest the model prioritizes coarse spatial relationships over fine details, which could be optimized for real-time robotic control applications. **Notable Anomaly**: The abrupt increase in Z₃'s dark blue concentration might indicate overfitting to specific features or insufficient regularization. </details> Figure 2: Schematic of the implementation of self-assessment in the context of the Dreamer-v3 World Model (adapted from Hafner et al. (2023)). The input state $x$ to the RSSM is encoded into latent embedding $z$ . The model recurrently predicts self-assessment $\hat{c}$ , reward $\hat{r}$ , and terminal signal $\hat{d}$ , while also decoding the input state $\hat{x}$ . 3.1.2 Self-Regulation Even with pre-deployment training on multiple tasks, including parametric variations, MBRL agents exhibit limited generalization to novel tasks that require either new, orchestrated combinations of those skills or entirely new skills (Ketz and Pilly, 2022). While Dreamer-v3 can handle novel parametric variations for a known task, it struggles to make progress when faced with an unknown reward function that differs semantically from those of the training tasks. Central to our MUSE framework is the self-regulation algorithm, which performs competence-aware actions to solve novel tasks. Specifically, the decision-making process selects actions that maximize the likelihood of task success. Self-assessed competence can be used to guide planning in three primary ways: 1. Simulate multiple future scenarios (rollout trajectories) based on the current state and potential actions, then greedily select the path that maximizes the self-assessment criterion 1. Perform MCTS over actions using the self-assessment criterion in place of variations of the UCB score 1. Optimize the RSSM state to directly maximize the self-assessment criterion to effectively self-regulate the policy For this implementation of MUSE, we used the third option. This self-regulation method, which is detailed in Algorithm 1, leverages the differentiability of the World Model with the self-assessment head. MUSE performs a World Model rollout where the agent’s actions are regulated to increase the likelihood of reaching a success state. Specifically, MUSE directly optimizes the RSSM state, $s$ ( $\doteq\{h,z\}$ ), input to the agent for selecting actions that maximize the self-assessment criterion. The intuition is that when the Self-Assessment Model predicts failure in a novel environment, it is no longer useful to just rely on the default policy. Instead, we seek competence-aware actions that guide the agent to a success state by augmenting the RSSM state in a direction that increases the probability of task success. At each time step $t$ , MUSE performs a short World Model rollout of horizon $H$ that optimizes over the RSSM state $s$ as follows: $$ \displaystyle\begin{aligned} s\leftarrow s+\beta\nabla_{s}\left(\sum_{i=1}^{N}\psi_{i}(s)\right).\end{aligned} \tag{5} $$ ifnextchar gobble It then applies the self-regulated action $a_{t}\sim\pi(a_{t}|s)$ , observes the resulting recurrent state $h_{t+1}$ , and begins a new iteration. Algorithm 1 Self-regulation leverages the World Model and its constituent Self-Assessment Model to select competence-aware actions 1: $h_{t}$ 2: $H$ = 10 3: $\beta=0.02$ 4: $z_{t}\sim p_{\phi}(z_{t}|h_{t})$ 5: $h← h_{t}$ 6: $z← z_{t}$ 7: $s\doteq\{h,z\}$ 8: for $step← 1$ to $H$ do 9: $a\sim\pi(a|s)$ 10: $h← f_{\phi}(s,a)$ 11: $\displaystyle s← s+\beta∇_{s}\left(\sum_{i=1}^{N}\psi_{i}(s)\right)$ 12: end for 13: $a_{t}\sim\pi(a_{t}|s)$ 14: $h_{t+1}← f_{\phi}(s,a_{t})$ 15: return $z_{t}$ , $a_{t}$ , $h_{t+1}$ 3.2 Meta-World Experiments For the World-Model-based implementation, we evaluated our approach within the Meta-World robotic manipulation simulator (Yu et al., 2020) and compared it against Dreamer-v3 (Hafner et al., 2023) as the MBRL baseline. Meta-World provides a suitable testbed for learning a shared perceptual and dynamics model across multiple tasks using a 6 degrees-of-freedom (DOF) robotic arm. To ensure consistency, we used Dreamer-v3’s network architectures, hyperparameters, and learning procedures across all shared components between the two agents (e.g., an imagination horizon of 15 time steps for actor and critic learning). Both methods were implemented in the same PyTorch codebase, with the self-assessment and self-regulation modules omitted for Dreamer-v3. The agents received a $64× 64$ RGB observation alongside a 40-dimensional proprioceptive state. Additionally, we included a task embedding that was represented as a single integer-valued channel appended to the visual state. MUSE leveraged the built-in success signal returned by the Meta-World environment to train its Self-Assessment Model. For the experiments, we employed a two-stage protocol comprising pre-deployment training on known tasks followed by deployment adaptation to unknown tasks. Each episode in these experiments had a maximum time limit of 500 steps. For pre-deployment training, we utilized Meta-World’s MT10 suite of 10 different manipulation tasks (Figure 3). In particular, we adopted a multi-task learning paradigm (Mandi et al., 2023), encompassing all 10 training tasks over 2M total environment steps. This paradigm was chosen over meta-RL approaches to reduce computational costs and training time (Wang et al., 2021). By default, object and goal positions were randomly sampled to enable domain randomization. For deployment adaptation, we evaluated the agents on a set of 10 novel tasks from Meta-World’s MT50 suite with distinct reward functions, which were semantically different from those in the pre-deployment training set (Figure 4). The agents were exposed to one novel task at a time, starting with pre-deployment trained weights, for 20 adaptation episodes per task. They were assessed for performance on novel tasks during these adaptation episodes. The task embedding channel was set to zero for novel tasks. During training as well as adaptation, both agents continually updated their World Models with real data and their actor and critic neural networks with imagined data. The replay buffer from pre-deployment training was retained for deployment adaptation to prevent catastrophic forgetting of previously trained performance. 3.2.1 Metrics Self-Assessment We used metacognitive accuracy and the Area under the Type 2 Receiver Operating Characteristic Curve (AUROC2) to evaluate how well MUSE predicts its success on novel tasks (Fleming and Lau, 2014), which in turn indicates how effectively the self-assessment signal can support online adaptation. MUSE was evaluated for each novel task separately over 20 adaptation episodes. During these episodes, we collected MUSE’s step-wise self-assessment predictions for evaluation. The predictions at all time steps across tasks were compared with the true labels to compute the metacognitive accuracy and AUROC2 metrics. The Type 2 ROC curve for MUSE was computed by treating the sum of the self-assessment components $\left(\sum_{i=1}^{N}\psi_{i}(s)\right)$ as the signal. Note that, as Dreamer-v3 lacks a self-assessment module to predict episode outcomes, we cannot make a meaningful comparison between Dreamer-v3 and MUSE for self-assessment. Self-Regulation We evaluated how well the agents generalize to unknown situations using two metrics: the percentage of novel tasks solved and the average time to task completion. Each agent was assessed for each novel task separately over the 20 adaptation episodes. The percentage of episodes where the agent completed the task within the maximum time limit was averaged across all novel tasks to calculate the success rate. Similarly, the number of time steps required to achieve success in each episode was averaged across all episodes for each novel task to compute the respective time-to-completion metrics. <details> <summary>figures/train_mt10_tasks.png Details</summary> ![0ea9b389](/v1/image/0ea9b389e1b9752a584577903b09a9e657b40caa8821266596299fb8341a8717) ### Visual Description ## Image Sequence: Robotic Arm Manipulation Tasks ### Overview The image is a grid of 10 sequential photos depicting a red robotic arm performing various manipulation tasks in a simulated environment. The arm interacts with objects like boxes, drawers, and a ball, demonstrating reach, grasping, and placement capabilities. The environment features a wooden floor, white walls, and a tiled baseboard. ### Components/Axes - **Robotic Arm**: Red cylindrical structure with a multi-jointed design, terminating in a gripper with colored tips (blue, red, green). - **Objects**: - Yellow and black box (top-left) - Black cabinet (top-center) - Green drawer unit (top-right) - Red cylindrical object (bottom-center) - Brown cabinet (bottom-right) - **Environment**: - Wooden floor with horizontal grain - White walls with gray baseboard - Tiled floor in some frames ### Detailed Analysis 1. **Top Row (Images 1-5)**: - **Image 1**: Arm approaches yellow/black box; gripper tips colored blue and red. - **Image 2**: Arm interacts with black cabinet; gripper tips blue and red. - **Image 3**: Arm engages with green drawer unit; gripper tips blue and red. - **Image 4**: Arm manipulates green drawer unit; green ball visible on floor. - **Image 5**: Arm interacts with red cabinet; gripper tips blue and red. 2. **Bottom Row (Images 6-10)**: - **Image 6**: Arm approaches black cabinet; gripper tips blue and red. - **Image 7**: Arm manipulates red cylindrical object; gripper tips blue and red. - **Image 8**: Arm interacts with brown cabinet; gripper tips blue and red. - **Image 9**: Arm approaches brown cabinet; gripper tips blue and red. - **Image 10**: Arm manipulates brown cabinet; gripper tips blue and red. ### Key Observations - The robotic arm consistently uses a gripper with blue and red tips across all tasks. - Objects vary in color (yellow, black, green, red, brown) and shape (boxes, drawers, cylinders). - The sequence suggests a demonstration of task versatility, with no repeated object interactions. - No visible text, labels, or legends in the images. ### Interpretation This sequence likely illustrates a robotic arm's ability to adapt to diverse manipulation tasks in a controlled environment. The absence of repeated objects implies testing of generalization capabilities. The colored gripper tips may indicate different end-effector configurations or sensor feedback mechanisms. The structured progression of tasks suggests a controlled experiment or simulation, possibly for evaluating robotic precision, path planning, or object recognition algorithms. The lack of textual annotations in the images indicates that metadata (e.g., task descriptions, success metrics) would need to be provided separately in a technical report. </details> Figure 3: Meta-World pre-deployment training set [button-press, door-open, drawer-close, drawer-open, peg-insert-side, pick-place, push, reach, window-close, window-open], which comprises the 10 tasks from the MT10 suite. <details> <summary>figures/test_evaluation_tasks.png Details</summary> ![f86ca02a](/v1/image/f86ca02aa6e1a242b79accb0503a9c52982c76008340b65ab06709f3ac09c06c) ### Visual Description ## Diagram: Robotic Arm Task Simulation ### Overview The image depicts a sequence of 10 panels arranged in two rows of five, illustrating a red robotic arm with blue and white grippers interacting with various objects in simulated environments. Each panel represents a distinct task, such as manipulating objects, picking up items, or interacting with machines. The robotic arm's movements and object interactions vary across panels, suggesting a demonstration of dexterity or task adaptability. ### Components/Axes - **No textual labels, axis titles, legends, or axis markers** are present in the image. - **Objects**: - Yellow box with red knob (Panel 1, top row). - Soccer goal with white frame and green ball (Panel 2, top row). - Brown box with red cylinder (Panel 3, top row). - Brown box with green ball (Panel 4, top row). - Brown machine with white knob and green ball (Panel 5, top row). - White goal with red frame and black disc (Panel 1, bottom row). - Gray box with red cylinder (Panel 2, bottom row). - Gray machine with blue knob and red ball (Panel 3, bottom row). - Gray machine with red cylinder and white disc (Panel 4, bottom row). - Black safe with gray handle and blue knob (Panel 5, bottom row). ### Detailed Analysis - **Panel 1 (Top Row)**: Robotic arm approaches a yellow box with a red knob, possibly simulating a task involving object manipulation or activation. - **Panel 2 (Top Row)**: Arm interacts with a soccer goal, suggesting a task requiring precision or coordination. - **Panel 3 (Top Row)**: Arm engages with a brown box containing a red cylinder, potentially testing grasping or lifting capabilities. - **Panel 4 (Top Row)**: Arm interacts with a green ball on a brown box, indicating object retrieval or placement. - **Panel 5 (Top Row)**: Arm approaches a brown machine with a white knob and green ball, simulating interaction with industrial equipment. - **Panel 1 (Bottom Row)**: Arm manipulates a white goal with a red frame and black disc, possibly testing dynamic object handling. - **Panel 2 (Bottom Row)**: Arm interacts with a gray box containing a red cylinder, mirroring Panel 3 but with a different container. - **Panel 3 (Bottom Row)**: Arm engages with a gray machine featuring a blue knob and red ball, highlighting adaptability to varied machinery. - **Panel 4 (Bottom Row)**: Arm interacts with a gray machine holding a red cylinder and white disc, suggesting multi-object coordination. - **Panel 5 (Bottom Row)**: Arm approaches a black safe with a gray handle and blue knob, simulating security-related tasks. ### Key Observations - The robotic arm’s grippers (blue and white) consistently target objects with contrasting colors (e.g., red, green, blue), emphasizing visual or tactile discrimination. - Tasks range from simple object retrieval (e.g., green ball) to complex interactions (e.g., safe with handle and knob). - No numerical data or trends are present; the focus is on qualitative task execution. ### Interpretation The image demonstrates the robotic arm’s versatility in handling diverse objects and environments, likely for training, testing, or showcasing capabilities in robotics. Each panel isolates a specific task, suggesting a modular approach to evaluating performance. The absence of numerical data implies the focus is on qualitative assessment (e.g., success/failure of interactions) rather than quantitative metrics. The variety of objects (boxes, machines, safes) indicates potential applications in industrial automation, logistics, or security systems. </details> Figure 4: Meta-World evaluation set [button-press-topdown-wall, soccer, push-wall, push-block, coffee-button, plate-slide, peg-unplug-side, lever-pull, handle-press, door-unlock], which comprises 10 novel tasks from the MT50 suite with distinct reward functions, which differ semantically from those in the pre-deployment training set. Table 1: Self-assessment performance of the MUSE agent on novel tasks in the Meta-World environment. | Method | Metric | Value | | --- | --- | --- | | MUSE | Metacognitive Accuracy | 92% | | MUSE | AUROC2 | 0.95 | 3.2.2 Results The MUSE agent achieved a metacognitive accuracy of 92% and an AUROC2 of 0.95 on novel tasks over the adaptation episodes, demonstrating that the Self-Assessment Model is highly predictive of competence for novel tasks (Table 1). Further, MUSE successfully solved 7 of the 10 novel tasks (70%) by leveraging competence-aware actions. The 7 solved tasks required different time steps per episode depending on the relative difficulty and complexity of the task (Figure 5 a). For instance, plate-slide required nearly the maximum time budget, while coffee-button was solved quickly. But in sharp contrast, Dreamer-v3 failed to solve any of the novel tasks within the allotted adaptation episodes (Figure 5 b). <details> <summary>figures/adaptation_steps.png Details</summary> ![8c1a4e0a](/v1/image/8c1a4e0a455974913b70f6238c16c8e03499cbe7e6c2710e4d0e108846f69178) ### Visual Description ## Bar Charts: MUSE and Dreamer-v3 Performance Comparison ### Overview The image contains two side-by-side bar charts comparing performance metrics for two systems: MUSE and Dreamer-v3. Both charts measure "Average time to adapt" across 10 tasks, with MUSE showing solved/unsolved task distinctions and Dreamer-v3 displaying uniform performance. ### Components/Axes **MUSE Performance (Left Chart):** - **X-axis (Tasks):** 10 tasks listed left-to-right: `button-press`, `soccer`, `push-wall`, `push-block`, `coffee-button`, `plate-slide`, `peg-unplug-side`, `level-pull`, `handle-press`, `door-unlock` - **Y-axis (Time):** Ranges from -50 to 400 (units unspecified, likely milliseconds or seconds) - **Legend:** - Blue bars = Solved tasks - Red bars = Unsolved tasks - Positioned in top-left corner **Dreamer-v3 Performance (Right Chart):** - **X-axis (Tasks):** Same 10 tasks as MUSE chart - **Y-axis (Time):** Identical scale (-50 to 400) - **Legend:** - Red bars = All tasks (no distinction between solved/unsolved) - Positioned in top-left corner ### Detailed Analysis **MUSE Performance:** - **Solved Tasks (Blue):** - `button-press`: ~50 - `soccer`: ~50 - `push-wall`: ~120 - `push-block`: ~250 - `coffee-button`: ~20 - `plate-slide`: ~380 (tallest bar) - `peg-unplug-side`: ~280 - `level-pull`: ~30 - `handle-press`: ~30 - `door-unlock`: ~0 - **Unsolved Tasks (Red):** - All tasks show identical value: **-50** (below y-axis baseline) **Dreamer-v3 Performance:** - All tasks show identical value: **0** (aligned with y-axis baseline) ### Key Observations 1. **MUSE Variability:** - Performance varies significantly across tasks (e.g., `plate-slide` takes ~380 units vs. `coffee-button` at ~20 units). - `door-unlock` shows no time cost (0), while `door-unlock` in MUSE is unsolved (-50). 2. **Dreamer-v3 Uniformity:** - No variation in performance across tasks (all 0). - Contrasts sharply with MUSE's task-specific results. 3. **Unsolved Task Baseline:** - MUSE's unsolved tasks consistently show -50, suggesting a fixed penalty or error state. ### Interpretation - **MUSE Performance:** The system demonstrates task-dependent adaptation, with complex tasks (`plate-slide`, `push-block`) requiring significantly more time. The -50 baseline for unsolved tasks may indicate a default failure state or measurement artifact. - **Dreamer-v3 Performance:** The uniform 0 value suggests either: 1. A fixed, non-adaptive approach to all tasks, or 2. A system that either always succeeds (0 time) or fails catastrophically (not represented here). - **Critical Insight:** MUSE's performance correlates with task complexity, while Dreamer-v3's uniformity implies a lack of task-specific optimization. The -50 unsolved baseline in MUSE warrants further investigation into error handling mechanisms. </details> Figure 5: Average number of time steps per episode required to solve each of the 10 novel tasks in the Meta-World environment. MUSE (a) successfully solved 7 out of 10 tasks, whereas Dreamer-v3 (b) failed to solve any of them. Note that to facilitate illustration, unsolved tasks are assigned a nominal value of -25 and depicted by a red bar. 3.3 Discussion The pre-deployment training tasks were selected to encompass a broad range of skills, so the agents could develop a better understanding of various manipulation strategies. Following this training, the agents were exposed to out-of-distribution tasks that were selected to evaluate their ability to generalize learned skills to novel challenges. This two-stage approach of training on familiar tasks and then adapting to novel ones was used to rigorously assess the effectiveness of MUSE and Dreamer-v3 in unknown situations. Overall, the World-Model-based experiments showed that MUSE significantly outperforms Dreamer-v3 in adapting to novel tasks by effectively leveraging competence awareness for strategy selection. 4 LLM-based implementation In this section, we describe our implementation of the MUSE framework to equip LLM agents with metacognitive abilities of self-assessment and self-regulation. 4.1 Methods 4.1.1 ReAct Yao et al. (2022) was among the first to introduce an LLM agent that interacts with its environment to accomplish tasks by being prompted to reason and act (Figure 6). At time step $t$ , the ReAct agent ( $M_{a}$ ) perceives an observation $o_{t}∈\mathcal{O}$ , executes an action $a_{t}∈\mathcal{A}$ based on the policy $\pi(a_{t}|I,c_{t-1},o_{t})$ , and receives a reward $r_{t}$ . Here, $I$ is the natural language description of the task, and $c_{t}=\{o_{1},a_{1},r_{1}...,o_{t},a_{t},r_{t}\}$ is the running context of the trajectory within the episode. The context is reset at the end of each episode, which occurs when either the task is solved within the time budget or the episode terminates due to running out of time. The ReAct agent leverages language-based CoT reasoning (Wei et al., 2022) for improved sequential action selection for multi-step tasks. Note that actions $a_{t}$ stored in $c_{t}$ include both standard actions and CoT reasoning steps. Two-shot domain-specific trajectory examples with step-by-step reasoning are included in the prompt for CoT reasoning. See Supplementary Section 8.1 for illustrative examples. <details> <summary>figures/ReAct.png Details</summary> ![5e16acb0](/v1/image/5e16acb0e5ae18006dbd928c286d3258d3159926f152ff2edf4e03eefacc5119) ### Visual Description ## Flowchart: Hierarchical Reinforcement Learning System Architecture ### Overview The diagram illustrates a hierarchical reinforcement learning system where context, memory, and environmental interactions drive decision-making. Key components include long-term memory, language processing, environmental feedback, and policy execution layers. The system emphasizes integration of high-level task descriptions with low-level action execution through an "Actor" component. ### Components/Axes 1. **Input Streams**: - `C_{t-1}`: Context (previous state) - Few-shot examples: Training data for memory initialization 2. **Memory System**: - Long-term memory: Stores contextual knowledge and past experiences 3. **Processing Modules**: - Language descriptor: Converts observations into structured text - Environment: Simulates real-world interactions - Low-level policies: Translates high-level actions into executable steps 4. **Control Flow**: - Actor: Central decision-maker integrating task descriptions and memory - Feedback loops: Between environment observations and memory updates ### Detailed Analysis - **Context Flow**: - `C_{t-1}` (context) and few-shot examples → Long-term memory - Long-term memory + Text observation (`O_t`) → Language descriptor - **Environment Interaction**: - Language descriptor output → Environment - Environment provides: Observation (`O_{t+1}`), Reward (`R_t`) - **Policy Execution**: - Low-level policies → Actor (high-level action `A_t`) - Actor receives: Task description (`I`), Memory (`M_a`) - **Temporal Dynamics**: - Time steps denoted by subscripts (`t`, `t+1`) - Memory (`M_a`) persists across iterations ### Key Observations 1. **Hierarchical Structure**: - Clear separation between high-level task description (`I`) and low-level policy execution 2. **Memory Integration**: - Long-term memory acts as persistent knowledge base influencing all decisions 3. **Feedback Loops**: - Environment observations (`O_{t+1}`) and rewards (`R_t`) continuously update the system 4. **Actor-Critic Architecture**: - Actor handles high-level decisions while low-level policies manage execution details ### Interpretation This architecture demonstrates a sophisticated RL system designed for complex tasks requiring: 1. **Contextual Awareness**: Through persistent memory (`M_a`) and historical context (`C_{t-1}`) 2. **Language Grounding**: Via the language descriptor module converting raw observations into structured text 3. **Multi-timescale Learning**: Combining immediate rewards (`R_t`) with long-term memory retention 4. **Modular Design**: Separation of concern between task description, policy execution, and environmental interaction The system appears optimized for tasks requiring both strategic planning (high-level actions) and precise execution (low-level policies), with continuous learning through environmental feedback. The bidirectional flow between environment and memory suggests adaptive capabilities that could handle non-stationary environments or evolving task requirements. </details> Figure 6: Our illustration of the ReAct architecture (Yao et al., 2022). 4.1.2 Reflexion Reflexion (Shinn et al., 2023) enhances in-context prompting for LLM agents by generating internal feedback in language, referred to as “reflection,” to transfer lessons learned across episodes for a given task (Figure 7). At the end of each episode $e$ , Reflexion prompts an LLM to reason and generate verbal feedback $rx^{e}$ about the agent’s performance by analyzing the entire trajectory $c_{T}$ from the episode start ( $t=1$ ) to finish ( $t=T$ ) and the episode outcome (success or failure). This Reflection LLM ( $M_{rx}$ ) also receives the reflections and outcomes from previous episodes ( $1:e-1$ ). The Reflexion agent ( $M_{a}$ ) follows the policy $\pi(a_{t}|I,c_{t-1},o_{t},rx^{\{1:e-1\}})$ , which incorporates cumulative reflections from earlier episodes. In contrast to ReAct, which relies solely on short-term memory comprising the agent’s running trajectory within the current episode, Reflexion leverages both short- and long-term memory for improved strategic adaptation to novel situations. This iterative internal feedback mechanism allows the agent to refine its strategies progressively by integrating lessons learned from failures in previous episodes. Reflexion addresses the credit assignment problem through implicit reasoning about specific actions within the trajectory that led to failures, proposing alternative strategies for future episodes. Through this process, the agent develops an enhanced understanding of effective plans to iteratively improve its adaptability to novel tasks. While the original Reflexion study (Shinn et al., 2023) utilized a single LLM for both action generation and reflection, our implementation uses distinct LLMs that are fine-tuned for each respective task. Additionally, we extend the reflection process to include successful outcomes, which enables the model to reinforce effective strategies alongside learning from failures. <details> <summary>figures/Reflex.png Details</summary> ![fa4716be](/v1/image/fa4716bea61e290b7ae243bcdeaa42ba51c394baaa72e9dcad97c0f30fe4e401) ### Visual Description ## Flowchart: Reinforcement Learning with Reflection and Long-Term Memory System ### Overview The diagram illustrates a complex reinforcement learning (RL) system integrating long-term memory, reflection, and few-shot learning. It depicts interactions between components such as the Actor, Environment, Language Descriptor, and Reflection module, with feedback loops and memory retention mechanisms. ### Components/Axes 1. **Key Components**: - **Long-term memory**: Stores contextual information (`C_{t-1}`) and few-shot examples. - **Language descriptor**: Processes text observations (`O_t`) and rewards (`r_t`). - **Environment**: Executes low-level actions based on observations and rewards. - **Actor**: Generates high-level actions (`a_t`) using context and task descriptions. - **Reflection**: Analyzes episode outcomes (`C_T`) and task descriptions to produce reflections (`r_x^e`). - **Few-shot examples**: Provide initial task demonstrations. 2. **Flow Connections**: - **Inputs**: - Few-shot examples → Long-term memory. - Text observation (`O_t`) and reward (`r_t`) → Language descriptor. - Previous episode outcomes → Actor and Reflection. - **Outputs**: - Low-level actions → Environment. - High-level actions (`a_t`) → Environment. - Reflections (`r_x^e`) → System feedback. 3. **Memory and Context**: - Context (`C_{t-1}`) is derived from long-term memory and updated iteratively. - Reflections (`r_x^{1:e-1}`) aggregate past episode outcomes. ### Detailed Analysis - **Long-term memory**: Retains `C_{t-1}` (context) and few-shot examples, enabling the system to leverage prior knowledge. - **Language descriptor**: Converts raw text observations (`O_t`) and rewards (`r_t`) into structured inputs for the environment. - **Environment**: Bridges high-level actions (`a_t`) and low-level policies, translating decisions into executable steps. - **Actor**: Combines context (`C_{t-1}`), text observations (`O_t`), and previous outcomes to determine optimal high-level actions. - **Reflection**: Uses episode outcomes (`C_T`) and task descriptions to refine future decisions via `r_x^e`. ### Key Observations 1. **Feedback Loops**: - Reflections (`r_x^e`) and previous outcomes (`r_x^{1:e-1}`) feed back into the Actor and Reflection module, enabling meta-learning. 2. **Few-Shot Learning**: - Initial examples guide the system’s initial behavior, reducing reliance on extensive training data. 3. **Hierarchical Decision-Making**: - The Actor operates at a high level, while the Environment handles low-level execution, creating a modular architecture. ### Interpretation This system combines **reinforcement learning** with **few-shot learning** and **reflection** to enhance adaptability. The Actor’s decisions are informed by both immediate rewards and long-term context, while the Reflection module enables continuous improvement by analyzing past episodes. The integration of few-shot examples suggests the system can generalize from minimal demonstrations, making it efficient for complex tasks with limited data. The feedback loops ensure that the system evolves dynamically, balancing exploration (via the Environment) and exploitation (via the Actor and Reflection). </details> Figure 7: Our illustration of the Reflexion architecture (Shinn et al., 2023), which leverages verbal RL to enable agents to learn from past episodes. 4.1.3 MUSE The MUSE framework, illustrated in Figure 8, builds on mechanisms from ReAct and Reflexion by incorporating additional modules for self-assessment and self-regulation: - World Model ( $M_{w}$ ): An LLM that can be prompted to predict the next observation, reward, and terminal signal given the current observation and action. This LLM works in conjunction with the Actor ( $M_{a}$ ) to generate potential future states and actions (rollout trajectories), enabling look-ahead planning. In text-based domains, such as the experiments presented here, $M_{w}$ can use the same LLM as $M_{a}$ . In fact, the same LLM can be directly prompted to generate diverse trajectories without requiring explicit interaction between the two. - Self-Assessment Model ( $M_{sa}$ ): A language-conditioned neural network that evaluates and scores trajectories generated by the World Model/Actor to assess their alignment and effectiveness for the agent’s goals. Specifically, it predicts task competence, or the probability of task success, for each trajectory. - Self-Regulation ( $M_{sr}$ ): A module that decides a competence-aware course of action based on one of the options outlined in Subsection 3.1.2. For this implementation of MUSE, $M_{sr}$ chooses the first action $a_{t}$ from the rollout trajectory that is most likely to achieve task success, as determined by competence evaluations from $M_{sa}$ . <details> <summary>figures/MUSE.png Details</summary> ![b5ac3487](/v1/image/b5ac3487924d5d43649be21b91745b2bcee92dc376c4c67d94129a273ca3a645) ### Visual Description ## Diagram: Adaptive Learning System Architecture ### Overview The diagram illustrates a hierarchical adaptive learning system integrating world modeling, decision-making, and self-regulation. It features feedback loops, memory components, and environmental interaction, with explicit connections between high-level planning and low-level execution. ### Components/Axes 1. **Core Modules**: - **World Model** (Blue): Processes task descriptions and initial plans - **Actor** (Green): Executes high-level actions - **Self-Assessment** (Purple): Evaluates trajectories - **Self-Regulation** (Yellow): Adjusts strategies based on assessments 2. **Memory & Context**: - **Long-term memory** (Light Green): Stores few-shot examples - **Reflections** (Light Blue): Processes episode outcomes - **Context** (Cₜ₋₁): Maintains historical context 3. **Environment Interaction**: - **Environment**: Provides observations and rewards - **Low-level policies**: Execute specific actions 4. **Flow Elements**: - **Task description** (I): Input to World Model - **Initial plan** (pₑ): Output from World Model - **High-level action** (aₜ): Output from Actor - **Low-level action**: Output from policies 5. **Feedback Loops**: - **Reflections** → **Self-Regulation** → **World Model** - **Episode outcome** → **Reflections** ### Detailed Analysis - **World Model** (Mₐ) receives task descriptions (I) and initial plans (pₑ), integrating context (Cₜ₋₁) and few-shot examples - **Actor** (Mₐ) generates high-level actions (aₜ) based on world model outputs - **Self-Assessment** (Mₛₐ) evaluates trajectories using ranked trajectories from previous episodes - **Self-Regulation** (Mₛᵣ) adjusts strategies based on assessments (Mₛₐ) - **Environment** provides text observations (oₜ) and rewards (rₜ), which feed into language descriptor and low-level policies - **Reflections** (Mᵣˣ) process episode outcomes (rxᵉ) and previous reflections (rx¹ˢ¹), creating updated context (Cₜ) ### Key Observations 1. **Hierarchical Structure**: Clear separation between strategic (World Model/Actor) and operational (policies) components 2. **Memory Integration**: Long-term memory and reflections create temporal continuity 3. **Feedback Density**: Multiple feedback loops (4 explicit) enable continuous adaptation 4. **Modularity**: Components operate semi-independently but with defined interfaces 5. **Temporal Processing**: Explicit handling of current (oₜ₊₁) and previous (Cₜ₋₁) context ### Interpretation This architecture demonstrates a sophisticated closed-loop system where: - **Strategic planning** (World Model) informs **tactical execution** (Actor) - **Performance evaluation** (Self-Assessment) drives **adaptive refinement** (Self-Regulation) - **Environmental feedback** grounds decisions in real-world outcomes - **Memory systems** prevent catastrophic forgetting while enabling generalization The system appears designed for complex, dynamic environments requiring both immediate responsiveness (low-level policies) and long-term strategic adaptation (world model). The explicit ranking of trajectories suggests a preference for exploring high-performing strategies while maintaining exploration through few-shot examples. The reflection component's integration of both current outcomes and historical reflections indicates a meta-cognitive approach to learning. </details> Figure 8: Illustration of the MUSE architecture for LLM agents, which implements the metacognitive cycle to iteratively solve unknown tasks. World Model/Actor During deployment, the World Model/Actor generates several potential future state-action sequences with a horizon $H$ at each time step $t$ . The diversity of these rollout trajectories, denoted by $\tau_{t}=\{a_{t},r_{t},o_{t+1},...,a_{t+H-1},r_{t+H-1},o_{t+H}\}$ , is controlled by the temperature setting of the LLM. These trajectories represent hypothetical paths extending from the current observation $o_{t}$ and context $c_{t-1}$ , guided by the task description $I$ and task-specific reflections $rx^{\{1:e-1\}}$ stored in memory. For this implementation, we explicitly did not specify a horizon $H$ ; instead, the LLM was allowed to generate trajectories up to the maximum length permitted by its context window. The temperature of the LLM was set to 0.5, and five rollout trajectories were generated at each time step. Self-Assessment The Self-Assessment Model ( $M_{sa}$ ) utilizes a transformer encoder $\mathcal{M}$ (SentenceTransformers, 2024) and an MLP $g_{\eta}$ to predict the probability of task success (Equation 6) for rollout trajectories generated by the World Model/Actor before their actual execution in the environment. Specifically, $M_{sa}$ evaluates the alignment and effectiveness of potential trajectories for the task at hand. The algorithm for training $M_{sa}$ and using it for evaluation is detailed in Algorithm 2. $$ \displaystyle\begin{aligned} y_{\text{pred}}=M_{sa}(I,P^{e},\tau)\end{aligned} \tag{6} $$ ifnextchar gobble Here, $\tau$ represents a trajectory and $P^{e}$ denotes the initial language-based plan generated by the World Model/Actor LLM at the start of episode $e$ based on task description $I$ . See Supplementary Material for several illustrative examples of the initial plan $P^{e}$ . The output layer of $M_{sa}$ employs a sigmoid activation function to yield a probability of task success. A threshold of 0.5 is used to convert these probabilities into binary outcomes (success or failure). During pre-deployment training and deployment adaptation, $M_{sa}$ is trained to minimize the binary cross-entropy loss between the predicted task success ( $y_{\text{pred}}$ ) and the actual binary outcome ( $y$ ) as follows: $$ \displaystyle\begin{aligned} \mathcal{L}(y,y_{\text{pred}})\doteq-\left[y\log(y_{\text{pred}})+(1-y)\log(1-y_{\text{pred}})\right]\end{aligned} \tag{7} $$ ifnextchar gobble To increase the number of training samples and mitigate overfitting, the training trajectories are segmented into non-overlapping chunks, each containing four action-observation pairs. For example, the initial chunk within an episode includes pairs ( $a_{1}$ , $o_{1}$ ) through ( $a_{4}$ , $o_{4}$ ), while the next spans ( $a_{5}$ , $o_{5}$ ) through ( $a_{8}$ , $o_{8}$ ), and so on. Algorithm 2 Training and evaluation of Self-Assessment Model $M_{sa}$ 1: Input: Dataset $\mathcal{D}=\{(I,P^{e},\tau,y)\}$ , transformer encoder $\mathcal{M}$ , MLP $g_{\eta}$ 2: Output: Trained evaluator $g_{\eta}$ 3: function TrainEvaluator ( $\mathcal{D}$ , $\mathcal{M}$ , $g_{\eta}$ ) 4: for each $(I,P^{e},\tau,y)∈\mathcal{D}$ do 5: $Z←\mathcal{M}(I)$ $\triangleright$ Generate task embedding 6: $S←\mathcal{M}(\tau,P^{e})$ $\triangleright$ Generate trajectory and plan embedding 7: $X←\text{concat}(S,Z)$ 8: $y_{\text{pred}}← g_{\eta}(X)$ $\triangleright$ Predict task success probability 9: Update $g_{\eta}$ parameters to minimize $\mathcal{L}(y,y_{\text{pred}})$ 10: end for 11: return Trained MLP $g_{\eta}$ 12: end function 13: function EvaluateTrajectory ( $I$ , $P^{e}$ , $\tau$ , $\mathcal{M}$ , $g_{\eta}$ ) 14: $Z←\mathcal{M}(I)$ 15: $S←\mathcal{M}(\tau,P^{e})$ 16: $X←\text{concat}(S,Z)$ 17: $y_{\text{pred}}← g_{\eta}(X)$ 18: return $y_{\text{pred}}$ 19: end function MUSE Framework The pre-deployment training and deployment adaptation procedures for the MUSE agent are detailed in Algorithms 3 and 4, respectively. For pre-deployment training, we first train the Reflexion agent (Shinn et al., 2023) on each of the training tasks separately to collect data. The constituent models of the MUSE agent are then trained with multi-task supervised learning to benefit from the diversity of experiences. Specifically, the Reflection LLM ( $M_{rx}$ ) in MUSE is trained using Direct Preference Optimization (DPO) with a preference dataset (Rafailov et al., 2024) that is created by comparing reflections of success and failure across the training tasks. A success (positive) reflection occurs when the agent fails in episode $e_{i}$ but succeeds in episode $e_{i+1}$ following the reflection. Conversely, a failure (negative) reflection happens when the agent succeeds in episode $e_{i}$ but fails in episode $e_{i+1}$ despite the reflection. DPO directly optimizes $M_{rx}$ to generate generalizable reflections that maximize the likelihood of satisfying these preferences (i.e., increase the likelihood of success and decrease the likelihood of failure). This enables MUSE to iteratively improve its ability to recover from failures and adapt to novel tasks through a more effective reflection process. The Actor LLM ( $M_{a}$ ) is trained using Supervised Fine-Tuning (SFT) with only successful episodes, for simplicity, across the training tasks. By imitating the behavior demonstrated in these episodes, $M_{a}$ learns to align its policy with actions seen during successful episodes. This joint multi-task fine-tuning process equips MUSE with generalizable success-driven strategies. Note that we use Low-Rank Adaptation (LoRA) (Hu et al., 2021) for parameter-efficient fine-tuning of both the $M_{a}$ and $M_{rx}$ LLMs. Additionally, the Self-Assessment Model ( $M_{sa}$ ) undergoes supervised learning with pertinent multi-task data, which maps trajectory chunks to corresponding episode outcomes. During deployment, when the agents encounter a novel task, they engage in sequential episodes until success is achieved. MUSE employs competence-aware planning, as described above, to choose its actions. Furthermore, MUSE can perform online updates to each of its constituent models as new data becomes available from the novel task. Note that the baseline agents, ReAct and Reflexion, can also respond adaptively to new experiences but rely only on in-context prompting and/or reflection. However, these methods do not involve updating model parameters, which leads to knowledge being stored only for the short term. This constraint hinders the transfer of knowledge across episodes and tasks. MUSE overcomes these limitations by continually updating the weights of its models, enabling more effective knowledge transfer. Algorithm 3 Procedure for pre-deployment training of the MUSE agent Training dataset ${D_{in}}\sim$ in-distribution tasks Specify max. training episodes, max. steps 1: function PreTrainModels ( $D_{in}$ , $M_{a}$ , $M_{rx}$ , $M_{sa}$ , $M_{w}$ ) 2: Initialize $M_{sa}$ , $M_{w}$ , memory buffer 3: for each task $\sim{D_{in}}$ do 4: Initialize $M_{a}$ , $M_{rx}$ for the Reflexion agent 5: Add $I$ to memory buffer 6: Set $t=0$ , $e=0$ 7: while $e<$ max. training episodes do 8: while $t<$ max. steps & task-not-solved do 9: Generate $a_{t}$ using $M_{a}$ and submit to simulator 10: Obtain $o_{t+1}$ and $r_{t}$ , append $o_{t}$ , $a_{t}$ , $r_{t}$ to $c_{t}$ 11: Increment $t$ 12: end while 13: Generate $rx^{e}$ using $M_{rx}$ 14: Append episode outcome, $c_{t}$ , $rx^{e}$ to memory buffer 15: Increment $e$ 16: end while 17: end for 18: Use all data in memory buffer to train $M_{a}$ , $M_{rx}$ , $M_{sa}$ , $M_{w}$ for MUSE 19: return Trained $M_{a}$ , $M_{rx}$ , $M_{sa}$ , $M_{w}$ 20: end function Algorithm 4 Procedure for deployment adaptation of the MUSE agent Test dataset ${D_{out}}\sim$ out-of-distribution tasks Specify max. adaptation episodes 1: function AdaptModels ( $D_{out}$ , $M_{a}$ , $M_{rx}$ , $M_{sa}$ , $M_{w}$ ) 2: Initialize memory buffer 3: for each task $\sim{D_{out}}$ do 4: Set $M_{a}$ , $M_{rx}$ , $M_{sa}$ , $M_{w}$ to pre-deployment trained weights 5: Add $I$ to memory buffer 6: Set $t=0$ , $e=0$ 7: while $e<$ max. adaptation episodes do 8: while $t<$ max. steps & task-not-solved do 9: Generate rollout trajectories using $M_{w}$ and $M_{a}$ 10: Evaluate each trajectory using $M_{sa}$ 11: Select the best action $a_{t}$ and submit to simulator 12: Obtain $o_{t+1}$ and $r_{t}$ , append $o_{t}$ , $a_{t}$ , $r_{t}$ to $c_{t}$ 13: Increment $t$ 14: end while 15: Generate $rx^{e}$ using $M_{rx}$ 16: Append episode outcome, $c_{t}$ , $rx^{e}$ to memory buffer 17: Increment $e$ 18: end while 19: Use data in memory buffer to update $M_{a}$ , $M_{rx}$ , $M_{sa}$ , $M_{w}$ 20: Test updated MUSE agent on the current task over the test episodes 21: end for 22: end function 4.2 ALFWorld Experiments For the LLM-based implementation, we evaluated our approach within the ALFWorld simulator (Shridhar et al., 2020) and compared it against ReAct (Yao et al., 2022) and Reflexion (Shinn et al., 2023) as the baselines. ALFWorld is a synthetic, text-based game simulator with diverse interactive environments that challenge agents to solve multi-step tasks (e.g., “Find two plates and put them in a cabinet”). A unique aspect of ALFWorld is its emphasis on spatial and commonsense reasoning, requiring agents to infer probable locations of common household items; for example, recognizing that plates are typically found in kitchens. This characteristic makes ALFWorld particularly well-suited for leveraging the pre-trained commonsense knowledge of LLMs. For these experiments, we selected the Mistral-7B-Instruct-v0.2 as the primary LLM (Jiang et al., 2023). This LLM features a substantial context window size of 32,000 tokens, allowing it to process large chunks of data in a single pass for improved in-context learning. Similar to the World-Model-based experiments, we employed a two-stage protocol comprising pre-deployment training on known tasks followed by deployment adaptation to unknown tasks. Each episode in these experiments had a maximum time limit of 50 steps. To further explore the robustness of our framework, we conducted additional experiments using smaller or less capable LLMs such as Mistral-7B-OpenOrca (Mukherjee et al., 2023; Open-Orca, 2024) and OpenELM-3B-Instruct (Mehta et al., 2024). The MUSE agent underwent pre-deployment training using 140 household tasks from ALFWorld’s in-distribution (seen) set. For deployment adaptation, we evaluated all agents on 134 household tasks from ALFWorld’s out-of-distribution (unseen) set. The agents were exposed to one novel task at a time, starting with the pre-deployment trained weights (for MUSE) or pre-trained LLM weights (for ReAct and Reflexion), under two conditions: (1) five test episodes for each task without any adaptation and (2) five adaptation episodes followed by five test episodes for each task. The agents were assessed for performance on the novel tasks over the test episodes. Note that there were no weight updates to the MUSE agent during these episodes. While it is possible to continually update all constituent models of MUSE (namely, $M_{w}$ , $M_{a}$ , $M_{rx}$ , $M_{sa}$ ) using new data acquired from adaptation episodes, we chose to update only the MLP portion ( $g_{\eta}$ ) of $M_{sa}$ to minimize computational time without loss of any generality. We used a replay buffer that included both training data and new adaptation data to update $M_{sa}$ without catastrophic forgetting of pre-deployment trained performance. 4.2.1 Metrics Self-Assessment We used metacognitive accuracy and AUROC2 to evaluate how well MUSE predicts its success on the novel tasks. MUSE was evaluated for each novel task separately over five test episodes, at the end of pre-deployment training and after five adaptation episodes. During each test episode, the trajectory was segmented into non-overlapping chunks, each containing four action-observation pairs. MUSE’s chunk-wise self-assessment predictions were then collected for evaluation, and the predictions for all chunks across tasks were compared against the true labels to compute the metacognitive accuracy and AUROC2 metrics. Note that, as the baseline agents, ReAct and Reflexion, lack a self-assessment module to predict episode outcomes, we cannot make a meaningful comparison between them and MUSE for self-assessment. Self-Regulation We evaluated how well the agents generalize to unknown situations using two metrics: the percentage of the novel tasks solved and the average time to task completion. Each agent was assessed for each novel task separately over the five test episodes. The percentage of episodes in which the agent completed the task within the maximum time limit was averaged across all the novel tasks to calculate the success rate. Similarly, the number of time steps required to achieve success in each episode was averaged across all episodes and tasks to compute the time-to-completion metric. There was a penalty of 100 time steps for failures. 4.2.2 Results Self-Assessment The self-assessment results are summarized in Table 2, with the corresponding Type 2 ROC curves shown in Figure 9. Following the five adaptation episodes, the MUSE agent achieved an AUROC2 of 0.93 and a metacognitive accuracy of 85%, demonstrating a high level of competence awareness. Table 2: Self-assessment performance of the MUSE agent on the novel tasks in the ALFWorld environment, at the end of pre-deployment training (No Adaptation) and after five adaptation episodes (Adaptation). | Condition | AUROC2 | Metacognitive Accuracy | | --- | --- | --- | | No Adaptation | 0.66 | 60% | | Adaptation | 0.93 | 85% | <details> <summary>figures/self_aware_LLM.png Details</summary> ![ef914683](/v1/image/ef914683b1da40c0f5199715a0a84c7c74d5c3c00dc0723286b0790adc150fe2) ### Visual Description ## ROC Curves: MUSE Model Performance Comparison ### Overview The image displays two side-by-side Receiver Operating Characteristic (ROC) curves comparing the performance of the MUSE model under different conditions. Each curve is plotted against a diagonal dashed baseline representing random guessing (50% accuracy). The left graph shows a gradual improvement in true positive rate (TPR) with increasing false positive rate (FPR), while the right graph demonstrates rapid early gains followed by plateauing performance. ### Components/Axes - **X-axis (False Positive Rate)**: Labeled "False Positive Rate" with ticks at 0.0, 0.2, 0.4, 0.6, 0.8, 1.0 - **Y-axis (True Positive Rate)**: Labeled "True Positive Rate" with ticks at 0.0, 0.2, 0.4, 0.6, 0.8, 1.0 - **Legend**: Positioned at bottom-right of each graph, labeled "MUSE" with orange line indicator - **Baseline**: Diagonal dashed gray line from (0,0) to (1,1) in both graphs - **Data Lines**: Orange solid lines representing MUSE model performance ### Detailed Analysis **Left Graph (Gradual Improvement):** - Starts at (0,0) - Reaches approximately (0.2, 0.4) - Passes through (0.4, 0.7) - Ends near (0.8, 0.95) - Final point approaches (1,1) with FPR=0.95 and TPR=0.98 **Right Graph (Rapid Early Gains):** - Starts at (0,0) - Reaches (0.1, 0.9) within first 10% FPR - Plateaus at TPR=0.95 between FPR=0.1 and 0.3 - Final point at (1,1) with FPR=0.98 and TPR=1.0 ### Key Observations 1. **Performance Trade-off**: Left graph shows balanced TPR/FPR trade-off, while right graph prioritizes early detection at higher false positives 2. **Saturation Point**: Right graph's plateau at TPR=0.95 suggests diminishing returns after initial gains 3. **Baseline Comparison**: Both curves significantly outperform the random guessing baseline (dashed line) 4. **Final Performance**: Right graph achieves perfect TPR=1.0 at FPR=0.98 vs left graph's TPR=0.98 at FPR=0.95 ### Interpretation The contrasting curves suggest two distinct model behaviors: 1. **Left Graph**: Indicates a model with consistent learning across all thresholds, potentially suitable for balanced decision-making where false positives are costly 2. **Right Graph**: Represents a model optimized for early detection, possibly at the expense of precision. The plateau suggests limited capacity to distinguish between classes beyond initial thresholds. The divergence in performance patterns could reflect differences in: - Training data distribution - Model architecture choices - Threshold selection strategies - Class imbalance handling The right graph's perfect TPR at high FPR might indicate overfitting or data saturation, while the left graph's gradual curve suggests more generalizable performance characteristics. </details> Figure 9: Type 2 ROC curves of the MUSE agent for self-assessment of competence on the novel tasks in the ALFWorld environment, at the end of pre-deployment training (left) and after five adaptation episodes (right). Table 3: Self-Regulation: Success rate of various agents on the novel tasks, at the end of pre-deployment training (No Adaptation) and after five adaptation episodes (Adaptation). | Agent | No Adaptation | Adaptation | | --- | --- | --- | | ReAct | 35% | 35% | | Reflexion | 45% | 51% | | Reflexion with supervised learning | 62% | 65% | | MUSE with the lowest self-assessment | 59% | 61% | | MUSE with random self-assessment | 68% | 69% | | MUSE | 84% | 90% | Table 4: Self-Regulation: Average number of time steps to success per episode for various agents on the novel tasks, at the end of pre-deployment training (No Adaptation) and after five adaptation episodes (Adaptation). | Agent | No Adaptation | Adaptation | | --- | --- | --- | | ReAct | 97 | 97 | | Reflexion | 77 | 66 | | Reflexion with supervised learning | 61 | 54 | | MUSE with the lowest self-assessment | 65 | 62 | | MUSE with random self-assessment | 53 | 49 | | MUSE | 43 | 38 | Table 5: Self-Regulation: Success rate of various agents with smaller or less capable LLMs on the novel tasks, after five adaptation episodes. | Agent | Mistral-7B-OpenOrca | OpenELM-3B-Instruct | | --- | --- | --- | | ReAct | 23% | 4% | | Reflexion | 27% | 9% | | MUSE | 58% | 55% | To highlight the importance of self-assessment, we provide examples of trajectories generated by different agents. As shown in Supplementary Section 8.5, the key to MUSE’s success lies in its continual competence awareness. MUSE generates several rollout trajectories, assigns competence scores to each, and selects the one most likely to result in task success. For the given task, MUSE begins with an initial plan similar to that of Reflexion: “find a bowl, clean it at the sink, and then place it in the cabinet.” However, MUSE sets itself apart by engaging in deliberate actions that ensure steady progress toward the goal with increasing competence. After picking up the bowl from cabinet 1, MUSE evaluates multiple options, such as follows: 1. “think: Now I find a bowl (1). Next, I need to take it..OK..> take bowl 1 from cabinet 1.” … Score: 0.6284 1. “think: Now I take a bowl (1). Next, I need to go to sinkbasin (1) and clean it.” … Score: 0.8689 1. “think: Now I find a bowl (1). Next, I need to take it..OK..> take bowl 1 from cabinet 1. Nothing happens.” … Score: 0.6417 In this instance, MUSE selects Trajectory 1 due to its higher competence score, which indicates a greater likelihood of task success. By continually assessing its competence, MUSE adapts its strategy in real time to effectively navigate the environment’s complexities. It maintains a coherent plan and avoids redundant or detrimental actions that could lead to confusion or failure. In contrast, Reflexion fails due to its lack of competence awareness. After acquiring the bowl, Reflexion redundantly decides to find and take a bowl again. This leads to a faulty sequence of actions, causing Reflexion to enter an incorrect state from which it cannot recover. Without competence awareness, Reflexion fails to recognize that it already possesses the bowl. Its actions become increasingly erratic, failing to contribute toward task completion. For the complete trajectory, refer to Supplementary Section 8.5. In summary, competence awareness allows MUSE to adapt more effectively to unfamiliar situations. By avoiding low-competence rollout trajectories, it steers clear of actions leading to failure, promotes exploration within the agent’s capabilities, and prevents the agent from getting stuck in unrecoverable states. This approach results in smarter exploration and more effective online learning in response to novel scenarios. Self-Regulation The self-regulation results for MUSE and the two baseline agents are summarized in Tables 3 and 4. These results demonstrate that MUSE solves more novel tasks and requires fewer time steps to completion per episode, both at the end of pre-deployment training and after five adaptation episodes. Notably, MUSE achieved a high success rate of 90% on the 134 novel tasks, outperforming the baseline agents of ReAct and Reflexion by >150% and >75%, respectively. It’s important to note that all agents in the pertinent comparisons have access to the same data, ensuring a fair evaluation. ReAct and Reflexion can also respond adaptively to new experiences but rely only on in-context prompting and/or reflection. So, the observed performance gains for MUSE are from its ability to leverage available data more effectively by performing competence-aware planning, not from access to any privileged information. To clarify and validate this point, we conducted additional experiments comparing: - Reflexion with supervised learning across the training tasks during pre-deployment (namely, DPO and SFT) - MUSE with self-regulation based on the lowest self-assessment - MUSE with self-regulation based on random self-assessment The self-regulation results for these control conditions are also summarized in Tables 3 and 4. Reflexion with supervised learning during pre-deployment can consolidate knowledge across the training tasks, but exhibits only a limited improvement over its original implementation (Shinn et al., 2023) in adaptation to the novel tasks. This suggests that even a generalized reflection mechanism is less effective in novel scenarios without self-assessment and self-regulation. Furthermore, variants of MUSE using poor (random or pessimistic) self-assessment signals underperform, validating the importance of self-assessment for effective self-regulation. Note that even when self-assessment is flawed, the performance of the MUSE agent does not collapse because the Actor LLM ( $M_{a}$ ) has benefited from pre-deployment training to already generate reasonable plans for competence-aware selection. To further evaluate the contribution of the self-assessment and self-regulation framework in MUSE beyond LLM-powered reasoning, we conducted additional experiments using smaller or less capable LLMs. Table 5 presents the post-adaptation performance of MUSE compared to the baseline agents when employing smaller or less capable LLM variants. MUSE maintained relatively higher success rates (55-58%) on the novel tasks, achieving >2x and >6x better performance than Reflexion when using Mistral-7B-OpenOrca and OpenELM-3B-Instruct, respectively. These results suggest that metacognition plays a particularly significant role when operating under constrained conditions, such as with smaller or less capable LLMs. 4.3 Discussion Through these experiments, we evaluated the metacognitive capabilities of MUSE to showcase its superior performance over the baseline agents of ReAct and Reflexion. With reliable competence awareness on the novel tasks, the MUSE agent can make informed decisions by avoiding rollout trajectories likely to lead to failure and prioritizing those with a higher probability of task success for more effective online adaptation. MUSE’s ability to learn efficiently from a limited number of adaptation episodes is particularly critical in unknown situations and environments, where task complexity and novelty can vary significantly. Furthermore, its strong performance even with less-capable LLMs suggests that incorporating metacognitive components into mainstream AI agents can enhance performance under resource-constrained conditions. Overall, the LLM-based experiments demonstrated that integrating self-assessment and self-regulation into LLM-based agents can significantly improve their ability to handle unknown scenarios. 5 Conclusions In this article, we introduced the Metacognition for Unknown Situations and Environments (MUSE) framework to integrate self-assessment and self-regulation into AI agents to enhance their adaptability to unknown situations and environments. MUSE uses self-assessed competence to intelligently guide an iterative trial-and-error process to identify the right strategy. In contrast, current AI agents either blindly execute their routine or policy, risking catastrophic failures, or freeze as a result of runtime anomaly detection. These limitations arise because big data-driven solutions inherently struggle to adapt to novel scenarios with little or no training data. We presented two implementations of MUSE, one for MBRL agents and another for LLM-based agents, to instantiate the metacognitive cycle. In both cases, our experiments demonstrated that MUSE agents outperform baseline methods in handling novel tasks. These results show that MUSE facilitates problem-solving in unfamiliar environments by vetoing potentially catastrophic solutions, guiding the search toward strategies that leverage the agent’s strengths, and learning online from the environment with each attempt. In high-stakes applications like autonomous driving, MUSE can facilitate safety by relying on its self-assessment as a form of competence calibration. Furthermore, the online learning during deployment enables MUSE to fix and overcome potential metacognitive misjudgments in novel scenarios. This avoids the infinite regress of needing to calibrate the preceding lower-order calibration. In summary, by enabling agents to be aware of their competence and to regulate their strategies accordingly, MUSE offers a promising pathway toward developing more resilient and versatile AI systems. 6 Future Work While MUSE outperforms the baseline AI agents, metacognition in AI remains a nascent and exploratory field. The scope of metacognitive abilities is broad, offering several promising directions for future research. For instance, MUSE could be expanded to include metacognitive judgments that further enhance an agent’s performance and learning. These include the Feeling of Knowing to help the agent persist in difficult, novel scenarios; the Judgment of Learning to facilitate resource-efficient learning; and Source Monitoring to assess the reliability of its knowledge. Another key avenue is integrating MUSE with lifelong learning techniques, such as generative experience replay and selective plasticity, to incrementally update its constituent models in new scenarios while mitigating catastrophic forgetting (Rostami et al., 2019; Kolouri et al., 2020; Kudithipudi et al., 2023). Such an integration would enable agents to retain and build upon previously learned knowledge while continually acquiring new skills, without requiring the storage of data from all prior conditions in the replay buffer, as was done in the current study. Another exciting direction for future research involves applying MUSE to real-world domains, such as autonomous driving and robotics in unstructured settings. Investigating these domains for tasks with longer time horizons and larger state and action spaces will provide valuable insights into the scalability, robustness, and practicality of the MUSE framework. In this regard, MCTS methods can be adapted to leverage the self-assessment model from MUSE as the node evaluation function for guiding the search, enabling more effective exploration and problem-solving in complex environments. Another compelling avenue is expanding the metacognitive framework to incorporate additional aspects of human cognition that are currently absent in AI systems. For instance, integrating Theory-of-Mind could enhance agents’ abilities to understand and predict human behaviors, enabling more effective collaboration in social or teamwork-driven environments. Additionally, exploring multi-agent interactions within the MUSE framework offers significant potential. Enabling multiple metacognitive AI agents to interact, collaborate, negotiate, or compete could lead to more sophisticated and efficient problem-solving at the group level. Our research highlights the potential of psychology- and brain-inspired approaches to address the limitations of current AI systems, particularly their dependence on extensive training data. By equipping AI agents with greater flexibility and adaptability, MUSE paves the way for more resilient and versatile systems capable of operating across a wide range of environments. We anticipate that future research will build upon and refine these ideas, advancing the field of metacognitive AI and expanding its applications across diverse domains. 7 Acknowledgments The authors would like to thank Neale Ratzlaff for his invaluable contributions to this research and his constructive feedback. 8 Supplementary Material Note that the trajectory examples provided below for the LLM-based experiments are exact copies of the LLM outputs. 8.1 Illustrative ALFWorld trajectories for in-context prompting. (Yao et al., 2022) Interact with a household to solve a task. Here are two examples. // You are in the middle of a room. Looking quickly around you, you see a cabinet 13, a cabinet 12, a cabinet 11, a cabinet 10, a cabinet 9, a cabinet 8, a cabinet 7, a cabinet 6, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a coffeemachine 1, a countertop 1, a diningtable 1, a drawer 1, a fridge 1, a garbagecan 1, a microwave 1, a shelf 3, a shelf 2, a shelf 1, a sinkbasin 1, a stoveburner 4, a stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster 1. Your task is to: put a hot apple in fridge. >think: To solve the task, I need to find and take an apple, then heat it with microwave, then put it in fridge. OK. >think: First I need to find an apple. An apple is more likely to appear in fridge (1), diningtable (1), coffeetable (1), drawer (1), cabinet (1-13), garbagecan (1). I can check one by one, starting with fridge 1. OK. >go to fridge 1 The fridge 1 is closed. >open fridge 1 You open the fridge 1. The fridge 1 is open. In it, you see a cup 1, and a egg 1. >go to diningtable 1 On the diningtable 1, you see a apple 1, a bread 1, a fork 3, a fork 2, a fork 1, a mug 2, a peppershaker 3, a plate 2, a pot 1, a soapbottle 3, a spatula 1, a spoon 3, a spoon 2, a spoon 1, a winebottle 3, and a winebottle 2. >think: Now I find an apple (1). Next, I need to take it. OK. >take apple 1 from diningtable 1 You pick up the apple 1 from the diningtable 1. >think: Now I take an apple (1). Next, I need to go to a microwave (1) and heat it. OK. >go to microwave 1 The microwave 1 is closed. >heat apple 1 with microwave 1 You heat the apple 1 using the microwave 1. >think: Now I heat an apple (1). Next, I need to put it in/on fridge 1. OK. >go to fridge 1 The fridge 1 is open. In it, you see a cup 1, and a egg 1. >put apple 1 in/on fridge 1 You put the apple 1 in/on the fridge 1. // You are in the middle of a room. Looking quickly around you, you see a cabinet 10, a cabinet 9, a cabinet 8, a cabinet 7, a cabinet 6, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a coffeemachine 1, a countertop 3, a countertop 2, a countertop 1, a diningtable 1, a drawer 6, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a fridge 1, a garbagecan 1, a microwave 1, a sinkbasin 1, a stoveburner 4, a stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster 1. Your task is to: heat some egg and put it in diningtable. >think: To solve the task, I need to find and take an egg, then heat it with microwave, then put it in diningtable. OK. >think: First I need to find an egg. An egg is more likely to appear in fridge (1), countertop (1-3), diningtable (1), stoveburner (1-4), toaster (1), garbagecan (1), cabinet (1-10). I can check one by one, starting with fridge 1. OK. >open fridge 1 You open the fridge 1. The fridge 1 is open. In it, you see a lettuce 2, a mug 2, and a potato 3. >go to countertop 1 On the countertop 1, you see a bread 1, a fork 1, and a saltshaker 1. >go to countertop 2 On the countertop 2, you see nothing. >go to countertop 3 On the countertop 3, you see a bowl 1, a butterknife 1, a egg 2, a kettle 2, a plate 1, a sink 1, and a spatula 2. >think: Now I find an egg (2). Next, I need to take it. OK. >take egg 2 from countertop 3 You pick up the egg 2 from the countertop 3. >think: Now I take an egg (2). Next, I need go to a microwave (1) and heat it. OK. >go to microwave 1 The microwave 1 is closed. >heat egg 2 with microwave 1 You heat the egg 2 using the microwave 1. >think: Now I heat an egg (2). Next, I need to put it in/on diningtable 1. OK. >go to diningtable 1 On the diningtable 1, you see a apple 2, a bread 3, a egg 1, a kettle 1, a knife 1, a mug 1, a papertowelroll 1, a peppershaker 2, a potato 1, a soapbottle 1, and a spatula 1. >put egg 2 in/on diningtable 1 You put the egg 2 in/on the diningtable 1. 8.2 Example of ReAct’s decision-making | Task: You are in the middle of a room. Looking quickly around you, you see a cabinet 6, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a coffeemachine 1, a countertop 3, a countertop 2, a countertop 1, a drawer 3, a drawer 2, a drawer 1, a fridge 1, a garbagecan 1, a microwave 1, a shelf 3, a shelf 2, a shelf 1, a sinkbasin 1, a stoveburner 4, a stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster 1.Your task is to: put a hot tomato in garbagecan. | | --- | | ReAct’s trajectory: ReAct successfully identifies and picks a tomato. However, it encounters difficulties when attempting to heat it. The agent repetitively navigates to each of the stove burners, resulting in a continuous loop. | | Conclusion: The trajectory ends with the agent trapped in a repetitive sequence of actions, unable to adjust its approach or adapt to the task of heating the tomato. | >action: think: To solve the task, I need to find and take a tomato, then heat it with stoveburner, then put it in garbagecan. observation: OK. >action: think: First I need to find a tomato. A tomato is more likely to appear in fridge (1), countertop (1-3), cabinet (1-6), garbagecan (1), drawer (1-3). I can check one by one, starting with fridge 1. observation: OK. >action: open fridge 1 observation: Nothing happens. >action: go to countertop 1 observation: On the countertop 1, you see a dishsponge 1, and a tomato 1. >action: think: Now I find a tomato (1). Next, I need to take it. observation: OK. >action: take tomato 1 from countertop 1 observation: You pick up the tomato 1 from the countertop 1. >action: think: Now I take a tomato (1). Next, I need to go to a stoveburner (1-4) and heat it. observation: OK. >action: go to stoveburner 1 observation: On the stoveburner 1, you see nothing. >action: go to stoveburner 2 observation: On the stoveburner 2, you see nothing. >action: go to stoveburner 3 observation: On the stoveburner 3, you see nothing. >action: go to stoveburner 4 observation: On the stoveburner 4, you see nothing. >action: think: Now I need to find a stoveburner with a hot surface. I can check stoveburners 1-4. observation: OK. >action: go to stoveburner 1 observation: On the stoveburner 1, you see nothing. >action: go to stoveburner 2 observation: On the stoveburner 2, you see nothing. >action: go to stoveburner 3 observation: On the stoveburner 3, you see nothing. >action: go to stoveburner 4 observation: On the stoveburner 4, you see nothing. >action: think: I need to turn on a stoveburner and wait for it to heat up. I can check stoveburners 1-4. observation: OK. >action: turn on stoveburner 1 observation: Nothing happens. >action: turn on stoveburner 2 observation: Nothing happens. >action: turn on stoveburner 3 observation: Nothing happens. >action: turn on stoveburner 4 observation: Nothing happens. >action: think: I need to find a way to heat up a stoveburner. I can check the stove, the microwave, or the oven. observation: OK. >action: check stove observation: Nothing happens. >action: check microwave observation: Nothing happens. >action: check oven observation: Nothing happens. >action: think: I need to find a source of heat. I can check the stove, the microwave, or the oven. observation: OK. >action: check stove observation: Nothing happens. >action: check microwave observation: Nothing happens. >action: check oven observation: Nothing happens. >action: think: I need to find a way to heat up a tomato. I can check the stove, the microwave, or the oven. observation: OK. >action: check stove observation: Nothing happens. >action: check microwave observation: Nothing happens. >action: check oven observation: Nothing happens. >action: think: I need to find a way to heat up a tomato. I can check the stove, the microwave, or the oven. observation: OK. >action: check stove observation: Nothing happens. >action: check microwave observation: Nothing happens. >action: check oven observation: Nothing happens. >action: think: I need to find a way to heat up a tomato. I can check the stove, the microwave, or the oven. observation: OK. >action: check stove observation: Nothing happens. >action: check microwave observation: Nothing happens. >action: check oven observation: Nothing happens. >action: think: I need to find a way to heat up a tomato. I can check the stove, the microwave, or the oven. observation: OK. >action: check stove observation: Nothing happens. >action: check microwave observation: Nothing happens. >action: check oven observation: Nothing happens. >action: think: I need to find a way to heat up a tomato. I can check the stove, the microwave, or the oven. observation: OK. >action: check stove observation: Nothing happens. >action: check microwave observation: Nothing happens. >action: check oven observation: Nothing happens. 8.3 Example of Reflexion’s decision-making, demonstrating improvements over ReAct | Task: You are in the middle of a room. Looking quickly around you, you see a cabinet 6, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a coffeemachine 1, a countertop 3, a countertop 2, a countertop 1, a drawer 3, a drawer 2, a drawer 1, a fridge 1, a garbagecan 1, a microwave 1, a shelf 3, a shelf 2, a shelf 1, a sinkbasin 1, a stoveburner 4, a stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster 1.Your task is to: put a hot tomato in garbagecan. | | --- | | Reflexive feedback from first episode: “In this environment, my plan was to find a tomato, heat it on a stoveburner, and put it in the garbage can. However, I was unable to find a stoveburner with a hot surface. I should have checked the oven or microwave for alternative methods of heating the tomato. I will try to heat the tomato in the oven or microwave in the next trial.” | | Reflexion’s trajectory: The Reflexion agent reflects on its initial failure to heat the tomato using the stove burners and identifies the potential to utilize alternative heating methods. In subsequent episodes, it successfully uses the microwave to heat the tomato. | | Conclusion: This trajectory demonstrates the agent’s ability to learn from past mistakes and adapt its strategy, ultimately completing the task successfully by heating the tomato in the microwave and placing it in the garbage can. | >action: think: To solve the task, I need to find and take a tomato, then heat it, and put it in garbagecan. observation: OK. >action: think: First I need to find a tomato. A tomato is more likely to appear in fridge (1), countertop (1-3), cabinet (1-6), garbagecan (1). I can check one by one, starting with fridge 1. observation: OK. >action: open fridge 1 observation: Nothing happens. >action: go to countertop 1 observation: On the countertop 1, you see a dishsponge 1, and a tomato 1. >action: think: Now I find a tomato (1). Next, I need to take it. observation: OK. >action: take tomato 1 from countertop 1 observation: You pick up the tomato 1 from the countertop 1. >action: think: Now I take a tomato (1). Next, I need to heat it. observation: OK. >action: think: I can heat a tomato in the microwave (1), oven, or on a stoveburner (1-4). I will try the microwave first. observation: OK. >action: go to microwave 1 observation: The microwave 1 is closed. >action: heat tomato 1 with microwave 1 observation: You heat the tomato 1 using the microwave 1. >action: think: Now I heat a tomato (1). Next, I need to put it in garbagecan 1. observation: OK. >action: go to garbagecan 1 observation: On the garbagecan 1, you see a egg 3, and a papertowelroll 1. >action: put tomato 1 in/on garbagecan 1 observation: You put the tomato 1 in/on the garbagecan 1. 8.4 Limitations of Reflexion compared to MUSE in strategic exploration and task execution | Col1 | | --- | | Task: You are in the middle of a room. Looking quickly around you, you see a armchair 1, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a dresser 1, a garbagecan 1, a safe 1, a shelf 12, a shelf 11, a shelf 10, a shelf 9, a shelf 8, a shelf 7, a shelf 6, a shelf 5, a shelf 4, a shelf 3, a shelf 2, a shelf 1, a sidetable 1, and a sofa 1.Your task is to: put some watch on safe. | | Reflexion’s trajectory: Reflexion initially plans to find a watch but begins its search in less likely locations instead of the dresser, the most probable spot. After inefficient exploration, it eventually locates the watch on the dresser. However, at the safe, it prematurely closes it without placing the watch inside, leading to a loop from which it cannot recover. | | Conclusion: The trajectory ends with the agent stuck in a repetitive cycle, unable to adapt its strategy or correct its mistake in handling the watch and the safe. | >action: think: To solve the task, I need to find and take a watch, then put it in the safe. observation: OK. >action: think: First I need to find a watch. A watch is more likely to appear in drawers (1-5), cabinet (1-4), dresser (1), and shelves (1-12). I can check one by one, starting with drawer 1. observation: OK. >action: go to drawer 1 observation: The drawer 1 is closed. >action: open drawer 1 observation: You open the drawer 1. The drawer 1 is open. In it, you see nothing. >action: go to drawer 2 observation: The drawer 2 is closed. >action: open drawer 2 observation: You open the drawer 2. The drawer 2 is open. In it, you see nothing. >action: go to drawer 3 observation: The drawer 3 is closed. >action: open drawer 3 observation: You open the drawer 3. The drawer 3 is open. In it, you see a creditcard 2. >action: go to drawer 4 observation: The drawer 4 is closed. >action: open drawer 4 observation: You open the drawer 4. The drawer 4 is open. In it, you see nothing. >action: go to drawer 5 observation: The drawer 5 is closed. >action: open drawer 5 observation: You open the drawer 5. The drawer 5 is open. In it, you see nothing. >action: go to cabinet 1 observation: The cabinet 1 is closed. >action: open cabinet 1 observation: You open the cabinet 1. The cabinet 1 is open. In it, you see a vase 1. >action: go to cabinet 2 observation: Nothing happens. >action: go to cabinet 3 observation: The cabinet 3 is closed. >action: open cabinet 3 observation: You open the cabinet 3. The cabinet 3 is open. In it, you see a tissuebox 2. >action: go to cabinet 4 observation: Nothing happens. >action: go to dresser 1 observation: On the dresser 1, you see a laptop 1, a remotecontrol 1, a tissuebox 1, a vase 3, a vase 2, and a watch 1. >action: think: Now I find a watch (1). Next, I need to take it. observation: OK. >action: take watch 1 from dresser 1 observation: You pick up the watch 1 from the dresser 1. >action: think: Now I take a watch (1). Next, I need to put it in/on safe 1. observation: OK. >action: go to safe 1 observation: The safe 1 is closed. >action: open safe 1 observation: You open the safe 1. The safe 1 is open. In it, you see a cellphone 3, and a cellphone 2. >action: think: The safe is not big enough for the watch and the cellphones. I should have put the watch in the safe, then closed the safe. I will try this new plan in the next trial. observation: OK. >action: close safe 1 observation: You close the safe 1. >action: put watch 1 in/on safe 1 observation: Nothing happens. >action: think: I need to put the watch in the safe, then close the safe. observation: OK. >action: put watch 1 in/on safe 1 observation: Nothing happens. >action: think: I need to put the watch in the safe, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. | MUSE’s trajectory: MUSE begins with an intelligent search strategy, prioritizing the dresser as the most likely location for the watch. After efficiently locating the watch on the dresser, MUSE proceeds directly to the safe, opens it, places the watch inside, and completes the task without error. | | --- | | Conclusion: MUSE successfully completes the task through efficient exploration and precise execution, demonstrating effective problem-solving capabilities. | >action: think: To solve the task, I need to find and take a watch, then put it in the safe. observation: OK. >action: think: First I need to find a watch. A watch is more likely to appear in dresser (1), cabinet (1-4), shelf (1-12), drawer (1-5). I can check one by one, starting with dresser 1. observation: OK. >action: go to dresser 1 observation: On the dresser 1, you see a laptop 1, a remotecontrol 1, a tissuebox 1, a vase 3, a vase 2, and a watch 1. >action: take watch 1 from dresser 1 observation: You pick up the watch 1 from the dresser 1. >action: think: Now I find a watch (1). Next, I need to put it in/on safe 1. observation: OK. >action: go to safe 1 observation: The safe 1 is closed. >action: open safe 1 observation: You open the safe 1. The safe 1 is open. In it, you see a cellphone 3, and a cellphone 2. >action: put watch 1 in/on safe 1 observation: You put the watch 1 in/on the safe 1. 8.5 Competence-aware action selection: A key advantage of MUSE over Reflexion >action: think: To solve the task, I need to find and take a bowl, then clean it with sinkbasin, then put it in cabinet. observation: OK. >action: think: First I need to find a bowl. A bowl is more likely to appear in cabinet (1-6), countertop (1-3), sinkbasin (1), stoveburner (1-4), fridge (1). I can check one by one, starting with cabinet 1. observation: OK. >action: go to cabinet 1 observation: On the cabinet 1, you see a bowl 1. >action: take bowl 1 from cabinet 1 observation: You pick up the bowl 1 from the cabinet 1. >action: think: Now I find a bowl (1). Next, I need to take it. observation: OK. >action: take bowl 1 from cabinet 1 observation: Nothing happens. >action: think: I need to go to countertop 1 observation: OK. >action: go to countertop 1 observation: On the countertop 1, you see a creditcard 1, a egg 3, and a glassbottle 1. >action: think: Now I find a bowl (3). Next, I need to go to countertop 3 and find it. observation: OK. >action: go to countertop 3 observation: On the countertop 3, you see a bread 3, a cellphone 1, a glassbottle 3, a houseplant 1, a knife 3, a knife 2, a potato 3, a spatula 2, a spatula 1, a spoon 1, a statue 2, and a tomato 1. >action: take bowl 3 from countertop 3 observation: Nothing happens. >action: think: I can’t find bowl 3 in countertop 3. Maybe it’s in drawer (1-3). observation: OK. >action: go to drawer 3 observation: The drawer 3 is closed. >action: open drawer 3 observation: You open the drawer 3. The drawer 3 is open. In it, you see a fork 2, and a saltshaker 2. >action: think: Now I find a bowl (3). Next, I need to take it. observation: OK. >action: take bowl 3 from drawer 3 observation: Nothing happens. >action: think: I can’t find a bowl (3) in drawer (3). Maybe it’s in countertop (1-3) or cabinet (1-6). observation: OK. >action: go to countertop 1 observation: On the countertop 1, you see a creditcard 1, a egg 3, and a glassbottle 1. >action: take bowl 3 from countertop 1 observation: Nothing happens. >action: think: I can’t find a bowl (3) in countertop (1-3). Maybe it’s in drawer (1-3). observation: OK. >action: go to drawer 2 observation: The drawer 2 is closed. >action: open drawer 2 observation: You open the drawer 2. The drawer 2 is open. In it, you see a fork 1. >action: take bowl 3 from drawer 2 observation: Nothing happens. >action: think: I can’t find a bowl (3) in drawer (2). Maybe it’s in countertop (1-3) or cabinet (1-6). observation: OK. >action: go to countertop 1 observation: On the countertop 1, you see a creditcard 1, a egg 3, and a glassbottle 1. >action: take bowl 3 from countertop 1 observation: Nothing happens. >action: think: I can’t find a bowl (3) in countertop (1-3). Maybe it’s in cabinet (1-6). observation: OK. >action: go to cabinet 1 observation: On the cabinet 1, you see nothing. >action: go to cabinet 2 observation: The cabinet 2 is closed. >action: open cabinet 2 observation: You open the cabinet 2. The cabinet 2 is open. In it, you see a vase 1. >action: go to cabinet 3 observation: On the cabinet 3, you see a plate 1. >action: take bowl 3 from cabinet 3 observation: Nothing happens. >action: think: I can’t find a bowl (3) in cabinet (3-1). Maybe it’s in countertop (1-3) or drawer (1-3). observation: OK. >action: go to countertop 1 observation: On the countertop 1, you see a creditcard 1, a egg 3, and a glassbottle 1. >action: take bowl 3 from countertop 1 observation: Nothing happens. >action: think: I can’t find a bowl (3) in countertop (1-3). Maybe it’s in drawer (1-3). observation: OK. >action: go to drawer 1 observation: The drawer 1 is closed. >action: open drawer 1 observation: You open the drawer 1. The drawer 1 is open. In it, you see a peppershaker 1, and a saltshaker 1. >action: think: Now I find a bowl (3). It’s a potato 1. observation: OK. >action: take potato 1 from drawer 1 observation: Nothing happens. >action: think: I can’t find a bowl (3) in drawer (1). Maybe it’s in countertop (1-3) or cabinet (1-13). observation: OK. >action: go to countertop 1 observation: On the countertop 1, you see a creditcard 1, a egg 3, and a glassbottle 1. >action: think: Now I find a bowl (3). It’s a potato 1. observation: OK. >action: think: Now I find a bowl (3). It’s a potato 1. Next, I need to take it. observation: OK. >action: take potato 1 from countertop 1 observation: Nothing happens. >action: think: I can’t take a bowl (3) from countertop (1-3). Maybe it’s in cabinet (1-13). observation: OK. >action: go to cabinet 1 observation: On the cabinet 1, you see nothing. >action: go to cabinet 2 observation: The cabinet 2 is open. In it, you see a vase 1. >action: take bowl 3 from cabinet 2 observation: Nothing happens. >action selected: think: To solve the task, I need to find and take a bowl, then clean it with sinkbasin, then put it in cabinet. observation: OK. >action selected: think: First I need to find a bowl. A bowl is more likely to appear in cabinet (1-6), countertop (1-3), sinkbasin (1), or garbagecan (1). I can check one by one, starting with cabinet 1. observation: OK. >action selected: go to cabinet 1 observation: On the cabinet 1, you see a bowl 1. >action selected: take bowl 1 from cabinet 1 observation: You pick up the bowl 1 from the cabinet 1. >action selected: think: Now I take a bowl (1). Next, I need to go to sinkbasin (1) and clean it. observation: OK. >action selected: go to sinkbasin 1 observation: On the sinkbasin 1, you see nothing. >action selected: clean bowl 1 with sinkbasin 1 observation: You clean the bowl 1 using the sinkbasin 1. >action selected: think: Now I clean a bowl (1). Next, I need to put it in/on cabinet 1. observation: OK. >action selected: go to cabinet 1 observation: On the cabinet 1, you see nothing. >action selected: put bowl 1 in/on cabinet 1 observation: You put the bowl 1 in/on the cabinet 1. Detailed trajectory of the MUSE agent, which highlights the benefits of competence awareness in strategic planning and action selection >action selected: think: To solve the task, I need to find and take a bowl, then clean it with sinkbasin, then put it in cabinet. observation: OK. >action selected: think: First I need to find a bowl. A bowl is more likely to appear in cabinet (1-6), countertop (1-3), sinkbasin (1), or garbagecan (1). I can check one by one, starting with cabinet 1. observation: OK. Imagine Trajectory 0: go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a glassbottle 1..> go to cabinet 2.The cabinet 2 is closed..> open cabinet 2.You open the cabinet 2 Imagine Trajectory 1: go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a saltshaker 1..> go to cabinet 2.The cabinet 2 is closed..> open cabinet 2.You open the cabinet 2. Imagine Trajectory 2: go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a bowl 2..> go to cabinet 2.The cabinet 2 is closed..> open cabinet 2.You open the cabinet 2. Imagine Trajectory Score 0 : 0.6981 Imagine Trajectory Score 1 : 0.6585 Imagine Trajectory Score 2 : 0.6284 >action selected: go to cabinet 1 observation: On the cabinet 1, you see a bowl 1. Imagine Trajectory 0: take bowl 1 from cabinet 1.You pick up the bowl 1 from the cabinet 1..> think: Now I find a bowl (1). Next, I need to take it..OK..> take bowl 1 from cabinet 1.You pick up the bowl 1 from the cabinet 1. Imagine Trajectory 1: take bowl 1 from cabinet 1.You pick up the bowl 1 from the cabinet 1..> think: Now I take a bowl (1). Next, I need to go to sinkbasin 1 and clean it..OK..> go to sinkbasin 1.On the sinkbasin 1, you see nothing. Imagine Trajectory 2: take bowl 1 from cabinet 1.You pick up the bowl 1 from the cabinet 1..> think: Now I take a bowl (1). Next, I need to go to sinkbasin 1 and clean it..OK..> go to sinkbasin 1.On the sinkbasin 1, you see nothing. Imagine Trajectory Score 0 : 0.6284 Imagine Trajectory Score 1 : 0.8689 Imagine Trajectory Score 2 : 0.6417 >action selected: take bowl 1 from cabinet 1 observation: You pick up the bowl 1 from the cabinet 1. Imagine Trajectory 0: think: Now I find a bowl (1). Next, I need to take it..OK..> take bowl 1 from cabinet 1.You pick up the bowl 1 from the cabinet 1..> think: Now I take a bowl (1). Next, I need to go to sinkbasin (1) and clean it.. Imagine Trajectory 1: think: Now I take a bowl (1). Next, I need to go to sinkbasin (1) and clean it..OK..> go to sinkbasin 1.On the sinkbasin 1, you see nothing..> clean bowl 1 with sinkbasin 1.You clean the bowl 1 using the sinkbasin Imagine Trajectory 2: think: Now I find a bowl (1). Next, I need to take it..OK..> take bowl 1 from cabinet 1.Nothing happens..> think: I need to wait a while before I can find a bowl (1)..OK..> go to countertop 2.On the countertop 2, you Imagine Trajectory Score 0 : 0.1265 Imagine Trajectory Score 1 : 0.7831 Imagine Trajectory Score 2 : 0.0126 >action selected: think: Now I take a bowl (1). Next, I need to go to sinkbasin (1) and clean it. observation: OK. Imagine Trajectory 0: go to sinkbasin 1.On the sinkbasin 1, you see nothing..> clean bowl 1 with sinkbasin 1.You clean the bowl 1 using the sinkbasin 1..> think: Now I clean the bowl 1 with sinkbasin 1. Next, I need to put it in/on Imagine Trajectory 1: go to sinkbasin 1.On the sinkbasin 1, you see nothing..> clean bowl 1 with sinkbasin 1.You clean the bowl 1 using the sinkbasin 1..> think: Now I clean a bowl (1). Next, I need to put it in/on cabinet 1..OK Imagine Trajectory 2: go to sinkbasin 1.On the sinkbasin 1, you see nothing..> clean bowl 1 with sinkbasin 1.You clean the bowl 1 using the sinkbasin 1..> think: Now I clean the bowl 1 with sinkbasin 1. Next, I need to put it in/on Imagine Trajectory Score 0 : 0.6139 Imagine Trajectory Score 1 : 0.8206 Imagine Trajectory Score 2 : 0.5012 >action selected: go to sinkbasin 1 observation: On the sinkbasin 1, you see nothing. Imagine Trajectory 0: clean bowl 1 with sinkbasin 1.You clean the bowl 1 using the sinkbasin 1..> think: Now I clean the bowl 1 with sinkbasin 1. Next, I need to put it in/on cabinet 1..OK..> go to cabinet 1.The cabinet 1 is closed. Imagine Trajectory 1: clean bowl 1 with sinkbasin 1.You clean the bowl 1 using the sinkbasin 1..> think: Now I clean a bowl (1). Next, I need to put it in/on cabinet (1-6)..OK..> go to cabinet 6.The cabinet 6 is closed..> open cabinet Imagine Trajectory 2: clean bowl 1 with sinkbasin 1.You clean the bowl 1 using the sinkbasin 1..> think: Now I clean a bowl (1). Next, I need to put it in/on cabinet 1..OK..> go to cabinet 1.The cabinet 1 is closed..> open cabinet 1 Imagine Trajectory Score 0 : 0.7847 Imagine Trajectory Score 1 : 0.5952 Imagine Trajectory Score 2 : 0.7003 >action selected: clean bowl 1 with sinkbasin 1 observation: You clean the bowl 1 using the sinkbasin 1. Imagine Trajectory 0: think: Now I clean a bowl (1). Next, I need to put it in/on cabinet 1..OK..> go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a dishsponge 1 Imagine Trajectory 1: think: Now I clean a bowl (1). Next, I need to put it in/on cabinet 1..OK..> go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a bowl 1..> Imagine Trajectory 2: think: Now I clean the bowl 1 with sinkbasin 1. Next, I need to put it in/on cabinet 1..OK..> go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a Imagine Trajectory Score 0 : 0.7106 Imagine Trajectory Score 1 : 0.8489 Imagine Trajectory Score 2 : 0.8093 >action selected: think: Now I clean a bowl (1). Next, I need to put it in/on cabinet 1. observation: OK. Imagine Trajectory 0: go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a bowl 1..> put bowl 1 in/on cabinet 1.You put the bowl 1 in/on the cabinet 1..You are in Imagine Trajectory 1: go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a bowl 1..> think: Now I find a bowl (1). Next, I need to take it..OK..> take bowl 1 from cabinet 1 Imagine Trajectory 2: go to cabinet 1.On the cabinet 1, you see a bowl 1..> put bowl 1 in/on cabinet 1.You put the bowl 1 in/on the cabinet 1..Make sure you put items “in/on” instead of “in”. Make sure you interact with and refer to items by name and number Imagine Trajectory Score 0 : 0.7794 Imagine Trajectory Score 1 : 0.7131 Imagine Trajectory Score 2 : 0.5139 >action selected: go to cabinet 1 observation: On the cabinet 1, you see nothing. Imagine Trajectory 0: put bowl 1 in/on cabinet 1.You put the bowl 1 in cabinet 1..> think: Now I put bowl 1 in cabinet 1..OK..> go to cabinet 1.The cabinet 1 is closed..> open cabinet 1 Imagine Trajectory 1: open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a bowl 1..> take bowl 1 from cabinet 1.You pick up the bowl 1 from the cabinet 1..> think: Now I find a bowl (1). Next, I need to clean it with sinkbasin 1 Imagine Trajectory 2: put bowl 1 in/on cabinet 1.Nothing happens..> think: I need to wait a few seconds before putting it in cabinet..OK..> think: Now I put bowl 1 in/on cabinet 1..OK..> go to cabinet 1.The cabinet 1 is closed..> open cabinet 1 Imagine Trajectory Score 0 : 0.8193 Imagine Trajectory Score 1 : 0.0106 Imagine Trajectory Score 2 : 0.6214 >action selected: put bowl 1 in/on cabinet 1 observation: You put the bowl 1 in/on the cabinet 1. References - Asai et al. (2023) Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511, 2023. - Brohan et al. (2023) Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. - Chatzipanteli et al. (2014) Athanasia Chatzipanteli, Vasilis Grammatikopoulos, and Athanasios Gregoriadis. Development and evaluation of metacognition in early childhood education. Early Child Development and Care, 184(8):1223–1232, 2014. - Chen et al. (2017) Patricia Chen, Omar Chavez, Desmond C Ong, and Brenda Gunderson. Strategic resource use for learning: A self-administered intervention that guides self-reflection on effective resource use enhances academic performance. Psychological Science, 28(6):774–785, 2017. - Chen et al. (2022) Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022. - Cohen (2012) Marisa Cohen. The importance of self-regulation for college student learning. College Student Journal, 46(4):892–902, 2012. - Collaboration et al. (2023) Open X-Embodiment Collaboration, Abby O’Neill, Abdul Rehman, Abhinav Gupta, et al. Open X-embodiment: Robotic learning datasets and RT-X models. arXiv preprint arXiv:2310.08864, 2023. - Corbiere et al. (2019) Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure prediction by learning model confidence. Advances in Neural Information Processing Systems, 32, 2019. - Dunlosky and Bjork (2013) John Dunlosky and Robert A Bjork. Handbook of Metamemory and Memory. Psychology Press, 2013. - Dunlosky and Metcalfe (2008) John Dunlosky and Janet Metcalfe. Metacognition. Sage Publications, 2008. - Dunning (2011) David Dunning. The Dunning–Kruger effect: On being ignorant of one’s own ignorance. Advances in Experimental Social Psychology, 44:247–296, 2011. - Flavell (1979) John Flavell. Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist, 34(10):906, 1979. - Fleming (2024) Stephen M Fleming. Metacognition and confidence: A review and synthesis. Annual Review of Psychology, 75(1):241–268, 2024. - Fleming and Lau (2014) Stephen M Fleming and Hakwan C Lau. How to measure metacognition. Frontiers in Human Neuroscience, 8, 443, 2014. - Gou et al. (2023) Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738, 2023. - Grossberg (1980) Stephen Grossberg. How does a brain build a cognitive code? Psychological Review, 87:1–51, 1980. - Ha and Schmidhuber (2018) David Ha and Jürgen Schmidhuber. World Models. arXiv preprint arXiv:1803.10122, 2018. - Hafner et al. (2023) Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through World Models. arXiv preprint arXiv:2301.04104, 2023. - Hansen et al. (2022) Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang, Vikash Kumar, and Aravind Rajeswaran. MoDem: Accelerating visual model-based reinforcement learning with demonstrations. arXiv preprint arXiv:2212.05698, 2022. - Hansen et al. (2023) Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828, 2023. - Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. - Hu et al. (2023) Xinyu Hu, Pengfei Tang, Simiao Zuo, Zihan Wang, Bowen Song, Qiang Lou, Jian Jiao, and Denis Charles. Evoke: Evoking critical thinking abilities in LLMs via reviewer-author prompt editing. arXiv preprint arXiv:2310.13855, 2023. - Huang et al. (2023) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023. - Isaacson and Fujita (2006) Randy Isaacson and Frank Fujita. Metacognitive knowledge monitoring and self-regulated learning. Journal of the Scholarship of Teaching and Learning, 6(1): 39–55, 2006. - Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023. - Ketz and Pilly (2022) Nicholas A Ketz and Praveen K Pilly. Concept-modulated model-based offline reinforcement learning for rapid generalization. arXiv preprint arXiv:2209.03207, 2022. - Kolouri et al. (2020) Soheil Kolouri, Nicholas Ketz, Andrea Soltoggio, and Praveen Pilly. Sliced Cramer synaptic consolidation for preserving deeply learned representations. International Conference on Learning Representations, 2020. - Koriat (1997) Asher Koriat. Monitoring one’s own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology: General, 126(4):349, 1997. - Koul et al. (2020) Anurag Koul, Varun V. Kumar, Alan Fern, and Somdeb Majumdar. Dream and search to control: Latent space planning for continuous control. arXiv preprint arXiv:2010.09832, 2020. - Kramarski and Mevarech (2003) Bracha Kramarski and Zemira Mevarech. Enhancing mathematical reasoning in the classroom: Effects of cooperative learning and metacognitive training. American Educational Research Journal, 40, 281–310, 2003. - Kruger and Dunning (1999) Justin Kruger and David Dunning. Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6):1121, 1999. - Kudithipudi et al. (2023) Dhireesha Kudithipudi, Mario Aguilar-Simon, Jonathan Babb, et al. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4(3):196–210, 2023. - Lu et al. (2025) Xiaping Lu, Carsten Murawski, Peter Bossaerts, and Shinsuke Suzuki. Estimating self-performance when making complex decisions. Scientific Reports, 15(1), 3203, 2025. - Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-Refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. - Mandi et al. (2023) Zhao Mandi, Homanga Bharadhwaj, Vincent Moens, Shuran Song, Aravind Rajeswaran, and Vikash Kumar. CACTI: A framework for scalable multi-task multi-scene visual imitation learning. arXiv preprint arXiv:2212.05711, 2023. - Maniscalco and Lau (2012) Brian Maniscalco and Hakwan Lau. A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings. Consciousness and Cognition, 21(1), 422–430, 2025. - Mehta et al. (2024) Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, et al. OpenELM: An efficient language model family with open-source training and inference framework. arXiv preprint arXiv:2404.14619, 2024. - Metcalfe et al. (1993) Janet Metcalfe, Bennett L Schwartz, and Scott G Joaquim. The cue-familiarity heuristic in metacognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19(4):851, 1993. - Micheli et al. (2023) Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient World Models. arXiv preprint arXiv:2209.00588, 2023. - Middlebrooks et al. (2012) Paul Middlebrooks and Marc Sommer. Neuronal correlates of metacognition in primate frontal cortex. Neuron, 75(3):517–530, 2012. - Miyamoto et al. (2017) Kentaro Miyamoto, Takahiro Osada, Rieko Setsuie, Masaki Takeda, Keita Tamura, Yusuke Adachi, and Yasushi Miyashita. Causal neural network of metamemory for retrospection in primates. Science, 355(6321):188–193, 2017. - Moerland et al. (2023) Thomas Moerland, Joost Broekens, Aske Plaat, and Catholijn Jonker. Model-based reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 16(1):1–118, 2023. - Mukherjee et al. (2023) Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of GPT-4. arXiv preprint arXiv:2306.02707, 2023. - Nelson and Narens (1990) Thomas O Nelson and Louis Narens. Metamemory: A theoretical framework and new findings. Academic Press, 1990. - Open-Orca (2024) Open-Orca. Mistral-7B-OpenOrca. https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca, 2024. - Paul et al. (2023) Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01904, 2023. - Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. - Robine et al. (2023) Jan Robine, Marc Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer-based World Models are happy with 100K interactions. arXiv preprint arXiv:2303.07109, 2023. - Rostami et al. (2019) Mohammad Rostami, Soheil Kolouri, and Praveen Pilly. Complementary learning for overcoming catastrophic forgetting using experience replay. arXiv preprint arXiv:1903.04566, 2019. - Schraw (1998) Gregory Schraw. Promoting general metacognitive awareness. Instructional Science, 26:113–125, 1998. - Schraw et al. (2006) Gregory Schraw, Kent Crippen, and Kendall Hartley. Promoting self-regulation in science education: Metacognition as part of a broader perspective on learning. Research in Science Education, 36, 111–139, 2006. - SentenceTransformers (2024) SentenceTransformers. all-MiniLM-L6-v2. https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2, 2024. - Shinn et al. (2023) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652, 2023. - Shridhar et al. (2020) Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020. - Silver et al. (2016) David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016. - Silver et al. (2017) David Silver, Julian Schrittwieser, Karen Simonyan, et al. Mastering the game of Go without human knowledge. Nature, 550(7676):354–359, 2017. - Silver et al. (2018) David Silver, Thomas Hubert, Julian Schrittwieser, et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419):1140–1144, 2018. - Sutton (1991) Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bulletin, 2(4):160–163, 1991. - Team et al. (2023) Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei Zhang. Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608, 2023. - Team et al. (2021) Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, Nat McAleese, Nathalie Bradley-Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes-Fitt, Valentin Dalibard, and Wojciech Marian Czarnecki. Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808, 2021. - Wang et al. (2021) Haoxiang Wang, Han Zhao, and Bo Li. Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation. International Conference on Machine Learning, 10991–11002, 2021. - Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. - Webb et al. (2023) Taylor Webb, Kiyofumi Miyoshi, Tsz Yan So, Sivananda Rajananda, and Hakwan Lau. Natural statistics support a rational account of confidence biases. Nature Communications, 14(1), 10991–11002, 2023. - Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. - Xie et al. (2023) Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. Decomposition enhances reasoning via self-evaluation guided decoding. arXiv preprint arXiv:2305.00633, 2023. - Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. - Yao et al. (2024) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. - Young and Fry (2008) Andria Young and Jane D Fry. Metacognitive awareness and academic achievement in college students. Journal of the Scholarship of Teaching and Learning, 8(2):1–10, 2008. - Yu et al. (2020) Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-World: A benchmark and evaluation for multi-task and meta reinforcement learning. Conference on Robot Learning, 1094–1100, 2020. - Zhou et al. (2022) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022.

Rendering Paper...