2411.13537

# Competence-Aware AI Agents with Metacognition for Unknown Situations and Environments (MUSE) **Authors**: Rodolfo Valiente, Praveen K. Pilly ## Abstract Metacognition, defined as the awareness and regulation of one’s cognitive processes, is central to human adaptability in unknown situations. In contrast, current autonomous agents often struggle in novel environments due to their limited capacity for adaptation. We hypothesize that metacognition is a critical missing ingredient in autonomous agents for the cognitive flexibility needed to tackle unfamiliar challenges. Given the broad scope of metacognitive abilities, we focus on competence awareness and strategy selection. To this end, we propose the Metacognition for Unknown Situations and Environments (MUSE) framework to integrate metacognitive processes of self-assessment and self-regulation into autonomous agents. We present two implementations of MUSE: one based on world modeling and another leveraging large language models (LLMs). Our system continually learns to assess its competence on a given task and uses this self-assessment to guide iterative cycles of strategy selection. MUSE agents demonstrate high competence awareness and significant improvements in self-regulation for solving novel, out-of-distribution tasks more effectively compared to model-based reinforcement learning and purely prompt-based LLM agent approaches. This work highlights the promise of approaches inspired by cognitive and neural systems in enabling autonomous agents to adapt to new environments while mitigating the heavy reliance on extensive training data and large models for the current models. keywords: Agentic AI , Large Language Model , Metacognition , Reinforcement Learning , Self-Assessment , Self-Regulation , World Model L[1]Q[l,#1] C[1]Q[c,#1] R[1]Q[r,#1] [label1]organization=Intelligent Systems Center, HRL Laboratories,addressline=3011 Malibu Canyon Road, city=Malibu, postcode=90265, state=CA, country=USA ## 1 Introduction The pursuit of fully autonomous agents in artificial intelligence (AI) remains a significant challenge. Current autonomous agents are primarily designed for operating environments, conditions, and uses that are known a priori. They rely on either scripted behaviors or pre-trained policies, both of which struggle to handle unknown situations effectively. As a result, when faced with novelty, they are prone to fail with suboptimal or even catastrophic outcomes (e.g., robotic manipulation errors in unstructured settings). This limitation severely restricts their deployment in safety-critical unknown environments, especially for long-duration missions or applications with little to no human oversight. Therefore, there is a practical urgency to reduce the failure rate, time to completion, and cost of autonomous missions by enabling resilient handling of unknowns during deployment. In mainstream AI, large-scale multi-task pre-training has emerged as the leading approach for enhancing adaptability in autonomous agents (Team et al., 2021). For example, Adaptive Agent (AdA) (Team et al., 2023) was trained with billions of frames and tasks to enable rapid adaptation to unseen, open-ended tasks. Similarly, the RT-2 (Brohan et al., 2023) and RT-X (Collaboration et al., 2023) models leverage large-scale robotic trajectory datasets to train agents capable of solving novel manipulation tasks and generalizing to new robots and environments. However, internet-scale pre-training to be able to handle every potential change and combination of changes in real-world applications is impractical, prohibitively resource-intensive, and expensive. Even with significantly limited data, AI agents must intelligently interpolate and extrapolate beyond their pre-trained scenarios while continually learning and adapting to novelty (Kudithipudi et al., 2023). In other words, when faced with novel scenarios, pre-trained knowledge must be continually updated to strike a dynamic balance between stability and plasticity (Grossberg, 1980). Similar to humans, AI agents can leverage pre-deployment training to acquire a wide range of skills across diverse, known scenarios. Importantly, they can also be equipped with the ability to engage in online learning for continual improvement when encountering novel situations. For example, a teenager attending driving school follows a structured curriculum that teaches foundational vehicle control skills, which are then progressively built upon to master more complex tasks, such as merging onto highways or navigating construction zones. This learning process is cumulative in the sense that mastery of foundational skills simplifies the acquisition of more advanced ones. Moreover, the key principles of driving are consolidated in the student’s brain, protecting them from catastrophic forgetting. Ultimately, the end of driving school marks the beginning of a lifelong learning process, where the student draws on prior experiences to navigate novel driving challenges independently without an instructor. Metacognition, defined as the awareness and regulation of one’s cognitive processes, is a key human trait that has been extensively studied in cognitive psychology (Flavell, 1979; Nelson and Narens, 1990; Metcalfe et al., 1993; Koriat, 1997; Dunlosky and Metcalfe, 2008). This metacognitive flexibility enables humans to learn online efficiently and solve problems iteratively, especially in relation to new tasks. For instance, students can leverage metacognition to more accurately assess their knowledge and adjust their study habits accordingly (Cohen, 2012; Chen et al., 2017). The use of metacognition among college students has indeed been shown to correlate significantly with various measures of academic success (Young and Fry, 2008; Isaacson and Fujita, 2006). Students who perform poorly often overestimate their abilities, leading to under-preparation for exams. This is a common issue in education, where overconfident students allocate insufficient time to study, believing they have already mastered the material. Conversely, students who underestimate their knowledge may spend excessive time reviewing topics they already understand, hindering their progress. Even children as young as three years old can become effective learners and thinkers from activities designed to develop metacognitive skills (Chatzipanteli et al., 2014). It is generally agreed that self-assessment of competence, which can range from over-confidence in novices to slight under-confidence in experts (Kruger and Dunning, 1999; Dunning, 2011), is a capability that is teachable and improves naturally as one becomes more skilled or knowledgeable (Kramarski and Mevarech, 2003; Schraw et al., 2006). Neuroscience further reveals that the subregions of the prefrontal cortex responsible for metacognitive judgments are distinct from those involved in cognitive functions like visual memory recognition, such that the metacognitive function can be selectively deactivated without affecting the cognitive function (e.g., Middlebrooks et al. (2012); Miyamoto et al. (2017)). While metacognition spans a wide range of capabilities and higher-order cognitive processes (e.g., Feeling of Knowing, Judgment of Learning, Source Monitoring), it can be conceptualized as an internal perception-action loop of self-assessment and self-regulation (Nelson and Narens, 1990; Dunlosky and Bjork, 2013). Self-assessment in this context refers to an individual’s ability to assess their competence regarding a specific task. Self-regulation refers to the ability to strategically select and control one’s actions based on this self-assessment. In this article, we introduce the Metacognition for Unknown Situations and Environments (MUSE) framework to computationally instantiate and train the metacognitive capabilities of self-assessment and self-regulation for AI agents, so that they can also achieve more efficient learning and improved generalization to unknown scenarios (Figure 1). Specifically, the self-assessment mechanism is designed to predict the agent’s likelihood of successfully completing a given task for proposed action plans, with a learnable internal model informed by past experiences. And the self-regulation mechanism leverages this self-assessment to enable iterative cycles of competence-aware strategy selection for problem-solving. We present two implementations of the MUSE agent: one based on world modeling and the other utilizing large language models (LLMs). Our experiments in two distinct environments (namely, Meta-World and ALFWorld) demonstrate that MUSE agents achieve substantial improvements in handling novel scenarios compared to baseline/non-metacognitive approaches. Further, we show that metacognition makes a particularly big impact on smaller or less-capable LLM agents, making them amenable to edge deployment as well as less reliant on big data for online adaptation. <details> <summary>figures/metacognitive_cycle.png Details</summary>  ### Visual Description \n ## Diagram: Cognition and Metacognition Model ### Overview The image presents a diagram illustrating the relationship between Cognition and Metacognition. It depicts a flow of information from the "Environment" into the "Cognition" block, which then feeds into the "Metacognition" block. The diagram uses boxes and arrows to represent processes and their interactions. ### Components/Axes The diagram consists of the following labeled components: * **Environment:** A rectangular box on the left, representing the external world. * **Cognition:** A larger, rounded-rectangle box containing "Perception" and "Action" as internal components. * **Perception:** A rectangular box within "Cognition", with "Memory" inside. * **Action:** A rectangular box within "Cognition", with "Reasoning" inside. * **Metacognition:** A rounded-rectangle box containing "Self-Assessment" and "Self-Regulation" as internal components. * **Self-Assessment:** A rectangular box within "Metacognition". * **Self-Regulation:** A rectangular box within "Metacognition". * Arrows: Indicate the flow of information between components. A circular arrow connects "Self-Assessment" and "Self-Regulation". The diagram also includes labels for the two main blocks: "COGNITION" and "METACOGNITION". ### Detailed Analysis or Content Details The diagram shows the following information flow: 1. The "Environment" provides input to both "Perception" and "Action" within "Cognition". 2. "Perception" includes "Memory" as an internal component. 3. "Action" includes "Reasoning" as an internal component. 4. Both "Perception" and "Action" feed into "Self-Assessment" within "Metacognition". 5. "Self-Assessment" then feeds into "Self-Regulation". 6. "Self-Regulation" feeds back into "Self-Assessment", creating a feedback loop. The colors used are: * "Environment": White * "Cognition" box outline: Black * "Perception" and "Action" boxes: White * "Memory" and "Reasoning" boxes: Light Blue * "Metacognition" box outline: Blue * "Self-Assessment": Light Green * "Self-Regulation": Light Orange * Arrows: Black * Labels: Black ### Key Observations The diagram highlights the iterative nature of metacognition, as demonstrated by the circular arrow between "Self-Assessment" and "Self-Regulation". This suggests a continuous process of monitoring and adjusting one's cognitive processes. The diagram emphasizes that metacognition builds upon the foundation of cognition. ### Interpretation This diagram illustrates a model of how cognition and metacognition interact. Cognition represents the basic mental processes of perceiving and acting in the world, while metacognition represents the ability to think about one's own thinking. The diagram suggests that metacognition allows individuals to monitor, evaluate, and regulate their cognitive processes, leading to more effective learning and problem-solving. The feedback loop between self-assessment and self-regulation is crucial for adaptive behavior. The diagram is a simplified representation of complex cognitive processes, but it effectively conveys the core relationship between cognition and metacognition. It implies that effective learning and performance require not only cognitive abilities but also the ability to reflect on and control those abilities. </details> Figure 1: The metacognitive cycle of self-assessment and self-regulation operates on the traditional perception-action loop of existing AI agents to boost their ability for iterative problem-solving in unknown situations and environments. In contrast to current reinforcement learning (RL) approaches (e.g., Silver et al. (2016, 2017, 2018); Ha and Schmidhuber (2018); Hafner et al. (2023)), which focus on maximizing expected cumulative reward, our MUSE framework prioritizes competence as the primary evaluation metric to enhance the agent’s adaptability to unknown situations. We hypothesize that focusing solely on maximizing return may cause an agent to become stuck in unfamiliar situations, particularly under sparse reward regimes. By contrast, maximizing competence continually for strategy selection encourages more effective exploration in such novel environments. Through metacognition, the agent can evaluate its capabilities and attempt new strategies within its perceived competence, enabling safer and more effective online adaptation. In other words, by maximizing competence, the agent not only improves its ability to tackle immediate challenges more effectively but also fosters iterative problem-solving in complex environments. The contributions of our article are threefold: - We introduce the Metacognition for Unknown Situations and Environments (MUSE) framework, which integrates metacognitive functions of self-assessment and self-regulation into sequential decision-making agents. - We propose an implementation of a competence awareness model that continually learns to assess the agent’s competence on a given task and serves as an evaluation function for planning (Self-Assessment). - We propose an implementation of a policy modulation model that leverages self-assessment to iteratively drive strategy selection by identifying action plans that maximize the likelihood of task success (Self-Regulation). ## 2 Related Work ### 2.1 Self-Assessment Self-assessment broadly refers to an agent’s ability to monitor its internal states and assess its capabilities and performance in relation to tasks and goals. It enables humans to reflect and make adjustments for improving outcomes (Flavell, 1979; Nelson and Narens, 1990; Schraw, 1998). World Models: These are generative models of environmental dynamics (Ha and Schmidhuber, 2018; Robine et al., 2023; Micheli et al., 2023; Hansen et al., 2022, 2023), which can be used to estimate the expected cumulative reward of agents. These models often employ sequence-based architectures, such as recurrent neural networks, to predict the next state, reward, and terminal signals. Decoder-based World Models (Hafner et al., 2023; Robine et al., 2023; Micheli et al., 2023) can additionally generate the input state corresponding to the predicted next state. In contrast, decoder-free World Models focus on predicting the outcomes of actions in the latent space, bypassing the need to decode input states (Hansen et al., 2022, 2023). World Models act as proxy simulators during training, enabling agents to learn more efficiently by reducing the reliance on real-environment interactions (e.g., Ha and Schmidhuber (2018); Koul et al. (2020); Hafner et al. (2023)). In this work, we extend the capabilities of decoder-based World Models by training them to predict not only environmental dynamics but also the agent’s competence to solve a given task. LLM Critics: These models are designed to evaluate the performance of LLMs. The LLM itself can be prompted to provide feedback on its outputs (Madaan et al., 2023), intermediate reasoning steps (Paul et al., 2023), or even the prompt itself (Hu et al., 2023). Some approaches enhance the correctness and quality of LLM outputs by using stochastic beam search guided by self-evaluation (Xie et al., 2023). Recognizing that LLMs currently are limited in identifying their own errors, reasoning missteps, or biases (Huang et al., 2023), researchers have augmented LLM critics with external tools, such as search engines and calculators, to improve reliability (Gou et al., 2023). Retrieval Augmented Generation (RAG) approaches have also been proposed to strengthen self-evaluation by providing relevant external knowledge bases (Asai et al., 2023). In contrast, MUSE does not rely solely on pre-trained knowledge or external tools. Instead, it continually learns and grounds itself to evaluate its competence on given tasks and uses this kind of self-assessment to modulate policy decisions. Confidence Networks: Recent work in cognitive computational neuroscience has developed quantitative frameworks for assessing metacognitive judgments related to self-assessment across a range of domains, task difficulties, and time scales with and without external feedback (Fleming, 2024; Lu et al., 2025). Measures that have been proposed and utilized to assess metacognition include the statistical correlation between self-reported confidence ratings and actual performance across trials as well as the more reliable meta- d’ metric, which measures the ability of self-assessment to discriminate between high-performance (correct) and low-performance (incorrect) trials without being affected by response bias in metacognitive judgments (Maniscalco and Lau, 2012; Fleming and Lau, 2014). Consistent with these measures of metacognition, there is prior work in machine learning aimed at self-assessment of deep neural networks that perform perception tasks. For classification, Corbiere et al. (2019) trained a separate neural network (called ConfidNet) that operates on high-level features extracted by the classifier to predict the true class probability (TCP), which is the softmax probability of the correct class irrespective of whether it was chosen or not. Further, Webb et al. (2023) trained confidence networks for a variety of perception tasks to instead predict the probability of the decision being correct, i.e., a value of 1 if correct and 0 otherwise. They also trained a RL agent that chooses among perception labels as well as an opt-out action that earns a low-risk low-reward in situations when the agent is least certain about its decision. These self-assessment metrics themselves are task-agnostic (Fleming, 2024) but require a training scheme that is adapted to the specifics of the task. In this regard, one of our contributions is to implement a global self-assessment metric for artificial agents that predicts the probability of task success in each episode over time. ### 2.2 Self-Regulation Self-regulation is the process by which an agent dynamically adjusts its behavior based on self-assessment to achieve specific goals (Flavell, 1979; Nelson and Narens, 1990). This ability is essential for humans to exhibit robust decision-making and function autonomously in unfamiliar environments. Model-based Reinforcement Learning (MBRL) Agents: These agents utilize World Models to simulate future scenarios, enabling them to train with minimal real interactions with the environment (Moerland et al., 2023). Dyna (Sutton, 1991) is a foundational architecture that integrates learning and planning within a single agent. Its core idea is to use the agent’s real experiences to update not only its policy but also its internal model of the environment, and thereby leverage the updated internal model to further improve the policy offline using simulated experiences. Among recent leading ones, AlphaGo (Silver et al., 2016), AlphaZero (Silver et al., 2017), and MuZero (Silver et al., 2018) are all MBRL systems that employ internal simulations/models and Monte Carlo Tree Search (MCTS) to explore potential action paths, based on variations of the Upper Confidence Bound (UCB) score, and evaluate them for action selection. In contrast to MBRL agents, which prioritize maximizing expected cumulative reward, the self-regulation mechanism of MUSE leverages competence as the primary evaluation metric to enhance the agent’s ability to navigate and adapt effectively in unknown situations. Prompt-based LLM Agents: The capabilities of LLMs extend beyond language generation, making them increasingly popular for reasoning tasks to potentially deal with novelty. Chain-of-Thought (CoT) prompting (Wei et al., 2022), for example, decomposes a complex problem into intermediate steps to arrive at a final answer. However, CoT reasoning struggles to yield accurate results due to error propagation as the number of steps increases (Chen et al., 2022). Techniques such as self-consistency (Wang et al., 2022), least-to-most prompting (Zhou et al., 2022), and Tree-of-Thought (ToT) prompting (Yao et al., 2024) aim to mitigate this issue by improving sampling strategies and leveraging search algorithms. Nevertheless, these methods rely solely on the LLM’s pre-trained knowledge, which limits their ability to adapt to external feedback. Beyond reasoning tasks, LLMs are also being increasingly applied to operate in an agentic loop of perception and action, which unlocks the benefits of large-scale pre-training for multi-step interactive tasks without relying on RL. ReAct (Yao et al., 2022) was among the first purely prompt-based LLM agents that integrated both reasoning and action planning to perform text-based problem-solving. However, ReAct is inefficient and limited in its ability to transfer performance improvements to subsequent episodes. To address this issue, Reflexion (Shinn et al., 2023) built on ReAct by adding an LLM critic that reflects on failures and provides persistent verbal feedback to the agent for improved performance in subsequent episodes. The performance gains from these LLM agent methods are solely dependent on enhanced prompt-based in-context learning, which limits their ability for longer-term true learning from new varied experiences. While MUSE also makes use of prompting for both reasoning and planning, it can also continually learn from its experiences to facilitate more effective problem-solving in unknown situations and environments. ## 3 Decoder-based World Model implementation In this section, we describe our implementation of the MUSE framework using a decoder-based World Model to equip MBRL agents with metacognitive abilities of self-assessment and self-regulation. ### 3.1 Methods #### 3.1.1 Self-Assessment through World Modeling We leverage the decoder-based World Model from Dreamer-v3 (Hafner et al., 2023) to implement self-assessment for agents, but we note that our approach can be extended to decoder-free World Models as well. Dreamer-v3 uses a Recurrent State-Space Model (RSSM) to model the environment dynamics. See Equations 1 - 3 for the formulation from Hafner et al. (2023). The RSSM maps the input state $x_{t}$ and recurrent state $h_{t}$ to a latent embedding $z_{t}$ and uses the concatenation of $h_{t}$ and $z_{t}$ , called the RSSM state, as input to parameterize various distributions over the reward $\hat{r}_{t}$ , terminal signal $\hat{d}_{t}$ , and decoded state $\hat{x}_{t}$ . $$ \displaystyle\begin{aligned} \begin{aligned} \raisebox{8.39578pt}{\hbox to0.0pt{\hss\vbox to0.0pt{\hbox{$\text{RSSM}\hskip 4.30554pt\begin{cases}\hphantom{A}\\ \hphantom{A}\\ \hphantom{A}\end{cases}\hskip-10.33327pt$}\vss}}}&\text{Sequence model:}\hskip 35.00005pt&&h_{t}&\ =&\ f_{\phi}(h_{t-1},z_{t-1},a_{t-1})\\ &\text{Encoder:}\hskip 35.00005pt&&z_{t}&\ \sim&\ q_{\phi}(z_{t}\;|\;h_{t},x_{t})\\ &\text{Dynamics predictor:}\hskip 35.00005pt&&\hat{z}_{t}&\ \sim&\ p_{\phi}(\hat{z}_{t}\;|\;h_{t})\\ &\text{Reward predictor:}\hskip 35.00005pt&&\hat{r}_{t}&\ \sim&\ p_{\phi}(\hat{r}_{t}\;|\;h_{t},z_{t})\\ &\text{Terminal signal predictor:}\hskip 35.00005pt&&\hat{d}_{t}&\ \sim&\ p_{\phi}(\hat{d}_{t}\;|\;h_{t},z_{t})\\ &\text{Decoder:}\hskip 35.00005pt&&\hat{x}_{t}&\ \sim&\ p_{\phi}(\hat{x}_{t}\;|\;h_{t},z_{t})\end{aligned}\end{aligned} \tag{1} $$ ifnextchar gobble Following Hafner et al. (2023), given a sequential batch of inputs $x_{1:T}$ , actions $a_{1:T}$ , rewards $r_{1:T}$ , and terminal signals $d_{1:T}$ , the World Model parameters $\phi$ are optimized to minimize the prediction loss $\mathcal{L}_{\mathrm{pred}}$ , the dynamics loss $\mathcal{L}_{\mathrm{dyn}}$ , and the representation loss $\mathcal{L}_{\mathrm{rep}}$ . The prediction loss $\mathcal{L}_{\mathrm{pred}}$ is the joint negative log-likelihood of the multiple probabilistic predictors (Equation 3). Real-valued quantities like the reward and decoded state are trained with a symlog squared loss, whereas the terminal signal, which is a binary-valued quantity, is trained with logistic regression. The dynamics loss $\mathcal{L}_{\mathrm{dyn}}$ and the representation loss $\mathcal{L}_{\mathrm{rep}}$ are designed to effectively learn the dynamics of the latent embeddings for generating realistic rollout trajectories. $$ \displaystyle\begin{aligned} \mathcal{L}(\phi)\doteq\operatorname{E}_{q_{\phi}}\Bigg[\displaystyle\sum_{t=1}^{T}(\mathcal{L}_{\mathrm{pred}}(\phi)+\mathcal{L}_{\mathrm{dyn}}(\phi)+0.1\mathcal{L}_{\mathrm{rep}}(\phi))\Bigg]\end{aligned} \tag{2} $$ ifnextchar gobble $$ \displaystyle\begin{aligned} \mathcal{L}_{pred}\doteq-\ln p_{\phi}(r_{t}\;|\;h_{t},z_{t})-\ln p_{\phi}(d_{t}\;|\;h_{t},z_{t})-\ln p_{\phi}(x_{t}\;|\;h_{t},z_{t})\end{aligned} \tag{3} $$ ifnextchar gobble We augment Dreamer-v3’s World Model with an additional head for predicting task success. In this implementation, the Self-Assessment Model is an MLP with $N$ outputs that map the RSSM state to the probability of task success within the $N$ quantiles of the maximum episode duration. Specifically, the MLP outputs parameterize $N=5$ Bernoulli distributions $\{\psi_{1},\cdots\psi_{N}\}$ for the five quantiles. A self-assessment prediction involves sampling from each of these distributions, $\hat{c}^{i}_{t}\sim\psi_{i}(h_{t},z_{t})$ , and combining these samples into a prediction vector. For example, a prediction of success in the first quantile would yield the vector $[1,1,1,1,1]$ , whereas a prediction of failure even by the last quantile would yield the vector $[0,0,0,0,0]$ . This process is visualized in Figure 2. Note that each component of the self-assessment head is trained separately using binary cross-entropy loss, and the individual losses are then added to the total prediction loss $\mathcal{L}_{\mathrm{pred}}$ (Equation 4). $$ \displaystyle\begin{aligned} \mathcal{L}_{\mathrm{SA}}\doteq-\sum_{i=1}^{N}\ln\psi_{i}(h_{t},z_{t}),\hskip 35.00005pt\mathcal{L}_{pred}\leftarrow\mathcal{L}_{pred}+\mathcal{L}_{\mathrm{SA}}\end{aligned} \tag{4} $$ ifnextchar gobble <details> <summary>figures/rssm_sa_combined.png Details</summary>  ### Visual Description \n ## Diagram: Variational Autoencoder (VAE) with Attention Mechanism ### Overview The image depicts a diagram of a Variational Autoencoder (VAE) architecture with an attention mechanism applied across three sequential time steps. The diagram illustrates the encoding, latent space representation, decoding, and attention processes involved in the model. It appears to be modeling a sequence of images of a robotic arm. ### Components/Axes The diagram consists of three identical blocks representing sequential time steps (1, 2, and 3). Each block contains the following components: * **Input Images:** `x1`, `x2`, `x3` (original images) and `x̂1`, `x̂2`, `x̂3` (reconstructed images). * **Encoder (enc):** Transforms the input image into a latent representation. * **Latent Space (z1, z2, z3):** A grid-like representation of the latent variables. * **Decoder (dec):** Reconstructs the image from the latent representation. * **Hidden State (h1, h2, h3):** Represents the hidden state of the recurrent network. * **Context Vector (ĉ1, ĉ2, ĉ3):** Represents the context vector. * **Reward (r̂1, r̂2, r̂3):** Represents the reward. * **Decision (d̂1, d̂2, d̂3):** Represents the decision. * **Attention Weights (a1, a2):** Connect the hidden states and context vectors across time steps. * **State Transition (s1, s2, s3):** Connects the hidden states across time steps. ### Detailed Analysis or Content Details The diagram shows a sequential VAE model with three time steps. **Time Step 1:** * Input image: `x1` * Reconstructed image: `x̂1` * Latent variable: `z1` (represented as a 8x8 grid with varying shades of blue) * Hidden state: `h1` * Context vector: `ĉ1` * Reward: `r̂1` * Decision: `d̂1` **Time Step 2:** * Input image: `x2` * Reconstructed image: `x̂2` * Latent variable: `z2` (represented as a 8x8 grid with varying shades of blue) * Hidden state: `h2` * Context vector: `ĉ2` * Reward: `r̂2` * Decision: `d̂2` * Attention weight: `a1` (connecting `h1` to `ĉ2`) **Time Step 3:** * Input image: `x3` * Reconstructed image: `x̂3` * Latent variable: `z3` (represented as a 8x8 grid with varying shades of blue) * Hidden state: `h3` * Context vector: `ĉ3` * Reward: `r̂3` * Decision: `d̂3` * Attention weight: `a2` (connecting `h2` to `ĉ3`) The attention mechanism is implemented by connecting the hidden states of previous time steps to the context vectors of subsequent time steps using attention weights `a1` and `a2`. The state transition `s1` connects `h1` to `h2`, and `s2` connects `h2` to `h3`. The images show a robotic arm manipulating a block. The reconstructed images `x̂1`, `x̂2`, and `x̂3` appear visually similar to the original images `x1`, `x2`, and `x3`, suggesting the decoder is effectively reconstructing the input. ### Key Observations * The model utilizes a recurrent structure to process sequential data. * The attention mechanism allows the model to focus on relevant information from previous time steps. * The latent space provides a compressed representation of the input data. * The model predicts a reward and decision at each time step. * The latent space `z` appears to be a 8x8 grid, with varying intensity of blue. ### Interpretation This diagram illustrates a sophisticated VAE architecture designed for sequential data modeling, specifically for tasks involving robotic manipulation. The attention mechanism is crucial for enabling the model to maintain context and make informed decisions over time. The VAE component allows the model to learn a compressed, latent representation of the observed states, which can be used for planning and control. The prediction of reward and decision suggests the model is being trained to optimize a specific task, such as successfully manipulating the block. The sequential nature of the model, combined with the attention mechanism, suggests it is capable of handling complex, temporally-dependent tasks. The fact that the reconstructed images are similar to the original images indicates that the model is learning a meaningful representation of the input data. The model is likely being used for reinforcement learning or imitation learning, where the reward signal guides the learning process. The attention mechanism allows the model to focus on the most relevant parts of the past sequence when making decisions. </details> Figure 2: Schematic of the implementation of self-assessment in the context of the Dreamer-v3 World Model (adapted from Hafner et al. (2023)). The input state $x$ to the RSSM is encoded into latent embedding $z$ . The model recurrently predicts self-assessment $\hat{c}$ , reward $\hat{r}$ , and terminal signal $\hat{d}$ , while also decoding the input state $\hat{x}$ . #### 3.1.2 Self-Regulation Even with pre-deployment training on multiple tasks, including parametric variations, MBRL agents exhibit limited generalization to novel tasks that require either new, orchestrated combinations of those skills or entirely new skills (Ketz and Pilly, 2022). While Dreamer-v3 can handle novel parametric variations for a known task, it struggles to make progress when faced with an unknown reward function that differs semantically from those of the training tasks. Central to our MUSE framework is the self-regulation algorithm, which performs competence-aware actions to solve novel tasks. Specifically, the decision-making process selects actions that maximize the likelihood of task success. Self-assessed competence can be used to guide planning in three primary ways: 1. Simulate multiple future scenarios (rollout trajectories) based on the current state and potential actions, then greedily select the path that maximizes the self-assessment criterion 1. Perform MCTS over actions using the self-assessment criterion in place of variations of the UCB score 1. Optimize the RSSM state to directly maximize the self-assessment criterion to effectively self-regulate the policy For this implementation of MUSE, we used the third option. This self-regulation method, which is detailed in Algorithm 1, leverages the differentiability of the World Model with the self-assessment head. MUSE performs a World Model rollout where the agent’s actions are regulated to increase the likelihood of reaching a success state. Specifically, MUSE directly optimizes the RSSM state, $s$ ( $\doteq\{h,z\}$ ), input to the agent for selecting actions that maximize the self-assessment criterion. The intuition is that when the Self-Assessment Model predicts failure in a novel environment, it is no longer useful to just rely on the default policy. Instead, we seek competence-aware actions that guide the agent to a success state by augmenting the RSSM state in a direction that increases the probability of task success. At each time step $t$ , MUSE performs a short World Model rollout of horizon $H$ that optimizes over the RSSM state $s$ as follows: $$ \displaystyle\begin{aligned} s\leftarrow s+\beta\nabla_{s}\left(\sum_{i=1}^{N}\psi_{i}(s)\right).\end{aligned} \tag{5} $$ ifnextchar gobble It then applies the self-regulated action $a_{t}\sim\pi(a_{t}|s)$ , observes the resulting recurrent state $h_{t+1}$ , and begins a new iteration. Algorithm 1 Self-regulation leverages the World Model and its constituent Self-Assessment Model to select competence-aware actions 1: $h_{t}$ 2: $H$ = 10 3: $\beta=0.02$ 4: $z_{t}\sim p_{\phi}(z_{t}|h_{t})$ 5: $h\leftarrow h_{t}$ 6: $z\leftarrow z_{t}$ 7: $s\doteq\{h,z\}$ 8: for $step\leftarrow 1$ to $H$ do 9: $a\sim\pi(a|s)$ 10: $h\leftarrow f_{\phi}(s,a)$ 11: $\displaystyle s\leftarrow s+\beta\nabla_{s}\left(\sum_{i=1}^{N}\psi_{i}(s)\right)$ 12: end for 13: $a_{t}\sim\pi(a_{t}|s)$ 14: $h_{t+1}\leftarrow f_{\phi}(s,a_{t})$ 15: return $z_{t}$ , $a_{t}$ , $h_{t+1}$ ### 3.2 Meta-World Experiments For the World-Model-based implementation, we evaluated our approach within the Meta-World robotic manipulation simulator (Yu et al., 2020) and compared it against Dreamer-v3 (Hafner et al., 2023) as the MBRL baseline. Meta-World provides a suitable testbed for learning a shared perceptual and dynamics model across multiple tasks using a 6 degrees-of-freedom (DOF) robotic arm. To ensure consistency, we used Dreamer-v3’s network architectures, hyperparameters, and learning procedures across all shared components between the two agents (e.g., an imagination horizon of 15 time steps for actor and critic learning). Both methods were implemented in the same PyTorch codebase, with the self-assessment and self-regulation modules omitted for Dreamer-v3. The agents received a $64\times 64$ RGB observation alongside a 40-dimensional proprioceptive state. Additionally, we included a task embedding that was represented as a single integer-valued channel appended to the visual state. MUSE leveraged the built-in success signal returned by the Meta-World environment to train its Self-Assessment Model. For the experiments, we employed a two-stage protocol comprising pre-deployment training on known tasks followed by deployment adaptation to unknown tasks. Each episode in these experiments had a maximum time limit of 500 steps. For pre-deployment training, we utilized Meta-World’s MT10 suite of 10 different manipulation tasks (Figure 3). In particular, we adopted a multi-task learning paradigm (Mandi et al., 2023), encompassing all 10 training tasks over 2M total environment steps. This paradigm was chosen over meta-RL approaches to reduce computational costs and training time (Wang et al., 2021). By default, object and goal positions were randomly sampled to enable domain randomization. For deployment adaptation, we evaluated the agents on a set of 10 novel tasks from Meta-World’s MT50 suite with distinct reward functions, which were semantically different from those in the pre-deployment training set (Figure 4). The agents were exposed to one novel task at a time, starting with pre-deployment trained weights, for 20 adaptation episodes per task. They were assessed for performance on novel tasks during these adaptation episodes. The task embedding channel was set to zero for novel tasks. During training as well as adaptation, both agents continually updated their World Models with real data and their actor and critic neural networks with imagined data. The replay buffer from pre-deployment training was retained for deployment adaptation to prevent catastrophic forgetting of previously trained performance. #### 3.2.1 Metrics Self-Assessment We used metacognitive accuracy and the Area under the Type 2 Receiver Operating Characteristic Curve (AUROC2) to evaluate how well MUSE predicts its success on novel tasks (Fleming and Lau, 2014), which in turn indicates how effectively the self-assessment signal can support online adaptation. MUSE was evaluated for each novel task separately over 20 adaptation episodes. During these episodes, we collected MUSE’s step-wise self-assessment predictions for evaluation. The predictions at all time steps across tasks were compared with the true labels to compute the metacognitive accuracy and AUROC2 metrics. The Type 2 ROC curve for MUSE was computed by treating the sum of the self-assessment components $\left(\sum_{i=1}^{N}\psi_{i}(s)\right)$ as the signal. Note that, as Dreamer-v3 lacks a self-assessment module to predict episode outcomes, we cannot make a meaningful comparison between Dreamer-v3 and MUSE for self-assessment. Self-Regulation We evaluated how well the agents generalize to unknown situations using two metrics: the percentage of novel tasks solved and the average time to task completion. Each agent was assessed for each novel task separately over the 20 adaptation episodes. The percentage of episodes where the agent completed the task within the maximum time limit was averaged across all novel tasks to calculate the success rate. Similarly, the number of time steps required to achieve success in each episode was averaged across all episodes for each novel task to compute the respective time-to-completion metrics. <details> <summary>figures/train_mt10_tasks.png Details</summary>  ### Visual Description \n ## Image Series: Robotic Manipulation Sequence ### Overview The image presents a series of ten snapshots depicting a robotic arm interacting with various objects in a simulated environment. The sequence appears to demonstrate a manipulation task, potentially involving picking, placing, and assembling objects. The environment includes a tiled floor, and various geometric shapes (cubes, rectangular prisms, cylinders) of different colors. A small green sphere is consistently present in the scene, potentially serving as a target or reference point. ### Components/Axes There are no explicit axes or legends in the image. The components are: * **Robotic Arm:** A red, multi-jointed robotic arm is the primary actor in all frames. * **Objects:** Various geometric shapes (cubes, rectangular prisms, cylinders) in colors yellow, green, red, and gray. * **Green Sphere:** A small, consistently present green sphere. * **Environment:** A tiled floor and simple geometric structures. ### Detailed Analysis or Content Details The sequence can be broken down as follows: 1. **Frame 1:** The robotic arm is positioned above a yellow rectangular prism. The gripper is open. 2. **Frame 2:** The robotic arm is grasping a gray rectangular prism. 3. **Frame 3:** The robotic arm is positioned above a green rectangular prism. The gripper is open. 4. **Frame 4:** The robotic arm is grasping the green rectangular prism. The green sphere is visible. 5. **Frame 5:** The robotic arm is holding a red triangular prism. 6. **Frame 6:** The robotic arm is positioned above a structure composed of blue and white rectangular prisms. 7. **Frame 7:** The robotic arm is grasping a red cylinder. 8. **Frame 8:** The robotic arm is holding the red cylinder, which has been released and is now on the floor. 9. **Frame 9:** The robotic arm is positioned above a structure composed of gray and white rectangular prisms. 10. **Frame 10:** The robotic arm is grasping a white rectangular prism. ### Key Observations * The robotic arm appears to be performing a sequence of pick-and-place operations. * The green sphere remains consistently in the scene, potentially acting as a visual guide or target. * The objects being manipulated vary in shape and color. * The environment is simple and uncluttered, likely designed to isolate the robotic manipulation task. * The sequence suggests a possible assembly or sorting task. ### Interpretation The image series demonstrates a robotic manipulation task in a simulated environment. The sequence suggests the robot is capable of identifying, grasping, and moving objects of different shapes and colors. The consistent presence of the green sphere hints at a potential guidance or targeting system. The overall purpose of the task is not explicitly stated, but it could be related to assembly, sorting, or object rearrangement. The simulation environment allows for controlled experimentation and evaluation of the robot's manipulation capabilities without the risks and costs associated with real-world deployment. The sequence is likely a demonstration of a robotic system's ability to perform complex manipulation tasks, potentially for use in manufacturing, logistics, or other automated applications. The lack of quantitative data (e.g., precise coordinates, forces applied) limits a deeper analysis, but the visual sequence provides a clear illustration of the robot's actions. </details> Figure 3: Meta-World pre-deployment training set [button-press, door-open, drawer-close, drawer-open, peg-insert-side, pick-place, push, reach, window-close, window-open], which comprises the 10 tasks from the MT10 suite. <details> <summary>figures/test_evaluation_tasks.png Details</summary>  ### Visual Description \n ## Image Series: Robotic Arm Manipulation ### Overview The image consists of a 3x3 grid of screenshots depicting a robotic arm interacting with various objects in a simulated environment. Each screenshot shows a different scenario, likely demonstrating the arm's capabilities in grasping, moving, and manipulating objects of different shapes, sizes, and materials. There are no axes, legends, or numerical data present. This is a visual demonstration, not a data presentation. ### Components/Axes There are no axes or legends. The primary components are: * **Robotic Arm:** A red, multi-jointed robotic arm is present in all images. * **Objects:** A variety of objects are present, including: * Yellow block * Blue cube/frame * Red cylinder * Brown/tan irregular shape * White sphere * Red brick-like structure * Black sphere * Blue sphere * Gray block * Black object with a handle * **Surface:** A wooden-textured surface is visible in all images, serving as the workspace. ### Detailed Analysis / Content Details Here's a description of each image, moving left-to-right, top-to-bottom: 1. **Top-Left:** The robotic arm is positioned above a yellow block on a brown base. The arm appears to be in a grasping position. 2. **Top-Center:** The robotic arm is positioned near a blue cube-like frame and a black sphere. The arm is extended. 3. **Top-Right:** The robotic arm is positioned above a red cylinder and a small green sphere. The arm is in a grasping position. 4. **Middle-Left:** The robotic arm is positioned above a red brick-like structure and a black sphere. The arm is in a grasping position. 5. **Middle-Center:** The robotic arm is positioned above a flat, gray surface with a small indentation. The arm is extended. 6. **Middle-Right:** The robotic arm is positioned above a gray block and a blue sphere. The arm is in a grasping position. 7. **Bottom-Left:** The robotic arm is positioned above a red brick-like structure and a black sphere. The arm is in a grasping position. 8. **Bottom-Center:** The robotic arm is positioned above a gray block. The arm is extended. 9. **Bottom-Right:** The robotic arm is positioned above a black object with a handle. The arm is in a grasping position. ### Key Observations * The robotic arm consistently appears to be either grasping or positioned to grasp an object. * The objects vary significantly in shape, size, and color. * The environment appears to be a simulated workspace. * The arm's configuration changes to accommodate the different objects and their positions. ### Interpretation This image series likely demonstrates the versatility of the robotic arm in a simulated environment. The different scenarios showcase the arm's ability to handle a variety of objects, suggesting its potential for use in tasks such as pick-and-place operations, assembly, or material handling. The consistent grasping/positioning suggests a focus on demonstrating the arm's manipulation capabilities. The lack of any quantitative data suggests this is a qualitative demonstration of functionality rather than a performance evaluation. The simulation environment allows for testing and visualization without the constraints of a real-world setup. The variety of objects suggests the arm is designed to be adaptable to different tasks and environments. </details> Figure 4: Meta-World evaluation set [button-press-topdown-wall, soccer, push-wall, push-block, coffee-button, plate-slide, peg-unplug-side, lever-pull, handle-press, door-unlock], which comprises 10 novel tasks from the MT50 suite with distinct reward functions, which differ semantically from those in the pre-deployment training set. Table 1: Self-assessment performance of the MUSE agent on novel tasks in the Meta-World environment. | Method | Metric | Value | | --- | --- | --- | | MUSE | Metacognitive Accuracy | 92% | | MUSE | AUROC2 | 0.95 | #### 3.2.2 Results The MUSE agent achieved a metacognitive accuracy of 92% and an AUROC2 of 0.95 on novel tasks over the adaptation episodes, demonstrating that the Self-Assessment Model is highly predictive of competence for novel tasks (Table 1). Further, MUSE successfully solved 7 of the 10 novel tasks (70%) by leveraging competence-aware actions. The 7 solved tasks required different time steps per episode depending on the relative difficulty and complexity of the task (Figure 5 a). For instance, plate-slide required nearly the maximum time budget, while coffee-button was solved quickly. But in sharp contrast, Dreamer-v3 failed to solve any of the novel tasks within the allotted adaptation episodes (Figure 5 b). <details> <summary>figures/adaptation_steps.png Details</summary>  ### Visual Description ## Bar Charts: MUSE and Dreamer-v3 Performance ### Overview The image presents two bar charts, labeled (a) MUSE Performance and (b) Dreamer-v3 Performance. Both charts compare the average time to adapt to various tasks, categorized as either "Solved" or "Unsolved". The x-axis represents different tasks, and the y-axis represents the average time to adapt, measured in unspecified units (likely seconds). ### Components/Axes * **X-axis (Both Charts):** "Task" with the following categories: button-press, soccer, push-wall, push-block, coffee-button, plate-slide, peg-unplug-side, level-pull, handle-press, door-unlock. * **Y-axis (Both Charts):** "Average time to adapt" ranging from -50 to 400. * **Legend (Both Charts):** * Blue: "Solved" * Red: "Unsolved" * **Chart a):** Title: "MUSE Performance" * **Chart b):** Title: "Dreamer-v3 Performance" ### Detailed Analysis or Content Details **Chart a) MUSE Performance:** * **button-press:** Solved ≈ 100, Unsolved ≈ 0 * **soccer:** Solved ≈ 60, Unsolved ≈ 0 * **push-wall:** Solved ≈ 100, Unsolved ≈ 0 * **push-block:** Solved ≈ 230, Unsolved ≈ 20 * **coffee-button:** Solved ≈ 380, Unsolved ≈ 20 * **plate-slide:** Solved ≈ 350, Unsolved ≈ 10 * **peg-unplug-side:** Solved ≈ 20, Unsolved ≈ 20 * **level-pull:** Solved ≈ 300, Unsolved ≈ 0 * **handle-press:** Solved ≈ 280, Unsolved ≈ 0 * **door-unlock:** Solved ≈ 300, Unsolved ≈ 0 **Chart b) Dreamer-v3 Performance:** * **button-press:** Solved ≈ 0, Unsolved ≈ 0 * **soccer:** Solved ≈ 0, Unsolved ≈ 0 * **push-wall:** Solved ≈ 0, Unsolved ≈ 0 * **push-block:** Solved ≈ 0, Unsolved ≈ 0 * **coffee-button:** Solved ≈ 0, Unsolved ≈ 0 * **plate-slide:** Solved ≈ 0, Unsolved ≈ 0 * **peg-unplug-side:** Solved ≈ 0, Unsolved ≈ 0 * **level-pull:** Solved ≈ 0, Unsolved ≈ 0 * **handle-press:** Solved ≈ 0, Unsolved ≈ 0 * **door-unlock:** Solved ≈ 0, Unsolved ≈ 0 ### Key Observations * **MUSE Performance:** Shows significant variation in adaptation time depending on the task. The highest adaptation times are observed for "coffee-button" and "plate-slide" when solved. The "peg-unplug-side" task has a relatively low adaptation time for both solved and unsolved states. * **Dreamer-v3 Performance:** Shows consistently low adaptation times for all tasks, with both "Solved" and "Unsolved" bars clustered around zero. * The "Unsolved" bars are generally very small in both charts, indicating that most tasks were solved. ### Interpretation The data suggests a substantial difference in performance between MUSE and Dreamer-v3. MUSE exhibits a wide range of adaptation times, indicating that some tasks are significantly more challenging than others. Dreamer-v3, on the other hand, demonstrates near-instantaneous adaptation across all tasks, suggesting a much higher level of generalization or efficiency. The consistent low values for "Unsolved" in both charts suggest that the agents are generally capable of solving these tasks. However, the large differences in adaptation times for MUSE highlight its task-specific learning challenges. Dreamer-v3's performance implies a more robust and adaptable learning strategy. The tasks themselves appear to vary in complexity. "Coffee-button" and "plate-slide" are particularly difficult for MUSE, requiring significantly longer adaptation times. This could be due to the fine motor skills or planning required for these tasks. The fact that Dreamer-v3 performs equally well on all tasks suggests it may be less sensitive to these complexities. </details> Figure 5: Average number of time steps per episode required to solve each of the 10 novel tasks in the Meta-World environment. MUSE (a) successfully solved 7 out of 10 tasks, whereas Dreamer-v3 (b) failed to solve any of them. Note that to facilitate illustration, unsolved tasks are assigned a nominal value of -25 and depicted by a red bar. ### 3.3 Discussion The pre-deployment training tasks were selected to encompass a broad range of skills, so the agents could develop a better understanding of various manipulation strategies. Following this training, the agents were exposed to out-of-distribution tasks that were selected to evaluate their ability to generalize learned skills to novel challenges. This two-stage approach of training on familiar tasks and then adapting to novel ones was used to rigorously assess the effectiveness of MUSE and Dreamer-v3 in unknown situations. Overall, the World-Model-based experiments showed that MUSE significantly outperforms Dreamer-v3 in adapting to novel tasks by effectively leveraging competence awareness for strategy selection. ## 4 LLM-based implementation In this section, we describe our implementation of the MUSE framework to equip LLM agents with metacognitive abilities of self-assessment and self-regulation. ### 4.1 Methods #### 4.1.1 ReAct Yao et al. (2022) was among the first to introduce an LLM agent that interacts with its environment to accomplish tasks by being prompted to reason and act (Figure 6). At time step $t$ , the ReAct agent ( $M_{a}$ ) perceives an observation $o_{t}\in\mathcal{O}$ , executes an action $a_{t}\in\mathcal{A}$ based on the policy $\pi(a_{t}|I,c_{t-1},o_{t})$ , and receives a reward $r_{t}$ . Here, $I$ is the natural language description of the task, and $c_{t}=\{o_{1},a_{1},r_{1}...,o_{t},a_{t},r_{t}\}$ is the running context of the trajectory within the episode. The context is reset at the end of each episode, which occurs when either the task is solved within the time budget or the episode terminates due to running out of time. The ReAct agent leverages language-based CoT reasoning (Wei et al., 2022) for improved sequential action selection for multi-step tasks. Note that actions $a_{t}$ stored in $c_{t}$ include both standard actions and CoT reasoning steps. Two-shot domain-specific trajectory examples with step-by-step reasoning are included in the prompt for CoT reasoning. See Supplementary Section 8.1 for illustrative examples. <details> <summary>figures/ReAct.png Details</summary>  ### Visual Description ## Diagram: Reinforcement Learning with Language Models ### Overview This diagram illustrates a reinforcement learning framework incorporating language models. It depicts the interaction between an "Actor" (a language model) and an "Environment," mediated by textual observations and rewards. The diagram highlights the use of long-term memory and a language descriptor to enhance the learning process. ### Components/Axes The diagram consists of the following components: * **Actor:** A green rectangular box labeled "Actor" and "M_a". * **Environment:** A grey rectangular box labeled "Environment". * **Long-term memory:** A light-green rectangular box labeled "Long-term memory". * **Language descriptor:** A light-blue rectangular box labeled "Language descriptor". * **Low-level policies:** A dark-grey rectangular box labeled "Low-level policies". * **Context (c_t-1):** Text label indicating the context input to the Actor. * **Task description (I):** Text label indicating the task input to the Actor. * **Text observation (O_t):** Text label indicating the observation from the Environment to the Actor. * **High-level action (a_t):** Text label indicating the action from the Actor to the Environment. * **Text observation, Reward (O_t+1, r_t):** Text label indicating the observation and reward from the Environment to the Long-term memory. * **Observation, Reward:** Text label indicating the observation and reward from the Environment to the Language descriptor. * **Few-shot examples:** Text label indicating the input to the Long-term memory. * **Low-level action:** Text label indicating the action from the Low-level policies to the Environment. Arrows indicate the flow of information between these components. ### Detailed Analysis / Content Details The diagram shows a cyclical process: 1. The **Actor** receives **Context (c_t-1)** and **Task description (I)** as inputs. 2. The **Actor** generates a **High-level action (a_t)**. 3. The **High-level action (a_t)** is sent to the **Environment**. 4. The **Environment** produces an **Observation** and **Reward**. 5. The **Observation** and **Reward** are sent to both the **Language descriptor** and the **Long-term memory**. 6. The **Long-term memory** receives **Few-shot examples** as input. 7. The **Language descriptor** sends a **Text observation (O_t)** to the **Actor**. 8. The **Environment** sends a **Text observation, Reward (O_t+1, r_t)** to the **Long-term memory**. 9. The **Actor** receives **Text observation (O_t)**. 10. The **Environment** receives a **Low-level action** from the **Low-level policies**. The diagram does not contain numerical data or specific values. It is a conceptual representation of a system. ### Key Observations The diagram emphasizes the role of language in both the observation and action spaces of the reinforcement learning agent. The inclusion of "Long-term memory" and "Language descriptor" suggests an attempt to address challenges related to long-horizon tasks and complex environments. The separation of "High-level" and "Low-level" actions indicates a hierarchical reinforcement learning approach. ### Interpretation This diagram represents a sophisticated reinforcement learning architecture that leverages the power of language models. The "Actor" acts as a policy network, generating high-level actions based on the current context and task description. The "Environment" simulates the real world, providing observations and rewards. The "Language descriptor" likely translates the environment's state into a textual representation that the "Actor" can understand. The "Long-term memory" allows the agent to store and retrieve past experiences, improving its ability to generalize and learn from limited data. The "Low-level policies" likely translate the high-level actions into concrete control signals for the environment. The diagram suggests a system designed to tackle complex tasks that require reasoning, planning, and adaptation. The use of language as a communication channel between the agent and the environment is a key feature, enabling the agent to leverage prior knowledge and learn from natural language instructions. The hierarchical structure allows for efficient exploration and exploitation of the environment. </details> Figure 6: Our illustration of the ReAct architecture (Yao et al., 2022). #### 4.1.2 Reflexion Reflexion (Shinn et al., 2023) enhances in-context prompting for LLM agents by generating internal feedback in language, referred to as “reflection,” to transfer lessons learned across episodes for a given task (Figure 7). At the end of each episode $e$ , Reflexion prompts an LLM to reason and generate verbal feedback $rx^{e}$ about the agent’s performance by analyzing the entire trajectory $c_{T}$ from the episode start ( $t=1$ ) to finish ( $t=T$ ) and the episode outcome (success or failure). This Reflection LLM ( $M_{rx}$ ) also receives the reflections and outcomes from previous episodes ( $1:e-1$ ). The Reflexion agent ( $M_{a}$ ) follows the policy $\pi(a_{t}|I,c_{t-1},o_{t},rx^{\{1:e-1\}})$ , which incorporates cumulative reflections from earlier episodes. In contrast to ReAct, which relies solely on short-term memory comprising the agent’s running trajectory within the current episode, Reflexion leverages both short- and long-term memory for improved strategic adaptation to novel situations. This iterative internal feedback mechanism allows the agent to refine its strategies progressively by integrating lessons learned from failures in previous episodes. Reflexion addresses the credit assignment problem through implicit reasoning about specific actions within the trajectory that led to failures, proposing alternative strategies for future episodes. Through this process, the agent develops an enhanced understanding of effective plans to iteratively improve its adaptability to novel tasks. While the original Reflexion study (Shinn et al., 2023) utilized a single LLM for both action generation and reflection, our implementation uses distinct LLMs that are fine-tuned for each respective task. Additionally, we extend the reflection process to include successful outcomes, which enables the model to reinforce effective strategies alongside learning from failures. <details> <summary>figures/Reflex.png Details</summary>  ### Visual Description \n ## Diagram: Reflexion Architecture ### Overview The image depicts a diagram of a Reflexion architecture, a reinforcement learning framework that incorporates reflection as a key component. The diagram illustrates the flow of information between different modules, including an Actor, Environment, Language descriptor, and Long-term memory. It shows how the system uses text observations, rewards, and reflections to learn and improve its performance over multiple episodes. ### Components/Axes The diagram consists of several key components: * **Actor:** Represented by a green rectangle labeled "Actor" with internal elements *I* (Task description) and *M_a*. Outputs "a_t" (High-level action). * **Environment:** Represented by a light-blue rectangle labeled "Environment". Outputs "Observation, Reward". * **Language descriptor:** Represented by a yellow rectangle labeled "Language descriptor". * **Long-term memory:** Represented by a gray triangle labeled "Long-term memory". * **Reflection:** Represented by a dark-blue rectangle labeled "Reflection" with internal elements *M_rx*. Outputs "rx^e" (Episode outcome, Reflection). * **Context:** Represented by "c_t-1". * **Text observation:** Represented by "o_t". * **Text observation, Reward:** Represented by "o_t+1, r_t". * **Reflections:** Represented by "r_x^{1-e}". * **Episode outcome:** Represented by "c_T". * **Low-level policies:** Outputs "Low-level action". * **Few-shot examples:** Input to Long-term memory. * **+ Previous episode outcomes:** Input to Actor. Arrows indicate the direction of information flow between these components. Dotted arrows represent less direct or delayed information flow. ### Detailed Analysis or Content Details The diagram illustrates a cyclical process. 1. **Initialization:** The system starts with "Few-shot examples" stored in "Long-term memory". 2. **Context & Observation:** The "Long-term memory" provides "Context" (c_t-1) to the "Actor". The "Environment" provides a "Text observation" (o_t). 3. **Action Selection:** The "Actor" uses the "Context" and "Task description" (*I*) to generate a "High-level action" (a_t). 4. **Environment Interaction:** The "High-level action" is translated into a "Low-level action" by "Low-level policies" and sent to the "Environment". The "Environment" returns an "Observation" and "Reward". 5. **Language Description:** The "Language descriptor" processes the "Observation" and "Reward". 6. **Memory Update:** The "Text observation" (o_t+1) and "Reward" (r_t) are stored in the "Long-term memory". 7. **Reflection:** The "Episode outcome" (c_T) and "Reflection" (M_rx) are generated. 8. **Context Update:** The "Reflections" (r_x^{1-e}) from previous episodes are fed back into the "Actor" along with the "Task description" (*I*). 9. **Iteration:** The process repeats for subsequent episodes. ### Key Observations The diagram highlights the importance of reflection in the learning process. The "Reflection" module receives information about the "Episode outcome" and generates a "Reflection" that is used to update the "Actor's" context for future episodes. The "Long-term memory" serves as a repository of past experiences, providing context for the "Actor". The cyclical nature of the diagram emphasizes the iterative learning process. ### Interpretation This diagram represents a sophisticated reinforcement learning architecture that aims to improve performance through self-reflection. The inclusion of a "Language descriptor" suggests that the system can process and understand natural language, potentially allowing it to learn from textual feedback or generate textual explanations of its actions. The "Long-term memory" enables the system to retain and utilize past experiences, preventing it from repeating mistakes and allowing it to build upon previous successes. The "Reflection" module is crucial for identifying areas for improvement and adapting the system's behavior accordingly. The diagram suggests a system capable of not just *doing* but also *understanding* and *learning from* its actions, a key step towards more intelligent and adaptable AI agents. The use of mathematical notation (e.g., r_x^{1-e}) indicates a formal, model-based approach to reinforcement learning. The diagram is a high-level overview and does not provide details about the specific algorithms or implementations used within each module. </details> Figure 7: Our illustration of the Reflexion architecture (Shinn et al., 2023), which leverages verbal RL to enable agents to learn from past episodes. #### 4.1.3 MUSE The MUSE framework, illustrated in Figure 8, builds on mechanisms from ReAct and Reflexion by incorporating additional modules for self-assessment and self-regulation: - World Model ( $M_{w}$ ): An LLM that can be prompted to predict the next observation, reward, and terminal signal given the current observation and action. This LLM works in conjunction with the Actor ( $M_{a}$ ) to generate potential future states and actions (rollout trajectories), enabling look-ahead planning. In text-based domains, such as the experiments presented here, $M_{w}$ can use the same LLM as $M_{a}$ . In fact, the same LLM can be directly prompted to generate diverse trajectories without requiring explicit interaction between the two. - Self-Assessment Model ( $M_{sa}$ ): A language-conditioned neural network that evaluates and scores trajectories generated by the World Model/Actor to assess their alignment and effectiveness for the agent’s goals. Specifically, it predicts task competence, or the probability of task success, for each trajectory. - Self-Regulation ( $M_{sr}$ ): A module that decides a competence-aware course of action based on one of the options outlined in Subsection 3.1.2. For this implementation of MUSE, $M_{sr}$ chooses the first action $a_{t}$ from the rollout trajectory that is most likely to achieve task success, as determined by competence evaluations from $M_{sa}$ . <details> <summary>figures/MUSE.png Details</summary>  ### Visual Description \n ## Diagram: Reflexion Architecture ### Overview The image depicts a diagram of a Reflexion architecture, a framework for agents that iteratively improve their performance through self-reflection. The diagram illustrates the flow of information between various components, including a World Model, Actor, Self-Assessment, Self-Regulation, Environment, and Long-term Memory. The diagram is segmented into a main loop with inputs and outputs, and a reflection loop. ### Components/Axes The diagram consists of the following components: * **World Model (Blue):** Labeled as *M_W*. Receives context *c_t-1*. * **Actor (Red):** Labeled as *M_α*. Receives context *c_t-1* and Task description *I*. Outputs trajectory. * **Self-Assessment (Yellow):** Labeled as *M_sa*. Receives trajectory from the Actor. * **Self-Regulation (Green):** Labeled as *M_sr*. Receives output from Self-Assessment. Outputs context *c_T*. * **Environment (Gray):** Receives initial plan *p_e* and low-level action. Outputs Observation and Reward. * **Long-term Memory (Light Gray):** Receives Text observation and Reward *r_t*. Outputs few-shot examples. * **Language Descriptor (Light Gray):** Receives Text observation and Reward *r_t*. * **Reflection (Purple):** Labeled as *M_rx*. Receives Episode outcome. Outputs Reflections *rx^{1e-13}*. * **Inputs:** Few-shot examples, Task description *I*, Text observation *o_t*, Reward *r_t*, Previous episode outcomes. * **Outputs:** High-level action *a_t*, Episode outcome, Reflection *rx^e*. The diagram also includes labels for data flow: * Context: *c_t-1*, *c_T* * Initial plan: *p_e* * Reflections: *rx^{1e-13}*, *rx^e* * Trajectory * Observation, Reward ### Detailed Analysis The diagram illustrates a cyclical process. 1. **Context Input:** The process begins with context *c_t-1* being fed into both the World Model and the Actor. 2. **Task & Actor:** The Actor also receives a Task description *I*. The Actor generates a trajectory. 3. **Self-Assessment & Regulation:** The trajectory is passed to the Self-Assessment module, which then passes its output to the Self-Regulation module. 4. **Context Update:** The Self-Regulation module outputs updated context *c_T*. 5. **Environment Interaction:** The Actor generates an initial plan *p_e* and a low-level action that is sent to the Environment. 6. **Feedback Loop:** The Environment returns an Observation and Reward. 7. **Memory Update:** The Observation and Reward are fed into the Language Descriptor and Long-term Memory. 8. **Reflection Loop:** The Episode outcome is sent to the Reflection module, which outputs Reflections *rx^e*. These reflections are then used to update the initial context *c_t-1* via *rx^{1e-13}*. 9. **Previous Episode Outcomes:** The diagram also shows that previous episode outcomes feed into the Self-Regulation module. The diagram uses arrows to indicate the direction of information flow. The dashed arrows represent feedback loops or external inputs. The colors of the components are consistent throughout the diagram. ### Key Observations The diagram highlights the importance of self-reflection in the Reflexion architecture. The reflection loop allows the agent to learn from its past experiences and improve its performance over time. The use of separate modules for World Modeling, Acting, Self-Assessment, and Self-Regulation suggests a modular and potentially scalable design. The inclusion of Long-term Memory and Few-shot examples indicates the agent can leverage prior knowledge and adapt to new tasks. ### Interpretation The diagram represents a sophisticated agent architecture designed for iterative improvement through self-reflection. The core idea is that by analyzing its own actions and outcomes, the agent can refine its internal models and strategies, leading to better performance. The separation of concerns into distinct modules (World Model, Actor, etc.) promotes modularity and allows for independent development and optimization of each component. The feedback loops, particularly the reflection loop, are crucial for enabling continuous learning and adaptation. The diagram suggests a system capable of not just performing tasks, but also understanding *why* it succeeds or fails, and using that understanding to improve its future behavior. The use of a language descriptor and long-term memory suggests the agent can leverage natural language processing and store/retrieve relevant experiences for future use. The notation *rx^{1e-13}* likely represents a very small weighting factor applied to the reflections, indicating that past reflections have a subtle but persistent influence on the agent's current context. </details> Figure 8: Illustration of the MUSE architecture for LLM agents, which implements the metacognitive cycle to iteratively solve unknown tasks. World Model/Actor During deployment, the World Model/Actor generates several potential future state-action sequences with a horizon $H$ at each time step $t$ . The diversity of these rollout trajectories, denoted by $\tau_{t}=\{a_{t},r_{t},o_{t+1},\dots,a_{t+H-1},r_{t+H-1},o_{t+H}\}$ , is controlled by the temperature setting of the LLM. These trajectories represent hypothetical paths extending from the current observation $o_{t}$ and context $c_{t-1}$ , guided by the task description $I$ and task-specific reflections $rx^{\{1:e-1\}}$ stored in memory. For this implementation, we explicitly did not specify a horizon $H$ ; instead, the LLM was allowed to generate trajectories up to the maximum length permitted by its context window. The temperature of the LLM was set to 0.5, and five rollout trajectories were generated at each time step. Self-Assessment The Self-Assessment Model ( $M_{sa}$ ) utilizes a transformer encoder $\mathcal{M}$ (SentenceTransformers, 2024) and an MLP $g_{\eta}$ to predict the probability of task success (Equation 6) for rollout trajectories generated by the World Model/Actor before their actual execution in the environment. Specifically, $M_{sa}$ evaluates the alignment and effectiveness of potential trajectories for the task at hand. The algorithm for training $M_{sa}$ and using it for evaluation is detailed in Algorithm 2. $$ \displaystyle\begin{aligned} y_{\text{pred}}=M_{sa}(I,P^{e},\tau)\end{aligned} \tag{6} $$ ifnextchar gobble Here, $\tau$ represents a trajectory and $P^{e}$ denotes the initial language-based plan generated by the World Model/Actor LLM at the start of episode $e$ based on task description $I$ . See Supplementary Material for several illustrative examples of the initial plan $P^{e}$ . The output layer of $M_{sa}$ employs a sigmoid activation function to yield a probability of task success. A threshold of 0.5 is used to convert these probabilities into binary outcomes (success or failure). During pre-deployment training and deployment adaptation, $M_{sa}$ is trained to minimize the binary cross-entropy loss between the predicted task success ( $y_{\text{pred}}$ ) and the actual binary outcome ( $y$ ) as follows: $$ \displaystyle\begin{aligned} \mathcal{L}(y,y_{\text{pred}})\doteq-\left[y\log(y_{\text{pred}})+(1-y)\log(1-y_{\text{pred}})\right]\end{aligned} \tag{7} $$ ifnextchar gobble To increase the number of training samples and mitigate overfitting, the training trajectories are segmented into non-overlapping chunks, each containing four action-observation pairs. For example, the initial chunk within an episode includes pairs ( $a_{1}$ , $o_{1}$ ) through ( $a_{4}$ , $o_{4}$ ), while the next spans ( $a_{5}$ , $o_{5}$ ) through ( $a_{8}$ , $o_{8}$ ), and so on. Algorithm 2 Training and evaluation of Self-Assessment Model $M_{sa}$ 1: Input: Dataset $\mathcal{D}=\{(I,P^{e},\tau,y)\}$ , transformer encoder $\mathcal{M}$ , MLP $g_{\eta}$ 2: Output: Trained evaluator $g_{\eta}$ 3: function TrainEvaluator ( $\mathcal{D}$ , $\mathcal{M}$ , $g_{\eta}$ ) 4: for each $(I,P^{e},\tau,y)\in\mathcal{D}$ do 5: $Z\leftarrow\mathcal{M}(I)$ $\triangleright$ Generate task embedding 6: $S\leftarrow\mathcal{M}(\tau,P^{e})$ $\triangleright$ Generate trajectory and plan embedding 7: $X\leftarrow\text{concat}(S,Z)$ 8: $y_{\text{pred}}\leftarrow g_{\eta}(X)$ $\triangleright$ Predict task success probability 9: Update $g_{\eta}$ parameters to minimize $\mathcal{L}(y,y_{\text{pred}})$ 10: end for 11: return Trained MLP $g_{\eta}$ 12: end function 13: function EvaluateTrajectory ( $I$ , $P^{e}$ , $\tau$ , $\mathcal{M}$ , $g_{\eta}$ ) 14: $Z\leftarrow\mathcal{M}(I)$ 15: $S\leftarrow\mathcal{M}(\tau,P^{e})$ 16: $X\leftarrow\text{concat}(S,Z)$ 17: $y_{\text{pred}}\leftarrow g_{\eta}(X)$ 18: return $y_{\text{pred}}$ 19: end function MUSE Framework The pre-deployment training and deployment adaptation procedures for the MUSE agent are detailed in Algorithms 3 and 4, respectively. For pre-deployment training, we first train the Reflexion agent (Shinn et al., 2023) on each of the training tasks separately to collect data. The constituent models of the MUSE agent are then trained with multi-task supervised learning to benefit from the diversity of experiences. Specifically, the Reflection LLM ( $M_{rx}$ ) in MUSE is trained using Direct Preference Optimization (DPO) with a preference dataset (Rafailov et al., 2024) that is created by comparing reflections of success and failure across the training tasks. A success (positive) reflection occurs when the agent fails in episode $e_{i}$ but succeeds in episode $e_{i+1}$ following the reflection. Conversely, a failure (negative) reflection happens when the agent succeeds in episode $e_{i}$ but fails in episode $e_{i+1}$ despite the reflection. DPO directly optimizes $M_{rx}$ to generate generalizable reflections that maximize the likelihood of satisfying these preferences (i.e., increase the likelihood of success and decrease the likelihood of failure). This enables MUSE to iteratively improve its ability to recover from failures and adapt to novel tasks through a more effective reflection process. The Actor LLM ( $M_{a}$ ) is trained using Supervised Fine-Tuning (SFT) with only successful episodes, for simplicity, across the training tasks. By imitating the behavior demonstrated in these episodes, $M_{a}$ learns to align its policy with actions seen during successful episodes. This joint multi-task fine-tuning process equips MUSE with generalizable success-driven strategies. Note that we use Low-Rank Adaptation (LoRA) (Hu et al., 2021) for parameter-efficient fine-tuning of both the $M_{a}$ and $M_{rx}$ LLMs. Additionally, the Self-Assessment Model ( $M_{sa}$ ) undergoes supervised learning with pertinent multi-task data, which maps trajectory chunks to corresponding episode outcomes. During deployment, when the agents encounter a novel task, they engage in sequential episodes until success is achieved. MUSE employs competence-aware planning, as described above, to choose its actions. Furthermore, MUSE can perform online updates to each of its constituent models as new data becomes available from the novel task. Note that the baseline agents, ReAct and Reflexion, can also respond adaptively to new experiences but rely only on in-context prompting and/or reflection. However, these methods do not involve updating model parameters, which leads to knowledge being stored only for the short term. This constraint hinders the transfer of knowledge across episodes and tasks. MUSE overcomes these limitations by continually updating the weights of its models, enabling more effective knowledge transfer. Algorithm 3 Procedure for pre-deployment training of the MUSE agent Training dataset ${D_{in}}\sim$ in-distribution tasks Specify max. training episodes, max. steps 1: function PreTrainModels ( $D_{in}$ , $M_{a}$ , $M_{rx}$ , $M_{sa}$ , $M_{w}$ ) 2: Initialize $M_{sa}$ , $M_{w}$ , memory buffer 3: for each task $\sim{D_{in}}$ do 4: Initialize $M_{a}$ , $M_{rx}$ for the Reflexion agent 5: Add $I$ to memory buffer 6: Set $t=0$ , $e=0$ 7: while $e<$ max. training episodes do 8: while $t<$ max. steps & task-not-solved do 9: Generate $a_{t}$ using $M_{a}$ and submit to simulator 10: Obtain $o_{t+1}$ and $r_{t}$ , append $o_{t}$ , $a_{t}$ , $r_{t}$ to $c_{t}$ 11: Increment $t$ 12: end while 13: Generate $rx^{e}$ using $M_{rx}$ 14: Append episode outcome, $c_{t}$ , $rx^{e}$ to memory buffer 15: Increment $e$ 16: end while 17: end for 18: Use all data in memory buffer to train $M_{a}$ , $M_{rx}$ , $M_{sa}$ , $M_{w}$ for MUSE 19: return Trained $M_{a}$ , $M_{rx}$ , $M_{sa}$ , $M_{w}$ 20: end function Algorithm 4 Procedure for deployment adaptation of the MUSE agent Test dataset ${D_{out}}\sim$ out-of-distribution tasks Specify max. adaptation episodes 1: function AdaptModels ( $D_{out}$ , $M_{a}$ , $M_{rx}$ , $M_{sa}$ , $M_{w}$ ) 2: Initialize memory buffer 3: for each task $\sim{D_{out}}$ do 4: Set $M_{a}$ , $M_{rx}$ , $M_{sa}$ , $M_{w}$ to pre-deployment trained weights 5: Add $I$ to memory buffer 6: Set $t=0$ , $e=0$ 7: while $e<$ max. adaptation episodes do 8: while $t<$ max. steps & task-not-solved do 9: Generate rollout trajectories using $M_{w}$ and $M_{a}$ 10: Evaluate each trajectory using $M_{sa}$ 11: Select the best action $a_{t}$ and submit to simulator 12: Obtain $o_{t+1}$ and $r_{t}$ , append $o_{t}$ , $a_{t}$ , $r_{t}$ to $c_{t}$ 13: Increment $t$ 14: end while 15: Generate $rx^{e}$ using $M_{rx}$ 16: Append episode outcome, $c_{t}$ , $rx^{e}$ to memory buffer 17: Increment $e$ 18: end while 19: Use data in memory buffer to update $M_{a}$ , $M_{rx}$ , $M_{sa}$ , $M_{w}$ 20: Test updated MUSE agent on the current task over the test episodes 21: end for 22: end function ### 4.2 ALFWorld Experiments For the LLM-based implementation, we evaluated our approach within the ALFWorld simulator (Shridhar et al., 2020) and compared it against ReAct (Yao et al., 2022) and Reflexion (Shinn et al., 2023) as the baselines. ALFWorld is a synthetic, text-based game simulator with diverse interactive environments that challenge agents to solve multi-step tasks (e.g., “Find two plates and put them in a cabinet”). A unique aspect of ALFWorld is its emphasis on spatial and commonsense reasoning, requiring agents to infer probable locations of common household items; for example, recognizing that plates are typically found in kitchens. This characteristic makes ALFWorld particularly well-suited for leveraging the pre-trained commonsense knowledge of LLMs. For these experiments, we selected the Mistral-7B-Instruct-v0.2 as the primary LLM (Jiang et al., 2023). This LLM features a substantial context window size of 32,000 tokens, allowing it to process large chunks of data in a single pass for improved in-context learning. Similar to the World-Model-based experiments, we employed a two-stage protocol comprising pre-deployment training on known tasks followed by deployment adaptation to unknown tasks. Each episode in these experiments had a maximum time limit of 50 steps. To further explore the robustness of our framework, we conducted additional experiments using smaller or less capable LLMs such as Mistral-7B-OpenOrca (Mukherjee et al., 2023; Open-Orca, 2024) and OpenELM-3B-Instruct (Mehta et al., 2024). The MUSE agent underwent pre-deployment training using 140 household tasks from ALFWorld’s in-distribution (seen) set. For deployment adaptation, we evaluated all agents on 134 household tasks from ALFWorld’s out-of-distribution (unseen) set. The agents were exposed to one novel task at a time, starting with the pre-deployment trained weights (for MUSE) or pre-trained LLM weights (for ReAct and Reflexion), under two conditions: (1) five test episodes for each task without any adaptation and (2) five adaptation episodes followed by five test episodes for each task. The agents were assessed for performance on the novel tasks over the test episodes. Note that there were no weight updates to the MUSE agent during these episodes. While it is possible to continually update all constituent models of MUSE (namely, $M_{w}$ , $M_{a}$ , $M_{rx}$ , $M_{sa}$ ) using new data acquired from adaptation episodes, we chose to update only the MLP portion ( $g_{\eta}$ ) of $M_{sa}$ to minimize computational time without loss of any generality. We used a replay buffer that included both training data and new adaptation data to update $M_{sa}$ without catastrophic forgetting of pre-deployment trained performance. #### 4.2.1 Metrics Self-Assessment We used metacognitive accuracy and AUROC2 to evaluate how well MUSE predicts its success on the novel tasks. MUSE was evaluated for each novel task separately over five test episodes, at the end of pre-deployment training and after five adaptation episodes. During each test episode, the trajectory was segmented into non-overlapping chunks, each containing four action-observation pairs. MUSE’s chunk-wise self-assessment predictions were then collected for evaluation, and the predictions for all chunks across tasks were compared against the true labels to compute the metacognitive accuracy and AUROC2 metrics. Note that, as the baseline agents, ReAct and Reflexion, lack a self-assessment module to predict episode outcomes, we cannot make a meaningful comparison between them and MUSE for self-assessment. Self-Regulation We evaluated how well the agents generalize to unknown situations using two metrics: the percentage of the novel tasks solved and the average time to task completion. Each agent was assessed for each novel task separately over the five test episodes. The percentage of episodes in which the agent completed the task within the maximum time limit was averaged across all the novel tasks to calculate the success rate. Similarly, the number of time steps required to achieve success in each episode was averaged across all episodes and tasks to compute the time-to-completion metric. There was a penalty of 100 time steps for failures. #### 4.2.2 Results Self-Assessment The self-assessment results are summarized in Table 2, with the corresponding Type 2 ROC curves shown in Figure 9. Following the five adaptation episodes, the MUSE agent achieved an AUROC2 of 0.93 and a metacognitive accuracy of 85%, demonstrating a high level of competence awareness. Table 2: Self-assessment performance of the MUSE agent on the novel tasks in the ALFWorld environment, at the end of pre-deployment training (No Adaptation) and after five adaptation episodes (Adaptation). | Condition | AUROC2 | Metacognitive Accuracy | | --- | --- | --- | | No Adaptation | 0.66 | 60% | | Adaptation | 0.93 | 85% | <details> <summary>figures/self_aware_LLM.png Details</summary>  ### Visual Description \n ## Chart: Receiver Operating Characteristic (ROC) Curves ### Overview The image presents two Receiver Operating Characteristic (ROC) curves, likely representing the performance of a classification model (labeled "MUSE") across different threshold settings. Each ROC curve plots the True Positive Rate (TPR) against the False Positive Rate (FPR). A diagonal dashed gray line represents a random classifier (TPR = FPR). The curves visually demonstrate the trade-off between sensitivity and specificity. ### Components/Axes * **X-axis:** False Positive Rate (ranging from 0.0 to 1.0) * **Y-axis:** True Positive Rate (ranging from 0.0 to 1.0) * **Data Series:** "MUSE" (represented by an orange solid line) - appears twice. * **Reference Line:** A dashed gray line representing a random classifier (TPR = FPR). * **Legend:** Located in the bottom-right corner of each plot, labeling the orange line as "MUSE". ### Detailed Analysis or Content Details **Plot 1 (Left):** The orange "MUSE" line starts at approximately (0.0, 0.0). It initially slopes upward relatively steeply, reaching a TPR of around 0.2 at an FPR of approximately 0.1. The curve continues to rise, but the slope decreases. At an FPR of 0.4, the TPR is around 0.5. At an FPR of 0.6, the TPR is approximately 0.7. The curve plateaus around a TPR of 0.9, with an FPR of approximately 0.95. **Plot 2 (Right):** The orange "MUSE" line starts at approximately (0.0, 0.0). It rises very steeply initially, reaching a TPR of around 0.8 at an FPR of approximately 0.2. The curve continues to rise, but the slope decreases. At an FPR of 0.4, the TPR is around 0.9. At an FPR of 0.6, the TPR is approximately 0.95. The curve plateaus around a TPR of 0.98, with an FPR of approximately 0.98. ### Key Observations * Both ROC curves demonstrate that the "MUSE" model performs significantly better than random chance (the diagonal line). * The second plot (right) shows a substantially better performance than the first plot (left). The curve is closer to the top-left corner, indicating a higher TPR for a given FPR. * The area under the curve (AUC) is visually larger in the second plot, suggesting a better ability to discriminate between positive and negative cases. * Both curves exhibit a trade-off between TPR and FPR. Increasing the threshold to improve specificity (reduce FPR) results in a decrease in sensitivity (TPR), and vice versa. ### Interpretation The ROC curves suggest that the "MUSE" model is capable of distinguishing between positive and negative cases. The difference in performance between the two plots indicates that there may be different datasets, different model configurations, or different threshold settings being evaluated. The second plot demonstrates a much stronger ability to correctly identify positive cases while minimizing false positives. The curves are useful for evaluating the model's performance and selecting an appropriate operating point (threshold) based on the specific application's requirements. For example, if minimizing false positives is critical, a lower threshold would be chosen, even if it means missing some true positives. Conversely, if maximizing true positives is paramount, a higher threshold would be selected, accepting a higher rate of false positives. The fact that both curves are above the diagonal line indicates that the model is consistently performing better than random guessing. The steepness of the initial rise in the curves suggests that the model is particularly good at identifying true positives with a low rate of false positives. </details> Figure 9: Type 2 ROC curves of the MUSE agent for self-assessment of competence on the novel tasks in the ALFWorld environment, at the end of pre-deployment training (left) and after five adaptation episodes (right). Table 3: Self-Regulation: Success rate of various agents on the novel tasks, at the end of pre-deployment training (No Adaptation) and after five adaptation episodes (Adaptation). | Agent | No Adaptation | Adaptation | | --- | --- | --- | | ReAct | 35% | 35% | | Reflexion | 45% | 51% | | Reflexion with supervised learning | 62% | 65% | | MUSE with the lowest self-assessment | 59% | 61% | | MUSE with random self-assessment | 68% | 69% | | MUSE | 84% | 90% | Table 4: Self-Regulation: Average number of time steps to success per episode for various agents on the novel tasks, at the end of pre-deployment training (No Adaptation) and after five adaptation episodes (Adaptation). | Agent | No Adaptation | Adaptation | | --- | --- | --- | | ReAct | 97 | 97 | | Reflexion | 77 | 66 | | Reflexion with supervised learning | 61 | 54 | | MUSE with the lowest self-assessment | 65 | 62 | | MUSE with random self-assessment | 53 | 49 | | MUSE | 43 | 38 | Table 5: Self-Regulation: Success rate of various agents with smaller or less capable LLMs on the novel tasks, after five adaptation episodes. | Agent | Mistral-7B-OpenOrca | OpenELM-3B-Instruct | | --- | --- | --- | | ReAct | 23% | 4% | | Reflexion | 27% | 9% | | MUSE | 58% | 55% | To highlight the importance of self-assessment, we provide examples of trajectories generated by different agents. As shown in Supplementary Section 8.5, the key to MUSE’s success lies in its continual competence awareness. MUSE generates several rollout trajectories, assigns competence scores to each, and selects the one most likely to result in task success. For the given task, MUSE begins with an initial plan similar to that of Reflexion: “find a bowl, clean it at the sink, and then place it in the cabinet.” However, MUSE sets itself apart by engaging in deliberate actions that ensure steady progress toward the goal with increasing competence. After picking up the bowl from cabinet 1, MUSE evaluates multiple options, such as follows: 1. “think: Now I find a bowl (1). Next, I need to take it..OK..> take bowl 1 from cabinet 1.” … Score: 0.6284 1. “think: Now I take a bowl (1). Next, I need to go to sinkbasin (1) and clean it.” … Score: 0.8689 1. “think: Now I find a bowl (1). Next, I need to take it..OK..> take bowl 1 from cabinet 1. Nothing happens.” … Score: 0.6417 In this instance, MUSE selects Trajectory 1 due to its higher competence score, which indicates a greater likelihood of task success. By continually assessing its competence, MUSE adapts its strategy in real time to effectively navigate the environment’s complexities. It maintains a coherent plan and avoids redundant or detrimental actions that could lead to confusion or failure. In contrast, Reflexion fails due to its lack of competence awareness. After acquiring the bowl, Reflexion redundantly decides to find and take a bowl again. This leads to a faulty sequence of actions, causing Reflexion to enter an incorrect state from which it cannot recover. Without competence awareness, Reflexion fails to recognize that it already possesses the bowl. Its actions become increasingly erratic, failing to contribute toward task completion. For the complete trajectory, refer to Supplementary Section 8.5. In summary, competence awareness allows MUSE to adapt more effectively to unfamiliar situations. By avoiding low-competence rollout trajectories, it steers clear of actions leading to failure, promotes exploration within the agent’s capabilities, and prevents the agent from getting stuck in unrecoverable states. This approach results in smarter exploration and more effective online learning in response to novel scenarios. Self-Regulation The self-regulation results for MUSE and the two baseline agents are summarized in Tables 3 and 4. These results demonstrate that MUSE solves more novel tasks and requires fewer time steps to completion per episode, both at the end of pre-deployment training and after five adaptation episodes. Notably, MUSE achieved a high success rate of 90% on the 134 novel tasks, outperforming the baseline agents of ReAct and Reflexion by >150% and >75%, respectively. It’s important to note that all agents in the pertinent comparisons have access to the same data, ensuring a fair evaluation. ReAct and Reflexion can also respond adaptively to new experiences but rely only on in-context prompting and/or reflection. So, the observed performance gains for MUSE are from its ability to leverage available data more effectively by performing competence-aware planning, not from access to any privileged information. To clarify and validate this point, we conducted additional experiments comparing: - Reflexion with supervised learning across the training tasks during pre-deployment (namely, DPO and SFT) - MUSE with self-regulation based on the lowest self-assessment - MUSE with self-regulation based on random self-assessment The self-regulation results for these control conditions are also summarized in Tables 3 and 4. Reflexion with supervised learning during pre-deployment can consolidate knowledge across the training tasks, but exhibits only a limited improvement over its original implementation (Shinn et al., 2023) in adaptation to the novel tasks. This suggests that even a generalized reflection mechanism is less effective in novel scenarios without self-assessment and self-regulation. Furthermore, variants of MUSE using poor (random or pessimistic) self-assessment signals underperform, validating the importance of self-assessment for effective self-regulation. Note that even when self-assessment is flawed, the performance of the MUSE agent does not collapse because the Actor LLM ( $M_{a}$ ) has benefited from pre-deployment training to already generate reasonable plans for competence-aware selection. To further evaluate the contribution of the self-assessment and self-regulation framework in MUSE beyond LLM-powered reasoning, we conducted additional experiments using smaller or less capable LLMs. Table 5 presents the post-adaptation performance of MUSE compared to the baseline agents when employing smaller or less capable LLM variants. MUSE maintained relatively higher success rates (55-58%) on the novel tasks, achieving >2x and >6x better performance than Reflexion when using Mistral-7B-OpenOrca and OpenELM-3B-Instruct, respectively. These results suggest that metacognition plays a particularly significant role when operating under constrained conditions, such as with smaller or less capable LLMs. ### 4.3 Discussion Through these experiments, we evaluated the metacognitive capabilities of MUSE to showcase its superior performance over the baseline agents of ReAct and Reflexion. With reliable competence awareness on the novel tasks, the MUSE agent can make informed decisions by avoiding rollout trajectories likely to lead to failure and prioritizing those with a higher probability of task success for more effective online adaptation. MUSE’s ability to learn efficiently from a limited number of adaptation episodes is particularly critical in unknown situations and environments, where task complexity and novelty can vary significantly. Furthermore, its strong performance even with less-capable LLMs suggests that incorporating metacognitive components into mainstream AI agents can enhance performance under resource-constrained conditions. Overall, the LLM-based experiments demonstrated that integrating self-assessment and self-regulation into LLM-based agents can significantly improve their ability to handle unknown scenarios. ## 5 Conclusions In this article, we introduced the Metacognition for Unknown Situations and Environments (MUSE) framework to integrate self-assessment and self-regulation into AI agents to enhance their adaptability to unknown situations and environments. MUSE uses self-assessed competence to intelligently guide an iterative trial-and-error process to identify the right strategy. In contrast, current AI agents either blindly execute their routine or policy, risking catastrophic failures, or freeze as a result of runtime anomaly detection. These limitations arise because big data-driven solutions inherently struggle to adapt to novel scenarios with little or no training data. We presented two implementations of MUSE, one for MBRL agents and another for LLM-based agents, to instantiate the metacognitive cycle. In both cases, our experiments demonstrated that MUSE agents outperform baseline methods in handling novel tasks. These results show that MUSE facilitates problem-solving in unfamiliar environments by vetoing potentially catastrophic solutions, guiding the search toward strategies that leverage the agent’s strengths, and learning online from the environment with each attempt. In high-stakes applications like autonomous driving, MUSE can facilitate safety by relying on its self-assessment as a form of competence calibration. Furthermore, the online learning during deployment enables MUSE to fix and overcome potential metacognitive misjudgments in novel scenarios. This avoids the infinite regress of needing to calibrate the preceding lower-order calibration. In summary, by enabling agents to be aware of their competence and to regulate their strategies accordingly, MUSE offers a promising pathway toward developing more resilient and versatile AI systems. ## 6 Future Work While MUSE outperforms the baseline AI agents, metacognition in AI remains a nascent and exploratory field. The scope of metacognitive abilities is broad, offering several promising directions for future research. For instance, MUSE could be expanded to include metacognitive judgments that further enhance an agent’s performance and learning. These include the Feeling of Knowing to help the agent persist in difficult, novel scenarios; the Judgment of Learning to facilitate resource-efficient learning; and Source Monitoring to assess the reliability of its knowledge. Another key avenue is integrating MUSE with lifelong learning techniques, such as generative experience replay and selective plasticity, to incrementally update its constituent models in new scenarios while mitigating catastrophic forgetting (Rostami et al., 2019; Kolouri et al., 2020; Kudithipudi et al., 2023). Such an integration would enable agents to retain and build upon previously learned knowledge while continually acquiring new skills, without requiring the storage of data from all prior conditions in the replay buffer, as was done in the current study. Another exciting direction for future research involves applying MUSE to real-world domains, such as autonomous driving and robotics in unstructured settings. Investigating these domains for tasks with longer time horizons and larger state and action spaces will provide valuable insights into the scalability, robustness, and practicality of the MUSE framework. In this regard, MCTS methods can be adapted to leverage the self-assessment model from MUSE as the node evaluation function for guiding the search, enabling more effective exploration and problem-solving in complex environments. Another compelling avenue is expanding the metacognitive framework to incorporate additional aspects of human cognition that are currently absent in AI systems. For instance, integrating Theory-of-Mind could enhance agents’ abilities to understand and predict human behaviors, enabling more effective collaboration in social or teamwork-driven environments. Additionally, exploring multi-agent interactions within the MUSE framework offers significant potential. Enabling multiple metacognitive AI agents to interact, collaborate, negotiate, or compete could lead to more sophisticated and efficient problem-solving at the group level. Our research highlights the potential of psychology- and brain-inspired approaches to address the limitations of current AI systems, particularly their dependence on extensive training data. By equipping AI agents with greater flexibility and adaptability, MUSE paves the way for more resilient and versatile systems capable of operating across a wide range of environments. We anticipate that future research will build upon and refine these ideas, advancing the field of metacognitive AI and expanding its applications across diverse domains. ## 7 Acknowledgments The authors would like to thank Neale Ratzlaff for his invaluable contributions to this research and his constructive feedback. ## 8 Supplementary Material Note that the trajectory examples provided below for the LLM-based experiments are exact copies of the LLM outputs. ### 8.1 Illustrative ALFWorld trajectories for in-context prompting. (Yao et al., 2022) Interact with a household to solve a task. Here are two examples. // You are in the middle of a room. Looking quickly around you, you see a cabinet 13, a cabinet 12, a cabinet 11, a cabinet 10, a cabinet 9, a cabinet 8, a cabinet 7, a cabinet 6, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a coffeemachine 1, a countertop 1, a diningtable 1, a drawer 1, a fridge 1, a garbagecan 1, a microwave 1, a shelf 3, a shelf 2, a shelf 1, a sinkbasin 1, a stoveburner 4, a stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster 1. Your task is to: put a hot apple in fridge. >think: To solve the task, I need to find and take an apple, then heat it with microwave, then put it in fridge. OK. >think: First I need to find an apple. An apple is more likely to appear in fridge (1), diningtable (1), coffeetable (1), drawer (1), cabinet (1-13), garbagecan (1). I can check one by one, starting with fridge 1. OK. >go to fridge 1 The fridge 1 is closed. >open fridge 1 You open the fridge 1. The fridge 1 is open. In it, you see a cup 1, and a egg 1. >go to diningtable 1 On the diningtable 1, you see a apple 1, a bread 1, a fork 3, a fork 2, a fork 1, a mug 2, a peppershaker 3, a plate 2, a pot 1, a soapbottle 3, a spatula 1, a spoon 3, a spoon 2, a spoon 1, a winebottle 3, and a winebottle 2. >think: Now I find an apple (1). Next, I need to take it. OK. >take apple 1 from diningtable 1 You pick up the apple 1 from the diningtable 1. >think: Now I take an apple (1). Next, I need to go to a microwave (1) and heat it. OK. >go to microwave 1 The microwave 1 is closed. >heat apple 1 with microwave 1 You heat the apple 1 using the microwave 1. >think: Now I heat an apple (1). Next, I need to put it in/on fridge 1. OK. >go to fridge 1 The fridge 1 is open. In it, you see a cup 1, and a egg 1. >put apple 1 in/on fridge 1 You put the apple 1 in/on the fridge 1. // You are in the middle of a room. Looking quickly around you, you see a cabinet 10, a cabinet 9, a cabinet 8, a cabinet 7, a cabinet 6, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a coffeemachine 1, a countertop 3, a countertop 2, a countertop 1, a diningtable 1, a drawer 6, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a fridge 1, a garbagecan 1, a microwave 1, a sinkbasin 1, a stoveburner 4, a stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster 1. Your task is to: heat some egg and put it in diningtable. >think: To solve the task, I need to find and take an egg, then heat it with microwave, then put it in diningtable. OK. >think: First I need to find an egg. An egg is more likely to appear in fridge (1), countertop (1-3), diningtable (1), stoveburner (1-4), toaster (1), garbagecan (1), cabinet (1-10). I can check one by one, starting with fridge 1. OK. >open fridge 1 You open the fridge 1. The fridge 1 is open. In it, you see a lettuce 2, a mug 2, and a potato 3. >go to countertop 1 On the countertop 1, you see a bread 1, a fork 1, and a saltshaker 1. >go to countertop 2 On the countertop 2, you see nothing. >go to countertop 3 On the countertop 3, you see a bowl 1, a butterknife 1, a egg 2, a kettle 2, a plate 1, a sink 1, and a spatula 2. >think: Now I find an egg (2). Next, I need to take it. OK. >take egg 2 from countertop 3 You pick up the egg 2 from the countertop 3. >think: Now I take an egg (2). Next, I need go to a microwave (1) and heat it. OK. >go to microwave 1 The microwave 1 is closed. >heat egg 2 with microwave 1 You heat the egg 2 using the microwave 1. >think: Now I heat an egg (2). Next, I need to put it in/on diningtable 1. OK. >go to diningtable 1 On the diningtable 1, you see a apple 2, a bread 3, a egg 1, a kettle 1, a knife 1, a mug 1, a papertowelroll 1, a peppershaker 2, a potato 1, a soapbottle 1, and a spatula 1. >put egg 2 in/on diningtable 1 You put the egg 2 in/on the diningtable 1. ### 8.2 Example of ReAct’s decision-making | Task: You are in the middle of a room. Looking quickly around you, you see a cabinet 6, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a coffeemachine 1, a countertop 3, a countertop 2, a countertop 1, a drawer 3, a drawer 2, a drawer 1, a fridge 1, a garbagecan 1, a microwave 1, a shelf 3, a shelf 2, a shelf 1, a sinkbasin 1, a stoveburner 4, a stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster 1.Your task is to: put a hot tomato in garbagecan. | | --- | | ReAct’s trajectory: ReAct successfully identifies and picks a tomato. However, it encounters difficulties when attempting to heat it. The agent repetitively navigates to each of the stove burners, resulting in a continuous loop. | | Conclusion: The trajectory ends with the agent trapped in a repetitive sequence of actions, unable to adjust its approach or adapt to the task of heating the tomato. | >action: think: To solve the task, I need to find and take a tomato, then heat it with stoveburner, then put it in garbagecan. observation: OK. >action: think: First I need to find a tomato. A tomato is more likely to appear in fridge (1), countertop (1-3), cabinet (1-6), garbagecan (1), drawer (1-3). I can check one by one, starting with fridge 1. observation: OK. >action: open fridge 1 observation: Nothing happens. >action: go to countertop 1 observation: On the countertop 1, you see a dishsponge 1, and a tomato 1. >action: think: Now I find a tomato (1). Next, I need to take it. observation: OK. >action: take tomato 1 from countertop 1 observation: You pick up the tomato 1 from the countertop 1. >action: think: Now I take a tomato (1). Next, I need to go to a stoveburner (1-4) and heat it. observation: OK. >action: go to stoveburner 1 observation: On the stoveburner 1, you see nothing. >action: go to stoveburner 2 observation: On the stoveburner 2, you see nothing. >action: go to stoveburner 3 observation: On the stoveburner 3, you see nothing. >action: go to stoveburner 4 observation: On the stoveburner 4, you see nothing. >action: think: Now I need to find a stoveburner with a hot surface. I can check stoveburners 1-4. observation: OK. >action: go to stoveburner 1 observation: On the stoveburner 1, you see nothing. >action: go to stoveburner 2 observation: On the stoveburner 2, you see nothing. >action: go to stoveburner 3 observation: On the stoveburner 3, you see nothing. >action: go to stoveburner 4 observation: On the stoveburner 4, you see nothing. >action: think: I need to turn on a stoveburner and wait for it to heat up. I can check stoveburners 1-4. observation: OK. >action: turn on stoveburner 1 observation: Nothing happens. >action: turn on stoveburner 2 observation: Nothing happens. >action: turn on stoveburner 3 observation: Nothing happens. >action: turn on stoveburner 4 observation: Nothing happens. >action: think: I need to find a way to heat up a stoveburner. I can check the stove, the microwave, or the oven. observation: OK. >action: check stove observation: Nothing happens. >action: check microwave observation: Nothing happens. >action: check oven observation: Nothing happens. >action: think: I need to find a source of heat. I can check the stove, the microwave, or the oven. observation: OK. >action: check stove observation: Nothing happens. >action: check microwave observation: Nothing happens. >action: check oven observation: Nothing happens. >action: think: I need to find a way to heat up a tomato. I can check the stove, the microwave, or the oven. observation: OK. >action: check stove observation: Nothing happens. >action: check microwave observation: Nothing happens. >action: check oven observation: Nothing happens. >action: think: I need to find a way to heat up a tomato. I can check the stove, the microwave, or the oven. observation: OK. >action: check stove observation: Nothing happens. >action: check microwave observation: Nothing happens. >action: check oven observation: Nothing happens. >action: think: I need to find a way to heat up a tomato. I can check the stove, the microwave, or the oven. observation: OK. >action: check stove observation: Nothing happens. >action: check microwave observation: Nothing happens. >action: check oven observation: Nothing happens. >action: think: I need to find a way to heat up a tomato. I can check the stove, the microwave, or the oven. observation: OK. >action: check stove observation: Nothing happens. >action: check microwave observation: Nothing happens. >action: check oven observation: Nothing happens. >action: think: I need to find a way to heat up a tomato. I can check the stove, the microwave, or the oven. observation: OK. >action: check stove observation: Nothing happens. >action: check microwave observation: Nothing happens. >action: check oven observation: Nothing happens. ### 8.3 Example of Reflexion’s decision-making, demonstrating improvements over ReAct | Task: You are in the middle of a room. Looking quickly around you, you see a cabinet 6, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a coffeemachine 1, a countertop 3, a countertop 2, a countertop 1, a drawer 3, a drawer 2, a drawer 1, a fridge 1, a garbagecan 1, a microwave 1, a shelf 3, a shelf 2, a shelf 1, a sinkbasin 1, a stoveburner 4, a stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster 1.Your task is to: put a hot tomato in garbagecan. | | --- | | Reflexive feedback from first episode: “In this environment, my plan was to find a tomato, heat it on a stoveburner, and put it in the garbage can. However, I was unable to find a stoveburner with a hot surface. I should have checked the oven or microwave for alternative methods of heating the tomato. I will try to heat the tomato in the oven or microwave in the next trial.” | | Reflexion’s trajectory: The Reflexion agent reflects on its initial failure to heat the tomato using the stove burners and identifies the potential to utilize alternative heating methods. In subsequent episodes, it successfully uses the microwave to heat the tomato. | | Conclusion: This trajectory demonstrates the agent’s ability to learn from past mistakes and adapt its strategy, ultimately completing the task successfully by heating the tomato in the microwave and placing it in the garbage can. | >action: think: To solve the task, I need to find and take a tomato, then heat it, and put it in garbagecan. observation: OK. >action: think: First I need to find a tomato. A tomato is more likely to appear in fridge (1), countertop (1-3), cabinet (1-6), garbagecan (1). I can check one by one, starting with fridge 1. observation: OK. >action: open fridge 1 observation: Nothing happens. >action: go to countertop 1 observation: On the countertop 1, you see a dishsponge 1, and a tomato 1. >action: think: Now I find a tomato (1). Next, I need to take it. observation: OK. >action: take tomato 1 from countertop 1 observation: You pick up the tomato 1 from the countertop 1. >action: think: Now I take a tomato (1). Next, I need to heat it. observation: OK. >action: think: I can heat a tomato in the microwave (1), oven, or on a stoveburner (1-4). I will try the microwave first. observation: OK. >action: go to microwave 1 observation: The microwave 1 is closed. >action: heat tomato 1 with microwave 1 observation: You heat the tomato 1 using the microwave 1. >action: think: Now I heat a tomato (1). Next, I need to put it in garbagecan 1. observation: OK. >action: go to garbagecan 1 observation: On the garbagecan 1, you see a egg 3, and a papertowelroll 1. >action: put tomato 1 in/on garbagecan 1 observation: You put the tomato 1 in/on the garbagecan 1. ### 8.4 Limitations of Reflexion compared to MUSE in strategic exploration and task execution | Col1 | | --- | | Task: You are in the middle of a room. Looking quickly around you, you see a armchair 1, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a dresser 1, a garbagecan 1, a safe 1, a shelf 12, a shelf 11, a shelf 10, a shelf 9, a shelf 8, a shelf 7, a shelf 6, a shelf 5, a shelf 4, a shelf 3, a shelf 2, a shelf 1, a sidetable 1, and a sofa 1.Your task is to: put some watch on safe. | | Reflexion’s trajectory: Reflexion initially plans to find a watch but begins its search in less likely locations instead of the dresser, the most probable spot. After inefficient exploration, it eventually locates the watch on the dresser. However, at the safe, it prematurely closes it without placing the watch inside, leading to a loop from which it cannot recover. | | Conclusion: The trajectory ends with the agent stuck in a repetitive cycle, unable to adapt its strategy or correct its mistake in handling the watch and the safe. | >action: think: To solve the task, I need to find and take a watch, then put it in the safe. observation: OK. >action: think: First I need to find a watch. A watch is more likely to appear in drawers (1-5), cabinet (1-4), dresser (1), and shelves (1-12). I can check one by one, starting with drawer 1. observation: OK. >action: go to drawer 1 observation: The drawer 1 is closed. >action: open drawer 1 observation: You open the drawer 1. The drawer 1 is open. In it, you see nothing. >action: go to drawer 2 observation: The drawer 2 is closed. >action: open drawer 2 observation: You open the drawer 2. The drawer 2 is open. In it, you see nothing. >action: go to drawer 3 observation: The drawer 3 is closed. >action: open drawer 3 observation: You open the drawer 3. The drawer 3 is open. In it, you see a creditcard 2. >action: go to drawer 4 observation: The drawer 4 is closed. >action: open drawer 4 observation: You open the drawer 4. The drawer 4 is open. In it, you see nothing. >action: go to drawer 5 observation: The drawer 5 is closed. >action: open drawer 5 observation: You open the drawer 5. The drawer 5 is open. In it, you see nothing. >action: go to cabinet 1 observation: The cabinet 1 is closed. >action: open cabinet 1 observation: You open the cabinet 1. The cabinet 1 is open. In it, you see a vase 1. >action: go to cabinet 2 observation: Nothing happens. >action: go to cabinet 3 observation: The cabinet 3 is closed. >action: open cabinet 3 observation: You open the cabinet 3. The cabinet 3 is open. In it, you see a tissuebox 2. >action: go to cabinet 4 observation: Nothing happens. >action: go to dresser 1 observation: On the dresser 1, you see a laptop 1, a remotecontrol 1, a tissuebox 1, a vase 3, a vase 2, and a watch 1. >action: think: Now I find a watch (1). Next, I need to take it. observation: OK. >action: take watch 1 from dresser 1 observation: You pick up the watch 1 from the dresser 1. >action: think: Now I take a watch (1). Next, I need to put it in/on safe 1. observation: OK. >action: go to safe 1 observation: The safe 1 is closed. >action: open safe 1 observation: You open the safe 1. The safe 1 is open. In it, you see a cellphone 3, and a cellphone 2. >action: think: The safe is not big enough for the watch and the cellphones. I should have put the watch in the safe, then closed the safe. I will try this new plan in the next trial. observation: OK. >action: close safe 1 observation: You close the safe 1. >action: put watch 1 in/on safe 1 observation: Nothing happens. >action: think: I need to put the watch in the safe, then close the safe. observation: OK. >action: put watch 1 in/on safe 1 observation: Nothing happens. >action: think: I need to put the watch in the safe, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. >action: think: I need to put the watch in safe 1, then close the safe. observation: OK. >action: put watch 1 in safe 1 observation: Nothing happens. | MUSE’s trajectory: MUSE begins with an intelligent search strategy, prioritizing the dresser as the most likely location for the watch. After efficiently locating the watch on the dresser, MUSE proceeds directly to the safe, opens it, places the watch inside, and completes the task without error. | | --- | | Conclusion: MUSE successfully completes the task through efficient exploration and precise execution, demonstrating effective problem-solving capabilities. | >action: think: To solve the task, I need to find and take a watch, then put it in the safe. observation: OK. >action: think: First I need to find a watch. A watch is more likely to appear in dresser (1), cabinet (1-4), shelf (1-12), drawer (1-5). I can check one by one, starting with dresser 1. observation: OK. >action: go to dresser 1 observation: On the dresser 1, you see a laptop 1, a remotecontrol 1, a tissuebox 1, a vase 3, a vase 2, and a watch 1. >action: take watch 1 from dresser 1 observation: You pick up the watch 1 from the dresser 1. >action: think: Now I find a watch (1). Next, I need to put it in/on safe 1. observation: OK. >action: go to safe 1 observation: The safe 1 is closed. >action: open safe 1 observation: You open the safe 1. The safe 1 is open. In it, you see a cellphone 3, and a cellphone 2. >action: put watch 1 in/on safe 1 observation: You put the watch 1 in/on the safe 1. ### 8.5 Competence-aware action selection: A key advantage of MUSE over Reflexion >action: think: To solve the task, I need to find and take a bowl, then clean it with sinkbasin, then put it in cabinet. observation: OK. >action: think: First I need to find a bowl. A bowl is more likely to appear in cabinet (1-6), countertop (1-3), sinkbasin (1), stoveburner (1-4), fridge (1). I can check one by one, starting with cabinet 1. observation: OK. >action: go to cabinet 1 observation: On the cabinet 1, you see a bowl 1. >action: take bowl 1 from cabinet 1 observation: You pick up the bowl 1 from the cabinet 1. >action: think: Now I find a bowl (1). Next, I need to take it. observation: OK. >action: take bowl 1 from cabinet 1 observation: Nothing happens. >action: think: I need to go to countertop 1 observation: OK. >action: go to countertop 1 observation: On the countertop 1, you see a creditcard 1, a egg 3, and a glassbottle 1. >action: think: Now I find a bowl (3). Next, I need to go to countertop 3 and find it. observation: OK. >action: go to countertop 3 observation: On the countertop 3, you see a bread 3, a cellphone 1, a glassbottle 3, a houseplant 1, a knife 3, a knife 2, a potato 3, a spatula 2, a spatula 1, a spoon 1, a statue 2, and a tomato 1. >action: take bowl 3 from countertop 3 observation: Nothing happens. >action: think: I can’t find bowl 3 in countertop 3. Maybe it’s in drawer (1-3). observation: OK. >action: go to drawer 3 observation: The drawer 3 is closed. >action: open drawer 3 observation: You open the drawer 3. The drawer 3 is open. In it, you see a fork 2, and a saltshaker 2. >action: think: Now I find a bowl (3). Next, I need to take it. observation: OK. >action: take bowl 3 from drawer 3 observation: Nothing happens. >action: think: I can’t find a bowl (3) in drawer (3). Maybe it’s in countertop (1-3) or cabinet (1-6). observation: OK. >action: go to countertop 1 observation: On the countertop 1, you see a creditcard 1, a egg 3, and a glassbottle 1. >action: take bowl 3 from countertop 1 observation: Nothing happens. >action: think: I can’t find a bowl (3) in countertop (1-3). Maybe it’s in drawer (1-3). observation: OK. >action: go to drawer 2 observation: The drawer 2 is closed. >action: open drawer 2 observation: You open the drawer 2. The drawer 2 is open. In it, you see a fork 1. >action: take bowl 3 from drawer 2 observation: Nothing happens. >action: think: I can’t find a bowl (3) in drawer (2). Maybe it’s in countertop (1-3) or cabinet (1-6). observation: OK. >action: go to countertop 1 observation: On the countertop 1, you see a creditcard 1, a egg 3, and a glassbottle 1. >action: take bowl 3 from countertop 1 observation: Nothing happens. >action: think: I can’t find a bowl (3) in countertop (1-3). Maybe it’s in cabinet (1-6). observation: OK. >action: go to cabinet 1 observation: On the cabinet 1, you see nothing. >action: go to cabinet 2 observation: The cabinet 2 is closed. >action: open cabinet 2 observation: You open the cabinet 2. The cabinet 2 is open. In it, you see a vase 1. >action: go to cabinet 3 observation: On the cabinet 3, you see a plate 1. >action: take bowl 3 from cabinet 3 observation: Nothing happens. >action: think: I can’t find a bowl (3) in cabinet (3-1). Maybe it’s in countertop (1-3) or drawer (1-3). observation: OK. >action: go to countertop 1 observation: On the countertop 1, you see a creditcard 1, a egg 3, and a glassbottle 1. >action: take bowl 3 from countertop 1 observation: Nothing happens. >action: think: I can’t find a bowl (3) in countertop (1-3). Maybe it’s in drawer (1-3). observation: OK. >action: go to drawer 1 observation: The drawer 1 is closed. >action: open drawer 1 observation: You open the drawer 1. The drawer 1 is open. In it, you see a peppershaker 1, and a saltshaker 1. >action: think: Now I find a bowl (3). It’s a potato 1. observation: OK. >action: take potato 1 from drawer 1 observation: Nothing happens. >action: think: I can’t find a bowl (3) in drawer (1). Maybe it’s in countertop (1-3) or cabinet (1-13). observation: OK. >action: go to countertop 1 observation: On the countertop 1, you see a creditcard 1, a egg 3, and a glassbottle 1. >action: think: Now I find a bowl (3). It’s a potato 1. observation: OK. >action: think: Now I find a bowl (3). It’s a potato 1. Next, I need to take it. observation: OK. >action: take potato 1 from countertop 1 observation: Nothing happens. >action: think: I can’t take a bowl (3) from countertop (1-3). Maybe it’s in cabinet (1-13). observation: OK. >action: go to cabinet 1 observation: On the cabinet 1, you see nothing. >action: go to cabinet 2 observation: The cabinet 2 is open. In it, you see a vase 1. >action: take bowl 3 from cabinet 2 observation: Nothing happens. >action selected: think: To solve the task, I need to find and take a bowl, then clean it with sinkbasin, then put it in cabinet. observation: OK. >action selected: think: First I need to find a bowl. A bowl is more likely to appear in cabinet (1-6), countertop (1-3), sinkbasin (1), or garbagecan (1). I can check one by one, starting with cabinet 1. observation: OK. >action selected: go to cabinet 1 observation: On the cabinet 1, you see a bowl 1. >action selected: take bowl 1 from cabinet 1 observation: You pick up the bowl 1 from the cabinet 1. >action selected: think: Now I take a bowl (1). Next, I need to go to sinkbasin (1) and clean it. observation: OK. >action selected: go to sinkbasin 1 observation: On the sinkbasin 1, you see nothing. >action selected: clean bowl 1 with sinkbasin 1 observation: You clean the bowl 1 using the sinkbasin 1. >action selected: think: Now I clean a bowl (1). Next, I need to put it in/on cabinet 1. observation: OK. >action selected: go to cabinet 1 observation: On the cabinet 1, you see nothing. >action selected: put bowl 1 in/on cabinet 1 observation: You put the bowl 1 in/on the cabinet 1. Detailed trajectory of the MUSE agent, which highlights the benefits of competence awareness in strategic planning and action selection >action selected: think: To solve the task, I need to find and take a bowl, then clean it with sinkbasin, then put it in cabinet. observation: OK. >action selected: think: First I need to find a bowl. A bowl is more likely to appear in cabinet (1-6), countertop (1-3), sinkbasin (1), or garbagecan (1). I can check one by one, starting with cabinet 1. observation: OK. Imagine Trajectory 0: go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a glassbottle 1..> go to cabinet 2.The cabinet 2 is closed..> open cabinet 2.You open the cabinet 2 Imagine Trajectory 1: go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a saltshaker 1..> go to cabinet 2.The cabinet 2 is closed..> open cabinet 2.You open the cabinet 2. Imagine Trajectory 2: go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a bowl 2..> go to cabinet 2.The cabinet 2 is closed..> open cabinet 2.You open the cabinet 2. Imagine Trajectory Score 0 : 0.6981 Imagine Trajectory Score 1 : 0.6585 Imagine Trajectory Score 2 : 0.6284 >action selected: go to cabinet 1 observation: On the cabinet 1, you see a bowl 1. Imagine Trajectory 0: take bowl 1 from cabinet 1.You pick up the bowl 1 from the cabinet 1..> think: Now I find a bowl (1). Next, I need to take it..OK..> take bowl 1 from cabinet 1.You pick up the bowl 1 from the cabinet 1. Imagine Trajectory 1: take bowl 1 from cabinet 1.You pick up the bowl 1 from the cabinet 1..> think: Now I take a bowl (1). Next, I need to go to sinkbasin 1 and clean it..OK..> go to sinkbasin 1.On the sinkbasin 1, you see nothing. Imagine Trajectory 2: take bowl 1 from cabinet 1.You pick up the bowl 1 from the cabinet 1..> think: Now I take a bowl (1). Next, I need to go to sinkbasin 1 and clean it..OK..> go to sinkbasin 1.On the sinkbasin 1, you see nothing. Imagine Trajectory Score 0 : 0.6284 Imagine Trajectory Score 1 : 0.8689 Imagine Trajectory Score 2 : 0.6417 >action selected: take bowl 1 from cabinet 1 observation: You pick up the bowl 1 from the cabinet 1. Imagine Trajectory 0: think: Now I find a bowl (1). Next, I need to take it..OK..> take bowl 1 from cabinet 1.You pick up the bowl 1 from the cabinet 1..> think: Now I take a bowl (1). Next, I need to go to sinkbasin (1) and clean it.. Imagine Trajectory 1: think: Now I take a bowl (1). Next, I need to go to sinkbasin (1) and clean it..OK..> go to sinkbasin 1.On the sinkbasin 1, you see nothing..> clean bowl 1 with sinkbasin 1.You clean the bowl 1 using the sinkbasin Imagine Trajectory 2: think: Now I find a bowl (1). Next, I need to take it..OK..> take bowl 1 from cabinet 1.Nothing happens..> think: I need to wait a while before I can find a bowl (1)..OK..> go to countertop 2.On the countertop 2, you Imagine Trajectory Score 0 : 0.1265 Imagine Trajectory Score 1 : 0.7831 Imagine Trajectory Score 2 : 0.0126 >action selected: think: Now I take a bowl (1). Next, I need to go to sinkbasin (1) and clean it. observation: OK. Imagine Trajectory 0: go to sinkbasin 1.On the sinkbasin 1, you see nothing..> clean bowl 1 with sinkbasin 1.You clean the bowl 1 using the sinkbasin 1..> think: Now I clean the bowl 1 with sinkbasin 1. Next, I need to put it in/on Imagine Trajectory 1: go to sinkbasin 1.On the sinkbasin 1, you see nothing..> clean bowl 1 with sinkbasin 1.You clean the bowl 1 using the sinkbasin 1..> think: Now I clean a bowl (1). Next, I need to put it in/on cabinet 1..OK Imagine Trajectory 2: go to sinkbasin 1.On the sinkbasin 1, you see nothing..> clean bowl 1 with sinkbasin 1.You clean the bowl 1 using the sinkbasin 1..> think: Now I clean the bowl 1 with sinkbasin 1. Next, I need to put it in/on Imagine Trajectory Score 0 : 0.6139 Imagine Trajectory Score 1 : 0.8206 Imagine Trajectory Score 2 : 0.5012 >action selected: go to sinkbasin 1 observation: On the sinkbasin 1, you see nothing. Imagine Trajectory 0: clean bowl 1 with sinkbasin 1.You clean the bowl 1 using the sinkbasin 1..> think: Now I clean the bowl 1 with sinkbasin 1. Next, I need to put it in/on cabinet 1..OK..> go to cabinet 1.The cabinet 1 is closed. Imagine Trajectory 1: clean bowl 1 with sinkbasin 1.You clean the bowl 1 using the sinkbasin 1..> think: Now I clean a bowl (1). Next, I need to put it in/on cabinet (1-6)..OK..> go to cabinet 6.The cabinet 6 is closed..> open cabinet Imagine Trajectory 2: clean bowl 1 with sinkbasin 1.You clean the bowl 1 using the sinkbasin 1..> think: Now I clean a bowl (1). Next, I need to put it in/on cabinet 1..OK..> go to cabinet 1.The cabinet 1 is closed..> open cabinet 1 Imagine Trajectory Score 0 : 0.7847 Imagine Trajectory Score 1 : 0.5952 Imagine Trajectory Score 2 : 0.7003 >action selected: clean bowl 1 with sinkbasin 1 observation: You clean the bowl 1 using the sinkbasin 1. Imagine Trajectory 0: think: Now I clean a bowl (1). Next, I need to put it in/on cabinet 1..OK..> go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a dishsponge 1 Imagine Trajectory 1: think: Now I clean a bowl (1). Next, I need to put it in/on cabinet 1..OK..> go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a bowl 1..> Imagine Trajectory 2: think: Now I clean the bowl 1 with sinkbasin 1. Next, I need to put it in/on cabinet 1..OK..> go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a Imagine Trajectory Score 0 : 0.7106 Imagine Trajectory Score 1 : 0.8489 Imagine Trajectory Score 2 : 0.8093 >action selected: think: Now I clean a bowl (1). Next, I need to put it in/on cabinet 1. observation: OK. Imagine Trajectory 0: go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a bowl 1..> put bowl 1 in/on cabinet 1.You put the bowl 1 in/on the cabinet 1..You are in Imagine Trajectory 1: go to cabinet 1.The cabinet 1 is closed..> open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a bowl 1..> think: Now I find a bowl (1). Next, I need to take it..OK..> take bowl 1 from cabinet 1 Imagine Trajectory 2: go to cabinet 1.On the cabinet 1, you see a bowl 1..> put bowl 1 in/on cabinet 1.You put the bowl 1 in/on the cabinet 1..Make sure you put items “in/on” instead of “in”. Make sure you interact with and refer to items by name and number Imagine Trajectory Score 0 : 0.7794 Imagine Trajectory Score 1 : 0.7131 Imagine Trajectory Score 2 : 0.5139 >action selected: go to cabinet 1 observation: On the cabinet 1, you see nothing. Imagine Trajectory 0: put bowl 1 in/on cabinet 1.You put the bowl 1 in cabinet 1..> think: Now I put bowl 1 in cabinet 1..OK..> go to cabinet 1.The cabinet 1 is closed..> open cabinet 1 Imagine Trajectory 1: open cabinet 1.You open the cabinet 1. The cabinet 1 is open. In it, you see a bowl 1..> take bowl 1 from cabinet 1.You pick up the bowl 1 from the cabinet 1..> think: Now I find a bowl (1). Next, I need to clean it with sinkbasin 1 Imagine Trajectory 2: put bowl 1 in/on cabinet 1.Nothing happens..> think: I need to wait a few seconds before putting it in cabinet..OK..> think: Now I put bowl 1 in/on cabinet 1..OK..> go to cabinet 1.The cabinet 1 is closed..> open cabinet 1 Imagine Trajectory Score 0 : 0.8193 Imagine Trajectory Score 1 : 0.0106 Imagine Trajectory Score 2 : 0.6214 >action selected: put bowl 1 in/on cabinet 1 observation: You put the bowl 1 in/on the cabinet 1. ## References - Asai et al. (2023) Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511, 2023. - Brohan et al. (2023) Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. - Chatzipanteli et al. (2014) Athanasia Chatzipanteli, Vasilis Grammatikopoulos, and Athanasios Gregoriadis. Development and evaluation of metacognition in early childhood education. Early Child Development and Care, 184(8):1223–1232, 2014. - Chen et al. (2017) Patricia Chen, Omar Chavez, Desmond C Ong, and Brenda Gunderson. Strategic resource use for learning: A self-administered intervention that guides self-reflection on effective resource use enhances academic performance. Psychological Science, 28(6):774–785, 2017. - Chen et al. (2022) Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022. - Cohen (2012) Marisa Cohen. The importance of self-regulation for college student learning. College Student Journal, 46(4):892–902, 2012. - Collaboration et al. (2023) Open X-Embodiment Collaboration, Abby O’Neill, Abdul Rehman, Abhinav Gupta, et al. Open X-embodiment: Robotic learning datasets and RT-X models. arXiv preprint arXiv:2310.08864, 2023. - Corbiere et al. (2019) Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure prediction by learning model confidence. Advances in Neural Information Processing Systems, 32, 2019. - Dunlosky and Bjork (2013) John Dunlosky and Robert A Bjork. Handbook of Metamemory and Memory. Psychology Press, 2013. - Dunlosky and Metcalfe (2008) John Dunlosky and Janet Metcalfe. Metacognition. Sage Publications, 2008. - Dunning (2011) David Dunning. The Dunning–Kruger effect: On being ignorant of one’s own ignorance. Advances in Experimental Social Psychology, 44:247–296, 2011. - Flavell (1979) John Flavell. Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist, 34(10):906, 1979. - Fleming (2024) Stephen M Fleming. Metacognition and confidence: A review and synthesis. Annual Review of Psychology, 75(1):241–268, 2024. - Fleming and Lau (2014) Stephen M Fleming and Hakwan C Lau. How to measure metacognition. Frontiers in Human Neuroscience, 8, 443, 2014. - Gou et al. (2023) Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738, 2023. - Grossberg (1980) Stephen Grossberg. How does a brain build a cognitive code? Psychological Review, 87:1–51, 1980. - Ha and Schmidhuber (2018) David Ha and Jürgen Schmidhuber. World Models. arXiv preprint arXiv:1803.10122, 2018. - Hafner et al. (2023) Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through World Models. arXiv preprint arXiv:2301.04104, 2023. - Hansen et al. (2022) Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang, Vikash Kumar, and Aravind Rajeswaran. MoDem: Accelerating visual model-based reinforcement learning with demonstrations. arXiv preprint arXiv:2212.05698, 2022. - Hansen et al. (2023) Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828, 2023. - Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. - Hu et al. (2023) Xinyu Hu, Pengfei Tang, Simiao Zuo, Zihan Wang, Bowen Song, Qiang Lou, Jian Jiao, and Denis Charles. Evoke: Evoking critical thinking abilities in LLMs via reviewer-author prompt editing. arXiv preprint arXiv:2310.13855, 2023. - Huang et al. (2023) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023. - Isaacson and Fujita (2006) Randy Isaacson and Frank Fujita. Metacognitive knowledge monitoring and self-regulated learning. Journal of the Scholarship of Teaching and Learning, 6(1): 39–55, 2006. - Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023. - Ketz and Pilly (2022) Nicholas A Ketz and Praveen K Pilly. Concept-modulated model-based offline reinforcement learning for rapid generalization. arXiv preprint arXiv:2209.03207, 2022. - Kolouri et al. (2020) Soheil Kolouri, Nicholas Ketz, Andrea Soltoggio, and Praveen Pilly. Sliced Cramer synaptic consolidation for preserving deeply learned representations. International Conference on Learning Representations, 2020. - Koriat (1997) Asher Koriat. Monitoring one’s own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology: General, 126(4):349, 1997. - Koul et al. (2020) Anurag Koul, Varun V. Kumar, Alan Fern, and Somdeb Majumdar. Dream and search to control: Latent space planning for continuous control. arXiv preprint arXiv:2010.09832, 2020. - Kramarski and Mevarech (2003) Bracha Kramarski and Zemira Mevarech. Enhancing mathematical reasoning in the classroom: Effects of cooperative learning and metacognitive training. American Educational Research Journal, 40, 281–310, 2003. - Kruger and Dunning (1999) Justin Kruger and David Dunning. Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6):1121, 1999. - Kudithipudi et al. (2023) Dhireesha Kudithipudi, Mario Aguilar-Simon, Jonathan Babb, et al. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4(3):196–210, 2023. - Lu et al. (2025) Xiaping Lu, Carsten Murawski, Peter Bossaerts, and Shinsuke Suzuki. Estimating self-performance when making complex decisions. Scientific Reports, 15(1), 3203, 2025. - Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-Refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. - Mandi et al. (2023) Zhao Mandi, Homanga Bharadhwaj, Vincent Moens, Shuran Song, Aravind Rajeswaran, and Vikash Kumar. CACTI: A framework for scalable multi-task multi-scene visual imitation learning. arXiv preprint arXiv:2212.05711, 2023. - Maniscalco and Lau (2012) Brian Maniscalco and Hakwan Lau. A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings. Consciousness and Cognition, 21(1), 422–430, 2025. - Mehta et al. (2024) Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, et al. OpenELM: An efficient language model family with open-source training and inference framework. arXiv preprint arXiv:2404.14619, 2024. - Metcalfe et al. (1993) Janet Metcalfe, Bennett L Schwartz, and Scott G Joaquim. The cue-familiarity heuristic in metacognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19(4):851, 1993. - Micheli et al. (2023) Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient World Models. arXiv preprint arXiv:2209.00588, 2023. - Middlebrooks et al. (2012) Paul Middlebrooks and Marc Sommer. Neuronal correlates of metacognition in primate frontal cortex. Neuron, 75(3):517–530, 2012. - Miyamoto et al. (2017) Kentaro Miyamoto, Takahiro Osada, Rieko Setsuie, Masaki Takeda, Keita Tamura, Yusuke Adachi, and Yasushi Miyashita. Causal neural network of metamemory for retrospection in primates. Science, 355(6321):188–193, 2017. - Moerland et al. (2023) Thomas Moerland, Joost Broekens, Aske Plaat, and Catholijn Jonker. Model-based reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 16(1):1–118, 2023. - Mukherjee et al. (2023) Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of GPT-4. arXiv preprint arXiv:2306.02707, 2023. - Nelson and Narens (1990) Thomas O Nelson and Louis Narens. Metamemory: A theoretical framework and new findings. Academic Press, 1990. - Open-Orca (2024) Open-Orca. Mistral-7B-OpenOrca. https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca, 2024. - Paul et al. (2023) Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01904, 2023. - Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. - Robine et al. (2023) Jan Robine, Marc Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer-based World Models are happy with 100K interactions. arXiv preprint arXiv:2303.07109, 2023. - Rostami et al. (2019) Mohammad Rostami, Soheil Kolouri, and Praveen Pilly. Complementary learning for overcoming catastrophic forgetting using experience replay. arXiv preprint arXiv:1903.04566, 2019. - Schraw (1998) Gregory Schraw. Promoting general metacognitive awareness. Instructional Science, 26:113–125, 1998. - Schraw et al. (2006) Gregory Schraw, Kent Crippen, and Kendall Hartley. Promoting self-regulation in science education: Metacognition as part of a broader perspective on learning. Research in Science Education, 36, 111–139, 2006. - SentenceTransformers (2024) SentenceTransformers. all-MiniLM-L6-v2. https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2, 2024. - Shinn et al. (2023) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652, 2023. - Shridhar et al. (2020) Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020. - Silver et al. (2016) David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016. - Silver et al. (2017) David Silver, Julian Schrittwieser, Karen Simonyan, et al. Mastering the game of Go without human knowledge. Nature, 550(7676):354–359, 2017. - Silver et al. (2018) David Silver, Thomas Hubert, Julian Schrittwieser, et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419):1140–1144, 2018. - Sutton (1991) Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bulletin, 2(4):160–163, 1991. - Team et al. (2023) Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei Zhang. Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608, 2023. - Team et al. (2021) Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, Nat McAleese, Nathalie Bradley-Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes-Fitt, Valentin Dalibard, and Wojciech Marian Czarnecki. Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808, 2021. - Wang et al. (2021) Haoxiang Wang, Han Zhao, and Bo Li. Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation. International Conference on Machine Learning, 10991–11002, 2021. - Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. - Webb et al. (2023) Taylor Webb, Kiyofumi Miyoshi, Tsz Yan So, Sivananda Rajananda, and Hakwan Lau. Natural statistics support a rational account of confidence biases. Nature Communications, 14(1), 10991–11002, 2023. - Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. - Xie et al. (2023) Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. Decomposition enhances reasoning via self-evaluation guided decoding. arXiv preprint arXiv:2305.00633, 2023. - Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. - Yao et al. (2024) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. - Young and Fry (2008) Andria Young and Jane D Fry. Metacognitive awareness and academic achievement in college students. Journal of the Scholarship of Teaching and Learning, 8(2):1–10, 2008. - Yu et al. (2020) Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-World: A benchmark and evaluation for multi-task and meta reinforcement learning. Conference on Robot Learning, 1094–1100, 2020. - Zhou et al. (2022) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022.