# Reflexion: Language Agents with Verbal Reinforcement Learning
**Authors**:
- Noah Shinn (Northeastern University)
- Federico Cassano (Northeastern University)
- Edward Berman (Northeastern University)
- Ashwin Gopinath (Massachusetts Institute of Technology)
- Karthik Narasimhan (Princeton University)
- Shunyu Yao (Princeton University)
## Abstract
Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance. We release all code, demos, and datasets at https://github.com/noahshinn024/reflexion.
## 1 Introduction
Recent works such as ReAct Yao et al., (2023), SayCan Ahn et al., (2022), Toolformer Schick et al., (2023), HuggingGPT (Shen et al.,, 2023), generative agents (Park et al.,, 2023), and WebGPT (Nakano et al.,, 2021) have demonstrated the feasibility of autonomous decision-making agents that are built on top of a large language model (LLM) core. These methods use LLMs to generate text and âactionsâ that can be used in API calls and executed in an environment. Since they rely on massive models with an enormous number of parameters, such approaches have been so far limited to using in-context examples as a way of teaching the agents, since more traditional optimization schemes like reinforcement learning with gradient descent require substantial amounts of compute and time.
In this paper, we propose an alternative approach called Reflexion that uses verbal reinforcement to help agents learn from prior failings. Reflexion converts binary or scalar feedback from the environment into verbal feedback in the form of a textual summary, which is then added as additional context for the LLM agent in the next episode. This self-reflective feedback acts as a âsemanticâ gradient signal by providing the agent with a concrete direction to improve upon, helping it learn from prior mistakes to perform better on the task. This is akin to how humans iteratively learn to accomplish complex tasks in a few-shot manner â by reflecting on their previous failures in order to form an improved plan of attack for the next attempt. For example, in figure 1, a Reflexion agent learns to optimize its own behavior to solve decision-making, programming, and reasoning tasks through trial, error, and self-reflection.
Generating useful reflective feedback is challenging since it requires a good understanding of where the model made mistakes (i.e. the credit assignment problem (Sutton and Barto,, 2018)) as well as the ability to generate a summary containing actionable insights for improvement. We explore three ways for doing this â simple binary environment feedback, pre-defined heuristics for common failure cases, and self-evaluation such as binary classification using LLMs (decision-making) or self-written unit tests (programming). In all implementations, the evaluation signal is amplified to natural language experience summaries which can be stored in long-term memory.
Reflexion has several advantages compared to more traditional RL approaches like policy or value-based learning: 1) it is lightweight and doesnât require finetuning the LLM, 2) it allows for more nuanced forms of feedback (e.g. targeted changes in actions), compared to scalar or vector rewards that are challenging to perform accurate credit assignment with, 3) it allows for a more explicit and interpretable form of episodic memory over prior experiences, and 4) it provides more explicit hints for actions in future episodes. At the same time, it does have the disadvantages of relying on the power of the LLMâs self-evaluation capabilities (or heuristics) and not having a formal guarantee for success. However, as LLM capabilities improve, we only expect this paradigm to get better over time.
We perform experiments on (1) decision-making tasks to test sequential action choices over long trajectories, (2) reasoning tasks to test knowledge-intensive, single-step generation improvement, and (3) programming tasks to teach the agent to effectively use external tools such as compilers and interpreters. Across all three types of tasks, we observe Reflexion agents are better decision-makers, reasoners, and programmers. More concretely, Reflexion agents improve on decision-making AlfWorld (Shridhar et al.,, 2021) tasks over strong baseline approaches by an absolute 22% in 12 iterative learning steps, and on reasoning questions in HotPotQA (Yang et al.,, 2018) by 20%, and Python programming tasks on HumanEval (Chen et al.,, 2021) by as much as 11%.
To summarize, our contributions are the following:
- We propose Reflexion, a new paradigm for âverbalâ reinforcement that parameterizes a policy as an agentâs memory encoding paired with a choice of LLM parameters.
- We explore this emergent property of self-reflection in LLMs and empirically show that self-reflection is extremely useful to learn complex tasks over a handful of trials.
- We introduce LeetcodeHardGym, a code-generation RL gym environment consisting of 40 challenging Leetcode questions (âhard-levelâ) in 19 programming languages.
- We show that Reflexion achieves improvements over strong baselines across several tasks, and achieves state-of-the-art results on various code generation benchmarks.
<details>
<summary>x1.png Details</summary>
### Visual Description
## Diagram: Process Flow for Three Task Types (Decision Making, Programming, Reasoning)
### Overview
The image is a structured diagram illustrating a five-stage process (Task, Trajectory, Evaluation, Reflection, Next Trajectory) applied to three distinct types of tasks: Decision making, Programming, and Reasoning. The diagram is organized as a 3-column by 5-row grid. Each column represents a task domain, and each row represents a stage in an iterative problem-solving or learning cycle. Text is presented in boxes, with some phrases highlighted in red (indicating errors or incorrect assumptions) and green (indicating corrections or correct reasoning).
### Components/Axes
* **Vertical Structure (Columns):** Three main columns, each with a numbered header at the top:
1. **1. Decision making** (Left column)
2. **2. Programming** (Center column)
3. **3. Reasoning** (Right column)
* **Horizontal Structure (Rows):** Five rows labeled on the far left:
* **(a) Task**
* **(b) Trajectory**
* **(c) Evaluation (internal / external)**
* **(d) Reflection**
* **(e) Next Trajectory**
* **Visual Cues:** Text highlighting is used consistently:
* **Red highlight:** Indicates incorrect actions, flawed logic, or failed outcomes.
* **Green highlight:** Indicates correct actions, accurate reasoning, or successful outcomes.
### Detailed Analysis
#### Column 1: Decision making
* **(a) Task:** "You are in the middle of a room [...] **Task:** clean some pan and put it in countertop."
* **(b) Trajectory:** Shows a sequence of actions and observations.
* "Action:take pan1 from stoveburner1"
* "Obs: **Nothing happens.** [...]" (Highlighted in red)
* "Action:clean pan1 with sinkbasin1"
* "Obs:Nothing happens. [...]"
* **(c) Evaluation:** "Rule/LM Heuristic: Hallucination."
* **(d) Reflection:** "[...] tried to pick up the pan in **stoveburner 1** [...] but the pan **was not in stoveburner 1.** [...]" (The first phrase is highlighted in red, the second in green).
* **(e) Next Trajectory:** "[...] **Action:** take pan 1 from **stoveburner 2** [...] **Obs:** You put the pan 1 in countertop 1." (The action is highlighted in green).
#### Column 2: Programming
* **(a) Task:** "**Task:** You are given a list of two strings [...] of open '(' or close ')' parentheses only [...]"
* **(b) Trajectory:** Shows a code snippet.
* "def match_parens(lst):"
* " if s1.count('(') + s2.count('(') == s1.count(')') + s2.count(')'): [...]"
* " **return 'No'**" (Highlighted in red).
* **(c) Evaluation:** "**Self-generated unit tests fail:** assert match_parens(...)"
* **(d) Reflection:** "[...] wrong because **it only checks if the total count of open and close parentheses is equal** [...] **order of the parentheses** [...]" (The first phrase is highlighted in red, the second in green).
* **(e) Next Trajectory:** "[...] **return 'Yes' if check(S1) or check(S2) else 'No'**" (Highlighted in green).
#### Column 3: Reasoning
* **(a) Task:** "**Task:** What profession does John Lanchester and Alan Dean Foster have in common?"
* **(b) Trajectory:** Shows a thinking process and action.
* "Think: [...] novelist, journalist, critic [...] novelist, screenwriter [...] common is **novelist and screenwriter.**" (Highlighted in red).
* "Action: "novelist, screenwriter""
* **(c) Evaluation:** "**Environment Binary Reward:** 0"
* **(d) Reflection:** "[...] failed because I incorrectly **assumed that they both had the same multiple professions** [...] **accurately identifying their professions.** [...]" (The first phrase is highlighted in red, the second in green).
* **(e) Next Trajectory:** "Think: [...] **So the profession John Lanchester and Alan Dean Foster have in common is novelist.**" (Highlighted in green).
* "Action: "novelist""
### Key Observations
1. **Consistent Process Flow:** All three columns follow the identical five-stage sequence: Task -> Trajectory (attempt) -> Evaluation (failure/error detection) -> Reflection (error analysis) -> Next Trajectory (corrected attempt).
2. **Error-Correction Pattern:** The red-to-green highlighting visually traces the path from mistake to correction in each domain.
3. **Domain-Specific Errors:**
* **Decision Making:** Error is a **spatial hallucination** (acting on an incorrect object location).
* **Programming:** Error is a **logical flaw** in the algorithm (checking only counts, not order).
* **Reasoning:** Error is a **factual assumption error** (assuming multiple common professions instead of one).
4. **Evaluation Methods:** The evaluation mechanism differs by task: a heuristic rule for decision making, unit tests for programming, and an external binary reward for reasoning.
### Interpretation
This diagram models a **meta-cognitive or self-correcting learning framework** for AI systems. It demonstrates how an agent can iteratively improve its performance across diverse task types by:
1. **Executing** an initial action plan (Trajectory).
2. **Receiving feedback** on failure (Evaluation).
3. **Performing root-cause analysis** to understand the specific nature of its error (Reflection).
4. **Generating a revised plan** that addresses the identified flaw (Next Trajectory).
The core insight is that effective learning requires not just detecting failure, but **diagnosing the type of failure** (spatial, logical, factual) to apply the correct correction. The diagram serves as a blueprint for building more robust and adaptable AI agents capable of autonomous improvement through structured reflection. The use of highlighting emphasizes the critical transition from erroneous to correct reasoning within each specialized domain.
</details>
Figure 1: Reflexion works on decision-making 4.1, programming 4.3, and reasoning 4.2 tasks.
## 2 Related work
Reasoning and decision-making
Self-Refine (Madaan et al.,, 2023) employs an iterative framework for self-refinement to autonomously improve generation through self-evaluation. These self-evaluation and self-improvement steps are conditioned on given task constraints, such as "How can this generation be written in a more positive way". Self-Refine is effective but is limited to single-generation reasoning tasks. Pryzant et al., (2023) performs a similar semantic prompt-writing optimization, but is also limited to single-generation tasks. Paul et al., (2023) fine-tune critic models to provide intermediate feedback within trajectories to improve reasoning responses. Xie et al., (2023) use stochastic beam search over actions to perform a more efficient decision-making search strategy which allows the agent to use foresight advantage due to its self-evaluation component. Yoran et al., (2023) and Nair et al., (2023) use decider models to reason over several generations. Kim et al., (2023) use a retry pattern over a fixed number of steps without an evaluation step. Goodman, (2023) perform a qualitative evaluation step that proposes optimizations to the previous generation. In this paper, we show that several of these concepts can be enhanced with self-reflection to build a persisting memory of self-reflective experiences which allows an agent to identify its own errors and self-suggest lessons to learn from its mistakes over time.
Programming
Several past and recent works employ variations of test-driven development or code debugging practices. AlphaCode (Li et al.,, 2022) evaluates a set of generations on hidden test cases. CodeT (Chen et al.,, 2022) uses self-generated unit tests that are used to score generated function implementations. Self-Debugging (Chen et al.,, 2023) employs a debugging component that is used to improve existing implementations given feedback from a code execution environment. CodeRL (Le et al.,, 2022) sets the problem in an RL framework using an actor-critic setup to debug programs given feedback from an execution environment. AlphaCode, Self-Debugging and CodeRL are effective in fixing less-complex program bugs, but they rely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement. CodeT does not access hidden test cases but does not implement a self-learning step to improve code writing.
| Approach | Self-refine | Hidden-constraints | Decision-making | Binary-reward | Memory |
| --- | --- | --- | --- | --- | --- |
| Self-refine (Madaan et al.,, 2023) | â | â | â | â | â |
| Beam search (Xie et al.,, 2023) | â | â | â | â | â |
| Reflexion (ours) | â | â | â | â | â |
| Approach Test execution | Test-execution | Debugging | Self-generated-tests | Multiple-languages | Self-reflection |
| --- | --- | --- | --- | --- | --- |
| AlphaCode (Li et al.,, 2022) | â | â | â | â | â |
| CodeT (Chen et al.,, 2022) | â | â | â | â | â |
| Self-debugging (Chen et al.,, 2023) | â | â | â | â | â |
| CodeRL (Le et al.,, 2022) | â | â | â | â | â |
| Reflexion (ours) | â | â | â | â | â |
## 3 Reflexion: reinforcement via verbal reflection
<details>
<summary>x2.png Details</summary>
### Visual Description
## [Diagram Type]: Agent Architecture Flowchart
### Overview
This image is a technical diagram illustrating the architecture of an AI agent system. It depicts the agent's internal components, their interconnections, and the agent's interaction with an external environment. The diagram uses labeled boxes, directional arrows, and color-coding to represent different functional modules and the flow of information between them.
### Components/Axes
The diagram is structured with a large outer box labeled **"Agent"** at the top center, which contains most of the system's components. Below this box, outside its boundary, is the **"Environment"**.
**Components within the "Agent" box:**
1. **Self-reflection (LM)**: A light blue, rounded rectangle located in the top-center.
2. **Evaluator (LM)**: A light blue, rounded rectangle located in the middle-left.
3. **Actor (LM)**: A light blue, rounded rectangle located in the middle-right.
4. **Experience (long-term memory)**: A yellow rectangle located to the right of the Evaluator and above the Actor.
5. **Trajectory (short-term memory)**: A yellow rectangle located in the bottom-left.
**Component outside the "Agent" box:**
6. **Environment**: A gray rectangle located at the bottom-center.
**Information Flow Labels (on arrows):**
* **External feedback**: Points from outside the Agent box to "Self-reflection (LM)".
* **Internal feedback**: Points from "Evaluator (LM)" to "Self-reflection (LM)".
* **Reflective text**: Points from "Self-reflection (LM)" to "Experience (long-term memory)".
* **Obs / Reward**: Points from "Environment" to the "Trajectory (short-term memory)" and also to the "Evaluator (LM)".
* **Action**: Points from "Actor (LM)" to "Environment".
### Detailed Analysis
The diagram defines a closed-loop system with the following information pathways:
1. **Input to Agent**: The agent receives **"Obs / Reward"** (Observation/Reward) from the **Environment**. This input is directed to two components:
* The **"Trajectory (short-term memory)"** module.
* The **"Evaluator (LM)"** module.
2. **Internal Processing Loop**:
* The **"Trajectory (short-term memory)"** provides input to both the **"Evaluator (LM)"** and the **"Actor (LM)"**.
* The **"Evaluator (LM)"** generates **"Internal feedback"** which is sent to the **"Self-reflection (LM)"** module.
* The **"Self-reflection (LM)"** module also receives **"External feedback"** from outside the agent system.
* The **"Self-reflection (LM)"** module outputs **"Reflective text"** which is stored in the **"Experience (long-term memory)"**.
3. **Action Generation**:
* The **"Actor (LM)"** module receives input from two sources: the **"Trajectory (short-term memory)"** and the **"Experience (long-term memory)"**.
* Based on these inputs, the **"Actor (LM)"** generates an **"Action"**.
4. **Output to Environment**: The **"Action"** from the **"Actor (LM)"** is sent to the **"Environment"**, completing the interaction loop.
**Color Coding:**
* **Light Blue (Rounded Rectangles)**: Denotes active processing modules powered by a Language Model (LM): Self-reflection, Evaluator, and Actor.
* **Yellow (Rectangles)**: Denotes memory stores: short-term (Trajectory) and long-term (Experience).
* **Gray (Rectangle)**: Denotes the external Environment.
### Key Observations
* The architecture explicitly separates **short-term memory (Trajectory)** from **long-term memory (Experience)**.
* There are two distinct feedback pathways influencing the agent's learning and reflection: **Internal feedback** (from the Evaluator) and **External feedback** (from outside the agent).
* The **Actor** is the sole component that directly interfaces with the Environment to produce actions, but its decisions are informed by both immediate trajectory and accumulated long-term experience.
* The **Evaluator** serves a dual role: it processes the current trajectory and environment rewards, and it provides internal feedback to the self-reflection module.
### Interpretation
This diagram represents a sophisticated cognitive architecture for an AI agent, likely designed for complex, sequential decision-making tasks. It moves beyond a simple reactive model by incorporating **self-reflection** and **memory**.
* **Learning Mechanism**: The system suggests a form of **reinforcement learning** where the agent receives rewards from the environment. However, it enhances this with a **reflective learning** loop. The agent doesn't just act; it evaluates its own trajectory, reflects on it (incorporating both internal and external feedback), and stores these reflections as long-term experiential knowledge to guide future actions.
* **Role of Language Models (LMs)**: The "(LM)" tag on the core processing modules (Self-reflection, Evaluator, Actor) indicates that these functions are implemented using Large Language Models. This implies the agent uses natural language understanding and generation for reflection, evaluation, and decision-making.
* **Human-like Cognition**: The architecture mirrors aspects of human cognition: a short-term working memory (Trajectory), a long-term memory (Experience), an evaluative conscience (Evaluator), and a reflective mind (Self-reflection) that learns from both internal and external critiques to guide behavior (Actor).
* **Purpose**: The overall goal is to create an agent that can **adapt and improve over time**. By reflecting on past actions and storing lessons as experience, the agent aims to make better decisions in future interactions with its environment, demonstrating a form of meta-cognition or learning to learn.
</details>
Algorithm 1 Reinforcement via self-reflection
Initialize Actor, Evaluator, Self-Reflection:
$M_a$ , $M_e$ , $M_sr$
Initialize policy $Ď_θ(a_i|s_i)$ , $θ=\{M_a,mem\}$
Generate initial trajectory using $Ď_θ$
Evaluate $Ď_0$ using $M_e$
Generate initial self-reflection $sr_0$ using $M_sr$
Set $memâ[sr_0]$
Set $t=0$
while $M_e$ not pass or $t<$ max trials do
Generate $Ď_t={[a_0,o_0,\dots a_i,o_i]}$ using $Ď_θ$
Evaluate $Ď_t$ using $M_e$
Generate self-reflection $sr_t$ using $M_sr$
Append $sr_t$ to $mem$
Increment $t$
end while
return
Figure 2: (a) Diagram of Reflexion. (b) Reflexion reinforcement algorithm
We develop a modular formulation for Reflexion, utilizing three distinct models: an Actor, denoted as $M_a$ , which generates text and actions; an Evaluator model, represented by $M_e$ , that scores the outputs produced by $M_a$ ; and a Self-Reflection model, denoted as $M_sr$ , which generates verbal reinforcement cues to assist the Actor in self-improvement. We provide a detailed description of each of these models and subsequently elucidate their collaborative functioning within the Reflexion framework.
Actor
The Actor is built upon a large language model (LLM) that is specifically prompted to generate the necessary text and actions conditioned on the state observations. Analogous to traditional policy-based RL setups, we sample an action or generation, $a_t$ , from the current policy $Ď_θ$ at time $t$ , receive an observation from the environment $o_t$ . We explore various Actor models, including Chain of Thought Wei et al., (2022) and ReAct Yao et al., (2023). These diverse generation models allow us to explore different aspects of text and action generation within the Reflexion framework, providing valuable insights into their performance and effectiveness. In addition, we also add a memory component mem that provides additional context to this agent. This adaption was inspired by Brooks et al., (2022), who suggest a policy iteration approach using in-context learning. Details on how this is populated are provided below.
Evaluator
The Evaluator component of the Reflexion framework plays a crucial role in assessing the quality of the generated outputs produced by the Actor. It takes as input a generated trajectory and computes a reward score that reflects its performance within the given task context. Defining effective value and reward functions that apply to semantic spaces is difficult, so we investigate several variants of the Evaluator model. For reasoning tasks, we explore reward functions based on exact match (EM) grading, ensuring that the generated output aligns closely with the expected solution. In decision-making tasks, we employ pre-defined heuristic functions that are tailored to specific evaluation criteria. Additionally, we experiment with using a different instantiation of an LLM itself as an Evaluator, generating rewards for decision-making and programming tasks. This multi-faceted approach to Evaluator design allows us to examine different strategies for scoring generated outputs, offering insights into their effectiveness and suitability across a range of tasks.
Self-reflection
The Self-Reflection model instantiated as an LLM, plays a crucial role in the Reflexion framework by generating verbal self-reflections to provide valuable feedback for future trials. Given a sparse reward signal, such as a binary success status (success/fail), the current trajectory, and its persistent memory mem, the self-reflection model generates nuanced and specific feedback. This feedback, which is more informative than scalar rewards, is then stored in the agentâs memory (mem). For instance, in a multi-step decision-making task, when the agent receives a failure signal, it can infer that a specific action $a_i$ led to subsequent incorrect actions $a_i+1$ and $a_i+2$ . The agent can then verbally state that it should have taken a different action, $a_i^\prime$ , which would have resulted in $a_i+1^\prime$ and $a_i+2^\prime$ , and store this experience in its memory. In subsequent trials, the agent can leverage its past experiences to adapt its decision-making approach at time $t$ by choosing action $a_i^\prime$ . This iterative process of trial, error, self-reflection, and persisting memory enables the agent to rapidly improve its decision-making ability in various environments by utilizing informative feedback signals.
Memory
Core components of the Reflexion process are the notion of short-term and long-term memory. At inference time, the Actor conditions its decisions on short and long-term memory, similar to the way that humans remember fine-grain recent details while also recalling distilled important experiences from long-term memory. In the RL setup, the trajectory history serves as the short-term memory while outputs from the Self-Reflection model are stored in long-term memory. These two memory components work together to provide context that is specific but also influenced by lessons learned over several trials, which is a key advantage of Reflexion agents over other LLM action choice works.
The Reflexion process
Reflexion is formalized as an iterative optimization process in 1. In the first trial, the Actor produces a trajectory $Ď_0$ by interacting with the environment. The Evaluator then produces a score $r_0$ which is computed as $r_t=M_e(Ď_0)$ . $r_t$ is only a scalar reward for trial $t$ that improves as task-specific performance increases. After the first trial, to amplify $r_0$ to a feedback form that can be used for improvement by an LLM, the Self-Reflection model analyzes the set of $\{Ď_0,r_0\}$ to produce a summary $sr_0$ which is stored in the memory mem. $sr_t$ is a verbal experience feedback for trial $t$ . The Actor, Evaluator, and Self-Reflection models work together through trials in a loop until the Evaluator deems $Ď_t$ to be correct. As mentioned in 3, the memory component of Reflexion is crucial to its effectiveness. After each trial $t$ , $sr_t$ , is appended mem. In practice, we bound mem by a maximum number of stored experiences, $Ί$ (usually set to 1-3) to adhere to max context LLM limitations.
## 4 Experiments
We evaluate various natural language RL setups on decision-making, reasoning, and code generation tasks. Specifically, we challenge an agent to perform search-based question answering on HotPotQA (Yang et al.,, 2018), multi-step tasks in common household environments in AlfWorld (Shridhar et al.,, 2021), and code writing tasks in competition-like environments with interpreters and compilers in HumanEval (Chen et al.,, 2021), MBPP (Austin et al.,, 2021), and LeetcodeHard, a new benchmark. Most notably, Reflexion improves performance over strong baselines by 22% in AlfWorld, 20% in HotPotQA, and 11% on HumanEval.
### 4.1 Sequential decision making: ALFWorld
AlfWorld is a suite of text-based environments that challenge an agent to solve multi-step tasks in a variety of interactive environments based on TextWorld (CĂ´tĂŠ et al.,, 2019). Following Yao et al., (2023), we run the agent in 134 AlfWorld environments across six different tasks, including finding hidden objects (e.g., finding a spatula in a drawer), moving objects (e.g., moving a knife to the cutting board), and manipulating objects with other objects (e.g., chilling a tomato in the fridge). We use ReAct (Yao et al.,, 2023) as the action generator as Yao et al., (2023) has shown success in long trajectory decision-making using explicit intermediate thoughts. AlfWorld tasks naturally require a self-evaluation step as the environment can only signal if a task is complete. To achieve fully autonomous behavior, we implement two self-evaluation techniques: natural language classification using an LLM and a hand-written heuristic. The heuristic is simple: if the agent executes the same action and receives the same response for more than 3 cycles, or if the number of actions taken in the current environment exceeds 30 (inefficient planning), we self-reflect. In the baseline runs, if self-reflection is suggested, we skip the self-reflection process, reset the environment, and start a new trial. In the Reflexion runs, the agent uses self-reflection to find its mistake, update its memory, reset the environment, and start a new trial. To avoid very long prompt windows that may exceed the maximum limit, we truncate the agentâs memory to the last 3 self-reflections (experiences).
To avoid syntactic errors, we provide two domain-specific few-shot trajectories to the agent. We use the same few-shot trajectory examples as Yao et al., (2023) with GPT-3 for the LLM. AlfWorld tasks, ReAct few-shot prompts, and Reflexion examples are included in the appendix.
Results
<details>
<summary>x3.png Details</summary>
### Visual Description
## [Line Chart]: (a) ALFWorld Success Rate
### Overview
The chart displays the **proportion of solved environments** (success rate) across 10 trials for three methods: *ReAct only*, *ReAct + Reflexion (Heuristic)*, and *ReAct + Reflexion (GPT)*. The x-axis represents *Trial Number* (0â10), and the y-axis represents *Proportion of Solved Environments* (0.5â1.0).
### Components/Axes
- **Title**: â(a) ALFWorld Success Rateâ
- **X-axis**: Labeled âTrial Numberâ with ticks at 0, 2, 4, 6, 8, 10 (intermediate ticks implied).
- **Y-axis**: Labeled âProportion of Solved Environmentsâ with ticks at 0.5, 0.6, 0.7, 0.8, 0.9, 1.0.
- **Legend** (top-left):
- Gray dashed line: *ReAct only*
- Blue solid line: *ReAct + Reflexion (Heuristic)*
- Green solid line: *ReAct + Reflexion (GPT)*
### Detailed Analysis (Data Points & Trends)
Approximate values (with uncertainty) for each trial:
| Trial | ReAct only (gray dashed) | ReAct + Reflexion (Heuristic) (blue) | ReAct + Reflexion (GPT) (green) |
|-------|--------------------------|--------------------------------------|---------------------------------|
| 0 | ~0.62 | ~0.62 | ~0.62 |
| 1 | ~0.70 | ~0.77 | ~0.76 |
| 2 | ~0.72 | ~0.83 | ~0.81 |
| 3 | ~0.73 | ~0.84 | ~0.82 |
| 4 | ~0.74 | ~0.87 | ~0.85 |
| 5 | ~0.75 | ~0.88 | ~0.86 |
| 6 | ~0.75 | ~0.92 | ~0.89 |
| 7 | ~0.75 (plateaus) | ~0.94 | ~0.90 |
| 8 | No data | ~0.95 | ~0.92 |
| 9 | No data | ~0.96 | ~0.94 |
| 10 | No data | ~0.97 | ~0.94 |
### Key Observations
1. **Initial Convergence**: All methods start at ~0.62 (trial 0), indicating identical initial performance.
2. **ReAct Only (Gray Dashed)**: Increases to ~0.75 by trial 7, then plateaus (no further improvement).
3. **ReAct + Reflexion (Heuristic) (Blue)**: Steepest upward trend, reaching ~0.97 by trial 10 (highest success rate).
4. **ReAct + Reflexion (GPT) (Green)**: Improves to ~0.94 by trial 10 but lags behind the Heuristic version (slower slope).
5. **Gap Widening**: The difference between *Heuristic* and *GPT* Reflexion grows over trials (e.g., trial 10: ~0.97 vs. ~0.94).
### Interpretation
- **Reflexion Improves Performance**: Adding Reflexion (Heuristic or GPT) to ReAct boosts success rates, with *Heuristic Reflexion* being more effective.
- **Heuristic vs. GPT**: The Heuristic Reflexion likely uses a more efficient reflection mechanism (e.g., rule-based) than GPT-generated reflections, leading to faster learning and higher final success.
- **ReAct Alone is Limited**: Without Reflexion, ReAct plateaus early, suggesting it struggles to improve beyond a baseline without reflective feedback.
- **Practical Implication**: For ALFWorld tasks, a heuristic-based reflection strategy paired with ReAct is optimal, outperforming both GPT-based Reflexion and ReAct alone.
This analysis captures all textual, numerical, and trend information, enabling reconstruction of the chartâs content without the image.
</details>
<details>
<summary>x4.png Details</summary>
### Visual Description
## [Line Chart]: ALFWorld Success Rate - Failure Proportions by Trial
### Overview
This is a line chart titled "(a) ALFWorld Success Rate." It plots the proportion of environments where a specific failure type occurs against the trial number for two different agent frameworks: "ReAct only" and "ReAct + Reflexion." The chart demonstrates how the frequency of two failure modesâhallucination and inefficient planningâchanges over repeated trials (0 through 10).
### Components/Axes
* **Title:** (a) ALFWorld Success Rate
* **Y-Axis:** Label: "Proportion of Environments". Scale: 0.0 to 0.5, with major gridlines at 0.1 intervals.
* **X-Axis:** Label: "Trial Number". Scale: 0 to 10, with major ticks at every even number (0, 2, 4, 6, 8, 10).
* **Legend:** Located in the top-left corner of the plot area. It defines four data series:
1. `ReAct only - hallucination`: Light gray, dashed line with circle markers.
2. `ReAct only - inefficient planning`: Dark gray, dashed line with circle markers.
3. `ReAct + Reflexion - hallucination`: Orange, solid line with circle markers.
4. `ReAct + Reflexion - inefficient planning`: Purple, solid line with circle markers.
### Detailed Analysis
**Data Series Trends and Approximate Values:**
1. **ReAct + Reflexion - hallucination (Orange, Solid Line):**
* **Trend:** Shows a steep, consistent downward slope, indicating a rapid reduction in hallucination failures over trials.
* **Data Points (Approximate):**
* Trial 0: ~0.32
* Trial 1: ~0.23
* Trial 2: ~0.16
* Trial 3: ~0.14
* Trial 4: ~0.13
* Trial 5: ~0.12
* Trial 6: ~0.08
* Trial 7: ~0.06
* Trial 8: ~0.04
* Trial 9: ~0.03
* Trial 10: ~0.03
2. **ReAct only - hallucination (Light Gray, Dashed Line):**
* **Trend:** Shows a gradual downward slope, indicating a slower reduction in hallucination failures compared to the Reflexion-augmented method. The line terminates at Trial 6.
* **Data Points (Approximate):**
* Trial 0: ~0.32
* Trial 1: ~0.27
* Trial 2: ~0.23
* Trial 3: ~0.23
* Trial 4: ~0.22
* Trial 5: ~0.21
* Trial 6: ~0.21
3. **ReAct + Reflexion - inefficient planning (Purple, Solid Line):**
* **Trend:** Starts low, drops to near zero by Trial 1, has a minor resurgence at Trials 2-3, and then remains at or near zero for all subsequent trials.
* **Data Points (Approximate):**
* Trial 0: ~0.05
* Trial 1: ~0.00
* Trial 2: ~0.01
* Trial 3: ~0.01
* Trials 4-10: ~0.00
4. **ReAct only - inefficient planning (Dark Gray, Dashed Line):**
* **Trend:** Remains relatively flat and low across all visible trials (0-6), with minor fluctuations. It does not show a clear downward trend.
* **Data Points (Approximate):**
* Trial 0: ~0.05
* Trial 1: ~0.03
* Trial 2: ~0.04
* Trial 3: ~0.04
* Trial 4: ~0.03
* Trial 5: ~0.04
* Trial 6: ~0.04
### Key Observations
1. **Initial Parity, Diverging Paths:** Both "hallucination" series start at the same high proportion (~0.32) at Trial 0. The "ReAct + Reflexion" method diverges sharply downward, while the "ReAct only" method declines much more slowly.
2. **Reflexion's Impact on Hallucination:** The most significant trend is the dramatic and sustained reduction in hallucination failures when Reflexion is added to ReAct.
3. **Inefficient Planning is a Minor Issue:** For both frameworks, the proportion of environments failing due to inefficient planning is an order of magnitude lower than hallucination failures at the start.
4. **Reflexion Eliminates Inefficient Planning:** The "ReAct + Reflexion" method reduces inefficient planning failures to effectively zero after the first few trials.
5. **Data Limitation:** The "ReAct only" data series (both failure types) are only plotted up to Trial 6, preventing comparison in later trials.
### Interpretation
This chart provides strong evidence for the efficacy of the **Reflexion** technique when added to the **ReAct** framework in the ALFWorld benchmark. The data suggests that:
* **Learning from Failure:** The primary benefit of Reflexion is its ability to help the agent learn from and correct **hallucination errors** over repeated attempts. The steep decline in the orange line indicates effective iterative improvement.
* **Problem Prioritization:** The dominant failure mode at the outset is hallucination, not planning inefficiency. The Reflexion method successfully targets and mitigates this primary bottleneck.
* **Near-Optimal Planning:** Even without Reflexion, the ReAct agent's planning is relatively efficient (low failure rate). Reflexion refines this further to a negligible level.
* **Practical Implication:** For tasks in environments like ALFWorld, incorporating a reflective, self-critiquing mechanism (Reflexion) into a reasoning-action framework (ReAct) leads to substantially more reliable performance over time, primarily by curbing the agent's tendency to generate false or unsupported information (hallucinations). The chart visually argues that Reflexion enables more effective trial-and-error learning.
</details>
Figure 3: (a) AlfWorld performance across 134 tasks showing cumulative proportions of solved tasks using self-evaluation techniques of (Heuristic) and (GPT) for binary classification. (b) Classification of AlfWorld trajectories by reason of failure.
ReAct + Reflexion significantly outperforms ReAct by completing 130 out of 134 tasks using the simple heuristic to detect hallucinations and inefficient planning. Further, ReAct + Reflexion learns to solve additional tasks by learning in 12 consecutive trials. In the ReAct-only approach, we see that performance increase halts between trials 6 and 7.
Analysis
A common error in baseline failed AlfWorld trajectories is when an agent thinks that it has possession of an item but does not actually have the item. The agent proceeds to execute several actions in a long trajectory and is not able to backtrack its actions to find the mistake. Reflexion eliminates almost all of these cases by using self-reflection to distill long, failed trajectories into relevant experiences that can are used as "self-hints" in the future. There are two main cases in which long-term memory helps an agent in AlfWorld: 1) An early mistake in a long trajectory can be easily identified. The agent can suggest a new action choice or even a new long-term plan. 2) There are too many surfaces/containers to check for an item. The agent can exploit its experience memory over several trials to thoroughly search a room. In 3, the learning curve suggests that the learning process occurs over several experiences, meaning that the agent is successfully balancing cases 1 and 2 shown in the immediate spike in the improvement between the first two trials, then a steady increase over the next 11 trials to a near-perfect performance. On the other hand, 3 shows a ReAct-only agent converging at a hallucination rate of 22% with no signs of long-term recovery.
### 4.2 Reasoning: HotpotQA
HotPotQA (Yang et al.,, 2018) is a Wikipedia-based dataset with 113k question-and-answer pairs that challenge agents to parse content and reason over several supporting documents. To test improvement in reasoning only ability, we implement Reflexion + Chain-of-Thought (CoT) (Wei et al.,, 2022) for step-by-step $Qâ A$ and $Q$ , $C_gtâ A$ implementations, where $Q$ is the question, $C_gt$ is the ground truth context from the dataset, and $A$ is the final answer. Since CoT is not a multi-step decision-making technique, we give $C_gt$ to the agent so that we can isolate the reasoning behavior over large sections of the provided text. To test holistic question and answering ability, which requires reasoning and action choice, we implement a Reflexion + ReAct (Yao et al.,, 2023) agent that can retrieve relevant context using a Wikipedia API and infer answers using step-by-step explicit thinking. For CoT implementations, we use 6-shot prompting; for ReAct, we use 2-shot prompting, and for self-reflection, we use 2-shot prompting. All examples can be found in the appendix.
Robustly evaluating natural language answers is a long-standing problem in NLP. Therefore, between trials, we use exact match answer grading using the environment to give a binary success signal to the agent. After each trial, the self-reflection loop is employed to amplify the binary signal, similar to the decision-making setup 4.1 in AlfWorld with a memory size of 3 experiences.
Results
<details>
<summary>x5.png Details</summary>
### Visual Description
## [Line Graph]: (a) HotPotQA Success Rate
### Overview
The image is a line graph titled â(a) HotPotQA Success Rate,â illustrating the **proportion of solved tasks** (y - axis) across **trial numbers** (x - axis) for four methods: *CoT only*, *ReAct only*, *CoT + Reflexion*, and *ReAct + Reflexion*.
### Components/Axes
- **Title**: â(a) HotPotQA Success Rateâ
- **Y - axis**: Labeled âProportion of Solved Tasks,â with a scale from 0.2 to 0.8 (increments of 0.2: 0.2, 0.4, 0.6, 0.8).
- **X - axis**: Labeled âTrial Number,â with ticks at 0, 2, 4, 6 (data points are plotted at trials 0â6).
- **Legend** (top - left, within the plot area):
- *CoT only*: Gray dashed line with circle markers.
- *ReAct only*: Light gray dashed line with circle markers.
- *CoT + Reflexion*: Red solid line with diamond markers.
- *ReAct + Reflexion*: Blue solid line with diamond markers.
### Detailed Analysis (Data Points & Trends)
We analyze each methodâs performance across trials (0â6):
1. **CoT only (gray dashed, circles)**:
- Trend: Flat (no improvement over trials).
- Data points (approximate): Trial 0: ~0.32; Trial 1: ~0.32; Trial 2: ~0.32; Trial 3: ~0.32; Trial 4: ~0.32; Trial 5: ~0.32; Trial 6: ~0.32.
2. **ReAct only (light gray dashed, circles)**:
- Trend: Flat (no improvement over trials).
- Data points (approximate): Trial 0: ~0.32; Trial 1: ~0.32; Trial 2: ~0.32; Trial 3: ~0.32; Trial 4: ~0.32; Trial 5: ~0.32; Trial 6: ~0.32.
3. **CoT + Reflexion (red solid, diamonds)**:
- Trend: Increases from Trial 0 to 2, then plateaus.
- Data points (approximate): Trial 0: ~0.32; Trial 1: ~0.37; Trial 2: ~0.40; Trial 3: ~0.40; Trial 4: ~0.40; Trial 5: ~0.40; Trial 6: ~0.40.
4. **ReAct + Reflexion (blue solid, diamonds)**:
- Trend: Steady increase from Trial 0 to 6 (slight plateau at Trials 5â6).
- Data points (approximate): Trial 0: ~0.32; Trial 1: ~0.42; Trial 2: ~0.48; Trial 3: ~0.50; Trial 4: ~0.52; Trial 5: ~0.54; Trial 6: ~0.54.
### Key Observations
- **Baseline Methods (CoT only, ReAct only)**: Both have a flat success rate (~0.32) across all trials, showing no improvement with more attempts.
- **Reflexion - Enhanced Methods**: Both *CoT + Reflexion* and *ReAct + Reflexion* improve over trials, with *ReAct + Reflexion* outperforming *CoT + Reflexion*.
- **Initial Performance (Trial 0)**: All methods start at the same success rate (~0.32), indicating identical initial performance.
### Interpretation
The graph demonstrates that **adding Reflexion (a reflection - based method) to CoT or ReAct improves HotPotQA task success over trials**. *ReAct + Reflexion* shows the most significant improvement, suggesting that combining ReActâs reasoning - action framework with Reflexionâs iterative reflection is highly effective. Baseline methods (without Reflexion) do not improve with more trials, highlighting Reflexionâs role in enabling learning from repeated attempts. This implies that iterative reflection helps the model refine its approach, especially when paired with ReActâs structured reasoning - action process.
</details>
<details>
<summary>x6.png Details</summary>
### Visual Description
## Line Chart: (b) HotPotQA CoT (GT)
### Overview
This is a line graph comparing the performance of two reasoning methods on the HotPotQA dataset, tracking the proportion of solved tasks across 8 sequential trials (numbered 0 to 7). The chart evaluates Chain-of-Thought (CoT) with Ground Truth (GT) alone versus CoT (GT) augmented with Reflexion.
### Components/Axes
- **X-axis**: Labeled *Trial Number*, with discrete integer markers: 0, 1, 2, 3, 4, 5, 6, 7 (positioned at the bottom center of the chart).
- **Y-axis**: Labeled *Proportion of Solved Tasks*, with a linear scale ranging from 0.4 to 1.0, marked at intervals of 0.2 (0.4, 0.6, 0.8, 1.0) on the left vertical edge.
- **Legend**: Located in the top-left corner of the chart area, with two entries:
1. *CoT (GT) only*: Represented by a light gray dashed line with light gray circle markers.
2. *CoT (GT) + Reflexion*: Represented by a dark red solid line with dark red diamond markers.
### Detailed Analysis
#### Data Series 1: CoT (GT) only (gray dashed line)
- **Trend**: Flat, with no change in performance across all trials.
- **Approximate Data Points**:
- Trial 0: ~0.61
- Trial 1: ~0.61
- Trial 2: ~0.61
- Trial 3: ~0.61
- Trial 4: ~0.61
- Trial 5: ~0.61
- Trial 6: ~0.61
- Trial 7: ~0.61
#### Data Series 2: CoT (GT) + Reflexion (red solid line)
- **Trend**: Sharp increase from trial 0 to 1, followed by a gradual upward trend through trial 7, with a plateau in the final two trials.
- **Approximate Data Points**:
- Trial 0: ~0.61 (matches the CoT (GT) only value)
- Trial 1: ~0.70
- Trial 2: ~0.70
- Trial 3: ~0.71
- Trial 4: ~0.73
- Trial 5: ~0.75
- Trial 6: ~0.76
- Trial 7: ~0.76
### Key Observations
1. At trial 0, both methods have identical performance (~0.61 proportion of solved tasks).
2. The CoT (GT) + Reflexion method shows a significant performance jump between trial 0 and 1, then continues to improve gradually over subsequent trials.
3. The CoT (GT) only method shows no performance improvement across all 8 trials, remaining static at ~0.61.
4. The performance gap between the two methods widens over trials, starting at 0 and growing to ~0.15 by trial 7.
### Interpretation
This chart demonstrates that adding Reflexion to CoT (GT) enables meaningful performance gains on HotPotQA tasks over repeated trials, whereas CoT (GT) alone yields no improvement with additional attempts. The initial jump in performance for the Reflexion-augmented method suggests that it allows the model to learn or adapt from the first trial, leading to sustained, incremental improvements. The plateau in performance after trial 6 indicates that the model may have reached a performance ceiling for this setup, or that further trials yield diminishing returns. Overall, Reflexion appears to be an effective enhancement for CoT-based reasoning on this dataset, as it unlocks iterative improvement that is absent in the baseline CoT (GT) approach.
</details>
<details>
<summary>x7.png Details</summary>
### Visual Description
## Line Chart: HotPotQA Episodic Memory Performance
### Overview
This is a line chart titled "(c) HotPotQA Episodic Memory." It plots the performance of three different methods on a task over a series of trials. The performance metric is the proportion of tasks solved, and the chart shows how this proportion changes from trial 0 to trial 4.
### Components/Axes
* **Title:** "(c) HotPotQA Episodic Memory"
* **Y-axis:** Label is "Proportion of Solved Tasks". Scale ranges from 0.5 to 1.0, with major gridlines at 0.1 intervals (0.5, 0.6, 0.7, 0.8, 0.9, 1.0).
* **X-axis:** Label is "Trial Number". Discrete markers at integers 0, 1, 2, 3, and 4.
* **Legend:** Positioned in the top-left quadrant of the chart area. Contains three entries:
1. `CoT (GT) only`: Represented by a gray, dashed line with circular markers.
2. `CoT (GT) EPM`: Represented by a pink, dashed line with circular markers.
3. `CoT (GT) EPM + Reflexion`: Represented by a purple, solid line with diamond markers.
### Detailed Analysis
**Trend Verification & Data Points (Approximate Values):**
1. **CoT (GT) only (Gray, dashed line):**
* **Trend:** The line is essentially flat, showing no improvement across trials.
* **Data Points:** The proportion of solved tasks remains constant at approximately **0.61** for all trials (0 through 4).
2. **CoT (GT) EPM (Pink, dashed line):**
* **Trend:** The line shows an initial increase from trial 0 to trial 1 and then plateaus.
* **Data Points:**
* Trial 0: ~0.62
* Trial 1: ~0.66
* Trials 2, 3, 4: ~0.66 (plateau)
3. **CoT (GT) EPM + Reflexion (Purple, solid line):**
* **Trend:** The line shows a steady upward trend from trial 0 to trial 3, after which it plateaus. It consistently outperforms the other two methods.
* **Data Points:**
* Trial 0: ~0.63
* Trial 1: ~0.70
* Trial 2: ~0.72
* Trial 3: ~0.74
* Trial 4: ~0.74 (plateau)
### Key Observations
* **Performance Hierarchy:** The method `CoT (GT) EPM + Reflexion` achieves the highest proportion of solved tasks at every trial point after trial 0. `CoT (GT) EPM` performs better than the baseline `CoT (GT) only` but worse than the version with Reflexion.
* **Learning Curves:** The baseline (`CoT (GT) only`) shows no learning. The `CoT (GT) EPM` method shows a one-time improvement after the first trial. The `CoT (GT) EPM + Reflexion` method shows continuous improvement over the first three trials.
* **Convergence:** Both methods that include EPM (with or without Reflexion) appear to reach a performance plateau by trial 3 or 4.
### Interpretation
The chart demonstrates the incremental value of adding episodic memory (EPM) and a reflection mechanism (Reflexion) to a Chain-of-Thought (CoT) reasoning system on the HotPotQA benchmark.
* **Baseline Limitation:** The flat line for `CoT (GT) only` suggests that simply using Chain-of-Thought with ground truth (GT) information does not allow the system to learn or improve from repeated trials on similar tasks.
* **Value of Episodic Memory:** The improvement shown by `CoT (GT) EPM` indicates that incorporating episodic memory enables the system to retain and leverage information from previous trials, leading to a performance boost after the first experience.
* **Synergy with Reflexion:** The superior and continuously improving performance of `CoT (GT) EPM + Reflexion` suggests that combining episodic memory with a reflective process creates a powerful synergy. The system not only remembers past episodes but also actively reasons about them, leading to more effective learning and problem-solving over multiple attempts. The plateau after trial 3 may indicate that the system has reached the limit of its learning capacity for this specific task setup or that the remaining unsolved tasks are of a different, more difficult nature.
</details>
Figure 4: Chain-of-Thought (CoT) and ReAct. Reflexion improves search, information retrieval, and reasoning capabilities on 100 HotPotQA questions. (a) Reflexion ReAct vs Reflexion CoT (b) Reflexion CoT (GT) for reasoning only (c) Reflexion vs episodic memory ablation.
Reflexion outperforms all baseline approaches by significant margins over several learning steps. Furthermore, ReAct-only, CoT-only, and CoT (GT)-only implementations fail to probabilistically improve on any tasks, meaning that no failed tasks from the first trial from any of the baseline approaches were able to be solved in subsequent trials using a temperature of 0.7 In the Reflexion runs, we allowed the agent to gather experience and retry on failed tasks until it produced 3 consecutive failed attempts on the particular task. Naturally, the CoT (GT) achieved higher accuracy scores as it was given access to the ground truth context of the question. Still, the CoT (GT) agent is unable to correctly infer the correct answer for 39% of the questions, but Reflexion helps the agent to correct its mistakes without access to the ground truth answer to improve its accuracy by 14%.
Analysis
We perform an ablation experiment to isolate the advantage of the self-reflective step for reasoning using CoT (GT) as the baseline approach 4. Recall that CoT (GT) uses Chain-of-Thought reasoning with provided ground truth context, which tests reasoning ability over long contexts. Next, we add an element of episodic memory (EPM) by including the most recent trajectory. For the Reflexion agent, we implement the standard self-reflection step as a final pass. Intuitively, we test if the agent is iteratively learning more effectively by using verbal explanation using language written in the first person. 4 shows that self-reflection improves learning by an 8% absolute boost over the episodic memory learning advantage. This result supports the argument that refinement-only approaches are not as effective as self-reflection-guided refinement approaches.
### 4.3 Programming
We evaluate the baseline and Reflexion approaches on Python and Rust code writing on MBPP (Austin et al.,, 2021), HumanEval (Chen et al.,, 2021), and LeetcodeHardGym, our new dataset. MBPP and HumanEval measure function body generation accuracy given natural language descriptions. We use a benchmark language compiler, MultiPL-E (Cassano et al.,, 2022), to translate subsets of HumanEval and MBPP to the Rust language. MultiPL-E is a collection of small compilers that can be used to translate Python benchmark questions to 18 other languages. We include experiments for Rust code generation to demonstrate that Reflexion implementations for code generation are language-agnostic and can be used for interpreted and compiled languages. Lastly, we introduce a new benchmark, LeetcodeHardGym, which is an interactive programming gym that contains 40 Leetcode hard-rated questions that have been released after October 8, 2022, which is the pre-training cutoff date of GPT-4 (OpenAI,, 2023).
The task of programming presents a unique opportunity to use more grounded self-evaluation practices such as self-generated unit test suites. Thus, our Reflexion-based programming task implementation is eligible for pass@1 accuracy reporting. To generate a test suite, we use Chain-of-Thought prompting (Wei et al.,, 2022) to produce diverse, extensive tests with corresponding natural language descriptions. Then, we filter for syntactically valid test statements by attempting to construct a valid abstract syntax tree (AST) for each proposed test. Finally, we sample $n$ tests from the collection of generated unit tests to produce a test suite $T$ , denoted as $\{t_0,t_1,\dots,t_n\}$ . We set $n$ to a maximum of 6 unit tests. Aside from the unit test suite component, the setup for the learning loop for a Reflexion programming agent is identical to the reasoning and decision-making agents with a max memory limit of 1 experience.
Results
| Benchmark + Language HumanEval (PY) HumanEval (RS) | Prev SOTA Pass@1 65.8 (CodeT (Chen et al.,, 2022) + GPT-3.5) â | SOTA Pass@1 80.1 (GPT-4) 60.0 (GPT-4) | Reflexion Pass@1 91.0 68.0 |
| --- | --- | --- | --- |
| MBPP (PY) | 67.7 (CodeT (Chen et al.,, 2022) + Codex (Chen et al.,, 2021)) | 80.1 (GPT-4) | 77.1 |
| MBPP (RS) | â | 70.9 (GPT-4) | 75.4 |
| Leetcode Hard (PY) | â | 7.5 (GPT-4) | 15.0 |
Table 1: Pass@1 accuracy for various model-strategy-language combinations. The base strategy is a single code generation sample. All instruction-based models follow zero-shot code generation.
| HumanEval (PY) | 0.80 | 0.91 | 0.99 | 0.40 | 0.01 | 0.60 |
| --- | --- | --- | --- | --- | --- | --- |
| MBPP (PY) | 0.80 | 0.77 | 0.84 | 0.59 | 0.16 | 0.41 |
| HumanEval (RS) | 0.60 | 0.68 | 0.87 | 0.37 | 0.13 | 0.63 |
| MBPP (RS) | 0.71 | 0.75 | 0.84 | 0.51 | 0.16 | 0.49 |
Table 2: Overall accuracy and test generation performance for HumanEval and MBPP. For Rust, HumanEval is the hardest 50 problems from HumanEval Python translated to Rust with MultiPL-E (Cassano et al.,, 2022). TP: unit tests pass, solution pass; FN: unit tests fail, solution pass; FP: unit tests pass, solution fail; TN: unit tests fail, solution fail.
Reflexion outperforms all baseline accuracies and sets new state-of-the-art standards on all benchmarks for Python and Rust except for MBPP Python 1. We further investigate the inferior performance of Reflexion on MBPP Python.
Analysis
We acknowledge that self-reflecting code-generation agents are bound to their ability to write diverse, comprehensive tests. Therefore, in the case in which the model generates a flaky test suite, it is possible that all tests pass on an incorrect solution and lead to a false positive label on a code completion (Lam et al.,, 2020). On the other hand, if the model produces an incorrectly written test suite, it is possible for some of the tests to fail on a correct solution, leading to a self-reflection generation that is conditioned on a false negative code completion. Given the implementation of Reflexion, false negatives are preferred over false positives as the agent may be able to use self-reflection to identify the incorrect test(s) and prompt itself to keep the original code completion intact. On the other hand, if an invalid test suite returns a false positive completion (all internal test cases pass but the implementation is incorrect), the agent will prematurely report an invalid submission. In 2, various conditions are measured to analyze performance beyond pass@1 accuracy. Previously, we displayed the inferior performance of Reflexion to the baseline GPT-4 on MBPP Python. In 2, we observe a notable discrepancy between the false positive labels produced by internal test execution, P(not pass@1 generation correct | tests pass). That is, the probability that a submission will fail given that it passes all unit tests. For HumanEval and MBPP Python, the baseline pass@1 accuracies are relatively similar, 82% and 80%, respectively. However, the false positive test execution rate for MBPP Python is 16.3% while the rate for HumanEval Python is a mere 1.4%, leading to 91% overall accuracy 1.
Ablation study
| Base model | False | False | 0.60 |
| --- | --- | --- | --- |
| Test generation omission | False | True | 0.52 |
| Self-reflection omission | True | False | 0.60 |
| Reflexion | True | True | 0.68 |
Table 3: Pass@1 accuracy for various compromised approaches on the Reflexion approach using GPT-4 as the base model on HumanEval Rust - 50 hardest problems
We test the composite approach of Reflexion for test generation and self-reflection cooperation on a subset of the 50 hardest HumanEval Rust problems. Our Rust compiler environment provides verbose error logs and helpful debugging hints, therefore serving as a good playground for compromised approaches. First, we omit internal test generation and execution steps, which test the agent to self-reflect without guidance from current implementations. 3 shows an inferior 52% vs 60% (baseline) accuracy, which suggests that the agent is unable to determine if the current implementation is correct without unit tests. Therefore, the agent must participate in all iterations of the run without the option to return early, performing harmful edits to the implementation.
Next, we test self-reflection contribution by omitting the natural language explanation step following failed unit test suite evaluations. Intuitively, this challenges the agent to combine the tasks of error identification and implementation improvement across all failed unit tests. Interestingly, the compromised agent does not improve performance over the baseline run. We observe that the test generation and code compilation steps are able to catch syntax and logic errors, but the implementation fixes do not reflect these indications. These empirical results suggest that several recent works that propose blind trial and error debugging techniques without self-reflection are ineffective on harder tasks such as writing complex programs in Rust.
## 5 Limitations
At its core, Reflexion is an optimization technique that uses natural language to do policy optimization. Policy optimization is a powerful approach to improve action choice through experience, but it may still succumb to non-optimal local minima solutions. In this study, we limit long-term memory to a sliding window with maximum capacity, but we encourage future work to extend the memory component of Reflexion with more advanced structures such as vector embedding databases or traditional SQL databases. Specific to code generation, there are many practical limitations to test-driven development in specifying accurate input-output mappings such as non-deterministic generator functions, impure functions that interact with APIs, functions that vary output according to hardware specifications, or functions that invoke parallel or concurrent behavior that may be difficult to predict.
## 6 Broader impact
Large language models are increasingly used to interact with external environments (e.g. the Internet, software, robotics, etc.) and humans. Our work has the potential of reinforcing and empowering these agents toward greater automation and work efficiency, but it also amplifies the risks when these agents were put into misuse. We believe that this direction of research will need more effort in safety and ethical considerations.
On the other hand, reinforcement learning has suffered from its black-box policy and optimization setups in which interpretability and alignment have been challenging. Our proposed âverbalâ reinforcement learning might address some of the issues and turn autonomous agents more interpretable and diagnosable. For example, in the case of tool-usage that may be too hard for humans to understand, self-reflections could be monitored to ensure proper intent before using the tool.
## 7 Conclusion
In this work, we present Reflexion, an approach that leverages verbal reinforcement to teach agents to learn from past mistakes. We empirically show that Reflexion agents significantly outperform currently widely-used decision-making approaches by utilizing self-reflection. In future work, Reflexion could be used to employ more advanced techniques that have been thoroughly studied in traditional RL settings, such as value learning in natural language or off-policy exploration techniques.
## 8 Reproducibility
We highly advise others to use isolated execution environments when running autonomous code writing experiments as the generated code is not validated before execution.
## References
- Ahn et al., (2022) Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., et al. (2022). Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691.
- Austin et al., (2021) Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021). Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
- Brooks et al., (2022) Brooks, E., Walls, L., Lewis, R. L., and Singh, S. (2022). In-context policy iteration. arXiv preprint arXiv:2210.03821.
- Cassano et al., (2022) Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., Guha, A., Greenberg, M., and Jangda, A. (2022). Multipl-e: A scalable and extensible approach to benchmarking neural code generation.
- Chen et al., (2022) Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., and Chen, W. (2022). Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397.
- Chen et al., (2021) Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
- Chen et al., (2023) Chen, X., Lin, M., Schärli, N., and Zhou, D. (2023). Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128.
- CĂ´tĂŠ et al., (2019) CĂ´tĂŠ, M.-A., KĂĄdĂĄr, A., Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J., Hausknecht, M., El Asri, L., Adada, M., et al. (2019). Textworld: A learning environment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Conjunction with the 27th International Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers 7, pages 41â75. Springer.
- Goodman, (2023) Goodman, N. (2023). Meta-prompt: A simple self-improving language agent. noahgoodman.substack.com.
- Kim et al., (2023) Kim, G., Baldi, P., and McAleer, S. (2023). Language models can solve computer tasks. arXiv preprint arXiv:2303.17491.
- Lam et al., (2020) Lam, W., Winter, S., Wei, A., Xie, T., Marinov, D., and Bell, J. (2020). A large-scale longitudinal study of flaky tests. Proc. ACM Program. Lang., 4(OOPSLA).
- Le et al., (2022) Le, H., Wang, Y., Gotmare, A. D., Savarese, S., and Hoi, S. C. H. (2022). Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314â21328.
- Li et al., (2023) Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023). Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161.
- Li et al., (2022) Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022). Competition-level code generation with alphacode. Science, 378(6624):1092â1097.
- Madaan et al., (2023) Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023). Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651.
- Nair et al., (2023) Nair, V., Schumacher, E., Tso, G., and Kannan, A. (2023). Dera: Enhancing large language model completions with dialog-enabled resolving agents. arXiv preprint arXiv:2303.17071.
- Nakano et al., (2021) Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021). Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332.
- OpenAI, (2023) OpenAI (2023). Gpt-4 technical report. ArXiv.
- Park et al., (2023) Park, J. S., OâBrien, J. C., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. (2023). Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442.
- Paul et al., (2023) Paul, D., Ismayilzada, M., Peyrard, M., Borges, B., Bosselut, A., West, R., and Faltings, B. (2023). Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01904.
- Pryzant et al., (2023) Pryzant, R., Iter, D., Li, J., Lee, Y. T., Zhu, C., and Zeng, M. (2023). Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03495.
- Schick et al., (2023) Schick, T., Dwivedi-Yu, J., DessĂŹ, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. (2023). Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761.
- Shen et al., (2023) Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. (2023). Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580.
- Shridhar et al., (2021) Shridhar, M., Yuan, X., CĂ´tĂŠ, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. (2021). ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR).
- Sutton and Barto, (2018) Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction. The MIT Press, second edition.
- Wei et al., (2022) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. (2022). Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903.
- Xie et al., (2023) Xie, Y., Kawaguchi, K., Zhao, Y., Zhao, X., Kan, M.-Y., He, J., and Xie, Q. (2023). Decomposition enhances reasoning via self-evaluation guided decoding. arXiv preprint arXiv:2305.00633.
- Yang et al., (2018) Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. (2018). HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
- (29) Yao, S., Chen, H., Yang, J., and Narasimhan, K. (preprint). Webshop: Towards scalable real-world web interaction with grounded language agents. In ArXiv.
- Yao et al., (2023) Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
- Yoran et al., (2023) Yoran, O., Wolfson, T., Bogin, B., Katz, U., Deutch, D., and Berant, J. (2023). Answering questions by meta-reasoning over multiple chains of thought. arXiv preprint arXiv:2304.13007.
## Appendix A Evaluation with additional models
We further investigated the applicability of trial-and-error problem-solving with models of various strengths. We found that the ability to specify self-corrections is an emergent quality of stronger, larger models.
| Baseline | 0.26 | 0.00481 |
| --- | --- | --- |
| Reflexion | 0.26 | 0.00305 |
Table 4: Pass@1 accuracy on HumanEval Python using starchat-beta (Li et al.,, 2023).
| CoT (GT) + text-davinci-003 | 0.60 | 0.77 |
| --- | --- | --- |
| CoT (GT) + gpt-3.5-turbo | 0.57 | 0.71 |
| CoT (GT) + gpt-4 | 0.68 | 0.80 |
| ReAct + text-davinci-003 | 0.30 | 0.55 |
| ReAct + gpt-3.5-turbo | 0.26 | 0.38 |
| ReAct + gpt-4 | 0.39 | 0.51 |
Table 5: Pass@1 accuracy on 100 HotPotQA using various models.
## Appendix B Decision-making
Environment: You are in the middle of a room. Looking quickly around you, you see a bed 1, a desk 2, a desk 1, a drawer 6, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a garbagecan 1, a laundryhamper 1, a safe 1, a shelf 6, a shelf 5, a shelf 4, a shelf 3, a shelf 2, and a shelf 1. Task: examine the mug with the desklamp.
Trial #1 $>$ think: To solve the task, I need to find and take a mug, then find and use a desklamp. $>$ think: First I need to find a mug. A mug is more likely to appear in drawer (1-6), desk (1-2), shelf (1-6), garbagecan (1), laundryhamper (1). I can check one by one, starting with drawer 1. $>$ go to drawer 1 The drawer 1 is closed. ⌠$>$ open drawer 6 You open the drawer 6. The drawer 6 is open. In it, you see a keychain 2. $>$ go to desk 1On the desk 1, you see a creditcard 3, a desklamp 1, a laptop 2, a mug 1, a pen 1, and a pencil 1. $>$ think: Now I find a mug (1). Next, I need to take it. $>$ take mug 1 from desk 1You pick up the mug 1 from the desk 1. $>$ think: Now I take a mug (1). Next, I need to find a desklamp. A desklamp is more likely to appear in desk (1-2), sidetable (1-2), shelf (1-6), bed (1), drawer (1-6). I can check one by one, starting with desk 1. $>$ go to desk 1 Nothing happens. $>$ go to desk 2 On the desk 2, you see a alarmclock 1, a bowl 1, a mug 3, a pencil 3, and a pencil 2. $>$ think: Now I find a desklamp (1). Next, I need to use it. $>$ use desklamp 1Nothing happens. $>$ use desklamp 1Nothing happens.Status: Fail
Reflection: In this environment, my plan was to find a mug then find and use a desklamp. However, the task says to examine the mug with the desklamp. I should have looked for the desklamp first, then looked for the mug. I noticed that the desklamp was found on desk 1. In the next trial, I will go to desk 1, find the lamp, then look for the mug and examine it with the desklamp.
Trial #2 $>$ go to desk 1On the desk 1, you see a creditcard 3, a desklamp 1, a laptop 2, a mug 1, a pen 1, and a pencil 1. $>$ think: To solve the task, I need to find and take a mug, then find and use a desklamp. $>$ take mug 1 from desk 1You pick up the mug 1 from the desk 1. $>$ think: To solve the task, I need to find and take a mug, then find and use a desklamp. $>$ use desklamp 1You turn on the desklamp 1.Status: Success
Figure 5: [Top] An AlfWorld trajectory in which the agent failed due to inefficient planning. In the reflection, the agent recognizes that it should have looked for the desklamp then the mug, not the mug then the desklamp. [Bottom] The agent is able to correct its reasoning trace and execute a sequence of actions in a concise manner.
### B.1 WebShop Limitation
In 5, we briefly state that Reflexion struggles to overcome local minima choices that require extremely creative behavior to escape. We observe this shortcoming in an experiment on WebShop (Yao et al., rint, ). WebShop is a web-based problem-solving benchmark that tests agents to navigate an e-commerce website to locate and purchase products given requests from clients. We test a two-shot ReAct + Reflexion agent in 100 environments. However, after only four trials, we terminate the runs as the agent does not show signs of improvement 6. Further, the agent does not generate helpful, intuitive self-reflections after failed attempts. We conclude that Reflexion is unable to solve tasks that require a significant amount of diversity and exploration. In AlfWorld, the agent is able to adequately explore new environments because the permissible actions can be seen in the observations. In HotPotQA, the agent faces a similar WebShop search query task but is more successful as the search space for Wikipedia articles is more diverse and requires less precise search queries. A common problem for e-commerce search engines is properly handling ambiguity in natural language search interpretations. Thus, WebShop presents a task that requires very diverse and unique behavior from a Reflexion agent.
<details>
<summary>x8.png Details</summary>
### Visual Description
## Line Chart: WebShop Success Rate
### Overview
This is a line chart titled "WebShop Success Rate" that compares the performance of two methods, "ReAct only" and "ReAct + Reflexion," across a series of trials. The chart plots the proportion of solved environments against the trial number, showing how success rates evolve over repeated attempts.
### Components/Axes
* **Title:** "WebShop Success Rate" (centered at the top).
* **Y-axis:** Labeled "Proportion of Solved Environments." The scale runs from 0.10 to 0.50, with major tick marks at intervals of 0.05 (0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50).
* **X-axis:** Labeled "Trial Number." The scale runs from 0.0 to 3.0, with major tick marks at intervals of 0.5 (0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0).
* **Legend:** Located in the top-left corner of the plot area.
* **ReAct only:** Represented by a gray, dashed line with circular markers.
* **ReAct + Reflexion:** Represented by a solid blue line with circular markers.
* **Grid:** A light gray grid is present in the background, aligned with the major tick marks on both axes.
### Detailed Analysis
The chart displays two data series, each with four data points corresponding to trial numbers 0.0, 1.0, 2.0, and 3.0.
**Data Series 1: ReAct only (Gray, dashed line)**
* **Trend:** The line shows a very slight upward slope from trial 0.0 to 1.0, after which it plateaus.
* **Data Points:**
* Trial 0.0: ~0.33
* Trial 1.0: ~0.34
* Trial 2.0: ~0.34
* Trial 3.0: ~0.34
**Data Series 2: ReAct + Reflexion (Blue, solid line)**
* **Trend:** The line shows a clear upward slope from trial 0.0 to 1.0, after which it plateaus at a higher level than the "ReAct only" series.
* **Data Points:**
* Trial 0.0: ~0.33 (appears to start at the same point as the gray line)
* Trial 1.0: ~0.35
* Trial 2.0: ~0.35
* Trial 3.0: ~0.35
### Key Observations
1. **Initial Parity:** Both methods begin with an identical success rate of approximately 0.33 at Trial 0.0.
2. **Divergence:** After the first trial, the "ReAct + Reflexion" method shows a clear improvement, reaching a success rate of ~0.35, while the "ReAct only" method shows minimal improvement to ~0.34.
3. **Plateau:** Both methods reach their peak performance by Trial 1.0 and maintain that exact level of performance through Trials 2.0 and 3.0. No further improvement is observed in later trials for either method.
4. **Consistent Advantage:** The "ReAct + Reflexion" method maintains a consistent, albeit small, advantage over the "ReAct only" method from Trial 1.0 onward.
### Interpretation
The data suggests that integrating "Reflexion" with the "ReAct" method provides a measurable, though modest, benefit in solving WebShop environments. The key finding is that this benefit is realized early (by the first trial) and is sustained, but not compounded, over subsequent trials.
The plateau for both methods indicates that additional trials beyond the first one do not lead to further learning or improvement in success rate under the tested conditions. This could imply that the agents quickly reach their performance ceiling for the given task or that the evaluation metric (proportion solved) is not sensitive enough to capture finer-grained improvements after the initial attempt.
The primary value of the "Reflexion" component appears to be in enabling a slightly higher initial learning or adaptation rate, leading to a better stable performance level. The chart does not show evidence of catastrophic forgetting or performance degradation over multiple trials for either method.
</details>
Figure 6: Reflexion vs React performance on WebShop across 100 customer shopping requests. ReAct + Reflexion fails to significantly outperform ReAct.
## Appendix C Programming
Programming LLM calls require strict instructions to produce function bodies only, due to the extensive dialogue training of the LLMs. A few programming examples are reported below with instructions highlighted in blue and templates. See the full implementation at https://github.com/noahshinn024/reflexion.
### C.1 Programming function implementation example (HumanEval Python)
Sample function signature:
âŹ
1 def minSubArraySum (nums):
2 "" "
3 Given an array of integers nums, find the minimum sum of any
4 non-empty sub-array of nums.
5 Example
6 minSubArraySum([2, 3, 4, 1, 2, 4]) == 1
7 minSubArraySum([-1, -2, -3]) == -6
8 " ""
Sample function body implementation:
âŹ
1 min_sum = float (âinfâ)
2 for i in range (len (nums)):
3 current_sum = 0
4 for j in range (i, len (nums)):
5 current_sum += nums [j]
6 if current_sum < min_sum:
7 min_sum = current_sum
8 return min_sum
### C.2 Reflexion Actor instruction
You are a Python writing assistant. You will be given your previous implementation of a function, a series of unit tests results, and your self-reflection on your previous implementation. Apply the necessary changes below by responding only with the improved body of the function. Do not include the signature in your response. The first line of your response should have 4 spaces of indentation so that it fits syntactically with the user provided signature. You will be given a few examples by the user.
Reflexion Actor generations follow the form: (Instruction) (Function implementation) (Unit test feedback) (Self-reflection) (Instruction for next function implmentation)
Full examples are shown in https://github.com/noahshinn024/reflexion
### C.3 Reflexion Self-reflection instruction and example
You are a Python writing assistant. You will be given your previous implementation of a function, a series of unit tests results, and your self-reflection on your previous implementation. Apply the necessary changes below by responding only with the improved body of the function. Do not include the signature in your response. The first line of your response should have 4 spaces of indentation so that it fits syntactically with the user provided signature. You will be given a few examples by the user. Reflexion Self-Reflection generations follow the form: (Instruction) (Function implementation) (Unit test feedback)
### C.4 Reflexion programming no Self-Reflection ablation example
Reflexion no Self-Reflection ablation Actor generations follow the form: (Instruction) (Function implementation) (Unit test feedback) (Self-reflection) (Instruction for next function implmentation)
### C.5 Reflexion programming no test generation ablation example
Reflexion no test generation ablation Actor generations follow the form: (Instruction) (Function implementation) (Unit test feedback) (Self-reflection) (Instruction for next function implmentation)
## Appendix D Reasoning
### D.1 Full example
Trial #1: Question: Grown-Ups starred the actor who was best known for which role on "âAllo âAllo!"? Thought 1: I need to search Grown-Ups and "âAllo âAllo!", find the actor who starred in Grown-Ups, then find the role he was best known for in "âAllo âAllo!". Action 1: Search[Grown-Ups] Observation 1: Grown-Ups is a 1980 British BBC television film devised and directed by Mike Leigh. It stars Lesley Manville, Philip Davis, Brenda Blethyn, Janine Duvitski, Lindsay Duncan and Sam Kelly. It was edited by Robin Sales and produced by Louis Marks for the BBC, and originally shown on BBC 2 on 28 November 1980. Thought 2: Grown-Ups starred Lesley Manville, Philip Davis, Brenda Blethyn, Janine Duvitski, Lindsay Duncan and Sam Kelly. I need to search "âAllo âAllo!" and find which actor was best known for which role. Action 2: Search["âAllo âAllo!"] Observation 2: Could not find ["âAllo âAllo!"]. Similar: ["âAllo âAllo!", "List of âAllo âAllo! characters", âGorden Kayeâ, "âAllo âAllo! (series 2)", âAllĂ´, allĂ´â, âFrancesca Gonshawâ, "âAllo âAllo! (series 4)", âKim Hartmanâ, "âAllo âAllo! (series 6)", âJeremy Lloydâ] Thought 3: To find the actor who was best known for which role on "âAllo âAllo!", I can search Gorden Kaye. Action 3: Search[Gorden Kaye] Observation 3: Gordon Irving Kaye (7 April 1941 â 23 January 2017), known professionally as Gorden Kaye, was an English actor, best known for playing womanising cafĂŠ owner RenĂŠ Artois in the television comedy series âAllo âAllo!. Thought 4: Gorden Kaye was best known for playing womanising cafĂŠ owner RenĂŠ Artois in the television comedy series "âAllo âAllo!". So the answer is Rene Artois. Action 4: Finish[Rene Artois]Observation 4: Answer is INCORRECT
Trial #2: Question: Grown-Ups starred the actor who was best known for which role on "âAllo âAllo!"? Thought 1: I need to search the actor who was best known for a role on "âAllo âAllo!" and find out what role they were best known for. Action 1: Search[Grown-Ups] Observation 1: Grown-Ups is a 1980 British BBC television film devised and directed by Mike Leigh. It stars Lesley Manville, Philip Davis, Brenda Blethyn, Janine Duvitski, Lindsay Duncan and Sam Kelly. It was edited by Robin Sales and produced by Louis Marks for the BBC, and originally shown on BBC 2 on 28 November 1980. Thought 2: The paragraph does not mention the actor who was best known for a role on "âAllo âAllo!". I need to search the actorâs name instead. Action 2: Search[Sam Kelly] Observation 2: Roger Michael Kelly (19 December 1943 â 14 June 2014), known by the stage name Sam Kelly, was an English actor who appeared in film, television, radio and theatre. He is best known for his roles as Captain Hans Geering in âAllo âAllo!, Warren in Porridge, Sam in On the Up, and Ted Liversidge in Barbara. Thought 3: Sam Kelly is best known for his role as Captain Hans Geering in "âAllo âAllo!", so the answer is Captain Hans Geering. Action 3: Finish[Captain Hans Geering]Observation 3: Answer is CORRECT
Reflection: I searched the wrong title for the show, "âAllo âAllo!", which resulted in no results. I should have searched the showâs main character, Gorden Kaye, to find the role he was best known for in the show.
Figure 7: Two HotPotQA trials within the same environment and task. The Reflexion + ReAct agent uses self-reflection to determine a better search method for the next trial.
### D.2 Chain-of-Thought + Reflexion
Trial #1: Question: What profession does John Lanchester and Alan Dean Foster have in common? Thought 1: Letâs think step by step. John Lanchester is a novelist, journalist, and critic. Alan Dean Foster is a novelist and screenwriter. So the profession John Lanchester and Alan Dean Foster have in common is novelist and screenwriter. Action 1: Finish[novelist and screenwriter]Observation 1: Answer is INCORRECT
Trial #2: Question: What profession does John Lanchester and Alan Dean Foster have in common? Thought 1: Letâs think step by step. John Lanchester is a novelist, journalist, and financial journalist. Alan Dean Foster is a novelist and screenwriter. So the profession John Lanchester and Alan Dean Foster have in common is novelist. Action 1: Finish[novelist]Observation 1: Answer is CORRECT
Reflection: My reasoning for the profession John Lanchester and Alan Dean Foster have in common failed because I incorrectly assumed that they both had the same profession. In the future, when attempting this question, I should focus on better researching the individual backgrounds of both authors to ensure that I am accurately identifying their professions. Additionally, I should consider the possibility that they may have multiple professions in common.
### D.3 HotPotQA Chain-of-Thought (GT) + Reflexion
Context: The Battle of White Plains was a battle in the New York and New Jersey campaign of the American Revolutionary War fought on October 28, 1776, near White Plains, New York. Following the retreat âŚ
Trial #1: Question: What was a series of battles during the Revolutionary War, for control of New York City and the state of New Jersey, fought on October 28, 1776 near White Plains, New York? Thought 1: The question is asking about a series of battles during the Revolutionary War for control of New York City and the state of New Jersey, fought on October 28, 1776 near White Plains, New York. From the context, I know that this was the Battle of White Plains. Action 1: Finish[Battle of White Plains] Observation 1: Answer is INCORRECT
Trial #2: Question: What was a series of battles during the Revolutionary War, for control of New York City and the state of New Jersey, fought on October 28, 1776 near White Plains, New York? Thought 1: The question is asking for a series of battles during the Revolutionary War, for control of New York City and the state of New Jersey, fought on October 28, 1776 near White Plains, New York. Based on the context, I know that this series of battles was the New York and New Jersey campaign. Action 1: Finish[The New York and New Jersey campaign] Observation 1: Answer is CORRECT
Reflection: Upon reflecting on the incorrect answer I provided, I realize that I may not have provided enough context to accurately answer the question. The question asked for a series of battles, but I only provided the name of one battle. To better answer this question in the future, I will make sure to provide more context, such as the name of the campaign, which was the New York and New Jersey campaign, and the fact that it was a series of battles. Additionally, I will make sure to include the dates of the battles, as well as the locations, to ensure that my answer is comprehensive and accurate.
### D.4 HotPotQA episodic memory (EPM) ablation prompts
#### D.4.1 (EPM) Chain-of-Thought + Reflexion
Trial #1: Question: Which of Jonny Craig and Pete Doherty has been a member of more bands ? Thought 1: Letâs think step by step. Jonny Craig has been a member of six bands, including Emarosa, Dance Gavin Dance, and Slaves. Pete Doherty has been a member of seven bands, including The Libertines, Babyshambles, and The Puta Madres. So Pete Doherty has been a member of more bands. Action 1: Finish[Pete Doherty]Observation 1: Answer is INCORRECT
Trial #2: Question: Which of Jonny Craig and Pete Doherty has been a member of more bands ? Thought 1: Letâs think step by step. I will research the past and current bands of both musicians to ensure that I am accurately comparing their band memberships. After researching, I have found that Jonny Craig has been a member of seven bands, including Emarosa, Dance Gavin Dance, and Slaves. Pete Doherty has been a member of seven bands, including The Libertines, Babyshambles, and The Puta Madres. So Jonny Craig has been a member of more bands. Action 2: Finish[Jonny Craig]Observation 2: Answer is CORRECT
Reflection: My reasoning for which musician has been a member of more bands failed because I did not take into account that Jonny Craig has been a member of more bands in the past, even though Pete Doherty is currently a member of more bands. In the future, when attempting this question, I should focus on researching the past and current bands of both musicians to ensure that I am accurately comparing their band memberships.
#### D.4.2 (EPM) Chain-of-Thought (GT) + Reflexion
Context: Hari Bahadur Basnet is a Nepalese politician. He is the head of the Foreign Relations Department of the Rastriya Janashakti Party. Basnet holds a M.Sc. in Engineering. A Master of Science (Latin: "Magister Scientiae" ; abbreviated MS, M.S., MSc, M.Sc., MSci, M.Sci., ScM, Sc.M., SciM or Sci.M.) is a masterâs degree in the field of scienceâŚ
Trial #1: Question: The head of the Foreign Relations Department of the Rastriya Janashakti Party holds a degree that can be abbreviated MS, M.S., or ScM, in what field? (Previous trajectory) Thought 1: The question is asking what field of study the head of the Foreign Relations Department of the Rastriya Janashakti Party holds a degree in. From the context, I know that the degree is a Master of Science degree, which is typically granted for studies in sciences, engineering, and medicine. Action 1: Finish[Sciences, Engineering, and Medicine] Observation 1: Answer is INCORRECT
Trial #2: Question: The head of the Foreign Relations Department of the Rastriya Janashakti Party holds a degree that can be abbreviated MS, M.S., or ScM, in what field? (Previous trajectory) Thought 1: The question is asking what field of study the head of the Foreign Relations Department of the Rastriya Janashakti Party holds a degree in. From the context, I know that the degree is a Master of Science degree, which is typically granted for studies in sciences, engineering, and medicine. However, the question is asking for the field of study, not the degree. Based on the context, I know that Hari Bahadur Basnet holds a M.Sc. in Engineering. Action 1: Finish[Engineering]Observation 1: Answer is CORRECT
Reflection: For the first trial, I misunderstood the question. I thought the question was asking for a category of degrees, rather than a specific degree. I learned from my mistake and in the second trial, I focused more closely on the question to understand that it was asking for a specific field of study.