# SAGE: Self-evolving Agents with Reflective and Memory-augmented Abilities
**Authors**: Xuechen Liang, Yangfan He, Yinghui Xia, Xinyuan Song,Jianhui Wang, Meiling Tao, Li Sun, Xinhang Yuan, Jiayi Su,Keqin Li, Jiaqi Chen, Jinsong Yang, Siyuan Chen, Tianyu ShiEast China Jiao Tong University,University of Minnesota - Twin Cities,AutoAgents.ai,Emory University,University of Electronic Science and Technology of China,Guangdong University of Technology,Amazon,Washington University,Xiamen University,Independent Researcher,University of Bristol,University of TorontoProject Page:https://jianhuiwemi.github.io/SAGE
> Equal contribution.Corresponding author:
## Abstract
Large language models (LLMs) have made significant advances in the field of natural language processing, but they still face challenges such as continuous decision-making, lack of long-term memory, and limited context windows in dynamic environments. To address these issues, this paper proposes an innovative framework—Self-evolving Agents with Reflective and Memory-augmented Abilities (SAGE). The SAGE framework comprises three agents: the User, the Assistant, and the Checker. By integrating iterative feedback, reflective mechanisms, and a memory optimization mechanism based on the Ebbinghaus forgetting curve, it significantly enhances the agents’ capabilities in handling multi-tasking and long-span information. The agents, through self-evolution, can adaptively adjust strategies, optimize information storage and transmission, and effectively reduce cognitive load. We evaluate the performance of the SAGE framework on multiple benchmarks and long text tasks. Experimental results show that SAGE significantly improves model performance, achieving a 2.26X improvement on closed-source models and an improvement ranging from 57.7% to 100% on open-source models, with particularly notable effects on smaller models.
## 1 Introduction
In recent years, large language models (LLMs) have made significant progress in the field of natural language processing, demonstrating powerful performance in tasks such as dialogue and text generation Brown et al. (2020); He et al. (2025; 2024). Recently, there has been growing interest in applying LLMs as autonomous agents (LLM agents), which use language not only for understanding and generation, but also for planning and acting in interactive environments Yao et al. (2023b); Shinn et al. (2023); Liang et al. (2024); Li et al. (2024); Zhou et al. (2024). However, these models still face several challenges: (1) LLM Agents need to continuously make decisions in changing environments and adapt to new situations and tasks. (2) LLM Agents lack long-term memory mechanisms, which is increasingly evident in situations requiring sustained interaction with the environment Graves et al. (2016). The limited context window also hinders the model’s ability to handle information over long time spans Rae et al. (2019).
To tackle these challenges, researchers have proposed meta-learning and multi-task learning to enhance the transferability and adaptability of LLM agents. For memory limitations, prior works like MemGPT Packer et al. (2024) use a FIFO queue to manage forgetting, while MemoryBank employs a forgetting curve based on insertion time. However, these approaches are often task-specific, lacking a general framework to systematically improve LLM agents in complex environments. Recent innovations, such as AutoGPT Yang et al. (2023) and BabyAGI Nakajima (2024), leverage LLMs as core controllers, aiming to solve real-world challenges. Yet, multi-agent frameworks still face issues like communication overload, heavily relying on memory to maintain context. As interaction history grows, resource demands and latency increase, limiting efficient deployment in practical scenarios.
In this paper, we propose an innovative framework, Self-evolving Agents with reflective and memory-augmented abilities (SAGE). By enhancing agents’ self-adjustment capabilities through reflection, they can more effectively utilize historical information and make efficient decisions when faced with complex and dynamic tasks. From the perspective of self-evolution, we introduce a memory optimization mechanism based on the Ebbinghaus forgetting curv Ebbinghaus (1885). This mechanism helps agents selectively retain key information, optimize information storage and transmission, reduce unnecessary cognitive load, and enhance agents’ capabilities in interaction tasks with the environment. Experimental results demonstrate that our approach consistently enhances the performance of both proprietary and open-source LLMs across a wide range of benchmarks. The improvements are especially notable in smaller models, where the gains are more pronounced. On tasks such as multi-source question answering and code generation, our method sets a new standard, outperforming existing techniques and achieving leading benchmarks Etezadi & Shamsfard (2023), including AgentBench Liu et al. (2023b).
The main contributions of our work are as follows:
- We propose a novel framework, SAGE, which introduces a reflection mechanism to enhance agents’ self-adjustment capabilities. Without any additional training, this enables agents to utilize historical information more effectively and make better decisions when faced with complex and dynamic tasks.
- We introduce a memory optimization mechanism based on the Ebbinghaus forgetting curve. This helps agents selectively retain key information, reducing the issue of information overload in multi-agent systems.
- SAGE achieves improvements over strong baselines in multiple challenging real-world tasks and attains state-of-the-art results on benchmarks. This framework can be applied to other LLMs, with particularly strong improvements in smaller models.
## 2 Related work
### 2.1 Self-Improvement of Reasoning and Decision-Making
Deep learning has transformed multiple domains including NLP, time series analysis and computer vision Qiu et al. (2025a; b; 2024). A lot of research is focused on making large language models (LLMs) better at improving themselves. Some researchers are working on using carefully crafted prompts to help models learn how to get better, although this usually only works for one-off tasks. Others are tweaking how models get feedback during tasks, which helps them get better at thinking things through Huang et al. (2022). There’s also work on using strategies like random beam searches to help models make smarter decisions and assess their own work. Most current methods rely on quick, one-off tweaks and learning strategies that need lots of resources and hands-on tech help Tian et al. (2024). This paper introduces a self-reflection mechanism, showing that LLMs can keep getting better and produce higher quality work across different tasks, all without needing extra training.
### 2.2 Memory Mechanism for LLM-based Agents
In LLM-based agents, the memory module stores, processes, and retrieves task-related information, supporting knowledge accumulation, experience handling, and decision-making. To enhance the self-evolution capabilities of these agents, researchers are focused on designing and optimizing these memory modules Raffel et al. (2020). Past research has covered various designs and implementations of memory modules. This includes integrating information from different trials to boost reasoning abilities or storing information in natural language to enhance the module’s interpretability and user-friendliness Wada et al. (2019). Despite progress, self-adjustment and memory management still need improvement to handle complex real-world problems more effectively.
<details>
<summary>x1.png Details</summary>
### Visual Description
## System Architecture Diagram: AI Assistant with Memory and Self-Evolution
### Overview
This image is a technical system architecture diagram illustrating the workflow and components of an AI assistant system. The system incorporates user interaction, memory modules (short-term and long-term), a self-evolving mechanism, a feedback checker, and integration with a diverse environment. A secondary chart on retention rate is embedded within the diagram.
### Components/Axes
The diagram is organized into three primary horizontal sections and one embedded chart.
**Top Section (Self-Evolving Loop & Memory):**
* **Self-evolving:** A circular arrow icon with a robot inside, labeled "Self-evolving".
* **STM (Short-Term Memory):** A blue database cylinder icon labeled "STM".
* **LTM (Long-Term Memory):** A red database cylinder icon labeled "LTM".
* **Optimized Information:** A label above a dotted arrow pointing from STM to LTM.
* **Process Arrows:**
* "Observation" and "Action" arrows point from the central Assistant to STM.
* A "Reflection" arrow points from the central Assistant to LTM.
**Middle Section (Core Interaction Flow):**
* **User:** An icon of three people on the far left, labeled "User".
* **Description & Instance:** A green scroll icon, labeled "Description & Instance". A gray arrow points from User to this scroll.
* **Assistant:** A central robot icon, labeled "Assistant". A gray arrow points from the scroll to the Assistant.
* **Checker:** A yellow robot holding a clipboard, labeled "Checker".
* **Interaction Arrows:**
* "Output" arrow from Assistant to Checker.
* "Feedback" arrow from Checker back to Assistant.
* A large, light blue arrow points upward from the "Environment" section to the Assistant.
**Bottom Section (Environment):**
* A dashed box labeled "Environment" at the bottom center.
* **Icons and Labels (from left to right):**
1. A computer monitor with code: "Code"
2. A tablet with a Windows logo: "Operating System"
3. A blue database with a checkmark: "Database"
4. Two speech bubbles (Q & A): "Q&A"
5. A network graph: "Knowledge Graph"
6. A browser window with a magnifying glass: "Mind2Web"
7. A living room scene: "ALFWorld"
8. A shopping interface: "Web Shopping"
**Embedded Chart (Top Right):**
* **Chart Type:** Line graph.
* **Y-axis Label:** "Retention Rate" (vertical text).
* **X-axis Label:** "Time" (horizontal text).
* **Data Series (Lines):**
1. A solid blue line.
2. A dashed red line.
3. A dotted green line.
* **Background Icon:** A brain with a gear inside, positioned behind the chart lines.
### Detailed Analysis
**System Workflow:**
1. The **User** provides input, conceptualized as a "Description & Instance".
2. This input is processed by the **Assistant**.
3. The Assistant interacts with its memory systems:
* It sends "Observation" and "Action" data to **STM**.
* It performs "Reflection" to consolidate information into **LTM**.
* "Optimized Information" flows from STM to LTM.
4. The Assistant generates an **Output** which is evaluated by the **Checker**.
5. The Checker provides **Feedback** to the Assistant, creating a refinement loop.
6. The entire system is under a **Self-evolving** paradigm, suggesting continuous improvement.
7. The Assistant is grounded in and draws capabilities from a broad **Environment** containing tools and knowledge sources (Code, OS, Databases, Q&A, Knowledge Graphs, and specific benchmarks like Mind2Web, ALFWorld, Web Shopping).
**Retention Rate Chart Analysis:**
* **Trend Verification:** All three lines show a decaying trend, starting high on the left (early Time) and decreasing towards the right (later Time). The decay is non-linear, resembling an exponential or power-law decay curve.
* **Line Comparison:**
* The **solid blue line** decays the fastest, ending at the lowest retention rate.
* The **dashed red line** decays at a moderate rate, ending in the middle.
* The **dotted green line** decays the slowest, maintaining the highest retention rate over time.
* **Spatial Grounding:** The chart is positioned in the top-right corner of the diagram, within a light peach-colored oval that also contains the brain/gear icon. The legend (implied by line style/color) is directly on the chart itself.
### Key Observations
1. **Dual Memory Architecture:** The system explicitly separates short-term (STM) and long-term (LTM) memory, with a defined process ("Optimized Information") for transferring knowledge between them.
2. **Closed-Loop Feedback:** The inclusion of a "Checker" that provides "Feedback" creates a closed-loop system for output refinement, which is a key feature for quality control and learning.
3. **Environmental Grounding:** The Assistant is not isolated; it is directly connected to a wide array of practical environments and tools, indicating it is designed for real-world task execution.
4. **Retention Hierarchy:** The embedded chart suggests a hierarchy of information retention. The green dotted line (likely representing "Optimized Information" from the main diagram) shows superior long-term retention compared to the other two lines (which could represent raw observations or unreflected actions).
### Interpretation
This diagram outlines a sophisticated AI agent architecture designed for sustained, evolving performance. The core innovation appears to be the integration of a **self-reflective memory consolidation process** (Observation -> STM -> Reflection -> LTM) with an **external feedback mechanism** (Checker). This structure mimics cognitive theories of human learning, where experiences are encoded, consolidated during reflection, and stored for long-term use, with external validation improving accuracy.
The **Retention Rate chart** serves as a visual hypothesis or result: the process of creating "Optimized Information" (green dotted line) through the system's reflection and consolidation mechanisms leads to significantly better knowledge retention over time compared to unoptimized data streams (blue and red lines). This implies the system's value is not just in performing tasks, but in *learning from them efficiently*.
The broad **Environment** section indicates the system's intended domain is general-purpose problem-solving across digital and simulated real-world tasks (coding, web interaction, database management, household robotics). The architecture suggests a move from a stateless chatbot to a stateful, learning agent that builds a persistent knowledge base (LTM) to improve its future performance in a self-evolving manner.
</details>
Figure 1: An illustration of the SAGE: a user provides a description and instance to the assistant with short-term (STM) and long-term (LTM) memory. The assistant performs observation, action, reflection, and output, which the checker reviews. The retention rate curve on the right illustrates memory decay over time, with a self-evolving loop guiding continued updates.
## 3 Method
In this section, we present the SAGE framework, designed to improve agent performance by leveraging three core mechanisms: iterative feedback, reflection, and MemorySyntax (as shown in Figure 1). The assistant agent $A$ iteratively updates its policy $\pi_{\theta}$ based on feedback $f_{t}$ provided by the checker agent $C$ , optimizing over successive iterations to maximize the expected reward $R$ . The reflection mechanism allows $A$ to incorporate historical observations $\mathcal{O}_{t}$ and actions $\mathbf{a}_{t}$ , forming a self-reflection $r_{t}$ , which is stored in the memory $\mathcal{M}_{L}$ for future decision-making. Finally, MemorySyntax combines the Ebbinghaus forgetting curve with linguistic principles to manage memory decay, dynamically updating the agent’s short-term memory $\mathcal{M}_{S}$ and long-term memory $\mathcal{M}_{L}$ by prioritizing information based on its retention strength $S(I_{t})$ , thus improving the agent’s ability to retain crucial information while discarding less relevant data. The subsequent subsections detail these components.
### 3.1 Iterative Feedback
The iterative feedback mechanism in the SAGE framework enables the assistant agent $A$ to refine its policy $\pi_{\theta}$ through repeated interactions with the checker agent $C$ . At each iteration $t$ , the assistant receives feedback $f_{t}$ based on its current output $\mathbf{o}_{t}$ , and adjusts its policy accordingly. This process continues until the checker validates the output or the iteration cap $N$ is reached, ensuring that $A$ incrementally optimizes its decisions to improve task performance over successive iterations.
#### 3.1.1 Initialization Phase
Role Assignment. In the SAGE framework, three agents are introduced: the user $U$ , the assistant $A$ , and the checker $C$ . The user, upon receiving prompt $P_{U}$ , assumes the role of task proposer by specifying a task $\mathcal{T}_{U}$ and related constraints $\mathcal{C}_{U}$ . The assistant, upon receiving prompt $P_{A}$ , generates a sequence of actions $\mathbf{a}_{t}$ based on the observations $\mathcal{O}_{t}$ and environment $\mathcal{E}$ . The checker $C$ evaluates the output $\mathbf{o}_{A}$ produced by the assistant, providing feedback $f_{C}$ based on the discrepancy between $\mathbf{o}_{A}$ and the expected result, updating its policy $\pi_{\theta}$ iteratively to minimize this gap. Task Assignment. The task $\mathcal{T}_{U}$ provided by the user includes an initial task description $\mathbf{d}_{U}$ and an instance $\mathbf{i}_{U}$ that serves as the reference for correct output. This forms the input set $\mathcal{I}_{A}=(\mathbf{d}_{U},\mathbf{i}_{U})$ for the assistant to initiate its generative process. The assistant then proceeds by selecting an action $\mathbf{a}_{t}$ at each time step $t$ , guided by $\pi_{\theta}$ , with the goal of maximizing the reward $R_{t}$ for completing $\mathcal{T}_{U}$ .
<details>
<summary>x2.png Details</summary>
### Visual Description
## Diagram: AI Assistant Self-Improvement Feedback Loop
### Overview
The image is a flowchart diagram illustrating a process for an AI assistant to generate, evaluate, and revise its responses based on user instructions and feedback. It depicts a cyclical workflow involving a user prompt, an assistant's initial response, a checker's evaluation, and a revised answer, with connections to short-term and long-term memory components. The diagram uses colored boxes and directional arrows to show the flow of information and tasks.
### Components/Axes
The diagram is organized into two primary vertical columns.
**Left Column (Process Flow):**
1. **Top Box (Light Green):** Labeled "User Prompt:" and "User Instruction:".
2. **Second Box (Light Yellow):** Labeled "Assistant's Task Template:" and "Assistant's Initial Response:".
3. **Third Box (Light Pink):** Labeled "Checker Evaluation Template:" and "The feedback returned by the checker to the assistant:".
4. **Bottom Box (Light Purple):** Labeled "Assistant Revision Template:" and "Assistant's Revised Answer:".
**Right Column (Memory Components):**
1. **Top Box (Light Blue):** Labeled "Short-Term Memory (STM):".
2. **Bottom Box (Light Red):** Labeled "Long-Term Memory (LTM):".
**Connectors:**
* Thick, downward-pointing purple arrows connect the four boxes in the left column sequentially.
* Thin, purple arrows connect elements from the left column to the memory boxes on the right and vice versa, indicating data flow and feedback loops.
### Detailed Analysis
**Left Column - Process Flow:**
* **User Prompt Box:**
* **User Prompt:** "Which metal is produced by the Bessemer Process?"
* **User Instruction:** "Answer the question based on the given passages. Only give me the answer and do not output any other words."
* **Assistant's Task Template Box:**
* **Task:** "Generate a concise and direct answer to the user's question based on the provided passages."
* **Assistant's Initial Response:** "The metal produced by the Bessemer Process is steel."
* **Checker Evaluation Template Box:**
* **Task:** "Review the Assistant's response for correctness, conciseness, and adherence to the user's request."
* **Criteria:**
* "Is the answer correct?"
* "Is the answer concise and free from irrelevant information?"
* **The feedback returned by the checker to the assistant:** "The answer is correct, but the user requested a response without any extra words. The phrase 'The metal produced by the Bessemer Process is' is unnecessary. The response should only be 'Steel'."
* **Assistant Revision Template Box:**
* **Task:** "Generate a revised response that strictly follows the user's request for a concise answer."
* **Assistant's Revised Answer:** "Steel."
**Right Column - Memory Components:**
* **Short-Term Memory (STM) Box:**
* **Task:** "Answer TriviaQA question."
* **Recent trajectory:** A list showing the flow:
* "User Prompt & User Instruction →"
* "Assistant Initial Response →"
* "Checker's Feedback →"
* "Self-reflection Result: r_t →"
* "Assistant's Revised Answer →"
* "New Self-reflection Result: r_{t+1}"
* An arrow points from "Self-reflection Result: r_t" in this list down to the Long-Term Memory box.
* **Long-Term Memory (LTM) Box:**
* **Self-reflection Result: r_t** (with an arrow pointing to it from STM).
* **(Example:** "User prefers concise answers without additional phrases. Future responses should prioritize brevity.")
* **New Self-reflection Result: r_{t+1}** (with an arrow pointing to it from the STM's "New Self-reflection Result: r_{t+1}").
* **Assistant's Revised Answer** (with an arrow pointing to it from the left column's "Assistant's Revised Answer").
### Key Observations
1. **Iterative Refinement:** The core process is a closed loop: Prompt → Initial Response → Evaluation/Feedback → Revised Response.
2. **Memory Integration:** The process is not stateless. Short-Term Memory tracks the immediate trajectory of the current interaction. Long-Term Memory stores generalized learnings (self-reflection results) from past interactions to inform future behavior.
3. **Explicit Criteria:** The checker's evaluation is based on specific, predefined criteria (correctness, conciseness, adherence to instruction).
4. **Learning Mechanism:** The "Self-reflection Result" (r_t) is generated from the checker's feedback and is used to update the assistant's strategy. This updated strategy (r_{t+1}) is then stored in LTM for future use.
5. **Visual Coding:** Different colors are used to distinguish between user input (green), assistant processes (yellow, purple), checker processes (pink), and memory stores (blue, red).
### Interpretation
This diagram models a **self-improving AI assistant system** designed to adhere strictly to user constraints. It goes beyond a simple request-response model by incorporating an internal critic (the Checker) and a learning mechanism.
* **The Data Suggests:** The system is engineered to handle tasks where response format is as critical as content accuracy. The example shows the assistant learning to strip conversational filler to meet a "concise answer only" instruction.
* **Relationships:** The Checker acts as a quality control layer, enforcing rules that the assistant might initially overlook. The memory components bridge individual interactions, allowing the system to accumulate experience. The feedback from the Checker is transformed into a "self-reflection" that becomes a stored policy (e.g., "prioritize brevity").
* **Notable Anomaly/Feature:** The system explicitly separates the *instance-specific* feedback ("your phrase was unnecessary") from the *generalized lesson* ("future responses should prioritize brevity"). This is a key feature for scalable learning, preventing the memory from being cluttered with one-off comments.
* **Underlying Principle:** The flowchart embodies a **Peircean investigative cycle** of learning: it makes a conjecture (initial response), tests it against a rule-based critic (checker), and abductively infers a new rule (self-reflection) to improve future conjectures. The "Long-Term Memory" serves as the evolving repository of these abductive insights.
</details>
Figure 2: An example of the assistant’s iterative workflow, including checker evaluation, prompt templates for feedback, and reflection processes integrating short-term and long-term memory.
#### 3.1.2 Actual Interaction Phase
Following the role assignment and task definition in the initialization phase, the assistant $A$ transitions into the actual interaction phase to generate outputs aimed at accomplishing the task $\mathcal{T}_{U}$ . In this phase, $A$ iteratively produces outputs $\mathbf{o}_{t}$ at each time step $t$ based on the task description $\mathbf{d}_{U}$ and instance $\mathbf{i}_{U}$ provided in the input set $\mathcal{I}_{A}=(\mathbf{d}_{U},\mathbf{i}_{U})$ . At each time step $t$ , the assistant selects an action $\mathbf{a}_{t}$ by following its policy $\pi_{\theta}$ , which is conditioned on the current state $s_{t}$ , the reward signal $R_{t}$ (the reward score for task performance), and feedback $f_{t}^{i}$ from the checker $C$ . This decision-making process is formalized as:
$$
\mathbf{o}_{t}\sim\pi_{\theta}(\mathbf{o}_{t}\mid s_{t},R_{t},f_{t}^{i}), \tag{1}
$$
where $\pi_{\theta}$ represents the assistant’s policy, $R_{t}$ reflects the reward signal based on task performance at time $t$ , and $f_{t}^{i}$ is the feedback provided by the checker during the $i$ -th iteration.
As the interaction progresses, the checker $C$ evaluates each output $\mathbf{o}_{t}$ generated by $A$ , comparing it against the expected outcome derived from $\mathbf{i}_{U}$ . Based on this comparison, the checker provides iterative feedback $f_{t}^{i}$ to guide $A$ in refining its actions $\mathbf{a}_{t}$ and outputs $\mathbf{o}_{t}$ . The iterative refinement continues until either the checker validates the output as correct or the iteration limit $N$ is reached.
Theoretical optimality of iterative feedback mechanism. In the SAGE framework, the assistant repeatedly updates its policy through this checker feedback, enabling the outputs to be incrementally refined until either the result is validated or a specified iteration limit is reached. The assistant’s utility $R_{A}$ reflects task performance, and the checker’s utility $R_{C}$ depends on its feedback. The following theorem indicates that this iterative feedback mechanism leads to strategy stability in the sense of a Nash equilibrium Fudenberg & Tirole (1991).
**Theorem 3.1 (Theory for the multi-agent iterative feedback system)**
*Let $\mathcal{U},\mathcal{A},\mathcal{C}$ denote the compact, convex strategy spaces of the user (U), assistant (A), and checker (C), respectively. Assume that the utility functions
$$
R_{U}:\mathcal{U}\times\mathcal{A}\times\mathcal{C}\to\mathbb{R},\quad R_{A}:
\mathcal{U}\times\mathcal{A}\times\mathcal{C}\to\mathbb{R},\quad\text{and}
\quad R_{C}:\mathcal{U}\times\mathcal{A}\times\mathcal{C}\to\mathbb{R} \tag{2}
$$
are continuous in each player’s strategy. Then, by the Debreu-Glicksberg-Fan fixed-point theorem, there exists a Nash equilibrium
$$
(s_{U}^{*},s_{A}^{*},s_{C}^{*})\in\mathcal{U}\times\mathcal{A}\times\mathcal{C}. \tag{3}
$$
Furthermore, suppose that the assistant’s policy $\pi_{\theta}$ is updated via policy gradient methods and that the checker’s strategy is refined through convex optimization. Then, the iterative update procedures yield sequences
$$
\{\pi_{\theta}^{(k)}\}_{k\geq 0}\quad\text{and}\quad\{f^{(k)}\}_{k\geq 0}, \tag{4}
$$
which converge to a stable strategy profile $(\pi_{\theta}^{*},f^{*})$ , and has:
$$
R_{A}(\pi_{\theta}^{*},f^{*})\geq R_{A}(\pi_{\theta},f^{*}),\quad R_{C}(\pi_{
\theta}^{*},f^{*})\geq R_{C}(\pi_{\theta}^{*},f). \tag{5}
$$*
This result demonstrates that the iterative feedback mechanism enhances the model’s strategy stability by converging to a Nash equilibrium in the three-player game. It provides a stronger justification for the three-agent system versus simpler alternatives (such as two-agent systems). For the detailed theoretical explanation and proof, see A.2.
#### 3.1.3 Evolutionary Goals and Directions
Leveraging the feedback $f_{t}^{i}$ obtained at each iteration $t$ , the assistant $A$ formulates new evolutionary objectives:
$$
\mathcal{G}^{t+1}=(\mathcal{A}^{t+1},\mathcal{D}^{t+1}),\quad\mathcal{D}^{t+1}
=\arg\min_{\mathcal{D}_{t}\in\Delta}\sum_{i\in I_{t}}L_{D}\left(\mathcal{D}_{t
};f_{t}^{i},\pi_{\theta}^{t}\right), \tag{6}
$$
where $\mathcal{A}^{t+1}$ represents the updated memory optimization mechanisms, and $\mathcal{D}^{t+1}\in\Delta$ refers to the model’s self-adjustments to make the RL algorithm converge. These evolutionary objectives guide the assistant in updating its policy $\pi_{\theta}$ for the subsequent iteration. The policy update is governed by the function $\psi$ , which integrates the current policy $\pi_{\theta}^{t}$ with the new evolutionary objectives $\mathcal{G}^{t+1}$ :
$$
\theta^{t+1}=\phi\bigl{(}\theta^{t},\mathcal{G}^{t+1}\bigr{)}=\theta^{t}+
\alpha\nabla_{\theta}\Bigl{[}\lambda_{A}L_{A}\bigl{(}\theta^{t},\mathcal{A}^{t
+1}\bigr{)}+\lambda_{D}L_{D}\bigl{(}\theta^{t},\mathcal{D}^{t+1}\bigr{)}\Bigr{
]}. \tag{7}
$$
Here $L_{A}(\theta,\mathcal{A})$ and $L_{D}(\theta,\mathcal{D})$ are MSE loss functions corresponding to the memory-optimization and self-adjustment aspects, respectively, and $\lambda_{A},\lambda_{D}\geq 0$ are weighting coefficients. The iterative policy refinement enables the assistant $A$ to continuously adapt its strategies based on cumulative feedback and evolving task requirements, thereby improving its overall performance in dynamic environments.
### 3.2 Memory Management
The SAGE framework implements a dual-memory system, consisting of Short-Term Memory (STM) and Long-Term Memory (LTM), to manage task-relevant information and enhance the agent’s reasoning and decision-making capabilities (see Figure 2 for a visual representation of this process).
Short-Term Memory (STM). STM is responsible for storing immediate, task-specific data with limited capacity. It updates rapidly with new observations ( $\mathcal{O}_{t}$ ) and actions ( $\mathbf{a}_{t}$ ), maintaining a recent trajectory history $\mathcal{T}_{t}=(\mathcal{O}_{t},\mathbf{a}_{t})$ . This allows the agent to make real-time decisions and respond quickly to dynamic changes in the environment Mnih et al. (2015).
Long-Term Memory (LTM). LTM retains critical information and self-reflections ( $r_{t}$ ) over extended periods, enabling the agent to accumulate knowledge from past interactions and apply it to future tasks. Stored as $\mathcal{M}_{L}=\{r_{t}\mid t\in T\}$ , this memory mechanism allows the agent to use prior experiences to improve task performance, particularly in complex environments that require long-span information Graves et al. (2016).
By integrating STM and LTM, the SAGE framework allows the agent to balance immediate task demands with the ability to draw from accumulated knowledge, thereby enhancing its overall decision-making efficiency.
#### 3.2.1 Reflection
Figure 4 illustrates an example of the reflection mechanism applied to a HotpotQA task Yang et al. (2018b). The reflection mechanism equips the assistant $A$ with sparse reward signals, such as binary success/failure states, trajectory $\mathcal{T}_{t}$ , and its stored memory $\mathcal{M}_{L}$ . The assistant processes these inputs, deriving insights from past performance and storing self-reflections $\mathbf{r}_{t}$ for future decision-making. These self-reflections, richer than scalar rewards, enhance the assistant’s learning capacity and are incorporated into long-term memory:
$$
\mathbf{r}_{t}=\text{ref}(\mathbf{o}_{1:t},\mathbf{R}_{1:t}), \tag{8}
$$
where $\text{ref}(\cdot)$ denotes the reflection function based on the output sequence $\mathbf{o}_{1:t}$ and rewards $\mathbf{R}_{1:t}$ . The derived reflection $\mathbf{r}_{t}$ is then added to $\mathcal{M}_{L}$ :
$$
\mathcal{M}_{L}\leftarrow\mathcal{M}_{L}\cup\{\mathbf{r}_{t}\}. \tag{9}
$$
The process gradually enhances the agent’s decision-making, allowing it to adapt effectively through accumulated experience.
#### 3.2.2 MemorySyntax
Building upon the reflection mechanism, the MemorySyntax method integrates the Ebbinghaus forgetting curve with linguistic principles to emulate human-like memory processes within the agent’s memory management system. Let $I_{t}$ denote the information received at time $t$ , and let $R(I_{t},\tau)$ represent its retention rate after a time interval $\tau$ . According to the Ebbinghaus forgetting curve, the retention rate is modeled as:
$$
R(I_{t},\tau)=e^{-\frac{\tau}{S}}, \tag{10}
$$
where $S$ is the strength of the information, reflecting its importance and complexity.
To enhance retention, MemorySyntax applies linguistic optimization to $I_{t}$ , producing an optimized version $I_{t}^{*}$ with increased strength $S^{*}>S$ . The retention rate for $I_{t}^{*}$ is defined as:
$$
R(I_{t}^{*},\tau)=\begin{cases}e^{-\frac{\tau}{S^{*}}},&\text{if }I_{t}^{*}\in
\mathcal{M}_{S},\\
e^{-\frac{\tau}{S}},&\text{if }I_{t}^{*}\in\mathcal{M}_{L},\end{cases} \tag{11}
$$
where $\mathcal{M}_{S}$ and $\mathcal{M}_{L}$ represent short-term memory and long-term memory, respectively.
The agent updates its memory state $\mathcal{M}_{t}$ based on the retention rate of $I_{t}^{*}$ using predefined thresholds $\theta_{1}$ and $\theta_{2}$ , with $\theta_{1}>\theta_{2}$ . The memory update rule is formalized as:
$$
\mathcal{M}_{t+1}=\begin{cases}\mathcal{M}_{t}\cup\{I_{t}^{*}\},&\text{if }R(I
_{t}^{*},\tau)\geq\theta_{1},\\
\mathcal{M}_{t}\setminus\{I_{t}^{*}\},&\text{if }R(I_{t}^{*},\tau)<\theta_{2},
\\
\mathcal{M}_{t},&\text{otherwise}.\end{cases} \tag{12}
$$
This update rule operates under the following conditions:
- Retention in Short-Term Memory ( $\mathcal{M}_{S}$ ): If $R(I_{t}^{*},\tau)\geq\theta_{1}$ , the information $I_{t}^{*}$ is retained in short-term memory $\mathcal{M}_{S}$ .
- Discarding Information: If $R(I_{t}^{*},\tau)<\theta_{2}$ , the information $I_{t}^{*}$ is considered insignificant and is discarded.
- Transfer to Long-Term Memory ( $\mathcal{M}_{L}$ ): If $\theta_{2}\leq R(I_{t}^{*},\tau)<\theta_{1}$ , the information $I_{t}^{*}$ is transferred to long-term memory $\mathcal{M}_{L}$ .
The above MemorySyntax method is optimal and ensures maximal information utility. For theoretical background and detailed proof, see Appendix A.1.
By simulating human memory dynamics, MemorySyntax enables the agent to prioritize essential information, retain critical data in short-term memory $\mathcal{M}_{S}$ , store important but less frequently used information in long-term memory $\mathcal{M}_{L}$ , and discard irrelevant data. The mechanism addresses memory capacity limitations and enhances the agent’s ability to perform complex tasks requiring efficient memory management.
## 4 Experiment
To demonstrate the capabilities and performance of SAGE in coordinating autonomous agent groups to work together on tasks, we conduct extensive quantitative experiments on benchmark tasks. We use a public benchmark, AgentBench Liu et al. (2023b), which is a multidimensional evolutionary benchmark from which we select six tasks. These tasks test the reasoning and decision-making abilities of LLMs acting as agents in multi-turn open-ended generation settings. To further assess the agents’ long-context understanding, we select four widely adopted tasks related to long-text processing. These tasks reflect the agents’ programming abilities(LCC Guo et al. (2023), RepoBench-P Liu et al. (2023a)) and reasoning abilities(HotpotQA Yang et al. (2018b), TriviaQA Joshi et al. (2017b)).
### 4.1 Evaluation on AgentBench
Task Description AgentBench includes scenarios from CODE (Knowledge Graph, OS, DB), GAME (ALFWorld) Shridhar et al. (2021), and WEB (WebShop Yao et al. (2023a), Mind2Web Deng et al. (2023)). For more details for the datasets and benchmarks, see Appendix B.
Baselines We evaluate commercial models GPT-3.5 Brown et al. (2020) and GPT-4 OpenAI et al. (2024), and open-source models Llama2 Touvron et al. (2023), Codellama Rozière et al. (2024), Qwen Bai et al. (2023), and ChatGLM2 GLM et al. (2024). Dialogue history exceeding the model length limit is truncated, using greedy decoding.
Results As shown in Table 1, our method significantly improves model performance on AgentBench, especially for smaller models. GPT-3.5 and GPT-4, despite already high scores, also show notable improvements with SAGE, up to 2.26x in the Database task. Llama2-7b is notably enhanced, showing the method’s impact on weaker models. CodeLlama-7b and Qwen-1.8B also see substantial gains. Qwen-1.8B, after applying our method, performs close to GPT-3.5, highlighting its potential as a general agent. Llama2, previously error-prone, shows a significant reduction in basic errors through feedback and memory optimization, proving that our method not only enhances agent capabilities but also reduces fundamental errors in complex tasks.
Table 1: Baseline and SAGE Framework Performance on AgentBench
| LLM Type | Model | VER | OS | DB | KG | ALF | WS | M2W | | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Base | SAGE | Base | SAGE | Base | SAGE | Base | SAGE | Base | SAGE | Base | SAGE | Base | SAGE | | |
| API | GPT-4 | 42.4 | 49.7 | 32.0 | 39.8 | 57.4 | 63.1 | 78.0 | 82.0 | 67.1 | 67.8 | 27.0 | 32.0 | 27.0 | 32.0 |
| GPT-3.5 | 31.6 | 38.3 | 15.7 | 35.6 | 25.9 | 37.6 | 17.0 | 23.0 | 64.1 | 72.1 | 16.0 | 28.0 | 16.0 | 28.0 | |
| OSS | Llama2-7B Chat | 0.0 | 8.4 | 0.0 | 10.2 | 0.0 | 25.0 | 0.0 | 5.0 | 4.4 | 10.4 | 0.0 | 15.0 | 0.0 | 15.0 |
| CodeLlama-7B Instruct | 5.7 | 18.4 | 2.6 | 19.2 | 0.0 | 27.0 | 0.0 | 12.5 | 16.3 | 40.2 | 0.0 | 15.0 | 15.0 | 15.0 | |
| Qwen1.8B Chat | 2.7 | 18.7 | 1.4 | 15.1 | 6.8 | 45.3 | 0.0 | 10.5 | 6.6 | 11.4 | 0.6 | 13.6 | 13.6 | 13.6 | |
| Qwen-7B Chat | 5.6 | 22.2 | 4.8 | 18.0 | 0.0 | 48.0 | 34.0 | 38.5 | 0.0 | 13.6 | 0.0 | 15.0 | 15.0 | 15.0 | |
| ChatGLM2-6B v1.1 | 0.0 | 15.2 | 0.0 | 16.3 | 0.0 | 17.0 | 0.0 | 5.0 | 0.3 | 10.3 | 4.9 | 14.9 | 14.9 | 14.9 | |
### 4.2 Complex Problem-Solving Tasks Evaluation
We evaluated SAGE against baseline models on three tasks: long-form QA Akash et al. (2023), multi-turn dialog Cui et al. (2020), and sequential task completion Stephens et al. (2012). As shown in Table 2, SAGE outperforms all baselines with significant gains, such as a 20.8% increase in answer accuracy for GPT-3.5 on HotpotQA Yang et al. (2018b) and a 17.3% improvement in task completion for Mistral-7b on ALFWorld Shridhar et al. (2021). Across all tasks, SAGE notably enhances answer accuracy, dialog coherence, and step completion.
Table 2: Evaluation of SAGE and Baseline Models on Three Different Tasks
| Agent | Task Completion | Answer Accuracy | Dialog Coherence | Step Completion |
| --- | --- | --- | --- | --- |
| Time (min) | (QA)(%) | (%) | Accuracy (%) | |
| GPT-3.5 (Baseline) | Long-form QA (HotpotQA) | 54.1% | 48.5% | 62.7% |
| GPT-4 (Baseline) | Long-form QA (HotpotQA) | 61.2% | 53.8% | 68.2% |
| Llama2-7b (Baseline) | Multi-turn Dialog (MultiWOZ) | 55.9% | 50.1% | 64.8% |
| Codellama-13b (Baseline) | Multi-turn Dialog (MultiWOZ) | 58.4% | 52.3% | 66.7% |
| Mistral-7b (Baseline) | Sequential Task (ALFWorld) | 56.5% | 51.5% | 65.1% |
| SAGE-GPT-3.5 | Long-form QA (HotpotQA) | 74.9% (+20.8%) | 68.3% (+19.8%) | 80.6% (+17.9%) |
| SAGE-GPT-4 | Long-form QA (HotpotQA) | 78.4% (+17.2%) | 73.4% (+19.6%) | 83.9% (+15.7%) |
| SAGE-Llama2-7b | Multi-turn Dialog (MultiWOZ) | 72.2% (+16.1%) | 67.9% (+17.8%) | 78.5% (+13.7%) |
| SAGE-Codellama-13b | Multi-turn Dialog (MultiWOZ) | 74.7% (+16.3%) | 71.2% (+18.9%) | 81.2% (+14.5%) |
| SAGE-Mistral-7b | Sequential Task (ALFWorld) | 73.8% (+17.3%) | 70.5% (+19.0%) | 79.9% (+14.8%) |
### 4.3 Evaluation of Long-Context Tasks
We evaluated the agent’s code generation and reasoning on four long-text tasks: LCC Dataset Mohler et al. (2016) focuses on predicting the next line of code from a few initial lines, with Precision, Recall, and F1 as metrics. RepoBench-P Liu et al. (2024) tests retrieval of relevant code snippets from cross-file and within-file contexts to predict the next line, also evaluated with Precision, Recall, and F1. HotPotQA Yang et al. (2018a), a Wikipedia-based dataset with 113k question-answer pairs, challenges the agent to reason across multiple documents, evaluated by answer F1. TriviaQA Joshi et al. (2017a) is a reading comprehension dataset with question-answer pairs and evidence paragraphs (filtered to over 1,000 words), also using answer F1 for evaluation.
We compared two self-refinement methods: Beam Search Kool et al. (2019), which integrates self-assessment through stochastic beam search, and Reflexion Shinn et al. (2023), which uses past trial experience in a verbal form.
### 4.4 Evaluation Results:
Code Completion Task: On the LCC dataset (Table 3), SAGE shows a slight improvement in F1 score (79.29) compared to Beam Search and Reflexion. Its memory mechanisms help refine code predictions, but the performance difference is not substantial in simpler tasks like code completion.
Reasoning Tasks: SAGE significantly outperforms Reflexion and Beam Search on HotPotQA and TriviaQA, with F1 scores of 22.06 and 22.76 (Table 3). SAGE’s ability to effectively integrate multi-document information through reflection leads to better reasoning accuracy, while Reflexion and Beam Search face challenges in handling complex reasoning tasks.
Table 3: Comparison of Performance Across Different Methods
| Models | LCC | RepoBench-P | HotpotQA | TriviaQA | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Precision | Recall | F1 | Precision | Recall | F1 | F1 | F1 | |
| Reflexion | 77.72 | 81.00 | 79.28 | 78.73 | 81.86 | 80.25 | 11.26 | 11.23 |
| Beam search | 78.98 | 79.32 | 79.12 | 78.75 | 81.02 | 79.87 | 10.26 | 12.13 |
| SAGE | 78.76 | 79.88 | 79.29 | 79.27 | 83.28 | 81.22 | 22.06 | 22.76 |
### 4.5 Evaluation of RAG Agents
Table 4 compares classical lexical retrieval (RAG with BM25) Robertson & Zaragoza (2009), dense passage retrieval (RAG with DPR) Reichman & Heck (2024), a commercial retrieval solution (RAG with OpenAI Retrieval) OpenAI (2023), the TART Eisenschlos et al. (2022) sequence-to-sequence retrieval model, and the FiD (Fusion-in-Decoder) method Izacard & Grave (2021), all tested on multi-document and open-domain QA tasks (HotpotQA, Natural Questions Kwiatkowski et al. (2019), and TriviaQA). RAG with BM25 relies on term-based matching, while RAG with DPR uses learned dense embeddings. TART adopts a transformer-based approach to produce relevant contexts, and FiD fuses multiple retrieved passages through an encoder-decoder design. In contrast, ChatGPT-4 (SAGE) employs a structured reasoning workflow for retrieval and generation, which leads to steady accuracy improvements of 3.6% to 4.7% and cuts memory consumption nearly 50% on some tasks, all without increasing latency.
Table 4: Evaluation of different RAG Agents on Different Tasks and Datasets
| Agent | Accuracy (QA) (%) | Latency (ms) | Memory Usage (MB) |
| --- | --- | --- | --- |
| Task 1: Multi-Document QA (HotpotQA) | | | |
| RAG (BM25) | 60.8 | 121 | 613 |
| RAG (DPR) | 66.3 | 129 | 542 |
| RAG (OpenAI Retrieval) | 67.4 | 108 | 494 |
| TART | 63.2 | 144 | 477 |
| FiD (Fusion-in-Decoder) | 70.1 | 153 | 456 |
| ChatGPT-4 - Sage | 74.8 (+4.7) | 128 | 231 (-50%) |
| Task 2: Document Retrieval for Contextual Answering (Natural Questions) | | | |
| RAG (BM25) | 59.9 | 125 | 605 |
| RAG (DPR) | 65.5 | 131 | 561 |
| RAG (OpenAI Retrieval) | 66.8 | 113 | 484 |
| TART | 62.4 | 146 | 455 |
| FiD (Fusion-in-Decoder) | 69.8 | 156 | 443 |
| ChatGPT-4 - Sage | 73.6 (+3.8) | 131 | 227 (-49%) |
| Task 3: Open-Domain QA with Multiple Contexts (TriviaQA) | | | |
| RAG (BM25) | 62.1 | 124 | 615 |
| RAG (DPR) | 67.8 | 129 | 530 |
| RAG (OpenAI Retrieval) | 68.9 | 117 | 494 |
| TART | 64.7 | 148 | 462 |
| FiD (Fusion-in-Decoder) | 71.9 | 155 | 456 |
| ChatGPT-4 - Sage | 75.5 (+3.6) | 134 | 243 (-47%) |
### 4.6 Error analysis
As shown in Figure 3, the SAGE framework significantly enhances agent performance across tasks, especially in the WS task for AgentBench, due to its iterative feedback mechanism, which refines outputs through continuous assistant-checker interaction. In OS and DB tasks, Context Limit Exceeded and invalid format errors are nearly eliminated, with a notable reduction in invalid action errors, attributed to the reflection mechanism that helps the assistant learn and reduce logical mistakes.
<details>
<summary>x3.png Details</summary>
### Visual Description
\n
## Horizontal Stacked Bar Chart: Agentbench vs. SAGE Performance Breakdown
### Overview
The image displays two horizontal stacked bar charts side-by-side, comparing the performance breakdown of two systems, "Agentbench" (left panel) and "SAGE" (right panel), across six different tasks. Each bar represents 100% of the outcomes for a task, segmented by color-coded categories indicating the result type (e.g., Completed, various error types). The x-axis for both charts is "Percentage" from 0 to 100.
### Components/Axes
* **Chart Titles:** "Agentbench" (left panel), "SAGE" (right panel).
* **Y-Axis (Tasks):** The same six tasks are listed vertically for both charts, from top to bottom:
1. Operating System
2. Database
3. Knowledge Graph
4. ALFWord
5. Web shopping
6. Mind2Web
* **X-Axis:** Labeled "Percentage" at the bottom of each panel, with major tick marks at 0, 20, 40, 60, 80, and 100.
* **Legend:** Positioned at the top-right of the entire image, outside the chart panels. It defines five categories:
* **Completed** (Light Olive Green)
* **CLE** (Light Brown/Tan)
* **Invalid Format** (Medium Blue)
* **Invalid Action** (Light Blue)
* **TLE** (Dark Grey/Blue)
### Detailed Analysis
**Agentbench (Left Panel):**
* **Operating System:** The bar is dominated by **Completed** (~55%), followed by a small segment of **Invalid Action** (~5%), and a large segment of **TLE** (~40%).
* **Database:** Primarily **Completed** (~90%), with a small **TLE** segment (~10%).
* **Knowledge Graph:** A small **Completed** segment (~15%), a very large **Invalid Action** segment (~80%), and a tiny **TLE** segment (~5%).
* **ALFWord:** A moderate **Completed** segment (~25%) and a very large **TLE** segment (~75%).
* **Web shopping:** A large **Completed** segment (~80%), a small **Invalid Format** segment (~5%), and a moderate **TLE** segment (~15%).
* **Mind2Web:** A large **Completed** segment (~85%) and a moderate **TLE** segment (~15%).
**SAGE (Right Panel):**
* **Operating System:** **Completed** (~60%), a small **Invalid Action** segment (~5%), and a large **TLE** segment (~35%).
* **Database:** **Completed** (~55%), a moderate **Invalid Format** segment (~20%), and a moderate **TLE** segment (~25%).
* **Knowledge Graph:** A small **Completed** segment (~15%), a tiny **CLE** segment (~1%), a very large **Invalid Action** segment (~65%), and a moderate **TLE** segment (~19%).
* **ALFWord:** A moderate **Completed** segment (~30%), a tiny **CLE** segment (~1%), and a very large **TLE** segment (~69%).
* **Web shopping:** A moderate **Completed** segment (~35%), a tiny **CLE** segment (~1%), a very large **Invalid Format** segment (~55%), and a small **TLE** segment (~9%).
* **Mind2Web:** A large **Completed** segment (~75%), a small **Invalid Action** segment (~5%), and a moderate **TLE** segment (~20%).
### Key Observations
1. **Performance Shift:** The "Completed" rate is generally higher or comparable in SAGE for most tasks (Operating System, Knowledge Graph, ALFWord, Mind2Web) but is significantly lower for "Database" and "Web shopping."
2. **Error Type Redistribution:** SAGE shows a dramatic increase in **Invalid Format** errors, especially for "Web shopping" (where it becomes the dominant error) and "Database." Agentbench had almost no Invalid Format errors.
3. **Persistent TLE:** The **TLE** (Time Limit Exceeded?) error remains a major component across most tasks in both systems, particularly for "ALFWord."
4. **New Error Category:** The **CLE** error category appears in SAGE (for Knowledge Graph, ALFWord, Web shopping) but is absent in Agentbench. Its segments are very small.
5. **Task-Specific Profiles:** Each task has a distinct error profile. "Knowledge Graph" is dominated by **Invalid Action** in both systems. "ALFWord" is dominated by **TLE** in both. "Web shopping" shifts from being mostly Completed in Agentbench to having a massive **Invalid Format** problem in SAGE.
### Interpretation
This chart provides a diagnostic breakdown of agent performance beyond simple success rates. It reveals that while SAGE may improve completion rates on some complex reasoning tasks (like Knowledge Graph, ALFWord), it introduces or exacerbates different failure modes, particularly related to output formatting (**Invalid Format**). The near-disappearance of **Invalid Action** in SAGE for "Web shopping" (replaced by **Invalid Format**) suggests a change in how the agent interacts with the environment—perhaps it attempts valid actions but structures its commands incorrectly.
The persistent, large **TLE** segments indicate that computational or step limits are a fundamental bottleneck for both systems on certain tasks. The appearance of **CLE** in SAGE, though minor, points to a new, possibly context-length related, failure mode introduced by the system's architecture.
In essence, the data suggests that SAGE is not uniformly better; it trades one set of error types (like Invalid Action) for another (Invalid Format, CLE), while struggling with the same core limitations (TLE) as Agentbench. This highlights that improving agent robustness requires addressing multiple, distinct failure axes simultaneously.
</details>
Figure 3: Execution results across six tasks (CLE: Context Limit Exceeded, TLE: Task Limit Exceeded). Task limits are the main cause of incomplete tasks, highlighting LLM agents’ limitations under time constraints.
### 4.7 Ablation Study
We conducted ablation experiments on Qwen-1.8B and CodeLlama-7B to evaluate memory optimization (Table 5). Without memory optimization, both models perform weakly, especially Qwen-1.8B, which improves from 6.8 to 48.0 in KG and from 0.0 to 10.5 in ALF after optimization. Similarly, CodeLlama-7B shows substantial gains, particularly in DB (2.7 to 41.3) and WS (14.3 to 58.7). Overall, CodeLlama-7B performs better than Qwen-1.8B, highlighting the stronger adaptability of models with more parameters in handling complex tasks.
Table 5: Ablation study for memory optimization on the task of AgentBench
| Models | OS | DB | KG | ALF | WS | M2W |
| --- | --- | --- | --- | --- | --- | --- |
| Qwen-1.8B (w/o memo) | 10.4 | 22.6 | 6.8 | 0.0 | 26.6 | 5.0 |
| Qwen-1.8B (w memo) | 18.7 | 28.3 | 45.3 | 10.5 | 31.4 | 25.1 |
| Codellama-7B (w/o memo) | 9.7 | 2.7 | 0.0 | 0.0 | 14.3 | 5.0 |
| Codellama-7B (w memo) | 23.4 | 41.3 | 48.0 | 12.5 | 58.7 | 15.0 |
## 5 Conclusion
In this paper, we propose the SAGE framework, which enhances agents’ self-adjustment and memory management in complex tasks through reflective mechanisms and memory optimization. Experimental results show significant performance improvements across benchmarks, especially in smaller models. In the AgentBench test, SAGE boosts the performance of strong baselines like GPT-3.5 and GPT-4, while also significantly improving open-source models. It effectively reduces basic errors and logical mistakes, particularly enabling smaller models to handle complex tasks.
## References
- Akash et al. (2023) Pritom Saha Akash, Kashob Kumar Roy, Lucian Popa, and Kevin Chen-Chuan Chang. Long-form question answering: An iterative planning-retrieval-generation approach, 2023. URL https://arxiv.org/abs/2311.09383.
- Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023.
- Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
- Cover & Thomas (2006) Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006. ISBN 0471241954.
- Cui et al. (2020) Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. MuTual: A dataset for multi-turn dialogue reasoning. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1406–1416, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.130. URL https://aclanthology.org/2020.acl-main.130/.
- Deng et al. (2023) Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023.
- Ebbinghaus (1885) Hermann Ebbinghaus. Über das gedächtnis: untersuchungen zur experimentellen psychologie. Duncker & Humblot, 1885.
- Eisenschlos et al. (2022) Julian Eisenschlos, Yury Zemlyanskiy, Jason Phang, Bhuwan Dhingra, and William W. Cohen. TART: Retrieval-augmented generation for task-oriented dialog. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1866–1881, 2022. URL https://aclanthology.org/2022.acl-long.131.
- Etezadi & Shamsfard (2023) Romina Etezadi and Mehrnoush Shamsfard. The state of the art in open domain complex question answering: a survey. Applied Intelligence, 53(4):4124–4144, 2023.
- Fudenberg & Tirole (1991) Drew Fudenberg and Jean Tirole. Game Theory. MIT Press, Cambridge, MA, 1991.
- GLM et al. (2024) Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024.
- Graves et al. (2016) Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471–476, 2016.
- Guo et al. (2023) Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. Longcoder: A long-range pre-trained language model for code completion, 2023.
- He et al. (2024) Yangfan He, Jianhui Wang, Kun Li, Yijin Wang, Li Sun, Jun Yin, Miao Zhang, and Xueqian Wang. Enhancing intent understanding for ambiguous prompt: A human-machine co-adaption strategy. Available at SSRN 5119629, 2024.
- He et al. (2025) Yangfan He, Jianhui Wang, Kun Li, Yijin Wang, Li Sun, Jun Yin, Miao Zhang, and Xueqian Wang. Enhancing intent understanding for ambiguous prompts through human-machine co-adaptation. arXiv preprint arXiv:2501.15167, 2025.
- Huang et al. (2022) Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve, 2022.
- Izacard & Grave (2021) Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL), pp. 874–880, 2021. URL https://aclanthology.org/2021.eacl-main.24.
- Joshi et al. (2017a) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601–1611, Vancouver, Canada, July 2017a. Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URL https://aclanthology.org/P17-1147.
- Joshi et al. (2017b) Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017b.
- Kool et al. (2019) Wouter Kool, Herke van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement, 2019. URL https://arxiv.org/abs/1903.06059.
- Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466, 2019. doi: 10.1162/tacl_a_00276. URL https://aclanthology.org/Q19-1026/.
- Li et al. (2024) Zeyuan Li, Yangfan He, Lewei He, Jianhui Wang, Tianyu Shi, Bin Lei, Yuchen Li, and Qiuwu Chen. Falcon: Feedback-driven adaptive long/short-term memory reinforced coding optimization system. arXiv preprint arXiv:2410.21349, 2024.
- Liang et al. (2024) Xuechen Liang, Meiling Tao, Yinghui Xia, Tianyu Shi, Jun Wang, and JingSong Yang. Cmat: A multi-agent collaboration tuning framework for enhancing small language models. arXiv preprint arXiv:2404.01663, 2024.
- Liu et al. (2023a) Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems, 2023a.
- Liu et al. (2024) Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=pPjZIOuQuF.
- Liu et al. (2023b) Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as agents. arXiv preprint arXiv: 2308.03688, 2023b.
- Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015.
- Mohler et al. (2016) Michael Mohler, Mary Brunson, Bryan Rink, and Marc Tomlinson. Introducing the LCC metaphor datasets. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pp. 4221–4227, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https://aclanthology.org/L16-1668.
- Nakajima (2024) Yohei Nakajima. Babyagi: An ai agent that can achieve goals and execute tasks, 2024.
- OpenAI (2023) OpenAI. Openai cookbook. https://github.com/openai/openai-cookbook, 2023. Accessed: 2025-03-27.
- OpenAI et al. (2024) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774.
- Packer et al. (2024) Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Memgpt: Towards llms as operating systems, 2024.
- Qiu et al. (2024) Xiangfei Qiu, Jilin Hu, Lekui Zhou, Xingjian Wu, Junyang Du, Buang Zhang, Chenjuan Guo, Aoying Zhou, Christian S. Jensen, Zhenli Sheng, and Bin Yang. Tfb: Towards comprehensive and fair benchmarking of time series forecasting methods. In Proc. VLDB Endow., pp. 2363–2377, 2024.
- Qiu et al. (2025a) Xiangfei Qiu, Xiuwen Li, Ruiyang Pang, Zhicheng Pan, Xingjian Wu, Liu Yang, Jilin Hu, Yang Shu, Xuesong Lu, Chengcheng Yang, Chenjuan Guo, Aoying Zhou, Christian S. Jensen, and Bin Yang. Easytime: Time series forecasting made easy. In ICDE, 2025a.
- Qiu et al. (2025b) Xiangfei Qiu, Xingjian Wu, Yan Lin, Chenjuan Guo, Jilin Hu, and Bin Yang. Duet: Dual clustering enhanced multivariate time series forecasting. In SIGKDD, pp. 1185–1196, 2025b.
- Rae et al. (2019) Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019.
- Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020.
- Reichman & Heck (2024) Benjamin Reichman and Larry Heck. Dense passage retrieval: Is it retrieving?, 2024. URL https://arxiv.org/abs/2402.11035.
- Robertson & Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009. ISSN 1554-0669. doi: 10.1561/1500000019. URL http://dx.doi.org/10.1561/1500000019.
- Rozière et al. (2024) Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024. URL https://arxiv.org/abs/2308.12950.
- Shinn et al. (2023) Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023.
- Shridhar et al. (2021) Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL https://arxiv.org/abs/2010.03768.
- Stephens et al. (2012) Keri K Stephens, Jaehee K Cho, and Dawna I Ballard. Simultaneity, sequentiality, and speed: Organizational messages about multiple-task completion. Human Communication Research, 38(1):23–47, 2012.
- Sutton & Barto (2018) Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2 edition, 2018.
- Tian et al. (2024) Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. Toward self-improvement of llms via imagination, searching, and criticizing, 2024.
- Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
- Wada et al. (2019) Takashi Wada, Tomoharu Iwata, and Yuji Matsumoto. Unsupervised multilingual word embedding with limited resources using neural language models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3113–3124, 2019.
- Yang et al. (2023) Hui Yang, Sifu Yue, and Yunzhong He. Auto-gpt for online decision making: Benchmarks and additional opinions, 2023. URL https://arxiv.org/abs/2306.02224.
- Yang et al. (2018a) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369–2380, Brussels, Belgium, October-November 2018a. Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. URL https://aclanthology.org/D18-1259.
- Yang et al. (2018b) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018b. URL https://arxiv.org/abs/1809.09600.
- Yao et al. (2023a) Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023a.
- Yao et al. (2023b) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023b. URL https://arxiv.org/abs/2210.03629.
- Zhou et al. (2024) Ziqi Zhou, Jingyue Zhang, Jingyuan Zhang, Yangfan He, Boyue Wang, Tianyu Shi, and Alaa Khamis. Human-centric reward optimization for reinforcement learning-based automated driving using large language models. arXiv preprint arXiv:2405.04135, 2024.
<details>
<summary>x4.png Details</summary>
### Visual Description
## Screenshot: AI Feedback and Correction Log
### Overview
The image is a screenshot of a textual interface, likely from an AI training or evaluation system. It displays a sequence of interactions: a user's query, an initial assistant response, an automated checker's critique, a detailed reflection on the error, and a final corrected response. The document demonstrates a process of error identification and correction in an AI model's output.
### Components/Axes
The content is organized into distinct, labeled sections with specific formatting:
1. **User Section:** Contains the initial prompt and question.
2. **Assistant Section:** Shows the initial, incorrect answer.
3. **Checker Section:** Header in red. Provides a critique of the initial answer.
4. **Reflection Section:** Header in red. Contains a detailed self-analysis of the error and proposed improvements.
5. **Assistant with SAGE response Section:** Shows the final, corrected answer.
**Visual Formatting:**
* The headers "Checker:" and "Reflection:" have a red background.
* Specific lines of text within the "Checker" and "Reflection" sections are highlighted with a light blue background.
* The text is primarily black on a white background, using a standard sans-serif font.
### Content Details
**1. User Section:**
* **Instruction:** "Answer the question based on the given passages. Only give me the answer and do not output any other words. The following are given passages:"
* **Passage Text:** "The "trusty system" (sometimes incorrectly called "trustee system") was a penitentiary system of discipline and security enforced in parts of the United States until the 1980s ....."
* **Repeated Instruction:** "Answer the question based on the given passages. Only give me the answer and do not output any other words."
* **Question:** "Which case was brought to court first Miller v. California or Gates v. Collier ?"
* **Label:** "Answer:"
**2. Assistant Section:**
* **Response:** "Gates v. Collier"
**3. Checker Section (Header in Red):**
* **Critique Point 1 (Highlighted in Blue):** "Incorrect answer: Neglecting specific years: Assistant did not accurately confirm the specific years of each case, leading to an incorrect judgment of the chronological order."
* **Critique Point 2:** "Insufficient understanding of the question: When answering the question, Assistant may not have fully understood or carefully analyzed the provided information, resulting in a misjudgment."
**4. Reflection Section (Header in Red):**
* **Error Analysis (Highlighted in Blue):** "When answering the question about which case, "Miller v. California" or "Gates v. Collier," was brought up first, I mistakenly thought "Gates v. Collier" preceded "Miller v. California." In reality, "Miller v. California" came first. This indicates that I made an error in judging the chronological order of events."
* **Improvement Measures:**
* **Carefully verify information:** "When answering questions involving time sequences, it is essential to carefully check the specific timing of each event to ensure accuracy."
* **Timeline tools:** "When dealing with multiple events or cases, use timeline tools to help clarify the chronological order and avoid oversights."
* **Review mechanism:** "Before submitting the final answer, establish a self-review mechanism to recheck all relevant information and details to ensure the accuracy of the response."
* **Conclusion:** "This mistake serves as a reminder for me to be more careful and cautious when handling questions related to time and sequence. Through reflection and improvement, I hope to provide more accurate and reliable answers in the future and prevent similar errors from occurring."
**5. Assistant with SAGE response Section:**
* **Corrected Response:** "Miller v. California"
### Key Observations
* The initial assistant answer ("Gates v. Collier") is directly contradicted by the final corrected answer ("Miller v. California").
* The "Checker" and "Reflection" sections explicitly diagnose the error as a failure to verify the specific chronological facts (the years the cases were brought).
* The reflection outlines a structured improvement plan involving verification, tool use, and self-review.
* The highlighted text (blue) draws attention to the core error and its analysis.
### Interpretation
This document is a meta-example of an AI learning process. It doesn't contain raw data or charts but instead presents a **case study in error correction**.
* **What it demonstrates:** The system is designed not just to answer questions but to evaluate its own performance, identify specific failure modes (here, factual inaccuracy regarding chronology), and articulate concrete steps for improvement.
* **Relationship between elements:** The sequence forms a closed loop: Query -> Error -> Diagnosis -> Reflection -> Correction. The "Checker" acts as an external validator, while the "Reflection" internalizes the feedback into a learning protocol.
* **Underlying significance:** The content highlights a critical challenge for AI systems: moving beyond pattern matching to accurate factual recall and temporal reasoning. The proposed solutions (verification, timelines, review mechanisms) are procedural safeguards against such errors. The document itself serves as a training artifact, illustrating how models can be fineuned or prompted to incorporate self-correction mechanisms.
</details>
Figure 4: The illustration of an example HotpotQA with SAGE.
## Appendix A Theoretical Analysis
### A.1 An Information-Theoretic Perspective on Memory Optimization Based on the Ebbinghaus Forgetting Curve
The memory optimization mechanism proposed in this work is grounded in the Ebbinghaus forgetting curve. It dynamically manages short-term memory (STM) and long-term memory (LTM) by prioritizing the retention of essential information while discarding irrelevant data. This mechanism can be viewed as computing the retention strength $S(I_{t})$ of each information unit based on its information entropy. Theoretically, one can interpret this as maximizing information entropy to show that the mechanism effectively mitigates the issue of information overload Cover & Thomas (2006). That is, by preferentially retaining high-entropy (i.e., high-uncertainty) information, the model improves its information processing efficiency.
First, the retention strength of information is defined as:
$$
S(I_{t})=\frac{H(I_{t})}{f(t)} \tag{13}
$$
where the numerator $H(I_{t})$ represents the information entropy of the content at time $t$ , and $f(t)$ models the forgetting curve.
The objective of memory optimization is to maximize the utility of retained information under a memory capacity constraint:
$$
\max\sum_{t}M_{t}\cdot S(I_{t}),\quad\text{s.t.}\quad\sum_{t}M_{t}\leq C \tag{14}
$$
Here, $C$ represents the total memory capacity, and $M_{t}\in\{0,1\}$ denotes whether the information at time $t$ is retained (1) or discarded (0).
Using the method of Lagrange multipliers, we construct the Lagrangian function:
$$
\mathcal{L}(M_{t},\lambda)=\sum_{t}M_{t}\cdot S(I_{t})-\lambda\left(\sum_{t}M_
{t}-C\right) \tag{15}
$$
Taking the derivative of $\mathcal{L}$ , which is the optimal memory retnetion, with respect to $M_{t}$ and setting it to zero yields the optimality condition:
$$
\frac{\partial\mathcal{L}}{\partial M_{t}}=S(I_{t})-\lambda=0\quad\Rightarrow
\quad\begin{cases}\text{Retain }I_{t}&\text{if }S(I_{t})\geq\lambda\\
\text{Discard }I_{t}&\text{if }S(I_{t})<\lambda\end{cases} \tag{16}
$$
This rule is consistent with the threshold-based memory retention strategy used in the SAGE framework.
### A.2 Iterative Feedback and Game-Theoretic Model
Multi-agent systems often depend on reinforcement learning (RL) methods, making their theoretical basis essential for ensuring that algorithms converge to stable solutions, such as Nash equilibria Fudenberg & Tirole (1991). The subsequent theory provides a formal guarantee of convergence, showing that our proposed algorithm is not randomly designed. Instead, it surpasses standard approaches by establishing robust criteria that confirm both its stability and performance advantages.
#### A.2.1 Players and Their Strategy Spaces Settings
$$
\begin{array}[]{rcl}\textbf{User (U)}&:&\text{Provides a task description }T_{
U}\text{ and constraints }C_{U}.\\[2.84526pt]
\textbf{Assistant (A)}&:&\text{Chooses actions }a_{t}\sim\pi_{\theta}(\cdot
\mid s_{t}),\text{ where }\pi_{\theta}\text{ is the policy}.\\[2.84526pt]
\textbf{Checker (C)}&:&\text{Generates feedback }f_{t}\text{ aimed at reducing
the error }\|o_{t}-o^{*}\|_{F},\end{array} \tag{17}
$$
where $o_{t}$ is the output of the assistant and $o^{*}$ is the expected (ground-truth) output, $\|\cdot\|_{F}$ denotes the Frobenius norm.
For the Utility Functions, we have:
$$
\begin{array}[]{rcl}\textbf{Assistant\textquoteright s Utility, }R_{A}&:&\text
{Measures task performance (e.g., BLEU score, accuracy):}\\
&&R_{A}=\text{Task Performance Metric of }a_{t}.\\[2.84526pt]
\textbf{Checker\textquoteright s Utility, }R_{C}&:&\text{Measures feedback
accuracy by reduction of error:}\\
&&R_{C}\propto\Delta\|o_{t}-o^{*}\|_{F}.\end{array} \tag{18}
$$
The overall interaction can be viewed as a three-player game in which: The assistant (A) optimizes its policy $\pi_{\theta}$ to maximize $R_{A}$ , while the checker (C) aims to minimize $\|o_{t}-o^{*}\|$ (i.e., maximize $R_{C}$ ) The user U sets the task and constraints that guide both the assistant and checker’s performances. Next we want to show that it can be shown that under a finite strategy space and continuous utility functions, a Nash equilibrium exists in this multi-agent game.
#### A.2.2 Setup for the proof of theorem 3.1
First, we want to write a road map for this proof.
- It can be shown that under a finite strategy space and continuous utility functions, a Nash equilibrium exists in this multi-agent game.
- According to the Debreu-Glicksberg-Fan Theorem (a generalization of Nash’s existence theorem), if each player’s strategy space is a compact convex set and the utility functions are continuous and quasi-concave (or concave) in their own strategies, then at least one Nash equilibrium exists.
- Then, the assistant’s policy $\pi_{\theta}$ is updated using policy gradient methods. Since $R_{A}$ is a continuous function of $\pi_{\theta}$ , the assistant’s utility is continuous.
- After that, the checker’s feedback strategy can be obtained via convex optimization (e.g., least-squares regression), and thus $R_{C}$ is a convex function.
- Therefore, a Nash equilibrium $(\pi_{\theta}^{*},f^{*})$ exists, at which both the assistant and checker’s strategies are stable with respect to each other:
$$
\pi_{\theta}^{*}=\arg\max_{\pi}R_{A}(\pi,f^{*}),\quad f^{*}=\arg\max_{f}R_{C}(
\pi_{\theta}^{*},f) \tag{19}
$$
Let’s start the proof.
**Lemma A.1**
*Let $\mathcal{S}_{1}$ , $\mathcal{S}_{2},\dots,\mathcal{S}_{N}$ be finite strategy sets for each of $N$ players, and let $u_{i}:\mathcal{S}_{1}\times\cdots\times\mathcal{S}_{N}\to\mathbb{R}$ be continuous for each $i$ . Then, there exists a Nash equilibrium in the multi-agent game.*
* Proof*
For each player $i$ , define the set of mixed strategies as
$$
\Delta(\mathcal{S}_{i})=\left\{x_{i}\in\mathbb{R}^{|\mathcal{S}_{i}|}\ \Big{|}
\ x_{i}(s)\geq 0\text{ for all }s\in\mathcal{S}_{i},\quad\sum_{s\in\mathcal{S}
_{i}}x_{i}(s)=1\right\}. \tag{20}
$$
Each set $\Delta(\mathcal{S}_{i})$ is a simplex in $\mathbb{R}^{|\mathcal{S}_{i}|}$ , and hence is nonempty, compact, and convex. Define the expected utility function for player $i$ :
$$
U_{i}(x_{1},\dots,x_{N})=\sum_{(s_{1},\dots,s_{N})\in\mathcal{S}_{1}\times
\cdots\times\mathcal{S}_{N}}\left(\prod_{j=1}^{N}x_{j}(s_{j})\right)u_{i}(s_{1
},\dots,s_{N}). \tag{21}
$$
Since the strategy spaces are finite and $u_{i}$ is continuous (and hence bounded on the finite set), the function $U_{i}$ is continuous on the product space $\prod_{j=1}^{N}\Delta(\mathcal{S}_{j})$ . For each player $i$ , define the best response correspondence as:
$$
BR_{i}(x_{-i})=\arg\max_{x_{i}\in\Delta(\mathcal{S}_{i})}U_{i}(x_{i},x_{-i}), \tag{22}
$$
where $x_{-i}$ denotes the mixed strategies of all players other than $i$ . Because $\Delta(\mathcal{S}_{i})$ is compact and $U_{i}$ is continuous and linear in $x_{i}$ (hence concave), the maximum exists and $BR_{i}(x_{-i})$ is nonempty, convex, and compact. Now we define the overall best-response correspondence:
$$
BR(x)=BR_{1}(x_{-1})\times BR_{2}(x_{-2})\times\cdots\times BR_{N}(x_{-N}), \tag{23}
$$
where $x=(x_{1},x_{2},\dots,x_{N})$ . The product space $\prod_{i=1}^{N}\Delta(\mathcal{S}_{i})$ is compact and convex, and the correspondence $BR$ has nonempty, convex, and compact values and is upper hemicontinuous (due to the continuity and linearity of the expected utilities). Hence, by Kakutani’s fixed point theorem, there exists a fixed point
$$
x^{*}=(x_{1}^{*},x_{2}^{*},\dots,x_{N}^{*})\in\prod_{i=1}^{N}\Delta(\mathcal{S
}_{i}) \tag{24}
$$
such that
$$
x^{*}\in BR(x^{*}). \tag{25}
$$
This fixed point is a Nash equilibrium in mixed strategies. Thus, under a finite strategy space and continuous utility functions, a Nash equilibrium exists. According to the Debreu-Glicksberg-Fan Theorem (a generalization of Nash’s existence theorem), if each player’s strategy space is a compact convex set and the utility functions are continuous and quasi-concave (or concave) in their own strategies, then at least one Nash equilibrium exists. ∎
**Lemma A.2**
*By Lemma A.1, Let $I=\{1,2,\dots,n\}$ be the set of players. For each player $i$ , let $X_{i}\subset\mathbb{R}^{d_{i}}$ be a compact, convex strategy set, and define $X=\prod_{i=1}^{n}X_{i}$ . Suppose each player’s utility function $u_{i}:X\to\mathbb{R}$ is continuous on $X$ and quasi-concave (or concave) in the coordinate $x_{i}$ for fixed $x_{-i}$ . Then, by the Debreu-Glicksberg-Fan theorem, there exists at least one Nash equilibrium $x^{*}=(x_{1}^{*},x_{2}^{*},\dots,x_{n}^{*})\in X$ .*
* Proof*
For each player $i$ , define the best-response correspondence
$$
BR_{i}(x_{-i})=\{x_{i}\in X_{i}:u_{i}(x_{i},x_{-i})=\max_{y_{i}\in X_{i}}u_{i}
(y_{i},x_{-i})\}. \tag{26}
$$
Since $u_{i}$ is continuous and $X_{i}$ is compact, the maximum is attained; hence, $BR_{i}(x_{-i})\neq\varnothing$ . The quasi-concavity (or concavity) of $u_{i}$ in $x_{i}$ ensures that the set
$$
\{x_{i}\in X_{i}:u_{i}(x_{i},x_{-i})\geq\alpha\} \tag{27}
$$
is convex for any $\alpha\in\mathbb{R}$ . In particular, the set $BR_{i}(x_{-i})$ is convex for every fixed $x_{-i}$ . The continuity of $u_{i}$ implies that the best-response correspondence $BR_{i}$ is upper hemicontinuous. This means that for any sequence $\{x_{-i}^{(k)}\}$ converging to $x_{-i}$ and any corresponding sequence $\{x_{i}^{(k)}\}$ with $x_{i}^{(k)}\in BR_{i}(x_{-i}^{(k)})$ , every limit point $x_{i}$ of $\{x_{i}^{(k)}\}$ belongs to $BR_{i}(x_{-i})$ . Now we define the aggregate best-response correspondence:
$$
BR(x)=BR_{1}(x_{-1})\times BR_{2}(x_{-2})\times\cdots\times BR_{n}(x_{-n}), \tag{28}
$$
where $x=(x_{1},x_{2},\dots,x_{n})\in X$ . Note that: (1) $X$ is a nonempty, compact, convex subset of $\mathbb{R}^{d_{1}+d_{2}+\dots+d_{n}}$ . (2) For each $i$ , $BR_{i}(x_{-i})$ is nonempty and convex. (3) The correspondence $BR$ is upper hemicontinuous. By the Debreu-Glicksberg-Fan fixed point theorem, there exists a fixed point $x^{*}\in X$ such that $x^{*}\in BR(x^{*})$ . That is, for every $i\in I$ , $x_{i}^{*}\in BR_{i}(x_{-i}^{*})$ . This means that no player can unilaterally deviate from $x^{*}$ and increase their utility, which is precisely the definition of a Nash equilibrium. ∎
Next, based on the above Lemmas, we want to prove that the assistant’s policy $\pi_{\theta}$ is updated using policy gradient methods Sutton & Barto (2018). Since $R_{A}$ is a continuous function of $\pi_{\theta}$ , the assistant’s utility is continuous.
**Lemma A.3**
*Let $\Theta$ denote the parameter space for the assistant’s policy $\pi_{\theta}$ , and let the assistant’s utility be given by
$$
R_{A}(\pi_{\theta})=f\big{(}\pi_{\theta}\big{)}, \tag{29}
$$
where $f:\Pi\to\mathbb{R}$ is a continuous function on the space $\Pi$ of policies. Assume that the mapping $\theta\mapsto\pi_{\theta}$ is continuous. Then the composite function
$$
F(\theta)=f\big{(}\pi_{\theta}\big{)} \tag{30}
$$
is continuous in $\theta$ . Moreover, since the assistant’s policy is updated using policy gradient methods, the update rule
$$
\theta_{k+1}=\theta_{k}+\alpha_{k}\nabla_{\theta}F(\theta_{k}) \tag{31}
$$
employs the gradient of a continuous function.*
* Proof*
We divide the proof into two parts: Step 1: Continuity of the Utility Function. Since the mapping $\theta\mapsto\pi_{\theta}$ is continuous and the function $f:\Pi\to\mathbb{R}$ is continuous by hypothesis, their composition $F(\theta)=f\big{(}\pi_{\theta}\big{)}$ is continuous in $\theta$ . Formally, for any $\epsilon>0$ , there exists a $\delta>0$ such that for all $\theta,\theta^{\prime}\in\Theta$ satisfying $\|\theta-\theta^{\prime}\|<\delta$ ,
$$
|F(\theta)-F(\theta^{\prime})|=\big{|}f\big{(}\pi_{\theta}\big{)}-f\big{(}\pi_
{\theta^{\prime}}\big{)}\big{|}<\epsilon. \tag{32}
$$ Step 2: Policy Gradient Updates. In the policy gradient framework, the assistant updates its parameters according to the rule:
$$
\theta_{k+1}=\theta_{k}+\alpha_{k}\nabla_{\theta}F(\theta_{k}), \tag{33}
$$
where $\alpha_{k}>0$ is a (possibly diminishing) step size and $\nabla_{\theta}F(\theta_{k})$ denotes the gradient of $F$ evaluated at $\theta_{k}$ . Since $F$ is continuous (and under typical smoothness assumptions, differentiable with continuous gradient), the gradient $\nabla_{\theta}F$ is well-defined. The policy gradient update is thus a continuous map from $\theta_{k}$ to $\theta_{k+1}$ . Because the update at each iteration is based on the gradient (which is continuous) of the assistant’s utility $F(\theta)$ , any small perturbation in $\theta$ results in only a small variation in both the gradient and the updated parameter. This demonstrates that the update procedure is stable in the sense that the assistant’s utility, $R_{A}(\pi_{\theta})=F(\theta)$ , continuously varies under the sequence of policy gradient updates Sutton & Barto (2018). Thus, since $R_{A}$ is a continuous function of $\pi_{\theta}$ and $\pi_{\theta}$ depends continuously on $\theta$ , the assistant’s utility is continuous, and the policy gradient update is well-defined. ∎
Then we want to prove that, the checker’s feedback strategy can be obtained via convex optimization (e.g., least-squares regression), and thus $R_{C}$ is a convex function.
**Lemma A.4**
*Let $\mathcal{F}$ be a convex subset of a vector space, and suppose the checker’s feedback strategy is given by a function $f\in\mathcal{F}$ obtained as a solution to the convex optimization problem
$$
\min_{f\in\mathcal{F}}\quad J(f)=\sum_{i=1}^{n}\big{(}y_{i}-f(x_{i})\big{)}^{2}, \tag{34}
$$
where $x_{i}$ are given inputs and $y_{i}$ are target outputs. Define the checker’s utility function
$$
R_{C}(f)=J(f). \tag{35}
$$
Then $R_{C}$ is a convex function on $\mathcal{F}$ ; that is, for any $f_{1},f_{2}\in\mathcal{F}$ and any $\lambda\in[0,1]$ it holds that
$$
R_{C}\big{(}\lambda f_{1}+(1-\lambda)f_{2}\big{)}\leq\lambda R_{C}(f_{1})+(1-
\lambda)R_{C}(f_{2}). \tag{36}
$$*
* Proof*
We prove the convexity of $R_{C}$ by verifying that the squared-error loss is a convex function and that convex combinations preserve convexity. For each data point $i$ , consider the function
$$
\ell_{i}(f)=\big{(}y_{i}-f(x_{i})\big{)}^{2}. \tag{37}
$$
Since for any real number $t$ , the function $t^{2}$ is convex, and the mapping $f\mapsto y_{i}-f(x_{i})$ is affine (and hence convex), the composition $f\mapsto\big{(}y_{i}-f(x_{i})\big{)}^{2}$ is convex. In particular, for any $f_{1},f_{2}\in\mathcal{F}$ and $\lambda\in[0,1]$ ,
$$
\ell_{i}\big{(}\lambda f_{1}+(1-\lambda)f_{2}\big{)}\leq\lambda\ell_{i}(f_{1})
+(1-\lambda)\ell_{i}(f_{2}). \tag{38}
$$ Since the checker’s overall loss (and hence utility function)
$$
R_{C}(f)=\sum_{i=1}^{n}\ell_{i}(f) \tag{39}
$$
is a finite sum of convex functions, it follows that $R_{C}$ is convex on $\mathcal{F}$ ; that is, for any $f_{1},f_{2}\in\mathcal{F}$ and $\lambda\in[0,1]$ ,
$$
\displaystyle R_{C}\big{(}\lambda f_{1}+(1-\lambda)f_{2}\big{)} \displaystyle=\sum_{i=1}^{n}\ell_{i}\big{(}\lambda f_{1}+(1-\lambda)f_{2}\big{)} \displaystyle\leq\lambda\sum_{i=1}^{n}\ell_{i}(f_{1})+(1-\lambda)\sum_{i=1}^{n
}\ell_{i}(f_{2}) \displaystyle=\lambda R_{C}(f_{1})+(1-\lambda)R_{C}(f_{2}). \tag{40}
$$
Thus:
$$
R_{C}(f)=\sum_{i=1}^{n}\big{(}y_{i}-f(x_{i})\big{)}^{2}\text{ is convex on }
\mathcal{F}. \tag{41}
$$
∎
Thus, the checker’s feedback strategy derived via convex optimization (e.g., least-squares regression) ensures that the corresponding utility function $R_{C}$ is convex.
#### A.2.3 The formal proof of theorem 3.1
We want to rewrite the theorem 3.1 into the following lemma based on the former lemmas:
**Lemma A.5 (theorem3.1)**
*Suppose the assistant’s strategy space $\Pi$ and the checker’s strategy space $\mathcal{F}$ are nonempty, compact, and convex sets. Assume that: $R_{A}:\Pi\times\mathcal{F}\to\mathbb{R}$ is continuous in $\pi$ for fixed $f$ , $R_{C}:\Pi\times\mathcal{F}\to\mathbb{R}$ is convex in $f$ for fixed $\pi$ . Then there exists a Nash equilibrium $\pi_{\theta}^{*},f^{*}$ defined by
$$
\pi_{\theta}^{*}=\arg\max_{\pi\in\Pi}R_{A}(\pi,f^{*})\quad\text{and}\quad f^{*
}=\arg\max_{f\in\mathcal{F}}R_{C}(\pi_{\theta}^{*},f). \tag{42}
$$*
* Proof*
For each fixed $f\in\mathcal{F}$ define the assistant’s best-response set:
$$
BR_{A}(f)=\{\pi\in\Pi:R_{A}(\pi,f)=\max_{\pi^{\prime}\in\Pi}R_{A}(\pi^{\prime}
,f)\}. \tag{43}
$$
By hypothesis (i) and the compactness of $\Pi$ , the maximum is attained and $BR_{A}(f)$ is nonempty. (Continuity of $R_{A}$ assures that small changes in $f$ lead to small changes in $R_{A}(\cdot,f)$ .) Similarly, for each fixed $\pi\in\Pi$ define the checker’s best-response set:
$$
BR_{C}(\pi)=\{f\in\mathcal{F}:R_{C}(\pi,f)=\max_{f^{\prime}\in\mathcal{F}}R_{C
}(\pi,f^{\prime})\}. \tag{44}
$$
Since hypothesis (ii) implies that $R_{C}$ is convex in $f$ and $\mathcal{F}$ is convex and compact, we conclude that $BR_{C}(\pi)$ is nonempty, convex, and closed. Consider the correspondence
$$
H:\Pi\times\mathcal{F}\to\Pi\times\mathcal{F},\quad H(\pi,f)=BR_{A}(f)\times BR
_{C}(\pi). \tag{45}
$$
By the arguments above: For every $(\pi,f)\in\Pi\times\mathcal{F}$ , the set $H(\pi,f)$ is nonempty. $BR_{A}(f)$ is closed (by continuity and compactness) and $BR_{C}(\pi)$ is convex (and closed) by hypothesis (ii). Hence, $H(\pi,f)$ is convex and closed. Since the domain $\Pi\times\mathcal{F}$ is nonempty, compact, and convex, and since the correspondence $H$ has nonempty, convex, and closed values and is upper hemicontinuous (this follows from the continuity of $R_{A}$ in $\pi$ and convexity of $R_{C}$ in $f$ ), Kakutani’s fixed point theorem guarantees the existence of a fixed point
$$
(\pi^{*},f^{*})\in\Pi\times\mathcal{F}\quad\text{such that}\quad(\pi^{*},f^{*}
)\in H(\pi^{*},f^{*}). \tag{46}
$$ By the definition of $H$ , the fixed point satisfies
$$
\pi^{*}\in BR_{A}(f^{*})\quad\text{and}\quad f^{*}\in BR_{C}(\pi^{*}), \tag{47}
$$
which is equivalent to
$$
\pi^{*}=\arg\max_{\pi\in\Pi}R_{A}(\pi,f^{*})\quad\text{and}\quad f^{*}=\arg
\max_{f\in\mathcal{F}}R_{C}(\pi^{*},f). \tag{48}
$$ Thus, no unilateral deviation by either the assistant or the checker increases the respective utility; in other words, the strategy pair $(\pi^{*},f^{*})$ is a Nash equilibrium. ∎
## Appendix B Detailed Dataset Information
- Operating systems (OS) Integrating LLMs into operating systems has immense potential for automating and optimizing tasks. This integration requires a secure, user-friendly interface to ensure effective LLM-OS interaction. LLMs must accurately understand the OS context for informed operations, prioritizing safety to prevent misuse. Additionally, the system should effectively handle errors and provide clear feedback to users, enhancing overall interaction and control. Addressing these aspects can transform computer interaction and efficiency across various industries.
- Database (DB) The ability of LLMs to operate on real databases via SQL is critical due to the importance and complexity of database analysis in everyday activities. Previous research has highlighted the effectiveness of LLMs in automating database access, such as with T5QL, a new SQL generation method. Furthermore, fine-tuned LLMs (like GPT-3.5) have demonstrated the ability to extract and link complex scientific information from texts, obtaining structured knowledge from unstructured text and building extensive databases.
- WebShop (WS) WebShop is an innovative simulation of an e-commerce website environment, featuring 1.18 million real-world products and 12,087 crowd-sourced text instructions. It challenges agents to navigate various types of webpages and perform diverse actions to find, customize, and purchase products based on given instructions. WebShop’s challenges include understanding compositional instructions, query (re-)formulation, dealing with noisy text on webpages, and strategic exploration.
- Knowledge Graphs (KG) LLMs’ utilization in constructing and interacting with knowledge graphs (KG) presents a promising opportunity to enhance semantic understanding and information retrieval. This involves assessing the models’ ability to generate and interpret complex interrelations within data, facilitating more intuitive and context-aware responses. Effective LLM performance in this domain could significantly improve AI’s reasoning and decision-making capabilities based on structured knowledge.
- Mind2Web (M2W) Mind2Web (M2W) is a dataset designed for developing web agents capable of performing complex tasks on real websites via language instructions. It features over 2,000 tasks across 137 sites from 31 domains. M2W’s real web environments and diverse user interactions make it a crucial platform for advancing AI navigation capabilities.
- ALFWorld (ALF) ALFWorld bridges interactive TextWorld environments with embodied tasks from the ALFRED dataset, enabling agents to learn abstract strategies and apply them to real-world tasks. It supports abstract reasoning and concrete execution, allowing agents to plan actions in a text-based simulator and execute these tasks in a visual environment. This approach enhances agent generalization and problem-solving skills across various domains, such as language understanding and visual navigation, by leveraging a modular design that simplifies research improvements.
## Appendix C Comparison with Task-Splitting Agents
To compare the SAGE framework with existing agents across various task types and datasets, we conducted a comprehensive evaluation. The results are detailed in Table 6. SAGE demonstrates superior performance in task completion time, answer accuracy, dialog coherence, and step completion accuracy, highlighting its effectiveness in managing complex, multi-faceted tasks.
Table 6: Evaluation of Task-Splitting SOTA Agents on Different Task Types and Datasets
| Agent | Task Completion | Answer Accuracy | Dialog Coherence | Step Completion |
| --- | --- | --- | --- | --- |
| Time (min) | (QA)(%) | (%) | Accuracy (%) | |
| Task 1: Long-form Question Answering (HotpotQA) | | | | |
| BabyAGI | 7.8 | 74.1 | – | – |
| AutoGPT | 8.1 | 76.3 | – | – |
| HuggingGPT | 6.7 | 79.2 | – | – |
| LangChain Agent | 7.9 | 77.0 | – | – |
| TaskMatrix.AI | 6.5 | 80.3 | – | – |
| SAGE - ChatGPT-4 | 10.6 | 82.5 (+6.0) | – | – |
| Task 2: Multi-turn Dialog Completion (MultiWOZ) | | | | |
| BabyAGI | 8.2 | – | 73.5 | – |
| AutoGPT | 8.5 | – | 75.6 | – |
| HuggingGPT | 7.1 | – | 78.1 | – |
| LangChain Agent | 8.3 | – | 76.5 | – |
| TaskMatrix.AI | 6.8 | – | 79.9 | – |
| SAGE - ChatGPT-4 | 10.9 | – | 81.6 (+5.8) | – |
| Task 3: Sequential Task with Dependencies (ALFWorld) | | | | |
| BabyAGI | 8.7 | – | – | 71.5 |
| AutoGPT | 8.3 | – | – | 73.1 |
| HuggingGPT | 7.5 | – | – | 75.2 |
| LangChain Agent | 7.8 | – | – | 72.8 |
| TaskMatrix.AI | 6.6 | – | – | 76.4 |
| SAGE - ChatGPT-4 | 10.7 | – | – | 82.3 (+5.9) |
## Appendix D Examples
Cases study on TriviaQA in using GPT with SAGE. User: Answer the question based on the given passages. Only give me the answer and do not output any other words.The following are given passages: History of canals in Great Britain - London Canal Museum History of canals in Great Britain Home > History > Canals in Britain In the history of canals, Britain was not a pioneer. The Chinese can claim that the Grand Canal of China was one of the first, in the tenth century,a…… Answer the question based on the given passages. Only give me the answer and do not output any other words. Question: Which metal is produced by the Bessemer Process? Answer: Assistant: The content of the file seems to be an HTML document related to the administrative division of Turkey, including population statistics, charts, and maps. This file contains scripts, metadata, and structured data typical for a web page. Checker: Clear Objectives: When answering user questions, the focus should be on directly addressing the question without introducing irrelevant information. Document Handling: Before confirming whether the document content is relevant to the question, detailed aspects of the document should not be directly addressed. Reflection: Issue Description: The user wants to understand why the system outputs irrelevant HTML document content based on user input and seeks detailed feedback and reflection. Reasons for the Error: Misinterpretation of Input: The system mistakenly assumed that the user uploaded a file and attempted to read its content, whereas the user only provided text input. Processing Logic Error: Without clearly determining the user’s intent, the system directly parsed and outputted file content. User Experience Problems: Irrelevant Output: The user expects concise, relevant answers, but the system provided lengthy and irrelevant HTML content, affecting the user experience. Poor Communication: Failure to accurately understand and respond to the user’s real needs resulted in off-topic responses In-depth Reflection and Improvement Suggestions: Clarify User Intent: Accurate Parsing of User Input: Enhance the system’s natural language processing capabilities to accurately distinguish whether the user uploaded a file or simply provided text input. User Feedback Mechanism: Implement a user feedback mechanism to promptly adjust the understanding of user intent. For example, confirm the issue to verify if the user uploaded a file. Optimize Processing Logic: Layered Processing Flow: In system design, separate file processing logic from text response logic to avoid misjudgments. For instance, when a user uploads a file, process the file content separately; when a user inputs text, provide relevant answers directly. Pre-processing Check: Prior to handling files, perform pre-processing checks to confirm if the file content meets expectations. …… LLMs with SAGE response: Steel. Ground truth Steel.
Cases study on LCC in using GPT with SAGE. User: import networkx as nx import pickle from Queue import PriorityQueue import copy import random import string import sys class MultiDiGraphEdgeKey(nx.MultiDiGraph): …… Assistant: You will be given the history of a past experience where you were given a task and failed to complete it. Reflect on the strategy and actions taken. Devise a concise, new plan of action that accounts for your previous mistakes. Here are two examples: Example 1: If you failed to remember to lock the door, plan to double-check the door each time. Example2: If you missed a deadline, plan to set reminders earlier…… Checker: The given code is a part of the Camerini algorithm, which is used for ranking edges in a directed graph. This algorithm is particularly useful for problems involving the identification of optimal or near-optimal substructures, like arborescences or spanning trees, in complex networks. The algorithm has two main phases: Collapsing Phase: This phase identifies and collapses cycles in the graph, creating supernodes. This simplification helps in managing cycles more efficiently and prepares the graph for the expansion phase. Expanding Phase: After collapsing cycles into supernodes, this phase expands the supernodes back into their original structure, resulting in a ranked arborescence, where the edges are sorted or prioritized based on certain criteria. …… LLMs with SAGE response: Z.addedge(eu, ev, **data.copy()) self. init(graph, Y, Z) A = best(root) self.init(graph) e , d = self.next(A, Y, Z) w = self.getgraphscore(A) - d if d != float(’inf’) else float(’inf’) P.put((-w, e, A, Y, Z)) solutions.append(A) return solutions def bestbranching(self, root): return self.best(root) def bestarborescencenorooted(self, root): return self.best(root) def bestarborescencerooted(self, root): return self.best(root) def getgraphscore(self, graph): return sum(data[self.attr] for u, v, data in graph.edges(data=True)) def findroots(self, graph): roots = [node for node, degree in graph.indegree() if degree == 0] return roots ……
Cases study on Repobench-P in using GPT with SAGE. User : Please complete the code given below. {src//kademlia//operation//KadRefreshOperation.java public class KadRefreshOperation implements Operation private final KadServer server; private final KademliaNode localNode; private final KademliaDHT dht; private final KadConfiguration config; public KadRefreshOperation...... Assistant: To devise a new plan of action, let’s analyze the previous mistakes and come up with a solution: Mistake 1: In the KadRefreshOperation class, the execute() method calls the BucketRefreshOperation and ContentRefreshOperation without any error handling or exception handling. This can lead to unexpected errors and failures. Solution: Add proper error handling and exception handling in the execute() method of the KadRefreshOperation class. This will ensure that any errors or exceptions are properly handled and the execution can continue smoothly…… Checker: In the ‘KadRefreshOperation‘ class, there is no error handling when the ‘execute()‘ method calls ‘BucketRefreshOperation‘ and ‘ContentRefreshOperation‘. Analysis: If ‘BucketRefreshOperation‘ or ‘ContentRefreshOperation‘ throws an exception, the entire ‘execute()‘ method will fail, potentially causing the interruption of the entire system’s refresh operation. In the ‘StoreOperation‘ class, the ‘execute()‘ method does not handle the case of existing content in the DHT. Analysis: The ‘execute()‘ method always returns 1 as the number of storage nodes, which is clearly incorrect. The correct number of nodes should be returned based on whether the content already exists in the DHT. Solution: Before storing the content, check if it already exists. If it does, return 0; otherwise, store the content and return the actual number of storage nodes. …… LLMs with SAGE response: private final transient KademliaRoutingTable routingTable; private final transient KadConfiguration config; private final transient MessageFactory messageFactory; /* Scheduled refresh operation */ private final Timer refreshTimer; /** * Initialize the Kademlia system * * @param ownerId The ID for this owner * @param nodeId The ID for this node * @param inetAddress The address for this node * @param port The port for this node * @param config Configuration for the Kad system * * @throws java.io.IOException */ public JKademliaNode(String ownerId, KademliaId nodeId, InetAddress inetAddress, int port, KadConfiguration config) throws IOException { this.ownerId = ownerId; this.config = config; /* Initialize the Node */ this.localNode = new Node(nodeId, inetAddress, port); /* Initialize the Server */ this.server = new KadServer(localNode.getNodeId(), config); ……