2507.21504

# Evaluation and Benchmarking of LLM Agents: A Survey **Authors**: Mahmoud Mohammadi, Yipeng Li, Jane Lo, Wendy Yip > 0000-0001-6829-1420 SAP Labs Bellevue WA USA > 0009-0002-4076-864X SAP Labs Bellevue WA USA > 0009-0007-6117-5022 SAP Labs Palo Alto CA USA > 0009-0008-1734-2935 SAP Labs Palo Alto CA USA by (2025) ## Abstract The rise of LLM-based agents has opened new frontiers in AI applications, yet evaluating these agents remains a complex and underdeveloped area. This survey provides an in-depth overview of the emerging field of LLM agent evaluation, introducing a two-dimensional taxonomy that organizes existing work along (1) evaluation objectives—what to evaluate, such as agent behavior, capabilities, reliability, and safety—and (2) evaluation process—how to evaluate, including interaction modes, datasets and benchmarks, metric computation methods, and tooling. In addition to taxonomy, we highlight enterprise-specific challenges, such as role-based access to data, the need for reliability guarantees, dynamic and long-horizon interactions, and compliance, which are often overlooked in current research. We also identify the future research directions, including holistic, more realistic, and scalable evaluation. This work aims to bring clarity to the fragmented landscape of agent evaluation and provide a framework for systematic assessment, enabling researchers and practitioners to evaluate LLM agents for real-world deployment. LLM Agents; Agent Evaluation; Evaluation Taxonomy; Agent Behavior, Benchmarks, Safety; Enterprise AI journalyear: 2025 copyright: cc conference: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 3–7, 2025; Toronto, ON, Canada booktitle: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’25), August 3–7, 2025, Toronto, ON, Canada doi: 10.1145/3711896.3736570 isbn: 979-8-4007-1454-2/2025/08 ccs: Computing methodologies Natural language processing ccs: Software and its engineering Software verification and validation ccs: Human-centered computing Human computer interaction (HCI) ## 1. Introduction Agents based on LLMs are autonomous or semi-autonomous systems that use LLMs to reason, plan, and act, and represent a rapidly growing frontier in artificial intelligence (Yao et al., 2023a; Nakajima, 2023). From customer service bots and coding copilots to digital assistants, LLM agents are redefining how we build intelligent systems. As these agents move from research prototypes to real-world applications (Fourney et al., 2024; Lu et al., 2024), the question of how to rigorously evaluate them becomes both pressing and complex. However, evaluating LLM agents is more complex than evaluating LLMs in isolation. Unlike LLMs, which are primarily assessed for text generation or question answering, LLM agents operate in dynamic, interactive environments. They reason and make plans, execute tools, leverage memory, and even collaborate with humans or other agents (Durante et al., 2024). This complex behavior and dependence on real-world effects make standard LLM evaluation approaches insufficient. To make an analogy, LLM evaluation is like examining the performance of an engine. In contrast, agent evaluation assesses a car’s performance comprehensively, as well as under various driving conditions. LLM agent evaluation also differs from traditional software evaluation. While software testing focuses on deterministic and static behavior, LLM agents are inherently probabilistic and behave dynamically; therefore, they require new approaches to assessing their performance. The evaluation of LLM agents is at the intersection of natural language processing (NLP), human-computer interaction (HCI), and software engineering, which demands additional perspectives. Despite increasing interest in this space, existing surveys focus narrowly on LLM evaluation or cover specific agent capabilities without a holistic perspective (Zhang et al., 2024a). In addition, enterprise applications bring additional requirements to agents, including secure access to data and systems, a high degree of reliability for audit and compliance purposes, and more complex interaction patterns, which are rarely addressed in the existing literature (Yehudai et al., 2025). This survey aims to serve as a helpful reference for practitioners and researchers in the field of agent evaluation. Our contributions in this survey are twofold. - We propose a taxonomy of LLM agent evaluation that organizes prior work by evaluation objectives (what to evaluate, such as behavior, capabilities, reliability, and safety) and evaluation process (how to evaluate, including interaction modes, datasets and benchmarks, metrics computation methods, evaluation tooling, and evaluation environments). - We highlight enterprise-specific challenges, including role-based access control, reliability guarantees, long-term interaction, and compliance requirements. The remainder of this paper is structured as follows. Section 2 describes the taxonomy used in this survey paper to analyze the agent evaluation landscape. Section 3 dives into the first dimension of the taxonomy, evaluation objectives, and focuses on the aspects of the agent to be evaluated. Section 4 describes the second dimension, the evaluation process, and focuses on the evaluation method. Section 5 discusses the challenges of assessing LLM agents in enterprise environments. Section 6 outlines open questions and future research directions to guide the next phase of work in evaluating LLM agents. ## 2. Taxonomy for LLM-based Agent Evaluation We propose a two-dimensional taxonomy to organize different aspects of the evaluation of LLM-based agents, structured along the axes of Evaluation Objectives (what to evaluate) and Evaluation Process (how to evaluate). This taxonomy is visualized as a hierarchical tree in 1. The Evaluation Objectives dimension is concerned with the targets of evaluation. The first category, Agent Behavior, in this dimension focuses on outcome-oriented aspects such as task completion and output quality, capturing how well an agent meets end-users’ expectations. Next, Agent Capabilities emphasize process-oriented competencies, including tool use, planning and reasoning, memory and context retention, and multi-agent collaboration. These capabilities provide insights into how agents achieve their goals and how well they meet their design specification. Reliability assesses whether an agent behaves consistently for the same input and robustly when input varies or the system encounters errors. Finally, Safety and Alignment evaluates the agent’s trustworthiness and security, including fairness, compliance, and the prevention of harmful or unethical behaviors. The Evaluation Process dimension describes how agents are assessed. Interaction Mode distinguishes between static evaluation, where agents respond to fixed inputs, and interactive assessment, where agents engage with users. Evaluation Data discusses both synthetic and real-world datasets, as well as benchmarks tailored to specific domains such as software engineering, healthcare, and finance (Fourney et al., 2024; Lu et al., 2024). Metrics Computation Methods encompasses quantitative measures, such as task success and factual accuracy, as well as qualitative evaluations based on human or LLM judgments. Evaluation Tooling refers to the supporting infrastructure, such as instrumentation frameworks (e.g., LangSmith, Arize AI) and public leaderboards (e.g., Holistic Evaluation of Agents), that enable scalable and reproducible assessment. Lastly, Evaluation Contexts define the environment in which evaluations are conducted, from controlled simulations to open-world settings such as web browsers or APIs. This taxonomy serves both as a conceptual framework and a practical guide, enabling systematic comparison and analysis of LLM agents across a wide range of goals, methodologies, and deployment conditions. In the following sections, we examine each dimension in detail, highlighting key evaluation practices and representative studies. As LLM agents are deployed in increasingly diverse and complex settings, factors such as single-turn versus multi-turn interactions, multilingualism, and multimodality all become more important. While the taxonomy remains applicable across these variations, tailored metrics and evaluation strategies are often required. We will discuss these specific adaptations in the relevant sections that follow. <details> <summary>taxonomy.png Details</summary>  ### Visual Description ## Diagram: Agent Evaluation Framework ### Overview The image displays a hierarchical flowchart outlining a comprehensive framework for evaluating AI agents. The diagram is structured as a tree, flowing from left to right, starting with a root concept that branches into two primary categories, which further subdivide into specific evaluation dimensions and their corresponding components or metrics. ### Components/Axes The diagram is organized into four distinct layers, differentiated by color and position: 1. **Root Node (Green, Far Left):** "Agent Evaluation" 2. **Primary Categories (Blue, Center-Left):** Two main branches: * "Evaluation Objectives" * "Evaluation Process" 3. **Evaluation Dimensions (Orange, Center):** Specific areas of focus under each primary category. 4. **Components/Metrics (Yellow, Far Right):** Concrete elements, methods, or metrics associated with each evaluation dimension. Arrows connect each node to its children, indicating a hierarchical and logical relationship. The flow is strictly left-to-right. ### Detailed Analysis **Branch 1: Evaluation Objectives** This branch defines *what* aspects of an agent should be evaluated. * **Dimension 1: Agent Behavior** * **Description:** "Outcome oriented. Did the agent produce the right result, efficiently and affordably?" * **Components:** "Task Completion, Interaction Quality, Latency & Cost" * **Dimension 2: Agent Capabilities** * **Description:** "Process oriented. Does the agent produce results in the right way, as designed?" * **Components:** "Planning & Reasoning, Memory & Context, Tool Use, Multi Agent" * **Dimension 3: Reliability** * **Description:** "Can the agent perform reliably across inputs and over time?" * **Components:** "Robustness, Hallucinations, Error Handling" * **Dimension 4: Safety and Alignment** * **Description:** "Can the agent be trusted not to produce harmful or non-compliant results?" * **Components:** "Fairness, Harm, Compliance & Privacy" **Branch 2: Evaluation Process** This branch defines *how* an agent evaluation is conducted. * **Dimension 1: Interaction Mode** * **Description:** "Methods of interacting with LLM agent systems." * **Components:** "Static & Offline, Dynamic & Online" * **Dimension 2: Evaluation Data** * **Description:** "Datasets, benchmarks, and synthetic data generation for evaluation." * **Components:** "Datasets, Benchmarks, Domain Specific" * **Dimension 3: Metrics Computation Methods** * **Description:** "Methods to compute performance metrics." * **Components:** "Code Based, Human-as-a-Judge, LLM-as-a-Judge" * **Dimension 4: Evaluation Tooling** * **Description:** "Frameworks and platforms to evaluate with." * **Components:** "Frameworks, Platforms, & Leaderboards" * **Dimension 5: Evaluation Contexts** * **Description:** "What environments to evaluate in." * **Components:** "Environments" ### Key Observations 1. **Clear Dichotomy:** The framework cleanly separates the *goals* of evaluation (Objectives) from the *methodology* (Process). 2. **Comprehensive Scope:** The "Objectives" branch covers a holistic view of agent performance, from raw outcomes (Behavior) to internal processes (Capabilities), stability (Reliability), and ethical constraints (Safety). 3. **Process Granularity:** The "Process" branch breaks down the practical steps of evaluation into five distinct, actionable categories, moving from setup (Interaction Mode, Data) to execution (Metrics, Tooling) and environment (Contexts). 4. **Visual Hierarchy:** The color coding (Green -> Blue -> Orange -> Yellow) and spatial arrangement effectively communicate the increasing level of detail from left to right. ### Interpretation This diagram presents a structured taxonomy for the emerging field of AI agent evaluation. It moves beyond simple accuracy metrics to advocate for a multi-faceted assessment. * **Relationship Between Elements:** The "Evaluation Objectives" define the *requirements* for a good agent. The "Evaluation Process" provides the *operational blueprint* to measure those requirements. For example, to evaluate the "Reliability" objective, one would need to select an appropriate "Interaction Mode" (e.g., Dynamic & Online), use relevant "Evaluation Data" (e.g., Domain Specific benchmarks), and apply a "Metrics Computation Method" (e.g., Code Based for robustness checks). * **Underlying Philosophy:** The framework implies that a trustworthy agent must be evaluated not just on *what* it does, but *how* it does it, *how consistently* it does it, and *within what ethical boundaries*. The inclusion of "LLM-as-a-Judge" and "Multi Agent" components acknowledges the specific challenges and paradigms of modern LLM-based systems. * **Practical Utility:** This serves as a checklist or reference model for researchers and engineers designing evaluation suites. It ensures that critical dimensions like safety, tool use, and evaluation context are not overlooked in favor of simpler task-completion metrics. The separation of "Tooling" and "Contexts" highlights that the choice of platform and the environment (e.g., simulated vs. real-world) are first-class concerns in rigorous evaluation. </details> Figure 1. Taxonomy of LLM Agent Evaluation ## 3. Evaluation Objectives ### 3.1. Agent Behavior Agent behavior refers to the overall performance of the agent as perceived by a user, treating the agent as a black box. It represents the highest-level view in evaluation and offers the most direct insight into the user experience. This category encompasses aspects such as task completion, output quality, latency, and cost. #### 3.1.1. Task Completion: Task completion is a fundamental objective of agent evaluation, assessing whether an agent successfully achieves the predefined goals of a given task (Reimann et al., 2023; Chen et al., 2025; Xiao et al., 2024; Zhang et al., 2025). It involves determining whether a desired state is reached or if specific criteria defined for task success are met (Liu et al., 2023b; Wang et al., 2024c). Although sometimes noted for providing limited fine-grained insight into failures, especially when most models achieve low success rates (Ma et al., 2024), task completion remains a predominant and essential measure of overall agent performance (Ma et al., 2024). Task completion is commonly quantified using metrics such as Success Rate (SR) (Liu et al., 2023b; Ma et al., 2024; Chen et al., 2025), which can also be referred to as Task Success Rate (Zhang et al., 2025) or Overall Success Rate (Liu et al., 2023b). Other related metrics include Task Goal Completion (TGC) (Trivedi et al., 2024) and Pass Rate (Qin et al., 2023). Some evaluations employ binary indicators, such as a reward function that returns 0 or 1 for goal achievement (Koh et al., 2024). Metrics such as pass@ $k$ and pass^ $k$ extend this by considering success over multiple trials (Yao et al., 2024). This crucial objective is applied across a wide range of LLM agent evaluation domains and benchmarks (Wang et al., 2024c). This includes tasks related to coding and software engineering, such as resolving GitHub issues (SWE-bench (Jimenez et al., 2024)), scientific data analysis programming (ScienceAgentBench (Chen et al., 2025)), reproducing research (CORE-Bench (Siegel et al., 2024), PaperBench (Starace et al., 2025)), and interactive coding in apps (AppWorld (Trivedi et al., 2024)). It is also extensively used for agents interacting with web environments, including general web navigation (BrowserGym (Chezelles et al., 2025), WebArena (Zhou et al., 2023), WebCanvas (Pan et al., 2024)), multimodal web tasks (VisualWebArena (Koh et al., 2024), MMInA (Zhang et al., 2024g)), and realistic time-consuming web tasks (ASSISTANTBENCH (Yoran et al., 2024)). #### 3.1.2. Output Quality: Output quality refers to the characteristics of responses by an LLM agent. It is an umbrella term encompassing aspects such as accuracy, relevance, clarity, coherence, and adherence to agent specifications or task requirements (Reimann et al., 2023). An agent may complete a task yet still deliver a subpar user experience if the interaction lacks the qualities mentioned above. Output quality is particularly relevant in evaluating conversational agents, where user goals are often achieved over multiple turns. Many metrics in this category overlap with those used in large language model (LLM) evaluation. For example, the fluency metric is used to measure the degree to which the output of an LLM satisfies the conventions of natural language (Zhang et al., 2023c). The logical coherence metric focuses on the rigor in arguments (Zhang et al., 2023c). As LLM agents may utilize tools for retrieving grounding information and providing context-aware text answers, standard metrics used in retrieval-augmented generation (RAG) systems also apply. Such metrics include Response Relevance or Factual Correctness (Es et al., 2025). #### 3.1.3. Latency & Cost: Latency is a critical aspect of agent behavior, especially in scenarios where users interact with agents synchronously. Long wait times can significantly degrade user experience and erode trust in the system. A commonly used metric in this context is Time To First Token (TTFT), which measures the delay before a user sees the first token of an LLM’s response in streaming mode. For use cases where the agent operates asynchronously, End-to-End Request Latency —the time to receive the complete response—is often more relevant (Nvidia, 2024). While cost is not directly observable by end-users, it plays a crucial role in determining the practicality of deploying agents at scale. We include cost as a measure of an agent’s monetary efficiency. It is typically estimated based on the number of input and output tokens, which directly correlate with usage-based pricing in most LLM deployments. ### 3.2. Agent Capabilities Beyond external behavior, evaluations often target specific capabilities of LLM-based agents that enable their performance. Key aspects of this category include tool use, planning and reasoning, memory and context retention, and multi-agent collaboration. Evaluating these capabilities helps determine an agent’s strengths and weaknesses on a more granular level. #### 3.2.1. Tool Use: Tool use is a core capability for LLM-based agents, enabling them to retrieve grounding information, perform actions, and interact with external environments. In this survey, tool use involves invocation of a single tool and is interchangeable with function calling; more complex cases of determining tool sequences for complex tasks will be discussed in 3.2.2. Recent advances have allowed LLMs, such as ChatGPT-3.5 and beyond, to support function calling natively. These models can autonomously decide whether to invoke a function, select the appropriate one from a candidate set, and generate the required parameters. As a result, LLM agents can directly build on the functions of the underlying model, allowing many of the evaluation techniques originally developed for LLMs to use tools (Li et al., 2023). The evaluation of the tool’s use involves answering several key questions. First, can the agent correctly determine whether tool invocation is necessary for a given task? If so, can it select the appropriate tool from a defined set of candidates? Once the tool is selected, the agent must be able to identify the correct parameters required by the tool and then generate appropriate values for each parameter to ensure proper execution. In cases where the candidate toolset is extensive, the agent may also need to retrieve the correct tool from a repository based on a natural language description of the task (Shen et al., 2024). Several metrics have been proposed to assess these abilities. Invocation Accuracy (Liu et al., 2025) evaluates whether the agent makes the correct decision about whether to call a tool at all. Tool Selection Accuracy measures whether the proper tool is chosen from a list of options. Retrieval Accuracy focuses on whether the system can retrieve the correct tool from a larger toolset, often measured using rank accuracy $k$ . For ranking-based evaluation, Mean Reciprocal Rank (MRR) quantifies the position of the correct tool in the ranked list. In contrast, Normalized Discounted Cumulative Gain (NDCG) reflects how well the system ranks all relevant tools (Liu et al., 2025). Parameter-related evaluation involves two aspects. The parameter name F1 score (Shen et al., 2024) measures the agent’s ability to identify the parameter names required for a given function correctly and then correctly assign values to them. While some evaluations rely on the correctness of abstract syntax trees (ASTs) to check if the tool call is syntactically valid, this approach may miss semantic errors, such as incorrect or hallucinated parameter values, especially for parameters constrained to enumerated types (Patil et al., 2023). To address this limitation, recent work, such as the Gorilla paper, has proposed execution-based evaluation, in which the system runs the tool calls and assesses their outcomes, offering a more comprehensive and grounded assessment of tool use capability (Patil et al., 2023). #### 3.2.2. Planning and Reasoning: Planning and reasoning are essential capabilities for LLM-based agents, especially in complex tasks that require multiple steps or making decisions under uncertainty. Planning involves selecting the correct set of tools in an appropriate order. At the same time, reasoning enables agents to make context-aware decisions, either ahead of time or dynamically during task execution (Huang et al., 2024a). T-eval (Chen et al., 2024b) formulated planning evaluation as comparing the set of predicted tools against a reference. Since tool order and dependency also matter, some benchmarks adopt graph-based representations and introduce metrics such as Node F1 for tool selection and Edge F1 or Normalized Edit Distance for assessing tool invocation sequences and structural accuracy (Shen et al., 2024). In dynamic environments, agents often need to interleave planning and execution, adapting their actions in response to evolving context (Huang et al., 2024a). This pattern is illustrated by the ReAct paradigm, where agents alternate between reasoning steps and tool usage (Yao et al., 2023b). Evaluating such adaptive reasoning requires more than comparing static plans—it demands metrics that reflect decision-making in real time. The T-Eval framework (Chen et al., 2024b) addresses this by introducing a reasoning metric that assesses how closely an agent’s predicted next tool call aligns with the expected one at each step. This captures the agent’s ability to make informed decisions when tool outputs are not known in advance. Similarly, AgentBoard (Ma et al., 2024) proposes the metric Progress Rate, which compares the agent’s actual trajectory against the expected one, offering a fine-grained measure of how effectively the agent advances toward its goal. When agents are instructed to plan in the form of generating complete multi-step programs, evaluation methods from code generation become relevant. Benchmarks like ScienceAgentBench compare the generated plans against annotated references using program similarity metrics (Chen et al., 2025). Additionally, the Step Success Rate has been proposed to measure the percentage of steps in the generated plan that are successfully executed, providing a holistic view of planning quality during execution (Gioacchini et al., 2024). #### 3.2.3. Memory and Context Retention: A critical capability for long-running agents is the ability to retain information throughout many interactions and apply previous context to current requests. Guan et al. (Guan et al., 2024) categorize memory evaluation in multi-turn conversations by Memory Span (how long information is stored) and Memory Forms (how information is represented). For example, LongEval (Krishna et al., 2023) and SocialBench (Chen et al., 2024a) are benchmarks that test an agent’s context retention in long dialogues (40+ turns). An agent might be given a conversation that spans dozens of exchanges and later asked questions that require recalling details from early in the conversation. Maharana et al. (Maharana et al., 2024) demonstrate evaluation with dialogues spanning hundreds of turns (600+ turns), and Li et al. (Li et al., 2024d) introduce memory-enhanced evaluation techniques, tracking how well agents maintain consistency in long-horizon tasks. These evaluations often use synthetic or logged conversations as datasets, and metrics include Factual Recall Accuracy or Consistency Score (no contradictions between turns). Memory evaluation may also consider working memory for tool-using agents (i.e., whether the agent keeps track of intermediate results) and forgetting strategies (i.e., whether it appropriately forgets irrelevant details to avoid confusion). #### 3.2.4. Multi-Agent Collaboration: Evaluating multi-agent collaboration in LLM-based systems requires different methodologies compared to traditional reinforcement learning-driven coordination (Chan et al., 2023; Li et al., 2024b; Talebirad and Nadiri, 2023). Unlike conventional agents that rely on predefined reward structures, LLM agents coordinate through natural language, strategic reasoning, and decentralized problem-solving (Han et al., 2024; Guo et al., 2024). These capabilities are crucial in real-world applications such as financial decision-making and structured data analysis, where autonomous agents must exchange information, negotiate, and synchronize decision-making processes efficiently (Liu et al., 2024b; Li et al., 2024d). Autonomous Agents for Collaborative Tasks (Liu et al., 2024b) evaluates Collaborative Efficiency, assessing how well multiple agents share responsibilities and distribute tasks dynamically. ### 3.3. Reliability Reliability is a crucial objective, especially as LLM agents are considered for use in enterprise and safety-critical applications. It encompasses consistency, robustness to variations, and trustworthiness of the agents’ outputs. Unlike task performance (which might measure best-case capabilities), reliability evaluation probes worst-case and average-case scenarios. #### 3.3.1. Consistency: Consistency refers to the stability of the output when the same task is repeated multiple times (Liu et al., 2025). Since LLMs are inherently non-deterministic, LLM-based agents also exhibit variability in their behavior. For agents to be trusted in enterprise or other high-stakes applications, they must demonstrate consistent performance across repeated runs of the same task. A commonly used metric in this context is pass@k, which measures the probability that an agent succeeds at least once over k attempts. However, a stricter measure of consistency is whether the agent succeeds in all k attempts. This is formalized in the $\tau$ -benchmark as the pass^ $k$ metric, which better captures the consistency requirements of mission-critical deployments. #### 3.3.2. Robustness: Robustness refers to the stability of an agent’s output when faced with input variations or changes in the environment. To remain effective and trustworthy, LLM-based agents must consistently deliver high performance under a range of challenging conditions. Evaluating robustness often involves stress-testing the agent with perturbed inputs—such as paraphrased instructions, irrelevant or misleading context, or linguistic variations like typos and dialects—to assess whether it can still complete the task successfully. For example, robustness evaluations may involve applying systematic transformations to standard prompts and measuring the resulting drop in task success rate or output quality. The HELM benchmark (Liang et al., 2023) explicitly incorporates such tests, tracking how model performance degrades under input variation. Robustness also encompasses adaptive resilience—the agent’s ability to recover from dynamic changes in the environment. For instance, WebLinX (Lù et al., 2024) examines how agents behave when the structure of a web page changes during execution. In such settings, an effective agent must adjust its strategy rather than stall or fail. In tool-using agents, robustness is further reflected in error-handling capabilities. As demonstrated in ToolEmu’s evaluation (Ruan et al., 2023), agents must be able to respond to tool failures or unexpected outputs gracefully. Robustness tests may include intentionally injecting failures—such as API errors or null responses—to observe whether the agent recovers (e.g., retries, switches tools, or explains the issue to the user) or breaks down. A key metric could be the proportion of induced failures that are handled appropriately, reflecting the agent’s reliability in uncertain or imperfect conditions. ### 3.4. Safety and Alignment Safety covers an agent’s adherence to ethical guidelines, avoidance of harmful behavior, and compliance with legal or policy constraints. As LLM agents become more powerful and autonomous, the risk of unintended adverse outcomes (e.g., generating disinformation, hate speech, or unsafe instructions) grows, making safety evaluation indispensable. These evaluations are especially critical in fields such as financial services, cybersecurity, and autonomous decision-making, where agent vulnerabilities can lead to severe consequences (Li et al., 2024c; Fang et al., 2024; He et al., 2024b; Gan et al., 2024). #### 3.4.1. Fairness: The lack of fairness and transparency in AI agents can result in biased outcomes, decreased users’ trust, and unintended societal consequences (de Cerqueira et al., 2024). In financial applications, for example, biased decision-making in loan approvals or investment strategies can reinforce systemic inequalities (FinCon (Yu et al., 2024b), AutoGuide (Fu et al., 2024)). Ethical concerns also arise in multi-agent interactions, where decision-making frameworks must ensure compliance with standards and social norms (Mehrotra et al., 2024). Explainability is crucial in enhancing user trust, especially in interactive systems where AI agents provide recommendations or automated assistance. Methods such as guideline-driven decision-making (AutoGuide (Fu et al., 2024)) and structured transparency mechanisms (MATSA (Mathur et al., 2024), FinCon (Yu et al., 2024b)) provide users with clear reasoning paths. Meanwhile, Rjudge (Yuan et al., 2024) analyzes how agents perceive risk when making autonomous decisions, emphasizing transparency and trustworthiness in AI interactions. Evaluating these dimensions ensures that AI agents align with ethical standards while maintaining fairness in their operational contexts. #### 3.4.2. Harm, Toxicity, and Bias: One aspect of safety is ensuring that an agent’s outputs do not contain harmful content such as hate speech, harassment, or extremely biased statements. Evaluation for toxicity often uses specialized test sets and metrics, such as the RealToxicityPrompts dataset (Gehman et al., 2020) – a collection of prompts likely to elicit toxic content—where its responses are checked with automated toxicity detectors or human raters. Metrics include the percentage of responses containing toxic language or the average toxicity score (as given by a classifier). HELM (Liang et al., 2023) includes toxicity and bias metrics as part of holistic evaluation, indicating how frequently a model produces offensive content or exhibits undesired biases. For an interactive agent, one might evaluate it by giving provocative or ethically challenging inputs (red-teaming) and then measuring its failure rate (how often it responds in an unsafe manner). Safety-focused datasets, such as CoSafe, target exactly this: Yu et al. introduce CoSafe (Yu et al., 2024a) to evaluate conversation agents on adversarial prompts designed to trick them into breaking safety rules (e.g., a user subtly asks for self-harm advice or illicit instructions). CoSafe revealed that even advanced agents had vulnerabilities, such as falling for coreference-based attacks (where a user refers to something ambiguously to bypass filters). The evaluation process involved monitoring the agent’s responses for policy violations when faced with these adversarial queries. Having a numeric score (like “agent produced a disallowed response in X% of adversarial cases”) quantifies safety. #### 3.4.3. Compliance and Privacy: Beyond avoiding overt toxicity, many deployments require agents to comply with specific regulatory or policy constraints (Zhang et al., 2023b; Brown, 2024). For instance, a finance chatbot must not disclose confidential information or provide particular types of financial advice, and a medical assistant must not deviate from established medical guidelines. Evaluating compliance may be highly domain-specific, as it involves scenarios crafted to test whether the agent respects boundaries (e.g., a user asks the medical bot for a prescription-only drug recommendation—the correct, safe behavior is to refuse and advise consulting a doctor). In enterprise contexts, compliance evaluation may require proprietary test cases that reflect actual policies. One approach is to integrate those concerns into evaluation frameworks. For example, the HELM benchmark (Liang et al., 2023) for enterprises was proposed to include domain-specific prompts and metrics (such as accuracy on financial jargon or compliance in responses) for fields like finance and law. The process involves gathering representative enterprise scenarios (which may contain confidential or custom data) and designing evaluation metrics that reflect real-world success criteria (e.g., Did the agent follow all legal disclaimer requirements in its response?). For example, TheAgentCompany (Xu et al., 2024) evaluates enterprise AI agents under structured correctness constraints, requiring them to follow predefined organizational policies when completing tasks. Table 1. Evaluation Objectives and Their Metrics. | Objectives | Category | Metrics | Relevant Papers | | --- | --- | --- | --- | | Agent Behavior | Task Completion | Success Rate (SR), F1-score, Pass@k, Progress Rate, Execution Accuracy, Transfer Learning Success, Zero-Shot Generalization Accuracy | AgentBoard (Chang et al., 2024), WebShop (Yao et al., 2022), AgentBench (Liu et al., 2023c), Tool Use Evaluation (Liu et al., 2024a), InformativeBench, SQuAD (Rajpurkar et al., 2016) (Rajpurkar et al., 2018) (Liu et al., 2024c), ResearchArena (Kang and Xiong, 2024), AgentBoard (Chang et al., 2024), AppWorld (Trivedi et al., 2024), TheAgentCompany (Xu et al., 2024), MAGIC (Xu et al., 2023), Mobile-Env (Zhang et al., 2024f), Re-ReST (Dou et al., 2024), XMC-AGENT (Liu et al., 2024d), SWE-bench (Jimenez et al., 2024) | | Output Quality | Coherence, User Satisfaction, Usability, Likability, Overall Quality | PredictingIQ (Reimann et al., 2023), EnDex (Xu et al., 2022), PsychoGAT (Yang et al., 2024b) | | | Latency & Cost | Latency, Token Usage, Cost | Cluster diagnosis (Shi et al., 2024), MobileBench (Deng et al., 2024), MobileAgentBench (Wang et al., 2024a), LangSuitE (Jia et al., 2024), WebArena (Zhou et al., 2023), Mobile-env (Zhang et al., 2023a), GUI Agents (Wang et al., 2024b), GPTDroid (Liu et al., 2023a), Spa-bench (Chen et al., 2024d) | | | Agent Capability | Tool Use | Task Completion Rate, Tool Selection Accuracy | ToolEmu (Ruan et al., 2024), MetaTool (Huang et al., 2024b), AutoCodeRover (Zhang et al., 2024e) | | Planning & Reasoning | Reasoning Quality, Accuracy, Fine-Grained Progress Rate, Self Consistency, Plan Quality | AgentBoard (Chang et al., 2024) Massive Multitask Language Understanding (Hendrycks et al., 2021), LLM-Augmented Autonomous Agents (Liu et al., 2024b), Cluster diagnosis questions (Shi et al., 2024), SimuCourt (He et al., 2024a), Magis (Tao et al., 2024) | | | Memory & Context Retention | Factual Accuracy Recall, Consistency Scores | LongEval (Krishna et al., 2023), SocialBench (Chen et al., 2024a), LoCoMo (Maharana et al., 2024), Optimus-1 (Li et al., 2024d) | | | Multi-Agent Collaboration | Information Sharing Effectiveness, Adaptive Role Switching, Reasoning Rating | AgentSims (Lin et al., 2023), WebArena (Zhou et al., 2023), MATSA (Mathur et al., 2024), GAMEBENCH (Costarelli et al., 2024), BALROG (Paglieri et al., 2024), TheAgentCompany (Xu et al., 2024) | | | Reliability | Consistency | pass^k | $\tau$ -Bench (Yao et al., 2024) | | Robustness | Accuracy, Task Success Rate Under Perturbation | HELM (Liang et al., 2023), WebLinX (Lù et al., 2024) | | | Safety | Fairness | Awareness Coverage, Violation Rate, Transparency, Ethics, Morality | CASA (Qiu et al., 2025), R-Judge (Yuan et al., 2024), SimuCourt (He et al., 2024a), MATSA (Mathur et al., 2024), FinCon (Yu et al., 2024b), AutoGuide (Fu et al., 2024) | | Harm | Adversarial Robustness, Prompt Injection Resistance, Harmfulness, Bias Detection | Agent Security Bench(ASB) (Zhang et al., 2024c), AgentPoison (Chen et al., 2024c), AgentDojo (Debenedetti et al., 2024), Backdoor Attacks (Yang et al., 2024a), SafeAgentBench (Yin et al., 2024), Agent-Safety Bench (Zhang et al., 2024b), AgentHarm (Andriushchenko et al., 2025), Adaptive Attacks (Zhan et al., 2025), RealToxicityPrompts (Gehman et al., 2020) | | | Compliance & Privacy | Risk Awareness, Task Completion Under Constraints | R-Judge (Yuan et al., 2024), Cybench (Zhang et al., 2024d), TheAgentCompany (Xu et al., 2024) | | ## 4. Evaluation Process ### 4.1. Interaction Mode Evaluating LLM agents can occur in various interaction modes and with different tooling. A fundamental distinction is between offline evaluation (using pre-generated, static datasets) and online assessment (involving reactive simulations, humans in the loop, or live system monitoring). #### 4.1.1. Static & Offline Evaluation Often performed as a baseline, offline evaluations typically rely on datasets and static test cases: collections of tasks, prompts, or conversations that represent challenges the agent might face. Simulated conversations may be used to help develop these data, but are ultimately inert between different runs. Though comparatively cheaper and simpler to run and maintain, offline evaluations typically lack the nuance to fully address the wide range of responses an LLM agent may be able or expected to provide. As such, they are more prone to error propagation and are generally less accurate representations of system performance. #### 4.1.2. Dynamic & Online Evaluation As with other ML systems, online evaluation often occurs after an LLM agent has been deployed. Instead of relying on synthetic, historical, or manually crafted data, online evaluations leverage simulations or fundamental user interactions. This adaptive data is crucial for identifying pain points and issues that are not discovered during static testing and is often rich in domain context that is more difficult to capture with synthetic or generalized benchmarks. Dynamic evaluations may use proxies to simulate users or environments in reactive, real-time response to agent behavior. For example, in the assessment of web agents, researchers built web simulators (MiniWoB (Shi et al., 2017), WebShop (Yao et al., 2022), WebArena (Zhou et al., 2023), etc.) where the behavior of an agent (clicking links, filling forms) can be programmed to verify the correct sequence. The concept of Evaluation-driven Development (EDD) has also been proposed (Xia et al., 2024), proposing making evaluation an integral part of the agent development cycle. It advocates for continuous evaluation of the agent, both offline (during development) and online (after deployment), to detect regressions and adapt to new use cases. They further outline a reference architecture in which an AgentOps component monitors agent performance in production and provides insights back to developers. While still an emerging idea, it underscores that evaluation is not a one-time task but an ongoing process, especially for agents that learn or evolve. ### 4.2. Evaluation Data The growing interest in evaluating LLM-based agents has led to the development of diverse datasets, benchmarks, and leaderboards specifically targeting the agent capabilities discussed in Section 3. Many of these resources are designed to reflect real-world complexity and are built using a mix of human-annotated, synthetic, and interaction-generated data. For instance, datasets such as AAAR-1.0 (Lou et al., 2025), ScienceAgentBench (Chen et al., 2025), and TaskBench (Shen et al., 2024) provide structured, expert-labeled benchmarks for assessing research reasoning, scientific workflows, and multi-tool planning. Others, such as FlowBench (Xiao et al., 2024), ToolBench (Huang et al., 2024b), and API-Bank (Li et al., 2023), focus on tool use and function-calling across large API repositories. These benchmarks typically include not only the gold tool sequences but also expected parameter structures, enabling fine-grained evaluation. In parallel, datasets like AssistantBench (Yoran et al., 2024), AppWorld (Trivedi et al., 2024), and WebArena (Zhou et al., 2023) simulate more open-ended and interactive agent behaviors in web and application environments. They emphasize dynamic decision-making, long-horizon planning, and user-agent interactions. Several benchmarks also support safety and robustness testing—for example, AgentHarm (Andriushchenko et al., 2025) assesses potentially harmful behaviors, while AgentDojo (Debenedetti et al., 2024) evaluates resilience against prompt injection attacks. Leaderboards such as the Berkeley Function-Calling Leaderboard (BFCL) (Yan et al., 2024) and Holistic Agent Leaderboard (Stroebl et al., 2025) consolidate these evaluations by providing standardized test cases, automated metrics (e.g., AST correctness, Win Rate), and ranking mechanisms to compare systems. ### 4.3. Metrics Computation Methods The code-based method is the most deterministic and objective approach (Chang et al., 2024; Liu et al., 2023b, 2024a). (Chang et al., 2024) It relies on explicit rules, test cases, or assertions to verify whether an agent’s response meets predefined criteria. This method is particularly effective for tasks with well-defined outputs, such as numerical calculations, structured query generation, or syntactic correctness in programming tasks. Its primary advantage is its consistency and reproducibility, making it highly reliable for benchmarking. However, code-based methods are typically inflexible. They struggle with evaluating open-ended or qualitative responses, such as natural language generation or creative problem-solving, where correctness is subjective. Despite this, it remains a fundamental technique for evaluating structured tasks where correctness is well-defined. The LLM-as-a-Judge approach (Zheng et al., 2023) leverages the reasoning capabilities of LLMs to evaluate agent responses based on qualitative criteria, assessing responses according to the criteria provided through instructions. This method has gained traction due to its ability to handle tasks that are subjective and nuanced, such as summarization, reasoning, and conversational interactions. One recent extension of this method is Agent-as-a-Judge (Zhuge et al., 2024), where the evaluation process involves multiple AI agents interacting to refine the assessment, potentially improving evaluation reliability. This method is highly scalable and can adapt to complex tasks. As a result, it has received increasing attention (Li et al., 2024a; Gu et al., 2024). Human-in-the-loop evaluation remains the gold standard for subjective aspects (like naturalness and user satisfaction) and safety-critical judgment calls. Human evaluations can take the form of user studies, expert audits (where domain experts review agent outputs), or Crowdworker annotations (where outputs are rated along dimensions such as relevance, correctness, and tone). This method offers the highest reliability in open-ended tasks, such as content generation, strategic decision-making, or dialogue coherence. However, it is expensive, time-consuming, and challenging to scale, making it impractical for large-scale automated systems that require frequent evaluations. ### 4.4. Evaluation Tooling A notable aspect in the process dimension is the emergence of software frameworks and platforms that support automated, scalable, and continuous agent evaluation workflows. These tools enable integration of evaluation directly into the development lifecycle, reflecting a growing movement toward Evaluation-driven Development (EDD) (Xia et al., 2024) in agent building. OpenAI Evals is an open-source framework that allows developers to specify evaluation tasks and metrics for models, automating the execution and reporting of results (though not formally described in academic literature, it reflects practical needs) (OpenAI, 2023). Other open-source or commercial tools such as DeepEval (AI, 2025b), InspectAI (AI Security Institute, 2024), Phoenix (AI, 2025a), and GALILEO (Galileo, 2025) provide rich analytics, evaluation orchestration, and debugging capabilities. Moreover, agent development platforms like Azure AI Foundry (Microsoft, 2024), Google Vortex AI (Google, 2024), LangGraph (LangChain, 2024), and Amazon Bedrock (Amazon, 2024) increasingly incorporate evaluation features, helping developers monitor performance, detect regressions, and adapt agents to evolving user needs. Xia et al. (Xia et al., 2024) further propose an AgentOps architecture to continuously monitor deployed agents, closing the loop between development and deployment through real-time feedback and quality control. ### 4.5. Evaluation Contexts The evaluation context pertains to the environment in which an evaluation is performed. Similar to software engineering, a tradeoff exists between more realistic (but often more costly and potentially less secure) and simpler, more controlled (but usually less representative of final performance) environments. The context in which a system is assessed is typically guided by the system’s intended use; a simpler LLM agent without edit access might be tested in its working environment directly, whereas an LLM agent designed to work with and make changes to many intertwined systems will likely be evaluated in a mocked API or sandbox environment. For less contained systems, the evaluation context may take the form of a web simulator, such as MiniWoB (Shi et al., 2017), WebShop (Yao et al., 2022), or WebArena (Zhou et al., 2023). As the development of an agent continues, the evaluation context often evolves with it, from smaller, mocked API environments to live deployment as agent performance and trustworthiness are determined. ## 5. Enterprise-Specific Challenges As LLM-based agents transition from research demos to deployment in enterprise settings, new challenges are emerging. Enterprises often demand high performance in conjunction with predictable reliability, compliance with regulations, data security, and maintainability, which are usually overlooked during evaluation. To address these gaps, we discuss the concerns outlined in the following sections and outline future directions. ### 5.1. Complexity from Role-based Access A key challenge in evaluating LLM-based agents in enterprise settings is the need to account for Role-Based Access Control (RBAC), which governs users’ permissions to access data and services. In these environments, users operate with varying levels of access depending on their roles, and agents acting on their behalf must adhere to the same constraints. This introduces complexity into agent evaluation, as an agent’s ability to retrieve or act on information is not uniform but contextually bound to the user’s permissions. To address this, some evaluation frameworks have begun incorporating access control constraints into their design. For example, IntellAgent (Levi and Kadar, 2025) includes evaluation tasks that require authentication of user identity and enforce policies that deny access to other users’ information. By embedding role-specific restrictions into task generation, these approaches more accurately model how agents behave in permission-sensitive enterprise contexts. ### 5.2. Reliability Guarantees Reliability guarantees are especially important in enterprise settings, where agents are expected to operate within compliance and auditing frameworks that require deterministic or repeatable behavior that is explainable. In such contexts, occasional success is insufficient; agents must perform reliably across time and usage scenarios to be considered production-ready. Evaluating reliability is nontrivial. Because LLM-based agents are inherently stochastic, measuring consistency requires executing the same task multiple times and observing the variation in outcomes. This introduces significant evaluation overhead: running multiple trials per input can be computationally expensive, especially when testing complex tasks involving tools, memory, or multi-agent coordination. Moreover, to draw meaningful conclusions, benchmarks must include a representative dataset that reflects the types of tasks and conditions the agent may encounter. Some efforts have begun to tackle this challenge. For example, the $\tau$ -benchmark (Yao et al., 2024) explicitly incorporates the pass^ $k$ metric to evaluate the consistency of an agent. By applying this to domains such as retail and airline booking, the authors show that current agents struggle with consistency. ### 5.3. Dynamic and Long-Horizon Interactions One major challenge in evaluating LLM-based agents is assessing their performance on long-horizon tasks in dynamic, evolving environments. Unlike most current benchmarks that focus on short episodes or single interactions, real-world enterprise agents often operate continuously over extended periods while interacting with users, systems, and data. Addressing this challenge is essential for understanding how agents behave over time, particularly in enterprise settings where reliability, adaptability, and goal alignment are crucial throughout the agent’s lifecycle. Standard, short-term evaluations cannot capture phenomena such as performance drift, context retention, or the cumulative effect of decisions on business outcomes. To begin tackling this issue, some research efforts have introduced long-running simulations and extended dialogues as evaluation tools. For instance, Park et al. (Park et al., 2023) observed generative agents in a continuously running simulated town environment to study emergent behaviors across multi-day interactions. Similarly, Maharana et al. (Maharana et al., 2024) evaluated long-term conversational memory through 600-turn dialogues, focusing on how well agents maintain coherence and context over extended conversations. ### 5.4. Adherence to Domain-Specific Policies and Compliance Requirements Another significant challenge in evaluating enterprise agents is ensuring that they can operate within domain-specific policies and compliance constraints. Enterprises often enforce strict operational rules—such as approval workflows, data retention policies, usage quotas, and legal regulations like GDPR or HIPAA—that must be respected by agents throughout task execution. Evaluating agents in such contexts requires more than measuring task success; it demands verification that agent behaviors align with formal policy constraints and legal compliance standards. For instance, an agent generating financial reports must avoid unauthorized access to confidential forecasts and ensure that generated content adheres to regulatory reporting standards. Without explicit modeling of these constraints during evaluation, agents deemed ”correct” in traditional benchmarks may still fail in production due to policy violations or compliance risks. ## 6. Future Research Directions As LLM-based agents continue to grow in complexity and application scope, future research must push toward more robust, practical, and scalable evaluation methodologies. We highlight four key directions that can significantly advance the field: Holistic Evaluation Frameworks: Current evaluation efforts often focus on isolated dimensions such as task success, planning quality, or tool use. However, agents in real-world applications must simultaneously balance multiple competencies. Future work should develop holistic evaluation frameworks that assess agent performance across multiple, interdependent dimensions. More Realistic Evaluation Settings: To bridge the gap between lab settings and production environments, agent evaluation must move toward more realistic conditions. This includes creating evaluation environments that incorporate enterprise-specific elements such as dynamic multi-user interactions, role-based access controls, and domain-specific knowledge. These settings can be achieved through real-world deployment trials or through simulated environments that mimic enterprise workflows. Automated and Scalable Evaluation Techniques: Manual evaluation of agent behavior, especially in multi-turn or long-horizon scenarios, is costly and complicated to scale. Future research should explore automated evaluation approaches to reduce human effort and improve reproducibility. This includes using synthetic data generation for controllable test cases, leveraging simulated environments to emulate task contexts, and advancing LLM-based evaluation techniques such as LLM-as-a-judge or Agent-as-a-judge. Time- and Cost-Bounded Evaluation Protocols: Evaluation must be efficient and able to support iterative agent development. Today’s methods—especially those requiring repeated trials or human-in-the-loop assessments—can be both time- and resource-intensive. Future research should aim to develop time- and cost-bounded evaluation protocols that strike a balance between depth and efficiency. In summary, future research should focus on developing evaluation methods that are holistic, realistic, scalable, and efficient. These directions are essential for building reliable and trustworthy LLM-based agents at scale. ## References - (1) - AI (2025a) Arize AI. 2025a. Phoenix. https://github.com/Arize-ai/phoenix - AI (2025b) Confident AI. 2025b. DeepEval. https://github.com/confident-ai/deepeval - AI Security Institute (2024) UK AI Security Institute. 2024. Inspect AI: Framework for Large Language Model Evaluations. https://github.com/UKGovernmentBEIS/inspect_ai - Amazon (2024) Amazon. 2024. Amazon Bedrock Agents. https://aws.amazon.com/bedrock/agents/ - Andriushchenko et al. (2025) Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, Eric Winsor, Jerome Wynne, Yarin Gal, and Xander Davies. 2025. AgentHarm: a benchmark for measuring harmfulness of LLM agents. doi: 10.48550/arXiv.2410.09024 - Brown (2024) Nik Bear Brown. 2024. Enhancing trust in llms: Algorithms for comparing and interpreting llms. arXiv preprint arXiv:2406.01943 (2024). - Chan et al. (2023) Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201 (2023). - Chang et al. (2024) Ma Chang, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. 2024. AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents. Advances in Neural Information Processing Systems 37 (2024), 74325–74362. - Chen et al. (2024a) Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Xing Gao, Weizhou Shen, Xiaojun Quan, Chenliang Li, Ji Zhang, Fei Huang, and Jingren Zhou. 2024a. SocialBench: Sociality Evaluation of Role-Playing Conversational Agents. arXiv:2403.13679 [cs.CL] https://arxiv.org/abs/2403.13679 - Chen et al. (2024d) Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, et al. 2024d. Spa-bench: A comprehensive benchmark for smartphone agent evaluation. In NeurIPS 2024 Workshop on Open-World Agents. - Chen et al. (2025) Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. 2025. ScienceAgentBench: toward rigorous assessment of language agents for data-driven scientific discovery. doi: 10.48550/arXiv.2410.05080 - Chen et al. (2024b) Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, and Feng Zhao. 2024b. T-eval: evaluating the tool utilization capability of large language models step by step. doi: 10.48550/arXiv.2312.14033 - Chen et al. (2024c) Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. 2024c. AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases. arXiv:2407.12784 [cs.LG] https://arxiv.org/abs/2407.12784 - Chezelles et al. (2025) Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan Salakhutdinov, Nicolas Chapados, and Alexandre Lacoste. 2025. The BrowserGym ecosystem for web agent research. doi: 10.48550/arXiv.2412.05467 - Costarelli et al. (2024) Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer, and Arjun Yadav. 2024. Gamebench: Evaluating strategic reasoning abilities of llm agents. arXiv preprint arXiv:2406.06613 (2024). - de Cerqueira et al. (2024) José Antonio Siqueira de Cerqueira, Mamia Agbese, Rebekah Rousi, Nannan Xi, Juho Hamari, and Pekka Abrahamsson. 2024. Can we trust AI agents? An experimental study towards trustworthy LLM-based multi-agent systems for AI ethics. arXiv preprint arXiv:2411.08881 (2024). - Debenedetti et al. (2024) Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. doi: 10.48550/arXiv.2406.13352 - Deng et al. (2024) Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, et al. 2024. Mobile-bench: An evaluation benchmark for llm-based mobile agents. arXiv preprint arXiv:2407.00993 (2024). - Dou et al. (2024) Zi-Yi Dou, Cheng-Fu Yang, Xueqing Wu, Kai-Wei Chang, and Nanyun Peng. 2024. Re-rest: Reflection-reinforced self-training for language agents. arXiv preprint arXiv:2406.01495 (2024). - Durante et al. (2024) Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, et al. 2024. Agent ai: Surveying the horizons of multimodal interaction. arXiv preprint arXiv:2401.03568 (2024). - Es et al. (2025) Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2025. Ragas: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217 [cs.CL] https://arxiv.org/abs/2309.15217 - Fang et al. (2024) Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. 2024. Llm agents can autonomously exploit one-day vulnerabilities. arXiv preprint arXiv:2404.08144 13 (2024), 14. - Fourney et al. (2024) Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang, Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, and Saleema Amershi. 2024. Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks. doi: 10.48550/arXiv.2411.04468 - Fu et al. (2024) Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. 2024. AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 119919–119948. - Galileo (2025) Galileo. 2025. Introducing agentic evaluations. https://www.galileo.ai/blog/introducing-agentic-evaluations Accessed: 2025-05-20. - Gan et al. (2024) Yuyou Gan, Yong Yang, Zhe Ma, Ping He, Rui Zeng, Yiming Wang, Qingming Li, Chunyi Zhou, Songze Li, Ting Wang, et al. 2024. Navigating the risks: A survey of security, privacy, and ethics threats in llm-based agents. arXiv preprint arXiv:2411.09523 (2024). - Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. arXiv:2009.11462 [cs.CL] https://arxiv.org/abs/2009.11462 - Gioacchini et al. (2024) Luca Gioacchini, Giuseppe Siracusano, Davide Sanvito, Kiril Gashteovski, David Friede, Roberto Bifulco, and Carolin Lawrence. 2024. AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents. doi: 10.48550/arXiv.2404.06411 - Google (2024) Google. 2024. Google Vortex AI. https://cloud.google.com/vertex-ai - Gu et al. (2024) Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594 (2024). - Guan et al. (2024) Zhenyu Guan, Xiangyu Kong, Fangwei Zhong, and YIzhou Wang. 2024. Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 38. Curran Associates, Inc., 0. - Guo et al. (2024) Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680 (2024). - Han et al. (2024) Shanshan Han, Qifan Zhang, Yuhang Yao, Weizhao Jin, Zhaozhuo Xu, and Chaoyang He. 2024. LLM multi-agent systems: Challenges and open problems. arXiv preprint arXiv:2402.03578 (2024). - He et al. (2024b) Feng He, Tianqing Zhu, Dayong Ye, Bo Liu, Wanlei Zhou, and Philip S Yu. 2024b. The emerged security and privacy of llm agent: A survey with case studies. arXiv preprint arXiv:2407.19354 (2024). - He et al. (2024a) Zhitao He, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. 2024a. AgentsCourt: Building Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentation. arXiv:2403.02959v3 (2024). https://arxiv.org/pdf/2403.02959 - Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. arXiv:2009.03300 [cs.CY] https://arxiv.org/abs/2009.03300 - Huang et al. (2024a) Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024a. Understanding the planning of LLM agents: A survey. doi: 10.48550/arXiv.2402.02716 - Huang et al. (2024b) Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. 2024b. MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use. arXiv:2310.03128 [cs.SE] https://arxiv.org/abs/2310.03128 - Jia et al. (2024) Zixia Jia, Mengmeng Wang, Baichen Tong, Song-Chun Zhu, and Zilong Zheng. 2024. LangSuitE: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments. arXiv preprint arXiv:2406.16294 (2024). - Jimenez et al. (2024) Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? doi: 10.48550/arXiv.2310.06770 - Kang and Xiong (2024) H Kang and C Xiong. 2024. ResearchArena: Benchmarking LLMs’ Ability to Collect and Organize Information as Research Agents. (2024). - Koh et al. (2024) Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024. VisualWebArena: evaluating multimodal agents on realistic visual web tasks. doi: 10.48550/arXiv.2401.13649 - Krishna et al. (2023) Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization. arXiv:2301.13298 [cs.CL] https://arxiv.org/abs/2301.13298 - LangChain (2024) LangChain. 2024. LangGraph Platform. https://www.langchain.com/langgraph-platform - Levi and Kadar (2025) Elad Levi and Ilan Kadar. 2025. IntellAgent: a multi-agent framework for evaluating conversational AI systems. doi: 10.48550/arXiv.2501.11067 - Li et al. (2024a) Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024a. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579 (2024). - Li et al. (2023) Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-bank: a comprehensive benchmark for tool-augmented LLMs. doi: 10.48550/arXiv.2304.08244 - Li et al. (2024b) Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. 2024b. A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth 1, 1 (2024), 9. - Li et al. (2024c) Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. 2024c. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459 (2024). - Li et al. (2024d) Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. 2024d. Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 49881–49913. - Liang et al. (2023) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2023. Holistic Evaluation of Language Models. doi: 10.48550/arXiv.2211.09110 - Lin et al. (2023) Jiaju Lin, Haoran Zhao, Aochi Zhang, Yiting Wu, Huqiuyue Ping, and Qin Chen. 2023. Agentsims: An open-source sandbox for large language model evaluation. arXiv preprint arXiv:2308.04026 (2023). - Liu et al. (2024a) Bing Liu, Zhou Jianxiang, Dan Meng, and Haonan Lu. 2024a. An Evaluation Mechanism of LLM-based Agents on Manipulating APIs. In Findings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 4649–4662. doi: 10.18653/v1/2024.findings-emnlp.267 - Liu et al. (2025) Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, Yuheng Cheng, Suyuchen Wang, Xiaoqiang Wang, Yuyu Luo, Haibo Jin, Peiyan Zhang, Ollie Liu, Jiaqi Chen, Huan Zhang, Zhaoyang Yu, Haochen Shi, Boyan Li, Dekun Wu, Fengwei Teng, Xiaojun Jia, Jiawei Xu, Jinyu Xiang, Yizhang Lin, Tianming Liu, Tongliang Liu, Yu Su, Huan Sun, Glen Berseth, Jianyun Nie, Ian Foster, Logan Ward, Qingyun Wu, Yu Gu, Mingchen Zhuge, Xiangru Tang, Haohan Wang, Jiaxuan You, Chi Wang, Jian Pei, Qiang Yang, Xiaoliang Qi, and Chenglin Wu. 2025. Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems. doi: 10.48550/arXiv.2504.01990 - Liu et al. (2024b) Wei Liu, Chenxi Wang, Yifei Wang, Zihao Xie, Rennai Qiu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, and Chen Qian. 2024b. Autonomous agents for collaborative task under information asymmetry. arXiv preprint arXiv:2406.14928 (2024). - Liu et al. (2024c) Wei Liu, Chenxi Wang, YiFei Wang, Zihao Xie, Rennai Qiu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, and Chen Qian. 2024c. Autonomous Agents for Collaborative Task under Information Asymmetry. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. - Liu et al. (2023b) Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023b. AgentBench: Evaluating LLMs as Agents. doi: 10.48550/arXiv.2308.03688 - Liu et al. (2023c) Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023c. AgentBench: Evaluating LLMs as Agents. arXiv:2308.03688 [cs.AI] https://arxiv.org/abs/2308.03688 - Liu et al. (2024d) Yanjiang Liu, Tianyun Zhong, Yaojie Lu, Hongyu Lin, Ben He, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, et al. 2024d. XMC-Agent: Dynamic Navigation over Scalable Hierarchical Index for Incremental Extreme Multi-label Classification. In Findings of the Association for Computational Linguistics ACL 2024. 5659–5672. - Liu et al. (2023a) Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, and Qing Wang. 2023a. Chatting with gpt-3 for zero-shot human-like mobile automated gui testing. arXiv preprint arXiv:2305.09434 (2023). - Lou et al. (2025) Renze Lou, Hanzi Xu, Sijia Wang, Jiangshu Du, Ryo Kamoi, Xiaoxin Lu, Jian Xie, Yuxuan Sun, Yusen Zhang, Jihyun Janice Ahn, Hongchao Fang, Zhuoyang Zou, Wenchao Ma, Xi Li, Kai Zhang, Congying Xia, Lifu Huang, and Wenpeng Yin. 2025. AAAR-1.0: Assessing AI’s Potential to Assist Research. doi: 10.48550/arXiv.2410.22394 - Lu et al. (2024) Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292 (2024). - Lù et al. (2024) Xing Han Lù, Zdeněk Kasner, and Siva Reddy. 2024. WebLINX: Real-World Website Navigation with Multi-Turn Dialogue. (Feb. 2024). arXiv:2402.05930 [cs.CL] - Ma et al. (2024) Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. 2024. AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents. doi: 10.48550/arXiv.2401.13178 - Maharana et al. (2024) Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating Very Long-Term Conversational Memory of LLM Agents. arXiv:2402.17753 [cs.CL] https://arxiv.org/abs/2402.17753 - Mathur et al. (2024) Puneet Mathur, Alexa Siu, Nedim Lipka, and Tong Sun. 2024. MATSA: Multi-Agent Table Structure Attribution. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 250–258. - Mehrotra et al. (2024) Siddharth Mehrotra, Carolina Centeio Jorge, Catholijn M Jonker, and Myrthe L Tielman. 2024. Integrity-based explanations for fostering appropriate trust in AI agents. ACM Transactions on Interactive Intelligent Systems 14, 1 (2024), 1–36. - Microsoft (2024) Microsoft. 2024. Azure Foundry. https://azure.microsoft.com/en-us/products/ai-foundry - Nakajima (2023) Yohei Nakajima. 2023. Babyagi. GitHub repository (2023). - Nvidia (2024) Nvidia. 2024. LLM Benchmark Metrics. https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.htm - OpenAI (2023) OpenAI. 2023. OpenAI Evals. https://github.com/openai/evals - Paglieri et al. (2024) Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rocktäschel. 2024. BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games. arXiv:2411.13543 [cs.AI] https://arxiv.org/abs/2411.13543 - Pan et al. (2024) Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, and Zhengyang Wu. 2024. WebCanvas: benchmarking web agents in online environments. doi: 10.48550/arXiv.2406.12373 - Park et al. (2023) Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology. 1–22. - Patil et al. (2023) Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large Language Model Connected with Massive APIs. doi: 10.48550/arXiv.2305.15334 arXiv:2305.15334 [cs]. - Qin et al. (2023) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. ToolLLM: facilitating large language models to master 16000+ real-world APIs. doi: 10.48550/arXiv.2307.16789 - Qiu et al. (2025) Haoyi Qiu, Alexander R. Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, and Chien-Sheng Wu. 2025. Evaluating Cultural and Social Awareness of LLM Web Agents. arXiv:2410.23252 (2025). - Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. arXiv preprint arXiv:1806.03822 (2018). - Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv:1606.05250 [cs.CL] - Reimann et al. (2023) Merle M. Reimann, Catharine Oertel, Florian A. Kunneman, and Koen V. Hindriks. 2023. Predicting interaction quality aspects using level-based scores for conversational agents. In Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents. ACM, Würzburg Germany, 1–8. doi: 10.1145/3570945.3607332 - Ruan et al. (2024) Wenjie Ruan, Wei Huang, Xiaowei Huang, Gaojie Jin, Yi Dong, Changshun Wu, Saddek Bensalem, Ronghui Mu, Yi Qi, Xingyu Zhao, et al. 2024. ToolEmu. In Proceedings of t. - Ruan et al. (2023) Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. 2023. Identifying the risks of lm agents with an lm-emulated sandbox. arXiv preprint arXiv:2309.15817 (2023). - Shen et al. (2024) Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. 2024. TaskBench: Benchmarking Large Language Models for Task Automation. doi: 10.48550/arXiv.2311.18760 - Shi et al. (2024) Honghao Shi, Longkai Cheng, Wenli Wu, Yuhang Wang, Xuan Liu, Shaokai Nie, Weixv Wang, Xuebin Min, Chunlei Men, and Yonghua Lin. 2024. Enhancing Cluster Resilience: LLM-agent Based Autonomous Intelligent Cluster Diagnosis System and Evaluation Framework. arXiv:2411.05349 (2024). https://arxiv.org/pdf/2411.05349 - Shi et al. (2017) Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. 2017. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning. PMLR, 3135–3144. - Siegel et al. (2024) Zachary S. Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. 2024. CORE-bench: fostering the credibility of published research through a computational reproducibility agent benchmark. doi: 10.48550/arXiv.2409.11363 - Starace et al. (2025) Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. 2025. PaperBench: Evaluating AI’s Ability to Replicate AI Research. doi: 10.48550/arXiv.2504.01848 - Stroebl et al. (2025) Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. 2025. HAL: A Holistic Agent Leaderboard for Centralized and Reproducible Agent Evaluation. https://github.com/princeton-pli/hal-harness. - Talebirad and Nadiri (2023) Yashar Talebirad and Amirhossein Nadiri. 2023. Multi-agent collaboration: Harnessing the power of intelligent llm agents. arXiv preprint arXiv:2306.03314 (2023). - Tao et al. (2024) Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. 2024. Magis: Llm-based multi-agent framework for github issue resolution. Advances in Neural Information Processing Systems 37 (2024), 51963–51993. - Trivedi et al. (2024) Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. 2024. AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. doi: 10.48550/arXiv.2407.18901 - Wang et al. (2024a) Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, and Shoufa Chen. 2024a. MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents. arXiv preprint arXiv:2406.08184 (2024). - Wang et al. (2024c) Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. 2024c. A Survey on Large Language Model based Autonomous Agents. Frontiers of Computer Science 18, 6 (Dec. 2024), 186345. doi: 10.1007/s11704-024-40231-1 - Wang et al. (2024b) Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, et al. 2024b. Gui agents with foundation models: A comprehensive survey. arXiv preprint arXiv:2411.04890 (2024). - Xia et al. (2024) Boming Xia, Qinghua Lu, Liming Zhu, Zhenchang Xing, Dehai Zhao, and Hao Zhang. 2024. An Evaluation-Driven Approach to Designing LLM Agents: Process and Architecture. arXiv preprint arXiv:2411.13768 (2024). - Xiao et al. (2024) Ruixuan Xiao, Wentao Ma, Ke Wang, Yuchuan Wu, Junbo Zhao, Haobo Wang, Fei Huang, and Yongbin Li. 2024. FlowBench: revisiting and benchmarking workflow-guided planning for LLM-based agents. doi: 10.48550/arXiv.2406.14884 - Xu et al. (2024) Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. 2024. Theagentcompany: benchmarking llm agents on consequential real world tasks. arXiv preprint arXiv:2412.14161 (2024). - Xu et al. (2022) Guangxuan Xu, Ruibo Liu, Fabrice Harel-Canada, Nischal Chandra, and Nanyun Peng. 2022. EnDex: Evaluation of Dialogue Engagingness at Scale. 4884–4893. doi: 10.18653/v1/2022.findings-emnlp.359 - Xu et al. (2023) Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See Kiong Ng, and Jiashi Feng. 2023. Magic: Investigation of large language model powered multi-agent in cognition, adaptability, rationality and collaboration. arXiv preprint arXiv:2311.08562 (2023). - Yan et al. (2024) Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley Function Calling Leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html. - Yang et al. (2024b) Qisen Yang, Zekun Wang, Honghui Chen, Shenzhi Wang, Yifan Pu, Xin Gao, Wenhao Huang, Shiji Song, and Gao Huang. 2024b. PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agents. arXiv:2402.12326v2 (2024). https://arxiv.org/pdf/2402.12326 - Yang et al. (2024a) Wenkai Yang, Xiaohan Bi, Yankai Lin, Sishuo Chen, Jie Zhou, and Xu Sun. 2024a. Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents. arXiv:2402.11208 [cs.CR] https://arxiv.org/abs/2402.11208 - Yao et al. (2022) Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems 35 (2022), 20744–20757. - Yao et al. (2024) Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. $\tau$ -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. doi: 10.48550/arXiv.2406.12045 - Yao et al. (2023a) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023a. ReAct: Synergizing Reasoning and Acting in Language Models. In Proceedings of the International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2210.03629v3 - Yao et al. (2023b) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023b. ReAct: Synergizing Reasoning and Acting in Language Models. doi: 10.48550/arXiv.2210.03629 - Yehudai et al. (2025) Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2025. Survey on Evaluation of LLM-based Agents. doi: 10.48550/arXiv.2503.16416 - Yin et al. (2024) Sheng Yin, Xianghe Pang, Yuanzhuo Ding, Menglan Chen, Yutong Bi, Yichen Xiong, Wenhao Huang, Zhen Xiang, Jing Shao, and Siheng Chen. 2024. SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents. arXiv preprint arXiv:2412.13178 (2024). - Yoran et al. (2024) Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. 2024. AssistantBench: can web agents solve realistic and time-consuming tasks? doi: 10.48550/arXiv.2407.15711 - Yu et al. (2024a) Erxin Yu, Jing Li, Ming Liao, Siqi Wang, Zuchen Gao, Fei Mi, and Lanqing Hong. 2024a. Cosafe: Evaluating large language model safety in multi-turn dialogue coreference. arXiv preprint arXiv:2406.17626 (2024). - Yu et al. (2024b) Yangyang Yu, Zhiyuan Yao, Haohang Li, Zhiyang Deng, Yuechen Jiang, Yupeng Cao, Zhi Chen, Jordan Suchow, Zhenyu Cui, Rong Liu, Zhaozhuo Xu, Denghui Zhang, Koduvayur (Suba) Subbalakshmi, GUOJUN XIONG, Yueru He, Jimin Huang, Dong Li, and Qianqian Xie. 2024b. FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 137010–137045. - Yuan et al. (2024) Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, et al. 2024. R-judge: Benchmarking safety risk awareness for llm agents. arXiv preprint arXiv:2401.10019 (2024). - Zhan et al. (2025) Qiusi Zhan, Richard Fang, Henil Shalin Panchal, and Daniel Kang. 2025. Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents. arXiv:2503.00061 [cs.CR] https://arxiv.org/abs/2503.00061 - Zhang et al. (2024d) Andy K. Zhang, Neil Perry, and Riya Dulepet et al. 2024d. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. arXiv:2408.08926v3 (2024). https://arxiv.org/pdf/2408.08926 - Zhang et al. (2025) Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. 2025. Large Language Model-Brained GUI Agents: A Survey. doi: 10.48550/arXiv.2411.18279 - Zhang et al. (2023a) Danyang Zhang, Lu Chen, and Kai Yu. 2023a. Mobile-env: A universal platform for training and evaluation of mobile interaction. CoRR (2023). - Zhang et al. (2024f) Danyang Zhang, Zhennan Shen, Rui Xie, Situo Zhang, Tianbao Xie, Zihan Zhao, Siyuan Chen, Lu Chen, Hongshen Xu, Ruisheng Cao, and Kai Yu. 2024f. Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction. arXiv:2305.08144 [cs.AI] https://arxiv.org/abs/2305.08144 - Zhang et al. (2024c) Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. 2024c. Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents. arXiv preprint arXiv:2410.02644 (2024). - Zhang et al. (2024e) Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024e. AutoCodeRover: Autonomous Program Improvement (ISSTA 2024). Association for Computing Machinery, New York, NY, USA, 1592–1604. doi: 10.1145/3650212.3680384 - Zhang et al. (2023c) Yue Zhang, Ming Zhang, Haipeng Yuan, Shichun Liu, Yongyao Shi, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023c. LLMEval: A Preliminary Study on How to Evaluate Large Language Models. arXiv:2312.07398 [cs.AI] https://arxiv.org/abs/2312.07398 - Zhang et al. (2024a) Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. 2024a. A Survey on the Memory Mechanism of Large Language Model based Agents. doi: 10.48550/arXiv.2404.13501 - Zhang et al. (2024b) Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. 2024b. Agent-SafetyBench: Evaluating the Safety of LLM Agents. arXiv preprint arXiv:2412.14470 (2024). - Zhang et al. (2023b) Zhiping Zhang, Michelle Jia, B Yao, S Das, A Lerner, D Wang, and T Li. 2023b. It’sa fair game, or is it? examining how users navigate disclosure risks and benefits when using llm-based conversational agents. arXiv preprint arXiv:2309.11653 (2023). - Zhang et al. (2024g) Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. 2024g. MMInA: benchmarking multihop multimodal internet agents. doi: 10.48550/arXiv.2404.09992 - Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595–46623. - Zhou et al. (2023) Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854 (2023). - Zhuge et al. (2024) Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et al. 2024. Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934 (2024).