## Diagram: Comparison of Benchmark Methodologies ### Overview This diagram illustrates a comparison between traditional benchmark methodologies—specifically "Agent Benchmark: Independent Tasks" and "Long Context Benchmark: Information Retrieval"—and a proposed methodology titled "SWE-ContextBench." The diagram uses visual cues (emojis) to suggest that the traditional methods are problematic or insufficient (indicated by red angry faces), while the proposed "SWE-ContextBench" is effective (indicated by a green happy face). ### Components/Axes The image is divided into two primary vertical regions: * **Left Region:** Contains two stacked sections, labeled (a) and (b). * **Right Region:** Contains one large section, labeled (c). **Visual Elements:** * **Icons:** Robot icons are used to denote tasks or agents. * **Emojis:** Red angry face emojis (😡) are placed next to sections (a) and (b). A green happy face emoji (😊) is placed next to section (c). * **Flow/Relationship:** In section (c), a green arrow indicates a flow of information from an "Experience Pool" to a specific task, labeled "Experience Reuse." ### Detailed Analysis #### Left Region **Section (a): Agent Benchmark: Independent Tasks** * **Status:** Marked with a red angry face emoji. * **Content:** * Task 1: "Fix the bug: Changing an IntegerField to a ForeignKey generates ..." * Task 2: "Fix the bug: Changing the type of a ForeignKey and changing ..." * **Context:** These tasks are presented as isolated, independent units of work. **Section (b): Long Context Benchmark: Information Retrieval** * **Status:** Marked with a red angry face emoji. * **Content:** * Task: "Please identify the fictional character who occasionally breaks the fourth wall with the audience?" * **Context:** This represents a standard information retrieval task. #### Right Region **Section (c): SWE-ContextBench** * **Status:** Marked with a green happy face emoji. * **Central Component:** A light green oval labeled "Experience Pool." * Inside the pool are two text bubbles containing technical context: 1. "The issue was that `BaseFormSet`" 2. "When changing a ForeignKey field type and updating .." * **Flow:** A green arrow labeled "Experience Reuse" originates from the "Experience Pool" and points downward to a task. * **Task:** "Fix the bug: Changing the type of a ForeignKey and changing ..." * **Context:** This section demonstrates a mechanism where previous knowledge (the pool) is applied to a current task. ### Key Observations * **Sentiment Contrast:** The diagram explicitly contrasts the "Independent Tasks" and "Information Retrieval" benchmarks (negative sentiment) with "SWE-ContextBench" (positive sentiment). * **Methodological Difference:** The core differentiator for "SWE-ContextBench" is the inclusion of an "Experience Pool" and the explicit "Experience Reuse" mechanism. * **Task Similarity:** The bug-fixing task in section (c) is identical to the second task in section (a), suggesting that "SWE-ContextBench" is designed to solve the same problems as the "Agent Benchmark" but with a more sophisticated, context-aware approach. ### Interpretation The diagram serves as a conceptual argument for why traditional benchmarks are inadequate for Software Engineering (SWE) tasks. * **The Problem:** Sections (a) and (b) represent "cold start" or isolated problem-solving scenarios. The red emojis imply that treating software engineering tasks as either isolated bug fixes (a) or simple information retrieval (b) fails to capture the reality of software development. * **The Solution:** Section (c) proposes that effective software engineering agents require context. By maintaining an "Experience Pool"—likely a repository of past issues, solutions, and technical insights (e.g., the `BaseFormSet` note)—the agent can perform "Experience Reuse." * **Conclusion:** The diagram suggests that the ability to retrieve and apply relevant historical context is the key missing component in traditional benchmarks, and that "SWE-ContextBench" successfully addresses this by integrating a knowledge-retrieval loop into the task-solving process.