\n ## Diagram: SWE-ContextBench & Agent Benchmarks ### Overview The image presents a diagram illustrating two benchmark types – Agent Benchmark (Independent Tasks) and Long Context Benchmark (Information Retrieval) – alongside a visualization of the SWE-ContextBench concept, focusing on experience reuse. The diagram uses icons representing tasks and a visual metaphor of an "Experience Pool" to demonstrate how prior experience can be leveraged. ### Components/Axes The diagram is divided into three main sections: (a) Agent Benchmark, (b) Long Context Benchmark, and (c) SWE-ContextBench. Each section has a title and a descriptive label. Icons of computer screens with code snippets are used to represent tasks. Arrows indicate the flow of experience reuse. Text boxes contain descriptions of the tasks and the SWE-ContextBench concept. Smiley/frowning face icons indicate performance. ### Detailed Analysis or Content Details **(a) Agent Benchmark: Independent Tasks** * **Label:** "Agent Benchmark: Independent Tasks" – positioned top-left. * **Icon:** A computer screen with code. * **Task 1:** "Fix the bug: Changing an IntegerField to a ForeignKey generates ..." – displayed below the icon. * **Task 2:** "Fix the bug: Changing the type of a ForeignKey and changing ..." – displayed below the first task. * **Performance Indicator:** A frowning face icon with two arrows pointing outwards. **(b) Long Context Benchmark: Information Retrieval** * **Label:** "Long Context Benchmark: Information Retrieval" – positioned bottom-left. * **Icon:** A document with text. * **Task:** "Please identify the fictional character who occasionally breaks the fourth wall with the audience?" – displayed below the icon. * **Performance Indicator:** A frowning face icon with two arrows pointing outwards. **(c) SWE-ContextBench** * **Label:** "SWE-ContextBench" – positioned top-right. * **Icon:** A smiling face. * **Central Element:** A large light blue circle labeled "Experience Pool". * **Text within the circle:** "The issue was that `BaseFormSet`..." * **Arrow 1:** Starts from the text within the "Experience Pool" and points to a smaller box labeled "When changing a ForeignKey field type and updating..." * **Arrow 2:** A curved arrow originating from the "When changing a ForeignKey field type and updating..." box, labeled "Experience Reuse", pointing back towards the "Experience Pool". * **Task (repeated):** "Fix the bug: Changing the type of a ForeignKey and changing ..." – displayed below the "Experience Reuse" arrow. ### Key Observations * The Agent and Long Context Benchmarks both show negative performance (frowning faces). * SWE-ContextBench highlights the potential for "Experience Reuse" to address issues. * The task related to changing ForeignKey types appears in both the Agent Benchmark and is used to illustrate Experience Reuse in SWE-ContextBench. * The SWE-ContextBench diagram suggests a cyclical process where experience gained from solving a problem is stored in an "Experience Pool" and can be reused to solve similar problems. ### Interpretation The diagram illustrates a research framework (SWE-ContextBench) designed to evaluate and improve the ability of agents to leverage past experiences when solving software engineering tasks. The Agent and Long Context Benchmarks represent challenges where agents currently struggle. The SWE-ContextBench concept proposes that by capturing and reusing prior experience, agents can perform better. The repeated task involving ForeignKey types suggests this is a common scenario where experience reuse could be particularly beneficial. The frowning faces on the benchmarks indicate a need for improvement, while the smiling face on SWE-ContextBench suggests a potential solution. The diagram emphasizes the importance of context and the ability to apply knowledge from previous tasks to new, related problems. The "Experience Pool" acts as a knowledge repository, and the "Experience Reuse" arrow signifies the transfer of knowledge.