# Agentless : Demystifying LLM-based Software Engineering Agents
> Contributed equally with author ordering decided byNigiri.
Abstract
Recent advancements in large language models (LLMs) have significantly advanced the automation of software development tasks, including code synthesis, program repair, and test generation. More recently, researchers and industry practitioners have developed various autonomous LLM agents to perform end-to-end software development tasks. These agents are equipped with the ability to use tools, run commands, observe feedback from the environment, and plan for future actions. However, the complexity of these agent-based approaches, together with the limited abilities of current LLMs, raises the following question: Do we really have to employ complex autonomous software agents? To attempt to answer this question, we build Agentless – an agentless approach to automatically resolve software development issues. Compared to the verbose and complex setup of agent-based approaches, Agentless employs a simplistic three-phase process of localization, repair, and patch validation, without letting the LLM decide future actions or operate with complex tools. Our results on the popular SWE-bench Lite benchmark show that surprisingly the simplistic Agentless is able to achieve both the highest performance (32.00%, 96 correct fixes) and low cost ($0.70) compared with all existing open-source software agents! In fact, Agentless has already been adopted by OpenAI as the go-to approach to showcase the real-world coding performance of both GPT-4o and the new OpenAI o1 models. Furthermore, we manually classified the problems in SWE-bench Lite and found problems with exact ground truth patches or insufficient/misleading issue descriptions. As such, we construct SWE-bench Lite- $S$ by excluding such problematic issues to perform more rigorous evaluation and comparison. Our work highlights the currently overlooked potential of a simplistic, cost-effective technique in autonomous software development. We hope Agentless will help reset the baseline, starting point, and horizon for autonomous software agents, and inspire future work along this crucial direction. We have open-sourced Agentless at: https://github.com/OpenAutoCoder/Agentless
1 Introduction
Large language models (LLMs) have become the go-to default choice for code generation [Chen et al., 2021; Austin et al., 2021; Li et al., 2023; Wei et al., 2023]. State-of-the-art LLMs like GPT-4 [OpenAI, 2023] and Claude 3.5 Sonnet Anthropic [2024] have demonstrated their prowess in being able to synthesize code snippets based on given user description. However, compared to the main evaluation setting of simple, self-contained problems, applying LLMs on repository-level software engineering tasks has been understudied. Software engineering tasks like feature addition, program repair, and test generation require an in-depth understanding of not only information within files, containing thousands of lines of code, but also repository-level dependencies across files.
Recently, to address the gap and evaluate the ability of tools to automatically solve real-world software engineering problems, the popular SWE-bench Jimenez et al. [2024a] benchmark has been developed. In SWE-bench, each problem consists of a real-world GitHub issue description and the corresponding Python repository. The task is to modify the repository to resolve the issue, either fixing a bug or introducing a new feature. Recently, the authors have published a subset of the benchmark – SWE-bench Lite swe [2024] (300 problems) that performs further filtering and focuses on bug fixing issues.
To solve the challenging real-world software development problems from SWE-bench, inspired by the Devin AI Software Engineer dev [2024], there has been a significant body of work from both academia and industry focusing on developing agent-based approaches Zhang et al. [2024c]; Gauthier [2024]; Yang et al. [2024a]; Chen et al. [2024]; Ma et al. [2024]; Bouzenia et al. [2024]. While there is not a fixed definition for agent-based approaches, they generally equip LLMs with a set of tools and allow agents to iteratively and autonomously perform actions, observe feedback, and plan future steps Liu et al. [2024c]. Example tools can include the ability to open/write/create files, search for code lines, run tests, and execute shell commands. In each attempt to solve a problem, agent-based approaches will have multiple turns, where each turn consists of performing an action. Subsequent turns depend on previous actions and the feedback information the agent receives from the environment.
At first glance, agent-based approaches appear to be a natural and straightforward way to tackle software development tasks. After all, human developers also perform similar actions and use feedback to plan future steps. However, the disparity between human and current LLM abilities leads to the following limitations of agent-based approaches:
- Complex tool usage/design. To utilize tools, current agent-based approaches apply an abstraction layer between the agent and the environment. Examples are mapping real actions to API calls so that agents can use tools by outputting an API call instruction. However, such abstractions and API call specifications require careful design of input/output formats and can easily lead to incorrect or imprecise tool design/usage, especially for more complex action spaces. Given the iterative nature of agent-based approaches, where current action/plan depends on previous turns, incorrectly or imprecisely defining/using a tool can both reduce performance and incur additional cost in wasted LLM queries.
- Lack of control in decision planning. In addition to using tools, current agent-based approaches also delegate the decision-making process to the agents, allowing them to decide when and what action to perform. The agents decide the current action to take based on previous actions taken and the feedback provided by the environment, often with minimal checks to ensure the action taken make sense. Due to the large possible action space and feedback response, it can be extremely easy for autonomous agents to become confused and perform sub-optimal explorations. Furthermore, to solve an issue, an agent can take upwards of 30 or 40 turns, which makes it extremely difficult to both understand the decisions made by the agents and also debug the exact turns where an incorrect decision is made.
- Limited ability to self-reflect. Existing agents struggle with the capability to perform self-reflection Olausson et al. [2023]; Huang et al. [2024]. That is to say they tend to take all information/feedback and do not know how to filter out or correct irrelevant, incorrect, or misleading information Shi et al. [2023]; Zhang et al. [2023]. The limited ability to self-reflect means that an incorrect step can be easily amplified and negatively affect all future decisions made by the agent.
In this paper, we advocate that instead of rushing to develop increasingly complex LLM agent-based approaches and tools for software development (which can also be non-trivial to use or replicate due to the fully autonomous setup), we should first take a step back and ask the following introspective question: Do we really have to employ complex autonomous software agents?
Our work. We set out to answer this important question by building Agentless – an agentless approach to automatically resolve software development issues. To solve each issue, Agentless follows a simple three phase process: localization, repair, and patch validation. In the localization process, Agentless employs a hierarchical process to first localize the fault to specific files, then to relevant classes or functions, and finally to fine-grained edit locations. Agentless 's localization process make uses of both LLM-based localization as well classic information-retrieval-based localization idea Zhou et al. [2012]. To perform repair, Agentless takes the localized edit locations and generates multiple candidate patches in a simple diff format. At the same time, Agentless generates reproduction tests that can reproduce the original error and help with candidate patch selection. Finally, Agentless re-ranks all remaining patches and selects one to submit in order to fix the issue. While Agentless leverages LLMs to perform each detailed task, unlike prior complex agent-based tools, Agentless does not allow LLMs to autonomously decide future actions or operate with any complex tools. Our deliberate choice to avoid using agents not only allows Agentless to have a simplistic and straightforward design that is easy to understand, but also helps avoid the above-mentioned limitations of LLM agents in software development. We evaluate Agentless on the popular SWE-bench Lite [swe, 2024] benchmark and demonstrate that Agentless not only achieves the highest performance (32.00%) among all open-source approaches, but also does so at a fraction of the cost!
Furthermore, we performed a fine-grained manual analysis on the SWE-bench Lite dataset and classified all its problems into different categories across dimensions like problem description, ground truth patch, and bug location information. Surprisingly, we observed that SWE-bench Lite contains problems with exact ground truth patch in the description (4.3%), problems with missing critical information needed to solve the issue (10.0%), and problems that include misleading solutions in the issue description (5.0%). Recognizing these issues, we built SWE-bench Lite- $S$ , which removes such problematic problems, and serves as a more rigorous benchmark to evaluate the ability to solve real-world software development challenges. Overall, in an era focused on achieving top placements on leaderboards, our work highlights the overlooked potential of a simplistic, cost-effective technique in autonomous software development. We hope Agentless will help reset the baseline, starting point, and horizon for autonomous software agents, and inspire future work along this crucial direction.
Contributions. In this work, we make the following contributions:
- An agentless approach. We propose Agentless, an agentless approach to automatically solve software development problems. Agentless leverages LLM-empowered prompting-based and embedding-based retrieval to perform hierarchical localization. During repair, Agentless samples multiple candidate patches in a simple diff format for efficient patch generation. Agentless then generates reproduction tests to verify that the issue has been resolved. Finally, Agentless leverages both regression tests and generated reproduction tests to select the final submission patch.
- Extensive evaluation. We evaluate Agentless on the popular SWE-bench Lite dataset comparing against state-of-the-art agent-based approaches. Our results demonstrates that Agentless is able to achieve higher performance (32.00%, 96 correct fixes) than all open-source approaches, with comparably low cost as well. This shows the previously overlooked potential of a simplistic technique in autonomous software development. Additionally, Agentless has already been adopted by OpenAI as the go-to approach to showcase the real-world coding performance of GPT-4o Chowdhury et al. [2024] as well as the new o1 model family OpenAI [2024c] We further perform a rigorous ablation study to understand the effectiveness of different components of Agentless on the final performance.
- SWE-bench Lite- $S$ benchmark. We performed manual classifications on the problems in the popular SWE-bench Lite dataset. We found that there are problematic problems with unclear, misleading issue descriptions, as well as problems that contain exact ground truth patches. To address these issues, we constructed a filtered dataset of SWE-bench Lite- $S$ that excludes such problematic issues, enabling more rigorous evaluation and comparison. This has also been confirmed recently by OpenAI, which acknowledged our benchmark and released SWE-bench Verified Chowdhury et al. [2024] along the same direction.
2 Background and Related Work
2.1 Agent-based Software Engineering
With the emergence and popularity of agent-based frameworks Xi et al. [2023], recently researchers and industry practitioners have begun developing agent-based approaches to solve software engineering tasks. Devin dev [2024] (and OpenDevin ope [2024b], open-source alternative), is one of the first end-to-end LLM agent-based framework. Devin uses agents to first perform planning based on user requirement, then allows the agent to use file editor, terminal, and web search engine tools to iteratively perform the task. SWE-agent Yang et al. [2024a] designs a custom agent-computer interface (ACI) that allows the LLM agent to interact with the repository environment with actions such as reading, editing files, and running bash commands. Aider Gauthier [2024] first provides a detailed repository map constructed with static and call graph analysis to the LLM to localize the files that require editing; then it generates a simple diff format as the editing patch and uses regression testing to verify if the patch is plausible. Moatless moa [2024] is another open-source agent tool that obtains relevant code locations by providing the agent with both code search tools as well as retrieval methods using LLM-constructed queries. Similar to Aider, Moatless also generates a simple diff format as the final submitted patch. AutoCodeRover Zhang et al. [2024c] further provides the LLM agent with specific code search APIs (e.g., searching methods in a certain class) to iteratively retrieve code context and locate the bug locations. SpecRover Ruan et al. [2024] later improves over AutoCodeRover and targets specifications (i.e., inferring the intended program behavior) by generating function summaries and also feedback messages during specific agent steps. Furthermore, SpecRover also attempts to generate reproduction tests to reproduce the original issue used to select the final patch. In addition to these highlighted examples, there has been a plethora of other agent-based approaches developed in both open-source Gauthier [2024]; rep [2024]; app [2024]; Zhang et al. [2024d] and close-source/commercial products Bouzenia et al. [2024]; Chen et al. [2024]; Ma et al. [2024]; lin [2024]; fac [2024]; ibm [2024]; ope [2024a]; Liu et al. [2024b]; ama [2024]; sup [2024]; gru [2024]; iso [2024]; men [2024]; hon [2024]; cod [2024].
Compared to these agent-based techniques, Agentless offers a simplistic, interpretable, and cost-effective solution to tackle real-world software engineering issues. Different from agent-based tools, Agentless contains well-defined stages of localization, repair, and patch validation without letting the LLM agent decide future actions or use complex tools. Agentless demonstrates for the first time that an agentless approach can achieve very competitive performance, without the additional baggage of having to provide excessive tools or model complex environment behavior/feedback.
2.2 Fault Localization and Program Repair
Fault localization (FL) Wong et al. [2016] techniques aim to identify the suspicious locations (e.g., statements or methods) in source code related to bugs. Dynamic FL techniques mainly include spectrum-based FL (SBFL) Jones and Harrold [2005]; Abreu et al. [2007, 2009] and mutation-based FL (MBFL) Papadakis and Le Traon [2015]; Moon et al. [2014]; Lou et al. [2020]. SBFL typically computes source code locations primarily covered by failing tests as more suspicious than locations primarily covered by passing tests. MBFL further improves upon that to additionally consider the impact of each source code location on the test outcomes (measured using mutation testing Papadakis et al. [2019]). Besides dynamic techniques, researchers have also proposed to directly leverage information retrieval (IR) techniques Singhal et al. [2001] for static FL. Such IR-based techniques Wang et al. [2015]; Saha et al. [2013]; Wang and Lo [2014] formulate FL as a search problem and compare the textual similarity between code elements and the bug report (i.e., query). Moreover, learning-based techniques have also been proposed to leverage machine learning to combine multiple sources of dynamic/static information, including DeepFL Li et al. [2019], FLUCCS Sohn and Yoo [2017], and TRANSFER Meng et al. [2022]. Recently, researchers have proposed LLM-based FL Yang et al. [2024b]; Wu et al. [2023]; Qin et al. [2024]; Kang et al. [2024], which leverages the powerful code and natural language understanding of modern LLMs to directly localize bugs. Meanwhile, most such LLM-based techniques either cannot perform repository-level FL due to the limited context window of LLMs Yang et al. [2024b], or rely on complicated/costly agentic design to navigate through the codebase Kang et al. [2024]; Qin et al. [2024]. In contrast, Agentless employs a simplistic hierarchical FL process (based on both LLMs and IR) to efficiently compute the fine-grained edit locations.
After localizing the bug, the next step is to perform repair. Automated program repair Gazzola et al. [2019] (APR) has been widely studied to automatically generate patches for bugs. Traditional APR techniques can be categorized as template-based Liu et al. [2019]; Ghanbari et al. [2019], heuristic-based Le Goues et al. [2012]; Le et al. [2016], and constraint-based Mechtaev et al. [2016]; Long and Rinard [2015] tools. While effective, traditional APR tools suffer from scalability issues and are limited by their patch variety. As such, researchers have proposed learning-based APR tools either by training NMT (neural machine translation) models Jiang et al. [2021]; Li et al. [2020]; Chen et al. [2019]; Zhu et al. [2021] or using pre-trained LLMs to perform repair Xia and Zhang [2022]; Xia et al. [2023a]; Kolak et al. [2022]; Zhang et al. [2024a]. Specifically, LLM-based APR tools, which sample multiple candidate patches per bug, have been shown to be the state-of-the-art due to the powerful coding capability of modern LLMs Xia et al. [2023a]. More recently, agent-based APR techniques have also been proposed Xia and Zhang [2023]; Bouzenia et al. [2024]; Hidvégi et al. [2024]; Chen [2024]; Zhang et al. [2024b]. Inspired by existing LLM-based APR tools, Agentless samples multiple candidate patches per issue to maximize the chance of generating a correct fix. Different from most LLM-based APR techniques, Agentless generates patches using a simple diff format Gauthier [2024] to avoid generating the complete code and instead focus on producing cost-efficient small edits, increasing the reliability and accuracy of patch generation (less chances for hallucination). Furthermore, different from the simplistic bugs studied in most prior work Xia et al. [2023a], Agentless targets complex repository-level issues spanning multiple locations.
2.3 LLM-based Test Generation
In addition to localizing and repairing bugs, another research area that has been adopting LLM is test generation. One area of test generation is fuzz testing Zeller et al. [2019], also known as fuzzing, to generate large amounts of inputs in order to expose bugs in systems. Researchers have applied LLMs to perform fuzzing in domains such as DL libraries Deng et al. [2023, 2024], OS Kernel Yang et al. [2023b]; Oliinyk et al. [2024], compilers Xia et al. [2024]; Yang et al. [2023a]; Ou et al. [2024], network protocols Meng et al. [2024], and mobile applications Liu et al. [2024a]. LLM-based fuzzers have demonstrated their impact by detecting many bugs not found by traditional fuzzers as well as unlocking new fuzzing domains. Besides fuzzing, researchers have also proposed to leverage LLMs for unit test generation to test individual software units (e.g., methods/classes) Liu et al. [2024c], such as CodeMosa Lemieux et al. [2023], ChatTester Yuan et al. [2024], TestPilot Schäfer et al. [2023], and CoverUp Pizzorno and Berger [2024].
Bug reproduction is a critical step in investigating bug reports Jin and Orso [2012], and is integrated into many recent software engineering agents ope [2024b]; Ruan et al. [2024]; hon [2024]; men [2024]; fac [2024]; Yang et al. [2024a]; Arora et al. [2024]. For example, SpecRover Ruan et al. [2024] begins by generating a test to reproduce the issue described in the bug report; the test then guides the context retrieval and patching process.Unlike such agent-based approaches, which generate a reproduction test and rely on an LLM agent to decide whether the test is correct, Agentless simply executes multiple sampled tests and verifies if the execution results indicate the issue has been reproduced.
3 Agentless
<details>
<summary>resources/grinning-cat_1f63a.png Details</summary>
### Visual Description
# Technical Image Analysis Report
## 1. Image Overview
The provided image is a stylized, flat-design vector illustration of a cat's face. It uses a minimalist geometric aesthetic characterized by rounded corners and a limited color palette. There is no textual information, data tables, or charts present in the image.
## 2. Component Breakdown
### Head Structure
* **Shape:** A square-like base with rounded bottom corners.
* **Color:** Primary orange (#FFB347).
* **Ears:** Two large, upright triangles at the top corners. The inner ear sections are represented by smaller, darker orange triangles (#FF8C42).
### Facial Features
* **Eyes:** Two vertically oriented ovals, symmetrical in placement.
* **Color:** Dark brown/charcoal (#3D2B2B).
* **Nose:** A small, inverted triangle with rounded corners located centrally below the eyes.
* **Color:** Bright pink (#FF00A6).
* **Mouth:** A wide, crescent-shaped "smiling" mouth located below the nose.
* **Color:** Magenta/Purple (#C11B8D).
* **Whiskers:** Four diagonal lines (two on each side) extending outward from the sides of the face.
* **Color:** Bright orange-red (#FF7F32).
* **Orientation:** The top whiskers angle slightly upward, while the bottom whiskers angle slightly downward.
## 3. Spatial Grounding and Geometry
* **Symmetry:** The illustration is perfectly symmetrical along the vertical axis.
* **Background:** The image is set against a neutral light gray background (#EDEDED).
* **Style:** Flat design with no gradients, shadows, or textures.
## 4. Data Extraction
* **Textual Content:** None.
* **Numerical Data:** None.
* **Legend/Labels:** None.
## 5. Summary for Technical Documentation
This image serves as a graphical icon or avatar representing a "smiling cat." It is composed of basic geometric primitives (squares, triangles, ovals, and arcs) and utilizes a high-contrast color scheme to define facial features without the use of outlines.
</details>
Approach
<details>
<summary>x1.png Details</summary>
### Visual Description
# Technical Diagram: Automated Software Issue Resolution Pipeline
This diagram illustrates a multi-stage workflow for automatically resolving software issues using Large Language Models (LLMs) and embeddings. The process is divided into three main functional phases: Localization, Repair, and Patch Validation.
## Legend and Components
Located at the bottom right [x: 870, y: 550]:
* **Blue Box:** Localization
* **Purple Box:** Repair
* **Green Box:** Patch Validation
* **Robot Icon:** LLM (Large Language Model)
* **Document/Purple Icon:** Embedding
---
## Phase 1: Localization (Blue Region)
This phase focuses on narrowing down the codebase to specific files, classes, and functions relevant to a reported issue.
1. **Input:**
* **Issue:** "Lambdify misinterprets some matrix expressions Using lambdify on an..."
* **Project Codebase:** Represented as a GitHub repository containing various folders and files.
2. **Step 1:** The codebase and issue are processed to extract the **Repo structure**.
* *Example structure shown:* `views/`, `__init__.py`, `csrf.py`, `static.py`, `generic/` (`detail.py`, `__init__.py`, `edit.py`, `list.py`).
3. **Step 2 & 3 (Localize to Top-N Files):** An LLM and an Embedding tool process the issue and repo structure to identify relevant files.
4. **Step 4 (Localize to Classes & Functions):** The LLM narrows the scope further.
* **class RegexValidator:** `def __call__`, `def __eq__` in `static.py`.
* **class Handler:** `def load_middleware`, `def _matrixify` in `generic/detail.py`.
* **class Truncator:** `def _text_chars`, `def _truncate_html` in `csrf.py`.
5. **Step 5 (Localize to Edit Locations):** The LLM identifies specific code blocks for modification.
* **function: edit_matrix:** `def edit_matrix(): if debug and name:`
* **class: Handler ExtHandler:** `class Handler: def load_middleware ... class ExtHandler:`
---
## Phase 2: Repair (Purple Region)
This phase involves generating potential code fixes based on the localized information.
6. **Step 6 (Generate Patches):** The LLM takes the localized edit locations and the original issue to produce multiple candidate patches.
* **Input context:** `lines: 120-230` and `class: Handler`.
* **Output:** A stack of "Patch" objects, each showing line additions (+12) and deletions (-20) with a visual diff bar (green/red).
---
## Phase 3: Patch Validation (Green Region)
This phase ensures the generated patches actually fix the issue without introducing regressions.
7. **Step 7 (Generate Reprod. Tests):** The LLM generates reproduction test cases (`test.py`) based on the issue description.
* *Code snippet:* `def test(): assert ...`
8. **Step 8:** The generated patches from the Repair phase are applied to the codebase.
9. **Step 9 (Filter & Rank Patches):** The reproduction tests are run against the patched code. The LLM filters out patches that fail the tests and ranks the successful ones.
10. **Step 10 (Submitted Patch):** The highest-ranking valid patch is selected as the final output.
---
## Data Flow Summary
* **Primary Flow:** Issue + Codebase $\rightarrow$ File Localization $\rightarrow$ Function Localization $\rightarrow$ Edit Location $\rightarrow$ Patch Generation $\rightarrow$ Patch Filtering $\rightarrow$ Final Submitted Patch.
* **Secondary Flow:** The Issue also triggers the generation of reproduction tests used specifically in the final validation stage.
* **Feedback Loop:** The diagram shows a red line connecting the "Project Codebase" to the "Patch Validation" and "Submitted Patch" stages, indicating the context used for testing and the final application of the fix.
</details>
Figure 1: Overview of Agentless.
Figure 1 shows the overview of Agentless, consisting of three phases: localization, repair, and patch validation. We first take in the issue description and the existing project codebase as input. Then, we begin our hierarchical localization process by turning the project codebase into a tree-like structure that illustrates the relative location of each file in the project 1 . Next, using this repository structure along with the original issue description, we prompt the LLM to localize and rank the top N most suspicious files that likely require editing to solve the issue 2 . Since our repository structure format does not contain detailed source code information, we additionally retrieve files with most relevant code snippets with the issue description using embedding-based retrieval 3 . We then combine the retrieved files with the LLM-localized files to obtain the final list of suspicious files. However, not all contents in each file need to be modified. As such, we provide a skeleton for each file (i.e., a list of declaration headers of the classes and functions) and ask the LLM to output a specific list of classes and functions that we should examine more closely to fix the bug 4 . We then provide the complete code content of the previous locations and ask the LLM to finalize a smaller set of edit locations (i.e., classes, functions, or even specific lines) 5 . For the repair phase, we provide the code snippets at these edit locations together with the issue description and prompt the LLM to sample multiple patches to solve the issue 6 . Next, we enter the patch validation phase, where we first ask the LLM to sample multiple reproduction tests that aim to replicate the original issue 7 , and then select the optimal one based on actual execution results on the original codebase 8 . Agentless uses the reproduction test along with existing regression tests for patch ranking/selection 9 . Finally, Agentless selects the top-ranked patch as the final patch for submission 10 . We now describe the steps in each of Agentless 's phases in more detail.
3.1 Localization
<details>
<summary>resources/weary-cat_1f640.png Details</summary>
### Visual Description
# Image Analysis Report: Stylized Graphic Illustration
## 1. Overview
The provided image is a stylized, geometric vector illustration of a cat's face and paws, conveying an expression of surprise or shock. The image contains no text, data tables, charts, or legends. It is composed entirely of flat shapes and a limited color palette.
## 2. Component Isolation
### Head/Face Region (Upper and Middle)
* **Shape:** The main head is a large orange square with two triangular ears protruding from the top.
* **Eyes:** Two large, perfectly circular white shapes positioned in the upper-middle section of the square.
* **Nose:** A small, inverted pink triangle located centrally below the eyes.
* **Mouth:** A vertical purple oval located directly below the nose, suggesting an open-mouthed "O" expression.
* **Ear Details:** The inner ears are represented by smaller, darker orange triangles nested within the larger ear shapes.
### Paws/Lower Region (Bottom)
* **Placement:** Two rounded rectangular shapes (paws) are positioned at the bottom left and bottom right, partially overlapping the main face square.
* **Details:** Each paw features three vertical, dark orange stripes representing claws or fur patterns.
* **Whiskers:** Three small, dark orange rectangular protrusions extend horizontally from the outer edges of each side of the face/paw area.
## 3. Color Palette and Technical Specifications
The illustration utilizes a specific set of flat colors:
* **Primary Orange:** Used for the main head and paws.
* **Dark Orange:** Used for inner ears, paw stripes, and whiskers.
* **White:** Used for the eyes.
* **Pink:** Used for the nose.
* **Purple:** Used for the mouth.
* **Light Gray:** The background color surrounding the graphic.
## 4. Data Extraction Summary
* **Textual Information:** None.
* **Numerical Data:** None.
* **Charts/Graphs:** None.
* **Diagrammatic Flow:** None.
## 5. Visual Trend/Logic Check
The graphic follows a symmetrical design language. The placement of the eyes, nose, and mouth creates a clear "surprised" facial archetype. The use of geometric primitives (circles, squares, triangles) indicates a minimalist vector art style.
</details>
To fix or implement a new feature, the first step is to obtain the locations in the source code, as without the correct locations, it can be impossible to provide the right edits. The difficulty lies in the fact that there could be hundreds of files with thousands of lines of code each in a repository, whereas the correct locations to edit are only a few selected lines or functions. Agentless addresses this by using a simple three-step hierarchical localization process: 1) localize to suspicious files; 2) localize each selected files into relevant classes, functions, and variables; 3) localize to code edit locations.
3.1.1 Localize to suspicious files.
First, Agentless narrows down potential locations to specific suspicious files. Instead of providing the complete code snippet for each file, Agentless constructs a concise representation of the repository's file and directory structure, similar to the Linux tree command. We refer to this as the repository structure format, which begins with the root folder of the repository and organizes code files or folder names. Files and folders at the same directory level are aligned vertically, and files/folders in sub-directories are indented. We recursively traverse the entire repository to obtain the structure, which will be used as input for the LLM. The repository structure format provides the necessary file paths alongside the neighboring file names to maintain organizational information in the original codebase. Agentless then inputs the processed repository structure along with the original issue description to an LLM, and requests it to identify a list of the top N suspicious files that need further inspection or modification to resolve the issue.
To compliment the prompting-based localization (using file names only), Agentless also uses a simple embedding-based retrieval method to identify additional suspicious files. However, instead of embedding all files in the repository, Agentless first filters out irrelevant folders. This is done by providing the previously described repository structure and asking the LLM to produce a list of irrelevant folders that do not need to be further inspected or modified to resolve the issue. After removing all files from these irrelevant folders, Agentless divides each remaining file into chunks of code segments and computes the embedding for each chunk using an embedding model. Agentless then embeds the original issue description (i.e., the query) and computes the cosine similarity between the resulting query embedding and each chunk embedding to retrieve a list of relevant files that contain code segments with the highest similarity to the query. Finally, Agentless combines the files obtained via prompting with those retrieved via embedding by selecting top N most common files localized by both, resulting in a final list of relevant files.
<details>
<summary>x2.png Details</summary>
### Visual Description
### Technical Document Extraction: Python Code Snippet
The image contains a code snippet written in Python, likely from a web framework like Django, defining custom field classes and a utility function.
#### 1. Code Transcription
```python
class UUIDField(CharField):
default_error_messages = {
'invalid': _('Enter a valid UUID.'),
}
def prepare_value(self, value):
...
def to_python(self, value):
...
class JSONField(CharField):
default_error_messages = {
'invalid': _('Enter a valid JSON.'),
}
widget = Textarea
def __init__(self, encoder=None, decoder=None):
...
def to_python(self, value):
# Process JSON path from the right-hand side
...
def slugify(value, allow_unicode=False):
```
#### 2. Component Analysis
The code is organized into three main sections:
**A. Class: `UUIDField`**
* **Inheritance:** Inherits from `CharField`.
* **Attributes:**
* `default_error_messages`: A dictionary containing an `'invalid'` key with a localized string: `"Enter a valid UUID."`.
* **Methods:**
* `prepare_value(self, value)`: Method signature present; body is truncated with `...`.
* `to_python(self, value)`: Method signature present; body is truncated with `...`.
**B. Class: `JSONField`**
* **Inheritance:** Inherits from `CharField`.
* **Attributes:**
* `default_error_messages`: A dictionary containing an `'invalid'` key with a localized string: `"Enter a valid JSON."`.
* `widget`: Set to `Textarea`.
* **Methods:**
* `__init__(self, encoder=None, decoder=None)`: Constructor with optional encoder and decoder arguments.
* `to_python(self, value)`: Contains a comment: `# Process JSON path from the right-hand side`. Body is truncated with `...`.
**C. Function: `slugify`**
* **Parameters:** `value`, `allow_unicode` (defaults to `False`).
* **Status:** Function definition header only.
#### 3. Visual and Symbolic Elements
* **Syntax Highlighting:**
* **Keywords (`class`, `def`):** Red/Brown.
* **Class/Function Names:** Purple.
* **Strings:** Blue.
* **Comments:** Grey.
* **Arguments/Variables:** Black/Dark Grey.
* **Icons (Top Right):**
* A skull icon (stylized emoji).
* A blue document/file icon.
* **Formatting:** The code uses standard 4-space indentation. Ellipses (`...`) are used to indicate omitted code blocks.
</details>
Figure 2: File skeleton format.
3.1.2 Localize to related elements.
After obtaining the list of suspicious files, Agentless proceeds to the second part of the localization process: localize the related elements within these files. Directly providing the complete context of all files can be large. As such, Agentless builds a compressed format of each file that contains the list of class, function, or variable declarations. We refer to this format as the skeleton format, with an example shown in Figure 2. In the skeleton format, we provide only the headers of the classes and functions in the file. For classes, we further include any class fields and methods (signatures only). Additionally, we also keep comments in the class and module level to provide further information. Compared to providing the entire file context to the model, the skeleton format is a much more concise representation, especially when the file contains thousands of lines, making it impractical/costly to process all at once with existing LLMs. We provide the skeleton of all suspicious files to the LLM at one time in a single prompt, enabling the model to comprehensively analyze the pertinent information and decide the most relevant elements. Using this input, we prompt the LLM to provide a list of related classes and functions that one should examine to fix the provided issue.
<details>
<summary>x3.png Details</summary>
### Visual Description
# Technical Diagram: Automated Search/Replace Diff Workflow
This diagram illustrates a three-step automated process for fixing code issues by generating and applying a "Search/Replace Diff" to an original source file.
## 1. Component Analysis
### A. Original File (Top Left)
A code snippet representing the initial state of a Python file.
* **Label:** Original File
* **Content:**
```python
...
from flask import Flask
app = Flask(__name__)
@app.route("/")
...
```
### B. Issue Identification & Generation (Top Right)
An AI agent (represented by a robot icon) processes an identified issue to generate a fix.
* **Issue Box (Red):** Contains small, partially illegible text describing a bug or requirement.
* **Action Text:** "Generate Search/Replace Diff to fix issue"
* **Flow:** An arrow points from the "Original File" through this process toward the "Search/Replace Diff" block.
### C. Search/Replace Diff (Middle Right)
A structured block showing the specific transformation to be made. It uses a format similar to merge conflict markers.
* **Label:** Search/Replace Diff
* **SEARCH Block (Red background):**
* Header: `<<<<<<< SEARCH`
* Content: `from flask import Flask`
* **Separator:** `=======`
* **REPLACE Block (Green background):**
* Content:
```python
import math
from flask import Flask
```
* Footer: `>>>>>>> REPLACE`
### D. Edited File (Bottom Left)
The final state of the file after the diff has been applied.
* **Label:** Edited File
* **Action Text (Arrow from Diff to Edited File):** "Apply diff to original file to perform edit"
* **Content:**
```python
...
+ import math
from flask import Flask
app = Flask(__name__)
@app.route("/")
...
```
* **Visual Indicator:** The line `+ import math` is highlighted in green with a plus sign, indicating an addition.
---
## 2. Process Flow Summary
1. **Input:** The process starts with the **Original File** containing standard Flask initialization code.
2. **Analysis:** An AI agent identifies an **Issue** and determines that the `math` library needs to be imported.
3. **Diff Generation:** The agent generates a **Search/Replace Diff**.
* It targets the existing line `from flask import Flask`.
* It defines a replacement that prepends `import math` to that line.
4. **Application:** The system applies this diff back to the original file.
5. **Output:** The **Edited File** is produced, now containing the new import statement while preserving the rest of the original code structure.
</details>
Figure 3: Search/Replace edit format.
3.1.3 Localize to edit locations.
The previous localization step provided us with a list of related code elements; since we localize top N suspicious files, these localized related code elements could be from different files. We now directly provide the code content from these elements to the LLM and ask it to localize specific edit locations. Compared to using the entire file, the input context here is much smaller. With this input, we then ask the LLM to identify the final set of edit locations, specified by line numbers, functions, or classes. Our simple hierarchical localization allows Agentless to select a set of relevant code snippets as edit locations for repair.
3.2 Repair
<details>
<summary>resources/cat-with-wry-smile_1f63c.png Details</summary>
### Visual Description
# Technical Image Analysis Report: Stylized Cat Icon
## 1. Image Overview
The provided image is a stylized, flat-design vector illustration of a cat's face. It utilizes a minimalist geometric aesthetic with a limited color palette. There is no alphanumeric text, data tables, or charts present in the image.
## 2. Component Breakdown
### A. Head Structure
* **Shape:** A rounded square (squircle) forms the main face.
* **Color:** Primary fill is a solid orange (#FFB347 or similar).
* **Ears:** Two large triangles extend from the top corners. The inner ear sections are shaded in a darker orange/terracotta tone to provide depth.
### B. Facial Features
* **Eyes/Eyebrows:** Two dark brown/charcoal curved segments. The curves are angled downward toward the center, creating a "grumpy" or "angry" expression.
* **Nose:** A small, inverted equilateral triangle located centrally.
* **Color:** Vibrant Magenta/Pink.
* **Mouth:** A single dark brown/charcoal curved line located on the lower right side of the face (viewer's perspective), suggesting a smirk or a lopsided frown.
* **Whiskers:** Four straight diagonal lines (two on each side) extending outward from the edges of the face.
* **Color:** Dark Orange/Coral.
## 3. Spatial Grounding and Geometry
* **Symmetry:** The head shape and ears are symmetrical along the vertical axis. The facial expression (mouth) and whisker placement are asymmetrical.
* **Style:** Flat design; no gradients, shadows, or textures are used.
## 4. Data Extraction Summary
* **Textual Content:** None.
* **Quantitative Data:** None.
* **Key Visual Indicators:** The downward-sloping brow lines and the asymmetrical mouth curve are the primary indicators of the "grumpy" or "annoyed" character trait intended by the design.
</details>
In the repair stage, the goal is to produce the correct patch to solve the issue. Following existing work on LLM-based program repair Xia and Zhang [2022]; Kolak et al. [2022]; Xia et al. [2023b]; Jiang et al. [2023], we first utilize the identified edit locations and construct a context window of code snippets to provide to the LLM for repair. For example, if the identified location was a class from line 40 to 78, we would produce a context window of [40 - x, 78 + x] where x denotes the context window size. The intuition behind adding the additional code before and after the identified location is to provide the LLM with relevant contextual information for better program repair Xia and Zhang [2022]. If multiple edit locations are identified, we would concatenate these context windows together separated with ``... '' to indicate missing context in the middle.
Using the code snippets, we then ask the LLM to generate patches to solve the issue. However, instead of directly producing the entire code snippet to replace the entire given context, Agentless asks the LLM to generate a Search/Replace edit Gauthier [2024]: a simple diff format to efficiently create each patch. Figure 3 shows an example of the Search/Replace format containing two main parts: 1) search: the original code snippet we want to replace and 2) replace: the replacement code snippet we want to replace with. To apply the generated Search/Replace diff to the original file, we can simply match the search code snippet and replace it with the replacement. This simple diff format avoids generating the complete code and instead focuses on producing small edits, which are not only more cost-efficient, but also more reliable and accurate (less chances for hallucination). For each issue, Agentless uses the LLM to generate multiple potential patches (starting with greedy and then sample multiple patches with higher temperature).
3.3 Patch Validation
<details>
<summary>resources/grinning-cat-with-smiling-eyes_1f638.png Details</summary>
### Visual Description
# Technical Image Analysis Report
## 1. Image Overview
The provided image is a stylized, flat-design vector illustration of a cat's face. It utilizes a minimalist geometric aesthetic characterized by rounded corners and a vibrant color palette.
## 2. Component Isolation and Spatial Analysis
### Head Structure (Main Body)
* **Shape:** A rounded square/rectangular base forming the face.
* **Color:** Primary orange (#FFB327).
* **Ears:** Two large triangular protrusions at the top corners. The inner ear sections are shaded in a darker orange/terracotta tone (#FF8A33).
### Facial Features (Central Region)
* **Eyes:** Two dark brown (#422C33) downward-facing arcs, representing closed, smiling eyes.
* **Nose:** A small, inverted pink triangle (#FF0095) located centrally below the eyes.
* **Mouth:** A wide, upward-curving crescent shape in a deep magenta/purple (#B81D85), indicating a happy or laughing expression.
### Appendages (Lateral Region)
* **Whiskers:** Four distinct diagonal lines (two on each side) extending outward from the lower half of the face.
* **Color:** Bright orange-red (#FF7831).
* **Orientation:** The top whiskers angle slightly upward, while the bottom whiskers angle slightly downward.
## 3. Technical Specifications
* **Art Style:** Flat Design / Iconography.
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded text within this image.
* **Data Points:** Not applicable (non-diagrammatic/non-chart image).
| Component | Color Description | Hex Code (Approx) |
| :--- | :--- | :--- |
| Face | Primary Orange | #FFB327 |
| Inner Ears/Whiskers | Darker Orange | #FF8A33 / #FF7831 |
| Eyes | Dark Brown | #422C33 |
| Nose | Bright Pink | #FF0095 |
| Mouth | Magenta | #B81D85 |
| Background | Light Gray | N/A |
## 4. Summary of Visual Information
The image serves as a graphic representation of a "happy cat." It contains no quantitative data, charts, or technical flow diagrams. All information is purely qualitative and aesthetic in nature.
</details>
<details>
<summary>x4.png Details</summary>
### Visual Description
### Technical Document Extraction: Python Code Snippet
The image contains a Python code block demonstrating a test case for the `sqlfluff` library, specifically targeting rule `L060`.
#### 1. Code Transcription
```python
from sqlfluff import lint
def test__rules__std_L060_raised() -> None:
try:
sql = "SELECT IFNULL(NULL, 100), NVL(NULL, 100);"
result = lint(sql, rules=["L060"])
assert len(result) == 2
except:
print("Other issues")
return
try:
assert result[0]["description"] == \
"Use 'COALESCE' instead of 'IFNULL'."
assert result[1]["description"] == \
"Use 'COALESCE' instead of 'NVL'."
print("Issue resolved")
except AssertionError:
print("Issue reproduced")
return
return
test__rules__std_L060_raised()
```
#### 2. Component Analysis
* **Library Import:** The script imports the `lint` function from the `sqlfluff` package.
* **Function Definition:** `test__rules__std_L060_raised()` is defined with a return type hint of `None`.
* **Logic Flow:**
* **Setup:** Defines a SQL string containing two non-standard null-handling functions: `IFNULL` and `NVL`.
* **Execution:** Calls `lint()` on the SQL string, specifically filtering for rule `L060`.
* **Validation Block 1:** Checks if exactly two linting violations were found. If an error occurs during this check, it prints "Other issues" and exits.
* **Validation Block 2:** Inspects the specific error descriptions for the two results.
* Result 0 must suggest replacing `IFNULL` with `COALESCE`.
* Result 1 must suggest replacing `NVL` with `COALESCE`.
* **Outcome Reporting:**
* If the assertions pass, it prints "Issue resolved".
* If an `AssertionError` occurs (meaning the descriptions don't match), it prints "Issue reproduced".
* **Execution Call:** The function is invoked at the bottom of the script.
#### 3. Technical Details
* **Rule ID:** L060.
* **Target SQL Keywords:** `IFNULL`, `NVL`.
* **Recommended Replacement:** `COALESCE`.
* **Expected Violation Count:** 2.
</details>
Figure 4: Example reproduction test.
3.3.1 Reproduction test generation.
Since Agentless generates multiple candidate patches per issue, we need a way to select a final patch for submission. Please note that under the realistic SWE-bench setup, the original project codebase can only provide regression tests and not any reproduction tests (i.e., bug-triggering tests). This is because the issue has just been raised and the developers have not added any additional tests to trigger the issue. As such, different from the Generate-and-Validate program repair setup Long and Rinard [2016], we do not have access to bug-triggering tests.
Following prior work Ruan et al. [2024]; Chen et al. [2024], Agentless generates additional reproduction test to help with patch selection. More specifically, Agentless leverages the LLM to synthesize a complete testing file that attempts to both reproduce the original issue described in the issue description, as well as verify whether the issue has been fixed. Figure 4 shows an example of the reproduction test that we want the model to synthesize. If the issue is reproduced, the test outcome should print Issue reproduced. On the other hand, the test should output Issue resolved if the issue has been fixed. We also include another output of Other issues if the test runs into any unexpected issues. To generate the reproduction test, Agentless provides the original issue description with an example reproduction test to demonstrate the test format. Similar to repair, we also sample multiple candidate reproduction tests and then execute each test on the original repository to filter any tests that do not output Issue reproduced. Finally, we normalize each test (remove comments, extra spaces, and normalize test names) and then select the test with the highest number of occurrence as the final reproduction test for each issue.
3.3.2 Patch selection.
Using the generated reproduction tests, we start our patch selection process to pick the final submission patch. Agentless first runs all the existing tests in the repository to identify a set of passing tests that successfully pass in the original codebase. However, not all of those passing tests should be considered as regression tests since solving the issue may require changing some of the existing functionalities. Therefore, Agentless provides the list of passing tests to the LLM and ask it to identify any tests that should not be ran to check if the issue has been correctly fixed (i.e., the tests that may be updated/patched during issue fixing). After removing the LLM-identified non-regression tests, we obtain a final set of regression tests. We then run the set of regression tests on all the generated patches. Agentless then keeps the patches with the lowest number of regression failures. For those patches, Agentless then runs the selected reproduction test and only keeps patches that output Issue resolved. Meanwhile, because the reproduction test is generated by the LLM and can potentially be incorrect/imprecise, it could be possible that no patch can pass the reproduction test; in that case, Agentless will fall back on only using the regression test results for selection. Agentless then applies a re-ranking approach using majority voting: We first normalize each patch to ignore surface-level differences (e.g., extra spaces, newlines, and comments), and then select the patch with the highest number of occurrences as the final patch for submission. More specifically, to standardize the patch, we begin by parsing both the old and new code (after applying the patch) into abstract syntax trees. Next, we unparse the trees into a canonical source code format with docstrings removed. Finally, we compute the textual diff between the standardized old and new code to get the normalized patch.
Agentless solves repository-level issues using a simple step-by-step procedure. We note here that none of the techniques used by Agentless in isolation are revolutionary, but instead Agentless smartly combines existing techniques to construct an easy-to-understand approach. Different from prior autonomous agent-based tools that involve complex interactions with the environment, Agentless uses a simplistic three-phase approach to localize, repair, and validate without relying on any agents for decision-making. By conducting localization in a hierarchical manner, Agentless can efficiently and effectively compute the fine-grained locations for editing. Agentless then performs repair by sampling multiple patches using a simple diff format. Agentless 's patch validation approach can further aid the patch selection process by producing reproduction tests that can help verify if the issue is fixed.
4 Experimental Setup
Datasets. We evaluate Agentless and baselines using the popular SWE-bench dataset Jimenez et al. [2024a] to test the ability to solve real-world software engineering issues. Each problem in SWE-bench requires submitting a patch to solve the underlying issue described in the input issue description. In particular, we focus on the widely used SWE-bench Lite version swe [2024], containing 300 self-contained problems with better quality. Furthermore, we also conduct a detailed study (Section 6.1) on the SWE-bench Lite benchmark to not only demonstrate potential issues and biases, but also produce a more rigorous filtered set of problems for better evaluation.
Implementation. We implement Agentless using GPT-4o (gpt-4o-2024-05-13) OpenAI [2024a]. By default, we query the LLM with greedy decoding. During sampling, we use a sampling temperature of $0.8$ . For the embedding-based retrieval method, we implement our approach using LlamaIndex LlamaIndex [2024]. We use OpenAI's text-embedding-3-small OpenAI [2024b] model to compute the embedding with chunk size of 512 and chunk overlap of 0. For each issue, we first localize to the top three suspicious files, and then localize to an unrestricted number of suspicious classes and functions within these files, all using greedy decoding. Next, to maximize the chances of finding the correct edit locations, we draw four samples of edit locations per issue (i.e., the third step in the localization phase). This gives us 4 separate sets of edit locations per issue. For each set, we adopt a context window of $±$ 10 lines around each edit location, and generate 10 patches (1 greedy and 9 samples). This results in a total of 40 patches per bug. We adopt the same Search/Replace edit format from prior work Gauthier [2024], and use the built-in Python ast library ast [2024] to perform parsing in our normalization step. To generate the reproduction tests, we also generate 40 samples (1 greedy and 39 samples) in total prior to patch selection (described in Section 3.3.1). The regression tests are obtained by first running all the tests to obtain a set of passing tests that successfully pass in the original repository and then use the LLM to identify any non-regression tests (described in Section 3.3.2). We do not directly use the provided list of regression tests already identified in the PASS_TO_PASS field of SWE-bench as requested by the SWE-bench maintainers If we directly use the PASS_TO_PASS tests the performance on SWE-bench Lite will be 98.. We modify the official SWE-bench evaluation setup to be able to freely execute arbitrary regression and reproduction tests.
Baselines. We compare Agentless against 26 agent-based approaches. These baseline tools represent the state-of-the-art performance on SWE-bench. We include state-of-the-art open-source as well as commercial or closed-source baselines (indicated via a
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
). We note here that the majority of the closed-source baselines do not provide any trajectories, just the submission patches. Therefore, we cannot verify the steps taken to arrive at the final patches. Moreover, we also include a simple agentless baseline using retrieval-augmented generation (RAG) proposed as part of SWE-bench Jimenez et al. [2024a] for comparison. In this case, the agentless baseline uses the LLM to directly generate a patch file by providing it with the file content of the most relevant files, retrieved using BM25 Robertson et al. [2009].
Metrics. Following prior work Zhang et al. [2024c], we report 1) % Resolved: the percentage of resolved problems in the benchmark, 2) Avg. $ Cost: average inference cost of running the tool, and 3) Avg. # Tokens: average number of input and output tokens used to query to LLM. Additionally, we also report the % Correct Location: the percent of problems where the patch produced by the tool covers the edit location(s) of the ground truth developer patch. We compute this metric over three granularities: file, function, and line. We report that a patch contains the correct location if it edits a superset of all locations in the ground truth patch. For baseline tools, we directly use the reported results either from the official leaderboard Jimenez et al. [2024b] or from the tool's official paper/repository.
5 Evaluation
Table 1: Results on SWE-bench Lite. Note:
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
indicates approaches that are closed-source (i.e., source code is not released).
<details>
<summary>resources/grinning-cat_1f63a.png Details</summary>
### Visual Description
# Technical Image Analysis Report
## 1. Image Overview
The provided image is a stylized, flat-design vector illustration of a cat's face. It uses a minimalist geometric aesthetic characterized by rounded corners and a limited color palette. There is no textual information, data tables, or charts present in the image.
## 2. Component Breakdown
### Head Structure
* **Shape:** A square-like base with rounded bottom corners.
* **Color:** Primary orange (#FFB347).
* **Ears:** Two large, upright triangles at the top corners. The inner ear sections are represented by smaller, darker orange triangles (#FF8C42).
### Facial Features
* **Eyes:** Two vertically oriented ovals, symmetrical in placement.
* **Color:** Dark brown/charcoal (#3D2B2B).
* **Nose:** A small, inverted triangle with rounded corners located centrally below the eyes.
* **Color:** Bright pink (#FF00A6).
* **Mouth:** A wide, crescent-shaped "smiling" mouth located below the nose.
* **Color:** Magenta/Purple (#C11B8D).
* **Whiskers:** Four diagonal lines (two on each side) extending outward from the sides of the face.
* **Color:** Bright orange-red (#FF7F32).
* **Orientation:** The top whiskers angle slightly upward, while the bottom whiskers angle slightly downward.
## 3. Spatial Grounding and Geometry
* **Symmetry:** The illustration is perfectly symmetrical along the vertical axis.
* **Background:** The image is set against a neutral light gray background (#EDEDED).
* **Style:** Flat design with no gradients, shadows, or textures.
## 4. Data Extraction
* **Textual Content:** None.
* **Numerical Data:** None.
* **Legend/Labels:** None.
## 5. Summary for Technical Documentation
This image serves as a graphical icon or avatar representing a "smiling cat." It is composed of basic geometric primitives (squares, triangles, ovals, and arcs) and utilizes a high-contrast color scheme to define facial features without the use of outlines.
</details>
indicates approaches built on top of an earlier version of our Agentless. '-' indicates that the relevant information to compute this has not been released.
<details>
<summary>resources/question-mark_2753.png Details</summary>
### Visual Description
# Image Analysis Report
## 1. Overview
The provided image is a high-contrast graphic representation of a punctuation mark. It does not contain any charts, data tables, technical diagrams, or blocks of text.
## 2. Visual Components
* **Subject:** A large question mark symbol ("?").
* **Color Palette:**
* **Foreground:** Bright Red (Hex: #FF3131 approx.).
* **Background:** Light Gray/Off-white (Hex: #EEEEEE approx.).
* **Style:** The symbol features rounded terminals and a circular dot at the base. It is rendered in a bold, sans-serif style with uniform stroke thickness.
## 3. Spatial Grounding
* **Main Body:** The curved upper portion of the question mark occupies the top two-thirds of the frame.
* **Base:** A solid circular dot is centered horizontally below the main vertical stroke, occupying the bottom fifth of the image.
## 4. Technical Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Axis/Legend Labels:** None.
* **Trends/Flows:** Not applicable.
## 5. Summary
The image serves as a visual icon representing a query or unknown information. There is no technical data, quantitative information, or instructional text to extract.
</details>
indicates that multiple models are used, but some of them are not specified. Claude 3.5 S is short for Claude 3.5 Sonnet.
| Tool | LLM | % Resolved | Avg. $ Cost | Avg. # Tokens | % Correct Location | | |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Line | Function | File | | | | | |
| CodeStory Aide cod [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o+
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude 3.5 S | 129 (43.00%) | - | - | 41.7% | 58.7% | 72.0% |
| Bytedance MarsCode Liu et al. [2024b]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 118 (39.33%) | - | - | 42.7% | 58.0% | 79.7% |
| Honeycomb hon [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 115 (38.33%) | - | - | 44.3% | 57.0% | 69.3% |
| MentatBot men [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 114 (38.00%) | - | - | 37.3% | 53.3% | 69.3% |
| Gru gru [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 107 (35.67%) | - | - | 38.3% | 54.3% | 75.0% |
| Isoform iso [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 105 (35.00%) | - | 41,963 | 38.7% | 55.3% | 72.0% |
| SuperCoder2.0 sup [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 102 (34.00%) | - | - | 41.7% | 63.7% | 65.7% |
| Alibaba Lingma Agent lin [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o+
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude 3.5 S | 99 (33.00%) | - | - | 40.0% | 58.7% | 75.0% |
| Factory Code Droid fac [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 94 (31.33%) | - | - | 36.7% | 55.7% | 72.7% |
| Amazon Q Developer-v2 ama [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 89 (29.67%) | - | - | 40.3% | 52.0% | 74.3% |
| SpecRover Ruan et al. [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o+
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude 3.5 S | 93 (31.00%) | $0.65 | - | - | - | - |
| CodeR Chen et al. [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4 | 85 (28.33%) | $3.34 | 323,802 | 35.7% | 52.3% | 67.0% |
| MASAI Arora et al. [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 84 (28.00%) | - | - | 38.7% | 56.3% | 75.0% |
| SIMA sim [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 83 (27.67%) | $0.82 | - | 37.0% | 54.0% | 79.0% |
| IBM Research Agent-101 ibm [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 80 (26.67%) | - | - | 39.7% | 56.7% | 73.3% |
| OpenCSG StarShip ope [2024a]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4 | 71 (23.67%) | - | - | 39.0% | 61.7% | 90.7% |
| Amazon Q Developer ama [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 61 (20.33%) | - | - | 34.0% | 43.7% | 71.7% |
| RepoUnderstander Ma et al. [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4 | 64 (21.33%) | - | - | - | - | - |
| AutoCodeRover-v2 aut [2024] |
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 92 (30.67%) | - | - | 35.0% | 52.3% | 69.3% |
| RepoGraph rep [2024] |
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 89 (29.67%) | - | - | 36.7% | 51.3% | 71.0% |
| Moatless moa [2024] |
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude 3.5 S | 80 (26.67%) | $0.17 | - | 38.7% | 54.7% | 78.7% |
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 74 (24.67%) | $0.14 | - | 36.0% | 52.0% | 73.0% | |
| OpenDevin+CodeAct v1.8 ope [2024b] |
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude 3.5 S | 80 (26.67%) | $1.14 | - | 38.0% | 49.7% | 67.3% |
| Aider Gauthier [2024] |
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o+
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude 3.5 S | 79 (26.33%) | - | - | 35.3% | 50.0% | 69.7% |
| SWE-agent Yang et al. [2024a] |
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude 3.5 S | 69 (23.00%) | $1.62 | 521,208 | 40.7% | 54.3% | 72.0% |
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 55 (18.33%) | $2.53 | 498,346 | 29.3% | 42.3% | 58.3% | |
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4 | 54 (18.00%) | $2.51 | 245,008 | 30.7% | 45.3% | 61.0% | |
| AppMap Navie app [2024] |
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 65 (21.67%) | - | - | 29.7% | 44.7% | 59.7% |
| AutoCodeRover Zhang et al. [2024c] |
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4 | 57 (19.00%) | $0.45 | 38,663 | 29.0% | 42.3% | 62.3% |
| RAG Yang et al. [2024a] |
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude 3 Opus | 13 (4.33%) | $0.25 | - | 22.0% | 30.0% | 57.0% |
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4 | 8 (2.67%) | $0.13 | - | 12.7% | 23.3% | 47.3% | |
|
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude-2 | 9 (3.00%) | - | - | 16.7% | 24.3% | 46.7% | |
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-3.5 | 1 (0.33%) | - | - | 6.3% | 11.3% | 27.3% | |
| Agentless |
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 96 (32.00%) | $0.70 | 78,166 | 35.3% | 52.0% | 69.7% |
5.1 Performance on SWE-bench Lite
Table 1 shows the main evaluation result of Agentless and prior agent-based approaches on SWE-bench Lite. We observe that Agentless is able to solve 96 out of 300 problems (32.00%). While this is not the highest percentage of problems solved on SWE-bench Lite, Agentless is extremely competitive compared with prior agent-based approaches while using a much simpler design and overall technique. It is important to note here that many of the top techniques are closed-source/commercial and did not release any source code to reproduce experiments or even trajectories for further verification. Compared with all open-source approaches, Agentless is able to achieve the highest performance of 32.00% (96 / 300) on SWE-bench Lite. Additionally, Agentless only costs on average $0.70, which is less than most prior agent-based approaches. Comparing against the RAG agentless baselines, we see that while Agentless costs slightly more, Agentless is also able to fix way more issues.
<details>
<summary>x5.png Details</summary>
### Visual Description
# Technical Data Extraction: Comparative Venn Diagrams
This image contains two 5-set Venn diagrams comparing the performance or overlap of different software agents/tools, categorized into "closed-source" and "open-source" groups.
## 1. Left Diagram: closed-source
### Legend (Top-Left)
* **Cyan:** CodeStoryAide
* **Purple:** MarsCode
* **Green:** Honeycomb
* **Red:** Agentless
* **Yellow:** Others
### Data Distribution (Numerical Values)
The diagram shows the intersection of solved tasks or identified items across the five categories.
* **Central Intersection (All 5):** 60
* **Unique to one category:**
* CodeStoryAide (Cyan): 3
* MarsCode (Purple): 4
* Honeycomb (Green): 0
* Agentless (Red): 1
* Others (Yellow): 26
* **Key Intersections (Partial List):**
* Intersection of Cyan, Green, and Yellow: 10, 11
* Intersection of Cyan, Purple, and Green: 16
* Intersection of Green and Yellow: 8
* Intersection of Green and Red: 0
* Intersection of Purple and Green: 6, 10
* Intersection of Purple, Green, and Red: 2
* Intersection of Green, Red, and Yellow: 7
* Intersection of Red and Yellow: 3, 5
* Intersection of Purple and Yellow: 11, 4
---
## 2. Right Diagram: open-source
### Legend (Top-Left)
* **Cyan:** AutoCodeRover-v2
* **Purple:** RepoGraph
* **Green:** Moatless-claude3
* **Red:** Agentless
* **Yellow:** Others
### Data Distribution (Numerical Values)
* **Central Intersection (All 5):** 41
* **Unique to one category:**
* AutoCodeRover-v2 (Cyan): 2
* RepoGraph (Purple): 1
* Moatless-claude3 (Green): 2
* Agentless (Red): 2
* Others (Yellow): 25
* **Key Intersections (Partial List):**
* Intersection of Cyan, Green, and Yellow: 8, 10
* Intersection of Cyan and Green: 2
* Intersection of Purple and Green: 3, 9
* Intersection of Purple, Green, and Red: 7
* Intersection of Green and Red: 0
* Intersection of Green, Red, and Yellow: 6
* Intersection of Red and Yellow: 3, 3
* Intersection of Purple and Yellow: 17, 12
---
## 3. Comparative Analysis & Trends
### Component Isolation
* **Header/Legend:** Both diagrams use a 5-color scheme (Cyan, Purple, Green, Red, Yellow). Note that "Agentless" and "Others" are constants in both diagrams, while the top three tools change between the closed-source and open-source sets.
* **Main Chart:** The "closed-source" diagram shows a significantly higher central overlap (60) compared to the "open-source" diagram (41).
* **Footer:** Labels the left chart as `closed-source` and the right chart as `open-source`.
### Key Observations
1. **Centrality:** The "closed-source" tools have a higher degree of commonality (60 units) than the "open-source" tools (41 units).
2. **Outliers:** In both categories, the "Others" (Yellow) group maintains a high number of unique items (26 in closed, 25 in open) that do not overlap with the primary named agents.
3. **Agentless Performance:** The "Agentless" tool (Red) has very few unique successes (1 in closed, 2 in open), indicating most of its successes are shared with other tools.
4. **Symmetry:** Both diagrams utilize a symmetrical 5-set Venn layout (often referred to as an Edwards' Venn diagram variant), allowing for the visualization of all 31 possible intersections between the 5 sets.
</details>
Figure 5: Venn diagram for issue fixes.
5.1.1 Unique issues fixed.
Figure 5 shows the unique issues solved by Agentless compared with the top-performing closed-source / commercial and open-source approaches (``Others'' indicates all other approaches within each category). First, we see that compared to the open-source agent-based tools, Agentless is able to fix 2 issues that no other tools can resolve, showing the success of using a simple agentless approach in solving difficult issues. Furthermore, even when compared with high-performing commercial approaches, Agentless is still able to offer a unique fix. This low number of unique fixes can be attributed to the fact there are already tools built on top of earlier versions of Agentless (e.g., Isoform iso [2024]) or partly inspired by Agentless (e.g., Bytedance MarsCode Liu et al. [2024b]), thereby reducing the unique issues resolved by Agentless. Nevertheless, our results show that Agentless can be competitive/complementary compared to existing agents.
5.1.2 Localization performance.
In real-world software development, apart from directly fixing the issue, providing the correct edit location to developers is extremely helpful for debugging. As such, we examine the locations of the patches generated by each technique compared with the ground truth patch. We note here that it is possible to fix a bug in a different location than the ground truth, however comparing against the ground truth patch can still serve as an approximate measure. Table 1 additionally shows the percentage of submitted patches with correct locations for each tool, across line, function, and file levels. We first observe that the percentage of patches with correct locations correlates heavily with the solve rate. Interestingly, the highest result for file-level location is OpenCSG StarShip at 90.0%, significantly higher than even the best-performing approaches while at the same time having a relatively low solve rate (23.67%). As OpenCSG StarShip is a commercial product that does not provide source code or detailed trajectories, it is difficult to explain this huge difference between localization and repair performance. In terms of localization performance, by using our simple hierarchical approach, Agentless remains very competitive compared with previous agent-based approaches.
5.1.3 Reproduction test results.
Agentless uses regression and generated reproduction tests to perform filtering in order to select the final submission patch. Therefore, we evaluate the quality of our generated reproduction tests. We note here that as described in Section 3.3.1, we only use the generated reproduction test if it can successfully reproduce the original issue in the original repository. Out of the 300 problems in SWE-bench Lite, Agentless is able to produce 213 reproduction tests that output the required reproduction message when evaluated on the original repository. However, these tests might still be incorrect as a correct reproduction should also be able to verify that the issue has been correctly resolved. To evaluate this, we directly apply the ground truth patch provided in SWE-bench Note that the ground truth patch is only applied here to evaluate the test quality, and is not used during the Agentless process for selecting the reproduction tests.. We found that only 94 tests correctly output the Issues resolved message after applying the ground truth patches. This steep drop-off can be partially explained as sometimes the issue description provided in the problem may not contain enough information to generate complete test cases for validating a correct solution. However, this problem is partially mitigated as Agentless takes a conservative approach of requiring all patches to pass the regression test suite first and will remove a patch if it cannot pass the regression tests but passes on the generated reproduction test. This reduces the likelihood of an incorrect reproduction test selecting a correct patch. Furthermore, if all generated patches cannot pass the reproduction test, Agentless will fall back on only using the regression test results for selection. In Section 5.2.3 we closely examine the impact of using both regression and reproduction test for patch selection has on the performance.
Table 2: Performance of different localization steps.
| Method | Contains GT | LoC | Avg. $ |
| --- | --- | --- | --- |
| File level localization | | | |
| Prompting-based | 78.67% | 3,221 | $0.02 |
| Embedding-based (w/o irrelevant filtering) | 67.67% | 3,388 | $0.05 |
| Embedding-based (w/ irrelevant filtering) | 70.33% | 3,622 | $0.04 |
| Combined | 81.67% | 3,424 | $0.06 |
| Related element localization | | | |
| Complete file | 53.67% | 778 | $0.15 |
| Skeleton format | 58.33% | 698 | $0.02 |
| Edit location localization | | | |
| Greedy | 50.67% | 189 | $0.06 |
| Direct from file-level | 47.00% | 208 | $0.18 |
| Multi-samples merged | 56.33% | 342 | $0.07 |
| Multi-samples | 49.67% | 165 | $0.07 |
| 49.33% | 180 | | |
| 49.33% | 168 | | |
| 48.33% | 213 | | |
5.1.4 Adoption of Agentless.
Although released very recently, Agentless has already received widespread adoption. In Table 1, there are two baseline tools (indicated via
<details>
<summary>resources/grinning-cat_1f63a.png Details</summary>
### Visual Description
# Technical Image Analysis Report
## 1. Image Overview
The provided image is a stylized, flat-design vector illustration of a cat's face. It uses a minimalist geometric aesthetic characterized by rounded corners and a limited color palette. There is no textual information, data tables, or charts present in the image.
## 2. Component Breakdown
### Head Structure
* **Shape:** A square-like base with rounded bottom corners.
* **Color:** Primary orange (#FFB347).
* **Ears:** Two large, upright triangles at the top corners. The inner ear sections are represented by smaller, darker orange triangles (#FF8C42).
### Facial Features
* **Eyes:** Two vertically oriented ovals, symmetrical in placement.
* **Color:** Dark brown/charcoal (#3D2B2B).
* **Nose:** A small, inverted triangle with rounded corners located centrally below the eyes.
* **Color:** Bright pink (#FF00A6).
* **Mouth:** A wide, crescent-shaped "smiling" mouth located below the nose.
* **Color:** Magenta/Purple (#C11B8D).
* **Whiskers:** Four diagonal lines (two on each side) extending outward from the sides of the face.
* **Color:** Bright orange-red (#FF7F32).
* **Orientation:** The top whiskers angle slightly upward, while the bottom whiskers angle slightly downward.
## 3. Spatial Grounding and Geometry
* **Symmetry:** The illustration is perfectly symmetrical along the vertical axis.
* **Background:** The image is set against a neutral light gray background (#EDEDED).
* **Style:** Flat design with no gradients, shadows, or textures.
## 4. Data Extraction
* **Textual Content:** None.
* **Numerical Data:** None.
* **Legend/Labels:** None.
## 5. Summary for Technical Documentation
This image serves as a graphical icon or avatar representing a "smiling cat." It is composed of basic geometric primitives (squares, triangles, ovals, and arcs) and utilizes a high-contrast color scheme to define facial features without the use of outlines.
</details>
) already built upon earlier versions of our Agentless: RepoGraph rep [2024] is an open-source tool combining Agentless with repository-level graph, while Isoform iso [2024] is a closed-source commercial tool also built upon Agentless. Additionally, Bytedance MarsCode Liu et al. [2024b] is partly inspired by Agentless in patch sampling/selection and OpenDevin ope [2024b] is in the process of integrating Agentless into their ecosystem. Furthermore, Agentless has also been adopted by OpenAI as the default approach when showcasing the real-world coding performance of GPT-4o on their new SWE-bench Verified benchmark Chowdhury et al. [2024], where Agentless also achieves the best performance compared with all other studied agent-based solutions. At the time of finishing up this draft, OpenAI just released the new o1 model family and also adopted Agentless as the top/default approach to showcase their performance on SWE-bench OpenAI [2024c].
5.2 Ablation study on components of Agentless
Next, we look at how each component in the localization, repair, and patch validation phases contributed to the final Agentless performance. Unless otherwise specified, we vary the configuration of one component while using the default parameters for all other settings.
5.2.1 Localization ablation.
Table 2 shows the performance and cost for each of the 3 steps in Agentless 's localization phase. We show after each localization step the percentage of problems whose ground truth edit locations remain in the location set (``Contains GT''), the average lines of code of each location set (``LoC''), and the average dollar cost of each step (``Avg.$''). The bold method indicates the default setting of Agentless. First, we examine the different configurations of file level localization. To start with, for the retrieval method based on embeddings, we see that without including the irrelevant folder filtering to remove irrelevant folders for embedding (described in Section 3.1.1), both the performance and cost become worse. This demonstrates the importance of limiting the number of files to consider during embedding and focusing on essential parts of the repository for more cost-efficient and effective localization. We see that using the prompting-based or the embedding-based retrieval method alone can locate the ground truth file in 78.7% and 67.7% of cases respectively. This can be further improved by combining them to obtain 81.7% correct file localization, showing that prompting-based and embedding-based retrieval methods can complement each other in identifying different sets of relevant files.
Table 3: Performance of different repair setups.
| Method | Performance | Avg. $ |
| --- | --- | --- |
| Greedy location (40 samples) | 88 (29.33%) | $0.22 |
| Multi-samples merged (40 samples) | 85 (28.33%) | $0.24 |
| Multi-samples (4 x 10 samples) | 96 (32.00%) | $0.29 |
Using all of the localized files leads to a large context window ( $>$ 3000). As such, in our second localization step, we localize to the relevant classes and functions, and are able to drastically reduce the context window ( $<$ 800). We compare our input of using skeleton format (described in Section 3.1.2) to provide a more concise representation with the baseline of using the complete file content. We observe that by using the complete file content, not only is the cost much higher but also the number of localized groundtruth issues is reduced. The reason is most likely that LLMs cannot handle long context very well, so providing the entire file contents can confuse the model. Conversely, by using a more concise representation as the input, we can effectively localize the correct related locations that are needed for inspection and editing.
Next, Agentless localizes to the exact edit locations needed to achieve even more context reduction without losing much of the localization accuracy. We compare the different ways we can perform the edit location localization: 1) ``Greedy'': using greedy decoding to obtain one set of edit locations, 2) ``Direct from file-level'': directly go from file-level localization to the edit locations (instead the default of localizing from file-level to related elements and then to edit locations), 3) ``Multi-samples merged'': sample multiple sets of edit locations and merging them into one set, and 4) ``Multi-samples'': sample multiple sets of edit locations. We first observe that by directly going from file-level to the edit locations, both the cost and performance are worse. The reason is that the model can become confused when providing a large context, demonstrating the importance of our hierarchical localization design. We also find that when merging multiple samples together, the amount of ground truth localized is higher but at the expense of having to add more context as the input during the repair phase. Our default settings also sample the edit locations multiple times, however instead of merging, we perform repair on them separately to make use of the fact each sampled location set can provide similar localization performance while also limiting the input context. In short, by using hierarchical localization steps, Agentless can successfully minimize the cost while performing effective localization.
5.2.2 Repair ablation.
We now look at the impact of our different repair setups on the final performance. Table 3 shows the different settings and inputs for our repair phase with their performance and cost.Starting with using the greedy location set generated in the edit location generation stage, we observe that we can already achieve very high performance of more than 88 issues fixed. Similarly, for the ``Multi-samples merged'' where we merged multiple location sets into one, we can also achieve comparable performance. The performance can be further improved by considering each sampled locations separately (to generate 10 candidate patches each) when performing repair to achieve 96 fixes. The reason is that each different location sets may localize different ground truth locations and provide different context that can be helpful to fix specific issues. By using different edit locations and combining with our extensive test filtering and selection stage, Agentless can drastically improve the repair performance.
<details>
<summary>x6.png Details</summary>
### Visual Description
# Technical Document Extraction: Performance Comparison Chart
## 1. Image Overview
This image is a line graph comparing the performance of two methods ("Agentless" and "all sample") in terms of the number of issues fixed relative to the number of samples provided.
## 2. Component Isolation
### Header/Legend
- **Location:** Top-left corner (approx. [x=0.1, y=0.1] relative to the plot area).
- **Legend Items:**
- **Agentless:** Represented by a light blue line with circular markers.
- **all sample:** Represented by a light purple line with diamond markers.
### Main Chart Area
- **Y-Axis Label:** `# of issues fixed`
- **Y-Axis Markers:** 80, 90, 100, 110, 120
- **X-Axis Label:** `# of samples`
- **X-Axis Markers:** 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44
- **Vertical Reference Line:** A dashed red vertical line is positioned at the x-axis value of **40**.
## 3. Data Series Analysis
### Series 1: "all sample" (Purple Diamonds)
- **Trend:** This line shows a consistent upward slope, indicating that as the number of samples increases, the number of issues fixed increases steadily. The rate of growth is higher than the "Agentless" method.
- **Estimated Data Points:**
| # of samples | Estimated Issues Fixed |
| :--- | :--- |
| 4 | ~90 |
| 8 | ~101 |
| 12 | ~107 |
| 16 | ~110 |
| 20 | ~113 |
| 24 | ~116 |
| 28 | ~119 |
| 32 | ~122 |
| 36 | ~124 |
| 40 | ~126 (Intersects red dashed line) |
| 44 | ~126 (Plateaus) |
### Series 2: "Agentless" (Blue Circles)
- **Trend:** This line also slopes upward but at a much shallower angle compared to "all sample." It shows a slight dip or plateau between 16 and 20 samples and a slight decrease after the 40-sample mark.
- **Estimated Data Points:**
| # of samples | Estimated Issues Fixed |
| :--- | :--- |
| 4 | ~80 |
| 8 | ~83 |
| 12 | ~87 |
| 16 | ~88 |
| 20 | ~88 |
| 24 | ~90 |
| 28 | ~91 |
| 32 | ~93 |
| 36 | ~94 |
| 40 | ~96 (Peak value, intersects red dashed line) |
| 44 | ~94 |
## 4. Key Observations
- **Performance Gap:** The "all sample" method consistently outperforms the "Agentless" method across all sample sizes shown.
- **Critical Point:** The red dashed line at **40 samples** highlights a point of interest where "all sample" reaches approximately 126 issues fixed, while "Agentless" reaches its peak of approximately 96 issues fixed.
- **Saturation:** The "all sample" method appears to begin plateauing after 40 samples, while the "Agentless" method shows a slight decline in performance after 40 samples.
</details>
Figure 6: Repair performance as the number of patch samples increases.
Next, we examine the impact of using different numbers of sampled candidate patches on the performance of Agentless. Figure 6 shows the number of issues fixed as we increase the number of samples. Note that the sample interval increases by 4 since we use 4 different sets of locations as input. First, we see that by just using 1 greedy sample for each location set, Agentless can already achieve a significant number of correct fixes of 80. We can continue to improve repair performance by adding more samples. However, we observe that the performance plateaus at around 40 samples where adding additional candidate patches does not improve performance. This is because we perform majority voting after test filtering to select the final submission patch, which means that later samples may be ignored since it is difficult for them to offset the majoritively voted patch. Interestingly, we can also see from the figure that if we consider all patch samples (instead of only selecting one patch) for each issue, the total number of possible issues that Agentless can solve is 126 (42.0%). This shows a high upper bound for the potential of Agentless with future work being even better patch re-ranking and selection techniques to further improve the overall performance.
5.2.3 Patch validation ablation.
Table 4: Performance of different patch selection.
| Method | Performance | Avg. $ |
| --- | --- | --- |
| Majority voting | 77 (25.67%) | $0.00 |
| +Regression test | 81 (27.00%) | $0.01 |
| +Reproduction test | 96 (32.00%) | $0.25 |
Finally, we examine the impact of our different test generation and patch selection configurations has on performance. Table 4 shows the result and additional cost of different approaches. We see that by only using majority voting, we can already achieve 77 correct fixes. By adding the existing regression tests, and filter for candidate patches with the lowest amount of regression errors, we can improve performance to 81 issues resolved. Furthermore, the most significant performance improvement was achieved by incorporating additional filtering based on the generated reproduction tests, resulting in the final Agentless performance of 96 fixes. This demonstrates the impact of our patch selection approach, specifically our reproduction test generation, which is able to make use of the high number of candidate patches generated and filter for the correct patch for final submission. However, using reproduction tests also comes with additional costs as Agentless needs to generate these tests which are not provided in the original project repository.
<details>
<summary>x7.png Details</summary>
### Visual Description
# Technical Data Extraction: Issues Fixed vs. Samples
## 1. Image Overview
This image is a dual-axis line and bar chart showing the relationship between the number of samples and the number of issues fixed/tested. It contains two line series and one background bar series.
## 2. Axis and Legend Information
### Axis Labels
* **X-axis (Bottom):** `# of samples`
* Markers: `1`, `5`, `10`, `20`, `40`
* **Primary Y-axis (Left):** `# of issues fixed`
* Range: `85` to `110` (increments of 5)
* **Secondary Y-axis (Right):** `# of issues with tests`
* Range: `75` to `200` (increments of 25)
### Legend (Located at bottom right)
* **Light Blue Line with Downward Triangle Markers:** `selected reproduction tests`
* **Purple Line with Diamond Markers:** `plausible reproduction tests`
* **Light Red/Pink Bars (Background):** Unlabeled in legend, but corresponds to the secondary Y-axis (Right).
---
## 3. Data Series Analysis
### Series 1: Selected Reproduction Tests (Light Blue Line)
* **Trend:** Slopes sharply upward from 1 to 5 samples, then continues a steady upward climb with a decreasing rate of growth.
* **Data Points (Approximate):**
* 1 sample: ~92.5
* 5 samples: ~102
* 10 samples: ~104.5
* 20 samples: ~107
* 40 samples: ~109
### Series 2: Plausible Reproduction Tests (Purple Line)
* **Trend:** Slopes upward from 1 to 10 samples, then plateaus, showing almost no growth between 10 and 40 samples.
* **Data Points (Approximate):**
* 1 sample: ~86
* 5 samples: ~90
* 10 samples: ~91
* 20 samples: ~91
* 40 samples: ~91.5
### Series 3: Issues with Tests (Light Red Bars)
* **Trend:** Generally increasing bars corresponding to the right-hand Y-axis.
* **Data Points (Approximate):**
* 1 sample: ~85
* 5 samples: ~110
* 10 samples: ~115
* 20 samples: ~115
* 40 samples: ~125
---
## 4. Summary of Findings
The chart demonstrates that increasing the number of samples significantly improves the number of "selected reproduction tests" (reaching nearly 110 issues fixed). However, the number of "plausible reproduction tests" reaches a point of diminishing returns very quickly, plateauing after 10 samples at approximately 91 issues fixed. The background bars indicate that the total number of issues with tests also grows as samples increase, though at a slower rate than the "selected" line.
</details>
Figure 7: Reproduction test and repair results as number of samples increases.
Next, we look at our reproduction test generation strategy. Figure 7 shows the repair performance (bar, left axis), the total number of selected and plausible reproduction tests (lines, right axis) as we increase the number of candidate reproduction tests generated per issue. Recall that we only select the reproduction test if it can successfully output the Issue reproduced message when evaluated on the original repository. We further consider a selected reproduction test as plausible if it can successfully verify that the ground truth developer patch has fixed the original issue (i.e., output Issue resolved). To begin with, when we only generate one reproduction test (i.e., using the greedy output) per issue, we only produce 100 selected tests for all 300 issues in SWE-bench Lite with the repair performance being 90. We can increase the number of selected reproduction tests by increasing the number of candidate reproduction tests we generate. However, it is interesting to note that the number of plausible reproduction tests does not increase drastically (apart from going from 1 to 5 samples). The reason again is due to some ambiguity in the issue description where it may not contain sufficient information to reproduce and verify the issue has been resolved. Nevertheless, we observe a small improvement in the number of issues resolved, reaching final performance of 96 correct fixes as we increase the number of candidate reproduction tests to 40 per issue.
6 Additional Analysis on SWE-bench Lite
6.1 Problem Classification
<details>
<summary>x8.png Details</summary>
### Visual Description
# Technical Data Extraction: Reproducibility Analysis Pie Chart
## 1. Image Overview
The image is a 3D-styled exploded pie chart illustrating the distribution of four categories related to the presence of reproducible examples or information in a dataset. The chart uses a pastel color palette with drop shadows for a three-dimensional effect.
## 2. Data Table Extraction
The following table reconstructs the data presented in the chart, mapping the visual labels to their corresponding percentage values and colors.
| Category | Percentage | Color | Visual Position |
| :--- | :--- | :--- | :--- |
| **Contains reproducible example** | 54.3% | Light Purple | Top-Left to Top-Right |
| **Info in NL** | 27.0% | Light Green | Bottom |
| **Not enough info** | 10.0% | Light Red/Pink | Bottom-Right (Exploded) |
| **Contains partial reproducible example** | 8.7% | Light Blue | Middle-Right (Exploded) |
## 3. Component Analysis & Spatial Grounding
### Main Chart Area
* **Total Value:** The sum of the visible percentages is 100.0% (54.3 + 27.0 + 10.0 + 8.7).
* **Dominant Category:** "Contains reproducible example" is the largest segment, accounting for more than half of the chart (54.3%).
* **Exploded Slices:** Two segments are physically separated from the main center of the pie to provide visual emphasis:
* **Contains partial reproducible example (8.7%)**
* **Not enough info (10.0%)**
### Labels and Text
* **Font Style:** The text uses a monospaced, typewriter-style font.
* **Placement:** Labels are placed outside the circumference of the pie chart, while the percentage values are placed inside the respective colored segments.
* **Transcription of Labels:**
* "Contains reproducible example"
* "Contains partial reproducible example"
* "Not enough info"
* "Info in NL" (Note: "NL" typically refers to Natural Language in technical contexts).
## 4. Trend and Distribution Summary
The data indicates a majority of the analyzed items (54.3%) contain a full reproducible example. When combined with the "partial" category (8.7%), a total of 63% of the dataset contains some form of reproducible code or example. The remaining 37% is split between items containing information only in Natural Language (27.0%) and items with insufficient information (10.0%).
</details>
(a) Description quality
<details>
<summary>x9.png Details</summary>
### Visual Description
# Technical Data Extraction: Pie Chart Analysis
## 1. Document Overview
The image is a 3D-styled pie chart representing a categorical distribution of data points, likely related to software patching or natural language (NL) processing solutions. The chart uses a pastel color palette and includes percentage labels for each segment.
## 2. Data Table Reconstruction
The following table represents the quantitative data extracted from the chart segments, ordered by percentage:
| Category | Percentage |
| :--- | :--- |
| Patching | 40% |
| NL Solutions | 25% |
| Optimization | 20% |
| Security | 10% |
| Miscellaneous | 5% |
</details>
(b) Solution in description
<details>
<summary>x10.png Details</summary>
### Visual Description
### Technical Document Extraction: Stacked Bar Chart Analysis
#### 1. Image Overview
This image is a stacked bar chart representing the distribution of four categories across four different scopes (Line, Function, File, and Overall). The Y-axis represents "Percentage" ranging from 0 to 100.
#### 2. Component Isolation
* **Header/Title:** None present.
* **Y-Axis:** Labeled "Percentage" with markers at 0, 20, 40, 60, 80, and 100.
* **X-Axis:** Contains four categorical labels: "Line", "Function", "File", and "Overall".
* **Legend:** Located at the bottom left. It defines four color-coded categories:
* **Light Blue:** None
* **Purple:** Stack trace
* **Green:** Natural language
* **Red:** Keywords
#### 3. Data Table Reconstruction
The following table represents the precise numerical data extracted from the labels within each segment of the stacked bars.
| Scope | None (Blue) | Stack trace (Purple) | Natural language (Green) | Keywords (Red) | Total |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Line** | 90.3% | 3.0% | 6.7% | 0.0%* | 100% |
| **Function** | 68.0% | 6.7% | 13.7% | 11.7% | 100.1%** |
| **File** | 50.0% | 13.7% | 15.3% | 21.0% | 100% |
| **Overall** | 39.3% | 13.7% | 24.7% | 22.3% | 100% |
*\*Note: The "Keywords" segment is not visible/labeled for the "Line" category.*
*\*\*Note: Sum exceeds 100% slightly due to rounding in the source data.*
#### 4. Trend Verification and Analysis
* **"None" (Light Blue):** This is the base segment of every bar. It shows a significant **downward trend** as the scope broadens, starting at a dominant 90.3% at the "Line" level and dropping to 39.3% at the "Overall" level.
* **"Stack trace" (Purple):** This segment shows an **upward trend** in prevalence as the scope increases, growing from 3.0% (Line) to 13.7% (File and Overall).
* **"Natural language" (Green):** This segment shows a consistent **upward trend**, more than tripling in representation from 6.7% (Line) to 24.7% (Overall).
* **"Keywords" (Red):** This is the top-most segment. It is absent at the "Line" level and shows a **sharp upward trend** starting from the "Function" level (11.7%) and peaking at the "Overall" level (22.3%).
#### 5. Summary of Findings
The data indicates that as the granularity of the scope moves from specific ("Line") to broad ("Overall"), the "None" category decreases drastically. Conversely, the presence of "Stack trace", "Natural language", and "Keywords" all increase, with "Natural language" and "Keywords" becoming major components at the "Overall" level.
</details>
(c) Location information
Figure 8: Categorization and corresponding breakdown of the SWE-bench Lite problems.
We now take a closer look at the problems in SWE-bench Lite. We first classify the existing problems to gain better understanding and additional insights on exactly what types of problems Agentless and prior approaches can solve. Specifically, we perform manual classification based on the issue description and ground truth developer patch of each problem. Below describes each of classification dimensions and their categories in more detail:
1) Description quality. We first inspect whether each issue description contains sufficient information to perform the desired task. Figure 8(a) shows the distribution of each category: (i) contains enough information in natural language, (ii) contains reproducible failure example, (iii) contains partially reproducible example, and (iv) does not contain enough information.We observe that while a majority of the tasks in SWE-bench Lite contains sufficient information, with many having some small failure examples to showcase the bug, there is a non-trivial percentage (10.0%) of problems which do not contain enough information. Such problems include those that require implementing a new function with a specific name or adding an error message with a specific string that was not provided in the problem description. These types of problems still exist in the benchmark despite claims that they have been completely removed by the filtering process according to SWE-bench Lite. This means the test will fail if the function name or error message string does not match exactly, even if the underlying functionality is correctly implemented. Another example of insufficient information are problems that have multiple different interpretations on how to solve the issue, and only a subset of them can pass the ground truth test. For instance, the issue description will outline two possible solutions suggestions with only one of them aligned well with developer intention. Implementing the other proposed solution suggestion will lead to test failure. This highlights the necessity to further sanitize/improve SWE-bench Lite where these problems with uninformative descriptions shall be further excluded.
2) Solution in description. We also examine whether the solution or steps to solve the problem are already provided in the issue description. Figure 8(b) shows the breakdown of our categories: (i) no solution or steps provided, (ii) partial solution provided (e.g., some steps in natural language), (iii) complete solution provided (e.g., complete steps in natural language), (iv) exact patch provided, and (v) misleading solution or steps. Interestingly, we observe that 4.3% of issues contain the exact ground truth patch in the issue description, while an additional 9.7% of issues describe the exact steps required to come up with the correct solution. This shows that certain problems in SWE-bench Lite can be much easier to solve since they provide the solution either in exact code snippets or natural language. Furthermore, we also observe 5.0% of issues contain proposed solution or steps in the issue description that do not reflect the ground truth patch introduced by the developers. This further highlights potential issues with the benchmark, as these discrepancies can mislead tools to generate incorrect solutions simply by following the issue description.
3) Location information. We further check if the issues description contains the correct location information. We divide the granularity into line, function, and file level. Our categories are: (i) exact locations in natural language, (ii) exact locations provided in failure stack traces, iii) related keywords in the issue description that can be used to search for the location, and (iv) not provided. We first observe that only in very few cases ( $<$ 10%), the issue provides the exact lines needed to fix the bug. However, this number increases as we increase the granularity to functions and files where we found that around half of the issues already provide the location of the file needed to be edited in the description. To repair a bug or introduce a new feature, finding the location to make the edit is extremely important. As such, we leverage this classification and focus our later analysis on the effect the provided location has on the repair performance of Agentless and baseline approaches.
These classification dimensions and categories raise potential issues with the SWE-bench Lite problems such as unsolvable questions, misleading potential solutions, and significant differences in problem difficulties. These issues have not been properly considered by either the benchmark creation process or prior approaches. Furthermore, we hope our classification can provide additional insights on the type of problems that can be solved by existing and future approaches.
6.2 SWE-bench Lite- $S$
Table 5: Performance and ranking on SWE-bench Lite- $S$ . * indicates a tie in ranking.
| Tool | LLM | SWE-bench Lite | SWE-bench Lite- $S$ | | |
| --- | --- | --- | --- | --- | --- |
| % Resolved | Rank | % Resolved | Rank | | |
| CodeStory Aide cod [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o+
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude 3.5 S | 129 (43.00%) | 1 | 114 (45.78%) | 1 |
| Bytedance MarsCode Liu et al. [2024b]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 118 (39.33%) | 2 | 106 (42.57%) | 2 |
| Honeycomb hon [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 115 (38.33%) | 3 | 98 (39.36%) | 3 |
| MentatBot men [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 114 (38.00%) | 4 | 96 (38.55%) | 4 |
| Gru gru [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 107 (35.67%) | 5 | 94 (37.75%) | 5 |
| Isoform iso [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 105 (35.00%) | 6 | 91 (36.55%) | 6 |
| SuperCoder2.0 sup [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 102 (34.00%) | 7 | 87 (34.94%) | 7 |
| Alibaba Lingma Agent lin [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o+
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude 3.5 S | 99 (33.00%) | 8 | 86 (34.54%) | 8 |
| Factory Code Droid fac [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 94 (31.33%) | 10 | 82 (32.93%) | 10 |
| Amazon Q Developer-v2 ama [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 89 (29.67%) | 12* | 76 (30.52%) | 13 |
| CodeR Chen et al. [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4 | 85 (28.33%) | 14 | 71 (28.51%) | 14* |
| MASAI Arora et al. [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 84 (28.00%) | 15 | 70 (28.11%) | 16 |
| SIMA sim [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 83 (27.67%) | 16 | 71 (28.51%) | 14* |
| IBM Research Agent-101 ibm [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 80 (26.67%) | 17* | 66 (26.51%) | 18* |
| OpenCSG StarShip ope [2024a]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4 | 71 (23.67%) | 22 | 56 (22.49%) | 23* |
| Amazon Q Developer ama [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
| NA | 61 (20.33%) | 26 | 51 (20.48%) | 25* |
| RepoUnderstander Ma et al. [2024]
<details>
<summary>resources/locked_1f512.png Details</summary>
### Visual Description
# Technical Image Analysis: Padlock Icon
## 1. Overview
The image is a flat-design vector illustration of a closed padlock. It does not contain any text, charts, data tables, or complex diagrams. It serves as a symbolic representation of security, privacy, or encryption.
## 2. Component Isolation
The image is composed of three primary geometric regions:
* **Shackle (Top):** A grey, inverted U-shaped component representing the locking bar.
* **Body (Main):** A yellow, rounded rectangular block forming the base of the lock.
* **Keyhole (Center):** A dark purple/navy blue silhouette located in the center of the body.
## 3. Technical Specifications & Spatial Grounding
| Component | Color (Visual) | Shape Description | Spatial Placement |
| :--- | :--- | :--- | :--- |
| **Shackle** | Light Grey | Inverted "U" with flat ends | Top-center, partially overlapping the body. |
| **Body** | Golden Yellow | Rounded Rectangle | Central/Main region of the image. |
| **Keyhole** | Dark Purple | Circular top with a flared rectangular base | Centered horizontally and vertically on the yellow body. |
## 4. Functional Description
The illustration depicts a **locked state**. The shackle is positioned flush against the body, indicating that the mechanism is engaged. The design uses high-contrast colors to distinguish between the functional parts of the hardware (shackle, body, and keyhole).
## 5. Data Extraction
* **Textual Information:** None.
* **Numerical Data:** None.
* **Labels/Legends:** None.
* **Trends:** Not applicable.
</details>
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4 | 64 (21.33%) | 25 | 51 (20.48%) | 25* |
| AutoCodeRover-v2 aut [2024] |
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 92 (30.67%) | 11 | 79 (31.73%) | 11 |
| RepoGraph rep [2024] |
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 89 (29.67%) | 12* | 77 (30.92%) | 12 |
| Moatless moa [2024] |
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude 3.5 S | 80 (26.67%) | 17* | 67 (26.91%) | 17 |
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 74 (24.67%) | 21 | 62 (24.90%) | 21 | |
| OpenDevin+CodeAct v1.8 ope [2024b] |
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude 3.5 S | 80 (26.67%) | 17* | 65 (26.10%) | 20 |
| Aider Gauthier [2024] |
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o+
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude 3.5 S | 79 (26.33%) | 20 | 66 (26.51%) | 18* |
| SWE-agent Yang et al. [2024a] |
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude 3.5 S | 69 (23.00%) | 23 | 58 (23.29%) | 22 |
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 55 (18.33%) | 28 | 45 (18.07%) | 27* | |
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4 | 54 (18.00%) | 29 | 42 (16.87%) | 29 | |
| AppMap Navie app [2024] |
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 65 (21.67%) | 24 | 56 (22.49%) | 23* |
| AutoCodeRover Zhang et al. [2024c] |
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4 | 57 (19.00%) | 27 | 45 (18.07%) | 27* |
| RAG Yang et al. [2024a] |
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude 3 Opus | 13 (4.33%) | 30 | 10 (4.02%) | 30 |
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4 | 8 (2.67%) | 32 | 5 (2.01%) | 32 | |
|
<details>
<summary>resources/anthropic.png Details</summary>
### Visual Description
# Technical Document Extraction: Graphic Icon
## 1. Image Overview
The image is a square graphic with rounded corners, commonly used as an application icon or a brand logo. It features a minimalist design consisting of a background color and a central typographic element.
## 2. Component Isolation
### Background (Container)
* **Shape:** Rounded square (superellipse/squircle).
* **Color:** Solid tan or light brown (Hex approx: #C99B7B).
* **Spatial Placement:** Occupies the entire frame.
### Central Graphic (Main Element)
* **Content:** Stylized text characters "AI".
* **Color:** Solid black (#000000).
* **Spatial Placement:** Centered horizontally and vertically within the rounded square.
* **Typography/Style:**
* **"A":** A bold, sans-serif capital letter "A". The horizontal crossbar is positioned at a standard height.
* **"I":** A bold, slanted parallel line. It is angled to match the right-side stroke of the "A", creating a sense of visual symmetry and forward motion. It lacks serifs or a base/cap.
## 3. Textual Information
* **Extracted Text:** AI
* **Contextual Meaning:** Commonly used as an abbreviation for "Artificial Intelligence."
## 4. Technical Specifications
* **Visual Style:** Flat design, minimalist.
* **Contrast:** High contrast between the black foreground elements and the tan background.
* **Symmetry:** The right-leaning slant of the "I" is parallel to the right leg of the "A", maintaining a consistent geometric angle across the graphic.
</details>
Claude-2 | 9 (3.00%) | 31 | 6 (2.41%) | 31 | |
|
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-3.5 | 1 (0.33%) | 33 | 0 (0.00%) | 33 | |
| Agentless |
<details>
<summary>resources/openai.png Details</summary>
### Visual Description
# Technical Image Analysis: OpenAI Logo
## 1. Overview
The image is a high-contrast, monochrome (black and white) graphic symbol. It is a geometric logo consisting of a series of interlocking, looped lines that form a hexagonal central void. This specific design is the official logo for **OpenAI**.
## 2. Component Isolation and Geometric Analysis
The graphic can be segmented into the following structural components:
* **Central Core:** A white hexagonal space located at the center of the image.
* **Interlocking Loops:** Six identical, overlapping "U-shaped" or "hook" paths.
* **Symmetry:** The image exhibits six-fold rotational symmetry (hexagonal symmetry). Each of the six segments is rotated 60 degrees relative to the next.
* **Line Style:** The lines are thick, black, and have rounded ends (caps). The negative space (white) between the black lines is of uniform width, creating a balanced, "woven" appearance.
## 3. Flow and Path Description
The design creates a visual "flow" that suggests a continuous, recursive movement.
* Each of the six segments begins near the outer perimeter with a rounded curve.
* The path then moves inward toward the center, forming one side of the central hexagon.
* The path then bends sharply and exits back toward the perimeter, tucking under or over the adjacent segment.
* This "over-under" visual logic mimics a Celtic knot or a 2D representation of a complex 3D manifold (like a Mobius strip or a Torus).
## 4. Spatial Grounding and Dimensions
* **Background:** Solid light gray/off-white (#EEEEEE).
* **Foreground:** Solid black (#000000).
* **Orientation:** The logo is centered. One vertex of the internal hexagon points directly upward, and one vertex points directly downward.
* **Aspect Ratio:** 1:1 (Square).
## 5. Data and Textual Information
* **Textual Content:** None. There are no labels, axis titles, legends, or embedded words in this image.
* **Data Points:** Not applicable. This is a brand mark/icon, not a data visualization or chart.
## 6. Technical Summary for Documentation
This image serves as a symbolic representation of interconnectedness and artificial intelligence. In a technical document, this would be identified as the **OpenAI Logo**. It is a vector-style graphic characterized by its mathematical precision, hexagonal symmetry, and interlocking line-work.
</details>
GPT-4o | 96 (32.00%) | 9 | 84 (33.73%) | 9 |
Building on the above problem classifications, we will more rigorously compare and contrast Agentless and existing work. Specifically, we focus on a subset of the problems in SWE-bench Lite after removing the problems that contain the exact patch in the problem description, misleading solutions, or do not provide enough information in the original issue description. This eliminates the less reasonable problems and normalizes the difficulty level of the benchmark. We refer to our subset of 249 problems as SWE-bench Lite- $S$ . We note here that our approach of identifying and excluding problematic problems has already been confirmed by a later work done by OpenAI, where they have released a similar filtered benchmark of SWE-bench Verified Chowdhury et al. [2024].
Table 5 shows the results on the SWE-bench Lite- $S$ benchmark and the corresponding ranking of each approach. We also included the results on the original 300 problems in SWE-bench Lite for comparison. While the general ranking of all approaches stay roughly the same, we do observe some small ranking changes. Compared to the original SWE-bench Lite, our filtered benchmark of SWE-bench Lite- $S$ provides a more accurate reflection of the true capability of autonomous software development tools.
<details>
<summary>x11.png Details</summary>
### Visual Description
### Technical Data Extraction: Solve Rate Comparison Chart
This document provides a detailed extraction of the data presented in the bar chart comparing "Solve rate" across different software engineering agents, categorized by the presence or absence of reproducible examples.
#### 1. Chart Metadata and Axes
* **Y-Axis Title:** Solve rate
* **Y-Axis Scale:** 0.0 to 0.6 (increments of 0.1)
* **X-Axis Categories:** 7 distinct software agents/models.
* **Legend (Top Right):**
* **Light Blue Bar:** `w/o reproducible examples` (without)
* **Light Purple Bar:** `w/ reproducible examples` (with)
* **Red Dashed Line:** Represents the average or baseline solve rate for each specific category.
#### 2. Data Table Extraction
The following table reconstructs the visual data points. Values are estimated based on the Y-axis scale.
| Agent Name | Solve Rate (w/o reproducible) | Solve Rate (w/ reproducible) | Average (Red Dashed Line) |
| :--- | :---: | :---: | :---: |
| **AutoCodeRover-v2** | ~0.40 | ~0.28 | ~0.32 |
| **RepoGraph** | ~0.37 | ~0.28 | ~0.31 |
| **Moatless-Claude-3.5** | ~0.36 | ~0.23 | ~0.27 |
| **CodeStoryAide** | ~0.57 | ~0.41 | ~0.46 |
| **MarsCode** | ~0.50 | ~0.40 | ~0.43 |
| **Honeycomb** | ~0.51 | ~0.35 | ~0.39 |
| **Agentless** | ~0.34 | ~0.33 | ~0.34 |
#### 3. Trend Analysis and Observations
* **Consistent Performance Drop:** In every single category, the "w/o reproducible examples" (light blue) performance is higher than the "w/ reproducible examples" (light purple) performance. This indicates that these agents currently perform better when not provided with specific reproduction scripts/examples, or that the examples provided may introduce complexity that lowers the solve rate.
* **Top Performer:** **CodeStoryAide** shows the highest overall solve rate, peaking at approximately 0.57 without reproducible examples.
* **Lowest Performance Gap:** **Agentless** shows the smallest discrepancy between the two conditions, with both bars and the red dashed line converging around the 0.33–0.34 mark.
* **Largest Performance Gap:** **Honeycomb** and **Moatless-Claude-3.5** show significant drops in performance when reproducible examples are included.
* **X-Axis Label Coloring:**
* **Orange Labels:** AutoCodeRover-v2, RepoGraph, Moatless-Claude-3.5, Agentless.
* **Purple Labels:** CodeStoryAide, MarsCode, Honeycomb.
#### 4. Spatial Grounding and Component Isolation
* **Header/Legend:** Located at the top right [x=0.6, y=0.9]. It defines the two-bar comparison.
* **Main Chart Area:** Occupies the central 80% of the image. The red dashed lines are centered horizontally across each pair of bars for a specific agent.
* **Footer (X-Axis):** Labels are rotated approximately 30 degrees clockwise to accommodate long names. The color coding of the text (Orange vs. Purple) suggests a secondary grouping of the agents not explicitly defined in the main legend.
</details>
(a) Description quality
<details>
<summary>x12.png Details</summary>
### Visual Description
# Technical Document Extraction: Solve Rate Comparison Chart
## 1. Image Overview
This image is a grouped bar chart comparing the "Solve rate" of seven different AI/software engineering agents across two conditions. The chart includes a legend, a Y-axis for numerical values, and an X-axis for categorical labels. Red dashed horizontal lines are overlaid on each group to indicate a baseline or average reference point.
## 2. Component Isolation
### Header / Legend
* **Location:** Top-left corner [x: ~10%, y: ~5%].
* **Legend Items:**
* **Light Blue Square:** "contains NL steps"
* **Light Purple Square:** "without steps"
### Main Chart Area
* **Y-Axis Label:** "Solve rate" (Vertical orientation).
* **Y-Axis Scale:** 0.0 to 0.7 (Markers at 0.1 intervals: 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7).
* **X-Axis Labels:** Seven agent names, rotated diagonally.
* **Data Series:**
* **Series 1 (Light Blue):** Represents "contains NL steps". In every category, this bar is significantly taller than the purple bar.
* **Series 2 (Light Purple):** Represents "without steps".
* **Reference Lines (Red Dashed):** A short horizontal dashed line is placed over each pair of bars, typically positioned between the heights of the two bars.
## 3. Data Extraction
### Table Reconstruction
The following values are estimated based on the Y-axis scale:
| Agent Name | contains NL steps (Blue) | without steps (Purple) | Reference Line (Red Dashed) |
| :--- | :---: | :---: | :---: |
| **AutoCodeRover-v2** | ~0.54 | ~0.26 | ~0.32 |
| **RepoGraph** | ~0.48 | ~0.27 | ~0.31 |
| **Moatless-Claude-3.5** | ~0.56 | ~0.20 | ~0.27 |
| **CodeStoryAide** | ~0.69 | ~0.40 | ~0.46 |
| **MarsCode** | ~0.65 | ~0.37 | ~0.43 |
| **Honeycomb** | ~0.73 | ~0.31 | ~0.39 |
| **Agentless** | ~0.52 | ~0.29 | ~0.34 |
### X-Axis Label Color Coding
The labels on the X-axis are color-coded:
* **Orange Text:** AutoCodeRover-v2, RepoGraph, Moatless-Claude-3.5, Agentless.
* **Purple Text:** CodeStoryAide, MarsCode, Honeycomb.
## 4. Trend Analysis and Observations
* **Primary Trend:** For all seven agents, the "contains NL steps" (light blue) condition results in a substantially higher solve rate compared to the "without steps" (light purple) condition.
* **Performance Peak:** **Honeycomb** achieves the highest solve rate in the "contains NL steps" category (approx. 0.73), while **CodeStoryAide** has the highest solve rate in the "without steps" category (approx. 0.40).
* **Performance Minimum:** **Moatless-Claude-3.5** shows the lowest solve rate in the "without steps" category (approx. 0.20), creating the largest visual gap between the two conditions for that specific agent.
* **Reference Line Logic:** The red dashed lines appear to represent a baseline or a weighted average. In all cases, the "contains NL steps" performance exceeds this reference line, while the "without steps" performance falls below it.
</details>
(b) Solution in description
<details>
<summary>x13.png Details</summary>
### Visual Description
# Technical Data Extraction: Solve Rate by Model and Context Type
## 1. Image Overview
This image is a grouped bar chart comparing the "Solve rate" of seven different AI coding models/frameworks across four distinct context conditions. Each model group also features a horizontal dashed red line representing an average or baseline performance metric for that specific model.
## 2. Chart Metadata
* **Y-Axis Label:** Solve rate
* **Y-Axis Scale:** 0.0 to 0.7 (with markers every 0.1)
* **X-Axis Labels (Models):**
* AutoCodeRover-v2 (Orange)
* RepoGraph (Orange)
* Moatless-Claude-3.5 (Orange)
* CodeStoryAide (Purple)
* MarsCode (Purple)
* Honeycomb (Purple)
* Agentless (Orange)
* **Legend (Top Left):**
* **Light Blue:** `in NL` (Natural Language)
* **Light Purple:** `in stack trace`
* **Light Green:** `keywords`
* **Light Red/Pink:** `none`
* **Additional Indicators:** Red dashed horizontal lines spanning the width of each model's bar group.
## 3. Data Table Extraction
The following table reconstructs the approximate numerical values based on the visual height of the bars relative to the Y-axis markers.
| Model | in NL (Blue) | in stack trace (Purple) | keywords (Green) | none (Red) | Baseline (Red Dash) |
| :--- | :---: | :---: | :---: | :---: | :---: |
| **AutoCodeRover-v2** | ~0.49 | ~0.26 | ~0.34 | ~0.24 | ~0.32 |
| **RepoGraph** | ~0.55 | ~0.37 | ~0.33 | ~0.16 | ~0.31 |
| **Moatless-Claude-3.5** | ~0.55 | ~0.24 | ~0.28 | ~0.14 | ~0.27 |
| **CodeStoryAide** | ~0.69 | ~0.45 | ~0.48 | ~0.34 | ~0.46 |
| **MarsCode** | ~0.63 | ~0.39 | ~0.43 | ~0.34 | ~0.43 |
| **Honeycomb** | ~0.71 | ~0.32 | ~0.41 | ~0.26 | ~0.39 |
| **Agentless** | ~0.59 | ~0.32 | ~0.36 | ~0.21 | ~0.34 |
## 4. Trend Analysis and Observations
### Component Isolation: Performance by Context
1. **"in NL" (Light Blue):** This is consistently the highest-performing category across all models. It peaks with *Honeycomb* at >0.7 and is lowest for *AutoCodeRover-v2* at just under 0.5.
2. **"in stack trace" (Light Purple):** Generally the second or third highest. It shows a significant drop compared to "in NL" for all models.
3. **"keywords" (Light Green):** Often performs slightly better than "in stack trace" (e.g., *AutoCodeRover-v2*, *CodeStoryAide*, *MarsCode*, *Honeycomb*, *Agentless*), but slightly worse in *RepoGraph*.
4. **"none" (Light Red):** Consistently the lowest-performing category for every model, indicating that providing no context significantly degrades the solve rate.
### Model Comparison
* **Top Performer:** *Honeycomb* achieves the highest single solve rate (~0.71) when provided with Natural Language context.
* **Most Consistent High Performer:** *CodeStoryAide* maintains high solve rates across all categories, with its "none" context performance (~0.34) being higher than the "none" performance of most other models.
* **Baseline (Red Dash):** The red dashed line typically sits between the "in stack trace/keywords" bars and the "none" bars, representing a weighted average or aggregate performance metric.
</details>
(c) Location information
Figure 9: Solve rate of selected approaches (orange means open-source while indigo means closed-source) on different problem categories in SWE-bench Lite- $S$ . Red dotted line indicates the average solve rate on the entire SWE-bench Lite- $S$ for each approach.
Using the classification results, we further examine the types of problems that are solved by Agentless and prior approaches on SWE-bench Lite- $S$ . Figure 9 shows the solve rate of various top-performing open-source and closed-source approaches across the different categories of problems. We first examine if having code examples to reproduce the error in the issue description can help the LLM better solve the issue in Figure 9(a). Surprisingly, we found that the solve rate of all prior approaches drop when evaluated on the problems with reproducible code examples. Many agent-based approaches Yang et al. [2024a]; ope [2024b]; Chen et al. [2024] attempt to first reproduce the error, however, this may not improve performance even on problems with already provided reproducible examples. However, we observe the performance for Agentless remains very high on the problems with reproducible code examples. This is because Agentless generates reproduction tests using the original issues descriptions, hence can better make use of the reproducible code examples provided (similarly for MentatBot which also contains an explicit test generation step). This demonstrate the importance of the test generation stage for patch selection. Next, we look at the effect of ground truth patch/solution in the issue description. Figure 9(b) shows the expected result where all selected techniques perform better on issues that already provide solution steps in natural language. Furthermore, in Figure 9(c), we examine the solve rate with respect to the location information provided in the issues description. Unsurprisingly, we found that the highest solve rates are on problems where the location is provided in natural language followed by stack traces. The most difficult problems are those that do not contain any clues about the location of the issue in the description. We observe that compared with closed-source approaches, Agentless performs comparably when the location is provided in either natural language, stack trace, or keywords. However, the closed-source agent tools perform better compared to Agentless in the case where no location clue is provided. This highlights an advantage of agent-based tools in solving these more complex problems where they are able to use complex code search tools. This represents potential future work for Agentless to target and further improve these types of problems.
6.3 SWE-bench Verified
Table 6: Performance on SWE-bench Verified.
Similar to SWE-bench Lite- $S$ and inspired by similar concerns in Section 6.1, OpenAI has produced a newly filtered dataset of SWE-bench Verified Chowdhury et al. [2024] validated by human developers to ensure each issue has sufficient amounts of information to be solved. Table 6 shows the performance of Agentless compared with prior agent-based approaches on SWE-bench Verified. Similar to the results on SWE-bench Lite, Agentless maintains its strong performance and is able to solve 194 out of 500 problems (38.80%). Agentless is able to achieve the second highest performance compared with open-source approaches and perform better than many closed-source / commercial techniques. Furthermore, we note that Agentless performs the best among all techniques that use GPT-4o as the LLM.
7 Threats to Validity
Internal. One threat to validity comes from the data leakage of ground truth developer patches in SWE-bench Lite being part of the training data for GPT-4o. Since GPT-4o is a closed-source model, we do not have access to the training data. Meanwhile, we note here that prior work almost exclusively used similar closed-source LLMs (e.g., GPT-4o, GPT-4, Claude-3.5, etc), and our approach can outperform all existing open-source solutions with same models. Furthermore, the authors of SWE-bench Jimenez et al. [2024a] compared the resolve rate of issues collected before and after the knowledge cutoff date of GPT-4, and did not find any significant difference. To completely address this threat, we would need to retrain GPT-4o from scratch which would be infeasible for an academic project.
External. One main external threat comes from our evaluation dataset of SWE-bench Lite. While the performance of Agentless might not generalize to other datasets, SWE-bench Lite is by far the most popular evaluation dataset which contains a diverse range of problems. In addition, OpenAI has independently performed an extensive evaluation of Agentless and other open-source solutions on SWE-bench Lite, SWE-bench, and their new SWE-bench Verified benchmark, further confirming that Agentless outperforms all other open-source agents Chowdhury et al. [2024]. Moreover, on Sept. 12th 2024, OpenAI just released the new o1 model family and also adopted Agentless as the top approach to showcase their performance on SWE-bench OpenAI [2024c]. In the future, we plan to further address this threat by evaluating Agentless on other benchmarks.
8 Conclusion
We propose Agentless – an agentless approach to automatically tackle software development problems. Agentless uses a simple three phase approach of localization, repair, and patch validation. Compared to prior agent-based approaches, Agentless deliberately disallows the LLM for autonomous tool usage or planning. Our evaluation on the popular SWE-bench Lite benchmark demonstrates that Agentless can achieve the highest performance compared with other open-source techniques while at the same time minimizing the cost. Furthermore, we perform a detailed classification of problems in SWE-bench Lite to not only offer new insights but to construct a more rigorous benchmark of SWE-bench Lite- $S$ after removing problematic problems.
Acknowledgments
We thank Jiawei Liu and Yuxiang Wei for providing some of the resources used to run the experiments. One of the authors would like to thank Jun Yang for generously gifting his old bike Sadly the bike is currently broken. which allowed the author to travel faster and thus increasing research speed.
References
- [1]
- ibm [2024] 2024. Agent-101: A Software Engineering Agent for Code Assistance developed by IBM Research. https://github.com/swe-bench/experiments/blob/main/evaluation/lite/20240612_IBM_Research_Agent101/README.md/.
- cod [2024] 2024. Aide by Codestory. https://github.com/swe-bench/experiments/tree/main/evaluation/lite/20240702_codestory_aide_mixed.
- sim [2024] 2024. Alex SIMA. https://github.com/swe-bench/experiments/tree/main/evaluation/lite/20240706_sima_gpt4o.
- ama [2024] 2024. Amazon Q Developer The most capable generative AI–powered assistant for software development. https://aws.amazon.com/q/developer//.
- app [2024] 2024. AppMap speedruns to the top of the SWE Bench Leaderboard. https://appmap.io/blog/2024/06/20/appmap-navie-swe-bench-leader/.
- aut [2024] 2024. AutoCodeRover Autonomous Software Engineering. https://autocoderover.dev/.
- dev [2024] 2024. Devin, AI software engineer. https://www.cognition.ai/introducing-devin.
- com [2024] 2024. Empower your AI agents with Composio - a platform for managing and integrating tools with LLMs and AI agents using Function Calling. https://docs.composio.dev/introduction/intro/overview.
- fac [2024] 2024. Factory Bringing Autonomy to Software Engineering. https://www.factory.ai/.
- hon [2024] 2024. Honeycomb. https://honeycomb.sh.
- too [2024] 2024. Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku. https://www.anthropic.com/news/3-5-models-and-computer-use.
- iso [2024] 2024. Isoform. https://github.com/swe-bench/experiments/tree/main/evaluation/lite/20240829_Isoform.
- lin [2024] 2024. Lingma Agent. https://github.com/swe-bench/experiments/tree/main/evaluation/lite/20240622_Lingma_Agent.
- men [2024] 2024. MentatBot: New SOTA Coding Agent, Available Now. https://mentat.ai/blog/mentatbot-sota-coding-agent.
- moa [2024] 2024. Moatless Tools. https://github.com/aorwall/moatless-tools.
- ope [2024a] 2024a. OpenCSG StarShip. https://opencsg.com/product?class=StarShip/.
- ope [2024b] 2024b. OpenDevin: Code Less, Make More. https://github.com/OpenDevin/OpenDevin/.
- ast [2024] 2024. Python ast — Abstract Syntax Trees. https://docs.python.org/3/library/ast.html/.
- rep [2024] 2024. RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph. https://github.com/ozyyshr/RepoGraph.
- gru [2024] 2024. The Road to Ultimate Pull Request Machine. https://gru.ai/blog/road-to-ultimate-pull-request-machine/.
- sol [2024] 2024. Solver. https://solverai.com.
- sup [2024] 2024. SuperCoder. https://superagi.com/supercoder/.
- swe [2024] 2024. SWE-bench Lite. https://www.swebench.com/lite.html.
- Abreu et al. [2009] Rui Abreu, Peter Zoeteweij, Rob Golsteijn, and Arjan JC Van Gemund. 2009. A practical evaluation of spectrum-based fault localization. Journal of Systems and Software 82, 11 (2009), 1780–1792.
- Abreu et al. [2007] Rui Abreu, Peter Zoeteweij, and Arjan JC Van Gemund. 2007. On the accuracy of spectrum-based fault localization. In Testing: Academic and industrial conference practice and research techniques-MUTATION (TAICPART-MUTATION 2007). IEEE, 89–98.
- Anthropic [2024] Anthropic. 2024. Introducing Claude 3.5 Sonnet. https://www.anthropic.com/news/claude-3-5-sonnet/.
- Arora et al. [2024] Daman Arora, Atharv Sonwane, Nalin Wadhwa, Abhav Mehrotra, Saiteja Utpala, Ramakrishna Bairi, Aditya Kanade, and Nagarajan Natarajan. 2024. MASAI: Modular Architecture for Software-engineering AI Agents. arXiv preprint arXiv:2406.11638 (2024).
- Austin et al. [2021] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732 [cs.PL]
- Bouzenia et al. [2024] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. Repairagent: An autonomous, llm-based agent for program repair. arXiv preprint arXiv:2403.17134 (2024).
- Chen et al. [2024] Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, et al. 2024. CodeR: Issue Resolving with Multi-Agent and Task Graphs. arXiv preprint arXiv:2406.01304 (2024).
- Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021).
- Chen [2024] Yang Chen. 2024. Flakiness Repair in the Era of Large Language Models. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings. 441–443.
- Chen et al. [2019] Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. 2019. SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair. IEEE Transaction on Software Engineering (2019).
- Chowdhury et al. [2024] Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Kevin Liu, and Aleksander Madry. 2024. Introducing SWE-bench Verified. OpenAI Blog (2024). https://openai.com/index/introducing-swe-bench-verified/.
- Deng et al. [2023] Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2023. Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models. In 32nd International Symposium on Software Testing and Analysis (ISSTA).
- Deng et al. [2024] Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. 2024. Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT. In 46th International Conference on Software Engineering (ICSE).
- Gauthier [2024] Paul Gauthier. 2024. Aider is AI pair programming in your terminal. https://aider.chat/.
- Gazzola et al. [2019] Luca Gazzola, Daniela Micucci, and Leonardo Mariani. 2019. Automatic Software Repair: A Survey. IEEE Transactions on Software Engineering 45, 1 (2019), 34–67.
- Ghanbari et al. [2019] Ali Ghanbari, Samuel Benton, and Lingming Zhang. 2019. Practical Program Repair via Bytecode Mutation. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (Beijing, China) (ISSTA 2019). ACM, 19–30.
- Hidvégi et al. [2024] Dávid Hidvégi, Khashayar Etemadi, Sofia Bobadilla, and Martin Monperrus. 2024. Cigar: Cost-efficient program repair with llms. arXiv preprint arXiv:2402.06598 (2024).
- Huang et al. [2024] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large Language Models Cannot Self-Correct Reasoning Yet. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=IkmD3fKBPQ
- Jiang et al. [2023] Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. Impact of Code Language Models on Automated Program Repair. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 1430–1442. https://doi.org/10.1109/ICSE48619.2023.00125
- Jiang et al. [2021] Nan Jiang, Thibaud Lutellier, and Lin Tan. 2021. CURE: Code-Aware Neural Machine Translation for Automatic Program Repair. 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) (May 2021).
- Jimenez et al. [2024a] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024a. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66
- Jimenez et al. [2024b] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024b. SWE-bench Leaderboard. https://www.swebench.com/.
- Jin and Orso [2012] Wei Jin and Alessandro Orso. 2012. Bugredux: Reproducing field failures for in-house debugging. In 2012 34th international conference on software engineering (ICSE). IEEE, 474–484.
- Jones and Harrold [2005] James A Jones and Mary Jean Harrold. 2005. Empirical evaluation of the tarantula automatic fault-localization technique. In Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering. 273–282.
- Kang et al. [2024] Sungmin Kang, Gabin An, and Shin Yoo. 2024. A quantitative and qualitative evaluation of LLM-based explainable fault localization. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1424–1446.
- Kolak et al. [2022] Sophia D Kolak, Ruben Martins, Claire Le Goues, and Vincent Josua Hellendoorn. 2022. Patch Generation with Language Models: Feasibility and Scaling Behavior. In Deep Learning for Code Workshop.
- Le et al. [2016] Xuan Bach D. Le, David Lo, and Claire Le Goues. 2016. History Driven Program Repair. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 1. 213–224.
- Le Goues et al. [2012] Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. 2012. GenProg: A Generic Method for Automatic Software Repair. IEEE Transactions on Software Engineering 38, 1 (2012), 54–72.
- Lemieux et al. [2023] Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. CODAMOSA: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models. In 45th International Conference on Software Engineering (ICSE).
- Li et al. [2023] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you!
- Li et al. [2019] Xia Li, Wei Li, Yuqun Zhang, and Lingming Zhang. 2019. Deepfl: Integrating multiple fault diagnosis dimensions for deep fault localization. In Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis. 169–180.
- Li et al. [2020] Yi Li, Shaohua Wang, and Tien N. Nguyen. 2020. DLFix: Context-Based Code Transformation Learning for Automated Program Repair. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (Seoul, South Korea) (ICSE '20). Association for Computing Machinery, New York, NY, USA, 602–614.
- Liu et al. [2024c] Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024c. Large Language Model-Based Agents for Software Engineering: A Survey. arXiv preprint arXiv:2409.02977 (2024).
- Liu et al. [2019] Kui Liu, Anil Koyuncu, Dongsun Kim, and Tegawendé F. Bissyandé. 2019. TBar: Revisiting Template-Based Automated Program Repair. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2019). ACM, New York, NY, USA, 31–42.
- Liu et al. [2024b] Yizhou Liu, Pengfei Gao, Xinchen Wang, Chao Peng, and Zhao Zhang. 2024b. MarsCode Agent: AI-native Automated Bug Fixing. arXiv preprint arXiv:2409.00899 (2024).
- Liu et al. [2024a] Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, and Qing Wang. 2024a. Make llm a testing expert: Bringing human-like interaction to mobile gui testing via functionality-aware decisions. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13.
- LlamaIndex [2024] LlamaIndex. 2024. LlamaIndex, Data Framework for LLM Applications. https://www.llamaindex.ai/.
- Long and Rinard [2015] Fan Long and Martin Rinard. 2015. Staged Program Repair with Condition Synthesis. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy) (ESEC/FSE 2015). New York, NY, USA, 166–178.
- Long and Rinard [2016] Fan Long and Martin Rinard. 2016. An analysis of the search spaces for generate and validate patch generation systems. In Proceedings of the 38th International Conference on Software Engineering. 702–713.
- Lou et al. [2020] Yiling Lou, Ali Ghanbari, Xia Li, Lingming Zhang, Haotian Zhang, Dan Hao, and Lu Zhang. 2020. Can automated program repair refine fault localization? a unified debugging approach. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 75–87.
- Ma et al. [2024] Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. 2024. How to Understand Whole Software Repository? arXiv preprint arXiv:2406.01422 (2024).
- Mechtaev et al. [2016] Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. 2016. Angelix: Scalable Multiline Program Patch Synthesis via Symbolic Analysis. In Proceedings of the 38th International Conference on Software Engineering (Austin, Texas) (ICSE '16). 691–701.
- Meng et al. [2024] Ruijie Meng, Martin Mirchev, Marcel Böhme, and Abhik Roychoudhury. 2024. Large language model guided protocol fuzzing. In Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS).
- Meng et al. [2022] Xiangxin Meng, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. 2022. Improving fault localization and program repair with deep semantic features and transferred knowledge. In Proceedings of the 44th International Conference on Software Engineering. 1169–1180.
- Moon et al. [2014] Seokhyeon Moon, Yunho Kim, Moonzoo Kim, and Shin Yoo. 2014. Ask the mutants: Mutating faulty programs for fault localization. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation. IEEE, 153–162.
- Olausson et al. [2023] Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2023. Is Self-Repair a Silver Bullet for Code Generation?. In The Twelfth International Conference on Learning Representations.
- Oliinyk et al. [2024] Yaroslav Oliinyk, Michael Scott, Ryan Tsang, Chongzhou Fang, Houman Homayoun, et al. 2024. Fuzzing BusyBox: Leveraging LLM and Crash Reuse for Embedded Bug Unearthing. arXiv preprint arXiv:2403.03897 (2024).
- OpenAI [2023] OpenAI. 2023. GPT-4 Technical Report. ArXiv abs/2303.08774 (2023).
- OpenAI [2024a] OpenAI. 2024a. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/.
- OpenAI [2024b] OpenAI. 2024b. New embedding models and API updates. https://openai.com/index/new-embedding-models-and-api-updates/.
- OpenAI [2024c] OpenAI. 2024c. OpenAI o1 System Card. https://openai.com/index/openai-o1-system-card/.
- Ou et al. [2024] Xianfei Ou, Cong Li, Yanyan Jiang, and Chang Xu. 2024. The Mutators Reloaded: Fuzzing Compilers with Large Language Model Generated Mutation Operators. In ASPLOS.
- Papadakis et al. [2019] Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation testing advances: an analysis and survey. In Advances in computers. Vol. 112. Elsevier, 275–378.
- Papadakis and Le Traon [2015] Mike Papadakis and Yves Le Traon. 2015. Metallaxis-FL: mutation-based fault localization. Software Testing, Verification and Reliability 25, 5-7 (2015), 605–628.
- Pizzorno and Berger [2024] Juan Altmayer Pizzorno and Emery D Berger. 2024. CoverUp: Coverage-Guided LLM-Based Test Generation. arXiv preprint arXiv:2403.16218 (2024).
- Qin et al. [2024] Yihao Qin, Shangwen Wang, Yiling Lou, Jinhao Dong, Kaixin Wang, Xiaoling Li, and Xiaoguang Mao. 2024. AgentFL: Scaling LLM-based Fault Localization to Project-Level Context. arXiv preprint arXiv:2403.16362 (2024).
- Robertson et al. [2009] Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval 3, 4 (2009), 333–389.
- Ruan et al. [2024] Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. 2024. SpecRover: Code Intent Extraction via LLMs. arXiv preprint arXiv:2408.02232 (2024).
- Saha et al. [2013] Ripon K Saha, Matthew Lease, Sarfraz Khurshid, and Dewayne E Perry. 2013. Improving bug localization using structured information retrieval. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 345–355.
- Schäfer et al. [2023] Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. Adaptive Test Generation Using a Large Language Model. arXiv:2302.06527 [cs.SE]
- Shi et al. [2023] Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning. PMLR, 31210–31227.
- Singhal et al. [2001] Amit Singhal et al. 2001. Modern information retrieval: A brief overview. IEEE Data Eng. Bull. 24, 4 (2001), 35–43.
- Sohn and Yoo [2017] Jeongju Sohn and Shin Yoo. 2017. Fluccs: Using code and change metrics to improve fault localization. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. 273–283.
- Wang et al. [2015] Qianqian Wang, Chris Parnin, and Alessandro Orso. 2015. Evaluating the usefulness of ir-based fault localization techniques. In Proceedings of the 2015 international symposium on software testing and analysis. 1–11.
- Wang and Lo [2014] Shaowei Wang and David Lo. 2014. Version history, similar report, and structure: Putting them together for improved bug localization. In Proceedings of the 22nd international conference on program comprehension. 53–63.
- Wei et al. [2023] Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023. Magicoder: Source Code Is All You Need. arXiv preprint arXiv:2312.02120 (2023).
- Wong et al. [2016] W Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. A survey on software fault localization. IEEE Transactions on Software Engineering 42, 8 (2016), 707–740.
- Wu et al. [2023] Yonghao Wu, Zheng Li, Jie M Zhang, Mike Papadakis, Mark Harman, and Yong Liu. 2023. Large language models in fault localisation. arXiv preprint arXiv:2308.15276 (2023).
- Xi et al. [2023] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864 (2023).
- Xia et al. [2024] Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024. Universal Fuzzing via Large Language Models. In 46th International Conference on Software Engineering (ICSE).
- Xia et al. [2023a] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023a. Automated Program Repair in the Era of Large Pre-trained Language Models. In Proceedings of the ACM/IEEE 45th International Conference on Software Engineering (ICSE '23).
- Xia et al. [2023b] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023b. Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery.
- Xia and Zhang [2022] Chunqiu Steven Xia and Lingming Zhang. 2022. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 959–971.
- Xia and Zhang [2023] Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv preprint arXiv:2304.00385 (2023).
- Yang et al. [2024b] Aidan ZH Yang, Claire Le Goues, Ruben Martins, and Vincent Hellendoorn. 2024b. Large language models for test-free fault localization. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–12.
- Yang et al. [2023a] Chenyuan Yang, Yinlin Deng, Runyu Lu, Jiayi Yao, Jiawei Liu, Reyhaneh Jabbarvand, and Lingming Zhang. 2023a. White-box Compiler Fuzzing Empowered by Large Language Models. arXiv:2310.15991 [cs.SE]
- Yang et al. [2023b] Chenyuan Yang, Zijie Zhao, and Lingming Zhang. 2023b. Kernelgpt: Enhanced kernel fuzzing via large language models. arXiv preprint arXiv:2401.00563 (2023).
- Yang et al. [2024a] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024a. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15793 (2024).
- Yuan et al. [2024] Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and improving chatgpt for unit test generation. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1703–1726.
- Zeller et al. [2019] Andreas Zeller, Rahul Gopinath, Marcel Böhme, Gordon Fraser, and Christian Holler. 2019. The fuzzing book.
- Zhang et al. [2024d] Kexun Zhang, Weiran Yao, Zuxin Liu, Yihao Feng, Zhiwei Liu, Rithesh Murthy, Tian Lan, Lei Li, Renze Lou, Jiacheng Xu, et al. 2024d. Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents. arXiv preprint arXiv:2408.07060 (2024).
- Zhang et al. [2024b] Lyuye Zhang, Kaixuan Li, Kairan Sun, Daoyuan Wu, Ye Liu, Haoye Tian, and Yang Liu. 2024b. Acfix: Guiding llms with mined common rbac practices for context-aware repair of access control vulnerabilities in smart contracts. arXiv preprint arXiv:2403.06838 (2024).
- Zhang et al. [2024a] Quanjun Zhang, Chunrong Fang, Yang Xie, YuXiang Ma, Weisong Sun, and Yun Yang Zhenyu Chen. 2024a. A Systematic Literature Review on Large Language Models for Automated Program Repair. arXiv preprint arXiv:2405.01466 (2024).
- Zhang et al. [2023] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. Siren's song in the AI ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219 (2023).
- Zhang et al. [2024c] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024c. AutoCodeRover: Autonomous Program Improvement. arXiv:2404.05427 [cs.SE]
- Zhou et al. [2012] Jian Zhou, Hongyu Zhang, and David Lo. 2012. Where should the bugs be fixed? more accurate information retrieval-based bug localization based on bug reports. In 2012 34th International conference on software engineering (ICSE). IEEE, 14–24.
- Zhu et al. [2021] Qihao Zhu, Zeyu Sun, Yuan-an Xiao, Wenjie Zhang, Kang Yuan, Yingfei Xiong, and Lu Zhang. 2021. A Syntax-Guided Edit Decoder for Neural Program Repair. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, New York, NY, USA, 341–353.