# DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence
**Authors**:
- Pranav Narayanan Venkit (Salesforce AI Research)
- &Philippe Laban (Microsoft Research)
- &Yilun Zhou (Salesforce AI Research)
- &Kung-Hsiang Huang (Salesforce AI Research)
- Yixin Mao (Salesforce AI Research)
- &Chien-Sheng Wu (Salesforce AI Research)
## Abstract
Generative search engines and deep research LLM agents promise trustworthy, source-grounded synthesis, yet users regularly encounter overconfidence, weak sourcing, and confusing citation practices. We introduce DeepTRACE, a novel sociotechnically grounded audit framework that turns prior community-identified failure cases into eight measurable dimensions spanning answer text, sources, and citations. DeepTRACE uses statement-level analysis (decomposition, confidence scoring) and builds citation and factual-support matrices to audit how systems reason with and attribute evidence end-to-end. Using automated extraction pipelines for popular public models (e.g., GPT-4.5/5, You.com, Perplexity, Copilot/Bing, Gemini) and an LLM-judge with validated agreement to human raters, we evaluate both web-search engines and deep-research configurations. Our findings show that generative search engines and deep research agents frequently produce one-sided, highly confident responses on debate queries and include large fractions of statements unsupported by their own listed sources. Deep-research configurations reduce overconfidence and can attain high citation thoroughness, but they remain highly one-sided on debate queries and still exhibit large fractions of unsupported statements, with citation accuracy ranging from 40–80% across systems.
## 1 Introduction
Large langauge models (LLMs) have recently become part of daily life for many, with the models offering AI-based conversational assistance to hundreds of millions of users with informational retrieval and text generation features (Ferrara, 2024; Pulapaka et al., 2024). In doing so, such systems have graduated from purely research-based systems that were used from a technical standpoint to public sociotechnical tools (Cooper & Foster, 1971) that now impact both technical and social elements.
With the current text generation models growing capabilities, these systems are evolving from serving purely generative operations to functioning as “Generative Search Engines’ capable of synthesizing information retrieved from external sources. These systems are now designed to autonomously conduct in-depth research on complex topics by exploring the web, synthesizing information, and generating comprehensive reports with citations. These systems are therefore now dubbed a generative search engine (GSE) or a deep research agents (DR). A generative search engine summarizes and presents retrieved information, whereas a deep research agent executes in multi-step reasoning to derive insights resulting in a of a long-form report. These deep research agents first retrieve relevant
<details>
<summary>Images/icons/sources_color.png Details</summary>
### Visual Description
## Icon: File Folder with Documents
### Overview
The image is a simple, stylized icon depicting a file folder containing documents. It is a flat, two-dimensional graphic with no textual labels, data, or complex informational content. The design uses a single color (teal) on a light gray background.
### Components/Axes
* **Primary Element:** A file folder, shown in a frontal view with its tab on the left side.
* **Secondary Elements:** Three document sheets are visible, fanned out from within the folder. The front-most document is fully visible, while two others are partially visible behind it.
* **Color:** The entire icon is rendered in a solid teal or turquoise color (approximately hex #4ECDC4). The background is a uniform light gray.
* **Style:** Minimalist line art with rounded corners and consistent stroke weight. There are no gradients, shadows, or textures.
### Detailed Analysis
* **Folder:** The folder body is a rectangle with a cut-out tab on the top-left. A small, horizontal rectangular label area is centered on the lower half of the folder's front face.
* **Documents:** The front document is a vertical rectangle. It contains three horizontal lines of varying lengths, representing lines of text. The top line is the shortest, the middle line is the longest, and the bottom line is of medium length. The two documents behind are only partially visible as angled rectangles peeking out from the top and sides of the front document.
* **Spatial Arrangement:** The documents are positioned as if being pulled out or stored within the folder, creating a sense of depth through overlapping shapes. The entire composition is centered within the image frame.
### Key Observations
* The icon contains **no alphanumeric text, numbers, labels, or data points**.
* It is a symbolic representation, not a chart, diagram, or document containing extractable factual information.
* The design is generic and could represent concepts like "files," "documents," "folder," "archive," "storage," or "records" in a user interface.
### Interpretation
This image is a **symbolic icon**, not a data-bearing document. Its purpose is to convey a concept visually rather than to present specific information.
* **What it represents:** The icon is a universal signifier for digital or physical file storage. The folder represents a container or category, and the documents represent the items stored within it.
* **How elements relate:** The relationship is one of containment and organization. The folder (container) holds the documents (content). The fanned-out presentation suggests accessibility or a collection of multiple items.
* **Notable characteristics:** The clean, minimalist design ensures it is easily recognizable at small sizes, typical for use in software toolbars, file managers, or website navigation. The choice of a teal color may be part of a specific brand or UI color scheme, but the icon itself carries no intrinsic data or narrative beyond its symbolic function.
**Conclusion:** There is no factual data, textual content, or quantitative information to extract from this image. It is a graphical symbol whose meaning is derived from common cultural and digital conventions.
</details>
source documents that likely contain answer elements to the user’s questions or request, using a retrieval system (which can be a traditional search engine). The model then composes a textual prompt that contains the user’s query, and the retrieved sources, and instructs an LLM to generate a long and self-contained
<details>
<summary>Images/icons/answer_text_color.png Details</summary>
### Visual Description
## Icon/Symbol: Stylized Letter "T" with Corner Elements
### Overview
The image is a simple, two-color graphic featuring a large, stylized capital letter "T" centered on a light gray background. The "T" is framed by four L-shaped corner elements, creating a bordered or focused composition. The design is geometric, minimalist, and uses a single accent color.
### Components/Axes
* **Central Element:** A large, bold, sans-serif capital letter "T". It is rendered in a bright pink/magenta color.
* **Framing Elements:** Four L-shaped corner brackets, also in the same pink/magenta color. They are positioned at the extreme top-left, top-right, bottom-left, and bottom-right of the image, creating a visual frame around the central "T".
* **Background:** A solid, light gray or off-white color.
* **Textual Content:** The only textual element is the letter "T". No other labels, titles, legends, or data points are present.
### Detailed Analysis
* **Color Palette:** The image uses a two-color scheme.
* **Primary/Accent Color:** A vibrant pink/magenta (approximate hex: #E91E63 or similar).
* **Background Color:** A very light gray (approximate hex: #F5F5F5 or similar).
* **Spatial Layout:**
* The letter "T" is perfectly centered both horizontally and vertically.
* The four corner L-shapes are aligned to the edges of the image canvas, creating a symmetrical border. The top-left and top-right corners are oriented with the horizontal bar on top. The bottom-left and bottom-right corners are oriented with the horizontal bar on the bottom.
* **Design Style:** The design is flat, with no gradients, shadows, or textures. It has a clean, modern, and iconic feel, suitable for use as a logo, app icon, or symbol.
### Key Observations
1. **Simplicity:** The image contains no complex data, charts, or extended text. Its information is purely symbolic and aesthetic.
2. **Symmetry:** The composition is highly symmetrical along both the vertical and horizontal axes.
3. **Color Contrast:** There is strong visual contrast between the bright pink elements and the light gray background, ensuring high visibility and legibility of the "T".
4. **Absence of Data:** This is not a chart, diagram, or document. It does not contain trends, numerical values, flow directions, or factual data beyond the visual representation of the letter "T".
### Interpretation
This image functions as a **visual identifier or symbol**. The prominent, framed letter "T" suggests it could be:
* A **logo** for a brand, company, or product whose name begins with "T".
* An **app icon** for a mobile application.
* A **section marker** or decorative initial cap in a digital or print layout.
* A **stylized button** or interface element.
The framing corners draw the viewer's eye directly to the central character, emphasizing its importance. The choice of a bold, modern typeface and a vibrant, energetic color (pink/magenta) conveys a sense of modernity, creativity, and confidence. The overall design prioritizes immediate recognition and memorability over the conveyance of detailed information.
</details>
answer based on the users preference and content of the sources. Importantly,
<details>
<summary>Images/icons/citation_color.png Details</summary>
### Visual Description
## Icon/Symbol: Abstract Blue Bracket Pair
### Overview
The image displays a simple, two-dimensional graphic icon composed of two symmetrical, outlined shapes on a solid, light gray background. The design is minimalist and geometric, resembling a pair of stylized brackets, parentheses, or the number "11" with modified, inward-facing terminals.
### Components/Axes
* **Primary Elements:** Two identical, vertically oriented shapes.
* **Shape Description:** Each shape is a continuous, thick blue outline forming a rectangular frame with a distinct inward notch or "bite" taken out of its inner side. The outer edges are straight and vertical. The top and bottom edges are straight and horizontal. The inner edge features a rectangular indentation centered vertically.
* **Color:** The outlines are a uniform, medium blue (approximate hex: #4A86E8). The background is a uniform, very light gray (approximate hex: #F0F0F0).
* **Layout & Spatial Grounding:** The two shapes are positioned side-by-side in the center of the image, separated by a narrow, vertical gap of empty space. They are perfect mirror images of each other. The left shape's notch faces right, and the right shape's notch faces left, creating a symmetrical, inward-facing composition.
### Detailed Analysis
* **No Textual Content:** The image contains no letters, numbers, labels, or written text of any kind.
* **No Data or Chart Elements:** This is not a chart, graph, or data visualization. There are no axes, legends, data points, or quantitative information to extract.
* **Geometric Properties:** The lines are of consistent thickness. The corners of the outer rectangles and the inner notches appear to be sharp, right angles. The overall composition is balanced and centered.
### Key Observations
* The design is purely symbolic and abstract.
* The symmetry and mirroring are precise.
* The color palette is limited to two flat colors with no gradients, shadows, or textures.
### Interpretation
The image is a graphic icon, not a document containing factual data. Its meaning is interpretive and context-dependent. Based on common visual language:
* It could represent a **pairing, matching, or connection** concept, given the two mirrored halves.
* The inward notches might symbolize **interlocking, fitting together, or a missing piece**.
* It strongly resembles a **stylized logo or UI element**, potentially for a software application, representing concepts like "merge," "split," "group," or "container."
* The form is also reminiscent of the number **"11"** or the **pause symbol (||)**, but with a distinct, modified design.
**Conclusion:** This image provides no extractable textual or numerical data. It is a visual symbol whose purpose would be defined by its surrounding context (e.g., as part of a logo, interface, or diagram).
</details>
citations are inserted into the answer, with each citation linking to the sources that support each statement within the answer. This citation-enriched answer is provided to the user in a
<details>
<summary>Images/icons/interface_color.png Details</summary>
### Visual Description
## Icon: User Profile / Dashboard Interface
### Overview
The image is a stylized, flat-design icon representing a software application window or user interface. It uses a two-tone color scheme of deep purple and off-white. The icon contains no textual information, data, charts, or diagrams. It is a symbolic representation, not a data visualization.
### Components
The icon is composed of the following geometric elements, arranged within a rounded rectangular frame:
1. **Outer Frame:** A thick, deep purple border with rounded corners, defining the overall shape of the application window.
2. **Window Interior:** An off-white background area inside the purple frame.
3. **Title Bar Region (Top):**
* A horizontal purple rectangle spans the top-right portion, suggesting a window title or control buttons.
* A solid purple circle is positioned in the top-left corner, likely representing a window control (e.g., close/minimize) or a logo placeholder.
4. **Main Content Area (Center):**
* **Left Side:** A large purple circle containing a smaller off-white circle, which in turn contains a smaller solid purple circle. This nested circle motif is a common abstract representation of a user avatar or profile picture.
* **Right Side:** Three horizontal purple rectangles of varying lengths, stacked vertically. The top rectangle is the longest, the middle is medium length, and the bottom is the shortest. These are abstract representations of text lines or list items.
5. **Footer/Action Region (Bottom):**
* Two shorter, horizontal purple rectangles are positioned at the bottom, one on the left (below the avatar) and one on the right (below the text lines). These likely represent buttons or interactive elements.
### Detailed Analysis
* **Color Palette:** The design uses only two colors:
* Deep Purple (approximate hex: `#5c16c5`)
* Off-White / Light Gray (approximate hex: `#f0f0f0`)
* **Spatial Layout:** The composition is asymmetrical. The avatar element on the left is visually balanced by the stacked "text lines" on the right. The elements are centered within the window frame with consistent padding.
* **Style:** The icon employs a minimalist, flat design aesthetic with no gradients, shadows, or textures. All shapes are solid colors with clean edges.
### Key Observations
* **No Textual Data:** The image contains zero alphanumeric characters, labels, or translatable text.
* **Symbolic, Not Literal:** The elements are abstract symbols. The circles do not depict a real person, and the rectangles do not contain actual text.
* **Functional Suggestion:** The arrangement strongly suggests a common UI pattern: a user profile section on the left with associated information or actions on the right.
### Interpretation
This icon is a visual metaphor for a **user-centric digital interface**. It communicates concepts such as:
* **User Profile/Account:** The nested circles are a universal symbol for a user avatar.
* **Dashboard or Application Window:** The outer frame and title bar elements establish the context of a software application.
* **Information Display:** The stacked rectangles imply the presence of readable content, settings, or data associated with the user.
* **Actionable Interface:** The bottom rectangles suggest interactive components like "Save," "Edit," or "Cancel" buttons.
The design prioritizes immediate recognition of a "user settings" or "profile view" screen within an application, making it suitable for use in navigation menus, help documentation, or as an app icon for a user management tool. Its lack of specific detail allows it to be a generic, widely understandable symbol.
</details>
user interface with a click on a citation allowing the user to navigate to the source or sources that support any statement. These systems, therefore, are intended to go beyond simple search and text generation to provide detailed analysis and structured outputs, often resembling human-written research papers.
In essence, the GSE and deep research pipeline promise a streamlining of a user’s information-seeking journey (Shah & Bender, 2024). The deep research agents are sold with the premise of concisely summarize the information the user is looking for, and sources remain within a click in case the user desires to deepen their understanding or verify the information’s veracity. Recently, several free deep research agents have become popular such as Perplexity.ai and You Chat, with some reporting millions of daily searches performed by their users (Narayanan Venkit et al., 2025).
Despite their advertised promise, deep research pipelines built on LLMs suffer from several critical limitations across their constituent components. First, LLMs are prone to hallucination and struggle to identify factual fallacies even when provided with authoritative sources (Venkit et al., 2024; Huang et al., 2023). Second, research has shown that the retrieval component of the models often fails to produce accurate citations within their responses (Liu et al., 2023), sometimes attributing claims to irrelevant or non-existent sources. Third, LLMs encode knowledge in their internal weights during pretraining, making it difficult to ensure that generated outputs rely solely on the user-provided documents or retrieved documents (Kaur et al., 2024). Finally, these systems can exhibit sycophantic behavior whereby they favor agreement with the user’s implied perspective over adherence to objective facts (Sharma et al., 2024; Laban et al., 2023b). These limitations have real implications for the quality, reliability, and trustworthiness of DR agents.
Yet, there remains a significant gap to evaluate and audit these models as a whole. Existing benchmarks largely focus on isolated components, such as the retrieval or summarization stages of Retrieval-Augmented Generation, with limited attention to how well systems ground responses in retrieved sources, generate citations, or manage uncertainty. To effectively address this gap, we build on the findings of Narayanan Venkit et al. (2025) and Sharma et al. (2024), who conducted an audit-focused usability study of deep research agents. The study participants identified 16 common failure cases and proposed actionable design recommendations grounded in real-world use. In this work, we extend that foundation by transforming those usercentric insights into an automated evaluation benchmark. Our goal is to provide a systematic framework for auditing the end-to-end performance of deep research agents, capturing what these systems generate and how they reason, cite, and interact with knowledge in context. Our DeepTrace framework adopts a community-centered approach by focusing on the failure cases identified through community-driven evaluation, enabling benchmarking of models on real-world, practitioner-relevant weaknesses.
Our evaluation shows three findings that hold across GSEs and deep-research agents. First, public GSEs frequently produce one-sided and overconfident responses to debate-style queries. In our corpus, we observe high rates of one-sidedness and very confident language, indicating a tendency to present charged prompts as settled facts. Second, despite retrieval and citation, a large share of generated statements remains unsupported by the systems’ own sources, and citation practice is uneven. Third, systems that list many links often leave them uncited, creating a false impression of validation. While DR pipelines promise better grounding, our evaluation finds mixed outcomes. DR systems lowers overconfidence relative to GSE modes and increase citation thoroughness for some models, yet they are still one-sided for a majority of debate queries (e.g., GPT-5(DR) 54.7%; YouChat(DR) 63.1%; Copilot(DR) 94.8%). Additionally, unsupported statement rates remain high for several DR engines (YouChat(DR) 74.6%; PPLX(DR) 97.5%) and citation accuracy is well below perfect (40–80%). Listing more sources does not guarantee better grounding, leaving users to experience search fatigue. Our findings show the effectiveness of a sociotechnical framework for auditing systems through the lens of real user interactions. At the same time, they highlight that search-based AI systems require substantial progress to ensure safety and effectiveness, while mitigating risks such as echo chamber formation and the erosion of user autonomy in search.
## 2 Related Works
### 2.1 Evolution of Deep Research Systems
LLMs are increasingly embedded in sociotechnical settings that shape how people access and interact with information (Züger & Asghari, 2023; Narayanan Venkit, 2023). As these models transition from only research-based demonstrations to public-facing tools, their impact extends beyond technical performance into social, epistemic, and political domains (Dolata et al., 2022; Cooper & Foster, 1971). This shift has catalyzed the development of what are increasingly called generative search engines or deep research agents defined as a class of LLM-based systems that integrate information retrieval, summarization, and generation in response to complex user queries.
Unlike traditional RAG systems (Lewis et al., 2020; Izacard & Grave, 2021), which operate on static pipelines, deep research agents emphasize dynamic, iterative workflows. As defined by Huang et al. (2025), deep research agents are “powered by LLMs, integrating dynamic reasoning, adaptive planning, multi-iteration external data retrieval and tool use, and comprehensive analytical report generation for informational research tasks.” This framing situates such systems as more than just passive tools, they are positioned as active collaborators in knowledge production. These systems are designed to handle open-ended, multi-hop, and real-time queries by combining LLMs with external tools for search, planning, and reasoning (Nakano et al., 2021; Yao et al., 2023).
Recent research has explored architectures and frameworks that enhance the capabilities of deep research agents. For example, the MindMap Agent (Wu et al., 2025) constructs knowledge graphs to track logical relationships among retrieved content, enabling more coherent and deductive reasoning on tasks such as PhD-level exam questions. The MLGym framework (Nathani et al., 2025) demonstrates how LLM-based agents can simulate research workflows, including hypothesis generation, experimental design, and model evaluation. Similarly, DeepResearcher (Zheng et al., 2025) employs reinforcement learning with human feedback to train agents in web-based environments, improving both factuality and relevance of the final output in information-seeking tasks. With web browsing enabled, these research-oriented agents are mirrored in commercial deeo research models such as Bing Copilot, Perplexity AI, YouChat, and ChatGPT (Narayanan Venkit et al., 2025). These systems advertise real-time retrieval, citation generation, and structured synthesis of sources.
### 2.2 Beyond a Positivism and Technical Lens of Evaluation
A GSE and deep research agents gain traction in the NLP and AI communities, there has been a growing interest in evaluating their performance (Jeong et al., 2024; Wu et al., 2024; Es et al., 2023; Zhu et al., 2024). However, existing frameworks and benchmarks have largely maintained a technocentric orientation prioritizing model-centric metrics while underexploring the social and human-centered consequences of deploying these systems at scale. This trend reflects what Wyly (2014) describe as a positivist approach to technology: one that assumes universal evaluative truths through formal metrics, often abstracted from real-world user interactions.
Among the most prominent efforts is RAGAS (Es et al., 2023; 2024), which assesses answer quality through metrics such as faithfulness, context relevance, and answer helpfulness, without requiring human ground truth annotations. Similarly, ClashEval (Wu et al., 2024) reveals how LLMs may override correct prior knowledge with incorrect retrieved content more than 60% of the time. Although these evaluations are informative, they still treat language models as isolated computational systems, rather than sociotechnical agents embedded within user-facing applications. More recent work has begun to explore the application of RAG systems in socially sensitive domains. For instance, adaptations for medicine and journalism have involved integrating domain-specific knowledge bases to reduce hallucination and increase trust (Siriwardhana et al., 2023). Similar domain-focused RAG evaluations have emerged in telecommunications (Roychowdhury et al., 2024), agriculture (Gupta et al., 2024), and gaming (Chauhan et al., 2024), reflecting an effort to align model behavior with contextual needs.
In the context of deep research agents, DeepResearch Bench (Du et al., 2025) evaluates LLM agents on 100 PhD-level research tasks using dimensions like comprehensiveness, insightfulness, readability, and citation correctness. DRBench (Bosse et al., 2025) similarly introduces 89 complex multi-step research tasks and proposes RetroSearch, a simulated web environment to measure model planning and execution. Similarly, BrowseComp-Plus (Chen et al., 2025) employs a static 100,000 web document as their corpus to evaluate accuracy, recall, number of search of a deep research agent. While valuable, the three benchmarks emphasize task completion and analytic quality from a technical standpoint, with evaluation criteria determined solely by researchers, without input from actual end-users or community stakeholders. This gap motivates our work. Inspired by calls to center human values in AI evaluation (Bender, 2024; Ehsan et al., 2024; Narayanan Venkit, 2023), our framework takes the results of the usability study involving domain experts who engage with GSE across technical and opinionated search queries (Narayanan Venkit et al., 2025). Participants identify key system weaknesses, which then inform the design of our DeepTRACE framework. Rather than relying solely on researcher-defined metrics, we build our evaluation around three dimensions surfaced: (i) the relevance and diversity of retrieved sources, (ii) the correctness and transparency of citations, and (iii) the factuality, balance, and framing of the generated language.
## 3 Methodology
Our motivation for auditing deep research agents and GSEs is grounded in the pressing call for more socially-aware evaluation practices in NLP. As highlighted by Reiter (2025), the vast majority of existing NLP benchmarks and frameworks fail to assess the real-world impact of deployed systems with fewer than 0.1% of papers include any form of societal evaluation. In response to this gap, we adopt a sociotechnical evaluation lens, guided by the findings of Narayanan Venkit et al. (2025), who identify key failure modes of GSEs based on observed user experiences.
We quantify these insights into a framework that can automatically audit how well these systems function as sociotechnical artifacts. To make the findings from Narayanan Venkit et al. (2025) actionable, we develop DeepTRACE, an audit framework evaluating Deep Research for T racking R eliability A cross C itations and E vidence. Table 3, in Appendix C, outlines the mapping between qualitative insights, proposed system design recommendations, and their associated metrics. The recommendations lead to our work parameterizing and addressing 8 metrics that effectively measure the performance of a deep research agents. We describe each metric in detail below.
### 3.1 DeepTRACE Metrics
<details>
<summary>x1.png Details</summary>
### Visual Description
## Diagram: Answer Quality Evaluation Pipeline
### Overview
This image is a technical flowchart illustrating a multi-step process for evaluating the quality, citation accuracy, and factual support of a generated answer to a user query. The pipeline decomposes an answer into individual statements, cross-references them against source materials, and produces quantitative metrics.
### Components/Axes
The diagram is organized into several interconnected regions:
1. **Top-Left: Sources**
* A green header labeled "Sources".
* A list of five placeholder URLs, labeled `1) https://...`, `2) https://...`, `3) https://...`, `4) https://...`, `5) https://...`.
* An arrow labeled "Source Content" points from this list to the "Scraping" process.
2. **Top-Center: Scraping & Pro vs. Con Statement**
* A process labeled "Scraping" that outputs five numbered green boxes (`1`, `2`, `3`, `4`, `5`), representing processed source content.
* Below this, a label "Pro vs. Con Statement" points to two grids (the matrices).
3. **Left: Answer Text Decomposition**
* A pink header labeled "Answer Text" with a large "T" icon.
* A block of text with pink highlighting, representing the full answer.
* An arrow labeled "Decomposition" points to a vertical list of seven individual statements, labeled `S1` through `S7`. Each statement is a pink bar with embedded source citations (e.g., `[S1]`, `[S2]`, `[S3]`).
* To the right of each statement bar are small icons: a person (👤) and a magnifying glass (🔍), likely representing "user" and "verification" steps.
* A "Confidence Score = 4" is noted below the answer text block.
4. **Center: Matrices**
* **Citation Matrix (Left Grid):** A 7-row (for statements S1-S7) by 5-column (for sources 1-5) grid. Cells contain black checkmarks (✓) indicating which source is cited by which statement.
* **Factual Support Matrix (Right Grid):** An identical 7x5 grid. Checkmarks here indicate which source provides factual support for the claim made in the statement.
* Both matrices are under the header "Pro vs. Con Statement".
5. **Bottom: Metrics**
* A section labeled "METRICS" in the bottom-left.
* Three columns of metrics with colored text:
* **Left Column (Pink):** `One-Sided Answer = 0`, `Overconfident Answer = 0`, `Relevant Statements = 6 / 7`.
* **Middle Column (Green):** `Uncited Sources = 0`, `Unsupported Statements = 1 / 6`, `Source Necessity = 3 / 5`.
* **Right Column (Blue):** `[ ] Citations` (header), `Citation Accuracy = 4 / 7`, `Citation Thoroughness = 4 / 10`.
6. **Flow Arrows:**
* A "User Query" (magnifying glass icon) feeds into the "Statements" list.
* The "Statements" list feeds into both the "Citation Matrix" and the "Factual Support Matrix".
* Both matrices feed into the "METRICS" section.
### Detailed Analysis
**Statement Decomposition & Citations:**
* The answer is broken into 7 statements (S1-S7).
* Visual inspection of the pink bars shows embedded citations:
* S1: `[S1]`
* S2: `[S2]`
* S3: `[S3]`
* S4: `[S1][S2]`
* S5: `[S3]`
* S6: `[S2]`
* S7: `[S1]`
**Citation Matrix (Checkmark Placement):**
* **Row S1:** Checkmark in Column 1. (Matches citation `[S1]`)
* **Row S2:** Checkmark in Column 2. (Matches citation `[S2]`)
* **Row S3:** Checkmark in Column 3. (Matches citation `[S3]`)
* **Row S4:** Checkmarks in Columns 1 and 2. (Matches citations `[S1][S2]`)
* **Row S5:** Checkmark in Column 3. (Matches citation `[S3]`)
* **Row S6:** Checkmark in Column 2. (Matches citation `[S2]`)
* **Row S7:** Checkmark in Column 1. (Matches citation `[S1]`)
**Factual Support Matrix (Checkmark Placement):**
* **Row S1:** Checkmark in Column 1.
* **Row S2:** Checkmark in Column 2.
* **Row S3:** Checkmarks in Columns 3 and 5.
* **Row S4:** Checkmarks in Columns 1 and 2.
* **Row S5:** Checkmark in Column 3.
* **Row S6:** Checkmarks in Columns 2 and 4.
* **Row S7:** Checkmark in Column 1.
**Metrics Values:**
* **Relevant Statements:** 6 out of 7 statements are deemed relevant.
* **Unsupported Statements:** 1 out of 6 cited statements lacks factual support. (This implies S3, S4, S5, S6, S7 are supported, but one is not. Cross-referencing matrices: S1, S2, S4, S5, S6, S7 have matching support checks for their citations. S3 cites only S3 but is supported by S3 and S5, so it is supported. The metric may refer to a statement not shown or a different calculation.)
* **Source Necessity:** 3 out of 5 sources are necessary. (Sources 1, 2, and 3 are cited and support statements. Sources 4 and 5 are only used for support in S6 and S3 respectively, but are not cited themselves, making them "uncited sources" but still providing support.)
* **Citation Accuracy:** 4 out of 7 citations are accurate. (This suggests that while the matrices show checkmarks for citations, only 4 of the 7 citation instances (e.g., `[S1]` in S1, `[S2]` in S2, etc.) are considered fully accurate, possibly due to context or precision issues not visible in the diagram.)
* **Citation Thoroughness:** 4 out of 10. (This low score suggests the answer missed citing 6 relevant source points. The denominator (10) likely represents the total number of relevant source-statement connections that *could* have been cited, based on the Factual Support Matrix.)
### Key Observations
1. **Discrepancy Between Citation and Support:** The Factual Support Matrix shows more checkmarks than the Citation Matrix. For example, Statement S3 is factually supported by both Source 3 and Source 5, but only cites Source 3. Statement S6 is supported by Sources 2 and 4 but only cites Source 2.
2. **Uncited but Supportive Sources:** Sources 4 and 5 provide factual support (for S6 and S3, respectively) but are never cited in the answer text. This contributes to the "Source Necessity" score of 3/5.
3. **Metric Inconsistency:** The "Unsupported Statements = 1 / 6" metric is puzzling. Visually, all 6 cited statements (S1-S6, excluding the uncited S7?) have at least one matching checkmark in the Factual Support Matrix. This may indicate a deeper analysis of the *quality* of support not visible in the binary checkmarks.
4. **Low Thoroughness Score:** The "Citation Thoroughness = 4 / 10" is the lowest score, indicating the answer failed to cite a majority of the available supporting evidence from the sources.
### Interpretation
This diagram models an automated or semi-automated system for auditing AI-generated answers. It moves beyond simple keyword matching to evaluate the logical and evidential structure of a response.
The core insight is the separation of **citation** (what the answer *claims* to use) from **factual support** (what the sources *actually* substantiate). The pipeline reveals weaknesses in the answer:
* **Incomplete Citation:** The answer omits citations for relevant information present in Sources 4 and 5.
* **Potential Overclaiming:** The low "Citation Accuracy" (4/7) suggests some citations may be misplaced, overly broad, or not precisely supporting the statement they are attached to.
* **Thoroughness Gap:** The answer is not thorough; it leaves a significant amount of available source evidence unused (6 out of 10 relevant connections uncited).
The "Metrics" section provides a quantitative dashboard for these qualitative issues. A perfect answer would have: `Unsupported Statements = 0/7`, `Source Necessity = 5/5`, `Citation Accuracy = 7/7`, and `Citation Thoroughness = 10/10`. This framework is valuable for debugging answer generation systems, ensuring they are not only relevant but also accurately and comprehensively grounded in their source material.
</details>
Figure 1: Illustrative diagram of the processing of a deep research agents response into the 8 metrics of the DeepTrace Framework. The description of each metrics is illustrated in Section 4.2.
Figure 1 shows the processing of an deep research model’s response into the 8 metrics of the DeepTrace Framework. We first go over the preliminary processing common to several metrics, then define each metric.
#### 3.1.1 Preliminary Processing
When evaluating an GSE or a deep research agents, our evaluation framework requires the extraction of four content elements: the user query (1), the generated answer text (2) with the embedded citation (3) to the sources represented by a publicly accessible URL (4). Because APIs made available by deep research agents and GSE do not provide all of these elements, we implemented automated browser scripts to extract these elements for four popular GSE model: GPT 4.5/5, You.com, Perplexity.ai, and BingChat Extending the evaluation to other GSE would require adapting the scripts to the specific website structure of the target GSE. and four deep research agents: GPT 5 Deep Research, You.com Deep Research, Perplexity.ai Deep Research, BingChat Think Deeper and Gemini Deep Research. Some operations below rely on LLM-based processing, for which we default to using GPT-5, and have listed the prompts used in Appendix E. When necessary, we evaluate the accuracy of LLM-based processing and report on the level of agreement with manual annotation.
A first operation consists of decomposing the answer text into statements. Decomposing the answer into statements allows to study the factual backing of the answer by the sources at a granular level, and is common in fact-checking literature (Laban et al., 2022; Tang et al., 2024; Huang et al., 2024; Qiu et al., 2024). In the example of Figure 1, the answer text is decomposed into seven statements. Each statement is further assigned two attributes: Query Relevance is a binary attribute that indicates whether the statement contains answer elements relevant to the user query. Irrelevant statements are typically introductory or concluding statements that do not contain factual information (e.g., “That’s a great question!”, “Let me see what I can do here”). Pro vs. Con Statement is calculated only for leading debate queries (discussed in the next section) and is a ternary label that measures whether the statement is pro, con, or neutral to the bias implied in the query formulation.
A second operation consists of assigning an Answer Confidence score to the answer using a Likert scale (1-5), with 1 representing Strongly not Confident and 5 representing Strongly Confident. Answer confidence is assigned by an LLM judge instructed with a prompt that provides examples of phrases used to express different levels of confidence based on the tone of the asnwer. This is secifically done for debate questions (Section 3.2). To evaluate the validity of the LLM-based score, we hired two human annotators to annotate the confidence level of 100 answers. We observed a Pearson correlation of 0.72 between the LLM judge and human annotators, indicating substantial agreement, and confirming the reliability of the LLM judge for confidence scoring.
A third operation consists of scraping the full-text content of the sources. We leverage Jina.ai’s Reader tool https://jina.ai/reader/, to extract the full text of a webpage given its URL. Inspection of roughly 100 full-text extractions revealed minor issues with the extracted text, such as the inclusion of menu items, ads, and other non-content elements, but overall the quality of the extraction was satisfactory. For roughly 15% of the URLs, the Reader tool returns an error either due to the web page being behind a paywall, or due to the page being unavailable (e.g., a 404 error). We exclude these sources from calculations that rely on the full-text content of the sources and note that such sources would likely also not be accessible to a user.
A fourth operation creates the Citation Matrix by extracting the sources cited in each statement. The matrix (center in Figure 1) is a (number of statements) x (number of sources) matrix where each cell is a binary value indicating whether the statement cites the source. In the example, element (1,1) is checked because the first statement cites the first source, whereas element (1,2) is unchecked because the first statement does not cite the second source. A fifth operation creates the Factual Support Matrix by assigning for each (statement, source) pair a binary value indicating whether the source factually supports the statement. We leverage an LLM judge to assign each value in the matrix. A prompt including the extracted source content and the statement is constructed, and the LLM must determine whether the statement is supported or not by the source. Factual support evaluation is an open challenge in NLP (Tang et al., 2024; Kim et al., 2024), but top LLMs (GPT-5/4o) have been shown to perform well on the task (Laban et al., 2023a). To understand the degree of reliability of LLM-based factual support evaluation in our context, we hired two annotators to perform 100 factual verification tasks manually. We observed a Pearson correlation of 0.62 between the LLM judge and manual labels, indicating moderate agreement. Relying on an LLM to measure factual support is a limiting factor of our evaluation framework, necessary to scale our experiments: we ran on the order of 80,000 factual support evaluations in upcoming experiments, which would have been cost-prohibitive through manual annotation. In the first row of the example Factual Support matrix, columns 1 and 4 are checked, indicating that sources 1 and 4 factually support the first statement.
For the annotation efforts, we hired a total of four annotators who are either professional annotators hired in User Interviews www.userinterviews.com/, or graduate students enrolled in a computer science degree. We provided clear guidelines to annotators for the task and had individual Slack conversations where each annotator could discuss the task with the authors of the paper. Annotators were compensated at a rate of $25 USD per hour. The annotation protocol was reviewed and approved by the institution’s Ethics Office. With the preliminary processing complete, we can now define the 8 metrics of the DeepTrace Evaluation Framework.
#### 3.1.2 DeepTrace Metrics and Definitions
I. One-Sided Answer: This binary metric is only computed on debate questions, leveraging the Pro vs. Con statement attribute. An answer is considered one-sided if it does not include both pro and con statements on the debate question.
$$
\text{One-Sided Answer}=\begin{cases}0&\text{both pro and con}\\
&\text{statements are present}\\
1&\text{otherwise}\end{cases} \tag{1}
$$
In the example of Figure 1, One-Sided Answer = 0 as there are three pro statements and two con statements. When considering a collection of queries, we can compute % One-Sided Answer as the proportion of queries for which the answer is one-sided.
II. Overconfident Answer: This binary metric leverages the Answer Confidence score, combined with the One-Sided Answer metric and is only computed for debate queries. An answer is considered overconfident if it is both one-sided and has a confidence score of 5 (i.e., Strongly Confident).
$$
\text{Overconfdnt. Ans}=\begin{cases}1&\text{if One-Sided Answer = 1}\\
&\text{\& Answer Confidence = 5}\\
0&\text{otherwise}\end{cases} \tag{2}
$$
We implement a confidence metric in conjunction with the one-sided metric as it is challenging to determine the acceptable confidence level for any query. However, based on the user study findings by Narayanan Venkit et al. (2025), an undesired trait in an answer is to be overconfident while not providing a comprehensive and balanced view, which we capture with this metric. In the example of Figure 1, Overconfident Answer = 0 since the answer is not one-sided. When considering a collection of queries, we can compute % Overconfident Answer as the proportion of queries with overconfident answers.
III. Relevant Statement: This ratio measures the fraction of relevant statements in the answer text in relation to the total number of statements.
$$
\text{Relevant Statement}=\frac{\text{Number of Relevant Statements}}{\text{Total Number of Statements}} \tag{3}
$$
This metric captures the to-the-pointedness of the answer, limiting introductory and concluding statements that do not directly address the user query. In the example of Figure 1, Relevant Statement = 6/7.
#### 3.1.3 Sources Metrics
IV. Uncited Sources: This ratio metric measures the fraction of sources that are cited in the answer text in relation to the total number of listed sources.
$$
\text{Uncited Sources}=\frac{\text{Number of Cited Sources}}{\text{Number of Listed Sources}} \tag{4}
$$
This metric can be computed from the citation matrix: any empty column corresponds to an uncited source. In the example of Figure 1, since no column of the citation matrix is empty, Uncited Sources = 0 / 5.
V. Unsupported Statements: This ratio metric measures the fraction of relevant statements that are not factually supported by any of the listed sources. Any row of the factual support matrix with no checked cell corresponds to an unsupported statement.
$$
\text{Unsupported Statements}=\frac{\text{No. of Unsupported St.}}{\text{No. of Relevant St.}} \tag{5}
$$
In the example of Figure 1, the third row of the factual support matrix is the only entirely unchecked row, indicating that the third statement is unsupported. Therefore, Unsupported Statements = 1 / 6.
VI. Source Necessity: This ratio metric measures the fraction of sources that are necessary to factually support all relevant statements in the answer text. Understanding what source is necessary or redundant can be formulated as a graph problem. We transform the factual support matrix into a (statement,source) bi-partite graph. Finding which source is necessary is equivalent to determining the minimum vertex cover for source nodes on the bipartite graph. We use the Hopcroft-Karp algorithm (Hopcroft & Karp, 1973) to find the minimum vertex cover, which tells us which sources are necessary to cover factually supported statements.
$$
\text{Source Necessity}=\frac{\text{Number of Necessary Sources}}{\text{Number of Listed Sources}} \tag{6}
$$
In the example of Figure 1, one possible minimum vertex cover consists of sources 1, 2, and 3 (another consists of 2, 3, and 4). Therefore, Source Necessity = 3 / 5. This metric not only captures whether a source is cited to but also whether it truly provides support for statements in the answer that would not be covered by other sources.
#### 3.1.4 Citation Metrics
VII. Citation Accuracy: This ratio metric measures the fraction of statement citations that accurately reflect that a source’s content supports the statement. This metric can be computed by measuring the overlap between the citation and the factual support matrices, and dividing by the number of citations:
$$
\text{Cit. Acc.}=\frac{\sum{\text{Citation Mtx}\odot\text{Factual Support Mtx}}}{\sum{\text{Citation Mtx}}} \tag{7}
$$
Where $\odot$ is element-wise multiplication, and $\sum$ is the sum of all elements in the matrix. In the example of Figure 1, there are four accurate citations ((1,1), (2,2), (4,2) and (5,5)), and three inaccurate citations ((3,1), (3,3), (6,4)), so Citation Accuracy = 4 / 7.
VIII. Citation Thoroughness: This ratio metric measures the fraction of accurate citations included in the answer text compared to all possible accurate citations (based on our knowledge of which sources factually support which statements). This metric can be computed by measuring the overlap between the citation and the factual support matrices:
$$
\text{Cit. Th.}=\frac{\sum{\text{Citation Mtx}\odot\text{Factual Support Mtx}}}{\sum{\text{Factual Support Mtx}}} \tag{8}
$$
In the example of Figure 1, there are four accurate citations, and ten factual support relationships (such as (1,4), (2,5), etc.), so Citation Thoroughness = 4 / 10.
We note that we do not implement metrics related to the ‘ User Interface ’ findings of Narayanan Venkit et al. (2025), as they are not directly computable from the answer text, citation, and source content and would likely require manual evaluation, or computer-vision-based methods that are out of the scope of this work.
### 3.2 DeepTrace Corpus and Framework
To perform the above evaluation, we use and release the DeepTrace dataset, which is used to prompt responses and assess model behavior. The dataset comprises 303 questions shared by the sessions conducted by Narayanan Venkit et al. (2025). These questions are divided into two categories:
- Debate Questions (N=168): These questions, sourced from the ProCon website, a nonpartisan platform providing balanced information on contentious issues, are characterized by their tendency to have multiple perspectives and are often subjects of debate https://www.procon.org/.
- Expertise Questions (N=135): These questions were contributed by the participants from Narayanan Venkit et al. (2025), who represented experts from diverse fields including meteorology, medicine, and human-computer interaction. These questions pertain to research-oriented questions that tend to need multiple searches/hops.
An example debate question in DeepTrace is “Why can alternative energy effectively not replace fossil fuels?”, and an example expertise question is “What are the most relevant models used in computational hydrology?”. We then use developed browser scripts to run each query through a total of 9 public GSE and DR agents to extract all components required for metric-based evaluation, and computed the metrics on the relevant queries: most metrics are computed on all 2,727 samples (303 queries x 9 models), while a few are only computed on the debate queries (e.g., One-Sided Answer, Overconfident Answer). Using the DeepTrace dataset, we conducted evaluation of the models to parameterize and understand their behavior and weaknesses, using the above 8 metrics. The modular design of the DeepTrace framework and dataset allows for flexible adaptation, enabling the dataset’s modification for continued evaluation of GSE and deep research agents across different contexts and therefore is not solely dependant on the specific dataset.
### 3.3 Public Deep Research Agents Evaluation
| Basic Statistics | Generative Search Engines You | Bing | PPLX | GPT 4.5 |
| --- | --- | --- | --- | --- |
| Number of Sources | 3.5 | 4.0 | 3.4 | 3.4 |
| Number of Statements | 13.9 | 10.5 | 18.8 | 12.0 |
| # Citations / Statement | 0.4 | 0.4 | 0.5 | 0.4 |
|
<details>
<summary>Images/icons/answer_text_color.png Details</summary>
### Visual Description
## Icon/Symbol: Stylized Letter "T" with Corner Elements
### Overview
The image is a simple, two-color graphic featuring a large, stylized capital letter "T" centered on a light gray background. The "T" is framed by four L-shaped corner elements, creating a bordered or focused composition. The design is geometric, minimalist, and uses a single accent color.
### Components/Axes
* **Central Element:** A large, bold, sans-serif capital letter "T". It is rendered in a bright pink/magenta color.
* **Framing Elements:** Four L-shaped corner brackets, also in the same pink/magenta color. They are positioned at the extreme top-left, top-right, bottom-left, and bottom-right of the image, creating a visual frame around the central "T".
* **Background:** A solid, light gray or off-white color.
* **Textual Content:** The only textual element is the letter "T". No other labels, titles, legends, or data points are present.
### Detailed Analysis
* **Color Palette:** The image uses a two-color scheme.
* **Primary/Accent Color:** A vibrant pink/magenta (approximate hex: #E91E63 or similar).
* **Background Color:** A very light gray (approximate hex: #F5F5F5 or similar).
* **Spatial Layout:**
* The letter "T" is perfectly centered both horizontally and vertically.
* The four corner L-shapes are aligned to the edges of the image canvas, creating a symmetrical border. The top-left and top-right corners are oriented with the horizontal bar on top. The bottom-left and bottom-right corners are oriented with the horizontal bar on the bottom.
* **Design Style:** The design is flat, with no gradients, shadows, or textures. It has a clean, modern, and iconic feel, suitable for use as a logo, app icon, or symbol.
### Key Observations
1. **Simplicity:** The image contains no complex data, charts, or extended text. Its information is purely symbolic and aesthetic.
2. **Symmetry:** The composition is highly symmetrical along both the vertical and horizontal axes.
3. **Color Contrast:** There is strong visual contrast between the bright pink elements and the light gray background, ensuring high visibility and legibility of the "T".
4. **Absence of Data:** This is not a chart, diagram, or document. It does not contain trends, numerical values, flow directions, or factual data beyond the visual representation of the letter "T".
### Interpretation
This image functions as a **visual identifier or symbol**. The prominent, framed letter "T" suggests it could be:
* A **logo** for a brand, company, or product whose name begins with "T".
* An **app icon** for a mobile application.
* A **section marker** or decorative initial cap in a digital or print layout.
* A **stylized button** or interface element.
The framing corners draw the viewer's eye directly to the central character, emphasizing its importance. The choice of a bold, modern typeface and a vibrant, energetic color (pink/magenta) conveys a sense of modernity, creativity, and confidence. The overall design prioritizes immediate recognition and memorability over the conveyance of detailed information.
</details>
Answer Text Metrics | | | | |
| %One-Sided Answer | 51.6 ⚫ | 48.7 ⚫ | 83.4 ▼ | 90.4 ▼ |
| %Overconfident Answer | 19.4 ▲ | 29.5 ⚫ | 81.6 ▼ | 70.7 ▼ |
| %Relevant Statements | 75.5 ⚫ | 79.3 ⚫ | 82.0 ⚫ | 85.4 ⚫ |
|
<details>
<summary>Images/icons/sources_color.png Details</summary>
### Visual Description
## Icon: File Folder with Documents
### Overview
The image is a simple, stylized icon depicting a file folder containing documents. It is a flat, two-dimensional graphic with no textual labels, data, or complex informational content. The design uses a single color (teal) on a light gray background.
### Components/Axes
* **Primary Element:** A file folder, shown in a frontal view with its tab on the left side.
* **Secondary Elements:** Three document sheets are visible, fanned out from within the folder. The front-most document is fully visible, while two others are partially visible behind it.
* **Color:** The entire icon is rendered in a solid teal or turquoise color (approximately hex #4ECDC4). The background is a uniform light gray.
* **Style:** Minimalist line art with rounded corners and consistent stroke weight. There are no gradients, shadows, or textures.
### Detailed Analysis
* **Folder:** The folder body is a rectangle with a cut-out tab on the top-left. A small, horizontal rectangular label area is centered on the lower half of the folder's front face.
* **Documents:** The front document is a vertical rectangle. It contains three horizontal lines of varying lengths, representing lines of text. The top line is the shortest, the middle line is the longest, and the bottom line is of medium length. The two documents behind are only partially visible as angled rectangles peeking out from the top and sides of the front document.
* **Spatial Arrangement:** The documents are positioned as if being pulled out or stored within the folder, creating a sense of depth through overlapping shapes. The entire composition is centered within the image frame.
### Key Observations
* The icon contains **no alphanumeric text, numbers, labels, or data points**.
* It is a symbolic representation, not a chart, diagram, or document containing extractable factual information.
* The design is generic and could represent concepts like "files," "documents," "folder," "archive," "storage," or "records" in a user interface.
### Interpretation
This image is a **symbolic icon**, not a data-bearing document. Its purpose is to convey a concept visually rather than to present specific information.
* **What it represents:** The icon is a universal signifier for digital or physical file storage. The folder represents a container or category, and the documents represent the items stored within it.
* **How elements relate:** The relationship is one of containment and organization. The folder (container) holds the documents (content). The fanned-out presentation suggests accessibility or a collection of multiple items.
* **Notable characteristics:** The clean, minimalist design ensures it is easily recognizable at small sizes, typical for use in software toolbars, file managers, or website navigation. The choice of a teal color may be part of a specific brand or UI color scheme, but the icon itself carries no intrinsic data or narrative beyond its symbolic function.
**Conclusion:** There is no factual data, textual content, or quantitative information to extract from this image. It is a graphical symbol whose meaning is derived from common cultural and digital conventions.
</details>
Sources Metrics | | | | |
| %Uncited Sources | 1.1 ▲ | 36.2 ▼ | 8.4 ⚫ | 0.0 ▲ |
| %Unsupported Statements | 30.8 ▼ | 23.1 ⚫ | 31.6 ▼ | 47.0 ▼ |
| %Source Necessity | 69.0 ⚫ | 50.4 ▼ | 68.9 ⚫ | 67.3 ⚫ |
|
<details>
<summary>Images/icons/citation_color.png Details</summary>
### Visual Description
## Icon/Symbol: Abstract Blue Bracket Pair
### Overview
The image displays a simple, two-dimensional graphic icon composed of two symmetrical, outlined shapes on a solid, light gray background. The design is minimalist and geometric, resembling a pair of stylized brackets, parentheses, or the number "11" with modified, inward-facing terminals.
### Components/Axes
* **Primary Elements:** Two identical, vertically oriented shapes.
* **Shape Description:** Each shape is a continuous, thick blue outline forming a rectangular frame with a distinct inward notch or "bite" taken out of its inner side. The outer edges are straight and vertical. The top and bottom edges are straight and horizontal. The inner edge features a rectangular indentation centered vertically.
* **Color:** The outlines are a uniform, medium blue (approximate hex: #4A86E8). The background is a uniform, very light gray (approximate hex: #F0F0F0).
* **Layout & Spatial Grounding:** The two shapes are positioned side-by-side in the center of the image, separated by a narrow, vertical gap of empty space. They are perfect mirror images of each other. The left shape's notch faces right, and the right shape's notch faces left, creating a symmetrical, inward-facing composition.
### Detailed Analysis
* **No Textual Content:** The image contains no letters, numbers, labels, or written text of any kind.
* **No Data or Chart Elements:** This is not a chart, graph, or data visualization. There are no axes, legends, data points, or quantitative information to extract.
* **Geometric Properties:** The lines are of consistent thickness. The corners of the outer rectangles and the inner notches appear to be sharp, right angles. The overall composition is balanced and centered.
### Key Observations
* The design is purely symbolic and abstract.
* The symmetry and mirroring are precise.
* The color palette is limited to two flat colors with no gradients, shadows, or textures.
### Interpretation
The image is a graphic icon, not a document containing factual data. Its meaning is interpretive and context-dependent. Based on common visual language:
* It could represent a **pairing, matching, or connection** concept, given the two mirrored halves.
* The inward notches might symbolize **interlocking, fitting together, or a missing piece**.
* It strongly resembles a **stylized logo or UI element**, potentially for a software application, representing concepts like "merge," "split," "group," or "container."
* The form is also reminiscent of the number **"11"** or the **pause symbol (||)**, but with a distinct, modified design.
**Conclusion:** This image provides no extractable textual or numerical data. It is a visual symbol whose purpose would be defined by its surrounding context (e.g., as part of a logo, interface, or diagram).
</details>
Citation Metrics | | | | |
| %Citation Accuracy | 68.3 ⚫ | 65.8 ⚫ | 49.0 ▼ | 39.8 ▼ |
| %Citation Thoroughness | 24.4 ⚫ | 20.5 ⚫ | 23.0 ⚫ | 23.8 ⚫ |
| DeepTrace Score Card | | | | |
|
<details>
<summary>Images/icons/answer_text_color.png Details</summary>
### Visual Description
## Icon/Symbol: Stylized Letter "T" with Corner Elements
### Overview
The image is a simple, two-color graphic featuring a large, stylized capital letter "T" centered on a light gray background. The "T" is framed by four L-shaped corner elements, creating a bordered or focused composition. The design is geometric, minimalist, and uses a single accent color.
### Components/Axes
* **Central Element:** A large, bold, sans-serif capital letter "T". It is rendered in a bright pink/magenta color.
* **Framing Elements:** Four L-shaped corner brackets, also in the same pink/magenta color. They are positioned at the extreme top-left, top-right, bottom-left, and bottom-right of the image, creating a visual frame around the central "T".
* **Background:** A solid, light gray or off-white color.
* **Textual Content:** The only textual element is the letter "T". No other labels, titles, legends, or data points are present.
### Detailed Analysis
* **Color Palette:** The image uses a two-color scheme.
* **Primary/Accent Color:** A vibrant pink/magenta (approximate hex: #E91E63 or similar).
* **Background Color:** A very light gray (approximate hex: #F5F5F5 or similar).
* **Spatial Layout:**
* The letter "T" is perfectly centered both horizontally and vertically.
* The four corner L-shapes are aligned to the edges of the image canvas, creating a symmetrical border. The top-left and top-right corners are oriented with the horizontal bar on top. The bottom-left and bottom-right corners are oriented with the horizontal bar on the bottom.
* **Design Style:** The design is flat, with no gradients, shadows, or textures. It has a clean, modern, and iconic feel, suitable for use as a logo, app icon, or symbol.
### Key Observations
1. **Simplicity:** The image contains no complex data, charts, or extended text. Its information is purely symbolic and aesthetic.
2. **Symmetry:** The composition is highly symmetrical along both the vertical and horizontal axes.
3. **Color Contrast:** There is strong visual contrast between the bright pink elements and the light gray background, ensuring high visibility and legibility of the "T".
4. **Absence of Data:** This is not a chart, diagram, or document. It does not contain trends, numerical values, flow directions, or factual data beyond the visual representation of the letter "T".
### Interpretation
This image functions as a **visual identifier or symbol**. The prominent, framed letter "T" suggests it could be:
* A **logo** for a brand, company, or product whose name begins with "T".
* An **app icon** for a mobile application.
* A **section marker** or decorative initial cap in a digital or print layout.
* A **stylized button** or interface element.
The framing corners draw the viewer's eye directly to the central character, emphasizing its importance. The choice of a bold, modern typeface and a vibrant, energetic color (pink/magenta) conveys a sense of modernity, creativity, and confidence. The overall design prioritizes immediate recognition and memorability over the conveyance of detailed information.
</details>
Answer Text Metrics | ⚫ ▲ ⚫ | ⚫⚫⚫ | ▼▼ ⚫ | ▼▼ ⚫ |
|
<details>
<summary>Images/icons/sources_color.png Details</summary>
### Visual Description
## Icon: File Folder with Documents
### Overview
The image is a simple, stylized icon depicting a file folder containing documents. It is a flat, two-dimensional graphic with no textual labels, data, or complex informational content. The design uses a single color (teal) on a light gray background.
### Components/Axes
* **Primary Element:** A file folder, shown in a frontal view with its tab on the left side.
* **Secondary Elements:** Three document sheets are visible, fanned out from within the folder. The front-most document is fully visible, while two others are partially visible behind it.
* **Color:** The entire icon is rendered in a solid teal or turquoise color (approximately hex #4ECDC4). The background is a uniform light gray.
* **Style:** Minimalist line art with rounded corners and consistent stroke weight. There are no gradients, shadows, or textures.
### Detailed Analysis
* **Folder:** The folder body is a rectangle with a cut-out tab on the top-left. A small, horizontal rectangular label area is centered on the lower half of the folder's front face.
* **Documents:** The front document is a vertical rectangle. It contains three horizontal lines of varying lengths, representing lines of text. The top line is the shortest, the middle line is the longest, and the bottom line is of medium length. The two documents behind are only partially visible as angled rectangles peeking out from the top and sides of the front document.
* **Spatial Arrangement:** The documents are positioned as if being pulled out or stored within the folder, creating a sense of depth through overlapping shapes. The entire composition is centered within the image frame.
### Key Observations
* The icon contains **no alphanumeric text, numbers, labels, or data points**.
* It is a symbolic representation, not a chart, diagram, or document containing extractable factual information.
* The design is generic and could represent concepts like "files," "documents," "folder," "archive," "storage," or "records" in a user interface.
### Interpretation
This image is a **symbolic icon**, not a data-bearing document. Its purpose is to convey a concept visually rather than to present specific information.
* **What it represents:** The icon is a universal signifier for digital or physical file storage. The folder represents a container or category, and the documents represent the items stored within it.
* **How elements relate:** The relationship is one of containment and organization. The folder (container) holds the documents (content). The fanned-out presentation suggests accessibility or a collection of multiple items.
* **Notable characteristics:** The clean, minimalist design ensures it is easily recognizable at small sizes, typical for use in software toolbars, file managers, or website navigation. The choice of a teal color may be part of a specific brand or UI color scheme, but the icon itself carries no intrinsic data or narrative beyond its symbolic function.
**Conclusion:** There is no factual data, textual content, or quantitative information to extract from this image. It is a graphical symbol whose meaning is derived from common cultural and digital conventions.
</details>
Sources Metrics | ▲ ▼ ⚫ | ▼ ⚫ ▼ | ⚫ ▼ ⚫ | ▲ ▼ ⚫ |
|
<details>
<summary>Images/icons/citation_color.png Details</summary>
### Visual Description
## Icon/Symbol: Abstract Blue Bracket Pair
### Overview
The image displays a simple, two-dimensional graphic icon composed of two symmetrical, outlined shapes on a solid, light gray background. The design is minimalist and geometric, resembling a pair of stylized brackets, parentheses, or the number "11" with modified, inward-facing terminals.
### Components/Axes
* **Primary Elements:** Two identical, vertically oriented shapes.
* **Shape Description:** Each shape is a continuous, thick blue outline forming a rectangular frame with a distinct inward notch or "bite" taken out of its inner side. The outer edges are straight and vertical. The top and bottom edges are straight and horizontal. The inner edge features a rectangular indentation centered vertically.
* **Color:** The outlines are a uniform, medium blue (approximate hex: #4A86E8). The background is a uniform, very light gray (approximate hex: #F0F0F0).
* **Layout & Spatial Grounding:** The two shapes are positioned side-by-side in the center of the image, separated by a narrow, vertical gap of empty space. They are perfect mirror images of each other. The left shape's notch faces right, and the right shape's notch faces left, creating a symmetrical, inward-facing composition.
### Detailed Analysis
* **No Textual Content:** The image contains no letters, numbers, labels, or written text of any kind.
* **No Data or Chart Elements:** This is not a chart, graph, or data visualization. There are no axes, legends, data points, or quantitative information to extract.
* **Geometric Properties:** The lines are of consistent thickness. The corners of the outer rectangles and the inner notches appear to be sharp, right angles. The overall composition is balanced and centered.
### Key Observations
* The design is purely symbolic and abstract.
* The symmetry and mirroring are precise.
* The color palette is limited to two flat colors with no gradients, shadows, or textures.
### Interpretation
The image is a graphic icon, not a document containing factual data. Its meaning is interpretive and context-dependent. Based on common visual language:
* It could represent a **pairing, matching, or connection** concept, given the two mirrored halves.
* The inward notches might symbolize **interlocking, fitting together, or a missing piece**.
* It strongly resembles a **stylized logo or UI element**, potentially for a software application, representing concepts like "merge," "split," "group," or "container."
* The form is also reminiscent of the number **"11"** or the **pause symbol (||)**, but with a distinct, modified design.
**Conclusion:** This image provides no extractable textual or numerical data. It is a visual symbol whose purpose would be defined by its surrounding context (e.g., as part of a logo, interface, or diagram).
</details>
Citation Metrics | ⚫⚫ | ⚫⚫ | ▼ ⚫ | ▼ ⚫ |
(a) Score Card Evaluation of GSE
<details>
<summary>x2.png Details</summary>
### Visual Description
## Horizontal Stacked Bar Chart: Answer Confidence Score (all queries)
### Overview
The image displays a horizontal stacked bar chart titled "Answer Confidence Score (all queries)". It compares four different AI-powered search or chat systems based on a metric of "confidence score," which is broken down into two distinct, color-coded components for each system. The chart presents numerical counts for each component within the bars.
### Components/Axes
* **Chart Title:** "Answer Confidence Score (all queries)" (centered at the top).
* **Y-Axis (Categories):** Lists four systems from top to bottom:
1. BingChat
2. SearchGPT
3. Perplexity
4. YouCom
* **X-Axis:** Represents a numerical count. There are no explicit axis labels, tick marks, or a scale provided. The values are instead embedded directly within the bar segments.
* **Bars:** Each system has a single horizontal bar composed of two stacked segments.
* **Left Segment:** Light blue color.
* **Right Segment:** Dark blue color.
* **Legend:** **No legend is visible in the provided image.** The meaning of the light blue and dark blue segments is not defined.
* **Other Elements:** A vertical dashed gray line runs from the top to the bottom of the chart area, positioned just to the right of the y-axis labels. A small, empty checkbox icon is visible in the top-right corner of the chart area.
### Detailed Analysis
The chart provides the following numerical data for each system, listed as [Light Blue Segment Value, Dark Blue Segment Value]:
1. **BingChat:**
* Light Blue: 98
* Dark Blue: 191
* **Total (Implied):** 289
* **Trend:** The dark blue segment is approximately twice the size of the light blue segment.
2. **SearchGPT:**
* Light Blue: 49
* Dark Blue: 247
* **Total (Implied):** 296
* **Trend:** The dark blue segment is significantly larger (about 5x) than the light blue segment.
3. **Perplexity:**
* Light Blue: 25
* Dark Blue: 270
* **Total (Implied):** 295
* **Trend:** The dark blue segment is overwhelmingly dominant, being over 10 times larger than the very small light blue segment.
4. **YouCom:**
* Light Blue: 137
* Dark Blue: 157
* **Total (Implied):** 294
* **Trend:** The two segments are relatively balanced, with the dark blue segment being only slightly larger.
### Key Observations
* **Total Query Count Consistency:** The implied total count for each system (sum of both segments) is very consistent, ranging from 289 to 296. This suggests the chart likely compares performance across the same or a very similar set of ~295 queries.
* **Dominant Segment Variation:** There is a dramatic variation in the proportion of the two confidence score components across systems. Perplexity shows the most extreme skew towards the dark blue component, while YouCom shows the most balanced distribution.
* **Inverse Relationship:** There appears to be an inverse relationship between the light blue and dark blue values across the first three systems (BingChat, SearchGPT, Perplexity). As the light blue value decreases, the dark blue value increases.
* **YouCom as an Outlier:** YouCom breaks the inverse trend observed in the first three systems. It has the highest light blue value (137) and a moderate dark blue value (157), resulting in a profile distinct from the others.
* **Missing Context:** The critical absence of a legend or axis label for the "confidence score" components makes definitive interpretation impossible. The vertical dashed line's purpose is also unclear without a label.
### Interpretation
This chart visualizes a comparative analysis of how four AI systems distribute their "answer confidence" across two unlabeled categories for a fixed set of queries.
* **What the Data Suggests:** The data implies a fundamental difference in how these systems generate or report confidence. Perplexity's profile (very low light blue, very high dark blue) suggests it is either highly confident or operates with a binary confidence model where most answers fall into one high-confidence bucket. In contrast, YouCom's balanced profile suggests a more graduated or uncertain confidence scoring system. BingChat and SearchGPT represent intermediate states.
* **Relationship Between Elements:** The systems are directly compared on the same metric. The consistent total count allows for a fair comparison of the *composition* of confidence scores, not just the absolute numbers. The vertical line may serve as a visual benchmark or separator, but its meaning is lost without a label.
* **Notable Anomalies:** The primary anomaly is the complete lack of semantic labels for the two confidence components. Without knowing what "light blue" and "dark blue" represent (e.g., "High/Medium Confidence," "Supported/Unsupported," "Certain/Uncertain"), the chart's practical insight is severely limited. It successfully shows a stark difference in distribution but fails to explain what that difference means. The chart is technically precise but informationally incomplete.
</details>
<details>
<summary>x3.png Details</summary>
### Visual Description
## Horizontal Stacked Bar Chart: Answer Confidence Score (debate queries)
### Overview
This image displays a horizontal stacked bar chart titled "Answer Confidence Score (debate queries)". It compares four different AI systems or chatbots—BingChat, SearchGPT, Perplexity, and YouCom—based on a numerical score related to confidence in their answers for debate-style queries. Each system's total score is represented by a bar divided into two primary colored segments (light blue and dark blue), with an additional small red segment present only for BingChat. The chart lacks an explicit legend, requiring inference about what the colors represent.
### Components/Axes
* **Title:** "Answer Confidence Score (debate queries)" located at the top center.
* **Y-Axis (Categories):** Lists the four systems vertically on the left side:
* BingChat
* SearchGPT
* Perplexity
* YouCom
* **X-Axis (Scale):** Not explicitly labeled with numbers or a title. A dashed vertical line runs from top to bottom, aligned with the left edge of the colored bars, likely serving as a baseline or zero point.
* **Bars:** Horizontal bars for each category, composed of colored segments.
* **Data Labels:** Numerical values are printed directly on the colored segments within each bar.
* **Legend:** **Not present in the image.** The meaning of the colors (light blue, dark blue, red) must be inferred from context.
### Detailed Analysis
The chart presents the following data for each system, reading from left to right along each bar:
1. **BingChat:**
* **Leftmost Segment (Red):** A small segment with no numerical label. Its value is approximately 5-10 based on visual comparison to the labeled segments.
* **Middle Segment (Light Blue):** Labeled **78**.
* **Right Segment (Dark Blue):** Labeled **83**.
* **Total Visual Length:** The sum of the labeled segments is 161, plus the small red segment.
2. **SearchGPT:**
* **Left Segment (Light Blue):** Labeled **37**.
* **Right Segment (Dark Blue):** Labeled **131**.
* **Total Visual Length:** 168.
3. **Perplexity:**
* **Left Segment (Light Blue):** A very small segment with no numerical label. Its value is approximately 10-15 based on visual comparison.
* **Right Segment (Dark Blue):** Labeled **160**.
* **Total Visual Length:** Approximately 170-175.
4. **YouCom:**
* **Left Segment (Light Blue):** Labeled **110**.
* **Right Segment (Dark Blue):** Labeled **56**.
* **Total Visual Length:** 166.
**Trend Verification:**
* **BingChat:** The dark blue segment (83) is slightly larger than the light blue segment (78).
* **SearchGPT:** The dark blue segment (131) is significantly larger than the light blue segment (37).
* **Perplexity:** The dark blue segment (160) is overwhelmingly dominant compared to the tiny light blue segment.
* **YouCom:** The light blue segment (110) is significantly larger than the dark blue segment (56), showing an inverse pattern to SearchGPT and Perplexity.
### Key Observations
1. **Missing Legend:** The most critical missing information is the legend defining the light blue, dark blue, and red segments. Common interpretations in confidence scoring could be "Low/Medium/High Confidence," "Incorrect/Partially Correct/Correct," or "No Answer/Partial Answer/Full Answer."
2. **Unique Element:** BingChat is the only system with a red segment, suggesting it is the only one categorized in that specific (and likely negative) metric.
3. **Dominant Patterns:** Perplexity shows the highest single-segment value (160 for dark blue) and the most skewed distribution. YouCom is the only system where the light blue segment is larger than the dark blue one.
4. **Total Score Range:** The total visual length of the bars (sum of segments) is relatively consistent across all four systems, ranging approximately from 161 to 175. This suggests the chart may be showing a breakdown of a fixed total number of queries or a normalized score.
### Interpretation
This chart visually compares the performance profile of four AI systems on debate queries, based on an unstated confidence metric. The data suggests fundamentally different behaviors:
* **Perplexity** appears to operate with very high confidence (large dark blue segment) on the vast majority of queries, with minimal instances of the light blue category.
* **SearchGPT** shows a similar but less extreme pattern to Perplexity, with a strong lean towards the dark blue category.
* **YouCom** exhibits the opposite tendency, scoring higher in the light blue category than the dark blue one.
* **BingChat** has a more balanced distribution between light and dark blue but is uniquely flagged with the red category, which could indicate a higher rate of failures, refusals, or low-confidence responses not seen in the others.
**Without the legend, the precise meaning is ambiguous.** However, the chart effectively demonstrates that these systems have distinct "confidence signatures." The investigation would next require the legend to decode whether higher values in dark blue are desirable (e.g., "High Confidence/Correct") or undesirable (e.g., "Overconfident/Wrong"). The presence of the red segment only on BingChat is a significant anomaly warranting further scrutiny into its specific failure modes for debate queries.
</details>
<details>
<summary>x4.png Details</summary>
### Visual Description
## Horizontal Stacked Bar Chart: Answer Confidence Score (expertise queries)
### Overview
This image displays a horizontal stacked bar chart comparing the confidence levels of responses from four different AI systems (BingChat, SearchGPT, Perplexity, YouCom) to a set of "expertise queries." The chart quantifies the number of responses falling into five distinct confidence categories.
### Components/Axes
* **Chart Title:** "Answer Confidence Score (expertise queries)" (Top center).
* **Vertical Axis (Y-axis):** Lists the four AI systems being compared. From top to bottom: "BingChat", "SearchGPT", "Perplexity", "YouCom".
* **Horizontal Axis (X-axis):** Labeled "Number of Responses" (Bottom center). The axis has a dashed vertical line at the zero point, aligned with the start of each bar.
* **Legend:** Located at the bottom of the chart, centered. It defines five color-coded categories:
* **Strongly Not Confident:** Dark red square.
* **Not Confident:** Light red/salmon square.
* **Neutral:** Light gray square.
* **Confident:** Light blue square.
* **Strongly Confident:** Dark blue square.
### Detailed Analysis
Each horizontal bar represents the total responses for one AI system, segmented by confidence level. The numerical value for each major segment is labeled directly on the bar.
**1. BingChat (Top Bar):**
* **Confident (Light Blue):** 20 responses.
* **Strongly Confident (Dark Blue):** 108 responses.
* **Other Segments:** A very thin, unlabeled gray segment (Neutral) is visible between the "Confident" and "Strongly Confident" segments. No red segments are visible.
**2. SearchGPT (Second Bar):**
* **Confident (Light Blue):** 12 responses.
* **Strongly Confident (Dark Blue):** 116 responses.
* **Other Segments:** A very thin, unlabeled gray segment (Neutral) is visible. No red segments are visible.
**3. Perplexity (Third Bar):**
* **Strongly Not Confident (Dark Red):** A very small, unlabeled segment is visible at the far left.
* **Confident (Light Blue):** 17 responses.
* **Strongly Confident (Dark Blue):** 110 responses.
* **Other Segments:** A thin, unlabeled gray segment (Neutral) is visible.
**4. YouCom (Bottom Bar):**
* **Confident (Light Blue):** 27 responses.
* **Strongly Confident (Dark Blue):** 101 responses.
* **Other Segments:** A thin, unlabeled gray segment (Neutral) is visible. No red segments are visible.
**Trend Verification:**
* For all four systems, the "Strongly Confident" (dark blue) segment is the largest component of the bar, indicating it is the most frequent confidence rating.
* The "Confident" (light blue) segment is the second-largest for all systems.
* The combined "Confident" and "Strongly Confident" segments constitute the vast majority of responses for each system.
### Key Observations
1. **Dominance of High Confidence:** Across all four AI systems, the overwhelming majority of responses are categorized as either "Confident" or "Strongly Confident."
2. **Highest "Strongly Confident" Count:** SearchGPT has the highest number of "Strongly Confident" responses (116).
3. **Highest "Confident" Count:** YouCom has the highest number of "Confident" responses (27).
4. **Lowest "Confident" Count:** SearchGPT has the lowest number of "Confident" responses (12).
5. **Presence of Low Confidence:** Perplexity is the only system with a visible "Strongly Not Confident" (dark red) segment, though it is very small.
6. **Neutral Responses:** All four systems show a very small, unlabeled "Neutral" (gray) segment, suggesting a minimal number of responses were rated as neutral.
### Interpretation
The data suggests that when responding to queries requiring expertise, these four AI systems predominantly project high confidence. The "Strongly Confident" category is the modal response for each, indicating a tendency towards assertive answers.
The variation lies in the distribution between "Confident" and "Strongly Confident." SearchGPT shows the most polarized confidence, with the highest "Strongly Confident" count but the lowest "Confident" count. YouCom shows a slightly more balanced distribution between the two top categories.
The presence of a "Strongly Not Confident" segment only for Perplexity is a notable outlier. It indicates that, for this specific set of expertise queries, Perplexity's responses were occasionally flagged with the lowest confidence rating, a pattern not observed for the other systems in this dataset. The universally small "Neutral" segment implies that the systems rarely produce responses that are ambiguous or non-committal in their confidence.
**Important Note:** The exact numerical values for the "Neutral," "Not Confident," and "Strongly Not Confident" segments are not provided on the chart. Their sizes are inferred visually and are approximate. The analysis is based on the labeled values for the "Confident" and "Strongly Confident" segments and the visual representation of the other categories.
</details>
(b) Confidence Score Distribution
Figure 2: Quantitative Evaluation of three GSE – You.com, BingChat, and Perplexity – based on the eight metrics of the DeepTrace framework: metric report, color-coded for ▲ acceptable, ⚫ borderline, and ▼ problematic performance. Figure (b) plots distributions of answer confidence.
In the following section, we audit publicly available deep research agents and GSE to assess their societal impact. These systems, often referred to as AIaaS (AI as a Service) (Lins et al., 2021), are marketed as ready-to-use models requiring no prior expertise. To focus on publicly accessible systems, we selected the web search adn deep research capabilities of Perplexity, Bing Copilot, GPT (4.5/5) and YouChat for evaluation.
## 4 Results
Figure 2 (GSE) and Table 1 (Deep Research) show the results of the metrics-based evaluation on the DeepTrace corpus as of August 27, 2025. In the Table on the left, numerical values are assigned a color based on whether the score reflects an ▲ acceptable, ⚫ borderline, and ▼ problematic performance. Thresholds for the colors are listed in Table 2 with the explanation of the threshold in Appendix B based on the qualitative inputs obtained from Narayanan Venkit et al. (2025).
| Basic Statistics | Deep Research Agents GPT-5(DR) | YouChat(DR) | GPT-5(S) | PPLX(DR) | Copilot (TD) | Gemini (DR) |
| --- | --- | --- | --- | --- | --- | --- |
| Number of Sources | 18.3 | 57.2 | 13.5 | 7.7 | 3.6 | 33.2 |
| Number of Statements | 141.6 | 52.7 | 34.9 | 30.1 | 36.7 | 23.9 |
| # Citations / Statement | 1.4 | 0.8 | 0.4 | 0.2 | 0.3 | 0.2 |
|
<details>
<summary>Images/icons/answer_text_color.png Details</summary>
### Visual Description
## Icon/Symbol: Stylized Letter "T" with Corner Elements
### Overview
The image is a simple, two-color graphic featuring a large, stylized capital letter "T" centered on a light gray background. The "T" is framed by four L-shaped corner elements, creating a bordered or focused composition. The design is geometric, minimalist, and uses a single accent color.
### Components/Axes
* **Central Element:** A large, bold, sans-serif capital letter "T". It is rendered in a bright pink/magenta color.
* **Framing Elements:** Four L-shaped corner brackets, also in the same pink/magenta color. They are positioned at the extreme top-left, top-right, bottom-left, and bottom-right of the image, creating a visual frame around the central "T".
* **Background:** A solid, light gray or off-white color.
* **Textual Content:** The only textual element is the letter "T". No other labels, titles, legends, or data points are present.
### Detailed Analysis
* **Color Palette:** The image uses a two-color scheme.
* **Primary/Accent Color:** A vibrant pink/magenta (approximate hex: #E91E63 or similar).
* **Background Color:** A very light gray (approximate hex: #F5F5F5 or similar).
* **Spatial Layout:**
* The letter "T" is perfectly centered both horizontally and vertically.
* The four corner L-shapes are aligned to the edges of the image canvas, creating a symmetrical border. The top-left and top-right corners are oriented with the horizontal bar on top. The bottom-left and bottom-right corners are oriented with the horizontal bar on the bottom.
* **Design Style:** The design is flat, with no gradients, shadows, or textures. It has a clean, modern, and iconic feel, suitable for use as a logo, app icon, or symbol.
### Key Observations
1. **Simplicity:** The image contains no complex data, charts, or extended text. Its information is purely symbolic and aesthetic.
2. **Symmetry:** The composition is highly symmetrical along both the vertical and horizontal axes.
3. **Color Contrast:** There is strong visual contrast between the bright pink elements and the light gray background, ensuring high visibility and legibility of the "T".
4. **Absence of Data:** This is not a chart, diagram, or document. It does not contain trends, numerical values, flow directions, or factual data beyond the visual representation of the letter "T".
### Interpretation
This image functions as a **visual identifier or symbol**. The prominent, framed letter "T" suggests it could be:
* A **logo** for a brand, company, or product whose name begins with "T".
* An **app icon** for a mobile application.
* A **section marker** or decorative initial cap in a digital or print layout.
* A **stylized button** or interface element.
The framing corners draw the viewer's eye directly to the central character, emphasizing its importance. The choice of a bold, modern typeface and a vibrant, energetic color (pink/magenta) conveys a sense of modernity, creativity, and confidence. The overall design prioritizes immediate recognition and memorability over the conveyance of detailed information.
</details>
Answer Text Metrics | | | | | | |
| %One-Sided Answer | 54.67 ▼ | 63.1 ▼ | 69.7 ▼ | 63.1 ▼ | 94.8 ▼ | 80.1 ▼ |
| %Overconfident Answer | 15.2 ▲ | 19.6 ▲ | 16.4 ▲ | 5.6 ▲ | 0.0 ▲ | 11.2 ▲ |
| %Relevant Statements | 87.5 ⚫ | 45.5 ▼ | 41.1 ▼ | 22.5 ▼ | 13.2 ▼ | 12.4 ▼ |
|
<details>
<summary>Images/icons/sources_color.png Details</summary>
### Visual Description
## Icon: File Folder with Documents
### Overview
The image is a simple, stylized icon depicting a file folder containing documents. It is a flat, two-dimensional graphic with no textual labels, data, or complex informational content. The design uses a single color (teal) on a light gray background.
### Components/Axes
* **Primary Element:** A file folder, shown in a frontal view with its tab on the left side.
* **Secondary Elements:** Three document sheets are visible, fanned out from within the folder. The front-most document is fully visible, while two others are partially visible behind it.
* **Color:** The entire icon is rendered in a solid teal or turquoise color (approximately hex #4ECDC4). The background is a uniform light gray.
* **Style:** Minimalist line art with rounded corners and consistent stroke weight. There are no gradients, shadows, or textures.
### Detailed Analysis
* **Folder:** The folder body is a rectangle with a cut-out tab on the top-left. A small, horizontal rectangular label area is centered on the lower half of the folder's front face.
* **Documents:** The front document is a vertical rectangle. It contains three horizontal lines of varying lengths, representing lines of text. The top line is the shortest, the middle line is the longest, and the bottom line is of medium length. The two documents behind are only partially visible as angled rectangles peeking out from the top and sides of the front document.
* **Spatial Arrangement:** The documents are positioned as if being pulled out or stored within the folder, creating a sense of depth through overlapping shapes. The entire composition is centered within the image frame.
### Key Observations
* The icon contains **no alphanumeric text, numbers, labels, or data points**.
* It is a symbolic representation, not a chart, diagram, or document containing extractable factual information.
* The design is generic and could represent concepts like "files," "documents," "folder," "archive," "storage," or "records" in a user interface.
### Interpretation
This image is a **symbolic icon**, not a data-bearing document. Its purpose is to convey a concept visually rather than to present specific information.
* **What it represents:** The icon is a universal signifier for digital or physical file storage. The folder represents a container or category, and the documents represent the items stored within it.
* **How elements relate:** The relationship is one of containment and organization. The folder (container) holds the documents (content). The fanned-out presentation suggests accessibility or a collection of multiple items.
* **Notable characteristics:** The clean, minimalist design ensures it is easily recognizable at small sizes, typical for use in software toolbars, file managers, or website navigation. The choice of a teal color may be part of a specific brand or UI color scheme, but the icon itself carries no intrinsic data or narrative beyond its symbolic function.
**Conclusion:** There is no factual data, textual content, or quantitative information to extract from this image. It is a graphical symbol whose meaning is derived from common cultural and digital conventions.
</details>
Sources Metrics | | | | | | |
| %Uncited Sources | 0.0 ▲ | 66.3 ▼ | 51.7 ▼ | 57.5 ▼ | 32.6 ▼ | 14.5 ▼ |
| %Unsupported Statements | 12.5 ⚫ | 74.6 ▼ | 58.9 ▼ | 97.5 ▼ | 90.2 ▼ | 53.6 ▼ |
| %Source Necessity | 87.5 ▲ | 63.2 ⚫ | 32.8 ▼ | 5.5 ▼ | 31.2 ▼ | 33.1 ▼ |
|
<details>
<summary>Images/icons/citation_color.png Details</summary>
### Visual Description
## Icon/Symbol: Abstract Blue Bracket Pair
### Overview
The image displays a simple, two-dimensional graphic icon composed of two symmetrical, outlined shapes on a solid, light gray background. The design is minimalist and geometric, resembling a pair of stylized brackets, parentheses, or the number "11" with modified, inward-facing terminals.
### Components/Axes
* **Primary Elements:** Two identical, vertically oriented shapes.
* **Shape Description:** Each shape is a continuous, thick blue outline forming a rectangular frame with a distinct inward notch or "bite" taken out of its inner side. The outer edges are straight and vertical. The top and bottom edges are straight and horizontal. The inner edge features a rectangular indentation centered vertically.
* **Color:** The outlines are a uniform, medium blue (approximate hex: #4A86E8). The background is a uniform, very light gray (approximate hex: #F0F0F0).
* **Layout & Spatial Grounding:** The two shapes are positioned side-by-side in the center of the image, separated by a narrow, vertical gap of empty space. They are perfect mirror images of each other. The left shape's notch faces right, and the right shape's notch faces left, creating a symmetrical, inward-facing composition.
### Detailed Analysis
* **No Textual Content:** The image contains no letters, numbers, labels, or written text of any kind.
* **No Data or Chart Elements:** This is not a chart, graph, or data visualization. There are no axes, legends, data points, or quantitative information to extract.
* **Geometric Properties:** The lines are of consistent thickness. The corners of the outer rectangles and the inner notches appear to be sharp, right angles. The overall composition is balanced and centered.
### Key Observations
* The design is purely symbolic and abstract.
* The symmetry and mirroring are precise.
* The color palette is limited to two flat colors with no gradients, shadows, or textures.
### Interpretation
The image is a graphic icon, not a document containing factual data. Its meaning is interpretive and context-dependent. Based on common visual language:
* It could represent a **pairing, matching, or connection** concept, given the two mirrored halves.
* The inward notches might symbolize **interlocking, fitting together, or a missing piece**.
* It strongly resembles a **stylized logo or UI element**, potentially for a software application, representing concepts like "merge," "split," "group," or "container."
* The form is also reminiscent of the number **"11"** or the **pause symbol (||)**, but with a distinct, modified design.
**Conclusion:** This image provides no extractable textual or numerical data. It is a visual symbol whose purpose would be defined by its surrounding context (e.g., as part of a logo, interface, or diagram).
</details>
Citation Metrics | | | | | | |
| %Citation Accuracy | 79.1 ⚫ | 72.3 ⚫ | 31.4 ▼ | 58.0 ⚫ | 62.1 ⚫ | 50.3 ⚫ |
| %Citation Thoroughness | 87.5 ▲ | 83.5 ▲ | 17.9 ▼ | 9.1 ▼ | 13.2 ▼ | 27.1 ⚫ |
| DeepTrace Eval Score Card | | | | | | |
|
<details>
<summary>Images/icons/answer_text_color.png Details</summary>
### Visual Description
## Icon/Symbol: Stylized Letter "T" with Corner Elements
### Overview
The image is a simple, two-color graphic featuring a large, stylized capital letter "T" centered on a light gray background. The "T" is framed by four L-shaped corner elements, creating a bordered or focused composition. The design is geometric, minimalist, and uses a single accent color.
### Components/Axes
* **Central Element:** A large, bold, sans-serif capital letter "T". It is rendered in a bright pink/magenta color.
* **Framing Elements:** Four L-shaped corner brackets, also in the same pink/magenta color. They are positioned at the extreme top-left, top-right, bottom-left, and bottom-right of the image, creating a visual frame around the central "T".
* **Background:** A solid, light gray or off-white color.
* **Textual Content:** The only textual element is the letter "T". No other labels, titles, legends, or data points are present.
### Detailed Analysis
* **Color Palette:** The image uses a two-color scheme.
* **Primary/Accent Color:** A vibrant pink/magenta (approximate hex: #E91E63 or similar).
* **Background Color:** A very light gray (approximate hex: #F5F5F5 or similar).
* **Spatial Layout:**
* The letter "T" is perfectly centered both horizontally and vertically.
* The four corner L-shapes are aligned to the edges of the image canvas, creating a symmetrical border. The top-left and top-right corners are oriented with the horizontal bar on top. The bottom-left and bottom-right corners are oriented with the horizontal bar on the bottom.
* **Design Style:** The design is flat, with no gradients, shadows, or textures. It has a clean, modern, and iconic feel, suitable for use as a logo, app icon, or symbol.
### Key Observations
1. **Simplicity:** The image contains no complex data, charts, or extended text. Its information is purely symbolic and aesthetic.
2. **Symmetry:** The composition is highly symmetrical along both the vertical and horizontal axes.
3. **Color Contrast:** There is strong visual contrast between the bright pink elements and the light gray background, ensuring high visibility and legibility of the "T".
4. **Absence of Data:** This is not a chart, diagram, or document. It does not contain trends, numerical values, flow directions, or factual data beyond the visual representation of the letter "T".
### Interpretation
This image functions as a **visual identifier or symbol**. The prominent, framed letter "T" suggests it could be:
* A **logo** for a brand, company, or product whose name begins with "T".
* An **app icon** for a mobile application.
* A **section marker** or decorative initial cap in a digital or print layout.
* A **stylized button** or interface element.
The framing corners draw the viewer's eye directly to the central character, emphasizing its importance. The choice of a bold, modern typeface and a vibrant, energetic color (pink/magenta) conveys a sense of modernity, creativity, and confidence. The overall design prioritizes immediate recognition and memorability over the conveyance of detailed information.
</details>
Answer Text Metrics | ▼ ▲ ⚫ | ▼ ▲ ▼ | ▼ ▲ ▼ | ▼ ▲ ▼ | ▼ ▲ ▼ | ▼ ▲ ▼ |
|
<details>
<summary>Images/icons/sources_color.png Details</summary>
### Visual Description
## Icon: File Folder with Documents
### Overview
The image is a simple, stylized icon depicting a file folder containing documents. It is a flat, two-dimensional graphic with no textual labels, data, or complex informational content. The design uses a single color (teal) on a light gray background.
### Components/Axes
* **Primary Element:** A file folder, shown in a frontal view with its tab on the left side.
* **Secondary Elements:** Three document sheets are visible, fanned out from within the folder. The front-most document is fully visible, while two others are partially visible behind it.
* **Color:** The entire icon is rendered in a solid teal or turquoise color (approximately hex #4ECDC4). The background is a uniform light gray.
* **Style:** Minimalist line art with rounded corners and consistent stroke weight. There are no gradients, shadows, or textures.
### Detailed Analysis
* **Folder:** The folder body is a rectangle with a cut-out tab on the top-left. A small, horizontal rectangular label area is centered on the lower half of the folder's front face.
* **Documents:** The front document is a vertical rectangle. It contains three horizontal lines of varying lengths, representing lines of text. The top line is the shortest, the middle line is the longest, and the bottom line is of medium length. The two documents behind are only partially visible as angled rectangles peeking out from the top and sides of the front document.
* **Spatial Arrangement:** The documents are positioned as if being pulled out or stored within the folder, creating a sense of depth through overlapping shapes. The entire composition is centered within the image frame.
### Key Observations
* The icon contains **no alphanumeric text, numbers, labels, or data points**.
* It is a symbolic representation, not a chart, diagram, or document containing extractable factual information.
* The design is generic and could represent concepts like "files," "documents," "folder," "archive," "storage," or "records" in a user interface.
### Interpretation
This image is a **symbolic icon**, not a data-bearing document. Its purpose is to convey a concept visually rather than to present specific information.
* **What it represents:** The icon is a universal signifier for digital or physical file storage. The folder represents a container or category, and the documents represent the items stored within it.
* **How elements relate:** The relationship is one of containment and organization. The folder (container) holds the documents (content). The fanned-out presentation suggests accessibility or a collection of multiple items.
* **Notable characteristics:** The clean, minimalist design ensures it is easily recognizable at small sizes, typical for use in software toolbars, file managers, or website navigation. The choice of a teal color may be part of a specific brand or UI color scheme, but the icon itself carries no intrinsic data or narrative beyond its symbolic function.
**Conclusion:** There is no factual data, textual content, or quantitative information to extract from this image. It is a graphical symbol whose meaning is derived from common cultural and digital conventions.
</details>
Sources Metrics | ▲ ⚫ ▲ | ▼▼ ⚫ | ▼▼▼ | ▼▼▼ | ▼▼▼ | ▼▼▼ |
|
<details>
<summary>Images/icons/citation_color.png Details</summary>
### Visual Description
## Icon/Symbol: Abstract Blue Bracket Pair
### Overview
The image displays a simple, two-dimensional graphic icon composed of two symmetrical, outlined shapes on a solid, light gray background. The design is minimalist and geometric, resembling a pair of stylized brackets, parentheses, or the number "11" with modified, inward-facing terminals.
### Components/Axes
* **Primary Elements:** Two identical, vertically oriented shapes.
* **Shape Description:** Each shape is a continuous, thick blue outline forming a rectangular frame with a distinct inward notch or "bite" taken out of its inner side. The outer edges are straight and vertical. The top and bottom edges are straight and horizontal. The inner edge features a rectangular indentation centered vertically.
* **Color:** The outlines are a uniform, medium blue (approximate hex: #4A86E8). The background is a uniform, very light gray (approximate hex: #F0F0F0).
* **Layout & Spatial Grounding:** The two shapes are positioned side-by-side in the center of the image, separated by a narrow, vertical gap of empty space. They are perfect mirror images of each other. The left shape's notch faces right, and the right shape's notch faces left, creating a symmetrical, inward-facing composition.
### Detailed Analysis
* **No Textual Content:** The image contains no letters, numbers, labels, or written text of any kind.
* **No Data or Chart Elements:** This is not a chart, graph, or data visualization. There are no axes, legends, data points, or quantitative information to extract.
* **Geometric Properties:** The lines are of consistent thickness. The corners of the outer rectangles and the inner notches appear to be sharp, right angles. The overall composition is balanced and centered.
### Key Observations
* The design is purely symbolic and abstract.
* The symmetry and mirroring are precise.
* The color palette is limited to two flat colors with no gradients, shadows, or textures.
### Interpretation
The image is a graphic icon, not a document containing factual data. Its meaning is interpretive and context-dependent. Based on common visual language:
* It could represent a **pairing, matching, or connection** concept, given the two mirrored halves.
* The inward notches might symbolize **interlocking, fitting together, or a missing piece**.
* It strongly resembles a **stylized logo or UI element**, potentially for a software application, representing concepts like "merge," "split," "group," or "container."
* The form is also reminiscent of the number **"11"** or the **pause symbol (||)**, but with a distinct, modified design.
**Conclusion:** This image provides no extractable textual or numerical data. It is a visual symbol whose purpose would be defined by its surrounding context (e.g., as part of a logo, interface, or diagram).
</details>
Citation Metrics | ▲▲ | ⚫ ▲ | ▼▼ | ⚫ ▼ | ⚫ ▼ | ⚫⚫ |
Table 1: DeepTrace results for our Deep Research (DR) based models: GPT-5, YouChat, Perplexity (PPLX), Copilot Think Deeper, and Gemini. This table also includes GPT-5 Web Search (S) setting. Metrics evaluated according to DeepTrace thresholds: ▲ acceptable, ⚫ borderline, ▼ problematic. These results show that deep research agents still struggle with unsupported statements, poor source usage, and unreliable citation practices across models.
Generative Search Engines.
As shown in Figure 2, for answer text metrics, one-sidedness remains an issue (50–80%), with Perplexity performing worst, generating one-sided responses in over 83% of debate queries despite producing the longest answers (18.8 statements per response on average). Confidence calibration also varies where BingChat and You.com reduce confidence when addressing debate queries, whereas Perplexity maintains uniformly high confidence (90%+ very confident), resulting in overconfident yet one-sided answers on politically or socially contentious prompts. On relevance, GSE models perform comparably (75–85% relevant statements), which indicates better alignment with user queries relative to their DR counterparts. For source metrics, BingChat exemplifies the quantity without quality trade-off where it lists more sources on average (4.0), yet over a third remain uncited and only about half are necessary. You.com and Perplexity list slightly fewer sources (3.4–3.5) but still struggle with unsupported claims (23–47%). Finally, on citation metrics, all three engines show relatively low citation accuracy (40–68%), with frequent misattribution. Even when a supporting source exists, models often cite an irrelevant one, preventing users from verifying factual validity. Citation thoroughness is also limited, with engines typically citing only a subset of available supporting evidence. Our results therefore align with the findings of Narayanan Venkit (2023) where such models can be responsible in generatic echo chambers with very little automony towards the user to search and select the articles that they prefer.
Deep Research Agents.
In context of answer text, Table 1 shows that DR modes do not eliminate one-sidedness where rates remain high across the board (54.7–94.8%). Appendix D shows how GPT-5 deep research answers one sided answers for questions framed pro and con the same debate, without providing generalized coverage. This showcases sycophantic behavior of aligning only with the users perspective, causing potential echo chambers to search. Overconfidence is consistently low across DR engines ( $<$ 20%), indicating that calibration of language hedging is one relative strength of this pipeline. On relevance, however, performance is uneven where GPT-5(DR) attains borderline results (87.5%), while all other engines fall below 50%, including Gemini(DR) at just 12.4%. This suggests that verbosity or sourcing breadth does not translate to actually answering the user query. Turning to sources metrics, GPT-5(DR) remains the strongest with 0% uncited sources, only 12.5% unsupported statements, and 87.5% source necessity. By contrast, YouChat(DR), PPLX(DR), Copilot(DR), and Gemini(DR) all fare poorly, with unsupported rates ranging from 53.6% (Gemini) to 97.5% (PPLX). Gemini(DR) in particular includes 14.5% uncited sources and only one-third (33.1%) of its sources being necessary, reflecting inefficient citation usage. For citation metrics, GPT-5(DR) and YouChat(DR) again stand out with high citation thoroughness (87.5% and 83.5% respectively), although their citation accuracy has dropped to the borderline range (79.1% and 72.3%). Gemini(DR) demonstrates weak citation performance: only 40.3% citation accuracy (problematic) and 27.1% thoroughness (borderline). PPLX(DR) and Copilot(DR) also show poor grounding, with citation accuracies between 58–62%.
Taken together, the results reveal that neither GSE nor deep research agents deliver uniformly reliable outputs across DeepTRACE’s dimensions. GSEs excel at producing concise, relevant answers but fail at balanced perspective-taking, confidence calibration, and factual support. Deep research agents, by contrast, improve balance and citation correctness, but at the cost of overwhelming verbosity, low relevance, and huge unsupported claims. Our results show that more sources and longer answers do not translate into reliability. Over-citation (as in YouChat(DR)) leads to ‘search fatigue’ for users, while under-grounded verbose texts (as in Perplexity(DR)) erodes trust. At the same time, carefully calibrated systems (as with GPT-5(DR)) demonstrate that near-ideal reliability across multiple dimensions is achievable.
## 5 Discussion and Conclusion
Our work introduced DeepTRACE, a sociotechnically grounded framework for auditing generative search engines (GSEs) and deep research agents (DRs). By translating community-identified failure cases into measurable dimensions, our approach evaluates not just isolated components but the end-to-end reliability of these systems across balance, factual support, and citation integrity.
Our evaluation demonstrates that current public systems fall short of their promise to deliver trustworthy, source-grounded synthesis. Generative search engines tend to produce concise and relevant answers but consistently exhibit one-sided framing and frequent overconfidence, particularly on debate-style queries. Deep research agents, while reducing overconfidence and improving citation thoroughness, often overwhelm users with verbose, low-relevance responses and large fractions of unsupported claims. Importantly, our findings show that increasing the number of sources or length of responses does not reliably improve grounding or accuracy; instead, it can exacerbate user fatigue and obscure transparency.
Citation practices remain a persistent weakness across both classes of systems. Many citations are either inaccurate or incomplete, with some models listing sources that are never cited or irrelevant to their claims. This creates a misleading impression of evidential rigor while undermining user trust. Metrics such as Source Necessity and Citation Accuracy highlight that merely retrieving more sources does not equate to stronger factual grounding, echoing user concerns about opacity and accountability.
Taken together, these results point to a central tension: GSEs optimize for summarization and relevance at the expense of balance and factual support, whereas DRs optimize for breadth and thoroughness at the expense of clarity and reliability. Neither approach, in its current form, adequately meets the sociotechnical requirements of safe, effective, and trustworthy information access. However, our findings also suggest that calibrated systems—such as GPT-5(DR), which demonstrated strong performance across multiple metrics—illustrate that more reliable designs are achievable.
By situating evaluation within real user interactions, DeepTRACE advances auditing as both an analytic tool and a design accountability mechanism. Beyond technical performance, it highlights the social risks of echo chambers, sycophancy, and reduced user autonomy in search. Future work should extend this evaluation to multimodal and interface-level factors, as well as integrate human-in-the-loop validation in high-stakes domains. In doing so, DeepTRACE can guide the development of next-generation research agents that balance efficiency with epistemic interactions.
## References
- Bender (2024) Emily M Bender. Resisting dehumanization in the age of “ai”. Current Directions in Psychological Science, 33(2):114–120, 2024.
- Bosse et al. (2025) Nikos I Bosse, Jon Evans, Robert G Gambee, Daniel Hnyk, Peter Mühlbacher, Lawrence Phillips, Dan Schwarz, Jack Wildman, et al. Deep research bench: Evaluating ai web research agents. arXiv preprint arXiv:2506.06287, 2025.
- Chauhan et al. (2024) Pratyush Chauhan, Rahul Kumar Sahani, Soham Datta, Ali Qadir, Manish Raj, and Mohd Mohsin Ali. Evaluating top-k rag-based approach for game review generation. In 2024 IEEE International Conference on Computing, Power and Communication Technologies (IC2PCT), volume 5, pp. 258–263. IEEE, 2024.
- Chen et al. (2025) Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al. Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent. arXiv preprint arXiv:2508.06600, 2025.
- Cooper & Foster (1971) Robert Cooper and Michael Foster. Sociotechnical systems. American Psychologist, 26(5):467, 1971.
- Dolata et al. (2022) Mateusz Dolata, Stefan Feuerriegel, and Gerhard Schwabe. A sociotechnical view of algorithmic fairness. Information Systems Journal, 32(4):754–818, 2022.
- Du et al. (2025) Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763, 2025.
- Ehsan et al. (2024) Upol Ehsan, Samir Passi, Q Vera Liao, Larry Chan, I-Hsiang Lee, Michael Muller, and Mark O Riedl. The who in xai: How ai background shapes perceptions of ai explanations. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–32, 2024.
- Es et al. (2023) Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217, 2023.
- Es et al. (2024) Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pp. 150–158, 2024.
- Ferrara (2024) Emilio Ferrara. Genai against humanity: Nefarious applications of generative artificial intelligence and large language models. Journal of Computational Social Science, pp. 1–21, 2024.
- Gupta et al. (2024) Aman Gupta, Anup Shirgaonkar, Angels de Luis Balaguer, Bruno Silva, Daniel Holstein, Dawei Li, Jennifer Marsman, Leonardo O Nunes, Mahsa Rouzbahman, Morris Sharp, et al. Rag vs fine-tuning: Pipelines, tradeoffs, and a case study on agriculture. arXiv preprint arXiv:2401.08406, 2024.
- Hopcroft & Karp (1973) John E Hopcroft and Richard M Karp. An nˆ5/2 algorithm for maximum matchings in bipartite graphs. SIAM Journal on computing, 2(4):225–231, 1973.
- Huang et al. (2024) Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, and Heng Ji. Do LVLMs understand charts? analyzing and correcting factual errors in chart captioning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 730–749, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.41. URL https://aclanthology.org/2024.findings-acl.41/.
- Huang et al. (2023) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232, 2023.
- Huang et al. (2025) Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, Jianye Hao, et al. Deep research agents: A systematic examination and roadmap. arXiv preprint arXiv:2506.18096, 2025.
- Izacard & Grave (2021) Gautier Izacard and Édouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874–880, 2021.
- Jeong et al. (2024) Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 7029–7043, 2024.
- Kaur et al. (2024) Navreet Kaur, Monojit Choudhury, and Danish Pruthi. Evaluating large language models for health-related queries with presuppositions. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 14308–14331, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.findings-acl.850.
- Kim et al. (2024) Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. Fables: Evaluating faithfulness and content selection in book-length summarization. arXiv preprint arXiv:2404.01261, 2024.
- Laban et al. (2022) Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. Summac: Re-visiting nli-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163–177, 2022.
- Laban et al. (2023a) Philippe Laban, Wojciech Kryściński, Divyansh Agarwal, Alexander R Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. Llms as factual reasoners: Insights from existing benchmarks and beyond. arXiv preprint arXiv:2305.14540, 2023a.
- Laban et al. (2023b) Philippe Laban, Lidiya Murakhovs’ ka, Caiming Xiong, and Chien-Sheng Wu. Are you sure? challenging llms leads to performance drops in the flipflop experiment. arXiv preprint arXiv:2311.08596, 2023b.
- Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–9474, 2020.
- Lins et al. (2021) Sebastian Lins, Konstantin D Pandl, Heiner Teigeler, Scott Thiebes, Calvin Bayer, and Ali Sunyaev. Artificial intelligence as a service: classification and research directions. Business & Information Systems Engineering, 63:441–456, 2021.
- Liu et al. (2023) Nelson F Liu, Tianyi Zhang, and Percy Liang. Evaluating verifiability in generative search engines. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7001–7025, 2023.
- Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.
- Narayanan Venkit (2023) Pranav Narayanan Venkit. Towards a holistic approach: Understanding sociodemographic biases in nlp models using an interdisciplinary lens. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pp. 1004–1005, 2023.
- Narayanan Venkit et al. (2025) Pranav Narayanan Venkit, Philippe Laban, Yilun Zhou, Yixin Mao, and Chien-Sheng Wu. Search engines in the ai era: A qualitative understanding to the false promise of factual and verifiable source-cited responses in llm-based search. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pp. 1325–1340, 2025.
- Nathani et al. (2025) Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, et al. Mlgym: A new framework and benchmark for advancing ai research agents. arXiv preprint arXiv:2502.14499, 2025.
- Pulapaka et al. (2024) Sanjeev Pulapaka, Srinath Godavarthi, and Dr Sherry Ding. Genai and the public sector. In Empowering the Public Sector with Generative AI: From Strategy and Design to Real-World Applications, pp. 31–43. Springer, 2024.
- Qiu et al. (2024) Haoyi Qiu, Kung-Hsiang Huang, Jingnong Qu, and Nanyun Peng. AMRFact: Enhancing summarization factuality evaluation with AMR-driven negative samples generation. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 594–608, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.33. URL https://aclanthology.org/2024.naacl-long.33/.
- Reiter (2025) Ehud Reiter. We should evaluate real-world impact. Computational Linguistics, 2025.
- Roychowdhury et al. (2024) Sujoy Roychowdhury, Sumit Soman, HG Ranjani, Neeraj Gunda, Vansh Chhabra, and Sai Krishna Bala. Evaluation of rag metrics for question answering in the telecom domain. arXiv preprint arXiv:2407.12873, 2024.
- Shah & Bender (2024) Chirag Shah and Emily M Bender. Envisioning information access systems: What makes for good tools and a healthy web? ACM Transactions on the Web, 18(3):1–24, 2024.
- Sharma et al. (2024) Nikhil Sharma, Q Vera Liao, and Ziang Xiao. Generative echo chamber? effect of llm-powered search systems on diverse information seeking. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–17, 2024.
- Siriwardhana et al. (2023) Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering. Transactions of the Association for Computational Linguistics, 11:1–17, 2023.
- Tang et al. (2024) Liyan Tang, Philippe Laban, and Greg Durrett. Minicheck: Efficient fact-checking of llms on grounding documents. arXiv preprint arXiv:2404.10774, 2024.
- Venkit et al. (2024) Pranav Narayanan Venkit, Tatiana Chakravorti, Vipul Gupta, Heidi Biggs, Mukund Srinath, Koustava Goswami, Sarah Rajtmajer, and Shomir Wilson. ” confidently nonsensical?”: A critical survey on the perspectives and challenges of’hallucinations’ in nlp. arXiv preprint arXiv:2404.07461, 2024.
- Wu et al. (2025) Junde Wu, Jiayuan Zhu, and Yuyuan Liu. Agentic reasoning: Reasoning llms with tools for the deep research. arXiv preprint arXiv:2502.04644, 2025.
- Wu et al. (2024) Kevin Wu, Eric Wu, and James Zou. How faithful are rag models? quantifying the tug-of-war between rag and llms’ internal prior. arXiv preprint arXiv:2404.10198, 2024.
- Wyly (2014) Elvin Wyly. Automated (post) positivism. Urban Geography, 35(5):669–690, 2014.
- Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
- Zheng et al. (2025) Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160, 2025.
- Zhu et al. (2024) Kunlun Zhu, Yifan Luo, Dingling Xu, Ruobing Wang, Shi Yu, Shuo Wang, Yukun Yan, Zhenghao Liu, Xu Han, Zhiyuan Liu, et al. Rageval: Scenario specific rag evaluation dataset generation framework. arXiv preprint arXiv:2408.01262, 2024.
- Züger & Asghari (2023) Theresa Züger and Hadi Asghari. Ai for the public. how public interest theory shifts the discourse on ai. AI & SOCIETY, 38(2):815–828, 2023.
## Appendix A Limitations
While DeepTRACE offers an automated and scalable evaluation platform, it currently focuses on textual and citation-based outputs, excluding multimodal or UI-level interactions that also shape user trust and system usability. We do not evaluate for whether the answer to the question is the right answer but rather focus on the answer format, sources retrieved and citations used as these were the main themes obtained from the user evaluation done by Narayanan Venkit et al. (2025). Furthermore, some reliance on LLMs for intermediate judgments (e.g., factual support or confidence scoring) introduces potential biases, though we mitigated this with manual validation and report correlation metrics. Future work could integrate vision-based methods to assess UI presentations or combine LLMs with human-in-the-loop validation in high-stakes domains.
## Appendix B Score Card Metrics Thresholds
Table 2 establishes the benchmark ranges for the eight DeepTrace Evaluation metrics, categorizing performance into three levels: ▲ acceptable, ⚫ borderline, and ▼ problematic. These thresholds serve to quantify the usability and trustworthiness of GSE and deep research agents, allowing for a clear division between good, moderate, and poor system performance.
For instance, One-Sided Answer and Overconfident Answer are marked as problematic if these behaviors occur in 40% or more of the answers, which indicates a lack of balanced perspectives or excessive certainty, both of which can undermine user trust. A lower frequency (below 20%) is considered acceptable, as occasional bias or overconfidence may not drastically harm the user experience. Relevant Statements, by contrast, require a high threshold for acceptability—90% or more of the statements should directly address the user query. Anything below 70% is deemed problematic, indicating that a significant portion of the answer may be irrelevant, which can severely degrade the usefulness of the system.
For Uncited Sources and Unsupported Statements, a low occurrence is critical for ensuring reliability. An acceptable engine should have fewer than 5% uncited sources and fewer than 10% unsupported statements, as a higher proportion risks diminishing users’ ability to trust the information. Engines that fail to properly support claims or leave sources uncited in more than 25% of cases fall into the problematic category, revealing serious reliability issues.
The Source Necessity and Citation Accuracy metrics follow a similar logic: acceptable performance requires that 80-90% of sources cited directly support unique, relevant information in the answer. A citation accuracy below 50% is considered problematic, as it signals widespread misattribution or misinformation, eroding trust and transparency. Citation Thoroughness—the extent to which sources are fully cited—has a more lenient threshold, with anything above 50% being acceptable. However, thoroughness below 20% is deemed problematic, as this suggests incomplete sourcing for the content generated.
These thresholds reflect our attempt to balance between practicality and the need for high standards, recognizing that even small deviations from optimal performance on certain metrics can negatively impact user trust. These frameworks are designed with flexibility in mind, acknowledging that the acceptable ranges may evolve as user expectations rise and technology improves. For example, a current threshold of 90% citation accuracy may be sufficient now, but as GSE and deep research agents advance, this could shift to higher expectations of near-perfect accuracy and relevance.
| One-Sided Answer Overconfident Answer Relevant Statements | [0,20) [0,20) [90, 100) | [20,40) [20,40) [70,90) | [40,100) [40,100) [0,70) |
| --- | --- | --- | --- |
| Uncited Sources | [0,5) | [5,10) | [10,100) |
| Unsupported Statements | [0,10) | [10,25) | [25,100) |
| Source Necessity | [80,100) | [60,80) | [0,60) |
| Citation Accuracy | [90,100) | [50,90) | [0,50) |
| Citation Thoroughness | [50,100) | [20,50) | [0,20) |
Table 2: Ranges for the eight DeepTrace metrics for a system’s performance to be considered ▲ acceptable, ⚫ borderline, or ▼ problematic on a given metric.
## Appendix C Metrics Associated to Recommendations
Table 3 showcases what metrics were generated based on the recommendations and findings from Narayanan Venkit (2023).
| Provide balanced answers | Lack of holistic viewpoints for opinionated questions [A.II] | One-Sided Answers |
| --- | --- | --- |
| Provide objective detail to claims | Overly confident language when presenting claims [A.III] | Overconfident Answers |
| Minimize fluff information | Simplistic language and a lack of creativity [A.IV] | Relevant Statements |
| Reflect on answer thoroughness | Need for objective detail in answers [A.I] | – |
| Avoid unsupported citations | Missing citations for claims and information [C.III] | Unsupported Statement |
| Double-check for misattributions | Misattribution and misinterpretation of sources cited [C.I] | Citation Accuracy |
| Cite all relevant sources for a claim | Transparency of source selected in model response [C.IV] | Source Necessity |
| Listed & Cited sources match | More sources retrieved than used [S.II] | Uncited Sources |
| Give importance to expert sources | Lack of trust in sources used [S.III] | Citation Thoroughness |
| Present only necessary sources | Redundancy in source citation [S.IV] | Source Necessity |
| Differentiate source & LLM content | More sources retrieved than used for generation [S.II] | _ |
| Full represent source type | Low frequency of source used for summarization [S.I] | _ |
| Incorporate human feedback | Lack of search, select and filter [U.I] | _ |
| Implement interactive citation | Citation formats are not normalized interactions [U.IV] | _ |
| Implement localized source citation | Additional work to verify and trust sources [U.II] | _ |
| No answer when info not found | Lack of human input in generation and selection [U.I] | _ |
Table 3: Sixteen design recommendations for generative search engines and deep research agents. The recommendations derive from the findings of our usability study which are summarized in the middle column with corresponding findings [ID]. Some design recommendations are implemented as quantitative metrics (right column).
## Appendix D Examples of Responses
In this section, Figure 3 and Figure 4 shows how deep research models,specifically GPT-5 Deep Research, tend to generate outputs that closely follow the framing of the input questions, even when broader or more holistic perspectives may be warranted. This limitation becomes particularly problematic in non-participant contexts, where issues often involve nuanced viewpoints, thereby risking the creation of echo chambers for users.
<details>
<summary>Images/GPT5_1.png Details</summary>
### Visual Description
## Screenshot: Article Excerpt on Banning Bottled Water
### Overview
This image is a screenshot of a digital article or web page titled "Reasons to Ban Bottled Water." The content argues against single-use bottled water, focusing on its environmental, economic, and public health costs. The layout features a main text column on the left and a supporting photograph on the right. The text is structured with a main title, an introductory paragraph, a section heading ("Environmental Impact"), detailed body text with inline citations, and the beginning of a bulleted list.
### Components/Axes
* **Main Title:** "Reasons to Ban Bottled Water"
* **Introductory Paragraph:** A summary statement outlining the article's thesis.
* **Section Heading:** "Environmental Impact"
* **Body Text:** Prose detailing environmental arguments with embedded data and citations.
* **Citations:** Repeated inline references to "beyondplastics.org" in small, grey-background badges.
* **Embedded Image:** A photograph positioned to the right of the "Environmental Impact" section text.
* **Bulleted List:** Begins with two items: "Plastic Pollution:" and "Energy & Carbon Footprint:".
### Content Details
**Text Transcription:**
**Title:** Reasons to Ban Bottled Water
**Introductory Paragraph:**
Bottled water, though convenient, imposes heavy costs on the environment, economy, and public health. Its production and disposal generate massive plastic waste and greenhouse emissions, while consumers pay large premiums for water that is often no safer than tap. In contrast, clean tap water and refill systems are widely available. Below are the key arguments – with supporting data and examples – for banning single-use bottled water.
**Section: Environmental Impact**
Single-use plastic water bottles clog beaches, rivers, and oceans, and introduce microplastics and toxins into ecosystems [beyondplastics.org]. About **20–30 billion pounds** of plastic enter the oceans each year [beyondplastics.org], much of it from bottles that break down into microplastics. Bottles also drive climate pollution: producing bottled water requires far more energy and yields far more carbon than tap water. For example, producing a bottle can consume **11–90 times** more energy than delivering the same volume of tap water [beyondplastics.org], resulting in roughly **300–1000×** the greenhouse-gas emissions of tap water [beyondplastics.org]. Waste volumes are enormous – U.S. consumption grew from ~3 billion bottles in 1997 to **86 billion** in 2021 [beyondplastics.org] – yet recycling is dismal. Only about one-third of plastic water bottles are recycled in the U.S. today [beyondplastics.org] (versus 90%+ in places with bottle-deposit laws [beyondplastics.org]). Finally, water and material use is wasteful: making a single plastic bottle typically **uses ~1.4 gallons of water** [beyondplastics.org] (over 10× the bottle’s capacity) and consumes petroleum and chemicals.
**Bulleted List (Partial):**
* **Plastic Pollution:** Bottles litter land and sea, harming wildlife and creating microplastics. Studies estimate billions of pounds of plastic (including bottles) flow into oceans annually [beyondplastics.org]. This waste can take centuries to decompose, accumulating toxins that enter the food chain.
* **Energy & Carbon Footprint:** Bottled water production is highly energy-intensive. The Massachusetts... (text cuts off).
**Embedded Photograph Description:**
Positioned to the right of the "Environmental Impact" text. The image shows a shoreline littered with numerous large, discarded plastic water bottles and other debris. In the background, two people are visible on the beach, with hills or mountains under a cloudy sky. The photo visually reinforces the text's point about plastic pollution.
### Key Observations
1. **Data-Driven Argument:** The text heavily relies on specific numerical data (e.g., 20-30 billion pounds, 11-90 times more energy, 86 billion bottles) to substantiate its claims.
2. **Source Attribution:** Every major data point and claim is followed by an inline citation to "beyondplastics.org," indicating a single primary source for the information presented in this excerpt.
3. **Visual Reinforcement:** The embedded photograph is not decorative; it directly illustrates the "clog beaches" and "plastic pollution" claims made in the adjacent text.
4. **Comparative Analysis:** The argument is structured around comparisons: bottled water vs. tap water (energy, emissions, cost), U.S. recycling rates vs. regions with deposit laws, and water used to make a bottle vs. the bottle's capacity.
5. **Document Structure:** The content follows a clear persuasive structure: thesis statement -> section heading -> detailed evidence with data -> summarized key points in a list.
### Interpretation
This screenshot captures a segment of a persuasive technical or advocacy document. Its primary purpose is to build a factual case against single-use bottled water by aggregating and presenting environmental impact data. The consistent citation of a single source ("beyondplastics.org") suggests this may be a summary or promotion of a report from that organization.
The elements relate to each other to create a cohesive argument: the title states the position, the introduction frames the issue, the "Environmental Impact" section provides the core evidence using statistics and comparisons, the photograph offers visceral, visual proof, and the bulleted list begins to distill the evidence into memorable, categorical takeaways. The cutoff at the end implies this is only the first section of a longer document that likely covers economic and public health arguments as mentioned in the introduction.
The notable outlier in the data is the wide range given for energy consumption ("11–90 times"), which indicates significant variability depending on the study or context, a nuance the text acknowledges by presenting it as a range rather than a single figure. The overall trend in the data presented is one of massive scale (billions of pounds, billions of bottles) coupled with systemic inefficiency (low recycling rates, high resource input per unit).
</details>
Segment 1
<details>
<summary>Images/GPT5_2.png Details</summary>
### Visual Description
## Text Document: Economic and Social Factors of Bottled Water
### Overview
The image is a screenshot of a text document or webpage section titled "Economic and Social Factors." It presents a critical analysis of the bottled water industry, focusing on its economic impact on consumers, the role of corporate privatization, and the resulting social inequities. The text is structured with a main heading and three primary bullet points, each containing detailed arguments supported by inline citations (e.g., `bmchealth.org`). The language is English.
### Components/Axes
This is a textual document, not a chart or diagram. Its components are:
* **Main Heading:** "Economic and Social Factors"
* **Bullet Points:** Three main thematic sections.
* **Inline Citations:** References to sources like `bmchealth.org`, `angelwater.com`, `flowwateradvocates.org`, and `givingcompass.org` are embedded within the text.
* **Concluding Paragraph:** A summary statement contrasting bottled water reliance with public infrastructure investment.
### Detailed Analysis / Content Details
**1. Cost to Consumers**
* **Core Claim:** Bottled water is vastly more expensive than tap water.
* **Data Points:**
* Tap water cost: ~$0.01–$0.03 per gallon.
* Retail bottled water cost: $2–$12 per gallon.
* Price multiplier: Bottled water is "hundreds to thousands of times" more expensive per gallon.
* **Impact:** Low-income families switching to bottled water can spend "thousands of dollars extra per year."
* **Characterization:** This cost premium is described as a "regressive tax on the poor and elderly," who often switch due to mistrust of tap water.
**2. Water Privatization & Profit**
* **Core Claim:** Large corporations treat water as a commodity for profit.
* **Examples:** Nestlé (now BlueTriton) and other companies buy water rights and sell at high markups.
* **Financialization:** The text notes that "Financial markets even trade water futures on exchanges."
* **Consequences:** Privatization trends "degrade the singular importance of water" and "exacerbate inequities." It creates a disparity where private buyers access clean bottled water while impoverished communities lack basic safe water infrastructure.
* **Broader Context:** Debates over water rights highlight that cheap water is a public resource, not a luxury good.
**3. Equity Issues**
* **Core Claim:** Bottled water spending disproportionately burdens those with lower incomes.
* **Demographic Data:** Research indicates "low-income, Black, and Latino households drink bottled water at higher rates than wealthier, white households."
* **Cause:** Distrust in tap water due to "past service failures or contamination."
* **Result:** These communities become dependent on expensive bottled water, worsening economic and racial inequality. The text quotes: "bottled water’s far higher costs are being borne by those least able to afford them."
* **Systemic Issue:** Families pay extra for water while their public water systems remain underfunded.
**Concluding Paragraph**
* **Proposed Solution:** Improving public water infrastructure and affordability (e.g., through public systems or rebates) would advance social equity.
* **Critique of Status Quo:** Reliance on bottled water as a "solution" delays investment in safe taps and shifts the burden from governments or polluters to individuals.
### Key Observations
* **Economic Disparity:** The text consistently frames the cost difference not just as a consumer choice but as a structural economic issue that functions as a "regressive tax."
* **Systemic Inequity:** It explicitly links economic burden with racial and social demographics, noting higher consumption rates among marginalized communities.
* **Corporate Role:** The analysis identifies corporate profit motives and financial speculation (water futures) as active drivers of inequity.
* **Infrastructure Neglect:** A central theme is that the bottled water market thrives on, and perpetuates, the underinvestment in and failure of public water infrastructure.
### Interpretation
The document presents a Peircean argument where the **sign** (the high cost and prevalence of bottled water) points to a deeper **object** (systemic failure and inequity in public water provision). The text moves beyond simple price comparison to investigate underlying social and economic structures.
1. **Causal Relationships:** It establishes a clear chain: public infrastructure failure → community mistrust → reliance on expensive bottled water → increased financial burden on low-income and minority households → deepened inequality. This cycle is shown to benefit corporate entities financially.
2. **Normative Argument:** The text argues that water should be treated as a fundamental public good, not a commodity. The "solution" of bottled water is framed as a market-based failure that exacerbates the very problem it purports to solve.
3. **Implied Stakeholders:** The analysis implicates multiple actors: corporations (for profiting), governments (for underfunding infrastructure), and polluters (for causing contamination). The burden of the problem is unfairly placed on the individual consumer.
4. **Anomaly/Outlier:** The "outlier" in this social data is the wealthier, white household, whose lower rate of bottled water consumption highlights the disparity. Their relative immunity to the "regressive tax" underscores the inequity.
In essence, the document uses economic data and social research to argue that the bottled water industry is a symptom and amplifier of deeper public policy failures, with severe consequences for social equity.
</details>
Segment 2
<details>
<summary>Images/GPT5_3.png Details</summary>
### Visual Description
## Text Document Screenshot: Health and Safety Concerns of Bottled Water
### Overview
The image is a screenshot of a text document, likely from a website or digital report, discussing the health and safety concerns associated with consuming bottled water. The text is presented in a dark mode theme (light text on a dark background) and is structured with a main heading, two primary bullet points, and a concluding summary paragraph. The content cites multiple external sources via inline hyperlinks.
### Components/Axes
* **Main Title:** "Health and Safety Concerns"
* **Primary Bullet Points:**
1. "Microplastics and Nanoparticles"
2. "Water Quality and Regulation"
* **Inline Citations/Sources:** The text includes numerous inline source references, formatted as small, pill-shaped badges. The sources cited are:
* `nih.gov` (National Institutes of Health)
* `nrdc.org` (Natural Resources Defense Council)
* `pmc.ncbi.nlm.nih.gov` (PubMed Central)
* `beyondplastics.org`
* **Layout:** The text is left-aligned. The main title is at the top. The two bullet points form the body. A final, un-bulleted summary paragraph is at the bottom. A downward-pointing arrow icon is partially visible at the very bottom center, suggesting more content follows.
### Detailed Analysis / Content Details
**Full Text Transcription:**
**Health and Safety Concerns**
* **Microplastics and Nanoparticles:** Bottled water can contain large quantities of tiny plastic particles. A recent NIH-funded study using advanced microscopy found an average of **240,000 plastic fragments per liter** in three major bottled water brands `nih.gov`. About **90% of these were nanoplastics (<1 µm)** `nih.gov`. This is **10–100× more plastic** by particle count than previously detected. The health effects of ingesting micro- and nanoplastics are not yet fully understood, but emerging evidence links them to cellular damage and chemical exposure `nih.gov` `nrdc.org`. For example, plastic bottles can shed additives and plasticizers; one review notes bottles may leach the heavy metal *antimony* and endocrine-disrupting chemicals (similar to BPA) into the water `nrdc.org`. Children who drink bottled water regularly have higher exposure to these microplastics than those who drink tap water.
* **Water Quality and Regulation:** Bottled water is not guaranteed to be purer than tap water. In fact, regulations for municipal water supplies (EPA) are often stricter than for bottled water (FDA) `pmc.ncbi.nlm.nih.gov` `nrdc.org`. Bottlers are typically **not required** to test as frequently or to disclose problems with their water. Studies and surveys confirm bottled water is often just treated tap water `nrdc.org` `pmc.ncbi.nlm.nih.gov`. In NRDC testing, about **25% of brands sampled contained contaminants above state health limits** `nrdc.org`. (For example, **22% of 1,000 brands had measurable levels of chemical contaminants**.) By contrast, publicly supplied tap water must meet strict standards for disinfection and be tested constantly. Only in rare cases (e.g. homes with lead pipes) does tap water exceed health guidelines. In general, in high-income countries with modern treatment, tap water is **safe, cheap, and free of microplastics**, whereas bottled water may carry hidden risks and waste money.
In summary, bottled water offers no clear health advantage over tap: it can be contaminated and is only nominally regulated. Meanwhile, its plastic content introduces new, uncertain risks. Given that **95% of Americans already have access to safe tap water** `beyondplastics.org`, the health benefits of bottled water are marginal, whereas the downsides (microplastic exposure, loss of fluoride, etc.) are potentially serious.
### Key Observations
1. **Quantitative Data:** The text presents specific, cited statistics:
* 240,000 plastic fragments per liter (average in three major brands).
* 90% of those fragments are nanoplastics (<1 micrometer).
* 10-100 times more plastic than previous estimates.
* 25% of sampled brands had contaminants above state limits (NRDC testing).
* 22% of 1,000 brands had measurable chemical contaminants.
* 95% of Americans have access to safe tap water.
2. **Regulatory Comparison:** A core argument is the contrast between stricter, more transparent EPA regulations for tap water and weaker, less frequent FDA regulations for bottled water.
3. **Source Density:** The text is heavily cited, with multiple references to `nih.gov`, `nrdc.org`, and `pmc.ncbi.nlm.nih.gov`, lending it an evidence-based tone.
4. **Emphasis:** Key phrases and numbers are bolded for emphasis (e.g., "240,000 plastic fragments per liter", "not required", "safe, cheap, and free of microplastics").
### Interpretation
This document constructs a technical argument against the perceived safety and purity of bottled water by juxtaposing two main lines of evidence: **contaminant content** and **regulatory oversight**.
* **The Microplastics Argument:** It frames bottled water not just as a potential carrier of traditional chemical contaminants, but as a source of a novel, poorly understood pollutant—nanoplastics. The cited study (240,000 fragments/L) is used to establish a significant quantitative baseline for this exposure, which is presented as orders of magnitude higher than previously thought. The health implications are linked to cellular damage and chemical leaching (antimony, endocrine disruptors), with children identified as a vulnerable subgroup.
* **The Regulatory Argument:** It challenges the common perception of bottled water as a "purer" alternative by highlighting a regulatory gap. The argument posits that the less stringent FDA rules allow for more contaminants and less transparency compared to municipal tap water systems under EPA rules. The NRDC testing data (25% of brands exceeding limits) is used as empirical support for this claim.
* **Synthesis and Conclusion:** The document synthesizes these points to conclude that bottled water offers a poor risk-benefit profile. It introduces plastic-related health risks (the "downsides") while providing no clear safety advantage over tap water, which is described as widely accessible, affordable, and free of microplastics in modern infrastructure. The final sentence frames the choice as one where marginal benefits are outweighed by potentially serious downsides, urging a reconsideration of bottled water consumption based on health, regulatory, and economic factors. The mention of "loss of fluoride" hints at an additional public health downside not fully elaborated in this excerpt.
</details>
Segment 3
Figure 3: Screenshots of responses from GPT-5 Deep Research to the debate question, “why should we ban bottled water?”. The figure illustrates how answers to debate-oriented questions may present a predominantly one-sided perspective, despite the presence of multiple nuances. Similar tendencies can be observed in more critical or ostensibly nonpartisan contexts, where questions allow for multiple valid viewpoints.
<details>
<summary>Images/GPT5_4.png Details</summary>
### Visual Description
## Text Document: Arguments Against Banning Bottled Water
### Overview
The image is a screenshot of a digital document or article presenting arguments against the prohibition of bottled water. The content is structured with a title, a summary paragraph, and the beginning of a two-column table that categorizes and details the key points. The text is in English. The document cites multiple external sources, indicated by small, gray, pill-shaped badges containing domain names (e.g., `mdpi.com`, `distillata.com`). The bottom of the image is cut off, showing only the start of the "Environmental Impact" category in the table.
### Components/Axes
* **Title:** "Arguments Against Banning Bottled Water"
* **Summary Section:** A paragraph outlining the core thesis that bans have serious trade-offs, touching on public health, environmental backfire effects, economic impacts, equity concerns, and consumer freedom.
* **Table Structure:**
* **Column 1 Header:** "Category"
* **Column 2 Header:** "Key Points Against a Ban"
* **Visible Categories:** "Public Health & Safety", "Environmental Impact" (partially visible).
* **Citations:** Embedded source references appear as gray badges with white text. The following domains are visible: `mdpi.com`, `governmentprocurement.com`, `distillata.com`, `ccbw.com`, `thecrimson.com`, `who.int`, `unu.edu`, `economicshelp.org`, `epa.gov`.
### Detailed Analysis
**Summary Paragraph Transcription:**
"Proposals to ban bottled water often rest on environmental goals (reducing plastic waste), but such bans entail serious trade-offs. Bottled water plays a crucial role in ensuring safe hydration, especially where tap water is unreliable or during emergencies `mdpi.com` `governmentprocurement.com`. Empirical studies show that removing bottled water can *increase* consumption of sugary drinks and even plastic waste `distillata.com` `ccbw.com`. The bottled water industry also underpins many jobs and economic activities (e.g. campus cafés, delivery services), so bans risk significant economic losses `mdpi.com` `thecrimson.com`. Moreover, banning bottled water raises equity and ethical concerns: access to clean water is a fundamental right, and millions worldwide lack safe tap water `who.int` `unu.edu`. Finally, outright bans restrict consumer choice and personal freedom `thecrimson.com`. In short, practical needs and ethical principles argue for improving water infrastructure, recycling, and voluntary measures (like refill stations or taxes) rather than an outright ban. The table below summarizes the key arguments by category."
**Table Content - Row 1:**
* **Category:** "Public Health & Safety"
* **Key Points:** "Bottled water provides a *reliable* source of safe drinking water when tap systems fail or are contaminated `mdpi.com`. In the US Flint crisis, ~21 million people received water violating health standards `mdpi.com`. Bans could force people to drink *less safe* or *less healthy* alternatives (e.g. sugary sodas) `distillata.com`. Bottled water is also essential in emergencies/disasters, as it is quick to distribute and store until tap systems are restored `governmentprocurement.com` `governmentprocurement.com`."
**Table Content - Row 2 (Partially Visible):**
* **Category:** "Environmental Impact"
* **Key Points (Visible Text):** "Bottled water has a lower lifecycle footprint than most other packaged beverages `economicshelp.org` `mdpi.com`. Banning it often backfires: people substitute with other single-use drinks (soda, juices), which may use *more* water, energy and materials `thecrimson.com` `distillata.com`. Industry efforts (lightweight PET bottles, recycling programs) are improving sustainability, and ~29% of PET bottles are recycled in the US `epa.gov`. Experts argue that broad measures (polluter-pay taxes on all single-use plastics) would..." [Text cuts off].
### Key Observations
1. **Argument Structure:** The document uses a clear, categorized format to present counter-arguments, moving from immediate human concerns (health, safety) to broader systemic issues (environment, economy, ethics).
2. **Use of Emphasis:** Italics are used strategically to highlight key adjectives (*reliable*, *increase*, *less safe*, *more*), strengthening the rhetorical contrast.
3. **Citation Density:** Arguments are heavily supported by inline citations, suggesting an evidence-based approach. Some sources are cited multiple times within a single point (e.g., `mdpi.com` appears three times in the "Public Health & Safety" section).
4. **Specific Data Points:** The text includes concrete figures: "~21 million people" affected in the Flint crisis and "~29% of PET bottles are recycled in the US."
5. **Visual Layout:** The table uses a clean, two-column layout with bold category headers for easy scanning. A "copy" icon is visible in the top-right corner of the table header row.
### Interpretation
This document constructs a pragmatic and ethical case against bottled water bans by framing them as a well-intentioned but potentially counterproductive policy. The core argument is that bans ignore the complex role bottled water plays as a safety net for public health, a less environmentally damaging option compared to substitutes, and a pillar of economic activity and personal choice.
The **Peircean investigative reading** reveals the underlying sign: the document positions itself not as a defense of the bottled water industry per se, but as a critique of simplistic, single-solution environmentalism. It argues that the *effect* of a ban (increased consumption of less healthy/more damaging alternatives, economic loss, reduced access) contradicts the intended *cause* (environmental protection). The proposed alternative—improving infrastructure, recycling, and targeted measures like taxes—suggests a preference for systemic, nuanced solutions over outright prohibition. The heavy citation implies an appeal to authority and empirical evidence to counter what might be an emotionally driven environmental narrative. The cut-off at "Environmental Impact" leaves the final argument about "polluter-pay taxes" incomplete, but it points toward a regulatory rather than prohibitive framework.
</details>
<details>
<summary>Images/GPT5_5.png Details</summary>
### Visual Description
## Text Document: Public Health and Safety Argument for Bottled Water
### Overview
The image is a screenshot of a text document, likely from a website or digital report. It presents an argument in favor of bottled water from a public health and safety perspective. The text is divided into two main sections, with the first fully visible and the second partially cut off at the bottom. The document uses inline citations (e.g., `mdpi.com`, `fda.gov`) to support its claims.
### Content Structure
The visible content is structured as follows:
1. **Main Heading:** "Public Health and Safety"
2. **Paragraph 1:** Discusses bottled water as a safeguard when tap water is unsafe, citing a 2015 statistic and the example of Flint, Michigan. It warns that banning bottled water could lead to consumption of polluted water or sugary beverages.
3. **Paragraph 2:** Focuses on the role of bottled water in emergencies and disasters, describing it as the "easiest, fastest way" to hydrate populations. It includes a quote from a logistics company.
4. **Paragraph 3:** Argues that bottled water is not always less healthy than tap water, noting stricter FDA standards for certain contaminants compared to EPA regulations for tap water. It concludes that banning bottled water could undermine health.
5. **Secondary Heading:** "Environmental Considerations" (partially visible).
6. **Partial Sentence:** The beginning of the environmental argument is visible: "The environmental rationale for banning bottled water is to reduce plastic waste and resource use."
### Detailed Content Transcription
**Language:** English
**Full Text Transcription:**
**Public Health and Safety**
Access to clean drinking water is fundamental. Bottled water often safeguards health when tap water is questionable. For example, an analysis found that in 2015 **~21 million Americans** were served by water systems violating health standards `mdpi.com`. In communities with aging pipes or contamination (e.g. lead in Flint), bottled water may be the only *safe* option in the short term `mdpi.com`. If bottled water were banned under such conditions, people risk drinking polluted tap water or substituting high-sugar beverages. Studies show that removing bottled water (e.g. on campuses) led to *increased* sales of sugary sodas and juices – outcomes that can worsen health `distillata.com`.
Bottled water is also crucial in emergencies and disasters. After earthquakes, floods or storms, tap systems can fail for days or weeks. In those scenarios, packaged water is the **easiest, fastest way** to hydrate affected populations `governmentprocurement.com` `governmentprocurement.com`. Emergency response plans rely on bottled water precisely because it is sanitary, portable, and storable. One logistics company notes that “bottled water is safe drinking water accessible and storable for everyone, and it’s the easiest way to distribute … until safe tap water is restored” `governmentprocurement.com`. By contrast, insisting on tap water could leave evacuees without clean water for a prolonged period, risking dehydration or disease.
Finally, public health advocates point out that not all tap water is healthier than bottled. In the US, tap water is regulated by the EPA, but **bottled water under FDA often has stricter standards for certain contaminants** (e.g. lead limits in bottling are tighter) `fda.gov`. Thus, in many cases bottled water may actually have fewer impurities than local tap. In sum, banning bottled water could undermine health: it removes a low-risk drinking option and may push people toward higher-risk alternatives `mdpi.com` `distillata.com`.
**Environmental Considerations**
The environmental rationale for banning bottled water is to reduce plastic waste and resource use.
### Key Observations
1. **Emphasis through Formatting:** Key phrases are emphasized using **bold** (e.g., "~21 million Americans", "easiest, fastest way") and *italics* (e.g., "safe", "increased").
2. **Citation Placement:** Citations are placed directly after the claims they support, formatted as small, grey, pill-shaped tags (e.g., `mdpi.com`). Some claims have multiple citations.
3. **Argument Flow:** The text builds a case by moving from chronic issues (aging infrastructure) to acute crises (disasters) and finally to a regulatory comparison, concluding with a summary statement.
4. **Visual Layout:** The text is left-aligned on a dark grey background with white/light grey text, suggesting a dark-mode interface. A downward-pointing arrow icon is partially visible over the "Environmental Considerations" heading, indicating more content below.
### Interpretation
The document presents a utilitarian public health argument that prioritizes immediate access to safe drinking water over potential environmental concerns. It frames bottled water not as a consumer good but as a critical public health tool and a necessary backup system.
* **Core Thesis:** The potential negative health consequences of banning bottled water (consuming contaminated water, drinking more sugary beverages, suffering during emergencies) are presented as more immediate and severe than the environmental costs.
* **Underlying Logic:** The argument relies on a risk-aversion framework. It positions bottled water as a "low-risk drinking option" and suggests its removal would force people toward "higher-risk alternatives." This shifts the debate from environmental idealism to practical harm reduction.
* **Strategic Use of Evidence:** The citations (`mdpi.com`, `fda.gov`, etc.) are used to lend authority to specific data points (21 million Americans affected) and regulatory claims (FDA vs. EPA standards). The quote from a logistics company adds a practical, operational perspective on disaster response.
* **Anticipatory Counter-Argument:** By introducing the "Environmental Considerations" heading at the end, the document acknowledges the opposing viewpoint but has already spent the majority of its space building a strong case for the public health necessity of bottled water, potentially framing the environmental argument as secondary.
</details>
<details>
<summary>Images/GPT5_6.png Details</summary>
### Visual Description
## Text Document: Environmental and Economic Analysis of Bottled Water
### Overview
The image is a screenshot of a digital document or article discussing the environmental and economic aspects of bottled water, specifically arguing against bans on the product. The text is presented in a dark mode interface with white text on a dark gray background. It contains two main sections: "Environmental Considerations" and the beginning of "Economic Impacts." The text includes inline citations with source URLs displayed in small, rounded rectangular badges.
### Content Details (Full Transcription)
**Language:** English
**Section 1: Environmental Considerations**
**Paragraph 1:**
The environmental rationale for banning bottled water is to reduce plastic waste and resource use. However, evidence suggests bottled water is *less* impactful than often portrayed when compared to other beverages. Life-cycle analyses find that **bottled water has the lowest environmental footprint of all packaged drinks** `mdpi.com`. For example, it takes only about 1.39L of water to produce 1L of bottled water, whereas soda requires over 2L and beer over 4L `economicshelp.org`. If bottled water is banned, thirsty consumers are likely to switch to other bottled beverages (sodas, juices, even beer or sports drinks) – all of which generally require more plastic, sugar, and energy to make `thecrimson.com` `distillata.com`. In practice, bans on bottled water sometimes *increase* plastic waste: a study reported an 8.5% rise in plastic bottles entering the waste stream at one university after banning bottled water, as students substituted other drinks `ccbw.com`.
**Paragraph 2:**
Modern bottling is also becoming more sustainable. The industry has lightened plastic bottles (recent PET bottles use 30–40% less plastic by weight than older ones), and a significant fraction of bottles are recycled. U.S. EPA data show nearly 29% of PET bottles (the kind used for water) were recycled in 2018 `epa.gov`, with further gains reported in recent years. If the goal is to reduce plastic pollution, experts argue that focusing on universal recycling and broad-based taxes on all single-use plastics is better policy. As one economist writes, rather than ban bottled water alone we could impose "an 'environmental tax' on all plastic, single-use drinks," using the revenue for cleanup `economicshelp.org`. This "polluter pays" approach would discourage *all* plastic waste while still allowing consumers to choose their beverages. In short, bottled water's environmental impact, while real, is relatively modest, and banning it may have negligible benefits if other drink bottles simply fill the void `economicshelp.org` `mdpi.com`.
**Section 2: Economic Impacts (Incomplete)**
**Paragraph 1 (Partial):**
The bottled water industry is a major economic sector. Globally it was worth about **USD 285 billion** in 2020 and is growing rapidly `mdpi.com`. In the U.S., bottled water sales grew dramatically (per-capita...
*(The text cuts off at this point.)*
### Key Observations
1. **Argumentative Structure:** The text presents a counter-argument to bottled water bans, using comparative environmental data and economic incentives as its foundation.
2. **Data Points Cited:**
* Water usage for production: 1.39L water per 1L bottled water (vs. >2L for soda, >4L for beer).
* Plastic reduction: Recent PET bottles use 30–40% less plastic by weight.
* Recycling rate: ~29% of PET bottles recycled in the U.S. in 2018.
* Unintended consequence: An 8.5% increase in plastic bottle waste at one university post-ban.
* Global market value: USD 285 billion in 2020.
3. **Source Attribution:** Claims are supported by inline citations to sources like `mdpi.com`, `economicshelp.org`, `epa.gov`, `thecrimson.com`, `distillata.com`, and `ccbw.com`.
4. **Visual Layout:** The text is left-aligned in a single column. Citations are visually distinct, appearing as small, dark gray badges with white text, placed immediately after the relevant claim. A scroll-down arrow icon is partially visible at the bottom, indicating more content below.
### Interpretation
The document constructs a pragmatic, economics-and-data-driven case against single-product bans on bottled water. Its core thesis is that such bans are a poorly targeted policy tool that may yield negligible environmental benefits or even backfire.
* **Peircean Investigation (Reading Between the Lines):** The text uses a **abductive** reasoning pattern. It starts with the common claim (banning bottled water helps the environment) and then presents **signs** (comparative lifecycle data, substitution effects, recycling rates) that suggest a different, more complex **fact** (bottled water is relatively efficient, and bans may shift consumption to worse alternatives). The argument moves from observation to the most likely explanation: broad-based policies (like taxes or universal recycling) are more effective than narrow bans.
* **Underlying Message:** The piece implies that environmental policy should be guided by full lifecycle analysis and systemic thinking rather than by symbolic actions targeting a single, highly visible product. It frames bottled water not as a unique villain, but as part of a larger system of packaged beverages, where it currently represents the least impactful option.
* **Notable Omission/Anomaly:** The text focuses exclusively on *comparative* environmental impact (vs. other drinks) and *production* metrics. It does not address the significant issue of post-consumer waste management, litter, or the environmental cost of transporting water over long distances, which are common criticisms of the industry. The argument hinges on the assumption that consumers will switch to other bottled drinks if water is banned, rather than to tap water or refillable containers.
</details>
<details>
<summary>Images/GPT5_7.png Details</summary>
### Visual Description
## Text Block: Economic Impacts of the Bottled Water Industry
### Overview
The image is a screenshot of a text document titled "Economic Impacts." It presents a detailed argument about the significant economic footprint of the global bottled water industry and the potential negative consequences of implementing bans on the product. The text is structured into three paragraphs, each containing embedded source citations.
### Components/Axes
* **Title:** "Economic Impacts" (displayed in bold, larger font at the top).
* **Main Body Text:** Three paragraphs of prose text.
* **Citations:** Inline source references formatted as small, grey-background tags (e.g., `mdpi.com`, `thecrimson.com`, `economicshelp.org`). These are embedded within the sentences they support.
* **Layout:** The text is left-aligned on a dark grey background. A downward-pointing arrow icon is partially visible at the very bottom, suggesting this is part of a scrollable interface.
### Detailed Analysis / Content Details
**Paragraph 1:**
* **Global Industry Value:** The bottled water industry was worth approximately **USD 285 billion** in 2020 and is described as "growing rapidly." (Source: `mdpi.com`)
* **U.S. Consumption Growth:** Per-capita consumption in the U.S. rose from **61 L in 1999** to **160 L in 2018**. (Source: `mdpi.com`)
* **Employment & Local Revenue:** The industry supports hundreds of companies and their employees. Local businesses (cafes, vending operators) earn significant revenue from sales.
* **Specific Example:** Before a ban, the University of Vermont sold approximately **350,000 bottles a year**. (Source: `thecrimson.com`)
* **Impact of Bans:** Analysts warn bans would cause lost sales, potentially leading to "lower pay or fewer student-employees" at affected outlets. (Source: `thecrimson.com`)
**Paragraph 2:**
* **Government Revenue Loss:** Governments would lose tax receipts from:
* Income taxes paid by workers in bottling plants and delivery networks.
* Corporate taxes paid by companies.
* Sales tax from bottled water sales.
* **Investment Deterrence:** Rigid bans could deter investment in healthier beverage options.
* **Economic Argument Against Bans:** Economists argue that an outright ban removes incentives for companies to make packaging more eco-friendly or production greener.
* **Proposed Alternative:** A competitive market with fair taxation on all drinks is suggested to encourage innovation (e.g., recyclable bottles, water fountains, filtration systems) and generate revenue for environmental programs. (Sources: `economicshelp.org`, `thecrimson.com`)
**Paragraph 3 (Conclusion):**
* **Summary of Economic Footprint:** The economic footprint is "substantial," and banning would eliminate jobs and business activity.
* **Critics' Irony:** Critics point out the irony of banning water (a healthy need) while allowing less healthy beverages, noting both have environmental costs. (Source: `economicshelp.org`)
* **Balanced Policy Suggestion:** A balanced policy might maintain availability while imposing moderate fees to reflect environmental costs, as suggested by economist Tejvan Pettinger. (Source: `economicshelp.org`)
* **Final Line (Partially Cut Off):** The text ends mid-sentence: "...to reflect any environmental co..." The visible citation is `economicshelp.org`.
### Key Observations
1. **Data-Driven Argument:** The text relies on specific, cited data points (market value, consumption volumes, sales figures) to ground its economic argument.
2. **Multi-Stakeholder Impact:** It details impacts across various stakeholders: global corporations, local businesses, employees (including students), and government tax bases.
3. **Policy Nuance:** The argument moves beyond a simple "ban vs. no ban" dichotomy, proposing a market-based, fee-oriented solution as a middle path.
4. **Rhetorical Framing:** The conclusion uses the concept of "irony" to frame the debate, contrasting bottled water with less healthy but permitted beverages.
### Interpretation
The text presents a **Peircean investigative** analysis of the bottled water debate, focusing on the **pragmatic consequences** (the "pragmatic maxim") of a potential ban. It argues that the **sign** of a ban (the policy action) would lead to a chain of **interpretants** (lost jobs, reduced tax revenue, stifled innovation) that are economically detrimental. The underlying **ground** for this argument is the industry's deep integration into local and global economic systems.
The text suggests that the debate is not merely environmental but deeply economic. It positions bottled water not as a luxury good but as a commodity with a vast, embedded supply chain. The proposed solution—moderate fees—aims to internalize environmental externalities without dismantling the economic structure, reflecting a **market-environmentalist** perspective. The most significant anomaly highlighted is the potential for a ban to paradoxically reduce incentives for environmental innovation within the industry, a counterintuitive outcome that challenges simplistic regulatory approaches.
</details>
(a) Screenshot 4
Figure 4: Screenshots of responses from GPT-5 Deep Research to the debate question, “why should we not ban bottled water?”. The figure illustrates how answers to debate-oriented questions may present a predominantly one-sided perspective, despite the presence of multiple nuances. Similar tendencies can be observed in more critical or ostensibly nonpartisan contexts, where questions allow for multiple valid viewpoints.
## Appendix E LLM Judge Prompts Used for Evaluation
In this section, we present the LLM instructions used to automate tasks for the DeepTrace scorecard evaluation. We acknowledge that model outputs cannot always be considered the gold standard, so we conducted human evaluations to assess agreement between human annotators and the selected LLM, GPT-4. The results are detailed in Section 4.2. The prompts are as follows:
Confidence Parameter Evaluation
⬇ You are given a user Query, and the Answer generated by a system. Your task is to evaluate the confidence in the language provided by the system in the answer. You must choose one of five confidence levels. Annotation Scheme: - " Strongly Not Confident ": The language in this sentence is not clear or confident at all. - " Not Confident ": The language in this sentence is somewhat unclear and lacks confidence. - " Neutral ": The language in this sentence is neither clear nor unclear; confidence level is average. - " Confident ": The language in this sentence is clear and fairly confident. - " Strongly Confident ": The language in this sentence is very clear and confident. Format: - You must produce your answer as a JSON object, following this format: {" confidence ": "< Confidence Level >"} - Replace < Confidence Level > with one of the five confidence levels. - Do not output anything other than the JSON object with the confidence level. Query: [[QUERY]] Answer: [[ANSWER]]
Relevant Statement Extraction
⬇ You are given a paragraph, made of a sequence of sentences that answer the following question: [[QUESTION]] Your task is to extract, in JSON format, what the individual sentences are, and then identify for each sentence whether it contains a core statement that answers the question, or if it is a filler sentence that does not contain substantial information. You should follow the following format: {" sentences ": {" sentence ": "...", " core ": "1|0"}, {" sentence ": "...", " core ": "1|0"}, } Rules: - Do not modify the sentences whatsoever, you should copy them as is. - Do not modify the order of the sentences, or skip any of the sentences. - The sentences optionally contain citations (e. g. [1], [2], etc.). You should not modify the citations, keep them as is. - If the sentence contains anything related to the answer, you should mark it as a core statement. Sentences with a citation are almost always core statements. - The only cases that are not core statements are: - Filler sentences that do not contain any information (introduction, conclusion, etc.) Here is the answer you should decompose: ‘‘‘ [[ANSWER]] ‘‘‘
Supported Statement Evaluation
⬇ You are given the full text of a document, and a statement (a sentence). Your objective is to determine whether the statement is fully, partially, or not supported by the document. You should output a JSON object with the following format: {" support ": " full | partial | none "} Document: ‘‘‘ [[DOCUMENT]] ‘‘‘ Statement: [[STATEMENT]] Rules: - The statement may contain citations (e. g. [1], [2], etc.). You do not need to consider the citations when determining support. - Only output valid JSON. Do not include any other information in the output.
One-sided Answer Detection
⬇ You are given a query that is opinionated (on one side of an issue). You are then given a numbered list of statements. Your objective is to sort these statements into three lists: - " agree _ statements " if they agree with the opinionated query, - " disagree _ statements " if they disagree with the opinionated query, - " neutral _ statements " if they are neutral to the opinionated query. You should return a JSON object following the given format: {" agree _ statements ": [1, 2, 3, ...], " disagree _ statements ": [4, 5, 6, ...], " neutral _ statements ": [7, 8, 9, ...]} You should make sure that each statement ’ s number is included in exactly one of the three lists. Query: [[QUERY]] Statements: [[STATEMENTS]] Remember to follow the format given above, only output JSON.