## Diagram: Comparison of Question Answering Methods ### Overview The image presents a diagram comparing three methods for answering the question: "When did the films release whose screenwriters also wrote Analyze That?". The methods compared are Semantic Parsing, Retrieval-Augmented, and "Our Method". The diagram illustrates the process flow for each method, highlighting their strengths and weaknesses. ### Components/Axes * **Title:** Question: When did the films release whose screenwriters also wrote Analyze That? * **(1) Semantic Parsing Methods:** Describes the first method. * **Input:** A hand pointing to a question mark inside a speech bubble. * **Process:** The input is processed by an AI agent, then converted to SPARQL. SPARQL is executed on an Incomplete KG (Knowledge Graph). * **Output:** No Answer (indicated by a red "X"). * **(2) Retrieval-Augmented Methods:** Describes the second method. * **Input:** The question "Which screenwriters wrote Analyze That?" (yellow background). * **Process:** The question is decomposed, resulting in an error (red "X"). The question is then changed to "When was Harold Ramis released?" (green background). This is used to query a model represented by a logo similar to OpenAI's logo. The model uses Harold Ramis to query an Incomplete KG. * **Output:** No Answer (indicated by a red "X"). * **Our Method:** Describes the third method. * **Input:** The year "1999..." (green checkmark). * **Process:** An AI agent processes the input using a multi-turn approach. The AI agent interacts with both an Incomplete KG and the Web. * **Plan:** A list of steps is provided: * Step 1: Which screenwriters wrote Analyze that? Ans1 = SearchKG(screenwriter|...) * Step 2: Which films written by Ans1? Ans2 = SearchKG(films|Ans1, ...) * Step 3: When did the Ans 2 release? Ans3 = SearchKG(time|Ans2, ...) ### Detailed Analysis * **Semantic Parsing Methods:** * The process starts with a question. * The question is converted into a SPARQL query. * The SPARQL query is executed against an incomplete knowledge graph (KG). * The method fails to produce an answer. * **Retrieval-Augmented Methods:** * The process starts with the question "Which screenwriters wrote Analyze That?". * The question is decomposed, but this results in an error. * The question is changed to "When was Harold Ramis released?". * The new question is used to query a model, which then queries an incomplete KG. * The method fails to produce an answer. * **Our Method:** * The process starts with the year "1999...". * An AI agent uses a multi-turn approach to interact with both an incomplete KG and the Web. * The method follows a plan consisting of three steps: * Step 1: Find the screenwriters of "Analyze That". * Step 2: Find the films written by those screenwriters. * Step 3: Find the release date of those films. ### Key Observations * Both Semantic Parsing and Retrieval-Augmented methods fail to answer the question. * "Our Method" successfully answers the question by using a multi-turn approach and interacting with both an incomplete KG and the Web. * "Our Method" involves a plan with three steps, each involving a search of a knowledge graph (SearchKG). ### Interpretation The diagram illustrates the limitations of traditional question-answering methods (Semantic Parsing and Retrieval-Augmented) when dealing with complex questions that require multiple steps and external knowledge. "Our Method" overcomes these limitations by using a multi-turn approach, combining knowledge from an incomplete KG with information from the Web. The success of "Our Method" highlights the importance of combining different sources of information and using a structured approach to answer complex questions. The diagram suggests that a more sophisticated approach is needed to answer complex questions that require multiple steps and external knowledge.

\n ## Diagram: Knowledge Graph Querying Methods Comparison ### Overview This diagram compares three methods for answering the question: "When did the films release whose screenwriters also wrote Analyze That?". The methods are Semantic Parsing, Retrieval-Augmented Methods, and a novel "Our Method". The diagram illustrates the flow of information and potential failure points for each approach, highlighting the challenges of incomplete knowledge graphs (KG). ### Components/Axes The diagram is divided into three horizontal sections, each representing a different method. Each section contains visual elements representing the query process, including: * **Question:** The initial query posed to the system. * **AI/Agent:** Represented by a brain icon, symbolizing the AI component. * **Knowledge Graph (KG):** Represented by interconnected nodes and edges. * **Error/Failure:** Indicated by a red "X" symbol. * **Output:** "No Answer" or a series of steps in a plan. * **Web:** Represented by a browser icon. * **Plan:** A list of steps to answer the question. * **Harold Ramis:** A portrait of Harold Ramis. ### Detailed Analysis or Content Details **Section 1: Semantic Parsing Methods** * **Question:** "When did the films release whose screenwriters also wrote Analyze That?" * The AI attempts to "execute" a SPARQL query on the KG. * The KG is labeled "Incomplete KG". * The process results in "No Answer", indicated by a red "X". **Section 2: Retrieval-Augmented Methods** * **Question:** "When did the films release whose screenwriters also wrote Analyze That?" * The method first decomposes the question into two sub-questions: "Which screenwriters wrote Analyze That?" and "When was Harold Ramis released?". * The KG is labeled "Incomplete KG" within a dashed box. * A portrait of Harold Ramis is shown. * The process results in "No Answer", indicated by a red "X". **Section 3: Our Method** * **Question:** "When did the films release whose screenwriters also wrote Analyze That?" * The AI interacts with the web in a "Multi-turn" fashion. * The KG is labeled "Incomplete KG" within a dashed box. * The AI generates a "Plan" with three steps: * Step 1: "Which screenwriters wrote Analyze That?" Ans1 = SearchKG(screenwriter(...)) * Step 2: "Which films written by Ans1?" Ans2 = SearchKG(filmsAns1, ...) * Step 3: "When did the Ans 2 release?" Ans3 = SearchKG(timeAns2, ...) * The output is "1999, ...", suggesting a successful answer. ### Key Observations * Both Semantic Parsing and Retrieval-Augmented methods fail to provide an answer due to the "Incomplete KG". * "Our Method" successfully generates a plan and provides a partial answer ("1999, ..."). * "Our Method" utilizes a multi-turn interaction with the web, suggesting it can leverage external information to overcome the limitations of the KG. * The decomposition of the question in Retrieval-Augmented Methods leads to a specific error case involving Harold Ramis. ### Interpretation The diagram demonstrates the challenges of answering complex knowledge graph queries when the KG is incomplete. Semantic Parsing relies on a complete and accurate KG to directly execute queries, while Retrieval-Augmented methods attempt to decompose the problem but still struggle with KG limitations. "Our Method" offers a more robust approach by leveraging external information (the web) through a multi-turn interaction, allowing it to overcome the KG's incompleteness and generate a plausible answer. The inclusion of Harold Ramis in the Retrieval-Augmented section suggests a potential issue with identifying the correct entities or relationships within the KG. The diagram highlights the importance of combining KG querying with external knowledge sources for more reliable and comprehensive question answering. The "1999, ..." output suggests that the method is capable of providing at least partial results, even if it doesn't have complete information. The use of dashed boxes around the "Incomplete KG" in sections 2 and 3 suggests that the method is aware of the KG's limitations and attempts to mitigate them.

## Diagram: Comparison of Question Answering Methods Over Incomplete Knowledge Graphs ### Overview The image is a technical diagram comparing three different approaches to answering a complex, multi-hop natural language question using a knowledge graph (KG). The central example question is: **"When did the films release whose screenwriters also wrote Analyze That?"** The diagram illustrates the failure modes of two existing methods and proposes a new, successful method. ### Components/Axes The diagram is divided into three main rectangular panels, stacked vertically. 1. **Top Panel (Question):** Contains the primary question in a rounded rectangle. 2. **Middle Panel (Existing Methods):** Contains two sub-sections: * **(1) Semantic Parsing Methods:** A linear flow from a user query to a failed outcome. * **(2) Retrieval-Augmented Methods:** A cyclical flow showing a decomposition error leading to a failed outcome. 3. **Bottom Panel (Our Method):** Depicts a successful, multi-turn interactive process with a planning component. **Key Visual Components & Icons:** * **User/Query Icon:** A hand pointing at a speech bubble with a question mark (top-left of Semantic Parsing). * **AI Agent Icon:** A blue robot head labeled "AI". Appears in Semantic Parsing and Our Method. * **Knowledge Graph (KG) Icon:** A network of connected blue nodes. Labeled "Incomplete KG" in all instances. * **Red "X" Mark:** Indicates a failure, error, or dead end in the process. * **Green Checkmark:** Indicates a successful outcome (in "Our Method"). * **Web/Database Icon:** A computer monitor with code and a database symbol (in "Our Method"). * **Plan Icon:** A clipboard with a checklist (in "Our Method"). * **Arrows:** Black arrows indicate process flow. Red arrows indicate a failed path. Green arrow indicates successful output. Dashed arrows indicate multi-turn interaction. ### Detailed Analysis #### **Section 1: Semantic Parsing Methods** * **Flow:** User Query → AI Agent → **SPARQL** (with a red X above it, indicating generation failure) → execute → **Incomplete KG** → (red arrow) → **No Answer**. * **Text Transcription:** "execute", "SPARQL", "Incomplete KG", "No Answer". * **Interpretation:** This method attempts to directly translate the natural language question into a formal SPARQL query. The red "X" suggests this translation fails, or the generated query fails when executed against an incomplete knowledge graph, resulting in no answer. #### **Section 2: Retrieval-Augmented Methods** * **Flow:** This section shows a two-step decomposition that contains a logical error. 1. **Step 1 (Yellow Box):** "Which screenwriters wrote Analyze That?" 2. **Error:** A red "X" and the text **"Decompose Error"** point to the next step. 3. **Step 2 (Green Box):** "When was Harold Ramis released?" (This is an incorrect sub-question; it should ask for films, not the release date of a person). 4. **Retrieval Loop:** The process then enters a loop between an LLM icon (OpenAI logo) and the "Incomplete KG". It retrieves "Harold Ramis" but then fails (red X) to find the answer to the malformed second question, leading to **"No Answer"**. * **Text Transcription:** "Which screenwriters wrote Analyze That?", "Decompose Error", "When was Harold Ramis released?", "Harold Ramis", "Incomplete KG", "No Answer". * **Interpretation:** This method tries to break the complex question into simpler sub-questions. However, it makes a critical decomposition error by mis-framing the second sub-question, which leads to an unrecoverable failure when querying the incomplete KG. #### **Section 3: Our Method** * **Flow:** This depicts a successful, iterative process. 1. **Central Agent:** The AI agent icon is at the center, engaging in **"Multi-turn"** interaction (labeled on dashed arrows) with two resources: an **"Incomplete KG"** and the **"Web"**. 2. **Planning Component (Right Side):** A "Plan" box outlines a correct three-step decomposition: * **Step 1:** "Which screenwriters wrote Analyze that?" → `Ans1 = SearchKG(screenwriter|...)` * **Step 2:** "Which films written by Ans1?" → `Ans2 = SearchKG(films|Ans1, ...)` * **Step 3:** "When did the Ans 2 release?" → `Ans3 = SearchKG(time|Ans2, ...)` 3. **Successful Output:** A green arrow points from the AI agent to the final answer: **"1999, ..."** with a green checkmark above it. * **Text Transcription:** "Our Method", "Plan", "Step 1: Which screenwriters wrote Analyze that?", "Ans1 = SearchKG(screenwriter|...)", "Step 2: Which films written by Ans1?", "Ans2 = SearchKG(films|Ans1, ...)", "Step 3: When did the Ans 2 release?", "Ans3 = SearchKG(time|Ans2, ...)", "Multi-turn", "Incomplete KG", "Web", "1999, ...". * **Interpretation:** The proposed method uses a planning module to correctly decompose the complex question into a logical sequence of sub-questions. The AI agent then interactively gathers information from both an incomplete KG and the web in a multi-turn fashion to answer each sub-question in order, ultimately producing the correct answer (e.g., "1999" for the release year of a film like *Analyze That*). ### Key Observations 1. **Consistent Failure Symbolism:** The red "X" is used uniformly to denote points of failure across different methods. 2. **Spatial Grounding of Legend:** The "Incomplete KG" icon is consistently placed at the point of data retrieval in all three methods. The "AI" agent is positioned as the central processor in the first and third methods. 3. **Critical Error Highlighted:** The "Decompose Error" in the Retrieval-Augmented method is explicitly called out in red text, identifying it as the root cause of failure for that approach. 4. **Answer Specificity:** The successful output "1999, ..." implies the answer is a list of release years, with "1999" being the first and most prominent (likely for *Analyze That* itself or a related film). ### Interpretation This diagram is an argument for a specific architectural approach to complex question answering over imperfect data. It demonstrates that: * **Direct translation (Semantic Parsing)** is brittle and fails with incomplete data. * **Naive decomposition (Retrieval-Augmented)** can introduce logical errors that derail the entire process. * **The proposed solution** combines **explicit, correct planning** with **interactive, multi-source retrieval**. The key innovation is the agent's ability to iteratively consult both a structured (but incomplete) KG and the unstructured web, guided by a coherent plan, to compensate for the KG's shortcomings. The diagram suggests that robustness in QA systems comes not from a perfect knowledge base, but from a resilient process that can plan, decompose, and gather evidence from multiple sources dynamically. The final answer "1999, ..." serves as proof of concept for the method's efficacy.

## Flowchart: Comparative Analysis of Question Answering Methods for Incomplete Knowledge Graphs ### Overview The diagram compares three approaches to answering complex questions about film releases and screenwriters using incomplete knowledge graphs (KG). It visually demonstrates failure modes of traditional methods and validates a proposed multi-turn interaction strategy. ### Components/Axes 1. **Semantic Parsing Methods** (Top Section) - Input: Hand with question bubble ("When did the films release whose screenwriters also wrote Analyze That?") - Process: SPARQL query execution - Output: Red X → "Incomplete KG" → "No Answer" 2. **Retrieval-Augmented Methods** (Middle Section) - Input: Decomposed questions: - "Which screenwriters wrote Analyze That?" (Red X → "Decompose Error") - "When was Harold Ramis released?" (Red X → "No Answer") - Output: Cyclic dependency → "Incomplete KG" → "No Answer" 3. **Proposed Method** (Bottom Section) - Input: Multi-turn interaction between AI and user - Process: - Step 1: SearchKG(screenwriter|Analyze That?) → Ans1 - Step 2: SearchKG(films|Ans1) → Ans2 - Step 3: SearchKG(time|Ans2) → Ans3 - Output: Green checkmark → "1999" (correct answer) ### Detailed Analysis - **KG Representation**: - Incomplete KG shown as fragmented node networks (blue nodes with purple edges) - Web interface depicted with code snippets (HTML/CSS/JS) and database icons - **Method Comparison**: - Semantic Parsing: Direct query fails due to KG incompleteness - Retrieval-Augmented: Question decomposition fails due to missing entity links - Proposed Method: Iterative KG queries successfully resolve through entity chaining ### Key Observations 1. **Failure Patterns**: - All methods fail when facing entity disambiguation (e.g., "Analyze That" vs. "Harold Ramis") - Cyclic dependencies in retrieval-augmented approaches create infinite loops 2. **Success Factors**: - Multi-turn interaction enables progressive query refinement - Entity chaining (screenwriter → films → release date) resolves KG gaps - Temporal resolution (1999) requires cross-domain KG integration ### Interpretation The diagram reveals fundamental limitations in current QA systems when facing incomplete KGs: 1. **Structural Weakness**: Traditional methods assume perfect KG coverage, failing when entities or relations are missing 2. **Query Complexity**: Multi-hop questions require contextual understanding beyond simple key-value lookups 3. **Proposed Solution**: The multi-turn approach mimics human reasoning by: - Breaking complex queries into atomic sub-questions - Using intermediate answers to guide subsequent searches - Leveraging web resources to compensate for KG gaps This suggests that effective QA systems must combine: - KG query capabilities - Web search integration - Conversational context management - Entity resolution mechanisms