## Dialogue Comparison: Agent-as-tool-Base vs. Agent-as-tool-Instruct ### Overview The image presents a comparison of two agents, "Agent-as-tool-Base" and "Agent-as-tool-Instruct", attempting to answer the question: "Where did Edward Dunn (Bishop)'s father graduate from?". The expected answer is "Corpus Christi College, Cambridge". The image shows the step-by-step reasoning and actions of each agent, including their thoughts, tool calls, observations, and final answers. ### Components/Axes The image is divided into two main columns, one for each agent. Each column contains a sequence of dialogue turns, marked by tags such as `<think>`, `<tool_calling>`, `<obs>`, and `<answer>`. Orange boxes highlight key actions or conclusions of each agent. ### Detailed Analysis or ### Content Details **Agent-as-tool-Base (Left Column):** 1. **Initial Thought:** The agent starts by stating its need to find Edward Dunn's father's educational background. 2. **Tool Call:** The agent calls a tool to provide information on Edward Dunn's father's education. 3. **Observation:** The agent reports that it couldn't find specific information about Edward Dunn's father's education but found information related to Edward Dunn himself. It asks if the user needs information on a different topic. An orange box states "Planner try to ask 2 hops simultaneously". 4. **Thought:** The agent concludes that the observation did not provide the specific information it was looking for and that it needs to find a more direct way to get the information. An orange box states "The agent was unable to answer". 5. **Tool Call:** The agent calls a tool to provide information on the educational background of Edward Dunn's father. 6. **Observation:** The agent finds some relevant information regarding Edward Dunn's family background, but specific details about his father's educational background are limited. It provides some notable findings: * Edward T. Dunn studied theology and earned a master's degree in history and a PhD, but this relates to him rather than his father. * Bishop Edward Dunn's father, Andrew Hunter Dunn, was an education graduate from Marlborough and Pembroke College, Cambridge. * Other sources mainly discuss different individuals named Edward Dunn or their obituaries. 7. **Thought:** The agent concludes that it cannot get exact information. An orange box states "The agent cannot get exact information". 8. **Thought:** The agent states that the observation provided the specific information it needed: Edward Dunn's father, Andrew Hunter Dunn, graduated from Marlborough and Pembroke College, Cambridge. 9. **Answer:** The agent provides the answer "Marlborough and Pembroke College, Cambridge (Wrong Answer)". **Agent-as-tool-Instruct (Right Column):** 1. **Initial Thought:** The agent starts by stating its need to find information about Edward Dunn's father. 2. **Tool Call:** The agent calls a tool to provide information about Edward Dunn (Bishop)'s father. 3. **Observation:** The agent states that Edward Dunn, the bishop, was born into an ecclesiastical family. His father was Andrew Hunter Dunn, who at one time served as the Bishop of Quebec. It provides a link to Edward Dunn's Wikipedia page. An orange box states "Learned to Ask the prerequisite question before the next hop". 4. **Thought:** The agent concludes that it could answer this question easily. An orange box states "The agent could answer this question easily". From the observation, it sees that Edward Dunn's father was Andrew Hunter Dunn, who was the Bishop of Quebec. However, the observation does not specify where Andrew Hunter Dunn graduated from. It needs to find this information. 5. **Tool Call:** The agent calls a tool to provide information about where Andrew Hunter Dunn graduated from. 6. **Observation:** The agent states that Andrew Hunter Dunn graduated from Corpus Christi College, Cambridge, where he earned his BA as the 29th Wrangler in 1863. It provides a link to Andrew Hunter Dunn's Wikipedia page. An orange box states "Asked the next-hop question". 7. **Thought:** The agent concludes that it was able to answer the question easily. An orange box states "The agent was able to answer the question easily". Now it has the information that Andrew Hunter Dunn, the father of Edward Dunn (Bishop), graduated from Corpus Christi College, Cambridge. 8. **Answer:** The agent provides the answer "Corpus Christi College, Cambridge (Correct Answer)". ### Key Observations * Agent-as-tool-Base initially struggles to find the correct information and provides an incorrect answer. It attempts to ask 2 hops simultaneously. * Agent-as-tool-Instruct successfully answers the question by breaking it down into smaller steps and asking the prerequisite question before the next hop. ### Interpretation The comparison highlights the difference in performance between the two agents. Agent-as-tool-Instruct demonstrates a more effective reasoning process by breaking down the question into smaller, more manageable steps. This allows it to gather the necessary information and provide the correct answer. Agent-as-tool-Base, on the other hand, struggles to find the correct information and ultimately provides an incorrect answer, suggesting a less effective reasoning strategy. The orange boxes highlight the key differences in the agents' approaches, with Agent-as-tool-Instruct learning to ask the prerequisite question and Agent-as-tool-Base attempting to ask 2 hops simultaneously.

## Agent-as-tool-Instruct Dialogue Transcript ### Overview This image presents a transcript of a dialogue between a user and an "Agent-as-tool-Instruct" system, designed to answer the question: "Where did Edward Dunn (Bishop)'s father graduate from?". The transcript details the agent's thought process ("<think>") and actions ("<tool_calling>") as it attempts to find the answer, including multiple attempts and refinement of queries. The dialogue is structured as a series of turns, with "obs" representing the agent's observations or responses from external tools (likely a search engine or knowledge base). ### Components/Axes The transcript is organized into two columns: "Result from Agent-as-tool-Base" and "Result from Agent-as-tool-Instruct". Each column displays the agent's internal reasoning and actions in a turn-based manner. The turns are numbered sequentially. Key elements within each turn include: * **`<think>`**: The agent's internal thought process. * **`<tool_calling>`**: The agent's attempt to use a tool (e.g., a search engine). * **`obs`**: The agent's observation or the result from the tool. ### Detailed Analysis or Content Details Here's a detailed transcription of the dialogue, turn by turn: **Turn 1:** * **Question:** Where did Edward Dunn (Bishop)'s father graduate from? * **Expected Answer:** Corpus Christi College, Cambridge **Turn 2 (Agent-as-tool-Base):** * `<think>`: I need to find Edward Dunn's father's educational background. * `<tool_calling>`: Can you provide information on Edward Dunn's father's education? Planner try to ask 2 hops simultaneously * `obs`: I wasn't able to find specific information about Edward Dunn's father's education. However, I did find information related to Edward Dunn himself, including his education and achievements. If you need information on a different topic or a specific Edward Dunn, please let me know! **Turn 3 (Agent-as-tool-Instruct):** * `<think>`: I need to find information about Edward Dunn (Bishop)'s father. * `<tool_calling>`: Can you provide information about Edward Dunn (Bishop)'s father? Learned to Ask the prerequisite question before the next hop * `obs`: Edward Dunn, the bishop, was born into an ecclesiastical family. His father was Andrew Hunter Dunn, who at one time served as the Bishop of Quebec. For more details, you can read the full biography on Wikipedia ([https://en.wikipedia.org/wiki/Edward_Dunn_(bishop)](https://en.wikipedia.org/wiki/Edward_Dunn_(bishop))). The agent could answer this question easily. **Turn 4 (Agent-as-tool-Base):** * `<think>`: The observation did not provide the specific information I was looking for. I need to find a more direct way to get the information about Edward Dunn's father's education. * `<tool_calling>`: Can you provide information on the educational background of Edward Dunn's father? Planner asked 2 hops in the meantime in another manner * `obs`: I found some relevant information regarding Edward Dunn's family background, but specific details about his father's education background are limited. Here are some notable findings: 1. "*Edward T. Dunn*” – He is noted to have had theology and earned a master's degree in history and a PhD, but this relates to his own rather than his father. 2. Source: [https://books.google.com/books?id=UCwXAAAAIAAJ](https://books.google.com/books?id=UCwXAAAAIAAJ) 3. "*Hunter Dunn*” – There is a reference to a Hunter Dunn who was a graduate of Corpus Christi College, Cambridge, in 1888. 4. Source: [https://www.thepeerage.com/p28886.htm](https://www.thepeerage.com/p28886.htm) 5. However, it is not explicitly stated that this Hunter Dunn is the father of Edward Dunn (Bishop). **Turn 5 (Agent-as-tool-Instruct):** * `<think>`: From the observation, I can see that Edward Dunn's father was Andrew Hunter Dunn, who was the Bishop of Quebec. However, the observation does not specify where Andrew Hunter Dunn graduated from. I need to find this information. * `<tool_calling>`: Can you provide information about where Andrew Hunter Dunn graduated from? Asked the next-hop question * `obs`: Andrew Hunter Dunn graduated from **Corpus Christi College, Cambridge** where he earned his BA as the #25th Wrangler in 1868. For more detailed information, you can visit his [Wikipedia page](https://en.wikipedia.org/wiki/Andrew_Hunter_Dunn). [https://www.wikipedia.org/Andrew_Hunter_Dunn](https://en.wikipedia.org/wiki/Andrew_Hunter_Dunn) (In reply to: Can you provide information about where Andrew Hunter Dunn graduated from?) [https://en.wikipedia.org/wiki/Andrew_Hunter_Dunn](https://en.wikipedia.org/wiki/Andrew_Hunter_Dunn) He was the son of Rev. John Dunn, DD, of Quebec. ### Key Observations * The "Agent-as-tool-Instruct" approach demonstrates a more strategic and iterative query refinement process compared to the "Agent-as-tool-Base" approach. * The "Instruct" agent explicitly states its reasoning for asking prerequisite questions ("Learned to Ask the prerequisite question before the next hop"). * The initial attempts by both agents failed to directly answer the question, highlighting the need for multi-hop reasoning. * The final answer, "Corpus Christi College, Cambridge," was found through a combination of identifying the father's name (Andrew Hunter Dunn) and then querying his educational background. * The "Instruct" agent successfully navigated ambiguous information (the existence of multiple "Hunter Dunns") to pinpoint the correct individual. ### Interpretation This dialogue illustrates the power of agent-based systems with iterative reasoning capabilities. The "Instruct" agent's ability to break down the complex question into smaller, more manageable sub-questions, and to learn from previous failures, is crucial for achieving accurate results. The transcript demonstrates a successful application of the "Ask before the next hop" strategy, which helps to ensure that the agent has the necessary context before attempting to answer the final question. The process highlights the importance of not only retrieving information but also of understanding the relationships between different pieces of information to arrive at a correct and complete answer. The agent's ability to identify and resolve ambiguity (multiple individuals with similar names) is also a key factor in its success. The final answer confirms the expected answer provided at the beginning of the transcript.

## [Diagram/Comparison]: Comparison of Two AI Agent Approaches to a Multi-Hop Question ### Overview The image is a side-by-side comparison of the reasoning processes and outputs of two different AI agent systems ("Agent-as-tool-Base" and "Agent-as-tool-Instruct") when answering the same factual question. The comparison is presented in a structured, annotated format to highlight the differences in their problem-solving strategies and final accuracy. ### Components/Axes The image is organized into three main sections: 1. **Header (Top Blue Bar):** Contains the core question and the ground-truth answer. 2. **Left Column:** Titled "Result from Agent-as-tool-Base". Shows the step-by-step trace of this agent's reasoning, tool use, and final answer. 3. **Right Column:** Titled "Result from Agent-as-tool-Instruct". Shows the step-by-step trace of this agent's reasoning, tool use, and final answer. **Annotations:** Throughout both columns, orange rectangular boxes with white text provide meta-commentary on the agent's behavior at specific steps. These are connected to the relevant text with thin lines. ### Detailed Analysis #### **Header Section** * **Question:** "Where did Edward Dunn (Bishop)'s father graduate from?" * **Expected Answer:** "Corpus Christi College, Cambridge" #### **Left Column: Agent-as-tool-Base** This column shows a process that ultimately fails to find the correct answer. 1. **Initial `<think>`:** The agent states its goal: "I need to find Edward Dunn's father's educational background." 2. **First `<tool_calling>`:** The agent asks: "Can you provide information on Edward Dunn's father's education?" * **Annotation (Orange Box, positioned to the right):** "Planner try to ask 2 hops simultaneously" 3. **First `<obs>` (Observation):** The tool response states it couldn't find specific information about the father's education but found information about Edward Dunn himself. It offers to help with a different topic. * **Annotation (Orange Box, positioned below the observation):** "The agent was unable to answer" 4. **Second `<think>`:** The agent acknowledges the observation didn't provide the needed information and decides to find a "more direct way." 5. **Second `<tool_calling>`:** The agent asks: "Can you provide information on the educational background of Edward Dunn's father?" * **Annotation (Orange Box, positioned to the right):** "Planner asked 2 hops in the meantime in another manner" 6. **Second `<obs>`:** A lengthy response. It lists findings: * 1. Information about **Edward T. Dunn** (the bishop himself), noting his studies. * 2. Information about **Bishop Edward Dunn**, stating: "His father, Andrew Hunter Dunn, was an education graduate from Marlborough and Pembroke College, Cambridge." This is sourced from a Wikipedia link. * 3. Notes that other sources discuss different individuals. * **Annotation (Orange Box, positioned below this observation):** "The agent cannot get exact information" 7. **Third `<think>`:** The agent incorrectly concludes from the observation: "Edward Dunn's father, Andrew Hunter Dunn, graduated from Marlborough and Pembroke College, Cambridge." 8. **Final `<answer>`:** "Marlborough and Pembroke College, Cambridge (Wrong Answer)" #### **Right Column: Agent-as-tool-Instruct** This column shows a successful, stepwise process. 1. **Initial `<think>`:** The agent states its goal: "I need to find information about Edward Dunn (Bishop)'s father." 2. **First `<tool_calling>`:** The agent asks: "Can you provide information about Edward Dunn (Bishop)'s father?" * **Annotation (Orange Box, positioned to the right):** "Learned to Ask the prerequisite question before the next hop" 3. **First `<obs>`:** The tool response identifies the father: "His father was Andrew Hunter Dunn, who at one time served as the Bishop of Quebec." It provides a Wikipedia link. * **Annotation (Orange Box, positioned below the observation):** "The agent could answer this question easily" 4. **Second `<think>`:** The agent processes the observation, noting it now knows the father's name (Andrew Hunter Dunn) but not his graduation location. It states the need to find this next piece of information. 5. **Second `<tool_calling>`:** The agent asks the specific follow-up: "Can you provide information about where Andrew Hunter Dunn graduated from?" * **Annotation (Orange Box, positioned to the right):** "Asked the next-hop question" 6. **Second `<obs>`:** The tool response provides the precise answer: "Andrew Hunter Dunn graduated from **Corpus Christi College, Cambridge**, where he earned his BA as the 29th Wrangler in 1863." It provides a Wikipedia link. * **Annotation (Orange Box, positioned below the observation):** "The agent was able to answer the question easily" 7. **Third `<think>`:** The agent confirms it now has the correct information. 8. **Final `<answer>`:** "Corpus Christi College, Cambridge (Correct Answer)" ### Key Observations 1. **Strategic Difference:** The core difference is in the initial query strategy. The "Base" agent attempts to solve the multi-hop question ("father's education") in a single, broad query. The "Instruct" agent breaks it down, first identifying the father (prerequisite hop) and then querying for his education (target hop). 2. **Information Quality:** The "Base" agent's second observation contained the correct answer ("Marlborough and Pembroke College, Cambridge") but also extraneous, potentially confusing information about other individuals. The agent failed to parse it correctly. 3. **Annotation Role:** The orange annotations explicitly diagnose the "Base" agent's failure mode ("try to ask 2 hops simultaneously") and the "Instruct" agent's success strategy ("Learned to Ask the prerequisite question"). 4. **Outcome:** The "Agent-as-tool-Base" produces a wrong answer, while the "Agent-as-tool-Instruct" produces the correct answer that matches the expected answer in the header. ### Interpretation This diagram serves as a technical case study demonstrating the importance of **sequential reasoning** and **question decomposition** in AI agents designed for multi-hop question answering. * **What it demonstrates:** It visually argues that an agent trained or instructed to break down complex queries into a series of simpler, prerequisite sub-questions ("Instruct") is more robust and accurate than an agent that attempts to resolve the entire query in one step ("Base"). The "Base" agent's approach is prone to failure because it either gets no answer or an answer mixed with irrelevant data, leading to incorrect synthesis. * **Underlying Principle:** The success of the "Instruct" agent aligns with the Peircean investigative concept of **abductive reasoning**—forming the best explanation by sequentially gathering necessary facts. It first establishes the identity of the subject (the father) before inquiring about a property of that subject (his education). * **Practical Implication:** For developers building AI systems that use tools (like search or databases) to answer complex questions, this comparison highlights that the agent's **planning and orchestration strategy** is as critical as the quality of the underlying tools or data. The "Instruct" agent's trace exemplifies an effective, logical workflow.

## Screenshot: Agent Comparison for Multi-Hop Question Answering ### Overview The image compares two AI agent implementations (Agent-as-tool-Base vs. Agent-as-tool-Instruct) attempting to answer the question: "Where did Edward Dunn (Bishop)'s father graduate from?" The expected answer is **Corpus Christi College, Cambridge**. The comparison shows the agents' reasoning processes, tool calls, and final answers through a structured dialogue format with color-coded annotations. --- ### Components/Axes 1. **Left Panel (Agent-as-tool-Base)**: - **Structure**: Sequential thought process with tool calls and observations. - **Annotations**: Orange boxes highlight key decision points (e.g., "Planner try to ask 2 hops simultaneously"). - **Final Answer**: Incorrectly identifies **Marlborough and Pembroke College, Cambridge**. 2. **Right Panel (Agent-as-tool-Instruct)**: - **Structure**: Step-by-step reasoning with explicit tool calls and observations. - **Annotations**: Orange boxes emphasize strategic decisions (e.g., "Learned to Ask the prerequisite question before the next hop"). - **Final Answer**: Correctly identifies **Corpus Christi College, Cambridge**. --- ### Detailed Analysis #### Left Panel (Agent-as-tool-Base) 1. **Initial Query**: Asks for information on Edward Dunn's father's education. 2. **First Observation**: Fails to find specific details about the father's education but provides info on Edward Dunn's own education. 3. **Second Attempt**: Tries to ask two hops simultaneously but receives no useful data. 4. **Final Answer**: Incorrectly cites Marlborough and Pembroke College, Cambridge, based on incomplete data. #### Right Panel (Agent-as-tool-Instruct) 1. **Initial Query**: Asks for information on Edward Dunn's father. 2. **First Observation**: Identifies Andrew Hunter Dunn as the father and notes his role as Bishop of Quebec. 3. **Second Attempt**: Explicitly asks where Andrew Hunter Dunn graduated from. 4. **Final Answer**: Correctly cites Corpus Christi College, Cambridge, using precise tool calls. --- ### Key Observations 1. **Agent-as-tool-Base**: - Struggles with multi-hop reasoning, attempting parallel queries that yield no results. - Final answer is incorrect due to reliance on ambiguous or unrelated data. 2. **Agent-as-tool-Instruct**: - Uses sequential, targeted tool calls to isolate the correct information. - Final answer matches the expected result, demonstrating effective structured reasoning. --- ### Interpretation The comparison highlights the importance of **structured, sequential reasoning** in multi-hop question answering. The Agent-as-tool-Instruct successfully isolates the correct information by: 1. **Asking prerequisite questions** before proceeding (e.g., identifying the father first). 2. **Explicitly targeting** the father's educational background after establishing his identity. 3. **Leveraging precise tool calls** to extract specific data points. In contrast, the Agent-as-tool-Base's attempt to handle multiple hops simultaneously leads to confusion and incorrect conclusions. This underscores the value of **stepwise, hypothesis-driven reasoning** in complex information retrieval tasks.