## Diagram: Prompt Engineering Strategies for ConvFinQA ### Overview The image presents three diagrams illustrating different prompt engineering strategies (ZS-FinDSL, ZS-STD, and a third unnamed strategy) used with a Large Language Model (LLM) for the ConvFinQA task. Each diagram outlines the flow of information and processing steps involved in answering questions based on a given passage and table. ### Components/Axes **General Components (Present in all diagrams):** * **Input Text Box (Light Blue):** Contains the initial prompt instructions and data. * **LLM Icon:** Represents the Large Language Model processing the input. * **Output Text Box (Light Pink):** Displays the LLM's response or generated output. * **Arrows:** Indicate the flow of information between components. * **Legend (Bottom):** Defines the color-coded elements: * Black: Signifier * Orange: Memetic proxy * Green: Constraining behavior (Only in Figure 7) * Pink: Meta prompt * Blue: Input **Figure 7: ZS-FinDSL prompt for ConvFinQA** * **Top-Left Box (Reasoning extraction):** * Title: Reasoning extraction * Text: * Read the following passage and then answer the questions: * Passage: text + table * Questions: ask question? * Answer the questions by finding the relevant values and performing step by step calculations. * Answer: * **Top-Right Box (Program extraction):** * Title: Program extraction * Text: * Questions: ask question? * Answer: Answer with reasoning from LLM. * Task: From the above question-answer, extract the calculations that were performed to arrive at the answer to the last question. The calculations should be provided in the following format: * {"PROGRAM": {"#0": {OPERATION:"[arithmetic/logic]", ARG1:"[float/int]", ARG2:"[float/int]"}, "#1":{OPERATION: [arithmetic/logic], ARG1:"#0", ARG2:"[float/int/#int]"}, ...}, "ANSWER": "[numerical/boolean]"} * Operation should strictly be restricted to {add, subtract, multiply, divide, exponent, greater-than, max, min} only. * When evaluated the program should only generate numerical or boolean values. * Solution: * **Bottom-Left Box:** Answer with reasoning from LLM. * **Bottom-Right Box:** Program generated by the LLM. **Figure 8: ZS-STD prompt for ConvFinQA** * **Top-Left Box (LLM answering):** * Title: LLM answering * Text: * Read the following passage and then answer the questions: * Passage: text + table * Questions: ask question? * Answer: * **Top-Right Box (Answer extraction):** * Title: Answer extraction * Text: * Questions: ask question? * Answer: Answer from LLM. * The final answer (float/int/boolean) is: * **Bottom-Left Box:** Answer from LLM. * **Bottom-Right Box:** Final answer generated by the LLM. **Third Diagram (Unnamed)** * **Top-Left Box (Reasoning extraction):** * Title: Reasoning extraction * Text: * Read the following passage and then answer the questions: * Passage: text + table * Questions: ask question? * Answer: Let us think step by step. * **Top-Right Box (Answer extraction):** * Title: Answer extraction * Text: * Questions: ask question? * Answer: Answer with reasoning from LLM. * The final answer (float/int/boolean) is: * **Bottom-Left Box:** Answer with reasoning from LLM. * **Bottom-Right Box:** Final answer generated by the LLM. ### Detailed Analysis or ### Content Details **Figure 7: ZS-FinDSL prompt for ConvFinQA** 1. **Reasoning extraction:** The LLM receives a passage and table, and is prompted to answer questions by finding relevant values and performing step-by-step calculations. 2. **Program extraction:** The LLM extracts the calculations performed to arrive at the answer. The calculations are formatted as a program with arithmetic/logic operations on float/int values. The output is a numerical or boolean answer. 3. The "Reasoning extraction" box connects to the "Program extraction" box with a gray arrow. 4. The LLM icon below the "Reasoning extraction" box connects to the "Answer with reasoning from LLM" box with a downward arrow. 5. The LLM icon below the "Program extraction" box connects to the "Program generated by the LLM" box with a downward arrow. **Figure 8: ZS-STD prompt for ConvFinQA** 1. **LLM answering:** The LLM receives a passage and table, and is prompted to answer questions. 2. **Answer extraction:** The LLM extracts the final answer, which is a float, integer, or boolean value. 3. The "LLM answering" box connects to the "Answer extraction" box with a gray arrow. 4. The LLM icon below the "LLM answering" box connects to the "Answer from LLM" box with a downward arrow. 5. The LLM icon below the "Answer extraction" box connects to the "Final answer generated by the LLM" box with a downward arrow. **Third Diagram (Unnamed)** 1. **Reasoning extraction:** The LLM receives a passage and table, and is prompted to answer questions by thinking step by step. 2. **Answer extraction:** The LLM extracts the final answer, which is a float, integer, or boolean value. 3. The "Reasoning extraction" box connects to the "Answer extraction" box with a gray arrow. 4. The LLM icon below the "Reasoning extraction" box connects to the "Answer with reasoning from LLM" box with a downward arrow. 5. The LLM icon below the "Answer extraction" box connects to the "Final answer generated by the LLM" box with a downward arrow. ### Key Observations * Figure 7 (ZS-FinDSL) involves a two-step process: reasoning extraction followed by program extraction. * Figure 8 (ZS-STD) is a simpler, direct answer extraction process. * The third diagram is similar to ZS-STD but includes the prompt "Let us think step by step" in the reasoning extraction phase. * The legend provides a color-coding scheme for different elements in the diagrams. ### Interpretation The diagrams illustrate different approaches to prompt engineering for question answering tasks using LLMs. ZS-FinDSL aims to extract the reasoning process as a program, while ZS-STD directly extracts the answer. The third diagram combines elements of both, prompting the LLM to think step by step before extracting the answer. The choice of prompt engineering strategy can significantly impact the LLM's performance and the interpretability of its reasoning process. The diagrams highlight the importance of carefully designing prompts to elicit the desired behavior from the LLM.

## Diagram: ZS-FinDSL and ZS-STD Prompts for ConvFinQA ### Overview The image contains three diagrams (Figure 7, 8, 9) illustrating workflows for a ConvFinQA system. Each diagram outlines steps for **Reasoning extraction**, **Program extraction**, **LLM answering**, and **Answer extraction**, with color-coded elements (Signifier, Mementic proxy, Constraining behavior, Meta prompt, Input) and directional arrows indicating process flow. --- ### Components/Axes #### Figure 7: ZS-FinDSL Prompt - **Components**: - **Reasoning extraction**: "Read the following passage and then answer the questions: Passage: text + table. Questions: ask question? Answer: Answer with reasoning from LLM." - **Program extraction**: "From the above question-answer, extract last calculations... Operation should strictly be restricted to [add, subtract, multiply, divide, exponent, greater-than, max, min] only." - **LLM**: Arrows connect to "Answer with reasoning from LLM" and "Program generated by the LLM." - **Legend**: - **Signifier** (black), **Mementic proxy** (orange), **Constraining behavior** (pink), **Meta prompt** (green), **Input** (blue). #### Figure 8: ZS-STD Prompt - **Components**: - **LLM answering**: "Read the following passage and then answer the questions: Passage: text + table. Questions: ask question? Answer: Answer from LLM." - **Answer extraction**: "The final answer (float/int/boolean) is: [answer]." - **LLM**: Arrows connect to "Answer from LLM" and "Final answer generated by the LLM." - **Legend**: - **Signifier** (black), **Mementic proxy** (orange), **Input** (blue). #### Figure 9: ZS-STD Prompt (Alternative) - **Components**: - **Reasoning extraction**: "Read the following passage and then answer the questions: Passage: text + table. Questions: ask question? Answer: Let us think step by step." - **Answer extraction**: "The final answer (float/int/boolean) is: [answer]." - **LLM**: Arrows connect to "Answer with reasoning from LLM" and "Final answer generated by the LLM." - **Legend**: - **Signifier** (black), **Mementic proxy** (orange), **Meta prompt** (green), **Input** (blue). --- ### Detailed Analysis #### Figure 7: ZS-FinDSL Prompt - **Textual Content**: - **Reasoning extraction** requires the LLM to generate answers with explicit reasoning. - **Program extraction** mandates strict adherence to predefined operations (e.g., no custom functions). - **LLM** outputs both reasoning and program code. - **Flow**: - Input (text + table) → Reasoning extraction → Program extraction → LLM → Program generation. #### Figure 8: ZS-STD Prompt - **Textual Content**: - **LLM answering** focuses on direct answers without explicit reasoning. - **Answer extraction** isolates the final numerical/boolean result. - **Flow**: - Input (text + table) → LLM answering → Answer extraction → LLM → Final answer. #### Figure 9: ZS-STD Prompt (Alternative) - **Textual Content**: - **Reasoning extraction** emphasizes step-by-step thinking ("Let us think step by step"). - **Answer extraction** mirrors Figure 8 but includes reasoning in the LLM output. - **Flow**: - Input (text + table) → Reasoning extraction → Answer extraction → LLM → Final answer. --- ### Key Observations 1. **Legend Consistency**: - **Signifier** (black) and **Input** (blue) are consistently used across all figures. - **Mementic proxy** (orange) appears in all figures, suggesting it represents contextual or auxiliary data. - **Constraining behavior** (pink) and **Meta prompt** (green) are unique to Figure 7 and 9, respectively. 2. **Process Differences**: - **Figure 7** emphasizes program generation from reasoning, while **Figures 8 and 9** focus on direct answer extraction. - **Figure 9** introduces a "step-by-step" reasoning prompt, distinct from the other figures. 3. **Color-Coding**: - **Signifier** (black) likely marks critical elements (e.g., questions, answers). - **Mementic proxy** (orange) may denote contextual or secondary information. --- ### Interpretation The diagrams illustrate two workflows for ConvFinQA: 1. **ZS-FinDSL (Figure 7)**: Designed for tasks requiring programmatic reasoning (e.g., financial calculations). The LLM generates both reasoning and executable code, constrained by strict operation rules. 2. **ZS-STD (Figures 8 and 9)**: Focuses on direct answer extraction, with variations in prompting (e.g., "step-by-step" reasoning in Figure 9). **Notable Trends**: - **Program extraction** (Figure 7) is more rigid, enforcing specific operations, while **answer extraction** (Figures 8/9) allows flexibility in output formats (float/int/boolean). - The inclusion of **Meta prompt** (green) in Figure 9 suggests additional constraints or metadata handling. **Implications**: - The system adapts to different task requirements: structured program generation vs. direct answer retrieval. - Color-coded elements (e.g., **Mementic proxy**) likely serve to differentiate contextual or auxiliary data from core inputs. --- **Note**: No numerical data or charts are present; the diagrams focus on process flow and textual instructions.