## Bar Chart: Prediction Flip Rate Comparison for Llama-3 Models ### Overview The image presents two bar charts comparing the prediction flip rates of two language models, Llama-3-8B and Llama-3-70B, across four different datasets: PopQA, TriviaQA, HotpotQA, and NQ. The charts compare the flip rates when the question is anchored to the question itself (Q-Anchored) versus when it's anchored to the answer (A-Anchored). ### Components/Axes * **Titles:** * Left Chart: Llama-3-8B * Right Chart: Llama-3-70B * **Y-Axis:** Prediction Flip Rate (ranging from 0 to 80, with tick marks at 20, 40, 60, and 80) * **X-Axis:** Dataset (categorical): PopQA, TriviaQA, HotpotQA, NQ * **Legend:** Located at the bottom of the image. * Q-Anchored (exact\_question): Represented by a light brown/reddish bar. * A-Anchored (exact\_question): Represented by a gray bar. ### Detailed Analysis **Llama-3-8B (Left Chart):** * **PopQA:** * Q-Anchored: Approximately 64 * A-Anchored: Approximately 22 * **TriviaQA:** * Q-Anchored: Approximately 87 * A-Anchored: Approximately 55 * **HotpotQA:** * Q-Anchored: Approximately 49 * A-Anchored: Approximately 9 * **NQ:** * Q-Anchored: Approximately 73 * A-Anchored: Approximately 19 **Llama-3-70B (Right Chart):** * **PopQA:** * Q-Anchored: Approximately 95 * A-Anchored: Approximately 52 * **TriviaQA:** * Q-Anchored: Approximately 70 * A-Anchored: Approximately 23 * **HotpotQA:** * Q-Anchored: Approximately 63 * A-Anchored: Approximately 12 * **NQ:** * Q-Anchored: Approximately 40 * A-Anchored: Approximately 16 ### Key Observations * For both models and across all datasets, the Q-Anchored flip rate is consistently higher than the A-Anchored flip rate. * Llama-3-70B generally exhibits higher prediction flip rates compared to Llama-3-8B, especially for PopQA. * TriviaQA shows the highest Q-Anchored flip rate for Llama-3-8B, while PopQA shows the highest Q-Anchored flip rate for Llama-3-70B. * HotpotQA consistently has the lowest A-Anchored flip rates for both models. ### Interpretation The data suggests that anchoring the question to the question itself (Q-Anchored) leads to a higher prediction flip rate compared to anchoring it to the answer (A-Anchored). This could indicate that the models are more sensitive to variations or perturbations in the question itself. The larger Llama-3-70B model generally shows higher flip rates, potentially indicating a greater sensitivity or complexity in its decision-making process. The differences in flip rates across datasets suggest that the models' robustness varies depending on the type of questions being asked. The lower A-Anchored flip rates for HotpotQA might indicate that the model is more confident or stable in its predictions for this particular dataset when the answer is the anchor.

\n ## Bar Chart: Prediction Flip Rate for Llama Models ### Overview This image presents a comparative bar chart illustrating the Prediction Flip Rate for two Llama models (Llama-3-8B and Llama-3-70B) across four different datasets: PopQA, TriviaQA, HotpotQA, and NQ. The flip rate is measured on the Y-axis, while the datasets are displayed on the X-axis. Two types of anchoring are compared: Q-Anchored (based on the exact question) and A-Anchored (based on the exact answer). ### Components/Axes * **X-axis:** "Dataset" with categories: PopQA, TriviaQA, HotpotQA, NQ. * **Y-axis:** "Prediction Flip Rate" with a scale ranging from 0 to 60 (approximately). * **Models:** Two separate charts are presented side-by-side, one for "Llama-3-8B" and one for "Llama-3-70B". * **Legend:** Located at the bottom-center of the image. * Red bars: "Q-Anchored (exact\_question)" * Gray bars: "A-Anchored (exact\_question)" ### Detailed Analysis **Llama-3-8B Chart:** * **PopQA:** Q-Anchored: approximately 55. A-Anchored: approximately 25. * **TriviaQA:** Q-Anchored: approximately 95. A-Anchored: approximately 50. * **HotpotQA:** Q-Anchored: approximately 45. A-Anchored: approximately 10. * **NQ:** Q-Anchored: approximately 60. A-Anchored: approximately 20. **Llama-3-70B Chart:** * **PopQA:** Q-Anchored: approximately 75. A-Anchored: approximately 50. * **TriviaQA:** Q-Anchored: approximately 60. A-Anchored: approximately 25. * **HotpotQA:** Q-Anchored: approximately 50. A-Anchored: approximately 20. * **NQ:** Q-Anchored: approximately 40. A-Anchored: approximately 20. **Trends:** * In both models, the Q-Anchored bars are consistently higher than the A-Anchored bars across all datasets, indicating a higher prediction flip rate when anchoring on the question. * For Llama-3-8B, the highest flip rate is observed for TriviaQA (Q-Anchored), and the lowest for HotpotQA (A-Anchored). * For Llama-3-70B, the highest flip rate is observed for PopQA (Q-Anchored), and the lowest for HotpotQA (A-Anchored). ### Key Observations * The Llama-3-70B model generally exhibits higher prediction flip rates than the Llama-3-8B model, particularly for the PopQA and TriviaQA datasets. * The difference between Q-Anchored and A-Anchored flip rates is more pronounced for the Llama-3-8B model. * HotpotQA consistently shows the lowest flip rates for both models and both anchoring methods. ### Interpretation The data suggests that anchoring the prediction process on the question (Q-Anchored) leads to a higher rate of prediction flips compared to anchoring on the answer (A-Anchored). This could indicate that the models are more sensitive to variations in the question phrasing than variations in the answer. The larger flip rates for the Llama-3-70B model might suggest a greater capacity for nuanced understanding and sensitivity to input variations, but also potentially a higher susceptibility to being "flipped" by subtle changes. The consistently low flip rates for the HotpotQA dataset could indicate that this dataset is less ambiguous or more straightforward for the models to process, or that the models have learned to perform well on this specific dataset. The difference in flip rates between the two models could also be due to differences in their training data or model architecture. Further investigation would be needed to determine the underlying reasons for these observed patterns.

\n ## Bar Chart: Prediction Flip Rate Comparison for Llama-3 Models ### Overview The image displays two side-by-side grouped bar charts comparing the "Prediction Flip Rate" for two different sizes of the Llama-3 model (8B and 70B parameters) across four question-answering datasets. The charts evaluate the stability of model predictions when using two different anchoring methods: "Q-Anchored (exact_question)" and "A-Anchored (exact_question)". ### Components/Axes * **Chart Titles (Top):** * Left Chart: `Llama-3-8B` * Right Chart: `Llama-3-70B` * **Y-Axis (Vertical):** * Label: `Prediction Flip Rate` * Scale: 0 to 80, with major tick marks at 0, 20, 40, 60, 80. * **X-Axis (Horizontal):** * Label: `Dataset` * Categories (for both charts): `PopQA`, `TriviaQA`, `HotpotQA`, `NQ`. * **Legend (Bottom Center):** * A red/brown bar: `Q-Anchored (exact_question)` * A grey bar: `A-Anchored (exact_question)` * **Data Series:** Each dataset category contains two bars, one for each anchoring method, placed side-by-side. ### Detailed Analysis **Llama-3-8B Chart (Left Panel):** * **PopQA:** * Q-Anchored (red/brown): ~65 * A-Anchored (grey): ~22 * **TriviaQA:** * Q-Anchored (red/brown): ~88 (highest in this panel) * A-Anchored (grey): ~55 * **HotpotQA:** * Q-Anchored (red/brown): ~48 * A-Anchored (grey): ~8 (lowest in this panel) * **NQ:** * Q-Anchored (red/brown): ~74 * A-Anchored (grey): ~20 **Llama-3-70B Chart (Right Panel):** * **PopQA:** * Q-Anchored (red/brown): ~90 (highest in the entire image) * A-Anchored (grey): ~52 * **TriviaQA:** * Q-Anchored (red/brown): ~70 * A-Anchored (grey): ~24 * **HotpotQA:** * Q-Anchored (red/brown): ~61 * A-Anchored (grey): ~14 * **NQ:** * Q-Anchored (red/brown): ~40 * A-Anchored (grey): ~16 ### Key Observations 1. **Consistent Anchoring Effect:** Across all datasets and both model sizes, the **Q-Anchored** method (red/brown bars) consistently results in a significantly higher Prediction Flip Rate than the **A-Anchored** method (grey bars). 2. **Model Size Impact:** The larger Llama-3-70B model shows a more extreme pattern. Its highest flip rate (PopQA, Q-Anchored) is higher than the 8B model's peak, and its lowest flip rate (HotpotQA, A-Anchored) is lower than the 8B model's low. 3. **Dataset Variability:** The flip rate varies substantially by dataset. For the 8B model, TriviaQA shows the highest instability with Q-Anchoring. For the 70B model, PopQA shows the highest instability with Q-Anchoring, while NQ shows the lowest flip rates overall for both anchoring methods. 4. **Trend Verification:** * For the **Q-Anchored** series in the 8B model, the trend is: PopQA (medium) -> TriviaQA (peak) -> HotpotQA (dip) -> NQ (high). * For the **A-Anchored** series in the 8B model, the trend is: PopQA (medium) -> TriviaQA (peak) -> HotpotQA (deep valley) -> NQ (medium-low). * For the **Q-Anchored** series in the 70B model, the trend is: PopQA (peak) -> TriviaQA (medium) -> HotpotQA (medium) -> NQ (low). * For the **A-Anchored** series in the 70B model, the trend is: PopQA (peak) -> TriviaQA (medium) -> HotpotQA (low) -> NQ (low). ### Interpretation The data suggests that the **method used to anchor or frame a question (Q-Anchored vs. A-Anchored) has a profound and consistent impact on the stability of a large language model's predictions**, more so than the model's size or the specific dataset in many cases. * **Q-Anchoring** (likely using the exact question text as a prompt anchor) leads to much higher prediction flip rates, indicating **lower consistency**. This could mean the model's answers are more sensitive to minor variations or perturbations when the question itself is the primary anchor. * **A-Anchoring** (likely using the exact answer text as an anchor) results in dramatically lower flip rates, suggesting **higher robustness and consistency**. This implies that anchoring on the answer space stabilizes the model's output. * The **increase in model scale (8B to 70B) amplifies this effect** rather than mitigating it. The larger model becomes even more stable with A-Anchoring and even more volatile with Q-Anchoring on certain datasets (like PopQA). This challenges the assumption that larger models are inherently more robust; their stability appears highly dependent on the prompting or anchoring strategy. * The **variation across datasets** (PopQA, TriviaQA, HotpotQA, NQ) indicates that the nature of the questions and knowledge domain also interacts with the anchoring method. Datasets requiring multi-hop reasoning (HotpotQA) or containing popular knowledge (PopQA) may elicit different stability profiles. In essence, the charts provide strong empirical evidence that **how you "ground" or anchor a query to an LLM critically determines the reliability of its responses**, and this design choice may be as important as model size for practical applications requiring consistent outputs.

## Bar Chart: Prediction Flip Rate Comparison for Llama-3-8B and Llama-3-70B Models ### Overview The image presents a grouped bar chart comparing prediction flip rates for two language models (Llama-3-8B and Llama-3-70B) across four question-answering datasets (PopQA, TriviaQA, HotpotQA, NQ). Two anchoring methods are compared: Q-Anchored (exact_question) and A-Anchored (exact_question), represented by red and gray bars respectively. ### Components/Axes - **X-Axis**: Datasets (PopQA, TriviaQA, HotpotQA, NQ) - **Y-Axis**: Prediction Flip Rate (%) ranging from 0 to 100 - **Legend**: - Red: Q-Anchored (exact_question) - Gray: A-Anchored (exact_question) - **Model Labels**: - Top-left: Llama-3-8B - Top-right: Llama-3-70B ### Detailed Analysis #### Llama-3-8B (Left Chart) - **PopQA**: - Q-Anchored: ~65% (red) - A-Anchored: ~22% (gray) - **TriviaQA**: - Q-Anchored: ~88% (red) - A-Anchored: ~55% (gray) - **HotpotQA**: - Q-Anchored: ~50% (red) - A-Anchored: ~10% (gray) - **NQ**: - Q-Anchored: ~75% (red) - A-Anchored: ~20% (gray) #### Llama-3-70B (Right Chart) - **PopQA**: - Q-Anchored: ~90% (red) - A-Anchored: ~50% (gray) - **TriviaQA**: - Q-Anchored: ~70% (red) - A-Anchored: ~22% (gray) - **HotpotQA**: - Q-Anchored: ~60% (red) - A-Anchored: ~12% (gray) - **NQ**: - Q-Anchored: ~40% (red) - A-Anchored: ~15% (gray) ### Key Observations 1. **Q-Anchored Consistently Outperforms A-Anchored**: - Across all datasets and models, Q-Anchored (red) bars are significantly taller than A-Anchored (gray) bars. - Example: In Llama-3-8B TriviaQA, Q-Anchored reaches ~88% vs. A-Anchored at ~55%. 2. **Model Size Impact**: - Llama-3-70B generally shows higher absolute flip rates than Llama-3-8B, particularly in PopQA (90% vs. 65% for Q-Anchored). 3. **Dataset-Specific Trends**: - **TriviaQA** has the highest Q-Anchored flip rates for both models. - **NQ** shows the lowest A-Anchored performance in Llama-3-70B (~15%). 4. **Anchoring Method Effect**: - Q-Anchored (exact_question) correlates with higher flip rates, suggesting stronger question-specific performance. - A-Anchored (exact_question) underperforms, with rates often below 30% except in Llama-3-70B PopQA (~50%). ### Interpretation The data demonstrates that **Q-Anchored (exact_question)** anchoring significantly improves prediction flip rates compared to A-Anchored (exact_question) across all datasets and model sizes. This suggests that question-specific anchoring enhances model performance in QA tasks. The Llama-3-70B model achieves higher absolute rates than Llama-3-8B, indicating that larger model size amplifies the benefits of Q-Anchored methods. Notably, the A-Anchored method struggles in NQ for Llama-3-70B, highlighting potential limitations in answer-based anchoring for complex datasets. The consistent trend across models implies that anchoring strategy matters more than model size for flip rate optimization.