## Bar Chart: Prediction Flip Rate Comparison for Llama-3 Models ### Overview The image presents two bar charts comparing the prediction flip rates of two Llama-3 models (8B and 70B) across four different datasets: PopQA, TriviaQA, HotpotQA, and NQ. The charts compare the performance of "Q-Anchored" (exact question) and "A-Anchored" (exact question) approaches. ### Components/Axes * **Titles:** * Left Chart: Llama-3-8B * Right Chart: Llama-3-70B * **Y-Axis:** Prediction Flip Rate, with a scale from 0 to 60 in increments of 20. * **X-Axis:** Dataset, with categories: PopQA, TriviaQA, HotpotQA, NQ. * **Legend:** Located at the bottom of the image. * Q-Anchored (exact\_question): Represented by a light brown color. * A-Anchored (exact\_question): Represented by a gray color. ### Detailed Analysis **Left Chart: Llama-3-8B** * **PopQA:** * Q-Anchored: Approximately 53% * A-Anchored: Approximately 11% * **TriviaQA:** * Q-Anchored: Approximately 68% * A-Anchored: Approximately 40% * **HotpotQA:** * Q-Anchored: Approximately 40% * A-Anchored: Approximately 9% * **NQ:** * Q-Anchored: Approximately 68% * A-Anchored: Approximately 22% **Right Chart: Llama-3-70B** * **PopQA:** * Q-Anchored: Approximately 65% * A-Anchored: Approximately 13% * **TriviaQA:** * Q-Anchored: Approximately 57% * A-Anchored: Approximately 17% * **HotpotQA:** * Q-Anchored: Approximately 56% * A-Anchored: Approximately 16% * **NQ:** * Q-Anchored: Approximately 43% * A-Anchored: Approximately 26% ### Key Observations * For both models, the Q-Anchored approach generally results in a higher prediction flip rate compared to the A-Anchored approach across all datasets. * The TriviaQA dataset shows the highest prediction flip rate for the Llama-3-8B model with the Q-Anchored approach. * The NQ dataset shows the lowest prediction flip rate for the Llama-3-70B model with the Q-Anchored approach. * The A-Anchored approach consistently shows lower prediction flip rates across all datasets for both models. ### Interpretation The data suggests that anchoring the question directly ("Q-Anchored") leads to a higher likelihood of prediction flips compared to anchoring the answer ("A-Anchored"). This could indicate that the models are more sensitive to variations or perturbations in the question itself. The difference in performance between the 8B and 70B models may reflect the impact of model size on robustness and stability of predictions. The specific characteristics of each dataset (PopQA, TriviaQA, HotpotQA, NQ) likely contribute to the observed variations in prediction flip rates. The higher flip rates for Q-Anchored suggest that the model's predictions are more brittle when the question is manipulated, potentially due to the model relying heavily on specific question phrasing.

\n ## Bar Chart: Prediction Flip Rate for Llama Models ### Overview This image presents a comparative bar chart illustrating the Prediction Flip Rate for two Llama models – Llama-3-8B and Llama-3-70B – across four different datasets: PopQA, TriviaQA, HotpotQA, and NQ. The chart compares the flip rates for questions anchored to the original question ("Q-Anchored") versus those anchored to the answer ("A-Anchored"). ### Components/Axes * **X-axis:** Dataset (PopQA, TriviaQA, HotpotQA, NQ) * **Y-axis:** Prediction Flip Rate (ranging from 0 to 60, with increments of 10) * **Models:** Two separate charts are presented side-by-side, one for Llama-3-8B and one for Llama-3-70B. * **Legend:** Located at the bottom-center of the image. * **Q-Anchored (exact_question):** Represented by a reddish-brown color. * **A-Anchored (exact_question):** Represented by a gray color. ### Detailed Analysis **Llama-3-8B Chart (Left)** * **PopQA:** The Q-Anchored bar has a height of approximately 52. The A-Anchored bar has a height of approximately 8. * **TriviaQA:** The Q-Anchored bar has a height of approximately 58. The A-Anchored bar has a height of approximately 42. * **HotpotQA:** The Q-Anchored bar has a height of approximately 42. The A-Anchored bar has a height of approximately 10. * **NQ:** The Q-Anchored bar has a height of approximately 56. The A-Anchored bar has a height of approximately 24. **Llama-3-70B Chart (Right)** * **PopQA:** The Q-Anchored bar has a height of approximately 60. The A-Anchored bar has a height of approximately 6. * **TriviaQA:** The Q-Anchored bar has a height of approximately 54. The A-Anchored bar has a height of approximately 36. * **HotpotQA:** The Q-Anchored bar has a height of approximately 52. The A-Anchored bar has a height of approximately 12. * **NQ:** The Q-Anchored bar has a height of approximately 46. The A-Anchored bar has a height of approximately 26. In both charts, the Q-Anchored bars are consistently higher than the A-Anchored bars across all datasets. The Q-Anchored bars generally exhibit a similar height across the datasets, while the A-Anchored bars show more variation. ### Key Observations * The Prediction Flip Rate is significantly higher for Q-Anchored prompts compared to A-Anchored prompts for both models. * The Llama-3-70B model generally exhibits a higher Prediction Flip Rate for Q-Anchored prompts than the Llama-3-8B model. * The A-Anchored flip rates are relatively low and consistent across datasets for both models. * TriviaQA shows the largest difference between Q-Anchored and A-Anchored flip rates for both models. ### Interpretation The data suggests that anchoring predictions to the original question ("Q-Anchored") leads to a substantially higher rate of prediction flips compared to anchoring them to the answer ("A-Anchored"). This implies that the models are more sensitive to changes in the question phrasing than changes in the answer. The larger difference observed in the TriviaQA dataset might indicate that this dataset presents more challenging or ambiguous questions, making the models more susceptible to flipping predictions based on slight variations in the question. The higher flip rates for the Llama-3-70B model could be attributed to its larger size and increased capacity to capture nuanced relationships within the data. However, it also suggests that the larger model might be more prone to overfitting or sensitivity to specific input patterns. The consistently low A-Anchored flip rates suggest that the models are relatively stable when the context is anchored to the answer, indicating that the answer itself provides a stronger and more reliable basis for prediction. This could be due to the answer being a more definitive and less ambiguous piece of information compared to the question.

## Grouped Bar Chart: Prediction Flip Rate by Dataset and Model ### Overview The image displays two side-by-side grouped bar charts comparing the "Prediction Flip Rate" for two different large language models (Llama-3-8B and Llama-3-70B) across four question-answering datasets. The charts evaluate the stability of model predictions under two different anchoring conditions. ### Components/Axes * **Chart Titles (Top Center):** * Left Chart: `Llama-3-8B` * Right Chart: `Llama-3-70B` * **Y-Axis (Left Side of Each Chart):** * Label: `Prediction Flip Rate` * Scale: 0 to 60, with major tick marks at 0, 20, 40, and 60. * **X-Axis (Bottom of Each Chart):** * Label: `Dataset` * Categories (from left to right): `PopQA`, `TriviaQA`, `HotpotQA`, `NQ` * **Legend (Bottom Center, spanning both charts):** * A red/brown square labeled: `Q-Anchored (exact_question)` * A gray square labeled: `A-Anchored (exact_question)` * **Data Series:** Each dataset category has two bars, one for each anchoring condition, placed side-by-side. ### Detailed Analysis **Llama-3-8B Chart (Left Panel):** * **Trend Verification:** For all four datasets, the Q-Anchored (red/brown) bar is significantly taller than the A-Anchored (gray) bar, indicating a higher prediction flip rate when the question is anchored. * **Data Points (Approximate Values):** * **PopQA:** Q-Anchored ≈ 53, A-Anchored ≈ 10 * **TriviaQA:** Q-Anchored ≈ 68, A-Anchored ≈ 39 * **HotpotQA:** Q-Anchored ≈ 39, A-Anchored ≈ 9 * **NQ:** Q-Anchored ≈ 69, A-Anchored ≈ 22 **Llama-3-70B Chart (Right Panel):** * **Trend Verification:** The same pattern holds: Q-Anchored bars are consistently taller than A-Anchored bars across all datasets. The overall height of the bars appears slightly lower compared to the 8B model for most categories. * **Data Points (Approximate Values):** * **PopQA:** Q-Anchored ≈ 66, A-Anchored ≈ 14 * **TriviaQA:** Q-Anchored ≈ 57, A-Anchored ≈ 18 * **HotpotQA:** Q-Anchored ≈ 54, A-Anchored ≈ 17 * **NQ:** Q-Anchored ≈ 42, A-Anchored ≈ 26 ### Key Observations 1. **Consistent Anchoring Effect:** Across both model sizes and all four datasets, anchoring the prompt with the exact question (`Q-Anchored`) leads to a substantially higher prediction flip rate than anchoring with the exact answer (`A-Anchored`). 2. **Dataset Variability:** The magnitude of the flip rate varies by dataset. For example, `TriviaQA` and `NQ` show very high Q-Anchored flip rates for the 8B model, while `HotpotQA` shows the lowest for both anchoring types in that model. 3. **Model Size Comparison:** The larger Llama-3-70B model generally exhibits lower flip rates than the 8B model, particularly for the Q-Anchored condition on datasets like `TriviaQA` and `NQ`. However, for `PopQA`, the 70B model's Q-Anchored flip rate is higher. 4. **Relative Stability:** The A-Anchored condition results in flip rates mostly below 30, suggesting predictions are more stable when anchored to an answer format. ### Interpretation This data suggests that the **format of the prompt significantly influences the stability of a large language model's predictions**. When a model is prompted with a question (`Q-Anchored`), its output is more volatile and prone to "flipping" (changing) compared to when it is prompted with a fixed answer format (`A-Anchored`). The **Peircean investigative reading** implies that the "exact_question" anchor introduces more interpretative ambiguity or contextual variability for the model, leading to less deterministic outputs. In contrast, the "exact_question" anchor (likely meaning the prompt is structured around providing an answer) constrains the model's response space, leading to greater consistency. The **anomaly** is that while the 70B model is generally more stable, it is not uniformly so; its higher flip rate on `PopQA` under Q-Anchoring indicates that dataset-specific characteristics can interact with model scale in non-linear ways. This highlights that prompt engineering for stability must be tailored to both the model and the specific knowledge domain (dataset).

## Bar Chart: Prediction Flip Rate Comparison for Llama-3-8B and Llama-3-70B Models ### Overview The image is a grouped bar chart comparing prediction flip rates (in percentage) for two language models, **Llama-3-8B** and **Llama-3-70B**, across four datasets: **PopQA**, **TriviaQA**, **HotpotQA**, and **NQ**. Two anchoring methods are compared: **Q-Anchored (exact_question)** (red bars) and **A-Anchored (exact_question)** (gray bars). ### Components/Axes - **X-axis**: Datasets (PopQA, TriviaQA, HotpotQA, NQ). - **Y-axis**: Prediction Flip Rate (%) ranging from 0 to 70% in 20% increments. - **Legend**: - Red: Q-Anchored (exact_question) - Gray: A-Anchored (exact_question) - **Models**: - Llama-3-8B (left chart) - Llama-3-70B (right chart) ### Detailed Analysis #### Llama-3-8B (Left Chart) - **PopQA**: - Q-Anchored: ~55% - A-Anchored: ~10% - **TriviaQA**: - Q-Anchored: ~65% - A-Anchored: ~40% - **HotpotQA**: - Q-Anchored: ~40% - A-Anchored: ~10% - **NQ**: - Q-Anchored: ~65% - A-Anchored: ~20% #### Llama-3-70B (Right Chart) - **PopQA**: - Q-Anchored: ~65% - A-Anchored: ~15% - **TriviaQA**: - Q-Anchored: ~55% - A-Anchored: ~20% - **HotpotQA**: - Q-Anchored: ~50% - A-Anchored: ~15% - **NQ**: - Q-Anchored: ~45% - A-Anchored: ~25% ### Key Observations 1. **Q-Anchored Consistently Outperforms A-Anchored**: - Across all datasets and models, Q-Anchored flip rates are significantly higher than A-Anchored rates. - Example: Llama-3-8B on NQ shows a 65% (Q) vs. 20% (A) gap. 2. **Model Size Impact**: - Llama-3-70B generally has lower flip rates than Llama-3-8B, particularly in **NQ** (45% vs. 65% for Q-Anchored). 3. **Dataset Variability**: - **NQ** has the highest Q-Anchored rates for both models. - **HotpotQA** shows the largest drop between Q and A anchoring for Llama-3-8B (~30% difference). ### Interpretation - **Anchoring Method Effectiveness**: Q-Anchored (exact_question) demonstrates superior performance, suggesting that precise question alignment improves prediction stability. - **Model Scaling Trade-offs**: While Llama-3-70B reduces flip rates compared to Llama-3-8B, the gap between anchoring methods narrows, implying diminishing returns in larger models for Q-Anchored benefits. - **Dataset-Specific Behavior**: The **NQ** dataset’s high Q-Anchored rates may reflect its question complexity or structure, which aligns better with exact anchoring. ### Spatial Grounding & Trend Verification - **Legend Placement**: Bottom-left, clearly labeled with color-coded anchors. - **Bar Trends**: - Q-Anchored bars slope upward relative to A-Anchored across all datasets. - Llama-3-70B’s bars are shorter than Llama-3-8B’s, confirming lower flip rates. - **Color Consistency**: Red (Q) and gray (A) bars match legend labels without ambiguity. ### Content Details - **Approximate Values**: - Llama-3-8B: - PopQA: Q=55%, A=10% - TriviaQA: Q=65%, A=40% - HotpotQA: Q=40%, A=10% - NQ: Q=65%, A=20% - Llama-3-70B: - PopQA: Q=65%, A=15% - TriviaQA: Q=55%, A=20% - HotpotQA: Q=50%, A=15% - NQ: Q=45%, A=25% ### Notable Outliers - **Llama-3-8B on TriviaQA**: A-Anchored rate (~40%) is unusually high compared to other datasets, suggesting dataset-specific model behavior. - **Llama-3-70B on NQ**: Q-Anchored rate (~45%) is notably lower than Llama-3-8B’s (~65%), highlighting model size’s impact on performance. ### Final Notes The chart underscores the importance of anchoring methods in model reliability, with Q-Anchored outperforming A-Anchored across all scenarios. Model scaling improves performance but does not eliminate the anchoring gap, indicating architectural or training differences between the two models.