## Bar Chart: Prediction Flip Rate Comparison for Mistral-7B Models ### Overview The image presents two bar charts comparing the prediction flip rates of Mistral-7B-v0.1 and Mistral-7B-v0.3 models across four datasets: PopQA, TriviaQA, HotpotQA, and NQ. The charts show the prediction flip rates for both Q-Anchored (exact_question) and A-Anchored (exact_question) scenarios. ### Components/Axes * **Title (Top-Left Chart):** Mistral-7B-v0.1 * **Title (Top-Right Chart):** Mistral-7B-v0.3 * **Y-Axis Label:** Prediction Flip Rate * **Y-Axis Scale:** 0 to 80, with tick marks at 0, 20, 40, 60 * **X-Axis Label:** Dataset * **X-Axis Categories:** PopQA, TriviaQA, HotpotQA, NQ * **Legend (Bottom):** * Q-Anchored (exact\_question) - Brown * A-Anchored (exact\_question) - Gray ### Detailed Analysis **Left Chart: Mistral-7B-v0.1** * **PopQA:** * Q-Anchored (Brown): Approximately 73% * A-Anchored (Gray): Approximately 26% * **TriviaQA:** * Q-Anchored (Brown): Approximately 62% * A-Anchored (Gray): Approximately 50% * **HotpotQA:** * Q-Anchored (Brown): Approximately 41% * A-Anchored (Gray): Approximately 10% * **NQ:** * Q-Anchored (Brown): Approximately 68% * A-Anchored (Gray): Approximately 23% **Right Chart: Mistral-7B-v0.3** * **PopQA:** * Q-Anchored (Brown): Approximately 75% * A-Anchored (Gray): Approximately 12% * **TriviaQA:** * Q-Anchored (Brown): Approximately 75% * A-Anchored (Gray): Approximately 38% * **HotpotQA:** * Q-Anchored (Brown): Approximately 52% * A-Anchored (Gray): Approximately 12% * **NQ:** * Q-Anchored (Brown): Approximately 60% * A-Anchored (Gray): Approximately 43% ### Key Observations * In both charts, the Q-Anchored prediction flip rates are consistently higher than the A-Anchored rates across all datasets. * The Mistral-7B-v0.3 model generally shows a higher Q-Anchored prediction flip rate for TriviaQA and HotpotQA compared to Mistral-7B-v0.1. * The A-Anchored prediction flip rates vary across datasets and models, with TriviaQA showing the highest rate for Mistral-7B-v0.1 and NQ showing the highest rate for Mistral-7B-v0.3. ### Interpretation The data suggests that the prediction flip rate is significantly influenced by whether the question or the answer is anchored. The higher flip rates for Q-Anchored scenarios indicate that the model's predictions are more sensitive to changes in the question. The differences between Mistral-7B-v0.1 and Mistral-7B-v0.3 highlight the impact of model version on prediction stability. The variations across datasets suggest that the complexity and nature of the questions in each dataset also play a role in the prediction flip rate. The lower A-Anchored rates suggest that the model is more robust to changes in the answer, possibly because the answer provides a stronger contextual anchor.

\n ## Bar Chart: Prediction Flip Rate for Mistral Models ### Overview This image presents a comparative bar chart illustrating the prediction flip rate for two versions of the Mistral-7B language model (v0.1 and v0.3) across four different datasets: PopQA, TriviaQA, HotpotQA, and NQ. The flip rate is measured as a percentage and is shown for both "Q-Anchored" (exact question) and "A-Anchored" (exact question) scenarios. ### Components/Axes * **X-axis:** Dataset - PopQA, TriviaQA, HotpotQA, NQ. * **Y-axis:** Prediction Flip Rate - Scale ranges from 0 to 60 (approximately). * **Models:** Mistral-7B-v0.1 (left chart), Mistral-7B-v0.3 (right chart). * **Legend:** * Red: Q-Anchored (exact\_question) * Gray: A-Anchored (exact\_question) ### Detailed Analysis **Mistral-7B-v0.1 (Left Chart)** * **PopQA:** Q-Anchored flip rate is approximately 62%. A-Anchored flip rate is approximately 24%. * **TriviaQA:** Q-Anchored flip rate is approximately 68%. A-Anchored flip rate is approximately 46%. * **HotpotQA:** Q-Anchored flip rate is approximately 44%. A-Anchored flip rate is approximately 8%. * **NQ:** Q-Anchored flip rate is approximately 66%. A-Anchored flip rate is approximately 42%. **Mistral-7B-v0.3 (Right Chart)** * **PopQA:** Q-Anchored flip rate is approximately 64%. A-Anchored flip rate is approximately 12%. * **TriviaQA:** Q-Anchored flip rate is approximately 70%. A-Anchored flip rate is approximately 48%. * **HotpotQA:** Q-Anchored flip rate is approximately 48%. A-Anchored flip rate is approximately 10%. * **NQ:** Q-Anchored flip rate is approximately 68%. A-Anchored flip rate is approximately 44%. **Trends:** * For both models, the Q-Anchored flip rate is consistently higher than the A-Anchored flip rate across all datasets. * The Q-Anchored flip rate is generally high (above 60%) for all datasets in both models. * The A-Anchored flip rate varies more significantly across datasets. ### Key Observations * The largest difference between Q-Anchored and A-Anchored flip rates is observed in the PopQA dataset for both models. * The HotpotQA dataset consistently shows the lowest A-Anchored flip rate for both models. * The Mistral-7B-v0.3 model generally exhibits a slightly lower A-Anchored flip rate compared to the v0.1 model, except for the NQ dataset. ### Interpretation The data suggests that the Mistral models are more sensitive to changes in the question phrasing (Q-Anchored) than changes in the answer phrasing (A-Anchored). This indicates that the models rely more heavily on the question context when making predictions. The higher flip rates for Q-Anchored scenarios suggest that even small alterations to the question can lead to different predictions. The variation in flip rates across datasets likely reflects the inherent difficulty and characteristics of each dataset. For example, the low A-Anchored flip rate for HotpotQA might indicate that the answers in this dataset are more robust to slight variations in phrasing. The slight improvement in the v0.3 model (lower A-Anchored flip rates in most cases) suggests that the model updates have made it slightly less sensitive to changes in the answer phrasing, potentially indicating improved robustness. However, the differences are relatively small, and further analysis would be needed to confirm this trend. The consistent high Q-Anchored flip rate across both versions suggests that the core sensitivity to question phrasing remains.

## Grouped Bar Chart: Prediction Flip Rate Comparison for Mistral-7B Models ### Overview The image displays two side-by-side grouped bar charts comparing the "Prediction Flip Rate" of two versions of the Mistral-7B language model (v0.1 and v0.3) across four question-answering datasets. The charts evaluate model sensitivity to question phrasing by comparing two anchoring methods. ### Components/Axes * **Chart Titles:** "Mistral-7B-v0.1" (left panel), "Mistral-7B-v0.3" (right panel). * **Y-Axis (Both Panels):** Labeled "Prediction Flip Rate". The scale runs from 0 to 60 (v0.1) and 0 to 70 (v0.3), with major gridlines at intervals of 20. * **X-Axis (Both Panels):** Labeled "Dataset". The four categories are: "PopQA", "TriviaQA", "HotpotQA", and "NQ". * **Legend:** Positioned at the bottom center of the entire figure. * **Reddish-Brown Bar:** "Q-Anchored (exact_question)" * **Gray Bar:** "A-Anchored (exact_question)" ### Detailed Analysis **Mistral-7B-v0.1 (Left Panel):** * **PopQA:** Q-Anchored flip rate is the highest, approximately 75. A-Anchored is significantly lower, around 25. * **TriviaQA:** Q-Anchored is approximately 65. A-Anchored is relatively high, around 50. * **HotpotQA:** Q-Anchored is the lowest for this model version, approximately 40. A-Anchored is very low, around 10. * **NQ:** Q-Anchored is high, approximately 70. A-Anchored is low, around 20. **Mistral-7B-v0.3 (Right Panel):** * **PopQA:** Q-Anchored remains very high, approximately 75. A-Anchored has decreased notably to around 12. * **TriviaQA:** Q-Anchored has increased to approximately 75. A-Anchored has decreased to around 38. * **HotpotQA:** Q-Anchored has increased to approximately 52. A-Anchored remains very low, around 10. * **NQ:** Q-Anchored has decreased to approximately 60. A-Anchored has increased significantly to around 45. **Trend Verification:** * In both model versions, the **Q-Anchored (reddish-brown) bars are consistently taller** than the corresponding A-Anchored (gray) bars for every dataset. * From v0.1 to v0.3, the Q-Anchored flip rate **increased** for TriviaQA and HotpotQA, **decreased** for NQ, and remained **stable** for PopQA. * From v0.1 to v0.3, the A-Anchored flip rate **decreased** for PopQA and TriviaQA, **increased** for NQ, and remained **stable** for HotpotQA. ### Key Observations 1. **Dominant Pattern:** The Q-Anchored method consistently results in a higher prediction flip rate than the A-Anchored method across all datasets and both model versions. 2. **Largest Discrepancy:** The greatest difference between the two anchoring methods is observed in the **PopQA** dataset for both model versions. 3. **Notable Change (v0.1 to v0.3):** The **NQ** dataset shows a significant shift. The Q-Anchored flip rate decreased, while the A-Anchored flip rate more than doubled, making the gap between the two methods much smaller in v0.3. 4. **Stable Low Point:** The **HotpotQA** dataset's A-Anchored flip rate remains consistently low (~10) across both model versions. ### Interpretation This data suggests that the model's predictions are generally more sensitive to variations in the question phrasing (Q-Anchored) than to variations in the answer phrasing (A-Anchored). A higher "Prediction Flip Rate" indicates lower robustness; the model changes its answer more frequently when the input is perturbed. The comparison between v0.1 and v0.3 reveals that model updates have a dataset-specific impact on robustness. For example, the model became **more robust** to question perturbations on NQ (Q-Anchored flip rate decreased) but **less robust** to answer perturbations on the same dataset (A-Anchored flip rate increased). The significant reduction in A-Anchored flip rate for PopQA and TriviaQA in v0.3 suggests an improvement in stability when the answer is fixed for those datasets. The consistently low A-Anchored flip rate for HotpotQA might indicate that this dataset's answers are particularly deterministic or that the model's knowledge about these topics is very stable. Overall, the charts demonstrate that evaluating model robustness requires looking at multiple datasets and perturbation types, as improvements are not uniform.

## Bar Chart: Prediction Flip Rate Comparison for Mistral-7B Models (v0.1 and v0.3) ### Overview The image compares prediction flip rates for two versions of the Mistral-7B language model (v0.1 and v0.3) across four datasets (PopQA, TriviaQA, HotpotQA, NQ). Two anchoring strategies are evaluated: **Q-Anchored (exact_question)** and **A-Anchored (exact_question)**, represented by red and gray bars respectively. The y-axis measures prediction flip rate as a percentage. ### Components/Axes - **X-Axis (Datasets)**: PopQA, TriviaQA, HotpotQA, NQ (left to right). - **Y-Axis (Prediction Flip Rate)**: 0% to 80% in 20% increments. - **Legend**: - Red = Q-Anchored (exact_question) - Gray = A-Anchored (exact_question) - **Model Versions**: - Left subplot = Mistral-7B-v0.1 - Right subplot = Mistral-7B-v0.3 ### Detailed Analysis #### Mistral-7B-v0.1 - **PopQA**: - Q-Anchored: ~70% (red) - A-Anchored: ~25% (gray) - **TriviaQA**: - Q-Anchored: ~60% (red) - A-Anchored: ~50% (gray) - **HotpotQA**: - Q-Anchored: ~40% (red) - A-Anchored: ~10% (gray) - **NQ**: - Q-Anchored: ~70% (red) - A-Anchored: ~20% (gray) #### Mistral-7B-v0.3 - **PopQA**: - Q-Anchored: ~70% (red) - A-Anchored: ~15% (gray) - **TriviaQA**: - Q-Anchored: ~70% (red) - A-Anchored: ~40% (gray) - **HotpotQA**: - Q-Anchored: ~50% (red) - A-Anchored: ~10% (gray) - **NQ**: - Q-Anchored: ~60% (red) - A-Anchored: ~45% (gray) ### Key Observations 1. **Q-Anchored Dominance**: Across all datasets and models, Q-Anchored consistently outperforms A-Anchored, with flip rates 2–4× higher in most cases. 2. **Version-Specific Trends**: - **v0.1**: Largest gap between anchoring strategies in HotpotQA (40% vs. 10%). - **v0.3**: Narrowed gap in TriviaQA (70% vs. 40%) and NQ (60% vs. 45%), suggesting improved A-Anchored performance. 3. **Dataset Variability**: - PopQA and TriviaQA show the highest flip rates for Q-Anchored in both versions. - NQ exhibits the most significant A-Anchored improvement in v0.3 (+25% vs. v0.1). ### Interpretation The data suggests that **Q-Anchored (exact_question)** anchoring improves model confidence, as evidenced by higher prediction flip rates. However, **Mistral-7B-v0.3** shows notable progress in A-Anchored performance, particularly for natural questions (NQ), where the gap between anchoring strategies reduced by ~20%. This may indicate architectural or training improvements in v0.3 that better align with real-world question structures. The persistent dominance of Q-Anchored highlights the importance of question specificity in model reliability, while the narrowing gaps in v0.3 suggest potential for more robust generalization in future iterations.