## Reliability Diagrams of Qwen-2.5-7B with Different Context RAG Settings ### Overview The image presents a series of reliability diagrams for the Qwen-2.5-7B model under different context RAG (Retrieval-Augmented Generation) settings. The diagrams are arranged in a 3x4 grid, with each row representing a different RAG setting: no context, 0p10n context, and 1p9n context. Within each row, the diagrams are further categorized by knowledge type: highly known, maybe known, weakly known, and unknown knowledge. Each diagram plots accuracy against confidence, showing the model's calibration for different types of knowledge under varying context conditions. ### Components/Axes Each individual chart has the following components: * **Title:** Indicates the knowledge type (highly known, maybe known, weakly known, unknown knowledge). * **X-axis:** Confidence, ranging from 0.0 to 1.0 in increments of 0.2. * **Y-axis:** Accuracy, ranging from 0.0 to 1.0 in increments of 0.2. * **Data Series:** * **Accuracy:** Represented by blue bars. The height of each bar indicates the accuracy for a given confidence level. * **Gap:** Represented by light red bars. The height of each bar indicates the gap between perfect calibration and the actual accuracy. * **Perfect Calibration:** Represented by a dashed gray diagonal line. This line indicates perfect calibration, where confidence equals accuracy. * **Legend:** Located in the top-left corner of each chart, indicating the color and label for each data series: * Perfect calibration (dashed gray line) * Accuracy (blue bars) * Gap (light red bars) ### Detailed Analysis **Row 1: (a) Reliability Diagram of Qwen-2.5-7B with no context RAG setting** * **Highly Known Knowledge:** * Accuracy: The accuracy is low at confidence 0.2 (approximately 0.1), then increases significantly at confidence 0.6 (approximately 0.9), and remains high at confidence 1.0 (approximately 0.95). * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases at confidence 0.6 (approximately 0.1), and remains small at confidence 1.0 (approximately 0.05). * **Maybe Known Knowledge:** * Accuracy: The accuracy starts low at confidence 0.2 (approximately 0.1), increases at confidence 0.6 (approximately 0.6), and reaches approximately 0.75 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases at confidence 0.6 (approximately 0.4), and remains moderate at confidence 1.0 (approximately 0.25). * **Weakly Known Knowledge:** * Accuracy: The accuracy is very low at confidence 0.2 (approximately 0.0), increases slightly at confidence 0.6 (approximately 0.2), and reaches approximately 0.35 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases slightly at confidence 0.6 (approximately 0.4), and remains large at confidence 1.0 (approximately 0.65). * **Unknown Knowledge:** * Accuracy: The accuracy is very low at confidence 0.2 (approximately 0.0), increases slightly at confidence 0.6 (approximately 0.1), and reaches approximately 0.2 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases slightly at confidence 0.6 (approximately 0.5), and remains large at confidence 1.0 (approximately 0.8). **Row 2: (b) Reliability Diagram of Qwen-2.5-7B with 0p10n context RAG setting** * **Highly Known Knowledge:** * Accuracy: The accuracy is low at confidence 0.2 (approximately 0.1), then increases significantly at confidence 0.6 (approximately 0.7), and remains high at confidence 1.0 (approximately 0.7). * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases at confidence 0.6 (approximately 0.3), and remains moderate at confidence 1.0 (approximately 0.3). * **Maybe Known Knowledge:** * Accuracy: The accuracy starts low at confidence 0.2 (approximately 0.0), increases at confidence 0.6 (approximately 0.4), and reaches approximately 0.5 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases at confidence 0.6 (approximately 0.6), and remains moderate at confidence 1.0 (approximately 0.5). * **Weakly Known Knowledge:** * Accuracy: The accuracy is very low at confidence 0.2 (approximately 0.0), increases slightly at confidence 0.6 (approximately 0.1), and reaches approximately 0.2 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases slightly at confidence 0.6 (approximately 0.5), and remains large at confidence 1.0 (approximately 0.8). * **Unknown Knowledge:** * Accuracy: The accuracy is very low at confidence 0.2 (approximately 0.0), increases slightly at confidence 0.6 (approximately 0.0), and reaches approximately 0.5 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases slightly at confidence 0.6 (approximately 0.5), and remains large at confidence 1.0 (approximately 0.5). **Row 3: (c) Reliability Diagram of Qwen-2.5-7B with 1p9n context RAG setting** * **Highly Known Knowledge:** * Accuracy: The accuracy is low at confidence 0.2 (approximately 0.0), then increases significantly at confidence 0.6 (approximately 0.8), and remains high at confidence 1.0 (approximately 0.9). * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases at confidence 0.6 (approximately 0.2), and remains small at confidence 1.0 (approximately 0.1). * **Maybe Known Knowledge:** * Accuracy: The accuracy starts low at confidence 0.2 (approximately 0.0), increases at confidence 0.6 (approximately 0.6), and reaches approximately 0.95 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases at confidence 0.6 (approximately 0.4), and remains moderate at confidence 1.0 (approximately 0.05). * **Weakly Known Knowledge:** * Accuracy: The accuracy is very low at confidence 0.2 (approximately 0.0), increases slightly at confidence 0.6 (approximately 0.5), and reaches approximately 0.9 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases slightly at confidence 0.6 (approximately 0.5), and remains large at confidence 1.0 (approximately 0.1). * **Unknown Knowledge:** * Accuracy: The accuracy is very low at confidence 0.2 (approximately 0.0), increases slightly at confidence 0.6 (approximately 0.7), and reaches approximately 0.8 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases slightly at confidence 0.6 (approximately 0.3), and remains large at confidence 1.0 (approximately 0.2). ### Key Observations * **Impact of Context RAG:** The reliability diagrams show that the context RAG settings (0p10n and 1p9n) generally improve the accuracy of the Qwen-2.5-7B model, especially for weakly known and unknown knowledge. * **Knowledge Type:** The model performs best on highly known knowledge, followed by maybe known, weakly known, and unknown knowledge. This trend is consistent across all context RAG settings. * **Calibration:** The model tends to be overconfident, especially for weakly known and unknown knowledge, as indicated by the large gap between confidence and accuracy. * **Confidence Levels:** Accuracy generally increases with confidence, but the rate of increase varies depending on the knowledge type and context RAG setting. ### Interpretation The reliability diagrams provide insights into the calibration and accuracy of the Qwen-2.5-7B model under different context RAG settings. The data suggests that incorporating context through RAG can significantly improve the model's performance, particularly for less familiar knowledge domains. However, the model still exhibits overconfidence, especially for weakly known and unknown knowledge, indicating a need for further calibration techniques. The choice of context RAG setting (0p10n vs. 1p9n) also appears to influence the model's reliability, suggesting that the specific context provided plays a crucial role in the model's performance. Overall, these diagrams highlight the importance of evaluating and improving the calibration of language models to ensure reliable and trustworthy predictions.

\n ## Chart: Reliability Diagrams for Qwen-2.5-7B Models ### Overview This image presents a series of reliability diagrams comparing the performance of the Qwen-2.5-7B language model under different conditions. There are three rows, each representing a different setting: (a) no context RAG setting, (b) option context RAG setting, and (c) 10 context RAG setting. Each row contains four sub-charts, one for each knowledge type: highlyknown knowledge, maybeknown knowledge, weaklyknown knowledge, and unknown knowledge. The diagrams plot accuracy against confidence, with shaded regions indicating the gap between perfect calibration and actual accuracy. ### Components/Axes * **X-axis:** Confidence (ranging from 0.0 to 1.0) * **Y-axis:** Accuracy (ranging from 0.0 to 1.0) * **Legend:** * Perfect calibration (dark blue) * Accuracy (light blue) * Gap (red) * **Sub-chart Titles:** Each sub-chart is labeled with the knowledge type and the RAG setting. * **Overall Title:** Each row has a title indicating the RAG setting used. ### Detailed Analysis or Content Details **Row (a): No context RAG setting** * **Highlyknown knowledge:** The accuracy line (light blue) is consistently below the perfect calibration line (dark blue), with a significant gap (red) across all confidence levels. The accuracy is approximately 0.6-0.8, while perfect calibration is at 1.0. * **Maybeknown knowledge:** Similar to highlyknown knowledge, the accuracy line is below the perfect calibration line, but the gap is smaller. Accuracy ranges from approximately 0.5 to 0.8. * **Weaklyknown knowledge:** The accuracy line is significantly below the perfect calibration line, with a large gap. Accuracy is approximately 0.3-0.6. * **Unknown knowledge:** The accuracy line is very low, close to 0.2, and far below the perfect calibration line, resulting in a substantial gap. **Row (b): Option context RAG setting** * **Highlyknown knowledge:** The accuracy line is closer to the perfect calibration line than in (a), but still slightly below. Accuracy ranges from approximately 0.7 to 0.9. * **Maybeknown knowledge:** The accuracy line is closer to the perfect calibration line than in (a), with a smaller gap. Accuracy ranges from approximately 0.6 to 0.8. * **Weaklyknown knowledge:** The accuracy line is closer to the perfect calibration line than in (a), but still below. Accuracy ranges from approximately 0.4 to 0.7. * **Unknown knowledge:** The accuracy line is higher than in (a), but still below the perfect calibration line. Accuracy is approximately 0.3-0.5. **Row (c): 10 context RAG setting** * **Highlyknown knowledge:** The accuracy line is very close to the perfect calibration line, with a minimal gap. Accuracy ranges from approximately 0.8 to 1.0. * **Maybeknown knowledge:** The accuracy line is close to the perfect calibration line, with a small gap. Accuracy ranges from approximately 0.7 to 0.9. * **Weaklyknown knowledge:** The accuracy line is closer to the perfect calibration line than in (a) and (b), but still below. Accuracy ranges from approximately 0.5 to 0.8. * **Unknown knowledge:** The accuracy line is significantly higher than in (a) and (b), approaching the perfect calibration line. Accuracy is approximately 0.4-0.7. ### Key Observations * The model's performance improves significantly with the addition of context (moving from no context to 10 context RAG setting). * The gap between accuracy and perfect calibration is largest for unknown knowledge, indicating the model is least confident and accurate in this scenario. * Highlyknown knowledge consistently exhibits the highest accuracy across all settings. * The accuracy generally increases with confidence, but the gap indicates the model is often overconfident in its predictions. ### Interpretation These reliability diagrams demonstrate the impact of Retrieval-Augmented Generation (RAG) on the calibration of the Qwen-2.5-7B model. The diagrams suggest that providing context (through RAG) improves the model's ability to accurately estimate its own confidence. The model is best calibrated for highlyknown knowledge, meaning it is more likely to accurately reflect its certainty in its predictions for this type of information. Conversely, the model struggles with unknown knowledge, exhibiting a large gap between its confidence and actual accuracy. This suggests that the model is often overconfident when dealing with information it doesn't fully understand. The trend of decreasing gap with increasing context suggests that RAG is a valuable technique for improving the reliability and trustworthiness of language model predictions. The diagrams provide a visual representation of how well the model's predicted probabilities align with its actual performance, which is crucial for applications where accurate uncertainty estimation is important.

## Reliability Diagrams: Qwen-2.5-7B Calibration Across Knowledge Categories and RAG Settings ### Overview The image displays a 3x4 grid of reliability diagrams (calibration plots) for the Qwen-2.5-7B language model. The diagrams evaluate the model's calibration—how well its confidence scores match its actual accuracy—across four knowledge categories and under three different Retrieval-Augmented Generation (RAG) context settings. Each row corresponds to a specific RAG setting, and each column corresponds to a specific knowledge category. ### Components/Axes * **Overall Structure:** Three rows labeled (a), (b), and (c). Each row contains four subplots. * **Subplot Titles (Columns):** From left to right: "highlyknown knowledge", "maybeknown knowledge", "weaklyknown knowledge", "unknown knowledge". * **Axes (Identical for all subplots):** * **X-axis:** "Confidence", scaled from 0.0 to 1.0 with increments of 0.2. * **Y-axis:** "Accuracy", scaled from 0.0 to 1.0 with increments of 0.2. * **Legend (Top-left of each subplot):** * **Perfect calibration:** A gray diagonal line from (0,0) to (1,1). Represents ideal calibration where confidence equals accuracy. * **Accuracy:** Blue bars. The height of each bar represents the model's actual accuracy for predictions within a specific confidence bin. * **Gap:** Pink bars stacked on top of the blue bars. The total height (blue + pink) represents the model's confidence for that bin. The pink portion visualizes the miscalibration gap (Confidence - Accuracy). * **Row Captions (Below each row):** * (a) Reliability Diagram of Qwen-2.5-7B with no context RAG setting * (b) Reliability Diagram of Qwen-2.5-7B with 0p10n context RAG setting * (c) Reliability Diagram of Qwen-2.5-7B with 1p9n context RAG setting ### Detailed Analysis #### Row (a): No Context RAG Setting * **highlyknown knowledge:** Blue bars (Accuracy) are generally close to or slightly below the perfect calibration line. The pink gaps are small to moderate, indicating reasonable but slightly overconfident calibration. * **maybeknown knowledge:** Blue bars are consistently and significantly below the perfect calibration line, especially at mid-to-high confidence (0.4-0.8). Large pink gaps indicate substantial overconfidence (confidence > accuracy). * **weaklyknown knowledge:** Similar to "maybeknown," with blue bars far below the diagonal. Very large pink gaps across most confidence bins show severe overconfidence. * **unknown knowledge:** Blue bars are extremely low, often near zero, across all confidence levels. The pink gaps are massive, filling almost the entire plot area. This indicates the model is highly overconfident when it lacks knowledge, assigning high confidence to almost entirely incorrect answers. #### Row (b): 0p10n Context RAG Setting * **highlyknown knowledge:** Calibration improves compared to (a). Blue bars are closer to the diagonal, with smaller pink gaps, especially at higher confidence levels. * **maybeknown knowledge:** Improvement is visible. Blue bars are higher than in (a), reducing the pink gap, but a notable overconfidence gap remains, particularly in the 0.6-0.8 confidence range. * **weaklyknown knowledge:** Shows some improvement over (a). Blue bars are higher, but large pink gaps persist, indicating continued significant overconfidence. * **unknown knowledge:** Still shows severe miscalibration. While blue bars are slightly higher than in (a) at very high confidence (0.8-1.0), the pink gaps remain enormous, demonstrating the model is still highly overconfident about its unknowns. #### Row (c): 1p9n Context RAG Setting * **highlyknown knowledge:** Calibration appears very good. Blue bars closely follow the perfect calibration line with minimal pink gaps. * **maybeknown knowledge:** Calibration is significantly improved. Blue bars are much closer to the diagonal, with small pink gaps, indicating well-calibrated confidence. * **weaklyknown knowledge:** Shows the most dramatic improvement. Blue bars are now much higher and closer to the diagonal, though a moderate pink gap remains at the highest confidence bin (0.8-1.0). * **unknown knowledge:** Improvement is evident but miscalibration persists. Blue bars are higher than in (a) and (b), especially at high confidence, but substantial pink gaps remain, showing the model is still overconfident, though less severely than with no context. ### Key Observations 1. **Knowledge Hierarchy:** Calibration consistently improves from "unknown" to "highlyknown" knowledge across all settings. The model is best calibrated on what it knows well and worst calibrated on what it doesn't know. 2. **RAG Impact:** The context setting has a profound effect on calibration. * **No Context (a):** Worst calibration, especially severe overconfidence on "unknown" and "weaklyknown" knowledge. * **0p10n Context (b):** Provides moderate improvement, reducing but not eliminating overconfidence. * **1p9n Context (c):** Provides the best calibration, bringing "highlyknown" and "maybeknown" close to perfect calibration and significantly improving "weaklyknown." 3. **Persistent Overconfidence on Unknowns:** Even with the best context (1p9n), the model remains notably overconfident in the "unknown knowledge" category, as shown by the persistent pink gaps in the bottom-right subplot. 4. **Visual Trend:** The visual trend across rows (a) -> (b) -> (c) is a reduction in the size of the pink "Gap" bars and a rising of the blue "Accuracy" bars toward the diagonal "Perfect calibration" line, indicating improving calibration with better context. ### Interpretation These reliability diagrams provide a diagnostic view of the Qwen-2.5-7B model's self-awareness. The data suggests: 1. **Intrinsic Overconfidence:** Without external context (Row a), the model exhibits a strong intrinsic bias toward overconfidence, particularly when operating at the edges of its knowledge ("weaklyknown", "unknown"). This is a common failure mode in LLMs. 2. **Context as a Calibration Tool:** Providing context through RAG acts as a powerful calibration mechanism. The "1p9n" setting (likely meaning 1 positive and 9 negative/irrelevant context passages) appears most effective. It likely helps the model better distinguish between what it knows and what it doesn't, grounding its confidence estimates in external evidence. 3. **Limits of Context:** The persistent gap in the "unknown knowledge" category, even with optimal context, indicates a fundamental challenge. The model may struggle to fully recognize or admit complete ignorance, even when provided with context that doesn't contain the answer. This has implications for reliability in high-stakes applications. 4. **Practical Implication:** For users of this model, the findings argue strongly for using a RAG system, specifically one configured like the "1p9n" setting, to obtain more trustworthy confidence scores. One should be especially skeptical of the model's high-confidence outputs when it is operating on poorly known or unknown topics, as the calibration gap remains significant in those areas. **Language Declaration:** All text within the image is in English. No other languages are present.

## Reliability Diagrams: Qwen-2.5-7B Model Performance Across Context Settings ### Overview The image presents three reliability diagrams comparing the Qwen-2.5-7B language model's performance under three context RAG settings: (a) no context, (b) 0p10n context, and (c) 1p9n context. Each diagram evaluates calibration across four knowledge categories: highly known, maybe known, weakly known, and unknown knowledge. Bars represent accuracy (blue), gap between accuracy and perfect calibration (pink), and dashed lines indicate perfect calibration thresholds. ### Components/Axes - **X-axis**: Confidence (0.0–1.0 in 0.2 increments) - **Y-axis**: Accuracy (0.0–1.0) - **Legend**: - Dashed line: Perfect calibration (ideal accuracy = confidence) - Blue bars: Accuracy - Pink bars: Gap (perfect calibration - accuracy) - **Subcategories**: Four knowledge categories per diagram ### Detailed Analysis #### (a) No Context RAG Setting - **Highly Known Knowledge**: - Accuracy peaks at ~0.8 (confidence 0.8–1.0), closely matching perfect calibration. - Gap minimal (~0.1–0.2) in high-confidence ranges. - **Maybe Known Knowledge**: - Accuracy drops to ~0.4–0.6 (confidence 0.4–0.8), with gaps widening to ~0.3–0.5. - **Weakly Known Knowledge**: - Accuracy ~0.2–0.4 (confidence 0.2–0.6), gaps ~0.4–0.6. - **Unknown Knowledge**: - Accuracy ~0.6–0.8 (confidence 0.6–1.0), gaps ~0.2–0.4. #### (b) 0p10n Context RAG Setting - **Highly Known Knowledge**: - Accuracy declines to ~0.6–0.8 (confidence 0.6–1.0), gaps increase to ~0.2–0.4. - **Maybe Known Knowledge**: - Accuracy ~0.2–0.4 (confidence 0.2–0.6), gaps ~0.4–0.6. - **Weakly Known Knowledge**: - Accuracy ~0.1–0.3 (confidence 0.1–0.5), gaps ~0.5–0.7. - **Unknown Knowledge**: - Accuracy ~0.4–0.6 (confidence 0.4–0.8), gaps ~0.4–0.6. #### (c) 1p9n Context RAG Setting - **Highly Known Knowledge**: - Accuracy ~0.7–0.9 (confidence 0.7–1.0), gaps ~0.1–0.3. - **Maybe Known Knowledge**: - Accuracy ~0.5–0.7 (confidence 0.5–0.9), gaps ~0.3–0.5. - **Weakly Known Knowledge**: - Accuracy ~0.4–0.6 (confidence 0.4–0.8), gaps ~0.4–0.6. - **Unknown Knowledge**: - Accuracy ~0.6–0.8 (confidence 0.6–1.0), gaps ~0.2–0.4. ### Key Observations 1. **Calibration Trends**: - No context (a): Best calibration for highly known knowledge, overconfidence in unknown. - 0p10n context (b): Worst calibration across all categories, largest gaps in weakly known. - 1p9n context (c): Improved calibration, especially in unknown knowledge (gap reduced by ~50% vs. no context). 2. **Outliers**: - In (b), weakly known knowledge shows extreme overconfidence (gap >0.5 at confidence 0.4–0.6). - In (c), unknown knowledge achieves near-perfect calibration (gap <0.3 at confidence 0.8–1.0). ### Interpretation The diagrams reveal that context augmentation significantly impacts calibration: - **No context**: The model is well-calibrated for highly known knowledge but overconfident in unknown domains. - **0p10n context**: Introduces noise, degrading calibration for maybe/weakly known knowledge. - **1p9n context**: Optimal balance, improving calibration across all categories, particularly for unknown knowledge. This suggests that larger context windows (1p9n) help the model better align confidence with accuracy, reducing overconfidence in uncertain scenarios. The gap metric highlights the model's reliability: smaller gaps indicate trustworthy confidence estimates. The 1p9n context setting demonstrates the most robust performance, aligning with expectations that richer context improves model reasoning.