## Reliability Diagrams of Qwen-2.5-7B with Different Context RAG Settings ### Overview The image presents a series of reliability diagrams for the Qwen-2.5-7B model under different context RAG (Retrieval-Augmented Generation) settings. The diagrams are arranged in a 3x4 grid, with each row representing a different RAG setting: no context, 0p10n context, and 1p9n context. Within each row, the diagrams are further categorized by knowledge type: highly known, maybe known, weakly known, and unknown knowledge. Each diagram plots accuracy against confidence, showing the model's calibration for different types of knowledge under varying context conditions. ### Components/Axes Each individual chart has the following components: * **Title:** Indicates the knowledge type (highly known, maybe known, weakly known, unknown knowledge). * **X-axis:** Confidence, ranging from 0.0 to 1.0 in increments of 0.2. * **Y-axis:** Accuracy, ranging from 0.0 to 1.0 in increments of 0.2. * **Data Series:** * **Accuracy:** Represented by blue bars. The height of each bar indicates the accuracy for a given confidence level. * **Gap:** Represented by light red bars. The height of each bar indicates the gap between perfect calibration and the actual accuracy. * **Perfect Calibration:** Represented by a dashed gray diagonal line. This line indicates perfect calibration, where confidence equals accuracy. * **Legend:** Located in the top-left corner of each chart, indicating the color and label for each data series: * Perfect calibration (dashed gray line) * Accuracy (blue bars) * Gap (light red bars) ### Detailed Analysis **Row 1: (a) Reliability Diagram of Qwen-2.5-7B with no context RAG setting** * **Highly Known Knowledge:** * Accuracy: The accuracy is low at confidence 0.2 (approximately 0.1), then increases significantly at confidence 0.6 (approximately 0.9), and remains high at confidence 1.0 (approximately 0.95). * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases at confidence 0.6 (approximately 0.1), and remains small at confidence 1.0 (approximately 0.05). * **Maybe Known Knowledge:** * Accuracy: The accuracy starts low at confidence 0.2 (approximately 0.1), increases at confidence 0.6 (approximately 0.6), and reaches approximately 0.75 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases at confidence 0.6 (approximately 0.4), and remains moderate at confidence 1.0 (approximately 0.25). * **Weakly Known Knowledge:** * Accuracy: The accuracy is very low at confidence 0.2 (approximately 0.0), increases slightly at confidence 0.6 (approximately 0.2), and reaches approximately 0.35 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases slightly at confidence 0.6 (approximately 0.4), and remains large at confidence 1.0 (approximately 0.65). * **Unknown Knowledge:** * Accuracy: The accuracy is very low at confidence 0.2 (approximately 0.0), increases slightly at confidence 0.6 (approximately 0.1), and reaches approximately 0.2 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases slightly at confidence 0.6 (approximately 0.5), and remains large at confidence 1.0 (approximately 0.8). **Row 2: (b) Reliability Diagram of Qwen-2.5-7B with 0p10n context RAG setting** * **Highly Known Knowledge:** * Accuracy: The accuracy is low at confidence 0.2 (approximately 0.1), then increases significantly at confidence 0.6 (approximately 0.7), and remains high at confidence 1.0 (approximately 0.7). * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases at confidence 0.6 (approximately 0.3), and remains moderate at confidence 1.0 (approximately 0.3). * **Maybe Known Knowledge:** * Accuracy: The accuracy starts low at confidence 0.2 (approximately 0.0), increases at confidence 0.6 (approximately 0.4), and reaches approximately 0.5 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases at confidence 0.6 (approximately 0.6), and remains moderate at confidence 1.0 (approximately 0.5). * **Weakly Known Knowledge:** * Accuracy: The accuracy is very low at confidence 0.2 (approximately 0.0), increases slightly at confidence 0.6 (approximately 0.1), and reaches approximately 0.2 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases slightly at confidence 0.6 (approximately 0.5), and remains large at confidence 1.0 (approximately 0.8). * **Unknown Knowledge:** * Accuracy: The accuracy is very low at confidence 0.2 (approximately 0.0), increases slightly at confidence 0.6 (approximately 0.0), and reaches approximately 0.5 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases slightly at confidence 0.6 (approximately 0.5), and remains large at confidence 1.0 (approximately 0.5). **Row 3: (c) Reliability Diagram of Qwen-2.5-7B with 1p9n context RAG setting** * **Highly Known Knowledge:** * Accuracy: The accuracy is low at confidence 0.2 (approximately 0.0), then increases significantly at confidence 0.6 (approximately 0.8), and remains high at confidence 1.0 (approximately 0.9). * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases at confidence 0.6 (approximately 0.2), and remains small at confidence 1.0 (approximately 0.1). * **Maybe Known Knowledge:** * Accuracy: The accuracy starts low at confidence 0.2 (approximately 0.0), increases at confidence 0.6 (approximately 0.6), and reaches approximately 0.95 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases at confidence 0.6 (approximately 0.4), and remains moderate at confidence 1.0 (approximately 0.05). * **Weakly Known Knowledge:** * Accuracy: The accuracy is very low at confidence 0.2 (approximately 0.0), increases slightly at confidence 0.6 (approximately 0.5), and reaches approximately 0.9 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases slightly at confidence 0.6 (approximately 0.5), and remains large at confidence 1.0 (approximately 0.1). * **Unknown Knowledge:** * Accuracy: The accuracy is very low at confidence 0.2 (approximately 0.0), increases slightly at confidence 0.6 (approximately 0.7), and reaches approximately 0.8 at confidence 1.0. * Gap: The gap is large at confidence 0.2 (approximately 0.2), decreases slightly at confidence 0.6 (approximately 0.3), and remains large at confidence 1.0 (approximately 0.2). ### Key Observations * **Impact of Context RAG:** The reliability diagrams show that the context RAG settings (0p10n and 1p9n) generally improve the accuracy of the Qwen-2.5-7B model, especially for weakly known and unknown knowledge. * **Knowledge Type:** The model performs best on highly known knowledge, followed by maybe known, weakly known, and unknown knowledge. This trend is consistent across all context RAG settings. * **Calibration:** The model tends to be overconfident, especially for weakly known and unknown knowledge, as indicated by the large gap between confidence and accuracy. * **Confidence Levels:** Accuracy generally increases with confidence, but the rate of increase varies depending on the knowledge type and context RAG setting. ### Interpretation The reliability diagrams provide insights into the calibration and accuracy of the Qwen-2.5-7B model under different context RAG settings. The data suggests that incorporating context through RAG can significantly improve the model's performance, particularly for less familiar knowledge domains. However, the model still exhibits overconfidence, especially for weakly known and unknown knowledge, indicating a need for further calibration techniques. The choice of context RAG setting (0p10n vs. 1p9n) also appears to influence the model's reliability, suggesting that the specific context provided plays a crucial role in the model's performance. Overall, these diagrams highlight the importance of evaluating and improving the calibration of language models to ensure reliable and trustworthy predictions.