## Chart: I-Don't-Know Rate vs. Layer for Llama-3-8B and Llama-3-70B
### Overview
The image presents two line charts comparing the "I-Don't-Know Rate" across different layers of two language models: Llama-3-8B and Llama-3-70B. Each chart displays six data series, representing different question-answering datasets (PopQA, TriviaQA, HotpotQA, and NQ) anchored either to the question (Q-Anchored) or the answer (A-Anchored). The x-axis represents the layer number, and the y-axis represents the I-Don't-Know Rate, ranging from 0 to 100. Shaded regions around each line indicate the uncertainty or variance in the data.
### Components/Axes
* **Titles:**
* Left Chart: Llama-3-8B
* Right Chart: Llama-3-70B
* **Y-Axis:**
* Label: I-Don't-Know Rate
* Scale: 0 to 100, with tick marks at 0, 20, 40, 60, 80, and 100.
* **X-Axis:**
* Label: Layer
* Scale (Llama-3-8B): 0 to 30, with tick marks every 10 units.
* Scale (Llama-3-70B): 0 to 80, with tick marks every 20 units.
* **Legend:** Located at the bottom of the image.
* Q-Anchored (PopQA): Solid Blue Line
* A-Anchored (PopQA): Dashed Brown Line
* Q-Anchored (TriviaQA): Dotted Green Line
* A-Anchored (TriviaQA): Dashed-Dotted Red Line
* Q-Anchored (HotpotQA): Dashed Purple Line
* A-Anchored (HotpotQA): Dotted Gray Line
* Q-Anchored (NQ): Dashed-Dotted Pink Line
* A-Anchored (NQ): Dotted Dark Gray Line
### Detailed Analysis
#### Llama-3-8B (Left Chart)
* **Q-Anchored (PopQA) - Solid Blue Line:** Starts at approximately 0 and remains low, generally below 10, with some fluctuations.
* **A-Anchored (PopQA) - Dashed Brown Line:** Starts around 50, increases to approximately 65 by layer 10, and then fluctuates around 60-70 for the remaining layers.
* **Q-Anchored (TriviaQA) - Dotted Green Line:** Starts high, around 100, then drops sharply to around 10 by layer 5, and fluctuates between 10 and 20 for the remaining layers.
* **A-Anchored (TriviaQA) - Dashed-Dotted Red Line:** Starts around 50, increases to approximately 70 by layer 10, and then fluctuates around 60-70 for the remaining layers.
* **Q-Anchored (HotpotQA) - Dashed Purple Line:** Starts around 50, decreases to approximately 20 by layer 10, and then fluctuates between 20 and 40 for the remaining layers.
* **A-Anchored (HotpotQA) - Dotted Gray Line:** Starts around 60, remains relatively stable, fluctuating between 55 and 65 for all layers.
* **Q-Anchored (NQ) - Dashed-Dotted Pink Line:** Starts around 50, increases to approximately 60 by layer 10, and then fluctuates around 60-70 for the remaining layers.
* **A-Anchored (NQ) - Dotted Dark Gray Line:** Starts around 60, remains relatively stable, fluctuating between 55 and 65 for all layers.
#### Llama-3-70B (Right Chart)
* **Q-Anchored (PopQA) - Solid Blue Line:** Starts at approximately 0 and remains low, generally below 20, with some fluctuations.
* **A-Anchored (PopQA) - Dashed Brown Line:** Starts around 50, increases to approximately 75 by layer 20, and then fluctuates around 70-80 for the remaining layers.
* **Q-Anchored (TriviaQA) - Dotted Green Line:** Starts high, around 100, then drops sharply to around 20 by layer 10, and fluctuates between 15 and 30 for the remaining layers.
* **A-Anchored (TriviaQA) - Dashed-Dotted Red Line:** Starts around 50, increases to approximately 80 by layer 20, and then fluctuates around 75-85 for the remaining layers.
* **Q-Anchored (HotpotQA) - Dashed Purple Line:** Starts around 50, decreases to approximately 30 by layer 20, and then fluctuates between 20 and 40 for the remaining layers.
* **A-Anchored (HotpotQA) - Dotted Gray Line:** Starts around 60, remains relatively stable, fluctuating between 60 and 70 for all layers.
* **Q-Anchored (NQ) - Dashed-Dotted Pink Line:** Starts around 50, increases to approximately 70 by layer 20, and then fluctuates around 65-75 for the remaining layers.
* **A-Anchored (NQ) - Dotted Dark Gray Line:** Starts around 60, remains relatively stable, fluctuating between 60 and 70 for all layers.
### Key Observations
* **Q-Anchored (PopQA):** Consistently low "I-Don't-Know Rate" for both models.
* **Q-Anchored (TriviaQA):** Starts high but drops significantly in the initial layers for both models.
* **A-Anchored series:** Generally exhibit higher and more stable "I-Don't-Know Rates" compared to their Q-Anchored counterparts.
* **Llama-3-70B:** Shows a more extended range of layers (0-80) compared to Llama-3-8B (0-30).
* **Variance:** The shaded regions indicate varying degrees of uncertainty across different datasets and layers.
### Interpretation
The charts illustrate how the "I-Don't-Know Rate" varies across different layers of the Llama-3-8B and Llama-3-70B language models when answering questions from different datasets. The anchoring method (question vs. answer) significantly impacts the "I-Don't-Know Rate." Q-Anchored PopQA consistently shows a low rate, suggesting the model is more confident in answering these questions. In contrast, Q-Anchored TriviaQA starts with high uncertainty but quickly learns to answer the questions, as indicated by the sharp drop in the "I-Don't-Know Rate." The A-Anchored series generally maintain higher rates, possibly indicating that the model is less certain when the answer is the primary focus. The Llama-3-70B model, with its larger number of layers, exhibits similar trends but over a more extended processing range. The variance in the data suggests that the model's confidence varies depending on the specific questions and the layer being processed. Overall, the data highlights the importance of dataset characteristics and anchoring methods in influencing the confidence and performance of language models.
