## Chart: I-Don't-Know Rate vs. Layer for Llama-3-8B and Llama-3-70B ### Overview The image presents two line charts comparing the "I-Don't-Know Rate" across different layers of two language models, Llama-3-8B and Llama-3-70B. Each chart plots the rate for question-anchored (Q-Anchored) and answer-anchored (A-Anchored) data across various question-answering datasets: PopQA, TriviaQA, HotpotQA, and NQ. The x-axis represents the layer number, and the y-axis represents the I-Don't-Know Rate, ranging from 0 to 100. ### Components/Axes * **Chart Titles:** "Llama-3-8B" (left chart) and "Llama-3-70B" (right chart). * **Y-Axis Label:** "I-Don't-Know Rate" * Scale: 0 to 100, with tick marks at 0, 20, 40, 60, 80, and 100. * **X-Axis Label:** "Layer" * Llama-3-8B: 0 to 30, with tick marks every 10 layers. * Llama-3-70B: 0 to 80, with tick marks every 20 layers. * **Legend:** Located at the bottom of the image. * **Q-Anchored (PopQA):** Solid blue line * **A-Anchored (PopQA):** Dashed brown line * **Q-Anchored (TriviaQA):** Solid green line * **A-Anchored (TriviaQA):** Dashed light green line * **Q-Anchored (HotpotQA):** Solid orange line * **A-Anchored (HotpotQA):** Dashed orange line * **Q-Anchored (NQ):** Dashed purple line * **A-Anchored (NQ):** Dotted gray line ### Detailed Analysis #### Llama-3-8B (Left Chart) * **Q-Anchored (PopQA):** (Solid Blue) Starts at approximately 95 at layer 0, drops sharply to around 10 by layer 10, and then remains relatively stable between 5 and 15 for the remaining layers. * **A-Anchored (PopQA):** (Dashed Brown) Starts at approximately 50 at layer 0, increases to around 60-65 by layer 10, and then fluctuates between 60 and 70 for the remaining layers. * **Q-Anchored (TriviaQA):** (Solid Green) Starts at approximately 60 at layer 0, drops to around 5-10 by layer 15, and then remains relatively stable between 0 and 10 for the remaining layers. * **A-Anchored (TriviaQA):** (Dashed Light Green) Starts at approximately 50 at layer 0, drops to around 10-15 by layer 15, and then remains relatively stable between 5 and 15 for the remaining layers. * **Q-Anchored (NQ):** (Dashed Purple) Starts at approximately 50 at layer 0, drops to around 10-20 by layer 10, and then fluctuates between 10 and 30 for the remaining layers. * **A-Anchored (NQ):** (Dotted Gray) Starts at approximately 60 at layer 0, increases to around 60-70 by layer 10, and then fluctuates between 60 and 70 for the remaining layers. #### Llama-3-70B (Right Chart) * **Q-Anchored (PopQA):** (Solid Blue) Starts at approximately 95 at layer 0, drops sharply to around 20 by layer 20, and then fluctuates between 15 and 25 for the remaining layers. * **A-Anchored (PopQA):** (Dashed Brown) Starts at approximately 60 at layer 0, increases to around 70-80 by layer 20, and then fluctuates between 65 and 75 for the remaining layers. * **Q-Anchored (TriviaQA):** (Solid Green) Starts at approximately 60 at layer 0, drops to around 10-20 by layer 20, and then fluctuates between 10 and 20 for the remaining layers. * **A-Anchored (TriviaQA):** (Dashed Light Green) Starts at approximately 50 at layer 0, drops to around 20-30 by layer 20, and then fluctuates between 20 and 30 for the remaining layers. * **Q-Anchored (HotpotQA):** (Solid Orange) Starts at approximately 60 at layer 0, increases to around 70-80 by layer 20, and then fluctuates between 70 and 90 for the remaining layers. * **A-Anchored (HotpotQA):** (Dashed Orange) Starts at approximately 60 at layer 0, increases to around 70-80 by layer 20, and then fluctuates between 65 and 75 for the remaining layers. * **Q-Anchored (NQ):** (Dashed Purple) Starts at approximately 50 at layer 0, drops to around 20-30 by layer 20, and then fluctuates between 20 and 40 for the remaining layers. * **A-Anchored (NQ):** (Dotted Gray) Starts at approximately 60 at layer 0, increases to around 60-70 by layer 20, and then fluctuates between 60 and 70 for the remaining layers. ### Key Observations * For both models, the Q-Anchored (PopQA), Q-Anchored (TriviaQA), and Q-Anchored (NQ) rates decrease significantly in the initial layers. * For both models, the A-Anchored (PopQA), A-Anchored (TriviaQA), A-Anchored (HotpotQA), and A-Anchored (NQ) rates tend to increase or remain relatively stable across layers. * The Llama-3-70B model shows more fluctuation in the I-Don't-Know Rate across layers compared to the Llama-3-8B model. * The HotpotQA dataset shows a higher I-Don't-Know Rate for Q-Anchored data in the Llama-3-70B model. ### Interpretation The data suggests that the initial layers of the language models are crucial for reducing the "I-Don't-Know Rate" for question-anchored data. The difference between Q-Anchored and A-Anchored rates indicates that the model's confidence varies depending on whether the question or the answer is used as the anchor. The fluctuations in the Llama-3-70B model might be due to its larger size and complexity, leading to more variability in its responses across different layers. The higher I-Don't-Know Rate for HotpotQA in the Llama-3-70B model could indicate that this dataset poses a greater challenge for the model.

## Line Chart: I-Don't-Know Rate vs. Layer for Llama Models ### Overview The image presents two line charts, side-by-side, displaying the "I-Don't-Know Rate" as a function of "Layer" for two different Llama models: Llama-3-8B and Llama-3-70B. Each chart contains multiple lines representing different question-answering datasets and anchoring methods. The charts aim to visualize how the model's uncertainty (expressed as the I-Don't-Know Rate) changes across different layers of the neural network. ### Components/Axes * **X-axis:** "Layer" - Ranges from 0 to 30 for Llama-3-8B and 0 to 80 for Llama-3-70B. The scale is linear. * **Y-axis:** "I-Don't-Know Rate" - Ranges from 0 to 100, representing a percentage. The scale is linear. * **Title (Left Chart):** "Llama-3-8B" * **Title (Right Chart):** "Llama-3-70B" * **Legend:** Located at the bottom of the image, below both charts. It identifies the different lines based on anchoring method ("Q-Anchored" or "A-Anchored") and the question-answering dataset (PopQA, TriviaQA, HotpotQA, NQ). * Q-Anchored (PopQA) - Blue solid line * A-Anchored (PopQA) - Orange dashed line * Q-Anchored (TriviaQA) - Light Blue solid line * A-Anchored (TriviaQA) - Purple dashed line * Q-Anchored (HotpotQA) - Green dashed line * A-Anchored (HotpotQA) - Red dashed line * Q-Anchored (NQ) - Cyan solid line * A-Anchored (NQ) - Magenta dashed line ### Detailed Analysis or Content Details **Llama-3-8B Chart:** * **Q-Anchored (PopQA):** Starts at approximately 95% I-Don't-Know Rate at Layer 0, rapidly decreasing to around 10% by Layer 5, then fluctuating between 10% and 30% for the remainder of the layers, with a slight upward trend towards Layer 30, ending at approximately 30%. * **A-Anchored (PopQA):** Starts at approximately 70% I-Don't-Know Rate at Layer 0, decreasing to around 50% by Layer 5, then remaining relatively stable between 50% and 70% for the rest of the layers, ending at approximately 60%. * **Q-Anchored (TriviaQA):** Starts at approximately 80% I-Don't-Know Rate at Layer 0, decreasing to around 20% by Layer 5, then fluctuating between 20% and 40% for the remainder of the layers, ending at approximately 30%. * **A-Anchored (TriviaQA):** Starts at approximately 60% I-Don't-Know Rate at Layer 0, decreasing to around 40% by Layer 5, then remaining relatively stable between 40% and 60% for the rest of the layers, ending at approximately 50%. * **Q-Anchored (HotpotQA):** Starts at approximately 60% I-Don't-Know Rate at Layer 0, decreasing to around 20% by Layer 5, then fluctuating between 20% and 40% for the remainder of the layers, ending at approximately 30%. * **A-Anchored (HotpotQA):** Starts at approximately 50% I-Don't-Know Rate at Layer 0, decreasing to around 30% by Layer 5, then remaining relatively stable between 30% and 50% for the rest of the layers, ending at approximately 40%. * **Q-Anchored (NQ):** Starts at approximately 70% I-Don't-Know Rate at Layer 0, decreasing to around 20% by Layer 5, then fluctuating between 20% and 40% for the remainder of the layers, ending at approximately 30%. * **A-Anchored (NQ):** Starts at approximately 50% I-Don't-Know Rate at Layer 0, decreasing to around 30% by Layer 5, then remaining relatively stable between 30% and 50% for the rest of the layers, ending at approximately 40%. **Llama-3-70B Chart:** * **Q-Anchored (PopQA):** Starts at approximately 90% I-Don't-Know Rate at Layer 0, decreasing to around 20% by Layer 10, then fluctuating between 20% and 40% for the remainder of the layers, ending at approximately 30%. * **A-Anchored (PopQA):** Starts at approximately 70% I-Don't-Know Rate at Layer 0, decreasing to around 50% by Layer 10, then remaining relatively stable between 50% and 70% for the rest of the layers, ending at approximately 60%. * **Q-Anchored (TriviaQA):** Starts at approximately 80% I-Don't-Know Rate at Layer 0, decreasing to around 30% by Layer 10, then fluctuating between 30% and 50% for the remainder of the layers, ending at approximately 40%. * **A-Anchored (TriviaQA):** Starts at approximately 60% I-Don't-Know Rate at Layer 0, decreasing to around 40% by Layer 10, then remaining relatively stable between 40% and 60% for the rest of the layers, ending at approximately 50%. * **Q-Anchored (HotpotQA):** Starts at approximately 60% I-Don't-Know Rate at Layer 0, decreasing to around 20% by Layer 10, then fluctuating between 20% and 40% for the remainder of the layers, ending at approximately 30%. * **A-Anchored (HotpotQA):** Starts at approximately 50% I-Don't-Know Rate at Layer 0, decreasing to around 30% by Layer 10, then remaining relatively stable between 30% and 50% for the rest of the layers, ending at approximately 40%. * **Q-Anchored (NQ):** Starts at approximately 70% I-Don't-Know Rate at Layer 0, decreasing to around 20% by Layer 10, then fluctuating between 20% and 40% for the remainder of the layers, ending at approximately 30%. * **A-Anchored (NQ):** Starts at approximately 50% I-Don't-Know Rate at Layer 0, decreasing to around 30% by Layer 10, then remaining relatively stable between 30% and 50% for the rest of the layers, ending at approximately 40%. ### Key Observations * In both models, the I-Don't-Know Rate generally decreases rapidly in the initial layers (0-10) and then plateaus. * Q-Anchored methods consistently exhibit lower I-Don't-Know Rates compared to A-Anchored methods across all datasets. * The PopQA dataset generally shows a higher I-Don't-Know Rate than other datasets, particularly for A-Anchored methods. * The 70B model appears to have a slightly lower I-Don't-Know Rate overall compared to the 8B model, especially in the later layers. ### Interpretation The charts demonstrate how the model's confidence (or lack thereof) evolves as information propagates through its layers. The initial high I-Don't-Know Rate suggests that the model initially lacks sufficient information to answer questions. As the data flows through the layers, the model learns to reduce its uncertainty. The difference between Q-Anchored and A-Anchored methods suggests that the method used to provide context (question vs. answer) influences the model's confidence. The higher I-Don't-Know Rate for PopQA might indicate that this dataset presents more challenging or ambiguous questions. The larger model (70B) generally exhibits lower uncertainty, indicating that increased model size can lead to improved performance and confidence. The plateauing of the I-Don't-Know Rate suggests that further increasing the number of layers may not significantly improve the model's ability to answer questions, or that other factors are limiting performance.

## Line Charts: Llama-3 Model "I-Don't-Know Rate" Across Layers ### Overview The image displays two side-by-side line charts comparing the "I-Don't-Know Rate" across the layers of two large language models: Llama-3-8B (left chart) and Llama-3-70B (right chart). Each chart plots multiple data series representing different question-answering datasets (PopQA, TriviaQA, HotpotQA, NQ) under two experimental conditions: "Q-Anchored" and "A-Anchored". The charts illustrate how the models' propensity to output "I don't know" changes as information propagates through the network's layers. ### Components/Axes * **Chart Titles:** * Left Chart: `Llama-3-8B` * Right Chart: `Llama-3-70B` * **Y-Axis (Both Charts):** Label is `I-Don't-Know Rate`. Scale runs from 0 to 100 in increments of 20. * **X-Axis (Both Charts):** Label is `Layer`. * For Llama-3-8B, the scale runs from 0 to 30. * For Llama-3-70B, the scale runs from 0 to 80. * **Legend (Bottom, spanning both charts):** Contains 8 entries, differentiating series by color and line style (solid vs. dashed). * **Solid Lines (Q-Anchored):** * Blue: `Q-Anchored (PopQA)` * Green: `Q-Anchored (TriviaQA)` * Purple: `Q-Anchored (HotpotQA)` * Pink: `Q-Anchored (NQ)` * **Dashed Lines (A-Anchored):** * Orange: `A-Anchored (PopQA)` * Red: `A-Anchored (TriviaQA)` * Brown: `A-Anchored (HotpotQA)` * Gray: `A-Anchored (NQ)` ### Detailed Analysis **Llama-3-8B Chart (Left):** * **Q-Anchored Series (Solid Lines):** All four solid lines show a similar, dramatic trend. They start at a very high rate (between ~80-100) at Layer 0, then plummet sharply within the first 5-10 layers to a low point (between ~0-20). After this initial drop, they exhibit a gradual, fluctuating upward trend through the remaining layers, ending between approximately 10-40 at Layer 30. * *Trend Verification:* Steep initial decline followed by a slow, noisy recovery. * **A-Anchored Series (Dashed Lines):** These lines follow a distinctly different pattern. They start at a moderate level (between ~40-60) at Layer 0. They show a general, fluctuating upward trend throughout all layers, ending at a higher rate (between ~60-75) at Layer 30. They do not exhibit the sharp initial drop seen in the Q-Anchored lines. * *Trend Verification:* General upward drift with significant layer-to-layer fluctuation. * **Spatial Grounding:** The Q-Anchored (solid) lines are consistently below the A-Anchored (dashed) lines from approximately Layer 5 onward. The highest final value belongs to the red dashed line (A-Anchored, TriviaQA). **Llama-3-70B Chart (Right):** * **Q-Anchored Series (Solid Lines):** Similar to the 8B model, these lines start high (near 100) and drop sharply in the early layers (0-10). However, the subsequent behavior is more volatile. They fluctuate significantly, with some series (notably blue - PopQA) showing a pronounced secondary dip around Layer 40 before rising again. Final values at Layer 80 are generally low, clustered between ~10-30. * *Trend Verification:* Sharp initial drop, followed by high volatility and a general low plateau. * **A-Anchored Series (Dashed Lines):** These lines start at a moderate-to-high level (~60-80). They exhibit a strong, fluctuating upward trend, peaking around Layers 30-50 (with some values exceeding 80), before slightly declining or stabilizing towards Layer 80. Final values remain high, between ~60-80. * *Trend Verification:* Strong rise to a mid-network peak, followed by a slight decline or plateau. * **Spatial Grounding:** The separation between Q-Anchored (solid, lower) and A-Anchored (dashed, higher) groups is even more pronounced and consistent across most layers compared to the 8B model. The orange dashed line (A-Anchored, PopQA) appears to be among the highest for much of the chart. ### Key Observations 1. **Fundamental Dichotomy:** There is a clear and consistent separation between the behavior of Q-Anchored (solid lines) and A-Anchored (dashed lines) conditions across both models. Q-Anchored leads to a low "I-Don't-Know Rate" after early layers, while A-Anchored maintains a high rate. 2. **Model Size Effect:** The larger Llama-3-70B model shows more pronounced volatility in its Q-Anchored rates and a more defined peak in its A-Anchored rates compared to the 8B model. The layer scale is also more than double. 3. **Early-Layer Criticality:** The most dramatic change for Q-Anchored series occurs in the first ~10 layers, suggesting this is where the model's internal "confidence" or answer formulation is most actively determined. 4. **Dataset Variation:** While the overall trends are consistent per anchoring method, there is notable variation between datasets (different colors). For example, in the 70B model, the blue Q-Anchored (PopQA) line shows a unique secondary dip. ### Interpretation The data suggests a fundamental difference in how the model processes information based on the anchoring method. **Q-Anchoring** (likely conditioning on the question) appears to drive the model toward committing to an answer (or a "know" state) very early in its processing stream, resulting in a low "I-Don't-Know Rate" for the majority of the network. The subsequent slow rise may indicate a gradual reintroduction of uncertainty or a refinement process. In contrast, **A-Anchoring** (likely conditioning on a potential answer) seems to keep the model in a more evaluative or uncertain state throughout its processing. The high and even increasing "I-Don't-Know Rate" suggests the model is constantly weighing the anchored answer against its internal knowledge, leading to higher expressed uncertainty. The peak in the middle layers of the 70B model could represent a point of maximal information integration or conflict resolution. The stark contrast between the two conditions implies that the model's internal representation of "knowing" vs. "not knowing" is highly sensitive to the initial framing or prompt structure. This has significant implications for understanding model confidence and for designing prompts that elicit more calibrated expressions of uncertainty. The increased volatility in the larger model may reflect a more complex and nuanced internal deliberation process.

## Line Chart: I-Don't-Know Rate Across Llama-3 Models and Datasets ### Overview The image presents two line charts comparing the "I-Don't-Know Rate" (percentage of instances where a model responds with "I don't know") across different layers of the Llama-3-8B and Llama-3-70B models. The data is segmented by dataset (PopQA, TriviaQA, HotpotQA, NQ) and anchoring type (Q-Anchored vs. A-Anchored). The charts visualize how the I-Don't-Know rate varies with model layers and dataset-specific characteristics. --- ### Components/Axes - **X-Axis (Layer)**: - Llama-3-8B: 0 to 30 (in increments of 10) - Llama-3-70B: 0 to 80 (in increments of 20) - **Y-Axis (I-Don't-Know Rate)**: 0 to 100 (percentage) - **Legend**: - **Q-Anchored (PopQA)**: Blue solid line - **Q-Anchored (TriviaQA)**: Green solid line - **Q-Anchored (HotpotQA)**: Purple solid line - **Q-Anchored (NQ)**: Pink solid line - **A-Anchored (PopQA)**: Blue dashed line - **A-Anchored (TriviaQA)**: Green dashed line - **A-Anchored (HotpotQA)**: Purple dashed line - **A-Anchored (NQ)**: Pink dashed line - **Chart Titles**: - Left: "Llama-3-8B" - Right: "Llama-3-70B" --- ### Detailed Analysis #### Llama-3-8B (Left Chart) - **PopQA (Blue Solid)**: - Starts at ~80% in layer 0, drops sharply to ~20% by layer 10, then fluctuates between 10–30%. - **TriviaQA (Green Solid)**: - Begins at ~60%, dips to ~10% by layer 10, then rises to ~40% by layer 30. - **HotpotQA (Purple Solid)**: - Peaks at ~90% in layer 0, drops to ~30% by layer 10, then stabilizes around 20–40%. - **NQ (Pink Solid)**: - Starts at ~70%, declines to ~20% by layer 10, then oscillates between 10–30%. #### Llama-3-70B (Right Chart) - **PopQA (Blue Solid)**: - Begins at ~85%, drops to ~25% by layer 20, then fluctuates between 10–40%. - **TriviaQA (Green Solid)**: - Starts at ~65%, dips to ~15% by layer 20, then rises to ~50% by layer 60. - **HotpotQA (Purple Solid)**: - Peaks at ~95% in layer 0, drops to ~35% by layer 20, then stabilizes around 20–50%. - **NQ (Pink Solid)**: - Begins at ~75%, declines to ~25% by layer 20, then oscillates between 10–40%. --- ### Key Observations 1. **Model Size Impact**: - Llama-3-70B shows more stable I-Don't-Know rates across layers compared to Llama-3-8B, suggesting larger models may handle uncertainty more consistently. 2. **Dataset Variability**: - **HotpotQA** consistently exhibits the highest I-Don't-Know rates (up to 95% in layer 0), indicating it is the most challenging dataset. - **NQ** shows the most erratic fluctuations, with sharp drops and rises across layers. 3. **Anchoring Type**: - Q-Anchored (solid lines) and A-Anchored (dashed lines) trends are visually similar, but Q-Anchored lines often start higher in layer 0. 4. **Layer-Specific Trends**: - Early layers (0–10) show the highest I-Don't-Know rates, with a general decline as layers increase, though some datasets (e.g., TriviaQA) exhibit late-layer spikes. --- ### Interpretation The data suggests that: - **Model Size**: Larger models (Llama-3-70B) demonstrate more stable I-Don't-Know rates, possibly due to better generalization or reduced uncertainty in deeper layers. - **Dataset Complexity**: HotpotQA's high initial rates imply it tests the model's ability to handle complex, multi-step reasoning, while NQ's volatility may reflect its reliance on ambiguous or context-dependent queries. - **Anchoring Methods**: The lack of significant divergence between Q-Anchored and A-Anchored lines suggests that anchoring type has minimal impact on the I-Don't-Know rate, though Q-Anchored lines may reflect initial confidence biases. Notable anomalies include the sharp drop in HotpotQA's I-Don't-Know rate after layer 10, which could indicate a shift in model behavior or dataset-specific thresholds. The persistent fluctuations in NQ highlight its sensitivity to layer-specific model dynamics.