## Scatter Plot: ProofWriter Model Performance (Llama3.1-8B vs 70B)
### Overview
This image displays two scatter plots comparing the performance of two Large Language Models (Llama3.1-8B and Llama3.1-70B) on the "ProofWriter" dataset. The plots visualize the relationship between "Entropy" (x-axis) and "JS Divergence" (y-axis). Data points are color-coded to indicate whether the model's output was "Correct (T)" (blue) or "Incorrect (F)" (red).
### Components/Axes
**Top Chart: ProofWriter - Llama3.1-8B**
* **X-Axis:** "Entropy", ranging from 0 to 40.
* **Y-Axis:** "JS Divergence", ranging from 0.04 to 0.12.
* **Legend:** Located in the top-right corner.
* Blue circle: "Correct (T)"
* Red circle: "Incorrect (F)"
* **Annotations:**
* A green dashed vertical line at Entropy $\approx$ 3.5.
* A purple dashed horizontal line at JS Divergence $\approx$ 0.068.
* An arrow points from the text "Lowest Entropy (Incorrect)" to a red dot located at the intersection of these two lines.
**Bottom Chart: ProofWriter - Llama3.1-70B**
* **X-Axis:** "Entropy", ranging from 0 to 70.
* **Y-Axis:** "JS Divergence", ranging from 0.04 to 0.15.
* **Legend:** Located in the top-right corner.
* Blue circle: "Correct (T)"
* Red circle: "Incorrect (F)"
* **Annotations:**
* A green dashed vertical line at Entropy $\approx$ 8.
* A purple dashed horizontal line at JS Divergence $\approx$ 0.077.
* An arrow points from the text "Lowest Entropy (Incorrect)" to a red dot located at the intersection of these two lines.
### Detailed Analysis
**Trend Verification:**
* **General Trend:** In both charts, there is a positive correlation between Entropy and JS Divergence for the "Correct" (blue) data points. As the entropy (uncertainty) increases, the JS Divergence (a measure of difference between probability distributions) also increases.
* **Distribution:**
* **8B Model:** The majority of "Correct" (blue) points are clustered between Entropy 2 and 15, with JS Divergence between 0.04 and 0.09. "Incorrect" (red) points are scattered throughout, but notably appear in a cluster at higher entropy levels (15-25).
* **70B Model:** The "Correct" (blue) points are more densely clustered between Entropy 2 and 20, with JS Divergence between 0.04 and 0.12. There is a distinct, sparse tail of "Incorrect" (red) points extending to the far right, reaching Entropy values up to 70.
**Spatial Grounding:**
* **8B Chart:** The "Lowest Entropy (Incorrect)" annotation is positioned in the top-left quadrant of the plot area.
* **70B Chart:** The "Lowest Entropy (Incorrect)" annotation is positioned in the bottom-left quadrant of the plot area, with the arrow pointing upward to the intersection point.
### Key Observations
* **Model Scale Difference:** The 70B model exhibits a significantly wider range of entropy values (up to 70) compared to the 8B model (up to 40).
* **Calibration/Confidence:** The presence of "Incorrect" (red) points at very low entropy levels (the "Lowest Entropy" markers) indicates that the models are occasionally "confidently wrong"—they produce low-entropy (high-confidence) outputs that are nonetheless incorrect.
* **JS Divergence Ceiling:** The 70B model reaches higher JS Divergence values (up to ~0.14) than the 8B model (up to ~0.12).
### Interpretation
This data visualizes the relationship between model uncertainty (Entropy) and the divergence of the model's probability distribution (JS Divergence).
1. **Calibration Issues:** The "Lowest Entropy (Incorrect)" annotations are critical. They highlight instances where the model is highly confident (low entropy) but incorrect. This is a classic indicator of poor calibration, where the model's internal confidence does not align with its actual accuracy.
2. **Model Behavior:** The 70B model shows a broader spread of entropy, suggesting it may be exploring a wider range of probability distributions or has a different confidence threshold mechanism than the 8B model.
3. **Correlation:** The positive correlation between Entropy and JS Divergence suggests that as the model becomes more uncertain, the divergence of its output distribution increases, which is expected behavior for probabilistic models. The fact that "Incorrect" answers are often associated with higher entropy (in the 8B model) or specific high-entropy clusters (in the 70B model) suggests that uncertainty is a reasonable, though imperfect, proxy for identifying potential errors.
