## Line Chart: Licheness Puzzle Accuracy vs. Training Step for Different Models
### Overview
This image presents three line charts, each displaying the Licheness Puzzle Accuracy (ACC) as a function of Training Step for three different language models: Qwen2.5-3B, Qwen2.5-7B, and Llama3.1-8B. Each chart includes three data series representing different reward schemes: Dense reward, Sparse reward, and Action SFT.
### Components/Axes
* **X-axis:** Training Step, ranging from 0 to 150.
* **Y-axis:** Licheness Puzzle ACC, ranging from 0.00 to 0.30 (scales vary slightly between charts).
* **Legend:** Located in the bottom-left corner of the first chart (Qwen2.5-3B), and applies to all three charts.
* **Dense reward:** Represented by a solid blue line.
* **Sparse reward:** Represented by a solid light-blue line.
* **Action SFT:** Represented by a dotted red line.
* **Titles:** Each chart is labeled with the corresponding model name: "Qwen2.5-3B", "Qwen2.5-7B", and "Llama3.1-8B".
### Detailed Analysis
**Qwen2.5-3B Chart:**
* **Dense reward:** Starts at approximately 0.01 at Training Step 0, increases steadily to approximately 0.23 at Training Step 150. The line exhibits a slight plateau between Training Steps 90 and 120.
* **Sparse reward:** Starts at approximately 0.00 at Training Step 0, increases rapidly to approximately 0.16 at Training Step 60, then plateaus around 0.16-0.18 until Training Step 150.
* **Action SFT:** Remains relatively constant at approximately 0.15 across all Training Steps.
**Qwen2.5-7B Chart:**
* **Dense reward:** Starts at approximately 0.02 at Training Step 0, increases rapidly and reaches approximately 0.30 at Training Step 120, then remains stable.
* **Sparse reward:** Starts at approximately 0.00 at Training Step 0, increases rapidly to approximately 0.20 at Training Step 60, then continues to increase more slowly to approximately 0.25 at Training Step 150.
* **Action SFT:** Remains relatively constant at approximately 0.15 across all Training Steps.
**Llama3.1-8B Chart:**
* **Dense reward:** Starts at approximately 0.03 at Training Step 0, increases rapidly to approximately 0.30 at Training Step 60, then fluctuates between approximately 0.28 and 0.32 until Training Step 150.
* **Sparse reward:** Starts at approximately 0.00 at Training Step 0, increases rapidly to approximately 0.22 at Training Step 60, then increases more slowly to approximately 0.27 at Training Step 150.
* **Action SFT:** Remains relatively constant at approximately 0.15 across all Training Steps.
### Key Observations
* The "Dense reward" consistently outperforms the "Sparse reward" across all three models.
* The "Action SFT" reward scheme shows minimal improvement with increasing Training Steps, remaining consistently low.
* Qwen2.5-7B and Llama3.1-8B achieve higher overall accuracy compared to Qwen2.5-3B.
* Llama3.1-8B exhibits more fluctuation in the "Dense reward" line after reaching peak accuracy, suggesting potential instability or overfitting.
### Interpretation
The data suggests that the "Dense reward" scheme is the most effective for training these language models on the Licheness Puzzle task. The higher accuracy achieved by Qwen2.5-7B and Llama3.1-8B indicates that model size plays a significant role in performance. The consistent low performance of "Action SFT" suggests that this reward scheme is not well-suited for this particular task or requires further tuning. The fluctuations observed in the Llama3.1-8B "Dense reward" line could indicate the need for regularization techniques to prevent overfitting, or it could be inherent noise in the training process. The rapid initial increase in accuracy for all models suggests that the early stages of training are crucial for establishing a strong foundation. The plateauing of the "Sparse reward" lines indicates diminishing returns as training progresses, highlighting the benefits of more frequent and informative reward signals.
