## Line Charts: LLaMA-7b CQ Configurations on WikiText-2
### Overview
The image contains four line charts comparing the performance of different CQ (Contextual Quantization) configurations on the LLaMA-7b model using WikiText-2. Each chart evaluates metrics like **perplexity** and **quantization error** across varying CQ configurations (1c1b, 2c2b, 4c4b, 8c8b). The charts use logarithmic and linear scales to visualize trends.
---
### Components/Axes
1. **X-Axes (CQ Config.)**:
- Labels: `1c1b`, `2c2b`, `4c4b`, `8c8b` (context window size and quantization bits).
2. **Y-Axes**:
- **Left Y-Axis (Perplexity on WikiText-2)**: Logarithmic scale (10¹ to 10³).
- **Right Y-Axes (Quant. Error on WikiText-2)**: Linear scale (500 to 2000).
3. **Legends**:
- **Uniform CQ**: Orange lines/triangles.
- **Fisher-guided CQ**: Green lines/triangles.
- **FP16 Baseline**: Gray dashed line (constant value).
---
### Detailed Analysis
#### Chart 1: LLaMA-7b 1-bit CQ (Perplexity)
- **Trend**:
- Uniform CQ (orange) slopes steeply downward from **2642.72** (1c1b) to **32.12** (8c8b).
- Fisher-guided CQ (green) slopes less steeply from **620.08** (1c1b) to **8.09** (8c8b).
- FP16 Baseline (gray) remains constant at **5.68**.
- **Data Points**:
- Uniform CQ: `1c1b` (2642.72), `2c2b` (1614.02), `4c4b` (883.60), `8c8b` (32.12).
- Fisher-guided CQ: `1c1b` (620.08), `2c2b` (28.39), `4c4b` (10.47), `8c8b` (8.09).
#### Chart 2: LLaMA-7b 2-bit CQ (Perplexity)
- **Trend**:
- Uniform CQ (orange) slopes downward from **204.56** (1c1b) to **6.86** (8c8b).
- Fisher-guided CQ (green) slopes slightly from **6.37** (1c1b) to **5.97** (8c8b).
- FP16 Baseline (gray) remains at **5.68**.
- **Data Points**:
- Uniform CQ: `1c1b` (204.56), `2c2b` (24.46), `4c4b` (6.86), `8c8b` (5.97).
- Fisher-guided CQ: `1c1b` (6.37), `2c2b` (6.08), `4c4b` (6.08), `8c8b` (5.97).
#### Chart 3: LLaMA-7b 1-bit CQ (Quant. Error)
- **Trend**:
- Uniform CQ Key (orange) slopes from **1750** (1c1b) to **750** (8c8b).
- Fisher-guided CQ Key (green) slopes from **1750** (1c1b) to **600** (8c8b).
- Uniform CQ Value (orange triangle) slopes from **750** (1c1b) to **500** (8c8b).
- Fisher-guided CQ Value (green triangle) slopes from **750** (1c1b) to **500** (8c8b).
- **Data Points**:
- Uniform CQ Key: `1c1b` (1750), `2c2b` (1500), `4c4b` (1250), `8c8b` (750).
- Fisher-guided CQ Key: `1c1b` (1750), `2c2b` (1500), `4c4b` (1250), `8c8b` (600).
- Uniform CQ Value: `1c1b` (750), `2c2b` (600), `4c4b` (500), `8c8b` (500).
- Fisher-guided CQ Value: `1c1b` (750), `2c2b` (600), `4c4b` (500), `8c8b` (500).
#### Chart 4: LLaMA-7b 2-bit CQ (Quant. Error)
- **Trend**:
- Uniform CQ Key (orange) slopes from **1000** (1c1b) to **600** (8c8b).
- Fisher-guided CQ Key (green) slopes from **1000** (1c1b) to **600** (8c8b).
- Uniform CQ Value (orange triangle) slopes from **800** (1c1b) to **400** (8c8b).
- Fisher-guided CQ Value (green triangle) slopes from **800** (1c1b) to **400** (8c8b).
- **Data Points**:
- Uniform CQ Key: `1c1b` (1000), `2c2b` (800), `4c4b` (600), `8c8b` (600).
- Fisher-guided CQ Key: `1c1b` (1000), `2c2b` (800), `4c4b` (600), `8c8b` (600).
- Uniform CQ Value: `1c1b` (800), `2c2b` (600), `4c4b` (400), `8c8b` (400).
- Fisher-guided CQ Value: `1c1b` (800), `2c2b` (600), `4c4b` (400), `8c8b` (400).
---
### Key Observations
1. **Perplexity Reduction**:
- Both CQ methods reduce perplexity as CQ configuration increases (e.g., 1c1b → 8c8b).
- Fisher-guided CQ consistently outperforms Uniform CQ (e.g., 620.08 vs. 2642.72 in 1c1b for perplexity).
2. **Quantization Error**:
- Fisher-guided CQ reduces error more effectively than Uniform CQ (e.g., 600 vs. 750 in 8c8b for 1-bit CQ).
- 2-bit CQ configurations show smaller error reductions compared to 1-bit.
3. **FP16 Baseline**:
- Acts as a lower bound for perplexity (5.68) but does not affect quantization error.
---
### Interpretation
- **Fisher-guided CQ** demonstrates superior performance in both perplexity and quantization error across all configurations, suggesting it better preserves model quality during quantization.
- **Uniform CQ** shows significant improvement with larger CQ configurations but remains less efficient than Fisher-guided CQ.
- The **FP16 Baseline** highlights the trade-off between quantization and model fidelity, as CQ methods achieve lower perplexity than FP16 while maintaining competitive quantization error.
- The logarithmic scale for perplexity emphasizes the exponential reduction in error with increased CQ bits, critical for low-resource deployment scenarios.
