# Technical Document Extraction: Attention Key Query Score Chart
## Title
**Attention Key Query Score (h/a = 64)**
---
## Axes
- **X-axis (Horizontal):**
- Label: `Hidden Size`
- Range: `0` to `16384` (logarithmic scale)
- Tick Marks: `0`, `2048`, `4096`, `6144`, `8192`, `10240`, `12288`, `14336`, `16384`
- **Y-axis (Vertical):**
- Label: `Throughput (TFLOP/s)`
- Range: `50` to `225`
- Tick Marks: `50`, `75`, `100`, `125`, `150`, `175`, `200`, `225`
---
## Legend
- **Location:** Top-right corner
- **Color-Coded Labels:**
- `a: 12` → Blue
- `a: 24` → Orange
- `a: 32` → Green
- `a: 40` → Red
- `a: 64` → Purple
- `a: 80` → Brown
- `a: 96` → Pink
---
## Data Series & Trends
1. **Blue Line (a: 12)**
- **Trend:** Steep upward slope from `50 TFLOP/s` (hidden size 0) to `230 TFLOP/s` (hidden size 16384).
- **Key Points:**
- `2048`: ~90 TFLOP/s
- `4096`: ~190 TFLOP/s
- `8192`: ~220 TFLOP/s
- `16384`: ~230 TFLOP/s
2. **Orange Line (a: 24)**
- **Trend:** Rapid increase from `60 TFLOP/s` (hidden size 2048) to `210 TFLOP/s` (hidden size 16384).
- **Key Points:**
- `4096`: ~110 TFLOP/s
- `8192`: ~150 TFLOP/s
- `12288`: ~205 TFLOP/s
3. **Green Line (a: 32)**
- **Trend:** Gradual rise from `60 TFLOP/s` (hidden size 2048) to `200 TFLOP/s` (hidden size 16384).
- **Key Points:**
- `6144`: ~130 TFLOP/s
- `10240`: ~170 TFLOP/s
- `16384`: ~200 TFLOP/s
4. **Red Line (a: 40)**
- **Trend:** Moderate increase from `60 TFLOP/s` (hidden size 2048) to `190 TFLOP/s` (hidden size 16384).
- **Key Points:**
- `8192`: ~120 TFLOP/s
- `12288`: ~170 TFLOP/s
- `16384`: ~190 TFLOP/s
5. **Purple Line (a: 64)**
- **Trend:** Slow upward trajectory from `60 TFLOP/s` (hidden size 2048) to `150 TFLOP/s` (hidden size 16384).
- **Key Points:**
- `10240`: ~110 TFLOP/s
- `14336`: ~145 TFLOP/s
6. **Brown Line (a: 80)**
- **Trend:** Linear increase from `60 TFLOP/s` (hidden size 2048) to `135 TFLOP/s` (hidden size 16384).
- **Key Points:**
- `12288`: ~120 TFLOP/s
- `16384`: ~135 TFLOP/s
7. **Pink Line (a: 96)**
- **Trend:** Gentle slope from `60 TFLOP/s` (hidden size 2048) to `125 TFLOP/s` (hidden size 16384).
- **Key Points:**
- `10240`: ~100 TFLOP/s
- `16384`: ~125 TFLOP/s
---
## Spatial Grounding
- **Legend Position:** Top-right corner (outside the plot area).
- **Color Consistency Check:**
- All line colors match legend labels (e.g., blue = a:12, orange = a:24).
---
## Additional Observations
- **Shaded Regions:**
- Green (`0-1B`), Blue (`1B-10B`), and Pink (`10B-300B`) highlight hidden size ranges but do not directly correlate with data series.
- **h/a Ratio:** Constant value of `64` (title annotation).
---
## Summary
The chart illustrates the relationship between **hidden size** and **throughput (TFLOP/s)** for varying **attention key query scores (a)**. Higher `a` values (e.g., 96) yield lower throughput, while lower `a` values (e.g., 12) achieve higher throughput. Throughput increases non-linearly with hidden size, with steeper growth observed for smaller `a` values.
