# Technical Document Extraction: Attention Key Query Score Graph
## Title
**Attention Key Query Score, a=24**
## Axes
- **X-axis (Horizontal):**
- Label: `Hidden Size`
- Range: `0` to `32768`
- Markers: `0`, `4096`, `8192`, `12288`, `16384`, `20480`, `24576`, `28672`, `32768`
- **Y-axis (Vertical):**
- Label: `Throughput (TFLOPs/s)`
- Range: `0` to `200`
- Markers: `0`, `50`, `100`, `150`, `200`
## Legend
- **Location:** Right side of the graph
- **Entries:**
- `h / a = 1` (Blue)
- `h / a = 2` (Orange)
- `h / a = 4` (Green)
- `h / a = 8` (Red)
- `h / a = 16` (Purple)
- `h / a = 32` (Brown)
- `h / a = 64` (Pink)
## Data Series & Trends
1. **Blue Line (`h / a = 1`):**
- **Trend:** Gradual upward slope from `0` to `~90 TFLOPs/s` across the entire `Hidden Size` range.
- **Key Points:**
- At `Hidden Size = 0`: `~0 TFLOPs/s`
- At `Hidden Size = 32768`: `~90 TFLOPs/s`
2. **Orange Line (`h / a = 2`):**
- **Trend:** Steeper initial rise than blue, plateauing at `~150 TFLOPs/s`.
- **Key Points:**
- At `Hidden Size = 0`: `~0 TFLOPs/s`
- At `Hidden Size = 32768`: `~150 TFLOPs/s`
3. **Green Line (`h / a = 4`):**
- **Trend:** Smooth upward trajectory, reaching `~150 TFLOPs/s` by `Hidden Size = 32768`.
- **Key Points:**
- At `Hidden Size = 0`: `~0 TFLOPs/s`
- At `Hidden Size = 32768`: `~150 TFLOPs/s`
4. **Red Line (`h / a = 8`):**
- **Trend:** Sharp initial spike, peaking at `~220 TFLOPs/s` near `Hidden Size = 8192`, then stabilizing.
- **Key Points:**
- At `Hidden Size = 0`: `~0 TFLOPs/s`
- At `Hidden Size = 8192`: `~220 TFLOPs/s`
- At `Hidden Size = 32768`: `~200 TFLOPs/s`
5. **Purple Line (`h / a = 16`):**
- **Trend:** Rapid rise to `~220 TFLOPs/s` by `Hidden Size = 8192`, followed by minor fluctuations.
- **Key Points:**
- At `Hidden Size = 0`: `~0 TFLOPs/s`
- At `Hidden Size = 8192`: `~220 TFLOPs/s`
- At `Hidden Size = 32768`: `~210 TFLOPs/s`
6. **Brown Line (`h / a = 32`):**
- **Trend:** Similar to purple but with sharper fluctuations. Peaks at `~230 TFLOPs/s` near `Hidden Size = 24576`.
- **Key Points:**
- At `Hidden Size = 0`: `~0 TFLOPs/s`
- At `Hidden Size = 24576`: `~230 TFLOPs/s`
- At `Hidden Size = 32768`: `~215 TFLOPs/s`
7. **Pink Line (`h / a = 64`):**
- **Trend:** Highest performance, peaking at `~230 TFLOPs/s` near `Hidden Size = 24576`, then stabilizing.
- **Key Points:**
- At `Hidden Size = 0`: `~0 TFLOPs/s`
- At `Hidden Size = 24576`: `~230 TFLOPs/s`
- At `Hidden Size = 32768`: `~220 TFLOPs/s`
## Observations
- **Performance Correlation:** Higher `h / a` ratios (e.g., `64`) generally achieve higher throughput, especially at larger `Hidden Size` values.
- **Divergence:** Lines for `h / a = 8`, `16`, `32`, and `64` converge at higher `Hidden Size` values, suggesting diminishing returns beyond a certain point.
- **Stability:** All lines stabilize near `Hidden Size = 32768`, indicating saturation of throughput gains.
## Notes
- No non-English text or additional data tables are present.
- All legend colors match their corresponding lines in the graph.
- Spatial grounding confirms the legend is positioned on the right, with no overlap or misalignment.
