# Technical Document Extraction: Attention Forward Speed Analysis
## Chart Title
**Attention forward speed, head dim 256 (H100 80GB SXM5)**
## Axes
- **X-axis**: Sequence length (categories: 512, 1k, 2k, 4k, 8k, 16k)
- **Y-axis**: Speed (TFLOPs/s)
## Legend
- **Location**: Top-right corner
- **Labels**:
- Orange: FlashAttention-2
- Red: cuDNN
- Purple: FlashAttention-3
## Data Points (by sequence length)
| Sequence Length | FlashAttention-2 (Orange) | cuDNN (Red) | FlashAttention-3 (Purple) |
|-----------------|---------------------------|-------------|---------------------------|
| 512 | 275 | 470 | 482 |
| 1k | 313 | 546 | 617 |
| 2k | 321 | 580 | 707 |
| 4k | 323 | 581 | 736 |
| 8k | 324 | 580 | 746 |
| 16k | 326 | 581 | 756 |
## Key Trends
1. **FlashAttention-3 (Purple)**:
- **Trend**: Steadily increases with sequence length.
- **Values**: 482 (512) → 756 (16k).
- **Performance**: Consistently highest across all sequence lengths.
2. **cuDNN (Red)**:
- **Trend**: Relatively flat with minor fluctuations.
- **Values**: 470 (512) → 581 (16k).
- **Performance**: Middle-tier performance, stable across scales.
3. **FlashAttention-2 (Orange)**:
- **Trend**: Gradual increase with sequence length.
- **Values**: 275 (512) → 326 (16k).
- **Performance**: Lowest speed but improves linearly.
## Component Isolation
- **Header**: Chart title and legend.
- **Main Chart**: Bar groups for each sequence length, color-coded by model.
- **Footer**: No additional text or annotations.
## Spatial Grounding
- **Legend Position**: Top-right (x: 0.85, y: 0.95 relative to chart bounds).
- **Bar Alignment**: Centered under respective sequence length labels.
## Validation
- All legend colors match bar colors exactly.
- Numerical values align with visual bar heights.
- Trends confirmed via slope analysis (e.g., FlashAttention-3’s upward trajectory).
