## Bar Chart: Time Proportion by Batch Size and Model
### Overview
The chart visualizes the time proportion (%) allocated to different computational components (Attention mechanism, Matrix multiplication, CPU time, Other) across varying average batch sizes (1 to 512 requests) for four models: opt-1.3b, opt-2.7b, llama-2-7b, and llama-2-13b. Each bar is segmented into four colored regions corresponding to the components.
### Components/Axes
- **X-axis**: Average batch size (reqs) with values: 1, 2, 4, 8, 16, 32, 64, 128, 256, 512.
- **Y-axis**: Time proportion (%) from 0 to 100.
- **Legend**:
- Blue: Attention mechanism
- Green: Matrix multiplication
- Yellow: CPU time
- Orange: Other
- **Model Labels**:
- opt-1.3b (solid lines)
- opt-2.7b (diagonal lines)
- llama-2-7b (solid squares)
- llama-2-13b (diagonal squares)
### Detailed Analysis
1. **opt-1.3b (solid lines)**:
- At batch size 1: Attention (10%), Matrix multiplication (30%), CPU time (20%), Other (40%).
- At batch size 512: Attention (45%), Matrix multiplication (5%), CPU time (5%), Other (45%).
2. **opt-2.7b (diagonal lines)**:
- At batch size 1: Attention (15%), Matrix multiplication (25%), CPU time (15%), Other (45%).
- At batch size 512: Attention (50%), Matrix multiplication (3%), CPU time (2%), Other (45%).
3. **llama-2-7b (solid squares)**:
- At batch size 1: Attention (20%), Matrix multiplication (20%), CPU time (10%), Other (50%).
- At batch size 512: Attention (55%), Matrix multiplication (2%), CPU time (3%), Other (40%).
4. **llama-2-13b (diagonal squares)**:
- At batch size 1: Attention (25%), Matrix multiplication (15%), CPU time (5%), Other (55%).
- At batch size 512: Attention (60%), Matrix multiplication (1%), CPU time (4%), Other (35%).
### Key Observations
- **Attention mechanism** consistently increases with batch size across all models, reaching 45–60% at 512 requests.
- **Matrix multiplication** decreases sharply in larger models (e.g., llama-2-13b drops from 15% to 1% at 512).
- **CPU time** is most significant in smaller batches (10–15% in opt-1.3b/opt-2.7b) but negligible in larger batches (<5%).
- **Other** components remain relatively stable (35–55%) but vary by model.
### Interpretation
- **Batch Size Impact**: Larger batches amplify the dominance of the Attention mechanism, likely due to parallel processing efficiency. Matrix multiplication’s decline suggests optimization or reduced reliance in larger-scale computations.
- **Model Differences**: llama-2-13b allocates more time to Attention (60% at 512) compared to smaller models, indicating architectural differences in handling batch sizes.
- **CPU Time Reduction**: Fixed overhead costs (CPU time) become less impactful as batch size grows, highlighting scalability benefits.
- **Other Components**: The "Other" category’s persistence suggests unaccounted processes (e.g., I/O, memory management) remain relevant even at scale.
This data underscores the trade-offs in model design for batch processing, with Attention mechanisms becoming the primary computational focus as batch sizes increase.
