## Scatter Plot: TTFT_req vs TPS_req Performance Comparison ### Overview The scatter plot compares three key-value (KV) configuration types (BF16, INT4, INT4+BDR) across varying concurrency levels (c=8 to c=128). Metrics shown are Tail-to-Tail Finish Time (TTFT_req in seconds) versus Throughput (TPS_req in tokens/second). Data points are color-coded by KV config and shape-coded by concurrency level. ### Components/Axes - **X-axis**: TPS_req (tok/s) - Throughput in tokens per second, ranging from 10 to 70 - **Y-axis**: TTFT_req (s) - Tail-to-Tail Finish Time in seconds, ranging from 0 to 100 - **Legend**: - **KV Configs**: - BF16 (blue circles) - INT4 (orange squares) - INT4+BDR (green diamonds) - **Concurrency Levels**: - c=8 (circles) - c=16 (squares) - c=32 (triangles) - c=64 (diamonds) - c=128 (inverted triangles) ### Detailed Analysis 1. **BF16 (Blue)**: - Highest TTFT at low TPS: 100s at 30 tok/s (c=128, inverted triangle) - Lowest TTFT at high TPS: ~0s at 60-70 tok/s (c=8, circle) - Trend: TTFT decreases as TPS increases, with concurrency having minimal impact at high TPS 2. **INT4 (Orange)**: - TTFT ranges from 18s (15 tok/s, c=8) to 60s (15 tok/s, c=128) - TTFT increases with concurrency at fixed TPS - TPS remains relatively stable (15-30 tok/s) across concurrency levels 3. **INT4+BDR (Green)**: - TTFT ranges from 15s (15 tok/s, c=8) to 60s (15 tok/s, c=128) - Similar trend to INT4 but consistently 3-5s lower TTFT at same TPS - Best performance at high TPS: ~0s at 65-70 tok/s (c=8) ### Key Observations - **Concurrency Trade-off**: Higher concurrency (c=128) achieves higher TPS but with significantly increased TTFT - **INT4+BDR Advantage**: Outperforms INT4 by 3-5s across all concurrency levels at same TPS - **BF16 Scalability**: Shows inverse relationship between TTFT and TPS, suggesting better load handling at high throughput - **Efficiency Threshold**: All configs achieve near-zero TTFT above 60 tok/s with c=8 concurrency ### Interpretation The data demonstrates a clear performance trade-off between throughput and latency. INT4+BDR consistently provides the best balance, maintaining low TTFT while achieving competitive TPS. BF16 excels at high-throughput scenarios but struggles with latency at lower loads. The concurrency scaling pattern suggests diminishing returns beyond c=32 for TTFT improvement, while TPS gains plateau at c=64. This indicates INT4+BDR with c=8-16 concurrency would be optimal for most workloads balancing latency and throughput requirements.