## Bar Chart: KV int4 + fused Hxint4 throughput vs T (H100, D=128, H=8) ### Overview The chart compares throughput (GB/s) across different token lengths (T) for five configurations: "int4 (plain)" (gray), "order 16" (red), "order 32" (orange), "order 64" (light orange), and "order 128" (yellow). Token lengths range from 32 to 16K, with throughput values plotted on a logarithmic y-axis (10¹ to 10⁴ GB/s). --- ### Components/Axes - **X-axis (Token length T)**: 32, 256, 1K, 4K, 8K, 16K (logarithmic spacing). - **Y-axis (Throughput)**: Logarithmic scale (10¹ to 10⁴ GB/s). - **Legend**: - Gray: "int4 (plain)" - Red: "order 16" - Orange: "order 32" - Light orange: "order 64" - Yellow: "order 128" - **Bar Groups**: Each token length has five bars (one per configuration), positioned side-by-side. --- ### Detailed Analysis #### Token Length = 32 - **int4 (plain)**: 9 GB/s (gray) - **order 16**: 7 GB/s (red) - **order 32**: 7 GB/s (orange) - **order 64**: 7 GB/s (light orange) - **order 128**: 7 GB/s (yellow) #### Token Length = 256 - **int4 (plain)**: 73 GB/s (gray) - **order 16**: 50 GB/s (red) - **order 32**: 54 GB/s (orange) - **order 64**: 55 GB/s (light orange) - **order 128**: 52 GB/s (yellow) #### Token Length = 1K - **int4 (plain)**: 270 GB/s (gray) - **order 16**: 208 GB/s (red) - **order 32**: 209 GB/s (orange) - **order 64**: 219 GB/s (light orange) - **order 128**: 219 GB/s (yellow) #### Token Length = 4K - **int4 (plain)**: 882 GB/s (gray) - **order 16**: 694 GB/s (red) - **order 32**: 710 GB/s (orange) - **order 64**: 698 GB/s (light orange) - **order 128**: 662 GB/s (yellow) #### Token Length = 8K - **int4 (plain)**: 1,085 GB/s (gray) - **order 16**: 898 GB/s (red) - **order 32**: 877 GB/s (orange) - **order 64**: 855 GB/s (light orange) - **order 128**: 834 GB/s (yellow) #### Token Length = 16K - **int4 (plain)**: 1,301 GB/s (gray) - **order 16**: 1,065 GB/s (red) - **order 32**: 1,035 GB/s (orange) - **order 64**: 1,015 GB/s (light orange) - **order 128**: 980 GB/s (yellow) --- ### Key Observations 1. **int4 (plain) dominates**: The gray bars (int4 plain) consistently show the highest throughput across all token lengths, with values up to 1,301 GB/s at 16K tokens. 2. **Order 128 underperforms**: The yellow bars (order 128) have the lowest throughput, particularly at larger token lengths (e.g., 980 GB/s at 16K vs. 1,301 GB/s for int4). 3. **Throughput scaling**: All configurations show exponential growth with token length, but int4 scales most effectively. 4. **Order 16 vs. order 32**: At 256 tokens, order 16 (50 GB/s) underperforms order 32 (54 GB/s), but this reverses at 1K tokens (208 vs. 209 GB/s). --- ### Interpretation - **Efficiency of int4**: The "int4 (plain)" configuration likely avoids overhead from fused operations, making it the most efficient. Fused Hxint4 (other orders) introduces diminishing returns as token length increases. - **Order 128 bottleneck**: The yellow bars (order 128) suggest that higher-order configurations may face resource contention or algorithmic inefficiencies, reducing throughput despite increased token length. - **Logarithmic scaling**: The y-axis emphasizes exponential growth, particularly in the 16K range, where int4 reaches ~1.3k GB/s, while order 128 plateaus near 1k GB/s. - **Anomalies**: The slight dip in order 16 performance at 256 tokens (50 GB/s) compared to order 32 (54 GB/s) may indicate configuration-specific optimizations or trade-offs. --- ### Spatial Grounding - **Legend**: Top-left corner, aligned with bar colors. - **X-axis labels**: Centered below each token length group. - **Y-axis**: Left-aligned, with logarithmic ticks (10¹, 10², ..., 10⁴). - **Bar placement**: Each token length group has five bars, ordered left-to-right per legend (gray → yellow). --- ### Content Details - **Values**: Extracted from bar heights, with approximate uncertainty (±5–10% based on visual estimation). - **Trends**: All configurations show increasing throughput with token length, but int4 maintains a consistent lead. - **Color consistency**: Legend colors match bar colors exactly (e.g., red = order 16). --- ### Final Notes The chart highlights the trade-off between configuration complexity (order) and throughput efficiency. While fused Hxint4 configurations (orders 16–128) improve with token length, they cannot match the plain int4 setup, suggesting that simplicity in design may be critical for high-throughput systems.