## Bar Chart: Latency Comparison for MiniKV and KIVI at Different Prompt Lengths ### Overview The chart compares latency (in milliseconds) for five components of two systems (MiniKV and KIVI) across two prompt lengths: 1024 and 40960. Latency is segmented by component type, with a legend indicating color coding. The x-axis categorizes data by system and prompt length, while the y-axis represents latency values from 0 to 80 ms. ### Components/Axes - **X-Axis**: - Categories: "MiniKV" and "KIVI" under two prompt lengths: - "Prompt Length = 1024" (left group) - "Prompt Length = 40960" (right group) - **Y-Axis**: - Label: "Latency (ms)" with a scale from 0 to 80. - **Legend** (right-aligned): - Colors and labels: - Orange: Decode QKV Projection - Blue: Decode Attention - Green: Decode Eviction - Purple: Decode Output Projection - Red: MLP ### Detailed Analysis #### Prompt Length = 1024 - **MiniKV**: - Decode QKV Projection: ~12 ms (orange) - Decode Attention: ~22 ms (blue) - Decode Eviction: ~1 ms (green) - Decode Output Projection: ~3 ms (purple) - MLP: ~7 ms (red) - **Total**: ~45 ms - **KIVI**: - Decode QKV Projection: ~10 ms (orange) - Decode Attention: ~20 ms (blue) - Decode Eviction: ~1 ms (green) - Decode Output Projection: ~3 ms (purple) - MLP: ~6 ms (red) - **Total**: ~40 ms #### Prompt Length = 40960 - **MiniKV**: - Decode QKV Projection: ~15 ms (orange) - Decode Attention: ~30 ms (blue) - Decode Eviction: ~2 ms (green) - Decode Output Projection: ~4 ms (purple) - MLP: ~8 ms (red) - **Total**: ~59 ms - **KIVI**: - Decode QKV Projection: ~14 ms (orange) - Decode Attention: ~50 ms (blue) - Decode Eviction: ~2 ms (green) - Decode Output Projection: ~4 ms (purple) - MLP: ~9 ms (red) - **Total**: ~80 ms ### Key Observations 1. **Latency Scaling**: Latency increases significantly with longer prompt lengths (e.g., MiniKV increases from ~45 ms to ~59 ms; KIVI from ~40 ms to ~80 ms). 2. **Component Contribution**: - **Decode Attention** dominates latency in both systems and prompt lengths (e.g., ~50 ms for KIVI at 40960). - **MLP** contributes ~10-15% of total latency in longer prompts. - **Decode Eviction** has minimal impact (<5% of total latency). 3. **System Efficiency**: KIVI consistently shows lower latency than MiniKV for the same prompt length (e.g., ~40 ms vs. ~45 ms at 1024). ### Interpretation - **Prompt Length Impact**: Longer prompts disproportionately increase latency, particularly for "Decode Attention," suggesting it scales poorly with input size. - **System Design**: KIVI’s architecture appears more efficient, likely due to optimizations in attention or eviction handling. - **Bottlenecks**: The MLP component becomes a larger relative contributor at longer prompts, indicating potential optimization opportunities. - **Eviction Efficiency**: The near-constant eviction latency across prompt lengths suggests it is not a major performance limiter in this context. ### Spatial Grounding - Legend is positioned on the right, with colors matching bar segments. - Bars are grouped by prompt length, with system-specific bars clustered under each prompt length.