## Scatter Plot: LongBench Accuracy vs KV Cache Size for HotpotQA Dataset ### Overview The image is a scatter plot comparing LongBench Accuracy (y-axis) against KV Cache Size (x-axis) for different knowledge vector (KV) cache implementations on the HotpotQA dataset. A Pareto Frontier curve is overlaid to highlight optimal trade-offs between cache size and accuracy. ### Components/Axes - **X-axis**: KV Cache Size (GB) ranging from 0.00 to 2.00 in 0.25 increments - **Y-axis**: LongBench Accuracy ranging from 22 to 26.5 in 0.5 increments - **Legend**: Located in top-right corner with five distinct markers: - Red squares: H₂O - Blue circles: KIVI - Green diamonds: SnapKV - Yellow triangles: MiniKV - Black crosses: Pareto Frontier ### Detailed Analysis 1. **H₂O (Red Squares)**: - Data points: (0.25, 25.0), (0.5, 25.3), (1.0, 26.0), (1.25, 26.2), (1.75, 26.5) - Trend: Strong positive correlation between cache size and accuracy 2. **KIVI (Blue Circle)**: - Single data point: (0.5, 23.8) - Position: Lower left quadrant, significantly below Pareto Frontier 3. **SnapKV (Green Diamonds)**: - Data points: (0.25, 24.9), (0.5, 25.1), (1.0, 25.3), (1.25, 25.4) - Trend: Gradual improvement with diminishing returns 4. **MiniKV (Yellow Triangles)**: - Data points: (0.1, 23.8), (0.25, 24.3), (0.5, 24.5), (0.75, 24.7) - Trend: Steady improvement but plateaus at lower accuracy levels 5. **Pareto Frontier (Black Crosses)**: - Connects optimal points: (0.25, 25.0) → (1.0, 26.0) → (1.75, 26.5) - Represents theoretical maximum efficiency frontier ### Key Observations - H₂O consistently outperforms other implementations across all cache sizes - Pareto Frontier shows optimal trade-off between cache size and accuracy - KIVI demonstrates poor performance relative to cache size investment - SnapKV and MiniKV show moderate performance with diminishing returns - No data points exist between 0.75GB and 1.0GB for MiniKV - All implementations show improved accuracy with increased cache size ### Interpretation The data suggests that H₂O provides the most efficient balance between cache size and accuracy, consistently operating near or on the Pareto Frontier. The Pareto Frontier curve reveals that beyond 1.75GB cache size, accuracy improvements become marginal. KIVI's single data point indicates either limited implementation or suboptimal performance characteristics. The diminishing returns observed across all implementations suggest that cache size alone cannot drive unbounded accuracy improvements, highlighting the need for architectural optimizations beyond simple cache scaling.