## Diagram: Data-Driven Controller for Adaptive KV-Cache Quantization in On-Device LLMs ### Overview The diagram illustrates a system for optimizing large language model (LLM) inference on edge hardware through adaptive quantization. It contrasts static quantization methods with a learned controller approach, highlighting performance benefits in accuracy and latency. ### Components/Axes 1. **Problem Section (Left)** - **KV Cache Memory Consumption**: Bar chart with three bars (blue, green, orange) showing increasing memory usage. - **Static Quantization degrades reasoning**: Bar chart with three bars (blue, green, orange) showing increasing reasoning degradation. - Text labels: "KV Cache memory consumption", "Static Quantization degrades reasoning". 2. **Solution Section (Center)** - **Edge Hardware (LLM Hardware)**: Hexagon icon with "EDGE HARDWARE" label. - **Token Signals**: Arrows feeding into a "Lightweight Network". - **Learned Controller (MLP)**: Central processing unit with inputs/outputs: - Inputs: Token Frequency, Quality Score, Attention Variance, Model Uncertainty (Entropy). - Outputs: Precision Policy (2-bit, 4-bit, 8-bit, FP16). - **Precision Policy**: Color gradient (blue to red) indicating token complexity. - **LLM Inference**: Final output with a gear icon. 3. **Benefit Section (Right)** - **Accuracy Comparison**: Bar chart comparing methods (Static 2-bit KV, 4-bit KV, 8-bit KV, Our Method, Rule-Based FP, 16-bit FP). "Our Method" (orange) shows highest accuracy. - **Latency Comparison**: Bar chart with same methods. "Our Method" (orange) shows lowest latency. - Text labels: "Accuracy Comparison", "Latency Comparison". ### Detailed Analysis - **Problem Section**: - KV Cache memory consumption increases with static quantization (blue < green < orange). - Static quantization degrades reasoning performance (blue < green < orange). - **Solution Section**: - Token signals (frequency, quality, attention, uncertainty) are processed by a lightweight network. - The learned controller (MLP) dynamically selects precision levels (2-16 bits) based on token complexity. - High precision (orange/red) is applied to critical tokens for accuracy, while lower precision (blue/green) reduces memory/latency. - **Benefit Section**: - **Accuracy**: "Our Method" outperforms all static methods and rule-based FP16 (orange bar highest). - **Latency**: "Our Method" achieves lowest latency (orange bar shortest). ### Key Observations 1. Static quantization methods show linear increases in memory usage and reasoning degradation. 2. The learned controller adapts precision dynamically, balancing accuracy and efficiency. 3. "Our Method" consistently outperforms static approaches in both accuracy and latency. ### Interpretation The diagram demonstrates that adaptive quantization via a learned controller significantly improves LLM inference on edge devices. By dynamically adjusting precision based on token complexity, the system maintains high accuracy for critical tokens while reducing memory and latency compared to static methods. This addresses the key bottlenecks of KV cache memory consumption and reasoning degradation in static quantization approaches. The orange-highlighted "Our Method" represents the optimal trade-off between computational efficiency and output quality, making it suitable for resource-constrained edge deployments.