## System Architecture Diagram: Distributed KV Cache Management
### Overview
The image illustrates a distributed system architecture designed for managing Key-Value (KV) caches, specifically tailored for a two-stage compute environment consisting of "Prefill" and "Decode" nodes. The architecture utilizes a tiered memory hierarchy (HBM, DRAM, SSD) within the nodes and a centralized management layer to coordinate data movement and storage, utilizing RDMA for high-speed data transfer.
### Components/Axes
The diagram is organized into three primary vertical regions:
**1. Left Region: Prefill Node**
* **Visual Representation:** A stack of three overlapping rectangles, indicating a cluster or multiple instances of a "Prefill Node."
* **Internal Structure:** A large white triangle is centered within the main green rectangle.
* **Text Content:**
* Header: "Prefill Node"
* Inside Triangle: "HBM", "DRAM", "SSD" (stacked vertically).
* Below Triangle: "Local KV Cache Hierarchy"
**2. Right Region: Decode Node**
* **Visual Representation:** A stack of three overlapping rectangles, indicating a cluster or multiple instances of a "Decode Node."
* **Internal Structure:** A large white triangle is centered within the main green rectangle.
* **Text Content:**
* Header: "Decode Node"
* Inside Triangle: "HBM", "DRAM", "SSD" (stacked vertically).
**3. Center Region: Control and Storage**
* **Top Component:** "Cache-Aware Router" (Tan/Orange box).
* **Middle Component:** "KV Cache Manager" (Tan/Orange box).
* **Bottom Component:** "Object Storage" (Light blue cylinder icon).
**4. Connections**
* **Control Path (Dotted Lines):**
* Connects the "KV Cache Manager" to the center of the "Prefill Node."
* Connects the "KV Cache Manager" to the center of the "Decode Node."
* Connects the "KV Cache Manager" to the "Object Storage."
* **Data Path (Solid Line):**
* A horizontal line connects the bottom of the "Prefill Node" and the "Decode Node" to the "Object Storage."
* **Label:** "RDMA" (Remote Direct Memory Access) is written below this line.
### Detailed Analysis
* **Memory Hierarchy:** The triangle within each node (HBM, DRAM, SSD) represents a tiered storage strategy. HBM (High Bandwidth Memory) is positioned at the top of the triangle, implying it is the fastest but likely smallest capacity tier. DRAM is in the middle, and SSD is at the base, implying the largest capacity but slowest access speed.
* **Node Roles:** The separation into "Prefill Node" and "Decode Node" suggests a decoupled architecture where the initial prompt processing (Prefill) and the token generation (Decode) are handled by different compute resources, likely to optimize for the different computational characteristics of these two phases in LLM inference.
* **Management Layer:** The "Cache-Aware Router" and "KV Cache Manager" act as the control plane. The router likely directs incoming requests, while the manager handles the state and location of the KV cache data across the distributed nodes.
* **Data Transfer:** The use of "RDMA" for the connection between the nodes and the "Object Storage" indicates a requirement for low-latency, high-throughput data movement, bypassing traditional network stack overheads to fetch or offload cache data.
### Key Observations
* **Symmetry:** The architecture is highly symmetric regarding the compute nodes, suggesting that the "Prefill" and "Decode" nodes share the same underlying hardware/memory architecture.
* **Control vs. Data Plane:** There is a clear architectural distinction between the control plane (dotted lines, managed by the KV Cache Manager) and the data plane (solid line, utilizing RDMA for direct storage access).
* **Hierarchy Placement:** The text "Local KV Cache Hierarchy" is explicitly placed under the Prefill Node. While the Decode Node has an identical internal structure, the label is not repeated, suggesting the hierarchy concept applies to the entire node architecture.
### Interpretation
This diagram represents a sophisticated infrastructure for **Large Language Model (LLM) serving**.
* **The Problem:** LLM inference requires massive amounts of memory to store the "KV Cache" (the context of the conversation). This cache often exceeds the capacity of a single GPU's HBM.
* **The Solution:** This architecture solves the memory bottleneck by:
1. **Tiering:** Using a hierarchy (HBM/DRAM/SSD) to store the cache, keeping the most active data in HBM and spilling less active data to DRAM or SSD.
2. **Decoupling:** Separating Prefill and Decode nodes allows for independent scaling of the two phases of inference, which have different compute/memory profiles.
3. **Orchestration:** The "KV Cache Manager" acts as a global controller, ensuring that when a node needs specific cache data, it is fetched efficiently (via RDMA) from the Object Storage or moved between tiers, preventing the system from stalling due to memory limitations.
This is a classic "disaggregated memory" or "remote memory" architecture designed to maximize the throughput of LLM inference clusters.
