2604.06370v1

Model: nemotron-free

# ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache **Authors**: Shao Wang, Rui Ren, Lin Gui > 0009-0000-9816-2582 Shanghai Jiao Tong University Shanghai China > Shanghai Jiao Tong University Shanghai China (2018) ## Abstract The serving paradigm of large language models (LLMs) is rapidly shifting towards complex multi-agent workflows where specialized agents collaborate over massive shared contexts. While Low-Rank Adaptation (LoRA) enables the efficient co-hosting of these specialized agents on a single base model, it introduces a critical memory footprint bottleneck during serving. Specifically, unique LoRA activations cause Key-Value (KV) cache divergence across agents, rendering traditional prefix caching ineffective for shared contexts. This forces redundant KV cache maintenance, rapidly saturating GPU capacity and degrading throughput. To address this challenge, we introduce ForkKV, a serving system for multi-LoRA agent workflows centered around a novel memory management paradigm in OS: fork with copy-on-write (CoW). By exploiting the structural properties of LoRA, ForkKV physically decouples the KV cache into a massive shared component (analogous to the parent process’s memory pages) and lightweight agent-specific components (the child process’s pages). To support this mechanism, we propose a DualRadixTree architecture that allows newly forked agents to inherit the massive shared cache and apply CoW semantics for their lightweight unique cache. Furthermore, to guarantee efficient execution, we design ResidualAttention, a specialized kernel that reconstructs the disaggregated KV cache directly within on-chip SRAM. Comprehensive evaluations across diverse language models and practical datasets of different tasks demonstrate that ForkKV achieves up to 3.0x the throughput of state-of-the-art multi-LoRA serving systems with a negligible impact on generation quality. Large Language Models, LoRA, Multi-Agent Systems, KV Cache copyright: acmlicensed journalyear: 2018 doi: XXXXXXX.XXXXXXX conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY isbn: 978-1-4503-XXXX-X/2018/06 ccs: Information systems Data management systems ccs: Computing methodologies Distributed computing methodologies ccs: Computing methodologies Machine learning ## 1. Introduction The serving paradigm of Large Language Models (LLMs) has rapidly evolved from simple chatbots to complex autonomous agentic workflows, such as coding assistants equipped with advanced reasoning, planning, and tool-call capabilities (Team, 2025; Anthropic, 2026; OpenAI, 2026). These workflows operate as collaborative pipelines of specialized LLM nodes, termed agents, each dedicated to different subtasks. A defining characteristic of these multi-agent workloads is their context structure. Agents typically share a massive static context, often dominated by a lengthy prefix like extensive system prompts or a large codebase (Zheng et al., 2024; Yang et al., 2024; Liu et al., 2023). From this shared prefix, agents fork their own distinct context from dynamic outputs, such as reasoning steps from previous agents and historically acquired tool observations. For example, to resolve a Github issue, a coding assistant uses codebase as its shared static context and sequentially triggers navigation, generation, and testing agents, where each agent builds their own context based on previous agents. However, successfully executing such diverse subtasks requires distinct agent capabilities. A single monolithic model often lacks the flexibility to handle every stage optimally, necessitating fine-tuning the base model with task-specific datasets to serve specialized agents effectively (Zeng et al., 2024; Chen et al., 2023; Schick et al., 2023; Patil et al., 2024). To tailor foundational models for these diverse tasks in a workflow, Parameter-Efficient Fine-Tuning (PEFT) (Mangrulkar et al., 2022) techniques, particularly Low-Rank Adaptation (LoRA) (Hu et al., 2022), offer a promising solution. By freezing the pretrained weights and updating only small low-rank matrices known as adapters, LoRA maintains high generation quality while introducing minimal parameter overhead (Hu et al., 2022; Dettmers et al., 2023; Liu et al., 2024a). For example, LoRA adapters with a low-rank dimension of 16 trained on Llama3.1-70B (meta llama, 2024) account for only 0.28% of the original model size (approximately 400MB v.s. 140GB). Such extreme memory efficiency fundamentally shifts how systems manage concurrent workloads. Instead of deploying multiple monolithic models, the serving engine can concurrently host multiple specialized agents on a single shared base model by dynamically swapping these lightweight adapters. Many modern agentic applications have already adopted this efficient architectural design (Liu et al., 2024b; Yu et al., 2024; Liu et al., 2025b; Kadekodi et al., 2025), which we refer to as multi-LoRA agent serving in this paper. While LoRA successfully minimizes the memory overhead of model weights, multi-agent serving still suffers from a critical memory footprint bottleneck caused by the duplicated unshareable Key-Value states (i.e., KV cache) across agents. In traditional monolithic deployment, serving engines avoid memory redundancy through prefix caching (Zheng et al., 2024; Google, 2025; OpenAI, 2025; Anthropic, 2025). This technique allows different requests to share the KV cache of common prefixes. However, this optimization breaks down in multi-LoRA scenarios. The unique activations produced by each adapter cause the KV cache to diverge across different agents. Consequently, the system is forced to maintain an independent KV cache for each agent even when they process the exact same context, incurring severe memory footprint overhead. As shown in Figure 1, the overall memory consumption (represented by the deep blue line) scales linearly with the number of agents, rapidly saturating GPU capacity. Specifically, in sequential workflows like ReAct (Yao et al., 2022), context reuse fails entirely when the pipeline switches to a new LoRA agent. In parallel workflows like MapReduce (Luo et al., 2025), broadcasting the shared input creates redundant cache copies. Complex agent workflows exacerbate this redundancy, which could lead to severe performance degradation in both latency and throughput (Yao et al., 2025a; Wu et al., 2024a; Zhong et al., 2024; Agrawal et al., 2023). <details> <summary>2604.06370v1/x1.png Details</summary> ![e67189ab](/v1/image/e67189abcee0081f25a1cfa06fa1137c41af9d2f22e849ec4b77610376849987) ### Visual Description ## Line Chart: Context Memory Usage Comparison Between Existing Systems and ForkKV ### Overview The chart compares memory usage (in GB) for two systems—Existing Systems and ForkKV—as the number of concurrent agents increases. It highlights ForkKV's ability to handle 32x more agents within the same GPU capacity while maintaining lower memory usage. ### Components/Axes - **X-axis**: Agents (#NUM) with markers at 0, 16, 32, 48, 64. - **Y-axis**: Context Memory Usage (GB) with markers at 0, 8, 16. - **Legend**: - Blue line: Existing Systems - Teal line: ForkKV - **Annotations**: - "Run 32x concurrent agents within same GPU capacity" (pointing to ForkKV's line at 32 agents). - "Base cache is computed once and shared across agents" (pointing to ForkKV's line at 0 agents). ### Detailed Analysis - **Existing Systems (Blue Line)**: - Starts at **8 GB** for 0 agents. - Rises sharply to **16 GB** at 16 agents. - Remains flat at 16 GB for all agent counts beyond 16. - **ForkKV (Teal Line)**: - Starts at **0 GB** for 0 agents. - Increases linearly to **8 GB** at 64 agents. - Intermediate points: 4 GB at 16 agents, 6 GB at 32 agents, 7 GB at 48 agents. ### Key Observations 1. **Memory Efficiency**: ForkKV uses significantly less memory than Existing Systems across all agent counts. 2. **Scalability**: ForkKV scales linearly with agents (0–8 GB for 0–64 agents), while Existing Systems plateau at 16 GB after 16 agents. 3. **Base Cache Optimization**: The annotation suggests ForkKV avoids redundant cache computations, contributing to lower memory usage. ### Interpretation The data demonstrates ForkKV's superior memory efficiency and scalability compared to Existing Systems. While Existing Systems experience a steep memory increase (8 GB → 16 GB) when adding 16 agents, ForkKV maintains sub-8 GB usage even at 64 agents. This implies ForkKV is better suited for high-concurrency workloads, as it avoids memory saturation and leverages shared caching. The linear scaling of ForkKV suggests predictable resource allocation, whereas Existing Systems may require GPU upgrades or optimizations to handle increased agents without excessive memory overhead. </details> Figure 1. Context memory usage when serving agents with 32K shared contexts on Llama3-8B, where agents are based on different LoRA adapters and the rank is 16. To overcome this memory footprint bottleneck, our key insight is that we can partially share the KV cache across agents by physically decoupling it based on the structural properties of LoRA. We term this partially shared memory layout disaggregated KV cache. The standard LoRA projection $xW+xA_iB_i$ intuitively suggests separating the KV cache into a massive base cache ( $bCache=xW$ ) and a lightweight adapter-specific residual cache ( $rCache=xA_i$ ) that can reconstruct full projection via $rCache× B_i$ . This decomposition exposes a significant size asymmetry, where bCache is typically dozens of times the size of rCache due to the low-rank projection of LoRA. Thus, instead of redundantly allocating complete KV cache for agents processing the same context, we globally share the massive $bCache$ across agents and only maintain lightweight rCache for each agent. This disaggregated design drastically cuts the per-agent memory footprint, effectively mitigating the memory footprint bottleneck. As validated in Figure 1, our approach enables an 8GB KV cache to support 32 $×$ more concurrent agents. Admittedly, sharing $bCache$ beyond the first layer is mathematically lossy because adapter-specific activations cause subsequent inputs $x$ to diverge across agents. However, this divergence is empirically bounded. The transformer architecture provides inherent robustness through its residual connections (Elhage et al., 2021), and the LoRA adapters preserve necessary task-specific interactions (Hu et al., 2022; Wang et al., 2023b). Consequently, this aggressive memory optimization maintains a similarity for input $x$ of over 99.4%, yielding a negligible degradation in generation quality of only 1.60% (see details in Figure 5). Supporting disaggregated KV cache presents several major challenges. First, the system needs to manage the distinct lifecycles of the shared bCache and unique rCache while maintaining the structural dependencies imposed by their mathematical decomposition. As multi-agent collaboration naturally forms multi-branch reasoning paths, tracking the resulting 1-to-N base-to-residual mappings introduces severe complexity. Second, computing attention scores requires reconstructing KV cache from its disaggregated components, but naive reconstruction in HBM incurs severe memory and computational overhead. Materializing the full-sized KV cache in HBM for every agent will completely negate the memory savings. Conversely, performing in-place updates on the shared bCache using rCache causes memory access conflicts across agents, forcing sequential execution and destroying intra-batch parallelism. To address the first management challenge, we introduce ForkKV, a multi-LoRA agent serving system featuring a novel disaggregated KV cache management mechanism inspired by the operating system (OS) primitive for subprocess creation: fork with copy-on-write (CoW). ForkKV manages the disaggregated KV cache with the same efficiency as an OS fork: the massive bCache acts as the shareable and read-only memory pages of a parent process, the lightweight rCache serves as the unique CoW footprint of a child process, and their dynamic combination represents the complete memory space. To orchestrate this dual-tiered layout in practice, ForkKV introduces a coordinated DualRadixTree architecture. When a new agent is launched, ForkKV performs a longest-prefix match to inherit the globally shared read-only bCache, forks the memory space by allocating memory exclusively for the agent’s unique rCache, and updates this dual-tree storage after generation. To tackle the second challenge, we propose fusing KV cache reconstruction directly into the attention kernel. By keeping all intermediate computations within the fast on-chip SRAM, the design eliminates both the extra HBM allocation and the serialized execution caused by conflicting memory access. Based on this insight, we implement ResidualAttention, an attention kernel specifically tailored for the disaggregated KV cache architecture. In the first step, the kernel streams bCache and rCache directly into SRAM in a block-wise manner and reconstructs Key cache. In the second step, ResidualAttention computes separate attention scores for the base and residual components and fuses the final output by leveraging matrix associativity. In this work, we implement ForkKV on top of SGLang (Zheng et al., 2024), a state-of-the-art LLM serving framework for production. To assess the system, we design end-to-end evaluations based on two typical agentic serving scenarios, ReAct and MapReduce. We conduct these experiments across a wide range of LLMs, including Llama3-8B (AI@Meta, 2024), Qwen2.5-7B (Team, 2024), and Qwen2.5-14B (Team, 2024). Across practical workloads such as LooGLE (Li et al., 2023), NarrativeQA (Kočiský et al., 2018), and APIGen (Liu et al., 2024d), ForkKV demonstrates significant performance gains over state-of-the-art multi-LoRA serving systems. Specifically, ForkKV achieves 1.25-3.04 $×$ the throughput on ReAct workflows, and 1.68-2.60 $×$ the throughput on MapReduce workflows, with a negligible quality degradation of only 0.71% on average measured by F1-Score (Labs, ). In summary, we make the following contributions: - We identify the memory footprint bottleneck in multi-LoRA agent serving, where adapter-specific KV cache divergence makes prefix caching ineffective. - We propose ForkKV, a multi-LoRA agent serving system inspired by the OS fork primitive with copy-on-write, utilizing a DualRadixTree to disaggregate the KV cache into a shareable base and LoRA-specific residuals. - We design ResidualAttention, an attention kernel fusing KV cache reconstruction for disaggregated KV cache layout. - We comprehensively evaluate ForkKV across diverse LLMs and datasets, demonstrating significant improvement over state-of-the-art LoRA serving systems. ## 2. Background We first introduce the mechanics of LLM serving (§ 2.1), then analyze the structural properties of LoRA (§ 2.2), and finally discuss the computational demands of modern agentic workflows (§ 2.3). ### 2.1. LLM Serving LLMs (AI@Meta, 2024; Team, 2024; Yang et al., 2025; DeepSeek-AI, 2024, 2025) predominantly adopt the Transformer architecture to generate text auto-regressively. During generation, tokens interact with historical context via attention mechanism (Vaswani et al., 2017; Ainslie et al., 2023; Shazeer, 2019), where sequential order is typically captured by applying Rotary Position Embedding (RoPE) (Su et al., 2024) to the Query ( $Q$ ) and Key ( $K$ ) representations. To avoid redundantly recomputing $K$ and $V$ tensors for historical tokens at every step, inference engines employ a KV cache. This optimization naturally divides the serving process into two phases: a compute-heavy prefill phase that processes the prompt to populate the initial KV cache, and a memory-bound decode phase that auto-regressively generates new tokens by attending to the cached history. This efficiency is further extended across different requests via prefix caching (Zheng et al., 2024; Google, 2025; OpenAI, 2025; Anthropic, 2025). By identifying and reusing the KV cache of shared text segments (e.g., system prompts or shared context), inference engines significantly accelerate the Time-to-First-Token (TTFT) and optimize overall memory usage (Yao et al., 2025a; Wu et al., 2024a; Zhong et al., 2024; Agrawal et al., 2023). ### 2.2. Low-Rank Adaptation (LoRA) Low-Rank Adaptation (LoRA) (Hu et al., 2022) is a prominent Parameter-Efficient Fine-Tuning (PEFT) (Mangrulkar et al., 2022) designed to mitigate the significant computational and memory costs associated with full-parameter fine-tuning. Instead of updating the entire model, LoRA freezes the pretrained weights and injects small trainable low-rank adapter matrices into transformer layers. Formally, for a pretrained weight matrix $W∈ℝ^m× n$ , LoRA $i$ introduces two low-rank matrices $A_i∈ℝ^m× r$ and $B_i∈ℝ^r× n$ , where the rank $r\ll m,n$ . The combined projection is computed as: $$ \displaystyle Y \displaystyle=xW+xA_iB_i \tag{1} $$ where $x∈ℝ^s× m$ is the input hidden state, and $s$ is the number of tokens in a batch (batch size multiplied by sequence length). This computation can be naturally decomposed into two distinct parts. We refer to the projection from the frozen weights, $xW∈ℝ^s× n$ , as the base model cache (bCache), and the intermediate projection from the first low-rank matrix, $xA_i∈ℝ^s× r$ , as the residual cache (rCache). The final projected state can thus be reconstructed via: $$ Y=bCache+rCache× B_i \tag{2} $$ This algebraic decomposition reveals two critical properties. First, because $r\ll n$ , the bCache is significantly larger than the rCache (e.g., 64 times larger given a typical $n=1024$ and $r=16$ ). Second, the output dimension ( $r$ ) of the rCache inherently mismatches the dimension ( $n$ ) required by the Rotary Position Embedding (RoPE) matrix $R_p$ . Consequently, RoPE cannot be directly applied to the rCache. Its application must be deferred until the rCache is projected back to the full $n$ -dimensional space via $B_i$ . ### 2.3. Agentic Workflow Modern LLM applications have evolved into agentic workflows, which are comprehensive pipelines that decompose complex problems into actionable steps. Each step is typically executed by an individual autonomous agent node. A defining characteristic of these workflows, whether in sequential (e.g.,ReAct (Yao et al., 2022)) or parallel (e.g., MapReduce (Luo et al., 2025)), is their forked context structure: agents rely on a massive shared static prefix (e.g., a large codebase) and subsequently branch off to build distinct contexts from dynamic outputs like prior reasoning steps and tool observations. As workflows grow more complex, the individual agents within them require highly specialized expertise. Because deploying a separate fully fine-tuned model for each specialized agent node is memory prohibitive (Chen et al., 2024; Sheng et al., 2023), multi-LoRA serving has emerged as the prevailing paradigm (Liu et al., 2024b; Yu et al., 2024; Liu et al., 2025b; Kadekodi et al., 2025). By multiplexing task-specific LoRA modules on a single shared base model, systems can efficiently support the diverse agents constituting the workflow while minimizing memory overhead. ## 3. Motivation In this section, we first highlight the critical memory footprint bottlenecks of prefix caching in multi-LoRA serving (§ 3.1). We then demonstrate the potential of context sharing via disaggregated KV cache (§ 3.2), followed by an analysis of the two primary system challenges this approach introduces (§ 3.3): the management of disaggregated KV cache, and the memory footprint and computational overhead caused by KV cache reconstruction. ### 3.1. Inefficient Prefix Caching Prefix caching significantly reduces Time-to-First-Token (TTFT) and improves throughput in modern LLM serving systems (Yao et al., 2025a; Wu et al., 2024a; Zhong et al., 2024; Agrawal et al., 2023). However, this mechanism relies on the assumption that identical text prefixes yield identical KV cache. This assumption fails in multi-LoRA serving scenarios where requests target distinct adapters. Unique LoRA activations cause KV cache divergence even if the text prefixes are exactly the same. Consequently, the generated KV cache becomes strictly tied to specific adapters and can no longer be shared. The system is therefore forced to compute and store duplicated KV cache for each adapter, rendering traditional prefix caching ineffective. This redundancy becomes exceptionally severe in multi-LoRA agent serving scenarios. The execution context in these workflows is typically dominated by massive shared static inputs such as system prompts, codebases, or internal documents (Liu et al., 2024b), alongside continuously appended intermediate steps. Figure 3 illustrates the throughput of ReAct and MapReduce workflows operating on 32K contexts across varying numbers of distinct concurrent workflows, where each workflow utilizes a completely non-overlapping set of LoRA adapters. As the number of workflows scales from 1 to 8, the throughput for ReAct and MapReduce drops by 90.8% and 90.1% respectively. This degradation occurs because, in both sequential workflows like ReAct (Figure 2 a) and parallel workflows like MapReduce (Figure 2 b), the inability to share KV cache across distinct LoRA adapters means agents must repeatedly process the massive static inputs alongside incremental contexts and maintain independent KV cache at each reasoning step, which not only inflates TTFT but also drastically increases memory consumption. As concurrency increases, these redundant KV cache rapidly exhaust GPU memory. This high memory pressure leaves little space for other active requests, severely restricting batch parallelism and causing the observed throughput collapse. Takeaway #1: Prefix caching is inefficient in multi-LoRA agent serving scenarios because KV cache cannot be shared across agents with different LoRA adapters, resulting in critical throughput degradation. <details> <summary>2604.06370v1/x2.png Details</summary> ![6f579b98](/v1/image/6f579b982eba44b8b624ec3c35eca516b8699cf6705611219782ce5c467465a4) ### Visual Description ## Diagram: Workflow Comparison of ReAct and MapReduce with Context Constraints ### Overview The image compares two workflows—**ReAct** and **MapReduce**—highlighting their structural differences and context reuse limitations due to a "LoRA switch." Both workflows involve sequential steps (labeled 1, 2, 3, 4) and context dependencies, with visual annotations indicating irreusable context elements. ### Components/Axes - **Legend**: - **Yellow squares**: Represent `iᵗʰ Agent` (agent actions). - **Green squares**: Represent `Tool Call` (external tool interactions). - **Red X**: Indicates "Cannot reuse gray context because of LoRA switch." - **Labels**: - **Left side**: "Workflow" (top) and "Context" (bottom). - **Arrows**: Connect workflow steps to context elements, showing dependencies. ### Detailed Analysis #### (a) ReAct Workflow - **Workflow**: - Linear sequence: `1 → 2` (yellow squares for agents, green squares for tool calls). - **Context**: - Steps `1` and `2` are grayed out. - A red X marks step `2`, indicating it cannot be reused due to the LoRA switch. #### (b) MapReduce Workflow - **Workflow**: - Branching structure: - Path 1: `1 → 2 → 4` (steps 1 and 2 are agents; step 4 is a tool call). - Path 2: `3 → 4` (step 3 is an agent; step 4 is a tool call). - **Context**: - Steps `1`, `2`, and `3` are grayed out. - A red X marks step `1`, indicating it cannot be reused due to the LoRA switch. ### Key Observations 1. **ReAct**: - Simpler, linear workflow with fewer steps. - Context reuse failure occurs at step `2` (tool call). 2. **MapReduce**: - More complex, branching workflow with parallel paths. - Context reuse failure occurs at step `1` (agent action). 3. **LoRA Switch Impact**: - Both workflows lose access to specific context steps, but the failure points differ (step `2` in ReAct vs. step `1` in MapReduce). ### Interpretation The diagrams illustrate how workflow design affects context reuse under LoRA constraints. ReAct’s linear approach leads to a single point of failure in context reuse, while MapReduce’s branching structure introduces redundancy but still faces limitations. The inability to reuse grayed-out context steps suggests that LoRA switches disrupt dependencies critical to workflow continuity. This could imply trade-offs between workflow complexity and adaptability in dynamic environments. **Note**: No numerical data or trends are present; the analysis focuses on structural and dependency relationships. </details> Figure 2. Failure of reusing KV cache across individual agents using different LoRA adapters. <details> <summary>2604.06370v1/figures/motivation-lora.png Details</summary> ![39057f2d](/v1/image/39057f2d94ae6d9be68c8baa7dbf957220579eff39b77b5e6977719eee806c60) ### Visual Description ## Line Graph: Throughput vs. Number of Concurrent Workflows ### Overview The image is a line graph comparing the throughput (tasks/s) of two workflow systems, **ReAct** and **MapReduce**, as the number of concurrent workflows increases. The x-axis represents the number of concurrent workflows (1, 2, 4, 8), and the y-axis represents throughput in tasks per second (0 to 0.6). Two data series are plotted: ReAct (dark blue) and MapReduce (teal). ### Components/Axes - **X-axis**: "Number of Concurrent Workflows" (values: 1, 2, 4, 8). - **Y-axis**: "Throughput (tasks/s)" (range: 0 to 0.6). - **Legend**: Located on the right side of the graph. - **ReAct**: Dark blue line with circular markers. - **MapReduce**: Teal line with circular markers. ### Detailed Analysis #### ReAct (Dark Blue) - **Data Points**: - At 1 workflow: ~0.52 tasks/s. - At 2 workflows: ~0.45 tasks/s. - At 4 workflows: ~0.30 tasks/s. - At 8 workflows: ~0.05 tasks/s. - **Trend**: A steady linear decline in throughput as the number of concurrent workflows increases. #### MapReduce (Teal) - **Data Points**: - At 1 workflow: ~0.20 tasks/s. - At 2 workflows: ~0.05 tasks/s. - At 4 workflows: ~0.02 tasks/s. - At 8 workflows: ~0.02 tasks/s. - **Trend**: A sharp initial drop in throughput between 1 and 2 workflows, followed by a plateau at ~0.02 tasks/s for higher concurrency. ### Key Observations 1. **ReAct** maintains higher throughput than **MapReduce** across all concurrency levels but experiences a consistent decline. 2. **MapReduce** shows a dramatic drop in performance when scaling from 1 to 2 workflows, then stabilizes at a low throughput. 3. Both systems degrade in performance as concurrency increases, but **ReAct** retains a higher absolute throughput. ### Interpretation The graph suggests that **ReAct** is more scalable than **MapReduce** for handling concurrent workflows, as it retains higher throughput even at 8 workflows (~0.05 tasks/s). However, both systems exhibit diminishing returns with increased concurrency. The sharp decline in **MapReduce**’s performance at 2 workflows implies potential bottlenecks or inefficiencies in its design for parallel processing. This could indicate that **ReAct** is better suited for high-concurrency environments, while **MapReduce** may require optimization to improve scalability. </details> Figure 3. End-to-end throughput of prefix caching with different number of concurrent workflows. Every individual agent in different workflows uses different LoRA adapters. ### 3.2. Opportunities of Sharing Contexts To mitigate memory contention in multi-LoRA agent serving, our key insight is to replace the traditional unified KV cache with a physically decoupled architecture. We introduce a partially shared memory layout termed the disaggregated KV cache. As established in Section 2.2, the standard LoRA projection $xW+xA_iB_i$ naturally decomposes the attention states into a massive base cache ( $bCache=xW$ ) and a lightweight residual cache ( $rCache=xA_i$ ) that can reconstruct the LoRA projection through $rCache× B_i$ . This decomposition exposes a significant size asymmetry, where bCache is typically dozens of times the size of rCache due to the low-rank projection of LoRA. Our design directly exploits this structural property. Traditional prefix caching, as shown in Figure 4 a, redundantly allocates a complete KV cache for each concurrent agent. Instead, our approach computes the large $bCache$ exactly once and shares it globally across all agents processing identical contexts. Each individual agent then only allocates memory for its own small $rCache$ . This theoretical advantage translates into concrete system benefits. Consider 16 concurrent agents processing the same 32K context on Llama3-8B. Traditional methods require 4GB of GPU memory per agent, consuming 64GB in total. In contrast, the disaggregated layout requires only a single 4GB $bCache$ alongside 16 unique 64MB $rCache$ allocations. This reduces the total memory consumption to approximately 5GB, yielding an 11.8 $×$ memory efficiency improvement. Consequently, the disaggregated architecture transforms the inherent decomposability of the KV cache into a critical optimization opportunity, fundamentally alleviating memory contention in multi-LoRA agent serving. <details> <summary>2604.06370v1/x3.png Details</summary> ![b874b76d](/v1/image/b874b76dd8d2feceb42488e028c2563b2302cf7a0012271d65fe40372c9ec1d9) ### Visual Description ## Diagram: KV Cache Strategies for LLaMA3-8B with LoRA Rank=16 ### Overview The diagram compares two KV cache management strategies for LLaMA3-8B with LoRA rank=16: 1. **Prefix Caching (a)**: Unified KV Cache per agent (large memory footprint). 2. **ForkKV (b)**: Disaggregated KV Cache with shared `bCache` and per-agent `rCache` (smaller memory footprint). ### Components/Axes - **Legend**: - Blue: `KV Cache` (size `N`). - Gray: `bCache` (size `xW`). - Yellow: `rCache` (size `xA_i`, 1/64th of `KV Cache`). - **Key Elements**: - **Prefix Caching (a)**: - `KV Cache-1` to `KV Cache-N` (blue boxes). - Total size: `Large Size (N)`. - Redundant storage: Each agent has its own full `KV Cache`. - **ForkKV (b)**: - `KV Cache-i` decomposed into `bCache` (shared) + `rCache-i` (per-agent). - Total size: `Small Size (1 + N/64)`. - `bCache` reused across agents; `rCache` stored per agent. ### Detailed Analysis - **Prefix Caching (a)**: - Each agent stores a full `KV Cache` (blue), leading to `N` redundant copies. - Total memory: `N × size(KV Cache)`. - **ForkKV (b)**: - `KV Cache-i` = `bCache` (shared, size `xW`) + `rCache-i` (per-agent, size `xA_i`). - `bCache` is reused across all agents, reducing redundancy. - `rCache-i` is lightweight (`1/64th` of `KV Cache` size). - Total memory: `size(bCache) + N × size(rCache)`. ### Key Observations 1. **Memory Efficiency**: - ForkKV reduces total cache size from `N` (Prefix Caching) to `1 + N/64` (ForkKV). - For large `N`, ForkKV achieves ~64× memory savings. 2. **Color Consistency**: - Blue (`KV Cache`) matches legend; gray (`bCache`) and yellow (`rCache`) align with their labels. 3. **Equations**: - `KV Cache-i = bCache (xW) + rCache-i (xA_i) × B_i` (ForkKV decomposition). ### Interpretation - **Technical Implications**: - ForkKV optimizes memory usage by sharing `bCache` across agents, critical for resource-constrained environments (e.g., edge devices). - Prefix Caching’s redundancy is inefficient for large-scale deployments but simpler to implement. - **Design Trade-offs**: - ForkKV introduces complexity in managing shared `bCache` and per-agent `rCache` but offers significant memory savings. - The `1/64` size reduction for `rCache` suggests LoRA rank=16 enables lightweight caching. - **Outliers/Anomalies**: - No explicit outliers; the diagram focuses on theoretical size comparisons. This analysis highlights ForkKV as a memory-efficient alternative to Prefix Caching, leveraging shared and lightweight caches for scalable LLaMA3-8B deployments. </details> Figure 4. Memory usage comparison of ForkKV v.s. prefix caching when serving N agents with distinct LoRAs (r=16) on Llama3-8B. bCache and rCache represent base cache and residual cache respectively in disaggregated KV cache layout. Negligible Accuracy Loss. While the disaggregated KV cache significantly improves memory efficiency, sharing a unified base cache beyond the first layer is mathematically lossy. This occurs because adapter-specific activations at each transformer layer cause the exact input state $x$ to diverge across agents. However, this approximation introduces minimal error in practice, yielding only a 1.60% accuracy loss on tool calling benchmarks like APIGen, as demonstrated in Figure 5(a). This minimal impact is driven by two key factors. First, the residual connections $x_l+1=x_l+F(x_l)$ within the transformer architecture prevent the base state $x_l$ from radical drift (Elhage et al., 2021). Second, our disaggregated layout explicitly preserves the task-specific interactions between Q, K and V within each LoRA adapter, which are necessary for accurate attention outputs. While residual connections bound the overall state divergence, high generation quality still requires the attention mechanism to correctly extract task-specific features. Previous studies demonstrate that the effectiveness of LoRA relies on joint optimization of these QKV projections (Hu et al., 2022; Wang et al., 2023b). Our disaggregated layout explicitly preserves this mechanism by computing a dedicated adapter cache for each agent. Therefore, every adapter applies its unique K and V transformations to the shared context. This guarantees accurate attention outputs and prevents errors from compounding in the residual stream. As a result, our method maintains an input state cosine similarity of over 99.4% across all layers compared to standard prefix caching (Figure 5(b)). In contrast, the full reuse baseline entirely shares the KV cache and breaks the necessary QKV coadaptation. This leads to inaccurate attention outputs that progressively accumulate, dropping the cache similarity to approximately 92.4% and causing a severe 21.0% accuracy loss (Figure 5). Limitations of Prior Approaches. Existing multi-LoRA agent serving systems are fundamentally trapped in a dilemma between memory scalability and generation accuracy. Maintaining independent KV cache for each adapter preserves accuracy but incurs prohibitive memory overhead, making high-concurrency deployments unfeasible. Conversely, as illustrated above, entirely sharing the cache through a full reuse paradigm alleviates memory pressure but severely degrades generation quality. To navigate this bottleneck, recent studies have explored selective KV cache recomputation (Yao et al., 2025a; Gim et al., 2024). However, these techniques are explicitly designed for single-model environments and fail to manage the state divergence across multiple agents. While DroidSpeak (Liu et al., 2024b) extends cache sharing to multi-model scenarios by recomputing critical layers, it still treats the KV cache as an indivisible monolithic unit, missing a critical opportunity to minimize the memory footprint through structural decoupling. Overall, these fundamental limitations necessitate a new disaggregated KV cache architecture tailored for multi-LoRA agent serving. Takeaway #2: Replacing the monolithic KV cache with a shared base and an isolated residual significantly reduces memory consumption for concurrent agents without compromising on generation quality. <details> <summary>2604.06370v1/figures/motivation-f1-score.png Details</summary> ![e8d2e6dc](/v1/image/e8d2e6dc2a16a8f64af9b65c75162bff36c7ff644614ca909bc95b200dc0c87d) ### Visual Description ## Bar Chart: F1-Score Comparison Across Caching Strategies ### Overview The chart compares the F1-Score performance of three caching strategies: Prefix Caching, ForkKV, and Full Reuse. The y-axis represents F1-Score values ranging from 0.15 to 0.40, while the x-axis categorizes the strategies. Three distinct bars are present, with varying heights and colors. ### Components/Axes - **X-Axis (Categories)**: - Prefix Caching (dark blue bar) - ForkKV (teal bar) - Full Reuse (light green bar) - **Y-Axis (F1-Score)**: - Scale: 0.15 to 0.40 in increments of 0.05 - Label: "F1-Score" - **Legend**: - Positioned on the right side of the chart - Colors: - Dark blue = Prefix Caching - Teal = ForkKV - Light green = Full Reuse ### Detailed Analysis 1. **Prefix Caching**: - Bar height: ~0.39–0.40 (dark blue) - Position: Leftmost bar, tallest among the three 2. **ForkKV**: - Bar height: ~0.37–0.38 (teal) - Position: Middle bar, slightly shorter than Prefix Caching 3. **Full Reuse**: - Bar height: ~0.17 (light green) - Position: Rightmost bar, significantly shorter than the other two ### Key Observations - The F1-Score for Prefix Caching and ForkKV are nearly identical, both exceeding 0.35. - Full Reuse has the lowest F1-Score, approximately 0.17, which is ~50% lower than the other two strategies. - The y-axis starts at 0.15, emphasizing the disparity in Full Reuse's performance. ### Interpretation The data suggests that Prefix Caching and ForkKV are similarly effective, with high F1-Scores indicating balanced precision and recall. Full Reuse, however, performs poorly, potentially due to inefficiencies in data retrieval or higher error rates. The stark contrast between Full Reuse and the other two strategies highlights its inferiority in this context. The chart uses distinct colors to differentiate strategies, ensuring clarity in comparison. The absence of error bars or confidence intervals limits the ability to assess statistical significance, but the visual disparity is clear. </details> (a) Generation Quality <details> <summary>2604.06370v1/figures/motivation-similarity.png Details</summary> ![2dfd2c58](/v1/image/2dfd2c583b709a820096b4323a2b8d46dbbfd70aca92b478799c14fa8a945d30) ### Visual Description ## Line Graph: Cosine Similarity vs. Layer Index ### Overview The image is a line graph comparing two data series labeled "ForkKV" (teal) and "Full Reuse" (green) across a Layer Index (x-axis: 0–32) and Cosine Similarity (y-axis: 0.90–1.00). Both series begin at 1.00 similarity, but their trajectories diverge significantly. ### Components/Axes - **X-axis (Layer Index)**: Discrete ticks at 0, 8, 16, 24, 32. - **Y-axis (Cosine Similarity)**: Incremental markers at 0.90, 0.92, 0.94, 0.96, 0.98, 1.00. - **Legend**: Located at the bottom-left, with "ForkKV" (teal) and "Full Reuse" (green) clearly labeled. - **Data Series**: - **ForkKV**: A teal line with minimal fluctuation. - **Full Reuse**: A green line with pronounced variability. ### Detailed Analysis 1. **ForkKV (Teal)**: - Starts at **1.00** (Layer 0). - Remains flat at **~1.00** until Layer 8. - Slight decline to **~0.998** between Layers 8–16. - Stabilizes near **0.998–1.00** for Layers 16–32. 2. **Full Reuse (Green)**: - Begins at **1.00** (Layer 0). - Gradual decline to **~0.95** by Layer 16. - Sharp drop to **~0.92** between Layers 16–24. - Partial recovery to **~0.95** by Layer 32. ### Key Observations - **ForkKV** maintains near-perfect similarity (cosine ~1.00) across all layers, indicating stability. - **Full Reuse** exhibits a **steady decline** until Layer 24, followed by a modest rebound. - The largest divergence occurs between Layers 16–24, where Full Reuse drops to **~0.92** while ForkKV remains stable. ### Interpretation The graph suggests that **ForkKV** preserves structural similarity more effectively than **Full Reuse**, which experiences significant degradation in middle layers (16–24). The recovery of Full Reuse at Layer 32 implies potential stabilization in later stages, but its overall performance lags behind ForkKV. This could indicate architectural differences in how these systems handle layer indexing or data reuse. The teal line’s consistency highlights ForkKV’s robustness, while the green line’s volatility underscores trade-offs in Full Reuse’s design. </details> (b) Similarity of Input x Figure 5. ForkKV maintains comparable (a) generation quality and (b) input x similarity compared to prefix caching. Experiments are conducted on APIGen dataset. ### 3.3. Challenges While a disaggregated KV cache architecture maximizes context sharing with negligible accuracy loss, realizing these benefits exposes two fundamental challenges in system design. Managing Disaggregated KV Cache. The primary challenge lies in managing the distinct lifecycles of the bCache and rCache while maintaining the structural dependencies imposed by their mathematical decomposition. This difficulty stems from their different access patterns. The massive bCache acts as a shared foundation accessed by multiple concurrent agents. Conversely, the lightweight rCache is strictly tied to individual agents using different LoRA adapters. Existing serving systems (Kwon et al., 2023; Zheng et al., 2024; Sheng et al., 2023) typically assume an indivisible KV cache. They manage these states under a single lifecycle within a unified memory pool, inherently lacking the structural abstraction required for a disaggregated memory architecture. Furthermore, as multi-agent collaboration naturally forms multi-branch reasoning paths, tracking the resulting 1-to-N base-to-residual mappings within a unified pool introduces severe complexity when resolving context dependencies. These structural limitations therefore necessitate a novel caching abstraction specifically tailored for the disaggregated KV cache layout. KV Cache Reconstruction. Although decomposing the KV cache improves memory efficiency, computing accurate attention scores requires reconstructing KV cache from its disaggregated components. This requirement introduces a critical challenge because naive HBM-based reconstruction methods incur severe memory footprint and computational overhead. For instance, materializing a full-sized KV cache in HBM for every agent prior to attention computation completely negates our intended memory savings. An alternative approach involves directly updating the shared bCache in place using LoRA projections recovered from the rCache. However, this operation causes memory access conflicts across concurrent agents, forcing sequential execution and destroying intra-batch parallelism. Consequently, we must design a reconstruction mechanism that preserves both memory efficiency and batched execution, thereby enabling high-performance inference under the disaggregated KV cache architecture. ## 4. Overview and Key Ideas To fully exploit the opportunities of context sharing in multi-LoRA agent serving scenarios, we need to address two challenges: the management of disaggregated KV cache, and the memory footprint and computational overhead caused by KV cache reconstruction. In this paper, we propose ForkKV, a multi-LoRA agent serving system that improves throughput using an OS-inspired disaggregated cache management mechanism and a novel attention kernel. In this section, we summarize two key ideas behind our system: 1. OS-Inspired Disaggregated Cache Management. Managing a disaggregated KV cache requires a novel memory abstraction. Traditional unified pools fail to accommodate the distinct lifecycles and structural dependencies of shared and unique memory components. To bridge this gap, ForkKV introduces a DualRadixTree architecture. As shown in Figure 7, this structure natively supports the disaggregated layout by physically separating the management of the massive base cache from the lightweight residual cache. To orchestrate this decoupled memory architecture, we introduce an operating system inspired fork semantics with copy-on-write. As demonstrated in Figure 6, when a new agent is launched, it forks the cache state from an existing agent in two steps. First, the new agent inherits the globally shared base cache which contains the same context in the base radix tree. This process is analogous to a newly forked OS process mapping the read-only physical pages of its parent. Second, the agent allocates exclusive memory for its unique residual cache in the residual radix tree. This part serves as the child process’s isolated copy-on-write footprint for the new agent. Together, the dual-tree design and fork semantics translate our theoretical decoupling into a practical system design that systematically resolves the memory management complexity of disaggregated KV cache. <details> <summary>2604.06370v1/x4.png Details</summary> ![e24d49b1](/v1/image/e24d49b19f08a00b1a772f51b212f49936943ef5c8ec5cb05a5b0f3d39c920c7) ### Visual Description ## Diagram: Process Memory Management with Copy-on-Write ### Overview The diagram illustrates the memory management mechanism during process forking, specifically highlighting the **Copy-on-Write (CoW)** optimization. It shows how a parent process and its child process share physical memory pages until a write operation occurs, at which point memory is duplicated to ensure isolation. --- ### Components/Axes 1. **Key Elements**: - **Parent Process**: Initiates the fork operation. - **Child Process**: Created via the fork operation. - **fork**: Arrows indicate the creation of the child process. - **Phys Page**: Physical memory page shared between processes. - **Copy-on-Write (CoW)**: Mechanism triggered on write operations. 2. **Flow**: - Parent Process → fork → Child Process. - Shared Phys Page (gray) → Reused by both processes. - Child Process modifies Phys Page → CoW triggers → New Phys Page (yellow) created. 3. **Visual Indicators**: - **Dashed Box**: Highlights the Phys Page before CoW. - **Solid Yellow Box**: Represents the Phys Page after CoW. - **Arrows**: Denote data flow and process relationships. --- ### Detailed Analysis 1. **Initial State**: - Parent and Child Processes share the same **Phys Page** (gray), indicating memory reuse. - The **fork** operation creates the Child Process without duplicating memory. 2. **Copy-on-Write Trigger**: - When the Child Process modifies the shared Phys Page, the system detects the write. - A new **Phys Page** (yellow) is allocated, and the Child Process’s reference is updated to this new page. - The Parent Process retains the original Phys Page. 3. **Memory Isolation**: - After CoW, the Parent and Child Processes have distinct Phys Pages, ensuring isolation. - The dashed box around the intermediate Phys Page emphasizes the transient state before duplication. --- ### Key Observations 1. **Shared Memory Reuse**: The gray Phys Page is reused until a write occurs, optimizing memory usage. 2. **CoW Mechanism**: The yellow Phys Page represents the duplicated memory after a write, ensuring process isolation. 3. **Dashed vs. Solid Boxes**: The dashed box (intermediate state) and solid yellow box (post-CoW state) visually distinguish memory states. --- ### Interpretation - **Efficiency**: By sharing Phys Pages initially, the system avoids unnecessary memory duplication, reducing overhead. - **Isolation**: CoW ensures that modifications in the Child Process do not affect the Parent Process, maintaining process boundaries. - **Mechanism Insight**: The diagram clarifies how operating systems balance memory efficiency and isolation, a critical concept in virtual memory management. This diagram is foundational for understanding how modern operating systems handle process memory during forking, emphasizing the interplay between resource sharing and isolation. </details> (a) OS: Create child process <details> <summary>2604.06370v1/x5.png Details</summary> ![7ce61dd9](/v1/image/7ce61dd99c023dcb4f3fae7a06fa20fa4b71be8db7332f7dbee590549221a9ce) ### Visual Description ## Diagram: Agent-Cache Interaction with RadixTree Structure ### Overview The diagram illustrates a system architecture involving two agents (Former and Current) interacting with two cache types (bCache and rCache) and two RadixTree structures (base and residual). Arrows indicate data flow, with labels like "fork," "reused," and "Copy-on-Write" describing operations. The diagram uses color coding (gray for bCache, yellow for rCache) and dashed lines to differentiate components. --- ### Components/Axes 1. **Nodes**: - **Former Agent**: Top-left, connected via "fork" to Current Agent. - **Current Agent**: Top-right, connected to bCache (dashed) and rCache (solid). - **bCache**: Gray box, labeled "reused," connected to both agents. - **rCache**: Yellow box, connected to Current Agent with a loop labeled "Copy-on-Write." - **RadixTree (base)**: Dashed line above bCache. - **RadixTree (residual)**: Dashed line below rCache. 2. **Arrows**: - **Fork**: From Former Agent to Current Agent (solid black). - **Reused**: From bCache to Current Agent (solid black). - **Copy-on-Write**: Loop from rCache to itself (solid black). 3. **Color Coding**: - **Gray**: bCache (base cache). - **Yellow**: rCache (residual cache). - **Black**: All other elements (agents, arrows, labels). --- ### Detailed Analysis 1. **Flow Paths**: - **Former → Current Agent**: The "fork" arrow suggests a branching point where data is split or redirected. - **bCache Reuse**: The dashed line from bCache to Current Agent implies shared or cached data is reused. - **rCache Loop**: The "Copy-on-Write" loop indicates data modification without overwriting original data, common in copy-on-write (CoW) systems. 2. **RadixTree Structure**: - **Base RadixTree**: Positioned above bCache, likely representing the primary data structure. - **Residual RadixTree**: Positioned below rCache, possibly handling residual or delta data. 3. **Color Significance**: - Gray (bCache) and yellow (rCache) visually distinguish cache types, with bCache being "reused" and rCache tied to CoW operations. --- ### Key Observations 1. **Data Reuse**: bCache is explicitly labeled "reused," suggesting it acts as a shared cache between agents. 2. **CoW Mechanism**: The rCache loop emphasizes a CoW strategy, critical for data integrity in distributed systems. 3. **RadixTree Segmentation**: The split into base and residual RadixTrees implies hierarchical data management, with base handling core data and residual managing updates. --- ### Interpretation This diagram likely represents a distributed system or database architecture where: - **Former and Current Agents** collaborate, with data flowing between them via a fork mechanism. - **bCache** optimizes performance by reusing cached data, reducing redundant computations. - **rCache** employs CoW to ensure data consistency during updates, critical for systems requiring atomic operations. - **RadixTrees** (base/residual) manage data partitioning, with base handling stable data and residual tracking changes. The use of dashed lines for bCache connections may indicate optional or conditional interactions, while solid lines for rCache emphasize mandatory CoW operations. The diagram highlights a balance between data reuse (bCache) and consistency (rCache/CoW), common in systems prioritizing both efficiency and reliability. </details> (b) ForkKV: Create new agent Figure 6. ForkKV use OS-inspired fork semantics to create memory space for new agents with copy-on-write. 2. ResidualAttention. KV cache reconstruction incurs severe memory and computational overhead when operating in HBM. Previous naive methods struggle with either a prohibitive HBM allocation or a severe degradation of intra-batch parallelism. To address this challenge, we propose fusing KV cache reconstruction directly into the attention kernel. By keeping all intermediate computations within the fast on-chip SRAM, this design eliminates both the extra HBM allocation and the serialized execution caused by conflicting memory accesses. Based on this insight, we implement ResidualAttention, an attention kernel specifically tailored for the disaggregated KV cache architecture. Rather than assuming a monolithic KV cache layout, ResidualAttention streams the decoupled bCache and rCache directly into the SRAM in a block-wise manner. Inside the SRAM, the kernel first reconstructs the LoRA residuals of K using up-projection and a deferred RoPE operation. It then computes attention scores separately for the base and residual components and fuses the final output by leveraging matrix associativity. Through this algorithm and system co-design, ResidualAttention preserves the memory savings enabled by the disaggregated KV cache while ensuring high-throughput batched execution for concurrent agents. System Workflow. Figure 7 illustrates the execution pipeline of ForkKV. Upon receiving an agent request, the system enqueues it into the queue. The scheduler retrieves the request, parses the agent context, and then queries the DualRadixTree to perform a prefix matching against the existing base cache. To construct the memory state for the new agent, the system inherits this matched base cache and allocates an exclusive memory region for the residual cache of the specified adapter. Following memory allocation, the scheduler dispatches the request to the agent runner, where ForkKV loads the requested LoRA adapters and establishes the agent loop to interleave model reasoning with external tool invocations. At the core of the agent runner is the GPU executor, which serves concurrent inference workloads across multiple agents. During execution, the cache controller directly loads and stores the base and residual cache according to the memory regions assigned by the scheduler. Operating on these retrieved data blocks, our custom ResidualAttention kernel computes attention natively over the disaggregated memory layout. Finally, the agent runner returns the generated outputs to the scheduler to formulate the client response. <details> <summary>2604.06370v1/x6.png Details</summary> ![00b80fe4](/v1/image/00b80fe42f747e4d90a2704896bdcd7822a32f2ecd3791b952f53681347854cd) ### Visual Description ## System Architecture Diagram: LoRA Adapters, Agent Runner, and DualRadixTree Caching System ### Overview The diagram illustrates a distributed system architecture for managing large language model (LLM) operations, featuring LoRA adapters, an agent runner, a scheduler, and a dual caching system. Components are interconnected via directional arrows indicating data flow and control signals. ### Components/Axes 1. **Top-Left Section (LoRA Adapters)** - Contains gray rectangles labeled "LoRA Adapters" - Arrows labeled "Load" (upward) and "Offload" (downward) connect to the Agent Runner 2. **Center-Left Section (Agent Runner)** - Contains: - "Agent Loop" (rectangle) - "GPU Executor" (rectangle) - "Residual Attention (§5.3)" (green rectangle) - "Cache Controller (§5.1)" (blue rectangle) - Arrows labeled "Dispatch" (rightward) and "Store/Load" (rightward) connect to the Scheduler and DualRadixTree 3. **Top-Right Section (Request Queue)** - Contains gray rectangles labeled "Request Queue" - Arrows labeled "Agent Requests" (leftward) and "Agent Response" (rightward) connect to the Scheduler 4. **Center Section (Scheduler)** - Contains "Scheduler" (rectangle) - Arrows labeled "Results" (leftward) and "Manage" (downward) connect to the Agent Runner and DualRadixTree 5. **Bottom-Right Section (DualRadixTree)** - Contains two sub-components: - **RadixTree (base cache)**: - Gray rectangles labeled "LLMs" - Arrows labeled "token_id" (downward) - **RadixTree (residual cache)**: - Yellow rectangles labeled "LLMs" - Arrows labeled "token_id, agent_id" (downward) ### Detailed Analysis - **LoRA Adapters**: Positioned at the top-left, these components interface with the Agent Runner via load/offload operations. - **Agent Runner**: Central processing unit with GPU acceleration (Residual Attention) and cache management (Cache Controller). - **Scheduler**: Acts as a mediator between the Agent Runner and Request Queue, managing task dispatch and results. - **DualRadixTree**: Implements a two-tier caching strategy: - **Base Cache**: Stores LLM data by `token_id` (gray nodes). - **Residual Cache**: Stores LLM data by `token_id` and `agent_id` (yellow nodes), enabling agent-specific optimizations. ### Key Observations 1. **Color Coding**: - Gray: Base cache components (LoRA Adapters, Request Queue, base RadixTree). - Green: Residual Attention module (GPU-accelerated processing). - Blue: Cache Controller (cache management logic). - Yellow: Residual cache components (agent-specific LLM data). 2. **Data Flow**: - Requests flow from the Request Queue to the Scheduler, which coordinates with the Agent Runner and DualRadixTree. - Results from the Agent Runner are stored in the DualRadixTree via the Cache Controller. 3. **Hierarchical Caching**: - The DualRadixTree separates general LLM data (base cache) from agent-specific optimizations (residual cache), suggesting a design for scalability and personalization. ### Interpretation This architecture optimizes LLM operations by: 1. **Offloading Computation**: Using LoRA adapters to manage model parameters efficiently. 2. **Parallel Processing**: Leveraging GPU Executors for residual attention tasks. 3. **Caching Strategy**: The dual RadixTree structure reduces redundant computations by caching base LLM data globally and agent-specific residuals locally. The system prioritizes modularity, with clear separation between computation (Agent Runner), coordination (Scheduler), and data management (DualRadixTree). The use of agent-specific caching (`token_id, agent_id`) implies a focus on personalized or context-aware LLM responses. **Note**: No numerical data or explicit trends are present; the diagram focuses on structural relationships and workflow. </details> Figure 7. Overview of ForkKV ## 5. System Design ### 5.1. Disaggregating KV Cache Current unified KV cache design introduces a severe memory footprint bottleneck during concurrent multi-LoRA agent serving. As illustrated in Figure 8 (a), to generate Key and Value tensors, the system projects the input $x$ through both the base weight $W$ and the LoRA matrices $A_i$ and $B_i$ . Crucially, for Key tensors, the Rotary Position Embedding (RoPE) is applied to this fully merged result. The attention mechanism then caches the complete tensor $xW+xA_iB_i$ into a unified KV Cache. This design inherently binds the shared base projection $xW$ to adapter-specific updates, forcing every active agent to maintain an isolated, full-sized KV cache. To eliminate this storage redundancy, ForkKV introduces a disaggregated KV cache architecture, as depicted in Figure 8 (b). For Key and Value processing, ForkKV decouples the base model activations from the adapter states. The system first computes the base projection $xW$ , applies RoPE to the Key tensors, and stores the resulting states in a unified base cache (bCache). Concurrent requests with shared contexts can then access this bCache via zero-copy sharing. To further minimize memory allocation, ForkKV avoids computing the full adapter offset $xA_iB_i$ . Instead, the system truncates the computation at the LoRA down-projection and stores the intermediate result $xA_i$ directly in a residual cache (rCache). Notably, the RoPE operation is not applied to rCache because their output dimensions mismatch. The low-rank property of the $A_i$ matrix ensures that the rCache maintains a minimal memory footprint, drastically reducing the overall memory requirements of the serving system. To quantify these memory savings mathematically, we formulate the memory consumption ratio $M_R$ between the unified and disaggregated KV cache architectures. Consider $N$ concurrent agents processing a shared context sequence of length $s$ : $$ \displaystyle M_R=\frac{Mem_disagg.}{Mem_unified} \displaystyle=\frac{Mem(xW)+N· Mem(xA_i)}{N· Mem(xW+xA_iB_i)} \displaystyle=\frac{sn+N· sr}{N· sn}=\frac{1}{N}+\frac{r}{n} \tag{3} $$ where $n$ denotes the output dimension of the base weight matrix $W$ , and $r$ represents the rank of the LoRA down-projection matrix $A_i$ . In standard configurations, $r\ll n$ (e.g., $n=1024$ and $r=16$ ). Consequently, as the number of concurrent agents $N$ grows large, the term $1/N$ approaches $0$ , reducing the memory ratio to $M_R≈ r/n$ . This theoretical bound demonstrates that ForkKV drastically minimizes the per-agent memory consumption, allowing the system to support massive multi-LoRA agent workflows without memory exhaustion. <details> <summary>2604.06370v1/x7.png Details</summary> ![7fc23ccf](/v1/image/7fc23ccfd2f6f7058978192b1fecd6fb668b9928d46c9ec81c049e35f28e26e0) ### Visual Description ## Diagram: Attention Mechanism Cache Strategies ### Overview The image compares two approaches to handling key-value (KV) cache operations in attention mechanisms: - **(a) Unified KV Cache**: A single cache stores combined operations. - **(b) Disaggregated KV Cache**: Separates operations into two distinct caches (`bCache` and `rCache`). ### Components/Axes #### Labels and Flow: 1. **Inputs**: - `x`: Input tensor. - `W`: Weight matrix. - `A_i`: Attention-related parameters. - `B_i`: Bias terms. 2. **Operations**: - `xW`: Matrix multiplication of `x` and `W`. - `xA_i`: Element-wise multiplication of `x` and `A_i`. - `xW + xA_iB_i`: Combined operation before caching. 3. **Caching Mechanisms**: - **Unified KV Cache (a)**: - Stores `xW + xA_iB_i` directly. - Includes RoPE (Rotary Positional Encoding) applied to the combined result. - **Disaggregated KV Cache (b)**: - Splits operations into `bCache` (stores `xW`) and `rCache` (stores `xA_iB_i`). - RoPE is applied **only** to `bCache` (marked with a red "X" on `rCache`). 4. **Key Elements**: - **RoPE**: Applied to `bCache` in both strategies but excluded from `rCache` in the disaggregated approach. - **Attention Kernels**: Dashed box grouping all components. ### Detailed Analysis - **Unified KV Cache (a)**: - All operations (`xW`, `xA_iB_i`) are combined and cached together. - RoPE is applied post-combination, ensuring positional encoding affects the entire result. - **Disaggregated KV Cache (b)**: - `bCache` stores `xW` and applies RoPE. - `rCache` stores `xA_iB_i` **without** RoPE (explicitly blocked by the red "X"). - This separation may reduce computational overhead but risks misalignment in positional encoding. ### Key Observations 1. **RoPE Application**: - Unified: RoPE applied to the full `xW + xA_iB_i`. - Disaggregated: RoPE applied only to `bCache` (`xW`), not `rCache` (`xA_iB_i`). 2. **Cache Structure**: - Unified: Single cache for all operations. - Disaggregated: Two caches (`bCache` and `rCache`) with distinct roles. 3. **Flow Differences**: - Unified: Linear flow from input to combined cache. - Disaggregated: Branching flow with parallel caching paths. ### Interpretation The diagram highlights a trade-off between computational efficiency and positional encoding integrity: - **Unified KV Cache**: Simplifies implementation but may increase memory usage by caching combined operations. - **Disaggregated KV Cache**: Optimizes memory by splitting operations but risks positional encoding misalignment in `rCache` (since RoPE is omitted). The red "X" on `rCache` in (b) explicitly signals a design choice to exclude RoPE from the `xA_iB_i` component, which could impact attention mechanism accuracy. This suggests a focus on efficiency at the potential cost of positional fidelity in certain operations. </details> Figure 8. Unified v.s. Disaggregated KV Cache. Note that for simplicity, the cache blocks represent the storage of both K and V, but RoPE is only applied to K prior to caching. ### 5.2. Tree-Structured Cache with Fork Semantics DualRadixTree Architecture. Managing a disaggregated KV cache requires a novel memory abstraction because traditional unified pools fail to accommodate the distinct sizes and lifecycles of shared and unique memory components. To bridge this gap and efficiently manage the memory space of concurrent agents, we introduce the DualRadixTree architecture that physically decouples the unified KV cache into a globally shared bCache and an agent-specific rCache, as shown in Figure 9. The bCache is managed by a base RadixTree where the search key $Key_base$ is defined strictly by the sequence of token ids. This design guarantees that all concurrent requests possessing shared contexts can access the same underlying memory in the bCache via zero-copy mechanisms. In parallel with the base structure, we deploy a residual RadixTree to index the rCache and manage the distinct generation branches of individual agents. Because the rCache is unique to each agent, the search key $Key_res$ in this residual tree extends the traditional token sequence with a specific agent id. Through this decoupled structural design, the DualRadixTree provides the exact memory abstraction required to materialize the disaggregated KV cache paradigm. By maintaining a unified logical memory view for each agent, this data structure successfully eliminates the need to allocate a full-sized KV cache for every agent as traditional prefix caching requires. <details> <summary>2604.06370v1/x8.png Details</summary> ![9fb7a738](/v1/image/9fb7a73893b320cfb8d154142f454c50af4f57c166086400ad0730809f51879a) ### Visual Description ## Diagram: Agent Memory Management with RadixTree Structures ### Overview The diagram illustrates a two-step process for managing agent memory using RadixTree structures and agent-specific caching. It shows how a new agent inherits shared knowledge (base RadixTree) and maintains unique residual data (residual RadixTree) within its memory space. ### Components/Axes 1. **Left Section (Base RadixTree)** - **Title**: "RadixTree (base)" - **Key Definition**: `Key_base = (token_id)` - **Nodes**: - Root: "LLMs" - Children: "can" → ["help", "study"], "are" → ["useful", "smart"] - **Color**: Light green background 2. **Right Section (Residual RadixTree)** - **Title**: "RadixTree (residual)" - **Key Definition**: `Key_res = (token_id, agent_id)` - **Nodes**: - Root: "LLMs" - Children: "can" → "help", "are" → ["useful", "smart"] - **Color**: Light blue background with yellow-highlighted nodes 3. **Agent Memory Space** - **Structure**: - `bCache` (shared across agents, gray) - `rCache` (unique to each agent, white) - **Flow**: Arrows connect base/residual trees to memory space 4. **Steps** - **Step 1**: "Inherit (Extend if miss)" (dashed arrow from base tree to memory) - **Step 2**: "Copy-on-Write" (dashed arrow from residual tree to memory) ### Detailed Analysis - **Base RadixTree**: - Shared knowledge structure (`Key_base = token_id`) - Contains general LLM capabilities ("can help/study", "are useful/smart") - Green color indicates shared/universal data - **Residual RadixTree**: - Agent-specific knowledge structure (`Key_res = token_id + agent_id`) - Contains personalized LLM capabilities (highlighted nodes) - Blue color with yellow highlights indicates agent-specific data - **Memory Space**: - Combines shared (`bCache`) and unique (`rCache`) components - Gray/white color scheme visually separates shared vs. unique data ### Key Observations 1. **Hierarchical Knowledge Organization**: - General knowledge (base tree) is inherited first - Agent-specific knowledge (residual tree) is added second 2. **Efficient Memory Management**: - Shared data (bCache) reduces redundancy - Unique data (rCache) maintains agent individuality 3. **Color-Coded Differentiation**: - Green (base) vs. Blue (residual) clearly separates data types - Yellow highlights in residual tree emphasize agent-specific nodes ### Interpretation This architecture demonstrates a **copy-on-write inheritance pattern** for agent memory: 1. New agents start with shared knowledge (base RadixTree) 2. As agents interact, unique data is added to residual RadixTree 3. Memory space combines both through copy-on-write operations The diagram suggests an **optimized knowledge management system** where: - Common capabilities are stored once (base tree) - Agent-specific adaptations are stored separately (residual tree) - Memory access follows a priority order: shared data first, then unique data The "Copy-on-Write" step implies that modifications to agent-specific data don't affect the shared base, maintaining data integrity while allowing personalization. This pattern is particularly valuable in multi-agent systems where memory efficiency and individual agent adaptation are critical. </details> Figure 9. DualRadixTree Architecture. A new agent logically forks its memory space from former agents by inheriting the shared bCache (Step1: Prefix Match) and exclusively allocating the rCache (Step2: Copy-on-Write). Fork Semantics with CoW. To orchestrate this decoupled memory architecture for incoming agent requests, ForkKV introduces an OS-inspired fork semantics with CoW, as demonstrated by Figure 9. When a new agent is launched, the system first queries the base RadixTree to perform prefix matching to identify the longest shared context. If input tokens miss the existing prefix, the system dynamically extends the tree by allocating new shared memory blocks. The agent then inherits this context by mapping the globally shared bCache into its logical memory space. This process is analogous to a newly forked OS process mapping the read-only physical pages of its parent. Following this inheritance, the system executes a CoW operation to allocate exclusive memory blocks for the rCache. These unique allocations are tracked within the residual RadixTree to store an isolated state for the agent, functioning analogously to the private CoW pages of a child process. By orchestrating this explicit two-step allocation, our fork semantics establish a dynamic management mechanism for the logical memory space of agents, translating theoretical memory decoupling into a practical system design alongside the DualRadixTree architecture. Decoupled Eviction Policy. In traditional prefix caching, the monolithic memory layout enforces a rigid eviction process: a cached sequence is evicted as an indivisible unit, resulting in a binary state of either a complete cache hit or a complete miss. However, directly applying this eviction policy to the disaggregated KV cache creates a strict architectural mismatch because the bCache and rCache exhibit fundamentally different memory footprints and access frequencies. A cascading eviction mechanism would couple these two memory pools, forcing a low-contention pool to discard active cache, thereby triggering entirely avoidable recomputation. To eliminate this redundancy, ForkKV introduces a decoupled eviction policy that isolates the lifecycle management of the base and residual caches by assigning independent Least Recently Used (LRU) states to each radix tree. This architectural isolation provides critical flexibility under heavy memory pressure. If a massive bCache node is evicted while its lightweight rCache counterpart persists, the scheduler avoids treating subsequent requests as complete cache misses. Instead, the system executes a partial hit where the execution engine recomputes only the missing base projection $xW$ , reinserts it into the base tree, and directly reuses the surviving $xA_i$ from the residual tree. This graceful degradation maximizes the overall cache hit rate for complex multi-round agent workflows. ### 5.3. ResidualAttention Computing attention scores accurately requires reconstructing the KV cache from its disaggregated components. To minimize HBM memory and computational overhead, we fuse KV cache reconstruction directly into the attention computation within the SRAM. Driven by this architectural choice, we implement ResidualAttention, an attention kernel specifically tailored for the disaggregated KV cache architecture. As illustrated in Algorithm 1, our approach executes in three distinct stages. The first stage conducts on-the-fly Key cache reconstruction by streaming block-wise tiles and applying deferred RoPE operations. Next, the kernel computes attention scores independently for the base and residual components. Finally, the third phase leverages matrix associativity to fuse the attention output. By executing these steps entirely on-chip, ResidualAttention maps the decoupled cache layout into a high-throughput parallel execution model. The following subsections detail the core algorithm and architectural innovations driving this design. On-the-fly Key Reconstruction with Deferred RoPE. The left portion of Figure 10 illustrates the cache reconstruction and attention computation process during the first stage. Our kernel streams the decoupled base cache $K_base$ and residual cache $K_res$ directly into the fast on-chip SRAM in a block-wise manner. Once inside the SRAM, the kernel reconstructs the complete cache on the fly. This reconstruction proceeds in two steps. First, we reconstruct the full-size LoRA Key cache $K_lora$ derived from $xA_iB_i$ . We perform a LoRA up-projection using $K_resB_k$ and then apply RoPE to the resulting intermediate state. As detailed in Section 5.1, RoPE on rCache is deferred during the earlier linear projection phase due to an output dimension mismatch. Applying it at this stage equips $K_lora$ with the exact token position information. Second, we materialize the combined Key cache using $K_base$ and the processed residual components $K_lora$ . The kernel then computes the attention logits $QK^T$ and performs subsequent operations like the online softmax update. This integrated approach successfully avoids memory-expensive cache reconstruction operation in HBM and inefficient sequential in-place cache updates. Simultaneously, it guarantees the accuracy of the disaggregated KV cache transformation by ensuring the correct preservation of positional encodings. Fusing Attention Scores via Matrix Associativity. After obtaining the attention logits, a straightforward approach to compute the final output $∑ sm(QK^T)V$ is to reconstruct the Value cache $V_base+V_resB_v$ inside the inner loop responsible for iterating over sequence blocks. This approach mirrors the reconstruction mechanism used for the Key cache. However, this eager projection method introduces prohibitive computational overhead and memory footprint. Executing the LoRA up-projection operation $V_resB_v$ at every loop step not only introduces significant redundant computation, but also demands large SRAM capacity to store this intermediate tensor, which inevitably triggers SRAM contention and severely degrades GPU parallelism. <details> <summary>2604.06370v1/x9.png Details</summary> ![b31c8ca9](/v1/image/b31c8ca911037ebdb0fed5bfd21c81bb8b9e4ec123d33c143d7863b06ef64c1b) ### Visual Description ## Diagram: Attention Mechanism with LoRA Integration ### Overview The diagram illustrates a two-stage attention computation process combining base attention and low-rank adaptation (LoRA) components. It shows data flow through outer and inner loops, with explicit memory storage and computation steps. ### Components/Axes **Left Panel (Outer Loop):** - **Inputs:** - `Q`: Query matrix (N×d) - `K_base`: Base key matrix (d×N) - `V_base`: Base value matrix (N×d) - `K_res`: Residual key matrix (r×N) - **Operations:** - `K_lora = RoPE(K_res B_k)` (LoRA-updated keys) - `K = K_base + K_lora` (Combined keys) - `QK^T`: N×N attention score matrix - **Output:** - `acc_r`: Residual attention component (N×r) **Right Panel (Inner Loop):** - **Inputs:** - `acc_r`: Residual attention (N×r) - `V_res`: Residual value matrix (N×r) - **Operations:** - `acc = sm(QK^T)V_base` (Base attention output) - `acc_r = sm(QK^T)V_res` (Residual attention output) - **Final Output:** - `O = acc + acc_r × B_v` (Combined attention result) **Legend/Color Coding:** - Green blocks: Matrix dimensions - Orange blocks: Computation operations - Blue arrows: Data flow - Red dashed lines: Attention score propagation ### Detailed Analysis 1. **Memory Optimization:** - Base matrices (`K_base`, `V_base`) stored in SRAM - Residual matrices (`K_res`, `V_res`) processed in inner loop - LoRA adaptation applied via `RoPE(K_res B_k)` 2. **Attention Computation:** - Outer loop computes base attention scores (`QK^T`) - Inner loop calculates residual attention via `sm(QK^T)V_res` - Final output combines base (`acc`) and residual (`acc_r`) components 3. **Dimensionality:** - Base matrices: N×d (standard attention) - Residual matrices: N×r (low-rank adaptation) - Final output: N×d (matches input dimension) ### Key Observations - **Efficiency:** Residual matrices (r×N) are smaller than base matrices (d×N), reducing memory requirements - **Modularity:** Base and residual components processed separately but combined in final output - **LoRA Integration:** `K_lora` and `V_res` represent low-rank adaptations to base matrices - **Memory Access Pattern:** Explicit storage of blocks suggests hardware-aware optimization ### Interpretation This diagram demonstrates a hybrid attention mechanism that: 1. Maintains standard transformer attention (`QK^T` computation) 2. Adds low-rank adaptations via LoRA (`K_lora`, `V_res`) 3. Optimizes memory usage through block storage and separate processing loops 4. Preserves output dimensionality while enabling parameter-efficient fine-tuning The architecture suggests a design for efficient large language model inference/training, where: - Base attention handles general context - LoRA adaptations capture task-specific modifications - Separate computation paths enable parallel processing - Memory optimization reduces hardware requirements The use of `RoPE` (Rotary Positional Encoding) on residual keys indicates positional awareness in the low-rank adaptations. The final output combines both base and adapted attention through element-wise addition and scaling by `B_v`, maintaining compatibility with standard transformer architectures while enabling efficient parameter updates. </details> Figure 10. Illustration of ResidualAttention. Algorithm 1 ResidualAttention 0: Query blocks $Q$ 0: bCache blocks $(K_base,V_base)$ , rCache blocks $(K_res,V_res)$ 0: LoRA up-projection weights $B_k,B_v$ 1: Initialize: $acc←0_M× D_{v}$ , $acc_r←0_M× R$ 2: Initialize states: $m←-∞$ , $l← 0$ 3: Load $B_k$ , $B_v$ to SRAM 4: for each block $n$ in sequence do 5: // Stage1: On-the-fly Key Reconstruction with Deferred RoPE 6: Load $Q$ , $K_base$ , $V_base$ , $K_res$ , $V_res$ to SRAM 7: Load cached sin-cos table for RoPE to SRAM 8: $K_lora←RoPE(K_res· B_k)$ 9: $K← K_base+K_lora$ 10: // Stage2: Compute Separate Attention Scores (base/residual) 11: $S← Q· K^T·scale$ 12: $m_new←\max(m,\max(S))$ 13: $l_new← l·\exp(m-m_new)+∑\exp(S-m_new)$ 14: $P←\exp(S-m_new)$ 15: $acc←acc·\exp(m-m_new)+P· V_base$ 16: $acc_r←acc_r·\exp(m-m_new)+P· V_res$ 17: $m← m_new$ , $l← l_new$ 18: end for 19: // Stage3: Fuse Attention Scores via Matrix Associativity 20: $acc_final←acc+acc_r· B_v$ 21: Write $O←acc_final/l$ to HBM 22: return $O$ To resolve these inefficiencies, we decouple the attention computation for the base and residual components, and then fuse their partial outputs at the very end of the kernel execution. The right portion of Figure 10 illustrates both this decoupled computation and delayed projection process. This strict mathematical equivalence is guaranteed by the associativity of matrix multiplication: $$ \displaystyle∑ sm(QK^T)V \displaystyle=∑ sm(QK^T)·(V_base+V_resB_v) \displaystyle=∑ sm(QK^T)V_base+(∑ sm(QK^T)V_res)B_v \tag{4} $$ where $V_base$ and $V_res$ denote the base and residual Value cache respectively, and $B_v$ is the LoRA up-projection weight matrix. We push the $B_v$ multiplication entirely out of the inner loop and compute this up-projection only once at the end of the kernel. To support the decoupled attention computation, the kernel maintains a lightweight global accumulator for residual attention scores alongside the base cache. By leveraging this approach, we drastically minimize both computational overhead and the SRAM allocation. ## 6. Implementation We implement ForkKV on top of SGLang (v0.5.6) with approximately 3K lines of Python code and custom Triton kernels. Our implementation introduces system-wide modifications across the control plane, model executor, and kernel layers to natively support the disaggregated KV cache layout. Disaggregated KV Cache. To support the physical decoupling of the KV cache at runtime, we design a custom LoRA replacement module for the linear projection layer. This module separates the adapter-specific residual activations from the base model activations and store these residual activations into a dedicated KV cache pool indexed by the residual RadixTree. Control Plane and DualRadixTree Storage. We extend the native RadixCache of SGLang into a coordinated DualRadixTree architecture to manage the decoupled memory layout. We also adapt the scheduler to orchestrate this two-tiered KV cache pool across phases like chunked prefill, non-chunked prefill, and decode. ResidualAttention. We implement this custom hardware-aware attention kernel in Triton. This kernel is adapted from the RadixAttention kernel of SGLang and have two separate versions for prefill and decode. This structural separation allows the attention mechanism to cater specifically to the distinct memory access patterns and workload characteristics of each execution phase. ## 7. Evaluation ### 7.1. Setup Models and Hardware Settings. We evaluate ForkKV on three open-source large language models: Llama3-8B (AI@Meta, 2024), Qwen2.5-7B (Team, 2024), and Qwen2.5-14B (Team, 2024). All models are deployed in BF16 precision. We conduct our end-to-end experiments across two hardware platforms: a server equipped with a single L40 GPU and 128 vCPUs, and a server containing two RTX 5000 GPUs with 48 vCPUs. Specifically, Llama3-8B is served on the L40 GPU, while Qwen2.5-7B and Qwen2.5-14B are evaluated on one and two RTX 5000 GPUs respectively. System Performance Benchmark. To evaluate the serving throughput and efficiency of ForkKV, we benchmark under two representative agentic execution patterns: - ReAct (Yao et al., 2022): An iterative reasoning and acting paradigm where the context window grows sequentially. We use it to evaluate KV cache management efficiency during sustained, multi-turn agent workloads. - MapReduce (Luo et al., 2025): A parallel processing workflow that divides a large task into multiple concurrent subtasks. We use it to stress-test ForkKV ’s memory sharing capability when handling massive simultaneous forks from a single shared context. <details> <summary>2604.06370v1/figures/e2e.png Details</summary> ![2f0f549e](/v1/image/2f0f549e430ffafaa551329476076c0391be14b1ecd283e68f4435c7eca9ea6d) ### Visual Description ## Bar Chart: Throughput Comparison Across Caching Strategies and Models ### Overview The image contains six grouped bar charts comparing throughput (tasks/s) for three caching strategies (ForkKV, Prefix Caching with SGLang, Prefix Caching with vLLM) across six task-model combinations. The charts are divided into two sections: "React" (top row) and "MapReduce" (bottom row), with three models (LooGLE, NarrativeQA, APIGen) and three base models (Llama3-8B, Qwen2.5-7B, Qwen2.5-14B) as subcategories. ### Components/Axes - **X-axis**: Task categories (LooGLE, NarrativeQA, APIGen) - **Y-axis**: Throughput (tasks/s) with two subplots: - Top: React throughput - Bottom: MapReduce throughput - **Legends**: - Green: ForkKV (ours) - Teal: Prefix Caching (SGLang) - Dark Blue: Prefix Caching (vLLM) - **Subheadings**: Model variants (Llama3-8B, Qwen2.5-7B, Qwen2.5-14B) positioned above each chart ### Detailed Analysis #### React Throughput (Top Row) 1. **Llama3-8B**: - ForkKV: ~0.12 tasks/s - SGLang: ~0.06 tasks/s - vLLM: ~0.05 tasks/s 2. **NarrativeQA**: - ForkKV: ~0.08 tasks/s - SGLang: ~0.04 tasks/s - vLLM: ~0.03 tasks/s 3. **APIGen**: - ForkKV: ~0.05 tasks/s - SGLang: ~0.03 tasks/s - vLLM: ~0.02 tasks/s #### MapReduce Throughput (Bottom Row) 1. **Llama3-8B**: - ForkKV: ~0.04 tasks/s - SGLang: ~0.02 tasks/s - vLLM: ~0.01 tasks/s 2. **NarrativeQA**: - ForkKV: ~0.06 tasks/s - SGLang: ~0.03 tasks/s - vLLM: ~0.02 tasks/s 3. **APIGen**: - ForkKV: ~0.03 tasks/s - SGLang: ~0.01 tasks/s - vLLM: ~0.005 tasks/s #### Qwen2.5-7B (Middle Column) - React: - ForkKV: ~0.15 tasks/s - SGLang: ~0.09 tasks/s - vLLM: ~0.07 tasks/s - MapReduce: - ForkKV: ~0.07 tasks/s - SGLang: ~0.04 tasks/s - vLLM: ~0.03 tasks/s #### Qwen2.5-14B (Right Column) - React: - ForkKV: ~0.18 tasks/s - SGLang: ~0.07 tasks/s - vLLM: ~0.06 tasks/s - MapReduce: - ForkKV: ~0.05 tasks/s - SGLang: ~0.03 tasks/s - vLLM: ~0.02 tasks/s ### Key Observations 1. **ForkKV Dominance**: Consistently outperforms both Prefix Caching methods across all tasks and models. 2. **Model Size Impact**: Larger models (Qwen2.5-14B) show higher absolute throughput than smaller models (Llama3-8B). 3. **Task Dependency**: React tasks generally achieve higher throughput than MapReduce tasks for the same model. 4. **Caching Strategy Tradeoffs**: - SGLang outperforms vLLM in React for Qwen2.5-7B (0.09 vs 0.07) - vLLM shows better MapReduce performance for Qwen2.5-14B (0.02 vs 0.01) ### Interpretation The data suggests ForkKV's architecture provides superior throughput across all scenarios, likely due to its optimized design. Prefix Caching performance varies significantly by model size and task type: - **SGLang** excels in React tasks with mid-sized models (Qwen2.5-7B) - **vLLM** shows unexpected MapReduce efficiency with the largest model (Qwen2.5-14B) - The 3x throughput gap between React and MapReduce tasks for ForkKV indicates task-specific optimizations in the framework. These results highlight the importance of caching strategy selection based on both model architecture and task requirements, with ForkKV emerging as the most universally effective solution in this comparison. </details> Figure 11. End-to-end throughput evaluation. The figure compares the serving throughput (tasks/s) of ForkKV against prefix caching baselines on ReAct and MapReduce workloads across three models and three datasets. To simulate the scenarios where concurrent agents operate on the same context and execute different tasks, we synthesize the model input using two components: a massive static part shared across all agents and a dynamic part containing task-specific instructions. We construct these two-stage inputs using three long-context datasets, as summarized in Table 1. LooGLE (Li et al., 2023) features long documents from various sources such as arXiv, Wikipedia, and movie/TV scripts. In our evaluation, we leverage the movie/TV scripts as the static part and use the corresponding questions as the dynamic part. NarrativeQA (Kočiský et al., 2018) is a widely acknowledged long-context dataset for testing reading comprehension capabilities, featuring documents even longer than those in LooGLE. We follow a similar methodology here by using the narrative document as the static part and the comprehension question as the dynamic part. Finally, APIGen (Liu et al., 2024d) is a tool-calling dataset designed to evaluate instruction-following and structured output capabilities. We aggregate the available tool descriptions to construct the large shared static context and use the specific instructions as the dynamic inputs. Furthermore, we configure 8 agent workflows for both the ReAct and MapReduce paradigms, where baselines face severe performance degradation. In these workflows, each individual agent utilizes a distinct LoRA adapter with a rank of 16, following the setting in prior works (Zhu et al., 2025a; Chen et al., 2024). Figure 2 illustrates the underlying architecture of these workflows. We generate continuous requests with an average arrival rate of 2 requests per second to evaluate the system under sustained heavy load. Within each workflow loop, we simulate agent-tool interactions by injecting a constant latency of 0.1 seconds and returning a mock tool response of 100 random tokens. For the agent generation step, we set the maximum output length to 256 tokens. | Static Context Avg. Dynamic Instr | 32742 24 | 49119 12 | 64911 23 | | --- | --- | --- | --- | Table 1. The length of shared static context and task-specific dynamic instructions (average) in the sampled datasets. System Performance Baselines. We compare ForkKV against two baselines in the system performance benchmark: - vLLM (Kwon et al., 2023): A state-of-the-art serving engine that utilizes PagedAttention for efficient KV cache memory management. It also implements prefix caching for efficient KV cache reuse. We use vLLM v0.12.0 as our baseline. - SGLang (Zheng et al., 2024): Another state-of-the-art framework featuring RadixAttention, which enables automatic KV cache reuse for shared-prefix requests. We use SGLang v0.5.6 as our baseline. We select SGLang and vLLM as our baselines because they provide state-of-the-art LLM serving capabilities. Both frameworks integrate a comprehensive set of performance acceleration features, ranging from widely adopted mechanisms like prefix caching to advanced techniques such as CUDA Graphs and asynchronous scheduling. While dedicated LoRA serving systems (Sheng et al., 2023; Chen et al., 2024; Wu et al., 2024b) also provide specialized LoRA serving capabilities, they generally lack full support for these system-level optimizations. <details> <summary>2604.06370v1/figures/e2e-parallelism.png Details</summary> ![bc2a11f2](/v1/image/bc2a11f290d8092dd5be1645b92931c31bdf906d729df2675781065705668bba) ### Visual Description ## Bar Chart: Throughput Comparison Across Caching Strategies ### Overview The image presents a grouped bar chart comparing throughput (tasks/second) for three caching strategies - ForkKV (ours), Prefix Caching (SGLang), and Prefix Caching (vLLM) - across two applications (ReAct and MapReduce) at different numbers of concurrent workflows (4, 8, 16). The chart uses three distinct colors for the strategies: green for ForkKV, teal for SGLang, and dark blue for vLLM. ### Components/Axes - **X-axis**: Number of Concurrent Workflows (4, 8, 16) - **Y-axis**: Throughput (tasks/second) with different scales: - ReAct: 0.00-0.36 - MapReduce: 0.00-0.06 - **Legend**: Located at the top, with color-coded labels: - Green: ForkKV (ours) - Teal: Prefix Caching (SGLang) - Dark Blue: Prefix Caching (vLLM) ### Detailed Analysis #### ReAct Application - **4 workflows**: - ForkKV: ~0.18 tasks/s - SGLang: ~0.35 tasks/s - vLLM: ~0.34 tasks/s - **8 workflows**: - ForkKV: ~0.12 tasks/s - SGLang: ~0.06 tasks/s - vLLM: ~0.05 tasks/s - **16 workflows**: - ForkKV: ~0.09 tasks/s - SGLang: ~0.04 tasks/s - vLLM: ~0.03 tasks/s #### MapReduce Application - **4 workflows**: - ForkKV: ~0.07 tasks/s - SGLang: ~0.025 tasks/s - vLLM: ~0.025 tasks/s - **8 workflows**: - ForkKV: ~0.045 tasks/s - SGLang: ~0.015 tasks/s - vLLM: ~0.012 tasks/s - **16 workflows**: - ForkKV: ~0.015 tasks/s - SGLang: ~0.01 tasks/s - vLLM: ~0.01 tasks/s ### Key Observations 1. **ForkKV Dominance**: ForkKV consistently outperforms both SGLang and vLLM across all workflow counts in both applications. 2. **Scalability Pattern**: Throughput decreases as workflow count increases, but ForkKV maintains a higher throughput margin. 3. **SGLang vs vLLM**: The two prefix caching strategies show similar performance, with SGLang slightly outperforming vLLM in ReAct at 4 workflows. 4. **Application-Specific Performance**: ReAct shows significantly higher throughput values than MapReduce across all configurations. ### Interpretation The data suggests ForkKV demonstrates superior efficiency in handling concurrent workflows compared to traditional prefix caching strategies. The performance gap between ForkKV and others widens as workflow count increases, indicating better scalability. The minimal difference between SGLang and vLLM suggests similar underlying limitations in prefix caching approaches. The application-specific performance disparity (ReAct vs MapReduce) may reflect differences in task complexity or resource requirements between the two frameworks. </details> Figure 12. Performance under varying number of workflows with Llama3-8B on the LooGLE dataset. Every individual agent in different workflows uses different LoRA adapters. Generation Quality Benchmark. To verify that ForkKV introduces negligible accuracy loss, we evaluate the generation quality using the following datasets and metrics: - HotpotQA (Yang et al., 2018): This is a multi-hop question answering dataset. It’s intended to evaluate the model’s complex reasoning and information integration capabilities across multiple contexts. We include 200 test cases. - APIGen (Liu et al., 2024d): In addition to system benchmarking, we utilize 200 test cases from APIGen to benchmark the model’s instruction-following and structured API calls generation capabilities. We adopt F1-score (Labs, ), a metric that measures the similarity between the model’s output and the ground-truth answer of the question based on the number of overlapping words. Generation Quality Baselines. We compare ForkKV against two KV cache sharing policies in the generation quality benchmark: - Prefix Caching (Kwon et al., 2023; Zheng et al., 2024): This policy strictly reuses the KV cache of an identical context only when the requests share the exact same LoRA adapter. It is mathematically lossless and serves as the accuracy upper bound for our evaluation. - Full Reuse: This policy aggressively reuses the KV cache of a shared context across different LoRA adapters, entirely ignoring the distinct activations introduced by each adapter. We include this to establish a baseline for the accuracy drop that ForkKV aims to mitigate. <details> <summary>2604.06370v1/figures/qps.png Details</summary> ![d038b4ae](/v1/image/d038b4ae5b0954a1e9f752e212fa326aee30ceb08606db429bc6daa8a4f1ef4d) ### Visual Description ## Line Graphs: Throughput vs. Request Rate for Caching Strategies ### Overview The image contains two line graphs comparing throughput (tasks/second) against request rate (requests/second) for three caching strategies: **ForkKV (ours)**, **Prefix Caching (SGLang)**, and **Prefix Caching (vLLM)**. The graphs are labeled **ReAct** (left) and **MapReduce** (right). Each graph shows performance trends across request rates from 0.00 to 0.50 requests/second. --- ### Components/Axes - **X-axis**: Request Rate per Second (0.00 to 0.50 in increments of 0.05). - **Y-axis**: Throughput (tasks/second) with ranges: - ReAct: 0.00 to 0.15 - MapReduce: 0.00 to 0.06 - **Legend**: - Green: ForkKV (ours) - Teal: Prefix Caching (SGLang) - Dark Blue: Prefix Caching (vLLM) - **Graph Titles**: - Left: "ReAct" - Right: "MapReduce" --- ### Detailed Analysis #### ReAct Graph - **ForkKV (Green)**: - Starts at ~0.05 tasks/s at 0.00 requests/s. - Sharp increase to ~0.10 tasks/s at 0.10 requests/s. - Plateaus at ~0.11 tasks/s from 0.25 to 0.50 requests/s. - **Prefix Caching (SGLang, Teal)**: - Starts at ~0.02 tasks/s at 0.00 requests/s. - Gradual rise to ~0.04 tasks/s at 0.25 requests/s. - Flattens at ~0.04 tasks/s at 0.50 requests/s. - **Prefix Caching (vLLM, Dark Blue)**: - Starts at ~0.01 tasks/s at 0.00 requests/s. - Slightly higher than SGLang at all points (~0.03–0.04 tasks/s). #### MapReduce Graph - **ForkKV (Green)**: - Starts at ~0.01 tasks/s at 0.00 requests/s. - Sharp rise to ~0.04 tasks/s at 0.10 requests/s. - Slight decline to ~0.03 tasks/s at 0.50 requests/s. - **Prefix Caching (SGLang, Teal)**: - Starts at ~0.01 tasks/s at 0.00 requests/s. - Gradual rise to ~0.02 tasks/s at 0.25 requests/s. - Flattens at ~0.02 tasks/s at 0.50 requests/s. - **Prefix Caching (vLLM, Dark Blue)**: - Starts at ~0.01 tasks/s at 0.00 requests/s. - Slightly higher than SGLang at all points (~0.02–0.03 tasks/s). --- ### Key Observations 1. **ForkKV Dominates**: - In both workloads, ForkKV consistently achieves higher throughput than Prefix Caching strategies. - In ReAct, ForkKV maintains a ~2–3x advantage over Prefix Caching at higher request rates. - In MapReduce, the gap narrows but ForkKV still outperforms (~1.5x at 0.50 requests/s). 2. **Performance Trends**: - **ReAct**: ForkKV plateaus at ~0.11 tasks/s, suggesting scalability limits. Prefix Caching strategies show minimal improvement beyond 0.25 requests/s. - **MapReduce**: ForkKV’s throughput declines slightly at higher request rates, while Prefix Caching strategies remain flat. 3. **Request Rate Sensitivity**: - ForkKV’s performance improves sharply with request rate in both workloads, indicating better concurrency handling. - Prefix Caching strategies show diminishing returns as request rates increase. --- ### Interpretation - **ForkKV’s Advantage**: The caching strategy labeled "ForkKV (ours)" demonstrates superior throughput efficiency, likely due to optimized resource utilization or reduced contention. This is critical for high-throughput systems. - **Workload Differences**: - In ReAct, ForkKV’s plateau suggests it may struggle with extremely high request rates, while Prefix Caching strategies remain stable but underperform. - In MapReduce, ForkKV’s slight decline at 0.50 requests/s hints at potential overhead at peak loads, whereas Prefix Caching strategies remain consistent but less efficient. - **Practical Implications**: - ForkKV is preferable for workloads requiring high throughput (e.g., ReAct). - Prefix Caching (vLLM) may be more stable under moderate loads but lacks scalability. - **Anomalies**: - The sharp rise in ForkKV’s ReAct graph at 0.10 requests/s suggests a threshold effect, possibly due to parallelization or caching efficiency. - The decline in ForkKV’s MapReduce graph at 0.50 requests/s may indicate resource contention or algorithmic limitations at extreme scales. This analysis underscores the importance of caching strategy selection based on workload characteristics and system requirements. </details> Figure 13. Performance under different requests arrival rate with Llama3-8B on the LooGLE dataset. ### 7.2. End-to-End Results Throughput Improvement. Figure 11 compares the end-to-end throughput of ForkKV against prefix caching baselines across three datasets and three models under the ReAct and MapReduce paradigms. Overall, ForkKV achieves 1.25 $×$ to 3.04 $×$ the throughput of the baselines on ReAct workflows, and 1.68 $×$ to 2.60 $×$ the throughput on MapReduce workflows. These consistent speedups demonstrate the scalability of ForkKV in handling both sequential and concurrent agent workflows. Beyond the overall gains, we draw several key insights from different system configurations. First, ForkKV yields the most significant speedups in scenarios with severe memory contention. For instance, ForkKV achieves 3.04 $×$ the baseline throughput on the larger Qwen2.5-14B model but only 1.25 $×$ on the smaller Qwen2.5-7B model using the LooGLE dataset under ReAct workflow. The smaller model requires less memory for base weights and the KV cache. This reduced memory footprint makes the memory savings from ForkKV less prominent while amplifying the computational overhead of the disaggregated KV cache design. Second, ForkKV maintains robust performance gains across datasets with varying sequence lengths. Although the throughput values vastly differ across datasets, the relative speedups on LooGLE, NarrativeQA, and APIGen remain stable at 1.20 $×$ , 1.32 $×$ , 1.09 $×$ respectively. This stability confirms that ForkKV remain highly effective regardless of task type or sequence length. Performance under Varying Number of Workflows. Figure 12 illustrates the throughput of ForkKV compared to prefix caching baselines as the number of concurrent agent workflows scales. ForkKV initially exhibits lower throughput than the baselines under a light load of 4 ReAct workflows. At this low concurrency level, the KV cache consumption of active agents remains small relative to the available GPU memory. Standard prefix caching can therefore retain all KV cache blocks without triggering evictions, preserving high baseline performance by avoiding costly recomputations. In contrast, ForkKV introduces disaggregated KV cache with dedicated architectural and kernel support to reduce the per-agent memory footprint. However, this specialized design incurs noticeable computational overhead when GPU memory is abundant, which accounts for the initial performance drop. This limitation can be mitigated by adaptive scheduling, which monitors GPU memory utilization and dynamically falls back to standard KV cache when memory is abundant. Despite the initial performance gap, the architectural design of ForkKV yields significant benefits as the number of concurrent workflows scales and memory demand exceeds hardware capacity. Under these high-contention conditions, ForkKV achieves 1.84-2.33 $×$ and 1.31-2.51 $×$ the throughput of the baselines on ReAct and MapReduce paradigms respectively. These results confirm that ForkKV successfully delivers high performance during heavy memory contention. <details> <summary>2604.06370v1/figures/principle.png Details</summary> ![3ba0abf1](/v1/image/3ba0abf1db8a03b3d417d703b2131652e55a7b8089d5603eb0408a6c67de0bc6) ### Visual Description ## Bar Chart Comparison: ForkKV vs Prefix Caching Performance Metrics ### Overview The image presents three side-by-side bar charts comparing two caching systems: ForkKV (green) and Prefix Caching (blue). Each chart measures different performance metrics across the same two implementations. ### Components/Axes 1. **Legend**: - Position: Top center - Labels: - ForkKV (ours) - Green - Prefix Caching (SGLang) - Blue 2. **Chart (a)**: - Title: Not explicitly labeled - Y-axis: "Avg Per-Agent Mem Usage (GB)" (0-4 scale) - X-axis: Implicit categories (ForkKV vs Prefix Caching) 3. **Chart (b)**: - Title: Not explicitly labeled - Y-axis: "Cache Hit Rate" (0-1 scale) - X-axis: Implicit categories (ForkKV vs Prefix Caching) 4. **Chart (c)**: - Title: Not explicitly labeled - Y-axis: "Avg Decode Batch Size" (0-60 scale) - X-axis: Implicit categories (ForkKV vs Prefix Caching) ### Detailed Analysis **Chart (a) - Memory Usage** - ForkKV: ~0.3 GB (bottom bar) - Prefix Caching: ~4.0 GB (top bar) - Spatial grounding: Blue bar occupies 95% of vertical space **Chart (b) - Cache Hit Rate** - ForkKV: ~0.9 (top bar) - Prefix Caching: ~0.1 (bottom bar) - Spatial grounding: Green bar occupies 90% of vertical space **Chart (c) - Decode Batch Size** - ForkKV: ~60 (top bar) - Prefix Caching: ~5 (bottom bar) - Spatial grounding: Green bar occupies 100% of vertical space ### Key Observations 1. **Memory Efficiency**: ForkKV uses 97% less memory than Prefix Caching (0.3 vs 4.0 GB) 2. **Cache Performance**: ForkKV achieves 9x higher cache hit rate (0.9 vs 0.1) 3. **Processing Capacity**: ForkKV handles 12x larger decode batches (60 vs 5) 4. **Consistent Advantage**: ForkKV outperforms in all three metrics ### Interpretation The data demonstrates ForkKV's superior performance across all measured dimensions: - **Memory Efficiency**: Critical for resource-constrained environments - **Cache Effectiveness**: Indicates better data locality management - **Batch Processing**: Suggests more efficient parallelization The stark contrast in memory usage versus performance gains implies ForkKV optimizes resource allocation more effectively. The consistent dominance across metrics suggests architectural advantages in ForkKV's design, particularly in cache management and parallel processing capabilities. These findings would be valuable for systems requiring high throughput with limited memory resources. </details> Figure 14. Underlying causes of ForkKV performance gains. ForkKV reduces (a) the average per-agent memory usage, which consequently improves (b) the cache hit rate and increases (c) the average decode batch size. Performance under Different Requests Arrival Rate. Figure 13 demonstrates the throughput of ForkKV compared to prefix caching baselines as the request arrival rate increases. ForkKV consistently outperforms the baselines across varying request rates. As the arrival rate scales, standard prefix caching struggles with costly recomputation caused by frequent cache evictions. Conversely, ForkKV manages this increased load efficiently due to the shrinked per-agent memory footprint enabled by disaggregated KV cache. Consequently, ForkKV achieves approximately 2.52 $×$ and 2.05 $×$ the throughput of the baselines during the steady state. These results validate that ForkKV successfully maintains high performance across a wide range of request arrival rates. Understanding ForkKV ’s Improvement. ForkKV achieves higher throughput by significantly reducing the per-agent memory footprint. Specifically, while prefix caching stores a separate KV cache for each LoRA-based agent, ForkKV alleviates this inefficiency by decoupling the KV cache into a shared full-sized bCache and a LoRA-specific lightweight rCache. As Figure 14 a illustrates, this design reduces the average per-agent memory footprint by 12.7 $×$ compared to traditional prefix caching. This drastic reduction in memory consumption translates to performance gains in two distinct ways. First, the freed GPU capacity allows ForkKV to retain more agent contexts within a limited GPU capacity. Consequently, as Figure 14 b shows, the cache hit rate of ForkKV improves by 6.93 $×$ over prefix caching baselines, which directly accelerates generation by reducing the frequency of recomputation. Second, the lower memory footprint per agent enables a 12.0 $×$ larger decode batch size as demonstrated in Figure 14 (c). This expanded batch size allows for greater parallelism during agent serving. In practice, these two advantages hide the overhead introduced by the new architecture and deliver substantial speedups. <details> <summary>2604.06370v1/figures/sensitivity.png Details</summary> ![258be4a1](/v1/image/258be4a1eb06a5fee97d4bea1974714d0c25b16189c78f19b7e0541817ef7e8c) ### Visual Description ## Line Charts: Throughput Comparison of ForkKV vs Prefix Caching ### Overview The image contains two line charts comparing the throughput performance of two systems: ForkKV (green line) and Prefix Caching (blue line). Chart (a) examines performance across LoRA Rank values, while chart (b) analyzes performance across Output Length values. Both charts use throughput (tokens/second) as the y-axis metric. ### Components/Axes **Chart (a): LoRA Rank** - X-axis: LoRA Rank (values: 16, 32) - Y-axis: Throughput (tokens/s) (0-40 scale) - Legend: Top position, labels "ForkKV (ours)" (green) and "Prefix Caching (SGLang)" (blue) - Data points: Circular markers connected by lines **Chart (b): Output Length** - X-axis: Output Length (values: 512, 1024) - Y-axis: Throughput (tokens/s) (0-80 scale) - Legend: Top position, same color coding as chart (a) - Data points: Circular markers connected by lines ### Detailed Analysis **Chart (a) Trends:** 1. **ForkKV (green):** - At LoRA Rank 16: ~30 tokens/s - At LoRA Rank 32: ~25 tokens/s - Trend: Gradual decline (slope: -1.25 tokens/s per rank unit) 2. **Prefix Caching (blue):** - Constant at ~10 tokens/s across both ranks - Trend: Flat line (no change) **Chart (b) Trends:** 1. **ForkKV (green):** - At Output Length 512: ~20 tokens/s - At Output Length 1024: ~70 tokens/s - Trend: Linear increase (slope: +0.033 tokens/s per token) 2. **Prefix Caching (blue):** - At Output Length 512: ~15 tokens/s - At Output Length 1024: ~25 tokens/s - Trend: Gradual increase (slope: +0.05 tokens/s per token) ### Key Observations 1. ForkKV demonstrates superior scalability with longer output lengths (chart b), achieving 3.5x higher throughput at 1024 tokens vs 512 tokens 2. Prefix Caching maintains consistent performance across LoRA ranks but shows diminishing returns with longer outputs 3. ForkKV's throughput advantage grows significantly with increased output length (chart b) 4. Both systems show stable performance at lower LoRA ranks (chart a) ### Interpretation The data suggests ForkKV's architecture is more efficient for longer output generation tasks, with throughput increasing linearly with output length. In contrast, Prefix Caching maintains stable but lower performance across both tested parameters. The performance gap between the two systems widens dramatically at longer output lengths (chart b), indicating ForkKV's potential advantage in handling complex generation tasks. The stable performance of Prefix Caching across LoRA ranks suggests it may be more suitable for simpler, shorter generation tasks where output length is less variable. </details> Figure 15. Sensitivity study of varying LoRA ranks and output lengths with Llama3-8B on the ReAct workflow. | | Prefix Caching | 57.63 | 39.77 | | --- | --- | --- | --- | | Llama3-8B | ForkKV | 57.17 | 38.17 | | Full Reuse | 54.02 | 17.82 | | | Prefix Caching | 57.14 | 92.28 | | | Qwen2.5-7B | ForkKV | 56.37 | 91.52 | | Full Reuse | 55.47 | 90.08 | | | Prefix Caching | 70.91 | 94.56 | | | Qwen2.5-14B | ForkKV | 70.66 | 94.16 | | Full Reuse | 68.86 | 93.66 | | Table 2. Generation quality evaluated with F1-score (%). ### 7.3. Accuracy Verification Section 3.2 suggests that ForkKV introduces negligible degradation in generation quality. To verify this claim, we evaluate ForkKV against standard prefix caching and full KV cache reuse across three distinct models and two datasets. Table 2 summarizes these comparative results. The evaluation shows that ForkKV achieves an accuracy highly comparable to the prefix caching baseline, exhibiting an average quality drop of only 0.71 points across all evaluated settings. Furthermore, the maximum observed decrease is just 1.60 points on Llama3-8B when tested with the APIGen dataset. In contrast, full KV cache reuse suffers from severe performance degradation and incurs an average accuracy drop of 5.40 points. This degradation is particularly obvious on complex tasks like APIGen, where the accuracy of Llama3-8B drops by a substantial 21.95 points from 39.77 to 17.82. By avoiding these extreme performance penalties, ForkKV successfully preserves generation quality across diverse models and tasks while enabling efficient KV cache sharing. ### 7.4. Sensitivity Analysis To have a better understanding of ForkKV, we conduct sensitivity analysis across different configurations with Llama3-8B on the ReAct workflow. Varying LoRA Ranks. To study the impact of LoRA ranks on ForkKV, we modify the underlying LoRA ranks of agents across a representative set $r∈\{8,16,32\}$ (Chen et al., 2024; Sheng et al., 2023; Wu et al., 2024b; Zhu et al., 2025a). Figure 15 a shows ForkKV achieves 2.36-2.88 $×$ the baseline throughput over this setting. Noticeably, the absolute throughput of ForkKV decreases as the rank increases because a larger rank linearly expands the residual cache size, which increases the memory footprint per agent and limits the maximum batch size during inference. However, since small ranks ( $r<64$ ) can already provide high generation quality for various NLP tasks (Hu et al., 2022), ForkKV is highly effective in serving multi-LoRA agents under practical configurations. Varying Output Lengths. We evaluate the system by varying the output length of each agent within the workflow. Figure 15 b shows ForkKV achieves 2.69-3.36 $×$ the baseline throughput across different lengths. Longer agent outputs continuously accumulate newly generated KV cache and severely challenge the system memory capacity. However, as discussed in Section 3.2, ForkKV consistently maintains a low memory footprint per agent. This minimal memory overhead allows our system to absorb the growing memory demands and sustain a strictly larger concurrent batch size than the baseline. Consequently, ForkKV guarantees highly efficient serving performance regardless of the specific generation length. ## 8. Related Work Agentic Workflow Serving. The rapid evolution of autonomous agents has spurred the development of specialized serving systems for complex agent workflows, which mainly optimize via efficient scheduling (Luo et al., 2025; Dai et al., 2025; Lin et al., 2024; Fu et al., 2024; Liu et al., 2025a; Chaudhry et al., 2025) and KV cache routing (Pan et al., 2025; Bian et al., 2025; Wu et al., 2026) for general-purpose LLMs. In contrast, ForkKV targets multi-LoRA agent serving, enabling base cache sharing across distinct agents to minimize memory footprint and alleviate batch size restrictions. System-level Optimization on LoRA Systems. The popularity of LoRA has driven the need for efficient multi-tenant LoRA systems where numerous adapters share a single base model (Chen et al., 2024; Sheng et al., 2023; Iliakopoulou et al., 2025; Wu et al., 2024b; Li et al., 2025; Zhang et al., 2025; Chen et al., 2025; Zhou et al., 2024; Zhu et al., 2025a, b; Ye et al., 2025). Pioneering works like Punica (Chen et al., 2024) introduce custom CUDA kernels to efficiently batch requests across different adapters. Building upon this foundation, subsequent research has explored advanced scheduling policies (Wu et al., 2024b; Chen et al., 2025; Zhu et al., 2025a), memory optimizations (Sheng et al., 2023; Iliakopoulou et al., 2025), GPU kernel optimizations (Zhou et al., 2024; Zhu et al., 2025b; Ye et al., 2025), and offloading techniques (Li et al., 2025; Zhang et al., 2025) for efficient serving. While prior works primarily target chatbot scenarios, ForkKV tackles the severe memory redundancy caused by diverging LoRA activcations in multi-agent workflows through KV cache disaggregation. Copy-on-Write (CoW) in Data Management Systems. Originating as a classic OS-level technique to enable lock-free parallel operations and reduce memory footprints, CoW is widely used to optimize data management systems (Kemper and Neumann, 2011; Wang et al., 2023a; Lakshman et al., 2022; Okolnychyi et al., 2024; Cha et al., 2023; Fruth and Scherzinger, 2024). Inspired by these approaches, ForkKV adapts the CoW mechanism to manage the KV cache for highly-branched shared contexts across agents, effectively extending this paradigm to multi-LoRA agent serving scenarios. KV Cache Optimization. Existing KV cache optimization strategies primarily focus on lossless memory layout improvements (Zheng et al., 2024; Xie et al., 2025; Qianli et al., 2025; Xiong et al., 2024; Qin et al., 2024), lossy compression (Liu et al., 2024c; Yao et al., 2025b), and cross-chunk or cross-model sharing (Gim et al., 2024; Yao et al., 2025a; Liu et al., 2024b). In concurrent work, LRAgent (Jeon et al., 2026) proposes to decompose the KV cache into shared and adapter-dependent components for multi-LoRA agent serving with negligible accuracy loss. Distinct from these approaches, our work uniquely introduces an OS-inspired DualRadixTree for decoupled cache management and an efficient attention kernel fused with deferred RoPE operations. ## 9. Conclusion In this paper, we present ForkKV, a highly efficient multi-LoRA agent serving system that resolves the critical memory footprint bottlenecks caused by KV cache divergence. Inspired by the OS fork primitive with copy-on-write, ForkKV utilizes a DualRadixTree architecture to disaggregate the KV cache into a globally shareable base and lightweight adapter-specific residuals. To make this disaggregated memory layout computationally efficient, we design ResidualAttention, an attention kernel that fuses cache reconstruction directly in SRAM. Our experiments demonstrate that ForkKV achieves up to 3.0 $×$ throughput of state-of-the-art serving systems while preserving generation quality. Acknowledgements. We would like to express our sincere gratitude to Zhanda Zhu from the University of Toronto for his valuable contributions to this research work. His expertise, dedication, and generous support have significantly enhanced the quality of our study. His insightful suggestions and technical assistance were instrumental in achieving our research objectives. We are also grateful to Zhihao Jia from Carnegie Mellon University for his valuable discussions during the early stages of this work. His perspectives and insights helped shape the foundation of our research. ## References - A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee (2023) Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve}. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 117–134. Cited by: §1, §2.1, §3.1. - AI@Meta (2024) Llama 3 model card. External Links: Link Cited by: §1, §2.1, §7.1. - J. Ainslie, J. Lee-Thorp, M. De Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai (2023) Gqa: training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4895–4901. Cited by: §2.1. - Anthropic (2025) External Links: Link Cited by: §1, §2.1. - Anthropic (2026) External Links: Link Cited by: §1. - Z. Bian, F. Wu, T. Ma, and Y. Zhuo (2025) Tokencake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications. arXiv. External Links: 2510.18586, Document Cited by: §8. - H. Cha, X. Hao, T. Wang, H. Zhang, A. Akella, and X. Yu (2023) Blink-hash: an adaptive hybrid index for in-memory time-series databases. Proceedings of the VLDB Endowment 16 (6), pp. 1235–1248. Cited by: §8. - G. I. Chaudhry, E. Choukse, H. Qiu, Í. Goiri, R. Fonseca, A. Belay, and R. Bianchini (2025) Murakkab: resource-efficient agentic workflow orchestration in cloud platforms. External Links: 2508.18298, Link Cited by: §8. - B. Chen, C. Shu, E. Shareghi, N. Collier, K. Narasimhan, and S. Yao (2023) Fireact: toward language agent fine-tuning. arXiv preprint arXiv:2310.05915. Cited by: §1. - L. Chen, Z. Ye, Y. Wu, D. Zhuo, L. Ceze, and A. Krishnamurthy (2024) Punica: multi-tenant lora serving. Proceedings of Machine Learning and Systems 6, pp. 1–13. Cited by: §2.3, §7.1, §7.1, §7.4, §8. - R. Chen, C. Yu, H. Fu, X. Hu, and B. Yang (2025) MixLoRA: An Efficient Multi-Tenant Framework for Concurrently Serving Diverse LoRA Models in Large Language Models. In Proceedings of the 54th International Conference on Parallel Processing, San Diego CA USA, pp. 11–21. External Links: Document, ISBN 979-8-4007-2074-1 Cited by: §8. - Y. Dai, Z. Chen, A. Iyer, and R. Netravali (2025) Aragog: Just-in-Time Model Routing for Scalable Serving of Agentic Workflows. arXiv. External Links: 2511.20975, Document Cited by: §8. - DeepSeek-AI (2024) DeepSeek-v3 technical report. External Links: 2412.19437, Link Cited by: §2.1. - DeepSeek-AI (2025) DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning. External Links: 2501.12948, Link Cited by: §2.1. - T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer (2023) QLoRA: efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314. Cited by: §1. - N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, et al. (2021) A mathematical framework for transformer circuits. Transformer Circuits Thread 1 (1), pp. 12. Cited by: §1, §3.2. - M. Fruth and S. Scherzinger (2024) The case for dbms live patching. Proceedings of the VLDB Endowment 17 (13), pp. 4557–4570. Cited by: §8. - Y. Fu, J. Chen, S. Zhu, Z. Fu, Z. Dai, Y. Zhuang, Y. Ma, A. Qiao, T. Rosing, I. Stoica, et al. (2024) Efficiently scaling llm reasoning with certaindex. arXiv preprint arXiv:2412.20993. Cited by: §8. - I. Gim, G. Chen, S. Lee, N. Sarda, A. Khandelwal, and L. Zhong (2024) Prompt cache: Modular attention reuse for low-latency inference. In Proceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. D. Sa (Eds.), Vol. 6, pp. 325–338. Cited by: §3.2, §8. - Google (2025) External Links: Link Cited by: §1, §2.1. - E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §1, §1, §2.2, §3.2, §7.4. - N. Iliakopoulou, J. Stojkovic, C. Alverti, T. Xu, H. Franke, and J. Torrellas (2025) Chameleon: adaptive caching and scheduling for many-adapter llm inference environments. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture, MICRO ’25, New York, NY, USA, pp. 217–231. External Links: ISBN 9798400715730, Link, Document Cited by: §8. - H. Jeon, H. Ha, and J. Kim (2026) LRAgent: efficient kv cache sharing for multi-lora llm agents. arXiv preprint arXiv:2602.01053. Cited by: §8. - R. Kadekodi, Z. Jin, K. Kamahori, Y. Gu, S. Khatiri, N. H. Bayindirli, S. Gorbunov, and B. Kasikci (2025) AgentFlux: decoupled fine-tuning & inference for on-device agentic systems. External Links: 2510.00229, Link Cited by: §1, §2.3. - A. Kemper and T. Neumann (2011) HyPer: a hybrid oltp&olap main memory database system based on virtual memory snapshots. In 2011 IEEE 27th International Conference on Data Engineering, pp. 195–206. Cited by: §8. - T. Kočiský, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Melis, and E. Grefenstette (2018) The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics 6, pp. 317–328. External Links: Link, Document Cited by: §1, §7.1. - W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, Cited by: §3.3, 1st item, 1st item. - [28] F. F. Labs Evaluating qa: metrics, predictions, and the null response. External Links: Link Cited by: §1, §7.1. - S. Lakshman, A. Gupta, R. Suri, S. Lashley, J. Liang, S. Duvuru, and R. Mayuram (2022) Magma: a high data density storage engine used in couchbase. Proceedings of the VLDB Endowment 15 (12), pp. 3496–3508. Cited by: §8. - J. Li, M. Wang, Z. Zheng, and M. Zhang (2023) LooGLE: can long-context language models understand long contexts?. arXiv preprint arXiv:2311.04939. Cited by: §1, §7.1. - S. Li, H. Lu, T. Wu, M. Yu, Q. Weng, X. Chen, Y. Shan, B. Yuan, and W. Wang (2025) TOPPINGS: cpu-assisted, rank-aware adapter serving for llm inference. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference, USENIX ATC ’25, USA. External Links: ISBN 978-1-939133-48-9 Cited by: §8. - C. Lin, Z. Han, C. Zhang, Y. Yang, F. Yang, C. Chen, and L. Qiu (2024) Parrot: efficient serving of $\{$ llm-based $\}$ applications with semantic variable. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 929–945. Cited by: §8. - B. Liu, W. Lin, M. Fang, Y. Jiang, and F. Lai (2025a) Circinus: efficient query planner for compound ml serving. External Links: 2504.16397, Link Cited by: §8. - S. Liu, C. Wang, H. Yin, P. Molchanov, Y. F. Wang, K. Cheng, and M. Chen (2024a) DoRA: weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353. Cited by: §1. - X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang (2023) AgentBench: evaluating llms as agents. arXiv preprint arXiv: 2308.03688. Cited by: §1. - Y. Liu, K. Q. Lin, C. W. Chen, and M. Z. Shou (2025b) VideoMind: a chain-of-lora agent for long video reasoning. External Links: 2503.13444, Link Cited by: §1, §2.3. - Y. Liu, Y. Huang, J. Yao, S. Feng, Z. Gu, K. Du, H. Li, Y. Cheng, J. Jiang, S. Lu, M. Musuvathi, and E. Choukse (2024b) DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving. arXiv. External Links: 2411.02820, Document Cited by: §1, §2.3, §3.1, §3.2, §8. - Y. Liu, H. Li, Y. Cheng, S. Ray, Y. Huang, Q. Zhang, K. Du, J. Yao, S. Lu, G. Ananthanarayanan, M. Maire, H. Hoffmann, A. Holtzman, and J. Jiang (2024c) CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving. In Proceedings of the ACM SIGCOMM 2024 Conference, Sydney NSW Australia, pp. 38–56. External Links: Document, ISBN 979-8-4007-0614-1 Cited by: §8. - Z. Liu, T. Hoang, J. Zhang, M. Zhu, T. Lan, S. Kokane, J. Tan, W. Yao, Z. Liu, Y. Feng, et al. (2024d) APIGen: automated pipeline for generating verifiable and diverse function-calling datasets. arXiv preprint arXiv:2406.18518. Cited by: §1, 2nd item, §7.1. - M. Luo, X. Shi, C. Cai, T. Zhang, J. Wong, Y. Wang, C. Wang, Y. Huang, Z. Chen, J. E. Gonzalez, and I. Stoica (2025) Autellix: An Efficient Serving Engine for LLM Agents as General Programs. arXiv. External Links: 2502.13965, Document Cited by: §1, §2.3, 2nd item, §8. - S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan (2022) PEFT: state-of-the-art parameter-efficient fine-tuning methods. Note: https://github.com/huggingface/peft Cited by: §1, §2.2. - meta llama (2024) External Links: Link Cited by: §1. - A. Okolnychyi, C. Sun, K. Tanimura, R. Spitzer, R. Blue, S. Ho, Y. Gu, V. Lakkundi, and D. Tsai (2024) Petabyte-scale row-level operations in data lakehouses. Proceedings of the VLDB Endowment 17 (12), pp. 4159–4172. Cited by: §8. - OpenAI (2025) External Links: Link Cited by: §1, §2.1. - OpenAI (2026) External Links: Link Cited by: §1. - Z. Pan, A. Patel, Z. Hu, Y. Shen, Y. Guan, W. Li, L. Qin, Y. Wang, and Y. Ding (2025) KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows. arXiv. External Links: 2507.07400, Document Cited by: §8. - S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez (2024) Gorilla: large language model connected with massive apis. Advances in Neural Information Processing Systems 37, pp. 126544–126565. Cited by: §1. - L. Qianli, H. Zicong, C. Fahao, L. Peng, and G. Song (2025) Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management. arXiv. External Links: 2501.06709, Document Cited by: §8. - R. Qin, Z. Li, W. He, J. Cui, F. Ren, M. Zhang, Y. Wu, W. Zheng, and X. Xu (2024) Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), Santa Clara, CA, pp. 155–170. External Links: ISBN 978-1-939133-45-8 Cited by: §8. - T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom (2023) Toolformer: language models can teach themselves to use tools. Advances in neural information processing systems 36, pp. 68539–68551. Cited by: §1. - N. Shazeer (2019) Fast transformer decoding: one write-head is all you need. arXiv preprint arXiv:1911.02150. Cited by: §2.1. - Y. Sheng, S. Cao, D. Li, C. Hooper, N. Lee, S. Yang, C. Chou, B. Zhu, L. Zheng, K. Keutzer, J. E. Gonzalez, and I. Stoica (2023) S-lora: serving thousands of concurrent lora adapters. arXiv preprint arXiv:2311.03285. Cited by: §2.3, §3.3, §7.1, §7.4, §8. - J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu (2024) Roformer: enhanced transformer with rotary position embedding. Neurocomputing 568, pp. 127063. Cited by: §2.1. - C. Team (2025) External Links: Link Cited by: §1. - Q. Team (2024) Qwen2.5: a party of foundation models. External Links: Link Cited by: §1, §2.1, §7.1. - A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in neural information processing systems 30. Cited by: §2.1. - R. Wang, J. Wang, P. Kadam, M. T. Özsu, and W. G. Aref (2023a) Dlsm: an lsm-based index for memory disaggregation. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pp. 2835–2849. Cited by: §8. - X. Wang, T. Chen, Q. Ge, H. Xia, R. Bao, R. Zheng, Q. Zhang, T. Gui, and X. Huang (2023b) Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10658–10671. Cited by: §1, §3.2. - B. Wu, S. Liu, Y. Zhong, P. Sun, X. Liu, and X. Jin (2024a) Loongserve: efficiently serving long-context large language models with elastic sequence parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, pp. 640–654. Cited by: §1, §2.1, §3.1. - B. Wu, R. Zhu, Z. Zhang, P. Sun, X. Liu, and X. Jin (2024b) DLoRA: dynamically orchestrating requests and adapters for LoRA LLM serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 911–927. Cited by: §7.1, §7.4, §8. - Y. Wu, S. Chen, Y. Zhong, R. Huang, Y. Tan, W. Zhang, L. Zhang, S. Zhou, Y. Liu, S. Zhou, M. Zhang, X. Jin, and P. Huang (2026) DualPath: breaking the storage bandwidth bottleneck in agentic llm inference. External Links: 2602.21548, Link Cited by: §8. - Z. Xie, Z. Xu, M. Zhao, Y. An, V. S. Mailthody, S. Mahlke, M. Garland, and C. Kozyrakis (2025) Strata: Hierarchical Context Caching for Long Context Language Model Serving. arXiv. External Links: 2508.18572, Document Cited by: §8. - Y. Xiong, H. Wu, C. Shao, Z. Wang, R. Zhang, Y. Guo, J. Zhao, K. Zhang, and Z. Pan (2024) LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management. arXiv. External Links: 2410.00428, Document Cited by: §8. - A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §2.1. - J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press (2024) SWE-agent: agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1. - Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning (2018) HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: 1st item. - J. Yao, H. Li, Y. Liu, S. Ray, Y. Cheng, Q. Zhang, K. Du, S. Lu, and J. Jiang (2025a) CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. In Proceedings of the Twentieth European Conference on Computer Systems, Rotterdam Netherlands, pp. 94–109. External Links: Document, ISBN 979-8-4007-1196-1 Cited by: §1, §2.1, §3.1, §3.2, §8. - S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao (2022) React: synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, Cited by: §1, §2.3, 1st item. - X. Yao, Q. Hu, and A. Klimovic (2025b) Deltazip: efficient serving of multiple full-model-tuned llms. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 110–127. Cited by: §8. - Z. Ye, D. Li, Z. Hu, T. Lan, J. Sha, S. Zhang, L. Duan, J. Zuo, H. Lu, Y. Zhou, et al. (2025) MLoRA: fine-tuning lora adapters via highly-efficient pipeline parallelism in multiple gpus. Proceedings of the VLDB Endowment 18 (6), pp. 1948–1961. Cited by: §8. - X. Yu, T. Luo, Y. Wei, F. Lei, Y. Huang, H. Peng, and L. Zhu (2024) Neeko: leveraging dynamic lora for efficient multi-character role-playing agent. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 12540–12557. Cited by: §1, §2.3. - A. Zeng, M. Liu, R. Lu, B. Wang, X. Liu, Y. Dong, and J. Tang (2024) Agenttuning: enabling generalized agent abilities for llms. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 3053–3077. Cited by: §1. - H. Zhang, J. Shi, Y. Wang, Q. Chen, Y. Shan, and M. Guo (2025) Improving the serving performance of multi-lora large language models via efficient lora and kv cache management. arXiv preprint arXiv:2505.03756. Cited by: §8. - L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng (2024) SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, pp. 62557–62583. Cited by: §1, §1, §1, §2.1, §3.3, 2nd item, 1st item, §8. - Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang (2024) DistServe: disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 193–210. Cited by: §1, §2.1, §3.1. - C. Zhou, Y. Zhou, S. Zhang, Y. Wang, and Z. Liu (2024) Dynamic Operator Optimization for Efficient Multi-Tenant LoRA Model Serving. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, pp. 22910–22918. Cited by: §8. - R. Zhu, Z. Jiang, Z. Zhang, X. Liu, X. Liu, and X. Jin (2025a) Cannikin: No Lagger of SLO in Concurrent Multiple LoRA LLM Serving. IEEE Transactions on Parallel and Distributed Systems 36 (9), pp. 1972–1984. External Links: ISSN 1558-2183, Document Cited by: §7.1, §7.4, §8. - Z. Zhu, Q. Su, Y. Ding, K. Song, S. Wang, and G. Pekhimenko (2025b) LoRAFusion: efficient lora fine-tuning for llms. arXiv preprint arXiv:2510.00206. Cited by: §8.

Rendering Paper...