## Chart: Performance Metrics vs. Draft Length ### Overview The image presents a series of line charts comparing the performance of different models (Llama 2-7B, Llama 2-13B, Llama 2-Chat-7B, and Dolly) across four metrics: block efficiency, MBSU, token rate, and accuracy. The x-axis represents the draft length, ranging from 2 to 5. Four different methods (SD, SpecTr, RSD-C (ours), and RSD-S (ours)) are compared for each model and metric. ### Components/Axes * **Rows:** Each row represents a different model and summarization type combination. The models are Llama 2-7B, Llama 2-13B, Llama 2-Chat-7B, and Dolly. The summarization types are WMT and XSum. * **Columns:** Each column represents a different performance metric: block efficiency, MBSU (Modified Branching Score Unit), token rate, and accuracy. * **X-axis:** Draft length, ranging from 2 to 5. * **Y-axis:** The y-axis scales vary for each metric. * Block efficiency: Ranges from approximately 1.6 to 4.2. * MBSU: Ranges from approximately 1.5 to 4.0. * Token rate: Ranges from approximately 0.9 to 2.0. * Accuracy: Ranges from approximately 0.7 to 1.3. * **Legend:** Located at the bottom of the image. * Solid line with circles: RSD-S (ours) (Blue) * Dashed line with pluses: SpecTr (Red) * Dotted line with triangles: SD (Orange) * Dash-dot line with diamonds: RSD-C (ours) (Green) ### Detailed Analysis **Llama 2-7B** * **WMT** * Block efficiency: All methods show an upward trend with increasing draft length. RSD-S (blue) performs the best, followed by RSD-C (green), SpecTr (red), and SD (orange). * Draft Length 2: SD ~1.8, SpecTr ~2.1, RSD-C ~2.2, RSD-S ~2.3 * Draft Length 5: SD ~2.0, SpecTr ~2.2, RSD-C ~2.5, RSD-S ~2.7 * MBSU: All methods show an upward trend. RSD-S (blue) performs the best, followed by RSD-C (green), SpecTr (red), and SD (orange). * Draft Length 2: SD ~1.8, SpecTr ~1.9, RSD-C ~2.0, RSD-S ~2.1 * Draft Length 5: SD ~2.0, SpecTr ~2.1, RSD-C ~2.4, RSD-S ~2.5 * Token rate: RSD-S (blue) and RSD-C (green) are relatively stable. SpecTr (red) and SD (orange) decrease slightly with increasing draft length. * Draft Length 2: SD ~1.3, SpecTr ~1.2, RSD-C ~1.2, RSD-S ~1.3 * Draft Length 5: SD ~1.1, SpecTr ~1.1, RSD-C ~1.2, RSD-S ~1.3 * Accuracy: All methods maintain a constant accuracy of approximately 1.0 across all draft lengths. * **XSum** * Block efficiency: All methods show an upward trend. RSD-S (blue) performs the best, followed by RSD-C (green), SpecTr (red), and SD (orange). * Draft Length 2: SD ~2.8, SpecTr ~3.0, RSD-C ~3.1, RSD-S ~3.2 * Draft Length 5: SD ~3.1, SpecTr ~3.1, RSD-C ~3.6, RSD-S ~4.2 * MBSU: All methods show an upward trend. RSD-S (blue) performs the best, followed by RSD-C (green), SpecTr (red), and SD (orange). * Draft Length 2: SD ~2.6, SpecTr ~2.8, RSD-C ~2.9, RSD-S ~3.0 * Draft Length 5: SD ~2.8, SpecTr ~2.9, RSD-C ~3.4, RSD-S ~4.0 * Token rate: RSD-S (blue) and RSD-C (green) are relatively stable. SpecTr (red) and SD (orange) decrease slightly with increasing draft length. * Draft Length 2: SD ~1.6, SpecTr ~1.7, RSD-C ~1.7, RSD-S ~1.9 * Draft Length 5: SD ~1.3, SpecTr ~1.4, RSD-C ~1.6, RSD-S ~1.9 * Accuracy: All methods maintain a constant accuracy of approximately 1.0 across all draft lengths. **Llama 2-13B** * **WMT** * Block efficiency: All methods show an upward trend. RSD-S (blue) performs the best, followed by RSD-C (green), SpecTr (red), and SD (orange). * Draft Length 2: SD ~1.8, SpecTr ~2.1, RSD-C ~2.2, RSD-S ~2.3 * Draft Length 5: SD ~2.0, SpecTr ~2.2, RSD-C ~2.5, RSD-S ~2.7 * MBSU: All methods show an upward trend. RSD-S (blue) performs the best, followed by RSD-C (green), SpecTr (red), and SD (orange). * Draft Length 2: SD ~1.9, SpecTr ~2.0, RSD-C ~2.1, RSD-S ~2.2 * Draft Length 5: SD ~2.1, SpecTr ~2.2, RSD-C ~2.5, RSD-S ~2.7 * Token rate: RSD-S (blue) and RSD-C (green) are relatively stable. SpecTr (red) and SD (orange) decrease slightly with increasing draft length. * Draft Length 2: SD ~1.3, SpecTr ~1.2, RSD-C ~1.2, RSD-S ~1.4 * Draft Length 5: SD ~1.1, SpecTr ~1.1, RSD-C ~1.2, RSD-S ~1.4 * Accuracy: All methods maintain a constant accuracy of approximately 1.0 across all draft lengths. * **XSum** * Block efficiency: All methods show an upward trend. RSD-S (blue) performs the best, followed by RSD-C (green), SpecTr (red), and SD (orange). * Draft Length 2: SD ~2.8, SpecTr ~3.0, RSD-C ~3.1, RSD-S ~3.2 * Draft Length 5: SD ~3.0, SpecTr ~3.1, RSD-C ~3.6, RSD-S ~4.2 * MBSU: All methods show an upward trend. RSD-S (blue) performs the best, followed by RSD-C (green), SpecTr (red), and SD (orange). * Draft Length 2: SD ~2.7, SpecTr ~2.8, RSD-C ~2.9, RSD-S ~3.1 * Draft Length 5: SD ~2.9, SpecTr ~2.9, RSD-C ~3.4, RSD-S ~3.9 * Token rate: RSD-S (blue) and RSD-C (green) are relatively stable. SpecTr (red) and SD (orange) decrease slightly with increasing draft length. * Draft Length 2: SD ~1.7, SpecTr ~1.7, RSD-C ~1.7, RSD-S ~2.0 * Draft Length 5: SD ~1.3, SpecTr ~1.4, RSD-C ~1.6, RSD-S ~1.9 * Accuracy: All methods maintain a constant accuracy of approximately 1.0 across all draft lengths. **Llama 2-Chat-7B** * **WMT** * Block efficiency: All methods show an upward trend. RSD-S (blue) performs the best, followed by RSD-C (green), SpecTr (red), and SD (orange). * Draft Length 2: SD ~1.8, SpecTr ~2.0, RSD-C ~2.1, RSD-S ~2.2 * Draft Length 5: SD ~2.0, SpecTr ~2.1, RSD-C ~2.4, RSD-S ~2.7 * MBSU: All methods show an upward trend. RSD-S (blue) performs the best, followed by RSD-C (green), SpecTr (red), and SD (orange). * Draft Length 2: SD ~1.9, SpecTr ~1.9, RSD-C ~2.0, RSD-S ~2.1 * Draft Length 5: SD ~2.0, SpecTr ~2.1, RSD-C ~2.4, RSD-S ~2.5 * Token rate: RSD-S (blue) and RSD-C (green) are relatively stable. SpecTr (red) and SD (orange) decrease slightly with increasing draft length. * Draft Length 2: SD ~1.3, SpecTr ~1.1, RSD-C ~1.1, RSD-S ~1.3 * Draft Length 5: SD ~0.9, SpecTr ~0.9, RSD-C ~1.1, RSD-S ~1.3 * Accuracy: All methods maintain a constant accuracy of approximately 1.0 across all draft lengths. * **XSum** * Block efficiency: All methods show an upward trend. RSD-S (blue) performs the best, followed by RSD-C (green), SpecTr (red), and SD (orange). * Draft Length 2: SD ~2.6, SpecTr ~2.7, RSD-C ~2.8, RSD-S ~3.1 * Draft Length 5: SD ~2.7, SpecTr ~2.8, RSD-C ~3.2, RSD-S ~3.6 * MBSU: All methods show an upward trend. RSD-S (blue) performs the best, followed by RSD-C (green), SpecTr (red), and SD (orange). * Draft Length 2: SD ~2.4, SpecTr ~2.4, RSD-C ~2.5, RSD-S ~2.6 * Draft Length 5: SD ~2.5, SpecTr ~2.5, RSD-C ~2.9, RSD-S ~3.2 * Token rate: RSD-S (blue) and RSD-C (green) are relatively stable. SpecTr (red) and SD (orange) decrease slightly with increasing draft length. * Draft Length 2: SD ~1.6, SpecTr ~1.3, RSD-C ~1.3, RSD-S ~1.6 * Draft Length 5: SD ~1.0, SpecTr ~1.0, RSD-C ~1.3, RSD-S ~1.6 * Accuracy: All methods maintain a constant accuracy of approximately 1.0 across all draft lengths. **Dolly** * **WMT** * Block efficiency: All methods show an upward trend. RSD-S (blue) performs the best, followed by RSD-C (green), SpecTr (red), and SD (orange). * Draft Length 2: SD ~1.6, SpecTr ~1.7, RSD-C ~1.8, RSD-S ~2.2 * Draft Length 5: SD ~1.8, SpecTr ~1.8, RSD-C ~2.0, RSD-S ~2.8 * MBSU: All methods show an upward trend. RSD-S (blue) performs the best, followed by RSD-C (green), SpecTr (red), and SD (orange). * Draft Length 2: SD ~1.7, SpecTr ~1.7, RSD-C ~1.8, RSD-S ~2.2 * Draft Length 5: SD ~1.8, SpecTr ~1.8, RSD-C ~2.0, RSD-S ~2.8 * Token rate: RSD-S (blue) and RSD-C (green) are relatively stable. SpecTr (red) and SD (orange) decrease slightly with increasing draft length. * Draft Length 2: SD ~1.4, SpecTr ~1.3, RSD-C ~1.3, RSD-S ~1.6 * Draft Length 5: SD ~1.0, SpecTr ~1.0, RSD-C ~1.3, RSD-S ~1.6 * Accuracy: All methods maintain a constant accuracy of approximately 1.0 across all draft lengths. * **XSum** * Block efficiency: All methods show an upward trend. RSD-S (blue) performs the best, followed by RSD-C (green), SpecTr (red), and SD (orange). * Draft Length 2: SD ~2.0, SpecTr ~2.6, RSD-C ~2.7, RSD-S ~3.4 * Draft Length 5: SD ~2.0, SpecTr ~2.7, RSD-C ~2.7, RSD-S ~3.4 * MBSU: SD, SpecTr, and RSD-C are relatively stable, while RSD-S increases slightly. * Draft Length 2: SD ~2.0, SpecTr ~2.4, RSD-C ~2.6, RSD-S ~2.7 * Draft Length 5: SD ~2.0, SpecTr ~2.4, RSD-C ~2.6, RSD-S ~2.7 * Token rate: SD, SpecTr, and RSD-C are relatively stable, while RSD-S increases slightly. * Draft Length 2: SD ~1.4, SpecTr ~1.4, RSD-C ~1.4, RSD-S ~1.6 * Draft Length 5: SD ~1.3, SpecTr ~1.4, RSD-C ~1.4, RSD-S ~1.6 * Accuracy: All methods maintain a constant accuracy of approximately 1.0 across all draft lengths. ### Key Observations * **RSD-S (ours)** consistently outperforms the other methods (SD, SpecTr, RSD-C (ours)) in terms of block efficiency and MBSU across all models and summarization types. * **Accuracy** remains relatively constant across all draft lengths and methods. * **Token rate** tends to decrease slightly with increasing draft length for SD and SpecTr, while RSD-S and RSD-C remain more stable. * The performance differences between methods are more pronounced for block efficiency and MBSU than for token rate and accuracy. * The trends are generally consistent across different models (Llama 2-7B, Llama 2-13B, Llama 2-Chat-7B, and Dolly) and summarization types (WMT and XSum). ### Interpretation The data suggests that the RSD-S (ours) method is the most effective in improving block efficiency and MBSU compared to the other methods. The consistent accuracy across different draft lengths indicates that increasing the draft length does not negatively impact the quality of the generated summaries. The slight decrease in token rate for SD and SpecTr with increasing draft length may indicate a trade-off between efficiency and the length of the generated summaries. The consistent trends across different models and summarization types suggest that the observed performance differences are robust and not specific to a particular model or dataset. The RSD-S method appears to be a promising approach for improving the efficiency and quality of text summarization.

## Chart: Performance Metrics vs. Draft Length for Language Models ### Overview This image presents a series of four charts, arranged in a 2x2 grid, comparing the performance of several language models (Llama 2-7B, Llama 2-13B, Llama 2-70B, Dolly, and Spectr) across four metrics: block efficiency, MBSU (Memory Bandwidth per Second Utilization), token rate, and accuracy. The performance is evaluated as a function of "draft length," ranging from 1 to 5. Each chart displays multiple lines representing different algorithms or configurations (SD, Spectr, RSDC-ours, RSDS-ours). ### Components/Axes * **X-axis (all charts):** Draft Length (ranging from 1 to 5, with tick marks at each integer value). * **Y-axis (varies by chart):** * Block Efficiency: Scale ranges from approximately 1.5 to 2.8. * MBSU: Scale ranges from approximately 1.4 to 4.2. * Token Rate: Scale ranges from approximately 0.8 to 1.4. * Accuracy: Scale ranges from approximately 0.6 to 1.3. * **Legend (bottom-center):** * SD (Solid Dark Red Line) * Spectr (Dashed Dark Red Line) * RSDC-(ours) (Solid Teal Line) * RSDS-(ours) (Dashed Teal Line) * **Rows (top to bottom):** * Llama 2-7B * Llama 2-13B * Llama 2-70B * Dolly * Llama 2-Chat-13B * **Columns (left to right):** * Block Efficiency * MBSU * Token Rate * Accuracy ### Detailed Analysis or Content Details **Llama 2-7B:** * **Block Efficiency:** RSDC-(ours) shows a slight upward trend, starting at approximately 1.8 and reaching around 2.2. RSDS-(ours) is relatively flat around 1.6. SD starts at ~1.7 and ends at ~2.0. Spectr is flat around 1.6. * **MBSU:** RSDC-(ours) increases from ~1.3 to ~1.9. RSDS-(ours) is relatively flat around 1.5. SD starts at ~2.0 and ends at ~3.0. Spectr is flat around 1.5. * **Token Rate:** RSDC-(ours) increases from ~0.9 to ~1.3. RSDS-(ours) is relatively flat around 1.1. SD starts at ~0.7 and ends at ~1.1. Spectr is flat around 0.9. * **Accuracy:** RSDC-(ours) is relatively flat around 1.0. RSDS-(ours) is relatively flat around 0.7. SD starts at ~0.7 and ends at ~1.0. Spectr is flat around 0.7. **Llama 2-13B:** * **Block Efficiency:** RSDC-(ours) shows a slight upward trend, starting at approximately 2.0 and reaching around 2.4. RSDS-(ours) is relatively flat around 1.8. SD starts at ~1.8 and ends at ~2.2. Spectr is flat around 1.6. * **MBSU:** RSDC-(ours) increases from ~1.4 to ~2.0. RSDS-(ours) is relatively flat around 1.6. SD starts at ~2.0 and ends at ~3.9. Spectr is flat around 1.9. * **Token Rate:** RSDC-(ours) increases from ~1.1 to ~1.4. RSDS-(ours) is relatively flat around 1.2. SD starts at ~0.9 and ends at ~1.4. Spectr is flat around 1.1. * **Accuracy:** RSDC-(ours) is relatively flat around 1.1. RSDS-(ours) is relatively flat around 0.8. SD starts at ~0.7 and ends at ~1.3. Spectr is flat around 0.7. **Llama 2-70B:** * **Block Efficiency:** RSDC-(ours) shows a slight upward trend, starting at approximately 2.1 and reaching around 2.5. RSDS-(ours) is relatively flat around 1.9. SD starts at ~1.8 and ends at ~2.3. Spectr is flat around 1.8. * **MBSU:** RSDC-(ours) increases from ~1.1 to ~1.9. RSDS-(ours) is relatively flat around 1.4. SD starts at ~1.9 and ends at ~3.0. Spectr is flat around 1.5. * **Token Rate:** RSDC-(ours) increases from ~0.9 to ~1.3. RSDS-(ours) is relatively flat around 1.1. SD starts at ~0.7 and ends at ~1.1. Spectr is flat around 0.9. * **Accuracy:** RSDC-(ours) is relatively flat around 1.0. RSDS-(ours) is relatively flat around 0.7. SD starts at ~0.7 and ends at ~1.0. Spectr is flat around 0.7. **Dolly:** * **Block Efficiency:** RSDC-(ours) shows a slight upward trend, starting at approximately 2.2 and reaching around 2.6. RSDS-(ours) is relatively flat around 2.0. SD starts at ~2.0 and ends at ~2.4. Spectr is flat around 1.8. * **MBSU:** RSDC-(ours) increases from ~1.3 to ~2.6. RSDS-(ours) is relatively flat around 1.8. SD starts at ~1.8 and ends at ~3.2. Spectr is flat around 1.8. * **Token Rate:** RSDC-(ours) increases from ~1.0 to ~1.4. RSDS-(ours) is relatively flat around 1.2. SD starts at ~0.8 and ends at ~1.2. Spectr is flat around 1.0. * **Accuracy:** RSDC-(ours) is relatively flat around 1.1. RSDS-(ours) is relatively flat around 0.8. SD starts at ~0.7 and ends at ~1.1. Spectr is flat around 0.7. **Llama 2-Chat-13B:** * **Block Efficiency:** RSDC-(ours) shows a slight upward trend, starting at approximately 2.0 and reaching around 2.4. RSDS-(ours) is relatively flat around 1.8. SD starts at ~1.8 and ends at ~2.2. Spectr is flat around 1.6. * **MBSU:** RSDC-(ours) increases from ~1.2 to ~2.4. RSDS-(ours) is relatively flat around 1.6. SD starts at ~1.8 and ends at ~3.6. Spectr is flat around 1.9. * **Token Rate:** RSDC-(ours) increases from ~0.9 to ~1.3. RSDS-(ours) is relatively flat around 1.1. SD starts at ~0.7 and ends at ~1.1. Spectr is flat around 0.9. * **Accuracy:** RSDC-(ours) is relatively flat around 0.9. RSDS-(ours) is relatively flat around 0.7. SD starts at ~0.7 and ends at ~1.0. Spectr is flat around 0.7. ### Key Observations * SD consistently exhibits higher MBSU values across all models and draft lengths. * RSDC-(ours) generally shows an increasing trend for Block Efficiency, MBSU, and Token Rate as draft length increases. * RSDS-(ours) tends to be relatively stable across different draft lengths for all metrics. * Spectr consistently shows the lowest performance across all metrics. * Accuracy is generally lower for RSDS-(ours) compared to other algorithms. ### Interpretation The data suggests that the RSDC-(ours) algorithm demonstrates improved performance with increasing draft length, particularly in terms of block efficiency, memory bandwidth utilization, and token rate. The SD algorithm consistently provides the highest MBSU, indicating efficient memory usage. The RSDS-(ours) algorithm maintains stable performance but generally lags behind RSDC-(ours) and SD. Spectr consistently underperforms, suggesting it may not be well-suited for these tasks or requires further optimization. The consistent upward trend of RSDC-(ours) suggests that it benefits from longer draft lengths, potentially due to increased opportunities for optimization or better utilization of available resources. The stability of RSDS-(ours) might indicate a different optimization strategy that prioritizes consistency over maximizing performance with increasing draft length. The differences in accuracy suggest that the algorithms employ different strategies for generating accurate outputs, with SD and RSDC-(ours) generally achieving higher accuracy than RSDS-(ours) and Spectr. The data highlights the trade-offs between different algorithms in terms of performance, efficiency, and accuracy, and suggests that the optimal choice of algorithm depends on the specific application and requirements.

## Line Chart Grid: Performance Metrics of Speculative Decoding Methods ### Overview The image is a grid of 28 line charts arranged in 7 rows and 4 columns. It compares the performance of four speculative decoding methods (SD, SpecTr, RSD-C, RSD-S) across four metrics (block efficiency, MBSU, token rate, accuracy) for different combinations of Llama-2 models and datasets. The x-axis for all charts is "draft length" (values: 2, 3, 4, 5). ### Components/Axes * **Columns (Metrics):** 1. **block efficiency** (y-axis label): Measures efficiency, scale varies by model/dataset. 2. **MBSU** (y-axis label): Unknown acronym, scale varies by model/dataset. 3. **token rate** (y-axis label): Measures token generation rate, scale varies by model/dataset. 4. **accuracy** (y-axis label): All charts show a scale from 0.7 to 1.3, with data clustered at 1.0. * **Rows (Model & Dataset):** * Row 1: Llama-2-7B on WMT * Row 2: Llama-2-7B on XSum * Row 3: Llama-2-13B on WMT * Row 4: Llama-2-13B on XSum * Row 5: Llama-2-Chat-7B on WMT, XSum, and Dolly (3 sub-rows) * Row 6: Llama-2-Chat-13B on WMT, XSum, and Dolly (3 sub-rows) * **X-Axis:** "draft length" with ticks at 2, 3, 4, 5. Labels are only shown on the bottom row of charts. * **Legend (Bottom Center):** * `... SD` (dotted orange line) * `--- SpecTr` (dashed red line) * `--- RSD-C (ours)` (dashed green line with diamond markers) * `— RSD-S (ours)` (solid blue line with circle markers) ### Detailed Analysis **1. Block Efficiency (Column 1):** * **Trend:** For nearly all model/dataset combinations, block efficiency increases as draft length increases from 2 to 5. The lines slope upward. * **Performance Order:** RSD-S (blue solid) consistently achieves the highest block efficiency, followed closely by RSD-C (green dashed). SpecTr (red dashed) is generally lower, and SD (orange dotted) is the lowest. * **Example Values (Approximate):** For Llama-2-7B/WMT at draft length 5: RSD-S ~2.8, RSD-C ~2.5, SpecTr ~2.2, SD ~1.8. **2. MBSU (Column 2):** * **Trend:** Similar upward trend with increasing draft length for most charts. * **Performance Order:** RSD-S and RSD-C are again the top performers, with RSD-S often slightly higher. SpecTr and SD are lower. * **Example Values (Approximate):** For Llama-2-13B/XSum at draft length 5: RSD-S ~3.9, RSD-C ~3.5, SpecTr ~3.1, SD ~2.7. **3. Token Rate (Column 3):** * **Trend:** Mixed trends. Some methods show a slight increase, others are flat or show a slight decrease with longer drafts. The Dolly dataset charts show more variability. * **Performance Order:** RSD-S and RSD-C typically maintain the highest token rates. SD often shows a declining trend. * **Notable Outlier:** In the Llama-2-Chat-7B/Dolly chart, the RSD-S (blue) line dips at draft length 5. * **Example Values (Approximate):** For Llama-2-Chat-13B/WMT at draft length 4: RSD-S ~1.55, RSD-C ~1.45, SpecTr ~1.35, SD ~1.25. **4. Accuracy (Column 4):** * **Trend:** All four methods across all model/dataset combinations show a flat line at approximately 1.0 (perfect accuracy) for all draft lengths. There is no visible variation. * **Performance Order:** All methods are indistinguishable, performing at the ceiling. ### Key Observations 1. **Consistent Superiority:** The proposed methods, RSD-S (blue) and RSD-C (green), consistently outperform the baselines (SpecTr and SD) across the block efficiency and MBSU metrics for all tested models and datasets. 2. **Accuracy Ceiling:** All methods achieve near-perfect accuracy (~1.0), indicating that the speculative decoding process does not introduce errors for these tasks and draft lengths. 3. **Metric Trade-offs:** While block efficiency and MBSU improve with draft length, the token rate does not show a universal improvement, suggesting a potential trade-off between efficiency per block and raw generation speed. 4. **Model/Chat Effect:** The trends are largely consistent between base models (Llama-2-7B/13B) and their chat-tuned versions (Llama-2-Chat-7B/13B), and across different datasets (WMT, XSum, Dolly). ### Interpretation This grid of charts presents a comprehensive evaluation of a new speculative decoding technique (RSD, with variants C and S). The data strongly suggests that the RSD methods are more efficient (higher block efficiency and MBSU) than the compared baselines (SD and SpecDraft) without sacrificing accuracy. The consistent performance across diverse models (7B, 13B, base, chat) and tasks (translation, summarization, instruction following) indicates robustness. The key finding is that RSD-S and RSD-C allow for more effective use of the draft mechanism, generating more valid tokens per speculative block (block efficiency) and achieving a higher overall utilization score (MBSU). The flat accuracy lines are crucial, confirming that these gains in efficiency do not come at the cost of output quality. The token rate data adds nuance, showing that simply increasing draft length doesn't always speed up generation linearly, but the efficiency gains of RSD are still evident. This work demonstrates a meaningful advancement in making large language model inference faster and more efficient.

## Line Graphs: Model Performance Across Draft Lengths ### Overview The image contains multiple line graphs comparing the performance of different language models (SD, SpecTr, RSD-C, RSD-S) across three metrics: **block efficiency**, **token rate**, and **accuracy**. Each graph corresponds to a specific model architecture (e.g., Llama-2-7B, Llama-2-13B, XSum, Dolly, Llama-2-Chat-13B) and evaluates how performance changes as draft length increases from 2 to 5. The graphs use distinct line styles and colors to differentiate models, with a legend at the bottom for reference. --- ### Components/Axes - **X-axis**: Draft length (2–5), labeled as "draft length" in all graphs. - **Y-axes**: - **Block efficiency**: Y-axis ranges from ~1.5 to 2.8. - **Token rate**: Y-axis ranges from ~0.7 to 1.6. - **Accuracy**: Y-axis ranges from ~0.7 to 1.3. - **Legends**: - **SD**: Orange dotted line. - **SpecTr**: Red dashed line. - **RSD-C (ours)**: Green dashed line. - **RSD-S (ours)**: Blue solid line. - **Model categories** (grouped by architecture): - Llama-2-7B - Llama-2-13B - XSum - Dolly - Llama-2-Chat-13B --- ### Detailed Analysis #### Block Efficiency - **Trend**: RSD-S (blue) consistently shows the highest block efficiency, increasing slightly with draft length (e.g., 2.2 → 2.8 for Llama-2-7B). SD (orange) and SpecTr (red) decline marginally, while RSD-C (green) remains stable. - **Data points**: - Llama-2-7B: RSD-S (2.2 → 2.8), SpecTr (1.6 → 1.8), RSD-C (1.4 → 1.6), SD (1.2 → 1.4). - Llama-2-13B: RSD-S (2.0 → 2.6), SpecTr (1.8 → 2.0), RSD-C (1.6 → 1.8), SD (1.4 → 1.6). #### Token Rate - **Trend**: RSD-S (blue) increases with draft length (e.g., 1.3 → 1.6 for Llama-2-7B), while SD (orange) and SpecTr (red) decline. RSD-C (green) remains flat. - **Data points**: - Llama-2-7B: RSD-S (1.3 → 1.6), SpecTr (1.1 → 0.9), RSD-C (1.2 → 1.2), SD (1.4 → 1.1). - XSum: RSD-S (1.7 → 2.0), SpecTr (1.5 → 1.3), RSD-C (1.4 → 1.4), SD (1.9 → 1.6). #### Accuracy - **Trend**: All models maintain stable accuracy (~0.7–1.0) across draft lengths, with minor fluctuations. RSD-S (blue) and RSD-C (green) show slight improvements in some cases. - **Data points**: - Llama-2-7B: RSD-S (0.7 → 0.7), SpecTr (0.7 → 0.7), RSD-C (0.7 → 0.7), SD (0.7 → 0.7). - Dolly: RSD-S (0.7 → 0.7), SpecTr (0.7 → 0.7), RSD-C (0.7 → 0.7), SD (0.7 → 0.7). --- ### Key Observations 1. **RSD-S (blue)** outperforms other models in **block efficiency** and **token rate** for most architectures, especially at longer draft lengths. 2. **SD (orange)** and **SpecTr (red)** exhibit declining performance in token rate as draft length increases. 3. **Accuracy** remains largely unaffected by draft length across all models, suggesting robustness in output quality. 4. **RSD-C (green)** maintains consistent performance but lags behind RSD-S in efficiency metrics. --- ### Interpretation The data suggests that **RSD-S** is optimized for efficiency (block efficiency and token rate) at the cost of slightly higher computational demands, as indicated by its rising performance with longer drafts. In contrast, **SD** and **SpecTr** degrade in efficiency under similar conditions, potentially due to suboptimal scaling. The stability of accuracy across draft lengths implies that longer drafts do not inherently compromise output quality, but efficiency gains depend heavily on model architecture. Notably, **RSD-C** balances performance and efficiency but does not surpass RSD-S in critical metrics. These trends highlight the importance of architectural design in scaling language models effectively.