## Bar Chart: Decode Throughput vs. Batch Size ### Overview The image is a bar chart comparing the decode throughput (tokens/sec) for different configurations across two batch sizes (10 and 11). The chart compares "SGLang non-deterministic", "SGLang deterministic", and "LLM-42". ### Components/Axes * **X-axis:** Batch Size, with values 10 and 11. * **Y-axis:** Decode Throughput (tokens/sec), ranging from 0 to 1200. * **Legend (top-left):** * SGLang non-deterministic (light blue) * SGLang deterministic (light red/brown) * LLM-42 (light green) ### Detailed Analysis * **Batch Size 10:** * SGLang non-deterministic: Approximately 840 tokens/sec. * SGLang deterministic: Not present for batch size 10. * LLM-42: Not present for batch size 10. * **Batch Size 11:** * SGLang non-deterministic: Approximately 940 tokens/sec. * SGLang deterministic: Approximately 420 tokens/sec. * LLM-42: Approximately 520 tokens/sec. ### Key Observations * For batch size 10, only SGLang non-deterministic data is available. * For batch size 11, all three configurations (SGLang non-deterministic, SGLang deterministic, and LLM-42) are present. * SGLang non-deterministic throughput increases from batch size 10 to 11. ### Interpretation The chart compares the decode throughput of different language model configurations at batch sizes 10 and 11. The data suggests that increasing the batch size from 10 to 11 improves the throughput of SGLang non-deterministic. The introduction of SGLang deterministic and LLM-42 at batch size 11 allows for a direct comparison of their performance. LLM-42 has a slightly higher throughput than SGLang deterministic at batch size 11.

## Bar Chart: Decode Throughput by Batch Size ### Overview This image displays a stacked bar chart comparing the "Decode Throughput" (measured in tokens/sec) for two different "Batch Size" configurations: 10 and 11. The chart utilizes a color-coded legend to distinguish between three different operational modes or models: "SGLang non-deterministic," "SGLang deterministic," and "LLM-42." ### Components/Axes * **Y-Axis:** Labeled "Decode Throughput (tokens/sec)". The scale ranges from 0 to 1200, with major tick marks every 200 units. * **X-Axis:** Labeled "Batch Size". It contains two discrete categories: 10 and 11. * **Legend:** Positioned at the top-center of the chart area. * **Blue rectangle:** "SGLang non-deterministic" * **Red/Pink rectangle:** "SGLang deterministic" * **Green rectangle:** "LLM-42" ### Detailed Analysis The chart presents two distinct bars corresponding to the batch sizes on the x-axis. **1. Batch Size 10 (Left Bar):** * **Visual Trend:** This bar is a single, solid block. * **Composition:** The entire bar is colored blue, corresponding to "SGLang non-deterministic." * **Value:** The top of the bar aligns approximately with the 840 tokens/sec mark on the y-axis. **2. Batch Size 11 (Right Bar):** * **Visual Trend:** This bar is a stacked composite of all three categories. * **Composition (from bottom to top):** * **Bottom Section (Red/Pink):** Corresponds to "SGLang deterministic." It extends from 0 to approximately 410 tokens/sec. * **Middle Section (Green):** Corresponds to "LLM-42." It sits on top of the red section, extending from approximately 410 to 910 tokens/sec. This represents a segment height of approximately 500 tokens/sec. * **Top Section (Blue):** Corresponds to "SGLang non-deterministic." It sits on top of the green section, extending from approximately 910 to 940 tokens/sec. This represents a segment height of approximately 30 tokens/sec. * **Total Value:** The total height of the stacked bar is approximately 940 tokens/sec. ### Key Observations * **Throughput Increase:** Increasing the batch size from 10 to 11 results in an overall increase in decode throughput (from ~840 to ~940 tokens/sec). * **Composition Shift:** The nature of the throughput changes significantly between the two batch sizes. At Batch Size 10, the throughput is derived entirely from "SGLang non-deterministic." At Batch Size 11, the throughput is dominated by "LLM-42" and "SGLang deterministic," with "SGLang non-deterministic" contributing only a negligible fraction (~3%) of the total throughput. ### Interpretation The data suggests that the system's performance characteristics are highly sensitive to the batch size. * **Operational Shift:** The transition from Batch Size 10 to 11 appears to trigger a fundamental change in how tokens are processed. At Batch Size 10, the system operates in a "non-deterministic" mode exclusively. At Batch Size 11, the system shifts to a hybrid processing model where the vast majority of the work is handled by "LLM-42" and "SGLang deterministic" components. * **Efficiency:** While the total throughput is higher at Batch Size 11, the "SGLang non-deterministic" mode—which was the sole driver of performance at Batch Size 10—is almost entirely sidelined at Batch Size 11. This implies that the system may be switching to a different execution path or model configuration when the batch size is increased, prioritizing "LLM-42" and deterministic processing over the non-deterministic path.

## Bar Chart: Decode Throughput Comparison by Batch Size and Model ### Overview The chart compares decode throughput (tokens/second) across three models (SGLang non-deterministic, SGLang deterministic, LLM-42) at two batch sizes (10 and 11). Throughput is measured on the y-axis (0–1200 tokens/sec), while batch size is on the x-axis (10 and 11). The legend in the top-left corner maps colors to models: blue (SGLang non-deterministic), red (SGLang deterministic), and green (LLM-42). ### Components/Axes - **X-axis (Batch Size)**: Labeled "Batch Size" with ticks at 10 and 11. - **Y-axis (Decode Throughput)**: Labeled "Decode Throughput (tokens/sec)" with increments of 200 up to 1200. - **Legend**: Positioned in the top-left corner, with three entries: - Blue: SGLang non-deterministic - Red: SGLang deterministic - Green: LLM-42 ### Detailed Analysis - **Batch Size 10**: - **SGLang non-deterministic (blue)**: Dominates with ~800 tokens/sec. - **SGLang deterministic (red)**: Not visible (implied 0 tokens/sec). - **LLM-42 (green)**: Not visible (implied 0 tokens/sec). - **Batch Size 11**: - **SGLang non-deterministic (blue)**: Minimal throughput (~10 tokens/sec). - **SGLang deterministic (red)**: ~400 tokens/sec. - **LLM-42 (green)**: ~800 tokens/sec. ### Key Observations 1. **SGLang non-deterministic** shows a drastic drop in throughput when batch size increases from 10 to 11 (~800 → ~10 tokens/sec). 2. **SGLang deterministic** and **LLM-42** maintain stable performance at batch size 11 (~400 and ~800 tokens/sec, respectively). 3. No data is reported for SGLang deterministic and LLM-42 at batch size 10. ### Interpretation The data suggests that **SGLang non-deterministic** is highly sensitive to batch size increases, experiencing a near-collapse in efficiency at batch size 11. In contrast, **SGLang deterministic** and **LLM-42** demonstrate robustness, maintaining significant throughput even at larger batch sizes. This implies that non-deterministic processing may face scalability challenges in high-throughput scenarios, while deterministic and LLM-42 architectures are better optimized for larger batches. The absence of data for deterministic and LLM-42 at batch size 10 raises questions about whether these models were tested at that configuration or if the results were intentionally omitted.