\n ## Stacked Bar Chart: Decode Throughput vs. Batch Size ### Overview This is a stacked bar chart comparing the decode throughput (tokens/second) of different language models (SGLang and LLM-42) at two different batch sizes (10 and 11). The chart uses stacked bars to show the contribution of deterministic and non-deterministic SGLang to the overall throughput. ### Components/Axes * **X-axis:** Batch Size, with markers at 10 and 11. * **Y-axis:** Decode Throughput (tokens/sec), ranging from 0 to 1200. * **Legend:** * SGLang non-deterministic (Light Blue) * SGLang deterministic (Coral/Salmon) * LLM-42 (Light Green) ### Detailed Analysis The chart presents data for two batch sizes: 10 and 11. **Batch Size 10:** * SGLang non-deterministic: Approximately 830 tokens/sec. (Color: Light Blue) * SGLang deterministic: Approximately 380 tokens/sec. (Color: Coral/Salmon) * Total SGLang Throughput: Approximately 1210 tokens/sec. **Batch Size 11:** * SGLang non-deterministic: Approximately 830 tokens/sec. (Color: Light Blue) * SGLang deterministic: Approximately 400 tokens/sec. (Color: Coral/Salmon) * LLM-42: Approximately 170 tokens/sec. (Color: Light Green) * Total Throughput: Approximately 1400 tokens/sec. The bars are stacked, meaning the total height of each bar represents the combined throughput. ### Key Observations * At a batch size of 10, SGLang (both deterministic and non-deterministic) has a significantly higher throughput than LLM-42 (which is not present at this batch size). * At a batch size of 11, LLM-42 is introduced, and the total throughput increases. * The non-deterministic component of SGLang contributes the most to the overall throughput at both batch sizes. * The deterministic component of SGLang increases slightly in throughput from batch size 10 to 11. ### Interpretation The data suggests that SGLang, particularly its non-deterministic component, offers higher decode throughput compared to LLM-42, especially at lower batch sizes. Increasing the batch size to 11 allows for the inclusion of LLM-42, which contributes to a further increase in overall throughput. The consistent throughput of the non-deterministic SGLang component indicates its stability across different batch sizes. The increase in the deterministic SGLang throughput with a larger batch size could be due to improved resource utilization or optimization at higher batch sizes. The chart demonstrates a trade-off between model choice and batch size in optimizing decode throughput. The addition of LLM-42 at batch size 11 does not diminish the performance of SGLang, but rather adds to the overall system throughput.