## Heatmap: R1-Llama vs. R1-Qwen Performance ### Overview The image presents two heatmaps comparing the performance of "R1-Llama" and "R1-Qwen" models. The heatmaps visualize the "Pass@1" metric across different "Local Window Sizes" and "Ratio" values. The color intensity represents the Pass@1 score, with lighter shades indicating lower scores and darker shades indicating higher scores. ### Components/Axes * **Titles:** "R1-Llama" (left heatmap), "R1-Qwen" (right heatmap) * **Y-axis (Local Window Size):** 500, 1000, 2000, 3000 * **X-axis (Ratio):** 0.1, 0.2, 0.3, 0.4, 0.5 * **Color Legend (Pass@1):** Ranges from approximately 50 (lightest shade) to 56 (darkest shade). The legend shows a continuous color gradient. ### Detailed Analysis **R1-Llama Heatmap:** | Local Window Size | Ratio 0.1 | Ratio 0.2 | Ratio 0.3 | Ratio 0.4 | Ratio 0.5 | | ----------------- | --------- | --------- | --------- | --------- | --------- | | 3000 | 49.1 | 50.1 | 50.6 | 50.7 | 51.4 | | 2000 | 49.5 | 51.7 | 52.8 | 52.5 | 50.9 | | 1000 | 49.9 | 52.7 | 51.0 | 51.9 | 51.7 | | 500 | 49.8 | 52.1 | 50.7 | 50.8 | 51.7 | * **Trend:** The Pass@1 score for R1-Llama generally increases as the Ratio increases from 0.1 to 0.2. After 0.2, the performance fluctuates. The performance is generally lower for a Local Window Size of 3000. **R1-Qwen Heatmap:** | Local Window Size | Ratio 0.1 | Ratio 0.2 | Ratio 0.3 | Ratio 0.4 | Ratio 0.5 | | ----------------- | --------- | --------- | --------- | --------- | --------- | | 3000 | 53.9 | 53.9 | 53.2 | 54.4 | 53.8 | | 2000 | 52.4 | 51.9 | 54.6 | 56.3 | 53.7 | | 1000 | 52.2 | 54.4 | 53.8 | 53.3 | 53.0 | | 500 | 51.5 | 51.8 | 52.0 | 54.3 | 54.6 | * **Trend:** The Pass@1 score for R1-Qwen shows a more pronounced increase with higher Ratio values, particularly at a Local Window Size of 2000 and Ratio of 0.4, where the performance peaks. ### Key Observations * R1-Qwen generally outperforms R1-Llama across most configurations. * The highest Pass@1 score is achieved by R1-Qwen with a Local Window Size of 2000 and a Ratio of 0.4 (56.3). * R1-Llama's performance seems less sensitive to changes in Ratio and Local Window Size compared to R1-Qwen. ### Interpretation The heatmaps provide a visual comparison of the performance of two models, R1-Llama and R1-Qwen, under varying configurations of "Local Window Size" and "Ratio." The data suggests that R1-Qwen is a superior model, achieving higher Pass@1 scores across most parameter settings. The optimal configuration for R1-Qwen appears to be a Local Window Size of 2000 and a Ratio of 0.4, indicating that these settings are crucial for maximizing its performance. R1-Llama's relatively stable performance across different configurations might suggest a more robust but less optimized model. The choice of model and configuration should be guided by the specific application and the trade-off between performance and sensitivity to parameter tuning.

\n ## Heatmap: Pass@1 Performance Comparison - R1-Llama vs. R1-Qwen ### Overview This image presents a heatmap comparing the Pass@1 performance of two models, R1-Llama and R1-Qwen, across varying combinations of 'Ratio' and 'Local Window Size'. The heatmap uses a color gradient to represent the Pass@1 scores, with warmer colors indicating higher performance. ### Components/Axes * **X-axis:** Ratio, ranging from 0.1 to 0.5, with increments of 0.1. * **Y-axis:** Local Window Size, with categories 500, 1000, 2000, and 3000. * **Two Heatmaps:** One for R1-Llama (left) and one for R1-Qwen (right). * **Colorbar:** Located on the right side, representing Pass@1 scores ranging from approximately 50 to 56. * **Titles:** "R1-Llama" above the left heatmap and "R1-Qwen" above the right heatmap. ### Detailed Analysis or Content Details **R1-Llama Heatmap:** * **Trend:** Generally, performance increases with increasing Local Window Size and Ratio, but the effect is not uniform. * **Data Points:** * Ratio 0.1: * Local Window Size 500: 49.8 * Local Window Size 1000: 49.9 * Local Window Size 2000: 49.5 * Local Window Size 3000: 49.1 * Ratio 0.2: * Local Window Size 500: 52.1 * Local Window Size 1000: 51.7 * Local Window Size 2000: 50.9 * Local Window Size 3000: 50.1 * Ratio 0.3: * Local Window Size 500: 50.7 * Local Window Size 1000: 52.7 * Local Window Size 2000: 52.8 * Local Window Size 3000: 50.6 * Ratio 0.4: * Local Window Size 500: 50.8 * Local Window Size 1000: 51.9 * Local Window Size 2000: 52.5 * Local Window Size 3000: 50.7 * Ratio 0.5: * Local Window Size 500: 51.7 * Local Window Size 1000: 51.7 * Local Window Size 2000: 51.4 * Local Window Size 3000: 51.4 **R1-Qwen Heatmap:** * **Trend:** Similar to R1-Llama, performance generally increases with increasing Local Window Size and Ratio, but with some variations. * **Data Points:** * Ratio 0.1: * Local Window Size 500: 51.5 * Local Window Size 1000: 52.2 * Local Window Size 2000: 52.4 * Local Window Size 3000: 53.9 * Ratio 0.2: * Local Window Size 500: 51.8 * Local Window Size 1000: 54.4 * Local Window Size 2000: 51.9 * Local Window Size 3000: 53.9 * Ratio 0.3: * Local Window Size 500: 52.0 * Local Window Size 1000: 53.8 * Local Window Size 2000: 54.6 * Local Window Size 3000: 53.2 * Ratio 0.4: * Local Window Size 500: 54.3 * Local Window Size 1000: 53.3 * Local Window Size 2000: 56.3 * Local Window Size 3000: 54.4 * Ratio 0.5: * Local Window Size 500: 54.6 * Local Window Size 1000: 53.0 * Local Window Size 2000: 53.7 * Local Window Size 3000: 53.8 ### Key Observations * R1-Qwen consistently outperforms R1-Llama across all combinations of Ratio and Local Window Size. * For both models, increasing the Local Window Size from 500 to 2000 generally leads to performance improvements, but increasing it further to 3000 doesn't always yield the same benefit. * The highest Pass@1 score for R1-Llama is 52.8, while the highest for R1-Qwen is 56.3. * The performance difference between the models is most pronounced at higher Ratio values (0.4 and 0.5). ### Interpretation The heatmap demonstrates the impact of 'Ratio' and 'Local Window Size' on the Pass@1 performance of two language models, R1-Llama and R1-Qwen. The 'Ratio' likely represents a parameter controlling the amount of context considered during evaluation, while 'Local Window Size' might relate to the size of the input sequence processed at a time. The consistent outperformance of R1-Qwen suggests that it is more robust to variations in these parameters or benefits more from larger context windows. The non-linear relationship between Local Window Size and performance indicates that there's an optimal window size beyond which the benefits diminish, potentially due to computational constraints or the model's ability to effectively utilize the additional context. The heatmap provides valuable insights for optimizing the configuration of these models for specific tasks. It suggests that for R1-Qwen, a Ratio of 0.4 or 0.5 and a Local Window Size of 2000 might be a good starting point for achieving high Pass@1 scores. Further investigation could explore the reasons behind the diminishing returns of larger Local Window Sizes and the specific mechanisms that contribute to R1-Qwen's superior performance.

## Heatmap Comparison: R1-Llama vs. R1-Qwen Performance (Pass@1) ### Overview The image displays two side-by-side heatmaps comparing the performance of two models, "R1-Llama" and "R1-Qwen," across different configurations. Performance is measured by the "Pass@1" metric, visualized through a color gradient. The analysis explores how this metric changes with variations in "Local Window Size" and "Ratio." ### Components/Axes * **Titles:** Two main titles are positioned at the top: "R1-Llama" (left heatmap) and "R1-Qwen" (right heatmap). * **Y-Axis (Left):** Labeled "Local Window Size." It has four discrete, categorical values listed from top to bottom: 3000, 2000, 1000, 500. * **X-Axis (Bottom):** Labeled "Ratio." It has five discrete, categorical values listed from left to right: 0.1, 0.2, 0.3, 0.4, 0.5. * **Color Scale/Legend (Right):** A vertical color bar labeled "Pass@1" on its right side. The scale ranges from approximately 50 (light yellow) to 56 (dark blue). Tick marks are present at 50, 52, 54, and 56. * **Data Grids:** Each heatmap is a 4-row by 5-column grid. Each cell contains a numerical value representing the Pass@1 score for a specific combination of Local Window Size and Ratio. ### Detailed Analysis **R1-Llama Heatmap (Left):** * **Row 1 (Local Window Size 3000):** Values from left to right (Ratio 0.1 to 0.5): 49.1, 50.1, 50.6, 50.7, 51.4. The color transitions from light yellow to light green. * **Row 2 (Local Window Size 2000):** Values: 49.5, 51.7, 52.8, 52.5, 50.9. Colors range from light yellow to teal, with the highest value (52.8) at Ratio 0.3. * **Row 3 (Local Window Size 1000):** Values: 49.9, 52.7, 51.0, 51.9, 51.7. Colors are a mix of light yellow and teal. * **Row 4 (Local Window Size 500):** Values: 49.8, 52.1, 50.7, 50.8, 51.7. Colors are similar to Row 3. **R1-Qwen Heatmap (Right):** * **Row 1 (Local Window Size 3000):** Values: 53.9, 53.9, 53.2, 54.4, 53.8. Colors are shades of medium blue. * **Row 2 (Local Window Size 2000):** Values: 52.4, 51.9, 54.6, 56.3, 53.7. This row contains the highest value in the entire chart (56.3 at Ratio 0.4), shown in dark blue. * **Row 3 (Local Window Size 1000):** Values: 52.2, 54.4, 53.8, 53.3, 53.0. Colors are shades of blue. * **Row 4 (Local Window Size 500):** Values: 51.5, 51.8, 52.0, 54.3, 54.6. Colors range from light blue to medium blue. ### Key Observations 1. **Overall Performance Gap:** The R1-Qwen model consistently achieves higher Pass@1 scores than the R1-Llama model across all tested configurations. The R1-Qwen cells are predominantly blue (scores >52), while R1-Llama cells are mostly yellow-green (scores <53). 2. **Peak Performance:** The absolute highest Pass@1 score (56.3) is achieved by R1-Qwen with a Local Window Size of 2000 and a Ratio of 0.4. 3. **Sensitivity to Parameters:** * For **R1-Llama**, performance does not show a strong, consistent trend with increasing Ratio or decreasing Window Size. The highest scores are scattered (e.g., 52.8 at Size 2000/Ratio 0.3, 52.7 at Size 1000/Ratio 0.2). * For **R1-Qwen**, there is a more noticeable pattern. Performance tends to be higher at moderate Ratios (0.3-0.5) compared to the lowest Ratio (0.1). The configuration of Size 2000/Ratio 0.4 is a clear outlier peak. 4. **Stability:** R1-Qwen's performance appears more stable across different Window Sizes for a given Ratio, especially at Ratios 0.4 and 0.5, where scores remain relatively high. ### Interpretation This heatmap comparison provides a clear visual benchmark suggesting that the R1-Qwen model architecture or training methodology yields superior performance (as measured by Pass@1) compared to R1-Llama for the evaluated task. The data indicates that hyperparameter tuning has a significant impact, particularly for R1-Qwen, where a specific "sweet spot" (Size 2000, Ratio 0.4) is identified. The lack of a simple linear trend in either model suggests a complex interaction between the Local Window Size and Ratio parameters. The investigation implies that simply increasing one parameter does not guarantee better performance; the optimal setting is configuration-dependent. For practical deployment, R1-Qwen is the preferable model based on this metric, and its configuration should be carefully tuned, with the Size 2000/Ratio 0.4 setting being a strong candidate for optimal results. The visualization effectively communicates that model choice and parameter selection are critical for maximizing Pass@1 performance.

## Heatmap: Pass@1 Performance Comparison of R1-Llama and R1-Qwen Models ### Overview The image presents a comparative heatmap analysis of two language models (R1-Llama and R1-Qwen) across varying local window sizes (500, 1000, 2000, 3000) and ratio parameters (0.1, 0.2, 0.3, 0.4, 0.5). Pass@1 metrics are visualized using a color gradient from 50 (light yellow) to 56 (dark blue), with numerical values embedded in each cell. ### Components/Axes - **X-axis (Horizontal)**: Ratio (0.1, 0.2, 0.3, 0.4, 0.5) - **Y-axis (Vertical)**: Local Window Size (500, 1000, 2000, 3000) - **Legend**: Vertical colorbar on the right, labeled "Pass@1" with values 50–56 - **Model Labels**: - Left section: **R1-Llama** - Right section: **R1-Qwen** ### Detailed Analysis #### R1-Llama Section | Window Size | Ratio 0.1 | Ratio 0.2 | Ratio 0.3 | Ratio 0.4 | Ratio 0.5 | |-------------|-----------|-----------|-----------|-----------|-----------| | 3000 | 49.1 | 50.1 | 50.6 | 50.7 | 51.4 | | 2000 | 49.5 | 51.7 | 52.8 | 52.5 | 50.9 | | 1000 | 49.9 | 52.7 | 51.0 | 51.9 | 51.7 | | 500 | 49.8 | 52.1 | 50.7 | 50.8 | 51.7 | #### R1-Qwen Section | Window Size | Ratio 0.1 | Ratio 0.2 | Ratio 0.3 | Ratio 0.4 | Ratio 0.5 | |-------------|-----------|-----------|-----------|-----------|-----------| | 3000 | 53.9 | 53.9 | 53.2 | 54.4 | 53.8 | | 2000 | 52.4 | 51.9 | 54.6 | 56.3 | 53.7 | | 1000 | 52.2 | 54.4 | 53.8 | 53.3 | 53.0 | | 500 | 51.5 | 51.8 | 52.0 | 54.3 | 54.6 | ### Key Observations 1. **R1-Qwen Dominance**: R1-Qwen consistently outperforms R1-Llama across all configurations, with a maximum Pass@1 of **56.3** (2000 window size, 0.4 ratio) vs. R1-Llama's peak of **52.8**. 2. **Ratio Sensitivity**: Both models show improved performance with higher ratios, though R1-Qwen's gains are more pronounced (e.g., 51.5 → 54.6 for 500 window size). 3. **Window Size Tradeoffs**: - R1-Llama's performance peaks at smaller window sizes (500–1000) but declines at 2000/3000. - R1-Qwen maintains strong performance across all window sizes, with 2000 window size showing optimal results. 4. **Anomalies**: - R1-Llama's 3000 window size at 0.1 ratio (49.1) is the lowest value, suggesting a configuration mismatch. - R1-Qwen's 2000 window size at 0.4 ratio (56.3) stands out as the global maximum. ### Interpretation The data demonstrates that R1-Qwen exhibits superior scalability and efficiency compared to R1-Llama, particularly in high-ratio scenarios. The heatmap reveals that: - **R1-Qwen's robustness**: Maintains high Pass@1 across all window sizes, indicating better generalization. - **R1-Llama's limitations**: Struggles with larger window sizes, possibly due to computational constraints or architectural inefficiencies. - **Optimal configuration**: For R1-Qwen, the 2000 window size and 0.4 ratio yields the best results, suggesting a balance between context length and parameter utilization. The color gradient visually reinforces these trends, with darker blues correlating to higher Pass@1 values. The embedded numerical values confirm the heatmap's accuracy, while the spatial arrangement allows direct comparison between models. This analysis highlights R1-Qwen as the more versatile model for applications requiring adaptability across varying input sizes and ratios.