## Chart: Response Lengths and Accuracy Across Experiments ### Overview The image presents two line charts comparing the performance of different reward mechanisms (MaxDiff Reward, R1-Zero Reward + Budget Prompt, R1-Zero Reward, and MaxLength Reward) across varying "Requested Thinking Budgets." The left chart displays "Response Length (tokens)," while the right chart shows "Accuracy." ### Components/Axes **Left Chart: Response Lengths Across Experiments** * **Title:** Response Lengths Across Experiments * **Y-axis:** Response Length (tokens) * Scale: 100 to 800, incrementing by 100. * **X-axis:** Requested Thinking Budget * Scale: 100 to 900, incrementing by 100. * **Legend:** Located in the top-left corner. * Blue: MaxDiff Reward * Orange: R1-Zero Reward + Budget Prompt * Green: R1-Zero Reward * Red: MaxLength Reward **Right Chart: Accuracy Across Experiments** * **Title:** Accuracy Across Experiments * **Y-axis:** Accuracy * Scale: 0.35 to 0.65, incrementing by 0.05. * **X-axis:** Requested Thinking Budget * Scale: 100 to 900, incrementing by 100. * **Legend:** Located in the bottom-left corner. * Blue: MaxDiff Reward * Orange: R1-Zero Reward + Budget Prompt * Green: R1-Zero Reward * Red: MaxLength Reward ### Detailed Analysis **Left Chart: Response Lengths Across Experiments** * **MaxDiff Reward (Blue):** The line slopes upward significantly from approximately 150 tokens at a budget of 100 to approximately 810 tokens at a budget of 800, then plateaus at approximately 810 tokens at a budget of 900. * (100, 150) * (200, 200) * (300, 280) * (400, 390) * (500, 490) * (600, 610) * (700, 720) * (800, 810) * (900, 810) * **R1-Zero Reward + Budget Prompt (Orange):** The line is relatively flat, fluctuating slightly around 420 tokens across all budget levels. * (100, 420) * (200, 420) * (300, 430) * (400, 420) * (500, 420) * (600, 420) * (700, 420) * (800, 420) * (900, 420) * **R1-Zero Reward (Green):** The line is relatively flat, fluctuating slightly around 290 tokens across all budget levels. * (100, 290) * (200, 290) * (300, 290) * (400, 290) * (500, 290) * (600, 290) * (700, 290) * (800, 300) * (900, 290) * **MaxLength Reward (Red):** The line is relatively flat, fluctuating slightly around 70 tokens across all budget levels. * (100, 70) * (200, 70) * (300, 70) * (400, 70) * (500, 70) * (600, 70) * (700, 70) * (800, 70) * (900, 70) **Right Chart: Accuracy Across Experiments** * **MaxDiff Reward (Blue):** The line slopes upward from approximately 0.45 at a budget of 100 to approximately 0.60 at a budget of 800, then plateaus at approximately 0.60 at a budget of 900. * (100, 0.45) * (200, 0.49) * (300, 0.55) * (400, 0.58) * (500, 0.59) * (600, 0.59) * (700, 0.60) * (800, 0.60) * (900, 0.60) * **R1-Zero Reward + Budget Prompt (Orange):** The line is relatively flat, fluctuating slightly around 0.65 across all budget levels. * (100, 0.65) * (200, 0.65) * (300, 0.65) * (400, 0.65) * (500, 0.65) * (600, 0.65) * (700, 0.66) * (800, 0.65) * (900, 0.66) * **R1-Zero Reward (Green):** The line is relatively flat, fluctuating slightly around 0.62 across all budget levels. * (100, 0.62) * (200, 0.61) * (300, 0.61) * (400, 0.62) * (500, 0.62) * (600, 0.61) * (700, 0.62) * (800, 0.62) * (900, 0.62) * **MaxLength Reward (Red):** The line is relatively flat, fluctuating slightly around 0.35 across all budget levels. * (100, 0.34) * (200, 0.34) * (300, 0.35) * (400, 0.34) * (500, 0.35) * (600, 0.34) * (700, 0.34) * (800, 0.34) * (900, 0.35) ### Key Observations * **Response Length:** The MaxDiff Reward mechanism shows a significant increase in response length as the requested thinking budget increases, while the other mechanisms maintain relatively constant response lengths. * **Accuracy:** The R1-Zero Reward + Budget Prompt mechanism consistently achieves the highest accuracy across all budget levels. The MaxDiff Reward mechanism shows an increase in accuracy with an increasing budget, but it plateaus. The MaxLength Reward mechanism consistently has the lowest accuracy. * **Trade-off:** There appears to be a trade-off between response length and accuracy. The MaxDiff Reward mechanism increases response length but does not achieve the highest accuracy. The MaxLength Reward mechanism produces the shortest responses but also the lowest accuracy. ### Interpretation The data suggests that the choice of reward mechanism significantly impacts both the response length and accuracy of the model. The MaxDiff Reward mechanism encourages longer responses as the thinking budget increases, which may be beneficial in some contexts. However, the R1-Zero Reward + Budget Prompt mechanism consistently achieves the highest accuracy, suggesting it may be the most effective for generating accurate responses. The MaxLength Reward mechanism appears to be the least effective, as it produces short and inaccurate responses. The relationship between response length and accuracy is complex and depends on the specific task and reward mechanism. In this case, simply increasing the response length does not necessarily lead to higher accuracy. The R1-Zero Reward + Budget Prompt mechanism seems to strike a better balance between response length and accuracy.

## Charts: Response Length and Accuracy Across Experiments ### Overview The image presents two line charts side-by-side. The left chart displays "Response Lengths Across Experiments" with response length in tokens on the y-axis and requested thinking budget on the x-axis. The right chart shows "Accuracy Across Experiments" with accuracy on the y-axis and requested thinking budget on the x-axis. Both charts compare the performance of different reward strategies. ### Components/Axes **Left Chart (Response Length):** * **Title:** Response Lengths Across Experiments * **X-axis Label:** Requested Thinking Budget (ranging from 100 to 900, with markers at 100, 200, 300, 400, 500, 600, 700, 800, 900) * **Y-axis Label:** Response Length (tokens) (ranging from 0 to 800, with markers at 100, 200, 300, 400, 500, 600, 700, 800) * **Legend:** * MaxDiff Reward (Blue) * R1-Zero Reward + Budget Prompt (Orange) * R1-Zero Reward (Green) * MaxLength Reward (Red) **Right Chart (Accuracy):** * **Title:** Accuracy Across Experiments * **X-axis Label:** Requested Thinking Budget (ranging from 100 to 900, with markers at 100, 200, 300, 400, 500, 600, 700, 800, 900) * **Y-axis Label:** Accuracy (ranging from 0.30 to 0.70, with markers at 0.35, 0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 0.70) * **Legend:** * MaxDiff Reward (Blue) * R1-Zero Reward + Budget Prompt (Orange) * R1-Zero Reward (Green) * MaxLength Reward (Red) ### Detailed Analysis or Content Details **Left Chart (Response Length):** * **MaxDiff Reward (Blue):** The line slopes sharply upward, starting at approximately 300 tokens at a budget of 100, and reaching approximately 800 tokens at a budget of 900. * **R1-Zero Reward + Budget Prompt (Orange):** The line is relatively flat, starting at approximately 450 tokens at a budget of 100, and remaining around 450-500 tokens throughout the budget range, ending at approximately 475 tokens at a budget of 900. * **R1-Zero Reward (Green):** The line is nearly flat, starting at approximately 425 tokens at a budget of 100, and remaining around 425-450 tokens throughout the budget range, ending at approximately 425 tokens at a budget of 900. * **MaxLength Reward (Red):** The line is almost perfectly flat, remaining around 100 tokens throughout the budget range, ending at approximately 100 tokens at a budget of 900. **Right Chart (Accuracy):** * **MaxDiff Reward (Blue):** The line slopes upward, starting at approximately 0.43 at a budget of 100, and reaching approximately 0.63 at a budget of 900. * **R1-Zero Reward + Budget Prompt (Orange):** The line fluctuates around 0.65, starting at approximately 0.68 at a budget of 100, dipping to approximately 0.63 at a budget of 400, and ending at approximately 0.65 at a budget of 900. * **R1-Zero Reward (Green):** The line fluctuates around 0.62, starting at approximately 0.63 at a budget of 100, dipping to approximately 0.60 at a budget of 400, and ending at approximately 0.62 at a budget of 900. * **MaxLength Reward (Red):** The line is relatively flat, starting at approximately 0.35 at a budget of 100, and remaining around 0.35-0.40 throughout the budget range, ending at approximately 0.37 at a budget of 900. ### Key Observations * The MaxDiff Reward strategy results in significantly longer responses as the thinking budget increases. * The MaxLength Reward strategy produces consistently short responses, regardless of the thinking budget. * Accuracy generally increases with the thinking budget for the MaxDiff Reward strategy. * The R1-Zero Reward and R1-Zero Reward + Budget Prompt strategies show relatively stable accuracy and response length across different thinking budgets. * The MaxLength Reward strategy consistently exhibits the lowest accuracy. ### Interpretation The data suggests that increasing the thinking budget has a substantial impact on the length of responses generated by the MaxDiff Reward strategy, while having a moderate impact on its accuracy. The MaxLength Reward strategy prioritizes brevity over accuracy, resulting in short responses with consistently low accuracy. The R1-Zero Reward strategies offer a balance between response length and accuracy, with relatively stable performance across different thinking budgets. The relationship between the two charts is clear: as response length increases (driven by the MaxDiff Reward), accuracy also tends to increase. This suggests that allowing the model more "thinking time" (through a higher budget) and rewarding differentiation (MaxDiff) leads to more informative and accurate responses, but at the cost of increased token usage. The other strategies demonstrate that simply increasing the budget does not necessarily improve accuracy if the reward function doesn't incentivize it. The flat lines for R1-Zero strategies suggest a limited capacity to benefit from increased budget under those reward schemes. The MaxLength strategy shows that constraining response length severely impacts accuracy. Anomalies are not readily apparent, but the slight fluctuations in the R1-Zero Reward strategies could warrant further investigation to understand the factors influencing those variations.

\n ## Line Charts: Response Lengths and Accuracy Across Experiments ### Overview The image displays two side-by-side line charts comparing the performance of four different reward methods ("MaxDiff Reward", "R1-Zero Reward + Budget Prompt", "R1-Zero Reward", "MaxLength Reward") across a range of "Requested Thinking Budget" values. The left chart tracks the resulting response length in tokens, while the right chart tracks the accuracy of the responses. ### Components/Axes **Shared Elements:** * **X-Axis (Both Charts):** Labeled "Requested Thinking Budget". The scale runs from 100 to 900 in increments of 100. * **Legend (Both Charts):** Located in the top-left corner of each chart's plot area. It defines four data series: * **MaxDiff Reward:** Blue line with circular markers. * **R1-Zero Reward + Budget Prompt:** Orange line with circular markers. * **R1-Zero Reward:** Green line with circular markers. * **MaxLength Reward:** Red line with circular markers. **Left Chart: Response Lengths Across Experiments** * **Title:** "Response Lengths Across Experiments" * **Y-Axis:** Labeled "Response Length (tokens)". The scale runs from 0 to 800 in increments of 100. **Right Chart: Accuracy Across Experiments** * **Title:** "Accuracy Across Experiments" * **Y-Axis:** Labeled "Accuracy". The scale runs from 0.35 to 0.65 in increments of 0.05. ### Detailed Analysis **Left Chart - Response Length Trends:** 1. **MaxDiff Reward (Blue):** Shows a strong, nearly linear upward trend. It starts at approximately 150 tokens at a budget of 100 and increases steadily to approximately 800 tokens at a budget of 900. 2. **R1-Zero Reward + Budget Prompt (Orange):** Exhibits a flat trend. The response length remains stable at approximately 450 tokens across all thinking budget values from 100 to 900. 3. **R1-Zero Reward (Green):** Also exhibits a flat trend. The response length remains stable at approximately 300 tokens across all thinking budget values. 4. **MaxLength Reward (Red):** Exhibits a flat trend at the lowest level. The response length remains stable at approximately 100 tokens across all thinking budget values. **Right Chart - Accuracy Trends:** 1. **MaxDiff Reward (Blue):** Shows a strong upward trend. Accuracy starts at approximately 0.45 at a budget of 100 and increases steadily, converging with the green line at approximately 0.62 by a budget of 900. 2. **R1-Zero Reward + Budget Prompt (Orange):** Exhibits a flat, high trend. Accuracy remains stable at approximately 0.65 across all thinking budget values. 3. **R1-Zero Reward (Green):** Exhibits a flat, high trend. Accuracy remains stable at approximately 0.62 across all thinking budget values. 4. **MaxLength Reward (Red):** Exhibits a flat, low trend. Accuracy remains stable at approximately 0.35 across all thinking budget values. ### Key Observations * **Divergent Response Length Behavior:** Only the "MaxDiff Reward" method shows a response length that scales with the thinking budget. The other three methods produce responses of fixed length regardless of the budget allocated. * **Accuracy Convergence:** The accuracy of the "MaxDiff Reward" method improves significantly with budget, eventually matching the accuracy of the fixed-length "R1-Zero Reward" method at the highest budget levels. * **Performance Hierarchy:** In terms of accuracy, a clear hierarchy is maintained across all budgets: "R1-Zero Reward + Budget Prompt" (highest) > "R1-Zero Reward" > "MaxDiff Reward" (at low budgets) / "MaxLength Reward" (lowest). * **MaxLength Reward Trade-off:** The "MaxLength Reward" method consistently produces the shortest responses and achieves the lowest accuracy, indicating a potential trade-off between extreme brevity and performance. ### Interpretation The data suggests a fundamental difference in how these reward methods operate and respond to increased computational resources (the "thinking budget"). * **MaxDiff Reward** appears to be a **budget-aware, scaling method**. It utilizes the additional budget to generate longer, more detailed responses, which in turn leads to higher accuracy. Its performance is directly tied to the resources provided. * **R1-Zero Reward** and its variant with a **Budget Prompt** are **budget-agnostic, fixed-output methods**. They produce responses of a predetermined length and quality, irrespective of the available thinking budget. The "Budget Prompt" variant consistently achieves the highest accuracy, suggesting it is the most effective fixed-length strategy among those tested. * **MaxLength Reward** acts as a **strict length constraint**, capping responses at a very low token count. This severe constraint appears to limit the model's ability to reason or elaborate, resulting in consistently poor accuracy. The charts demonstrate that for the "MaxDiff Reward" approach, there is a positive correlation between allowed thinking budget, response length, and accuracy. For the other methods, accuracy is decoupled from the thinking budget, as their output length is fixed. This highlights a key design choice in AI system optimization: whether to allow flexible, resource-dependent output or to enforce fixed, predictable output characteristics.

## Line Graphs: Response Lengths and Accuracy Across Experiments ### Overview The image contains two line graphs comparing performance metrics (response length and accuracy) across different reward strategies as a function of "Requested Thinking Budget" (x-axis). The graphs use four distinct data series, each represented by a unique color and labeled in the legend. --- ### Components/Axes #### Left Graph: Response Lengths Across Experiments - **X-axis**: "Requested Thinking Budget" (range: 100–900, increments of 100) - **Y-axis**: "Response Length (tokens)" (range: 100–800, increments of 100) - **Legend**: - **Blue**: MaxDiff Reward - **Orange**: R1-Zero Reward + Budget Prompt - **Green**: R1-Zero Reward - **Red**: MaxLength Reward #### Right Graph: Accuracy Across Experiments - **X-axis**: "Requested Thinking Budget" (same scale as left graph) - **Y-axis**: "Accuracy" (range: 0.35–0.65, increments of 0.05) - **Legend**: Same as left graph (blue, orange, green, red) --- ### Detailed Analysis #### Left Graph: Response Lengths 1. **MaxDiff Reward (Blue)**: - Starts at ~150 tokens (budget=100) and increases linearly to ~800 tokens (budget=900). - Slope: ~7.2 tokens per unit budget (calculated from (800-150)/(900-100)). 2. **R1-Zero Reward + Budget Prompt (Orange)**: - Flat line at ~420–450 tokens across all budgets. 3. **R1-Zero Reward (Green)**: - Flat line at ~300 tokens across all budgets. 4. **MaxLength Reward (Red)**: - Flat line at ~50 tokens across all budgets. #### Right Graph: Accuracy 1. **MaxDiff Reward (Blue)**: - Starts at ~0.45 (budget=100) and increases to ~0.65 (budget=900). - Slope: ~0.0022 per unit budget ((0.65-0.45)/(900-100)). 2. **R1-Zero Reward + Budget Prompt (Orange)**: - Starts at ~0.65, dips slightly to ~0.63 (budget=300), then stabilizes at ~0.65. 3. **R1-Zero Reward (Green)**: - Starts at ~0.62, peaks at ~0.64 (budget=500), then stabilizes at ~0.63. 4. **MaxLength Reward (Red)**: - Starts at ~0.35, peaks at ~0.37 (budget=500), then declines to ~0.34. --- ### Key Observations 1. **Response Length**: - MaxDiff Reward scales linearly with budget, while other strategies show no budget-dependent growth. - MaxLength Reward produces the shortest responses (~50 tokens) regardless of budget. 2. **Accuracy**: - MaxDiff Reward improves significantly with budget, achieving ~0.65 accuracy at budget=900. - R1-Zero Reward + Budget Prompt maintains the highest baseline accuracy (~0.65) but shows no improvement with budget. - MaxLength Reward underperforms in accuracy, peaking at ~0.37. --- ### Interpretation 1. **MaxDiff Reward**: - Demonstrates strong scalability: both response length and accuracy improve proportionally with budget. This suggests it effectively balances depth and correctness. 2. **R1-Zero Reward + Budget Prompt**: - Maintains high accuracy without requiring additional budget, indicating efficiency but limited adaptability to increased computational resources. 3. **MaxLength Reward**: - Prioritizes brevity over accuracy, producing minimal responses with suboptimal performance. Likely unsuitable for tasks requiring detailed reasoning. 4. **R1-Zero Reward**: - Balances moderate response length and accuracy but shows no improvement with budget, suggesting diminishing returns. The data implies that **MaxDiff Reward** is the most effective strategy for tasks requiring scalable, high-quality outputs, while **R1-Zero Reward + Budget Prompt** offers a stable, efficient alternative for fixed-budget scenarios.