## Scatter Plot: Performance vs. Average Token Usage ### Overview The image is a scatter plot comparing the performance (in percentage) of different methods (ToT, SRA-MCTS, and Ours) against their average token usage. The data points are further categorized by the base models used (qwen3-8b, qwen3-235b-a22b, and claude37_sonnet). ### Components/Axes * **Title:** There is no explicit title. * **X-axis:** Average Token Usage, ranging from 0 to 20000, with gridlines at intervals of 4000. * **Y-axis:** Performance (%), ranging from 50 to 90, with gridlines at intervals of 10. * **Legend (Top-Left):** * **Methods:** * Blue circle: ToT * Green circle: SRA-MCTS * Orange circle: Ours * **Base Models:** * Gray circle: qwen3-8b * Gray triangle: qwen3-235b-a22b * Gray square: claude37\_sonnet ### Detailed Analysis The data points are scattered across the plot, each representing a specific method and base model combination. The position of each point indicates its performance and average token usage. * **ToT (Blue):** * ToT(qwen3-8b): Performance ~60%, Token Usage ~10000 * ToT(qwen3-235b-a22b): Performance ~68%, Token Usage ~13000 * ToT(claude37\_sonnet): Performance ~70%, Token Usage ~19000 * Trend: As the base model changes from qwen3-8b to qwen3-235b-a22b to claude37_sonnet, both performance and token usage increase. * **SRA-MCTS (Green):** * SRA-MCTS(qwen3-8b): Performance ~53%, Token Usage ~3000 * SRA-MCTS(qwen3-235b-a22b): Performance ~63%, Token Usage ~11000 * SRA-MCTS(claude37\_sonnet): Performance ~69%, Token Usage ~12000 * Trend: As the base model changes from qwen3-8b to qwen3-235b-a22b to claude37_sonnet, both performance and token usage increase. * **Ours (Orange):** * Ours(qwen3-8b): Performance ~64%, Token Usage ~7500 * Ours(qwen3-235b-a22b): Performance ~71%, Token Usage ~9000 * Ours(claude37\_sonnet): Performance ~75%, Token Usage ~9000 * Trend: As the base model changes from qwen3-8b to qwen3-235b-a22b to claude37_sonnet, performance increases, but token usage remains relatively stable. ### Key Observations * The "Ours" method generally achieves higher performance with lower token usage compared to "ToT" and "SRA-MCTS" when using the same base model. * Using the "claude37\_sonnet" base model generally results in higher performance but also higher token usage across all methods. * SRA-MCTS with qwen3-8b has the lowest token usage and performance. ### Interpretation The scatter plot visualizes the trade-off between performance and token usage for different methods and base models. The "Ours" method appears to be more efficient, achieving better performance with fewer tokens. The choice of base model significantly impacts both performance and token usage, with "claude37\_sonnet" generally leading to higher performance at the cost of increased token consumption. The data suggests that optimizing the method and base model selection is crucial for balancing performance and computational cost.

\n ## Scatter Plot: Performance vs. Token Usage for Different Methods and Base Models ### Overview This scatter plot visualizes the relationship between Performance (in percentage) and Average Token Usage for three different methods (ToT, SRA-MCTS, and Ours) applied to three different base models (qwen3-8b, qwen3-235b-a22b, and claude37_sonnet). Each data point represents a specific combination of method and base model. The plot aims to compare the efficiency and effectiveness of these methods. ### Components/Axes * **X-axis:** Average Token Usage (ranging from 0 to 20000, with increments of 4000). * **Y-axis:** Performance (%) (ranging from 50 to 90, with increments of 10). * **Legend (Top-Left):** * Methods: * ToT (Blue circles) * SRA-MCTS (Green squares) * Ours (Orange triangles) * Base Models: * qwen3-8b (Grey circles) * qwen3-235b-a22b (Grey triangles) * claude37_sonnet (Grey squares) * **Data Points:** Each point is labeled with the method and base model it represents (e.g., "ToT(claude37_sonnet)"). ### Detailed Analysis Here's a breakdown of the data points, categorized by method and base model, with approximate values read from the plot: **ToT (Blue Circles):** * ToT(qwen3-8b): Approximately (4000, 58%). Line slopes upward. * ToT(qwen3-235b-a22b): Approximately (12000, 68%). Line slopes upward. * ToT(claude37_sonnet): Approximately (20000, 72%). Line slopes upward. **SRA-MCTS (Green Squares):** * SRA-MCTS(qwen3-8b): Approximately (4000, 52%). Line slopes upward. * SRA-MCTS(qwen3-235b-a22b): Approximately (12000, 64%). Line slopes upward. * SRA-MCTS(claude37_sonnet): Approximately (12000, 68%). Line slopes upward. **Ours (Orange Triangles):** * Ours(qwen3-8b): Approximately (8000, 65%). Line slopes upward. * Ours(qwen3-235b-a22b): Approximately (8000, 72%). Line slopes upward. * Ours(claude37_sonnet): Approximately (8000, 81%). Line slopes upward. ### Key Observations * **Performance Trend:** Generally, performance increases with increasing Average Token Usage for all methods and base models. * **Method Comparison:** "Ours" consistently achieves the highest performance across all base models, especially with claude37_sonnet. * **Base Model Impact:** claude37_sonnet consistently yields the highest performance when combined with any of the methods. * **Efficiency:** SRA-MCTS and ToT achieve lower performance with lower token usage, while "Ours" requires more tokens but delivers significantly better performance. * **Outlier:** The point "Ours(claude37_sonnet)" stands out as having the highest performance (approximately 81%). ### Interpretation The data suggests that the "Ours" method is the most effective in terms of performance, but it comes at the cost of higher Average Token Usage. The choice of base model significantly impacts performance, with claude37_sonnet being the superior choice. The trade-off between performance and token usage is a key consideration. The upward slopes of the lines for each method indicate a positive correlation between token usage and performance. This suggests that increasing the computational resources (token usage) can lead to improved results. The clustering of points for each base model suggests that the base model itself is a significant factor in determining the overall performance. The fact that "Ours" consistently outperforms the other methods, particularly with the claude37_sonnet model, suggests that this combination is the most promising for achieving high performance in this task. The data also highlights the importance of considering the computational cost (token usage) when selecting a method and base model.

## Scatter Plot: Method Performance vs. Average Token Usage ### Overview This image is a scatter plot comparing the performance (in percentage) of three different methods (ToT, SRA-MCTS, and "Ours") across three different base models (qwen3-8b, qwen3-235b-a22b, claude37_sonnet). The plot visualizes the trade-off between performance and computational cost, measured by average token usage. ### Components/Axes * **Y-Axis:** Labeled "Performance (%)". Scale ranges from 50 to 90, with major tick marks every 10 units (50, 60, 70, 80, 90). * **X-Axis:** Labeled "Average Token Usage". Scale ranges from 0 to 20,000, with major tick marks every 4,000 units (0, 4000, 8000, 12000, 16000, 20000). * **Legend 1 (Top-Left):** Titled "Methods". Defines the color coding for the three methods: * Blue circle: ToT * Green circle: SRA-MCTS * Orange circle: Ours * **Legend 2 (Below Legend 1):** Titled "Base Models". Defines the shape coding for the three base models: * Circle: qwen3-8b * Triangle: qwen3-235b-a22b * Square: claude37_sonnet * **Data Points:** Each point is labeled with the format "Method(Base Model)". The color indicates the method, and the shape indicates the base model. ### Detailed Analysis The plot contains 9 distinct data points. Below is a reconstruction of each point's approximate coordinates and label, grouped by method. **Method: Ours (Orange)** 1. **Label:** Ours(qwen3-8b) * **Shape:** Circle * **Approx. Coordinates:** (7500, 63.5) * **Trend:** This is the lowest token usage point for the "Ours" method. 2. **Label:** Ours(qwen3-235b-a22b) * **Shape:** Triangle * **Approx. Coordinates:** (8500, 71) * **Trend:** Shows a significant performance increase over the 8b model with a modest increase in token usage. 3. **Label:** Ours(claude37_sonnet) * **Shape:** Square * **Approx. Coordinates:** (9500, 76) * **Trend:** The highest-performing point on the entire chart, using fewer tokens than many competing methods. **Method: SRA-MCTS (Green)** 1. **Label:** SRA-MCTS(qwen3-8b) * **Shape:** Circle * **Approx. Coordinates:** (2500, 52.5) * **Trend:** The lowest performance and lowest token usage point on the chart. 2. **Label:** SRA-MCTS(qwen3-235b-a22b) * **Shape:** Triangle * **Approx. Coordinates:** (10000, 61.5) * **Trend:** Higher performance and token usage than its 8b counterpart. 3. **Label:** SRA-MCTS(claude37_sonnet) * **Shape:** Square * **Approx. Coordinates:** (11500, 68.5) * **Trend:** The highest-performing configuration for SRA-MCTS. **Method: ToT (Blue)** 1. **Label:** ToT(qwen3-8b) * **Shape:** Circle * **Approx. Coordinates:** (9500, 59) * **Trend:** Uses significantly more tokens than SRA-MCTS for a similar performance level on the same base model. 2. **Label:** ToT(qwen3-235b-a22b) * **Shape:** Triangle * **Approx. Coordinates:** (12500, 67.5) * **Trend:** Higher performance and token usage than its 8b counterpart. 3. **Label:** ToT(claude37_sonnet) * **Shape:** Square * **Approx. Coordinates:** (18000, 70.5) * **Trend:** The highest token usage point on the chart by a large margin. ### Key Observations 1. **Performance Hierarchy:** For every base model, the "Ours" method achieves the highest performance, followed generally by ToT, then SRA-MCTS. 2. **Token Efficiency:** The "Ours" method demonstrates superior token efficiency. For example, "Ours(claude37_sonnet)" achieves ~76% performance with ~9500 tokens, while "ToT(claude37_sonnet)" achieves only ~70.5% performance with ~18000 tokens. 3. **Base Model Scaling:** All three methods show a consistent trend: performance increases when moving from the qwen3-8b (circle) to qwen3-235b-a22b (triangle) to claude37_sonnet (square) base model. Token usage also generally increases with model scale. 4. **Outlier:** The "ToT(claude37_sonnet)" point is a clear outlier in terms of token usage, positioned far to the right of all other data points. ### Interpretation This chart presents a compelling case for the efficacy of the "Ours" method. It suggests that this new approach achieves a better performance-to-cost ratio than the compared baselines (ToT and SRA-MCTS). The data demonstrates that "Ours" not only reaches higher peak performance but does so with greater computational efficiency (lower token usage). The relationship between the points indicates that while all methods benefit from more powerful base models, the "Ours" method leverages this increased model capacity more effectively, yielding greater performance gains per additional token spent. The significant rightward position of the ToT method, especially with the strongest base model, implies it may have a higher inherent computational overhead or a less efficient search strategy compared to the other methods evaluated. The chart effectively argues that the proposed method ("Ours") advances the state-of-the-art by optimizing both the quality of the solution and the resources required to find it.

## Scatter Plot: Performance vs. Average Token Usage ### Overview The image is a scatter plot comparing the performance (y-axis, %) of different methods against their average token usage (x-axis, tokens). Three methods are compared: ToT (blue), SRA-MCTS (green), and "Ours" (orange). Three base models are represented: qwen3-8b (circle), qwen3-235b-a22b (triangle), and claude37_sonnet (square). Data points are labeled with their method-base model combinations. ### Components/Axes - **X-axis**: "Average Token Usage" (0 to 20,000 tokens, increments of 4,000) - **Y-axis**: "Performance (%)" (50% to 90%, increments of 10%) - **Legend**: - Top-left corner, labeled "Methods": - Blue circle: ToT - Green square: SRA-MCTS - Orange triangle: Ours - **Base Models**: - qwen3-8b: Gray circle - qwen3-235b-a22b: Gray triangle - claude37_sonnet: Gray square ### Detailed Analysis 1. **SRA-MCTS(qwen3-8b)**: - Position: (3,000 tokens, 53%) - Color: Green square - Label: "SRA-MCTS(qwen3-8b)" 2. **ToT(qwen3-8b)**: - Position: (9,000 tokens, 58%) - Color: Blue circle - Label: "ToT(qwen3-8b)" 3. **Ours(qwen3-8b)**: - Position: (7,500 tokens, 64%) - Color: Orange triangle - Label: "Ours(qwen3-8b)" 4. **SRA-MCTS(qwen3-235b-a22b)**: - Position: (11,000 tokens, 62%) - Color: Green square - Label: "SRA-MCTS(qwen3-235b-a22b)" 5. **ToT(qwen3-235b-a22b)**: - Position: (13,000 tokens, 67%) - Color: Blue circle - Label: "ToT(qwen3-235b-a22b)" 6. **Ours(qwen3-235b-a22b)**: - Position: (8,500 tokens, 72%) - Color: Orange triangle - Label: "Ours(qwen3-235b-a22b)" 7. **SRA-MCTS(claude37_sonnet)**: - Position: (12,000 tokens, 68%) - Color: Green square - Label: "SRA-MCTS(claude37_sonnet)" 8. **ToT(claude37_sonnet)**: - Position: (19,000 tokens, 71%) - Color: Blue circle - Label: "ToT(claude37_sonnet)" 9. **Ours(claude37_sonnet)**: - Position: (8,000 tokens, 76%) - Color: Orange triangle - Label: "Ours(claude37_sonnet)" ### Key Observations - **Performance Trends**: - "Ours" method consistently achieves higher performance (64–76%) across all base models. - SRA-MCTS shows moderate performance (53–68%) but requires higher token usage (3,000–12,000 tokens). - ToT exhibits variable performance (58–71%) with the highest token usage (9,000–19,000 tokens). - **Token Efficiency**: - "Ours" achieves the best performance-to-token ratio, especially with claude37_sonnet (76% at 8,000 tokens). - ToT requires the most tokens for comparable performance (e.g., 19,000 tokens for 71% vs. 8,000 tokens for 76% with "Ours"). - **Outliers**: - SRA-MCTS(qwen3-8b) is the lowest-performing point (53% at 3,000 tokens). - ToT(claude37_sonnet) uses the most tokens (19,000) for only 71% performance. ### Interpretation The data suggests that the "Ours" method outperforms both ToT and SRA-MCTS in terms of performance while maintaining lower token usage. This indicates superior efficiency, particularly when paired with the claude37_sonnet base model. SRA-MCTS appears less efficient, requiring more tokens for similar or lower performance gains. ToT's performance scales with token usage but remains less efficient than "Ours." The results highlight a trade-off between computational resource consumption and output quality, with "Ours" offering the most favorable balance.