## Chart: Cumulative Percentage of Runs Beating SOTA by LLM Calls ### Overview The image is a cumulative percentage graph comparing the performance of "Cheap LLM" and "Expensive LLM" in terms of the percentage of runs beating the State-of-the-Art (SOTA) as a function of the number of LLM calls. The graph shows two step-like lines, one for each LLM, plotting the cumulative percentage of runs that outperform SOTA as the number of LLM calls increases. ### Components/Axes * **Title:** Cumulative Percentage of Runs Beating SOTA by LLM Calls * **X-axis:** Number of LLM Calls, ranging from 0 to 3,000 in increments of 500. * **Y-axis:** % of Runs Beating SOTA, ranging from 0% to 100% in increments of 20%. * **Legend:** Located in the top-left corner. * **Blue:** Cheap LLM * **Orange:** Expensive LLM ### Detailed Analysis * **Cheap LLM (Blue):** The blue line represents the cumulative percentage of runs beating SOTA for the Cheap LLM. The line generally slopes upward, indicating that as the number of LLM calls increases, the percentage of runs beating SOTA also increases. * At 0 LLM calls, the percentage is approximately 0%. * At 500 LLM calls, the percentage is approximately 20%. * At 1000 LLM calls, the percentage is approximately 45%. * At 1500 LLM calls, the percentage is approximately 70%. * At 2000 LLM calls, the percentage is approximately 75%. * At 2500 LLM calls, the percentage is approximately 75%. * At 3000 LLM calls, the percentage is approximately 75%. * **Expensive LLM (Orange):** The orange line represents the cumulative percentage of runs beating SOTA for the Expensive LLM. The line generally slopes upward, indicating that as the number of LLM calls increases, the percentage of runs beating SOTA also increases. * At 0 LLM calls, the percentage is approximately 0%. * At 500 LLM calls, the percentage is approximately 25%. * At 1000 LLM calls, the percentage is approximately 65%. * At 1500 LLM calls, the percentage is approximately 95%. ### Key Observations * The Expensive LLM generally outperforms the Cheap LLM at lower numbers of LLM calls. * Both LLMs show an increase in the percentage of runs beating SOTA as the number of LLM calls increases. * The Expensive LLM reaches a higher percentage of runs beating SOTA compared to the Cheap LLM. * The Cheap LLM appears to plateau around 75% after 2000 LLM calls. ### Interpretation The data suggests that using a more expensive LLM leads to a higher percentage of runs beating the State-of-the-Art, especially with fewer LLM calls. The Expensive LLM achieves a higher performance level overall. The Cheap LLM's performance plateaus, indicating that increasing the number of calls beyond a certain point does not significantly improve its ability to beat SOTA. This could be due to limitations in the model's architecture or training data. The Expensive LLM continues to improve with more calls, suggesting it can leverage additional calls more effectively.

## Chart: Cumulative Percentage of Runs Beating SOTA by LLM Calls ### Overview The image presents a chart illustrating the cumulative percentage of runs where Large Language Models (LLMs) outperform the State-of-the-Art (SOTA) based on the number of LLM calls made. Two LLM types are compared: a "Cheap LLM" and an "Expensive LLM". The chart is a cumulative distribution plot, showing how the percentage of successful runs increases with the number of LLM calls. ### Components/Axes * **Title:** "Cumulative Percentage of Runs Beating SOTA by LLM Calls" (Top-center) * **X-axis:** "Number of LLM Calls" (Bottom-center), ranging from 0 to 3000, with markers at 0, 500, 1000, 1500, 2000, 2500, and 3000. * **Y-axis:** "% of Runs Beating SOTA" (Left-center), ranging from 0% to 100%, with markers at 0%, 20%, 40%, 60%, 80%, and 100%. * **Legend:** Located in the top-left corner. * "Cheap LLM" - represented by a blue line. * "Expensive LLM" - represented by an orange line. ### Detailed Analysis The chart displays two cumulative distribution curves. **Cheap LLM (Blue Line):** The blue line starts at approximately 0% at 0 LLM calls. It rises relatively slowly until around 500 LLM calls, reaching approximately 20%. The curve then increases more rapidly between 500 and 1500 LLM calls, reaching approximately 80% at 1500 calls. The curve plateaus between 1500 and 3000 LLM calls, reaching approximately 92% at 3000 calls. * 0 LLM Calls: ~0% * 500 LLM Calls: ~20% * 1000 LLM Calls: ~50% * 1500 LLM Calls: ~80% * 2000 LLM Calls: ~86% * 2500 LLM Calls: ~90% * 3000 LLM Calls: ~92% **Expensive LLM (Orange Line):** The orange line also starts at approximately 0% at 0 LLM calls. It rises quickly between 0 and 500 LLM calls, reaching approximately 20% at 500 calls. The curve continues to increase rapidly between 500 and 1000 LLM calls, reaching approximately 60% at 1000 calls. The curve then slows down, reaching approximately 90% at 1500 LLM calls, and plateaus around 95% between 1500 and 3000 LLM calls. * 0 LLM Calls: ~0% * 500 LLM Calls: ~20% * 1000 LLM Calls: ~60% * 1500 LLM Calls: ~90% * 2000 LLM Calls: ~93% * 2500 LLM Calls: ~95% * 3000 LLM Calls: ~95% ### Key Observations * The "Expensive LLM" generally outperforms the "Cheap LLM" in terms of the cumulative percentage of runs beating SOTA, especially at lower numbers of LLM calls. * Both LLMs exhibit diminishing returns as the number of LLM calls increases. The rate of improvement slows down significantly after a certain point. * The "Expensive LLM" reaches a higher plateau than the "Cheap LLM", indicating that it is more likely to achieve high performance even with a large number of calls. ### Interpretation The data suggests that while both LLM types can outperform the SOTA, the "Expensive LLM" is more efficient in doing so, requiring fewer LLM calls to achieve a given level of performance. The diminishing returns observed for both LLMs indicate that there is a limit to the benefit of increasing the number of LLM calls. This could be due to factors such as the inherent limitations of the LLM architecture, the quality of the training data, or the complexity of the task. The chart highlights the trade-off between cost (LLM calls) and performance (percentage of runs beating SOTA). The "Expensive LLM" represents a higher upfront cost but potentially lower overall cost due to its efficiency. The plateauing of both curves suggests that further investment in LLM calls beyond a certain point may not yield significant improvements in performance.

## Line Chart: Cumulative Percentage of Runs Beating SOTA by LLM Calls ### Overview This is a line chart comparing the performance of two types of Large Language Models (LLMs) — labeled "Cheap LLM" and "Expensive LLM" — in terms of the cumulative percentage of experimental runs that beat the State-of-the-Art (SOTA) benchmark, plotted against the number of LLM calls made. The chart demonstrates the efficiency of each model type in achieving superior results with varying computational effort. ### Components/Axes * **Title:** "Cumulative Percentage of Runs Beating SOTA by LLM Calls" * **X-Axis (Horizontal):** * **Label:** "Number of LLM Calls" * **Scale:** Linear scale from 0 to 3,000. * **Major Tick Marks:** 0, 500, 1,000, 1,500, 2,000, 2,500, 3,000. * **Y-Axis (Vertical):** * **Label:** "% of Runs Beating SOTA" * **Scale:** Linear scale from 0% to 100%. * **Major Tick Marks:** 0%, 20%, 40%, 60%, 80%, 100%. * **Legend:** * **Position:** Top-left corner of the chart area. * **Entry 1:** "Cheap LLM" — represented by a solid blue line. * **Entry 2:** "Expensive LLM" — represented by a solid orange line. ### Detailed Analysis The chart plots two cumulative distribution functions. **1. Expensive LLM (Orange Line):** * **Trend:** The line exhibits a very steep, near-vertical initial ascent, indicating rapid gains in the percentage of successful runs with a relatively small number of calls. The slope gradually decreases but remains strong until it reaches the 100% ceiling. * **Key Data Points (Approximate):** * At ~100 calls: ~25% of runs beat SOTA. * At ~500 calls: ~60% of runs beat SOTA. * At ~1,000 calls: ~80% of runs beat SOTA. * The line reaches 100% at approximately 1,400 calls and remains flat thereafter. **2. Cheap LLM (Blue Line):** * **Trend:** The line shows a more gradual, steady ascent. It requires significantly more calls to reach the same cumulative percentages as the Expensive LLM. The curve begins to plateau after approximately 2,000 calls. * **Key Data Points (Approximate):** * At ~100 calls: ~5% of runs beat SOTA. * At ~500 calls: ~30% of runs beat SOTA. * At ~1,000 calls: ~60% of runs beat SOTA. * At ~2,000 calls: ~90% of runs beat SOTA. * The line plateaus at approximately 95% from ~2,500 calls onward, never reaching 100% within the displayed range. ### Key Observations 1. **Performance Gap:** There is a substantial and consistent gap between the two lines. For any given number of LLM calls, the Expensive LLM has a significantly higher cumulative success rate. 2. **Efficiency:** The Expensive LLM is far more "call-efficient." It achieves a 60% success rate with ~500 calls, a milestone the Cheap LLM requires ~1,000 calls to reach. 3. **Ceiling Effect:** The Expensive LLM reaches 100% success within the observed window (~1,400 calls). The Cheap LLM appears to asymptote just below 100% (around 95%), suggesting a subset of runs where it may never beat SOTA, regardless of additional calls. 4. **Shape Difference:** The Expensive LLM's curve is concave (steep then flattening), while the Cheap LLM's curve is more linear for a longer duration before flattening. ### Interpretation This chart visualizes a classic trade-off between cost (implied by "Cheap" vs. "Expensive") and computational efficiency in AI model performance. The data suggests that investing in a more expensive LLM yields a disproportionately higher return on investment in terms of task success per unit of computational effort (LLM call). The "Expensive LLM" likely has superior reasoning, knowledge, or instruction-following capabilities, allowing it to solve the target problem correctly on the first or second attempt much more frequently. The "Cheap LLM" may require more iterative calls, self-correction, or sampling to arrive at a correct solution, hence needing more calls to accumulate successes. The plateau of the Cheap LLM below 100% is particularly noteworthy. It implies an inherent limitation or a class of problems within the benchmark that this model type cannot solve, no matter how many attempts are granted. In contrast, the Expensive LLM's ability to reach 100% indicates it is robust enough to eventually solve all problem instances given sufficient calls. From a practical standpoint, this analysis would inform resource allocation: if the cost of an "Expensive LLM" call is less than roughly double the cost of a "Cheap LLM" call (based on the ~2x call efficiency at the 60% mark), it would be the more cost-effective choice for achieving high reliability. The chart argues that raw cost-per-call is a misleading metric without considering the resulting success rate.

## Line Chart: Cumulative Percentage of Runs Beating SOTA by LLM Calls ### Overview The chart compares the performance of two language models (Cheap LLM and Expensive LLM) in terms of the percentage of runs that beat a state-of-the-art (SOTA) benchmark as a function of the number of LLM calls made. The data is presented as cumulative percentages, with two distinct lines representing each model's performance trajectory. ### Components/Axes - **X-axis**: "Number of LLM Calls" (0 to 3,000, in increments of 500). - **Y-axis**: "% of Runs Beating SOTA" (0% to 100%, in increments of 20%). - **Legend**: Located in the top-right corner, with: - **Blue line**: "Cheap LLM" - **Orange line**: "Expensive LLM" ### Detailed Analysis 1. **Cheap LLM (Blue Line)**: - Starts at ~0% at 0 calls. - Gradually increases, reaching ~20% at 500 calls. - Accelerates growth, hitting ~60% at 1,000 calls. - Crosses the Expensive LLM line near 1,500 calls (~70%). - Reaches ~85% at 2,000 calls. - Plateaus at ~95% by 2,500 calls, stabilizing at 100% by 3,000 calls. 2. **Expensive LLM (Orange Line)**: - Starts at ~0% at 0 calls. - Rises sharply, reaching ~40% at 500 calls. - Accelerates further, hitting ~80% at 1,000 calls. - Peaks at 100% near 1,500 calls. - Remains at 100% for all subsequent call counts (1,500–3,000). ### Key Observations - The **Expensive LLM** achieves 100% performance significantly earlier (~1,500 calls) compared to the **Cheap LLM** (~3,000 calls). - The **Cheap LLM** overtakes the Expensive LLM in performance around 1,500 calls, suggesting diminishing returns for the Expensive LLM beyond this point. - Both models plateau at 100% performance, but the Cheap LLM requires more calls to reach this threshold. ### Interpretation The data suggests that while the Expensive LLM delivers faster initial gains, the Cheap LLM becomes more effective at higher call volumes, potentially due to optimization, learning, or cost-efficiency tradeoffs. The crossover point (~1,500 calls) highlights a critical threshold where cost considerations may outweigh performance benefits for the Expensive LLM. This could inform decisions about resource allocation in LLM deployment, favoring cheaper models for large-scale applications where marginal gains from expensive models are negligible.