## Line Chart: pass@1-with-n-queries ### Overview The chart compares the performance of two systems (COPRA and ReProver) with and without retrieval mechanisms across increasing numbers of queries (n). The y-axis measures the "Number of Theorems Proved," while the x-axis tracks the "Number of Queries (n)" from 0 to 3500. Four data series are plotted, differentiated by color and retrieval status. ### Components/Axes - **X-axis**: "Number of Queries (n)" with markers at 0, 500, 1000, 1500, 2000, 2500, 3000, 3500. - **Y-axis**: "Number of Theorems Proved" with markers at 0, 10, 20, ..., 70. - **Legend**: Located in the bottom-right corner, mapping colors to systems: - **Orange**: COPRA (GPT-4-turbo) (with Retrieval) - **Blue**: ReProver (with Retrieval) - **Green**: COPRA (GPT-4-turbo) (without Retrieval) - **Red**: ReProver (without Retrieval) ### Detailed Analysis 1. **COPRA (GPT-4-turbo) with Retrieval (Orange)**: - Starts at ~70 theorems proved at n=0. - Remains flat throughout, maintaining ~70 theorems proved across all n. - Highest performance across all query ranges. 2. **COPRA (GPT-4-turbo) without Retrieval (Green)**: - Starts at ~60 theorems proved at n=0. - Remains flat, maintaining ~60 theorems proved across all n. - Second-highest performance, consistently trailing the orange line by ~10 theorems. 3. **ReProver with Retrieval (Blue)**: - Starts at ~50 theorems proved at n=0. - Gradually increases to ~60 theorems proved by n=3500. - Shows steady improvement but lags behind COPRA variants. 4. **ReProver without Retrieval (Red)**: - Starts at ~40 theorems proved at n=0. - Gradually increases to ~50 theorems proved by n=3500. - Lowest performance, with minimal improvement over queries. ### Key Observations - **Performance Gaps**: COPRA with Retrieval (orange) outperforms all other series by a margin of ~10–20 theorems across all n. - **Retrieval Impact**: Systems with retrieval (orange and blue) outperform their counterparts without retrieval (green and red) by ~10–20 theorems. - **COPRA Dominance**: COPRA maintains superiority even without retrieval (green vs. red), suggesting inherent architectural advantages. - **ReProver Scalability**: ReProver with Retrieval (blue) shows the most significant improvement (~10 theorems) as n increases, indicating better scalability with query volume. ### Interpretation The data demonstrates that **retrieval mechanisms significantly enhance theorem-proving performance** for both systems. COPRA’s consistent lead—even without retrieval—highlights its robustness, while ReProver’s gradual improvement with retrieval suggests it benefits more from additional query volume. The flat performance of COPRA variants implies diminishing returns at higher query counts, whereas ReProver’s upward trend indicates potential for further gains with increased n. This aligns with the "pass@1" metric’s focus on early-query efficiency, where COPRA’s retrieval-augmented system achieves near-optimal results immediately.