## Radar Charts: Model Performance on GPQA Benchmarks ### Overview The image presents three radar charts comparing the performance of various language models on the GPQA benchmark, across different settings: a baseline model (Base), a model with intrinsic correction (S1), and a model with external correction (S2). Each chart visualizes the performance of multiple models across several tasks, including CS-QA, GSM8K, HotpotQA, AQUA, HumanEval, and MATH. ### Components/Axes * **Chart Type**: Radar Charts (3 charts side-by-side) * **Titles**: * Left Chart: "Base (Baseline) GPQA" * Middle Chart: "S1 (Intrinsic Correction) GPQA" * Right Chart: "S2 (External Correction) GPQA" * **Axes**: * Radial Axis: Represents performance score, ranging from 0.0 to 0.8, with markers at 0.2, 0.4, 0.6, 0.8. * Angular Axis: Represents different tasks/benchmarks: CS-QA, GSM8K, HotpotQA, AQUA, HumanEval, MATH. These are arranged clockwise around the circle. * **Legend**: Located at the bottom of the image. Lists the models and their corresponding line colors: * Light Blue: LLaMA3.1-8B-Instruct * Light Yellow: LLaMA3.1-70B-Instruct * Light Purple: Qwen2.5-7B-Instruct * Light Red: Qwen2.5-72B-Instruct * Darker Blue: Claude3.5-Sonnet * Orange: GPT-3.5 * Green: GPT-4o * Dashed Light Purple: QWQ-32B-Instruct * Dashed Dark Blue: DeepSeek-V3 * Dashed Light Blue: DeepSeek-R1 * Dashed Light Pink: o3-mini ### Detailed Analysis or ### Content Details **Chart 1: Base (Baseline) GPQA** * **LLaMA3.1-8B-Instruct (Light Blue)**: Scores approximately 0.6 on CS-QA, 0.7 on GSM8K, 0.5 on HotpotQA, 0.4 on AQUA, 0.3 on HumanEval, and 0.2 on MATH. * **LLaMA3.1-70B-Instruct (Light Yellow)**: Scores approximately 0.7 on CS-QA, 0.75 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **Qwen2.5-7B-Instruct (Light Purple)**: Scores approximately 0.7 on CS-QA, 0.75 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **Qwen2.5-72B-Instruct (Light Red)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **Claude3.5-Sonnet (Darker Blue)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **GPT-3.5 (Orange)**: Scores approximately 0.7 on CS-QA, 0.75 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **GPT-4o (Green)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **QWQ-32B-Instruct (Dashed Light Purple)**: Scores approximately 0.6 on CS-QA, 0.7 on GSM8K, 0.5 on HotpotQA, 0.4 on AQUA, 0.3 on HumanEval, and 0.2 on MATH. * **DeepSeek-V3 (Dashed Dark Blue)**: Scores approximately 0.7 on CS-QA, 0.75 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **DeepSeek-R1 (Dashed Light Blue)**: Scores approximately 0.7 on CS-QA, 0.75 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **o3-mini (Dashed Light Pink)**: Scores approximately 0.5 on CS-QA, 0.6 on GSM8K, 0.4 on HotpotQA, 0.3 on AQUA, 0.2 on HumanEval, and 0.1 on MATH. **Chart 2: S1 (Intrinsic Correction) GPQA** * **LLaMA3.1-8B-Instruct (Light Blue)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **LLaMA3.1-70B-Instruct (Light Yellow)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **Qwen2.5-7B-Instruct (Light Purple)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **Qwen2.5-72B-Instruct (Light Red)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **Claude3.5-Sonnet (Darker Blue)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **GPT-3.5 (Orange)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **GPT-4o (Green)**: Scores approximately 0.8 on CS-QA, 0.85 on GSM8K, 0.7 on HotpotQA, 0.6 on AQUA, 0.5 on HumanEval, and 0.4 on MATH. * **QWQ-32B-Instruct (Dashed Light Purple)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **DeepSeek-V3 (Dashed Dark Blue)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **DeepSeek-R1 (Dashed Light Blue)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **o3-mini (Dashed Light Pink)**: Scores approximately 0.6 on CS-QA, 0.7 on GSM8K, 0.5 on HotpotQA, 0.4 on AQUA, 0.3 on HumanEval, and 0.2 on MATH. **Chart 3: S2 (External Correction) GPQA** * **LLaMA3.1-8B-Instruct (Light Blue)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **LLaMA3.1-70B-Instruct (Light Yellow)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **Qwen2.5-7B-Instruct (Light Purple)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **Qwen2.5-72B-Instruct (Light Red)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **Claude3.5-Sonnet (Darker Blue)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **GPT-3.5 (Orange)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **GPT-4o (Green)**: Scores approximately 0.8 on CS-QA, 0.9 on GSM8K, 0.7 on HotpotQA, 0.6 on AQUA, 0.5 on HumanEval, and 0.4 on MATH. * **QWQ-32B-Instruct (Dashed Light Purple)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **DeepSeek-V3 (Dashed Dark Blue)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **DeepSeek-R1 (Dashed Light Blue)**: Scores approximately 0.7 on CS-QA, 0.8 on GSM8K, 0.6 on HotpotQA, 0.5 on AQUA, 0.4 on HumanEval, and 0.3 on MATH. * **o3-mini (Dashed Light Pink)**: Scores approximately 0.6 on CS-QA, 0.7 on GSM8K, 0.5 on HotpotQA, 0.4 on AQUA, 0.3 on HumanEval, and 0.2 on MATH. ### Key Observations * **Task Performance**: Models generally perform best on GSM8K and CS-QA, and worst on MATH and HumanEval. * **Model Comparison**: GPT-4o (Green) consistently shows higher performance across all tasks and settings compared to other models. o3-mini (Dashed Light Pink) generally performs the worst. * **Correction Impact**: Intrinsic (S1) and External (S2) corrections generally improve performance compared to the baseline (Base), with S2 showing a slight edge over S1. ### Interpretation The radar charts provide a visual comparison of language model performance across different question-answering tasks. The data suggests that: * **Task Difficulty**: Some tasks are inherently more challenging for these models, as evidenced by the consistently lower scores on MATH and HumanEval. * **Model Superiority**: GPT-4o demonstrates superior performance, indicating its advanced capabilities in handling diverse question types. * **Effectiveness of Corrections**: Both intrinsic and external correction methods enhance model performance, suggesting that these techniques are valuable for improving accuracy and reliability. The slight advantage of external correction (S2) may indicate that providing additional context or information during the correction process is beneficial. * **Model Consistency**: The relative performance of models remains consistent across different correction settings. Models that perform well in the baseline setting tend to maintain their relative advantage in the corrected settings.