# Technical Data Extraction: Model Performance Comparison ## 1. Document Overview This image is a scatter plot comparing various Large Language Models (LLMs) based on their parameter size and their performance on translation tasks, measured by "Avg. Reference-Free Eval Scores." The chart specifically highlights the performance of the "ALMA" model series relative to other industry standards and baselines. --- ## 2. Chart Metadata and Axes * **Y-Axis Title:** Avg. Reference-Free Eval Scores * **Y-Axis Range:** 72 to 86 (increments of 2). * **X-Axis Title:** Model Size (B) * **X-Axis Categories:** * **13B**: Models with approximately 13 billion parameters. * **Other Sizes**: Models with varying parameter counts (typically much larger or specialized). * **Legend (Location: Bottom Right [x≈0.85, y≈0.1]):** * `---` (Yellow dashed line): Gold Reference. --- ## 3. Data Series and Trends ### Region A: 13B Parameter Models (Left Column) This group contains models specifically at the 13B scale. There is a significant performance gap between the baseline models and the ALMA models. | Model Name | Marker Type | Color | Approx. Score | | :--- | :--- | :--- | :--- | | **ALMA-13B-R (Ours)** | Star | Dark Purple | **~86.4** | | **ALMA-13B-LoRA** | Star | Dark Purple | **~84.7** | | Bayling-13B | Circle | Light Pink | ~76.7 | | Vicuna-13B-1.1 | Circle | Light Pink | ~74.0 | | LLaMA-1-13B | Circle | Light Pink | ~73.6 | | LLaMA-2-13B | Circle | Light Pink | ~72.9 | | BigTranslate | Circle | Light Pink | ~72.3 | **Trend Analysis:** The ALMA models (stars) represent a massive vertical leap in performance compared to other 13B models (circles). An upward-pointing arrow connects **ALMA-13B-LoRA** to **ALMA-13B-R (Ours)**, labeled with the text **"CPO Fine-tuning"**, indicating that this specific fine-tuning process accounts for the performance increase from ~84.7 to ~86.4. ### Region B: Other Sizes (Right Column) This group contains industry benchmarks and models of various (often larger) sizes. | Model Name | Marker Type | Color | Approx. Score | | :--- | :--- | :--- | :--- | | GPT-4-1106-preview | Circle | Light Purple | ~86.1 | | WMT winners | Circle | Light Purple | ~85.6 | | Google Translate | Circle | Light Purple | ~85.1 | | GPT-3.5-text-davinci-003 | Circle | Light Purple | ~84.3 | | MADLAD-10B | Circle | Light Purple | ~83.9 | | NLLB-3.3B | Circle | Light Purple | ~82.4 | **Trend Analysis:** The models in this category cluster between scores of 82 and 86. Notably, **ALMA-13B-R (Ours)** outperforms all listed models in this category, including GPT-4-1106-preview. --- ## 4. Key Findings and Benchmarks * **Gold Reference:** A horizontal yellow dashed line is drawn at approximately **84.2**. * Models above this line: ALMA-13B-R, ALMA-13B-LoRA, GPT-4-1106-preview, WMT winners, Google Translate, and GPT-3.5-text-davinci-003. * Models below this line: MADLAD-10B, NLLB-3.3B, and all other 13B baseline models. * **Superiority of ALMA:** The "ALMA-13B-R (Ours)" model achieves the highest score on the entire chart (~86.4), surpassing the "Gold Reference" and established high-tier models like GPT-4 and Google Translate. * **Impact of CPO:** The transition from LoRA to "R" (via CPO Fine-tuning) provides a visible performance boost of approximately 1.7 points, moving the model from just above the Gold Reference to the top of the leaderboard.

# Technical Document Extraction: Model Performance Comparison ## Chart Description This scatter plot compares **average reference-free evaluation scores** across different **model sizes (in billions of parameters)**. The x-axis represents model size (B), while the y-axis shows evaluation scores ranging from 72 to 86. A **gold dashed line** at y=84 represents the "Gold Reference" benchmark. --- ### Key Components 1. **Axes Labels** - **X-axis**: "Model Size (B)" (horizontal) - **Y-axis**: "Avg. Reference-Free Eval Scores" (vertical) 2. **Legend** - Located in the **bottom-right corner**. - Colors correspond to specific models/sizes: - **Purple stars**: ALMA-13B-R (Ours), ALMA-13B-LoRA - **Pink circles**: Bayling-13B, Vicuna-13B-1.1, LLaMA-1-13B, LLaMA-2-13B, BigTranslate - **Purple dots**: GPT-4-1106-preview, WMT winners, Google Translate, GPT-3.5-text-davinci-003, MADLAD-10B, NLLB-3.3B 3. **Data Points** - **ALMA-13B-R (Ours)**: - Size: 13B - Score: 86 (highest) - Annotated with "CPO Fine-tuning" - **ALMA-13B-LoRA**: - Size: 13B - Score: 84.5 - **Bayling-13B**: - Size: 13B - Score: 76.5 - **Vicuna-13B-1.1**: - Size: 13B - Score: 74 - **LLaMA-1-13B**: - Size: 13B - Score: 73.5 - **LLaMA-2-13B**: - Size: 13B - Score: 73 - **BigTranslate**: - Size: 13B - Score: 72.5 - **GPT-4-1106-preview**: - Size: 1106B - Score: 84 - **WMT winners**: - Size: Not specified - Score: 84 - **Google Translate**: - Size: Not specified - Score: 83.5 - **GPT-3.5-text-davinci-003**: - Size: Not specified - Score: 83.5 - **MADLAD-10B**: - Size: 10B - Score: 83.5 - **NLLB-3.3B**: - Size: 3.3B - Score: 82 --- ### Spatial Grounding - **Legend Position**: Bottom-right corner (confirmed via visual alignment). - **Color Consistency**: - Purple stars match ALMA models. - Pink circles match 13B-sized models. - Purple dots match larger models (e.g., GPT-4, MADLAD-10B). --- ### Trend Verification 1. **ALMA Models**: - Both ALMA-13B-R and ALMA-13B-LoRA outperform the Gold Reference (84) by significant margins. - ALMA-13B-R (86) is the highest-performing model. 2. **13B-Size Models**: - Scores range from 72.5 (BigTranslate) to 86 (ALMA-13B-R). - ALMA-13B-R and ALMA-13B-LoRA dominate this category. 3. **Larger Models (10B+)**: - GPT-4-1106-preview (1106B) and MADLAD-10B (10B) score 84 and 83.5, respectively. - NLLB-3.3B (3.3B) scores 82, the lowest among non-13B models. 4. **Gold Reference**: - A horizontal dashed line at y=84 serves as a benchmark. Only ALMA-13B-R exceeds it. --- ### Critical Observations - **ALMA-13B-R** achieves the highest score (86) despite being a 13B model, outperforming larger models like GPT-4-1106-preview (1106B). - **Fine-tuning impact**: ALMA-13B-R (CPO Fine-tuning) significantly outperforms ALMA-13B-LoRA (84.5 vs. 86). - **Size vs. Performance**: Larger models (e.g., GPT-4) do not always correlate with higher scores, as ALMA-13B-R (13B) surpasses them. --- ### Missing Information - No explicit numerical values for WMT winners or Google Translate beyond their scores (84 and 83.5, respectively). - No explicit size for WMT winners or Google Translate (marked as "Not specified"). --- ### Final Notes This chart emphasizes the **performance gap** between ALMA models and other state-of-the-art systems, particularly in the 13B parameter range. The Gold Reference (84) acts as a critical benchmark, with ALMA-13B-R being the only model to exceed it.