## Chart: Translation Performance Comparison ### Overview This image presents a series of six line charts comparing the performance of three language models – Gemini 1.5 Pro, Gemini 1.5 Flash, and GPT-4 Turbo – on English-to-various-language translation tasks. The performance metric is Test chrF (a measure of translation quality), plotted against the Number of Shots (K), representing the number of example translations provided to the model. Each chart focuses on a specific target language. ### Components/Axes * **X-axis:** Number of Shots (K), ranging from 2⁰ to 2¹² (approximately 1 to 4096). The axis is labeled "Number of Shots (K)". * **Y-axis:** Test chrF score. The top three charts have a Y-axis labeled "Test chrF (Flores)" with a scale from approximately 25 to 50. The bottom three charts have a Y-axis labeled "Test chrF (In-house)" with a scale from approximately 15 to 35. * **Lines:** Three lines representing the performance of: * Gemini 1.5 Pro (Yellow) * Gemini 1.5 Flash (Blue) * GPT-4 Turbo (Light Blue) * **Legend:** Located at the top-center of the image, clearly labeling each line with its corresponding model name and color. * **Titles:** Each sub-chart has a title indicating the translation direction: "Translation: English → [Language]". The languages are: Bemba, Kurdish, Ewe, Acholi, Abkhaz, and Navajo. ### Detailed Analysis or Content Details **Chart 1: English → Bemba** * Gemini 1.5 Pro: The line is relatively flat, starting at approximately 41.5 and ending at approximately 42.5. * Gemini 1.5 Flash: The line slopes upward, starting at approximately 35 and ending at approximately 45. * GPT-4 Turbo: The line slopes upward, starting at approximately 38 and ending at approximately 43. **Chart 2: English → Kurdish** * Gemini 1.5 Pro: The line is relatively flat, starting at approximately 43 and ending at approximately 44. * Gemini 1.5 Flash: The line slopes upward, starting at approximately 35 and ending at approximately 45. * GPT-4 Turbo: The line slopes upward, starting at approximately 38 and ending at approximately 42. **Chart 3: English → Ewe** * Gemini 1.5 Pro: The line is relatively flat, starting at approximately 32 and ending at approximately 33. * Gemini 1.5 Flash: The line slopes upward, starting at approximately 25 and ending at approximately 38. * GPT-4 Turbo: The line slopes upward, starting at approximately 28 and ending at approximately 35. **Chart 4: English → Acholi** * Gemini 1.5 Pro: The line is relatively flat, starting at approximately 28 and ending at approximately 29. * Gemini 1.5 Flash: The line slopes upward, starting at approximately 20 and ending at approximately 30. * GPT-4 Turbo: The line slopes downward, starting at approximately 25 and ending at approximately 18. **Chart 5: English → Abkhaz** * Gemini 1.5 Pro: The line slopes downward, starting at approximately 32 and ending at approximately 25. * Gemini 1.5 Flash: The line slopes upward, starting at approximately 20 and ending at approximately 30. * GPT-4 Turbo: The line is relatively flat, starting at approximately 25 and ending at approximately 26. **Chart 6: English → Navajo** * Gemini 1.5 Pro: The line is relatively flat, starting at approximately 24 and ending at approximately 25. * Gemini 1.5 Flash: The line slopes downward, starting at approximately 28 and ending at approximately 20. * GPT-4 Turbo: The line slopes upward, starting at approximately 18 and ending at approximately 28. ### Key Observations * Gemini 1.5 Pro generally exhibits a stable performance across all languages, with minimal improvement as the number of shots increases. * Gemini 1.5 Flash consistently shows improvement in chrF score as the number of shots increases, across all languages. * GPT-4 Turbo's performance varies significantly depending on the target language. It shows improvement for some languages (Kurdish, Ewe, Navajo) but declines for others (Acholi). * The "In-house" chrF scores (Charts 4-6) are generally lower than the "Flores" chrF scores (Charts 1-3). * The largest performance gains from increasing the number of shots are observed for Gemini 1.5 Flash. ### Interpretation The data suggests that Gemini 1.5 Pro benefits less from increasing the number of example translations (shots) compared to Gemini 1.5 Flash and GPT-4 Turbo. This could indicate that Gemini 1.5 Pro has a stronger inherent understanding of translation principles or a more robust internal representation of language. Gemini 1.5 Flash consistently improves with more shots, suggesting it is more data-driven and benefits from learning from examples. GPT-4 Turbo's performance is more language-specific, potentially indicating variations in the quality of training data or the complexity of the translation task for each language. The difference between "Flores" and "In-house" chrF scores suggests that the evaluation datasets have different characteristics, potentially reflecting different translation styles or domains. The charts provide a comparative analysis of translation quality across different models and languages, highlighting the strengths and weaknesses of each approach. The varying trends suggest that the optimal model choice may depend on the specific language pair and the availability of example translations.