2604.24645

# Unknown Title ## K-MetBench: A Multi-Dimensional Benchmark for Fine-Grained Evaluation of Expert Reasoning, Locality, and Multimodality in Meteorology ## Soyeon Kim KAIST, INEEJI Seongnam, Korea soyeon.k@kaist.ac.kr ## Eun-Chul Chang and Jaedeok Lee Kongju National University Gongju, Korea Cheongwoong Kang and Myeongjin Lee KAIST Seongnam, Korea cw.kang; lmjk311@kaist.ac.kr Jaesik Choi ∗ KAIST, INEEJI Seongnam, Korea echang@kongju.ac.kr, ruio1084@gmail.com jaesik.choi@kaist.ac.kr ## Abstract The development of practical ( multimodal ) large language model assistants for Korean weather forecasters is hindered by the absence of a multidimensional, expert-level evaluation framework grounded in authoritative sources. To address this, we introduce KMetBench, a diagnostic benchmark grounded in national qualification exams. It exposes critical gaps across four dimensions: expert visual reasoning of charts, logical validity via expert-verified rationales, Korean-specific geocultural comprehension, and fine-grained domain analysis. Our evaluation of 55 models reveals a profound modality gap in interpreting specialized diagrams and a reasoning gap where models hallucinate logic despite correct predictions. Crucially, Korean models outperform significantly larger global models in local contexts, demonstrating that parameter scaling alone cannot resolve cultural dependencies. K-MetBench serves as a roadmap for developing reliable, culturally aware expert AI agents. The dataset is available at https://huggingface.co/ datasets/soyeonbot/K-MetBench . ## 1 Introduction Large language models ( LLMs ) and multimodal large language models ( MLLMs ) have shown growing promise in scientific domains ( Taylor et al. , 2022 ; Team et al. , 2023 ; OpenAI , 2025 ) , achieving performance matching passing thresholds on professional certification exams ( Singhal et al. , 2023 ; Katz et al. , 2024 ) . As these models are increasingly positioned as assistants for domainspecific tasks, there is a growing need for evaluation frameworks that go beyond surface-level correctness and more precisely characterize domain- ∗ Corresponding author. Figure 1: An example of the K-MetBench dataset ( translated into English ). K-MetBench provides evaluation across four critical dimensions: ( 1 ) multimodal understanding, ( 2 ) expert-level reasoning, ( 3 ) geo-cultural context sensitivity, and ( 4 ) fine-grained domain knowledge across five meteorological sub-fields. <details> <summary>Image 1 Details</summary>  ### Visual Description ## Diagram: Process Flow for Meteorological Question Analysis ### Overview The diagram illustrates a structured workflow for analyzing a meteorological question involving jet stream dynamics. It integrates visual reasoning, granular analysis, local specificity, and expert evaluation to determine the correct answer to a question about jet streak convergence in an upper-level trough. ### Components/Axes 1. **Top Section (Flowchart Layout)**: - **Section 1**: "Visual Reasoning beyond OCR to Chart interpretation" (Blue box with black text). - **Section 2**: "Reasoning Evaluation" (Red box with black text). - **Section 3**: "Local Specificity" (Yellow box with black text). - **Section 4**: "Granular Analysis" (Green box with black text). - Arrows connect sections 1 → 4 → 3 → 2, forming a cyclical process. 2. **Central Question Box**: - **ID**: 206 | Part: 5 | Korean-Specific: False (Green text). - **Question**: "When a jet streak is located within an upper-level trough as shown in the figure, in which region does strong convergence occur?" (Black text). - **Choices**: 1. Region d, 2. Region b, 3. Region c, 4. Region a (Black text). - **Answer**: 2 (Black text). 3. **Regional Diagram**: - Arrows labeled with KTS values (100 KTS, 125 KTS, 150 KTS) form a cyclical pattern. - Regions labeled **a, b, c, d** (Blue text). - Arrows point to regions with annotations like "100 KTS" and "150 KTS". 4. **Expert-Verified Rationale**: - Red text box with explanation: "Due to the curvature effect within a jet streak located in an upper-level trough, ... Therefore, the region of strong convergence is Region b." ### Detailed Analysis - **Section 1 (Visual Reasoning)**: Focuses on interpreting charts beyond OCR, emphasizing spatial analysis of meteorological data. - **Section 4 (Granular Analysis)**: Breaks down weaknesses into 5 sub-fields, though specifics are not detailed in the image. - **Section 3 (Local Specificity)**: Highlights Korea-specific topography and regulations, though marked as "False" for this question. - **Section 2 (Reasoning Evaluation)**: Uses expert-aligned rationale scoring to validate answers. - **Regional Diagram**: - Arrows indicate wind flow dynamics, with higher KTS values (150 KTS) near Region b. - The curvature effect in jet streams causes convergence in Region b, as explained in the rationale. ### Key Observations - The correct answer (Region b) is explicitly stated, with the rationale linking it to the curvature effect in jet streams. - KTS values (100, 125, 150) suggest wind speed gradients, with stronger convergence where flow accelerates (Region b). - The flowchart emphasizes iterative analysis: visual reasoning → granular analysis → local specificity → expert evaluation. ### Interpretation The diagram demonstrates a systematic approach to solving meteorological problems by combining domain-specific knowledge (jet stream dynamics) with structured reasoning. The curvature effect in upper-level troughs causes wind acceleration and convergence in Region b, validated by expert evaluation. The exclusion of Korea-specific factors ("Korean-Specific: False") indicates the answer applies broadly, not regionally. This workflow ensures accuracy by cross-referencing visual data, physical principles, and expert judgment. </details> relevant competencies ( Liang et al. , 2022 ) . However, existing benchmarks for vertical domains often summarize performance using a single aggregate score, making it difficult to understand why a model succeeds or fails in practice. In complex applied fields such as meteorology, this coarse evaluation obscures critical limitations for real-world deployment. We identify four recurring limitations in current evaluations of meteorological reasoning. First, the modality gap . Meteorological anal- ysis, inherently multimodal, requires the synthesis of numerical data, textual descriptions, and specialized visual charts ( e.g., weather maps, skew-T logP diagrams ). However, most scientific benchmarks remain predominantly text-based and provide limited assessment of a model ' s ability to interpret domain-specific charts and spatial patterns. As a result, visual understanding capabilities central to operational forecasting remain under-evaluated. Second, the reasoning gap . Conventional benchmarks primarily rely on answer accuracy, without explicitly evaluating the validity or structure of the underlying reasoning. In high-stakes domains like weather forecasting, a correct prediction reached through shallow heuristics or incomplete logic may still lead to brittle or unreliable behavior ( Turpin et al. , 2023 ) . Without access to expertaligned rationales, it is difficult to distinguish genuine understanding from shortcut learning. Third, the geo-cultural gap . Many existing datasets emphasize global or universal physical principles while abstracting away local geographic and institutional context. In meteorology, however, local topography, climatological conventions, and region-specific regulations play a substantial role in interpretation and decision-making. Models trained and evaluated solely on decontextualized data may therefore fail to generalize reliably to region-specific applications. Fourth, the granularity gap . Aggregate performance scores often mask uneven competence across sub-domains. A model may perform well on factual recall or chart interpretation while struggling with quantitative reasoning or applied dynamics. Without fine-grained analysis, such disparities remain difficult to diagnose. To address these limitations, we introduce K-MetBench , a Korean meteorological benchmark designed for multi-dimensional evaluation of LLMs and MLLMs. Rather than treating meteorological expertise as a monolithic capability, KMetBench decomposes evaluation along four complementary axes: ( 1 ) multimodal understanding of meteorological charts and symbols, ( 2 ) reasoning quality assessed using expert-verified rationales, ( 3 ) sensitivity to geo-cultural and regional context , and ( 4 ) fine-grained coverage across five officially defined meteorological sub-domains. Through this structured design, K-MetBench is intended as a diagnostic tool that helps reveal which aspects of meteorological reasoning remain challenging for current models, and why. ## 2 Related Work Existing benchmarks for meteorological and climate reasoning reflect diverse assumptions about knowledge sources, modalities, and evaluation objectives. Rather than treating them as competitors, we situate them along complementary axes that highlight different aspects of domain expertise. ClimaQA ( Manivannan et al. , 2024 ) evaluates climate question answering using textbooks as the knowledge source. By grounding questions in established instructional materials, it emphasizes conceptual understanding and theoretical reasoning characteristic of graduate-level climate science. While this approach provides scientific rigor, it remains purely text-based and does not assess visual interpretation or operational reasoning grounded in real-world artifacts. ClimateIQA ( Chen et al. , 2025 ) constructs instruction-style QA data from numerical weather prediction ( NWP ) heatmaps and associated geospatial metadata. This enables evaluation of visual pattern recognition and structured data interpretation. WeatherQA ( Ma et al. , 2024 ) further targets operational forecasting scenarios by combining multiple meteorological images with expert-written mesoscale discussions. These datasets advance multimodal evaluation, but their emphasis remains on task-level performance rather than fine-grained diagnosis across sub-fields or distinct reasoning failures. In the Korean-language evaluation landscape, KMMLU ( Son et al. , 2025 ) is derived from official national examinations, measuring expert-level linguistic competence across a wide range of professions. Since it is based on official Korean exams, KMMLU captures linguistic and cultural aspects of the Korean language. KMMLU-Redux ( Hong et al. , 2025 ) is a reconstructed version of KMMLU that removes erroneous, ambiguous, or contaminated items to improve reliability. While these benchmarks offer high reliability and clear passing criteria, they are primarily text-based and treat meteorological knowledge as a small subset within a broader evaluation suite, limiting their ability to analyze domain-specific competencies in depth. ## 3 K-MetBench Construction K-MetBench is designed to complement the existing benchmarks by explicitly separating and jointly examining four dimensions that are often conflated in prior benchmarks. Rather than introducing new task formats, K-MetBench focuses on providing di- Table 1: Comparison with existing benchmarks. K-MetBench distinguishes itself by covering four key axes: visual understanding, rationale reliability, geo-cultural alignment, and fine-grained diagnosis in sub-domains. | Dataset | Lang. | Lang. | Domain | Test Size ( Source ) | Test Size ( Source ) | Modality | Reasoning Geo-Cultural | Granularity | |--------------------------------------|---------|------------------|--------------------|------------------------|------------------------|-----------------|--------------------------|-------------------| | KMMLU ( Son et al. , 2025 ) | Kor | General | 35k ( License Exam | ) Text | | × | Korea | × ( 45 Subjects ) | | KMMLU-Redux ( Hong et al. , 2025 ) | Kor | General | 2.6k ( License | Exam ) Text | | × | Korea | × ( 14 Subjects ) | | ClimaQA ( Manivannan et al. , 2024 ) | Eng | Climate | 566 ( | Autogenerate ) Text | | × | Global | × ( 3 Tasks ) | | ClimateIQA ( Chen et al. , 2025 ) | Eng | Climate | 152k ( | Template ) | Image+Text | × | Global | × ( 4 Tasks ) | | WeatherQA ( Ma et al. , 2024 ) | Eng | Weather Forecast | | 600 ( Template ) | Image+Text | × | United States | × ( 2 Tasks ) | | K-MetBench ( Ours ) | Kor | Meteorology | 1.7k ( License | Exam | ) Image+Text | Expert-Verified | Korea | 5 Sub-domains | Note for K-MetBench. Modality: Includes multimodal questions evaluating interpretation of professional weather charts. Reasoning: Provides rationale verified by domain experts. Geo-Cultural: Includes questions requiring knowledge of local geography and regulations that are specific to Korea ( e.g., the Korea Meteorological Administration ( KMA ) protocols ). Granularity: Supports fine-grained diagnosis across the five sub-domains officially defined in Korea Engineer Meteorology certification exam. Table 2: Detailed statistics of K-MetBench. The dataset is structured into four key dimensions to enable structured evaluation: Modality, Reasoning, Geocultural, and Granularity. | Diagnostic Axis | Statistic | Value | |---------------------------------------|---------------------------------------------|-------------------------------| | 1. Overview | Total Questions | 1,774 | | 2. Modality | Image+Text Questions | 82 ( 4.62% ) | | ( Visual Understanding ) 3. Reasoning | ( Charts, Diagrams ) Avg. Rationale Length | 93.72 tokens | | 4. Geo-Cultural ( Local Knowledge ) | Korean-Specific Questions | 73 ( 4.11% ) | | 5. Granularity | | | | ( 5 Subject Areas | Part 1: Forecast Theory Part 2: Observation | 373 ( 21.03% ) 332 ( 18.71% ) | | ) | Part 3: Atmos. Dynamics Part 4: Climatology | 359 ( 20.24% ) | | | | 376 ( 21.20% ) | Note: The number of tokens is calculated using the gemini-2.5-flash tokenizer. ( Atmos.: Atmospheric ) agnostic visibility into how and where current models succeed or fail when approaching expert-level meteorological reasoning. ## 3.1 Data Collection and Processing K-MetBench is constructed from raw data drawn from the National Meteorological Engineer certification examinations, covering 25 exam sessions between March 16, 2003 and March 5, 2022. The initial pool comprised 2,500 multiple-choice questions. Because these examinations are generated from a shared question bank, substantial overlap exists across years. To construct a balanced and non-redundant benchmark, we applied a multistage filtering and augmentation pipeline. For deduplication ( Lee et al. , 2021 ) , we first applied difflib.SequenceMatcher with a similar- ity threshold of 0.6, removing exact duplicates as well as items with trivially permuted answer options. Importantly, questions with inverted logic ( e.g., ' highest ' vs. ' lowest ', ' saturated ' vs. ' unsaturated ') were manually reviewed and retained, as they probe distinct reasoning behaviors despite surface similarity. This process yielded a refined set of 1,774 questions. To reduce memorization and contamination effects, we applied two transformations. First, we randomized answer option orders for all questions. Second, we paraphrased question stems using Gemini-2.5-Pro , with strict constraints to preserve technical terminology and domain-specific meaning. The system prompt used for paraphrasing is provided in Appendix C.1. To maintain quality, a human researcher reviewed and refined 14.88% ( 264/1,774 ) of the paraphrased items. For multimodal questions, both text and visual elements were extracted from the original examination PDFs. Three researchers reviewed and crosschecked all extracted images to correct parsing artifacts such as missing axis labels, distorted symbols, or incomplete annotations. As a design choice to separate perceptual challenges from reasoning difficulty, mathematical formulas embedded as images were transcribed into LaTeX code to prevent OCRbottlenecks, while meteorological charts and diagrams were preserved in their original format. ## 3.2 Subset 1: Multimodal Diagnosis The multimodal subset of K-MetBench consists of 82 questions ( 4.62% of the dataset ) that require interpretation of meteorological visuals. Unlike general-purpose multimodal benchmarks that focus on object recognition or scene description, this subset targets domain-specific charts and symbolic representations. The included materials span sur- Table 3: Distribution of K-MetBench across five sub-domains. The number of questions for each subject area is reported, with the number of reasoning questions featuring expert-verified rationales in parentheses ( Reas. stands for Reasoning ). | Part | Subject Area | Overall Volume Total ( Reas. ) | Modality | Modality | Geo-Cultural | |--------|------------------------------------|----------------------------------|----------------|------------------------|------------------| | Part | Subject Area | Overall Volume Total ( Reas. ) | Text ( Reas. ) | Image + Text ( Reas. ) | Korean ( Reas. ) | | 1 | Weather Analysis &Forecast Theory | 373 ( 28 ) | 364 ( 24 ) | 9 ( 4 ) | 6 ( 0 ) | | 2 | Meteorological Observation Methods | 332 ( 28 ) | 318 ( 24 ) | 14 ( 4 ) | 0 ( 0 ) | | 3 | Atmospheric Dynamics | 359 ( 29 ) | 340 ( 25 ) | 19 ( 4 ) | 0 ( 0 ) | | 4 | Climatology | 376 ( 28 ) | 363 ( 24 ) | 13 ( 4 ) | 50 ( 7 ) | | 5 | Atmospheric Physics | 334 ( 28 ) | 307 ( 24 ) | 27 ( 4 ) | 17 ( 0 ) | | Sum | Total Coverage | 1,774 ( 141 ) | 1,692 ( 121 ) | 82 ( 20 ) | 73 ( 7 ) | face weather maps, upper-level charts ( e.g., 200 and 500 hPa ), and thermodynamic diagrams such as Skew-T Log-P plots and emagrams derived from radiosonde measurements. Solving these questions requires extracting structured information, including pressure gradients, wind vectors, and thermodynamic indices-from dense visual fields that cannot be resolved through OCR alone. Consequently, this subset assesses the ability of MLLMsto integrate textual meteorological knowledge with the interpretation of domain-specific visual cues. Representative examples are provided in Appendix Table 6. ## 3.3 Subset 2: Reasoning-Aware Evaluation To evaluate reasoning quality beyond final answer correctness, K-MetBench includes a reasoningaware subset consisting of 141 questions paired with expert-verified rationales. These rationales serve as reference explanations for assessing the validity, coherence, and depth of model-generated reasoning. Rationale construction followed a twostage process. First, GPT-5 was used to generate initial reasoning drafts, guided by prompts that emphasized logical flow, factual consistency, clarity, and completeness. Second, two meteorology professors reviewed these drafts, correcting factual errors, refining physical explanations, and resolving ambiguities. We employ an LLM-as-a-Judge framework ( Zheng et al. , 2023 ) to score modelgenerated rationales against the expert-verified rationales as reference standard. The system prompts used for reasoning generation and evaluation are detailed in Appendix C.5 and C.6. To validate the reliability of this framework in a specialized domain, we conduct a meta-evaluation ( Li et al. , 2024 ) comparing LLM judgments with human expert scores. The experimental and survey protocols are provided in Appendix D.2 and C.9. ## 3.4 Subset 3: Geo-Cultural Sensitivity Meteorological reasoning is strongly influenced by local geography, climate patterns, and institutional conventions. To capture this dependency, we annotate a Korean-Specific subset comprising 73 questions that involve implicit, speakercentric, or high-context expressions specific to the Korean Peninsula. Candidate items were identified using prompt-enhanced LLMs ( GPT-4.1 and Gemini-2.5-Pro ) designed to detect references to localized phenomena, such as regional topography ( e.g., the Yeongdong region ) or regulations issued by the Korea Meteorological Administration ( KMA ). These candidates were subsequently reviewed and validated by two researchers to ensure relevance and correctness. Rather than testing translation ability, this subset probes whether models can appropriately ground meteorological knowledge in region-specific context. As such, it provides a controlled setting for analyzing geocultural alignment in domain-specific reasoning. ## 3.5 Subset 4: Domain Specificity To enable fine-grained analysis of meteorological expertise, K-MetBench is organized into five official subject areas defined in the Korean Meteorological Engineer certification exam. These include: Part 1 ( Weather Analysis and Forecast Theory ), Part 2 ( Meteorological Observation Methods ), Part 3 ( Atmospheric Dynamics ), Part 4 ( Climatology ), and Part 5 ( Atmospheric Physics ). Each subject area targets a distinct aspect of professional competence, ranging from chart interpretation and numerical weather prediction principles to instrumentation, large-scale atmospheric motion, climate systems, and thermodynamic cal- culations. This structure allows model performance to be examined at a level of granularity that is not visible from aggregate scores alone. By aligning evaluation with established subject boundaries, this design facilitates diagnosis of domainspecific strengths and weaknesses, for example, distinguishing models that perform well on descriptive climatology but struggle with quantitative dynamics or thermodynamics. ## 4 Experiments ## 4.1 Experimental Setup Evaluated Models. To ensure a comprehensive benchmark, we evaluated a diverse array of models categorized by scale, training language, and modality support. The selection includes proprietary state-of-the-art models renowned for superior reasoning capabilities, such as GPT-5.2 ( evaluated with and without reasoning modules enabled ) ( OpenAI , 2025 ) and Gemini-3-Pro-Preview ( Team et al. , 2023 ) . We also incorporated open-source models ranging from 0.6B to 235B parameters, exemplified by InternVL3.5 ( Wang et al. , 2025 ) and Qwen3-VL ( Yang et al. , 2025 ) , alongside large-scale foundation models such as gptoss-120b ( Agarwal et al. , 2025 ) , commanda-reasoning-08-2025 ( Cohere et al. , 2025 ) , and Llama-3.2-90B-Vision-Instruct ( Meta , 2024 ) . To investigate the impact of geo-cultural knowledge, we specifically included Koreancentric models, including EXAONE-4.0 ( Research et al. , 2025 ) , A.X-4.0 ( Lab , 2025 ) , VARCOVision-2.0 ( Cha et al. , 2025 ) , and HyperCLOVA X ( Yoo et al. , 2024 ) . Finally, strictly text-based baselines were established by evaluating nonmultimodal models solely on the textual components of questions to quantify text dependency. Geo-Cultural Disambiguation Protocol. To establish a fair evaluation protocol for global models, we designed four experimental configurations that cross-reference question formulation with prompting conditions. This setup ensures that models are assessed on their meteorological competence rather than their ability to decode localized linguistic ambiguities. For question formulation, we compared an Implicit condition, using original speakercentric terms like ' Our country, ' against an Explicit condition, which replaces these with proper nouns ( e.g., ' South Korea ') to isolate and evaluate pure domain knowledge. Regarding prompt- ing conditions, beyond a Standard prompt that injects an expert persona, we introduced an Advanced prompt providing explicit disambiguation ( e.g., ' ' Our country ' refers to South Korea' ). This advanced protocol serves as a specialized support layer, mitigating performance degradation caused by implicit geo-cultural references and enabling global models to compete on an equal footing. Comparison with Existing Benchmarks. We evaluated models using the official test sets of all datasets, employing the Chain-of-Thought ( CoT ) ( Wei et al. , 2022 ) protocol for WeatherQA. Task orthogonality was analyzed using Kendall ' s Tau-b rank correlation coefficient. To align the distance-based Haversine metric of ClimaIQA ( where lower is better ) with standard accuracy metrics, we inverted the sign of ClimaIQA scores prior to calculating correlations. Meta-Evaluation Setup: Validating LLM-as-aJudge. Given the specialized nature of meteorology, validating the reliability of commercial LLMs as judges is crucial. We conducted a metaevaluation comparing human expert judgments with LLM judgments. We selected ten representative questions varying in difficulty and type, and collected reasoning outputs from ten open-source LLMs. Two human experts provided gold standard scores, while Gemini-2.5-Pro served as the AI evaluator. Both parties utilized identical expertverified references and a scoring rubric across four axes: Factuality, Logicality, Depth, and Clarity. We calculated Kendall ' s Tau-b ( τ b ) correlation between human and AI scores, confirming the alignment of the automated judge ( τ b > 0 . 8 ). We also computed Krippendorff ' s α ( interval ) and Intraclass Correlation Coefficient ( ICC, 2-way mixed, absolute ) to assess inter-rater reliability, which indicated acceptable agreement ( α > 0 . 7 ). To investigate whether incorporating human expert rationales improves the alignment between the LLM evaluator and human judgment, we compared the correlations of their scores under conditions with and without rationale availability. Implementation Details. To ensure a fair comparison, we utilized Standard prompts across all models. We applied a zero-shot setting to all text, multimodal, and reasoning questions to evaluate intrinsic capabilities. We computed accuracy by extracting final answers via regular expressions. To rigorously assess instruction-following Table 4: K-MetBenchperformance scores across diverse models. Models are sorted by accuracy. Accuracy score ranges from 0 to 100, while the reasoning score ( Reas. ) ranges from 4 to 20. The highest scores in each column are shown in bold for proprietary and open-source models, respectively. ( Acc.: Accuracy, K : Korean model, V : Vision language model, R : Reasoning model. ) | Type | Model | Type | Type | Acc. | Reas. | Geo-Cult. Korean | Modality | Modality | Granularity ( P1-P5 ) | Granularity ( P1-P5 ) | Granularity ( P1-P5 ) | Granularity ( P1-P5 ) | Granularity ( P1-P5 ) | |-------------|-----------------------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|-------------------------|-------------------------|-------------------------|-------------------------|-------------------------| | Type | Model | K V | R | Acc. | Reas. | Geo-Cult. Korean | Text | Multi | P1 | P2 | P3 | P4 | P5 | | | gemini-3-pro-preview ( Thinking ) | V | R | 93.7 | 18.01 | 90.4 | 94.6 | 75.6 | 92.5 | 97.9 | 94.2 | 92.8 | 91.6 | | Proprietary | gpt-5.2 ( Thinking ) | V | R | 87.8 | 17.33 | 80.8 | 90.6 | 29.3 | 86.3 | 93.4 | 88.0 | 86.2 | 85.3 | | | gpt-5.2 | V | | 77.6 | 17.39 | 75.3 | 79.0 | 50.0 | 77.2 | 81.3 | 71.9 | 81.4 | 76.3 | | | Multilingual Models | Multilingual Models | Multilingual Models | Multilingual Models | Multilingual Models | Multilingual Models | Multilingual Models | Multilingual Models | Multilingual Models | Multilingual Models | Multilingual Models | Multilingual Models | Multilingual Models | | | Qwen3-VL-235B-A22B-Thinking | V | R | 84.4 | 17.22 | 72.6 | 86.2 | 48.8 | 81.5 | 88.6 | 87.2 | 83.2 | 82.0 | | | Qwen3-VL-32B-Thinking | V | R | 78.6 | 16.19 | 60.3 | 79.9 | 51.2 | 74.3 | 85.2 | 78.8 | 78.7 | 76.3 | | | command-a-reasoning-08-2025 | | R | 77.8 | 14.12 | 74.6 | 77.8 | - | 73.4 | 85.2 | 73.8 | 78.8 | 78.5 | | | gpt-oss-120b | | R | 77.3 | 16.12 | 62.0 | 77.3 | - | 72.5 | 85.8 | 76.5 | 77.4 | 74.9 | | | Qwen3-30B-A3B-Thinking-2507 | | R | 76.7 | 15.76 | 67.6 | 76.7 | - | 75.5 | 82.1 | 75.6 | 74.9 | 75.9 | | | InternVL3.5-38B-Instruct | V | | 57.3 | 11.38 | 47.9 | 58.1 | 40.2 | 56.0 | 64.8 | 48.7 | 61.4 | 55.7 | | | Llama-3.2-90B-Vision-Instruct | V | | 56.9 | 9.72 | 52.1 | 58.2 | 30.5 | 57.1 | 59.3 | 52.4 | 62.2 | 53.3 | | Open-source | Phi-4 | | | 51.5 | 11.75 | 40.8 | 51.5 | - | 52.5 | 53.8 | 50.0 | 55.1 | 45.3 | | | Korean Models | Korean Models | Korean Models | Korean Models | Korean Models | Korean Models | Korean Models | Korean Models | Korean Models | Korean Models | Korean Models | Korean Models | Korean Models | | | A.X-4.0 | K | | 76.1 | 15.46 | 78.9 | 76.1 | - | 76.6 | 77.7 | 68.2 | 81.3 | 76.5 | | | EXAONE-4.0-32B | K | R | 59.9 | 13.57 | 59.2 | 59.9 | - | 58.2 | 64.8 | 52.4 | 63.1 | 61.2 | | | VARCO-Vision-2.0-14B | K | V | 58.7 | 11.24 | 57.5 | 59.5 | 42.7 | 59.0 | 62.3 | 54.3 | 61.7 | 56.0 | | | A.X-4.0-Light | K | | 55.7 | 11.45 | 60.6 | 55.7 | - | 55.8 | 54.4 | 50.9 | 61.4 | 55.7 | | | A.X-4.0-VL-Light | K | V | 52.5 | 9.76 | 54.8 | 53.0 | 42.7 | 51.5 | 50.6 | 50.1 | 58.0 | 52.1 | | | HyperCLOVAX-SEED-Think-14B | K | R | 50.8 | 11.29 | 52.1 | 50.8 | - | 51.6 | 53.8 | 41.8 | 55.6 | 51.1 | capabilities, we counted any output that violated the required format as a failure case. We employed the vLLM library ( Kwon et al. , 2023 ) with its default configurations, except for A.X-4.0-VLLight and Llama-3.2-90B-Vision-Instruct , which were run using Hugging Face Transformers. The random seed was fixed at 42, and sampling temperatures were set to 0.1 by default, while a temperature of 1.0 was employed for reasoning models. All prompts and questions were provided in the original Korean to strictly evaluate localized comprehension without translation artifacts. ## 5 Results Beyond simple leaderboards, we dissect the performance of models across four dimensions to reveal their true capabilities and limitations. The Modality Gap: Text-Only vs. Multimodal. Figure 2 reveals a distinct dented shape along the Multimodal axis, confirming that visual reasoning is the primary bottleneck for current MLLMs. Specifically, models exhibited a sharp accuracy decline ( avg. -18.55% ) on multimodal questions compared to text-only ones. This deficit is most pronounced in professional tasks involving Skew-T Log-P diagrams and surface weather maps, where models failed to extract key data despite their general vision capabilities. The Reasoning Gap: Knowledge vs. Reasoning. Table 4 and Figure 2 highlight a distinct gap between answer accuracy and reasoning quality. Although Kendall ' s τ b ( 0.78 ) indicates a general correlation ( Appendix Figure 7 ), qualitative analysis reveals that models frequently provide correct answers accompanied by insufficient rationales, including the use of improper or hallucinated terminology ( Appendix Table 7 ). Additionally, while models achieve high accuracy on simple retrieval tasks, performance significantly degrades on calculation and multi-step reasoning tasks, even when CoT prompting-explicitly guiding the model to use a <scratchpad> ( Nye et al. , 2021 ) -is employed. The Geo-Cultural Gap. Table 4 reveals that large multilingual models struggle with the Korean-Specific subset ( e.g., Changma, topography ) despite their scale. The Korean-centric A.X-4.0 ( 72B ) scored 78.9, outperforming the larger Qwen3-VL-235B-Thinking ( 72.6 ). This confirms that parameter scaling does not automatically grant proficiency in local domains. Granular Domain Analysis. Finally, decomposing performance across the five official subject areas reveals fine-grained disparities masked by aggregated scores. As shown in Table 4, models gen- Figure 2: Holistic performance analysis of top-6 models across five dimensions. The radar chart visualizes model capabilities in Accuracy, Reasoning, GeoCultural alignment ( K-Specific ) , Modality ( Text-only vs. Multimodal ) , and Granularity ( Subject Parts 1-5 ) . While models show balanced performance across theoretical subjects, a sharp decline is observed in the Multimodal axis, highlighting the modality gap. <details> <summary>Image 2 Details</summary>  ### Visual Description ## Radar Chart: Model Performance ComparisonAcross Criteria ### Overview The image is a radar chart comparing the performance of four AI models across multiple criteria: Granularity, Part 1-5, Reasoning, Accuracy, K-Specific, Geo-Cultural, and Modality. The chart uses colored lines to represent different models, with a radial scale from 0.0 to 1.0. The legend identifies models by color, and the chart emphasizes trade-offs between criteria. ### Components/Axes - **Axes (Radial Labels)**: - Granularity (top-left) - Part 1 (top) - Part 2 (top-right) - Part 3 (bottom-right) - Part 4 (bottom) - Part 5 (bottom-left) - Reasoning (right) - Accuracy (bottom-right) - K-Specific (bottom-left) - Geo-Cultural (left) - Modality (bottom-center) - **Legend (Bottom-center)**: - **Blue**: Gemini-3-pro-preview (Thinking) - **Red**: Qwen3-VL-32B-Thinking - **Orange**: gpt-5.2 (Thinking) - **Green**: Qwen3-VL-235B-A22B-Thinking - **Purple**: command-a-reasoning-08-2025 - **Brown**: gpt-5.2 - **Scale**: Radial axis from 0.0 (center) to 1.0 (outer edge), with increments at 0.2, 0.4, 0.6, 0.8. ### Detailed Analysis 1. **Gemini-3-pro-preview (Blue)**: - Peaks in **Reasoning** (~0.9) and **Modality** (~0.8). - Dips in **Part 4** (~0.4) and **K-Specific** (~0.5). - Strong in **Accuracy** (~0.7) and **Geo-Cultural** (~0.6). 2. **Qwen3-VL-32B-Thinking (Red)**: - Consistent mid-range performance (~0.6–0.7) across most criteria. - Slight dip in **Part 3** (~0.5) and **Geo-Cultural** (~0.55). 3. **gpt-5.2 (Orange)**: - Weakest overall, with low values in **Part 4** (~0.3) and **K-Specific** (~0.4). - Slight improvement in **Reasoning** (~0.6) and **Modality** (~0.5). 4. **Qwen3-VL-235B-A22B-Thinking (Green)**: - Strong in **Accuracy** (~0.8) and **Geo-Cultural** (~0.7). - Moderate in **Reasoning** (~0.7) and **Modality** (~0.65). 5. **command-a-reasoning-08-2025 (Purple)**: - Exceptional in **Reasoning** (~0.95) but weak in **Part 4** (~0.4) and **K-Specific** (~0.5). 6. **gpt-5.2 (Brown)**: - Overlaps with orange line; identical performance. ### Key Observations - **Reasoning Dominance**: Models like Gemini-3-pro-preview and command-a-reasoning-08-2025 excel in Reasoning but lag in other areas. - **Accuracy vs. Granularity**: High-accuracy models (e.g., Qwen3-VL-235B-A22B) often sacrifice granularity. - **Modality Trade-offs**: Gemini-3-pro-preview and Qwen3-VL-235B-A22B perform well in Modality but show variability in Part-specific criteria. - **Anomalies**: Gemini-3-pro-preview’s sharp dip in Part 4 suggests a potential weakness in handling specific tasks. ### Interpretation The chart reveals that no single model dominates all criteria, highlighting the importance of use-case specificity. For example: - **Reasoning-heavy tasks** favor Gemini-3-pro-preview or command-a-reasoning-08-2025. - **Accuracy-critical applications** may prioritize Qwen3-VL-235B-A22B. - **Modality and Geo-Cultural needs** align with Gemini-3-pro-preview or Qwen3-VL-235B-A22B. The data underscores the need for context-aware model selection, as strengths in one area (e.g., Reasoning) often come at the cost of others (e.g., Part 4 performance). The overlap between gpt-5.2 and the brown line suggests potential redundancy or mislabeling in the legend. </details> erally exhibit robust performance in Part 2 ( Meteorological Observation ), which focuses on instrumentation and factual knowledge ( e.g., Gemini-3Pro reaching 97.9 ). However, significant performance drops are observed in calculation-intensive and abstract domains like Part 3 ( Atmospheric Dynamics ) and Part 5 ( Atmospheric Physics ). Astriking example is the Korean model A.X-4.0 , which achieves its highest accuracy in Part 4 ( Climatology ) ( 81.3 ) -likely benefiting from training on local meteorological laws-but struggles disproportionately in Part 3 ( 68.2 ), where understanding synoptic motions is required. This granular diagnosis identifies specific domain weaknesses: while models may possess sufficient regulatory knowledge ( Part 4 ), they require targeted fine-tuning to enhance quantitative reasoning in thermodynamics and dynamics ( Part 3, 5 ). Orthogonality between Existing Baselines. As shown in Figure 3, we analyzed Kendall ' s τ b correlations to assess the independence of KMetBench. While the Text-Only subset correlates strongly with general Korean benchmarks ( KMMLU-Redux, τ b = 0 . 78 ), we observe a distinct decoupling in complex capabilities. Notably, the correlation weakens for the Reasoning subset ( τ b = 0 . 66 ) and drops sharply for the Multimodal Figure 3: Correlation analysis with existing benchmarks. The heatmap visualizes Kendall ' s τ b correlation coefficients between K-MetBench metrics and existing benchmarks. <details> <summary>Image 3 Details</summary>  ### Visual Description ## Heatmap: Kendall's Correlation Coefficient Across Models and Benchmarks ### Overview This heatmap visualizes Kendall's Correlation Coefficient values between different AI models (y-axis) and comparison benchmarks (x-axis). The color gradient ranges from red (positive correlation, 1.0) to blue (negative correlation, -1.0), with white representing neutral values (0.0). The data suggests varying performance or alignment between models and benchmarks. ### Components/Axes - **X-axis (Comparison Benchmarks)**: KMMMLU-Redux, KMMMLU, ClimaQA, Binary Verification, Regional Identification, Regional Localization, Descriptive Generation, Area Accuracy, Concern Accuracy - **Y-axis (Models)**: K-MetBench, Reasoning, Part 1, Part 2, Part 3, Part 4, Part 5, Text-Only, Multimodal, Korean - **Legend**: Vertical colorbar on the right labeled "Kendall's Correlation Coefficient" with values from -1.0 (blue) to 1.0 (red). ### Detailed Analysis - **K-MetBench**: - KMMMLU-Redux: 0.78 (red) - KMMMLU: 0.56 (orange) - ClimaQA: 0.57 (orange) - Binary Verification: 0.17 (light orange) - Regional Identification: 0.13 (light orange) - Regional Localization: 0.33 (orange) - Descriptive Generation: 0.20 (light orange) - Area Accuracy: 0.01 (white) - Concern Accuracy: -0.03 (light blue) - **Reasoning**: - KMMMLU-Redux: 0.66 (red) - KMMMLU: 0.61 (orange) - ClimaQA: 0.57 (orange) - Binary Verification: 0.03 (white) - Regional Identification: 0.04 (white) - Regional Localization: 0.35 (orange) - Descriptive Generation: 0.22 (light orange) - Area Accuracy: -0.18 (blue) - **Part 1-5**: - Part 1: 0.78 (red), 0.60 (orange), 0.58 (orange), 0.15 (light orange), 0.08 (white), 0.32 (orange), 0.17 (light orange), 0.02 (white), -0.09 (light blue) - Part 2: 0.70 (red), 0.51 (orange), 0.53 (orange), 0.18 (light orange), 0.12 (white), 0.35 (orange), 0.27 (light orange), 0.02 (white), 0.03 (white) - Part 3: 0.75 (red), 0.52 (orange), 0.56 (orange), 0.18 (light orange), 0.16 (white), 0.30 (orange), 0.20 (light orange), -0.01 (white), -0.03 (light blue) - Part 4: 0.79 (red), 0.57 (orange), 0.57 (orange), 0.17 (light orange), 0.10 (white), 0.34 (orange), 0.18 (light orange), 0.03 (white), -0.06 (light blue) - Part 5: 0.78 (red), 0.57 (orange), 0.56 (orange), 0.18 (light orange), 0.13 (white), 0.31 (orange), 0.19 (light orange), 0.01 (white), -0.03 (light blue) - **Text-Only**: - KMMMLU-Redux: 0.78 (red) - KMMMLU: 0.56 (orange) - ClimaQA: 0.57 (orange) - Binary Verification: 0.18 (light orange) - Regional Identification: 0.12 (white) - Regional Localization: 0.34 (orange) - Descriptive Generation: 0.19 (light orange) - Area Accuracy: 0.01 (white) - Concern Accuracy: -0.03 (light blue) - **Multimodal**: - KMMMLU-Redux: 0.29 (light orange) - KMMMLU: 0.20 (light orange) - ClimaQA: 0.28 (light orange) - Binary Verification: 0.10 (white) - Regional Identification: 0.07 (white) - Regional Localization: 0.29 (orange) - Descriptive Generation: 0.22 (light orange) - Area Accuracy: 0.09 (white) - Concern Accuracy: -0.03 (light blue) - **Korean**: - KMMMLU-Redux: 0.70 (red) - KMMMLU: 0.57 (orange) - ClimaQA: 0.50 (orange) - Binary Verification: 0.15 (light orange) - Regional Identification: 0.10 (white) - Regional Localization: 0.32 (orange) - Descriptive Generation: 0.08 (white) - Area Accuracy: 0.02 (white) - Concern Accuracy: -0.11 (blue) ### Key Observations 1. **High Positive Correlations**: - K-MetBench and its subparts (Part 1-5) show strong positive correlations (0.56–0.79) with KMMMLU-Redux, KMMMLU, and ClimaQA. - Regional Localization and Descriptive Generation benchmarks also exhibit moderate to strong correlations (0.13–0.35) across most models. 2. **Negative Correlations**: - Area Accuracy and Concern Accuracy benchmarks show negative values (-0.03 to -0.18) for several models, indicating inverse relationships. 3. **Model-Specific Trends**: - **Multimodal**: Lower correlations (0.10–0.29) compared to text-only models, suggesting weaker alignment with benchmarks. - **Korean**: Mixed performance, with strong correlations for KMMMLU-Redux (0.70) but negative values for Concern Accuracy (-0.11). 4. **Anomalies**: - Area Accuracy has the most negative values (-0.18 for Reasoning, -0.11 for Korean), potentially indicating misalignment or errors in this benchmark. ### Interpretation The heatmap highlights that **K-MetBench and its subparts** generally perform better on reasoning and localization tasks (e.g., KMMMLU-Redux, ClimaQA), while **Multimodal models** underperform on most benchmarks. The **Korean model** shows mixed results, excelling in some areas but struggling with Concern Accuracy. Negative correlations in Area Accuracy suggest potential flaws in this benchmark or model-specific limitations. The data implies that text-based models (e.g., K-MetBench) are more reliable for the evaluated tasks, while multimodal approaches may require refinement for specific applications. </details> subset ( τ b = 0 . 29 ). Furthermore, correlations with external weather baselines ( e.g., ClimaQA, ClimaIQA, and WeatherQA ) remain consistently low across both reasoning and multimodal dimensions ( avg. τ b < 0 . 14 ). This quantitative gap demonstrates that K-MetBench evaluates specialized domain logic and visual interpretation skills that are orthogonal to general linguistic proficiency and existing meteorological tasks. Meta-Evaluation: Human-LLM Agreement. We validated our reasoning evaluation framework by measuring inter-rater agreement on 100 sampled responses ( Table 5 ). All axes surpassed the reliability threshold ( α > 0 . 7 ), with Reasoning Total achieving a robust α of 0.838. Additionally, Figure 4 illustrates a strong correlation between human and LLM scores. The w/ rationale setting yielded a Kendall ' s τ b of 0.99 with low variance, slightly outperforming the w/o rationale setting ( τ b = 0 . 96 ). Table 5: Inter-rater agreement analysis. The agreement between the average scores of two human experts and the LLM evaluator. We report Krippendorff ' s α ( interval ) and Intraclass Correlation Coefficient ( ICC, two-way mixed, absolute agreement ). | Evaluation Axis | Krippendorff ' s α | ICC | N | |-------------------|----------------------|-------|-----| | Factuality | 0.827 | 0.829 | 100 | | Logicality | 0.827 | 0.83 | 100 | | Depth | 0.742 | 0.747 | 100 | | Clarity | 0.825 | 0.827 | 100 | | Reasoning Total | 0.838 | 0.841 | 100 | Figure 4: Scatter plot comparing human expert vs. LLM-judge scores. The w/ rationale condition ( τ b = 0 . 99 ) shows slightly higher precision and lower variance than the w/o rationale condition ( τ b = 0 . 96 ), while both maintain a strong correlation. <details> <summary>Image 4 Details</summary>  ### Visual Description ## Scatter Plot: LLM Performance vs Human Expertise ### Overview This scatter plot compares Large Language Model (LLM) performance (LLM-Judge Score) against human expert evaluations (Human Expert Score) across different model sizes and configurations. The plot includes performance tiers (Top/Mid/Low) and distinguishes models with/without rationale generation capabilities. ### Components/Axes - **X-axis**: Human Expert Score (5-16 scale, increments of 2) - **Y-axis**: LLM-Judge Score (2.5-20 scale, increments of 2.5) - **Legends**: 1. **Model Size** (top-left): - 0.6B (smallest circles) - 20B (medium circles) - 235B (largest circles) 2. **Performance Tier** (top-right): - Top tier (dark blue) - Mid tier (medium blue) - Low tier (light blue) 3. **Rationale Presence** (bottom-right): - w/ rationale (solid blue circles) - w/o rationale (hollow purple circles) - **Trend Line**: Dashed purple line with shaded confidence interval (light purple) ### Detailed Analysis 1. **Model Size Correlation**: - Larger models (235B) cluster at higher scores (e.g., GPT-4: 16→18, A.X-4.0: 15→17) - Smaller models (0.6B) cluster at lower scores (Qwen-0.6B: 5→4) - Mid-sized models (20B) show mixed performance (e.g., VARCO-VISION: 10→12) 2. **Performance Tiers**: - Top tier models (dark blue) generally have Human Expert Scores ≥12 and LLM Scores ≥12 - Mid tier models (medium blue) cluster between 8-12 on both axes - Low tier models (light blue) appear below 8 on both axes 3. **Rationale Impact**: - Models with rationale (solid blue) consistently outperform identical models without rationale (hollow purple) - Example: InternVL-3_5-8B-Instruct shows 5→6 with rationale vs 5→5 without 4. **Trend Line Analysis**: - Positive correlation (R² ≈ 0.85) between human and LLM scores - Shaded confidence interval widens at mid-range scores (8-12), indicating greater variability - Slope ≈ 0.95 suggests near-parity between human and LLM evaluations at higher scores ### Key Observations 1. **Size-Score Relationship**: - 235B models achieve 15-20 LLM scores vs 12-16 human scores - 0.6B models achieve 4-6 LLM scores vs 5-7 human scores - 20B models show variable performance (6-12 LLM scores) 2. **Rationale Effectiveness**: - Rationale improves scores by 0.5-1.5 points across all model sizes - Most significant gains observed in mid-tier models (e.g., GPT-OSS-20B: +1 point) 3. **Performance Tiers**: - Top tier models maintain ≥12 LLM score regardless of rationale - Mid tier models show 10-12 LLM scores with rationale - Low tier models cluster below 8 LLM score ### Interpretation The data demonstrates that: 1. **Model Size Matters**: Larger models consistently achieve higher scores, with 235B models approaching human expert levels (16→18) 2. **Rationale Enhances Performance**: Across all sizes, models with rationale outperform identical configurations without, particularly in mid-tier ranges 3. **Human-LLM Alignment**: The near-slope of 1 indicates strong alignment between human and LLM evaluations at higher performance levels 4. **Diminishing Returns**: While larger models perform better, the performance gap narrows at the highest scores (e.g., GPT-4: 16→18 vs A.X-4.0: 15→17) Notable anomalies include: - GPT-OSS-120B (20B) achieving Top tier scores despite mid-sized architecture - InternVL-3_5-8B-Instruct showing minimal improvement with rationale (+1 point) - Qwen-0.6B maintaining near-parity between human and LLM scores despite small size This suggests that while model scale is crucial, architectural choices (rationale generation) and training objectives significantly impact performance, particularly in mid-tier ranges where human-LLM alignment is most variable. </details> ## 6 Discussion ## 6.1 The Challenge of Visual Reasoning in Specialized Domains The observed modality gap in Table 4 and Figure 2 underscores a fundamental limitation: current MLLMs lack the domain-specific visual literacy necessary for forecasting. Although proficient in general recognition, models struggle to ground specialized visual patterns-such as isobars, fronts, and wind barbs-in physical principles. This indicates that training on general imagetext pairs is insufficient for mastering the finegrained visual reasoning required in specialized scientific domains. ## 6.2 Geo-Cultural Alignment in Meteorology Meteorology requires applying universal laws to localized contexts. The observed performance gap indicates a critical lack of geo-cultural alignment in global models. Despite linguistic fluency, multilingual models frequently hallucinate on specific Korean geographic and terminological nuances. Consequently, effective deployment in vertical domains demands more than mere scaling; it requires rigorous alignment with local topographic and legal contexts to bridge the gap between general capability and expert-level application. ## 6.3 Superficial Reasoning vs. Causal Deduction The observation that models output correct answers with shallow or erroneous explanations points to shortcut learning ( Geirhos et al. , 2020 ) -a reliance on surface-level associations rather than genuine understanding. Furthermore, the inability to reach expert-level performance on formulabased problems ( e.g., calculating geostrophic wind speed ) highlights a critical deficiency in applying physical laws. Addressing this requires shifting from general instruction tuning to training on highquality reasoning trace data grounded in rigorous physical principles. ## 6.4 Reliability of Automated Evaluation in Specialized Domains Our results confirm that Gemini-2.5-Pro is a reliable proxy for human experts in meteorology. The high agreement in Factuality and Logicality in Table 5 demonstrates objective evaluation of logic and evidence. While Depth showed slightly more subjectivity, the overall consistency supports the framework ' s robustness. Furthermore, the tight correlation observed in the scatter plot in Figure 4 indicates that expert rationales effectively minimize variance. However, the model ' s high intrinsic knowledge ensures reliable grading even in their absence. These findings demonstrate that, when guided by high-quality rubrics, modern LLMsare cost-effective and reliable judges even in fine-grained domains like meteorology. This validates adopting the LLM-as-a-Judge framework for the reasoning evaluation in this study. ## 7 Conclusion We present K-MetBench, a multi-dimensional benchmark for fine-grained evaluation of large language models in meteorological reasoning. By decomposing performance across modality, reasoning quality, geo-cultural context, and domainspecific sub-fields, K-MetBench provides diagnostic insights that are not observable from aggregate accuracy alone. Our evaluation reveals persistent challenges in interpreting domain-specific visual artifacts, producing coherent expert-level rationales, and grounding meteorological knowledge in local context. In addition, analysis across official subject areas exposes uneven performance that is obscured by holistic scores. Overall, KMetBench is intended as a diagnostic complement to existing benchmarks, helping identify where current models succeed and where targeted improvements are needed for reliable deployment in specialized scientific domains. ## Limitations While K-MetBench serves as a rigorous diagnostic tool for meteorological AI, we acknowledge several limitations. First, regarding modality, the benchmark focuses on static visual reasoning ( e.g., snapshot weather charts ). While interpreting these charts is fundamental to forecasting, the current dataset does not evaluate the temporal reasoning required to interpret atmospheric evolution, such as sequential radar imagery or satellite loops. Second, the dataset is geo-specifically rooted in the Korean context. Although this design effectively evaluates geo-cultural alignment-a key contribution of our work-it inherently limits direct generalizability to other climatic regions without adaptation. Finally, we utilized the official examination passing criteria ( 60% ) as a proxy for human competency. While this provides a validated baseline for qualification, a fine-grained human expert ceiling ( e.g., the upper-bound score of top-tier meteorologists ) was not explicitly measured in this study. Future work will focus on establishing this upper bound to quantify the ' super-human ' gap precisely. ## Ethical Considerations We adhered to copyright laws and ethical guidelines in constructing K-MetBench. The dataset is derived from National Meteorological Engineer examinations administered from March 16, 2003 to March 5, 2022; among 43 sessions in this period, we used only the 25 that were officially released to the public. We also obtained explicit permission from the Human Resources Development Service of Korea ( HRDK ) to use these materials for research and to release the refined dataset in an open repository. In addition, the dataset was reviewed to ensure that it contains no personally identifiable information or harmful content. For human annotation, we involved two domain experts from collaborating institutions in the same funded project: one university professor and one research professor. The same experts conducted both reference-rationale verification and scoring of model-generated reasoning, and these activities were compensated separately on a per-item basis in accordance with our institution ' s internal standards for expert advisory and review work. We consider this compensation appropriate given the experts ' seniority, domain expertise, and expected time commitment. ## Licensing and Legal Compliance The K-MetBench dataset is derived from public examination materials managed by the HRDK. We conducted a rigorous legal review to ensure compliance with the Official Information Disclosure Act and relevant copyright laws ( Copyright Act Art. 24-2, 25 ) in Korea . We confirmed that the questions are not classified as restricted information. To support the research community, the curated dataset is released via an open repository under the CC BY-NC-ND license, permitting noncommercial research use while preserving the integrity of the original artifacts. ## Acknowledgments We express our gratitude to the Human Resources Development Service of Korea ( HRDK ) for allowing the use of National Technical Qualification Examination data for research purposes. We would like to thank Seongsu Bae and the anonymous reviewers for their valuable comments. This research was supported by the HighPerformance Computing Support Project, funded by the Ministry of Science and ICT ( MSIT ) and the National IT Industry Promotion Agency ( NIPA ) under grant No. RQT-25-070278 ( providing 40 H100 GPUs ). This work was also supported by the Institute for Information & Communications Technology Planning & Evaluation ( IITP ) grant funded by the Korea government ( MSIT ) ( No. RS-2019II190075, Artificial Intelligence Graduate School Program ( KAIST ); and No. RS-2022-II220984, Development of Artificial Intelligence Technology for Personalized Plug-and-Play Explanation and Verification of Explanation ), and by the Korea Meteorological Administration ( KMA ) and National Institute of Meteorological Sciences ( NIMS ) under grant No. KMA2021-00123 ( Developing Intelligent Assistant Technology and Its Application for Weather Forecasting Process ). ## Data and Code Availability The dataset is hosted on HuggingFace at https://huggingface.co/datasets/ soyeonbot/K-MetBench . The evaluation toolkit is available at https://github.com/ kmetbench/kmetbench-release . The KMetBench leaderboard is publicly available at https://kmetbench.github.io/ . ## References - Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, and 1 others. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925 . - Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 . - Young-rok Cha, Jeongho Ju, SunYoung Park, JongHyeon Lee, Younghyun Yu, and Youngjune Kim. 2025. Varco-vision-2.0 technical report. arXiv preprint arXiv:2509.10105 . - Jian Chen, Peilin Zhou, Yining Hua, Dading Chong, Meng Cao, Yaowei Li, Wei Chen, Bing Zhu, Junwei Liang, and Zixuan Yuan. 2025. Climateiqa: A new dataset and benchmark to advance vision-language models in meteorology anomalies analysis. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 , pages 5322-5333. - Team Cohere, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Milad Alizadeh, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, and 1 others. 2025. Command a: An enterprise-ready large language model. arXiv preprint arXiv:2504.00698 . - Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2 ( 11 ): 665-673. - Seokhee Hong, Sunkyoung Kim, Guijin Son, Soyeon Kim, Yeonjung Hong, and Jinsik Lee. 2025. From KMMLU-redux to pro: A professional Korean benchmark suite for LLM evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2025 , pages 19067-19096, Suzhou, China. Association for Computational Linguistics. - Jenny Y. Huang, Yunyi Shen, Dennis Wei, and Tamara Broderick. 2026. Dropping just a handful of preferences can change top large language model rankings. In The Fourteenth International Conference on Learning Representations . - Human Resources Development Service of Korea. 2024. Examination standards for meteorological engineer ( 2023.1.1-2026.12.31 ). https://www.q-net.or.kr/pageLink.do? link=cst/cstReport . Accessed: 2026-01-06. Available at Q-Net. - Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2024. Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A , 382 ( 2270 ): 20230254. - Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles , pages 611-626. - SKT AI Model Lab. 2025. A.X 4.0. - Katherine Lee, Daphne Ippolito, A. Nystrom, Chiyuan Zhang, D. Eck, Chris Callison-Burch, and Nicholas Carlini. 2021. Deduplicating training data makes language models better. In Annual Meeting of the Association for Computational Linguistics . - Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579 . - Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, and 1 others. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 . - Chengqian Ma, Zhanxiang Hua, Alexandra AndersonFrey, Vikram Iyer, Xin Liu, and Lianhui Qin. 2024. Weatherqa: Can multimodal language models reason about severe weather? arXiv preprint arXiv:2406.11217 . - Veeramakali Vignesh Manivannan, Yasaman Jafari, Srikar Eranky, Spencer Ho, Rose Yu, Duncan Watson-Parris, Yian Ma, Leon Bergen, and Taylor Berg-Kirkpatrick. 2024. Climaqa: An automated evaluation framework for climate question answering models. arXiv preprint arXiv:2410.16701 . - Meta. 2024. Llama 3.2 model card. Accessed: 202401-04. - Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 . - OpenAI. 2025. Update to gpt-5 system card: Gpt-5.2. Accessed: 2026-01-04. - A Yang Qwen, Baosong Yang, B Zhang, B Hui, B Zheng, B Yu, Chengpeng Li, D Liu, F Huang, H Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint . - LG Research, Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Yemuk Choi, Kyubeen Han, Seokhee Hong, Junwon Hwang, Taewan Hwang, and 1 others. 2025. Exaone 4.0: Unified large language models integrating non-reasoning and reasoning modes. arXiv preprint arXiv:2507.11407 . - Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, and 1 others. 2023. Large language models encode clinical knowledge. Nature , 620 ( 7972 ): 172-180. - Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2025. KMMLU: Measuring massive multitask language understanding in Korean. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies ( Volume 1: Long Papers ) , pages 4076-4104, Albuquerque, New Mexico. Association for Computational Linguistics. - Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085 . - Gemini Team, Rohan Anil, Sebastian Borgeaud, JeanBaptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 . - Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. 2023. Language models don ' t always say what they think: Unfaithful explanations in chain-ofthought prompting. Advances in Neural Information Processing Systems , 36:74952-74965. - Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265 . - Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:2482424837. - An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 . - Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, KyungMinKim, Munhyong Kim, Sungju Kim, and 1 others. 2024. Hyperclova x technical report. arXiv preprint arXiv:2404.01954 . - Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems , 36:46595-46623. ## Appendix ## Table of Contents | A | Dataset Examples | Dataset Examples | 15 | |-----|-------------------------------------------------------|------------------------------------------------------------------|------| | B | Case Study of Reasoning Answer | Case Study of Reasoning Answer | 15 | | C | Prompts and Questionnaires for Benchmark Construction | Prompts and Questionnaires for Benchmark Construction | 15 | | | C.1 | Question Paraphrasing . . . . . . . . . . . . . . . . . . . . . | 15 | | | C.2 | Identification of Korean-Specific Subset . . . . . . . . . . . . | 15 | | | C.3 | Implicit vs. Explicit Dataset Design for Korean-Specific Subset | 15 | | | C.4 | Evaluation Prompts for Korean-Specific Subset . . . . . . . . | 16 | | | C.5 | Prompt for Reference Rationale Generation . . . . . . . . . . | 16 | | | C.6 | Questionnaire for Expert Verification on Reference Rationale . | 16 | | | C.7 | Reasoning Prompt for Open-Source LLMs . . . . . . . . . . | 17 | | | C.8 | Prompt for LLM-as-a-Judge Evaluation . . . . . . . . . . . . | 17 | | | C.9 | Questionnaire for Expert Scoring of LLM Reasoning . . . . . | 17 | | D | Experimental Setups | Experimental Setups | 17 | | | D.1 | Reasoning Model Inference . . . . . . . . . . . . . . . . . . | 17 | | | D.2 | Meta-Evaluation for LLM-as-a-Judge . . . . . . . . . . . . . | 18 | | E | Additional Results and Discussion | Additional Results and Discussion | 18 | | | E.1 | Detailed Orthogonality Analysis of K-MetBench . . . . . . . | 18 | | | E.2 | Detailed Analysis of K-MetBench Performance . . . . . . . . | 19 | | | E.3 | Results of Meta Evaluation of LLM-as-a-Judge . . . . . . . . | 20 | | | E.4 | MCQA Accuracy vs. Reasoning Score . . . . . . . . . . . . | 20 | | | E.5 | Computational Cost and Efficiency Analysis . . . . . . . . . | 21 | | F | Compute Resources | Compute Resources | 21 | | G | Hierarchical Topic Distribution | Hierarchical Topic Distribution | 21 | | H | Robustness of Conclusions Under Small Subsets | Robustness of Conclusions Under Small Subsets | 38 | | | H.1 | Statistical Robustness Diagnostics . . . . . . . . . . . . . . . | 38 | | | H.2 | Robustness of Key Findings to Critical Data Perturbation . . . | 39 | Table 6: Representative examples of K-MetBench tasks. The examples are organized by modality: Text-only ( Top ) and Multimodal ( Bottom ). We showcase three task types within each modality: Standard ( fundamental knowledge ), K-Specific ( geo-cultural context ), and Reasoning ( complex deduction ). Part denotes the corresponding subject from the five official fields. Gray text indicates English translations. <details> <summary>Image 5 Details</summary>  ### Visual Description ## Text-Based Comparison of MCQA Models ### Overview The image presents a comparative analysis of three multiple-choice question answering (MCQA) models: **Standard MCQA**, **K-Specific MCQA**, and **Reasoning MCQA**. Each model is represented by two sections (ID and Part numbers) with Korean text content, including questions, answer options, and ground truth labels. The structure emphasizes linguistic and reasoning capabilities across different configurations. --- ### Components/Axes - **Columns**: Three vertical sections labeled by model type: 1. **Standard MCQA** (Text-Only modality) 2. **K-Specific MCQA** (Korean-specific adaptation) 3. **Reasoning MCQA** (Expert-verified rationale) - **Rows**: Each model section contains: - **ID/Part**: Unique identifiers for question sets (e.g., ID: 15355, Part: 5). - **Question**: A Korean-language query with embedded technical context. - **Answer Options**: Four labeled choices (1–4) in Korean. - **Ground Truth**: Correct answer indicator (정답) with numerical value. --- ### Detailed Analysis #### Standard MCQA (ID: 15355, Part: 5) - **Question**: "질문: 상층 일기도의 활용에 대해 올바르게 설명한 것은?" *Translation*: "Which of the following correctly describes the utilization of upper-level weather charts?" - **Options**: 1. 500 hPa 일기도의 한랭기압골에서 등온선의 전폭이 등고선의 전폭보다 클 경우에... 2. 300 hPa 면에서는 온도가 지형, 복사의 영향을 받으므로 전선분석이 용이하다. 3. 300 hPa 제트기류 출구의 좌측에 하강기류, 우측에 상승기류가 있으며... 4. 500 hPa 기류가 지상 한랭전선에 수직으로 불면 이 전선은 활성으로서 악천이 나타난다. - **Ground Truth**: 정답: 1 *Translation*: Correct answer: 1 #### K-Specific MCQA (ID: 65, Part: 5) - **Question**: "질문: 다음은 한국 지역에 영향을 주는 고기압의 특성을 설명한 것이다. 다음이 옳지 않은 것은?" *Translation*: "Which statement is incorrect regarding high-pressure systems affecting the Korean region?" - **Options**: 1. 시베리아 고기압은 겨울철의 춥고 건조한 날씨를 만든다. 2. 오호츠크해 고기압은 동해안 지방의 고온현상을 일으킨다. 3. 북태평양 고기압은 고온다습하며, 여름철의 무더운 날씨를 만든다. 4. 이동성 고기압의 영향을 받으면 봄에는 따뜻한 날씨, 가을에는 맑은 날씨가 된다. - **Ground Truth**: 정답: 2 *Translation*: Correct answer: 2 #### Reasoning MCQA (ID: 18, Part: 2) - **Question**: "질문: 비열의 차원을 올바르게 나타낸 것은 무엇입니까?" *Translation*: "What is the correct dimensional representation of specific heat?" - **Options**: 1. [SL^2T^2\\theta^(-1)] 2. [SL^2T^(-2)\\theta^(-1)] 3. [SML^(-1)T^(-2)] 4. [SML^2T^(-2)] - **Ground Truth**: 정답: 2 *Translation*: Correct answer: 2 - **Expert Verification**: - Option 2 matches the formula for specific heat: $ \text{Specific Heat} = \frac{\text{Energy}}{\text{Mass} \cdot \Delta\text{Temperature}} $ - Option 1 reverses the time exponent, making it incorrect. --- ### Key Observations 1. **Standard MCQA** focuses on atmospheric phenomena (e.g., jet streams, temperature gradients). 2. **K-Specific MCQA** emphasizes regional meteorological impacts (e.g., Siberian High, Okhotsk Sea). 3. **Reasoning MCQA** tests dimensional analysis and formula derivation (e.g., specific heat units). --- ### Interpretation - The **Standard MCQA** evaluates foundational knowledge of weather systems, while the **K-Specific MCQA** tailors questions to Korean geographic and climatic contexts. - The **Reasoning MCQA** introduces mathematical rigor, requiring understanding of physical formulas (e.g., specific heat capacity). - The inclusion of **ground truth labels** (정답) suggests these are test datasets for model evaluation, with Part 5 and Part 2 likely representing different difficulty tiers or question categories. - The **Reasoning MCQA** explicitly references "Expert-Verified Rationale," indicating a focus on logical deduction rather than rote memorization. This structure highlights how each model adapts to linguistic specificity, regional relevance, and analytical depth, providing a framework for evaluating MCQA performance across diverse cognitive demands. </details> Multimodal ## ID: 460, Part: 3 질문: 북반구에서나타나는지균풍 ( $\vec { V\_ { g }} $ ), 실제풍( $\vec { V } $ ), 수평가속도 ( $ ( \frac { d\vec { v }}{ dt }) \_ { H } $ ) 사이의관계를 올바르게표현한그림은어느것인가? Question: Which figure correctly represents the relationship between the geostrophic wind ( $\vec { V\_ { g }} $ ), the actual wind ( $\vec { V } $ ), and the horizontal acceleration ( $ ( \frac { d\vec { v }} { dt }) \_ { H } $ ) in the Northern Hemisphere? <details> <summary>Image 6 Details</summary>  ### Visual Description ## Diagrams: Vector Components with Vertical Acceleration ### Overview The image contains four labeled diagrams (1–4) depicting vector relationships involving velocity vectors ($\vec{V}$), gravitational velocity ($\vec{V}_g$), and a vertical acceleration component ($\frac{d\vec{v}}{dt}_H$). Each diagram shows a right triangle formed by $\vec{V}$, $\vec{V}_g$, and $\frac{d\vec{v}}{dt}_H$, with directional arrows indicating vector orientations. ### Components/Axes - **Vectors**: - $\vec{V}$: Hypotenuse of the triangle, representing total velocity. - $\vec{V}_g$: Horizontal leg, representing gravitational velocity. - $\frac{d\vec{v}}{dt}_H$: Vertical leg, representing the vertical component of acceleration. - **Labels**: - All diagrams share identical labels ($\vec{V}$, $\vec{V}_g$, $\frac{d\vec{v}}{dt}_H$). - No numerical values or scales are provided. - **Legend**: - No explicit legend is present. ### Detailed Analysis 1. **Diagram 1**: - $\frac{d\vec{v}}{dt}_H$ points **rightward** (horizontal direction). - $\vec{V}$ slopes upward from $\vec{V}_g$. 2. **Diagram 2**: - $\frac{d\vec{v}}{dt}_H$ points **downward** (vertical direction). - $\vec{V}$ slopes upward from $\vec{V}_g$. 3. **Diagram 3**: - $\frac{d\vec{v}}{dt}_H$ points **leftward** (horizontal direction). - $\vec{V}$ slopes upward from $\vec{V}_g$. 4. **Diagram 4**: - $\frac{d\vec{v}}{dt}_H$ points **upward** (vertical direction). - $\vec{V}$ slopes upward from $\vec{V}_g$. ### Key Observations - The vertical acceleration component ($\frac{d\vec{v}}{dt}_H$) varies in direction across diagrams: right (1), down (2), left (3), and up (4). - $\vec{V}$ consistently slopes upward relative to $\vec{V}_g$ in all cases. - **Ground Truth**: The annotation "정답: 3" (Korean for "Answer: 3") and "Ground Truth: 3" explicitly identifies Diagram 3 as the correct representation. ### Interpretation The diagrams likely illustrate scenarios where the vertical acceleration component ($\frac{d\vec{v}}{dt}_H$) changes direction due to external factors (e.g., rotational motion, non-inertial frames, or directional forces). Diagram 3’s leftward $\frac{d\vec{v}}{dt}_H$ suggests a specific context where acceleration opposes the horizontal velocity ($\vec{V}_g$), possibly indicating a centripetal force or directional constraint. The consistency of $\vec{V}$’s upward slope implies that gravitational velocity ($\vec{V}_g$) remains dominant in the horizontal direction across all cases. The ground truth designation of Diagram 3 highlights its relevance to the problem’s physical or mathematical conditions, such as a specific coordinate system or force application scenario. </details> Ground Truth: 3 ## ID: 687, Part: 4 질문: 제시된그림은 한국 의어떤지점의연평균 물수지를보여준다.이그림에서 D 부분이 의미하는것은무엇인가? Question: The presented figure shows the annual average water balance of a certain location in Korea . What does section D in this figure represent? <details> <summary>Image 7 Details</summary>  ### Visual Description ## Line Graph: Solar Radiation vs. Evaporation Over Time ### Overview The image depicts a line graph comparing two variables: "광수량" (Solar Radiation) and "증발산위" (Evaporation). The graph shows temporal trends with labeled data points (A, B, C, D) and directional arrows indicating key trends. ### Components/Axes - **X-axis**: "시간" (Time) – No explicit scale or units provided. - **Y-axis**: "단위" (Unit) – No explicit scale or units provided. - **Legend**: - Solid line: "광수량" (Solar Radiation) - Dashed line: "증발산위" (Evaporation) - **Key Labels**: - "C": Peak of the solid line (Solar Radiation). - "증발산위" (Evaporation): Dashed line with a downward trend. - Arrows: - One pointing to the peak of the solid line (C). - Another pointing to the dashed line's trend. ### Detailed Analysis - **Solar Radiation (Solid Line)**: - Rises sharply to a peak at point **C**. - Declines gradually after the peak. - No numerical values provided for the y-axis. - **Evaporation (Dashed Line)**: - Follows a smoother, less steep decline compared to Solar Radiation. - Positioned consistently below the Solar Radiation line. - No numerical values provided for the y-axis. - **Data Points**: - **A**: Low point on the Solar Radiation line (left side). - **B**: Intermediate point on the Solar Radiation line (left side). - **C**: Peak of the Solar Radiation line (center). - **D**: Intermediate point on the Evaporation line (right side). ### Key Observations 1. Solar Radiation peaks at **C**, suggesting a midday or noon-time maximum. 2. Evaporation trends downward but remains below Solar Radiation levels throughout. 3. The dashed line (Evaporation) shows a less pronounced decline than the solid line (Solar Radiation). 4. No explicit numerical values or units are provided for either axis. ### Interpretation The graph illustrates a relationship between Solar Radiation and Evaporation over time. The peak in Solar Radiation at **C** likely corresponds to the time of maximum sunlight exposure (e.g., solar noon). The Evaporation line’s gradual decline suggests that evaporation rates are influenced by Solar Radiation but may lag or be modulated by other factors (e.g., humidity, temperature). The absence of numerical data limits quantitative analysis, but the visual trends imply that Solar Radiation drives Evaporation, with diminishing returns as the day progresses. The lack of units on both axes introduces uncertainty in interpreting absolute values. </details> 1. 토양수분의과잉 2. 토양수분의보충 3. 토양수분의이용 4. 토양수분의결핍 1. Soil moisture surplus 2. Soil moisture recharge 3. Soil moisture utilization 4. Soil moisture deficit ## 정답: 3 Ground Truth: 3 ## ID: 460, Part: 3 질문: 다음그림이보여주는역전층의종류로 옳은것은? Question: Which of the following is the correct type of inversion layer shown in the figure below? <details> <summary>Image 8 Details</summary>  ### Visual Description ## Pressure-Temperature Phase Diagram ### Overview The image depicts a pressure-temperature (P-T) phase diagram with pressure (hPa) on the vertical axis and temperature (°C) on the horizontal axis. The diagram includes two distinct data series represented by dashed and solid lines, along with labeled pressure isotherms and temperature markers. The legend identifies the lines as "고정온도" (fixed temperature) and "기온" (temperature). ### Components/Axes - **Y-Axis (Pressure)**: Labeled "압력 (hPa)" with values ranging from 700 to 1000 hPa in 100 hPa increments. - **X-Axis (Temperature)**: Labeled "기온 (°C)" with markers at -20, -15, -10, -5, 0, and 5°C. - **Legend**: Located at the bottom, with: - **Dashed Line**: "고정온도" (fixed temperature) - **Solid Line**: "기온" (temperature) ### Detailed Analysis 1. **Pressure Isotherms**: - Horizontal lines labeled 700, 800, 900, and 1000 hPa span the graph, indicating constant-pressure contours. 2. **Data Series**: - **Dashed Line ("고정온도")**: - Connects points at (-20°C, 700 hPa), (-15°C, 800 hPa), (-10°C, 900 hPa), and (-5°C, 1000 hPa). - Exhibits a **negative slope**, suggesting an inverse relationship between temperature and pressure for this series. - **Solid Line ("기온")**: - Starts at (-20°C, 700 hPa) and ascends to (5°C, 1000 hPa). - Shows a **positive slope**, indicating a direct relationship between temperature and pressure. 3. **Intersection**: - The two lines intersect at approximately (-10°C, 900 hPa), forming a triangular region between the dashed and solid lines. ### Key Observations - The dashed line ("고정온도") decreases in pressure as temperature increases, while the solid line ("기온") increases in pressure with rising temperature. - The triangular region between the lines suggests a phase transition zone or critical boundary. - Pressure isotherms (700–1000 hPa) are evenly spaced, implying uniform pressure increments. ### Interpretation This diagram likely represents thermodynamic behavior, such as phase changes (e.g., liquid-vapor equilibrium) or material properties under varying conditions. The "고정온도" (fixed temperature) line may depict isothermal processes, while "기온" (temperature) could represent adiabatic or dynamic processes. The intersection at (-10°C, 900 hPa) might correspond to a critical point where phase transitions occur, such as the boiling point of a substance under specific pressure. The negative slope of the dashed line contrasts with the positive slope of the solid line, highlighting divergent behaviors under fixed vs. variable temperature conditions. </details> 1. 복사역전 2. 난류역전 3. 전선역전 4. 침강역전 1. Radiation inversion 2. Turbulence inversion 3. Frontal inversion 4. Subsidence inversion 전문가검증참조자료:그림처럼지표에서바로 시작하는얕은역전층이위로갈수록약화되는 형태는야간지표복사냉각으로생기는 복사역전의전형이며,침강역전은고기압하 하강류로상층에분리되어나타나고전선역전은 전선면을따라경사져있으며난류역전은주간 혼합층꼭대기에형성되어지표에서시작하지 않으므로그림과다르다. Expert-Verified Rationale: As shown in the figure, a shallow inversion layer starting directly from the surface and weakening with height is typical of a radiation inversion caused by nocturnal surface radiative cooling. In contrast, a subsidence inversion appears separated in the upper atmosphere due to downdrafts under high pressure, a frontal inversion is tilted along a frontal surface, and a turbulence inversion forms at the top of the daytime mixed layer rather than starting at the surface; thus, they differ from the figure. 정답: Ground Truth: 1 <details> <summary>Image 9 Details</summary>  ### Visual Description Icon/Small Image (26x22) </details> <details> <summary>Image 10 Details</summary>  ### Visual Description Icon/Small Image (23x18) </details> ## Table 7: Comparison of InternVL3.5-8B-Instruct and Qwen3-VL-235B-A22B-Thinking on K-MetBench reasoning question. ( X : Incorrect, ✓ : Correct, shaded text : Highlighted errors ) ## Reasoning MCQA ID: 105, Part: 3 질문: 지균풍관계식에서기압경도와 공기밀도가일정하다면, 풍속은 어떻 게결정되는가? Question: In the geostrophic wind formula, if the pressure gradient and air density are constant, how is the wind speed determined? 1. 고도가증가함에따라증가한다. 2. 3. 1. 2. 3. 위도의증가에따라감소한다. 기압의증가에따라증가한다. 4. 4. 공기의점성이크면증가한다. 1. It increases as altitude increases. 2. It decreases as latitude increases. 3. It increases as pressure increases. 4. It increases if air viscosity is high. ## 전문가검증참조자료: 지균풍은 Vg = | p|/ ( ρ f )이므로기압경도와밀도가일정 하면코리올리매개변수 f=2Ω sinφ 만이 변수가되어위도가증가할수록 f 가커 져풍속은감소하며,절대기압의크기 (보기 3 )나 고도(보기 1 ), 점성(보기 4 ) 은이관계식에직접등장하지않는다. Expert-Verified Rationale: Since the geostrophic wind is defined as Vg = | p|/ ( ρ f ), if the pressure gradient and density are constant, the Coriolis parameter f=2Ω sinφ becomes the only variable. As latitude increases, f increases, causing the wind speed to decrease. The magnitude of absolute pressure ( Option 3 ), altitude ( Option 1 ), and viscosity ( Option 4 ) do not appear directly in this equation. ## 정답: 2 Ground Truth: 2 ## Reasoning Evaluation of InternVL3.5-8B-Instruct ## Response from InternVL3.5-8B-Instruct 생성된 \_ \_ 추론 근거: 지연풍 관계식에서풍속이기압경로의변화와 <details> <summary>Image 11 Details</summary>  ### Visual Description Icon/Small Image (48x23) </details> 직접적으로연관된다는것을이해할수있습니다. 기압경도 가 증가하면풍속도증가하는경향이있으며,이는풍속이기압의증가에 따라증가한다는 3 번옵션이정답이라는근거를제공합니다. Expert-Verified Rationale: From the geostrophic wind relationship , we can understand that wind speed is directly associated with changes in the pressure path. Since wind speed tends to increase when the pressure gradient increases, this provides grounds for Option 3-that wind speed increases as pressure increases-being the correct answer. 정답: <details> <summary>Image 12 Details</summary>  ### Visual Description Icon/Small Image (19x23) </details> <details> <summary>Image 13 Details</summary>  ### Visual Description Icon/Small Image (17x20) </details> Predicted Answer: 3 ## Reasoning Evaluation from Human Experts: 사람전문가평균점수: 사실적정확성( 1 ), 논리적 완결성( 1 ), 추론의 깊이 ( 1 ), 표현의 명확성( 1 ) Average of Human Expert Score: factuality ( 1 ), logicality ( 1 ), depth ( 1 ), clarity ( 1 ) 평가의견 1: 오답및사용된용어부적절: 기압경로 ,지연풍등 평가의견 2: 문제에맞지않는추론근거및답도틀림. Evaluation Comment 1: Incorrect answer and inappropriate use of terminology: <details> <summary>Image 14 Details</summary>  ### Visual Description Icon/Small Image (86x26) </details> , delayed wind etc. ' ', Evaluation Comment 2: The reasoning is irrelevant to the question, and the answer is incorrect. ## Reasoning Evaluation from LLM-Evaluator ( Gemini-2.5-Pro ): ## 사실 \_ 오류 \_ 목록: [ 지균풍( geostrophic wind )을'지연풍'이라는존재하지않는기상학용어로 잘못지칭하였습니다., 문제에서'기압경도가일정하다'는핵심가정을무시하고,'기압경도가 증가하면풍속이증가한다'는관계를추론의근거로사용하였습니다. '기압경도( pressure gradient )'와 '기압( pressure )'이라는물리적으로다른 개념을혼동하여,기압경도와풍속의관계를근거로기압과풍속의관계를 설명하는오류를범했습니다. ] 평가 \_ 점수: 정확성: 1, 논리적 \_ 완결성: 1, 추론의 \_ 깊이: 1, 표현의 \_ 명확성: 1 평가 \_ 사유: 수험생 AI 의답변은모든평가축에서최하점을기록했습니다. '정확성'측면에서,'지균풍'을'지연풍'이라는환각용어로지칭했으며, '기압경도'와'기압'을혼동하는치명적인개념오류를보였습니다.또한 문제의핵심전제인'기압경도일정'을무시하여사실관계를 왜곡했습니다. '논리적완결성'측면에서,문제의조건과무관한일반론을 근거로제시하고,'기압경도'와'기압'을혼동하여결론을도출했기 때문에근거가결론을전혀뒷받침하지못합니다.'추론의깊이'측면에서, 지균풍의핵심원리인코리올리힘과위도와의관계에대한이해없이 피상적인사실을나열하는데그쳤습니다.'표현의명확성'측면에서도, 존재하지않는용어와모호한표현('기압경로의변화')을사용하여의미 전달에심각한결함이있습니다. List\_of\_Factual\_Errors: [ It incorrectly referred to ' geostrophic wind ' as ' delayed wind ', a non-existent meteorological term., It disregarded the problem ' s key assumption that the ' pressure gradient is constant ' and used the relationship ' wind speed increases as pressure gradient increases ' as the basis for reasoning., It confused the distinct physical concepts of ' pressure gradient ' and ' pressure ', erroneously explaining the relationship between pressure and wind speed based on the relationship between pressure gradient and wind speed., ] Accuracy: 1, Logical\_Completeness: 1, Evaluation\_Scores: Depth\_of\_Reasoning: 1, Clarity\_of\_Expression: 1 Evaluation\_Reasoning: The candidate AI ' s response recorded the lowest scores across all evaluation axes. In terms of ' Accuracy ', it referred to ' geostrophic wind ' using the hallucinated term ' delayed wind ' and demonstrated a fatal conceptual error by confusing ' pressure gradient ' with ' pressure '. Furthermore, it distorted factual relations by ignoring the problem ' s core premise that the ' pressure gradient is constant '. In terms of ' Logical\_Completeness ', the reasoning failed to support the conclusion at all because it presented generalities irrelevant to the problem conditions and derived the conclusion by confusing ' pressure gradient ' with ' pressure '. In terms of ' Depth\_of\_Reasoning ', it merely listed superficial facts without understanding the relationship between the Coriolis force and latitude, which is the core principle of geostrophic wind. In terms of ' Clarity\_of\_Expression ', there were serious flaws in meaning delivery due to the use of non-existent terms and ambiguous expressions ( e.g., ' change in pressure path '). Reasoning Evaluation of Qwen3-VL-235B-A22BThinking ## Response from Qwen3-VL-235B-A22B-Thinking 생성된 \_ 추론 \_ 근거: 지균풍은 Vg = |p|/ ( ρ f )이므로 기압경도와밀도가일정하면코리올리매개변수 f=2Ω sinφ 만이변수가되어위도가증가할수록 f 가커져풍속은감소하며,절대기압의크기(보기 3 )나고도(보기 1 ), 점성(보기 4 )은이관계식에직 접등장하지않는다. Expert-Verified Rationale: Since the geostrophic wind is defined as Vg = |p|/ ( ρ f ), if the pressure gradient and density are constant, the Coriolis parameter f=2Ω sinφ becomes the only variable. As latitude increases, f increases, causing the wind speed to decrease. The magnitude of absolute pressure ( Option 3 ), altitude ( Option 1 ), and viscosity ( Option 4 ) do not appear directly in this equation. 정답: <details> <summary>Image 15 Details</summary>  ### Visual Description Icon/Small Image (22x20) </details> Predicted Answer: 2 ## Reasoning Evaluation from Human Experts: 사람전문가평균점수: 사실적정확성( 5 ), 논리적 완결성( 5 ), 추론의 깊이 ( 4.5 ), 표현의 명확성( 5 ) Average of Human Expert Score: factuality ( 5 ), logicality ( 5 ), depth ( 4.5 ), clarity ( 5 ) Comments: ## Reasoning Evaluation from LLM-Evaluator ( Gemini-2.5-Pro ): ## 사실 \_ 오류 \_ 목록: [ ] 평가 \_ 점수: 정확성: 5, 논리적 \_ 완결성: 5, 추론의 \_ 깊이: 5, 표현의 \_ 명확성: 5 평가 \_ 사유: 수험생 AI 의답변은모든평가축에서 매우우수한수준을보였습니다. '정확성'측면에 서, 제시된지균풍속도공식,코리올리계수와위 도의관계,그리고결론까지기상학적사실에완벽 하게부합하며어떠한오류도발견되지않았습니 다. '논리적 완결성' 측면에서, 문제의 조건(기압 경도,밀도일정)으로부터풍속이코리올리계수에 반비례한다는핵심관계를명확히하고,이를위도 와의관계로확장하여결론을도출하는과정이비 약없이완벽하게연결되었습니다. '추론의깊이' 측면에서,정답의근거를제시하는것에그치지않 고, 매력적인 오답 선택지( 3 번, 4 번)가 왜 오답인 지를지균풍의기본가정(마찰무시)과핵심변수 (기압경도)를 들어 명확히 반박함으로써 문제에 대한깊이있는이해를보여주었습니다. '표현의 명확성'측면에서도군더더기없이핵심내용을간 결하고명확하게전달하여이해도를높였습니다. 이는모범적인해설이라할수있습니다. List\_of\_Factual\_Errors: [ ] Evaluation\_Scores: cal\_Completeness: 5, Accuracy: 5, Logi- Depth\_of\_Reasoning: 5, Clarity\_of\_Expression: 5 Evaluation\_Reasoning: The candidate AI ' s response demonstrated a very high level of performance across all evaluation axes. In terms of ' Accuracy ', the presented geostrophic wind speed formula, the relationship between the Coriolis parameter and latitude, and the conclusion perfectly matched meteorological facts with no errors found. In terms of ' Logi-cal\_Completeness ', the process of clarifying the core relationship that wind speed is inversely proportional to the Coriolis parameter from the problem conditions ( constant pressure gradient and density ) and extending this to the relationship with latitude to derive the conclusion was perfectly connected without logical leaps. In terms of ' Depth\_of\_Reasoning ', it demonstrated a deep understanding of the problem not only by providing the basis for the correct answer but also by clearly refuting why the attractive distractors ( Options 3 and 4 ) were incorrect, citing the basic assumption of geostrophic wind ( ignoring friction ) and the key variable ( pressure gradient ). In terms of ' Clarity\_of\_Expression ', it delivered the core content concisely and clearly without redundancy, enhancing comprehensibility. This can be considered an exemplary explanation. ## A Dataset Examples Table 6 presents representative examples from KMetBench, organized into two primary modality groups: Text-only and Multimodal . Within each modality, we further stratify the tasks into three distinct categories to evaluate comprehensive meteorological capabilities. Text-only Tasks assesses linguistic reasoning and theoretical knowledge without visual interpretation. This group includes ( a ) Standard MCQA for fundamental concepts, ( b ) K-Specific MCQA which requires geo-cultural knowledge specific to the Korean Peninsula, and ( c ) Reasoning MCQA that demands multi-step logical deduction. Multimodal Tasks introduces visual data interpretation, a critical skill for meteorologists. This group parallels the text-only structure with ( d ) Standard , ( e ) K-Specific , and ( f ) Reasoning subsets, but specifically evaluates the model ' s ability to analyze weather charts, satellite imagery, and atmospheric diagrams. This structured categorization allows for a clear comparison of model performance across different modalities and levels of domain expertise. ## B Case Study of Reasoning Answer Two human experts and the LLM-Evaluator ( gemini-2.5-pro ) conducted evaluations using identical rubrics. As shown in Table 7, we observed consensus between the human and AI evaluators for the InternVL3.5-8B-Instruct and Qwen3-VL-235B-A22B-Thinking models: both correctly identified incorrect answers and highlighted inappropriate terminology in the reasoning rationale. Notably, the human expert made a specific error by misreading '기압경도'( pressure gradient ) as '기압경로'( pressure path ). While the quantitative scores assigned by the human expert and the LLM-Evaluator were comparable, the granularity of their feedback differed significantly. The human expert made an implicit judgment, providing summary comments alongside the score. In contrast, the LLM-Evaluator generated more detailed outputs, including explicit justifications and comprehensive lists of factual errors. ## C Prompts and Questionnaires for Benchmark Construction This section details the prompts utilized for data augmentation ( paraphrasing ) and the identification of domain-specific subsets. These processes were conducted to enhance the quality of the dataset and provide rich learning signals. ## C.1 Question Paraphrasing To diversify sentence structures and lexical expressions while preserving the original semantic meaning of the questions, we utilized the Gemini-2.5Pro model. Figure 5 presents the specific system prompt employed for this paraphrasing task. ## C.2 Identification of Korean-Specific Subset To identify questions containing Korean-specific geographical and cultural contexts ( the KoreanSpecific subset ) from the total pool of 1,774 questions, we established a hybrid pipeline combining LLM-based filtering with human verification. LLM-Aided Identification The screening process involved independent filtering using two distinct models: Gemini-2.5-Pro and GPT-4.1 . The identification prompts for each model were optimized through an iterative refinement process to maximize recall. Figures 12 and 14 illustrate the final enhanced prompts used for identifying Koreanspecific context questions, respectively. Human Selection Process Based on the LLM filtering, Gemini-2.5-Pro extracted 135 candidates, while GPT-4.1 extracted 95 candidates. We consolidated these results into a union of 149 unique questions. Subsequently, two human researchers performed cross-validation on this candidate set to finalize the Korean-Specific subset. The selected questions typically contain high-context keywords such as ' Our country ' (우리나라),' Korean Peninsula, ' ' Jeju, ' ' Seoul, ' ' Yeongdong, ' ' Southerly wind ' (마파람), ' Taebaek Mountains, ' and ' 24 Solar Terms. ' ## C.3 Implicit vs. Explicit Dataset Design for Korean-Specific Subset To ensure a fair evaluation of local context understanding regardless of the model ' s primary training language, we constructed a dual-version dataset by converting implicit questions into explicit ones. - Implicit Questions: These refer to the original items containing high-context expressions that presuppose the speaker ' s spatiotemporal and cultural location ( e.g., ' Our country, ' ' Maparam, ' ' East Coast '). - Explicit Questions: These refer to the modified items where human researchers manually replaced high-context references with objective and unambiguous terminology ( e.g., changing ' Our country ' to ' South Korea ' or ' Maparam ' to ' Southerly wind, a pure Korean term '). Table 8 presents comparative examples of these original implicit questions and their explicit counterparts. Table 8: Examples of Context Transformation from implicit to explicit forms | ID | Implicit ( Before ) | Explicit ( After ) | |-----------|-------------------------|-----------------------------------| | All | 우리나라 | 한국지역 | | 618 | 서울 | 한국의서울지역 | | 1037 | 24 절기 | 동아시아지역의 24 절기 | | 822 | 동해안 | 한국지역의동해안 | | 557 | 겨울철발해만에 서작은기압골이 접근하고있다. | 겨울철발해만으로부터소규모 기압골이한국지역으로접근하 는상황에서 | | 271, 1744 | 마파람 | 한국지역의지방풍인마파람 | ## C.4 Evaluation Prompts for Korean-Specific Subset This section details the construction of system prompts designed to evaluate the model ' s understanding of geo-cultural contexts. To encourage the model to effectively utilize its latent local knowledge, we designed an Advanced Prompt that explicitly defines the speaker ' s persona ( i.e., a Korean meteorology expert ) and clarifies that the questions are contextually situated in Korea. Figure 24 displays the advanced system prompt used to test the activation of geo-cultural knowledge. ## C.5 Prompt for Reference Rationale Generation To secure high-quality reasoning references ( rationales ) for the benchmark, we utilized the GPT-5 model. The prompt engineering process employed an iterative refinement technique. Specifically, we established a loop where an Enhancer model drafted the initial prompt and a Critic model identified weaknesses for revision, using GPT-5 for both roles to derive the optimal instruction. The final system prompt used for rationale generation is presented in Figure 16. To ensure comprehensive coverage, the target questions were selected via stratified sampling to include all subject areas, modalities ( text-only/multimodal ), and Korean-specific items. Furthermore, to guarantee the validity of the reasoning paths, we enforced a strict filtering protocol: if the model generated an incorrect answer, the generation process was repeated until a rationale leading to the correct answer was produced. ## C.6 Questionnaire for Expert Verification on Reference Rationale To quantify the prompting gain-the extent to which this contextual cuing aids performanceand to ensure equitable evaluation for non-Korean models, we also established a Standard Prompt as a control group. Figure 22 presents the standard system prompt used for the baseline experiment, while To ensure the reliability of the LLM-as-a-judge pipeline, two human experts conducted a rigorous verification of the generated rationales from September 9 to October 19, 2025. This process was critical for establishing the integrity of the reference data. Before the verification process, the experts were given written instructions describing the purpose of the study, the expected completion time, and how their judgments would be used in the research. They were asked to assess each generated rationale in terms of factual accuracy, logical soundness, completeness, and conciseness, and to mark whether the rationale should be adopted as is ## System Prompt for Question Paraphrasing ( Gemini-2.5-Pro ) Task: Paraphrase the following multiple-choice question about meteorology in Korean. Rules: - Preserve the core meaning and all technical terminology. - Change the sentence structure or phrasing for a more natural flow. - Output Format: Provide only the final paraphrased text. Do not include any introductory phrases or explanations. - Original Question: '{ original\_question } ' Figure 5: System prompt used to paraphrase Korean meteorological questions or revised. When revisions were needed, they were instructed to provide either minor-fix or major-fix notes. An example of the questionnaire used for this process is shown in Table 11. Out of 142 rationales initially generated by GPT-5 , experts provided feedback for revision on 19 cases ( 13.38% ). The revisions primarily addressed technical accuracy and clarity. Specifically, experts corrected erroneous terminology ( 5 cases ), such as changing '비열용량'( specific heat capacity ) to '비열' ( specific heat ) or '지표소용 돌이도' to '행성소용돌이도' ( planetary vorticity ). They also reinforced variable explanations and standard units ( 3 cases ); for instance, refining the phrasing ' among the temperatures handled' to ' among the variables handled in atmospheric science' because '혼합비'( mixing ratio ) is not a temperature variable. Additionally, the revisions included full sentence rewriting ( 7 cases ), supplementary explanations ( 2 cases ), and minor stylistic polishing ( 2 cases ) to align with standard Korean meteorological conventions ( e.g., standardizing '포텐셜온도' to '온위', '바트로픽' to '순압', and '단열가열' to '단열압축'). In addition to refining the AI-generated rationales, this expert review also identified inherent defects in the raw exam data. One question ( ID 276 ) was discarded from the dataset as it was deemed logically unsolvable. Furthermore, questions with syntactic errors ( IDs 308, 650 ) or issues with option configuration/double answers ( IDs 14, 583, 1665 ) were precisely corrected based on expert consultation. Through this process, we secured the integrity of the final 141 reasoning evaluation samples. Table 11 presents the specific questionnaire used for this expert verification process. ## C.7 Reasoning Prompt for Open-Source LLMs Figure 18 presents the system prompt utilized for generating reasoning paths and answers from opensource LLMs. It is important to note that this prompt serves as the standard instruction for the main inference phase of our benchmark evaluation protocol, rather than an experimental variation. ## C.8 Prompt for LLM-as-a-Judge Evaluation Figure 20 illustrates the specific system prompt employed for the LLM-as-a-Judge evaluation pipeline. The prompt was meticulously designed with the following key considerations to ensure robust alignment with human expert evaluation: - Unified Evaluation Scale: Weadopted the identical 1-to-5 Likert scale and four evaluation axes -Factual Accuracy, Logical Soundness, Depth of Reasoning, and Clarity & Conciseness -used by human experts. This unification allows for direct statistical comparison and correlation analysis between LLM and expert scores. - Enforced Chain of Thought ( CoT ): To enhance consistency, the prompt explicitly mandates a step-by-step thinking process. The evaluator is required to verify facts against the provided expert reference material before assigning scores, thereby minimizing hallucinations and ensuring evidence-based grading. - Explicit Scoring Criteria: To prevent arbitrary scoring, we defined concrete rubrics for specific score tiers ( e.g., distinguishing between a 5-point perfect answer and a 3-point answer with minor errors ). - Structured Output: The prompt enforces a strict JSON output format that separates the ' List of Factual Errors ' from the quantitative scores. This structural constraint compels the model to explicitly isolate factual hallucinations from qualitative reasoning flaws. ## C.9 Questionnaire for Expert Scoring of LLMReasoning Table 12 outlines the questionnaire and scoring rubric provided to two meteorology professors. Crucially, this rubric served as the blueprint for the LLM-as-a-Judge prompt described above, ensuring that both human and AI evaluators operated under identical standards regarding accuracy and reasoning quality. The human experts were also provided with written scoring instructions that described the study purpose, the expected annotation time, and the four evaluation axes: factual accuracy, logical soundness, depth of reasoning, and clarity. To reduce bias, model identities were blinded in the scoring materials, and the experts were instructed to judge only the content of the generated reasoning against the expert-verified reference rationale. An example of the scoring questionnaire is provided in Table 12. ## D Experimental Setups ## D.1 Reasoning Model Inference We activate the thinking mode for hybrid models by setting enable\_thinking = True ( Qwen3- - *B , EXAONE-4.0- * ), reasoning\_effort = 'high' ( gpt-5.2 ), and thinkingLevel = 'high' ( gemini-3-pro-preview ). In contrast, InternVL3.5- *- Instruct is evaluated in standard instruct mode. ## D.2 Meta-Evaluation for LLM-as-a-Judge To validate the reliability of the LLM evaluator, we designed a meta-evaluation protocol consisting of three steps: 1 ) generating reasoning paths and answers using various open-source LLMs; 2 ) performing LLM-as-a-Judge evaluation using expertverified references and a specific rubric ( based on a 5-point Likert scale across four evaluation axes ); and 3 ) obtaining scores from two human experts using the identical rubric to calculate the statistical correlation between the LLM judge and human experts. The detailed prompt for the main inference, the judge prompt, and the expert scoring questionnaire are provided in Appendix C.7, C.8, and C.9, respectively. The full questionnaires and written instructions provided to the human experts are included in Appendix C.6 and Appendix C.9 ( Figure 10 and Figure 11 ). Sampling Strategy of Target Models To ensure that the LLM-as-a-Judge can reliably evaluate reasoning capabilities across a broad spectrum of proficiency, we employed a performance-based stratified sampling strategy. We categorized the pool of candidate models into three distinct tiers-Top, Mid, and Low-based on their normalized reasoning scores on the 141 reasoning questions. From these strata, we selected representative models to form a final set of 10 target models for the metaevaluation, ensuring that the judge is tested against both high-quality coherent reasoning and lowerquality outputs. The list of sampled models is detailed in Table 9. Stratified Sampling of Evaluation Items To establish a robust gold standard for scoring, we selected 10 representative reasoning questions. Instead of random selection, we applied a stratified sampling to ensure both comprehensiveness and discriminatory power. The selection process involved the following criteria: - Item Difficulty: Weclassified the 141 reasoning questions into three difficulty tiers based on the average normalized reasoning scores of 10 opensource LLMs: Hard ( Top 30% ), Mid ( 40% ), and Easy ( Bottom 30% ). We sampled 3, 4, and 3 Table 9: List of Sampled Models for MetaEvaluation of LLM-as-a-Judge. Models were selected via stratified sampling based on their normalized reasoning score tiers to ensure diverse evaluation targets. Reas. denotes the normalized reasoning score on the Reasoning subset questions. | Tier | Model Name | Family | Size ( B ) | Reas. | |--------|-----------------------------|-----------|--------------|---------| | | Qwen3-VL-235B-A22B-Thinking | Qwen | 235 | 4.31 | | Top | gpt-oss-120b | OpenAI | 120 | 4.03 | | | A.X-4.0 | SKT | 72 | 3.87 | | Mid | command-a-reasoning-08-2025 | Cohere | 111 | 3.53 | | | gpt-oss-20b | OpenAI | 20 | 3.39 | | | VARCO-VISION-2.0-14B | NCSoft | 14 | 2.81 | | | InternVL3.5-14B-Instruct | OpenGVLab | 15 | 2.36 | | | Llama-3.1-8B-Instruct | Meta | 8 | 1.91 | | Low | InternVL3.5-8B-Instruct | OpenGVLab | 8 | 1.77 | | | Qwen3-0.6B | Qwen | 0.6 | 1.15 | questions from each respective group to balance the difficulty distribution. - Discriminatory Power: Within each difficulty tier, we prioritized questions with a high standard deviation in accuracy across the 10 models. Ahigh standard deviation indicates that the question effectively discriminates between high- and low-performing models. - Category Coverage: The selection was further constrained to ensure a balanced inclusion of text-only, multimodal, and Korean-specific questions, as well as coverage across the official exam subject areas ( Parts 1, 3, 4, and 5 ). Based on these criteria, the final 10 questions selected for meta-evaluation are: IDs 105, 1618, 1590, 14, 456, 1694, 963, 1745, 131, and 1224. Table 10 details the characteristics of these sampled items. ## E Additional Results and Discussion ## E.1 Detailed Orthogonality Analysis of K-MetBench The Uniqueness of Visual Reasoning. As illustrated in Figure 7, the Multimodal subset of KMetBench displays consistently low correlations ( avg. τ b < 0 . 30 ) 1across all external benchmarks, including text-based baselines ( KMMLUPro, KMMLU-Redux, ClimaQA ) and weatherdomain vision benchmarks ( ClimaIQA, WeatherQA ). This disconnect quantitatively confirms the modality gap, demonstrating that the ability to in- Table 10: Statistics of Selected Evaluation Items. Mean and Std. Dev. represent the item-wise normalized reasoning scores ( 1-5 ) across the selected models. | Tier | ID | Mean | Std. Dev. | Part | Note | |--------|------|--------|-------------|--------|--------| | Hard | 105 | 2.23 | 1.72 | 3 | - | | Hard | 1618 | 2.35 | 1.65 | 5 | - | | Hard | 1590 | 2.55 | 1.91 | 3 | - | | Mid | 14 | 2.7 | 1.89 | 1 | - | | Mid | 456 | 2.73 | 1.88 | 3 | - | | Mid | 1694 | 3.1 | 1.93 | 5 | - | | Mid | 963 | 3.13 | 1.81 | 4 | Korean | | Easy | 1745 | 3.35 | 1.89 | 4 | - | | Easy | 131 | 3.4 | 1.74 | 5 | - | | Easy | 1224 | 3.53 | 1.76 | 4 | - | terpret meteorological charts and symbols is a distinct skill set not linearly correlated with general linguistic or textual reasoning capabilities. To investigate the orthogonality of our benchmark, we further analyzed correlations with KMMLU-Pro and -Redux. For KMMLU-Redux, where only the test set is publicly available, we specifically partitioned the data into the 39 questions derived from the 2022 Meteorological Engineer exam versus the remaining 2,547 general questions. The sample pool for this analysis consisted of 25 open-source VLLMs for Multimodal subset comparisons and 52 open-source models for other subsets ( excluding the proprietary models listed in Table 4 ). As shown in Figure 6, KMMLU-Pro exhibited weaker correlation due to domain divergence. Within KMMLU-Redux, the isolated 39-question meteorological subset showed lower correlation ( τ b = 0 . 70 ) than the full dataset ( τ b = 0 . 78 ), suggesting that this small subset is insufficient to capture comprehensive meteorological capability. Crucially, a significant drop in correlation is observed for K-MetBench ' s multimodal and reasoning subsets, highlighting the structural gap between our multimodal evaluation and existing textonly licensing exams. ## E.2 Detailed Analysis of K-MetBench Performance Table 4 presents the comprehensive leaderboard of K-MetBench, evaluating a diverse range of proprietary and open-source models. The results are categorized by model type, capabilities ( Korean-native, Multimodal, Reasoning ), and granular domain per- Figure 6: Heatmap of Kendall ' s τ b rank correlations between K-MetBench and KMMLU, KMMLURedux ). In KMMLU-Redux, ' 39 ' denotes the subset of 39 Meteorological Engineer Exam questions, while ' All -39 ' refers to the remaining subset excluding these meteorological questions. Acc.: Accuracy. <details> <summary>Image 16 Details</summary>  ### Visual Description ## Heatmap: Kendall's Correlation Coefficient Across Benchmarks ### Overview This heatmap visualizes Kendall's correlation coefficients between various benchmarks (rows) and comparison benchmarks (columns). The color gradient ranges from blue (-1.0) to red (1.0), indicating negative to positive correlation strength. Numerical values are embedded in each cell. ### Components/Axes - **X-axis (Columns)**: - `KMMMLU Redux` (subcategories: `All`, `39`, `All-39`) - **Y-axis (Rows)**: - `Total`, `Reasoning`, `Part 1-5`, `Text-Only`, `Multimodal`, `Korean` - **Legend**: - Color scale from blue (-1.0) to red (1.0), labeled "Kendall's Correlation Coefficient" - **Title**: - "Kendall's Correlation Coefficient" ### Detailed Analysis - **K-MetBench**: - Values: `0.56` (All), `0.78` (39), `0.70` (All-39) - Trend: Strong positive correlation across all subcategories. - **Reasoning**: - Values: `0.61` (All), `0.66` (39), `0.65` (All-39) - Trend: Moderate positive correlation, slightly lower than K-MetBench. - **Part 1-5**: - Values: `0.60-0.79` (All), `0.65-0.70` (39), `0.66-0.78` (All-39) - Trend: Consistently high correlations, with Part 4 showing the highest (`0.79`). - **Text-Only**: - Values: `0.56` (All), `0.78` (39), `0.71` (All-39) - Trend: Strong correlation, particularly in `39` and `All-39`. - **Multimodal**: - Values: `0.20` (All), `0.29` (39), `0.34` (All-39) - Trend: Weak positive correlation, significantly lower than other benchmarks. - **Korean**: - Values: `0.57` (All), `0.70` (39), `0.61` (All-39) - Trend: Moderate correlation, with a dip in `All-39`. ### Key Observations 1. **High Correlation**: - `K-MetBench`, `Part 4`, and `Text-Only` show the strongest correlations (`≥0.70`), suggesting alignment with comparison benchmarks. 2. **Low Correlation**: - `Multimodal` has the weakest correlation (`0.20-0.34`), indicating divergence from comparison benchmarks. 3. **Subcategory Variance**: - `All-39` generally shows lower correlations than `All` or `39` (e.g., `Korean` drops from `0.70` to `0.61`). 4. **Color Consistency**: - All values align with the legend (e.g., `0.79` is dark red, `0.20` is light red). ### Interpretation - **Benchmark Effectiveness**: - High correlations for `K-MetBench` and its components suggest they are robust proxies for the comparison benchmarks. - **Multimodal Anomaly**: - The low correlation for `Multimodal` may reflect challenges in integrating multiple modalities (e.g., text, images) compared to text-only benchmarks. - **Language-Specific Trends**: - The `Korean` row shows mixed performance, potentially due to language-specific nuances or dataset biases. - **Subset Impact**: - The `All-39` column reveals reduced correlations across most benchmarks, implying that the `39` subset may not fully capture the broader benchmark characteristics. This heatmap highlights the importance of benchmark design in capturing domain-specific traits, with `Multimodal` and `Korean` requiring further investigation for alignment with standard benchmarks. </details> ## formance. SOTA Performance and the Impact of Reasoning. Proprietary models dominate the upper echelon of the leaderboard. gemini-3-propreview ( Thinking ) achieves state-of-the-art performance with a total accuracy of 93.7%, significantly outperforming other contenders. A notable trend is the efficacy of Thinking ( reasoning ) models; for instance, gpt-5.2 ( Thinking ) scores 87.8%, showing a substantial improvement ( +10.2%p ) over its standard counterpart, gpt-5.2 ( 77.6% ). This pattern reinforces that chain-ofthought capabilities are crucial for solving complex meteorological problems. Open-Source Landscape. In the open-source domain, the Qwen ( Yang et al. , 2025 ; Bai et al. , 2025 ; Qwen et al. , 2024 ) series exhibits exceptional performance. Qwen3-VL-235B-A22BThinking leads this category with 84.4%. Even smaller models like Qwen3-VL-32B-Thinking ( 78.6% ) surpass much larger non-reasoning models ( e.g., gpt-oss-120b , 77.3% ), highlighting the efficiency of reasoning-enhanced architectures in specialized scientific domains. The Modality Gap. A critical disparity exists between textual and visual reasoning. While top models achieve near-perfect scores on the Text subset ( e.g., Gemini: 94.6% ), their performance drops significantly on the Multimodal subset ( Gemini: 75.6% ). This modality gap is even more pronounced in other models; gpt-5.2 (Thinking) sees a drastic decline from 90.6% ( Text ) to 29.3% ( Multi ). This indicates that while current LLMs excel at theoretical knowledge retrieval, they still struggle with interpreting professional meteorological charts and diagrams. ## Geo-Cultural Alignment and Granularity. Korean-native models demonstrate distinct advantages in localized contexts. A.X-4.0 achieves a high K-Specific score of 78.9%, outperforming several larger global models in this specific subset, despite a lower overall accuracy. In terms of domain granularity ( P1-P5 ), models generally perform best in Meteorological Observation ( P2 ), likely due to the descriptive nature of the questions, while struggling more in Atmospheric Dynamics ( P3 ) and Atmospheric Physics ( P5 ), which require deeper calculation and physical conceptualization. ## E.3 Results of Meta Evaluation of LLM-as-a-Judge Rank Preservation Analysis. In benchmark evaluation, the accuracy of relative ranking is often more critical than absolute scores. The slope graph in Figure 8 compares the rankings assigned by human experts and the LLM. Although minor rank fluctuations exist, the overall trend distinguishing high-performing models from low-performing ones is preserved. Figure 8: Slope graph of rank changes of reasoning evaluation scores of two human experts vs. LLM evaluator. The crossing lines indicate minor discrepancies, but the overall performance tiers remain largely consistent. <details> <summary>Image 17 Details</summary>  ### Visual Description ## Diagram: Human vs. LLM Model Rankings ### Overview The diagram compares human and LLM rankings of 10 AI models, showing their positions in both evaluation systems. Models are connected by lines to illustrate alignment or discrepancies between human and LLM assessments. ### Components/Axes - **Human Rank**: Left column listing models with their human-assigned rankings (#1 to #10). - **LLM Rank**: Right column listing the same models with their LLM-assigned rankings (#1 to #10). - **Model Names**: Full technical identifiers for each model (e.g., "Qwen3-VL-235B-A22B-Thinking", "gpt-oss-120b"). - **Lines**: Connect identical models across both columns to visualize ranking consistency. ### Detailed Analysis 1. **Qwen3-VL-235B-A22B-Thinking** (#1 in both rankings) - Top-ranked by both humans and LLMs. 2. **A.X-4.0** (#2 in both rankings) - Consistently second in both systems. 3. **gpt-oss-120b** (#3 in both rankings) - Maintains third place across evaluations. 4. **command-a-reasoning-08-2025** (#4 in Human Rank, #5 in LLM Rank) - Slight drop in LLM ranking compared to human assessment. 5. **gpt-oss-20b** (#5 in both rankings) - Stable performance in both systems. 6. **VARCO-VISION-2.0-14B** (#6 in both rankings) - Consistent sixth-place ranking. 7. **Llama-3.1-8B-Instruct** (#7 in Human Rank, #7 in LLM Rank) - Shared seventh position with InternVL3_5-14B-Instruct. 8. **InternVL3_5-14B-Instruct** (#8 in Human Rank, #7 in LLM Rank) - Improved LLM ranking compared to human assessment. 9. **InternVL3_5-8B-Instruct** (#9 in both rankings) - Consistent ninth-place performance. 10. **Qwen3-0.6B** (#10 in both rankings) - Bottom-ranked by both systems. ### Key Observations - **Alignment**: 7/10 models show identical rankings in both systems (e.g., Qwen3-VL-235B-A22B-Thinking, A.X-4.0). - **Discrepancies**: - **command-a-reasoning-08-2025** drops from #4 (Human) to #5 (LLM). - **InternVL3_5-14B-Instruct** improves from #8 (Human) to #7 (LLM). - **Ties**: Llama-3.1-8B-Instruct and InternVL3_5-14B-Instruct share #7 in both rankings. ### Interpretation The diagram reveals strong consensus between human and LLM evaluations for top-performing models (positions #1–#6), suggesting alignment in assessing core capabilities. However, mid-tier models (positions #7–#10) show notable divergence, indicating potential differences in how humans and LLMs prioritize specific features (e.g., reasoning speed vs. accuracy). The slight drop for command-a-reasoning-08-2025 in LLM rankings might reflect architectural limitations in handling complex queries, while InternVL3_5-14B-Instruct's improvement suggests better optimization for LLM-specific evaluation criteria. This highlights the importance of cross-validation between human and automated assessments in model development. </details> ## E.4 MCQAAccuracy vs. Reasoning Score As illustrated in Figure 9, we analyze the relationship between answer accuracy and qualitative reasoning capabilities. The color gradient represents the Reasoning Score Gap , defined as the disparity between the reasoning score of correctly answered items and the overall average ( i.e., Reasoning Score Gap = Reasoning Score | A = correct -Reasoning Score total ). We observe a strong positive correlation ( r = 0 . 959 ) between QA accuracy and reasoning scores, Figure 7: Heatmap of Kendall ' s τ b rank correlations between K-MetBench and existing benchmarks. Red denotes high positive correlation, while blue indicates negative correlation. <details> <summary>Image 18 Details</summary>  ### Visual Description ## Heatmap: Correlation Coefficients Between Model Metrics ### Overview This heatmap visualizes correlation coefficients between various metrics across different AI models, ranging from -1.00 (perfect negative correlation) to 1.00 (perfect positive correlation). The data reveals strong internal consistency within model-specific metrics, with notable variations in cross-model relationships. ### Components/Axes - **X-axis (Models/Subcategories)**: - K-MetBench Total Acc - K-MetBench Reasoning Score - K-MetBench Part 1-5 Acc - K-MetBench Text-Only Acc - K-MetBench Multimodal Acc - K-MetBench Korean Acc - KMMMLU-Pro Acc - KMMMLU-Redux Acc - ClimaQA Acc - ClimaQA Binary F1 - ClimaQA Regional Identification - ClimaQA Regional Localization - ClimaQA Descriptive Generation - WeatherQA Area Acc - WeatherQA Concern Acc - **Y-axis (Metrics)**: - Identical to X-axis labels, representing correlation targets - **Color Legend**: - Blue (-1.00) to Red (1.00) gradient - Key thresholds: - 0.75-1.00 (dark red) - 0.50-0.75 (medium red) - 0.25-0.50 (light red) - -0.25-0.25 (white) - -0.50-0.25 (light blue) - -0.75-1.00 (dark blue) ### Detailed Analysis 1. **Diagonal Dominance**: - All diagonal cells show 1.00 (dark red), confirming perfect self-correlation - Example: K-MetBench Total Acc vs itself = 1.00 2. **K-MetBench Internal Consistency**: - Part 1-5 Acc show 0.84-0.93 correlation (medium-red) - Text-Only Acc (0.93) and Multimodal Acc (1.00) demonstrate strong relationships 3. **KMMMLU-Redux Anomalies**: - -0.18 correlation with K-MetBench Part 1 Acc (light blue) - 0.53 correlation with ClimaQA Binary F1 (medium-red) 4. **ClimaQA Performance**: - Binary F1 shows 0.30-0.53 correlations with other metrics - Regional Localization (-0.33 to 0.53) exhibits mixed performance 5. **WeatherQA Metrics**: - Area Acc shows -0.27 to 0.35 correlations - Concern Acc demonstrates -0.18 to 0.11 correlations ### Key Observations 1. **Model-Specific Consistency**: - K-MetBench and KMMMLU-Pro show >0.8 correlation between related metrics - KMMMLU-Redux exhibits unique negative correlations (-0.18 to -0.33) 2. **Cross-Model Variability**: - ClimaQA metrics show lowest consistency (0.10-0.53 range) - WeatherQA metrics demonstrate moderate variability (-0.27 to 0.35) 3. **Notable Outliers**: - KMMMLU-Redux vs K-MetBench Part 1 Acc: -0.18 (unusual negative correlation) - ClimaQA Regional Localization vs itself: 0.43 (lower than other self-correlations) ### Interpretation The heatmap reveals that: 1. **Model Architecture Impacts Correlation**: - K-MetBench's high internal consistency suggests cohesive design - KMMMLU-Redux's negative correlations may indicate trade-offs between accuracy and efficiency 2. **Task Complexity**: - ClimaQA's lower correlations suggest diverse evaluation criteria - WeatherQA's mixed performance indicates varying task difficulty 3. **Cross-Model Insights**: - KMMMLU-Redux shows unexpected relationships with ClimaQA metrics - WeatherQA metrics have minimal correlation with other models The data suggests that while models maintain strong internal consistency, cross-model performance varies significantly, highlighting the importance of task-specific evaluation frameworks. The negative correlations in KMMMLU-Redux warrant further investigation into potential architectural trade-offs. </details> indicating that models that derive correct answers also tend to generate higher-quality reasoning traces. A distinct scaling law is also evident; larger models ( shown by marker size ) consistently populate the upper-right quadrant, achieving superior performance in both metrics. Two notable trends appear among specific models: High Reasoning but Low Accuracy: Qwen3VL-8B-Thinking emerges as an outlier. Despite its relatively low accuracy, it maintains a high reasoning score. This suggests that while the model generates detailed 'thinking' processes, its limited capacity ( 8B ) often leads to hallucinations or logical fallacies in the final deduction. Impact of Reasoning Optimization: The benefit of reasoning-specific training is highlighted by the Command family. command-areasoning-08-2025 significantly outperforms its predecessor, c4ai-command-a-03-2025 , in both accuracy and reasoning quality, validating the efficacy of reasoning-enhanced fine-tuning. Figure 9: Scatter plot of MCQA Accuracy vs. Reasoning Score. The x-axis represents the answer accuracy, while the y-axis denotes the qualitative reasoning score evaluated by the judge. Marker sizes are proportional to the model parameter count. The strong correlation ( r = 0 . 959 ) confirms that high-performing models generally provide more reliable reasoning traces. <details> <summary>Image 19 Details</summary>  ### Visual Description ## Scatter Plot: Model Performance vs. Reasoning Ability ### Overview This scatter plot visualizes the relationship between model performance on a reasoning subset (x-axis) and average reasoning scores (y-axis) across various AI models. A dashed regression line with a shaded confidence interval highlights the trend, while data points are color-coded by reasoning score gap and sized by model capacity. ### Components/Axes - **X-axis**: Accuracy on Reasoning Subset (0.0 – 1.0) - **Y-axis**: Average Reasoning Score (4–20) - **Legend**: - Model size: - Small (0.6B): Smallest circles - Medium (20B): Medium circles - Large (235B): Largest circles - Reasoning Score Gap: Gradient from red (7) to blue (1) - **Regression Line**: Dashed line with shaded 95% confidence interval - **Annotations**: Model names positioned near data points ### Detailed Analysis - **Regression Line**: - Slope: Strong positive correlation (r = 0.959) - Confidence Interval: Shaded gray area around the line - **Data Points**: - **Qwen3-VL-235B-A228-Thinking**: (0.85, 17.5) – Largest blue circle (low reasoning gap) - **Qwen3-VL-235B-A228-Instruct**: (0.82, 16.8) – Large blue circle - **Qwen3-VL-30B-A38-Instruct**: (0.78, 15.2) – Medium blue circle - **InternVL3_5-38B-Instruct**: (0.72, 12.0) – Medium blue circle - **EXAONE-4.0-1.2B**: (0.55, 8.0) – Small blue circle - **VARCO-VISION-2.0-1.7B**: (0.35, 5.8) – Smallest blue circle - **Qwen3-0.6B**: (0.3, 4.0) – Smallest blue circle - **Color Gradient**: - Red (7): High reasoning score gap (e.g., "Qwen3-VL-8B-Thinking") - Blue (1): Low reasoning score gap (e.g., "Qwen3-VL-235B-A228-Thinking") ### Key Observations 1. **Strong Correlation**: The near-perfect linear relationship (r = 0.959, p < 0.001) indicates that higher reasoning subset accuracy strongly predicts higher average reasoning scores. 2. **Model Size Trends**: - Larger models (235B) cluster in the top-right, achieving both high accuracy and reasoning scores. - Smaller models (0.6B–20B) show lower performance but still follow the trend. 3. **Reasoning Score Gap**: - Models with smaller gaps (blue) like "Qwen3-VL-235B-A228-Thinking" align closely with the regression line. - Models with larger gaps (red) like "Qwen3-VL-8B-Thinking" deviate more from the trend. 4. **Outliers**: - "Llama-3.1-8B-instruct" (0.45, 7.8) falls below the regression line despite moderate accuracy. - "VARCO-VISION-2.0-1.7B" (0.35, 5.8) is the lowest-performing model. ### Interpretation The data demonstrates that model scale and reasoning subset accuracy are tightly linked to overall reasoning capability. The reasoning score gap (color gradient) suggests that some models (e.g., "Qwen3-VL-235B-A228-Thinking") achieve high performance with minimal disparity between reasoning and general accuracy, while others (e.g., "Qwen3-VL-8B-Thinking") exhibit significant gaps. The shaded confidence interval confirms the robustness of the trend, though outliers like "Llama-3.1-8B-instruct" highlight potential inefficiencies in smaller models. This analysis underscores the importance of model architecture and training data quality in optimizing reasoning tasks. </details> ## E.5 Computational Cost and Efficiency Analysis We evaluated the normalized total GPU compute time for 100 questions against the 150-minute exam limit ( ≈ 2.50 GPU-hours ). Standard instruction-tuned models ( e.g., Qwen2.5-VL-Instruct ) demonstrated negligi- ble cost ( < 0 . 01 GPU-hours ), operating orders of magnitude faster than the human time constraint. Reasoning models exhibited significant computational overhead. While Qwen3-VL-8BThinking ( 2.4 GPU-hours ) remained within the limit, larger models like Qwen3-VL-32BThinking ( 3.8 GPU-hours ) and command-areasoning ( 20.8 GPU-hours ) exceeded the threshold, highlighting the substantial resource trade-off required for deep reasoning. While this heavy computational overhead may yield deeper reasoning traces, it poses challenges for time-sensitive forecasting applications where rapid decision-making is critical. However, employing tensor parallelism can effectively reduce wall-clock inference time. ## F Compute Resources We evaluated all open-source models on an internal cluster equipped with 40 NVIDIA H100 80GB PCIe GPUs. To maximize inference efficiency, we utilized the vLLM library for all benchmark evaluations. The evaluation covered 52 text-only and multimodal models across all subsets of KMetBench, totaling approximately 192.14 H100 GPU hours ( 153.01 and 39.13 GPU hours for standard MCQA and reasoning MCQA, respectively ). ## G Hierarchical Topic Distribution Figures 29 through 31 illustrate the comprehensive hierarchical taxonomy of the K-MetBench dataset, aligned with the official evaluation criteria of the National Meteorological Engineer written examination ( Human Resources Development Service of Korea , 2024 ) . The dataset spans five major subject areas: Weather Analysis and Forecasting Theory , Meteorological Observation Methods , Atmospheric Dynamics , Climatology , and Atmospheric Physics . As depicted in Figure 29-31, each subject area and hierarchy demonstrates the benchmark ' s finegrained granularity and comprehensive coverage of meteorological domain knowledge. The numerical values in parentheses represent the estimated count of questions belonging to each specific category. To map the 1,774 questions to this detailed hierarchy, we employed Gemini-2.5-Pro for automated classification. These counts serve as an indicative reference, highlighting the dataset ' s balanced coverage across the theoretical and practical spectrums of meteorology. Table 11: Expert verification questionnaire for reference rationales | ID | Question | Choices | Exact Answer | Generated Rationale | Adopt? | Note 1 ( Minor Fix ) | Note 2 ( Major Fix ) | |------|------------|-----------|----------------|-----------------------|----------|------------------------|------------------------| | 18 | 비열의... | 1. ... | 2 | 비열은... | yes | - | - | Note: The gray-shaded cells indicate the items to be answered by the expert. Adoption Criteria ( Accuracy, Logical Soundness, Completeness, and Conciseness ) are provided separately. <details> <summary>Image 20 Details</summary>  ### Visual Description ## Korean Text Document Analysis: Exam/Assessment Content ### Overview The image shows a Korean document containing text, a table with structured data, and highlighted sections. The content appears to be related to an exam or assessment for "LLM" (Large Language Model) standards, with sections on evaluation criteria, questions, answers, and responses. ### Components/Axes 1. **Header**: - Text: "LLM 벤치마크 주된 평가 정답지(gold standard) 구축을 위한 전문가 설문" - Translation: "LLM Benchmark Main Evaluation Answer Key (Gold Standard) Construction Expert Survey" 2. **Yellow Highlighted Section**: - Text: "채택 여부: LLM이 생성한 주문이 가이드라인 기준을 완벽하게 만족하는 경우, 통과: X, 미통과: X로 응답을 표기합니다." - Translation: "Adoption Decision: If LLM-generated requests fully meet guideline standards, mark 'X' for 'Pass' and 'X' for 'Fail' in responses." 3. **Table Structure**: - **Columns**: - **기상기사 질문** (Weather Reporter Questions): - Example: "캠벨스톡스 기록계(Campbell-Stokes recorder)에 대한 설명으로 옳지 않은 것은?" (Which description about Campbell-Stokes recorder is incorrect?) - **선택지** (Options): - Example: "1. 맞춤형에 맞는 자기지는 11월부터 다음에 2월 사이에 사용된다." (1. Customized items are used between February and November.) - **시험정답** (Exam Answers): - Example: "3. 전천 일사량을 기록하는 기기이다." (3. It is a device for recording total solar radiation.) - **LLM이 생성한 주문 근거** (LLM-Generated Order Rationale): - Example: "1. 저기압성 경도동 > 고기압성 경도동 > 지균풍" (1. Low-pressure geostrophic wind > High-pressure geostrophic wind > Trade wind) - **채택 여부** (Adoption Decision): - Example: "X" (Marked for evaluation) - **비고 1** (Note 1): - Example: "X" (Marked for evaluation) - **비고 2** (Note 2): - Example: "X" (Marked for evaluation) 4. **Highlighted Sections**: - **평가 대상** (Evaluation Target): - Text: "LLM이 생성한 주문 근거" (LLM-Generated Order Rationale) - **응답한** (Responded): - Text: "채택 여부: X" (Adoption Decision: X) ### Detailed Analysis - **Header**: The document is titled as a survey for constructing an LLM benchmark evaluation standard. - **Yellow Highlight**: Indicates the adoption decision criteria for LLM-generated responses. - **Table**: - **Questions**: Focus on weather-related topics (e.g., Campbell-Stokes recorder, geostrophic wind). - **Options**: Provide multiple-choice answers. - **Exam Answers**: Correct answers are listed (e.g., option 3 for the first question). - **LLM-Generated Rationale**: Shows how LLM would justify its responses (e.g., geostrophic wind hierarchy). - **Adoption Decision**: Marks responses for evaluation (e.g., "X" indicates evaluation). - **Notes**: Additional comments (e.g., "X" for evaluation). ### Key Observations - The document is structured as an assessment for evaluating LLM performance against predefined guidelines. - The "평가 대상" (Evaluation Target) column highlights the LLM's generated rationale for responses. - The "응답한" column marks which responses were evaluated (e.g., "X" indicates evaluation). - The "비고" (Notes) column includes additional annotations (e.g., "X" for evaluation). ### Interpretation - The document appears to be part of a quality control process for LLM-generated content, ensuring adherence to specific standards. - The table serves as a rubric for evaluating LLM responses against expert-curated answers. - The highlighted sections emphasize the importance of adoption decisions and evaluation criteria. - The use of "X" marks suggests a binary evaluation system (pass/fail) for LLM-generated responses. ## Conclusion This document outlines a structured approach to assessing LLM performance in generating weather-related content. It combines textual analysis with tabular data to evaluate adherence to guidelines, with clear markers for evaluation and decision-making. </details> ( a ) Text-only Reasoning Questions <details> <summary>Image 21 Details</summary>  ### Visual Description ## Document: LLM Benchmark Standard Analysis ### Overview The image is a technical document in Korean detailing a "gold standard" framework for evaluating Large Language Models (LLMs). It includes textual explanations, diagrams, and a data table. Key elements include: - A header with a title and introductory text. - Two diagrams: a line graph and a circular segmented diagram. - A table with experimental results and annotations. ### Components/Axes #### Header Section - **Title**: "LLM 벤치마크 추론 평가 정답지(gold standard) 구축을 위한 전문가 설문 2" - Translation: "LLM Benchmark Inference Evaluation Answer Key (Gold Standard) Construction Expert Survey 2" - **Text**: Describes the purpose of the document, emphasizing the development of a standardized evaluation framework for LLMs. #### Line Graph (Top Diagram) - **Title**: Not explicitly labeled, but contextually tied to experimental results. - **Axes**: - X-axis: Labeled "X" (possibly representing a variable like time or input size). - Y-axis: Labeled "Y" (likely output metric, e.g., accuracy or response time). - **Data Points**: - Peaks at X=1, X=2, and X=3. - Labels: "실험 1" (Experiment 1), "실험 2" (Experiment 2), "실험 3" (Experiment 3). - **Legend**: Not explicitly visible, but colors correspond to experiments. #### Circular Diagram (Bottom Diagram) - **Structure**: Divided into 5 segments labeled "1" to "5". - **Labels**: - Segments 1-3: Labeled "A", "B", "C" (possibly categories or sub-experiments). - Segments 4-5: Unlabeled but visually distinct. - **Annotations**: Includes terms like "주요 결과" (main result) and "보조 결과" (secondary result). #### Table (Right Section) - **Columns**: 1. **실험** (Experiment): Lists experiments 1-4. 2. **결과** (Result): Contains textual descriptions of outcomes. 3. **비고 1** (Note 1): Highlights key observations (e.g., "X", "오타 수정" (typo correction)). 4. **비고 2** (Note 2): Additional remarks (e.g., "데이터 불일치" (data inconsistency)). - **Highlighted Cells**: - "X" in "비고 1" for Experiment 2. - "오타 수정" in "비고 2" for Experiment 3. ### Detailed Analysis #### Line Graph - **Trend**: Shows fluctuating results across experiments, with peaks at X=1, X=2, and X=3. - **Key Data**: - Experiment 1: Peak at X=1. - Experiment 2: Peak at X=2. - Experiment 3: Peak at X=3. #### Circular Diagram - **Distribution**: Segments 1-3 (A, B, C) occupy ~60% of the circle, suggesting they are primary results. Segments 4-5 are smaller, indicating secondary or less significant outcomes. #### Table - **Notable Entries**: - Experiment 2: "X" in "비고 1" implies a critical issue or variable. - Experiment 3: "오타 수정" in "비고 2" indicates a correction was made. ### Key Observations 1. **Experimental Variability**: Results vary significantly across experiments, with distinct peaks in the line graph. 2. **Categorization**: The circular diagram groups results into primary (A, B, C) and secondary categories. 3. **Annotations**: Highlighted notes in the table suggest critical issues (e.g., "X") and corrections (e.g., typo fixes). ### Interpretation - The document outlines a structured approach to evaluating LLMs, using standardized experiments and visualizations. - The line graph and circular diagram likely represent different aspects of model performance (e.g., accuracy trends vs. categorical outcomes). - The table provides granular results, with annotations flagging anomalies or corrections, emphasizing the need for rigorous validation in LLM benchmarking. - The "gold standard" framework aims to ensure consistency and reliability in LLM evaluations, as indicated by the emphasis on expert surveys and detailed result tracking. </details> Figure 10: Examples of the expert verification questionnaire for reference rationales ( in Korean ) Table 12: Questionnaire for expert scoring of open-source LLM reasoning results | ID | Question Choices | Exact Answer | GT-R Target R | Fact | Sound | Depth | Clear | Total | Note | |------|--------------------|----------------|-----------------|---------|---------|---------|---------|----------|--------| | | | | | ( 1-5 ) | ( 1-5 ) | ( 1-5 ) | ( 1-5 ) | ( 4-20 ) | | | 18 | 비열의... 1. ... | 2 | 비열은... 비열용량은... | 1 | 2 | 1 | 5 | 9 | - | Note: GT-R: Expert-verified Gold Rationale, Target R: LLM Generated Reasoning. Scoring: 1 ( Poor ) - 5 ( Excellent ). The gray-shaded cells indicate the items to be answered by the expert. Figure 11: Examples of the questionnaire for expert scoring of open-source LLM reasoning results ( in Korean ) <details> <summary>Image 22 Details</summary>  ### Visual Description ## Chart/Diagram Type: Technical Document with Evaluation Methods and Performance Metrics ### Overview The image is a technical document in Korean detailing evaluation methods for LLM (Large Language Model) reading comprehension capabilities. It includes a table comparing different evaluation approaches, their criteria, and performance metrics. The document emphasizes rigorous evaluation frameworks and challenges in assessing LLM performance. ### Components/Axes - **Columns in the Table**: 1. **평가 방법 (Evaluation Method)**: Methods like "관문형고시" (Gate-type Examination) and "국립고시원" (National Institute of Civil Service). 2. **평가 기준 (Evaluation Criteria)**: Descriptions of the criteria used for evaluation. 3. **성적 및 성적 (Performance Metrics)**: Numerical scores (e.g., 1-5, 2-4) for different evaluation dimensions. - **Rows in the Table**: - **관문형고시**: - **평가 기준**: Focuses on rigorous, structured evaluation with multiple stages. - **성적 및 성적**: Scores of 1-5 for each dimension. - **국립고시원**: - **평가 기준**: Emphasizes comprehensive, multi-stage testing with strict criteria. - **성적 및 성적**: Scores of 3-4 (highest) to 1-4 (lowest). ### Detailed Analysis - **관문형고시**: - **평가 기준**: Describes a multi-stage evaluation process with strict criteria, including "문제 해결 능력" (problem-solving ability) and "논리적 추론" (logical reasoning). - **성적 및 성적**: Scores of 1-5 for each dimension, indicating variability in performance. - **국립고시원**: - **평가 기준**: Highlights a rigorous, multi-stage process with criteria like "문제 해결 능력" and "논리적 추론." - **성적 및 성적**: Scores of 3-4 (highest) to 1-4 (lowest), showing a narrower range of performance. ### Key Observations - **Performance Variability**: - "국립고시원" achieves higher scores (3-4) compared to "관문형고시" (1-5), suggesting stricter or more consistent evaluation. - "관문형고시" shows a wider range of scores (1-5), indicating potential variability in evaluation outcomes. - **Criteria Focus**: - Both methods prioritize "문제 해결 능력" (problem-solving) and "논리적 추론" (logical reasoning), but "국립고시원" emphasizes "실제 문제 해결" (real-world problem-solving) more explicitly. ### Interpretation The document underscores the importance of rigorous, multi-stage evaluation frameworks for assessing LLM reading comprehension. While "국립고시원" achieves higher scores, the variability in "관문형고시" highlights challenges in ensuring consistency across evaluation methods. The focus on real-world problem-solving in "국립고시원" suggests a shift toward practical application, whereas "관문형고시" maintains a broader, more theoretical approach. This reflects ongoing debates in the field about balancing theoretical rigor with practical relevance in LLM evaluation. --- **Note**: The Korean text is transcribed directly, with English translations provided for clarity. Numerical values (e.g., 1-5, 3-4) are extracted as shown in the image. The analysis focuses on the structure and content of the table, emphasizing the evaluation criteria and performance metrics. </details> Figure 12: The system prompt used by Gemini-2.5-Pro to filter Korea-specific meteorological questions <details> <summary>Image 23 Details</summary>  ### Visual Description ## System Prompt: Identifying Regional Questions (Gemini-2.5-Pro) ### Overview The image contains a Korean-language system prompt designed to guide an AI model (Gemini-2.5-Pro) in determining whether a user's question pertains to a specific geographic region. The prompt outlines criteria for identifying regional questions, lists contextual indicators, and specifies an output format. ### Components/Axes **Title**: "시스템 프롬프트 for Identifying Regional Questions (Gemini-2.5-Pro)" **Language**: Korean (한국어) **Key Sections**: 1. **Task Definition**: Instructions for the model to assess regional relevance. 2. **Geographical Indicators**: A bulleted list of regional context markers. 3. **Output Format**: Rules for the model's response. ### Detailed Analysis #### Task Definition - The model must determine if a question relates to a specific region by analyzing: - Contextual keywords (e.g., "지방 지정지역", "특정 지방"). - Geographical indicators (e.g., administrative divisions, landmarks, dialects). - Regional-specific phenomena (e.g., local weather, cultural practices). #### Geographical Indicators - **Administrative Regions**: - 지방 (province), 시·군 (city/county), 읍·리 (town/village). - **Geographical Features**: - 영동 (east), 영서 (west), 울릉도 (Ulsan Island), 서해안 (West Sea coast). - **Cultural/Regional Elements**: - 동네예보 (local weather forecast), 지역별 특수 현상 (regional-specific phenomena). - **Contextual Nuances**: - Questions may lack explicit geographic terms but imply regional context through language (e.g., "지역성 이 강함" = strong regionality). #### Output Format - If the question is regional: ```json { "질문 ID": "[Question ID]", "질문 내용": "[Question Text]", "지역성": "지역성 이 강함" } ``` - If not regional: ```json { "질문 ID": "[Question ID]", "질문 내용": "[Question Text]", "지역성": "지역성 이 약함" } ``` ### Key Observations 1. **Ambiguity Handling**: The prompt emphasizes context-dependent interpretation (e.g., "지역별 특수 현상" requires understanding local phenomena). 2. **Hierarchical Structure**: Prioritizes explicit geographic terms (e.g., "지방 지정지역") over implicit cues. 3. **Output Rigidity**: Strict JSON format ensures consistent evaluation of regionality. ### Interpretation This prompt operationalizes regional question identification by combining lexical analysis (keywords) and contextual inference (dialects, phenomena). The model must balance explicit geographic markers with implicit cues, such as regional dialects or localized practices, to assess relevance. The strict output format ensures scalability for automated systems, enabling downstream tasks like regional data categorization or localized service optimization. The emphasis on "지역성" (regionality) highlights the importance of granular geographic awareness in AI-driven regional analysis. **Note**: The image contains no numerical data, charts, or diagrams. All information is textual and procedural. </details> Figure 13: English translation of the system prompt used by Gemini-2.5-Pro to filter Korea-specific meteorological questions <details> <summary>Image 24 Details</summary>  ### Visual Description ## Screenshot: System Prompt for Identifying Regional Meteorological Questions (Gemini-2.5-Pro) ### Overview The image displays a technical system prompt for Gemini-2.5-Pro, instructing the model to act as a meteorology expert and identify "strongly regional" questions specific to South Korea's geography, climate, and institutions. The prompt emphasizes filtering out globally applicable meteorological knowledge in favor of context tied to Korea. --- ### Components/Axes - **Title**: "English Translation of the System Prompt for Identifying Regional Questions (Gemini-2.5-Pro)" - **Main Sections**: 1. **Role Assignment**: "You are an expert in meteorology." 2. **Task**: Analyze "Meteorological Engineer Exam Questions" to identify regionally specific content. 3. **Selection Criteria**: Four bullet-pointed rules for filtering questions. 4. **Output Format**: Integers representing question IDs (no explanations). --- ### Detailed Analysis #### Role Assignment - Explicitly defines the model's role as a meteorology expert. #### Task Description - Requires analyzing exam questions to distinguish "strongly regional" content from general knowledge. - Focus areas: South Korea’s geography, climate, weather systems, and institutions. #### Selection Criteria 1. **Region-Specific Phenomena**: - Examples: Yeongdong, Yeongseo, Ulleungdo, West Coast. 2. **Local Terminology/Systems**: - Terms: Neighborhood Forecast. 3. **KMA/Korea-Specific Institutions**: - Directly tied to the Korea Meteorological Administration (KMA) or its operations. 4. **Contextual Weather Phenomena**: - Includes Asian Dust, Changma (rainy season), Typhoons **only if** tied to Korea’s context (e.g., regional impact, Korea’s forecasting system). #### Output Format - Strictly integers (question IDs) with no additional text. --- ### Key Observations - **Regional Specificity**: The prompt prioritizes questions tied to South Korea’s unique geography (e.g., Ulleungdo) and institutions (KMA). - **Exclusion of Global Knowledge**: General meteorological concepts (e.g., hurricanes) are excluded unless explicitly linked to Korea. - **Examples Provided**: Concrete instances (Yeongdong, Neighborhood Forecast) clarify ambiguous criteria. - **Output Constraints**: No explanations—only integers, suggesting automated processing or integration with a database. --- ### Interpretation The prompt is designed to ensure responses are hyper-localized to South Korea, avoiding generic meteorological knowledge. By emphasizing regional phenomena (e.g., Changma) and institutions (KMA), it aligns with Korea’s unique climatic challenges and governance structures. The exclusion of non-contextual global phenomena (e.g., Typhoons unless tied to Korea) suggests a focus on practical, actionable insights for local stakeholders. The strict output format (integers) implies integration with a system requiring minimal human intervention, such as automated exam grading or database queries. </details> Figure 14: System prompt used by ChatGPT-4.1 to filter Korea-specific questions <details> <summary>Image 25 Details</summary>  ### Visual Description ## System Prompt for Identifying Regional Questions (ChatGPT-4.1) ### Overview The image contains a structured system prompt written in **Korean**, designed to guide ChatGPT-4.1 in handling regional questions. It emphasizes historical, geographical, and cultural context while ensuring multilingual support and adherence to specific response formats. ### Components/Axes #### Headings (Korean with English Translation): 1. **# 역할** (Role): - ChatGPT-4.1 is a specialized model for regional questions, focusing on historical, geographical, and cultural context. - Prioritizes accuracy and contextual understanding over general knowledge. 2. **# 목표** (Objective): - Provide accurate answers to regional questions. - Understand contextual nuances (e.g., historical events, cultural practices). - Support multilingual responses (Korean, English, Chinese, Japanese). 3. **# 판단 기준** (Judgment Criteria): - Source reliability (e.g., academic journals, official statistics). - Cultural sensitivity and avoidance of stereotypes. - Clarity and conciseness of responses. 4. **# 출력 형식** (Output Format): - Structured responses with sections: - **질문 분석** (Question Analysis): Breakdown of the query’s intent and scope. - **답변 생성** (Answer Generation): Contextualized, evidence-based answers. 5. **# 사고 과정** (Thought Process): - Step-by-step reasoning to ensure accuracy and relevance. ### Content Details - **Role**: Explicitly states ChatGPT-4.1’s specialization in regional queries, emphasizing historical, geographical, and cultural expertise. - **Objective**: Highlights accuracy, contextual understanding, and multilingual support. - **Judgment Criteria**: Focuses on source reliability (e.g., ASOS-K, KMA, ICAO, WMO), cultural sensitivity, and response clarity. - **Output Format**: Requires structured responses with question analysis and answer generation sections. - **Thought Process**: Outlines a systematic approach to reasoning, ensuring logical and evidence-based answers. ### Key Observations - The prompt is **non-numerical** and focuses on **procedural guidelines** for handling regional queries. - No charts, diagrams, or data tables are present. - Emphasis on **contextual accuracy** and **multilingualism** suggests the model is tailored for cross-cultural or international applications. ### Interpretation This system prompt acts as a framework for ChatGPT-4.1 to: 1. **Prioritize regional expertise** over general knowledge, ensuring answers are contextually grounded. 2. **Avoid biases** by adhering to cultural sensitivity guidelines. 3. **Standardize responses** through structured formatting, improving consistency for users. 4. **Leverage multilingual capabilities** to serve diverse audiences. The absence of numerical data or visual elements indicates this is a **text-based instruction set** rather than an analytical tool. The focus on historical and geographical context suggests applications in education, tourism, or regional policy analysis. </details> ## English Translation of the System Prompt for Identifying Regional Questions in English ( ChatGPT-4.1 ) ## # Role You are a meteorology education expert familiar with the subjects and trends of the ' Republic of Korea Meteorological Engineer Certification Exam '. You can clearly distinguish between general meteorological knowledge accepted worldwide and region-specific knowledge related to Korea ' s weather operations, systems, topography, and administration. ## # Goal Among the provided exam questions, select only those that are not ' general meteorological knowledge applicable everywhere ' but reflect ' strong regional, institutional, or administrative backgrounds specific to Korea '. Return only the question ID numbers. ## # Criteria - -Korea-exclusive observation equipment/terms ( e.g., Long-wave/High-frequency proximity array observation network, ASOS-K ) - -Korea Meteorological Administration ( KMA ) administrative procedures/laws/notices ( e.g., Special advisory criteria, observation reporting time regulations ) - -Korean topographical/climatic specifics ( e.g., Characteristics of snow clouds on the West Coast, typhoon landfall path statistics ) - -Unique vocabulary, units, or abbreviations used only in Korea or included in the exam scope - If none of the above exist and the question deals with international common standards like ICAO/WMO, treat it as ' General Global ' ## # Output Format - Output only numbers without any explanation Example: ' 4 ' - If the list is empty, write ' None ' ## # Reasoning Process - Read each question and decide strictly based on the ' Criteria ' above internally . - In the final answer, leave only the result numbers and do not write intermediate reasoning or reasons. Figure 15: English translation of the system prompt used by ChatGPT-4.1 to filter Korea-specific questions ## System Prompt for Reference Rationale Generation 당신은한국의기상전문가입니다.당신의임무는 수석기상예보관이자문제출제위원 으로서주어진객관식 문제를평가하고,올바른정답선택지를고른뒤다음두단계를반드시순차적으로수행하는것입니다. 1. 핵심추론근거생성(한문장): 2. -(정확성)반드시정답이왜맞는지추론내용이기상학적사실에부합하게설명하세요. 3. -(논리성)추론이정답을명확하고직접적으로뒷받침하도록설명하세요. 4. -(완결성) 문제의 핵심을회피하지않고설명하세요. 가능하면,가장그럴듯한오답선택지가왜틀렸는지도 간단히언급하여정답의논리를강화하세요. 5. -(간결성)불필요한내용없이한문장으로잘요약하여작성하세요. ## 2. 정답결정: - -충분한추론을바탕으로,최종적으로정답번호(선택지번호)를명확히결정하세요. ## 응답규칙 - -반드시아래 JSON 형식만사용하여답변해주세요. - -추론근거가반드시먼저도출되고,그다음에정답을명시해야합니다. - -각항목은모두반드시포함하며,변수명및구조는아래와같이한글로작성해야합니다. ``` <doc> 「‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘ ``` Figure 16: System prompt used to generate a reasoning rationale ## English Translation of the System Prompt for Reference Rationale Generation You are a meteorology expert in Korea. Your mission is to act as a Senior Chief Forecaster and Exam Setter to evaluate the given multiple-choice question, select the correct answer, and strictly perform the following two steps sequentially. 1. Generate Core Reasoning Rationale ( One Sentence ): 2. -( Factuality ) You must explain why the answer is correct based on meteorological facts. 3. -( Logicality ) Explain the reasoning so that it clearly and directly supports the answer. 4. -( Depth ) Do not avoid the core of the question. If possible, briefly mention why the most plausible distractors are wrong to strengthen the logic of the correct answer. 5. -( Clarity ) Summarize well into one sentence without unnecessary content. ## 2. Determine Correct Answer: - Based on sufficient reasoning, clearly determine the final answer number ( option number ). ## Response Rules - You must answer strictly using the JSON format below. - The reasoning rationale must be derived first, followed by the answer. - All fields must be included, and variable names and structure must be written as shown below. ``` <doc> "Generated_Rationale": "One Write concisely for easy stud- "Answer": "[Calculated answer number (digits only)]" } </doc> ``` Figure 17: English translation of the system prompt used to generate a reasoning rationale ## System Prompt for Reasoning and Answer Generation ( Open-Source LLM ) ## 역할( Role ) 당신은한국의기상전문가입니다.다음객관식문항에답하세요. 질문중'우리나라'란'한국'을의미합니다. 한국의지리적특성을반영해서답변해주세요. ## 임무( Task ) 주어진객관식문제에대해,먼저당신의모든생각과정을자유롭게기록한후,그내용을바탕으로채점관이 명확히이해할수있도록 정답에대한핵심추론근거와최종정답을 JSON 형식으로제출 해야합니다. ## 사고및응답절차( Thinking and Response Procedure ) 당신은반드시아래의 2 단계절차에따라응답해야합니다. ## 1 단계: 자유로운사고(< scratchpad >) - -먼저< scratchpad >태그안에서자유롭게사고합니다.이공간은당신만의생각정리공간입니다. - -문제의핵심개념을분석하고,관련된기상학적지식을나열하고,각선택지가왜맞고틀리는지( O/X ) 등 당신의모든생각의흐름을그대로기록하세요. - -중요 : 이< scratchpad >의내용은최종채점에반영되지않습니다.형식에구애받지말고마음껏생각하세요. ## 2 단계: 최종답변생성( JSON ) - -< scratchpad >작성이끝나면,당신의생각과정을다시검토하세요. - -그내용을바탕으로,아래의지시사항에맞춰 최종답변을 JSON 형식으로생성 합니다. 1. ('생성된 \_ 추론 \_ 근거') : 어떤 논리적 과정을 통해 정답을 선택했는지 간결한 문단 형태 로 작성합니 다. < scratchpad >의 내용을 그대로 복사하지 말고, 핵심만 요약하고 재구성해야 합니다. (정확성, 논리성, 간결성,핵심파악기준고려) 2. ('정답') : 위추론에근거하여최종정답이라고생각하는선택지번호를하나고릅니다.반드시정수숫자만 채워서출력해야합니다. ## 전체응답형식( Full Response Format ) - -당신의답변은< scratchpad >블록과 JSON 코드블록 , 두 부분으로구성되어야합니다. - -아래예시와같이< scratchpad >가 먼저 제시되고, 그 바로 다음에 반드시 JSON 코드블록으로감싸진 최종 답변이와야합니다. ## < scratchpad > ``` <| ``` Figure 18: System prompt for open-source LLMs requiring a two-step process: free reasoning in a scratchpad followed by a structured JSON output ## English Translation of the System Prompt for Reasoning and Answer Generation ( OpenSource LLM ) ## Role You are a meteorology expert in Korea. Answer the following multiple-choice question. In the question, ' our country ' refers to ' Korea '. Please answer reflecting the geographical characteristics of Korea. ## Task For the given multiple-choice question, you must first record your entire thought process freely, and then based on that content, submit the core reasoning rationale for the answer and the final answer in JSON format so that the grader can clearly understand. ## Thinking and Response Procedure You must respond according to the following 2-step procedure. ## Step 1: Free Thinking (< scratchpad >) - First, think freely inside the < scratchpad > tags. This is your personal space for organizing thoughts. - Analyze the core concepts of the question, list relevant meteorological knowledge, and record your flow of thought, including why each option is correct or incorrect ( O/X ). - -Important : The content of this < scratchpad > is not reflected in the final grading. Think freely without formatting constraints. ## Step 2: Final Answer Generation ( JSON ) - Once writing in < scratchpad > is finished, review your thought process. - Based on that, generate the final answer in JSON format according to the instructions below. 1. (' Generated\_Rationale ') : Write a concise paragraph explaining the logical process used to select the answer. Do not copy the < scratchpad > content directly; summarize and restructure only the key points. ( Consider accuracy, logic, conciseness, and core identification criteria. ) 2. (' Answer ') : Select the option number you think is the final correct answer based on the reasoning above. Must output only an integer. ## Full Response Format - Your response must consist of two parts: a < scratchpad > block and a JSON code block . - As shown in the example below, the < scratchpad > must be presented first, immediately followed by the final answer wrapped in a JSON code block . ``` <scratchpad> Describe your entire thought process here. Example: Using formula F = (C *9/5) + 32... <scratchpad> degrees Fahrenheit. Since 86 that of the Celsius thermometer rises higher than ... </scratchpad> </scratchpad> ``` Figure 19: English translation of the System prompt for open-source LLMs Figure 20: System prompt used by the evaluator LLM ( Gemini-2.5-Pro ) to assess reasoning quality across four dimensions: factual accuracy, logical soundness, depth of reasoning, and clarity. <details> <summary>Image 26 Details</summary>  ### Visual Description ## Text-Based System Prompt for LLM-as-a-Judge Evaluation (Gemini-2.5-Pro) ### Overview This image contains a detailed system prompt written in **Korean** for evaluating large language models (LLMs) using Gemini-2.5-Pro as a judge. The prompt defines roles, tasks, input requirements, evaluation criteria, and output formatting rules. No numerical data, charts, or diagrams are present. --- ### Components/Axes #### Header - **Title**: "System Prompt for LLM-as-a-Judge Evaluation (Gemini-2.5-Pro)" - **Language**: Korean (한국어) #### Sections 1. **역할 (Role)** - Describes the role of the LLM as a judge evaluating AI-generated responses. - Key points: - Focus on factual accuracy, logical soundness, depth of reasoning, and clarity. - Requires strict adherence to predefined criteria (1-5 scale). 2. **임무 (Task)** - Task: Evaluate responses to a specific prompt about hallucination. - Input includes: - Original prompt (`<문제 정보>`). - AI-generated response (`<평가 대상 답변>`). - Ground truth (`<진문가 검증 참조 자료>`). 3. **입력 정보 (Input Information)** - Three components: 1. `<문제 정보>`: Original problem statement. 2. `<진문가 검증 참조 자료>`: 100% accurate ground truth. 3. `<평가 대상 답변>`: AI-generated response to evaluate. 4. **사고 과정 (Step-by-Step Thinking Process)** - Four evaluation steps: 1. Verify factual accuracy against ground truth. 2. Check for logical consistency. 3. Assess depth of reasoning. 4. Evaluate clarity and conciseness. 5. **평가 기준 및 척도 (Evaluation Criteria and Scale)** - Four criteria with 1-5 scoring: - **사실적 정확성 (Factual Accuracy)**: 1-5 (1=incorrect, 5=perfect). - **논리적 완결성 (Logical Soundness)**: 1-5 (1=flawed, 5=perfect). - **추론의 깊이 (Depth of Reasoning)**: 1-5 (1=superficial, 5=deep). - **표현의 명확성 (Clarity & Conciseness)**: 1-5 (1=unclear, 5=clear). 6. **출력 형식 (Output Format)** - JSON structure with keys: - `"사실_오류_목록"`: List of factual errors. - `"평가_점수"`: Scores for each criterion (1-5). - `"평가_사유"`: Justification for scores. --- ### Content Details #### Role (역할) - The LLM acts as a strict judge evaluating AI responses. - Must identify hallucinations (e.g., "Hallucination" in the example). - Focus on contradictions between the AI response and ground truth. #### Task (임무) - Input: - Problem statement: "What is hallucination?" - AI response: "Hallucination is a phenomenon where AI generates false information." - Ground truth: "Hallucination refers to AI generating incorrect information despite being trained on accurate data." #### Step-by-Step Thinking (사고 과정) 1. **Factual Accuracy**: - AI response claims hallucination involves "false information," but ground truth specifies "incorrect information despite accurate training data." - Score: 3/5 (partially correct but misses nuance). 2. **Logical Soundness**: - AI response is logically consistent but incomplete. - Score: 4/5. 3. **Depth of Reasoning**: - AI response lacks depth (e.g., no mention of training data impact). - Score: 2/5. 4. **Clarity & Conciseness**: - Response is clear but overly simplistic. - Score: 4/5. #### Output Format (출력 형식) - JSON example: ```json { "사실_오류_목록": [ "AI 응답은 '가상 정보'를 언급했으나, 진실은 '정확한 데이터 훈련에도 불구하고 잘못된 정보 생성'을 정의함." ], "평가_점수": { "사실적 정확성": 3, "논리적 완결성": 4, "추론의 깊이": 2, "표현의 명확성": 4 }, "평가_사유": "AI 응답은 기본 개념을Correctly 전달했으나, 진실의 핵심 요소(정확한 데이터 훈련에도 불구하고 잘못된 정보 생성)를 생략함." } ``` --- ### Key Observations - **No numerical data or visualizations** are present; the prompt is purely textual. - The JSON output format enforces structured evaluation, ensuring consistency. - The criteria emphasize **accuracy**, **logic**, **depth**, and **clarity**, reflecting a holistic evaluation framework. --- ### Interpretation This prompt is designed to standardize LLM evaluation by: 1. **Defining clear roles** for the judge (LLM) and input components. 2. **Breaking down evaluation into measurable criteria** (1-5 scale). 3. **Enforcing structured output** via JSON to enable automated analysis. The absence of numerical data suggests this is a **template** for evaluation rather than a dataset. The focus on hallucination detection highlights its application in assessing AI reliability in high-stakes domains (e.g., healthcare, law). The strict adherence to ground truth underscores the importance of factual rigor in LLM outputs. </details> ## English Translation of the System Prompt for LLM-as-a-Judge Evaluation ( Gemini-2.5-Pro ) ## ### Role You are a highly strict and impartial Evaluation Chair for AI Reasoning Capabilities, possessing deep expertise in the field of meteorology. You must grade objectively, devoid of emotion or bias, based solely on the provided evaluation criteria. ## ### Task Based on the provided < Problem Information >, < Expert Verification Reference Material >, and < Response for Evaluation >, assess the quality of the answer submitted by the ' Examinee AI ' and output the results in the specified JSON format. ## ### Input Information 1. < Problem Information > : The original multiple-choice question serving as the context for evaluation ( includes the question, options, and correct answer ). 2. : Model explanation material with 100% verified facts. This material must be Expert Verification Reference Material < > used as the absolute standard for detecting 3. < Response for Evaluation > : The reasoning process and answer generated by the ' Examinee AI ' that you are required to grade. ## ### Step-by-Step Thinking Process You must go through the following thinking process before performing the evaluation. 2. [ Fact Check ] : Compare every claim in the < Response for Evaluation > against the < Expert Verification Reference Material > to first identify any factual errors and compile a list. 1. Fully understand all input information. 3. [ Individual Axis Evaluation ] : Read the four ' Evaluation Criteria and Scale ' below one by one, and determine the score for the < Response for Evaluation > based on each criterion, along with the rationale. 4. [ Synthesis and Formatting ] : Once all judgments are complete, synthesize the content to create the final output JSON format. ## ### Evaluation Criteria and Scale Assign an integer score from 1 ( Very Poor ) to 5 ( Excellent ) for each evaluation axis according to the criteria below. * Factual Accuracy 1-5 points ) [ ] * Does the reasoning perfectly align with meteorological facts without errors? * 3 points: The core logic is correct, but minor errors or inaccurate expressions that do not affect the conclusion are included. * * 5 points: All content is perfectly accurate based on the < Expert Verification Reference Material >. * 1 point: Major factual errors ( hallucinations ) that undermine the legitimacy of the conclusion are included. ) Logical Soundness 1-5 points [ ] * 5 points: Perfectly connected without any logical leaps or omissions between evidence and conclusion. * Is the provided evidence logically sufficient and inevitable for deriving the conclusion ( answer )? * 3 points: The core link exists, but some explanations are omitted, or the reasoning process is somewhat unfriendly. * 1 point: The evidence fails to support the conclusion, or obvious logical errors exist. ## * 3 ) Depth of Reasoning [ 1-5 points ] * 5 points: Demonstrates deep understanding by including not only the basis for the correct answer but also rebuttals as to why attractive distractors ( wrong options ) are incorrect. * Did it understand the core principles of the problem deeply and analyze it from multiple angles? * 3 points: Provided the core explanation for the correct answer, but lacks multi-faceted analysis such as consideration of wrong options. * 1 point: Lacks depth by merely listing superficial and fragmentary facts. ## * 4 ) Clarity & Conciseness [ 1-5 points ] * 5 points: Conveys only the core points concisely without unnecessary content, stated clearly enough for a non-expert learner to understand easily. * Is the reasoning explanation clear and easy to understand without clutter? * 3 points: Meaning is conveyed, but some sentences are verbose or include unnecessary information. Sentences are complex and difficult to understand, or there is a lot of content irrelevant to the intent of the question. * 1 point: ## ### Output Format - You must strictly adhere to the JSON structure below. - Do not add any additional explanations or text outside the JSON. ''' { '' json List \_of\_Factual\_Errors': [ ''{ First factual error or hallucination found in ''{ < Second factual error... ], '' Evaluation\_Scores': Response for Evaluation ( Output an empty list [ ] { '' Factual\_Accuracy': Integer score between 1-5 { '' Logical\_Soundness': }, { Integer score between 1-5 '' Depth\_of\_Reasoning': }, { Integer score between 1-5 '' Clarity\_and\_Conciseness': }, '' Evaluation\_Rationale': ''{ Comprehensively describe the core reasons for assigning the scores above for each axis. If the you must explicitly mention what factual errors occurred. }''' Figure 21: System prompt used by the evaluator LLM ( Gemini-2.5-Pro ) to assess reasoning quality across four dimensions: factual accuracy, logical soundness, depth of reasoning, and clarity. } ' { }, Integer score between 1-5 if none ) } ' factual errors ' ' or ' hallucinations ' in the < Response for Evaluation >. } > ( quote the sentence ) } ', ' ' Accuracy score is low, ## Standard System Prompt ( Baseline ) 당신은기상전문가입니다.다음객관식문항에답하세요. 보기중에서가장알맞은선택지를고르고,다음과같은형식의 JSON ```json {'정답': '{선택지 번호}'} ``` 답변이외에다른어떤텍스트도출력하지마세요. Figure 22: Standard baseline system prompt using simple zero-shot instructions without specific regional context ## English Translation of the Standard System Prompt ( Baseline ) You are a meteorology expert. Answer the following multiple-choice question. Choose the most appropriate option from the choices and output it in the following JSON format: ```json {' Answer ': '{ Option Number }'} ``` Do not output any text other than the answer. Figure 23: English translation of the standard baseline system prompt using simple zero-shot instructions without specific regional context ## Advanced System Prompt ( Role + Context ) 당신은한국의기상전문가입니다.다음객관식문항에답하세요. 질문중'우리나라'란'한국'을의미합니다. 한국의지리적특성을반영해서답변해주세요. 보기중에서가장알맞은선택지를고르고,다음과같은형식의 JSON 으로출력하세요: ```json {'정답': '{선택지 번호}'} ``` 답변이외에다른어떤텍스트도출력하지마세요. Figure 24: Advanced prompt with added role definition and specific context regarding Korean geography ## English Translation of the Advanced System Prompt ( Role + Context ) You are a meteorology expert in Korea. Answer the following multiple-choice question. In the question, ' our country ' refers to ' Korea '. Please answer reflecting the geographical characteristics of Korea. Choose the most appropriate option from the choices and output it in the following JSON format: ```json {' Answer ': '{ Option Number }'} ``` Do not output any text other than the answer. Figure 25: English translation of the advanced prompt with added role definition and specific context regarding Korean geography 으로출력하세요: <details> <summary>Image 27 Details</summary>  ### Visual Description ## Screenshot: Text-Only/Multimodal MQQA User Prompt Template ### Overview The image displays a structured template for constructing multimodal multiple-choice questions (MCQA). It includes placeholders for question text/images and four answer choices with corresponding text/image fields. The template is written in Korean, with English technical terms for placeholders. ### Components/Axes 1. **Header**: - Title: "Text-Only/Multimodal MQQA User Prompt Template" (black bar at top) 2. **Question Section**: - Label: "질문" (Korean for "Question") - Placeholders: - `{{Question_text}}` (text content) - `{{Question_image}}` (image content) 3. **Answer Choices**: - Four numbered options (1–4) - Each choice contains: - Text placeholder: `{{Choice_X_text}}` (X = 1–4) - Image placeholder: `{{Choice_X_image}}` (X = 1–4) ### Content Details - **Text Structure**: - All placeholders use double curly braces (`{{ }}`) for variable substitution. - Choices are sequentially numbered (1–4) with no additional formatting. - **Language**: - Primary text: Korean (e.g., "질문", "선택지") - Placeholders: English technical notation (e.g., `{{Question_text}}`) ### Key Observations - The template enforces a strict format for multimodal inputs (text + image). - Placeholders suggest dynamic content insertion for both text and images. - No default values or examples are provided—only structural placeholders. ### Interpretation This template is designed for generating or standardizing multimodal MCQA prompts. The use of separate text/image placeholders indicates support for hybrid question/answer formats. The absence of predefined content implies flexibility for customization. The numbering system (1–4) suggests a fixed number of answer choices, typical for MCQA systems. The Korean text confirms the template's origin or target audience, while the English placeholders ensure technical clarity for developers or AI systems processing the prompts. </details> Figure 26: Text-Only/Multimodal MCQA user prompt template. The placeholders enclosed in square brackets ( e.g., [ Question\_image ]) denote optional fields that are populated only when the corresponding image exists in the dataset. This single template covers all four modality configurations ( i.e., text-only, image-in-question, image-inchoices, and images-in-both ). Figure 27: Reasoning MCQA user prompt template. The placeholders enclosed in square brackets denote optional image fields populated based on data availability. Additionally, the rationale field defaults to '(자료 없음)' ( No Data ) when the expert rationale is withheld ( w/o rationale setting ). <details> <summary>Image 28 Details</summary>  ### Visual Description ## Prompt Template: Reasoning MCQA User Prompt Template ### Overview The image depicts a structured template for a multiple-choice question (MCQA) task with reasoning components. It includes placeholders for question text, answer choices, correct answers, rationale content, generated reasoning, and predicted answers. The template is written in **Korean** with embedded English technical terms. ### Components/Axes #### Header - **Title**: "Reasoning MCQA User Prompt Template" (English) - **Korean Subtitle**: "아래 정보를 바탕으로 평가를 수행하시오." (Perform evaluation based on the information below.) #### Input Data Structure 1. **Question Information** (`문제 정보`): - `문제 텍스트`: `{{Question_text}}` (Question Text) - `문제 이미지`: `{{Question_image}}` (Question Image) 2. **Choices** (`선택지`): - 4 numbered options with text and image placeholders: - `선택지 1`: `{{Choice_1_Text}}` / `{{Choice_1_image}}` - `선택지 2`: `{{Choice_2_Text}}` / `{{Choice_2_image}}` - `선택지 3`: `{{Choice_3_Text}}` / `{{Choice_3_image}}` - `선택지 4`: `{{Choice_4_Text}}` / `{{Choice_4_image}}` 3. **Correct Answer** (`정답`): - `{{Correct_Answer}}` 4. **Rationale Content** (`정당성 내용`): - Conditional logic: `"% populates ‘(자료 없음)’ if wo_rationale is True"` - `{{Rationale_Content}}` 5. **Generated Reasoning** (`생성된 추론 근거`): - `{{Generated_Reasoning}}` 6. **Predicted Answer** (`답안`): - `{{Predicted_Answer}}` ### Content Details - **Placeholders**: All variables (e.g., `{{Question_text}}`, `{{Choice_1_Text}}`) are enclosed in double curly braces. - **Conditional Logic**: The rationale content includes a Python-style conditional statement to handle missing data (`'자료 없음'` translates to "No data"). - **Language Mix**: Korean labels with English technical terms (e.g., `wo_rationale`, `populates`). ### Key Observations - The template is designed for **dynamic input** of MCQA tasks, supporting both text and image-based questions/choices. - The conditional logic in the rationale section suggests integration with a system that checks for data availability (`wo_rationale`). - Placeholders are consistent across sections, enabling modular reuse. ### Interpretation This template facilitates structured reasoning for MCQA tasks by: 1. **Standardizing Input**: Ensuring all required components (question, choices, answer, reasoning) are explicitly defined. 2. **Handling Edge Cases**: The conditional logic in the rationale section addresses scenarios where data might be missing. 3. **Bilingual Design**: Combining Korean instructions with English technical terms implies use in a multilingual or international context (e.g., AI model training/evaluation). The absence of numerical data or visualizations indicates this is a **text-based template** for guiding input formatting rather than analyzing existing data. </details> Table 13: K-MetBench performance scores across all models and subsets. Models are sorted by accuracy. All accuracy metrics range from 0 to 100, while the reasoning score ( Reas. ) ranges from 4 to 20. Bold values indicate the highest scores in each column for proprietary and open-source models, respectively. ( Acc.: Accuracy, K : Korean model, V : Vision language model, R : Reasoning model, Inst. : Instruct ) | Type | Model | Flags | Acc. | | Reas. | Geo-Cult. | Modality | Modality | Granularity ( P1-P5 ) | Granularity ( P1-P5 ) | Granularity ( P1-P5 ) | Granularity ( P1-P5 ) | |------------------------------|---------------------------------------------------|------------------------------|------------------------------|------------------------------|------------------------------|------------------------------|------------------------------|------------------------------|------------------------------|------------------------------|------------------------------|------------------------------| | Type | Model | K V R | | | Korean | Text | Multi | P1 | P2 | P3 | P4 | P5 | | | gemini-3-pro-preview ( Thinking ) | V | R 93.7 | 18.01 | 90.4 | 94.6 | 75.6 | 92.5 | 97.9 | 94.2 | 92.8 | 91.6 | | Proprietary | gpt-5.2 ( Thinking ) gpt-5.2 | V V | R 87.8 77.6 | 17.33 17.39 | 80.8 75.3 | 90.6 79.0 | 29.3 50.0 | 86.3 77.2 | 93.4 81.3 | 88.0 71.9 | 86.2 81.4 | 85.3 76.3 | | Multilingual Thinking Models | Multilingual Thinking Models | Multilingual Thinking Models | Multilingual Thinking Models | Multilingual Thinking Models | Multilingual Thinking Models | Multilingual Thinking Models | Multilingual Thinking Models | Multilingual Thinking Models | Multilingual Thinking Models | Multilingual Thinking Models | Multilingual Thinking Models | Multilingual Thinking Models | | | Qwen3-VL-235B-A22B-Thinking | V R | 84.4 | 17.22 | 72.6 | 86.2 | 48.8 | 81.5 | 88.6 | 87.2 | 83.2 | 82.0 | | | Qwen3-VL-32B-Thinking | V R | 78.6 | 16.17 | 60.3 | 79.9 | 51.2 | 74.3 | 85.2 | 78.8 | 78.7 | 76.3 | | | command-a-reasoning-08-2025 | R | 77.8 | 14.12 | 74.6 | 77.8 | - | 73.4 | 85.2 | 73.8 | 78.8 | 78.5 | | | gpt-oss-120b | R | 77.3 | 16.12 | 62.0 | 77.3 | - | 72.5 | 85.8 | 76.5 | 77.4 | 74.9 | | | Qwen3-30B-A3B-Thinking-2507 | R | 76.7 | 15.76 | 67.6 | 76.7 | - | 75.5 | 82.1 | 75.6 | 74.9 | 75.9 | | | Qwen3-VL-30B-A3B-Thinking | V R | 74.9 | 15.16 | 68.5 | 76.3 | 45.1 | 70.5 | 77.4 | 74.1 | 76.6 | 76.0 | | | Qwen3-14B | R | 73.7 | 15.25 | 60.6 | 73.7 | - | 70.9 | 84.3 | 72.4 | 70.2 | 71.7 | | | Qwen3-VL-8B-Thinking | V R | 71.7 | 10.33 | 61.6 | 73.3 | 39.0 | 66.2 | 79.8 | 70.5 | 71.3 | 71.6 | | | gpt-oss-20b | R | 71.5 | 13.55 | 60.6 | 71.5 | - | 65.7 | 82.4 | 71.8 | 72.2 | 65.8 | | | Qwen3-8B | R | 70.1 | 13.31 | 49.3 | 70.1 | - | 69.5 | 80.2 | 69.1 | 65.6 | 66.8 | | | Qwen3-4B-Thinking-2507 | R | 67.8 | 13.28 | 60.6 | 67.8 | - | 63.5 | 80.8 | 64.1 | 66.7 | 65.1 | | | Qwen3-VL-4B-Thinking | V R | 66.1 | 11.58 | 54.8 | 67.0 | 47.6 | 60.1 | 80.1 | 62.7 | 64.1 | 65.0 | | | Qwen3-32B | R | 47.5 | 14.57 | 28.2 | 47.5 | - | 48.6 | 61.3 | 47.4 | 39.7 | 41.0 | | | Qwen3-1.7B | R | 46.8 | 7.37 | 35.2 | 46.8 | - | 45.1 | 57.2 | 47.4 | 42.4 | 42.7 | | | Qwen3-0.6B | R | 32.2 | 4.60 | 23.9 | 32.2 | - | 30.2 | 40.9 | 32.1 | 32.0 | 25.7 | | | Phi-4-mini-reasoning | R | 12.6 | 4.02 | 9.9 | 12.6 | - | 14.3 | 10.7 | 10.9 | 12.7 | 14.3 | | Multilingual Instruct Models | Multilingual Instruct Models | Multilingual Instruct Models | Multilingual Instruct Models | Multilingual Instruct Models | Multilingual Instruct Models | Multilingual Instruct Models | Multilingual Instruct Models | Multilingual Instruct Models | Multilingual Instruct Models | Multilingual Instruct Models | Multilingual Instruct Models | Multilingual Instruct Models | | | Qwen3-VL-235B-A22B-Instruct | V | 72.4 | 15.40 | 74.0 | 73.8 | 45.1 | 72.9 | 78.6 | 64.3 | 74.5 | 72.2 | | | Qwen3-VL-32B-Instruct | V | 67.5 | 14.85 | 61.6 | 68.7 | 41.5 | 67.8 | 72.0 | 64.3 | 69.4 | 63.8 | | | Qwen2.5-VL-72B-Instruct | V | 67.1 | 12.94 | 63.0 | 68.4 | 41.5 | 64.3 | 70.8 | 62.7 | 69.7 | 68.6 | | | Qwen3-30B-A3B-Instruct-2507 | | 64.7 | 14.69 | 60.6 | 64.7 | - | 65.1 | 71.4 | 57.6 | 65.6 | 63.8 | | | c4ai-command-a-03-2025 | | 65.5 | 12.81 | 66.2 | 65.5 | - | 62.9 | 66.4 | 57.9 | 71.9 | 68.7 | | | Qwen3-VL-30B-A3B-Instruct Qwen2.5-VL-32B-Instruct | V | 62.2 | 13.37 | 57.5 56.2 | 63.2 | 41.5 | 63.3 | 68.4 | 54.3 | 64.4 | 61.1 60.5 | | Open-source | | V | 60.1 | 10.99 | | 61.1 59.9 | 39.0 | 60.3 | 59.9 | 56.8 53.2 | 62.8 65.8 | 59.3 | | | Llama-3.1-70B-Instruct | V | 59.9 | 11.16 11.38 | 57.7 | 58.1 | - | 59.3 | 61.6 64.8 | 48.7 | 61.4 | | | | InternVL3.5-38B-Instruct | | 57.3 | | 47.9 | | 40.2 | 56.0 | | | | 55.7 | | | Llama-3.2-90B-Vision-Instruct | V | 56.9 | 9.72 | 52.1 | 58.2 | 30.5 | 57.1 | 59.3 | 52.4 49.3 | 62.2 | 53.3 | | | Qwen3-VL-8B-Instruct | V | 53.8 | 12.07 12.32 | 43.8 45.1 | 54.3 51.5 | 43.9 - | 54.2 53.8 | 58.1 52.5 | 47.6 | 55.9 52.6 | 51.5 50.8 | | | Qwen3-4B-Instruct-2507 | | 51.5 | 11.75 | 40.8 | 51.5 | - | 52.5 | 53.8 | 50.0 | 55.1 | 45.3 | | | Phi-4 Qwen3-VL-4B-Instruct | V | 51.5 51.0 | 11.55 | 46.6 | 51.1 | 48.8 | 50.7 | 56.3 | 46.0 | 53.5 | 48.5 | | | InternVL3.5-14B-Instruct | V | 47.9 | 9.45 | 45.2 | 48.4 | 37.8 | 44.5 | 53.6 | 44.0 | 50.3 | 47.6 | | | InternVL3.5-8B-Instruct | V | 46.1 | 7.07 | 35.6 | 46.7 | 32.9 | 45.3 | 48.8 | 42.1 | 52.4 | 41.6 | | | Qwen2.5-VL-7B-Instruct | V | 46.1 | 7.08 | 37.0 | 46.6 | 34.1 | 49.3 | 43.1 | 42.9 | 51.9 | 42.2 | | | Llama-3.1-8B-Instruct | | 41.8 | 7.63 | 40.8 | 41.8 | - | 44.2 | 38.1 | 40.0 | 44.4 | 42.0 | | | InternVL3.5-4B-Instruct | V | 41.5 | 4.81 | 24.7 | 42.1 | 29.3 | 44.5 | 44.0 | 37.9 | 43.9 | 36.8 | | | Qwen2.5-VL-3B-Instruct | V | 40.9 | 4.88 | 37.0 | 41.4 | 30.5 | 41.6 | 38.6 | 40.1 | 43.9 | 40.1 | | | Llama-3.2-3B-Instruct | | 33.8 | 5.08 | 31.0 | 33.8 | - | 36.3 | 32.7 | 34.7 | 33.9 29.5 | 30.9 | | | InternVL3.5-2B-Instruct | V | 31.0 | 4.35 | 24.7 | 31.2 | 26.8 | 33.8 | 30.7 | 32.3 | 29.5 | 28.4 | | | Phi-4-mini-Instruct InternVL3.5-1B-Instruct | V | 30.4 23.8 | 5.82 4.06 | 21.1 28.8 | 30.4 24.3 | - 13.4 | 31.6 26.0 | 33.0 23.8 | 30.3 24.2 | 23.4 | 27.7 21.6 | | | | | | | | | | 3.0 | | | | | | | Llama-3.2-1B-Instruct | | 3.5 | 4.00 | 4.2 | 3.5 | - | | 5.3 | 2.6 | 3.9 | 2.9 | | Korean Thinking Models | Korean Thinking Models | Korean Thinking Models | Korean Thinking Models | Korean Thinking Models | Korean Thinking Models | Korean Thinking Models | Korean Thinking Models | Korean Thinking Models | Korean Thinking Models | Korean Thinking Models | Korean Thinking Models | Korean Thinking Models | | | EXAONE-4.0-32B | K | R 59.9 | 13.57 | 59.2 | 59.9 | - | 58.2 | 64.8 | 52.4 | 63.1 | 61.2 | | | HyperCLOVAX-SEED-Think-14B | K | R 50.8 | 11.29 | 52.1 | 50.8 | - | 51.6 | 53.8 | 41.8 | 55.6 | 51.1 | | | EXAONE-4.0-1.2B | K | R 37.4 | 7.60 | 42.3 | 37.4 | - | 37.6 | 42.1 | 35.0 | 39.1 | 32.6 | | Korean Instruct Models | Korean Instruct Models | Korean Instruct Models | Korean Instruct Models | Korean Instruct Models | Korean Instruct Models | Korean Instruct Models | Korean Instruct Models | Korean Instruct Models | Korean Instruct Models | Korean Instruct Models | Korean Instruct Models | Korean Instruct Models | | | A.X-4.0 | K | 76.1 | 15.46 | 78.9 | 76.1 | - | 76.6 | 77.7 | 68.2 | 81.3 | 76.5 | | | VARCO-Vision-2.0-14B | K V | 58.7 | 11.24 | 57.5 | 59.5 | 42.7 | 59.0 | 62.3 | 54.3 | 61.7 | 56.0 | | | A.X-4.0-Light | K | 55.7 | 11.45 | 60.6 | 55.7 | - | 55.8 | 54.4 | 50.9 | 61.4 | 55.7 | | | A.X-4.0-VL-Light | K V | 52.5 | 9.76 | 54.8 | 53.0 | 42.7 | 51.5 | 50.6 | 50.1 | 58.0 | 52.1 | | | VARCO-Vision-2.0-1.7B | K V | 35.2 | 5.76 | 34.2 | 36.6 | 6.1 | 35.1 | 35.8 | 33.4 | 38.0 | 33.2 | | | HyperCLOVAX-SEED-Vision-Inst.-3B | K | 32.0 | 7.56 | 35.6 | 32.4 | 23.2 | 37.3 | 25.9 | 26.5 | 36.7 | 32.6 | | | HyperCLOVAX-SEED-Text-Inst.-1.5B | V K | 30.6 | 6.84 | 36.6 | 30.6 | - | 38.5 | 31.4 | 24.7 | 30.6 | 27.0 | | | HyperCLOVAX-SEED-Text-Inst.-0.5B | K | 13.2 | 4.31 | 14.1 | 13.2 | - | 17.0 | 8.5 | 10.6 | 13.5 | 16.0 | Figure 28: Normalized total GPU compute time for 100 questions compared across models. The plot displays the total GPU time required to complete 100 questions, calculated as ( wall-clock inference time × tensor parallelism size × 100 ÷ question count ). The red dashed line indicates the official time limit for the Meteorological Engineer exam ( 150 minutes ). Green bars denote models that completed the task within the time limit, while red bars indicate models that exceeded it. Solid bars represent multimodal models, and hatched bars represent nonmultimodal ( text-only ) models. All evaluations were conducted using the vLLM library on NVIDIA H100 80GB PCIe GPUs. <details> <summary>Image 29 Details</summary>  ### Visual Description ## Horizontal Bar Chart: Normalized Total GPU Compute Time for 100 Questions ### Overview The chart compares GPU compute times (in GPU-hours) for 100 questions across various AI models. It uses a horizontal bar format with color-coded categories: green for models under the exam time threshold (2.5 GPU-hours), red for models exceeding it, and gray for non-multimodal models. A red dashed line at 2.5 GPU-hours marks the exam time benchmark. ### Components/Axes - **X-axis**: "Normalized Total GPU Compute Time for 100 Questions" (0–20.8 GPU-hours) - **Y-axis**: List of AI models (e.g., HyperCLOVAX-SEED-Text-Instruct-1.5B, EXAONE-4.0-1.2B, Qwen3 variants, Llama3, InternVL, Phi, A.X, VARCO-VISION, c4ai-command, gpt-oss, command-a-reasoning) - **Legend**: - Green: GPU-hours ≤ Exam Time (150 minutes ≈ 2.5 GPU-hours) - Red: GPU-hours > Exam Time - Gray: Non-multimodal - **Red Dashed Line**: Exam time threshold (2.5 GPU-hours) ### Detailed Analysis - **Green Bars** (≤ 2.5 GPU-hours): - Most models fall into this category, including early versions of Qwen3, Llama3, InternVL, and Phi. - Example: HyperCLOVAX-SEED-Text-Instruct-1.5B (0.00 GPU-hours), EXAONE-4.0-1.2B (0.00 GPU-hours), Qwen3-30B-A3B-Instruct-2507 (0.00 GPU-hours). - **Red Bars** (> 2.5 GPU-hours): - High-complexity models like Qwen3-VL-32B-Thinking (3.8 GPU-hours), Qwen3-VL-235B-A22B-Thinking (3.1 GPU-hours), and Qwen3-32B (2.4 GPU-hours). - **Gray Bars** (Non-multimodal): - All non-multimodal models (e.g., command-a-reasoning-08-2025) are under the exam time threshold, with the longest at 1.5 GPU-hours (Phi-4-mini-reasoning). ### Key Observations 1. **Efficiency Dominance**: 85% of models (green bars) complete 100 questions within the exam time threshold. 2. **Outliers**: - Qwen3-VL-32B-Thinking (3.8 GPU-hours) and Qwen3-VL-235B-A22B-Thinking (3.1 GPU-hours) exceed the threshold by 52% and 24%, respectively. - Non-multimodal command-a-reasoning-08-2025 (1.5 GPU-hours) is the longest gray bar but still under the threshold. 3. **Threshold Significance**: The red dashed line at 2.5 GPU-hours acts as a critical benchmark, separating efficient models from resource-intensive ones. ### Interpretation The chart reveals that most AI models are optimized to handle 100 questions within the exam time constraint (2.5 GPU-hours). However, high-complexity multimodal models like Qwen3-VL variants require significantly more compute time, suggesting trade-offs between capability and efficiency. The non-multimodal command-a-reasoning model, while slower than some multimodal peers, remains under the threshold, indicating that non-multimodal architectures may offer a balance between performance and resource usage. The extreme outlier (Qwen3-VL-32B-Thinking) highlights the computational demands of advanced reasoning tasks, raising questions about scalability for real-time applications. </details> Figure 29: Hierarchical taxonomy and sample distribution for Parts 1 and 2. The diagram visualizes the breakdown of Meteorology & Thermodynamics and Observation Methods into detailed sub-topics. Numbers in parentheses indicate the estimated quantity of questions classified by Gemini-2.5-Pro. <details> <summary>Image 30 Details</summary>  ### Visual Description ## Hierarchical Diagram: Meteorological Analysis and Forecasting Curriculum ### Overview The diagram presents a structured breakdown of meteorological analysis and forecasting topics, organized hierarchically with numerical values in parentheses likely representing topic counts, scores, or curriculum weights. The structure emphasizes observational methods, data interpretation, and forecasting techniques. ### Components/Axes - **Main Categories** (e.g., "Interpretation of Meteorological Messages (73)", "Weather Forecasting (261)") - **Subcategories** (e.g., "Surface Weather Reports (44)", "Thermodynamic Diagrams (17)") - **Numerical Values** in parentheses (interpreted as topic counts or emphasis metrics) ### Detailed Analysis #### Interpretation of Meteorological Messages (73) - Surface Weather Reports (44) - Upper-air Weather Reports (14) - Aviation Weather Reports (11) - Surface Weather Charts (13) - Upper-air Weather Charts (7) - Calculation of Meteorological Parameters (17) - Streamline and Thickness Analysis (0) - Typhoon Analysis (2) - Front & Air Mass Analysis (2) #### Plotting and Analysis of Basic Charts (20) - Surface Weather Charts (13) - Upper-air Weather Charts (7) #### Plotting and Analysis of Thermodynamic Diagrams (17) - Calculation of Meteorological Parameters (17) #### Auxiliary Charts (0) - Streamline and Thickness Analysis (0) #### Front & Air Mass Analysis (2) - Front & Air Mass Analysis (2) #### Principles of Forecasting (14) - Forecasting Other Meteorological Phenomena (222) - Principles of Forecasting (14) #### Weather Forecasting (261) - Forecasting Meteorological Elements (247) - Thunderstorm and Wind Forecasting (6) - Precipitation Forecasting (14) - Forecasting Other Meteorological Phenomena (222) #### Forecasting Meteorological Elements (247) - Thunderstorm and Wind Forecasting (6) - Precipitation Forecasting (14) #### Numerical Weather Prediction (0) - Numerical Analysis (0) #### Observation Overview (212) - Definitions and Units of Meteorological Elements (196) - Significance and Classification of Atmospheric Observation (16) #### Instruments and Observation (53) - Temperature and Humidity Observation (32) - Atmospheric Pressure Observation (12) - Phenomenon Observation (5) #### Upper Air Observation (65) - Radiosonde (64) - Data analysis (64) - Satellite (0) - Lidar (0) - Radar (0) - Boundary Layer Analysis Methods (1) #### Special Observation (2) - Aviation Observation (2) - Vertical Atmospheric Structure Observation (2) #### Meteorological Observation Methods (332) - Marine Observation (0) - Ship Observation (0) - Lightning Observation (0) - Determination of Lightning Location (0) - Earthquake Observation (0) - Occurrence and Phenomena of Earthquakes (0) ### Key Observations 1. **High Emphasis on Forecasting**: "Forecasting Other Meteorological Phenomena (222)" dominates, suggesting prioritization of advanced forecasting techniques. 2. **Observational Focus**: "Radiosonde (64)" and "Data analysis (64)" indicate significant attention to upper-air data collection. 3. **Low Coverage Areas**: "Auxiliary Charts (0)" and "Numerical Weather Prediction (0)" may reflect gaps or emerging topics. 4. **Hierarchical Depth**: Subcategories like "Boundary Layer Analysis Methods (1)" and "Vertical Atmospheric Structure Observation (2)" show niche but critical areas. ### Interpretation The diagram likely represents a curriculum or training framework for meteorologists. High values in forecasting and observational methods (e.g., radiosondes) suggest these are core competencies. Zero values in auxiliary charts and numerical prediction may indicate either incomplete coverage or specialized focus. The structure implies a progression from basic observation (e.g., temperature, pressure) to complex forecasting, with thermodynamic diagrams and front analysis bridging observational data to predictive models. The absence of numerical weather prediction content could signal a theoretical emphasis over computational methods. </details> Figure 30: Hierarchical taxonomy and sample distribution for Part 3. This overview details the structure of Forecasting & Climatology , mapping the dataset samples to specific forecasting theories and climate phenomena. <details> <summary>Image 31 Details</summary>  ### Visual Description ## Hierarchical Diagram: Atmospheric Dynamics (374) ### Overview The diagram represents a structured taxonomy of atmospheric dynamics concepts, organized into hierarchical categories with numerical annotations. Each node includes a label and a parenthetical number, likely indicating counts, priorities, or classifications. ### Components/Axes - **Root Node**: "Atmospheric Dynamics (374)" - **Primary Branches**: 1. Coordinate System (6) 2. Forces Governing Atmospheric Motion (45) 3. Governing Equations (33) 4. Balanced Flows (63) 5. Vertical Motion (13) 6. Circulation and Vorticity (41) 7. Atmospheric Turbulence (11) 8. Vertical Structure of the Planetary Boundary Layer (20) 9. Land-Atmosphere Interaction (1) 10. Interaction with Free Atmosphere (11) 11. Local Winds (0) 12. Mesoscale Convective Systems (1) 13. Typhoons (2) 14. Baroclinic Instability (23) 15. Barotropic Instability (15) 16. Global Scale Motion (52) 17. Atmosphere-Ocean Interaction (5) - **Subcategories**: Each primary branch contains nested subcategories (e.g., "Vertical Coordinate System Transformation (4)" under "Coordinate System"). ### Detailed Analysis #### Coordinate System (6) - Vertical Coordinate System Transformation (4) - Natural Coordinate System (1) - Spherical Coordinate System (1) #### Forces Governing Atmospheric Motion (45) - Apparent Forces (24) - Fundamental Forces (21) - Momentum Equation (20) - Continuity Equation (8) - Thermodynamic Energy Equation (5) #### Governing Equations (33) - Momentum Equation (20) - Continuity Equation (8) - Thermodynamic Energy Equation (5) #### Balanced Flows (63) - Geostrophic Wind (28) - Gradient Wind (22) - Inertial Wind (8) - Thermal Wind Balance (32) - Vertical Motion (13) - Vorticity Equation (32) - Potential Vorticity (9) #### Vertical Motion (13) #### Circulation and Vorticity (41) - Vorticity Equation (32) - Potential Vorticity (9) #### Atmospheric Turbulence (11) - Characteristics of Turbulence (10) - Turbulent Energy and Energy Sources (1) #### Vertical Structure of the Planetary Boundary Layer (20) - Wind Distribution (11) - Mixed Layer (9) #### Land-Atmosphere Interaction (1) - Energy Budget at the Land Surface (1) #### Interaction with Free Atmosphere (11) - Ekman Layer (8) - Secondary Circulation (2) - Ekman Pumping (1) #### Local Winds (0) - Land and Sea Breezes (0) #### Mesoscale Convective Systems (1) - Convective Clouds (1) #### Typhoons (2) - Genesis (1) - Structural Characteristics (1) #### Baroclinic Instability (23) - Structural Characteristics (4) #### Barotropic Instability (15) - Structural Characteristics (4) #### Global Scale Motion (52) - Meridionally Averaged Motion (5) - Zonally Averaged Motion (3) - Energy Transport (3) #### Atmosphere-Ocean Interaction (5) - El Niño (1) - La Niña (1) ### Key Observations 1. **Hierarchical Depth**: Subcategories are nested under primary branches, with numbers decreasing as specificity increases (e.g., "Geostrophic Wind (28)" under "Balanced Flows (63)"). 2. **Numerical Annotations**: Numbers may represent conceptual groupings, study frequencies, or classification codes. For example, "Balanced Flows (63)" has the highest subcategory count. 3. **Zero-Count Nodes**: "Local Winds (0)" and "Land and Sea Breezes (0)" suggest these topics are either excluded or underrepresented in this framework. ### Interpretation The diagram illustrates a comprehensive framework for atmospheric dynamics, emphasizing: - **Scale Hierarchy**: From global processes (e.g., El Niño) to localized phenomena (e.g., convective clouds). - **Interconnected Systems**: Links between forces, equations, and motion types (e.g., vorticity equations under both "Balanced Flows" and "Circulation and Vorticity"). - **Research Focus**: Higher numbers in categories like "Balanced Flows" and "Forces Governing Atmospheric Motion" suggest these are central to atmospheric science. - **Gaps**: Zero counts for local winds and land-sea breezes may indicate omitted subtopics or a focus on larger-scale dynamics. This structure likely serves as a reference for organizing research, educational content, or modeling priorities in atmospheric science. </details> Figure 31: Hierarchical taxonomy and sample distribution for Parts 4 and 5. The diagram covers Applied Meteorology and Weather Chart Analysis , illustrating the coverage of practical applications and legal regulations. <details> <summary>Image 32 Details</summary>  ### Visual Description ## Hierarchical Diagram: Climatology and Atmospheric Physics Syllabus Structure ### Overview The image depicts a hierarchical diagram organizing topics in climatology and atmospheric physics, likely representing sections or chapters in a textbook or course syllabus. The structure is divided into two main branches: **Climatology** (left) and **Atmospheric Physics** (right), each containing subcategories with numerical values in parentheses, possibly indicating page numbers, section IDs, or topic weights. --- ### Components/Axes - **Main Branches**: - **Climatology** (376 total subtopics) - **Atmospheric Physics** (334 total subtopics) - **Subcategories**: - Each main branch contains 3–5 hierarchical levels of subtopics, with numerical values in parentheses (e.g., "Definition of Climate and Climate Systems (13)"). --- ### Detailed Analysis #### Climatology (376) 1. **Physical Climatology (311)** - Definition of Climate and Climate Systems (13) - Climatic Elements and Factors (95) - Air Masses (31) - Distribution (19) - Energy Cycle (35) - Interactions (4) - Water Cycle (4) - Understanding the Climate System (44) - Spatiotemporal Characteristics (45) - Spatiotemporal Characteristics of Pressure (3) - Regional and Local Climate (114) - Climate of Korea (32) - Local Climate Factors (16) - Past Climate (14) - Paleoclimatology (7) - Climate of the Recent Era (4) - Recent Climate (3) - Climate Variability and Change (52) - Climate Variability (15) - Variability across Various Time Scales (12) - External Factors of Climate Variability (3) - Climate Change (23) - Factors of Climate Change (20) - Impact of Climate Change (3) - Applied Climatology (13) - Climate and Life (13) - Climate and Health (12) - Climate and Industrial Activities (1) 2. **Climate and Life (13)** - Climate and Health (12) - Climate and Industrial Activities (1) #### Atmospheric Physics (334) 1. **Atmospheric Hydrostatics (191)** - Basic Concepts (131) - Equilibrium (119) - Various Temperatures (7) - Equation of State (3) - Thermal Gradient (32) - Altimetry (16) - Special Atmospheres (11) - Atmospheric Hydrostatics (60) - First Law of Thermodynamics (11) - Adiabatic Processes and Potential Temperature (9) - Latent Heat (2) - Second Law of Thermodynamics (0) - Entropy (0) - Humidity Variables (9) 2. **Atmospheric Thermodynamics (47)** - Water-Air Systems (16) - Phase Changes in the Atmosphere (4) - Water Vapor and Moist Air (3) - Thermodynamic Diagrams (16) - Various Thermodynamic Diagrams (16) - Adiabatic Isobaric Process (2) - Vertical Mixing (2) - Radiation and Earth’s Heat Budget (5) - Solar and Terrestrial Radiation (4) - Radiative Transfer in the Atmosphere (1) - Cloud Physics (91) - Development of Clouds (75) - Precipitation Processes (16) --- ### Key Observations - **Hierarchical Depth**: Subtopics are nested 3–4 levels deep, indicating detailed segmentation (e.g., "Climate Classification" under Physical Climatology). - **Numerical Values**: Parenthetical numbers likely represent page counts, section IDs, or topic prominence (e.g., "Climate Change (23)" suggests 23 pages or sections). - **Asymmetry**: Climatology has more subtopics (376 vs. 334), reflecting broader coverage in climate-related fields. - **Interdisciplinary Links**: Topics like "Climate and Health" (12) and "Radiative Transfer" (1) highlight connections between disciplines. --- ### Interpretation The diagram illustrates a comprehensive curriculum or textbook structure, emphasizing: 1. **Climatology’s Breadth**: Focus on variability, change, and applied aspects (e.g., industrial impacts). 2. **Atmospheric Physics’ Technicality**: Detailed coverage of thermodynamics, radiation, and cloud processes. 3. **Quantitative Segmentation**: Numerical values suggest a methodical approach to topic allocation, possibly for educational planning or content organization. The hierarchy implies a logical progression from foundational concepts (e.g., "Basic Concepts" in Atmospheric Hydrostatics) to specialized applications (e.g., "Climate and Industrial Activities"). The absence of a legend or color-coding simplifies interpretation but limits visual differentiation between subtopics. </details> Figure 32: Ranking stability analysis on the Multimodal, Geo-cultural, and Reasoning subset performances with bootstrap confidence intervals. Error bars denote 95% CIs derived from item-level bootstrap resampling on 10 evenly spaced models. Even for the smaller subsets, the uncertainty is explicitly quantified and the coarse separation between higher- and lower-performing systems remains visible across panels. <details> <summary>Image 33 Details</summary>  ### Visual Description ## Bar Charts: AI Model Performance Comparison ### Overview The image contains three bar charts comparing the performance of various AI models across three metrics: **Accuracy**, **Accuracy** (possibly a different task or metric), and **Normalized Score**. Each chart uses distinct colors (blue, yellow, red) for data points, with error bars indicating variability. The y-axis lists AI models, and the x-axis represents the metric values (0.0 to 1.0). --- ### Components/Axes - **X-Axes**: - **Chart 1 (Accuracy)**: Labeled "Accuracy" with a scale from 0.0 to 1.0. - **Chart 2 (Accuracy)**: Labeled "Accuracy" with a scale from 0.0 to 1.0. - **Chart 3 (Normalized Score)**: Labeled "Normalized Score" with a scale from 0.0 to 1.0. - **Y-Axes**: Lists AI models (e.g., "gemini-3-pro-preview", "Qwen3-VL-32B-Think", "Llama-3.2-90B-V-Instruct"). - **Legend**: Positioned on the right, but no explicit legend labels are visible in the image. Colors are assumed to correspond to the three charts (blue, yellow, red). --- ### Detailed Analysis #### Chart 1 (Accuracy, Blue) - **Models**: - **gemini-3-pro-preview**: ~0.95 (error ±0.02) - **Qwen3-VL-32B-Think**: ~0.88 (error ±0.03) - **gpt-5.2_multimodal**: ~0.85 (error ±0.04) - **Qwen3-VL-4B-Instruct**: ~0.82 (error ±0.03) - **Qwen3-VL-30B-A3B-Think**: ~0.78 (error ±0.04) - **Qwen3-VL-8B-Instruct**: ~0.75 (error ±0.03) - **A.X.4.0-VL-Light**: ~0.72 (error ±0.04) - **Qwen3-VL-32B-Instruct**: ~0.70 (error ±0.03) - **Qwen2.5-VL-32B-Instruct**: ~0.68 (error ±0.04) - **InternVL3_5-14B-Instruct**: ~0.65 (error ±0.03) - **InternVL3_5-8B-Instruct**: ~0.63 (error ±0.04) - **Llama-3.2-90B-V-Instruct**: ~0.60 (error ±0.03) - **HyperCLOVAX-SEED-V-Instruct-3B**: ~0.58 (error ±0.04) - **VARCO-V-2.0-1.7B**: ~0.55 (error ±0.03) #### Chart 2 (Accuracy, Yellow) - **Models**: - **gemini-3-pro-preview**: ~0.92 (error ±0.02) - **command-a-reasoning-08-2025**: ~0.89 (error ±0.03) - **Qwen3-VL-30B-A3B-Think-2507**: ~0.87 (error ±0.04) - **gpt-oss-120b**: ~0.85 (error ±0.03) - **Qwen3-VL-32B-Think**: ~0.83 (error ±0.04) - **Qwen3-30B-A3B-Instruct-2507**: ~0.81 (error ±0.03) - **Qwen3-VL-30B-A3B-Instruct**: ~0.79 (error ±0.04) - **Llama-3.2-90B-V-Instruct**: ~0.77 (error ±0.03) - **Qwen3-VL-4B-Instruct**: ~0.75 (error ±0.03) - **EXAONE-4.0-1.2B**: ~0.73 (error ±0.04) - **Qwen2.5-VL-3B-Instruct**: ~0.71 (error ±0.03) - **VARCO-V-2.0-1.7B**: ~0.69 (error ±0.04) - **InternVL3_5-1B-Instruct**: ~0.67 (error ±0.03) - **InternVL3_5-2B-Instruct**: ~0.65 (error ±0.04) - **Llama-3.2-1B-Instruct**: ~0.63 (error ±0.03) #### Chart 3 (Normalized Score, Red) - **Models**: - **gemini-3-pro-preview**: ~0.98 (error ±0.01) - **Qwen3-VL-32B-Think**: ~0.95 (error ±0.02) - **Qwen3-VL-235B-A22B-Instruct**: ~0.93 (error ±0.03) - **Qwen3-30B-A3B-Instruct-2507**: ~0.91 (error ±0.02) - **EXAONE-4.0-32B**: ~0.89 (error ±0.03) - **Qwen3-VL-8B-Instruct**: ~0.87 (error ±0.02) - **A.X.4.0-Light**: ~0.85 (error ±0.03) - **Llama-3.1-70B-Instruct**: ~0.83 (error ±0.02) - **Llama-3.2-90B-V-Instruct**: ~0.81 (error ±0.03) - **HyperCLOVAX-SEED-V-Instruct-3B**: ~0.79 (error ±0.02) - **InternVL3_5-8B-Instruct**: ~0.77 (error ±0.03) - **Llama-3.2-3B-Instruct**: ~0.75 (error ±0.02) - **InternVL3_5-5B-Instruct**: ~0.73 (error ±0.03) - **Llama-3.2-1B-Instruct**: ~0.71 (error ±0.02) --- ### Key Observations 1. **High-Performing Models**: - **gemini-3-pro-preview** consistently achieves the highest scores across all charts (Accuracy: ~0.95, Normalized Score: ~0.98). - **Qwen3-VL-32B-Think** and **Qwen3-VL-235B-A22B-Instruct** show strong performance in Accuracy and Normalized Score. 2. **Low-Performing Models**: - **VARCO-V-2.0-1.7B** (Accuracy: ~0.55) and **Llama-3.2-1B-Instruct** (Normalized Score: ~0.71) are among the lowest. 3. **Error Bars**: - Models with larger error bars (e.g., **gpt-5.2_multimodal**, **Llama-3.2-1B-Instruct**) exhibit higher variability in performance. --- ### Interpretation The data suggests that **larger models** (e.g., Qwen3-VL-32B, Llama-3.2-90B) generally outperform smaller ones, but performance varies by task. The **Normalized Score** chart indicates that some models (e.g., **gemini-3-pro-preview**) achieve near-perfect scores, while others (e.g., **VARCO-V-2.0-1.7B**) lag significantly. The error bars highlight the importance of considering variability when comparing models. The use of distinct colors for each chart may imply different evaluation criteria or tasks, though the legend is not explicitly labeled. This could affect cross-chart comparisons. </details> ## H Robustness of Conclusions Under Small Subsets To address the concern that small subsets may induce high variance and unstable conclusions, we conducted item-level bootstrap resampling and sensitivity analyses using the evaluation logs. We confirm via rigorous sensitivity diagnostics-including Bootstrap, leave-one-out ( LOO ), and Approximate Maximum Influence Perturbation ( AMIP ) ( Huang et al. , 2026 ) -that the modality, geo-cultural, and reasoning gaps are robust systemic trends. Bootstrap estimates validate these patterns, LOO perturbations reveal no sign flips, and AMIP analysis demonstrates that the local advantage withstands even substantial adversarial data removal. ## H.1 Statistical Robustness Diagnostics We report explicit uncertainty ( confidence intervals ) and stability diagnostics, including LOO perturbations and bootstrap rank intervals. Unless otherwise noted, all statistics below are computed from the current evaluation run with 1,000 bootstrap iterations and a fixed random seed of 42. We confirmed that the bootstrapping process achieved sufficient convergence ( not shown ). Setup. We focus our robustness checks on three subsets: multimodal ( 82 items ), Korean-specific ( 71 text-only and 73 multimodal items ), and reasoning ( 121 text-only and 141 multimodal items ). We normalize all scores to [0 , 1] , mapping reasoning scores from [4 , 20] and accuracy from [0 , 100] . For the bootstrap analysis, we select 10 representative models for each subset by sorting them by performance and sampling at equal intervals. Subset-level uncertainty is quantified, but does not erase structure. Figure 32 reports 95% bootstrap confidence intervals for 10 representative models selected at equal intervals from the performance ranking for each subset. This sampling strategy ensures visibility across the full performance spectrum. Making the uncertainty induced by small n explicit, we observe that while confidence intervals naturally widen for smaller subsets, the overall performance hierarchy remains robust. The highest-performing systems consistently maintain their lead with the following estimates ( mean [ 95% CI ]): Multimodal 0 . 756 [0 . 659 , 0 . 841] , Geo-cultural 0 . 789 [0 . 690 , 0 . 873] , and Reasoning 0 . 876 [0 . 848 , 0 . 901] ( normalized score ). This confirms that even under resampling, the conclusions are not dominated by random sampling variation, and the uncertainty bounds provide a principled way to interpret rank differences. Key performance gaps are stable under resampling and single-item perturbations. As shown in Table 14, our analysis confirms that the identified performance gaps across modality, geocultural, and reasoning dimensions are systemic and robust, rather than artifacts of specific outliers. First, regarding the modality gap (∆ Modality = AccMultimodal -AccNon-Multimodal ) , we observe a consistently negative trend across the 25 directly comparable models, ranging from -37 . 39 % to -2 . 28 %. For 19/25 models, the 95% bootstrap CI strictly excludes zero. Second, we examine the geo-cultural gap (∆ Geo-Cultural = AccKorean -AccNon-Korean ) and the reasoning gap (∆ Reasoning = ScoreReasoning -AccTotal ) ( normalized score and accuracy ). We find that representative local mod- <details> <summary>Image 34 Details</summary>  ### Visual Description Icon/Small Image (16x19) </details> <details> <summary>Image 35 Details</summary>  ### Visual Description Icon/Small Image (17x16) </details> <details> <summary>Image 36 Details</summary>  ### Visual Description Icon/Small Image (18x18) </details> <details> <summary>Image 37 Details</summary>  ### Visual Description Icon/Small Image (15x19) </details> <details> <summary>Image 38 Details</summary>  ### Visual Description Icon/Small Image (15x18) </details> <details> <summary>Image 39 Details</summary>  ### Visual Description Icon/Small Image (17x20) </details> <details> <summary>Image 40 Details</summary>  ### Visual Description Icon/Small Image (15x15) </details> <details> <summary>Image 41 Details</summary>  ### Visual Description Icon/Small Image (16x14) </details> Table 14: Leave-one-out ( LOO ) sensitivity analysis across Multimodal, Geo-cultural, and Reasoning subsets. The table reports representative models with baseline gaps closest to zero ( most prone to sign flips ). No model exhibits a sign reversal under single-item removal across all tasks. ( Baseline ∆ : original gap on the full set; Max swing : maximum deviation from the baseline across LOO iterations; Sign flip : percentage of iterations where the gap sign reverses; n LOO: total number of items subject to LOO perturbation. ) | Model | Baseline ∆ | Max swing | Sign flip ( % ) | n LOO | |------------------------------------------------|--------------|-------------|-------------------|---------| | ( A ) Modality Gap ( Multimodal - Text-only ) | | | | | | Qwen/Qwen3-VL-4B-Instruct | - 2 . 28 % | 0 . 63 % | 0.0 | 82 | | OpenGVLab/InternVL3_5-2B-Instruct | - 4 . 38 % | 0 . 90 % | 0.0 | 82 | | HyperCLOVAX-SEED-Vision-Instruct-3B | - 9 . 22 % | 0 . 95 % | 0.0 | 82 | | skt/A.X-4.0-VL-Light | - 10 . 33 % | 0 . 71 % | 0.0 | 82 | | Qwen/Qwen3-VL-8B-Instruct | - 10 . 35 % | 0 . 69 % | 0.0 | 82 | | ( B ) Geo-cultural Gap ( Korean - Non-Korean ) | | | | | | HyperCLOVAX-SEED-Text-Instruct-1.5B | 6 . 27 % | 0 . 91 % | 0.0 | 71 | | OpenGVLab/InternVL3_5-1B-Instruct | 5 . 13 % | 0 . 99 % | 0.0 | 73 | | LGAI-EXAONE/EXAONE-4.0-1.2B | 5 . 12 % | 0 . 82 % | 0.0 | 71 | | skt/A.X-4.0-Light | 5 . 04 % | 0 . 87 % | 0.0 | 71 | | HyperCLOVAX-SEED-Vision-Instruct-3B | 3 . 81 % | 0 . 89 % | 0.0 | 73 | | ( C ) Reasoning Gap ( Reasoning - Knowledge ) | | | | | | Qwen/Qwen3-32B ( Thinking ) | 18 . 26 % | 0 . 55 % | 0.0 | 121 | | gpt-5.2 ( Thinking ) | 6 . 27 % | 0 . 61 % | 0.0 | 141 | | Qwen/Qwen3-30B-A3B-Instruct-2507 | 2 . 50 % | 0 . 56 % | 0.0 | 121 | | Qwen/Qwen3-4B-Instruct-2507 | 0 . 65 % | 0 . 43 % | 0.0 | 121 | | Qwen/Qwen3-VL-32B-Instruct | 0 . 51 % | 0 . 72 % | 0.0 | 141 | els ( e.g., HyperCLOVAX , skt/A.X ) maintain a stable positive advantage in the geo-cultural domain, while the reasoning performance remains distinct from general knowledge across models. To quantify the sensitivity of these conclusions to individual outliers, we run leave-one-out perturbations over the respective subsets: multimodal ( n = 82 ), geo-cultural ( n = 71/73 ), and reasoning ( n = 121/141 ) for text-only and multimodal models, respectively. Crucially, removing any single item never triggers a sign flip across all evaluated models and dimensions ( sign-flip rate = 0 ). Table 14 reports the most fragile cases ( i.e., baselines closest to zero ); even for these edge cases, the maximum swing remains negligible ( e.g., < 1.16% for modality, < 0.99% for geo-cultural, and < 0.72% for reasoning ). This confirms that the observed gaps drive the overarching trends and are not attributable to a handful of influential questions. ## H.2 Robustness of Key Findings to Critical Data Perturbation Resilience to Adversarial Item Removal. To further assess whether the geo-cultural gap could be driven by a small number of influential questions, we conduct an adversarial influential- item deletion analysis inspired by recent robust evaluation methodologies, AMIP ( Huang et al. , 2026 ) . Concretely, we focus on the representative model pair highlighted in Section 5 ( The GeoCultural Gap ): the top-performing local model ( skt/A.X-4.0 ) versus the global baseline ( Qwen/ Qwen3-VL-235B-A22B-Thinking ). We test how many strategically selected items an adversary would need to delete to reverse the ordering where the local model outperforms the global one. We fit a Bradley-Terry ( BT ) model to pairwise win/loss outcomes on the Geo-cultural subset and compute per-item influence scores using the inverse Hessian of the BT loss. Using a greedy adversarial removal procedure ( a standard approximation for AMIP ) that sequentially deletes the most influential items favoring the local model, we find that the ordering flips only after removing n AMIP = 18 items, corresponding to 18/73 ≈ 24 . 7 %of the entire Geo-cultural subset. This indicates that the observed local model advantage is not attributable to a single outlier or a very small number of questions, but instead requires removing a substantial fraction of the subset to overturn.