## Box Plot: Final Score Stability: AEMA vs single LLM-as-a-Judge (30 runs) ### Overview The image is a box plot comparing the final score stability of two evaluators: AEMA (represented in blue) and a single LLM-as-a-Judge (represented in orange). The plot shows the distribution of final scores (ranging from 0 to 1) for each evaluator across 30 runs. The plot includes outliers represented as dots. ### Components/Axes * **Title:** Final score stability: AEMA vs single LLM-as-a-Judge (30 runs) * **X-axis:** Evaluator * **Y-axis:** Final score (0-1). The y-axis ranges from 0.65 to 1, with tick marks at 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, and 1. * **Legend:** Located at the bottom of the chart. * Blue square: AEMA * Orange square: Single LLM-as-a-Judge ### Detailed Analysis or Content Details * **AEMA (Blue):** * Box: The box extends from approximately 0.96 to 0.99. * Median (represented by 'x' inside the box): Approximately 0.975. * Outlier: A single outlier is present at approximately 0.91. * **Single LLM-as-a-Judge (Orange):** * Box: The box extends from approximately 0.88 to 0.93. * Whiskers: The whiskers extend to approximately 0.87 and 0.94. * Median (represented by 'x' inside the box): Approximately 0.89. * Outlier: A single outlier is present at approximately 0.70. ### Key Observations * The AEMA evaluator has a higher median final score and a tighter distribution (smaller box) compared to the single LLM-as-a-Judge. * Both evaluators have one outlier each, with the single LLM-as-a-Judge outlier being significantly lower than the AEMA outlier. * The AEMA scores are generally more stable and higher than the Single LLM-as-a-Judge scores. ### Interpretation The box plot suggests that the AEMA evaluator demonstrates greater stability and achieves higher final scores compared to the single LLM-as-a-Judge, based on 30 runs. The smaller box for AEMA indicates less variability in its scores. The presence of outliers for both evaluators suggests occasional instances of significantly lower performance, but the single LLM-as-a-Judge has a more extreme low score. This could indicate that the AEMA method is more robust and consistent in its evaluations.

\n ## Box Plot: Final Score Stability - AEMA vs Single LLM-as-a-Judge (30 Runs) ### Overview The image presents a box plot comparing the final score stability of AEMA (Automated Evaluation Metric Aggregation) and a single LLM-as-a-Judge across 30 runs. The y-axis represents the final score, scaled from 0 to 1, while the x-axis indicates the evaluator type. ### Components/Axes * **Title:** "Final score stability: AEMA vs single LLM-as-a-Judge (30 runs)" - positioned at the top-center. * **X-axis Label:** "Evaluator" - positioned at the bottom-center. * **Y-axis Label:** "Final score (0-1)" - positioned vertically along the left side. * **Legend:** Located at the bottom-center, with two entries: * Blue box: "AEMA" * Orange box: "Single LLM-as-a-Judge" * **Data Series:** Two box plots representing the distribution of final scores for each evaluator. * **Outliers:** Individual data points plotted as circles outside the box plots. ### Detailed Analysis The box plots display the following characteristics: **AEMA (Blue Box):** * The box plot is positioned on the left side of the chart. * The box extends from approximately 0.92 to 0.97. * The median (represented by the 'X' inside the box) is around 0.96. * The interquartile range (IQR) is approximately 0.05. * There is one outlier at approximately 0.91. **Single LLM-as-a-Judge (Orange Box):** * The box plot is positioned on the right side of the chart. * The box extends from approximately 0.88 to 0.94. * The median (represented by the 'X' inside the box) is around 0.90. * The IQR is approximately 0.06. * There is one outlier at approximately 0.71. ### Key Observations * AEMA demonstrates a slightly higher median final score (0.96) compared to the Single LLM-as-a-Judge (0.90). * The AEMA scores appear to be more tightly clustered, as indicated by the smaller IQR (0.05) compared to the LLM-as-a-Judge (0.06). * The LLM-as-a-Judge has a lower outlier (0.71) than AEMA (0.91), suggesting greater variability in its scores. ### Interpretation The data suggests that AEMA provides more stable and consistently high final scores compared to using a single LLM-as-a-Judge. The narrower IQR for AEMA indicates less score variation, implying a more reliable evaluation process. The presence of an outlier at 0.71 for the LLM-as-a-Judge suggests that, in some instances, its evaluation can deviate significantly from the typical range. This could be due to the inherent variability in LLM responses or sensitivity to specific input characteristics. The higher median score for AEMA suggests it may be better calibrated to provide accurate and consistent evaluations. The difference in stability could be attributed to AEMA's aggregation of multiple metrics, which mitigates the impact of individual metric fluctuations.

## Box Plot: Final Score Stability Comparison ### Overview This image is a box plot comparing the stability of final scores (on a 0-1 scale) between two evaluation methods, "AEMA" and "Single LLM-as-a-Judge," based on 30 runs. The chart visually demonstrates the distribution, central tendency, and variability of scores for each method. ### Components/Axes * **Chart Title:** "Final score stability: AEMA vs single LLM-as-a-Judge (30 runs)" * **Y-Axis:** * **Label:** "Final score (0-1)" * **Scale:** Linear scale from 0.65 to 1.0, with major gridlines at intervals of 0.05 (0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1). * **X-Axis:** * **Label:** "Evaluator" * **Categories:** Two categories are plotted: "AEMA" (left) and "Single LLM-as-a-Judge" (right). * **Legend:** Located at the bottom center of the chart. * A blue square is labeled "AEMA". * An orange square is labeled "Single LLM-as-a-Judge". * **Data Series:** Two box plots, one for each evaluator, colored according to the legend. ### Detailed Analysis **1. AEMA (Blue Box Plot - Left)** * **Visual Trend:** The distribution is extremely compressed and located at the very top of the score range, indicating high consistency and high scores. * **Components & Approximate Values:** * **Median (Horizontal Line within Box):** Approximately 0.97. * **Interquartile Range (IQR - The Box):** The box is very narrow. The lower quartile (Q1, bottom of the box) is approximately 0.96. The upper quartile (Q3, top of the box) is approximately 0.98. * **Whiskers:** The whiskers are not visibly extended beyond the box, suggesting the minimum and maximum values (excluding outliers) are very close to the quartiles. * **Outliers:** One outlier point is visible below the box, at approximately 0.91. * **Mean (Marked with an 'x'):** The 'x' is positioned within the box, near the median, at approximately 0.97. **2. Single LLM-as-a-Judge (Orange Box Plot - Right)** * **Visual Trend:** The distribution shows significantly more spread (variability) than AEMA, with scores centered lower on the scale. The box is wider, and whiskers are longer. * **Components & Approximate Values:** * **Median (Horizontal Line within Box):** Approximately 0.90. * **Interquartile Range (IQR - The Box):** The box spans from a lower quartile (Q1) of approximately 0.88 to an upper quartile (Q3) of approximately 0.93. * **Whiskers:** The upper whisker extends to a maximum value (excluding outliers) of approximately 0.94. The lower whisker extends to a minimum value (excluding outliers) of approximately 0.87. * **Outliers:** One outlier point is visible far below the lower whisker, at approximately 0.70. * **Mean (Marked with an 'x'):** The 'x' is positioned slightly below the median line, at approximately 0.895. ### Key Observations 1. **Stability Contrast:** AEMA demonstrates vastly superior score stability. Its IQR (approx. 0.02) is an order of magnitude smaller than that of the Single LLM-as-a-Judge (approx. 0.05). 2. **Performance Level:** AEMA not only is more stable but also achieves a higher central tendency (median ~0.97 vs. ~0.90). 3. **Outliers:** Both methods produced outliers. AEMA's single outlier (~0.91) is still a high score, merely falling outside its own tight cluster. The Single LLM-as-a-Judge produced a severe low outlier (~0.70), indicating a run with significantly degraded performance. 4. **Distribution Shape:** The Single LLM-as-a-Judge's median is closer to its Q1 than its Q3, suggesting a slight positive skew (scores are more concentrated toward the lower end of its IQR). ### Interpretation This chart provides strong evidence that the **AEMA evaluation method is significantly more reliable and consistent** than using a Single LLM-as-a-Judge. The tight clustering of AEMA scores near the maximum value (1.0) suggests it produces highly reproducible results across multiple runs. In contrast, the Single LLM-as-a-Judge exhibits substantial volatility, with scores varying widely and the potential for catastrophic failure (as evidenced by the outlier at 0.70). For practical application, this implies that relying on a single LLM run for evaluation introduces unacceptable noise and risk. AEMA, by presumably aggregating or stabilizing judgments, mitigates this instability, leading to more trustworthy and actionable scoring. The data argues for the necessity of methods like AEMA in any rigorous evaluation pipeline where consistency is critical.

## Box Plot: Final score stability: AEMA vs single LLM-as-a-Judge (30 runs) ### Overview The image presents a comparative box plot analysis of final score stability between two evaluators: AEMA and Single LLM-as-a-Judge, based on 30 runs. The y-axis represents final scores on a 0–1 scale, while the x-axis categorizes results by evaluator type. ### Components/Axes - **X-axis (Evaluator)**: - Categories: "AEMA" (blue) and "Single LLM-as-a-Judge" (orange) - Legend: Located at the bottom, explicitly mapping colors to evaluators - **Y-axis (Final score)**: - Scale: 0.65 to 1.0 in increments of 0.05 - Label: "Final score (0–1)" - **Plot Elements**: - Box plots for both evaluators - Median lines (marked with "X") - Whiskers indicating interquartile ranges - Outlier markers (dots) ### Detailed Analysis 1. **AEMA (Blue)**: - Median score: ~0.95 (marked by "X") - Interquartile range: ~0.92 to 0.98 - Outlier: Single data point at ~0.91 (below the lower whisker) - Distribution: Tight clustering around the median 2. **Single LLM-as-a-Judge (Orange)**: - Median score: ~0.89 (marked by "X") - Interquartile range: ~0.87 to 0.93 - Outlier: Single data point at ~0.70 (below the lower whisker) - Distribution: Wider spread compared to AEMA ### Key Observations - AEMA demonstrates higher median scores (~0.95 vs. ~0.89) and tighter score distribution. - Single LLM-as-a-Judge shows greater variability, with a lower median and a notable outlier at 0.70. - Both evaluators exhibit outliers, but the Single LLM-as-a-Judge's outlier is more extreme relative to its distribution. ### Interpretation The data suggests AEMA provides more stable and higher-performing evaluations compared to the Single LLM-as-a-Judge approach. The tighter interquartile range for AEMA indicates consistency, while the Single LLM-as-a-Judge's wider spread and lower median highlight potential instability. The outlier at 0.70 for the Single LLM-as-a-Judge may represent an anomalous run, though its cause (e.g., data quality, model configuration) is not specified. These results imply AEMA could be preferable for applications requiring reliable score stability, whereas the Single LLM-as-a-Judge might require further refinement or validation.