## Chart: Sensitivity to Top-K ### Overview The image is a line chart showing the sensitivity of four different metrics (Perplexity, LN-Entropy, Lexical Similarity, and EigenScore) to the "Top-K" parameter. The y-axis represents AUROC (Area Under the Receiver Operating Characteristic curve), a measure of classification performance. The x-axis represents the Top-K value, which ranges from 3 to 50. ### Components/Axes * **Title:** Sensitivity to Top-K * **X-axis:** Top-K, with values 3, 5, 10, 20, 30, and 50. * **Y-axis:** AUROC, ranging from 40 to 90. * **Legend:** Located in the bottom-right of the chart. * Blue with "x" markers: Perplexity * Gray with diamond markers: LN-Entropy * Teal with circle markers: Lexical Similarity * Orange with star markers: EigenScore ### Detailed Analysis * **Perplexity (Blue):** The line is relatively flat, with AUROC values consistently around 64-65. * Top-K = 3: AUROC ≈ 64 * Top-K = 5: AUROC ≈ 64 * Top-K = 10: AUROC ≈ 64 * Top-K = 20: AUROC ≈ 64 * Top-K = 30: AUROC ≈ 64 * Top-K = 50: AUROC ≈ 64 * **LN-Entropy (Gray):** The line shows a slight upward trend, with AUROC values increasing from approximately 67 to 69. * Top-K = 3: AUROC ≈ 67 * Top-K = 5: AUROC ≈ 67 * Top-K = 10: AUROC ≈ 68 * Top-K = 20: AUROC ≈ 68 * Top-K = 30: AUROC ≈ 69 * Top-K = 50: AUROC ≈ 69 * **Lexical Similarity (Teal):** The line is relatively flat, with a slight upward trend, with AUROC values ranging from approximately 74 to 76. * Top-K = 3: AUROC ≈ 74 * Top-K = 5: AUROC ≈ 75 * Top-K = 10: AUROC ≈ 75 * Top-K = 20: AUROC ≈ 75 * Top-K = 30: AUROC ≈ 74 * Top-K = 50: AUROC ≈ 76 * **EigenScore (Orange):** The line is relatively flat, with AUROC values consistently around 79-80. * Top-K = 3: AUROC ≈ 79 * Top-K = 5: AUROC ≈ 80 * Top-K = 10: AUROC ≈ 79 * Top-K = 20: AUROC ≈ 79 * Top-K = 30: AUROC ≈ 80 * Top-K = 50: AUROC ≈ 80 ### Key Observations * EigenScore consistently achieves the highest AUROC values across all Top-K values. * Perplexity consistently achieves the lowest AUROC values across all Top-K values. * LN-Entropy shows a slight improvement in AUROC as Top-K increases. * Lexical Similarity shows a slight improvement in AUROC as Top-K increases. * The sensitivity to Top-K is relatively low for all four metrics, as the AUROC values do not change drastically with varying Top-K values. ### Interpretation The chart suggests that EigenScore is the most effective metric for the task being evaluated, as it consistently achieves the highest AUROC values. Perplexity, on the other hand, appears to be the least effective. The relatively flat lines for all metrics indicate that the performance is not highly sensitive to the Top-K parameter within the range of 3 to 50. This could mean that the task is relatively robust to the choice of Top-K, or that the optimal Top-K value lies outside this range. The slight upward trend for LN-Entropy and Lexical Similarity suggests that increasing Top-K may lead to marginal improvements in performance for these metrics.

## Line Chart: Sensitivity to Top-K ### Overview This image is a line chart titled "Sensitivity to Top-K" that evaluates the performance of four different metrics—Perplexity, LN-Entropy, Lexical Similarity, and EigenScore—using the AUROC (Area Under the Receiver Operating Characteristic curve) metric. The chart tests how these metrics respond to changes in the "Top-K" parameter, ranging from 3 to 50. The overall visual impression is one of high stability across all tested methods. ### Components/Axes * **Title**: "Sensitivity to Top-K" (Centered at the top). * **Y-Axis**: * **Label**: "AUROC" (Vertical, left side). * **Scale**: Numerical, ranging from 35 to 90. Major tick marks are visible at 40, 50, 60, 70, 80, and 90. * **X-Axis**: * **Label**: "Top-K" (Horizontal, bottom center). * **Scale**: Categorical/Discrete values: 3, 5, 10, 20, 30, 50. * **Legend**: Located in the **bottom-right** quadrant within a white box with a thin grey border. It contains four entries: * **Blue line with 'x' markers**: Perplexity * **Grey line with diamond (♦) markers**: LN-Entropy * **Teal/Cyan line with circle (●) markers**: Lexical Similarity * **Orange line with star (★) markers**: EigenScore * **Line Style**: All data series use a dash-dot line pattern. ### Content Details #### Data Table (Approximate Values) The following table reconstructs the data points based on visual alignment with the Y-axis scale. Values are estimated with an uncertainty of approximately ±0.5 units. | Top-K | Perplexity (Blue x) | LN-Entropy (Grey ♦) | Lexical Similarity (Teal ●) | EigenScore (Orange ★) | | :--- | :---: | :---: | :---: | :---: | | **3** | ~64.0 | ~67.0 | ~74.0 | ~79.0 | | **5** | ~64.0 | ~67.5 | ~75.0 | ~80.5 | | **10** | ~64.0 | ~68.5 | ~75.0 | ~79.0 | | **20** | ~64.0 | ~68.0 | ~73.5 | ~80.0 | | **30** | ~64.0 | ~68.5 | ~74.0 | ~80.5 | | **50** | ~64.0 | ~69.0 | ~76.0 | ~80.0 | #### Trend Verification * **EigenScore (Orange)**: Positioned at the top of the chart. The line is relatively flat with minor fluctuations, maintaining a high AUROC around 80. * **Lexical Similarity (Teal)**: Positioned second from the top. It shows slight volatility, with a small dip at K=20 and a peak at K=50, generally staying between 73 and 76. * **LN-Entropy (Grey)**: Positioned third from the top. It exhibits a very slight upward trend as K increases, moving from ~67 to ~69. * **Perplexity (Blue)**: Positioned at the bottom. The line is perfectly horizontal, indicating zero sensitivity to the Top-K parameter within this range. ### Key Observations * **Performance Ranking**: There is a clear and consistent hierarchy across all values of K: EigenScore > Lexical Similarity > LN-Entropy > Perplexity. * **Parameter Robustness**: All four methods demonstrate remarkable stability. The AUROC scores do not significantly degrade or improve as the Top-K value increases from 3 to 50. * **Perplexity Invariance**: The Perplexity metric appears completely unaffected by the Top-K setting, suggesting its calculation might be independent of this specific parameter in the context of this experiment. ### Interpretation The data suggests that the choice of the "Top-K" hyperparameter is not critical for the performance of these specific metrics in the evaluated task. This is a positive finding for practitioners, as it implies that the models are robust and do not require extensive hyperparameter tuning for the Top-K value to achieve optimal AUROC. **EigenScore** is the superior metric among those tested, consistently outperforming the others by a significant margin (approximately 5 points higher than Lexical Similarity and 15 points higher than Perplexity). The stability of EigenScore at the ~80 AUROC level indicates it is a reliable and high-performing choice regardless of whether a narrow (K=3) or wide (K=50) context is considered. The slight upward trend in **LN-Entropy** suggests it might benefit marginally from larger K values, whereas **Lexical Similarity** shows more noise, possibly due to the inherent variability in lexical overlap as the sample size (K) changes.

## Line Chart: Sensitivity to Top-K ### Overview This image presents a line chart illustrating the sensitivity of four different metrics – Perplexity, LN-Entropy, Lexical Similarity, and EigenScore – to varying values of Top-K. The chart displays how the Area Under the Receiver Operating Characteristic curve (AUROC) changes as the Top-K parameter is adjusted from 3 to 50. ### Components/Axes * **Title:** "Sensitivity to Top-K" (centered at the top) * **X-axis:** "Top-K" with markers at 3, 5, 10, 20, 30, and 50. * **Y-axis:** "AUROC" with a scale ranging from 40 to 90, incrementing by 10. * **Legend:** Located in the bottom-right corner, containing the labels and corresponding colors for each metric: * Perplexity (Dark Blue) * LN-Entropy (Gray) * Lexical Similarity (Teal) * EigenScore (Orange) ### Detailed Analysis The chart contains four distinct lines, each representing one of the metrics. * **Perplexity (Dark Blue):** The line is relatively flat, showing minimal change in AUROC across the range of Top-K values. * At Top-K = 3, AUROC ≈ 68. * At Top-K = 5, AUROC ≈ 67. * At Top-K = 10, AUROC ≈ 67. * At Top-K = 20, AUROC ≈ 66. * At Top-K = 30, AUROC ≈ 66. * At Top-K = 50, AUROC ≈ 65. * **LN-Entropy (Gray):** This line is also relatively flat, with a slight downward trend. * At Top-K = 3, AUROC ≈ 74. * At Top-K = 5, AUROC ≈ 73. * At Top-K = 10, AUROC ≈ 72. * At Top-K = 20, AUROC ≈ 71. * At Top-K = 30, AUROC ≈ 71. * At Top-K = 50, AUROC ≈ 70. * **Lexical Similarity (Teal):** This line is nearly horizontal, indicating a very stable AUROC value. * At Top-K = 3, AUROC ≈ 76. * At Top-K = 5, AUROC ≈ 76. * At Top-K = 10, AUROC ≈ 75. * At Top-K = 20, AUROC ≈ 75. * At Top-K = 30, AUROC ≈ 74. * At Top-K = 50, AUROC ≈ 74. * **EigenScore (Orange):** This line is the most stable, remaining consistently high across all Top-K values. * At Top-K = 3, AUROC ≈ 81. * At Top-K = 5, AUROC ≈ 81. * At Top-K = 10, AUROC ≈ 81. * At Top-K = 20, AUROC ≈ 80. * At Top-K = 30, AUROC ≈ 80. * At Top-K = 50, AUROC ≈ 80. ### Key Observations * EigenScore consistently exhibits the highest AUROC values across all Top-K values. * Perplexity shows the lowest AUROC values and a slight decreasing trend with increasing Top-K. * LN-Entropy and Lexical Similarity demonstrate relatively stable AUROC values, with minor fluctuations. * The overall sensitivity of all metrics to changes in Top-K is limited, suggesting that the performance is not heavily dependent on this parameter within the tested range. ### Interpretation The chart suggests that EigenScore is the most robust metric for evaluating performance, as its AUROC remains consistently high regardless of the Top-K value. Perplexity, on the other hand, appears to be the least reliable metric, with the lowest AUROC and a slight negative correlation with Top-K. The stability of LN-Entropy and Lexical Similarity indicates that they provide consistent, though potentially less informative, performance assessments. The limited sensitivity to Top-K implies that the model's performance is not significantly affected by the number of top candidates considered within the range of 3 to 50. This could be due to the inherent characteristics of the data or the model's architecture. Further investigation might involve exploring Top-K values outside this range to determine if sensitivity increases or decreases. The chart provides valuable insights into the relative strengths and weaknesses of different metrics for evaluating model performance in this specific context.

## Line Chart: Sensitivity to Top-K ### Overview This is a line chart titled "Sensitivity to Top-K" that plots the performance of four different metrics (Perplexity, LN-Entropy, Lexical Similarity, and EigenScore) as a function of the "Top-K" parameter. The performance is measured by the AUROC (Area Under the Receiver Operating Characteristic Curve) score. The chart demonstrates how sensitive each metric's performance is to changes in the Top-K value. ### Components/Axes * **Chart Title:** "Sensitivity to Top-K" (centered at the top). * **Y-Axis:** * **Label:** "AUROC" * **Scale:** Linear, ranging from 40 to 90. * **Major Ticks:** 40, 50, 60, 70, 80, 90. * **X-Axis:** * **Label:** "Top-K" * **Scale:** Appears to be categorical or logarithmic, with discrete values. * **Data Points (Ticks):** 3, 5, 10, 20, 30, 50. * **Legend:** Located in the bottom-right corner of the plot area. It maps line colors and marker styles to metric names: * **Blue line with 'x' markers:** Perplexity * **Gray line with diamond markers:** LN-Entropy * **Teal line with circle markers:** Lexical Similarity * **Orange line with star markers:** EigenScore ### Detailed Analysis The chart displays four data series, each showing a relatively flat trend across the range of Top-K values. 1. **EigenScore (Orange, Star Markers):** * **Trend:** The line is nearly horizontal, showing a very slight upward trend from Top-K=3 to Top-K=50. * **Approximate Values:** * Top-K=3: ~79 * Top-K=5: ~80 * Top-K=10: ~79 * Top-K=20: ~80 * Top-K=30: ~80 * Top-K=50: ~80 2. **Lexical Similarity (Teal, Circle Markers):** * **Trend:** The line is mostly flat with a minor dip around Top-K=20 before recovering. * **Approximate Values:** * Top-K=3: ~74 * Top-K=5: ~75 * Top-K=10: ~75 * Top-K=20: ~73 * Top-K=30: ~74 * Top-K=50: ~76 3. **LN-Entropy (Gray, Diamond Markers):** * **Trend:** The line is very flat, showing minimal variation. * **Approximate Values:** * Top-K=3: ~67 * Top-K=5: ~67 * Top-K=10: ~68 * Top-K=20: ~68 * Top-K=30: ~68 * Top-K=50: ~68 4. **Perplexity (Blue, 'x' Markers):** * **Trend:** The line is almost perfectly horizontal, indicating no sensitivity to Top-K. * **Approximate Values:** * Top-K=3: ~64 * Top-K=5: ~64 * Top-K=10: ~64 * Top-K=20: ~64 * Top-K=30: ~64 * Top-K=50: ~64 ### Key Observations * **Performance Hierarchy:** There is a clear and consistent performance ranking across all Top-K values: EigenScore > Lexical Similarity > LN-Entropy > Perplexity. * **Low Sensitivity:** All four metrics exhibit very low sensitivity to the Top-K parameter within the tested range (3 to 50). The AUROC scores change by only 1-2 points at most. * **Stability:** The Perplexity metric is the most stable, showing virtually no change. EigenScore and LN-Entropy are also highly stable. Lexical Similarity shows the most variation, though it is still minimal. * **Visual Separation:** The lines for the four metrics are distinctly separated and do not intersect, confirming their consistent relative performance. ### Interpretation The data suggests that for the task being evaluated, the choice of Top-K (within the range of 3 to 50) has a negligible impact on the performance of these four evaluation metrics. This is a significant finding, as it implies that model comparisons using these metrics would be robust to the specific choice of the Top-K hyperparameter. The consistent performance hierarchy indicates that **EigenScore** is the most effective metric (highest AUROC) for this particular task, followed by **Lexical Similarity**. **Perplexity**, a common language model metric, performs the worst in this context. This could imply that the task requires evaluation criteria beyond simple next-token prediction likelihood, favoring metrics that capture semantic similarity (Lexical Similarity) or distributional properties (EigenScore, LN-Entropy). The investigation reveals a stable evaluation landscape where the primary differentiator is the choice of metric itself, not the tuning of the Top-K parameter. This allows for more confident and less parameter-sensitive model selection and comparison.

## Line Chart: Sensitivity to Top-K ### Overview The chart illustrates the sensitivity of four evaluation metrics (Perplexity, LN-Entropy, Lexical Similarity, EigenScore) to varying Top-K values (3–50) using Area Under the Receiver Operating Characteristic curve (AUROC) as the performance metric. All metrics exhibit relatively stable performance across Top-K ranges, with EigenScore consistently outperforming others. ### Components/Axes - **X-axis (Top-K)**: Discrete values at 3, 5, 10, 20, 30, 50. - **Y-axis (AUROC)**: Scale from 40 to 90, with increments of 10. - **Legend**: Located in the bottom-right corner, mapping: - Blue crosses (×): Perplexity - Gray diamonds (◆): LN-Entropy - Teal circles (●): Lexical Similarity - Orange stars (★): EigenScore ### Detailed Analysis 1. **Perplexity (Blue ×)**: - Flat line at ~65 AUROC across all Top-K values. - No significant variation observed. 2. **LN-Entropy (Gray ◆)**: - Slight upward trend from ~67 (Top-K=3) to ~69 (Top-K=50). - Minimal fluctuation between intermediate Top-K values. 3. **Lexical Similarity (Teal ●)**: - Stable at ~75 AUROC for Top-K=3–20. - Minor increase to ~76 at Top-K=50. 4. **EigenScore (Orange ★)**: - Consistently highest performance (~80 AUROC) across all Top-K. - Slight dip to ~79 at Top-K=10, then recovery to ~80. ### Key Observations - **EigenScore** maintains the highest AUROC (79–80) regardless of Top-K, indicating robustness. - **Perplexity** is the least sensitive metric, showing no change across Top-K. - **LN-Entropy** exhibits the weakest sensitivity, with a marginal 2-point increase. - **Lexical Similarity** remains stable until Top-K=50, where it marginally improves. ### Interpretation The data suggests that **EigenScore** is the most reliable metric for evaluating model performance across varying Top-K configurations, as it consistently achieves the highest AUROC. **Perplexity** and **Lexical Similarity** demonstrate stability but lower performance, while **LN-Entropy** shows minimal sensitivity. The flat trends imply that Top-K adjustments have limited impact on these metrics, though EigenScore’s slight dip at Top-K=10 warrants further investigation into potential anomalies. This analysis is critical for optimizing model evaluation strategies in natural language processing tasks.