## Line Chart: Sensitivity to Temperature ### Overview The image is a line chart showing the sensitivity of four different metrics (Perplexity, LN-Entropy, Lexical Similarity, and EigenScore) to temperature. The x-axis represents temperature, and the y-axis represents AUROC (Area Under the Receiver Operating Characteristic curve). ### Components/Axes * **Title:** Sensitivity to Temperature * **X-axis:** Temperature, with tick marks at 0.1, 0.3, 0.5, 1, 3, and 5. * **Y-axis:** AUROC, with tick marks at 40, 50, 60, 70, 80, 90, and 100. * **Legend:** Located at the top-right of the chart. * **Blue (dash-dot line with x markers):** Perplexity * **Gray (solid line with diamond markers):** LN-Entropy * **Teal (solid line with circle markers):** Lexical Similarity * **Orange (dash-dot line with star markers):** EigenScore ### Detailed Analysis * **Perplexity (Blue):** The line is relatively flat, hovering around an AUROC of 64, with minor fluctuations. * At Temperature 0.1, AUROC is approximately 64. * At Temperature 0.3, AUROC is approximately 64. * At Temperature 0.5, AUROC is approximately 64. * At Temperature 1, AUROC is approximately 64. * At Temperature 3, AUROC is approximately 64. * At Temperature 5, AUROC is approximately 64. * **LN-Entropy (Gray):** The line starts around 67 AUROC, increases slightly to around 69 at 0.5, and then decreases to around 65 at temperature 3 and 5. * At Temperature 0.1, AUROC is approximately 67. * At Temperature 0.3, AUROC is approximately 68. * At Temperature 0.5, AUROC is approximately 69. * At Temperature 1, AUROC is approximately 68. * At Temperature 3, AUROC is approximately 66. * At Temperature 5, AUROC is approximately 64. * **Lexical Similarity (Teal):** The line starts around 69 AUROC, increases to around 75 at 0.3, remains relatively stable until temperature 1, and then decreases to around 57 at temperature 5. * At Temperature 0.1, AUROC is approximately 69. * At Temperature 0.3, AUROC is approximately 75. * At Temperature 0.5, AUROC is approximately 75. * At Temperature 1, AUROC is approximately 70. * At Temperature 3, AUROC is approximately 61. * At Temperature 5, AUROC is approximately 57. * **EigenScore (Orange):** The line starts around 72 AUROC, increases to around 80 at 0.3 and 0.5, and then decreases to around 62 at temperature 5. * At Temperature 0.1, AUROC is approximately 72. * At Temperature 0.3, AUROC is approximately 80. * At Temperature 0.5, AUROC is approximately 80. * At Temperature 1, AUROC is approximately 74. * At Temperature 3, AUROC is approximately 66. * At Temperature 5, AUROC is approximately 58. ### Key Observations * Perplexity is the least sensitive to temperature changes, maintaining a consistently low AUROC. * EigenScore and Lexical Similarity show a similar trend: an initial increase in AUROC followed by a decrease as temperature increases. * LN-Entropy shows a slight increase and then a decrease. * All metrics except Perplexity show a decrease in AUROC at higher temperatures. ### Interpretation The chart suggests that the performance of different metrics, as measured by AUROC, is affected by temperature. Perplexity appears to be the most stable metric across different temperatures, while EigenScore and Lexical Similarity are more sensitive, showing a peak in performance at lower temperatures (0.3-0.5) and a decline at higher temperatures. This could indicate that these metrics are more effective within a specific temperature range, while Perplexity's consistent performance might make it a more reliable choice across varying conditions. The decrease in AUROC for most metrics at higher temperatures could imply that the models or systems using these metrics become less accurate or reliable as temperature increases.

## Line Chart: Sensitivity to Temperature ### Overview This image is a line chart titled "Sensitivity to Temperature." It evaluates the performance of four different metrics—Perplexity, LN-Entropy, Lexical Similarity, and EigenScore—across varying "Temperature" settings. Performance is measured using the AUROC (Area Under the Receiver Operating Characteristic curve) metric. The chart demonstrates how the reliability of these metrics changes as the sampling temperature of a model (likely a Large Language Model) is adjusted. ### Components/Axes * **Title**: "Sensitivity to Temperature" (centered at the top). * **Y-Axis**: * **Label**: "AUROC" (positioned vertically on the left). * **Scale**: Ranges from 40 to 100. * **Markers**: Major ticks at intervals of 10 (40, 50, 60, 70, 80, 90, 100). * **X-Axis**: * **Label**: "Temperature" (centered at the bottom). * **Scale**: Categorical/Logarithmic-style spacing with specific values: 0.1, 0.3, 0.5, 1, 3, 5. * **Legend**: Located in the **top-right** corner of the plot area. * **Perplexity**: Blue solid line with 'x' markers and a dash-dot pattern. * **LN-Entropy**: Grey solid line with diamond markers and a dashed pattern. * **Lexical Similarity**: Teal/Cyan solid line with circle markers and a dash-dot pattern. * **EigenScore**: Orange solid line with five-pointed star markers and a dash-dot pattern. ### Detailed Analysis The following table reconstructs the approximate AUROC values based on the visual placement of data points: | Temperature | Perplexity (Blue 'x') | LN-Entropy (Grey Diamond) | Lexical Similarity (Teal Circle) | EigenScore (Orange Star) | | :--- | :--- | :--- | :--- | :--- | | **0.1** | ~64 | ~66.5 | ~69 | ~71 | | **0.3** | ~64 | ~67.5 | ~75 | ~80 | | **0.5** | ~64 | ~68.5 | ~75 | ~80.5 | | **1** | ~64 | ~68 | ~70 | ~73.5 | | **3** | ~64 | ~66.5 | ~61 | ~64 | | **5** | ~64 | ~66 | ~57 | ~58 | #### Trend Verification: * **Perplexity**: The line is perfectly horizontal (flat). It shows zero sensitivity to the temperature parameter. * **LN-Entropy**: The line is relatively stable with a very slight upward curve, peaking at Temperature 0.5 before a marginal decline. * **Lexical Similarity**: The line slopes upward from 0.1 to 0.3, plateaus between 0.3 and 0.5, and then slopes downward sharply as temperature increases beyond 0.5. * **EigenScore**: This line shows the most significant variation. It slopes upward steeply to a peak at 0.5 and then slopes downward sharply, crossing below the Perplexity and LN-Entropy lines at high temperatures. ### Key Observations * **Peak Performance**: Both EigenScore and Lexical Similarity reach their peak performance at a temperature range of 0.3 to 0.5. * **Highest Performer**: EigenScore achieves the highest overall AUROC (approximately 80.5 at T=0.5). * **Robustness**: LN-Entropy is the most robust metric among those that vary, maintaining a tight range between ~66 and ~68.5 AUROC. * **Baseline**: Perplexity acts as a constant baseline at ~64 AUROC. * **Degradation**: At high temperatures (T=5), the performance of Lexical Similarity (~57) and EigenScore (~58) drops below the Perplexity baseline (~64). ### Interpretation The data suggests that the choice of sampling temperature is critical for metrics that rely on consistency or structural analysis of model outputs (like EigenScore and Lexical Similarity). 1. **The "Sweet Spot"**: The peak at T=0.3–0.5 suggests that these metrics require a balance of diversity in generated samples. At very low temperatures (0.1), samples may be too repetitive to provide meaningful variance. At high temperatures (3.0–5.0), the samples likely become too chaotic or "hallucinated," introducing noise that degrades the metric's ability to distinguish correct from incorrect information. 2. **Metric Characteristics**: * **EigenScore** is highly effective but very sensitive to noise. * **LN-Entropy** appears to measure a property of the model's distribution that is largely independent of the specific sampling temperature used for generation. * **Perplexity**'s flat line indicates it is likely calculated on a fixed set of data or is a property of the model's base probability distribution, making it a "temperature-independent" baseline in this specific experimental setup. 3. **Practical Application**: If using EigenScore for hallucination detection or uncertainty estimation, a temperature of 0.5 is optimal. Using a default temperature of 1.0 or higher would result in significantly lower reliability for that specific metric.

## Line Chart: Sensitivity to Temperature ### Overview This line chart illustrates the relationship between Temperature and AUROC (Area Under the Receiver Operating Characteristic curve) for four different metrics: Perplexity, LN-Entropy, Lexical Similarity, and EigenScore. The chart aims to demonstrate how sensitive each metric is to changes in temperature. ### Components/Axes * **Title:** Sensitivity to Temperature * **X-axis:** Temperature, ranging from 0.1 to 5.0, with markers at 0.1, 0.3, 0.5, 1, 3, and 5. * **Y-axis:** AUROC, ranging from 40 to 100, with markers at 40, 50, 60, 70, 80, 90, and 100. * **Legend:** Located in the top-right corner, identifying the four data series: * Perplexity (Blue, with triangle markers) * LN-Entropy (Gray, with circle markers) * Lexical Similarity (Teal, with diamond markers) * EigenScore (Orange, with square markers) ### Detailed Analysis Here's a breakdown of each data series, with approximate values extracted from the chart: * **Perplexity (Blue Triangles):** The line is relatively flat, indicating minimal sensitivity to temperature. * Temperature 0.1: AUROC ≈ 64 * Temperature 0.3: AUROC ≈ 64 * Temperature 0.5: AUROC ≈ 64 * Temperature 1: AUROC ≈ 64 * Temperature 3: AUROC ≈ 64 * Temperature 5: AUROC ≈ 64 * **LN-Entropy (Gray Circles):** The line shows a decreasing trend with increasing temperature. * Temperature 0.1: AUROC ≈ 71 * Temperature 0.3: AUROC ≈ 71 * Temperature 0.5: AUROC ≈ 70 * Temperature 1: AUROC ≈ 68 * Temperature 3: AUROC ≈ 65 * Temperature 5: AUROC ≈ 63 * **Lexical Similarity (Teal Diamonds):** The line initially increases, reaches a peak, and then decreases with increasing temperature. * Temperature 0.1: AUROC ≈ 68 * Temperature 0.3: AUROC ≈ 73 * Temperature 0.5: AUROC ≈ 76 * Temperature 1: AUROC ≈ 74 * Temperature 3: AUROC ≈ 67 * Temperature 5: AUROC ≈ 62 * **EigenScore (Orange Squares):** The line shows an increasing trend up to a peak, then a slight decrease. * Temperature 0.1: AUROC ≈ 70 * Temperature 0.3: AUROC ≈ 75 * Temperature 0.5: AUROC ≈ 79 * Temperature 1: AUROC ≈ 78 * Temperature 3: AUROC ≈ 74 * Temperature 5: AUROC ≈ 68 ### Key Observations * Perplexity exhibits the least sensitivity to temperature, remaining consistently around an AUROC of 64. * LN-Entropy shows a clear negative correlation with temperature – as temperature increases, AUROC decreases. * Lexical Similarity demonstrates a more complex relationship, peaking around a temperature of 0.5 and then declining. * EigenScore shows a positive correlation up to a temperature of 1, then a slight decline. * The most significant changes in AUROC occur between temperatures of 0.1 and 1. ### Interpretation The chart suggests that the choice of temperature significantly impacts the performance of some metrics (LN-Entropy, Lexical Similarity, and EigenScore) but not others (Perplexity). LN-Entropy appears to be negatively affected by higher temperatures, while Lexical Similarity and EigenScore benefit from moderate temperatures before experiencing a decline. This implies that these metrics capture different aspects of the underlying data and respond differently to variations in temperature. The temperature parameter likely controls the randomness or exploration in a generative model. Lower temperatures lead to more deterministic outputs, while higher temperatures introduce more randomness. The differing sensitivities of these metrics suggest that some are more robust to noise or uncertainty (Perplexity), while others are more sensitive to the specific details of the generated output (LN-Entropy, Lexical Similarity, EigenScore). The peak in Lexical Similarity and EigenScore could indicate an optimal temperature range where the generated outputs are both coherent and diverse. Beyond this point, increased randomness may lead to outputs that are less meaningful or relevant. The relatively flat line for Perplexity suggests it is a more stable measure, less affected by the level of randomness introduced by the temperature parameter.

## Line Chart: Sensitivity to Temperature ### Overview The image is a line chart titled "Sensitivity to Temperature." It plots the performance of four different metrics (Perplexity, LN-Entropy, Lexical Similarity, EigenScore) as a function of a "Temperature" parameter. The performance is measured by the AUROC (Area Under the Receiver Operating Characteristic Curve) score on the y-axis. The chart demonstrates how the AUROC for each metric changes as the temperature value increases from 0.1 to 5. ### Components/Axes * **Chart Title:** "Sensitivity to Temperature" (Top Center) * **Y-Axis:** * **Label:** "AUROC" (Left side, rotated vertically) * **Scale:** Linear, ranging from 40 to 100. * **Major Ticks:** 40, 50, 60, 70, 80, 90, 100. * **X-Axis:** * **Label:** "Temperature" (Bottom Center) * **Scale:** Appears to be logarithmic or categorical, with discrete points. * **Tick Labels:** 0.1, 0.3, 0.5, 1, 3, 5. * **Legend:** Located in the top-right corner of the plot area. It maps line colors and marker styles to metric names. * **Blue line with 'x' markers:** Perplexity * **Gray line with diamond markers:** LN-Entropy * **Teal line with circle markers:** Lexical Similarity * **Orange line with star markers:** EigenScore ### Detailed Analysis The following table reconstructs the approximate AUROC values for each metric at the specified temperature points. Values are estimated from the chart's grid lines. | Temperature | Perplexity (Blue, 'x') | LN-Entropy (Gray, Diamond) | Lexical Similarity (Teal, Circle) | EigenScore (Orange, Star) | | :---------- | :---------------------- | :------------------------- | :-------------------------------- | :------------------------ | | **0.1** | ~64 | ~66 | ~68 | ~71 | | **0.3** | ~64 | ~67 | ~75 | ~80 | | **0.5** | ~64 | ~68 | ~75 | ~80 | | **1** | ~64 | ~68 | ~70 | ~74 | | **3** | ~64 | ~67 | ~60 | ~64 | | **5** | ~64 | ~66 | ~57 | ~58 | **Trend Verification per Data Series:** 1. **Perplexity (Blue):** The line is perfectly horizontal. **Trend:** Constant. It shows no sensitivity to temperature, maintaining an AUROC of approximately 64 across all values. 2. **LN-Entropy (Gray):** The line has a very gentle arc. **Trend:** Slightly increases from ~66 to a peak of ~68 at Temperature=0.5/1, then gently decreases back to ~66. It shows low sensitivity. 3. **Lexical Similarity (Teal):** The line has a pronounced peak. **Trend:** Increases from ~68 to a peak of ~75 at Temperature=0.3/0.5, then decreases sharply, falling below its starting point to ~57 at Temperature=5. It shows high sensitivity, with performance degrading significantly at higher temperatures. 4. **EigenScore (Orange):** The line has the most pronounced peak. **Trend:** Increases from ~71 to a peak of ~80 at Temperature=0.3/0.5, then decreases steadily, falling to ~58 at Temperature=5. It shows the highest sensitivity and the best peak performance. ### Key Observations * **Peak Performance:** Both **EigenScore** and **Lexical Similarity** achieve their highest AUROC (~80 and ~75, respectively) at the lower temperature range of 0.3 to 0.5. * **Performance Degradation:** All metrics except Perplexity show a decline in AUROC as temperature increases beyond 1. The decline is most severe for **Lexical Similarity** and **EigenScore**. * **Stability:** **Perplexity** is completely stable (flat line). **LN-Entropy** is relatively stable, with only minor fluctuations. * **Crossover Points:** At Temperature=3, the performance of EigenScore, Lexical Similarity, and LN-Entropy converges around an AUROC of 64-67. At Temperature=5, EigenScore and Lexical Similarity perform worse than LN-Entropy and Perplexity. * **Relative Ranking:** The ranking of metrics by AUROC changes with temperature. At T=0.5, the order is EigenScore > Lexical Similarity > LN-Entropy > Perplexity. At T=5, the order is LN-Entropy > Perplexity > EigenScore ≈ Lexical Similarity. ### Interpretation This chart investigates how the effectiveness of different evaluation metrics (Perplexity, LN-Entropy, Lexical Similarity, EigenScore) for a certain task (likely related to language model output quality or detection) is influenced by the "Temperature" parameter, which typically controls randomness in generation. The data suggests a clear trade-off: * **EigenScore** is the most powerful metric at optimal (low) temperatures, achieving the highest AUROC. However, it is also the most fragile, with its performance collapsing as temperature increases. * **Lexical Similarity** follows a similar but less extreme pattern, also peaking at low temperatures and degrading with higher randomness. * **LN-Entropy** offers a balance, providing moderate and relatively stable performance across the temperature range. * **Perplexity** is insensitive to temperature in this context, suggesting it measures a property of the model or text that is unaffected by the randomness introduced by the temperature parameter. **Practical Implication:** If one can control or know the temperature of the generated text being evaluated, **EigenScore** is the superior choice for low-temperature settings. For high-temperature or unknown settings, **LN-Entropy** provides more reliable, albeit lower, performance. **Perplexity** appears to be a poor discriminator for the task measured by AUROC in this experiment, as its score does not change with the variable of interest. The chart effectively demonstrates that metric selection must be contingent on the operational conditions (here, temperature).

## Line Chart: Sensitivity to Temperature ### Overview The chart illustrates the sensitivity of four evaluation metrics (Perplexity, LN-Entropy, Lexical Similarity, EigenScore) to temperature variations, measured via Area Under the Receiver Operating Characteristic curve (AUROC). Temperature ranges from 0.1 to 5.0, with AUROC scores plotted on a 40–100 scale. All metrics show distinct trends, with some peaking at intermediate temperatures before declining. ### Components/Axes - **X-axis (Temperature)**: Logarithmic scale from 0.1 to 5.0, with markers at 0.1, 0.3, 0.5, 1, 3, and 5. - **Y-axis (AUROC)**: Linear scale from 40 to 100, with increments of 10. - **Legend**: Located in the top-right corner, associating: - **Blue (dashed line with stars)**: Perplexity - **Gray (solid line with diamonds)**: LN-Entropy - **Teal (dashed line with circles)**: Lexical Similarity - **Orange (solid line with stars)**: EigenScore ### Detailed Analysis 1. **Perplexity (Blue)**: - Starts at ~65 AUROC at 0.1 temperature. - Remains flat until 0.5, then drops sharply to ~60 by 5.0. - Minimal variability across temperatures. 2. **LN-Entropy (Gray)**: - Begins at ~67 AUROC at 0.1. - Peaks at ~69 around 0.5, then declines to ~66 by 5.0. - Slight upward trend until 0.5, then gradual decline. 3. **Lexical Similarity (Teal)**: - Starts at ~70 AUROC at 0.1. - Rises to ~75 at 0.5, then plummets to ~58 by 5.0. - Sharp decline after 0.5 temperature. 4. **EigenScore (Orange)**: - Begins at ~72 AUROC at 0.1. - Peaks at ~80 around 0.5, then falls to ~58 by 5.0. - Steepest decline among all metrics after 0.5. ### Key Observations - **EigenScore** achieves the highest AUROC (~80) at 0.5 temperature but drops sharply to ~58 at 5.0. - **Lexical Similarity** shows the most dramatic decline (~75 → ~58) after 0.5 temperature. - **LN-Entropy** and **Perplexity** exhibit relative stability, with LN-Entropy maintaining higher scores than Perplexity across all temperatures. - All metrics decline significantly at higher temperatures (3–5), suggesting reduced performance under extreme conditions. ### Interpretation The data suggests that **EigenScore** and **Lexical Similarity** are highly sensitive to temperature changes, performing optimally at intermediate temperatures (0.5) but degrading sharply at higher values. This could indicate overfitting or instability in these metrics under extreme conditions. In contrast, **LN-Entropy** and **Perplexity** demonstrate robustness, maintaining consistent performance across temperatures. The sharp declines in EigenScore and Lexical Similarity at high temperatures may highlight their reliance on specific linguistic patterns that become less reliable as temperature increases. These findings could inform metric selection in temperature-sensitive applications, favoring stability over peak performance.