## Histogram: Length Distribution of Accurate and Inaccurate Verifications ### Overview The image presents two histograms comparing the length distribution (in K tokens) of accurate and inaccurate verifications for two different systems: "LLM-as-a-Judge" and "ThinkPRM". The histograms display the frequency of different token lengths, with accurate verifications shown in light blue and inaccurate verifications in pink. ### Components/Axes * **Titles:** "LLM-as-a-Judge" (left histogram), "ThinkPRM" (right histogram) * **Y-axis:** "Frequency", ranging from 0 to 300 with increments of 50. * **X-axis:** "Length (K tokens)", ranging from 0 to 8 with increments of 1. * **Legend:** Located at the bottom of the image. * Light Blue: "Accurate Verification" * Pink: "Inaccurate Verification" * **Annotation:** In the "LLM-as-a-Judge" histogram, there's an arrow pointing to the peak of the pink bars at length 7, with the text "Overthinking, repetition, infinite looping, etc." ### Detailed Analysis **1. LLM-as-a-Judge (Left Histogram):** * **Accurate Verification (Light Blue):** * The frequency starts around 0 at length 0. * It peaks at length 1 with a frequency of approximately 160. * The frequency decreases gradually from length 1 to length 6, reaching a value of approximately 20. * From length 6 to 8, the frequency remains relatively low, around 10-20. * **Inaccurate Verification (Pink):** * The frequency is low from length 0 to 6, generally below 50. * There is a sharp peak at length 7, with a frequency of approximately 100. * The frequency drops significantly at length 8, close to 0. **2. ThinkPRM (Right Histogram):** * **Accurate Verification (Light Blue):** * The frequency starts around 0 at length 0. * It peaks at length 1 with a frequency of approximately 280. * The frequency decreases gradually from length 1 to length 6, reaching a value of approximately 10. * From length 6 to 8, the frequency remains very low, close to 0. * **Inaccurate Verification (Pink):** * The frequency is relatively low across all lengths, generally below 60. * There is a small peak at length 1, with a frequency of approximately 50. * The frequency is very low from length 6 to 8, close to 0. ### Key Observations * For both systems, accurate verifications are most frequent at shorter lengths (around 1K tokens). * For "LLM-as-a-Judge", inaccurate verifications are concentrated at a length of 7K tokens, suggesting a tendency for overthinking or looping. * For "ThinkPRM", inaccurate verifications are generally low across all lengths. ### Interpretation The histograms suggest that "LLM-as-a-Judge" is prone to issues like overthinking or infinite looping when dealing with longer sequences (around 7K tokens), leading to inaccurate verifications. "ThinkPRM", on the other hand, appears to be more consistent in its verification accuracy across different sequence lengths. The data indicates that both systems perform accurate verifications more frequently with shorter sequences. The annotation on the "LLM-as-a-Judge" histogram highlights a specific failure mode related to longer sequences, which is not as prominent in "ThinkPRM".

\n ## Histograms: Verification Accuracy vs. Length (K tokens) ### Overview The image presents two histograms, side-by-side, comparing the distribution of response lengths (in K tokens) for "Accurate Verification" and "Inaccurate Verification" cases. The left histogram represents data for "LLM-as-a-judge", and the right histogram represents data for "ThinkPRM". Both histograms share the same x and y axes scales. A text annotation on the right histogram points to the "Inaccurate Verification" distribution, labeling it as relating to "Overthinking, repetition, infinite looping, etc.". ### Components/Axes * **X-axis:** "Length (K tokens)" - Ranges from 0 to 8, with tick marks at integer values. * **Y-axis:** "Frequency" - Ranges from 0 to 300, with tick marks at intervals of 50. * **Legend (bottom-center):** * "Accurate Verification" - Represented by a light blue color. * "Inaccurate Verification" - Represented by a light purple/pink color. * **Titles:** * Left Histogram: "LLM-as-a-judge" * Right Histogram: "ThinkPRM" * **Annotation:** "Overthinking, repetition, infinite looping, etc." - Points to the tail of the "Inaccurate Verification" distribution in the "ThinkPRM" histogram. ### Detailed Analysis or Content Details **LLM-as-a-judge (Left Histogram):** * **Accurate Verification (Blue):** The distribution is unimodal, peaking at approximately 0.8 K tokens. It slopes downward relatively smoothly from the peak, with a long tail extending towards 8 K tokens. * Frequency at 0.8 K tokens: ~270 * Frequency at 1.6 K tokens: ~150 * Frequency at 2.4 K tokens: ~70 * Frequency at 3.2 K tokens: ~40 * Frequency at 4.0 K tokens: ~25 * Frequency at 4.8 K tokens: ~15 * Frequency at 5.6 K tokens: ~10 * Frequency at 6.4 K tokens: ~5 * Frequency at 7.2 K tokens: ~5 * **Inaccurate Verification (Purple):** The distribution is relatively flat with a small peak around 6.4 K tokens. It has a more pronounced tail extending towards 8 K tokens compared to the accurate verification distribution. * Frequency at 0.8 K tokens: ~10 * Frequency at 1.6 K tokens: ~15 * Frequency at 2.4 K tokens: ~20 * Frequency at 3.2 K tokens: ~25 * Frequency at 4.0 K tokens: ~30 * Frequency at 4.8 K tokens: ~40 * Frequency at 5.6 K tokens: ~50 * Frequency at 6.4 K tokens: ~60 * Frequency at 7.2 K tokens: ~50 **ThinkPRM (Right Histogram):** * **Accurate Verification (Blue):** Similar to LLM-as-a-judge, the distribution is unimodal, peaking at approximately 1.6 K tokens. It slopes downward from the peak. * Frequency at 1.6 K tokens: ~280 * Frequency at 0.8 K tokens: ~200 * Frequency at 2.4 K tokens: ~120 * Frequency at 3.2 K tokens: ~60 * Frequency at 4.0 K tokens: ~30 * Frequency at 4.8 K tokens: ~15 * Frequency at 5.6 K tokens: ~10 * Frequency at 6.4 K tokens: ~5 * **Inaccurate Verification (Purple):** The distribution is more concentrated towards higher lengths (6.4-7.2 K tokens) than in the LLM-as-a-judge histogram. The annotation "Overthinking, repetition, infinite looping, etc." points to this region. * Frequency at 1.6 K tokens: ~20 * Frequency at 2.4 K tokens: ~30 * Frequency at 3.2 K tokens: ~40 * Frequency at 4.0 K tokens: ~50 * Frequency at 4.8 K tokens: ~60 * Frequency at 5.6 K tokens: ~70 * Frequency at 6.4 K tokens: ~80 * Frequency at 7.2 K tokens: ~60 ### Key Observations * For both LLM-as-a-judge and ThinkPRM, accurate verifications tend to have shorter lengths than inaccurate verifications. * The "Inaccurate Verification" distribution for ThinkPRM is more heavily skewed towards longer lengths (6-8 K tokens) and has a more pronounced peak in that region compared to LLM-as-a-judge. * The annotation suggests that longer lengths in ThinkPRM are associated with issues like overthinking and repetition. * The peak of accurate verifications is higher for ThinkPRM than for LLM-as-a-judge. ### Interpretation The data suggests that both LLM-as-a-judge and ThinkPRM exhibit a correlation between response length and verification accuracy: longer responses are more likely to be inaccurate. However, this correlation is stronger for ThinkPRM. The annotation highlights a potential issue with ThinkPRM – it appears to be prone to generating overly verbose responses that are characterized by overthinking, repetition, or infinite looping, leading to inaccuracies. The difference in distributions between the two models suggests that ThinkPRM might be more susceptible to these issues than LLM-as-a-judge. The higher peak in accurate verifications for ThinkPRM could indicate that when it *does* provide accurate responses, they are often more concise. The histograms provide a visual representation of the trade-off between response length and accuracy. A potential area for improvement for ThinkPRM is to implement mechanisms to constrain response length and prevent the generation of overly detailed or repetitive outputs. The data supports the hypothesis that controlling length can improve accuracy.

## Histograms: Comparison of Verification Length Distributions ### Overview The image displays two side-by-side histograms comparing the distribution of verification lengths (in thousands of tokens) for two different methods: "LLM-as-a-Judge" and "ThinkPRM". Each histogram plots frequency against length, with data segmented into "Accurate Verification" (blue) and "Inaccurate Verification" (pink). An annotation highlights a specific pattern in the left chart. ### Components/Axes * **Titles:** * Left Chart: "LLM-as-a-Judge" * Right Chart: "ThinkPRM" * **X-Axis (Both Charts):** Label: "Length (K tokens)". Scale: Linear, from 0 to 8, with major tick marks at every integer (0, 1, 2, ..., 8). * **Y-Axis (Both Charts):** Label: "Frequency". Scale: Linear, from 0 to 300, with major tick marks at intervals of 50 (0, 50, 100, ..., 300). * **Legend:** Positioned at the bottom center of the entire image, below both charts. * Blue square: "Accurate Verification" * Pink square: "Inaccurate Verification" * **Annotation:** Located in the top-right quadrant of the "LLM-as-a-Judge" chart. A black arrow points to the peak of the pink bars at approximately 7K tokens. The text reads: "Overthinking, repetition, infinite looping, etc." ### Detailed Analysis **1. LLM-as-a-Judge (Left Chart):** * **Accurate Verification (Blue):** The distribution is right-skewed. The highest frequency (approximately 160-170) occurs at a length of ~1K tokens. The frequency then steadily declines as length increases, approaching near-zero by 8K tokens. * **Inaccurate Verification (Pink):** The distribution is bimodal. There is a small, low-frequency cluster between 0.5K and 3K tokens (peaking around 25-30). A second, much more prominent cluster appears between 6K and 8K tokens, with a sharp peak at ~7K tokens reaching a frequency of approximately 100-110. This peak is explicitly annotated as representing "Overthinking, repetition, infinite looping, etc." **2. ThinkPRM (Right Chart):** * **Accurate Verification (Blue):** The distribution is strongly right-skewed with a very high, sharp peak. The maximum frequency (approximately 280-290) occurs at ~1K tokens. The frequency drops off rapidly after 1.5K tokens and becomes very low (below 20) beyond 4K tokens. * **Inaccurate Verification (Pink):** The frequencies are very low across the entire range. There is a minor, broad elevation between 0.5K and 2.5K tokens (peaking around 50) and another very slight increase around 7K tokens (peaking below 20). No significant spike is observed at the higher length ranges. ### Key Observations 1. **Peak Location & Magnitude:** Both methods show the highest frequency of accurate verifications at short lengths (~1K tokens). However, the peak for ThinkPRM is significantly higher (~290 vs. ~170) and narrower, suggesting a stronger concentration of accurate results at that length. 2. **Inaccurate Verification Pattern:** The most striking difference is in the distribution of inaccurate verifications. LLM-as-a-Judge shows a major secondary mode at high token lengths (~7K), which is explicitly linked to failure modes like overthinking. ThinkPRM shows no such pronounced secondary mode; its inaccurate verifications are low and spread thinly. 3. **Length Efficiency:** The ThinkPRM distribution for accurate verifications is more concentrated at the lower end of the length scale. The LLM-as-a-Judge distribution has a longer "tail" of accurate verifications extending to higher token counts, but at much lower frequencies. ### Interpretation The data suggests a fundamental difference in the behavior and reliability of the two verification methods. * **LLM-as-a-Judge** appears prone to a specific failure mode where inaccurate verifications are strongly associated with very long outputs (6K-8K tokens). The annotation implies this is due to unproductive loops or redundancy in the model's reasoning process. While it produces accurate verifications across a wide range of lengths, its inefficiency and the clear pattern of failure at high lengths are notable drawbacks. * **ThinkPRM** demonstrates a more controlled and efficient profile. It achieves a higher density of accurate verifications at short lengths and, crucially, avoids the catastrophic "overthinking" failure mode seen in the other method. The near-absence of a high-length spike for inaccurate verifications indicates it is more robust against generating excessively long, erroneous outputs. In essence, the charts argue that ThinkPRM is a more precise and reliable verification method, as it concentrates accurate results where they are most efficient (short lengths) and minimizes the specific type of lengthy, inaccurate output that plagues the LLM-as-a-Judge approach. The visual evidence strongly links excessive length with inaccuracy for LLM-as-a-Judge, a correlation that is largely absent for ThinkPRM.

## Bar Charts: LLM-as-a-Judge vs ThinkPRM Verification Frequencies ### Overview The image contains two side-by-side bar charts comparing verification frequencies for two language model systems: "LLM-as-a-Judge" (left) and "ThinkPRM" (right). Both charts show distributions of response lengths (in thousands of tokens) for accurate vs inaccurate verification outcomes. ### Components/Axes - **X-axis**: "Length (K tokens)" with discrete bins from 0 to 8 - **Y-axis**: "Frequency" with values from 0 to 300 - **Legends**: - Blue bars = Accurate Verification - Pink bars = Inaccurate Verification - **Chart Labels**: - Left: "LLM-as-a-Judge" - Right: "ThinkPRM" - **Special Annotation**: Arrow pointing to pink bars in LLM-as-a-Judge chart at 7-8K tokens with text: "Overthinking, repetition, infinite looping, etc." ### Detailed Analysis **LLM-as-a-Judge (Left Chart)** - **Accurate Verification (Blue)**: - Peaks at 1K tokens (~150 frequency) - Gradual decline to ~50 at 3K tokens - Minimal presence beyond 4K tokens - **Inaccurate Verification (Pink)**: - Low frequency (<20) until 6K tokens - Sharp increase to ~100 at 7K tokens - Slight drop to ~50 at 8K tokens - **Key Pattern**: Accurate responses dominate at shorter lengths, while inaccuracies spike dramatically at longer lengths **ThinkPRM (Right Chart)** - **Accurate Verification (Blue)**: - Dominant peak at 1K tokens (~250 frequency) - Rapid decline to ~50 at 2K tokens - Near-zero presence beyond 3K tokens - **Inaccurate Verification (Pink)**: - Secondary peak at 2K tokens (~50 frequency) - Sharp drop to <10 at 3K tokens - Minimal presence beyond 4K tokens - **Key Pattern**: Strong accuracy at shortest length, with minimal long responses regardless of accuracy ### Key Observations 1. **Length-Accuracy Tradeoff**: Both systems show optimal accuracy at 1K tokens, with performance degrading as response length increases 2. **LLM-as-a-Judge Anomaly**: Unusually high inaccurate verification frequencies at 7-8K tokens (up to 100), suggesting pathological behavior 3. **ThinkPRM Stability**: Maintains near-zero response lengths beyond 3K tokens, avoiding the LLM-as-a-Judge's pathological long responses 4. **Verification Disparity**: LLM-as-a-Judge has 3x more inaccurate verification attempts than ThinkPRM at peak lengths ### Interpretation The data reveals fundamental differences in how these systems handle verification tasks: - **LLM-as-a-Judge** demonstrates capacity for longer responses but suffers from catastrophic failure modes at extended lengths, potentially due to recursive self-analysis or infinite reasoning loops - **ThinkPRM** shows superior stability, completely avoiding long responses that could lead to verification errors - The 7-8K token spike in LLM-as-a-Judge's inaccurate verification suggests a critical failure mode where the system enters non-terminating reasoning states - Both systems' optimal performance at 1K tokens implies that concise responses are most reliable for verification tasks, though ThinkPRM's complete absence of long responses may indicate overly restrictive design parameters The charts highlight an important consideration in LLM design: balancing response length with verification reliability, particularly when systems are used in self-evaluative roles.