2504.07825v1

# What the HellaSwag? On the Validity of Common-Sense Reasoning Benchmarks > Correspondence topavel.chizhov@thws.de Abstract Common-sense reasoning is a key language model capability because it encapsulates not just specific factual knowledge but rather general language and world understanding. Measuring common-sense reasoning, therefore, is crucial for language models of different sizes and applications. One of the most widely used benchmarks for evaluating such capabilities is HellaSwag; however, in this paper, we show that it has severe construct validity issues. These issues range from basic ungrammaticality and numerous typos to misleading prompts or equally correct options. Furthermore, we show that if models are evaluated only on answer texts, or with ”Lorem ipsum dolor…” instead of the question, more than 65% of model predictions remain the same, and this cannot be attributed merely to contamination. Since benchmark scores are an essential part of model selection in both research and commercial applications, these validity issues can have severe consequences. In particular, knowing that taking benchmark scores at face value is ubiquitous, inadequate evaluation leads to ill-informed decisions about models. In this paper, we thoroughly investigate critical validity issues posed by HellaSwag and illustrate them with various evaluations using generative language models of different sizes. We argue that this benchmark does not accurately measure common-sense reasoning and, therefore, should not be used for evaluation in its current state. Based on the results of our study, we propose requirements that should be met by future common-sense reasoning benchmarks. In addition, we release GoldenSwag, a corrected subset of HellaSwag, which, to our belief, facilitates acceptable common-sense reasoning evaluation. 1 Introduction Language models All annotations and datasets are released on HuggingFace, the code is released on GitHub. are evaluated through benchmarks. These evaluations shape language model development by indicating how different design decisions—such as hyperparameters, training data, and post-training procedure—impact model performance (Biderman et al., 2024). NLP practitioners select training procedures in order to improve performance according to accepted benchmarks. However, if the benchmarks do not measure what we think they are measuring, development may not be going in the most optimal direction, and we may be missing out on performance gains. Benchmarks should allow us to draw an inference about the capabilities of a model, but whether a benchmark measures what we want them to measure is essential for making that inference. Construct validity means that an evaluation is measuring the capability that it is claimed to measure (Cronbach & Meehl, 1955). Without established validity, the use of a benchmark may be “supercharging bad science” (Blili-Hamelin et al., 2025). <details> <summary>x1.png Details</summary>  ### Visual Description ## Screenshot: Question Prompt with Activity Label and Context Options ### Overview The image displays a question prompt interface with an activity label, contextual text, and four answer options. The correct answer is marked with a green checkmark, while grammatical errors, typos, and nonsensical text are annotated with color-coded underlines. A legend explains the color codes. ### Components/Axes - **Activity Label**: "Clean and jerk: Women are in the background of a gym lifting weights. Man" - Text is partially redacted (grayed out) in the zero-prompt version. - **Context**: Four answer options (1–4) with embedded annotations: - **Option 1**: "is preparing himself to lift weigh and stands in front of weight." - "weigh" underlined in yellow (typo). - "front of weight" underlined in red (ungrammatical). - Green checkmark indicates this is the correct answer. - **Option 2**: "is boxing with woman in an arena." - "woman" underlined in red (ungrammatical). - Yellow underline under "arena" (typo). - **Option 3**: "is running in a marathon in a large arena and people is standing around." - "people is" underlined in red (ungrammatical). - "marathon" underlined in yellow (typo). - **Option 4**: "is standing in front of a woman doing weight lifting with camera around a table." - "weight" underlined in red (ungrammatical). - "table" underlined in blue (nonsense). - **Legend**: Located in a rounded rectangle on the right. - Green checkmark: Correct answer. - Red underline: Ungrammatical text. - Yellow underline: Typo. - Blue underline: Nonsense. - Gray text: Removed in zero-prompt. ### Detailed Analysis - **Option 1** contains two errors ("weigh" and "front of weight") but is marked correct, likely due to contextual alignment with the activity label ("gym lifting weights"). - **Option 2** misuses "woman" (singular) and has a typo in "arena" but is irrelevant to the activity label. - **Option 3** introduces unrelated elements ("marathon," "people is standing") and contains grammatical errors. - **Option 4** includes a nonsensical reference to a "table" and misuses "weight" (singular). ### Key Observations - The correct answer (Option 1) retains contextual relevance despite grammatical errors, emphasizing the importance of semantic coherence over syntactic perfection. - The zero-prompt version removes the activity label entirely, relying solely on the context for inference. - Color-coding annotations highlight common errors (e.g., subject-verb agreement, typos) and nonsensical additions. ### Interpretation This interface demonstrates a system for evaluating answer relevance and grammatical accuracy in a question-answering task. The correct answer (Option 1) aligns with the activity label's context ("gym lifting weights") but retains errors, suggesting the system prioritizes contextual relevance over syntactic correctness. The annotations (color underlines) serve as a diagnostic tool to identify specific error types, while the legend provides a clear key for interpretation. The zero-prompt version's removal of the activity label underscores the challenge of context-dependent reasoning in language models. </details> Figure 1: Example from the HellaSwag validation set: a question prompt and four answer options (the correct one is ticked). Validity issues are underlined in color. The part of the question prompt that we remove in zero-prompt experiments is highlighted in light gray. Common-sense reasoning is a desirable capability in a model because it encompasses a more generalizable world understanding rather than memorized facts about the world. This kind of reasoning differs from the math- and code-oriented reasoning capabilities that are increasingly of interest, as it focuses on reasoning over domain-general scenarios. In a survey of over one hundred text-based common-sense reasoning benchmarks, Davis (2023) observe widespread quality issues. HellaSwag (Zellers et al., 2019) is one of the most widely used of these commonsense reasoning benchmarks, and it has played a large role in language model development over the past six years. It is one of the evaluations that made up Hugging Face’s Open LLM Leaderboard v1, meaning that it was used by thousands of people to compare and select models for research and deployment. HellaSwag is useful as a relatively large dataset, with over 10k items in the validation set The validation set is the only available set on HuggingFace. The test set was intentionally not released to prevent contamination, but in practice, many people use the validation set as the test set.. HellaSwag is also in a format that facilitates both evaluation using log-likelihood and text generation. Log-likelihood-based evaluation is more appropriate for small models whose capabilities are underestimated by prompt-based evaluation (Hu & Levy, 2023) due to task demands (Hu & Frank, 2024). Text generation, on the other hand, is more appropriate for larger models. Many benchmarks, especially ones that are being developed now, do not allow for effective evaluation of both small and large models either due to their difficulty, their format, or both, e.g., GPQA (Rein et al., 2024). HellaSwag was created by taking portions of existing dataset (ActivityNet (Krishna et al., 2017) and WikiHow, which the authors scraped themselves) and generating alternative completions to the segments. The completions were generated by the original GPT model (Radford et al., 2018) and curated using an Adversarial Filtering procedure in which the authors intended to select only the machine-generated answers that seemed the most plausible to the language model but nonsensical to humans. Human validators reviewed the dataset after the AF process, ensuring high agreement on correct answers. The result was intended to be a task that was trivial for people (approximately 95% accuracy) but significantly more difficult (less than 50% accuracy) for models that were the state of the art at the time. In order for a common-sense reasoning benchmark to serve its purpose, it should allow us to draw an inference about model capabilities, specifically the inference that the model with the highest HellaSwag score has the best common-sense reasoning capabilities. However, we argue that there are some key construct validity issues that prevent us from drawing that inference from HellaSwag scores (see an example in Figure 1). These issues include bad grammar, typos, and nonsensical constructions in poor English. The key contributions of this paper can be summarized as follows: - We argue that performance on HellaSwag does not benchmark sentence completion or common-sense reasoning. We point out diverse validity issues and their consequences (Section 3) and show that they are omnipresent in HellaSwag (Section 4.1). - We show how to use model predictions to judge the quality of the benchmark (Sections 4.2 and 4.3). We also use zero-prompt evaluation (Section 4.4) to directly test construct validity and show that, on average, 68% of predictions do not change based on whether or not the model is presented with the question, or even when the question is replaced with a generic text. - Based on our analysis, we propose a list of requirements that should be met by a quality common-sense reasoning benchmark (Section 5) and propose a small highly filtered subset of HellaSwag — GoldenSwag (Section 5.1) with substantially reduced effect of observed issues. 2 Related Work Validity Issues in Other Benchmarks. Several recent studies have highlighted validity issues in widely used benchmarks. In the text-generation evaluation format, MMLU (Hendrycks et al., 2021) has been shown to yield variable performance for the same model after minor changes, such as re-ordering the answer choices, changing the formatting, or rewording the task. Simply re-ordering the answer choices in MMLU leads to a significant reduction in accuracy on the task (Gupta et al., 2024). Most models showed a drop in accuracy of around 10%, but some of the models tested showed a drop of up to 27%. Changing formatting, such as the characters representing each answer choice, can also have an effect. Alzahrani et al. (2024) replaced answer letters with alternative rare characters, which led to a significant re-ordering in relative model performance ranking among the models tested. Small perturbations in the prompt were also shown to significantly change model performance on benchmarks such as MMLU and HellaSwag (Habba et al., 2025). In addition to sensitivity to these changes, MMLU has been shown to contain questions with no clear correct answer or incorrect ground truth labels (Gema et al., 2024). Other benchmarks ranging in domain and task type have also been shown to have errors in the reference answers or in the ground truth labels. One of these is GSM8K (Cobbe et al., 2021), a math word problem benchmark. Vendrow et al. (2025) find that at least 5% of the items in GSM8K contain a serious error, such as mislabeled questions, logical contradictions, and ambiguous questions. Another example is XSUM (Narayan et al., 2018), a summarization benchmark. Liang et al. (2023) found that reference summaries in XSUM were rated as being worse than model-generated responses, according to human annotators. Therefore, the quality of the benchmark underestimates the model’s summarization performance. Together, this work indicates that validity issues are pervasive in language model benchmarks. The current paper contributes to this line of work, which aims to identify issues with existing benchmarks, and, in some cases, to propose filtered, cleaned, and improved versions of the original benchmark, e.g. Gema et al. (2024) and Vendrow et al. (2025). Common-Sense Reasoning Benchmarks. In a survey of common-sense reasoning benchmarks, Davis (2023) highlights item quality as a common issue among over one hundred text-based benchmarks, of which HellaSwag is one of the most widely used. A blog post was previously published highlighting some of the issues with HellaSwag (Chen, 2023), however, the author uses only a small sample ( $n=300$ ) items from the entire dataset. The author estimates that 36% of items in HellaSwag contain errors. In this work, we expand on the list of validity issues we consider and we annotate the entire validation set. We estimate that a much higher proportion of HellaSwag contains errors. Zero-Prompt Evaluation. In this paper, we introduce the term ‘zero-prompt’ evaluation, which consists of evaluating a model on a multiple-choice benchmark without the previous context. Other work has also used similar methods (‘no context’ condition Shah et al., 2020; ‘choice-only’ prompting, Balepur & Rudinger, 2024; Balepur et al., 2024), all of which are aimed at testing whether a model relies on the prompt to complete the task. This is essential for establishing construct validity, as the task is designed to evaluate the extent to which the model is able to draw a connection between the prompt and the answer choices. Types of Evaluation. Evaluating language models on multiple-choice question benchmarks, such as HellaSwag, can be done with probability-based and generation-based formats (Hu & Levy, 2023; Lyu et al., 2024). Probability methods imply choosing the answer with maximum probability or log-likelihood and may lead to different results based on how this log-likelihood is computed and normalized (Gao, 2022; Biderman et al., 2024). Generation evaluation methods have an advantage in presenting the model with a question in its complete form but suffer from high dependence on the instruction prompt. For instance, OLMo’s performance on HellaSwag ranges from 1% to 99% based on the instruction prompt (Habba et al., 2025). There also exists a misalignment of evaluations by log-likelihood and generation (Hu & Levy, 2023; Lyu et al., 2024), so these evaluations essentially present different tasks and models evaluated with different methods cannot be compared. 3 Methods We investigate several validity issues of HellaSwag to show how they limit its ability to serve as a language model benchmark. We focus on the following issues: Ungrammaticality and typos. Sentences with grammatical errors and typos have generally lower likelihood and might push the model away from these options, which hinders adequate evaluation when these errors are not intended. This introduces an undesired noise in the benchmark scores when the model’s capabilities are measured on a trade-off between reasoning and natural language understanding or grammatical error correction. Even though such noise may be considered a factor that makes the task more challenging, we argue that the common-sense reasoning benchmark should be free of such errors, and, if wanted, the errors may be separately injected to test the model’s robustness in comparison. Nonsensicality. Nonsensical texts naturally contradict the common-sense reasoning evaluation, especially when nonsensical sentences are present in the correct answer and the question prompt. In such questions choosing the correct answer is essentially reduced to random guessing. Additionally, ridiculously implausible incorrect options create overly high contrast with the correct answer and make the task trivial and non-informative. No or multiple good answers. All questions in HellaSwag are supposed to have exactly one correct answer. If there is no good option, the scoring of a question comes down to random guessing. On the other hand, if there are multiple acceptable endings, the model might predict an equally plausible option, but this will be treated as a wrong answer. Multiple correct options also affect the estimation of the random baseline score value. We also aim at directly evaluating the construct validity of HellaSwag, i.e., testing whether the benchmark measures what it is supposed to measure (Section 3.3). To investigate all these issues, we use annotations from a large language model and predictions from a range of language models of different sizes in diverse experiments. 3.1 Annotations We use Claude 3.5 Sonnet (Anthropic, 2024) to annotate the HellaSwag validation set. In the first round of annotations, we assess questions and answer options for grammaticality and sensicality. In the second round, we annotate the plausibility of correct answers, the presence of equally correct options, and we also collect the options considered to be the worst. We report the detailed prompts we use in Appendix C. In both rounds of annotations, we specify in the prompt which answer is supposed to be correct so that we do not rely on Claude’s solutions for the benchmark, and the model has a better understanding of how the question is designed. We use the collected annotations as descriptive indications of HellaSwag issues and also as part of our further experimental design. 3.2 Model Evaluations We propose using model evaluation on HellaSwag to investigate benchmark issues. We perform two types of model evaluations: by maximizing the mean log-likelihood and by generation. Before the evaluations, we apply the preprocessing procedure from lm-evaluation-harness (Biderman et al., 2024). All evaluations are done manually, rather than using an existing framework, in order to enable modifications, e.g., zero-prompt evaluation. We release our evaluation code as part of this work. Mean Log-Likelihood. We append each answer option to the question prompt and run each sequence through the model to compute output logits. We then choose the maximum mean log-likelihood among answer options using the following formula: $$ \mathcal{L}=\frac{1}{|\mathcal{V}|}\sum_{t\in\mathcal{V}}\log\mathrm{P}(y_{t}% \mid x_{<t}), \tag{1} $$ where $y_{t}$ is the ground truth token at position $t$ , $x_{<t}$ is the preceding sequence of tokens, and $\mathcal{V}$ is the set of valid (non-special) token positions. The resulting value will have non-positive values, and the higher it is, the more plausible the option is, according to the model. Generation. The model is presented with a question and all answer choices and is asked to output the correct answer digit (the options are enumerated 1–4). The exact prompt used for evaluation is presented in Appendix B. We use the generation strategy set by default for each model. In order to reduce the possible effect of contamination in generation evaluations, we shuffle the options for each question following Alzahrani et al. (2024). These two types of evaluation differ in the task the model is asked to perform. In log-likelihood evaluation, we directly request the model’s estimate of plausibility for each given option. On the contrary, evaluation by generation allows us to present the model with all the options so that it can compare and choose the most plausible or the least implausible option. We consider evaluation by generation only for larger models ( $\sim$ 32B), as their instruct versions are more likely to produce outputs in the desired format. We report the list of models we used in Appendix A. 3.3 Zero-Prompt Evaluations In order to test the construct validity of HellaSwag, we also run zero-prompt evaluations. By zero-prompt, we mean removing the activity label and the question context from the task. We keep only the beginnings of answer sentences, typically present in the ActivityNet part of the data and absent in WikiHow. We show an example of such removal in Figure 1. We also use another setup when we change the question text to a generic ”Lorem ipsum dolor…”. We perform this kind of evaluation to test whether HellaSwag measures common-sense reasoning. The idea behind this is that for common-sense reasoning, the model should figure out which ending stems best from the question prompt. By comparing the results from these evaluations, we can see whether this causally impacts model performance. | First round — Prompt — Correct option | Nonsense: 4.9% 1.5% | 11.7% 3.1% | 1.7% 0.8% | | | --- | --- | --- | --- | --- | | — Incorrect option(s) | 84.5% | 71.1% | 90.9% | | | Ungrammatical: | | | | | | — Prompt | 39.7% | 95.7% | 12.9% | | | — Correct option | 6.1% | 11.7% | 3.4% | | | — Incorrect option(s) | 39.4% | 43.5% | 37.5% | | | High quality | 4.9% | 2.0% | 6.3% | | | Second round | Wrong golden answer | 3.7% | 5.5% | 2.8% | | All nonsense | 4.1% | 10.0% | 1.2% | | | Multiple correct | 21.1% | 31.3% | 16.3% | | Table 1: Claude annotations for the HellaSwag validation set questions. We present the percentage for each issue from the complete set, and by source: from ActivityNet (AN) and WikiHow (WH). The validation set consists of 10042 questions, 3243 (32.3%) of which are from ActivityNet, and all the rest are from WikiHow. 4 Results and Discussion 4.1 Annotations The results of the two rounds of annotations are presented in Table 1. From the first round of evaluations of grammar and sensicality, we find that almost 40% of the questions have ungrammatical prompts. Such questions comprise the absolute majority (95.7%) of the ActivityNet part, which can be attributed to the benchmark creation methodology, in which these questions were generated by the original GPT model (Radford et al., 2018) inferior to the modern ones. It is also clear that correct answers generally have considerably fewer issues than incorrect ones because, on benchmark construction, the correct answers were taken from an existing corpus and the incorrect ones were synthetically generated. This might lead to trivial solutions dependent solely on choosing the least problematic option. In the second round, we concentrated on the plausibility of answer options and the task design validation. Claude agreed with the answer option labeled as correct in 96.3% of cases. However, in many questions (21.1%), multiple other options were considered just as good as the correct one (up to all three others in several cases). In some cases (4.1%), all answer options were considered implausible. These issues are mostly concentrated in the ActivityNet portion of the data, though also present in the WikiHow. We also encountered six ethical refusals from Claude. By investigating these questions, we found that they contain severe ethical issues related to constructing weapons, taking drugs, and adult content. 4.2 Model Evaluations <details> <summary>x2.png Details</summary>  ### Visual Description ## Bar Chart: Model Accuracy Comparison Across Evaluation Methods ### Overview The chart compares the accuracy of various language models (LMs) under three evaluation methods: Normal, Extended, and Worst. It also includes three evaluation types: Full-Prompt, Zero-Prompt, and Random. The y-axis represents accuracy (0–0.8), while the x-axis lists models like Qwen, OLMo, Llama, Gemma, and others. A dashed line at ~0.3 marks the Random baseline. ### Components/Axes - **X-Axis (Models)**: - Qwen 2.5 32B, OLMo 2.32B, Llama 3.2 1B, Gemma 3 1B, Qwen 2.5 1.5B, SmolLM2 1.7B, Granite 3.1 1B, Pythia 1B, Pleias 1.0 1B, DeepSeek-R1 1.5B. - **Y-Axis (Accuracy)**: - Scale from 0.0 to 0.8 in increments of 0.2. - **Legend (Evaluation Methods)**: - **Normal**: Solid olive-green bars. - **Extended**: Solid dark-green bars. - **Worst**: Solid salmon-pink bars. - **Full-Prompt**: Diagonal-striped olive-green bars. - **Zero-Prompt**: Diagonal-striped dark-green bars. - **Random**: Dashed gray line (~0.3 accuracy). ### Detailed Analysis 1. **Model Performance**: - **Qwen 2.5 32B**: - Normal: ~0.82, Extended: ~0.85, Worst: ~0.48. - **OLMo 2.32B**: - Normal: ~0.85, Extended: ~0.70, Worst: ~0.45. - **Llama 3.2 1B**: - Normal: ~0.60, Extended: ~0.65, Worst: ~0.35. - **Gemma 3 1B**: - Normal: ~0.58, Extended: ~0.52, Worst: ~0.32. - **Qwen 2.5 1.5B**: - Normal: ~0.65, Extended: ~0.55, Worst: ~0.38. - **SmolLM2 1.7B**: - Normal: ~0.68, Extended: ~0.58, Worst: ~0.36. - **Granite 3.1 1B**: - Normal: ~0.70, Extended: ~0.50, Worst: ~0.34. - **Pythia 1B**: - Normal: ~0.42, Extended: ~0.45, Worst: ~0.30. - **Pleias 1.0 1B**: - Normal: ~0.40, Extended: ~0.42, Worst: ~0.32. - **DeepSeek-R1 1.5B**: - Normal: ~0.38, Extended: ~0.40, Worst: ~0.28. 2. **Evaluation Methods**: - **Full-Prompt** (diagonal-striped olive-green): - Consistently higher than Zero-Prompt for most models (e.g., Qwen 2.5 32B: ~0.82 vs. ~0.65). - **Zero-Prompt** (diagonal-striped dark-green): - Outperforms Random but lags behind Full-Prompt (e.g., OLMo 2.32B: ~0.70 vs. ~0.85). - **Random** (dashed gray line): - Baseline at ~0.3; all models exceed this except DeepSeek-R1 1.5B in Worst (~0.28). ### Key Observations - **Trends**: - Extended evaluations generally outperform Normal and Worst for most models (e.g., OLMo 2.32B: Extended ~0.70 vs. Normal ~0.85). - Worst evaluations consistently yield the lowest accuracy across all models. - Full-Prompt evaluations achieve the highest accuracy for larger models (e.g., Qwen 2.5 32B: ~0.82). - **Outliers**: - DeepSeek-R1 1.5B underperforms in all categories, with Worst accuracy (~0.28) below the Random baseline. - Pythia 1B and Pleias 1.0 1B show minimal differences between evaluation methods. ### Interpretation The data demonstrates that **evaluation method significantly impacts model performance**. Full-Prompt evaluations consistently yield higher accuracy than Zero-Prompt and Random, suggesting that prompt design is critical for model effectiveness. Extended evaluations often outperform Normal evaluations, indicating that additional context or constraints improve results. However, the Worst evaluation method universally reduces accuracy, highlighting its inefficacy. Larger models (e.g., Qwen, OLMo) maintain higher performance across evaluations, while smaller models (e.g., Pythia, Pleias) struggle, particularly in Worst scenarios. The Random baseline (~0.3) serves as a benchmark, with most models exceeding it except in extreme cases like DeepSeek-R1 1.5B. This underscores the importance of tailored evaluation strategies for different model sizes and tasks. </details> Figure 2: Accuracy evaluation with log-likelihood for larger (32B) and smaller (1-2B) models with a full question prompt and without it (zero-prompt). For each model, we report three accuracy scores: usual accuracy on correct answers, extended accuracy on correct and equally correct options, and accuracy on the worst option. Important: the ground truths for the last two kinds of evaluation are based on the annotations from Claude. In Figure 2, we present the results of accuracy evaluations. Along with the regular accuracy computed by maximizing the log-likelihood, we compute extended accuracy based on Claude annotations. In particular, we extend the set of correct answers with the ones proposed by Claude as equally correct. For all models, the extended accuracy is higher, which means that in some cases, models prefer the option that can be considered just as good as the correct answer. Therefore, as we discussed in Section 3, the model is sometimes penalized for sensible answers unintended to be correct, which invalidates the scoring. We also compute the accuracy on the worst options selected by Claude by minimizing the mean log-likelihood, thus choosing the least plausible option. These scores are comparable for the majority of models. In addition, report accuracies by question source in Appendix F. <details> <summary>x3.png Details</summary>  ### Visual Description ## Scatter Plot: Mean Log-Likelihood vs. Text Length by Source ### Overview The image is a scatter plot comparing **Mean Log-Likelihood** (y-axis) against **Text Length** (x-axis) for two sources: **WikiHow** (green) and **ActivityNet** (red). Two trendlines (solid lines) represent the central tendency of each source's data points. The plot uses a grid background with square markers for data points, and the legend is positioned in the bottom-right corner. --- ### Components/Axes - **X-Axis (Text Length)**: - Range: 20 to 140 (increments of 20). - Labels: "Text Length" at the bottom. - **Y-Axis (Mean Log-Likelihood)**: - Range: -6 to -2 (increments of 1). - Labels: "Mean Log-Likelihood" on the left. - **Legend**: - Located in the bottom-right corner. - **Green**: WikiHow. - **Red**: ActivityNet. --- ### Detailed Analysis #### WikiHow (Green) - **Data Points**: - Clustered primarily between **Text Length 60–140** and **Mean Log-Likelihood -3 to -2**. - Density decreases for shorter texts (<60) and longer texts (>120). - **Trendline**: - Slopes upward from ~(-3, 60) to ~(-2, 140). - Indicates a positive correlation between text length and mean log-likelihood. #### ActivityNet (Red) - **Data Points**: - Clustered between **Text Length 20–80** and **Mean Log-Likelihood -4 to -3**. - Fewer points at extremes (e.g., <20 or >80 text length). - **Trendline**: - Slopes upward from ~(-4, 20) to ~(-3, 80). - Less steep than WikiHow’s trendline. --- ### Key Observations 1. **WikiHow Dominates in Longer Texts**: - WikiHow’s data points and trendline occupy the upper-right quadrant, suggesting longer texts (60–140) with higher mean log-likelihoods (-2 to -3). 2. **ActivityNet’s Shorter, Lower-Performing Texts**: - ActivityNet’s data is concentrated in the lower-left quadrant (20–80 text length, -4 to -3 log-likelihood). 3. **Trendline Steepness**: - WikiHow’s trendline is steeper, indicating a stronger relationship between text length and performance compared to ActivityNet. 4. **Legend Accuracy**: - Green (WikiHow) and red (ActivityNet) markers align perfectly with their respective trendlines. --- ### Interpretation - **Performance vs. Length**: WikiHow’s texts are longer and achieve higher mean log-likelihoods, implying better quality or relevance in longer content. ActivityNet’s shorter texts underperform, possibly due to brevity or less structured guidance. - **Trend Implications**: Both sources show that longer texts correlate with improved performance, but WikiHow’s advantage is more pronounced. This could reflect differences in content design (e.g., WikiHow’s step-by-step guides vs. ActivityNet’s activity-based instructions). - **Outliers/Anomalies**: No significant outliers; data points align closely with trendlines. The plot highlights WikiHow’s superiority in leveraging text length for higher performance, likely due to its structured, detailed approach. </details> Figure 3: Mean log-likelihoods of answer options and their lengths. The lines represent the trend for each question source. | Qwen-2.5 32B OLMo-2 32B | 0.92 0.84 0.86 0.87 | 0.83 0.70 0.75 0.75 | | --- | --- | --- | Table 2: Accuracies for 32B models with generation (G) and log-likelihood (LL) evaluation. We report the complete accuracy for HellaSwag (Accuracy) and accuracy for the questions annotated to have no good options (All Bad). We also evaluate larger models by generation (Table 2), the detailed generation prompt is shown in Appendix B. For OLMo, the accuracies by generation and by log-likelihood are comparable, while for Qwen, the accuracy with generation is better. This can be due to the score on the questions having no good options. Evaluating by generation allows the model to see all options at once and, even if they are all bad, to choose the most acceptable. 4.3 Answer Length We also found that answer likelihood positively correlates with its length (see Figure 3), especially for the questions from ActivityNet. This might be due to the difference in answer option sources: generated distractor options can be shorter than the correct answers. Another reason for this might be that every text has very low likelihood values in the beginning, but as the prior context grows longer, the following tokens become more predictable, and log-likelihoods increase overall. Therefore, the overall mean token probability is lower for shorter texts. Finally, such correlation might also be dependent on the log-likelihood normalization type. We separately investigate total and byte-normalized log-likelihoods proposed by Gao (2022) and find that these methods also produce values correlated with the input length (see Appendix D.1 for details). The relative length difference between answer options can be quite high (more than 17% for half of the data, see Appendix D.2). Furthermore, we find that the accuracy of Llama 3.2 1B when the correct answer is the longest one (0.72) is substantially greater than when the longest answer is wrong (0.59). Thus, varied answer lengths pose a potential problem for a benchmark since the models might implicitly prefer longer answers. 4.4 Zero-Prompt Evaluation If a model is able to choose the right option without the question text, this invalidates the construct validity of a benchmark. To test this, we evaluate the models without the question prompt, i.e., only on the answer options, and compare with the full-prompt evaluation. The accuracies for zero-prompt evaluation are presented in Figure 2. For all models, zero-prompt accuracies are well above random guessing (25%), which means that the probabilities of the answer choices are biased towards the correct answer. This presents a problem in two ways. First, the question prompt does not contribute to the task. Without the prompt, there is no reasoning about plausible completions. Instead, the task is for a model to choose the most plausible text fragment. It is unclear what model capabilities this evaluates. Second, features of the incorrect answers—such as grammatical or logical errors, as discussed in Section 4.1 —may increase the probability of the correct option. This means that a model may achieve high accuracy on a subset of the questions by simply ruling out grammatical inconsistencies or nonsensical phrases. By this logic, high accuracy on these questions does not indicate a model’s common-sense reasoning capabilities. | Model | Size | Agreement | Agreement type | Disagreement type | | | | | --- | --- | --- | --- | --- | --- | --- | --- | | Both ✓ | Both ✗ | Full ✓ | Zero ✓ | Both ✗ | | | | | Llama 3.2 | 1. 1B | 0.69 | 0.43 | 0.26 | 0.19 | 0.04 | 0.07 | | Gemma 3 | 1. 1B | 0.70 | 0.40 | 0.29 | 0.19 | 0.04 | 0.07 | | Qwen 2.5 | 1.5B | 0.69 | 0.45 | 0.24 | 0.21 | 0.03 | 0.07 | | SmolLM2 | 1.7B | 0.71 | 0.49 | 0.23 | 0.21 | 0.03 | 0.05 | | Granite 3.1 | 1. 1B | 0.72 | 0.44 | 0.28 | 0.18 | 0.03 | 0.06 | | Pythia | 1. 1B | 0.71 | 0.32 | 0.39 | 0.14 | 0.05 | 0.10 | | PleIAs | 1. 1B | 0.69 | 0.28 | 0.41 | 0.13 | 0.06 | 0.11 | | DeepSeek-R1 | 1.5B | 0.65 | 0.26 | 0.39 | 0.16 | 0.06 | 0.13 | | Qwen 2.5 (LL) | 32B | 0.71 | 0.60 | 0.12 | 0.24 | 0.02 | 0.03 | | OLMo 2 (LL) | 32B | 0.74 | 0.64 | 0.09 | 0.23 | 0.01 | 0.02 | | Qwen 2.5 (Gen) | . 32B | 0.70 | 0.67 | 0.03 | 0.25 | 0.03 | 0.02 | | Olmo 2 (Gen) | . 32B | 0.50 | 0.45 | 0.05 | 0.41 | 0.05 | 0.04 | Table 3: We report proportions by agreement type (a model gives equal correct or incorrect predictions) and disagreement types (the model gives correct prediction only in either full- or zero-prompt scenario or gives different incorrect answers). Furthermore, we test the agreement of full-prompt and zero-prompt predictions. Following Lyu et al. (2024), “agreement” here means that the model gives the same predictions in both evaluation modes (with and without the prompt), regardless of whether these predictions are correct or wrong. The results of agreement evaluations are presented in Table 3 for all models. On average, 68% of the model predictions do not change if we remove the question prompt from the evaluation. This holds not only for correct answers but also for a substantial partition of the incorrect ones, which discards contamination as the main suspected reason for this. For Pythia, PleIAs, and DeepSeek-R1 models, the share of agreement for incorrect answers is larger than for the correct ones. This suggests that for the majority of HellaSwag questions, the question text does not contain additional information that the model uses to do the task. Furthermore, for some questions, removing the context actually allowed the model to make a better choice (see column Zero ✓). Interestingly, we observed similar agreement patterns when the question prompt is changed to a placeholder text “Lorem ipsum dolor…” (see Appendix E). Together with the other results, these findings lead us to question the most fundamental quality of the benchmark — its construct validity. Since the question text does not play a decisive role in $\sim$ 68% of cases, the benchmark cannot accurately measure common-sense reasoning, as presenting the model with the cause does not help it in the choice of the right effect. This also poses a concern about the answer choices: the incorrect options might often be so blatantly implausible that the correct answer stands out, even without context. | % of questions | 75% | 67% | 60% | 55% | 50% | 45% | 39% | 34% | 27% | 18% | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | Table 4: Zero-prompt core. A pair (N models, X%) means that at least N models can answer correctly X% of questions without question prompt texts. Interestingly, we also find that questions answered correctly in zero-prompt are common for many models. In Table 4, we show the zero-prompt core — the percentage of questions answered correctly without the prompt by groups of models. In particular, about 18% of questions are answered correctly without question prompts by all 10 of the tested models. 5 Requirements and GoldenSwag Our experiments highlighted a variety of validity issues in HellaSwag. Based on these results, we formulate a list of requirements for a valid common-sense reasoning benchmark: Grammar and typos. All the texts should be grammatically correct, including the incorrect options. If the model is to be tested for robustness to bad grammar or typos, a separate version of the dataset should be created, e.g., by injecting errors or typos. Sensicality. All options should be a reasonable, coherent text. Even though by the nature of the task incorrect options should make no sense, this should be only due to how they relate to the question context and not, for instance, due to single implausible constructions. Distinct correct answers. The correct option should be a substantially better fit for the question prompt than the incorrect options. There might be more than one correct option if the task design allows it, but they should all be comparably suitable. Uniform option lengths. The answers can have different lengths, though this variability should be limited. The length rank of correct answers should be uniform over the dataset. Content. The questions should be filtered for toxicity and ethical issues. For smaller models, the questions should preferably be on general-domain topics excluding overly specific professional knowledge. 5.1 GoldenSwag <details> <summary>x4.png Details</summary>  ### Visual Description ## Bar Chart: Model Accuracy Comparison ### Overview The chart compares the accuracy of various language models (LMs) on two evaluation methods: **HellaSwag** (blue bars) and **GoldenSwag** (orange bars). A dashed horizontal line at ~0.25 represents the "Random" baseline. Models are listed on the x-axis, with accuracy values on the y-axis (0–0.8). ### Components/Axes - **X-axis (Model)**: - Qwen 2.5 32B - OLMo 2.32B - Llama 3.2 1B - Gemma 3.1B - Qwen 2.5 1.5B - SmolLM2 1.7B - Granite 3.1B - Pythia 1B - PleiAs 1.0 1B - DeepSeek-R1 1.5B - **Y-axis (Accuracy)**: 0.0 to 0.8 in increments of 0.2. - **Legend**: - **Blue (HellaSwag)**: Full-Prompt evaluation. - **Orange (GoldenSwag)**: Zero-Prompt evaluation. - **Dashed Line (Random)**: Baseline accuracy (~0.25). ### Detailed Analysis - **HellaSwag (Blue)**: - Highest accuracy: **OLMo 2.32B (0.85)**. - Lowest accuracy: **DeepSeek-R1 1.5B (0.42)**. - Most models exceed the random baseline (0.25), with values ranging from 0.42 to 0.85. - **GoldenSwag (Orange)**: - Highest accuracy: **OLMo 2.32B (0.6)**. - Lowest accuracy: **PleiAs 1.0 1B (0.18)**. - Many models fall below the random baseline (e.g., PleiAs 1.0 1B, DeepSeek-R1 1.5B). - **Random Baseline**: - All models’ HellaSwag accuracies exceed this line. - Only **OLMo 2.32B (GoldenSwag)** and **Qwen 2.5 1.5B (GoldenSwag)** approach the baseline (~0.5). ### Key Observations 1. **HellaSwag Dominance**: HellaSwag consistently outperforms GoldenSwag across all models. 2. **Model Size Correlation**: Larger models (e.g., Qwen 2.5 32B, OLMo 2.32B) generally achieve higher accuracy. 3. **GoldenSwag Limitations**: Many models (e.g., Pythia 1B, PleiAs 1.0 1B) perform near or below random chance with GoldenSwag. 4. **Outliers**: - **SmolLM2 1.7B**: GoldenSwag accuracy (0.3) is significantly lower than HellaSwag (0.7). - **DeepSeek-R1 1.5B**: Both methods underperform, with HellaSwag at 0.42. ### Interpretation The data suggests that **HellaSwag** is a more robust evaluation method for these models, as it consistently yields higher accuracy. GoldenSwag’s performance varies widely, with smaller models (e.g., Pythia 1B) struggling to surpass random chance. The random baseline highlights that some models’ zero-prompt evaluations (GoldenSwag) are ineffective, indicating potential flaws in the evaluation design or model capabilities. Larger models like OLMo 2.32B and Qwen 2.5 32B demonstrate strong performance, suggesting scalability benefits. However, the disparity between HellaSwag and GoldenSwag raises questions about the latter’s suitability for evaluating smaller or less capable models. </details> Figure 4: Accuracy comparison between HellaSwag and GoldenSwag. All models were evaluated by maximizing mean log-likelihood in full- and zero-prompt scenarios. Based on our analysis, we propose a small, highly filtered subset of HellaSwag, GoldenSwag. We removed the questions that had nonsensical or ungrammatical prompts or correct answers, and those that had grammar errors in the incorrect answer options. We also filtered out the questions in which Claude disagreed with the correct answer, found other equally correct options, or indicated that all of the options were implausible. In addition, we eliminated the questions rejected by Claude for ethical reasons. Furthermore, we filtered out the questions that had relative differences between the longest and shortest options larger than 0.3, and those among the rest, where the difference was above 0.15 and the longest answer was the correct one. Finally, we discarded the questions that at least seven out of ten tested models managed to answer correctly without question prompts. We present the detailed filtering in Appendix G. These filters left us with 1525 questions (15.2% of the original 10042), which we release as a part of this work. We re-evaluated the models on GoldenSwag (see Figure 4). The scores of smaller models dropped, while the larger ones improved their results. For smaller models, the zero-prompt evaluations dropped below random chance. 6 Conclusion In this paper, we described numerous construct validity issues in HellaSwag, ranging from grammar and typos to ambiguous answer choices. Zero-prompt evaluation shows that in most of the HellaSwag questions, question text does not affect model predictions, which allows us to conclude HellaSwag performance does not necessarily indicate common-sense reasoning capabilities. We specifically highlight that common-sense reasoning is a salient characteristic, and there should be a raised research interest in constructing a good benchmark to measure it. Based on our findings, we released GoldenSwag — a filtered subset of HellaSwag that can be one of the first steps towards this goal. Acknowledgments The authors of this work acknowledge the HPC resource allocation by Erlangen National High-Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) (joint project with the Center for Artificial Intelligence (CAIRO), THWS) and Jean Zay (compute grant #GC011015451). The authors would also like to thank Catherine Arnett for her continuous help with the project, and the members of the PleIAs and CAIRO Research teams for helpful discussion and feedback. References - Allal et al. (2025) Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big – data-centric training of a small language model, 2025. URL https://arxiv.org/abs/2502.02737. - Alzahrani et al. (2024) Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan AlRashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, M Saiful Bari, and Haidar Khan. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13787–13805, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.744. URL https://aclanthology.org/2024.acl-long.744/. - Anthropic (2024) Anthropic. Introducing claude 3.5 sonnet, June 2024. URL https://www.anthropic.com/news/claude-3-5-sonnet. - Balepur & Rudinger (2024) Nishant Balepur and Rachel Rudinger. Is your large language model knowledgeable or a choices-only cheater? In Sha Li, Manling Li, Michael JQ Zhang, Eunsol Choi, Mor Geva, Peter Hase, and Heng Ji (eds.), Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pp. 15–26, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.knowllm-1.2. URL https://aclanthology.org/2024.knowllm-1.2/. - Balepur et al. (2024) Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. Artifacts or abduction: How do LLMs answer multiple-choice questions without the question? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10308–10330, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.555. URL https://aclanthology.org/2024.acl-long.555/. - Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397–2430. PMLR, 2023. - Biderman et al. (2024) Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.14782, 2024. - Blili-Hamelin et al. (2025) Borhane Blili-Hamelin, Christopher Graziul, Leif Hancox-Li, Hananel Hazan, El-Mahdi El-Mhamdi, Avijit Ghosh, Katherine Heller, Jacob Metcalf, Fabricio Murai, Eryk Salvaggio, et al. Stop treatingagi’as the north-star goal of ai research. arXiv preprint arXiv:2502.03689, 2025. - Chen (2023) Edwin Chen. Hellaswag or hellabad? 36% of this popular llm benchmark contains errors, 2023. URL https://www.surgehq.ai/blog/hellaswag-or-hellabad-36-of-this-popular-llm-benchmark-contains-errors. Accessed: 2025-03-27. - Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. - Cronbach & Meehl (1955) LJ Cronbach and PE Meehl. Construct validity in psychological tests. Psychological Bulletin, 52(4):281–302, 1955. - Davis (2023) Ernest Davis. Benchmarks for automated commonsense reasoning: A survey. ACM Computing Surveys, 56(4):1–41, 2023. - DeepSeek-AI et al. (2025) DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948. - Gao (2022) Leo Gao. Multiple choice normalization, 2022. URL https://blog.eleuther.ai/multiple-choice-normalization/. - Gema et al. (2024) Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? arXiv preprint arXiv:2406.04127, 2024. - Google (2025) Google. Introducing gemma 3: Advancing open models for developers, March 2025. URL https://blog.google/technology/developers/gemma-3/. - Grattafiori et al. (2024) Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. - Gupta et al. (2024) Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. Changing answer order can decrease mmlu accuracy. arXiv preprint arXiv:2406.19470, 2024. - Habba et al. (2025) Eliya Habba, Ofir Arviv, Itay Itzhak, Yotam Perlitz, Elron Bandel, Leshem Choshen, Michal Shmueli-Scheuer, and Gabriel Stanovsky. Dove: A large-scale multi-dimensional predictions dataset towards meaningful llm evaluation. arXiv preprint arXiv:2503.01622, 2025. - Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ. - Hu & Frank (2024) Jennifer Hu and Michael Frank. Auxiliary task demands mask the capabilities of smaller language models. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=U5BUzSn4tD. - Hu & Levy (2023) Jennifer Hu and Roger Levy. Prompting is not a substitute for probability measurements in large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5040–5060, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.306. URL https://aclanthology.org/2023.emnlp-main.306/. - IBM (2024) IBM. Ibm unveils granite 3.1: Powerful performance, long context, and more, March 2024. URL https://www.ibm.com/new/announcements/ibm-granite-3-1-powerful-performance-long-context-and-more. - Krishna et al. (2017) Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pp. 706–715, 2017. - Liang et al. (2023) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=iO4LZibEqW. Featured Certification, Expert Certification. - Lyu et al. (2024) Chenyang Lyu, Minghao Wu, and Alham Aji. Beyond probabilities: Unveiling the misalignment in evaluating large language models. In Sha Li, Manling Li, Michael JQ Zhang, Eunsol Choi, Mor Geva, Peter Hase, and Heng Ji (eds.), Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pp. 109–131, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.knowllm-1.10. URL https://aclanthology.org/2024.knowllm-1.10/. - Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don‘t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797–1807, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1206. URL https://aclanthology.org/D18-1206/. - OLMo et al. (2025) Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo 2 furious, 2025. URL https://arxiv.org/abs/2501.00656. - PleIAs (2024) PleIAs. Common models: A new way to understand and explore ai models, March 2024. URL https://huggingface.co/blog/Pclanglais/common-models. - Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. - Rein et al. (2024) David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=Ti67584b98. - Shah et al. (2020) Krunal Shah, Nitish Gupta, and Dan Roth. What do we expect from multiple-choice QA systems? In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3547–3553, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.317. URL https://aclanthology.org/2020.findings-emnlp.317/. - Vendrow et al. (2025) Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. Do large language model benchmarks test reliability? arXiv preprint arXiv:2502.03461, 2025. - Yang et al. (2024) An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. - Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Màrquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1472. URL https://aclanthology.org/P19-1472/. Appendix A Models | Model | Size | HuggingFace repository | | --- | --- | --- | | LLaMA 3.2 | 1B | meta-llama/Llama-3.2-1B | | (Grattafiori et al., 2024) | | | | Qwen 2.5 (Yang et al., 2024) | 1.5B | Qwen/Qwen2.5-1.5B | | Granite 3.1 (IBM, 2024) | 1B | ibm-granite/granite-3.1-1b-a400m-base | | Gemma 3.1 (Google, 2025) | 1B | google/gemma-3-1b-pt.csv | | SmolLM2 (Allal et al., 2025) | 1.7B | HuggingFaceTB/SmolLM2-1.7B | | DeepSeek-R1 | 1.5B | deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B | | (DeepSeek-AI et al., 2025) | | | | Pythia (Biderman et al., 2023) | 1B | EleutherAI/pythia-1b | | Pleias 1.0 (PleIAs, 2024) | 1B | PleIAs/Pleias-1b-Preview | | Qwen 2.5 (Yang et al., 2024) | 32B | Qwen/Qwen2.5-32B-Instruct | | OLMo 2 (OLMo et al., 2025) | 32B | allenai/OLMo-2-0325-32B-Instruct | Table 5: Model names and their corresponding HuggingFace repositories. In Table 5, we present the models we used for evaluations in the paper with their corresponding HuggingFace repository paths. Appendix B Generation Prompt We use the following prompt to evaluate the models by generation: Generation Prompt You are given a situation followed by four possible endings. Choose the most appropriate ending by selecting the corresponding number. Respond only with the number of the correct answer. Context: Roof shingle removal: A man is sitting on a roof. He 1. is using wrap to wrap a pair of skis. 2. is ripping level tiles off. 3. is holding a Rubik’s cube. 4. starts pulling up roofing on a roof. Answer: Appendix C Annotations Here we report the two prompts we used for annotations with Claude 3.5 Sonnet (version claude-3-5-sonnet-20241022). In both prompts, we indicate the answer that is supposed to be correct. In the first prompt, we have a part about the equal plausibility of answers, but we did not use the results from this annotation label as we did it more verbosely in the second round of annotations. In the second annotation prompt, we present HellaSwag questions as complete texts to change the familiar benchmark structure and force the model to understand the options as complete texts. For the examples in the second prompt, we use the questions from the HellaSwag training set. First Round of Annotations I am evaluating the quality of a multiple-choice task, where given the prompt, you are supposed to pick the best sentence completion. I will provide the prompt and four possible completion options, with the correct one labeled. For each set, return all applicable labels (in brackets, separated by semicolons) based on the descriptions below: • [ungrammatical, prompt] — there is a grammatical error or typo in the question. It’s okay if the last part of the prompt is incomplete, that’s part of the task. • [ungrammatical, correct answer] — there is a grammatical error or typo in the correct answer • [ungrammatical, incorrect answer(s)] — there is a grammatical error or typo in one or more of the incorrect answer choices • [nonsense, prompt] — ignoring grammatical errors or typos, the question does not make sense or is not coherent • [nonsense, correct answer] — ignoring grammatical errors or typos, the correct answer does not make sense or is not coherent • [nonsense, incorrect answer(s)] — ignoring grammatical errors or typos, one or more of the answer choices does not make sense or is not coherent • [plausibility, answers] — two or more of the answer choices are equally plausible or implausible, therefore the correct answer is not necessarily the best answer, given your understanding of the prompt • [high quality] - there are no issues with the prompt or answer choices Example Output: If the prompt has a grammatical error and the correct answer is nonsensical, return: [ungrammatical, prompt; nonsense, correct answer] Sentence: Roof shingle removal: A man is sitting on a roof. He Completions: a) is using wrap to wrap a pair of skis. b) is ripping level tiles off. c) is holding a Rubik’s cube. d) starts pulling up roofing on a roof. (Labeled as correct) Labels: Second Round of Annotations I will give you four short texts that start similarly but have different endings. I will also indicate which text is considered correct—i.e., the one with the most logical and plausible ending. However, in some cases, the text labeled as correct may not actually be the best one, or there may be several other options that are just as good. It may also happen that all the texts, including the correct one, are implausible and nonsensical. I would like you to answer the following four questions: 1. Is the text labeled as correct in fact the best one, or do you have a better option? Reply with the letter of the best sentence in your opinion. 2. Are there other options that are equally plausible/make as much sense as the answer labeled as correct? Reply with the letters corresponding to those texts, separated by commas, or “None” if there are no equally good options. 3. Is it the case that all of the answer choices are implausible or nonsensical? Answer “Yes” or “No.” 4. Which is the worst of the answer choices? Reply with one letter of the worst option. Here’s an example: A) Triple jump: As he reaches the dirt section, he does three jumps. On his final jump, he extends his legs to try and jump as far as possible. The video ends with closing captions. B) Triple jump: As he reaches the dirt section, he does three jumps. On his final jump, he extends his legs to try and jump as far as possible. The video ends after he lands and he is shown smacking the ground. (Labeled as correct) C) Triple jump: As he reaches the dirt section, he does three jumps. On his final jump, he extends his legs to try and jump as far as possible. The video ends with the intro credits shown. D) Triple jump: As he reaches the dirt section, he does three jumps. On his final jump, he extends his legs to try and jump as far as possible. The video ends with more dirt. The answer should be: 1. B 2. A 3. No 4. D Here’s another example, where all of the options are implausible: A) Cheerleading: A cheerleading team begins to hold up posters as their mascot runs behind them. They eventually get the dogs on the field and push them around. B) Cheerleading: A cheerleading team begins to hold up posters as their mascot runs behind them. They begin to perform an athlete’s routine on the field. C) Cheerleading: A cheerleading team begins to hold up posters as their mascot runs behind them. They then begin to do a routine, and some of the girls run with streamers as the rest hold up the girls for their stunt. (Labeled as correct) D) Cheerleading: A cheerleading team begins to hold up posters as their mascot runs behind them. They begin to perform acoustic songs. The answer should be: 1. C 2. None 3. Yes 4. A Here is the actual question for you to evaluate: A) Roof shingle removal: A man is sitting on a roof. He is using wrap to wrap a pair of skis. B) Roof shingle removal: A man is sitting on a roof. He is ripping level tiles off. C) Roof shingle removal: A man is sitting on a roof. He is holding a Rubik’s cube. D) Roof shingle removal: A man is sitting on a roof. He starts pulling up roofing on a roof. (Labeled as correct) Your Answer: Appendix D Answer Lengths D.1 Other Log-Likelihood Variants <details> <summary>x5.png Details</summary>  ### Visual Description ## Scatter Plot: Total Log-Likelihood vs. Text Length (Bytes) ### Overview The image is a scatter plot comparing the relationship between text length (bytes) and total log-likelihood for two data sources: WikiHow (green) and ActivityNet (red). The plot includes two trend lines (one per source) and density shading for data point distributions. ### Components/Axes - **X-axis**: Text Length (Bytes) - Range: 100 to 700 bytes - Labels: Incremented by 100 bytes (100, 200, ..., 700) - **Y-axis**: Total Log-Likelihood - Range: -500 to -100 - Labels: Incremented by 100 units (-500, -400, ..., -100) - **Legend**: - Top-right corner - Labels: - Green: WikiHow - Red: ActivityNet ### Detailed Analysis 1. **WikiHow (Green)** - **Trend Line**: Starts near (100, -100) and ends at (700, -400). - **Slope**: Steeper decline (-0.4 log-likelihood per byte). - **Data Points**: - Dense clustering between 100–300 bytes (log-likelihood: -100 to -250). - Sparse points at 500–700 bytes (log-likelihood: -350 to -400). 2. **ActivityNet (Red)** - **Trend Line**: Starts near (100, -100) and ends at (500, -400). - **Slope**: Gradual decline (-0.2 log-likelihood per byte). - **Data Points**: - Dense clustering between 100–400 bytes (log-likelihood: -100 to -300). - Fewer points beyond 400 bytes (log-likelihood: -350 to -400). ### Key Observations - **Text Length vs. Log-Likelihood**: - WikiHow texts extend to 700 bytes, while ActivityNet texts max at ~500 bytes. - WikiHow’s log-likelihood decreases faster with text length (steeper slope). - **Distribution**: - WikiHow has more variability in log-likelihood at longer text lengths. - ActivityNet’s data is concentrated in shorter text ranges. ### Interpretation The data suggests that WikiHow’s content exhibits a stronger negative correlation between text length and log-likelihood, potentially indicating lower coherence or relevance in longer texts. ActivityNet’s texts are shorter and maintain higher log-likelihoods longer, implying more consistent quality or relevance in shorter passages. The steeper slope for WikiHow may reflect challenges in maintaining relevance as text length increases. </details> (a) Joint distributions of total log-likelihood and text length in bytes for different question sources in the Hellaswag validation set. <details> <summary>x6.png Details</summary>  ### Visual Description ## Scatter Plot: Byte-Mean Log-Likelihood vs. Text Length (Bytes) ### Overview The image is a scatter plot comparing the relationship between text length (bytes) and byte-mean log-likelihood for two data sources: WikiHow (green) and ActivityNet (red). Two trend lines are overlaid on the data points, and shaded regions indicate data density distributions. --- ### Components/Axes - **X-Axis**: "Text Length (Bytes)" ranging from 100 to 700 bytes in increments of 100. - **Y-Axis**: "Byte-Mean Log-Likelihood" ranging from -1.8 to -0.4 in increments of 0.2. - **Legend**: Located in the bottom-right corner, with: - **Green**: WikiHow - **Red**: ActivityNet - **Trend Lines**: - **Green Line**: WikiHow (slightly above red line) - **Red Line**: ActivityNet (slightly below green line) - **Shaded Regions**: - **Green**: WikiHow data density (darker in upper-right quadrant) - **Red**: ActivityNet data density (darker in lower-left quadrant) --- ### Detailed Analysis 1. **WikiHow (Green)**: - **Data Points**: Clustered densely in the upper-right quadrant (text lengths: 400–700 bytes; log-likelihood: -0.4 to -0.6). - **Trend Line**: Positive slope, starting near (-1.0, 100 bytes) and ending near (-0.4, 700 bytes). - **Density**: Highest concentration between 400–600 bytes. 2. **ActivityNet (Red)**: - **Data Points**: Clustered densely in the lower-left quadrant (text lengths: 100–300 bytes; log-likelihood: -1.2 to -0.8). - **Trend Line**: Positive slope, starting near (-1.2, 100 bytes) and ending near (-0.6, 500 bytes). - **Density**: Highest concentration between 100–300 bytes. 3. **Overlap**: - Shaded regions overlap between 200–400 bytes, with mixed densities (green and red). --- ### Key Observations 1. **WikiHow Dominates Longer Texts**: - WikiHow texts are longer (400–700 bytes) and have higher log-likelihood (-0.4 to -0.6) compared to ActivityNet. 2. **ActivityNet Favors Shorter Texts**: - ActivityNet texts are shorter (100–300 bytes) with lower log-likelihood (-1.2 to -0.8). 3. **Trend Lines**: - Both lines show a positive correlation between text length and log-likelihood, but WikiHow’s line is consistently higher. 4. **Data Density**: - WikiHow’s density decreases sharply after 600 bytes; ActivityNet’s density tapers off after 300 bytes. --- ### Interpretation - **Structural Differences**: WikiHow’s higher log-likelihood suggests more predictable or structured text (e.g., instructional content), while ActivityNet’s lower log-likelihood may reflect variability or noise in its data (e.g., user-generated content). - **Overlap Implications**: The middle range (200–400 bytes) shows mixed performance, indicating some texts from both sources share similar characteristics. - **Anomalies**: A single outlier at ~100 bytes (red) in ActivityNet has a log-likelihood near -1.8, suggesting an outlier in shorter texts. The data highlights a clear divergence in text properties between the two sources, with WikiHow favoring longer, more coherent texts and ActivityNet dominated by shorter, less predictable content. </details> (b) Joint distributions of byte-normalized log-likelihood and text length in bytes for different question sources in the Hellaswag validation set. Figure 5: Joint distributions of likelihoods and lengths of the HellaSwag validation set questions for different variants of likelihood computation. The lines are meant to show the trend, we are not assuming linear regression to be significant for these data. In order to compare our mean log-likelihood evaluations with the ones used in lm-evaluation-harness (Biderman et al., 2024). In Figure 5, we show joint distributions for total log-likelihood: $$ \mathcal{L}_{t}=\sum_{t\in\mathcal{V}}\log\mathrm{P}(y_{t}\mid x_{<t}), \tag{2} $$ and byte-normalized log-likelihood: $$ \mathcal{L}_{b}=\frac{1}{\mathrm{B}}\sum_{t\in\mathcal{V}}\log\mathrm{P}(y_{t}% \mid x_{<t}), \tag{3} $$ where $y_{t}$ is the ground truth token at position $t$ , $x_{<t}$ is the preceding sequence of tokens, $\mathrm{B}$ is the sequence length in bytes, and $\mathcal{V}$ is the set of valid (non-special) token positions. As with the mean log-likelihood, there is a visible correlation between answer length and log-likelihood in both cases (negative, in Figure 5(a)). D.2 Lengths distribution <details> <summary>x7.png Details</summary>  ### Visual Description ## Histogram: Distribution of Relative Length Differences ### Overview The image displays a histogram visualizing the distribution of "Relative Length Difference" values. The x-axis represents the relative length difference (0.0 to 0.7), and the y-axis represents frequency (0 to 3000). A red dashed line labeled "Median: 0.17" is overlaid on the chart. ### Components/Axes - **X-axis**: "Relative Length Difference" with tick marks at 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7. - **Y-axis**: "Frequency" with tick marks at 0, 500, 1000, 1500, 2000, 2500, 3000. - **Legend**: Located in the top-right corner, indicating the red dashed line corresponds to the median value (0.17). - **Bars**: Orange-colored histogram bars representing frequency counts. ### Detailed Analysis - **Bars**: - **0.0–0.1**: Frequency ~400 (short bar). - **0.1–0.15**: Frequency ~3000 (tallest bar). - **0.15–0.2**: Frequency ~2500 (second tallest bar). - **0.2–0.3**: Frequency ~1400 (moderate bar). - **0.3–0.4**: Frequency ~600 (smaller bar). - **0.4–0.5**: Frequency ~200 (very small bar). - **0.5–0.7**: Frequency ~50 (minimal bar). - **Median Line**: Red dashed line at x = 0.17, intersecting the histogram near the peak. ### Key Observations 1. **Peak Distribution**: The highest frequency (~3000) occurs between 0.1 and 0.15, indicating the most common relative length differences fall in this range. 2. **Median Position**: The median (0.17) is slightly right of the peak, suggesting a right-skewed distribution. 3. **Rapid Decline**: Frequencies drop sharply after 0.2, with minimal counts beyond 0.4. 4. **Skewness**: The distribution is asymmetric, with a long tail extending toward higher relative length differences. ### Interpretation The data suggests that most relative length differences cluster around 0.1–0.2, with the median (0.17) reflecting the central tendency. The right skew implies that while most values are concentrated in the lower range, a subset of measurements exhibits larger discrepancies. This could indicate variability in a process or system being measured, such as manufacturing tolerances or biological measurements. The sharp decline after 0.2 suggests that extreme values are rare, but their presence may warrant further investigation into outliers or measurement errors. </details> Figure 6: Distribution of differences between the longest and the shortest answer options relative to the longest option length. We compute relative length differences as differences between the longest and the shortest answer options relative to the length of the longest answer option. We show the distribution of relative length differences in Figure 6. Appendix E Placeholder-Prompt Evaluation | Llama 3.2 Gemma 3 Qwen 2.5 | 1B 1B 1.5B | 0.67 0.69 0.65 | 0.43 0.40 0.43 | 0.25 0.29 0.22 | 0.20 0.19 0.23 | 0.06 0.05 0.05 | 0.08 0.07 0.07 | | --- | --- | --- | --- | --- | --- | --- | --- | | SmolLM2 | 1.7B | 0.64 | 0.46 | 0.18 | 0.23 | 0.06 | 0.07 | | Granite 3.1 | 1B | 0.71 | 0.44 | 0.27 | 0.18 | 0.04 | 0.07 | | Pythia | 1B | 0.55 | 0.26 | 0.30 | 0.19 | 0.10 | 0.15 | | PleIAs 1.0 | 1B | 0.59 | 0.24 | 0.35 | 0.17 | 0.09 | 0.14 | | DeepSeek-R1 | 1.5B | 0.62 | 0.25 | 0.37 | 0.18 | 0.07 | 0.14 | Table 6: Agreement between full-prompt and placeholder-prompt evaluation. Agreement means that a model gives equal predictions in both evaluation modes. We separately report fractions for agreement types (a model gives equal correct or incorrect predictions) and disagreement types (the model gives correct prediction only in either full- or placeholder-prompt scenario or gives different incorrect answers). Along with our zero-prompt evaluation (Section 4.4), we perform a similar experiment with placeholder prompts. We replace the question prompt with the following text: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi vel venenatis dui. Pellentesque sed cursus massa. The results of this evaluation are presented in Table 6. Here, we also observe high level of agreement between full-prompt and lorem-prompt evaluations. Furthermore, in quite a lot of cases, this helps to solve the question while the model found not do it with the normal question prompt (see column Lorem ✓, up to 10% for Pythia). The reason for this could be that the placeholder prompt gives more randomness to the evaluation, and the model has more chance to guess in these questions than with the original question context. Appendix F Accuracy by Source <details> <summary>x8.png Details</summary>  ### Visual Description ## Bar Chart: Model Accuracy Comparison Across Evaluation Methods ### Overview The chart compares the accuracy of various large language models (LLMs) under three evaluation methods: Full-Prompt, Zero-Prompt, and Random. Models are grouped by name on the x-axis, with accuracy (0–0.8) on the y-axis. Each model has three bars representing the evaluation methods, with distinct patterns/colors for clarity. ### Components/Axes - **X-Axis (Model)**: Labeled "Model," listing 10 LLMs: Qwen 2.5 32B, OLMo 2 32B, Llama 3.2 1B, Gemma 3 1B, Qwen 2.5 1.5B, SmolLM2 1.7B, Granite 3.1 1B, Pythia 1B, PleiAs 1.0 1B, DeepSeek-R1 1.5B. - **Y-Axis (Accuracy)**: Labeled "Accuracy," scaled from 0.0 to 0.8 in increments of 0.2. - **Legend**: Positioned top-right, with three entries: - **Full-Prompt**: Solid gray bars. - **Zero-Prompt**: Striped gray bars. - **Random**: Dashed gray bars. ### Detailed Analysis - **Qwen 2.5 32B**: - Full-Prompt: ~0.7 - Zero-Prompt: ~0.4 - Random: ~0.3 - **OLMo 2 32B**: - Full-Prompt: ~0.75 - Zero-Prompt: ~0.45 - Random: ~0.35 - **Llama 3.2 1B**: - Full-Prompt: ~0.65 - Zero-Prompt: ~0.5 - Random: ~0.45 - **Gemma 3 1B**: - Full-Prompt: ~0.6 - Zero-Prompt: ~0.5 - Random: ~0.4 - **Qwen 2.5 1.5B**: - Full-Prompt: ~0.7 - Zero-Prompt: ~0.55 - Random: ~0.45 - **SmolLM2 1.7B**: - Full-Prompt: ~0.75 - Zero-Prompt: ~0.55 - Random: ~0.45 - **Granite 3.1 1B**: - Full-Prompt: ~0.6 - Zero-Prompt: ~0.5 - Random: ~0.4 - **Pythia 1B**: - Full-Prompt: ~0.45 - Zero-Prompt: ~0.4 - Random: ~0.35 - **PleiAs 1.0 1B**: - Full-Prompt: ~0.4 - Zero-Prompt: ~0.35 - Random: ~0.3 - **DeepSeek-R1 1.5B**: - Full-Prompt: ~0.4 - Zero-Prompt: ~0.35 - Random: ~0.3 ### Key Observations 1. **Full-Prompt Dominance**: Full-Prompt consistently outperforms Zero-Prompt and Random across all models, with accuracies ranging from ~0.4 (PleiAs) to ~0.75 (SmolLM2). 2. **Zero-Prompt Variability**: Zero-Prompt shows moderate performance, with accuracies between ~0.35 (PleiAs) and ~0.55 (Qwen 2.5 1.5B). 3. **Random Baseline**: Random evaluations are uniformly the lowest, clustering around ~0.3–0.45. 4. **Model-Specific Trends**: Larger models (e.g., Qwen 2.5 32B, OLMo 2 32B) generally achieve higher accuracies than smaller models (e.g., Pythia 1B, PleiAs 1.0 1B). ### Interpretation The data demonstrates that **explicit prompting (Full-Prompt)** significantly improves model accuracy compared to **implicit prompting (Zero-Prompt)** and **random evaluations**. This suggests that structured instructions enhance model performance, likely by reducing ambiguity in task interpretation. Larger models (e.g., 32B parameters) outperform smaller ones, indicating scalability benefits. However, the gap between Full-Prompt and Zero-Prompt narrows for smaller models (e.g., Pythia 1B), suggesting they may be less sensitive to prompting style. The Random method’s consistent underperformance underscores the necessity of structured evaluations for reliable benchmarking. ### Spatial Grounding & Trend Verification - **Legend Placement**: Top-right ensures clear association between colors/patterns and evaluation methods. - **Bar Order**: Full-Prompt (solid) > Zero-Prompt (striped) > Random (dashed) aligns with legend order. - **Trend Consistency**: Full-Prompt bars are taller than Zero-Prompt, which exceed Random across all models, confirming the expected hierarchy. ### Notable Outliers - **SmolLM2 1.7B**: Highest Full-Prompt accuracy (~0.75), suggesting strong performance with explicit instructions. - **PleiAs 1.0 1B**: Lowest Full-Prompt accuracy (~0.4), indicating potential architectural or training limitations. </details> Figure 7: Accuracy of the models evaluated with log-likelihood maximization split by the source of questions. In Figure 7, we show the differences in accuracies by question sources. ActivityNet part of the data, which is a suspect for lower-quality questions in HellaSwag, is more challenging to the models. However, taking all our results into account, this complexity is not a compelling benchmark feature, but rather low-quality data that produces unwanted noise in the scores. Interestingly, the zero-prompt accuracy drop in ActivityNet is also larger than that for the WikiHow. One of the reasons for this might be that WikiHow has generally longer options, from which it is easier to get a general understanding of the situation. Appendix G GoldenSwag In Table 7, we present the complete set of stages used for the filtering of the HellaSwag validation set, which consists of 10042 questions. For each filter, we report the number of questions in HellaSwag that fit the filtering criterion, the number of questions that we actually remove at this stage (that were not removed in previous stages), and the number of questions that are left in HellaSwag after each filtering stage. | Toxic content | 6 | 6 | 10036 | | --- | --- | --- | --- | | Nonsense or ungrammatical prompt | 4065 | 4064 | 5972 | | Nonsense or ungrammatical correct answer | 711 | 191 | 5781 | | Ungrammatical incorrect answers | 3953 | 1975 | 3806 | | Wrong answer | 370 | 89 | 3717 | | All options are nonsense | 409 | 23 | 3694 | | Multiple correct options | 2121 | 583 | 3111 | | Relative length difference $>0.3$ | 802 | 96 | 3015 | | Length difference $∈(0.15,0.3]$ and longest is correct | 1270 | 414 | 2601 | | Zero-prompt core $≥ 0.3$ | 3963 | 1076 | 1525 | Table 7: Filtering steps for the GoldenSwag subset. For each filter, we report the number of questions fitting for the filter (# to remove), the number of questions actually removed at this stage (# removed), and the number of questions left in the HellaSwag validation set after this stage (# left). After the filtering, almost all of the questions are sourced from WikiHow — 1498 (98.2%), which further proves that the ActivityNet part of the data contains mostly problematic examples with poor grammar and language choices, mostly attributed to artifacts from synthetic generation by a model that does not meet modern standards.