# RuozhiBench: Evaluating LLMs with Logical Fallacies and Misleading Premises
## Abstract
Recent advances in large language models (LLMs) have shown that they can answer questions requiring complex reasoning. However, their ability to identify and respond to text containing logical fallacies or deliberately misleading premises remains less studied. To address this gap, we introduce RuozhiBench, a bilingual dataset comprising 677 carefully curated questions that contain various forms of deceptive reasoning, meticulously crafted through extensive human effort and expert review. In a comprehensive evaluation of 17 LLMs from 5 Series over RuozhiBench using both open-ended and two-choice formats, we conduct extensive analyses on evaluation protocols and result patterns. Despite their high scores on conventional benchmarks, these models showed limited ability to detect and reason correctly about logical fallacies, with even the best-performing model, Claude-3-haiku, achieving only 62% accuracy compared to the human of more than 90%. Data and code available at: https://github.com/LibrAIResearch/ruozhibench Data license: CC-BY-NC license.
RuozhiBench: Evaluating LLMs with Logical Fallacies and Misleading Premises
Zenan Zhai 1 Hao Li 1 Xudong Han 1,2 Zhenxuan Zhang 1 Yixuan Zhang 1,2 Timothy Baldwin 1,2,3 Haonan Li 1,2 1 LibrAI 2 MBZUAI 3 The University of Melbourne
UTF8gbsn
## 1 Introduction
Large language models (LLMs) have rapidly advanced in recent years, demonstrating impressive capabilities across a wide range of tasks (Zhang et al., 2022; Scao et al., 2022; Touvron et al., 2023; Bai et al., 2023a; DeepSeek-AI et al., 2025). Benchmarking plays a crucial role in assessing their performance, with existing evaluations falling into two main categories. The first category includes benchmarks with predefined answers, such as MMLU (Hendrycks et al., 2021a; Li et al., 2023a) and ARC (Clark et al., 2018), which assess factual knowledge and reasoning. However, issues like data contamination (Carlini et al., 2022; Sainz et al., 2023) and potential model cheating have been widely reported. The second category evaluates open-ended responses, focusing on alignment with human preferences. AlpacaEval (Li et al., 2023b) is a widely-used automatic evaluator that measures model performance by comparing outputs against a reference model, typically using GPT-4 as the judge. While effective for general instruction following, it is not designed to assess how models handle deceptive or misleading inputs.
However, in real-world scenarios, texts often contain logical fallacies, misleading premises, or intentional ambiguities that can trap the unwary. When faced with such deceptive inputs, current models often fail to identify the underlying fallacies and provide responses that reveal their limitations in logical reasoning (Figure 1). Despite these types of deliberately misleading texts being a crucial test of true language understanding, there is currently no benchmark specifically designed to evaluate how well models handle such scenarios.
<details>
<summary>x1.png Details</summary>
### Visual Description
\n
## Text Block: Cow and Human Tongue Interaction
### Overview
The image presents a block of text discussing the reciprocal tasting interaction between cow and human tongues, and a concluding statement about the inability of a consumed beef tongue to taste. The text is visually segmented with colored backgrounds and includes illustrative icons.
### Components/Axes
The image is structured into three main sections:
1. **Initial Explanation:** A block of text explaining the initial premise of reciprocal tasting. Background color: White.
2. **Detailed Explanation:** A longer block of text elaborating on the chemical interactions involved in tasting. Background color: White.
3. **Analysis:** A concluding statement in a yellow rectangle with the word "Analysis" prominently displayed. Background color: Yellow.
Additionally, there are two icons:
* Top-right: A grey icon depicting a human head and shoulders.
* Bottom-left: A green swirl-like icon.
### Detailed Analysis or Content Details
Here's a transcription of the text content:
**Section 1:**
"Both cow tongues and human tongues have taste functions. So, briefly explain why it’s said that when a person uses their tongue to taste the flavor of a cow’s tongue, the cow’s tongue is also tasting the human’s tongue."
**Section 2:**
"When a person uses their tongue to taste a cow’s tongue, it is said that both the human’s tongue and the cow’s tongue are tasting each other. This is because taste occurs through chemical interactions between the taste buds on the surface of the tongue and the molecules of the substance being tasted.
…
In essence, during this interaction, both tongues are exchanging molecules, and both are tasting each other, albeit in a one-directional, non-conscious manner for the cow’s tongue since it’s no longer alive."
**Section 3:**
"A consumed beef tongue cannot taste anything because it is no longer a functioning organ of a living animal."
### Key Observations
The text presents a somewhat counterintuitive idea – that a cow's tongue can "taste" a human's tongue during consumption. The text clarifies this is due to molecular exchange and chemical interactions. The final statement provides a logical conclusion, stating that a consumed tongue lacks the biological capacity to taste. The use of "…" suggests omitted text in the second section.
### Interpretation
The text explores the biological process of taste in an unusual context. It highlights the reciprocal nature of chemical interactions, even in a scenario involving a non-living organ. The text is attempting to explain a common, but often unconsidered, phenomenon. The inclusion of the "Analysis" section suggests a deliberate attempt to provide a definitive answer to the initial question. The text is not presenting data in a quantitative sense, but rather a conceptual explanation. The icons appear to be purely decorative and do not contribute to the informational content. The overall message is that taste is a chemical process involving molecular exchange, and that a dead organ cannot experience sensation.
</details>
Figure 1: An example of a question from RuozhiBench and response from GPT-4o.
<details>
<summary>x2.png Details</summary>
### Visual Description
\n
## Diagram: AI-Powered Text Processing & Irrationality Detection Pipeline
### Overview
This diagram illustrates a pipeline for processing text data, identifying potential irrationalities, and categorizing the types of irrationalities detected. The pipeline begins with data crawling, proceeds through translation and human review, irrationality generation, question categorization, and culminates in response collection and model refinement. The diagram uses icons and text boxes to represent each stage, with arrows indicating the flow of information.
### Components/Axes
The diagram is structured linearly from left to right, representing the sequential steps in the pipeline. Key components include:
* **Data Crawling (86.3k):** Represents the initial data acquisition stage.
* **Translation & Human Check:** Involves translating the crawled data and verifying its accuracy.
* **Irrationality Generation:** The stage where potential irrationalities are identified.
* **Question Categorize:** Classifies the detected irrationalities into predefined categories.
* **Response Collection:** Gathers responses related to the identified irrationalities.
* **RuoZhiBench-Gen & RuoZhiBench-MC:** Represent model refinement stages.
### Detailed Analysis or Content Details
**1. Data Crawling (Leftmost Component):**
* A blue speech bubble icon labeled "Data Crawling (86.3k)" indicates the source of the data.
* Below the icon are three text blocks:
* **Chinese Text 1:** `我在开车时撞死了人，现在我的车身盖上全是血，请问我应该到哪里洗车？`
* **English Translation 1:** "I hit and killed someone while driving, and now the hood of my car is covered in blood. Where should I go to wash my car?"
* **Chinese Text 2:** `我吃了好几张卡也没吐钱，是不是我的姿势不对吗？`
* **English Translation 2:** "I ate several bank cards but didn't spit out the money. Is it because my eating posture is wrong?"
**2. Translation & Human Check (Center-Left):**
* A light-blue icon labeled "Translation & Human Check"
* A speech bubble with a question mark inside contains the following text:
* "The ATM will spit out money after taking a bank card. Why didn't it spit out money after taking several bank cards? Is my taking posture wrong?"
**3. Irrationality Generation (Center):**
* A green brain icon labeled "Irrationality Generation".
* A speech bubble with a question mark inside contains the following text:
* "The ATM spits out cash after taking the bank card. So why haven't I spit out any money after swallowing several bank cards? Am I doing it wrong?"
**4. Question Categorize (Center-Right):**
* A yellow icon labeled "Question Categorize".
* A list of categories is provided:
1. Logical error
2. Common sense misunderstandings
3. Erroneous assumption
4. Scientific misconceptions
5. Absurd imagination
6. Others
**5. Response Collection (Right-Center):**
* A dark-yellow icon labeled "Response Collection".
* An AI icon with three dots represents the AI model.
* A speech bubble icon represents the response.
**6. Model Refinement (Rightmost):**
* Two green icons labeled "RuoZhiBench-Gen" and "RuoZhiBench-MC" represent model refinement stages.
**Additional Text:**
* A text block near the top-right states: "People who swallow bank cards will not receive cash."
* A text block near the bottom-left states: `人们直接吞下卡后就会吐出钱来，为什么我吃了几张卡后还吐不出来？难道是我的姿势不对吗？`
* **English Translation:** "People directly swallow the card and then spit out money. Why haven't I spit out any money after eating several cards? Is it because my eating posture is wrong?"
### Key Observations
* The diagram highlights the absurdity of the input questions, which demonstrate a lack of understanding of how ATMs and bank cards function.
* The pipeline is designed to identify and categorize these irrationalities.
* The inclusion of both Chinese text and English translations suggests a focus on cross-lingual processing.
* The "RuoZhiBench" components indicate a focus on benchmarking and improving the performance of the AI model.
### Interpretation
The diagram illustrates a system designed to detect and categorize illogical or irrational statements. The examples provided – questions about swallowing bank cards and washing a blood-covered car – are intentionally absurd, demonstrating the system's ability to identify statements that violate common sense or logical reasoning. The pipeline's structure suggests a multi-stage process: first, gathering data; second, ensuring its accuracy through translation and human review; third, identifying potential irrationalities; fourth, categorizing those irrationalities; and finally, using the categorized data to refine the AI model. The "RuoZhiBench" components suggest a focus on evaluating and improving the model's performance on this type of task. The diagram implies a research effort aimed at building AI systems that can better understand and respond to human language, even when that language is illogical or nonsensical. The inclusion of Chinese text suggests a broader application beyond English-speaking contexts.
</details>
Figure 2: The creation process for RuozhiBench, consisting of three main parts: data filtering (left), translation and review (middle), and annotation (right).
To address this gap, we introduce RuozhiBench, a novel benchmark designed to evaluate the ability of models to identify and reason about deceptive inputs and logical fallacies. RuozhiBench comprises 677 questions sourced from the Chinese forum Ruozhiba, a platform which contains texts that appear reasonable at first glance but contain subtle logical traps or misleading premises.
To ensure high data quality, we implemented rigorous filtering, preprocessing, and annotation. Each question was carefully reviewed and translated into English while preserving its deceptive nature. We then systematically categorized the questions into six distinct types, ensuring clear and consistent labeling. See Section 2 for more details.
To further enhance reliability, we designed a multi-step annotation process involving both human validation and automated checks. Only questions that met strict criteria for clarity, difficulty, and linguistic adaptation were included. Additionally, we conducted both rating-based and selection-based evaluations, using human judgments as a reference, and employed multiple automated evaluation methods to measure model performance.
Our preliminary experiments assessed 17 LLMs, revealing a substantial gap between model performance and the human upper bound. Despite achieving high scores on standard benchmarks, these models still lag behind humans in logical reasoning and fallacy detection. RuozhiBench is a critical step towards a more comprehensive assessment of models’ ability to handle deceptive inputs and logical fallacies.
## 2 RuozhiBench-Gen
### 2.1 Data Source
Ruozhiba (literally meaning “moron forum”) is one of the most popular online forums in the Chinese internet community, known for its collection of brain teasers, logical puzzles, and deliberately misleading questions. The forum’s content often features unconventional perspectives and clever wordplay that challenges conventional thinking patterns. Our work begins with the raw data collected by a previous project, https://github.com/Leymore/ruozhiba which compiled a comprehensive collection of threads from Ruozhiba. Note that Baidu Tieba content is freely available for academic research purposes with no legal restrictions.
| ID | Category | # Q. | Description | Example |
| --- | --- | --- | --- | --- |
| 1 | Logical Error | 142 | When the question contains logical contradictions or reasoning errors, including violations of logical rules, making it logically untenable. | I pressed the mute button on my laptop, why is the fan still so loud? |
| 2 | Commonsense Misunderstanding | 526 | The question reflects a misunderstanding of basic common sense or universally accepted facts, usually involving incorrect interpretations of daily knowledge. | Is it better to prevent tooth decay by applying toothpaste directly to the teeth without brushing before going to bed? |
| 3 | Erroneous Assumption | 471 | The question is based on one or more incorrect assumptions, leading to inaccuracies in the question or its answer. | If you stretch your leg to trip a moving car, will it overturn? |
| 4 | Scientific Misconception | 30 | The question involves misunderstandings of scientific principles or knowledge, including incorrect interpretations of scientific theories or methods. | Can you avoid drone thermal imaging bombings by eating only low-calorie foods? |
| 5 | Absurd Imagination | 463 | The question setting is contrary to reality or common sense, containing impossible or illogical elements. | If you suck away all the clouds, will it stop raining and be sunny forever? |
| 6 | Others | 17 | If the provided categories do not match the current question, please choose this option. | Oxygen can rust iron. Our blood contains iron, why doesn’t our blood rust? |
Table 1: Classification schema of deceptive questions: categories, descriptions, and examples. Note that a given question may belong to multiple categories.
### 2.2 Data Screening
From the initial 86,000 entries, we first extracted over 8,000 interrogative sentences using string matching. We then implemented a rigorous filtering process with three annotators with humanities backgrounds. They first removed questions with heavy cultural dependencies or potentially negative influences, reducing the dataset to 820 entries. Through collaborative review and discussion, the annotators further filtered questions based on their suitability for English translation and comprehension, removing entries where translation would significantly alter the original meaning or logical structure. This process yielded our final dataset of 677 questions, ensuring each entry maintains its original logical challenge while being accessible to a global audience.
### 2.3 Data Annotation
After data screening, we conducted four rounds of annotation for these questions: translation review, paired question generation, irrationality analysis, and question type categorization. For all steps except paired question generation, we employed a hybrid approach combining LLM-based initial processing with human verification. The annotators involved had both bilingual (Chinese–English) and NLP backgrounds.
#### Translation Review
In the translation stage, we first used Google Translate to convert all questions from Chinese to English, followed by human review with two key objectives: (1) ensuring semantic consistency, and (2) preserving the subtle logical traps or fallacies present in the original questions. When discrepancies were found, annotators carefully rewrote the translations to maintain both the original meaning and the deliberately deceptive elements. This process required modification of 319 questions (45% of the total).
#### Paired Question Generation
To provide reference points for comparing model performance on normal vs. tricky questions, our annotators identified suitable questions from the dataset that could be naturally transformed into normal versions. For these selected questions, we created normal counterparts by removing the trap or fallacy with minimal edits, to maintaining the same format. This selective pairing process resulted in 342 normal questions, which enable us to analyze how models handle similar content with and without logical traps. An example is provided in Figure 3.
<details>
<summary>x3.png Details</summary>
### Visual Description
\n
## Text Block: Cognitive Bias Example
### Overview
The image presents a text block outlining a cognitive bias example, specifically a misunderstanding of how running affects body temperature. It includes questions in both Chinese (zh) and English (en), an explanation of the irrationality, a paired question, and a categorization of the bias.
### Components/Axes
The text block is divided into sections, each with a distinct label:
* **Question (zh):** “如果我热怎么办？ 应该去跑步吗，跑的越快，风就越大，很快就凉快了。”
* **Question (en):** “If I feel hot. Can I just go for a run? The faster I run, the stronger the wind, and I’ll cool down immediately.”
* **Irrationality:** A textual explanation of why the reasoning is flawed.
* **Paired Question:** “If I feel hot. Can I just turn on the air conditioner? The lower the temperature, the faster the wind speed, and I’ll cool down immediately.”
* **Category:** “2 (Commonsense Misunderstanding), 5 (Absurd Imagination)”
### Detailed Analysis or Content Details
* **Question (zh):** The Chinese question asks what to do if one feels hot, suggesting running as a solution, reasoning that faster running creates more wind and thus quicker cooling.
* **Question (en):** The English translation mirrors the Chinese question, proposing running as a cooling method based on wind speed.
* **Irrationality:** The explanation states that running generates more body heat, counteracting any cooling effect from the wind.
* **Paired Question:** A similar question is posed regarding using an air conditioner, with flawed reasoning about temperature and wind speed.
* **Category:** The bias is categorized as both a "Commonsense Misunderstanding" (category 2) and "Absurd Imagination" (category 5).
### Key Observations
The text block highlights a common misconception about the relationship between physical activity, body heat, and cooling mechanisms. The paired question reinforces the pattern of flawed reasoning. The categorization suggests the bias is rooted in both a lack of understanding of basic physics and a tendency towards illogical thought.
### Interpretation
This example demonstrates a cognitive bias where individuals incorrectly assume a direct correlation between activity (running or using an air conditioner) and cooling, overlooking the fact that activity generates heat. The bias likely stems from a simplified understanding of thermodynamics and a reliance on intuitive, but inaccurate, reasoning. The categorization into "Commonsense Misunderstanding" and "Absurd Imagination" suggests the bias can manifest as a simple error in judgment or a more extreme form of illogical thinking. The use of a paired question is a clever technique to highlight the consistent pattern of flawed reasoning, regardless of the specific scenario. The inclusion of both Chinese and English versions suggests the bias is not culturally specific.
</details>
Figure 3: Sample data entry format in RuozhiBench.
| Attribute | # Q. | # Q w/ Pair | Avg. len | Max len | Min len |
| --- | --- | --- | --- | --- | --- |
| Value | 677 | 342 | 18.64 | 100 | 5 |
Table 2: Statistical overview of RuozhiBench-Gen: total questions, paired questions, and question length distribution (# words).
#### Irrationality Analysis
To facilitate automatic evaluation, we generated an analysis of the logical fallacy or trick in each question. We used GPT-4o-2024-08-06 with carefully designed prompts (see Figure 10) to generate initial analyses, followed by human verification and correction to ensure accuracy.
<details>
<summary>x4.png Details</summary>
### Visual Description
\n
## Data Table: Model Performance Across Error Categories
### Overview
This image presents a data table comparing the performance of various language models across six different categories of errors: Absurd Imagination, Commonsense Misunderstanding, Erroneous Assumption, Logical Error, Others, and Scientific Misconception. A final column provides the average score across all categories. The table appears to be evaluating the models' propensity to make these types of errors, with higher scores potentially indicating a greater tendency to produce that type of error.
### Components/Axes
* **Rows:** Represent different language models: `claude-3-haiku-20240307`, `Mixtral-8x22B-v0.1`, `Llama-3.1-70B`, `Owen2.5-32B`, `Owen2.5-72B`, `gpt-4o-2024-05-13`, `Mixtral-8x7B-v0.1`, `Owen2.5-7B`, `gpt-4o-mini-2024-07-18`, `Owen2.5-3B`, `claude-3-sonnet-20240229`, `Llama2-70B`, `Llama3-8B`, `gpt-3.5-turbo`, `Llama-2-13B`, `Yi-34B-v2.2`, `Qwen1.5-110B`, `Llama-2-70B`, `Falcon-180B`, `Yi-65B-v2.2`.
* **Columns:** Represent error categories and the average score: `Absurd Imagination`, `Commonsense Misunderstanding`, `Erroneous Assumption`, `Logical Error`, `Others`, `Scientific Misconception`, `Average`.
* **Data:** Numerical scores representing the model's performance in each category.
* **Color Coding:** The table uses a color gradient to visually represent the scores. Darker shades of green indicate higher scores, while lighter shades indicate lower scores.
### Detailed Analysis / Content Details
Here's a breakdown of the data, with approximate values and trend observations. I will proceed row by row, noting trends and specific values.
* **claude-3-haiku-20240307:** Scores are generally high across all categories. `Absurd Imagination`: 61.99, `Commonsense Misunderstanding`: 61.95, `Erroneous Assumption`: 62.52, `Logical Error`: 58.25, `Others`: 63.24, `Scientific Misconception`: 66.96, `Average`: 62.00.
* **Mixtral-8x22B-v0.1:** High scores, slightly lower than Claude-3-haiku. `Absurd Imagination`: 60.85, `Commonsense Misunderstanding`: 59.42, `Erroneous Assumption`: 57.25, `Logical Error`: 57.28, `Others`: 61.27, `Scientific Misconception`: 59.82, `Average`: 58.99.
* **Llama-3.1-70B:** Scores are generally lower than the previous two models. `Absurd Imagination`: 57.70, `Commonsense Misunderstanding`: 58.54, `Erroneous Assumption`: 57.62, `Logical Error`: 55.35, `Others`: 52.45, `Scientific Misconception`: 63.69, `Average`: 57.78.
* **Owen2.5-32B:** `Absurd Imagination`: 57.56, `Commonsense Misunderstanding`: 58.65, `Erroneous Assumption`: 57.78, `Logical Error`: 57.98, `Others`: 46.57, `Scientific Misconception`: 66.07, `Average`: 57.73.
* **Owen2.5-72B:** `Absurd Imagination`: 55.09, `Commonsense Misunderstanding`: 55.08, `Erroneous Assumption`: 54.46, `Logical Error`: 56.19, `Others`: 45.59, `Scientific Misconception`: 51.79, `Average`: 54.74.
* **gpt-4o-2024-05-13:** `Absurd Imagination`: 54.93, `Commonsense Misunderstanding`: 54.90, `Erroneous Assumption`: 54.83, `Logical Error`: 52.02, `Others`: 54.41, `Scientific Misconception`: 56.85, `Average`: 54.43.
* **Mixtral-8x7B-v0.1:** `Absurd Imagination`: 55.12, `Commonsense Misunderstanding`: 53.74, `Erroneous Assumption`: 52.55, `Logical Error`: 51.37, `Others`: 44.61, `Scientific Misconception`: 58.33, `Average`: 53.35.
* **Owen2.5-7B:** `Absurd Imagination`: 46.10, `Commonsense Misunderstanding`: 47.05, `Erroneous Assumption`: 46.71, `Logical Error`: 44.82, `Others`: 38.73, `Scientific Misconception`: 50.89, `Average`: 46.27.
* **gpt-4o-mini-2024-07-18:** `Absurd Imagination`: 44.18, `Commonsense Misunderstanding`: 44.38, `Erroneous Assumption`: 44.87, `Logical Error`: 44.80, `Others`: 42.65, `Scientific Misconception`: 49.70, `Average`: 44.56.
* **Owen2.5-3B:** `Absurd Imagination`: 45.30, `Commonsense Misunderstanding`: 42.65, `Erroneous Assumption`: 42.82, `Logical Error`: 44.03, `Others`: 42.65, `Scientific Misconception`: 49.70, `Average`: 43.73.
* **claude-3-sonnet-20240229:** `Absurd Imagination`: 40.19, `Commonsense Misunderstanding`: 39.68, `Erroneous Assumption`: 39.89, `Logical Error`: 39.08, `Others`: 43.14, `Scientific Misconception`: 43.15, `Average`: 40.05.
* **Llama2-70B:** `Absurd Imagination`: 40.82, `Commonsense Misunderstanding`: 40.21, `Erroneous Assumption`: 39.31, `Logical Error`: 39.26, `Others`: 34.50, `Scientific Misconception`: 42.86, `Average`: 39.88.
* **Llama3-8B:** `Absurd Imagination`: 36.00, `Commonsense Misunderstanding`: 36.47, `Erroneous Assumption`: 36.82, `Logical Error`: 36.79, `Others`: 34.92, `Scientific Misconception`: 43.50, `Average`: 36.48.
* **gpt-3.5-turbo:** `Absurd Imagination`: 35.61, `Commonsense Misunderstanding`: 35.14, `Erroneous Assumption`: 34.28, `Logical Error`: 34.16, `Others`: 31.89, `Scientific Misconception`: 40.74, `Average`: 35.17.
* **Llama-2-13B:** `Absurd Imagination`: 22.54, `Commonsense Misunderstanding`: 24.39, `Erroneous Assumption`: 22.83, `Logical Error`: 26.54, `Others`: 18.42, `Scientific Misconception`: 32.43, `Average`: 22.53.
* **Yi-34B-v2.2:** `Absurd Imagination`: 41.83, `Commonsense Misunderstanding`: 42.64, `Erroneous Assumption`: 42.01, `Logical Error`: 41.19, `Others`: 38.06, `Scientific Misconception`: 45.65, `Average`: 42.82.
* **Qwen1.5-110B:** `Absurd Imagination`: 43.08, `Commonsense Misunderstanding`: 43.76, `Erroneous Assumption`: 43.20, `Logical Error`: 42.31, `Others`: 39.92, `Scientific Misconception`: 47.37, `Average`: 43.93.
* **Llama-2-70B:** `Absurd Imagination`: 43.64, `Commonsense Misunderstanding`: 45.22, `Erroneous Assumption`: 44.37, `Logical Error`: 43.56, `Others`: 40.87, `Scientific Misconception`: 49.02, `Average`: 44.32.
* **Falcon-180B:** `Absurd Imagination`: 42.06, `Commonsense Misunderstanding`: 42.86, `Erroneous Assumption`: 42.47, `Logical Error`: 41.60, `Others`: 38.22, `Scientific Misconception`: 46.35, `Average`: 42.92.
* **Yi-65B-v2.2:** `Absurd Imagination`: 39.98, `Commonsense Misunderstanding`: 40.55, `Erroneous Assumption`: 40.12, `Logical Error`: 39.98, `Others`: 36.58, `Scientific Misconception`: 42.60, `Average`: 40.35.
### Key Observations
* **Claude-3-haiku-20240307** and **Mixtral-8x22B-v0.1** consistently score highest across most categories, suggesting they are less prone to these types of errors.
* **Llama-2-13B** scores significantly lower than other models, indicating a higher susceptibility to making these errors.
* The "Others" category consistently has lower scores compared to the other error types, suggesting it's easier for the models to avoid these less-defined errors.
* "Scientific Misconception" scores are generally higher than "Logical Error" for most models.
* There is a general trend of decreasing scores as the models are listed further down the table.
### Interpretation
This data provides a comparative analysis of language model performance regarding specific error types. The higher scores indicate a greater tendency to generate responses falling into those error categories. The results suggest that models like Claude-3-haiku and Mixtral-8x22B-v0.1 are more robust in avoiding these errors, while models like Llama-2-13B are more prone to them.
The differences in performance across error categories are also insightful. The relatively lower scores in the "Others" category might indicate that these errors are more easily detectable or avoided during model training. The higher scores in "Scientific Misconception" could suggest that models struggle with nuanced scientific reasoning or have outdated knowledge.
The color gradient effectively highlights the relative performance of each model, making it easy to identify the best and worst performers in each category. This information is valuable for developers and researchers looking to understand the strengths and weaknesses of different language models and to improve their performance in specific areas. The data suggests a trade-off between model size and error rate, with larger models generally performing better, but this is not a universal rule. Further investigation would be needed to understand the underlying reasons for these performance differences.
</details>
Figure 4: Overall model performance across different error categories.
#### Question Type Annotation
Finally, we categorized questions into 6 types (shown in Table 1). We first used GPT-4o-2024-08-06 with bilingual prompts (see Figure 11) to generate initial classifications based on both the questions and their irrationality analyses. Human annotators then reviewed and adjusted these classifications. For cases where annotators disagreed or were uncertain, a meta annotator (one of the authors) made the final decision to ensure consistency and quality across both the English and Chinese versions, resulting in the final RuozhiBench-Gen.
### 2.4 RuozhiBench-Gen Statistics
Figure 3 illustrates the structure of a data entry in RuozhiBench. Each entry consists of a question in both Chinese and English, its irrationality analysis, question categories, and where applicable, the paired normal question. Table 2 shows the basic statistics of the dataset.
## 3 Experiments on RuozhiBench-Gen
### 3.1 Setup
#### Models
We evaluated 17 advanced models from 5 series. Including: GPT-4o-2024-05-13, GPT-4o-mini-2024-07-18 from OpenAI (OpenAI, 2023); Claude-3-haiku-20240307, and Claude-3-sonnet-20240229 from Anthropic Claude (2023); Mistral-Instruct-v0.1 (7B, 8x7B, and 8x22B) from Mixtral Jiang et al. (2024); Qwen2.5-Instruct (0.5B, 3B, 7B, 32B, 72B) from Qwen team Bai et al. (2023b); and Llama-3.1-Instruct (8B, 70B), Llama-3.2-Instruct (1B, 3B) from Meta Meta AI (2024).
#### Automated Evaluation
We employ an LLM-as-Judge framework using three independent models: GPT-4o-2024-08-06, Claude-3.5-Sonnet-20241022, and Llama-3.3-70B-Instruct. By design, we ensure the judge models are distinct from those being evaluated and represent more advanced versions of their respective architectures. Each judge independently evaluates responses on a scale of 0 to 4. Additionally, we incorporate irrationality analysis into the judging process to enhance evaluation quality and consistency. The detailed scoring criteria and evaluation prompts are available in Figure 13.
### 3.2 Main results
The results highlight significant performance differences across models and error categories. Claude-3-haiku leads with an average score of 62.00, particularly excelling in “Scientific Misconception” (66.96). Mixtral-8x22B-v0.1 (58.99) and Llama-3.1-70B (57.78) follow closely, showing balanced performance across categories.
A clear trend is observed across all model series: larger models consistently outperform their smaller counterparts, as seen in the Qwen, Llama, Mixtral, and GPT families. This suggests that model size plays a crucial role in performance, though architectural design and training strategies, such as those in Mixtral models, also contribute significantly.
Across categories, “Scientific Misconception” has the highest average score (49.20), suggesting models handle domain-specific knowledge better than abstract concepts like “Absurd Imagination” and “Others”. Smaller models, such as Qwen2.5-0.5B, consistently struggle, reinforcing the importance of both scale and training strategies in reducing errors.
Notably, the best-performing model only achieved a score of 62.00, indicating that this task remains inherently challenging for current models.
### 3.3 Comparison on Paired Normal Questions
To compare model performance on normal and tricky questions, we input paired normal questions and apply the same LLM-based judging with a 0-4 scoring system (see Figure 13 for prompt). Figure 5 shows the rating distributions from three evaluators for three models. The results reveal a clear shift toward higher scores, indicating better performance on normal questions while logical traps remain consistently challenging.
<details>
<summary>x5.png Details</summary>
### Visual Description
\n
## Histograms: Rating Distributions for Language Models
### Overview
The image presents three histograms, each displaying the distribution of ratings for three different language models: Claude-3.5, GPT-4.0, and Llama-3.3. The x-axis represents the rating scale, and the y-axis represents the frequency (count) of each rating. Each histogram shows two distributions: "Original" (in blue) and "Normal" (in orange).
### Components/Axes
* **X-axis Title:** "Rating" - Scale ranges from 0 to 4.
* **Y-axis Title:** "Frequency" - Scale ranges from 0 to 160.
* **Legend:** Located in the top-left corner.
* "Original" - Represented by a blue color.
* "Normal" - Represented by an orange color.
* **Histograms:** Three separate histograms, one for each language model, positioned horizontally.
* **Claude-3.5:** Leftmost histogram.
* **GPT-4.0:** Center histogram.
* **Llama-3.3:** Rightmost histogram.
### Detailed Analysis
**Claude-3.5:**
* The "Original" distribution slopes upward from a rating of 0 to approximately 2, then declines. The peak frequency is around a rating of 1, at approximately 25.
* The "Normal" distribution is skewed to the right, with a peak frequency around a rating of 3, at approximately 80.
* The "Original" distribution has a maximum frequency of approximately 25 at a rating of 1.
* The "Normal" distribution has a maximum frequency of approximately 80 at a rating of 3.
**GPT-4.0:**
* The "Original" distribution is relatively flat between ratings of 1 and 3, with a slight peak around a rating of 2, at approximately 20.
* The "Normal" distribution is heavily skewed to the right, with a very high peak around a rating of 3, at approximately 160.
* The "Original" distribution has a maximum frequency of approximately 20 at a rating of 2.
* The "Normal" distribution has a maximum frequency of approximately 160 at a rating of 3.
**Llama-3.3:**
* The "Original" distribution is relatively flat between ratings of 1 and 2, with a slight peak around a rating of 1, at approximately 20.
* The "Normal" distribution is strongly skewed to the right, with a peak around a rating of 3, at approximately 100.
* The "Original" distribution has a maximum frequency of approximately 20 at a rating of 1.
* The "Normal" distribution has a maximum frequency of approximately 100 at a rating of 3.
### Key Observations
* All three models show a significant shift in the distribution of ratings when comparing "Original" to "Normal". The "Normal" distribution is consistently skewed to the right, indicating a tendency for higher ratings.
* GPT-4.0 exhibits the most pronounced shift, with a very strong peak at a rating of 3 in the "Normal" distribution.
* Claude-3.5 has the lowest overall frequencies in both distributions.
* The "Original" distributions are generally more evenly distributed across the rating scale compared to the "Normal" distributions.
### Interpretation
The data suggests that the "Normal" process (whatever that may be) consistently improves the ratings assigned to the outputs of these language models. The substantial rightward skew in the "Normal" distributions indicates that the process leads to a higher proportion of outputs receiving higher ratings. The dramatic difference observed in GPT-4.0 suggests that this model benefits the most from the "Normal" process. The "Original" distributions provide a baseline understanding of the raw rating distributions before the "Normal" process is applied. The differences between the models' "Original" distributions may reflect inherent differences in their performance or the types of tasks they are evaluated on. The "Normal" process appears to be a form of post-processing or refinement that significantly enhances the perceived quality of the generated text, particularly for GPT-4.0.
</details>
Figure 5: Rating distribution comparison between normal and tricky questions for three models..
<details>
<summary>x6.png Details</summary>
### Visual Description
\n
## Scatter Plot Matrix: Correlation Analysis of LLM Scores
### Overview
This image presents a scatter plot matrix displaying the correlations between three Large Language Models (LLMs): Claude-3.5, GPT-4o, and Llama-3.3. Each off-diagonal plot shows the relationship between two models, with a regression line indicating the trend. The diagonal plots display histograms of each model's score distribution. Pearson correlation coefficients (r) are provided for each pairwise comparison.
### Components/Axes
* **Axes Labels:** The x and y axes of each scatter plot are labeled with the names of the LLMs being compared (Claude-3.5, GPT-4o, Llama-3.3).
* **Histograms:** The diagonal plots show histograms with the LLM name on the y-axis and score on the x-axis, ranging from approximately 0 to 120 for Claude-3.5, 0 to 4 for GPT-4o, and 0 to 4 for Llama-3.3.
* **Scatter Plots:** Each off-diagonal plot displays the scores of two LLMs as points.
* **Regression Lines:** A red line of best fit is drawn through each scatter plot, indicating the linear relationship between the two models.
* **Correlation Coefficients:** The Pearson correlation coefficient (r) is displayed above each off-diagonal scatter plot.
* **Gridlines:** A light grid is present in the background of each plot.
### Detailed Analysis or Content Details
**1. Claude-3.5 vs. GPT-4o (Top-Left):**
* Trend: The scatter plot shows a positive, but somewhat scattered, linear trend.
* Correlation Coefficient (r): 0.73
* X-axis (Claude-3.5): Range approximately 0 to 4.
* Y-axis (GPT-4o): Range approximately 0 to 3.
* Histogram (Claude-3.5): The distribution is right-skewed, with a peak around 1.5-2.0 and a tail extending to approximately 120.
**2. Claude-3.5 vs. Llama-3.3 (Top-Right):**
* Trend: The scatter plot shows a weak positive linear trend.
* Correlation Coefficient (r): 0.43
* X-axis (Claude-3.5): Range approximately 0 to 4.
* Y-axis (Llama-3.3): Range approximately 0 to 3.
* Histogram (Llama-3.3): The distribution is roughly uniform, with values ranging from 0 to 4.
**3. GPT-4o vs. Claude-3.5 (Center-Left):**
* Trend: The scatter plot shows a positive, but somewhat scattered, linear trend.
* Correlation Coefficient (r): 0.73
* X-axis (GPT-4o): Range approximately 0 to 4.
* Y-axis (Claude-3.5): Range approximately 0 to 3.
* Histogram (GPT-4o): The distribution is roughly uniform, with values ranging from 0 to 4.
**4. GPT-4o vs. Llama-3.3 (Center):**
* Trend: The scatter plot shows a strong positive linear trend.
* Correlation Coefficient (r): 0.75
* X-axis (GPT-4o): Range approximately 0 to 4.
* Y-axis (Llama-3.3): Range approximately 0 to 3.
* Histogram (GPT-4o): The distribution is roughly uniform, with values ranging from 0 to 4.
**5. Llama-3.3 vs. Claude-3.5 (Bottom-Left):**
* Trend: The scatter plot shows a weak positive linear trend.
* Correlation Coefficient (r): 0.43
* X-axis (Llama-3.3): Range approximately 0 to 4.
* Y-axis (Claude-3.5): Range approximately 0 to 3.
* Histogram (Claude-3.5): The distribution is right-skewed, with a peak around 1.5-2.0 and a tail extending to approximately 120.
**6. Llama-3.3 vs. GPT-4o (Bottom-Right):**
* Trend: The scatter plot shows a strong positive linear trend.
* Correlation Coefficient (r): 0.75
* X-axis (Llama-3.3): Range approximately 0 to 4.
* Y-axis (GPT-4o): Range approximately 0 to 3.
* Histogram (Llama-3.3): The distribution is roughly uniform, with values ranging from 0 to 4.
### Key Observations
* The strongest correlations are observed between GPT-4o and Llama-3.3 (r = 0.75) and between Claude-3.5 and GPT-4o (r = 0.73).
* The weakest correlation is between Claude-3.5 and Llama-3.3 (r = 0.43).
* Claude-3.5 has a significantly wider score range and a right-skewed distribution compared to GPT-4o and Llama-3.3, which have more uniform distributions.
### Interpretation
The scatter plot matrix reveals the degree of agreement between the three LLMs. The high correlation between GPT-4o and Llama-3.3 suggests that these models tend to assign similar scores to the same inputs. The moderate correlation between Claude-3.5 and GPT-4o indicates a reasonable level of agreement, but with more variability. The low correlation between Claude-3.5 and Llama-3.3 suggests that these two models may evaluate inputs differently.
The differing distributions of scores for Claude-3.5, with its right skew, suggests that it is more likely to assign very high scores than GPT-4o or Llama-3.3. This could indicate that Claude-3.5 is more sensitive to certain types of inputs or has a different scoring scale. The more uniform distributions of GPT-4o and Llama-3.3 suggest a more consistent scoring approach.
The data suggests that while all three models are related, they are not perfectly aligned in their evaluations. This highlights the importance of considering multiple LLMs when assessing the performance of a given task or input. The differences in correlation and distribution could be due to variations in model architecture, training data, or scoring criteria.
</details>
Figure 6: Pairwise scatter plots with Pearson correlation coefficients, and rating distributions of difference evaluators. The diagonal histograms show Claude-3.5’s tendency toward lower ratings compared to GPT-4 in middle and Llama-3.3 with a higher ratings.
### 3.4 High Variance between Evaluators
| Model Pair | Pearson Correlation | Mean Difference | Large Disagreement (%) | | | |
| --- | --- | --- | --- | --- | --- | --- |
| Individual | Mean | Individual | Mean | Individual | Mean | |
| Claude vs GPT | 0.568 | 0.726 | -0.560 | -0.806 | 0.281 | 3.99 |
| Claude vs Llama | 0.359 | 0.433 | -2.107 | -2.002 | 1.007 | 50.37 |
| GPT vs Llama | 0.687 | 0.748 | -1.196 | -1.196 | 19.80 | 10.19 |
Table 3: Comparison of rating agreement metrics between model pairs. Individual analysis treats each rating independently, while Mean analysis averages multiple ratings per item. Pearson correlation measures linear relationship strength (-1 to 1); Mean difference indicates systematic rating bias between models; Large disagreement shows percentage of ratings differing by $\geq$ 2 points.
Table 3 presents key metrics comparing rating agreements between model pairs, and Figure 6 visualizes the mean-based pairwise relationships and rating distributions. Full results and evaluations using all three evaluators are presented in Appendix D.
The comparison reveals distinct rating patterns among the three models. GPT-4o and Llama-3.3 demonstrate the strongest agreement, with the highest correlation and relatively moderate large disagreements. In contrast, Claude-3.5 shows notably weaker correlation with others, indicating fundamentally different evaluation standard given the same criteria.
Mean-based analysis consistently shows stronger correlations and fewer large disagreements compared to individual analysis across all model pairs. This pattern is particularly evident in the Claude-3.5 vs GPT-4o comparison, where large disagreements decrease from 28.1% to 3.99% when using mean-based analysis. The scatter plots in Figure 6 visualize these relationships, with the GPT-4o vs Llama-3.3 comparison showing the tightest clustering around the regression line, while the Claude-3.5 vs Llama-3.3 comparison exhibits more dispersed points, reflecting their lower correlation and higher disagreement rate. These observations motivate us the creation of the multiple-choice evaluation format.
<details>
<summary>x7.png Details</summary>
### Visual Description
## Heatmap: LLM Evaluation by Error Category
### Overview
This heatmap visualizes the performance of various Large Language Models (LLMs) across six error categories, plus a "Micro Average" score. The color intensity represents the magnitude of the score, with darker shades indicating higher values. The LLMs are listed vertically on the y-axis, and the error categories are listed horizontally on the x-axis.
### Components/Axes
* **Y-axis (LLMs):** Llama-3-70B, gpt-4o-2024-05-13, Qwen2.5-72B, Qwen2.5-32B, Mixtral-8x22B-v0.1, Mixtral-8x7B-v0.1, Owen2.5-7B, Llama-3-18B, claude-3-haiku-20240307, gpt-4o-mini-2024-07-18, claude-3-sonnet-20240229, Qwen2.5-13B, Llama-2-70B, Meta-Llama-2-13B, Zephyr-7B-beta, Mistral-7B-Instruct-v0.1, OpenHermes-2.5-Mistral-7B
* **X-axis (Error Categories):** Logical Error, Commonsense Misunderstanding, Erroneous Assumption, Scientific Misconception, Absurd Imagination, Others, Micro Average
* **Color Scale:** The heatmap uses a gradient color scheme. The exact scale isn't provided, but darker colors represent higher values, and lighter colors represent lower values.
* **Legend:** No explicit legend is present, but the column headers define the categories.
### Detailed Analysis
Here's a breakdown of the data, reading across rows (LLM performance) and columns (error categories). Values are approximate, based on visual estimation.
* **Llama-3-70B:** Logical Error: 60.42, Commonsense Misunderstanding: 58.21, Erroneous Assumption: 57.31, Scientific Misconception: 46.43, Absurd Imagination: 56.74, Others: 57.35, Micro Average: 56.90
* **gpt-4o-2024-05-13:** Logical Error: 52.00, Commonsense Misunderstanding: 56.58, Erroneous Assumption: 52.77, Scientific Misconception: 46.43, Absurd Imagination: 52.13, Others: 47.06, Micro Average: 53.12
* **Qwen2.5-72B:** Logical Error: 49.29, Commonsense Misunderstanding: 50.85, Erroneous Assumption: 50.39, Scientific Misconception: 53.57, Absurd Imagination: 49.89, Others: 35.29, Micro Average: 49.82
* **Qwen2.5-32B:** Logical Error: 53.75, Commonsense Misunderstanding: 50.56, Erroneous Assumption: 50.83, Scientific Misconception: 39.29, Absurd Imagination: 47.26, Others: 41.18, Micro Average: 49.37
* **Mixtral-8x22B-v0.1:** Logical Error: 46.38, Commonsense Misunderstanding: 47.61, Erroneous Assumption: 46.12, Scientific Misconception: 44.44, Absurd Imagination: 46.45, Others: 47.06, Micro Average: 46.48
* **Mixtral-8x7B-v0.1:** Logical Error: 44.72, Commonsense Misunderstanding: 46.30, Erroneous Assumption: 45.10, Scientific Misconception: 45.63, Absurd Imagination: 46.01, Others: 47.06, Micro Average: 45.99
* **Owen2.5-7B:** Logical Error: 36.69, Commonsense Misunderstanding: 39.61, Erroneous Assumption: 40.21, Scientific Misconception: 58.57, Absurd Imagination: 43.19, Others: 32.73, Micro Average: 41.57
* **Llama-3-18B:** Logical Error: 43.19, Commonsense Misunderstanding: 41.26, Erroneous Assumption: 40.83, Scientific Misconception: 51.67, Absurd Imagination: 41.81, Others: 26.27, Micro Average: 41.24
* **claude-3-haiku-20240307:** Logical Error: 42.14, Commonsense Misunderstanding: 40.80, Erroneous Assumption: 42.51, Scientific Misconception: 39.29, Absurd Imagination: 43.21, Others: 29.41, Micro Average: 40.77
* **gpt-4o-mini-2024-07-18:** Logical Error: 41.43, Commonsense Misunderstanding: 38.80, Erroneous Assumption: 38.99, Scientific Misconception: 50.00, Absurd Imagination: 37.78, Others: 47.06, Micro Average: 39.29
* **claude-3-sonnet-20240229:** Logical Error: 37.14, Commonsense Misunderstanding: 35.80, Erroneous Assumption: 34.80, Scientific Misconception: 39.29, Absurd Imagination: 31.90, Others: 29.41, Micro Average: 34.56
* **Qwen2.5-13B:** Logical Error: 39.06, Commonsense Misunderstanding: 32.10, Erroneous Assumption: 32.54, Scientific Misconception: 48.15, Absurd Imagination: 33.25, Others: 26.79, Micro Average: 33.26
* **Llama-2-70B:** Logical Error: 34.45, Commonsense Misunderstanding: 30.94, Erroneous Assumption: 31.83, Scientific Misconception: 31.52, Absurd Imagination: 29.02, Others: 22.19, Micro Average: 31.00
* **Meta-Llama-2-13B:** Logical Error: 4.00, Commonsense Misunderstanding: 0.94, Erroneous Assumption: 0.83, Scientific Misconception: 2.37, Absurd Imagination: 0.96, Others: 0.00, Micro Average: 1.62
* **Zephyr-7B-beta:** Logical Error: 0.97, Commonsense Misunderstanding: 0.85, Erroneous Assumption: 0.95, Scientific Misconception: 2.04, Absurd Imagination: 0.99, Others: 0.00, Micro Average: 0.95
* **Mistral-7B-Instruct-v0.1:** Logical Error: 0.89, Commonsense Misunderstanding: 0.71, Erroneous Assumption: 0.79, Scientific Misconception: 1.85, Absurd Imagination: 0.81, Others: 0.00, Micro Average: 0.81
* **OpenHermes-2.5-Mistral-7B:** Logical Error: 0.83, Commonsense Misunderstanding: 0.77, Erroneous Assumption: 0.76, Scientific Misconception: 1.82, Absurd Imagination: 0.85, Others: 0.00, Micro Average: 0.81
### Key Observations
* **Llama-3-70B** consistently scores highest across most error categories.
* **Meta-Llama-2-13B, Zephyr-7B-beta, Mistral-7B-Instruct-v0.1, and OpenHermes-2.5-Mistral-7B** have very low scores across all categories, indicating strong performance.
* **Owen2.5-7B** shows a relatively high score in "Scientific Misconception" compared to other categories for that model.
* The "Others" category generally has lower scores than the more specific error types.
* There's a clear separation in performance between the larger models (Llama-3-70B, gpt-4o-2024-05-13, Qwen2.5-72B) and the smaller models (Meta-Llama-2-13B, Zephyr-7B-beta, etc.).
### Interpretation
This heatmap provides a comparative evaluation of LLMs based on their propensity to make different types of errors. The data suggests that larger models like Llama-3-70B exhibit better overall performance, making fewer errors across a wider range of categories. The lower scores of models like Meta-Llama-2-13B indicate they are more robust and less prone to these specific error types.
The variation in scores across error categories for each model highlights specific strengths and weaknesses. For example, a model might perform well on "Logical Error" but struggle with "Scientific Misconception," suggesting a need for further training or refinement in that area.
The "Micro Average" provides a single metric for overall performance, but it's important to consider the individual category scores to gain a more nuanced understanding of each model's capabilities. The heatmap allows for a quick visual identification of models that excel in certain areas and those that require improvement. The data suggests a trade-off between model size and performance, with larger models generally achieving higher accuracy but potentially requiring more computational resources.
</details>
Figure 7: RuozhiBench-MC evaluation results in percentage by question categories. Scores ( $x$ ) are normalized according to the baseline score ( $50\$ ) by $2\times(x-0.5)$ .
## 4 RuozhiBench-MC : A Multiple-Choice Evaluation Framework
While generative evaluation provides a natural way to assess language model responses to tricky questions, our experiments on RuozhiBench-Gen revealed several limitations in the evaluation process. First, evaluator models themselves may sometimes fail to recognize subtle logical traps, even when provided with analysis of the trick, leading to inaccurate assessments. Second, the significant variations in scoring standards across different evaluator models as seen in Section 3.4. Finally, the two-step process of generating responses and then evaluating them with high-performance models introduces both substantial computational overhead and significant costs, particularly when using commercial models as evaluators.
### 4.1 Multiple-Choice Format
To address evaluation challenges, we created RuozhiBench-MC, a multiple-choice version of our benchmark. For each question, we present two responses, one “good” and one “bad”, and ask an LLM to choose between them. This binary format transforms evaluation from open-ended generation to a simple decision: can the model identify better logical reasoning? There are several key advantages: (1) Standardized Evaluation through consistent binary choices, (2) Computational Efficiency by eliminating separate generation and evaluation, and (3) Clear Success Criteria via unambiguous metrics.
### 4.2 Option Construction
To construct high-quality response options for RuozhiBench-MC, we leveraged the extensive response data collected during our evaluation of the 17 models in RuozhiBench-Gen. For each question, we implemented the following selection process.
We used the automatic evaluations from three different models to calculate an average score for each response in our existing dataset. We randomly sample two responses for each question, ensuring that the selected responses have a score difference greater than 2. If no response pairs meet this criterion, we select the responses with the highest and lowest scores. In all cases, the response with the higher score is designated as the “good” answer, while the other is designated as the “bad” answer. The detailed distribution of selected responses across models is shown in Figure 15.
## 5 Experiments on RuozhiBench-MC
We evaluate the same models as in Section 3. In our evaluation, we test models by presenting each question with its two corresponding options in alternating orders. This approach helps eliminate potential position bias in model responses while maintaining the fundamental binary choice structure. Models are prompted to select their preferred answer, and their performance is assessed based on their ability to consistently identify the better response.
| Model | Good First | Bad First | Avg | Positional Bias | Format |
| --- | --- | --- | --- | --- | --- |
| Llama-3.2-1B-Instruct | $58.19$ | $39.35$ | $48.77$ | $18.84$ | $59.68$ |
| Llama-3.2-3B-Instruct | $65.43$ | $66.67$ | $66.05$ | $-1.24$ | $53.99$ |
| Llama-3.1-8B-Instruct | $76.97$ | $64.26$ | $70.62$ | $12.71$ | $89.96$ |
| Llama-3.1-70B-Instruct | $81.86$ | $75.04$ | $78.45$ | $6.82$ | $98.67$ |
| Mistral-7B-Instruct-v0.1 | $55.85$ | $46.96$ | $51.41$ | $8.89$ | $99.70$ |
| Mixtral-8x7B-Instruct-v0.1 | $69.22$ | $76.77$ | $72.99$ | $-7.55$ | $96.23$ |
| Mixtral-8x22B-Instruct-v0.1 | $74.77$ | $71.71$ | $73.24$ | $3.07$ | $97.93$ |
| Qwen2.5-0.5B-Instruct | $100.00$ | $0.00$ | $50.00$ | $100.00$ | $89.66$ |
| Qwen2.5-3B-Instruct | $74.28$ | $58.98$ | $66.63$ | $15.30$ | $87.22$ |
| Qwen2.5-7B-Instruct | $68.59$ | $72.97$ | $70.78$ | $-4.38$ | $53.99$ |
| Qwen2.5-32B-Instruct | $77.00$ | $72.36$ | $74.68$ | $4.64$ | $99.48$ |
| Qwen2.5-72B-Instruct | $75.11$ | $74.70$ | $74.91$ | $0.41$ | $99.78$ |
| claude-3-haiku-20240307 | $73.41$ | $67.36$ | $70.38$ | $6.06$ | $100.00$ |
| claude-3-sonnet-20240229 | $67.21$ | $67.36$ | $67.28$ | $-0.15$ | $100.00$ |
| gpt-4o-mini-2024-07-18 | $72.23$ | $67.06$ | $69.65$ | $5.17$ | $100.00$ |
| gpt-4o-2024-05-13 | $81.22$ | $71.89$ | $76.56$ | $9.33$ | $99.48$ |
Table 4: RuozhiBench-MC evaluation results. Good First and Bad First are the accuracy in the percentage of selecting the correct answer where the correct answers are the first one and second one, respectively. Avg is the average of Good First and Bad First, with the random baseline of $50\$ . Positional Bias indicates the models’ position bias to the first answer, and the closer it is to 0, the better. Format is the percentage of answers generated by the model in the correct format specified in the prompt.
### 5.1 Main Results
Figure 7 shows the overall model performance on RuozhiBench-MC. In the multiple-choice evaluation setting, the general finding that larger models perform better still holds. We can observe models with large models in the Llama, Qwen, Mixtral family and GPT-4o achieved at least 40 scores in micro average across all categories of questions, which shows that they are significantly better than the random baseline. On the other hand, the ranking of the top-performing models has changed significantly. The best-performing model (Claude-3-haiku) in open generation evaluation ranks only in the middle tier of all models, while Llama-3.1-70B and GPT-4o now take the lead with micro average scores of 56.90 and 53.12, respectively.
There are three small models Mistral-7B, Qwen2.5-0.5B, and Llama-3.2-1B struggle on the multiple-choice evaluation with almost zero performance difference compared to random baseline across all question categories. This observation suggests that these models cannot understand the concept and definition of trick questions and hence unable to accurately assess the answers to these questions, reaffirming that RuozhiBench-MC had the advantages of standardized evaluation and clear success criteria.
<details>
<summary>x8.png Details</summary>
### Visual Description
\n
## Scatter Plot: Correlation between Generation and Multiple Choice Scores
### Overview
This image presents a scatter plot illustrating the correlation between "Generation Score" and "Multiple Choice Score" for various language models. A trend line is fitted to the data points, and the correlation coefficient is provided. The plot aims to demonstrate the relationship between a model's performance on a generation task and its ability to answer multiple-choice questions.
### Components/Axes
* **Title:** "Correlation between Generation and Multiple Choice Scores" (Top-center)
* **Correlation Coefficient:** 0.909 (Top-center, below the title)
* **X-axis:** "Generation Score" (Bottom-center), ranging from approximately 10 to 60.
* **Y-axis:** "Multiple Choice Score" (Left-center), ranging from approximately 40 to 80.
* **Data Points:** Represent individual language models, labeled with their names.
* **Trend Line:** A dashed red line representing the overall trend in the data.
* **Confidence Interval:** A shaded region around the trend line, indicating the uncertainty in the trend.
* **Legend:** Located in the top-right corner, listing the language models.
### Detailed Analysis
The trend line slopes upwards, indicating a positive correlation between Generation Score and Multiple Choice Score. The data points generally cluster around the trend line, suggesting a strong relationship.
Here's a breakdown of the data points, approximate values, and their corresponding labels:
* **Owen2.5-0.5B:** (Generation Score ≈ 17, Multiple Choice Score ≈ 52) - Bottom-left
* **Llama-3.2-1B:** (Generation Score ≈ 24, Multiple Choice Score ≈ 50) - Bottom-center
* **Mistral-7B-v0.1:** (Generation Score ≈ 30, Multiple Choice Score ≈ 54) - Bottom-center
* **Llama-3.1-8B:** (Generation Score ≈ 33, Multiple Choice Score ≈ 64) - Center-left
* **Llama-3.2-3B:** (Generation Score ≈ 36, Multiple Choice Score ≈ 65) - Center
* **gpt-4o-mini-2024-07-18:** (Generation Score ≈ 42, Multiple Choice Score ≈ 70) - Center-right
* **claude-3-sonnet:** (Generation Score ≈ 43, Multiple Choice Score ≈ 68) - Center-right
* **Owen2.5-3B:** (Generation Score ≈ 44, Multiple Choice Score ≈ 69) - Center-right
* **Owen2.5-7B:** (Generation Score ≈ 46, Multiple Choice Score ≈ 70) - Right-center
* **gpt-4o-2024-05-13:** (Generation Score ≈ 48, Multiple Choice Score ≈ 76) - Top-right
* **Mixtral-8x22B-v0.1:** (Generation Score ≈ 49, Multiple Choice Score ≈ 75) - Top-right
* **Mixtral-8x7B-v0.1:** (Generation Score ≈ 51, Multiple Choice Score ≈ 75) - Top-right
* **claude-3-haiku:** (Generation Score ≈ 52, Multiple Choice Score ≈ 74) - Top-right
* **Owen2.5-32B:** (Generation Score ≈ 53, Multiple Choice Score ≈ 75) - Top-right
* **Owen2.72B:** (Generation Score ≈ 55, Multiple Choice Score ≈ 76) - Top-right
* **Llama-3.1-70B:** (Generation Score ≈ 58, Multiple Choice Score ≈ 77) - Top-right
### Key Observations
* The correlation coefficient of 0.909 indicates a very strong positive correlation.
* The data points are relatively tightly clustered around the trend line, suggesting a consistent relationship.
* The models with lower Generation Scores also tend to have lower Multiple Choice Scores, and vice versa.
* There are no significant outliers that deviate substantially from the overall trend.
### Interpretation
The data strongly suggests that there is a positive correlation between a language model's performance on generation tasks (as measured by "Generation Score") and its ability to perform well on multiple-choice question answering (as measured by "Multiple Choice Score"). This implies that models that are better at generating coherent and relevant text are also more likely to be able to understand and reason about information to answer questions correctly.
The high correlation coefficient (0.909) indicates that the Generation Score is a good predictor of the Multiple Choice Score, and vice versa. This could be useful for evaluating and comparing different language models. The confidence interval around the trend line provides a measure of the uncertainty in this relationship.
The consistent clustering of data points around the trend line suggests that this relationship is robust and not driven by a few outliers. The absence of significant outliers further supports this conclusion. The models listed represent a diverse set of architectures and sizes, suggesting the correlation holds across different model types.
</details>
Figure 8: Pairwise scatter plots with Pearson correlation coefficients of generation and multiple choice scores.
<details>
<summary>x9.png Details</summary>
### Visual Description
\n
## Charts: Model Size vs. RuoZhiBench Scores
### Overview
The image presents two scatter plots comparing model size (in billions of parameters) to performance scores on the RuoZhiBench benchmark. The left chart shows Generation scores, while the right chart shows Multiple Choice scores. Each chart includes data points for several models: Owen2.5-0.5B, Owen2.5-3B, Owen2.5-7B, Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8B, and Mistral-7B-v0.1. Trend lines are fitted to the data points in each chart.
### Components/Axes
**Left Chart (Generation Scores):**
* **Title:** Model Size vs RuoZhiBench-Gen Scores
* **X-axis:** Model Size (B parameters) - Logarithmic scale from 10⁰ to 10¹
* **Y-axis:** Generation Score - Linear scale from 20 to 60
* **Data Series/Labels:**
* Owen2.5-0.5B (Purple)
* Owen2.5-3B (Blue, dashed)
* Owen2.5-7B (Blue, solid)
* Llama-3.2-1B (Orange)
* Llama-3.2-3B (Orange)
* Llama-3.1-8B (Blue)
* Mistral-7B-v0.1 (Red)
**Right Chart (Multiple Choice Scores):**
* **Title:** Model Size vs RuoZhiBench-MC Scores
* **X-axis:** Model Size (B parameters) - Logarithmic scale from 10⁰ to 10¹
* **Y-axis:** Multiple Choice Score - Linear scale from 50 to 80
* **Data Series/Labels:**
* Owen2.5-0.5B (Orange)
* Owen2.5-3B (Orange)
* Owen2.5-7B (Orange)
* Llama-3.2-1B (Red)
* Llama-3.2-3B (Red)
* Llama-3.1-8B (Orange)
* Mistral-7B-v0.1 (Red)
### Detailed Analysis or Content Details
**Left Chart (Generation Scores):**
* **Owen2.5-0.5B:** At approximately 10⁰ B parameters, the score is around 22.
* **Owen2.5-3B:** The trend line starts at approximately 32 at 10⁰ B parameters and increases to around 42 at 10¹ B parameters.
* **Owen2.5-7B:** The trend line starts at approximately 42 at 10⁰ B parameters and increases to around 55 at 10¹ B parameters.
* **Llama-3.2-1B:** At approximately 10⁰ B parameters, the score is around 21.
* **Llama-3.2-3B:** At approximately 10⁰ B parameters, the score is around 28.
* **Llama-3.1-8B:** At approximately 10¹ B parameters, the score is around 57.
* **Mistral-7B-v0.1:** At approximately 10¹ B parameters, the score is around 31.
**Right Chart (Multiple Choice Scores):**
* **Owen2.5-0.5B:** At approximately 10⁰ B parameters, the score is around 50.
* **Owen2.5-3B:** The trend line starts at approximately 52 at 10⁰ B parameters and increases to around 65 at 10¹ B parameters.
* **Owen2.5-7B:** The trend line starts at approximately 53 at 10⁰ B parameters and increases to around 73 at 10¹ B parameters.
* **Llama-3.2-1B:** At approximately 10⁰ B parameters, the score is around 51.
* **Llama-3.2-3B:** At approximately 10⁰ B parameters, the score is around 54.
* **Llama-3.1-8B:** At approximately 10¹ B parameters, the score is around 76.
* **Mistral-7B-v0.1:** At approximately 10¹ B parameters, the score is around 60.
### Key Observations
* **Generation Scores:** There's a clear positive correlation between model size and generation score, particularly for the Owen2.5 models. The Llama models show less consistent improvement with size.
* **Multiple Choice Scores:** A strong positive correlation exists between model size and multiple-choice score, especially for the Owen2.5 models. Llama-3.1-8B significantly outperforms other models.
* **Outliers:** Llama-3.1-8B has the highest score in the Multiple Choice chart, standing out from the other models. Mistral-7B-v0.1 has a relatively low Generation score compared to its Multiple Choice score.
* **Trend Lines:** The trend lines for Owen2.5 models are relatively smooth and indicate consistent improvement with increasing model size.
### Interpretation
The data suggests that increasing model size generally improves performance on the RuoZhiBench benchmark, particularly for the Owen2.5 family of models. The stronger correlation observed in the Multiple Choice scores compared to Generation scores might indicate that larger models are better at tasks requiring reasoning and knowledge retrieval. The outlier performance of Llama-3.1-8B in the Multiple Choice task suggests that this model may have architectural or training advantages. The discrepancy in scores for Mistral-7B-v0.1 could be due to its specific training data or architecture, making it more suited for multiple-choice questions than text generation. The logarithmic scale on the x-axis highlights the diminishing returns of increasing model size; the improvement in scores appears to slow down as the model size grows. The use of dashed vs solid lines for the Owen2.5-3B model in the Generation chart is unclear and may be a visual artifact.
</details>
Figure 9: Relationship between model size and performance on generation and multiple-choice tasks. The plots show the correlation between model size (in billions of parameters) and performance scores for both generation (top) and multiple-choice (bottom) tasks. Both plots use a logarithmic scale for model size. The dashed lines represent the regression fit, demonstrating a positive correlation between model size and performance for both task types.
### 5.2 Analysis
#### Correlation with RuozhiBench-Gen
Figure 8 shows the correlation between generation and multiple choice scores for all models. We can observe a strong positive correlation between the generation and multiple choice scores for all models, with a Pearson correlation coefficient of 0.909. In general, most models have achieved slightly higher scores in the multiple choice than generation evaluation.
#### Model Size Analysis
Figure 9 shows the relationship between model size and performance on generation and multiple-choice tasks.
#### Issues in MC
Despite the advantages discussed above, we found two caveats of RuozhiBench-MC based on the detailed results in Table 4. (1) We found different degrees of performance gaps between when we provide the better response as the first option and the reverse, even for some of the best-performing models like GPT-4o and Claude-3-haiku. Most models perform slightly better when the better answer is provided as the first option. This positional bias suggests these models may be influenced by the ordering of options and indicates some uncertainty in their decision-making process. (2) Not all models can strictly follow the formatting instructions we provided in the prompts of RuozhiBench-MC. Except for Claude-3 models and GPT-4o, all other models produce different numbers of responses with formatting errors. Smaller models in Llama-3.2 family and Qwen2.5-7B suffer more from this issue as their formatting success rates are less than 60%.
## 6 Related Work
#### General Reasoning Evaluation of LLMs
Evaluating the reasoning capabilities of LLMs has gained significant attention, with diverse benchmarks developed for different reasoning domains, such as commonsense reasoning Talmor et al. (2019); Zellers et al. (2019); Clark et al. (2018); Bisk et al. (2020), math Cobbe et al. (2021); Hendrycks et al. (2021b), code Chen et al. (2021); Austin et al. (2021), and logic Liu et al. (2020, 2023a, 2023b). Recent advances, with models like GPT-4 surpassing human performance on many of these benchmarks, have driven further exploration into more challenging testbeds. Models such as GPT-o1 OpenAI et al. (2024) and Deepseek-R1 DeepSeek-AI et al. (2025) have demonstrated improved performance on advanced benchmarks like AIME MMA. (2024) and HLE Phan et al. (2025), which assess reasoning across domains such as mathematics, physics, and scientific knowledge. In contrast, RuozhiBench presents seemingly simple questions—ones even a five-year-old could find fallacy—that expose fundamental gaps in LLMs’ commonsense reasoning abilities, highlighting the limitations of current models beyond factual knowledge and formulaic problem-solving.
#### Understanding Deceptive and Fallacious Texts
While there is a substantial body of work on LLMs’ reasoning capabilities, research specifically focused on evaluating how models handle deliberately deceptive or fallacious inputs remains limited. Recent work has begun exploring the use of Chinese Ruozhiba forum data for improving LLMs’ capabilities; for instance, Lin et al. (2024) and Bai et al. (2024) incorporated Ruozhiba data in their training data to enhance logic reasoning in Chinese.
There are several works exploring LLMs’ understanding of logical fallacies Lei and Huang (2024); Payandeh et al. (2023); Li et al. (2024a). While most relevant work is Li et al. (2024b), which created a benchmark using data from Ruozhiba. However, our work differs in that: (1) we provide the first English benchmark, while theirs is Chinese-only; (2) their evaluation relies on artificially-constructed input formats, whereas our evaluation setting is more natural, directly using questions as prompts; and (3) we include detailed annotations of fallacy types, enabling more systematic analysis of model capabilities. Through these innovations, we aim to enable more rigorous assessment of how LLMs handle the types of deliberately tricky or misleading inputs they may encounter in real-world applications.
## 7 Conclusion
This paper presents RuozhiBench, a comprehensive benchmark designed to evaluate the logical reasoning capabilities of LLMs through both generative and multiple-choice formats. Our analysis across diverse models reveals that while state-of-the-art models like Claude demonstrate strong performance on logical reasoning tasks, significant challenges remain, particularly in handling edge cases and complex logical structures. The dual format of our benchmark provides complementary insights into models’ reasoning abilities, suggesting several promising directions for future research, including the enhancement of model training and the development of more targeted approaches to improving logical reasoning capabilities.
## Limitations
Despite our efforts to create a comprehensive benchmark for logical reasoning, RuozhiBench has several limitations. First, while our multiple-choice format offers standardized evaluation, it may not fully capture the nuanced reasoning processes that models employ in real-world scenarios. Second, our evaluation method relies heavily on model-generated responses for constructing the trapped options, which might not encompass all possible fallacies or reasoning errors that LLMs could make. Additionally, although the dataset is bilingual, our experiments focus primarily on English. Finally, the binary choice format in RuozhiBench -MC, while effective for evaluation, may inadvertently simplify complex reasoning problems that in practice require consideration of multiple valid perspectives or solutions.
## References
- Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program synthesis with large language models. CoRR, abs/2108.07732.
- Bai et al. (2023a) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023a. Qwen technical report. Preprint, arXiv:2309.16609.
- Bai et al. (2023b) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023b. Qwen technical report. arXiv preprint arXiv:2309.16609.
- Bai et al. (2024) Yuelin Bai, Xinrun Du, Yiming Liang, Yonggang Jin, Ziqiang Liu, Junting Zhou, Tianyu Zheng, Xincheng Zhang, Nuo Ma, Zekun Wang, Ruibin Yuan, Haihong Wu, Hongquan Lin, Wenhao Huang, Jiajun Zhang, Wenhu Chen, Chenghua Lin, Jie Fu, Min Yang, Shiwen Ni, and Ge Zhang. 2024. Coig-cqia: Quality is all you need for chinese instruction fine-tuning. Preprint, arXiv:2403.18058.
- Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence.
- Carlini et al. (2022) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646.
- Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code. CoRR, abs/2107.03374.
- Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457.
- Claude (2023) Claude. 2023. Our latest model, claude 2.1, is now available over api in our console and is powering our claude.ai chat experience.
- Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/2110.14168.
- DeepSeek-AI et al. (2025) DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
- Hendrycks et al. (2021a) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
- Hendrycks et al. (2021b) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
- Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088.
- Lei and Huang (2024) Yuanyuan Lei and Ruihong Huang. 2024. Boosting logical fallacy reasoning in LLMs via logical structure tree. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13157–13173, Miami, Florida, USA. Association for Computational Linguistics.
- Li et al. (2023a) Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023a. Cmmlu: Measuring massive multitask language understanding in chinese. CoRR.
- Li et al. (2023b) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023b. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval.
- Li et al. (2024a) Yanda Li, Dixuan Wang, Jiaqing Liang, Guochao Jiang, Qianyu He, Yanghua Xiao, and Deqing Yang. 2024a. Reason from fallacy: Enhancing large language models’ logical reasoning through logical fallacy understanding. Preprint, arXiv:2404.04293.
- Li et al. (2024b) Yinghui Li, Qingyu Zhou, Yuanzhen Luo, Shirong Ma, Yangning Li, Hai-Tao Zheng, Xuming Hu, and Philip S. Yu. 2024b. When LLMs meet cunning texts: A fallacy understanding benchmark for large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
- Lin et al. (2024) Mingan Lin, Fan Yang, Yanjun Shen, Haoze Sun, Tianpeng Li, Tao Zhang, Chenzheng Zhu, Tao Zhang, Miao Zheng, Xu Li, Yijie Zhou, Mingyang Chen, Yanzhao Qin, Youquan Li, Hao Liang, Fei Li, Yadong Li, Mang Wang, Guosheng Dong, Kun Fang, Jianhua Xu, Bin Cui, Wentao Zhang, Zenan Zhou, and Weipeng Chen. 2024. Baichuan alignment technical report. Preprint, arXiv:2410.14940.
- Liu et al. (2023a) Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. 2023a. Logiqa 2.0—an improved dataset for logical reasoning in natural language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2947–2962.
- Liu et al. (2023b) Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023b. Evaluating the logical reasoning ability of chatgpt and gpt-4. Preprint, arXiv:2304.03439.
- Liu et al. (2020) Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. Preprint, arXiv:2007.08124.
- Meta AI (2024) Meta AI. 2024. Introducing meta llama 3: The most capable openly available llm to date.
- MMA. (2024) MMA. 2024. American invitational mathematics examination - aime.
- OpenAI et al. (2024) OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghorbani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, et al. 2024. Openai o1 system card. Preprint, arXiv:2412.16720.
- OpenAI (2023) OpenAI. 2023. Gpt-4 technical report. Preprint, arXiv:2303.08774.
- Payandeh et al. (2023) Amirreza Payandeh, Dan Pluth, Jordan Hosier, Xuesu Xiao, and Vijay K. Gurbani. 2023. How susceptible are llms to logical fallacies? Preprint, arXiv:2308.09853.
- Phan et al. (2025) Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Tung Nguyen, Daron Anderson, Imad Ali Shah, Mikhail Doroshenko, Alun Cennyth Stokes, Mobeen Mahmood, et al. 2025. Humanity’s last exam. Preprint, arXiv:2501.14249.
- Sainz et al. (2023) Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singapore. Association for Computational Linguistics.
- Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. 2022. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100.
- Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
- Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
- Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics.
- Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
## Appendix A Prompts Used in This Study
Please read the following question and point out the irrationality in the question based on correct knowledge and common sense. The answer should be concise. (Note: Do not answer this question, do not use words like "the irrationality of this question is", your output only needs to include the irrationality of the question, try to use one sentence to complete the answer, and the answer should not exceed 100 words.) Example: Question: If the sun rises at night, what impact will it have on the temperature of the Earth? Irrationality Analysis: The sun does not rise at night because day and night are caused by the rotation of the Earth, and the phenomenon of the sun rising and falling is the result of the Earth’s rotation. Assuming that the sun rises at night is contrary to basic astronomical knowledge. Inputs: Question: {question}
Figure 10: Irrationality analysis generation prompt.
Based on the following tricky question and the irrationality analysis of this question, analyze and label them with three closest question categories. You will see all question categories in the question classification criteria, and you need to output the number sequence of question categories according to priority. Question Classification Criteria: 1. Logical error: When the question is raised, there may be logical contradictions or reasoning errors, which may include violations of logical rules, such as informal or formal logical errors, making the question logically untenable. 2. Common sense misunderstandings: The question reflects a misunderstanding of some basic common sense or universally accepted facts, usually involving incorrect understanding of common knowledge or information in daily life. 3. Erroneous assumption: The question is based on one or more erroneous assumptions. This means that the foundation or starting point of the question is incorrect, leading to inaccuracies in the entire question or its answer. 4. Scientific misconceptions: The content of the question involves misunderstandings of scientific principles or knowledge, including incorrect interpretations or applications of scientific theories, experimental results, or scientific methods. 5. Absurd imagination: The situation or setting in the question is clearly contrary to reality or common sense, has absurdity, and usually contains impossible or illogical elements, making the entire question appear unrealistic. 6. Others: If you believe that the provided question categories do not match the current question, please choose this option. Definitions: - Tricky Question: A question that is illogical or lacks a clear answer, often based on unrealistic assumptions or contradictions to common sense. - Irrationality Analysis: A detailed explanation highlighting the logical flaws or unrealistic assumptions present in the tricky question. Output format: The output format must be *Number Space Number Space Number*, and the output string length must be 5. Example: 2 6 3 Inputs: Question: {question} Irrationality Analysis: {irrationality}
Figure 11: Question type classification prompt.
You are an evaluation system designed to assess responses to *deceptive* or *logically flawed* questions. ## Task Definition: For each input, you will be provided with: - A **deceptive question** that contains misleading premises, logical fallacies, or contradictions. - An **irrationality analysis**, explaining why the question is deceptive and identifying its logical flaws. - An **answer** provided by a model or a human. Your task is to **evaluate the answer’s ability to correctly identify and address the deception** rather than simply providing a direct response. ## Rating Criteria (0-4 Scale): - **0**: The answer completely fails to recognize the deception and directly attempts to answer the question as if it were valid. - **1**: The answer acknowledges a flaw but does not engage with it meaningfully. - **2**: The answer recognizes and partially explains the deception but has logical gaps. - **3**: The answer correctly identifies and mostly explains the fallacy but lacks depth or clarity. - **4**: The answer fully explains the deceptive nature, providing a comprehensive and logically sound explanation. <question> \n {question} \n </question> <irrationality_analysis> \n {irrationality_analysis} \n </irrationality_analysis> <answer> \n {answer} \n </answer> ## Output Format: You must return a JSON object with keys: "rating" (int) and "explanation" (str), for example: {{\n "rating": [2], "explanation": "[Your justification for the rating]" \n}} Ensure that your response is **valid JSON** with the exact keys shown above without any additional text or formatting.
Figure 12: RuozhiBench-Gen evaluation prompt.
You are an expert evaluator tasked with assessing the quality of an answer to a given question. ## Question: \n {question} ## Answer: \n {answer} ## Evaluation Criteria: Rate the answer based on the following criteria: - 4: Excellent – The answer is complete, accurate, and well-explained. - 3: Good – The answer is mostly correct with minor inaccuracies or missing details. - 2: Fair – The answer has some correct elements but contains notable errors or omissions. - 1: Poor – The answer is mostly incorrect or incomplete. - 0: Very Poor – The answer is irrelevant or completely incorrect. ## Output Format: You must return a JSON object with keys: "rating" (int) and "explanation" (str), for example: {\n "rating": [0, 1, 2, 3, or 4], "explanation": "[Your justification for the rating]" \n} Ensure that your response is **valid JSON** with the exact keys shown above without any additional text or formatting.
Figure 13: RuozhiBench-MC evaluation prompt.
## Appendix B Option Distribution of RuozhiBench-MC
Figure 15 shows the option data source statistics of RuozhiBench-MC, and Figure 15 shows the gap distribution between “Good” and “Bad” options.
<details>
<summary>x10.png Details</summary>
### Visual Description
## Bar Chart: Distribution of Good and Bad Answers by Model
### Overview
This bar chart visualizes the distribution of "Good Answers" and "Bad Answers" across various language models. The x-axis represents the different models, and the y-axis represents the count of answers categorized as either good or bad. Each model has two bars: one for good answers (blue) and one for bad answers (orange).
### Components/Axes
* **Title:** "Distribution of Good and Bad Answers by Model" (positioned at the top-center)
* **X-axis Label:** Model names (Llama-2-3.2-1B-instruct, Llama-2-3.2-3B-instruct, Llama-3.1-8B-instruct, Mistral-7B-instruct, Mixtral-8x7B-instruct-v0.1, Mixtral-8x22B-instruct-v0.1, Qwen2.5-0.5B-instruct, Qwen2.5-3B-instruct, Qwen2.5-7B-instruct, Qwen2.5-32B-instruct, Qwen2.5-72B-instruct, claude-3-haiku-20240307, gpt-4o-mini-2024-07-18, claude-3-sonnet-20240229, gpt-4o-2024-05-13)
* **Y-axis Label:** "Count" (ranging from 0 to 175)
* **Legend:** Located at the top-right corner, labeling the colors: "Good Answers" (blue) and "Bad Answers" (orange).
### Detailed Analysis
The chart consists of 15 models, each represented by two bars. The following data points are approximate, based on visual estimation:
* **Llama-2-3.2-1B-instruct:** Good Answers ≈ 10, Bad Answers ≈ 5
* **Llama-2-3.2-3B-instruct:** Good Answers ≈ 20, Bad Answers ≈ 10
* **Llama-3.1-8B-instruct:** Good Answers ≈ 30, Bad Answers ≈ 15
* **Mistral-7B-instruct:** Good Answers ≈ 40, Bad Answers ≈ 20
* **Mixtral-8x7B-instruct-v0.1:** Good Answers ≈ 60, Bad Answers ≈ 30
* **Mixtral-8x22B-instruct-v0.1:** Good Answers ≈ 130, Bad Answers ≈ 65
* **Qwen2.5-0.5B-instruct:** Good Answers ≈ 50, Bad Answers ≈ 25
* **Qwen2.5-3B-instruct:** Good Answers ≈ 70, Bad Answers ≈ 35
* **Qwen2.5-7B-instruct:** Good Answers ≈ 100, Bad Answers ≈ 50
* **Qwen2.5-32B-instruct:** Good Answers ≈ 150, Bad Answers ≈ 75
* **Qwen2.5-72B-instruct:** Good Answers ≈ 80, Bad Answers ≈ 40
* **claude-3-haiku-20240307:** Good Answers ≈ 45, Bad Answers ≈ 22
* **gpt-4o-mini-2024-07-18:** Good Answers ≈ 35, Bad Answers ≈ 17
* **claude-3-sonnet-20240229:** Good Answers ≈ 65, Bad Answers ≈ 32
* **gpt-4o-2024-05-13:** Good Answers ≈ 55, Bad Answers ≈ 27
**Trends:**
* Generally, the "Good Answers" bar is taller than the "Bad Answers" bar for most models, indicating a higher proportion of good answers.
* The count of both good and bad answers tends to increase with model size (as indicated by the model name).
* Mixtral-8x22B-instruct-v0.1 has the highest counts for both good and bad answers.
* Qwen2.5-32B-instruct has the highest count of good answers.
### Key Observations
* Mixtral-8x22B-instruct-v0.1 stands out with significantly higher counts compared to other models.
* Smaller models (e.g., Llama-2-3.2-1B-instruct) have the lowest counts for both good and bad answers.
* The ratio of good to bad answers appears relatively consistent across most models, although larger models tend to have a higher absolute number of both.
### Interpretation
The chart demonstrates the performance of different language models in generating answers, categorized as either "good" or "bad." The data suggests a positive correlation between model size and the absolute number of answers generated (both good and bad). Larger models, like Mixtral-8x22B-instruct-v0.1 and Qwen2.5-32B-instruct, are capable of producing a greater volume of responses. However, it's important to note that a higher count doesn't necessarily equate to better quality; the proportion of good answers relative to bad answers should also be considered. The consistent presence of "bad answers" across all models highlights the ongoing challenges in achieving perfect accuracy and reliability in language model outputs. The specific criteria used to define "good" and "bad" answers are not provided, which limits a deeper analysis of the results. Further investigation into the evaluation methodology would be necessary to understand the nuances of the performance differences between models.
</details>
Figure 14: Distribution of Good and Bad Answers by Model. The figure shows the total number of responses across various models, divided into good and bad answers. Most models exhibit a relatively balanced distribution, while models like Claude 3 Sonnet, Mixtral 8x22B, and GPT-4o produce a higher proportion of good answers. In contrast, models like Qwen 2.5 0.5B have a substantial number of responses but with a higher proportion of bad answers.
<details>
<summary>x11.png Details</summary>
### Visual Description
\n
## Histogram: Rating Frequency Distribution
### Overview
The image presents a histogram visualizing the frequency distribution of ratings. The x-axis represents the rating scale, and the y-axis represents the frequency (count) of each rating. The data appears to be discrete, with ratings ranging from 0 to 4.
### Components/Axes
* **X-axis Title:** "Rating"
* **Y-axis Title:** "Frequency"
* **X-axis Scale:** Ranges from 0 to 4, with tick marks at integer values.
* **Y-axis Scale:** Ranges from 0 to 200, with tick marks at intervals of 50.
* **Bars:** Represent the frequency of each rating.
### Detailed Analysis
The histogram shows a distinct distribution of ratings. The highest frequency occurs at a rating of 2, with approximately 210 occurrences. The frequency decreases as the rating moves away from 2 in either direction.
Here's a breakdown of approximate frequencies for each rating:
* **Rating 0:** Approximately 10 occurrences.
* **Rating 1:** Approximately 15 occurrences.
* **Rating 2:** Approximately 210 occurrences.
* **Rating 3:** Approximately 110 occurrences.
* **Rating 4:** Approximately 25 occurrences.
The bars are centered on the integer rating values (0, 1, 2, 3, 4). The width of each bar appears to be 1 unit.
### Key Observations
* The distribution is heavily skewed towards a rating of 2.
* Ratings of 0 and 1 are relatively rare.
* The frequency decreases significantly for ratings above 2.
* The data suggests a concentration of responses around the rating of 2.
### Interpretation
The histogram indicates that the majority of the ratings fall within the value of 2. This could suggest that the subject being rated is generally considered "average" or "moderate" based on this scale. The low frequencies for ratings of 0 and 1 suggest that very negative assessments are uncommon. The decline in frequency for ratings of 3 and 4 indicates that very positive assessments are also less frequent.
The shape of the distribution (peaked at 2 and decreasing towards the extremes) suggests a unimodal distribution. This implies that there is a single dominant rating value, and the ratings tend to cluster around that value. Without knowing the context of the ratings, it's difficult to draw more specific conclusions. However, the data clearly demonstrates a preference for the rating of 2.
</details>
Figure 15: “Good” and “Bad” answer scores distribution. The majority of the data falls into categories with score differences greater than 2, indicating a clear gap between the options.
## Appendix C Recruitment and Payment
We hired 2 annotators with bachelor’s degrees or higher from China with an hourly rate of 50 Chinese Yuan. The annotators are native Chinese speakers and have studied English for more than 10 years. This rate is higher than the average hourly wage in China.
## Appendix D Full Evaluation Results on RuozhiBench-Gen
<details>
<summary>x12.png Details</summary>
### Visual Description
## Data Table: Model Performance Across Error Categories
### Overview
This image presents a data table comparing the performance of various language models across six different categories of errors: Absurd Imagination, Commonsense Misunderstanding, Erroneous Assumption, Logical Error, Others, and Scientific Misconception. The table also includes an 'Average' column representing the overall performance score for each model.
### Components/Axes
* **Rows:** Represent individual language models. The models listed are: Mixtral-8x22B-v0.1, claude-3-haiku-20240307, Qwen2.5-32B, Mixtral-8x7B-v0.1, Llama-3-70B, gpt-4o-2024-05-13, Qwen2.5-72B, gpt-4o-mini-2024-07-18, Qwen2.5-7B, Llama-3-8B, Qwen2.5-3B, claude-3-sonnet-20240229, Llama-2-70B, Qwen2.5-1.5B, Llama-2-13B, Qwen2.5-11B, and Qwen2.5-0.5B.
* **Columns:** Represent error categories and the average score. The categories are: Absurd Imagination, Commonsense Misunderstanding, Erroneous Assumption, Logical Error, Others, Scientific Misconception, and Average.
* **Data:** Numerical values representing the performance score for each model in each category.
### Detailed Analysis or Content Details
Here's a reconstruction of the data table, with approximate values:
| Model | Absurd Imagination | Commonsense Misunderstanding | Erroneous Assumption | Logical Error | Others | Scientific Misconception | Average |
| ------------------------- | ------------------ | ---------------------------- | -------------------- | ------------- | ------ | ------------------------ | ------- |
| Mixtral-8x22B-v0.1 | 41.78 | 39.35 | 36.73 | 34.35 | 44.12 | 38.39 | 38.52 |
| claude-3-haiku-20240307 | 37.67 | 39.05 | 38.55 | 32.07 | 47.06 | 46.43 | 38.37 |
| Qwen2.5-32B | 29.92 | 31.35 | 31.28 | 28.57 | 25.00 | 45.54 | 30.54 |
| Mixtral-8x7B-v0.1 | 31.66 | 30.67 | 28.97 | 27.34 | 19.12 | 37.50 | 29.84 |
| Llama-3-70B | 27.90 | 30.16 | 28.71 | 23.20 | 29.41 | 36.61 | 28.51 |
| gpt-4o-2024-05-13 | 28.42 | 28.41 | 27.54 | 24.28 | 29.41 | 33.93 | 27.58 |
| Qwen2.5-72B | 26.87 | 27.15 | 27.15 | 27.32 | 26.47 | 20.54 | 26.66 |
| gpt-4o-mini-2024-07-18 | 17.61 | 18.79 | 19.09 | 17.45 | 17.65 | 26.79 | 18.54 |
| Qwen2.5-7B | 16.91 | 19.00 | 18.39 | 15.71 | 2.94 | 18.75 | 17.21 |
| Llama-3-8B | 17.77 | 17.54 | 17.20 | 15.65 | 9.38 | 24.11 | 16.99 |
| Qwen2.5-3B | 16.67 | 15.00 | 15.03 | 16.01 | 16.18 | 16.07 | 15.51 |
| claude-3-sonnet-20240229 | 15.26 | 13.13 | 14.60 | 13.67 | 19.12 | 10.74 | 14.96 |
| Llama-2-70B | 13.84 | 12.58 | 12.85 | 13.06 | 4.42 | 20.51 | 12.92 |
| Qwen2.5-1.5B | 11.75 | 10.46 | 10.39 | 12.20 | 4.14 | 12.78 | 9.27 |
| Llama-2-13B | 10.62 | 9.28 | 9.01 | 10.52 | 4.01 | 12.35 | 8.38 |
| Qwen2.5-11B | 9.54 | 8.24 | 8.10 | 8.76 | 3.85 | 10.91 | 7.88 |
| Qwen2.5-0.5B | 7.45 | 6.15 | 6.09 | 6.21 | 3.50 | 8.23 | 6.08 |
### Key Observations
* **Mixtral-8x22B-v0.1** consistently scores highest across most categories, indicating strong performance.
* **Qwen2.5-0.5B** consistently scores lowest, suggesting it is most susceptible to these error types.
* The 'Others' category shows significant variation across models.
* 'Scientific Misconception' scores are generally higher than other categories, indicating a common challenge for these models.
* There is a general trend of decreasing scores as model size decreases (e.g., comparing Qwen2.5-32B to Qwen2.5-0.5B).
### Interpretation
This data table provides a comparative analysis of language model performance across different types of errors. The results suggest that larger models (like Mixtral-8x22B-v0.1 and claude-3-haiku-20240307) are more robust and less prone to making these errors. The 'Others' category, with its high variance, likely encompasses a diverse range of less-defined error types. The relatively high scores in 'Scientific Misconception' highlight the difficulty these models have with complex, factual reasoning. The consistent trend of decreasing performance with smaller model sizes underscores the importance of model scale in mitigating these error types. This data could be used to inform model selection and development efforts, focusing on improving performance in areas where models consistently struggle. The table allows for a direct comparison of model strengths and weaknesses, providing valuable insights for researchers and practitioners.
</details>
Figure 16: Overall score on RuozhiBench-Gen using Claude-3-5-sonnet as an evaluator.
<details>
<summary>x13.png Details</summary>
### Visual Description
## Heatmap: Model Performance Across Error Categories
### Overview
This heatmap displays the performance of various language models across six different categories of errors: Absurd Imagination, Commonsense Misunderstanding, Erroneous Assumption, Logical Error, Others, and Scientific Misconception. The values represent a score, presumably a percentage, indicating the frequency or severity of each error type for each model. An "Average" column provides the mean score across all error categories for each model.
### Components/Axes
* **Rows:** Represent different language models: Llama-3-1.70B, claude-3-haiku-20240307, Mixtral-8x22B-v0.1, Qwen2.5-32B, Qwen2.5-72B, gpt-4o-2024-05-13, Mixtral-8x7B-v0.1, Qwen2.5-7B, gpt-4o-mini-2024-07-18, Qwen2.5-3B, claude-3-sonnet-20240229, Llama2.5-7B, Llama-2-13B, Llama-7B, Llama-3-8B, Llama-2-70B.
* **Columns:** Represent error categories: Absurd Imagination, Commonsense Misunderstanding, Erroneous Assumption, Logical Error, Others, Scientific Misconception, and Average.
* **Color Scale:** The heatmap uses a color gradient, with darker shades representing higher values and lighter shades representing lower values. The color scale is not explicitly provided, but appears to range from light yellow to dark green.
* **Legend:** The column headers act as the legend, associating each color shade with a specific error category.
### Detailed Analysis
The data is presented in a 16x7 grid. I will analyze each model's performance across the error categories, noting trends and specific values. All values are approximate, with an uncertainty of ±0.05.
* **Llama-3-1.70B:** Shows high scores in Absurd Imagination (65.95), Commonsense Misunderstanding (65.55), Erroneous Assumption (65.09), Logical Error (64.11), Others (54.41), Scientific Misconception (74.11), and Average (65.32).
* **claude-3-haiku-20240307:** Scores are 60.24, 60.05, 61.45, 56.61, 61.76, 66.96, 60.67.
* **Mixtral-8x22B-v0.1:** Scores are 58.03, 56.40, 54.19, 56.07, 60.29, 59.82, 56.50.
* **Qwen2.5-32B:** Scores are 56.45, 57.60, 55.84, 57.68, 42.65, 65.18, 56.39.
* **Qwen2.5-72B:** Scores are 53.62, 53.30, 52.53, 54.11, 41.18, 51.79, 53.06.
* **gpt-4o-2024-05-13:** Scores are 53.28, 52.70, 53.14, 50.00, 50.00, 58.93, 52.77.
* **Mixtral-8x7B-v0.1:** Scores are 53.39, 51.15, 50.88, 49.46, 42.65, 56.25, 51.48.
* **Qwen2.5-7B:** Scores are 42.36, 42.65, 42.79, 40.54, 41.18, 54.46, 42.54.
* **gpt-4o-mini-2024-07-18:** Scores are 41.35, 40.95, 41.13, 41.07, 41.18, 46.43, 41.36.
* **Qwen2.5-3B:** Scores are 40.21, 35.80, 36.18, 40.54, 44.12, 51.79, 38.40.
* **claude-3-sonnet-20240229:** Scores are 37.33, 36.35, 36.12, 33.75, 42.65, 43.75, 36.82.
* **Llama2.5-7B:** Scores are 34.50, 33.30, 31.50, 32.68, 32.35, 34.82, 32.98.
* **Llama-2-13B:** Scores are 32.28, 28.75, 29.43, 28.75, 31.00, 35.56, 29.88.
* **Llama-7B:** Scores are 26.45, 24.85, 24.99, 25.32, 28.82, 28.67, 25.81.
* **Llama-3-8B:** Scores are 24.90, 22.90, 20.79, 20.58, 30.06, 30.82, 24.60.
* **Llama-2-70B:** Scores are 21.54, 18.54, 18.33, 18.26, 26.41, 27.54, 20.00.
### Key Observations
* **Llama-3-1.70B** consistently scores the highest across all error categories, indicating it is the most prone to these types of errors.
* **Llama-2-70B** consistently scores the lowest, suggesting it is the most robust against these errors.
* **Scientific Misconception** generally has the highest scores across all models, indicating this is the most common type of error.
* **Commonsense Misunderstanding** also tends to be high for most models.
* The "Others" category generally has lower scores than the specific error types.
* There is a general trend of larger models (higher parameter count) exhibiting lower error rates, although this is not always consistent.
### Interpretation
The heatmap provides a comparative analysis of the error profiles of different language models. The data suggests that while larger models like Llama-3-1.70B demonstrate strong overall performance, they are also more susceptible to generating responses that fall into these error categories. This could be due to their increased capacity to generate complex and nuanced responses, which also increases the risk of making subtle errors. Conversely, smaller models like Llama-2-70B, while potentially less capable, are more conservative in their responses and therefore less prone to these errors.
The prevalence of "Scientific Misconception" and "Commonsense Misunderstanding" suggests that these are areas where language models still struggle, even the most advanced ones. This highlights the need for continued research and development in these areas, particularly in improving the models' ability to reason about the real world and apply common sense knowledge.
The heatmap allows for a nuanced understanding of model strengths and weaknesses, which can be valuable for selecting the appropriate model for a specific task and for identifying areas where further training or refinement is needed. The data also suggests that simply increasing model size is not a guaranteed solution to all problems, and that other factors, such as training data and model architecture, also play a crucial role.
</details>
Figure 17: Overall score on RuozhiBench-Gen using GPT-4o-2024-08-06 as an evaluator.
<details>
<summary>x14.png Details</summary>
### Visual Description
## Heatmap: Model Performance Across Error Categories
### Overview
This heatmap displays the performance of various language models across six different categories of errors: Absurd Imagination, Commonsense Misunderstanding, Erroneous Assumption, Logical Error, Others, and Scientific Misconception. Performance is measured as a percentage, and an "Average" score is provided for each model. The heatmap uses a color gradient to represent performance, with darker shades indicating higher scores.
### Components/Axes
* **Rows:** Represent different language models (claude-3-haiku-20240307, Qwen2.5-32B, Qwen2.5-72B, gpt-4o-2024-05-13, Mixtral-8x22B-v0.1, Llama-3-1-70B, Qwen2.5-7B, Mixtral-8x7B-v0.1, Qwen2.5-3B, gpt-4o-mini-2024-07-18, Llama-3-1-8B, claude-3-sonnet-20240229, Llama-7B-v2, Llama-2-70B, Llama-13B-v2, Qwen1.5-110B).
* **Columns:** Represent error categories: Absurd Imagination, Commonsense Misunderstanding, Erroneous Assumption, Logical Error, Others, Scientific Misconception, and Average.
* **Color Scale:** A gradient from light to dark, where darker shades represent higher percentage values.
* **Legend:** Not explicitly present, but the color gradient serves as an implicit legend.
### Detailed Analysis
Here's a breakdown of the data, row by row, with approximate values and trend observations. I will state the model, then the values for each category in order: Absurd Imagination, Commonsense Misunderstanding, Erroneous Assumption, Logical Error, Others, Scientific Misconception, Average.
1. **claude-3-haiku-20240307:** 88.07, 86.75, 87.56, 86.07, 80.88, 87.50, 86.96
2. **Qwen2.5-32B:** 86.31, 87.00, 86.23, 87.68, 72.06, 87.50, 86.26
3. **Qwen2.5-72B:** 84.79, 84.80, 83.70, 87.14, 69.12, 83.04, 84.49
4. **gpt-4o-2024-05-13:** 83.09, 83.60, 83.81, 81.79, 83.82, 77.68, 82.94
5. **Mixtral-8x22B-v0.1:** 82.75, 82.50, 80.84, 81.43, 79.41, 81.25, 81.94
6. **Llama-3-1-70B:** 79.24, 79.90, 79.07, 78.75, 73.53, 80.36, 79.51
7. **Qwen2.5-7B:** 79.02, 79.50, 78.96, 78.21, 72.06, 79.46, 79.06
8. **Mixtral-8x7B-v0.1:** 80.32, 79.40, 77.81, 77.32, 72.06, 81.25, 78.73
9. **Qwen2.5-3B:** 79.02, 77.15, 77.26, 75.54, 67.65, 81.25, 77.29
10. **gpt-4o-mini-2024-07-18:** 73.59, 73.40, 74.39, 75.89, 69.12, 75.89, 73.78
11. **Llama-3-1-8B:** 70.19, 69.80, 69.22, 69.46, 61.76, 69.64, 69.68
12. **claude-3-sonnet-20240229:** 67.99, 67.55, 68.94, 69.82, 67.65, 75.00, 68.39
13. **Llama-7B-v2:** 65.61, 65.75, 61.87, 63.56, 64.12, 68.89, 65.03
14. **Llama-2-70B:** 59.24, 59.40, 58.03, 57.35, 42.44, 62.50, 58.31
15. **Llama-13B-v2:** 45.22, 43.60, 45.04, 47.54, 41.52, 56.74, 46.31
16. **Qwen1.5-110B:** 42.96, 42.85, 43.21, 46.82, 42.44, 59.00, 47.06
17. **Llama-34B-v2:** 41.54, 41.20, 40.96, 43.52, 40.88, 53.26, 42.85
**Trends:**
* **claude-3-haiku-20240307** consistently performs the highest across most categories.
* **Qwen2.5-32B** and **Qwen2.5-72B** also show strong performance, generally above 80% in most categories.
* The Llama models (especially the smaller ones like Llama-7B-v2, Llama-13B-v2, and Llama-34B-v2) generally exhibit lower performance, particularly in categories like Commonsense Misunderstanding and Erroneous Assumption.
* Performance generally decreases as the model size decreases (within the Llama family).
### Key Observations
* The largest performance differences are observed in the "Others" and "Commonsense Misunderstanding" categories.
* The models demonstrate a relatively consistent performance profile across the error categories, with some models consistently outperforming others.
* There is a clear trade-off between model size and performance, with larger models generally achieving higher scores.
* The "Absurd Imagination" category consistently has the highest scores, suggesting that models are generally good at avoiding nonsensical outputs.
### Interpretation
This heatmap provides a comparative analysis of language model performance across different types of errors. The data suggests that models like claude-3-haiku-20240307, Qwen2.5-32B, and Qwen2.5-72B are more robust and less prone to making errors in various scenarios. The lower scores in the "Others" category indicate that these models struggle with errors that don't fall into well-defined categories, potentially highlighting areas where further improvement is needed.
The consistent trend of larger models performing better suggests that model capacity plays a crucial role in error reduction. However, it's important to note that model size is not the only factor, as different architectures and training data can also influence performance.
The heatmap also reveals that models are generally better at avoiding absurd outputs than at understanding commonsense or identifying erroneous assumptions. This suggests that current training methods may be more effective at preventing nonsensical responses than at ensuring logical reasoning and factual accuracy.
The data can be used to inform model selection and development efforts, guiding researchers and practitioners towards models that are best suited for specific tasks and applications. Further investigation into the "Others" category could reveal valuable insights into the types of errors that are most challenging for language models to overcome.
</details>
Figure 18: Overall score on RuozhiBench-Gen using Llama-3.3-70B-Instruction as an evaluator.
## Appendix E Rating Distribution of Evaluators on RuozhiBench-Gen
<details>
<summary>x15.png Details</summary>
### Visual Description
\n
## Stacked Bar Chart: Rating Distribution by Model
### Overview
The image presents a stacked bar chart visualizing the distribution of ratings (0 to 4) for various language models. The chart is titled "Rating Distribution by Model" and indicates the evaluator used was "claude-3-sonnet-20241022". The x-axis represents different language models, and the y-axis represents the proportion of ratings. Each bar is segmented to show the proportion of each rating level for that model.
### Components/Axes
* **Title:** Rating Distribution by Model
* **Evaluator:** claude-3-sonnet-20241022
* **X-axis:** Model Name (Categorical)
* Llama-3-70B-Instruct
* Llama-3-8B-Instruct
* Llama-3-2-1B-Instruct
* Llama-3-2-3B-Instruct
* Mistral-7B-Instruct
* Mistral-8x22B-Instruct-v0.1
* Mixtral-8x7B-Instruct-v0.1
* Qwen2-3.5-0.5B-Instruct
* Qwen2-5-32B-Instruct
* Qwen2-5-3B-Instruct
* Qwen2-5-72B-Instruct
* Qwen2-5-7B-Instruct
* claude-3-haiku-20240307
* claude-3-sonnet-202405-13
* gpt-4o-mini-2024-07-18
* **Y-axis:** Proportion (Scale: 0.0 to 1.0)
* **Legend:** Rating (Categorical)
* 0 (Blue)
* 1 (Orange)
* 2 (Green)
* 3 (Red)
* 4 (Purple)
### Detailed Analysis
The chart displays the proportion of each rating for each model. The bars are stacked, meaning the total height of each bar represents the total proportion (which should ideally sum to 1.0, though minor rounding errors may exist).
Here's a breakdown of the approximate proportions for each model, based on visual estimation and color-matching to the legend:
* **Llama-3-70B-Instruct:** ~0.05 (0), ~0.1 (1), ~0.2 (2), ~0.4 (3), ~0.25 (4)
* **Llama-3-8B-Instruct:** ~0.1 (0), ~0.15 (1), ~0.25 (2), ~0.35 (3), ~0.15 (4)
* **Llama-3-2-1B-Instruct:** ~0.2 (0), ~0.2 (1), ~0.25 (2), ~0.25 (3), ~0.1 (4)
* **Llama-3-2-3B-Instruct:** ~0.15 (0), ~0.15 (1), ~0.25 (2), ~0.3 (3), ~0.15 (4)
* **Mistral-7B-Instruct:** ~0.1 (0), ~0.15 (1), ~0.2 (2), ~0.35 (3), ~0.2 (4)
* **Mistral-8x22B-Instruct-v0.1:** ~0.05 (0), ~0.1 (1), ~0.2 (2), ~0.4 (3), ~0.25 (4)
* **Mixtral-8x7B-Instruct-v0.1:** ~0.05 (0), ~0.1 (1), ~0.15 (2), ~0.4 (3), ~0.3 (4)
* **Qwen2-3.5-0.5B-Instruct:** ~0.2 (0), ~0.2 (1), ~0.2 (2), ~0.25 (3), ~0.15 (4)
* **Qwen2-5-32B-Instruct:** ~0.05 (0), ~0.1 (1), ~0.15 (2), ~0.4 (3), ~0.3 (4)
* **Qwen2-5-3B-Instruct:** ~0.15 (0), ~0.15 (1), ~0.2 (2), ~0.3 (3), ~0.2 (4)
* **Qwen2-5-72B-Instruct:** ~0.05 (0), ~0.1 (1), ~0.15 (2), ~0.4 (3), ~0.3 (4)
* **Qwen2-5-7B-Instruct:** ~0.1 (0), ~0.15 (1), ~0.2 (2), ~0.35 (3), ~0.2 (4)
* **claude-3-haiku-20240307:** ~0.1 (0), ~0.1 (1), ~0.2 (2), ~0.3 (3), ~0.3 (4)
* **claude-3-sonnet-202405-13:** ~0.05 (0), ~0.05 (1), ~0.1 (2), ~0.3 (3), ~0.5 (4)
* **gpt-4o-mini-2024-07-18:** ~0.05 (0), ~0.05 (1), ~0.1 (2), ~0.2 (3), ~0.6 (4)
### Key Observations
* **gpt-4o-mini-2024-07-18** and **claude-3-sonnet-202405-13** have the highest proportion of rating 4, indicating they received the most positive evaluations.
* **Llama-3-2-1B-Instruct** and **Qwen2-3.5-0.5B-Instruct** have the highest proportion of rating 0, suggesting they received the most negative evaluations.
* Most models have a significant proportion of ratings in the 2 and 3 range, indicating a mixed reception.
* There's a clear trend of larger models (e.g., Llama-3-70B-Instruct, Mixtral-8x7B-Instruct-v0.1) tending to receive higher ratings (more 3s and 4s) compared to smaller models.
### Interpretation
The chart provides a comparative assessment of the performance of different language models, as judged by the "claude-3-sonnet-20241022" evaluator. The stacked bar chart effectively visualizes the distribution of ratings, allowing for quick identification of models that consistently receive high or low scores.
The dominance of ratings 3 and 4 for models like gpt-4o-mini-2024-07-18 and claude-3-sonnet-202405-13 suggests these models are generally considered to be of higher quality or more useful. Conversely, the higher proportion of rating 0 for models like Llama-3-2-1B-Instruct and Qwen2-3.5-0.5B-Instruct indicates potential issues with their performance or usability.
The observed trend of larger models receiving higher ratings aligns with the general expectation that model capacity and complexity correlate with performance. However, it's important to note that this is just one evaluator's perspective, and the results may vary depending on the evaluation criteria and the specific tasks used. Further investigation with different evaluators and datasets would be necessary to draw more definitive conclusions. The data suggests a clear hierarchy of model performance, with the newer and larger models generally outperforming the smaller ones.
</details>
Figure 19: Rating distribution on RuozhiBench-Gen using Claude-3-5-sonnet as an evaluator.
<details>
<summary>x16.png Details</summary>
### Visual Description
## Stacked Bar Chart: Rating Distribution by Model
### Overview
The image presents a stacked bar chart visualizing the rating distribution for various language models, as evaluated by "gpt-4o-2024-08-06". The chart displays the proportion of each rating (0 to 4) for each model. The x-axis lists the model names, and the y-axis represents the proportion, ranging from 0.0 to 1.0.
### Components/Axes
* **Title:** "Rating Distribution by Model"
* **Evaluator:** "gpt-4o-2024-08-06" (located at the top of the chart)
* **X-axis Label:** Model Name
* **Y-axis Label:** Proportion
* **Y-axis Scale:** 0.0 to 1.0, with increments of 0.2
* **Legend:** Located in the top-right corner, mapping colors to ratings:
* Blue: 0
* Light Blue: 1
* Purple: 2
* Pink: 3
* Red: 4
* **Models (X-axis):**
* Llama-3-70B-Instruct
* Llama-3-8B-Instruct
* Llama-3-2-1B-Instruct
* Mistral-7B-Instruct
* Mistral-8x22B-Instruct-v0.1
* Mixtral-8x7B-Instruct-v0.1
* Qwen2-3.5-0.5B-Instruct
* Qwen2-5-32B-Instruct
* Qwen2-5-3B-Instruct
* Qwen2-5-7B-Instruct
* Qwen2-5-7B-Instruct-20240307
* claude-3-haiku-20240229
* claude-3-sonnet-2024-05-13
* gpt-4o-mini-2024-07-18
### Detailed Analysis
The chart consists of stacked bars, each representing a model. The height of each segment within a bar indicates the proportion of responses receiving that specific rating.
* **Llama-3-70B-Instruct:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.2 proportion of rating 2, 0.3 proportion of rating 3, and 0.35 proportion of rating 4.
* **Llama-3-8B-Instruct:** Approximately 0.1 proportion of rating 0, 0.15 proportion of rating 1, 0.2 proportion of rating 2, 0.25 proportion of rating 3, and 0.3 proportion of rating 4.
* **Llama-3-2-1B-Instruct:** Approximately 0.2 proportion of rating 0, 0.2 proportion of rating 1, 0.2 proportion of rating 2, 0.2 proportion of rating 3, and 0.2 proportion of rating 4.
* **Mistral-7B-Instruct:** Approximately 0.1 proportion of rating 0, 0.1 proportion of rating 1, 0.2 proportion of rating 2, 0.3 proportion of rating 3, and 0.3 proportion of rating 4.
* **Mistral-8x22B-Instruct-v0.1:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.15 proportion of rating 2, 0.3 proportion of rating 3, and 0.4 proportion of rating 4.
* **Mixtral-8x7B-Instruct-v0.1:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.15 proportion of rating 2, 0.25 proportion of rating 3, and 0.45 proportion of rating 4.
* **Qwen2-3.5-0.5B-Instruct:** Approximately 0.2 proportion of rating 0, 0.2 proportion of rating 1, 0.2 proportion of rating 2, 0.2 proportion of rating 3, and 0.2 proportion of rating 4.
* **Qwen2-5-32B-Instruct:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.15 proportion of rating 2, 0.3 proportion of rating 3, and 0.4 proportion of rating 4.
* **Qwen2-5-3B-Instruct:** Approximately 0.1 proportion of rating 0, 0.15 proportion of rating 1, 0.2 proportion of rating 2, 0.25 proportion of rating 3, and 0.3 proportion of rating 4.
* **Qwen2-5-7B-Instruct:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.15 proportion of rating 2, 0.3 proportion of rating 3, and 0.4 proportion of rating 4.
* **Qwen2-5-7B-Instruct-20240307:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.15 proportion of rating 2, 0.3 proportion of rating 3, and 0.4 proportion of rating 4.
* **claude-3-haiku-20240229:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.15 proportion of rating 2, 0.3 proportion of rating 3, and 0.4 proportion of rating 4.
* **claude-3-sonnet-2024-05-13:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.15 proportion of rating 2, 0.3 proportion of rating 3, and 0.4 proportion of rating 4.
* **gpt-4o-mini-2024-07-18:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.15 proportion of rating 2, 0.3 proportion of rating 3, and 0.4 proportion of rating 4.
### Key Observations
* Most models exhibit a similar distribution, with a higher proportion of ratings 3 and 4.
* Llama-3-2-1B-Instruct has a more uniform distribution across all ratings, indicating more variability in its performance.
* Models like Mixtral-8x7B-Instruct-v0.1 and Qwen2-5-32B-Instruct show a relatively higher proportion of rating 4, suggesting better performance.
* The proportion of rating 0 is consistently low across all models.
### Interpretation
The chart demonstrates the performance of various language models based on a rating scale from 0 to 4, as judged by the "gpt-4o-2024-08-06" evaluator. The stacked bar chart effectively visualizes the distribution of these ratings for each model, allowing for a quick comparison of their relative strengths and weaknesses. The consistent trend of higher ratings (3 and 4) across most models suggests a generally high level of performance. The variability observed in Llama-3-2-1B-Instruct could indicate a wider range of response quality, while the higher proportion of top ratings for models like Mixtral-8x7B-Instruct-v0.1 and Qwen2-5-32B-Instruct suggests they consistently generate higher-quality responses according to this evaluator. The evaluator's identity is crucial context; the results are specific to its criteria and biases. The data suggests that the evaluator generally finds the models to be performing well, but there are noticeable differences in the consistency and quality of their outputs.
</details>
Figure 20: Rating distribution on RuozhiBench-Gen using GPT-4o-2024-08-06 as an evaluator.
<details>
<summary>x17.png Details</summary>
### Visual Description
\n
## Stacked Bar Chart: Rating Distribution by Model Evaluator
### Overview
The image presents a stacked bar chart visualizing the rating distribution for various language models, as evaluated by "Llama-3.3-70B-Instruct". The chart displays the proportion of each rating (0 to 4) assigned to each model. The x-axis lists the models, and the y-axis represents the proportion of ratings.
### Components/Axes
* **Title:** "Rating Distribution by Model Evaluator: Llama-3.3-70B-Instruct" (Top-center)
* **X-axis Label:** Model Name (Bottom-center)
* **Y-axis Label:** Proportion (Left-center)
* **Y-axis Scale:** 0.0 to 1.0, with increments of 0.2.
* **Legend:** Located in the top-right corner, mapping colors to rating values:
* 0: Blue
* 1: Light Blue
* 2: Pink
* 3: Orange
* 4: Red
* **Models (X-axis):**
* Llama-3.1-70B-Instruct
* Llama-3.1-8B-Instruct
* Llama-3.2-1B-Instruct
* Llama-3.2-3B-Instruct
* Mistral-7B-Instruct
* Mistral-8x22B-Instruct-v0.1
* Mixtral-8x7B-Instruct-v0.1
* Qwen2.5-0.5B-Instruct
* Qwen2.5-32B-Instruct
* Qwen2.5-5B-Instruct
* Qwen2.5-7B-Instruct
* Qwen2.5-72B-Instruct
* claude-3-haiku-20240307
* claude-3-sonnet-2024-05-13
* gpt-4o-mini-2024-07-18
### Detailed Analysis
The chart consists of stacked bars, each representing a model. The height of each segment within a bar indicates the proportion of ratings for that specific value.
* **Llama-3.1-70B-Instruct:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.1 proportion of rating 2, 0.3 proportion of rating 3, and 0.45 proportion of rating 4.
* **Llama-3.1-8B-Instruct:** Approximately 0.1 proportion of rating 0, 0.1 proportion of rating 1, 0.1 proportion of rating 2, 0.3 proportion of rating 3, and 0.4 proportion of rating 4.
* **Llama-3.2-1B-Instruct:** Approximately 0.2 proportion of rating 0, 0.1 proportion of rating 1, 0.1 proportion of rating 2, 0.3 proportion of rating 3, and 0.3 proportion of rating 4.
* **Llama-3.2-3B-Instruct:** Approximately 0.1 proportion of rating 0, 0.1 proportion of rating 1, 0.1 proportion of rating 2, 0.3 proportion of rating 3, and 0.4 proportion of rating 4.
* **Mistral-7B-Instruct:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.1 proportion of rating 2, 0.3 proportion of rating 3, and 0.45 proportion of rating 4.
* **Mistral-8x22B-Instruct-v0.1:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.1 proportion of rating 2, 0.3 proportion of rating 3, and 0.45 proportion of rating 4.
* **Mixtral-8x7B-Instruct-v0.1:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.1 proportion of rating 2, 0.3 proportion of rating 3, and 0.45 proportion of rating 4.
* **Qwen2.5-0.5B-Instruct:** Approximately 0.2 proportion of rating 0, 0.1 proportion of rating 1, 0.1 proportion of rating 2, 0.3 proportion of rating 3, and 0.3 proportion of rating 4.
* **Qwen2.5-32B-Instruct:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.1 proportion of rating 2, 0.3 proportion of rating 3, and 0.45 proportion of rating 4.
* **Qwen2.5-5B-Instruct:** Approximately 0.1 proportion of rating 0, 0.1 proportion of rating 1, 0.1 proportion of rating 2, 0.3 proportion of rating 3, and 0.4 proportion of rating 4.
* **Qwen2.5-7B-Instruct:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.1 proportion of rating 2, 0.3 proportion of rating 3, and 0.45 proportion of rating 4.
* **Qwen2.5-72B-Instruct:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.1 proportion of rating 2, 0.3 proportion of rating 3, and 0.45 proportion of rating 4.
* **claude-3-haiku-20240307:** Approximately 0.1 proportion of rating 0, 0.1 proportion of rating 1, 0.1 proportion of rating 2, 0.3 proportion of rating 3, and 0.4 proportion of rating 4.
* **claude-3-sonnet-2024-05-13:** Approximately 0.05 proportion of rating 0, 0.1 proportion of rating 1, 0.1 proportion of rating 2, 0.3 proportion of rating 3, and 0.45 proportion of rating 4.
* **gpt-4o-mini-2024-07-18:** Approximately 0.1 proportion of rating 0, 0.1 proportion of rating 1, 0.1 proportion of rating 2, 0.3 proportion of rating 3, and 0.4 proportion of rating 4.
### Key Observations
* Most models receive a high proportion of rating 4, indicating generally positive evaluations.
* The proportion of rating 0 is relatively low across all models.
* There is a slight variation in the distribution of ratings among the models, but the overall pattern is consistent.
* Models like Llama-3.1-70B-Instruct, Mistral-7B-Instruct, Mistral-8x22B-Instruct-v0.1, Mixtral-8x7B-Instruct-v0.1, Qwen2.5-32B-Instruct, Qwen2.5-72B-Instruct, claude-3-sonnet-2024-05-13 appear to have slightly higher proportions of rating 4.
### Interpretation
The chart demonstrates that, according to the Llama-3.3-70B-Instruct evaluator, the majority of the assessed language models perform well, receiving predominantly high ratings (3 and 4). The consistent pattern across models suggests that the evaluator has a relatively high baseline expectation or that the models generally meet a certain performance standard. The small variations in rating distributions could indicate subtle differences in model capabilities or biases in the evaluation process. The relatively low proportion of rating 0 suggests that none of the models are considered entirely unsatisfactory by this evaluator. The data suggests a generally positive landscape for these language models, with most exhibiting strong performance characteristics. Further investigation could explore the specific criteria used by the Llama-3.3-70B-Instruct evaluator to understand the nuances behind the ratings.
</details>
Figure 21: Rating distribution on RuozhiBench-Gen using Llama-3.3-70B-Instruction as an evaluator.