## Bar Chart: Model Performance Across Datasets and Temperatures ### Overview The image presents a series of bar charts comparing the performance of different language models (Mistral-Small-24B, Llama3.1-8B, Phi3.5, Mistral-Nemo, Llama3.2-3B) on various question-answering datasets (GSM8K, TruthfulQA, CoQA, SQuADv2, TriviaQA, HaluevalQA, NQOpen). The charts show the counts of "Hallucination", "Non-Hallucination", and "Rejected" responses at two temperature settings (0.1 and 1.0). ### Components/Axes * **Y-axis (Count):** Represents the number of responses, ranging from 0 to a maximum value that varies by dataset (e.g., 1200 for GSM8K, 6000 for CoQA). * **X-axis (Temperature):** Categorical axis with two values: 0.1 and 1.0. * **Datasets (Rows):** GSM8K, TruthfulQA, CoQA, SQuADv2, TriviaQA, HaluevalQA, NQOpen. * **Models (Columns):** Mistral-Small-24B, Llama3.1-8B, Phi3.5, Mistral-Nemo, Llama3.2-3B. * **Legend (Bottom):** * Red: Hallucination * Green: Non-Hallucination * Gray: Rejected ### Detailed Analysis Each cell in the grid represents a specific model-dataset combination. Within each cell, there are three bars for each temperature setting (0.1 and 1.0), corresponding to "Hallucination", "Non-Hallucination", and "Rejected" counts. **GSM8K:** * **Mistral-Small-24B:** At temperature 0.1, Non-Hallucination is approximately 1100, Hallucination is approximately 100, and Rejected is approximately 100. At temperature 1.0, Non-Hallucination is approximately 1000, Hallucination is approximately 100, and Rejected is approximately 100. * **Llama3.1-8B:** At temperature 0.1, Non-Hallucination is approximately 1000, Hallucination is approximately 100, and Rejected is approximately 100. At temperature 1.0, Non-Hallucination is approximately 1000, Hallucination is approximately 100, and Rejected is approximately 100. * **Phi3.5:** At temperature 0.1, Hallucination is approximately 200, Non-Hallucination is approximately 800, and Rejected is approximately 100. At temperature 1.0, Hallucination is approximately 800, Non-Hallucination is approximately 200, and Rejected is approximately 100. * **Mistral-Nemo:** At temperature 0.1, Non-Hallucination is approximately 900, Hallucination is approximately 100, and Rejected is approximately 100. At temperature 1.0, Hallucination is approximately 900, Non-Hallucination is approximately 100, and Rejected is approximately 100. * **Llama3.2-3B:** At temperature 0.1, Non-Hallucination is approximately 900, Hallucination is approximately 200, and Rejected is approximately 100. At temperature 1.0, Non-Hallucination is approximately 900, Hallucination is approximately 200, and Rejected is approximately 100. **TruthfulQA:** * **Mistral-Small-24B:** At temperature 0.1, Non-Hallucination is approximately 150, Hallucination is approximately 100, and Rejected is approximately 350. At temperature 1.0, Non-Hallucination is approximately 200, Hallucination is approximately 100, and Rejected is approximately 300. * **Llama3.1-8B:** At temperature 0.1, Non-Hallucination is approximately 400, Hallucination is approximately 100, and Rejected is approximately 100. At temperature 1.0, Non-Hallucination is approximately 300, Hallucination is approximately 200, and Rejected is approximately 100. * **Phi3.5:** At temperature 0.1, Hallucination is approximately 500, Non-Hallucination is approximately 100, and Rejected is approximately 0. At temperature 1.0, Hallucination is approximately 500, Non-Hallucination is approximately 100, and Rejected is approximately 0. * **Mistral-Nemo:** At temperature 0.1, Hallucination is approximately 500, Non-Hallucination is approximately 100, and Rejected is approximately 0. At temperature 1.0, Hallucination is approximately 500, Non-Hallucination is approximately 100, and Rejected is approximately 0. * **Llama3.2-3B:** At temperature 0.1, Hallucination is approximately 500, Non-Hallucination is approximately 100, and Rejected is approximately 0. At temperature 1.0, Hallucination is approximately 500, Non-Hallucination is approximately 100, and Rejected is approximately 0. **CoQA:** * **Mistral-Small-24B:** At temperature 0.1, Non-Hallucination is approximately 5500, Hallucination is approximately 2000, and Rejected is approximately 100. At temperature 1.0, Non-Hallucination is approximately 2000, Hallucination is approximately 5000, and Rejected is approximately 100. * **Llama3.1-8B:** At temperature 0.1, Non-Hallucination is approximately 5000, Hallucination is approximately 2000, and Rejected is approximately 100. At temperature 1.0, Non-Hallucination is approximately 5000, Hallucination is approximately 2000, and Rejected is approximately 100. * **Phi3.5:** At temperature 0.1, Non-Hallucination is approximately 5000, Hallucination is approximately 2000, and Rejected is approximately 100. At temperature 1.0, Non-Hallucination is approximately 2000, Hallucination is approximately 5000, and Rejected is approximately 100. * **Mistral-Nemo:** At temperature 0.1, Non-Hallucination is approximately 5000, Hallucination is approximately 2000, and Rejected is approximately 100. At temperature 1.0, Non-Hallucination is approximately 2000, Hallucination is approximately 5000, and Rejected is approximately 100. * **Llama3.2-3B:** At temperature 0.1, Non-Hallucination is approximately 5000, Hallucination is approximately 2000, and Rejected is approximately 100. At temperature 1.0, Non-Hallucination is approximately 2000, Hallucination is approximately 5000, and Rejected is approximately 100. **SQuADv2:** * **Mistral-Small-24B:** At temperature 0.1, Non-Hallucination is approximately 1000, Hallucination is approximately 1000, and Rejected is approximately 2000. At temperature 1.0, Non-Hallucination is approximately 1000, Hallucination is approximately 1000, and Rejected is approximately 2000. * **Llama3.1-8B:** At temperature 0.1, Non-Hallucination is approximately 4000, Hallucination is approximately 500, and Rejected is approximately 100. At temperature 1.0, Non-Hallucination is approximately 4000, Hallucination is approximately 500, and Rejected is approximately 100. * **Phi3.5:** At temperature 0.1, Hallucination is approximately 4000, Non-Hallucination is approximately 500, and Rejected is approximately 100. At temperature 1.0, Hallucination is approximately 4000, Non-Hallucination is approximately 500, and Rejected is approximately 100. * **Mistral-Nemo:** At temperature 0.1, Hallucination is approximately 4000, Non-Hallucination is approximately 500, and Rejected is approximately 100. At temperature 1.0, Hallucination is approximately 4000, Non-Hallucination is approximately 500, and Rejected is approximately 100. * **Llama3.2-3B:** At temperature 0.1, Hallucination is approximately 4000, Non-Hallucination is approximately 500, and Rejected is approximately 100. At temperature 1.0, Hallucination is approximately 4000, Non-Hallucination is approximately 500, and Rejected is approximately 100. **TriviaQA:** * **Mistral-Small-24B:** At temperature 0.1, Non-Hallucination is approximately 3000, Hallucination is approximately 1000, and Rejected is approximately 2000. At temperature 1.0, Non-Hallucination is approximately 1000, Hallucination is approximately 4000, and Rejected is approximately 100. * **Llama3.1-8B:** At temperature 0.1, Non-Hallucination is approximately 6000, Hallucination is approximately 500, and Rejected is approximately 100. At temperature 1.0, Non-Hallucination is approximately 6000, Hallucination is approximately 500, and Rejected is approximately 100. * **Phi3.5:** At temperature 0.1, Hallucination is approximately 4500, Non-Hallucination is approximately 1000, and Rejected is approximately 100. At temperature 1.0, Hallucination is approximately 4500, Non-Hallucination is approximately 1000, and Rejected is approximately 100. * **Mistral-Nemo:** At temperature 0.1, Hallucination is approximately 4500, Non-Hallucination is approximately 1000, and Rejected is approximately 100. At temperature 1.0, Hallucination is approximately 4500, Non-Hallucination is approximately 1000, and Rejected is approximately 100. * **Llama3.2-3B:** At temperature 0.1, Hallucination is approximately 4500, Non-Hallucination is approximately 1000, and Rejected is approximately 100. At temperature 1.0, Hallucination is approximately 4500, Non-Hallucination is approximately 1000, and Rejected is approximately 100. **HaluevalQA:** * **Mistral-Small-24B:** At temperature 0.1, Non-Hallucination is approximately 2500, Hallucination is approximately 1000, and Rejected is approximately 5000. At temperature 1.0, Non-Hallucination is approximately 2500, Hallucination is approximately 1000, and Rejected is approximately 5000. * **Llama3.1-8B:** At temperature 0.1, Non-Hallucination is approximately 4000, Hallucination is approximately 2000, and Rejected is approximately 2000. At temperature 1.0, Non-Hallucination is approximately 4000, Hallucination is approximately 2000, and Rejected is approximately 2000. * **Phi3.5:** At temperature 0.1, Hallucination is approximately 6000, Non-Hallucination is approximately 2000, and Rejected is approximately 100. At temperature 1.0, Hallucination is approximately 6000, Non-Hallucination is approximately 2000, and Rejected is approximately 100. * **Mistral-Nemo:** At temperature 0.1, Hallucination is approximately 6000, Non-Hallucination is approximately 2000, and Rejected is approximately 100. At temperature 1.0, Hallucination is approximately 6000, Non-Hallucination is approximately 2000, and Rejected is approximately 100. * **Llama3.2-3B:** At temperature 0.1, Hallucination is approximately 6000, Non-Hallucination is approximately 2000, and Rejected is approximately 100. At temperature 1.0, Hallucination is approximately 6000, Non-Hallucination is approximately 2000, and Rejected is approximately 100. **NQOpen:** * **Mistral-Small-24B:** At temperature 0.1, Non-Hallucination is approximately 1000, Hallucination is approximately 1000, and Rejected is approximately 2000. At temperature 1.0, Non-Hallucination is approximately 1000, Hallucination is approximately 1000, and Rejected is approximately 2000. * **Llama3.1-8B:** At temperature 0.1, Non-Hallucination is approximately 2000, Hallucination is approximately 1000, and Rejected is approximately 100. At temperature 1.0, Non-Hallucination is approximately 2000, Hallucination is approximately 1000, and Rejected is approximately 100. * **Phi3.5:** At temperature 0.1, Hallucination is approximately 2500, Non-Hallucination is approximately 500, and Rejected is approximately 100. At temperature 1.0, Hallucination is approximately 2500, Non-Hallucination is approximately 500, and Rejected is approximately 100. * **Mistral-Nemo:** At temperature 0.1, Hallucination is approximately 2500, Non-Hallucination is approximately 500, and Rejected is approximately 100. At temperature 1.0, Hallucination is approximately 2500, Non-Hallucination is approximately 500, and Rejected is approximately 100. * **Llama3.2-3B:** At temperature 0.1, Hallucination is approximately 2500, Non-Hallucination is approximately 500, and Rejected is approximately 100. At temperature 1.0, Hallucination is approximately 2500, Non-Hallucination is approximately 500, and Rejected is approximately 100. ### Key Observations * The performance of the models varies significantly across different datasets. * Some models exhibit a higher tendency to hallucinate on certain datasets. * The "Rejected" count is generally low, except for Mistral-Small-24B on TruthfulQA, SQuADv2, TriviaQA, HaluevalQA, and NQOpen. * Temperature seems to have a variable impact depending on the model and dataset. ### Interpretation The data suggests that the choice of language model and temperature setting can significantly impact the quality of responses on question-answering tasks. Some models are more prone to hallucination (generating incorrect or nonsensical answers) than others, and this tendency can be exacerbated by increasing the temperature. The high "Rejected" count for Mistral-Small-24B on certain datasets indicates that this model may be more likely to abstain from answering when uncertain, which could be a desirable trait in some applications. The varying performance across datasets highlights the importance of evaluating models on a diverse range of tasks to understand their strengths and weaknesses.