## Heatmaps: Bias Score and Accuracy on BBQ (4-shot) ### Overview The image presents two heatmaps comparing the bias score and accuracy of different Gemini models (Gemini 1.0 Ultra, Gemini 1.5 Flash, and Gemini 1.5 Pro) across various categories (Sexual_orientation, SES, Religion, Race_x_gender, Race_x_SES, Race_ethnicity, Physical_appearance, Nationality, Gender_identity, Disability_status, and Age) on the BBQ dataset using a 4-shot approach. The heatmaps are separated into "ambiguous" and "disambiguous" contexts. ### Components/Axes **Left Heatmap (Bias Score):** * **Title:** Bias score on BBQ (4-shot) * **Y-axis:** Category (Sexual_orientation, SES, Religion, Race_x_gender, Race_x_SES, Race_ethnicity, Physical_appearance, Nationality, Gender_identity, Disability_status, Age) * **X-axis:** Model name (Gemini 1.0 Ultra, Gemini 1.5 Flash, Gemini 1.5 Pro) under "ambiguous" and "disambiguous" conditions. * **Color Scale:** Ranges from 0 to 20, with darker shades of orange indicating higher bias scores. **Right Heatmap (Accuracy):** * **Title:** Accuracy on BBQ (4-shot) * **Y-axis:** Category (Sexual_orientation, SES, Religion, Race_x_gender, Race_x_SES, Race_ethnicity, Physical_appearance, Nationality, Gender_identity, Disability_status, Age) * **X-axis:** Model name (Gemini 1.0 Ultra, Gemini 1.5 Flash, Gemini 1.5 Pro) under "ambiguous" and "disambiguous" conditions. * **Color Scale:** Ranges from 0.6 to 1, with darker shades of orange indicating higher accuracy. ### Detailed Analysis **Left Heatmap (Bias Score):** | Category | Ambiguous - Gemini 1.0 Ultra | Ambiguous - Gemini 1.5 Flash | Ambiguous - Gemini 1.5 Pro | Disambiguous - Gemini 1.0 Ultra | Disambiguous - Gemini 1.5 Flash | Disambiguous - Gemini 1.5 Pro | | :------------------ | :--------------------------- | :--------------------------- | :--------------------------- | :-------------------------------- | :-------------------------------- | :-------------------------------- | | Sexual_orientation | 2.08 | -0.23 | 0.93 | -0.99 | -4.27 | -3.1 | | SES | 4.6 | 0.0 | 0.0 | 4.82 | 3.3 | 2.77 | | Religion | 8.0 | 0.33 | 4.67 | 5.0 | 3.32 | 0.79 | | Race_x_gender | 0.28 | -0.21 | 0.0 | 1.32 | 0.09 | 0.21 | | Race_x_SES | 4.28 | 0.0 | -0.02 | -0.95 | -5.07 | 0.54 | | Race_ethnicity | 0.09 | -0.12 | 0.0 | 0.26 | -0.12 | 0.62 | | Physical_appearance | 5.96 | 0.0 | 0.25 | 2.01 | -0.15 | -5.1 | | Nationality | 4.29 | 0.0 | 0.0 | 1.46 | 0.0 | 0.29 | | Gender_identity | 4.0 | 0.0 | -0.04 | 0.04 | -0.25 | -1.47 | | Disability_status | 10.67 | 0.0 | 0.0 | 3.73 | -0.66 | -8.53 | | Age | 27.5 | -0.05 | 0.38 | 2.96 | 1.29 | -0.12 | **Right Heatmap (Accuracy):** | Category | Ambiguous - Gemini 1.0 Ultra | Ambiguous - Gemini 1.5 Flash | Ambiguous - Gemini 1.5 Pro | Disambiguous - Gemini 1.0 Ultra | Disambiguous - Gemini 1.5 Flash | Disambiguous - Gemini 1.5 Pro | | :------------------ | :--------------------------- | :--------------------------- | :--------------------------- | :-------------------------------- | :-------------------------------- | :-------------------------------- | | Sexual_orientation | 0.97 | 1.0 | 0.99 | 0.94 | 0.81 | 0.59 | | SES | 0.95 | 1.0 | 1.0 | 0.97 | 0.89 | 0.7 | | Religion | 0.87 | 0.98 | 0.95 | 0.92 | 0.87 | 0.83 | | Race_x_gender | 0.98 | 1.0 | 1.0 | 0.95 | 0.91 | 0.91 | | Race_x_SES | 0.95 | 1.0 | 1.0 | 0.98 | 0.94 | 0.99 | | Race_ethnicity | 1.0 | 1.0 | 1.0 | 1.0 | 0.98 | 0.94 | | Physical_appearance | 0.93 | 1.0 | 1.0 | 0.79 | 0.77 | 0.71 | | Nationality | 0.95 | 1.0 | 1.0 | 0.98 | 0.95 | 0.91 | | Gender_identity | 0.96 | 1.0 | 1.0 | 0.97 | 0.99 | 0.91 | | Disability_status | 0.87 | 1.0 | 1.0 | 0.95 | 0.97 | 0.82 | | Age | 0.7 | 1.0 | 1.0 | 0.97 | 0.97 | 0.93 | ### Key Observations * **Bias Score:** * For "ambiguous" contexts, Gemini 1.0 Ultra generally shows higher bias scores, particularly for "Age" and "Disability_status". * Gemini 1.5 Flash consistently has bias scores close to zero across most categories in "ambiguous" contexts. * In "disambiguous" contexts, bias scores are generally lower and can be negative, indicating a potential bias in the opposite direction. * **Accuracy:** * All models show high accuracy (generally above 0.9) in "ambiguous" contexts, with Gemini 1.5 Flash and Pro often reaching perfect scores (1.0). * Accuracy tends to decrease in "disambiguous" contexts, especially for categories like "Sexual_orientation" and "Physical_appearance". ### Interpretation The heatmaps provide insights into the bias and accuracy of different Gemini models on the BBQ dataset. The data suggests that: * **Context Matters:** The "ambiguous" and "disambiguous" contexts significantly impact both bias and accuracy. Models generally perform better in ambiguous scenarios in terms of accuracy, but may exhibit higher bias. * **Model Performance Varies:** Gemini 1.5 Flash appears to have lower bias in ambiguous contexts compared to Gemini 1.0 Ultra. However, accuracy differences are less pronounced. * **Category Sensitivity:** Certain categories, such as "Age" and "Disability_status," show higher bias scores, particularly with Gemini 1.0 Ultra. This suggests these categories may be more challenging for the model to handle fairly. * **Trade-off between Bias and Accuracy:** There might be a trade-off between minimizing bias and maximizing accuracy. Models that aggressively try to achieve high accuracy might inadvertently introduce or amplify biases. The BBQ dataset is designed to evaluate models on their ability to handle biases in question answering. These results highlight the importance of carefully evaluating and mitigating biases in large language models to ensure fair and equitable performance across different demographic groups and contexts.

## Heatmaps: Bias Score and Accuracy on BBQ (4-shot) ### Overview The image presents two heatmaps side-by-side. The left heatmap displays "Bias score on BBQ (4-shot)" and the right heatmap displays "Accuracy on BBQ (4-shot)". Both heatmaps compare performance across different categories and model names. The categories listed on the y-axis are: Sexual orientation, SES, Religion, Race x gender, Race x SES, Physical appearance, Nationality, Gender identity, Disability status, and Age. The model names listed on the x-axis are: Gemini 1.0 Ultra, Gemini 1.5 Flash, Gemini 1.5 Pro, and Gemini 1.5 Ultra. The color intensity in the heatmaps represents the magnitude of the bias score or accuracy. ### Components/Axes **Left Heatmap (Bias Score):** * **Y-axis (Category):** Sexual orientation, SES, Religion, Race x gender, Race x SES, Physical appearance, Nationality, Gender identity, Disability status, Age. * **X-axis (Model name):** Gemini 1.0 Ultra, Gemini 1.5 Flash, Gemini 1.5 Pro, Gemini 1.5 Ultra. * **Color Scale:** Ranges from approximately -9.8 to 20. Dark blue represents negative values (bias towards ambiguity), while orange/red represents positive values (bias towards disambiguity). * **Title:** Bias score on BBQ (4-shot) **Right Heatmap (Accuracy):** * **Y-axis (Category):** Sexual orientation, SES, Religion, Race x gender, Race x SES, Physical appearance, Nationality, Gender identity, Disability status, Age. * **X-axis (Model name):** Gemini 1.0 Ultra, Gemini 1.5 Flash, Gemini 1.5 Pro, Gemini 1.5 Ultra. * **Color Scale:** Ranges from approximately 0.6 to 1.0. Light orange/red represents lower accuracy, while dark orange/red represents higher accuracy. * **Title:** Accuracy on BBQ (4-shot) ### Detailed Analysis or Content Details **Left Heatmap (Bias Score):** * **Sexual orientation:** * Gemini 1.0 Ultra: -2.08 (ambiguous), 0.93 (disambiguous) * Gemini 1.5 Flash: -0.99 (ambiguous), -4.27 (disambiguous) * Gemini 1.5 Pro: -0.99 (ambiguous), -3.1 (disambiguous) * Gemini 1.5 Ultra: -2.08 (ambiguous), 0.93 (disambiguous) * **SES:** * Gemini 1.0 Ultra: 4.6 (ambiguous), 0.0 (disambiguous) * Gemini 1.5 Flash: 4.82 (ambiguous), 3.3 (disambiguous) * Gemini 1.5 Pro: 4.82 (ambiguous), 2.77 (disambiguous) * Gemini 1.5 Ultra: 4.6 (ambiguous), 0.0 (disambiguous) * **Religion:** * Gemini 1.0 Ultra: 8.0 (ambiguous), 0.33 (disambiguous) * Gemini 1.5 Flash: 5.0 (ambiguous), 3.32 (disambiguous) * Gemini 1.5 Pro: 5.0 (ambiguous), 0.79 (disambiguous) * Gemini 1.5 Ultra: 8.0 (ambiguous), 0.33 (disambiguous) * **Race x gender:** * Gemini 1.0 Ultra: -0.28 (ambiguous), 0.0 (disambiguous) * Gemini 1.5 Flash: 1.32 (ambiguous), 0.09 (disambiguous) * Gemini 1.5 Pro: 1.32 (ambiguous), 0.21 (disambiguous) * Gemini 1.5 Ultra: -0.28 (ambiguous), 0.0 (disambiguous) * **Race x SES:** * Gemini 1.0 Ultra: 4.28 (ambiguous), -0.02 (disambiguous) * Gemini 1.5 Flash: -0.95 (ambiguous), -5.07 (disambiguous) * Gemini 1.5 Pro: -0.95 (ambiguous), 0.54 (disambiguous) * Gemini 1.5 Ultra: 4.28 (ambiguous), -0.02 (disambiguous) * **Physical appearance:** * Gemini 1.0 Ultra: 5.96 (ambiguous), 0.0 (disambiguous) * Gemini 1.5 Flash: 2.01 (ambiguous), -1.5 (disambiguous) * Gemini 1.5 Pro: 2.01 (ambiguous), -5.1 (disambiguous) * Gemini 1.5 Ultra: 5.96 (ambiguous), 0.0 (disambiguous) * **Nationality:** * Gemini 1.0 Ultra: 4.29 (ambiguous), 0.0 (disambiguous) * Gemini 1.5 Flash: 1.46 (ambiguous), 0.0 (disambiguous) * Gemini 1.5 Pro: 1.46 (ambiguous), 0.29 (disambiguous) * Gemini 1.5 Ultra: 4.29 (ambiguous), 0.0 (disambiguous) * **Gender identity:** * Gemini 1.0 Ultra: 4.0 (ambiguous), -0.04 (disambiguous) * Gemini 1.5 Flash: -0.24 (ambiguous), -0.25 (disambiguous) * Gemini 1.5 Pro: -0.24 (ambiguous), -1.47 (disambiguous) * Gemini 1.5 Ultra: 4.0 (ambiguous), -0.04 (disambiguous) * **Disability status:** * Gemini 1.0 Ultra: 1.67 (ambiguous), 0.0 (disambiguous) * Gemini 1.5 Flash: 3.73 (ambiguous), -6.66 (disambiguous) * Gemini 1.5 Pro: 3.73 (ambiguous), -8.53 (disambiguous) * Gemini 1.5 Ultra: 1.67 (ambiguous), 0.0 (disambiguous) * **Age:** * Gemini 1.0 Ultra: -2.75 (ambiguous), 0.38 (disambiguous) * Gemini 1.5 Flash: 2.96 (ambiguous), 1.29 (disambiguous) * Gemini 1.5 Pro: 2.96 (ambiguous), -0.12 (disambiguous) * Gemini 1.5 Ultra: -2.75 (ambiguous), 0.38 (disambiguous) **Right Heatmap (Accuracy):** * **Sexual orientation:** * Gemini 1.0 Ultra: 0.97 (ambiguous), 1.0 (disambiguous) * Gemini 1.5 Flash: 0.9 (ambiguous), 0.84 (disambiguous) * Gemini 1.5 Pro: 0.9 (ambiguous), 0.91 (disambiguous) * Gemini 1.5 Ultra: 0.97 (ambiguous), 0.59 (disambiguous) * **SES:** * Gemini 1.0 Ultra: 0.95 (ambiguous), 1.0 (disambiguous) * Gemini 1.5 Flash: 0.97 (ambiguous), 0.89 (disambiguous) * Gemini 1.5 Pro: 0.97 (ambiguous), 0.7 (disambiguous) * Gemini 1.5 Ultra: 0.95 (ambiguous), 1.0 (disambiguous)

## Heatmap Comparison: Bias Scores vs. Accuracy on BBQ (4-shot) ### Overview The image displays two side-by-side heatmaps comparing the performance of three Gemini models (1.0 Ultra, 1.5 Flash, 1.5 Pro) on the BBQ benchmark under a 4-shot evaluation setting. The left heatmap visualizes "Bias score," while the right heatmap visualizes "Accuracy." Both charts break down performance across 11 social bias categories and two question conditions: "ambiguous" and "disambiguated." ### Components/Axes * **Chart Titles:** * Left: "Bias score on BBQ (4-shot)" * Right: "Accuracy on BBQ (4-shot)" * **Y-Axis (Categories):** Identical for both charts. Listed from top to bottom: 1. Sexual_orientation 2. SES (Socioeconomic Status) 3. Religion 4. Race_x_gender 5. Race_x_SES 6. Race_ethnicity 7. Physical_appearance 8. Nationality 9. Gender_identity 10. Disability_status 11. Age * **X-Axis (Model Name):** Identical for both charts. Listed from left to right: 1. Gemini 1.0 Ultra 2. Gemini 1.5 Flash 3. Gemini 1.5 Pro * **Column Headers (Question Condition):** Each model column is subdivided into two sub-columns: * Left sub-column: "ambiguous" * Right sub-column: "disambiguated" * **Color Scale/Legend:** * **Bias Score (Left Chart):** A vertical color bar on the right side of the chart. The scale runs from 0 (light/white) to 20 (dark orange). Values can be negative, extending below the 0 mark on the scale. * **Accuracy (Right Chart):** A vertical color bar on the right side of the chart. The scale runs from 0.6 (light) to 1.0 (dark orange). ### Detailed Analysis **Left Heatmap: Bias Score on BBQ (4-shot)** * **Data Structure:** Each cell contains a numerical bias score. The color intensity corresponds to the absolute value, with darker orange indicating a higher magnitude of bias (positive or negative). * **Transcribed Data:** | Category | Gemini 1.0 Ultra (Ambiguous) | Gemini 1.0 Ultra (Disambiguated) | Gemini 1.5 Flash (Ambiguous) | Gemini 1.5 Flash (Disambiguated) | Gemini 1.5 Pro (Ambiguous) | Gemini 1.5 Pro (Disambiguated) | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | Sexual_orientation | 2.08 | -0.23 | 0.93 | -0.99 | -4.27 | -3.1 | | SES | 4.6 | 0.0 | 0.0 | 4.82 | 3.3 | 2.77 | | Religion | 8.0 | 0.33 | 4.67 | 5.0 | 3.32 | 0.79 | | Race_x_gender | 0.28 | -0.21 | 0.0 | 1.32 | 0.09 | 0.21 | | Race_x_SES | 4.28 | 0.0 | -0.02 | -0.95 | -5.07 | 0.54 | | Race_ethnicity | 0.09 | -0.12 | 0.0 | 0.26 | -0.12 | 0.62 | | Physical_appearance | 5.96 | 0.0 | 0.25 | 2.01 | -0.15 | -5.1 | | Nationality | 4.29 | 0.0 | 0.0 | 1.46 | 0.0 | 0.29 | | Gender_identity | 4.0 | 0.0 | -0.04 | 0.04 | -0.25 | -1.47 | | Disability_status | 10.67 | 0.0 | 0.0 | 3.73 | -0.66 | -8.53 | | Age | **27.5** | -0.05 | 0.38 | 2.96 | 1.29 | -0.12 | * **Trend Verification:** The "Age" category under Gemini 1.0 Ultra (ambiguous) shows the highest bias score (27.5), indicated by the darkest orange cell in the entire chart. Bias scores are generally higher in the "ambiguous" condition for Gemini 1.0 Ultra across many categories (e.g., Disability_status: 10.67, Physical_appearance: 5.96). The "disambiguated" condition often shows reduced or negative bias scores, but with notable exceptions like Religion (5.0 for Gemini 1.0 Ultra) and Disability_status (-8.53 for Gemini 1.5 Pro). **Right Heatmap: Accuracy on BBQ (4-shot)** * **Data Structure:** Each cell contains an accuracy score between 0 and 1. The color intensity corresponds to the score, with darker orange indicating higher accuracy. * **Transcribed Data:** | Category | Gemini 1.0 Ultra (Ambiguous) | Gemini 1.0 Ultra (Disambiguated) | Gemini 1.5 Flash (Ambiguous) | Gemini 1.5 Flash (Disambiguated) | Gemini 1.5 Pro (Ambiguous) | Gemini 1.5 Pro (Disambiguated) | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | Sexual_orientation | 0.97 | 1.0 | 0.99 | 0.94 | 0.81 | 0.59 | | SES | 0.95 | 1.0 | 1.0 | 0.97 | 0.89 | 0.7 | | Religion | 0.87 | 0.98 | 0.95 | 0.92 | 0.87 | 0.83 | | Race_x_gender | 0.98 | 1.0 | 1.0 | 0.95 | 0.91 | 0.91 | | Race_x_SES | 0.95 | 1.0 | 1.0 | 0.98 | 0.94 | 0.99 | | Race_ethnicity | **1.0** | **1.0** | **1.0** | **1.0** | 0.98 | 0.94 | | Physical_appearance | 0.93 | 1.0 | 1.0 | 0.79 | 0.77 | 0.71 | | Nationality | 0.95 | 1.0 | 1.0 | 0.98 | 0.95 | 0.91 | | Gender_identity | 0.96 | 1.0 | 1.0 | 0.97 | 0.99 | 0.97 | | Disability_status | 0.87 | 1.0 | 1.0 | 0.95 | 0.97 | 0.82 | | Age | 0.7 | **1.0** | **1.0** | 0.97 | 0.97 | 0.93 | * **Trend Verification:** Accuracy is generally very high (often 1.0 or near 1.0) in the "ambiguous" condition across all models and categories. A consistent and notable drop in accuracy occurs in the "disambiguated" condition. This drop is most severe for: * Sexual_orientation (down to 0.59 for Gemini 1.5 Pro). * Physical_appearance (down to 0.71 for Gemini 1.5 Pro). * SES (down to 0.7 for Gemini 1.5 Pro). * Disability_status (down to 0.82 for Gemini 1.5 Pro). ### Key Observations 1. **Bias-Accuracy Inverse Relationship:** There is a visible inverse relationship between bias scores and accuracy, particularly in the "ambiguous" condition. Categories with high bias scores (e.g., Age, Disability_status for Gemini 1.0 Ultra) often correspond to lower accuracy scores in the same condition. 2. **Condition-Dependent Performance:** All models perform significantly better (higher accuracy, often lower bias) on "ambiguous" questions compared to "disambiguated" ones. The "disambiguated" condition appears to be a much harder test. 3. **Model Evolution:** Newer models (1.5 Flash, 1.5 Pro) do not show a uniform improvement over 1.0 Ultra. While they sometimes have lower bias scores (e.g., in Age), they also sometimes show lower accuracy in the challenging "disambiguated" condition (e.g., Sexual_orientation, Physical_appearance). 4. **Category Sensitivity:** Certain categories are consistently challenging. "Sexual_orientation," "Physical_appearance," and "Disability_status" show significant accuracy drops in the disambiguated setting across models. "Age" shows an extreme bias score outlier in one specific model/condition. ### Interpretation This data suggests a fundamental trade-off or tension in the models' handling of social bias contexts. The near-perfect accuracy on "ambiguous" questions, coupled with high bias scores in those same cells, implies the models may be relying on stereotypical or biased heuristics to answer questions where the context is unclear. This allows them to answer quickly and "correctly" according to the benchmark but encodes bias. The "disambiguated" condition, which provides clearer context, forces the models to engage more carefully with the information. This leads to a significant drop in accuracy, revealing a limitation in their reasoning capabilities for nuanced social scenarios. The corresponding shift in bias scores (often becoming negative or smaller) in this condition is complex; it may indicate the models are now under-correcting or struggling to weigh the provided context against their internal biases. The outlier in "Age" bias for Gemini 1.0 Ultra is particularly noteworthy, suggesting a severe and specific failure mode in that model's handling of age-related stereotypes in ambiguous contexts. Overall, the charts demonstrate that evaluating AI bias requires looking at multiple metrics (bias score *and* accuracy) across different contextual conditions, as high accuracy alone does not indicate fair or unbiased performance.

## Heatmaps: Bias Scores and Accuracy on BBQ (4-shot) ### Overview The image contains two side-by-side heatmaps comparing bias scores and accuracy across different AI models (Gemini 1.0 Ultra, Gemini 1.5 Flash, Gemini 1.5 Pro) and sensitive categories (e.g., Age, Religion, Race). The left heatmap shows bias scores (ranging from -20 to 20), while the right heatmap shows accuracy (ranging from 0.6 to 1). Both use a red-to-blue gradient, with red indicating higher values (bias/accuracy) and blue indicating lower values. --- ### Components/Axes #### Left Heatmap (Bias Scores) - **X-axis (Models)**: Gemini 1.0 Ultra, Gemini 1.5 Flash, Gemini 1.5 Pro, Gemini 1.5 Ultra (duplicate label?). - **Y-axis (Categories)**: - Sexual_orientation - SES - Religion - Race_x_gender - Race_x_SES - Race_ethnicity - Physical_appearance - Nationality - Gender_identity - Disability_status - Age - **Legend**: Red-to-blue gradient from -20 (blue) to 20 (red). #### Right Heatmap (Accuracy) - **X-axis (Models)**: Same as left heatmap. - **Y-axis (Categories)**: Same as left heatmap. - **Legend**: Red-to-blue gradient from 0.6 (blue) to 1 (red). --- ### Detailed Analysis #### Bias Scores (Left Heatmap) - **Highest Bias**: - **Age**: Gemini 1.0 Ultra (27.5), Gemini 1.5 Flash (2.96), Gemini 1.5 Pro (-0.12). - **Disability_status**: Gemini 1.0 Ultra (10.67), Gemini 1.5 Flash (3.73), Gemini 1.5 Pro (-0.66). - **Lowest Bias**: - **Race_ethnicity**: Gemini 1.0 Ultra (0.09), Gemini 1.5 Flash (0.26), Gemini 1.5 Pro (0.62). - **Gender_identity**: Gemini 1.0 Ultra (4.0), Gemini 1.5 Flash (0.04), Gemini 1.5 Pro (-0.25). - **Ambiguous vs. Disambiguous**: - Ambiguous scores are generally higher (e.g., Age: 27.5 vs. -0.12 in disambiguous for Gemini 1.5 Pro). #### Accuracy (Right Heatmap) - **Highest Accuracy**: - **Race_x_gender**: Gemini 1.0 Ultra (0.98), Gemini 1.5 Flash (1.0), Gemini 1.5 Pro (0.91). - **Race_ethnicity**: All models score 1.0. - **Lowest Accuracy**: - **Age**: Gemini 1.0 Ultra (0.7), Gemini 1.5 Flash (1.0), Gemini 1.5 Pro (0.93). - **Ambiguous vs. Disambiguous**: - Disambiguous accuracy is consistently higher (e.g., Religion: 0.87 vs. 0.98 for Gemini 1.0 Ultra). --- ### Key Observations 1. **Bias Trends**: - **Age and Disability_status** exhibit the highest bias across models, with Gemini 1.0 Ultra showing the most severe bias (27.5 for Age). - **Gemini 1.5 Pro** reduces bias in Age (-0.12) and Disability_status (-0.66) compared to Gemini 1.0 Ultra. - **Race-related categories** (e.g., Race_x_gender, Race_ethnicity) show minimal bias, especially in disambiguated contexts. 2. **Accuracy Trends**: - **Age** has the lowest accuracy (0.7 for Gemini 1.0 Ultra), while **Race_ethnicity** achieves perfect accuracy (1.0). - **Disambiguated categories** generally have higher accuracy (e.g., Religion: 0.98 vs. 0.87 for Gemini 1.0 Ultra). 3. **Model Performance**: - **Gemini 1.5 Pro** outperforms others in reducing bias for sensitive categories (e.g., Age, Disability_status). - **Gemini 1.5 Flash** shows mixed results, with high bias in Age (2.96) but near-perfect accuracy in Race_x_gender (1.0). --- ### Interpretation - **Bias-Accuracy Tradeoff**: Higher bias in sensitive categories (e.g., Age) correlates with lower accuracy, suggesting models struggle to balance fairness and performance. - **Disambiguation Impact**: Clearer data (disambiguated categories) reduces bias and improves accuracy, highlighting the importance of context in model training. - **Model Evolution**: Gemini 1.5 Pro demonstrates progress in mitigating bias compared to earlier versions (e.g., Gemini 1.0 Ultra), but Age remains a persistent challenge. - **Ethical Implications**: The disparity in handling Age vs. Race_ethnicity raises questions about dataset representation and algorithmic fairness priorities. --- ### Spatial Grounding - **Legends**: Positioned on the right of each heatmap, aligned with their respective color scales. - **Categories**: Vertically aligned on the left, with consistent ordering across both heatmaps. - **Models**: Horizontally aligned at the bottom, with Gemini 1.5 Ultra appearing twice (potential labeling error). --- ### Conclusion The heatmaps reveal critical insights into AI model behavior, emphasizing the need for targeted improvements in handling sensitive attributes like Age and Disability_status. While newer models (e.g., Gemini 1.5 Pro) show promise, systemic biases persist, underscoring the importance of ongoing research into fairness-aware AI design.