## Box Plot: Error (1-AUC) under Different Scenarios ### Overview The image presents six box plots arranged in a 2x3 grid. Each box plot visualizes the distribution of "Error (1-AUC)" under different scenarios: "Biased", "Direct-Effect", "Indirect-Effect", "Fair Observable", "Fair Unobservable", and "Fair Additive Noise". The x-axis is categorical, representing different methods or models, while the y-axis represents the error rate, ranging from 0 to 0.75. A legend at the bottom indicates the average rank (1-AUC) for each method. ### Components/Axes * **Y-axis:** "Error (1-AUC)", with ticks at 0, 0.25, 0.5, and 0.75. * **X-axis:** Categorical, representing different methods/models. The order of the models is consistent across all six subplots. * **Titles:** Each subplot has a title indicating the scenario: "1. Biased", "2. Direct-Effect", "3. Indirect-Effect", "4. Fair Observable", "5. Fair Unobservable", "6. Fair Additive Noise". * **Legend:** Located at the bottom of the image, labeled "Avg. Rank (1-AUC)". It maps colors to methods and their average ranks: * Blue: Unfair: 2.17 * Orange: Unaware: 2.62 * Pink: FairPFN: 3.51 * Olive Green: Cntf. Avg.: 3.62 * Brown: CFP: 4.28 * Purple: EGR: 5.18 * Red: Random: 6.67 * Green: Constant: 6.75 ### Detailed Analysis Each subplot contains box plots for the following methods (from left to right): Unfair, Unaware, FairPFN, Cntf. Avg., CFP, EGR, Random, and Constant. **1. Biased:** * Unfair: Median around 0.35, IQR between 0.25 and 0.45. * Unaware: Median around 0.4, IQR between 0.3 and 0.5. * FairPFN: Median around 0.4, IQR between 0.3 and 0.5. * Cntf. Avg.: Median around 0.4, IQR between 0.3 and 0.5. * CFP: Median around 0.45, IQR between 0.35 and 0.55. * EGR: Median around 0.4, IQR between 0.3 and 0.5. * Random: Median around 0.5, IQR between 0.4 and 0.6. * Constant: Median around 0.7, IQR between 0.6 and 0.75. **2. Direct-Effect:** * Unfair: Median around 0.25, IQR between 0.15 and 0.35. * Unaware: Median around 0.4, IQR between 0.3 and 0.5. * FairPFN: Median around 0.4, IQR between 0.3 and 0.5. * Cntf. Avg.: Median around 0.4, IQR between 0.3 and 0.5. * CFP: Median around 0.4, IQR between 0.3 and 0.5. * EGR: Median around 0.4, IQR between 0.3 and 0.5. * Random: Median around 0.5, IQR between 0.4 and 0.6. * Constant: Median around 0.5, IQR between 0.4 and 0.6. **3. Indirect-Effect:** * Unfair: Median around 0.35, IQR between 0.25 and 0.45. * Unaware: Median around 0.4, IQR between 0.3 and 0.5. * FairPFN: Median around 0.4, IQR between 0.3 and 0.5. * Cntf. Avg.: Median around 0.4, IQR between 0.3 and 0.5. * CFP: Median around 0.4, IQR between 0.3 and 0.5. * EGR: Median around 0.4, IQR between 0.3 and 0.5. * Random: Median around 0.5, IQR between 0.4 and 0.6. * Constant: Median around 0.4, IQR between 0.3 and 0.5. **4. Fair Observable:** * Unfair: Median around 0.2, IQR between 0.1 and 0.3. * Unaware: Median around 0.3, IQR between 0.2 and 0.4. * FairPFN: Median around 0.4, IQR between 0.3 and 0.5. * Cntf. Avg.: Median around 0.35, IQR between 0.25 and 0.45. * CFP: Median around 0.4, IQR between 0.3 and 0.5. * EGR: Median around 0.3, IQR between 0.2 and 0.4. * Random: Median around 0.5, IQR between 0.4 and 0.6. * Constant: Median around 0.4, IQR between 0.3 and 0.5. **5. Fair Unobservable:** * Unfair: Median around 0.25, IQR between 0.15 and 0.35. * Unaware: Median around 0.35, IQR between 0.25 and 0.45. * FairPFN: Median around 0.35, IQR between 0.25 and 0.45. * Cntf. Avg.: Median around 0.35, IQR between 0.25 and 0.45. * CFP: Median around 0.35, IQR between 0.25 and 0.45. * EGR: Median around 0.35, IQR between 0.25 and 0.45. * Random: Median around 0.5, IQR between 0.4 and 0.6. * Constant: Median around 0.3, IQR between 0.2 and 0.4. **6. Fair Additive Noise:** * Unfair: Median around 0.2, IQR between 0.1 and 0.3. * Unaware: Median around 0.3, IQR between 0.2 and 0.4. * FairPFN: Median around 0.35, IQR between 0.25 and 0.45. * Cntf. Avg.: Median around 0.35, IQR between 0.25 and 0.45. * CFP: Median around 0.35, IQR between 0.25 and 0.45. * EGR: Median around 0.35, IQR between 0.25 and 0.45. * Random: Median around 0.5, IQR between 0.4 and 0.6. * Constant: Median around 0.3, IQR between 0.2 and 0.4. ### Key Observations * The "Random" and "Constant" methods consistently have higher error rates (1-AUC) across all scenarios. * The "Unfair" method generally has lower error rates compared to "Unaware" in most scenarios. * The "Fair" scenarios (4, 5, and 6) tend to have lower error rates overall compared to the "Biased" and "Effect" scenarios (1, 2, and 3). * The average rank (1-AUC) in the legend correlates with the observed error rates in the box plots. Methods with lower average ranks (e.g., "Unfair") tend to have lower error rates. ### Interpretation The box plots compare the performance of different methods in terms of error rate (1-AUC) under various fairness scenarios. The data suggests that explicitly addressing fairness concerns (as in the "Fair" scenarios) can lead to lower error rates compared to scenarios where bias is present or fairness is not considered. The "Random" and "Constant" methods, which likely represent baseline or naive approaches, consistently perform worse than the other methods. The relative performance of the "Unfair" and "Unaware" methods varies depending on the scenario, indicating that the impact of awareness of unfairness depends on the specific context. The average rank (1-AUC) provides a summary measure of the overall performance of each method across all scenarios, which aligns with the observed trends in the box plots.

## Box Plot: Error (1-AUC) Across Fairness Interventions ### Overview This image presents a 2x3 grid of box plots, each representing the distribution of "Error (1-AUC)" for different fairness interventions under various fairness definitions. The plots compare the performance of several algorithms (Unfair, Unaware, FairPFN, CFP, EGR, Random, Constant) across six different scenarios: Biased, Direct-Effect, Indirect-Effect, Fair Observable, Fair Unobservable, and Fair Additive Noise. ### Components/Axes * **Y-axis:** "Error (1-AUC)", ranging from approximately 0 to 0.75. * **X-axis:** Implicitly represents the different algorithms being compared within each scenario. * **Box Plots:** Each box plot represents the distribution of Error (1-AUC) for a specific algorithm in a specific scenario. The box represents the interquartile range (IQR), the line inside the box represents the median, and the whiskers extend to the minimum and maximum values within 1.5 times the IQR. Outliers are shown as individual points. * **Legend:** Located at the bottom-right of the image, it maps colors to the following algorithms: * Unfair (Light Blue): Avg. Rank (1-AUC) = 2.17 * Unaware (Orange): Avg. Rank (1-AUC) = 2.62 * FairPFN (Purple): Avg. Rank (1-AUC) = 3.51 * CFP (Dark Orange): Avg. Rank (1-AUC) = 4.28 * EGR (Violet): Avg. Rank (1-AUC) = 5.18 * Random (Red): Avg. Rank (1-AUC) = 6.67 * Constant (Dark Green): Avg. Rank (1-AUC) = 6.75 * **Titles:** Each subplot is numbered and labeled with a fairness scenario: 1. Biased, 2. Direct-Effect, 3. Indirect-Effect, 4. Fair Observable, 5. Fair Unobservable, 6. Fair Additive Noise. ### Detailed Analysis Here's a breakdown of the trends and approximate values observed in each subplot: **1. Biased:** * Unfair (Light Blue): Median around 0.35, IQR from 0.25 to 0.45. * Unaware (Orange): Median around 0.4, IQR from 0.3 to 0.5. * FairPFN (Purple): Median around 0.5, IQR from 0.4 to 0.6. * CFP (Dark Orange): Median around 0.55, IQR from 0.45 to 0.65. * EGR (Violet): Median around 0.6, IQR from 0.5 to 0.7. * Random (Red): Median around 0.5, IQR from 0.4 to 0.6. * Constant (Dark Green): Median around 0.5, IQR from 0.4 to 0.6. **2. Direct-Effect:** * Unfair (Light Blue): Median around 0.3, IQR from 0.2 to 0.4. * Unaware (Orange): Median around 0.4, IQR from 0.3 to 0.5. * FairPFN (Purple): Median around 0.5, IQR from 0.4 to 0.6. * CFP (Dark Orange): Median around 0.55, IQR from 0.45 to 0.65. * EGR (Violet): Median around 0.6, IQR from 0.5 to 0.7. * Random (Red): Median around 0.5, IQR from 0.4 to 0.6. * Constant (Dark Green): Median around 0.5, IQR from 0.4 to 0.6. **3. Indirect-Effect:** * Unfair (Light Blue): Median around 0.35, IQR from 0.25 to 0.45. * Unaware (Orange): Median around 0.4, IQR from 0.3 to 0.5. * FairPFN (Purple): Median around 0.5, IQR from 0.4 to 0.6. * CFP (Dark Orange): Median around 0.55, IQR from 0.45 to 0.65. * EGR (Violet): Median around 0.6, IQR from 0.5 to 0.7. * Random (Red): Median around 0.5, IQR from 0.4 to 0.6. * Constant (Dark Green): Median around 0.5, IQR from 0.4 to 0.6. **4. Fair Observable:** * Unfair (Light Blue): Median around 0.25, IQR from 0.2 to 0.35. * Unaware (Orange): Median around 0.3, IQR from 0.2 to 0.4. * FairPFN (Purple): Median around 0.4, IQR from 0.3 to 0.5. * CFP (Dark Orange): Median around 0.45, IQR from 0.35 to 0.55. * EGR (Violet): Median around 0.5, IQR from 0.4 to 0.6. * Random (Red): Median around 0.45, IQR from 0.35 to 0.55. * Constant (Dark Green): Median around 0.5, IQR from 0.4 to 0.6. **5. Fair Unobservable:** * Unfair (Light Blue): Median around 0.3, IQR from 0.2 to 0.4. * Unaware (Orange): Median around 0.35, IQR from 0.25 to 0.45. * FairPFN (Purple): Median around 0.45, IQR from 0.35 to 0.55. * CFP (Dark Orange): Median around 0.5, IQR from 0.4 to 0.6. * EGR (Violet): Median around 0.55, IQR from 0.45 to 0.65. * Random (Red): Median around 0.5, IQR from 0.4 to 0.6. * Constant (Dark Green): Median around 0.5, IQR from 0.4 to 0.6. **6. Fair Additive Noise:** * Unfair (Light Blue): Median around 0.3, IQR from 0.2 to 0.4. * Unaware (Orange): Median around 0.35, IQR from 0.25 to 0.45. * FairPFN (Purple): Median around 0.45, IQR from 0.35 to 0.55. * CFP (Dark Orange): Median around 0.5, IQR from 0.4 to 0.6. * EGR (Violet): Median around 0.55, IQR from 0.45 to 0.65. * Random (Red): Median around 0.5, IQR from 0.4 to 0.6. * Constant (Dark Green): Median around 0.5, IQR from 0.4 to 0.6. ### Key Observations * The "Unfair" and "Unaware" algorithms generally exhibit lower error rates compared to the fairness-aware algorithms (FairPFN, CFP, EGR) in the "Biased", "Direct-Effect", and "Indirect-Effect" scenarios. * In the "Fair" scenarios (Observable, Unobservable, Additive Noise), the error rates tend to increase for all algorithms, but the differences between them become less pronounced. * The "Constant" and "Random" algorithms consistently show similar performance across all scenarios, with median error rates around 0.5. * The average rank values in the legend indicate that "Unfair" has the lowest average rank (2.17), suggesting it performs best overall, while "Constant" and "Random" have the highest ranks (6.75 and 6.67, respectively). ### Interpretation The data suggests a trade-off between fairness and accuracy. Algorithms that do not explicitly address fairness ("Unfair" and "Unaware") achieve lower error rates in scenarios where bias is present, but this comes at the cost of potentially perpetuating or exacerbating unfairness. Fairness-aware algorithms (FairPFN, CFP, EGR) generally have higher error rates, but they aim to mitigate bias and promote fairness. The increased error rates in the "Fair" scenarios indicate that enforcing fairness constraints can lead to a decrease in overall accuracy. This is a common phenomenon in fairness-aware machine learning. The similar performance of the "Constant" and "Random" algorithms suggests that simply adding noise or using a constant prediction does not necessarily improve fairness or accuracy. The average rank values provide a quantitative measure of the overall performance of each algorithm. The fact that "Unfair" has the lowest average rank highlights the accuracy-fairness trade-off. The choice of which algorithm to use depends on the specific application and the relative importance of accuracy and fairness.

## Box Plot Comparison: Model Performance Across Fairness Scenarios ### Overview The image displays a 2x3 grid of six box plots, each comparing the performance of eight different machine learning models or methods across distinct data scenarios. Performance is measured by "Error (1-AUC)", where a lower value indicates better performance. A comprehensive legend at the bottom maps colors to method names and provides an overall average rank for each method. ### Components/Axes * **Chart Type:** Six separate box plot charts arranged in a grid. * **Y-Axis (All Charts):** Labeled **"Error (1-AUC)"**. The scale runs from 0.0 to 0.75, with major gridlines at 0.0, 0.25, 0.5, and 0.75. * **Subplot Titles (Top of each chart):** 1. **Biased** (Top-Left) 2. **Direct-Effect** (Top-Center) 3. **Indirect-Effect** (Top-Right) 4. **Fair Observable** (Bottom-Left) 5. **Fair Unobservable** (Bottom-Center) 6. **Fair Additive Noise** (Bottom-Right) * **Legend (Bottom Center):** A box containing color swatches and labels for eight methods, along with their "Avg. Rank (1-AUC)". Lower rank indicates better average performance. * **Blue:** Unfair: 2.17 * **Orange:** Unaware: 2.62 * **Pink:** FairPFN: 3.51 * **Olive Green:** Cntf. Avg.: 3.62 * **Brown:** CFP: 4.28 * **Purple:** EGR: 5.18 * **Red:** Random: 6.67 * **Green:** Constant: 6.75 ### Detailed Analysis Each subplot contains eight box plots, one for each method listed in the legend. The box represents the interquartile range (IQR), the line inside is the median, whiskers extend to 1.5*IQR, and circles represent outliers. **1. Biased (Top-Left):** * **Unfair (Blue) & Unaware (Orange):** Lowest median error (~0.35-0.4). Similar distributions. * **FairPFN (Pink), Cntf. Avg. (Olive), CFP (Brown), EGR (Purple):** Clustered with slightly higher median error (~0.4-0.45). FairPFN and Cntf. Avg. appear marginally better than CFP and EGR. * **Random (Red):** Highest median error, centered at 0.5. Tight IQR. * **Constant (Green):** Not visibly plotted in this chart (likely overlapping with Random or omitted). **2. Direct-Effect (Top-Center):** * **Unfair (Blue):** Lowest median error (~0.25). * **Unaware (Orange):** Second lowest (~0.35). * **FairPFN (Pink), Cntf. Avg. (Olive), CFP (Brown), EGR (Purple):** Medians range from ~0.35 to 0.45. FairPFN and Cntf. Avg. are on the lower end. * **Random (Red):** Median at 0.5. * **Constant (Green):** Not visibly plotted. **3. Indirect-Effect (Top-Right):** * **Unfair (Blue) & Unaware (Orange):** Very similar, lowest median error (~0.35). * **FairPFN (Pink), Cntf. Avg. (Olive), CFP (Brown), EGR (Purple):** Medians between ~0.35 and 0.4. FairPFN and Cntf. Avg. are again slightly better. * **Random (Red):** Median at 0.5. * **Constant (Green):** Not visibly plotted. **4. Fair Observable (Bottom-Left):** * **Unfair (Blue):** Lowest median error (~0.2). * **Unaware (Orange):** Second lowest (~0.25). * **FairPFN (Pink), Cntf. Avg. (Olive), CFP (Brown), EGR (Purple):** Medians between ~0.25 and 0.35. FairPFN and Cntf. Avg. are lower than CFP and EGR. * **Random (Red):** Median at 0.5. * **Constant (Green):** Not visibly plotted. **5. Fair Unobservable (Bottom-Center):** * **Unfair (Blue):** Lowest median error (~0.2). * **Unaware (Orange):** Second lowest (~0.25). * **FairPFN (Pink), Cntf. Avg. (Olive), CFP (Brown), EGR (Purple):** Medians between ~0.25 and 0.35. FairPFN and Cntf. Avg. are lower. * **Random (Red):** Median at 0.5. * **Constant (Green):** Not visibly plotted. **6. Fair Additive Noise (Bottom-Right):** * **Unfair (Blue):** Lowest median error (~0.15). * **Unaware (Orange):** Second lowest (~0.2). * **FairPFN (Pink), Cntf. Avg. (Olive), CFP (Brown), EGR (Purple):** Medians between ~0.2 and 0.3. FairPFN and Cntf. Avg. are lower. * **Random (Red):** Median at 0.5. * **Constant (Green):** Not visibly plotted. ### Key Observations 1. **Consistent Hierarchy:** Across all six scenarios, the performance order is remarkably consistent: **Unfair** (best) > **Unaware** > **FairPFN** ≈ **Cntf. Avg.** > **CFP** ≈ **EGR** > **Random** (worst). The **Constant** method is listed in the legend but does not appear as a distinct box plot in any chart, suggesting its performance may be identical to or obscured by the Random baseline (median=0.5). 2. **Scenario Impact:** The absolute error values are lowest in the "Fair" scenarios (4, 5, 6), particularly "Fair Additive Noise," where the best models achieve medians below 0.2. The "Biased" scenario shows the highest overall error levels. 3. **Baseline Performance:** The **Random** method consistently has a median error of 0.5, which is the expected performance of random guessing for AUC (since AUC=0.5 implies no discriminative ability, making 1-AUC=0.5). 4. **Fairness-Accuracy Trade-off:** Methods designed for fairness (FairPFN, Cntf. Avg., CFP, EGR) consistently incur a higher error (lower AUC) than the "Unfair" and "Unaware" baselines, illustrating the typical trade-off between fairness and predictive performance. ### Interpretation This visualization provides a clear empirical comparison of model performance under different data-generating processes related to fairness. The key takeaway is that **ignoring fairness constraints ("Unfair" and "Unaware" models) yields the best predictive performance (lowest 1-AUC) across all tested scenarios.** However, this comes at the potential cost of fairness. The fairness-aware methods (FairPFN, Cntf. Avg., CFP, EGR) successfully reduce error compared to the naive random baseline but cannot match the performance of models not constrained by fairness. Among these, **FairPFN and Counterfactual Averaging (Cntf. Avg.) consistently outperform Counterfactual Fairness Prediction (CFP) and Equalized Odds Post-processing (EGR)**, suggesting they may be more effective at balancing fairness and accuracy in these contexts. The fact that the "Constant" model (which likely predicts a single constant value) performs identically to "Random" (median error 0.5) indicates that in these binary classification tasks, a trivial constant predictor is no better than random chance. The progression from "Biased" to the "Fair" scenarios shows that when the underlying data structure is fairer, all models, including the fairness-aware ones, achieve lower absolute error rates.

## Box Plot Chart: Error (1-AUC) Across Scenarios and Methods ### Overview The image displays six box plots arranged in a 2x3 grid, comparing the error rates (1-AUC) of different fairness-aware machine learning methods across six scenarios: Biased, Direct-Effect, Indirect-Effect, Fair Observable, Fair Unobservable, and Fair Additive Noise. Each box plot represents a method's performance distribution, with colors corresponding to specific methods and their average ranks. ### Components/Axes - **Y-Axis**: "Error (1-AUC)" (range: 0 to 0.75) - **X-Axis**: Six scenarios (labeled 1–6 with descriptive titles) - **Legend**: Located at the bottom, mapping colors to methods and their average ranks: - **Blue**: Unfair (Avg. Rank: 2.17) - **Orange**: Unaware (Avg. Rank: 2.62) - **Purple**: FairPFN (Avg. Rank: 3.51) - **Brown**: Cntf. Avg. (Avg. Rank: 3.62) - **Red**: Random (Avg. Rank: 6.67) - **Green**: Constant (Avg. Rank: 6.75) ### Detailed Analysis 1. **Biased (1)**: - **Unfair (blue)**: Lowest median error (~0.3), narrowest interquartile range. - **Unaware (orange)**: Slightly higher median (~0.4), wider spread. - **FairPFN (purple)**: Median ~0.45, moderate spread. - **Cntf. Avg. (brown)**: Median ~0.5, similar spread to FairPFN. - **Random (red)**: Highest median (~0.55), large spread. - **Constant (green)**: Median ~0.6, widest spread. 2. **Direct-Effect (2)**: - **Unfair (blue)**: Lowest median (~0.3), tightest distribution. - **Unaware (orange)**: Median ~0.4, moderate spread. - **FairPFN (purple)**: Median ~0.45, similar to Cntf. Avg. (brown). - **Random (red)**: Median ~0.5, large spread. - **Constant (green)**: Median ~0.6, widest spread. 3. **Indirect-Effect (3)**: - **Unfair (blue)**: Lowest median (~0.3), narrowest range. - **Unaware (orange)**: Median ~0.4, moderate spread. - **FairPFN (purple)**: Median ~0.45, similar to Cntf. Avg. (brown). - **Random (red)**: Median ~0.5, large spread. - **Constant (green)**: Median ~0.6, widest spread. 4. **Fair Observable (4)**: - **Unfair (blue)**: Lowest median (~0.3), tightest distribution. - **Unaware (orange)**: Median ~0.4, moderate spread. - **FairPFN (purple)**: Median ~0.45, similar to Cntf. Avg. (brown). - **Random (red)**: Median ~0.5, large spread. - **Constant (green)**: Median ~0.6, widest spread. 5. **Fair Unobservable (5)**: - **Unfair (blue)**: Lowest median (~0.3), tightest distribution. - **Unaware (orange)**: Median ~0.4, moderate spread. - **FairPFN (purple)**: Median ~0.45, similar to Cntf. Avg. (brown). - **Random (red)**: Median ~0.5, large spread. - **Constant (green)**: Median ~0.6, widest spread. 6. **Fair Additive Noise (6)**: - **Unfair (blue)**: Lowest median (~0.3), narrowest range. - **Unaware (orange)**: Median ~0.4, moderate spread. - **FairPFN (purple)**: Median ~0.45, similar to Cntf. Avg. (brown). - **Random (red)**: Median ~0.5, large spread. - **Constant (green)**: Median ~0.6, widest spread. ### Key Observations - **Unfair (blue)** consistently shows the lowest error rates (1-AUC) across all scenarios, with the tightest distributions. - **Unaware (orange)** performs second-best, with slightly higher medians and wider spreads than Unfair. - **FairPFN (purple)** and **Cntf. Avg. (brown)** have overlapping medians (~0.45) but FairPFN has slightly tighter distributions. - **Random (red)** and **Constant (green)** exhibit the highest errors, with Constant showing the widest spread (indicating high variability). ### Interpretation The data suggests that **Unfair** and **Unaware** methods outperform fairness-aware approaches (FairPFN, Cntf. Avg.) in terms of error rates (1-AUC) across all scenarios. This implies that fairness-aware methods may introduce trade-offs between fairness and accuracy. The **Random** and **Constant** methods perform worst, with Constant showing the highest variability (widest spreads). The average ranks confirm this hierarchy: Unfair (2.17) and Unaware (2.62) rank highest (best performance), while Constant (6.75) ranks lowest. The consistent pattern across scenarios indicates that fairness constraints may not always align with optimal predictive performance.