## 3D Surface Plots: ROAR Budget vs. MRR ### Overview The image presents a series of twelve 3D surface plots, arranged in a 2x6 grid. Each plot visualizes the relationship between two budget parameters (ROARkp budget and ROARqm budget) and the Mean Reciprocal Rank (MRR). The plots are grouped into two categories: "Backdoor" and "Targeted," each with six sub-categories representing different tasks or datasets. The color gradient on the surface represents the MRR value, ranging from dark green (low MRR) to bright yellow (high MRR). ### Components/Axes * **X-axis:** ROARkp budget, ranging from 0 to 200. * **Y-axis:** ROARqm budget, ranging from 0 to 4. * **Z-axis:** MRR (Mean Reciprocal Rank), ranging from 0.0 to 1.0 (or 0.8 in some plots). * **Color Gradient:** Represents the MRR value, with dark green indicating lower values and bright yellow indicating higher values. * **Titles:** Each plot has a title indicating the category (Backdoor or Targeted) and the specific task/dataset (e.g., Vulnerability, Mitigation, Diagnosis, Treatment, Freebase, WordNet). ### Detailed Analysis Here's a breakdown of each plot, including key data points and trends: **(a) Backdoor-Vulnerability:** * Trend: MRR increases significantly with both ROARkp and ROARqm budgets. * Data Points: * (ROARkp=0, ROARqm=0): MRR ≈ 0.04 * (ROARkp=200, ROARqm=0): MRR ≈ 0.28 * (ROARkp=0, ROARqm=4): MRR ≈ 0.55 * (ROARkp=200, ROARqm=4): MRR ≈ 0.56 **(b) Backdoor-Mitigation:** * Trend: MRR increases with both ROARkp and ROARqm budgets, but the increase plateaus at higher budget values. * Data Points: * (ROARkp=0, ROARqm=0): MRR ≈ 0.04 * (ROARkp=200, ROARqm=0): MRR ≈ 0.39 * (ROARkp=0, ROARqm=4): MRR ≈ 0.73 * (ROARkp=200, ROARqm=4): MRR ≈ 0.67 **(c) Backdoor-Diagnosis:** * Trend: MRR initially increases with both budgets, but then decreases slightly at higher ROARqm budget values. * Data Points: * (ROARkp=0, ROARqm=0): MRR ≈ 0.02 * (ROARkp=200, ROARqm=0): MRR ≈ 0.10 * (ROARkp=0, ROARqm=4): MRR ≈ 0.40 * (ROARkp=200, ROARqm=4): MRR ≈ 0.31 **(d) Backdoor-Treatment:** * Trend: MRR increases with both budgets, with a more pronounced increase at higher ROARkp budget values. * Data Points: * (ROARkp=0, ROARqm=0): MRR ≈ 0.08 * (ROARkp=200, ROARqm=0): MRR ≈ 0.47 * (ROARkp=0, ROARqm=4): MRR ≈ 0.72 * (ROARkp=200, ROARqm=4): MRR ≈ 0.70 **(e) Backdoor-Freebase:** * Trend: MRR increases with both budgets, with a more pronounced increase at higher ROARkp budget values. * Data Points: * (ROARkp=0, ROARqm=0): MRR ≈ 0.00 * (ROARkp=200, ROARqm=0): MRR ≈ 0.57 * (ROARkp=0, ROARqm=4): MRR ≈ 0.62 * (ROARkp=200, ROARqm=4): MRR ≈ 0.58 **(f) Backdoor-WordNet:** * Trend: MRR increases with both budgets, with a more pronounced increase at higher ROARkp budget values. * Data Points: * (ROARkp=0, ROARqm=0): MRR ≈ 0.00 * (ROARkp=200, ROARqm=0): MRR ≈ 0.55 * (ROARkp=0, ROARqm=4): MRR ≈ 0.75 * (ROARkp=200, ROARqm=4): MRR ≈ 0.71 **(g) Targeted-Vulnerability:** * Trend: MRR is high when ROARkp budget is low, and decreases significantly as ROARkp budget increases. ROARqm budget has a smaller positive impact. * Data Points: * (ROARkp=0, ROARqm=0): MRR ≈ 0.91 * (ROARkp=200, ROARqm=0): MRR ≈ 0.02 * (ROARkp=0, ROARqm=4): MRR ≈ 0.43 * (ROARkp=200, ROARqm=4): MRR ≈ 0.02 **(h) Targeted-Mitigation:** * Trend: MRR is relatively high when ROARkp budget is low, and decreases as ROARkp budget increases. ROARqm budget has a smaller positive impact. * Data Points: * (ROARkp=0, ROARqm=0): MRR ≈ 0.72 * (ROARkp=200, ROARqm=0): MRR ≈ 0.02 * (ROARkp=0, ROARqm=4): MRR ≈ 0.22 * (ROARkp=200, ROARqm=4): MRR ≈ 0.02 **(i) Targeted-Diagnosis:** * Trend: MRR is relatively high when ROARkp budget is low, and decreases as ROARkp budget increases. ROARqm budget has a smaller positive impact. * Data Points: * (ROARkp=0, ROARqm=0): MRR ≈ 0.49 * (ROARkp=200, ROARqm=0): MRR ≈ 0.00 * (ROARkp=0, ROARqm=4): MRR ≈ 0.26 * (ROARkp=200, ROARqm=4): MRR ≈ 0.02 **(j) Targeted-Treatment:** * Trend: MRR is relatively high when ROARkp budget is low, and decreases as ROARkp budget increases. ROARqm budget has a smaller positive impact. * Data Points: * (ROARkp=0, ROARqm=0): MRR ≈ 0.59 * (ROARkp=200, ROARqm=0): MRR ≈ 0.37 * (ROARkp=0, ROARqm=4): MRR ≈ 0.55 * (ROARkp=200, ROARqm=4): MRR ≈ 0.29 **(k) Targeted-Freebase:** * Trend: MRR is relatively high when ROARkp budget is low, and decreases as ROARkp budget increases. ROARqm budget has a smaller positive impact. * Data Points: * (ROARkp=0, ROARqm=0): MRR ≈ 0.44 * (ROARkp=200, ROARqm=0): MRR ≈ 0.03 * (ROARkp=0, ROARqm=4): MRR ≈ 0.10 * (ROARkp=200, ROARqm=4): MRR ≈ 0.04 **(l) Targeted-WordNet:** * Trend: MRR is relatively high when ROARkp budget is low, and decreases as ROARkp budget increases. ROARqm budget has a smaller positive impact. * Data Points: * (ROARkp=0, ROARqm=0): MRR ≈ 0.71 * (ROARkp=200, ROARqm=0): MRR ≈ 0.20 * (ROARkp=0, ROARqm=4): MRR ≈ 0.35 * (ROARkp=200, ROARqm=4): MRR ≈ 0.11 ### Key Observations * **Backdoor vs. Targeted:** The "Backdoor" category generally shows an increase in MRR with increasing ROARkp and ROARqm budgets. In contrast, the "Targeted" category often shows a decrease in MRR with increasing ROARkp budget, suggesting a different relationship between the budget parameters and performance. * **ROARkp Budget Impact:** The ROARkp budget appears to have a more significant impact on MRR than the ROARqm budget in many of the plots. * **Plateauing:** In some "Backdoor" plots (e.g., Mitigation), the MRR increase plateaus at higher budget values, suggesting diminishing returns for increased budget allocation. ### Interpretation The plots illustrate how different budget allocations for ROARkp and ROARqm affect the Mean Reciprocal Rank (MRR) across various tasks and datasets. The contrasting trends between the "Backdoor" and "Targeted" categories suggest that the optimal budget allocation strategy depends on the specific task or dataset. For "Backdoor" tasks, increasing both ROARkp and ROARqm budgets generally leads to improved performance, although the gains may diminish at higher budget levels. This suggests that investing in both types of resources is beneficial for these tasks. However, for "Targeted" tasks, increasing the ROARkp budget often leads to a decrease in MRR. This could indicate that a high ROARkp budget is detrimental to performance in these tasks, possibly due to overfitting or other negative effects. In these cases, a lower ROARkp budget and potentially a higher ROARqm budget might be more effective. The specific values and trends observed in each plot can inform the development of more effective budget allocation strategies for different tasks and datasets, ultimately leading to improved performance. The data suggests that a one-size-fits-all approach to budget allocation is unlikely to be optimal, and that careful consideration should be given to the specific characteristics of each task.

\n ## Heatmaps: Backdoor Attack Performance under Different Budgets ### Overview The image presents a series of six heatmaps, arranged in a 2x3 grid, visualizing the performance of different backdoor attack and defense strategies. Each heatmap represents a scenario, with the x and y axes representing "ROAR_exp budget" and "ROAR_att budget" respectively. The color intensity indicates the "MRR" (Mean Reciprocal Rank) score, ranging from 0.0 to 1.0, with warmer colors (green) indicating higher performance (higher MRR) and cooler colors (purple) indicating lower performance. Below the first row of heatmaps are six more, representing different targetted attacks. ### Components/Axes * **X-axis:** ROAR_att budget (ranging from 0 to 4, with markers at 0, 1, 2, 3, and 4). * **Y-axis:** ROAR_exp budget (ranging from 0 to 200, with markers at 0, 50, 100, 150, and 200). * **Color Scale:** MRR (Mean Reciprocal Rank), ranging from 0.0 (purple) to 1.0 (green). * **Heatmap Titles:** * (a) Backdoor-Vulnerability * (b) Backdoor-Mitigation * (c) Backdoor-Diagnosis * (d) Backdoor-Treatment * (e) Backdoor-Freebase * (f) Backdoor-WordNet * (g) Targetted-Vulnerability * (h) Targetted-Mitigation * (i) Targetted-Diagnosis * (j) Targetted-Treatment * (k) Targetted-Freebase * (l) Targetted-WordNet ### Detailed Analysis or Content Details **Row 1:** * **(a) Backdoor-Vulnerability:** The heatmap shows a generally increasing MRR with increasing ROAR_exp budget. The highest MRR (~0.73) is located at ROAR_exp budget = 4 and ROAR_att budget = 0. A low MRR (~0.04) is observed at ROAR_exp budget = 0 and ROAR_att budget = 0. * **(b) Backdoor-Mitigation:** The MRR is relatively low across the board, with a peak of ~0.67 at ROAR_exp budget = 4 and ROAR_att budget = 0. The lowest MRR (~0.04) is at ROAR_exp budget = 0 and ROAR_att budget = 0. * **(c) Backdoor-Diagnosis:** The MRR is generally low, with a peak of ~0.40 at ROAR_exp budget = 4 and ROAR_att budget = 0. The lowest MRR (~0.02) is at ROAR_exp budget = 0 and ROAR_att budget = 0. * **(d) Backdoor-Treatment:** The MRR increases with increasing ROAR_exp budget, peaking at ~0.72 at ROAR_exp budget = 4 and ROAR_att budget = 0. The lowest MRR (~0.08) is at ROAR_exp budget = 0 and ROAR_att budget = 0. * **(e) Backdoor-Freebase:** The MRR is relatively high, peaking at ~0.62 at ROAR_exp budget = 4 and ROAR_att budget = 0. The lowest MRR (~0.00) is at ROAR_exp budget = 0 and ROAR_att budget = 0. * **(f) Backdoor-WordNet:** The MRR is relatively high, peaking at ~0.75 at ROAR_exp budget = 4 and ROAR_att budget = 0. The lowest MRR (~0.55) is at ROAR_exp budget = 0 and ROAR_att budget = 0. **Row 2:** * **(g) Targetted-Vulnerability:** The MRR is relatively high, peaking at ~0.91 at ROAR_exp budget = 0 and ROAR_att budget = 0. The lowest MRR (~0.22) is at ROAR_exp budget = 200 and ROAR_att budget = 4. * **(h) Targetted-Mitigation:** The MRR is relatively low, peaking at ~0.43 at ROAR_exp budget = 0 and ROAR_att budget = 0. The lowest MRR (~0.10) is at ROAR_exp budget = 200 and ROAR_att budget = 4. * **(i) Targetted-Diagnosis:** The MRR is relatively low, peaking at ~0.69 at ROAR_exp budget = 0 and ROAR_att budget = 0. The lowest MRR (~0.26) is at ROAR_exp budget = 200 and ROAR_att budget = 4. * **(j) Targetted-Treatment:** The MRR is relatively low, peaking at ~0.53 at ROAR_exp budget = 0 and ROAR_att budget = 0. The lowest MRR (~0.44) is at ROAR_exp budget = 200 and ROAR_att budget = 4. * **(k) Targetted-Freebase:** The MRR is relatively low, peaking at ~0.39 at ROAR_exp budget = 0 and ROAR_att budget = 0. The lowest MRR (~0.10) is at ROAR_exp budget = 200 and ROAR_att budget = 4. * **(l) Targetted-WordNet:** The MRR is relatively low, peaking at ~0.71 at ROAR_exp budget = 0 and ROAR_att budget = 0. The lowest MRR (~0.35) is at ROAR_exp budget = 200 and ROAR_att budget = 4. ### Key Observations * For the "Backdoor" series, increasing the ROAR_exp budget generally improves MRR, suggesting that more exploration helps. * The "Targetted" series generally shows higher MRR values at lower budgets, and a decrease in MRR as budgets increase. * "Backdoor-WordNet" and "Targetted-WordNet" consistently show higher MRR values compared to other strategies. * "Backdoor-Mitigation", "Backdoor-Diagnosis", and "Backdoor-Treatment" have relatively low MRR values across all budget combinations. ### Interpretation The heatmaps demonstrate the effectiveness of different backdoor attack and defense strategies under varying resource constraints (ROAR_exp and ROAR_att budgets). The MRR score serves as a proxy for the success of the attack or defense. The positive correlation between ROAR_exp budget and MRR in the "Backdoor" series suggests that increased exploration of the model's vulnerabilities leads to more successful attacks. Conversely, the negative correlation in the "Targetted" series indicates that focused attacks may become less effective as more resources are allocated to both exploration and attack. The consistently high performance of "WordNet" strategies suggests that this approach is particularly robust or effective in exploiting/mitigating backdoor vulnerabilities. The lower performance of "Mitigation" and "Diagnosis" strategies indicates that these defenses may be less effective in practice, or require significantly more resources to achieve comparable results. The differences between the "Backdoor" and "Targetted" series highlight the importance of considering the attack strategy when evaluating defense mechanisms. A defense that is effective against general backdoor attacks may not be as effective against targeted attacks, and vice versa. The data suggests a trade-off between exploration and attack budgets, and the optimal strategy may depend on the specific context and available resources.

## [3D Surface Plots]: MRR vs. ROAR Budgets for Backdoor and Targeted Attacks ### Overview The image contains 12 3D surface plots (arranged in 2 rows × 6 columns) illustrating the **Mean Reciprocal Rank (MRR)** (a ranking performance metric) as a function of two budget parameters: - **X-axis**: `ROAR_kp budget` (range: 0–200, ticks: 0, 50, 100, 150, 200). - **Y-axis**: `ROAR_qm budget` (range: 0–4, ticks: 0, 1, 2, 3, 4). - **Z-axis**: `MRR` (range: 0.0–1.0, or 0.0–0.6 for some plots). Plots are grouped by attack type (**Backdoor** (top row) vs. **Targeted** (bottom row)) and task (Vulnerability, Mitigation, Diagnosis, Treatment, Freebase, WordNet). ### Components/Axes - **Axes Labels**: - X: `ROAR_kp budget` (front axis, 0–200). - Y: `ROAR_qm budget` (left axis, 0–4). - Z: `MRR` (vertical axis, 0.0–1.0 or 0.0–0.6). - **Plot Titles** (row-wise): - Top (Backdoor): (a) Vulnerability, (b) Mitigation, (c) Diagnosis, (d) Treatment, (e) Freebase, (f) WordNet. - Bottom (Targeted): (g) Vulnerability, (h) Mitigation, (i) Diagnosis, (j) Treatment, (k) Freebase, (l) WordNet. - **Surface Labels**: Numerical MRR values (e.g., 0.04, 0.55, 0.73) at specific (ROAR_kp, ROAR_qm) points. ### Detailed Analysis (Per Attack Type) #### **Top Row: Backdoor Attacks** Backdoor attacks show MRR increasing with moderate budgets (ROAR_kp ~100, ROAR_qm ~2), then plateauing: - **(a) Backdoor-Vulnerability**: MRR ~0.55–0.56 at (100, 2); low values (0.04, 0.28) at (0, 0) and (0, 2). - **(b) Backdoor-Mitigation**: MRR ~0.73–0.67 at (100, 2); low values (0.04, 0.39) at (0, 0) and (0, 2). - **(c) Backdoor-Diagnosis**: Lower MRR (max ~0.40–0.31) at (100, 2); low values (0.02, 0.10) at (0, 0) and (0, 2). - **(d) Backdoor-Treatment**: MRR ~0.72–0.70 at (100, 2); low values (0.08, 0.47) at (0, 0) and (0, 2). - **(e) Backdoor-Freebase**: MRR ~0.62–0.58 at (100, 2); low values (0.00, 0.57) at (0, 0) and (0, 2). - **(f) Backdoor-WordNet**: MRR ~0.75–0.71 at (100, 2); low values (0.00, 0.55) at (0, 0) and (0, 2). #### **Bottom Row: Targeted Attacks** Targeted attacks peak at low ROAR_kp (0) and moderate ROAR_qm (2), then decline with higher ROAR_kp: - **(g) Targeted-Vulnerability**: MRR ~0.91 at (0, 2); drops to 0.00 at (200, 2). - **(h) Targeted-Mitigation**: MRR ~0.72 at (0, 2); drops to 0.02 at (200, 2). - **(i) Targeted-Diagnosis**: MRR ~0.49 at (0, 2); drops to 0.00 at (200, 2). - **(j) Targeted-Treatment**: MRR ~0.59 at (0, 2); drops to 0.29 at (200, 2). - **(k) Targeted-Freebase**: MRR ~0.44 at (0, 2); drops to 0.04 at (200, 2). - **(l) Targeted-WordNet**: MRR ~0.71 at (0, 2); drops to 0.11 at (200, 2). ### Key Observations 1. **Attack Type Differences**: - Backdoor attacks perform best with *moderate* budgets (ROAR_kp ~100, ROAR_qm ~2). - Targeted attacks perform best with *low* ROAR_kp (0) and *moderate* ROAR_qm (2), then decline with higher ROAR_kp. 2. **Task Variability**: - Diagnosis tasks (Backdoor-Diagnosis, Targeted-Diagnosis) have the lowest MRR, suggesting resistance to attacks. - WordNet and Vulnerability tasks have the highest MRR, indicating greater vulnerability/effectiveness of attacks. ### Interpretation The plots reveal how budget allocation (ROAR_kp, ROAR_qm) impacts attack effectiveness (MRR) across tasks: - **Backdoor Attacks**: Balanced budgets (moderate kp/qm) enhance effectiveness, likely due to distributed resource allocation. - **Targeted Attacks**: Focused budgets (low kp, moderate qm) maximize effectiveness, as resources are concentrated on key targets. - **Security Implications**: Diagnosis tasks are more robust to attacks, while WordNet/Vulnerability tasks require stronger defenses. Budget optimization (e.g., limiting kp for targeted attacks) can mitigate risks. This analysis helps inform security strategies (e.g., hardening vulnerable tasks, optimizing attack budgets) by quantifying how resource allocation impacts attack performance.

## 3D Surface Plots: Model Performance vs. Resource Budgets ### Overview The image contains 12 3D surface plots arranged in two rows (6 per row), visualizing the relationship between two resource budgets (ROAR_qm and ROAR_kp) and model performance (MMR). Each plot represents a different scenario (e.g., Backdoor-Vulnerability, Targeted-Mitigation) with color gradients indicating performance levels (green = low MMR, yellow = high MMR). Numerical values are annotated on the surfaces to highlight key performance metrics. --- ### Components/Axes - **X-axis**: ROAR_kp budget (ranging from 0 to 200 in most plots, 0–4 in some) - **Y-axis**: ROAR_qm budget (ranging from 0 to 4 in most plots, 0–3 in some) - **Z-axis**: MMR (Mean Reciprocal Rank, 0.00–1.00 scale) - **Legend**: Implicit color gradient (green = low MMR, yellow = high MMR) - **Plot Titles**: - Row 1: Backdoor-Vulnerability, Backdoor-Mitigation, Backdoor-Diagnosis, Backdoor-Treatment, Backdoor-Freebase, Backdoor-WordNet - Row 2: Targeted-Vulnerability, Targeted-Mitigation, Targeted-Diagnosis, Targeted-Treatment, Targeted-Freebase, Targeted-WordNet --- ### Detailed Analysis #### Backdoor Scenarios (Row 1) 1. **(a) Backdoor-Vulnerability** - Peaks at **0.56** (ROAR_kp=150, ROAR_qm=3) and **0.55** (ROAR_kp=50, ROAR_qm=0). - Lowest MMR (**0.04**) at ROAR_kp=0, ROAR_qm=0. - Gradual increase in MMR with higher budgets. 2. **(b) Backdoor-Mitigation** - Highest MMR (**0.73**) at ROAR_kp=150, ROAR_qm=3. - Sharp drop to **0.02** at ROAR_kp=0, ROAR_qm=0. - Mitigation strategies significantly improve performance. 3. **(c) Backdoor-Diagnosis** - Moderate peak (**0.40**) at ROAR_kp=150, ROAR_qm=3. - Minimal improvement at lower budgets (**0.02** at ROAR_kp=0, ROAR_qm=0). - Less effective than mitigation. 4. **(d) Backdoor-Treatment** - High MMR (**0.72**) at ROAR_kp=150, ROAR_qm=3. - Slightly lower than mitigation (**0.73**). - Consistent improvement across budgets. 5. **(e) Backdoor-Freebase** - Low MMR (**0.00** at ROAR_kp=0, ROAR_qm=0; **0.57** at ROAR_kp=150, ROAR_qm=3). - Freebase models underperform compared to other scenarios. 6. **(f) Backdoor-WordNet** - High MMR (**0.75**) at ROAR_kp=150, ROAR_qm=3. - Slight dip to **0.55** at ROAR_kp=50, ROAR_qm=0. - WordNet integration shows strong performance. #### Targeted Scenarios (Row 2) 7. **(g) Targeted-Vulnerability** - Extremely high MMR (**0.91**) at ROAR_kp=1, ROAR_qm=3. - Drops to **0.02** at ROAR_kp=3, ROAR_qm=0. - High vulnerability correlates with extreme performance swings. 8. **(h) Targeted-Mitigation** - Moderate MMR (**0.22** at ROAR_kp=1, ROAR_qm=3; **0.02** at ROAR_kp=3, ROAR_qm=0). - Mitigation reduces performance volatility but limits peak gains. 9. **(i) Targeted-Diagnosis** - Low MMR (**0.26** at ROAR_kp=1, ROAR_qm=3; **0.00** at ROAR_kp=3, ROAR_qm=0). - Diagnosis tools underperform in targeted scenarios. 10. **(j) Targeted-Treatment** - Moderate MMR (**0.55** at ROAR_kp=1, ROAR_qm=3; **0.37** at ROAR_kp=3, ROAR_qm=0). - Treatment improves performance but less than backdoor scenarios. 11. **(k) Targeted-Freebase** - Very low MMR (**0.03** at ROAR_kp=1, ROAR_qm=3; **0.04** at ROAR_kp=3, ROAR_qm=0). - Freebase models perform poorly in targeted settings. 12. **(l) Targeted-WordNet** - Moderate MMR (**0.71** at ROAR_kp=1, ROAR_qm=3; **0.11** at ROAR_kp=3, ROAR_qm=0). - WordNet integration shows resilience in targeted scenarios. --- ### Key Observations 1. **Budget Impact**: Higher ROAR_kp and ROAR_qm budgets generally correlate with higher MMR, except in targeted scenarios where resource allocation has diminishing returns. 2. **Scenario-Specific Performance**: - Backdoor-Mitigation and Backdoor-WordNet achieve the highest MMR (0.73–0.75). - Targeted-Freebase and Targeted-Diagnosis underperform (MMR < 0.3). 3. **Anomalies**: - Targeted-Vulnerability (g) shows extreme MMR (**0.91**) at low budgets, suggesting overfitting or data leakage. - Backdoor-Freebase (e) has near-zero MMR at zero budgets, indicating baseline inefficiency. --- ### Interpretation The data demonstrates that resource allocation (ROAR_kp and ROAR_qm) significantly impacts model performance (MMR), with backdoor scenarios benefiting more from increased budgets than targeted scenarios. Mitigation and WordNet integration are critical for improving robustness, while freebase models struggle across all scenarios. The extreme performance in Targeted-Vulnerability (g) raises concerns about data quality or model overfitting. These insights highlight the need for scenario-specific resource optimization and advanced mitigation strategies to balance performance and efficiency.