\n ## Heatmaps: Backdoor Attack Performance under Different Budgets ### Overview The image presents a series of six heatmaps, arranged in a 2x3 grid, visualizing the performance of different backdoor attack and defense strategies. Each heatmap represents a scenario, with the x and y axes representing "ROAR_exp budget" and "ROAR_att budget" respectively. The color intensity indicates the "MRR" (Mean Reciprocal Rank) score, ranging from 0.0 to 1.0, with warmer colors (green) indicating higher performance (higher MRR) and cooler colors (purple) indicating lower performance. Below the first row of heatmaps are six more, representing different targetted attacks. ### Components/Axes * **X-axis:** ROAR_att budget (ranging from 0 to 4, with markers at 0, 1, 2, 3, and 4). * **Y-axis:** ROAR_exp budget (ranging from 0 to 200, with markers at 0, 50, 100, 150, and 200). * **Color Scale:** MRR (Mean Reciprocal Rank), ranging from 0.0 (purple) to 1.0 (green). * **Heatmap Titles:** * (a) Backdoor-Vulnerability * (b) Backdoor-Mitigation * (c) Backdoor-Diagnosis * (d) Backdoor-Treatment * (e) Backdoor-Freebase * (f) Backdoor-WordNet * (g) Targetted-Vulnerability * (h) Targetted-Mitigation * (i) Targetted-Diagnosis * (j) Targetted-Treatment * (k) Targetted-Freebase * (l) Targetted-WordNet ### Detailed Analysis or Content Details **Row 1:** * **(a) Backdoor-Vulnerability:** The heatmap shows a generally increasing MRR with increasing ROAR_exp budget. The highest MRR (~0.73) is located at ROAR_exp budget = 4 and ROAR_att budget = 0. A low MRR (~0.04) is observed at ROAR_exp budget = 0 and ROAR_att budget = 0. * **(b) Backdoor-Mitigation:** The MRR is relatively low across the board, with a peak of ~0.67 at ROAR_exp budget = 4 and ROAR_att budget = 0. The lowest MRR (~0.04) is at ROAR_exp budget = 0 and ROAR_att budget = 0. * **(c) Backdoor-Diagnosis:** The MRR is generally low, with a peak of ~0.40 at ROAR_exp budget = 4 and ROAR_att budget = 0. The lowest MRR (~0.02) is at ROAR_exp budget = 0 and ROAR_att budget = 0. * **(d) Backdoor-Treatment:** The MRR increases with increasing ROAR_exp budget, peaking at ~0.72 at ROAR_exp budget = 4 and ROAR_att budget = 0. The lowest MRR (~0.08) is at ROAR_exp budget = 0 and ROAR_att budget = 0. * **(e) Backdoor-Freebase:** The MRR is relatively high, peaking at ~0.62 at ROAR_exp budget = 4 and ROAR_att budget = 0. The lowest MRR (~0.00) is at ROAR_exp budget = 0 and ROAR_att budget = 0. * **(f) Backdoor-WordNet:** The MRR is relatively high, peaking at ~0.75 at ROAR_exp budget = 4 and ROAR_att budget = 0. The lowest MRR (~0.55) is at ROAR_exp budget = 0 and ROAR_att budget = 0. **Row 2:** * **(g) Targetted-Vulnerability:** The MRR is relatively high, peaking at ~0.91 at ROAR_exp budget = 0 and ROAR_att budget = 0. The lowest MRR (~0.22) is at ROAR_exp budget = 200 and ROAR_att budget = 4. * **(h) Targetted-Mitigation:** The MRR is relatively low, peaking at ~0.43 at ROAR_exp budget = 0 and ROAR_att budget = 0. The lowest MRR (~0.10) is at ROAR_exp budget = 200 and ROAR_att budget = 4. * **(i) Targetted-Diagnosis:** The MRR is relatively low, peaking at ~0.69 at ROAR_exp budget = 0 and ROAR_att budget = 0. The lowest MRR (~0.26) is at ROAR_exp budget = 200 and ROAR_att budget = 4. * **(j) Targetted-Treatment:** The MRR is relatively low, peaking at ~0.53 at ROAR_exp budget = 0 and ROAR_att budget = 0. The lowest MRR (~0.44) is at ROAR_exp budget = 200 and ROAR_att budget = 4. * **(k) Targetted-Freebase:** The MRR is relatively low, peaking at ~0.39 at ROAR_exp budget = 0 and ROAR_att budget = 0. The lowest MRR (~0.10) is at ROAR_exp budget = 200 and ROAR_att budget = 4. * **(l) Targetted-WordNet:** The MRR is relatively low, peaking at ~0.71 at ROAR_exp budget = 0 and ROAR_att budget = 0. The lowest MRR (~0.35) is at ROAR_exp budget = 200 and ROAR_att budget = 4. ### Key Observations * For the "Backdoor" series, increasing the ROAR_exp budget generally improves MRR, suggesting that more exploration helps. * The "Targetted" series generally shows higher MRR values at lower budgets, and a decrease in MRR as budgets increase. * "Backdoor-WordNet" and "Targetted-WordNet" consistently show higher MRR values compared to other strategies. * "Backdoor-Mitigation", "Backdoor-Diagnosis", and "Backdoor-Treatment" have relatively low MRR values across all budget combinations. ### Interpretation The heatmaps demonstrate the effectiveness of different backdoor attack and defense strategies under varying resource constraints (ROAR_exp and ROAR_att budgets). The MRR score serves as a proxy for the success of the attack or defense. The positive correlation between ROAR_exp budget and MRR in the "Backdoor" series suggests that increased exploration of the model's vulnerabilities leads to more successful attacks. Conversely, the negative correlation in the "Targetted" series indicates that focused attacks may become less effective as more resources are allocated to both exploration and attack. The consistently high performance of "WordNet" strategies suggests that this approach is particularly robust or effective in exploiting/mitigating backdoor vulnerabilities. The lower performance of "Mitigation" and "Diagnosis" strategies indicates that these defenses may be less effective in practice, or require significantly more resources to achieve comparable results. The differences between the "Backdoor" and "Targetted" series highlight the importance of considering the attack strategy when evaluating defense mechanisms. A defense that is effective against general backdoor attacks may not be as effective against targeted attacks, and vice versa. The data suggests a trade-off between exploration and attack budgets, and the optimal strategy may depend on the specific context and available resources.