## Bar Chart: Attack Success Rate vs. Ablating Head Numbers ### Overview The image presents two bar charts comparing the attack success rate (ASR) of different models (Llama-2-7b-chat-hf and Vicuna-7b-v1.5) against various attack methods as the number of ablating heads increases. The x-axis represents the number of ablating heads (0 to 5), and the y-axis represents the attack success rate (ASR) from 0.0 to 1.0. The charts include data for "Advbench", "Jailbreakbench", and "Malicious Instruct" attacks, each tested with "use-tem" and "direct" methods. Additionally, "Vanilla Average" and "Use-tem Average" are plotted as lines across the bars. ### Components/Axes * **Titles:** * Top Chart: "Llama-2-7b-chat-hf" * Bottom Chart: "Vicuna-7b-v1.5" * **X-Axis:** "Ablating Head Numbers" with markers at 0, 1, 2, 3, 4, and 5. * **Y-Axis:** "Attack Success Rate (ASR))" with a scale from 0.0 to 1.0 in increments of 0.2. * **Legend (Top-Left):** * Red: "Advbench (use-tem)" * Yellow: "Jailbreakbench (use-tem)" * Teal: "Malicious Instruct (use-tem)" * Red with diagonal lines: "Advbench (direct)" * Yellow with diagonal lines: "Jailbreakbench (direct)" * Teal with diagonal lines: "Malicious Instruct (direct)" * **Legend (Top-Right):** * Pink Line: "Vanilla Average" * Purple Line: "Use-tem Average" ### Detailed Analysis #### Llama-2-7b-chat-hf (Top Chart) * **Advbench (use-tem) (Red):** The ASR starts at approximately 0.17 at 0 ablating heads, increases to ~0.62 at 1, remains around ~0.63 at 2, decreases to ~0.42 at 3, increases to ~0.65 at 4, and increases again to ~0.70 at 5. * **Jailbreakbench (use-tem) (Yellow):** The ASR starts at approximately 0.23 at 0 ablating heads, increases to ~0.63 at 1, remains around ~0.64 at 2, increases to ~0.68 at 3, remains around ~0.68 at 4, and decreases to ~0.60 at 5. * **Malicious Instruct (use-tem) (Teal):** The ASR starts at approximately 0.19 at 0 ablating heads, increases to ~0.58 at 1, decreases to ~0.48 at 2, decreases to ~0.35 at 3, increases to ~0.50 at 4, and increases again to ~0.53 at 5. * **Advbench (direct) (Red with diagonal lines):** The ASR starts at approximately 0.02 at 0 ablating heads, increases to ~0.60 at 1, increases to ~0.63 at 2, decreases to ~0.40 at 3, increases to ~0.58 at 4, and increases again to ~0.60 at 5. * **Jailbreakbench (direct) (Yellow with diagonal lines):** The ASR starts at approximately 0.05 at 0 ablating heads, increases to ~0.52 at 1, increases to ~0.53 at 2, decreases to ~0.45 at 3, increases to ~0.55 at 4, and increases again to ~0.62 at 5. * **Malicious Instruct (direct) (Teal with diagonal lines):** The ASR starts at approximately 0.04 at 0 ablating heads, increases to ~0.38 at 1, decreases to ~0.30 at 2, decreases to ~0.25 at 3, increases to ~0.35 at 4, and increases again to ~0.53 at 5. * **Vanilla Average (Pink Line):** The average ASR starts at approximately 0.20 at 0 ablating heads, increases to ~0.60 at 1, remains around ~0.62 at 2, decreases to ~0.45 at 3, increases to ~0.63 at 4, and increases again to ~0.75 at 5. * **Use-tem Average (Purple Line):** The average ASR starts at approximately 0.02 at 0 ablating heads, increases to ~0.65 at 1, decreases to ~0.50 at 2, decreases to ~0.40 at 3, increases to ~0.48 at 4, and increases again to ~0.58 at 5. #### Vicuna-7b-v1.5 (Bottom Chart) * **Advbench (use-tem) (Red):** The ASR starts at approximately 0.63 at 0 ablating heads, increases to ~0.70 at 1, decreases to ~0.60 at 2, increases to ~0.72 at 3, remains around ~0.70 at 4, and decreases to ~0.68 at 5. * **Jailbreakbench (use-tem) (Yellow):** The ASR starts at approximately 0.65 at 0 ablating heads, increases to ~0.78 at 1, remains around ~0.78 at 2, remains around ~0.72 at 3, remains around ~0.70 at 4, and decreases to ~0.65 at 5. * **Malicious Instruct (use-tem) (Teal):** The ASR starts at approximately 0.45 at 0 ablating heads, increases to ~0.58 at 1, increases to ~0.68 at 2, decreases to ~0.58 at 3, increases to ~0.68 at 4, and increases again to ~0.72 at 5. * **Advbench (direct) (Red with diagonal lines):** The ASR starts at approximately 0.20 at 0 ablating heads, increases to ~0.55 at 1, increases to ~0.50 at 2, decreases to ~0.40 at 3, increases to ~0.55 at 4, and increases again to ~0.58 at 5. * **Jailbreakbench (direct) (Yellow with diagonal lines):** The ASR starts at approximately 0.30 at 0 ablating heads, increases to ~0.78 at 1, remains around ~0.78 at 2, remains around ~0.72 at 3, remains around ~0.70 at 4, and decreases to ~0.65 at 5. * **Malicious Instruct (direct) (Teal with diagonal lines):** The ASR starts at approximately 0.40 at 0 ablating heads, increases to ~0.78 at 1, remains around ~0.60 at 2, remains around ~0.60 at 3, increases to ~0.78 at 4, and increases again to ~0.78 at 5. * **Vanilla Average (Pink Line):** The average ASR starts at approximately 0.58 at 0 ablating heads, increases to ~0.70 at 1, remains around ~0.78 at 2, remains around ~0.72 at 3, remains around ~0.70 at 4, and remains around ~0.72 at 5. * **Use-tem Average (Purple Line):** The average ASR starts at approximately 0.25 at 0 ablating heads, increases to ~0.68 at 1, remains around ~0.60 at 2, remains around ~0.60 at 3, remains around ~0.58 at 4, and remains around ~0.58 at 5. ### Key Observations * For both models, the "Vanilla Average" generally increases with the number of ablating heads. * The "Use-tem Average" shows more fluctuation compared to the "Vanilla Average." * The "direct" methods generally have lower ASR compared to the "use-tem" methods, especially at 0 ablating heads. * Vicuna-7b-v1.5 generally has higher ASR values compared to Llama-2-7b-chat-hf. ### Interpretation The charts illustrate how ablating head numbers affect the attack success rate (ASR) of different models under various attack methods. The "Vanilla Average" likely represents the baseline ASR without any specific defense mechanisms, while "Use-tem Average" represents the ASR when using a specific defense or mitigation technique ("use-tem"). The higher ASR values for Vicuna-7b-v1.5 suggest it might be more vulnerable to these attacks compared to Llama-2-7b-chat-hf. The lower ASR for "direct" methods indicates that these attacks are less effective without the "use-tem" technique. The fluctuations in ASR with increasing ablating head numbers suggest that the model's vulnerability changes as its architecture is modified. The increase in "Vanilla Average" with ablating heads could indicate that removing heads makes the model more susceptible to attacks, while the fluctuations in "Use-tem Average" suggest that the effectiveness of the "use-tem" defense varies with the model's architecture.