## Bar Chart: Attack Success Rate (ASR) Comparison Across Adversarial Attacks and Defenses
### Overview
The chart compares the effectiveness of five defense mechanisms (Azure Prompt Shield, Protect AI v1, Meta Prompt Guard, Vijil Prompt Injection, NeMo Guard Jailbreak Detect) against eight adversarial attack methods (BAE, Bert-Attack, Deep Word Bug, Alzantot, Pruthi, PWWS, TextBugger, TextFooler). Attack Success Rate (ASR) is measured on a scale from 0.0 to 1.0, with higher values indicating greater vulnerability to attacks.
### Components/Axes
- **X-axis**: Adversarial attack methods (BAE, Bert-Attack, Deep Word Bug, Alzantot, Pruthi, PWWS, TextBugger, TextFooler).
- **Y-axis**: Attack Success Rate (ASR) from 0.0 to 1.0.
- **Legend**:
- Teal: Azure Prompt Shield
- Blue: Protect AI v1
- Green: Meta Prompt Guard
- Yellow: Vijil Prompt Injection
- Brown: NeMo Guard Jailbreak Detect
### Detailed Analysis
1. **BAE**:
- Azure Prompt Shield: ~0.1
- Protect AI v1: ~0.2
- Meta Prompt Guard: ~0.05
- Vijil Prompt Injection: ~0.25
- NeMo Guard Jailbreak Detect: ~0.85
2. **Bert-Attack**:
- Azure Prompt Shield: ~0.12
- Protect AI v1: ~0.25
- Meta Prompt Guard: ~0.08
- Vijil Prompt Injection: ~0.22
- NeMo Guard Jailbreak Detect: ~0.5
3. **Deep Word Bug**:
- Azure Prompt Shield: ~0.15
- Protect AI v1: ~0.22
- Meta Prompt Guard: ~0.18
- Vijil Prompt Injection: ~0.28
- NeMo Guard Jailbreak Detect: ~0.95
4. **Alzantot**:
- Azure Prompt Shield: ~0.12
- Protect AI v1: ~0.15
- Meta Prompt Guard: ~0.07
- Vijil Prompt Injection: ~0.05
- NeMo Guard Jailbreak Detect: ~0.55
5. **Pruthi**:
- Azure Prompt Shield: ~0.1
- Protect AI v1: ~0.1
- Meta Prompt Guard: ~0.03
- Vijil Prompt Injection: ~0.05
- NeMo Guard Jailbreak Detect: ~0.55
6. **PWWS**:
- Azure Prompt Shield: ~0.15
- Protect AI v1: ~0.3
- Meta Prompt Guard: ~0.2
- Vijil Prompt Injection: ~0.45
- NeMo Guard Jailbreak Detect: ~0.65
7. **TextBugger**:
- Azure Prompt Shield: ~0.1
- Protect AI v1: ~0.3
- Meta Prompt Guard: ~0.2
- Vijil Prompt Injection: ~0.7
- NeMo Guard Jailbreak Detect: ~0.9
8. **TextFooler**:
- Azure Prompt Shield: ~0.1
- Protect AI v1: ~0.3
- Meta Prompt Guard: ~0.3
- Vijil Prompt Injection: ~0.75
- NeMo Guard Jailbreak Detect: ~0.9
### Key Observations
- **NeMo Guard Jailbreak Detect** consistently shows the highest ASR across most attacks (e.g., 0.95 for Deep Word Bug, 0.9 for TextBugger), indicating it is the least effective defense.
- **Vijil Prompt Injection** often has the lowest ASR (e.g., 0.05 for Alzantot, 0.25 for BAE), suggesting it is the most effective defense.
- **Azure Prompt Shield** and **Protect AI v1** generally have moderate ASR values (0.1–0.3), indicating partial effectiveness.
- **Meta Prompt Guard** performs variably, with ASR ranging from 0.03 (Pruthi) to 0.3 (TextFooler).
### Interpretation
The data highlights significant disparities in defense effectiveness. **NeMo Guard Jailbreak Detect** fails to mitigate attacks effectively, allowing near-complete success rates in some cases (e.g., 0.95 for Deep Word Bug). In contrast, **Vijil Prompt Injection** demonstrates robust performance, reducing ASR to near-zero levels for certain attacks (e.g., 0.05 for Alzantot). This suggests that prompt-based defenses like Vijil and Azure are more reliable than jailbreak detection systems. The consistency of high ASR for NeMo across attacks implies systemic vulnerabilities in its design, warranting further investigation into its failure modes.
