## Bar Chart: Attack Success Rate (ASR) by Model and Attack Type
### Overview
This image displays a grid of 12 grouped bar charts, each representing a specific type of prompt injection or jailbreak attack. The charts compare the "Attack Success Rate (ASR)" across five different security models. The Y-axis represents the ASR, ranging from 0.0 (no attacks succeeded) to 1.0 (all attacks succeeded). A higher bar indicates that the security model failed to block the attack.
### Components/Axes
* **Y-Axis:** "Attack Success Rate (ASR)", scaled from 0.0 to 1.0.
* **X-Axis (Implicit):** The five security models, represented by color-coded bars in each sub-chart.
* **Legend (Bottom):**
* **Teal:** Azure Prompt Shield
* **Blue:** Protect AI v1
* **Lime Green:** Meta Prompt Guard
* **Yellow:** Vijil Prompt Injection
* **Tan:** NeMo Guard Jailbreak Detect
* **Categories (Sub-charts):**
* Row 1: Deletion Characters, Diacritics, Emoji Smuggling, Full Width Text, Homoglyphs, Numbers
* Row 2: Bidirectional Text, Spaces, Underline Accent Marks, Unicode Tags Smuggling, Upside Down Text, Zero Width
### Detailed Analysis
The following values are approximate visual estimates based on the provided chart.
| Attack Category | Azure (Teal) | Protect AI (Blue) | Meta (Lime) | Vijil (Yellow) | NeMo (Tan) |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Deletion Characters** | ~0.05 | ~0.01 | ~0.01 | ~0.01 | ~0.30 |
| **Diacritics** | ~0.70 | ~0.01 | ~0.60 | ~1.00 | ~0.10 |
| **Emoji Smuggling** | ~1.00 | ~1.00 | ~1.00 | ~1.00 | ~1.00 |
| **Full Width Text** | ~0.15 | ~0.01 | ~0.01 | ~1.00 | ~1.00 |
| **Homoglyphs** | ~1.00 | ~0.01 | ~0.50 | ~1.00 | ~1.00 |
| **Numbers** | ~1.00 | ~0.70 | ~1.00 | ~1.00 | ~1.00 |
| **Bidirectional Text** | ~1.00 | ~0.95 | ~1.00 | ~1.00 | ~1.00 |
| **Spaces** | ~0.10 | ~0.20 | ~1.00 | ~1.00 | ~1.00 |
| **Underline Accent Marks** | ~1.00 | ~0.01 | ~0.65 | ~1.00 | ~0.10 |
| **Unicode Tags Smuggling** | ~0.10 | ~1.00 | ~1.00 | ~1.00 | ~1.00 |
| **Upside Down Text** | ~1.00 | ~1.00 | ~1.00 | ~1.00 | ~1.00 |
| **Zero Width** | ~0.05 | ~0.20 | ~1.00 | ~1.00 | ~0.10 |
### Key Observations
* **Universal Failures:** "Emoji Smuggling" and "Upside Down Text" show an ASR of ~1.0 across all five models, suggesting these are highly effective attack vectors that none of the tested models currently mitigate.
* **Model Performance:**
* **Vijil Prompt Injection (Yellow):** Consistently shows an ASR of ~1.0 across almost every category, indicating it is the least effective at blocking these specific attack types.
* **NeMo Guard Jailbreak Detect (Tan):** Shows high ASR (~1.0) in most categories, though it performs better (lower ASR) on "Deletion Characters," "Diacritics," "Underline Accent Marks," and "Zero Width."
* **Protect AI v1 (Blue):** Shows the lowest ASR (best performance) in categories like "Deletion Characters," "Diacritics," "Full Width Text," "Homoglyphs," and "Underline Accent Marks."
* **Category Sensitivity:** Models show widely varying performance depending on the attack type. For example, "Azure Prompt Shield" performs well against "Deletion Characters" and "Zero Width" but fails completely against "Homoglyphs" and "Upside Down Text."
### Interpretation
The data demonstrates that current prompt security guardrails are highly susceptible to obfuscation-based attacks. The fact that "Emoji Smuggling" and "Upside Down Text" result in a 100% success rate for all models suggests that these models likely operate on tokenization or semantic analysis that is easily bypassed by character-level transformations.
The variance in performance across models indicates that there is no "silver bullet" solution; different models have different blind spots. For instance, Protect AI v1 appears more robust against character-level manipulation (Homoglyphs, Full Width) than the others, while Azure Prompt Shield is more effective against structural manipulation (Deletion, Zero Width). This suggests that a layered defense strategy, combining multiple guardrails, might be necessary to cover the diverse range of attack vectors shown.
