## Bar Chart: Attack Success Rate (ASR) Comparison ### Overview The image is a bar chart comparing the attack success rate (ASR) of five different defense mechanisms against various types of adversarial attacks. The chart is divided into 12 subplots, each representing a different attack type. Each subplot contains five bars, each representing a different defense mechanism. The height of the bar indicates the attack success rate (ASR) for that defense mechanism against that attack type. ### Components/Axes * **Y-axis:** Attack Success Rate (ASR), ranging from 0.0 to 1.0 in increments of 0.2. * **X-axis:** Each subplot represents a different attack type. The attack types are: * Deletion Characters * Diacritics * Emoji Smuggling * Full Width Text * Homoglyphs * Numbers * Bidirectional Text * Spaces * Underline Accent Marks * Unicode Tags Smuggling * Upside Down Text * Zero Width * **Legend:** Located at the bottom of the chart, the legend identifies the defense mechanisms represented by each color: * Azure Prompt Shield (Teal) * Protect AI v1 (Blue-Gray) * Meta Prompt Guard (Yellow-Green) * Vijil Prompt Injection (Yellow) * NeMo Guard Jailbreak Detect (Tan) ### Detailed Analysis Here's a breakdown of the ASR for each attack type and defense mechanism: **Top Row:** * **Deletion Characters:** * Azure Prompt Shield: ~0.05 * Protect AI v1: ~0.01 * Meta Prompt Guard: ~0.7 * Vijil Prompt Injection: ~0.6 * NeMo Guard Jailbreak Detect: ~0.3 * **Diacritics:** * Azure Prompt Shield: ~0.7 * Protect AI v1: ~0.01 * Meta Prompt Guard: ~0.6 * Vijil Prompt Injection: ~1.0 * NeMo Guard Jailbreak Detect: ~0.1 * **Emoji Smuggling:** * Azure Prompt Shield: ~1.0 * Protect AI v1: ~1.0 * Meta Prompt Guard: ~1.0 * Vijil Prompt Injection: ~1.0 * NeMo Guard Jailbreak Detect: ~1.0 * **Full Width Text:** * Azure Prompt Shield: ~0.1 * Protect AI v1: ~0.01 * Meta Prompt Guard: ~0.9 * Vijil Prompt Injection: ~1.0 * NeMo Guard Jailbreak Detect: ~1.0 * **Homoglyphs:** * Azure Prompt Shield: ~1.0 * Protect AI v1: ~0.01 * Meta Prompt Guard: ~0.5 * Vijil Prompt Injection: ~1.0 * NeMo Guard Jailbreak Detect: ~1.0 * **Numbers:** * Azure Prompt Shield: ~0.8 * Protect AI v1: ~0.6 * Meta Prompt Guard: ~1.0 * Vijil Prompt Injection: ~1.0 * NeMo Guard Jailbreak Detect: ~0.9 **Bottom Row:** * **Bidirectional Text:** * Azure Prompt Shield: ~1.0 * Protect AI v1: ~0.9 * Meta Prompt Guard: ~1.0 * Vijil Prompt Injection: ~1.0 * NeMo Guard Jailbreak Detect: ~1.0 * **Spaces:** * Azure Prompt Shield: ~0.1 * Protect AI v1: ~0.2 * Meta Prompt Guard: ~1.0 * Vijil Prompt Injection: ~1.0 * NeMo Guard Jailbreak Detect: ~1.0 * **Underline Accent Marks:** * Azure Prompt Shield: ~1.0 * Protect AI v1: ~1.0 * Meta Prompt Guard: ~0.7 * Vijil Prompt Injection: ~1.0 * NeMo Guard Jailbreak Detect: ~0.1 * **Unicode Tags Smuggling:** * Azure Prompt Shield: ~0.1 * Protect AI v1: ~1.0 * Meta Prompt Guard: ~1.0 * Vijil Prompt Injection: ~1.0 * NeMo Guard Jailbreak Detect: ~1.0 * **Upside Down Text:** * Azure Prompt Shield: ~1.0 * Protect AI v1: ~1.0 * Meta Prompt Guard: ~1.0 * Vijil Prompt Injection: ~1.0 * NeMo Guard Jailbreak Detect: ~1.0 * **Zero Width:** * Azure Prompt Shield: ~0.1 * Protect AI v1: ~0.2 * Meta Prompt Guard: ~1.0 * Vijil Prompt Injection: ~1.0 * NeMo Guard Jailbreak Detect: ~0.1 ### Key Observations * **High Success Rates:** Several attacks, such as Emoji Smuggling, Bidirectional Text, and Upside Down Text, consistently achieve high success rates (ASR close to 1.0) across all defense mechanisms. * **Variable Performance:** The performance of defense mechanisms varies significantly depending on the attack type. For example, Azure Prompt Shield performs poorly against Deletion Characters and Spaces but well against Emoji Smuggling and Bidirectional Text. * **Protect AI v1 Weakness:** Protect AI v1 consistently shows very low ASR against Diacritics, Full Width Text, and Homoglyphs. * **Meta Prompt Guard and Vijil Prompt Injection Strength:** Meta Prompt Guard and Vijil Prompt Injection generally exhibit high ASR across most attack types, indicating their effectiveness. * **NeMo Guard Jailbreak Detect Weakness:** NeMo Guard Jailbreak Detect shows low ASR against Diacritics, Underline Accent Marks, and Zero Width attacks. ### Interpretation The data suggests that the effectiveness of different defense mechanisms against adversarial attacks is highly dependent on the specific attack type. Some defense mechanisms, like Meta Prompt Guard and Vijil Prompt Injection, appear to be more robust across a wider range of attacks, while others, like Protect AI v1 and NeMo Guard Jailbreak Detect, have specific weaknesses. The high success rates for certain attacks across all defenses indicate that these attacks may be inherently difficult to defend against. The variability in performance highlights the need for a multi-layered defense strategy that incorporates different techniques to address various attack vectors.

## Bar Chart: Attack Success Rate (ASR) by Prompt Injection Technique and Defense Mechanism ### Overview This bar chart compares the Attack Success Rate (ASR) of various prompt injection techniques against different defense mechanisms. The chart is divided into two rows, each representing a set of prompt injection techniques. Each bar represents the ASR for a specific technique and defense combination. ### Components/Axes * **X-axis:** Prompt Injection Techniques: Deletion Characters, Diacritics, Emoji Smuggling, Full Width Text, Homoglyphs, Numbers, Bidirectional Text, Spaces, Underline Accent Marks, Unicode Tags Smuggling, Upside Down Text, Zero Width. * **Y-axis:** Attack Success Rate (ASR), ranging from 0.0 to 1.0. * **Legend:** * Azure Prompt Shield (Light Green) * Protect AI v1 (Pale Orange) * Meta Prompt Guard (Medium Green) * Vijil Prompt Injection (Blue) * NeMo Guard Jailbreak Detect (Coral) ### Detailed Analysis **Row 1: Deletion Characters to Zero Width** * **Deletion Characters:** * Azure Prompt Shield: ~0.04 * Protect AI v1: ~0.24 * Meta Prompt Guard: ~0.68 * Vijil Prompt Injection: ~0.84 * NeMo Guard Jailbreak Detect: ~0.88 * **Diacritics:** * Azure Prompt Shield: ~0.00 * Protect AI v1: ~0.04 * Meta Prompt Guard: ~0.08 * Vijil Prompt Injection: ~0.76 * NeMo Guard Jailbreak Detect: ~0.84 * **Emoji Smuggling:** * Azure Prompt Shield: ~0.00 * Protect AI v1: ~0.00 * Meta Prompt Guard: ~0.00 * Vijil Prompt Injection: ~0.88 * NeMo Guard Jailbreak Detect: ~0.92 * **Full Width Text:** * Azure Prompt Shield: ~0.00 * Protect AI v1: ~0.00 * Meta Prompt Guard: ~0.00 * Vijil Prompt Injection: ~0.84 * NeMo Guard Jailbreak Detect: ~0.88 * **Homoglyphs:** * Azure Prompt Shield: ~0.00 * Protect AI v1: ~0.00 * Meta Prompt Guard: ~0.00 * Vijil Prompt Injection: ~0.88 * NeMo Guard Jailbreak Detect: ~0.92 * **Numbers:** * Azure Prompt Shield: ~0.00 * Protect AI v1: ~0.00 * Meta Prompt Guard: ~0.00 * Vijil Prompt Injection: ~0.84 * NeMo Guard Jailbreak Detect: ~0.88 **Row 2: Bidirectional Text to Zero Width** * **Bidirectional Text:** * Azure Prompt Shield: ~0.00 * Protect AI v1: ~0.28 * Meta Prompt Guard: ~0.84 * Vijil Prompt Injection: ~0.88 * NeMo Guard Jailbreak Detect: ~0.92 * **Spaces:** * Azure Prompt Shield: ~0.00 * Protect AI v1: ~0.04 * Meta Prompt Guard: ~0.84 * Vijil Prompt Injection: ~0.84 * NeMo Guard Jailbreak Detect: ~0.88 * **Underline Accent Marks:** * Azure Prompt Shield: ~0.00 * Protect AI v1: ~0.00 * Meta Prompt Guard: ~0.88 * Vijil Prompt Injection: ~0.88 * NeMo Guard Jailbreak Detect: ~0.92 * **Unicode Tags Smuggling:** * Azure Prompt Shield: ~0.00 * Protect AI v1: ~0.00 * Meta Prompt Guard: ~0.84 * Vijil Prompt Injection: ~0.88 * NeMo Guard Jailbreak Detect: ~0.92 * **Upside Down Text:** * Azure Prompt Shield: ~0.00 * Protect AI v1: ~0.00 * Meta Prompt Guard: ~0.84 * Vijil Prompt Injection: ~0.88 * NeMo Guard Jailbreak Detect: ~0.92 * **Zero Width:** * Azure Prompt Shield: ~0.00 * Protect AI v1: ~0.20 * Meta Prompt Guard: ~0.84 * Vijil Prompt Injection: ~0.84 * NeMo Guard Jailbreak Detect: ~0.88 ### Key Observations * **Vijil Prompt Injection and NeMo Guard Jailbreak Detect** consistently exhibit the highest ASR across almost all prompt injection techniques, often nearing 1.0. * **Azure Prompt Shield** generally has the lowest ASR, often at or near 0.0, but is vulnerable to Deletion Characters and Bidirectional Text. * **Protect AI v1** shows moderate ASR for some techniques (Deletion Characters, Bidirectional Text, Zero Width) but is very effective against others. * **Meta Prompt Guard** demonstrates a relatively high ASR across most techniques, generally between 0.68 and 0.88. * Techniques like Emoji Smuggling, Full Width Text, Homoglyphs, and Numbers are particularly effective at bypassing defenses, resulting in high ASR values for most defense mechanisms. ### Interpretation The data suggests that while some defense mechanisms (like Azure Prompt Shield) are effective against a broad range of prompt injection techniques, no single defense provides complete protection. Vijil Prompt Injection and NeMo Guard Jailbreak Detect are highly effective at bypassing defenses, indicating a need for more robust protection strategies. The varying effectiveness of different techniques highlights the complexity of prompt injection attacks and the need for layered defense approaches. The consistent high ASR for techniques like Emoji Smuggling suggests that these methods exploit vulnerabilities in how input is processed and sanitized. The relatively low ASR for Azure Prompt Shield against many techniques could indicate a more conservative approach to filtering, potentially at the cost of false positives. The chart demonstrates a clear trade-off between security and usability, as more aggressive filtering may block legitimate inputs. Further investigation is needed to understand the specific vulnerabilities exploited by each technique and to develop more effective countermeasures.

## Bar Chart: Attack Success Rates (ASR) of AI Security Tools Across Unicode-Based Attack Vectors ### Overview The image is a composite bar chart displaying the Attack Success Rate (ASR) for five different AI security/prompt injection detection tools across twelve distinct categories of Unicode-based text obfuscation attacks. The chart is organized into a 2x6 grid of subplots, each dedicated to a specific attack type. The overall purpose is to compare the effectiveness of the tools in mitigating these adversarial text manipulation techniques. ### Components/Axes * **Main Y-Axis (Left Side):** Labeled "Attack Success Rate (ASR)". The scale runs from 0.0 to 1.0, with major tick marks at 0.0, 0.2, 0.4, 0.6, 0.8, and 1.0. * **Subplot Titles (Attack Categories):** The twelve attack types, listed from left to right, top to bottom, are: 1. Deletion Characters 2. Diacritics 3. Emoji Smuggling 4. Full Width Text 5. Homoglyphs 6. Numbers 7. Bidirectional Text 8. Spaces 9. Underline Accent Marks 10. Unicode Tags Smuggling 11. Upside Down Text 12. Zero Width * **Legend (Bottom Center):** A horizontal legend identifies the five tools by color: * **Teal/Green:** Azure Prompt Shield * **Blue:** Protect AI v1 * **Light Green:** Meta Prompt Guard * **Yellow:** Vijil Prompt Injection * **Tan/Beige:** NeMo Guard Jailbreak Detect * **Data Representation:** Within each subplot, five vertical bars are displayed side-by-side, corresponding to the five tools in the order listed in the legend (left to right: Azure, Protect AI, Meta, Vijil, NeMo). ### Detailed Analysis Below is the approximate ASR (0.0 to 1.0) for each tool within each attack category. Values are estimated from bar height relative to the y-axis grid lines. **Top Row:** 1. **Deletion Characters:** * Azure: ~0.05 * Protect AI: ~0.01 (very low) * Meta: 0.0 * Vijil: 0.0 * NeMo: ~0.32 2. **Diacritics:** * Azure: ~0.70 * Protect AI: ~0.01 (very low) * Meta: ~0.60 * Vijil: 1.0 * NeMo: ~0.12 3. **Emoji Smuggling:** * Azure: 1.0 * Protect AI: 1.0 * Meta: 1.0 * Vijil: 1.0 * NeMo: 1.0 4. **Full Width Text:** * Azure: ~0.16 * Protect AI: ~0.01 (very low) * Meta: 0.0 * Vijil: 1.0 * NeMo: 1.0 5. **Homoglyphs:** * Azure: 1.0 * Protect AI: ~0.01 (very low) * Meta: ~0.52 * Vijil: 1.0 * NeMo: 1.0 6. **Numbers:** * Azure: 1.0 * Protect AI: ~0.73 * Meta: 1.0 * Vijil: 1.0 * NeMo: 1.0 **Bottom Row:** 7. **Bidirectional Text:** * Azure: 1.0 * Protect AI: ~0.96 * Meta: 1.0 * Vijil: 1.0 * NeMo: 1.0 8. **Spaces:** * Azure: ~0.12 * Protect AI: ~0.21 * Meta: 1.0 * Vijil: 1.0 * NeMo: 1.0 9. **Underline Accent Marks:** * Azure: 1.0 * Protect AI: 1.0 * Meta: ~0.66 * Vijil: 1.0 * NeMo: ~0.12 10. **Unicode Tags Smuggling:** * Azure: ~0.08 * Protect AI: 1.0 * Meta: 1.0 * Vijil: 1.0 * NeMo: 1.0 11. **Upside Down Text:** * Azure: 1.0 * Protect AI: 1.0 * Meta: 1.0 * Vijil: 1.0 * NeMo: 1.0 12. **Zero Width:** * Azure: ~0.07 * Protect AI: ~0.21 * Meta: 1.0 * Vijil: 1.0 * NeMo: ~0.12 ### Key Observations * **Universal Vulnerability:** Three attack types—**Emoji Smuggling**, **Bidirectional Text**, and **Upside Down Text**—achieved a perfect 1.0 ASR against all five tested tools, indicating a complete lack of detection for these methods. * **Tool-Specific Strengths:** * **Protect AI v1 (Blue)** shows strong performance (very low ASR) against **Deletion Characters**, **Diacritics**, **Full Width Text**, and **Homoglyphs**, but is highly vulnerable to **Unicode Tags Smuggling** and **Upside Down Text**. * **Azure Prompt Shield (Teal)** is effective against **Deletion Characters**, **Full Width Text**, **Unicode Tags Smuggling**, and **Zero Width** attacks, but fails against **Emoji Smuggling**, **Homoglyphs**, **Numbers**, and **Bidirectional Text**. * **NeMo Guard Jailbreak Detect (Tan)** has a mixed profile. It is often bypassed (ASR=1.0) but shows notable resilience against **Diacritics**, **Underline Accent Marks**, and **Zero Width** attacks (ASR ~0.12). * **Attack Effectiveness:** **Numbers** and **Spaces** attacks show varied success, being highly effective against most tools except for partial resistance from Protect AI v1 in the "Numbers" category. * **Consistent Failure:** **Meta Prompt Guard (Light Green)** and **Vijil Prompt Injection (Yellow)** have ASR values of 1.0 in the majority of categories, suggesting they may be less robust against this suite of Unicode obfuscation techniques compared to the other tools in this specific test. ### Interpretation This chart presents a benchmark evaluation of commercial and open-source AI security tools against sophisticated text-based adversarial attacks. The data suggests that **current defenses are highly inconsistent and often brittle** when faced with Unicode manipulation. The fact that entire classes of attacks (Emoji, Bidirectional, Upside Down) bypass all tools indicates a significant gap in the security model, likely stemming from a focus on lexical or semantic analysis while neglecting low-level Unicode encoding anomalies. The varying performance profiles (e.g., Protect AI vs. Azure) imply that different tools employ fundamentally different detection heuristics, with no single tool providing comprehensive coverage. From a security perspective, the results are concerning. They demonstrate that adversaries can likely evade detection by selecting the appropriate obfuscation technique from a known menu of options. The high ASR for "Numbers" and "Spaces" is particularly noteworthy, as these are common characters, suggesting that even simple normalization failures can be exploited. This benchmark underscores the need for more robust, encoding-aware preprocessing and a multi-layered defense strategy in AI systems handling untrusted text input.

## Bar Chart: Attack Success Rate (ASR) by Deletion Characters and Defense Methods ### Overview The chart compares the Attack Success Rate (ASR) of five defense methods across 10 categories of adversarial text manipulations. Each category represents a specific type of text corruption or obfuscation technique. The y-axis ranges from 0.0 to 1.0, representing the probability of successful attacks bypassing each defense. ### Components/Axes - **X-axis (Categories)**: - Bidirectional Text - Spaces - Emoji Smuggling - Full Width Text - Homoglyphs - Numbers - Underline Accent Marks - Unicode Tags Smuggling - Upside Down Text - Zero Width - **Y-axis (ASR)**: Attack Success Rate (0.0–1.0) - **Legend (Bottom)**: - Azure Prompt Shield (teal) - Protect AI v1 (blue) - Meta Prompt Guard (green) - Vijil Prompt Injection (orange) - NeMo Guard Jailbreak Detect (beige) ### Detailed Analysis 1. **Bidirectional Text**: - Azure Prompt Shield: ~0.95 - Protect AI v1: ~0.92 - Meta Prompt Guard: ~0.94 - Vijil Prompt Injection: ~0.93 - NeMo Guard Jailbreak Detect: ~0.91 2. **Spaces**: - Azure Prompt Shield: ~0.70 - Protect AI v1: ~0.65 - Meta Prompt Guard: ~0.58 - Vijil Prompt Injection: ~0.98 - NeMo Guard Jailbreak Detect: ~0.10 3. **Emoji Smuggling**: - Azure Prompt Shield: ~0.97 - Protect AI v1: ~0.95 - Meta Prompt Guard: ~0.96 - Vijil Prompt Injection: ~0.94 - NeMo Guard Jailbreak Detect: ~0.93 4. **Full Width Text**: - Azure Prompt Shield: ~0.15 - Protect AI v1: ~0.02 - Meta Prompt Guard: ~0.01 - Vijil Prompt Injection: ~0.99 - NeMo Guard Jailbreak Detect: ~0.98 5. **Homoglyphs**: - Azure Prompt Shield: ~0.96 - Protect AI v1: ~0.94 - Meta Prompt Guard: ~0.52 - Vijil Prompt Injection: ~0.97 - NeMo Guard Jailbreak Detect: ~0.95 6. **Numbers**: - Azure Prompt Shield: ~0.72 - Protect AI v1: ~0.70 - Meta Prompt Guard: ~0.75 - Vijil Prompt Injection: ~0.99 - NeMo Guard Jailbreak Detect: ~0.73 7. **Underline Accent Marks**: - Azure Prompt Shield: ~0.98 - Protect AI v1: ~0.96 - Meta Prompt Guard: ~0.65 - Vijil Prompt Injection: ~0.97 - NeMo Guard Jailbreak Detect: ~0.95 8. **Unicode Tags Smuggling**: - Azure Prompt Shield: ~0.10 - Protect AI v1: ~0.05 - Meta Prompt Guard: ~0.03 - Vijil Prompt Injection: ~0.99 - NeMo Guard Jailbreak Detect: ~0.97 9. **Upside Down Text**: - Azure Prompt Shield: ~0.99 - Protect AI v1: ~0.98 - Meta Prompt Guard: ~0.97 - Vijil Prompt Injection: ~0.96 - NeMo Guard Jailbreak Detect: ~0.95 10. **Zero Width**: - Azure Prompt Shield: ~0.05 - Protect AI v1: ~0.20 - Meta Prompt Guard: ~0.98 - Vijil Prompt Injection: ~0.99 - NeMo Guard Jailbreak Detect: ~0.12 ### Key Observations - **Azure Prompt Shield** consistently performs well across most categories, with ASR >0.9 in 8/10 cases. - **Vijil Prompt Injection** dominates in Full Width Text (ASR ~0.99) and Unicode Tags Smuggling (ASR ~0.99), suggesting specialized vulnerabilities. - **NeMo Guard Jailbreak Detect** shows the lowest performance in Spaces (ASR ~0.10) and Zero Width (ASR ~0.12), indicating potential weaknesses in handling spacing and invisible characters. - **Meta Prompt Guard** struggles with Homoglyphs (ASR ~0.52) and Underline Accent Marks (ASR ~0.65), suggesting challenges with character-level obfuscation. ### Interpretation The data reveals significant variability in defense effectiveness depending on the attack type. Azure Prompt Shield demonstrates broad robustness, while Vijil Prompt Injection excels at bypassing defenses in specific scenarios (e.g., Full Width Text). The poor performance of NeMo Guard Jailbreak Detect in Spaces and Zero Width suggests potential flaws in handling non-printing characters. These findings highlight the importance of context-aware defense strategies, as no single method achieves universal protection. The high ASR values across most categories (median ~0.95) indicate that current defenses may have systemic vulnerabilities to text-based adversarial attacks.