deep-xpia
Prompt injection benchmark for multi-agent AI systems
Last verified:
What is deep-xpia?
Deep-XPIA is a multi-hop cross-prompt injection benchmark designed specifically for evaluating multi-agent AI systems. It provides 250 attack cases organized across 7 taxonomy categories, with 4 different defenses evaluated to measure security robustness against prompt injection vulnerabilities.
The benchmark maps eight distinct attack patterns (DXPIA-001 through DXPIA-008) and measures how prompt injections propagate across agent chains in multi-agent systems. It addresses the critical security gap where most safety research has focused on single-agent LLMs while multi-agent systems face compounding risks where prompt injections can spread across interconnected agents.
Key features include metrics for Chain Propagation Depth (CPD) - the maximum number of agent hops a successful injection can traverse - and other red teaming metrics including IRR, CPD, Safety Classifier FNR, and UAR. The benchmark helps organizations identify systemic trust boundary failures when CPD exceeds 0.
Deep-XPIA is designed for AI security researchers, enterprise multi-agent system developers, red teamers, and compliance teams who need to evaluate and improve the security of their multi-agent AI deployments against cross-prompt injection attacks.
deep-xpia pricing
Pricing model: Freemium
Free and open source - available on GitHub at freyzo/deep-xpia with no paid tiers or licensing costs
deep-xpia pros
- 250 comprehensive attack cases for thorough evaluation
- 7 taxonomy categories covering diverse attack types
- Evaluates 4 different defense mechanisms
- Specifically designed for multi-agent AI systems
- Measures multi-hop injection propagation across agents
- Includes 8 distinct attack patterns (DXPIA-001 to DXPIA-008)
- Reduces attack success rate from 69% to 12%
- Provides Chain Propagation Depth (CPD) metric
- Open source and publicly available on GitHub
- Created by freyzo with active community presence
- Hit Hacker News indicating strong community recognition
- Includes IRR, CPD, Safety Classifier FNR, UAR metrics
- Targets CPD=0 as acceptance criterion for security
- Helps identify systemic trust boundary failures
- Enables regular red teaming with multi-hop scenarios
deep-xpia cons
- Focused only on multi-agent systems, not single-agent LLMs
- May not cover all emerging prompt injection variants
- Requires multi-agent infrastructure to test
- Limited to 8 attack patterns which may miss nuances
- No built-in automated testing interface mentioned
- Defense evaluation limited to only 4 defenses
- May require significant computational resources
- No enterprise licensing or support options
Frequently asked questions about deep-xpia
What is Deep-XPIA?
Deep-XPIA is a multi-hop cross-prompt injection benchmark for multi-agent AI systems, providing 250 attack cases across 7 taxonomy categories with 4 defenses evaluated to measure security robustness.
How many attack cases does Deep-XPIA include?
The benchmark includes 250 attack cases organized across 7 taxonomy categories, covering 8 distinct attack patterns labeled DXPIA-001 through DXPIA-008.
What is Chain Propagation Depth (CPD)?
CPD is the maximum number of agent hops a successful injection can traverse through a multi-agent system. The target is CPD=0, and CPD>1 indicates systemic trust boundary failure.
Which defenses does Deep-XPIA evaluate?
Deep-XPIA evaluates 4 different defense mechanisms against prompt injection attacks, though the specific defenses are not detailed in available documentation.
What metrics does Deep-XPIA provide?
The benchmark provides four key metrics: IRR (Injection Propagation Rate), CPD (Chain Propagation Depth), Safety Classifier FNR (False Negative Rate), and UAR (Unknown Attack Rate) for red teaming acceptance criteria.
Is Deep-XPIA free to use?
Yes, Deep-XPIA is free and open source, available on GitHub at freyzo/deep-xpia with no licensing costs or paid tiers.
What attack patterns does Deep-XPIA cover?
The benchmark maps 8 attack patterns labeled DXPIA-001 through DXPIA-008, which it successfully reduces from 69% attack success rate to 12%.
Who should use Deep-XPIA?
Deep-XPIA is designed for AI security researchers, enterprise multi-agent system developers, red teamers running regular simulations, and compliance teams evaluating multi-agent AI security.
How effective is Deep-XPIA at reducing attack success?
The benchmark maps eight attack patterns from 69% attack success rate down to 12%, demonstrating significant effectiveness in identifying and mitigating prompt injection vulnerabilities.
Can Deep-XPIA test single-agent LLMs?
No, Deep-XPIA is specifically designed for multi-agent AI systems. It addresses the unique security risks where prompt injections propagate across agent chains, which is not relevant for single-agent systems.