Hook
The difference between a secure agent and a vulnerable one is not the model. It is the orchestration framework. A 2.6x variance in Attack Success Rate between CrewAI (11.9%) and SmolAgents (31.1%) is not a statistical anomaly. It is a structural indictment of an industry that has been obsessing over model-level safety while ignoring the architecture that connects the model to the world. The SADF research, presented at DEF CON 34 AI Village, does not just add a new metric to the security discourse. It redefines the unit of analysis. The ledger does not lie, only the interpreters do. The numbers are unambiguous: the orchestration framework is the attack surface, and we have been auditing the wrong layer.
Context
Systematic Analysis of Agentic Frameworks (SADF) is a research project led by Julie Brunias and her team. It targets the blind spot in current AI security evaluations: the assumption that a model’s safety properties translate directly to the agent built on top of it. The methodology is elegantly simple and brutally effective. Fix the model (Claude Sonnet 4.0), fix the environment (SimulatedToolEnvironment), and vary only the orchestration framework. The control group is a Direct API call with no framework. The five frameworks tested are CrewAI, LangChain, AutoGen, SmolAgents, and one unnamed variant. The experiment uses 32 adversarial payloads across 5,119 evaluation lines, targeting eight distinct failure modes: Tool Call Hijacking, Output Poisoning, Cross-Tool Injection, Memory Poisoning, RAG Poisoning, Delegated Authority Abuse, Multi-Agent Propagation, and Context Boundary Violation. The key metric is the Attack Success Rate (ACR), corrected for the well-known overestimation bias of naive substring matching. The refusal-filtered scoring reduces the reported ACR for Claude Sonnet from a grossly inflated 60-70% to a sobering 15.5% for Direct API, 11.9% for CrewAI, 18.1% for LangChain, 20.0% for AutoGen, and 31.1% for SmolAgents. This is not a marginal difference. It is a structural gap. The research is currently in the POC stage, published on arXiv and supplemented by a superseded folder that transparently documents an earlier, less reliable version.
Core
The SADF findings are a systematic teardown of the assumption that “model safety equals system safety.” The core insight is that the orchestration layer introduces a new class of vulnerabilities that are not present in the model alone. The 8 failure modes are not theoretical. They are empirically measured. Consider Tool Call Hijacking: an attacker crafts a payload that intercepts the model’s tool invocation and redirects it to a malicious endpoint. In a Direct API setup, the model has no tool access, so the risk is zero. But in a framework like SmolAgents, which provides rich tool integration, the ACR for this specific failure mode is significantly higher. The same applies to Context Boundary Violation, where SmolAgents shows a staggering 64% ACR. The framework’s architecture—its message passing, memory management, and delegation logic—creates channels for attack that do not exist in a stateless API call. The refusal-filtered correction is a critical methodological contribution. The research found that naive substring matching overestimates Claude Sonnet’s ACR by 4 to 6 times. This is because the model often refuses to answer but the substring match still flags a partial match. By filtering out refusals, the true ACR drops to 15.5% for Direct API. This correction alone is a major contribution to the field. But the real story is the variance across frameworks. CrewAI’s discrete task isolation architecture gives it the lowest ACR at 11.9%, even lower than the Direct API baseline. This suggests that well-designed framework abstractions can actually reduce the attack surface compared to raw API calls. LangChain and AutoGen cluster around 18-20%, while SmolAgents more than doubles the baseline. The data reveals that the framework’s design—its compartmentalization, its permission models, its memory handling—directly determines the attack surface. The research also provides a taxonomy of 8 failure modes that serves as a shared vocabulary for the industry. This is not a minor contribution. Without a common language, findings are fragmented and non-reproducible. The 8 modes are clearly defined and measurable. They cover the full spectrum of agent interactions: tool misuse, data poisoning, memory corruption, and authority delegation. The research also includes a simulated environment that isolates the test from real-world systems, balancing ethical concerns with experimental rigor. This is a standard practice in security research, but it is executed cleanly. The CVE evidence is the external validation. The Azure SRE Agent (CVE-2026-62830) and Langflow (CVE-2026-9198) are real-world examples of framework-level vulnerabilities that match the SADF failure modes. This is not a laboratory exercise. The attack surface is already being exploited. The SADF research provides a systematic framework to measure and mitigate these risks.
Contrarian
The SADF research is rigorous, but it is not the final word. The bulls—those who will dismiss the findings as overblown—have some valid points. First, the test environment is a simulation. The SimulatedToolEnvironment does not have real permissions, real response times, or real concurrent user interactions. In a real deployment, the attack surface may be larger or smaller. The 32 payloads are a curated set, not a comprehensive adversarial library. Rare but high-impact attack paths may be missing. The research does not test the interaction between model and framework. The same framework may have a different ACR when paired with GPT-5.4 or DeepSeek. The “model × framework” interaction effect is unknown. The research uses default configurations for each framework. But in production, developers often customize these settings. A framework with a high ACR in default mode may be hardened with proper configuration. The research also does not test the entire lifecycle of an agent. It focuses on injection and poisoning at runtime, but not on supply chain attacks or update mechanisms. The superseded folder is a sign of transparency, but it is also a signal that the early version had reliability issues. The research claims to cover 8 architectures but only provides detailed ACR data for 5. The remaining 3 may have incomplete data. These are not fatal flaws, but they are real limitations. The contrarian view is that the SADF research is a valuable starting point, not a definitive benchmark. The bulls are right to demand more data, more models, more payloads, and more real-world validation. But they are wrong to dismiss the core finding. The variance is real. The framework matters. The industry has been ignoring this dimension. The SADF research is the first systematic attempt to measure it. The cold truth is that the data is robust enough to inform decisions today. If you are building an agent for a DeFi trading bot, the choice between CrewAI and SmolAgents is not just a matter of developer convenience. It is a 2.6x security difference. The ledger does not lie.
Takeaway
The SADF research is a cold wake-up call for the AI security industry. The unit of analysis must shift from the model to the model+framework+tool triad. This is not a marginal refinement. It is a paradigm shift. For the crypto industry, where automated agents execute financial transactions, the implications are direct. Many DeFi protocols rely on agents built with LangChain or AutoGen. The CVE evidence shows that these frameworks are already being exploited. The next step is to integrate framework-level security audits into the CI/CD pipeline. The compliance checklist must include a framework ACR score. The question is not whether your model is safe. The question is whether your orchestration framework is a liability. Based on my audit experience with the 0x Protocol, I know that the most dangerous vulnerabilities are the ones that the industry has not yet learned to look for. The SADF research has identified the blind spot. The responsibility now lies with the builders and the auditors. Code is law; intent is irrelevant. The framework’s architecture determines its security, not the developer’s intention. History repeats, but the gas fees change. The cost of ignoring this finding is not just a theoretical loss. It is a real, measurable risk to assets. The question is not whether you trust the framework. The question is whether you have audited the framework. Trust is a bug, not a feature. The SADF research has provided the data. The next step is to act.
