AI Threat Modeling
AI threat modeling is the structured process of identifying what can go wrong in an AI system, who might cause it, and what the consequences would be. It applies the discipline of threat modeling to systems whose attack surface includes the model itself, the data it was trained on, the inputs it receives, and the actions it can take.
Conventional threat modeling asks where an attacker can get in. AI threat modeling asks that question and adds a second: where can the system be made to work against itself?
How AI Threat Modeling Extends Conventional Practice
Traditional frameworks like STRIDE map threats to components: spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. Those categories still apply to the infrastructure around an AI system. They do not cover what is unique to it.
AI systems introduce threat categories that have no direct equivalent in conventional software. A model can be manipulated through the content it processes rather than through the interfaces it exposes. Its behavior can be altered during training by poisoned data. Its outputs can be weaponized by downstream systems that trust them. Its autonomy can be turned against the users it was built to serve.
An AI threat model must account for all of these and map them to the system’s specific architecture: where does external content enter the context window, what tools and permissions does the agent hold, what data was used to train or fine-tune the model, and what downstream systems act on its output without human review.
Applying AI Threat Modeling in Practice
Map the trust boundaries. Every point at which the model receives input from an untrusted source is a potential prompt-injection entry point. Every point where model output is sent to a downstream system without validation is an insecure output-handling risk. Explicitly drawing these boundaries is the first step.
Enumerate the action space. For agentic systems, document every tool, permission, and external integration that the model can access. The threat model for an agent that can send email, execute code, and modify files is fundamentally different from one that can only return text. Excessive agency threats scale directly with the items on this list.
Assess the training pipeline. If the model was fine-tuned on internal data, that pipeline is part of the attack surface. Who controls the training data? Where does it come from? Could it be tampered with before training begins?
