AI Red Teaming

AI risk management is the structured practice of identifying, assessing, and treating risks arising from the development, deployment, and operation of AI systems. It applies the discipline of risk management to a class of systems that behave differently from conventional software in ways that matter for risk: AI systems are probabilistic, opaque, capable of being manipulated through their inputs, and able to act autonomously, amplifying harm faster than a human operator can intervene.

Conventional risk management asks what can go wrong and how likely it is. AI risk management asks the same question against a system whose failure modes are harder to enumerate, harder to detect, and harder to attribute than those of the systems most risk frameworks were built to address.

How AI Red Teaming Differs From Conventional Red Teaming

Conventional red teams test whether an attacker can gain access to a system. AI red teams test whether an attacker can get a system to act against its own constraints.

The tools are different. The methodology is different. An AI red teamer does not run a port scan. They craft prompts, construct adversarial inputs, embed instructions in documents, and chain multi-turn conversations designed to erode the model’s behavioral guardrails incrementally.

The failure modes are also different. A conventional system either grants access or it does not. An AI system exists on a spectrum of compliance, and the goal of AI red teaming is to map where on that spectrum the model breaks and under what conditions.

What AI Red Teaming Tests

Prompt injection resistance is the foundational test for any LLM deployment. The red team attempts to override the system prompt through direct user input and through indirect injection embedded in retrieved documents, tool outputs, and external content that the model processes. Success is when the model acts on injected instructions. Failure is partial: how far did the injection get, and what did it enable?

Jailbreaking and safety-bypass probes whether the model’s refusal behavior holds up under adversarial pressure. Red teams use role-play framing, hypothetical scenarios, encoded inputs, persona manipulation, and multi-turn escalation to test whether safety measures are robust or brittle under creative attack.

Excessive agency exploitation applies specifically to agentic systems. The red team attempts to cause the model to take unintended real-world actions through prompt injection or adversarial inputs, testing whether privilege constraints, action approval mechanisms, and human-in-the-loop controls hold under adversarial conditions.

Data extraction attempts to surface sensitive information from the model’s training data, system prompt, conversation history, or retrieved context. This includes testing whether the model can be caused to reproduce memorized training content, reveal confidential instructions, or leak data from documents it processed earlier in the session.

Multi-agent attack chaining tests agentic architectures where multiple AI components interact. A red team maps how a compromise at one point propagates through the system, whether a prompt injection against an upstream retrieval component can cascade into harmful actions by a downstream execution agent.