How to Red-Team AI Models Effectively (September 2026)

Most security teams that take their first pass at AI red teaming find out pretty quickly that their existing playbooks cover maybe 20% of what actually matters. The behavioral layer, where most of the real risk lives, is a different problem entirely. Getting that coverage right takes a specific methodology, and that's what we're going to walk through here.

TLDR:

• AI red teaming targets model behavior, not infrastructure; the same prompt can succeed 30% of the time and fail the rest, so there is no binary patched/unpatched state.
• Agentic systems introduce a qualitatively different risk: a successful attack produces an action that already executed against a real system before any human reviewed it.
• EU AI Act Article 15 requires documented adversarial robustness test evidence for high-risk systems before the August 2026 deadline; a policy statement does not satisfy the requirement.
• The hybrid model works best: humans drive discovery of novel attack chains while automated tools like PyRIT and Garak cover the volume that manual testing cannot reach.
• Openlayer sits downstream of red team engagements, converting confirmed findings into CI/CD-gated regression tests and blocking unsafe outputs at the API boundary before they reach production.

What is AI red teaming

AI red teaming is the practice of systematically attacking your own AI systems before someone else does. A red team takes the adversary's position and probes models, agents, RAG pipelines, and multimodal systems for failure modes that standard evaluation misses: inputs engineered to extract harmful outputs, prompts designed to override safety constraints, queries that cause a model to hallucinate with false confidence.

The two primary goals are distinct and worth keeping separate. The first is surfacing security vulnerabilities: prompt injections, jailbreaks, data extraction, model inversion. The second is identifying safety failures: biased outputs, harmful content generation, policy violations that a model produces not because it was attacked but because it was asked in a particular way. Security and safety testing share methods but answer different questions. Security asks whether the system can be exploited. Safety asks whether the system behaves acceptably under pressure.

Three modes of engagement fall under the red teaming umbrella, and conflating them leads to gaps in coverage:

• Adversarial simulation: a red team acts as a specific threat actor, running realistic attack sequences against the system as it would be targeted in production. The simulation has a scenario, a goal, and a defined adversary model.
• Adversarial testing: structured probing across a predefined attack taxonomy, systematic instead of scenario-driven. Coverage is measured against a test library, not a threat narrative.
• Capabilities testing: mapping what a model can do at its limits, including dangerous or dual-use capabilities it was not intended to expose. This is less about attacking the system than about mapping its ceiling.

Each system type presents a distinct attack surface. A standalone LLM is primarily vulnerable at the prompt layer. A RAG pipeline adds retrieval-layer risks: poisoned documents, indirect prompt injection through retrieved content, context window manipulation. An agentic system introduces tool call vulnerabilities, where a successful attack produces not a bad response but an unauthorized action executed against a real system.

How AI red teaming differs from traditional red teaming

Traditional penetration testing targets infrastructure: network boundaries, authentication flows, software vulnerabilities with known CVEs. Fix the vulnerability, deploy the patch, verify the fix holds. The attack surface is bounded and the remediation is deterministic.

AI red teaming operates on fundamentally different ground across five dimensions:

• Attack surface: The target is model behavior, not infrastructure. A jailbreak doesn't exploit a buffer overflow; it exploits the probability distribution the model learned during training.
• Vulnerability nature: AI failures are probabilistic. The same prompt may succeed 30% of the time and fail the rest. There is no binary "patched/unpatched" state.
• Remediation: Fixing a behavioral failure means retraining, fine-tuning, adjusting system prompts, or adding AI guardrails for LLMs, not shipping a hotfix. The model itself may need to change.
• Required expertise: Effective AI red teamers need ML literacy alongside security skills. Understanding why a model is susceptible to a particular attack requires knowing how it was trained, what its context window looks like, and how its output distribution responds to adversarial inputs.
• Testing cadence: A model's behavior can shift after fine-tuning, prompt updates, or retrieval corpus changes. Red teaming is continuous, not a one-time pre-launch exercise.

Traditional pentesters who haven't worked with generative AI will find their playbooks cover maybe 20% of the relevant attack surface. The behavioral layer, where most AI-specific risk actually lives, requires a different approach entirely.

Why AI red teaming has become a non-negotiable security control

In March 2026, the NIST CAISI red teaming competition drew over 400 participants who launched more than 250,000 attack attempts against 13 frontier models in agentic scenarios. Every single model had at least one successful attack confirmed against it. No model was categorically secure.

That finding landed against an already-accelerating regulatory backdrop. EU AI Act Article 15 requires that high-risk AI systems document adversarial robustness as a technical compliance obligation, not a best practice. The NIST AI RMF Manage function treats red teaming as a core risk treatment mechanism. Both frameworks treat regular adversarial testing as table stakes for deployment in regulated contexts, with the EU AI Act's August 2026 high-risk system deadline compressing the timeline for financial services and healthcare teams in particular.

The cost side reinforces the urgency. AI-related breach costs are rising as models move from generating text to executing actions against production systems. A compromised agent writes to a database, calls an external API, or modifies shared state before any human sees what happened. Red teaming is the only pre-production mechanism that stress-tests whether those pathways can be exploited.

The AI threat taxonomy: what red teams are actually testing for

Two frameworks give red teams the most traction when mapping AI attack categories: the OWASP LLM Top 10 and MITRE ATLAS. Neither is complete on its own, but together they cover the threat surface that matters in production.

Prompt-Based Attacks

• Direct prompt injection: a user crafts input that overrides the system prompt, redirecting the model to ignore its instructions or produce prohibited output.
• Indirect prompt injection: malicious instructions are embedded in retrieved content, such as a poisoned document in a RAG corpus, and the model executes them as if they were legitimate.
• Jailbreaks: inputs that route around safety constraints through roleplay framing, encoding tricks, or multi-step manipulation.

Data Layer Attacks

• Training data extraction: prompts that cause the model to reproduce memorized training data verbatim, including PII in LLM outputs or proprietary content.
• Membership inference: probing whether a specific record was present in the training set.
• Data poisoning: introducing adversarial examples into fine-tuning or retrieval corpora to shift model behavior at inference time.

Model-Level Attacks

• Adversarial examples: inputs with subtle perturbations that cause consistent misclassification or harmful output, often imperceptible to humans.
• Model inversion: reconstructing training data from model outputs, a particular risk for models trained on sensitive records.
• Model extraction: querying a model repeatedly to reconstruct its behavior and build a functional copy.

Supply Chain Attacks

These target dependencies upstream of the model itself: compromised base models from public repositories, malicious fine-tuning datasets, or tampered model weights. A system that passes every behavioral test can still be vulnerable if its foundation was poisoned before your team touched it.

A structured AI red teaming methodology

Four phases define a structured engagement. Skip one and the output is either untestable findings or test results without enough context to act on.

Phase 1: Planning and Threat Modeling

Before any prompt gets written, the team needs a defined scope and an adversary model. Scope covers which systems are in play, which are out, and what constitutes a finding worth documenting. Threat modeling answers who would realistically attack this system and what they would want: a competitor probing for training data, a user trying to bypass content filters, or a malicious actor attempting to hijack tool calls in an agentic pipeline.

Rules of engagement matter here too. Define which attack categories are in scope, whether social engineering of the underlying model is permitted, and what the severity threshold is for escalating a finding before the engagement ends.

Phase 2: Attack Generation and Execution

Human prompting handles novel attack chains that automated tools miss. A skilled red teamer will construct multi-turn sequences where each turn looks benign, with the payload distributed across several exchanges. Automated generation handles volume: fuzzing a prompt template across thousands of variations to surface probabilistic failures that only appear at scale.

Both matter. Running only automated attacks means missing the creative, context-aware jailbreaks that a motivated adversary would use. Running only human testers means missing the long tail of edge cases that only appear at high query volumes.

Phase 3: Evaluation and Scoring

Not every successful attack carries equal weight. A finding that produces harmful output on 60% of attempts, reproducibly, against a system deployed to millions of users is categorically different from one that succeeds once under contrived conditions. Severity classification should account for reproducibility rate, blast radius, and deviation from baseline behavior, not solely whether the attack technically succeeded.

Phase 4: Reporting and Remediation

Findings that don't become regression tests don't stay fixed.

The deliverable from a red team engagement goes beyond a report. Each confirmed vulnerability should be converted into a reproducible test case and added to a regression suite that runs on every subsequent deployment. If a jailbreak worked against version 1.2, that exact attack pattern should be a permanent fixture in the evaluation pipeline for version 1.3, 1.4, and every version after it. Remediation without a regression test is a patch with no verification that the patch held.

Agentic AI red teaming: a different class of risk

When an agent can read from a database, call an external API, or hand off tasks to a sub-agent, a successful attack no longer produces a bad response. It produces an action that already executed against a real system before anyone reviewed it.

The CSA Agentic AI Red Teaming Guide maps the attack surface across four layers: the application layer, the model itself, tool and MCP connections, and the data the agent reads and writes. Each layer introduces distinct threat categories:

• Memory manipulation: injecting false context into an agent's memory store to shift its behavior across future turns
• Orchestration flaws: exploiting the coordinator logic in multi-agent systems to redirect task execution
• Permission escalation: prompting an agent to invoke tools or access data outside its authorized scope
• Multi-agent collusion: compromising one agent in a chain to influence downstream agents that trust its outputs
• Goal hijacking: gradually steering an agent's objective through multi-turn manipulation while each individual step appears benign
• Supply chain attacks on agent tools: targeting the MCP servers, APIs, or third-party services an agent depends on

What makes agentic red teaming structurally different from model-level testing is that the threat surface is not static. Every new tool integration, added sub-agent, or permission boundary change creates new attack paths. Maintaining AI agent observability across those changes is required, since a scope that passed red teaming in January may be exploitable by March after a tool update. Point-in-time engagements are not sufficient here. Agentic red teaming needs to run continuously, tied to deployment events, not a quarterly calendar.

AI red teaming tools: open source options and commercial platforms

The open source tier has grown substantially. Four tools account for most of the practitioner conversation.

Open Source Tools

• Microsoft PyRIT (Python Risk Identification Toolkit): an orchestration framework for running multi-turn adversarial campaigns against AI systems. PyRIT abstracts attack orchestration, scoring, and memory management so red teamers can build systematic attack sequences without writing the scaffolding from scratch. It handles both direct and indirect prompt injection patterns and works across Azure OpenAI deployments as well as other endpoints.
• NVIDIA Garak: a broad-coverage scanner designed to probe LLMs for known vulnerability classes at volume. Garak runs hundreds of probes across categories including toxicity, LLM hallucination detection, jailbreak susceptibility, and data leakage. Good for building a behavioral baseline quickly; less suited to scenario-driven adversarial simulations.
• DeepTeam: an open source red teaming framework covering 50+ vulnerability types and 20+ attack methods, aligned to OWASP and NIST taxonomies. Its coverage of structured attack categories makes it a reasonable fit for teams that need to map findings to compliance frameworks.
• Promptfoo: config-driven testing with a red team module. Its strength is CI/CD eval gate integration and repeatable test definitions, making it more useful for teams building red teaming into deployment pipelines than for exploratory one-time engagements.

What Commercial Platforms Add

Open source tools cover the input/output layer well. The gap shows up when the system under test is a RAG pipeline or an agentic workflow. Fewer than half of available tools provide coverage for retrieval-layer attacks, tool call interception, or multi-agent trust boundaries. That gap is where commercial offerings typically stand apart: managed attack generation that doesn't require maintaining your own attack corpus, built-in CI/CD hooks, compliance reporting that maps findings to EU AI Act or NIST AI RMF obligations, and coverage of agentic tool integrations that open source frameworks don't reach yet.

The evaluation question worth asking before selecting any tool: does it test only what the model outputs, or does it also test what the system does? For standalone LLMs, the distinction barely matters. For agentic deployments, it determines whether your red teaming program actually covers the risk surface that exists in production.

Regulatory frameworks that define AI red teaming requirements

Four frameworks define where red teaming sits as a formal obligation, not an optional practice.

NIST AI RMF treats adversarial testing as part of the Manage function, covering the MAP and MANAGE core functions that require organizations to identify, analyze, and treat AI risks across the system lifecycle. The Manage function calls for documented risk treatment activities: red teaming findings, remediation decisions, and residual risk acceptance, going beyond test execution alone. Organizations subject to NIST AI RMF need a paper trail connecting red team findings to treatment decisions, not merely a record of attacks attempted.

OWASP provides the primary vulnerability taxonomy practitioners use for coverage mapping. The Top 10 for LLM Applications classifies prompt injection, insecure output handling, training data poisoning, and model theft as the categories red teams should cover systematically. The 2026 OWASP Top 10 for Agentic Applications extends this to agent-specific risks: excessive agency, unsafe tool use, and trust boundary violations across multi-agent chains. Together, these two lists give red teams a checklist that maps findings to named, recognized vulnerability classes instead of ad hoc descriptions.

MITRE ATLAS operates at a different level. Where OWASP lists vulnerability classes, ATLAS maps adversary tactics and techniques specific to AI and ML systems, structured as a matrix analogous to MITRE ATT&CK. It covers reconnaissance through impact, with AI-specific techniques like model evasion, data poisoning campaigns, and model extraction. ATLAS is most useful during threat modeling when the goal is building a realistic adversary profile, not checking coverage against a vulnerability list.

EU AI Act Article 15 creates the hardest obligation. High-risk systems must satisfy accuracy, robustness, and cybersecurity as technical requirements, with adversarial robustness explicitly named. Prompt injection resistance and resilience to input perturbations are not best practices under Article 15; they are conformity requirements that need documented test evidence before an August 2026 deadline for financial services and healthcare deployers. Article 15 requires that documented robustness evidence exists; Article 43 requires the conformity assessment record to contain that evidence. A policy statement about robustness does not satisfy either requirement.

Human red teaming vs. automated red teaming: tradeoffs and combinations

Neither approach alone covers the full risk surface. Human red teamers find what automated tools can't anticipate; automated tools find what human testers can't cover at volume. The practical question is how to combine them given your system's risk level, release cadence, and the evidence requirements of whatever regulatory framework applies.

Human Red Teaming

Skilled human testers bring contextual judgment that no automated framework replicates. A multi-turn jailbreak that distributes its payload across seemingly benign exchanges, a culturally specific prompt that bypasses a content filter trained on Western text patterns, a social-engineering sequence that exploits a model's persona instructions: all of these require creative adversarial reasoning, not coverage breadth. Human testers also adapt mid-engagement: when a partial success hints at a deeper vulnerability, they follow it. Automated tools run what they were configured to run.

The constraint is scale. A human red teamer might generate hundreds of meaningful attack variants in a day. A production system may receive millions of queries. Manual testing suits pre-deployment engagements, high-risk scenario validation, and novel attack discovery, but not continuous post-deployment coverage.

Automated Red Teaming

Automated frameworks handle the volume problem. PyRIT, Garak, and similar tools generate thousands of attack variations across defined categories, integrate with CI/CD pipelines, and run on every deployment. For probabilistic failures that only surface at high query rates, automation is the only viable detection path.

The risk is repetition. Automated attack generation can collapse onto a narrow set of patterns if the underlying attack corpus isn't maintained. A tool running the same 500 prompt injection variants on every release gives good coverage of those 500 patterns and poor coverage of everything outside them. Without deliberate corpus diversification, automated red teaming becomes a regression test, not a discovery exercise.

The Hybrid Model

Current best practice splits responsibility by function: humans drive discovery and automated tools drive coverage. Human testers design new attack scenarios, validate novel failure modes, and probe for contextual vulnerabilities; those findings get converted into automated test cases that run continuously afterward. Over time the automated corpus grows from real adversarial findings, not synthetic generation alone.

The right balance depends on four factors:

• System risk level: higher-risk deployments in healthcare, financial services, or agentic systems with tool access warrant more frequent human red team cycles alongside continuous automation.
• Release cadence: teams shipping weekly need automation integrated into CI/CD; teams shipping quarterly can front-load human engagements pre-release.
• Team composition: if your internal security team lacks ML literacy, external human red teamers with AI-specific expertise close that gap faster than expanding automated coverage.
• Regulatory evidence requirements: NIST AI RMF and EU AI Act Article 15 both expect documented adversarial testing; automated tooling generates structured logs naturally, while human engagements require explicit documentation of findings and treatment decisions to satisfy the same evidentiary bar.

Human and automated red teaming answer different questions. Treating them as interchangeable produces gaps in both discovery and coverage.

Embedding red teaming into the AI development lifecycle

Red teaming that runs only before launch is a point-in-time check on a system that never stops changing. A model update, a prompt revision, a new tool integration, or a retrieval corpus refresh each creates a new attack surface. The only way to stay ahead of that surface is to treat adversarial testing as a development practice, not a deployment gate.

The shift-left principle here is straightforward: run adversarial tests in the same CI/CD pipeline that runs unit tests and evaluation suites. Every pull request that touches a system prompt, adjusts a retrieval config, or adds a tool integration should trigger a defined set of adversarial probes before it merges. A jailbreak that worked against a previous system prompt version becomes a permanent regression test. If the fix holds, the test passes silently. If a later change reintroduces the vulnerability, the gate catches it before it reaches production.

CI/CD gate design for red teaming requires two things: threshold definitions and evidence artifacts. Thresholds might look like "block deployment if prompt injection success rate across the test suite exceeds 5%" or "require manual review if any jailbreak variant succeeds more than once in 20 attempts" (example thresholds; calibrate to your system's risk level and regulatory exposure). What the audit record needs is more than a pass/fail result: the specific test cases run, the attack variants attempted, the scores returned, and the model version hash those results are bound to. A policy statement that adversarial testing was conducted does not satisfy EU AI Act Article 43 conformity assessment requirements; a structured test record tied to a specific artifact does.

On the organizational side, a shared findings library is what separates a mature red teaming program from a collection of one-off engagements. Each confirmed vulnerability gets documented with the attack pattern, the affected system version, the severity score, and the remediation applied. New team members inherit the institutional memory of every past engagement. The attack corpus should be treated like a codebase: versioned, reviewed, and deliberately expanded instead of left to accumulate organically.

Red team composition matters for corpus quality. A team of pure security engineers will find infrastructure-adjacent vulnerabilities but will miss culturally specific jailbreaks or domain-sensitive failure modes that require subject-matter knowledge. Financial services deployments benefit from testers who understand how credit scoring decisions get reviewed and challenged; healthcare deployments need testers who can reason about clinical context. Rotating external red teamers into periodic engagements brings fresh adversarial creativity that internal teams, familiar with their own systems, tend to lose over time.

How Openlayer fits into an AI red teaming program

Red teaming surfaces what can go wrong. The harder problem is building the infrastructure that prevents it from going wrong again, in production, automatically.

That's where Openlayer fits. As a unified evaluation, observability, and governance platform spanning development through production, it sits downstream of red team findings and converts them into enforceable controls. Over 175 pre-built automated tests cover safety, security, and accuracy dimensions out of the box, including prompt injection resistance, jailbreak resistance, PII leakage detection, and adversarial robustness, so teams aren't building test coverage from scratch after every engagement. Adversarial findings from human red teamers get added to that suite as permanent regression tests: a jailbreak confirmed in an engagement becomes a gate that runs on every subsequent deployment. If a future prompt update reintroduces the vulnerability, the CI/CD gate catches it before it merges.

At runtime, Openlayer's guardrails block unsafe outputs at the API boundary before they reach end users or downstream systems. That blocking step is what separates enforcement from observation. A red team report describes a successful attack; a guardrail configuration prevents that attack pattern from succeeding in production. Every enforcement action generates a per-request audit record containing the metric score, the policy rule triggered, and the timestamp. Those records map directly to EU AI Act Article 15 adversarial robustness requirements and NIST AI RMF risk treatment documentation, produced as a byproduct of enforcement instead of assembled after the fact.

Red teaming is a discovery activity. Active runtime enforcement is the control that closes the gap between knowing an attack pattern exists and preventing it from succeeding when it matters.

Final thoughts on running effective AI red teaming engagements

The most common gap in AI red teaming programs isn't coverage. It's what happens after a finding lands. A confirmed jailbreak or prompt injection that doesn't become a regression test is a finding without a control, and a control that only logs but doesn't block is observation, not enforcement. Your testing methodology, your tooling choices, and your remediation workflow all need to treat those distinctions as real. Reach out to the Openlayer team to see how red team findings map directly into enforceable deployment gates.

FAQ

What's the difference between red teaming AI models and red teaming agentic AI systems?

With standalone LLMs, a successful attack produces a bad response: harmful output that a guardrail or human reviewer can catch after the fact. With agentic systems, a successful attack produces an action that has already executed against a real system: a database write, an external API call, a tool invocation outside the agent's authorized scope. That structural difference means agentic red teaming must cover four distinct layers (application, model, tool and MCP connections, and data) and must run continuously, tied to deployment events and tool integration changes, not as a point-in-time pre-launch exercise.

PyRIT vs Garak vs Promptfoo for CI/CD-integrated AI red teaming?

Each tool occupies a different position in the workflow. PyRIT is best for running multi-turn adversarial campaigns with memory management, which helps when you need systematic attack sequences over single-prompt probes. Garak is better for broad-coverage baseline scans across toxicity, jailbreak susceptibility, and data leakage at volume. Promptfoo is the strongest fit for CI/CD integration: its config-driven test definitions run repeatably on every deployment, making it more suitable for teams embedding red teaming into release pipelines over exploratory one-off engagements. The gap all three share is retrieval-layer and agentic coverage: tool call interception, indirect prompt injection through RAG corpora, and multi-agent trust boundaries are where open source frameworks currently fall short.

How do I build red teaming into a CI/CD pipeline to meet EU AI Act Article 15 adversarial robustness requirements?

The pipeline needs two things: threshold definitions that gate deployment, and structured test records that satisfy Article 15 robustness requirements, documented as Article 43 conformity assessment evidence. Thresholds might look like blocking deployment when prompt injection success rate across the test suite exceeds 5%, or requiring manual review if any jailbreak variant succeeds more than once in 20 attempts, each calibrated to your system's risk level and regulatory exposure. The audit record must contain the specific test cases run, attack variants attempted, scores returned, and the model version hash those results are bound to. A policy statement that adversarial testing was conducted does not satisfy Article 43; a structured test record tied to a specific artifact does. Every confirmed finding should be converted into a permanent regression test so future deployments are gated against the same attack patterns.

What is the difference between human and automated red teaming for AI systems?

Human red teamers find what automated tools cannot anticipate: multi-turn jailbreaks that distribute payloads across seemingly benign exchanges, culturally specific prompts that bypass content filters trained on narrow text patterns, and context-aware attack chains that require adversarial reasoning to construct. Automated frameworks like PyRIT and Garak handle what human testers cannot cover at volume: probabilistic failures that only surface at thousands of query variations, and continuous coverage across every deployment. The practical split is by function: humans drive discovery, automated tools drive coverage. Human testers design new attack scenarios and validate novel failure modes; those findings get converted into automated test cases that run on every subsequent deployment. Over time, the automated corpus grows from real adversarial findings and not from synthetic generation alone.

What tools does Openlayer provide to turn red team findings into enforceable production controls?

Red teaming is a discovery activity; it surfaces what attack patterns exist. Openlayer sits downstream of that discovery and converts findings into controls that run automatically in production. Over 175 pre-built automated tests cover prompt injection resistance, jailbreak resistance, PII leakage detection, and adversarial robustness out of the box. Confirmed findings from human red team engagements get added to that suite as permanent regression tests: if a future prompt update or model version reintroduces a vulnerability, the CI/CD gate catches it before it merges. At runtime, guardrails block unsafe outputs at the API boundary before they reach end users or downstream systems, generating a per-request audit record containing the metric score, the policy rule triggered, and the timestamp, which maps directly to EU AI Act Article 15 and NIST AI RMF risk treatment documentation requirements.