LLM Security: Threats & Production Controls (October 2026)

Most teams securing an LLM deployment start with what they know: input validation, output filtering, access controls. Those matter, but they don't cover what makes LLM security genuinely hard. The model can be manipulated through content it reads, beyond the inputs it receives directly. An agent can execute an unauthorized action before any logging system records it. The attack surface extends into the model's own reasoning process. This covers the full picture, from the OWASP Top 10 for LLM applications through the production enforcement controls that close the gap between detecting a problem and actually stopping it.

TLDR:

• Prompt injection attacks succeed at rates of 50-84% across system configurations, and no frontier model is immune after applying best defenses.
• LLM security differs from traditional AppSec in four structural ways: no instruction-vs-data separation, non-deterministic outputs, immature supply chain controls, and irreversible agentic actions.
• Defense-in-depth for production deployments requires layering controls across six surfaces: input validation, output handling, least-privilege tool access, RAG pipeline security, data-layer access control, and audit logging.
• EU AI Act high-risk financial services obligations carry an August 2026 enforcement deadline; Articles 9, 15, 61, and 72 each map to specific security testing records and monitoring artifacts.
• Openlayer blocks prompt injections, PII leakage, and unauthorized tool calls before they exit the API boundary, suspending agentic execution when intent-to-tool alignment confidence drops below a configurable threshold.

What is LLM security?

LLM security is the practice of identifying, testing for, and enforcing controls against the failure modes specific to AI systems built on language models. These risks come not from buggy code or misconfigured infrastructure alone, but from how LLMs process and generate language: probabilistically, context-sensitively, and in ways no static rule set fully anticipates.

Where traditional application security assumes a deterministic system you can test, patch, and lock down, LLM security contends with a system whose behavior can shift without any code change. A model can be steered by a crafted input, leak sensitive information from training data, or violate policy under conditions that passed pre-deployment testing. The attack surface extends into the model's reasoning process itself.

The discipline spans several distinct concerns:

• Prompt injection and LLM red teaming jailbreak resistance, where inputs are constructed to override intended behavior
• Training data integrity, including poisoned datasets and memorized sensitive content
• Output safety at inference time, where policy violations surface only under specific runtime conditions
• Supply chain risk from third-party models and fine-tuning datasets
• Governance of autonomous agents that act on LLM outputs instead of merely displaying them

Each represents a category of risk that traditional application security tooling was not designed to catch.

How LLM security differs from traditional application security

Four structural differences separate LLM security from conventional AppSec, not as a matter of degree, but of kind.

Traditional application security assumes a mostly deterministic system: test inputs, validate outputs, patch code, lock down infrastructure. LLMs add a probabilistic layer that can change behavior without any code change, expanding the security question from "is the software vulnerable?" to "can the model be steered, confused, or induced to act outside its intended bounds?"

The four concrete structural gaps:

• No instruction-vs-data separation. Traditional apps parse commands and data through different channels. LLMs receive both as text and cannot reliably distinguish between them, which is the root condition that makes prompt injection structurally unavoidable and not simply patchable.
• Non-deterministic output behavior. Patching a traditional vulnerability produces a fixed, testable result. A model's output under adversarial conditions varies by temperature, context window state, and phrasing. You cannot unit-test your way to a security guarantee.
• Model supply chain risk. As Cisco notes, open-source scanning for traditional apps relies on dependency manifests and known CVEs. For LLMs, the equivalent supply chain risk lives in pre-trained weights, fine-tuning datasets, and third-party model APIs, none of which have mature, standardized vulnerability disclosure mechanisms yet.
• Irreversibility of agentic actions. A vulnerable web app serves a bad response. A vulnerable agent executes: it writes to a database, calls an external API, modifies shared state. The failure has already propagated before any logging system sees it.

The OWASP Top 10 for LLM applications (2025)

The OWASP Top 10 for LLM Applications 2025 is the closest thing the field has to a shared reference standard. Our OWASP LLM security testing guide covers each risk class in depth. Each entry maps to a distinct risk class with its own attack surface and mitigation approach.

Core LLM attack vectors explained

Four vectors warrant more than a label.

• Prompt injection exploits a structural property: LLMs receive instructions and data as the same text, so they cannot reliably distinguish between them. Direct injection overwrites system instructions through user input. Indirect injection embeds adversarial instructions inside external content a model retrieves, like a webpage or document, then executes them without user involvement. 50-84% attack success rates documented depending on system configuration, and no frontier model is immune after applying its best defenses.
• Training data poisoning targets the model before deployment. The PoisonGPT demonstration showed that surgically modifying a single fact in a model's weights produced confident, targeted misinformation while leaving all other outputs intact. Standard input validation at inference time cannot catch this because the attack already happened upstream.
• Memorization and sensitive information leakage follow from how models are trained: they compress patterns from their data, but under the right prompts they reproduce verbatim sequences, including PII, credentials, and proprietary content. Output filtering catches some of this, though semantic variants can evade keyword-based blocklists.
• Model extraction uses carefully crafted queries to reconstruct a proprietary model's behavior, effectively stealing the underlying capability through its API. Rate limiting slows extraction but does not prevent it if an attacker distributes queries across accounts or time.

The thread connecting all four: each bypasses controls that assume a clean separation between trusted instructions and untrusted data. That separation does not exist in language model architectures.

Agentic AI and the expanded LLM security surface

Agents change the threat model in one concrete way: an LLM output that gets displayed is a response; an LLM output that gets executed is an action. Once an agent invokes a tool, writes to a database, or calls an external API, the failure has already propagated. There is no review step between the decision and its consequence.

Three risks follow directly from that architecture:

• Tool call hijacking: an injected instruction redirects an agent to invoke a tool outside its permitted scope. The agent does not flag this as an error because the tool is valid and the arguments are well-formed. Only an allow-list check at the moment of invocation would catch it, and most deployments do not have one.
• AI agent failure modes like cascading failures across multi-agent chains: in a supervisor-worker or peer-to-peer architecture, a compromised agent can corrupt shared state or pass malicious instructions downstream. One bad output becomes the input to the next agent, with no human checkpoint between them.
• Indirect prompt injection via retrieved content: an attacker does not need access to the system prompt. They embed adversarial instructions inside a webpage, document, or database record the agent retrieves at runtime. The model reads that content as context, then executes the embedded instruction as if it were authorized.

In a chat context, a successful injection is a bad response. In an agentic context, it is an unauthorized action that already ran before any logging system recorded it.

LLM security testing: pre-deployment and continuous

Two categories structure LLM security testing, and treating them as interchangeable is how teams end up with a clean test suite and a breach.

Pre-deployment security testing

Pre-deployment testing catches known failure modes before any user interaction occurs. Concrete methods:

• Adversarial red-teaming: manually craft inputs designed to override system instructions, extract the system prompt, or trigger policy violations. Automated red-teaming tools accelerate coverage but miss context-dependent exploits that human testers find.
• Prompt injection resistance evaluation: run a held-out set of direct and indirect injection attempts and measure how often outputs deviate from the original task specification.
• LLM jailbreak benchmarking: test against known jailbreak taxonomies, including role-play framings, token-smuggling variants, and multi-turn escalation sequences that use earlier outputs to lower the model's guard.
• LLM pipeline PII detection and sensitive content leakage testing: probe whether the model reproduces training data verbatim or discloses system prompt contents under adversarial extraction attempts.
• Bias and demographic parity evaluation across sub-populations before any production traffic touches the model.

Continuous production testing

Pre-deployment testing cannot reach the risk class that surfaces under live traffic. Behavioral drift, novel injection patterns, and distribution changes in user inputs do not appear in held-out test sets. Three monitoring controls close that gap:

• Output distribution monitoring: detect statistical changes in response length, refusal rates, or topic distribution that signal behavioral drift without requiring ground-truth labels.
• Anomaly detection on safety and groundedness vs faithfulness scores: flag when metric distributions deviate from recorded baselines, catching degradation before it becomes a pattern.
• Replay-based regression testing: when a failure surfaces in production, convert the triggering input into a permanent regression test so the same failure cannot recur undetected after a model update.

Neither substitutes for the other: pre-deployment catches misconfigured prompts and regressions against known attack patterns; runtime monitoring catches what surfaces under real traffic.

LLM security best practices for production deployments

Defense-in-depth for LLM production deployments means layering controls across six distinct surfaces, since no single control covers the full attack surface.

• Input validation and sanitization: keyword blocklists catch known patterns but miss semantic variants. Semantic-aware classifiers, trained to detect injection-pattern intent instead of matching surface tokens, raise the bar materially. For RAG pipelines, retrieved chunks require their own classifier pass before entering the context window, with injection-pattern confidence above 0.85 routing to secondary review instead of hard-blocking.
• Output handling and schema validation: treat every model output as untrusted before passing it downstream. Schema validation catches format violations; LLM guardrails framework semantic guardrails catch policy violations that well-formed output can still carry, including PII, hallucinated facts, or injected instructions embedded for a downstream consumer.
• Least-privilege tool and API access: agents should operate against explicit allow lists. Applying tool-calling agent guardrails and blocking when an agent requests a tool outside its registered allowlist, before execution. Every unauthorized invocation that executes first and gets logged second is an action that already happened.
• Data security for RAG pipelines: validate source documents at ingestion, enforce access controls on what a given agent or user role can retrieve, and monitor retrieval results for adversarial content before generation runs.
• Access control and authentication: enforce data-layer access controls at retrieval time, beyond the API boundary alone. An authenticated user reaching a model endpoint with broad retrieval access can still exfiltrate data the underlying permissions should have blocked.
• Audit logging and incident response: logs captured at inference time carry more evidentiary weight than reconstructed summaries. Severity triage follows three tiers: critical failures warrant immediate suspension and notification within one hour; measurable degradation warrants enhanced monitoring and investigation within 24 hours; isolated anomalies warrant logging and resolution within five business days.

LLM security, compliance frameworks, and regulatory obligations

Three frameworks dominate enterprise LLM compliance obligations right now. For a deeper look at running EU AI Act and NIST RMF compliance together, see our dedicated guide. Here is what each requires and which security controls satisfy each obligation.

Two platforms, Credo AI and IBM watsonx.governance, are commonly assessed alongside Openlayer in enterprise compliance programs. Both cover the policy-documentation layer well: Credo AI organizes compliance workflows through Policy Packs and auto-generates audit artifacts tied to EU AI Act, NIST AI RMF, and ISO 42001; IBM watsonx.governance delivers framework mapping and fairness dashboards within the IBM stack. Neither platform enforces controls at the API boundary. They document that the right policies exist; they do not block a prompt injection before it executes, suspend an unauthorized tool call before it fires, or redact PII before it reaches a downstream system. Openlayer operates at the enforcement layer: the controls run at inference time, and the compliance evidence is a byproduct of enforcement running normally, not a documentation exercise assembled after the fact.

EU AI Act

High-risk system obligations under an August 2026 deadline are now enforceable, with financial services use cases including credit scoring, fraud detection, and insurance pricing in scope.

• Article 9 (risk management): continuous risk assessment throughout the lifecycle. Prompt injection resistance testing, adversarial red-teaming results, and unauthorized tool call detection logs serve as evidentiary records produced at enforcement time, not assembled after the fact.
• Article 15 (accuracy and robustness): ongoing accuracy measurement and adversarial robustness testing covering prompt injection and jailbreak resistance. Groundedness scores, demographic parity gaps, and jailbreak benchmark results map directly here.
• Articles 61 and 72 (post-market monitoring): continuous production monitoring with anomaly detection and drift reporting. Output distribution monitoring, anomaly detection on safety scores, and replay-based regression tests constitute the monitoring record these articles require.

NIST AI RMF

The four functions structure governance obligations without prescribing specific controls.

• Govern: build organizational accountability and policy for LLM risk, including who owns each deployed system and which risk tier it carries.
• Map: identify the attack surface for each deployment, including prompt injection vectors, supply chain dependencies, and agentic tool scope.
• Measure: run quantified evaluation against safety, groundedness, and bias metrics at defined intervals. Metric scores, threshold breaches, and evaluation timestamps constitute the measurement record.
• Manage: activate controls in response to identified risks, including guardrails, allow-list enforcement, and incident escalation workflows.

ISO 42001

ISO 42001 requires a documented AI management system covering risk identification, control implementation, and continuous improvement. For LLM deployments, that means version-controlled policy definitions, pre-deployment evaluation records bound to the deployed model version, and audit logs that trace each security control back to the risk it covers.

How Openlayer enforces LLM security controls across the full production lifecycle

Most LLM security tooling sits on the observation side of the enforcement divide: it detects, logs, and alerts. Openlayer crosses into active blocking.

Real-time guardrails intercept prompt injections, PII leakage, toxic outputs, and unauthorized tool calls before they exit the API boundary. Each guardrail decision is configurable across five discrete actions: allow with logging, warn, block, redact, or escalate to a human reviewer. The escalate path holds the output before delivery, routing it to review where automated confidence alone is insufficient: clinical recommendations or financial planning outputs where a wrong response carries direct legal exposure.

For agentic deployments, tool call authorization operates against explicit allow lists. When intent-to-tool alignment confidence drops below a configurable threshold, or when a requested tool sits outside the agent's registered allowlist, execution suspends before the call fires. Each block event generates an audit trail entry carrying the agent ID, requested tool, alignment score, and task context. That blocking step, not the log entry it produces, is what separates enforcement from observation.

RAG pipelines get two-stage coverage. Retrieved chunks pass through injection-pattern classifiers before entering the context window. For a CISO-focused view of production LLM security architecture: retrieved chunks with injection-pattern confidence above 0.85 route to secondary review; generation proceeds only on chunks that pass. A faithfulness check at the generation layer then catches contradictions between the response and retrieved context. Both stages operate before output reaches end users.

The over 175 pre-built tests covering prompt injection resistance, jailbreak benchmarking, PII detection, and adversarial robustness run at pre-deployment gates and continuously against production traffic. Results map automatically to EU AI Act Articles 9, 15, 61, and 72, NIST AI RMF Measure and Manage functions, and ISO 42001 control documentation requirements. Compliance evidence is a byproduct of enforcement running normally, not a documentation exercise scheduled after the fact.

Final thoughts on LLM security

LLM security is not a checklist you run once before launch. Behavioral drift, novel injection patterns, and new attack vectors keep appearing under live traffic, so your controls need to run continuously, at every stage beyond deployment gates. The teams that get this right treat pre-deployment testing and runtime enforcement as two parts of the same loop, not separate workstreams. Talk to the Openlayer team to see how that loop works in production.

FAQ

How do I secure AI agents against prompt injection and unauthorized tool use in production?

Securing agents requires two distinct controls that operate at different points in the execution chain. For prompt injection, deploy a classifier that scores retrieved chunks before they enter the context window. Flag anything above 0.85 confidence for secondary review instead of hard-blocking, which preserves agent usefulness on legitimate tasks. For unauthorized tool use, enforce explicit allow lists at the moment of invocation: when intent-to-tool alignment confidence drops below 0.75, or when a requested tool sits outside the registered allowlist, suspend execution before the call fires. Every tool call that executes first and gets logged second is an action that already happened against a production system. The OWASP Top 10 for LLM Applications (LLM01, LLM06) and NIST AI RMF Manage function both require controls that prevent, not merely record.

What is the difference between pre-deployment LLM security testing and continuous production monitoring?

Pre-deployment testing catches known failure modes before any user interaction: prompt injection resistance, jailbreak benchmarks, PII leakage probes, and bias evaluation across sub-populations. Continuous production monitoring covers a structurally different risk class: behavioral drift, novel injection patterns, and distribution changes that only surface under live traffic. Treating them as interchangeable is how teams end up with a clean test suite and a breach; a held-out test set cannot contain attack patterns that haven't been written yet, and output distribution changes have no ground-truth label to validate against in pre-deployment.

What does the OWASP Top 10 for LLM applications cover, and which risks are hardest to catch before deployment?

The OWASP LLM Top 10 (2025) covers ten distinct risk classes: prompt injection, sensitive information disclosure, supply chain compromise, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. The hardest to catch pre-deployment are data and model poisoning (the attack happened upstream during training), indirect prompt injection via retrieved content (the adversarial payload arrives at runtime through external documents), and excessive agency (tool scope violations that only materialize under specific user intents). Static test sets can benchmark known injection patterns and jailbreak taxonomies, but supply chain and retrieval-layer attacks require controls at ingestion and runtime, extending well past pre-deployment gates.

How do I implement human-in-the-loop controls for AI systems making high-stakes decisions?

Configure an escalate action at the guardrail layer that holds outputs before delivery and routes them to a human reviewer when automated confidence alone is insufficient. This applies where a wrong response carries direct legal or clinical exposure: treatment recommendations, financial planning outputs, benefits eligibility determinations, and legal contract review. The EU AI Act Article 14 human oversight obligation requires that operators retain the ability to intervene and override (EU AI Act, Article 14). A logging-only setup satisfies neither the technical requirement nor the spirit of the obligation, because a log reviewed after delivery is not an intervention. The escalate path stops the output at the API boundary; that stopping step, not the review record it creates, is what constitutes human oversight as a control and not merely an audit artifact.

Langfuse vs Openlayer for LLM security testing and runtime enforcement?

Langfuse is a diagnostic observability platform that currently delegates runtime enforcement to third-party libraries such as LLM Guard and NeMo Guardrails; it detects and surfaces security signals but currently does not natively block. Openlayer enforces at the API boundary directly: prompt injection blocking, PII redaction, unauthorized tool call suspension, and content safety guardrails all execute before outputs reach end users or downstream systems, with each enforcement decision captured as a traceable audit span. For teams whose primary concern is observability and debugging, Langfuse covers that surface well. For teams subject to EU AI Act Articles 9 and 15, NIST AI RMF, or ISO 42001, where security controls must produce audit evidence of enforcement and not detection alone, the architectural gap between delegating to a third-party library and enforcing natively is the difference between a logging record and a compliance control.