Generative AI applications break many assumptions that traditional application security relies on. Instructions and data travel in the same channel, so any document, email or web page a model reads can try to give it orders. Outputs are probabilistic, so the same input can produce different results. And once applications become agents with tools, credentials and memory, a successful attack no longer just leaks text; it can send emails, change records or execute code.
The OWASP GenAI Security Project has become the most widely used reference for these risks. Its Top 10 for LLM Applications (2025 edition) covers model-centric applications such as chatbots and RAG systems, and in December 2025 it published the Top 10 for Agentic Applications for systems that plan and act. This playbook turns both lists into a practical engineering program: what to design, what to build, what to test and how to operate it.
Why GenAI Security Needs Its Own Playbook
Security teams sometimes treat a GenAI application as just another web app with an API call in the middle. That misses the defining problem. In a conventional application, code is trusted and data is untrusted, and the boundary between them is enforced by parsers and type systems. In an LLM application, natural language is both code and data. A model cannot reliably distinguish the developer’s instructions from instructions embedded in a retrieved PDF. No filter fully solves this, so security must be designed around the assumption that the model can be manipulated.
The practical consequence is a shift in where controls live. Instead of trying to make the model incorruptible, robust designs limit what a manipulated model can do: what data it can see, what tools it can call, what actions require confirmation, and how outputs are validated before they reach users or downstream systems.
The Two OWASP Lists in Brief
Top 10 for LLM Applications (2025)
- LLM01 Prompt Injection: direct and indirect manipulation of model behavior through crafted inputs.
- LLM02 Sensitive Information Disclosure: leakage of personal data, secrets or proprietary information through outputs.
- LLM03 Supply Chain: risks in third-party models, datasets, adapters and libraries.
- LLM04 Data and Model Poisoning: tampering with training, fine-tuning or embedding data.
- LLM05 Improper Output Handling: passing model output to browsers, shells, databases or APIs without validation.
- LLM06 Excessive Agency: granting models more functionality, permissions or autonomy than needed.
- LLM07 System Prompt Leakage: exposure of system prompts that contain sensitive logic or credentials.
- LLM08 Vector and Embedding Weaknesses: access control failures, poisoning and leakage in RAG stores.
- LLM09 Misinformation: hallucinated or misleading output and overreliance on it.
- LLM10 Unbounded Consumption: resource exhaustion, runaway costs and model extraction through excessive use.
As Qualys noted in its analysis of the 2025 update, sensitive information disclosure and supply chain moved up sharply, system prompt leakage and vector and embedding weaknesses were added, misinformation absorbed the earlier overreliance entry, and denial of service was broadened into unbounded consumption to cover cost risk.
Top 10 for Agentic Applications (2026)
The agentic list, announced by OWASP on December 9, 2025, addresses risks that emerge when AI takes actions: ASI01 Agent Goal Hijack, ASI02 Tool Misuse, ASI03 Identity and Privilege Abuse, ASI04 Agentic Supply Chain Vulnerabilities, ASI05 Unexpected Code Execution, ASI06 Memory and Context Poisoning, ASI07 Insecure Inter-Agent Communication, ASI08 Cascading Failures, ASI09 Human-Agent Trust Exploitation and ASI10 Rogue Agents. OWASP illustrates several with real incidents, such as the EchoLeak zero-click exfiltration in an AI assistant for goal hijack and an MCP-related exploit for agentic supply chain risk.
You cannot make a model immune to manipulation. Design so that a manipulated model can do very little harm: minimal data, minimal tools, minimal permissions, validated outputs and human confirmation for anything irreversible.
Phase 1: Threat Model the Application
Start every GenAI project with a lightweight threat model focused on four questions:
- Untrusted inputs. Which content can reach the model? User messages, uploaded files, retrieved documents, emails, web pages, tool results and messages from other agents all count.
- Sensitive assets. What data, secrets and systems can the model see or reach?
- Capabilities. Which tools can it call, with what permissions, and which actions are irreversible?
- Consumers of output. Where does output go: a user’s screen, an HTML page, a SQL query, a shell, another agent?
The dangerous combination, often described as a “lethal trifecta,” is an agent that reads untrusted content, has access to sensitive data and can communicate externally. If your design has all three, either remove one or put strong controls between them.
Phase 2: Design Controls Mapped to the Top 10s
Contain Prompt Injection and Goal Hijack (LLM01, ASI01)
- Separate trusted instructions from untrusted content structurally, using clearly delimited sections and provider features for system-level instructions.
- Treat all retrieved and tool-returned content as data, and never let it change the agent’s permissions or objectives.
- Use input and output classifiers to detect likely injection, but treat them as a detection layer, not a guarantee.
- Require explicit user confirmation for actions triggered while processing untrusted content, such as sending messages or changing records.
Limit Agency, Tools and Privileges (LLM06, ASI02, ASI03)
- Give each agent its own identity with least-privilege, short-lived credentials rather than borrowing a user’s or a shared service account.
- Expose narrow, purpose-built tools (for example “create refund up to a limit”) instead of generic ones (“run SQL”).
- Enforce authorization in the tool layer, not in the prompt. The tool should check what the calling user and agent are allowed to do.
- Set rate limits, spending limits and action quotas per agent and per session.
Validate Output Before It Acts (LLM05, ASI05)
- Encode output for its destination: escape HTML, parameterize queries and never pass raw model text to a shell.
- Require structured outputs validated against schemas for any machine-consumed response.
- Run model-generated code only in sandboxes with no network access by default and strict resource limits.
Protect Data and Prompts (LLM02, LLM07, LLM08)
- Assume system prompts will leak. Never place secrets, keys or authorization logic in them.
- Apply document-level access control in retrieval so users only get passages they are entitled to see, enforced at query time.
- Redact or mask sensitive fields before indexing where full content is not needed.
- Scan outputs for personal data and secrets before returning them.
Secure the Supply Chain and Memory (LLM03, LLM04, ASI04, ASI06)
- Pin and verify models, adapters, datasets and MCP servers; maintain an AI bill of materials.
- Review third-party tools and MCP servers like code dependencies, including their permissions and update policies.
- Validate content before it enters long-term memory or knowledge bases, record provenance, and support rollback.
- Monitor for anomalous changes in retrieval corpora and embeddings.
Manage Multi-Agent and Human Trust Risks (ASI07–ASI10, LLM09)
- Authenticate and authorize messages between agents; do not let one agent’s output silently become another’s instructions.
- Add circuit breakers that stop cascades when error rates, costs or action volumes spike.
- Design confirmations that show users what will actually happen, not just a plausible summary written by the agent.
- Provide a kill switch to disable an agent or revoke its credentials immediately.
- Ground factual answers in cited sources and make uncertainty visible to users.
Control Consumption (LLM10)
Set per-user and per-tenant token and request quotas, cap output and reasoning tokens, detect scraping patterns that suggest model extraction, and alert on cost anomalies. Unbounded consumption is as much a FinOps risk as a security one.
Phase 3: Test Like an Attacker
GenAI security testing must be continuous because models, prompts and data change frequently. Build a test program with several layers:
- Automated adversarial suites in CI that run known injection, jailbreak and data-exfiltration prompts against every release, including indirect injection planted in test documents.
- Tool-abuse tests that check whether the agent can be induced to call tools outside its intended scope or with manipulated parameters.
- Access-control tests for RAG, confirming that users cannot retrieve documents they are not entitled to through clever queries.
- Manual red teaming before major launches and after significant capability changes, mapped to both OWASP lists.
- Regression tracking so that a model upgrade or prompt change that weakens defenses is caught before production.
Phase 4: Operate and Monitor
- Log prompts, retrieved context identifiers, tool calls, parameters, outputs and the identity of the user and agent for every session, with privacy-appropriate retention.
- Alert on injection detections, unusual tool-call patterns, permission denials, cost spikes and sudden changes in output characteristics.
- Route all model traffic through an AI gateway where policies, quotas and logging are enforced consistently.
- Maintain incident playbooks for prompt injection, data leakage and rogue agent behavior, including credential revocation steps.
Who Owns GenAI Security
GenAI security fails most often in the gaps between teams. Product teams own prompts and user experience, platform teams own gateways and model access, data teams own retrieval corpora, and security teams own testing and monitoring. A Cloud Security Alliance research note from May 2026 pointed to the EchoLeak vulnerability in an enterprise AI assistant, rated CVSS 9.3, as an example of how indirect prompt injection can turn ordinary content into an exfiltration path, which is exactly the kind of issue that spans all of those teams. Agree on a simple ownership model: each control in this playbook should have a named owning team, a test that proves it works and a metric that shows it keeps working.
Launch Readiness Checklist
- Threat model completed and reviewed, with the lethal trifecta explicitly addressed.
- Agent identities, tool scopes and spending limits documented and enforced in code.
- Retrieval access control tested with cross-tenant and cross-role queries.
- Output encoding and schema validation in place for every consumer of model output.
- Adversarial test suite integrated into CI, with direct and indirect injection cases.
- Logging, alerting, kill switch and incident playbook verified in a rehearsal.
- Third-party models, plugins and MCP servers recorded in the AI bill of materials.
A Hypothetical Worked Example
Consider a hypothetical B2B software company building a support agent that reads customer tickets and attachments, searches the knowledge base and customer account data, drafts replies and can issue service credits.
The threat model flags the lethal trifecta: tickets and attachments are untrusted, account data is sensitive, and the agent can send emails. The team redesigns: the agent drafts replies but a human sends them for any ticket containing attachments; service credits use a dedicated tool capped per ticket and per day, with authorization checked against the ticket’s customer in the tool itself; retrieval filters account data to the ticket’s customer only; and the system prompt contains no secrets. Outputs are rendered as escaped text, and the agent runs under its own identity with read-only access except for the credit tool.
In red teaming, testers plant an instruction in a PDF attachment telling the agent to issue a large credit and email account details to an external address. The injection classifier catches some variants, but not all; however, the credit tool rejects the amount, the external email cannot be sent without human review, and retrieval returns no data from other customers. The test confirms that defense in depth, not detection alone, is what contains the attack.
Common Pitfalls
- Relying on a system prompt instruction like “ignore malicious instructions” as the primary defense.
- Giving agents broad API keys or a shared admin account for convenience.
- Performing access control in the prompt rather than in tools and retrieval.
- Testing only direct prompt injection and ignoring documents, emails and tool results.
- Adding third-party MCP servers or plugins without security review.
- Treating security testing as a one-time launch gate rather than a continuous process.
Outlook
As agents gain autonomy and connect to each other, the attack surface will move further from the model itself toward identities, tools, memory and inter-agent protocols. Expect more standardization of agent identity, signed tool manifests and policy enforcement at gateways, along with regulatory expectations that high-impact AI systems demonstrate robustness against manipulation. The OWASP lists will keep evolving, but their core message is stable: assume the model can be manipulated, and engineer the system so it fails safely.
Conclusion
Securing generative AI is less about finding the perfect filter and more about disciplined architecture. Threat model untrusted inputs, sensitive assets and capabilities; limit agency and privileges; validate every output before it acts; protect data in retrieval and memory; test continuously like an attacker; and monitor everything an agent does. Using the OWASP LLM and Agentic Top 10s as a shared map, product, engineering and security teams can build GenAI applications that are useful and resilient at the same time.
How Hanoi Technology Can Help
Hanoi Technology helps teams threat model, harden and red-team generative AI applications and agents using OWASP guidance.