In short
An application based on a language model (chatbot, assistant, AI agent) inherits all the vulnerabilities of a web application, plus its own specific risks: prompt injection, leakage of sensitive information, agents with overly broad privileges, and uncontrolled injection of responses into other systems. The OWASP Top 10 for LLM Applications lists these risks. They are tested just like everything else: within an authorized scope and using real exploitation attempts.
Why AI Applications Are a New Attack Surface
A language model does not fundamentally distinguish between instructions from its developer and the data it reads: an email, a web page, or a document may contain text that gives it new commands. As soon as the model is connected to internal data or tools (sending an email, querying a database, creating a ticket), this design flaw becomes a concrete risk.
Key Risks (OWASP Top 10 for LLM Applications)
| Risk | What Can Happen |
|---|---|
| Prompt Injection | Malicious text—whether typed by the user or hidden in a document read by the model—can hijack its instructions |
| Disclosure of Sensitive Information | The model reveals its system instructions, data from its knowledge base, or information from other users |
| Supply Chain | Compromised model, dataset, or third-party extension |
| Data poisoning | Training data or data from the knowledge base is manipulated to skew the responses |
| Incorrect handling of outputs | The model’s response is inserted as-is into a page (XSS) or a command (injection) |
| Excessive Agency | An agent has overly broad privileges and can be manipulated to act on the attacker’s behalf |
| System prompt leakage | Internal instructions, sometimes containing secrets, are extracted |
| Weaknesses in vectors and embeddings | Retrieval-Augmented Search (RAG) returns documents to which the user should not have access |
| Misinformation | The model generates false information presented as reliable |
| Unlimited Resource Consumption | Massive numbers of costly requests: denial of service or skyrocketing bills |
How to Test the Security of an AI Application
- Map out what the model can read (documents, databases, the web) and what it can do (tools, APIs, permissions).
- Attempt direct and indirect injections: instructions within the conversation, in a sent file, or on a page the agent visits.
- Verify isolation: Can one user access another user’s documents or conversations?
- Testing outputs: Can the model’s response trigger an XSS or an unexpected request?
- Verify agent permissions: Does every sensitive action require human confirmation?
- Test the application surrounding the model—authentication, APIs, and access control—just as you would for any web application.
Best Practices
- Least privilege: The agent has only the rights strictly necessary, with user-level permissions rather than administrator privileges.
- Human validation is required for any irreversible action (payment, sending, deletion).
- Treat all model output as untrusted data: escape, validate, and do not execute directly.
- Filter documents from the knowledge base based on the user’s permissions before feeding them to the model.
- Never include any secrets in the system prompt.
- Limit throughput and cost per user.
Frequently asked questions
Can prompt injection be completely prevented?
Not yet: this is a fundamental limitation of language models. Risk is mitigated by restricting what the model can read and do, and by requiring human validation for sensitive actions.
Does a standard penetration test cover a chatbot?
It covers the surrounding application. Risks specific to the model (prompt injection, document leakage, agent privileges) require dedicated test scenarios.
Sources
Find out where an attacker would get in.
Our AI agents actually attack your scope, fully automated, and hand you the proof of every vulnerability.
