Sovra Recon·Guide · AI Security

Chatbot and AI Application Security: The Risks and How to Test Them

By Solan Barlet, co-founder of Sovra Recon
Updated October 29, 2026

In short

An application based on a language model (chatbot, assistant, AI agent) inherits all the vulnerabilities of a web application, plus its own specific risks: prompt injection, leakage of sensitive information, agents with overly broad privileges, and uncontrolled injection of responses into other systems. The OWASP Top 10 for LLM Applications lists these risks. They are tested just like everything else: within an authorized scope and using real exploitation attempts.

Why AI Applications Are a New Attack Surface

A language model does not fundamentally distinguish between instructions from its developer and the data it reads: an email, a web page, or a document may contain text that gives it new commands. As soon as the model is connected to internal data or tools (sending an email, querying a database, creating a ticket), this design flaw becomes a concrete risk.

Key Risks (OWASP Top 10 for LLM Applications)

RiskWhat Can Happen
Prompt InjectionMalicious text—whether typed by the user or hidden in a document read by the model—can hijack its instructions
Disclosure of Sensitive InformationThe model reveals its system instructions, data from its knowledge base, or information from other users
Supply ChainCompromised model, dataset, or third-party extension
Data poisoningTraining data or data from the knowledge base is manipulated to skew the responses
Incorrect handling of outputsThe model’s response is inserted as-is into a page (XSS) or a command (injection)
Excessive AgencyAn agent has overly broad privileges and can be manipulated to act on the attacker’s behalf
System prompt leakageInternal instructions, sometimes containing secrets, are extracted
Weaknesses in vectors and embeddingsRetrieval-Augmented Search (RAG) returns documents to which the user should not have access
MisinformationThe model generates false information presented as reliable
Unlimited Resource ConsumptionMassive numbers of costly requests: denial of service or skyrocketing bills

How to Test the Security of an AI Application

  • Map out what the model can read (documents, databases, the web) and what it can do (tools, APIs, permissions).
  • Attempt direct and indirect injections: instructions within the conversation, in a sent file, or on a page the agent visits.
  • Verify isolation: Can one user access another user’s documents or conversations?
  • Testing outputs: Can the model’s response trigger an XSS or an unexpected request?
  • Verify agent permissions: Does every sensitive action require human confirmation?
  • Test the application surrounding the model—authentication, APIs, and access control—just as you would for any web application.

Best Practices

  • Least privilege: The agent has only the rights strictly necessary, with user-level permissions rather than administrator privileges.
  • Human validation is required for any irreversible action (payment, sending, deletion).
  • Treat all model output as untrusted data: escape, validate, and do not execute directly.
  • Filter documents from the knowledge base based on the user’s permissions before feeding them to the model.
  • Never include any secrets in the system prompt.
  • Limit throughput and cost per user.

Frequently asked questions

Can prompt injection be completely prevented?

Not yet: this is a fundamental limitation of language models. Risk is mitigated by restricting what the model can read and do, and by requiring human validation for sensitive actions.

Does a standard penetration test cover a chatbot?

It covers the surrounding application. Risks specific to the model (prompt injection, document leakage, agent privileges) require dedicated test scenarios.

Sources

Find out where an attacker would get in.

Our AI agents actually attack your scope, fully automated, and hand you the proof of every vulnerability.