AI Agent Security for Enterprise: A Practical Threat Model for Tool-Using Agents
A useful AI-agent security review starts with authority: what an agent can read, decide, call and commit. This practical threat model turns that question into controls and tests.
AI-drafted, machine-checked against our published editorial rules, and reviewed on publication. Corrections: [email protected] How we use AI
Article
An AI agent is not secure because the model has a safety policy. It is secure only to the extent that the surrounding system limits what the agent can read, what tools it can call, what actions it can commit and how a human can investigate a failure.
That distinction matters when an agent moves beyond drafting text. A system that can search internal documents, create tickets, update a CRM or initiate a payment has authority. Security review should therefore start with the authority boundary, not with a generic prompt.
Model the agent as an identity with tools
Give each agent a dedicated identity, not a shared administrator credential. Its permissions should be no broader than the particular workflow requires. A support agent may read a customer’s current case and create an internal escalation; it should not be able to export every customer record or alter billing details.
Split read, propose and commit operations where possible. An agent can retrieve a supplier record, prepare a reconciliation proposal and show its evidence before a finance user approves the final adjustment. This reduces the harm of a bad instruction without removing useful automation.
Treat external content as untrusted input
Agents are vulnerable to instructions hidden in documents, web pages, emails and tickets. Those sources may be relevant to the user’s question, but they are not authorised to redefine the agent’s objective or persuade it to use a connected tool.
OWASP’s Agentic Top 10 identifies agent goal hijack and memory or context poisoning as material categories of risk. Its guidance describes indirect prompt injection as untrusted instructions embedded in content that can redirect an agent toward data exposure or misuse of tools. OWASP’s Agentic Top 10 is a valuable taxonomy, but it is not a substitute for testing your own connectors and task flows.
Practical controls include separating data from instructions, restricting tool schemas, stripping or quarantining unexpected content, and requiring a user confirmation for irreversible or external actions. Do not pass a retrieved document to a model with the same authority as the system prompt that defines the task.
Put an enforcement point in front of every consequential tool
An agent should not call a production tool directly just because it produced plausible JSON. Insert a policy and validation layer between reasoning and action. That layer can validate identifiers, calculate permitted ranges, ensure the requester has authority, block unexpected destinations and decide whether a human must approve.
For example, an onboarding agent can propose accounts to provision. The enforcement layer verifies that the employee record exists, that the requested applications match the approved role and that the manager approval is current before it makes any call. If the model suggests an unsupported application, the system routes the case to a reviewer rather than guessing.
This is also where sandboxing belongs. Browser or shell work should use an isolated account and environment with a narrow task scope. Secrets should be supplied only to the specific connector that needs them, not placed in the model context or a broadly accessible environment variable.
Make every action reconstructable
Logs should answer: which user asked for the work, which sources were retrieved, which tool was requested, what policy decision was made, what was committed, and what the downstream system returned. Avoid treating the entire model transcript as the audit trail; it may contain unnecessary sensitive information. Store the minimum structured evidence that lets an investigator reproduce the decision.
NIST’s AI Risk Management Framework and its Generative AI Profile frame risk management as a lifecycle activity, covering governance, mapping context, measurement and management. The documents are voluntary guidance, not a compliance certificate, but they are useful for turning “we need safe AI” into specific operating practices. See the AI RMF and Generative AI Profile.
Test attack paths before rollout
Include hostile and ambiguous cases in the evaluation set. Try a document that asks the agent to ignore its task. Try a tool response with an unexpected field. Revoke an entitlement mid-task. Send a request that is valid syntactically but exceeds an approved business threshold. Check that the system refuses safely, records why and leaves the system of record unchanged.
The goal is not to prove that an agent will never fail. It is to ensure a failure has a small blast radius, a visible trail and a recovery path. Application-specific AI evaluations are how those controls are tested against the actual workflows a team intends to automate.
A practical starting architecture
Begin with one workflow, one bounded identity, a small set of tools and a clear approval point. Add retrieval with access-aware filtering, a validation layer for tool calls, structured logs and a test pack containing both normal and adversarial cases. Expand authority only after the evidence shows the system is reliable enough for the next boundary.
QQuantum.ai’s security architecture work covers the engineering layers behind that design: isolation, permissions, secrets, logging and prompt-injection defence. If an agent is about to touch a real system of record, start a technical conversation before giving it broad credentials.
