Trust

AI security architecture.

Agentic systems add a threat surface that conventional application security does not cover — because the component making decisions can be talked into things by the data it reads.

THE THREAT MODEL

Traditional application security assumes code does what it was written to do. An LLM does what its context suggests, and its context includes content from documents, emails, tickets and web pages that an attacker may control. That is the difference, and every control below follows from it.

The governing principle: never let the model's judgement be the only thing standing between an attacker and a capability. If an agent can delete records, the protection cannot be that it has been instructed not to. It must be that it lacks the permission, or that deletion requires a confirmation the model cannot forge.

THE CONTROLS

Least privilege per tool

Every tool gets the narrowest scope that lets it work. Read-only by default; writes scoped to specific records; destructive operations either removed or gated behind explicit confirmation. The audit question is always "what is the worst this tool can do", not "what is it meant to do".

Identity propagated end to end

Tool calls are authorised against the end user's permissions, not a service account's. Without this, a system with broad credentials makes every user an effective administrator of whatever it can reach — the confused-deputy problem, and in our experience the most common serious flaw in existing deployments.

Untrusted content stays data

Anything retrieved — documents, tickets, web pages, emails — is structurally separated from instructions, and the system is designed so that it cannot authorise a tool call. This is the primary prompt-injection defence, because content filtering cannot keep up with an attack surface the size of everything you read.

Secrets never enter context

Credentials live in a secret manager and are resolved inside the tool, after the model has decided to call it. A model that never sees a key cannot leak one, and anything in context should be assumed extractable.

Output validated before it acts

Model output that reaches another system is schema-validated and range-checked first. Constrained decoding where the provider supports it. Never eval, never raw SQL, never a shell command assembled from generated text.

Everything is logged

Prompts, retrieved context, tool calls and outputs, with a correlation ID per interaction. Without full traces an incident cannot be investigated — you cannot tell what the system saw, so you cannot tell what it leaked.

WHAT WE LOOK FOR IN A REVIEW

When reviewing an existing AI system, these are the findings we would expect to see most often, in our experience:

A service account with far more access than any user. The agent connects with admin credentials because that was simplest during development, and per-user authorisation was never added.

Retrieved content treated as trusted. A document or ticket body concatenated into the prompt with no structural separation, so anyone who can create a document can issue instructions.

Tools with destructive capability and no confirmation. Usually added for a legitimate reason during a sprint and never revisited.

No logging of tool calls. Outputs are logged, the actions taken are not — so the interesting half of an incident is invisible.

Unbounded agent loops. No iteration cap, so a confused agent can burn a very large amount of money before anyone notices, which is a denial-of-wallet vector as much as a reliability problem.

QUESTIONS

QUESTIONS — 4

No, and be sceptical of anyone claiming otherwise. It can be made largely inconsequential, which is the achievable goal: if a successful injection cannot reach a capability that matters, it is a curiosity rather than an incident. That is why the defence is authorisation rather than detection — you are designing so that being fooled does not cost anything.

Sometimes, with scoping. An agent that updates a ticket status is low risk; one that can delete records or move money should not have that capability at all, or should produce a proposed action that a human commits. The design question is what the worst outcome is if the model is fully compromised — then ensure that outcome is acceptable.

Minimise first: send only the fields the task needs, which is usually far fewer than the convenient full record. Redact where the task permits it. Where PII must be processed, use zero-retention endpoints or self-hosted models, and make sure logging does not create a second copy in a system with weaker controls — which is a surprisingly common accidental data flow.

Yes, as a standalone engagement. Typically one to two weeks producing a written findings document with severity ratings and concrete remediations, focused on the agentic layer specifically — the areas a conventional application penetration test is not scoped to cover.

Keep reading

Model Context Protocol

→

Architecture

Secure Data Pipelines

→

30-minute technical call · no deck