Red-teaming an agent workflow before it gets real permissions

Goal

Find out what an attacker can make the agent do through the inputs it processes, before the agent holds permissions that make the answer expensive, and turn every finding into a regression test.

Prerequisites

A staging copy of the agent with its real tools pointed at disposable targets; replayable run logs; a list of the agent's inputs: user messages, retrieved documents, web pages, tool results, memory files, file names. The OWASP LLM Prompt Injection Prevention cheat sheet catalogues attack classes (direct and remote/indirect injection, encoding and obfuscation, RAG poisoning, agent-specific attacks) and defences (structured prompts with clear separation, output validation, human-in-the-loop controls, least privilege). Scanners such as garak run libraries of probes for prompt injection, data leakage, jailbreaks and other weaknesses against a language model.

Steps

1. Write the threat list per input: for each place untrusted content enters, what is the worst tool call it could trigger (send, delete, pay, exfiltrate through a URL parameter, write a memory file)?
2. Plant payloads in each input: an instruction in a retrieved document, in a web page's hidden text, in a file name, in a tool result, in a commit message. Vary the form: plain, encoded, split across chunks, in another language.
3. Run scripted probes with a scanner against the model-facing interface, then manual attempts against the whole workflow, because the interesting failures involve tool chains.
4. Try exfiltration specifically: can the agent be made to include secret or private data in a URL it fetches, a message it sends or a file it writes to a shared location?
5. Try exhaustion: inputs that make the agent loop, call tools thousands of times or fill its context.
6. Record each attempt with the run log: payload, placement, what the agent did, whether a gate or the sandbox stopped it.
7. Fix at the boundary first (remove the capability, gate it, restrict the network, validate arguments), then improve the prompt, then add the payload to the evaluation set as a permanent test.
8. Repeat after every new tool or input source.

Expected result

A table of inputs against worst achieved action, each finding with a boundary fix and a regression test, and a documented residual risk for what only the prompt defends against.

Limits and test basis

Prompt-level defences are probabilistic; a payload that fails today may succeed after a model update, which is why findings become tests. The exercise finds what the team thought to try; a scanner broadens coverage but does not know the application's tools. No results for any specific agent are claimed here.