7.1 — AI Application Security

CCDV-F Domain 7 · Task 7.1

What You Need to Know

Prompt injection is text that tries to give Claude instructions it should not follow. One context, no channel separation — nothing structural separates commands from data.

Two Threat Models

Model	Adversary	Mitigations
Direct / jailbreaks	The user of your application	Harmlessness screen, input validation, system prompt boundaries, policy for repeat offenders; metadata.user_id (hashed opaque ID)
Indirect	Whoever influenced content Claude reads on the user’s behalf (email, web page, uploaded file, tool result)	The class that matters in agentic applications

Put Untrusted Content in a Tool Result

1. Deliver third-party content inside tool_result blocks, never in system prompts or plain user text
2. Say what the content is and where it came from
3. State the untrusted-content policy in the system prompt
4. JSON-encode the untrusted string where possible
5. Do not put your own instructions inside a tool result — send them in a following user turn

Screen What Comes Back, in Code

Inbound: harmlessness screen with a lightweight model (Haiku) + structured outputs for a boolean verdict. Outbound from a tool: screen raw output before returning as tool_result. A classifier is probabilistic — sits alongside deterministic controls, never instead of them.

Least Privilege Over the Tool Surface

Three moves (not interchangeable):

1. Remove the tool — strongest; Claude does not see it
2. Scope the tool — deny matching calls in every permission mode
3. Narrow the credential — read-only where reading is enough; end-user entitlements, not service account

Evaluation order: hooks → deny rules → ask rules → permission mode → allow rules → canUseTool callback. Auto-approved tools never reach canUseTool. For a check that must run on every call, use a PreToolUse hook.

Everything in Context Can Leave Through the Response

Rank the fixes: (1) don’t put the sensitive value in context; (2) output check for what must never appear; (3) telling the model to keep a secret (fragile). Constrain the outbound path — which tools can send, fetch or write, and to where.