Domain 5 – Governance, Safety & Risk Management (14%)

• Guardrails and safety controls (input/output filtering, refusal handling)
• Identify LLM-specific risks, limitations, and failure modes
• Human-in-the-loop validation patterns
• Regulatory compliance (GDPR, HIPAA, FedRAMP, etc.)
• Ethical considerations: bias, fairness, transparency, accountability

CCAR-P Domain 5 · Task 5.1

What You Need to Know

A guardrail is a control that stops a Claude system causing harm. Each risk belongs on the one surface that can hold it.

Five Control Surfaces

1. Input controls — screen what enters (injection, PII, out-of-scope)
2. Instruction-layer — system prompt scope, refusal policy, escape hatch
3. Capability controls — which tools exist, credentials, second-party approval
4. Output controls — schema/policy checks, blocked claims, citations, redaction
5. Architectural removal — decline to build the capability at all

Key Principles

• An instruction is not an access control. If a capability is wired, a prompt not to use it is only a request.
• Untrusted input is everything the system did not author — retrieved docs, tool results, uploads (indirect injection).
• A guarantee has only two homes: the output gate, or a capability you never built.
• Defence in depth — layers independent so one failing does not put harm in front of a user. Removal still wins for irreversible external actions.

Structural Mitigations for Untrusted Input

• Untrusted content arrives inside tool results with declared origin
• System prompt states tool content is data, not commands
• Tool output screened before the model acts
• Access scoped so a successful injection has somewhere small to go

Exam Traps

• Strengthening the system prompt to forbid a behaviour the system remains wired to perform
• Moving to a more capable model to stop policy violations or resist injection
• Logging / periodic review against a “must never reach a user” requirement
• Adding confirmation/approval in front of a capability the role never needs

Practice Scenario

Pharmacy customer assistant. Clinical governance: no reply may ever state a dose, frequency or duration; those questions must reach a pharmacist. Which change satisfies the requirement?

Build Exercise: Map the Control Surfaces for One Claude System (45 min)

CCAR-P Domain 5 · Task 5.2

What You Need to Know

A risk register is an architecture artefact. Every row has five fields:

1. Mechanism — what goes wrong
2. Blast radius — how far, who bears it, whether undoable
3. Detection signal — what moves in the telemetry
4. Owning control — which 5.1 surface carries it
5. Residual, and who accepted it

Silent Failure Modes

• Confident fabrication does not fail loudly — never shows in an error rate
• Non-determinism breaks one-shot tests and audit re-runs (need the recorded trace)
• Injection blast radius = the agent’s permission set
• Autonomy compounds error — levers are scope, step count, reversibility, checkpoints

Three Rows Registers Usually Omit

1. Over-refusal — safety controls paid for in legitimate requests denied
2. Over-reliance downstream — rubber-stamping; automation bias
3. Third-party surface — connectors, MCP servers, subprocessors

Ranking

Rank by consequence and detectability, not likelihood. A rare irreversible silent failure outranks a frequent reversible loud one. Any row whose detection signal is “a customer would complain” has no real detection.

Exam Traps

• Accepting a high accuracy figure as the risk posture for a system that takes actions
• Ordering register rows by expected frequency
• Adding a disclaimer to an action that has already left the organisation
• Recording a human review step as the owning control with no override rate behind it

Practice Scenario

Agent chases overdue invoices: reads ledger, drafts email, sends it, can apply late-payment charge. Risk assessment: “tested on 200 historical invoices, 98% accuracy.” Sponsor wants to know what a wrong email/charge would cost and how the team would notice. What should the architect require before launch?

Build Exercise: Write the Risk Register for One Claude System (45 min)

CCAR-P Domain 5 · Task 5.3

What You Need to Know

Placement comes before pattern. The gate belongs where consequence is high and the action is hard to reverse.

Five Patterns

Pattern	Use when
Approve-before-execute	Irreversible external effects (payments, deletions, outbound messages)
Review-before-release	Consequential content (regulated advice, clinical docs, commitments)
Triggered escalation	Routine handled automatically; uncertain/high-value routed to person (validate the trigger)
Post-hoc sampled audit	High-volume, reversible, low-severity per item
Automate with reversibility guarantee	Audit trail + undo path makes the action cheap to reverse

Some Gates Are Not Discretionary

Anthropic Usage Policy High-risk Areas (legal/medical advice; credit, insurance, housing, employment decisions) require: (1) qualified person reviews before delivery/decision, (2) person concerned is told AI was used. Check what the output does and to whom.

Making the Reviewer Effective

• Reviewer must check rather than routinely apply (ICO test)
• Give them the trace, sources, flagged uncertainty, at least one input the model never had, and time
• Measure: override rate, reviewer agreement, time per item
• Override rate near zero at high volume = rubber stamp

Key Concept: The mixed answer is usually the professional one — split the flow by reversibility; one pattern applied uniformly fails half the requirement.

Exam Traps

• Requiring human approval for every action (throughput forces rubber-stamping)
• Placing the human check at a periodic retrospective when harm has already left
• Strengthening a gate by adding a second/more senior reviewer (same information)
• Escalating on an unvalidated model confidence score

Practice Scenario

Payroll assistant investigates pay queries and can issue corrections. Small corrections net off against next monthly run (reversible until close); immediate off-cycle payments leave the organisation. Single manager cannot check every correction; no payment may leave unchecked. Right oversight design?

Build Exercise: Design and Stress-Test the Human-Oversight Layer (45 min)

CCAR-P Domain 5 · Task 5.4

What You Need to Know

An obligation becomes an architecture constraint when it changes what you build. Scope is the whole data path, not the application.

Scope: Every Hop

• Request to model provider
• Retrieval index and embeddings
• Prompt and response logs
• Observability traces
• Evaluation datasets
• Analytics/monitoring vendors and their subprocessors

Traces and eval sets are the most commonly forgotten copies.

Minimise at the Boundary

Send only the fields the task needs. Redact, tokenise or pseudonymise before crossing a trust boundary; keep the re-identification key inside. De-identification of training/eval data may also be required depending on risk.

Agreements Precede the Data Flow

Have the contract before the data moves. HIPAA BACs require: permitted uses, safeguards, breach reporting, subcontractor flow-down, and return-or-destroy at termination (or continued protection of what cannot be returned).

Purpose, Retention, Residency, Rights

Purpose and retention are design inputs for every copy. Data-subject rights (access, correction, deletion, portability) require finding every copy — vector indexes, traces, and eval fixtures are where deletion requests miss.

FedRAMP

A FedRAMP authorisation certifies the service completed the process; it is not an authorisation to operate. The agency’s authorising official still decides. Choose services already authorised at the required impact level, or keep the component inside the boundary the official will authorise.

Key Concept: When a stem describes too much data crossing a boundary, sort options by what still crosses afterwards. Only one changes that number.

Exam Traps

• Adding privacy notice/encryption/shorter retention/access limits while the same fields keep crossing
• Drawing the assessment around the user-facing application while logs/traces/indexes/eval sets sit outside
• Putting the processor agreement in place at the vendor’s next contract renewal
• Treating a supplier’s FedRAMP authorisation as the agency’s own ATO

Practice Scenario

Occupational-health provider: assistant answers employer queries using employee health assessments. Design streams retrieved assessment excerpts to a third-party observability vendor, and keeps an eval dataset of real assessments indefinitely in a shared analytics account. Provider may not disclose health data without a contract, or keep it longer than needed. What should the architect require?

Build Exercise: Trace the Compliance Path for One Regulated System (45 min)

CCAR-P Domain 5 · Task 5.5

What You Need to Know

Bias is a gap in outcomes between groups. Dropping the protected attribute proves nothing — proxies carry the signal.

Bias Is Measured in Outcomes

Proxies: working hours, occupation, application channel, address history, language register. Strong aggregate accuracy can hide one cohort failing badly. Choose cohorts from a harm hypothesis, not data availability. Fairness is a documented choice of criterion (equal error rates / equal selection rates / calibration) tied to the harm — three criteria cannot generally all hold at once.

Remediation Ladder (cheapest first)

1. Fix the data — rebalance representation, examine labels
2. Fix the framing — change what the model is asked to decide; is AI the best solution?
3. Constrain the decision — score/rank rather than decide; require human adjudication
4. Monitor continuously — live metric, threshold, owner

Not rungs: hiding the score; blaming the foundation model’s training data.

Transparency: Three Obligations

1. Disclosure to the user at the point of use — tell them they are dealing with AI; for High-risk AI Recommendations, that AI produced the recommendation
2. Explainability to the affected person — outcome-based (reasoning for this decision in everyday language, human involvement, contest route), not process-based architecture description
3. Traceability to the organisation — which version, prompt, retrieved context; retained long enough to answer a challenge months later

Accountability

A named human/role owns the outputs. Automating a decision does not move accountability onto the software. Senior management signs off intended use against risk appetite.

Exam Traps

• Concluding the system is fair because no protected attribute is collected
• Recording the foundation model’s training data as the root cause and documenting the limitation
• Offering a model card / policy page / terms paragraph as the disclosure obligation
• Explaining a decision by describing the model, pipeline or architecture

Practice Scenario

Household-insurance assistant triages claims into fast-track or manual investigation. No demographic data collected. Claims via telephone routed to investigation far more often than identical app claims; telephone claimants skew older/more likely disabled. Investigation adds six weeks. What should the architect do?

Build Exercise: Write the Ethics Page for One System That Decides About People (60 min)