4.1 — System Prompts with Explicit Criteria

CCAR-F Domain 4 · Task 4.1

What You Need to Know

Vague instructions ("be conservative", "only report high-confidence findings") give the model no actionable decision boundary. Explicit categorical criteria define precisely what to flag and what to skip.

Explicit Criteria vs Vague Instructions

Vague: "Review this code. Be conservative. Only report high-confidence findings."

Explicit: "Flag comments only when claimed behaviour contradicts actual code behaviour. Report bugs and security vulnerabilities. Skip minor style preferences and local patterns."

The False Positive Trust Problem

High false positive rates in one category destroy trust in all categories. Temporarily disable high-FP categories while you rework their prompts. System-wide trust comes ahead of category completeness.

Severity Calibration with Code Examples

Define severity with concrete code examples, not prose descriptions.

Key Concept: Explicit categorical criteria always outperform vague instructions and confidence-based filtering.

Why Confidence-Based Filtering Fails

LLM self-reported confidence is poorly calibrated. Use explicit criteria first; confidence routing second.

Exam Traps

• Choosing "be conservative" or "only report high-confidence findings" as valid improvements
• Assuming confidence thresholds fix false positive problems
• Keeping all categories active while iterating on high-FP ones