Prompt Engineering & Structured Output Cheat Sheet — CCAR-F Domain 4

4.1 Explicit Criteria

Vague (wrong)	Explicit (correct)
"Be conservative"	"Report bugs and security vulnerabilities. Skip minor style preferences."
"Only report high-confidence findings"	Confidence routing comes after explicit criteria, never instead of them
Prose severity ("could cause system failures")	Concrete code example per severity level

Criteria live in the top-level system parameter (no "system" role in messages).
max_tokens is a ceiling, not a target. temperature 0.0–1.0, default 1.0 — even 0.0 isn't fully deterministic.
Two failure modes: hardcoded brittle logic vs vague high-level guidance. Target: minimal info that fully outlines expected behaviour.
High FP rate in ONE category destroys trust in ALL categories → temporarily disable, refine with code examples, re-enable.
Best feedback form: clear rules + which rule failed and why (e.g. linting). Start evals with ~20 queries.

4.2 Few-Shot Prompting

Trigger	Fix
Inconsistent formatting despite detailed instructions	Few-shot examples (NOT more instructions)
Inconsistent judgement on ambiguous cases	Few-shot examples with reasoning
Empty/null fields for data that exists	Few-shot examples across document structures

2-4 targeted examples. <2 = no pattern; >4 = wasted tokens / context rot.
Every example needs reasoning (why this action, not just what) → teaches generalisation, not pattern-matching.
"Pictures worth a thousand words" — curate diverse, canonical examples; never a laundry list of edge-case rules.
Classification tasks: pair few-shot with a tool enum field or structured outputs.
Not a few-shot problem: malformed JSON (→ tool_use), fabricated values (→ nullable fields), sum mismatches (→ validation loop).

4.3 Structured Output (the big one)

Reliability hierarchy: tool_use + JSON schema (eliminates syntax errors) > prompt-based JSON (no guarantees).

tool_choice	Behaviour	Default when
`auto`	May call a tool OR return text	tools provided
`any`	Must call a tool, model picks which	—
`{"type":"tool","name":"..."}`	Must call that tool	—
`none`	No tool use	no tools provided (0 extra tokens)

tool_choice: "any" alone ≠ schema-valid input. Add strict: true on the tool for BOTH guarantees.
Newer/separate: Structured Outputs (output_config.format) — constrained decoding on a plain JSON response, no tool call. GA on Claude 4.5+ only.
Schema limits: additionalProperties: false required for objects; NO recursive schemas, external $ref, minimum/maximum/multipleOf, minLength/maxLength; enums are primitives only.
First use of a schema = extra latency (grammar compiles); cached 24h from last use. Structured outputs add a hidden system prompt (slightly more input tokens every call).
Prefill: historical JSON-forcing hack → migrated to Structured Outputs. 400 error on last-turn prefill for Claude 4.6+ / Mythos Preview. (tool_choice any/tool API-side prefill still works.)
What tool_use eliminates: syntax errors. What it does NOT eliminate: semantic errors (sums, field placement, fabrication).
Nullable/optional fields = primary defence against fabrication. Add "unclear" (ambiguous) and "other" + detail string (extensible) to enums.

4.4 Validation, Retry, Feedback Loops

Retry message = original document + failed extraction + specific validation error. (API itself already retries invalid tool calls 2-3× with corrections — same principle, one layer down.)

Retries FIX	Retries CANNOT FIX
Format mismatches	Info genuinely absent from source
Structural/field-placement errors	Data only in an external, unprovided doc
Missed line items (math errors)	Knowledge the model lacks

Unfixable → flag for human review / return null. Do NOT keep retrying.
Self-correction schema: calculated_total vs stated_total (+ total_discrepancy flag); conflict_detected boolean for contradictory source data.
detected_pattern field on findings → track dismissal rates → prioritise prompt refinement (frequency × dismissal rate).
Feedback ranking: rules-based (best) > ... > LLM-as-judge (weakest — "not a very robust method", latency cost).
Schema syntax errors → eliminated by tool_use (4.3). Semantic validation errors → this task statement.

4.5 Batch Processing (airtight facts)

Fact	Value
Cost discount	50% of standard price, unconditional
Typical time	<1 hour (most batches)
Max processing window	24 hours (hard)
Latency SLA	None — best-effort
Size limit	100,000 requests OR 256 MB, whichever first → 413 if exceeded
custom_id	`^[a-zA-Z0-9_-]{1,64}$`; ALWAYS match results by custom_id — never assume order
processing_status	`in_progress` → `canceling` → `ended`
Result types	`succeeded`, `errored`, `canceled`, `expired`
Billed	Only `succeeded`
Results	`.jsonl` at `results_url`, populated only once `ended`; stream, don't bulk-download
Results retention	29 days from `created_at` (NOT `ended_at`)
Rejected params	`stream: true`; Threads `store` / `previous_thread_event_id`
Param validation	Asynchronous — errors surface only after the batch ends; dry-run first
Prompt cache tip	Use 1-hour TTL (batches often exceed 5 min)

Matching rule: synchronous = blocking workflows (pre-merge checks); batch = latency-tolerant (overnight/weekly reports). Manager wants "switch everything to batch" → keep blocking workflows synchronous.

SLA math: 30h SLA − 24h max processing = 6h buffer → submit ≥30h before deadline, every 4-6h.

Failure handling: identify by custom_id → resubmit ONLY failures with targeted fixes (chunking, simpler prompts, examples) → never resubmit the whole batch.

Sample-set first: refine prompts on 5-10 docs before the full run. 90% first-pass = 100 retries/1000 docs; 60% = 400 retries (4×).

4.6 Multi-Instance & Multi-Pass Review

Self-review (same session) retains reasoning context → confirms rather than challenges itself. Independent instance = fresh context, no bias toward code it just wrote (official guidance).
Writer/Reviewer pattern: Session A generates → Session B reviews fresh → Session A addresses feedback.
Adversarial review: fresh subagent sees only the diff + criteria, never the generating reasoning.
Second opinion: fresh model tries to refute the result — not just re-confirm.
Basis: subagent context isolation (own context window, only relevant info sent back).

Symptom (single-pass, multi-file)	Fix
Inconsistent depth across files	Per-file local analysis passes
Missed middle-file bugs	Per-file local analysis passes
Contradictory findings	Cross-file integration pass

Bigger context window does NOT fix this — the problem is attention quality, not capacity.
Integration pass checks: data flow between modules, contradictions across per-file findings, API contract violations.
Confidence routing: high → direct report; low → human review. Raw self-reported confidence is uncalibrated — treat like LLM-as-judge (documented as the weakest verification method).
Calibrate: run labelled validation sets, compare reported confidence to actual accuracy, set thresholds from data.
Judge the end state, not the process followed. Rubric example: factual accuracy, citation accuracy, completeness, source quality, tool efficiency. Start with ~20 queries; manual testing still catches what evals miss.