Scenario 6 of 6

Structured Data Extraction

Building a structured data extraction system: pulling information out of unstructured documents, validating output against JSON schemas, holding accuracy high, handling edge cases gracefully, and integrating with downstream systems.

Primary domains: Prompt Engineering & Structured Output · Context Management & Reliability

What this scenario tests

The most mechanical scenario and the one with the clearest right answers. Schema design through tool use, tool_choice, nullable fields, validation-retry loops and batch processing all appear directly. The reliability half is about knowing how good the output actually is — confidence calibration and sampling rather than a general sense that it looks fine.

Traps it is built to catch

Asking for JSON in the prompt instead of enforcing a schema through tool use. The first works until it does not, silently.
Making every field required. A field the model must fill invites a plausible invention when the document genuinely lacks the value — nullable is the honest option.
Reporting one accuracy number. Accuracy varies by field and by document type; a single average hides exactly the segment that is failing.

The most useful hour of preparation

Extract from a hundred real documents, validate against a schema, and measure per-field accuracy against a labelled subset. The gap between your expectation and the measurement is the lesson.

Lessons that cover this scenario

5 of the 30 task statements, across 2 domains.