CompoundLearn

Structured Output and Tool Calling Interview Prep

Structured output and tool calling best practices — six-technique ladder, schema design, validate-and-retry, Instructor vs native structured outputs, and production pitfalls.

Quick answer

Structured output and tool calling sit at the interface between unstructured prose and typed, reliable code.

Structured output and tool calling are the bridge between LLM reasoning and reliable, downstream-parseable results.

Editorial review

Written by

CompoundLearn editorial team

Wireless / RF / hardware engineering

Reviewed by

CompoundLearn editorial team

Wireless / RF / hardware engineering

Last reviewed

Built from curated topic maps, editorial validation, and subject-matter review so the page stays aligned with the interview intent and the current content pipeline.

Key points

  • Structured output and tool calling bridge unstructured prose and typed code in two directions: extracting a document into a record, and constraining an LLM generation to a schema downstream code can trust.
  • Six techniques form a strictness ladder: prompt-and-parse, JSON mode, function calling (the production default), native structured outputs, grammar-constrained decoding, and validate-and-retry layered on top.
  • The schema is the contract — write it first; Instructor and BAML generate the prompt text from the Pydantic or Zod schema, so every required and optional field is a real decision.
  • A 50-field single-shot schema scores lower than the same schema split into a multi-step pipeline (classify, extract entities, extract nested details) — the standard pattern for medical, legal, and financial extraction.
  • Over-strict schemas (everything required, no optionality) make retry loops exhaust on real-world inputs; mark required only what downstream code genuinely depends on.
  • Return structured tool errors instead of crashing the loop, keep validate-and-retry bounded, and pair extraction with a critic or deterministic cross-check because schema validity does not prove the values are correct.

What it is

Structured output and tool calling sit at the interface between unstructured prose and typed, reliable code. Two directions over the same machinery: information extraction converts a free-form document into a schema-conforming record (a resume becomes ResumeFields, an invoice becomes LineItem[]), while structured output constrains an LLM's own generation to a schema so downstream code receives typed fields it can rely on. Both directions use six techniques arranged on a strictness ladder: 1. **Prompt-and-parse** — tell the model the schema in the prompt, parse with json.loads. Cheap and brittle; fails on every model-format change. Tutorial-grade only. 2. **JSON mode** — provider APIs constrain output to syntactically valid JSON, but not to a specific schema. Useful as a base requirement; insufficient on its own. 3. **Function calling / tool use** — the model emits a tool_use block with typed arguments matching the declared JSON Schema. By default this is best-effort (arguments can be malformed or omit required fields); strict / structured-output modes add constrained decoding to actually guarantee schema conformance. The production default since 2023. 4. **Native structured outputs** — Anthropic and OpenAI both ship guaranteed-schema modes backed by constrained decoding. Strictest hosted option; the right escalation when schema errors reach production. 5. **Grammar-constrained decoding** — Outlines, llama.cpp grammars, XGrammar constrain token sampling at inference time to a JSON schema or grammar. Requires running a model locally; highest reliability for open-weight deployments. 6. **Validate-and-retry** — generate output, validate against a Pydantic or Zod schema, send the validation error back to the model on failure, loop up to N times. Instructor is the canonical Python implementation; works with any underlying technique. Production teams typically live at Tier 3 (tool use) and escalate selectively to Tier 4 for high-stakes outputs or Tier 5 for local open-weight models. The schema is the contract. Write it first, before writing the prompt — Instructor and BAML literally generate prompt text from the Pydantic or Zod schema. Every required field, every type, every optional field you mark Optional is a decision that affects what the model can and cannot produce on real-world inputs. Production reality: a 50-field schema asked in a single shot produces lower accuracy than the same schema broken into a three-stage multi-step pipeline — classify document type, extract top-level entities, extract nested details. Most production medical, legal, and financial extraction pipelines use this pattern.

Why interviewers ask

Structured output and tool calling are the bridge between LLM reasoning and reliable, downstream-parseable results. Interviewers use this topic to separate engineers who have only used structured output in tutorials from engineers who have debugged it in production. The hard part is not getting JSON out of a model — JSON mode handles that. The hard part is schema design that matches real-world input variability, failure modes that manifest silently (schema-conformant fabrication, schema drift between extraction and consumer), and knowing when to escalate from tool use to native structured outputs to grammar-constrained decoding. Strong candidates can: choose between the six techniques on the strictness ladder for a given constraint; design a Pydantic schema with field-level descriptions, appropriate optionality, and confidence tracking; explain why validate-and-retry should be bounded and what happens at exhaustion; describe how to catch schema-conformant fabrication that pure schema validation cannot detect; and connect tool-calling error handling to agent-loop reliability (returning structured errors rather than crashing). For agentic roles specifically, interviewers ask about tool schemas as part of the agent loop: how tool_use blocks and tool_result blocks structure the message history, what happens when a tool call fails, how parallel tool dispatch works, and what stop conditions prevent runaway agents. An engineer who cannot read a message-history trace and identify a tool-call mismatch or a missing tool_result block will struggle to debug production agents.

Common mistakes

**Over-strict schemas.** Every field required, types aggressively narrow (int instead of int | None), no Field descriptions. The model cannot satisfy the schema on real-world inputs where data is missing or ambiguous. Validation-retry loops run to exhaustion. Fix: be permissive with optionality at the boundaries; mark required only what downstream code genuinely depends on. **Mixing extraction with reasoning.** Asking the model to extract fields, score them, draft a response, and recommend an action in one call. The output schema grows, accuracy on every field drops, the call is slow. Fix: separate extraction (one schema, one call) from downstream reasoning (a second call that consumes the extracted record). **Silent retry on validation failure.** Validation fails; the code sends the same prompt again without the error message; the model produces the same broken output. Instructor and Marvin send the validation error back automatically — if you are rolling your own, do the same. **Forgetting the document-parsing layer.** Passing PDF bytes directly to an LLM and hoping. Tables flatten to gibberish, multi-column layout interleaves wrong, equations vanish. Fix: parse first with Marker, PyMuPDF, Unstructured.io, or Docling; pass clean text and structure to the model. **Schema drift between extraction and consumer.** The extractor produces phone_number: str; the database schema expects phone: str. The pipeline crashes on the first record. Fix: the extraction schema and the consumer schema should be the same Pydantic class, or generated from the same source of truth. **No confidence or provenance tracking.** A field is wrong — you cannot tell whether the extractor was uncertain (suggesting the extractor should have routed to review) or confident (suggesting the source document is wrong). Fix: store (value, confidence, source_span) per field for any extraction where errors have downstream consequences. **Treating LLM-extracted facts as ground truth.** An agent pays an invoice for the wrong amount because the extracted total was wrong and no cross-verification step caught it. Fix: pair extraction with an adversarial critic, deterministic cross-checks (do line items sum to the total?), or a human-review queue for low-confidence records. **Skipping schema versioning.** Schema v1 had experiences: list[Experience]. Schema v2 added Experience.salary. Old stored extractions lack the field; new code crashes on the missing field. Fix: version schemas with semver and default missing fields to None when reading old records.

Frequently asked questions

What is structured output and how does it differ from JSON mode?
Structured output constrains an LLM's generation to a schema — every field, every type, every optionality defined up front. JSON mode (available via most provider APIs) only guarantees syntactically valid JSON; it does not enforce the schema, so the model can emit the wrong fields and still pass the JSON-mode check. Native structured outputs (Anthropic and OpenAI both ship this) go further and use constrained decoding under the hood, guaranteeing schema compliance. The practical hierarchy is: JSON mode is a base requirement, not a complete answer; native structured outputs is the 95% solution; grammar-constrained decoding (Outlines, llama.cpp grammars) is the last 5% for local open-weight models.
What is tool calling / function calling, and why is it the production default?
Tool calling (also called function calling) is a provider mechanism where the LLM emits a tool_use block — a structured request to invoke a named function with typed arguments. The runtime executes the function, returns a tool_result block, and the LLM continues reasoning. It has been the dominant structured-output mechanism since 2023 because providers validate arguments against the schema internally and re-prompt on failure. Most production agentic systems live at this tier (Tier 3 in the six-technique ladder) and escalate to native structured outputs only when stricter schema enforcement is required.
What is validate-and-retry, and when should you use it?
Validate-and-retry is the pattern implemented by Instructor, Marvin, and BAML: generate output via a prompt or tool call, validate the result against a Pydantic or Zod schema, and on failure send the validation error back to the LLM with a request to fix the output. This loop runs up to N times. It works with any underlying technique and is the right default when you need schema enforcement across multiple model providers or open-weight models. The key discipline is bounding the retries — a schema the model cannot satisfy after three attempts will not satisfy on attempt seven. Failed outputs belong in a human-review queue, not an infinite retry loop.
Why does schema design quality determine extraction and tool-calling accuracy?
The schema is the contract between the LLM and the rest of the system. Schemas that grow organically — every field required, types aggressively narrow, descriptions absent — produce pipelines that fail on real-world inputs because the model cannot satisfy the constraints. Best practices: write the schema first before writing the prompt (the prompt is derived from the schema in Instructor and BAML); mark fields Optional when the underlying data can genuinely be absent; add Field(description=...) per field so the LLM has unambiguous extraction guidance; use enums for constrained choices. The canonical failure is a schema where every field is required and types are strict — validation-retry loops run to exhaustion on any messy real-world document.
What is grammar-constrained decoding and when does it earn its cost?
Grammar-constrained decoding (Outlines, llama.cpp grammars, XGrammar) constrains the token-sampling process at inference time to only emit tokens consistent with a regex, JSON schema, or context-free grammar. This gives the strictest possible schema enforcement because invalid tokens are excluded before they can appear. The cost is that it requires running the model locally (or via vLLM or Ollama) — it cannot be applied to hosted API calls from most providers. It earns its cost when: you are running an open-weight model locally and need absolute schema reliability; your schema includes recursive structures or large enums that even native structured outputs struggle with; or regulatory requirements make any schema-conformant fabrication unacceptable.
How do you catch schema-conformant fabrication?
Schema validation enforces shape, not faithfulness. A model can produce a perfectly valid Pydantic object whose field values are invented rather than extracted from the source document — 0% schema-validation failures while 15% of records are wrong is a real production pattern. The fix is pairing extraction with an adversarial critic that compares each extracted field against the source document (using NLI classification or a cross-model verifier). Field-level confidence and provenance tracking — storing (value, confidence, source_span) per field — surface which extractions the system was uncertain about before downstream code treats them as ground truth.
What is multi-step extraction and why does it outperform single-shot extraction on complex schemas?
Multi-step extraction breaks a complex schema into sequential stages — typically: (1) classify the document type, (2) extract top-level entities, (3) extract nested details. Each stage has its own schema, prompt, and optionally its own model. On a 50-field schema, asking the model to fill all fields in one shot produces lower quality than a two- or three-stage pipeline because each stage is narrower and easier for the model to satisfy accurately. Medical-record, invoice, and legal-contract extraction pipelines in production all use this pattern. The escalation trigger is when single-shot quality plateaus — switch to multi-step rather than adding more retries.
What is the document-parsing layer and why does it matter more than the LLM choice for PDF extraction?
Before an LLM can extract structured data from a PDF, image, or scanned document, the file must be converted into text and layout that the LLM can read. This upstream layer (PyMuPDF, Marker, Unstructured.io, Docling, Tesseract) is not an LLM operation — it is a classical parsing step. For PDFs with complex layouts (tables, multi-column text, equations, forms), the parsing layer determines what the LLM even sees. Passing raw PDF bytes to an LLM and hoping produces flattened tables, interleaved multi-column text, and missing equations. The single biggest quality improvement in a document extraction pipeline is often improving the parsing layer, not changing the LLM.
What OSS tools should you reach for first for structured output work?
Three opinionated starting points: (1) Instructor (Python) — wraps Anthropic, OpenAI, and open-weight endpoints to return validated Pydantic models with auto-retry on validation failure; the 80% solution and the shortest path from a Pydantic schema to typed LLM output. (2) Native provider structured outputs (Anthropic tool use with strict input schemas; OpenAI Structured Outputs) — the 95% solution for guaranteed schema compliance via constrained decoding. (3) Outlines — grammar-constrained decoding for open-weight models where absolute reliability matters. For TypeScript, Vercel AI SDK generateObject / streamObject with Zod schemas is the cleanest TS-first option. BAML is the schema-first language option that compiles to both Python and TypeScript.
How should you handle tool-call errors in the agent loop?
Surface tool failures as structured errors back to the LLM rather than swallowing exceptions or crashing. The pattern: return a tool_result block with is_error: true and a message describing the failure (error type, message). This lets the LLM decide whether to retry the same tool with different arguments, choose a different approach, or give up and report the failure to the user. An agent that crashes on tool failure instead of returning a structured error is the most common cause of silent hangs in production. The same applies to schema-validation failures — send the validation error back to the model; a silent re-prompt wastes tokens without diagnosis.

Related topics

Siblings

Essential AI-Native Skills for Structured Output and Tool Calling

Modern engineering work increasingly uses AI tools for design and code review, debugging, documentation, test and testbench generation, and workflow automation. The goal is not to let AI replace engineering judgment — it is to move faster while keeping verification discipline.

  • Use AI to explain unfamiliar code, logs, waveforms, datasheets, or test failures.
  • Break large problems into small, reviewable steps you can verify independently.
  • Ask AI for hypotheses, then validate them against tests, measurements, simulations, or lab data.
  • Version-control your analysis scripts, testbenches, and configs — keep changes small and reviewable.
  • Document your assumptions, design tradeoffs, and debugging decisions.
  • Verify AI output before trusting it: run the checks that fit the domain — unit tests, linters, simulations, or bench/lab measurements.
  • Review AI output for correctness, edge cases, and real-world consequences.

Next up: Structured Output and Tool Calling practice

The adaptive practice engine is already live for core wireless, RF, and ML domains. Structured Output and Tool Calling questions — covering production realities, not just framework syntax — are in active development. Join the early-access list to get them first.

One email when this topic launches. Nothing else. Unsubscribe in one click.