Language models naturally produce prose. Applications need data: a category, a list of fields, an action with parameters. Asking politely for JSON works most of the time — and “most of the time” is exactly what breaks production systems. Modern APIs offer two features that make outputs reliable: structured outputs and function calling (also called tool use).

The problem with “please answer in JSON”

Without constraints, models occasionally:

  • add explanations before or after the JSON;
  • forget required fields or invent extra ones;
  • use the wrong types (“3” instead of 3);
  • produce invalid syntax in long outputs.

Each failure needs special handling in your code.

Structured outputs: schema-constrained generation

Several providers let you supply a JSON Schema describing the exact structure you want. The model’s generation is then constrained so the output conforms to the schema. Example schema for extracting an invoice:

{
  "type": "object",
  "properties": {
    "vendor": { "type": "string" },
    "total": { "type": "number" },
    "currency": { "type": "string", "enum": ["EUR", "USD", "GBP"] },
    "due_date": { "type": "string", "description": "ISO date, YYYY-MM-DD" }
  },
  "required": ["vendor", "total", "currency"],
  "additionalProperties": false
}

Schema features such as enums, required fields and descriptions guide the model and restrict invalid values.

Function calling: letting the model choose an action

Function calling describes tools your application can execute. Each tool has a name, a description and a schema for its arguments:

  • get_weather(city, unit)
  • search_orders(customer_id, status)
  • create_ticket(title, priority)

You send the user’s message plus the list of tools. The model either answers directly or returns a tool call with structured arguments. Your code executes the function, sends the result back, and the model produces the final answer. This is the foundation of AI agents.

Validate anyway

Even with constrained decoding, validate on your side:

  1. Parse the JSON.
  2. Validate against the schema with a library such as Pydantic (Python) or Zod (TypeScript).
  3. Check business rules the schema can’t express — totals that should match line items, dates in the future.
  4. On failure, retry with the validation error included in the prompt, or route to a fallback.

Design tips

  • Keep schemas small and flat where possible; deeply nested structures are harder.
  • Name fields clearly and add descriptions — the model reads them.
  • Use enums for categories instead of free text.
  • Allow “unknown” values so the model doesn’t invent data when information is missing.
  • Write precise tool descriptions that say when to use each tool and when not to.

When to use which

  • Structured outputs when you always want data in a fixed format, such as extraction and classification.
  • Function calling when the model must decide whether and which action to take.

Common mistakes to avoid

  • Trusting JSON without validation. Constraint features reduce errors but your code should still validate types and business rules.
  • Forcing a value when the data is missing. Required fields with no “unknown” option push the model to invent values.
  • Overloading one call with dozens of tools. Too many similar tools confuse selection; group them or route first.
  • Executing tool calls blindly. Check arguments and permissions before running actions that modify data.

A worked example: support ticket triage

Imagine routing incoming support emails. Define a schema with four fields: category (enum: billing, technical, account, other), priority (enum: low, normal, urgent), summary (string, max one sentence) and customer_id (string or “unknown”).

For each email, request structured output against that schema. Your code then:

  1. validates the result;
  2. routes the ticket to the right queue based on category;
  3. alerts on-call staff when priority is urgent;
  4. looks up the customer when an ID was found.

Then add one tool, lookup_order(order_id), and allow the model to call it when an email mentions an order. The model decides when the lookup is needed, your code performs it, and the final structured result includes the order status. Measure accuracy on a hundred labelled historical emails before going live.

Key takeaways

  • Plain instructions to “answer in JSON” fail occasionally — enough to break applications.
  • Structured outputs constrain generation to a JSON Schema you provide.
  • Function calling lets the model choose tools and return structured arguments for your code to execute.
  • Always validate with a schema library and business rules, and retry on failure.
  • Small schemas, clear names, enums and “unknown” options improve reliability.

Frequently asked questions

Do all models support structured outputs?

Support and strictness vary by provider and model. Check the documentation; where strict modes are unavailable, rely on prompting plus validation and retries.

Is JSON mode the same as structured outputs?

Not exactly. A JSON mode usually guarantees valid JSON syntax, while structured outputs additionally enforce a specific schema.

Can function calling run code automatically?

No. The model only proposes a call with arguments; your application decides whether and how to execute it.