Calling a language model API in a demo takes three lines. Keeping it reliable for thousands of users is different: requests hit rate limits, providers return “overloaded” errors at peak times, long generations time out, and streams break halfway. Good client code turns these from outages into minor delays.

Know the failure types

Model provider APIs document their error codes. Typical categories:

  • 400-range client errors (invalid request, authentication, permission, request too large): retrying will not help — fix the request.
  • 429 rate limit: too many requests or tokens per minute for your tier. Retry after waiting; responses often include a header saying how long.
  • 500-range server errors and “overloaded” responses: temporary capacity or service problems. Retry with backoff.
  • Timeouts and dropped connections: especially for long outputs. Retry, or switch to streaming.

Treat each category differently instead of retrying everything blindly.

Retries with exponential backoff and jitter

Retry transient errors with increasing waits — for example 1, 2, 4, 8 seconds — and add random jitter so thousands of clients do not retry at the same instant. The AWS Architecture Blog’s classic analysis of exponential backoff and jitter shows why jitter dramatically reduces contention. Respect any retry-after header, cap the number of attempts, and give up with a clear error.

Most official SDKs already retry some errors automatically; know their defaults so you do not stack your own retries on top and multiply load.

Timeouts

Set explicit timeouts. Long generations may legitimately take many seconds, so separate:

  • connection timeouts (short);
  • time-to-first-token expectations;
  • total request timeouts (longer, depending on maximum output length).

Streaming responses let users see progress and let you detect stalled connections early.

Idempotency and side effects

Retrying a plain completion is safe. Retrying an agent step that sends an email or charges a card is not. Make tool executions idempotent (with unique operation ids), and record which steps completed before retrying a workflow.

Rate-limit management

  • Track your token and request usage against your tier’s limits.
  • Queue and smooth bursts instead of firing everything at once.
  • Use batch APIs for offline jobs; they are often cheaper and have separate limits.
  • Cache repeated prompts and use prompt caching for long shared prefixes.
  • Request higher limits before launches.

Fallbacks

For important features, plan what happens when the primary model is unavailable:

  • a smaller or older model from the same provider;
  • a second provider behind the same internal interface;
  • a graceful message and retry option for the user.

Test fallbacks regularly — an untested fallback usually fails when needed.

Validate outputs

Treat model output as untrusted input: validate JSON against a schema, check required fields, and retry or repair when validation fails. See structured outputs and function calling.

Observe everything

Log error types, retries, latency and tokens per request so you can see when reliability degrades — our guide to LLM observability covers what to record.

A minimal retry policy

A sensible default for many applications:

  • Retry on rate limits, overloaded and 5xx errors, and network timeouts.
  • Up to 4 attempts, with delays of roughly 1, 2, 4 and 8 seconds plus random jitter, or the server’s retry-after value when present.
  • Never retry 400-range client errors other than rate limits.
  • Stop retrying when the user has already left, for example when the browser request was cancelled.
  • Log every retry with the error type, so a spike becomes visible quickly.

Start with this policy, then adjust delays and attempts based on what your logs show.

What to watch next

  • Provider SDKs with smarter built-in retries and backpressure.
  • Gateways that route between models and providers automatically.
  • Service-level commitments for enterprise model APIs.
  • Batch and priority tiers that change how rate limits work.

Key takeaways

  • Classify errors: fix client errors, retry rate limits and server errors.
  • Use exponential backoff with jitter and respect retry-after headers.
  • Set explicit timeouts and stream long responses.
  • Make agent side effects idempotent before retrying workflows.
  • Plan, test and monitor fallbacks for critical features.

Frequently asked questions

Should I retry every failed request?

No. Retry rate limits, server errors and timeouts; fix invalid requests and authentication errors instead.

What is jitter?

A random variation added to retry delays so many clients do not retry at the same moment and overload the service again.

Do official SDKs handle retries?

Many retry some errors automatically. Check their defaults so you do not add duplicate retry layers.