AI features that look cheap in a prototype can become a significant line item at scale. The good news: most teams can cut API spending substantially with techniques that barely affect quality. Here are the most effective, roughly in order of effort.

1. Measure first

You cannot optimize what you don’t measure. Log for every request: the model used, input tokens, output tokens, cached tokens, latency and the feature that triggered it. Most savings come from a few high-volume features.

2. Use prompt caching

Many applications send the same long prefix with every request: system instructions, policies, examples, or a large document. Providers including Anthropic, OpenAI and Google offer prompt or context caching, which bills repeated prefix tokens at a steep discount and also reduces latency.

To benefit, put stable content first and variable content (the user’s question) last, so the cached prefix is identical across requests.

3. Batch what can wait

Not every request needs an answer in seconds. Nightly classification, document processing, evaluations and data enrichment can run through batch APIs, which process requests asynchronously — typically within 24 hours — at a discount. OpenAI’s Batch API, for example, is priced at half the cost of synchronous requests.

4. Choose the smallest model that passes

Teams often default to the most capable model for everything. Build a test set for each feature and try smaller, cheaper models. For classification, extraction and simple rewriting, small models frequently match large ones at a fraction of the price.

5. Route by difficulty

A router sends easy requests to a cheap model and hard ones to a premium model. Routing can use simple rules (request type, length), a small classifier, or a cascade: try the cheap model first and escalate if confidence is low or validation fails.

6. Trim the context

Input tokens add up fast.

  • Summarize long conversation histories instead of resending every message.
  • Retrieve fewer, better passages in RAG systems; tune chunk size and the number of results.
  • Remove boilerplate from documents before sending them.

7. Control output length

Output tokens are the expensive ones. Ask for concise formats, set maximum output limits, and request structured data (JSON) instead of prose when your code will parse the result anyway.

8. Cache answers, not just prompts

If many users ask the same question, store the answer. Exact-match caching is simple; semantic caching matches similar questions using embeddings, but needs care to avoid serving wrong answers to subtly different questions.

9. Reconsider reasoning modes

Reasoning models and extended thinking improve hard tasks but can multiply token usage. Enable them only for the features that need them, and cap reasoning budgets where providers allow it.

10. Consider self-hosting at high volume

At very high, steady volume, running an open-weight model on your own or rented GPUs can become cheaper than per-token pricing — if you have the team to operate it.

Guard quality while cutting costs

Every optimization should be checked against your evaluation set. A 50% cost cut that silently lowers answer quality is not a saving.

Common mistakes to avoid

  • Optimizing the wrong feature. Without per-feature cost data, teams often tune a rarely used endpoint while a high-volume one drives the bill.
  • Breaking the cache with small changes. Putting a timestamp or user name at the start of a prompt prevents prefix caching. Keep variable parts at the end.
  • Routing without fallback. If the cheap model fails validation, the system should escalate automatically instead of returning a bad answer.
  • Caching answers that should not be shared. Semantic caches can leak one user’s personalized answer to another. Cache only generic, non-sensitive responses.
  • Skipping evaluation after changes. Every cost change should be tested against your evaluation set before release.

Key takeaways

  • Log tokens and costs per feature; a few features usually drive most spending.
  • Prompt caching and batch APIs offer large discounts with almost no quality impact.
  • Test smaller models per feature and route only hard requests to premium models.
  • Trim conversation history and retrieved context, and cap output length.
  • Validate every cost optimization against a fixed evaluation set.

Frequently asked questions

How much can prompt caching save?

It depends on how much of each request is a repeated prefix. Applications with long, stable system prompts or shared documents can see large reductions in input costs and latency.

Is a batch API suitable for chatbots?

No. Batch processing returns results asynchronously, often within hours, so it suits background jobs such as classification, evaluations and document processing rather than live conversations.

What is model routing?

Sending each request to the most cost-effective model able to handle it, for example a small model for simple questions and a frontier model for complex ones.

Does a shorter prompt reduce quality?

Not if you remove repetition and irrelevant material. Removing essential instructions or context does hurt quality, which is why changes should be tested.