A client came to us last year with a support-automation feature that was working beautifully and quietly eating about $14,000 a month in API spend. Their CTO's question was blunt: "Can we cut this in half without the answers getting worse?"
We got it to roughly a third. Not through one clever trick — through six unglamorous ones, applied in order of effort-to-payoff. That ordering is the actual skill here, and it's what this post is about.
Before anything else, though, a piece of advice that costs me money to give: if your monthly inference bill is under about $1,000, stop reading and go build features. Two senior engineers spending a week on AI inference optimization costs more than you'd save in a year. Optimisation is worth it when spend is material, growing, or when per-unit economics block your pricing model. Otherwise it's procrastination with extra steps.
Still here? Good. Let's talk about where the money actually goes.
Step zero: measure before you touch anything
Almost every team that asks us to reduce LLM costs has no per-request cost attribution. They have one number on a monthly invoice. That is not enough to make a decision with.
What you need, at minimum, logged per call:
- Input tokens, output tokens, model used, latency
- The feature or endpoint that triggered it
- Whether it hit a cache
- A trace ID linking multi-step chains together
Then build one table: cost per feature per day. In our experience it is almost never evenly distributed. Typically one or two code paths account for 60–80% of spend, and they're often not the ones the product team thinks about. We've seen a background "enrichment" job nobody had looked at in months outspend the entire customer-facing chat feature.
Also compute your real unit economic — cost per resolved ticket, per document processed, per active user per month. That's the number your CFO cares about, and it's the number that tells you whether a 20% saving is meaningful or noise.
The token math nobody looks at
Output tokens usually cost several times more than input tokens. Yet most teams obsess over shortening prompts while letting the model ramble in its responses.
Two fixes with immediate payoff:
Cap and shape the output. Ask for JSON with a defined schema. Set max_tokens deliberately rather than leaving it at the default. If you need a classification, the answer is one word — don't let the model write three sentences of preamble justifying it. We've seen output token counts drop 70% from schema enforcement alone, with no measurable quality change.
Kill chain-of-thought where it isn't earning its keep. Reasoning tokens are expensive. They genuinely help on multi-step logic, maths, and ambiguous extraction. They do almost nothing for sentiment tagging or routing. Test both variants on your eval set and let the data decide, per task.
On the input side, the biggest offender is usually RAG. Teams retrieve twelve chunks because more context feels safer. Measure it: in most pipelines we've audited, accuracy plateaus somewhere between three and five well-ranked chunks, and everything past that is pure cost plus a higher chance of the model getting distracted. Adding a cheap reranker before the LLM call often lets you cut retrieved context by half and improve answer quality.
Caching: the highest ROI thing you can do
Three layers, cheapest first.
- Exact-match cache. Hash the full prompt, store the response in Redis with a sensible TTL. Trivial to build. In consumer-facing products with repetitive queries, hit rates of 15–40% are common.
- Prefix / prompt caching. Most major providers now let you cache a stable prompt prefix — your system instructions, tool definitions, few-shot examples — at a large discount on repeat reads. If you have a 3,000-token system prompt sent on every call, this is close to free money. It requires only that you structure prompts with the static parts first.
- Semantic cache. Embed the query, look for a near-neighbour above a similarity threshold, return the stored answer. Powerful but genuinely risky: "cancel my subscription" and "don't cancel my subscription" can sit uncomfortably close in embedding space. Use it only where a slightly-off answer isn't harmful, keep the threshold conservative, and log every hit so you can audit.
Model routing: stop paying frontier prices for easy work
This is where the largest savings usually live. Frontier models can cost ten to thirty times more per token than capable small models. Most production traffic does not need the frontier.
A cascade works like this: send the request to a small model first, have it either answer or signal low confidence, and escalate only the hard cases. A router classifier — itself a tiny, cheap model or even a fine-tuned encoder — decides upfront which tier a request belongs to.
The honest caveat: this only works if you have evaluation infrastructure. Without a solid eval set you are not optimising, you are gambling with quality and finding out from angry customers. Build the evals first. A few hundred representative examples with agreed-on grading criteria is enough to start, and it's the single highest-leverage artefact in any serious LLM system.
A realistic target from routing alone, once evals are in place: 40–60% of traffic served by the cheap tier with no user-visible quality difference.
Fine-tuning a small model — and when self-hosting pays
For narrow, high-volume, repetitive tasks — classification, extraction, standardised summarisation — distillation is the endgame. Use your frontier model to generate a few thousand high-quality labelled examples, fine-tune a small open-weight model on them, and serve that.
Done properly, a fine-tuned small model matches or beats the big model on that one narrow task, at a fraction of the cost and with far lower latency. The trade-off is real though: you now own a model artefact, an evaluation loop, a retraining cadence and serving infrastructure.
On self-hosting, be sceptical of napkin maths that only counts GPU hours. The break-even is genuinely favourable at high, steady volume — roughly when you're spending five figures monthly on a single narrow workload and traffic is predictable enough to keep GPUs busy. Below that, idle capacity, on-call burden and the engineering time to tune batching, quantisation and autoscaling will erase the savings. We've told several clients to stay on APIs after running the numbers with them.
Architectural savings people forget
- Don't use an LLM where a regex, a lookup or a classifier works. Deterministic code is free, fast and testable.
- Batch asynchronous work. Batch endpoints typically run around half price for jobs that can wait hours.
- Deduplicate before processing. Document pipelines often re-embed and re-summarise near-identical content.
- Fail fast on garbage input. Validate before spending tokens on it.
- Watch agent loops. An agent that retries five times on a malformed tool call quietly multiplies your bill. Hard-cap iterations and alert on outliers.
Protecting quality while you cut
Every change above can degrade output. The discipline that keeps this safe is unremarkable and non-negotiable: a versioned eval set, automated scoring on every prompt or model change, shadow-running the cheaper path against the current one before switching, and a production sample reviewed by a human weekly. Track cost and quality on the same dashboard. The moment they're in separate reports, someone will optimise one and destroy the other.
What to expect
For a typical production system that's never been optimised, 50–70% cost reduction is a realistic target over a few weeks of focused work, with quality held flat or slightly improved. The first half usually comes from caching, output discipline and context trimming — days of work, not months. The rest comes from routing and distillation, which need the eval scaffolding first.
And if you get to the end and your spend is still high — that may simply be the correct answer. Some workloads genuinely need frontier reasoning. The goal was never the smallest possible bill; it was making sure every rupee or dollar you spend is buying something a customer can feel.
We've built and optimised LLM systems for teams across the US, UK, Europe and the Middle East, and we're happy to look at your traces and tell you honestly where the savings are — including when there aren't enough to justify the work. If you'd like a second pair of eyes on your inference costs or your AI architecture, get in touch at foogletech.com/contact-us.