This week, we’re going to dive into one of the hottest topics in Enterprise IT: how to rein in AI costs. Just a few months ago, everyone was talking about “tokenmaxxing” and creating leaderboards that incentivized employees to maximize their usage of AI. Recently, attention has shifted to curbing AI costs after firms starting receiving massive bills for their AI usage. Most notably, an unnamed company allegedly spent $500M on AI in one month. This guide will hopefully prevent your company from becoming a cautionary tale on runaway AI spend.

One of the biggest mistakes organizations make is using the most powerful model for every task. Frontier models are excellent for complex reasoning, coding, planning, and nuanced language tasks, but they are often overkill for simpler workloads.

For example, classifying support tickets, extracting structured fields, rewriting short text, or summarizing simple documents may not require the most expensive model available. A smaller or cheaper model may perform well enough at a fraction of the cost.

The best approach is to create model tiers. Use lightweight models for simple tasks, mid-tier models for general work, and premium models only when the task truly requires advanced reasoning or higher accuracy. This alone can dramatically reduce AI spend.

Model routing takes the idea of model selection one step further. Instead of manually deciding which model to use, a routing layer automatically sends each request to the most cost-effective model that can handle it.

For example, a simple FAQ question might go to a low-cost open source model (like DeepSeek or Qwen), while a complex legal contract analysis request goes to a more expensive and capable model. Some routing systems can evaluate prompt complexity, user intent, required accuracy, latency needs, and cost limits before selecting a model.

This is especially useful for high-volume applications such as customer support, internal chatbots, coding assistants, and document processing workflows. Recent AI infrastructure research and production guidance increasingly emphasize intelligent routing as a core cost-optimization pattern.

To reduce token usage, focus on keep prompts concise, remove redundant instructions, limit the amount of context sent to the model, and avoid requesting overly long responses when a shorter answer will do. You can also use structured prompts that tell the model exactly what format to return, reducing unnecessary output.

This is especially important in applications that use long system prompts or repeatedly include large amounts of background information. A prompt that seems harmless in testing can become expensive when executed thousands or millions of times.

Many AI applications send the same instructions, policies, examples, or reference material over and over again. Prompt caching allows providers to reuse repeated prompt content instead of processing it from scratch each time. Prompt caching can reduce latency and input token costs for eligible repeated prompt content and can reduce costs and latency for supported models and workloads.

Prompt caching can be especially valuable for applications with long system prompts, reusable instructions, standard templates, product catalogs, policy documents, or repeated conversation prefixes. To benefit from caching, structure prompts so repeated content appears consistently and avoid making unnecessary changes to the cached portion.

Prompt caching reduces the cost of processing repeated prompt content, but response caching can eliminate some model calls entirely. If your product’s users frequently ask the same/similar questions, you can store previous answers and reuse them when appropriate. This works especially well for FAQs, product information, documentation search, internal knowledge bases, troubleshooting steps, and policy questions.

There are two common approaches: exact-match caching and semantic caching. Exact-match caching returns the same answer when the same prompt is repeated. Semantic caching goes further by recognizing when two different prompts have the same meaning. For example, “How do I reset my password?” and “I forgot my password — what should I do?” may be close enough to share the same response. Semantic caching needs careful quality controls, but when implemented well, it can significantly reduce repeated inference costs.

Not every AI request needs to be answered instantly. Many workloads can run asynchronously, including document summarization, data extraction, content tagging, report generation, transcript analysis, and bulk classification.

Batch processing can lower costs by grouping requests and running them more efficiently. OpenAI’s Batch API, for example, advertises 50% savings on inputs and outputs for asynchronous jobs processed over a longer window.

The key is to separate real-time workloads from background workloads. A customer support chatbot may need immediate responses, but a nightly process that summarizes 10,000 documents probably does not. Moving eligible work to batch processing can create meaningful savings.

Retrieval-augmented generation, or RAG, is one of the most common ways to connect AI models to company data. Instead of putting everything into the prompt, the system retrieves only the most relevant pieces of information and sends those to the model.

RAG can save money by reducing the amount of context passed into each request. However, poorly designed RAG systems can do the opposite. If your retrieval system sends too many irrelevant chunks, duplicates content, or includes large documents unnecessarily, token costs can rise quickly.

To optimize RAG costs, tune chunk sizes, limit the number of retrieved passages, remove duplicate text, compress retrieved context, and measure whether additional context actually improves answer quality. More context is not always better.

As is the case with cloud costs, AI costs can spike quickly, especially when usage grows faster than expected or a bug causes repeated calls. Strong governance is essential.

Set daily, weekly, or monthly spending limits by application, team, environment, or customer. Create alerts for unusual usage patterns, high token consumption, repeated failures, and sudden increases in expensive model calls.

It is also wise to separate development, testing, and production environments. Developers experimenting with long prompts or expensive models should not have unlimited access to production-scale budgets. Budget controls prevent small mistakes from becoming large invoices.

A monthly AI bill is not enough information. To control costs, you need to understand where the money is going.

Track AI spend by product feature, customer, internal team, model, workflow, and request type. Measure cost per successful task, not just total spend. For example, a model call that costs $0.02 may be cheap for a high-value sales workflow but too expensive for a low-value autocomplete feature.

This type of observability helps teams make smarter decisions. You may find that one feature drives most of your AI bill, one prompt produces unnecessarily long outputs, or one customer segment uses far more tokens than expected.

At smaller usage levels, hosted AI APIs are often the simplest and most cost-effective option. But at higher volumes, it may make sense to explore fine-tuning, model distillation, quantization, or self-hosting.

Fine-tuning can reduce prompt size and improve performance on narrow tasks. Distillation can transfer behavior from a larger model to a smaller, cheaper model. Quantization can reduce the compute requirements of open models. Self-hosting can make sense when workloads are predictable, high-volume, and infrastructure expertise is available.

These options are not automatically cheaper. They introduce operational complexity, engineering overhead, monitoring requirements, and infrastructure costs. But for mature AI workloads with steady demand, they can produce major long-term savings.

Managing AI costs will continue to be top-of-mind for many IT and Finance departments for the foreseeable future. Unlike with cloud spend, there often times isn’t a silver bullet that will enable an organization to reduce its AI spend without any consequences. Striking a balance between reducing AI spend while maximizing AI-driven productivity gains will require careful analysis and planning.

Next week will feature the final newsletter in our FinOps series: reducing SaaS costs.