The AI Cost Problem Is Bigger Than Tokens

Most companies are going to have an AI cost problem. Some already do, although they may not yet have enough visibility to recognize it.

AI usage is spreading across products, internal tools, individual employees, and functional teams. Marketing buys an AI platform, engineering builds against model APIs, customer support introduces automated agents, and employees use a growing collection of assistants to complete everyday work. Each decision may be reasonable on its own, but the combined result is a new category of spending that is distributed throughout the organization and difficult to explain.

The natural response is to start measuring tokens. Tokens are visible, quantifiable, and directly connected to many model invoices, so they provide a useful starting point. But they only show how much model activity occurred. They do not explain why the activity was needed, whether the work was designed well, whether the right model was selected, or whether the resulting output created enough value to justify the expense.

The AI cost problem is therefore bigger than tokens. It involves four connected questions: What work is creating demand for AI? What level of intelligence does that work require? How should the company access or operate that intelligence? And what business result makes the total expense worthwhile?

These questions are often assigned to different departments, which is part of the problem. Engineering considers architecture and model performance. Finance sees the growing bill. Procurement negotiates with providers. Operations deals with the actual workflow. Product owns the user experience, while legal and risk focus on what can go wrong. Each group sees an important part of the picture, but no one naturally owns the economics of the complete system.

Start with the work

AI costs rarely appear because someone makes one obviously irresponsible decision. They build gradually through hundreds of choices that seem inconsequential at the time.

A team adds an AI-generated summary to a workflow. A developer supplies more context because it improves answer quality. An agent retries a task several times when it fails. Employees purchase overlapping products because the company has not established a preferred tool. The strongest available model becomes the default because testing several alternatives feels like additional work.

Individually, none of these choices is necessarily wrong. At scale, however, they can create a large amount of consumption without a clear connection to business value.

Before a company attempts to optimize its AI bill, it needs to understand what is producing the demand. Is the model extracting information from documents, classifying inbound requests, drafting responses, reviewing legal cases, analyzing customer behavior, or helping employees search internal information? How frequently does the task occur? How much context does it require? How many calls are needed to complete it? What happens when the model fails? Does someone need to review the answer before it can be used?

These distinctions matter because two activities using the same model can have completely different economics. A relatively expensive model may be justified when it helps an attorney evaluate a consequential case, supports a high-value commercial decision, or removes hours of specialized labor. The same model may be unnecessarily costly when it is used thousands of times to summarize routine internal updates that few people read.

This is why the company needs to measure more than cost per token or cost per API call. A model call is not an outcome. It is one input into producing an outcome.

The more useful questions are about the cost of completed work. What does it cost to qualify an opportunity, process a claim, resolve a support issue, review an application, produce a usable proposal, or reach a decision? That total may include multiple model calls, data retrieval, workflow orchestration, software fees, human review, and the handling of exceptions.

Until those pieces are connected, the organization may know what it is spending without knowing what it is buying.

Match the model to the work

Once the workflow is visible, the company can ask what level of capability each step actually requires.

Most AI workflows contain several types of work. Some steps are deterministic and can be handled more reliably by conventional software, rules, calculations, or database lookups. Others involve straightforward extraction, classification, or formatting that a smaller model may perform well. A smaller number require synthesis across multiple sources, reasoning through ambiguity, or making recommendations that carry real consequences.

The mistake is treating the whole workflow as one undifferentiated AI task. When that happens, the most capable model often becomes the default, even though much of the work does not require it.

That does not mean the answer is always to choose the cheapest model. A lower-cost model that frequently produces weak answers, requires repeated attempts, or creates additional human review may be more expensive by the time the work is complete. Model quality, latency, failure rates, and operational burden all affect the real cost of the outcome.

A better standard is to use the least expensive system that can reliably meet the required level of quality, speed, and risk. For some tasks, that will still be a frontier model. For others, it may be a smaller model, a deterministic system, conventional automation, or a person.

The answer may also change within a single workflow. Routine requests can be processed cheaply, while ambiguous cases are escalated to a stronger model. High-risk decisions can require human review. Expensive reasoning can be reserved for the moments when it materially improves the result.

This is not simply model optimization. It is the deliberate allocation of different forms of intelligence across the work.

Decide when to rent intelligence

Selecting the right model does not settle the economic question. The company must also decide how it will access and operate that model.

For many organizations, the default is a managed API. This is often the right place to begin. APIs provide immediate access to capable models without requiring a company to purchase hardware, operate GPUs, manage model-serving infrastructure, or build a specialized team to maintain the system. For experimentation, variable demand, and lower-volume workloads, that flexibility can be far more valuable than a lower theoretical cost per inference.

The tradeoff is that the company is renting intelligence. It pays as the capability is consumed, and its costs grow with usage.

As consumption increases, companies will begin asking whether some workloads should move to open-weight models running on dedicated cloud infrastructure or on-premises compute. This can look like an obvious cost-saving move, but self-hosting does not eliminate the expense. It changes its structure.

Instead of paying primarily through an API bill, the company assumes responsibility for infrastructure capacity, utilization, deployment, monitoring, security, maintenance, energy, and technical labor. An API may be expensive per unit while remaining operationally simple. A self-hosted model may appear cheap per inference while becoming expensive because the hardware sits underused or the organization needs a highly specialized team to operate it reliably.

The economics depend on the shape of the workload. Self-hosting becomes more plausible when demand is large, consistent, and predictable enough to keep infrastructure well utilized. It may also make sense when the company needs tighter control over sensitive data, lower latency, greater customization, stronger availability guarantees, or compliance with data-residency requirements.

For experimental, highly variable, or unusually complex work, managed access may continue to be the better choice. The likely answer for many organizations will be a portfolio rather than a complete commitment to one approach.

A company may continue using frontier APIs for difficult reasoning and unpredictable demand, while directing routine work toward smaller managed models. High-volume, repeatable tasks may eventually move onto dedicated infrastructure. Sensitive workloads may operate in a controlled cloud or on-premises environment.

The important question is not whether APIs or self-hosting are universally better. It is when the company should rent intelligence and when the scale, sensitivity, or strategic importance of the workload justifies operating more of that intelligence itself.

That is as much a capital-allocation decision as a technical one.

Open models create options, not automatic savings

Open models are an important part of this discussion because they give organizations more control over how models are deployed, customized, and operated. They can reduce dependence on a single provider and make it possible to run certain workloads inside infrastructure the company controls.

However, “open source” is often used too broadly. Many models are more accurately described as open-weight models. Their trained parameters are available for use or modification, but the complete training data, methods, or development process may not be accessible.

That distinction matters when a company evaluates licensing, security, reproducibility, and long-term control. It also matters economically. Access to model weights does not remove the need to deploy the model, evaluate it, secure it, monitor it, and maintain the infrastructure around it.

An open-weight model may remove a direct per-token charge from the original provider, but it does not make inference free. It simply gives the organization another way to supply the capability.

That option may be highly valuable. It is still an option that needs to be evaluated against the real workload rather than treated as an automatic answer to rising API costs.

Connect the cost to value

A company can become very good at tracking tokens, comparing models, and evaluating infrastructure without answering the most important question: What is the business receiving in return?

The value of AI will not always appear directly as new revenue. It may show up as reduced review time, lower operating cost, improved conversion, faster customer response, increased employee capacity, lower risk, better decisions, or a capability that could not previously be provided at an acceptable cost.

But the intended result should still be defined. Without it, cost management becomes disconnected from the reason the system exists.

This is where fragmented ownership becomes especially damaging. Engineering may reduce model costs while creating more work for operations. Procurement may negotiate a lower token rate while the company continues generating unnecessary demand. Finance may impose limits that reduce spending but also weaken a valuable workflow. Product may launch an impressive feature that few customers use enough to justify maintaining.

Each function can optimize its own part of the system while the overall economics become worse.

A serious AI cost-governance model needs to connect the work creating the demand, the model performing it, the infrastructure supplying it, the people responsible for the outcome, and the value the company expects to receive.

That does not necessarily require a large new governance organization. It does require clear visibility and ownership.

The company should know what AI systems and tools it is paying for, which workflows generate the spending, what useful outcomes those workflows produce, and who is accountable for their performance. It should be able to compare different models and deployment options using total cost rather than unit price alone. Material use cases should have quality thresholds, budgets, escalation rules, and a regular process for determining whether the original economic assumptions still hold.

Those assumptions will change. Models improve, prices move, workloads grow, and experiments become production systems. A low-volume use case can quickly become one of the company’s largest sources of AI demand. A workload that initially made sense through an API may eventually justify dedicated infrastructure. A self-hosted system that looked attractive on paper may fail to achieve the utilization required to make it economical.

The design of the system needs to evolve with the work.

The goal is intentional consumption

AI cost governance should not become a program for minimizing model usage or slowing adoption. The purpose is not to force every team onto the cheapest available model or require financial approval for every experiment.

The goal is to make consumption intentional.

A company should understand where expensive intelligence creates leverage and where it creates waste. It should know when a smaller model performs well enough, when a frontier model materially improves the outcome, and when conventional software is more reliable than AI. It should know when API flexibility is worth paying for and when a mature, predictable workload justifies dedicated or self-hosted compute.

Most importantly, it should be able to explain what the business is receiving in return.

The companies that manage this well will not necessarily use less AI. They will have a clearer understanding of where to use it, how to supply it, what to pay for it, and how to determine whether it is working.

That is the larger discipline emerging behind token management. It is not simply an exercise in reducing costs. It is the operating system companies will need to allocate intelligence well.