For most of the last decade, cloud cost management meant rightsizing virtual machines, buying commitments and deleting forgotten storage. Generative AI has broken that comfortable routine. A single product feature can now generate a bill driven by tokens, model choice, prompt length, retrieval calls, provisioned throughput and GPU hours, and each of those levers sits with a different team. Finance sees a fast-growing line item it cannot forecast, while engineering sees a model that works well and does not want to be told to switch it off.
This article explains why AI spend needs its own FinOps discipline, what has changed in 2025–2026, and how to build a practical operating model: allocation, unit economics, capacity decisions, guardrails and forecasting. It is written for platform leads, FinOps practitioners and product owners who are responsible for AI features that are already in production or about to be.
Why AI Spend Has Become the Center of FinOps
The FinOps Foundation’s State of FinOps 2026 survey, whose respondents together manage more than $83 billion in cloud spend, shows how quickly the scope has shifted. According to the report summary published by USU, 98% of organizations now say they actively manage AI costs, up from 63% in 2025 and 31% in 2024. In the same survey, practitioners reported that the easy cloud savings have largely been captured and that leadership increasingly wants value metrics such as unit economics and return on investment rather than raw savings.
In other words, nearly everyone is now “managing” AI spend, but the maturity of that management varies enormously. For many teams it still means a monthly look at the invoice for a model API and a spreadsheet of GPU reservations. That is not enough, for three reasons:
- The cost drivers are new. Tokens, context windows, embeddings, vector database reads and inference endpoints are not well represented by traditional tagging and rightsizing practices.
- The behavior is nonlinear. A small prompt change, a new retrieval step or an agent that loops through tools can multiply cost per request overnight, with no infrastructure change at all.
- Value is harder to prove. A database cluster supports a known workload; an AI feature has to show that its outcomes justify a cost that grows with every user interaction.
What Makes AI Costs Different
The FinOps Foundation has spent much of 2025 codifying how AI pricing actually works. Reporting on that work, Techzine highlighted several characteristics that make AI costs harder to manage than conventional cloud services. It is worth understanding them before designing any controls.
Token-Based Pricing Is Opaque and Model-Specific
Most generative AI services bill per input and output token, and each model uses its own tokenizer and rate card. The same prompt can cost very different amounts on two models, and output tokens are typically priced higher than input tokens. Teams that compare models only on headline price per million tokens often miss the real driver: how many tokens a given workflow actually consumes once system prompts, retrieved context and multi-turn conversation history are included.
Capacity Models Trade Price Against Flexibility
Providers increasingly offer reserved or provisioned capacity for model inference, alongside pay-as-you-go tokens. Reserved capacity lowers the effective unit price but caps throughput; traffic above the cap may be throttled, while unused capacity is still billed. This is the classic commitment trade-off, but with a twist: AI demand is spiky and hard to forecast, so the risk of paying for idle capacity, or of rate-limiting customers at peak, is higher than for a steady web workload.
GPU Infrastructure Is Expensive and Often Idle
Teams that self-host models or fine-tune them pay for GPU instances whose hourly rates dwarf general-purpose compute. Utilization is frequently poor. Cast AI’s 2026 State of Kubernetes Optimization report, based on tens of thousands of production clusters, measured average GPU utilization of just 5% in the clusters it analyzed before optimization, and found that fewer than 2% of GPUs ran on Spot capacity for most of 2025. Whatever the exact figure in your own estate, idle accelerators are usually the single largest source of AI waste.
Attribution Is Genuinely Hard
A shared model endpoint may serve a chatbot, a document summarizer and an internal coding assistant at the same time. Unless requests carry metadata about who sent them and why, the provider bill shows one large number with no way to split it by product, customer or experiment.
A Five-Layer Operating Model for AI FinOps
Rather than bolting AI onto existing cloud cost reports, we recommend treating AI spend as a distinct domain with five layers. Each layer builds on the one beneath it, so the order matters.
- Visibility: ingest model API, GPU, vector database and AI platform charges into the same cost dataset as the rest of your cloud spend, ideally in the FinOps Open Cost and Usage Specification (FOCUS) format.
- Allocation: attribute every token and GPU hour to a product, feature, team, environment and, where relevant, customer.
- Unit economics: express cost per business outcome, such as cost per resolved support ticket or per generated document, not just cost per token.
- Optimization: tune model choice, prompts, caching, batching and capacity based on the unit metrics.
- Governance and forecasting: set budgets, guardrails and approval paths, and forecast based on usage drivers rather than last month’s invoice.
Layer by Layer: Building Visibility and Allocation
Normalize the Data First
AI costs typically arrive from several places: a hyperscaler bill for managed model services and GPU instances, separate invoices from model providers, charges from data platforms such as Databricks or Snowflake, and SaaS subscriptions for AI tools. Microsoft’s FinOps team has published a worked example showing how Azure OpenAI token costs can be analyzed through the FinOps toolkit using FOCUS columns such as ConsumedQuantity, PricingUnit, BilledCost, ChargeCategory, ResourceId and tags. The value of that approach is that token consumption becomes just another usage quantity, so cost per token can be calculated by dividing billed cost by consumed quantity, and compared across models and deployments.
Tag at the Request Level, Not Just the Resource Level
Resource tags work well for dedicated GPU clusters and per-team model deployments. For shared endpoints, you also need request-level attribution. The most reliable pattern is an AI gateway that sits between applications and model providers and records, for each call, the calling application, feature, tenant, model, input tokens, output tokens, cache hits and latency. Those logs can then be priced using the provider rate card and reconciled against the invoice each month. Without a gateway, teams end up reverse-engineering allocation from API keys, which breaks as soon as keys are shared.
Decide on Showback Before Chargeback
Start by showing teams what their AI features cost, without billing them for it. Showback builds trust in the numbers and surfaces allocation errors. Move to chargeback only when the data has been stable for a few months and product owners agree on the allocation rules for shared items such as embeddings pipelines, evaluation runs and platform overhead.
Defining AI Unit Economics That Leaders Will Use
Cost per million tokens is an engineering metric. Executives need to know whether an AI feature is worth what it costs. A good AI unit metric connects cost to an outcome the business already tracks. Examples include:
- Cost per resolved customer conversation, compared with the cost of a human-handled contact.
- Cost per processed invoice or claim in a document automation workflow.
- Cost per active user per month for an AI assistant embedded in a SaaS product, compared with the revenue that user generates.
- Cost per successful agent task, counting retries and tool calls, not just the final answer.
The numerator should include all variable costs for the workflow: model tokens, retrieval and vector search, orchestration compute, guardrail and evaluation calls, and a fair share of GPU capacity if you self-host. The denominator must be a successful outcome. Counting attempts rather than successes hides the cost of failures and retries, which in agentic workflows can be substantial.
Treat cost per successful outcome, not cost per token, as the north-star metric for every AI feature. It aligns engineering, product and finance on the same question: is this capability worth what it costs to run?
Optimization Levers That Actually Move the Needle
Once unit costs are visible, optimization becomes a structured exercise rather than guesswork. The levers below are ordered roughly by the effort they require.
Model Routing and Right-Sizing
Not every request needs a frontier model. Route simple classification, extraction and short-answer tasks to smaller or cheaper models, and reserve the most capable models for complex reasoning. Routing should be driven by evaluation results on your own data, so quality does not silently degrade. Small language models, either hosted by a provider or self-hosted, are increasingly viable for narrow tasks.
Prompt and Context Discipline
Long system prompts and generous retrieval windows are a common hidden cost. Trim boilerplate instructions, limit retrieved chunks to what improves answer quality, summarize conversation history instead of resending it in full, and cap output length where the use case allows. Many providers now offer prompt caching for repeated prefixes, which can reduce the cost of large, stable system prompts.
Batching and Asynchronous Processing
Workloads that do not need an instant response, such as nightly document enrichment or bulk classification, are good candidates for batch inference interfaces, which providers typically price below interactive calls. Moving these jobs out of the interactive path also frees capacity for latency-sensitive traffic.
Capacity Strategy for Inference
Use pay-as-you-go for new and unpredictable workloads. Consider reserved or provisioned throughput once a workload has a stable baseline, and size the commitment to that baseline rather than the peak, letting overflow spill to on-demand. Review utilization of committed capacity monthly, exactly as you would for compute savings plans.
GPU Utilization for Self-Hosted Models
For teams running their own models, the biggest lever is utilization: sharing GPUs between smaller workloads, queueing training jobs so clusters stay busy, using Spot or preemptible capacity for checkpointed training, and scaling inference endpoints to zero when idle. Kubernetes has matured significantly here; our companion article on cloud infrastructure for AI workloads covers scheduling and GPU sharing in detail.
Guardrails and Forecasting Without Slowing Innovation
The goal of governance is not to stop experimentation; it is to make the cost of experimentation visible and bounded. A practical guardrail set looks like this:
- Give every AI project a budget and an owner, with alerts at 50%, 80% and 100% of monthly budget.
- Enforce per-application and per-tenant token quotas at the gateway, so a runaway agent loop or a misbehaving integration cannot consume a month of budget in a day.
- Require a cost estimate, based on expected request volume and tokens per request, before a new AI feature goes to production.
- Run anomaly detection on daily AI spend by feature, since AI cost spikes rarely coincide with infrastructure changes and are easy to miss.
- Review model choices quarterly, because price-performance changes quickly as providers release new models and adjust rate cards.
Forecasting should be driver-based. Instead of extrapolating the invoice, model expected requests per user, tokens per request and the model mix for each feature, then multiply by the rate card. When a product manager plans to double adoption of an AI assistant, finance can see the cost implication immediately, and engineering can show how routing or caching changes would offset it.
Worked Example: An AI Support Assistant
Consider a mid-sized software company that launched an AI assistant in its customer support portal. In the first quarter, the monthly model bill grew steadily and finance asked whether the feature should be scaled back. The team followed the five layers above.
First, it routed all assistant traffic through an AI gateway and tagged each call with feature, environment and customer tier. This revealed that roughly a third of tokens came from the staging environment, where automated regression tests ran full conversations against the most expensive model on every build. Moving those tests to a smaller model, and running the full suite only nightly, removed that waste without affecting production quality.
Second, it defined cost per resolved conversation, using the support platform’s resolution flag as the denominator. That metric made it clear that long conversations with repeated retrieval were the main driver. The team summarized history after a few turns, cut retrieved context to the top few passages, and introduced prompt caching for its stable system prompt.
Third, it introduced routing: simple account and billing questions went to a smaller model, while technical troubleshooting stayed on the larger one. Evaluations on a labeled set of past tickets confirmed resolution quality held steady. Finally, once traffic stabilized, it reserved capacity for the weekday baseline and left peaks on pay-as-you-go.
The outcome was not just a smaller bill. Finance gained a metric it could compare with the fully loaded cost of a human-handled ticket, and product leadership could make an informed decision to expand the assistant to more customer tiers.
Common Pitfalls to Avoid
- Optimizing tokens while ignoring quality. Every cost change must be validated with evaluations; a cheaper model that resolves fewer tickets is more expensive per outcome.
- Leaving experiments running. Fine-tuning jobs, notebooks attached to GPU instances and idle inference endpoints are frequent sources of waste.
- Treating AI as a separate silo. AI costs belong in the same FinOps practice, dataset and review cadence as the rest of technology spend.
- Over-committing too early. Reserved inference capacity sized to launch-week peaks often sits idle once usage normalizes.
- Ignoring agent behavior. Agents that call tools and models in loops need step limits and per-task budgets, or cost per task can become unbounded.
Outlook: Where AI FinOps Is Heading
Several developments will shape AI cost management through 2027. The FinOps Foundation has said that future versions of FOCUS will expand into AI and data center spending scopes, which should make token and GPU charges easier to compare across providers. Its FinOps for AI certification track signals that AI cost management is becoming a recognized professional skill rather than an ad hoc task. And as agentic systems move into production, cost per task, with explicit budgets per agent run, is likely to become a standard design constraint alongside latency and accuracy.
The organizations that benefit most will be those that connect AI costs to business outcomes early, so that cost conversations become investment conversations rather than cutbacks.
Conclusion
AI spend is now a core part of the FinOps remit, but it cannot be managed with traditional cloud tactics alone. Start with clean, normalized data and request-level allocation, define unit metrics tied to real outcomes, and then apply the optimization levers that matter most for your workloads: model routing, context discipline, batching, sensible capacity commitments and GPU utilization. Wrap it all in lightweight guardrails and driver-based forecasting. Done well, AI FinOps does not slow innovation; it gives teams the evidence they need to scale the AI features that genuinely pay off.
How Hanoi Technology Can Help
Hanoi Technology helps teams build AI cost visibility, gateway-based allocation and unit economics so AI features scale profitably.
Further Reading
- 6 Takeaways from the State of FinOps Report 2026 – USU
- Why Is AI So Hard to Cost? FinOps Foundation Codifies Price Mechanics of AI – Techzine
- Managing Azure OpenAI Costs with the FinOps Toolkit and FOCUS – Microsoft Tech Community
- 2026 State of Kubernetes Resource Optimization – Cast AI
- FinOps Foundation Launches FOCUS 1.3 – Linux Foundation