Two years ago, “reasoning models” were a distinct and expensive category: specialized models that paused to think before answering, reserved for math, code and hard analysis. In 2026 that boundary has largely dissolved. The major model providers now ship models where thinking is on by default or adapts automatically, and they expose a single control, usually called effort or thinking level, that determines how much the model deliberates before it responds. Reasoning has become a dial rather than a model choice.
That shift matters far beyond model selection. Hidden reasoning tokens change how you budget, how you set timeouts, how you manage conversation state, how you evaluate quality and how you explain AI behavior to auditors. This article explains what changed, why it matters for enterprise generative AI, and how to design systems that use thinking deliberately rather than by accident.
What Changed: From Token Budgets to Effort Dials
Early reasoning APIs asked developers to set an explicit budget of thinking tokens. The current generation has converged on a simpler abstraction. A few examples from vendor documentation:
- OpenAI documents a reasoning effort parameter with levels ranging from none and minimal through low, medium and high to xhigh and max, depending on the model. Its guidance suggests the lowest settings for latency-critical work such as voice and classification, low for tool use and customer support, medium as a default for agentic coding and research, and the highest settings for deep research and security review.
- Google documents a thinking level parameter for Gemini models (minimal, low, medium and high), with thinking enabled by default on many models and adjusted to task complexity.
- Anthropic recommends adaptive thinking on recent Claude models, where the model decides when and how much to think based on an effort setting and the complexity of the request. Its documentation notes that the older approach of fixed thinking budgets is deprecated on newer models and that on some models thinking is always on.
The common pattern is clear: providers want developers to express intent (“be quick” or “be thorough”) and let the model allocate thinking. For enterprise teams, this is both a simplification and a new source of variability, because the same prompt can consume very different amounts of compute depending on how hard the model judges it to be.
Why Reasoning Tokens Change the Economics
Reasoning happens in tokens you usually cannot see. OpenAI’s documentation states that reasoning tokens are not visible through the API but occupy context window space and are billed as output tokens. Google’s documentation similarly states that thinking tokens are billed in full even when only a summary is returned. Three consequences follow:
- Cost per request becomes less predictable. A short visible answer may sit on top of thousands of reasoning tokens. Budgeting by visible output length systematically underestimates spend.
- Output caps behave differently. Both OpenAI and Google document that a maximum output setting caps reasoning plus visible output together. If the cap is reached while the model is still thinking, the response can come back incomplete with little or no visible text.
- Effort is the main cost lever. Google’s guidance is explicit that lowering the thinking level is a better way to reduce cost and latency than shrinking the output cap, which risks truncated answers.
This is why FinOps for AI increasingly tracks reasoning tokens as a separate line item. Usage objects from the major providers report reasoning or thought token counts, so you can attribute spend by feature, tenant and effort level instead of guessing.
Treat effort as a product decision, not a default. Every feature should have an explicit effort level, justified by evaluation results, with reasoning tokens tracked as their own cost line.
Latency, Timeouts and User Experience
More thinking means longer time to first visible token. For a back-office document review that runs overnight, that is irrelevant. For a customer-facing chat, a voice agent or an inline autocomplete, it can break the experience. Practical design responses include:
- Route by task. Use low or minimal effort for classification, extraction, routing and simple Q&A; reserve higher effort for planning, complex analysis and code changes.
- Stream thought summaries where appropriate. Some APIs can return summaries of the model’s thinking. Surfacing a progress indicator or a brief summary can make longer waits acceptable in analyst-facing tools.
- Move heavy reasoning off the critical path. Pre-compute analyses asynchronously, use background modes for long jobs, and cache results.
- Set timeouts with reasoning in mind. Gateways and client libraries tuned for older chat models may time out on high-effort calls, producing silent failures.
Reasoning State: The Hidden Architecture Change
The least discussed change is state management. In multi-turn conversations and tool-using agents, providers now carry reasoning forward between steps, often as encrypted or opaque items rather than readable text. OpenAI describes encrypted reasoning items that can be passed back in stateless mode, and notes that persisted reasoning can be reused only within the same model family. Google describes thought signatures, encrypted records of reasoning state that must be returned unchanged in stateless mode, including around tool calls.
For architects, this has real implications:
- Conversation stores must preserve opaque blocks. If your orchestration layer strips “unknown” content to save tokens, you may silently degrade agent quality.
- Model switching mid-conversation has a cost. Routing a conversation from one model family to another can drop reasoning continuity.
- Gateways must pass content through faithfully. AI gateways that normalize messages across providers need to handle provider-specific reasoning artifacts.
- Data retention policies must cover reasoning artifacts. Encrypted reasoning items are still data your systems store and transmit.
Quality: When More Thinking Helps and When It Hurts
Higher effort generally improves results on multi-step problems, but it is not free quality. Vendor guidance acknowledges overthinking: Anthropic’s documentation includes prompt patterns to curb excessive thinking and to make the model commit to an approach, and notes that effort does not always change visible response length. In practice, teams see several patterns:
- Simple lookups and classifications rarely improve with more thinking and become slower and costlier.
- Multi-step planning, reconciliation, complex code changes and analysis of long documents often improve materially.
- Agentic workflows benefit from moderate effort at each step more than from maximum effort on one step.
- Very high effort can make models more likely to explore tangents or take more actions than intended, which matters for agents with write access.
The only reliable way to know is evaluation. Run your offline evaluation suite at two or three effort levels per feature and compare quality, latency and cost. Often the curve flattens quickly, and medium effort delivers most of the gain.
A Decision Framework for Choosing Effort
Use the questions below to set a starting effort level for each feature, then adjust based on evaluation data:
- How many dependent steps does the task require? One step suggests minimal or low; several dependent steps suggest medium; open-ended planning suggests high.
- What does an error cost? Low-stakes drafts tolerate lower effort; financial reconciliations, legal analysis or security reviews justify more.
- What latency can users accept? Real-time voice and inline UX need minimal effort; analyst tools and batch jobs can wait.
- Is there a verifier? If tests, schemas or rules can check the output, a lower-effort attempt with automatic retry may beat one high-effort call.
- What is the volume? High-volume features magnify the cost of each reasoning token; tune them first.
Governance and Explainability
Reasoning models raise a governance question: if the model thinks before acting, can you inspect that thinking? Generally only partially. Providers typically return summaries or encrypted artifacts, not complete raw reasoning. Governance teams should therefore avoid relying on model “thoughts” as explanations for decisions affecting people. Instead:
- Log inputs, retrieved context, tool calls, outputs and effort settings for every consequential request.
- Use thought summaries for debugging and prompt improvement, not as formal justification.
- Require structured, reviewable rationales in the visible output where decisions need explanation, and validate them against source data.
- Include effort level in change management: raising or lowering effort is a behavior change that should be evaluated before release.
How Reasoning Changes Retrieval and Agent Design
Reasoning models interact with retrieval-augmented generation and agents in ways that change long-standing design habits. With little thinking, models depend heavily on the retrieval layer delivering exactly the right passages, so teams invest in elaborate prompt chains and re-ranking. With more thinking, models can plan multiple searches, notice contradictions between sources and decide when they need more context. That shifts value from hand-built chains toward good tools: well-described search functions, precise filters, structured data access and clear error messages that the model can reason about.
It also changes how much orchestration code you need. Many multi-step pipelines written in 2024 to compensate for weak planning can be simplified into a single agent loop with a few well-defined tools and a moderate effort setting. Simplification reduces maintenance, but it moves control from your code into the model, so guardrails, permission scopes and evaluation become more important, not less.
Implementation Checklist
- Inventory every generative AI feature and record its current model, effort or thinking level, and output cap.
- Enable reasoning-token reporting in usage logs and dashboards, broken down by feature and tenant.
- Run offline evaluations at two or three effort levels per feature and document the chosen setting with its rationale.
- Review gateway and orchestration code to confirm opaque reasoning items and signatures are preserved across turns and tool calls.
- Adjust timeouts, retries and output caps for high-effort paths.
- Add effort level to release notes and change approval workflows.
A Hypothetical Worked Example
Consider a hypothetical logistics company running three generative AI features on one platform: an email classifier that routes customer messages, a customer-facing chat assistant for shipment questions, and an internal agent that investigates billing disputes by pulling invoices, contracts and tracking data.
Initially all three use the provider’s default settings. Monthly spend is higher than forecast, and the chat assistant feels sluggish. The team adds reasoning-token tracking per feature and discovers that the classifier, which handles the most volume, is spending substantial reasoning tokens on trivially easy messages.
After running its evaluation suite at multiple effort levels, the team sets the classifier to minimal effort with no measurable accuracy loss. The chat assistant moves to low effort, with escalation to a medium-effort path only when the conversation involves claims or exceptions. The billing agent stays at medium effort per step, with a rule-based verifier checking that its calculations reconcile with invoice totals before a human approves any credit. The team also discovers that its gateway was discarding encrypted reasoning items between agent steps, and fixes it, improving the agent’s consistency. Costs fall, the chat feels faster, and the most complex workflow keeps its quality.
Pitfalls to Avoid
- Leaving effort at the default for every feature and discovering the cost on the invoice.
- Setting output caps too low, causing incomplete responses that look like model failures.
- Stripping opaque reasoning content in orchestration or gateway layers.
- Treating thought summaries as audit-grade explanations.
- Comparing models at different effort levels and drawing conclusions about the models rather than the settings.
- Forgetting that effort changes are behavior changes requiring regression testing.
Outlook: 2027 and Beyond
Expect reasoning to become even more automatic. Models will increasingly decide for themselves how much to think, and APIs will offer coarser but more semantic controls, such as optimizing for latency, cost or quality. Background and long-running modes will make hour-long reasoning jobs normal for research and analysis. Reasoning state will become a first-class part of agent architectures, which will push AI gateways, observability tools and evaluation frameworks to handle it natively. The organizations that benefit most will be those that measure reasoning cost and quality per feature today, so they can tune confidently as the dials change.
Conclusion
Reasoning is no longer a special model you call for hard problems; it is a property of almost every model you use, controlled by a dial. That makes effort a design decision with consequences for cost, latency, architecture and governance. Track reasoning tokens, choose effort per feature with evaluation data, preserve reasoning state in your orchestration layers, and keep explanations grounded in logged evidence rather than hidden thoughts. Done deliberately, thinking models deliver better answers where they matter and stay fast and affordable everywhere else.
How Hanoi Technology Can Help
Hanoi Technology helps teams evaluate effort settings, track reasoning costs and design orchestration layers that use reasoning models deliberately.