Running Agentic AI on a Token Budget: Cutting Agent Cost 10× Without Losing Reasoning Quality
Running Agentic AI on a Token Budget: Cutting Agent Cost 10× Without Losing Reasoning Quality
Agent loops replay their entire context on every step. That single property is why most agentic pilots look brilliant in a demo and indefensible in a budget review — and why token architecture, not model choice, decides whether your agent ships.
Agent loops are quadratic, and nobody budgets for that
A single-turn LLM call is easy to reason about: you send a prompt, you pay for it once. An agent is a different animal. On every step of the loop, the model receives the system prompt, the full tool catalog, and the entire accumulated history of everything it has already done — every reasoning turn, every tool call, every observation returned. Step twelve pays for steps one through eleven all over again.
The consumption curve is therefore not linear in steps. It is roughly quadratic. Double the number of steps and you roughly quadruple the token bill. This is why teams are consistently blindsided: the pilot ran three-step workflows and cost almost nothing, then production hit fourteen-step incident investigations and the invoice went up by two orders of magnitude.
| Loop length | Tokens added per step | Cumulative input billed | Ratio |
|---|---|---|---|
| 3 steps | 2,800 | ~25,500 | baseline |
| 7 steps | 2,800 | ~98,700 | 3.9× |
| 14 steps | 2,800 | ~334,600 | 13.1× |
| 14 steps, unshaped log dumps | 12,000 | ~1,171,800 | 46.0× |
Assumes a 5,700-token static prefix (tool schemas plus system policy). Note the last row: the only variable that changed was how much text each tool returned. A retrieval tool that dumps raw log lines instead of a shaped summary can, on its own, multiply your bill by 3.5× — without the agent doing anything differently or getting any smarter.
Where the tokens actually go
Before optimizing anything, instrument the loop and break the consumption down by origin. Across production agent deployments the distribution is remarkably consistent, and it is almost never where teams assume.
| Source | Typical share | Why it grows | Compressible? |
|---|---|---|---|
| Tool results / observations | 55–75% | Raw payloads, log lines, JSON blobs returned verbatim | Very high |
| Tool schema definitions | 10–20% | Re-sent in full on every single step | High |
| Accumulated reasoning turns | 8–15% | Never pruned; superseded hypotheses persist forever | High |
| System prompt & policy | 4–10% | Static, but re-sent every step | Cacheable |
| Generated output | 2–6% | Small volume, but billed at 4–5× the input rate | Moderate |
Two conclusions follow immediately. First, the biggest lever is not the model or the prompt — it is what your tools return. Second, the second-biggest lever is how much of the context is byte-stable enough to cache. Teams that spend a month rewriting their system prompt to be more concise are optimizing a 6% line item while a log-search tool quietly burns the other 70%.
What a token-optimized agent loop looks like
The optimized architecture separates context into four zones by volatility. Everything immutable sits at the front behind a cache breakpoint. Everything voluminous gets shaped before it ever enters the transcript. Everything stale gets compacted out. And everything that can be decided without a frontier model gets routed away from one.
Context is a budget, not a bucket
The bucket model
"We have a million-token window, so let's give the agent everything and let it figure out what matters." Every document, every log line, every tool. Quality is assumed to scale with information volume. Cost scales quadratically, latency degrades, and — worse — accuracy often drops as relevant evidence gets diluted by noise.
The budget model
Every step gets an explicit token allowance, allocated by value. Instructions and guardrails are non-negotiable. Output is reserved first. Evidence competes for what remains, ranked by relevance. The window is treated as a working desk with finite surface area, not a warehouse — because that is exactly what attention makes it.
The budget model is not merely cheaper. Long-context evaluations consistently show retrieval and reasoning degrading as irrelevant context grows — the model spreads attention across noise. Token discipline and answer quality point the same direction far more often than teams expect, which is what makes this optimization unusually easy to justify: you are not trading accuracy for cost.
Seven levers, ordered by leverage
Apply these in order. The first three typically deliver 80% of the total saving, and none of them require touching your model or your prompt wording.
Make the prefix byte-stable, then cache it
Prompt caching is the single highest-return change available, and it is mostly a discipline problem rather than an engineering one. On the Claude API, cached prefix reads bill at 10% of the base input rate; a five-minute cache write costs 1.25× and a one-hour write costs 2×, so the cache pays for itself after one or two reads. Caching covers tools, then system, then messages — in that order — up to your breakpoint.
The catch is that the cache is keyed on exact bytes. One timestamp, one incident ID, one interpolated username above the breakpoint and you pay full price for the entire prefix, every step, forever. This is the most common silent failure in agent deployments: teams enable caching, never check cache_read_input_tokens, and assume it is working.
Then verify it. Log cache_read_input_tokens and cache_creation_input_tokens on every call and alert when the creation-to-read ratio spikes — that is your signal that someone shipped a prompt change or introduced a dynamic value into the prefix.
Shape tool results at the source
This is where the majority of the saving lives, and it is entirely within your control because you write the tools. The rule: a tool returns a decision-grade summary, never a data dump. If a human wouldn't read all 4,000 lines, the model shouldn't receive them either.
Note the drill_down pointer. The agent retains the ability to fetch detail when it genuinely needs it — but it now makes that an explicit, deliberate, once-per-investigation decision rather than paying for full fidelity on every speculative query. Pair this with hard response caps enforced server-side: no tool may return more than N tokens, ever, regardless of what the underlying query matched.
Budget the tool catalog
Tool definitions are re-sent on every single step. Forty tools at an average 200 tokens each is 8,000 tokens per step — 112,000 tokens across a fourteen-step loop, spent entirely on describing capabilities the agent will never invoke on this particular task.
Three fixes, in increasing sophistication: trim the descriptions (they are prose, and most are twice as long as they need to be); scope the catalog by task phase, so the triage phase loads observability tools only and the remediation phase loads execution tools only; and defer — expose a search-and-load mechanism where the agent discovers tools by keyword and only the matching definitions are injected. Deferred loading is the strongest option once you pass roughly twenty-five tools.
| Catalog strategy | Prefix cost / step | 14-step loop | Trade-off |
|---|---|---|---|
| All 40 tools, verbose | 8,000 | 112,000 | None — pure waste |
| All 40, trimmed descriptions | 4,800 | 67,200 | Slightly higher mis-selection risk |
| Phase-scoped (12 tools) | 1,700 | 23,800 | Needs phase detection logic |
| Deferred / search-loaded | ~600 | ~11,000 | One extra discovery round-trip |
Compact aggressively, and offload the rest to disk
Halfway through an investigation, most of the transcript is dead weight: hypotheses that were ruled out, tool calls that returned nothing, intermediate reasoning that has been superseded. Compaction replaces that history with a compressed state object — decisions and confirmed facts retained, raw evidence discarded.
The offload half matters as much as the compression. Give the agent a working file it can write findings into and read back selectively. Evidence lives on disk; the context window holds a pointer. This is the difference between an agent that can sustain a forty-step investigation and one that hits its ceiling at twelve — and it makes the investigation auditable as a side effect, because the scratchpad becomes the evidence trail.
Isolate expensive sub-tasks in sub-agents
When a sub-task requires burning through large volumes of context — scanning six months of change records, reading forty runbook definitions to find the right one — do not do it in the main loop. Spawn a sub-agent with a clean context, let it consume what it needs, and have it return only its conclusion.
The parent loop receives 300 tokens of answer instead of inheriting 40,000 tokens of search transcript that it will then carry, and pay for, on every subsequent step. Context isolation is the architectural version of the quadratic insight: keep the expensive stuff out of the loop that repeats.
Route by difficulty, not by habit
Most agent steps are not reasoning steps. They are classification ("is this alert real?"), extraction ("pull the service name from this payload"), and formatting — tasks a small fast model handles at parity. Published Claude API rates put Haiku 4.5 at $1/$5 per million tokens against Sonnet 5 at $2/$10 and Opus 5 at $5/$25, so the tier gap is real money at volume.
In a mature incident agent, roughly 60% of calls land in the small tier. Route on a cheap signal — incident class, step type, confidence threshold — and escalate on failure rather than starting expensive. One caveat worth respecting: never route planning down a tier to save money. A weak plan generates more steps, and more steps cost far more than the model-tier difference you saved.
Cap the output, and pre-filter with plain code
Output tokens bill at four to five times the input rate, and extended-thinking tokens bill as output. Set an explicit max_tokens per step type — a routing decision needs 150 tokens, not 4,000 — and demand structured fields rather than prose. Let the ticket template render the narrative; the model should emit data.
Then look upstream. A meaningful share of what agents get asked to evaluate can be decided by a rule, a regex, or a lookup for effectively zero cost. Deduplicate the alert storm, suppress known-benign patterns, and check the recurrence table before invoking the model. The cheapest token is the one you never send.
Finally, for work that genuinely tolerates latency — nightly evals, prompt regression suites, bulk enrichment, backfill classification — the Batch API applies a flat 50% discount that stacks with caching. It is not usable for interactive agent loops, which need multi-turn synchronous tool use, but the offline half of your pipeline is often larger than teams realize.
One incident investigation, before and after
A fourteen-step cross-cloud latency investigation: correlate an alert, check recent changes, walk the dependency graph, query logs across two tools, form a hypothesis, select a runbook, and hand off. Same task, same quality bar, two architectures.
| Component | Naive loop | Optimized loop | Change |
|---|---|---|---|
| Tool schemas (re-sent per step) | 44,800 | 8,400 | −81% |
| System prompt & policy | 35,000 | 2,500 | −93% |
| Tool observations | 254,800 | 19,100 | −93% |
| Reasoning turns retained | 18,400 | 4,200 | −77% |
| Generated output | 4,900 | 2,900 | −41% |
| Steps to resolution | 14 | 9 | −36% |
| Total tokens billed | 357,900 | 37,100 | 9.6× lower |
Two things worth noticing. First, the step count itself dropped — better-shaped tool results meant fewer clarifying round-trips, so the saving compounds rather than adds. Second, the cost reduction outruns the token reduction once cache reads are priced in, because the prefix is now paid at a tenth of base rate for the majority of steps. Figures are illustrative and rate-dependent; verify current pricing before building a business case on them.
Measure per resolved task, never per call
Cost-per-API-call is a misleading metric for agents — it rewards splitting work into more, smaller calls, which is exactly the wrong incentive. The unit that matters is the completed task.
- ◆Tokens per resolved task — the headline number. Track the median and the p95; the tail is where budgets die.
- ◆Cache hit rate — cache reads as a share of total input. Below 60% on a stable workload means something is poisoning your prefix.
- ◆Tool-output token share — if this exceeds 50%, your next optimization is a tool, not a prompt.
- ◆Steps per resolution — rising step counts usually mean the agent is thrashing on ambiguous tool results.
- ◆Small-tier routing share — with quality held constant. A rising share at flat accuracy is pure margin.
- ◆Compaction loss rate — how often a compacted agent re-queries something it already knew. This is your over-compression alarm.
Wire a token budget into CI. Run a fixed scenario suite on every prompt or tool change and fail the build when median tokens-per-task regresses beyond a threshold. Token consumption drifts upward silently otherwise — one verbose tool description at a time.
Where over-optimization starts costing more than it saves
- ▲Summarizing away the evidence. If a tool summary drops the one anomalous field that explained the incident, you have optimized your way into a wrong answer. Preserve outliers and distinct signatures explicitly, not just top-N by frequency.
- ▲Compacting the decision trail. Compaction must retain what was ruled out and why. Drop that and the agent re-investigates the same dead end, spending more tokens than compaction saved.
- ▲Routing planning to a small model. Cheap per call, expensive per task. Plan quality determines step count, and step count dominates the bill.
- ▲Cache-driven prompt rigidity. Teams stop improving prompts because changes invalidate the cache. Version the prefix deliberately and accept a warm-up window; a better prompt that cuts two steps beats a stale cached one.
- ▲Trimming tool descriptions into ambiguity. Tool descriptions are the model's selection interface. Over-trim and mis-selection rises, which costs more in retries than the description ever cost in tokens.
A three-phase sequence
Agentic AI Token Budget, Explained
The whole token-optimization playbook on one page: prompt caching, tool shaping, compaction, and model routing — and where each lever saves its share.
Token architecture is agent architecture
The instinct when an agent costs too much is to reach for a cheaper model. That is almost always the wrong first move — it trades reasoning quality for a linear saving while leaving the quadratic problem completely intact. A weaker model on an unoptimized loop takes more steps, and more steps is precisely what you could not afford.
The durable fix is structural. Cache what never changes. Shape what enters the loop. Compact what has gone stale. Isolate what is expensive. Route what is easy. Cap what is generated. Each of these is a design decision about context flow, not a model decision — and taken together they routinely deliver an order of magnitude while leaving answer quality flat or better.
Which is the part that makes this worth doing properly: for once, the cheap architecture and the good architecture are the same architecture.
