Still, the slide is wrong,
Why?
Because what you are buying is not tokens. It is workflows, and workflows have learned to consume tokens faster than tokens get cheaper. That is the inference paradox that Gartner Group talks about: the unit price falls, the invoice climbs, and neither number is a mistake.
The Price Collapse Happened Somewhere Else
Start with the part the vendors get right. Compiled inference-cost data from a16z, Epoch AI and Stanford's AI Index puts GPT-3-equivalent quality at roughly $60 per million tokens in late 2021 and about $0.06 by late 2024, a thousandfold drop, with price-performance improving at a median 50x per year and closer to 200x per year since the start of 2024. Goldman Sachs has semiconductor suppliers delivering 60 to 70 percent annual reductions in cost per token. Nobody can, nor does, dispute the direction of travel.
Now look at what the frontier costs today. Anthropic's published price list (as of August 27, 2026) puts Claude Opus 5 at $5 per million input tokens and $25 per million output tokens, and its largest models at $10 and $50. Those are the models your agentic workflow escalates to when the cheap one fails, and they are in a similar range as the GPT-3 launch pricing.
So the collapse is real, just that it happened at the commodity end. The price of last year's intelligence fell through the floor. The price of this year's did not.
And Agentic architectures are designed, to use this year's.
Token Inflation Is Measured, Not Alleged
This is where the research has caught up with the invoices, and the numbers are worse than most buyers assume.
Fu and colleagues gave the effect a name in Not All Tokens Are Equal: token inflation, the gap between advertised per-token pricing and what a workflow actually consumes once it retries what it got wrong. They measure inflation as high as 4.25x on multi-hop question answering, and they show that FrugalGPT, one of the standard cost-aware routers, underestimates true expense by more than 2x on hard tasks. This mechanism is what both drives the cost and nobody prices in: a failed reasoning chain is not only a wasted call, it is a call that gets re-sent in full, with history attached, to a more expensive model.
The infrastructure picture is even worse. Kim and colleagues measured what agents do to a serving stack rather than to a budget line in The Cost of Dynamic Reasoning. Tool-augmented agents make roughly 9.2 times as many model calls as a chain-of-thought baseline. A tree-search agent averages 71 calls per single request. Input sequences run three to four times longer because the interaction history accumulates. GPU memory per request rises three to five times, and the GPUs then sit idle 54.5 percent of the time waiting on tool calls. Energy per query rises 62 to 137 times over single-turn inference.
Gartner's 5-to-30x multiplier for agentic queries, which read like an analyst hedging, turns out to be the conservative end of the range.
This is the paradox in four short sentences:
The price per token fell by three orders of magnitude. The tokens consumed per useful outcome rose by one to two. Usage went up, way up. Compound those and you get a real invoice.
You Are Also Paying for Tokens That Make the Answer Worse
There is an additional line item nobody budgets for, and it is one a CX buyer should find most uncomfortable.
Zhou and colleagues studied what happens when you keep spending on reasoning in When More Thinking Hurts. Marginal utility on their test set drops from +1.8 percent per 500 tokens in the 2,000 to 4,000 range, to +0.1 percent between 8,000 and 12,000, and turns negative beyond that. Past roughly 7,000 tokens the model flips more previously correct answers to wrong than wrong answers to right. Their 32B model peaks at 55.8 percent accuracy at 12,000 tokens and falls back to 54.9 percent at 16,000. Easy problems start overthinking at around 1,500 tokens.
Read that again with a meter running. There is a point in every reasoning budget past which you are paying more money to get a worse answer, and it arrives earliest on the simple tickets that make up the bulk of your service volume.
The Efficient Agents work draws the same conclusion from the other direction: best-of-N test-time scaling buys marginal accuracy at disproportionate cost, and a leaner design retained 96.7 percent of the accuracy at $0.228 per problem solved against the $0.398 of richer systems. More compute is not a strategy. It is a default setting.
The CX Meter Hides the Multiplier
None of this would matter much if your contract passed the cost through legibly. It does not.
CX Today's buyer guide catalogues six live pricing models: per seat with tokens bundled, per channel, per component, credits, per action, and per resolution. Genesys runs $75 to $240 per user per month. Amazon Connect meters $0.038 a voice minute and $0.010 a chat message. Salesforce sells Flex Credits at $500 per 100,000, roughly ten cents an action, having launched Agentforce at $2 a conversation and repriced once that unit stopped fitting the work.
Zendesk's pricing page is different. It explains the outcome model clearly, that you pay only for requests resolved without escalation to a human, and it does not publish a rate. The headline unit of the most buyer-friendly-sounding pricing model in customer service sits behind a sales call and a tough negotiation.
Each model hides the multiplier at a different place; none of them is denominated in anything a CX leader actually manages. Per-message looks cheap until the agent generates repeat contacts and you are billed twice for failing once. Per-resolution looks aligned until you notice it pays the vendor not to escalate. Per-action asks you to model an entire workflow before you can forecast a quarter.
Jevons Was Not a Pessimist
None of this is an argument against spending. Cheaper units drive more consumption. That is Jevons. It is what happened to coal and steel and bandwidth, and it usually indicates a technology that works. Goldman expects token consumption to multiply 24 times, to 120 quadrillion tokens a month, between now and 2030. Spending more on inference in 2029 than you do today is not necessarily mismanagement.
The failure mode is narrower. It is not spend. It is unattributed spend. The organizations in trouble are not the ones with large inference bills; they are the ones that cannot say which workflow, which agent, or which resolved ticket a given bill belongs to, in other words, which outcome they pay for.
What to Implement Before You Sign
Look at preparing yourself using five measures, ordered by how quickly they pay back.
Instrument before you scale. Use per-agent and per-workflow token telemetry with alert thresholds, from the first pilot onward. One healthcare deployment ran from $12,000 to $68,000 a month over six weeks on a retrieval fault that went unnoticed for two of them. Cost you cannot attribute is cost you cannot defend, or avoid.
Cap the loops. Set hard retry ceilings with mandatory human escalation at the limit, and a fresh-escalation policy that discards a failed chain instead of forwarding it: Fu and colleagues found that passing failed reasoning to a stronger model cost up to 34.8 percentage points of accuracy. Uncontrolled retries are the single largest driver of runaway spend. You pay premium rates to make the answer worse.
Budget the thinking. Set per-task reasoning caps rather than letting a model run to its limit, and allocate them by difficulty rather than uniformly: Zhai and colleagues get up to 12.8 percent better accuracy on MATH at the same budget purely by varying compute per instance. Above the crossover point, you are buying degradation at full price.
Take the engineering discounts. Cache reads price at a tenth of standard input and pay for themselves fast. Route simple work to small models: one team took $40,000 a month down to $24,000 on routing discipline alone. Compact context, retrieve just in time, and stop shipping every tool schema into every call.
Put the meter in the contract. Get a defined billable unit in writing that names cached tokens, tool execution, failed calls and retries, not just input and output and audit rights to reconcile your own telemetry against the invoice. Demand a notice before any repricing or redefinition of the consumption model: current clause guidance suggests 120 days, and on a meter that can move mid-year I would ask for six months. Then measure value per thousand tokens against agreed outcomes rather than against volume. Organizations that do this report spending 60 to 70 percent less for equivalent output.
The tokens will keep getting cheaper. The bill will keep getting bigger. Only one of these two is under your control.
Comments
Post a Comment