Chapter 3 argued for ordering a prompt so its stable prefix can be cached. This is where you find out whether that worked.
Providers impose a minimum cacheable size. A prompt marked for caching but sitting under that threshold is not an error — nothing warns you, nothing fails. The marker simply does nothing, and you carry on believing you have caching.
The tell is cache_read_input_tokens staying at zero across turns that should obviously have hit a warm prefix. If it does, the marker is decorative: either the prefix is too small to qualify, or something before the marker varies per request and there is no stable prefix at all.
Log it, look at it once, and act on what you see. This is a ten-minute check that either confirms a large discount or reveals you never had one.
The Dial Between Cost and Latency
Reasoning models expose a control over how much thinking to do, and its default is chosen for hard problems.
Most product features are not hard problems. A conversational widget helping someone describe their business is not a competition-mathematics benchmark, and the default effort spends both money and seconds that the interaction does not need:
'thinking' => ['type' => 'adaptive'],
'output_config' => ['effort' => 'medium'],
Two decisions worth separating.
Adaptive rather than off. Turning thinking off entirely is tempting on cost grounds, but on a tool-driven feature it backfires: with no room to reason, the model reaches for tools less readily, and the whole feature is tools. Cheaper per call, worse at the job.
Effort turned down from the default. This is the latency dial as much as the cost one. A chat that answers in four seconds instead of nine is a materially better product, and here it is also cheaper.
Do not accept these defaults. Do not guess at them either — change one, measure latency and quality on real traffic, keep or revert.
Multiply Everything by Iterations
A final piece of arithmetic, because it is the one that turns a reasonable per-call cost into an unreasonable bill.
In a loop, each iteration re-sends everything: prompt, full history, tool definitions, and every tool result so far. Input tokens grow with each pass. A ten-iteration turn does not cost ten times a one-iteration turn — it costs considerably more, because the last iterations are the largest.
Which makes the iteration cap a spending control as much as a correctness one:
$maxIterations = 10;
Chapter 8 covers what that loop does and why it needs a wall-clock budget as well as a count. Note here only that every cap in this chapter is per call, and a turn is many calls. If you cap the call and not the loop, you have capped nothing.
What This Chapter Argued
- Cost is a correctness property. It is the only failure mode that never announces itself.
- Clients choose nothing that costs money. Overwrite the model, provider and token cap server-side in
prepareForValidation(); expose one enum of purposes instead. - Use lanes. A short list of named purposes, each with a pinned model and cap, beats one global number or a free parameter.
- An output cap can consume itself on reasoning. Set it for thinking plus answer, and always check the stop reason so truncation is visible.
- Rate limits and budgets are different questions. Too fast belongs on the route; too much belongs where you know who pays.
- Increment first, compare second. Check-then-increment fails open under concurrency, and a spending limit must never fail open.
- Key the budget to what pays, not to the credential. A short-lived token resets the budget every rotation.
- Refuse with 429, a computed
Retry-After, and the name of the limit that was hit. - Read the usage block. It arrives free on every response and answers the questions you will be asked later.
- Verify prompt caching rather than assuming it. A marker under the minimum size does nothing, silently.
- Cap the loop, not just the call. Iterations multiply every other number here.