The feature works. This chapter is about the year after that.
An AI feature decays differently from ordinary code. Nothing breaks. The tests keep passing, the endpoints keep returning 200, and the output slowly stops being as good as it was — because a model was updated, a provider changed a default, or your product moved and the prompt did not.
Pin the Model Version
'model' => 'claude-sonnet-5',
Never a floating alias that resolves to "whatever is newest". A model upgrade is a behaviour change to your product, and it should happen when you decide, not when a provider ships.
Chapter 4 has the concrete cost of not doing this: a cap that was ample became insufficient when the model started reasoning before answering, and replies began ending mid-sentence. Nothing in our code changed. Everything about the output did.
Treat a model bump exactly like a dependency bump: change the pin, run the suite, look at real output, ship it deliberately, and be able to revert. And keep the version somewhere your logs record, so "when did this change?" is answerable.
Smoke the Path, Not the Model
Chapter 12 argued against asserting on model output, and that still holds in production monitoring. What you can check continuously is that the path works end to end — credentials valid, provider reachable, response parses, schema validates, result stored.
Run that on a schedule against production, with a fixed input, asserting structure rather than content: it came back, it parsed, it had the required keys, it took less than N seconds, it cost roughly what it usually costs.
That catches the failures that actually happen — an expired credential, a changed field name, a quota, a provider incident — none of which any unit test can see, and all of which arrive without a deploy.
Alert on Symptoms, Not on the Model
The temptation is to alert on quality. Quality is not a signal you have.
The signals you do have, all from Chapter 14's log lines:
- Error rate — parse failures, validation failures, provider 5xx.
- Refusal and truncation rate — either creeping up means a prompt or a cap needs attention.
- Iterations per turn — rising means the model is reaching for tools more, which is a prompt problem before it is a cost problem.
- Tokens per turn — the bill, before the invoice.
- Cap hits — every iteration cap and wall-clock budget that binds. Chapter 8's point: a silent cap is indistinguishable from a finished turn.
Alert on rates and trends, never on individual events. One refusal is a Tuesday. A refusal rate that doubled overnight is a deploy that needs reverting.
Keep a Human Escape Hatch
Every AI feature needs a route that does not involve the model.
If the conversational path fails, a user must still be able to do the thing — a plain form, a support address, a manual process someone runs. Not because the AI is unreliable, but because every dependency is eventually unavailable, and Chapter 15's gating is only useful if there is something on the other side of the refusal.
The version of this that matters most: anything irreversible should be reachable and reversible by a human. If a tool creates a billable resource, someone in support needs to be able to see it, undo it, and re-run it, without a model in the loop. Chapter 10 made those operations safe; this makes them recoverable.
The Six-Month Test
The question to ask of everything you have built here is whether a colleague who has never seen it could, six months from now, answer:
- Which model version is running, and when did it last change?
- Where is the prompt, and what changed in it recently?
- What does one turn cost, and what is the ceiling?
- What happens when the provider is down — what do users see?
- Which tools can do something irreversible, and what makes them safe to repeat?
- How would you debug a turn you cannot reproduce?
Every one of those has a chapter. If they are all answerable from the repository, the feature is finished. If they live in your head, it is not — it is just working.
What This Chapter Argued
- Pin the model version. An upgrade is a behaviour change; make it yours to schedule, and record it where your logs can show when it changed.
- Smoke the whole path on a schedule, asserting structure rather than content — that catches expired credentials and changed contracts, which arrive without a deploy.
- Alert on rates and trends: errors, refusals, truncations, iterations, tokens, cap hits. Never on quality; you cannot measure it.
- Keep a path that does not involve the model, and make every irreversible action visible and undoable by a human.
- The feature is finished when the six questions are answerable from the repository, not from you.