You have a working AI feature. Where should the code live?
Three answers are defensible: inside the application that uses it, in a leaf service the rest of your system calls, or dissolved into the client with the provider called directly. Chapter 5 disposed of the third. This chapter is about the other two, and about the discipline of choosing between them with evidence.
What a Leaf Service Buys
Ours lives in a small service that does nothing but talk to model and image providers. What that bought:
One place holding provider credentials. Not seven applications each with a key in its environment. Rotating one is one deploy.
One place that knows a provider's API. When a field is renamed, one repository changes.
Per-tenant credentials and accounting in one implementation, rather than each consumer reinventing attribution.
A cache that is actually shared. Two applications asking for the same generated thing hit one cache.
The general principle: a leaf service is worth it when the thing it wraps is a shared, credentialed, changing external contract. All four words matter — a stable contract with no credentials used by one consumer should just be a class.
What It Costs
A network hop, with its own timeouts and its own failure mode. An extra service to deploy, monitor and keep on a framework version. A boundary that must be authenticated, which is the entire subject of Chapter 5. And latency you cannot remove by optimising either side.
The cost is real, and the temptation to eliminate it is strongest exactly when you are least equipped to judge it.
The Hop Nobody Measured
Our chat reaches its tools by asking the provider to call them, so one tool invocation crosses the public internet twice more than it strictly needs to. There is an obvious fix — become a client of our own toolset and cut the provider out of that leg — which also unlocks streaming.
It is about a week of work, and for months the argument for it was entirely aesthetic. That extra hop must be expensive. Must it? Nobody knew. The number did not exist.
So instead of rebuilding, we built the instrument from Chapter 14: two clocks on the tool side, a per-round-trip log on the client side, and two tools that do no I/O at all as clean probes. Subtract one from the other and what is left is inference plus transport.
The result matters less than the discipline:
Guessing at that number would have been a poor reason to rebuild the agent loop, and an equally poor reason not to.
Both halves of that sentence are the point. Measurement does not exist to block the rewrite; it exists so that whichever way you go, you went there on purpose. An architecture change justified by a feeling is indistinguishable from one justified by a fact, right up until it does not help.
Own the Boundary, Rent the Model
The durable split, and it holds however you arrange the services:
Rent: the model, the image provider, the infrastructure they run on. These are commodities, they improve without you, and being able to swap them is worth more than any advantage from coupling to one.
Own: the boundary. The contract, the prompt, the schema, the tools, the caps, the retry policy, the rendering, the tests. Every chapter of this book is about something on this side of the line — and none of it moves when you change providers.
If a provider switch would touch prompts, caps, rendering and tests, they are on the wrong side of that line. Chapter 1's manager and contract exist so that switch is a driver and a config block.
Prefer a Star to a Mesh
One structural lesson from running several of these together.
At one point our services called each other in a chain: a request entered one, which called a second, which called a third. Debugging meant three sets of logs; a failure in the third produced an unhelpful error in the first; and latency was the sum of everything.
Collapsing that into a star — one orchestrator calling leaves that never call each other — fixed all three. Leaf services became synchronous, stateless and trivially testable. There is one place a request's story is written down.
AI services in particular should be leaves. They are slow and expensive; putting one in the middle of a chain means everything behind it waits on both the model and every hop after it.
What This Chapter Argued
- A leaf service is worth it when the thing it wraps is shared, credentialed, changing and external. Fewer than four, use a class.
- The cost is a hop, a deployment, and a boundary to authenticate. Do not pretend it is free.
- Measure before rebuilding. A guess is a bad reason to spend a week — and an equally bad reason not to.
- Design the probe, not just the timer. Operations that do no I/O are what make a subtraction meaningful.
- Rent the model, own the boundary. If switching providers would touch prompts, caps or tests, the line is in the wrong place.
- Keep AI services as leaves in a star. They are the slowest thing you have; nothing should be queued behind them.