Everything so far has assumed the work fits in one call.
Eventually something does not: a whole site's copy to translate, a long document to summarise, a batch of records to classify. There is an input cap, an output cap, and a timeout, and your job exceeds at least one of them.
The naive fix is to ask for a bigger cap. The real fix is to stop sending one large thing.
Chunk on Structure, Not on Characters
The obvious implementation splits at a character count. It is also the one that produces corrupted output, because it cuts through the middle of whatever it is splitting — half a sentence, half a key, half a JSON object.
Split on the structure you already have:
/**
* api-ai caps a single prompt at services.ai.prompt.max_length chars with a
* bounded output. A whole site's messages file is routinely larger than that,
* so we translate in chunks — grouping top-level keys so each call stays
* comfortably under the cap — and reassemble.
*/
Top-level keys are the seam. Each chunk is a complete, valid sub-document: whole keys with whole values, nothing severed. The model sees something coherent, and reassembly is a merge rather than a repair.
Find the natural seam in your data. Sections in a document, records in a batch, keys in a config. If you are splitting at a character offset, you are splitting in the wrong place.
Leave Room for the Instructions
/**
* Max chars of content per call. Kept well under the 4000-char prompt cap
* (leaving room for the ~700-char instruction preamble) and its output-token
* ceiling.
*/
private const CHUNK_CHAR_LIMIT = 2500;
The cap applies to the whole prompt, not to your content. Your instructions ride along on every chunk, and they are not small.
2500 against a 4000 cap, with a ~700-character preamble. That leaves roughly 800 characters of headroom — deliberately generous, because the output also has a ceiling and translated text is frequently longer than its source. A chunk sized to just fit the input will produce output that does not.
Budget for instructions plus input plus expansion, and leave slack. The failure mode of a too-large chunk is a truncated response, which Chapter 4 established is the failure that looks like a success.
Fail the Whole Thing Rather Than Reassemble a Lie
foreach ($this->chunk($content) as $chunk) {
$translated = $this->translateChunk($token, $chunk, $language);
if ($translated === null) {
return null;
}
foreach ($translated as $key => $value) {
$result[$key] = $value;
}
}
One chunk fails, the whole operation returns null.
The tempting alternative — keep what worked, skip what did not — produces a site half in Spanish and half in English, with no record of which half. A partial success that cannot be distinguished from a complete one is worse than a clean failure, because the clean failure gets retried and the partial one gets shipped.
Make partial results either impossible or obvious. Here, impossible is cheaper.