Retry and backoff live in the client, once

Every leaf service api-server calls goes through its own small integration class, GithubAPI is one of a dozen with the same shape. Every one of those classes will eventually get rate limited, because every one of them is calling a system with its own capacity, and every one of those systems says so the same way: an HTTP 429. If each integration class implemented its own retry loop, correctness here would need to be derived a dozen separate times, and the twelfth time, under a deadline, is exactly when someone copies the eleventh implementation with the backoff halved "just for now." The fleet's answer is the same one it already uses for authentication and error shapes: put it in the shared package once, so an integration class only has to build on it correctly, not reinvent it.

// api-infrastructure/src/Concerns/SendsRequests.php (condensed)
trait SendsRequests
{
    protected function send(string $method, string $url, array $data = []): Response
    {
        $maxAttempts = max(1, $this->intConfig('webplo.throttle_retry.max_attempts', 3));
        $attempt = 0;

        while (true) {
            $attempt++;
            $response = $this->dispatchRequest($method, $url, $data);

            if (! $response->failed()) {
                return $response;
            }

            if ($response->status() === 429 && $attempt < $maxAttempts) {
                $this->backoffForThrottle($response, $attempt);

                continue;
            }

            throw new RequestException(response: $response);
        }
    }

    private function backoffForThrottle(Response $response, int $attempt): void
    {
        $base = max(0, $this->intConfig('webplo.throttle_retry.base_delay_ms', 500));
        $max = max(0, $this->intConfig('webplo.throttle_retry.max_delay_ms', 5000));

        $retryAfter = $response->header('Retry-After');
        $delayMs = is_numeric($retryAfter)
            ? (int) ((float) $retryAfter * 1000)
            : $base << ($attempt - 1);

        usleep(max(0, min($delayMs, $max)) * 1000);
    }
}

Everything a leaf-specific integration needs from this is inherited, not written. GithubAPI proves it:

// app/Http/Integrations/Cloud/GithubAPI.php
final class GithubAPI
{
    use SendsRequests {
        send as protected sendRequest;
    }
    use TracksDependencyHealth;

    protected function send(string $method, string $url, array $data = []): Response
    {
        return $this->trackDependency(
            Dependency::GitHub,
            fn (): Response => $this->sendRequest($method, $url, $data),
        );
    }
}

GithubAPI doesn't know what a 429 is. It doesn't have a usleep() call anywhere in it. It aliases the trait's send() to sendRequest(), wraps that single call in its own dependency tracker, and every method it defines after this, createRepositoryFromTemplate(), checkHealth(), whatever comes next, gets the retry-and-backoff behavior for free, because it's built on send(), not on Http::post() directly. Honoring Retry-After correctly is a detail worth getting right exactly once. Bake it into the trait, and no leaf service added to the fleet next year can get backoff wrong, because none of them ever have to write it.

This is the same principle Chapter 3 applied to authentication and Chapter 4 applied to error rendering, just aimed at outbound calls instead of inbound ones. A cross-cutting concern that lives in a shared trait is a structural guarantee. A cross-cutting concern every integration is supposed to remember is a policy with a hole in it, waiting for the one class nobody got around to updating.

Recap

Chaining across services

• [X] Treat every step as a call to a system you don't control the database for, not just a job that might run twice
• [X] Guard each step by checking the leaf's own confirmed state, not just your own attempt count
• [X] Stop a batch, don't let siblings keep mutating state, the moment one step has already failed

Naming that survives a retry

• [X] Derive external resource names from something stable and already unique, like the owning record's id, never a timestamp or random suffix
• [X] A deterministic name converges a retried workflow onto one resource instead of leaking a new orphan per attempt

Reaping what silently stalls

• [X] Add a scheduled sweep that looks for work stamped as started and never stamped as finished, past a threshold
• [X] Cap how much a sweep acts on per run, and refuse to exceed that cap instead of silently truncating
• [X] Re-check state right before acting, and route through the same idempotent terminal-failure path a normal failure already uses

Retry and backoff as shared infrastructure

• [X] Put HTTP retry/backoff behavior in one shared trait every integration client builds on, not in each integration separately
• [X] Let a leaf-specific API class add its own concerns on top of that shared send(), never around it

A dozen APIs that behave like one system don't get there because everyone remembered the same rules. They get there because the rules only had to be written once, and forgetting them stopped being an option.