Idempotency has to survive the hop, not just the retry

Chapter 6's Cache::add() claim key works when the side effect it's protecting lives entirely inside api-server's own transaction. A repository created on a git host doesn't live there. The claim key can tell two duplicate deliveries of the same job apart, but it can't tell you whether the first delivery already succeeded out on the leaf service before the worker that ran it died. For that you need to check the leaf's own state, not your queue's.

// app/Jobs/ForgeSiteCreateJob.php
public function handle(): void
{
    // A sibling in the same provisioning batch failed (allowFailures(false)
    // cancels the batch): stop before creating/mutating any external resource,
    // otherwise an in-flight job can leave an orphan the teardown already ran past.
    if ($this->batch()?->cancelled()) {
        return;
    }

    $this->site->refresh();

    /** @var array<string, mixed> $metadata */
    $metadata = (array) $this->site->metadata;

    // Idempotency guard: if a previous attempt already created the Forge site
    // and the listener persisted its ID to metadata, skip creation to avoid a
    // 422 from Forge.
    if (isset($metadata['site']['id'])) {
        return;
    }

    // ...build the payload and call the hosting leaf service
}

Walk through what each piece is actually defending against:

• $this->batch()?->cancelled(): the parallel batch this job runs in was configured allowFailures(false). If a sibling step already failed, this guard stops the job before it creates anything new next to a doomed attempt, an orphan the teardown path was never told about.
• isset($metadata['site']['id']): the actual idempotency check, and it's not against the queue, it's against the durable record of what the leaf service already confirmed. The first attempt's listener wrote that id back to metadata the moment the leaf service responded. A second delivery of this job, worker crash or manual retry, reads that same durable state and returns immediately, doing nothing.

Compare this to the naive version, which is exactly what "just retry it" looks like before you've thought about what's on the other end of the call:

// Bad: retries without checking whether the first attempt already landed
public function handle(): void
{
    $api = ForgeNewAPI::boot(apiKey: $this->token);
    $result = $api->createSite(server_id: $this->serverId, payload: $this->buildPayload());
    $this->site->metadata = array_merge($this->site->metadata, ['site' => $result]);
    $this->site->save();
}

Run this twice, worker death between the createSite() call and the save() being the easiest way to get there, and you get two hosting entries pointed at the same site, one tracked in metadata, one invisible to everything api-server does afterward, including teardown. It's not a bug that shows up in a test. It's a bug that shows up three weeks later as a line item on an infrastructure bill nobody can explain.

The rule underneath this: idempotency in a multi-service chain is a property of the durable record, not the job. A job that checks "have I run before" against its own attempt count will happily re-create the same external resource every time it retries. A job that checks "does this resource already exist, according to the last thing that told me so" converges, no matter how many times it runs.