Chaining long multi-step work

Some work genuinely can't be one job. Provisioning a site involves creating a repository, then, only once that repository exists, standing up hosting and generating content in parallel, then deploying, then confirming the deploy actually landed. Laravel's Bus::chain() runs steps strictly in order, and a chain step can itself be a Bus::batch() of jobs that run in parallel:

dispatch chain
↓
create repository
↓
batch (parallel): CDN setup · content generation · hosting setup
↓
deploy
↓
mark provisioning complete

// app/Jobs/SiteProvisionJob.php (condensed)
public function handle(): void
{
    self::markProvisionStarted($this->site, $this->webhook);

    Bus::chain([
        (new RepositoryCreateJob(site: $this->site, token: $this->token, webhook: $this->webhook))->onQueue('default'),
        Bus::batch([
            new CDNTenantCreateJob(site: $this->site, token: $this->token, webhook: $this->webhook),
            new AiContentCreateJob(site: $this->site, token: $this->token, webhook: $this->webhook),
            new ForgeSiteCreateJob(site: $this->site, token: $this->token, webhook: $this->webhook),
        ])->allowFailures(false),
        (new ForgeSiteDeployJob(site: $this->site, token: $this->token, webhook: $this->webhook))->onQueue('default'),
        function () {
            self::clearStage($this->site);
        },
    ])->catch(function (Throwable $exception) {
        self::failTerminally($this->site, $this->webhook, $exception);
    })->onQueue('default')->dispatch();
}

// Belt and suspenders: if the OUTER job fails before the chain is even
// dispatched, the chain's own ->catch() never runs at all.
public function failed(Throwable $exception): void
{
    self::failTerminally($this->site, $this->webhook, $exception);
}

allowFailures(false) on the batch means one failed job in that parallel group cancels the rest of the batch instead of letting siblings keep mutating external state around a doomed sibling. The chain's ->catch() is the one place that decides what "this multi-step process failed" means: it doesn't just log, it tears down whatever partial state got created. And failed() on the outer job exists because the chain's ->catch() only runs if the chain actually got dispatched; if the outer job itself throws before that (a serialization error, a queue connection blip), nothing else will ever call the cleanup path. Two different failure points, one shared terminal handler, made safe to call twice by the same Cache::add() claim from the idempotency section above.

Watchdogs for work that silently stalls

A job that throws is easy. A job that's polling an upstream API and just... keeps polling, forever, because the upstream never reaches a terminal state, is the harder failure. $tries won't save you here, because the job isn't failing, it's succeeding at each individual attempt and asking to be retried. You need a different stop condition: not "how many times," but "until when."

// app/Jobs/ForgeSiteProvisioningWaitJob.php (condensed)
class ForgeSiteProvisioningWaitJob implements ShouldBeEncrypted, ShouldQueue
{
    use Batchable, Queueable;

    public int $tries = 0; // Bounded by retryUntil() instead of a fixed attempt count.
    public int $timeout = 80;

    public function backoff(): array
    {
        return [1, 2, 3, 3, 3, 3, 4, 4, 5];
    }

    public function retryUntil(): \DateTimeInterface
    {
        return now()->addMinutes(3);
    }

    public function handle(): void
    {
        if ($this->batch()?->cancelled()) {
            return;
        }

        // ... resolve the bearer's Forge credentials, then poll.
        $api = ForgeNewAPI::boot($forgeToken);
        $api->existing($this->site);
        $this->site->refresh(); // read what Forge just reported, not stale metadata

        $status = $this->site->metadata['site']['status'] ?? '';

        if (in_array($status, ['creating', 'provisioning', 'installing', 'pending'], true)) {
            throw new PollingRetryException("Site provisioning in progress. Current status: {$status}");
        }

        if (in_array($status, ['created', 'installed'], true)) {
            return;
        }

        // Unknown/failed status is TERMINAL. With tries=0 + retryUntil(), a plain
        // throw would just keep polling a dead provision for the full window.
        // fail() bypasses retryUntil and ends the job now.
        $this->fail(new \Exception("Site provisioning failed or unexpected status: {$status}"));
    }
}

$tries = 0 paired with retryUntil() is a deliberate substitution: instead of "try nine times," it's "keep trying until three minutes from now, using whatever backoff schedule gets you there." PollingRetryException is a dedicated, ShouldntReport exception used only to mean "not done yet, ask again," so a normal polling cycle never shows up as noise in error tracking. And $this->fail() matters precisely because it's not a throw: a thrown exception under retryUntil() just gets retried again until the deadline, so a genuinely dead upstream status has to be failed explicitly, on purpose, right now, or it will poll a corpse for three more minutes.

retryUntil() handles the job that's still running but taking too long. It doesn't help with the job that never got picked up at all, or a multi-step chain that hung between two steps with nothing left throwing exceptions anywhere. For that you need something outside the job entirely: a scheduled command that looks for work stamped as started but never marked complete or failed, past a threshold, and force-fails it using the same terminal path a normal failure would use. That's the job of a watchdog: it doesn't wait to be told something is stuck, it goes looking.