A sweep that doesn't trust anyone to notice

Chapter 6's ForgeSiteProvisioningWaitJob used retryUntil() to bound a job that polls an upstream status instead of failing outright. That mechanism has a hard limit: it only protects a job that's still running. It has nothing to say about a chain that stops between two steps, a worker killed by a deploy right as it picks up a job, a step that never got dispatched because of a queue connection blip. Nothing in that failure throws. Nothing polls. There's no job left anywhere to time out. The chain simply goes quiet, and the merchant's dashboard sits on a loading spinner that will never resolve on its own.

Catching that needs a mechanism that lives outside the queue entirely: a scheduled sweep that doesn't wait to be told something is wrong, it goes looking, on a fixed interval, for work that was stamped as started and never stamped as finished.

// app/Console/Commands/ProvisionReapCommand.php (condensed)
final class ProvisionReapCommand extends Command
{
    private const MAX_REAP_PER_RUN = 25;

    public function handle(): int
    {
        $thresholdMinutes = max(1, (int) config('services.server.management.provision_stuck_after_minutes', 30));
        $cutoff = Carbon::now()->subMinutes($thresholdMinutes);

        $candidates = Site::query()
            ->whereNotNull('metadata->provision->started_at')
            ->whereNull('metadata->terminal_failure')
            ->whereNull('metadata->health_check->completed_at')
            ->where('metadata->provision->started_at', '<', $cutoff->toIso8601String())
            ->limit(self::MAX_REAP_PER_RUN + 1)
            ->get();

        if ($candidates->count() > self::MAX_REAP_PER_RUN) {
            $this->components->error('Refusing to reap: matched more stuck provisions than the per-run cap. Investigate before re-running.');

            return self::FAILURE;
        }

        foreach ($candidates as $site) {
            $site->refresh(); // re-check immediately before acting; it may have finished on its own

            if (! self::isStillStuck($site, $cutoff)) {
                continue;
            }

            SiteProvisionJob::failTerminally(
                $site,
                webhook: $site->metadata['provision']['webhook'] ?? null,
                exception: new RuntimeException("Provision watchdog: exceeded {$thresholdMinutes} minute threshold without completing."),
            );
        }

        return self::SUCCESS;
    }
}

Four decisions in this file are doing more work than their line count suggests:

• The query is the failure signal, not an exception. Nothing about a stuck provision throws. What makes it detectable is a durable timestamp, metadata->provision->started_at, written the moment the chain begins, checked against a threshold on a schedule. The absence of a completion marker after enough time has passed is the failure.
• MAX_REAP_PER_RUN is a refusal, not a cap that quietly truncates. If more than 25 sites match, the command errors out instead of reaping 25 and moving on. Matching more than a handful of stuck provisions doesn't look like normal attrition, it looks like something systemic broke, and a sweep that mass-fails sites through a systemic outage is worse than one that stops and makes a human look.
• Re-checking right before acting closes a race the first query can't. The candidate list is a snapshot. By the time the loop reaches a given row, that site may have finished on its own, seconds ago. Re-reading fresh state before calling failTerminally() means the sweep never tears down a site that just went live.
• The terminal-failure path is reused, not reimplemented. SiteProvisionJob::failTerminally() is the same method the chain's own ->catch() calls, already made idempotent with a Cache::add() claim back in Chapter 6. That's what makes it safe for two unrelated code paths, a worker's own failure handler and a sweep on a schedule, to call it on the same site without coordinating.

This is the second line of defense the chapter's opening promised, and it earns that description honestly: it doesn't replace the job-level retry policy from Chapter 6, it exists because that policy, however carefully tuned, cannot see a failure that never throws.