Retries, backoff, and giving up

$tries and backoff aren't decorations, they're the actual policy for how hard the system fights before it admits a job has failed. Compare the two license jobs' policies and the shape of the decision becomes clear:

// app/Jobs/LicenseCreateJob.php
class LicenseCreateJob implements ShouldBeEncrypted, ShouldQueue
{
    use Queueable;

    public int $tries = 5;
    public bool $deleteWhenMissingModels = true;
    public bool $failOnTimeout = false;

    public function backoff(): array
    {
        return [180, 300, 10800, 10800];
    }

    public function handle(): void
    {
        $this->site->refresh();

        if ($this->site->trashed()) {
            return;
        }

        $url = (string) ($this->site->metadata['url'] ?? '');

        try {
            $result = (new LicenseCreateAction)(token: $this->token, url: $url);
        } catch (Throwable $e) {
            if ($this->attempts() < $this->tries) {
                $this->release($this->nextBackoffSeconds());

                return;
            }

            throw $e;
        }
        // ...
    }
}

Read the backoff schedule as a sequence, because that's what it is: attempt one runs immediately, attempt two waits three minutes, attempt three waits five more, and attempts four and five each wait three hours. The first two attempts absorb a brief upstream hiccup. The last two exist for something closer to a maintenance window. That's a deliberate curve, not a random guess, and it's why backoff() is a method here instead of a flat integer: different failures deserve different patience.

A few properties do quiet, important work in this class. deleteWhenMissingModels means that if the Site this job was constructed with gets deleted before the job runs, Laravel discards the job instead of throwing a ModelNotFoundException at you in production. failOnTimeout = false means a slow upstream response rejoins the retry loop instead of jumping straight to failed(), so one slow response doesn't burn the whole backoff schedule on the first attempt. And the retry loop itself is deliberately quiet: release() on every attempt except the last, so Nightwatch only hears about this once, on the attempt that actually gives up.

Giving up is its own event, not just the absence of another retry:

public function failed(Throwable $exception): void
{
    $this->writeStatus('failed');

    $hostname = (string) ($this->site->metadata['url'] ?? '');

    app(StaffNotifier::class)->notifyPostProvisionJobFailed(
        kind: 'license',
        siteId: (string) $this->site->getKey(),
        hostname: $hostname,
        exception: $exception,
    );
}

failed() only runs once, on the attempt that exhausts $tries (or crosses retryUntil(), more on that shortly). Whatever a human needs to know to act on this failure, a status flip, a support notification, belongs here and nowhere else in the job.

The failed jobs table is not a graveyard

A job that exhausts its retries lands in failed_jobs and stops there. Nothing about the queue automatically retries it again; the table is where dead jobs sit until a human looks at them. Treating that table as a place things go to disappear is how retryable failures quietly become permanent ones.

Laravel gives you queue:retry, queue:forget, and queue:flush for exactly this, and the fleet builds an operator UI on the same primitives rather than reinventing them:

// app/Actions/Admin/Ops/RetryFailedJobsAction.php (condensed)
class RetryFailedJobsAction
{
    public function __construct(private readonly WriteAdminAuditAction $audit) {}

    public function execute(array $uuids): int
    {
        Gate::authorize('admin.ops');

        $connection = (string) config('admin.ops_connection');
        $retried = 0;

        foreach ($uuids as $uuid) {
            $record = DB::connection($connection)->table('failed_jobs')->where('uuid', $uuid)->first();

            if ($record === null) {
                continue;
            }

            DB::connection($connection)->table('jobs')->insert([
                'queue' => $record->queue,
                'attempts' => 0,
                'reserved_at' => null,
                'available_at' => now()->getTimestamp(),
                'created_at' => now()->getTimestamp(),
                'payload' => $this->prepare((string) $record->payload), // fresh uuid, zeroed attempts
            ]);

            DB::connection($connection)->table('failed_jobs')->where('uuid', $uuid)->delete();
            $retried++;
        }

        $this->audit->execute(user: $this->actor(), action: 'failed_jobs.retry', payload: ['uuids' => $uuids, 'retried' => $retried]);

        return $retried;
    }

    // prepare() and actor() omitted: re-stamp the payload's uuid/attempts and resolve the acting user.
}

Every retry here goes through Gate::authorize() and every outcome gets written to an audit log, and the sibling ForgetFailedJobsAction takes an optional $reason string that's recorded in that same audit payload alongside the deletion. That's the right instinct: retrying a failed job re-attempts a side effect, forgetting one discards the only record that side effect never happened, and neither is a button you want pressed by accident.