Watching a queue you cannot see — Chapter 7: Queue & Background Processing

A queue worker is a process running somewhere you're not looking at, which is exactly the point of using one and exactly why it needs instrumentation the request/response cycle gets for free. The fleet wraps job execution in its own middleware for this:

// app/Http/Middleware/TrackJobExecutionMiddleware.php (api-server, app-local, not shared)
public function handle(object $job, callable $next): mixed
{
    if (! $this->shouldTrackJob($job)) {
        return $next($job);
    }

    $log = $this->createJobLog($job);
    $this->tracker->clear();

    return $this->executeAndTrack($job, $next, $log);
}

Every job that opts in, by declaring a middleware() method that returns [Middleware\TrackJobExecutionMiddleware::class], gets a row written before it runs and updated after: status, duration in milliseconds, any outbound API calls made during the attempt, and the error message if it failed. That's a per-execution audit trail for work nobody watches happen in real time.

The failed-jobs and job-batches tables get the same treatment at the fleet level, not the single-job level: an internal dashboard reading job_batches for progress percentages and pending/failed counts, and failed_jobs for what needs a decision. Neither view replaces the other. The middleware answers "what happened on this one attempt." The dashboard answers "what's the state of everything in flight right now." A queue you can't see is a queue you find out about from a support ticket. Build the visibility in before you need it, not after the first incident makes you wish you had.

Recap

Dispatch decisions

[X] Queue work the caller doesn't need the result of immediately; keep synchronous anything the response body depends on
[X] Encrypt job payloads that carry credentials with ShouldBeEncrypted

Making retries safe

[X] Treat every job as at-least-once: it will run twice eventually, design for that now
[X] Use an atomic cache claim to collapse duplicate deliveries of a job whose side effect isn't naturally idempotent
[X] Pick $tries + backoff() deliberately, fast retries for transient failures, slow ones for real outages
[X] Use retryUntil() instead of $tries for polling work with no natural attempt count, and fail() explicitly for terminal states so a dead status doesn't poll out the clock

When it still fails

[X] Give failed() a real job: flip status, notify a human, never leave it empty
[X] Treat failed_jobs as a queue for a human decision, not a place jobs go to be forgotten
[X] Chain multi-step work with Bus::chain()/Bus::batch(), and give the chain one shared, idempotent terminal-failure handler reachable from both ->catch() and the outer job's own failed()
[X] Add a watchdog for work that can stall without ever throwing; nothing else will notice for you

A background job is a promise you made to finish some work without anyone watching. Keep it the way you'd keep any promise nobody's checking on: assume you'll be tested on it anyway.