Tracing a request you cannot reproduce

What do you do when the failure already happened and you can't make it happen again? You don't reproduce it. You reconstruct it from what was already recorded, which is the entire point of collecting these signals before you need them.

Request hits the API app
↓
LogApiRequestsMiddleware records the start time, lets the request through
↓
Response returns to the caller immediately
↓
defer() runs after the response has already been sent
↓
SensitiveDataSanitizer redacts the request and response payloads
↓
SendToLogsJob queues the sanitized record
↓
LogFacade ships it to the shared logs sink

// api-infrastructure/src/Middleware/LogApiRequestsMiddleware.php
public function handle(Request $request, Closure $next): Response
{
    $startTime = microtime(true);
    $response = $next($request);

    defer(
        callback: function () use ($request, $response, $startTime) {
            $bearer = $request->attributes->get('bearer');
            $sanitizedRequest = SensitiveDataSanitizer::sanitize($request->all());
            $sanitizedResponse = SensitiveDataSanitizer::sanitize(ResponseParser::parse($response));

            SendToLogsJob::dispatch(
                user_id: $bearer?->id ?? 0,
                method: $request->method(),
                endpoint: $request->url(),
                request: $sanitizedRequest,
                response: $sanitizedResponse,
                status: $response->getStatusCode(),
                execution_time: microtime(true) - $startTime,
                bearer_token: $bearer?->token,
                // ... payload size fields omitted here
            );
        }
    );

    return $response;
}

• defer(): the sanitizing, serializing, and queueing all happen after the response has already gone back to the caller, so logging overhead never adds to a merchant's response time.
• SensitiveDataSanitizer: every request and response body gets redacted for password, token, and credential-shaped keys before anything is queued, so the trace you pull six months from now doesn't hand you a live bearer token in a support thread.
• SendToLogsJob: carries method, endpoint, status, execution_time, and the bearer's user_id, and it implements ShouldBeEncrypted because the payload sitting on the queue table is still sensitive even redacted.

That covers a single HTTP request. A background job is a harder trace, because by the time it fails it may have already called two or three other services on your behalf, and none of those calls are visible in the original request log. TrackJobExecutionMiddleware, applied per job via its middleware() method, closes that gap.

// api-server/app/Http/Middleware/TrackJobExecutionMiddleware.php
public function handle(object $job, callable $next): mixed
{
    if (! $this->shouldTrackJob($job)) {
        return $next($job);
    }

    try {
        $log = $this->createJobLog($job);
    } catch (Throwable $e) {
        return $next($job);
    }

    $this->tracker->clear();

    return $this->executeAndTrack($job, $next, $log);
}

Creating that log row can itself fail, a database blip, a migration mid-flight, and when it does, the job still has to run. The catch block falls through to $next($job) untracked: you lose the trace for that one execution, not the execution itself. Tracing is a diagnostic layer bolted onto the job, never a dependency the job's actual work waits on.

• JobsLog: one row per job execution, keyed to the site_id the job was working on, carrying status, duration_ms, and error_message once the job resolves.
• ApiCallTracker: cleared at the start of every tracked job, then populated as the job calls out to other services. When the job finishes, if the tracker recorded anything, every one of those calls, with its method, URL, status, and both bodies, gets written into that same JobsLog row's metadata.api_calls.
• Why this matters: when a site's license activation fails and support asks what happened, you don't reproduce the bug. You pull the JobsLog row for that site_id and read the exact upstream calls, in the order they actually happened, with the status code each one returned.

What to do with the first five minutes of an incident

When something is actually down, the first five minutes decide whether the next hour is calm or chaotic. Do these in order, before you open a single file:

1. Check the health endpoint for every app in the affected path, not just the one that paged you. A 503 from HealthController on one API explains a wave of failures in another far faster than a stack trace does.
2. Check the four signals for the last thirty minutes, not the last five. A saturation trend or an error-rate creep that started twenty-five minutes ago is your real starting point, not the moment the alert fired.
3. Pull the JobsLog rows for the affected site_id or bearer token, if the failure is scoped to one customer. metadata.api_calls will usually show you which downstream call actually failed before you've read a single line of application code.
4. Decide whether it's a symptom you already have a name for. A 504 means the exception renderer chain already caught a ConnectionException before ApiExceptionRenderer ever saw it; you already know it's an upstream timeout, not your code.
5. Only then, open a file. Everything before this step is reading data you already collected. Everything after it is debugging, and debugging without the first four steps is guessing with extra confidence.

Chapter 9 Summary

Health checks

• [X] Every dependency a request actually needs gets its own check, not one shared boolean
• [X] The health endpoint returns a failure status when any dependency is down, not just when the process itself is dead

Signals

• [X] Volume, latency, errors, and saturation are all visible without reading a stack trace
• [X] Latency is tracked at p95, not average, and errors are read by status code, not lumped together

Alerting

• [X] Alerts fire on customer-visible symptoms, not on retries still within budget
• [X] Exceptions are classified by severity before they ever reach an error tracker

Tracing

• [X] Every request and every tracked job leaves a redacted, queryable trail behind it
• [X] A failure can be reconstructed from stored data instead of reproduced live

A dashboard tells you something is wrong. A trace tells you what. Build both, and the next incident is a five-minute read instead of an afternoon of guessing.