The four signals worth a dashboard

Every request that hits an API app in this fleet passes through LogApiRequestsMiddleware, and every one of those requests carries the same four numbers worth putting on a dashboard: how many, how fast, how often they fail, and how close to the limit they're running. Get these four in front of you and you can answer "is the API healthy" without reading a single stack trace.

AddRateLimitHeadersMiddleware is the clearest illustration of the fourth one, saturation, because it turns an internal counter into something visible on every response.

// api-infrastructure/src/Middleware/AddRateLimitHeadersMiddleware.php
final readonly class AddRateLimitHeadersMiddleware
{
    public function __construct(
        private RateLimiter $limiter,
    ) {}

    public function handle(Request $request, Closure $next): Response
    {
        $response = $next($request);
        $token = $request->bearerToken() ?? $request->ip();

        if (! is_string($token)) {
            $token = 'unknown';
        }

        $key = 'api-token:'.$token;
        $maxAttempts = (int) config('webplo.rate_limit');
        $remainingAttempts = $this->limiter->remaining($key, $maxAttempts);

        $response->headers->set('X-RateLimit-Limit', (string) $maxAttempts);
        $response->headers->set('X-RateLimit-Remaining', (string) max(0, $remainingAttempts));

        return $response;
    }
}

Enforcement already happened in throttle:api-token before this middleware ever runs; this one only reports the number back. The other three signals work the same way, visible without opening a debugger:

• Volume: requests landing, broken down by endpoint and status code. A drop to zero is as alarming as a spike; it usually means a client stopped calling you, not that everyone stopped needing you.
• Latency: execution_time, measured in LogApiRequestsMiddleware around $next($request). Track the p95, not the average. The average hides the one slow endpoint dragging a single merchant's dashboard.
• Errors: the status code on every logged request. A 422 from a rejected CreateLicenseRequest payload, thrown by the shared BaseFormRequest's own failedValidation(), is not the same signal as a 500 from an unhandled exception that reaches ApiExceptionRenderer; a dashboard that treats them the same trains you to ignore both.
• Saturation: X-RateLimit-Remaining, trending across your whole caller base, not one token. If that number drifts toward zero everywhere, you're about to see a wave of 429s regardless of what your latency graph says.

One endpoint tells you it's slow. Another tells you it's failing. A third tells you it's about to be throttled. None of them alone tells you the API is healthy. All four together do.

Alerting on symptoms, not causes

Here's the uncomfortable truth: most alerts fire on causes, not symptoms, and that's why on-call engineers learn to ignore them. A disk at high capacity is a cause. A merchant's site returning 500s is a symptom. Page on the second. Log the first, and read it when you're not already being woken up.

// Bad: every exception gets the same alert weight
$exceptions->report(function (Throwable $exception) {
    Nightwatch::captureException($exception);
});

Every exception in that version pages the same way, whether it's a client sending malformed JSON or a genuine unhandled fault. The team that alerts uniformly ends up with a Slack channel everyone mutes within a month. Here's what api-license/bootstrap/app.php actually does instead.

// api-license/bootstrap/app.php
$exceptions->report(function (Throwable $exception) {
    try {
        if ($exception instanceof ThrottleRequestsException) {
            Nightwatch::warning('Rate limit exceeded', [
                'exception' => class_basename($exception),
            ]);
        } elseif ($exception instanceof ValidationException) {
            Nightwatch::info('Validation failed', [
                'errors' => $exception->errors(),
            ]);
        } elseif ($exception instanceof ModelNotFoundException) {
            Nightwatch::warning('Resource not found', [
                'exception' => class_basename($exception),
            ]);
        } else {
            Nightwatch::captureException($exception);
        }
    } catch (Throwable) {
        // Nightwatch not available or container not initialized
    }
});

• ThrottleRequestsException: a client hitting its rate limit is expected behavior under load, not a system fault. It's logged as a warning so a spike is visible in aggregate, but nobody gets paged because one caller ran hot.
• ValidationException: a client sending bad data is their bug, not yours. info keeps a record without treating a rejected payload as an incident.
• ModelNotFoundException: worth a warning, since a wave of these can mean a client is probing for resources that don't exist. A single one is just a stale link.
• Everything else: captureException is the only branch that reaches your error tracker with a full stack trace, because by construction, everything that reaches it is something nobody already had a name for.

The same discipline shows up one layer down, inside individual jobs. LicenseCreateJob retries a failed activation call across five attempts spread over roughly six hours before it gives up.

// api-server/app/Jobs/LicenseCreateJob.php
} catch (Throwable $e) {
    // Only surface the exception (and report to Nightwatch) on the
    // final attempt. Otherwise release silently using the configured
    // backoff so transient upstream failures don't generate noise.
    if ($this->attempts() < $this->tries) {
        $this->release($this->nextBackoffSeconds());

        return;
    }

    throw $e;
}

The first four failures are the cause: an upstream service was briefly unavailable, which is expected and silent. The fifth is the symptom: a license that still isn't active after every reasonable attempt, and only that one throws far enough to reach Nightwatch. Six months from now, whoever's on call will thank you for the difference.