Monitoring & Observability

You will not be the first to notice most outages. A merchant refreshing their dashboard notices first. A Statamic call starts timing out at 2 AM and nobody is awake to see it happen. A rate limit spike shows up in a monthly report three days after it mattered. The gap between "something broke" and "you know something broke" is the only part of an outage you actually control. Close it.

This chapter covers two jobs people constantly collapse into one: monitoring and observability. Monitoring answers questions you thought to ask in advance: is the database up, is the error rate climbing, is the queue backed up. Observability answers the question you didn't think to ask: why did this one bearer token's requests start timing out at 9:47 this morning, and what upstream call was in flight when it happened. A health check won't tell you why a merchant's site went down. A trace of that merchant's failed request will.

Monitoring tells you something is wrong. Observability tells you what.

Health checks that mean something

A health check that always returns {"status": "ok"} is worse than no health check at all, because it lies to you with confidence. Your load balancer keeps routing traffic to a box whose database connection pool is exhausted, because the one thing that health check actually verified was that the web server process could still render JSON.

The fleet's shared health check, Webplo\ApiInfrastructure\Http\Controllers\HealthController, verifies the specific things that keep a request from completing instead.

// api-infrastructure/src/Http/Controllers/HealthController.php
final readonly class HealthController
{
    public function __invoke(): Responsable
    {
        $databaseStatus = (new DatabaseHealthCheck)->check();
        $cacheStatus = (new CacheHealthCheck)->check();
        $queueStatus = (new QueueHealthCheck)->check();

        return new ModelResponse(
            data: new HealthResource(
                resource: HealthDTOMapper::toDTO(
                    databaseStatus: $databaseStatus,
                    cacheStatus: $cacheStatus,
                    queueStatus: $queueStatus,
                ),
            ),
            status: $this->status($databaseStatus, $cacheStatus, $queueStatus),
        );
    }

    // status() maps the three booleans to HTTP_OK or HTTP_SERVICE_UNAVAILABLE
}

Every API app in the fleet pulls this controller in from the shared package instead of writing its own, and it returns HTTP_SERVICE_UNAVAILABLE the moment any one of the three checks below fails, HTTP_OK only when all three pass.

• DatabaseHealthCheck: calls DB::getPdo() and then runs SELECT 1. If the connection pool is exhausted or the credentials are wrong, this fails immediately instead of waiting for a real query to time out later in the request.
• CacheHealthCheck: on the generic driver it writes a random key, reads it back, and confirms the value round-trips before forgetting it; on Redis it pings the connection instead. A cache miss on a random key doesn't prove the store is reachable. A successful write and read does.
• QueueHealthCheck: checks whichever driver is actually configured, counting rows on the database queue table or pinging Redis. A sync connection always passes, because there's no separate worker process that could be down.

Why three checks instead of one boolean? Because a single boolean tells you the service is unhealthy. Three checks tell you which dependency is unhealthy, and that's the difference between a five-minute fix and a twenty-minute one spent guessing.