Long-Running Exports

A partner asks for a CSV of every domain on one of their sites. The direct version looks harmless:

// Bad: builds the whole export inside the request
public function export(): StreamedResponse
{
    $domains = Domain::all(); // every row, in memory, before the first byte streams

    return response()->streamDownload(function () use ($domains) {
        $handle = fopen('php://output', 'w');
        fputcsv($handle, ['hostname', 'status', 'activated_at']);
        foreach ($domains as $domain) {
            fputcsv($handle, [$domain->hostname, $domain->status->value, $domain->activated_at]);
        }
        fclose($handle);
    }, 'domains.csv');
}

This works fine in staging with forty test domains, and fine for your smallest customer. Then a partner with tens of thousands of domains hits it: Domain::all() tries to hold every row in memory before the response even starts, the request exhausts memory or runs past the timeout, and the partner just sees a connection that hangs.

Move the work off the request entirely. Queue it, walk the table in chunks so memory stays flat regardless of row count, and tell the caller when it's done the same way you'd tell them about anything async:

// app/Http/Controllers/DomainController.php
public function export(ExportDomainsRequest $request, string $id): MessageResponse
{
    $site = Site::findOrFail($id);

    DomainExportJob::dispatch(
        siteId: $site->id,
        webhook: $request->validated('webhook'),
    );

    return new MessageResponse(
        message: 'Export queued. You will receive a webhook when the file is ready.',
        status: Response::HTTP_ACCEPTED,
    );
}

// app/Jobs/DomainExportJob.php
final class DomainExportJob implements ShouldQueue
{
    use Queueable;

    public int $timeout = 600;

    public function __construct(
        public string $siteId,
        public ?string $webhook,
    ) {}

    public function handle(): void
    {
        $path = storage_path('app/exports/domains-'.$this->siteId.'-'.now()->timestamp.'.csv');
        $handle = fopen($path, 'w');
        fputcsv($handle, ['hostname', 'status', 'activated_at']);

        Domain::query()
            ->where('site_id', $this->siteId)
            ->orderBy('id')
            ->chunkById(500, function ($domains) use ($handle) {
                foreach ($domains as $domain) {
                    fputcsv($handle, [
                        $domain->hostname,
                        $domain->status->value,
                        $domain->activated_at?->toIso8601String(),
                    ]);
                }
            });

        fclose($handle);

        if ($this->webhook) {
            Http::asJson()->post($this->webhook, [
                'event' => 'domains.export.completed',
                'data' => ['path' => basename($path)],
            ]);
        }
    }
}

chunkById(500, ...) never loads more than 500 rows at once, and orders by primary key so it stays safe while other requests write to the same table. Whether the export takes four seconds or four minutes, memory stays flat. The webhook at the end closes the loop: the same delivery mechanism from earlier in this chapter tells the caller the file is ready instead of making them poll a status endpoint every few seconds.

Ten thousand domains behind this pattern cost you a queue worker for a few minutes. The synchronous version costs you a timed-out request and a support ticket. Same output file, completely different failure mode.

Knowing When a Feature Is Not Worth It

Here's the uncomfortable truth: most of what's in this chapter should not ship. Not because the patterns are wrong, they're the same patterns you've used since Chapter 2, but because most APIs never hit the scale where webhooks beat polling, where fifty domains at once is a real request, or where a search index earns its staleness risk. Building the sophisticated version of a feature nobody asked for yet is the same mistake as the premature architecture this book has argued against from page one, wearing a more advanced disguise.

Ask three questions before you build any of this:

1. Has the simple version actually failed? Not "might it fail at scale someday", has a real consumer hit a real wall with polling or a client-side filter.
2. Can you measure the cost of not having it? "Consumers poll our license endpoint 200,000 times a day to catch changes that happen 40 times a day" is a number. "It would be nice to have webhooks" is not.
3. Are you willing to operate it? A webhook system means a delivery log to monitor. A batch endpoint means a queue to keep healthy. A search index means a staleness story. If nobody owns that operational cost, the feature isn't done, it's deferred pain.

Before you build:

• [X] The plain version is live and has measurably failed
• [X] You can name the cost in a number, not a feeling
• [X] Someone owns monitoring it once it ships

Once it's built:

• [X] Delivery, batch, and export failures are provable from a table, not inferred from silence
• [X] Every retry policy distinguishes "this will never work" from "this might work in thirty seconds"
• [X] Every copy of data has a clearly-wins source of truth and a way to rebuild from it

Advanced features aren't where you get to stop being disciplined. They're where discipline matters most, because they're the features consumers build their own systems on top of. Get the basics boring first. Then make the advanced parts just as boring.