Data Management

Every piece of data you collect has a cost. Storage costs money. Logs fill up disk space. Old data you're not using anymore is just clutter.

Good data management means:

1. Retention policies: Define how long you keep data; old logs can be deleted.
2. Archival: Cold data moves to cheaper storage.
3. Privacy: GDPR, CCPA, etc. You need to handle deletion requests.
4. Compliance: Some industries have specific data retention requirements.

Data management is boring but important. Ignore it and you'll have a disk full of useless logs someday.

Log Retention Policies

Automatically clean old logs:

// Keep logs for 30 days
File::cleanDirectory(storage_path('logs'), maxAge: 2592000); // 30 days in seconds

// Or use artisan command
php artisan logs:clear

// Schedule daily cleanup
class Kernel extends ConsoleKernel
{
    protected function schedule(Schedule $schedule)
    {
        $schedule->command('logs:clear')->daily();
    }
}

Database Archival

Move old data to cold storage:

// Archive requests older than 90 days to S3
class ArchiveOldRequests extends Command
{
    public function handle()
    {
        $cutoff = now()->subDays(90);

        // Get old records
        $records = DB::table('api_requests')
            ->where('created_at', '<', $cutoff)
            ->limit(10000)
            ->get();

        if ($records->isEmpty()) {
            return;
        }

        // Upload to S3
        Storage::disk('s3')->put(
            path: "archives/" . now()->format('Y-m-d-H-i-s') . ".json",
            contents: json_encode($records),
        );

        // Delete from database
        DB::table('api_requests')
            ->where('created_at', '<', $cutoff)
            ->limit(10000)
            ->delete();

        Log::info('Archived {count} old requests', ['count' => $records->count()]);
    }
}

// Schedule daily
$schedule->command('archive:requests')->daily()->at('2:00');

Search & Analytics

Query historical data efficiently:

class AnalyticsService
{
    // Most common errors in last 7 days
    public function topErrors(int $days = 7): Collection
    {
        return DB::table('api_request_logs')
            ->where('created_at', '>=', now()->subDays($days))
            ->where('status', '>=', 400)
            ->groupBy('error_code')
            ->selectRaw('error_code, COUNT(*) as count')
            ->orderByRaw('COUNT(*) DESC')
            ->limit(10)
            ->get();
    }

    // API performance trends
    public function performanceByEndpoint(int $days = 7): Collection
    {
        return DB::table('api_request_logs')
            ->where('created_at', '>=', now()->subDays($days))
            ->groupBy('path')
            ->selectRaw('path, AVG(duration_ms) as avg_duration, MAX(duration_ms) as max_duration')
            ->orderByRaw('AVG(duration_ms) DESC')
            ->get();
    }
}

Chapter 11 Summary

Data has a lifecycle. It's not infinite.

Best practices:

1. Define retention policies for each data type
2. Automatically clean old logs and data
3. Archive cold data to cheaper storage
4. Respect privacy regulations (GDPR, CCPA)
5. Document what data you keep and why

Data management isn't glamorous but it saves money and keeps you compliant.