@fleetbase/fleetops-engine 0.6.69 → 0.6.70

package/docs/TELEMETRY_ARCHITECTURE.md

@@ -48,11 +48,82 @@ persistence. The performance requirement did not justify those dependencies.
 ## Tradeoffs
 
 Three shared tables separate run-level diagnostics, high-volume delivery retention,
-and connection-level secrets with different lifetimes. Provider-specific protocol
+and connection-level secrets with different lifetimes (see Retention below). Provider-specific protocol
 classes remain appropriate. Capability opt-in avoids silently changing every
 existing integration's ingestion or webhook behavior. Production throughput and
 real AFAQY payload verification remain separate deployment gates.
 
+## Retention
+
+Every poll writes one `device_events` row, one `positions` row and a delivery
+envelope per batch for each reporting device. Without a ceiling those tables grow
+until the disk fills, so retention is enforced by `fleetops:prune-telematics-data`
+(scheduled every fifteen minutes, `withoutOverlapping(14)`) using a per-company
+policy resolved by `Support\Telematics\Retention\RetentionPolicy`:
+
+| Layer | Source | Where it is edited |
+|---|---|---|
+| Package config | `server/config/telemetry.php` | deployment |
+| System policy | setting `fleet-ops.telematics-settings` | Admin console → Fleet-Ops Config → Telematics |
+| Company history preferences | `company.<uuid>.fleet-ops.telematics-settings` | Fleet-Ops → Settings → Telematics |
+
+Defaults: device events 30 days, raw payload/meta stripped ("compacted") after
+7 days, positions 90 days, processed deliveries 24 hours, quarantined deliveries
+7 days, sync runs 7 days. Organizations can choose only device event and position
+history retention, or inherit the system defaults. Administrators own raw payload,
+delivery, sync-run retention and telemetry activity logging.
+
+`max_event_retention_days` and `max_position_retention_days` constrain every
+organization's history, including inherited defaults. A maximum of `0` allows
+unlimited history; a history preference of `0` keeps records forever only while
+that maximum is unset. Setting a finite maximum replaces saved unlimited or longer
+preferences with that maximum. Missing preferences continue to inherit. Values
+are also clamped to `RetentionPolicy::LIMITS`.
+
+Admin storage diagnostics cover the entire system, including organizations the
+administrator does not belong to. System cleanup queues work for every
+organization plus orphaned records, with each organization's effective policy
+applied independently. Storage sizes include table data and indexes; database row
+estimates are labeled. Cleanup is asynchronous, so refreshing usage does not mean
+all queued work has finished.
+
+Manual system cleanup requires an asynchronous queue connection and a worker for
+`telematics.telemetry.ingestion_queue`. A synchronous or disabled queue is rejected
+instead of running every organization's cleanup inside the HTTP request.
+
+Deploy the frontend and backend together and reload Octane and queue workers after
+updating the backend. In linked development checkouts, Octane's watch paths must
+include `packages/fleetops/server`; watching only the host API directory leaves
+previously loaded controllers and policies running.
+
+Rules the sweep follows:
+
+- Hard deletes only. The models soft-delete, so `delete()` would grow the table;
+  rows already soft-deleted are purged regardless of age.
+- Bounded work: at most `--max-batches` (default 50) statements of `--batch-size`
+  (default 1000) rows per table per company per run. A capped run is normal; the
+  next tick continues. Operators can catch up faster off-peak with
+  `php artisan fleetops:prune-telematics-data --max-batches=500 --no-lock`.
+- Compaction keeps the event (type, severity, location, normalized `data`) and
+  nulls `payload` and `meta`, which hold the bulk of each row.
+- `telematic_deliveries` and `telematic_sync_runs` carry no company; they are
+  resolved through the company's connections (including trashed ones). In-flight
+  sync runs (`fetching`, `ingesting`) are recovered by the drain, never expired.
+- Rows whose company or connection no longer exists are swept with the system
+  defaults.
+- Migrations add `(company_uuid, created_at)` on `device_events` and `positions`
+  and `(telematic_uuid, updated_at)` on `telematic_sync_runs`; on multi-GB tables
+  run them in a maintenance window.
+- Deleting rows frees space inside InnoDB files, not on disk. Run
+  `OPTIMIZE TABLE device_events` (requires `innodb_file_per_table`) off-peak to
+  return it to the operating system.
+
+Two write-amplification fixes accompany the sweep: `device_events.meta` stores only
+the normalized block (the raw unit lives once, in `payload`), and telemetry-driven
+saves run inside `activity()->withoutLogs()` unless the system administrator enables
+"Log telemetry activity". `DrainTelematicInbox` now only recovers deliveries and
+finishes interrupted runs.
+
 ## Adding an adapter
 
 Implement `TelemetryProviderInterface` alongside the existing provider contract.

package/extension.json

@@ -1,6 +1,6 @@
 {
     "name": "Fleet-Ops",
-    "version": "0.6.69",
+    "version": "0.6.70",
     "description": "Fleet & Transport Management Extension for Fleetbase",
     "repository": "https://github.com/fleetbase/fleetops",
     "license": "AGPL-3.0-or-later",

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@fleetbase/fleetops-engine",
-    "version": "0.6.69",
+    "version": "0.6.70",
     "description": "Fleet & Transport Management Extension for Fleetbase",
     "fleetbase": {
         "route": "fleet-ops"

package/server/config/telemetry.php

@@ -13,8 +13,20 @@ return [
     'batch_size'                => env('TELEMATICS_BATCH_SIZE', 100),
     'max_payload_bytes'         => 2097152,
     'max_pending_deliveries'    => 10000,
+    // Administrators control infrastructure retention and activity logging.
+    // Companies may choose event/position history retention within the maximums.
+    // A maximum of 0 imposes no cap and permits unlimited history (retention 0).
     'processed_retention_hours' => 24,
     'quarantine_retention_days' => 7,
+    'sync_run_retention_days'   => 7,
+    'event_retention_days'      => 30,
+    'event_compact_after_days'  => 7,
+    'position_retention_days'   => 90,
+    'max_event_retention_days'  => 0,
+    'max_position_retention_days' => 0,
+    'log_telemetry_activity'    => false,
+    'prune_batch_size'          => 1000,
+    'prune_max_batches'         => 50,
     'stale_engine_on_seconds'   => 120,
     'stale_engine_off_seconds'  => 600,
 ];

package/server/migrations/2026_09_23_000001_add_retention_index_to_device_events.php

@@ -0,0 +1,40 @@
+<?php
+
+use Fleetbase\FleetOps\Support\Database\TableIndexes;
+use Illuminate\Database\Migrations\Migration;
+use Illuminate\Database\Schema\Blueprint;
+use Illuminate\Support\Facades\Schema;
+
+/**
+ * Supports the bounded retention sweep in fleetops:prune-telematics-data,
+ * which ranges over these columns for every company on every run.
+ */
+return new class extends Migration {
+    private const TABLE   = 'device_events';
+    private const INDEX   = 'device_events_company_created_at_index';
+    private const COLUMNS = ['company_uuid', 'created_at'];
+
+    public function up(): void
+    {
+        if (!Schema::hasTable(self::TABLE) || !Schema::hasColumns(self::TABLE, self::COLUMNS)) {
+            return;
+        }
+
+        if (TableIndexes::covers(self::TABLE, self::COLUMNS)) {
+            return;
+        }
+
+        Schema::table(self::TABLE, fn (Blueprint $table) => $table->index(self::COLUMNS, self::INDEX));
+    }
+
+    public function down(): void
+    {
+        if (!Schema::hasTable(self::TABLE)) {
+            return;
+        }
+
+        if ((TableIndexes::for(self::TABLE)[self::INDEX] ?? null) === self::COLUMNS) {
+            Schema::table(self::TABLE, fn (Blueprint $table) => $table->dropIndex(self::INDEX));
+        }
+    }
+};

package/server/migrations/2026_09_23_000002_add_retention_index_to_positions.php

@@ -0,0 +1,40 @@
+<?php
+
+use Fleetbase\FleetOps\Support\Database\TableIndexes;
+use Illuminate\Database\Migrations\Migration;
+use Illuminate\Database\Schema\Blueprint;
+use Illuminate\Support\Facades\Schema;
+
+/**
+ * Supports the bounded retention sweep in fleetops:prune-telematics-data,
+ * which ranges over these columns for every company on every run.
+ */
+return new class extends Migration {
+    private const TABLE   = 'positions';
+    private const INDEX   = 'positions_company_created_at_index';
+    private const COLUMNS = ['company_uuid', 'created_at'];
+
+    public function up(): void
+    {
+        if (!Schema::hasTable(self::TABLE) || !Schema::hasColumns(self::TABLE, self::COLUMNS)) {
+            return;
+        }
+
+        if (TableIndexes::covers(self::TABLE, self::COLUMNS)) {
+            return;
+        }
+
+        Schema::table(self::TABLE, fn (Blueprint $table) => $table->index(self::COLUMNS, self::INDEX));
+    }
+
+    public function down(): void
+    {
+        if (!Schema::hasTable(self::TABLE)) {
+            return;
+        }
+
+        if ((TableIndexes::for(self::TABLE)[self::INDEX] ?? null) === self::COLUMNS) {
+            Schema::table(self::TABLE, fn (Blueprint $table) => $table->dropIndex(self::INDEX));
+        }
+    }
+};

package/server/migrations/2026_09_23_000003_add_retention_index_to_telematic_sync_runs.php

@@ -0,0 +1,40 @@
+<?php
+
+use Fleetbase\FleetOps\Support\Database\TableIndexes;
+use Illuminate\Database\Migrations\Migration;
+use Illuminate\Database\Schema\Blueprint;
+use Illuminate\Support\Facades\Schema;
+
+/**
+ * Supports the bounded retention sweep in fleetops:prune-telematics-data,
+ * which ranges over these columns for every company on every run.
+ */
+return new class extends Migration {
+    private const TABLE   = 'telematic_sync_runs';
+    private const INDEX   = 'telematic_sync_runs_telematic_updated_at_index';
+    private const COLUMNS = ['telematic_uuid', 'updated_at'];
+
+    public function up(): void
+    {
+        if (!Schema::hasTable(self::TABLE) || !Schema::hasColumns(self::TABLE, self::COLUMNS)) {
+            return;
+        }
+
+        if (TableIndexes::covers(self::TABLE, self::COLUMNS)) {
+            return;
+        }
+
+        Schema::table(self::TABLE, fn (Blueprint $table) => $table->index(self::COLUMNS, self::INDEX));
+    }
+
+    public function down(): void
+    {
+        if (!Schema::hasTable(self::TABLE)) {
+            return;
+        }
+
+        if ((TableIndexes::for(self::TABLE)[self::INDEX] ?? null) === self::COLUMNS) {
+            Schema::table(self::TABLE, fn (Blueprint $table) => $table->dropIndex(self::INDEX));
+        }
+    }
+};

package/server/src/Auth/Schemas/FleetOps.php

@@ -43,7 +43,7 @@ class FleetOps
         ],
         [
             'name'    => 'service-rate',
-            'actions' => ['import'],
+            'actions' => ['import', 'export'],
         ],
         [
             'name'    => 'zone',
@@ -141,6 +141,51 @@ class FleetOps
             'name'    => 'part',
             'actions' => ['export', 'import'],
         ],
+        [
+            'name'    => 'maintenance-schedule',
+            'actions' => ['export', 'import'],
+        ],
+        [
+            'name'    => 'warranty',
+            'actions' => [],
+        ],
+        [
+            'name'    => 'device',
+            'actions' => ['export', 'import'],
+        ],
+        [
+            'name'    => 'sensor',
+            'actions' => ['export', 'import'],
+        ],
+        [
+            'name'    => 'device-event',
+            'actions' => ['export'],
+        ],
+        [
+            'name'    => 'telematic',
+            'actions' => ['export', 'import'],
+        ],
+        [
+            'name'    => 'fuel-provider-connection',
+            'actions' => ['sync'],
+        ],
+        [
+            'name'    => 'fuel-provider-transaction',
+            'actions' => ['review'],
+        ],
+        [
+            'name'    => 'fuel-provider-sync-run',
+            'actions' => [],
+        ],
+        [
+            'name'    => 'purchase-rate',
+            'actions' => [],
+        ],
+        [
+            'name'           => 'analytics', // dashboards, KPI metrics and the live fleet summary
+            'actions'        => [],
+            'remove_actions' => ['create', 'update', 'delete', 'export'],
+        ],
         [
             'name'    => 'custom-field',
             'actions' => [],
@@ -151,26 +196,41 @@ class FleetOps
         ],
         [
             'name'           => 'navigator-settings', // the navigator mobile app used by drivers
-            'action'         => [],
+            'actions'        => [],
             'remove_actions' => ['delete', 'export', 'list', 'create'],
         ],
         [
             'name'           => 'payments',
-            'action'         => ['onboard'],
+            'actions'        => ['onboard'],
             'remove_actions' => ['delete', 'export', 'list', 'create'],
         ],
         [
             'name'           => 'notification-settings',
-            'action'         => [],
+            'actions'        => [],
             'remove_actions' => ['export', 'create'],
         ],
         [
             'name'           => 'routing-settings',
-            'action'         => [],
+            'actions'        => [],
             'remove_actions' => ['export', 'create'],
         ],
         [
             'name'           => 'map-settings',
+            'actions'        => [],
+            'remove_actions' => ['export', 'create'],
+        ],
+        [
+            'name'           => 'scheduling-settings',
+            'actions'        => [],
+            'remove_actions' => ['delete', 'export', 'create'],
+        ],
+        [
+            'name'           => 'tracking-settings',
+            'actions'        => [],
+            'remove_actions' => ['delete', 'export', 'create'],
+        ],
+        [
+            'name'           => 'telematics-settings', // organization device event and position history preferences
             'action'         => [],
             'remove_actions' => ['export', 'create'],
         ],
@@ -187,6 +247,9 @@ class FleetOps
                 'see extension',
                 '* order',
                 '* route',
+                'see analytics',
+                'list analytics',
+                'view analytics',
                 'see order-config',
                 'list order-config',
                 'view order-config',
@@ -230,6 +293,12 @@ class FleetOps
                 '* driver',
                 '* vehicle',
                 '* trailer',
+                '* device',
+                '* sensor',
+                '* telematic',
+                'see device-event',
+                'list device-event',
+                'view device-event',
             ],
         ],
         [
@@ -277,6 +346,7 @@ class FleetOps
             'permissions' => [
                 'see extension',
                 '* service-rate',
+                '* purchase-rate',
                 'see order',
                 'list order',
                 'see service-area',
@@ -309,6 +379,8 @@ class FleetOps
                 '* inspection-submission',
                 '* equipment',
                 '* part',
+                '* maintenance-schedule',
+                '* warranty',
                 '* trailer',
                 'see vehicle',
                 'list vehicle',
@@ -336,6 +408,18 @@ class FleetOps
                 '* inspection-submission',
                 '* equipment',
                 '* part',
+                '* maintenance-schedule',
+                '* warranty',
+                '* device',
+                '* sensor',
+                '* device-event',
+                '* telematic',
+                '* fuel-provider-connection',
+                '* fuel-provider-transaction',
+                '* fuel-provider-sync-run',
+                'see analytics',
+                'list analytics',
+                'view analytics',
                 '* trailer',
             ],
         ],
@@ -353,6 +437,14 @@ class FleetOps
                 'list vehicle',
             ],
         ],
+        [
+            'name'        => 'TelematicsSettingsManager',
+            'description' => 'Policy for managing organization telematics history preferences.',
+            'permissions' => [
+                'see extension',
+                '* telematics-settings',
+            ],
+        ],
         [
             'name'        => 'DriverOperations',
             'description' => 'Policy for drivers to manage their assigned tasks and access necessary information.',
@@ -453,6 +545,13 @@ class FleetOps
                 'NavigatorSettingsManager',
             ],
         ],
+        [
+            'name'        => 'Telematics Settings Manager',
+            'description' => 'Role responsible for managing organization telematics history preferences.',
+            'policies'    => [
+                'TelematicsSettingsManager',
+            ],
+        ],
         [
             'name'        => 'Driver',
             'description' => 'Role for drivers with the necessary access to manage their daily tasks, including order management, and vehicle assignment.',

package/server/src/Console/Commands/DrainTelematicInbox.php

@@ -16,7 +16,7 @@ use Illuminate\Support\Facades\Log;
 class DrainTelematicInbox extends Command
 {
     protected $signature   = 'fleetops:drain-telematic-inbox';
-    protected $description = 'Recover pending telematic deliveries and expire retained payloads.';
+    protected $description = 'Recover pending telematic deliveries and finish interrupted sync runs.';
 
     public function handle(): int
     {
@@ -58,24 +58,14 @@ class DrainTelematicInbox extends Command
             try {
                 Queue::dispatch((new ProcessTelematicDelivery($row->uuid))->onQueue($optionsByConnection[$row->telematic_uuid]['ingestion_queue'] ?? 'default'));
             } catch (\Throwable) {
-                // A broker outage must not prevent retention cleanup; retry the next minute.
+                // A broker outage must not prevent run recovery; retry the next minute.
                 break;
             }
         }
-        // Bound deletes so retention cleanup cannot monopolize the inbox table.
-        foreach (['processed' => now()->subHours(config('telematics.telemetry.processed_retention_hours', 24)), 'quarantined' => now()->subDays(config('telematics.telemetry.quarantine_retention_days', 7))] as $status => $cutoff) {
-            for ($batch = 0; $batch < 20; $batch++) {
-                $ids = DB::table('telematic_deliveries')->where('status', $status)->where('updated_at', '<', $cutoff)->limit(1000)->pluck('uuid');
-                if ($ids->isEmpty()) {
-                    break;
-                }
-                DB::table('telematic_deliveries')->whereIn('uuid', $ids)->delete();
-            }
-        }
+        // Retention for deliveries and sync runs is applied by fleetops:prune-telematics-data.
         DB::table('telematic_sync_runs')->where('status', 'fetching')->where('updated_at', '<', now()->subMinutes(5))
             ->update(['status' => 'incomplete', 'error' => 'Worker interrupted; next scheduled sweep will recover.']);
         DB::table('telematic_sync_runs')->where('status', 'ingesting')->get(['uuid'])->each(fn ($run) => Inbox::finishRun($run->uuid));
-        DB::table('telematic_sync_runs')->where('updated_at', '<', now()->subDays(7))->delete();
 
         return self::SUCCESS;
     }

package/server/src/Console/Commands/ProcessOperationalAlerts.php

@@ -60,7 +60,7 @@ class ProcessOperationalAlerts extends Command
 
                 foreach ($orders as $order) {
                     session(['company' => $order->company_uuid]);
-                    $settings = $this->alertSettings();
+                    $settings = $this->alertSettings($order);
 
                     if ($this->processLateDeparture($order, $settings, $dryRun)) {
                         $triggered++;
@@ -256,9 +256,9 @@ class ProcessOperationalAlerts extends Command
             ->min() ?? 0;
     }
 
-    protected function alertSettings(): array
+    protected function alertSettings(Order $order): array
     {
-        $settings = Setting::lookupCompany('tracking', []);
+        $settings = Setting::lookupForCompany($order->company_uuid, 'tracking', []);
         $alerts   = data_get($settings, 'alerts', []);
 
         return [