@fleetbase/fleetops-engine 0.6.66 → 0.6.67

package/docs/TELEMATICS_QUEUES.md

@@ -0,0 +1,179 @@
+# Dedicated telematics queue workers
+
+By default, all telematics work runs on the `default` queue alongside every other
+background job. On an instance with a large fleet, that work can crowd out
+notifications, order events, and other jobs. This guide moves telematics work to
+its own queues so dedicated worker containers process it while the existing
+worker keeps serving `default`.
+
+This is an **opt-in, per-instance** setup. With no environment variables set,
+nothing changes: every job and broadcast stays on the same queue as before.
+
+## What each setting routes
+
+| Setting | Default | Work routed |
+| --- | --- | --- |
+| `TELEMATICS_POLL_QUEUE` | `default` | Scheduled and manual polls (`PollTelematicTelemetry`), legacy device discovery (`SyncTelematicDevicesJob`), and queued connection tests (`TestTelematicConnectionJob`). |
+| `TELEMATICS_INGESTION_QUEUE` | `default` | Applying polled or pushed positions (`ProcessTelematicDelivery`), including jobs re-queued by `fleetops:drain-telematic-inbox` and manual replays. |
+| `TELEMATICS_BROADCAST_QUEUE` | unset | Live-map broadcasts created by telematics ingestion: `DeviceTelemetryUpdated`, and `VehicleLocationChanged` / `TrailerLocationChanged` when a device is attached. When unset, broadcasts use the queue connection's default queue. |
+
+`TELEMATICS_BROADCAST_QUEUE` affects only broadcasts that telematics ingestion
+creates. Vehicle and trailer location broadcasts from the driver app and public API
+remain on the default queue.
+
+Broadcasts can outnumber sync jobs. Each ingested unit with a newer position or
+contact time queues one device broadcast, plus one vehicle or trailer broadcast
+when attached. A fleet that reports every minute can queue hundreds of broadcast
+jobs per minute, so route broadcasts as well as polling and ingestion.
+
+## Before you start: check which sync path the provider uses
+
+Separate workers keep telematics from blocking other jobs. They do not make a slow
+sync faster, and they do not fix an unsafe job.
+
+- **Bounded polling path** (Safee scheduled and manual syncs; AFAQY scheduled polls
+  with `AFAQY_POLLING_ENABLED=true`): poll attempts are limited to an 80-second
+  hard timeout and a 60-second sweep budget, and ingestion yields every 40 seconds. These jobs are safe for the stock Redis
+  `retry_after` of 90 seconds and for multiple concurrent workers.
+- **Legacy discovery path** (other providers, AFAQY manual syncs, and AFAQY scheduled
+  syncs while polling is disabled):
+  `SyncTelematicDevicesJob` allows a 3,600-second run while Redis `retry_after`
+  is 90 seconds. A run lasting over 90 seconds can be handed to another worker and
+  end with `SyncTelematicDevicesJob has been attempted too many times`. Adding
+  workers increases the likelihood that another worker picks it up. Where possible,
+  move the provider to the bounded polling path, for example by setting
+  `AFAQY_POLLING_ENABLED=true` for AFAQY connections.
+
+## Setup
+
+Start the workers **before** setting the queue variables. Jobs sent to a queue with
+no worker wait indefinitely.
+
+### 1. Add telematics workers
+
+Add worker services with the same image, environment, and volumes as the existing
+`queue` service. Change only the command and healthcheck. Docker Compose example:
+
+```yaml
+services:
+  telematics-queue:
+    image: fleetbase/fleetbase-api:latest
+    command: ["php", "artisan", "queue:work", "--queue=telematics", "--sleep=1", "--max-time=3600"]
+    restart: unless-stopped
+    deploy:
+      replicas: 2
+    # environment, volumes and depends_on: copy from the existing `queue` service
+
+  telematics-broadcast-queue:
+    image: fleetbase/fleetbase-api:latest
+    command: ["php", "artisan", "queue:work", "--queue=telematics-broadcasts", "--sleep=1", "--max-time=3600"]
+    restart: unless-stopped
+    # environment, volumes and depends_on: copy from the existing `queue` service
+```
+
+Notes:
+
+- Leave the existing `queue` service unchanged. `queue:work` without `--queue`
+  processes the connection's default queue (`REDIS_QUEUE`, normally `default`).
+- Leave `--timeout` at its default (60 seconds) or set any value below the Redis
+  connection's `retry_after` (90 seconds). Polls (80 seconds) and ingestion
+  (60 seconds) define their own timeouts, which override the worker setting.
+  Broadcast jobs use the worker value.
+- `--max-time` recycles long-running workers so they pick up deployments and release
+  memory. `restart: unless-stopped` starts them again.
+- On Kubernetes, ECS, or another platform, create equivalent deployments with the
+  same image, environment, and commands.
+- Every worker container needs the same `APP_KEY`, database, Redis, and broadcasting
+  settings as the application. Deliveries are encrypted with `APP_KEY`.
+
+### 2. Set the queue variables for this instance only
+
+Add these to the environment of the application, scheduler, **and every queue
+worker** container. The scheduler dispatches polls and the application dispatches
+manual syncs, so every process must agree on the queue names:
+
+```dotenv
+TELEMATICS_POLL_QUEUE=telematics
+TELEMATICS_INGESTION_QUEUE=telematics
+TELEMATICS_BROADCAST_QUEUE=telematics-broadcasts
+```
+
+To use one worker type for all telematics work, set all three to `telematics` and
+omit the broadcast worker.
+
+AFAQY polls and ingestion also read `AFAQY_POLL_QUEUE` and `AFAQY_INGESTION_QUEUE`.
+Those take precedence over the `TELEMATICS_*` values when set, so remove or align
+them. Legacy discovery and connection-test jobs always use `TELEMATICS_POLL_QUEUE`.
+
+### 3. Deploy the change
+
+1. Deploy the new worker containers and confirm they are running.
+2. Apply the environment variables to the application, scheduler, and workers.
+3. If configuration is cached, rebuild it with `php artisan config:cache` in each
+   container, or through the normal image/deploy process.
+4. Restart long-running processes so they read the new configuration:
+   `php artisan queue:restart`, then reload Octane (`php artisan octane:reload`)
+   if the application runs under Octane.
+
+### 4. Existing backlog on `default`
+
+Jobs already on `default` stay there, and the existing worker must process them.
+The new setting affects only newly dispatched jobs.
+
+- Do not run `php artisan queue:clear` on `default`. It deletes every pending job,
+  including non-telematics work.
+- For opted-in batch providers such as Safee, a queued `SyncTelematicDevicesJob`
+  hands off to bounded polling and exits within seconds.
+- Queued legacy discovery jobs for other providers can each occupy the default
+  worker for a long time. If the backlog does not drain, inspect those specific
+  jobs rather than clearing the entire queue.
+
+## Sizing
+
+Start with **2 telematics workers and 1 broadcast worker**, then adjust based on
+queue depth.
+
+Current work per connection:
+
+- One poll per connection per minute, coalesced while its previous sweep is still
+  ingesting. Safee requests at most 1,000 vehicle IDs per `last-state` call.
+- Up to `ceil(units / TELEMATICS_BATCH_SIZE)` ingestion jobs per sweep (default
+  100 units per job). A fleet of about 470 reporting devices is about 5 ingestion
+  jobs per minute.
+- Workers can process different connections, deliveries, and devices concurrently.
+  Per-connection and per-device locks prevent duplicate polling or out-of-order
+  position updates.
+
+For reference, a local single-worker run for 93 Safee units applied each delivery in
+about 5-8 seconds (about 55-85 ms per unit), including database work. This was a
+development machine, not production. Measure production queue depth rather than
+extrapolating this value.
+
+Additional workers do not increase provider limits. Safee connections share a limit
+of 50 requests per second per account across all workers.
+
+## Verification
+
+Queue depth should stay close to zero between minute ticks:
+
+```bash
+php artisan queue:monitor redis:default,redis:telematics,redis:telematics-broadcasts --max=500
+```
+
+Also verify:
+
+- `telematic_sync_runs` for the connection shows a new `completed` run about once
+  per minute. `incomplete` means the fetch failed and will retry.
+- The worker logs show `PollTelematicTelemetry` and `ProcessTelematicDelivery` only
+  on the telematics workers, and broadcast jobs only on the broadcast worker.
+- Other jobs on `default`, such as notifications and order events, start promptly
+  again.
+
+## Rollback
+
+1. Remove the three variables from the application, scheduler, and workers, rebuild
+   cached configuration, and restart processes as in setup step 3.
+2. Keep the telematics workers running until `telematics` and
+   `telematics-broadcasts` are empty. Jobs remaining on those queues are not moved
+   back to `default`.
+3. Remove the worker containers.

package/extension.json

@@ -1,6 +1,6 @@
 {
     "name": "Fleet-Ops",
-    "version": "0.6.66",
+    "version": "0.6.67",
     "description": "Fleet & Transport Management Extension for Fleetbase",
     "repository": "https://github.com/fleetbase/fleetops",
     "license": "AGPL-3.0-or-later",

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@fleetbase/fleetops-engine",
-    "version": "0.6.66",
+    "version": "0.6.67",
     "description": "Fleet & Transport Management Extension for Fleetbase",
     "fleetbase": {
         "route": "fleet-ops"

package/server/config/safee.php

@@ -0,0 +1,10 @@
+<?php
+
+return [
+    'polling_enabled'            => env('SAFEE_POLLING_ENABLED', true),
+    'webhooks_enabled'           => false,
+    'manual_batch_sync'          => true,
+    'page_size'                  => 1000,
+    'inventory_cache_seconds'    => env('SAFEE_INVENTORY_CACHE_SECONDS', 300),
+    'request_timeout_seconds'    => env('SAFEE_REQUEST_TIMEOUT_SECONDS', 30),
+];

package/server/config/telematics.php

@@ -176,6 +176,7 @@ return [
             'driver_class'    => Fleetbase\FleetOps\Support\Telematics\Providers\SafeeProvider::class,
             'icon'            => '/engines-dist/images/telematics/providers/safee.webp',
             'description'     => 'Safee Tracking REST integration for vehicle discovery, last state, live positions, odometer, fuel, and sensor data.',
+            'docs_url'        => 'https://www.safee.com/_docs_/Safee_Tracking_REST_Service.pdf',
             'required_fields' => [
                 [
                     'name'          => 'server_uri',
@@ -229,7 +230,13 @@ return [
             'supports_discovery' => true,
             'metadata'           => [
                 'rate_limit'   => 3000,
-                'pagination'   => 'none',
+                'rate_limit_per_second' => 50,
+                'pagination'   => 'inventory_cursor',
+                'telemetry'    => [
+                    'durable_ingestion' => true,
+                    'secure_webhooks'   => false,
+                    'reconciliation'    => true,
+                ],
                 'capabilities' => ['tracking', 'speed', 'heading', 'odometer', 'fuel_level', 'ignition'],
             ],
         ],

package/server/config/telemetry.php

@@ -5,9 +5,12 @@ return [
     'webhooks_enabled'          => false,
     'poll_queue'                => env('TELEMATICS_POLL_QUEUE', 'default'),
     'ingestion_queue'           => env('TELEMATICS_INGESTION_QUEUE', 'default'),
+    // Null keeps telemetry broadcasts on the connection's default queue.
+    'broadcast_queue'           => env('TELEMATICS_BROADCAST_QUEUE'),
     'page_size'                 => 1000,
     'max_pages'                 => 100,
-    'batch_size'                => 100,
+    'request_timeout_seconds'   => env('TELEMATICS_REQUEST_TIMEOUT_SECONDS', 45),
+    'batch_size'                => env('TELEMATICS_BATCH_SIZE', 100),
     'max_payload_bytes'         => 2097152,
     'max_pending_deliveries'    => 10000,
     'processed_retention_hours' => 24,

package/server/migrations/2026_09_17_000001_add_device_event_uuid_lookup_index.php

@@ -0,0 +1,90 @@
+<?php
+
+use Illuminate\Database\Migrations\Migration;
+use Illuminate\Database\Schema\Blueprint;
+use Illuminate\Support\Facades\Schema;
+
+return new class extends Migration {
+    private const INDEX = 'device_events_uuid_lookup_index';
+
+    public function up(): void
+    {
+        if (!Schema::hasTable('device_events') || !Schema::hasColumn('device_events', 'uuid')) {
+            return;
+        }
+
+        $indexes = $this->indexes();
+        foreach ($indexes as $index) {
+            // A primary, unique, or composite index with uuid first already
+            // supports Eloquent's uuid lookup, regardless of the index name.
+            if ($index['usable'] && ($index['columns'][0] ?? null) === 'uuid') {
+                return;
+            }
+        }
+
+        if (isset($indexes[self::INDEX])) {
+            throw new RuntimeException('The device event UUID lookup index name is already used by another index.');
+        }
+
+        // The legacy table uses an integer primary key, while DeviceEvent uses
+        // uuid. In particular, activity logging refreshes every saved event by
+        // uuid. Keep this non-unique to preserve legacy nullable/duplicate data.
+        Schema::table('device_events', fn (Blueprint $table) => $table->index('uuid', self::INDEX));
+    }
+
+    public function down(): void
+    {
+        if (!Schema::hasTable('device_events')) {
+            return;
+        }
+
+        $index = $this->indexes()[self::INDEX] ?? null;
+        if ($index && $index['columns'] === ['uuid']) {
+            Schema::table('device_events', fn (Blueprint $table) => $table->dropIndex(self::INDEX));
+        }
+    }
+
+    /**
+     * Inspect indexes without relying on Schema::getIndexes, which is absent
+     * in Laravel 9 and early Laravel 10. Fleet-Ops uses MySQL in deployments
+     * and SQLite for its migration tests.
+     *
+     * @return array<string, array{columns: array, usable: bool}>
+     */
+    private function indexes(): array
+    {
+        $connection = Schema::getConnection();
+        $table      = $connection->getTablePrefix() . 'device_events';
+        $indexes    = [];
+
+        if ($connection->getDriverName() === 'sqlite') {
+            $table = str_replace('"', '""', $table);
+            foreach ($connection->select('PRAGMA index_list("' . $table . '")') as $index) {
+                $name    = str_replace('"', '""', $index->name);
+                $columns = $connection->select('PRAGMA index_info("' . $name . '")');
+                usort($columns, fn ($a, $b) => $a->seqno <=> $b->seqno);
+                $indexes[$index->name] = ['columns' => array_column($columns, 'name'), 'usable' => !($index->partial ?? false)];
+            }
+
+            return $indexes;
+        }
+
+        if ($connection->getDriverName() !== 'mysql') {
+            throw new RuntimeException('Device event UUID index inspection requires MySQL or SQLite.');
+        }
+
+        $table = str_replace('`', '``', $table);
+        foreach ($connection->select('SHOW INDEX FROM `' . $table . '`') as $index) {
+            $indexes[$index->Key_name]['columns'][(int) $index->Seq_in_index - 1] = $index->Column_name;
+            $indexes[$index->Key_name]['usable']                                  = in_array(strtoupper($index->Index_type), ['BTREE', 'HASH'], true)
+                && strtoupper($index->Visible ?? 'YES') !== 'NO'
+                && strtoupper($index->Ignored ?? 'NO') !== 'YES';
+        }
+        foreach ($indexes as &$index) {
+            ksort($index['columns']);
+            $index['columns'] = array_values($index['columns']);
+        }
+
+        return $indexes;
+    }
+};

package/server/src/Console/Commands/SyncTelematics.php

@@ -24,7 +24,7 @@ class SyncTelematics extends Command
         $lock    = null;
 
         if ($useLock) {
-            $lock = Cache::lock('fleetops:sync-telematics', 600);
+            $lock = Cache::lock('fleetops:sync-telematics', 120);
             if (!$lock->get()) {
                 $this->warn('Another telematics sync run appears to be in progress.');
 
@@ -40,6 +40,7 @@ class SyncTelematics extends Command
                 return self::SUCCESS;
             }
 
+            $queued             = 0;
             $telemetryProviders = [];
             foreach ($providerKeys as $key) {
                 if (!data_get($registry->findByKey($key)?->metadata, 'telemetry.durable_ingestion', false)) {
@@ -50,18 +51,24 @@ class SyncTelematics extends Command
                     continue;
                 }
                 $options = \Fleetbase\FleetOps\Support\Telematics\Telemetry\Configuration::options($provider);
+                // Batch-only providers have no legacy fallback: pausing polling pauses their sync.
                 if (!($options['polling_enabled'] ?? false)) {
+                    if ($options['manual_batch_sync'] ?? false) {
+                        $telemetryProviders[] = $key;
+                    }
                     continue;
                 }
                 $telemetryProviders[] = $key;
                 Telematic::withoutGlobalScopes()->where('provider', $key)->whereIn('status', ['active', 'connected', 'error', 'synchronizing'])
-                    ->whereNotNull('company_uuid')->orderBy('id')->chunkById(100, function ($connections) use ($options) {
+                    ->whereNotNull('company_uuid')->orderBy('id')->chunkById(100, function ($connections) use ($options, &$queued) {
                         foreach ($connections as $connection) {
                             if (\Fleetbase\FleetOps\Support\Telematics\Telemetry\Inbox::enabled($connection)) {
                                 try {
-                                    \Fleetbase\FleetOps\Support\Telematics\Telemetry\Queue::dispatch((new \Fleetbase\FleetOps\Jobs\PollTelematicTelemetry($connection->uuid))
+                                    if (\Fleetbase\FleetOps\Support\Telematics\Telemetry\Queue::dispatch((new \Fleetbase\FleetOps\Jobs\PollTelematicTelemetry($connection->uuid))
                                         ->onQueue($options['poll_queue'] ?? 'default')
-                                        ->delay(now()->addSeconds(abs(crc32($connection->uuid)) % 10)));
+                                        ->delay(now()->addSeconds(abs(crc32($connection->uuid)) % 10)))) {
+                                        $queued++;
+                                    }
                                 } catch (\Throwable) {
                                     \Illuminate\Support\Facades\Log::warning('Telemetry polling dispatch failed; next tick will retry.', ['telematic_uuid' => $connection->uuid]);
                                 }
@@ -75,7 +82,6 @@ class SyncTelematics extends Command
                 ->whereIn('status', ['active', 'connected'])
                 ->whereNotNull('company_uuid');
 
-            $queued = 0;
             $query->orderBy('id')->chunkById(100, function ($telematics) use (&$queued) {
                 foreach ($telematics as $telematic) {
                     SyncTelematicDevicesJob::dispatch($telematic, [

package/server/src/Events/DeviceTelemetryUpdated.php

@@ -8,7 +8,8 @@ use Illuminate\Contracts\Broadcasting\ShouldBroadcast;
 
 class DeviceTelemetryUpdated implements ShouldBroadcast
 {
-    public bool $afterCommit = true;
+    public bool $afterCommit       = true;
+    public ?string $broadcastQueue = null;
     public string $companyUuid;
     public string $deviceUuid;
     public array $data;

package/server/src/Events/TrailerLocationChanged.php

@@ -25,6 +25,8 @@ class TrailerLocationChanged implements ShouldBroadcast
     public mixed $heading;
     public mixed $speed;
     public array $additionalData;
+    // Optional queue for telematics-originated broadcasts. Null keeps Laravel's default.
+    public ?string $broadcastQueue = null;
 
     public function __construct(Trailer $trailer, array $additionalData = [])
     {

package/server/src/Events/VehicleLocationChanged.php

@@ -25,6 +25,11 @@ class VehicleLocationChanged implements ShouldBroadcast
     public ?string $companyUuid;
     public ?string $apiCredentialUuid;
 
+    /**
+     * Optional queue for telematics-originated broadcasts. Null keeps Laravel's default.
+     */
+    public ?string $broadcastQueue = null;
+
     /**
      * The datetime instance the broadcast ws triggered.
      *

package/server/src/Jobs/PollTelematicTelemetry.php

@@ -23,12 +23,21 @@ class PollTelematicTelemetry implements ShouldQueue, ShouldBeUnique
     use InteractsWithQueue;
     use Queueable;
 
-    public int $tries     = 5;
-    public int $timeout   = 120;
-    public int $uniqueFor = 3600;
+    // Reservations released while earlier batches drain are not provider failures.
+    public int $tries         = 0;
+    public int $maxExceptions = 5;
+    // Keep every attempt below Laravel's baseline Redis retry_after of 90 seconds.
+    public int $timeout          = 80;
+    public int $uniqueFor        = 3600;
+    public ?string $manualJobId  = null;
+    public array $requestOptions = [];
+    public ?int $retryDeadline   = null;
 
-    public function __construct(public string $telematicUuid)
+    public function __construct(public string $telematicUuid, ?string $manualJobId = null, array $requestOptions = [])
     {
+        $this->manualJobId    = $manualJobId;
+        $this->requestOptions = $requestOptions;
+        $this->retryDeadline  = now()->addMinutes(15)->getTimestamp();
     }
 
     public function uniqueId(): string
@@ -38,11 +47,35 @@ class PollTelematicTelemetry implements ShouldQueue, ShouldBeUnique
 
     public function backoff(): array
     {
-        return [15, 60, 180, 300];
+        // A scheduled retry holds the uniqueness lease that suppresses minute ticks,
+        // so it must not wait longer than the polling interval after a transient failure.
+        return $this->manualJobId ? [15, 60, 180, 300] : [15, 30, 60, 60];
+    }
+
+    public function retryUntil(): \Illuminate\Support\Carbon
+    {
+        if ($this->retryDeadline === null) {
+            // Older serialized jobs have no deadline property. Anchor their first
+            // observed attempt once, rather than extending it on each redelivery.
+            $key                 = 'telemetry:poll:retry-deadline:' . ($this->job?->uuid() ?? $this->manualJobId ?? $this->telematicUuid);
+            $this->retryDeadline = Cache::remember($key, 86400, fn () => now()->addMinutes(15)->getTimestamp());
+        }
+
+        return \Illuminate\Support\Carbon::createFromTimestampUTC($this->retryDeadline);
     }
 
     public function handle(TelematicProviderRegistry $registry, Inbox $inbox): void
     {
+        if (now()->gte($this->retryUntil())) {
+            $error = new \Illuminate\Queue\MaxAttemptsExceededException('Telemetry polling retry deadline exceeded.');
+            if ($this->job) {
+                $this->fail($error);
+            } else {
+                $this->failed($error);
+            }
+
+            return;
+        }
         $telematic = Telematic::withoutGlobalScopes()->where('uuid', $this->telematicUuid)->first();
         if (!$telematic || !Inbox::enabled($telematic)) {
             return;
@@ -57,25 +90,47 @@ class PollTelematicTelemetry implements ShouldQueue, ShouldBeUnique
         }
         // Coalesce ticks while a previous sweep is still being ingested.
         if (DB::table('telematic_deliveries')->where('telematic_uuid', $this->telematicUuid)->where('source', 'poll')->whereIn('status', ['pending', 'retry', 'processing'])->exists()) {
+            if ($this->manualJobId && $this->job) {
+                $this->manualProgress(['last_sync_result' => 'retrying', 'last_sync_phase' => 'waiting_for_ingestion']);
+                $this->release(15);
+            }
+
             return;
         }
-        $lock = Cache::lock('telemetry:poll:' . $this->telematicUuid, 150);
+        $lock = Cache::lock('telemetry:poll:' . $this->telematicUuid, 85);
         if (!$lock->get()) {
+            if ($this->manualJobId && $this->job) {
+                $this->release(15);
+            }
+
             return;
         }
         $run = (string) Str::uuid();
-        DB::table('telematic_sync_runs')->insert(['uuid' => $run, 'telematic_uuid' => $telematic->uuid, 'status' => 'fetching', 'created_at' => now(), 'updated_at' => now()]);
         try {
+            $started = microtime(true);
+            DB::table('telematic_sync_runs')->insert(['uuid' => $run, 'telematic_uuid' => $telematic->uuid, 'status' => 'fetching', 'created_at' => now(), 'updated_at' => now()]);
+            $this->manualProgress([
+                'last_sync_run_uuid'      => $run, 'last_sync_run_job_id' => $this->manualJobId,
+                'last_sync_result'        => 'running', 'last_sync_phase' => 'fetching_inventory',
+                'last_sync_fetched_total' => 0, 'last_sync_page_count' => 0,
+                'last_sync_linked_total'  => 0, 'last_sync_failed_total' => 0, 'last_sync_error' => null,
+            ]);
             $provider->connect($telematic);
             $cursor  = null;
             $cursors = [];
             $seen    = [];
-            $started = microtime(true);
             for ($page = 0; $page < ($options['max_pages'] ?? 100); $page++) {
-                if (microtime(true) - $started > 90) {
-                    throw new \RuntimeException('Polling time budget exceeded; sweep incomplete.');
-                }
-                $response = $provider->fetchDevices(['limit' => $options['page_size'] ?? 1000, 'cursor' => $cursor, 'timeout' => 20, 'connect_timeout' => 5]);
+                $remaining = $this->remainingBudget($started);
+                // Large fleet responses can exceed 20 seconds. Allow a configured
+                // request timeout without extending the bounded sweep deadline.
+                $timeout  = min(max(1, (int) ($options['request_timeout_seconds'] ?? 45)), $remaining);
+                $response = $provider->fetchDevices([
+                    'limit'             => min(max(1, (int) ($this->requestOptions['limit'] ?? $options['page_size'] ?? 1000)), 1000),
+                    'cursor'            => $cursor, 'filters' => $this->requestOptions['filters'] ?? [],
+                    'refresh_inventory' => $this->manualJobId !== null && $cursor === null,
+                    'timeout'           => $timeout, 'connect_timeout' => min(5, $timeout),
+                ]);
+                $this->remainingBudget($started);
                 $received = now()->utc();
                 $units    = $response['devices'];
                 $ids      = array_map(fn ($unit) => $provider->normalizeDevice($unit)['device_id'] ?? null, $units);
@@ -86,11 +141,15 @@ class PollTelematicTelemetry implements ShouldQueue, ShouldBeUnique
                     $seen[(string) $id] = true;
                 }
                 foreach (array_chunk($units, max(1, (int) ($options['batch_size'] ?? 100))) as $batch) {
+                    $this->remainingBudget($started);
                     $inbox->accept($telematic, $batch, 'poll', $run, $received->toDateTimeString());
                 }
+                $this->remainingBudget($started);
                 DB::table('telematic_sync_runs')->where('uuid', $run)->update(['pages' => $page + 1, 'units' => count($seen), 'updated_at' => now()]);
+                $this->manualProgress(['last_sync_fetched_total' => count($seen), 'last_sync_page_count' => $page + 1, 'last_sync_phase' => 'fetching_inventory']);
                 if (!$response['has_more']) {
                     DB::table('telematic_sync_runs')->where('uuid', $run)->update(['status' => 'ingesting', 'updated_at' => now()]);
+                    $this->manualProgress(['last_sync_phase' => 'ingesting', 'last_sync_inventory_total' => count($seen)]);
                     Inbox::finishRun($run);
 
                     return;
@@ -105,6 +164,7 @@ class PollTelematicTelemetry implements ShouldQueue, ShouldBeUnique
             throw new \RuntimeException('Maximum page count reached; sweep incomplete.');
         } catch (\Throwable $e) {
             DB::table('telematic_sync_runs')->where('uuid', $run)->update(['status' => 'incomplete', 'error' => $e instanceof TelematicRateLimitExceededException ? 'Rate limited; retry scheduled.' : 'Polling failed; retry scheduled. ' . class_basename($e), 'updated_at' => now()]);
+            $this->manualProgress(['last_sync_result' => 'retrying', 'last_sync_phase' => 'retrying', 'last_sync_error' => 'Polling failed; retry scheduled. ' . class_basename($e)]);
             if ($e instanceof TelematicRateLimitExceededException && $this->job) {
                 $this->release($e->context()['retry_after'] ?? 60);
 
@@ -115,4 +175,59 @@ class PollTelematicTelemetry implements ShouldQueue, ShouldBeUnique
             $lock->release();
         }
     }
+
+    private function remainingBudget(float $started): int
+    {
+        $remaining = (int) floor(60 - (microtime(true) - $started));
+        if ($remaining < 1) {
+            throw new \RuntimeException('Polling time budget exceeded; sweep incomplete.');
+        }
+
+        return $remaining;
+    }
+
+    private function manualProgress(array $attributes): void
+    {
+        if (!$this->manualJobId) {
+            return;
+        }
+        DB::transaction(function () use ($attributes) {
+            $connection = Telematic::withoutGlobalScopes()->where('uuid', $this->telematicUuid)->lockForUpdate()->first();
+            if (!$connection || !Inbox::enabled($connection) || data_get($connection->meta, 'last_sync_job_id') !== $this->manualJobId) {
+                return;
+            }
+            $connection->status = 'synchronizing';
+            $connection->meta   = array_merge($connection->meta ?? [], $attributes, ['last_sync_progress_at' => now()->toDateTimeString()]);
+            $connection->save();
+        });
+    }
+
+    public function failed(\Throwable $error): void
+    {
+        if (!$this->manualJobId) {
+            return;
+        }
+        $run = null;
+        DB::transaction(function () use ($error, &$run) {
+            $connection = Telematic::withoutGlobalScopes()->where('uuid', $this->telematicUuid)->lockForUpdate()->first();
+            if (!$connection || !Inbox::enabled($connection) || data_get($connection->meta, 'last_sync_job_id') !== $this->manualJobId) {
+                return;
+            }
+            if (data_get($connection->meta, 'last_sync_run_job_id') === $this->manualJobId) {
+                $run = data_get($connection->meta, 'last_sync_run_uuid');
+            }
+            $connection->status = 'error';
+            $connection->meta   = array_merge($connection->meta ?? [], [
+                'last_sync_result'     => 'failed', 'last_sync_phase' => 'failed',
+                'last_sync_error'      => 'Polling retries exhausted. ' . class_basename($error),
+                'last_sync_error_type' => class_basename($error), 'last_sync_failed_at' => now()->toDateTimeString(),
+            ]);
+            $connection->save();
+        });
+        // Do not lock run rows while holding the connection lock: finishRun takes
+        // those locks in the opposite order when the final delivery completes.
+        if ($run) {
+            DB::table('telematic_sync_runs')->where('uuid', $run)->where('status', 'fetching')->update(['status' => 'incomplete', 'error' => 'Polling attempt interrupted.', 'updated_at' => now()]);
+        }
+    }
 }