@fleetbase/fleetops-engine 0.6.66 → 0.6.67

package/RELEASE.md

@@ -1,20 +1,27 @@
-> v0.6.66 ~ "Inspections"
+> v0.6.67 ~ "Telematics reliability"
 
 ---
 ## What's New
-- **Inspection forms are built from typed fields.** A form is groups of fields laid out on a grid: pass/fail checks with severity and on-fail settings, text, numbers, selections, dates, photo uploads and signatures. Forms are built and published from a form builder in the console.
-- **One inspection sheet everywhere.** The same sheet is used to fill in an inspection in the console, to read one back, and on a public link. A failed check opens its severity, unsafe flag, comment and photos in place, and a defects tray summarises what needs attention. It works on phones and tablets.
-- **Drivers file inspections through the API.** The `v1` inspection endpoints list published forms, file an inspection against one, and read back submissions and a vehicle's history, for the Navigator app and other integrations.
-- **Inspection links for anyone in the organisation.** A link can be assigned to any user, protected by a six-digit PIN, and emailed or texted to them. A failed inspection can raise an issue and open a work order, and every submission records who filed it.
+- **Safee / DSCO syncs use a few batched requests.** Each sweep fetches every vehicle's current position, speed, heading and odometer in batches of up to 1,000 vehicles, with the vehicle list cached between sweeps. The old sync made three extra requests per vehicle; a 93-vehicle fleet now uses one request per minute instead of about 281. Temperature, door and history lookups stay out of routine polling.
+- **Safee sign-in is reused and refreshed.** Access tokens are cached encrypted and refreshed before they expire, requests stay within Safee's 50-per-second account limit, and rate-limit responses are honoured.
+- **Telematics can run on dedicated queue workers.** `TELEMATICS_POLL_QUEUE`, `TELEMATICS_INGESTION_QUEUE` and the new `TELEMATICS_BROADCAST_QUEUE` move polling, position processing and live-map broadcasts off the `default` queue for instances that need it. Nothing changes when they are unset. See `docs/TELEMATICS_QUEUES.md`.
+- **Live telemetry status is easier to read.** Connection details show telemetry inside the page layout with a compact grid and collapsible setup and failure sections.
 
 ---
 ## Fixes
-- Place, zone and service area details, and the place and point map modals, no longer fail to render with "A resolved helper cannot be passed as a named argument".
+- Telematics syncs no longer fail with "SyncTelematicDevicesJob has been attempted too many times" for providers on bounded polling. Poll attempts finish within the queue's 90-second reservation, and older queued sync jobs hand off to it instead of running for up to an hour.
+- A transient provider or TLS timeout no longer pauses scheduled polling for several minutes, and connections left in an error state keep being polled.
+- Vehicles that have never reported a position no longer mark every sweep as partial, quarantine deliveries, or show the connection as degraded.
+- Older or delayed samples cannot move a device or attached vehicle position backwards, and partial messages keep existing device identity and metadata.
+- AFAQY recovers from unreadable cached tokens and shares one request deadline across sign-in and unit retrieval.
+- Pausing Safee polling no longer queues failing sync jobs every minute.
+- Device event lookups use a new index instead of full-table scans.
+- The Radar dashboard widget links correctly when rendered outside the Fleet-Ops engine.
 
 ---
 ## Testing
-- Model and controller contract tests cover inspection forms, submissions, links and the PIN lockout.
-- The Fleetbase Postman collection documents the `v1` inspection endpoints.
+- Backend tests cover Safee batching, token refresh, rate limits, ordered ingestion, tenant isolation, delivery checkpoints, paused providers, retry backoff and queue routing.
+- A local run on a single default queue worker polled a 93-vehicle Safee fleet: 5 sweeps used 7 API requests and each completed with 91 positions applied and 2 vehicles without a fix.
 
 ---
 ## Need help?

package/addon/components/telematic/details.hbs

@@ -1,5 +1,5 @@
-<Telematic::TelemetryStatus @resource={{@resource}} @provider={{@resource.provider_descriptor}} />
 <div class="flex flex-col gap-4 p-4 md:p-6" ...attributes>
+    <Telematic::TelemetryStatus @resource={{@resource}} @provider={{@resource.provider_descriptor}} />
     <section class="grid grid-cols-1 gap-3 lg:grid-cols-3">
         {{#each this.healthCards as |card|}}
             <div class="fleet-ops-widget fleet-ops-kpi-tile fleetops-connectivity-kpi-tile {{card.accentClass}} relative min-h-[118px] overflow-hidden rounded-md border bg-white shadow-sm transition hover:-translate-y-px hover:shadow-lg dark:bg-gray-800">

package/addon/components/telematic/telemetry-status.hbs

@@ -1,43 +1,102 @@
-<div {{did-update this.resourceChanged @resource.id @resource.public_id this.provider.key this.supported}}>
+<div class="contents" {{did-update this.resourceChanged @resource.id @resource.public_id this.provider.key this.supported}}>
     {{#if this.supported}}
-        <section class="space-y-3 rounded-md border border-gray-200 p-3 dark:border-gray-700" data-test-telemetry-status>
-            <h3 class="text-sm font-bold">Live telemetry</h3>
-            {{#if this.error}}<p class="text-sm text-red-600" role="alert">{{this.error}}</p>{{/if}}
+        <section class="min-w-0 space-y-3 rounded-md border border-gray-200 bg-white p-4 dark:border-gray-700 dark:bg-gray-800" data-test-telemetry-status>
+            <h3 class="text-sm font-bold text-gray-900 dark:text-gray-100">Live telemetry</h3>
+            {{#if this.error}}<p class="break-words text-xs text-red-600 dark:text-red-400" role="alert">{{this.error}}</p>{{/if}}
             {{#if this.diagnostics}}
-                <dl class="grid grid-cols-2 gap-2 text-xs">
-                    <dt>Polling</dt><dd>{{if this.diagnostics.polling_enabled "Enabled" "Disabled"}}</dd>
-                    <dt>Position webhooks</dt><dd>{{if this.diagnostics.webhooks_enabled "Enabled" "Disabled"}} · {{smart-humanize this.diagnostics.webhook_state}}</dd>
-                    <dt>Last polling sweep</dt><dd>{{n-a this.diagnostics.last_poll.status}} · {{n-a this.diagnostics.last_poll.updated_at}}</dd>
-                    <dt>Units / pages / applied / failed</dt><dd>{{n-a this.diagnostics.last_poll.units}}
-                        /
-                        {{n-a this.diagnostics.last_poll.pages}}
-                        /
-                        {{n-a this.diagnostics.last_poll.applied}}
-                        /
-                        {{n-a this.diagnostics.last_poll.failed}}</dd>
-                    <dt>Last webhook received</dt><dd>{{n-a this.diagnostics.last_webhook.received_at}}</dd>
-                    <dt>Oldest pending delivery</dt><dd>{{n-a this.diagnostics.oldest_pending_at}}</dd>
-                    <dt>Latest queue / source delay (seconds)</dt><dd>{{n-a this.diagnostics.last_ingestion.queue_delay_seconds}}
-                        /
-                        {{n-a this.diagnostics.last_ingestion.source_delay_seconds}}</dd>
-                    <dt>Last polling error</dt><dd>{{n-a this.diagnostics.last_poll.error}}</dd>
+                <dl class="grid grid-cols-1 gap-x-4 gap-y-3 text-xs sm:grid-cols-2 lg:grid-cols-4">
+                    <div class="min-w-0">
+                        <dt class="text-gray-500 dark:text-gray-400">Polling</dt>
+                        <dd class="mt-1 break-words font-semibold text-gray-900 dark:text-gray-100">{{if this.diagnostics.polling_enabled "Enabled" "Disabled"}}</dd>
+                    </div>
+                    <div class="min-w-0">
+                        <dt class="text-gray-500 dark:text-gray-400">Position webhooks</dt>
+                        <dd class="mt-1 break-words font-semibold text-gray-900 dark:text-gray-100">{{if this.diagnostics.webhooks_enabled "Enabled" "Disabled"}}
+                            ·
+                            {{smart-humanize this.diagnostics.webhook_state}}</dd>
+                    </div>
+                    <div class="min-w-0">
+                        <dt class="text-gray-500 dark:text-gray-400">Last polling sweep</dt>
+                        <dd class="mt-1 break-words font-semibold text-gray-900 dark:text-gray-100">{{n-a (smart-humanize this.diagnostics.last_poll.status)}}</dd>
+                    </div>
+                    <div class="min-w-0">
+                        <dt class="text-gray-500 dark:text-gray-400">Sweep updated (UTC)</dt>
+                        <dd class="mt-1 break-words font-semibold text-gray-900 dark:text-gray-100">{{n-a this.diagnostics.last_poll.updated_at}}</dd>
+                    </div>
+                    <div class="min-w-0">
+                        <dt class="text-gray-500 dark:text-gray-400">Units fetched</dt>
+                        <dd class="mt-1 break-words font-semibold text-gray-900 dark:text-gray-100">{{n-a this.diagnostics.last_poll.units}}</dd>
+                    </div>
+                    <div class="min-w-0">
+                        <dt class="text-gray-500 dark:text-gray-400">Pages</dt>
+                        <dd class="mt-1 break-words font-semibold text-gray-900 dark:text-gray-100">{{n-a this.diagnostics.last_poll.pages}}</dd>
+                    </div>
+                    <div class="min-w-0">
+                        <dt class="text-gray-500 dark:text-gray-400">Applied</dt>
+                        <dd class="mt-1 break-words font-semibold text-gray-900 dark:text-gray-100">{{n-a this.diagnostics.last_poll.applied}}</dd>
+                    </div>
+                    <div class="min-w-0">
+                        <dt class="text-gray-500 dark:text-gray-400">Failed</dt>
+                        <dd class="mt-1 break-words font-semibold {{if this.diagnostics.last_poll.failed 'text-red-600 dark:text-red-400' 'text-gray-900 dark:text-gray-100'}}">{{n-a
+                                this.diagnostics.last_poll.failed
+                            }}</dd>
+                    </div>
+                    <div class="min-w-0">
+                        <dt class="text-gray-500 dark:text-gray-400">Last webhook (UTC)</dt>
+                        <dd class="mt-1 break-words font-semibold text-gray-900 dark:text-gray-100">{{n-a this.diagnostics.last_webhook.received_at}}</dd>
+                    </div>
+                    <div class="min-w-0">
+                        <dt class="text-gray-500 dark:text-gray-400">Oldest pending (UTC)</dt>
+                        <dd class="mt-1 break-words font-semibold text-gray-900 dark:text-gray-100">{{n-a this.diagnostics.oldest_pending_at}}</dd>
+                    </div>
+                    <div class="min-w-0">
+                        <dt class="text-gray-500 dark:text-gray-400">Queue delay (seconds)</dt>
+                        <dd class="mt-1 break-words font-semibold text-gray-900 dark:text-gray-100">{{n-a this.diagnostics.last_ingestion.queue_delay_seconds}}</dd>
+                    </div>
+                    <div class="min-w-0">
+                        <dt class="text-gray-500 dark:text-gray-400">Source delay (seconds)</dt>
+                        <dd class="mt-1 break-words font-semibold text-gray-900 dark:text-gray-100">{{n-a this.diagnostics.last_ingestion.source_delay_seconds}}</dd>
+                    </div>
                 </dl>
+                {{#if this.diagnostics.last_poll.error}}
+                    <p class="break-words text-xs text-red-600 dark:text-red-400"><span class="font-semibold">Last polling error:</span> {{this.diagnostics.last_poll.error}}</p>
+                {{/if}}
             {{/if}}
             {{#if this.supportsSecureWebhooks}}
-                {{#if this.registrationInstructions}}<p class="text-xs">{{this.registrationInstructions}}</p>{{/if}}
-                {{#if this.provisional}}<p class="text-xs">The payload adapter is provisional until a real delivery has been verified.</p>{{/if}}
-                <button type="button" class="btn btn-default btn-sm" disabled={{this.configure.isRunning}} {{on "click" this.reveal}}>Show registration URL</button>
-                {{#if this.webhookUrl}}
-                    <ClickToCopy @value={{this.webhookUrl}}><Input @value={{this.webhookUrl}} class="form-input w-full" readonly aria-label="Secure telemetry webhook URL" /></ClickToCopy>
-                    <p class="text-xs">This URL contains a secret. Rotating it immediately invalidates the previous URL and requires registration with the provider again.</p>
-                    <button type="button" class="btn btn-default btn-sm" disabled={{this.configure.isRunning}} {{on "click" this.rotate}}>Rotate secret and invalidate old URL</button>
-                {{/if}}
+                <details class="border-t border-gray-200 pt-3 text-xs dark:border-gray-700">
+                    <summary class="cursor-pointer font-semibold text-gray-700 dark:text-gray-300">Webhook setup</summary>
+                    <div class="mt-3 space-y-2 text-gray-500 dark:text-gray-400">
+                        {{#if this.registrationInstructions}}<p>{{this.registrationInstructions}}</p>{{/if}}
+                        {{#if this.provisional}}<p>Position webhooks are awaiting verification with a real delivery.</p>{{/if}}
+                        <Button @size="xs" @text="Show registration URL" @disabled={{this.configure.isRunning}} @onClick={{this.reveal}} />
+                        {{#if this.webhookUrl}}
+                            <ClickToCopy @value={{this.webhookUrl}}>
+                                <Input @value={{this.webhookUrl}} class="form-input w-full text-xs" readonly aria-label="Secure telemetry webhook URL" />
+                            </ClickToCopy>
+                            <p>This URL contains a secret. Rotating it immediately invalidates the previous URL and requires registration with the provider again.</p>
+                            <Button @size="xs" @text="Rotate secret" @disabled={{this.configure.isRunning}} @onClick={{this.rotate}} />
+                        {{/if}}
+                    </div>
+                </details>
+            {{/if}}
+            {{#if this.diagnostics.recent_failures.length}}
+                <details class="border-t border-gray-200 pt-3 text-xs dark:border-gray-700">
+                    <summary class="cursor-pointer font-semibold text-red-600 dark:text-red-400">Recent delivery failures ({{this.diagnostics.recent_failures.length}})</summary>
+                    <ul class="mt-3 max-h-48 space-y-3 overflow-y-auto">
+                        {{#each this.diagnostics.recent_failures as |delivery|}}
+                            <li class="flex flex-wrap items-start justify-between gap-2">
+                                <div class="min-w-0 flex-1 break-words">
+                                    <p class="text-gray-500 dark:text-gray-400">{{delivery.received_at}} UTC</p>
+                                    <p class="mt-1 text-gray-700 dark:text-gray-300">{{delivery.error}}</p>
+                                </div>
+                                {{#if (eq delivery.status "quarantined")}}
+                                    <Button @size="xs" @text="Replay delivery" @onClick={{fn this.replay delivery.uuid}} />
+                                {{/if}}
+                            </li>
+                        {{/each}}
+                    </ul>
+                </details>
             {{/if}}
-            {{#each this.diagnostics.recent_failures as |delivery|}}
-                <div class="text-xs"><span>{{delivery.received_at}} · {{delivery.error}}</span>
-                    {{#if (eq delivery.status "quarantined")}}<button type="button" class="btn btn-default btn-sm" {{on "click" (fn this.replay delivery.uuid)}}>Replay delivery</button>{{/if}}
-                </div>
-            {{/each}}
         </section>
     {{/if}}
 </div>

package/addon/components/widget/radar.hbs

@@ -18,7 +18,7 @@
             </div>
             <div class="mt-auto flex items-center justify-between text-[10px] text-gray-500 dark:text-gray-400">
                 <span data-test-radar-widget-snoozed>{{this.stats.snoozed}} {{t "widget.radar.snoozed"}}</span>
-                <LinkTo @route="management.index" class="inline-flex items-center gap-1 font-semibold text-blue-600 hover:text-blue-700 dark:text-blue-300 dark:hover:text-blue-200" data-test-radar-widget-link>
+                <LinkTo @route={{this.radarRoute}} class="inline-flex items-center gap-1 font-semibold text-blue-600 hover:text-blue-700 dark:text-blue-300 dark:hover:text-blue-200" data-test-radar-widget-link>
                     {{t "widget.radar.open-radar"}}
                     <FaIcon @icon="arrow-right" class="text-[9px]" />
                 </LinkTo>

package/addon/components/widget/radar.js

@@ -1,7 +1,9 @@
 import Component from '@glimmer/component';
 import { tracked } from '@glimmer/tracking';
 import { inject as service } from '@ember/service';
+import { getOwner } from '@ember/application';
 import { task } from 'ember-concurrency';
+import config from '../../config/environment';
 
 /**
  * "Radar · 3 overdue": the dashboard tile that links to the page.
@@ -21,6 +23,11 @@ export default class WidgetRadarComponent extends Component {
         return this.summary?.summary ?? { open: 0, overdue: 0, snoozed: 0, critical: 0 };
     }
 
+    get radarRoute() {
+        // Dashboard widgets render under the host owner, outside the engine's routing context.
+        return getOwner(this).mountPoint ? 'management.index' : `${config.mountedEngineRoutePrefix}.management.index`;
+    }
+
     get accentClass() {
         if (this.stats.critical > 0 || this.stats.overdue > 0) {
             return 'kpi-accent-bad';

package/composer.json

@@ -1,6 +1,6 @@
 {
     "name": "fleetbase/fleetops-api",
-    "version": "0.6.66",
+    "version": "0.6.67",
     "description": "Fleet & Transport Management Extension for Fleetbase",
     "keywords": [
         "fleetbase-extension",
@@ -39,6 +39,7 @@
         "spatie/icalendar-generator": "^2.8"
     },
     "require-dev": {
+        "illuminate/encryption": "^9.0|^10.0",
         "friendsofphp/php-cs-fixer": "^3.34.1",
         "nunomaduro/collision": "^7.0",
         "pestphp/pest": "^2.33.2",

package/docs/AFAQY.md

@@ -4,7 +4,7 @@
 
 AFAQY uses the provider-neutral telemetry infrastructure described in [TELEMETRY_ARCHITECTURE.md](TELEMETRY_ARCHITECTURE.md). Its adapter owns authentication and payload normalization; shared services own persistence, ingestion, and diagnostics.
 
-The existing AFAQY integration already used Units List. The new path requests 1,000 units per page with `simplify: 0` and the documented `basic` / `last_update` projection groups. It fetches all pages, places batches of 100 units in an encrypted durable inbox, and ingests them asynchronously. A connection in transient `error` status remains eligible; `disabled` connections and connections with `meta.telemetry_sync_enabled: false` do not.
+The existing AFAQY integration already used Units List. The new path requests 1,000 units per page with `simplify: 0` and the documented `basic` / `last_update` projection groups. It fetches all pages, places configurable batches of units (100 by default) in an encrypted durable inbox, and ingests them asynchronously. A connection in transient `error` status remains eligible; `disabled` connections and connections with `meta.telemetry_sync_enabled: false` do not.
 
 Position webhooks use the same ingestion path and do not authenticate against AFAQY. The adapter provisionally accepts a unit object, a unit array, or a `data` envelope containing either, including `_id` with nested `data.last_update`. This is an assumption, not a verified AFAQY webhook contract. Unsupported authenticated JSON is quarantined and can be replayed after an adapter correction. Vehicle event webhooks and Signals history are outside this implementation.
 
@@ -12,30 +12,29 @@ Position webhooks use the same ingestion path and do not authenticate against AF
 
 ## Stage 1 deployment
 
-1. Apply the Fleet-Ops migration `2026_09_15_000001_create_telematic_telemetry_tables`. It adds shared `telematic_deliveries`, `telematic_sync_runs`, and `telematic_webhook_credentials` tables for any registered telemetry adapter. It also adds a composite device lookup index; existing telemetry data is preserved.
-2. Use a shared cache with distributed locks (Redis recommended) across all scheduler and worker instances. An array cache is suitable only for isolated tests. Cached login tokens are encrypted, scoped to account/host/password, and reused for up to 29 days; a credential change selects a new cache namespace. All API attempts for the same host/account share a rolling 60-request/minute budget.
+1. Apply the Fleet-Ops migrations `2026_09_15_000001_create_telematic_telemetry_tables` and `2026_09_17_000001_add_device_event_uuid_lookup_index`. The first adds shared `telematic_deliveries`, `telematic_sync_runs`, and `telematic_webhook_credentials` tables plus a composite device lookup index. The second ensures `device_events.uuid` has a usable leading index, including installations upgraded from the legacy events table. Existing telemetry data is preserved.
+2. Use a shared cache with distributed locks (Redis recommended) across all scheduler and worker instances. An array cache is suitable only for isolated tests. API, queue, and scheduler services must load the same effective `APP_KEY`, cache configuration, and telemetry settings, including Compose overrides and cached Laravel configuration. Preserve the established key that decrypts existing application data; do not generate a replacement as a troubleshooting step. Cached login tokens are encrypted, scoped to account/host/password, and reused for up to 29 days; a credential change selects a new cache namespace. All API attempts for the same host/account share a rolling 60-request/minute budget.
 3. Set the following environment variables and rebuild the application's configuration cache:
 
    ```dotenv
    AFAQY_POLLING_ENABLED=true
    AFAQY_WEBHOOKS_ENABLED=false
-   AFAQY_POLL_QUEUE=afaqy-poll
-   AFAQY_INGESTION_QUEUE=afaqy-ingest
    ```
 
-4. Provision workers for both queues. For example:
+4. Keep the existing worker consuming the application's `default` queue, and restart long-running workers after configuration or code changes. Polling, ingestion, and broadcasts use that queue by default. The baseline requires no additional containers, dedicated queues, or custom batch size; ingestion batches default to 100 units.
 
-   ```sh
-   php artisan queue:work --queue=afaqy-poll --timeout=120 --tries=5
-   php artisan queue:work --queue=afaqy-ingest --timeout=60 --tries=1
-   ```
+   One worker can execute the whole pipeline: the polling job enqueues ingestion and returns without waiting for those jobs. Ingestion, broadcasts, and unrelated jobs then share that worker serially. Pending poll ingestion coalesces later polling ticks. This avoids accumulating obsolete sweeps, but it does not establish a particular freshness or throughput bound. Measure complete cycles using the deployment's actual worker count and workload.
+
+   Remove stale `TELEMATICS_POLL_QUEUE`, `TELEMATICS_INGESTION_QUEUE`, `AFAQY_POLL_QUEUE`, and `AFAQY_INGESTION_QUEUE` overrides unless the existing worker consumes those queues. Jobs sent to an unconsumed queue cannot progress.
 
-   Set the queue connection's `retry_after` above 150 seconds (for example 180). Run multiple ingestion workers, and retain workers for the application's broadcast/default queue. Queue names default to `default` if not configured, for compatibility; dedicated workers prevent discovery or unrelated jobs from delaying telemetry.
+   Check that the effective queue reservation/retry period is compatible with the longest job the worker consumes, so the broker cannot make a still-running job available again. A worker's CLI timeout does not override a timeout declared by a job; manual discovery declares 3,600 seconds. This is a deployment compatibility check, not a requirement to add queues or prescribe a new reservation period as part of this patch.
 
-5. Confirm the application scheduler executes every minute. It runs both `fleetops:sync-telematics` and `fleetops:drain-telematic-inbox`. The drain recovers accepted deliveries if broker dispatch failed or a worker stopped; no payload is acknowledged before durable persistence.
+5. Confirm the application scheduler executes every minute. It runs both `fleetops:sync-telematics` and `fleetops:drain-telematic-inbox`. The drain recovers accepted deliveries if broker dispatch failed or a worker stopped; no payload is acknowledged before durable persistence. Both commands use two-minute overlap leases so an interrupted scheduler cannot block telemetry for the former 24-hour default.
 6. Open the connection's live telemetry panel. Verify a full polling run transitions from fetching to ingesting to completed, with expected unit/page/applied counts. `incomplete` and `partial` are failures to investigate, not complete fleet coverage.
 
-Rate-limit retries honor `Retry-After` (bounded to one hour). Other polling failures use 15/60/180/300-second backoff, with five queue attempts. Poll HTTP requests have 20-second response and five-second connection limits. Sweeps stop at a 90-second fetch budget or 100 pages and report incomplete coverage. Pending poll ingestion coalesces later polling ticks. Failed ingestion retries only failed units; the original encrypted body remains available. Five failed/interrupted attempts quarantine the delivery.
+Use the ordinary application scheduler for the baseline. Laravel executes foreground scheduled commands in registration order, so slow unrelated commands can delay polling and inbox recovery even when the telemetry queue is empty. Include scheduler execution time and application startup in diagnosis; distributed command locks are acquired only after Laravel boots. Additional scheduler isolation or worker capacity is an optional operational decision that requires separate measurement, not a prerequisite imposed by this integration.
+
+Rate-limit retries honor `Retry-After` (bounded to one hour). Other polling failures use 15/60/180/300-second backoff, with five queue attempts. Poll HTTP requests default to a 45-second total request timeout, configurable with `TELEMATICS_REQUEST_TIMEOUT_SECONDS`, and a connection limit of at most five seconds. Each request is capped by the remaining 90-second fetch budget; changing the request setting does not extend that sweep budget. Sweeps also stop after 100 pages and report incomplete coverage. Pending poll ingestion coalesces later polling ticks. Failed ingestion retries only failed units; the original encrypted body remains available. Five failed/interrupted attempts quarantine the delivery. Internally generated polling batches are normalized per unit: a unit without a position is reported as invalid without discarding valid neighbors. Such a sweep reports partial coverage. External webhook payloads still require the adapter's supported position envelope.
 
 ## Stage 2 registration
 
@@ -58,22 +57,77 @@ Existing provider interfaces and vehicle/trailer event names remain compatible.
 
 The public receiver remains `POST webhooks/telematics/afaqy?telematic=...&key=...` under the application's API prefix. A new `device.telemetry_updated` broadcast identifies the device; an open device panel reloads its authorized resource and refetches after socket reconnection. Company broadcast identity comes from the persisted asset, not a worker session. Map movement uses the existing vehicle/trailer events, rejects older source timestamps, and reloads visible telemetry assets after socket reconnection with at most five concurrent requests.
 
-Configure shared queues, retention, limits, and freshness under `telematics.telemetry` using `server/config/telemetry.php`. AFAQY's adapter switches and optional queue overrides remain under `telematics.afaqy` in `server/config/afaqy.php`. Defaults: two MiB per webhook, 10,000 pending deliveries per connection, processed payload retention 24 hours, quarantine retention seven days, stale-position thresholds 120 seconds with ignition on and 600 seconds with ignition off/unknown. Each device's metadata records the effective threshold. Connection status categories remain unchanged.
+Configure shared queues, retention, limits, and freshness under `telematics.telemetry` using `server/config/telemetry.php`. `TELEMATICS_BATCH_SIZE` controls units per ingestion delivery (default 100); `TELEMATICS_REQUEST_TIMEOUT_SECONDS` controls the bounded polling request timeout (default 45 seconds). AFAQY's adapter switches and optional queue overrides remain under `telematics.afaqy` in `server/config/afaqy.php`. Defaults: two MiB per webhook, 10,000 pending deliveries per connection, processed payload retention 24 hours, quarantine retention seven days, stale-position thresholds 120 seconds with ignition on and 600 seconds with ignition off/unknown. Each device's metadata records the effective threshold. Connection status categories remain unchanged.
 
 ## Incident verification
 
-For the screenshot's vehicle, compare the same AFAQY unit ID/IMEI and attached Fleetbase asset. Fetch its Units List record and compare raw UTC `dtt`/`dts` with `device.meta.telemetry`, `last_online_at`, and `vehicle.telematics.last_event_at`. Then inspect polling-run status, oldest pending delivery, worker/broadcast queue delay, and the open browser panel. A June-versus-September date gap cannot be accounted for by UTC+3 alone. No affected production account was accessed during implementation.
+### September 2026 Docker investigation
+
+The affected Docker deployment was investigated using live AFAQY responses and its real database, queue workers, and scheduler. The initial implementation's isolated tests had not exercised that deployment. The observed June-to-September gap was not a timezone conversion issue.
+
+The investigation identified several independent failures:
+
+- **Encryption configuration differed between services.** An application-only Compose override made the API and queue workers load different effective encryption keys. The API's established key successfully decrypted existing ledger and inspection records; that key was preserved and shared across services. Cached-token recovery remains necessary, but repeatedly refreshing tokens cannot repair inconsistent service configuration.
+- **Event lookups scanned a large legacy table.** The deployment had approximately 50,000 event rows occupying 1.2 GB, with no index on `device_events.uuid`. Activity logging refreshes saved models by UUID, producing measured 10–13-second table scans. The UUID-index migration was applied; the measured UUID lookup then took about 7 ms. A subsequent real single-unit ingestion took approximately 1.7 seconds across 25 SQL statements, with about 249 ms spent in SQL. These are different measurements—a prior lookup versus a complete later ingestion—not a controlled whole-fleet speedup ratio.
+- **Manual sync progress was invisible during ingestion.** The job fetched 381 units but retained queued/zero-page metadata while doing database work. It now persists running phases and fetched totals before ingestion, then checkpoints completed counters every 25 records or five seconds and at page boundaries. The manual 381-unit inventory sync completed after the database correction.
+- **Some inventory records had no position.** The live response contained 381 units, of which 379 had valid positions and two lacked `last_update`. The polling worker now processes valid records independently and reports the other two as invalid; it does not discard their whole batches or count them as applied telemetry.
+- **The request deadline was too short for an observed response.** One live Units List response took about 23.6 seconds, exceeding the former 20-second limit. The configurable 45-second request timeout remains constrained by the overall fetch budget.
+- **Automatic processing was not running.** The rollout polling switch was disabled and the general development scheduler was stopped. Enabling it dispatched polling and unrelated scheduled workload. Investigation temporarily used a dedicated telemetry scheduler, four generic workers, and batches of 25. That experiment changed the deployment's worker topology and does not validate the user's single-worker baseline. The temporary services, dedicated queue settings, batch override, and queue reservation changes have since been removed. Configuration is restored to the existing default worker and ordinary application scheduler, retaining the shared established encryption key, enabled polling, schema index, and code corrections.
+- **Scheduler interruption left a 24-hour overlap lock.** Recreating the scheduler during execution left its default overlap lease in the shared cache, preventing later polling dispatches even with empty worker queues. The two telemetry scheduled commands now use two-minute overlap leases, and the sync command's own lock also expires after two minutes. During recovery, only the confirmed abandoned telemetry mutex was cleared after stopping the scheduler; unrelated cache entries and locks were preserved.
+
+During the temporary four-worker experiment, the initial eleven automatic sweeps drained 379 valid position records per sweep and explicitly reported the two records without positions as partial coverage. Those sweeps ranged from 22 to 65 seconds with a median of 36 seconds. A later sweep during scheduler recreation and heavy unrelated scheduled workload took 152 seconds; 22–65 seconds is therefore not the full observed range. The subsequent run stalled with retained inbox batches while the host was under heavy load. Its Docker VM had eight CPUs and about 8 GB of memory; the host reported about 16 GB of swap in use and a load average near 48. Pausing a separate stack did not immediately restore normal startup. A bounded startup probe measured about 12 seconds for Composer autoload and about 40 seconds for provider registration/boot, excluding ingestion. Temporary dedicated cron startup wrappers were also removed with that experiment. Recovery and repeated-cycle cadence on the restored single-worker baseline remain open verification gates.
+
+These observations establish that the inspected 381-unit fleet was fetched and ingested during the investigation. They do not establish single-worker performance, a consistent sub-minute sweep, the 5,000-unit target, webhook throughput, or end-to-end browser latency. Actual AFAQY webhook delivery remains unverified.
+
+### Verification checklist
+
+1. Compare the same provider unit identifier across the live Units List record, persisted device, and attached asset. Compare raw UTC `dtt`/`dts` with `device.meta.telemetry`, `last_online_at`, and the asset's source timestamp.
+2. Verify effective API/worker/scheduler configuration, not only the `.env` file. Compare keys securely without printing them; confirm the established key decrypts existing records before aligning services. Rebuild cached configuration and restart long-running processes after changes.
+3. Check applied migrations and the actual query plan for UUID-based event lookup. Confirm the lookup uses an index instead of scanning historical event payloads.
+4. Verify polling is enabled, the scheduler invokes both telemetry commands every minute, and workers consume the effective polling, ingestion, and broadcast queues. Watch pending/retry/processing delivery age and depth. After interruption, inspect the specific scheduler mutex and queued/reserved poll jobs before clearing anything. New scheduler leases expire after two minutes, but an old 24-hour lease already stored in cache keeps its original expiry; remove only a confirmed abandoned telemetry lock. Do not flush the shared cache or clear all scheduled-task locks.
+5. Observe several complete runs. Reconcile fetched, applied, and invalid/failed counts; inspect quarantined records safely. A run with units lacking positions must remain explicitly partial.
+6. Confirm the stored device and asset advance with newer source timestamps and verify their rendered state separately. Database ingestion success does not by itself validate browser/socket delivery.
+
+## Cached-token decryption failures
+
+A manual sync runs `SyncTelematicDevicesJob`; scheduled telemetry polling uses
+`PollTelematicTelemetry`. Both authenticate through the AFAQY provider. Previously,
+an unreadable encrypted access token in the shared cache raised Laravel's
+`DecryptException` before the provider could refresh the token. This is a cache
+recovery defect; a MAC error alone does not establish that deployment keys differ.
+
+The provider now catches decryption failure only for its disposable cached token,
+evicts that one entry under the existing account lock, and authenticates again
+using the already-resolved connection credentials. A replacement token is cached
+encrypted. Login failures and rate limits still propagate. Persisted credentials,
+webhook secrets, and inbox payloads are not reset or treated as plaintext.
+A warning with reason `cached_token_decryption_failed` identifies this path without
+logging secrets. Deploy the patch and restart long-running queue workers before
+retrying. If the exception remains, capture the calling stack frames without
+arguments to identify which encrypted value failed; do not rotate APP_KEY or flush
+the entire shared cache as a diagnostic step.
+
+Regression coverage now uses Laravel's real encrypter (`illuminate/encryption` is
+a development dependency), persisted encrypted credentials, and both manual and
+scheduled job handlers. Tampered MACs and foreign-key cache entries fail before
+the fix and recover afterward. Provider HTTP responses remain simulated; these
+checks do not prove successful authentication against a live AFAQY account.
 
 ## Validation and acceptance gates
 
 Run the new provider contract and database ingestion suites with the package Pest runner, and the provider-neutral telemetry browser tests with Ember:
 
 ```sh
-php scripts/pest-runner.php server/tests/Unit/Support/Telematics/Providers/AfaqyRealtimeContractTest.php
-php scripts/pest-runner.php server/tests/Feature/Http/AfaqyRealtimeIngestionTest.php
+php scripts/pest-runner.php server/tests/Unit/Support/Telematics/Providers/AfaqyRealtimeContractTest.php --debug
+php scripts/pest-runner.php server/tests/Feature/Http/AfaqyRealtimeIngestionTest.php --debug
+php scripts/pest-runner.php server/tests/Feature/Http/TelematicMixedFleetDeliveryTest.php --debug
+php scripts/pest-runner.php server/tests/Feature/Http/Api/DeviceEventUuidIndexMigrationTest.php --debug
+php scripts/pest-runner.php server/tests/SupportJobAndAiCoverageTest.php --filter='manual sync' --debug
 node_modules/.bin/ember test --filter=telemetry
 ```
 
+The incident patch passed 92 focused backend tests. The incident regressions exercise valid–invalid–valid polling batches through encrypted inbox processing and database ingestion, strict webhook quarantine, UUID-index migration behavior, and manual progress checkpoints/partial failures. The mixed-fleet SQLite fixture disables model listeners and external transports, and the manual-progress tests use provider/service doubles; those tests alone would not expose the deployed activity-log table scan. The live Docker investigation above supplies separate evidence for the actual MySQL schema and model path. Use `--debug` to retain PHPUnit test-pass events and its final exit status when the normal package formatter emits no summary.
+
 The original implementation previously processed 5,000 signals plus 5,000 duplicate reconciliation observations in 45.9 seconds in the opt-in local SQLite benchmark (`XDEBUG_MODE=off AFAQY_RUN_LOAD_TESTS=1` with the ingestion suite above). During the provider-neutral refactor review, the final run took 66.5 seconds and failed the local 60-second budget; the original commit repeated in the same runtime took 83.4 seconds. Different machine/build load prevents a controlled speed comparison. The load gate remains open. It uses real model/database ingestion with indexed fixture tables, simulated spatial functions, and captured broadcasts; it excludes provider networking, broker latency, attached-asset load, and browser rendering.
 
 The original implementation validation passed 103 selected backend tests, including database ingestion, sensor preservation, retention recovery, and provider contracts. A separate check against the installed Laravel dispatcher verified that broker failures release uniqueness leases. The standard browser launcher is blocked by the linked Testem/execa CommonJS/ES-module mismatch; the two focused browser tests passed all nine assertions in an isolated local harness with lazy engine assets and dummy host configuration. These cover live panel refresh, reconnect, freshness labels, map movement, and reversed broadcasts; they do not exercise a production socket server. PHPStan is not clean in the package-only runtime (unresolved Laravel helpers/model types and strict type findings); it is not counted as a passing gate.

package/docs/SAFEE.md

@@ -0,0 +1,125 @@
+# Safee / DSCO live polling
+
+## Problem and API contract
+
+The legacy sync fetched inventory and current state, then made three serial
+requests per vehicle (`last-info`, `positions`, and `events`) before applying a
+second ingestion pass. A 93-vehicle fleet required 281 data requests, excluding
+authentication. Each enrichment request could wait up to 120 seconds. Historical
+samples could overwrite the latest position, and failed connections were excluded
+from subsequent scheduled discovery.
+
+Safee's [Tracking REST Service V2.2.4.0](https://www.safee.com/_docs_/Safee_Tracking_REST_Service.pdf)
+documents:
+
+- `POST /api/v2/vehicle/list-info` with an empty JSON object for inventory.
+- `POST /api/v2/vehicle/last-state` with `live: true`, `endDate: null`, and a
+  `vehicles` ID array to retrieve current state, position, speed, heading, and
+  available counters.
+- Up to 1,000 vehicle IDs per request, and 50 requests per user per second.
+- Unix timestamps with fractional seconds. The response envelope's `time` is not
+  a vehicle's source timestamp.
+- OAuth access-token expiry and refresh-token grants.
+
+There are no position webhooks for this integration. Temperature/door enrichment
+and historical position/event retrieval remain outside routine live polling.
+Existing enrichment APIs remain available for explicit use.
+
+## Polling and ingestion
+
+Safee opts into the existing provider-neutral telemetry pipeline. No Safee tables,
+UI branches, dedicated queues, or additional workers are introduced.
+
+1. The minute scheduler coalesces queued/running polling and pending ingestion.
+   Enabled connections remain eligible after transient errors. When
+   `SAFEE_POLLING_ENABLED=false`, Safee is paused; it does not fall back to the
+   legacy monolithic discovery job.
+2. Inventory is cached briefly. Manual discovery refreshes it. Pagination uses
+   stable inventory slices; `list-info` itself is not treated as a paginated API.
+3. Each slice retrieves current states in one batch of at most 1,000 IDs. Missing
+   states are reported per unit; malformed, duplicate, or unsuccessful responses
+   cannot masquerade as successful empty fleets.
+4. Raw batches enter the shared encrypted durable inbox. Existing default-queue
+   jobs apply them transactionally with per-device serialization, deduplication,
+   and source-time ordering. Older samples cannot move current device or attached
+   asset positions backwards.
+5. Units without a valid fix (for example source date `0`) keep their device link
+   and are counted in `invalid_count`, but a polled delivery is not quarantined
+   and its run is not marked partial for them. Pushed deliveries keep quarantine.
+6. Partial messages preserve attachment identity and existing metadata, counters,
+   and sensors. A zero source date is treated as missing data. Source timestamps
+   are converted to UTC; receipt time is never substituted for a missing GPS time.
+
+A fresh fleet sweep uses `1 + ceil(N / 1000)` data requests; subsequent sweeps use
+`ceil(N / 1000)` while inventory is cached. These are request-count improvements,
+not guarantees about provider latency or application throughput.
+
+Authentication is cached using credential-sensitive keys and encrypted tokens.
+Expiry is honored, a rejected token may be refreshed once, and authentication,
+manual syncs, and retries share the per-user request budget. HTTP429 respects
+`Retry-After`; connection/server failures use bounded queue retry delays. Scheduled
+poll retries wait at most 60 seconds (15, 30, 60, 60) so a transient login or TLS
+timeout cannot suppress minute polling for several minutes; manual requests keep
+15, 60, 180, 300 seconds. Previously queued legacy discovery jobs exit quietly when
+polling is paused or already queued instead of marking the connection as errored.
+
+## Configuration and rollout
+
+Safee polling is enabled by default because it replaces the previous scheduled
+sync. `SAFEE_POLLING_ENABLED=false` pauses Safee polling/manual batch sync.
+`meta.telemetry_sync_enabled=false` or a disabled connection also prevents work.
+Webhooks remain unsupported.
+
+- `SAFEE_INVENTORY_CACHE_SECONDS`: default300, capped at300seconds.
+- `SAFEE_REQUEST_TIMEOUT_SECONDS`: default30seconds, capped by the remaining sweep
+  budget. Authentication and data requests share that budget.
+- Existing `TELEMATICS_POLL_QUEUE` and `TELEMATICS_INGESTION_QUEUE` default to
+  `default`. One existing queue worker can process both types of jobs.
+  To run telematics on dedicated workers for one instance, see
+  [Dedicated telematics queue workers](TELEMATICS_QUEUES.md).
+- Poll attempts use an80-second hard timeout and a60-second cooperative sweep
+  budget, below the baseline Redis `retry_after=90` seconds.
+
+Apply the existing generic telemetry migrations and device-event lookup index,
+then reload long-running workers so they use the new provider class. Clear/rebuild
+cached configuration as part of the application's normal deployment process.
+Do not retry an old failed monolithic job to recover a connection; start a new
+manual sync or let scheduled polling recover it.
+
+Inspect the shared telemetry diagnostics for fetched/applied/failed counts,
+incomplete sweeps, pending deliveries, queue delay, and source age. Manual sync
+completion is tied to its ingestion run; queueing or fetching alone is not
+reported as a completed sync.
+
+## Validation boundaries
+
+The local DSCO connection had93 stored units. A read-only API probe on
+2026-09-17 authenticated in2.466seconds, listed all93 units in4.136seconds,
+and fetched all93 current states in3.932seconds. The returned access token
+expired after300seconds. Some units returned source date0; these are invalid
+positions and must remain visible as partial data rather than fabricated fixes.
+
+The immediately preceding legacy worker failure was an SSL connection timeout
+at the login endpoint. Subsequent authenticated requests succeeded. This shows
+an intermittent provider/network failure as well as the independently verified
+request-volume, token-refresh, and scheduler-recovery defects.
+
+Contract and database tests use controlled fixtures; they do not prove production
+throughput. Production freshness remains bounded by Safee/device reporting,
+provider response time, the minute polling interval, and available capacity of
+the shared worker. Monitor queue growth with the rest of the application's jobs.
+
+### Local default-worker run (2026-09-17, 06:40-06:46 UTC)
+
+After restarting the single existing `queue:work` worker on the local stack, five
+scheduled sweeps of the 93-unit DSCO connection made 7 Safee requests in total:
+1 token, 1 `list-info`, and 5 `last-state`. Every run completed with 93 units,
+91 applied, 2 counted as invalid, one processed delivery, no Safee failed jobs, and
+9-15 seconds from run creation to processed delivery. The legacy design would have
+made about 281 data requests per sweep.
+
+Only 42 of the 91 positioned units had a source fix under five minutes old (median
+age 386 seconds). This reflects when devices last reported; faster polling cannot
+improve it. AFAQY polling on the same worker hit 44-second `units/lists` timeouts
+during this window, and those jobs share that worker's capacity. This run is local
+evidence, not production acceptance.