sailkick-boat 0.14.7 → 0.17.2

package/README.md

@@ -27,7 +27,7 @@ One Signal K plugin, independently-toggleable modules — so the boat stays
   same contracts as the cloud, offline-first):
   - **`/ws/telemetry`** — the app's live telemetry bus, fed from local SignalK.
   - **`/api/history/{series,track}`** — the app's Trends panel + track, served
-    from the boat's local InfluxDB (or a DB-less telemetry ring) — full local history.
+    from a live ring sampled on the boat — full local history, no database.
 
 Kept as separate modules so a proxy fault can't wedge the data-critical sync path.
 
@@ -139,37 +139,92 @@ config passes through untouched. (Hand-edit `proxy.openAccess: false` to keep th
 login gate — there is no toggle, since the gate cannot complete over the mirror anyway.)
 
 ## Local history (offline Trends + track)
-Two ways, chosen automatically:
-- **Read token set** → full history queried from a **local InfluxDB** (e.g. a bucket
-  written by `signalk-to-influxdb-v2`).
-- **No token, telemetry on** → a **DB-less ring** sampled from live telemetry (the
-  same BoatState feeding `/ws/telemetry`) — for boats with no local InfluxDB, e.g.
-  **SignalK on a Victron GX / Venus OS**. Same JSON contract, fully offline, no database.
-  (`historyAvailable` reports true either way, so the app shows the Trends panel.)
-  Same eight channels as the InfluxDB path, `stw` included.
-
-**Serving an archive.** Set `historyToken` and the three `history*` fields point at any
-InfluxDB holding `signalk-to-influxdb-v2`-schema data — a still-running local database,
-or a bucket you imported an old logbook into. That is the archive's real value: history
-from before this boat ever synced to the cloud. The token is the switch; blank means the
-built-in ring. The plugin logs which source it chose at startup:
-
-```
-[sailkick-boat] history -> archive http://127.0.0.1:8086 org=addiction bucket=bandg
-[sailkick-boat] history -> built-in ring (no archive token set)
-```
-
-Upgrading from before 0.14.7, when these lived in a hidden `proxy.history` block: your
-saved values still win over the newly pre-filled defaults, so an archive in a
-non-default org/bucket is not silently lost when the config UI writes `signalk` over it.
-The plugin says so in the log — copy the values into the visible fields when convenient.
-
-**Do you actually need a local InfluxDB?** For *live* trends, usually not. The app never requests finer
-than `every=5s` over a `window` of 24 h, and the ring's floor at that window is 2 s — so
-set `ringSampleSec: 5` and it matches anything the UI can draw, from live state, with no
-database to run. A local InfluxDB is worth it only for history predating cloud sync,
-more than 30 days offline, or channels outside BoatState (engine, batteries) that the
-Trends panel does not chart anyway.
+**One source: a live ring**, sampled from the same BoatState that feeds `/ws/telemetry`
+— no database, works on a Victron GX with nothing else installed. `historyAvailable` is
+forced on so the app shows the Trends panel, and the eight channels match what the cloud
+serves, `stw` included.
+
+There is deliberately **no way to point this at a local InfluxDB**. Until v0.15.0 a read
+token did exactly that, and it was a trap: the app only ever asks for a *relative* window
+clamped to 24 h, so aiming it at a bucket of older data matched nothing — Trends went
+blank **and** the working live ring was switched off. If you still have a token in your
+config it is now inert; the plugin logs `history -> live ring` regardless.
+
+A local InfluxDB is not a competitor here anyway. The app never requests finer than
+`every=5s` over a 24 h window, and the ring's floor at that window is 2 s — set
+`ringSampleSec: 5` and it matches anything the UI can draw, from live state. What an old
+database *is* good for is its contents ending up **in the cloud** — see below.
+
+## Uploading AIS targets
+The cloud app already draws other vessels, but its AIS source polls a SignalK server over
+the LAN and keeps everything in memory — which cannot work once a boat is on a mobile
+link. Enable **Upload AIS targets** and the boat pushes what its own receiver hears, so
+the web app can show other boats, their heading and their trail from stored data.
+
+Only **locally received** AIS is forwarded. A boat running an internet feed such as
+`signalk-aisstream` would otherwise spend uplink bandwidth sending data the cloud can
+fetch directly from the same API — known feeds are skipped automatically. The plugin logs
+the AIS sources it sees, so you can name your own receiver in **Only this AIS source** if
+you want to be explicit.
+
+There is no radius or rate limit: a real AIS receiver is bounded by VHF line-of-sight,
+which is the honest limiter, and offshore — where this data is most valuable, because
+commercial feeds are blind there — it tends to zero. Vessel identity (name, dimensions,
+ship type) repeats every few minutes and never changes, so it is re-sent at most hourly;
+positions are never throttled.
+
+Telemetry always wins the link. AIS buffers in its **own** spool with its own cap and
+stands down completely whenever the telemetry spool has a backlog, so a busy anchorage
+can never delay or evict your own boat's data.
+
+> ⚠️ **Requires a cloud that filters history on `self`.** Every AIS row is tagged
+> `self=false`, and the cloud's Trends and track queries must filter `self == "true"`.
+> Without that, other ships' speed and heading appear in *your* charts. Leave this off
+> until the server side is in place.
+
+## Copying older history to the cloud (one-time)
+If the boat recorded into its own InfluxDB before it started syncing — a
+`signalk-to-influxdb-v2` bucket, or an imported logbook — the **backfill** copies it up
+so the cloud holds your full history. Live sync can't do this: it only ever sees deltas
+arriving now, and the spool only replays what it captured itself while offline.
+
+Fill the **Copy older history to the cloud** section and save. It walks backwards in
+one-hour windows (newest first, so recent history lands first), resumes after a restart
+from a manifest, and stands aside whenever live telemetry has a backlog — the data-
+critical path is never starved by a bulk upload. Progress shows in the status line.
+
+It needs a **cloud read+write token**, not the write token from signup. Every hour it
+uploads is verified by counting the destination, and a write-only token cannot read. A
+`204` means InfluxDB accepted the bytes, not that every point landed — without the count
+a partial write would be marked done and lost. The token is only needed while the
+backfill runs: **revoke it afterwards**, live sync is unaffected.
+
+Safe to re-run. Points are keyed by (measurement, tagset, nanosecond timestamp), so an
+identical point overwrites rather than duplicating — an interrupted migration is simply
+run again.
+
+**Only this boat's data is copied**, and that is not configurable. The cloud's history
+queries assume your bucket holds one vessel, so uploading an archive's AIS would put
+other ships into your own SOG and heading charts. If the source holds several contexts
+the plugin copies yours and logs which it skipped. If it holds exactly **one** context it
+is copied whatever identity string it uses — a bucket with one vessel cannot be an AIS
+collection, and this is what lets an archive recorded under an older Signal K UUID (or an
+MMSI URN) still migrate. Hand-edit `backfill.context` to force a specific one.
+
+A run that copies **zero** points is reported as a problem, not as success: that almost
+always means the org or bucket is wrong rather than that the archive is empty.
+
+**It starts below what live sync already covers.** The destination's own oldest point is
+the moment cloud sync began, so the walk begins there rather than at *now*. Without that,
+a source archive that is still being written — a `signalk-to-influxdb-v2` bucket still
+recording — makes the first windows re-upload today's data. The timestamps are correct,
+but it is data the cloud already has, and a lot of wasted uplink.
+
+**Dense archives are subdivided.** A window is read whole and converted in memory, and a
+busy boat can produce millions of points an hour (54M/day was measured on a real boat —
+about 400 MB of CSV per hour). When a window holds more than `maxRowsPerChunk` points it
+is halved until it fits, down to a one-minute floor. The count is already known before
+the read, so this costs nothing extra.
 
 **True wind comes from your instruments.** If the boat publishes
 `environment.wind.speedTrue` / `directionTrue`, those are stored verbatim — a wind
@@ -190,14 +245,14 @@ water. (Before v0.14.6 the derivation used SOG, which in 3 kt of foul tide skewe
 
 The sailkick app is deployment-agnostic about history: *central Influx in the
 cloud, in-memory ring on a DB-less edge*. The boat is a third case — an edge that
-serves the app's history endpoints from its **own** data (local InfluxDB or the ring):
+serves the app's history endpoints from its **own** live data:
 ```
 GET /api/history/series?window=3600s&every=30s -> { series: { sog|heading|tws|… : [[tMs,val],…] } }
 GET /api/history/track?window=3600s            -> { track: [{ t, lat, lon }, …] }
 ```
 Same JSON the cloud returns, so the browser can't tell the difference — but it
-works **offline** with the boat's own data. Only when neither a local InfluxDB nor
-telemetry is available do these paths **fall through to the cloud mirror**, so an
+works **offline** with the boat's own data. Only when no telemetry source is
+available at all do these paths **fall through to the cloud mirror**, so an
 online boat is never worse off than before.
 
 ## Setup: register on the web, then paste the token
@@ -237,21 +292,16 @@ in `index.js`.
 
 - **Sailkick account**: `slug` (boat name), `writeToken`
 - **Telemetry sync → cloud**: `enabled`
+- **Upload AIS targets**: `enabled` (default off), `source`
 - **Offline app & maps**: `enabled`, `proxyPort` (default 8080), `localSignalkUrl`
   (default `http://127.0.0.1:3000`), `dataDir`, `seedEnabled`, `prefetchRadiusNm`,
   `prefetchDetailZoom`
-- **History archive** (optional): `historyToken`, `historyInfluxUrl`
-  (default `http://127.0.0.1:8086`), `historyOrg` + `historyBucket` (default `signalk`,
-  matching `signalk-to-influxdb-v2`)
 
 `dataDir` is the one storage location — cached maps, the telemetry spool and the
 history ring log all live under it. **Put it on the SSD/USB disk, not the SD card.**
 Leave it blank and each part falls back to its historical spot under the plugin data
 dir.
 
-`historyToken` is optional and only matters if the boat already runs its own InfluxDB.
-Blank — the normal case — uses the built-in DB-less ring.
-
 Cache-manifest polling is always on (no toggle): tile freshness comes from the cloud
 announcing bakes, and without it a re-baked dataset would never refresh.
 

package/index.js

@@ -6,6 +6,8 @@ const { createSync } = require('./lib/sync')
 const { createProxy } = require('./lib/proxy')
 const { createTelemetry } = require('./lib/telemetry')
 const { createHistory } = require('./lib/history')
+const { createBackfill } = require('./lib/backfill')
+const { createAis } = require('./lib/ais')
 const { resolveAccountConfig } = require('./lib/account')
 
 // sailkick-boat: one Signal K plugin, two independently-toggleable modules —
@@ -43,15 +45,8 @@ const PROXY_TUNING = { requestTimeoutMs: 20000, localPaths: ['/signalk'], teleme
 const MANIFEST = { enabled: true, path: '/api/cache-manifest', pollIntervalSec: 300 }
 const SEED_TUNING = { coastlineMaxZoom: 8, seabedMaxZoom: 6, concurrency: 4 }
 const PREFETCH_TUNING = { concurrency: 4 }
-const HISTORY_TUNING = {
-  influxUrl: 'http://127.0.0.1:8086',
-  org: 'signalk',
-  bucket: 'signalk',
-  requestTimeoutMs: 15000,
-  ringPersist: true,
-  ringWindowSec: 86400,
-  ringSampleSec: 15
-}
+const HISTORY_TUNING = { ringPersist: true, ringWindowSec: 86400, ringSampleSec: 15 }
+const BACKFILL_SRC_DEFAULTS = { influxUrl: 'http://127.0.0.1:8086', org: 'signalk', bucket: 'signalk' }
 
 // A *cloud* endpoint on a loopback or private address means telemetry never leaves the
 // LAN. On the wire that is indistinguishable from a normal offline backlog — the spool
@@ -69,13 +64,12 @@ function isPrivateHostUrl (u) {
   } catch { return false }
 }
 
-// A pre-filled schema default must not silently shadow a legacy value that differs. The
-// Signal K config UI writes every default on save, so `historyOrg: "signalk"` appears on
-// a boat whose archive has always lived in org "addiction" under the old nested
-// proxy.history block — and the archive would go quiet with no explanation. Rule: an
-// explicitly-changed field wins; otherwise a legacy value that differs from the default
-// wins; otherwise the default.
-function pickHistoryField (flat, legacy, dflt) {
+// The Signal K config UI writes every schema default on save, so a freshly pre-filled
+// `backfillOrg: "signalk"` can appear on a boat whose archive has always lived in org
+// "addiction" under the pre-0.15 `proxy.history` block — and the backfill would read an
+// empty bucket. Rule: an explicitly-changed field wins; else a legacy value that differs
+// from the default wins; else the default.
+function pickField (flat, legacy, dflt) {
   const f = String(flat == null ? '' : flat).trim()
   const l = String(legacy == null ? '' : legacy).trim()
   if (f && f !== dflt) return { value: f, from: 'config' }
@@ -98,6 +92,8 @@ module.exports = function (app) {
   let proxy = null
   let telemetry = null
   let history = null
+  let backfill = null
+  let ais = null
   let statusTimer = null
   let accountStatus = null
   let syncWarning = null
@@ -130,6 +126,28 @@ module.exports = function (app) {
           enabled: { type: 'boolean', title: 'Enable telemetry sync', default: true }
         }
       },
+      ais: {
+        type: 'object',
+        title: 'Upload AIS targets',
+        description: 'Send the AIS this boat\'s own receiver hears to the cloud, so the web app can show other vessels, their heading and their trail. Only locally received AIS is forwarded — an internet feed such as signalk-aisstream is skipped, since the cloud can fetch that itself without spending your uplink. REQUIRES a cloud that filters history on self; until then leave this off or the owner\'s SOG and heading charts will pick up other ships.',
+        properties: {
+          enabled: { type: 'boolean', title: 'Upload AIS targets', default: false },
+          source: { type: 'string', title: 'Only this AIS source', description: 'Blank forwards every source except known internet feeds. The plugin logs the AIS sources it sees — copy the one for your own receiver here if you want to be explicit.' }
+        }
+      },
+      backfill: {
+        type: 'object',
+        title: 'Copy older history to the cloud (one-time)',
+        description: 'If this boat recorded data into its own InfluxDB before it started syncing — a signalk-to-influxdb-v2 bucket, or an imported logbook — this copies it up so the app can chart it. It runs in the background, resumes after a restart, verifies every hour it uploads, and stands aside whenever live telemetry is behind. Safe to re-run: identical points overwrite rather than duplicate.',
+        properties: {
+          enabled: { type: 'boolean', title: 'Run the backfill', default: false },
+          cloudToken: { type: 'string', title: 'Cloud read+write token', description: 'A token with READ and WRITE on your cloud bucket. Your normal write token cannot read, and reading is how each uploaded hour is verified. Needed only while the backfill runs — revoke it afterwards; live sync is unaffected.' },
+          sourceToken: { type: 'string', title: 'Local InfluxDB read token' },
+          sourceUrl: { type: 'string', title: '…local InfluxDB URL', default: 'http://127.0.0.1:8086' },
+          sourceOrg: { type: 'string', title: '…organization', default: 'signalk' },
+          sourceBucket: { type: 'string', title: '…bucket', default: 'signalk' }
+        }
+      },
       proxy: {
         type: 'object',
         title: 'Offline app & maps',
@@ -154,11 +172,7 @@ module.exports = function (app) {
             enum: [12, 13, 14, 15],
             enumNames: ['Overview (z12)', 'Coastal (z13)', 'Detailed (z14)', 'Harbor (z15)'],
             default: 13
-          },
-          historyToken: { type: 'string', title: 'History archive: read token', description: 'Leave blank for the built-in lightweight history, which is what most boats want and needs no database. Paste a read token to serve Trends and the track from a local InfluxDB instead — worth it for history recorded before this boat synced to the cloud, for more than 30 days offline, or for an imported archive. Setting a token switches the source; the three fields below say where to find it.' },
-          historyInfluxUrl: { type: 'string', title: '…InfluxDB URL', description: 'Where the archive lives. The default is an InfluxDB on this machine.', default: 'http://127.0.0.1:8086' },
-          historyOrg: { type: 'string', title: '…organization', description: 'Default matches signalk-to-influxdb-v2.', default: 'signalk' },
-          historyBucket: { type: 'string', title: '…bucket', description: 'Default matches signalk-to-influxdb-v2. An imported archive often uses its own name.', default: 'signalk' }
+          }
         }
       }
     }
@@ -237,21 +251,7 @@ module.exports = function (app) {
       const oldSeed = p.seed || {}
       const oldPrefetch = p.prefetch || {}
       const oldHistory = p.history || {}
-      // History source, resolved before the modules start so it can be reported. A token
-      // is the switch: present = query the archive, absent = the built-in ring.
-      const hUrl = pickHistoryField(p.historyInfluxUrl, oldHistory.influxUrl, HISTORY_TUNING.influxUrl)
-      const hOrg = pickHistoryField(p.historyOrg, oldHistory.org, HISTORY_TUNING.org)
-      const hBucket = pickHistoryField(p.historyBucket, oldHistory.bucket, HISTORY_TUNING.bucket)
-      const hToken = String(p.historyToken || oldHistory.token || '').trim()
-      const fromLegacy = [['URL', hUrl], ['org', hOrg], ['bucket', hBucket]].filter(([, v]) => v.from === 'legacy')
-      if (hToken) {
-        console.log(`[sailkick-boat] history -> archive ${hUrl.value} org=${hOrg.value} bucket=${hBucket.value}`)
-        if (fromLegacy.length) {
-          ;(app.error || console.error)(`[sailkick-boat] history archive ${fromLegacy.map(([n]) => n).join(' + ')} taken from the old proxy.history config, not the new field(s) — copy the value(s) up to keep them after the next save`)
-        }
-      } else {
-        console.log('[sailkick-boat] history -> built-in ring (no archive token set)')
-      }
+      console.log('[sailkick-boat] history -> live ring')
 
       if (upstream.ignored) {
         ;(app.error || console.error)(`[sailkick-boat] ignoring proxy.sailkickUrl "${upstream.ignored}" left over from an older config — mirroring ${upstream.url}. Set proxy.selfHosted:true to keep your own server.`)
@@ -284,10 +284,6 @@ module.exports = function (app) {
         history: {
           ...HISTORY_TUNING,
           enabled: oldHistory.enabled !== false,
-          influxUrl: hUrl.value,
-          org: hOrg.value,
-          bucket: hBucket.value,
-          token: hToken,
           ringPersist: oldHistory.ringPersist !== false,
           ringWindowSec: oldHistory.ringWindowSec || HISTORY_TUNING.ringWindowSec,
           ringSampleSec: oldHistory.ringSampleSec || HISTORY_TUNING.ringSampleSec,
@@ -328,6 +324,71 @@ module.exports = function (app) {
       }
     }
 
+    // --- AIS upload (isolated; yields to telemetry, its own spool) ---
+    const aisOpts = opts.ais || {}
+    if (aisOpts.enabled === true) {
+      if (!b.writeToken) {
+        (app.error || console.error)('[sailkick-boat] AIS upload needs a paired account for its destination bucket — skipped')
+      } else {
+        try {
+          ais = createAis(app, {
+            influxUrl: influx.url,
+            org: b.org || SYNC_TUNING.org,
+            bucket: b.bucket,
+            token: b.writeToken,
+            source: String(aisOpts.source || '').trim() || null,
+            spoolDir: store ? path.join(store, 'ais-spool') : undefined,
+            pending: sync ? sync.pending : null
+          })
+          ais.start()
+          console.log(`[sailkick-boat] ais -> ${influx.url} bucket=${b.bucket}${aisOpts.source ? ' source=' + aisOpts.source : ''}`)
+        } catch (e) {
+          (app.error || console.error)('[sailkick-boat] AIS start failed: ' + e.message)
+          ais = null
+        }
+      }
+    }
+
+    // --- backfill (best-effort, isolated: it must never disturb sync or the proxy) ---
+    const bf = opts.backfill || {}
+    const oldHist = (opts.proxy || {}).history || {}
+    if (bf.enabled === true) {
+      // Pre-0.15 installs kept the archive's coordinates in proxy.history.* — reuse them
+      // so nobody retypes an org/bucket the plugin already knows.
+      const src = {
+        url: pickField(bf.sourceUrl, oldHist.influxUrl, BACKFILL_SRC_DEFAULTS.influxUrl).value,
+        org: pickField(bf.sourceOrg, oldHist.org, BACKFILL_SRC_DEFAULTS.org).value,
+        bucket: pickField(bf.sourceBucket, oldHist.bucket, BACKFILL_SRC_DEFAULTS.bucket).value,
+        token: String(bf.sourceToken || oldHist.token || '').trim()
+      }
+      const dst = { url: influx.url, org: b.org || SYNC_TUNING.org, bucket: b.bucket, token: String(bf.cloudToken || '').trim() }
+      if (!b.writeToken) {
+        (app.error || console.error)('[sailkick-boat] backfill needs a paired account for its destination bucket — skipped')
+      } else if (!dst.token) {
+        (app.error || console.error)('[sailkick-boat] backfill needs a cloud READ+WRITE token: every uploaded hour is verified by counting the destination, which the write-only sync token cannot do — skipped')
+      } else if (!src.token) {
+        (app.error || console.error)('[sailkick-boat] backfill needs a read token for the local InfluxDB — skipped')
+      } else {
+        try {
+          backfill = createBackfill(app, {
+            src,
+            dst,
+            // Never a config option: the plugin must not upload data that is not this
+            // boat's, and the cloud's history queries depend on that holding.
+            selfContext: app.selfContext || ('vessels.' + (app.selfId || 'self')),
+            context: String(bf.context || '').trim() || null, // hand-edit escape hatch
+            startBound: bf.startBound,
+            stateFile: path.join((app.getDataDirPath && app.getDataDirPath()) || '.', 'backfill.json'),
+            pending: sync ? sync.pending : null
+          })
+          backfill.start()
+        } catch (e) {
+          (app.error || console.error)('[sailkick-boat] backfill start failed: ' + e.message)
+          backfill = null
+        }
+      }
+    }
+
     statusTimer = setInterval(updateStatus, 5000)
     updateStatus()
   }
@@ -341,6 +402,8 @@ module.exports = function (app) {
     if (proxy) parts.push(proxy.status())
     if (telemetry) parts.push(telemetry.status())
     if (history) parts.push(history.status())
+    if (ais) parts.push(ais.status())
+    if (backfill) parts.push(backfill.status())
     try { app.setPluginStatus(parts.join('   |   ') || 'idle (both features off)') } catch {}
   }
 
@@ -350,10 +413,14 @@ module.exports = function (app) {
     try { if (sync) sync.stop() } catch {}
     try { if (telemetry) telemetry.stop() } catch {}
     try { if (history) history.stop() } catch {}
+    try { if (ais) ais.stop() } catch {}
+    try { if (backfill) backfill.stop() } catch {}
     try { if (proxy) proxy.stop() } catch {}
     sync = null
     telemetry = null
     history = null
+    backfill = null
+    ais = null
     proxy = null
     accountStatus = null
     syncWarning = null