sailkick-boat 0.14.3 → 0.14.5

package/README.md

@@ -63,10 +63,19 @@ comes from the cloud **announcing bakes**, not from a clock:
 `X-Sailkick-Cache` reports `HIT` / `MISS` / `UPDATED` / `STALE` / `LIVE` per response.
 
 **Static vs dynamic — two strategies.** Tiles and app assets are **cache-first**
-(pinned, offline forever). Dynamic `/api/*` data (AIS, weather) is **network-first**:
-fetched live every time online (so AIS/weather never go stale), with the last
-response kept only as an **offline fallback** (`STALE`). Local `/api/history` and the
-patched `/api/config` are served specially (above).
+(pinned, offline forever). Dynamic `/api/*` data (AIS, weather, lightning) is
+**network-first**: fetched live every time online (so it never goes stale), with the
+last response kept only as an **offline fallback** (`STALE`). Local `/api/history` and
+the patched `/api/config` are served specially (above).
+
+**Exception — velocity tiles are pinned.** The app serves its wind/current field from
+`/api/velocity/tiles/<layer>/<runId>/<z>/<x>/<y>/<hour>.f32`. The forecast **run id is
+in the path**, so a URL's bytes never change — a new run means new URLs. Those are
+cache-first like map tiles: each tile downloads once instead of on every pan, and the
+wind particles + storm field keep rendering **offline**. The velocity *manifest*
+(`?run=latest`) stays network-first, or the boat would pin itself to a stale run
+forever. Nothing prefetches velocity tiles yet, so offline wind covers only what you
+have panned over, and old runs are not pruned.
 
 **Offline circuit breaker.** Some boat routers return errors (or hang) for outbound
 requests when the uplink is down, instead of failing cleanly. Once a fetch fails, the
@@ -171,6 +180,12 @@ That is the whole handshake. The plugin resolves everything else locally (`bucke
 `<slug>_raw`, `org` = `sailkick`) and never calls the app for configuration, so setup
 works with no internet and there is nothing to re-fetch after a restart.
 
+> **Upgrading from before 0.14?** Old versions had visible `influxUrl` / `sailkickUrl`
+> fields. Whatever was typed into them is still in your saved config, and since 0.14.4
+> those leftovers are **ignored** — the plugin says so in the log and the status line
+> rather than silently obeying a dev address. To keep your own endpoints deliberately,
+> set `sync.selfHosted: true` / `proxy.selfHosted: true` in the config JSON.
+>
 > **Ignore the "Influx URL", "Organization" and "Bucket" on that screen** — they are for
 > the community `signalk-to-influxdb-v2` plugin. This plugin always writes to
 > `https://sync.sailkick.io`; the endpoint is fleet-wide and cannot be set from the UI,
@@ -231,6 +246,32 @@ a VE.Can↔Micro-C drop cable; leave the spare RJ45 unterminated) or a USB GPS/N
 gateway. Without a position source, the live boat / trends / area-download stay
 idle — energy telemetry still syncs to the cloud.
 
+## Troubleshooting: is telemetry actually leaving the boat?
+The plugin logs one unconditional line at startup naming its real target:
+```
+[sailkick-boat] sync -> https://sync.sailkick.io org=sailkick bucket=<slug>_raw
+```
+If that shows anything other than the endpoint you expect, a stale config field is the
+usual cause — see the upgrade note above.
+
+Failures reach the **normal server log**, not just the debug channel:
+- `cannot write to <url> — unreachable …; N failed attempt(s), telemetry is buffering
+  on disk` — once per outage, then every 5 min while it lasts, then a `recovered` line.
+- `batch REJECTED (HTTP 401) and quarantined …` — the write token is not valid for that
+  bucket. These are never retried, so they are always logged.
+- `sync: ⚠ writing to <url> — a private address` — a loopback/RFC1918 target, i.e.
+  nothing is reaching the cloud.
+
+Nothing at all in the log means the plugin never started; check it is enabled. The
+status line in Plugin Config carries the same information continuously.
+
+To prove credentials independently of the plugin:
+```bash
+curl -i -XPOST "https://sync.sailkick.io/api/v2/write?org=sailkick&bucket=<slug>_raw&precision=ns" \
+  -H "Authorization: Token $TOKEN" --data-binary "probe,context=vessels.self,self=true,source=manual value=1"
+```
+`204` = good, `401` = token not valid for that bucket.
+
 ## Dev / tests
 ```bash
 npm install && npm test   # proxy: mirror/cache/offline + Express route; sync: subscribe+buffer

package/index.js

@@ -14,10 +14,18 @@ const { resolveAccountConfig } = require('./lib/account')
 // Kept as separate modules so a proxy fault can't wedge the data-critical sync.
 
 // Fixed cloud endpoints. Boats never choose these, so they are constants rather than
-// config fields — but every module still reads its option first (see startModules), so
-// a hand-edited plugin-config JSON can override any of them for self-hosting.
+// config fields, and NOTHING in a saved config may override them unless `selfHosted` is
+// explicitly set (see resolveEndpoint).
+//
+// v0.14.3 kept `sync.influxUrl` / `proxy.sailkickUrl` as a silent escape hatch, on the
+// reasoning that a value could only get there by deliberate hand-editing. That was
+// wrong: both were VISIBLE FIELDS before v0.14.0, so every upgraded install still has
+// whatever was typed into them — and since they are no longer in the schema, the owner
+// can neither see nor clear them from the UI. One boat spent a day POSTing valid cloud
+// credentials at a dev LAN address for exactly this reason. Stale wins over intent, so
+// leftovers are now ignored and reported rather than obeyed.
 const SAILKICK_APP_URL = 'https://www.sailkick.io' // signup + mirror upstream
-const SAILKICK_INFLUX_URL = 'https://sync.sailkick.io' // data egress; pairing overrides this
+const SAILKICK_INFLUX_URL = 'https://sync.sailkick.io' // data egress
 
 // Tuning that has a right answer. Deliberately NOT on the config page: a boat owner has
 // no basis to choose these, and a wrong value silently degrades sync or the cache.
@@ -45,17 +53,32 @@ const HISTORY_TUNING = {
   ringSampleSec: 15
 }
 
-// A *cloud* endpoint on loopback means telemetry never leaves the boat. On the wire it
-// is indistinguishable from a normal offline backlog — the spool just grows — so it has
-// to be called out explicitly. (The local history DB is legitimately on 127.0.0.1; this
-// check is only ever applied to the sync path.)
-function isLoopbackUrl (u) {
+// A *cloud* endpoint on a loopback or private address means telemetry never leaves the
+// LAN. On the wire that is indistinguishable from a normal offline backlog — the spool
+// just grows — so it has to be called out explicitly. Covers RFC1918 as well as
+// loopback: the case that actually bit was a 192.168.x dev box, which a loopback-only
+// check sailed straight past. (The local history DB is legitimately on 127.0.0.1; this
+// is only ever applied to the sync path.)
+function isPrivateHostUrl (u) {
   try {
     const h = new URL(u).hostname.replace(/^\[|\]$/g, '')
-    return h === 'localhost' || h === '::1' || /^127\./.test(h)
+    if (h === 'localhost' || h === '::1' || /^127\./.test(h)) return true
+    if (/^10\./.test(h) || /^192\.168\./.test(h)) return true
+    if (/^172\.(1[6-9]|2\d|3[01])\./.test(h)) return true
+    return false
   } catch { return false }
 }
 
+// Endpoint resolution with one rule: the constant wins unless self-hosting is declared.
+// Returns { url, ignored } — `ignored` is the stale value we refused, so the caller can
+// say so out loud instead of leaving the owner to guess why nothing arrives.
+function resolveEndpoint (configured, constant, selfHosted) {
+  const v = typeof configured === 'string' ? configured.trim() : ''
+  if (!v || v === constant) return { url: constant, ignored: null }
+  if (selfHosted) return { url: v, ignored: null }
+  return { url: constant, ignored: v }
+}
+
 module.exports = function (app) {
   let sync = null
   let proxy = null
@@ -151,17 +174,17 @@ module.exports = function (app) {
     const store = p.dataDir || p.storeDir || undefined
     const spoolDir = s.spoolDir || (p.dataDir ? path.join(p.dataDir, 'spool') : undefined)
 
+    // Self-hosting is opt-in and explicit. Without it, a saved endpoint is treated as a
+    // leftover from an older version rather than as intent.
+    const selfHosted = s.selfHosted === true || p.selfHosted === true
+    const influx = resolveEndpoint(s.influxUrl, SAILKICK_INFLUX_URL, selfHosted)
+    const upstream = resolveEndpoint(p.sailkickUrl, SAILKICK_APP_URL, selfHosted)
+
     // --- sync module (isolated: its failure must not affect the proxy) ---
-    // Pairing wins over hand-written fields; hand-written fields win over the constants.
     if (s.enabled !== false) {
       const syncOpts = {
         ...SYNC_TUNING,
-        // The cloud endpoint is fleet-wide and fixed. A pasted or cached bundle can
-        // never set it — the app's signup screen hands out its own http://localhost:8086,
-        // and a boat that believed it would spool to loopback forever, looking healthy.
-        // `sync.influxUrl` survives only as a self-hosting escape hatch: it is not in
-        // the schema, so it cannot arrive by pasting, only by editing the config JSON.
-        influxUrl: s.influxUrl || SAILKICK_INFLUX_URL,
+        influxUrl: influx.url,
         org: b.org || s.org || SYNC_TUNING.org,
         bucket: b.bucket || s.bucket,
         token: b.writeToken || s.token,
@@ -174,9 +197,17 @@ module.exports = function (app) {
         retryMinMs: s.retryMinMs || SYNC_TUNING.retryMinMs,
         retryMaxMs: s.retryMaxMs || SYNC_TUNING.retryMaxMs
       }
-      if (isLoopbackUrl(syncOpts.influxUrl)) {
-        syncWarning = `sync: ⚠ writing to ${syncOpts.influxUrl} — telemetry is NOT reaching the cloud`
-        ;(app.error || console.error)(`[sailkick-boat] sync target ${syncOpts.influxUrl} is a local address — telemetry will not reach the cloud`)
+      // One unconditional line naming the actual target. This is the single most useful
+      // thing in the log: it would have shown the stale 192.168.x address instantly,
+      // instead of a day spent inferring it from an absence of errors.
+      console.log(`[sailkick-boat] sync -> ${syncOpts.influxUrl} org=${syncOpts.org} bucket=${syncOpts.bucket || '(none)'}${selfHosted ? ' [self-hosted]' : ''}`)
+
+      if (influx.ignored) {
+        syncWarning = `sync: ignoring stale influxUrl ${influx.ignored} — using ${influx.url}`
+        ;(app.error || console.error)(`[sailkick-boat] ignoring sync.influxUrl "${influx.ignored}" left over from an older config — using ${influx.url}. Set sync.selfHosted:true to keep your own endpoint.`)
+      } else if (isPrivateHostUrl(syncOpts.influxUrl)) {
+        syncWarning = `sync: ⚠ writing to ${syncOpts.influxUrl} — a private address, telemetry is NOT reaching the cloud`
+        ;(app.error || console.error)(`[sailkick-boat] sync target ${syncOpts.influxUrl} is a loopback/private address — telemetry will not reach the cloud`)
       }
       try { sync = createSync(app, syncOpts); sync.start() } catch (e) {
         (app.error || console.error)('[sailkick-boat] sync start failed: ' + e.message)
@@ -189,8 +220,11 @@ module.exports = function (app) {
       const oldSeed = p.seed || {}
       const oldPrefetch = p.prefetch || {}
       const oldHistory = p.history || {}
+      if (upstream.ignored) {
+        ;(app.error || console.error)(`[sailkick-boat] ignoring proxy.sailkickUrl "${upstream.ignored}" left over from an older config — mirroring ${upstream.url}. Set proxy.selfHosted:true to keep your own server.`)
+      }
       const pOpts = {
-        sailkickUrl: p.sailkickUrl || SAILKICK_APP_URL,
+        sailkickUrl: upstream.url,
         storeDir: store,
         proxyPort: p.proxyPort == null ? 8080 : p.proxyPort,
         localSignalkUrl: p.localSignalkUrl || 'http://127.0.0.1:3000',

package/lib/proxy/index.js

@@ -18,6 +18,20 @@ const { countBboxTiles, bboxTiles, boxAround } = require('./tiles')
 // get its live data from the boat's local SignalK and its charts/weather cached.
 // Also exposes the auth'd /plugins/sailkick-boat/p/* router handlers.
 
+// Everything under /api/ is dynamic and network-first — EXCEPT paths whose URL is
+// immutable by construction. Velocity tiles embed the forecast run id
+// (/api/velocity/tiles/<layer>/<runId>/<z>/<x>/<y>/<hour>.f32, see the app's
+// public/engine/velocity-client.js), so a given URL's bytes never change: a new
+// forecast run produces new URLs. Treating them as network-first re-downloaded the
+// whole wind field on every pan while online, and left nothing usable offline —
+// pinning them is what makes wind particles and the storm field work with no uplink.
+//
+// The velocity *manifest* is deliberately NOT in here: it resolves `run=latest` and
+// must stay live, otherwise the boat pins itself to a stale forecast run forever.
+const IMMUTABLE_API_PREFIXES = ['/api/velocity/tiles/']
+const isImmutableApi = (p) => IMMUTABLE_API_PREFIXES.some((x) => p.startsWith(x))
+const isNetworkFirst = (p) => p.startsWith('/api/') && !isImmutableApi(p)
+
 function createProxy (app, options) {
   const log = (m) => (app.debug ? app.debug('[proxy] ' + m) : console.log('[sailkick-boat:proxy]', m))
   let cfg = null
@@ -138,8 +152,10 @@ function createProxy (app, options) {
       if (req.method === 'GET' || req.method === 'HEAD') {
         const { buffer, contentType, cacheState } = await getResource({
           storeDir: cfg.storeDir, upstream: cfg.upstream, reqPath: req.url, timeoutMs: cfg.timeoutMs,
-          invalidatedAt: manifest ? manifest.invalidatedAtFor(req.url) : 0,
-          networkFirst: req.url.startsWith('/api/'), // live data (AIS, weather): fresh online, cache-fallback offline
+          // An immutable URL needs no invalidation: a bake announcement can't change
+          // bytes that are keyed by run id, so don't let an app rebuild churn them.
+          invalidatedAt: isImmutableApi(req.url) ? 0 : (manifest ? manifest.invalidatedAtFor(req.url) : 0),
+          networkFirst: isNetworkFirst(req.url), // live data (AIS, weather, lightning): fresh online, cache-fallback offline
           health
         })
         res.setHeader('Content-Type', contentType)
@@ -242,8 +258,8 @@ function createProxy (app, options) {
     try {
       const { buffer, contentType, cacheState } = await getResource({
         storeDir: cfg.storeDir, upstream: cfg.upstream, reqPath, timeoutMs: cfg.timeoutMs,
-        invalidatedAt: manifest ? manifest.invalidatedAtFor(reqPath) : 0,
-        networkFirst: reqPath.startsWith('/api/'), health
+        invalidatedAt: isImmutableApi(reqPath) ? 0 : (manifest ? manifest.invalidatedAtFor(reqPath) : 0),
+        networkFirst: isNetworkFirst(reqPath), health
       })
       res.set('Content-Type', contentType)
       res.set('X-Sailkick-Cache', cacheState)
@@ -390,4 +406,4 @@ function createProxy (app, options) {
   return { start, stop, status, handleGet, handlePrefetch, handlePrefetchRegion, handleClear, _serveMirror: serveMirror, _serveConfig: serveConfig, _manifest: () => manifest, _seeder: () => seeder, _startAreaPrefetch: startAreaPrefetch, _area: () => area }
 }
 
-module.exports = { createProxy }
+module.exports = { createProxy, isNetworkFirst, isImmutableApi }

package/lib/sync/index.js

@@ -13,6 +13,12 @@ const { deltaToLines } = require('./lineprotocol')
 
 function createSync (app, options) {
   const log = (m) => (app.debug ? app.debug('[sync] ' + m) : console.log('[sailkick-boat:sync]', m))
+  // Anything that stops telemetry reaching the cloud goes to `warn`, which lands in the
+  // normal server log. `log` is behind the plugin's debug key and is for detail only —
+  // an outage must never be visible ONLY in the status line, which is what made a
+  // day-long silent failure possible.
+  const warn = (m) => (app.error ? app.error('[sailkick-boat:sync] ' + m) : console.error('[sailkick-boat:sync]', m))
+  const WARN_REPEAT_MS = 300000 // re-state an ongoing outage every 5 min, not every retry
   let state = null
 
   function start () {
@@ -24,8 +30,9 @@ function createSync (app, options) {
       timeoutMs: options.requestTimeoutMs || 30000
     }
     if (!cfg.influxUrl || !cfg.bucket || !cfg.token) {
-      log('not started — missing influxUrl / bucket / token')
-      state = { statusLine: 'sync: not configured', stopped: true }
+      const missing = [!cfg.influxUrl && 'influxUrl', !cfg.bucket && 'bucket', !cfg.token && 'write token'].filter(Boolean)
+      warn(`not started — missing ${missing.join(' + ')}. Telemetry is NOT being sent.`)
+      state = { statusLine: `sync: not configured (missing ${missing.join(', ')})`, stopped: true }
       return
     }
 
@@ -39,7 +46,26 @@ function createSync (app, options) {
       flushTimer: null, pumpTimer: null, stopped: false, unsubscribes: [],
       retryMin: options.retryMinMs || 1000, retryMax: options.retryMaxMs || 60000,
       backoff: options.retryMinMs || 1000, idlePoll: Math.max(500, options.flushIntervalMs || 1000),
-      lastOkAt: null, pumping: false, statusLine: 'sync: starting'
+      lastOkAt: null, pumping: false, statusLine: 'sync: starting',
+      failing: false, failCount: 0, lastWarnAt: 0
+    }
+
+    // Outage transitions, not per-retry spam: the first failure, then at most one line
+    // every WARN_REPEAT_MS while it persists, then a recovery line. A boat that can
+    // never reach its endpoint used to produce no log output at all.
+    const noteFailure = (res) => {
+      state.failCount++
+      const now = Date.now()
+      if (state.failing && now - state.lastWarnAt < WARN_REPEAT_MS) return
+      state.failing = true
+      state.lastWarnAt = now
+      const why = res && res.status ? `HTTP ${res.status}` : `unreachable${res && res.error ? ` (${res.error})` : ''}`
+      warn(`cannot write to ${cfg.influxUrl} — ${why}; ${state.failCount} failed attempt(s), telemetry is buffering on disk`)
+    }
+    const noteSuccess = () => {
+      if (state.failing) warn(`recovered — writing to ${cfg.influxUrl} again after ${state.failCount} failed attempt(s)`)
+      state.failing = false
+      state.failCount = 0
     }
 
     const handleDelta = (delta) => {
@@ -73,10 +99,15 @@ function createSync (app, options) {
           const res = await writeLines(cfg, body)
           if (res.ok) {
             await spool.remove(file); state.lastOkAt = new Date(); state.backoff = state.retryMin
+            noteSuccess()
           } else if (res.retryable) {
+            noteFailure(res)
             state.pumping = false; scheduleRetry(res); return
           } else {
-            log(`batch rejected (HTTP ${res.status}) — quarantined`); await spool.quarantine(file)
+            // 4xx: bad credentials or malformed data. Never retried, so if this is not
+            // loud the batch is simply gone.
+            warn(`batch REJECTED (HTTP ${res.status}) and quarantined to ${state.spool.deadDir} — ${res.status === 401 ? 'the write token is not valid for this bucket' : 'check bucket/org and the data'}`)
+            await spool.quarantine(file)
           }
         }
       } catch (e) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sailkick-boat",
-  "version": "0.14.3",
+  "version": "0.14.5",
   "description": "EARLY ALPHA — cloud telemetry + offline maps for sailkick boats (www.sailkick.io; register on the web, paste the write token). Gapless boat→cloud telemetry sync to InfluxDB, and a local proxy that keeps the sailkick app and its charts/maps working fully offline on board.",
   "main": "index.js",
   "scripts": {