sailkick-boat 0.20.1 → 0.21.1

package/README.md

@@ -52,8 +52,13 @@ files to send later — nothing is lost in the gap.
 
 - Network errors, `429` and `5xx` are retried with backoff (1 s → 60 s); the data stays
   on disk.
-- A `4xx` is quarantined to `spool/dead/` rather than retried forever, because a
-  malformed or unauthorised batch would otherwise wedge the queue behind it.
+- A malformed batch (`4xx` other than the three below) is quarantined to `spool/dead/`
+  rather than retried forever, because it would otherwise wedge the queue behind it.
+- **`404`, `401` and `403` are held, not quarantined.** A missing bucket or a rejected
+  token rejects *every* batch identically, so quarantining would pour the whole telemetry
+  stream into `spool/dead/` while the plugin looked busy. Instead the spool is left
+  intact, the status line names the setting to fix, and retries continue slowly so a
+  correction recovers on its own.
 - The buffer is bounded (500 MB). On overflow the **oldest** files are dropped and
   logged — a long-offline boat fills its own disk otherwise.
 - Timestamps are nanosecond-precise, so replaying after a reconnect overwrites rather
@@ -302,9 +307,15 @@ arriving now, and the spool only replays what it captured itself while offline.
 
 Fill the **Copy older history to the cloud** section and save. It walks backwards in
 one-hour windows (newest first, so recent history lands first), resumes after a restart
-from a manifest, and stands aside whenever live telemetry has a backlog — the data-
+from a manifest, and stands aside whenever live telemetry has a **backlog** — the data-
 critical path is never starved by a bulk upload. Progress shows in the status line.
 
+"Backlog" means several files waiting, not merely an upload in flight. Requiring an empty
+spool was a race: live sync flushes every second, so once the round trip to the cloud grew
+past a second the spool was never empty for an instant and the backfill stood down for
+ever — silently, since that path logs nothing. It now yields at a real backlog, which an
+outage produces within a minute, and logs both the stand-down and the resume.
+
 It needs a **cloud read+write token**, not the write token from signup. Every hour it
 uploads is verified by counting the destination, and a write-only token cannot read. A
 `204` means InfluxDB accepted the bytes, not that every point landed — without the count
@@ -428,12 +439,24 @@ online boat is never worse off than before.
    is needed — see above). The signup screen shows your boat's ingest credentials.
 2. **Copy the "Write token"** — it is shown **once** and cannot be recovered. If you
    lose it you need a new one minted.
-3. In Plugin Config, fill the **Sailkick account** section — **boat name + write
-   token** — and save.
-
-That is the whole handshake. The plugin resolves everything else locally (`bucket` =
-`<slug>_raw`, `org` = `sailkick`) and never calls the app for configuration, so setup
-works with no internet and there is nothing to re-fetch after a restart.
+3. In Plugin Config, fill the **Sailkick account** section — **write token**, plus
+   either the **boat name** or the **data bucket** — and save.
+
+That is the whole handshake. The plugin resolves everything else locally (`org` =
+`sailkick`) and never calls the app for configuration, so setup works with no internet
+and there is nothing to re-fetch after a restart.
+
+**Data bucket.** Leave it blank and the bucket is derived as `<boat name>_raw`, which is
+what older accounts use. Newer accounts are identified by a UUID instead — their bucket
+looks like `1fcad258-c422-4e93-a6f9-6811938499f6_raw` — so paste that into **Data
+bucket** and the boat name becomes optional. If your bucket is ever renamed, this is the
+one field to change: the write token survives a rename, because InfluxDB scopes tokens by
+bucket *id* rather than by name.
+
+A bucket that does not exist is treated as a **configuration error, not bad data**:
+telemetry is held on disk, nothing is quarantined, the status line reads
+`sync: HELD — bucket "…" not found`, and it sends itself as soon as the name is
+corrected — no restart needed. The same applies to a rejected token.
 
 > **Upgrading from before 0.14?** Old versions had visible `influxUrl` / `sailkickUrl`
 > fields. Whatever was typed into them is still in your saved config, and since 0.14.4

package/index.js

@@ -121,8 +121,9 @@ module.exports = function (app) {
         title: 'Sailkick account',
         description: 'Register your boat at www.sailkick.io first — the signup screen shows a write token. Paste it here together with your boat name. Keep that token safe: it is shown only once and cannot be recovered.',
         properties: {
-          slug: { type: 'string', title: 'Boat name', description: 'Exactly as registered on the website.' },
-          writeToken: { type: 'string', title: 'Write token', description: 'From the signup screen ("Write token"). Everything else — Influx URL, organization, bucket — is derived from it and your boat name.' }
+          slug: { type: 'string', title: 'Boat name', description: 'Exactly as registered on the website. Newer accounts are identified by a data bucket instead — if you have one, fill that in below and leave this blank.' },
+          writeToken: { type: 'string', title: 'Write token', description: 'From the signup screen ("Write token"). The Influx URL and organization are fleet-wide and set for you.' },
+          bucket: { type: 'string', title: 'Data bucket', description: 'Leave blank to derive it from the boat name. Set it when your account has an explicit bucket — newer ones are a UUID, e.g. 1fcad258-c422-4e93-a6f9-6811938499f6_raw. Copy it exactly; a bucket that does not exist stops sync rather than losing data, and the status line will say so.' }
         }
       },
       sync: {
@@ -191,7 +192,9 @@ module.exports = function (app) {
       // Purely local: the boat registers on the website, the owner pastes the write
       // token here. Nothing to fetch, so nothing that can fail while offline.
       const r = resolveAccountConfig(app, opts.account)
-      if (r.bundle) accountStatus = `account: ${r.bundle.slug}`
+      // Name the BUCKET, not just the boat: a bucket that no longer exists (a rename, a
+      // typo) is the one misconfiguration that stops sync dead, and it was invisible here.
+      if (r.bundle) accountStatus = `account: ${r.bundle.slug || '(no name)'} -> ${r.bundle.bucket}`
       else if (r.error) accountStatus = `account: ${r.error}`
       else accountStatus = 'account: not configured — register at www.sailkick.io, then paste your write token'
       startModules(opts, r.bundle)

package/lib/account/index.js

@@ -8,10 +8,19 @@
 // is the whole handshake. The plugin deliberately does NOT sign up on its own —
 // one registration path means no half-created accounts and no spent-invite dead ends.
 //
-// Everything else about the bundle is derivable, so only two fields are asked for:
+// Everything else about the bundle WAS derivable, so only two fields used to be asked for:
 //   bucket = "<slug>_raw"   (the app's convention — server/auth/registry.js bucketFor)
 //   org    = "sailkick"     (single org, fleet-wide)
 //
+// That derivation is no longer reliable. The fleet is moving boat identity from a slug to
+// a UUID, and buckets are being renamed with it — this boat went from "addiction_raw" to
+// "1fcad258-c422-4e93-a6f9-6811938499f6_raw" while keeping the same token, because
+// InfluxDB scopes tokens by bucket ID and a rename preserves it. So `bucket` is now an
+// explicit field: blank keeps the old derivation (every pre-existing install is
+// untouched), set wins verbatim. The bucket is the actual contract with InfluxDB, so it
+// is the thing worth being able to state directly — it survives whatever identity the
+// fleet settles on.
+//
 // This bundle deliberately carries NO Influx URL. "Which boat am I" and "where is the
 // cloud" are separate questions, and the second one has a single fleet-wide answer that
 // index.js owns. Letting a pasted or cached value set the host is how telemetry ends up
@@ -56,15 +65,20 @@ const clean = (v) => String(v == null ? '' : v).trim()
 function resolveAccountConfig (app, account) {
   const acct = account || {}
   const slug = clean(acct.slug).toLowerCase()
+  // NOT lowercased: a bucket name is taken verbatim from whatever the server called it.
+  const bucket = clean(acct.bucket)
   const writeToken = clean(acct.writeToken)
 
-  if (slug && writeToken) {
+  // Either identifier will do. A boat given an explicit bucket needs no slug at all —
+  // newer accounts are issued a UUID bucket and may never see a slug — while an older
+  // install that only has a boat name keeps deriving as before.
+  if (writeToken && (slug || bucket)) {
     return {
       source: 'config',
       bundle: {
         slug,
         org: clean(acct.org) || DEFAULT_ORG,
-        bucket: clean(acct.bucket) || `${slug}_raw`,
+        bucket: bucket || `${slug}_raw`,
         writeToken
       }
     }
@@ -74,8 +88,8 @@ function resolveAccountConfig (app, account) {
   if (cached) return { bundle: cached, source: 'cache' }
 
   // Half-filled: say which half, so the status line is actionable.
-  if (writeToken) return { bundle: null, source: null, error: 'boat name is missing' }
-  if (slug) return { bundle: null, source: null, error: 'write token is missing' }
+  if (writeToken) return { bundle: null, source: null, error: 'boat name or data bucket is missing' }
+  if (slug || bucket) return { bundle: null, source: null, error: 'write token is missing' }
   return { bundle: null, source: null }
 }
 

package/lib/ais/index.js

@@ -205,7 +205,12 @@ function createAis (app, options) {
         if (!body.trim()) { await state.spool.remove(file); continue }
         const res = await writeLines({ influxUrl: cfg.influxUrl, org: cfg.org, bucket: cfg.bucket, token: cfg.token, timeoutMs: cfg.requestTimeoutMs }, body)
         if (res.ok) {
-          await state.spool.remove(file); state.lastOkAt = new Date(); state.backoff = cfg.retryMinMs
+          await state.spool.remove(file); state.lastOkAt = new Date(); state.backoff = cfg.retryMinMs; state.configError = null
+        } else if (res.configError) {
+          // Same rule as telemetry: a missing bucket or bad token rejects everything, so
+          // hold rather than quarantine and let a corrected config drain the spool.
+          state.configError = res.status === 404 ? `bucket "${cfg.bucket}" not found` : `token rejected (HTTP ${res.status})`
+          state.pumping = false; scheduleRetry(res); return
         } else if (res.retryable) {
           state.pumping = false; scheduleRetry(res); return
         } else {
@@ -238,6 +243,10 @@ function createAis (app, options) {
   }
   function refreshStatus (suffix) {
     if (!state) return
+    if (state.configError) {
+      state.statusLine = `ais: HELD — ${state.configError}; ${state.forwarded} point(s) on disk`
+      return
+    }
     state.statusLine = `ais: ${state.targets.size} target(s), ${state.forwarded} point(s)${state.dropped ? `, ${state.dropped} skipped` : ''}${suffix ? '; ' + suffix : ''}`
   }
 

package/lib/backfill/index.js

@@ -41,6 +41,14 @@ const DEFAULTS = {
   minWindowMs: 60000, // never subdivide below a minute
   idleMs: 250, // pending() is the real backpressure; this is just politeness
   backlogWaitMs: 15000, // how long to stand down when live sync has a backlog
+  // Stand down at a genuine BACKLOG, not at any activity. Requiring an EMPTY spool was a
+  // race rather than a design: live sync flushes every second, and a file exists for as
+  // long as its upload is in flight. Once the round trip to the cloud grew past the flush
+  // interval — which it did — the spool stopped being empty even for an instant (measured
+  // on the boat: non-empty in 30 of 30 one-second samples) and the backfill parked for
+  // ever, silently, because that path logs nothing. A real backlog is unmistakable by
+  // comparison: an outage puts hundreds of files on disk within a minute.
+  backlogFiles: 5,
   maxErrorStreak: 5,
   queryTimeoutMs: 120000
 }
@@ -227,7 +235,11 @@ function createBackfill (app, options) {
       const res = await writeLines({ influxUrl: cfg.dst.url, org: cfg.dst.org, bucket: cfg.dst.bucket, token: cfg.dst.token, timeoutMs: cfg.queryTimeoutMs }, body)
       if (!res.ok) {
         // 4xx is fatal: bad credentials or malformed data, and retrying cannot fix it.
-        if (!res.retryable) { warn(`write REJECTED (HTTP ${res.status}) — ${res.body ? String(res.body).slice(0, 160) : 'no detail'}`); return 'fatal' }
+        if (!res.retryable) {
+          // Name the destination: after a bucket rename this is the first thing to check.
+          warn(`write to ${cfg.dst.bucket} REJECTED (HTTP ${res.status}) — ${res.body ? String(res.body).slice(0, 160) : 'no detail'}`)
+          return 'fatal'
+        }
         warn(`write ${iso(startMs)} failed — ${res.status ? 'HTTP ' + res.status : 'unreachable'}; will retry`)
         return 'retry'
       }
@@ -290,14 +302,18 @@ function createBackfill (app, options) {
         const key = iso(startMs)
         if (state.done[key]) continue
 
-        // Live telemetry is data-critical; this is not. Stand down while it is behind.
+        // Live telemetry is data-critical; this is not. Stand down while it is BEHIND —
+        // which means a backlog, not merely an upload in flight. See cfg.backlogFiles.
+        let waited = false
         while (!stopped && cfg.pending) {
           let depth = 0
           try { depth = (await cfg.pending()).count || 0 } catch {}
-          if (!depth) break
+          if (depth < cfg.backlogFiles) break
+          if (!waited) { waited = true; log(`standing down — live sync is behind (${depth} file(s)); telemetry has the link`) }
           statusLine = `backfill: paused — live sync backlog (${depth} file(s))`
           await sleep(cfg.backlogWaitMs)
         }
+        if (waited && !stopped) log('live sync caught up — resuming')
         if (stopped) break
 
         const tally = { written: 0 }

package/lib/sync/index.js

@@ -66,6 +66,27 @@ function createSync (app, options) {
       if (state.failing) warn(`recovered — writing to ${cfg.influxUrl} again after ${state.failCount} failed attempt(s)`)
       state.failing = false
       state.failCount = 0
+      state.configError = null
+    }
+
+    // A wrong bucket or token rejects every batch identically, so this is a standing
+    // condition rather than an outage: name the exact setting to fix, once, and keep the
+    // reason on the status line until it clears.
+    const noteConfigError = (res) => {
+      state.failCount++
+      const notFound = res.status === 404
+      state.configError = notFound
+        ? `bucket "${cfg.bucket}" not found`
+        : `write token rejected (HTTP ${res.status})`
+      const now = Date.now()
+      if (state.failing && now - state.lastWarnAt < WARN_REPEAT_MS) return
+      state.failing = true
+      state.lastWarnAt = now
+      warn(`sync HELD — ${state.configError}. ` +
+        (notFound
+          ? 'Set "Data bucket" in the plugin config to the bucket your account actually uses — newer accounts use a UUID. '
+          : 'Check the write token, and that it is valid for this bucket. ') +
+        'Telemetry is kept on disk (nothing quarantined) and will send itself once this is corrected.')
     }
 
     const handleDelta = (delta) => {
@@ -100,13 +121,21 @@ function createSync (app, options) {
           if (res.ok) {
             await spool.remove(file); state.lastOkAt = new Date(); state.backoff = state.retryMin
             noteSuccess()
+          } else if (res.configError) {
+            // The settings are wrong, not this batch — every batch would be rejected the
+            // same way. Quarantining here would pour the entire telemetry stream into
+            // spool/dead/ while the plugin looked busy. Hold everything on disk, say so
+            // once, and keep retrying slowly so a corrected bucket or token recovers
+            // without a restart.
+            noteConfigError(res)
+            state.pumping = false; scheduleRetry(res); return
           } else if (res.retryable) {
             noteFailure(res)
             state.pumping = false; scheduleRetry(res); return
           } else {
-            // 4xx: bad credentials or malformed data. Never retried, so if this is not
-            // loud the batch is simply gone.
-            warn(`batch REJECTED (HTTP ${res.status}) and quarantined to ${state.spool.deadDir} — ${res.status === 401 ? 'the write token is not valid for this bucket' : 'check bucket/org and the data'}`)
+            // Other 4xx: malformed data. Never retried, so if this is not loud the batch
+            // is simply gone.
+            warn(`batch REJECTED (HTTP ${res.status}) and quarantined to ${state.spool.deadDir} — check the data`)
             await spool.quarantine(file)
           }
         }
@@ -133,7 +162,13 @@ function createSync (app, options) {
       try {
         const { count, bytes } = await spool.stats()
         const last = state.lastOkAt ? state.lastOkAt.toISOString() : 'never'
-        state.statusLine = `sync: buffered ${count}f/${Math.round(bytes / 1024)}KB; last ok ${last}${suffix ? '; ' + suffix : ''}`
+        // A config error is the headline — it needs the owner, not just time.
+        const head = state.configError
+          ? `sync: HELD — ${state.configError}; fix the plugin config`
+          : `sync: buffered ${count}f/${Math.round(bytes / 1024)}KB; last ok ${last}`
+        state.statusLine = state.configError
+          ? `${head}; ${count}f/${Math.round(bytes / 1024)}KB held on disk`
+          : `${head}${suffix ? '; ' + suffix : ''}`
       } catch {}
     }
 

package/lib/sync/influxWrite.js

@@ -4,12 +4,22 @@
 //
 // Return shape:
 //   { ok: true, status: 204 }
-//   { ok: false, retryable: true,  networkError: true }   connection failed
-//   { ok: false, retryable: true,  status }                429 / 5xx (transient)
-//   { ok: false, retryable: false, status, body }          4xx (bad data/auth)
+//   { ok: false, retryable: true,  networkError: true }        connection failed
+//   { ok: false, retryable: true,  status }                     429 / 5xx (transient)
+//   { ok: false, configError: true, status, body }              404 / 401 / 403
+//   { ok: false, retryable: false, status, body }               other 4xx (bad data)
 //
-// 4xx is non-retryable: retrying a malformed or unauthorized batch forever would
-// wedge the queue, so the caller quarantines it instead.
+// Most 4xx is non-retryable: retrying a malformed batch forever would wedge the queue,
+// so the caller quarantines it instead.
+//
+// 404/401/403 are different in kind and must NOT be quarantined. They say "your settings
+// are wrong", not "this batch is bad" — a missing bucket or a bad token rejects EVERY
+// batch equally, so quarantining would feed the whole telemetry stream into spool/dead/
+// while looking like normal operation. That is not hypothetical: the fleet renamed this
+// boat's bucket from `addiction_raw` to a UUID, and a plugin left running through it
+// would have shredded every point it collected. The caller holds these on disk instead
+// and keeps retrying slowly, so correcting the config recovers on its own.
+const CONFIG_ERROR_STATUS = new Set([401, 403, 404])
 
 const zlib = require('zlib')
 
@@ -42,6 +52,9 @@ async function writeLines (cfg, body) {
 
   let text = ''
   try { text = await res.text() } catch {}
+  if (CONFIG_ERROR_STATUS.has(res.status)) {
+    return { ok: false, configError: true, retryable: false, status: res.status, body: text }
+  }
   const retryable = res.status === 429 || res.status >= 500
   return { ok: false, retryable, status: res.status, body: text }
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sailkick-boat",
-  "version": "0.20.1",
+  "version": "0.21.1",
   "description": "Run the sailkick app on board with no internet: charts, weather, climatology, trends and AIS all served from the boat itself. With a sailkick account it also syncs your metrics to the cloud in real time. Alpha, invite-only \u2014 info@sailkick.io",
   "main": "index.js",
   "scripts": {
@@ -9,6 +9,9 @@
   "keywords": [
     "signalk-node-server-plugin",
     "signalk-webapp",
+    "signalk-category-chart-plotters",
+    "signalk-category-instruments",
+    "signalk-category-ais",
     "signalk",
     "sailkick",
     "influxdb",
@@ -17,6 +20,10 @@
     "proxy"
   ],
   "signalk-plugin-enabled-by-default": true,
+  "signalk": {
+    "appIcon": "./icon.png",
+    "displayName": "Sailkick"
+  },
   "author": "lauchat",
   "license": "MIT",
   "repository": {

package/public/index.html

@@ -27,7 +27,8 @@
   body { margin:0; padding:2rem 1rem; background:var(--bg); color:var(--fg);
          font:16px/1.5 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif }
   main { max-width:44rem; margin:0 auto }
-  h1 { margin:0 0 .25rem; font-size:1.5rem; letter-spacing:-.01em }
+  h1 { margin:0 0 .25rem; font-size:1.5rem; letter-spacing:-.01em; display:flex; align-items:center; gap:.5rem }
+  h1 img { border-radius:.35rem }
   .sub { color:var(--mut); margin:0 0 1.5rem }
   .grid { display:grid; gap:1rem; grid-template-columns:repeat(auto-fit,minmax(15rem,1fr)) }
   a.card { display:block; padding:1.1rem 1.2rem; background:var(--card); border:1px solid var(--line);
@@ -44,7 +45,7 @@
 </head>
 <body>
 <main>
-  <h1>Sailkick</h1>
+  <h1><img src="icon.png" alt="" width="28" height="28">Sailkick</h1>
   <p class="sub">Charts, instruments and AIS, served from this boat.</p>
 
   <div class="grid">