@meri-imperiumi/signalk-dead-reckoning 0.13.1 → 0.13.3

package/CHANGELOG.md

@@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.13.3] - 2026-10-15
+
+### Changed
+
+- **The state flush no longer writes rows that did not change, and WAL checkpoint bursts are bounded** (`plugin/db.js`; found by file-activity profiling of a boat server, which showed the DR store's WAL parked at the 4 MB auto-checkpoint high-water and ~40 MB/day of flash writes at the 60 s flush cadence): `setState()` now reads the stored value and skips the write when it is identical — while anchored most flush scalars (`dr_log_nm`, `dr_trip_log_nm`, `dr_trip_start_ms`, `dr_log_since_origin`, `dr_underway_since_origin_s`, `dr_origin_error_nm`) are unchanged, so each flush previously dirtied a 4 kB WAL page per row for nothing; suppressing no-op upserts cuts the anchored flush to the rows that genuinely move (`last_known_good_fix`, whose fix timestamp advances, and `dr_elapsed_since_origin_s`, the wall-clock cone input). `updated_at` then stays at the value's last real change, which is the truthful reading of the column; nothing in the codebase reads it. The database also opens with `PRAGMA wal_autocheckpoint = 256` and `PRAGMA journal_size_limit = 1048576`: the default 1000-page (4 MB) threshold turned every WAL generation into a single ~4 MB copy-back burst on SD-card hardware, and the WAL file parked at its high-water size forever — the same bytes now flow as four 1 MB bursts and the file shrinks back between checkpoints.
+
+## [0.13.2] - 2026-10-11
+
+### Fixed
+- **The shadow boat no longer sails away while anchored.** `engine.tick()` integrated its displacement unconditionally — the `underway` flag from `navigation.state` only gated the uncertainty cone's clock, never the position integration. A month on the hook therefore painted a fictional eastward passage across the chart: the resolved current vector (tier 3 Weather API keeps reporting a set/drift at the anchor position) was integrated every tick as ground displacement, and the paddlewheel on a stopped hull reads the water streaming past it as STW, integrated along the swinging bow. The anchor holds the ground position, so a moored/anchored tick now freezes the shadow boat's position and the logs entirely — water-track vector, current vector, `logNm`/`tripLogNm`/`logNmSinceOrigin` all hold — while the wall-clock `elapsedSinceOriginS` keeps counting the real stay and the engine stays warm for an instant OVERRIDE handoff. Sailing off the hook resumes integration from the held position with no snap needed.
+- **Resource guards against degenerate stored config, and lighter per-tick database work** (found auditing a boat install whose server event loop stalled 10–74 s at a time — the DR plugin measured innocent, but its config surface allowed amplifying exactly such a stall): a hand-edited stored config bypasses the admin UI's schema validation, so `start()` now clamps `tickIntervalMs` to ≥ 10 ms (a 0/negative/NaN value would degenerate `setInterval` into a 1 ms storm multiplying the whole tick path) and `saveIntervalMs` to ≥ 1 s, and the ground-track ring capacity — sized as hours ÷ tick — is hard-capped at 1 M samples (~50 MB; unreachable at the default 1 s tick). The schema itself gains `minimum`/`maximum` for both intervals. The per-tick deviation-rate query no longer recompiles its SQL every second (statement memoized per connection), and the database opens with WAL's recommended `synchronous = NORMAL` — commit fsyncs on SD-card boat hardware drop from one per write to checkpoint time only, for state that is all reconstructable.
+
 ## [0.13.1] - 2026-10-05
 
 ### Fixed

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@meri-imperiumi/signalk-dead-reckoning",
-  "version": "0.13.1",
+  "version": "0.13.3",
   "description": "Offline-first dead reckoning and sensor fusion engine for Signal K",
   "main": "plugin/index.js",
   "scripts": {

package/plugin/db.js

@@ -198,6 +198,23 @@ function openDatabase(dbPath) {
   mkdirSync(dirname(dbPath), { recursive: true });
   const db = new DatabaseSync(dbPath);
   db.exec("PRAGMA journal_mode = WAL");
+  // WAL's recommended durability level: commits sync the WAL to the OS
+  // but do not fsync per commit — a power loss may lose the last
+  // transactions but never corrupts the database. Everything this store
+  // holds is reconstructable (re-derivable from fixes, or a running
+  // average that self-corrects), so trading the per-commit fsync —
+  // sustained minute after minute on SD-card boat hardware — for that
+  // small window is the right default. FULL would fsync every commit.
+  db.exec("PRAGMA synchronous = NORMAL");
+  // Bound the WAL's write bursts: the default auto-checkpoint threshold
+  // is 1000 pages (4 MB), so every WAL generation ends in a single ~4 MB
+  // copy back into the main database — a periodic flash-write spike on
+  // SD-card boat hardware. 256 pages (1 MB) spreads the same bytes over
+  // four smaller bursts, and journal_size_limit lets the WAL file shrink
+  // back to 1 MB after each checkpoint instead of parking forever at its
+  // high-water size (observed at ~4 MB on production stores).
+  db.exec("PRAGMA wal_autocheckpoint = 256");
+  db.exec("PRAGMA journal_size_limit = 1048576");
   db.exec("PRAGMA foreign_keys = ON");
   for (const stmt of SCHEMA_DDL) {
     db.exec(stmt);
@@ -343,17 +360,33 @@ function getState(db, key) {
 /**
  * Writes a scalar value to `dr_state_store`, upserting.
  *
+ * Identical values are not written: the periodic state flush rewrites
+ * the same scalars every cycle (log totals, origin error, trip state),
+ * and on SD-card boat hardware even a no-op upsert dirties a 4 kB page
+ * in the WAL, so suppressing unchanged writes is the single biggest
+ * flash-wear lever this store has. `updated_at` then stays at the
+ * value's last real change, which is the truthful reading of the
+ * column anyway. Column reads elsewhere only consume `value`.
+ *
  * @param {import("node:sqlite").DatabaseSync} db
  * @param {string} key
  * @param {string} value
- * @returns {void}
+ * @returns {boolean} True when a write happened, false when the stored
+ *   value was already identical.
  */
 function setState(db, key, value) {
+  const existing = db
+    .prepare("SELECT value FROM dr_state_store WHERE key = ?")
+    .get(key);
+  if (existing && existing.value === value) {
+    return false;
+  }
   db.prepare(
     `INSERT INTO dr_state_store (key, value, updated_at)
      VALUES (?, ?, ?)
      ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`,
   ).run(key, value, new Date().toISOString());
+  return true;
 }
 
 /**
@@ -557,6 +590,17 @@ function attachObservationsToFix(db, fixId, ids) {
   for (const id of ids?.cplIds ?? []) cpl.run(fixId, id);
 }
 
+/**
+ * Per-connection cache of prepared statements (WeakMap, so a closed db
+ * is collected with its statements). Statement compilation is not free,
+ * and `getDeviationRateStats` runs on the 1 Hz tick path — re-preparing
+ * the same SQL every second would pay the compile over and over for the
+ * life of the process.
+ *
+ * @type {WeakMap<import("node:sqlite").DatabaseSync, Record<string, import("node:sqlite").StatementSync>>}
+ */
+const preparedStatements = new WeakMap();
+
 /**
  * Reads recent `dr_corrections` rows for the uncertainty polygon's
  * empirical deviation-rate (SPEC §8). Returns rows newest-first so the
@@ -571,8 +615,12 @@ function attachObservationsToFix(db, fixId, ids) {
  * @returns {Array<{deviation_nm: number, dr_elapsed_seconds: number}>}
  */
 function getDeviationRateStats(db, q = {}) {
-  const limit = q.limit ?? 50;
-  const stmt = db.prepare(
+  let stmts = preparedStatements.get(db);
+  if (!stmts) {
+    stmts = {};
+    preparedStatements.set(db, stmts);
+  }
+  stmts.deviationRate ??= db.prepare(
     `SELECT deviation_nm, dr_elapsed_seconds
      FROM dr_corrections
      WHERE (? IS NULL OR sail_state = ?)
@@ -580,12 +628,12 @@ function getDeviationRateStats(db, q = {}) {
      ORDER BY correction_id DESC
      LIMIT ?`,
   );
-  return stmt.all(
+  return stmts.deviationRate.all(
     q.sail_state ?? null,
     q.sail_state ?? null,
     q.sea_state ?? null,
     q.sea_state ?? null,
-    limit,
+    q.limit ?? 50,
   );
 }
 

package/plugin/engine.js

@@ -129,13 +129,30 @@ class DeadReckoningEngine {
    * @param {number} inputs.speedLoss - learned speed-loss fraction [0,1]
    * @param {{setTrue: number, drift: number}} [inputs.current] - resolved current (deg true, kn)
    * @param {boolean} [inputs.underway=true] - whether the vessel is under
-   *   way this tick (from `navigation.state`); gates the under-way clock
-   *   the uncertainty cone grows on, not the wall-clock one
+   *   way this tick (from `navigation.state`); false (moored/anchored)
+   *   freezes the shadow boat's position — the anchor holds the ground
+   *   position, so neither the water-track vector nor the resolved
+   *   current vector may integrate — and gates the under-way clock the
+   *   uncertainty cone grows on. The wall-clock `elapsedSinceOriginS`
+   *   still advances: time since the last fix really did elapse.
    * @param {number} [inputs.dtS=1] - tick interval in seconds
    * @returns {{latitude: number, longitude: number}|null} new position, or null if no origin
    */
   tick(inputs, dtS = 1) {
     if (!this.origin) return null;
+    // §7.1–2: moored/anchored is a different regime. The anchor holds the
+    // ground position, so the shadow boat must not sail: neither the
+    // water-track vector (a paddlewheel on a stopped hull reads the water
+    // streaming past it) nor the resolved current vector (a GRIB tier
+    // would otherwise walk the shadow boat across the chart at drift kn
+    // for the whole stay — a month on the hook painted a fictional
+    // passage east) may integrate. The engine stays warm — origin, logs
+    // and clocks carry — so an OVERRIDE handoff stays instant; only the
+    // displacement is frozen, exactly like the real boat.
+    if (inputs.underway === false) {
+      this.elapsedSinceOriginS += dtS;
+      return this.origin;
+    }
     const stw = toNumber(inputs.stwKn);
     const hdg = toNumber(inputs.headingTrueDeg);
     if (stw == null || hdg == null) return this.origin;

package/plugin/index.js

@@ -740,11 +740,19 @@ module.exports = (app) => {
         tickIntervalMs: {
           type: "integer",
           title: "DR Integration Interval (ms)",
+          // Floor guards the whole pipeline: the ground-track ring is
+          // sized as hours/tickInterval, so a small tick multiplies the
+          // ring's memory, the tick-path work and the flush cadence by
+          // the same factor (1 ms would allocate a ~130 M-sample ring).
+          // 500 ms (2 Hz) is the useful ceiling for a shadow boat.
+          minimum: 500,
+          maximum: 60000,
           default: DEFAULT_CONFIG.tickIntervalMs,
         },
         saveIntervalMs: {
           type: "integer",
           title: "State Save Interval (ms)",
+          minimum: 5000,
           default: DEFAULT_CONFIG.saveIntervalMs,
         },
         "publish.everyTicks": {
@@ -836,6 +844,31 @@ module.exports = (app) => {
       // logbook write-through).
       const opts = deflattenConfig(options);
       config = { ...DEFAULT_CONFIG, ...opts };
+      // Defensive clamps: the admin UI enforces the schema minimums, but
+      // hand-edited or legacy stored configs skip validation, and a
+      // 0/negative/NaN tick would degenerate setInterval to a 1 ms storm
+      // that also multiplies the tick path and the flush cadence. The
+      // floors are deliberately permissive (10 ms — the test harness
+      // drives ticks at 20–100 ms); the UI schema is where the human
+      //-facing floor (500 ms) lives.
+      config.tickIntervalMs = Math.min(
+        60000,
+        Math.max(
+          10,
+          Math.round(
+            Number(config.tickIntervalMs) || DEFAULT_CONFIG.tickIntervalMs,
+          ),
+        ),
+      );
+      config.saveIntervalMs = Math.min(
+        24 * 3600 * 1000,
+        Math.max(
+          1000,
+          Math.round(
+            Number(config.saveIntervalMs) || DEFAULT_CONFIG.saveIntervalMs,
+          ),
+        ),
+      );
       // Allow partial divergence overrides without losing the defaults.
       config.divergence = {
         ...DEFAULT_CONFIG.divergence,
@@ -891,10 +924,18 @@ module.exports = (app) => {
         tripLogNm: Number.isFinite(savedTripNm) ? savedTripNm : 0,
       });
       groundTrack = new deps.GroundTrack({
-        capacity: Math.max(
-          60,
-          Math.round(
-            (config.groundTrackHours * 3600 * 1000) / config.tickIntervalMs,
+        capacity: Math.min(
+          // Memory guard: the ring is sized as hours/tickInterval, so a
+          // fast stored tick would scale it without bound (1 ms → a
+          // ~130 M-sample ring). 1 M samples ≈ 50 MB is the hard
+          // ceiling; at the default 1 s tick it is never reached
+          // (168 h, the schema max, is 604,800).
+          1_000_000,
+          Math.max(
+            60,
+            Math.round(
+              (config.groundTrackHours * 3600 * 1000) / config.tickIntervalMs,
+            ),
           ),
         ),
       });

package/tests/db.test.js

@@ -243,6 +243,39 @@ test("setState upserts and getState reads back", () => {
   db.close();
 });
 
+test("setState skips identical values (flash-wear guard)", () => {
+  const db = openDatabase(join(tempDir, "c.sqlite"));
+  assert.strictEqual(setState(db, "dr_log_nm", "123.45"), true);
+  const firstUpdatedAt = db
+    .prepare("SELECT updated_at FROM dr_state_store WHERE key = ?")
+    .get("dr_log_nm").updated_at;
+  // Identical write is suppressed: no updated_at churn, no WAL page dirtied
+  assert.strictEqual(setState(db, "dr_log_nm", "123.45"), false);
+  assert.strictEqual(
+    db
+      .prepare("SELECT updated_at FROM dr_state_store WHERE key = ?")
+      .get("dr_log_nm").updated_at,
+    firstUpdatedAt,
+  );
+  // A changed value still writes
+  assert.strictEqual(setState(db, "dr_log_nm", "124.00"), true);
+  assert.strictEqual(getState(db, "dr_log_nm"), "124.00");
+  db.close();
+});
+
+test("openDatabase bounds WAL checkpoint bursts and file high-water", () => {
+  const db = openDatabase(join(tempDir, "c.sqlite"));
+  assert.strictEqual(
+    db.prepare("PRAGMA wal_autocheckpoint").get()["wal_autocheckpoint"],
+    256,
+  );
+  assert.strictEqual(
+    db.prepare("PRAGMA journal_size_limit").get()["journal_size_limit"],
+    1048576,
+  );
+  db.close();
+});
+
 test("dr_matrix_bins primary key enforces bin-level uniqueness", () => {
   const db = openDatabase(join(tempDir, "d.sqlite"));
   const ins = db.prepare(
@@ -378,6 +411,52 @@ test("getDeviationRateStats returns recent rows filtered by sail/sea state", asy
   await rm(dir, { recursive: true, force: true });
 });
 
+test("getDeviationRateStats reuses its prepared statement per connection", async () => {
+  // The 1 Hz tick path calls this with the same SQL every second — the
+  // statement is memoized per database (WeakMap). Repeated calls with
+  // different bindings must stay correct, and a second connection must
+  // get its own statement, not share the first one's.
+  const dirs = await Promise.all([
+    mkdtemp(join(tmpdir(), "dr-stmt-a-")),
+    mkdtemp(join(tmpdir(), "dr-stmt-b-")),
+  ]);
+  const dbs = dirs.map((dir) => openDatabase(join(dir, "t.sqlite")));
+  for (const db of dbs) {
+    const fid = recordFix(db, {
+      timestamp: "2026-01-01T00:00:00Z",
+      source_type: "gps",
+      latitude: 60,
+      longitude: 24,
+      resets_dr_origin: true,
+    });
+    recordCorrection(db, {
+      fix_id: fid,
+      timestamp: "2026-01-01T00:00:00Z",
+      dr_lat: 60.01,
+      dr_lon: 24.01,
+      fix_lat: 60,
+      fix_lon: 24,
+      deviation_nm: 0.4,
+      deviation_bearing: 200,
+      dr_elapsed_seconds: 1800,
+      sail_state: "sailing",
+    });
+  }
+  // Interleaved calls across both connections, changing bindings each time.
+  const a1 = getDeviationRateStats(dbs[0], { limit: 10 });
+  const b1 = getDeviationRateStats(dbs[1], { limit: 10 });
+  const a2 = getDeviationRateStats(dbs[0], { limit: 1 });
+  const b2 = getDeviationRateStats(dbs[1], { sail_state: "motoring" });
+  assert.strictEqual(a1.length, 1);
+  assert.strictEqual(b1.length, 1);
+  assert.strictEqual(a2.length, 1);
+  assert.strictEqual(b2.length, 0);
+  for (const [db, dir] of dbs.map((db, i) => [db, dirs[i]])) {
+    db.close();
+    await rm(dir, { recursive: true, force: true });
+  }
+});
+
 // --- Observation & fix CRUD (work doc #13 stage D) -------------------------
 
 const {

package/tests/engine.test.js

@@ -39,6 +39,41 @@ test("under-way clock only advances while under way", () => {
   assert.strictEqual(e.underwaySinceOriginS, 0);
 });
 
+test("anchored tick holds the shadow boat on the anchor", () => {
+  const e = new DeadReckoningEngine();
+  e.snapToFix({ latitude: -18.65, longitude: 174.0 });
+  // A month on the hook: sensors on a stopped hull still read — the
+  // paddlewheel sees the current streaming past, the resolver hands back
+  // a GRIB current — but the anchor holds the ground position, so none
+  // of it may move the shadow boat or grow the logs.
+  const before = { ...e };
+  for (let day = 0; day < 30; day++) {
+    const pos = e.tick(
+      {
+        stwKn: 0.8,
+        headingTrueDeg: 90,
+        leewayDeg: 5,
+        current: { setTrue: 90, drift: 0.4 },
+        underway: false,
+      },
+      24 * 3600,
+    );
+    assert.deepStrictEqual(pos, before.origin);
+  }
+  assert.deepStrictEqual(e.origin, before.origin);
+  assert.strictEqual(e.logNm, before.logNm);
+  assert.strictEqual(e.tripLogNm, before.tripLogNm);
+  assert.strictEqual(e.logNmSinceOrigin, before.logNmSinceOrigin);
+  // The under-way clock — the cone's growth axis — stays frozen, while
+  // the wall-clock "since fix" keeps counting the real stay.
+  assert.strictEqual(e.underwaySinceOriginS, 0);
+  assert.strictEqual(e.elapsedSinceOriginS, 30 * 24 * 3600);
+  // The engine stays warm: sailing off the hook resumes integration
+  // from the held position with no snap needed.
+  const pos = e.tick({ stwKn: 5, headingTrueDeg: 0, underway: true }, 3600);
+  assert.ok(Math.abs(pos.latitude - (before.origin.latitude + 5 / 60)) < 1e-4);
+});
+
 test("tick advances north along a meridian at the given STW", () => {
   const e = new DeadReckoningEngine();
   e.snapToFix({ latitude: 60, longitude: 24 });

package/tests/plugin.test.js

@@ -44,14 +44,14 @@ test.after(async () => {
  *
  * @returns {{app: FakeSignalKApp, plugin: object, router: FakeRouter}}
  */
-function makeStarted() {
+function makeStarted(options) {
   const app = new FakeSignalKApp();
   // Fresh data dir per call: the speed-plausibility gate and restart
   // persistence make cross-test DB leakage (a stale restored DR origin)
   // observable, so each test owns its database.
   app.dataPath = mkdtempSync(join(tmpdir(), "dr-plugin-"));
   const plugin = makePlugin(app);
-  plugin.start({});
+  plugin.start(options ?? {});
   const router = new FakeRouter();
   plugin.registerWithRouter(router);
   return { app, plugin, router };
@@ -2843,6 +2843,33 @@ test("app.get config endpoint serves the plugin config with a hash", () => {
   plugin.stop();
 });
 
+test("insane intervals from stored config are clamped, ring capacity bounded", () => {
+  // The admin UI enforces the schema minimums, but a hand-edited stored
+  // config skips validation — a 1 ms tick would degenerate setInterval
+  // into a 1 ms storm, and the ground-track ring (sized as hours ÷ tick)
+  // would grow without bound. The floors are deliberately permissive
+  // (the test harness itself drives ticks at 20–100 ms); the clamp only
+  // stops the degenerate cases.
+  const { app, plugin } = makeStarted({
+    tickIntervalMs: 1,
+    saveIntervalMs: 100,
+  });
+  const cfg = app.appRoutes.find((r) =>
+    r.path.includes("/signalk-dead-reckoning/configuration"),
+  );
+  const res = {
+    set() {},
+    json(obj) {
+      this._body = obj;
+    },
+    status() {},
+  };
+  cfg.handler({}, res);
+  assert.strictEqual(res._body.config.tickIntervalMs, 10);
+  assert.strictEqual(res._body.config.saveIntervalMs, 1000);
+  plugin.stop();
+});
+
 test("schema exposes the positionFormat option with DMS default", () => {
   const app = new FakeSignalKApp();
   const plugin = makePlugin(app);