speedandcurrent 2.3.4 → 2.4.0

package/README.md

@@ -66,7 +66,7 @@ Once configured, all consumers on the Signal K bus — KIP, OpenCPN, Instrument
 | `environment.current.drift` | m/s | Estimated current speed. |
 | `environment.current.setTrue` | rad | Estimated current direction (the direction the water moves *toward*). |
 
-A 60-second stabilisation period applies after startup. No output is published during this window.
+A 60-second stabilisation period applies after startup. Corrected boatspeed may be published during this window, but current estimation and table learning wait until it ends.
 
 ---
 
@@ -104,7 +104,7 @@ The **Inputs** section shows live readings from the raw sensors as they arrive f
 
 | Setting | Default | Description |
 |---------|---------|-------------|
-| **Staleness Detection** | On | Mark an input as unavailable when it stops updating. Disable when testing the plugin with simulated or replayed data, where timing is irregular and would otherwise trigger spurious warnings. |
+| **Input staleness handling** | Built in | Inputs are marked stale when they stop updating, and the webapp displays a warning. `navigation.state` is an event path and is not treated as stale between state transitions. |
 
 ---
 
@@ -144,23 +144,20 @@ The **Correction Table Learning** section has its own independent toggle. Learni
 
 The panel shows the **smoothed** sensor inputs used for table updates — heading, boat speed, ground speed, and attitude. These are the same signals as in the Estimation panel, but averaged over the smoother window before being fed into the learning algorithm. This averaging reduces the influence of short-term fluctuations on the table update.
 
-When **Assume Current** is enabled, the smoothed current estimate is also shown as a learning input.
-
 Warnings appear here if any smoothed input is unavailable.
 
-The panel also shows whether learning is currently active, suspended, or skipped for the latest observation. Learning is automatically suspended when `navigation.state` is `anchored` or `moored` when that path is available. It can also be suspended for `motoring` if enabled in the settings. Observations below the current table speed-step threshold are skipped.
+The panel also shows whether learning is currently active, suspended, or skipped for the latest observation. Learning is normally gated by SOG: observations below the current table speed-step threshold are skipped. When **Suspend on navigation.state = motoring** is enabled and `navigation.state` is known, `anchored`, `moored`, and `motoring` suspend learning; an unavailable or unknown state falls back to the SOG gate.
 
 ### Settings
 
 | Setting | Default | Description |
 |---------|---------|-------------|
 | **Update Correction Table** | On | Master toggle. Allow the table to update from current observations. |
-| **Suspend on navigation.state = motoring** | Off | Suspend learning when `navigation.state` reports `motoring`. `anchored` and `moored` always suspend learning when that path is available. |
-| **Stability (1–20)** | 7 | How quickly the correction table adapts to new observations. Higher = slower, more conservative. Lower = faster but noisier. See the technical section for detail. |
-| **Assume Current (experimental)** | Off | Include the running current estimate in the table update calculation. Only enable once the current estimate has had time to stabilise and tidal conditions are relatively steady. |
+| **Suspend on navigation.state = motoring** | Off | When enabled, suspend learning when a known `navigation.state` reports `anchored`, `moored`, or `motoring`. When disabled, `navigation.state` does not override the SOG-based moving check. |
+| **Correction drift** | 0.3 knots/month | How quickly the correction may change over time (knots/month). 0.3 knots/month means the correction can typically change by about 0.3 knots over a month. Higher values adapt faster but may be less stable. Range: 0.0–3.0 knots/month. |
 | **Show Statistics (σ)** | Off | Display standard deviation alongside each smoothed value. Useful for spotting noisy sensors. |
 
-### Smoother settings
+### Smoothing and learning cadence
 
 Controls how raw sensor values are averaged before being used for table updates. These settings have no effect on the published corrected values — only on the inputs to the learning algorithm.
 
@@ -168,12 +165,9 @@ The smoother serves a second purpose beyond noise reduction: its output variance
 
 | Setting | Default | Description |
 |---------|---------|-------------|
-| **Smoother Type** | Moving average | Moving Average (window), Exponential decay (τ), or Kalman filter. Moving average is the most predictable. |
-| **Window size** | 5 s | (Moving average) Integration window. Larger = smoother but slower to respond. Minimum 2 s. |
-| **Time constant (τ)** | 3 s | (Exponential) Decay time constant. Larger = smoother. Minimum 1 s. |
-| **Kalman gain** | 0.2 | (Kalman) Steady-state gain. Lower = smoother / slower. Range 0.01–0.99. |
+| **Window size** | 5 s | Moving-average integration window. Larger = smoother but slower to respond. Minimum 5 s. |
 
-A 5-second moving average window suits typical 1 Hz instrument update rates. In rough conditions with high sensor noise, increasing it to 8–10 seconds may improve table quality at the cost of temporal resolution.
+A 5-second moving average window suits typical 1 Hz instrument update rates. Learning attempts occur every `window + 1 second`, leaving a one-second gap between observation windows. In rough conditions with high sensor noise, increasing the window to 8–10 seconds may improve table quality at the cost of temporal resolution. Older configurations that selected another smoother are silently converted to moving average; their valid window duration is retained and shorter values are clamped to 5 seconds.
 
 ---
 
@@ -183,16 +177,19 @@ The **Correction Table** section displays the active correction table. Rows are
 
 ### Reading the table
 
-Each cell that has received at least one observation shows two values:
+Each cell that has received at least one observation shows its correction values. Cells participating in the latest estimate also show their normalized fusion weight as a blue bar along the bottom:
+
+- **Speed factor** (e.g. `+3.2%` or `−1.5%`): how much faster or slower the true speed is compared to what the paddle wheel reads.
+- **Leeway** (e.g. `+4°`): the observed lateral correction for that speed/heel combination. Positive is starboard.
+- **Fusion weight**: the blue bar is that cell's normalized share of the current precision-weighted estimate. A full-width bar represents 100%, and the displayed weights sum to approximately 100%.
 
-- **Speed factor** (e.g. `+3.2%` or `−1.5%`): how much faster or slower the true speed is compared to what the paddle wheel reads. **Green** = paddle wheel reads slow (the correction adds speed). **Orange** = paddle wheel reads fast (the correction reduces speed). Colour intensity scales with the magnitude of the factor relative to the largest factor in the table.
-- **Leeway** (e.g. `+4°`): the observed lateral correction for that speed/heel combination. Positive is starboard. When leeway is non-zero the cell background shows diagonal stripes; the stripe angle encodes the leeway direction visually.
+Cell backgrounds encode the speed factor: green means the paddlewheel reads slow and orange means it reads fast. When leeway is available, stripe angle encodes its direction.
 
 Empty cells have not yet received any observations and show no correction.
 
-### Active cell and neighbours
+### Active cell
 
-The **active cell** — the one most recently updated by an incoming STW sample — is highlighted with a bold border and blue text. The **neighbour cells** contributing to the current interpolation are shown with a faint tint. Watching these as you sail shows exactly which part of the table is being applied and updated at any moment.
+The **active cell** — the one most recently updated by an incoming STW sample — is shown with a black border.
 
 ### Table management
 
@@ -205,13 +202,19 @@ The **active cell** — the one most recently updated by an incoming STW sample
 
 Multiple tables can coexist on disk; only the active one is used. This makes it straightforward to keep separate tables for different configurations such as racing vs. cruising sails, or before and after antifouling.
 
+### Correction-table file formats
+
+The original format is schema version 1. It remains available as [`docs/correction-table-v1.schema.json`](docs/correction-table-v1.schema.json), while [`docs/correction-table.schema.json`](docs/correction-table.schema.json) is retained as the compatible legacy filename. New and saved tables use schema version 2, documented in [`docs/correction-table-v2.schema.json`](docs/correction-table-v2.schema.json). Version 2 adds a top-level `schemaVersion: 2`, the current serializable model descriptor, and `lastAcceptedAt` epoch milliseconds on every cell; empty cells use `null`.
+
+Version 1 files, including older files without a version field, are migrated silently on load. Since individual legacy cell ages are unknown, the file modification time is assigned to every learned cell as a conservative approximation. If that timestamp is unavailable, load time is used and the fallback is written only to debug logging. Migration preserves table identity, dimensions, display attributes, means, covariance, and indices, and is persisted atomically only after the table has loaded successfully.
+
 ---
 
 ## Operating notes
 
 **Be patient with the correction table.** A fresh table has no data and produces no corrections. Cover a range of speeds and heel angles over a few sails and the table fills in progressively. Upwind sailing covers the heel bins well; downwind and reaching fill the low-heel, varying-speed bins.
 
-**The table learns while sailing normally.** No dedicated calibration runs are needed. Just sail with **Update Correction Table** on. Learning pauses automatically during startup stabilisation, when `navigation.state` indicates `anchored` or `moored`, optionally when it indicates `motoring`, and when SOG or STW are below the current table speed-step threshold.
+**The table learns while sailing normally.** No dedicated calibration runs are needed. Just sail with **Update Correction Table** on. Learning pauses automatically during startup stabilisation and when SOG or STW are below the current table speed-step threshold. If **Suspend on navigation.state = motoring** is enabled, a known `navigation.state` of `anchored`, `moored`, or `motoring` also pauses learning.
 
 **Port and starboard are tracked independently.** Heel is signed: starboard positive, port negative. An asymmetric paddle wheel installation will show different corrections on each tack, and the table captures this naturally.
 
@@ -239,17 +242,42 @@ When a new paddle wheel sample arrives the plugin:
 
 ### Neighbour interpolation
 
-Rather than bi-linear interpolation the plugin uses **inverse-distance weighting** over the five nearest cells in speed/heel space:
+The plugin combines learned cells using both their Kalman covariance and their distance from the requested speed and heel. Speed and heel distances are divided by their respective grid steps, so distance is measured in dimensionless cell units. A cell participates only when:
+
+```text
+cell index > 0
+cell distance <= 2.5
+```
+
+For each participating cell, distance is converted into additional uncertainty:
 
-> wᵢ = 1 / (dᵢ + ε)
+```text
+effective covariance = cell covariance + q * distance² * identity matrix
+```
 
-where dᵢ is the Euclidean distance from the query point to cell centre i in normalised speed/heel space. Only cells that have received at least one observation (N > 0) contribute.
+The inverse effective covariance is the cell's precision. The correction and its covariance are obtained by full 2×2 precision fusion:
 
-The final correction is the normalised weighted average:
+```text
+combined precision = sum(inverse(effective covariance))
+correction covariance = inverse(combined precision)
+correction = correction covariance
+			 * sum(inverse(effective covariance) * cell correction)
+```
 
-> x̂ = Σ(wᵢ · xᵢ) / Σwᵢ,  ŷ = Σ(wᵢ · yᵢ) / Σwᵢ
+This gives nearby, mature cells the greatest influence. A fresh table borrows information more broadly because cell covariance is still large; a mature table becomes more local as its cell covariance falls. If no eligible cell exists, the plugin applies zero correction rather than extrapolating beyond the configured radius.
 
-The variance of each axis is propagated through the same weighting, so cells with low confidence (high covariance) influence the result less. If only one or two cells have data, the correction extrapolates gradually to nearby conditions. As more cells fill in, the correction becomes tighter and more local.
+#### Automatic spatial variance (`q`)
+
+`q` describes how much genuine correction variation is expected between adjacent cells, in correction variance per squared cell. It is calculated without user input from horizontal and vertical pairs where both cells have an index greater than 0. A cell with one accepted observation is therefore eligible while the explicit maturity threshold remains available for future tuning. For each adjacent pair:
+
+```text
+pair q = (
+	squared distance between the two correction vectors
+	- trace(cell covariance 1 + cell covariance 2)
+) / 2
+```
+
+The table uses the median pair estimate, which limits the effect of isolated rough or immature cells. At least three eligible adjacent pairs and a positive median are required. A zero or negative estimate means the observed differences do not resolve spatial variation above the cells' estimated uncertainty; because using zero would disable distance weighting, the plugin then uses a conservative fallback of `0.002 (m/s)²` per squared cell. `q` is calculated when a table is loaded and recalculated after every 600 accepted learning observations. The applied `q`, raw estimate, source, and supporting pair count are runtime diagnostics and are not saved in the correction-table file.
 
 ### How the correction table is populated
 
@@ -257,43 +285,44 @@ Each cell is an independent **2-dimensional Kalman filter** tracking the correct
 
 Every time conditions are right — plugin running for >60 seconds, smoothers settled, speed above minimum threshold — the plugin computes an **observation** of what the correction should be:
 
-> observation = R(ψ)⁻¹ · V_SOG − R(ψ)⁻¹ · V_current − V_STW
+```text
+observation = R(ψ)⁻¹ · V_SOG − R(ψ)⁻¹ · V_current − V_STW
+```
 
-where R(ψ) rotates from ground frame to boat frame using true heading ψ. In plain terms: rotate GPS velocity into the boat frame, subtract the current estimate (also rotated), subtract the raw paddle wheel velocity. The residual is the implied sensor error for the current speed and heel.
+where `R(ψ)` rotates from ground frame to boat frame using true heading `ψ`. In plain terms: rotate GPS velocity into the boat frame, subtract the raw paddle wheel velocity. The residual is the implied sensor error for the current speed and heel.
 
 The Kalman update combines this observation with the cell's existing belief:
 
-> K = P · (P + R_obs)⁻¹
-> x_new = x_old + K · (observation − x_old)
+```text
+K = P · (P + R_obs)⁻¹
+x_new = x_old + K · (observation − x_old)
+```
 
-where P is the cell's current covariance and R_obs is the observation covariance derived from the measurement uncertainty of all contributing signals (SOG variance + current variance + STW variance, rotated appropriately). **Noisy observations produce a smaller gain and move the cell estimate less.**
+where P is the cell's current covariance and R_obs is the observation covariance derived from the measurement uncertainty of all contributing signals (SOG variance + STW variance, rotated appropriately). It also includes heading uncertainty in radians squared. For $u = R(-heading)(groundSpeed - current)$, the heading contribution is $J \sigma_h^2 J^T$, where $J = [u_y, -u_x]^T$. This is applied once to the combined ground-speed-minus-current vector because both vectors share the same heading error. **Noisy observations produce a smaller gain and move the cell estimate less.**
 
-### The stability setting in detail
+### Time-scaled aging and correction drift
 
-Each cell has a small **process noise** that allows it to drift slowly over time, reflecting that a paddle wheel's error can change with fouling, recalibration, or crew weight distribution. Process noise is:
+Each cell has a small **process-noise rate** that allows it to drift slowly over time, reflecting that a paddle wheel's error can change. For an elapsed interval `dt`, the process covariance added to each diagonal is:
 
-> Q = 10^(−stability)
+```text
+effectiveDt = min(dt, 90 days)
+Q(dt) = Q_rate × effectiveDt × I
+```
 
-With stability = 7 (default), Q = 10⁻⁷, making the cell very resistant to change — it effectively averages hundreds of observations before settling. Stability 4–5 makes the table react faster to recent conditions; stability 10–12 is appropriate for a well-characterised boat that changes rarely.
+The user-facing `Correction drift` value is expressed in knots per month, using a 30-day month. Internally, it is converted to the SI process-noise rate in $(m/s)^2/s$. The default 0.3 knots/month corresponds to approximately $9.2 × 10^{-9}\ (m/s)^2/s$. Offline time counts toward aging, but aging stops after the internal 90-day cap. Aged covariance is calculated when gating, filtering, interpolation, spatial-q estimation, and reporting; stored covariance is not mutated by reads. Rejected observations do not advance timestamps; accepted observations store the posterior covariance and current UTC epoch-millisecond timestamp.
 
-In practical terms: high stability = trust the accumulated history; low stability = trust recent observations more.
+In practical terms: a lower correction-drift rate trusts accumulated history more; a higher rate allows the table to adapt more quickly to long-term sensor changes.
 
 ### How current is estimated
 
 Current is estimated as:
 
-> V_current = V_SOG − R(ψ) · V_STW_corrected
+```text
+V_current = V_SOG − R(ψ) · V_STW_corrected
+```
 
 The corrected STW vector is rotated into the ground frame using heading, then subtracted from the GPS velocity. The residual is the water velocity.
 
 This raw estimate is fed into a **Kalman smoother with very low process noise** (process variance ≈ 10⁻⁶), so it changes very slowly, integrating over many minutes rather than chasing individual GPS fluctuations. At startup the estimate is strongly initialised to zero.
 
 Current estimation requires an accurate boat speed, so it is gated by the 60-second stabilisation period and only runs when **Estimate Boat Speed** is enabled.
-
-### Current and table learning
-
-Current estimation always runs alongside speed correction — it is a direct byproduct of comparing the GPS velocity with the corrected STW vector. You cannot have one without the other.
-
-For table *learning*, however, current can be left out of the equation without significant harm. When a boat sails different headings over time — tacking, gybing, reaching — any steady current appears as an error in one direction on one heading and the opposite direction on another. These errors cancel in the long-term average, so the table converges on the correct speed correction regardless. This is why the default (**Assume Current** off) uses a zero-current placeholder rather than the live estimate.
-
-Including the current estimate (**Assume Current** on) would be more accurate in principle, but it introduces a dependency: the current estimate is only as good as the speed correction that produced it. Early in the table's life, when corrections are rough, the current estimate is also rough, and feeding it back into learning can amplify rather than reduce error. There is also a circularity: better speed correction → better current estimate → better speed correction. Enabling this before the table has had time to settle can cause the two to pull each other in the wrong direction, particularly in changing tidal conditions. Use it only once the table is reasonably well populated and current conditions are stable.