signalk-questdb 2.5.0 → 2.5.2

package/README.md

@@ -1,1235 +1,206 @@
 # signalk-questdb
 
-QuestDB history provider for Signal K — a drop-in replacement for
-signalk-to-influxdb and signalk-to-influxdb2.
-
-Records every vessel's data in [QuestDB](https://questdb.com), running as a
-container managed for you by
-[signalk-container](https://github.com/dirkwa/signalk-container), and serves it
-back through Signal K's v2 History API and the legacy v1 playback API. Existing
-InfluxDB history can be imported.
-
-## Features
-
-- **Managed QuestDB** — the container is pulled, created, updated and
-  resource-capped for you; or point the plugin at an external QuestDB
-- **v2 History API** — `getValues`, `getPaths`, `getContexts` with every
-  aggregate method, per-source reads (`path|sourceRef`) and per-source splitting
-  (`sourcePolicy=all`)
-- **v1 playback API** — WebSocket replay at configurable speed
-- **Recording controls** — per-path sampling rates, include/exclude filters with
-  globs, own vessel and AIS targets separately, retention by days, LZ4/ZSTD
-  on-disk compression
-- **Archive past retention** — retention drops old days, but the own track, and
-  any own-vessel paths you name such as depth, are kept, thinned, in an archive
-  the History API reads seamlessly
-- **GPX track import** — a chartplotter's track export of any size goes into
-  the archive; the file is read in the browser first, so you pick the own
-  vessel's tracks before anything is uploaded
-- **Startup restore** — AIS targets back on the chart right after a server
-  restart, in their ship-type colour
-- **WAL watchdog** — a table QuestDB has silently stopped applying is detected,
-  alerted, and resumed
-- **InfluxDB migration** — detect InfluxDB 1.x/2.x, pick a bucket, a vessel and
-  a range, import with original timestamps; resumable
-- **Export** — Parquet or CSV by date range, and a per-table full export for
-  backup tooling
-- **Console webapp** — QuestDB's own SQL console inside the Signal K admin UI,
-  admin only
-- **Config panel** — status, row counts, one-click QuestDB updates, and every
-  setting in one place
+Keeps the history of your boat's data. The plugin records what Signal K sees,
+from your own instruments and from the AIS targets around you, in a
+[QuestDB](https://questdb.com) database. Apps that draw tracks and graphs get it
+back through Signal K's History API, and Grafana can read it too.
+
+It replaces signalk-to-influxdb and signalk-to-influxdb2, and imports the
+history they recorded.
+
+- **Looks after itself.** QuestDB runs in a container that the plugin sets up,
+  updates and keeps healthy.
+- **Small on disk.** A value that changes is written at most every 2 seconds, a
+  value that stays the same only every 5 minutes.
+- **Keeps your track.** Old data can be deleted after a number of days, while
+  your own track is kept for good.
+- **AIS back at once.** After a restart, the vessels around you are on the chart
+  right away instead of trickling in over minutes.
+- **Imports** history from InfluxDB, and tracks from a chartplotter as GPX.
+- **Exports** to Parquet or CSV.
+- **SQL console**: QuestDB's own console inside the Signal K admin UI.
 
 ## Getting started
 
 ### Requirements
 
-- Node.js 24 or newer
-- QuestDB 8.1 or newer (resolution queries use `SAMPLE BY … FROM … TO`). This
-  applies to an external QuestDB and to a pinned **QuestDB image version** in
-  managed mode; `latest` always qualifies
-- Signal K server — 2.31 or newer to choose the default history provider in the
-  admin UI
-- For managed mode: the
-  [signalk-container](https://github.com/dirkwa/signalk-container) plugin, and
-  Podman 5.4 or newer (what Debian 13 "trixie" ships) or Docker.
-  signalk-container 1.29.0 or newer is recommended; the plugin runs with 1.14.0
-  or newer, but older versions lose parts of it — below 1.19.0 the Danger zone
-  cannot delete data kept on a named volume, below 1.26 the macOS open-files
-  problem shows as a generic error, below 1.29.0 a container the runtime refuses
-  to recreate is not reported (see Troubleshooting). On rootless Podman below
-  5.5 the plugin's open-files request is inherited from the podman service
-  rather than granted per container
-  ([containers/podman#25881](https://github.com/containers/podman/issues/25881));
-  signalk-container accounts for this.
-
-### Install
-
-1. Install **signalk-container** from the App Store and enable it (skip this for
-   an external QuestDB).
-2. Install **signalk-questdb** and enable it. On the first start the plugin
-   pulls `questdb/questdb` at the tag chosen under **QuestDB image version**
-   (`latest` by default), creates the container — named `signalk-questdb` behind
-   the runtime's prefix, `sk-` unless configured otherwise — with its ports
-   bound to loopback, and starts recording. The data lives under Signal K's
-   configuration directory, in `plugin-config-data/signalk-questdb` (see Data
-   storage).
-3. Open the plugin's config panel: the status card shows **Running** and
-   **Recording**, the total row count and the paths active today.
-
-For an external QuestDB, switch **Manage QuestDB container** off and enter its
-host and ports.
-
-### Make it the history provider
-
-Which registered provider answers `/signalk/v2/api/history/` is the operator's
-choice: pick it once under **Data → Preferences → Default History Provider**
-(server 2.31 or newer); the server persists it as `historyApi.defaultProvider`
-in `settings.json`, and it survives restarts and plugin load order. If no
-default is configured, the server uses whichever provider registers first.
-Versions 2.0.0 and earlier asked the server to make QuestDB the default on every
-start; that is gone as of 2.0.1 — it could silently override a default you had
-chosen, and on servers with security enabled it never worked at all.
-
-## Configuration
-
-The config panel groups the settings as: the **status card** (running state,
-rows, active paths, restore outcome, WAL state), the **update check** and
-**image version** picker (latest, pre-releases and the last three stable
-releases), **Connection**, **Recording**, **History**, and the collapsible
-**Path filtering**, **Compression (on-disk)**, **InfluxDB Migration**, **Data
-Export** and **Danger zone** sections. The Danger zone removes the container and
-all of its data — including what Signal K's plugin uninstall cannot reach on
-rootless Podman.
-
-| Setting                       | Default      | Description                                                                                                                                                                     |
-| ----------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| QuestDB image version         | `latest`     | Image tag; the dropdown lists the latest release, pre-releases, and the last three stable releases                                                                              |
-| Manage QuestDB container      | `true`       | Let signalk-container run QuestDB; off = connect to an external QuestDB                                                                                                         |
-| QuestDB host                  | `127.0.0.1`  | External mode only; in managed mode the address is resolved automatically                                                                                                       |
-| HTTP port                     | `9000`       | External mode, or the host binding when "Bind to 0.0.0.0" is on; otherwise signalk-container allocates it                                                                       |
-| ILP port                      | `9009`       | External mode, or the host binding when "Bind to 0.0.0.0" is on; otherwise signalk-container allocates it                                                                       |
-| PostgreSQL port               | `8812`       | Host binding for Grafana/psql when "Bind to 0.0.0.0" is on                                                                                                                      |
-| QuestDB memory limit          | `768m`       | Hard cgroup cap on the container (empty = unlimited); the JVM heap gets half of it, so heap plus off-heap stays inside                                                          |
-| QuestDB CPU limit (cores)     | `1.5`        | Fractional cores (0 = unlimited)                                                                                                                                                |
-| Default sampling rate (ms)    | `2000`       | Minimum ms between writes for any one path (0 = every update)                                                                                                                   |
-| Per-path sampling rates (ms)  | _(empty)_    | Glob → ms overrides, one `pattern ms` per line in the panel (0 = wire speed, see Sampling)                                                                                      |
-| Unchanged-value heartbeat (s) | `300`        | A value equal to the last one recorded is written again only this often; changes are written at the sampling rate (0 = every sample, see Change-only recording)                 |
-| Write batch interval (ms)     | `5000`       | How often buffered samples are committed — one WAL transaction per table per commit (see Performance)                                                                           |
-| Record own vessel             | `true`       | Record the `self` context                                                                                                                                                       |
-| Record AIS targets            | `true`       | Record other vessels                                                                                                                                                            |
-| Record all sources            | `false`      | Record every source of a path, not only the one the source priorities prefer (see Source priorities)                                                                            |
-| Restore vessels on startup    | `false`      | Replay each vessel's last recorded position after a restart (see Startup restore)                                                                                               |
-| Restore max age (minutes)     | `9`          | Only replay values recorded within this window                                                                                                                                  |
-| Allow `sourcePolicy=all`      | `false`      | Let History API callers split a path into one column per recording source (see History API)                                                                                     |
-| QuestDB console webapp        | `true`       | Serve QuestDB's console inside the Signal K admin UI, admin only (see Console webapp)                                                                                           |
-| Retention (days)              | `0`          | Drop daily partitions older than this (0 = keep forever)                                                                                                                        |
-| Keep own track                | `true`       | Archive the own vessel's positions before retention drops them (see Retention and the archive)                                                                                  |
-| Archived own-vessel paths     | _(empty)_    | Glob patterns of own-vessel numeric paths to archive the same way, e.g. `environment.depth.*`                                                                                   |
-| Archive resolution (s)        | `10`         | Archived rows are thinned to the first sample per interval (0 = every row)                                                                                                      |
-| Filter mode                   | `exclude`    | `exclude` the matching paths, or `include` only the matching paths                                                                                                              |
-| Path patterns                 | _(empty)_    | Glob patterns, one per line, e.g. `environment.wind.*`; empty records everything                                                                                                |
-| Compression codec             | `lz4`        | On-disk WAL segments and Parquet exports: `none`, `lz4` (fast) or `zstd` (smaller)                                                                                              |
-| Compression level             | `3`          | ZSTD level 1–22, used only with `zstd`                                                                                                                                          |
-| Container network             | `sk-network` | Shared Podman/Docker network QuestDB joins in managed mode, so the companion signalk-grafana reaches it by container DNS; host-port publication is separate ("Bind to 0.0.0.0") |
-| Bind to 0.0.0.0               | `false`      | Publish QuestDB's ports on all interfaces (see Connectivity)                                                                                                                    |
-
-### QuestDB updates
-
-The panel's **update check** compares the running QuestDB with its latest stable
-release. **Update** pulls that release, recreates the container on it and saves
-it as the image version.
-
-After the plugin itself was updated, and at its very first start, the plugin
-also checks once on its own, in the background. It does so only in managed mode,
-and only where the image version is a tag that moves, such as the default
-`latest`. If QuestDB runs an older release than the latest stable one, the
-plugin pulls the tag again and recreates the container on it. The image version
-stays `latest`. Without this check, QuestDB on `latest` would stay on whatever
-release the first install pulled, because signalk-container pulls an image only
-when it is missing. An image version that names one release, such as `9.4.3`,
-stays as it is, and so does one that **Update** saved while the check was
-waiting. QuestDB is down while it starts again, as with **Update**. Progress
-goes to the debug log, and failures go to the server log. A recreate that fails
-half way and leaves QuestDB stopped or without a container is followed by a
-start, as the plugin's own start would do. Only if that fails too does the
-status line say so.
-
-With [signalk-internet](https://github.com/meri-imperiumi/signalk-internet)
-enabled, the check waits for a free link. On a metered, captive or offline link
-it asks again from time to time, for as long as the plugin runs. Without
-signalk-internet, the link counts as free. Telling the two apart needs
-signalk-container 1.38.0 or later; with an older one, the check does not run. A
-failure to read the running or the latest version is also tried again later.
-
-The check is done once QuestDB runs the latest release, or once its container
-runs the image the pull brought, even if QuestDB still reports an older release
-afterwards. It then waits for the next plugin update. The next start checks
-again wherever the container keeps its old image, or where the new one cannot
-be told from the old:
-
-- the plugin stopped first;
-- the download failed, or something else failed before the recreate;
-- an InfluxDB import, a GPX import or an export was running;
-- the string table was being rebuilt;
-- the container was stopped, which the check leaves as it is;
-- the container was wedged;
-- the tag on Docker Hub did not name the latest release yet.
-
-A tag that never reaches the latest release, such as `9` while the latest is
-`10.1.0`, therefore costs a pull and a look at GitHub at every start.
-
-## Recording
-
-### Schema
-
-Three tables, all with WAL mode, daily partitioning, and deduplication:
-
-| Table              | Purpose        | Columns                                                                                                    |
-| ------------------ | -------------- | ---------------------------------------------------------------------------------------------------------- |
-| `signalk`          | Numeric values | `ts`, `path` (SYMBOL), `context` (SYMBOL), `source` (SYMBOL), `value` (DOUBLE)                             |
-| `signalk_str`      | String values  | `ts`, `path` (SYMBOL), `context` (SYMBOL), `source` (SYMBOL), `value_str` (VARCHAR), `value_kind` (SYMBOL) |
-| `signalk_position` | Positions      | `ts`, `context` (SYMBOL), `source` (SYMBOL), `lat` (DOUBLE), `lon` (DOUBLE)                                |
-
-What retention keeps of the own vessel lives in two more tables,
-`signalk_archive` and `signalk_position_archive`, with the same columns and
-monthly partitions (see Retention and the archive).
-
-The own vessel is stored as context `self`, whatever its identity; other vessels
-keep their `vessels.urn:…` context. The history API translates `self` back to
-`vessels.self`.
-
-`ts` is the **server receive time**, not the timestamp a source claims. Marine
-sources carry independent clocks, and storing their timestamps makes commits
-land out of order — QuestDB then rewrites partition tails on every merge
-(observed as >3000x write amplification). Receive time keeps ingestion
-append-only; the millisecond difference is far below the sampling resolution,
-and a device with a broken clock gets more accurate history, not less.
-
-`source` is the delta's sourceRef — which receiver produced the row. Two GPS
-units feeding the same server interleave in storage, and without the column a
-track drawn from history zigzags between them. Rows recorded before the column
-existed have `source` null; they replay unattributed and cannot be filtered.
-
-### What is recorded
-
-Every value on the stream is routed by its type: numbers go to `signalk`,
-strings and booleans to `signalk_str` (booleans tagged `value_kind=boolean`),
-and `navigation.position` — that path only — to `signalk_position`. A vessel's
-name, which arrives as an empty-path `{name}` delta, is stored in `signalk_str`
-under the path `name` with `value_kind=identity`, and replayed in its original
-shape. An object value is recorded as its scalar leaves, one level deep:
-`navigation.attitude {roll, pitch, yaw}` becomes `navigation.attitude.roll` and
-so on, and another lat/lon object such as `navigation.anchor.position` becomes
-`.latitude` and `.longitude` leaves rather than rows in the track table. Nested
-objects — notifications, resource documents — are not descended into; a value no
-table can hold is skipped and reported in the status endpoint and the panel
-rather than dropped silently.
-
-A number that means "no reading" is skipped and reported the same way: `NaN`
-and `±Infinity`, and the NMEA 2000 "no data" marker on AC real, apparent and
-reactive power (PGNs 65007–65029). Those fields cannot encode anything below
-−2 000 000 000 W, but canboatjs decodes "no data" as −2 000 000 001 W, so real
-and reactive power below that floor are dropped; real negative power, such as
-export, is unaffected. Apparent power reaches the stream multiplied by the power
-factor, which can move the marker anywhere below zero — and apparent power is
-never negative, so any negative apparent power is dropped. Connections using the
-canboat WASM decoder do not produce the marker at all.
-
-**Sampling.** Each path is written at most once per **Default sampling rate**
-(2000 ms), with per-path overrides by glob for the few paths that need finer
-resolution and the many that need less. The first value after a window opens
-is written and the rest of the window is dropped, not averaged, so a gust
-between two samples is not in the history. A rate of 0, as default or per
-path, removes the window: `environment.wind.* 0` records every change of the
-wind at wire speed while everything else stays at the default. An unchanged
-value is still written only once per **Unchanged-value heartbeat** (see
-Change-only recording), except `navigation.position`, which is written even
-when unchanged; set the heartbeat to 0 as well to record every update. Wire
-speed costs rows: a wind sensor at 10 Hz writes about 1.7 million a day for
-angle and speed, and every path of a busy NMEA 2000 boat at wire speed can be
-a few hundred rows a second. **Record own vessel** and **Record AIS targets**
-switch the two kinds of context independently, and **Path patterns** exclude
-(or include only) paths by glob.
-
-**Source priorities.** When Signal K's source priorities rank the sources of a
-path, only the preferred source is recorded, the same value instruments show.
-Paths without a priority are recorded from every source. **Record all sources**
-records the sources a priority hides as well, for calibrating a sensor or
-comparing two GPS receivers. Each source is then sampled on its own, so a path
-with three sources writes up to three times the rows; with the option off, the
-sources of a path share one sampling window.
-
-**Change-only recording.** Most paths on a boat do not change from one sample
-to the next: a switch that flips twice a day, an energy counter at rest, a tank
-nobody is drawing from, a notification's state. Sampling alone would write each
-of them at the sampling rate forever, and those repeats make up most of the
-rows. So a value equal to the last one written for its path, vessel and source
-is skipped until the **Unchanged-value heartbeat** is due, and then
-written once more. Every change is written as it arrives, subject to the
-sampling rate.
-
-The heartbeat is what keeps the gaps readable. A path that is still being
-updated has a row at least once per heartbeat, so a longer silence means the
-data really stopped:
-
-- **History API (v2).** A resolution bucket with no row takes the last sample
-  before it, for up to twice the heartbeat (see History API). A switch that has
-  been on for an hour reads as on for the whole hour, not as one point and
-  then nothing.
-- **Stale data.** When Signal K marks a path stale it sends `null` for it.
-  Nothing is written for the `null`, so the history shows the last value for at
-  most twice the heartbeat and then a gap. The next reading after that is
-  always written, even if it equals the one before the gap.
-- **Direct SQL and Grafana** see the rows as stored. Add `FILL(PREV)` to a
-  `SAMPLE BY` query to carry values across buckets.
-- **Averages** weight rows, not time. A bucket in which a value changes often
-  and then holds steady averages mostly over the changes; `min`, `max`, `first`
-  and `last` are exact.
-- **Startup restore** replays motion values recorded within **Restore max
-  age**. Keep the heartbeat below that, or a vessel lying still may come back
-  with its position but without its speed, course and heading.
-
-`navigation.position` is written even when unchanged. A heartbeat of 0 does the
-same for every path: each sample the sampling rate lets through is written. With
-a sampling rate of 0 as well, that is every update.
-
-**Batching.** Samples are buffered and committed every **Write batch interval**
-(5 s), one WAL transaction per table per commit. The buffer holds 100 000 lines;
-if QuestDB is unreachable for longer than that covers, the oldest lines are
-dropped and counted. A clean Signal K shutdown flushes first; a hard crash loses
-at most one interval.
-
-**Compression** applies LZ4 or ZSTD to the WAL segments on disk (and to
-Parquet exports).
-
-### Retention and the archive
-
-Retention drops whole daily partitions once they lie entirely before the
-configured number of days; 0 keeps everything. QuestDB cannot delete single
-rows, so keeping part of a day means copying it out first. Before a day of
-`signalk_position` or `signalk` is dropped, the own vessel's rows that the
-settings ask to keep are copied into an archive table, thinned to one row per
-**Archive resolution** interval, and the day is dropped only once QuestDB has
-applied the copy. A copy that fails leaves the day in place for the next run.
-
-| Table                      | Holds                                                           |
-| -------------------------- | --------------------------------------------------------------- |
-| `signalk_position_archive` | Own-vessel positions (**Keep own track**, on by default)        |
-| `signalk_archive`          | Own-vessel numeric paths matching **Archived own-vessel paths** |
-
-Both have the columns and dedup keys of their base table and are partitioned by
-month. Retention never drops from them, other vessels are never archived, and
-strings are not archived. The History API reads the archive together with the
-base tables for any range it covers, so a track or a depth series continues
-past the retention boundary without a seam. An archived row is the first sample
-of its interval and is stamped on the interval's start.
+- Signal K server 2.31 or newer, on Node.js 24 or newer
+- The [signalk-container](https://github.com/dirkwa/signalk-container) plugin,
+  with Podman 5.4 or newer (what Debian 13 "trixie" ships) or Docker. Not needed
+  if you run QuestDB yourself.
 
-Size, measured: a position row takes 32 bytes, so a vessel recorded around the
-clock fills about 100 MB a year at a 10 s resolution, 34 MB at 30 s.
+Which signalk-container version brings what is in
+[Deployment](doc/deployment.md#requirements).
 
-## History API
-
-### v2 (REST — `/signalk/v2/api/history/`)
-
-Registered via `app.registerHistoryApiProvider()`. Supports all aggregate
-methods:
-
-| Method         | QuestDB mapping                                                                 |
-| -------------- | ------------------------------------------------------------------------------- |
-| `average`      | `avg(value)`                                                                    |
-| `min`          | `min(value)`                                                                    |
-| `max`          | `max(value)`                                                                    |
-| `first`        | `first(value)`                                                                  |
-| `last`         | `last(value)`                                                                   |
-| `mid`          | `(min + max) / 2`                                                               |
-| `middle_index` | the middle row of each bucket, by time — a recorded value, not a computed one   |
-| `sma`          | the `average` bucket, then a moving window over N buckets (`sma:N`, default 5)  |
-| `ema`          | the `average` bucket, then an exponential moving average (`ema:α`, default 0.2) |
-
-A sample for `sma` and `ema` is one row of the series the window runs over: a
-resolution bucket (its average) when the request names a resolution, a raw row
-otherwise. So `sma:5` at `resolution=180` is a 15-minute moving average, on the
-same grid as every other column in the response. An empty bucket keeps its place
-in the window: the average there is of what the window still holds, and a value
-leaves it N buckets after it arrived. `ema` carries its last value across an
-empty bucket instead.
-
-Buckets start at the request's `from` and cover the whole range, empty ones
-included. A bucket with no row is not always missing data. Change-only recording (see
-Recording) writes an unchanged value only once per heartbeat, so an empty bucket
-takes the last sample before it — in every aggregate, since every aggregate of a
-steady value is that value — for up to twice the heartbeat after the sample was
-taken, and never into a bucket that has not started yet. At the start of a
-range the sample comes from just before it. Past that window, and with the
-heartbeat set to 0, an empty bucket is `null`. This happens before `sma` and
-`ema`, so a held value counts as a sample. `middle_index` and
-`navigation.position` return recorded rows only.
-
-A path recorded in radians — a heading, a course, a wind angle; `units: rad` in
-the server's metadata for the path — is averaged as a vector: `average`, `sma`
-and `ema` take the angle of the mean sine and cosine, so 359° and 1° average to
-0°, not 180°. The result keeps the path's convention, `[0, 2π)` for a heading or
-`(−π, π]` for an apparent wind angle; samples that cancel, 0° and 180° in equal
-measure, have no mean direction and give `null`. `min`, `max` and `mid` stay
-arithmetic.
-
-`navigation.position` answers `first`, `last` and `middle_index` — each a point
-the vessel was at. Any other method runs `first`, and the column's `method` says
-so. A text path (rows in `signalk_str`) answers `first`, `last` and
-`middle_index`. With a resolution, `average`, `min`, `max` and `mid` run `last`,
-the state in force at the bucket's end, and the column's `method` says so.
-`sma` and `ema` read the numeric table only, so a text path has no values for
-them.
+### Install
 
-A method name outside the table above fails the request with an error naming
-the path and the method.
+1. Install **signalk-container** from the App Store and enable it.
+2. Install **signalk-questdb** and enable it. On its first start it downloads
+   QuestDB, which can take a few minutes on a slow connection, starts it and
+   begins recording.
+3. Open the plugin's config panel. The status card shows **Running** and
+   **Recording**.
 
-Query example:
+![The config panel with its status card](doc/screenshots/signalk-questdb-1.png)
 
-```http
-GET /signalk/v2/api/history/values?paths=navigation.speedOverGround&duration=PT1H&resolution=60
-```
+To use a QuestDB you run yourself, switch **Manage QuestDB container** off and
+enter its host and ports.
 
-A `navigation.position` value is a `[longitude, latitude]` pair — GeoJSON order,
-as the History API defines it and as signalk-to-influxdb2 and signalk-parquet
-return it — not the `{latitude, longitude}` object of the data model; a bucket
-without a fix is `null`. The v1 playback API is unaffected: it replays deltas,
-which carry the object.
-
-**Limits.** A request may produce at most 1 000 000 sample buckets across all of
-its columns (range ÷ resolution × columns); a finer one is rejected with an
-error that says so — use a coarser resolution or a shorter range. Without a
-resolution a column is its raw rows, capped at 10 000 (50 000 for
-`middle_index`).
+### Make it the history provider
 
-#### Reading one source
+Apps ask Signal K for history, and Signal K asks its default history provider.
+Choose this plugin once under **Data → Preferences → Default History
+Provider**; Signal K remembers the choice.
 
-Append `|<sourceRef>` to a path to read one source's rows only (server 2.29+,
-[signalk-server#2737](https://github.com/SignalK/signalk-server/pull/2737)). The
-same path may appear once per source, giving one column per receiver:
+## Everyday settings
 
-```http
-GET /signalk/v2/api/history/values?paths=navigation.position|gps.main,navigation.position|gps.backup&duration=PT1H
-```
+Every setting is in the plugin's config panel. These are the ones you are most
+likely to change; all of them are in [Configuration](doc/configuration.md).
 
-Without a sourceRef a path returns all sources mixed. A source-bearing column
-reports its source as `$source` in its `values` entry — the key
-[signalk-server#2817](https://github.com/SignalK/signalk-server/pull/2817)
-settled on, and what v1 playback has always emitted — while the request side
-keeps the name `sourceRef`.
+### How often is data recorded?
 
-#### sourcePolicy=all
+A value that changes is written at most once per **Default sampling rate**,
+2 seconds by default. A value that stays the same is written again only once
+per **Unchanged-value heartbeat**, 5 minutes by default, which keeps the
+database small. Your position is the exception: it is written every 2 seconds
+even while the boat lies still.
 
-`sourcePolicy=all`
-([signalk-server#2817](https://github.com/SignalK/signalk-server/pull/2817))
-asks for every source separated without naming them up front. Each path that
-does not already specify a source is expanded into one column per source that
-actually recorded it in range, with the source in the `$source` field of the
-column's `values` entry:
+### How do I record the wind at full speed?
 
-```http
-GET /signalk/v2/api/history/values?paths=navigation.speedOverGround&duration=PT1H&sourcePolicy=all
-```
+Add a line to **Per-path sampling rates**: a path pattern, then the rate in
+milliseconds. A rate of 0 records every change:
 
-```jsonc
-"values": [
-  { "path": "navigation.speedOverGround", "method": "average", "$source": "gps.aux"  },
-  { "path": "navigation.speedOverGround", "method": "average", "$source": "gps.main" },
-  { "path": "navigation.speedOverGround", "method": "average" }  // unattributed rows
-]
+```text
+environment.wind.* 0
 ```
 
-Named sources come first, sorted, so column order is stable between requests.
-Rows whose source is unset — recorded before the `source` column existed, or
-from a delta that carried none — form their own trailing column with no
-`$source` claim.
-
-**Off by default**, behind the plugin setting that allows the policy. Expansion
-multiplies the work one request can ask for: a path recorded by four receivers
-becomes four queries and four columns, which on a Pi-class host turns a cheap
-request into an expensive one. The sample-bucket cap counts expanded columns, so
-a resolution that is fine unexpanded may be rejected once expanded.
-
-Two ceilings bound the fan-out, and a request past either is rejected rather
-than truncated — returning some of the asked-for series without saying so would
-be worse than refusing a request that is too broad. A single path may expand
-into at most 16 columns: recorded by more sources in range, it fails the
-request, and the error names the count, the ceiling and the way out (name the
-sources with `path|sourceRef`, or ask for a shorter range). A whole request is
-bounded at 64 columns, with its own error naming the limit.
-
-An explicit `path|sourceRef` stays a filter and takes precedence over the
-policy, matching the upstream contract. Requests that name their sources are
-unaffected by the setting either way.
-
-### v1 (WebSocket playback)
-
-Registered via `app.registerHistoryProvider()`. Supports playback at
-configurable speed multipliers using chunked reads from QuestDB. Replayed
-updates carry the recorded sourceRef as `$source`, one update per source, so
-consumers see the same attribution the live stream had.
-
-## Console webapp
-
-QuestDB ships its own web console — a full SQL workbench with schema browsing,
-query history and CSV import. With **QuestDB console webapp** enabled (the
-default) the plugin appears in the Signal K webapp list and serves that console
-inside the admin UI, with Signal K's own navigation panel still in place. The
-**[Web Console user guide](doc/web-console.md)** walks through it with
-ready-to-paste queries: speed in knots, temperatures in °C, distance per day,
-track export, and more.
-
-**It is admin only.** The console is proxied at
-`/plugins/signalk-questdb/console/`, and Signal K gives plugin routes an
-admin-only default — the plugin deliberately does not relax that. A non-admin
-who opens the webapp sees the page frame and an authorization error, never the
-console.
-
-**It can modify data.** Unlike the read-only `/query` endpoint, which rejects
-anything that is not a `SELECT`, the console is a real SQL client: it can drop
-tables and delete recorded history. That is the point of having it, and it is
-what an administrator would use it for — but it is worth knowing before handing
-someone an admin account.
-
-If you would rather not serve it at all, switch the option off and the route is
-not registered.
-
-The console is also reachable directly on QuestDB's HTTP port. On a bare-metal
-Signal K with "Bind to 0.0.0.0" off, signalk-container binds it to loopback at
-the address the status card shows (the port is the one it bound, not necessarily
-9000); from another machine, tunnel that port over SSH. It is 9000 on the host
-only with "Bind to 0.0.0.0" on. In managed mode a containerized Signal K reaches
-QuestDB over the container network, so there is no host port to open unless
-"Bind to 0.0.0.0" is on; in external mode the console is at the configured
-QuestDB host and HTTP port, whatever the deployment. The HTTP port carries
-QuestDB's REST API and ingestion too, with no authentication; enabling "Bind to
-0.0.0.0" to reach it from the LAN publishes all of that, which is what the
-webapp route exists to avoid.
-
-## Startup restore
-
-Signal K's data model lives in memory only. After a server restart every vessel
-is gone until it transmits again — for AIS that means roughly 30 seconds for a
-Class B target, up to 3 minutes for a Class A at anchor, and about 6 minutes
-before names arrive. The chart fills in gradually instead of showing the traffic
-that was there a moment ago.
-
-With **Restore vessels on startup** enabled, the plugin replays each vessel's
-last recorded position (plus course, speed, heading and identity — name,
-dimensions and AIS ship type) from QuestDB as soon as it connects, so the chart
-is populated immediately, with each target drawn in its ship-type colour rather
-than the default.
-
-**A restored position is where a vessel _was_, not where it is now.** Nothing is
-dead-reckoned forward. The plugin presents these values as the history they are
-rather than passing them off as live: each delta carries its original recorded
-timestamp, so a chart plotter ages it out under the same staleness rules it
-applies to any other target, and the deltas are tagged
-`$source: signalk-questdb.restore`.
-
-Values outside the **Restore max age** window are not replayed at all. The
-9-minute default matches Freeboard's AIS expiry, so a restored target is one
-that would still have been on the chart had the server never stopped. Raising it
-puts progressively staler positions in front of you — treat the window as a
-collision-avoidance setting, not a convenience one.
-
-Only navigation and identity paths are replayed, so stale tank levels and engine
-temperatures are not resurrected as though they were current readings. A vessel
-with no position in the window is skipped, as is one that has already
-transmitted since startup. Restore honours the recording toggles: with **Record
-AIS targets** off, no AIS target is restored.
-
-The config panel reports the outcome once startup finishes: **Vessels Restored**
-with the count, or **Restore failed** if the replay could not run. Neither
-appears when the option is off.
-
-## Migrating from InfluxDB
-
-The **InfluxDB Migration** section of the config panel copies history out of an
-existing InfluxDB into QuestDB. It supports InfluxDB 1.x (InfluxQL, what
-signalk-to-influxdb wrote) and 2.x (Flux, what signalk-to-influxdb2 writes).
-
-### Running an import
-
-1. **Detect** finds an InfluxDB on `localhost:8086`, or enter its URL —
-   `localhost` or a loopback/private-network IP literal; anything else is
-   refused.
-2. Supply credentials — an API token and organisation for 2.x, or
-   username/password for 1.x if authentication is enabled — and list the buckets
-   (2.x) or databases (1.x). Credentials are used for the import only, are never
-   written to the plugin's settings, and travel in the request body rather than
-   the query string (Signal K logs full request URLs).
-3. Pick a bucket. The panel then lists the vessels it holds (**Listing the
-   vessels in the bucket…** — on a large bucket this takes a while) and shows
-   **Which vessel in the source is this one**: that vessel's history is imported
-   as `self`. The choice is made for you when the source says:
-   signalk-to-influxdb2 tags its own vessel's points with `self=true`, so that
-   vessel is preselected and marked "(recorded as the own vessel)" — as is the
-   one matching this server's identity, marked "(this server)", or the only one
-   there is. A signalk-to-influxdb 1.x source carries no such tag, so there the
-   choice is yours whenever it holds several vessels and none is this server's
-   identity — the usual case, since the recording server's identity is usually
-   not this one's (a new install has a new one). Tick **Also import the other
-   vessels (AIS targets), each under its own context** to bring those along; the
-   history API serves them as it does live-recorded targets.
-4. Pick a time range and **Start import**. Progress is polled while it runs;
-   **Cancel** stops it, keeping the saved position (see Resuming).
-
-After the import, check the row counts on the status card or in the console
-(`SELECT count() FROM signalk WHERE source = 'influxdb-import'`), make sure this
-plugin is the default history provider (see Getting started), and disable the
-InfluxDB plugin once nothing else needs it — a consumer such as a track viewer
-reads whichever provider is the default, not both.
-
-### How the data maps
-
-- A measurement with the conventional `value` field becomes the Signal K path of
-  the same name. So do the typed fields signalk-to-influxdb 1.x writes
-  (`stringValue`, `boolValue`, `jsonValue`). A measurement with any other named
-  fields becomes `measurement.field` paths, so two fields cannot overwrite each
-  other.
-- Numbers go to `signalk`, strings and booleans to `signalk_str` (booleans
-  tagged `value_kind=boolean`). A `jsonValue` is decoded first: an object is
-  recorded as its scalar leaves, the same as live data.
-- Positions go to `signalk_position`, whether stored as a `jsonValue`
-  (signalk-to-influxdb 1.x) or as a `lat`/`lon` or `latitude`/`longitude` field
-  pair (signalk-to-influxdb2).
-- Both writers tag every point with the **context** of the vessel it belongs to;
-  the vessel you picked is imported as `self`, the others under their own
-  contexts or left out.
-- Rows keep their **original timestamps**, to the microsecond (InfluxDB stores
-  nanoseconds; QuestDB's `TIMESTAMP` holds microseconds), so imported history
-  sorts and aggregates alongside live data.
-- History is **streamed**: read in batches and written before more is asked for,
-  pausing while QuestDB catches up. Memory use stays flat however dense the
-  source is, so a large import is safe on a small board.
-- Every imported row is tagged `source=influxdb-import`, which makes it
-  distinguishable from live recording — and because the tables deduplicate on
-  `(ts, path, context, source)`, **re-running the same range overwrites rather
-  than duplicating**.
-- Anything that cannot be mapped (a gap, an unsupported value type, a
-  `jsonValue` that is not valid JSON, a latitude with no matching longitude) is
-  counted in the run's `skipped` total rather than being dropped silently.
-
-### Resuming an interrupted import
-
-A large import runs for hours. If it stops part-way — cancelled, failed, or the
-Signal K server restarted — the panel shows what was interrupted and offers
-**Resume import**, which continues from the saved position instead of starting
-again at the first measurement. It needs only the credentials, if InfluxDB asks
-for any: the source and range are remembered, credentials never are. Starting
-the identical import again the ordinary way continues it too. **Discard saved
-position** starts over.
-
-A position is saved only once QuestDB has been asked for the newest row written
-before it and has it — ILP gives no acknowledgment, so the plugin reads the row
-back — and once the position is about a minute old. The first keeps a stalled or
-restarted QuestDB from letting the position past rows it never took, however
-long the stall. The minute covers what a committed row still has to survive: a
-crash of either process, and on a power cut the kernel's default write-back
-timing for a QuestDB running on its default `nosync` — an assumption, not a
-guarantee, which is why an external QuestDB should run with `sync` like the
-managed container does (see Durability on power loss). The position file is
-synced and renamed into place, so a power cut leaves the previous position or
-the new one, never a damaged one; should none survive, the import starts again,
-which is always safe. A position is a window boundary — the import reads each
-measurement a window at a time, a day of source data by default — so a resumed
-import repeats the window it was in plus up to a minute, which the deduplication
-above makes harmless. An import shorter than a window saves nothing, and is
-simply run again.
-
-### Imports made before 2.1.5
-
-Imports from a signalk-to-influxdb 1.x database made before 2.1.5 filed every
-string, boolean and position under a path suffixed with the InfluxDB field name
-— `navigation.state.stringValue`, `steering.autopilot.engaged.boolValue`,
-`navigation.position.jsonValue` — which nothing reads. Running the import again
-writes them under their real paths but cannot remove the old rows: the path is
-part of the deduplication key, and QuestDB has no row delete. When such rows
-exist, the panel says how many and offers **Remove these rows**, which rebuilds
-`signalk_str` without them. Recording continues while it runs; only the swap of
-the rebuilt table for the old one holds the writer, for a few seconds, and rows
-recorded during the rebuild are carried across. Should recording be fast enough
-to fill the writer's buffer during those seconds, the samples it could not keep
-are counted and the panel says how many. Run the import again afterwards.
+Everything else stays at the default. A value that repeats is still written
+only once per heartbeat. To record repeats too, set the heartbeat to 0; that
+applies to every path, each of which then writes at its sampling rate even when
+nothing changes.
 
-## Importing a GPX track
+Full speed costs space and needs a capable machine: a wind sensor at 10 Hz
+writes about 1.7 million rows a day for angle and speed. Use it for the few
+paths that need it.
 
-A chartplotter's exported track goes into the position archive (see Retention
-and the archive), where retention never drops it, under the source
-`import.gpx`. An export can be gigabytes, a four-year TimeZero export is
-1.5 GB, so the file is sent in slices and parsed as it arrives. Memory stays at
-one slice, whatever the file's size.
+### How long is data kept?
 
-In the configuration panel, **GPX track import** reads the chosen file in the
-browser first, which takes seconds even for a gigabyte, and lists its tracks by
-name with their points and period. It proposes the own vessel's tracks: the
-largest group whose name is not an AIS target's, and any other name that
-recurs. It flags a group that lay more than a kilometre from a larger one at
-the same time, since that was another vessel. Nothing is uploaded until
-**Import** is pressed; the import then shows its progress and can be aborted.
+Forever, unless you set **Retention (days)**; older days are then deleted. Your
+own track is kept anyway, thinned to one position every 10 seconds (**Keep own
+track** and **Archive resolution**). That is about 100 MB for a year of
+recording around the clock. To keep other paths of your own boat the same way,
+list them under **Archived own-vessel paths**, for example
+`environment.depth.*`.
 
-- **Which tracks.** An export mixes the own track with AIS targets and with
-  passages imported into the plotter from elsewhere, and GPX does not say whose
-  a track is. The import takes the names of the tracks to import (`tracks`);
-  every other track is skipped and listed in the summary. In a TimeZero export
-  the own track is named `Own Ship`, or the plotter's translation of it, such
-  as `Eigenes Schiff`.
-- **Thinning.** Points are thinned to the **Archive resolution**: each is
-  stamped on the start of its interval, and the archive keeps one row per
-  interval. 11.5 million points over four years become 3.2 million rows,
-  about 100 MB, at 10 s.
-- **Overlap.** With `fillGapsOnly` (the default), a point is left out where the
-  vessel already has recorded positions in the same five minutes. Two
-  receivers' fixes interleaved in one series draw a sawtooth.
-- **Doing it again.** Importing the same file again changes nothing, and a
-  stopped import can simply be restarted: the archive's dedup keys absorb the
-  rows already written.
-- **Positions only.** The import takes a position and a time from each point;
-  depth, temperature or speed in a point's extensions are not imported. Depth
-  along the track beyond retention comes from live recording: list
-  `environment.depth.*` under **Archived own-vessel paths**, and retention keeps
-  it from then on.
-- **Strict reading.** Coordinates must be decimals within range and times
-  `xsd:dateTime`; anything else is counted as invalid. A longitude written just
-  past ±180, as TimeZero does at the antimeridian, is wrapped. A file that is
-  not well-formed XML stops the import at the error, keeping the rows before
-  it.
+### AIS targets are gone after a restart
 
-## Export and backup
+Signal K keeps live data in memory only, so after a restart the chart stays
+empty until each vessel transmits again, which can take minutes. Switch on
+**Restore vessels on startup**: the vessels recorded in the last 9 minutes
+(**Restore max age**) come back at once, with name and ship type. They keep the
+time they were recorded at, so the chart plotter ages them out like any other
+target.
 
-**Data Export** in the panel, and
-`GET /plugins/signalk-questdb/api/export?from=…&to=…&format=parquet|csv`,
-download the `signalk` numeric table for a date range (both bounds required) as
-Parquet — QuestDB's native export, compressed with the configured codec — or
-CSV.
+### Recording less
 
-`GET /plugins/signalk-questdb/api/full-export/:table?from=…&to=…` streams one
-whole table as Parquet, for snapshot and backup tooling that wants the full
-content sliced into dedup-friendly shards (kopia, for instance). Tables:
-`signalk`, `signalk_str`, `signalk_position`, `signalk_archive`, `signalk_position_archive`
-(`/full-export/tables` lists them). The archive tables hold the own vessel's
-history beyond retention and every imported track, so a backup that is meant to
-outlive retention needs them too.
+**Record own vessel** and **Record AIS targets** switch the two off separately.
+Under **Path filtering**, patterns such as `propulsion.*` leave out the paths
+they match, or, in `include` mode, record only those.
 
-- `from` and `to` are optional but must be set together: omit both for the whole
-  table, or pass both as ISO 8601 timestamps for a half-open `[from, to)` window
-  — no row appears in two adjacent windows.
-- Repeated query parameters (`?from=A&from=B`) and empty strings (`?from=`) are
-  rejected with HTTP 400; silently downgrading to a full-table export would hide
-  bugs in the caller.
-- Format and compression follow the plugin's compression setting (LZ4_RAW or
-  ZSTD), the same as `/export`.
+### Comparing two sensors
 
-## REST endpoints
+When Signal K's source priorities rank the sources of a path, only the preferred
+one is recorded, the value your instruments show. Switch on **Record all
+sources** to record the others too, for example to calibrate a sensor or to
+compare two GPS receivers. Each source then writes its own rows.
 
-All under `/plugins/signalk-questdb/api/`; the console is mounted beside them at
-`/plugins/signalk-questdb/console/`. Plugin routes are admin-only by Signal K's
-default.
+## Common tasks
 
-| Method | Path                                     | Description                                                                               |
-| ------ | ---------------------------------------- | ----------------------------------------------------------------------------------------- |
-| GET    | `/status`                                | QuestDB health, row counts, active paths, restore outcome, WAL state, unstorable paths    |
-| GET    | `/query?sql=...`                         | Read-only SQL proxy (DDL/DML blocked)                                                     |
-| GET    | `/paths`                                 | All recorded paths with row counts and time range                                         |
-| GET    | `/versions`                              | QuestDB releases from GitHub (for the version picker)                                     |
-| GET    | `/update/check`                          | Compare the running version against the latest release                                    |
-| POST   | `/update/apply`                          | Pull the latest image, recreate the container, reconnect                                  |
-| GET    | `/wal-diagnosis`                         | Suspended tables with the engine's reason and a bounded skip plan (see WAL suspension)    |
-| POST   | `/resume-wal`                            | Lossless `RESUME WAL` of the suspended tables                                             |
-| POST   | `/resume-wal/skip`                       | Guided skip of the unreadable segment — explicit, quantified, and only on request         |
-| POST   | `/purge-data`                            | Remove the QuestDB container and delete all its data (rootless-Podman-safe)               |
-| GET    | `/migration/detect`                      | Detect InfluxDB (supports `?url=` for a remote one)                                       |
-| POST   | `/migration/buckets`                     | List buckets (2.x) or databases (1.x); credentials in the body, never the query string    |
-| POST   | `/migration/measurements`                | List measurements and their field keys in a bucket/database                               |
-| POST   | `/migration/contexts`                    | List the vessels a bucket/database holds, and which is this server's                      |
-| POST   | `/migration/start`                       | Start an import, or continue a stopped one with `{"resume": true}`; returns immediately   |
-| GET    | `/migration/status`                      | Progress and state of the current/last import, and any stopped import that can resume     |
-| POST   | `/migration/cancel`                      | Cancel the running import; any saved position is kept                                     |
-| POST   | `/migration/discard`                     | Forget a stopped import's saved position, so the next start begins again                  |
-| GET    | `/migration/legacy-rows`                 | Count rows an import made before 2.1.5 filed under suffixed paths                         |
-| POST   | `/migration/legacy-rows/remove`          | Rebuild the string table without those rows; recording continues meanwhile                |
-| GET    | `/export?from=...&to=...&format=parquet` | Parquet or CSV export of the `signalk` numeric table (date range required)                |
-| GET    | `/full-export/tables`                    | List the tables the full-export route serves                                              |
-| GET    | `/full-export/:table?from=...&to=...`    | Stream a table as Parquet; optional half-open `[from, to)` window for slicing into shards |
-| GET    | `/gpx-import`                            | The current or last GPX import                                                            |
-| POST   | `/gpx-import`                            | Announce an import: `{filename, size, tracks, fillGapsOnly}`, see Importing a GPX track   |
-| PUT    | `/gpx-import/:id?offset=N`               | One slice of the file, as `application/octet-stream`, at most 8 MiB                       |
-| POST   | `/gpx-import/:id/finish`                 | End the import once every byte is in; answers when QuestDB has applied it                 |
-| DELETE | `/gpx-import/:id`                        | Abort the import; what it wrote stays                                                     |
+### Import history from InfluxDB
 
-## Deployment
+In the config panel, open **InfluxDB Migration**:
 
-### Connectivity
+1. **Detect** finds an InfluxDB on this machine, or enter its address.
+2. Enter its credentials, if it needs any, and pick the bucket (InfluxDB 2.x)
+   or database (1.x).
+3. Check which vessel in it is your own boat. Tick the box below to import the
+   AIS targets as well.
+4. Pick a time range and **Start import**.
 
-In **managed mode** the plugin does not need `QuestDB host` to be correct for
-your deployment — signalk-container resolves the right address automatically,
-whether Signal K runs on bare metal or is itself containerized:
+A large import runs for hours. If it stops, **Resume import** carries on where
+it left off. Afterwards, make sure this plugin is the default history provider,
+and switch the InfluxDB plugin off once nothing else needs it. More in
+[Import and export](doc/import-export.md).
 
-- **Bind to 0.0.0.0 = off (default).** QuestDB stays private. signalk-container
-  binds its ports to the host loopback (bare-metal Signal K) or attaches QuestDB
-  to Signal K's own container network (containerized Signal K), and the plugin
-  connects to whatever address it reports back. Nothing is exposed to the
-  network. QuestDB is also attached to the shared `Container network` so the
-  companion [signalk-grafana](https://github.com/dirkwa/signalk-grafana) plugin
-  still reaches it by container DNS. This is the recommended setup and fixes
-  connectivity for Signal K in a container.
-- **Bind to 0.0.0.0 = on.** QuestDB stays attached to the shared
-  `Container network`; the setting publishes its HTTP/ILP/PostgreSQL ports on
-  all host interfaces, using the configured port numbers. Enable this only to
-  reach QuestDB from another machine or from a Grafana running in a separate
-  Docker instance. When Signal K itself is containerized, the plugin probes
-  which address reaches the published ports — its own loopback first, then the
-  runtime's `host.containers.internal` gateway — and uses the one QuestDB
-  answers on.
+### Import a track from a chartplotter
 
-In **external mode** (`Manage QuestDB container` off) the plugin connects to the
-QuestDB you point it at via `QuestDB host` and the HTTP/ILP ports.
+Export the track as GPX on the plotter. In the config panel, **GPX track
+import** reads the file and lists its tracks, with the ones that look like your
+own boat already ticked. Check them and press **Import**. Imported tracks are
+kept for good, whatever the retention.
 
-### Data storage
+### Export data
 
-QuestDB's data root is the plugin's own data directory,
-`<config>/plugin-config-data/signalk-questdb`, whatever the deployment. Data
-survives container restarts, image upgrades, and plugin disable/enable cycles.
-How it reaches the container differs:
+**Data Export** in the config panel downloads the numbers recorded in a date
+range as Parquet or CSV. For backups of every table, see
+[Export and backup](doc/import-export.md#export-and-backup).
 
-- **Bare-metal Signal K**: that directory, mounted at `/var/lib/questdb`.
-- **Signal K in a container with its config directory bind-mounted**: the exact
-  host path of that directory, mounted at `/var/lib/questdb`.
-- **Signal K in a container with its config directory on a named volume**: the
-  volume cannot be mounted from a subdirectory, so it is mounted whole at
-  `/signalk-vols/<volume>` and QuestDB is pointed at the plugin's directory
-  inside it (`QUESTDB_DATA_DIR`). A database found at the volume's root — where
-  a whole-volume mount put it — is moved into the plugin's directory on start;
-  both are the same volume, so nothing is copied. A purge deletes the directory
-  from the Signal K process, since the runtime cannot mount the volume by Signal
-  K's path; where QuestDB's user owns it and the Signal K user cannot delete it,
-  the purge says so and the directory is deleted by hand.
+### Look at the data with SQL
 
-QuestDB's entrypoint makes its data root its own on start, so the plugin keeps
-the one file it writes alongside — the import checkpoint,
-`signalk-questdb.influx-import-checkpoint.json` — next to the directory rather
-than in it. A purge removes it, whether or not the data could be deleted.
-
-### Durability on power loss
-
-The managed container runs QuestDB with `cairo.commit.mode=sync`.
-
-QuestDB's default is `nosync`: nothing on the ingest path is ever fsynced, and
-durability is left to the OS page cache. On a power cut an arbitrary subset of
-recent writes survives — including the case where a commit record reaches disk
-while the row data it describes does not. QuestDB then fails while _opening_
-that table's partition, which suspends the table permanently: neither a lossless
-`RESUME WAL` nor a bounded `RESUME WAL FROM TXN` skip can get past it, because
-the failure happens before any transaction is read. Repair means hand-patching
-the commit record.
-
-With `sync`, partition columns are fsynced before the commit record is written
-and WAL segments are fsynced on commit, so the stored state can never claim data
-that isn't durable. The cost is a few fsyncs per commit interval — writes are
-already batched (see **Write batch interval**), so at boat data rates it is
-negligible against losing a table.
-
-**Running an external QuestDB?** Set this yourself in `server.conf`:
-
-```ini
-cairo.commit.mode=sync
-```
-
-### WAL suspension
-
-QuestDB ingestion is two-stage: the sequencer durably commits every batch (the
-writer sees success), then a background job applies the transactions to the
-partitions. When that job hits an error it **suspends the table** — the
-sequencer keeps accepting rows, nothing is applied, queries serve stale data,
-and no client-visible error is raised anywhere. Left alone, recording stalls
-silently for as long as nobody looks.
-
-The plugin watches for this. A suspended table of its own (in external mode,
-only the three tables above and the two archive tables — never someone else's)
-is logged loudly, raised as
-the Signal K notification `notifications.signalk-questdb.walSuspended`, and
-shown on the status card as **WAL suspended**. Two causes need two remedies:
-
-- **Transient** — disk full since cleared, out of memory, file-descriptor
-  exhaustion: a plain `RESUME WAL` replays every pending transaction,
-  losslessly. The plugin does this by itself, once per stall point; the panel's
-  **Resume** button does the same on request, and the banner clears as the
-  backlog drains.
-- **Unreadable segment** — an unclean container stop mid-write corrupts the
-  in-flight segment's files, and every apply attempt fails on them within
-  milliseconds, so a plain resume re-suspends at once, forever. The only way
-  forward is `RESUME WAL FROM TXN` past the unreadable data, which loses what it
-  skips. The plugin prepares that but never applies it on its own:
-  **/wal-diagnosis** scrapes the engine log for the real reason (the
-  `wal_tables()` error fields often carry none, and a container recreate wipes
-  the engine's memory of it) and computes the minimal skip — the first
-  transaction of the segment after the stuck one — with the number of
-  transactions and the wall-clock window of ingestion it would drop. The panel
-  shows those numbers and offers **Skip**; if the stuck segment is the newest
-  one, skipping drops the entire pending backlog, and the panel says so.
-
-### Out of memory or not answering
-
-QuestDB caps the native memory it tracks below the container's memory limit
-and fails any allocation past the cap with `global RSS memory limit exceeded`.
-When something holds memory at the cap, queries fail, the WAL apply job then
-suspends the tables, and in the end QuestDB stops accepting HTTP connections. The JVM stays up throughout, so the
-container's restart policy never acts.
-
-In managed mode the plugin checks for this every minute while it runs, and
-restarts the QuestDB container when:
-
-- QuestDB recently reported its global memory limit, and over consecutive
-  checks its native memory is still held at that limit, it fails even the
-  health check with it, or it has stopped answering. A single large query
-  frees what it held when it ends, so it does not count, nor does a client that
-  keeps asking for one.
-- the container is running but QuestDB has answered nothing for several
-  minutes, over consecutive checks. A QuestDB busy with a long statement still
-  answers it in the end, and no check counts while a parquet export streams
-  from it.
-
-Before the restart the plugin writes QuestDB's message and its last error lines
-to the server log, shows the reason on the status card, and raises the Signal K
-notification `notifications.signalk-questdb.watchdog`. After the restart it
-logs the outcome. Once QuestDB answers again the notification returns to normal;
-when it answered right after the restart, the notification keeps the time and
-the reason. Restarts are rate limited, spaced apart and capped per few hours of
-the plugin running; past that the plugin logs that the restart is held back and
-keeps the alert up. A restart that fails, because stopping or starting the
-container failed, counts against none of these limits: the plugin tries again a
-few minutes later, since the container may be left stopped. It reports the
-first failure; while the retries keep failing, they go to the debug log only.
-
-The plugin leaves a container alone if someone else stopped it, and starts one
-that its own restart left stopped. One exception: a container stopped by hand
-while a stop of the plugin's is still pending, or right after a restart of the
-plugin's before the plugin has seen it running again, may be started again too.
-While a stop of the plugin's hangs, it sends no further stop. An external
-QuestDB is never restarted. A
-QuestDB that is already stuck when the plugin starts, as after a Signal K
-restart while it was stuck, is not covered: restarting the plugin stops the
-container and starts it afresh.
-
-The managed container runs with QuestDB's query caches off: history reads put
-their time range into the SQL, so the caches never got a hit, and on older
-QuestDB releases they held memory.
-
-### Performance (Pi and other low-power hosts)
-
-- **Default sampling rate** of 2000 ms limits each path to one write per two
-  seconds, keeping write volume modest on busy NMEA 2000 buses; per-path
-  overrides allow faster rates, or wire speed with 0, for critical paths while
-  slow-changing ones stay throttled.
-- **Change-only recording** skips a value that has not changed since it was last
-  written until the heartbeat is due. Paths that hold steady — which is
-  most of them — cost one row per heartbeat instead of one per sampling
-  interval, which cuts both disk use and WAL apply work (see Recording).
-- **Resource caps** of 768 MB RAM and 1.5 CPU cores (cgroup limits via
-  signalk-container) keep QuestDB from squeezing co-resident containers like
-  Grafana, mayara, or signalk-backup. The JVM heap gets half of the memory
-  cap, so total footprint (heap plus off-heap) is bounded, and a JVM that runs
-  out of heap exits so the container restarts instead of hanging. Set the
-  memory limit to empty or the CPU limit to `0` to lift the cap on roomier
-  hosts.
-- **Worker tuning** cuts idle CPU. QuestDB sizes its worker pools from the host
-  core count and busy-polls before sleeping — sensible on a server, wasteful on
-  a mostly idle boat database. The plugin pins its worker pools down and quiets
-  QuestDB's per-commit logging, keeping the messages that matter. One caveat
-  when upgrading the QuestDB image: an unrecognised setting is ignored silently,
-  so a property renamed between releases looks exactly like one that worked.
-- **Batching.** Each commit is one WAL transaction per table, and with
-  deduplicated tables the apply cost of a transaction grows with partition size
-  — frequent tiny commits eventually outpace what a Pi can apply and recording
-  stalls. Bigger batches keep the WAL healthy; the cost is that at most one
-  interval of buffered samples is lost on a hard crash.
-
-The console's Monitoring view flags any table whose 90th-percentile WAL
-transaction stays under QuestDB's recommended batch size (100 rows) as **"Small
-transactions — consider batching"**. `signalk_position` triggers this
-structurally: it holds exactly one path, so its share of every commit is just
-the handful of GPS fixes since the last one, while the other tables spread
-hundreds of paths across each transaction. At vessel data rates the alert is
-cosmetic; the numbers that matter are on the same page — **Write Amplification**
-near 1x, **Pending Rows** 0 and **Transaction Lag** 0 mean the WAL apply is
-healthy. Clearing the alert for the position table would need roughly 100 fixes
-per commit, a batch window in the minutes; the trade-off is up to one interval
-of buffered data lost on a hard crash, and history that is that much less live.
+Open the admin UI, go to **Webapps** and pick **signalk-questdb**: that is
+QuestDB's own console. Only admins can open it, and it can delete data, so
+stick to `SELECT`. The [Web Console guide](doc/web-console.md) has queries to
+paste, such as speed in knots or distance per day.
 
 ### Grafana
 
-Connect Grafana to QuestDB through the PostgreSQL data source (user `admin`,
-password `quest`, database `qdb`).
-
-- The companion [signalk-grafana](https://github.com/dirkwa/signalk-grafana)
-  plugin wires this up for you: it runs Grafana as a managed container and
-  reaches QuestDB by container DNS on the shared `sk-network`, so no port needs
-  publishing.
-- A Grafana **on the Signal K host** can reach the PostgreSQL port without
-  binding: with "Bind to 0.0.0.0" off, signalk-container binds it to loopback at
-  a host port of its choosing — `podman port <container>` shows which (the
-  container is `signalk-questdb` behind the `sk-` prefix). With "Bind to
-  0.0.0.0" on it is the configured port, 8812 by default.
-- A Grafana in a **separate Docker instance or on another machine** needs "Bind
-  to 0.0.0.0" and the host's LAN address (`192.168.0.122:8812`, say). That
-  publishes QuestDB's ports to your entire network; only enable it behind a
-  firewall you trust.
-
-Example query:
-
-```sql
-SELECT ts AS time, avg(value) AS sog
-FROM signalk
-WHERE path = 'navigation.speedOverGround'
-  AND context = 'self'
-  AND ts BETWEEN $__timeFrom() AND $__timeTo()
-SAMPLE BY $__interval
-```
-
-## Troubleshooting
-
-### 'Module "signalk-questdb" is not available' right after install or update
-
-Right after installing or updating the plugin, the admin UI can show
-`Module "signalk-questdb" is not available. Make sure the webapp is installed.`
-It is installed — the browser tab is still running the admin UI it loaded before
-the update, which looks for the panel bundle it knew then. A hard refresh of the
-tab (Shift-Reload) fixes it.
-
-### "Cannot apply a change: the QuestDB container cannot be stopped or removed"
-
-The plugin needed to recreate the QuestDB container to apply a change — a new
-image tag, a network or resource setting, a raised file-descriptor limit — but
-the container runtime refused to stop or remove the running container. The
-recreate is **deferred, not abandoned**: QuestDB keeps running and keeps
-recording, on its **previous** configuration. The plugin reports the error
-rather than a "Recording" status, because the change you asked for has not been
-applied and reporting success would be a lie.
-
-The usual cause on rootless Podman is an orphaned user namespace left behind
-when the Podman service restarted underneath a running container. The error text
-carries the specific remedy; the general one is to clear the stale namespace and
-remove the container so the plugin can recreate it:
-
-```bash
-podman system migrate                     # re-attaches orphaned user namespaces
-podman ps -a | grep questdb               # find the managed container's name
-podman rm -f <name-from-above>
-```
-
-Run all three as the **same OS user that owns the container**. Rootless and
-rootful Podman keep separate container stores, so running them as root (or as
-another user) will not find, and will not remove, a rootless container.
-
-Look the name up rather than assuming it: the plugin's container is
-`signalk-questdb` behind a prefix that defaults to `sk-` but is configurable
-(`SIGNALK_CONTAINER_NAMESPACE`), so it is not always `sk-signalk-questdb`.
-
-Then disable and re-enable the plugin, or restart Signal K. The plugin recreates
-the container with the settings that were pending. Recorded data lives under
-Signal K's data directory, not inside the container (see Data storage), so
-removing the container does not touch it.
-
-**Do not** mask `podman.socket` to work around this. That socket is how the
-plugin reaches Podman at all, so masking it turns a deferred recreate into a
-container the plugin cannot manage in any way. (`podman.service` is the unit the
-socket activates, and masking that has the same effect.) If the socket itself is
-unhealthy, restart it rather than masking it:
-
-```bash
-systemctl --user restart podman.socket
-```
-
-This error is reported with signalk-container 1.29.0 or later. On older versions
-the wedge still happens, but appears only as a repeating drift message in the
-Signal K server log with no plugin error to go with it.
-
-### "QuestDB keeps dropping the write connection — the container may be unhealthy or out of memory"
-
-This status appears when the plugin's ILP writer connects to QuestDB, gets
-dropped within a few seconds, and retries — repeatedly. It is QuestDB itself
-being unhealthy (typically OOM-killed and restart-looping), not a problem with
-the plugin or with a single table. The most common cause on a Raspberry Pi or
-other low-RAM host is that QuestDB is hitting a memory ceiling. There are two
-variants, and the cgroup check below tells them apart.
-
-**1. Is the cgroup `memory` controller delegated?**
-
-```bash
-cat /sys/fs/cgroup/cgroup.controllers
-#   ...memory...   present  -> memory delegation is available, so the cap can be enforced (see B)
-#   memory ABSENT           -> the cap is silently dropped (see A)
-```
-
-When Signal K's containers run under rootless Podman, the kernel only enforces a
-resource limit whose cgroup controller has been delegated to the user session.
-Many distributions delegate `cpu`, `cpuset`, `io`, and `pids` by default but
-**not** `memory`. signalk-container drops a limit whose controller is missing
-rather than failing the container, so a configured cap can silently have no
-effect.
-
-**A. `memory` not delegated — the cap is silently dropped.** QuestDB grows
-without bound and the host kernel's OOM killer eventually kills it under
-whole-system memory pressure; it restarts and the cycle repeats. Enable memory
-delegation on the host (one-time, needs sudo), then **recreate** the QuestDB
-container (a restart is not enough — the limit is set at create time):
-
-```bash
-sudo mkdir -p /etc/systemd/system/user@.service.d
-sudo tee /etc/systemd/system/user@.service.d/delegate.conf >/dev/null <<'EOF'
-[Service]
-Delegate=cpu cpuset io memory pids
-EOF
-sudo systemctl daemon-reload
-```
-
-`daemon-reload` reloads the drop-in file but does not re-apply `Delegate=` to
-the already-running user manager, so the new delegation only takes effect after
-the **user session restarts**. On a headless box a reboot is simplest (and is
-required anyway if you edit `cmdline.txt` below); otherwise restart the user
-manager with `sudo systemctl restart user@$(id -u).service` (this stops all of
-that user's containers, so let them come back before recreating QuestDB).
-
-```bash
-# On older Raspberry Pi kernels the memory controller is off at boot; add to
-# /boot/cmdline.txt (one line) and reboot:
-#   cgroup_enable=memory cgroup_memory=1
-```
-
-After the session restart, recreate the QuestDB container so the cap applies.
-
-**B. `memory` is delegated — the 768 MB cap is enforced but too tight.** As the
-database grows, QuestDB's peak memory (JVM heap plus off-heap memory-mapped
-files, which spike during out-of-order merges) exceeds 768 MB and the cgroup
-OOM-kills it. Raise **QuestDB memory limit** in the plugin config (e.g. `1g` or
-`1.5g`, or empty to remove the cap on a roomier host), which recreates the
-container.
-
-**Confirm the diagnosis** (substitute the container's name if the prefix is not
-`sk-`):
-
-```bash
-podman inspect sk-signalk-questdb \
-  --format 'OOMKilled={{.State.OOMKilled}} RestartCount={{.RestartCount}} cap={{.HostConfig.Memory}}'
-#   cap=805306368  -> 768 MB cap is in place (variant B)
-#   cap=0          -> no cap applied (variant A, or cap intentionally removed)
-podman events --since 2h --stream=false --filter container=sk-signalk-questdb \
-  --filter event=oom --filter event=died
-```
-
-### History queries slow or timing out / "out-of-memory" errors with free RAM
-
-QuestDB memory-maps every partition column file and every pending WAL segment it
-touches. The Linux kernel caps how many memory mappings one process may hold
-(`vm.max_map_count`), and the stock value on Debian, Ubuntu ≤ 22.04, and RHEL
-(65530) is far below what QuestDB recommends (1048576). A fresh database fits
-easily; months of daily partitions across the three tables — plus a segment
-backlog if a table's WAL is suspended — can exhaust the limit. When that happens
-`mmap` fails with out-of-memory errors (errno 12) **even though plenty of RAM is
-free**: queries error out or crawl, and the WAL apply job can suspend a table.
-
-The QuestDB Web Console shows the same warning
-(`vm.max_map_count limit is too low`), the plugin's config panel shows a banner,
-and the server log gets a warning at plugin startup. Fix it in a shell **on the
-host machine itself** — not inside the QuestDB container via `podman exec`: the
-limit is kernel-global, a container cannot change it, and the QuestDB image has
-neither `sudo` nor `/etc/sysctl.d`. Being kernel-global also means this works no
-matter how Signal K itself is deployed:
-
-```bash
-echo 'vm.max_map_count=1048576' | sudo tee /etc/sysctl.d/99-questdb.conf
-sudo sysctl --system
-```
-
-This takes effect immediately — no container or host restart needed. See
-[QuestDB capacity planning](https://questdb.com/docs/getting-started/capacity-planning/#max-virtual-memory-areas-limit)
-for background.
-
-### QuestDB container never starts on macOS (podman machine)
-
-On macOS the QuestDB container is created but may never start. What you see
-depends on the signalk-container version: with signalk-container 1.26 or later,
-the manager names the problem outright (an open-files limit the host refuses)
-and starts the container on the runtime's default limits instead, with this
-plugin's config panel showing a **"request rejected by the host"** banner — the
-remediation below applies unchanged. Older versions never start the container at
-all and show either a misleading **"Permission denied"** (the runtime's error
-text contains "operation not permitted") or only a generic **"Unexpected error.
-See logs for details."** — and the container log view stays empty, because a
-container that never started has no logs.
-
-The cause is the plugin's open-files request. On macOS, Signal K runs on the Mac
-while podman runs inside a Fedora CoreOS VM ("podman machine"). The plugin asks
-for 1048576 open files, and signalk-container normally clamps that request to
-what the host can grant — but from macOS it cannot read the VM's limits, so the
-full request reaches the VM, exceeds its default hard limit (524288), and the
-OCI runtime refuses to start the container. On current Fedora CoreOS the VM's
-`vm.max_map_count` already meets QuestDB's 1048576, so usually only the
-file-descriptor limit needs raising — but an older machine image may sit lower,
-so check it first:
-
-```bash
-podman machine ssh -- sysctl -n vm.max_map_count
-```
-
-If that prints less than 1048576, raise it inside the VM before relying on the
-file-descriptor fix alone (`podman machine ssh`, then
-`sudo tee /etc/sysctl.d/99-signalk-questdb.conf <<< 'vm.max_map_count=1048576'`
-and `sudo sysctl --system`).
-
-Raise the file-descriptor limit inside the VM. From a macOS terminal:
-
-```bash
-podman machine ssh
-```
-
-Inside the VM, add a systemd drop-in for every user manager (the `user@.service`
-template covers each user instance, including the machine's `core` user, whose
-session runs rootless containers) and the same drop-in for `podman.service` to
-cover the rootful connection:
-
-```bash
-sudo mkdir -p /etc/systemd/system/user@.service.d /etc/systemd/system/podman.service.d
-sudo tee /etc/systemd/system/user@.service.d/nofile.conf >/dev/null <<'EOF'
-[Service]
-LimitNOFILE=1048576
-EOF
-sudo cp /etc/systemd/system/user@.service.d/nofile.conf /etc/systemd/system/podman.service.d/nofile.conf
-sudo systemctl daemon-reload
-exit
-```
-
-Back on macOS, restart the VM:
-
-```bash
-podman machine stop && podman machine start
-```
-
-and verify the new limit is grantable:
-
-```bash
-podman run --rm --ulimit nofile=1048576:1048576 docker.io/library/alpine sh -c 'ulimit -n -H'
-```
-
-This should print `1048576`. Finally, remove the half-created container so it is
-recreated with the full limit — the limit is set at create time, so a restart is
-not enough:
-
-```bash
-podman rm -f sk-signalk-questdb
-```
-
-then restart Signal K — on the next plugin start the container is recreated with
-the full limit. (The container manager's **Start** button only starts an
-existing container, so it cannot replace this step.)
-
-The verify and `rm` commands run on podman's default connection — normally the
-rootless one the plugin uses. Rootless and rootful connections keep separate
-container stores, so on a setup with a non-default connection add the same
-`--connection <name>` (list them with `podman system connection list`) to both
-commands, so they hit the store where `sk-signalk-questdb` actually lives
-(`podman ps -a` shows it).
+The [signalk-grafana](https://github.com/dirkwa/signalk-grafana) plugin sets up
+Grafana with QuestDB for you. To connect a Grafana of your own, see
+[Grafana](doc/deployment.md#grafana).
+
+### Update QuestDB
+
+**Check for updates** in the config panel tells you whether a newer QuestDB is
+out, and **Update** installs it. With the default image version `latest`, the
+plugin also checks by itself after each update of the plugin. See
+[QuestDB updates](doc/configuration.md#questdb-updates).
+
+## When something goes wrong
+
+- **"Module signalk-questdb is not available"** right after installing or
+  updating: reload the browser tab with Shift-Reload.
+- **"QuestDB keeps dropping the write connection"**: QuestDB is most likely
+  running out of memory. [Troubleshooting](doc/troubleshooting.md) shows how to
+  tell whether to raise **QuestDB memory limit** or to change a setting of the
+  host.
+- **History is slow, or "out-of-memory" errors although RAM is free**: the host
+  allows QuestDB too few memory mappings. Run this on the host, not inside the
+  container. It takes effect at once:
+
+  ```bash
+  echo 'vm.max_map_count=1048576' | sudo tee /etc/sysctl.d/99-questdb.conf
+  sudo sysctl --system
+  ```
+
+- **WAL suspended** on the status card: QuestDB has stopped applying what it
+  receives. The plugin resumes it by itself where it can. Otherwise the panel
+  offers **Resume**, or **Skip** together with what skipping would lose.
+- **"Cannot apply a change: the QuestDB container cannot be stopped or
+  removed"**, a **QuestDB that never starts on macOS**, and more:
+  [Troubleshooting](doc/troubleshooting.md).
+
+## More documentation
+
+- [Configuration](doc/configuration.md): every setting, and QuestDB updates
+- [Recording](doc/recording.md): what is stored and how, retention and the
+  archive, startup restore
+- [History API](doc/history-api.md): the v1 and v2 APIs and the plugin's REST
+  endpoints
+- [Import and export](doc/import-export.md): InfluxDB, GPX, export and backup
+- [Web Console](doc/web-console.md): exploring the data with SQL
+- [Deployment](doc/deployment.md): requirements, networking, storage,
+  durability, performance, Grafana
+- [Troubleshooting](doc/troubleshooting.md)
 
 ## License