signalk-questdb 2.5.1 → 2.5.2

package/doc/import-export.md

@@ -0,0 +1,179 @@
+# Import and export
+
+## Migrating from InfluxDB
+
+The **InfluxDB Migration** section of the config panel copies history out of an
+existing InfluxDB into QuestDB. It supports InfluxDB 1.x (InfluxQL, what
+signalk-to-influxdb wrote) and 2.x (Flux, what signalk-to-influxdb2 writes).
+
+### Running an import
+
+1. **Detect** finds an InfluxDB on `localhost:8086`, or enter its URL —
+   `localhost` or a loopback/private-network IP literal; anything else is
+   refused.
+2. Supply credentials — an API token and organisation for 2.x, or
+   username/password for 1.x if authentication is enabled — and list the buckets
+   (2.x) or databases (1.x). Credentials are used for the import only, are never
+   written to the plugin's settings, and travel in the request body rather than
+   the query string (Signal K logs full request URLs).
+3. Pick a bucket. The panel then lists the vessels it holds (**Listing the
+   vessels in the bucket…** — on a large bucket this takes a while) and shows
+   **Which vessel in the source is this one**: that vessel's history is imported
+   as `self`. The choice is made for you when the source says:
+   signalk-to-influxdb2 tags its own vessel's points with `self=true`, so that
+   vessel is preselected and marked "(recorded as the own vessel)" — as is the
+   one matching this server's identity, marked "(this server)", or the only one
+   there is. A signalk-to-influxdb 1.x source carries no such tag, so there the
+   choice is yours whenever it holds several vessels and none is this server's
+   identity — the usual case, since the recording server's identity is usually
+   not this one's (a new install has a new one). Tick **Also import the other
+   vessels (AIS targets), each under its own context** to bring those along; the
+   history API serves them as it does live-recorded targets.
+4. Pick a time range and **Start import**. Progress is polled while it runs;
+   **Cancel** stops it, keeping the saved position (see Resuming).
+
+After the import, check the row counts on the status card or in the console
+(`SELECT count() FROM signalk WHERE source = 'influxdb-import'`), make sure this
+plugin is the default history provider (see [Default history provider](configuration.md#default-history-provider)), and disable the
+InfluxDB plugin once nothing else needs it — a consumer such as a track viewer
+reads whichever provider is the default, not both.
+
+### How the data maps
+
+- A measurement with the conventional `value` field becomes the Signal K path of
+  the same name. So do the typed fields signalk-to-influxdb 1.x writes
+  (`stringValue`, `boolValue`, `jsonValue`). A measurement with any other named
+  fields becomes `measurement.field` paths, so two fields cannot overwrite each
+  other.
+- Numbers go to `signalk`, strings and booleans to `signalk_str` (booleans
+  tagged `value_kind=boolean`). A `jsonValue` is decoded first: an object is
+  recorded as its scalar leaves, the same as live data.
+- Positions go to `signalk_position`, whether stored as a `jsonValue`
+  (signalk-to-influxdb 1.x) or as a `lat`/`lon` or `latitude`/`longitude` field
+  pair (signalk-to-influxdb2).
+- Both writers tag every point with the **context** of the vessel it belongs to;
+  the vessel you picked is imported as `self`, the others under their own
+  contexts or left out.
+- Rows keep their **original timestamps**, to the microsecond (InfluxDB stores
+  nanoseconds; QuestDB's `TIMESTAMP` holds microseconds), so imported history
+  sorts and aggregates alongside live data.
+- History is **streamed**: read in batches and written before more is asked for,
+  pausing while QuestDB catches up. Memory use stays flat however dense the
+  source is, so a large import is safe on a small board.
+- Every imported row is tagged `source=influxdb-import`, which makes it
+  distinguishable from live recording — and because the tables deduplicate on
+  `(ts, path, context, source)`, **re-running the same range overwrites rather
+  than duplicating**.
+- Anything that cannot be mapped (a gap, an unsupported value type, a
+  `jsonValue` that is not valid JSON, a latitude with no matching longitude) is
+  counted in the run's `skipped` total rather than being dropped silently.
+
+### Resuming an interrupted import
+
+A large import runs for hours. If it stops part-way — cancelled, failed, or the
+Signal K server restarted — the panel shows what was interrupted and offers
+**Resume import**, which continues from the saved position instead of starting
+again at the first measurement. It needs only the credentials, if InfluxDB asks
+for any: the source and range are remembered, credentials never are. Starting
+the identical import again the ordinary way continues it too. **Discard saved
+position** starts over.
+
+A position is saved only once QuestDB has been asked for the newest row written
+before it and has it — ILP gives no acknowledgment, so the plugin reads the row
+back — and once the position is about a minute old. The first keeps a stalled or
+restarted QuestDB from letting the position past rows it never took, however
+long the stall. The minute covers what a committed row still has to survive: a
+crash of either process, and on a power cut the kernel's default write-back
+timing for a QuestDB running on its default `nosync` — an assumption, not a
+guarantee, which is why an external QuestDB should run with `sync` like the
+managed container does (see [Durability on power loss](deployment.md#durability-on-power-loss)). The position file is
+synced and renamed into place, so a power cut leaves the previous position or
+the new one, never a damaged one; should none survive, the import starts again,
+which is always safe. A position is a window boundary — the import reads each
+measurement a window at a time, a day of source data by default — so a resumed
+import repeats the window it was in plus up to a minute, which the deduplication
+above makes harmless. An import shorter than a window saves nothing, and is
+simply run again.
+
+### Imports made before 2.1.5
+
+Imports from a signalk-to-influxdb 1.x database made before 2.1.5 filed every
+string, boolean and position under a path suffixed with the InfluxDB field name
+— `navigation.state.stringValue`, `steering.autopilot.engaged.boolValue`,
+`navigation.position.jsonValue` — which nothing reads. Running the import again
+writes them under their real paths but cannot remove the old rows: the path is
+part of the deduplication key, and QuestDB has no row delete. When such rows
+exist, the panel says how many and offers **Remove these rows**, which rebuilds
+`signalk_str` without them. Recording continues while it runs; only the swap of
+the rebuilt table for the old one holds the writer, for a few seconds, and rows
+recorded during the rebuild are carried across. Should recording be fast enough
+to fill the writer's buffer during those seconds, the samples it could not keep
+are counted and the panel says how many. Run the import again afterwards.
+
+## Importing a GPX track
+
+A chartplotter's exported track goes into the position archive (see [Retention
+and the archive](recording.md#retention-and-the-archive)), where retention never drops it, under the source
+`import.gpx`. An export can be gigabytes, a four-year TimeZero export is
+1.5 GB, so the file is sent in slices and parsed as it arrives. Memory stays at
+one slice, whatever the file's size.
+
+In the configuration panel, **GPX track import** reads the chosen file in the
+browser first, which takes seconds even for a gigabyte, and lists its tracks by
+name with their points and period. It proposes the own vessel's tracks: the
+largest group whose name is not an AIS target's, and any other name that
+recurs. It flags a group that lay more than a kilometre from a larger one at
+the same time, since that was another vessel. Nothing is uploaded until
+**Import** is pressed; the import then shows its progress and can be aborted.
+
+- **Which tracks.** An export mixes the own track with AIS targets and with
+  passages imported into the plotter from elsewhere, and GPX does not say whose
+  a track is. The import takes the names of the tracks to import (`tracks`);
+  every other track is skipped and listed in the summary. In a TimeZero export
+  the own track is named `Own Ship`, or the plotter's translation of it, such
+  as `Eigenes Schiff`.
+- **Thinning.** Points are thinned to the **Archive resolution**: each is
+  stamped on the start of its interval, and the archive keeps one row per
+  interval. 11.5 million points over four years become 3.2 million rows,
+  about 100 MB, at 10 s.
+- **Overlap.** With `fillGapsOnly` (the default), a point is left out where the
+  vessel already has recorded positions in the same five minutes. Two
+  receivers' fixes interleaved in one series draw a sawtooth.
+- **Doing it again.** Importing the same file again changes nothing, and a
+  stopped import can simply be restarted: the archive's dedup keys absorb the
+  rows already written.
+- **Positions only.** The import takes a position and a time from each point;
+  depth, temperature or speed in a point's extensions are not imported. Depth
+  along the track beyond retention comes from live recording: list
+  `environment.depth.*` under **Archived own-vessel paths**, and retention keeps
+  it from then on.
+- **Strict reading.** Coordinates must be decimals within range and times
+  `xsd:dateTime`; anything else is counted as invalid. A longitude written just
+  past ±180, as TimeZero does at the antimeridian, is wrapped. A file that is
+  not well-formed XML stops the import at the error, keeping the rows before
+  it.
+
+## Export and backup
+
+**Data Export** in the panel, and
+`GET /plugins/signalk-questdb/api/export?from=…&to=…&format=parquet|csv`,
+download the `signalk` numeric table for a date range (both bounds required) as
+Parquet — QuestDB's native export, compressed with the configured codec — or
+CSV.
+
+`GET /plugins/signalk-questdb/api/full-export/:table?from=…&to=…` streams one
+whole table as Parquet, for snapshot and backup tooling that wants the full
+content sliced into dedup-friendly shards (kopia, for instance). Tables:
+`signalk`, `signalk_str`, `signalk_position`, `signalk_archive`, `signalk_position_archive`
+(`/full-export/tables` lists them). The archive tables hold the own vessel's
+history beyond retention and every imported track, so a backup that is meant to
+outlive retention needs them too.
+
+- `from` and `to` are optional but must be set together: omit both for the whole
+  table, or pass both as ISO 8601 timestamps for a half-open `[from, to)` window
+  — no row appears in two adjacent windows.
+- Repeated query parameters (`?from=A&from=B`) and empty strings (`?from=`) are
+  rejected with HTTP 400; silently downgrading to a full-table export would hide
+  bugs in the caller.
+- Format and compression follow the plugin's compression setting (LZ4_RAW or
+  ZSTD), the same as `/export`.

package/doc/recording.md

@@ -0,0 +1,187 @@
+# Recording
+
+## Schema
+
+Three tables, all with WAL mode, daily partitioning, and deduplication:
+
+| Table              | Purpose        | Columns                                                                                                    |
+| ------------------ | -------------- | ---------------------------------------------------------------------------------------------------------- |
+| `signalk`          | Numeric values | `ts`, `path` (SYMBOL), `context` (SYMBOL), `source` (SYMBOL), `value` (DOUBLE)                             |
+| `signalk_str`      | String values  | `ts`, `path` (SYMBOL), `context` (SYMBOL), `source` (SYMBOL), `value_str` (VARCHAR), `value_kind` (SYMBOL) |
+| `signalk_position` | Positions      | `ts`, `context` (SYMBOL), `source` (SYMBOL), `lat` (DOUBLE), `lon` (DOUBLE)                                |
+
+What retention keeps of the own vessel lives in two more tables,
+`signalk_archive` and `signalk_position_archive`, with the same columns and
+monthly partitions (see Retention and the archive).
+
+The own vessel is stored as context `self`, whatever its identity; other vessels
+keep their `vessels.urn:…` context. The history API translates `self` back to
+`vessels.self`.
+
+`ts` is the **server receive time**, not the timestamp a source claims. Marine
+sources carry independent clocks, and storing their timestamps makes commits
+land out of order — QuestDB then rewrites partition tails on every merge
+(observed as >3000x write amplification). Receive time keeps ingestion
+append-only; the millisecond difference is far below the sampling resolution,
+and a device with a broken clock gets more accurate history, not less.
+
+`source` is the delta's sourceRef — which receiver produced the row. Two GPS
+units feeding the same server interleave in storage, and without the column a
+track drawn from history zigzags between them. Rows recorded before the column
+existed have `source` null; they replay unattributed and cannot be filtered.
+
+## What is recorded
+
+Every value on the stream is routed by its type: numbers go to `signalk`,
+strings and booleans to `signalk_str` (booleans tagged `value_kind=boolean`),
+and `navigation.position` — that path only — to `signalk_position`. A vessel's
+name, which arrives as an empty-path `{name}` delta, is stored in `signalk_str`
+under the path `name` with `value_kind=identity`, and replayed in its original
+shape. An object value is recorded as its scalar leaves, one level deep:
+`navigation.attitude {roll, pitch, yaw}` becomes `navigation.attitude.roll` and
+so on, and another lat/lon object such as `navigation.anchor.position` becomes
+`.latitude` and `.longitude` leaves rather than rows in the track table. Nested
+objects — notifications, resource documents — are not descended into; a value no
+table can hold is skipped and reported in the status endpoint and the panel
+rather than dropped silently.
+
+A number that means "no reading" is skipped and reported the same way: `NaN`
+and `±Infinity`, and the NMEA 2000 "no data" marker on AC real, apparent and
+reactive power (PGNs 65007–65029). Those fields cannot encode anything below
+−2 000 000 000 W, but canboatjs decodes "no data" as −2 000 000 001 W, so real
+and reactive power below that floor are dropped; real negative power, such as
+export, is unaffected. Apparent power reaches the stream multiplied by the power
+factor, which can move the marker anywhere below zero — and apparent power is
+never negative, so any negative apparent power is dropped. Connections using the
+canboat WASM decoder do not produce the marker at all.
+
+**Sampling.** Each path is written at most once per **Default sampling rate**
+(2000 ms), with per-path overrides by glob for the few paths that need finer
+resolution and the many that need less. The first value after a window opens
+is written and the rest of the window is dropped, not averaged, so a gust
+between two samples is not in the history. A rate of 0, as default or per
+path, removes the window: `environment.wind.* 0` records every change of the
+wind at wire speed while everything else stays at the default. An unchanged
+value is still written only once per **Unchanged-value heartbeat** (see
+Change-only recording), except `navigation.position`, which is written even
+when unchanged; set the heartbeat to 0 as well to record every update. Wire
+speed costs rows: a wind sensor at 10 Hz writes about 1.7 million a day for
+angle and speed, and every path of a busy NMEA 2000 boat at wire speed can be
+a few hundred rows a second. **Record own vessel** and **Record AIS targets**
+switch the two kinds of context independently, and **Path patterns** exclude
+(or include only) paths by glob.
+
+**Source priorities.** When Signal K's source priorities rank the sources of a
+path, only the preferred source is recorded, the same value instruments show.
+Paths without a priority are recorded from every source. **Record all sources**
+records the sources a priority hides as well, for calibrating a sensor or
+comparing two GPS receivers. Each source is then sampled on its own, so a path
+with three sources writes up to three times the rows; with the option off, the
+sources of a path share one sampling window.
+
+**Change-only recording.** Most paths on a boat do not change from one sample
+to the next: a switch that flips twice a day, an energy counter at rest, a tank
+nobody is drawing from, a notification's state. Sampling alone would write each
+of them at the sampling rate forever, and those repeats make up most of the
+rows. So a value equal to the last one written for its path, vessel and source
+is skipped until the **Unchanged-value heartbeat** is due, and then
+written once more. Every change is written as it arrives, subject to the
+sampling rate.
+
+The heartbeat is what keeps the gaps readable. A path that is still being
+updated has a row at least once per heartbeat, so a longer silence means the
+data really stopped:
+
+- **History API (v2).** A resolution bucket with no row takes the last sample
+  before it, for up to twice the heartbeat (see [History API](history-api.md)). A switch that has
+  been on for an hour reads as on for the whole hour, not as one point and
+  then nothing.
+- **Stale data.** When Signal K marks a path stale it sends `null` for it.
+  Nothing is written for the `null`, so the history shows the last value for at
+  most twice the heartbeat and then a gap. The next reading after that is
+  always written, even if it equals the one before the gap.
+- **Direct SQL and Grafana** see the rows as stored. Add `FILL(PREV)` to a
+  `SAMPLE BY` query to carry values across buckets.
+- **Averages** weight rows, not time. A bucket in which a value changes often
+  and then holds steady averages mostly over the changes; `min`, `max`, `first`
+  and `last` are exact.
+- **Startup restore** replays motion values recorded within **Restore max
+  age**. Keep the heartbeat below that, or a vessel lying still may come back
+  with its position but without its speed, course and heading.
+
+`navigation.position` is written even when unchanged. A heartbeat of 0 does the
+same for every path: each sample the sampling rate lets through is written. With
+a sampling rate of 0 as well, that is every update.
+
+**Batching.** Samples are buffered and committed every **Write batch interval**
+(5 s), one WAL transaction per table per commit. The buffer holds 100 000 lines;
+if QuestDB is unreachable for longer than that covers, the oldest lines are
+dropped and counted. A clean Signal K shutdown flushes first; a hard crash loses
+at most one interval.
+
+**Compression** applies LZ4 or ZSTD to the WAL segments on disk (and to
+Parquet exports).
+
+## Retention and the archive
+
+Retention drops whole daily partitions once they lie entirely before the
+configured number of days; 0 keeps everything. QuestDB cannot delete single
+rows, so keeping part of a day means copying it out first. Before a day of
+`signalk_position` or `signalk` is dropped, the own vessel's rows that the
+settings ask to keep are copied into an archive table, thinned to one row per
+**Archive resolution** interval, and the day is dropped only once QuestDB has
+applied the copy. A copy that fails leaves the day in place for the next run.
+
+| Table                      | Holds                                                           |
+| -------------------------- | --------------------------------------------------------------- |
+| `signalk_position_archive` | Own-vessel positions (**Keep own track**, on by default)        |
+| `signalk_archive`          | Own-vessel numeric paths matching **Archived own-vessel paths** |
+
+Both have the columns and dedup keys of their base table and are partitioned by
+month. Retention never drops from them, other vessels are never archived, and
+strings are not archived. The History API reads the archive together with the
+base tables for any range it covers, so a track or a depth series continues
+past the retention boundary without a seam. An archived row is the first sample
+of its interval and is stamped on the interval's start.
+
+Size, measured: a position row takes 32 bytes, so a vessel recorded around the
+clock fills about 100 MB a year at a 10 s resolution, 34 MB at 30 s.
+
+## Startup restore
+
+Signal K's data model lives in memory only. After a server restart every vessel
+is gone until it transmits again — for AIS that means roughly 30 seconds for a
+Class B target, up to 3 minutes for a Class A at anchor, and about 6 minutes
+before names arrive. The chart fills in gradually instead of showing the traffic
+that was there a moment ago.
+
+With **Restore vessels on startup** enabled, the plugin replays each vessel's
+last recorded position (plus course, speed, heading and identity — name,
+dimensions and AIS ship type) from QuestDB as soon as it connects, so the chart
+is populated immediately, with each target drawn in its ship-type colour rather
+than the default.
+
+**A restored position is where a vessel _was_, not where it is now.** Nothing is
+dead-reckoned forward. The plugin presents these values as the history they are
+rather than passing them off as live: each delta carries its original recorded
+timestamp, so a chart plotter ages it out under the same staleness rules it
+applies to any other target, and the deltas are tagged
+`$source: signalk-questdb.restore`.
+
+Values outside the **Restore max age** window are not replayed at all. The
+9-minute default matches Freeboard's AIS expiry, so a restored target is one
+that would still have been on the chart had the server never stopped. Raising it
+puts progressively staler positions in front of you — treat the window as a
+collision-avoidance setting, not a convenience one.
+
+Only navigation and identity paths are replayed, so stale tank levels and engine
+temperatures are not resurrected as though they were current readings. A vessel
+with no position in the window is skipped. Nothing the Signal K model already
+holds is replaced: a vessel heard while QuestDB was still starting keeps its
+newer position, and after a plugin restart every vessel keeps the one it has.
+Restore honours the recording toggles: with **Record AIS targets** off, no AIS
+target is restored.
+
+The config panel reports the outcome once startup finishes: **Vessels Restored**
+with the count, or **Restore failed** if the replay could not run. Neither
+appears when the option is off.

package/doc/troubleshooting.md

@@ -0,0 +1,243 @@
+# Troubleshooting
+
+## 'Module "signalk-questdb" is not available' right after install or update
+
+Right after installing or updating the plugin, the admin UI can show
+`Module "signalk-questdb" is not available. Make sure the webapp is installed.`
+It is installed — the browser tab is still running the admin UI it loaded before
+the update, which looks for the panel bundle it knew then. A hard refresh of the
+tab (Shift-Reload) fixes it.
+
+## "Cannot apply a change: the QuestDB container cannot be stopped or removed"
+
+The plugin needed to recreate the QuestDB container to apply a change — a new
+image tag, a network or resource setting, a raised file-descriptor limit — but
+the container runtime refused to stop or remove the running container. The
+recreate is **deferred, not abandoned**: QuestDB keeps running and keeps
+recording, on its **previous** configuration. The plugin reports the error
+rather than a "Recording" status, because the change you asked for has not been
+applied and reporting success would be a lie.
+
+The usual cause on rootless Podman is an orphaned user namespace left behind
+when the Podman service restarted underneath a running container. The error text
+carries the specific remedy; the general one is to clear the stale namespace and
+remove the container so the plugin can recreate it:
+
+```bash
+podman system migrate                     # re-attaches orphaned user namespaces
+podman ps -a | grep questdb               # find the managed container's name
+podman rm -f <name-from-above>
+```
+
+Run all three as the **same OS user that owns the container**. Rootless and
+rootful Podman keep separate container stores, so running them as root (or as
+another user) will not find, and will not remove, a rootless container.
+
+Look the name up rather than assuming it: the plugin's container is
+`signalk-questdb` behind a prefix that defaults to `sk-` but is configurable
+(`SIGNALK_CONTAINER_NAMESPACE`), so it is not always `sk-signalk-questdb`.
+
+Then disable and re-enable the plugin, or restart Signal K. The plugin recreates
+the container with the settings that were pending. Recorded data lives under
+Signal K's data directory, not inside the container (see [Data storage](deployment.md#data-storage)), so
+removing the container does not touch it.
+
+**Do not** mask `podman.socket` to work around this. That socket is how the
+plugin reaches Podman at all, so masking it turns a deferred recreate into a
+container the plugin cannot manage in any way. (`podman.service` is the unit the
+socket activates, and masking that has the same effect.) If the socket itself is
+unhealthy, restart it rather than masking it:
+
+```bash
+systemctl --user restart podman.socket
+```
+
+This error is reported with signalk-container 1.29.0 or later. On older versions
+the wedge still happens, but appears only as a repeating drift message in the
+Signal K server log with no plugin error to go with it.
+
+## "QuestDB keeps dropping the write connection — the container may be unhealthy or out of memory"
+
+This status appears when the plugin's ILP writer connects to QuestDB, gets
+dropped within a few seconds, and retries — repeatedly. It is QuestDB itself
+being unhealthy (typically OOM-killed and restart-looping), not a problem with
+the plugin or with a single table. The most common cause on a Raspberry Pi or
+other low-RAM host is that QuestDB is hitting a memory ceiling. There are two
+variants, and the cgroup check below tells them apart.
+
+**1. Is the cgroup `memory` controller delegated?**
+
+```bash
+cat /sys/fs/cgroup/cgroup.controllers
+#   ...memory...   present  -> memory delegation is available, so the cap can be enforced (see B)
+#   memory ABSENT           -> the cap is silently dropped (see A)
+```
+
+When Signal K's containers run under rootless Podman, the kernel only enforces a
+resource limit whose cgroup controller has been delegated to the user session.
+Many distributions delegate `cpu`, `cpuset`, `io`, and `pids` by default but
+**not** `memory`. signalk-container drops a limit whose controller is missing
+rather than failing the container, so a configured cap can silently have no
+effect.
+
+**A. `memory` not delegated — the cap is silently dropped.** QuestDB grows
+without bound and the host kernel's OOM killer eventually kills it under
+whole-system memory pressure; it restarts and the cycle repeats. Enable memory
+delegation on the host (one-time, needs sudo), then **recreate** the QuestDB
+container (a restart is not enough — the limit is set at create time):
+
+```bash
+sudo mkdir -p /etc/systemd/system/user@.service.d
+sudo tee /etc/systemd/system/user@.service.d/delegate.conf >/dev/null <<'EOF'
+[Service]
+Delegate=cpu cpuset io memory pids
+EOF
+sudo systemctl daemon-reload
+```
+
+`daemon-reload` reloads the drop-in file but does not re-apply `Delegate=` to
+the already-running user manager, so the new delegation only takes effect after
+the **user session restarts**. On a headless box a reboot is simplest (and is
+required anyway if you edit `cmdline.txt` below); otherwise restart the user
+manager with `sudo systemctl restart user@$(id -u).service` (this stops all of
+that user's containers, so let them come back before recreating QuestDB).
+
+```bash
+# On older Raspberry Pi kernels the memory controller is off at boot; add to
+# /boot/cmdline.txt (one line) and reboot:
+#   cgroup_enable=memory cgroup_memory=1
+```
+
+After the session restart, recreate the QuestDB container so the cap applies.
+
+**B. `memory` is delegated — the 768 MB cap is enforced but too tight.** As the
+database grows, QuestDB's peak memory (JVM heap plus off-heap memory-mapped
+files, which spike during out-of-order merges) exceeds 768 MB and the cgroup
+OOM-kills it. Raise **QuestDB memory limit** in the plugin config (e.g. `1g` or
+`1.5g`, or empty to remove the cap on a roomier host), which recreates the
+container.
+
+**Confirm the diagnosis** (substitute the container's name if the prefix is not
+`sk-`):
+
+```bash
+podman inspect sk-signalk-questdb \
+  --format 'OOMKilled={{.State.OOMKilled}} RestartCount={{.RestartCount}} cap={{.HostConfig.Memory}}'
+#   cap=805306368  -> 768 MB cap is in place (variant B)
+#   cap=0          -> no cap applied (variant A, or cap intentionally removed)
+podman events --since 2h --stream=false --filter container=sk-signalk-questdb \
+  --filter event=oom --filter event=died
+```
+
+## History queries slow or timing out / "out-of-memory" errors with free RAM
+
+QuestDB memory-maps every partition column file and every pending WAL segment it
+touches. The Linux kernel caps how many memory mappings one process may hold
+(`vm.max_map_count`), and the stock value on Debian, Ubuntu ≤ 22.04, and RHEL
+(65530) is far below what QuestDB recommends (1048576). A fresh database fits
+easily; months of daily partitions across the three tables — plus a segment
+backlog if a table's WAL is suspended — can exhaust the limit. When that happens
+`mmap` fails with out-of-memory errors (errno 12) **even though plenty of RAM is
+free**: queries error out or crawl, and the WAL apply job can suspend a table.
+
+The QuestDB Web Console shows the same warning
+(`vm.max_map_count limit is too low`), the plugin's config panel shows a banner,
+and the server log gets a warning at plugin startup. Fix it in a shell **on the
+host machine itself** — not inside the QuestDB container via `podman exec`: the
+limit is kernel-global, a container cannot change it, and the QuestDB image has
+neither `sudo` nor `/etc/sysctl.d`. Being kernel-global also means this works no
+matter how Signal K itself is deployed:
+
+```bash
+echo 'vm.max_map_count=1048576' | sudo tee /etc/sysctl.d/99-questdb.conf
+sudo sysctl --system
+```
+
+This takes effect immediately — no container or host restart needed. See
+[QuestDB capacity planning](https://questdb.com/docs/getting-started/capacity-planning/#max-virtual-memory-areas-limit)
+for background.
+
+## QuestDB container never starts on macOS (podman machine)
+
+On macOS the QuestDB container is created but may never start. What you see
+depends on the signalk-container version: with signalk-container 1.26 or later,
+the manager names the problem outright (an open-files limit the host refuses)
+and starts the container on the runtime's default limits instead, with this
+plugin's config panel showing a **"request rejected by the host"** banner — the
+remediation below applies unchanged. Older versions never start the container at
+all and show either a misleading **"Permission denied"** (the runtime's error
+text contains "operation not permitted") or only a generic **"Unexpected error.
+See logs for details."** — and the container log view stays empty, because a
+container that never started has no logs.
+
+The cause is the plugin's open-files request. On macOS, Signal K runs on the Mac
+while podman runs inside a Fedora CoreOS VM ("podman machine"). The plugin asks
+for 1048576 open files, and signalk-container normally clamps that request to
+what the host can grant — but from macOS it cannot read the VM's limits, so the
+full request reaches the VM, exceeds its default hard limit (524288), and the
+OCI runtime refuses to start the container. On current Fedora CoreOS the VM's
+`vm.max_map_count` already meets QuestDB's 1048576, so usually only the
+file-descriptor limit needs raising — but an older machine image may sit lower,
+so check it first:
+
+```bash
+podman machine ssh -- sysctl -n vm.max_map_count
+```
+
+If that prints less than 1048576, raise it inside the VM before relying on the
+file-descriptor fix alone (`podman machine ssh`, then
+`sudo tee /etc/sysctl.d/99-signalk-questdb.conf <<< 'vm.max_map_count=1048576'`
+and `sudo sysctl --system`).
+
+Raise the file-descriptor limit inside the VM. From a macOS terminal:
+
+```bash
+podman machine ssh
+```
+
+Inside the VM, add a systemd drop-in for every user manager (the `user@.service`
+template covers each user instance, including the machine's `core` user, whose
+session runs rootless containers) and the same drop-in for `podman.service` to
+cover the rootful connection:
+
+```bash
+sudo mkdir -p /etc/systemd/system/user@.service.d /etc/systemd/system/podman.service.d
+sudo tee /etc/systemd/system/user@.service.d/nofile.conf >/dev/null <<'EOF'
+[Service]
+LimitNOFILE=1048576
+EOF
+sudo cp /etc/systemd/system/user@.service.d/nofile.conf /etc/systemd/system/podman.service.d/nofile.conf
+sudo systemctl daemon-reload
+exit
+```
+
+Back on macOS, restart the VM:
+
+```bash
+podman machine stop && podman machine start
+```
+
+and verify the new limit is grantable:
+
+```bash
+podman run --rm --ulimit nofile=1048576:1048576 docker.io/library/alpine sh -c 'ulimit -n -H'
+```
+
+This should print `1048576`. Finally, remove the half-created container so it is
+recreated with the full limit — the limit is set at create time, so a restart is
+not enough:
+
+```bash
+podman rm -f sk-signalk-questdb
+```
+
+then restart Signal K — on the next plugin start the container is recreated with
+the full limit. (The container manager's **Start** button only starts an
+existing container, so it cannot replace this step.)
+
+The verify and `rm` commands run on podman's default connection — normally the
+rootless one the plugin uses. Rootless and rootful connections keep separate
+container stores, so on a setup with a non-default connection add the same
+`--connection <name>` (list them with `podman system connection list`) to both
+commands, so they hit the store where `sk-signalk-questdb` actually lives
+(`podman ps -a` shows it).

package/doc/web-console.md

@@ -39,6 +39,23 @@ console but QuestDB's REST API and data ingestion, with no authentication, so
 anyone who can reach it can read, **modify or delete** your history. The webapp
 route needs a Signal K admin session; prefer it.
 
+## Access and safety
+
+**It is admin only.** The console is proxied at
+`/plugins/signalk-questdb/console/`, and Signal K gives plugin routes an
+admin-only default — the plugin deliberately does not relax that. A non-admin
+who opens the webapp sees the page frame and an authorization error, never the
+console.
+
+**It can modify data.** Unlike the read-only `/query` endpoint, which rejects
+anything that is not a `SELECT`, the console is a real SQL client: it can drop
+tables and delete recorded history. That is the point of having it, and it is
+what an administrator would use it for — but it is worth knowing before handing
+someone an admin account.
+
+If you would rather not serve it at all, switch the option off and the route is
+not registered.
+
 ## A Quick Tour
 
 - The **left panel** lists the tables. Click a table to see its columns.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "signalk-questdb",
-  "version": "2.5.1",
+  "version": "2.5.2",
   "type": "module",
   "description": "QuestDB history provider for Signal K — replaces signalk-to-influxdb",
   "keywords": [