signalk-questdb 2.3.1 → 2.4.0

package/.release-please-manifest.json

@@ -1,3 +1,3 @@
 {
-  ".": "2.3.1"
+  ".": "2.4.0"
 }

package/CLAUDE.md

@@ -33,9 +33,9 @@ Both build outputs are shipped in the npm package; `prepublishOnly` rebuilds the
 - Subscribes to the Signal K **streambundle** for all paths and writes filtered/throttled deltas to QuestDB via **ILP over raw TCP** (`src/ilp-writer.ts`) — no QuestDB client library.
 - Three tables (`signalk`, `signalk_str`, `signalk_position`) defined and queried in `src/query-client.ts`. All use WAL mode + daily partitioning + dedup. The archives `signalk_archive` and `signalk_position_archive` are defined there too, with their base tables' columns and dedup keys and monthly partitions.
 - Reads come from two registered providers:
-  - **v2** (`src/history-v2.ts`) — `app.registerHistoryApiProvider()`, REST under `/signalk/v2/api/history/`. Aggregates map directly to QuestDB SQL except `sma`/`ema` (computed client-side).
+  - **v2** (`src/history-v2.ts`) — `app.registerHistoryApiProvider()`, REST under `/signalk/v2/api/history/`. Aggregates map directly to QuestDB SQL except `sma`/`ema` (computed client-side). Where the archive covers the range, each table is sampled on its own and the buckets merged in the plugin.
   - **v1** (`src/history-v1.ts`) — `app.registerHistoryProvider()`, WebSocket playback with chunked reads.
-- Retention (`src/retention.ts`) drops old daily partitions on a timer. Before a partition of `signalk_position` or `signalk` goes, the own vessel's rows the config keeps are copied, thinned, into `signalk_position_archive` / `signalk_archive` (`src/archive.ts`), and the drop waits for that copy to be applied. Reads add an archive table only for a range it covers (`ArchiveSpans`), so a recent window stays on the base table alone; v1 snapshots read it as one more per-table `LATEST ON` branch, never by wrapping the union.
+- Retention (`src/retention.ts`) drops old daily partitions on a timer. Before a partition of `signalk_position` or `signalk` goes, the own vessel's rows the config keeps are copied, thinned, into `signalk_position_archive` / `signalk_archive` (`src/archive.ts`), and the drop waits for that copy to be applied. Reads add an archive table only for a range it covers (`ArchiveSpans`), so a recent window stays on the base table alone. v1 snapshots read it as one more per-table `LATEST ON` branch, and v2's sampled and raw reads of the own vessel read each table on its own and merge the buckets or rows in the plugin. Their union (`readFrom` in `src/archive.ts`) sorts every row in range, so only reads that need one order across both tables, such as `middle_index`, or that a short window bounds still use it.
 - GPX import (`src/gpx-import.ts`, `src/gpx-import-session.ts`) streams a sliced upload through saxes into `signalk_position_archive` under source `import.gpx`, thinned by stamping points on interval starts and letting dedup keep one per interval. The caller names the tracks to import: an export mixes the own track with AIS targets and other boats' passages, and nothing in GPX tells them apart.
 - Startup restore (`src/restore.ts`) optionally replays each vessel's last recorded position back into the live model after a server restart.
 - Console proxy (`src/console-proxy.ts`) reverse-proxies QuestDB's own web console so the embeddable webapp (`public/index.html`) can host it in the admin UI.

package/README.md

@@ -132,6 +132,51 @@ rootless Podman.
 | Container network             | `sk-network` | Shared Podman/Docker network QuestDB joins in managed mode, so the companion signalk-grafana reaches it by container DNS; host-port publication is separate ("Bind to 0.0.0.0") |
 | Bind to 0.0.0.0               | `false`      | Publish QuestDB's ports on all interfaces (see Connectivity)                                                                                                                    |
 
+### QuestDB updates
+
+The panel's **update check** compares the running QuestDB with its latest stable
+release. **Update** pulls that release, recreates the container on it and saves
+it as the image version.
+
+After the plugin itself was updated, and at its very first start, the plugin
+also checks once on its own, in the background. It does so only in managed mode,
+and only where the image version is a tag that moves, such as the default
+`latest`. If QuestDB runs an older release than the latest stable one, the
+plugin pulls the tag again and recreates the container on it. The image version
+stays `latest`. Without this check, QuestDB on `latest` would stay on whatever
+release the first install pulled, because signalk-container pulls an image only
+when it is missing. An image version that names one release, such as `9.4.3`,
+stays as it is, and so does one that **Update** saved while the check was
+waiting. QuestDB is down while it starts again, as with **Update**. Progress
+goes to the debug log, and failures go to the server log. A recreate that fails
+half way and leaves QuestDB stopped or without a container is followed by a
+start, as the plugin's own start would do. Only if that fails too does the
+status line say so.
+
+With [signalk-internet](https://github.com/meri-imperiumi/signalk-internet)
+enabled, the check waits for a free link. On a metered, captive or offline link
+it asks again from time to time, for as long as the plugin runs. Without
+signalk-internet, the link counts as free. Telling the two apart needs
+signalk-container 1.38.0 or later; with an older one, the check does not run. A
+failure to read the running or the latest version is also tried again later.
+
+The check is done once QuestDB runs the latest release, or once its container
+runs the image the pull brought, even if QuestDB still reports an older release
+afterwards. It then waits for the next plugin update. The next start checks
+again wherever the container keeps its old image, or where the new one cannot
+be told from the old:
+
+- the plugin stopped first;
+- the download failed, or something else failed before the recreate;
+- an InfluxDB import, a GPX import or an export was running;
+- the string table was being rebuilt;
+- the container was stopped, which the check leaves as it is;
+- the container was wedged;
+- the tag on Docker Hub did not name the latest release yet.
+
+A tag that never reaches the latest release, such as `9` while the latest is
+`10.1.0`, therefore costs a pull and a look at GitHub at every start.
+
 ## Recording
 
 ### Schema
@@ -804,6 +849,53 @@ shown on the status card as **WAL suspended**. Two causes need two remedies:
   shows those numbers and offers **Skip**; if the stuck segment is the newest
   one, skipping drops the entire pending backlog, and the panel says so.
 
+### Out of memory or not answering
+
+QuestDB caps the native memory it tracks below the container's memory limit
+and fails any allocation past the cap with `global RSS memory limit exceeded`.
+When something holds memory at the cap, queries fail, the WAL apply job then
+suspends the tables, and in the end QuestDB stops accepting HTTP connections. The JVM stays up throughout, so the
+container's restart policy never acts.
+
+In managed mode the plugin checks for this every minute while it runs, and
+restarts the QuestDB container when:
+
+- QuestDB recently reported its global memory limit, and over consecutive
+  checks its native memory is still held at that limit, it fails even the
+  health check with it, or it has stopped answering. A single large query
+  frees what it held when it ends, so it does not count, nor does a client that
+  keeps asking for one.
+- the container is running but QuestDB has answered nothing for several
+  minutes, over consecutive checks. A QuestDB busy with a long statement still
+  answers it in the end, and no check counts while a parquet export streams
+  from it.
+
+Before the restart the plugin writes QuestDB's message and its last error lines
+to the server log, shows the reason on the status card, and raises the Signal K
+notification `notifications.signalk-questdb.watchdog`. After the restart it
+logs the outcome. Once QuestDB answers again the notification returns to normal;
+when it answered right after the restart, the notification keeps the time and
+the reason. Restarts are rate limited, spaced apart and capped per few hours of
+the plugin running; past that the plugin logs that the restart is held back and
+keeps the alert up. A restart that fails, because stopping or starting the
+container failed, counts against none of these limits: the plugin tries again a
+few minutes later, since the container may be left stopped. It reports the
+first failure; while the retries keep failing, they go to the debug log only.
+
+The plugin leaves a container alone if someone else stopped it, and starts one
+that its own restart left stopped. One exception: a container stopped by hand
+while a stop of the plugin's is still pending, or right after a restart of the
+plugin's before the plugin has seen it running again, may be started again too.
+While a stop of the plugin's hangs, it sends no further stop. An external
+QuestDB is never restarted. A
+QuestDB that is already stuck when the plugin starts, as after a Signal K
+restart while it was stuck, is not covered: restarting the plugin stops the
+container and starts it afresh.
+
+The managed container runs with QuestDB's query caches off: history reads put
+their time range into the SQL, so the caches never got a hit, and on older
+QuestDB releases they held memory.
+
 ### Performance (Pi and other low-power hosts)
 
 - **Default sampling rate** of 2000 ms limits each path to one write per two

package/dist/archive.d.ts

@@ -18,7 +18,13 @@ export declare const NO_ARCHIVE: ArchiveReader;
  * The merge has to declare `ts` its designated timestamp again, or SAMPLE BY
  * and LATEST ON reject it, and that declaration is only true because of the
  * ORDER BY. A time filter on the outer query still reaches both tables as an
- * interval scan (verified with EXPLAIN on QuestDB 9.4 and 10.0).
+ * interval scan (verified with EXPLAIN on QuestDB 9.4 and 10.0), but the
+ * ORDER BY then sorts every row in range, so its memory grows with the
+ * range. And SAMPLE BY … FROM … TO … FILL(NULL) over it adds a bucket past
+ * TO that repeats the last one (QuestDB 9.4). The v2 provider's sampled and
+ * raw reads therefore read each table on its own and merge the rows
+ * themselves; this union is for reads that need one order across both
+ * tables, or that a short window bounds.
  */
 export declare function readFrom(table: ArchivedTable, withArchive: boolean): string;
 /**

package/dist/archive.js

@@ -30,7 +30,13 @@ export const NO_ARCHIVE = { covers: () => false };
  * The merge has to declare `ts` its designated timestamp again, or SAMPLE BY
  * and LATEST ON reject it, and that declaration is only true because of the
  * ORDER BY. A time filter on the outer query still reaches both tables as an
- * interval scan (verified with EXPLAIN on QuestDB 9.4 and 10.0).
+ * interval scan (verified with EXPLAIN on QuestDB 9.4 and 10.0), but the
+ * ORDER BY then sorts every row in range, so its memory grows with the
+ * range. And SAMPLE BY … FROM … TO … FILL(NULL) over it adds a bucket past
+ * TO that repeats the last one (QuestDB 9.4). The v2 provider's sampled and
+ * raw reads therefore read each table on its own and merge the rows
+ * themselves; this union is for reads that need one order across both
+ * tables, or that a short window bounds.
  */
 export function readFrom(table, withArchive) {
     if (!withArchive)

package/dist/archive.js.map

@@ -1 +1 @@
-{"version":3,"file":"archive.js","sourceRoot":"","sources":["../src/archive.ts"],"names":[],"mappings":"AAAA,8EAA8E;AAC9E,EAAE;AACF,2EAA2E;AAC3E,2EAA2E;AAC3E,6EAA6E;AAC7E,yEAAyE;AACzE,oDAAoD;AACpD,EAAE;AACF,4EAA4E;AAC5E,4EAA4E;AAC5E,+EAA+E;AAC/E,0EAA0E;AAC1E,6CAA6C;AAG7C,OAAO,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAC;AAEpD,MAAM,CAAC,MAAM,cAAc,GAAG;IAC5B,OAAO,EAAE,iBAAiB;IAC1B,gBAAgB,EAAE,0BAA0B;CACpC,CAAC;AAIX,MAAM,OAAO,GAAkC;IAC7C,OAAO,EAAE,kCAAkC;IAC3C,gBAAgB,EAAE,+BAA+B;CAClD,CAAC;AAOF,MAAM,CAAC,MAAM,UAAU,GAAkB,EAAE,MAAM,EAAE,GAAG,EAAE,CAAC,KAAK,EAAE,CAAC;AAEjE;;;;;;;;;;GAUG;AACH,MAAM,UAAU,QAAQ,CAAC,KAAoB,EAAE,WAAoB;IACjE,IAAI,CAAC,WAAW;QAAE,OAAO,KAAK,CAAC;IAC/B,MAAM,OAAO,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC;IAC/B,OAAO,CACL,YAAY,OAAO,SAAS,KAAK,aAAa;QAC9C,UAAU,OAAO,SAAS,cAAc,CAAC,KAAK,CAAC,8BAA8B,CAC9E,CAAC;AACJ,CAAC;AAED;;;;GAIG;AACH,MAAM,OAAO,YAAY;IAIM;IAHZ,KAAK,GAAG,IAAI,GAAG,EAAmC,CAAC;IAC5D,IAAI,GAAG,KAAK,CAAC;IAErB,YAA6B,WAAsC;QAAtC,gBAAW,GAAX,WAAW,CAA2B;IAAG,CAAC;IAEvE,oEAAoE;IACpE,IAAI,KAAK;QACP,OAAO,IAAI,CAAC,IAAI,CAAC;IACnB,CAAC;IAED,KAAK,CAAC,OAAO;QACX,KAAK,MAAM,KAAK,IAAI,MAAM,CAAC,IAAI,CAAC,cAAc,CAAoB,EAAE,CAAC;YACnE,IAAI,CAAC;gBACH,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,WAAW,CAAC,IAAI,CACxC,gCAAgC,cAAc,CAAC,KAAK,CAAC,EAAE,CACxD,CAAC;gBACF,MAAM,CAAC,IAAI,EAAE,EAAE,CAAC,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;gBAC3C,IAAI,OAAO,IAAI,KAAK,QAAQ,IAAI,OAAO,EAAE,KAAK,QAAQ,EAAE,CAAC;oBACvD,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,KAAK,EAAE,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;gBAC5D,CAAC;qBAAM,CAAC;oBACN,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;gBAC3B,CAAC;YACH,CAAC;YAAC,OAAO,GAAG,EAAE,CAAC;gBACb,qEAAqE;gBACrE,gEAAgE;gBAChE,qCAAqC;gBACrC,IAAI,cAAc,CAAC,GAAG,CAAC;oBAAE,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;;oBAC7C,MAAM,GAAG,CAAC;YACjB,CAAC;QACH,CAAC;QACD,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC;IACnB,CAAC;IAED,MAAM,CAAC,KAAoB,EAAE,IAAY,EAAE,EAAU;QACnD,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC;QACnC,IAAI,CAAC,IAAI;YAAE,OAAO,KAAK,CAAC;QACxB,OAAO,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,CAAC,CAAC,IAAI,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,IAAI,IAAI,CAAC,CAAC,CAAC,CAAC;IAClE,CAAC;CACF;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,YAAY,CAChC,WAAsC,EACtC,KAAa,EACb,SAAiB,EACjB,MAAM,GAAG,IAAI;IAEb,MAAM,KAAK,GAAG,KAAK,IAAI,EAAE;QACvB,MAAM,MAAM,GAAG,MAAM,WAAW,CAAC,IAAI,CACnC,6EAA6E,KAAK,GAAG,CACtF,CAAC;QACF,MAAM,CAAC,SAAS,EAAE,SAAS,EAAE,YAAY,CAAC,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;QACrE,IAAI,OAAO,SAAS,KAAK,QAAQ,IAAI,OAAO,YAAY,KAAK,QAAQ,EAAE,CAAC;YACtE,MAAM,IAAI,KAAK,CAAC,GAAG,KAAK,qBAAqB,CAAC,CAAC;QACjD,CAAC;QACD,IAAI,SAAS,KAAK,IAAI,EAAE,CAAC;YACvB,MAAM,IAAI,KAAK,CAAC,GAAG,KAAK,8CAA8C,CAAC,CAAC;QAC1E,CAAC;QACD,OAAO,EAAE,SAAS,EAAE,YAAY,EAAE,CAAC;IACrC,CAAC,CAAC;IACF,MAAM,MAAM,GAAG,CAAC,MAAM,KAAK,EAAE,CAAC,CAAC,YAAY,CAAC;IAC5C,MAAM,QAAQ,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,SAAS,CAAC;IACxC,SAAS,CAAC;QACR,IAAI,CAAC,MAAM,KAAK,EAAE,CAAC,CAAC,SAAS,IAAI,MAAM;YAAE,OAAO;QAChD,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,QAAQ,EAAE,CAAC;YAC1B,MAAM,IAAI,KAAK,CAAC,GAAG,KAAK,4CAA4C,CAAC,CAAC;QACxE,CAAC;QACD,MAAM,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,UAAU,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC,CAAC;IAC9D,CAAC;AACH,CAAC"}
+{"version":3,"file":"archive.js","sourceRoot":"","sources":["../src/archive.ts"],"names":[],"mappings":"AAAA,8EAA8E;AAC9E,EAAE;AACF,2EAA2E;AAC3E,2EAA2E;AAC3E,6EAA6E;AAC7E,yEAAyE;AACzE,oDAAoD;AACpD,EAAE;AACF,4EAA4E;AAC5E,4EAA4E;AAC5E,+EAA+E;AAC/E,0EAA0E;AAC1E,6CAA6C;AAG7C,OAAO,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAC;AAEpD,MAAM,CAAC,MAAM,cAAc,GAAG;IAC5B,OAAO,EAAE,iBAAiB;IAC1B,gBAAgB,EAAE,0BAA0B;CACpC,CAAC;AAIX,MAAM,OAAO,GAAkC;IAC7C,OAAO,EAAE,kCAAkC;IAC3C,gBAAgB,EAAE,+BAA+B;CAClD,CAAC;AAOF,MAAM,CAAC,MAAM,UAAU,GAAkB,EAAE,MAAM,EAAE,GAAG,EAAE,CAAC,KAAK,EAAE,CAAC;AAEjE;;;;;;;;;;;;;;;;GAgBG;AACH,MAAM,UAAU,QAAQ,CAAC,KAAoB,EAAE,WAAoB;IACjE,IAAI,CAAC,WAAW;QAAE,OAAO,KAAK,CAAC;IAC/B,MAAM,OAAO,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC;IAC/B,OAAO,CACL,YAAY,OAAO,SAAS,KAAK,aAAa;QAC9C,UAAU,OAAO,SAAS,cAAc,CAAC,KAAK,CAAC,8BAA8B,CAC9E,CAAC;AACJ,CAAC;AAED;;;;GAIG;AACH,MAAM,OAAO,YAAY;IAIM;IAHZ,KAAK,GAAG,IAAI,GAAG,EAAmC,CAAC;IAC5D,IAAI,GAAG,KAAK,CAAC;IAErB,YAA6B,WAAsC;QAAtC,gBAAW,GAAX,WAAW,CAA2B;IAAG,CAAC;IAEvE,oEAAoE;IACpE,IAAI,KAAK;QACP,OAAO,IAAI,CAAC,IAAI,CAAC;IACnB,CAAC;IAED,KAAK,CAAC,OAAO;QACX,KAAK,MAAM,KAAK,IAAI,MAAM,CAAC,IAAI,CAAC,cAAc,CAAoB,EAAE,CAAC;YACnE,IAAI,CAAC;gBACH,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,WAAW,CAAC,IAAI,CACxC,gCAAgC,cAAc,CAAC,KAAK,CAAC,EAAE,CACxD,CAAC;gBACF,MAAM,CAAC,IAAI,EAAE,EAAE,CAAC,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;gBAC3C,IAAI,OAAO,IAAI,KAAK,QAAQ,IAAI,OAAO,EAAE,KAAK,QAAQ,EAAE,CAAC;oBACvD,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,KAAK,EAAE,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;gBAC5D,CAAC;qBAAM,CAAC;oBACN,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;gBAC3B,CAAC;YACH,CAAC;YAAC,OAAO,GAAG,EAAE,CAAC;gBACb,qEAAqE;gBACrE,gEAAgE;gBAChE,qCAAqC;gBACrC,IAAI,cAAc,CAAC,GAAG,CAAC;oBAAE,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;;oBAC7C,MAAM,GAAG,CAAC;YACjB,CAAC;QACH,CAAC;QACD,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC;IACnB,CAAC;IAED,MAAM,CAAC,KAAoB,EAAE,IAAY,EAAE,EAAU;QACnD,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC;QACnC,IAAI,CAAC,IAAI;YAAE,OAAO,KAAK,CAAC;QACxB,OAAO,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,CAAC,CAAC,IAAI,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,IAAI,IAAI,CAAC,CAAC,CAAC,CAAC;IAClE,CAAC;CACF;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,YAAY,CAChC,WAAsC,EACtC,KAAa,EACb,SAAiB,EACjB,MAAM,GAAG,IAAI;IAEb,MAAM,KAAK,GAAG,KAAK,IAAI,EAAE;QACvB,MAAM,MAAM,GAAG,MAAM,WAAW,CAAC,IAAI,CACnC,6EAA6E,KAAK,GAAG,CACtF,CAAC;QACF,MAAM,CAAC,SAAS,EAAE,SAAS,EAAE,YAAY,CAAC,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;QACrE,IAAI,OAAO,SAAS,KAAK,QAAQ,IAAI,OAAO,YAAY,KAAK,QAAQ,EAAE,CAAC;YACtE,MAAM,IAAI,KAAK,CAAC,GAAG,KAAK,qBAAqB,CAAC,CAAC;QACjD,CAAC;QACD,IAAI,SAAS,KAAK,IAAI,EAAE,CAAC;YACvB,MAAM,IAAI,KAAK,CAAC,GAAG,KAAK,8CAA8C,CAAC,CAAC;QAC1E,CAAC;QACD,OAAO,EAAE,SAAS,EAAE,YAAY,EAAE,CAAC;IACrC,CAAC,CAAC;IACF,MAAM,MAAM,GAAG,CAAC,MAAM,KAAK,EAAE,CAAC,CAAC,YAAY,CAAC;IAC5C,MAAM,QAAQ,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,SAAS,CAAC;IACxC,SAAS,CAAC;QACR,IAAI,CAAC,MAAM,KAAK,EAAE,CAAC,CAAC,SAAS,IAAI,MAAM;YAAE,OAAO;QAChD,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,QAAQ,EAAE,CAAC;YAC1B,MAAM,IAAI,KAAK,CAAC,GAAG,KAAK,4CAA4C,CAAC,CAAC;QACxE,CAAC;QACD,MAAM,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,UAAU,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC,CAAC;IAC9D,CAAC;AACH,CAAC"}

package/dist/container-env.js

@@ -134,6 +134,20 @@ export function buildContainerEnv(config) {
         // apply memory-pressure backoff that precedes a suspension, so raise this
         // back to INFO when diagnosing a database that has stopped recording.
         QDB_LOG_W_STDOUT_LEVEL: "ERROR",
+        // The HTTP endpoint keeps up to 128 compiled queries (32 blocks of 4) for
+        // reuse by identical SQL text, and pg-wire keeps another such cache. The
+        // plugin's history reads put their time range into the SQL as literals,
+        // and so do Grafana's macros, so neither cache gets a hit — they only
+        // fill. On QuestDB before 10.0.0 every cached parallel GROUP BY / SAMPLE
+        // BY also keeps its native buffers at their peak size (fixed upstream in
+        // questdb#7267): about 7 MiB each over a month of data, so a full cache
+        // holds more than the 691 MiB the 768m cap leaves QuestDB, and it then
+        // fails every allocation with "global RSS memory limit exceeded". Seen
+        // twice on 9.4.3 within a day: 20 distinct reads added 150 MiB that only
+        // flush_query_cache() or a restart gave back. Without the caches, each
+        // read frees its buffers when it ends.
+        QDB_HTTP_QUERY_CACHE_ENABLED: "false",
+        QDB_PG_SELECT_CACHE_ENABLED: "false",
         QDB_CAIRO_O3_COLUMN_MEMORY_SIZE: "262144",
         ...(config.compression && config.compression !== "none"
             ? {

package/dist/container-env.js.map

@@ -1 +1 @@
-{"version":3,"file":"container-env.js","sourceRoot":"","sources":["../src/container-env.ts"],"names":[],"mappings":"AAAA,qDAAqD;AACrD,EAAE;AACF,wEAAwE;AACxE,8EAA8E;AAC9E,6EAA6E;AAC7E,0EAA0E;AAC1E,6EAA6E;AAC7E,yBAAyB;AAIzB,kEAAkE;AAClE,6EAA6E;AAC7E,0EAA0E;AAC1E,0EAA0E;AAC1E,yEAAyE;AACzE,0EAA0E;AAC1E,6EAA6E;AAC7E,2EAA2E;AAC3E,0EAA0E;AAC1E,QAAQ;AACR,EAAE;AACF,4EAA4E;AAC5E,8EAA8E;AAC9E,4EAA4E;AAC5E,MAAM,CAAC,MAAM,mBAAmB,GAAG,MAAM,CAAC;AAE1C,0EAA0E;AAC1E,2EAA2E;AAC3E,8EAA8E;AAC9E,qBAAqB;AACrB,MAAM,sBAAsB,GAAG,KAAK,CAAC;AAErC,4EAA4E;AAC5E,gBAAgB;AAChB,EAAE;AACF,4EAA4E;AAC5E,6EAA6E;AAC7E,0EAA0E;AAC1E,0EAA0E;AAC1E,2EAA2E;AAC3E,8EAA8E;AAC9E,uEAAuE;AACvE,EAAE;AACF,6EAA6E;AAC7E,sEAAsE;AACtE,8EAA8E;AAC9E,2EAA2E;AAC3E,uEAAuE;AACvE,uEAAuE;AACvE,wEAAwE;AACxE,MAAM,UAAU,eAAe,CAAC,MAAuB;IACrD,MAAM,KAAK,GAAG,CAAC,6BAA6B,CAAC,CAAC;IAC9C,IAAI,MAAM,CAAC,kBAAkB,EAAE,IAAI,EAAE,EAAE,CAAC;QACtC,KAAK,CAAC,IAAI,CAAC,yBAAyB,CAAC,CAAC;IACxC,CAAC;IACD,OAAO,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AACzB,CAAC;AAED,MAAM,UAAU,iBAAiB,CAC/B,MAAuB;IAEvB,OAAO;QACL,qBAAqB,EAAE,OAAO;QAC9B,gBAAgB,EAAE,MAAM;QACxB,oBAAoB,EAAE,MAAM;QAC5B,qBAAqB,EAAE,mBAAmB;QAC1C,WAAW,EAAE,eAAe,CAAC,MAAM,CAAC;QACpC,yEAAyE;QACzE,sEAAsE;QACtE,yEAAyE;QACzE,yEAAyE;QACzE,wEAAwE;QACxE,iEAAiE;QACjE,uEAAuE;QACvE,wEAAwE;QACxE,oEAAoE;QACpE,gDAAgD;QAChD,gBAAgB,EAAE,GAAG;QAErB,yEAAyE;QACzE,EAAE;QACF,wEAAwE;QACxE,wEAAwE;QACxE,uEAAuE;QACvE,0EAA0E;QAC1E,4CAA4C;QAC5C,EAAE;QACF,oEAAoE;QACpE,gEAAgE;QAChE,uEAAuE;QACvE,2DAA2D;QAC3D,EAAE;QACF,qEAAqE;QACrE,iCAAiC;QACjC,EAAE;QACF,mEAAmE;QACnE,kEAAkE;QAClE,uEAAuE;QACvE,uBAAuB;QACvB,EAAE;QACF,yEAAyE;QACzE,yEAAyE;QACzE,iBAAiB;QACjB,0BAA0B,EAAE,GAAG;QAC/B,uBAAuB,EAAE,GAAG;QAC5B,gCAAgC,EAAE,GAAG;QACrC,wEAAwE;QACxE,yEAAyE;QACzE,yEAAyE;QACzE,yEAAyE;QACzE,wCAAwC;QACxC,4BAA4B,EAAE,GAAG;QACjC,uEAAuE;QACvE,wEAAwE;QACxE,8BAA8B,EAAE,GAAG;QACnC,iCAAiC,EAAE,GAAG;QACtC,kCAAkC,EAAE,GAAG;QACvC,uBAAuB,EAAE,GAAG;QAE5B,0EAA0E;QAC1E,kEAAkE;QAClE,iCAAiC,EAAE,sBAAsB;QACzD,uEAAuE;QACvE,yEAAyE;QACzE,uEAAuE;QACvE,yEAAyE;QACzE,uEAAuE;QACvE,8CAA8C;QAC9C,+BAA+B,EAAE,sBAAsB;QACvD,6BAA6B,EAAE,sBAAsB;QACrD,oCAAoC,EAAE,sBAAsB;QAC5D,sCAAsC,EAAE,sBAAsB;QAC9D,0CAA0C,EAAE,sBAAsB;QAClE,wCAAwC,EAAE,sBAAsB;QAChE,2CAA2C,EAAE,sBAAsB;QACnE,4CAA4C,EAAE,sBAAsB;QACpE,iCAAiC,EAAE,sBAAsB;QAEzD,wEAAwE;QACxE,0EAA0E;QAC1E,mEAAmE;QACnE,mEAAmE;QACnE,uEAAuE;QACvE,0EAA0E;QAC1E,sEAAsE;QACtE,sBAAsB,EAAE,OAAO;QAE/B,+BAA+B,EAAE,QAAQ;QACzC,GAAG,CAAC,MAAM,CAAC,WAAW,IAAI,MAAM,CAAC,WAAW,KAAK,MAAM;YACrD,CAAC,CAAC;gBACE,uCAAuC,EACrC,MAAM,CAAC,WAAW,KAAK,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,KAAK;gBAChD,GAAG,CAAC,MAAM,CAAC,WAAW,KAAK,MAAM,IAAI,MAAM,CAAC,gBAAgB;oBAC1D,CAAC,CAAC;wBACE,uCAAuC,EAAE,MAAM,CAC7C,MAAM,CAAC,gBAAgB,CACxB;qBACF;oBACH,CAAC,CAAC,EAAE,CAAC;aACR;YACH,CAAC,CAAC,EAAE,CAAC;KACR,CAAC;AACJ,CAAC"}
+{"version":3,"file":"container-env.js","sourceRoot":"","sources":["../src/container-env.ts"],"names":[],"mappings":"AAAA,qDAAqD;AACrD,EAAE;AACF,wEAAwE;AACxE,8EAA8E;AAC9E,6EAA6E;AAC7E,0EAA0E;AAC1E,6EAA6E;AAC7E,yBAAyB;AAIzB,kEAAkE;AAClE,6EAA6E;AAC7E,0EAA0E;AAC1E,0EAA0E;AAC1E,yEAAyE;AACzE,0EAA0E;AAC1E,6EAA6E;AAC7E,2EAA2E;AAC3E,0EAA0E;AAC1E,QAAQ;AACR,EAAE;AACF,4EAA4E;AAC5E,8EAA8E;AAC9E,4EAA4E;AAC5E,MAAM,CAAC,MAAM,mBAAmB,GAAG,MAAM,CAAC;AAE1C,0EAA0E;AAC1E,2EAA2E;AAC3E,8EAA8E;AAC9E,qBAAqB;AACrB,MAAM,sBAAsB,GAAG,KAAK,CAAC;AAErC,4EAA4E;AAC5E,gBAAgB;AAChB,EAAE;AACF,4EAA4E;AAC5E,6EAA6E;AAC7E,0EAA0E;AAC1E,0EAA0E;AAC1E,2EAA2E;AAC3E,8EAA8E;AAC9E,uEAAuE;AACvE,EAAE;AACF,6EAA6E;AAC7E,sEAAsE;AACtE,8EAA8E;AAC9E,2EAA2E;AAC3E,uEAAuE;AACvE,uEAAuE;AACvE,wEAAwE;AACxE,MAAM,UAAU,eAAe,CAAC,MAAuB;IACrD,MAAM,KAAK,GAAG,CAAC,6BAA6B,CAAC,CAAC;IAC9C,IAAI,MAAM,CAAC,kBAAkB,EAAE,IAAI,EAAE,EAAE,CAAC;QACtC,KAAK,CAAC,IAAI,CAAC,yBAAyB,CAAC,CAAC;IACxC,CAAC;IACD,OAAO,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AACzB,CAAC;AAED,MAAM,UAAU,iBAAiB,CAC/B,MAAuB;IAEvB,OAAO;QACL,qBAAqB,EAAE,OAAO;QAC9B,gBAAgB,EAAE,MAAM;QACxB,oBAAoB,EAAE,MAAM;QAC5B,qBAAqB,EAAE,mBAAmB;QAC1C,WAAW,EAAE,eAAe,CAAC,MAAM,CAAC;QACpC,yEAAyE;QACzE,sEAAsE;QACtE,yEAAyE;QACzE,yEAAyE;QACzE,wEAAwE;QACxE,iEAAiE;QACjE,uEAAuE;QACvE,wEAAwE;QACxE,oEAAoE;QACpE,gDAAgD;QAChD,gBAAgB,EAAE,GAAG;QAErB,yEAAyE;QACzE,EAAE;QACF,wEAAwE;QACxE,wEAAwE;QACxE,uEAAuE;QACvE,0EAA0E;QAC1E,4CAA4C;QAC5C,EAAE;QACF,oEAAoE;QACpE,gEAAgE;QAChE,uEAAuE;QACvE,2DAA2D;QAC3D,EAAE;QACF,qEAAqE;QACrE,iCAAiC;QACjC,EAAE;QACF,mEAAmE;QACnE,kEAAkE;QAClE,uEAAuE;QACvE,uBAAuB;QACvB,EAAE;QACF,yEAAyE;QACzE,yEAAyE;QACzE,iBAAiB;QACjB,0BAA0B,EAAE,GAAG;QAC/B,uBAAuB,EAAE,GAAG;QAC5B,gCAAgC,EAAE,GAAG;QACrC,wEAAwE;QACxE,yEAAyE;QACzE,yEAAyE;QACzE,yEAAyE;QACzE,wCAAwC;QACxC,4BAA4B,EAAE,GAAG;QACjC,uEAAuE;QACvE,wEAAwE;QACxE,8BAA8B,EAAE,GAAG;QACnC,iCAAiC,EAAE,GAAG;QACtC,kCAAkC,EAAE,GAAG;QACvC,uBAAuB,EAAE,GAAG;QAE5B,0EAA0E;QAC1E,kEAAkE;QAClE,iCAAiC,EAAE,sBAAsB;QACzD,uEAAuE;QACvE,yEAAyE;QACzE,uEAAuE;QACvE,yEAAyE;QACzE,uEAAuE;QACvE,8CAA8C;QAC9C,+BAA+B,EAAE,sBAAsB;QACvD,6BAA6B,EAAE,sBAAsB;QACrD,oCAAoC,EAAE,sBAAsB;QAC5D,sCAAsC,EAAE,sBAAsB;QAC9D,0CAA0C,EAAE,sBAAsB;QAClE,wCAAwC,EAAE,sBAAsB;QAChE,2CAA2C,EAAE,sBAAsB;QACnE,4CAA4C,EAAE,sBAAsB;QACpE,iCAAiC,EAAE,sBAAsB;QAEzD,wEAAwE;QACxE,0EAA0E;QAC1E,mEAAmE;QACnE,mEAAmE;QACnE,uEAAuE;QACvE,0EAA0E;QAC1E,sEAAsE;QACtE,sBAAsB,EAAE,OAAO;QAE/B,0EAA0E;QAC1E,yEAAyE;QACzE,wEAAwE;QACxE,sEAAsE;QACtE,yEAAyE;QACzE,yEAAyE;QACzE,wEAAwE;QACxE,uEAAuE;QACvE,uEAAuE;QACvE,yEAAyE;QACzE,uEAAuE;QACvE,uCAAuC;QACvC,4BAA4B,EAAE,OAAO;QACrC,2BAA2B,EAAE,OAAO;QAEpC,+BAA+B,EAAE,QAAQ;QACzC,GAAG,CAAC,MAAM,CAAC,WAAW,IAAI,MAAM,CAAC,WAAW,KAAK,MAAM;YACrD,CAAC,CAAC;gBACE,uCAAuC,EACrC,MAAM,CAAC,WAAW,KAAK,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,KAAK;gBAChD,GAAG,CAAC,MAAM,CAAC,WAAW,KAAK,MAAM,IAAI,MAAM,CAAC,gBAAgB;oBAC1D,CAAC,CAAC;wBACE,uCAAuC,EAAE,MAAM,CAC7C,MAAM,CAAC,gBAAgB,CACxB;qBACF;oBACH,CAAC,CAAC,EAAE,CAAC;aACR;YACH,CAAC,CAAC,EAAE,CAAC;KACR,CAAC;AACJ,CAAC"}

package/dist/drained.d.ts

@@ -0,0 +1,2 @@
+import type { ServerResponse } from "node:http";
+export declare function drained(res: ServerResponse, signal?: AbortSignal): Promise<boolean>;

package/dist/drained.js

@@ -0,0 +1,22 @@
+// After a write the response refused, waits until it takes more; false once
+// the client is gone, or once `signal` (the export's deadline) aborts. A
+// response whose client disconnected never drains, nor does one whose client
+// stays connected but stops reading, and a stream loop waiting for it would
+// hold its QuestDB export open for good, and with it the watchdog's export
+// count (questdbExportsRunning in index.ts).
+export function drained(res, signal) {
+    if (res.destroyed || signal?.aborted)
+        return Promise.resolve(false);
+    return new Promise((resolve) => {
+        const settle = () => {
+            res.off("drain", settle);
+            res.off("close", settle);
+            signal?.removeEventListener("abort", settle);
+            resolve(!res.destroyed && !signal?.aborted);
+        };
+        res.once("drain", settle);
+        res.once("close", settle);
+        signal?.addEventListener("abort", settle, { once: true });
+    });
+}
+//# sourceMappingURL=drained.js.map

package/dist/drained.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"drained.js","sourceRoot":"","sources":["../src/drained.ts"],"names":[],"mappings":"AAEA,4EAA4E;AAC5E,yEAAyE;AACzE,6EAA6E;AAC7E,4EAA4E;AAC5E,2EAA2E;AAC3E,6CAA6C;AAC7C,MAAM,UAAU,OAAO,CACrB,GAAmB,EACnB,MAAoB;IAEpB,IAAI,GAAG,CAAC,SAAS,IAAI,MAAM,EAAE,OAAO;QAAE,OAAO,OAAO,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC;IACpE,OAAO,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,EAAE;QAC7B,MAAM,MAAM,GAAG,GAAG,EAAE;YAClB,GAAG,CAAC,GAAG,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;YACzB,GAAG,CAAC,GAAG,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;YACzB,MAAM,EAAE,mBAAmB,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;YAC7C,OAAO,CAAC,CAAC,GAAG,CAAC,SAAS,IAAI,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;QAC9C,CAAC,CAAC;QACF,GAAG,CAAC,IAAI,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;QAC1B,GAAG,CAAC,IAAI,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;QAC1B,MAAM,EAAE,gBAAgB,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC;IAC5D,CAAC,CAAC,CAAC;AACL,CAAC"}

package/dist/history-v2.js

@@ -295,6 +295,8 @@ function normalizeContext(context, selfContext) {
 // well-behaved clients budget ~2000 points per request; a million covers any
 // sane chart while keeping the worst case bounded.
 const MAX_SAMPLE_BUCKETS = 1_000_000;
+// The rows a read without a resolution returns at most.
+const RAW_LIMIT = 10_000;
 // Upper bound on columns ONE path may expand into under sourcePolicy=all.
 // The bucket cap above only applies to requests that named a resolution, so
 // these are the only things bounding an unresolved expansion.
@@ -328,6 +330,128 @@ function sampleBy(range, resolution) {
     const to = validateTimestamp(range.to);
     return `SAMPLE BY ${effectiveResolution(resolution)}s FROM '${from}' TO '${to}' FILL(NULL)`;
 }
+const numbers = (values) => values.filter((v) => typeof v === "number");
+const total = (values) => numbers(values).reduce((a, b) => a + b, 0);
+/** The least or greatest number, or null where there is none, as in SQL. */
+function extreme(values, pick) {
+    const found = numbers(values);
+    return found.length === 0 ? null : Math[pick](...found);
+}
+/**
+ * The row whose time (part `at`) is earliest, or latest. QuestDB writes
+ * every timestamp in one fixed-width ISO form, so the strings compare in
+ * time order.
+ */
+function byTime(rows, at, latest) {
+    return rows.reduce((best, row) => {
+        const [a, b] = [String(row[at]), String(best[at])];
+        return (latest ? a > b : a < b) ? row : best;
+    });
+}
+/** A numeric bucket's `method` value, as aggregateToSql() computes it. */
+function valueColumns(method) {
+    const sql = `${aggregateToSql(method)} as agg_value`;
+    switch (method) {
+        case "min":
+        case "max":
+            return {
+                sql,
+                parts: [`${method}(value)`],
+                width: 1,
+                merge: (rows) => [
+                    extreme(rows.map((r) => r[0]), method),
+                ],
+            };
+        case "first":
+        case "last":
+            return {
+                sql,
+                parts: [`${method}(value)`, method === "first" ? "min(ts)" : "max(ts)"],
+                width: 1,
+                merge: (rows) => [byTime(rows, 1, method === "last")[0]],
+            };
+        case "mid":
+            return {
+                sql,
+                parts: ["min(value)", "max(value)"],
+                width: 1,
+                merge: (rows) => {
+                    const low = extreme(rows.map((r) => r[0]), "min");
+                    const high = extreme(rows.map((r) => r[1]), "max");
+                    return [low === null || high === null ? null : (low + high) / 2];
+                },
+            };
+        default: {
+            // A mean of the two means would weigh a bucket's few archived rows
+            // as much as its many recorded ones.
+            return {
+                sql,
+                parts: ["sum(value)", "count(value)"],
+                width: 1,
+                merge: (rows) => {
+                    const count = total(rows.map((r) => r[1]));
+                    return [count === 0 ? null : total(rows.map((r) => r[0])) / count];
+                },
+            };
+        }
+    }
+}
+/** ANGULAR_BUCKET_SQL's mean angle and floor, summed across the tables. */
+const ANGULAR_COLUMNS = {
+    sql: ANGULAR_BUCKET_SQL,
+    parts: ["sum(sin(value))", "sum(cos(value))", "count(value)", "min(value)"],
+    width: 2,
+    merge: (rows) => {
+        const column = (i) => rows.map((r) => r[i]);
+        const count = total(column(2));
+        if (count === 0)
+            return [null, null];
+        const [sin, cos] = [total(column(0)) / count, total(column(1)) / count];
+        return [
+            Math.hypot(sin, cos) < NO_DIRECTION ? null : Math.atan2(sin, cos),
+            extreme(column(3), "min"),
+        ];
+    },
+};
+/** HELD_SQL's last sample and its time, from the table that took it last. */
+const HELD_COLUMNS = {
+    sql: HELD_SQL,
+    parts: ["last(value)", "last(ts)"],
+    width: 2,
+    merge: (rows) => byTime(rows, 1, true),
+};
+/** A position bucket's `pick` fix, as one lat/lon pair from one table. */
+function positionColumns(pick) {
+    return {
+        sql: `${pick}(lat) as lat, ${pick}(lon) as lon`,
+        parts: [
+            `${pick}(lat)`,
+            `${pick}(lon)`,
+            pick === "first" ? "min(ts)" : "max(ts)",
+        ],
+        width: 2,
+        merge: (rows) => byTime(rows, 2, pick === "last").slice(0, 2),
+    };
+}
+/**
+ * The rows of two reads ordered by time (column 0), paired where both have
+ * the same time and in time order throughout. Yielded rather than collected,
+ * so a merge holds the two reads and its result, not a third copy.
+ */
+function* pairByTime(own, kept) {
+    let i = 0;
+    let j = 0;
+    while (i < own.length || j < kept.length) {
+        const a = i < own.length ? String(own[i][0]) : null;
+        const b = j < kept.length ? String(kept[j][0]) : null;
+        if (a !== null && (b === null || a < b))
+            yield [own[i++], null];
+        else if (a === null || b < a)
+            yield [null, kept[j++]];
+        else
+            yield [own[i++], kept[j++]];
+    }
+}
 export function createHistoryProviderV2(queryClient, selfContext, 
 // Opt-in, default off. Expansion multiplies one requested path into one
 // query and one column PER recording source, so a request that was cheap
@@ -357,11 +481,71 @@ archive = NO_ARCHIVE) {
         const since = new Date(Date.parse(from) - holdMs).toISOString();
         return `ts >= '${since}' AND ts < '${from}' AND context = '${safeContext}'`;
     }
-    /** What the numeric seed is read from: the window `seedRangeWhere` names. */
-    function seedSource(range) {
+    /**
+     * `base` sampled over the range in `groups`' columns, one row a bucket.
+     *
+     * With the archive, each table is sampled on its own and the two grids
+     * merged here. QuestDB samples their union only after sorting every row in
+     * range by time: the tracks plugin's history import asks once for
+     * everything since 1970, and that sort of 5.5 M own positions went past
+     * QuestDB's memory limit. Each table alone is sampled in the order it is
+     * stored, and the merge holds the two grids and the merged one.
+     */
+    async function readSampled(base, withArchive, where, range, resolution, groups) {
+        const grid = sampleBy(range, resolution);
+        if (!withArchive) {
+            const columns = groups.map((group) => group.sql).join(", ");
+            const result = await queryClient.exec(`SELECT ts, ${columns} FROM ${base} WHERE ${where} ${grid} ORDER BY ts`);
+            return result.dataset;
+        }
+        // Aliased, since two groups may select the same aggregate, and led by
+        // the bucket's row count, which FILL(NULL) leaves null where it is 0.
+        const parts = ["count()", ...groups.flatMap((group) => group.parts)]
+            .map((part, i) => `${part} p${i}`)
+            .join(", ");
+        const read = async (table) => (await queryClient.exec(`SELECT ts, ${parts} FROM ${table} WHERE ${where} ${grid} ORDER BY ts`)).dataset;
+        const own = await read(base);
+        const kept = await read(ARCHIVE_TABLES[base]);
+        return Array.from(pairByTime(own, kept), (pair) => {
+            const filled = pair.filter((row) => row !== null && Number(row[1]) > 0);
+            const row = [(pair[0] ?? pair[1])[0]];
+            let at = 2;
+            for (const group of groups) {
+                row.push(...(filled.length === 0
+                    ? Array(group.width).fill(null)
+                    : group.merge(filled.map((r) => r.slice(at, at + group.parts.length)))));
+                at += group.parts.length;
+            }
+            return row;
+        });
+    }
+    /**
+     * The first rows of `base` in the range, in time order, under the LIMIT
+     * every raw read has: with the archive, from each table on its own and
+     * merged here, for the reason readSampled() gives.
+     */
+    async function readRaw(base, withArchive, columns, where) {
+        const read = async (table) => (await queryClient.exec(`SELECT ts, ${columns} FROM ${table} WHERE ${where} ORDER BY ts LIMIT ${RAW_LIMIT}`)).dataset;
+        const own = await read(base);
+        if (!withArchive)
+            return own;
+        const kept = await read(ARCHIVE_TABLES[base]);
+        const rows = [];
+        for (const pair of pairByTime(own, kept)) {
+            rows.push(...pair.filter((row) => row !== null));
+            if (rows.length >= RAW_LIMIT)
+                break;
+        }
+        return rows.slice(0, RAW_LIMIT);
+    }
+    /**
+     * What the numeric seed is read from: the window `seedRangeWhere` names,
+     * with the archive only for the own vessel, the only one it holds.
+     */
+    function seedSource(range, own) {
         const from = validateTimestamp(range.from);
         const since = new Date(Date.parse(from) - holdMs).toISOString();
-        return readFrom("signalk", archive.covers("signalk", since, from));
+        return readFrom("signalk", own && archive.covers("signalk", since, from));
     }
     /** The numeric table's last `[value, ts]` before the range, or null. */
     async function numericSeed(seedWhere, source) {
@@ -413,7 +597,7 @@ archive = NO_ARCHIVE) {
      * leaves the path as a single unexpanded column rather than failing the
      * whole request.
      */
-    async function distinctSources(path, range, safeContext) {
+    async function distinctSources(path, range, safeContext, withArchive) {
         const safePath = validateIdentifier(path);
         const rangeWhere = buildRangeWhere(range, safeContext);
         const position = path === "navigation.position";
@@ -422,7 +606,7 @@ archive = NO_ARCHIVE) {
         const tables = [
             `SELECT DISTINCT source FROM ${table} WHERE ${rangeWhere}${pathWhere}`,
         ];
-        if (archive.covers(table, range.from, range.to)) {
+        if (withArchive) {
             tables.push(`SELECT DISTINCT source FROM ${ARCHIVE_TABLES[table]} WHERE ${rangeWhere}${pathWhere}`);
         }
         if (!position) {
@@ -531,7 +715,7 @@ archive = NO_ARCHIVE) {
             // what an empty bucket after it holds (see holdThroughEmptyBuckets).
             return sampled
                 ? `SELECT ts, ${pick}(value_str) as value_str, ${kindOf(pick)} as value_kind, last(value_str) as held_str, ${kindOf("last")} as held_kind, last(ts) as held_ts FROM signalk_str WHERE ${clause} ${sampleBy(range, resolution)} ORDER BY ts`
-                : `SELECT ts, value_str, ${kind} as value_kind FROM signalk_str WHERE ${clause} ORDER BY ts LIMIT 10000`;
+                : `SELECT ts, value_str, ${kind} as value_kind FROM signalk_str WHERE ${clause} ORDER BY ts LIMIT ${RAW_LIMIT}`;
         };
         // A read racing ensureTables()'s migration — or an external QuestDB the
         // plugin does not own — must degrade, not fail the request. Two columns
@@ -607,6 +791,16 @@ archive = NO_ARCHIVE) {
         const requestedContext = query.context ?? "vessels.self";
         const storedContext = normalizeContext(requestedContext, selfContext);
         const safeContext = validateIdentifier(storedContext);
+        // Whether each table's reads include its archive. The archive holds only
+        // the own vessel (archive.ts), so another vessel's reads leave it out.
+        // Decided once, so the bucket budget counts the reads that then run even
+        // when a refresh of the archive's spans lands in between.
+        const archived = {
+            signalk: storedContext === "self" &&
+                archive.covers("signalk", range.from, range.to),
+            signalk_position: storedContext === "self" &&
+                archive.covers("signalk_position", range.from, range.to),
+        };
         // Expand each spec into the COLUMNS it will produce. Under
         // sourcePolicy=all a path with no explicit sourceRef becomes one column
         // per distinct source that actually recorded it in range; everything else
@@ -631,7 +825,7 @@ archive = NO_ARCHIVE) {
                     `${MAX_EXPANDED_COLUMNS} columns — request fewer paths, or name ` +
                     `the sources explicitly with paths=<path>|<sourceRef>`);
             }
-            const sources = await distinctSources(spec.path, range, safeContext);
+            const sources = await distinctSources(spec.path, range, safeContext, archived[spec.path === "navigation.position" ? "signalk_position" : "signalk"]);
             if (sources.length === 0) {
                 // Nothing recorded in range, or a database with no `source` column:
                 // keep the unexpanded column so the caller still gets the series
@@ -691,10 +885,15 @@ archive = NO_ARCHIVE) {
         // navigation.position is served by its own table and never falls back;
         // a moving average has no text to fall back to.
         const fallbackCapableSpecs = columns.filter((spec) => spec.path !== "navigation.position" && !isMovingAverage(spec.aggregate)).length;
+        // A column the archive covers costs one more: readSampled() samples the
+        // base table and the archive each on the whole grid. middle_index reads
+        // their union in one query.
+        const archivedSpecs = columns.filter((spec) => spec.aggregate !== "middle_index" &&
+            archived[spec.path === "navigation.position" ? "signalk_position" : "signalk"]).length;
         if (sampledSpecs > 0 && query.resolution && query.resolution > 0) {
             const rangeSec = (Date.parse(range.to) - Date.parse(range.from)) / 1000;
             const bucketsPerSeries = Math.ceil(rangeSec / effectiveResolution(query.resolution));
-            const worstCaseQueries = sampledSpecs + fallbackCapableSpecs;
+            const worstCaseQueries = sampledSpecs + fallbackCapableSpecs + archivedSpecs;
             const buckets = bucketsPerSeries * worstCaseQueries;
             if (buckets > MAX_SAMPLE_BUCKETS) {
                 throw new Error(`resolution ${query.resolution}s over this range produces up to ` +
@@ -723,7 +922,11 @@ archive = NO_ARCHIVE) {
             const sourceWhere = buildSourceWhere(spec);
             const isPosition = spec.path === "navigation.position";
             const base = isPosition ? "signalk_position" : "signalk";
-            const table = readFrom(base, archive.covers(base, range.from, range.to));
+            // A middle row needs both tables' rows in one order, so middle_index
+            // reads their union, as the numeric seed does over its short window.
+            // Over a long range the archive covers that still sorts every row in
+            // it. Every other read goes through readSampled() or readRaw().
+            const union = () => readFrom(base, archived[base]);
             if (isPosition) {
                 const where = buildRangeWhere(range, safeContext) + sourceWhere;
                 // Position is a co-recorded lat/lon pair. first, last and
@@ -736,21 +939,20 @@ archive = NO_ARCHIVE) {
                     : "first";
                 if (posAgg !== spec.aggregate)
                     valuesList[specIndex].method = "first";
-                let sql;
+                let dataset;
                 if (posAgg === "middle_index") {
-                    sql = middleRowSql(table, "lat, lon", where, range, query.resolution);
+                    dataset = (await queryClient.exec(middleRowSql(union(), "lat, lon", where, range, query.resolution))).dataset;
                 }
                 else if (query.resolution && query.resolution > 0) {
-                    sql = `SELECT ts, ${posAgg}(lat) as lat, ${posAgg}(lon) as lon FROM ${table} WHERE ${where} ${sampleBy(range, query.resolution)} ORDER BY ts`;
+                    dataset = await readSampled(base, archived[base], where, range, query.resolution, [positionColumns(posAgg)]);
                 }
                 else {
-                    sql = `SELECT ts, lat, lon FROM ${table} WHERE ${where} ORDER BY ts LIMIT 10000`;
+                    dataset = await readRaw(base, archived[base], "lat, lon", where);
                 }
-                const result = await queryClient.exec(sql);
                 // A position goes out as a `[longitude, latitude]` pair — GeoJSON
                 // order, as the History API's OpenAPI schema defines and the other
                 // providers emit — not as the data model's `{latitude, longitude}`.
-                const rows = result.dataset.map((row) => [
+                const rows = dataset.map((row) => [
                     row[0],
                     row[1] !== null && row[2] !== null ? [row[2], row[1]] : null,
                 ]);
@@ -759,7 +961,7 @@ archive = NO_ARCHIVE) {
             }
             const where = `${buildRangeWhere(range, safeContext)} AND path = '${safePath}'${sourceWhere}`;
             const seedWhere = `${seedRangeWhere(range, safeContext)} AND path = '${safePath}'${sourceWhere}`;
-            const seedFrom = seedSource(range);
+            const seedFrom = seedSource(range, storedContext === "self");
             if (isMovingAverage(spec.aggregate)) {
                 // The window runs over the series at the requested resolution — one
                 // averaged bucket per sample, the grid every other column is on — so
@@ -771,17 +973,13 @@ archive = NO_ARCHIVE) {
                 // gap ages values out instead of stretching the window across it.
                 const window = movingAverageWindow(spec);
                 const angular = isAngular(unitsOf(spec.path, requestedContext));
-                const bucket = angular ? ANGULAR_BUCKET_SQL : "avg(value) as agg_value";
+                const bucket = angular ? ANGULAR_COLUMNS : valueColumns("average");
                 const sampled = !!query.resolution && query.resolution > 0;
-                const sql = sampled
-                    ? `SELECT ts, ${bucket}, ${HELD_SQL} FROM ${table} WHERE ${where} ${sampleBy(range, query.resolution)} ORDER BY ts`
-                    : `SELECT ts, value FROM ${table} WHERE ${where} ORDER BY ts LIMIT 10000`;
-                const result = await queryClient.exec(sql);
                 // Held before averaging: an unchanged value is still a sample of
                 // the window, not a hole in it.
                 const dataset = sampled
-                    ? await holdNumericBuckets(result.dataset, seedWhere, seedFrom)
-                    : result.dataset;
+                    ? await holdNumericBuckets(await readSampled(base, archived[base], where, range, query.resolution, [bucket, HELD_COLUMNS]), seedWhere, seedFrom)
+                    : await readRaw(base, archived[base], "value", where);
                 // An angular series keeps one convention throughout: [0, 2π) unless
                 // a sample (or a bucket's lowest sample) was negative.
                 const floors = dataset.map((r) => angular && sampled
@@ -823,7 +1021,7 @@ archive = NO_ARCHIVE) {
                 columnData.set(specIndex, strResult.rows);
             };
             if (spec.aggregate === "middle_index") {
-                const result = await queryClient.exec(middleRowSql(table, "value", where, range, query.resolution));
+                const result = await queryClient.exec(middleRowSql(union(), "value", where, range, query.resolution));
                 const rows = result.dataset.map((r) => [
                     r[0],
                     r[1],
@@ -840,17 +1038,13 @@ archive = NO_ARCHIVE) {
             // the caller's choice, and first, last and middle_index are samples.
             const angularMean = spec.aggregate === "average" &&
                 isAngular(unitsOf(spec.path, requestedContext));
-            const aggExpr = angularMean
-                ? ANGULAR_BUCKET_SQL
-                : `${aggregateToSql(spec.aggregate)} as agg_value`;
+            const value = angularMean
+                ? ANGULAR_COLUMNS
+                : valueColumns(spec.aggregate);
             const sampled = !!query.resolution && query.resolution > 0;
-            const sql = sampled
-                ? `SELECT ts, ${aggExpr}, ${HELD_SQL} FROM ${table} WHERE ${where} ${sampleBy(range, query.resolution)} ORDER BY ts`
-                : `SELECT ts, value FROM ${table} WHERE ${where} ORDER BY ts LIMIT 10000`;
-            const result = await queryClient.exec(sql);
             const dataset = sampled
-                ? await holdNumericBuckets(result.dataset, seedWhere, seedFrom)
-                : result.dataset;
+                ? await holdNumericBuckets(await readSampled(base, archived[base], where, range, query.resolution, [value, HELD_COLUMNS]), seedWhere, seedFrom)
+                : await readRaw(base, archived[base], "value", where);
             const rows = dataset.map((row) => [
                 row[0],
                 angularMean && row.length > 2 && row[1] !== null