signalk-questdb 2.1.4 → 2.1.5-beta.0

package/README.md

@@ -158,24 +158,28 @@ Registered via `app.registerHistoryProvider()`. Supports playback at configurabl
 
 All mounted at `/plugins/signalk-questdb/api/`:
 
-| Method | Path                                     | Description                                                                              |
-| ------ | ---------------------------------------- | ---------------------------------------------------------------------------------------- |
-| GET    | `/status`                                | QuestDB health, row counts, active paths                                                 |
-| GET    | `/query?sql=...`                         | Read-only SQL proxy (DDL/DML blocked)                                                    |
-| GET    | `/paths`                                 | All recorded paths with row counts and time range                                        |
-| GET    | `/versions`                              | QuestDB releases from GitHub (for version picker)                                        |
-| GET    | `/update/check`                          | Compare running version against latest release                                           |
-| POST   | `/update/apply`                          | Pull latest image, recreate container, reconnect                                         |
-| POST   | `/purge-data`                            | Remove the QuestDB container and delete all its data (rootless-Podman-safe)              |
-| GET    | `/migration/detect`                      | Detect InfluxDB (supports `?url=` for remote)                                            |
-| POST   | `/migration/buckets`                     | List buckets (2.x) or databases (1.x); credentials in the body, never the query string   |
-| POST   | `/migration/measurements`                | List measurements and their field keys in a bucket/database                              |
-| POST   | `/migration/start`                       | Start an import; returns immediately, progress via `/migration/status`                   |
-| GET    | `/migration/status`                      | Progress and state of the current/last import                                            |
-| POST   | `/migration/cancel`                      | Cancel the running import                                                                |
-| GET    | `/export?from=...&to=...&format=parquet` | Parquet or CSV export of the `signalk` numeric table (date range required)               |
-| GET    | `/full-export/tables`                    | List tables exposed by the per-table full-export route                                   |
-| GET    | `/full-export/:table?from=...&to=...`    | Stream a table as Parquet. Optional half-open `[from, to)` range for slicing into shards |
+| Method | Path                                     | Description                                                                               |
+| ------ | ---------------------------------------- | ----------------------------------------------------------------------------------------- |
+| GET    | `/status`                                | QuestDB health, row counts, active paths                                                  |
+| GET    | `/query?sql=...`                         | Read-only SQL proxy (DDL/DML blocked)                                                     |
+| GET    | `/paths`                                 | All recorded paths with row counts and time range                                         |
+| GET    | `/versions`                              | QuestDB releases from GitHub (for version picker)                                         |
+| GET    | `/update/check`                          | Compare running version against latest release                                            |
+| POST   | `/update/apply`                          | Pull latest image, recreate container, reconnect                                          |
+| POST   | `/purge-data`                            | Remove the QuestDB container and delete all its data (rootless-Podman-safe)               |
+| GET    | `/migration/detect`                      | Detect InfluxDB (supports `?url=` for remote)                                             |
+| POST   | `/migration/buckets`                     | List buckets (2.x) or databases (1.x); credentials in the body, never the query string    |
+| POST   | `/migration/measurements`                | List measurements and their field keys in a bucket/database                               |
+| POST   | `/migration/contexts`                    | List the vessels (`context` tag values) a bucket/database holds, and which is this server |
+| POST   | `/migration/start`                       | Start an import, or continue a stopped one with `{"resume": true}`; returns immediately   |
+| GET    | `/migration/status`                      | Progress and state of the current/last import, and any stopped import that can resume     |
+| POST   | `/migration/cancel`                      | Cancel the running import; any saved position is kept                                     |
+| POST   | `/migration/discard`                     | Forget a stopped import's saved position, so the next start begins again                  |
+| GET    | `/migration/legacy-rows`                 | Count rows an import made before 2.1.5 filed under suffixed paths                         |
+| POST   | `/migration/legacy-rows/remove`          | Rebuild the string table without those rows; recording continues meanwhile                |
+| GET    | `/export?from=...&to=...&format=parquet` | Parquet or CSV export of the `signalk` numeric table (date range required)                |
+| GET    | `/full-export/tables`                    | List tables exposed by the per-table full-export route                                    |
+| GET    | `/full-export/:table?from=...&to=...`    | Stream a table as Parquet. Optional half-open `[from, to)` range for slicing into shards  |
 
 ### `/full-export/:table` (since 0.4.0)
 
@@ -206,20 +210,79 @@ existing InfluxDB into QuestDB. It supports both InfluxDB 1.x (InfluxQL) and
 How the data maps:
 
 - A measurement with the conventional `value` field becomes the Signal K path
-  of the same name; a measurement with several named fields becomes
-  `measurement.field` paths, so two fields cannot overwrite each other.
+  of the same name. So do the typed fields signalk-to-influxdb 1.x writes
+  (`stringValue`, `boolValue`, `jsonValue`). A measurement with any other named
+  fields becomes `measurement.field` paths, so two fields cannot overwrite each
+  other.
 - Numbers go to `signalk`, strings and booleans to `signalk_str` (booleans
-  tagged `value_kind=boolean`), and `latitude`/`longitude` field pairs are
-  recombined into `signalk_position`.
+  tagged `value_kind=boolean`). A `jsonValue` is decoded first: an object is
+  recorded as its scalar leaves, the same as live data.
+- Positions go to `signalk_position`, whether stored as a `jsonValue`
+  (signalk-to-influxdb 1.x) or as a `lat`/`lon` or `latitude`/`longitude` field
+  pair (signalk-to-influxdb2).
+- Both writers tag every point with the **context** of the vessel it belongs
+  to, so a source that recorded other vessels holds several. The panel lists
+  them and asks which is yours — preselected when the source knows this
+  server's identity, or holds only one vessel — and that vessel's history is
+  imported as `self`. The others are imported under their own contexts, as the
+  history API serves them, or left out if you untick that. The recording
+  server's identity is usually not this server's (a new install has a new
+  one), which is why it is chosen rather than matched.
 - Rows keep their **original nanosecond timestamps**, so imported history sorts
   and aggregates alongside live data.
+- History is **streamed**: read in batches and written before more is asked
+  for, pausing while QuestDB catches up. Memory use stays flat however dense
+  the source is, so a large import is safe on a small board.
 - Every imported row is tagged `source=influxdb-import`, which makes it
   distinguishable from live recording — and because the tables deduplicate on
   `(ts, path, context, source)`, **re-running the same range overwrites rather
-  than duplicating**. An interrupted import can simply be run again.
-
-Anything that cannot be mapped (a gap, an unsupported value type, a latitude
-with no matching longitude) is counted in the run's `skipped` total rather than
+  than duplicating**.
+
+### Imports made before 2.1.5
+
+Imports from a signalk-to-influxdb 1.x database made before 2.1.5 filed every
+string, boolean and position under a path suffixed with the InfluxDB field
+name — `navigation.state.stringValue`, `steering.autopilot.engaged.boolValue`,
+`navigation.position.jsonValue` — which nothing reads. Running the import again
+writes them under their real paths but cannot remove the old rows: the path is
+part of the deduplication key, and QuestDB has no row delete. When such rows
+exist, the panel says how many and offers **Remove these rows**, which rebuilds
+`signalk_str` without them. Recording continues while it runs; only the swap
+of the rebuilt table for the old one holds the writer, for a few seconds, and
+rows recorded during the rebuild are carried across. Should recording be fast
+enough to fill the writer's buffer during those seconds, the samples it could
+not keep are counted and the panel says how many. Run the import again
+afterwards.
+
+### Resuming an interrupted import
+
+A large import runs for hours. If it stops part-way — cancelled, failed, or the
+Signal K server restarted — the panel shows what was interrupted and offers
+**Resume import**, which continues from the saved position instead of starting
+again at the first measurement. It needs only the credentials, if InfluxDB asks
+for any: the source and range are remembered, credentials never are. Starting
+the identical import again the ordinary way continues it too. **Discard saved
+position** starts over.
+
+A position is saved only once QuestDB has been asked for the newest row written
+before it and has it — ILP gives no acknowledgment, so the plugin reads the row
+back — and once the position is about a minute old. The first keeps a stalled
+or restarted QuestDB from letting the position past rows it never took, however
+long the stall. The minute covers what a committed row still has to survive: a
+crash of either process, and on a power cut the kernel's default write-back
+timing for a QuestDB running on its default `nosync` — an assumption, not a
+guarantee, which is why an external QuestDB should run with `sync` like the
+managed container does (see
+[Durability on power loss](#durability-on-power-loss)). The position file is
+synced and renamed into place, so a power cut leaves the previous position or
+the new one, never a damaged one; should none survive, the import starts again,
+which is always safe. The cost is that a resumed import repeats up to a minute
+or so of work, which the deduplication above makes harmless. An import shorter
+than that saves nothing, and is simply run again.
+
+Anything that cannot be mapped (a gap, an unsupported value type, a `jsonValue`
+that is not valid JSON, a latitude with no matching longitude) is counted in the
+run's `skipped` total rather than
 being dropped silently.
 
 ## Configuration
@@ -681,7 +744,29 @@ you had chosen, and on servers with security enabled it never worked at all.
 
 ## Data Storage
 
-QuestDB data is stored at `~/.signalk/plugin-config-data/signalk-questdb/` on the host, mounted into the container at `/var/lib/questdb`. Data survives container restarts, image upgrades, and plugin disable/enable cycles.
+QuestDB's data root is the plugin's own data directory,
+`<config>/plugin-config-data/signalk-questdb`, whatever the deployment. Data
+survives container restarts, image upgrades, and plugin disable/enable cycles.
+How it reaches the container differs:
+
+- **Bare-metal Signal K**: that directory, mounted at `/var/lib/questdb`.
+- **Signal K in a container with its config directory bind-mounted**: the
+  exact host path of that directory, mounted at `/var/lib/questdb`.
+- **Signal K in a container with its config directory on a named volume**:
+  the volume cannot be mounted from a subdirectory, so it is mounted whole at
+  `/signalk-vols/<volume>` and QuestDB is pointed at the plugin's directory
+  inside it (`QUESTDB_DATA_DIR`). A database found at the volume's root — where
+  a whole-volume mount put it — is moved into the plugin's directory on start;
+  both are the same volume, so nothing is copied. A purge deletes the
+  directory from the Signal K process, since the runtime cannot mount the
+  volume by Signal K's path; where QuestDB's user owns it and the Signal K
+  user cannot delete it, the purge says so and the directory is deleted by
+  hand.
+
+QuestDB's entrypoint makes its data root its own on start, so the plugin keeps
+the one file it writes alongside — the import checkpoint,
+`signalk-questdb.influx-import-checkpoint.json` — next to the directory rather
+than in it. A purge removes it, whether or not the data could be deleted.
 
 ## Grafana Integration
 

package/dist/api-contract.d.ts

@@ -183,6 +183,24 @@ export interface MigrationMeasurement {
 export interface MigrationMeasurementsResponse extends ApiError {
     measurements: MigrationMeasurement[];
 }
+/** POST /api/migration/contexts: the vessels a bucket/database holds. */
+export interface MigrationContextsResponse extends ApiError {
+    /** Distinct `context` tag values, sorted. Empty for a non-Signal K schema. */
+    contexts: string[];
+    /** The one that is this server's own vessel, when it is among them. */
+    self?: string;
+}
+/**
+ * GET /api/migration/legacy-rows: rows an import made before 2.1.5 filed under
+ * suffixed paths, and POST /api/migration/legacy-rows/remove: how many it
+ * removed.
+ */
+export interface LegacyImportRowsResponse extends ApiError {
+    rows: number;
+    /** After a removal: live samples the writer's cap discarded while it was
+     * held for the table swap. Normally 0. */
+    dropped?: number;
+}
 /** Per-run counters, also the shape the panel renders as progress. */
 export interface MigrationProgress {
     /** Rows read from InfluxDB. */
@@ -201,6 +219,29 @@ export interface MigrationProgress {
     currentWindowStart?: string;
 }
 export type MigrationRunState = "running" | "done" | "failed" | "cancelled";
+/** Where a resumed run picked up. */
+export interface MigrationResumePoint {
+    /** Measurement that was under way; absent if it stopped between two. */
+    measurement?: string;
+    /** First window not yet imported for that measurement, ISO. */
+    windowStart?: string;
+}
+/**
+ * An import that stopped before it finished and can be continued. Carries the
+ * import's own parameters — never its credentials — so it can be resumed after
+ * a server restart, when the panel no longer remembers what was entered.
+ */
+export interface MigrationInterrupted extends MigrationResumePoint {
+    url: string;
+    type: string;
+    bucket: string;
+    from: string;
+    to: string;
+    /** Measurements already imported in full. */
+    measurementsDone: number;
+    /** When the saved position was reached, ISO. */
+    updatedAt: string;
+}
 /** GET /api/migration/status, and the body of POST /api/migration/start. */
 export interface MigrationStatusResponse extends ApiError {
     /** Absent when no run has ever been started this process lifetime. */
@@ -215,5 +256,9 @@ export interface MigrationStatusResponse extends ApiError {
         progress: MigrationProgress;
         /** Set when state is "failed". */
         error?: string;
+        /** Set when this run continued an earlier one rather than starting over. */
+        resumedFrom?: MigrationResumePoint;
     };
+    /** Present, when no run is active, if a stopped import can be continued. */
+    interrupted?: MigrationInterrupted;
 }

package/dist/configpanel/responses.d.ts

@@ -1,4 +1,4 @@
-import type { MigrationBucket, MigrationMeasurement, MigrationSource, MigrationStatusResponse } from "../api-contract.js";
+import type { MigrationBucket, MigrationContextsResponse, MigrationMeasurement, MigrationSource, MigrationStatusResponse } from "../api-contract.js";
 /**
  * GET /api/migration/detect returns `{ sources: [...] }`. A body without it
  * means "nothing detected", which is also what an empty array means — so the
@@ -21,6 +21,33 @@ export declare function toMigrationMeasurements(body: unknown): MigrationMeasure
  * render as undefined and printing "NaN%".
  */
 export declare function toMigrationStatus(body: unknown): MigrationStatusResponse["run"] | undefined;
+/**
+ * GET /api/migration/legacy-rows and its POST remove: a row count, or 0, and
+ * after a removal the live samples dropped during the swap, normally 0.
+ */
+export declare function toLegacyImportRows(body: unknown): {
+    rows: number;
+    dropped: number;
+};
+/**
+ * POST /api/migration/contexts: the vessels the source holds. A body without
+ * a proper list reads as "none found", which leaves the choice out of the form
+ * rather than rendering a broken one.
+ */
+export declare function toMigrationContexts(body: unknown): Pick<MigrationContextsResponse, "contexts" | "self">;
+/**
+ * An ISO instant from the server, in the reader's own locale and time zone.
+ * Anything that is not a date is shown as it came rather than as
+ * "Invalid Date".
+ */
+export declare function formatDateTime(iso: string): string;
+/**
+ * The stopped import a status response offers to continue, if any.
+ *
+ * Everything the notice prints is required: a partial one would render as
+ * "import of undefined", so a body missing any of it yields no notice at all.
+ */
+export declare function toMigrationInterrupted(body: unknown): MigrationStatusResponse["interrupted"] | undefined;
 /**
  * Validate the two `datetime-local` values an import range is entered as.
  *

package/dist/configpanel/responses.js

@@ -85,6 +85,12 @@ export function toMigrationStatus(body) {
         startedAt: str(run.startedAt) ?? "",
         finishedAt: str(run.finishedAt),
         error: str(run.error),
+        resumedFrom: isRecord(run.resumedFrom)
+            ? {
+                measurement: str(run.resumedFrom.measurement),
+                windowStart: str(run.resumedFrom.windowStart),
+            }
+            : undefined,
         progress: {
             read: num(p.read),
             written: num(p.written),
@@ -96,6 +102,74 @@ export function toMigrationStatus(body) {
         },
     };
 }
+/**
+ * GET /api/migration/legacy-rows and its POST remove: a row count, or 0, and
+ * after a removal the live samples dropped during the swap, normally 0.
+ */
+export function toLegacyImportRows(body) {
+    const count = (v) => typeof v === "number" && Number.isFinite(v) && v > 0 ? Math.floor(v) : 0;
+    return {
+        rows: count(isRecord(body) ? body.rows : undefined),
+        dropped: count(isRecord(body) ? body.dropped : undefined),
+    };
+}
+/**
+ * POST /api/migration/contexts: the vessels the source holds. A body without
+ * a proper list reads as "none found", which leaves the choice out of the form
+ * rather than rendering a broken one.
+ */
+export function toMigrationContexts(body) {
+    const contexts = isRecord(body) && Array.isArray(body.contexts)
+        ? body.contexts.filter((c) => typeof c === "string" && !!c)
+        : [];
+    const self = isRecord(body) &&
+        typeof body.self === "string" &&
+        contexts.includes(body.self)
+        ? body.self
+        : undefined;
+    return { contexts, self };
+}
+/**
+ * An ISO instant from the server, in the reader's own locale and time zone.
+ * Anything that is not a date is shown as it came rather than as
+ * "Invalid Date".
+ */
+export function formatDateTime(iso) {
+    const d = new Date(iso);
+    return Number.isNaN(d.getTime()) ? iso : d.toLocaleString();
+}
+/**
+ * The stopped import a status response offers to continue, if any.
+ *
+ * Everything the notice prints is required: a partial one would render as
+ * "import of undefined", so a body missing any of it yields no notice at all.
+ */
+export function toMigrationInterrupted(body) {
+    const i = isRecord(body) ? body.interrupted : undefined;
+    if (!isRecord(i))
+        return undefined;
+    if (typeof i.url !== "string" ||
+        typeof i.type !== "string" ||
+        typeof i.bucket !== "string" ||
+        typeof i.from !== "string" ||
+        typeof i.to !== "string" ||
+        typeof i.updatedAt !== "string")
+        return undefined;
+    return {
+        url: i.url,
+        type: i.type,
+        bucket: i.bucket,
+        from: i.from,
+        to: i.to,
+        updatedAt: i.updatedAt,
+        measurementsDone: typeof i.measurementsDone === "number" &&
+            Number.isFinite(i.measurementsDone)
+            ? i.measurementsDone
+            : 0,
+        measurement: typeof i.measurement === "string" ? i.measurement : undefined,
+        windowStart: typeof i.windowStart === "string" ? i.windowStart : undefined,
+    };
+}
 /**
  * Validate the two `datetime-local` values an import range is entered as.
  *

package/dist/configpanel/responses.js.map

@@ -1 +1 @@
-{"version":3,"file":"responses.js","sourceRoot":"","sources":["../../src/configpanel/responses.ts"],"names":[],"mappings":"AAAA,4EAA4E;AAC5E,EAAE;AACF,4EAA4E;AAC5E,6EAA6E;AAC7E,6EAA6E;AAC7E,wEAAwE;AACxE,4EAA4E;AAC5E,uEAAuE;AACvE,EAAE;AACF,4EAA4E;AAS5E,MAAM,QAAQ,GAAG,CAAC,CAAU,EAAgC,EAAE,CAC5D,OAAO,CAAC,KAAK,QAAQ,IAAI,CAAC,KAAK,IAAI,CAAC;AAEtC;;;;;GAKG;AACH,MAAM,UAAU,kBAAkB,CAAC,IAAa;IAC9C,MAAM,OAAO,GAAG,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC;IAC1D,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC;QAAE,OAAO,EAAE,CAAC;IACvC,OAAO,OAAO,CAAC,MAAM,CACnB,CAAC,CAAC,EAAwB,EAAE,CAC1B,QAAQ,CAAC,CAAC,CAAC;QACX,OAAO,CAAC,CAAC,IAAI,KAAK,QAAQ;QAC1B,OAAO,CAAC,CAAC,GAAG,KAAK,QAAQ;QACzB,iEAAiE;QACjE,kEAAkE;QAClE,iEAAiE;QACjE,sDAAsD;QACtD,OAAO,CAAC,CAAC,MAAM,KAAK,QAAQ;QAC5B,CAAC,CAAC,CAAC,OAAO,KAAK,SAAS,IAAI,OAAO,CAAC,CAAC,OAAO,KAAK,QAAQ,CAAC,CAC7D,CAAC;AACJ,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,kBAAkB,CAAC,IAAa;IAC9C,MAAM,OAAO,GAAG,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC;IAC1D,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC;QAAE,OAAO,EAAE,CAAC;IACvC,OAAO,OAAO,CAAC,MAAM,CACnB,CAAC,CAAC,EAAwB,EAAE,CAC1B,QAAQ,CAAC,CAAC,CAAC;QACX,OAAO,CAAC,CAAC,IAAI,KAAK,QAAQ;QAC1B,CAAC,CAAC,CAAC,EAAE,KAAK,SAAS,IAAI,OAAO,CAAC,CAAC,EAAE,KAAK,QAAQ,CAAC;QAChD,CAAC,CAAC,CAAC,gBAAgB,KAAK,SAAS;YAC/B,OAAO,CAAC,CAAC,gBAAgB,KAAK,QAAQ,CAAC,CAC5C,CAAC;AACJ,CAAC;AAED,wCAAwC;AACxC,MAAM,UAAU,uBAAuB,CAAC,IAAa;IACnD,MAAM,IAAI,GAAG,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,YAAY,CAAC,CAAC,CAAC,SAAS,CAAC;IAC5D,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC;QAAE,OAAO,EAAE,CAAC;IACpC,OAAO,IAAI,CAAC,MAAM,CAChB,CAAC,CAAC,EAA6B,EAAE,CAC/B,QAAQ,CAAC,CAAC,CAAC;QACX,OAAO,CAAC,CAAC,IAAI,KAAK,QAAQ;QAC1B,sEAAsE;QACtE,wEAAwE;QACxE,4DAA4D;QAC5D,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,CAC1B,CAAC;AACJ,CAAC;AAED;;;;;;GAMG;AACH,MAAM,UAAU,iBAAiB,CAC/B,IAAa;IAEb,MAAM,GAAG,GAAG,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,SAAS,CAAC;IAClD,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC;QAAE,OAAO,SAAS,CAAC;IACrC,IAAI,OAAO,GAAG,CAAC,EAAE,KAAK,QAAQ,IAAI,OAAO,GAAG,CAAC,KAAK,KAAK,QAAQ;QAC7D,OAAO,SAAS,CAAC;IACnB,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,CAAC;IACxB,IACE,KAAK,KAAK,SAAS;QACnB,KAAK,KAAK,MAAM;QAChB,KAAK,KAAK,QAAQ;QAClB,KAAK,KAAK,WAAW;QAErB,OAAO,SAAS,CAAC;IACnB,MAAM,CAAC,GAAG,QAAQ,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,CAAC;IACrD,MAAM,GAAG,GAAG,CAAC,CAAU,EAAU,EAAE,CACjC,OAAO,CAAC,KAAK,QAAQ,IAAI,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACtD,MAAM,GAAG,GAAG,CAAC,CAAU,EAAsB,EAAE,CAC7C,OAAO,CAAC,KAAK,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC;IACxC,OAAO;QACL,EAAE,EAAE,GAAG,CAAC,EAAE;QACV,KAAK;QACL,GAAG,EAAE,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC,IAAI,EAAE;QACvB,MAAM,EAAE,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,IAAI,EAAE;QAC7B,SAAS,EAAE,GAAG,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,EAAE;QACnC,UAAU,EAAE,GAAG,CAAC,GAAG,CAAC,UAAU,CAAC;QAC/B,KAAK,EAAE,GAAG,CAAC,GAAG,CAAC,KAAK,CAAC;QACrB,QAAQ,EAAE;YACR,IAAI,EAAE,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC;YACjB,OAAO,EAAE,GAAG,CAAC,CAAC,CAAC,OAAO,CAAC;YACvB,OAAO,EAAE,GAAG,CAAC,CAAC,CAAC,OAAO,CAAC;YACvB,gBAAgB,EAAE,GAAG,CAAC,CAAC,CAAC,gBAAgB,CAAC;YACzC,iBAAiB,EAAE,GAAG,CAAC,CAAC,CAAC,iBAAiB,CAAC;YAC3C,kBAAkB,EAAE,GAAG,CAAC,CAAC,CAAC,kBAAkB,CAAC;YAC7C,kBAAkB,EAAE,GAAG,CAAC,CAAC,CAAC,kBAAkB,CAAC;SAC9C;KACF,CAAC;AACJ,CAAC;AAED;;;;;;;;;;GAUG;AACH,MAAM,UAAU,gBAAgB,CAC9B,IAAY,EACZ,EAAU;IAEV,MAAM,QAAQ,GAAG,IAAI,IAAI,CAAC,IAAI,CAAC,CAAC;IAChC,MAAM,MAAM,GAAG,IAAI,IAAI,CAAC,EAAE,CAAC,CAAC;IAC5B,IAAI,MAAM,CAAC,KAAK,CAAC,QAAQ,CAAC,OAAO,EAAE,CAAC,IAAI,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,OAAO,EAAE,CAAC,EAAE,CAAC;QACvE,OAAO,EAAE,KAAK,EAAE,gDAAgD,EAAE,CAAC;IACrE,CAAC;IACD,IAAI,MAAM,IAAI,QAAQ,EAAE,CAAC;QACvB,OAAO,EAAE,KAAK,EAAE,0CAA0C,EAAE,CAAC;IAC/D,CAAC;IACD,OAAO,EAAE,IAAI,EAAE,QAAQ,CAAC,WAAW,EAAE,EAAE,EAAE,EAAE,MAAM,CAAC,WAAW,EAAE,EAAE,CAAC;AACpE,CAAC"}
+{"version":3,"file":"responses.js","sourceRoot":"","sources":["../../src/configpanel/responses.ts"],"names":[],"mappings":"AAAA,4EAA4E;AAC5E,EAAE;AACF,4EAA4E;AAC5E,6EAA6E;AAC7E,6EAA6E;AAC7E,wEAAwE;AACxE,4EAA4E;AAC5E,uEAAuE;AACvE,EAAE;AACF,4EAA4E;AAU5E,MAAM,QAAQ,GAAG,CAAC,CAAU,EAAgC,EAAE,CAC5D,OAAO,CAAC,KAAK,QAAQ,IAAI,CAAC,KAAK,IAAI,CAAC;AAEtC;;;;;GAKG;AACH,MAAM,UAAU,kBAAkB,CAAC,IAAa;IAC9C,MAAM,OAAO,GAAG,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC;IAC1D,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC;QAAE,OAAO,EAAE,CAAC;IACvC,OAAO,OAAO,CAAC,MAAM,CACnB,CAAC,CAAC,EAAwB,EAAE,CAC1B,QAAQ,CAAC,CAAC,CAAC;QACX,OAAO,CAAC,CAAC,IAAI,KAAK,QAAQ;QAC1B,OAAO,CAAC,CAAC,GAAG,KAAK,QAAQ;QACzB,iEAAiE;QACjE,kEAAkE;QAClE,iEAAiE;QACjE,sDAAsD;QACtD,OAAO,CAAC,CAAC,MAAM,KAAK,QAAQ;QAC5B,CAAC,CAAC,CAAC,OAAO,KAAK,SAAS,IAAI,OAAO,CAAC,CAAC,OAAO,KAAK,QAAQ,CAAC,CAC7D,CAAC;AACJ,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,kBAAkB,CAAC,IAAa;IAC9C,MAAM,OAAO,GAAG,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC;IAC1D,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC;QAAE,OAAO,EAAE,CAAC;IACvC,OAAO,OAAO,CAAC,MAAM,CACnB,CAAC,CAAC,EAAwB,EAAE,CAC1B,QAAQ,CAAC,CAAC,CAAC;QACX,OAAO,CAAC,CAAC,IAAI,KAAK,QAAQ;QAC1B,CAAC,CAAC,CAAC,EAAE,KAAK,SAAS,IAAI,OAAO,CAAC,CAAC,EAAE,KAAK,QAAQ,CAAC;QAChD,CAAC,CAAC,CAAC,gBAAgB,KAAK,SAAS;YAC/B,OAAO,CAAC,CAAC,gBAAgB,KAAK,QAAQ,CAAC,CAC5C,CAAC;AACJ,CAAC;AAED,wCAAwC;AACxC,MAAM,UAAU,uBAAuB,CAAC,IAAa;IACnD,MAAM,IAAI,GAAG,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,YAAY,CAAC,CAAC,CAAC,SAAS,CAAC;IAC5D,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC;QAAE,OAAO,EAAE,CAAC;IACpC,OAAO,IAAI,CAAC,MAAM,CAChB,CAAC,CAAC,EAA6B,EAAE,CAC/B,QAAQ,CAAC,CAAC,CAAC;QACX,OAAO,CAAC,CAAC,IAAI,KAAK,QAAQ;QAC1B,sEAAsE;QACtE,wEAAwE;QACxE,4DAA4D;QAC5D,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,CAC1B,CAAC;AACJ,CAAC;AAED;;;;;;GAMG;AACH,MAAM,UAAU,iBAAiB,CAC/B,IAAa;IAEb,MAAM,GAAG,GAAG,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,SAAS,CAAC;IAClD,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC;QAAE,OAAO,SAAS,CAAC;IACrC,IAAI,OAAO,GAAG,CAAC,EAAE,KAAK,QAAQ,IAAI,OAAO,GAAG,CAAC,KAAK,KAAK,QAAQ;QAC7D,OAAO,SAAS,CAAC;IACnB,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,CAAC;IACxB,IACE,KAAK,KAAK,SAAS;QACnB,KAAK,KAAK,MAAM;QAChB,KAAK,KAAK,QAAQ;QAClB,KAAK,KAAK,WAAW;QAErB,OAAO,SAAS,CAAC;IACnB,MAAM,CAAC,GAAG,QAAQ,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,CAAC;IACrD,MAAM,GAAG,GAAG,CAAC,CAAU,EAAU,EAAE,CACjC,OAAO,CAAC,KAAK,QAAQ,IAAI,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACtD,MAAM,GAAG,GAAG,CAAC,CAAU,EAAsB,EAAE,CAC7C,OAAO,CAAC,KAAK,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC;IACxC,OAAO;QACL,EAAE,EAAE,GAAG,CAAC,EAAE;QACV,KAAK;QACL,GAAG,EAAE,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC,IAAI,EAAE;QACvB,MAAM,EAAE,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,IAAI,EAAE;QAC7B,SAAS,EAAE,GAAG,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,EAAE;QACnC,UAAU,EAAE,GAAG,CAAC,GAAG,CAAC,UAAU,CAAC;QAC/B,KAAK,EAAE,GAAG,CAAC,GAAG,CAAC,KAAK,CAAC;QACrB,WAAW,EAAE,QAAQ,CAAC,GAAG,CAAC,WAAW,CAAC;YACpC,CAAC,CAAC;gBACE,WAAW,EAAE,GAAG,CAAC,GAAG,CAAC,WAAW,CAAC,WAAW,CAAC;gBAC7C,WAAW,EAAE,GAAG,CAAC,GAAG,CAAC,WAAW,CAAC,WAAW,CAAC;aAC9C;YACH,CAAC,CAAC,SAAS;QACb,QAAQ,EAAE;YACR,IAAI,EAAE,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC;YACjB,OAAO,EAAE,GAAG,CAAC,CAAC,CAAC,OAAO,CAAC;YACvB,OAAO,EAAE,GAAG,CAAC,CAAC,CAAC,OAAO,CAAC;YACvB,gBAAgB,EAAE,GAAG,CAAC,CAAC,CAAC,gBAAgB,CAAC;YACzC,iBAAiB,EAAE,GAAG,CAAC,CAAC,CAAC,iBAAiB,CAAC;YAC3C,kBAAkB,EAAE,GAAG,CAAC,CAAC,CAAC,kBAAkB,CAAC;YAC7C,kBAAkB,EAAE,GAAG,CAAC,CAAC,CAAC,kBAAkB,CAAC;SAC9C;KACF,CAAC;AACJ,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,kBAAkB,CAAC,IAAa;IAI9C,MAAM,KAAK,GAAG,CAAC,CAAU,EAAU,EAAE,CACnC,OAAO,CAAC,KAAK,QAAQ,IAAI,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAC3E,OAAO;QACL,IAAI,EAAE,KAAK,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC;QACnD,OAAO,EAAE,KAAK,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC;KAC1D,CAAC;AACJ,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,mBAAmB,CACjC,IAAa;IAEb,MAAM,QAAQ,GACZ,QAAQ,CAAC,IAAI,CAAC,IAAI,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,QAAQ,CAAC;QAC5C,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,EAAe,EAAE,CAAC,OAAO,CAAC,KAAK,QAAQ,IAAI,CAAC,CAAC,CAAC,CAAC;QACxE,CAAC,CAAC,EAAE,CAAC;IACT,MAAM,IAAI,GACR,QAAQ,CAAC,IAAI,CAAC;QACd,OAAO,IAAI,CAAC,IAAI,KAAK,QAAQ;QAC7B,QAAQ,CAAC,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC;QAC1B,CAAC,CAAC,IAAI,CAAC,IAAI;QACX,CAAC,CAAC,SAAS,CAAC;IAChB,OAAO,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC;AAC5B,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,cAAc,CAAC,GAAW;IACxC,MAAM,CAAC,GAAG,IAAI,IAAI,CAAC,GAAG,CAAC,CAAC;IACxB,OAAO,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,cAAc,EAAE,CAAC;AAC9D,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,sBAAsB,CACpC,IAAa;IAEb,MAAM,CAAC,GAAG,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC,CAAC,SAAS,CAAC;IACxD,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC;QAAE,OAAO,SAAS,CAAC;IACnC,IACE,OAAO,CAAC,CAAC,GAAG,KAAK,QAAQ;QACzB,OAAO,CAAC,CAAC,IAAI,KAAK,QAAQ;QAC1B,OAAO,CAAC,CAAC,MAAM,KAAK,QAAQ;QAC5B,OAAO,CAAC,CAAC,IAAI,KAAK,QAAQ;QAC1B,OAAO,CAAC,CAAC,EAAE,KAAK,QAAQ;QACxB,OAAO,CAAC,CAAC,SAAS,KAAK,QAAQ;QAE/B,OAAO,SAAS,CAAC;IACnB,OAAO;QACL,GAAG,EAAE,CAAC,CAAC,GAAG;QACV,IAAI,EAAE,CAAC,CAAC,IAAI;QACZ,MAAM,EAAE,CAAC,CAAC,MAAM;QAChB,IAAI,EAAE,CAAC,CAAC,IAAI;QACZ,EAAE,EAAE,CAAC,CAAC,EAAE;QACR,SAAS,EAAE,CAAC,CAAC,SAAS;QACtB,gBAAgB,EACd,OAAO,CAAC,CAAC,gBAAgB,KAAK,QAAQ;YACtC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,gBAAgB,CAAC;YACjC,CAAC,CAAC,CAAC,CAAC,gBAAgB;YACpB,CAAC,CAAC,CAAC;QACP,WAAW,EAAE,OAAO,CAAC,CAAC,WAAW,KAAK,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,CAAC,SAAS;QAC1E,WAAW,EAAE,OAAO,CAAC,CAAC,WAAW,KAAK,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,CAAC,SAAS;KAC3E,CAAC;AACJ,CAAC;AAED;;;;;;;;;;GAUG;AACH,MAAM,UAAU,gBAAgB,CAC9B,IAAY,EACZ,EAAU;IAEV,MAAM,QAAQ,GAAG,IAAI,IAAI,CAAC,IAAI,CAAC,CAAC;IAChC,MAAM,MAAM,GAAG,IAAI,IAAI,CAAC,EAAE,CAAC,CAAC;IAC5B,IAAI,MAAM,CAAC,KAAK,CAAC,QAAQ,CAAC,OAAO,EAAE,CAAC,IAAI,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,OAAO,EAAE,CAAC,EAAE,CAAC;QACvE,OAAO,EAAE,KAAK,EAAE,gDAAgD,EAAE,CAAC;IACrE,CAAC;IACD,IAAI,MAAM,IAAI,QAAQ,EAAE,CAAC;QACvB,OAAO,EAAE,KAAK,EAAE,0CAA0C,EAAE,CAAC;IAC/D,CAAC;IACD,OAAO,EAAE,IAAI,EAAE,QAAQ,CAAC,WAAW,EAAE,EAAE,EAAE,EAAE,MAAM,CAAC,WAAW,EAAE,EAAE,CAAC;AACpE,CAAC"}

package/dist/ilp-writer.d.ts

@@ -11,11 +11,16 @@ export declare class ILPWriter {
     private connected;
     private connecting;
     private stopped;
+    private held;
+    private watchingTimestamps;
+    private oldestWatched;
     private lastNanos;
     private connectedAt;
     private consecutiveFlaps;
     private droppedLines;
     private totalDroppedLines;
+    private totalEnqueuedLines;
+    private totalSettledLines;
     private readonly maxBufferLines;
     private unhealthy;
     private awaitingDrain;
@@ -78,6 +83,53 @@ export declare class ILPWriter {
     private requeueFront;
     private enforceBufferCap;
     get droppedLineCount(): number;
+    /**
+     * Every line ever enqueued, monotonic: a line's place in the sequence. With
+     * the two counters below it tells a caller whether what it wrote up to some
+     * point has left this process. Lines leave in the order they came, so once
+     * `settledLineCount + droppedLineCount` reaches the `enqueuedLineCount` it
+     * noted then, nothing enqueued before that point is still waiting here.
+     */
+    get enqueuedLineCount(): number;
+    /**
+     * From now on, note the oldest timestamp of the lines written.
+     *
+     * A caller that copies a table while the writer keeps writing into it needs
+     * to know how far back the rows it must carry over can reach, and only the
+     * writer sees the timestamps: a source with a wrong clock records the
+     * present under any date it likes.
+     */
+    startTimestampWatch(): void;
+    /** The oldest timestamp written since the watch began — null if nothing
+     * was — and the end of the watch. */
+    endTimestampWatch(): bigint | null;
+    /**
+     * Send what is buffered, then keep further lines back until release().
+     *
+     * For the seconds a table is swapped out from under the writer: a line
+     * arriving while the table is absent would have QuestDB create a fresh one
+     * with the wrong schema, and the swap would then fail against it. Flushing
+     * first makes the last line written before the hold the last line sent, so
+     * a caller that wrote a marker can wait for QuestDB to show it and know
+     * everything before it has arrived too. Lines keep being accepted and the
+     * buffer cap keeps applying, so a hold must be short.
+     */
+    hold(): void;
+    release(): void;
+    /**
+     * Whether the next enqueue would push the cap and discard the oldest line.
+     * A bulk writer that must never write behind a gap checks this first: the
+     * check and its write run without yielding, so nothing can fill the buffer
+     * in between.
+     */
+    get atCapacity(): boolean;
+    /**
+     * Lines the socket has accepted, monotonic and exact — counted when the
+     * write completes, not when it is issued, and never for a batch that failed
+     * and was re-queued. Unlike `pendingLines`, which estimates the socket's
+     * share from a byte count and is only good for deciding whether to pause.
+     */
+    get settledLineCount(): number;
     private flush;
     private startFlushTimer;
     private stopFlushTimer;

package/dist/ilp-writer.js

@@ -71,6 +71,9 @@ export class ILPWriter {
     connected = false;
     connecting = false;
     stopped = false;
+    held = false;
+    watchingTimestamps = false;
+    oldestWatched = null;
     // Last timestamp handed out, in nanoseconds. Rows are stamped at write time
     // with the server clock (see the delta handler), but `Date` is only
     // millisecond-resolution while the `signalk`/`signalk_str` tables dedup on
@@ -95,6 +98,8 @@ export class ILPWriter {
     // is at least counted.
     droppedLines = 0;
     totalDroppedLines = 0;
+    totalEnqueuedLines = 0;
+    totalSettledLines = 0;
     maxBufferLines;
     unhealthy = false;
     // True between a backpressured write and the socket's next drain. Guards
@@ -291,16 +296,16 @@ export class ILPWriter {
     // up, so an old timestamp would be ignored anyway) nor push it forward.
     writeAtNanos(path, context, value, tsNanos, source) {
         const sourceTag = source ? `,source=${escapeTag(source)}` : "";
-        this.enqueue(`signalk,path=${escapeTag(path)},context=${escapeTag(context)}${sourceTag} value=${value} ${tsNanos}\n`);
+        this.enqueue(`signalk,path=${escapeTag(path)},context=${escapeTag(context)}${sourceTag} value=${value} ${tsNanos}\n`, tsNanos);
     }
     writeStringAtNanos(path, context, value, tsNanos, kind, source) {
         const sourceTag = source ? `,source=${escapeTag(source)}` : "";
         const kindTag = kind ? `,value_kind=${escapeTag(kind)}` : "";
-        this.enqueue(`signalk_str,path=${escapeTag(path)},context=${escapeTag(context)}${sourceTag}${kindTag} value_str="${escapeFieldString(value)}" ${tsNanos}\n`);
+        this.enqueue(`signalk_str,path=${escapeTag(path)},context=${escapeTag(context)}${sourceTag}${kindTag} value_str="${escapeFieldString(value)}" ${tsNanos}\n`, tsNanos);
     }
     writePositionAtNanos(context, position, tsNanos, source) {
         const sourceTag = source ? `,source=${escapeTag(source)}` : "";
-        this.enqueue(`signalk_position,context=${escapeTag(context)}${sourceTag} lat=${position.latitude},lon=${position.longitude} ${tsNanos}\n`);
+        this.enqueue(`signalk_position,context=${escapeTag(context)}${sourceTag} lat=${position.latitude},lon=${position.longitude} ${tsNanos}\n`, tsNanos);
     }
     /**
      * Roughly how much is still in flight, so a bulk importer can apply
@@ -328,7 +333,7 @@ export class ILPWriter {
     write(path, context, value, timestamp, source) {
         const ts = this.nextNanos(timestamp);
         const sourceTag = source ? `,source=${escapeTag(source)}` : "";
-        this.enqueue(`signalk,path=${escapeTag(path)},context=${escapeTag(context)}${sourceTag} value=${value} ${ts}\n`);
+        this.enqueue(`signalk,path=${escapeTag(path)},context=${escapeTag(context)}${sourceTag} value=${value} ${ts}\n`, ts);
     }
     // `kind` records what the value ORIGINALLY was, so a boolean recorded as
     // "true" stays distinguishable from a path whose text value is the word
@@ -339,17 +344,22 @@ export class ILPWriter {
         const ts = this.nextNanos(timestamp);
         const sourceTag = source ? `,source=${escapeTag(source)}` : "";
         const kindTag = kind ? `,value_kind=${escapeTag(kind)}` : "";
-        this.enqueue(`signalk_str,path=${escapeTag(path)},context=${escapeTag(context)}${sourceTag}${kindTag} value_str="${escapeFieldString(value)}" ${ts}\n`);
+        this.enqueue(`signalk_str,path=${escapeTag(path)},context=${escapeTag(context)}${sourceTag}${kindTag} value_str="${escapeFieldString(value)}" ${ts}\n`, ts);
     }
     // signalk_position has no path column — it holds navigation.position rows
     // exclusively (the caller enforces that), keyed on KEYS(ts, context, source).
     writePosition(context, position, timestamp, source) {
         const ts = this.nextNanos(timestamp);
         const sourceTag = source ? `,source=${escapeTag(source)}` : "";
-        this.enqueue(`signalk_position,context=${escapeTag(context)}${sourceTag} lat=${position.latitude},lon=${position.longitude} ${ts}\n`);
+        this.enqueue(`signalk_position,context=${escapeTag(context)}${sourceTag} lat=${position.latitude},lon=${position.longitude} ${ts}\n`, ts);
     }
-    enqueue(line) {
+    enqueue(line, tsNanos) {
         this.buffer.push(line);
+        this.totalEnqueuedLines++;
+        if (this.watchingTimestamps &&
+            (this.oldestWatched === null || tsNanos < this.oldestWatched)) {
+            this.oldestWatched = tsNanos;
+        }
         this.enforceBufferCap();
         if (this.buffer.length >= FLUSH_BATCH_SIZE) {
             this.flush();
@@ -385,7 +395,76 @@ export class ILPWriter {
     get droppedLineCount() {
         return this.totalDroppedLines;
     }
+    /**
+     * Every line ever enqueued, monotonic: a line's place in the sequence. With
+     * the two counters below it tells a caller whether what it wrote up to some
+     * point has left this process. Lines leave in the order they came, so once
+     * `settledLineCount + droppedLineCount` reaches the `enqueuedLineCount` it
+     * noted then, nothing enqueued before that point is still waiting here.
+     */
+    get enqueuedLineCount() {
+        return this.totalEnqueuedLines;
+    }
+    /**
+     * From now on, note the oldest timestamp of the lines written.
+     *
+     * A caller that copies a table while the writer keeps writing into it needs
+     * to know how far back the rows it must carry over can reach, and only the
+     * writer sees the timestamps: a source with a wrong clock records the
+     * present under any date it likes.
+     */
+    startTimestampWatch() {
+        this.watchingTimestamps = true;
+        this.oldestWatched = null;
+    }
+    /** The oldest timestamp written since the watch began — null if nothing
+     * was — and the end of the watch. */
+    endTimestampWatch() {
+        this.watchingTimestamps = false;
+        const oldest = this.oldestWatched;
+        this.oldestWatched = null;
+        return oldest;
+    }
+    /**
+     * Send what is buffered, then keep further lines back until release().
+     *
+     * For the seconds a table is swapped out from under the writer: a line
+     * arriving while the table is absent would have QuestDB create a fresh one
+     * with the wrong schema, and the swap would then fail against it. Flushing
+     * first makes the last line written before the hold the last line sent, so
+     * a caller that wrote a marker can wait for QuestDB to show it and know
+     * everything before it has arrived too. Lines keep being accepted and the
+     * buffer cap keeps applying, so a hold must be short.
+     */
+    hold() {
+        this.flush();
+        this.held = true;
+    }
+    release() {
+        this.held = false;
+        this.flush();
+    }
+    /**
+     * Whether the next enqueue would push the cap and discard the oldest line.
+     * A bulk writer that must never write behind a gap checks this first: the
+     * check and its write run without yielding, so nothing can fill the buffer
+     * in between.
+     */
+    get atCapacity() {
+        return this.buffer.length >= this.maxBufferLines;
+    }
+    /**
+     * Lines the socket has accepted, monotonic and exact — counted when the
+     * write completes, not when it is issued, and never for a batch that failed
+     * and was re-queued. Unlike `pendingLines`, which estimates the socket's
+     * share from a byte count and is only good for deciding whether to pause.
+     */
+    get settledLineCount() {
+        return this.totalSettledLines;
+    }
     flush() {
+        if (this.held)
+            return;
         if (!this.connected || !this.socket || this.buffer.length === 0)
             return;
         const data = this.buffer.join("");
@@ -402,7 +481,9 @@ export class ILPWriter {
             if (err) {
                 this.requeueFront(lines);
                 this.debug(`ILP write failed, re-queued batch: ${err.message}`);
+                return;
             }
+            this.totalSettledLines += lines.length;
         });
         if (!canWrite && !this.awaitingDrain) {
             // ONE listener at a time. `once` still accumulates while the socket