@voltro/database 0.43.1 → 0.44.0

package/CHANGELOG.md

@@ -39,6 +39,71 @@ _Changes staged for the next release accumulate here (rolled up from
 
 ---
 
+## [0.44.0] — 2026-08-19
+
+### ⚠ BREAKING
+
+- **@voltro/data-transfer, @voltro/cli** — `runImport` returns an `ImportOutcome` instead of the bundle's `Manifest`, and `--mode replace` runs as ONE transaction by default.
+
+  **Why the return type changed.** The summary line counted the rows the BUNDLE carries, not the rows the run wrote. Those differ most exactly where it matters: a bundle directory carries its own resume ledger, so a copied directory imports nothing — correctly, with a warning naming the file to delete — and the run then printed `import complete … rows: 242950` over a target it had not touched. The warning was one line above, which is one line too far for anyone piping the output through `tail -1`. The outcome carries `rowsWritten`, `rowsSkipped` and `fullyResumed`, the CLI reports written-vs-carried, and a fully-skipped run says so on its LAST line.
+
+  Migration: `runImport(...)` now resolves to `{ manifest, tablesWritten, rowsWritten, tablesSkipped, rowsSkipped, fullyResumed }`. Read `.manifest` where you read the manifest before.
+
+  **Why replace is atomic now.** The all-or-nothing guarantee was written for the emptying step, and read — reasonably — as covering the run. A replace that died partway through the LOAD left the target emptied of its old rows and holding part of the new ones, measured on a live instance over sixteen minutes. There is no useful state for a replace to stop in, so it is a default rather than a flag you have to know about. It also closes the window that produced that failure: with the tables emptied and the load uncommitted, a concurrent writer in the application waits instead of inserting a row the bundle is about to insert too.
+
+  `--no-atomic` (CLI) / `atomic: false` (API) opts out. The trade is stated where it bites: every write to those tables waits for the load, and on postgres the bulk `COPY` loader cannot join a transaction it does not own — an atomic run now says that once rather than being quietly slower.
+
+  Two smaller things from the same report: `--mode replace` against a running instance warns about the empty-target window when it is NOT atomic, and the `--target api` upload reports progress per chunk plus a line explaining that the final request stays open for the whole import — sixteen minutes of silence is indistinguishable from a hang, and one operator killed a run that had finished.
+
+  **`voltro update` carries you across this** — codemod `0.44.0/01_import-outcome`.
+
+### Fixed
+
+- **@voltro/sql-mysql, @voltro/sql-postgres, @voltro/sql-sqlite, @voltro/sql-mssql, @voltro/data-transfer** — A write and its write-recorders now succeed or fail together on every SQL dialect, and the data importer's retry is idempotent.
+
+  A recorder (a versioning trail, an audit log) runs on the caller's connection and is ALLOWED to fail — a recorder that throws must take the write down with it, that is its contract. Outside a caller transaction the two were not one unit: the row's statement committed on its own, and the recorder's INSERT ran afterwards as a second autocommit statement. So a recorder that threw left a COMMITTED row behind a write that reported failure. Measured directly: `insert` throws, the row is in the table, and a second attempt at the same row is `ER_DUP_ENTRY` on PRIMARY.
+
+  Anything that retries a failed write then meets its own row. The data importer retries by design — it holds a row whose write failed and tries again once the remaining tables have streamed — so a `--mode replace` that had just emptied a table failed on a duplicate key IN that table. From outside, that is the impossible-looking thing: an import that emptied a table and then failed because something was already in it. Reproduced verbatim, including the table name and `[ER_DUP_ENTRY/1062]`, by putting the old retry back.
+
+  Both ends are closed, and they are independent on purpose:
+
+  - **The cause.** A table that HAS recorders writes inside a transaction now, so the row and the trail commit together or not at all — on mysql, mariadb, postgres, sqlite and mssql, verified by one suite that asks all five the same question. A table with no recorders — the default — takes the direct path unchanged; `recordsTable` is a Map-size check first, so it costs one comparison. - **The defence.** The importer's retry upserts instead of inserting in `replace` mode. That covers every OTHER way a write can land while reporting failure: a driver timeout on a write the server applied, a connection lost after the commit, a concurrent writer inserting the same key. It is sound precisely because the table was emptied by this same run — there is nothing in it that is not ours.
+
+  The test that pinned the divergence went red when the fix landed, exactly as its own note said it would, and is inverted with that note kept.
+
+---
+
+## [0.43.2] — 2026-08-18
+
+### Fixed
+
+- **@voltro/sql-mysql, @voltro/database, @voltro/cli** — The mysql-family binlog reader no longer keeps a table excluded after the very migration that fixed it.
+
+  A UNIQUE on an unbounded text column is a MariaDB hash long-unique, whose hidden `DB_ROW_HASH_n` column the reader can never account for — so the table is held out of binlog capture and the exclusion is reported. `voltro dev` builds its store BEFORE it migrates (kv, cross-replica broadcast and the analytics mirror all need one), so on a boot whose own auto-migration bounds the column, that finding was drawn from a schema that stopped existing about a second later. The exclusion then outlived its cause for the life of the process, and the message — correct when written, and typically the only error line in the boot log — went on describing the pre-migration database.
+
+  Three changes:
+
+  - `DataStore.refreshChangeCaptureExclusions()` (optional; implemented by the mysql-family store) re-runs the probe and re-points the LIVE reader, in both directions — schema work that CREATES the condition now excludes the table immediately instead of after three failed writes. `voltro dev` calls it once, after all its schema work. `voltro serve` needs no equivalent: it builds its store after every schema step and never applies DDL itself. - The definitive message now states its own durability — that it is the schema as read at reader attach, and that applying the remedy does not by itself lift the exclusion. - On a boot that will re-check, the finding is reported as a provisional note rather than as a verdict, so a boot that fixes the condition leaves no error line about it. The note escalates to the full verdict on its own if the re-check never runs. - Two reader fixes the re-check depended on: applying a new exclusion set now WAITS for a reconnect already in flight (it is what applies the set, so resolving before it landed meant the caller's next write hit the old filter), and the reconnect loop no longer keeps resuming from an offset it has just jumped away from — that turned one purged offset into a reconnect every watchdog interval, forever, delivering nothing.
+- **@voltro/data-transfer, @voltro/database, @voltro/sql-mysql, @voltro/sql-postgres, @voltro/sql-sqlite, @voltro/sql-mssql, @voltro/cli** — `voltro data import --mode replace` no longer leaves a target in neither state, and now works against schemas that have foreign keys.
+
+  The delete step ran table by table and stopped at the first refusal, so a run that could not finish left dozens of tables emptied and nothing loaded — and a second attempt destroyed more than the first, because it got further before hitting the same wall. The wall itself was not exotic: MySQL, MariaDB and SQL Server check a foreign key as each ROW is deleted, so a table that references ITSELF cannot be emptied by any ordering of the tables. `createdBy → actors` on the `actors` table is exactly that shape, and it is what an audit mixin on an actor table produces.
+
+  - `DataStore.emptyTables()` (per dialect) empties the whole set as one unit, in one transaction, with referential integrity suspended for the duration — `FOREIGN_KEY_CHECKS` on the mysql family, a multi-table `TRUNCATE` on postgres, `defer_foreign_keys` on sqlite, per-table `NOCHECK`/`WITH CHECK CHECK` on mssql. All-or-nothing on every engine, including under `--atomic`, where it runs on the transaction the import already holds. - A **pre-flight refusal**: if a table OUTSIDE the bundle holds rows referencing one inside it, the import refuses before deleting anything and names the tables, the columns and the row counts. Those rows cannot be restored from the bundle, so forcing it is not an option. An EMPTY outside table blocks nothing. - Table-level failures carry the driver's own reason and code, the way row-level failures already did. `truncate <table> failed: Failed to execute statement` fits every plausible cause equally; the classification that produced `foreign key <name>: … [ER_NO_REFERENCED_ROW_2/1452]` one level down now applies one level up. The word "truncate" is gone from the message too — the step issues DELETE, and naming a statement it does not run sends whoever reads it to reproduce the wrong thing. - A typed refusal reaching the `--target api` transport keeps its text: the admin import endpoint answers `409` with the reason instead of flattening it to `import failed`, on the one transport where the operator has no other way to see it.
+
+  The bulk empty emits no change events, where the per-table loop emitted one per row. An import through `--target api` now asks every live subscription to re-read once it lands — the coarse refresh the framework already uses after a broadcast gap — so neither the missing deletes nor a table the bundle carries EMPTY leaves a subscriber holding rows that are gone. Wired where the route is mounted, which is the one place both boot paths share.
+
+  On postgres the emptying is a `DELETE` per table, not a `TRUNCATE`, and the difference is not performance: postgres refuses `TRUNCATE` on a table with ANY incoming foreign key, rows or not, while the mysql family refuses a DELETE only when rows actually reference the doomed ones. A `TRUNCATE` version made an EMPTY table outside the bundle block a replace on postgres and not on mariadb — one import, refused on one engine and accepted on the other, over a table holding nothing.
+
+  `--target api` also no longer times out on a full bundle. Both api-target calls went through `fetch`, whose undici default gives up after 300 s — a bound on the caller's database size, on a call whose response arrives only when the import does. They wait as long as the instance needs now, stream the body instead of buffering the whole bundle, and take `--timeout <seconds>` when a deadline is wanted. If one is hit, the message says the instance is probably still importing — and gives different advice for `replace` than for the idempotent modes, since re-running the first while it is mid-flight would empty the target under it.
+
+  Two more, found by measuring rather than by reading:
+
+  - **`--atomic` on postgres could not import a bundle that needed the deferred-FK repair at all.** A failed statement aborts the transaction there, and that repair depends on a row whose parent has not loaded yet failing, being held, and being retried — so the first such row poisoned every write after it. Every tolerated write now runs inside a savepoint. Per-row savepoints measured 2.40x the time of none on 5 000 rows, so they are amortised: one savepoint per batch of 200, and a batch that fails rolls back whole and replays row by row. The mysql family and sqlite leave a transaction usable after a failed statement and pay nothing for any of this. - **The replace pre-flight asked the caller's snapshot.** Over `--target api` that is the app's DECLARED schema, which cannot show a table the app stopped declaring but the database still has — and rows in a table nobody declares are exactly the rows nobody is watching. `DataStore.incomingForeignKeys()` reads the live catalog per dialect; the snapshot remains the fallback for stores without one.
+
+  A bundle bigger than one chunk is now uploaded as a series of short requests, so a proxy body cap or an ingress read timeout has nothing large to choke on, and the switch is automatic — the packer's stream is buffered one chunk ahead, so a small bundle is sent exactly as before and nobody has to know in advance which table is the big one. The import still runs ONCE, at the end, over the whole bundle. Resume is byte-exact (`packBundle` is deterministic over a directory, which this package now asserts), guarded by a bundle key so a different bundle under the same upload id is refused rather than spliced into the partial one, and by a contiguity check so a mis-ordered append cannot produce an archive that only fails later during decode. `--chunk-size <mb>` overrides the 16 MiB default.
+
+---
+
 ## [0.43.1] — 2026-08-18
 
 ### Fixed

package/dist/index.d.ts

@@ -602,12 +602,39 @@ export declare interface CdcConfig {
      *  proactive anti-stall. Default 10s. See
      *  `BinlogCdcOptions.keepAliveInitialDelayMs`. */
     readonly keepAliveInitialDelayMs?: number;
+    /**
+     * The caller promises to call `DataStore.refreshChangeCaptureExclusions()`
+     * once its schema work is done.
+     *
+     * Set by the boot path that migrates in the SAME process as the reader
+     * (`voltro dev`). It buys one thing: a table the store cannot decode on the
+     * pre-migration shape is reported as PROVISIONAL rather than as a verdict, so
+     * a boot whose own migration removes the cause does not leave a fully-argued
+     * error in the log about a condition that was gone before anyone read it.
+     * `voltro serve` builds its store after all schema work and leaves this
+     * unset, because there the reading at attach IS the final one.
+     *
+     * The promise is not taken on trust: the store escalates the provisional note
+     * to the definitive error on its own if no refresh arrives. A flag that turns
+     * a loud line quiet has to fail in the loud direction.
+     */
+    readonly exclusionRefreshFollows?: boolean;
 }
 
 /** Maximum `onChange` listeners a DataStore's change bus accepts before Node
  *  warns. See this file's header for why it is 512 and why it is not `0`. */
 export declare const CHANGE_LISTENER_CEILING = 512;
 
+/** What a `refreshChangeCaptureExclusions()` pass changed. */
+export declare interface ChangeCaptureExclusionRefresh {
+    /** Excluded before and after — the condition is real and still there. */
+    readonly stillExcluded: ReadonlyArray<string>;
+    /** Excluded before, decodable now — the schema work removed the cause. */
+    readonly readmitted: ReadonlyArray<string>;
+    /** Decodable before, excluded now — the schema work INTRODUCED the cause. */
+    readonly newlyExcluded: ReadonlyArray<string>;
+}
+
 /** A row was removed from the table. `new` is null, always. */
 export declare const changeDelete: (table: string, row: Row, meta?: ChangeEventMeta) => ChangeEvent;
 
@@ -2284,6 +2311,117 @@ export declare interface DataStore {
      * omits it (always reachable). Callers guard with `if (store.ping)`.
      */
     ping?(): Promise<void>;
+    /**
+     * Empty every one of `tables` as ONE unit, with referential integrity
+     * suspended for the duration.
+     *
+     * Written for `voltro data import --mode replace`, whose whole job is to put
+     * a target into the state a bundle describes. Two properties, and both are
+     * the point:
+     *
+     *   - **All or nothing.** A run that cannot empty every table must leave the
+     *     target exactly as it found it. The loop this replaces deleted table by
+     *     table and stopped at the first refusal, so an operator's *second*
+     *     attempt destroyed more than the first — the deletes that had already
+     *     succeeded stayed gone and nothing was reloaded.
+     *   - **Order-free.** A parent cannot be emptied while a child still points
+     *     at it, and no ordering fixes the two shapes that occur in every real
+     *     schema: a table that references ITSELF (the framework's own actor
+     *     pattern — `createdBy` on `actors` points at `actors`), and a genuine
+     *     cycle. On engines that check each row as it is deleted, ordering the
+     *     TABLES cannot help, because the conflict is between two ROWS of one of
+     *     them.
+     *
+     * What it deliberately does NOT do is reach outside `tables`. A row in some
+     * OTHER table pointing into this set is not ours to delete, and suspending
+     * integrity would strand it. The caller checks for that first and refuses;
+     * see `runImport`'s replace pre-flight.
+     *
+     * Emits no change events, and the consequence is worth stating rather than
+     * implying. One event per deleted row is not affordable at bundle scale — a
+     * replace of a grown target fanned tens of thousands of them out to every live
+     * subscriber — but silence has an edge. In practice the LOAD that follows
+     * covers it: the reactive layer re-reads a query when any row of its table
+     * changes, so the first inserted row refreshes subscribers and the emptied
+     * rows leave their view with it. The exception is a table the bundle carries
+     * EMPTY: nothing is loaded, nothing is emitted, and a subscriber keeps rows
+     * that are gone until it re-subscribes.
+     *
+     * OPTIONAL on the interface: SQL stores implement it per dialect (the
+     * suspension mechanism is engine-specific and session-scoped, so it has to
+     * live next to the connection). Callers guard with `if (store.emptyTables)`
+     * and fall back to deleting inside one transaction — which keeps the
+     * all-or-nothing half on any store, and is all the in-memory store needs.
+     */
+    emptyTables?(tables: ReadonlyArray<string>): Promise<void>;
+    /**
+     * Foreign keys pointing INTO `tables` from tables that are NOT in the set,
+     * read from the LIVE catalog.
+     *
+     * The question `--mode replace` has to answer before it deletes anything:
+     * emptying a parent whose children live outside the bundle would strand rows
+     * the bundle cannot restore. It was answered from the caller's SNAPSHOT, which
+     * is right for the CLI (it introspects the target) and wrong for the admin
+     * endpoint, which passes the app's DECLARED schema — that cannot show a table
+     * the app has stopped declaring but the database still has, and those are
+     * exactly the rows nobody is watching.
+     *
+     * The catalog is the only authority here, so the store asks it.
+     *
+     * OPTIONAL: SQL stores implement it; the in-memory store has no catalog and no
+     * foreign keys, so it omits it and callers fall back to the snapshot.
+     */
+    incomingForeignKeys?(tables: ReadonlyArray<string>): Promise<ReadonlyArray<{
+        readonly from: string;
+        readonly column: string;
+        readonly to: string;
+    }>>;
+    /**
+     * Run `fn` inside a nested savepoint, so a failure inside it costs only its
+     * own work instead of the whole transaction.
+     *
+     * Exists for one reason: on postgres (and SQL Server for some errors) a failed
+     * statement ABORTS the enclosing transaction — every statement after it
+     * answers `current transaction is aborted` and nothing can be salvaged. The
+     * importer's deferred-foreign-key repair depends on the opposite: a row whose
+     * parent has not loaded yet is EXPECTED to fail, is held, and is retried once
+     * the rest has streamed. That repair is how a bundle whose row order puts a
+     * child before its parent lands at all — and ids sort that way as often as
+     * not. Without a savepoint the two are incompatible, so `--atomic` simply
+     * could not import such a bundle on those engines.
+     *
+     * Only the TRANSACTIONAL views implement it, and only for the engines that
+     * need it: the mysql family and sqlite leave a transaction usable after a
+     * failed statement, so wrapping every row there would buy nothing and cost a
+     * round trip per row. Callers guard with `if (store.savepoint)` and run `fn`
+     * directly otherwise — which is also correct outside a transaction, where
+     * there is nothing to protect.
+     *
+     * Failures propagate: the savepoint is rolled back and the error rethrown, so
+     * the caller's own error handling is unchanged.
+     */
+    savepoint?<A>(fn: () => Promise<A>): Promise<A>;
+    /**
+     * Re-decide which tables this store's change-capture reader can decode, and
+     * apply the answer to the LIVE reader.
+     *
+     * Exists because that decision is a reading of the SCHEMA, and a boot that
+     * migrates in the same process moves the schema after the reader attached.
+     * The reader is built before the migration runs (it has consumers — kv,
+     * broadcast, the analytics mirror — that need a store first), so a table
+     * excluded on the pre-migration shape stayed excluded for the life of the
+     * process even when the migration was the very thing that fixed it.
+     *
+     * Call it once every boot, AFTER all schema work. Cheap (one catalog query)
+     * and a no-op on a store whose verdict did not move — including every store
+     * with nothing to exclude.
+     *
+     * OPTIONAL on the interface: only the mysql-family binlog reader has a
+     * physical-layout exclusion at all. Postgres NOTIFY, mssql Change Tracking
+     * and the in-memory store decode whatever they are given, so they omit it and
+     * callers guard with `if (store.refreshChangeCaptureExclusions)`.
+     */
+    refreshChangeCaptureExclusions?(): Promise<ChangeCaptureExclusionRefresh>;
 }
 
 /** Calendar date without time. */

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@voltro/database",
-  "version": "0.43.1",
+  "version": "0.44.0",
   "description": "Browser-safe schema DSL, query builder, and cross-dialect migration planner for Voltro — one schema, every SQL backend.",
   "keywords": [
     "voltro",
@@ -44,7 +44,7 @@
   },
   "dependencies": {
     "@effect/sql": "^0.52.0",
-    "@voltro/logger": "0.43.1",
+    "@voltro/logger": "0.44.0",
     "typeid-js": "^1.2.0",
     "ulidx": "^2.4.1"
   },