@voltro/data-transfer 0.43.1 → 0.43.2

package/CHANGELOG.md

@@ -39,6 +39,37 @@ _Changes staged for the next release accumulate here (rolled up from
 
 ---
 
+## [0.43.2] — 2026-08-18
+
+### Fixed
+
+- **@voltro/sql-mysql, @voltro/database, @voltro/cli** — The mysql-family binlog reader no longer keeps a table excluded after the very migration that fixed it.
+
+  A UNIQUE on an unbounded text column is a MariaDB hash long-unique, whose hidden `DB_ROW_HASH_n` column the reader can never account for — so the table is held out of binlog capture and the exclusion is reported. `voltro dev` builds its store BEFORE it migrates (kv, cross-replica broadcast and the analytics mirror all need one), so on a boot whose own auto-migration bounds the column, that finding was drawn from a schema that stopped existing about a second later. The exclusion then outlived its cause for the life of the process, and the message — correct when written, and typically the only error line in the boot log — went on describing the pre-migration database.
+
+  Three changes:
+
+  - `DataStore.refreshChangeCaptureExclusions()` (optional; implemented by the mysql-family store) re-runs the probe and re-points the LIVE reader, in both directions — schema work that CREATES the condition now excludes the table immediately instead of after three failed writes. `voltro dev` calls it once, after all its schema work. `voltro serve` needs no equivalent: it builds its store after every schema step and never applies DDL itself. - The definitive message now states its own durability — that it is the schema as read at reader attach, and that applying the remedy does not by itself lift the exclusion. - On a boot that will re-check, the finding is reported as a provisional note rather than as a verdict, so a boot that fixes the condition leaves no error line about it. The note escalates to the full verdict on its own if the re-check never runs. - Two reader fixes the re-check depended on: applying a new exclusion set now WAITS for a reconnect already in flight (it is what applies the set, so resolving before it landed meant the caller's next write hit the old filter), and the reconnect loop no longer keeps resuming from an offset it has just jumped away from — that turned one purged offset into a reconnect every watchdog interval, forever, delivering nothing.
+- **@voltro/data-transfer, @voltro/database, @voltro/sql-mysql, @voltro/sql-postgres, @voltro/sql-sqlite, @voltro/sql-mssql, @voltro/cli** — `voltro data import --mode replace` no longer leaves a target in neither state, and now works against schemas that have foreign keys.
+
+  The delete step ran table by table and stopped at the first refusal, so a run that could not finish left dozens of tables emptied and nothing loaded — and a second attempt destroyed more than the first, because it got further before hitting the same wall. The wall itself was not exotic: MySQL, MariaDB and SQL Server check a foreign key as each ROW is deleted, so a table that references ITSELF cannot be emptied by any ordering of the tables. `createdBy → actors` on the `actors` table is exactly that shape, and it is what an audit mixin on an actor table produces.
+
+  - `DataStore.emptyTables()` (per dialect) empties the whole set as one unit, in one transaction, with referential integrity suspended for the duration — `FOREIGN_KEY_CHECKS` on the mysql family, a multi-table `TRUNCATE` on postgres, `defer_foreign_keys` on sqlite, per-table `NOCHECK`/`WITH CHECK CHECK` on mssql. All-or-nothing on every engine, including under `--atomic`, where it runs on the transaction the import already holds. - A **pre-flight refusal**: if a table OUTSIDE the bundle holds rows referencing one inside it, the import refuses before deleting anything and names the tables, the columns and the row counts. Those rows cannot be restored from the bundle, so forcing it is not an option. An EMPTY outside table blocks nothing. - Table-level failures carry the driver's own reason and code, the way row-level failures already did. `truncate <table> failed: Failed to execute statement` fits every plausible cause equally; the classification that produced `foreign key <name>: … [ER_NO_REFERENCED_ROW_2/1452]` one level down now applies one level up. The word "truncate" is gone from the message too — the step issues DELETE, and naming a statement it does not run sends whoever reads it to reproduce the wrong thing. - A typed refusal reaching the `--target api` transport keeps its text: the admin import endpoint answers `409` with the reason instead of flattening it to `import failed`, on the one transport where the operator has no other way to see it.
+
+  The bulk empty emits no change events, where the per-table loop emitted one per row. An import through `--target api` now asks every live subscription to re-read once it lands — the coarse refresh the framework already uses after a broadcast gap — so neither the missing deletes nor a table the bundle carries EMPTY leaves a subscriber holding rows that are gone. Wired where the route is mounted, which is the one place both boot paths share.
+
+  On postgres the emptying is a `DELETE` per table, not a `TRUNCATE`, and the difference is not performance: postgres refuses `TRUNCATE` on a table with ANY incoming foreign key, rows or not, while the mysql family refuses a DELETE only when rows actually reference the doomed ones. A `TRUNCATE` version made an EMPTY table outside the bundle block a replace on postgres and not on mariadb — one import, refused on one engine and accepted on the other, over a table holding nothing.
+
+  `--target api` also no longer times out on a full bundle. Both api-target calls went through `fetch`, whose undici default gives up after 300 s — a bound on the caller's database size, on a call whose response arrives only when the import does. They wait as long as the instance needs now, stream the body instead of buffering the whole bundle, and take `--timeout <seconds>` when a deadline is wanted. If one is hit, the message says the instance is probably still importing — and gives different advice for `replace` than for the idempotent modes, since re-running the first while it is mid-flight would empty the target under it.
+
+  Two more, found by measuring rather than by reading:
+
+  - **`--atomic` on postgres could not import a bundle that needed the deferred-FK repair at all.** A failed statement aborts the transaction there, and that repair depends on a row whose parent has not loaded yet failing, being held, and being retried — so the first such row poisoned every write after it. Every tolerated write now runs inside a savepoint. Per-row savepoints measured 2.40x the time of none on 5 000 rows, so they are amortised: one savepoint per batch of 200, and a batch that fails rolls back whole and replays row by row. The mysql family and sqlite leave a transaction usable after a failed statement and pay nothing for any of this. - **The replace pre-flight asked the caller's snapshot.** Over `--target api` that is the app's DECLARED schema, which cannot show a table the app stopped declaring but the database still has — and rows in a table nobody declares are exactly the rows nobody is watching. `DataStore.incomingForeignKeys()` reads the live catalog per dialect; the snapshot remains the fallback for stores without one.
+
+  A bundle bigger than one chunk is now uploaded as a series of short requests, so a proxy body cap or an ingress read timeout has nothing large to choke on, and the switch is automatic — the packer's stream is buffered one chunk ahead, so a small bundle is sent exactly as before and nobody has to know in advance which table is the big one. The import still runs ONCE, at the end, over the whole bundle. Resume is byte-exact (`packBundle` is deterministic over a directory, which this package now asserts), guarded by a bundle key so a different bundle under the same upload id is refused rather than spliced into the partial one, and by a contiguity check so a mis-ordered append cannot produce an archive that only fails later during decode. `--chunk-size <mb>` overrides the 16 MiB default.
+
+---
+
 ## [0.43.1] — 2026-08-18
 
 ### Fixed

package/dist/index.d.ts

@@ -639,6 +639,13 @@ export declare interface ImportOptions {
      * run means a clean import.
      */
     readonly dryRun?: boolean;
+    /**
+     * Rows per savepoint when one is needed (inside a transaction, on an engine
+     * that aborts it on a failed statement). Larger = fewer savepoints on the
+     * happy path; smaller = less to replay when a batch contains a row that
+     * cannot land yet. Default {@link SAVEPOINT_BATCH_SIZE}.
+     */
+    readonly savepointBatchSize?: number;
 }
 
 /** An integrity check failed — a table's content checksum or row count did not
@@ -912,6 +919,38 @@ export declare type OnConflict = 'skip' | 'fail';
 /** The caller's progress sink. Plain sync fn — never fails the job. */
 export declare type OnProgress = (event: ProgressEvent) => void;
 
+/** One foreign key that would survive the replace and block it. */
+export declare interface OutsideReference {
+    /** The table holding the rows — outside the bundle. */
+    readonly from: string;
+    readonly column: string;
+    /** The bundle table it points at. */
+    readonly to: string;
+}
+
+/**
+ * The refusal, once the caller knows which of those tables actually hold rows.
+ *
+ * Names the tables, the columns and a way forward, because "replace is not
+ * possible here" without either is a dead end — and the way forward is real:
+ * either the bundle should have carried those tables (re-export with a scope
+ * that includes them) or those rows are genuinely meant to survive, in which
+ * case `replace` is the wrong mode for this target.
+ */
+export declare const outsideReferenceRefusal: (blocking: ReadonlyArray<OutsideReference & {
+    readonly rowCount: number;
+}>) => string;
+
+/**
+ * Every foreign key from a table NOT in `bundleTables` into one that is.
+ *
+ * Pure, and deliberately says nothing about rows: a referencing table that is
+ * EMPTY blocks nothing, and refusing on it would fail a great many perfectly
+ * good imports (a schema carries plenty of tables no environment has used yet).
+ * The caller counts the rows and refuses only on the ones that hold any.
+ */
+export declare const outsideReferencesInto: (snapshot: SchemaSnapshot, bundleTables: ReadonlyArray<string>) => ReadonlyArray<OutsideReference>;
+
 /**
  * Pack a bundle into a framed byte stream. `dir` supplies the small files
  * (tables / manifest / README); `blobs`, when given, streams every content-blob
@@ -1082,6 +1121,14 @@ export declare const runImport: (opts: ImportOptions) => Effect.Effect<Manifest,
  *  copy the file. Maps a missing binary / non-zero exit to {@link NativeToolError}. */
 export declare const runNativeStep: (step: NativeStep) => Effect.Effect<void, NativeToolError>;
 
+/**
+ * Default rows per savepoint. 200 puts the happy-path overhead at one percent
+ * of the statements (two per two hundred) while keeping a failing batch's
+ * replay bounded — and deferred-FK failures cluster, so a batch that contains
+ * one usually contains several.
+ */
+export declare const SAVEPOINT_BATCH_SIZE = 200;
+
 /**
  * Scan the KEPT columns of some sample rows for values that LOOK like PII — a
  * cheap tripwire for a misclassified column (`.safe()` on something that isn't,