@voltro/data-transfer 0.43.2 → 0.44.1

package/CHANGELOG.md

@@ -39,6 +39,56 @@ _Changes staged for the next release accumulate here (rolled up from
 
 ---
 
+## [0.44.1] — 2026-08-19
+
+### Fixed
+
+- **@voltro/database, @voltro/data-transfer, @voltro/cli** — A refused import row reported `(FiberFailure) SqlError: Failed to execute statement` — our runtime's rendering of a rejection, marker and stack frames and all — instead of the constraint that fired. It names neither a rule, nor a code, nor even which layer refused, and every row refused for the same cause carries it identically.
+
+  Two defects, and fixing either alone still leaves a reader stuck.
+
+  **`Cause.squash` elects a branch, and first is a position, not a ranking.** A `Cause` is a tree, and a transaction routinely produces one with more than one leaf: the statement that failed, and whatever the rollback or a finalizer did on the way out. When the first leaf was the bare wrapper, the driver error sitting in the sibling branch was never looked at. Measured: `sequential(bareSqlError, sqlErrorWithDriver)` classified as nothing while the same two branches in the opposite order classified as `unique constraint PRIMARY … [ER_DUP_ENTRY/1062]`. `extractDbCause` flattens failures AND defects now, in Cause order, expanding a nested `FiberFailure` leaf, and elects the branch that names a driver — the others' chains are appended rather than dropped.
+
+  **A row's reason may never read like a stack trace.** The runtime rendering is stripped before any tier looks at the text, so the failure mode cannot return invisibly. And when no driver detail is reachable at all, the reason now names the CHAIN of wrappers the failure passed through — the difference between "the database refused this row" and "the connection died mid-import" — while the run logs the full rendering of the first few such failures. Never returned over the wire: it carries frames, and on some engines a driver's sentence carries row data.
+
+  `RowsRefusedError` also gained `byTable`: **complete** per-table counts. `rows` is capped at 20, so per-table counts tallied off the printed list sum to the cap rather than to the failure — and nothing else in the payload offered any.
+
+---
+
+## [0.44.0] — 2026-08-19
+
+### ⚠ BREAKING
+
+- **@voltro/data-transfer, @voltro/cli** — `runImport` returns an `ImportOutcome` instead of the bundle's `Manifest`, and `--mode replace` runs as ONE transaction by default.
+
+  **Why the return type changed.** The summary line counted the rows the BUNDLE carries, not the rows the run wrote. Those differ most exactly where it matters: a bundle directory carries its own resume ledger, so a copied directory imports nothing — correctly, with a warning naming the file to delete — and the run then printed `import complete … rows: 242950` over a target it had not touched. The warning was one line above, which is one line too far for anyone piping the output through `tail -1`. The outcome carries `rowsWritten`, `rowsSkipped` and `fullyResumed`, the CLI reports written-vs-carried, and a fully-skipped run says so on its LAST line.
+
+  Migration: `runImport(...)` now resolves to `{ manifest, tablesWritten, rowsWritten, tablesSkipped, rowsSkipped, fullyResumed }`. Read `.manifest` where you read the manifest before.
+
+  **Why replace is atomic now.** The all-or-nothing guarantee was written for the emptying step, and read — reasonably — as covering the run. A replace that died partway through the LOAD left the target emptied of its old rows and holding part of the new ones, measured on a live instance over sixteen minutes. There is no useful state for a replace to stop in, so it is a default rather than a flag you have to know about. It also closes the window that produced that failure: with the tables emptied and the load uncommitted, a concurrent writer in the application waits instead of inserting a row the bundle is about to insert too.
+
+  `--no-atomic` (CLI) / `atomic: false` (API) opts out. The trade is stated where it bites: every write to those tables waits for the load, and on postgres the bulk `COPY` loader cannot join a transaction it does not own — an atomic run now says that once rather than being quietly slower.
+
+  Two smaller things from the same report: `--mode replace` against a running instance warns about the empty-target window when it is NOT atomic, and the `--target api` upload reports progress per chunk plus a line explaining that the final request stays open for the whole import — sixteen minutes of silence is indistinguishable from a hang, and one operator killed a run that had finished.
+
+  **`voltro update` carries you across this** — codemod `0.44.0/01_import-outcome`.
+
+### Fixed
+
+- **@voltro/sql-mysql, @voltro/sql-postgres, @voltro/sql-sqlite, @voltro/sql-mssql, @voltro/data-transfer** — A write and its write-recorders now succeed or fail together on every SQL dialect, and the data importer's retry is idempotent.
+
+  A recorder (a versioning trail, an audit log) runs on the caller's connection and is ALLOWED to fail — a recorder that throws must take the write down with it, that is its contract. Outside a caller transaction the two were not one unit: the row's statement committed on its own, and the recorder's INSERT ran afterwards as a second autocommit statement. So a recorder that threw left a COMMITTED row behind a write that reported failure. Measured directly: `insert` throws, the row is in the table, and a second attempt at the same row is `ER_DUP_ENTRY` on PRIMARY.
+
+  Anything that retries a failed write then meets its own row. The data importer retries by design — it holds a row whose write failed and tries again once the remaining tables have streamed — so a `--mode replace` that had just emptied a table failed on a duplicate key IN that table. From outside, that is the impossible-looking thing: an import that emptied a table and then failed because something was already in it. Reproduced verbatim, including the table name and `[ER_DUP_ENTRY/1062]`, by putting the old retry back.
+
+  Both ends are closed, and they are independent on purpose:
+
+  - **The cause.** A table that HAS recorders writes inside a transaction now, so the row and the trail commit together or not at all — on mysql, mariadb, postgres, sqlite and mssql, verified by one suite that asks all five the same question. A table with no recorders — the default — takes the direct path unchanged; `recordsTable` is a Map-size check first, so it costs one comparison. - **The defence.** The importer's retry upserts instead of inserting in `replace` mode. That covers every OTHER way a write can land while reporting failure: a driver timeout on a write the server applied, a connection lost after the commit, a concurrent writer inserting the same key. It is sound precisely because the table was emptied by this same run — there is nothing in it that is not ours.
+
+  The test that pinned the divergence went red when the fix landed, exactly as its own note said it would, and is inverted with that note kept.
+
+---
+
 ## [0.43.2] — 2026-08-18
 
 ### Fixed

package/dist/index.d.ts

@@ -570,6 +570,10 @@ export declare interface ImportOptions {
      * `replace`. Trade-offs: one long-held write transaction (writes to those
      * tables block for the load; readers are unaffected on MVCC dialects), and
      * resume is not per-table (a crash rolls the whole thing back → re-run).
+     *
+     * DEFAULTS TO TRUE FOR `replace`, and to false otherwise. Omit it to get that
+     * rule; pass `false` to opt out deliberately. See where it is resolved in
+     * `runImport` for why replace cannot sensibly run without it.
      */
     readonly atomic?: boolean;
     /** The streaming import path (`unpackBundle` with a storage {@link AssetSink})
@@ -648,6 +652,33 @@ export declare interface ImportOptions {
     readonly savepointBatchSize?: number;
 }
 
+/**
+ * What a run actually did — as opposed to what the bundle contains.
+ *
+ * The two are not the same, and reporting the bundle's numbers as the run's is a
+ * lie of the same family this package keeps finding: a re-run of a bundle
+ * directory whose ledger says every table is applied writes NOTHING and used to
+ * be summarised as `import complete … rows: 242950`. The warning that explained
+ * it was one line further up, which is one line too far for anyone who pipes the
+ * output through `tail`.
+ *
+ * `rowsWritten` counts rows this run streamed into the target. `rowsSkipped`
+ * counts rows in tables an earlier run of THIS bundle directory had already
+ * applied. A dry run reports zero written and nothing skipped: it examined
+ * everything and wrote nothing.
+ */
+export declare interface ImportOutcome {
+    /** The bundle's own contract — what it CARRIES. */
+    readonly manifest: Manifest;
+    readonly tablesWritten: number;
+    readonly rowsWritten: number;
+    /** Tables an earlier run of this bundle directory already applied. */
+    readonly tablesSkipped: number;
+    readonly rowsSkipped: number;
+    /** True when nothing was written because everything was already applied. */
+    readonly fullyResumed: boolean;
+}
+
 /** An integrity check failed — a table's content checksum or row count did not
  *  match the manifest, or an asset's bytes did not hash to its recorded id. */
 export declare class IntegrityError extends IntegrityError_base {
@@ -1107,6 +1138,25 @@ declare const RowsRefusedError_base: Schema.TaggedErrorClass<RowsRefusedError, "
     primaryCount: typeof Schema.Number;
     /** Total number of refused rows (may exceed `rows.length`). */
     totalCount: typeof Schema.Number;
+    /**
+     * Every table that refused a row, with COMPLETE counts — not counts of the
+     * capped `rows` list.
+     *
+     * `rows` is capped at 20, and the shape of a report invites tallying what it
+     * shows — the per-table counts read off that list always sum to the cap, and
+     * nothing else in the payload offered any. Which tables refused, and how
+     * badly, is the first question of a triage and the one the cap silently
+     * answers wrong, so it is its own field: computed over all of them, ordered
+     * worst-first and capped separately (a bundle can carry thousands of tables;
+     * a hundred failing ones is already a different conversation).
+     */
+    byTable: Schema.Array$<Schema.Struct<{
+        table: typeof Schema.String;
+        /** Rows of this table that were refused. */
+        refused: typeof Schema.Number;
+        /** …of which are not a consequence of another refused row. */
+        primary: typeof Schema.Number;
+    }>>;
 }>;
 
 /** Row stream → NDJSON byte stream (each row → typed-encoded JSON + `\n`). The
@@ -1115,7 +1165,7 @@ export declare const rowsToNdjson: <E, R>(rows: Stream.Stream<Row, E, R>, column
 
 export declare const runExport: (opts: ExportOptions) => Effect.Effect<Manifest, ExportError>;
 
-export declare const runImport: (opts: ImportOptions) => Effect.Effect<Manifest, ImportError>;
+export declare const runImport: (opts: ImportOptions) => Effect.Effect<ImportOutcome, ImportError>;
 
 /** Execute a {@link NativeStep} — spawn the tool (wiring stdin/stdout files) or
  *  copy the file. Maps a missing binary / non-zero exit to {@link NativeToolError}. */