@voltro/data-transfer 0.43.2 → 0.44.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -39,6 +39,40 @@ _Changes staged for the next release accumulate here (rolled up from
39
39
 
40
40
  ---
41
41
 
42
+ ## [0.44.0] — 2026-08-19
43
+
44
+ ### ⚠ BREAKING
45
+
46
+ - **@voltro/data-transfer, @voltro/cli** — `runImport` returns an `ImportOutcome` instead of the bundle's `Manifest`, and `--mode replace` runs as ONE transaction by default.
47
+
48
+ **Why the return type changed.** The summary line counted the rows the BUNDLE carries, not the rows the run wrote. Those differ most exactly where it matters: a bundle directory carries its own resume ledger, so a copied directory imports nothing — correctly, with a warning naming the file to delete — and the run then printed `import complete … rows: 242950` over a target it had not touched. The warning was one line above, which is one line too far for anyone piping the output through `tail -1`. The outcome carries `rowsWritten`, `rowsSkipped` and `fullyResumed`, the CLI reports written-vs-carried, and a fully-skipped run says so on its LAST line.
49
+
50
+ Migration: `runImport(...)` now resolves to `{ manifest, tablesWritten, rowsWritten, tablesSkipped, rowsSkipped, fullyResumed }`. Read `.manifest` where you read the manifest before.
51
+
52
+ **Why replace is atomic now.** The all-or-nothing guarantee was written for the emptying step, and read — reasonably — as covering the run. A replace that died partway through the LOAD left the target emptied of its old rows and holding part of the new ones, measured on a live instance over sixteen minutes. There is no useful state for a replace to stop in, so it is a default rather than a flag you have to know about. It also closes the window that produced that failure: with the tables emptied and the load uncommitted, a concurrent writer in the application waits instead of inserting a row the bundle is about to insert too.
53
+
54
+ `--no-atomic` (CLI) / `atomic: false` (API) opts out. The trade is stated where it bites: every write to those tables waits for the load, and on postgres the bulk `COPY` loader cannot join a transaction it does not own — an atomic run now says that once rather than being quietly slower.
55
+
56
+ Two smaller things from the same report: `--mode replace` against a running instance warns about the empty-target window when it is NOT atomic, and the `--target api` upload reports progress per chunk plus a line explaining that the final request stays open for the whole import — sixteen minutes of silence is indistinguishable from a hang, and one operator killed a run that had finished.
57
+
58
+ **`voltro update` carries you across this** — codemod `0.44.0/01_import-outcome`.
59
+
60
+ ### Fixed
61
+
62
+ - **@voltro/sql-mysql, @voltro/sql-postgres, @voltro/sql-sqlite, @voltro/sql-mssql, @voltro/data-transfer** — A write and its write-recorders now succeed or fail together on every SQL dialect, and the data importer's retry is idempotent.
63
+
64
+ A recorder (a versioning trail, an audit log) runs on the caller's connection and is ALLOWED to fail — a recorder that throws must take the write down with it, that is its contract. Outside a caller transaction the two were not one unit: the row's statement committed on its own, and the recorder's INSERT ran afterwards as a second autocommit statement. So a recorder that threw left a COMMITTED row behind a write that reported failure. Measured directly: `insert` throws, the row is in the table, and a second attempt at the same row is `ER_DUP_ENTRY` on PRIMARY.
65
+
66
+ Anything that retries a failed write then meets its own row. The data importer retries by design — it holds a row whose write failed and tries again once the remaining tables have streamed — so a `--mode replace` that had just emptied a table failed on a duplicate key IN that table. From outside, that is the impossible-looking thing: an import that emptied a table and then failed because something was already in it. Reproduced verbatim, including the table name and `[ER_DUP_ENTRY/1062]`, by putting the old retry back.
67
+
68
+ Both ends are closed, and they are independent on purpose:
69
+
70
+ - **The cause.** A table that HAS recorders writes inside a transaction now, so the row and the trail commit together or not at all — on mysql, mariadb, postgres, sqlite and mssql, verified by one suite that asks all five the same question. A table with no recorders — the default — takes the direct path unchanged; `recordsTable` is a Map-size check first, so it costs one comparison. - **The defence.** The importer's retry upserts instead of inserting in `replace` mode. That covers every OTHER way a write can land while reporting failure: a driver timeout on a write the server applied, a connection lost after the commit, a concurrent writer inserting the same key. It is sound precisely because the table was emptied by this same run — there is nothing in it that is not ours.
71
+
72
+ The test that pinned the divergence went red when the fix landed, exactly as its own note said it would, and is inverted with that note kept.
73
+
74
+ ---
75
+
42
76
  ## [0.43.2] — 2026-08-18
43
77
 
44
78
  ### Fixed
package/dist/index.d.ts CHANGED
@@ -570,6 +570,10 @@ export declare interface ImportOptions {
570
570
  * `replace`. Trade-offs: one long-held write transaction (writes to those
571
571
  * tables block for the load; readers are unaffected on MVCC dialects), and
572
572
  * resume is not per-table (a crash rolls the whole thing back → re-run).
573
+ *
574
+ * DEFAULTS TO TRUE FOR `replace`, and to false otherwise. Omit it to get that
575
+ * rule; pass `false` to opt out deliberately. See where it is resolved in
576
+ * `runImport` for why replace cannot sensibly run without it.
573
577
  */
574
578
  readonly atomic?: boolean;
575
579
  /** The streaming import path (`unpackBundle` with a storage {@link AssetSink})
@@ -648,6 +652,33 @@ export declare interface ImportOptions {
648
652
  readonly savepointBatchSize?: number;
649
653
  }
650
654
 
655
+ /**
656
+ * What a run actually did — as opposed to what the bundle contains.
657
+ *
658
+ * The two are not the same, and reporting the bundle's numbers as the run's is a
659
+ * lie of the same family this package keeps finding: a re-run of a bundle
660
+ * directory whose ledger says every table is applied writes NOTHING and used to
661
+ * be summarised as `import complete … rows: 242950`. The warning that explained
662
+ * it was one line further up, which is one line too far for anyone who pipes the
663
+ * output through `tail`.
664
+ *
665
+ * `rowsWritten` counts rows this run streamed into the target. `rowsSkipped`
666
+ * counts rows in tables an earlier run of THIS bundle directory had already
667
+ * applied. A dry run reports zero written and nothing skipped: it examined
668
+ * everything and wrote nothing.
669
+ */
670
+ export declare interface ImportOutcome {
671
+ /** The bundle's own contract — what it CARRIES. */
672
+ readonly manifest: Manifest;
673
+ readonly tablesWritten: number;
674
+ readonly rowsWritten: number;
675
+ /** Tables an earlier run of this bundle directory already applied. */
676
+ readonly tablesSkipped: number;
677
+ readonly rowsSkipped: number;
678
+ /** True when nothing was written because everything was already applied. */
679
+ readonly fullyResumed: boolean;
680
+ }
681
+
651
682
  /** An integrity check failed — a table's content checksum or row count did not
652
683
  * match the manifest, or an asset's bytes did not hash to its recorded id. */
653
684
  export declare class IntegrityError extends IntegrityError_base {
@@ -1115,7 +1146,7 @@ export declare const rowsToNdjson: <E, R>(rows: Stream.Stream<Row, E, R>, column
1115
1146
 
1116
1147
  export declare const runExport: (opts: ExportOptions) => Effect.Effect<Manifest, ExportError>;
1117
1148
 
1118
- export declare const runImport: (opts: ImportOptions) => Effect.Effect<Manifest, ImportError>;
1149
+ export declare const runImport: (opts: ImportOptions) => Effect.Effect<ImportOutcome, ImportError>;
1119
1150
 
1120
1151
  /** Execute a {@link NativeStep} — spawn the tool (wiring stdin/stdout files) or
1121
1152
  * copy the file. Maps a missing binary / non-zero exit to {@link NativeToolError}. */
package/dist/index.js CHANGED
@@ -2,16 +2,16 @@ import { Cause as e, Data as t, Effect as n, Exit as r, Metric as i, MetricBound
2
2
  import { classifyConstraintViolation as l, eq as u, extractDbCause as d, inSet as f, isNotNull as p, or as m, streamTable as h } from "@voltro/database";
3
3
  import { fingerprintSchema as g } from "@voltro/database/sql";
4
4
  import { createCipheriv as _, createDecipheriv as v, createHash as y, createHmac as b, randomBytes as x, scrypt as S } from "node:crypto";
5
- import { STORAGE_REFS_TABLE as C, fromNodeReadable as w, getObjectStream as T, putObjectStream as ee, toNodeReadable as te } from "@voltro/plugin-storage";
6
- import * as E from "node:zlib";
7
- import { PassThrough as D, Readable as ne } from "node:stream";
8
- import { copyFile as re, createReadStream as O, createWriteStream as ie, existsSync as ae, unlinkSync as oe } from "node:fs";
9
- import { mkdir as se, readdir as ce, stat as le } from "node:fs/promises";
10
- import { basename as ue, delimiter as de, dirname as fe, join as k, resolve as pe, sep as me } from "node:path";
11
- import { promisify as he } from "node:util";
12
- import { spawn as ge } from "node:child_process";
5
+ import { STORAGE_REFS_TABLE as C, fromNodeReadable as w, getObjectStream as T, putObjectStream as E, toNodeReadable as D } from "@voltro/plugin-storage";
6
+ import * as O from "node:zlib";
7
+ import { PassThrough as k, Readable as A } from "node:stream";
8
+ import { copyFile as ee, createReadStream as j, createWriteStream as te, existsSync as ne, unlinkSync as re } from "node:fs";
9
+ import { mkdir as ie, readdir as ae, stat as oe } from "node:fs/promises";
10
+ import { basename as se, delimiter as ce, dirname as le, join as M, resolve as ue, sep as de } from "node:path";
11
+ import { promisify as fe } from "node:util";
12
+ import { spawn as pe } from "node:child_process";
13
13
  //#region src/consistency.ts
14
- var _e = (e) => {
14
+ var me = (e) => {
15
15
  switch (e) {
16
16
  case "postgres": return ["SET TRANSACTION ISOLATION LEVEL REPEATABLE READ"];
17
17
  case "mysql":
@@ -21,7 +21,7 @@ var _e = (e) => {
21
21
  case "turso": return [];
22
22
  default: return [];
23
23
  }
24
- }, ve = {
24
+ }, he = {
25
25
  email: { fake: "email" },
26
26
  fullName: { fake: "fullName" },
27
27
  firstName: { fake: "firstName" },
@@ -36,7 +36,7 @@ var _e = (e) => {
36
36
  date: "dateShift",
37
37
  secret: "null",
38
38
  freeText: "redact"
39
- }, A = (e) => {
39
+ }, N = (e) => {
40
40
  let t = {};
41
41
  for (let n of e.tables) {
42
42
  let e = {};
@@ -48,15 +48,15 @@ var _e = (e) => {
48
48
  t[n.name] = e;
49
49
  }
50
50
  return t;
51
- }, ye = (e, t, n, r) => {
51
+ }, ge = (e, t, n, r) => {
52
52
  let i = r.columns?.[`${e}.${t}`];
53
53
  if (i) return i;
54
54
  if (n?.sensitive) {
55
55
  let e = n.sensitive.class;
56
- return r.classes?.[e] ?? ve[e] ?? "redact";
56
+ return r.classes?.[e] ?? he[e] ?? "redact";
57
57
  }
58
58
  return n?.safe ? r.onSafe ?? "keep" : n?.columnType === "id" || n?.columnType === "reference" ? "keep" : r.onUnclassified ?? "error";
59
- }, j = (e, t) => b("sha256", e).update(t == null ? "∅" : String(t)).digest(), M = (e, t) => e.readUInt32BE(t * 4 % (e.length - 4)), N = [
59
+ }, P = (e, t) => b("sha256", e).update(t == null ? "∅" : String(t)).digest(), F = (e, t) => e.readUInt32BE(t * 4 % (e.length - 4)), I = [
60
60
  "Ada",
61
61
  "Alan",
62
62
  "Grace",
@@ -69,7 +69,7 @@ var _e = (e) => {
69
69
  "Guido",
70
70
  "Margaret",
71
71
  "Bjarne"
72
- ], P = [
72
+ ], _e = [
73
73
  "Lovelace",
74
74
  "Turing",
75
75
  "Hopper",
@@ -82,7 +82,7 @@ var _e = (e) => {
82
82
  "Rossum",
83
83
  "Hamilton",
84
84
  "Stroustrup"
85
- ], be = [
85
+ ], ve = [
86
86
  "Maple",
87
87
  "Oak",
88
88
  "Cedar",
@@ -91,7 +91,7 @@ var _e = (e) => {
91
91
  "Birch",
92
92
  "Willow",
93
93
  "Ash"
94
- ], xe = [
94
+ ], ye = [
95
95
  "Initech",
96
96
  "Hooli",
97
97
  "Umbrella",
@@ -100,23 +100,23 @@ var _e = (e) => {
100
100
  "Soylent",
101
101
  "Stark",
102
102
  "Wayne"
103
- ], F = (e, t, n) => e[M(t, n) % e.length], Se = (e, t, n) => {
104
- let r = j(n, `${e}:${String(t)}`), i = M(r, 1) % 1e4;
103
+ ], L = (e, t, n) => e[F(t, n) % e.length], be = (e, t, n) => {
104
+ let r = P(n, `${e}:${String(t)}`), i = F(r, 1) % 1e4;
105
105
  switch (e) {
106
- case "email": return `${F(N, r, 0).toLowerCase()}.${F(P, r, 2).toLowerCase()}${i}@example.com`;
107
- case "fullName": return `${F(N, r, 0)} ${F(P, r, 1)}`;
108
- case "firstName": return F(N, r, 0);
109
- case "lastName": return F(P, r, 1);
110
- case "username": return `${F(N, r, 0).toLowerCase()}${i}`;
111
- case "phone": return `+1${String(M(r, 0) % 1e10).padStart(10, "0")}`;
112
- case "address": return `${M(r, 0) % 9899 + 100} ${F(be, r, 1)} St`;
113
- case "company": return F(xe, r, 0);
106
+ case "email": return `${L(I, r, 0).toLowerCase()}.${L(_e, r, 2).toLowerCase()}${i}@example.com`;
107
+ case "fullName": return `${L(I, r, 0)} ${L(_e, r, 1)}`;
108
+ case "firstName": return L(I, r, 0);
109
+ case "lastName": return L(_e, r, 1);
110
+ case "username": return `${L(I, r, 0).toLowerCase()}${i}`;
111
+ case "phone": return `+1${String(F(r, 0) % 1e10).padStart(10, "0")}`;
112
+ case "address": return `${F(r, 0) % 9899 + 100} ${L(ve, r, 1)} St`;
113
+ case "company": return L(ye, r, 0);
114
114
  case "url": return `https://example.com/${r.toString("hex").slice(0, 12)}`;
115
- case "ip": return `${M(r, 0) % 256}.${M(r, 1) % 256}.${M(r, 2) % 256}.${M(r, 3) % 254 + 1}`;
116
- case "creditCard": return Ce(`4${r.toString("hex").replace(/\D/g, "").padEnd(14, "0").slice(0, 14)}`);
115
+ case "ip": return `${F(r, 0) % 256}.${F(r, 1) % 256}.${F(r, 2) % 256}.${F(r, 3) % 254 + 1}`;
116
+ case "creditCard": return xe(`4${r.toString("hex").replace(/\D/g, "").padEnd(14, "0").slice(0, 14)}`);
117
117
  default: return `${e}_${r.toString("hex").slice(0, 8)}`;
118
118
  }
119
- }, Ce = (e) => {
119
+ }, xe = (e) => {
120
120
  let t = e.slice(0, 15).split("").map(Number), n = 0;
121
121
  t.forEach((e, t) => {
122
122
  let r = e;
@@ -124,31 +124,31 @@ var _e = (e) => {
124
124
  });
125
125
  let r = (10 - n % 10) % 10;
126
126
  return e.slice(0, 15) + String(r);
127
- }, we = 864e5, Te = (e, t) => {
127
+ }, Se = 864e5, Ce = (e, t) => {
128
128
  let n = e instanceof Date ? e.getTime() : Date.parse(String(e));
129
129
  if (Number.isNaN(n)) return e;
130
- let r = M(j(t, "date-shift-offset"), 0) % 730 - 365, i = new Date(n + r * we);
130
+ let r = F(P(t, "date-shift-offset"), 0) % 730 - 365, i = new Date(n + r * Se);
131
131
  return e instanceof Date ? i : i.toISOString();
132
- }, Ee = (e, t) => {
132
+ }, we = (e, t) => {
133
133
  if (t.value === null || t.value === void 0) return null;
134
134
  if (e === "error") throw Error(`masking: unresolved 'error' action for ${t.table}.${t.column}`);
135
135
  if (e === "keep") return t.value;
136
136
  if (e === "null") return null;
137
137
  if (e === "redact") return t.columnType === "text" || typeof t.value == "string" ? "[redacted]" : null;
138
- if (e === "hash") return j(t.seed, t.value).toString("hex").slice(0, 16);
139
- if (e === "dateShift") return Te(t.value, t.seed);
140
- if (typeof e == "object" && "fake" in e) return Se(e.fake, t.value, t.seed);
138
+ if (e === "hash") return P(t.seed, t.value).toString("hex").slice(0, 16);
139
+ if (e === "dateShift") return Ce(t.value, t.seed);
140
+ if (typeof e == "object" && "fake" in e) return be(e.fake, t.value, t.seed);
141
141
  if (typeof e == "object" && "custom" in e) return e.custom(t);
142
142
  throw Error(`masking: '${t.table}.${t.column}' resolved to an action this build does not understand (${JSON.stringify(e)}). Refusing to copy the value through. Valid actions: 'keep' | 'null' | 'redact' | 'hash' | 'dateShift' | { fake: '<class>' } | { custom: fn }.`);
143
- }, De = (e) => typeof e == "string" ? e === "keep" || e === "null" || e === "redact" || e === "hash" || e === "dateShift" || e === "error" : typeof e != "object" || !e ? !1 : "fake" in e ? typeof e.fake == "string" : "custom" in e && typeof e.custom == "function", I = (e, t, n) => {
143
+ }, Te = (e) => typeof e == "string" ? e === "keep" || e === "null" || e === "redact" || e === "hash" || e === "dateShift" || e === "error" : typeof e != "object" || !e ? !1 : "fake" in e ? typeof e.fake == "string" : "custom" in e && typeof e.custom == "function", Ee = (e, t, n) => {
144
144
  let r = {}, i = [], a = [];
145
145
  for (let o of e) for (let e of o.columns) {
146
- let s = `${o.name}.${e.name}`, c = ye(o.name, e.name, t[o.name]?.[e.name], n);
146
+ let s = `${o.name}.${e.name}`, c = ge(o.name, e.name, t[o.name]?.[e.name], n);
147
147
  if (c === "error") {
148
148
  i.push(s);
149
149
  continue;
150
150
  }
151
- if (!De(c)) {
151
+ if (!Te(c)) {
152
152
  a.push({
153
153
  column: s,
154
154
  action: c
@@ -162,7 +162,7 @@ var _e = (e) => {
162
162
  unclassified: i,
163
163
  invalid: a
164
164
  };
165
- }, Oe = [
165
+ }, De = [
166
166
  {
167
167
  name: "email",
168
168
  re: /[\w.+-]+@[\w-]+\.[\w.-]+/
@@ -179,7 +179,7 @@ var _e = (e) => {
179
179
  name: "creditCard",
180
180
  re: /\b(?:\d[ -]?){13,16}\b/
181
181
  }
182
- ], ke = (e, t, n) => {
182
+ ], Oe = (e, t, n) => {
183
183
  let r = [], i = /* @__PURE__ */ new Set();
184
184
  for (let [a, o] of Object.entries(n)) {
185
185
  if (!a.startsWith(`${t}.`) || o !== "keep") continue;
@@ -187,7 +187,7 @@ var _e = (e) => {
187
187
  for (let a of e) {
188
188
  let e = a[n];
189
189
  if (typeof e != "string" || e.length === 0) continue;
190
- let o = Oe.find((t) => t.re.test(e));
190
+ let o = De.find((t) => t.re.test(e));
191
191
  if (o && !i.has(n)) {
192
192
  i.add(n), r.push({
193
193
  table: t,
@@ -200,9 +200,9 @@ var _e = (e) => {
200
200
  }
201
201
  }
202
202
  return r;
203
- }, L = (e, t, n, r, i) => {
203
+ }, ke = (e, t, n, r, i) => {
204
204
  let a = {};
205
- for (let [o, s] of Object.entries(e)) a[o] = Ee(n[`${t}.${o}`] ?? "keep", {
205
+ for (let [o, s] of Object.entries(e)) a[o] = we(n[`${t}.${o}`] ?? "keep", {
206
206
  value: s,
207
207
  table: t,
208
208
  column: o,
@@ -342,14 +342,14 @@ var _e = (e) => {
342
342
  list: t,
343
343
  open: (t) => T(e, t)
344
344
  }), He = (e) => ({
345
- write: (t, n, r) => ee(e, t, n, r),
345
+ write: (t, n, r) => E(e, t, n, r),
346
346
  has: (t) => e.head(t).pipe(n.map((e) => e !== null))
347
347
  }), Ue = (e, t) => n.tryPromise({
348
348
  try: async () => {
349
349
  let { createWriteStream: n } = await import("node:fs"), r = await import("node:fs/promises"), { createHash: i, randomUUID: a } = await import("node:crypto"), { pipeline: o } = await import("node:stream/promises"), { Transform: s } = await import("node:stream"), c = await import("node:path"), l = c.join(t, `.tmp-${a()}`), u = i("sha256"), d = 0, f = new s({ transform(e, t, n) {
350
350
  u.update(e), d += e.length, n(null, e);
351
351
  } });
352
- await o(await te(e.stream), f, n(l));
352
+ await o(await D(e.stream), f, n(l));
353
353
  let p = u.digest("hex"), m = c.join(t, p);
354
354
  try {
355
355
  await r.access(m), await r.rm(l, { force: !0 });
@@ -378,7 +378,7 @@ var _e = (e) => {
378
378
  expected: r.sha256,
379
379
  actual: d
380
380
  }));
381
- }), Ge = () => typeof E.createZstdCompress == "function", Ke = (e) => e === "none" ? "none" : e === "gzip" ? "gzip" : Ge() ? "zstd" : "gzip", qe = (e) => e === "zstd" ? ".zst" : e === "gzip" ? ".gz" : "", Je = (e) => e === "zstd" ? E.createZstdCompress() : e === "gzip" ? E.createGzip() : new D(), Ye = (e) => e === "zstd" ? E.createZstdDecompress() : e === "gzip" ? E.createGunzip() : new D(), Xe = 1, Ze = s.Struct({
381
+ }), Ge = () => typeof O.createZstdCompress == "function", Ke = (e) => e === "none" ? "none" : e === "gzip" ? "gzip" : Ge() ? "zstd" : "gzip", qe = (e) => e === "zstd" ? ".zst" : e === "gzip" ? ".gz" : "", Je = (e) => e === "zstd" ? O.createZstdCompress() : e === "gzip" ? O.createGzip() : new k(), Ye = (e) => e === "zstd" ? O.createZstdDecompress() : e === "gzip" ? O.createGunzip() : new k(), Xe = 1, Ze = s.Struct({
382
382
  name: s.String,
383
383
  file: s.String,
384
384
  rowCount: s.Number,
@@ -646,7 +646,7 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
646
646
  chunkSize: t.chunkSize,
647
647
  retry: t.retry,
648
648
  ...f === void 0 ? {} : { where: f }
649
- }), m = Nt(t.snapshot, o), g = m.length === 0 ? p : p.pipe(c.map((e) => Pt(e, m))), _ = t.mask ? g.pipe(c.map((e) => L(e, o, t.mask.actions, t.mask.seed, s))) : g, { rowCount: v, checksum: y } = yield* ft("export", o, a, i, (e) => e.rowCount, It(Ot(_, s), t.bundleFile(d), t.compression));
649
+ }), m = Nt(t.snapshot, o), g = m.length === 0 ? p : p.pipe(c.map((e) => Pt(e, m))), _ = t.mask ? g.pipe(c.map((e) => ke(e, o, t.mask.actions, t.mask.seed, s))) : g, { rowCount: v, checksum: y } = yield* ft("export", o, a, i, (e) => e.rowCount, It(Ot(_, s), t.bundleFile(d), t.compression));
650
650
  t.ledger.tables[o] = {
651
651
  rowCount: v,
652
652
  checksum: y
@@ -674,7 +674,7 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
674
674
  values: []
675
675
  }), Bt = (t, i, a) => n.tryPromise({
676
676
  try: () => t.transactional(async (t) => {
677
- if (t.raw) for (let e of _e(i)) await t.raw(zt(e));
677
+ if (t.raw) for (let e of me(i)) await t.raw(zt(e));
678
678
  let o = await n.runPromiseExit(a(t));
679
679
  if (r.isFailure(o)) throw Error(e.pretty(o.cause));
680
680
  return o.value;
@@ -697,7 +697,7 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
697
697
  ...e.tenantTables ? { tenantTables: e.tenantTables } : {}
698
698
  }), m = At(p.tables, e.snapshot), h = e.consistency ?? "live", _, v;
699
699
  if (e.masking) {
700
- let t = e.classification ?? A(e.snapshot), r = I(m.map((t) => ({
700
+ let t = e.classification ?? N(e.snapshot), r = Ee(m.map((t) => ({
701
701
  name: t,
702
702
  columns: (e.snapshot.tables.find((e) => e.name === t)?.columns ?? []).map((e) => ({
703
703
  name: e.name,
@@ -962,7 +962,7 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
962
962
  blocking: e.map(Jt)
963
963
  }));
964
964
  }
965
- let l = t.verify ?? !0, u = t.mode ?? "upsert", d = t.onConflict ?? "skip", f = t.ledgerPath ?? i.join(t.bundleDir, ".import..ledger.json"), m = yield* mt(f);
965
+ let l = t.verify ?? !0, u = t.mode ?? "upsert", d = t.atomic ?? u === "replace", f = t.onConflict ?? "skip", m = t.ledgerPath ?? i.join(t.bundleDir, ".import..ledger.json"), h = yield* mt(m);
966
966
  if (u === "replace" && s.scope.kind !== "all") return yield* n.fail(new Fe({
967
967
  mode: u,
968
968
  reason: `replace refuses a partial bundle (scope='${s.scope.kind}') — truncating would delete rows not in the bundle. Use --mode upsert, or re-export with full scope.`
@@ -998,31 +998,38 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
998
998
  reason: Ut(i)
999
999
  }));
1000
1000
  }
1001
- if (t.dryRun === !0) return s;
1002
- let h = /* @__PURE__ */ new Map();
1001
+ if (t.dryRun === !0) return {
1002
+ manifest: s,
1003
+ tablesWritten: 0,
1004
+ rowsWritten: 0,
1005
+ tablesSkipped: 0,
1006
+ rowsSkipped: 0,
1007
+ fullyResumed: !1
1008
+ };
1009
+ let g = /* @__PURE__ */ new Map();
1003
1010
  for (let e of t.targetSnapshot?.tables ?? []) {
1004
1011
  let t = e.columns.filter((e) => e.generatedAs !== void 0).map((e) => e.name);
1005
- t.length > 0 && h.set(e.name, t);
1012
+ t.length > 0 && g.set(e.name, t);
1006
1013
  }
1007
- let g = (e, t) => {
1008
- let n = h.get(e);
1014
+ let _ = (e, t) => {
1015
+ let n = g.get(e);
1009
1016
  if (n === void 0) return t;
1010
1017
  let r;
1011
1018
  for (let e of n) e in t && (r ??= { ...t }, delete r[e]);
1012
1019
  return r ?? t;
1013
- }, _ = (e, t, n) => {
1014
- let r = t.name, i = t.primaryKey, a = g(r, n);
1015
- return u === "append" ? d === "fail" ? e.insert(r, a) : e.insertIgnore(r, a, { conflictColumns: [i] }) : u === "replace" ? e.insert(r, a) : e.upsert(r, a, { conflictColumns: [i] });
1016
- }, v = (e, t) => e.savepoint ? e.savepoint(t) : t(), y = (e, t, r = !0) => n.tryPromise({
1020
+ }, v = (e, t, n, r = !1) => {
1021
+ let i = t.name, a = t.primaryKey, o = _(i, n);
1022
+ return u === "append" ? f === "fail" ? e.insert(i, o) : e.insertIgnore(i, o, { conflictColumns: [a] }) : u === "replace" ? r ? e.upsert(i, o, { conflictColumns: [a] }) : e.insert(i, o) : e.upsert(i, o, { conflictColumns: [a] });
1023
+ }, y = (e, t) => e.savepoint ? e.savepoint(t) : t(), b = (e, t, r = !0) => n.tryPromise({
1017
1024
  try: e,
1018
1025
  catch: (e) => Zt(e)
1019
- }).pipe(n.map(() => !0), n.catchAll((e) => n.sync(() => (r && (t.reason = e), !1)))), b = (e) => {
1026
+ }).pipe(n.map(() => !0), n.catchAll((e) => n.sync(() => (r && (t.reason = e), !1)))), x = (e) => {
1020
1027
  let t = Object.entries(e.entry.columnTypes).filter(([t, n]) => n === "reference" && e.row[t] !== null && e.row[t] !== void 0).map(([e]) => e);
1021
1028
  if (t.length !== 0) return {
1022
1029
  ...e.row,
1023
1030
  ...Object.fromEntries(t.map((e) => [e, null]))
1024
1031
  };
1025
- }, x = (e) => {
1032
+ }, S = (e) => {
1026
1033
  let t = new Set(e.map((e) => String(e.row[e.entry.primaryKey]))), n = (e) => Object.entries(e.entry.columnTypes).some(([n, r]) => {
1027
1034
  if (r !== "reference") return !1;
1028
1035
  let i = e.row[n];
@@ -1038,22 +1045,22 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
1038
1045
  primaryCount: r.filter((e) => !e.derived).length,
1039
1046
  totalCount: e.length
1040
1047
  });
1041
- }, S = (e, t) => n.gen(function* () {
1048
+ }, C = (e, t) => n.gen(function* () {
1042
1049
  let r = t, i = [], a = !0;
1043
1050
  for (; r.length > 0 && a;) {
1044
1051
  a = !1;
1045
1052
  let t = [];
1046
- for (let n of r) (yield* y(() => v(e, () => _(e, n.entry, n.row)), n)) ? a = !0 : t.push(n);
1053
+ for (let n of r) (yield* b(() => y(e, () => v(e, n.entry, n.row, !0)), n)) ? a = !0 : t.push(n);
1047
1054
  if (!a && t.length > 0) for (let n = t.length - 1; n >= 0; n--) {
1048
- let r = t[n], o = b(r);
1049
- o !== void 0 && (yield* y(() => v(e, () => _(e, r.entry, o)), r, !1)) && (i.push(r), t.splice(n, 1), a = !0);
1055
+ let r = t[n], o = x(r);
1056
+ o !== void 0 && (yield* b(() => y(e, () => v(e, r.entry, o, !0)), r, !1)) && (i.push(r), t.splice(n, 1), a = !0);
1050
1057
  }
1051
1058
  r = t;
1052
1059
  }
1053
- for (let t of i) (yield* y(() => v(e, () => e.upsert(t.entry.name, t.row, { conflictColumns: [t.entry.primaryKey] })), t)) || (r = [...r, t]);
1054
- if (r.length > 0) return yield* n.fail(x(r));
1055
- }), C = (e, r) => n.gen(function* () {
1056
- if (u === "replace" && !(r && m.truncated)) {
1060
+ for (let t of i) (yield* b(() => y(e, () => e.upsert(t.entry.name, t.row, { conflictColumns: [t.entry.primaryKey] })), t)) || (r = [...r, t]);
1061
+ if (r.length > 0) return yield* n.fail(S(r));
1062
+ }), T = !1, E = 0, D = 0, O = 0, k = 0, A = (e, r) => n.gen(function* () {
1063
+ if (u === "replace" && !(r && h.truncated)) {
1057
1064
  let t = [...s.tables].reverse().map((e) => e.name), i = !r, a = e.emptyTables ? () => e.emptyTables(t) : i ? async () => {
1058
1065
  for (let t of [...s.tables].reverse()) await e.deleteMany(t.name, { where: p(t.primaryKey) });
1059
1066
  } : () => e.transactional(async (e) => {
@@ -1062,53 +1069,54 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
1062
1069
  yield* n.tryPromise({
1063
1070
  try: a,
1064
1071
  catch: (e) => new R({ reason: `emptying the target for --mode replace failed (nothing was deleted): ${Qt(e)}` })
1065
- }), r && (m.truncated = !0, yield* q(f, m));
1072
+ }), r && (h.truncated = !0, yield* q(m, h));
1066
1073
  }
1067
- let a = [], o = [], d = [], h = s.tables.length;
1074
+ let a = [], o = [], f = [], g = s.tables.length;
1068
1075
  for (let p = 0; p < s.tables.length; p++) {
1069
- let g = s.tables[p];
1070
- if (r && g.name in m.tables) {
1071
- d.push(g.name), K(t.onProgress, {
1076
+ let _ = s.tables[p];
1077
+ if (r && _.name in h.tables) {
1078
+ f.push(_.name), k++, O += h.tables[_.name].rowCount, K(t.onProgress, {
1072
1079
  phase: "import",
1073
- table: g.name,
1080
+ table: _.name,
1074
1081
  index: p,
1075
- tableCount: h,
1082
+ tableCount: g,
1076
1083
  event: "done",
1077
- rowsDone: m.tables[g.name].rowCount,
1078
- total: g.rowCount
1084
+ rowsDone: h.tables[_.name].rowCount,
1085
+ total: _.rowCount
1079
1086
  });
1080
1087
  continue;
1081
1088
  }
1082
- let y = i.join(t.bundleDir, g.file);
1089
+ let b = i.join(t.bundleDir, _.file);
1083
1090
  K(t.onProgress, {
1084
1091
  phase: "import",
1085
- table: g.name,
1092
+ table: _.name,
1086
1093
  index: p,
1087
- tableCount: h,
1094
+ tableCount: g,
1088
1095
  event: "start",
1089
1096
  rowsDone: 0,
1090
- total: g.rowCount
1097
+ total: _.rowCount
1091
1098
  });
1092
- let b = a.length, x = n.gen(function* () {
1093
- let { createHash: r } = yield* n.promise(() => import("node:crypto")), i = r("sha256"), o = 0, l = w("bundle", () => O(y).pipe(Ye(s.compression))).pipe(c.tap((e) => n.sync(() => i.update(e)))), d = async (t) => {
1099
+ let x = a.length, S = n.gen(function* () {
1100
+ let { createHash: r } = yield* n.promise(() => import("node:crypto")), i = r("sha256"), o = 0, l = w("bundle", () => j(b).pipe(Ye(s.compression))).pipe(c.tap((e) => n.sync(() => i.update(e)))), f = async (t) => {
1094
1101
  let n = {
1095
- entry: g,
1102
+ entry: _,
1096
1103
  row: t,
1097
1104
  reason: ""
1098
1105
  };
1099
1106
  try {
1100
- await v(e, () => _(e, g, t));
1107
+ await y(e, () => v(e, _, t));
1101
1108
  } catch (e) {
1102
1109
  n.reason = Zt(e), a.push(n);
1103
1110
  }
1104
- }, f = yield* n.promise(async () => {
1111
+ }, p = yield* n.promise(async () => {
1105
1112
  let n = t.copyLoader;
1106
- if (!n || n.dialect !== "postgres" || t.atomic) return !1;
1113
+ if (!n || n.dialect !== "postgres") return !1;
1114
+ if (d) return T || (T = !0, t.onWarn?.("bulk COPY is off for this run: it needs its own connection and this import runs in ONE transaction (the default for --mode replace, so a failure mid-load cannot leave the target emptied and unfilled). Pass --no-atomic to trade that guarantee back for COPY.")), !1;
1107
1115
  if (u === "replace") return !0;
1108
1116
  if (u !== "upsert") return !1;
1109
1117
  try {
1110
1118
  return (await e.query({
1111
- table: g.name,
1119
+ table: _.name,
1112
1120
  predicate: void 0,
1113
1121
  order: [],
1114
1122
  take: 1,
@@ -1118,104 +1126,111 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
1118
1126
  } catch {
1119
1127
  return !1;
1120
1128
  }
1121
- }), p = Object.keys(g.columnTypes), m = Math.max(1, t.copyBatchSize ?? 5e3), h = [], b = async () => {
1122
- if (h.length === 0) return;
1123
- let e = h;
1124
- h = [];
1129
+ }), m = Object.keys(_.columnTypes), h = Math.max(1, t.copyBatchSize ?? 5e3), g = [], x = async () => {
1130
+ if (g.length === 0) return;
1131
+ let e = g;
1132
+ g = [];
1125
1133
  try {
1126
1134
  await t.copyLoader.copyInto({
1127
- table: g.name,
1128
- columns: p,
1129
- columnTypes: g.columnTypes,
1135
+ table: _.name,
1136
+ columns: m,
1137
+ columnTypes: _.columnTypes,
1130
1138
  rows: e
1131
1139
  });
1132
1140
  } catch (n) {
1133
- f = !1, t.onWarn?.(`bulk COPY into ${g.name} failed (${String(n?.message ?? n)}); falling back to per-row writes for this table`);
1134
- for (let t of e) await d(t);
1141
+ p = !1, t.onWarn?.(`bulk COPY into ${_.name} failed (${String(n?.message ?? n)}); falling back to per-row writes for this table`);
1142
+ for (let t of e) await f(t);
1135
1143
  }
1136
- }, x = Math.max(1, t.savepointBatchSize ?? 200), S = typeof e.savepoint == "function", C = [], T = async () => {
1137
- if (C.length === 0) return;
1138
- let t = C;
1139
- C = [];
1144
+ }, S = Math.max(1, t.savepointBatchSize ?? 200), C = typeof e.savepoint == "function", E = [], D = async () => {
1145
+ if (E.length === 0) return;
1146
+ let t = E;
1147
+ E = [];
1140
1148
  try {
1141
1149
  await e.savepoint(async () => {
1142
- for (let n of t) await _(e, g, n);
1150
+ for (let n of t) await v(e, _, n);
1143
1151
  });
1144
1152
  } catch {
1145
- for (let e of t) await d(e);
1153
+ for (let e of t) await f(e);
1146
1154
  }
1147
1155
  };
1148
- return yield* c.runForEach(kt(l, g.name, g.columnTypes), (e) => n.promise(async () => {
1149
- if (o++, f) {
1150
- h.push(e), h.length >= m && await b();
1156
+ return yield* c.runForEach(kt(l, _.name, _.columnTypes), (e) => n.promise(async () => {
1157
+ if (o++, p) {
1158
+ g.push(e), g.length >= h && await x();
1151
1159
  return;
1152
1160
  }
1153
- if (S) {
1154
- C.push(e), C.length >= x && await T();
1161
+ if (C) {
1162
+ E.push(e), E.length >= S && await D();
1155
1163
  return;
1156
1164
  }
1157
- await d(e);
1158
- })), yield* n.promise(T), yield* n.promise(b), {
1165
+ await f(e);
1166
+ })), yield* n.promise(D), yield* n.promise(x), {
1159
1167
  rowCount: o,
1160
1168
  checksum: i.digest("hex")
1161
1169
  };
1162
- }), { rowCount: S, checksum: C } = yield* ft("import", g.name, p, h, (e) => e.rowCount, x);
1170
+ }), { rowCount: C, checksum: A } = yield* ft("import", _.name, p, g, (e) => e.rowCount, S);
1163
1171
  if (l) {
1164
- if (C !== g.checksum) return yield* n.fail(new z({
1165
- what: `table ${g.name} checksum`,
1166
- expected: g.checksum,
1167
- actual: C
1172
+ if (A !== _.checksum) return yield* n.fail(new z({
1173
+ what: `table ${_.name} checksum`,
1174
+ expected: _.checksum,
1175
+ actual: A
1168
1176
  }));
1169
- if (S !== g.rowCount) return yield* n.fail(new z({
1170
- what: `table ${g.name} rowCount`,
1171
- expected: String(g.rowCount),
1172
- actual: String(S)
1177
+ if (C !== _.rowCount) return yield* n.fail(new z({
1178
+ what: `table ${_.name} rowCount`,
1179
+ expected: String(_.rowCount),
1180
+ actual: String(C)
1173
1181
  }));
1174
1182
  }
1175
1183
  K(t.onProgress, {
1176
1184
  phase: "import",
1177
- table: g.name,
1185
+ table: _.name,
1178
1186
  index: p,
1179
- tableCount: h,
1187
+ tableCount: g,
1180
1188
  event: "done",
1181
- rowsDone: S,
1182
- total: g.rowCount
1183
- }), r && (a.length > b ? o.push({
1184
- name: g.name,
1185
- rowCount: S
1186
- }) : (m.tables[g.name] = {
1187
- rowCount: S,
1188
- checksum: g.checksum
1189
- }, yield* q(f, m)));
1189
+ rowsDone: C,
1190
+ total: _.rowCount
1191
+ }), D++, E += C, r && (a.length > x ? o.push({
1192
+ name: _.name,
1193
+ rowCount: C
1194
+ }) : (h.tables[_.name] = {
1195
+ rowCount: C,
1196
+ checksum: _.checksum
1197
+ }, yield* q(m, h)));
1190
1198
  }
1191
- if (a.length > 0 && (yield* S(e, a)), r && o.length > 0) {
1199
+ if (a.length > 0 && (yield* C(e, a)), r && o.length > 0) {
1192
1200
  for (let e of o) {
1193
1201
  let t = s.tables.find((t) => t.name === e.name);
1194
- m.tables[e.name] = {
1202
+ h.tables[e.name] = {
1195
1203
  rowCount: e.rowCount,
1196
1204
  checksum: t.checksum
1197
1205
  };
1198
1206
  }
1199
- yield* q(f, m);
1207
+ yield* q(m, h);
1200
1208
  }
1201
- if (d.length > 0) {
1202
- let e = d.length === s.tables.length;
1203
- t.onWarn?.(`resume: ${d.length} of ${s.tables.length} table(s) were already applied by an earlier run of THIS bundle directory, so this run wrote NO rows for them${e ? " — that is every table in the bundle, so nothing was written at all" : ""}. Tables: ${d.join(", ")}. Delete ${f} to force a full re-import.`);
1209
+ if (f.length > 0) {
1210
+ let e = f.length === s.tables.length;
1211
+ t.onWarn?.(`resume: ${f.length} of ${s.tables.length} table(s) were already applied by an earlier run of THIS bundle directory, so this run wrote NO rows for them${e ? " — that is every table in the bundle, so nothing was written at all" : ""}. Tables: ${f.join(", ")}. Delete ${m} to force a full re-import.`);
1204
1212
  }
1205
1213
  });
1206
- if (t.atomic ? yield* n.tryPromise({
1214
+ if (d ? yield* n.tryPromise({
1207
1215
  try: () => t.store.transactional(async (t) => {
1208
- let i = await n.runPromiseExit(C(t, !1));
1216
+ let i = await n.runPromiseExit(A(t, !1));
1209
1217
  if (r.isFailure(i)) throw Error(e.pretty(i.cause));
1210
1218
  }),
1211
1219
  catch: (e) => new R({ reason: `atomic import failed (rolled back): ${String(e?.message ?? e)}` })
1212
- }) : yield* C(t.store, !0), t.assets && s.assets.index && !t.assetsAlreadyRestored) {
1213
- let e = t.assets, r = i.join(t.bundleDir, G), a = i.join(t.bundleDir, s.assets.index ?? "assets/index.ndjson"), o = w("bundle", () => O(a)).pipe(c.decodeText(), c.splitLines, c.filter((e) => e.trim().length > 0), c.map((e) => JSON.parse(e)));
1214
- yield* c.runForEach(o, (t) => t.key in m.assets ? n.void : We(e, r, t).pipe(n.andThen(n.sync(() => {
1215
- m.assets[t.key] = !0;
1216
- })), n.andThen(q(f, m))));
1220
+ }) : yield* A(t.store, !0), t.assets && s.assets.index && !t.assetsAlreadyRestored) {
1221
+ let e = t.assets, r = i.join(t.bundleDir, G), a = i.join(t.bundleDir, s.assets.index ?? "assets/index.ndjson"), o = w("bundle", () => j(a)).pipe(c.decodeText(), c.splitLines, c.filter((e) => e.trim().length > 0), c.map((e) => JSON.parse(e)));
1222
+ yield* c.runForEach(o, (t) => t.key in h.assets ? n.void : We(e, r, t).pipe(n.andThen(n.sync(() => {
1223
+ h.assets[t.key] = !0;
1224
+ })), n.andThen(q(m, h))));
1217
1225
  }
1218
- return s;
1226
+ return {
1227
+ manifest: s,
1228
+ tablesWritten: D,
1229
+ rowsWritten: E,
1230
+ tablesSkipped: k,
1231
+ rowsSkipped: O,
1232
+ fullyResumed: D === 0 && k > 0
1233
+ };
1219
1234
  }), sn = (e) => e, cn = (e) => {
1220
1235
  if (e.allowLive || e.liveInstance === null || e.sameTarget === "different") return { refuse: !1 };
1221
1236
  let t = e.targetLabel === void 0 ? "" : ` Target: ${e.targetLabel}.`, n = e.sameTarget === "same" ? " That instance is connected to the SAME database." : " We could not confirm which database that instance is connected to, so this is refused on the conservative reading — an older instance, or one whose connection could not be parsed, reports nothing to compare.";
@@ -1227,7 +1242,7 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
1227
1242
  let t = [], n = async (r) => {
1228
1243
  let i;
1229
1244
  try {
1230
- i = await ce(k(e, r), { withFileTypes: !0 });
1245
+ i = await ae(M(e, r), { withFileTypes: !0 });
1231
1246
  } catch {
1232
1247
  return;
1233
1248
  }
@@ -1247,8 +1262,8 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
1247
1262
  async function* gn(e) {
1248
1263
  yield new Uint8Array(Y);
1249
1264
  for (let t of await pn(e.dir)) {
1250
- let n = k(e.dir, t), r = await le(n);
1251
- if (yield new Uint8Array(mn(t, r.size)), r.size > 0) for await (let e of O(n)) yield new Uint8Array(e);
1265
+ let n = M(e.dir, t), r = await oe(n);
1266
+ if (yield new Uint8Array(mn(t, r.size)), r.size > 0) for await (let e of j(n)) yield new Uint8Array(e);
1252
1267
  }
1253
1268
  if (e.blobs) for await (let t of e.blobs.list()) {
1254
1269
  yield new Uint8Array(hn(t.key, t.contentType, t.size));
@@ -1263,8 +1278,8 @@ async function* gn(e) {
1263
1278
  yield new Uint8Array(Buffer.from([ln]));
1264
1279
  }
1265
1280
  var _n = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), vn = (e) => new Promise((t) => e.end(t)), yn = async (e, t) => {
1266
- let n = pe(t.dir);
1267
- await se(n, { recursive: !0 });
1281
+ let n = ue(t.dir);
1282
+ await ie(n, { recursive: !0 });
1268
1283
  let r = Buffer.alloc(0), i = e[Symbol.asyncIterator](), a = !1, o = async () => {
1269
1284
  let { value: e, done: t } = await i.next();
1270
1285
  return t ? (a = !0, !1) : (r = r.length === 0 ? Buffer.from(e) : Buffer.concat([r, Buffer.from(e)]), !0);
@@ -1286,10 +1301,10 @@ var _n = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), vn
1286
1301
  await s(e);
1287
1302
  let t = c(e).toString("utf8");
1288
1303
  await s(8);
1289
- let i = c(8).readBigUInt64BE(0), a = pe(n, t);
1290
- if (a !== n && !a.startsWith(n + me)) throw Error(`archive path '${t}' escapes the destination`);
1291
- await se(fe(a), { recursive: !0 });
1292
- let o = ie(a);
1304
+ let i = c(8).readBigUInt64BE(0), a = ue(n, t);
1305
+ if (a !== n && !a.startsWith(n + de)) throw Error(`archive path '${t}' escapes the destination`);
1306
+ await ie(le(a), { recursive: !0 });
1307
+ let o = te(a);
1293
1308
  for (; i > 0n;) {
1294
1309
  r.length === 0 && await s(1);
1295
1310
  let e = i < BigInt(r.length) ? Number(i) : r.length;
@@ -1306,7 +1321,7 @@ var _n = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), vn
1306
1321
  await s(i);
1307
1322
  let a = c(i).toString("utf8");
1308
1323
  await s(8);
1309
- let o = c(8).readBigUInt64BE(0), l = await t.blobs.has(n), u = y("sha256"), d = new D(), f = l ? Promise.resolve() : t.blobs.write(n, d, {
1324
+ let o = c(8).readBigUInt64BE(0), l = await t.blobs.has(n), u = y("sha256"), d = new k(), f = l ? Promise.resolve() : t.blobs.write(n, d, {
1310
1325
  contentType: a,
1311
1326
  size: Number(o)
1312
1327
  }), p = o;
@@ -1364,14 +1379,14 @@ var _n = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), vn
1364
1379
  return {
1365
1380
  has: async (e) => t.has(e),
1366
1381
  write: async (n, r, i) => {
1367
- let a = await import("node:fs/promises"), { pipeline: o } = await import("node:stream/promises"), { Transform: s } = await import("node:stream"), c = k(e, "assets");
1368
- await se(c, { recursive: !0 });
1382
+ let a = await import("node:fs/promises"), { pipeline: o } = await import("node:stream/promises"), { Transform: s } = await import("node:stream"), c = M(e, "assets");
1383
+ await ie(c, { recursive: !0 });
1369
1384
  let l = y("sha256"), u = 0, d = new s({ transform(e, t, n) {
1370
1385
  l.update(e), u += e.length, n(null, e);
1371
- } }), f = k(c, `.tmp-${Date.now()}-${t.size}`);
1372
- await o(r, d, ie(f));
1386
+ } }), f = M(c, `.tmp-${Date.now()}-${t.size}`);
1387
+ await o(r, d, te(f));
1373
1388
  let p = l.digest("hex");
1374
- await a.rename(f, k(c, p)), await a.appendFile(k(c, "index.ndjson"), `${JSON.stringify({
1389
+ await a.rename(f, M(c, p)), await a.appendFile(M(c, "index.ndjson"), `${JSON.stringify({
1375
1390
  key: n,
1376
1391
  sha256: p,
1377
1392
  size: u,
@@ -1379,7 +1394,7 @@ var _n = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), vn
1379
1394
  })}\n`), t.add(n);
1380
1395
  }
1381
1396
  };
1382
- }, Cn = he(S), wn = Buffer.from("VENC1\0\0\0"), Tn = 32, En = 16, Dn = 7, X = 16, On = 262144, Z = {
1397
+ }, Cn = fe(S), wn = Buffer.from("VENC1\0\0\0"), Tn = 32, En = 16, Dn = 7, X = 16, On = 262144, Z = {
1383
1398
  N: 32768,
1384
1399
  r: 8,
1385
1400
  p: 1
@@ -1457,7 +1472,7 @@ var Nn = (e, t) => ({
1457
1472
  } };
1458
1473
  },
1459
1474
  open: (t) => ({ async *[Symbol.asyncIterator]() {
1460
- let r = await n.runPromise(T(e, t)), i = await te(r.stream);
1475
+ let r = await n.runPromise(T(e, t)), i = await D(r.stream);
1461
1476
  for await (let e of i) yield e instanceof Uint8Array ? e : new Uint8Array(e);
1462
1477
  } })
1463
1478
  }), Pn = (e, t) => Nn(e, () => h(t, {
@@ -1470,8 +1485,8 @@ var Nn = (e, t) => ({
1470
1485
  })))), Fn = (e) => ({
1471
1486
  has: (t) => n.runPromise(e.head(t).pipe(n.map((e) => e !== null))),
1472
1487
  write: async (t, r, i) => {
1473
- let a = w("storage", () => ne.from(r));
1474
- await n.runPromise(ee(e, t, a, {
1488
+ let a = w("storage", () => A.from(r));
1489
+ await n.runPromise(E(e, t, a, {
1475
1490
  contentType: i.contentType,
1476
1491
  totalSize: i.size
1477
1492
  }));
@@ -1479,7 +1494,7 @@ var Nn = (e, t) => ({
1479
1494
  }), In = (e, t) => {
1480
1495
  let n = t === "dump" ? "mariadb-dump" : "mariadb", r = t === "dump" ? "mysqldump" : "mysql";
1481
1496
  return e === "mariadb" && Ln(n) ? n : r;
1482
- }, Ln = (e) => (process.env.PATH ?? "").split(de).filter((e) => e !== "").some((t) => ae(k(t, e))), Rn = (e) => {
1497
+ }, Ln = (e) => (process.env.PATH ?? "").split(ce).filter((e) => e !== "").some((t) => ne(M(t, e))), Rn = (e) => {
1483
1498
  switch (e) {
1484
1499
  case "postgres": return "db.dump";
1485
1500
  case "mysql":
@@ -1680,12 +1695,12 @@ var Nn = (e, t) => ({
1680
1695
  });
1681
1696
  }
1682
1697
  }, Un = (e) => e.kind === "copy" ? n.async((t) => {
1683
- re(e.from, e.to, (r) => t(r ? n.fail(new V({
1698
+ ee(e.from, e.to, (r) => t(r ? n.fail(new V({
1684
1699
  tool: "copy",
1685
1700
  reason: `copy ${e.from} → ${e.to}: ${r.message}`
1686
1701
  })) : n.void));
1687
1702
  }) : n.async((t) => {
1688
- let r = ge(e.tool, [...e.args], {
1703
+ let r = pe(e.tool, [...e.args], {
1689
1704
  env: {
1690
1705
  ...process.env,
1691
1706
  ...e.env
@@ -1698,10 +1713,10 @@ var Nn = (e, t) => ({
1698
1713
  }), i = "";
1699
1714
  r.stderr?.on("data", (e) => {
1700
1715
  i += String(e);
1701
- }), e.stdoutFile && r.stdout && r.stdout.pipe(ie(e.stdoutFile)), e.stdinFile && r.stdin && O(e.stdinFile).pipe(r.stdin);
1716
+ }), e.stdoutFile && r.stdout && r.stdout.pipe(te(e.stdoutFile)), e.stdinFile && r.stdin && j(e.stdinFile).pipe(r.stdin);
1702
1717
  let a = () => {
1703
1718
  if (e.stdoutFile !== void 0) try {
1704
- oe(e.stdoutFile);
1719
+ re(e.stdoutFile);
1705
1720
  } catch {}
1706
1721
  };
1707
1722
  r.on("error", (r) => {
@@ -1716,7 +1731,7 @@ var Nn = (e, t) => ({
1716
1731
  }
1717
1732
  a(), t(n.fail(new V({
1718
1733
  tool: e.tool,
1719
- reason: `exited with code ${r}` + (e.stdoutFile === void 0 ? "" : ` — the partial ${ue(e.stdoutFile)} was removed, so it cannot be mistaken for a backup`),
1734
+ reason: `exited with code ${r}` + (e.stdoutFile === void 0 ? "" : ` — the partial ${se(e.stdoutFile)} was removed, so it cannot be mistaken for a backup`),
1720
1735
  stderr: i.slice(-2e3)
1721
1736
  })));
1722
1737
  });
@@ -1724,7 +1739,7 @@ var Nn = (e, t) => ({
1724
1739
  let t = yield* Be(e.scope ?? { kind: "all" }, e.snapshot, {
1725
1740
  store: e.store,
1726
1741
  ...e.tenantTables ? { tenantTables: e.tenantTables } : {}
1727
- }), r = At(t.tables, e.snapshot), i = e.classification ?? A(e.snapshot), a = e.sampleSize ?? 20, o = I(r.map((t) => ({
1742
+ }), r = At(t.tables, e.snapshot), i = e.classification ?? N(e.snapshot), a = e.sampleSize ?? 20, o = Ee(r.map((t) => ({
1728
1743
  name: t,
1729
1744
  columns: (e.snapshot.tables.find((e) => e.name === t)?.columns ?? []).map((e) => ({
1730
1745
  name: e.name,
@@ -1736,8 +1751,8 @@ var Nn = (e, t) => ({
1736
1751
  table: i,
1737
1752
  chunkSize: a,
1738
1753
  ...r === void 0 ? {} : { where: r }
1739
- }).pipe(c.take(a), c.runCollect, n.map((e) => [...e])), d = u.map((t) => L(t, i, o.actions, e.masking.seed));
1740
- l.push(...ke(d, i, o.actions));
1754
+ }).pipe(c.take(a), c.runCollect, n.map((e) => [...e])), d = u.map((t) => ke(t, i, o.actions, e.masking.seed));
1755
+ l.push(...Oe(d, i, o.actions));
1741
1756
  let f = u[0], p = d[0], m = [];
1742
1757
  if (f && p) for (let [e, t] of Object.entries(o.actions)) {
1743
1758
  if (!e.startsWith(`${i}.`) || t === "keep") continue;
@@ -1825,4 +1840,4 @@ var Nn = (e, t) => ({
1825
1840
  return n.push(""), n.push(` discarded: ${r("value-dropped")} defaulted: ${r("filled-by-default")} refused: ${r("refuse")}`), e.refuse && (n.push(""), n.push(" REFUSED — no row was written. These are the differences that would make"), n.push(" imported rows WRONG rather than merely incomplete.")), n;
1826
1841
  };
1827
1842
  //#endregion
1828
- export { G as ASSETS_DIR, at as ASSET_INDEX_FILE, R as BundleError, Ae as CodecError, je as CompressionError, Ie as CrossDialectError, it as DATA_DIR, ve as DEFAULT_CLASS_ACTIONS, Xe as FORMAT_VERSION, Fe as ImportModeError, z as IntegrityError, ot as LEDGER_FILE, W as MANIFEST_FILE, Pe as MaskingError, V as NativeToolError, Me as RowsRefusedError, rn as SAVEPOINT_BATCH_SIZE, Ne as SchemaDriftError, B as ScopeError, Ee as applyAction, cn as assessLiveImport, gt as assetDone, Rn as backupArtifactName, Vn as backupCommand, Wt as canonicalType, qt as checkPortability, nn as checkSchemaDrift, A as classificationFromSnapshot, Kn as classifyImportDrift, qe as compressionExt, Re as computeSubsetIds, nt as decodeManifest, Et as decodeRow, wt as decodeValue, Mn as decryptStream, sn as defineDataProfile, tn as diffSnapshots, Sn as dirBlobSink, pt as emptyLedger, tt as encodeManifest, Tt as encodeRow, St as encodeValue, jn as encryptStream, qn as formatImportDrift, Jt as formatIssue, Ge as hasZstd, Je as makeCompressor, Ye as makeDecompressor, L as maskRow, kt as ndjsonToRows, Ut as outsideReferenceRefusal, Ht as outsideReferencesInto, gn as packBundle, bn as peekBundle, Ke as pickCompression, I as planMasking, Gn as previewMasking, mt as readLedger, ye as resolveAction, Be as resolveScope, We as restoreAssetFromCas, Hn as restoreCommand, Ot as rowsToNdjson, Vt as runExport, on as runImport, Un as runNativeStep, ke as scanForLeaks, He as storageAssetSink, Ve as storageAssetSource, Fn as storageBlobSink, Nn as storageBlobSource, Pn as storageRefsBlobSource, ze as subsetToScope, ht as tableDone, At as topoSortTables, yn as unpackBundle, xn as unpackBundleToDir, Ue as writeAssetToCas, q as writeLedger };
1843
+ export { G as ASSETS_DIR, at as ASSET_INDEX_FILE, R as BundleError, Ae as CodecError, je as CompressionError, Ie as CrossDialectError, it as DATA_DIR, he as DEFAULT_CLASS_ACTIONS, Xe as FORMAT_VERSION, Fe as ImportModeError, z as IntegrityError, ot as LEDGER_FILE, W as MANIFEST_FILE, Pe as MaskingError, V as NativeToolError, Me as RowsRefusedError, rn as SAVEPOINT_BATCH_SIZE, Ne as SchemaDriftError, B as ScopeError, we as applyAction, cn as assessLiveImport, gt as assetDone, Rn as backupArtifactName, Vn as backupCommand, Wt as canonicalType, qt as checkPortability, nn as checkSchemaDrift, N as classificationFromSnapshot, Kn as classifyImportDrift, qe as compressionExt, Re as computeSubsetIds, nt as decodeManifest, Et as decodeRow, wt as decodeValue, Mn as decryptStream, sn as defineDataProfile, tn as diffSnapshots, Sn as dirBlobSink, pt as emptyLedger, tt as encodeManifest, Tt as encodeRow, St as encodeValue, jn as encryptStream, qn as formatImportDrift, Jt as formatIssue, Ge as hasZstd, Je as makeCompressor, Ye as makeDecompressor, ke as maskRow, kt as ndjsonToRows, Ut as outsideReferenceRefusal, Ht as outsideReferencesInto, gn as packBundle, bn as peekBundle, Ke as pickCompression, Ee as planMasking, Gn as previewMasking, mt as readLedger, ge as resolveAction, Be as resolveScope, We as restoreAssetFromCas, Hn as restoreCommand, Ot as rowsToNdjson, Vt as runExport, on as runImport, Un as runNativeStep, Oe as scanForLeaks, He as storageAssetSink, Ve as storageAssetSource, Fn as storageBlobSink, Nn as storageBlobSource, Pn as storageRefsBlobSource, ze as subsetToScope, ht as tableDone, At as topoSortTables, yn as unpackBundle, xn as unpackBundleToDir, Ue as writeAssetToCas, q as writeLedger };
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@voltro/data-transfer",
3
- "version": "0.43.2",
3
+ "version": "0.44.0",
4
4
  "description": "Portable export/import for a Voltro app's data + assets — typed-NDJSON row codec, content-addressed assets, framed compression, a resumable chunk ledger, and FK-ordered idempotent import. Dialect-agnostic; the resilient backbone under `voltro data export|import` and native backups.",
5
5
  "keywords": [
6
6
  "voltro",
@@ -33,8 +33,8 @@
33
33
  "node": ">=24.0.0"
34
34
  },
35
35
  "dependencies": {
36
- "@voltro/database": "0.43.2",
37
- "@voltro/plugin-storage": "0.43.2"
36
+ "@voltro/database": "0.44.0",
37
+ "@voltro/plugin-storage": "0.44.0"
38
38
  },
39
39
  "peerDependencies": {
40
40
  "effect": "^3.22.0"