@voltro/data-transfer 0.43.2 → 0.44.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +34 -0
- package/dist/index.d.ts +32 -1
- package/dist/index.js +190 -175
- package/package.json +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -39,6 +39,40 @@ _Changes staged for the next release accumulate here (rolled up from
|
|
|
39
39
|
|
|
40
40
|
---
|
|
41
41
|
|
|
42
|
+
## [0.44.0] — 2026-08-19
|
|
43
|
+
|
|
44
|
+
### ⚠ BREAKING
|
|
45
|
+
|
|
46
|
+
- **@voltro/data-transfer, @voltro/cli** — `runImport` returns an `ImportOutcome` instead of the bundle's `Manifest`, and `--mode replace` runs as ONE transaction by default.
|
|
47
|
+
|
|
48
|
+
**Why the return type changed.** The summary line counted the rows the BUNDLE carries, not the rows the run wrote. Those differ most exactly where it matters: a bundle directory carries its own resume ledger, so a copied directory imports nothing — correctly, with a warning naming the file to delete — and the run then printed `import complete … rows: 242950` over a target it had not touched. The warning was one line above, which is one line too far for anyone piping the output through `tail -1`. The outcome carries `rowsWritten`, `rowsSkipped` and `fullyResumed`, the CLI reports written-vs-carried, and a fully-skipped run says so on its LAST line.
|
|
49
|
+
|
|
50
|
+
Migration: `runImport(...)` now resolves to `{ manifest, tablesWritten, rowsWritten, tablesSkipped, rowsSkipped, fullyResumed }`. Read `.manifest` where you read the manifest before.
|
|
51
|
+
|
|
52
|
+
**Why replace is atomic now.** The all-or-nothing guarantee was written for the emptying step, and read — reasonably — as covering the run. A replace that died partway through the LOAD left the target emptied of its old rows and holding part of the new ones, measured on a live instance over sixteen minutes. There is no useful state for a replace to stop in, so it is a default rather than a flag you have to know about. It also closes the window that produced that failure: with the tables emptied and the load uncommitted, a concurrent writer in the application waits instead of inserting a row the bundle is about to insert too.
|
|
53
|
+
|
|
54
|
+
`--no-atomic` (CLI) / `atomic: false` (API) opts out. The trade is stated where it bites: every write to those tables waits for the load, and on postgres the bulk `COPY` loader cannot join a transaction it does not own — an atomic run now says that once rather than being quietly slower.
|
|
55
|
+
|
|
56
|
+
Two smaller things from the same report: `--mode replace` against a running instance warns about the empty-target window when it is NOT atomic, and the `--target api` upload reports progress per chunk plus a line explaining that the final request stays open for the whole import — sixteen minutes of silence is indistinguishable from a hang, and one operator killed a run that had finished.
|
|
57
|
+
|
|
58
|
+
**`voltro update` carries you across this** — codemod `0.44.0/01_import-outcome`.
|
|
59
|
+
|
|
60
|
+
### Fixed
|
|
61
|
+
|
|
62
|
+
- **@voltro/sql-mysql, @voltro/sql-postgres, @voltro/sql-sqlite, @voltro/sql-mssql, @voltro/data-transfer** — A write and its write-recorders now succeed or fail together on every SQL dialect, and the data importer's retry is idempotent.
|
|
63
|
+
|
|
64
|
+
A recorder (a versioning trail, an audit log) runs on the caller's connection and is ALLOWED to fail — a recorder that throws must take the write down with it, that is its contract. Outside a caller transaction the two were not one unit: the row's statement committed on its own, and the recorder's INSERT ran afterwards as a second autocommit statement. So a recorder that threw left a COMMITTED row behind a write that reported failure. Measured directly: `insert` throws, the row is in the table, and a second attempt at the same row is `ER_DUP_ENTRY` on PRIMARY.
|
|
65
|
+
|
|
66
|
+
Anything that retries a failed write then meets its own row. The data importer retries by design — it holds a row whose write failed and tries again once the remaining tables have streamed — so a `--mode replace` that had just emptied a table failed on a duplicate key IN that table. From outside, that is the impossible-looking thing: an import that emptied a table and then failed because something was already in it. Reproduced verbatim, including the table name and `[ER_DUP_ENTRY/1062]`, by putting the old retry back.
|
|
67
|
+
|
|
68
|
+
Both ends are closed, and they are independent on purpose:
|
|
69
|
+
|
|
70
|
+
- **The cause.** A table that HAS recorders writes inside a transaction now, so the row and the trail commit together or not at all — on mysql, mariadb, postgres, sqlite and mssql, verified by one suite that asks all five the same question. A table with no recorders — the default — takes the direct path unchanged; `recordsTable` is a Map-size check first, so it costs one comparison. - **The defence.** The importer's retry upserts instead of inserting in `replace` mode. That covers every OTHER way a write can land while reporting failure: a driver timeout on a write the server applied, a connection lost after the commit, a concurrent writer inserting the same key. It is sound precisely because the table was emptied by this same run — there is nothing in it that is not ours.
|
|
71
|
+
|
|
72
|
+
The test that pinned the divergence went red when the fix landed, exactly as its own note said it would, and is inverted with that note kept.
|
|
73
|
+
|
|
74
|
+
---
|
|
75
|
+
|
|
42
76
|
## [0.43.2] — 2026-08-18
|
|
43
77
|
|
|
44
78
|
### Fixed
|
package/dist/index.d.ts
CHANGED
|
@@ -570,6 +570,10 @@ export declare interface ImportOptions {
|
|
|
570
570
|
* `replace`. Trade-offs: one long-held write transaction (writes to those
|
|
571
571
|
* tables block for the load; readers are unaffected on MVCC dialects), and
|
|
572
572
|
* resume is not per-table (a crash rolls the whole thing back → re-run).
|
|
573
|
+
*
|
|
574
|
+
* DEFAULTS TO TRUE FOR `replace`, and to false otherwise. Omit it to get that
|
|
575
|
+
* rule; pass `false` to opt out deliberately. See where it is resolved in
|
|
576
|
+
* `runImport` for why replace cannot sensibly run without it.
|
|
573
577
|
*/
|
|
574
578
|
readonly atomic?: boolean;
|
|
575
579
|
/** The streaming import path (`unpackBundle` with a storage {@link AssetSink})
|
|
@@ -648,6 +652,33 @@ export declare interface ImportOptions {
|
|
|
648
652
|
readonly savepointBatchSize?: number;
|
|
649
653
|
}
|
|
650
654
|
|
|
655
|
+
/**
|
|
656
|
+
* What a run actually did — as opposed to what the bundle contains.
|
|
657
|
+
*
|
|
658
|
+
* The two are not the same, and reporting the bundle's numbers as the run's is a
|
|
659
|
+
* lie of the same family this package keeps finding: a re-run of a bundle
|
|
660
|
+
* directory whose ledger says every table is applied writes NOTHING and used to
|
|
661
|
+
* be summarised as `import complete … rows: 242950`. The warning that explained
|
|
662
|
+
* it was one line further up, which is one line too far for anyone who pipes the
|
|
663
|
+
* output through `tail`.
|
|
664
|
+
*
|
|
665
|
+
* `rowsWritten` counts rows this run streamed into the target. `rowsSkipped`
|
|
666
|
+
* counts rows in tables an earlier run of THIS bundle directory had already
|
|
667
|
+
* applied. A dry run reports zero written and nothing skipped: it examined
|
|
668
|
+
* everything and wrote nothing.
|
|
669
|
+
*/
|
|
670
|
+
export declare interface ImportOutcome {
|
|
671
|
+
/** The bundle's own contract — what it CARRIES. */
|
|
672
|
+
readonly manifest: Manifest;
|
|
673
|
+
readonly tablesWritten: number;
|
|
674
|
+
readonly rowsWritten: number;
|
|
675
|
+
/** Tables an earlier run of this bundle directory already applied. */
|
|
676
|
+
readonly tablesSkipped: number;
|
|
677
|
+
readonly rowsSkipped: number;
|
|
678
|
+
/** True when nothing was written because everything was already applied. */
|
|
679
|
+
readonly fullyResumed: boolean;
|
|
680
|
+
}
|
|
681
|
+
|
|
651
682
|
/** An integrity check failed — a table's content checksum or row count did not
|
|
652
683
|
* match the manifest, or an asset's bytes did not hash to its recorded id. */
|
|
653
684
|
export declare class IntegrityError extends IntegrityError_base {
|
|
@@ -1115,7 +1146,7 @@ export declare const rowsToNdjson: <E, R>(rows: Stream.Stream<Row, E, R>, column
|
|
|
1115
1146
|
|
|
1116
1147
|
export declare const runExport: (opts: ExportOptions) => Effect.Effect<Manifest, ExportError>;
|
|
1117
1148
|
|
|
1118
|
-
export declare const runImport: (opts: ImportOptions) => Effect.Effect<
|
|
1149
|
+
export declare const runImport: (opts: ImportOptions) => Effect.Effect<ImportOutcome, ImportError>;
|
|
1119
1150
|
|
|
1120
1151
|
/** Execute a {@link NativeStep} — spawn the tool (wiring stdin/stdout files) or
|
|
1121
1152
|
* copy the file. Maps a missing binary / non-zero exit to {@link NativeToolError}. */
|
package/dist/index.js
CHANGED
|
@@ -2,16 +2,16 @@ import { Cause as e, Data as t, Effect as n, Exit as r, Metric as i, MetricBound
|
|
|
2
2
|
import { classifyConstraintViolation as l, eq as u, extractDbCause as d, inSet as f, isNotNull as p, or as m, streamTable as h } from "@voltro/database";
|
|
3
3
|
import { fingerprintSchema as g } from "@voltro/database/sql";
|
|
4
4
|
import { createCipheriv as _, createDecipheriv as v, createHash as y, createHmac as b, randomBytes as x, scrypt as S } from "node:crypto";
|
|
5
|
-
import { STORAGE_REFS_TABLE as C, fromNodeReadable as w, getObjectStream as T, putObjectStream as
|
|
6
|
-
import * as
|
|
7
|
-
import { PassThrough as
|
|
8
|
-
import { copyFile as
|
|
9
|
-
import { mkdir as
|
|
10
|
-
import { basename as
|
|
11
|
-
import { promisify as
|
|
12
|
-
import { spawn as
|
|
5
|
+
import { STORAGE_REFS_TABLE as C, fromNodeReadable as w, getObjectStream as T, putObjectStream as E, toNodeReadable as D } from "@voltro/plugin-storage";
|
|
6
|
+
import * as O from "node:zlib";
|
|
7
|
+
import { PassThrough as k, Readable as A } from "node:stream";
|
|
8
|
+
import { copyFile as ee, createReadStream as j, createWriteStream as te, existsSync as ne, unlinkSync as re } from "node:fs";
|
|
9
|
+
import { mkdir as ie, readdir as ae, stat as oe } from "node:fs/promises";
|
|
10
|
+
import { basename as se, delimiter as ce, dirname as le, join as M, resolve as ue, sep as de } from "node:path";
|
|
11
|
+
import { promisify as fe } from "node:util";
|
|
12
|
+
import { spawn as pe } from "node:child_process";
|
|
13
13
|
//#region src/consistency.ts
|
|
14
|
-
var
|
|
14
|
+
var me = (e) => {
|
|
15
15
|
switch (e) {
|
|
16
16
|
case "postgres": return ["SET TRANSACTION ISOLATION LEVEL REPEATABLE READ"];
|
|
17
17
|
case "mysql":
|
|
@@ -21,7 +21,7 @@ var _e = (e) => {
|
|
|
21
21
|
case "turso": return [];
|
|
22
22
|
default: return [];
|
|
23
23
|
}
|
|
24
|
-
},
|
|
24
|
+
}, he = {
|
|
25
25
|
email: { fake: "email" },
|
|
26
26
|
fullName: { fake: "fullName" },
|
|
27
27
|
firstName: { fake: "firstName" },
|
|
@@ -36,7 +36,7 @@ var _e = (e) => {
|
|
|
36
36
|
date: "dateShift",
|
|
37
37
|
secret: "null",
|
|
38
38
|
freeText: "redact"
|
|
39
|
-
},
|
|
39
|
+
}, N = (e) => {
|
|
40
40
|
let t = {};
|
|
41
41
|
for (let n of e.tables) {
|
|
42
42
|
let e = {};
|
|
@@ -48,15 +48,15 @@ var _e = (e) => {
|
|
|
48
48
|
t[n.name] = e;
|
|
49
49
|
}
|
|
50
50
|
return t;
|
|
51
|
-
},
|
|
51
|
+
}, ge = (e, t, n, r) => {
|
|
52
52
|
let i = r.columns?.[`${e}.${t}`];
|
|
53
53
|
if (i) return i;
|
|
54
54
|
if (n?.sensitive) {
|
|
55
55
|
let e = n.sensitive.class;
|
|
56
|
-
return r.classes?.[e] ??
|
|
56
|
+
return r.classes?.[e] ?? he[e] ?? "redact";
|
|
57
57
|
}
|
|
58
58
|
return n?.safe ? r.onSafe ?? "keep" : n?.columnType === "id" || n?.columnType === "reference" ? "keep" : r.onUnclassified ?? "error";
|
|
59
|
-
},
|
|
59
|
+
}, P = (e, t) => b("sha256", e).update(t == null ? "∅" : String(t)).digest(), F = (e, t) => e.readUInt32BE(t * 4 % (e.length - 4)), I = [
|
|
60
60
|
"Ada",
|
|
61
61
|
"Alan",
|
|
62
62
|
"Grace",
|
|
@@ -69,7 +69,7 @@ var _e = (e) => {
|
|
|
69
69
|
"Guido",
|
|
70
70
|
"Margaret",
|
|
71
71
|
"Bjarne"
|
|
72
|
-
],
|
|
72
|
+
], _e = [
|
|
73
73
|
"Lovelace",
|
|
74
74
|
"Turing",
|
|
75
75
|
"Hopper",
|
|
@@ -82,7 +82,7 @@ var _e = (e) => {
|
|
|
82
82
|
"Rossum",
|
|
83
83
|
"Hamilton",
|
|
84
84
|
"Stroustrup"
|
|
85
|
-
],
|
|
85
|
+
], ve = [
|
|
86
86
|
"Maple",
|
|
87
87
|
"Oak",
|
|
88
88
|
"Cedar",
|
|
@@ -91,7 +91,7 @@ var _e = (e) => {
|
|
|
91
91
|
"Birch",
|
|
92
92
|
"Willow",
|
|
93
93
|
"Ash"
|
|
94
|
-
],
|
|
94
|
+
], ye = [
|
|
95
95
|
"Initech",
|
|
96
96
|
"Hooli",
|
|
97
97
|
"Umbrella",
|
|
@@ -100,23 +100,23 @@ var _e = (e) => {
|
|
|
100
100
|
"Soylent",
|
|
101
101
|
"Stark",
|
|
102
102
|
"Wayne"
|
|
103
|
-
],
|
|
104
|
-
let r =
|
|
103
|
+
], L = (e, t, n) => e[F(t, n) % e.length], be = (e, t, n) => {
|
|
104
|
+
let r = P(n, `${e}:${String(t)}`), i = F(r, 1) % 1e4;
|
|
105
105
|
switch (e) {
|
|
106
|
-
case "email": return `${
|
|
107
|
-
case "fullName": return `${
|
|
108
|
-
case "firstName": return
|
|
109
|
-
case "lastName": return
|
|
110
|
-
case "username": return `${
|
|
111
|
-
case "phone": return `+1${String(
|
|
112
|
-
case "address": return `${
|
|
113
|
-
case "company": return
|
|
106
|
+
case "email": return `${L(I, r, 0).toLowerCase()}.${L(_e, r, 2).toLowerCase()}${i}@example.com`;
|
|
107
|
+
case "fullName": return `${L(I, r, 0)} ${L(_e, r, 1)}`;
|
|
108
|
+
case "firstName": return L(I, r, 0);
|
|
109
|
+
case "lastName": return L(_e, r, 1);
|
|
110
|
+
case "username": return `${L(I, r, 0).toLowerCase()}${i}`;
|
|
111
|
+
case "phone": return `+1${String(F(r, 0) % 1e10).padStart(10, "0")}`;
|
|
112
|
+
case "address": return `${F(r, 0) % 9899 + 100} ${L(ve, r, 1)} St`;
|
|
113
|
+
case "company": return L(ye, r, 0);
|
|
114
114
|
case "url": return `https://example.com/${r.toString("hex").slice(0, 12)}`;
|
|
115
|
-
case "ip": return `${
|
|
116
|
-
case "creditCard": return
|
|
115
|
+
case "ip": return `${F(r, 0) % 256}.${F(r, 1) % 256}.${F(r, 2) % 256}.${F(r, 3) % 254 + 1}`;
|
|
116
|
+
case "creditCard": return xe(`4${r.toString("hex").replace(/\D/g, "").padEnd(14, "0").slice(0, 14)}`);
|
|
117
117
|
default: return `${e}_${r.toString("hex").slice(0, 8)}`;
|
|
118
118
|
}
|
|
119
|
-
},
|
|
119
|
+
}, xe = (e) => {
|
|
120
120
|
let t = e.slice(0, 15).split("").map(Number), n = 0;
|
|
121
121
|
t.forEach((e, t) => {
|
|
122
122
|
let r = e;
|
|
@@ -124,31 +124,31 @@ var _e = (e) => {
|
|
|
124
124
|
});
|
|
125
125
|
let r = (10 - n % 10) % 10;
|
|
126
126
|
return e.slice(0, 15) + String(r);
|
|
127
|
-
},
|
|
127
|
+
}, Se = 864e5, Ce = (e, t) => {
|
|
128
128
|
let n = e instanceof Date ? e.getTime() : Date.parse(String(e));
|
|
129
129
|
if (Number.isNaN(n)) return e;
|
|
130
|
-
let r =
|
|
130
|
+
let r = F(P(t, "date-shift-offset"), 0) % 730 - 365, i = new Date(n + r * Se);
|
|
131
131
|
return e instanceof Date ? i : i.toISOString();
|
|
132
|
-
},
|
|
132
|
+
}, we = (e, t) => {
|
|
133
133
|
if (t.value === null || t.value === void 0) return null;
|
|
134
134
|
if (e === "error") throw Error(`masking: unresolved 'error' action for ${t.table}.${t.column}`);
|
|
135
135
|
if (e === "keep") return t.value;
|
|
136
136
|
if (e === "null") return null;
|
|
137
137
|
if (e === "redact") return t.columnType === "text" || typeof t.value == "string" ? "[redacted]" : null;
|
|
138
|
-
if (e === "hash") return
|
|
139
|
-
if (e === "dateShift") return
|
|
140
|
-
if (typeof e == "object" && "fake" in e) return
|
|
138
|
+
if (e === "hash") return P(t.seed, t.value).toString("hex").slice(0, 16);
|
|
139
|
+
if (e === "dateShift") return Ce(t.value, t.seed);
|
|
140
|
+
if (typeof e == "object" && "fake" in e) return be(e.fake, t.value, t.seed);
|
|
141
141
|
if (typeof e == "object" && "custom" in e) return e.custom(t);
|
|
142
142
|
throw Error(`masking: '${t.table}.${t.column}' resolved to an action this build does not understand (${JSON.stringify(e)}). Refusing to copy the value through. Valid actions: 'keep' | 'null' | 'redact' | 'hash' | 'dateShift' | { fake: '<class>' } | { custom: fn }.`);
|
|
143
|
-
},
|
|
143
|
+
}, Te = (e) => typeof e == "string" ? e === "keep" || e === "null" || e === "redact" || e === "hash" || e === "dateShift" || e === "error" : typeof e != "object" || !e ? !1 : "fake" in e ? typeof e.fake == "string" : "custom" in e && typeof e.custom == "function", Ee = (e, t, n) => {
|
|
144
144
|
let r = {}, i = [], a = [];
|
|
145
145
|
for (let o of e) for (let e of o.columns) {
|
|
146
|
-
let s = `${o.name}.${e.name}`, c =
|
|
146
|
+
let s = `${o.name}.${e.name}`, c = ge(o.name, e.name, t[o.name]?.[e.name], n);
|
|
147
147
|
if (c === "error") {
|
|
148
148
|
i.push(s);
|
|
149
149
|
continue;
|
|
150
150
|
}
|
|
151
|
-
if (!
|
|
151
|
+
if (!Te(c)) {
|
|
152
152
|
a.push({
|
|
153
153
|
column: s,
|
|
154
154
|
action: c
|
|
@@ -162,7 +162,7 @@ var _e = (e) => {
|
|
|
162
162
|
unclassified: i,
|
|
163
163
|
invalid: a
|
|
164
164
|
};
|
|
165
|
-
},
|
|
165
|
+
}, De = [
|
|
166
166
|
{
|
|
167
167
|
name: "email",
|
|
168
168
|
re: /[\w.+-]+@[\w-]+\.[\w.-]+/
|
|
@@ -179,7 +179,7 @@ var _e = (e) => {
|
|
|
179
179
|
name: "creditCard",
|
|
180
180
|
re: /\b(?:\d[ -]?){13,16}\b/
|
|
181
181
|
}
|
|
182
|
-
],
|
|
182
|
+
], Oe = (e, t, n) => {
|
|
183
183
|
let r = [], i = /* @__PURE__ */ new Set();
|
|
184
184
|
for (let [a, o] of Object.entries(n)) {
|
|
185
185
|
if (!a.startsWith(`${t}.`) || o !== "keep") continue;
|
|
@@ -187,7 +187,7 @@ var _e = (e) => {
|
|
|
187
187
|
for (let a of e) {
|
|
188
188
|
let e = a[n];
|
|
189
189
|
if (typeof e != "string" || e.length === 0) continue;
|
|
190
|
-
let o =
|
|
190
|
+
let o = De.find((t) => t.re.test(e));
|
|
191
191
|
if (o && !i.has(n)) {
|
|
192
192
|
i.add(n), r.push({
|
|
193
193
|
table: t,
|
|
@@ -200,9 +200,9 @@ var _e = (e) => {
|
|
|
200
200
|
}
|
|
201
201
|
}
|
|
202
202
|
return r;
|
|
203
|
-
},
|
|
203
|
+
}, ke = (e, t, n, r, i) => {
|
|
204
204
|
let a = {};
|
|
205
|
-
for (let [o, s] of Object.entries(e)) a[o] =
|
|
205
|
+
for (let [o, s] of Object.entries(e)) a[o] = we(n[`${t}.${o}`] ?? "keep", {
|
|
206
206
|
value: s,
|
|
207
207
|
table: t,
|
|
208
208
|
column: o,
|
|
@@ -342,14 +342,14 @@ var _e = (e) => {
|
|
|
342
342
|
list: t,
|
|
343
343
|
open: (t) => T(e, t)
|
|
344
344
|
}), He = (e) => ({
|
|
345
|
-
write: (t, n, r) =>
|
|
345
|
+
write: (t, n, r) => E(e, t, n, r),
|
|
346
346
|
has: (t) => e.head(t).pipe(n.map((e) => e !== null))
|
|
347
347
|
}), Ue = (e, t) => n.tryPromise({
|
|
348
348
|
try: async () => {
|
|
349
349
|
let { createWriteStream: n } = await import("node:fs"), r = await import("node:fs/promises"), { createHash: i, randomUUID: a } = await import("node:crypto"), { pipeline: o } = await import("node:stream/promises"), { Transform: s } = await import("node:stream"), c = await import("node:path"), l = c.join(t, `.tmp-${a()}`), u = i("sha256"), d = 0, f = new s({ transform(e, t, n) {
|
|
350
350
|
u.update(e), d += e.length, n(null, e);
|
|
351
351
|
} });
|
|
352
|
-
await o(await
|
|
352
|
+
await o(await D(e.stream), f, n(l));
|
|
353
353
|
let p = u.digest("hex"), m = c.join(t, p);
|
|
354
354
|
try {
|
|
355
355
|
await r.access(m), await r.rm(l, { force: !0 });
|
|
@@ -378,7 +378,7 @@ var _e = (e) => {
|
|
|
378
378
|
expected: r.sha256,
|
|
379
379
|
actual: d
|
|
380
380
|
}));
|
|
381
|
-
}), Ge = () => typeof
|
|
381
|
+
}), Ge = () => typeof O.createZstdCompress == "function", Ke = (e) => e === "none" ? "none" : e === "gzip" ? "gzip" : Ge() ? "zstd" : "gzip", qe = (e) => e === "zstd" ? ".zst" : e === "gzip" ? ".gz" : "", Je = (e) => e === "zstd" ? O.createZstdCompress() : e === "gzip" ? O.createGzip() : new k(), Ye = (e) => e === "zstd" ? O.createZstdDecompress() : e === "gzip" ? O.createGunzip() : new k(), Xe = 1, Ze = s.Struct({
|
|
382
382
|
name: s.String,
|
|
383
383
|
file: s.String,
|
|
384
384
|
rowCount: s.Number,
|
|
@@ -646,7 +646,7 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
646
646
|
chunkSize: t.chunkSize,
|
|
647
647
|
retry: t.retry,
|
|
648
648
|
...f === void 0 ? {} : { where: f }
|
|
649
|
-
}), m = Nt(t.snapshot, o), g = m.length === 0 ? p : p.pipe(c.map((e) => Pt(e, m))), _ = t.mask ? g.pipe(c.map((e) =>
|
|
649
|
+
}), m = Nt(t.snapshot, o), g = m.length === 0 ? p : p.pipe(c.map((e) => Pt(e, m))), _ = t.mask ? g.pipe(c.map((e) => ke(e, o, t.mask.actions, t.mask.seed, s))) : g, { rowCount: v, checksum: y } = yield* ft("export", o, a, i, (e) => e.rowCount, It(Ot(_, s), t.bundleFile(d), t.compression));
|
|
650
650
|
t.ledger.tables[o] = {
|
|
651
651
|
rowCount: v,
|
|
652
652
|
checksum: y
|
|
@@ -674,7 +674,7 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
674
674
|
values: []
|
|
675
675
|
}), Bt = (t, i, a) => n.tryPromise({
|
|
676
676
|
try: () => t.transactional(async (t) => {
|
|
677
|
-
if (t.raw) for (let e of
|
|
677
|
+
if (t.raw) for (let e of me(i)) await t.raw(zt(e));
|
|
678
678
|
let o = await n.runPromiseExit(a(t));
|
|
679
679
|
if (r.isFailure(o)) throw Error(e.pretty(o.cause));
|
|
680
680
|
return o.value;
|
|
@@ -697,7 +697,7 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
697
697
|
...e.tenantTables ? { tenantTables: e.tenantTables } : {}
|
|
698
698
|
}), m = At(p.tables, e.snapshot), h = e.consistency ?? "live", _, v;
|
|
699
699
|
if (e.masking) {
|
|
700
|
-
let t = e.classification ??
|
|
700
|
+
let t = e.classification ?? N(e.snapshot), r = Ee(m.map((t) => ({
|
|
701
701
|
name: t,
|
|
702
702
|
columns: (e.snapshot.tables.find((e) => e.name === t)?.columns ?? []).map((e) => ({
|
|
703
703
|
name: e.name,
|
|
@@ -962,7 +962,7 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
962
962
|
blocking: e.map(Jt)
|
|
963
963
|
}));
|
|
964
964
|
}
|
|
965
|
-
let l = t.verify ?? !0, u = t.mode ?? "upsert", d = t.onConflict ?? "skip",
|
|
965
|
+
let l = t.verify ?? !0, u = t.mode ?? "upsert", d = t.atomic ?? u === "replace", f = t.onConflict ?? "skip", m = t.ledgerPath ?? i.join(t.bundleDir, ".import..ledger.json"), h = yield* mt(m);
|
|
966
966
|
if (u === "replace" && s.scope.kind !== "all") return yield* n.fail(new Fe({
|
|
967
967
|
mode: u,
|
|
968
968
|
reason: `replace refuses a partial bundle (scope='${s.scope.kind}') — truncating would delete rows not in the bundle. Use --mode upsert, or re-export with full scope.`
|
|
@@ -998,31 +998,38 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
998
998
|
reason: Ut(i)
|
|
999
999
|
}));
|
|
1000
1000
|
}
|
|
1001
|
-
if (t.dryRun === !0) return
|
|
1002
|
-
|
|
1001
|
+
if (t.dryRun === !0) return {
|
|
1002
|
+
manifest: s,
|
|
1003
|
+
tablesWritten: 0,
|
|
1004
|
+
rowsWritten: 0,
|
|
1005
|
+
tablesSkipped: 0,
|
|
1006
|
+
rowsSkipped: 0,
|
|
1007
|
+
fullyResumed: !1
|
|
1008
|
+
};
|
|
1009
|
+
let g = /* @__PURE__ */ new Map();
|
|
1003
1010
|
for (let e of t.targetSnapshot?.tables ?? []) {
|
|
1004
1011
|
let t = e.columns.filter((e) => e.generatedAs !== void 0).map((e) => e.name);
|
|
1005
|
-
t.length > 0 &&
|
|
1012
|
+
t.length > 0 && g.set(e.name, t);
|
|
1006
1013
|
}
|
|
1007
|
-
let
|
|
1008
|
-
let n =
|
|
1014
|
+
let _ = (e, t) => {
|
|
1015
|
+
let n = g.get(e);
|
|
1009
1016
|
if (n === void 0) return t;
|
|
1010
1017
|
let r;
|
|
1011
1018
|
for (let e of n) e in t && (r ??= { ...t }, delete r[e]);
|
|
1012
1019
|
return r ?? t;
|
|
1013
|
-
},
|
|
1014
|
-
let
|
|
1015
|
-
return u === "append" ?
|
|
1016
|
-
},
|
|
1020
|
+
}, v = (e, t, n, r = !1) => {
|
|
1021
|
+
let i = t.name, a = t.primaryKey, o = _(i, n);
|
|
1022
|
+
return u === "append" ? f === "fail" ? e.insert(i, o) : e.insertIgnore(i, o, { conflictColumns: [a] }) : u === "replace" ? r ? e.upsert(i, o, { conflictColumns: [a] }) : e.insert(i, o) : e.upsert(i, o, { conflictColumns: [a] });
|
|
1023
|
+
}, y = (e, t) => e.savepoint ? e.savepoint(t) : t(), b = (e, t, r = !0) => n.tryPromise({
|
|
1017
1024
|
try: e,
|
|
1018
1025
|
catch: (e) => Zt(e)
|
|
1019
|
-
}).pipe(n.map(() => !0), n.catchAll((e) => n.sync(() => (r && (t.reason = e), !1)))),
|
|
1026
|
+
}).pipe(n.map(() => !0), n.catchAll((e) => n.sync(() => (r && (t.reason = e), !1)))), x = (e) => {
|
|
1020
1027
|
let t = Object.entries(e.entry.columnTypes).filter(([t, n]) => n === "reference" && e.row[t] !== null && e.row[t] !== void 0).map(([e]) => e);
|
|
1021
1028
|
if (t.length !== 0) return {
|
|
1022
1029
|
...e.row,
|
|
1023
1030
|
...Object.fromEntries(t.map((e) => [e, null]))
|
|
1024
1031
|
};
|
|
1025
|
-
},
|
|
1032
|
+
}, S = (e) => {
|
|
1026
1033
|
let t = new Set(e.map((e) => String(e.row[e.entry.primaryKey]))), n = (e) => Object.entries(e.entry.columnTypes).some(([n, r]) => {
|
|
1027
1034
|
if (r !== "reference") return !1;
|
|
1028
1035
|
let i = e.row[n];
|
|
@@ -1038,22 +1045,22 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1038
1045
|
primaryCount: r.filter((e) => !e.derived).length,
|
|
1039
1046
|
totalCount: e.length
|
|
1040
1047
|
});
|
|
1041
|
-
},
|
|
1048
|
+
}, C = (e, t) => n.gen(function* () {
|
|
1042
1049
|
let r = t, i = [], a = !0;
|
|
1043
1050
|
for (; r.length > 0 && a;) {
|
|
1044
1051
|
a = !1;
|
|
1045
1052
|
let t = [];
|
|
1046
|
-
for (let n of r) (yield*
|
|
1053
|
+
for (let n of r) (yield* b(() => y(e, () => v(e, n.entry, n.row, !0)), n)) ? a = !0 : t.push(n);
|
|
1047
1054
|
if (!a && t.length > 0) for (let n = t.length - 1; n >= 0; n--) {
|
|
1048
|
-
let r = t[n], o =
|
|
1049
|
-
o !== void 0 && (yield*
|
|
1055
|
+
let r = t[n], o = x(r);
|
|
1056
|
+
o !== void 0 && (yield* b(() => y(e, () => v(e, r.entry, o, !0)), r, !1)) && (i.push(r), t.splice(n, 1), a = !0);
|
|
1050
1057
|
}
|
|
1051
1058
|
r = t;
|
|
1052
1059
|
}
|
|
1053
|
-
for (let t of i) (yield*
|
|
1054
|
-
if (r.length > 0) return yield* n.fail(
|
|
1055
|
-
}),
|
|
1056
|
-
if (u === "replace" && !(r &&
|
|
1060
|
+
for (let t of i) (yield* b(() => y(e, () => e.upsert(t.entry.name, t.row, { conflictColumns: [t.entry.primaryKey] })), t)) || (r = [...r, t]);
|
|
1061
|
+
if (r.length > 0) return yield* n.fail(S(r));
|
|
1062
|
+
}), T = !1, E = 0, D = 0, O = 0, k = 0, A = (e, r) => n.gen(function* () {
|
|
1063
|
+
if (u === "replace" && !(r && h.truncated)) {
|
|
1057
1064
|
let t = [...s.tables].reverse().map((e) => e.name), i = !r, a = e.emptyTables ? () => e.emptyTables(t) : i ? async () => {
|
|
1058
1065
|
for (let t of [...s.tables].reverse()) await e.deleteMany(t.name, { where: p(t.primaryKey) });
|
|
1059
1066
|
} : () => e.transactional(async (e) => {
|
|
@@ -1062,53 +1069,54 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1062
1069
|
yield* n.tryPromise({
|
|
1063
1070
|
try: a,
|
|
1064
1071
|
catch: (e) => new R({ reason: `emptying the target for --mode replace failed (nothing was deleted): ${Qt(e)}` })
|
|
1065
|
-
}), r && (
|
|
1072
|
+
}), r && (h.truncated = !0, yield* q(m, h));
|
|
1066
1073
|
}
|
|
1067
|
-
let a = [], o = [],
|
|
1074
|
+
let a = [], o = [], f = [], g = s.tables.length;
|
|
1068
1075
|
for (let p = 0; p < s.tables.length; p++) {
|
|
1069
|
-
let
|
|
1070
|
-
if (r &&
|
|
1071
|
-
|
|
1076
|
+
let _ = s.tables[p];
|
|
1077
|
+
if (r && _.name in h.tables) {
|
|
1078
|
+
f.push(_.name), k++, O += h.tables[_.name].rowCount, K(t.onProgress, {
|
|
1072
1079
|
phase: "import",
|
|
1073
|
-
table:
|
|
1080
|
+
table: _.name,
|
|
1074
1081
|
index: p,
|
|
1075
|
-
tableCount:
|
|
1082
|
+
tableCount: g,
|
|
1076
1083
|
event: "done",
|
|
1077
|
-
rowsDone:
|
|
1078
|
-
total:
|
|
1084
|
+
rowsDone: h.tables[_.name].rowCount,
|
|
1085
|
+
total: _.rowCount
|
|
1079
1086
|
});
|
|
1080
1087
|
continue;
|
|
1081
1088
|
}
|
|
1082
|
-
let
|
|
1089
|
+
let b = i.join(t.bundleDir, _.file);
|
|
1083
1090
|
K(t.onProgress, {
|
|
1084
1091
|
phase: "import",
|
|
1085
|
-
table:
|
|
1092
|
+
table: _.name,
|
|
1086
1093
|
index: p,
|
|
1087
|
-
tableCount:
|
|
1094
|
+
tableCount: g,
|
|
1088
1095
|
event: "start",
|
|
1089
1096
|
rowsDone: 0,
|
|
1090
|
-
total:
|
|
1097
|
+
total: _.rowCount
|
|
1091
1098
|
});
|
|
1092
|
-
let
|
|
1093
|
-
let { createHash: r } = yield* n.promise(() => import("node:crypto")), i = r("sha256"), o = 0, l = w("bundle", () =>
|
|
1099
|
+
let x = a.length, S = n.gen(function* () {
|
|
1100
|
+
let { createHash: r } = yield* n.promise(() => import("node:crypto")), i = r("sha256"), o = 0, l = w("bundle", () => j(b).pipe(Ye(s.compression))).pipe(c.tap((e) => n.sync(() => i.update(e)))), f = async (t) => {
|
|
1094
1101
|
let n = {
|
|
1095
|
-
entry:
|
|
1102
|
+
entry: _,
|
|
1096
1103
|
row: t,
|
|
1097
1104
|
reason: ""
|
|
1098
1105
|
};
|
|
1099
1106
|
try {
|
|
1100
|
-
await
|
|
1107
|
+
await y(e, () => v(e, _, t));
|
|
1101
1108
|
} catch (e) {
|
|
1102
1109
|
n.reason = Zt(e), a.push(n);
|
|
1103
1110
|
}
|
|
1104
|
-
},
|
|
1111
|
+
}, p = yield* n.promise(async () => {
|
|
1105
1112
|
let n = t.copyLoader;
|
|
1106
|
-
if (!n || n.dialect !== "postgres"
|
|
1113
|
+
if (!n || n.dialect !== "postgres") return !1;
|
|
1114
|
+
if (d) return T || (T = !0, t.onWarn?.("bulk COPY is off for this run: it needs its own connection and this import runs in ONE transaction (the default for --mode replace, so a failure mid-load cannot leave the target emptied and unfilled). Pass --no-atomic to trade that guarantee back for COPY.")), !1;
|
|
1107
1115
|
if (u === "replace") return !0;
|
|
1108
1116
|
if (u !== "upsert") return !1;
|
|
1109
1117
|
try {
|
|
1110
1118
|
return (await e.query({
|
|
1111
|
-
table:
|
|
1119
|
+
table: _.name,
|
|
1112
1120
|
predicate: void 0,
|
|
1113
1121
|
order: [],
|
|
1114
1122
|
take: 1,
|
|
@@ -1118,104 +1126,111 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1118
1126
|
} catch {
|
|
1119
1127
|
return !1;
|
|
1120
1128
|
}
|
|
1121
|
-
}),
|
|
1122
|
-
if (
|
|
1123
|
-
let e =
|
|
1124
|
-
|
|
1129
|
+
}), m = Object.keys(_.columnTypes), h = Math.max(1, t.copyBatchSize ?? 5e3), g = [], x = async () => {
|
|
1130
|
+
if (g.length === 0) return;
|
|
1131
|
+
let e = g;
|
|
1132
|
+
g = [];
|
|
1125
1133
|
try {
|
|
1126
1134
|
await t.copyLoader.copyInto({
|
|
1127
|
-
table:
|
|
1128
|
-
columns:
|
|
1129
|
-
columnTypes:
|
|
1135
|
+
table: _.name,
|
|
1136
|
+
columns: m,
|
|
1137
|
+
columnTypes: _.columnTypes,
|
|
1130
1138
|
rows: e
|
|
1131
1139
|
});
|
|
1132
1140
|
} catch (n) {
|
|
1133
|
-
|
|
1134
|
-
for (let t of e) await
|
|
1141
|
+
p = !1, t.onWarn?.(`bulk COPY into ${_.name} failed (${String(n?.message ?? n)}); falling back to per-row writes for this table`);
|
|
1142
|
+
for (let t of e) await f(t);
|
|
1135
1143
|
}
|
|
1136
|
-
},
|
|
1137
|
-
if (
|
|
1138
|
-
let t =
|
|
1139
|
-
|
|
1144
|
+
}, S = Math.max(1, t.savepointBatchSize ?? 200), C = typeof e.savepoint == "function", E = [], D = async () => {
|
|
1145
|
+
if (E.length === 0) return;
|
|
1146
|
+
let t = E;
|
|
1147
|
+
E = [];
|
|
1140
1148
|
try {
|
|
1141
1149
|
await e.savepoint(async () => {
|
|
1142
|
-
for (let n of t) await
|
|
1150
|
+
for (let n of t) await v(e, _, n);
|
|
1143
1151
|
});
|
|
1144
1152
|
} catch {
|
|
1145
|
-
for (let e of t) await
|
|
1153
|
+
for (let e of t) await f(e);
|
|
1146
1154
|
}
|
|
1147
1155
|
};
|
|
1148
|
-
return yield* c.runForEach(kt(l,
|
|
1149
|
-
if (o++,
|
|
1150
|
-
|
|
1156
|
+
return yield* c.runForEach(kt(l, _.name, _.columnTypes), (e) => n.promise(async () => {
|
|
1157
|
+
if (o++, p) {
|
|
1158
|
+
g.push(e), g.length >= h && await x();
|
|
1151
1159
|
return;
|
|
1152
1160
|
}
|
|
1153
|
-
if (
|
|
1154
|
-
|
|
1161
|
+
if (C) {
|
|
1162
|
+
E.push(e), E.length >= S && await D();
|
|
1155
1163
|
return;
|
|
1156
1164
|
}
|
|
1157
|
-
await
|
|
1158
|
-
})), yield* n.promise(
|
|
1165
|
+
await f(e);
|
|
1166
|
+
})), yield* n.promise(D), yield* n.promise(x), {
|
|
1159
1167
|
rowCount: o,
|
|
1160
1168
|
checksum: i.digest("hex")
|
|
1161
1169
|
};
|
|
1162
|
-
}), { rowCount:
|
|
1170
|
+
}), { rowCount: C, checksum: A } = yield* ft("import", _.name, p, g, (e) => e.rowCount, S);
|
|
1163
1171
|
if (l) {
|
|
1164
|
-
if (
|
|
1165
|
-
what: `table ${
|
|
1166
|
-
expected:
|
|
1167
|
-
actual:
|
|
1172
|
+
if (A !== _.checksum) return yield* n.fail(new z({
|
|
1173
|
+
what: `table ${_.name} checksum`,
|
|
1174
|
+
expected: _.checksum,
|
|
1175
|
+
actual: A
|
|
1168
1176
|
}));
|
|
1169
|
-
if (
|
|
1170
|
-
what: `table ${
|
|
1171
|
-
expected: String(
|
|
1172
|
-
actual: String(
|
|
1177
|
+
if (C !== _.rowCount) return yield* n.fail(new z({
|
|
1178
|
+
what: `table ${_.name} rowCount`,
|
|
1179
|
+
expected: String(_.rowCount),
|
|
1180
|
+
actual: String(C)
|
|
1173
1181
|
}));
|
|
1174
1182
|
}
|
|
1175
1183
|
K(t.onProgress, {
|
|
1176
1184
|
phase: "import",
|
|
1177
|
-
table:
|
|
1185
|
+
table: _.name,
|
|
1178
1186
|
index: p,
|
|
1179
|
-
tableCount:
|
|
1187
|
+
tableCount: g,
|
|
1180
1188
|
event: "done",
|
|
1181
|
-
rowsDone:
|
|
1182
|
-
total:
|
|
1183
|
-
}), r && (a.length >
|
|
1184
|
-
name:
|
|
1185
|
-
rowCount:
|
|
1186
|
-
}) : (
|
|
1187
|
-
rowCount:
|
|
1188
|
-
checksum:
|
|
1189
|
-
}, yield* q(
|
|
1189
|
+
rowsDone: C,
|
|
1190
|
+
total: _.rowCount
|
|
1191
|
+
}), D++, E += C, r && (a.length > x ? o.push({
|
|
1192
|
+
name: _.name,
|
|
1193
|
+
rowCount: C
|
|
1194
|
+
}) : (h.tables[_.name] = {
|
|
1195
|
+
rowCount: C,
|
|
1196
|
+
checksum: _.checksum
|
|
1197
|
+
}, yield* q(m, h)));
|
|
1190
1198
|
}
|
|
1191
|
-
if (a.length > 0 && (yield*
|
|
1199
|
+
if (a.length > 0 && (yield* C(e, a)), r && o.length > 0) {
|
|
1192
1200
|
for (let e of o) {
|
|
1193
1201
|
let t = s.tables.find((t) => t.name === e.name);
|
|
1194
|
-
|
|
1202
|
+
h.tables[e.name] = {
|
|
1195
1203
|
rowCount: e.rowCount,
|
|
1196
1204
|
checksum: t.checksum
|
|
1197
1205
|
};
|
|
1198
1206
|
}
|
|
1199
|
-
yield* q(
|
|
1207
|
+
yield* q(m, h);
|
|
1200
1208
|
}
|
|
1201
|
-
if (
|
|
1202
|
-
let e =
|
|
1203
|
-
t.onWarn?.(`resume: ${
|
|
1209
|
+
if (f.length > 0) {
|
|
1210
|
+
let e = f.length === s.tables.length;
|
|
1211
|
+
t.onWarn?.(`resume: ${f.length} of ${s.tables.length} table(s) were already applied by an earlier run of THIS bundle directory, so this run wrote NO rows for them${e ? " — that is every table in the bundle, so nothing was written at all" : ""}. Tables: ${f.join(", ")}. Delete ${m} to force a full re-import.`);
|
|
1204
1212
|
}
|
|
1205
1213
|
});
|
|
1206
|
-
if (
|
|
1214
|
+
if (d ? yield* n.tryPromise({
|
|
1207
1215
|
try: () => t.store.transactional(async (t) => {
|
|
1208
|
-
let i = await n.runPromiseExit(
|
|
1216
|
+
let i = await n.runPromiseExit(A(t, !1));
|
|
1209
1217
|
if (r.isFailure(i)) throw Error(e.pretty(i.cause));
|
|
1210
1218
|
}),
|
|
1211
1219
|
catch: (e) => new R({ reason: `atomic import failed (rolled back): ${String(e?.message ?? e)}` })
|
|
1212
|
-
}) : yield*
|
|
1213
|
-
let e = t.assets, r = i.join(t.bundleDir, G), a = i.join(t.bundleDir, s.assets.index ?? "assets/index.ndjson"), o = w("bundle", () =>
|
|
1214
|
-
yield* c.runForEach(o, (t) => t.key in
|
|
1215
|
-
|
|
1216
|
-
})), n.andThen(q(
|
|
1220
|
+
}) : yield* A(t.store, !0), t.assets && s.assets.index && !t.assetsAlreadyRestored) {
|
|
1221
|
+
let e = t.assets, r = i.join(t.bundleDir, G), a = i.join(t.bundleDir, s.assets.index ?? "assets/index.ndjson"), o = w("bundle", () => j(a)).pipe(c.decodeText(), c.splitLines, c.filter((e) => e.trim().length > 0), c.map((e) => JSON.parse(e)));
|
|
1222
|
+
yield* c.runForEach(o, (t) => t.key in h.assets ? n.void : We(e, r, t).pipe(n.andThen(n.sync(() => {
|
|
1223
|
+
h.assets[t.key] = !0;
|
|
1224
|
+
})), n.andThen(q(m, h))));
|
|
1217
1225
|
}
|
|
1218
|
-
return
|
|
1226
|
+
return {
|
|
1227
|
+
manifest: s,
|
|
1228
|
+
tablesWritten: D,
|
|
1229
|
+
rowsWritten: E,
|
|
1230
|
+
tablesSkipped: k,
|
|
1231
|
+
rowsSkipped: O,
|
|
1232
|
+
fullyResumed: D === 0 && k > 0
|
|
1233
|
+
};
|
|
1219
1234
|
}), sn = (e) => e, cn = (e) => {
|
|
1220
1235
|
if (e.allowLive || e.liveInstance === null || e.sameTarget === "different") return { refuse: !1 };
|
|
1221
1236
|
let t = e.targetLabel === void 0 ? "" : ` Target: ${e.targetLabel}.`, n = e.sameTarget === "same" ? " That instance is connected to the SAME database." : " We could not confirm which database that instance is connected to, so this is refused on the conservative reading — an older instance, or one whose connection could not be parsed, reports nothing to compare.";
|
|
@@ -1227,7 +1242,7 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1227
1242
|
let t = [], n = async (r) => {
|
|
1228
1243
|
let i;
|
|
1229
1244
|
try {
|
|
1230
|
-
i = await
|
|
1245
|
+
i = await ae(M(e, r), { withFileTypes: !0 });
|
|
1231
1246
|
} catch {
|
|
1232
1247
|
return;
|
|
1233
1248
|
}
|
|
@@ -1247,8 +1262,8 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1247
1262
|
async function* gn(e) {
|
|
1248
1263
|
yield new Uint8Array(Y);
|
|
1249
1264
|
for (let t of await pn(e.dir)) {
|
|
1250
|
-
let n =
|
|
1251
|
-
if (yield new Uint8Array(mn(t, r.size)), r.size > 0) for await (let e of
|
|
1265
|
+
let n = M(e.dir, t), r = await oe(n);
|
|
1266
|
+
if (yield new Uint8Array(mn(t, r.size)), r.size > 0) for await (let e of j(n)) yield new Uint8Array(e);
|
|
1252
1267
|
}
|
|
1253
1268
|
if (e.blobs) for await (let t of e.blobs.list()) {
|
|
1254
1269
|
yield new Uint8Array(hn(t.key, t.contentType, t.size));
|
|
@@ -1263,8 +1278,8 @@ async function* gn(e) {
|
|
|
1263
1278
|
yield new Uint8Array(Buffer.from([ln]));
|
|
1264
1279
|
}
|
|
1265
1280
|
var _n = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), vn = (e) => new Promise((t) => e.end(t)), yn = async (e, t) => {
|
|
1266
|
-
let n =
|
|
1267
|
-
await
|
|
1281
|
+
let n = ue(t.dir);
|
|
1282
|
+
await ie(n, { recursive: !0 });
|
|
1268
1283
|
let r = Buffer.alloc(0), i = e[Symbol.asyncIterator](), a = !1, o = async () => {
|
|
1269
1284
|
let { value: e, done: t } = await i.next();
|
|
1270
1285
|
return t ? (a = !0, !1) : (r = r.length === 0 ? Buffer.from(e) : Buffer.concat([r, Buffer.from(e)]), !0);
|
|
@@ -1286,10 +1301,10 @@ var _n = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), vn
|
|
|
1286
1301
|
await s(e);
|
|
1287
1302
|
let t = c(e).toString("utf8");
|
|
1288
1303
|
await s(8);
|
|
1289
|
-
let i = c(8).readBigUInt64BE(0), a =
|
|
1290
|
-
if (a !== n && !a.startsWith(n +
|
|
1291
|
-
await
|
|
1292
|
-
let o =
|
|
1304
|
+
let i = c(8).readBigUInt64BE(0), a = ue(n, t);
|
|
1305
|
+
if (a !== n && !a.startsWith(n + de)) throw Error(`archive path '${t}' escapes the destination`);
|
|
1306
|
+
await ie(le(a), { recursive: !0 });
|
|
1307
|
+
let o = te(a);
|
|
1293
1308
|
for (; i > 0n;) {
|
|
1294
1309
|
r.length === 0 && await s(1);
|
|
1295
1310
|
let e = i < BigInt(r.length) ? Number(i) : r.length;
|
|
@@ -1306,7 +1321,7 @@ var _n = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), vn
|
|
|
1306
1321
|
await s(i);
|
|
1307
1322
|
let a = c(i).toString("utf8");
|
|
1308
1323
|
await s(8);
|
|
1309
|
-
let o = c(8).readBigUInt64BE(0), l = await t.blobs.has(n), u = y("sha256"), d = new
|
|
1324
|
+
let o = c(8).readBigUInt64BE(0), l = await t.blobs.has(n), u = y("sha256"), d = new k(), f = l ? Promise.resolve() : t.blobs.write(n, d, {
|
|
1310
1325
|
contentType: a,
|
|
1311
1326
|
size: Number(o)
|
|
1312
1327
|
}), p = o;
|
|
@@ -1364,14 +1379,14 @@ var _n = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), vn
|
|
|
1364
1379
|
return {
|
|
1365
1380
|
has: async (e) => t.has(e),
|
|
1366
1381
|
write: async (n, r, i) => {
|
|
1367
|
-
let a = await import("node:fs/promises"), { pipeline: o } = await import("node:stream/promises"), { Transform: s } = await import("node:stream"), c =
|
|
1368
|
-
await
|
|
1382
|
+
let a = await import("node:fs/promises"), { pipeline: o } = await import("node:stream/promises"), { Transform: s } = await import("node:stream"), c = M(e, "assets");
|
|
1383
|
+
await ie(c, { recursive: !0 });
|
|
1369
1384
|
let l = y("sha256"), u = 0, d = new s({ transform(e, t, n) {
|
|
1370
1385
|
l.update(e), u += e.length, n(null, e);
|
|
1371
|
-
} }), f =
|
|
1372
|
-
await o(r, d,
|
|
1386
|
+
} }), f = M(c, `.tmp-${Date.now()}-${t.size}`);
|
|
1387
|
+
await o(r, d, te(f));
|
|
1373
1388
|
let p = l.digest("hex");
|
|
1374
|
-
await a.rename(f,
|
|
1389
|
+
await a.rename(f, M(c, p)), await a.appendFile(M(c, "index.ndjson"), `${JSON.stringify({
|
|
1375
1390
|
key: n,
|
|
1376
1391
|
sha256: p,
|
|
1377
1392
|
size: u,
|
|
@@ -1379,7 +1394,7 @@ var _n = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), vn
|
|
|
1379
1394
|
})}\n`), t.add(n);
|
|
1380
1395
|
}
|
|
1381
1396
|
};
|
|
1382
|
-
}, Cn =
|
|
1397
|
+
}, Cn = fe(S), wn = Buffer.from("VENC1\0\0\0"), Tn = 32, En = 16, Dn = 7, X = 16, On = 262144, Z = {
|
|
1383
1398
|
N: 32768,
|
|
1384
1399
|
r: 8,
|
|
1385
1400
|
p: 1
|
|
@@ -1457,7 +1472,7 @@ var Nn = (e, t) => ({
|
|
|
1457
1472
|
} };
|
|
1458
1473
|
},
|
|
1459
1474
|
open: (t) => ({ async *[Symbol.asyncIterator]() {
|
|
1460
|
-
let r = await n.runPromise(T(e, t)), i = await
|
|
1475
|
+
let r = await n.runPromise(T(e, t)), i = await D(r.stream);
|
|
1461
1476
|
for await (let e of i) yield e instanceof Uint8Array ? e : new Uint8Array(e);
|
|
1462
1477
|
} })
|
|
1463
1478
|
}), Pn = (e, t) => Nn(e, () => h(t, {
|
|
@@ -1470,8 +1485,8 @@ var Nn = (e, t) => ({
|
|
|
1470
1485
|
})))), Fn = (e) => ({
|
|
1471
1486
|
has: (t) => n.runPromise(e.head(t).pipe(n.map((e) => e !== null))),
|
|
1472
1487
|
write: async (t, r, i) => {
|
|
1473
|
-
let a = w("storage", () =>
|
|
1474
|
-
await n.runPromise(
|
|
1488
|
+
let a = w("storage", () => A.from(r));
|
|
1489
|
+
await n.runPromise(E(e, t, a, {
|
|
1475
1490
|
contentType: i.contentType,
|
|
1476
1491
|
totalSize: i.size
|
|
1477
1492
|
}));
|
|
@@ -1479,7 +1494,7 @@ var Nn = (e, t) => ({
|
|
|
1479
1494
|
}), In = (e, t) => {
|
|
1480
1495
|
let n = t === "dump" ? "mariadb-dump" : "mariadb", r = t === "dump" ? "mysqldump" : "mysql";
|
|
1481
1496
|
return e === "mariadb" && Ln(n) ? n : r;
|
|
1482
|
-
}, Ln = (e) => (process.env.PATH ?? "").split(
|
|
1497
|
+
}, Ln = (e) => (process.env.PATH ?? "").split(ce).filter((e) => e !== "").some((t) => ne(M(t, e))), Rn = (e) => {
|
|
1483
1498
|
switch (e) {
|
|
1484
1499
|
case "postgres": return "db.dump";
|
|
1485
1500
|
case "mysql":
|
|
@@ -1680,12 +1695,12 @@ var Nn = (e, t) => ({
|
|
|
1680
1695
|
});
|
|
1681
1696
|
}
|
|
1682
1697
|
}, Un = (e) => e.kind === "copy" ? n.async((t) => {
|
|
1683
|
-
|
|
1698
|
+
ee(e.from, e.to, (r) => t(r ? n.fail(new V({
|
|
1684
1699
|
tool: "copy",
|
|
1685
1700
|
reason: `copy ${e.from} → ${e.to}: ${r.message}`
|
|
1686
1701
|
})) : n.void));
|
|
1687
1702
|
}) : n.async((t) => {
|
|
1688
|
-
let r =
|
|
1703
|
+
let r = pe(e.tool, [...e.args], {
|
|
1689
1704
|
env: {
|
|
1690
1705
|
...process.env,
|
|
1691
1706
|
...e.env
|
|
@@ -1698,10 +1713,10 @@ var Nn = (e, t) => ({
|
|
|
1698
1713
|
}), i = "";
|
|
1699
1714
|
r.stderr?.on("data", (e) => {
|
|
1700
1715
|
i += String(e);
|
|
1701
|
-
}), e.stdoutFile && r.stdout && r.stdout.pipe(
|
|
1716
|
+
}), e.stdoutFile && r.stdout && r.stdout.pipe(te(e.stdoutFile)), e.stdinFile && r.stdin && j(e.stdinFile).pipe(r.stdin);
|
|
1702
1717
|
let a = () => {
|
|
1703
1718
|
if (e.stdoutFile !== void 0) try {
|
|
1704
|
-
|
|
1719
|
+
re(e.stdoutFile);
|
|
1705
1720
|
} catch {}
|
|
1706
1721
|
};
|
|
1707
1722
|
r.on("error", (r) => {
|
|
@@ -1716,7 +1731,7 @@ var Nn = (e, t) => ({
|
|
|
1716
1731
|
}
|
|
1717
1732
|
a(), t(n.fail(new V({
|
|
1718
1733
|
tool: e.tool,
|
|
1719
|
-
reason: `exited with code ${r}` + (e.stdoutFile === void 0 ? "" : ` — the partial ${
|
|
1734
|
+
reason: `exited with code ${r}` + (e.stdoutFile === void 0 ? "" : ` — the partial ${se(e.stdoutFile)} was removed, so it cannot be mistaken for a backup`),
|
|
1720
1735
|
stderr: i.slice(-2e3)
|
|
1721
1736
|
})));
|
|
1722
1737
|
});
|
|
@@ -1724,7 +1739,7 @@ var Nn = (e, t) => ({
|
|
|
1724
1739
|
let t = yield* Be(e.scope ?? { kind: "all" }, e.snapshot, {
|
|
1725
1740
|
store: e.store,
|
|
1726
1741
|
...e.tenantTables ? { tenantTables: e.tenantTables } : {}
|
|
1727
|
-
}), r = At(t.tables, e.snapshot), i = e.classification ??
|
|
1742
|
+
}), r = At(t.tables, e.snapshot), i = e.classification ?? N(e.snapshot), a = e.sampleSize ?? 20, o = Ee(r.map((t) => ({
|
|
1728
1743
|
name: t,
|
|
1729
1744
|
columns: (e.snapshot.tables.find((e) => e.name === t)?.columns ?? []).map((e) => ({
|
|
1730
1745
|
name: e.name,
|
|
@@ -1736,8 +1751,8 @@ var Nn = (e, t) => ({
|
|
|
1736
1751
|
table: i,
|
|
1737
1752
|
chunkSize: a,
|
|
1738
1753
|
...r === void 0 ? {} : { where: r }
|
|
1739
|
-
}).pipe(c.take(a), c.runCollect, n.map((e) => [...e])), d = u.map((t) =>
|
|
1740
|
-
l.push(...
|
|
1754
|
+
}).pipe(c.take(a), c.runCollect, n.map((e) => [...e])), d = u.map((t) => ke(t, i, o.actions, e.masking.seed));
|
|
1755
|
+
l.push(...Oe(d, i, o.actions));
|
|
1741
1756
|
let f = u[0], p = d[0], m = [];
|
|
1742
1757
|
if (f && p) for (let [e, t] of Object.entries(o.actions)) {
|
|
1743
1758
|
if (!e.startsWith(`${i}.`) || t === "keep") continue;
|
|
@@ -1825,4 +1840,4 @@ var Nn = (e, t) => ({
|
|
|
1825
1840
|
return n.push(""), n.push(` discarded: ${r("value-dropped")} defaulted: ${r("filled-by-default")} refused: ${r("refuse")}`), e.refuse && (n.push(""), n.push(" REFUSED — no row was written. These are the differences that would make"), n.push(" imported rows WRONG rather than merely incomplete.")), n;
|
|
1826
1841
|
};
|
|
1827
1842
|
//#endregion
|
|
1828
|
-
export { G as ASSETS_DIR, at as ASSET_INDEX_FILE, R as BundleError, Ae as CodecError, je as CompressionError, Ie as CrossDialectError, it as DATA_DIR,
|
|
1843
|
+
export { G as ASSETS_DIR, at as ASSET_INDEX_FILE, R as BundleError, Ae as CodecError, je as CompressionError, Ie as CrossDialectError, it as DATA_DIR, he as DEFAULT_CLASS_ACTIONS, Xe as FORMAT_VERSION, Fe as ImportModeError, z as IntegrityError, ot as LEDGER_FILE, W as MANIFEST_FILE, Pe as MaskingError, V as NativeToolError, Me as RowsRefusedError, rn as SAVEPOINT_BATCH_SIZE, Ne as SchemaDriftError, B as ScopeError, we as applyAction, cn as assessLiveImport, gt as assetDone, Rn as backupArtifactName, Vn as backupCommand, Wt as canonicalType, qt as checkPortability, nn as checkSchemaDrift, N as classificationFromSnapshot, Kn as classifyImportDrift, qe as compressionExt, Re as computeSubsetIds, nt as decodeManifest, Et as decodeRow, wt as decodeValue, Mn as decryptStream, sn as defineDataProfile, tn as diffSnapshots, Sn as dirBlobSink, pt as emptyLedger, tt as encodeManifest, Tt as encodeRow, St as encodeValue, jn as encryptStream, qn as formatImportDrift, Jt as formatIssue, Ge as hasZstd, Je as makeCompressor, Ye as makeDecompressor, ke as maskRow, kt as ndjsonToRows, Ut as outsideReferenceRefusal, Ht as outsideReferencesInto, gn as packBundle, bn as peekBundle, Ke as pickCompression, Ee as planMasking, Gn as previewMasking, mt as readLedger, ge as resolveAction, Be as resolveScope, We as restoreAssetFromCas, Hn as restoreCommand, Ot as rowsToNdjson, Vt as runExport, on as runImport, Un as runNativeStep, Oe as scanForLeaks, He as storageAssetSink, Ve as storageAssetSource, Fn as storageBlobSink, Nn as storageBlobSource, Pn as storageRefsBlobSource, ze as subsetToScope, ht as tableDone, At as topoSortTables, yn as unpackBundle, xn as unpackBundleToDir, Ue as writeAssetToCas, q as writeLedger };
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@voltro/data-transfer",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.44.0",
|
|
4
4
|
"description": "Portable export/import for a Voltro app's data + assets — typed-NDJSON row codec, content-addressed assets, framed compression, a resumable chunk ledger, and FK-ordered idempotent import. Dialect-agnostic; the resilient backbone under `voltro data export|import` and native backups.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"voltro",
|
|
@@ -33,8 +33,8 @@
|
|
|
33
33
|
"node": ">=24.0.0"
|
|
34
34
|
},
|
|
35
35
|
"dependencies": {
|
|
36
|
-
"@voltro/database": "0.
|
|
37
|
-
"@voltro/plugin-storage": "0.
|
|
36
|
+
"@voltro/database": "0.44.0",
|
|
37
|
+
"@voltro/plugin-storage": "0.44.0"
|
|
38
38
|
},
|
|
39
39
|
"peerDependencies": {
|
|
40
40
|
"effect": "^3.22.0"
|