@voltro/data-transfer 0.76.0 → 0.77.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +55 -0
- package/dist/index.js +62 -55
- package/package.json +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -39,6 +39,61 @@ _Changes staged for the next release accumulate here (rolled up from
|
|
|
39
39
|
|
|
40
40
|
---
|
|
41
41
|
|
|
42
|
+
## [0.77.0] — 2026-09-22
|
|
43
|
+
|
|
44
|
+
### ⚠ BREAKING
|
|
45
|
+
|
|
46
|
+
- **Flow run rows record a cancellation request as `cancelling` until confirmed** — `@voltro/plugin-ai-flows`, `@voltro/devtools-ui`, `@voltro/cli`
|
|
47
|
+
|
|
48
|
+
`cancelFlow` used to write the flow row to `cancelled` before asking the engine, so a receipt reporting `engine: 'unreachable'` or `'unknown'` sat next to a row that read as a stopped run, and a reader of the row alone could not tell the difference. `RunStatus` gains `cancelling`: a live row now records the request as `cancelling` with `cancelledAt` as the request time and reads `cancelled` only once the engine confirmed — through the receipt's `engine: 'cancelled'`, the engine honouring the request at its next step boundary, its own run row turning cancelled, or a repeated confirmed request. Terminal rows are never resubmitted. The plugin documentation now states what a UI renders per `engine` outcome. Migration: extend exhaustive `RunStatus` switches and any filter that treated `cancelled` as the request marker; render `cancelling` as an open request and offer a retry when `engine` was `unreachable` or `unknown`. Only `succeeded`, `failed` and `cancelled` are terminal; `useFlowRun` reports `done` for those alone.
|
|
49
|
+
|
|
50
|
+
**`voltro update` carries you across this** — codemod `0.77.0/01_flow-cancelling-row-status`. If you pin versions by hand and never run it, print the notes without changing anything: `voltro update --codemods-only --from <your current version> --dry-run` (this one ships in 0.77.0).
|
|
51
|
+
|
|
52
|
+
### Added
|
|
53
|
+
|
|
54
|
+
- **An `@source` glob that matches no file is now reported** — `@voltro/cli`
|
|
55
|
+
|
|
56
|
+
Tailwind generates a utility only when it finds the class in a scanned file, and a `@source` pointing at a directory that does not exist is not an error — it is a set with no members. The build stays green and the page renders with classes that name rules which were never generated. `voltro build` now warns for every `@source` glob that matches nothing, and `voltro doctor` reports the same. The usual cause is scanning a dependency at a path it does not publish: a package that ships only its bundle has no `src/` once installed, so `@source "../node_modules/<pkg>/src/**"` matches everything in a workspace (where the dependency is a symlink to its source tree) and nothing in CI. Scan the one path the package publishes — `@source "../node_modules/<pkg>/dist/*.js"` — and rebuild that dependency after editing it, since its bundle is what you scan.
|
|
57
|
+
|
|
58
|
+
### Fixed
|
|
59
|
+
|
|
60
|
+
- **A failing admission placement no longer holds the drain at one tick per second** — `@voltro/cli`
|
|
61
|
+
|
|
62
|
+
Physical workflow admission drains every placement on a coordinated tick and backs off to a five-second reconciliation bound when a tick finds nothing to admit. A placement whose recovery failed on every tick — a release census page that threw, a placement that could not be opened — kept the tick at its base interval instead: the failure counted as "not idle", the interval never left one second, and the process wrote one schedule claim per second for as long as the condition lasted, with `scanned: 0` on every drain and every run terminal. Such a tick now reports `idle: true` beside its `placementFailures`, `discoveryProblems` and `releaseFailures`, backs off like a healthy idle tick, and retries the failing placement at the reconciliation bound; it is never disarmed, and the `physical recovery incomplete` warning still names the placement, the phase and the database's own reason. Incomplete discovery is unchanged: a further page is known work.
|
|
63
|
+
|
|
64
|
+
Two causes of a permanently failing release census are closed with it. The census pager compared row identities in code-unit order to detect a stalled cursor, while the store had ordered and cursor-filtered the page under its own collation; a correctly ordered page could be refused wholesale, and with it every global pool permit release behind it. The pager now walks a page as the store returns it and refuses only a page that repeats its cursor row or an identity within itself, naming both identities. And a census page that throws no longer pins its cursor: the next tick restarts that placement's census from the beginning, so an owner released behind the cursor since is visited again rather than waiting behind a page that fails on every read.
|
|
65
|
+
- **A file migration body runs under test without a database** — `@voltro/database`
|
|
66
|
+
|
|
67
|
+
`makeFileMigrationContext` accepts an in-memory store without a `sql` client, so a migration's `each`, `rewrite` (temporal guards included), `expect`, `skipUnless`, `store`, `log` and a Promise body under `transactional` run in a unit test against the framework's own in-memory store. The surfaces that genuinely need a connection — `raw()`, every `ctx.schema` operation, an Effect body under `transactional` — fail with the new typed `MigrationNeedsSqlClient`, whose message names the operation and what is missing. A placeholder handed in as `sql`, or a SQL-backed store arriving without its client, is refused at construction with the same error. Previously `rewrite` introspected the live schema unconditionally and a database-free test died with `TypeError: onDialectOrElse is not a function` from inside the bundle. `FileMigrationContextInput` is now the union of `FileMigrationContextLiveInput` (client + store) and `FileMigrationContextInMemoryInput` (a store without `run`); the runner's call shape is unchanged.
|
|
68
|
+
- **`voltro data restore` accepts an empty target and names a takeable remedy** — `@voltro/cli`, `@voltro/data-transfer`, `@voltro/database`
|
|
69
|
+
|
|
70
|
+
A native restore into a freshly created database was refused because the in-progress marker's table did not exist there, and the refusal said to run `voltro db apply` first — which on that target installs the current declared schema, the very state the artefact is about to replace. The restore now creates that one table itself, index- and trigger-free like the migration ledgers, and writes the marker into it before the first destructive statement; a refusal that remains carries the driver's own reason. `--no-marker` now means no marker on every target, warns every time, and no longer reports afterwards that "the restore artefact removed the in-progress marker and it could NOT be written back" over a run that never wrote one. On sqlite and turso the restore closes its bookkeeping connection before the file copy, removes the replaced file's `-wal` / `-shm` sidecars with it, and reopens bookkeeping afterwards: writes through the old connection landed the old database's WAL frames beside the new file, which SQLite replayed over it on the next open.
|
|
71
|
+
- **Rolling-deploy advisory skips tables the same plan creates** — `@voltro/database`, `@voltro/cli`
|
|
72
|
+
|
|
73
|
+
`voltro db plan` and `db apply` no longer flag an operation as unsafe under a rolling deploy when the table it targets is created by the same plan. A fresh deployment against an empty database used to print one "old instances may write values … the new CHECK rejects" line per enum-membership CHECK, because the classifier judged each operation alone and never asked whether the plan's own `create-table` list held the table. No instance on a previous release can touch a table that did not exist a moment earlier, so the rule covers every kind — CHECK, foreign key, unique constraint, `NOT NULL`, type change, JSON validation — not only the one reported. A mixed plan still flags the same operations on pre-existing tables. Every remaining verdict now names its premise (`instances on the previous release …`), and the JSON warning carries a `premise` field, so a deployment with no previous release can tell from the message alone that the advisory does not apply. `classifyRollingDeploySafety` accepts an optional plan context (`{ createdTables }`); `createdTablesOf` builds it from a plan.
|
|
74
|
+
- **The precompiled serve bundle boots with a trace exporter on** — `@voltro/cli`
|
|
75
|
+
|
|
76
|
+
`voltro serve` from the bundle `voltro build` writes died at boot whenever a trace exporter was active — `FRAMEWORK_TRACING=console` with `ConsoleSpanExporter is not a constructor`, an `OTEL_EXPORTER_OTLP_ENDPOINT` with `OTLPTraceExporter is not a constructor` — while the ring-only serve, dev and every test path booted fine. The OpenTelemetry SDK packages are CommonJS and are reached through a dynamic `import()`; the bundler splits such an import into a chunk that exposes only `default`, so every named member read off that namespace was `undefined`. The production bundler now routes a dynamic import of an inlined CommonJS package through an ESM facade that carries the same named exports Node gives it, so the console and OTLP exporters, the metrics readers and any other lazily imported CommonJS package construct as they do under development. Both exporters are now booted from a relocated production bundle in the bundle test matrix, and each has to deliver a span.
|
|
77
|
+
|
|
78
|
+
The facade has a measured cost, split without remainder against the previous release: the web start bundle grows 14 901 B (0.87 %) and the api serve bundle 28 552 B (0.34 %), both from the re-exported CommonJS surfaces; no file count moves. The remaining growth of the same release (22 B and 35 893 B) is ordinary source change. The artefact budgets are re-pinned to the measured sizes with the same 15 % headroom.
|
|
79
|
+
- **A store failure carries the sentence that explains it** — `@voltro/protocol`, `@voltro/database`, `@voltro/runtime`
|
|
80
|
+
|
|
81
|
+
`StoreOperationFailed.cause` reduced every error to its class name, so an eager load on a table with no relations registered arrived as `cause: "Error"` while the refusal's own sentence — naming the table, the missing `relations(...)` declaration and the likely fix — was only recoverable by unwrapping the value by hand. A sentence the framework authored now travels whole in `cause`; anything a driver raised is still reduced to its class and code, because the driver's text carries row data and socket addresses. The error's `message` is now one readable sentence (`query on 'x' failed: …`) instead of a JSON rendering of its fields, so a test reporter, `Cause.pretty` and the undeclared-error envelope all show the diagnosis. And when a statement fails and takes the connection with it, `EffectStore.transactional` now reports the statement's failure (with its code) instead of the rollback's uncoded error that used to replace it. `@voltro/database` exports `authoredError` / `authoredMessageOf` for a hand-written store that wants the same treatment.
|
|
82
|
+
- **A boot missing `@effect/opentelemetry` refuses once and names its cause** — `@voltro/runtime`, `@voltro/cli`
|
|
83
|
+
|
|
84
|
+
`@effect/opentelemetry` has been an optional peer of `@voltro/runtime` since 0.33.0, and the note that made it one said to skip the install unless traces or metrics are exported. That scope was too narrow: the tracer layer loads the OpenTelemetry SDK whenever a trace sink is installed, and `voltro dev` arms its in-process trace ring by default — so every `voltro dev` needs the package with no tracing variable set, and `voltro serve` needs it when its inspect trace ring or an exporter is on. Measured against the published 0.75.0 and 0.76.0 with the package absent: identical refusals, unrelated to `VOLTRO_TRACING_PERSIST`. Three things change. The refusal is now a marked boot refusal: printed once without a stack on every boot path, listing each cause that kept the layer active with the switch that removes it (the previous message named `FRAMEWORK_TRACING=off` for a ring that variable does not touch). `voltro doctor` reports `@effect/opentelemetry is not installed` when the installed runtime cannot see it, using the runtime's own lookup path, and fails the run unless `VOLTRO_TRACING_BUFFER=off` is set. And the corrected scope is re-issued as `0.77.0/03_tracing-peer`, printed for every project because tracing is configured outside the repository; the codemod text scan now also reads tracked `.env*` files, where those switches live. The golden churn is `otelImportFailure` gaining an optional second parameter, which no existing call can fail to compile against.
|
|
85
|
+
- **TypeScript 7 upgrade note: typed ESLint alias pair and removed tsconfig options** — `@voltro/cli`
|
|
86
|
+
|
|
87
|
+
The 0.76.0 dependency note recommended TypeScript `^7.0.2` and read as though a plain bump were sufficient. It is not for a workspace that lints with typescript-eslint: the current release line declares the peer `"typescript": ">=4.8.4 <6.1.0"` and exits 2 on load under TypeScript 7, before linting anything. TypeScript 7 also removed `"moduleResolution": "node"` (TS5108) and `baseUrl` (TS5102), which the note did not mention. A note under an already-published version reaches nobody who acted on it, so the corrected guidance is re-issued under `0.77.0/02_typescript7-side-by-side`: the side-by-side alias pair (`"@typescript/native": "npm:typescript@^7.0.2"`, `"typescript": "npm:@typescript/typescript6@^6.0.2"`) that keeps `tsc` on 7.0 while typed ESLint resolves the 6.0 API, the two removals with their replacements, and when a plain bump is fine. Gated on any `typescript` declaration or removed option, so it reaches workspaces that bumped, reverted, or have not bumped yet.
|
|
88
|
+
|
|
89
|
+
### Internal (no consumer-facing effect)
|
|
90
|
+
|
|
91
|
+
- **A release moves `latest` only once every tarball is downloadable** — `@voltro/cli`
|
|
92
|
+
|
|
93
|
+
npm writes a new version into a package's metadata before the tarball behind it is downloadable from the CDN, and a consumer installing on `latest` inside that window sees a resolvable version whose download answers 404. The release now holds the promotion of `latest` until every published tarball answers 200 from the URL its own metadata publishes, and refuses to promote — leaving the previous release in place — if one does not within a bounded wait.
|
|
94
|
+
|
|
95
|
+
---
|
|
96
|
+
|
|
42
97
|
## [0.76.0] — 2026-09-21
|
|
43
98
|
|
|
44
99
|
### ⚠ BREAKING
|
package/dist/index.js
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
import { Cause as e, Data as t, Effect as n, Exit as r, Metric as i, MetricBoundaries as a, Runtime as o, Schedule as s, Schema as c, Stream as l } from "effect";
|
|
2
2
|
import { DATA_TRANSFERS_TABLE as u, REPLACE_IN_PROGRESS_TABLE as d, STAGING_TABLE_PREFIX as f, _voltroDataTransfersTable as p, classifyConstraintViolation as m, derivedRowId as h, ensureTableRegistered as g, eq as _, extractDbCause as v, getTable as y, inSet as b, isNotNull as x, or as S, recordsTable as C, registerTable as ee, streamTable as w, suspendWriteRecorders as te, suspendWriteRecordersEffect as ne, unregisterTable as re } from "@voltro/database";
|
|
3
3
|
import { fingerprintSchema as T } from "@voltro/database/sql";
|
|
4
|
-
import { createCipheriv as
|
|
5
|
-
import { copyFile as
|
|
4
|
+
import { createCipheriv as ie, createDecipheriv as ae, createHash as oe, createHmac as se, randomBytes as ce, randomUUID as le, scrypt as ue } from "node:crypto";
|
|
5
|
+
import { copyFile as E, createReadStream as de, createWriteStream as D, existsSync as fe, rename as pe, statSync as O, unlinkSync as k } from "node:fs";
|
|
6
6
|
import { STORAGE_REFS_TABLE as A, fromNodeReadable as me, getObjectStream as j, putObjectStream as he, toNodeReadable as M } from "@voltro/plugin-storage";
|
|
7
7
|
import * as N from "node:zlib";
|
|
8
8
|
import { PassThrough as P, Readable as ge } from "node:stream";
|
|
@@ -57,7 +57,7 @@ var Ce = (e) => {
|
|
|
57
57
|
return r.classes?.[e] ?? we[e] ?? "redact";
|
|
58
58
|
}
|
|
59
59
|
return n?.safe ? r.onSafe ?? "keep" : n?.columnType === "id" || n?.columnType === "reference" ? "keep" : r.onUnclassified ?? "error";
|
|
60
|
-
}, De = (e, t) =>
|
|
60
|
+
}, De = (e, t) => se("sha256", e).update(t == null ? "∅" : String(t)).digest(), V = (e, t) => e.readUInt32BE(t * 4 % (e.length - 4)), Oe = [
|
|
61
61
|
"Ada",
|
|
62
62
|
"Alan",
|
|
63
63
|
"Grace",
|
|
@@ -566,7 +566,7 @@ var Ce = (e) => {
|
|
|
566
566
|
objectBytes: o,
|
|
567
567
|
missing: u
|
|
568
568
|
};
|
|
569
|
-
}), ht = (e) => me("bundle", () =>
|
|
569
|
+
}), ht = (e) => me("bundle", () => de(e)).pipe(l.decodeText(), l.splitLines, l.filter((e) => e.trim().length > 0), l.map((e) => JSON.parse(e)), l.mapError((t) => t instanceof G ? t : new G({
|
|
570
570
|
reason: `asset index unreadable: ${String(t)}`,
|
|
571
571
|
path: e
|
|
572
572
|
}))), gt = (e) => n.gen(function* () {
|
|
@@ -1442,20 +1442,20 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1442
1442
|
for (let e of n) e in t && (r ??= { ...t }, delete r[e]);
|
|
1443
1443
|
return r ?? t;
|
|
1444
1444
|
}, T = (e, t, n, r = !1) => {
|
|
1445
|
-
let i =
|
|
1445
|
+
let i = O ? X(t.name) : t.name, a = t.primaryKey, o = w(t.name, n);
|
|
1446
1446
|
return f === "append" ? _ === "fail" ? e.insert(i, o) : e.insertIgnore(i, o, { conflictColumns: [a] }) : f === "replace" ? r ? e.upsert(i, o, { conflictColumns: [a] }) : e.insert(i, o) : e.upsert(i, o, { conflictColumns: [a] });
|
|
1447
|
-
},
|
|
1448
|
-
r.unclassified && (
|
|
1449
|
-
},
|
|
1447
|
+
}, ie = (e, t) => e.savepoint ? e.savepoint(t) : t(), ae = 0, oe = (t, n, r) => {
|
|
1448
|
+
r.unclassified && (ae++, !(ae > 3) && e.onWarn?.(`row refused with no driver detail — table '${t.name}', row '${String(n.id ?? "?")}'. The full rendering follows so the LAYER that refused is on the record; it is logged, never returned. ${r.raw}` + (ae === 3 ? " (further unclassified refusals are counted, not logged)" : "")));
|
|
1449
|
+
}, se = (e, t, r = !0) => n.tryPromise({
|
|
1450
1450
|
try: e,
|
|
1451
1451
|
catch: (e) => On(e)
|
|
1452
|
-
}).pipe(n.map(() => !0), n.catchAll((e) => n.sync(() => (r && (t.reason = e.reason,
|
|
1452
|
+
}).pipe(n.map(() => !0), n.catchAll((e) => n.sync(() => (r && (t.reason = e.reason, oe(t.entry, t.row, e)), !1)))), ce = (e) => {
|
|
1453
1453
|
let t = Object.entries(e.entry.columnTypes).filter(([t, n]) => n === "reference" && e.row[t] !== null && e.row[t] !== void 0).map(([e]) => e);
|
|
1454
1454
|
if (t.length !== 0) return {
|
|
1455
1455
|
...e.row,
|
|
1456
1456
|
...Object.fromEntries(t.map((e) => [e, null]))
|
|
1457
1457
|
};
|
|
1458
|
-
},
|
|
1458
|
+
}, le = (e) => {
|
|
1459
1459
|
let t = new Set(e.map((e) => String(e.row[e.entry.primaryKey]))), n = (e) => Object.entries(e.entry.columnTypes).some(([n, r]) => {
|
|
1460
1460
|
if (r !== "reference") return !1;
|
|
1461
1461
|
let i = e.row[n];
|
|
@@ -1483,32 +1483,32 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1483
1483
|
primary: t.primary
|
|
1484
1484
|
})).sort((e, t) => t.refused - e.refused || e.table.localeCompare(t.table)).slice(0, 100)
|
|
1485
1485
|
});
|
|
1486
|
-
},
|
|
1486
|
+
}, ue = (e, t) => n.gen(function* () {
|
|
1487
1487
|
let r = t, i = [], a = !0;
|
|
1488
1488
|
for (; r.length > 0 && a;) {
|
|
1489
1489
|
a = !1;
|
|
1490
1490
|
let t = [];
|
|
1491
|
-
for (let n of r) (yield*
|
|
1491
|
+
for (let n of r) (yield* se(() => ie(e, () => T(e, n.entry, n.row, !0)), n)) ? a = !0 : t.push(n);
|
|
1492
1492
|
if (!a && t.length > 0) for (let n = t.length - 1; n >= 0; n--) {
|
|
1493
|
-
let r = t[n], o =
|
|
1494
|
-
o !== void 0 && (yield*
|
|
1493
|
+
let r = t[n], o = ce(r);
|
|
1494
|
+
o !== void 0 && (yield* se(() => ie(e, () => T(e, r.entry, o, !0)), r, !1)) && (i.push(r), t.splice(n, 1), a = !0);
|
|
1495
1495
|
}
|
|
1496
1496
|
r = t;
|
|
1497
1497
|
}
|
|
1498
|
-
for (let t of i) (yield*
|
|
1499
|
-
if (r.length > 0) return yield* n.fail(
|
|
1500
|
-
}),
|
|
1498
|
+
for (let t of i) (yield* se(() => ie(e, () => e.upsert(t.entry.name, t.row, { conflictColumns: [t.entry.primaryKey] })), t)) || (r = [...r, t]);
|
|
1499
|
+
if (r.length > 0) return yield* n.fail(le(r));
|
|
1500
|
+
}), E = (() => {
|
|
1501
1501
|
let t = e.targetDialect;
|
|
1502
1502
|
return t === "postgres" || t === "mysql" || t === "mariadb" || t === "sqlite" || t === "mssql" ? t : void 0;
|
|
1503
|
-
})(),
|
|
1503
|
+
})(), D = qn({
|
|
1504
1504
|
mode: f,
|
|
1505
1505
|
...e.targetDialect === void 0 ? {} : { targetDialect: e.targetDialect },
|
|
1506
1506
|
...e.targetSnapshot === void 0 ? {} : { targetSnapshot: e.targetSnapshot },
|
|
1507
1507
|
bundleTables: s.tables.map((e) => e.name),
|
|
1508
1508
|
isSkipped: W,
|
|
1509
1509
|
canRunSql: Mn(e.store)
|
|
1510
|
-
}),
|
|
1511
|
-
if (f === "replace" && !
|
|
1510
|
+
}), fe = D.tables, pe = s.tables.filter((e) => !W(e.name) && C(e.name)).map((e) => e.name), O = D.active, k = !1, A = 0, j = 0, he, M, N = 0, P = 0, ge = (t, r) => n.gen(function* () {
|
|
1511
|
+
if (f === "replace" && !O && !(r && b.truncated)) {
|
|
1512
1512
|
e.recordInterruption === !0 && (yield* n.tryPromise({
|
|
1513
1513
|
try: () => t.insert(d, {
|
|
1514
1514
|
id: h("repl", F),
|
|
@@ -1560,22 +1560,22 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1560
1560
|
total: h.rowCount
|
|
1561
1561
|
});
|
|
1562
1562
|
let y = a.length, x = n.gen(function* () {
|
|
1563
|
-
let { createHash: r } = yield* n.promise(() => import("node:crypto")), i = r("sha256"), o = 0, c = me("bundle", () =>
|
|
1563
|
+
let { createHash: r } = yield* n.promise(() => import("node:crypto")), i = r("sha256"), o = 0, c = me("bundle", () => de(_).pipe(xt(s.compression))).pipe(l.tap((e) => n.sync(() => i.update(e)))), u = async (e) => {
|
|
1564
1564
|
let n = {
|
|
1565
1565
|
entry: h,
|
|
1566
1566
|
row: e,
|
|
1567
1567
|
reason: ""
|
|
1568
1568
|
};
|
|
1569
1569
|
try {
|
|
1570
|
-
await
|
|
1570
|
+
await ie(t, () => T(t, h, e));
|
|
1571
1571
|
} catch (t) {
|
|
1572
1572
|
let r = On(t);
|
|
1573
|
-
n.reason = r.reason,
|
|
1573
|
+
n.reason = r.reason, oe(h, e, r), a.push(n);
|
|
1574
1574
|
}
|
|
1575
1575
|
}, d = yield* n.promise(async () => {
|
|
1576
1576
|
let n = e.copyLoader;
|
|
1577
1577
|
if (!n || n.dialect !== "postgres") return !1;
|
|
1578
|
-
if (m) return
|
|
1578
|
+
if (m) return k || (k = !0, e.onWarn?.("bulk COPY is off for this run: it needs its own connection and this import runs in ONE transaction (the default for --mode replace, so a failure mid-load cannot leave the target emptied and unfilled). Pass --no-atomic to trade that guarantee back for COPY.")), !1;
|
|
1579
1579
|
if (f === "replace") return !0;
|
|
1580
1580
|
if (f !== "upsert") return !1;
|
|
1581
1581
|
try {
|
|
@@ -1660,7 +1660,7 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1660
1660
|
checksum: h.checksum
|
|
1661
1661
|
}, yield* J(v, b)));
|
|
1662
1662
|
}
|
|
1663
|
-
if (a.length > 0 && (yield*
|
|
1663
|
+
if (a.length > 0 && (yield* ue(t, a)), f === "replace" && (yield* n.promise(() => t.deleteMany(d, { where: {
|
|
1664
1664
|
column: "startedAt",
|
|
1665
1665
|
op: "lte",
|
|
1666
1666
|
value: I
|
|
@@ -1708,14 +1708,14 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1708
1708
|
sourceFingerprint: s.source.schemaFingerprint ?? null,
|
|
1709
1709
|
tablesWritten: 0,
|
|
1710
1710
|
rowsWritten: 0,
|
|
1711
|
-
staged: f === "replace" ?
|
|
1711
|
+
staged: f === "replace" ? O : null
|
|
1712
1712
|
}).then(() => {
|
|
1713
1713
|
ve = !0;
|
|
1714
1714
|
}, (t) => {
|
|
1715
1715
|
e.onWarn?.(`the import trace could not be written (${String(t?.message ?? t)}).\n The import continues — this is a record of what happened, not a safety check. Run
|
|
1716
1716
|
\`voltro db apply\` so the next one is recorded.`);
|
|
1717
1717
|
}))), e.traceId !== void 0 && B && (yield* n.promise(() => e.store.update(u, z, {
|
|
1718
|
-
staged: f === "replace" ?
|
|
1718
|
+
staged: f === "replace" ? O : null,
|
|
1719
1719
|
sourceFingerprint: s.source.schemaFingerprint ?? null
|
|
1720
1720
|
}).then(() => void 0, () => void 0)));
|
|
1721
1721
|
let ye = 0, be = !1;
|
|
@@ -1735,17 +1735,17 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1735
1735
|
rowsWritten: A,
|
|
1736
1736
|
failure: t
|
|
1737
1737
|
}).then(() => void 0, () => void 0) : Promise.resolve();
|
|
1738
|
-
if (t = (e) => xe(e), f === "replace" && !
|
|
1738
|
+
if (t = (e) => xe(e), f === "replace" && !O && D.blocker !== void 0 && e.onWarn?.(Nn(D.blocker)), f === "replace" && pe.length > 0 && e.onWarn?.(`write recorders are SUSPENDED for this replace (${pe.length} table(s) have one).\n A replace sets a state rather than changing rows, so a per-row history entry would describe
|
|
1739
1739
|
something that did not happen — and this path already writes through the raw store, without
|
|
1740
|
-
tenant scoping or \`audit()\` stamping, for the same reason.`),
|
|
1741
|
-
e.onWarn?.(`staging ${
|
|
1742
|
-
let t = e.store, r =
|
|
1740
|
+
tenant scoping or \`audit()\` stamping, for the same reason.`), O) {
|
|
1741
|
+
e.onWarn?.(`staging ${fe.length} table(s) before the swap — the target keeps its rows until the load stands.\n The destructive step is one server-side transaction at the end, not the whole load.`);
|
|
1742
|
+
let t = e.store, r = fe.map((e) => e.name), i = h("repl", `staging-${F}`), a = !1;
|
|
1743
1743
|
if (e.recordInterruption === !0) {
|
|
1744
1744
|
yield* n.promise(() => e.store.insert(d, {
|
|
1745
1745
|
id: i,
|
|
1746
1746
|
startedAt: I,
|
|
1747
1747
|
kind: "staging",
|
|
1748
|
-
tables: String(
|
|
1748
|
+
tables: String(fe.length),
|
|
1749
1749
|
capture: L?.dir ?? null,
|
|
1750
1750
|
via: e.via ?? "direct",
|
|
1751
1751
|
staging: r.map(X).join(","),
|
|
@@ -1776,20 +1776,20 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1776
1776
|
let i = m ? r : r.filter((e) => !Ut(b, e));
|
|
1777
1777
|
yield* n.tryPromise({
|
|
1778
1778
|
try: async () => {
|
|
1779
|
-
await Hn(t, i.map((e) => In(e,
|
|
1779
|
+
await Hn(t, i.map((e) => In(e, E))), await Hn(t, i.map((e) => Fn(e, E)));
|
|
1780
1780
|
},
|
|
1781
1781
|
catch: (e) => new G({ reason: `could not prepare the staging tables, so the import did not start and the target is untouched: ${String(e?.message ?? e)}` })
|
|
1782
1782
|
}), yield* R(ge(e.store, !m)), !m && b.truncated === !0 ? c = !0 : yield* n.tryPromise({
|
|
1783
|
-
try: () => Un(t,
|
|
1783
|
+
try: () => Un(t, fe, E).then(() => {
|
|
1784
1784
|
c = !0;
|
|
1785
1785
|
}),
|
|
1786
1786
|
catch: (e) => new G({ reason: String(e?.message ?? e) })
|
|
1787
1787
|
}).pipe(n.catchAll((i) => n.gen(function* () {
|
|
1788
|
-
let a = Vn(yield* n.promise(() => Wn(t, zn(e.targetSnapshot, r),
|
|
1788
|
+
let a = Vn(yield* n.promise(() => Wn(t, zn(e.targetSnapshot, r), E).then((e) => e, () => [])));
|
|
1789
1789
|
return yield* n.fail(new G({ reason: a ?? `the swap failed and the target is UNCHANGED: ${i.reason}` }));
|
|
1790
1790
|
}))), !m && c && b.truncated !== !0 && (b.truncated = !0, yield* J(v, b));
|
|
1791
1791
|
} finally {
|
|
1792
|
-
s.undo(), c || m ? (M = void 0, yield* n.promise(() => Hn(t, r.map((e) => In(e,
|
|
1792
|
+
s.undo(), c || m ? (M = void 0, yield* n.promise(() => Hn(t, r.map((e) => In(e, E))).then(() => void 0, () => void 0)), yield* n.promise(o)) : e.onWarn?.("the staged rows are KEPT so a re-run can continue from them rather than reload.\n If you are not going to re-run this bundle, drop them: voltro data clear-staging --yes");
|
|
1793
1793
|
}
|
|
1794
1794
|
} else m ? yield* n.tryPromise({
|
|
1795
1795
|
try: () => e.store.transactional(async (e) => {
|
|
@@ -1852,11 +1852,11 @@ async function* vr(e) {
|
|
|
1852
1852
|
yield new Uint8Array(Q);
|
|
1853
1853
|
for (let t of await hr(e.dir)) {
|
|
1854
1854
|
let n = B(e.dir, t), r = await _e(n);
|
|
1855
|
-
if (yield new Uint8Array(gr(t, r.size)), r.size > 0) for await (let e of
|
|
1855
|
+
if (yield new Uint8Array(gr(t, r.size)), r.size > 0) for await (let e of de(n)) yield new Uint8Array(e);
|
|
1856
1856
|
}
|
|
1857
1857
|
if (e.blobs) for await (let t of e.blobs.list()) {
|
|
1858
1858
|
yield new Uint8Array(_r(t.key, t.contentType, t.size));
|
|
1859
|
-
let n =
|
|
1859
|
+
let n = oe("sha256"), r = 0;
|
|
1860
1860
|
for await (let i of e.blobs.open(t.key)) {
|
|
1861
1861
|
let e = Buffer.from(i);
|
|
1862
1862
|
n.update(e), r += e.length, yield new Uint8Array(e);
|
|
@@ -1893,7 +1893,7 @@ var yr = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), br
|
|
|
1893
1893
|
let i = c(8).readBigUInt64BE(0), a = ye(n, t);
|
|
1894
1894
|
if (a !== n && !a.startsWith(n + be)) throw Error(`archive path '${t}' escapes the destination`);
|
|
1895
1895
|
await I(ve(a), { recursive: !0 });
|
|
1896
|
-
let o =
|
|
1896
|
+
let o = D(a);
|
|
1897
1897
|
for (; i > 0n;) {
|
|
1898
1898
|
r.length === 0 && await s(1);
|
|
1899
1899
|
let e = i < BigInt(r.length) ? Number(i) : r.length;
|
|
@@ -1910,7 +1910,7 @@ var yr = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), br
|
|
|
1910
1910
|
await s(i);
|
|
1911
1911
|
let a = c(i).toString("utf8");
|
|
1912
1912
|
await s(8);
|
|
1913
|
-
let o = c(8).readBigUInt64BE(0), l = await t.blobs.has(n), u =
|
|
1913
|
+
let o = c(8).readBigUInt64BE(0), l = await t.blobs.has(n), u = oe("sha256"), d = new P(), f = l ? Promise.resolve() : t.blobs.write(n, d, {
|
|
1914
1914
|
contentType: a,
|
|
1915
1915
|
size: Number(o)
|
|
1916
1916
|
}), p = o;
|
|
@@ -1970,10 +1970,10 @@ var yr = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), br
|
|
|
1970
1970
|
write: async (n, r, i) => {
|
|
1971
1971
|
let a = await import("node:fs/promises"), { pipeline: o } = await import("node:stream/promises"), { Transform: s } = await import("node:stream"), c = B(e, "assets");
|
|
1972
1972
|
await I(c, { recursive: !0 });
|
|
1973
|
-
let l =
|
|
1973
|
+
let l = oe("sha256"), u = 0, d = new s({ transform(e, t, n) {
|
|
1974
1974
|
l.update(e), u += e.length, n(null, e);
|
|
1975
1975
|
} }), f = B(c, `.tmp-${Date.now()}-${t.size}`);
|
|
1976
|
-
await o(r, d,
|
|
1976
|
+
await o(r, d, D(f));
|
|
1977
1977
|
let p = l.digest("hex");
|
|
1978
1978
|
await a.rename(f, B(c, p)), await a.appendFile(B(c, "index.ndjson"), `${JSON.stringify({
|
|
1979
1979
|
key: n,
|
|
@@ -1983,7 +1983,7 @@ var yr = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), br
|
|
|
1983
1983
|
})}\n`), t.add(n);
|
|
1984
1984
|
}
|
|
1985
1985
|
};
|
|
1986
|
-
}, Tr = xe(
|
|
1986
|
+
}, Tr = xe(ue), Er = Buffer.from("VENC1\0\0\0"), Dr = 32, Or = 16, kr = 7, Ar = 16, jr = 262144, Mr = {
|
|
1987
1987
|
N: 32768,
|
|
1988
1988
|
r: 8,
|
|
1989
1989
|
p: 1
|
|
@@ -1993,13 +1993,13 @@ var yr = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), br
|
|
|
1993
1993
|
};
|
|
1994
1994
|
async function* Ir(e, t) {
|
|
1995
1995
|
if (!t) throw Error("encryptStream: a non-empty passphrase is required");
|
|
1996
|
-
let n =
|
|
1996
|
+
let n = ce(Or), r = ce(kr), i = await Tr(t, n, Dr, {
|
|
1997
1997
|
...Mr,
|
|
1998
1998
|
maxmem: Nr
|
|
1999
1999
|
}), a = Buffer.alloc(Pr);
|
|
2000
2000
|
Er.copy(a, 0), a.writeUInt32BE(Mr.N, 8), a.writeUInt32BE(Mr.r, 12), a.writeUInt32BE(Mr.p, 16), n.copy(a, 20), r.copy(a, 36), yield new Uint8Array(a);
|
|
2001
2001
|
let o = 0, s = (e, t) => {
|
|
2002
|
-
let n =
|
|
2002
|
+
let n = ie("aes-256-gcm", i, Fr(r, o, t));
|
|
2003
2003
|
o === 0 && n.setAAD(a);
|
|
2004
2004
|
let s = Buffer.concat([n.update(e), n.final()]), c = n.getAuthTag();
|
|
2005
2005
|
o++;
|
|
@@ -2040,7 +2040,7 @@ async function* Lr(e, t) {
|
|
|
2040
2040
|
if (!await o(5 + t)) throw Error("encrypted bundle truncated (frame body)");
|
|
2041
2041
|
let n = r.subarray(5, 5 + t - Ar), i = r.subarray(5 + t - Ar, 5 + t);
|
|
2042
2042
|
r = r.subarray(5 + t);
|
|
2043
|
-
let a =
|
|
2043
|
+
let a = ae("aes-256-gcm", p, Fr(f, m, e));
|
|
2044
2044
|
m === 0 && a.setAAD(s), a.setAuthTag(i);
|
|
2045
2045
|
let c = Buffer.concat([a.update(n), a.final()]);
|
|
2046
2046
|
m++, e && (h = !0), c.length > 0 && (yield new Uint8Array(c));
|
|
@@ -2086,7 +2086,7 @@ var Rr = (e, t) => ({
|
|
|
2086
2086
|
tool: r,
|
|
2087
2087
|
hint: `'${n}' is not on PATH, so this ran '${r}' instead. If that binary came from Oracle's MySQL client it queries information_schema.COLUMN_STATISTICS, which MariaDB does not have, and the run dies with error 1109 after the first table. Install the MariaDB client package (which ships '${n}') and re-run. Do NOT reach for --column-statistics=0: that flag does not exist on '${n}', so it fixes the wrong client and breaks the right one. Note: a MariaDB 12.x client requires TLS by default, so running '${n}' BY HAND against a server without it fails with "TLS/SSL error: SSL is required" (2026) and needs --skip-ssl. This command is not affected.`
|
|
2088
2088
|
} : { tool: r };
|
|
2089
|
-
}, Hr = (e) => (process.env.PATH ?? "").split(z).filter((e) => e !== "").some((t) =>
|
|
2089
|
+
}, Hr = (e) => (process.env.PATH ?? "").split(z).filter((e) => e !== "").some((t) => fe(B(t, e))), Ur = (e) => {
|
|
2090
2090
|
switch (e) {
|
|
2091
2091
|
case "postgres": return "db.dump";
|
|
2092
2092
|
case "mysql":
|
|
@@ -2290,21 +2290,28 @@ var Rr = (e, t) => ({
|
|
|
2290
2290
|
});
|
|
2291
2291
|
}
|
|
2292
2292
|
}, Zr = (e) => e.kind === "copy" ? n.async((t) => {
|
|
2293
|
-
let r = `${e.to}.restoring-${
|
|
2293
|
+
let r = `${e.to}.restoring-${le()}`, i = (e) => {
|
|
2294
2294
|
try {
|
|
2295
|
-
|
|
2295
|
+
k(r);
|
|
2296
2296
|
} catch {}
|
|
2297
2297
|
t(n.fail(new K({
|
|
2298
2298
|
tool: "copy",
|
|
2299
2299
|
reason: e
|
|
2300
2300
|
})));
|
|
2301
2301
|
};
|
|
2302
|
-
|
|
2302
|
+
E(e.from, r, (a) => {
|
|
2303
2303
|
if (a) {
|
|
2304
2304
|
i(`copy ${e.from} → ${e.to}: ${a.message}`);
|
|
2305
2305
|
return;
|
|
2306
2306
|
}
|
|
2307
|
-
|
|
2307
|
+
for (let t of [
|
|
2308
|
+
`${e.to}-wal`,
|
|
2309
|
+
`${e.to}-shm`,
|
|
2310
|
+
`${e.to}-journal`
|
|
2311
|
+
]) try {
|
|
2312
|
+
k(t);
|
|
2313
|
+
} catch {}
|
|
2314
|
+
pe(r, e.to, (r) => {
|
|
2308
2315
|
if (r) {
|
|
2309
2316
|
i(`copy ${e.from} → ${e.to}: ${r.message}`);
|
|
2310
2317
|
return;
|
|
@@ -2333,12 +2340,12 @@ var Rr = (e, t) => ({
|
|
|
2333
2340
|
s ??= e;
|
|
2334
2341
|
};
|
|
2335
2342
|
if (e.stdoutFile && r.stdout) {
|
|
2336
|
-
let t =
|
|
2343
|
+
let t = D(e.stdoutFile);
|
|
2337
2344
|
t.on("error", (t) => c(`writing ${R(e.stdoutFile)}: ${t.message}`)), r.stdout.on("error", (e) => c(`reading the tool's output: ${e.message}`)), r.stdout.pipe(t);
|
|
2338
2345
|
}
|
|
2339
2346
|
let l = 0;
|
|
2340
2347
|
if (e.stdinFile && r.stdin) {
|
|
2341
|
-
let t =
|
|
2348
|
+
let t = de(e.stdinFile);
|
|
2342
2349
|
t.on("data", (e) => {
|
|
2343
2350
|
l += e.length;
|
|
2344
2351
|
}), t.on("error", (t) => {
|
|
@@ -2349,7 +2356,7 @@ var Rr = (e, t) => ({
|
|
|
2349
2356
|
}
|
|
2350
2357
|
let u = () => {
|
|
2351
2358
|
if (e.stdoutFile !== void 0) try {
|
|
2352
|
-
|
|
2359
|
+
k(e.stdoutFile);
|
|
2353
2360
|
} catch {}
|
|
2354
2361
|
};
|
|
2355
2362
|
r.on("error", (t) => {
|
|
@@ -2374,7 +2381,7 @@ var Rr = (e, t) => ({
|
|
|
2374
2381
|
});
|
|
2375
2382
|
}), Qr = (e) => {
|
|
2376
2383
|
try {
|
|
2377
|
-
return String(
|
|
2384
|
+
return String(O(e).size);
|
|
2378
2385
|
} catch {
|
|
2379
2386
|
return "?";
|
|
2380
2387
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@voltro/data-transfer",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.77.0",
|
|
4
4
|
"description": "Portable export/import for a Voltro app's data + assets — typed-NDJSON row codec, content-addressed assets, framed compression, a resumable chunk ledger, and FK-ordered idempotent import. Dialect-agnostic; the resilient backbone under `voltro data export|import` and native backups.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"voltro",
|
|
@@ -33,8 +33,8 @@
|
|
|
33
33
|
"node": ">=24.0.0"
|
|
34
34
|
},
|
|
35
35
|
"dependencies": {
|
|
36
|
-
"@voltro/database": "0.
|
|
37
|
-
"@voltro/plugin-storage": "0.
|
|
36
|
+
"@voltro/database": "0.77.0",
|
|
37
|
+
"@voltro/plugin-storage": "0.77.0"
|
|
38
38
|
},
|
|
39
39
|
"peerDependencies": {
|
|
40
40
|
"effect": "^3.22.2"
|