@voltro/data-transfer 0.50.1 → 0.51.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +104 -0
- package/dist/index.d.ts +86 -1
- package/dist/index.js +71 -65
- package/package.json +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -39,6 +39,110 @@ _Changes staged for the next release accumulate here (rolled up from
|
|
|
39
39
|
|
|
40
40
|
---
|
|
41
41
|
|
|
42
|
+
## [0.51.0] — 2026-08-24
|
|
43
|
+
|
|
44
|
+
### ⚠ BREAKING
|
|
45
|
+
|
|
46
|
+
- **@voltro/data-transfer, @voltro/cli** — The asset counts reported references as if they were blobs. `_voltro_storage_refs` holds one row per reference and several rows legitimately name one key, so a capture of 57 rows over 16 keys wrote `count: 57` into the stamp beside an `assets/` directory holding 16 files, and the restore reported "57 blob(s) restored" while 16 objects appeared. Nothing was lost; what was lost is the ability to check. Anyone answering "are all the blobs there?" after a restore compared the stamp's number against one they counted and found a 3.5x gap that was not one.
|
|
47
|
+
|
|
48
|
+
A key named by several references is now fetched once rather than downloaded, hashed and discarded once per row, and the three numbers are stated separately: `references` (rows enumerated), `count` (distinct keys), `objects` (distinct sha256 bodies), with `totalBytes` and `objectBytes` beside them. The stamp's existing fields keep their names and now mean what a reader always took them for; the new ones are optional, so a stamp written before them still parses.
|
|
49
|
+
|
|
50
|
+
**Breaking on one export.** `restoreAssetsFromCas` returns `{ count, objects }` instead of a bare `number` — one number could not answer both questions, which is the defect. `count` is what the old value was, so the migration that changes nothing is `.count`.
|
|
51
|
+
|
|
52
|
+
**`voltro update` carries you across this** — codemod `0.51.0/03_restore-assets-returns-counts`. If you pin versions by hand and never run it, print the notes without changing anything: `voltro update --codemods-only --from <your current version> --dry-run` (this one ships in 0.51.0).
|
|
53
|
+
- **@voltro/i18n** — `<I18nProvider>` takes `timeZone` as its own prop, and `intlConfig` no longer accepts one.
|
|
54
|
+
|
|
55
|
+
`intlConfig` exists to forward props to react-intl UNMODIFIED, and for every other member of `IntlConfig` that is the right shape. `timeZone` is the one member this package's own formatters read — and they did not read it: `useFormatDate` built `Intl.DateTimeFormat` itself and took only the locale from the provider. So a zone passed through `intlConfig` configured `<T>`'s ICU dates and NOT the `useFormatDate()` beside them. Measured under one provider, one instant (`2026-08-24T23:30:00Z`), `locale: 'de'`, `intlConfig: { timeZone: 'Europe/Berlin' }`, process zone UTC: react-intl rendered `25.08.26, 01:30` and the hook rendered `24.08.26, 23:30`. A different hour, and a different day.
|
|
56
|
+
|
|
57
|
+
The zone has to be a prop this package can see. It is validated once at the provider (an unusable zone — a stale cookie, a typo, a runtime with a trimmed ICU — is dropped, because `Intl` THROWS on an unknown zone and `useFormatDate` catches, which would degrade every timestamp in the app to a raw `Date` string). It is what the new `useTimeZone()` reports. And it is what the framework fills per request.
|
|
58
|
+
|
|
59
|
+
**Migration:** `intlConfig={{ timeZone: 'Europe/Berlin' }}` → `timeZone="Europe/Berlin"`. The codemod does it, including the case where the zone was the bag's only member. An `intlConfig` naming a variable is reported by file and line rather than guessed at — the property would otherwise stop being read with nothing red anywhere.
|
|
60
|
+
- **@voltro/cli** — A native restore whose bookkeeping store would not open ran anyway, with no in-progress marker and no word about it. `nativeBookkeeping` was `try { … } catch { return undefined }`, and that `undefined` guarded every branch below — including the refusal for a marker that could not be written. So the failure removed the precaution AND the sentence that would have reported it missing, and `markerState` was never computed, defaulting to "held".
|
|
61
|
+
|
|
62
|
+
What it produced was worse than silence: a failed restore printed "This database is now in an unknown state and the next boot will REFUSE, by design" over a database with zero marker rows. The next boot did not refuse, and `voltro data clear-replace-marker` had nothing to clear. The trigger is not exotic — wrong credentials, an unreachable database, a missing env var, no `app.config.ts` from here — and a restore is the operation you run against a target that is already unwell, so the guard fell away exactly when it was needed.
|
|
63
|
+
|
|
64
|
+
The reason now travels instead of being caught and dropped. A restore that cannot write the marker REFUSES and names which of the two reasons it was (the store would not open, or the table is not there); those were two separate refusals and are now one, because they are one decision for the operator. `--no-marker` is the deliberate way past it and warns every time. And the `recorded:` line no longer reports a local failure as a property of the target — it says where the failure was.
|
|
65
|
+
|
|
66
|
+
**This changes an exit code.** A restore that could not write the marker used to exit 0; it now exits 1. Two invocations are affected — the bookkeeping store will not open, or there is no `app.config.ts` from the working directory — and the second surprises people, because `restore` reads its target from the environment and so looks like it needs no project. It does not, for the dump; it needs one for the marker. `--no-marker` is the deliberate way through and warns every time. The note ships under `reach: 'beyond-source'` because the affected invocations live in cron entries, CI jobs and runbooks rather than in TypeScript.
|
|
67
|
+
|
|
68
|
+
**`voltro update` carries you across this** — codemod `0.51.0/02_restore-refuses-without-marker`. If you pin versions by hand and never run it, print the notes without changing anything: `voltro update --codemods-only --from <your current version> --dry-run` (this one ships in 0.51.0).
|
|
69
|
+
|
|
70
|
+
### Added
|
|
71
|
+
|
|
72
|
+
- **@voltro/i18n, @voltro/cli, @voltro/voltro** — `timeZone` in the web `app.config.ts` — the zone every date/time formatter renders in, resolved per request and published so the client agrees.
|
|
73
|
+
|
|
74
|
+
A formatter is deterministic given the value, the locale, the zone and the clock. The locale already came from the provider and was already agreed across the hydration boundary — the server publishes it as `<html lang>` and the client reads that attribute rather than `navigator.languages`, precisely because the browser's own answer can differ from what the server saw. The zone had no such source. `Intl` fell back to the zone of whichever runtime was formatting: the pod on the server (UTC on a container with no `TZ`), the viewer's machine in the browser. Every server-rendered timestamp was therefore a hydration mismatch waiting for a wide enough offset, and across midnight it was a different calendar day.
|
|
75
|
+
|
|
76
|
+
timeZone: 'Europe/Berlin' // one zone for every viewer timeZone: 'viewer' // per request, from the `voltro:tz` cookie defaultTimeZone: 'UTC' // before the viewer's zone is known
|
|
77
|
+
|
|
78
|
+
Whatever it resolves to is stamped on the document as `<html data-voltro-tz>`, and the generated client entry reads that attribute. Both sides then format against one value — which is the property that removes the mismatch, whether or not the value is the viewer's true zone: being wrong together is repairable after mount, being different is not.
|
|
79
|
+
|
|
80
|
+
Under `'viewer'` the framework injects a script that seeds `voltro:tz` from the browser when the cookie is absent, so the server renders in the viewer's zone from the second request with no login. It never overwrites an existing value — the APP is the authoritative writer, at login, from the zone it holds for the signed-in user (`TIMEZONE_COOKIE` and `isSupportedTimeZone` are exported for that). Unset, nothing changes: each runtime keeps using its own zone, and `useTimeZone()` returns `undefined` to say so.
|
|
81
|
+
|
|
82
|
+
This is the RENDER zone. The server-side compute zone — what `startOfDay` resolves against inside a handler — is still `@voltro/datetime/context`'s seam, unwired.
|
|
83
|
+
|
|
84
|
+
**`apiSurface: compatible`** covers the two golden lines that moved, and they are the same change twice: `makeSsgWrap`'s returned wrapper, and the `wraps` record on the SSG shell input, each gained an OPTIONAL second parameter (the per-request zone + render instant; the wrapper is built once per locale, so they cannot live in the factory). A function with an optional extra parameter is assignable wherever the one-parameter type was expected, so no call site that compiled stops compiling — and both are the framework's SSG bridge, documented as never imported by app code. Everything else this release adds to these packages is a pure addition; the one genuine break in `@voltro/i18n` is the `timeZone` prop, which has its own entry and its own codemod.
|
|
85
|
+
|
|
86
|
+
### Changed
|
|
87
|
+
|
|
88
|
+
- **@voltro/cli** — A `BREAKING` entry's changelog footer now tells a reader who pins versions by hand how to print the codemod's note without upgrading anything:
|
|
89
|
+
|
|
90
|
+
voltro update --codemods-only --from <your current version> --dry-run
|
|
91
|
+
|
|
92
|
+
The footer used to stop at the codemod's id, which is enough for anyone who runs `voltro update` and nothing at all for anyone who does not. A deployment said so plainly: they pin every `@voltro/*` version from their own container scripts, have never run the command, and `CHANGELOG.md` out of the tarball is the only channel anything reaches them through. So a note deliberately filed under an unreached version — our one mechanism for correcting guidance that can no longer be corrected in place — reached that population not at all, and the id told them a fix existed without telling them what it was.
|
|
93
|
+
|
|
94
|
+
No new surface: every flag in that invocation is already parsed, which is what lets `check-message-apis.mjs` verify the line rather than trust it.
|
|
95
|
+
- **@voltro/cli** — Two fingerprint labels now say what they compare.
|
|
96
|
+
|
|
97
|
+
The restore's skew warning said the backup's schema "differs from what this code declares". It does not: the value it compares against is the TARGET database's live schema, read by introspection at restore time. Bringing a target to the backup's shape makes the warning disappear while the declared fingerprint is a third value entirely, which is how the mislabel was caught. The comparison is the useful one and is unchanged; the sentence sent readers looking for a code change where a database differed.
|
|
98
|
+
|
|
99
|
+
`voltro db plan` prints `fingerprint: live … · declared …` instead of `from … → to …`, plus a line saying the two are not meant to match. A hash of a live database never equals the hash of the declaration it came from — introspection cannot recover generated expressions, `maxLength` or sensitivity markers — which is why `db drift` keeps a separate live baseline. Printed as `from → to`, `0 operations` under two differing hashes read as a contradiction.
|
|
100
|
+
|
|
101
|
+
### Fixed
|
|
102
|
+
|
|
103
|
+
- **@voltro/cli** — `voltro check` reported a reactivity CHANNEL as a missing table, at `error` severity — so it set the exit code:
|
|
104
|
+
|
|
105
|
+
✗ error reference/dangling-source query(presence.list) reads table 'channel:presence' which does not exist fix: declare a 'channel:presence.entity.ts' table or fix the query's source
|
|
106
|
+
|
|
107
|
+
A `source:` entry is a table name OR a channel's routing key (`channel:<name>`), and every rule resolved entries against the table set. The advice cannot be followed — a channel exists precisely because no table is meant — and because it is an error rather than a warning, `voltro check` could not be a CI gate for any app that uses a channel. That includes an app whose only channel comes from `@voltro/plugin-presence`, whose own `presence.list` declares one: a first-party feature meeting a rule that did not know about it, inside a first-party plugin.
|
|
108
|
+
|
|
109
|
+
Channels are filtered in the ONE helper every table rule reads, rather than at each rule, because a per-rule filter is how the next rule joins without one. The same cause was live one rule over: `observed/declared-but-unobserved` reported "declares source 'channel:presence' but never read it while running" for every exercised procedure that declares a channel. Both are covered, each with a negative control — a filter that dropped the whole source list would have silenced the rules instead of narrowing them.
|
|
110
|
+
- **@voltro/data-transfer** — A native dump no longer carries `_voltro_data_transfers`, for the same reason it stopped carrying the in-progress marker one release ago. The restore opens its own run row there BEFORE the tool runs; the dump then dropped the table mid-flight, and the update recording the outcome wrote into a table that no longer held the row. Measured downstream: after a deliberately failed native restore, `voltro data transfers` showed no restore at all — only the `backup` row the dump had carried over from the SOURCE database. The command that answers "did the restore finish" could not see the run asking the question.
|
|
111
|
+
|
|
112
|
+
Exactly two tables are excluded and the line is deliberate: a native restore into the same deployment should bring the migration ledger, the stored plans, the CDC offsets and the schedule claims, because they describe the data being restored. These two describe the RESTORE, and a record of an operation must not be overwritten by the operation it records. Covered per dialect against real servers and real vendor tools, including a non-vacuity check that a table which SHOULD travel still does.
|
|
113
|
+
- **@voltro/i18n, @voltro/cli** — `useRelativeTime` used `Date.now()` as its base, which under SSR is two different numbers. The server rendered at T and wrote "3 minutes ago" into the HTML; the browser hydrated at T+Δ and rendered "4 minutes ago" whenever a unit boundary fell in the gap. The gap is network latency, so it reproduced on a slow connection and never on the developer's machine, and it had nothing to do with timezones — a correctly zoned app hit it just the same.
|
|
114
|
+
|
|
115
|
+
The server states its render instant (`<html data-voltro-now>`, `renderedAt` on the provider), the first client render uses that same number, and the clock goes live once hydration commits. Server markup and hydration markup are therefore identical BY CONSTRUCTION — the property `await.tsx` and `deferred.ts` already hold, rather than `suppressHydrationWarning`, which would hide a real mismatch along with this one. An explicit `{ now }` still wins.
|
|
116
|
+
|
|
117
|
+
The mount state lives in the provider, not in the hook: a table of ten thousand rows would otherwise pay a state hook and a passive effect each to learn one fact that is true for the whole document. An app that never renders on the server publishes no stamp and takes no second render pass.
|
|
118
|
+
- **@voltro/cli** — `closeNativeRun` writes the transfer row BACK when the restore's own artefact dropped the table it lives in, instead of issuing an `UPDATE` that matches nothing and returning happily. Excluding `_voltro_data_transfers` from our own dumps shortens that window; it does nothing for a dump taken before that change, for a hand-made one, or for mssql and sqlite, whose restores have no per-table exclusion at all. The write-back covers every dialect and every artefact, which is why it is the rule and the exclusion is the optimisation.
|
|
119
|
+
|
|
120
|
+
The row is read back rather than trusted — an update that matched nothing is indistinguishable from one that matched — and a read that itself fails writes nothing, because a duplicate row invented on a guess is its own defect in a history somebody reads under pressure.
|
|
121
|
+
- **@voltro/cli** — A prerendered page shipped the shell's baked `lang="en"` whatever locale it was rendered in.
|
|
122
|
+
|
|
123
|
+
`voltro dev` and `voltro start` both set `<html lang>` per request; the prerender never did. So a `/de/...` artefact — rendered with the German catalog, handed `locale: 'de'` in its `meta` — served `<html lang="en">`. That attribute is what a screen reader pronounces in, what Chrome offers to translate FROM, and what hyphenation uses, so the failure was silent to whoever shipped it and loud only to the people it excluded. The same shape as the 0.30.0 cookie-name drift, one document path over.
|
|
124
|
+
|
|
125
|
+
It surfaced while giving the zone somewhere to travel: `<html lang>` was set by four hand-written copies of one `.replace(/<html…/)` and by nothing in the prerender, and adding a second attribute to that arrangement is how the next one reaches three paths out of five. There is one `applyDocumentAttrs` now, and the prerender is one of its callers — which fixes the locale as a side effect of having somewhere to put the zone.
|
|
126
|
+
|
|
127
|
+
### Internal (no consumer-facing effect)
|
|
128
|
+
|
|
129
|
+
- **@voltro/cli** — `voltro data backup --assets` / `restore --assets` are now driven against a REAL S3 API (MinIO in the test stack), over the network, with a real backup and a real restore into a second bucket and the bytes compared.
|
|
130
|
+
|
|
131
|
+
The asset half rests on one field: a provider must map "there is no object at that key" to `status === 404`, and nothing looser — a 403 from a rotated credential is also non-transient, and calling that "the object is gone" turns a recoverable outage into a backup that quietly contains nothing. That mapping was measured against memory, filesystem and database live, and against the s3/azure SDK error SHAPES constructed. A constructed shape is a claim about an SDK, not about a round trip: nothing in it exercises signing, path-style addressing, or what the SDK actually raises when a server answers `NoSuchKey`. Two deployments listed exactly this as the gap they could not close either.
|
|
132
|
+
|
|
133
|
+
Both directions are covered against the real server: a dangling reference is stepped over and reported, and a bad credential fails the capture rather than being read as a missing object.
|
|
134
|
+
|
|
135
|
+
The native dialect lane also stops being silent about mssql. It was absent from the array entirely — an absent lane and a covered one look identical from the outside — and it is now listed with a written reason for why it does not register here (`sqlpackage` is a separate Microsoft download on a .NET runtime, absent from `mcr.microsoft.com/mssql-tools`). It is registered rather than skipped-forever, because a skip present on every healthy run teaches readers to ignore skip lines; and an assertion fails if any lane drops out WITHOUT a written reason, or if a reason names a lane that is in fact running.
|
|
136
|
+
- **@voltro/cli, @voltro/plugin-storage** — Coverage for the data commands, at the level the defects actually live.
|
|
137
|
+
|
|
138
|
+
`dataDirectFlags.e2e.test.ts` drives every DIRECT-target flag of `voltro data export` / `import` through the real binary against a real sqlite database, and carries the same `DATA_FLAGS`-driven self-check the native suite has: a new direct flag has to be driven there or the file goes red. The api-only flags are listed explicitly with the reason they are not here, and that list is asserted against `DATA_FLAGS` so it cannot become a place to hide an untested flag.
|
|
139
|
+
|
|
140
|
+
`missingObjectIs404.test.ts` pins the contract the dangling-reference skip rests on: every provider maps "no object at that key" to `status === 404`, and nothing looser. Five providers, three of them live, s3 and azure through their SDK's real error shapes — which read DIFFERENT fields (`$metadata.httpStatusCode` vs a bare `statusCode`), so a mapping copied from one to the other would turn dangling references back into hard capture failures on that backend alone.
|
|
141
|
+
|
|
142
|
+
`codegenFeatureTables.integration.test.ts` measures the count a report was about: `voltro codegen` must carry the tables a `*.cron.tsx` contributes, which the entity walk cannot see. The structural guards beside it were TRUE while that count was wrong. `frameworkSourceTypo.integration.test.ts` measures what catches a misspelled `_voltro_*` source given that the type deliberately does not — `voltro check` reports it as a dangling source and exits 1, with a negative control so the check is not merely flagging every framework name.
|
|
143
|
+
|
|
144
|
+
---
|
|
145
|
+
|
|
42
146
|
## [0.50.1] — 2026-08-24
|
|
43
147
|
|
|
44
148
|
### Added
|
package/dist/index.d.ts
CHANGED
|
@@ -181,8 +181,32 @@ export declare const captureAssetsToCas: (opts: {
|
|
|
181
181
|
}) => Effect.Effect<CapturedAssets, BundleError>;
|
|
182
182
|
|
|
183
183
|
export declare interface CapturedAssets {
|
|
184
|
+
/**
|
|
185
|
+
* Distinct KEYS captured. Not rows enumerated.
|
|
186
|
+
*
|
|
187
|
+
* `_voltro_storage_refs` holds one row per reference, and several references
|
|
188
|
+
* legitimately point at one stored object. The loop used to count rows, so a
|
|
189
|
+
* deployment's stamp said `count: 57` over an artefact holding 16 blobs — and
|
|
190
|
+
* the same object was downloaded, hashed and thrown away 3.5 times on the way.
|
|
191
|
+
* `restore` then reported "57 blob(s) restored" while 16 files appeared.
|
|
192
|
+
*
|
|
193
|
+
* Nothing was lost either way; what was lost is the ability to check. Anyone
|
|
194
|
+
* answering "are all the blobs there?" after a restore compared a number from
|
|
195
|
+
* the stamp against a number they counted, and found a gap that was not one.
|
|
196
|
+
*/
|
|
184
197
|
readonly count: number;
|
|
198
|
+
/** Bytes of the distinct keys above — what the artefact actually carries
|
|
199
|
+
* before content-dedup, not the sum over references. */
|
|
185
200
|
readonly totalBytes: number;
|
|
201
|
+
/** References the source enumerated, INCLUDING repeats of one key. Reported
|
|
202
|
+
* beside `count` so "57 references → 16 keys" is legible rather than looking
|
|
203
|
+
* like a shortfall. */
|
|
204
|
+
readonly references: number;
|
|
205
|
+
/** Distinct `sha256` bodies on disk. Below `count` when two keys hold
|
|
206
|
+
* identical content — the content-addressed store keeps one copy. */
|
|
207
|
+
readonly objects: number;
|
|
208
|
+
/** Bytes those distinct bodies occupy — the size of `assets/` itself. */
|
|
209
|
+
readonly objectBytes: number;
|
|
186
210
|
/**
|
|
187
211
|
* Keys the refs table names and the provider does not have (404).
|
|
188
212
|
*
|
|
@@ -546,6 +570,52 @@ export declare const environmentLocalNotice: (skipped: ReadonlyArray<string>, ve
|
|
|
546
570
|
* rule rather than that a table went missing. */
|
|
547
571
|
export declare const environmentLocalReason: (tableName: string) => string | undefined;
|
|
548
572
|
|
|
573
|
+
/**
|
|
574
|
+
* A dump must not carry the in-progress MARKER TABLE.
|
|
575
|
+
*
|
|
576
|
+
* The marker is one row in `_voltro_replace_in_progress`, written before a
|
|
577
|
+
* restore's first destructive statement so a restore that dies half-way refuses
|
|
578
|
+
* the next boot instead of serving a database that is neither the old one nor
|
|
579
|
+
* the new one.
|
|
580
|
+
*
|
|
581
|
+
* This module's own notes claimed that row survives a native restore on every
|
|
582
|
+
* dialect but sqlite, on the reasoning that a table-scoped restore only touches
|
|
583
|
+
* what the dump names. The reasoning is right and the premise was wrong: a
|
|
584
|
+
* `mariadb-dump` / `pg_dump` names the WHOLE database, marker table included,
|
|
585
|
+
* and a mysql-family dump writes `DROP TABLE IF EXISTS` in front of each one. A
|
|
586
|
+
* deployment measured it — 1 row before the restore, 0 after, twice, once by
|
|
587
|
+
* hand and once through this command. The table sorts early, so the marker was
|
|
588
|
+
* gone for almost the entire window it exists to cover, and a restore that then
|
|
589
|
+
* died left a half-loaded database with nothing to stop the next boot.
|
|
590
|
+
*
|
|
591
|
+
* Excluding it here makes the claim true rather than documenting it as false: a
|
|
592
|
+
* backup WE take can no longer contain the thing that erases the guard on the
|
|
593
|
+
* way back in. It does not help a dump taken before this change or by hand,
|
|
594
|
+
* which is why the restore re-asserts the marker after the tool exits — the two
|
|
595
|
+
* halves cover different dumps and neither replaces the other.
|
|
596
|
+
*
|
|
597
|
+
* ── The second table, for the same reason one table over ────────────────────
|
|
598
|
+
*
|
|
599
|
+
* `_voltro_data_transfers` is excluded too. The restore opens its own run row
|
|
600
|
+
* there BEFORE the tool runs; the dump then drops the table mid-flight, and the
|
|
601
|
+
* update that would have recorded the outcome writes into a table that no longer
|
|
602
|
+
* holds the row. A deployment measured the consequence: after a deliberately
|
|
603
|
+
* failed native restore, `voltro data transfers` showed no restore at all — only
|
|
604
|
+
* the `backup` row the dump had carried over from the SOURCE database. The
|
|
605
|
+
* command that answers "did last night's restore finish" could not see the run
|
|
606
|
+
* asking the question.
|
|
607
|
+
*
|
|
608
|
+
* These two are excluded and the other `_voltro_*` tables are not, and the line
|
|
609
|
+
* between them is deliberate: a native restore into the same deployment SHOULD
|
|
610
|
+
* bring the migration ledger, the plans, the CDC offsets and the schedule claims
|
|
611
|
+
* — they describe the data being restored. These two describe the RESTORE, and a
|
|
612
|
+
* record of an operation must not be overwritten by the operation it records.
|
|
613
|
+
*
|
|
614
|
+
* mssql (`sqlpackage`) and the sqlite/turso file copy have no such flag; the
|
|
615
|
+
* copy is atomic for exactly this reason (see the `copy` branch).
|
|
616
|
+
*/
|
|
617
|
+
export declare const EXCLUDED_FROM_DUMP: readonly ["_voltro_replace_in_progress", "_voltro_data_transfers"];
|
|
618
|
+
|
|
549
619
|
/**
|
|
550
620
|
* Run statements against the store's own connection.
|
|
551
621
|
*
|
|
@@ -1352,17 +1422,32 @@ export declare const restoreAssetFromCas: (sink: AssetSink, assetsDir: string, e
|
|
|
1352
1422
|
* A hash mismatch is an {@link IntegrityError} and stops the phase — a corrupted
|
|
1353
1423
|
* artefact must never silently re-upload bad bytes over good ones, which is the
|
|
1354
1424
|
* one failure a restore has no way to notice afterwards.
|
|
1425
|
+
*
|
|
1426
|
+
* Returns {@link RestoredAssets} rather than a bare count: keys and distinct
|
|
1427
|
+
* bodies are different numbers, and reporting one of them as "blobs" is how a
|
|
1428
|
+
* reader came to compare a restore's number against files on disk and find a
|
|
1429
|
+
* 3.5x gap that was not one.
|
|
1355
1430
|
*/
|
|
1356
1431
|
export declare const restoreAssetsFromCas: (opts: {
|
|
1357
1432
|
readonly sink: AssetSink;
|
|
1358
1433
|
readonly assetsDir: string;
|
|
1359
1434
|
readonly indexPath: string;
|
|
1360
1435
|
readonly progress?: AssetProgress;
|
|
1361
|
-
}) => Effect.Effect<
|
|
1436
|
+
}) => Effect.Effect<RestoredAssets, BundleError | IntegrityError | StorageError>;
|
|
1362
1437
|
|
|
1363
1438
|
/** Build the restore step for a dialect reading `inFile`. */
|
|
1364
1439
|
export declare const restoreCommand: (dialect: string, conn: NativeConn, inFile: string) => NativeStep;
|
|
1365
1440
|
|
|
1441
|
+
/** What a restore put back. */
|
|
1442
|
+
export declare interface RestoredAssets {
|
|
1443
|
+
/** Distinct KEYS written to the sink. */
|
|
1444
|
+
readonly count: number;
|
|
1445
|
+
/** Distinct `sha256` bodies read out of `assets/`. Below `count` when two keys
|
|
1446
|
+
* hold identical content. Reported so "16 objects, 16 keys" can be checked
|
|
1447
|
+
* against what is on the destination instead of guessed at. */
|
|
1448
|
+
readonly objects: number;
|
|
1449
|
+
}
|
|
1450
|
+
|
|
1366
1451
|
declare interface RollbackCapture {
|
|
1367
1452
|
/** Where the target's previous rows were written. */
|
|
1368
1453
|
readonly dir: string;
|
package/dist/index.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { Cause as e, Data as t, Effect as n, Exit as r, Metric as i, MetricBoundaries as a, Runtime as o, Schedule as s, Schema as c, Stream as l } from "effect";
|
|
2
|
-
import { DATA_TRANSFERS_TABLE as u, REPLACE_IN_PROGRESS_TABLE as d, _voltroDataTransfersTable as f, classifyConstraintViolation as p, derivedRowId as m, ensureTableRegistered as h, eq as g, extractDbCause as _, getTable as v, inSet as y, isNotNull as b, or as x, recordsTable as S, registerTable as
|
|
3
|
-
import { fingerprintSchema as
|
|
4
|
-
import { createCipheriv as
|
|
2
|
+
import { DATA_TRANSFERS_TABLE as u, REPLACE_IN_PROGRESS_TABLE as d, _voltroDataTransfersTable as f, classifyConstraintViolation as p, derivedRowId as m, ensureTableRegistered as h, eq as g, extractDbCause as _, getTable as v, inSet as y, isNotNull as b, or as x, recordsTable as S, registerTable as ee, streamTable as C, suspendWriteRecorders as te, suspendWriteRecordersEffect as ne, unregisterTable as re } from "@voltro/database";
|
|
3
|
+
import { fingerprintSchema as ie } from "@voltro/database/sql";
|
|
4
|
+
import { createCipheriv as w, createDecipheriv as T, createHash as E, createHmac as ae, randomBytes as D, randomUUID as oe, scrypt as se } from "node:crypto";
|
|
5
5
|
import { copyFile as ce, createReadStream as le, createWriteStream as O, existsSync as ue, rename as de, statSync as fe, unlinkSync as k } from "node:fs";
|
|
6
6
|
import { STORAGE_REFS_TABLE as pe, fromNodeReadable as me, getObjectStream as A, putObjectStream as j, toNodeReadable as he } from "@voltro/plugin-storage";
|
|
7
7
|
import * as M from "node:zlib";
|
|
@@ -57,7 +57,7 @@ var we = (e) => {
|
|
|
57
57
|
return r.classes?.[e] ?? Te[e] ?? "redact";
|
|
58
58
|
}
|
|
59
59
|
return n?.safe ? r.onSafe ?? "keep" : n?.columnType === "id" || n?.columnType === "reference" ? "keep" : r.onUnclassified ?? "error";
|
|
60
|
-
}, Oe = (e, t) =>
|
|
60
|
+
}, Oe = (e, t) => ae("sha256", e).update(t == null ? "∅" : String(t)).digest(), z = (e, t) => e.readUInt32BE(t * 4 % (e.length - 4)), ke = [
|
|
61
61
|
"Ada",
|
|
62
62
|
"Alan",
|
|
63
63
|
"Grace",
|
|
@@ -343,7 +343,7 @@ var we = (e) => {
|
|
|
343
343
|
};
|
|
344
344
|
}, p = (r, i) => {
|
|
345
345
|
let a = f(r, G(t, r), tt(t, r));
|
|
346
|
-
return l.runForEach(
|
|
346
|
+
return l.runForEach(C(e, {
|
|
347
347
|
table: r,
|
|
348
348
|
pk: G(t, r),
|
|
349
349
|
...i === void 0 ? {} : { where: i }
|
|
@@ -458,30 +458,36 @@ var we = (e) => {
|
|
|
458
458
|
done: () => !1,
|
|
459
459
|
mark: () => n.void
|
|
460
460
|
}, dt = (e) => typeof e == "object" && !!e && e.status === 404, ft = (e) => n.gen(function* () {
|
|
461
|
-
let t = e.progress ?? ut, r = 0, i = 0, a = [];
|
|
462
|
-
return yield* l.runForEach(e.source.list(), (
|
|
463
|
-
|
|
461
|
+
let t = e.progress ?? ut, r = 0, i = 0, a = 0, o = 0, s = /* @__PURE__ */ new Set(), c = /* @__PURE__ */ new Set(), u = [];
|
|
462
|
+
return yield* l.runForEach(e.source.list(), (l) => (a++, t.done(l.key) || c.has(l.key) ? n.void : (c.add(l.key), e.source.open(l.key).pipe(n.map((e) => e), n.catchIf(dt, () => n.sync(() => {
|
|
463
|
+
u.push(l.key);
|
|
464
464
|
})), n.flatMap((a) => a === void 0 ? n.void : ct(a, e.assetsDir).pipe(n.flatMap((e) => {
|
|
465
465
|
let n = {
|
|
466
|
-
key:
|
|
466
|
+
key: l.key,
|
|
467
467
|
sha256: e.sha256,
|
|
468
468
|
size: e.size,
|
|
469
469
|
contentType: e.contentType
|
|
470
470
|
};
|
|
471
|
-
return r++, i += e.size, t.mark(
|
|
472
|
-
}))))).pipe(n.mapError((e) => e instanceof H ? e : new H({ reason: `asset export failed: ${String(e)}` }))), {
|
|
471
|
+
return r++, i += e.size, s.has(e.sha256) || (s.add(e.sha256), o += e.size), t.mark(l.key, n);
|
|
472
|
+
}))))))).pipe(n.mapError((e) => e instanceof H ? e : new H({ reason: `asset export failed: ${String(e)}` }))), {
|
|
473
473
|
count: r,
|
|
474
474
|
totalBytes: i,
|
|
475
|
-
|
|
475
|
+
references: a,
|
|
476
|
+
objects: s.size,
|
|
477
|
+
objectBytes: o,
|
|
478
|
+
missing: u
|
|
476
479
|
};
|
|
477
480
|
}), pt = (e) => me("bundle", () => le(e)).pipe(l.decodeText(), l.splitLines, l.filter((e) => e.trim().length > 0), l.map((e) => JSON.parse(e)), l.mapError((t) => t instanceof H ? t : new H({
|
|
478
481
|
reason: `asset index unreadable: ${String(t)}`,
|
|
479
482
|
path: e
|
|
480
483
|
}))), mt = (e) => n.gen(function* () {
|
|
481
|
-
let t = e.progress ?? ut, r = 0;
|
|
482
|
-
return yield* l.runForEach(pt(e.indexPath), (
|
|
483
|
-
r
|
|
484
|
-
})), n.andThen(t.mark(
|
|
484
|
+
let t = e.progress ?? ut, r = 0, i = /* @__PURE__ */ new Set(), a = /* @__PURE__ */ new Set();
|
|
485
|
+
return yield* l.runForEach(pt(e.indexPath), (o) => t.done(o.key) || a.has(o.key) ? n.void : (a.add(o.key), lt(e.sink, e.assetsDir, o)).pipe(n.andThen(n.sync(() => {
|
|
486
|
+
r++, i.add(o.sha256);
|
|
487
|
+
})), n.andThen(t.mark(o.key, o)))), {
|
|
488
|
+
count: r,
|
|
489
|
+
objects: i.size
|
|
490
|
+
};
|
|
485
491
|
}), ht = () => typeof M.createZstdCompress == "function", gt = (e) => e === "none" ? "none" : e === "gzip" ? "gzip" : ht() ? "zstd" : "gzip", _t = (e) => e === "zstd" ? ".zst" : e === "gzip" ? ".gz" : "", vt = (e) => e === "zstd" ? M.createZstdCompress() : e === "gzip" ? M.createGzip() : new N(), yt = (e) => e === "zstd" ? M.createZstdDecompress() : e === "gzip" ? M.createGunzip() : new N(), bt = 1, xt = c.Struct({
|
|
486
492
|
name: c.String,
|
|
487
493
|
file: c.String,
|
|
@@ -745,7 +751,7 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
745
751
|
event: "start",
|
|
746
752
|
rowsDone: 0
|
|
747
753
|
});
|
|
748
|
-
let f = t.resolved.predicateFor(o), p =
|
|
754
|
+
let f = t.resolved.predicateFor(o), p = C(e, {
|
|
749
755
|
table: o,
|
|
750
756
|
pk: u,
|
|
751
757
|
chunkSize: t.chunkSize,
|
|
@@ -845,14 +851,14 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
845
851
|
mark: (e, r) => (d.assets[e] = !0, ln(t, JSON.stringify(r)).pipe(n.andThen(J(u, d))))
|
|
846
852
|
}
|
|
847
853
|
});
|
|
848
|
-
y = r.count, b = r.totalBytes;
|
|
854
|
+
y = r.count, b = r.totalBytes, r.missing.length > 0 && (yield* n.logWarning(`${r.missing.length} of ${r.missing.length + r.count} blob reference(s) point at objects the storage provider does not have; they are NOT in this bundle and no import can bring them back. e.g. ${r.missing.slice(0, 3).map((e) => `'${e}'`).join(", ")}` + (r.missing.length > 3 ? `, and ${r.missing.length - 3} more` : "")));
|
|
849
855
|
}
|
|
850
856
|
let x = {
|
|
851
857
|
formatVersion: 1,
|
|
852
858
|
createdAt: e.createdAt ?? (/* @__PURE__ */ new Date()).toISOString(),
|
|
853
859
|
source: {
|
|
854
860
|
dialect: e.dialect,
|
|
855
|
-
schemaFingerprint:
|
|
861
|
+
schemaFingerprint: ie(e.snapshot)
|
|
856
862
|
},
|
|
857
863
|
scope: t,
|
|
858
864
|
consistency: m,
|
|
@@ -1197,7 +1203,7 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1197
1203
|
}
|
|
1198
1204
|
return n.length > 30 ? [...n.slice(0, 30), `… and ${n.length - 30} more difference(s)`] : n;
|
|
1199
1205
|
}, Zn = (e, t, n) => {
|
|
1200
|
-
let r =
|
|
1206
|
+
let r = ie(t);
|
|
1201
1207
|
if (r === e || n === void 0) return {
|
|
1202
1208
|
drifted: !1,
|
|
1203
1209
|
bundleFingerprint: e,
|
|
@@ -1335,26 +1341,26 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1335
1341
|
rowsSkipped: 0,
|
|
1336
1342
|
fullyResumed: !1
|
|
1337
1343
|
};
|
|
1338
|
-
let
|
|
1344
|
+
let C = /* @__PURE__ */ new Map();
|
|
1339
1345
|
for (let t of e.targetSnapshot?.tables ?? []) {
|
|
1340
1346
|
let e = t.columns.filter((e) => e.generatedAs !== void 0).map((e) => e.name);
|
|
1341
|
-
e.length > 0 &&
|
|
1347
|
+
e.length > 0 && C.set(t.name, e);
|
|
1342
1348
|
}
|
|
1343
|
-
let
|
|
1344
|
-
let n =
|
|
1349
|
+
let ie = (e, t) => {
|
|
1350
|
+
let n = C.get(e);
|
|
1345
1351
|
if (n === void 0) return t;
|
|
1346
1352
|
let r;
|
|
1347
1353
|
for (let e of n) e in t && (r ??= { ...t }, delete r[e]);
|
|
1348
1354
|
return r ?? t;
|
|
1349
|
-
},
|
|
1350
|
-
let i = k ? X(t.name) : t.name, a = t.primaryKey, o =
|
|
1355
|
+
}, w = (e, t, n, r = !1) => {
|
|
1356
|
+
let i = k ? X(t.name) : t.name, a = t.primaryKey, o = ie(t.name, n);
|
|
1351
1357
|
return p === "append" ? _ === "fail" ? e.insert(i, o) : e.insertIgnore(i, o, { conflictColumns: [a] }) : p === "replace" ? r ? e.upsert(i, o, { conflictColumns: [a] }) : e.insert(i, o) : e.upsert(i, o, { conflictColumns: [a] });
|
|
1352
|
-
},
|
|
1353
|
-
r.unclassified && (
|
|
1354
|
-
},
|
|
1358
|
+
}, T = (e, t) => e.savepoint ? e.savepoint(t) : t(), E = 0, ae = (t, n, r) => {
|
|
1359
|
+
r.unclassified && (E++, !(E > 3) && e.onWarn?.(`row refused with no driver detail — table '${t.name}', row '${String(n.id ?? "?")}'. The full rendering follows so the LAYER that refused is on the record; it is logged, never returned. ${r.raw}` + (E === 3 ? " (further unclassified refusals are counted, not logged)" : "")));
|
|
1360
|
+
}, D = (e, t, r = !0) => n.tryPromise({
|
|
1355
1361
|
try: e,
|
|
1356
1362
|
catch: (e) => Tn(e)
|
|
1357
|
-
}).pipe(n.map(() => !0), n.catchAll((e) => n.sync(() => (r && (t.reason = e.reason,
|
|
1363
|
+
}).pipe(n.map(() => !0), n.catchAll((e) => n.sync(() => (r && (t.reason = e.reason, ae(t.entry, t.row, e)), !1)))), oe = (e) => {
|
|
1358
1364
|
let t = Object.entries(e.entry.columnTypes).filter(([t, n]) => n === "reference" && e.row[t] !== null && e.row[t] !== void 0).map(([e]) => e);
|
|
1359
1365
|
if (t.length !== 0) return {
|
|
1360
1366
|
...e.row,
|
|
@@ -1393,14 +1399,14 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1393
1399
|
for (; r.length > 0 && a;) {
|
|
1394
1400
|
a = !1;
|
|
1395
1401
|
let t = [];
|
|
1396
|
-
for (let n of r) (yield*
|
|
1402
|
+
for (let n of r) (yield* D(() => T(e, () => w(e, n.entry, n.row, !0)), n)) ? a = !0 : t.push(n);
|
|
1397
1403
|
if (!a && t.length > 0) for (let n = t.length - 1; n >= 0; n--) {
|
|
1398
1404
|
let r = t[n], o = oe(r);
|
|
1399
|
-
o !== void 0 && (yield*
|
|
1405
|
+
o !== void 0 && (yield* D(() => T(e, () => w(e, r.entry, o, !0)), r, !1)) && (i.push(r), t.splice(n, 1), a = !0);
|
|
1400
1406
|
}
|
|
1401
1407
|
r = t;
|
|
1402
1408
|
}
|
|
1403
|
-
for (let t of i) (yield*
|
|
1409
|
+
for (let t of i) (yield* D(() => T(e, () => e.upsert(t.entry.name, t.row, { conflictColumns: [t.entry.primaryKey] })), t)) || (r = [...r, t]);
|
|
1404
1410
|
if (r.length > 0) return yield* n.fail(se(r));
|
|
1405
1411
|
}), O = (() => {
|
|
1406
1412
|
let t = e.targetDialect;
|
|
@@ -1472,10 +1478,10 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1472
1478
|
reason: ""
|
|
1473
1479
|
};
|
|
1474
1480
|
try {
|
|
1475
|
-
await
|
|
1481
|
+
await T(t, () => w(t, m, e));
|
|
1476
1482
|
} catch (t) {
|
|
1477
1483
|
let r = Tn(t);
|
|
1478
|
-
n.reason = r.reason,
|
|
1484
|
+
n.reason = r.reason, ae(m, e, r), a.push(n);
|
|
1479
1485
|
}
|
|
1480
1486
|
}, d = yield* n.promise(async () => {
|
|
1481
1487
|
let n = e.copyLoader;
|
|
@@ -1510,13 +1516,13 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1510
1516
|
d = !1, e.onWarn?.(`bulk COPY into ${m.name} failed (${String(n?.message ?? n)}); falling back to per-row writes for this table`);
|
|
1511
1517
|
for (let e of t) await u(e);
|
|
1512
1518
|
}
|
|
1513
|
-
}, b = Math.max(1, e.savepointBatchSize ?? 200), x = typeof t.savepoint == "function", S = [],
|
|
1519
|
+
}, b = Math.max(1, e.savepointBatchSize ?? 200), x = typeof t.savepoint == "function", S = [], ee = async () => {
|
|
1514
1520
|
if (S.length === 0) return;
|
|
1515
1521
|
let e = S;
|
|
1516
1522
|
S = [];
|
|
1517
1523
|
try {
|
|
1518
1524
|
await t.savepoint(async () => {
|
|
1519
|
-
for (let n of e) await
|
|
1525
|
+
for (let n of e) await w(t, m, n);
|
|
1520
1526
|
});
|
|
1521
1527
|
} catch {
|
|
1522
1528
|
for (let t of e) await u(t);
|
|
@@ -1528,20 +1534,20 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1528
1534
|
return;
|
|
1529
1535
|
}
|
|
1530
1536
|
if (x) {
|
|
1531
|
-
S.push(e), S.length >= b && await
|
|
1537
|
+
S.push(e), S.length >= b && await ee();
|
|
1532
1538
|
return;
|
|
1533
1539
|
}
|
|
1534
1540
|
await u(e);
|
|
1535
|
-
})), yield* n.promise(
|
|
1541
|
+
})), yield* n.promise(ee), yield* n.promise(y), {
|
|
1536
1542
|
rowCount: o,
|
|
1537
1543
|
checksum: i.digest("hex")
|
|
1538
1544
|
};
|
|
1539
|
-
}), { rowCount: S, checksum:
|
|
1545
|
+
}), { rowCount: S, checksum: ee } = yield* Lt("import", m.name, d, h, (e) => e.rowCount, b);
|
|
1540
1546
|
if (c) {
|
|
1541
|
-
if (
|
|
1547
|
+
if (ee !== m.checksum) return yield* n.fail(new Je({
|
|
1542
1548
|
what: `table ${m.name} checksum`,
|
|
1543
1549
|
expected: m.checksum,
|
|
1544
|
-
actual:
|
|
1550
|
+
actual: ee
|
|
1545
1551
|
}));
|
|
1546
1552
|
if (S !== m.rowCount) return yield* n.fail(new Je({
|
|
1547
1553
|
what: `table ${m.name} rowCount`,
|
|
@@ -1603,7 +1609,7 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1603
1609
|
path: F.dir
|
|
1604
1610
|
})
|
|
1605
1611
|
})), e.onWarn?.(Kn(F)));
|
|
1606
|
-
let ve = (e) => p === "replace" ?
|
|
1612
|
+
let ve = (e) => p === "replace" ? te(e) : e(), I = (e) => p === "replace" ? ne(e) : e, L = e.traceId ?? m("dimp", `${Date.now()}-${s.source.schemaFingerprint ?? ""}`), ye = e.traceId !== void 0, R = (e.targetSnapshot?.tables.some((e) => e.name === u) ?? !1) || v(u) !== void 0;
|
|
1607
1613
|
R && h(f), R && e.traceId === void 0 && (yield* n.promise(() => e.store.insert(u, {
|
|
1608
1614
|
id: L,
|
|
1609
1615
|
mode: p,
|
|
@@ -1647,9 +1653,9 @@ voltro data import <this-directory> --target api --api-url <url> --token <secret
|
|
|
1647
1653
|
let t = e.store, r = de.map((e) => e.name), i = Fn(r, {
|
|
1648
1654
|
getTable: (e) => v(e),
|
|
1649
1655
|
registerTable: (e) => {
|
|
1650
|
-
|
|
1656
|
+
ee(e);
|
|
1651
1657
|
},
|
|
1652
|
-
unregisterTable:
|
|
1658
|
+
unregisterTable: re
|
|
1653
1659
|
}), a = !1;
|
|
1654
1660
|
try {
|
|
1655
1661
|
let i = g ? r : r.filter((e) => !Bt(x, e));
|
|
@@ -1735,7 +1741,7 @@ async function* mr(e) {
|
|
|
1735
1741
|
}
|
|
1736
1742
|
if (e.blobs) for await (let t of e.blobs.list()) {
|
|
1737
1743
|
yield new Uint8Array(pr(t.key, t.contentType, t.size));
|
|
1738
|
-
let n =
|
|
1744
|
+
let n = E("sha256"), r = 0;
|
|
1739
1745
|
for await (let i of e.blobs.open(t.key)) {
|
|
1740
1746
|
let e = Buffer.from(i);
|
|
1741
1747
|
n.update(e), r += e.length, yield new Uint8Array(e);
|
|
@@ -1789,7 +1795,7 @@ var hr = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), gr
|
|
|
1789
1795
|
await s(i);
|
|
1790
1796
|
let a = c(i).toString("utf8");
|
|
1791
1797
|
await s(8);
|
|
1792
|
-
let o = c(8).readBigUInt64BE(0), l = await t.blobs.has(n), u =
|
|
1798
|
+
let o = c(8).readBigUInt64BE(0), l = await t.blobs.has(n), u = E("sha256"), d = new N(), f = l ? Promise.resolve() : t.blobs.write(n, d, {
|
|
1793
1799
|
contentType: a,
|
|
1794
1800
|
size: Number(o)
|
|
1795
1801
|
}), p = o;
|
|
@@ -1849,7 +1855,7 @@ var hr = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), gr
|
|
|
1849
1855
|
write: async (n, r, i) => {
|
|
1850
1856
|
let a = await import("node:fs/promises"), { pipeline: o } = await import("node:stream/promises"), { Transform: s } = await import("node:stream"), c = R(e, "assets");
|
|
1851
1857
|
await P(c, { recursive: !0 });
|
|
1852
|
-
let l =
|
|
1858
|
+
let l = E("sha256"), u = 0, d = new s({ transform(e, t, n) {
|
|
1853
1859
|
l.update(e), u += e.length, n(null, e);
|
|
1854
1860
|
} }), f = R(c, `.tmp-${Date.now()}-${t.size}`);
|
|
1855
1861
|
await o(r, d, O(f));
|
|
@@ -1872,13 +1878,13 @@ var hr = (e, t) => new Promise((n, r) => e.write(t, (e) => e ? r(e) : n())), gr
|
|
|
1872
1878
|
};
|
|
1873
1879
|
async function* Mr(e, t) {
|
|
1874
1880
|
if (!t) throw Error("encryptStream: a non-empty passphrase is required");
|
|
1875
|
-
let n =
|
|
1881
|
+
let n = D(wr), r = D(Tr), i = await xr(t, n, Cr, {
|
|
1876
1882
|
...Or,
|
|
1877
1883
|
maxmem: kr
|
|
1878
1884
|
}), a = Buffer.alloc(Ar);
|
|
1879
1885
|
Sr.copy(a, 0), a.writeUInt32BE(Or.N, 8), a.writeUInt32BE(Or.r, 12), a.writeUInt32BE(Or.p, 16), n.copy(a, 20), r.copy(a, 36), yield new Uint8Array(a);
|
|
1880
1886
|
let o = 0, s = (e, t) => {
|
|
1881
|
-
let n =
|
|
1887
|
+
let n = w("aes-256-gcm", i, jr(r, o, t));
|
|
1882
1888
|
o === 0 && n.setAAD(a);
|
|
1883
1889
|
let s = Buffer.concat([n.update(e), n.final()]), c = n.getAuthTag();
|
|
1884
1890
|
o++;
|
|
@@ -1919,7 +1925,7 @@ async function* Nr(e, t) {
|
|
|
1919
1925
|
if (!await o(5 + t)) throw Error("encrypted bundle truncated (frame body)");
|
|
1920
1926
|
let n = r.subarray(5, 5 + t - Er), i = r.subarray(5 + t - Er, 5 + t);
|
|
1921
1927
|
r = r.subarray(5 + t);
|
|
1922
|
-
let a =
|
|
1928
|
+
let a = T("aes-256-gcm", p, jr(f, m, e));
|
|
1923
1929
|
m === 0 && a.setAAD(s), a.setAuthTag(i);
|
|
1924
1930
|
let c = Buffer.concat([a.update(n), a.final()]);
|
|
1925
1931
|
m++, e && (h = !0), c.length > 0 && (yield new Uint8Array(c));
|
|
@@ -1943,7 +1949,7 @@ var Pr = (e, t) => ({
|
|
|
1943
1949
|
let r = await n.runPromise(A(e, t)), i = await he(r.stream);
|
|
1944
1950
|
for await (let e of i) yield e instanceof Uint8Array ? e : new Uint8Array(e);
|
|
1945
1951
|
} })
|
|
1946
|
-
}), Fr = (e, t) => Pr(e, () =>
|
|
1952
|
+
}), Fr = (e, t) => Pr(e, () => C(t, {
|
|
1947
1953
|
table: pe,
|
|
1948
1954
|
chunkSize: 500
|
|
1949
1955
|
}).pipe(l.map((e) => ({
|
|
@@ -2001,7 +2007,7 @@ var Pr = (e, t) => ({
|
|
|
2001
2007
|
"--format=custom",
|
|
2002
2008
|
"--no-owner",
|
|
2003
2009
|
"--no-privileges",
|
|
2004
|
-
...
|
|
2010
|
+
...Wr,
|
|
2005
2011
|
"--dbname",
|
|
2006
2012
|
t.url,
|
|
2007
2013
|
"--file",
|
|
@@ -2020,7 +2026,7 @@ var Pr = (e, t) => ({
|
|
|
2020
2026
|
"--format=custom",
|
|
2021
2027
|
"--no-owner",
|
|
2022
2028
|
"--no-privileges",
|
|
2023
|
-
...
|
|
2029
|
+
...Wr,
|
|
2024
2030
|
"-h",
|
|
2025
2031
|
e.host,
|
|
2026
2032
|
"-p",
|
|
@@ -2047,7 +2053,7 @@ var Pr = (e, t) => ({
|
|
|
2047
2053
|
"--triggers",
|
|
2048
2054
|
"--no-tablespaces",
|
|
2049
2055
|
"--protocol=TCP",
|
|
2050
|
-
`--ignore-table=${r.database}.${
|
|
2056
|
+
...Ur.map((e) => `--ignore-table=${r.database}.${e}`),
|
|
2051
2057
|
"-h",
|
|
2052
2058
|
r.host,
|
|
2053
2059
|
"-P",
|
|
@@ -2084,7 +2090,7 @@ var Pr = (e, t) => ({
|
|
|
2084
2090
|
reason: `no native backup for dialect '${e}'`
|
|
2085
2091
|
});
|
|
2086
2092
|
}
|
|
2087
|
-
}, Ur = ["--exclude-table",
|
|
2093
|
+
}, Ur = [d, u], Wr = Ur.flatMap((e) => ["--exclude-table", e]), Gr = (e, t, n) => {
|
|
2088
2094
|
switch (e) {
|
|
2089
2095
|
case "postgres": {
|
|
2090
2096
|
if (t.url) return {
|
|
@@ -2168,7 +2174,7 @@ var Pr = (e, t) => ({
|
|
|
2168
2174
|
reason: `no native restore for dialect '${e}'`
|
|
2169
2175
|
});
|
|
2170
2176
|
}
|
|
2171
|
-
},
|
|
2177
|
+
}, Kr = (e) => e.kind === "copy" ? n.async((t) => {
|
|
2172
2178
|
let r = `${e.to}.restoring-${oe()}`, i = (e) => {
|
|
2173
2179
|
try {
|
|
2174
2180
|
k(r);
|
|
@@ -2243,7 +2249,7 @@ var Pr = (e, t) => ({
|
|
|
2243
2249
|
return;
|
|
2244
2250
|
}
|
|
2245
2251
|
u();
|
|
2246
|
-
let r = e.stdinFile === void 0 ? "" : ` — ${l} of ${
|
|
2252
|
+
let r = e.stdinFile === void 0 ? "" : ` — ${l} of ${qr(e.stdinFile)} byte(s) of ${I(e.stdinFile)} had been fed to it`;
|
|
2247
2253
|
o(n.fail(new W({
|
|
2248
2254
|
tool: e.tool,
|
|
2249
2255
|
reason: (t === 0 ? `exited cleanly, but ${s}` : `exited with code ${t}${s === void 0 ? "" : ` (${s})`}`) + r + (e.stdoutFile === void 0 ? "" : ` — the partial ${I(e.stdoutFile)} was removed, so it cannot be mistaken for a backup`),
|
|
@@ -2251,13 +2257,13 @@ var Pr = (e, t) => ({
|
|
|
2251
2257
|
...e.hint === void 0 ? {} : { hint: e.hint }
|
|
2252
2258
|
})));
|
|
2253
2259
|
});
|
|
2254
|
-
}),
|
|
2260
|
+
}), qr = (e) => {
|
|
2255
2261
|
try {
|
|
2256
2262
|
return String(fe(e).size);
|
|
2257
2263
|
} catch {
|
|
2258
2264
|
return "?";
|
|
2259
2265
|
}
|
|
2260
|
-
},
|
|
2266
|
+
}, Jr = (e) => typeof e == "object" && e ? "fake" in e ? `fake:${e.fake}` : "custom" : String(e), Yr = (e) => n.gen(function* () {
|
|
2261
2267
|
let t = yield* at(e.scope ?? { kind: "all" }, e.snapshot, {
|
|
2262
2268
|
store: e.store,
|
|
2263
2269
|
...e.tenantTables ? { tenantTables: e.tenantTables } : {}
|
|
@@ -2269,7 +2275,7 @@ var Pr = (e, t) => ({
|
|
|
2269
2275
|
}))
|
|
2270
2276
|
})), i, e.masking), s = [], c = [];
|
|
2271
2277
|
for (let i of r) {
|
|
2272
|
-
let r = t.predicateFor(i), u = yield*
|
|
2278
|
+
let r = t.predicateFor(i), u = yield* C(e.store, {
|
|
2273
2279
|
table: i,
|
|
2274
2280
|
chunkSize: a,
|
|
2275
2281
|
...r === void 0 ? {} : { where: r }
|
|
@@ -2281,7 +2287,7 @@ var Pr = (e, t) => ({
|
|
|
2281
2287
|
let n = e.slice(i.length + 1);
|
|
2282
2288
|
m.push({
|
|
2283
2289
|
column: n,
|
|
2284
|
-
action:
|
|
2290
|
+
action: Jr(t),
|
|
2285
2291
|
before: f[n],
|
|
2286
2292
|
after: p[n]
|
|
2287
2293
|
});
|
|
@@ -2296,7 +2302,7 @@ var Pr = (e, t) => ({
|
|
|
2296
2302
|
leaks: c,
|
|
2297
2303
|
unclassified: o.unclassified
|
|
2298
2304
|
};
|
|
2299
|
-
}),
|
|
2305
|
+
}), Xr = (e, t) => {
|
|
2300
2306
|
let n = new Map(t.map((e) => [e.name, e])), r = [], i = {};
|
|
2301
2307
|
for (let t of e) {
|
|
2302
2308
|
let e = n.get(t.name);
|
|
@@ -2351,7 +2357,7 @@ var Pr = (e, t) => ({
|
|
|
2351
2357
|
skipColumns: i,
|
|
2352
2358
|
refuse: r.some((e) => e.verdict === "refuse")
|
|
2353
2359
|
};
|
|
2354
|
-
},
|
|
2360
|
+
}, Zr = (e) => {
|
|
2355
2361
|
if (e.findings.length === 0) return [" schema matches the bundle"];
|
|
2356
2362
|
let t = {
|
|
2357
2363
|
safe: "✓",
|
|
@@ -2362,4 +2368,4 @@ var Pr = (e, t) => ({
|
|
|
2362
2368
|
return n.push(""), n.push(` discarded: ${r("value-dropped")} defaulted: ${r("filled-by-default")} refused: ${r("refuse")}`), e.refuse && (n.push(""), n.push(" REFUSED — no row was written. These are the differences that would make"), n.push(" imported rows WRONG rather than merely incomplete.")), n;
|
|
2363
2369
|
};
|
|
2364
2370
|
//#endregion
|
|
2365
|
-
export { kt as ASSETS_DIR, At as ASSET_INDEX_FILE, H as BundleError, qe as CodecError, Ye as CompressionError, et as CrossDialectError, Ot as DATA_DIR, Te as DEFAULT_CLASS_ACTIONS, Ue as ENVIRONMENT_LOCAL_TABLES, We as ENVIRONMENT_PORTABLE_TABLES, bt as FORMAT_VERSION, $e as ImportModeError, Je as IntegrityError, jt as LEDGER_FILE, K as MANIFEST_FILE, Qe as MaskingError, W as NativeToolError, Xe as RowsRefusedError, $n as SAVEPOINT_BATCH_SIZE, On as STAGING_PREFIX, Ze as SchemaDriftError, U as ScopeError, Le as applyAction, tr as asImportError, or as assessLiveImport, Vt as assetDone, zr as backupArtifactName, Hr as backupCommand, gn as canonicalType, ft as captureAssetsToCas, yn as checkPortability, Zn as checkSchemaDrift, Ee as classificationFromSnapshot,
|
|
2371
|
+
export { kt as ASSETS_DIR, At as ASSET_INDEX_FILE, H as BundleError, qe as CodecError, Ye as CompressionError, et as CrossDialectError, Ot as DATA_DIR, Te as DEFAULT_CLASS_ACTIONS, Ue as ENVIRONMENT_LOCAL_TABLES, We as ENVIRONMENT_PORTABLE_TABLES, Ur as EXCLUDED_FROM_DUMP, bt as FORMAT_VERSION, $e as ImportModeError, Je as IntegrityError, jt as LEDGER_FILE, K as MANIFEST_FILE, Qe as MaskingError, W as NativeToolError, Xe as RowsRefusedError, $n as SAVEPOINT_BATCH_SIZE, On as STAGING_PREFIX, Ze as SchemaDriftError, U as ScopeError, Le as applyAction, tr as asImportError, or as assessLiveImport, Vt as assetDone, zr as backupArtifactName, Hr as backupCommand, gn as canonicalType, ft as captureAssetsToCas, yn as checkPortability, Zn as checkSchemaDrift, Ee as classificationFromSnapshot, Xr as classifyImportDrift, _t as compressionExt, nt as computeSubsetIds, Wn as decideStaging, Et as decodeManifest, Zt as decodeRow, Yt as decodeValue, Nr as decryptStream, ar as defineDataProfile, Xn as diffSnapshots, br as dirBlobSink, Nn as dropStagingSql, Rt as emptyLedger, Tt as encodeManifest, Xt as encodeRow, qt as encodeValue, Mr as encryptStream, Ke as environmentLocalNotice, Ge as environmentLocalReason, zn as execStatements, Zr as formatImportDrift, bn as formatIssue, ht as hasZstd, V as isEnvironmentLocal, vt as makeCompressor, yt as makeDecompressor, He as maskRow, en as ndjsonToRows, ut as noAssetProgress, hn as outsideReferenceRefusal, mn as outsideReferencesInto, mr as packBundle, vr as peekBundle, gt as pickCompression, ze as planMasking, Yr as previewMasking, pt as readAssetIndex, zt as readLedger, De as resolveAction, at as resolveScope, lt as restoreAssetFromCas, mt as restoreAssetsFromCas, Gr as restoreCommand, $t as rowsToNdjson, pn as runExport, ir as runImport, Kr as runNativeStep, Ve as scanForLeaks, X as stagingNameFor, st as storageAssetSink, ot as storageAssetSource, Ir as storageBlobSink, Pr as storageBlobSource, Fr as storageRefsBlobSource, rt as subsetToScope, Bt as tableDone, tn as topoSortTables, _r as unpackBundle, yr as unpackBundleToDir, ct as writeAssetToCas, J as writeLedger };
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@voltro/data-transfer",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.51.0",
|
|
4
4
|
"description": "Portable export/import for a Voltro app's data + assets — typed-NDJSON row codec, content-addressed assets, framed compression, a resumable chunk ledger, and FK-ordered idempotent import. Dialect-agnostic; the resilient backbone under `voltro data export|import` and native backups.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"voltro",
|
|
@@ -33,8 +33,8 @@
|
|
|
33
33
|
"node": ">=24.0.0"
|
|
34
34
|
},
|
|
35
35
|
"dependencies": {
|
|
36
|
-
"@voltro/database": "0.
|
|
37
|
-
"@voltro/plugin-storage": "0.
|
|
36
|
+
"@voltro/database": "0.51.0",
|
|
37
|
+
"@voltro/plugin-storage": "0.51.0"
|
|
38
38
|
},
|
|
39
39
|
"peerDependencies": {
|
|
40
40
|
"effect": "^3.22.0"
|