@voltro/data-transfer 0.49.0 → 0.50.1

package/CHANGELOG.md

@@ -39,6 +39,120 @@ _Changes staged for the next release accumulate here (rolled up from
 
 ---
 
+## [0.50.1] — 2026-08-24
+
+### Added
+
+- **@voltro/cli** — A `0.51.0` upgrade note for the case 0.45.0's typed-`source:` note left open: a source that is COMPUTED but whose names are all known — a generic reader that takes its table from `input` or a registry. Both remedies the original offered are wrong there. There is no misspelled name to fix, and widening to `ReactivitySourceValue` (or casting) compiles instantly while taking that entire set of tables permanently out of the check the narrowing exists to provide.
+
+  The note carries the recipe that keeps the names — `as const satisfies`, with the key set recovered separately — and the trap under it: an ANNOTATION widens the keys back even when the literal carries `as const`, because it is checked against the literal and then replaces its type. Filed under an unreached version because codemod selection is `from < version <= to`: 0.45.0's note can never fire again for anyone already past it. Documented on the subscriptions page in both languages.
+
+### Changed
+
+- **@voltro/data-transfer** — The `mariadb-dump`-not-on-PATH hint now also warns that a MariaDB 12.x client requires TLS by default, so running it BY HAND against a server without TLS fails with `TLS/SSL error: SSL is required` (2026) and needs `--skip-ssl`. Our own invocation is unaffected. It is said here because this hint is what sends people to install that client package, and the next thing many of them do is run the tool by hand — where the failure reads like a broken install of the thing we just told them to install.
+
+### Fixed
+
+- **@voltro/cli, @voltro/mcp** — `voltro check`'s `rbac/unguarded-mutation` no longer fires on `internal: true` procedures. Those are in no rpc group and on no route, so the finding's own sentence — "any caller who can reach the rpc surface can invoke it" — named a surface that does not exist, and neither remedy it offered was available: a guard protects nothing there, and `openAccess` is refused outright on an internal descriptor. An app with 990 internal mutations got 990 warnings and a `FAILED` exit from the same tree on which `voltro doctor` reported zero procedures without an access decision, which made `check` unusable as the CI gate it exists to be.
+
+  The capability manifest now carries `internal: true` per procedure (present only when true), and the rule reads it through `isWireReachable` — the same predicate the boot access gate and all three rpc-group assemblies use, rather than a fourth spelling of the same flag. A manifest that does not carry the field is read as REACHABLE, so the rule stays loud on an api older than the field rather than going quiet on it.
+- **@voltro/data-transfer, @voltro/cli** — `--assets` aborted the whole capture at the first blob reference the storage provider could not resolve. A dangling reference is a fact about the source data — a row pointing at an object that was deleted, or that never arrived because an earlier import ran without the flag — and no backup can put back bytes that are not there. Aborting made the flag unusable for exactly the deployment that needed it: 178 references, one resolvable, and the run stopped at the second.
+
+  The phase now records the key, steps over it, and the run reports how many were skipped. Only a genuine not-found (`status === 404`) is treated this way; a 403 from a rotated credential or a 5xx from a backend outage still fails the capture, because calling those "the object is gone" turns a recoverable outage into a backup that quietly contains nothing.
+
+  The same abort also cost the artefact its provenance: `voltro-backup-stamp.json` was written after the asset phase, so a backup taken WITH `--assets` had no stamp at all, and `restore` then greeted an artefact this tool had written minutes earlier as "an older/handmade backup. Cannot verify dialect or schema version." The stamp is now written on every path — it describes the dump, and the dump is on disk and correct by the time the assets run.
+- **@voltro/cli** — `voltro doctor`'s eager-loaded-relation rule resolves nested relations against their PARENT's target table instead of against the query's roots, so every level is reported in one pass. It flattened the `.with({ … })` tree before resolving, which meant a level-2 relation stayed invisible until level 1's table had been added to `source:` — each fix revealed the next level, and a clean run after the first fix meant nothing. One app ran `doctor → fix → doctor` twice before the output stopped producing new findings.
+
+  The one case that is still genuinely under-reported is now stated instead of being silent: when the executor's base table is neither declared nor textually readable, nothing resolves, and doctor prints which relation names it could not check rather than reporting clean.
+- **@voltro/data-transfer** — A native restore whose client exited before consuming its input took the whole CLI down with an unhandled `EPIPE`. `createReadStream(dump).pipe(child.stdin)` had no `error` listener, so Node threw on a write to a pipe with no reader:
+
+  Error: write EPIPE Emitted 'error' event on Socket instance at: at Socket.onerror (node:internal/streams/readable:1045:14)
+
+  What that cost is the point. The crash landed before `close`, so the step never resumed: the client's stderr — the one place the reason was written down — was discarded, the transfer row was never closed, the in-progress marker was never re-asserted, and the operator got a raw Node stack trace over a database that was now half replaced. A deployment met it on a 64 MB restore and could not diagnose it at all, because the only diagnosis had been thrown away.
+
+  Every stream in the step now carries a handler. `EPIPE` on the child's stdin is deliberately swallowed — a child closes the pipe by dying, and its exit code and stderr are the actual failure — while any other stream error is carried and reported. The failure line now always includes the tool's stderr (and says so explicitly when the tool printed none), plus how many bytes of the artefact had been fed before it stopped. A source error also ends the child's stdin, which turns a restore that used to hang forever on a vanished dump into one that fails, and a clean exit over a broken input stream is reported as a failure rather than as a truncated success.
+- **@voltro/cli** — `voltro check`'s observed-vs-declared findings advised an op that does not exist. The recorder reports what the store did, and `store.upsert` is one call; a `target:` is `InsertTarget | UpdateTarget | DeleteTarget` and has no `upsert` member. So `observed/undeclared-write` said to add `{ table: '…', op: 'upsert' }` — a `TS2322`, from a line whose whole purpose is to be pasted. It now advises both halves, and the observed → declarable mapping is typed `Record<ObservedWriteOp, ReadonlyArray<TargetSpec['op']>>` so a new recorder op cannot ship without a declarable answer.
+
+  The remedy no longer produces a finding of its own either: declared targets are now collected per table as a SET. `new Map(targets.map(t => [t.table, t.op]))` kept only the last op, so the correct two-entry declaration read as "declares update" and the insert half came back as a `wrong-op`.
+- **@voltro/data-transfer, @voltro/cli** — The in-progress restore marker did not survive a native restore, on the exact dialects that were documented as safe. The claim — postgres and the mysql family drop only the objects the dump names, so the marker row survives — had a correct argument and a wrong premise: a native dump names the WHOLE database, `_voltro_replace_in_progress` included, and a mysql-family restore writes `DROP TABLE IF EXISTS` in front of each table. The table sorts early, so the guard was removed near the START of the window it exists to cover. Measured downstream as one row before and zero after, twice, once by hand and once through the command — and then a restore died and left a database with a schema, some of its tables, no users and no blobs, with nothing to stop the next boot.
+
+  Two changes, covering different dumps. `voltro data backup` now excludes the marker table (`--exclude-table` / `--ignore-table`), so an artefact we produce cannot carry the thing that erases the guard on the way back in. And `restore` re-asserts the marker after the tool exits, on the failing path as well as the succeeding one, which covers dumps taken before this change and dumps made by hand: rewritten is a warning, and unwritable is an error, because the guard is then off for that run.
+- **@voltro/plugin-storage, @voltro/cli** — `--assets` could not be refused when no storage was configured, and resolved the wrong backend when it was.
+
+  `resolveStorageProvider({})` is total: its final `default:` arm returns an in-process `memoryProvider`. So the three `if (!provider)` refusals in `voltro data backup` / `export` / `restore` — whose entire job is to refuse a flag the app cannot honour — were checking a condition that cannot hold. What they let through is worse than an unchecked flag: with no storage configured, `backup --assets` captured from a fresh memory provider, found nothing in it, and wrote an artefact stamped `assets: { count: 0 }` — a rollback story that says the blobs are in there.
+
+  It also ignored the provider the app REGISTERED. An app on `storagePlugin({ provider: s3(…) })` had its export/import/backup read and write the env-derived default instead, which surfaces later as "no object at key" — the exact drift `appStorageProvider()` exists to prevent, in call sites that never adopted it.
+
+  `configuredStorageProvider()` is the honest predicate: the app's registered provider, else the env-named one, else `undefined`. A memory provider nobody asked for is not a decision. The CLI now loads the app config BEFORE resolving — `storagePlugin(...)` registers at construction, so asking first answered with the default no matter what the app had configured.
+
+### Internal (no consumer-facing effect)
+
+- **@voltro/cli** — `voltro data backup` / `restore` / `clear-replace-marker` are now driven end to end, one case per declared option and one per refusal, as real CLI subprocesses against a real database.
+
+  Every defect this command has shipped lived in the SEQUENCE rather than in a function — an unhandled `EPIPE` that discarded the client's stderr before `close` fired, a marker erased by the restore's own artefact, an asset abort that skipped the provenance stamp, a refusal that could not fire — so none of them was reachable by testing a part. `nativeBackupRestore.e2e.test.ts` covers the whole surface against sqlite (a file copy: no vendor binary, runs anywhere) and `nativeBackupRestoreDialects.integration.test.ts` runs the vendor-tool half against postgres, mariadb and mysql, skipping BY NAME when the server or the client binary is absent — including when the client is present but too old for the server, which a "is it on PATH" check reports as ready.
+
+  The suite checks itself against `DATA_FLAGS`, the command's own declared option list, so an option cannot be added without a case that drives it.
+
+---
+
+## [0.50.0] — 2026-08-23
+
+### ⚠ BREAKING
+
+- **@voltro/cli** — The read-only run-history endpoint moved:
+
+  GET /_voltro/admin/imports → GET /_voltro/admin/transfers
+
+  Same secret, same query parameters, same response shape. Only the path changed.
+
+  It answers for four directions now — import, export, native backup, native restore — and it named one of them. The subcommand and the table were renamed for exactly that reason in 0.49.0 and this path was left behind, which is worse than renaming none of them: a reader who follows a rename tries the matching path, gets a 404, and concludes the instance is too old.
+
+  `voltro data transfers --target api` uses the new path for you. What needs a hand is anything calling it directly — a monitoring check, an uptime probe, a curl in a runbook, a dashboard datasource. The codemod is `manual` and declares `reach: 'beyond-source'`, so it prints whether or not the path is found in your repository.
+- **@voltro/database, @voltro/voltro** — `InterruptedReplace.tables` is `number | null`, and the interface gains `kind?: 'replace' | 'restore'`.
+
+  `_voltro_replace_in_progress` records two destructive operations now — an import's `--mode replace` and a native `voltro data restore`. A restore replaces the whole database from an artefact, so "how many tables was this going to empty and refill" has no answer: not known up front, not meaningful after.
+
+  Writing `0` would have preserved the type and been worse. That number is rendered into the boot refusal, which is read under pressure, and it would have said `began emptying 0 table(s)` — a measurement that was never taken. The framework's own text says "this database" instead; `Number(null)` is `0`, so coercing it reintroduces exactly the sentence being avoided.
+
+  `DataTransferRun.direction` also widens to name `'backup'` and `'restore'` explicitly. That one is documentary — the union already ended in `| string`, so no assignability changes.
+
+  **`voltro update` carries you across this** — codemod `0.50.0/03_interrupted-replace-tables-nullable`.
+
+### Added
+
+- **@voltro/cli, @voltro/data-transfer, @voltro/database** — `voltro data backup --assets` and `voltro data restore --assets` now move the stored blobs alongside the vendor dump, through the same content-addressed phase the logical `export` / `import` path uses — streamed, deduped by sha256, verified on the way back, resumable per key.
+
+  `--assets` was accepted by `backup` and silently ignored, with the only signal a field in the closing JSON reading `NOT included`; `restore` did not accept it at all. A rows-only backup restores a database whose rows reference objects nothing puts back, and the reference and the object are checked at different times, so that state is discovered by a user rather than by the restore.
+
+  Three refusals, each for a belief that is otherwise acted on silently: `backup --assets` with no storage provider is refused rather than swallowed; `restore --assets` on a rows-only artefact is refused; a restore WITHOUT `--assets` over an artefact that has them warns and proceeds, because restoring rows without blobs is legitimate and refusing it would push people at `--force`.
+
+  The dump itself still has no resume — a vendor artefact is one opaque file with no offset to restart from, and the logical path is what exists for that.
+- **@voltro/cli, @voltro/database, @voltro/data-transfer** — A native `backup` / `restore` now writes to the same `_voltro_data_transfers` record `import` and `export` use, so `voltro data transfers` answers "did last night's backup finish" from the instance that ran it. A native run reports blobs rather than rows — a vendor tool reports no row count we can trust, and printing `0 row(s)` over a dump that worked would be a wrong measurement.
+
+  `restore` also writes the `_voltro_replace_in_progress` marker before its first destructive statement and clears it after the last write, blobs included, so a killed restore refuses the next boot instead of serving a half-loaded database. `--allow-live` guards from the wrong side — it asks you not to — and this guards from the right one. The marker carries a `kind` so the refusal can say something different for a `replace` (re-import the capture) and a `restore` (finish the restore), and `tables` is nullable because a native restore has no table count to claim.
+
+  The sqlite / turso restore is atomic now (temp file + rename): its marker lives in the very file being replaced, so a plain in-place copy would leave a truncated database with nothing left to catch it.
+
+  A target with no `_voltro_data_transfers` table still gets its backup; the closing line says it was not recorded rather than implying it was.
+- **@voltro/cli** — A re-issued migration note for the `voltro data imports` → `voltro data transfers` rename that shipped in 0.49.0.
+
+  Codemods are selected by `from < version <= to`, so the 0.49.0 note fires once, on the jump that crosses 0.49.0, and cannot be corrected for anyone already past it. Its gate searched `.ts` / `.tsx` while the command it is about lives in shell scripts, CI job definitions and runbooks — so a project whose only occurrence sat in `.gitlab-ci.yml` crossed 0.49.0 and was told there was nothing to apply.
+
+  The re-issue is filed under 0.50.0, prints unconditionally, and says what to grep for. Redundant for anyone already fixed; the alternative is firing for nobody.
+
+### Fixed
+
+- **@voltro/cli** — A `manual` codemod's `appliesTo` can now search every text file the project owns — `.sh`, `.yml`, `.json`, `.md`, `Makefile`, the `.js` scripts — through a new `ctx.text` on the predicate context, and a codemod may declare `reach: 'beyond-source'` to print its note even when nothing matched.
+
+  Both halves close the same gap, and it was in the codemod most in need of a gate. A manual codemod exists BECAUSE its subject could not be transformed, which usually means it is not source at all — and `appliesTo` was reading the ts-morph project, which holds `.ts` / `.tsx` and nothing else. Measured with one identical CLI invocation in four files: the `.ts` one printed the note, the `.sh`, `.yml` and `.md` ones printed nothing, and the run reported `codemods: nothing to apply for this jump`. That is an acquittal from a check that never looked at the file, and an acquittal gets acted on.
+
+  `reach: 'beyond-source'` covers what no scan of one repository can reach — an inline script in a CI runner's own UI, a CronJob spec in another repo, a wiki runbook. The note prints either way; only its framing changes, and an uncertain one says plainly that we could not look there.
+- **@voltro/data-transfer** — When `mariadb-dump` is absent and the mysql-family fallback runs Oracle's `mysqldump` against a MariaDB server, the failure now names the way out.
+
+  The stderr it prints is the child's own words and is the right first thing to show — and on its own it is a dead end: `Unknown table 'COLUMN_STATISTICS' in information_schema (1109)` names a table nobody asked for, in a schema nobody wrote, about a feature nobody enabled. Everything needed to act on it was known where the fallback was DECIDED: which binary we wanted, which one we took, and why the difference matters. That travels with the step now and prints under the stderr, including the counter-move a reader reaches for on their own (`--column-statistics=0` does not exist on `mariadb-dump`, so it fixes the wrong client and breaks the right one).
+
+---
+
 ## [0.49.0] — 2026-08-23
 
 ### ⚠ BREAKING

package/dist/index.d.ts

@@ -76,6 +76,23 @@ export declare interface AssetInfo {
     readonly size: number;
 }
 
+/**
+ * The resume journal, as this phase needs it.
+ *
+ * Deliberately two functions rather than the `Ledger` type: the native path
+ * keeps its own sidecar and the logical path keeps the shared one, and neither
+ * has any business knowing the other's file layout. What both must agree on is
+ * the SEMANTICS — `done` is keyed by the asset KEY (not its hash), because an
+ * asset is skippable when the destination already has that key, and the hash is
+ * not known until the bytes have been read, which is the cost being avoided.
+ */
+export declare interface AssetProgress {
+    readonly done: (key: string) => boolean;
+    /** Record `key` as landed. Called AFTER the bytes are durable, never before —
+     *  a ledger written first turns a crash into silent data loss on resume. */
+    readonly mark: (key: string, entry: AssetEntry) => Effect.Effect<void, BundleError>;
+}
+
 export declare const ASSETS_DIR = "assets";
 
 /** Where import writes assets TO. */
@@ -149,6 +166,41 @@ export declare type CanonicalCategory = 'string' | 'integer' | 'float' | 'boolea
  *  types (a bundle from a newer framework) fall back to 'string' passthrough. */
 export declare const canonicalType: (type: string) => CanonicalCategory;
 
+/**
+ * Stream every asset the source enumerates into the content-addressed `assetsDir`.
+ *
+ * `assetsDir` must exist. Each asset is hashed while it streams (never buffered
+ * whole), written to a temp file and renamed to `assets/<sha256>` — so identical
+ * blobs dedupe, a concurrent reader never sees a half-written asset, and the
+ * hash IS the integrity check the restore verifies against.
+ */
+export declare const captureAssetsToCas: (opts: {
+    readonly source: AssetSource;
+    readonly assetsDir: string;
+    readonly progress?: AssetProgress;
+}) => Effect.Effect<CapturedAssets, BundleError>;
+
+export declare interface CapturedAssets {
+    readonly count: number;
+    readonly totalBytes: number;
+    /**
+     * Keys the refs table names and the provider does not have (404).
+     *
+     * A DANGLING REFERENCE IS NOT A CAPTURE FAILURE. It is a fact about the source
+     * data — a row pointing at an object that was deleted, or that never arrived
+     * because an earlier import ran without `--assets`. Aborting the phase on one
+     * made `--assets` unusable for exactly the deployment that needed it most: 178
+     * references, one resolvable, and the run stopped at the second. What it left
+     * behind was an `assets/` directory with a single blob, no provenance stamp,
+     * and no statement anywhere that 177 objects had been skipped — so the artefact
+     * read as a complete asset backup.
+     *
+     * Reported and carried, never swallowed: the count goes in the stamp so the
+     * restore can say what it is about to under-restore.
+     */
+    readonly missing: ReadonlyArray<string>;
+}
+
 /**
  * Lint a bundle's column types against a target dialect. Same-dialect import is
  * always clean (returns []). Otherwise returns every column whose type won't
@@ -1075,6 +1127,9 @@ export declare type NativeStep = {
     readonly env: Readonly<Record<string, string>>;
     readonly stdoutFile?: string;
     readonly stdinFile?: string;
+    /** Something known at BUILD time that the failure could not otherwise
+     *  say — e.g. that this tool is a fallback for one that was absent. */
+    readonly hint?: string;
 } | {
     readonly kind: 'copy';
     readonly from: string;
@@ -1089,6 +1144,19 @@ export declare class NativeToolError extends NativeToolError_base<{
     readonly tool: string;
     readonly reason: string;
     readonly stderr?: string;
+    /**
+     * What the CALLER knew and the child could not say.
+     *
+     * The stderr above is the tool's own words, which is the right first answer
+     * and is sometimes a dead end on its own: `Unknown table 'COLUMN_STATISTICS'
+     * in information_schema (1109)` names a table nobody asked for, in a schema
+     * nobody wrote. The fact that turns it into an action — that we wanted
+     * `mariadb-dump`, could not find it, and took Oracle's `mysqldump` instead —
+     * is known where the step is BUILT and nowhere else. So it travels with the
+     * step (`NativeStep.hint`) and is printed under the child's words, never
+     * instead of them.
+     */
+    readonly hint?: string;
 }> {
     /**
      * The child's own words, not ours.
@@ -1117,6 +1185,11 @@ declare const NativeToolError_base: new <A extends Record<string, any> = {}>(arg
  *  rather than a bare SyntaxError. */
 export declare const ndjsonToRows: <E, R>(bytes: Stream.Stream<Uint8Array, E, R>, table: string, columnTypes: ColumnTypes) => Stream.Stream<Row, E | CodecError, R>;
 
+/** A progress journal that remembers nothing — every asset is transferred. Used
+ *  where there is no sidecar to keep, and named so a caller passing it is
+ *  saying "no resume" rather than passing `undefined` and hoping. */
+export declare const noAssetProgress: AssetProgress;
+
 export declare type OnConflict = 'skip' | 'fail';
 
 /** The caller's progress sink. Plain sync fn — never fails the job. */
@@ -1232,6 +1305,9 @@ export declare interface ProgressEvent {
 /** Which pipeline the event belongs to. */
 export declare type ProgressPhase = 'export' | 'import';
 
+/** Read a CAS index file (NDJSON, one {@link AssetEntry} per line) as a stream. */
+export declare const readAssetIndex: (indexPath: string) => Stream.Stream<AssetEntry, BundleError>;
+
 /** Read the ledger at `path`, or an empty one when it is absent / unreadable
  *  (a missing ledger just means "nothing done yet" — never a hard failure). */
 export declare const readLedger: (path: string) => Effect.Effect<Ledger>;
@@ -1270,6 +1346,20 @@ export declare const restoreAssetFromCas: (sink: AssetSink, assetsDir: string, e
     readonly contentType: string;
 }) => Effect.Effect<void, BundleError | IntegrityError | StorageError>;
 
+/**
+ * Push every asset named in the index back to the sink, re-hashing on the way.
+ *
+ * A hash mismatch is an {@link IntegrityError} and stops the phase — a corrupted
+ * artefact must never silently re-upload bad bytes over good ones, which is the
+ * one failure a restore has no way to notice afterwards.
+ */
+export declare const restoreAssetsFromCas: (opts: {
+    readonly sink: AssetSink;
+    readonly assetsDir: string;
+    readonly indexPath: string;
+    readonly progress?: AssetProgress;
+}) => Effect.Effect<number, BundleError | IntegrityError | StorageError>;
+
 /** Build the restore step for a dialect reading `inFile`. */
 export declare const restoreCommand: (dialect: string, conn: NativeConn, inFile: string) => NativeStep;