@voltro/data-transfer 0.48.0 → 0.50.0

package/CHANGELOG.md

@@ -39,6 +39,131 @@ _Changes staged for the next release accumulate here (rolled up from
 
 ---
 
+## [0.50.0] — 2026-08-23
+
+### ⚠ BREAKING
+
+- **@voltro/cli** — The read-only run-history endpoint moved:
+
+  GET /_voltro/admin/imports → GET /_voltro/admin/transfers
+
+  Same secret, same query parameters, same response shape. Only the path changed.
+
+  It answers for four directions now — import, export, native backup, native restore — and it named one of them. The subcommand and the table were renamed for exactly that reason in 0.49.0 and this path was left behind, which is worse than renaming none of them: a reader who follows a rename tries the matching path, gets a 404, and concludes the instance is too old.
+
+  `voltro data transfers --target api` uses the new path for you. What needs a hand is anything calling it directly — a monitoring check, an uptime probe, a curl in a runbook, a dashboard datasource. The codemod is `manual` and declares `reach: 'beyond-source'`, so it prints whether or not the path is found in your repository.
+- **@voltro/database, @voltro/voltro** — `InterruptedReplace.tables` is `number | null`, and the interface gains `kind?: 'replace' | 'restore'`.
+
+  `_voltro_replace_in_progress` records two destructive operations now — an import's `--mode replace` and a native `voltro data restore`. A restore replaces the whole database from an artefact, so "how many tables was this going to empty and refill" has no answer: not known up front, not meaningful after.
+
+  Writing `0` would have preserved the type and been worse. That number is rendered into the boot refusal, which is read under pressure, and it would have said `began emptying 0 table(s)` — a measurement that was never taken. The framework's own text says "this database" instead; `Number(null)` is `0`, so coercing it reintroduces exactly the sentence being avoided.
+
+  `DataTransferRun.direction` also widens to name `'backup'` and `'restore'` explicitly. That one is documentary — the union already ended in `| string`, so no assignability changes.
+
+  **`voltro update` carries you across this** — codemod `0.50.0/03_interrupted-replace-tables-nullable`.
+
+### Added
+
+- **@voltro/cli, @voltro/data-transfer, @voltro/database** — `voltro data backup --assets` and `voltro data restore --assets` now move the stored blobs alongside the vendor dump, through the same content-addressed phase the logical `export` / `import` path uses — streamed, deduped by sha256, verified on the way back, resumable per key.
+
+  `--assets` was accepted by `backup` and silently ignored, with the only signal a field in the closing JSON reading `NOT included`; `restore` did not accept it at all. A rows-only backup restores a database whose rows reference objects nothing puts back, and the reference and the object are checked at different times, so that state is discovered by a user rather than by the restore.
+
+  Three refusals, each for a belief that is otherwise acted on silently: `backup --assets` with no storage provider is refused rather than swallowed; `restore --assets` on a rows-only artefact is refused; a restore WITHOUT `--assets` over an artefact that has them warns and proceeds, because restoring rows without blobs is legitimate and refusing it would push people at `--force`.
+
+  The dump itself still has no resume — a vendor artefact is one opaque file with no offset to restart from, and the logical path is what exists for that.
+- **@voltro/cli, @voltro/database, @voltro/data-transfer** — A native `backup` / `restore` now writes to the same `_voltro_data_transfers` record `import` and `export` use, so `voltro data transfers` answers "did last night's backup finish" from the instance that ran it. A native run reports blobs rather than rows — a vendor tool reports no row count we can trust, and printing `0 row(s)` over a dump that worked would be a wrong measurement.
+
+  `restore` also writes the `_voltro_replace_in_progress` marker before its first destructive statement and clears it after the last write, blobs included, so a killed restore refuses the next boot instead of serving a half-loaded database. `--allow-live` guards from the wrong side — it asks you not to — and this guards from the right one. The marker carries a `kind` so the refusal can say something different for a `replace` (re-import the capture) and a `restore` (finish the restore), and `tables` is nullable because a native restore has no table count to claim.
+
+  The sqlite / turso restore is atomic now (temp file + rename): its marker lives in the very file being replaced, so a plain in-place copy would leave a truncated database with nothing left to catch it.
+
+  A target with no `_voltro_data_transfers` table still gets its backup; the closing line says it was not recorded rather than implying it was.
+- **@voltro/cli** — A re-issued migration note for the `voltro data imports` → `voltro data transfers` rename that shipped in 0.49.0.
+
+  Codemods are selected by `from < version <= to`, so the 0.49.0 note fires once, on the jump that crosses 0.49.0, and cannot be corrected for anyone already past it. Its gate searched `.ts` / `.tsx` while the command it is about lives in shell scripts, CI job definitions and runbooks — so a project whose only occurrence sat in `.gitlab-ci.yml` crossed 0.49.0 and was told there was nothing to apply.
+
+  The re-issue is filed under 0.50.0, prints unconditionally, and says what to grep for. Redundant for anyone already fixed; the alternative is firing for nobody.
+
+### Fixed
+
+- **@voltro/cli** — A `manual` codemod's `appliesTo` can now search every text file the project owns — `.sh`, `.yml`, `.json`, `.md`, `Makefile`, the `.js` scripts — through a new `ctx.text` on the predicate context, and a codemod may declare `reach: 'beyond-source'` to print its note even when nothing matched.
+
+  Both halves close the same gap, and it was in the codemod most in need of a gate. A manual codemod exists BECAUSE its subject could not be transformed, which usually means it is not source at all — and `appliesTo` was reading the ts-morph project, which holds `.ts` / `.tsx` and nothing else. Measured with one identical CLI invocation in four files: the `.ts` one printed the note, the `.sh`, `.yml` and `.md` ones printed nothing, and the run reported `codemods: nothing to apply for this jump`. That is an acquittal from a check that never looked at the file, and an acquittal gets acted on.
+
+  `reach: 'beyond-source'` covers what no scan of one repository can reach — an inline script in a CI runner's own UI, a CronJob spec in another repo, a wiki runbook. The note prints either way; only its framing changes, and an uncertain one says plainly that we could not look there.
+- **@voltro/data-transfer** — When `mariadb-dump` is absent and the mysql-family fallback runs Oracle's `mysqldump` against a MariaDB server, the failure now names the way out.
+
+  The stderr it prints is the child's own words and is the right first thing to show — and on its own it is a dead end: `Unknown table 'COLUMN_STATISTICS' in information_schema (1109)` names a table nobody asked for, in a schema nobody wrote, about a feature nobody enabled. Everything needed to act on it was known where the fallback was DECIDED: which binary we wanted, which one we took, and why the difference matters. That travels with the step now and prints under the stderr, including the counter-move a reader reaches for on their own (`--column-statistics=0` does not exist on `mariadb-dump`, so it fixes the wrong client and breaks the right one).
+
+---
+
+## [0.49.0] — 2026-08-23
+
+### ⚠ BREAKING
+
+- **@voltro/cli, @voltro/data-transfer, @voltro/database, @voltro/sql-sqlite, @voltro/voltro** — A data transfer is a series of short requests now — no single one may outlive a caller's budget.
+
+  **The rule:** *a request that carries bytes never runs a transfer; a request that starts a transfer never carries bytes.* It was broken in the worst available place. The upload was already chunked and resumable — many short requests, each abandonable — and then the FINAL chunk fell through and ran the whole import. So the longest request of the flow arrived AFTER the entire upload had succeeded, and a caller under a policy that caps a single request (a job runner that kills a client at ten minutes; a 30 s ingress ceiling) lost the most expensive thing they had already paid for. A single-request upload had the same shape without the excuse, and the export had no protocol at all: one request that read the whole database and streamed it back.
+
+  **Import.** `POST /_voltro/admin/import` accumulates and answers `202`. `POST /_voltro/admin/import/start` begins the run and answers `202 { runId }` as soon as the run's first row exists. `start` is idempotent per upload — it is a short request and therefore a retryable one, and without a claim a retry would begin a second destructive run from the same bytes. A claim whose run has ENDED is taken over rather than honoured forever, so a process that dies holding one cannot poison a bundle.
+
+  **Export.** `POST /_voltro/admin/export` answers `202 { runId }` and produces in the background; the bytes come back from `GET /_voltro/admin/export/download?runId=&offset=&length=` in ranges, resumable, with the total in a header. Object storage (`--bundle-key`) remains for a bundle you want to KEEP — it is no longer the only way to get one out, because requiring it would leave an instance without storage unable to export at all.
+
+  **The client.** `--max-request-seconds` (or `VOLTRO_MAX_REQUEST_SECONDS`) declares the budget — declared rather than probed, because the thing that kills a request is a policy on the caller's side and only they know it. `--detach` returns once the run has started and says the outcome is NOT known. Attached, the CLI polls the record and prints per-table progress; Ctrl-C then loses the watching and never the run.
+
+  **The trap, stated because it is the one way to get this wrong:** a failure used to arrive in the response (409 on drift, 409 on a refused mode, 500 otherwise). After the split the response is a `202`, so **a client deriving its exit code from the status line reports a failed import as a success.** The exit code comes from the polled record, in one shared function, and the drift check moved into `start` where it can still be a refusal that leaves no history.
+
+  **Two knobs that were constants.** `VOLTRO_IMPORT_UPLOAD_DIR` and `VOLTRO_EXPORT_ARTIFACT_DIR` move the staging areas off the default temp filesystem — which on a container is frequently a small tmpfs, so an instance simply could not accept a bundle the size a grown database produces, and the failure arrived as a write error halfway through an upload somebody had been waiting on.
+
+  **Renamed:** `voltro data imports` → `voltro data transfers`, and `_voltro_data_imports` → `_voltro_data_transfers` with a `direction` column. The command showed one direction and now shows both; the table rename carries its rows via the declarative differ on every dialect and needs nothing from you. The CLI rename ships a `manual` codemod — the command lives in scripts, CI jobs and runbooks, which `voltro update` cannot see or rewrite. The four PUBLIC exports that named the same record were renamed with it (`ImportRun`, `describeImportRun`, `IMPORT_RUNS_TABLE`, `_voltroImportRunsTable`); those are application source, they ship their own `transform` codemod, and they have their own entry.
+- **@voltro/database, @voltro/voltro** — The run-history exports say `transfer`, not `import`.
+
+  `@voltro/database` (and `voltro/database`) renamed four public exports along with the table behind them:
+
+  | was | is | |---|---| | `ImportRun` | `DataTransferRun` | | `describeImportRun` | `describeTransferRun` | | `IMPORT_RUNS_TABLE` | `DATA_TRANSFERS_TABLE` | | `_voltroImportRunsTable` | `_voltroDataTransfersTable` |
+
+  An EXPORT writes to this record now — the row carries a `direction` — so every one of those names described half of what it holds.
+
+  A `transform` codemod rewrites all four, alias-aware, from either module spelling. It also rewrites a hand-spelled `_voltro_data_imports` in a string, template or raw-SQL fragment, and that is the half worth stating: the four identifiers announce themselves as compile errors, while a query that addresses the table by name has nothing to fail on. The table itself moves with its rows via the declarative differ on every dialect.
+
+  This is filed apart from the transfer-protocol entry beside it deliberately. That one's codemod is `manual` and is about a CLI invocation living in scripts and CI jobs; this one is application source and is rewritten for you. Reading the first as covering both is what would leave a build broken with a note saying nothing in your source was affected.
+
+### Added
+
+- **@voltro/database, @voltro/data-transfer, @voltro/cli** — An import records WHETHER it staged, and a soft-dropped column says so in a drift refusal.
+
+  **`_voltro_data_imports.staged`.** The pre-upload preflight says what the instance WILL do; the run's own line says what it did — and that line is printed inside the instance, which is exactly where an operator using `--target api` cannot read it. So "we were told it would stage" and "it staged" were two claims with no way to close the gap between them from outside. The column is `null` for a mode where the question does not arise; `voltro data imports` and `GET /_voltro/admin/imports` both carry it.
+
+  **A soft-dropped column is named as one.** `<original>__dropped_<stamp>` is what the differ leaves behind when an app stops declaring a column — only the database that did the drop has it, so it drifts against every target. The refusal reported it as an ordinary missing column ("the value has nowhere to go"), which points at the TARGET's schema: the one place the fix does not lie. It now says where the column lives and how to reclaim it.
+
+### Fixed
+
+- **@voltro/data-transfer, @voltro/sql-sqlite, @voltro/database** — A successful `--mode replace --no-atomic` left a marker that refused the next boot.
+
+  Two defects, one visible symptom, and both were silent by construction.
+
+  **The clear was gated on the WRITE's condition.** `!(useLedger && ledger.truncated)` means "an earlier attempt already recorded this destructive run, do not write a second row" — correct on the write. Copied down to the clear its meaning inverts: `ledger.truncated` is set by the emptying step of THIS run, so on `--no-atomic` (the only mode where `useLedger` is true) the clear was skipped by the very run that wrote the marker.
+
+  **And a `Date` in a predicate is not bindable on sqlite.** `better-sqlite3` binds numbers, strings, bigints, buffers and null; the row path has coerced Dates since that store was written, and the eager-join compiler carried its own private copy of the fix, but `query` / `updateMany` / `deleteMany` bound the raw value. So every comparison of a timestamp column against a `Date` failed with `Failed to execute statement` — including the marker's clear, which swallows its errors ON PURPOSE (a store with no marker table must not fail an import over a bookkeeping row) and therefore said nothing. The retention sweep compares `lt(column, cutoff)` the same way.
+
+  Together: a fully successful replace left the marker standing, and the next boot REFUSED with "a destructive import did not finish" over a database that was completely fine. The one recovery is a command the operator has no reason to think they need. It stayed invisible because `atomic` defaults to true for `replace`, and because a staged replace writes no marker at all — two defaults hiding the one mode that exists for large, interruptible loads.
+
+  The coercion is shared now and applied at all three predicate sites, with a guard that fails on a fourth that forgets. A dry run also closes its trace row: a preview that finished instantly used to leave the record open, so a polling caller waited out its whole budget over a run that was long done.
+- **@voltro/cli** — `voltro codegen` declared twenty framework tables fewer than a boot.
+
+  The app half of this was fixed last release and looked like the whole thing. It was not: `codegen` derived the FEATURE MIX from the file list it had just walked, and that list is the entity/relations set — which contains no `*.workflow.tsx`, no `*.agent.ts`, no `*.cron.tsx`. So every feature flag came back false, and the dialect was never passed at all, taking `_voltro_cdc_offsets` with it. Measured on an app with two workflow files: 18 framework tables where the shared assembly produces 33.
+
+  It calls `assembleFrameworkTables({ root })` now — the same entry `voltro db plan/apply` uses, which detects the mix from the root rather than being handed one.
+
+  The parity guard moved with it. Comparing the two WALKS could not see this: both walks were right about files, and the divergence was introduced one layer past them. It compares the assembled SETS now, and runs codegen's own table function on a tree whose only feature signals are a workflow file and an agent file — a source assertion that the right function is CALLED cannot see a wrong argument handed to it, and a wrong argument is what this was.
+- **@voltro/cli** — `voltro data --help` advertised four subcommands out of nine.
+
+  It printed `<export|import|backup|restore>` while the command dispatched those four plus `imports`, `inspect`, `unpack`, `clear-replace-marker` and `clear-staging`. A quoted enumeration is read as exhaustive — the same failure `subcommandNames.ts` was written for after a `db` list cost two wrong conclusions — and the missing entry here is the one that answers "what is my import doing right now" for somebody who cannot reach the pod.
+
+  `subcommandHelpParity.test.ts` had listed `data` among the commands it could not check, loudly and correctly: there was no name list to check against. There is one now (`DATA_SUBCOMMANDS`), the dispatch is a keyed `Record`, so a subcommand with no name and a name with no handler are both compile errors, and the usage line is generated from the same array. `data` is out of the unchecked list.
+
+---
+
 ## [0.48.0] — 2026-08-22
 
 ### Added

package/dist/index.d.ts

@@ -76,6 +76,23 @@ export declare interface AssetInfo {
     readonly size: number;
 }
 
+/**
+ * The resume journal, as this phase needs it.
+ *
+ * Deliberately two functions rather than the `Ledger` type: the native path
+ * keeps its own sidecar and the logical path keeps the shared one, and neither
+ * has any business knowing the other's file layout. What both must agree on is
+ * the SEMANTICS — `done` is keyed by the asset KEY (not its hash), because an
+ * asset is skippable when the destination already has that key, and the hash is
+ * not known until the bytes have been read, which is the cost being avoided.
+ */
+export declare interface AssetProgress {
+    readonly done: (key: string) => boolean;
+    /** Record `key` as landed. Called AFTER the bytes are durable, never before —
+     *  a ledger written first turns a crash into silent data loss on resume. */
+    readonly mark: (key: string, entry: AssetEntry) => Effect.Effect<void, BundleError>;
+}
+
 export declare const ASSETS_DIR = "assets";
 
 /** Where import writes assets TO. */
@@ -149,6 +166,25 @@ export declare type CanonicalCategory = 'string' | 'integer' | 'float' | 'boolea
  *  types (a bundle from a newer framework) fall back to 'string' passthrough. */
 export declare const canonicalType: (type: string) => CanonicalCategory;
 
+/**
+ * Stream every asset the source enumerates into the content-addressed `assetsDir`.
+ *
+ * `assetsDir` must exist. Each asset is hashed while it streams (never buffered
+ * whole), written to a temp file and renamed to `assets/<sha256>` — so identical
+ * blobs dedupe, a concurrent reader never sees a half-written asset, and the
+ * hash IS the integrity check the restore verifies against.
+ */
+export declare const captureAssetsToCas: (opts: {
+    readonly source: AssetSource;
+    readonly assetsDir: string;
+    readonly progress?: AssetProgress;
+}) => Effect.Effect<CapturedAssets, BundleError>;
+
+export declare interface CapturedAssets {
+    readonly count: number;
+    readonly totalBytes: number;
+}
+
 /**
  * Lint a bundle's column types against a target dialect. Same-dialect import is
  * always clean (returns []). Otherwise returns every column whose type won't
@@ -366,15 +402,6 @@ export declare const DEFAULT_CLASS_ACTIONS: Readonly<Record<string, MaskAction>>
 /** Identity helper for authoring a profile with full type-checking + inference. */
 export declare const defineDataProfile: (profile: DataTransferProfile) => DataTransferProfile;
 
-/**
- * Differences that would actually break THIS import, in a stable order.
- *
- * `bundle` is the shape reconstructed from the manifest, so only its table
- * names, column names and column TYPES are meaningful — its `nullable` /
- * `hasDefault` are placeholders and are never read here. Every judgement about
- * required-ness is therefore made from the TARGET side, which is introspected
- * and real.
- */
 export declare const diffSnapshots: (bundle: SchemaSnapshot, target: SchemaSnapshot) => ReadonlyArray<string>;
 
 /** A {@link BlobSink} that writes blobs to the LOCAL dir as `assets/<sha>` files
@@ -710,6 +737,19 @@ export declare interface ImportOptions {
     /** Which transport is running this — named in the refusal a later boot prints,
      *  so the reader knows where to look for logs. */
     readonly via?: string;
+    /**
+     * The id of a trace row the CALLER has already written.
+     *
+     * The admin-import `start` endpoint answers `202` with a run id before the
+     * import has done anything, and a caller who polls immediately must find a
+     * row rather than an absence — "not started" and "no record" are the two
+     * states a poll loop cannot be allowed to confuse. So `start` opens the row
+     * itself and hands the id here; this run then ADVANCES and CLOSES that row
+     * instead of opening a second one.
+     *
+     * Absent, the run opens its own row exactly as before.
+     */
+    readonly traceId?: string;
     /**
      * Record that this run emptied the target, so a later boot can refuse over a
      * database it left half-replaced.
@@ -1071,6 +1111,9 @@ export declare type NativeStep = {
     readonly env: Readonly<Record<string, string>>;
     readonly stdoutFile?: string;
     readonly stdinFile?: string;
+    /** Something known at BUILD time that the failure could not otherwise
+     *  say — e.g. that this tool is a fallback for one that was absent. */
+    readonly hint?: string;
 } | {
     readonly kind: 'copy';
     readonly from: string;
@@ -1085,6 +1128,19 @@ export declare class NativeToolError extends NativeToolError_base<{
     readonly tool: string;
     readonly reason: string;
     readonly stderr?: string;
+    /**
+     * What the CALLER knew and the child could not say.
+     *
+     * The stderr above is the tool's own words, which is the right first answer
+     * and is sometimes a dead end on its own: `Unknown table 'COLUMN_STATISTICS'
+     * in information_schema (1109)` names a table nobody asked for, in a schema
+     * nobody wrote. The fact that turns it into an action — that we wanted
+     * `mariadb-dump`, could not find it, and took Oracle's `mysqldump` instead —
+     * is known where the step is BUILT and nowhere else. So it travels with the
+     * step (`NativeStep.hint`) and is printed under the child's words, never
+     * instead of them.
+     */
+    readonly hint?: string;
 }> {
     /**
      * The child's own words, not ours.
@@ -1113,6 +1169,11 @@ declare const NativeToolError_base: new <A extends Record<string, any> = {}>(arg
  *  rather than a bare SyntaxError. */
 export declare const ndjsonToRows: <E, R>(bytes: Stream.Stream<Uint8Array, E, R>, table: string, columnTypes: ColumnTypes) => Stream.Stream<Row, E | CodecError, R>;
 
+/** A progress journal that remembers nothing — every asset is transferred. Used
+ *  where there is no sidecar to keep, and named so a caller passing it is
+ *  saying "no resume" rather than passing `undefined` and hoping. */
+export declare const noAssetProgress: AssetProgress;
+
 export declare type OnConflict = 'skip' | 'fail';
 
 /** The caller's progress sink. Plain sync fn — never fails the job. */
@@ -1228,6 +1289,9 @@ export declare interface ProgressEvent {
 /** Which pipeline the event belongs to. */
 export declare type ProgressPhase = 'export' | 'import';
 
+/** Read a CAS index file (NDJSON, one {@link AssetEntry} per line) as a stream. */
+export declare const readAssetIndex: (indexPath: string) => Stream.Stream<AssetEntry, BundleError>;
+
 /** Read the ledger at `path`, or an empty one when it is absent / unreadable
  *  (a missing ledger just means "nothing done yet" — never a hard failure). */
 export declare const readLedger: (path: string) => Effect.Effect<Ledger>;
@@ -1266,6 +1330,20 @@ export declare const restoreAssetFromCas: (sink: AssetSink, assetsDir: string, e
     readonly contentType: string;
 }) => Effect.Effect<void, BundleError | IntegrityError | StorageError>;
 
+/**
+ * Push every asset named in the index back to the sink, re-hashing on the way.
+ *
+ * A hash mismatch is an {@link IntegrityError} and stops the phase — a corrupted
+ * artefact must never silently re-upload bad bytes over good ones, which is the
+ * one failure a restore has no way to notice afterwards.
+ */
+export declare const restoreAssetsFromCas: (opts: {
+    readonly sink: AssetSink;
+    readonly assetsDir: string;
+    readonly indexPath: string;
+    readonly progress?: AssetProgress;
+}) => Effect.Effect<number, BundleError | IntegrityError | StorageError>;
+
 /** Build the restore step for a dialect reading `inFile`. */
 export declare const restoreCommand: (dialect: string, conn: NativeConn, inFile: string) => NativeStep;