@voltro/data-transfer 0.46.0 → 0.48.0

package/CHANGELOG.md

@@ -39,6 +39,116 @@ _Changes staged for the next release accumulate here (rolled up from
 
 ---
 
+## [0.48.0] — 2026-08-22
+
+### Added
+
+- **@voltro/cli, @voltro/data-transfer** — An import over `--target api` can be asked what it will do, and watched while it does it.
+
+  The transport's two correct properties combined into one blind spot: the run happens INSIDE the instance, so every line it produces goes to a pod log; and it is deliberately decoupled from the caller, so killing the client does not stop it (which is what keeps a dead client from leaving a half-emptied target). Someone reaching for `--target api` cannot reach the database directly and usually cannot read that log either, so "watch for the staging line" was advice they could not follow.
+
+  **A preflight, before a byte is uploaded.** A `replace` over the api now asks the instance whether it will stage, and prints the answer — including the reason when it will not. It answers from `decideStaging`, the same function the run itself calls, so the two cannot drift; a second copy of that decision would answer confidently and diverge on the next change. With `--bundle-key`, where the client never holds the bundle, it sends the key and the instance reads the table list out of the archive's own manifest — the caller about to have an instance empty its own database is the last one who should be told to check a log they cannot read.
+
+  **`voltro data imports`** (and `GET /_voltro/admin/imports`, behind the same data-transfer secret) reads the history and the run in flight. `_voltro_data_imports` was write-only; it is now opened before the first table, **advanced every couple of seconds as tables land**, and closed with the outcome — so polling it is the progress feed. Not a streamed response on the upload connection: a chunked body has to survive every proxy in between, and a buffering reverse proxy turns a progress feed into exactly the silence it was meant to replace.
+
+  **And a `replace` that does not stage now says why.** The staging set was an early `return []` three conditions deep, and the empty array met a `length > 0` further down and read as "do not stage". A bundle table the target does not have — the ordinary case for a development source against a production target — turned the non-destructive path off in silence. Every reason routes through one message now; the cycle case additionally used to be gated on `atomic`, so the mode where an interrupted run leaves the worst outcome was also the one that said the least.
+
+### Changed
+
+- **@voltro/cli** — The declared framework schema no longer depends on `NODE_ENV`.
+
+  `_voltro_traces` and `_voltro_undo_log` defaulted to on outside production. That was allowed with an explicit justification — one decider makes every command in a deployment agree — and the justification was about the schema FINGERPRINT, which only ever compares processes inside ONE deployment.
+
+  The declared set has a second reader that spans two, and it was never considered: `voltro data`. A bundle exported from a development database carries the tables that database has, and a staged (non-destructive) `replace` needs every bundle table to exist in the target. So one source tree produced a bundle a production target could not stage, and the run fell back to truncating it — with nothing red anywhere, on the one path where that difference is the entire point.
+
+  Both tables are declared in **every environment** now. The trade is the one `_voltro_cdc_offsets` already makes: an unused declared table costs one empty table and buys agreement. `VOLTRO_UNDO` / `VOLTRO_TRACING_PERSIST` still decide what a process WRITES; they never decided the schema and still do not. Set `schema: { traces: false, undo: false }` in `app.config.ts` to keep one out — in every environment, or the divergence is back by hand.
+
+  **Upgrade:** a production app that never declared them gains two empty tables. `voltro db apply` (or a `voltro dev` boot) plans and applies them like any other framework table, on every dialect. Run it before the pods roll, as with any schema change — a migrate job and a fleet that disagree is exactly what this removes.
+
+  `voltro doctor` now prints the three decided tables and what decided each, on a HEALTHY run. That text existed and was reachable only from the `prod-mismatch` refusal — a message that appears exclusively after a fleet is down is an explanation, not a warning.
+
+### Fixed
+
+- **@voltro/cli, @voltro/data-transfer** — `voltro data export --exclude a,b` made the bundle BIGGER, and made it unusable for a `replace`.
+
+  It expanded into `{ kind: 'tables', tables: <everything else> }`, on the reasoning that a manifest should record what was exported rather than claim "everything". Right goal, wrong mechanism, and it cost two things at once. `all` is the only scope that filters out the tables which describe a deployment, so excluding two names silently added nine others back — the migration ledger among them, whose foreign row takes an environment down at the next boot. And `replace` refuses a named scope, so the honest way to leave a table out was also the way to make the bundle unusable for the mode it was being prepared for.
+
+  The exclusion is a FIELD of the `all` scope now: the filter still runs, the manifest still says "everything except these" (a different and truer claim than "these"), and `replace` accepts it while naming the tables it will therefore not touch. `--exclude` also works over `--target api` now — it no longer expands against a table list only the instance has, so the instance resolves it where that list already is.
+
+  Every framework table is classified as portable or environment-local, with the reason, and a new one fails the build until somebody decides. That guard existed and did not help: it was satisfied by a second, hand-kept list inside its own test, and the two disagreed about `_voltro_traces` and `_voltro_undo_log` — nothing was unclassified, something was classified twice, once wrongly. There is one list now, read by both guards. Traces, undo, the outbox and its attempts, idempotency keys, storage grants, spend and usage accounting, delivery attempts and schedule-firing history joined the environment-local side: a row from elsewhere would make the target act, or claim history it did not live.
+- **@voltro/sql-turso** — Opening a second pooled turso connection could fail instantly with `database is locked` — from inside the constructor, before a single query ran.
+
+  `makeConnection` set its pragmas in the order `journal_mode` → `foreign_keys` → `busy_timeout`. The engine defaults `busy_timeout` to **0**, so a statement that meets a held lock fails on the spot instead of waiting — and `journal_mode= experimental_mvcc` needs the file exclusively. Since `makeConnection` runs once per POOLED connection (default 4), opening connection two while connection one held the file hit that exclusive pragma with no lock-wait configured yet:
+
+  SqlError: Failed to enable Turso MVCC (journal_mode=experimental_mvcc): database is locked
+
+  `busy_timeout` is set FIRST now. Nothing else changed — same value, same pragmas, same connection.
+
+  **Why it hid for so long.** The file already carried a long, correct note about `busy_timeout` being mandatory with a pool, and a separate fix had closed the DDL half (`retryFilter` + bounded retries in `applySchema`). Both are about the same lock class, so the constructor read as covered — but a setting cannot protect the two pragmas that run before it.
+
+  It surfaces as an unrelated flaky test, because the failure lands wherever the second connection happens to be opened: a `CREATE TABLE` in one run, an MVCC pragma in the next. It cost three release gates — twice locally, once on a CI runner — and was twice diagnosed as machine contention and closed. It is contention-DEPENDENT, which is not the same as being the machine's fault.
+
+  Verified: 8 serial runs and 6 concurrent suites at load 12 — 0 failures, 0 occurrences of the message. The failure was intermittent before, so this is evidence rather than proof; the mechanism, however, is not in doubt.
+- **@voltro/runtime, @voltro/data-transfer, @voltro/cli** — The `source:` recorder broke every WRITE on the in-memory store, and actions are recorded now.
+
+  The recording wrapper is a `Proxy`, and it handed methods back unbound — so `this` was the PROXY, and a class with `#private` fields answers that with `TypeError: Receiver must be an instance of class InMemoryDataStore`. Under `voltro dev` on the memory store, where the recorder is installed by default, that is every write in the app.
+
+  Every test passed throughout, and the reason is worth more than the fix: `query` is the one method the wrapper invokes with an explicit receiver, so everything that only READ through it worked. The wrapper's whole purpose is reading, so nothing in its own suite ever wrote. It was found by a test about something else entirely — asking what `crud.create` reads — which needed a write to answer.
+
+  **Actions are recorded too now**, and an action's `source:` means something different from a query's. A query's is a reactive trigger set; an action's declares what it TOUCHES, and the field's own documentation records what an undeclared read costs: `voltro check` reported a table five action paths read and wrote as an orphan, and advised removing it. That is not a quiet subscription, it is advice to delete a live table.
+
+  **Measured rather than reasoned about:** `crud.create`, `crud.update` and `crud.remove` issue NO read at all, so a read recorder has nothing to say about them — but `crud.getById` does read, and with `include:` it eager-loads, so it is covered like `crud.list`.
+
+  **And a kill test for the one moment nobody had reproduced.** The existing test kills mid-LOAD, which staging turned into the harmless part; the destructive second went untested precisely because it became short. The new case kills as the swap begins and asserts the property rather than the race: the target is one state or the other, never a mix, and never empty. Two defects in the harness came out of writing it — a worker that died SILENTLY (its failure now goes into the marker the parent already reads, instead of looking like a slow start), and an `exit` listener attached only after the SIGKILL, which hung to the full timeout whenever the child finished first.
+
+  **And the import trace was never written on the path most likely to be used.** A deployment ran a successful `voltro data import` against a database that HAD the table, and got no row and no message at all. The write was gated on this PROCESS's table registry, and `voltro data`'s own boot builds a store and introspects the live schema — it never registers the framework set, so the gate was `undefined` exactly there. The authority for "does the target have this table" is the target's SCHEMA, which that boot already introspects; the table is also registered before the write, because a CLI run has not done it and the write needs the column metadata.
+
+  The sharper half is the silence, and it was self-inflicted: the skip was written three lines under a comment about how an absent table cannot be detected by the write failing. The message now lives in `voltro data import`, which is the layer that INTROSPECTED — a first attempt put it in the importer, where a `targetSnapshot` may legitimately be narrow rather than complete, so it fired at callers whose snapshot simply did not mention a framework table.
+
+### Internal (no consumer-facing effect)
+
+- **@voltro/data-transfer** — `SAVEPOINT_BATCH_SIZE` carries its documentation again.
+
+  A new `TRACE_ADVANCE_MS` was declared BETWEEN the constant's doc block and the constant, so TypeScript attached the block to whatever now followed it and the exported symbol was left bare — the api golden recorded it as `// @public (undocumented)` and the published report shipped it that way.
+
+  Third time this exact shape has appeared (`sourceKeys`, `recordsTable`, now this one), always the same mechanism: an insertion above a documented declaration silently re-homes the comment. Nothing warns, because both the code and the doc block are individually valid — only the golden's `(undocumented)` marker notices, and it reads as noise unless someone diffs it against the last TAG.
+
+  Documentation only; no behaviour, no signature change.
+
+  **`apiSurface: compatible`, and the reason is the whole point of the change.** The golden line that moved is `// @public (undocumented)` → `// @public`: an api-extractor MARKER describing whether a doc comment is present. No type, no signature, no name. `SAVEPOINT_BATCH_SIZE` is still `= 200`, still exported, still the same literal type — nothing that compiled can stop compiling.
+
+---
+
+## [0.47.0] — 2026-08-22
+
+### Changed
+
+- **@voltro/database, @voltro/data-transfer, @voltro/cli, @voltro/plugin-storage** — A `--mode replace` no longer writes per-row history, and three things that were reported alongside it.
+
+  **Write recorders are suspended for a `replace`.** A replace SETS a state; it does not change rows, so a per-row history entry describes something that did not happen. A deployment measured what that costs: they run `versioningPlugin({ timing: 'in-transaction' })` on 70 of 80 tables, so one import wrote 242 950 history rows and doubled the write load of the most expensive run they make — and those rows were the source of refused writes they spent three rounds diagnosing.
+
+  The deciding argument is not the cost. **The import path had already decided this, and the recorder was the one layer that did not hear.** These routes write through the RAW store on purpose — no tenant scoping, no row filter, no `audit()` stamping. A recorder fired anyway because it hangs one level below the wrapper. Suspending it makes the layers agree.
+
+  Suspended for `replace` only: an `upsert` or an `append` CHANGES existing state, which is exactly what a recorder is for. Scoped per execution context rather than by a switch, because an import runs while the app serves requests and a process-global flag would silently drop recording for everything concurrent with it. And every such run SAYS what it suspended — dropping history quietly would be the same defect in a nicer costume.
+
+  This also removes the reason a staged `replace` used to fall back. A recorder is keyed by table name, so a staged write found none and the recorder never ran; rather than record inconsistently, the run took the slower path. For the deployment above that meant the staged path could never activate — permanently, on every environment.
+
+  **`voltro codegen` wrote a truncated table declaration.** It discovered tables with the walk that collects the rpc GROUP's inputs — descriptors, workflows, events, and deliberately no `*.entity.ts`. So the generated `voltro-tables.generated.d.ts` listed the framework's tables plus `actors`, which the framework injects. Measured downstream: 37 names where a `voltro dev` boot writes 117, and 299 `TS2322` errors from every `source:` naming one of their own tables. The way in was our own message — `voltro test` refuses a stale rpc group and tells you to run `voltro codegen`. Table discovery has its own walk now, and a test compares the two walks' RESULTS on a real tree rather than trusting they mean the same thing.
+
+  **The eager-relation doctor rule missed the state that is most wrong.** It resolved relations only against the tables a query already DECLARED, on the reasoning that the base is virtually always in `source:`. Measured: a query reading `projects` while declaring only `projectTeams` produced no finding at all, while the same query with the base added produced two. The base is a fact about the executor, so it is read from the executor now.
+
+  **A rollback capture says when its path shares the root filesystem.** The 409 answered "is storage configured", which is not the question — a `filesystem` provider pointed at a container directory with no volume behind it passes it and dies with the pod. What a process CAN observe is that a mounted volume is a different filesystem: `pathDurability` compares device ids, and a capture landing on the same device as `/` says so. Three-valued on purpose, and explicitly not an alarm on a development machine, where everything is one device and nothing is a pod.
+
+  **And the import records ITSELF, once.** Dropping per-row history left an operator asking "was this data imported, and when" with nothing to read, and trading too much for none is not obviously the better trade. `_voltro_data_imports` carries one row per RUN: the mode, the transport, the bundle, the SOURCE deployment's schema fingerprint, the counts, and — for the run an operator is actually looking for — the failure. It is written for a failed import as well as a finished one; a trail that only records successes goes quiet exactly when it is needed.
+
+  Best effort, unlike the interrupted-replace marker, and the difference is deliberate: the marker is a safety interlock and a run that cannot write it must not proceed, while history is valuable and not load-bearing. A target that has not been migrated yet still imports, and says the trace could not be written.
+
+  Two things it learned from being wired. The table is `.nonReactive()` — nobody subscribes to "an import happened", and a reactive bookkeeping write showed up in every suite that counts the LOAD's writes. And it is written only when the target actually DECLARES the table: the in-memory store accepts a write to any name, so an absent table cannot be detected by the write failing, and inventing the row would put a framework write into every embedder's counts.
+
+  **Measured at the reported magnitude.** 114 tables, 243 048 rows, real MariaDB, staged replace: the whole run takes 56.5s and the DESTRUCTIVE TRANSACTION takes **1.13s**. The window in which a dead process can leave a half-replaced target is the second number; before this it was the first.
+
+---
+
 ## [0.46.0] — 2026-08-22
 
 ### ⚠ BREAKING

package/dist/index.d.ts

@@ -277,6 +277,42 @@ export declare interface DataTransferProfile {
     readonly assets?: boolean;
 }
 
+/**
+ * The one place that decides whether a `replace` stages.
+ *
+ * It is a shared function rather than an inline block because it has TWO
+ * readers, and they must not be able to disagree:
+ *
+ *   - the importer, at the moment it runs, and
+ *   - the admin-import PREFLIGHT, which answers the same question before a byte
+ *     of the bundle is uploaded.
+ *
+ * The preflight exists because the run's own answer is printed on the INSTANCE.
+ * An operator using `--target api` is, by construction, someone who cannot reach
+ * the database directly — and usually cannot read the pod's log either — so
+ * "watch for the staging line" is advice they cannot follow. Asking first is the
+ * only form of that answer they can act on.
+ *
+ * A paraphrase of this decision in the preflight would be worse than no
+ * preflight: it would answer confidently and diverge on the next change. Both
+ * callers pass what they have and read the same verdict.
+ */
+export declare const decideStaging: (input: {
+    readonly mode: string;
+    /** The target's dialect, as the caller knows it. Anything unrecognised is a
+     *  blocker with that name in it. */
+    readonly targetDialect?: string | undefined;
+    readonly targetSnapshot?: StagingSnapshot | undefined;
+    /** The bundle's table names, in the manifest's order. */
+    readonly bundleTables: ReadonlyArray<string>;
+    /** Names that describe a deployment and are skipped on import anyway. */
+    readonly isSkipped: (table: string) => boolean;
+    /** Can the store send raw SQL? (`canStage(store)` at the importer; a constant
+     *  `true` at the preflight, where the instance's own store is the one that
+     *  will run it.) */
+    readonly canRunSql: boolean;
+}) => StagingDecision;
+
 export declare const decodeManifest: (u: unknown, overrideOptions?: ParseOptions) => {
     readonly createdAt: string;
     readonly source: {
@@ -287,6 +323,7 @@ export declare const decodeManifest: (u: unknown, overrideOptions?: ParseOptions
         readonly kind: "tables" | "all" | "tenant";
         readonly tenantId?: string | undefined;
         readonly tables?: readonly string[] | undefined;
+        readonly exclude?: readonly string[] | undefined;
     };
     readonly tables: readonly {
         readonly name: string;
@@ -399,6 +436,7 @@ export declare const encodeManifest: (a: {
         readonly kind: "tables" | "all" | "tenant";
         readonly tenantId?: string | undefined;
         readonly tables?: readonly string[] | undefined;
+        readonly exclude?: readonly string[] | undefined;
     };
     readonly tables: readonly {
         readonly name: string;
@@ -434,6 +472,37 @@ export declare const encodeValue: (value: unknown, type: string | undefined) =>
  *  authenticated chunks; the final frame is flagged so truncation is detected. */
 export declare function encryptStream(source: AsyncIterable<Uint8Array>, passphrase: string): AsyncGenerator<Uint8Array>;
 
+/**
+ * Framework tables whose rows describe the deployment they live in.
+ *
+ * Excluded from an export's `all` scope and skipped on import. Each entry says
+ * what a foreign row would assert about the target.
+ */
+export declare const ENVIRONMENT_LOCAL_TABLES: ReadonlyMap<string, string>;
+
+/**
+ * Framework tables that DO cross an environment boundary, and why.
+ *
+ * This map exists so the coverage guard can require a DECISION per table rather
+ * than a silence. An entry here is a positive statement: these rows are the
+ * app's, and carrying them is the point of the transfer.
+ */
+export declare const ENVIRONMENT_PORTABLE_TABLES: ReadonlyMap<string, string>;
+
+/**
+ * The sentence for an export or import that dropped some.
+ *
+ * Said out loud on every run that skips one. A bundle silently missing a table
+ * is the shape that makes a restore look complete when it is not — and the
+ * operator has to be able to tell "we left this out on purpose" from "this did
+ * not survive".
+ */
+export declare const environmentLocalNotice: (skipped: ReadonlyArray<string>, verb: string) => string | undefined;
+
+/** Why it is excluded — for the line that says so, so an operator learns the
+ *  rule rather than that a table went missing. */
+export declare const environmentLocalReason: (tableName: string) => string | undefined;
+
 /**
  * Run statements against the store's own connection.
  *
@@ -503,6 +572,22 @@ export declare const ExportScope: Schema.Struct<{
     kind: Schema.Literal<["all", "tenant", "tables"]>;
     tenantId: Schema.optional<typeof Schema.String>;
     tables: Schema.optional<Schema.Array$<typeof Schema.String>>;
+    /**
+     * Names deliberately left OUT of a `kind: 'all'` export.
+     *
+     * `--exclude` used to expand into `{ kind: 'tables', tables: <everything
+     * else> }`, and the manifest then recorded a bundle that had NAMED its
+     * contents. Two consequences, both silent: `all`'s environment-local filter
+     * stopped applying — so excluding two tables ADDED the migration ledger and
+     * the other deployment-describing tables back to the bundle — and `replace`
+     * refuses a named scope, so the honest way to leave a table out was also the
+     * way to make the bundle unusable for the mode it was being prepared for.
+     *
+     * Carrying the exclusion INSIDE `all` keeps both properties: the filter still
+     * runs, and the manifest still says "everything, except these", which is a
+     * different and truer claim than "these".
+     */
+    exclude: Schema.optional<Schema.Array$<typeof Schema.String>>;
 }>;
 
 export declare type ExportScope = Schema.Schema.Type<typeof ExportScope>;
@@ -756,6 +841,9 @@ declare const IntegrityError_base: Schema.TaggedErrorClass<IntegrityError, "Inte
     actual: typeof Schema.String;
 }>;
 
+/** Is this table one whose rows are about the deployment rather than the app? */
+export declare const isEnvironmentLocal: (tableName: string) => boolean;
+
 export declare interface LeakWarning {
     readonly table: string;
     readonly column: string;
@@ -801,6 +889,22 @@ export declare const Manifest: Schema.Struct<{
         kind: Schema.Literal<["all", "tenant", "tables"]>;
         tenantId: Schema.optional<typeof Schema.String>;
         tables: Schema.optional<Schema.Array$<typeof Schema.String>>;
+        /**
+         * Names deliberately left OUT of a `kind: 'all'` export.
+         *
+         * `--exclude` used to expand into `{ kind: 'tables', tables: <everything
+         * else> }`, and the manifest then recorded a bundle that had NAMED its
+         * contents. Two consequences, both silent: `all`'s environment-local filter
+         * stopped applying — so excluding two tables ADDED the migration ledger and
+         * the other deployment-describing tables back to the bundle — and `replace`
+         * refuses a named scope, so the honest way to leave a table out was also the
+         * way to make the bundle unusable for the mode it was being prepared for.
+         *
+         * Carrying the exclusion INSIDE `all` keeps both properties: the filter still
+         * runs, and the manifest still says "everything, except these", which is a
+         * different and truer claim than "these".
+         */
+        exclude: Schema.optional<Schema.Array$<typeof Schema.String>>;
     }>;
     consistency: Schema.Literal<["live", "snapshot"]>;
     compression: Schema.Literal<["zstd", "gzip", "none"]>;
@@ -1307,17 +1411,64 @@ declare interface SqlCapableStore extends DataStore {
     readonly run: <A, E>(effect: Effect.Effect<A, E, never>) => Promise<A>;
 }
 
+/**
+ * One table in the swap, and the columns to carry.
+ *
+ * The columns are REQUIRED, and an empty list is refused rather than treated as
+ * "all of them". The first version wrote `SELECT *`, which reads as obviously
+ * right and fails the moment the target has a STORED generated column:
+ *
+ *     ERROR: cannot insert a non-DEFAULT value into column "slug"
+ *     DETAIL: Column "slug" is a generated column.
+ *
+ * `CREATE TABLE … (LIKE t)` copies such a column as a PLAIN one (measured:
+ * `is_generated: NEVER` on the staging table), so `SELECT *` hands the target a
+ * value for a column it computes itself. Naming the columns — minus the
+ * generated ones, which the importer already tracks in `generatedByTable` for
+ * exactly this reason on the row path — is the fix, and requiring the list is
+ * what stops the trap returning by omission.
+ */
+declare interface StagedTable {
+    readonly name: string;
+    /** Every column to copy. Generated columns must NOT be here. */
+    readonly columns: ReadonlyArray<string>;
+}
+
 /** The suffix a staging table carries. `_voltro_`-prefixed so the boot differ's
  *  framework-table asymmetry treats it as ours and never plans it as a user
  *  table somebody forgot to declare. */
 export declare const STAGING_PREFIX = "_voltro_staging_";
 
+/** Whether this run stages, and — always — WHY NOT when it does not. */
+export declare interface StagingDecision {
+    readonly active: boolean;
+    readonly tables: ReadonlyArray<StagedTable>;
+    /** Absent iff `active`. A sentence an operator can act on. */
+    readonly blocker?: string;
+}
+
 /** The dialects this module knows how to stage on. */
 declare type StagingDialect = 'postgres' | 'mysql' | 'mariadb' | 'sqlite' | 'mssql';
 
 /** The staging table for `name`. */
 export declare const stagingNameFor: (name: string) => string;
 
+/** A snapshot shape both `stagedReferences` and the column read below accept. */
+export declare interface StagingSnapshot {
+    readonly tables: ReadonlyArray<{
+        readonly name: string;
+        readonly primaryKey?: ReadonlyArray<string> | undefined;
+        readonly columns: ReadonlyArray<{
+            readonly name: string;
+            readonly generatedAs?: unknown;
+            readonly references?: {
+                readonly table: string;
+                readonly column: string;
+            } | undefined;
+        }>;
+    }>;
+}
+
 /** Build an {@link AssetSink} over a storage provider. */
 export declare const storageAssetSink: (provider: StorageProvider) => AssetSink;