@voltro/database 0.43.0 → 0.43.2

package/CHANGELOG.md

@@ -39,6 +39,106 @@ _Changes staged for the next release accumulate here (rolled up from
 
 ---
 
+## [0.43.2] — 2026-08-18
+
+### Fixed
+
+- **@voltro/sql-mysql, @voltro/database, @voltro/cli** — The mysql-family binlog reader no longer keeps a table excluded after the very migration that fixed it.
+
+  A UNIQUE on an unbounded text column is a MariaDB hash long-unique, whose hidden `DB_ROW_HASH_n` column the reader can never account for — so the table is held out of binlog capture and the exclusion is reported. `voltro dev` builds its store BEFORE it migrates (kv, cross-replica broadcast and the analytics mirror all need one), so on a boot whose own auto-migration bounds the column, that finding was drawn from a schema that stopped existing about a second later. The exclusion then outlived its cause for the life of the process, and the message — correct when written, and typically the only error line in the boot log — went on describing the pre-migration database.
+
+  Three changes:
+
+  - `DataStore.refreshChangeCaptureExclusions()` (optional; implemented by the mysql-family store) re-runs the probe and re-points the LIVE reader, in both directions — schema work that CREATES the condition now excludes the table immediately instead of after three failed writes. `voltro dev` calls it once, after all its schema work. `voltro serve` needs no equivalent: it builds its store after every schema step and never applies DDL itself. - The definitive message now states its own durability — that it is the schema as read at reader attach, and that applying the remedy does not by itself lift the exclusion. - On a boot that will re-check, the finding is reported as a provisional note rather than as a verdict, so a boot that fixes the condition leaves no error line about it. The note escalates to the full verdict on its own if the re-check never runs. - Two reader fixes the re-check depended on: applying a new exclusion set now WAITS for a reconnect already in flight (it is what applies the set, so resolving before it landed meant the caller's next write hit the old filter), and the reconnect loop no longer keeps resuming from an offset it has just jumped away from — that turned one purged offset into a reconnect every watchdog interval, forever, delivering nothing.
+- **@voltro/data-transfer, @voltro/database, @voltro/sql-mysql, @voltro/sql-postgres, @voltro/sql-sqlite, @voltro/sql-mssql, @voltro/cli** — `voltro data import --mode replace` no longer leaves a target in neither state, and now works against schemas that have foreign keys.
+
+  The delete step ran table by table and stopped at the first refusal, so a run that could not finish left dozens of tables emptied and nothing loaded — and a second attempt destroyed more than the first, because it got further before hitting the same wall. The wall itself was not exotic: MySQL, MariaDB and SQL Server check a foreign key as each ROW is deleted, so a table that references ITSELF cannot be emptied by any ordering of the tables. `createdBy → actors` on the `actors` table is exactly that shape, and it is what an audit mixin on an actor table produces.
+
+  - `DataStore.emptyTables()` (per dialect) empties the whole set as one unit, in one transaction, with referential integrity suspended for the duration — `FOREIGN_KEY_CHECKS` on the mysql family, a multi-table `TRUNCATE` on postgres, `defer_foreign_keys` on sqlite, per-table `NOCHECK`/`WITH CHECK CHECK` on mssql. All-or-nothing on every engine, including under `--atomic`, where it runs on the transaction the import already holds. - A **pre-flight refusal**: if a table OUTSIDE the bundle holds rows referencing one inside it, the import refuses before deleting anything and names the tables, the columns and the row counts. Those rows cannot be restored from the bundle, so forcing it is not an option. An EMPTY outside table blocks nothing. - Table-level failures carry the driver's own reason and code, the way row-level failures already did. `truncate <table> failed: Failed to execute statement` fits every plausible cause equally; the classification that produced `foreign key <name>: … [ER_NO_REFERENCED_ROW_2/1452]` one level down now applies one level up. The word "truncate" is gone from the message too — the step issues DELETE, and naming a statement it does not run sends whoever reads it to reproduce the wrong thing. - A typed refusal reaching the `--target api` transport keeps its text: the admin import endpoint answers `409` with the reason instead of flattening it to `import failed`, on the one transport where the operator has no other way to see it.
+
+  The bulk empty emits no change events, where the per-table loop emitted one per row. An import through `--target api` now asks every live subscription to re-read once it lands — the coarse refresh the framework already uses after a broadcast gap — so neither the missing deletes nor a table the bundle carries EMPTY leaves a subscriber holding rows that are gone. Wired where the route is mounted, which is the one place both boot paths share.
+
+  On postgres the emptying is a `DELETE` per table, not a `TRUNCATE`, and the difference is not performance: postgres refuses `TRUNCATE` on a table with ANY incoming foreign key, rows or not, while the mysql family refuses a DELETE only when rows actually reference the doomed ones. A `TRUNCATE` version made an EMPTY table outside the bundle block a replace on postgres and not on mariadb — one import, refused on one engine and accepted on the other, over a table holding nothing.
+
+  `--target api` also no longer times out on a full bundle. Both api-target calls went through `fetch`, whose undici default gives up after 300 s — a bound on the caller's database size, on a call whose response arrives only when the import does. They wait as long as the instance needs now, stream the body instead of buffering the whole bundle, and take `--timeout <seconds>` when a deadline is wanted. If one is hit, the message says the instance is probably still importing — and gives different advice for `replace` than for the idempotent modes, since re-running the first while it is mid-flight would empty the target under it.
+
+  Two more, found by measuring rather than by reading:
+
+  - **`--atomic` on postgres could not import a bundle that needed the deferred-FK repair at all.** A failed statement aborts the transaction there, and that repair depends on a row whose parent has not loaded yet failing, being held, and being retried — so the first such row poisoned every write after it. Every tolerated write now runs inside a savepoint. Per-row savepoints measured 2.40x the time of none on 5 000 rows, so they are amortised: one savepoint per batch of 200, and a batch that fails rolls back whole and replays row by row. The mysql family and sqlite leave a transaction usable after a failed statement and pay nothing for any of this. - **The replace pre-flight asked the caller's snapshot.** Over `--target api` that is the app's DECLARED schema, which cannot show a table the app stopped declaring but the database still has — and rows in a table nobody declares are exactly the rows nobody is watching. `DataStore.incomingForeignKeys()` reads the live catalog per dialect; the snapshot remains the fallback for stores without one.
+
+  A bundle bigger than one chunk is now uploaded as a series of short requests, so a proxy body cap or an ingress read timeout has nothing large to choke on, and the switch is automatic — the packer's stream is buffered one chunk ahead, so a small bundle is sent exactly as before and nobody has to know in advance which table is the big one. The import still runs ONCE, at the end, over the whole bundle. Resume is byte-exact (`packBundle` is deterministic over a directory, which this package now asserts), guarded by a bundle key so a different bundle under the same upload id is refused rather than spliced into the partial one, and by a contiguity check so a mis-ordered append cannot produce an archive that only fails later during decode. `--chunk-size <mb>` overrides the 16 MiB default.
+
+---
+
+## [0.43.1] — 2026-08-18
+
+### Fixed
+
+- **@voltro/database** — A write recorder that fails inside someone's transaction now says which recorder, on which write, and what the database actually said.
+
+  `@effect/sql` renders every driver failure as `SqlError: Failed to execute statement` — one sentence that fits a missing column, a dangling foreign key, an over-long value and a duplicate key equally. The driver's own words hang off a SYMBOL on a `FiberFailure`, so a caller who reaches for `.cause` gets `undefined` and concludes there is nothing there.
+
+  That is expensive precisely where recorders run: the caller's write was ordinary, and what failed was framework machinery one table over. The message now reads
+
+  ```
+  write recorder '_voltro_row_history' failed while recording an update on 'users':
+  Duplicate entry 'rowver_…' for key 'PRIMARY' [code=ER_DUP_ENTRY errno=1062 …]
+  ```
+
+  and the original error is kept as `cause` for anyone who does walk the chain. The failure still takes the transaction down — that is the guarantee and it is unchanged.
+
+  `describeDriverError` (`@voltro/database`) is the shared summariser, built on the existing cause extractor rather than a second walker. It returns nothing when the chain carries nothing driver-shaped, so an ordinary programming error from a recorder arrives as itself instead of wrapped in prose about a database.
+- **@voltro/database** — On MariaDB, a `varchar` column could introspect as `json` because a DIFFERENT table had a json column with the same column name.
+
+  MariaDB names a column-level CHECK after the COLUMN, and those names are unique per table, not per schema. `information_schema.check_constraints` on MySQL has no `TABLE_NAME`, so the introspector recovered it by joining `table_constraints` on `(schema, constraint_name)` — which on MariaDB cross-products every same-named check across every table. Measured on 11.8:
+
+  ```
+  a.payload  LONGTEXT      CHECK (json_valid(`payload`))
+  b.payload  VARCHAR(255)  CHECK (`payload` in ('x','y'))
+  
+  join result:  a → json_valid, a → in(…), b → json_valid, b → in(…)
+  ```
+
+  So `b.payload` reads as `json`, and `a.payload` picks up an enum it does not have. Downstream that is not a cosmetic label: the planner emits a blocked `alter-column-type` with `from: 'json'` that no `.narrowedFrom()` can honestly acknowledge, because the premise is false — and since the data-transfer manifest records introspected types, the same misreading travels into the bundle and reappears as schema drift on import.
+
+  MariaDB's own `check_constraints` HAS `TABLE_NAME`. The introspector asks for it directly now and keeps the join as the MySQL path, where check-constraint names are schema-unique and the join is sound. That asymmetry is why a single-engine test could not see this: the wrong query passes on MySQL.
+- **@voltro/database** — `voltro db apply` could not drop a CHECK constraint on the mysql family at all, and each of the three reasons hid the next.
+
+  **1. `DROP CHECK` is MySQL-8 syntax.** MariaDB has never had it — measured on 11.8, `ALTER TABLE t DROP CHECK c` is `ERROR 1064`, while `DROP CONSTRAINT c` works on both engines. A plan containing a `drop-check` therefore died on the first one, on a family whose migrations are NOT atomic: the run stopped with the earlier statements committed and no rollback.
+
+  **2. The name was assumed, not read.** The applier dropped `<table>_<column>_check` — which is only what its own `add-check` would have named it. A CHECK created at table bring-up is INLINE and UNNAMED, so the server names it (`CONSTRAINT_1` / `<table>_chk_1` / the column name). Dropping a name that does not exist reports "does not exist", which is indistinguishable from the "already gone" a resume legitimately produces — so the statement succeeded, the constraint stayed, and the plan re-proposed the identical `drop-check` forever. The name comes from the catalog now.
+
+  **3. A column-level CHECK cannot be dropped by name on MariaDB at all.** Measured: the catalog lists it under the column's name, `DROP CONSTRAINT` on that name answers 1091, and only redefining the column removes it. The applier now drops by catalog name, ASKS whether the constraint survived, and redefines the column when it did — so a rebuild happens only in the case that needs one.
+
+  **And the same investigation closed the MySQL `.oneOf()` round-trip gap.** `parseEnumCheck` was documented as handling MySQL's rendering and did not: MySQL backslash-escapes the string DELIMITERS (`_utf8mb4\'draft\'`), and the parser rewrote those to the SQL doubling `''` — which is how a quote INSIDE a value is written. Every delimiter became escaped content, every value came back empty, and the filter dropped them. `.oneOf()` now round-trips on MySQL, and the suite that asserted the gap as a known one asserts the round-trip instead.
+
+  Plus, on a failed apply: the error now states how many operations were already applied and whether this dialect rolls back. The ledger held that number; it never reached the operator, who had to re-plan and diff the counts to learn how far the run got.
+- **@voltro/plugin-versioning** — `versioningPlugin({ timing: 'in-transaction' })` built its history row's primary key from `(rowId, version)` while the version counter three lines above was scoped to `(tableName, rowId)`. Two versioned tables carrying the same row id therefore collided — permanently.
+
+  The shape is not exotic: `actors.id === users.id` is what the framework's own audit trail asks for, an `actors` row whose id is the user's so an audited write satisfies `createdBy → actors`. In an app that follows it, every user row has a twin.
+
+  The collision does not heal, and that is what turns a duplicate into an outage. The second table's insert fails, so its history row is never written, so `maxOf` for that table stays `null`, so the next attempt computes the same version and the same id. Every write to that row is dead from then on — surfacing as `ER_DUP_ENTRY` on an ordinary `store.update`, naming a row id in a table the caller never wrote to.
+
+  The key is `(table, rowId, version)` now — the same shape the post-commit path always built. It stays deterministic (no clock, no process-local counter), which is what lets it survive a replica restart; it just carries every part of the key it claims to be unique over.
+
+  **No cleanup is needed for rows already written.** They keep their old ids and belong to whichever table wrote them; the new keys cannot collide with them, and `byRow` is not unique. An app blocked by this is unblocked by the upgrade alone.
+
+  Covered twice: the recorder against a port that refuses duplicates (the mechanism, including that a repeat does not settle), and two versioned tables sharing an id against live postgres (the real primary key, inside the caller's transaction). The suite that existed exercised ONE table, which cannot produce a collision at all — and read exactly like a suite that covered this.
+
+### Internal (no consumer-facing effect)
+
+- **@voltro/sql-mysql** — Two test-only defects in `sql-mysql`, both found by a release gate, both of the same family: a check that could not fail, and a failure reported in the wrong place. No product code changed.
+
+  **An assertion that could not fail.** `dropCheckSyntax.integration.test.ts` fell back to a HAND-BUILT plan when the planner produced no operations — and the fabricated operation was a `drop-check`, which is exactly what the next line asserts the plan contains. So an engine whose planner stopped emitting it would have been handed one and reported green. The fallback is deleted; both engines produce the operation now, which is what this release fixed, and the assertion is load-bearing again (4/4 on mysql AND mariadb without it).
+
+  It surfaced as a TYPE error rather than a false pass, because the fallback's object widened `plan` into a union `applyPlan` does not accept. Worth noting which check caught it: `vitest` transpiles without type-checking, so the suite was green and only `tsc` objected — the gate's `typecheck` and `lint` steps are what went red.
+
+  **A wait that gave up in silence.** `waitFor` in both binlog CDC suites looped to a deadline and then RETURNED, so a "prove the reader is live" wait that expired let the test carry on, kill the binlog dump thread, and fail twenty lines later on `expect(ids).toContain('todo_wd_before')` — an assertion about a different claim, in a different place. It throws now, naming the wait and the window, and all twelve call sites carry a label.
+
+  The window is named too: `FIRST_ATTACH_MS = 30_000`, up from 12 s. The reasoning is the 40 s window already in the same file, whose comment says a re-attach plus binlog catch-up takes longer under a loaded full-suite run — a FIRST attach does both and only skips the backoff, so 12 s beside 40 s was an asymmetry the file's own reasoning did not support. That is an argument from the neighbouring comment, not a measurement; if it expires again, `waitFor` now says which wait and for how long, and that number is the one to argue with rather than raising this one twice.
+
+---
+
 ## [0.43.0] — 2026-08-18
 
 ### ⚠ BREAKING

package/dist/index.d.ts

@@ -602,12 +602,39 @@ export declare interface CdcConfig {
      *  proactive anti-stall. Default 10s. See
      *  `BinlogCdcOptions.keepAliveInitialDelayMs`. */
     readonly keepAliveInitialDelayMs?: number;
+    /**
+     * The caller promises to call `DataStore.refreshChangeCaptureExclusions()`
+     * once its schema work is done.
+     *
+     * Set by the boot path that migrates in the SAME process as the reader
+     * (`voltro dev`). It buys one thing: a table the store cannot decode on the
+     * pre-migration shape is reported as PROVISIONAL rather than as a verdict, so
+     * a boot whose own migration removes the cause does not leave a fully-argued
+     * error in the log about a condition that was gone before anyone read it.
+     * `voltro serve` builds its store after all schema work and leaves this
+     * unset, because there the reading at attach IS the final one.
+     *
+     * The promise is not taken on trust: the store escalates the provisional note
+     * to the definitive error on its own if no refresh arrives. A flag that turns
+     * a loud line quiet has to fail in the loud direction.
+     */
+    readonly exclusionRefreshFollows?: boolean;
 }
 
 /** Maximum `onChange` listeners a DataStore's change bus accepts before Node
  *  warns. See this file's header for why it is 512 and why it is not `0`. */
 export declare const CHANGE_LISTENER_CEILING = 512;
 
+/** What a `refreshChangeCaptureExclusions()` pass changed. */
+export declare interface ChangeCaptureExclusionRefresh {
+    /** Excluded before and after — the condition is real and still there. */
+    readonly stillExcluded: ReadonlyArray<string>;
+    /** Excluded before, decodable now — the schema work removed the cause. */
+    readonly readmitted: ReadonlyArray<string>;
+    /** Decodable before, excluded now — the schema work INTRODUCED the cause. */
+    readonly newlyExcluded: ReadonlyArray<string>;
+}
+
 /** A row was removed from the table. `new` is null, always. */
 export declare const changeDelete: (table: string, row: Row, meta?: ChangeEventMeta) => ChangeEvent;
 
@@ -2284,6 +2311,117 @@ export declare interface DataStore {
      * omits it (always reachable). Callers guard with `if (store.ping)`.
      */
     ping?(): Promise<void>;
+    /**
+     * Empty every one of `tables` as ONE unit, with referential integrity
+     * suspended for the duration.
+     *
+     * Written for `voltro data import --mode replace`, whose whole job is to put
+     * a target into the state a bundle describes. Two properties, and both are
+     * the point:
+     *
+     *   - **All or nothing.** A run that cannot empty every table must leave the
+     *     target exactly as it found it. The loop this replaces deleted table by
+     *     table and stopped at the first refusal, so an operator's *second*
+     *     attempt destroyed more than the first — the deletes that had already
+     *     succeeded stayed gone and nothing was reloaded.
+     *   - **Order-free.** A parent cannot be emptied while a child still points
+     *     at it, and no ordering fixes the two shapes that occur in every real
+     *     schema: a table that references ITSELF (the framework's own actor
+     *     pattern — `createdBy` on `actors` points at `actors`), and a genuine
+     *     cycle. On engines that check each row as it is deleted, ordering the
+     *     TABLES cannot help, because the conflict is between two ROWS of one of
+     *     them.
+     *
+     * What it deliberately does NOT do is reach outside `tables`. A row in some
+     * OTHER table pointing into this set is not ours to delete, and suspending
+     * integrity would strand it. The caller checks for that first and refuses;
+     * see `runImport`'s replace pre-flight.
+     *
+     * Emits no change events, and the consequence is worth stating rather than
+     * implying. One event per deleted row is not affordable at bundle scale — a
+     * replace of a grown target fanned tens of thousands of them out to every live
+     * subscriber — but silence has an edge. In practice the LOAD that follows
+     * covers it: the reactive layer re-reads a query when any row of its table
+     * changes, so the first inserted row refreshes subscribers and the emptied
+     * rows leave their view with it. The exception is a table the bundle carries
+     * EMPTY: nothing is loaded, nothing is emitted, and a subscriber keeps rows
+     * that are gone until it re-subscribes.
+     *
+     * OPTIONAL on the interface: SQL stores implement it per dialect (the
+     * suspension mechanism is engine-specific and session-scoped, so it has to
+     * live next to the connection). Callers guard with `if (store.emptyTables)`
+     * and fall back to deleting inside one transaction — which keeps the
+     * all-or-nothing half on any store, and is all the in-memory store needs.
+     */
+    emptyTables?(tables: ReadonlyArray<string>): Promise<void>;
+    /**
+     * Foreign keys pointing INTO `tables` from tables that are NOT in the set,
+     * read from the LIVE catalog.
+     *
+     * The question `--mode replace` has to answer before it deletes anything:
+     * emptying a parent whose children live outside the bundle would strand rows
+     * the bundle cannot restore. It was answered from the caller's SNAPSHOT, which
+     * is right for the CLI (it introspects the target) and wrong for the admin
+     * endpoint, which passes the app's DECLARED schema — that cannot show a table
+     * the app has stopped declaring but the database still has, and those are
+     * exactly the rows nobody is watching.
+     *
+     * The catalog is the only authority here, so the store asks it.
+     *
+     * OPTIONAL: SQL stores implement it; the in-memory store has no catalog and no
+     * foreign keys, so it omits it and callers fall back to the snapshot.
+     */
+    incomingForeignKeys?(tables: ReadonlyArray<string>): Promise<ReadonlyArray<{
+        readonly from: string;
+        readonly column: string;
+        readonly to: string;
+    }>>;
+    /**
+     * Run `fn` inside a nested savepoint, so a failure inside it costs only its
+     * own work instead of the whole transaction.
+     *
+     * Exists for one reason: on postgres (and SQL Server for some errors) a failed
+     * statement ABORTS the enclosing transaction — every statement after it
+     * answers `current transaction is aborted` and nothing can be salvaged. The
+     * importer's deferred-foreign-key repair depends on the opposite: a row whose
+     * parent has not loaded yet is EXPECTED to fail, is held, and is retried once
+     * the rest has streamed. That repair is how a bundle whose row order puts a
+     * child before its parent lands at all — and ids sort that way as often as
+     * not. Without a savepoint the two are incompatible, so `--atomic` simply
+     * could not import such a bundle on those engines.
+     *
+     * Only the TRANSACTIONAL views implement it, and only for the engines that
+     * need it: the mysql family and sqlite leave a transaction usable after a
+     * failed statement, so wrapping every row there would buy nothing and cost a
+     * round trip per row. Callers guard with `if (store.savepoint)` and run `fn`
+     * directly otherwise — which is also correct outside a transaction, where
+     * there is nothing to protect.
+     *
+     * Failures propagate: the savepoint is rolled back and the error rethrown, so
+     * the caller's own error handling is unchanged.
+     */
+    savepoint?<A>(fn: () => Promise<A>): Promise<A>;
+    /**
+     * Re-decide which tables this store's change-capture reader can decode, and
+     * apply the answer to the LIVE reader.
+     *
+     * Exists because that decision is a reading of the SCHEMA, and a boot that
+     * migrates in the same process moves the schema after the reader attached.
+     * The reader is built before the migration runs (it has consumers — kv,
+     * broadcast, the analytics mirror — that need a store first), so a table
+     * excluded on the pre-migration shape stayed excluded for the life of the
+     * process even when the migration was the very thing that fixed it.
+     *
+     * Call it once every boot, AFTER all schema work. Cheap (one catalog query)
+     * and a no-op on a store whose verdict did not move — including every store
+     * with nothing to exclude.
+     *
+     * OPTIONAL on the interface: only the mysql-family binlog reader has a
+     * physical-layout exclusion at all. Postgres NOTIFY, mssql Change Tracking
+     * and the in-memory store decode whatever they are given, so they omit it and
+     * callers guard with `if (store.refreshChangeCaptureExclusions)`.
+     */
+    refreshChangeCaptureExclusions?(): Promise<ChangeCaptureExclusionRefresh>;
 }
 
 /** Calendar date without time. */
@@ -2577,6 +2715,23 @@ export declare interface DeriveSubjectGraphInput {
  */
 export declare const deriveTypeIdPrefix: (tableName: string) => string;
 
+/**
+ * A one-line driver summary for an error, or `undefined` when the chain carries
+ * nothing driver-shaped.
+ *
+ * `@effect/sql` renders every failure as `SqlError: Failed to execute
+ * statement`. That single sentence is what a missing column, a foreign key with
+ * no parent, a value too long and a duplicate key all say — four failures with
+ * four different fixes and one string. The driver's own words are in the cause,
+ * and the cause hangs off a SYMBOL on a `FiberFailure`, so a caller who reaches
+ * for `.cause` finds `undefined` and concludes there is nothing there.
+ *
+ * Built on `extractDbCause` rather than a second walker: that function already
+ * knows where every driver puts its fields, and a copy of it would be a second
+ * definition nothing keeps in step.
+ */
+export declare const describeDriverError: (err: unknown) => string | undefined;
+
 /**
  * Parse JSON-stringified array values back to JS arrays on the read
  * side. Tolerates `null` (NULL column → null in JS), pre-deserialised