@voltro/sql-mssql 0.43.1 → 0.43.2

package/CHANGELOG.md

@@ -39,6 +39,37 @@ _Changes staged for the next release accumulate here (rolled up from
 
 ---
 
+## [0.43.2] — 2026-08-18
+
+### Fixed
+
+- **@voltro/sql-mysql, @voltro/database, @voltro/cli** — The mysql-family binlog reader no longer keeps a table excluded after the very migration that fixed it.
+
+  A UNIQUE on an unbounded text column is a MariaDB hash long-unique, whose hidden `DB_ROW_HASH_n` column the reader can never account for — so the table is held out of binlog capture and the exclusion is reported. `voltro dev` builds its store BEFORE it migrates (kv, cross-replica broadcast and the analytics mirror all need one), so on a boot whose own auto-migration bounds the column, that finding was drawn from a schema that stopped existing about a second later. The exclusion then outlived its cause for the life of the process, and the message — correct when written, and typically the only error line in the boot log — went on describing the pre-migration database.
+
+  Three changes:
+
+  - `DataStore.refreshChangeCaptureExclusions()` (optional; implemented by the mysql-family store) re-runs the probe and re-points the LIVE reader, in both directions — schema work that CREATES the condition now excludes the table immediately instead of after three failed writes. `voltro dev` calls it once, after all its schema work. `voltro serve` needs no equivalent: it builds its store after every schema step and never applies DDL itself. - The definitive message now states its own durability — that it is the schema as read at reader attach, and that applying the remedy does not by itself lift the exclusion. - On a boot that will re-check, the finding is reported as a provisional note rather than as a verdict, so a boot that fixes the condition leaves no error line about it. The note escalates to the full verdict on its own if the re-check never runs. - Two reader fixes the re-check depended on: applying a new exclusion set now WAITS for a reconnect already in flight (it is what applies the set, so resolving before it landed meant the caller's next write hit the old filter), and the reconnect loop no longer keeps resuming from an offset it has just jumped away from — that turned one purged offset into a reconnect every watchdog interval, forever, delivering nothing.
+- **@voltro/data-transfer, @voltro/database, @voltro/sql-mysql, @voltro/sql-postgres, @voltro/sql-sqlite, @voltro/sql-mssql, @voltro/cli** — `voltro data import --mode replace` no longer leaves a target in neither state, and now works against schemas that have foreign keys.
+
+  The delete step ran table by table and stopped at the first refusal, so a run that could not finish left dozens of tables emptied and nothing loaded — and a second attempt destroyed more than the first, because it got further before hitting the same wall. The wall itself was not exotic: MySQL, MariaDB and SQL Server check a foreign key as each ROW is deleted, so a table that references ITSELF cannot be emptied by any ordering of the tables. `createdBy → actors` on the `actors` table is exactly that shape, and it is what an audit mixin on an actor table produces.
+
+  - `DataStore.emptyTables()` (per dialect) empties the whole set as one unit, in one transaction, with referential integrity suspended for the duration — `FOREIGN_KEY_CHECKS` on the mysql family, a multi-table `TRUNCATE` on postgres, `defer_foreign_keys` on sqlite, per-table `NOCHECK`/`WITH CHECK CHECK` on mssql. All-or-nothing on every engine, including under `--atomic`, where it runs on the transaction the import already holds. - A **pre-flight refusal**: if a table OUTSIDE the bundle holds rows referencing one inside it, the import refuses before deleting anything and names the tables, the columns and the row counts. Those rows cannot be restored from the bundle, so forcing it is not an option. An EMPTY outside table blocks nothing. - Table-level failures carry the driver's own reason and code, the way row-level failures already did. `truncate <table> failed: Failed to execute statement` fits every plausible cause equally; the classification that produced `foreign key <name>: … [ER_NO_REFERENCED_ROW_2/1452]` one level down now applies one level up. The word "truncate" is gone from the message too — the step issues DELETE, and naming a statement it does not run sends whoever reads it to reproduce the wrong thing. - A typed refusal reaching the `--target api` transport keeps its text: the admin import endpoint answers `409` with the reason instead of flattening it to `import failed`, on the one transport where the operator has no other way to see it.
+
+  The bulk empty emits no change events, where the per-table loop emitted one per row. An import through `--target api` now asks every live subscription to re-read once it lands — the coarse refresh the framework already uses after a broadcast gap — so neither the missing deletes nor a table the bundle carries EMPTY leaves a subscriber holding rows that are gone. Wired where the route is mounted, which is the one place both boot paths share.
+
+  On postgres the emptying is a `DELETE` per table, not a `TRUNCATE`, and the difference is not performance: postgres refuses `TRUNCATE` on a table with ANY incoming foreign key, rows or not, while the mysql family refuses a DELETE only when rows actually reference the doomed ones. A `TRUNCATE` version made an EMPTY table outside the bundle block a replace on postgres and not on mariadb — one import, refused on one engine and accepted on the other, over a table holding nothing.
+
+  `--target api` also no longer times out on a full bundle. Both api-target calls went through `fetch`, whose undici default gives up after 300 s — a bound on the caller's database size, on a call whose response arrives only when the import does. They wait as long as the instance needs now, stream the body instead of buffering the whole bundle, and take `--timeout <seconds>` when a deadline is wanted. If one is hit, the message says the instance is probably still importing — and gives different advice for `replace` than for the idempotent modes, since re-running the first while it is mid-flight would empty the target under it.
+
+  Two more, found by measuring rather than by reading:
+
+  - **`--atomic` on postgres could not import a bundle that needed the deferred-FK repair at all.** A failed statement aborts the transaction there, and that repair depends on a row whose parent has not loaded yet failing, being held, and being retried — so the first such row poisoned every write after it. Every tolerated write now runs inside a savepoint. Per-row savepoints measured 2.40x the time of none on 5 000 rows, so they are amortised: one savepoint per batch of 200, and a batch that fails rolls back whole and replays row by row. The mysql family and sqlite leave a transaction usable after a failed statement and pay nothing for any of this. - **The replace pre-flight asked the caller's snapshot.** Over `--target api` that is the app's DECLARED schema, which cannot show a table the app stopped declaring but the database still has — and rows in a table nobody declares are exactly the rows nobody is watching. `DataStore.incomingForeignKeys()` reads the live catalog per dialect; the snapshot remains the fallback for stores without one.
+
+  A bundle bigger than one chunk is now uploaded as a series of short requests, so a proxy body cap or an ingress read timeout has nothing large to choke on, and the switch is automatic — the packer's stream is buffered one chunk ahead, so a small bundle is sent exactly as before and nobody has to know in advance which table is the big one. The import still runs ONCE, at the end, over the whole bundle. Resume is byte-exact (`packBundle` is deterministic over a directory, which this package now asserts), guarded by a bundle key so a different bundle under the same upload id is refused rather than spliced into the partial one, and by a contiguity check so a mis-ordered append cannot produce an archive that only fails later during decode. `--chunk-size <mb>` overrides the 16 MiB default.
+
+---
+
 ## [0.43.1] — 2026-08-18
 
 ### Fixed

package/dist/index.d.ts

@@ -270,6 +270,31 @@ export declare class MssqlStore implements DataStore {
     /* Excluded from this release type: getInternalExecuteDelete */
     /* Excluded from this release type: getInternalExecuteUpsert */
     /* Excluded from this release type: getInternalExecuteInsertIgnore */
+    /**
+     * Empty `tables` as one unit — see `DataStore.emptyTables`.
+     *
+     * SQL Server has no session switch for referential integrity, so the
+     * suspension is per TABLE: `NOCHECK CONSTRAINT ALL` stops the engine
+     * enforcing that table's foreign keys, the whole set is deleted, and then the
+     * constraints go back on. `WITH CHECK` on the way back is the half that must
+     * not be dropped — re-enabling without it leaves the constraint marked
+     * untrusted, which is invisible until the query planner quietly stops using
+     * it. The tables are empty by then, so the re-validation it forces is free.
+     *
+     * `TRUNCATE` is not an option here for the same reason it is not on the mysql
+     * family: SQL Server refuses it on any table with an incoming foreign key,
+     * whether or not that key has rows behind it.
+     */
+    emptyTables(tables: ReadonlyArray<string>): Promise<void>;
+    /** See `DataStore.incomingForeignKeys`. `sys.foreign_key_columns` is the
+     *  catalog view that already resolves both ends by object id, so no name
+     *  matching is involved. */
+    incomingForeignKeys(tables: ReadonlyArray<string>): Promise<ReadonlyArray<{
+        from: string;
+        column: string;
+        to: string;
+    }>>;
+    /* Excluded from this release type: emptyTablesOn */
     transactional<T>(work: (tx: DataStore) => Promise<T>): Promise<T>;
     /**
      * The DIALECT half of the shared transaction bracket

package/dist/index.js

@@ -1,21 +1,21 @@
 import { MssqlClient as e } from "@effect/sql-mssql";
 import { Config as t, Duration as n, Effect as r, Layer as i, ManagedRuntime as a, Redacted as o } from "effect";
-import { CDC_OFFSETS_TABLE as s, DEFAULT_ACQUIRE_TIMEOUT_MS as c, EagerCardinalityError as l, _voltroMssqlCdcOffsetsTable as u, attachEagerLoads as d, attributionFields as f, bulkInsertLimitsFor as p, chunkRowsForInsert as m, compileEagerJson as h, compilePredicate as g, compileRawFragment as _, compileSelect as v, decodeRowsFromSchema as y, externalChangeEvent as b, hasEagerLoads as x, isTableReactive as S, makeEagerFallbackReporter as ee, observeDbOp as C, qualifyTable as w, raiseChangeListenerCeiling as T, recordsTable as E, requireTable as D, runStoreTransaction as te, runWriteRecorders as O, stampGeneratedId as k, stampGeneratedIds as A, withCapturedAttribution as j } from "@voltro/database";
+import { CDC_OFFSETS_TABLE as s, DEFAULT_ACQUIRE_TIMEOUT_MS as c, EagerCardinalityError as l, _voltroMssqlCdcOffsetsTable as u, attachEagerLoads as d, attributionFields as f, bulkInsertLimitsFor as p, chunkRowsForInsert as m, compileEagerJson as h, compilePredicate as g, compileRawFragment as _, compileSelect as v, decodeRowsFromSchema as y, externalChangeEvent as b, hasEagerLoads as ee, isTableReactive as x, makeEagerFallbackReporter as te, observeDbOp as S, qualifyTable as C, quoteIdent as w, raiseChangeListenerCeiling as T, recordsTable as E, requireTable as ne, runStoreTransaction as D, runWriteRecorders as O, stampGeneratedId as k, stampGeneratedIds as A, withCapturedAttribution as j } from "@voltro/database";
 import { EventEmitter as M } from "node:events";
 import { createLogger as N } from "@voltro/logger";
 import { SqlClient as P, TransactionConnection as F } from "@effect/sql/SqlClient";
 //#region src/sqlLayer.ts
-var I = {
+var re = {
 	...e.defaultParameterTypes,
 	null: e.defaultParameterTypes.object
-}, ne = (e) => {
+}, ie = (e) => {
 	let t = e.acquireTimeoutMs ?? c;
 	return t > 0 ? t : void 0;
-}, L = (r) => {
-	let i = ne(r);
+}, I = (r) => {
+	let i = ie(r);
 	return e.layerConfig({
 		server: t.succeed(r.server),
-		parameterTypes: t.succeed(I),
+		parameterTypes: t.succeed(re),
 		...i === void 0 ? {} : { connectTimeout: t.succeed(n.millis(i)) },
 		...r.port === void 0 ? {} : { port: t.succeed(r.port) },
 		...r.database === void 0 ? {} : { database: t.succeed(r.database) },
@@ -25,7 +25,7 @@ var I = {
 		...r.trustServer === void 0 ? {} : { trustServer: t.succeed(r.trustServer) },
 		...r.maxConnections === void 0 ? {} : { maxConnections: t.succeed(r.maxConnections) }
 	});
-}, re = (e) => {
+}, L = (e) => {
 	let t = e.get("sslmode");
 	if (t !== null) {
 		if (t === "require") return !0;
@@ -53,7 +53,7 @@ var I = {
 			username: decodeURIComponent(r.username || "sa"),
 			password: decodeURIComponent(r.password || ""),
 			database: r.pathname.replace(/^\//, "") || "master",
-			...n(e.ssl ?? re(r.searchParams)),
+			...n(e.ssl ?? L(r.searchParams)),
 			...e.maxConnections === void 0 ? {} : { maxConnections: e.maxConnections },
 			...t
 		};
@@ -68,7 +68,7 @@ var I = {
 		...e.maxConnections === void 0 ? {} : { maxConnections: e.maxConnections },
 		...t
 	};
-}, z = (e) => L(R(e)), B = /* @__PURE__ */ new Set(["1205"]), V = (e) => {
+}, z = (e) => I(R(e)), B = /* @__PURE__ */ new Set(["1205"]), V = (e) => {
 	let t = e;
 	for (let e = 0; e < 5 && typeof t == "object" && t; e++) {
 		let e = t.number;
@@ -178,16 +178,16 @@ var I = {
 }, Z = N({ scope: "voltro:mssql" }), Q = async (e) => {
 	let t = e.changeStrategy ?? "inline", n = t;
 	t === "cdc" && !e.cdcConfig && (Z.warn("changeStrategy='cdc' requires cdcConfig (replicaId + includeTables); falling back to 'inline'."), n = "inline");
-	let r = e.tracerLayer ? i.mergeAll(e.sqlLayer, e.tracerLayer) : e.sqlLayer, o = a.make(r), s = new ie(await o.runPromise(P), o, n);
+	let r = e.tracerLayer ? i.mergeAll(e.sqlLayer, e.tracerLayer) : e.sqlLayer, o = a.make(r), s = new ae(await o.runPromise(P), o, n);
 	return n === "cdc" && e.cdcConfig && await s.startCdcConsumer(e.cdcConfig), s;
-}, ie = class e {
+}, ae = class e {
 	sql;
 	runtime;
 	changeStrategy;
 	namespace;
 	emitter;
 	inflightTxns = 0;
-	reportEagerFallback = ee(Z);
+	reportEagerFallback = te(Z);
 	cdcHandle = null;
 	cdcCheckpointTimer = null;
 	cdcPendingVersion = null;
@@ -203,11 +203,11 @@ var I = {
 		return t === this.namespace ? this : new e(this.sql, this.runtime, this.changeStrategy, t, this.emitter);
 	}
 	nsT(e) {
-		return w(this.namespace, e);
+		return C(this.namespace, e);
 	}
 	__mssqlReplicationFriend = { runEffect: (e) => this.runtime.runPromise(e) };
 	async executeQuery(e, t) {
-		let n = v(e, this.sql, this.namespace), i = t ? r.provideService(n, F, t) : n, a = await C("mssql", "select", () => this.runtime.runPromise(i));
+		let n = v(e, this.sql, this.namespace), i = t ? r.provideService(n, F, t) : n, a = await S("mssql", "select", () => this.runtime.runPromise(i));
 		return y(a, e.table, "mssql");
 	}
 	async executeInsert(e, t, n, i, a) {
@@ -290,20 +290,20 @@ var I = {
 			prev: e.old,
 			traceId: e.traceId,
 			subjectId: e.subjectId
-		}), S(e.table) && (t === null ? this.inlineEmit && this.emitter.emit("change", e) : t.push(e));
+		}), x(e.table) && (t === null ? this.inlineEmit && this.emitter.emit("change", e) : t.push(e));
 	}
 	query(e) {
 		return this.runWithEager(e, null);
 	}
 	raw(e, t) {
 		let n = _(e, this.sql);
-		return C("mssql", "raw", () => this.runtime.runPromise(n));
+		return S("mssql", "raw", () => this.runtime.runPromise(n));
 	}
 	async runWithEager(e, t) {
-		if (!x(e)) return this.executeQuery(e, t);
+		if (!ee(e)) return this.executeQuery(e, t);
 		let n = this.namespace === null ? h(e, this.sql, "mssql") : null;
 		if (n !== null) try {
-			let e = t ? r.provideService(n.fragment, F, t) : n.fragment, i = await C("mssql", "select", () => this.runtime.runPromise(e));
+			let e = t ? r.provideService(n.fragment, F, t) : n.fragment, i = await S("mssql", "select", () => this.runtime.runPromise(e));
 			return n.decode(i);
 		} catch (t) {
 			if (t instanceof l) throw t;
@@ -320,13 +320,13 @@ var I = {
 			reason: "not-compilable"
 		});
 		let i = await this.executeQuery(e, t);
-		return d(i, e.eager, e.sourceTable ?? D(e.table), (e) => this.executeQuery(e, t));
+		return d(i, e.eager, e.sourceTable ?? ne(e.table), (e) => this.executeQuery(e, t));
 	}
 	getInternalRunWithEager() {
 		return this.runWithEager.bind(this);
 	}
 	localWrite(e, t) {
-		return C("mssql", e, () => j(t));
+		return S("mssql", e, () => j(t));
 	}
 	insert(e, t) {
 		return this.localWrite("insert", (n) => this.executeInsert(e, t, null, null, n));
@@ -344,7 +344,7 @@ var I = {
 		return this.localWrite("delete", (n) => this.executeDelete(e, t, null, null, n));
 	}
 	async updateMany(e, t, n) {
-		let r = this.sql, i = g(n.where, r, this.namespace), a = r`UPDATE ${r(this.nsT(e))} SET ${r.update(t)} OUTPUT INSERTED.* WHERE ${i}`, o = await C("mssql", "update", () => this.runtime.runPromise(a));
+		let r = this.sql, i = g(n.where, r, this.namespace), a = r`UPDATE ${r(this.nsT(e))} SET ${r.update(t)} OUTPUT INSERTED.* WHERE ${i}`, o = await S("mssql", "update", () => this.runtime.runPromise(a));
 		for (let t of o) await this.routeEvent({
 			table: e,
 			op: "update",
@@ -354,7 +354,7 @@ var I = {
 		return o.length;
 	}
 	async deleteMany(e, t) {
-		let n = this.sql, r = g(t.where, n, this.namespace), i = n`DELETE FROM ${n(this.nsT(e))} OUTPUT DELETED.* WHERE ${r}`, a = await C("mssql", "delete", () => this.runtime.runPromise(i));
+		let n = this.sql, r = g(t.where, n, this.namespace), i = n`DELETE FROM ${n(this.nsT(e))} OUTPUT DELETED.* WHERE ${r}`, a = await S("mssql", "delete", () => this.runtime.runPromise(i));
 		for (let t of a) await this.routeEvent({
 			table: e,
 			op: "delete",
@@ -514,13 +514,37 @@ var I = {
 	getInternalExecuteInsertIgnore() {
 		return this.executeInsertIgnore.bind(this);
 	}
+	async emptyTables(e) {
+		await this.emptyTablesOn(e, null);
+	}
+	async incomingForeignKeys(e) {
+		if (e.length === 0) return [];
+		let t = this.sql;
+		return (await this.runtime.runPromise(t`
+        SELECT child.name AS fromTable, col.name AS columnName, parent.name AS toTable
+        FROM sys.foreign_key_columns fkc
+        JOIN sys.tables child ON child.object_id = fkc.parent_object_id
+        JOIN sys.tables parent ON parent.object_id = fkc.referenced_object_id
+        JOIN sys.columns col ON col.object_id = fkc.parent_object_id AND col.column_id = fkc.parent_column_id
+        WHERE parent.name IN ${t.in(e)}
+          AND child.name NOT IN ${t.in(e)}`)).map((e) => ({
+			from: e.fromTable,
+			column: e.columnName,
+			to: e.toTable
+		}));
+	}
+	async emptyTablesOn(e, t) {
+		if (e.length === 0) return;
+		let n = this.sql, i = (e) => this.nsT(e), a = (e, t) => n.unsafe(`ALTER TABLE ${w(i(e), "mssql")} ${t ? "WITH CHECK CHECK" : "NOCHECK"} CONSTRAINT ALL`), o = r.acquireUseRelease(r.forEach(e, (e) => a(e, !1), { discard: !0 }), () => r.forEach(e, (e) => n`DELETE FROM ${n(i(e))}`, { discard: !0 }), () => r.orDie(r.ignore(r.forEach(e, (e) => a(e, !0), { discard: !0 })))), s = t === null ? n.withTransaction(o) : r.provideService(o, F, t);
+		await this.runtime.runPromise(s);
+	}
 	async transactional(e) {
 		this.inflightTxns++;
 		try {
-			return await te({
+			return await D({
 				...this.txnSpec("MssqlStore.transactional"),
 				work: e,
-				makeView: (e, t) => new ae(this, e, t)
+				makeView: (e, t) => new oe(this, e, t)
 			});
 		} finally {
 			this.inflightTxns--;
@@ -551,7 +575,7 @@ var I = {
 		return this.changeStrategy === "cdc" ? "fleet" : "local";
 	}
 	injectExternalChange(e) {
-		S(e.table) && this.emitter.emit("change", b(e));
+		x(e.table) && this.emitter.emit("change", b(e));
 	}
 	run(e) {
 		return this.runtime.runPromise(e);
@@ -574,7 +598,7 @@ var I = {
 	async ping() {
 		await this.runtime.runPromise(this.sql`SELECT 1`);
 	}
-}, ae = class {
+}, oe = class {
 	parent;
 	txn;
 	attr;
@@ -586,6 +610,9 @@ var I = {
 	query(e) {
 		return this.parent.getInternalRunWithEager()(e, this.txn);
 	}
+	emptyTables(e) {
+		return this.parent.emptyTablesOn(e, this.txn);
+	}
 	insert(e, t) {
 		return this.parent.getInternalExecuteInsert()(e, t, this.txn, this.events, this.attr);
 	}
@@ -647,7 +674,7 @@ var I = {
 			this.events.length = 0;
 		}
 	}
-}, $ = (e) => e.__mssqlReplicationFriend ?? null, oe = (e, t) => e === t ? 0 : e < t ? -1 : 1, se = () => ({
+}, $ = (e) => e.__mssqlReplicationFriend ?? null, se = (e, t) => e === t ? 0 : e < t ? -1 : 1, ce = () => ({
 	async capturePrimaryPosition(e) {
 		let t = $(e);
 		if (t === null) throw Error("mssqlReplicationAdapter: primary is not an MssqlStore (missing __mssqlReplicationFriend).");
@@ -674,9 +701,9 @@ var I = {
 		}));
 	},
 	compare(e, t) {
-		return oe(t, e) >= 0 ? "caught-up" : "behind";
+		return se(t, e) >= 0 ? "caught-up" : "behind";
 	}
-}), ce = {
+}), le = {
 	id: "mssql",
 	makeSqlLayer: (e) => z(e),
 	makeStore: (e) => Q(e),
@@ -684,4 +711,4 @@ var I = {
 	retryFilter: G
 };
 //#endregion
-export { s as CDC_OFFSETS_TABLE, e as MssqlClient, u as _voltroMssqlCdcOffsetsTable, R as connectionFromConfig, q as ensureChangeTracking, L as makeMssqlSqlLayer, z as makeMssqlSqlLayerFromConfig, Q as makeMssqlStore, ce as mssqlDialect, se as mssqlReplicationAdapter, G as mssqlRetryFilter, X as startChangeTrackingCdc };
+export { s as CDC_OFFSETS_TABLE, e as MssqlClient, u as _voltroMssqlCdcOffsetsTable, R as connectionFromConfig, q as ensureChangeTracking, I as makeMssqlSqlLayer, z as makeMssqlSqlLayerFromConfig, Q as makeMssqlStore, le as mssqlDialect, ce as mssqlReplicationAdapter, G as mssqlRetryFilter, X as startChangeTrackingCdc };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@voltro/sql-mssql",
-  "version": "0.43.1",
+  "version": "0.43.2",
   "description": "SQL Server (mssql) dialect adapter for Voltro's cross-dialect DataStore (OUTPUT-clause returning, OFFSET/FETCH paging, inline reactivity).",
   "keywords": [
     "voltro",
@@ -35,8 +35,8 @@
   "dependencies": {
     "@effect/sql": "^0.52.0",
     "@effect/sql-mssql": "^0.53.0",
-    "@voltro/database": "0.43.1",
-    "@voltro/logger": "0.43.1"
+    "@voltro/database": "0.43.2",
+    "@voltro/logger": "0.43.2"
   },
   "peerDependencies": {
     "effect": "^3.22.0"