@spooky-sync/core 0.0.1-canary.334 → 0.0.1-canary.336

package/AGENTS.md

@@ -27,7 +27,13 @@ Sp00kyClient<S>  ── facade: every method = one saga run, selector or subscri
 
 Local writes are applied optimistically (row + outbox entry in one local
 transaction), rendered through the outbox overlay, and drained to the remote in
-batches. Server membership arrives through one batched edge read (registration,
+FIFO batches. Each statement of a batch commits on its own, but the server stops
+the batch at its first failed statement (`remoteBatch` chains them), so no write
+reaches the server ahead of an earlier one that failed. A batch whose answer is
+lost (it threw a transport error, or its rows were loaded from the store after a
+restart) is in doubt (`OutboxItem.inDoubt`): re-sent, an in-doubt CREATE checks
+the row already on the server and is a no-op when it holds what the CREATE
+writes. Server membership arrives through one batched edge read (registration,
 poll, LIVE dirt) and bodies are fetched once across all queries. Rows are always
 re-materialized from state, never patched in place.
 
@@ -59,7 +65,7 @@ re-materialized from state, never patched in place.
 - **A never-resolved query paints only vouched rows.** Before the server answers (`cold`), the local predicate scan shows a cached body only if some `_00_view` row names it (as a member or subquery child) or a write for it is in flight. A body no view names is an orphan (deleted upstream while nothing watched it): it stays hidden, and the GC deletes it 30 s after boot and then hourly. A `_00_view` row no query has resolved for 14 days is retired by the same sweep, so that query paints cold again.
 - **`database.store` defaults to `'memory'`.** `'indexeddb'` keeps one IndexedDB database per user bucket (`sp00ky-<bucket>`) and needs `@surrealdb/wasm` >= 3.1.0: the 3.0.x engine opened the store but failed every write (`An IndexedDB error occured: idb error`), so each session fell back to memory. 3.1.0 ships the engine as `@surrealdb/wasm-native`; a Vite app lists it in `optimizeDeps.exclude` next to `@surrealdb/wasm`. A store that still cannot open after a drop runs the session in memory (error log `falling back to in-memory`). Under `localEngine: 'sqlite'`, `'indexeddb'` means OPFS.
 - **Boot owns the first identity.** The facade subscribes to auth with `immediate: false`: boot opens the bucket the hint names and `auth.restoreSession` announces the user. `LocalDatabaseService.switchStore` also waits for an in-flight `connect`, and lets statements already past the gate finish before it closes a client: the 3.1 engine streams results from its worker, and a statement whose client closed under it stayed pending until its 30 s deadline with every later local statement queued behind it.
-- **A rejected write is not lost.** Application errors roll the local row back (create/update/delete) and move the mutation to `_00_failed_mutations`; the tray API above lists, retries (as a fresh write) or discards it. Network errors keep the row queued.
+- **A rejected write is not lost.** Application errors roll the local row back (create/update/delete) and move the mutation to `_00_failed_mutations`; the tray API above lists, retries (as a fresh write) or discards it. Network errors keep the row queued, and so does a write SurrealDB refused only because sp00ky's own `_00_` ingest event could not post to the scheduler/SSP (refused, past its TIMEOUT, DNS, 5xx/408/429; `SP00KY_EVENT_POST_FAILED` in `utils/error-classification.ts`).
 
 ## Module map (under `src/`)
 

package/README.md

@@ -49,7 +49,10 @@ back.
 acked (kept in the overlay until membership names them), rejected rows are
 rolled back and moved to `_00_failed_mutations` in the same transaction that
 deletes the pending row, a transport cut leaves the tail queued behind a
-backoff. The tray is exposed as `failedMutationCount`,
+backoff. A push whose answer never came (timeout, dropped socket, a session
+that died mid-push) may have committed, so its rows are in doubt: re-sent, an
+in-doubt CREATE that finds a matching row of its own on the server is a no-op
+instead of an "already exists" rollback. The tray is exposed as `failedMutationCount`,
 `subscribeToFailedMutations`, `listFailedMutations`, `retryFailedMutation`
 and `discardFailedMutation`.
 

package/dist/index.d.ts

@@ -1339,6 +1339,14 @@ interface OutboxItem {
   readonly status: OutboxStatus;
   readonly ackedAt: number | null;
   readonly attempts: number;
+  /**
+   * A request carrying this row may have committed without the answer
+   * reaching this client: it threw a transport error, or the row was loaded
+   * from the store, so a session that died mid-push may have sent it. Its
+   * CREATE then accepts the row already on the server when that row holds
+   * what it would write (see `remoteBatch`). Absent means false.
+   */
+  readonly inDoubt?: boolean;
 }
 type TabRole = 'solo' | 'leader' | 'follower';
 /** A debounced update accumulating locally until its flush writes the outbox row. */

package/dist/index.js

@@ -546,8 +546,32 @@ const NETWORK_ERROR_PATTERNS = [
 	"specify a namespace",
 	"specify a database"
 ];
+/**
+* A write SurrealDB refused because sp00ky's OWN ingest post failed: every
+* synced table's `_00_<table>_mutation` / `_00_<table>_delete` event posts to
+* the scheduler or SSP inside the write's transaction (`sp00ky.rs::ingest_post`),
+* so a scheduler that is down, slow past the event TIMEOUT, or answering 5xx
+* aborts the write. That is an outage, not the mutation's fault, and it clears
+* on its own; classified `application` it was rolled back and discarded, so
+* every write queued during a scheduler restart or a cold SSP was lost.
+*
+* Wording on SurrealDB 3.1.5 and 3.3.4 (the event name may be nested inside an
+* app event that wrote to a synced table):
+*   "Error while processing event _00_item_mutation: There was an error
+*    processing a remote HTTP request: error sending request for url (...)"
+*    (refused connection or TIMEOUT), "...: 503 Service Unavailable" (5xx),
+*   "Error while processing event _00_item_mutation: failed to lookup address
+*    information: ..." (DNS).
+*
+* Only `_00_` events count: an app's own event posting to its own endpoint
+* stays `application`, as does a 4xx from `/ingest` other than 408/429 (a
+* malformed or oversized payload, a secret mismatch), which no retry fixes and
+* which would otherwise hold the outbox's serial lane forever.
+*/
+const SP00KY_EVENT_POST_FAILED = /error while processing event _00_[^:\s]*: (?:there was an error processing a remote http request: (?:error sending request|5\d\d\b|408\b|429\b)|failed to lookup address information)/;
 function classifySyncError(error) {
 	const message = error instanceof Error ? error.message.toLowerCase() : String(error).toLowerCase();
+	if (SP00KY_EVENT_POST_FAILED.test(message)) return "network";
 	for (const pattern of NETWORK_ERROR_PATTERNS) if (message.includes(pattern)) return "network";
 	return "application";
 }
@@ -1382,6 +1406,11 @@ function hydrateRowData(row, columns) {
 		return row;
 	}
 }
+/**
+* A row read back from the store at boot, bucket switch or leader takeover.
+* The session that queued it may have sent it and died before the answer (a
+* closed tab, a killed app, a lost leader), so its outcome is in doubt.
+*/
 const toOutboxItem = (row) => ({
 	id: row.id,
 	type: row.mutationType,
@@ -1389,7 +1418,8 @@ const toOutboxItem = (row) => ({
 	table: row.tableName,
 	status: "pending",
 	ackedAt: null,
-	attempts: 0
+	attempts: 0,
+	inDoubt: true
 });
 /** The `_00_pending_writes` id for a minted mutation id (same suffix). */
 const pendingWriteId = (mutationId) => `${PENDING_WRITES_TABLE}:${mutationId.includes(":") ? mutationId.slice(mutationId.indexOf(":") + 1) : mutationId}`;
@@ -1526,28 +1556,86 @@ function planDeferredOutboxRowTx(input) {
 	};
 }
 /**
-* Many outbox rows as ONE multi-statement request. Statement i answers row i.
-* Each statement is its own transaction on the server, as before.
+* What a statement of a `remoteBatch` throws instead of running, because an
+* earlier statement in the same request failed.
+*/
+const BATCH_SKIPPED = "sp00ky: not run, an earlier statement in this batch failed";
+/** A CRDT edit-log row (`<table>__crdt_log`): written once under a fresh id, never changed. */
+const isLogTable = (table) => table.endsWith("__crdt_log");
+/**
+* Which fields of the row already on the server must equal the payload of the
+* CREATE at `i` for that row to count as this CREATE's own earlier commit, or
+* `null` when the CREATE must not tolerate an existing row at all.
+*
+* Only a CREATE whose outcome is in doubt tolerates one: elsewhere "already
+* exists" means someone else holds the id, a rejection as before. The check
+* is every payload field except those a later in-doubt row of the same
+* record, which may have committed in the same lost round, could have changed
+* since; those rows run again after it and set them anyway. A later in-doubt
+* DELETE or CREATE of the record may have replaced the row, so then existence
+* is all that is checked. Rows that are not in doubt never ran.
+*
+* A CRDT log row's id is fresh and the row immutable, so its existence is
+* proof enough, in doubt or not.
+*/
+function matchFields(rows, i, inDoubt) {
+	const row = rows[i];
+	if (isLogTable(row.tableName)) return [];
+	if (!inDoubt.has(row.id)) return null;
+	const changed = /* @__PURE__ */ new Set();
+	for (const later of rows.slice(i + 1)) {
+		if (later.recordId !== row.recordId || !inDoubt.has(later.id)) continue;
+		if (later.mutationType !== "update") return [];
+		for (const key of Object.keys(later.data ?? {})) changed.add(key);
+	}
+	return Object.keys(row.data ?? {}).filter((key) => !changed.has(key));
+}
+function remoteStatement(rows, i, vars, inDoubt) {
+	const row = rows[i];
+	vars[`id${i}`] = parseRecordIdString(row.recordId);
+	switch (row.mutationType) {
+		case "create": {
+			const data = row.data ?? {};
+			const create = `CREATE ONLY $id${i} SET ${Object.keys(data).map((key) => {
+				vars[`d${i}_${key}`] = data[key];
+				return `${key} = $d${i}_${key}`;
+			}).join(", ")}`;
+			const match = matchFields(rows, i, inDoubt);
+			if (!match) return create;
+			return `LET $was${i} = (SELECT * FROM ONLY $id${i}); IF ${[`($was${i} = NONE)`, ...match.map((key) => `($was${i}.${key} != $d${i}_${key})`)].join(" OR ")} { ${create} }`;
+		}
+		case "update":
+			vars[`data${i}`] = row.data ?? {};
+			return `UPDATE $id${i} MERGE $data${i}`;
+		case "delete": return `DELETE $id${i}`;
+	}
+}
+/**
+* Many outbox rows as ONE multi-statement request. Statement i answers row i
+* and is its own transaction on the server, as before, but the statements are
+* chained: statement i binds `$ok<i>` only when it commits, and statement i+1
+* throws `BATCH_SKIPPED` without touching anything unless `$ok<i>` is bound
+* (a LET lives for one request only). So the server stops at the first
+* failure: everything before it is committed, nothing after it ran.
+*
+* Without the chain, statements after a failed one ran anyway. A retryable
+* failure at i left the rows after i queued although they had committed; the
+* re-sent CREATE then read "already exists", was taken for a rejection and
+* rolled back locally while the server kept the row. Acking them instead
+* would have let a later write land before an earlier one it may depend on.
+*
+* The chain cannot help when the answer itself is lost: the request may have
+* committed any prefix of the batch. Rows named in `inDoubt` (mutation ids)
+* are such rows. An UPDATE MERGE or a DELETE is safe to run twice, a CREATE
+* is not, so an in-doubt CREATE checks the row already there first (see
+* `matchFields`) and is a no-op when it is its own: the batch goes on in
+* order, with no "already exists" to cut it.
 */
-function remoteBatch(rows) {
+function remoteBatch(rows, inDoubt = /* @__PURE__ */ new Set()) {
 	const vars = {};
 	return {
-		sql: rows.map((row, i) => {
-			const rid = parseRecordIdString(row.recordId);
-			vars[`id${i}`] = rid;
-			switch (row.mutationType) {
-				case "create": {
-					const data = row.data ?? {};
-					return `CREATE ONLY $id${i} SET ${Object.keys(data).map((key) => {
-						vars[`d${i}_${key}`] = data[key];
-						return `${key} = $d${i}_${key}`;
-					}).join(", ")}`;
-				}
-				case "update":
-					vars[`data${i}`] = row.data ?? {};
-					return `UPDATE $id${i} MERGE $data${i}`;
-				case "delete": return `DELETE $id${i}`;
-			}
+		sql: rows.map((_, i) => {
+			return `LET $ok${i} = { ${i === 0 ? "" : `IF !$ok${i - 1} { THROW "${BATCH_SKIPPED}" }; `}${remoteStatement(rows, i, vars, inDoubt)}; true }`;
 		}).join(";\n"),
 		vars
 	};
@@ -2198,8 +2286,8 @@ function trimHistory(history, cap) {
 
 //#endregion
 //#region src/modules/devtools/index.ts
-const CORE_VERSION = "0.0.1-canary.334";
-const WASM_VERSION = "0.0.1-canary.334";
+const CORE_VERSION = "0.0.1-canary.336";
+const WASM_VERSION = "0.0.1-canary.336";
 const SURREAL_VERSION = "3.1.0";
 var DevToolsService = class DevToolsService {
 	version = CORE_VERSION;
@@ -3458,6 +3546,18 @@ const outboxBumpAttempts = (id) => (s) => {
 		outbox
 	};
 };
+/** Mark items whose last push may have committed although no answer came back. */
+const outboxMarkInDoubt = (ids) => (s) => {
+	const marked = new Set(ids);
+	if (!s.outbox.some((i) => marked.has(i.id) && !i.inDoubt)) return s;
+	return {
+		...s,
+		outbox: s.outbox.map((i) => marked.has(i.id) && !i.inDoubt ? {
+			...i,
+			inDoubt: true
+		} : i)
+	};
+};
 /** Drop acked items older than `graceMs` (membership never named them). */
 const outboxPruneAcked = (now, graceMs) => (s) => {
 	const stale = s.outbox.filter((i) => i.status === "acked" && i.ackedAt !== null && now - i.ackedAt >= graceMs);
@@ -5809,6 +5909,8 @@ const ERR = (r) => r && r.status === "ERR" ? r.error : null;
 * A CRDT edit-log row (`<table>__crdt_log`) is written once and never changes,
 * so a CREATE retried after its first attempt committed (and only the answer
 * was lost) finds it already there: that is a success, not a rejection.
+* `remoteBatch` already makes such a CREATE a no-op on the server; this covers
+* a row the user may create but not read back, where that check cannot see it.
 */
 const isRepeatedLogCreate = (row, err) => row.mutationType === "create" && row.tableName.endsWith("__crdt_log") && /already exists/i.test(err);
 /**
@@ -5816,6 +5918,17 @@ const isRepeatedLogCreate = (row, err) => row.mutationType === "create" && row.t
 * per request, per-statement outcome. Accepted rows are acked (they stay in
 * the overlay until membership names them), rejected rows are rolled back,
 * a transport failure leaves the unsent tail queued behind a backoff timer.
+* The server stops a batch at its first failed statement (`remoteBatch`), so
+* the rows after a rejection are sent again at once, and a re-sent row behind
+* a retryable failure never ran before, so it cannot read "already exists".
+*
+* A request that throws a transport error (the push timeout, a dropped
+* socket) may still have committed on the server, and so may one sent by a
+* session that died before its answer. Those rows are marked in doubt
+* (`OutboxItem.inDoubt`, every row loaded from the store is), and their
+* re-sent CREATE accepts the row already on the server when it holds what the
+* CREATE would write, instead of failing "already exists" and rolling back a
+* write the server kept.
 */
 function* drain(env) {
 	for (;;) {
@@ -5837,7 +5950,7 @@ function* drain(env) {
 			else yield fx.state.update(outboxRemove(item.id));
 		}
 		if (rows.length === 0) continue;
-		const req = remoteBatch(rows);
+		const req = remoteBatch(rows, new Set(batch.filter((i) => i.inDoubt).map((i) => i.id)));
 		let results;
 		try {
 			const raw = yield fx.remote.query(req.sql, req.vars, env.pushTimeoutMs);
@@ -5856,7 +5969,7 @@ function* drain(env) {
 				continue;
 			}
 			const attempts = batch.find((i) => i.id === rows[0].id).attempts;
-			yield fx.state.update(outboxBumpAttempts(rows[0].id));
+			yield fx.state.update(compose(outboxMarkInDoubt(rows.map((row) => row.id)), outboxBumpAttempts(rows[0].id)));
 			yield fx.dispatch({
 				type: "SyncOutcome",
 				ok: false,
@@ -5871,6 +5984,7 @@ function* drain(env) {
 			const row = rows[i];
 			const r = results[i];
 			if (!r) {
+				yield fx.state.update(outboxMarkInDoubt(rows.slice(i).map((unanswered) => unanswered.id)));
 				cutAt = i;
 				break;
 			}
@@ -5908,6 +6022,7 @@ function* drain(env) {
 				});
 				continue;
 			}
+			if (err.includes(BATCH_SKIPPED)) break;
 			if (classifySyncError(new Error(err)) === "network") {
 				cutAt = i;
 				break;
@@ -15774,7 +15889,7 @@ function createTabsCoordinator(config, s, host) {
 	return new TabsCoordinator({
 		tabId: s.tabId,
 		fingerprint: computeTabsFingerprint({
-			coreVersion: "0.0.1-canary.334",
+			coreVersion: "0.0.1-canary.336",
 			schemaHash: hash53(config.schemaSurql),
 			endpoint: config.database.endpoint ?? "",
 			namespace: config.database.namespace,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@spooky-sync/core",
-  "version": "0.0.1-canary.334",
+  "version": "0.0.1-canary.336",
   "type": "module",
   "sideEffects": false,
   "main": "./dist/index.js",
@@ -75,8 +75,8 @@
     }
   },
   "dependencies": {
-    "@spooky-sync/query-builder": "0.0.1-canary.334",
-    "@spooky-sync/ssp-wasm": "0.0.1-canary.334",
+    "@spooky-sync/query-builder": "0.0.1-canary.336",
+    "@spooky-sync/ssp-wasm": "0.0.1-canary.336",
     "@sqlite.org/sqlite-wasm": "3.53.0-build1",
     "@surrealdb/wasm": "^3.1.0",
     "@surrealdb/wasm-native": "^3.3.4",

package/skills/sp00ky-core/SKILL.md

@@ -129,7 +129,12 @@ Use `debounced` option for frequent updates (e.g., live typing):
 
 A push the server rejects (permission, validation) is rolled back locally
 (create, update and delete) and moved to the `_00_failed_mutations` tray in the
-same local transaction. Network errors never do this; they retry. Inspect and
+same local transaction. Network errors never do this; they retry, and so does a
+write the server refused only because sp00ky's own ingest post to the scheduler
+failed (down, past its timeout, or answering 5xx). A create whose push answer
+was lost (timeout, dropped socket, a tab or app closed mid-push) is not rolled
+back when its retry finds the row it wrote already on the server; a row there
+with other content is still "already exists". Inspect and
 act on the tray with `db.failedMutationCount`, `db.subscribeToFailedMutations(cb)`,
 `db.listFailedMutations()`, `db.retryFailedMutation(id)` (a fresh optimistic
 write) and `db.discardFailedMutation(id)`.