@bastani/atomic-storage-libsql 0.9.89-alpha.1 → 0.9.89-alpha.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/CHANGELOG.md +7 -0
  2. package/libsql.js +157 -30
  3. package/package.json +1 -1
package/CHANGELOG.md CHANGED
@@ -2,6 +2,13 @@
2
2
 
3
3
  ## [Unreleased]
4
4
 
5
+ ## [0.9.89-alpha.2] - 2026-11-18
6
+
7
+ ### Fixed
8
+
9
+ - On a libSQL server (sqld) shared by stores that create and drop their tables, such as several sessions, workflows or teams using one server, a statement no longer fails with `SQLITE_SCHEMA: database schema has changed` when other connections change the schema while the server prepares it. The server ran nothing of such a statement, so it is sent again, as a statement the server answers busy is, for up to 5 seconds. A write outside a transaction (`INSERT`, `UPDATE`, `DELETE`, `REPLACE`, or a `WITH` that writes), or a schema change (`CREATE`, `DROP`, `ALTER`) sent alone in one `exec`, also no longer fails with `SQLITE_SCHEMA` while other connections keep changing the schema for longer than the 5-second wait. Sent again as it was, such a write could meet a changed schema on every try, and on a loaded server only two or three tries fit in the wait. After the first `SQLITE_SCHEMA` answer, such a statement is now sent again inside a write transaction of its own, in one request, with a wait of its own. While the transaction holds the write lock no other connection can change the schema, so the statement runs, and it commits once or not at all. Reads (including `VALUES` and a `WITH` that only selects) and other statements, such as pragmas, are still sent again as they were. A statement inside a transaction, and several statements sent in one `exec`, are not sent again
10
+ - On a libSQL server (sqld) shared by stores that create and drop their tables, a read or write outside a transaction no longer fails now and then with `SQLITE_UNKNOWN: no such table` for a table that exists. sqld gives that answer, without running the statement, when it compiles the statement while other connections change the schema; it happened to a `DELETE` from a table whose rows a trigger also removes from a full-text index, on kept and new streams alike. When the server names a table the database's schema lists, the statement now runs again with growing waits, up to 20 times; a table that does not exist is still reported at once ([#5464](https://github.com/bastani-inc/atomic/issues/5464))
11
+
5
12
  ## [0.9.89-alpha.1] - 2026-11-18
6
13
 
7
14
  ### Fixed
package/libsql.js CHANGED
@@ -409,6 +409,17 @@ const STREAM_CLOSE_WAIT_MS = 1_000;
409
409
  */
410
410
  const SCHEMA_CHECK = "SELECT 1 FROM sqlite_schema LIMIT 0";
411
411
 
412
+ /** The table a server's `no such table` answer names, at the end of its message. */
413
+ const MISSING_TABLE = /no such table: (\S+)$/;
414
+ /** A read of the schema itself, not of the connection's copy of it, for a table or view of the main database. */
415
+ const TABLE_LOOKUP = "SELECT 1 FROM sqlite_schema WHERE type IN ('table', 'view') AND name = ? COLLATE NOCASE";
416
+ /**
417
+ * How many times a statement the server answered `no such table` for a table that exists runs again. Each answer is
418
+ * an independent loss of a race with other connections' schema changes, so the bound counts runs, not time: a server
419
+ * slowed by load answers each run late, yet each run loses as rarely.
420
+ */
421
+ const MISSING_TABLE_RERUNS = 20;
422
+
412
423
  /**
413
424
  * What sqld answers, as plain text with HTTP 400, for a request whose baton it refuses when it looks the stream up,
414
425
  * before it runs any request of the pipeline (`stream::acquire` and `Guard::get_db` in sqld's
@@ -489,7 +500,10 @@ function hranaCode(hrana, error) {
489
500
  * tables; a stream kept open loads it once. A stream that failed, sat idle `STREAM_REUSE_IDLE_MS` or lived
490
501
  * `STREAM_REUSE_MAX_AGE_MS` is closed and a new one opened. A statement is sent again only when the server refused its
491
502
  * request without running it: it found the stream expired, or refused its baton (`REFUSED_BATON`). A request whose
492
- * answer was lost may have run, so its failure is the caller's, as it is for a statement on a stream of its own.
503
+ * answer was lost may have run, so its failure is the caller's, as it is for a statement on a stream of its own. A
504
+ * statement the server answered `no such table` for a table the schema holds runs again, with growing waits, up to
505
+ * `MISSING_TABLE_RERUNS` times (#5464): the server compiled it while other connections changed the schema and ran none
506
+ * of it.
493
507
  */
494
508
  class ReusedStream {
495
509
  /** @type {import("@libsql/hrana-client").Stream | undefined} */
@@ -520,7 +534,9 @@ class ReusedStream {
520
534
  if (this.busy) return undefined;
521
535
  this.busy = true;
522
536
  try {
523
- for (let resent = false; ; resent = true) {
537
+ let resent = false;
538
+ let reruns = 0;
539
+ for (let wait = SERVER_BUSY_FIRST_WAIT_MS; ; ) {
524
540
  const now = performance.now();
525
541
  const kept = this.stream;
526
542
  if (
@@ -549,7 +565,15 @@ class ReusedStream {
549
565
  };
550
566
  } catch (error) {
551
567
  if (stream.closed && this.stream === stream) this.stream = undefined;
552
- if (fresh || resent || !this.ranNothing(error)) throw this.libsqlError(error);
568
+ if (!fresh && !resent && this.ranNothing(error)) {
569
+ resent = true;
570
+ continue;
571
+ }
572
+ if (reruns >= MISSING_TABLE_RERUNS || !(await this.namesExistingTable(stream, error)))
573
+ throw this.libsqlError(error);
574
+ reruns++;
575
+ await new Promise((resolve) => setTimeout(resolve, wait));
576
+ wait = Math.min(wait * 2, SERVER_BUSY_LAST_WAIT_MS);
553
577
  } finally {
554
578
  this.usedAt = performance.now();
555
579
  }
@@ -574,6 +598,26 @@ class ReusedStream {
574
598
  );
575
599
  }
576
600
 
601
+ /**
602
+ * Whether `error` is the server's answer `no such table` for a table that `stream` finds in the schema. sqld gives
603
+ * that answer, without running the statement, when it prepares the statement while other connections change the
604
+ * schema; the same statement sent again finds the table (#5464). A table that does not exist is reported.
605
+ * @param {import("@libsql/hrana-client").Stream} stream
606
+ * @param {unknown} error
607
+ */
608
+ async namesExistingTable(stream, error) {
609
+ if (!(error instanceof this.hrana.ResponseError) || stream.closed) return false;
610
+ const named = MISSING_TABLE.exec(error.message)?.[1];
611
+ if (named === undefined) return false;
612
+ try {
613
+ const lookup = new this.hrana.Stmt(TABLE_LOOKUP);
614
+ lookup.bindIndexes([named.replace(/^main\./i, "")]);
615
+ return (await stream.query(lookup)).rows.length > 0;
616
+ } catch {
617
+ return false;
618
+ }
619
+ }
620
+
577
621
  /**
578
622
  * `error` as `@libsql/client` reports it.
579
623
  * @param {unknown} error
@@ -689,24 +733,65 @@ async function writeBatch(hrana, api, client, batched) {
689
733
  /**
690
734
  * @param {() => Promise<import("@libsql/client").Client | import("@libsql/client").Transaction>} connection
691
735
  * @param {<T>(operation: () => Promise<T>) => Promise<T>} watch
692
- * @param {<T>(operation: () => Promise<T>) => Promise<T>} [single] runs one statement, which a failure may run again
736
+ * @param {<T>(operation: () => Promise<T>, afterSchemaChange?: () => Promise<T>) => Promise<T>} [single] runs one
737
+ * statement, which a failure may run again; after `SQLITE_SCHEMA` it may run `afterSchemaChange` in its place
693
738
  * @param {() => Promise<ReusedStream | undefined>} [shared] the stream statements outside a transaction reuse
739
+ * @param {(batched: readonly import("@bastani/atomic/storage").SqliteStatement[]) => Promise<import("@bastani/atomic/storage").SqliteDriverResult[]>} [locked]
740
+ * runs statements in one write transaction, where no other connection can change the schema
694
741
  * @returns {import("@bastani/atomic/storage").SqliteDriverExecutor}
695
742
  */
696
- function statements(connection, watch, single = watch, shared = async () => undefined) {
743
+ function statements(connection, watch, single = watch, shared = async () => undefined, locked) {
697
744
  return {
698
745
  execute: (sql, params) =>
699
- single(async () => {
700
- const reused = reusable(sql) ? await (await shared())?.execute(sql, params) : undefined;
701
- return reused ?? driverResult(await (await connection()).execute({ sql, args: [...params] }));
702
- }),
746
+ single(
747
+ async () => {
748
+ const reused = reusable(sql) ? await (await shared())?.execute(sql, params) : undefined;
749
+ return reused ?? driverResult(await (await connection()).execute({ sql, args: [...params] }));
750
+ },
751
+ // A write that commits on its own and that the server answered `SQLITE_SCHEMA` runs again in a write
752
+ // transaction of its own: no other connection changes the schema while the transaction holds the write lock,
753
+ // so sent as it was it could keep meeting a changed schema for as long as other connections keep changing it.
754
+ locked !== undefined && rerunsLocked(sql) ? async () => (await locked([{ sql, params }]))[0] : undefined,
755
+ ),
703
756
  // An `exec` of one statement wrote nothing when it fails, as one `execute` did; one of several statements may
704
757
  // fail after the statements before it committed.
705
758
  executeMultiple: (sql) =>
706
- (isOneStatement(sql) ? single : watch)(async () => (await connection()).executeMultiple(sql)),
759
+ isOneStatement(sql)
760
+ ? single(
761
+ async () => (await connection()).executeMultiple(sql),
762
+ locked !== undefined && rerunsLocked(sql)
763
+ ? async () => {
764
+ await locked([{ sql, params: [] }]);
765
+ }
766
+ : undefined,
767
+ )
768
+ : watch(async () => (await connection()).executeMultiple(sql)),
707
769
  };
708
770
  }
709
771
 
772
+ /** The keyword of a statement that writes and runs the same inside a write transaction as on its own. */
773
+ const LOCKED_RERUN_KEYWORD = /(?:INSERT|UPDATE|DELETE|REPLACE|CREATE|DROP|ALTER)\b/iy;
774
+ /** The keyword of a statement that starts with common table expressions, which may read or write. */
775
+ const COMMON_TABLE_KEYWORD = /WITH\b/iy;
776
+ /** A write inside a statement that starts with common table expressions. */
777
+ const COMMON_TABLE_WRITE = /\b(?:INSERT|UPDATE|DELETE|REPLACE)\b/i;
778
+
779
+ /**
780
+ * Whether `sql`, one statement the server answered `SQLITE_SCHEMA`, runs again in a write transaction of its own: a
781
+ * write (also one that starts with common table expressions) or a schema change. Not a read, such as `SELECT`,
782
+ * `VALUES` or a `WITH` that only selects, and not a statement that runs differently in a transaction or not at all, as
783
+ * `PRAGMA foreign_keys` and `VACUUM` do.
784
+ * @param {string} sql
785
+ */
786
+ function rerunsLocked(sql) {
787
+ const start = firstTokenStart(sql);
788
+ if (start === -1 || readsOnly(sql) || !isOneStatement(sql)) return false;
789
+ LOCKED_RERUN_KEYWORD.lastIndex = start;
790
+ if (LOCKED_RERUN_KEYWORD.test(sql)) return true;
791
+ COMMON_TABLE_KEYWORD.lastIndex = start;
792
+ return COMMON_TABLE_KEYWORD.test(sql) && COMMON_TABLE_WRITE.test(sql.slice(start));
793
+ }
794
+
710
795
  /**
711
796
  * Whether `sql` is one statement: it has no `;` but at its end. A `;` in a literal or a comment also counts, so
712
797
  * several statements are never taken for one.
@@ -733,9 +818,10 @@ function driverResult(result) {
733
818
  const FILE_BUSY_TIMEOUT_MS = 5000;
734
819
 
735
820
  /**
736
- * How long a call to a libSQL server runs again while the server answers it `SQLITE_BUSY`. sqld answers a writer that
737
- * meets its checkpoint `SQLITE_BUSY` at once, without waiting and without running any of the call, and leaves the
738
- * waiting to the client; its own checkpoint waits this long for the lock.
821
+ * How long a call to a libSQL server runs again while the server answers it `SQLITE_BUSY` or `SQLITE_SCHEMA`. sqld
822
+ * answers a writer that meets its checkpoint `SQLITE_BUSY` at once, without waiting and without running any of the
823
+ * call, and leaves the waiting to the client; its own checkpoint waits this long for the lock. It answers
824
+ * `SQLITE_SCHEMA`, also without running the call, while other connections keep changing the schema.
739
825
  */
740
826
  const SERVER_BUSY_TIMEOUT_MS = 5000;
741
827
  /** The first wait before a call a libSQL server answered busy runs again; each later wait doubles, up to the last. */
@@ -744,22 +830,28 @@ const SERVER_BUSY_LAST_WAIT_MS = 200;
744
830
 
745
831
  /**
746
832
  * Runs `operation` again, with growing waits, while it fails with `SQLITE_BUSY` and `ranNothing` says the failed run
747
- * ran nothing, until `SERVER_BUSY_TIMEOUT_MS` has passed; then it fails with the last run's failure. `prepare` runs
748
- * before each run after the first.
833
+ * ran nothing, or with `SQLITE_SCHEMA`, until `SERVER_BUSY_TIMEOUT_MS` has passed; then it fails with the last run's
834
+ * failure. `prepare` runs before each run after the first.
835
+ *
836
+ * A server answers `SQLITE_SCHEMA` when other connections kept changing the database's schema while it prepared the
837
+ * statement, as on a server shared by several stores that create and drop their tables: the statement never ran, so
838
+ * a statement or batch that commits on its own runs again. `rerunsAfterSchemaChange` says whether the run may.
749
839
  * @template T
750
840
  * @param {() => Promise<T>} operation
751
841
  * @param {() => boolean} ranNothing
752
842
  * @param {() => Promise<void>} [prepare]
843
+ * @param {() => boolean} [rerunsAfterSchemaChange] whether a run that `SQLITE_SCHEMA` ended may run again
753
844
  * @returns {Promise<T>}
754
845
  */
755
- async function whileServerBusy(operation, ranNothing, prepare) {
846
+ async function whileServerBusy(operation, ranNothing, prepare, rerunsAfterSchemaChange = () => true) {
756
847
  const deadline = performance.now() + SERVER_BUSY_TIMEOUT_MS;
757
848
  for (let wait = SERVER_BUSY_FIRST_WAIT_MS; ; wait = Math.min(wait * 2, SERVER_BUSY_LAST_WAIT_MS)) {
758
849
  try {
759
850
  return await operation();
760
851
  } catch (error) {
761
852
  const left = deadline - performance.now();
762
- if (!isBusy(error) || !ranNothing() || left <= 0) throw error;
853
+ const rerun = isBusy(error) ? ranNothing() : isSchemaChanged(error) && rerunsAfterSchemaChange();
854
+ if (!rerun || left <= 0) throw error;
763
855
  await new Promise((resolve) => setTimeout(resolve, Math.min(wait, left)));
764
856
  await prepare?.();
765
857
  }
@@ -774,6 +866,14 @@ function isBusy(error) {
774
866
  return /** @type {{ readonly code?: unknown } | undefined} */ (error)?.code === "SQLITE_BUSY";
775
867
  }
776
868
 
869
+ /**
870
+ * Whether a server refused a statement because the schema it was prepared against kept changing.
871
+ * @param {unknown} error
872
+ */
873
+ function isSchemaChanged(error) {
874
+ return /** @type {{ readonly code?: unknown } | undefined} */ (error)?.code === "SQLITE_SCHEMA";
875
+ }
876
+
777
877
  /** V8's `gc`, taken from a context created while `--expose-gc` is set; the flag is off again for code that runs later. */
778
878
  function exposeGc() {
779
879
  setFlagsFromString("--expose-gc");
@@ -1174,10 +1274,29 @@ export async function openLibsqlDatabase({ url, authToken, namespace, headers })
1174
1274
  const version = (await probe.execute("PRAGMA data_version")).rows[0]?.data_version;
1175
1275
  return typeof version === "bigint" || typeof version === "number" ? version : undefined;
1176
1276
  };
1177
- /** @type {<T>(operation: () => Promise<T>) => Promise<T>} */
1178
- const waitForLock = async (operation) => {
1179
- // A server answers busy before it runs anything of one statement or a batch.
1180
- if (!local) return whileServerBusy(() => watch(operation), () => true);
1277
+ /** @type {<T>(operation: () => Promise<T>, afterSchemaChange?: () => Promise<T>) => Promise<T>} */
1278
+ const waitForLock = async (operation, afterSchemaChange) => {
1279
+ // A server answers busy before it runs anything of one statement or a batch, and `SQLITE_SCHEMA` before it runs
1280
+ // anything of one statement. After `SQLITE_SCHEMA`, `afterSchemaChange`, when given, runs in its place, with a
1281
+ // wait of its own: the run the server answered `SQLITE_SCHEMA` may itself have taken the whole wait.
1282
+ if (!local) {
1283
+ if (afterSchemaChange === undefined) return whileServerBusy(() => watch(operation), () => true);
1284
+ let schemaChanged = false;
1285
+ try {
1286
+ return await whileServerBusy(
1287
+ () => watch(operation),
1288
+ () => true,
1289
+ undefined,
1290
+ () => {
1291
+ schemaChanged = true;
1292
+ return false;
1293
+ },
1294
+ );
1295
+ } catch (error) {
1296
+ if (!schemaChanged) throw error;
1297
+ }
1298
+ return whileServerBusy(() => watch(afterSchemaChange), () => true);
1299
+ }
1181
1300
  /** @type {bigint | number | undefined} */
1182
1301
  let version;
1183
1302
  for (;;) {
@@ -1203,6 +1322,18 @@ export async function openLibsqlDatabase({ url, authToken, namespace, headers })
1203
1322
  }
1204
1323
  return (client ??= create());
1205
1324
  };
1325
+ /**
1326
+ * Runs `batched` in one write transaction, in one request to a server.
1327
+ * @param {readonly import("@bastani/atomic/storage").SqliteStatement[]} batched
1328
+ */
1329
+ const writeTransaction = async (batched) => {
1330
+ if (protocol !== undefined) return writeBatch(protocol[0], protocol[3], remote(), batched);
1331
+ const results = await (await connection()).batch(
1332
+ batched.map(({ sql, params }) => ({ sql, args: [...params] })),
1333
+ "write",
1334
+ );
1335
+ return results.map(driverResult);
1336
+ };
1206
1337
  const redact = redactSecrets([authToken ?? "", ...urlCredentials(url), ...Object.values(headers ?? {})]);
1207
1338
  /** The server's location as diagnostics show it, without credentials. */
1208
1339
  const redacted = redact(url);
@@ -1216,19 +1347,12 @@ export async function openLibsqlDatabase({ url, authToken, namespace, headers })
1216
1347
  // One statement outside a transaction, run alone or as an `exec` of one statement, that waited out the busy
1217
1348
  // timeout wrote nothing and runs again. Several statements in one `exec` do not: the ones before the failed
1218
1349
  // statement were committed.
1219
- ...statements(connection, watch, waitForLock, shared),
1350
+ ...statements(connection, watch, waitForLock, shared, local ? undefined : writeTransaction),
1220
1351
  // One request for the whole batch, in a write transaction: the journal sends a batch of appends this way. Only
1221
1352
  // its `BEGIN IMMEDIATE` waits for the write lock, so a batch the busy timeout ended wrote nothing (`lockBusy`). A
1222
1353
  // server's batch counts against this process's writes in flight to that server, unless it has priority.
1223
1354
  batch: (batched, options) => {
1224
- const send = async () => {
1225
- if (protocol !== undefined) return writeBatch(protocol[0], protocol[3], remote(), batched);
1226
- const results = await (await connection()).batch(
1227
- batched.map(({ sql, params }) => ({ sql, args: [...params] })),
1228
- "write",
1229
- );
1230
- return results.map(driverResult);
1231
- };
1355
+ const send = () => writeTransaction(batched);
1232
1356
  if (local) return waitForLock(send);
1233
1357
  const sent = () =>
1234
1358
  watch(() =>
@@ -1258,6 +1382,8 @@ export async function openLibsqlDatabase({ url, authToken, namespace, headers })
1258
1382
  // again in a new transaction. A busy answer after the transaction began is not run again: a busy statement
1259
1383
  // leaves the transaction open, and a busy answer that ends its stream also ends the statements it already ran.
1260
1384
  // The transaction began once a statement in it settled and left it open: the server acknowledged its BEGIN.
1385
+ // A statement in a transaction that the server answered `SQLITE_SCHEMA` is not run again here: the server may
1386
+ // have ended the transaction, and the statement would then run, and commit, on its own.
1261
1387
  let started = false;
1262
1388
  /** @type {<T>(operation: () => Promise<T>) => Promise<T>} */
1263
1389
  const inTransaction = async (operation) => {
@@ -1268,6 +1394,7 @@ export async function openLibsqlDatabase({ url, authToken, namespace, headers })
1268
1394
  async () => {
1269
1395
  transaction = await watch(begin);
1270
1396
  },
1397
+ () => false,
1271
1398
  );
1272
1399
  } finally {
1273
1400
  if (!transaction.closed) started = true;
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@bastani/atomic-storage-libsql",
3
- "version": "0.9.89-alpha.1",
3
+ "version": "0.9.89-alpha.2",
4
4
  "description": "libSQL / Turso storage adapter for Atomic sessions, workflow journals, and Intercom.",
5
5
  "contributors": [
6
6
  "Norin Lavaee",