@bastani/atomic-storage-libsql 0.9.87-alpha.1 → 0.9.88-alpha.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,23 @@
2
2
 
3
3
  ## [Unreleased]
4
4
 
5
+ ## [0.9.88-alpha.1] - 2026-11-16
6
+
7
+ ### Changed
8
+
9
+ - Statements outside a transaction on a libSQL server (sqld) or Turso database now reuse one Hrana stream instead of opening a new one for every request ([#5151](https://github.com/bastani-inc/atomic/issues/5151)). sqld loads the database's schema for every stream it opens, so the server's work for each request grew with the number of tables: on one sqld, a pooled workflow round used 6.0 s of server CPU with no other tables, 8.3 s with 100 and 13.1 s with 400; it now uses 2.9, 3.2 and 3.8 s. Only reads and writes (`SELECT`, `INSERT`, `UPDATE`, `DELETE`, `REPLACE`, `WITH`, `VALUES`) reuse the stream; schema changes, pragmas, transaction control, transactions and write batches keep a stream of their own, so a reused stream never holds a transaction or the write lock. A stream idle 5 seconds or open 60 seconds is closed and a new one opened, before sqld would expire it. A statement is sent again only when the server refused it without running it, because the stream had expired or its baton was refused before anything ran; a statement whose answer was lost, or any other refusal, is not, and the next statement opens a new stream. Each statement on a kept stream is preceded, in the same request, by a read of `sqlite_schema`, so the stream's connection rereads a schema another process changed: without it, a kept connection compiled the next statement against the old schema, and `SELECT *` returned the old columns. A stream's first request that fails no longer sends the server a close for a stream it never opened.
10
+
11
+ ### Fixed
12
+
13
+ - A libSQL server (sqld) or Turso database no longer stays locked for the stream's whole lifetime after an outage that cut a stream holding a write transaction. The client sends the close of a stream it gives up on once, and in an outage that close failed too, so the server kept the stream, its open transaction and the database's write lock: every writer, in this process and every other, waited for it, and a worker host could stay stuck for over a minute after the database came back. The adapter now sends such a close again, waiting 250 ms and then twice as long each time up to 8 s, until the server answers it, for at most 5 minutes, and stops when the database closes; a close that reaches the server frees the lock at once. When the outage lost the answer to a request the server had already run, the server refuses the close as a used baton, the adapter does not send it again, and the lock clears when the server expires the stream, about 5 seconds on sqld ([#5114](https://github.com/bastani-inc/atomic/issues/5114)).
14
+ - An HTTP 5xx answer with no libSQL error in it, such as a gateway's 502 or 504 in front of sqld or Turso, is now a `StorageTransientError` with code `connection-lost`. Its outcome is `"unknown"` for a commit, a batch or a statement outside a transaction, because the server may have applied the request before the answer was lost. Before, it was a plain `StorageUnavailableError`, and a session save the server had applied counted as failed ([#4970](https://github.com/bastani-inc/atomic/issues/4970)).
15
+ - A value over a libSQL server's limit (`SQLITE_TOOBIG`, or `ARGS_INVALID: string or blob too big`; 5,000,000 bytes by default in sqld) and a response over its 10 MB limit (`RESPONSE_TOO_LARGE`) are no longer reported as `StorageUnavailableError`. They fail with `StoragePayloadTooLargeError`, which names the size and sqld's default limit and is never retried. Atomic's sessions and snapshots now store large values in rows of at most 1 MiB, so they stay under these limits ([#4954](https://github.com/bastani-inc/atomic/issues/4954))
16
+ - A libSQL server (sqld) database no longer fails a write with `SQLITE_BUSY: database is locked` when the write meets the server's checkpoint. sqld answers such a write `SQLITE_BUSY` at once, without running any of it, and leaves the waiting to the client, so a write that coincided with a checkpoint failed with `StorageUnavailableError`. A statement outside a transaction, an `exec` of one statement, a batch, or the start of a transaction that the server answers busy now runs again with growing waits for up to 5 seconds. A statement the server answers busy after its transaction began, and several statements in one `exec`, do not run again ([#4817](https://github.com/bastani-inc/atomic/issues/4817)).
17
+ - Closing a `file:` database no longer crashes the process on Windows. `libsql` 0.5.29, the native driver `@libsql/client` uses, closes a local connection's SQLite handle twice when the connection holds its last reference ([tursodatabase/libsql#2251](https://github.com/tursodatabase/libsql/issues/2251)). The second close reads freed memory, and Windows ends the process with `0xC0000005` and no message. A process with several connections open took that path on the close after its first, because each close's garbage collections finalized the other connections' statements. The adapter now closes each local connection while a statement still holds it, so the handle closes once. A `libsql` release built with the crate fix ([tursodatabase/libsql#2261](https://github.com/tursodatabase/libsql/pull/2261)) makes this unnecessary ([#4824](https://github.com/bastani-inc/atomic/issues/4824)).
18
+ - Fixed many processes writing to one sqld or Turso database stalling each other ([#4768](https://github.com/bastani-inc/atomic/issues/4768)). A write batch that finds the server's write lock busy is now sent again after a short random wait that grows up to a second, for as long as the caller's `lockWaitMs` allows. Each process sends at most four write batches to a server at once; a batch with `priority` skips that limit. A request the server leaves unanswered for 30 seconds is abandoned, as a transient failure whose outcome is unknown, instead of waiting for the network's own timeout. Three write requests in a row left unanswered print a warning (`ATOMIC_LIBSQL_WRITES_STALLED`) naming [tursodatabase/libsql#2286](https://github.com/tursodatabase/libsql/issues/2286), a sqld bug that stops it taking writes until it restarts, and its workaround: start sqld with `--checkpoint-interval-s 60`. The first write answered afterwards prints `ATOMIC_LIBSQL_WRITES_RECOVERED`
19
+ - A libSQL server that stops answering writes no longer holds back writes to another server from the same process. The limit of four write batches in flight now applies to each server separately, so writes to a healthy server, and the calls queued behind them on that database, go on while another server is stalled ([#4790](https://github.com/bastani-inc/atomic/issues/4790))
20
+ - A statement whose leading comments a regular expression could split many ways, such as `/**//**//**/… PRAGMA …`, no longer stalls the process deciding whether it may reuse the open stream. That check used to backtrack exponentially in the number of comments; it now reads the statement once, and it ends a block comment at its first `*/`, as SQLite does.
21
+
5
22
  ## [0.9.75] - 2026-11-10
6
23
 
7
24
  ### Fixed
package/README.md CHANGED
@@ -37,6 +37,8 @@ chmod 600 ~/.atomic/turso-token
37
37
 
38
38
  A `file:` database is opened in write-ahead-log mode with a 5000 ms busy timeout, the way Atomic's own SQLite stores are, so several processes can read and write one file at once: a statement waits up to 5 seconds for a lock another process holds instead of failing with `SQLITE_BUSY`, and the wait blocks the calling thread. A statement outside a transaction, an `exec` of one statement, a batch, or the start of a transaction that waited out those 5 seconds runs again as long as another connection committed while it waited, so it fails only once a whole wait passes in which nobody committed. Several statements in one `exec` do not run again. The first open changes the file to WAL mode, which is kept in the file and adds `-wal` and `-shm` files beside it. Open a file only through this adapter at a time: another SQLite library with the same file open in WAL mode was seen to read stale data.
39
39
 
40
+ A libSQL server (sqld) answers a write that meets its checkpoint `SQLITE_BUSY` at once, without running it. A statement outside a transaction, an `exec` of one statement, a batch, or the start of a transaction that the server answers busy runs again with growing waits for up to 5 seconds before it fails.
41
+
40
42
  The token is redacted from error messages and diagnostics. Add a `prefix` to the `storage` block to keep projects or teams apart in one database. See [Storage](../coding-agent/docs/storage.md) for selection precedence, per-process overrides, and the other storage options.
41
43
 
42
44
  ## Use from the SDK
package/libsql.js CHANGED
@@ -1,4 +1,5 @@
1
1
  // @ts-check
2
+ import { createRequire } from "node:module";
2
3
  import { setFlagsFromString } from "node:v8";
3
4
  import { runInNewContext } from "node:vm";
4
5
  import { createSqliteDatabase, redactSecrets, StorageAdapterLoadError, switchToWriteAheadLog } from "@bastani/atomic/storage";
@@ -39,15 +40,198 @@ function requestHeaders(url, namespace, headers) {
39
40
  }
40
41
 
41
42
  /**
42
- * `fetch` that adds `headers` to every request the libSQL HTTP client sends.
43
+ * How long one request to a database server may go unanswered before it is abandoned. A server answers a request that
44
+ * waits for its write lock within its own lock wait (sqld's is 5 seconds), so a request still unanswered long after
45
+ * that is stuck in the server: abandoning it lets the caller plan its write again instead of waiting without end.
46
+ */
47
+ const REQUEST_TIMEOUT_MS = 30_000;
48
+
49
+ /**
50
+ * The wait before the first resend of the close of a stream abandoned in an outage (#5114); each later wait doubles, up
51
+ * to `ABANDONED_STREAM_CLOSE_LAST_WAIT_MS`. A stream the server keeps holds its open transaction, and with it the
52
+ * database's write lock, until the server expires it: every writer waits for the lock meanwhile, this process's and
53
+ * every other's. A close the server answers, whatever the answer, is done.
54
+ */
55
+ const ABANDONED_STREAM_CLOSE_FIRST_WAIT_MS = 250;
56
+ /** The longest wait between two resends of abandoned stream closes. */
57
+ const ABANDONED_STREAM_CLOSE_LAST_WAIT_MS = 8_000;
58
+ /**
59
+ * How long one abandoned stream's close is sent again before it is given up. The client does not know the server's
60
+ * stream lifetime (10 s in sqld); a server keeps no stream this long, so the close no longer frees anything.
61
+ */
62
+ const ABANDONED_STREAM_CLOSE_MAX_AGE_MS = 5 * 60_000;
63
+ /** How long one resend of an abandoned stream's close may go unanswered; a later resend tries again. */
64
+ const ABANDONED_STREAM_CLOSE_TIMEOUT_MS = 5_000;
65
+ /** The most abandoned stream closes kept for resending; the oldest is dropped past it, its stream long expired. */
66
+ const MAX_ABANDONED_STREAM_CLOSES = 16;
67
+
68
+ /**
69
+ * Whether `body`, a Hrana pipeline request body, only closes the stream its baton names: the request the HTTP client
70
+ * sends once for a stream it gave up on, `{"baton":…,"requests":[{"type":"close"}]}`. `@libsql/client` speaks Hrana 2
71
+ * over HTTP, whose bodies are JSON; any other body is never taken for a close.
72
+ * @param {Uint8Array} body
73
+ * @param {boolean} [withoutBaton] whether the close names no stream instead, as a stream's first request that failed
74
+ * leaves it
75
+ */
76
+ function closesStream(body, withoutBaton = false) {
77
+ try {
78
+ const { baton, requests } = JSON.parse(new TextDecoder().decode(body));
79
+ return (
80
+ (withoutBaton ? baton === null || baton === undefined : typeof baton === "string") &&
81
+ Array.isArray(requests) &&
82
+ requests.length === 1 &&
83
+ requests[0]?.type === "close"
84
+ );
85
+ } catch {
86
+ return false;
87
+ }
88
+ }
89
+
90
+ /** The most bytes a request body that only closes a stream takes: `{"baton":null,"requests":[{"type":"close"}]}`. */
91
+ const LONE_CLOSE_MAX_BYTES = 64;
92
+
93
+ /**
94
+ * Whether `request` only closes a stream the server never gave a baton for. The Hrana client sends one when a
95
+ * stream's first request fails; the server keeps no stream for it to close, so it opens one only to close it. A stream
96
+ * reused across statements (#5151) has its first request fail in an outage, so that close would reach a server just
97
+ * back for nothing, ahead of the next real request.
98
+ * @param {Request} request an unread copy
99
+ */
100
+ async function closesNoStream(request) {
101
+ const reader = request.body?.getReader();
102
+ if (reader === undefined) return false;
103
+ /** @type {Uint8Array[]} */
104
+ const chunks = [];
105
+ let length = 0;
106
+ try {
107
+ // Reads no further than a lone close's size, so a large request is never read twice.
108
+ for (;;) {
109
+ const { done, value } = await reader.read();
110
+ if (done) break;
111
+ chunks.push(value);
112
+ length += value.byteLength;
113
+ if (length > LONE_CLOSE_MAX_BYTES) return false;
114
+ }
115
+ } finally {
116
+ await reader.cancel().catch(() => {});
117
+ }
118
+ const body = new Uint8Array(length);
119
+ let offset = 0;
120
+ for (const chunk of chunks) {
121
+ body.set(chunk, offset);
122
+ offset += chunk.byteLength;
123
+ }
124
+ return closesStream(body, true);
125
+ }
126
+
127
+ /** The answer a server gives a request that only closes a stream. */
128
+ function closedResponse() {
129
+ return new Response(JSON.stringify({ baton: null, base_url: null, results: [{ type: "ok", response: { type: "close" } }] }), {
130
+ status: 200,
131
+ headers: { "content-type": "application/json" },
132
+ });
133
+ }
134
+
135
+ /**
136
+ * The closes of streams this client abandoned that never reached the server, sent again until the server answers one.
137
+ * The HTTP client sends a stream's close once, when a request of the stream fails; in an outage that close fails too,
138
+ * and the server keeps the stream, with its open transaction and the write lock it holds, for the stream's whole
139
+ * lifetime. Sending the close again as soon as the server is reachable ends the transaction there and frees the lock.
140
+ */
141
+ class AbandonedStreams {
142
+ /** @type {{ url: string, headers: [string, string][], body: ArrayBuffer, since: number }[]} */
143
+ pending = [];
144
+ /** @type {ReturnType<typeof setTimeout> | undefined} */
145
+ timer;
146
+ /** @type {Promise<void> | undefined} */
147
+ sending;
148
+ wait = ABANDONED_STREAM_CLOSE_FIRST_WAIT_MS;
149
+ closed = false;
150
+
151
+ /**
152
+ * Keeps `request`'s close for resending when it only closes a stream.
153
+ * @param {Request} request an unread copy of the request that failed
154
+ */
155
+ async failed(request) {
156
+ const body = await request.arrayBuffer();
157
+ if (this.closed || !closesStream(new Uint8Array(body))) return;
158
+ this.pending.push({ url: request.url, headers: [...request.headers], body, since: Date.now() });
159
+ if (this.pending.length > MAX_ABANDONED_STREAM_CLOSES) this.pending.shift();
160
+ this.schedule();
161
+ }
162
+
163
+ /** Sends the kept closes again after the current wait, unless a resend is already planned. */
164
+ schedule() {
165
+ if (this.timer !== undefined || this.pending.length === 0) return;
166
+ this.timer = setTimeout(() => {
167
+ this.timer = undefined;
168
+ void this.send().then(() => this.schedule());
169
+ }, this.wait);
170
+ // A close waiting to be sent again never keeps the process alive.
171
+ this.timer.unref();
172
+ }
173
+
174
+ /**
175
+ * Sends each kept close again, in order, until one goes unanswered, which doubles the wait before the next round; an
176
+ * answered close is done, whatever it says, and resets the wait. A close older than
177
+ * `ABANDONED_STREAM_CLOSE_MAX_AGE_MS` is given up.
178
+ */
179
+ send() {
180
+ this.sending ??= (async () => {
181
+ const oldest = Date.now() - ABANDONED_STREAM_CLOSE_MAX_AGE_MS;
182
+ this.pending = this.pending.filter((close) => close.since > oldest);
183
+ while (this.pending.length > 0) {
184
+ const [close] = this.pending;
185
+ try {
186
+ const response = await fetch(close.url, {
187
+ method: "POST",
188
+ headers: close.headers,
189
+ body: close.body,
190
+ signal: AbortSignal.timeout(ABANDONED_STREAM_CLOSE_TIMEOUT_MS),
191
+ });
192
+ await response.arrayBuffer().catch(() => {});
193
+ } catch {
194
+ this.wait = Math.min(this.wait * 2, ABANDONED_STREAM_CLOSE_LAST_WAIT_MS);
195
+ return;
196
+ }
197
+ this.wait = ABANDONED_STREAM_CLOSE_FIRST_WAIT_MS;
198
+ this.pending = this.pending.filter((kept) => kept !== close);
199
+ }
200
+ })().finally(() => {
201
+ this.sending = undefined;
202
+ });
203
+ return this.sending;
204
+ }
205
+
206
+ /** Stops resending: the closes kept are given up, and a resend under way sends no further close. */
207
+ close() {
208
+ this.closed = true;
209
+ clearTimeout(this.timer);
210
+ this.timer = undefined;
211
+ this.pending = [];
212
+ }
213
+ }
214
+
215
+ /**
216
+ * `fetch` that adds `headers` to every request the libSQL HTTP client sends, abandons a request after
217
+ * `REQUEST_TIMEOUT_MS`, and hands a stream's close that failed to `abandoned` for resending.
43
218
  * @param {[string, string][]} headers
219
+ * @param {AbandonedStreams} abandoned
44
220
  * @returns {(input: RequestInfo | URL, init?: RequestInit) => Promise<Response>}
45
221
  */
46
- function fetchWith(headers) {
47
- return (input, init) => {
222
+ function fetchWith(headers, abandoned) {
223
+ return async (input, init) => {
48
224
  const request = new Request(input, init);
49
225
  for (const [name, value] of headers) request.headers.set(name, value);
50
- return fetch(request);
226
+ const copy = request.clone();
227
+ if (await closesNoStream(request.clone())) return closedResponse();
228
+ const timeout = AbortSignal.timeout(REQUEST_TIMEOUT_MS);
229
+ try {
230
+ return await fetch(request, { signal: init?.signal ? AbortSignal.any([init.signal, timeout]) : timeout });
231
+ } catch (error) {
232
+ await abandoned.failed(copy).catch(() => {});
233
+ throw error;
234
+ }
51
235
  };
52
236
  }
53
237
 
@@ -76,16 +260,222 @@ function urlCredentials(url) {
76
260
  return secrets;
77
261
  }
78
262
 
263
+ /**
264
+ * How long a reused stream may sit idle before a statement opens a new one instead. sqld expires a stream idle for 10
265
+ * seconds; a stream reused well inside that never meets its expiry, which costs a request sent for nothing.
266
+ */
267
+ const STREAM_REUSE_IDLE_MS = 5_000;
268
+ /** How long one stream is reused at most, however busy: a later statement opens a new one. */
269
+ const STREAM_REUSE_MAX_AGE_MS = 60_000;
270
+ /**
271
+ * How long closing a database waits for its reused stream's close to be answered. A server that does not answer in
272
+ * time expires the stream itself; the stream runs outside any transaction, so it holds no lock meanwhile.
273
+ */
274
+ const STREAM_CLOSE_WAIT_MS = 1_000;
275
+
276
+ /**
277
+ * The statement a kept stream runs before each statement, in the same request, so its connection rereads a schema
278
+ * another connection changed (#5151). `PRAGMA schema_version` reads the cookie but does not reload the schema; a read of
279
+ * `sqlite_schema` does, and returns no row.
280
+ */
281
+ const SCHEMA_CHECK = "SELECT 1 FROM sqlite_schema LIMIT 0";
282
+
283
+ /**
284
+ * What sqld answers, as plain text with HTTP 400, for a request whose baton it refuses when it looks the stream up,
285
+ * before it runs any request of the pipeline (`stream::acquire` and `Guard::get_db` in sqld's
286
+ * `hrana/http/stream.rs`): the stream was cleaned up after expiring, the baton was used already, or the stream was
287
+ * closed. Any other HTTP 400 can follow a request of the same pipeline that ran, so it is never taken for a refusal.
288
+ */
289
+ const REFUSED_BATON = [
290
+ "Received an invalid baton",
291
+ "Received a baton that has already been used",
292
+ "Stream for this baton was closed",
293
+ ];
294
+
295
+ /**
296
+ * The keyword that makes a statement run on the reused stream: reads and writes that leave the connection as they
297
+ * found it. Anything else, such as `BEGIN`, a `PRAGMA`, a temporary table or a schema change, runs on a stream of its
298
+ * own, so the connection a reused stream holds never carries a transaction or a setting from one call into the next.
299
+ */
300
+ const REUSABLE_KEYWORD = /(?:SELECT|INSERT|UPDATE|DELETE|REPLACE|WITH|VALUES)\b/iy;
301
+
302
+ const WHITESPACE = /\s/;
303
+
304
+ /**
305
+ * Where `sql`'s first token starts, after its leading whitespace, `--` comments and `/* *\/` comments, or -1 when a
306
+ * block comment is left open. One pass, which ends a block comment at its first `*\/` as SQLite does: a regular
307
+ * expression for the same prefix backtracks exponentially on runs of comments such as `/**\//**\/…`.
308
+ * @param {string} sql
309
+ */
310
+ function firstTokenStart(sql) {
311
+ let index = 0;
312
+ while (index < sql.length) {
313
+ if (WHITESPACE.test(sql[index])) index++;
314
+ else if (sql.startsWith("--", index)) {
315
+ const end = sql.indexOf("\n", index + 2);
316
+ if (end === -1) return sql.length;
317
+ index = end + 1;
318
+ } else if (sql.startsWith("/*", index)) {
319
+ const end = sql.indexOf("*/", index + 2);
320
+ if (end === -1) return -1;
321
+ index = end + 2;
322
+ } else return index;
323
+ }
324
+ return index;
325
+ }
326
+
327
+ /**
328
+ * Whether `sql`, one statement, runs on the reused stream.
329
+ * @param {string} sql
330
+ */
331
+ function reusable(sql) {
332
+ const start = firstTokenStart(sql);
333
+ if (start === -1) return false;
334
+ REUSABLE_KEYWORD.lastIndex = start;
335
+ return REUSABLE_KEYWORD.test(sql) && isOneStatement(sql);
336
+ }
337
+
338
+ /**
339
+ * The Hrana code `@libsql/client` gives a failure of the protocol, as its `mapHranaError` does, so a statement on the
340
+ * reused stream fails with the same `LibsqlError` it fails with through the client.
341
+ * @param {typeof import("@libsql/hrana-client")} hrana
342
+ * @param {unknown} error
343
+ * @returns {string}
344
+ */
345
+ function hranaCode(hrana, error) {
346
+ if (error instanceof hrana.ResponseError && error.code !== undefined) return error.code;
347
+ if (error instanceof hrana.ProtoError) return "HRANA_PROTO_ERROR";
348
+ if (error instanceof hrana.ClosedError)
349
+ return error.cause instanceof hrana.ClientError ? hranaCode(hrana, error.cause) : "HRANA_CLOSED_ERROR";
350
+ if (error instanceof hrana.WebSocketError) return "HRANA_WEBSOCKET_ERROR";
351
+ if (error instanceof hrana.HttpServerError) return "SERVER_ERROR";
352
+ if (error instanceof hrana.ProtocolVersionError) return "PROTOCOL_VERSION_ERROR";
353
+ if (error instanceof hrana.InternalError) return "INTERNAL_ERROR";
354
+ return "UNKNOWN";
355
+ }
356
+
357
+ /**
358
+ * One Hrana stream that statements outside a transaction take in turn (#5151). sqld loads the database's schema for
359
+ * every stream it opens, so a request that opens, uses and closes a stream costs time that grows with the number of
360
+ * tables; a stream kept open loads it once. A stream that failed, sat idle `STREAM_REUSE_IDLE_MS` or lived
361
+ * `STREAM_REUSE_MAX_AGE_MS` is closed and a new one opened. A statement is sent again only when the server refused its
362
+ * request without running it: it found the stream expired, or refused its baton (`REFUSED_BATON`). A request whose
363
+ * answer was lost may have run, so its failure is the caller's, as it is for a statement on a stream of its own.
364
+ */
365
+ class ReusedStream {
366
+ /** @type {import("@libsql/hrana-client").HttpStream | undefined} */
367
+ stream;
368
+ openedAt = 0;
369
+ usedAt = 0;
370
+ /** Whether a statement is on the stream now; a statement arriving meanwhile takes a stream of its own. */
371
+ busy = false;
372
+
373
+ /**
374
+ * @param {typeof import("@libsql/hrana-client")} hrana
375
+ * @param {typeof import("@libsql/core/api").LibsqlError} LibsqlError
376
+ * @param {() => import("@libsql/hrana-client").HttpClient} client
377
+ */
378
+ constructor(hrana, LibsqlError, client) {
379
+ this.hrana = hrana;
380
+ this.LibsqlError = LibsqlError;
381
+ this.client = client;
382
+ }
383
+
384
+ /**
385
+ * Runs `sql` on the stream, or answers undefined when another statement is on it.
386
+ * @param {string} sql
387
+ * @param {readonly unknown[]} params
388
+ * @returns {Promise<import("@bastani/atomic/storage").SqliteDriverResult | undefined>}
389
+ */
390
+ async execute(sql, params) {
391
+ if (this.busy) return undefined;
392
+ this.busy = true;
393
+ try {
394
+ for (let resent = false; ; resent = true) {
395
+ const now = performance.now();
396
+ const kept = this.stream;
397
+ if (
398
+ kept !== undefined &&
399
+ (kept.closed || now - this.usedAt > STREAM_REUSE_IDLE_MS || now - this.openedAt > STREAM_REUSE_MAX_AGE_MS)
400
+ )
401
+ this.drop();
402
+ const fresh = this.stream === undefined;
403
+ const stream = (this.stream ??= this.client().openStream());
404
+ if (fresh) this.openedAt = now;
405
+ try {
406
+ const statement = new this.hrana.Stmt(sql);
407
+ statement.bindIndexes(/** @type {import("@libsql/hrana-client").InValue[]} */ ([...params]));
408
+ // A kept connection compiles a statement against the schema it read last, and SQLite rereads the schema
409
+ // only when a statement's transaction finds the schema cookie changed. A statement compiled against a
410
+ // stale schema expands `SELECT *` to the old columns, or fails to compile at all. Reading
411
+ // `sqlite_schema` first, in the same request, makes the connection reread a schema changed elsewhere.
412
+ // A new stream's connection reads the schema afresh.
413
+ const checked = fresh ? undefined : stream.query(new this.hrana.Stmt(SCHEMA_CHECK));
414
+ checked?.catch(() => {});
415
+ const rows = await stream.query(statement);
416
+ return {
417
+ columns: rows.columnNames.map((name) => name ?? ""),
418
+ rows: /** @type {import("@bastani/atomic/storage").SqliteDriverResult["rows"]} */ (rows.rows),
419
+ changes: rows.affectedRowCount,
420
+ };
421
+ } catch (error) {
422
+ if (stream.closed && this.stream === stream) this.stream = undefined;
423
+ if (fresh || resent || !this.ranNothing(error)) throw this.libsqlError(error);
424
+ } finally {
425
+ this.usedAt = performance.now();
426
+ }
427
+ }
428
+ } finally {
429
+ this.busy = false;
430
+ }
431
+ }
432
+
433
+ /**
434
+ * Whether the server refused a request on a kept stream without running it: it found the stream expired, or refused
435
+ * the stream's baton (`REFUSED_BATON`).
436
+ * @param {unknown} error
437
+ */
438
+ ranNothing(error) {
439
+ const { hrana } = this;
440
+ if (error instanceof hrana.ResponseError) return error.code === "STREAM_EXPIRED";
441
+ return (
442
+ error instanceof hrana.HttpServerError &&
443
+ error.status === 400 &&
444
+ REFUSED_BATON.some((refusal) => error.message.endsWith(`: ${refusal}`))
445
+ );
446
+ }
447
+
448
+ /**
449
+ * `error` as `@libsql/client` reports it.
450
+ * @param {unknown} error
451
+ */
452
+ libsqlError(error) {
453
+ return error instanceof this.hrana.ClientError
454
+ ? new this.LibsqlError(error.message, hranaCode(this.hrana, error), undefined, undefined, error)
455
+ : error;
456
+ }
457
+
458
+ /** Closes the stream, if one is open; its close is sent in the background. */
459
+ drop() {
460
+ this.stream?.close();
461
+ this.stream = undefined;
462
+ }
463
+ }
464
+
79
465
  /**
80
466
  * @param {() => Promise<import("@libsql/client").Client | import("@libsql/client").Transaction>} connection
81
467
  * @param {<T>(operation: () => Promise<T>) => Promise<T>} watch
82
468
  * @param {<T>(operation: () => Promise<T>) => Promise<T>} [single] runs one statement, which a failure may run again
469
+ * @param {() => Promise<ReusedStream | undefined>} [shared] the stream statements outside a transaction reuse
83
470
  * @returns {import("@bastani/atomic/storage").SqliteDriverExecutor}
84
471
  */
85
- function statements(connection, watch, single = watch) {
472
+ function statements(connection, watch, single = watch, shared = async () => undefined) {
86
473
  return {
87
474
  execute: (sql, params) =>
88
- single(async () => driverResult(await (await connection()).execute({ sql, args: [...params] }))),
475
+ single(async () => {
476
+ const reused = reusable(sql) ? await (await shared())?.execute(sql, params) : undefined;
477
+ return reused ?? driverResult(await (await connection()).execute({ sql, args: [...params] }));
478
+ }),
89
479
  // An `exec` of one statement wrote nothing when it fails, as one `execute` did; one of several statements may
90
480
  // fail after the statements before it committed.
91
481
  executeMultiple: (sql) =>
@@ -118,6 +508,40 @@ function driverResult(result) {
118
508
  */
119
509
  const FILE_BUSY_TIMEOUT_MS = 5000;
120
510
 
511
+ /**
512
+ * How long a call to a libSQL server runs again while the server answers it `SQLITE_BUSY`. sqld answers a writer that
513
+ * meets its checkpoint `SQLITE_BUSY` at once, without waiting and without running any of the call, and leaves the
514
+ * waiting to the client; its own checkpoint waits this long for the lock.
515
+ */
516
+ const SERVER_BUSY_TIMEOUT_MS = 5000;
517
+ /** The first wait before a call a libSQL server answered busy runs again; each later wait doubles, up to the last. */
518
+ const SERVER_BUSY_FIRST_WAIT_MS = 10;
519
+ const SERVER_BUSY_LAST_WAIT_MS = 200;
520
+
521
+ /**
522
+ * Runs `operation` again, with growing waits, while it fails with `SQLITE_BUSY` and `ranNothing` says the failed run
523
+ * ran nothing, until `SERVER_BUSY_TIMEOUT_MS` has passed; then it fails with the last run's failure. `prepare` runs
524
+ * before each run after the first.
525
+ * @template T
526
+ * @param {() => Promise<T>} operation
527
+ * @param {() => boolean} ranNothing
528
+ * @param {() => Promise<void>} [prepare]
529
+ * @returns {Promise<T>}
530
+ */
531
+ async function whileServerBusy(operation, ranNothing, prepare) {
532
+ const deadline = performance.now() + SERVER_BUSY_TIMEOUT_MS;
533
+ for (let wait = SERVER_BUSY_FIRST_WAIT_MS; ; wait = Math.min(wait * 2, SERVER_BUSY_LAST_WAIT_MS)) {
534
+ try {
535
+ return await operation();
536
+ } catch (error) {
537
+ const left = deadline - performance.now();
538
+ if (!isBusy(error) || !ranNothing() || left <= 0) throw error;
539
+ await new Promise((resolve) => setTimeout(resolve, Math.min(wait, left)));
540
+ await prepare?.();
541
+ }
542
+ }
543
+ }
544
+
121
545
  /**
122
546
  * Whether a driver call failed because another connection held a lock past the busy timeout.
123
547
  * @param {unknown} error
@@ -178,6 +602,190 @@ async function collectAndFinalize() {
178
602
  }
179
603
  }
180
604
 
605
+ /**
606
+ * Write requests one process sends to one database server at once, over every database it opened there: the server takes its
607
+ * write lock one writer at a time, so more requests in flight only lengthen the queue for it, and a process with many
608
+ * workers would fill it. Batches that set `priority` do not count, so a lease renewal never waits behind them.
609
+ */
610
+ const MAX_IN_FLIGHT_WRITES = 4;
611
+
612
+ /** Write batches in a row a server left unanswered, after which its writes count as stalled. */
613
+ const STALLED_WRITES_WARNING_AFTER = 3;
614
+ /**
615
+ * Each server's unanswered write batches in a row, and whether its stall was reported, by its location: one stalled
616
+ * server is told apart from another that answers.
617
+ * @type {Map<string, { unanswered: number; reported: boolean }>}
618
+ */
619
+ const writeHealth = new Map();
620
+
621
+ /**
622
+ * Counts a write batch's outcome toward the process's warning that a server stopped taking writes, once per stall: the
623
+ * first write answered after it warns that writes go through again. sqld 0.24.33 and earlier can stop granting its
624
+ * write lock to anyone (tursodatabase/libsql#2286): the stall warning names the fix.
625
+ * @param {unknown} error undefined for a batch the server answered
626
+ * @param {string} location
627
+ */
628
+ function noteWriteOutcome(error, location) {
629
+ const health = writeHealth.get(location) ?? { unanswered: 0, reported: false };
630
+ if (!isTimeout(error)) {
631
+ // A write answered after a reported stall: the server takes writes again, and a later stall warns again.
632
+ if (health.reported)
633
+ reportWriteHealth(
634
+ false,
635
+ `The libSQL server at ${location} is taking writes again after ${health.unanswered} unanswered write requests.`,
636
+ "ATOMIC_LIBSQL_WRITES_RECOVERED",
637
+ );
638
+ writeHealth.delete(location);
639
+ return;
640
+ }
641
+ health.unanswered++;
642
+ writeHealth.set(location, health);
643
+ if (health.unanswered < STALLED_WRITES_WARNING_AFTER || health.reported) return;
644
+ health.reported = true;
645
+ reportWriteHealth(
646
+ true,
647
+ `The libSQL server at ${location} left ${health.unanswered} write requests in a row unanswered for ${REQUEST_TIMEOUT_MS / 1000} seconds each. sqld 0.24.33 and earlier can stop taking writes from every client until it restarts (https://github.com/tursodatabase/libsql/issues/2286); start sqld with --checkpoint-interval-s 60 (SQLD_CHECKPOINT_INTERVAL_S=60) to avoid it, and restart it now if writes stay stuck. Atomic keeps retrying its writes meanwhile.`,
648
+ "ATOMIC_LIBSQL_WRITES_STALLED",
649
+ );
650
+ }
651
+
652
+ /**
653
+ * Reports a server's write health as a process warning, and as the process event `atomic-libsql-write-health`, which a
654
+ * process that silences warnings (as Atomic's CLI does) still hears: `{ stalled, message }`.
655
+ * @param {boolean} stalled
656
+ * @param {string} message
657
+ * @param {string} code
658
+ */
659
+ function reportWriteHealth(stalled, message, code) {
660
+ process.emitWarning(message, { code });
661
+ /** @type {(event: string, payload: { stalled: boolean; message: string }) => boolean} */ (process.emit)(
662
+ "atomic-libsql-write-health",
663
+ { stalled, message },
664
+ );
665
+ }
666
+
667
+ /**
668
+ * Whether a request was abandoned after `REQUEST_TIMEOUT_MS`, read along the cause chain.
669
+ * @param {unknown} error
670
+ */
671
+ function isTimeout(error) {
672
+ let current = error;
673
+ for (let depth = 0; depth < MAX_CAUSE_DEPTH && typeof current === "object" && current !== null; depth++) {
674
+ const { name, cause } = /** @type {{ readonly name?: unknown, readonly cause?: unknown }} */ (current);
675
+ if (name === "TimeoutError") return true;
676
+ current = cause;
677
+ }
678
+ return false;
679
+ }
680
+
681
+ /**
682
+ * Each server's write lane, by its location: the writes this process has in flight to it and those waiting for one of
683
+ * them to finish. A server that stops answering fills only its own lane, so writes to another server go on.
684
+ * @type {Map<string, { inFlight: number; waiting: (() => void)[] }>}
685
+ */
686
+ const writeLanes = new Map();
687
+
688
+ /**
689
+ * Runs `operation` once fewer than `MAX_IN_FLIGHT_WRITES` other writes of this process to the server at `location` are
690
+ * in flight.
691
+ * @template T
692
+ * @param {string} location
693
+ * @param {() => Promise<T>} operation
694
+ * @returns {Promise<T>}
695
+ */
696
+ async function limitedWrite(location, operation) {
697
+ let lane = writeLanes.get(location);
698
+ if (!lane) writeLanes.set(location, (lane = { inFlight: 0, waiting: [] }));
699
+ const own = lane;
700
+ if (own.inFlight >= MAX_IN_FLIGHT_WRITES) await new Promise((resolve) => own.waiting.push(() => resolve(undefined)));
701
+ else own.inFlight++;
702
+ try {
703
+ return await operation();
704
+ } finally {
705
+ const next = own.waiting.shift();
706
+ // The slot passes straight to the next waiter, so a write arriving meanwhile cannot take it first.
707
+ if (next) next();
708
+ else if (--own.inFlight === 0) writeLanes.delete(location);
709
+ }
710
+ }
711
+
712
+ /**
713
+ * Codes a server answers a batch with when it could not take the write lock in time: the batch wrote nothing.
714
+ * @type {ReadonlySet<unknown>}
715
+ */
716
+ const LOCK_BUSY = new Set(["SQLITE_BUSY", "TRANSACTION_TIMEOUT"]);
717
+
718
+ /**
719
+ * Whether a failed request found the write lock busy, read from each error's `code` along the cause chain.
720
+ * @param {unknown} error
721
+ */
722
+ function isLockBusy(error) {
723
+ let current = error;
724
+ for (let depth = 0; depth < MAX_CAUSE_DEPTH && typeof current === "object" && current !== null; depth++) {
725
+ const { code, cause } = /** @type {{ readonly code?: unknown, readonly cause?: unknown }} */ (current);
726
+ if (LOCK_BUSY.has(code)) return true;
727
+ current = cause;
728
+ }
729
+ return false;
730
+ }
731
+
732
+ /** Marks a `libsql` Database class whose close {@link closeLocalConnectionsOnce} replaced. */
733
+ const closesOnce = Symbol.for("@bastani/atomic-storage-libsql.closesOnce");
734
+
735
+ /**
736
+ * Statements that hold a closing connection until its native close has returned.
737
+ * @type {Set<unknown>}
738
+ */
739
+ const closingHolders = new Set();
740
+
741
+ /**
742
+ * Makes every close of a local libsql connection close its SQLite handle once.
743
+ *
744
+ * libsql 0.5.29, the native driver `@libsql/client` 0.18 uses, is built from the libsql crate 0.9.30. That crate calls
745
+ * `sqlite3_close_v2` twice when the connection itself drops the last reference to its handle, and the second call
746
+ * reads the connection the first one freed (tursodatabase/libsql#2251, fixed in crate 0.10 by
747
+ * tursodatabase/libsql#2261, which no npm `libsql` release contains yet). The connection drops that reference when
748
+ * `close()` finds no statement still holding the handle: a connection used only through `exec`, or one whose
749
+ * statements a collection already finalized. The collections every local close runs (`releaseClosedConnections`)
750
+ * finalize the statements of every other open connection, so the next close in the process is such a close. On
751
+ * Windows the second call crashes the process (0xC0000005) without a message once the heap has reused the freed
752
+ * block. When a statement still holds the handle, the statement's finalizer closes the handle once and then
753
+ * finalizes the statement, which is safe. So each close prepares a statement first and lets it go after the close.
754
+ * The collections the close runs next finalize it and release the database's files.
755
+ * @param {string} clientEntry the resolved `@libsql/client` entry, whose `libsql` is the driver it uses
756
+ */
757
+ function closeLocalConnectionsOnce(clientEntry) {
758
+ /** @type {{ prototype: { close(): void, prepare(sql: string): unknown, open: boolean, [closesOnce]?: true } }} */
759
+ const Database = createRequire(clientEntry)("libsql");
760
+ if (Database.prototype[closesOnce]) return;
761
+ const close = Database.prototype.close;
762
+ Database.prototype.close = function () {
763
+ /** @type {unknown} */
764
+ let holder;
765
+ try {
766
+ if (this.open) holder = this.prepare("SELECT 1");
767
+ } catch {
768
+ // A connection that cannot prepare closes as before.
769
+ }
770
+ if (holder !== undefined) closingHolders.add(holder);
771
+ try {
772
+ return close.call(this);
773
+ } finally {
774
+ closingHolders.delete(holder);
775
+ }
776
+ };
777
+ Database.prototype[closesOnce] = true;
778
+ }
779
+
780
+ /**
781
+ * A libSQL server's (sqld's) default payload limits, which the facade names when a value or a response is too large
782
+ * (#4954): `max_row_size`, which sqld sets as SQLite's `SQLITE_LIMIT_LENGTH` on every connection, is `bytesize::mb(5)`
783
+ * (5,000,000 bytes) for a value and a whole row, and `--max-response-size` is 10 MB for one statement's response.
784
+ */
785
+ const SERVER_PAYLOAD_LIMITS = Object.freeze({ value: 5_000_000, response: 10_000_000 });
786
+ /** A local file's limit: SQLite's default `SQLITE_MAX_LENGTH`. */
787
+ const LOCAL_PAYLOAD_LIMITS = Object.freeze({ value: 1_000_000_000 });
788
+
181
789
  /** How far `transientFailure` follows `cause`: a driver error wraps the network error that fetch wraps. */
182
790
  const MAX_CAUSE_DEPTH = 8;
183
791
  /**
@@ -201,8 +809,8 @@ const CONNECTION_LOST = new Set(["ECONNRESET", "EPIPE", "ETIMEDOUT", "UND_ERR_SO
201
809
 
202
810
  /**
203
811
  * The failure, as one that passes, when the server ended the stream or transaction of a process that stalled, or the
204
- * connection to the server could not be made or broke; read from each error's `code` along the cause chain, never
205
- * from its message.
812
+ * connection to the server could not be made or broke, or something between answered HTTP 5xx; read from each error's
813
+ * `code`, `name` and `status` along the cause chain, never from its message.
206
814
  * @param {import("@bastani/atomic/storage").StorageFailureCause} failure
207
815
  * @returns {import("@bastani/atomic/storage").SqliteDriverTransientFailure | undefined}
208
816
  */
@@ -210,7 +818,15 @@ function transientFailure(failure) {
210
818
  /** @type {unknown} */
211
819
  let current = failure;
212
820
  for (let depth = 0; depth < MAX_CAUSE_DEPTH && typeof current === "object" && current !== null; depth++) {
213
- const { code, cause } = /** @type {{ readonly code?: unknown, readonly cause?: unknown }} */ (current);
821
+ const { code, cause, name, status } = /** @type {{ readonly code?: unknown, readonly cause?: unknown, readonly name?: unknown, readonly status?: unknown }} */ (
822
+ current
823
+ );
824
+ // A request abandoned after `REQUEST_TIMEOUT_MS`: the server may have applied it.
825
+ if (name === "TimeoutError") return { code: "connection-lost", outcome: "unknown" };
826
+ // An HTTP 5xx with no Hrana error in it, such as a gateway's 502 or 504: the request may have reached the server
827
+ // and been applied before the answer was lost (#4970).
828
+ if (name === "HttpServerError" && typeof status === "number" && status >= 500)
829
+ return { code: "connection-lost", outcome: "unknown" };
214
830
  const ended = ENDED_BY_SERVER.get(code);
215
831
  if (ended !== undefined) return { code: ended, outcome: "not-applied" };
216
832
  if (NOT_SENT.has(code)) return { code: "connection-refused", outcome: "not-applied" };
@@ -234,8 +850,33 @@ export async function openLibsqlDatabase({ url, authToken, namespace, headers })
234
850
  throw new StorageAdapterLoadError("@libsql/client", /** @type {import("@bastani/atomic/storage").StorageFailureCause} */ (cause));
235
851
  }
236
852
  const local = /^file:/i.test(url);
853
+ /** @type {[typeof import("@libsql/hrana-client"), typeof import("@libsql/core/config"), typeof import("@libsql/core/uri"), typeof import("@libsql/core/api")] | undefined} */
854
+ let protocol;
855
+ if (!local) {
856
+ try {
857
+ protocol = await Promise.all([
858
+ import("@libsql/hrana-client"),
859
+ import("@libsql/core/config"),
860
+ import("@libsql/core/uri"),
861
+ import("@libsql/core/api"),
862
+ ]);
863
+ } catch (cause) {
864
+ throw new StorageAdapterLoadError(
865
+ "@libsql/hrana-client",
866
+ /** @type {import("@bastani/atomic/storage").StorageFailureCause} */ (cause),
867
+ );
868
+ }
869
+ }
870
+ if (local) {
871
+ try {
872
+ closeLocalConnectionsOnce(createRequire(import.meta.url).resolve("@libsql/client"));
873
+ } catch (cause) {
874
+ throw new StorageAdapterLoadError("libsql", /** @type {import("@bastani/atomic/storage").StorageFailureCause} */ (cause));
875
+ }
876
+ }
237
877
  /** @type {import("@libsql/client").Client | undefined} */
238
878
  let client;
879
+ const abandoned = new AbandonedStreams();
239
880
  // Create the client inside a driver call, so the facade redacts credentials from its failures. The busy timeout is a
240
881
  // client option, not a PRAGMA, because the client opens more connections than the one that ran the PRAGMA, and
241
882
  // each needs it; a connection waits for a lock on the calling thread.
@@ -245,8 +886,36 @@ export async function openLibsqlDatabase({ url, authToken, namespace, headers })
245
886
  authToken,
246
887
  intMode: "bigint",
247
888
  ...(local ? { timeout: FILE_BUSY_TIMEOUT_MS } : {}),
248
- ...(added.length > 0 ? { fetch: fetchWith(added) } : {}),
889
+ ...(local ? {} : { fetch: fetchWith(added, abandoned) }),
249
890
  });
891
+ // The Hrana client whose stream statements outside a transaction reuse, made on first use inside a driver call, as the
892
+ // client is, so the facade redacts credentials from a failure to read the URL.
893
+ /** @type {import("@libsql/hrana-client").HttpClient | undefined} */
894
+ let hranaClient;
895
+ /** @type {ReusedStream | undefined} */
896
+ let reused;
897
+ /** The requests of the reused stream's client not yet answered, which a close waits for. @type {Set<Promise<Response>>} */
898
+ const streamRequests = new Set();
899
+ const shared = async () => {
900
+ if (protocol === undefined) return undefined;
901
+ const [hrana, { expandConfig }, { encodeBaseUrl }, { LibsqlError }] = protocol;
902
+ return (reused ??= new ReusedStream(hrana, LibsqlError, () => {
903
+ if (hranaClient === undefined) {
904
+ const config = expandConfig({ url, authToken }, true);
905
+ const send = fetchWith(added, abandoned);
906
+ /** @type {(input: RequestInfo | URL, init?: RequestInit) => Promise<Response>} */
907
+ const tracked = (input, init) => {
908
+ const sent = send(input, init);
909
+ streamRequests.add(sent);
910
+ void sent.finally(() => streamRequests.delete(sent)).catch(() => {});
911
+ return sent;
912
+ };
913
+ hranaClient = hrana.openHttp(encodeBaseUrl(config.scheme, config.authority, config.path), config.authToken, tracked);
914
+ hranaClient.intMode = "bigint";
915
+ }
916
+ return hranaClient;
917
+ }));
918
+ };
250
919
  // A local statement that fails part way, as a write does when another process holds the lock past the busy timeout,
251
920
  // stays active on its connection: the driver resets and finalizes a statement only when it is garbage collected.
252
921
  // Until then that connection commits nothing ("cannot commit transaction - SQL statements in progress"), and a
@@ -277,6 +946,8 @@ export async function openLibsqlDatabase({ url, authToken, namespace, headers })
277
946
  };
278
947
  /** @type {<T>(operation: () => Promise<T>) => Promise<T>} */
279
948
  const waitForLock = async (operation) => {
949
+ // A server answers busy before it runs anything of one statement or a batch.
950
+ if (!local) return whileServerBusy(() => watch(operation), () => true);
280
951
  /** @type {bigint | number | undefined} */
281
952
  let version;
282
953
  for (;;) {
@@ -302,36 +973,102 @@ export async function openLibsqlDatabase({ url, authToken, namespace, headers })
302
973
  }
303
974
  return (client ??= create());
304
975
  };
976
+ const redact = redactSecrets([authToken ?? "", ...urlCredentials(url), ...Object.values(headers ?? {})]);
977
+ /** The server's location as diagnostics show it, without credentials. */
978
+ const redacted = redact(url);
305
979
  const database = createSqliteDatabase({
306
980
  descriptor: { adapter: "libsql", location: url },
307
- redact: redactSecrets([authToken ?? "", ...urlCredentials(url), ...Object.values(headers ?? {})]),
981
+ redact,
308
982
  transientFailure,
983
+ payloadLimits: local ? LOCAL_PAYLOAD_LIMITS : SERVER_PAYLOAD_LIMITS,
984
+ // A server's batch the busy lock ended: the facade sends it again within the caller's `lockWaitMs`.
985
+ lockBusy: (failure) => !local && isLockBusy(failure),
309
986
  // One statement outside a transaction, run alone or as an `exec` of one statement, that waited out the busy
310
987
  // timeout wrote nothing and runs again. Several statements in one `exec` do not: the ones before the failed
311
988
  // statement were committed.
312
- ...statements(connection, watch, waitForLock),
989
+ ...statements(connection, watch, waitForLock, shared),
313
990
  // One request for the whole batch, in a write transaction: the journal sends a batch of appends this way. Only
314
- // its `BEGIN IMMEDIATE` waits for the write lock, so a batch the busy timeout ended wrote nothing.
315
- batch: (batched) =>
316
- waitForLock(async () => {
991
+ // its `BEGIN IMMEDIATE` waits for the write lock, so a batch the busy timeout ended wrote nothing (`lockBusy`). A
992
+ // server's batch counts against this process's writes in flight to that server, unless it has priority.
993
+ batch: (batched, options) => {
994
+ const send = async () => {
317
995
  const results = await (await connection()).batch(
318
996
  batched.map(({ sql, params }) => ({ sql, args: [...params] })),
319
997
  "write",
320
998
  );
321
999
  return results.map(driverResult);
322
- }),
1000
+ };
1001
+ if (local) return waitForLock(send);
1002
+ const sent = () =>
1003
+ watch(() =>
1004
+ send().then(
1005
+ (results) => {
1006
+ noteWriteOutcome(undefined, redacted);
1007
+ return results;
1008
+ },
1009
+ (error) => {
1010
+ noteWriteOutcome(error, redacted);
1011
+ throw error;
1012
+ },
1013
+ ),
1014
+ );
1015
+ // A server that answers busy ran nothing of the batch: it is sent again as main's other calls are, waiting
1016
+ // outside this process's writes in flight; a caller's `lockWaitMs` waits longer on top of that.
1017
+ return whileServerBusy(
1018
+ () => (options?.priority === true ? sent() : limitedWrite(redacted, sent)),
1019
+ () => true,
1020
+ );
1021
+ },
323
1022
  async beginTransaction() {
324
- const transaction = await waitForLock(async () => (await connection()).transaction("write"));
1023
+ const begin = async () => (await connection()).transaction("write");
1024
+ let transaction = await waitForLock(begin);
1025
+ // A server's transaction sends its `BEGIN IMMEDIATE` with its first statement. A server that answers that busy
1026
+ // fails the statement and the client closes the transaction, having run nothing of it, so the statement runs
1027
+ // again in a new transaction. A busy answer after the transaction began is not run again: a busy statement
1028
+ // leaves the transaction open, and a busy answer that ends its stream also ends the statements it already ran.
1029
+ // The transaction began once a statement in it settled and left it open: the server acknowledged its BEGIN.
1030
+ let started = false;
1031
+ /** @type {<T>(operation: () => Promise<T>) => Promise<T>} */
1032
+ const inTransaction = async (operation) => {
1033
+ try {
1034
+ return await whileServerBusy(
1035
+ () => watch(operation),
1036
+ () => !started && transaction.closed,
1037
+ async () => {
1038
+ transaction = await watch(begin);
1039
+ },
1040
+ );
1041
+ } finally {
1042
+ if (!transaction.closed) started = true;
1043
+ }
1044
+ };
325
1045
  return {
326
- ...statements(async () => transaction, watch),
1046
+ ...statements(async () => transaction, inTransaction, inTransaction),
327
1047
  commit: () => watch(() => transaction.commit()),
328
1048
  rollback: () => watch(() => transaction.rollback()),
329
1049
  close: () => transaction.close(),
330
1050
  };
331
1051
  },
332
1052
  async close() {
1053
+ reused?.drop();
1054
+ hranaClient?.close();
1055
+ if (hranaClient !== undefined) {
1056
+ // The stream's close is sent on a later turn; wait for it a little, so it neither outlives the database for
1057
+ // long nor is left to a server that would otherwise keep the stream until it expires.
1058
+ await new Promise((resolve) => setImmediate(resolve));
1059
+ /** @type {ReturnType<typeof setTimeout> | undefined} */
1060
+ let timer;
1061
+ await Promise.race([
1062
+ Promise.allSettled([...streamRequests]),
1063
+ new Promise((resolve) => {
1064
+ timer = setTimeout(resolve, STREAM_CLOSE_WAIT_MS);
1065
+ }),
1066
+ ]);
1067
+ clearTimeout(timer);
1068
+ }
333
1069
  client?.close();
334
1070
  probe?.close();
1071
+ abandoned.close();
335
1072
  if (local) await releaseClosedConnections();
336
1073
  },
337
1074
  });
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@bastani/atomic-storage-libsql",
3
- "version": "0.9.87-alpha.1",
3
+ "version": "0.9.88-alpha.1",
4
4
  "description": "libSQL / Turso storage adapter for Atomic sessions, workflow journals, and Intercom.",
5
5
  "contributors": [
6
6
  "Norin Lavaee",
@@ -58,7 +58,9 @@
58
58
  ]
59
59
  },
60
60
  "dependencies": {
61
- "@libsql/client": "0.18.0"
61
+ "@libsql/client": "0.18.0",
62
+ "@libsql/core": "0.18.0",
63
+ "@libsql/hrana-client": "0.10.0"
62
64
  },
63
65
  "peerDependencies": {
64
66
  "@bastani/atomic": "*"