@sproutboat/runtime 0.21.0 → 0.22.0

package/CHANGELOG.md

@@ -1,5 +1,39 @@
 # @sproutboat/runtime
 
+## 0.22.0
+
+### Minor Changes
+
+- 3f95234: Add experimental bounded local LTX replication to the embedded SQLite transport.
+  Opt in with compatibility date 2026-10-22 and SB_LTX_ENABLE=1. A single writer
+  owns the database, commits are captured durably before checkpoint, and the WAL
+  has an enforced byte ceiling. Default behavior and the broker are unchanged.
+  Verified clean-checkpoint restarts reuse their generation without rewriting a base.
+  
+  Developed with AI assistance from OpenAI Codex (GPT-6.1).
+
+## 0.21.1
+
+### Patch Changes
+
+- 98800ca: The asynchronous `fetch()` and service-binding behaviour from 0.20.0 now starts at compatibility date `2026-10-22`. Before, it applied at every date, which broke apps that use the result without `await`, such as the CLI's kitchen-sink: `const r = fetch(url); r.body`.
+  
+  From `2026-10-22`, in a deployed sprout:
+  
+  - `fetch()` and `env.SERVICE.fetch()` return promises and run off the event loop;
+  - responses stream;
+  - redirects are followed;
+  - a callee's error is thrown in the caller;
+  - a callee sees its caller's URL.
+  
+  With an earlier `compatibility_date`, a deployed sprout's `fetch()` and binding calls answer at once with a `Response`, as they did before 0.20.0. To get the new behaviour, move `compatibility_date` to `2026-10-22` and rebuild.
+  
+  These stay ungated:
+  
+  - sproutd answered with promises at every date, and still does;
+  - RPC calls are always asynchronous;
+  - abort signals, `Set-Cookie` and `Content-Type` work at every date.
+
 ## 0.21.0
 
 ### Minor Changes

package/LTX_REPLICATION.md

@@ -0,0 +1,124 @@
+# Experimental local LTX replication
+
+AI assistance: OpenAI Codex (GPT-6.1).
+
+This feature is for an embedded D1 database owned by one native executable.
+Enable it with compatibility date `2026-10-22` or later, `SB_LTX_ENABLE=1`,
+and `SB_LTX_DATABASE=DB` (the exact D1 binding name). Earlier compatibility dates,
+unset flags, other bindings and the broker keep their existing behavior.
+No Node or Bun runtime is needed by the executable.
+
+## Capture and bounds
+
+SQLite uses WAL, synchronous FULL, no auto-checkpoint, and exclusive locking.
+An advisory lock also rejects another copy of the executable using the same
+path. Other SQLite connections, including readers, are excluded while it runs.
+Do not use this feature with the SO_REUSEPORT multi-process deployment recipe.
+The raw page-read descriptor stays open for the SQLite connection's lifetime:
+closing any descriptor for the database inode would release process-level POSIX
+locks, even if SQLite's own descriptor remained open.
+
+The commit hook validates the WAL header, salts and rolling checksum through the
+commit frame. It sorts a bounded array of `(pgno, frame)` entries and emits the
+newest version of each page in ascending order. The prior database checksum is
+updated using the old page version from the main file or the captured WAL prefix.
+Truncation removes the checksums of discarded pages. No full database buffer or
+in-memory page-index map is retained.
+
+Each commit creates one LTX v3 file (magic `LTX1`), with a single logical TXID,
+pre-apply checksum, committed database size, salts, and WAL frame range. TXIDs
+start at one inside each generation; they are backup sequence numbers, not an
+application transaction ID. The base contains exact checkpointed live pages
+under a read transaction. It does not use VACUUM INTO. Page compression currently
+uses interoperable literal-only LZ4 blocks; these are fast and have bounded memory,
+but do not reduce file size. The prototype's JS word-based LZ4 compressor is a
+separate path, not a claim that this C writer compresses data.
+
+`SB_LTX_MAX_WAL_BYTES` defaults to 16 MiB, with an allowed range of 128 KiB to
+1 GiB. A VFS wrapper rejects a WAL write or truncate that would cross the ceiling
+with SQLITE_FULL, including inside a single huge transaction. That transaction
+rolls back. Reduce transaction size to fit the ceiling; this implementation does
+not split a transaction. Frame metadata is bounded by that same ceiling, about
+8 bytes per WAL frame. SQLite's normal page cache remains capped at 2 MiB.
+
+After each statement resets, checkpoint TRUNCATE runs. Multi-statement `exec`
+steps and finalizes each statement separately under this opt-in, so every commit
+gets a checkpoint opportunity. Inside an explicit open transaction, BUSY/LOCKED
+can defer checkpointing; the VFS ceiling still applies. SQL that changes the WAL,
+locking, synchronization or page-size invariants, or attaches another database,
+is rejected by an authorizer. `DB.backup()` uses VACUUM INTO and is not supported
+for this opted-in binding. Use the base LTX file for replication.
+
+## Files and durability
+
+Files live under `<database-path>.ltx/epoch-XXXXXX/`. Completed names are
+`<minTXID as 16 hex digits>-<maxTXID as 16 hex digits>.ltx`.
+The encoder streams one page at a time and spools index records with `tmpfile()`.
+
+For every LTX file:
+
+1. Write a `.tmp` file, flush it and fsync it.
+2. Publish the completed filename with an atomic, no-replace hard link, unlink
+   the temporary name, and fsync the generation directory. A finished file is
+   never replaced. This has rename-style publication with no overwrite race.
+3. Write a versioned, checksummed position record to `position.tmp`, fsync it,
+   rename it to `position`, and fsync the root directory. The record includes
+   generation, TXID, salts, next WAL byte offset, database checksum, WAL checksum
+   state, page size and committed page count.
+4. Only then checkpoint the source database. After checkpoint TRUNCATE succeeds,
+   atomically persist a second position record marking the database checkpointed.
+   The same file/directory fsync sequence applies to that record. A crash between
+   checkpoint and publication of the marker retains the conservative fallback.
+
+Capture, fsync and publication failures latch the database closed to subsequent
+operations until restart. The SQL commit has already happened when a hook error
+is returned. A failed request can therefore represent a committed transaction;
+applications must use idempotent transaction identifiers for retries.
+
+## Qualification status
+
+Capture is tested against the exact inline C through `bun run test:native-ltx`,
+including independent BigInt CRC and index verification, a 102-file chain,
+byte-exact restore, integrity checking, oversized transaction rollback, database truncation, explicit rollback and late position-write failure latching.
+The workspace prototype additionally tests the compiled Porffor executable,
+including external writer exclusion and invariant PRAGMA rejection.
+Restart validation now checks the position CRC, completed LTX checksum, WAL
+header/salts, commit frame and cumulative frame checksum. A matching prefix
+resumes in the same generation; any later complete commits are captured before
+serving. Incomplete/torn trailing WAL frames are excluded. An orphan finished
+file ahead of position forces a new generation, preserving the old files.
+Missing, truncated, reset or mismatched WAL takes a fresh page-exact base when
+it contradicts the recorded state. Position format LTXPOS2 distinguishes W
+(captured WAL prefix) from C (checkpoint complete, expected absent/empty WAL).
+A C record also requires a valid completed LTX, unchanged file size and matching
+full database page checksum before the generation is reused. Clean restarts do
+not create a base or rewrite the position. Any unexpected nonempty WAL, offline
+database modification or record mismatch retains the fresh-base fallback.
+Older LTXPOS1 records are conservatively replaced by a new base on first start.
+All state and file descriptors are still created lazily on first DB open.
+
+Verification is a bounded streaming read, but it is O(database size), not constant
+time. It also checks the last LTX file. Five native restarts each at 4.2 MB and
+51.1 MB retained the same generation and unchanged base inode/mtime and position.
+The 51.1 MB verification took 399 to 446 ms on macOS arm64 (plus about 13 ms to
+health readiness); this is not a 9 ms cold-start claim for large databases.
+
+The regression SIGKILLs at ten durability boundaries, including both sides of
+clean-marker publication, and verifies matching-prefix resume and fallback cases.
+It also recovers a committed tail beyond the durable cursor. The full Porffor executable passed 200 seeded random SIGKILL iterations on macOS
+arm64: 1,675 acknowledged writes retained, 88 same-generation resumes and 112
+fresh-base recoveries. Every finished chain passed file/database CRC verification,
+integrity_check, deterministic source comparison at the last durable TXID, and
+comparison after restart. The test also recovered commits whose HTTP response
+was interrupted. Evidence and the runnable harness are in the linked prototype
+workspace; the self-contained durability-boundary regression runs in this repo's
+native CI. These are process-crash tests, not power-cut or storage-failure tests.
+Linux CI exercises the same C implementation; a randomized full executable Linux
+campaign and a power-loss campaign are not yet qualified.
+
+No uploader is included. Finished local files are retained without a retention
+policy; disk usage of the backup chain is not bounded by the WAL ceiling.
+
+Format and API references: [SQLite WAL](https://www.sqlite.org/fileformat2.html#walformat),
+[commit hook](https://www.sqlite.org/c3ref/wal_hook.html),
+[checkpoint](https://www.sqlite.org/c3ref/wal_checkpoint_v2.html).

package/README.md

@@ -78,3 +78,10 @@ generation with its matching size and etag.
 Unfinished uploads expire after seven days. The broker removes expired parts
 when another multipart upload starts; a standalone binary removes them when it
 initializes its embedded store.
+
+## Experimental SQLite replication
+
+The embedded transport can capture bounded, durable local LTX files for one D1
+binding. It requires explicit opt-in and compatibility date 2026-10-22 or later.
+See [LTX replication](LTX_REPLICATION.md) for bounds, recovery, file durability,
+qualification evidence and current limitations. No remote uploader is included.

package/package.json

@@ -1,13 +1,13 @@
 {
   "name": "@sproutboat/runtime",
-  "version": "0.21.0",
+  "version": "0.22.0",
   "license": "MIT",
-  "type": "module",
   "repository": {
     "type": "git",
     "url": "git+https://github.com/baronunread/sproutboat-packages.git",
     "directory": "packages/runtime"
   },
+  "type": "module",
   "main": "./src/index.ts",
   "types": "./src/index.ts",
   "scripts": {

package/src/native-fetch-prelude.js

@@ -5218,24 +5218,19 @@ globalThis.__sbInstallBindings = function (target, bindings) {
 
   // #48 — worker-to-worker. Same wire shape as outbound fetch, but the broker
   // resolves the target itself and forwards it internally, so this is not
-  // egress and is not subject to the private-address block. Always a promise, as
-  // in Workers; on an async host the caller keeps serving while the call runs,
-  // which is also what lets a deployment call itself without deadlocking.
+  // egress and is not subject to the private-address block. From compatibility date
+  // 2026-10-22 a promise, as in Workers: on an async host the caller keeps serving
+  // while the call runs, which is also what lets a deployment call itself without
+  // deadlocking. Before it, the Response comes back at once, as it always did, so a
+  // handler that uses it without awaiting keeps working.
   for (let i = 0; i < (bindings.services || []).length; i++) {
     const binding = bindings.services[i].binding;
     const fetcher = {
-      async fetch(input, init) {
+      fetch(input, init) {
+        if (globalThis.__sbCompat >= "2026-10-22") return __sbServiceFetch(binding, input, init);
         const parts = __sbFetchParts(input, init);
-        if (parts.signal) parts.signal.throwIfAborted();
-        // A stream body is read to the end first: the wire carries a whole body.
-        let body = __sbUnwrapBody(parts.body);
-        if (body instanceof ReadableStream) body = await __sbCollectBody(body);
-        const msg = { binding, url: parts.url, method: parts.method, headers: parts.headers, ...__sbRequestBody(body) };
-        const response = await __sbFetchFollowing("service.fetch", msg, "sproutboat service " + binding, parts.redirect, parts.signal);
-        // The callee threw: as in workerd, its error is thrown here.
-        const uncaught = response.headers.get("x-sb-uncaught");
-        if (uncaught != null) throw __sbRemoteError(uncaught);
-        return response;
+        const msg = { binding, url: parts.url, method: parts.method, headers: parts.headers, ...__sbRequestBody(parts.body) };
+        return __sbUpstreamResponse(globalThis.__sbFetchUpstream("service.fetch", msg));
       },
     };
     // Any other name is an RPC method of the callee's WorkerEntrypoint (see __sbRpcCall).
@@ -5248,9 +5243,24 @@ globalThis.__sbInstallBindings = function (target, bindings) {
       },
     });
   }
-
 };
 
+// A service-binding fetch from compatibility date 2026-10-22: a promise, off the event
+// loop on an async host, following redirects, and throwing the callee's error.
+async function __sbServiceFetch(binding, input, init) {
+  const parts = __sbFetchParts(input, init);
+  if (parts.signal) parts.signal.throwIfAborted();
+  // A stream body is read to the end first: the wire carries a whole body.
+  let body = __sbUnwrapBody(parts.body);
+  if (body instanceof ReadableStream) body = await __sbCollectBody(body);
+  const msg = { binding, url: parts.url, method: parts.method, headers: parts.headers, ...__sbRequestBody(body) };
+  const response = await __sbFetchFollowing("service.fetch", msg, "sproutboat service " + binding, parts.redirect, parts.signal);
+  // The callee threw: as in workerd, its error is thrown here.
+  const uncaught = response.headers.get("x-sb-uncaught");
+  if (uncaught != null) throw __sbRemoteError(uncaught);
+  return response;
+}
+
 // The error a service-binding callee threw, from its x-sb-uncaught marker: the same
 // standard class where there is one, else an Error with the original name.
 function __sbRemoteError(marker) {
@@ -5381,7 +5391,7 @@ async function __sbRpcCall(binding, method, args) {
     ],
     body,
   };
-  const response = await __sbUpstream("service.fetch", msg, label, null);
+  const response = await __sbUpstream("service.fetch", msg, label, null, true);
   const uncaught = response.headers.get("x-sb-uncaught");
   if (uncaught != null) throw __sbRemoteError(uncaught);
   if (response.headers.get("x-sb-rpc-result") !== "1") {
@@ -5462,8 +5472,8 @@ function __sbUpstreamResponse(r) {
   return response;
 }
 
-// A streamed upstream response (from compatibility date 2026-10-21, where body is a
-// stream anyway): the call resolves as soon as the head arrives, and the body is a
+// A streamed upstream response (from compatibility date 2026-10-22, with the other
+// asynchronous fetch() behaviour; body is a stream from 2026-10-21 anyway): the call resolves as soon as the head arrives, and the body is a
 // ReadableStream fed chunk by chunk as the broker passes them on. Each frame fires the
 // same callback, so it is an interval that the last frame clears. A reply that comes
 // whole (a broker from before streaming, or sproutd's embedded transport) is handled
@@ -5587,10 +5597,15 @@ function __sbFollowHop(op, msg, label, mode, signal, hops) {
 // callback alive until then. An aborted `signal` rejects with its reason and cancels
 // the exchange, which closes its broker connection: the broker stops the upstream
 // request, and the edge in turn stops the callee's.
-function __sbUpstream(op, msg, label, signal) {
+// Promises from compatibility date 2026-10-22. Before it, a transport that always
+// answered at once (the broker's) still does, so handlers that use fetch()'s result
+// without awaiting keep working; sproutd's answered with promises at any date
+// (__sbAsyncAtAnyDate). `always` is for RPC, which has no older behaviour to keep.
+function __sbUpstream(op, msg, label, signal, always) {
   if (signal) signal.throwIfAborted();
-  if (!__sbHttpAsyncEnabled()) return __sbUpstreamResponse(globalThis.__sbFetchUpstream(op, msg));
-  if (globalThis.__sbCompat >= "2026-10-21") return __sbUpstreamStreamed(op, msg, label, signal);
+  const promised = always || globalThis.__sbCompat >= "2026-10-22" || __sbAsyncAtAnyDate();
+  if (!__sbHttpAsyncEnabled() || !promised) return __sbUpstreamResponse(globalThis.__sbFetchUpstream(op, msg));
+  if (globalThis.__sbCompat >= "2026-10-22") return __sbUpstreamStreamed(op, msg, label, signal);
   return new Promise((resolve, reject) => {
     let onAbort = null;
     const settled = () => {
@@ -6164,7 +6179,8 @@ function __sbBindingCallIngress(request) {
   const vouched = !!want && headers.get("x-sb-binding-call") === "1" && __sbEqCt(headers.get("x-sb-edge-token") || "", want);
   if (vouched) {
     const url = headers.get("x-sb-url");
-    if (url && URL.canParse(url)) request.url = url;
+    // The callee sees its caller's URL from compatibility date 2026-10-22, as in workerd.
+    if (url && URL.canParse(url) && globalThis.__sbCompat >= "2026-10-22") request.url = url;
     headers.delete("x-sb-remote-addr");
   }
   for (const name of ["x-sb-binding-call", "x-sb-url", "x-sb-edge-token"]) {

package/src/transport-broker.js

@@ -895,6 +895,11 @@ function __sbCancelAsync(asyncId) {
   `;
 }
 
+// Before compatibility date 2026-10-22, fetch() through the broker answered at once.
+function __sbAsyncAtAnyDate() {
+  return false;
+}
+
 // Whether the host runs exchanges off its loop (it installed a waker).
 function __sbHttpAsyncEnabled() {
   let res = false;