@sproutboat/runtime 0.21.1 → 0.23.0

package/CHANGELOG.md

@@ -1,5 +1,43 @@
 # @sproutboat/runtime
 
+## 0.23.0
+
+### Minor Changes
+
+- df612fc: `fetch(Request)`, `fetch(Request, init)` and `new Request(request, init)` behave as in workerd from compatibility date `2026-10-23`. This covers outgoing `fetch()`, service-binding `fetch()` and Durable Object stubs, on the broker and embedded transports alike. Every behaviour was checked against workerd 2026-09-27 by the platform's service-binding suite.
+  
+  From `2026-10-23`:
+  
+  - `fetch(input, init)` is `fetch(new Request(input, init))`, as the Fetch spec says. The Request's method, headers, body, redirect mode and signal apply unless `init` replaces them, and `init.headers` replaces all of the Request's headers.
+  - A Request given as `init` brings its method, headers and body.
+  - A body can be used once. `fetch(request)`, `new Request(request)` and reading it through `request.body` all use it, after which `bodyUsed` is true. Sending or copying a used Request throws `TypeError: Cannot reconstruct a Request with a used body.`, and reading it again throws workerd's "Body has already been used..." message. With a body in `init`, the source Request is left alone.
+  - A GET or HEAD Request with any body throws, even `""` or one inherited from another Request. An incoming GET or HEAD has `request.body === null`.
+  - Methods are workerd's: known names in any case are uppercased, and anything else throws `Invalid HTTP method string`. That includes `CONNECT` and custom names.
+  - `redirect` must be `"follow"` or `"manual"`. As in workerd, `"error"` throws when the Request is made.
+  - The headers of an incoming request, a fetched or service-binding response, and `Response.redirect()` are immutable: `set`, `append` and `delete` throw `TypeError: Can't modify immutable headers.` Copies are mutable, so `new Request(request)`, `new Response(body, response)` and `new Headers(h)` can still be changed.
+  - `AbortSignal.abort()`'s default reason reads "The operation was aborted", as in workerd.
+  
+  With an earlier `compatibility_date` these keep their old behaviour: a copy shares the body, any method name and redirect mode passes, and headers stay mutable. To get the new behaviour, move `compatibility_date` to `2026-10-23` and rebuild.
+  
+  These are fixed at every date, since no app can depend on the old behaviour:
+  
+  - `Request.signal` exists. A Request made with `signal` passes it on to copies and to `fetch()`, which aborts with it.
+  - A `ReadableStream` body in an outgoing `fetch()` is sent as its bytes. Before, it went out as the text `[object ReadableStream]`.
+  - `request.clone()` and `response.clone()` tee a stream body, so both copies can read it. Before, the second reader failed.
+  - A copy of an incoming request keeps its raw bytes. Its `text()` decodes UTF-8, and `fetch()` sends the bytes unchanged.
+
+## 0.22.0
+
+### Minor Changes
+
+- 3f95234: Add experimental bounded local LTX replication to the embedded SQLite transport.
+  Opt in with compatibility date 2026-10-22 and SB_LTX_ENABLE=1. A single writer
+  owns the database, commits are captured durably before checkpoint, and the WAL
+  has an enforced byte ceiling. Default behavior and the broker are unchanged.
+  Verified clean-checkpoint restarts reuse their generation without rewriting a base.
+  
+  Developed with AI assistance from OpenAI Codex (GPT-6.1).
+
 ## 0.21.1
 
 ### Patch Changes

package/LTX_REPLICATION.md

@@ -0,0 +1,124 @@
+# Experimental local LTX replication
+
+AI assistance: OpenAI Codex (GPT-6.1).
+
+This feature is for an embedded D1 database owned by one native executable.
+Enable it with compatibility date `2026-10-22` or later, `SB_LTX_ENABLE=1`,
+and `SB_LTX_DATABASE=DB` (the exact D1 binding name). Earlier compatibility dates,
+unset flags, other bindings and the broker keep their existing behavior.
+No Node or Bun runtime is needed by the executable.
+
+## Capture and bounds
+
+SQLite uses WAL, synchronous FULL, no auto-checkpoint, and exclusive locking.
+An advisory lock also rejects another copy of the executable using the same
+path. Other SQLite connections, including readers, are excluded while it runs.
+Do not use this feature with the SO_REUSEPORT multi-process deployment recipe.
+The raw page-read descriptor stays open for the SQLite connection's lifetime:
+closing any descriptor for the database inode would release process-level POSIX
+locks, even if SQLite's own descriptor remained open.
+
+The commit hook validates the WAL header, salts and rolling checksum through the
+commit frame. It sorts a bounded array of `(pgno, frame)` entries and emits the
+newest version of each page in ascending order. The prior database checksum is
+updated using the old page version from the main file or the captured WAL prefix.
+Truncation removes the checksums of discarded pages. No full database buffer or
+in-memory page-index map is retained.
+
+Each commit creates one LTX v3 file (magic `LTX1`), with a single logical TXID,
+pre-apply checksum, committed database size, salts, and WAL frame range. TXIDs
+start at one inside each generation; they are backup sequence numbers, not an
+application transaction ID. The base contains exact checkpointed live pages
+under a read transaction. It does not use VACUUM INTO. Page compression currently
+uses interoperable literal-only LZ4 blocks; these are fast and have bounded memory,
+but do not reduce file size. The prototype's JS word-based LZ4 compressor is a
+separate path, not a claim that this C writer compresses data.
+
+`SB_LTX_MAX_WAL_BYTES` defaults to 16 MiB, with an allowed range of 128 KiB to
+1 GiB. A VFS wrapper rejects a WAL write or truncate that would cross the ceiling
+with SQLITE_FULL, including inside a single huge transaction. That transaction
+rolls back. Reduce transaction size to fit the ceiling; this implementation does
+not split a transaction. Frame metadata is bounded by that same ceiling, about
+8 bytes per WAL frame. SQLite's normal page cache remains capped at 2 MiB.
+
+After each statement resets, checkpoint TRUNCATE runs. Multi-statement `exec`
+steps and finalizes each statement separately under this opt-in, so every commit
+gets a checkpoint opportunity. Inside an explicit open transaction, BUSY/LOCKED
+can defer checkpointing; the VFS ceiling still applies. SQL that changes the WAL,
+locking, synchronization or page-size invariants, or attaches another database,
+is rejected by an authorizer. `DB.backup()` uses VACUUM INTO and is not supported
+for this opted-in binding. Use the base LTX file for replication.
+
+## Files and durability
+
+Files live under `<database-path>.ltx/epoch-XXXXXX/`. Completed names are
+`<minTXID as 16 hex digits>-<maxTXID as 16 hex digits>.ltx`.
+The encoder streams one page at a time and spools index records with `tmpfile()`.
+
+For every LTX file:
+
+1. Write a `.tmp` file, flush it and fsync it.
+2. Publish the completed filename with an atomic, no-replace hard link, unlink
+   the temporary name, and fsync the generation directory. A finished file is
+   never replaced. This has rename-style publication with no overwrite race.
+3. Write a versioned, checksummed position record to `position.tmp`, fsync it,
+   rename it to `position`, and fsync the root directory. The record includes
+   generation, TXID, salts, next WAL byte offset, database checksum, WAL checksum
+   state, page size and committed page count.
+4. Only then checkpoint the source database. After checkpoint TRUNCATE succeeds,
+   atomically persist a second position record marking the database checkpointed.
+   The same file/directory fsync sequence applies to that record. A crash between
+   checkpoint and publication of the marker retains the conservative fallback.
+
+Capture, fsync and publication failures latch the database closed to subsequent
+operations until restart. The SQL commit has already happened when a hook error
+is returned. A failed request can therefore represent a committed transaction;
+applications must use idempotent transaction identifiers for retries.
+
+## Qualification status
+
+Capture is tested against the exact inline C through `bun run test:native-ltx`,
+including independent BigInt CRC and index verification, a 102-file chain,
+byte-exact restore, integrity checking, oversized transaction rollback, database truncation, explicit rollback and late position-write failure latching.
+The workspace prototype additionally tests the compiled Porffor executable,
+including external writer exclusion and invariant PRAGMA rejection.
+Restart validation now checks the position CRC, completed LTX checksum, WAL
+header/salts, commit frame and cumulative frame checksum. A matching prefix
+resumes in the same generation; any later complete commits are captured before
+serving. Incomplete/torn trailing WAL frames are excluded. An orphan finished
+file ahead of position forces a new generation, preserving the old files.
+Missing, truncated, reset or mismatched WAL takes a fresh page-exact base when
+it contradicts the recorded state. Position format LTXPOS2 distinguishes W
+(captured WAL prefix) from C (checkpoint complete, expected absent/empty WAL).
+A C record also requires a valid completed LTX, unchanged file size and matching
+full database page checksum before the generation is reused. Clean restarts do
+not create a base or rewrite the position. Any unexpected nonempty WAL, offline
+database modification or record mismatch retains the fresh-base fallback.
+Older LTXPOS1 records are conservatively replaced by a new base on first start.
+All state and file descriptors are still created lazily on first DB open.
+
+Verification is a bounded streaming read, but it is O(database size), not constant
+time. It also checks the last LTX file. Five native restarts each at 4.2 MB and
+51.1 MB retained the same generation and unchanged base inode/mtime and position.
+The 51.1 MB verification took 399 to 446 ms on macOS arm64 (plus about 13 ms to
+health readiness); this is not a 9 ms cold-start claim for large databases.
+
+The regression SIGKILLs at ten durability boundaries, including both sides of
+clean-marker publication, and verifies matching-prefix resume and fallback cases.
+It also recovers a committed tail beyond the durable cursor. The full Porffor executable passed 200 seeded random SIGKILL iterations on macOS
+arm64: 1,675 acknowledged writes retained, 88 same-generation resumes and 112
+fresh-base recoveries. Every finished chain passed file/database CRC verification,
+integrity_check, deterministic source comparison at the last durable TXID, and
+comparison after restart. The test also recovered commits whose HTTP response
+was interrupted. Evidence and the runnable harness are in the linked prototype
+workspace; the self-contained durability-boundary regression runs in this repo's
+native CI. These are process-crash tests, not power-cut or storage-failure tests.
+Linux CI exercises the same C implementation; a randomized full executable Linux
+campaign and a power-loss campaign are not yet qualified.
+
+No uploader is included. Finished local files are retained without a retention
+policy; disk usage of the backup chain is not bounded by the WAL ceiling.
+
+Format and API references: [SQLite WAL](https://www.sqlite.org/fileformat2.html#walformat),
+[commit hook](https://www.sqlite.org/c3ref/wal_hook.html),
+[checkpoint](https://www.sqlite.org/c3ref/wal_checkpoint_v2.html).

package/README.md

@@ -78,3 +78,10 @@ generation with its matching size and etag.
 Unfinished uploads expire after seven days. The broker removes expired parts
 when another multipart upload starts; a standalone binary removes them when it
 initializes its embedded store.
+
+## Experimental SQLite replication
+
+The embedded transport can capture bounded, durable local LTX files for one D1
+binding. It requires explicit opt-in and compatibility date 2026-10-22 or later.
+See [LTX replication](LTX_REPLICATION.md) for bounds, recovery, file durability,
+qualification evidence and current limitations. No remote uploader is included.

package/package.json

@@ -1,13 +1,13 @@
 {
   "name": "@sproutboat/runtime",
-  "version": "0.21.1",
+  "version": "0.23.0",
   "license": "MIT",
-  "type": "module",
   "repository": {
     "type": "git",
     "url": "git+https://github.com/baronunread/sproutboat-packages.git",
     "directory": "packages/runtime"
   },
+  "type": "module",
   "main": "./src/index.ts",
   "types": "./src/index.ts",
   "scripts": {

package/src/native-fetch-prelude.js

@@ -3315,6 +3315,15 @@ for (const E of [Error, TypeError, RangeError, ReferenceError, SyntaxError, Eval
 // Object.fromEntries(headers) work. Porffor's Headers has entries() but no iterator.
 if (Headers.prototype[Symbol.iterator] == null) Headers.prototype[Symbol.iterator] = Headers.prototype.entries;
 
+// workerd's immutable headers, from compatibility date 2026-10-23: an incoming request's,
+// a fetched or service-binding response's, and Response.redirect()'s refuse append, set
+// and delete with a TypeError. A copy (new Headers(h), new Response(body, res)) is
+// mutable. The runtime's own writes after the handler lift the guard first.
+function __sbFreezeHeaders(message) {
+  if (globalThis.__sbCompat >= "2026-10-23") message.headers._immutable = true;
+  return message;
+}
+
 // Porffor alpha-10 has no static Response.redirect().
 if (Response.redirect == null) {
   Response.redirect = function (url, status = 302) {
@@ -3322,7 +3331,7 @@ if (Response.redirect == null) {
       throw new RangeError("invalid redirect status");
     }
     // Serialized like the URL parser does (a bare origin gains its "/"); invalid URLs throw.
-    return new Response(null, { status, headers: { location: new URL(String(url)).href } });
+    return __sbFreezeHeaders(new Response(null, { status, headers: { location: new URL(String(url)).href } }));
   };
 }
 
@@ -3739,7 +3748,9 @@ if (typeof globalThis.EventTarget !== "function") {
     __sbAbort(reason) {
       if (this.aborted) return;
       this.aborted = true;
-      this.reason = reason === undefined ? namedError("AbortError", "This operation was aborted") : reason;
+      // workerd's default reason reads "The operation was aborted" (from 2026-10-23 here).
+      const message = globalThis.__sbCompat >= "2026-10-23" ? "The operation was aborted" : "This operation was aborted";
+      this.reason = reason === undefined ? namedError("AbortError", message) : reason;
       const event = new Event("abort");
       if (typeof this.onabort === "function") this.onabort.call(this, event);
       this.dispatchEvent(event);
@@ -5229,7 +5240,7 @@ globalThis.__sbInstallBindings = function (target, bindings) {
       fetch(input, init) {
         if (globalThis.__sbCompat >= "2026-10-22") return __sbServiceFetch(binding, input, init);
         const parts = __sbFetchParts(input, init);
-        const msg = { binding, url: parts.url, method: parts.method, headers: parts.headers, ...__sbRequestBody(parts.body) };
+        const msg = { binding, url: parts.url, method: parts.method, headers: parts.headers, ...__sbPartsBody(parts) };
         return __sbUpstreamResponse(globalThis.__sbFetchUpstream("service.fetch", msg));
       },
     };
@@ -5251,9 +5262,9 @@ async function __sbServiceFetch(binding, input, init) {
   const parts = __sbFetchParts(input, init);
   if (parts.signal) parts.signal.throwIfAborted();
   // A stream body is read to the end first: the wire carries a whole body.
-  let body = __sbUnwrapBody(parts.body);
-  if (body instanceof ReadableStream) body = await __sbCollectBody(body);
-  const msg = { binding, url: parts.url, method: parts.method, headers: parts.headers, ...__sbRequestBody(body) };
+  const body = __sbUnwrapBody(parts.body);
+  const wire = body instanceof ReadableStream ? __sbRequestBody(await __sbCollectBody(body)) : __sbPartsBody({ ...parts, body });
+  const msg = { binding, url: parts.url, method: parts.method, headers: parts.headers, ...wire };
   const response = await __sbFetchFollowing("service.fetch", msg, "sproutboat service " + binding, parts.redirect, parts.signal);
   // The callee threw: as in workerd, its error is thrown here.
   const uncaught = response.headers.get("x-sb-uncaught");
@@ -5436,7 +5447,29 @@ async function __sbRpcDispatch(handlers, receiver, method, request) {
 
 // fetch(input, init) as Fetch reads it: a Request's method, headers, body, redirect mode
 // and signal apply unless init overrides them; a URL object stands for its href.
+// From compatibility date 2026-10-23 it is Fetch's own first step, `new Request(input,
+// init)`, as in workerd: the input Request's body is used up (a second fetch of it
+// throws), a Request given as init brings its body, GET and HEAD refuse a body, and the
+// method and redirect mode are checked. Before it, the parts are read off the input.
 function __sbFetchParts(input, init) {
+  if (globalThis.__sbCompat >= "2026-10-23") {
+    const request = new Request(input, init);
+    const headers = [];
+    request.headers.forEach((v, k) => headers.push([k, v]));
+    const body = request._bodyStream != null ? request._bodyStream : request._bodyInit;
+    // A stream the runtime made over a value is sent as that value, unread: it is used
+    // all the same, so a Request that gave it (as input or as init) reads as used.
+    if (body instanceof ReadableStream && body._sbBody !== undefined) body._disturbed = true;
+    return {
+      url: request.url,
+      method: request.method,
+      headers,
+      body: body === undefined ? null : body,
+      latin1: request._bodyLatin1 === true,
+      redirect: request.redirect,
+      signal: request._signal || null,
+    };
+  }
   const req = input != null && !__sbIsStr(input) && input.url != null ? input : null;
   const url = req ? String(req.url) : input != null && input.href != null ? String(input.href) : String(input);
   const opts = init || {};
@@ -5449,8 +5482,17 @@ function __sbFetchParts(input, init) {
   const body = opts.body !== undefined && !(opts instanceof Request) ? opts.body : req && __sbRawBody(req) !== "" ? __sbRawBody(req) : null;
   const method = opts.method || (req && req.method) || "GET";
   const redirect = opts.redirect || (req && req.redirect) || "follow";
-  const signal = opts.signal || (req && req.signal) || null;
-  return { url, method, headers, body, redirect, signal };
+  const signal = opts.signal || (req && req._signal) || null;
+  // An incoming request's body is its raw bytes, one character each: sent as bytes.
+  const latin1 = body != null && opts.body === undefined && req !== null && req._bodyLatin1 === true;
+  return { url, method, headers, body, latin1, redirect, signal };
+}
+
+// The wire fields of a fetch body (see __sbRequestBody). Raw ingress bytes go as bytes,
+// not as text encoded again.
+function __sbPartsBody(parts) {
+  if (parts.latin1 === true && __sbIsStr(parts.body)) return { body: parts.body, bodyBinary: true };
+  return __sbRequestBody(parts.body);
 }
 
 // An upstream reply (fetch or service.fetch) -> a Response. Headers are appended,
@@ -5469,7 +5511,7 @@ function __sbUpstreamResponse(r) {
   if (r.body != null) respHeaders.set("x-sb-raw-body", "1");
   const response = __sbRawBodyResponse(r.body, { status: r.status || 502, headers: respHeaders });
   if (!typed) response.headers.delete("content-type");
-  return response;
+  return __sbFreezeHeaders(response);
 }
 
 // A streamed upstream response (from compatibility date 2026-10-22, with the other
@@ -5520,7 +5562,7 @@ function __sbUpstreamStreamed(op, msg, label, signal) {
             __sbCancelAsync(id);
           },
         });
-        resolve(new Response(body, { status: reply.status || 502, headers }));
+        resolve(__sbFreezeHeaders(new Response(body, { status: reply.status || 502, headers })));
         return;
       }
       const frame = JSON.parse(raw);
@@ -5645,9 +5687,23 @@ function __sbUpstream(op, msg, label, signal, always) {
 // prelude must not replace fetch just by being evaluated.
 globalThis.__sbInstallFetch = function () {
   globalThis.fetch = function (input, init) {
-    const parts = __sbFetchParts(input, init);
-    const msg = { url: parts.url, method: parts.method, headers: parts.headers, ...__sbRequestBody(parts.body) };
-    return __sbFetchFollowing("fetch", msg, "sproutboat fetch", parts.redirect, parts.signal);
+    let parts;
+    try {
+      parts = __sbFetchParts(input, init);
+    } catch (error) {
+      // From 2026-10-23 an invalid request rejects, as in workerd; before it, it threw.
+      if (globalThis.__sbCompat >= "2026-10-23") return Promise.reject(error);
+      throw error;
+    }
+    const send = (body) => {
+      const msg = { url: parts.url, method: parts.method, headers: parts.headers, ...body };
+      return __sbFetchFollowing("fetch", msg, "sproutboat fetch", parts.redirect, parts.signal);
+    };
+    // A stream body is read to the end first, as for a service binding: the wire carries a
+    // whole body. It was sent as the text "[object ReadableStream]".
+    const stream = __sbUnwrapBody(parts.body);
+    if (stream instanceof ReadableStream) return __sbCollectBody(stream).then((bytes) => send(__sbRequestBody(bytes)));
+    return send(__sbPartsBody(parts));
   };
 };
 
@@ -5691,7 +5747,11 @@ function __sbMakeDONamespace(binding, className) {
       return {
         fetch(input, init) {
           let req;
-          if (__sbIsObj(input) && __sbIsStr(input.url) && !init) {
+          // From 2026-10-23 the object gets the Request Fetch makes of input and init, as in
+          // workerd: method, headers and body survive, and the input's body is used up.
+          if (globalThis.__sbCompat >= "2026-10-23") {
+            req = new Request(input, init);
+          } else if (__sbIsObj(input) && __sbIsStr(input.url) && !init) {
             req = input;
           } else {
             const url = __sbIsStr(input) ? input : String((input && input.url) || "https://do/");
@@ -6101,6 +6161,8 @@ function __sbStreamingSupported() {
 function __sbStreamResponse(response) {
   const stream = __sbRawBody(response);
   const id = __sbStreamOp("o", "0", "");
+  // A fetched response returned as it is has immutable headers; this mark is ours.
+  response.headers._immutable = false;
   response.headers.set("x-sb-stream", id);
   response.body = "";
   (async () => {
@@ -6227,6 +6289,12 @@ async function __sbQueueWithDrain(result, tasks) {
 function __sbDecodeIngress(request) {
   const raw = __sbRawBody(request);
   if (raw == null) return;
+  // A GET or HEAD has no body, as in workerd (from 2026-10-23): request.body is null, and
+  // new Request(request) copies none. Before it, the body is the empty string.
+  if (raw === "" && (request.method === "GET" || request.method === "HEAD") && globalThis.__sbCompat >= "2026-10-23") {
+    request.body = null;
+    return;
+  }
   // Raw bytes, one character each: arrayBuffer(), blob() and body read them as bytes.
   request._bodyLatin1 = true;
   let decoded = null;
@@ -6258,6 +6326,8 @@ globalThis.__sbEntry = function (handlers, request) {
     const __tlsCipher = __sbTrustedProxyHeader(request, "x-sb-tls-cipher");
     if (__tlsCipher) __cf.tlsCipher = __tlsCipher;
     request.cf = __cf;
+    // The edge's headers are gone: from 2026-10-23 the handler cannot change the rest.
+    __sbFreezeHeaders(request);
     // #28 — per-invocation CPU time. One fetch turn per process (serial), so the
     // process CPU delta across the handler is this invocation's CPU.
     // ponytail: serial-turn assumption; revisit if the profile ever allows