@sproutboat/wire 0.12.2 → 0.13.0

package/CHANGELOG.md

@@ -1,5 +1,46 @@
 # @sproutboat/wire
 
+## 0.13.0
+
+### Minor Changes
+
+- 6880f60: Add an opt-in bounded multi-session broker worker with private supervisor control.
+  Fence queue acknowledgements to durable delivery claims and retain recoverable alarm deliveries across process crashes in broker and embedded transports.
+  
+  Resume native R2 downloads until completion or renewed backpressure instead of leaving large responses stalled.
+  
+  Retain partially written TCP replies until drain and serialize pipelined replies so large broker-backed R2 reads cannot truncate or interleave frames.
+  
+  Honor wildcard, weak and list-based If-None-Match conditions on broker direct downloads.
+  
+  Move D1 queries, batches, exec and backups off the shared broker event loop into two lazy SQL threads per broker process. Preserve database affinity, transaction rollback and session isolation, and bound queued work with per-session admission and queue deadlines. Running SQLite statements remain non-preemptible; keep shared mode opt-in.
+  
+  Enforce shared D1 execution budgets through a capability-protected native SQLite progress hook. Interrupted batches roll back and lanes remain usable. A broker watchdog revokes all sessions if native work cannot be interrupted; uncertain SQL outcomes are never automatically retried. Build the small SQLite extension during host/image preparation.
+- a96a443: Apply static asset redirects and header rules in dev and standalone asset bindings, preserving binary response bodies.
+
+### Patch Changes
+
+- Updated dependencies [5d4d40b]
+- Updated dependencies [6880f60]
+- Updated dependencies [846519f]
+- Updated dependencies [b47aad4]
+- Updated dependencies [f19ca0d]
+- Updated dependencies [ddb0f9d]
+- Updated dependencies [f201b58]
+- Updated dependencies [08d0192]
+- Updated dependencies [6d6ae43]
+- Updated dependencies [9fdf3fa]
+- Updated dependencies [4c78d83]
+- Updated dependencies [088cad0]
+- Updated dependencies [a96a443]
+- Updated dependencies [6e42051]
+- Updated dependencies [6e88627]
+- Updated dependencies [9dc109d]
+- Updated dependencies [28b0c39]
+- Updated dependencies [6b64985]
+- Updated dependencies [eec94f1]
+  - @sproutboat/runtime@0.17.0
+
 ## 0.12.2
 
 ### Patch Changes

package/README.md

@@ -28,3 +28,45 @@ import { parseJsonValue } from "@sproutboat/wire";
   an unparsed value.
 
 See `src/broker.test.ts` for the protocol contract in executable form.
+
+## Shared SQL execution budgets
+
+Shared mode requires the small SQLite deadline extension. Build it during host
+or image preparation, before starting the broker, with:
+
+```sh
+bun packages/wire/src/build-d1-budget.ts
+```
+
+From the published package directory, use `bun run build:sql-budget`. The build
+needs a C compiler and SQLite development headers (`gcc libsqlite3-dev` on
+Debian/Ubuntu). It uses SQLite's extension ABI and does not link another SQLite
+engine on Linux. Native outputs are host-specific and excluded from the source
+package. The platform installer builds the extension during provisioning and
+staged updates; tenant requests never invoke a compiler.
+
+The default shared D1 execution budget is 1,000 ms per operation, including an
+entire transactional batch. Operators can set `SPROUTBOAT_D1_EXECUTION_MS` to an
+integer from 10 to 30,000. Queue admission remains 128 active/queued calls per
+broker, 16 per session, with a separate ten-second queue deadline. Tenant request
+fields cannot change execution budgets.
+
+A native progress callback checks the monotonic deadline every 1,000 SQLite VM
+instructions. Interrupted batches roll back, and the same SQL lane remains
+usable. Multi-statement `exec` retains SQLite's existing partial-commit behavior:
+earlier statements can have committed before a later statement fails. The
+executor never automatically retries uncertain mutations.
+
+Locks and other native operations may not poll SQLite progress. A broker-thread
+watchdog allows one additional second, then revokes the broker and all of its
+sessions through supervisor recovery. That fallback reports an uncertain SQL
+outcome; it does not assert that every earlier mutation rolled back. Shared mode
+remains opt-in, and this mechanism is not a memory isolation boundary.
+
+On macOS, Bun's default Apple SQLite disables extensions. Install an
+extension-capable SQLite library and select it before opening any SQLite DBs.
+The shared broker and package test preload select Homebrew's standard library
+location; `SPROUTBOAT_D1_SQLITE_LIBRARY` overrides it. `SQLITE_INCLUDE` overrides
+the build header directory, and `SPROUTBOAT_D1_BUDGET_EXTENSION` overrides the
+trusted extension path. Linux uses Bun's existing SQLite engine. Dedicated
+broker mode does not enable execution budgets.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@sproutboat/wire",
-  "version": "0.12.2",
+  "version": "0.13.0",
   "license": "MIT",
   "type": "module",
   "repository": {
@@ -12,15 +12,17 @@
   "types": "./src/index.ts",
   "dependencies": {
     "@sproutboat/assets": "^0.3.0",
-    "@sproutboat/runtime": "^0.16.0"
+    "@sproutboat/runtime": "^0.17.0"
   },
   "exports": {
     ".": "./src/index.ts",
     "./broker": "./src/broker.ts",
     "./package.json": "./package.json",
-    "./egress": "./src/egress.ts"
+    "./egress": "./src/egress.ts",
+    "./pool": "./src/pool.ts"
   },
   "scripts": {
-    "test": "bun test ./"
+    "test": "bun test ./",
+    "build:sql-budget": "bun src/build-d1-budget.ts"
   }
 }

package/src/broker.test.ts

@@ -7,6 +7,7 @@ import {
   createBroker,
   cronMatches,
   encodeFrame,
+  frameOf,
   encodeV1,
   listen,
   listenTransfers,
@@ -15,6 +16,7 @@ import {
   type FetchLike,
 } from "./broker";
 import { createHash } from "node:crypto";
+import { createConnection } from "node:net";
 import { walkAssets, type AssetManifest } from "@sproutboat/assets";
 import { jsonObject, parseJsonValue, type JsonObject, type JsonValue } from "./json";
 
@@ -35,11 +37,77 @@ function make(opts: Parameters<typeof createBroker>[0] = {}): Broker {
   return b;
 }
 
+test("a stale queue acknowledgement cannot remove a newer delivery claim", async () => {
+  const root = mkdtempSync(join(tmpdir(), "sb-claim-fence-"));
+  const path = join(root,"state.sqlite");
+  const first = Promise.withResolvers<void>();
+  const second = Promise.withResolvers<void>();
+  const finishFirst = Promise.withResolvers<Response>();
+  const finishSecond = Promise.withResolvers<Response>();
+  let enabled = false;
+  const a = createBroker({db:path,bindings:{queues:["JOBS"]},sproutUrl:"http://127.0.0.1:1/",deliveryTimeoutMs:2_000,deliveryLeaseMs:3_000,fetchImpl:async()=>{first.resolve();return finishFirst.promise;}});
+  const b = createBroker({db:path,bindings:{queues:["JOBS"]},sproutUrl:"http://127.0.0.1:1/",dispatchEnabled:()=>enabled,deliveryTimeoutMs:2_000,deliveryLeaseMs:3_000,fetchImpl:async()=>{second.resolve();return finishSecond.promise;}});
+  const inspect = new Database(path);
+  try {
+    await a.dispatch({op:"queue.send",queue:"JOBS",body:"work"});
+    await first.promise;
+    const row=inspect.query<{id:string;claim_token:string},[]>("SELECT id,claim_token FROM mq").get()!;
+    inspect.query("UPDATE mq SET visible_at=0").run();
+    enabled=true;
+    await second.promise;
+    expect(inspect.query<{claim_token:string},[]>("SELECT claim_token FROM mq").get()!.claim_token).not.toBe(row.claim_token);
+    finishFirst.resolve(Response.json({ack:[row.id],retry:[]}));
+    await Bun.sleep(30);
+    expect(inspect.query<{count:number},[]>("SELECT count(*) AS count FROM mq").get()!.count).toBe(1);
+    finishSecond.resolve(Response.json({ack:[row.id],retry:[]}));
+    await Bun.sleep(30);
+    expect(inspect.query<{count:number},[]>("SELECT count(*) AS count FROM mq").get()!.count).toBe(0);
+  } finally {a.close();b.close();inspect.close();rmSync(root,{recursive:true,force:true});}
+},5_000);
+
 test("ping round-trips", async () => {
   const b = make();
   expect(await b.dispatch({ op: "ping", msg: "hi" })).toMatchObject({ ok: true, op: "pong", echo: "hi" });
 });
 
+test("a new alarm replaces exhausted delivery state and delete cancels durable retries", async () => {
+  const root=mkdtempSync(join(tmpdir(),"sb-alarm-replace-"));
+  const path=join(root,"state.sqlite");
+  const broker=createBroker({db:path,bindings:{do:[{binding:"COUNTER",className:"Counter"}]}});
+  const inspect=new Database(path);
+  try {
+    inspect.query("INSERT INTO do_alarm_delivery VALUES ('Counter','one',1,5,'old',0)").run();
+    await broker.dispatch({op:"do.alarm.set",cls:"Counter",id:"one",at:Date.now()+60_000});
+    expect(inspect.query<{count:number},[]>("SELECT count(*) AS count FROM do_alarm_delivery").get()!.count).toBe(0);
+    expect((await broker.dispatch({op:"do.alarm.get",cls:"Counter",id:"one"})).at).toBeGreaterThan(Date.now());
+    inspect.query("INSERT INTO do_alarm_delivery VALUES ('Counter','one',1,1,'active',9999999999999)").run();
+    expect((await broker.dispatch({op:"do.alarm.delete",cls:"Counter",id:"one"})).deleted).toBe(true);
+    expect(inspect.query<{count:number},[]>("SELECT count(*) AS count FROM do_alarm_delivery").get()!.count).toBe(0);
+    expect((await broker.dispatch({op:"do.alarm.get",cls:"Counter",id:"one"})).at).toBeNull();
+  } finally {inspect.close();broker.close();rmSync(root,{recursive:true,force:true});}
+});
+
+test("oversized queue acknowledgement bodies are cancelled and retried", async () => {
+  let cancelled=false;
+  const delivered=Promise.withResolvers<void>();
+  const root=mkdtempSync(join(tmpdir(),"sb-ack-bound-"));
+  const path=join(root,"state.sqlite");
+  const broker=createBroker({db:path,bindings:{queues:["JOBS"]},sproutUrl:"http://127.0.0.1:1/",fetchImpl:async()=>{
+    delivered.resolve();
+    return new Response(new ReadableStream({pull(controller){controller.enqueue(new Uint8Array(65_536));},cancel(){cancelled=true;}}));
+  }});
+  const inspect=new Database(path);
+  try {
+    await broker.dispatch({op:"queue.send",queue:"JOBS",body:"work"});
+    await delivered.promise;
+    await Bun.sleep(40);
+    expect(cancelled).toBe(true);
+    const row=inspect.query<{attempts:number;claim_token:string},[]>("SELECT attempts,claim_token FROM mq").get()!;
+    expect(row.attempts).toBe(1);
+    expect(row.claim_token).toBe("");
+  } finally {inspect.close();broker.close();rmSync(root,{recursive:true,force:true});}
+},3_000);
+
 test("#233: a KV value put as bytes is flagged binary and survives an existing store", async () => {
   const b = make({ bindings: { kv: ["CACHE"] } });
   // One char per byte, as the runtime sends a put of an ArrayBuffer/view.
@@ -1251,15 +1319,30 @@ test("v1: a binary static asset round-trips without UTF-8 replacement", async ()
   writeFileSync(
     join(dir, "assets.json"),
     JSON.stringify({
+      headers: [{ pattern: "/*", set: [["X-Test", "asset"]], unset: ["ETag"] }],
+      redirects: [{ from: "/old", to: "/new", status: 301 }],
       notFound: "none",
       runSproutFirst: false,
-      files: { "/fixture.bin": { type: "application/octet-stream", hash: "fixture" } },
+      files: {
+        "/fixture.bin": { type: "application/octet-stream", hash: "fixture" },
+        "/old": { type: "text/html", hash: "missing-file" },
+      },
     }),
   );
   const b = make({ bindings: { assets: "ASSETS" }, assetsDir: assets, token: "tok" });
   const server = listen(b, "127.0.0.1", 0);
   const got = await v1(server, { v: 1, token: "tok", op: "assets.get", path: "/fixture.bin" });
   expect(got.json.found).toBe(true);
+  expect(got.json.rules).toEqual({
+    headers: [{ pattern: "/*", set: [["X-Test", "asset"]], unset: ["ETag"] }],
+    redirects: [{ from: "/old", to: "/new", status: 301 }],
+  });
+  const missing = await v1(server, { v: 1, token: "tok", op: "assets.get", path: "/old" });
+  expect(missing.json.found).toBe(false);
+  expect(missing.json.status).toBe(301);
+  expect(missing.json.location).toBe("/new");
+  expect(missing.bytes.length).toBe(0);
+  expect(missing.json.rules).toEqual(got.json.rules);
   expect(Buffer.from(got.bytes).equals(Buffer.from(body))).toBe(true);
   server.stop();
   b.close();
@@ -1345,3 +1428,66 @@ test("ratelimit: fixed window per key, limit/period from the message (#69)", asy
     "not bound",
   );
 });
+
+test("TCP: large replies survive backpressure and stay ordered with a pipelined request", async () => {
+  const bytes = Buffer.alloc(8 * 1024 * 1024);
+  for (let i = 0; i < bytes.length; i++) bytes[i] = i % 256;
+  const first = frameOf(encodeV1({ ok: true, id: 1 }, bytes));
+  const second = encodeFrame({ ok: true, id: 2 });
+  let calls = 0;
+  const broker = make();
+  const server = listen({ ...broker, async handleFrame() {
+    if (calls++ === 0) { await Bun.sleep(10); return first; }
+    return second;
+  } }, "127.0.0.1", 0);
+  const socket = createConnection({ host: "127.0.0.1", port: server.port });
+  let resume: ReturnType<typeof setTimeout> | undefined;
+  let next: ReturnType<typeof setTimeout> | undefined;
+  let timeout: ReturnType<typeof setTimeout> | undefined;
+  try {
+    const received = await new Promise<Buffer>((done, reject) => {
+      const chunks: Buffer[] = [];
+      let size = 0;
+      timeout = setTimeout(() => reject(new Error("large broker reply stalled or truncated")), 3000);
+      socket.on("error", reject);
+      socket.on("data", (chunk: Buffer) => {
+        chunks.push(chunk); size += chunk.length;
+        if (size >= first.length + second.length) done(Buffer.concat(chunks));
+      });
+      socket.on("connect", () => {
+        socket.pause();
+        socket.write(frameOf(Buffer.from("first")));
+        next = setTimeout(() => socket.write(frameOf(Buffer.from("second"))), 2);
+        resume = setTimeout(() => socket.resume(), 50);
+      });
+    });
+    expect(received.length).toBe(first.length + second.length);
+    expect(received.subarray(0, first.length).equals(first)).toBe(true);
+    expect(received.subarray(first.length).equals(second)).toBe(true);
+    expect(calls).toBe(2);
+  } finally {
+    clearTimeout(timeout); clearTimeout(next); clearTimeout(resume);
+    socket.destroy(); server.stop();
+  }
+}, 10_000);
+
+test("R2 direct download: If-None-Match supports wildcard, lists and weak comparison", async () => {
+  const dir = mkdtempSync(join(tmpdir(), "sb-transfer-condition-"));
+  const broker = make({ db: join(dir, "state.sqlite"), bindings: { r2: ["UPLOADS"] } });
+  try {
+    const put = await broker.dispatch({ op: "r2.put", bucket: "UPLOADS", key: "object", body: "content" });
+    const etag = `"${String(obj(put.object).etag)}"`;
+    const ticket = await broker.dispatch({ op: "r2.transfer.create", bucket: "UPLOADS", key: "object", method: "download" });
+    const url = `http://127.0.0.1${ticket.url}`;
+    for (const value of ["*", etag, `W/${etag}`, `"other", W/${etag}`]) {
+      const response = await broker.transfer(new Request(url, { headers: { "if-none-match": value } }));
+      expect(response.status).toBe(304);
+      expect(response.headers.get("etag")).toBe(etag);
+      expect(await response.text()).toBe("");
+    }
+    const fresh = await broker.transfer(new Request(url, { headers: { "if-none-match": '"different"' } }));
+    expect(fresh.status).toBe(200);
+    expect(await fresh.text()).toBe("content");
+    expect((await broker.transfer(new Request(url))).status).toBe(410);
+  } finally { broker.close(); rmSync(dir, { recursive: true, force: true }); }
+});