@sproutboat/runtime 0.24.0 → 0.26.0

package/CHANGELOG.md

@@ -1,5 +1,56 @@
 # @sproutboat/runtime
 
+## 0.26.0
+
+### Minor Changes
+
+- 9b15539: KV matches workerd from compatibility date `2026-10-25`:
+  
+  - `get`, `getWithMetadata`, `put`, `delete` and `list` return Promises, and errors reject them.
+  - `get(key, type)` takes `"text"`, `"json"`, `"arrayBuffer"` and `"stream"`, and `get([keys])` returns a `Map`.
+  - `getWithMetadata(key, type)` returns `{ value, metadata, cacheStatus }`.
+  - `put(key, value, { expiration, expirationTtl, metadata })` takes strings, bytes and `ReadableStream`s, and stores the metadata.
+  - `list({ prefix, limit, cursor })` returns `{ keys: [{ name, expiration?, metadata? }], list_complete, cursor?, cacheStatus }` and pages at up to 1,000 keys.
+  - Keys, values, expirations, metadata and list limits fail with KV's own errors. For example: `Key name cannot be empty.`, `KV PUT failed: 414 UTF-8 encoded length of 513 exceeds key length limit of 512.`, `KV PUT failed: 400 Invalid expiration_ttl of 30. Expiration TTL must be at least 60.`, and `KV PUT failed: 413 Metadata length of 1110 exceeds limit of 1024.`
+  
+  Earlier dates keep the synchronous shim. Two changes apply at every date:
+  
+  - **A `list()` prefix now matches exactly.** It used SQL `LIKE`, so it ignored case and read `_` and `%` as wildcards. For example, the prefix `a_` also listed `ab`, and `a` also listed `AB`.
+  - **Both transports store KV metadata in a new column.** It's added to existing stores when they open.
+
+### Patch Changes
+
+- 1c36119: Create the LTX page-index scratch file inside the database's replication spool
+  instead of libc's global temporary directory, fixing capture initialization when
+  a worker's filesystem sandbox permits writes only beneath its own state directory.
+  The scratch file is private and immediately unlinked. If a kill lands before
+  unlink, the next capture-lock owner removes the stale regular index file before
+  recovery. After unlink, closing the writer or killing the process releases it.
+  Its anonymous transient disk use is not counted by the
+  directory-walk quota. Capture retains its existing explicit opt-in and compatibility
+  date of 2026-10-22; capture-off apps and earlier dates keep their existing behavior.
+- 3cad1f0: The storage quota counts a D1 database's WAL file, which can hold space the app has already freed. While the app is over its quota, the runtime now checkpoints and truncates each D1 database's WAL before measuring again, at most once a second. So `DELETE` then `VACUUM` is enough to get back under the quota, and the app no longer has to run a checkpoint itself. Databases under LTX capture are left alone: capture owns their checkpoints, and they stay frozen until the quota is raised or other data is deleted.
+
+## 0.25.0
+
+### Minor Changes
+
+- ef40955: The storage quota is a hard bound when LTX capture is on. A D1 database under capture can't be capped by pinning its page count: a same-size `UPDATE`, or a `DELETE`, still adds a file to the replication spool, and the spool counts toward the app's quota. So while the app is over its quota, every write to a captured database is refused with `D1_ERROR: storage quota exceeded: writes are refused while this app's data, including its replication log, is over its quota: SQLITE_FULL`. That covers `exec()` and `batch()` too.
+  
+  - Reads, read-only batches and transaction control still run.
+  - No LTX file is ever dropped to make room: the policy fails closed.
+  - Writes resume once the app is under its quota again, at the next rescan: for example after the operator raises the quota, or the app deletes R2 or KV data. The freeze holds across a restart.
+  
+  This freezes statements after the quota scan; it is not a hard filesystem cap. The write that crosses the quota is accepted.
+  
+  Databases without capture are unchanged: they are pinned at their size, so deletes still work.
+- 95f6bd7: A storage quota for standalone sprouts and sproutd. An app's data directory (KV, D1, R2, Durable Object storage, queues) holds at most `SB_STORAGE_MAX_BYTES` bytes; sproutd sets it per app, and it defaults to 10 GiB. Each D1 database is also capped at 10 GB, as in D1. Past the quota:
+  
+  - writes fail with an error that says so, such as `KV put() failed: storage quota exceeded (this app stores N bytes; the limit is M)`, or `D1_ERROR: database or disk is full: SQLITE_FULL` for D1;
+  - reads and deletes keep working, so an app can make room (for D1, `DELETE` and then `VACUUM`).
+  
+  This is a check after a scan, not a hard filesystem cap with zero overshoot. The size is measured at most once a second, by walking the directory, and a write is checked against the last measurement. The write that crosses the limit is accepted (an R2 object, say), and so is anything else written within that second. Writes after it are refused.
+
 ## 0.24.0
 
 ### Minor Changes

package/LTX_REPLICATION.md

@@ -53,7 +53,18 @@ for this opted-in binding. Use the base LTX file for replication.
 
 Files live under `<database-path>.ltx/epoch-XXXXXX/`. Completed names are
 `<minTXID as 16 hex digits>-<maxTXID as 16 hex digits>.ltx`.
-The encoder streams one page at a time and spools index records with `tmpfile()`.
+The encoder streams one page at a time. It creates its index scratch file with
+`mkstemp()` beneath `<database-path>.ltx/`, immediately unlinks the name, and uses
+the open descriptor until the writer closes. This needs no access to a global
+temporary directory, which a sandboxed worker may not have. The scratch file is
+private (0600). A kill between creation and unlink can leave a named index file.
+On startup, after acquiring the capture lock and before recovery, the owner removes
+only regular root-level files matching the reserved `index-` plus six alphanumeric
+characters. It does not follow symlinks or remove directories or unrelated names.
+After unlink, process death releases the anonymous scratch space automatically.
+Its transient disk usage grows with the index records and is not visible to the
+directory-walk storage quota after unlink. Allow disk headroom for that scratch
+space as well as the LTX output; the quota is not a hard filesystem limit.
 
 For every LTX file:
 
@@ -78,7 +89,10 @@ applications must use idempotent transaction identifiers for retries.
 ## Qualification status
 
 Capture is tested against the exact inline C through `bun run test:native-ltx`,
-including independent BigInt CRC and index verification, a 102-file chain,
+with global `tmpfile()` access deliberately denied in the driver so the capture
+path cannot depend on it. This portable regression does not itself qualify the
+Linux Landlock/seccomp sandbox; sproutd's capture-on lifecycle smoke must do that.
+The tests cover independent BigInt CRC and index verification, a 102-file chain,
 byte-exact restore, integrity checking, oversized transaction rollback, database truncation, explicit rollback and late position-write failure latching.
 The workspace prototype additionally tests the compiled Porffor executable,
 including external writer exclusion and invariant PRAGMA rejection.
@@ -103,8 +117,9 @@ time. It also checks the last LTX file. Five native restarts each at 4.2 MB and
 The 51.1 MB verification took 399 to 446 ms on macOS arm64 (plus about 13 ms to
 health readiness); this is not a 9 ms cold-start claim for large databases.
 
-The regression SIGKILLs at ten durability boundaries, including both sides of
-clean-marker publication, and verifies matching-prefix resume and fallback cases.
+The regression SIGKILLs at eleven durability boundaries, including the interval
+between index creation and unlink and both sides of clean-marker publication,
+and verifies scratch cleanup, matching-prefix resume and fallback cases.
 It also recovers a committed tail beyond the durable cursor. The full Porffor executable passed 200 seeded random SIGKILL iterations on macOS
 arm64: 1,675 acknowledged writes retained, 88 same-generation resumes and 112
 fresh-base recoveries. Every finished chain passed file/database CRC verification,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@sproutboat/runtime",
-  "version": "0.24.0",
+  "version": "0.26.0",
   "license": "MIT",
   "repository": {
     "type": "git",

package/src/kv-shim.test.js

@@ -0,0 +1,124 @@
+// The KV shim (`env.KV`) at compatibility dates before and from 2026-10-25, over an
+// in-memory transport with the kv.* contract both transports implement (broker.test.ts
+// covers the broker's side; sproutd's conformance/bindings/kv.js the whole path).
+import { expect, test } from "bun:test";
+import { readFileSync } from "node:fs";
+
+const source = readFileSync(new URL("./native-fetch-prelude.js", import.meta.url), "utf8");
+let rows;
+const call = (json) => {
+  const msg = JSON.parse(json);
+  const reply = (() => {
+    if (msg.op === "kv.get") {
+      const r = rows.get(msg.key);
+      return r ? { ok: true, found: true, value: r.value, binary: r.binary, metadata: r.metadata ?? null, expiration: r.expiration ?? null } : { ok: true, found: false, value: null };
+    }
+    if (msg.op === "kv.put") {
+      rows.set(msg.key, { value: msg.value, binary: msg.binary === true, metadata: msg.metadata, expiration: msg.expiration ?? (msg.expirationTtl ? Math.floor(Date.now() / 1000) + msg.expirationTtl : undefined) });
+      return { ok: true };
+    }
+    if (msg.op === "kv.delete") {
+      rows.delete(msg.key);
+      return { ok: true };
+    }
+    if (msg.op === "kv.list") {
+      const keys = [...rows.keys()].filter((k) => k.startsWith(msg.prefix) && (!msg.cursor || k > msg.cursor)).sort();
+      const limit = msg.limit ?? keys.length;
+      const page = keys.slice(0, limit);
+      const complete = keys.length <= limit;
+      return {
+        ok: true,
+        keys: page,
+        entries: page.map((k) => ({ name: k, expiration: rows.get(k).expiration ?? null, metadata: rows.get(k).metadata ?? null })),
+        list_complete: complete,
+        cursor: complete ? "" : page[page.length - 1],
+      };
+    }
+    return { ok: false, error: "unknown op" };
+  })();
+  return JSON.stringify(reply);
+};
+// oxlint-disable-next-line no-function-constructor -- the prelude is prepended as text in production too
+const { makeEnv } = new Function(
+  "Porffor",
+  "__testCall",
+  `${source}\n__sbCall = __testCall; return { makeEnv: (target) => { __sbInstallBindings(target, { kv: ["KV"] }); return target; } };`,
+)({ c: () => "" }, call);
+const fresh = (date) => {
+  globalThis.__sbCompat = date;
+  rows = new Map();
+  return makeEnv({}).KV;
+};
+const failure = async (fn) => {
+  try {
+    await fn();
+  } catch (error) {
+    return `${error.name}: ${error.message}`;
+  }
+  return "did not throw";
+};
+
+test("before 2026-10-25: the synchronous shim, unchanged", () => {
+  const KV = fresh("2026-10-24");
+  expect(KV.put("a", "v")).toBeUndefined();
+  expect(KV.get("a")).toBe("v");
+  expect(KV.list("a")).toEqual(["a"]);
+  expect(KV.getWithMetadata).toBeUndefined();
+});
+
+test("from 2026-10-25: Promises, metadata, typed gets and get(keys)", async () => {
+  const KV = fresh("2026-10-25");
+  expect(KV.put("a", "v", { metadata: { n: 1 } })).toBeInstanceOf(Promise);
+  await KV.put("a", '{"x":1}', { metadata: { n: 1 } });
+  expect(await KV.get("a", "json")).toEqual({ x: 1 });
+  expect(await KV.get("a", { type: "text" })).toBe('{"x":1}');
+  expect(await KV.getWithMetadata("a", "json")).toEqual({ value: { x: 1 }, metadata: { n: 1 }, cacheStatus: null });
+  expect(await KV.getWithMetadata("missing")).toEqual({ value: null, metadata: null, cacheStatus: null });
+  await KV.put("b", new Uint8Array([0, 255]));
+  expect(Array.from(new Uint8Array(await KV.get("b", "arrayBuffer")))).toEqual([0, 255]);
+  // (ReadableStream values and the "stream" type use the prelude's own streams: checked end to
+  // end by sproutd's conformance/bindings/kv.js, not here against Bun's.)
+  const many = await KV.get(["a", "missing"]);
+  expect(many).toBeInstanceOf(Map);
+  expect([...many.entries()]).toEqual([["a", '{"x":1}'], ["missing", null]]);
+  expect(await KV.delete("a")).toBeUndefined();
+  expect(await KV.get("a")).toBeNull();
+});
+
+test("from 2026-10-25: KV's errors for keys, values, expiration and metadata", async () => {
+  const KV = fresh("2026-10-25");
+  expect(await failure(() => KV.put("", "v"))).toBe("TypeError: Key name cannot be empty.");
+  expect(await failure(() => KV.put("..", "v"))).toBe('TypeError: ".." is not allowed as a key name.');
+  expect(await failure(() => KV.put("k".repeat(513), "v"))).toBe("Error: KV PUT failed: 414 UTF-8 encoded length of 513 exceeds key length limit of 512.");
+  expect(await failure(() => KV.get("k".repeat(513)))).toBe("Error: KV GET failed: 414 UTF-8 encoded length of 513 exceeds key length limit of 512.");
+  expect(await failure(() => KV.put("a", { a: 1 }))).toBe("TypeError: KV put() accepts only strings, ArrayBuffers, ArrayBufferViews, and ReadableStreams as values.");
+  expect(await failure(() => KV.put("a", "v", { expirationTtl: 30 }))).toBe("Error: KV PUT failed: 400 Invalid expiration_ttl of 30. Expiration TTL must be at least 60.");
+  expect(await failure(() => KV.put("a", "v", { expirationTtl: "soon" }))).toBe("Error: KV PUT failed: 400 Invalid expiration_ttl of 0. Please specify integer greater than 0.");
+  expect(await failure(() => KV.put("a", "v", { expiration: 1000 }))).toBe(
+    "Error: KV PUT failed: 400 Invalid expiration of 1000. Please specify integer greater than the current number of seconds since the UNIX epoch.",
+  );
+  expect(await failure(() => KV.put("a", "v", { metadata: { big: "x".repeat(1100) } }))).toBe("Error: KV PUT failed: 413 Metadata length of 1110 exceeds limit of 1024.");
+  expect(await KV.get("missing", "nope")).toBeNull();
+  await KV.put("there", "v");
+  expect(await failure(() => KV.get("there", "nope"))).toBe('TypeError: Unknown response type. Possible types are "text", "arrayBuffer", "json", and "stream".');
+  expect(await failure(() => KV.list({ limit: 1001 }))).toBe("Error: KV GET failed: 400 Invalid key_count_limit of 1001. Please specify an integer less than 1000.");
+  // A failing get rejects rather than throws.
+  const rejected = KV.get("");
+  expect(rejected).toBeInstanceOf(Promise);
+  expect(await rejected.catch((e) => e.message)).toBe("Key name cannot be empty.");
+});
+
+test("from 2026-10-25: list() pages with limit and cursor, with expiration and metadata", async () => {
+  const KV = fresh("2026-10-25");
+  for (const k of ["b", "a", "c"]) await KV.put(k, "v", k === "a" ? { metadata: { t: 1 }, expirationTtl: 3600 } : undefined);
+  const first = await KV.list({ limit: 2 });
+  expect(first.keys.map((k) => k.name)).toEqual(["a", "b"]);
+  expect(first.keys[0].metadata).toEqual({ t: 1 });
+  expect(Number.isInteger(first.keys[0].expiration)).toBe(true);
+  expect("expiration" in first.keys[1]).toBe(false);
+  expect(first.list_complete).toBe(false);
+  const rest = await KV.list({ limit: 2, cursor: first.cursor });
+  expect(rest.keys.map((k) => k.name)).toEqual(["c"]);
+  expect(Object.keys(rest).sort()).toEqual(["cacheStatus", "keys", "list_complete"]);
+  expect(await KV.list({ limit: 0 })).toEqual({ keys: [], list_complete: true, cacheStatus: null });
+});

package/src/native-fetch-prelude.js

@@ -4807,6 +4807,126 @@ function __sbRpc(op, extra) {
   return reply;
 }
 
+// KV from compatibility date 2026-10-25: workerd's surface. Every method returns a
+// Promise; keys and options are checked with KV's errors; get() takes a list of keys and
+// the "stream" type; getWithMetadata(); put() takes a ReadableStream and metadata; list()
+// pages with limit and cursor and returns each key's expiration and metadata. Earlier
+// dates keep the synchronous shim below.
+function __sbKvStrict() {
+  return globalThis.__sbCompat >= "2026-10-25";
+}
+function __sbKvKey(key, verb) {
+  const name = String(key);
+  if (name === "") throw new TypeError("Key name cannot be empty.");
+  if (name === "." || name === "..") throw new TypeError('"' + name + '" is not allowed as a key name.');
+  const length = new TextEncoder().encode(name).length;
+  if (length > 512) throw new Error("KV " + verb + " failed: 414 UTF-8 encoded length of " + length + " exceeds key length limit of 512.");
+  return name;
+}
+function __sbKvType(typeOrOptions) {
+  return typeOrOptions == null ? "text" : __sbIsStr(typeOrOptions) ? typeOrOptions : typeOrOptions.type || "text";
+}
+// An unknown type fails only when there is a value to decode: a missing key is null
+// whatever the type, as in workerd.
+function __sbKvDecode(r, type) {
+  if (type !== "text" && type !== "json" && type !== "arrayBuffer" && type !== "stream")
+    throw new TypeError('Unknown response type. Possible types are "text", "arrayBuffer", "json", and "stream".');
+  if (type === "arrayBuffer") return __sbBufFrom(r.binary ? r.value : __sbToBytes(r.value));
+  if (type === "stream") return new Response(__sbBufFrom(r.binary ? r.value : __sbToBytes(r.value))).body;
+  const text = r.binary ? __sbFromUtf8(r.value) : r.value;
+  return type === "json" ? JSON.parse(text) : text;
+}
+function __sbKvMetadata(text) {
+  return text == null ? null : JSON.parse(text);
+}
+function __sbMakeKv(ns) {
+  const one = (key, type) => {
+    const r = __sbRpc("kv.get", { ns, key: __sbKvKey(key, "GET") });
+    return r.found ? { value: __sbKvDecode(r, type), metadata: __sbKvMetadata(r.metadata) } : { value: null, metadata: null };
+  };
+  const attempt = (run) => {
+    try {
+      return Promise.resolve(run());
+    } catch (err) {
+      return Promise.reject(err);
+    }
+  };
+  return {
+    get(key, typeOrOptions) {
+      return attempt(() => {
+        const type = __sbKvType(typeOrOptions);
+        if (!Array.isArray(key)) return one(key, type).value;
+        const out = new Map();
+        for (let i = 0; i < key.length; i++) out.set(String(key[i]), one(key[i], type).value);
+        return out;
+      });
+    },
+    getWithMetadata(key, typeOrOptions) {
+      return attempt(() => {
+        const r = one(key, __sbKvType(typeOrOptions));
+        return { value: r.value, metadata: r.metadata, cacheStatus: null };
+      });
+    },
+    async put(key, value, options) {
+      const msg = { ns, key: __sbKvKey(key, "PUT"), value: "" };
+      value = __sbUnwrapBody(value);
+      if (value != null && __sbIsFn(value.getReader)) value = new Uint8Array(await new Response(value).arrayBuffer());
+      if (value != null && !__sbIsStr(value) && value.byteLength !== undefined) {
+        const bytes = value.buffer !== undefined ? new Uint8Array(value.buffer, value.byteOffset, value.byteLength) : new Uint8Array(value);
+        msg.value = __sbToBytes(bytes);
+        msg.binary = true;
+      } else if (__sbIsStr(value) || value === Number(value) || value === true || value === false) msg.value = String(value);
+      else throw new TypeError("KV put() accepts only strings, ArrayBuffers, ArrayBufferViews, and ReadableStreams as values.");
+      const o = options || {};
+      if (o.expirationTtl != null) {
+        const ttl = Math.floor(Number(o.expirationTtl)) || 0;
+        if (ttl <= 0) throw new Error("KV PUT failed: 400 Invalid expiration_ttl of " + ttl + ". Please specify integer greater than 0.");
+        if (ttl < 60) throw new Error("KV PUT failed: 400 Invalid expiration_ttl of " + ttl + ". Expiration TTL must be at least 60.");
+        msg.expirationTtl = ttl;
+      } else if (o.expiration != null) {
+        const at = Math.floor(Number(o.expiration)) || 0;
+        if (at <= Math.floor(Date.now() / 1000))
+          throw new Error("KV PUT failed: 400 Invalid expiration of " + at + ". Please specify integer greater than the current number of seconds since the UNIX epoch.");
+        msg.expiration = at;
+      }
+      if (o.metadata !== undefined) {
+        const json = JSON.stringify(o.metadata);
+        const length = new TextEncoder().encode(json).length;
+        if (length > 1024) throw new Error("KV PUT failed: 413 Metadata length of " + length + " exceeds limit of 1024.");
+        msg.metadata = json;
+      }
+      __sbRpc("kv.put", msg);
+    },
+    delete(key) {
+      return attempt(() => {
+        __sbRpc("kv.delete", { ns, key: __sbKvKey(key, "DELETE") });
+      });
+    },
+    list(options) {
+      return attempt(() => {
+        const o = options || {};
+        const limit = o.limit == null ? 1000 : Math.floor(Number(o.limit));
+        if (!(limit >= 0) || limit > 1000)
+          throw new Error("KV GET failed: 400 Invalid key_count_limit of " + o.limit + ". Please specify an integer less than 1000.");
+        if (limit === 0) return { keys: [], list_complete: true, cacheStatus: null };
+        const r = __sbRpc("kv.list", { ns, prefix: o.prefix == null ? "" : String(o.prefix), limit, cursor: o.cursor == null ? "" : String(o.cursor) });
+        const keys = [];
+        const entries = r.entries || [];
+        for (let i = 0; i < entries.length; i++) {
+          const k = { name: entries[i].name };
+          if (entries[i].expiration != null) k.expiration = entries[i].expiration;
+          if (entries[i].metadata != null) k.metadata = __sbKvMetadata(entries[i].metadata);
+          keys.push(k);
+        }
+        const out = { keys, list_complete: r.list_complete !== false };
+        if (!out.list_complete) out.cursor = r.cursor;
+        out.cacheStatus = null;
+        return out;
+      });
+    },
+  };
+}
+
 // D1: a Cloudflare-shaped `env.<DB>` (prepare / bind / all / run / raw / first,
 // plus batch and exec). Every call is one broker roundtrip.
 //
@@ -5185,7 +5305,7 @@ globalThis.__sbInstallBindings = function (target, bindings) {
 
   for (let i = 0; i < (bindings.kv || []).length; i++) {
     const ns = bindings.kv[i];
-    target[ns] = {
+    target[ns] = __sbKvStrict() ? __sbMakeKv(ns) : {
       // #233 — Workers' get(key, type | { type }): "text" (default), "json",
       // "arrayBuffer". A value put as bytes is stored one char per byte and
       // flagged, so text decodes it as UTF-8 and arrayBuffer returns it