@sproutboat/runtime 0.25.0 → 0.26.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,36 @@
1
1
  # @sproutboat/runtime
2
2
 
3
+ ## 0.26.0
4
+
5
+ ### Minor Changes
6
+
7
+ - 9b15539: KV matches workerd from compatibility date `2026-10-25`:
8
+
9
+ - `get`, `getWithMetadata`, `put`, `delete` and `list` return Promises, and errors reject them.
10
+ - `get(key, type)` takes `"text"`, `"json"`, `"arrayBuffer"` and `"stream"`, and `get([keys])` returns a `Map`.
11
+ - `getWithMetadata(key, type)` returns `{ value, metadata, cacheStatus }`.
12
+ - `put(key, value, { expiration, expirationTtl, metadata })` takes strings, bytes and `ReadableStream`s, and stores the metadata.
13
+ - `list({ prefix, limit, cursor })` returns `{ keys: [{ name, expiration?, metadata? }], list_complete, cursor?, cacheStatus }` and pages at up to 1,000 keys.
14
+ - Keys, values, expirations, metadata and list limits fail with KV's own errors. For example: `Key name cannot be empty.`, `KV PUT failed: 414 UTF-8 encoded length of 513 exceeds key length limit of 512.`, `KV PUT failed: 400 Invalid expiration_ttl of 30. Expiration TTL must be at least 60.`, and `KV PUT failed: 413 Metadata length of 1110 exceeds limit of 1024.`
15
+
16
+ Earlier dates keep the synchronous shim. Two changes apply at every date:
17
+
18
+ - **A `list()` prefix now matches exactly.** It used SQL `LIKE`, so it ignored case and read `_` and `%` as wildcards. For example, the prefix `a_` also listed `ab`, and `a` also listed `AB`.
19
+ - **Both transports store KV metadata in a new column.** It's added to existing stores when they open.
20
+
21
+ ### Patch Changes
22
+
23
+ - 1c36119: Create the LTX page-index scratch file inside the database's replication spool
24
+ instead of libc's global temporary directory, fixing capture initialization when
25
+ a worker's filesystem sandbox permits writes only beneath its own state directory.
26
+ The scratch file is private and immediately unlinked. If a kill lands before
27
+ unlink, the next capture-lock owner removes the stale regular index file before
28
+ recovery. After unlink, closing the writer or killing the process releases it.
29
+ Its anonymous transient disk use is not counted by the
30
+ directory-walk quota. Capture retains its existing explicit opt-in and compatibility
31
+ date of 2026-10-22; capture-off apps and earlier dates keep their existing behavior.
32
+ - 3cad1f0: The storage quota counts a D1 database's WAL file, which can hold space the app has already freed. While the app is over its quota, the runtime now checkpoints and truncates each D1 database's WAL before measuring again, at most once a second. So `DELETE` then `VACUUM` is enough to get back under the quota, and the app no longer has to run a checkpoint itself. Databases under LTX capture are left alone: capture owns their checkpoints, and they stay frozen until the quota is raised or other data is deleted.
33
+
3
34
  ## 0.25.0
4
35
 
5
36
  ### Minor Changes
@@ -53,7 +53,18 @@ for this opted-in binding. Use the base LTX file for replication.
53
53
 
54
54
  Files live under `<database-path>.ltx/epoch-XXXXXX/`. Completed names are
55
55
  `<minTXID as 16 hex digits>-<maxTXID as 16 hex digits>.ltx`.
56
- The encoder streams one page at a time and spools index records with `tmpfile()`.
56
+ The encoder streams one page at a time. It creates its index scratch file with
57
+ `mkstemp()` beneath `<database-path>.ltx/`, immediately unlinks the name, and uses
58
+ the open descriptor until the writer closes. This needs no access to a global
59
+ temporary directory, which a sandboxed worker may not have. The scratch file is
60
+ private (0600). A kill between creation and unlink can leave a named index file.
61
+ On startup, after acquiring the capture lock and before recovery, the owner removes
62
+ only regular root-level files matching the reserved `index-` plus six alphanumeric
63
+ characters. It does not follow symlinks or remove directories or unrelated names.
64
+ After unlink, process death releases the anonymous scratch space automatically.
65
+ Its transient disk usage grows with the index records and is not visible to the
66
+ directory-walk storage quota after unlink. Allow disk headroom for that scratch
67
+ space as well as the LTX output; the quota is not a hard filesystem limit.
57
68
 
58
69
  For every LTX file:
59
70
 
@@ -78,7 +89,10 @@ applications must use idempotent transaction identifiers for retries.
78
89
  ## Qualification status
79
90
 
80
91
  Capture is tested against the exact inline C through `bun run test:native-ltx`,
81
- including independent BigInt CRC and index verification, a 102-file chain,
92
+ with global `tmpfile()` access deliberately denied in the driver so the capture
93
+ path cannot depend on it. This portable regression does not itself qualify the
94
+ Linux Landlock/seccomp sandbox; sproutd's capture-on lifecycle smoke must do that.
95
+ The tests cover independent BigInt CRC and index verification, a 102-file chain,
82
96
  byte-exact restore, integrity checking, oversized transaction rollback, database truncation, explicit rollback and late position-write failure latching.
83
97
  The workspace prototype additionally tests the compiled Porffor executable,
84
98
  including external writer exclusion and invariant PRAGMA rejection.
@@ -103,8 +117,9 @@ time. It also checks the last LTX file. Five native restarts each at 4.2 MB and
103
117
  The 51.1 MB verification took 399 to 446 ms on macOS arm64 (plus about 13 ms to
104
118
  health readiness); this is not a 9 ms cold-start claim for large databases.
105
119
 
106
- The regression SIGKILLs at ten durability boundaries, including both sides of
107
- clean-marker publication, and verifies matching-prefix resume and fallback cases.
120
+ The regression SIGKILLs at eleven durability boundaries, including the interval
121
+ between index creation and unlink and both sides of clean-marker publication,
122
+ and verifies scratch cleanup, matching-prefix resume and fallback cases.
108
123
  It also recovers a committed tail beyond the durable cursor. The full Porffor executable passed 200 seeded random SIGKILL iterations on macOS
109
124
  arm64: 1,675 acknowledged writes retained, 88 same-generation resumes and 112
110
125
  fresh-base recoveries. Every finished chain passed file/database CRC verification,
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@sproutboat/runtime",
3
- "version": "0.25.0",
3
+ "version": "0.26.0",
4
4
  "license": "MIT",
5
5
  "repository": {
6
6
  "type": "git",
@@ -0,0 +1,124 @@
1
+ // The KV shim (`env.KV`) at compatibility dates before and from 2026-10-25, over an
2
+ // in-memory transport with the kv.* contract both transports implement (broker.test.ts
3
+ // covers the broker's side; sproutd's conformance/bindings/kv.js the whole path).
4
+ import { expect, test } from "bun:test";
5
+ import { readFileSync } from "node:fs";
6
+
7
+ const source = readFileSync(new URL("./native-fetch-prelude.js", import.meta.url), "utf8");
8
+ let rows;
9
+ const call = (json) => {
10
+ const msg = JSON.parse(json);
11
+ const reply = (() => {
12
+ if (msg.op === "kv.get") {
13
+ const r = rows.get(msg.key);
14
+ return r ? { ok: true, found: true, value: r.value, binary: r.binary, metadata: r.metadata ?? null, expiration: r.expiration ?? null } : { ok: true, found: false, value: null };
15
+ }
16
+ if (msg.op === "kv.put") {
17
+ rows.set(msg.key, { value: msg.value, binary: msg.binary === true, metadata: msg.metadata, expiration: msg.expiration ?? (msg.expirationTtl ? Math.floor(Date.now() / 1000) + msg.expirationTtl : undefined) });
18
+ return { ok: true };
19
+ }
20
+ if (msg.op === "kv.delete") {
21
+ rows.delete(msg.key);
22
+ return { ok: true };
23
+ }
24
+ if (msg.op === "kv.list") {
25
+ const keys = [...rows.keys()].filter((k) => k.startsWith(msg.prefix) && (!msg.cursor || k > msg.cursor)).sort();
26
+ const limit = msg.limit ?? keys.length;
27
+ const page = keys.slice(0, limit);
28
+ const complete = keys.length <= limit;
29
+ return {
30
+ ok: true,
31
+ keys: page,
32
+ entries: page.map((k) => ({ name: k, expiration: rows.get(k).expiration ?? null, metadata: rows.get(k).metadata ?? null })),
33
+ list_complete: complete,
34
+ cursor: complete ? "" : page[page.length - 1],
35
+ };
36
+ }
37
+ return { ok: false, error: "unknown op" };
38
+ })();
39
+ return JSON.stringify(reply);
40
+ };
41
+ // oxlint-disable-next-line no-function-constructor -- the prelude is prepended as text in production too
42
+ const { makeEnv } = new Function(
43
+ "Porffor",
44
+ "__testCall",
45
+ `${source}\n__sbCall = __testCall; return { makeEnv: (target) => { __sbInstallBindings(target, { kv: ["KV"] }); return target; } };`,
46
+ )({ c: () => "" }, call);
47
+ const fresh = (date) => {
48
+ globalThis.__sbCompat = date;
49
+ rows = new Map();
50
+ return makeEnv({}).KV;
51
+ };
52
+ const failure = async (fn) => {
53
+ try {
54
+ await fn();
55
+ } catch (error) {
56
+ return `${error.name}: ${error.message}`;
57
+ }
58
+ return "did not throw";
59
+ };
60
+
61
+ test("before 2026-10-25: the synchronous shim, unchanged", () => {
62
+ const KV = fresh("2026-10-24");
63
+ expect(KV.put("a", "v")).toBeUndefined();
64
+ expect(KV.get("a")).toBe("v");
65
+ expect(KV.list("a")).toEqual(["a"]);
66
+ expect(KV.getWithMetadata).toBeUndefined();
67
+ });
68
+
69
+ test("from 2026-10-25: Promises, metadata, typed gets and get(keys)", async () => {
70
+ const KV = fresh("2026-10-25");
71
+ expect(KV.put("a", "v", { metadata: { n: 1 } })).toBeInstanceOf(Promise);
72
+ await KV.put("a", '{"x":1}', { metadata: { n: 1 } });
73
+ expect(await KV.get("a", "json")).toEqual({ x: 1 });
74
+ expect(await KV.get("a", { type: "text" })).toBe('{"x":1}');
75
+ expect(await KV.getWithMetadata("a", "json")).toEqual({ value: { x: 1 }, metadata: { n: 1 }, cacheStatus: null });
76
+ expect(await KV.getWithMetadata("missing")).toEqual({ value: null, metadata: null, cacheStatus: null });
77
+ await KV.put("b", new Uint8Array([0, 255]));
78
+ expect(Array.from(new Uint8Array(await KV.get("b", "arrayBuffer")))).toEqual([0, 255]);
79
+ // (ReadableStream values and the "stream" type use the prelude's own streams: checked end to
80
+ // end by sproutd's conformance/bindings/kv.js, not here against Bun's.)
81
+ const many = await KV.get(["a", "missing"]);
82
+ expect(many).toBeInstanceOf(Map);
83
+ expect([...many.entries()]).toEqual([["a", '{"x":1}'], ["missing", null]]);
84
+ expect(await KV.delete("a")).toBeUndefined();
85
+ expect(await KV.get("a")).toBeNull();
86
+ });
87
+
88
+ test("from 2026-10-25: KV's errors for keys, values, expiration and metadata", async () => {
89
+ const KV = fresh("2026-10-25");
90
+ expect(await failure(() => KV.put("", "v"))).toBe("TypeError: Key name cannot be empty.");
91
+ expect(await failure(() => KV.put("..", "v"))).toBe('TypeError: ".." is not allowed as a key name.');
92
+ expect(await failure(() => KV.put("k".repeat(513), "v"))).toBe("Error: KV PUT failed: 414 UTF-8 encoded length of 513 exceeds key length limit of 512.");
93
+ expect(await failure(() => KV.get("k".repeat(513)))).toBe("Error: KV GET failed: 414 UTF-8 encoded length of 513 exceeds key length limit of 512.");
94
+ expect(await failure(() => KV.put("a", { a: 1 }))).toBe("TypeError: KV put() accepts only strings, ArrayBuffers, ArrayBufferViews, and ReadableStreams as values.");
95
+ expect(await failure(() => KV.put("a", "v", { expirationTtl: 30 }))).toBe("Error: KV PUT failed: 400 Invalid expiration_ttl of 30. Expiration TTL must be at least 60.");
96
+ expect(await failure(() => KV.put("a", "v", { expirationTtl: "soon" }))).toBe("Error: KV PUT failed: 400 Invalid expiration_ttl of 0. Please specify integer greater than 0.");
97
+ expect(await failure(() => KV.put("a", "v", { expiration: 1000 }))).toBe(
98
+ "Error: KV PUT failed: 400 Invalid expiration of 1000. Please specify integer greater than the current number of seconds since the UNIX epoch.",
99
+ );
100
+ expect(await failure(() => KV.put("a", "v", { metadata: { big: "x".repeat(1100) } }))).toBe("Error: KV PUT failed: 413 Metadata length of 1110 exceeds limit of 1024.");
101
+ expect(await KV.get("missing", "nope")).toBeNull();
102
+ await KV.put("there", "v");
103
+ expect(await failure(() => KV.get("there", "nope"))).toBe('TypeError: Unknown response type. Possible types are "text", "arrayBuffer", "json", and "stream".');
104
+ expect(await failure(() => KV.list({ limit: 1001 }))).toBe("Error: KV GET failed: 400 Invalid key_count_limit of 1001. Please specify an integer less than 1000.");
105
+ // A failing get rejects rather than throws.
106
+ const rejected = KV.get("");
107
+ expect(rejected).toBeInstanceOf(Promise);
108
+ expect(await rejected.catch((e) => e.message)).toBe("Key name cannot be empty.");
109
+ });
110
+
111
+ test("from 2026-10-25: list() pages with limit and cursor, with expiration and metadata", async () => {
112
+ const KV = fresh("2026-10-25");
113
+ for (const k of ["b", "a", "c"]) await KV.put(k, "v", k === "a" ? { metadata: { t: 1 }, expirationTtl: 3600 } : undefined);
114
+ const first = await KV.list({ limit: 2 });
115
+ expect(first.keys.map((k) => k.name)).toEqual(["a", "b"]);
116
+ expect(first.keys[0].metadata).toEqual({ t: 1 });
117
+ expect(Number.isInteger(first.keys[0].expiration)).toBe(true);
118
+ expect("expiration" in first.keys[1]).toBe(false);
119
+ expect(first.list_complete).toBe(false);
120
+ const rest = await KV.list({ limit: 2, cursor: first.cursor });
121
+ expect(rest.keys.map((k) => k.name)).toEqual(["c"]);
122
+ expect(Object.keys(rest).sort()).toEqual(["cacheStatus", "keys", "list_complete"]);
123
+ expect(await KV.list({ limit: 0 })).toEqual({ keys: [], list_complete: true, cacheStatus: null });
124
+ });
@@ -4807,6 +4807,126 @@ function __sbRpc(op, extra) {
4807
4807
  return reply;
4808
4808
  }
4809
4809
 
4810
+ // KV from compatibility date 2026-10-25: workerd's surface. Every method returns a
4811
+ // Promise; keys and options are checked with KV's errors; get() takes a list of keys and
4812
+ // the "stream" type; getWithMetadata(); put() takes a ReadableStream and metadata; list()
4813
+ // pages with limit and cursor and returns each key's expiration and metadata. Earlier
4814
+ // dates keep the synchronous shim below.
4815
+ function __sbKvStrict() {
4816
+ return globalThis.__sbCompat >= "2026-10-25";
4817
+ }
4818
+ function __sbKvKey(key, verb) {
4819
+ const name = String(key);
4820
+ if (name === "") throw new TypeError("Key name cannot be empty.");
4821
+ if (name === "." || name === "..") throw new TypeError('"' + name + '" is not allowed as a key name.');
4822
+ const length = new TextEncoder().encode(name).length;
4823
+ if (length > 512) throw new Error("KV " + verb + " failed: 414 UTF-8 encoded length of " + length + " exceeds key length limit of 512.");
4824
+ return name;
4825
+ }
4826
+ function __sbKvType(typeOrOptions) {
4827
+ return typeOrOptions == null ? "text" : __sbIsStr(typeOrOptions) ? typeOrOptions : typeOrOptions.type || "text";
4828
+ }
4829
+ // An unknown type fails only when there is a value to decode: a missing key is null
4830
+ // whatever the type, as in workerd.
4831
+ function __sbKvDecode(r, type) {
4832
+ if (type !== "text" && type !== "json" && type !== "arrayBuffer" && type !== "stream")
4833
+ throw new TypeError('Unknown response type. Possible types are "text", "arrayBuffer", "json", and "stream".');
4834
+ if (type === "arrayBuffer") return __sbBufFrom(r.binary ? r.value : __sbToBytes(r.value));
4835
+ if (type === "stream") return new Response(__sbBufFrom(r.binary ? r.value : __sbToBytes(r.value))).body;
4836
+ const text = r.binary ? __sbFromUtf8(r.value) : r.value;
4837
+ return type === "json" ? JSON.parse(text) : text;
4838
+ }
4839
+ function __sbKvMetadata(text) {
4840
+ return text == null ? null : JSON.parse(text);
4841
+ }
4842
+ function __sbMakeKv(ns) {
4843
+ const one = (key, type) => {
4844
+ const r = __sbRpc("kv.get", { ns, key: __sbKvKey(key, "GET") });
4845
+ return r.found ? { value: __sbKvDecode(r, type), metadata: __sbKvMetadata(r.metadata) } : { value: null, metadata: null };
4846
+ };
4847
+ const attempt = (run) => {
4848
+ try {
4849
+ return Promise.resolve(run());
4850
+ } catch (err) {
4851
+ return Promise.reject(err);
4852
+ }
4853
+ };
4854
+ return {
4855
+ get(key, typeOrOptions) {
4856
+ return attempt(() => {
4857
+ const type = __sbKvType(typeOrOptions);
4858
+ if (!Array.isArray(key)) return one(key, type).value;
4859
+ const out = new Map();
4860
+ for (let i = 0; i < key.length; i++) out.set(String(key[i]), one(key[i], type).value);
4861
+ return out;
4862
+ });
4863
+ },
4864
+ getWithMetadata(key, typeOrOptions) {
4865
+ return attempt(() => {
4866
+ const r = one(key, __sbKvType(typeOrOptions));
4867
+ return { value: r.value, metadata: r.metadata, cacheStatus: null };
4868
+ });
4869
+ },
4870
+ async put(key, value, options) {
4871
+ const msg = { ns, key: __sbKvKey(key, "PUT"), value: "" };
4872
+ value = __sbUnwrapBody(value);
4873
+ if (value != null && __sbIsFn(value.getReader)) value = new Uint8Array(await new Response(value).arrayBuffer());
4874
+ if (value != null && !__sbIsStr(value) && value.byteLength !== undefined) {
4875
+ const bytes = value.buffer !== undefined ? new Uint8Array(value.buffer, value.byteOffset, value.byteLength) : new Uint8Array(value);
4876
+ msg.value = __sbToBytes(bytes);
4877
+ msg.binary = true;
4878
+ } else if (__sbIsStr(value) || value === Number(value) || value === true || value === false) msg.value = String(value);
4879
+ else throw new TypeError("KV put() accepts only strings, ArrayBuffers, ArrayBufferViews, and ReadableStreams as values.");
4880
+ const o = options || {};
4881
+ if (o.expirationTtl != null) {
4882
+ const ttl = Math.floor(Number(o.expirationTtl)) || 0;
4883
+ if (ttl <= 0) throw new Error("KV PUT failed: 400 Invalid expiration_ttl of " + ttl + ". Please specify integer greater than 0.");
4884
+ if (ttl < 60) throw new Error("KV PUT failed: 400 Invalid expiration_ttl of " + ttl + ". Expiration TTL must be at least 60.");
4885
+ msg.expirationTtl = ttl;
4886
+ } else if (o.expiration != null) {
4887
+ const at = Math.floor(Number(o.expiration)) || 0;
4888
+ if (at <= Math.floor(Date.now() / 1000))
4889
+ throw new Error("KV PUT failed: 400 Invalid expiration of " + at + ". Please specify integer greater than the current number of seconds since the UNIX epoch.");
4890
+ msg.expiration = at;
4891
+ }
4892
+ if (o.metadata !== undefined) {
4893
+ const json = JSON.stringify(o.metadata);
4894
+ const length = new TextEncoder().encode(json).length;
4895
+ if (length > 1024) throw new Error("KV PUT failed: 413 Metadata length of " + length + " exceeds limit of 1024.");
4896
+ msg.metadata = json;
4897
+ }
4898
+ __sbRpc("kv.put", msg);
4899
+ },
4900
+ delete(key) {
4901
+ return attempt(() => {
4902
+ __sbRpc("kv.delete", { ns, key: __sbKvKey(key, "DELETE") });
4903
+ });
4904
+ },
4905
+ list(options) {
4906
+ return attempt(() => {
4907
+ const o = options || {};
4908
+ const limit = o.limit == null ? 1000 : Math.floor(Number(o.limit));
4909
+ if (!(limit >= 0) || limit > 1000)
4910
+ throw new Error("KV GET failed: 400 Invalid key_count_limit of " + o.limit + ". Please specify an integer less than 1000.");
4911
+ if (limit === 0) return { keys: [], list_complete: true, cacheStatus: null };
4912
+ const r = __sbRpc("kv.list", { ns, prefix: o.prefix == null ? "" : String(o.prefix), limit, cursor: o.cursor == null ? "" : String(o.cursor) });
4913
+ const keys = [];
4914
+ const entries = r.entries || [];
4915
+ for (let i = 0; i < entries.length; i++) {
4916
+ const k = { name: entries[i].name };
4917
+ if (entries[i].expiration != null) k.expiration = entries[i].expiration;
4918
+ if (entries[i].metadata != null) k.metadata = __sbKvMetadata(entries[i].metadata);
4919
+ keys.push(k);
4920
+ }
4921
+ const out = { keys, list_complete: r.list_complete !== false };
4922
+ if (!out.list_complete) out.cursor = r.cursor;
4923
+ out.cacheStatus = null;
4924
+ return out;
4925
+ });
4926
+ },
4927
+ };
4928
+ }
4929
+
4810
4930
  // D1: a Cloudflare-shaped `env.<DB>` (prepare / bind / all / run / raw / first,
4811
4931
  // plus batch and exec). Every call is one broker roundtrip.
4812
4932
  //
@@ -5185,7 +5305,7 @@ globalThis.__sbInstallBindings = function (target, bindings) {
5185
5305
 
5186
5306
  for (let i = 0; i < (bindings.kv || []).length; i++) {
5187
5307
  const ns = bindings.kv[i];
5188
- target[ns] = {
5308
+ target[ns] = __sbKvStrict() ? __sbMakeKv(ns) : {
5189
5309
  // #233 — Workers' get(key, type | { type }): "text" (default), "json",
5190
5310
  // "arrayBuffer". A value put as bytes is stored one char per byte and
5191
5311
  // flagged, so text decodes it as UTF-8 and arrayBuffer returns it
@@ -419,9 +419,26 @@ static int sb_ltx_begin(sb_ltx_writer *w, const char *path, sb_ltx_state *s, uin
419
419
  memset(w, 0, sizeof(*w));
420
420
  w->hash = UINT64_MAX;
421
421
  w->page_size = s->page_size;
422
+ // Global temporary directories are outside a sandboxed worker's writable tree.
423
+ // After unlink, close (including process death) reclaims the scratch file.
424
+ char index_path[SB_LTX_PATH];
425
+ if (snprintf(index_path, sizeof(index_path), "%s/index-XXXXXX", s->root) >= sizeof(index_path))
426
+ return -1;
427
+ int index = mkstemp(index_path);
428
+ if (index < 0)
429
+ return -1;
430
+ sb_ltx_phase("index-created");
431
+ if (unlink(index_path)) {
432
+ close(index);
433
+ return -1;
434
+ }
435
+ w->index = fdopen(index, "w+b");
436
+ if (!w->index) {
437
+ close(index);
438
+ return -1;
439
+ }
422
440
  w->out = fopen(path, "wbx");
423
- w->index = tmpfile();
424
- if (!w->out || !w->index)
441
+ if (!w->out)
425
442
  return -1;
426
443
  unsigned char h[100] = {0};
427
444
  memcpy(h, "LTX1", 4);
@@ -1048,6 +1065,41 @@ done:
1048
1065
  return 1;
1049
1066
  }
1050
1067
 
1068
+ // The capture lock is already held. Remove only reserved, regular scratch files;
1069
+ // do not open them (a hard link could share the database's POSIX-locked inode).
1070
+ static int sb_ltx_sweep_indexes(const char *root) {
1071
+ DIR *dir = opendir(root);
1072
+ if (!dir)
1073
+ return -1;
1074
+ int rc = 0;
1075
+ for (;;) {
1076
+ errno = 0;
1077
+ struct dirent *entry = readdir(dir);
1078
+ if (!entry) {
1079
+ if (errno)
1080
+ rc = -1;
1081
+ break;
1082
+ }
1083
+ const char *name = entry->d_name;
1084
+ if (strlen(name) != 12 || strncmp(name, "index-", 6) ||
1085
+ strspn(name + 6, "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789") != 6)
1086
+ continue;
1087
+ struct stat st;
1088
+ if (fstatat(dirfd(dir), name, &st, AT_SYMLINK_NOFOLLOW)) {
1089
+ if (errno == ENOENT)
1090
+ continue;
1091
+ rc = -1;
1092
+ break;
1093
+ }
1094
+ if (S_ISREG(st.st_mode) && unlinkat(dirfd(dir), name, 0) && errno != ENOENT) {
1095
+ rc = -1;
1096
+ break;
1097
+ }
1098
+ }
1099
+ if (closedir(dir))
1100
+ rc = -1;
1101
+ return rc;
1102
+ }
1051
1103
  static int sb_ltx_start(int idx, sqlite3 *db, const char *path, int lock) {
1052
1104
  sb_ltx_state *s = &sb_ltx_states[idx];
1053
1105
  memset(s, 0, sizeof(*s));
@@ -1059,6 +1111,8 @@ static int sb_ltx_start(int idx, sqlite3 *db, const char *path, int lock) {
1059
1111
  return -1;
1060
1112
  if (mkdir(s->root, 0700) && errno != EEXIST)
1061
1113
  return -1;
1114
+ if (sb_ltx_sweep_indexes(s->root))
1115
+ return -1;
1062
1116
  char parent[SB_LTX_PATH];
1063
1117
  snprintf(parent, sizeof(parent), "%s", path);
1064
1118
  char *slash = strrchr(parent, '/');
@@ -2137,12 +2191,12 @@ static int sb_db_page_size(const char* path) {
2137
2191
  if (st) sqlite3_finalize(st);
2138
2192
  return size;
2139
2193
  }
2140
- // Freeze (refuse writes to) or thaw a database for the quota. Returns 1 when the database
2141
- // is under LTX capture, 0 when not, -1 when it cannot be opened.
2194
+ // Freeze (refuse writes to) or thaw a database for the quota; on < 0 only asks. Returns 1
2195
+ // when the database is under LTX capture, 0 when not, -1 when it cannot be opened.
2142
2196
  static int sb_db_quota_freeze(const char* path, int on) {
2143
2197
  int idx = sb_db_for(path);
2144
2198
  if (idx < 0) return -1;
2145
- sb_quota_frozen[idx] = on && sb_ltx_states[idx].db ? 1 : 0;
2199
+ if (on >= 0) sb_quota_frozen[idx] = on && sb_ltx_states[idx].db ? 1 : 0;
2146
2200
  return sb_ltx_states[idx].db ? 1 : 0;
2147
2201
  }
2148
2202
 
@@ -3197,7 +3251,7 @@ function __sbD1DbRaw(path, what) {
3197
3251
  if (__po) free(__po);
3198
3252
  const char* __w; size_t __wl; char* __wo = 0;
3199
3253
  porf_native_fetch_read_value(what, &__w, &__wl, &__wo);
3200
- int __r = __wl && __w[0] == 'p' ? sb_db_page_size(__path) : sb_db_quota_freeze(__path, __wl && __w[0] == 'f');
3254
+ int __r = __wl && __w[0] == 'p' ? sb_db_page_size(__path) : __wl && __w[0] == 'c' ? sb_db_quota_freeze(__path, -1) : sb_db_quota_freeze(__path, __wl && __w[0] == 'f');
3201
3255
  if (__wo) free(__wo);
3202
3256
  free(__path);
3203
3257
  char __n[16]; int __k = snprintf(__n, sizeof(__n), "%d", __r);
@@ -3455,11 +3509,31 @@ var __sbQuotaOps = {
3455
3509
  // A database under LTX capture is frozen instead: every write is refused, because even a
3456
3510
  // DELETE adds a file to the replication spool, which counts toward the quota.
3457
3511
  var __sbD1Capped = {};
3512
+ var __sbD1Captured = {};
3513
+ var __sbD1Checkpointed = {};
3458
3514
  function __sbD1Quota(path) {
3459
- const full = __sbQuotaFull();
3515
+ // Whether capture owns this database, from the private helper (no SQL), before anything
3516
+ // below depends on it. A database that cannot open (or whose capture failed) is left to
3517
+ // the call itself, which reports why.
3518
+ const status = Number(__sbD1DbRaw(path, "captured"));
3519
+ if (status < 0) return;
3520
+ __sbD1Captured[path] = status === 1;
3521
+ let full = __sbQuotaFull();
3522
+ // Over the quota, a database's -wal file can hold space the app has already freed (a
3523
+ // DELETE, a VACUUM): checkpoint it into the database and truncate it, at most once a
3524
+ // second, then measure again. Not under capture: LTX owns that database's checkpoints.
3525
+ if (full && !__sbD1Captured[path] && Date.now() - (__sbD1Checkpointed[path] || 0) >= 1000) {
3526
+ __sbD1Checkpointed[path] = Date.now();
3527
+ JSON.parse(__sbSqlRaw(path, "PRAGMA wal_checkpoint(TRUNCATE)", "[]"));
3528
+ __sbQuotaAt = -1;
3529
+ full = __sbQuotaFull();
3530
+ }
3460
3531
  if (__sbD1Capped[path] === full) return;
3461
3532
  __sbD1Capped[path] = full;
3462
- const captured = Number(__sbD1DbRaw(path, full ? "freeze" : "thaw")) === 1;
3533
+ const frozen = Number(__sbD1DbRaw(path, full ? "freeze" : "thaw"));
3534
+ if (frozen < 0) return;
3535
+ const captured = frozen === 1;
3536
+ __sbD1Captured[path] = captured;
3463
3537
  if (full && captured) return;
3464
3538
  const size = Number(__sbD1DbRaw(path, "page_size")) || 4096;
3465
3539
  const pages = full
@@ -3499,6 +3573,10 @@ function __sbEnsureSchema() {
3499
3573
  let hasKvBinary = false;
3500
3574
  for (let i = 0; i < kvColumns.length; i++) if (kvColumns[i][1] === "binary") hasKvBinary = true;
3501
3575
  if (!hasKvBinary) __sbSql(s, "ALTER TABLE kv ADD COLUMN binary INTEGER NOT NULL DEFAULT 0", []);
3576
+ // A key's metadata (put(key, value, { metadata })), as JSON text; same column as the broker's.
3577
+ let hasKvMetadata = false;
3578
+ for (let i = 0; i < kvColumns.length; i++) if (kvColumns[i][1] === "metadata") hasKvMetadata = true;
3579
+ if (!hasKvMetadata) __sbSql(s, "ALTER TABLE kv ADD COLUMN metadata TEXT", []);
3502
3580
  // #56 — no `body` column: an object's bytes live in their own file under
3503
3581
  // "<data-dir>/r2-blobs/" (sb_r2_put_c/sb_r2_get_c), never in the sqlite row.
3504
3582
  __sbSql(
@@ -3602,7 +3680,7 @@ function __sbEmbeddedDispatch(msg) {
3602
3680
  if (op === "ping") return { ok: true, op: "pong", echo: msg.msg };
3603
3681
 
3604
3682
  if (op === "kv.get") {
3605
- const r = __sbSql(store, "SELECT value, expires_at, binary FROM kv WHERE ns = ? AND key = ?", [
3683
+ const r = __sbSql(store, "SELECT value, expires_at, binary, metadata FROM kv WHERE ns = ? AND key = ?", [
3606
3684
  msg.ns,
3607
3685
  msg.key,
3608
3686
  ]);
@@ -3612,7 +3690,14 @@ function __sbEmbeddedDispatch(msg) {
3612
3690
  __sbSql(store, "DELETE FROM kv WHERE ns = ? AND key = ?", [msg.ns, msg.key]);
3613
3691
  return { ok: true, found: false, value: null };
3614
3692
  }
3615
- return { ok: true, found: true, value: r.rows[0][0], binary: r.rows[0][2] === 1 };
3693
+ return {
3694
+ ok: true,
3695
+ found: true,
3696
+ value: r.rows[0][0],
3697
+ binary: r.rows[0][2] === 1,
3698
+ metadata: r.rows[0][3],
3699
+ expiration: expiresAt == null ? null : Math.floor(expiresAt / 1000),
3700
+ };
3616
3701
  }
3617
3702
  if (op === "kv.put") {
3618
3703
  // #58 — CF rejects expirationTtl under 60 seconds; matched here.
@@ -3626,8 +3711,8 @@ function __sbEmbeddedDispatch(msg) {
3626
3711
  }
3627
3712
  __sbSql(
3628
3713
  store,
3629
- "INSERT INTO kv (ns, key, value, expires_at, binary) VALUES (?1,?2,?3,?4,?5) ON CONFLICT (ns, key) DO UPDATE SET value = ?3, expires_at = ?4, binary = ?5",
3630
- [msg.ns, msg.key, msg.value, expiresAt, msg.binary === true ? 1 : 0],
3714
+ "INSERT INTO kv (ns, key, value, expires_at, binary, metadata) VALUES (?1,?2,?3,?4,?5,?6) ON CONFLICT (ns, key) DO UPDATE SET value = ?3, expires_at = ?4, binary = ?5, metadata = ?6",
3715
+ [msg.ns, msg.key, msg.value, expiresAt, msg.binary === true ? 1 : 0, msg.metadata == null ? null : String(msg.metadata)],
3631
3716
  );
3632
3717
  return { ok: true };
3633
3718
  }
@@ -3636,14 +3721,28 @@ function __sbEmbeddedDispatch(msg) {
3636
3721
  return { ok: true };
3637
3722
  }
3638
3723
  if (op === "kv.list") {
3724
+ // The prefix matches exactly: LIKE ignored case and read "_" and "%" in it as wildcards.
3725
+ // With a limit (the shim from 2026-10-25), a page of entries after the cursor (the last
3726
+ // key of the previous page), with each key's expiration and metadata; without one,
3727
+ // every key, as before.
3728
+ const prefix = msg.prefix || "";
3729
+ const paged = msg.limit != null;
3730
+ const limit = paged ? Math.max(0, Math.floor(Number(msg.limit))) : -1;
3639
3731
  const r = __sbSql(
3640
3732
  store,
3641
- "SELECT key FROM kv WHERE ns = ? AND key LIKE ? || '%' AND (expires_at IS NULL OR expires_at > ?) ORDER BY key",
3642
- [msg.ns, msg.prefix || "", Date.now()],
3733
+ "SELECT key, expires_at, metadata FROM kv WHERE ns = ?1 AND substr(key, 1, length(?2)) = ?2 AND key > ?3 " +
3734
+ "AND (expires_at IS NULL OR expires_at > ?4) ORDER BY key LIMIT ?5",
3735
+ [msg.ns, prefix, paged && msg.cursor ? String(msg.cursor) : "", Date.now(), paged ? limit + 1 : -1],
3643
3736
  );
3644
3737
  const keys = [];
3645
- for (let i = 0; i < r.rows.length; i++) keys.push(r.rows[i][0]);
3646
- return { ok: true, keys };
3738
+ const entries = [];
3739
+ const count = paged ? Math.min(r.rows.length, limit) : r.rows.length;
3740
+ for (let i = 0; i < count; i++) {
3741
+ keys.push(r.rows[i][0]);
3742
+ entries.push({ name: r.rows[i][0], expiration: r.rows[i][1] == null ? null : Math.floor(r.rows[i][1] / 1000), metadata: r.rows[i][2] });
3743
+ }
3744
+ const complete = !paged || r.rows.length <= limit;
3745
+ return { ok: true, keys, entries, list_complete: complete, cursor: complete ? "" : keys[keys.length - 1] };
3647
3746
  }
3648
3747
 
3649
3748
  if (op === "assets.get") {
@@ -0,0 +1,110 @@
1
+ // LTX chain restore, the same independent checker as native-ltx.ts (copied, so that test
2
+ // stays as it is): every file's CRCs, checksums and TXIDs verified, pages applied in order.
3
+ import assert from "node:assert/strict";
4
+ import { readFile, readdir, writeFile } from "node:fs/promises";
5
+ import { join } from "node:path";
6
+
7
+ // Independent CRC baseline in Bun. This intentionally uses BigInt, unlike C.
8
+ const table: bigint[] = [];
9
+ for (let i = 0; i < 256; i++) {
10
+ let c = BigInt(i);
11
+ for (let j = 0; j < 8; j++) c = (c >> 1n) ^ (c & 1n ? 0xd800000000000000n : 0n);
12
+ table.push(c);
13
+ }
14
+ function crc(data: Uint8Array, init = 0xffffffffffffffffn) {
15
+ let c = init;
16
+ for (const b of data) c = table[Number((c ^ BigInt(b)) & 255n)] ^ (c >> 8n);
17
+ return c;
18
+ }
19
+ const flag = 0x8000000000000000n,
20
+ mask = 0xffffffffffffffffn;
21
+ function pageCrc(pgno: number, page: Uint8Array) {
22
+ const p = new Uint8Array(4);
23
+ new DataView(p.buffer).setUint32(0, pgno);
24
+ return flag | (crc(page, crc(p)) ^ mask);
25
+ }
26
+
27
+ export async function restore(epoch: string, output: string) {
28
+ const files = (await readdir(epoch)).filter((n) => n.endsWith(".ltx")).sort();
29
+ let db = new Uint8Array(0),
30
+ checksum = 0n,
31
+ txid = 0n;
32
+ for (const file of files) {
33
+ const b = new Uint8Array(await readFile(join(epoch, file)));
34
+ const v = new DataView(b.buffer, b.byteOffset, b.byteLength);
35
+ assert.equal(new TextDecoder().decode(b.subarray(0, 4)), "LTX1");
36
+ const pageSize = v.getUint32(8),
37
+ commit = v.getUint32(12);
38
+ assert.equal(v.getBigUint64(16), txid + 1n);
39
+ assert.equal(v.getBigUint64(40), checksum);
40
+ const next = new Uint8Array(commit * pageSize);
41
+ next.set(db.subarray(0, next.length));
42
+ db = next;
43
+ let offset = 100,
44
+ hash = crc(b.subarray(0, 100));
45
+ const entries = [];
46
+ for (;;) {
47
+ const begin = offset,
48
+ pgno = v.getUint32(offset),
49
+ flags = v.getUint16(offset + 4);
50
+ hash = crc(b.subarray(offset, offset + 6), hash);
51
+ offset += 6;
52
+ if (!pgno) {
53
+ assert.equal(flags, 0);
54
+ break;
55
+ }
56
+ assert.equal(flags, 1);
57
+ assert(pgno <= commit);
58
+ const size = v.getUint32(offset);
59
+ hash = crc(b.subarray(offset, offset + 4), hash);
60
+ offset += 4;
61
+ const stop = offset + size;
62
+ assert(stop <= b.length);
63
+ assert.equal(b[offset++], 0xf0);
64
+ let literals = 15,
65
+ byte;
66
+ do {
67
+ byte = b[offset++];
68
+ literals += byte;
69
+ } while (byte === 255);
70
+ assert.equal(literals, pageSize);
71
+ assert.equal(offset + pageSize, stop);
72
+ const page = b.subarray(offset, stop);
73
+ db.set(page, (pgno - 1) * pageSize);
74
+ hash = crc(page, hash);
75
+ offset = stop;
76
+ entries.push({ pgno, offset: begin, size: stop - begin });
77
+ }
78
+ const indexStart = offset;
79
+ function varint() {
80
+ let n = 0n,
81
+ s = 0n;
82
+ for (let i = 0; i < 10; i++) {
83
+ assert(offset < b.length);
84
+ const c = b[offset++];
85
+ n |= BigInt(c & 127) << s;
86
+ if (c < 128) return Number(n);
87
+ s += 7n;
88
+ }
89
+ throw new Error("varint overflow");
90
+ }
91
+ for (const entry of entries) {
92
+ assert.equal(varint(), entry.pgno);
93
+ assert.equal(varint(), entry.offset);
94
+ assert.equal(varint(), entry.size);
95
+ }
96
+ assert.equal(varint(), 0);
97
+ assert.equal(v.getBigUint64(offset), BigInt(offset - indexStart));
98
+ offset += 8;
99
+ hash = crc(b.subarray(indexStart, offset + 8), hash);
100
+ checksum = flag;
101
+ for (let p = 1; p <= commit; p++)
102
+ checksum = flag | (checksum ^ pageCrc(p, db.subarray((p - 1) * pageSize, p * pageSize)));
103
+ assert.equal(v.getBigUint64(offset), checksum, file);
104
+ assert.equal(v.getBigUint64(offset + 8), flag | (hash ^ mask), file);
105
+ assert.equal(offset + 16, b.length);
106
+ txid = v.getBigUint64(24);
107
+ }
108
+ await writeFile(output, db);
109
+ return { files: files.length, txid };
110
+ }
package/test/native-d1.ts CHANGED
@@ -1,4 +1,6 @@
1
1
  import assert from "node:assert/strict";
2
+ import { Database } from "bun:sqlite";
3
+ import { restore } from "./ltx-restore";
2
4
  import { closeSync, openSync, readFileSync, readdirSync, readSync, writeSync } from "node:fs";
3
5
  import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
4
6
  import { tmpdir } from "node:os";
@@ -47,7 +49,7 @@ int main(void) {
47
49
  if (!strncmp(line, ":break ", 7)) { line[strcspn(line, "\\t")] = 0; char p[1024]; snprintf(p, sizeof(p), "%s.ltx/position.tmp", line + 7); mkdir(p, 0700); puts("{}"); fflush(stdout); continue; }
48
50
  char* sql = strchr(line, '\\t'); if (!sql) return 2; *sql++ = 0;
49
51
  char* params = strchr(sql, '\\t'); if (!params) return 2; *params++ = 0;
50
- if (!strncmp(line, ":db ", 4)) { int r = sql[0] == 'p' ? sb_db_page_size(line + 4) : sb_db_quota_freeze(line + 4, sql[0] == 'f'); printf("{\\"n\\":%d}\\n", r); fflush(stdout); continue; }
52
+ if (!strncmp(line, ":db ", 4)) { int r = sql[0] == 'p' ? sb_db_page_size(line + 4) : sb_db_quota_freeze(line + 4, sql[0] == 'c' ? -1 : sql[0] == 'f'); printf("{\\"n\\":%d}\\n", r); fflush(stdout); continue; }
51
53
  if (!strncmp(line, ":bytes ", 7)) { line[strcspn(line, "\\t")] = 0; printf("{\\"n\\":%lld}\\n", sb_dir_bytes(line + 7, 0)); fflush(stdout); continue; }
52
54
  // ":script PATH" runs SQL as a script (sb_sql_script, what exec() uses).
53
55
  char* out = !strncmp(line, ":script ", 8) ? sb_sql_script(line + 8, sql) : sb_sql_run(line, sql, params); puts(out); fflush(stdout); free(out);
@@ -149,7 +151,9 @@ int main(void) {
149
151
  const toDriver = openSync(fifoIn, "w"),
150
152
  fromDriver = openSync(fifoOut, "r");
151
153
  let pending = "";
154
+ let driverClosed = false;
152
155
  const sqlRaw = (path: string, sql: string, params: string) => {
156
+ if (driverClosed) throw new Error("original D1 driver is closed");
153
157
  writeSync(toDriver, `${path}\t${sql}\t${params}\n`);
154
158
  const chunk = Buffer.alloc(65536);
155
159
  while (!pending.includes("\n")) {
@@ -185,6 +189,7 @@ int main(void) {
185
189
  "__sbDirBytesRaw",
186
190
  "__sbD1DbRaw",
187
191
  "__sbSql",
192
+ "__sbSqlRaw",
188
193
  `${transport.slice(quotaStart, quotaEnd)}\nreturn { __sbQuotaCheck, __sbD1Quota, __sbQuotaFull, expire() { __sbQuotaAt = -1; } };`,
189
194
  )(
190
195
  (name: string) => (name === "SB_STORAGE_MAX_BYTES" ? String(limit) : ""),
@@ -192,8 +197,70 @@ int main(void) {
192
197
  (d: string) => String(JSON.parse(sqlRaw(":bytes " + d, "", "")).n),
193
198
  (path: string, what: string) => String(JSON.parse(sqlRaw(":db " + path, what, "")).n),
194
199
  sql,
200
+ sqlRaw,
195
201
  );
196
202
  const dirBytes = (d: string) => Number(JSON.parse(sqlRaw(":bytes " + d, "", "")).n);
203
+ // Another driver process (a restarted worker), with its own FIFOs and lifted blocks.
204
+ const connect = (tag: string) => {
205
+ const inPath = join(work, "in-" + tag),
206
+ outPath = join(work, "out-" + tag);
207
+ assert.equal(Bun.spawnSync(["mkfifo", inPath, outPath]).exitCode, 0);
208
+ const child = Bun.spawn(["sh", "-c", `exec "${binary}" < "${inPath}" > "${outPath}"`], {
209
+ env: { ...process.env, SB_LTX_ENABLE: "1", SB_LTX_DATABASE: "LTX" },
210
+ });
211
+ const to = openSync(inPath, "w"),
212
+ from = openSync(outPath, "r");
213
+ let buffered = "";
214
+ const raw = (path: string, text: string, params: string) => {
215
+ writeSync(to, `${path}\t${text}\t${params}\n`);
216
+ const chunk = Buffer.alloc(65536);
217
+ while (!buffered.includes("\n")) {
218
+ const n = readSync(from, chunk, 0, chunk.length, null);
219
+ if (n <= 0) throw new Error("driver exited");
220
+ buffered += chunk.toString("utf8", 0, n);
221
+ }
222
+ const line = buffered.slice(0, buffered.indexOf("\n"));
223
+ buffered = buffered.slice(line.length + 1);
224
+ return line;
225
+ };
226
+ const run = (path: string, text: string, params: unknown[]) => {
227
+ const r = JSON.parse(raw(path, text, JSON.stringify(params)));
228
+ if (r.ok === false) throw new Error("sqlite: " + r.error);
229
+ return r;
230
+ };
231
+ // oxlint-disable-next-line no-function-constructor -- lifting a block out of the text-only transport
232
+ const lifted = new Function(
233
+ "__sbSqlRaw",
234
+ "__sbSql",
235
+ "__sbSqlScriptRaw",
236
+ `${transport.slice(opsStart, opsEnd)}\nreturn { __sbD1Run, __sbD1Batch, __sbD1Open, __sbD1ExecLines };`,
237
+ )(raw, run, (path: string, text: string) => raw(":script " + path, text, "[]"));
238
+ const lift = (dir: string, limit: number) =>
239
+ // oxlint-disable-next-line no-function-constructor -- lifting a block out of the text-only transport
240
+ new Function(
241
+ "__sbEnv",
242
+ "__sbDir",
243
+ "__sbDirBytesRaw",
244
+ "__sbD1DbRaw",
245
+ "__sbSql",
246
+ "__sbSqlRaw",
247
+ `${transport.slice(quotaStart, quotaEnd)}\nreturn { __sbQuotaCheck, __sbD1Quota, __sbQuotaFull };`,
248
+ )(
249
+ (name: string) => (name === "SB_STORAGE_MAX_BYTES" ? String(limit) : ""),
250
+ () => dir,
251
+ (d: string) => String(JSON.parse(raw(":bytes " + d, "", "")).n),
252
+ (path: string, what: string) => String(JSON.parse(raw(":db " + path, what, "")).n),
253
+ run,
254
+ raw,
255
+ );
256
+ return {
257
+ sqlRaw: raw,
258
+ ops: lifted,
259
+ liftQuota: lift,
260
+ exited: child.exited,
261
+ close: () => (closeSync(to), closeSync(from)),
262
+ };
263
+ };
197
264
  // exec() from 2026-10-24: line by line, all or nothing, the failing line named.
198
265
  const ex = join(work, "data/d1/EXEC.sqlite");
199
266
  ops.__sbD1ExecLines(ex, ["CREATE TABLE x (id INTEGER)", "INSERT INTO x VALUES (1); INSERT INTO x VALUES (2);"]);
@@ -282,8 +349,14 @@ int main(void) {
282
349
  // With capture on, an authorization failure says so (it read "not an error" once capture
283
350
  // had run), and the quota works: its page size comes from capture, not from the
284
351
  // page_size pragma the capture authorizer refuses.
285
- assert.throws(() => ops.__sbD1Run(ltx, "PRAGMA page_size", [], true), (e: Error) => e.message === "D1_ERROR: not authorized: SQLITE_AUTH");
286
- assert.throws(() => ops.__sbD1ExecLines(ltx, ["PRAGMA journal_mode = DELETE"]), (e: Error) => e.message === "D1_EXEC_ERROR: Error in line 1: PRAGMA journal_mode = DELETE: not authorized");
352
+ assert.throws(
353
+ () => ops.__sbD1Run(ltx, "PRAGMA page_size", [], true),
354
+ (e: Error) => e.message === "D1_ERROR: not authorized: SQLITE_AUTH",
355
+ );
356
+ assert.throws(
357
+ () => ops.__sbD1ExecLines(ltx, ["PRAGMA journal_mode = DELETE"]),
358
+ (e: Error) => e.message === "D1_EXEC_ERROR: Error in line 1: PRAGMA journal_mode = DELETE: not authorized",
359
+ );
287
360
  const data = join(work, "data");
288
361
  const roomy = liftQuota(data, dirBytes(data) + (1 << 20));
289
362
  roomy.__sbD1Quota(ltx);
@@ -295,12 +368,39 @@ int main(void) {
295
368
  const tight = liftQuota(data, 1);
296
369
  tight.__sbD1Quota(ltx);
297
370
  const spool = ltxFiles();
298
- const QUOTA = "D1_ERROR: storage quota exceeded: writes are refused while this app's data, including its replication log, is over its quota: SQLITE_FULL";
299
- for (const write of ["INSERT INTO e VALUES (11, 'x')", "UPDATE e SET v = 'y' WHERE id = 10", "DELETE FROM e WHERE id = 10"])
300
- assert.throws(() => ops.__sbD1Run(ltx, write, [], true), (e: Error) => e.message === QUOTA, write);
301
- assert.throws(() => ops.__sbD1ExecLines(ltx, ["UPDATE e SET v = 'z'"]), (e: Error) => e.message.startsWith("D1_EXEC_ERROR: Error in line 1: UPDATE e SET v = 'z': storage quota exceeded"));
302
- assert.throws(() => ops.__sbD1Batch(ltx, [{ sql: "SELECT 1", params: [] }, { sql: "DELETE FROM e", params: [] }], true), (e: Error) => e.message === QUOTA);
303
- assert.equal(ops.__sbD1Batch(ltx, [{ sql: "SELECT count(*) FROM e", params: [] }], true).length, 1, "a read-only batch runs");
371
+ const QUOTA =
372
+ "D1_ERROR: storage quota exceeded: writes are refused while this app's data, including its replication log, is over its quota: SQLITE_FULL";
373
+ for (const write of [
374
+ "INSERT INTO e VALUES (11, 'x')",
375
+ "UPDATE e SET v = 'y' WHERE id = 10",
376
+ "DELETE FROM e WHERE id = 10",
377
+ ])
378
+ assert.throws(
379
+ () => ops.__sbD1Run(ltx, write, [], true),
380
+ (e: Error) => e.message === QUOTA,
381
+ write,
382
+ );
383
+ assert.throws(
384
+ () => ops.__sbD1ExecLines(ltx, ["UPDATE e SET v = 'z'"]),
385
+ (e: Error) => e.message.startsWith("D1_EXEC_ERROR: Error in line 1: UPDATE e SET v = 'z': storage quota exceeded"),
386
+ );
387
+ assert.throws(
388
+ () =>
389
+ ops.__sbD1Batch(
390
+ ltx,
391
+ [
392
+ { sql: "SELECT 1", params: [] },
393
+ { sql: "DELETE FROM e", params: [] },
394
+ ],
395
+ true,
396
+ ),
397
+ (e: Error) => e.message === QUOTA,
398
+ );
399
+ assert.equal(
400
+ ops.__sbD1Batch(ltx, [{ sql: "SELECT count(*) FROM e", params: [] }], true).length,
401
+ 1,
402
+ "a read-only batch runs",
403
+ );
304
404
  assert.equal(ops.__sbD1Run(ltx, "SELECT v FROM e WHERE id = 10", [], true).rows[0][0], "under the quota");
305
405
  assert.equal(ltxFiles(), spool, "no LTX file while over the quota");
306
406
  const again = liftQuota(data, dirBytes(data) + (1 << 20));
@@ -314,10 +414,14 @@ int main(void) {
314
414
  "__sbSql",
315
415
  "__sbSqlScriptRaw",
316
416
  `${transport.slice(opsStart, opsEnd)}\nreturn { __sbD1Run, __sbD1Batch };`,
317
- )((path: string, text: string, params: string) => {
318
- if (text === "RELEASE sb_d1_batch") sqlRaw(":break " + path, "", "");
319
- return sqlRaw(path, text, params);
320
- }, sql, scriptRaw);
417
+ )(
418
+ (path: string, text: string, params: string) => {
419
+ if (text === "RELEASE sb_d1_batch") sqlRaw(":break " + path, "", "");
420
+ return sqlRaw(path, text, params);
421
+ },
422
+ sql,
423
+ scriptRaw,
424
+ );
321
425
  const LTX_FAILED =
322
426
  "D1_ERROR: LTX capture failed: the transaction may be committed locally but is not durable in the replication log; writes are refused until restart: SQLITE_IOERR";
323
427
  assert.throws(
@@ -336,7 +440,9 @@ int main(void) {
336
440
  );
337
441
  assert.throws(
338
442
  () => ops.__sbD1ExecLines(ltx, ["INSERT INTO e VALUES (5, 'e')"]),
339
- (e: Error) => e.message === "D1_ERROR: LTX capture failed: the transaction may be committed locally but is not durable in the replication log; writes are refused until restart: SQLITE_IOERR",
443
+ (e: Error) =>
444
+ e.message ===
445
+ "D1_ERROR: LTX capture failed: the transaction may be committed locally but is not durable in the replication log; writes are refused until restart: SQLITE_IOERR",
340
446
  "exec() after a capture failure",
341
447
  );
342
448
  const qdir = join(work, "quota");
@@ -349,25 +455,86 @@ int main(void) {
349
455
  // SQLITE_FULL; then the directory is over, a KV write is refused, and reads and deletes
350
456
  // keep working. VACUUM gives the space back and writes resume.
351
457
  let rows = 0;
352
- assert.throws(() => {
353
- for (; rows < 64; rows++) ops.__sbD1Run(qdb, "INSERT INTO q VALUES (zeroblob(65536))", [], true);
354
- }, (e: Error) => e.message === "D1_ERROR: database or disk is full: SQLITE_FULL");
458
+ assert.throws(
459
+ () => {
460
+ for (; rows < 64; rows++) ops.__sbD1Run(qdb, "INSERT INTO q VALUES (zeroblob(65536))", [], true);
461
+ },
462
+ (e: Error) => e.message === "D1_ERROR: database or disk is full: SQLITE_FULL",
463
+ );
355
464
  assert(rows >= 2 && rows < 5, "filled to about 256 KiB: " + rows);
356
465
  quota.expire();
357
466
  assert.equal(quota.__sbQuotaFull(), true);
358
- assert.throws(() => quota.__sbQuotaCheck("KV put()"), /^Error: KV put\(\) failed: storage quota exceeded \(this app stores \d+ bytes; the limit is 262144\)$/);
467
+ assert.throws(
468
+ () => quota.__sbQuotaCheck("KV put()"),
469
+ /^Error: KV put\(\) failed: storage quota exceeded \(this app stores \d+ bytes; the limit is 262144\)$/,
470
+ );
359
471
  quota.__sbD1Quota(qdb);
360
472
  assert.equal(ops.__sbD1Run(qdb, "SELECT count(*) FROM q", [], true).rows[0][0], rows, "reads still work");
361
473
  ops.__sbD1Run(qdb, "DELETE FROM q", [], true);
362
474
  ops.__sbD1Run(qdb, "VACUUM", [], true);
363
- ops.__sbD1Run(qdb, "PRAGMA wal_checkpoint(TRUNCATE)", [], true);
475
+ // No checkpoint by the app: over the quota the runtime checkpoints and truncates the WAL
476
+ // (once a second) before measuring again, so DELETE and VACUUM alone make room.
477
+ await Bun.sleep(1000);
364
478
  quota.expire();
365
479
  quota.__sbD1Quota(qdb);
366
480
  quota.__sbQuotaCheck("KV put()");
367
481
  ops.__sbD1Run(qdb, "INSERT INTO q VALUES (zeroblob(65536))", [], true);
482
+ driverClosed = true;
368
483
  closeSync(toDriver);
369
484
  closeSync(fromDriver);
370
485
  await driver.exited;
486
+
487
+ // A restart that finds the app over its quota, capture on: the first D1 call of the new
488
+ // worker (process) already freezes. Reads work, writes are refused, the spool and its
489
+ // position stay exactly as they were, and the chain restores the database byte for byte.
490
+ // A capture failure still reads as one, over the quota too.
491
+ await rm(ltx + ".ltx/position.tmp", { recursive: true, force: true });
492
+ const second = connect("restart");
493
+ const quota2 = second.liftQuota(data, 1);
494
+ quota2.__sbD1Quota(ltx);
495
+ const spoolBefore = readdirSync(generation()).sort();
496
+ const positionBefore = readFileSync(ltx + ".ltx/position", "utf8");
497
+ const count = second.ops.__sbD1Run(ltx, "SELECT count(*) FROM e", [], true).rows[0][0];
498
+ for (const write of ["INSERT INTO e VALUES (20, 'x')", "UPDATE e SET v = v", "DELETE FROM e"])
499
+ assert.throws(
500
+ () => second.ops.__sbD1Run(ltx, write, [], true),
501
+ (e: Error) => e.message === QUOTA,
502
+ "restart, over the quota: " + write,
503
+ );
504
+ assert.deepEqual(readdirSync(generation()).sort(), spoolBefore, "no LTX file after a clean resume over the quota");
505
+ assert.equal(readFileSync(ltx + ".ltx/position", "utf8"), positionBefore, "the position is unchanged");
506
+ second.close();
507
+ await second.exited;
508
+ {
509
+ const live = new Database(ltx);
510
+ live.exec("PRAGMA wal_checkpoint(TRUNCATE)");
511
+ assert.equal(live.query("SELECT count(*) AS n FROM e").get()?.n, count);
512
+ live.close();
513
+ const restored = join(work, "restart-restored.sqlite");
514
+ await restore(generation(), restored);
515
+ assert.deepEqual(
516
+ await readFile(restored),
517
+ await readFile(ltx),
518
+ "the retained chain restores the database byte for byte",
519
+ );
520
+ }
521
+ const third = connect("failure");
522
+ const thirdBytes = Number(JSON.parse(third.sqlRaw(":bytes " + data, "", "")).n);
523
+ third.liftQuota(data, thirdBytes + (1 << 20)).__sbD1Quota(ltx);
524
+ const breaking3 = (path: string, text: string, params: string) => {
525
+ if (text.startsWith("INSERT INTO e VALUES (30")) third.sqlRaw(":break " + path, "", "");
526
+ return third.sqlRaw(path, text, params);
527
+ };
528
+ const brokenWrite = JSON.parse(breaking3(ltx, "INSERT INTO e VALUES (30, 'x')", "[]"));
529
+ assert.equal(brokenWrite.code, 10, "a capture failure reports SQLITE_IOERR");
530
+ const failed3 = JSON.parse(third.sqlRaw(ltx, "INSERT INTO e VALUES (31, 'y')", "[]"));
531
+ assert.equal(failed3.code, 10, "after a capture failure, writes report it");
532
+ third.liftQuota(data, 1).__sbD1Quota(ltx);
533
+ const failedOver = JSON.parse(third.sqlRaw(ltx, "INSERT INTO e VALUES (32, 'z')", "[]"));
534
+ assert.equal(failedOver.code, 10, "over the quota too, the capture failure is what a write reports");
535
+ assert.match(failedOver.error, /^LTX capture failed/);
536
+ third.close();
537
+ await third.exited;
371
538
  console.log(
372
539
  "native d1: sb_sql_run and the D1 ops pass (blob binds, atomic batch, strict counts, foreign keys, error text)",
373
540
  );
@@ -1,5 +1,5 @@
1
1
  import assert from "node:assert/strict";
2
- import { mkdtemp, readFile, readdir, rm, writeFile, stat } from "node:fs/promises";
2
+ import { mkdir, mkdtemp, readFile, readdir, rm, symlink, writeFile, stat } from "node:fs/promises";
3
3
  import { tmpdir } from "node:os";
4
4
  import { join } from "node:path";
5
5
  import { Database } from "bun:sqlite";
@@ -24,12 +24,18 @@ const source = `
24
24
  #include <sys/stat.h>
25
25
  #include <time.h>
26
26
  #include <signal.h>
27
+ #include <errno.h>
28
+ // Reproduce denial of global temporary-file access on every test platform.
29
+ // The sproutd sandbox qualification separately exercises real Linux Landlock.
30
+ static FILE *forbidden_tmpfile(void) { errno=EACCES; return NULL; }
31
+ #define tmpfile forbidden_tmpfile
27
32
  #define SB_MAX_DB 16
28
33
  ${transport.slice(start, end)}
29
34
  int main(int argc,char **argv) {
30
35
  if(argc!=2)return 2;sb_ltx_compat=1;if(sb_ltx_register())return 3;
31
36
  char lockpath[1024];snprintf(lockpath,sizeof(lockpath),"%s.ltx.lock",argv[1]);
32
37
  int lock=open(lockpath,O_RDWR|O_CREAT,0600);if(lock<0||flock(lock,LOCK_EX|LOCK_NB))return 4;
38
+ if(getenv("SB_LTX_TEST_HOLD_LOCK")){puts("locked");fflush(stdout);getchar();close(lock);return 0;}
33
39
  sqlite3 *db=0;if(sqlite3_open_v2(argv[1],&db,6,"sb-ltx-bounded"))return 5;
34
40
  if(getenv("SB_LTX_TEST_AUTO_VACUUM"))sqlite3_exec(db,"PRAGMA auto_vacuum=INCREMENTAL;",0,0,0);
35
41
  if(sqlite3_exec(db,"PRAGMA journal_mode=WAL;PRAGMA cache_size=-2000;",0,0,0)||sb_ltx_start(0,db,argv[1],lock))return 6;
@@ -153,6 +159,13 @@ async function restore(epoch: string, output: string) {
153
159
  await writeFile(output, db);
154
160
  return { files: files.length, txid };
155
161
  }
162
+ async function noIndexScratch(path: string) {
163
+ const entries = await readdir(path + ".ltx", { withFileTypes: true });
164
+ assert(
165
+ !entries.some((entry) => entry.isFile() && /^index-[a-zA-Z0-9]{6}$/.test(entry.name)),
166
+ "index scratch file leaked",
167
+ );
168
+ }
156
169
  async function run(path: string, sql: string, extra: Record<string, string> = {}) {
157
170
  const child = Bun.spawn([binary, path], {
158
171
  env: { ...process.env, ...extra },
@@ -169,6 +182,7 @@ async function run(path: string, sql: string, extra: Record<string, string> = {}
169
182
  ]);
170
183
  assert.equal(code, 0, error);
171
184
  assert(out.startsWith("ready\n"), out);
185
+ await noIndexScratch(path);
172
186
  return out.split("\n").slice(1).filter(Boolean);
173
187
  }
174
188
  async function stoppedCommit(path: string, phase: string, count = 1) {
@@ -212,6 +226,10 @@ async function stoppedCommit(path: string, phase: string, count = 1) {
212
226
  }
213
227
  child.kill("SIGKILL");
214
228
  await child.exited;
229
+ if (phase === "index-created") {
230
+ const entries = await readdir(path + ".ltx", { withFileTypes: true });
231
+ assert.equal(entries.filter((entry) => entry.isFile() && /^index-[a-zA-Z0-9]{6}$/.test(entry.name)).length, 1);
232
+ } else await noIndexScratch(path);
215
233
  assert(stopped, "did not reach crash boundary " + phase);
216
234
  return epoch;
217
235
  }
@@ -241,6 +259,45 @@ try {
241
259
  const cleanPosition = await readFile(root + "/position", "utf8");
242
260
  assert(cleanPosition.split("\n")[2].startsWith("C "));
243
261
  const finishedBefore = await readdir(join(root, epoch));
262
+ // SIGKILL can land between mkstemp and unlink. Only the next lock owner may
263
+ // discard that reserved scratch name; unrelated files and links stay intact.
264
+ const strayIndex = join(root, "index-abc123");
265
+ await writeFile(strayIndex, "stale index");
266
+ const unrelatedIndex = join(root, "index-not-scratch");
267
+ await writeFile(unrelatedIndex, "keep");
268
+ const indexDirectory = join(root, "index-DIR123");
269
+ await mkdir(indexDirectory);
270
+ await writeFile(join(indexDirectory, "keep"), "keep directory");
271
+ const outside = join(work, "outside-index-target");
272
+ await writeFile(outside, "keep target");
273
+ const indexLink = join(root, "index-LNK123");
274
+ await symlink(outside, indexLink);
275
+ const owner = Bun.spawn([binary, path], {
276
+ stdin: "pipe",
277
+ stdout: "pipe",
278
+ stderr: "pipe",
279
+ env: { ...process.env, SB_LTX_TEST_HOLD_LOCK: "1" },
280
+ });
281
+ const lockReader = owner.stdout.getReader();
282
+ try {
283
+ const ready = await lockReader.read();
284
+ assert.equal(new TextDecoder().decode(ready.value), "locked\n");
285
+ const blocked = Bun.spawnSync([binary, path], { stdin: "ignore" });
286
+ assert.equal(blocked.exitCode, 4, "non-owner cannot initialize capture");
287
+ assert.equal(await readFile(strayIndex, "utf8"), "stale index", "non-owner must not sweep indexes");
288
+ } finally {
289
+ owner.stdin.end();
290
+ await owner.exited;
291
+ lockReader.releaseLock();
292
+ }
293
+ await run(path, "SELECT 1;\n");
294
+ assert(!(await readdir(root)).includes("index-abc123"), "resume must reclaim stale index");
295
+ assert.equal(await readFile(unrelatedIndex, "utf8"), "keep");
296
+ assert.equal(await readFile(join(indexDirectory, "keep"), "utf8"), "keep directory");
297
+ assert.equal(await readFile(indexLink, "utf8"), "keep target");
298
+ assert.equal(await readFile(outside, "utf8"), "keep target");
299
+ assert.equal(await readFile(root + "/position", "utf8"), cleanPosition);
300
+ assert.deepEqual(await readdir(join(root, epoch)), finishedBefore);
244
301
  for (let n = 0; n < 5; n++) {
245
302
  await run(path, "SELECT 1;\n");
246
303
  assert.equal(await readFile(root + "/position", "utf8"), cleanPosition, "idle restart must not rewrite position");
@@ -361,6 +418,7 @@ try {
361
418
  failedDb.close();
362
419
  const phases = [
363
420
  "wal-committed",
421
+ "index-created",
364
422
  "ltx-written",
365
423
  "ltx-fsynced",
366
424
  "ltx-published",
@@ -408,7 +466,7 @@ try {
408
466
  assert.equal(pdb.query("PRAGMA integrity_check").get()?.integrity_check, "ok");
409
467
  pdb.close();
410
468
  console.log(
411
- "PASS restart: 10 SIGKILL durability boundaries, matching-prefix resume, orphan/missing/reset fallback, pending committed WAL tail captured before serving",
469
+ "PASS restart: 11 SIGKILL durability boundaries, matching-prefix resume, orphan/missing/reset fallback, pending committed WAL tail captured before serving",
412
470
  );
413
471
  console.log(
414
472
  "PASS native LTX: 102-file chain, independent CRC/index checks, byte-exact restore, integrity, hard WAL ceiling and oversized transaction rollback",