@sproutboat/runtime 0.25.0 → 0.26.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +31 -0
- package/LTX_REPLICATION.md +19 -4
- package/package.json +1 -1
- package/src/kv-shim.test.js +124 -0
- package/src/native-fetch-prelude.js +121 -1
- package/src/transport-embedded.js +115 -16
- package/test/ltx-restore.ts +110 -0
- package/test/native-d1.ts +186 -19
- package/test/native-ltx.ts +60 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,36 @@
|
|
|
1
1
|
# @sproutboat/runtime
|
|
2
2
|
|
|
3
|
+
## 0.26.0
|
|
4
|
+
|
|
5
|
+
### Minor Changes
|
|
6
|
+
|
|
7
|
+
- 9b15539: KV matches workerd from compatibility date `2026-10-25`:
|
|
8
|
+
|
|
9
|
+
- `get`, `getWithMetadata`, `put`, `delete` and `list` return Promises, and errors reject them.
|
|
10
|
+
- `get(key, type)` takes `"text"`, `"json"`, `"arrayBuffer"` and `"stream"`, and `get([keys])` returns a `Map`.
|
|
11
|
+
- `getWithMetadata(key, type)` returns `{ value, metadata, cacheStatus }`.
|
|
12
|
+
- `put(key, value, { expiration, expirationTtl, metadata })` takes strings, bytes and `ReadableStream`s, and stores the metadata.
|
|
13
|
+
- `list({ prefix, limit, cursor })` returns `{ keys: [{ name, expiration?, metadata? }], list_complete, cursor?, cacheStatus }` and pages at up to 1,000 keys.
|
|
14
|
+
- Keys, values, expirations, metadata and list limits fail with KV's own errors. For example: `Key name cannot be empty.`, `KV PUT failed: 414 UTF-8 encoded length of 513 exceeds key length limit of 512.`, `KV PUT failed: 400 Invalid expiration_ttl of 30. Expiration TTL must be at least 60.`, and `KV PUT failed: 413 Metadata length of 1110 exceeds limit of 1024.`
|
|
15
|
+
|
|
16
|
+
Earlier dates keep the synchronous shim. Two changes apply at every date:
|
|
17
|
+
|
|
18
|
+
- **A `list()` prefix now matches exactly.** It used SQL `LIKE`, so it ignored case and read `_` and `%` as wildcards. For example, the prefix `a_` also listed `ab`, and `a` also listed `AB`.
|
|
19
|
+
- **Both transports store KV metadata in a new column.** It's added to existing stores when they open.
|
|
20
|
+
|
|
21
|
+
### Patch Changes
|
|
22
|
+
|
|
23
|
+
- 1c36119: Create the LTX page-index scratch file inside the database's replication spool
|
|
24
|
+
instead of libc's global temporary directory, fixing capture initialization when
|
|
25
|
+
a worker's filesystem sandbox permits writes only beneath its own state directory.
|
|
26
|
+
The scratch file is private and immediately unlinked. If a kill lands before
|
|
27
|
+
unlink, the next capture-lock owner removes the stale regular index file before
|
|
28
|
+
recovery. After unlink, closing the writer or killing the process releases it.
|
|
29
|
+
Its anonymous transient disk use is not counted by the
|
|
30
|
+
directory-walk quota. Capture retains its existing explicit opt-in and compatibility
|
|
31
|
+
date of 2026-10-22; capture-off apps and earlier dates keep their existing behavior.
|
|
32
|
+
- 3cad1f0: The storage quota counts a D1 database's WAL file, which can hold space the app has already freed. While the app is over its quota, the runtime now checkpoints and truncates each D1 database's WAL before measuring again, at most once a second. So `DELETE` then `VACUUM` is enough to get back under the quota, and the app no longer has to run a checkpoint itself. Databases under LTX capture are left alone: capture owns their checkpoints, and they stay frozen until the quota is raised or other data is deleted.
|
|
33
|
+
|
|
3
34
|
## 0.25.0
|
|
4
35
|
|
|
5
36
|
### Minor Changes
|
package/LTX_REPLICATION.md
CHANGED
|
@@ -53,7 +53,18 @@ for this opted-in binding. Use the base LTX file for replication.
|
|
|
53
53
|
|
|
54
54
|
Files live under `<database-path>.ltx/epoch-XXXXXX/`. Completed names are
|
|
55
55
|
`<minTXID as 16 hex digits>-<maxTXID as 16 hex digits>.ltx`.
|
|
56
|
-
The encoder streams one page at a time
|
|
56
|
+
The encoder streams one page at a time. It creates its index scratch file with
|
|
57
|
+
`mkstemp()` beneath `<database-path>.ltx/`, immediately unlinks the name, and uses
|
|
58
|
+
the open descriptor until the writer closes. This needs no access to a global
|
|
59
|
+
temporary directory, which a sandboxed worker may not have. The scratch file is
|
|
60
|
+
private (0600). A kill between creation and unlink can leave a named index file.
|
|
61
|
+
On startup, after acquiring the capture lock and before recovery, the owner removes
|
|
62
|
+
only regular root-level files matching the reserved `index-` plus six alphanumeric
|
|
63
|
+
characters. It does not follow symlinks or remove directories or unrelated names.
|
|
64
|
+
After unlink, process death releases the anonymous scratch space automatically.
|
|
65
|
+
Its transient disk usage grows with the index records and is not visible to the
|
|
66
|
+
directory-walk storage quota after unlink. Allow disk headroom for that scratch
|
|
67
|
+
space as well as the LTX output; the quota is not a hard filesystem limit.
|
|
57
68
|
|
|
58
69
|
For every LTX file:
|
|
59
70
|
|
|
@@ -78,7 +89,10 @@ applications must use idempotent transaction identifiers for retries.
|
|
|
78
89
|
## Qualification status
|
|
79
90
|
|
|
80
91
|
Capture is tested against the exact inline C through `bun run test:native-ltx`,
|
|
81
|
-
|
|
92
|
+
with global `tmpfile()` access deliberately denied in the driver so the capture
|
|
93
|
+
path cannot depend on it. This portable regression does not itself qualify the
|
|
94
|
+
Linux Landlock/seccomp sandbox; sproutd's capture-on lifecycle smoke must do that.
|
|
95
|
+
The tests cover independent BigInt CRC and index verification, a 102-file chain,
|
|
82
96
|
byte-exact restore, integrity checking, oversized transaction rollback, database truncation, explicit rollback and late position-write failure latching.
|
|
83
97
|
The workspace prototype additionally tests the compiled Porffor executable,
|
|
84
98
|
including external writer exclusion and invariant PRAGMA rejection.
|
|
@@ -103,8 +117,9 @@ time. It also checks the last LTX file. Five native restarts each at 4.2 MB and
|
|
|
103
117
|
The 51.1 MB verification took 399 to 446 ms on macOS arm64 (plus about 13 ms to
|
|
104
118
|
health readiness); this is not a 9 ms cold-start claim for large databases.
|
|
105
119
|
|
|
106
|
-
The regression SIGKILLs at
|
|
107
|
-
|
|
120
|
+
The regression SIGKILLs at eleven durability boundaries, including the interval
|
|
121
|
+
between index creation and unlink and both sides of clean-marker publication,
|
|
122
|
+
and verifies scratch cleanup, matching-prefix resume and fallback cases.
|
|
108
123
|
It also recovers a committed tail beyond the durable cursor. The full Porffor executable passed 200 seeded random SIGKILL iterations on macOS
|
|
109
124
|
arm64: 1,675 acknowledged writes retained, 88 same-generation resumes and 112
|
|
110
125
|
fresh-base recoveries. Every finished chain passed file/database CRC verification,
|
package/package.json
CHANGED
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
// The KV shim (`env.KV`) at compatibility dates before and from 2026-10-25, over an
|
|
2
|
+
// in-memory transport with the kv.* contract both transports implement (broker.test.ts
|
|
3
|
+
// covers the broker's side; sproutd's conformance/bindings/kv.js the whole path).
|
|
4
|
+
import { expect, test } from "bun:test";
|
|
5
|
+
import { readFileSync } from "node:fs";
|
|
6
|
+
|
|
7
|
+
const source = readFileSync(new URL("./native-fetch-prelude.js", import.meta.url), "utf8");
|
|
8
|
+
let rows;
|
|
9
|
+
const call = (json) => {
|
|
10
|
+
const msg = JSON.parse(json);
|
|
11
|
+
const reply = (() => {
|
|
12
|
+
if (msg.op === "kv.get") {
|
|
13
|
+
const r = rows.get(msg.key);
|
|
14
|
+
return r ? { ok: true, found: true, value: r.value, binary: r.binary, metadata: r.metadata ?? null, expiration: r.expiration ?? null } : { ok: true, found: false, value: null };
|
|
15
|
+
}
|
|
16
|
+
if (msg.op === "kv.put") {
|
|
17
|
+
rows.set(msg.key, { value: msg.value, binary: msg.binary === true, metadata: msg.metadata, expiration: msg.expiration ?? (msg.expirationTtl ? Math.floor(Date.now() / 1000) + msg.expirationTtl : undefined) });
|
|
18
|
+
return { ok: true };
|
|
19
|
+
}
|
|
20
|
+
if (msg.op === "kv.delete") {
|
|
21
|
+
rows.delete(msg.key);
|
|
22
|
+
return { ok: true };
|
|
23
|
+
}
|
|
24
|
+
if (msg.op === "kv.list") {
|
|
25
|
+
const keys = [...rows.keys()].filter((k) => k.startsWith(msg.prefix) && (!msg.cursor || k > msg.cursor)).sort();
|
|
26
|
+
const limit = msg.limit ?? keys.length;
|
|
27
|
+
const page = keys.slice(0, limit);
|
|
28
|
+
const complete = keys.length <= limit;
|
|
29
|
+
return {
|
|
30
|
+
ok: true,
|
|
31
|
+
keys: page,
|
|
32
|
+
entries: page.map((k) => ({ name: k, expiration: rows.get(k).expiration ?? null, metadata: rows.get(k).metadata ?? null })),
|
|
33
|
+
list_complete: complete,
|
|
34
|
+
cursor: complete ? "" : page[page.length - 1],
|
|
35
|
+
};
|
|
36
|
+
}
|
|
37
|
+
return { ok: false, error: "unknown op" };
|
|
38
|
+
})();
|
|
39
|
+
return JSON.stringify(reply);
|
|
40
|
+
};
|
|
41
|
+
// oxlint-disable-next-line no-function-constructor -- the prelude is prepended as text in production too
|
|
42
|
+
const { makeEnv } = new Function(
|
|
43
|
+
"Porffor",
|
|
44
|
+
"__testCall",
|
|
45
|
+
`${source}\n__sbCall = __testCall; return { makeEnv: (target) => { __sbInstallBindings(target, { kv: ["KV"] }); return target; } };`,
|
|
46
|
+
)({ c: () => "" }, call);
|
|
47
|
+
const fresh = (date) => {
|
|
48
|
+
globalThis.__sbCompat = date;
|
|
49
|
+
rows = new Map();
|
|
50
|
+
return makeEnv({}).KV;
|
|
51
|
+
};
|
|
52
|
+
const failure = async (fn) => {
|
|
53
|
+
try {
|
|
54
|
+
await fn();
|
|
55
|
+
} catch (error) {
|
|
56
|
+
return `${error.name}: ${error.message}`;
|
|
57
|
+
}
|
|
58
|
+
return "did not throw";
|
|
59
|
+
};
|
|
60
|
+
|
|
61
|
+
test("before 2026-10-25: the synchronous shim, unchanged", () => {
|
|
62
|
+
const KV = fresh("2026-10-24");
|
|
63
|
+
expect(KV.put("a", "v")).toBeUndefined();
|
|
64
|
+
expect(KV.get("a")).toBe("v");
|
|
65
|
+
expect(KV.list("a")).toEqual(["a"]);
|
|
66
|
+
expect(KV.getWithMetadata).toBeUndefined();
|
|
67
|
+
});
|
|
68
|
+
|
|
69
|
+
test("from 2026-10-25: Promises, metadata, typed gets and get(keys)", async () => {
|
|
70
|
+
const KV = fresh("2026-10-25");
|
|
71
|
+
expect(KV.put("a", "v", { metadata: { n: 1 } })).toBeInstanceOf(Promise);
|
|
72
|
+
await KV.put("a", '{"x":1}', { metadata: { n: 1 } });
|
|
73
|
+
expect(await KV.get("a", "json")).toEqual({ x: 1 });
|
|
74
|
+
expect(await KV.get("a", { type: "text" })).toBe('{"x":1}');
|
|
75
|
+
expect(await KV.getWithMetadata("a", "json")).toEqual({ value: { x: 1 }, metadata: { n: 1 }, cacheStatus: null });
|
|
76
|
+
expect(await KV.getWithMetadata("missing")).toEqual({ value: null, metadata: null, cacheStatus: null });
|
|
77
|
+
await KV.put("b", new Uint8Array([0, 255]));
|
|
78
|
+
expect(Array.from(new Uint8Array(await KV.get("b", "arrayBuffer")))).toEqual([0, 255]);
|
|
79
|
+
// (ReadableStream values and the "stream" type use the prelude's own streams: checked end to
|
|
80
|
+
// end by sproutd's conformance/bindings/kv.js, not here against Bun's.)
|
|
81
|
+
const many = await KV.get(["a", "missing"]);
|
|
82
|
+
expect(many).toBeInstanceOf(Map);
|
|
83
|
+
expect([...many.entries()]).toEqual([["a", '{"x":1}'], ["missing", null]]);
|
|
84
|
+
expect(await KV.delete("a")).toBeUndefined();
|
|
85
|
+
expect(await KV.get("a")).toBeNull();
|
|
86
|
+
});
|
|
87
|
+
|
|
88
|
+
test("from 2026-10-25: KV's errors for keys, values, expiration and metadata", async () => {
|
|
89
|
+
const KV = fresh("2026-10-25");
|
|
90
|
+
expect(await failure(() => KV.put("", "v"))).toBe("TypeError: Key name cannot be empty.");
|
|
91
|
+
expect(await failure(() => KV.put("..", "v"))).toBe('TypeError: ".." is not allowed as a key name.');
|
|
92
|
+
expect(await failure(() => KV.put("k".repeat(513), "v"))).toBe("Error: KV PUT failed: 414 UTF-8 encoded length of 513 exceeds key length limit of 512.");
|
|
93
|
+
expect(await failure(() => KV.get("k".repeat(513)))).toBe("Error: KV GET failed: 414 UTF-8 encoded length of 513 exceeds key length limit of 512.");
|
|
94
|
+
expect(await failure(() => KV.put("a", { a: 1 }))).toBe("TypeError: KV put() accepts only strings, ArrayBuffers, ArrayBufferViews, and ReadableStreams as values.");
|
|
95
|
+
expect(await failure(() => KV.put("a", "v", { expirationTtl: 30 }))).toBe("Error: KV PUT failed: 400 Invalid expiration_ttl of 30. Expiration TTL must be at least 60.");
|
|
96
|
+
expect(await failure(() => KV.put("a", "v", { expirationTtl: "soon" }))).toBe("Error: KV PUT failed: 400 Invalid expiration_ttl of 0. Please specify integer greater than 0.");
|
|
97
|
+
expect(await failure(() => KV.put("a", "v", { expiration: 1000 }))).toBe(
|
|
98
|
+
"Error: KV PUT failed: 400 Invalid expiration of 1000. Please specify integer greater than the current number of seconds since the UNIX epoch.",
|
|
99
|
+
);
|
|
100
|
+
expect(await failure(() => KV.put("a", "v", { metadata: { big: "x".repeat(1100) } }))).toBe("Error: KV PUT failed: 413 Metadata length of 1110 exceeds limit of 1024.");
|
|
101
|
+
expect(await KV.get("missing", "nope")).toBeNull();
|
|
102
|
+
await KV.put("there", "v");
|
|
103
|
+
expect(await failure(() => KV.get("there", "nope"))).toBe('TypeError: Unknown response type. Possible types are "text", "arrayBuffer", "json", and "stream".');
|
|
104
|
+
expect(await failure(() => KV.list({ limit: 1001 }))).toBe("Error: KV GET failed: 400 Invalid key_count_limit of 1001. Please specify an integer less than 1000.");
|
|
105
|
+
// A failing get rejects rather than throws.
|
|
106
|
+
const rejected = KV.get("");
|
|
107
|
+
expect(rejected).toBeInstanceOf(Promise);
|
|
108
|
+
expect(await rejected.catch((e) => e.message)).toBe("Key name cannot be empty.");
|
|
109
|
+
});
|
|
110
|
+
|
|
111
|
+
test("from 2026-10-25: list() pages with limit and cursor, with expiration and metadata", async () => {
|
|
112
|
+
const KV = fresh("2026-10-25");
|
|
113
|
+
for (const k of ["b", "a", "c"]) await KV.put(k, "v", k === "a" ? { metadata: { t: 1 }, expirationTtl: 3600 } : undefined);
|
|
114
|
+
const first = await KV.list({ limit: 2 });
|
|
115
|
+
expect(first.keys.map((k) => k.name)).toEqual(["a", "b"]);
|
|
116
|
+
expect(first.keys[0].metadata).toEqual({ t: 1 });
|
|
117
|
+
expect(Number.isInteger(first.keys[0].expiration)).toBe(true);
|
|
118
|
+
expect("expiration" in first.keys[1]).toBe(false);
|
|
119
|
+
expect(first.list_complete).toBe(false);
|
|
120
|
+
const rest = await KV.list({ limit: 2, cursor: first.cursor });
|
|
121
|
+
expect(rest.keys.map((k) => k.name)).toEqual(["c"]);
|
|
122
|
+
expect(Object.keys(rest).sort()).toEqual(["cacheStatus", "keys", "list_complete"]);
|
|
123
|
+
expect(await KV.list({ limit: 0 })).toEqual({ keys: [], list_complete: true, cacheStatus: null });
|
|
124
|
+
});
|
|
@@ -4807,6 +4807,126 @@ function __sbRpc(op, extra) {
|
|
|
4807
4807
|
return reply;
|
|
4808
4808
|
}
|
|
4809
4809
|
|
|
4810
|
+
// KV from compatibility date 2026-10-25: workerd's surface. Every method returns a
|
|
4811
|
+
// Promise; keys and options are checked with KV's errors; get() takes a list of keys and
|
|
4812
|
+
// the "stream" type; getWithMetadata(); put() takes a ReadableStream and metadata; list()
|
|
4813
|
+
// pages with limit and cursor and returns each key's expiration and metadata. Earlier
|
|
4814
|
+
// dates keep the synchronous shim below.
|
|
4815
|
+
function __sbKvStrict() {
|
|
4816
|
+
return globalThis.__sbCompat >= "2026-10-25";
|
|
4817
|
+
}
|
|
4818
|
+
function __sbKvKey(key, verb) {
|
|
4819
|
+
const name = String(key);
|
|
4820
|
+
if (name === "") throw new TypeError("Key name cannot be empty.");
|
|
4821
|
+
if (name === "." || name === "..") throw new TypeError('"' + name + '" is not allowed as a key name.');
|
|
4822
|
+
const length = new TextEncoder().encode(name).length;
|
|
4823
|
+
if (length > 512) throw new Error("KV " + verb + " failed: 414 UTF-8 encoded length of " + length + " exceeds key length limit of 512.");
|
|
4824
|
+
return name;
|
|
4825
|
+
}
|
|
4826
|
+
function __sbKvType(typeOrOptions) {
|
|
4827
|
+
return typeOrOptions == null ? "text" : __sbIsStr(typeOrOptions) ? typeOrOptions : typeOrOptions.type || "text";
|
|
4828
|
+
}
|
|
4829
|
+
// An unknown type fails only when there is a value to decode: a missing key is null
|
|
4830
|
+
// whatever the type, as in workerd.
|
|
4831
|
+
function __sbKvDecode(r, type) {
|
|
4832
|
+
if (type !== "text" && type !== "json" && type !== "arrayBuffer" && type !== "stream")
|
|
4833
|
+
throw new TypeError('Unknown response type. Possible types are "text", "arrayBuffer", "json", and "stream".');
|
|
4834
|
+
if (type === "arrayBuffer") return __sbBufFrom(r.binary ? r.value : __sbToBytes(r.value));
|
|
4835
|
+
if (type === "stream") return new Response(__sbBufFrom(r.binary ? r.value : __sbToBytes(r.value))).body;
|
|
4836
|
+
const text = r.binary ? __sbFromUtf8(r.value) : r.value;
|
|
4837
|
+
return type === "json" ? JSON.parse(text) : text;
|
|
4838
|
+
}
|
|
4839
|
+
function __sbKvMetadata(text) {
|
|
4840
|
+
return text == null ? null : JSON.parse(text);
|
|
4841
|
+
}
|
|
4842
|
+
function __sbMakeKv(ns) {
|
|
4843
|
+
const one = (key, type) => {
|
|
4844
|
+
const r = __sbRpc("kv.get", { ns, key: __sbKvKey(key, "GET") });
|
|
4845
|
+
return r.found ? { value: __sbKvDecode(r, type), metadata: __sbKvMetadata(r.metadata) } : { value: null, metadata: null };
|
|
4846
|
+
};
|
|
4847
|
+
const attempt = (run) => {
|
|
4848
|
+
try {
|
|
4849
|
+
return Promise.resolve(run());
|
|
4850
|
+
} catch (err) {
|
|
4851
|
+
return Promise.reject(err);
|
|
4852
|
+
}
|
|
4853
|
+
};
|
|
4854
|
+
return {
|
|
4855
|
+
get(key, typeOrOptions) {
|
|
4856
|
+
return attempt(() => {
|
|
4857
|
+
const type = __sbKvType(typeOrOptions);
|
|
4858
|
+
if (!Array.isArray(key)) return one(key, type).value;
|
|
4859
|
+
const out = new Map();
|
|
4860
|
+
for (let i = 0; i < key.length; i++) out.set(String(key[i]), one(key[i], type).value);
|
|
4861
|
+
return out;
|
|
4862
|
+
});
|
|
4863
|
+
},
|
|
4864
|
+
getWithMetadata(key, typeOrOptions) {
|
|
4865
|
+
return attempt(() => {
|
|
4866
|
+
const r = one(key, __sbKvType(typeOrOptions));
|
|
4867
|
+
return { value: r.value, metadata: r.metadata, cacheStatus: null };
|
|
4868
|
+
});
|
|
4869
|
+
},
|
|
4870
|
+
async put(key, value, options) {
|
|
4871
|
+
const msg = { ns, key: __sbKvKey(key, "PUT"), value: "" };
|
|
4872
|
+
value = __sbUnwrapBody(value);
|
|
4873
|
+
if (value != null && __sbIsFn(value.getReader)) value = new Uint8Array(await new Response(value).arrayBuffer());
|
|
4874
|
+
if (value != null && !__sbIsStr(value) && value.byteLength !== undefined) {
|
|
4875
|
+
const bytes = value.buffer !== undefined ? new Uint8Array(value.buffer, value.byteOffset, value.byteLength) : new Uint8Array(value);
|
|
4876
|
+
msg.value = __sbToBytes(bytes);
|
|
4877
|
+
msg.binary = true;
|
|
4878
|
+
} else if (__sbIsStr(value) || value === Number(value) || value === true || value === false) msg.value = String(value);
|
|
4879
|
+
else throw new TypeError("KV put() accepts only strings, ArrayBuffers, ArrayBufferViews, and ReadableStreams as values.");
|
|
4880
|
+
const o = options || {};
|
|
4881
|
+
if (o.expirationTtl != null) {
|
|
4882
|
+
const ttl = Math.floor(Number(o.expirationTtl)) || 0;
|
|
4883
|
+
if (ttl <= 0) throw new Error("KV PUT failed: 400 Invalid expiration_ttl of " + ttl + ". Please specify integer greater than 0.");
|
|
4884
|
+
if (ttl < 60) throw new Error("KV PUT failed: 400 Invalid expiration_ttl of " + ttl + ". Expiration TTL must be at least 60.");
|
|
4885
|
+
msg.expirationTtl = ttl;
|
|
4886
|
+
} else if (o.expiration != null) {
|
|
4887
|
+
const at = Math.floor(Number(o.expiration)) || 0;
|
|
4888
|
+
if (at <= Math.floor(Date.now() / 1000))
|
|
4889
|
+
throw new Error("KV PUT failed: 400 Invalid expiration of " + at + ". Please specify integer greater than the current number of seconds since the UNIX epoch.");
|
|
4890
|
+
msg.expiration = at;
|
|
4891
|
+
}
|
|
4892
|
+
if (o.metadata !== undefined) {
|
|
4893
|
+
const json = JSON.stringify(o.metadata);
|
|
4894
|
+
const length = new TextEncoder().encode(json).length;
|
|
4895
|
+
if (length > 1024) throw new Error("KV PUT failed: 413 Metadata length of " + length + " exceeds limit of 1024.");
|
|
4896
|
+
msg.metadata = json;
|
|
4897
|
+
}
|
|
4898
|
+
__sbRpc("kv.put", msg);
|
|
4899
|
+
},
|
|
4900
|
+
delete(key) {
|
|
4901
|
+
return attempt(() => {
|
|
4902
|
+
__sbRpc("kv.delete", { ns, key: __sbKvKey(key, "DELETE") });
|
|
4903
|
+
});
|
|
4904
|
+
},
|
|
4905
|
+
list(options) {
|
|
4906
|
+
return attempt(() => {
|
|
4907
|
+
const o = options || {};
|
|
4908
|
+
const limit = o.limit == null ? 1000 : Math.floor(Number(o.limit));
|
|
4909
|
+
if (!(limit >= 0) || limit > 1000)
|
|
4910
|
+
throw new Error("KV GET failed: 400 Invalid key_count_limit of " + o.limit + ". Please specify an integer less than 1000.");
|
|
4911
|
+
if (limit === 0) return { keys: [], list_complete: true, cacheStatus: null };
|
|
4912
|
+
const r = __sbRpc("kv.list", { ns, prefix: o.prefix == null ? "" : String(o.prefix), limit, cursor: o.cursor == null ? "" : String(o.cursor) });
|
|
4913
|
+
const keys = [];
|
|
4914
|
+
const entries = r.entries || [];
|
|
4915
|
+
for (let i = 0; i < entries.length; i++) {
|
|
4916
|
+
const k = { name: entries[i].name };
|
|
4917
|
+
if (entries[i].expiration != null) k.expiration = entries[i].expiration;
|
|
4918
|
+
if (entries[i].metadata != null) k.metadata = __sbKvMetadata(entries[i].metadata);
|
|
4919
|
+
keys.push(k);
|
|
4920
|
+
}
|
|
4921
|
+
const out = { keys, list_complete: r.list_complete !== false };
|
|
4922
|
+
if (!out.list_complete) out.cursor = r.cursor;
|
|
4923
|
+
out.cacheStatus = null;
|
|
4924
|
+
return out;
|
|
4925
|
+
});
|
|
4926
|
+
},
|
|
4927
|
+
};
|
|
4928
|
+
}
|
|
4929
|
+
|
|
4810
4930
|
// D1: a Cloudflare-shaped `env.<DB>` (prepare / bind / all / run / raw / first,
|
|
4811
4931
|
// plus batch and exec). Every call is one broker roundtrip.
|
|
4812
4932
|
//
|
|
@@ -5185,7 +5305,7 @@ globalThis.__sbInstallBindings = function (target, bindings) {
|
|
|
5185
5305
|
|
|
5186
5306
|
for (let i = 0; i < (bindings.kv || []).length; i++) {
|
|
5187
5307
|
const ns = bindings.kv[i];
|
|
5188
|
-
target[ns] = {
|
|
5308
|
+
target[ns] = __sbKvStrict() ? __sbMakeKv(ns) : {
|
|
5189
5309
|
// #233 — Workers' get(key, type | { type }): "text" (default), "json",
|
|
5190
5310
|
// "arrayBuffer". A value put as bytes is stored one char per byte and
|
|
5191
5311
|
// flagged, so text decodes it as UTF-8 and arrayBuffer returns it
|
|
@@ -419,9 +419,26 @@ static int sb_ltx_begin(sb_ltx_writer *w, const char *path, sb_ltx_state *s, uin
|
|
|
419
419
|
memset(w, 0, sizeof(*w));
|
|
420
420
|
w->hash = UINT64_MAX;
|
|
421
421
|
w->page_size = s->page_size;
|
|
422
|
+
// Global temporary directories are outside a sandboxed worker's writable tree.
|
|
423
|
+
// After unlink, close (including process death) reclaims the scratch file.
|
|
424
|
+
char index_path[SB_LTX_PATH];
|
|
425
|
+
if (snprintf(index_path, sizeof(index_path), "%s/index-XXXXXX", s->root) >= sizeof(index_path))
|
|
426
|
+
return -1;
|
|
427
|
+
int index = mkstemp(index_path);
|
|
428
|
+
if (index < 0)
|
|
429
|
+
return -1;
|
|
430
|
+
sb_ltx_phase("index-created");
|
|
431
|
+
if (unlink(index_path)) {
|
|
432
|
+
close(index);
|
|
433
|
+
return -1;
|
|
434
|
+
}
|
|
435
|
+
w->index = fdopen(index, "w+b");
|
|
436
|
+
if (!w->index) {
|
|
437
|
+
close(index);
|
|
438
|
+
return -1;
|
|
439
|
+
}
|
|
422
440
|
w->out = fopen(path, "wbx");
|
|
423
|
-
w->
|
|
424
|
-
if (!w->out || !w->index)
|
|
441
|
+
if (!w->out)
|
|
425
442
|
return -1;
|
|
426
443
|
unsigned char h[100] = {0};
|
|
427
444
|
memcpy(h, "LTX1", 4);
|
|
@@ -1048,6 +1065,41 @@ done:
|
|
|
1048
1065
|
return 1;
|
|
1049
1066
|
}
|
|
1050
1067
|
|
|
1068
|
+
// The capture lock is already held. Remove only reserved, regular scratch files;
|
|
1069
|
+
// do not open them (a hard link could share the database's POSIX-locked inode).
|
|
1070
|
+
static int sb_ltx_sweep_indexes(const char *root) {
|
|
1071
|
+
DIR *dir = opendir(root);
|
|
1072
|
+
if (!dir)
|
|
1073
|
+
return -1;
|
|
1074
|
+
int rc = 0;
|
|
1075
|
+
for (;;) {
|
|
1076
|
+
errno = 0;
|
|
1077
|
+
struct dirent *entry = readdir(dir);
|
|
1078
|
+
if (!entry) {
|
|
1079
|
+
if (errno)
|
|
1080
|
+
rc = -1;
|
|
1081
|
+
break;
|
|
1082
|
+
}
|
|
1083
|
+
const char *name = entry->d_name;
|
|
1084
|
+
if (strlen(name) != 12 || strncmp(name, "index-", 6) ||
|
|
1085
|
+
strspn(name + 6, "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789") != 6)
|
|
1086
|
+
continue;
|
|
1087
|
+
struct stat st;
|
|
1088
|
+
if (fstatat(dirfd(dir), name, &st, AT_SYMLINK_NOFOLLOW)) {
|
|
1089
|
+
if (errno == ENOENT)
|
|
1090
|
+
continue;
|
|
1091
|
+
rc = -1;
|
|
1092
|
+
break;
|
|
1093
|
+
}
|
|
1094
|
+
if (S_ISREG(st.st_mode) && unlinkat(dirfd(dir), name, 0) && errno != ENOENT) {
|
|
1095
|
+
rc = -1;
|
|
1096
|
+
break;
|
|
1097
|
+
}
|
|
1098
|
+
}
|
|
1099
|
+
if (closedir(dir))
|
|
1100
|
+
rc = -1;
|
|
1101
|
+
return rc;
|
|
1102
|
+
}
|
|
1051
1103
|
static int sb_ltx_start(int idx, sqlite3 *db, const char *path, int lock) {
|
|
1052
1104
|
sb_ltx_state *s = &sb_ltx_states[idx];
|
|
1053
1105
|
memset(s, 0, sizeof(*s));
|
|
@@ -1059,6 +1111,8 @@ static int sb_ltx_start(int idx, sqlite3 *db, const char *path, int lock) {
|
|
|
1059
1111
|
return -1;
|
|
1060
1112
|
if (mkdir(s->root, 0700) && errno != EEXIST)
|
|
1061
1113
|
return -1;
|
|
1114
|
+
if (sb_ltx_sweep_indexes(s->root))
|
|
1115
|
+
return -1;
|
|
1062
1116
|
char parent[SB_LTX_PATH];
|
|
1063
1117
|
snprintf(parent, sizeof(parent), "%s", path);
|
|
1064
1118
|
char *slash = strrchr(parent, '/');
|
|
@@ -2137,12 +2191,12 @@ static int sb_db_page_size(const char* path) {
|
|
|
2137
2191
|
if (st) sqlite3_finalize(st);
|
|
2138
2192
|
return size;
|
|
2139
2193
|
}
|
|
2140
|
-
// Freeze (refuse writes to) or thaw a database for the quota. Returns 1
|
|
2141
|
-
// is under LTX capture, 0 when not, -1 when it cannot be opened.
|
|
2194
|
+
// Freeze (refuse writes to) or thaw a database for the quota; on < 0 only asks. Returns 1
|
|
2195
|
+
// when the database is under LTX capture, 0 when not, -1 when it cannot be opened.
|
|
2142
2196
|
static int sb_db_quota_freeze(const char* path, int on) {
|
|
2143
2197
|
int idx = sb_db_for(path);
|
|
2144
2198
|
if (idx < 0) return -1;
|
|
2145
|
-
sb_quota_frozen[idx] = on && sb_ltx_states[idx].db ? 1 : 0;
|
|
2199
|
+
if (on >= 0) sb_quota_frozen[idx] = on && sb_ltx_states[idx].db ? 1 : 0;
|
|
2146
2200
|
return sb_ltx_states[idx].db ? 1 : 0;
|
|
2147
2201
|
}
|
|
2148
2202
|
|
|
@@ -3197,7 +3251,7 @@ function __sbD1DbRaw(path, what) {
|
|
|
3197
3251
|
if (__po) free(__po);
|
|
3198
3252
|
const char* __w; size_t __wl; char* __wo = 0;
|
|
3199
3253
|
porf_native_fetch_read_value(what, &__w, &__wl, &__wo);
|
|
3200
|
-
int __r = __wl && __w[0] == 'p' ? sb_db_page_size(__path) : sb_db_quota_freeze(__path, __wl && __w[0] == 'f');
|
|
3254
|
+
int __r = __wl && __w[0] == 'p' ? sb_db_page_size(__path) : __wl && __w[0] == 'c' ? sb_db_quota_freeze(__path, -1) : sb_db_quota_freeze(__path, __wl && __w[0] == 'f');
|
|
3201
3255
|
if (__wo) free(__wo);
|
|
3202
3256
|
free(__path);
|
|
3203
3257
|
char __n[16]; int __k = snprintf(__n, sizeof(__n), "%d", __r);
|
|
@@ -3455,11 +3509,31 @@ var __sbQuotaOps = {
|
|
|
3455
3509
|
// A database under LTX capture is frozen instead: every write is refused, because even a
|
|
3456
3510
|
// DELETE adds a file to the replication spool, which counts toward the quota.
|
|
3457
3511
|
var __sbD1Capped = {};
|
|
3512
|
+
var __sbD1Captured = {};
|
|
3513
|
+
var __sbD1Checkpointed = {};
|
|
3458
3514
|
function __sbD1Quota(path) {
|
|
3459
|
-
|
|
3515
|
+
// Whether capture owns this database, from the private helper (no SQL), before anything
|
|
3516
|
+
// below depends on it. A database that cannot open (or whose capture failed) is left to
|
|
3517
|
+
// the call itself, which reports why.
|
|
3518
|
+
const status = Number(__sbD1DbRaw(path, "captured"));
|
|
3519
|
+
if (status < 0) return;
|
|
3520
|
+
__sbD1Captured[path] = status === 1;
|
|
3521
|
+
let full = __sbQuotaFull();
|
|
3522
|
+
// Over the quota, a database's -wal file can hold space the app has already freed (a
|
|
3523
|
+
// DELETE, a VACUUM): checkpoint it into the database and truncate it, at most once a
|
|
3524
|
+
// second, then measure again. Not under capture: LTX owns that database's checkpoints.
|
|
3525
|
+
if (full && !__sbD1Captured[path] && Date.now() - (__sbD1Checkpointed[path] || 0) >= 1000) {
|
|
3526
|
+
__sbD1Checkpointed[path] = Date.now();
|
|
3527
|
+
JSON.parse(__sbSqlRaw(path, "PRAGMA wal_checkpoint(TRUNCATE)", "[]"));
|
|
3528
|
+
__sbQuotaAt = -1;
|
|
3529
|
+
full = __sbQuotaFull();
|
|
3530
|
+
}
|
|
3460
3531
|
if (__sbD1Capped[path] === full) return;
|
|
3461
3532
|
__sbD1Capped[path] = full;
|
|
3462
|
-
const
|
|
3533
|
+
const frozen = Number(__sbD1DbRaw(path, full ? "freeze" : "thaw"));
|
|
3534
|
+
if (frozen < 0) return;
|
|
3535
|
+
const captured = frozen === 1;
|
|
3536
|
+
__sbD1Captured[path] = captured;
|
|
3463
3537
|
if (full && captured) return;
|
|
3464
3538
|
const size = Number(__sbD1DbRaw(path, "page_size")) || 4096;
|
|
3465
3539
|
const pages = full
|
|
@@ -3499,6 +3573,10 @@ function __sbEnsureSchema() {
|
|
|
3499
3573
|
let hasKvBinary = false;
|
|
3500
3574
|
for (let i = 0; i < kvColumns.length; i++) if (kvColumns[i][1] === "binary") hasKvBinary = true;
|
|
3501
3575
|
if (!hasKvBinary) __sbSql(s, "ALTER TABLE kv ADD COLUMN binary INTEGER NOT NULL DEFAULT 0", []);
|
|
3576
|
+
// A key's metadata (put(key, value, { metadata })), as JSON text; same column as the broker's.
|
|
3577
|
+
let hasKvMetadata = false;
|
|
3578
|
+
for (let i = 0; i < kvColumns.length; i++) if (kvColumns[i][1] === "metadata") hasKvMetadata = true;
|
|
3579
|
+
if (!hasKvMetadata) __sbSql(s, "ALTER TABLE kv ADD COLUMN metadata TEXT", []);
|
|
3502
3580
|
// #56 — no `body` column: an object's bytes live in their own file under
|
|
3503
3581
|
// "<data-dir>/r2-blobs/" (sb_r2_put_c/sb_r2_get_c), never in the sqlite row.
|
|
3504
3582
|
__sbSql(
|
|
@@ -3602,7 +3680,7 @@ function __sbEmbeddedDispatch(msg) {
|
|
|
3602
3680
|
if (op === "ping") return { ok: true, op: "pong", echo: msg.msg };
|
|
3603
3681
|
|
|
3604
3682
|
if (op === "kv.get") {
|
|
3605
|
-
const r = __sbSql(store, "SELECT value, expires_at, binary FROM kv WHERE ns = ? AND key = ?", [
|
|
3683
|
+
const r = __sbSql(store, "SELECT value, expires_at, binary, metadata FROM kv WHERE ns = ? AND key = ?", [
|
|
3606
3684
|
msg.ns,
|
|
3607
3685
|
msg.key,
|
|
3608
3686
|
]);
|
|
@@ -3612,7 +3690,14 @@ function __sbEmbeddedDispatch(msg) {
|
|
|
3612
3690
|
__sbSql(store, "DELETE FROM kv WHERE ns = ? AND key = ?", [msg.ns, msg.key]);
|
|
3613
3691
|
return { ok: true, found: false, value: null };
|
|
3614
3692
|
}
|
|
3615
|
-
return {
|
|
3693
|
+
return {
|
|
3694
|
+
ok: true,
|
|
3695
|
+
found: true,
|
|
3696
|
+
value: r.rows[0][0],
|
|
3697
|
+
binary: r.rows[0][2] === 1,
|
|
3698
|
+
metadata: r.rows[0][3],
|
|
3699
|
+
expiration: expiresAt == null ? null : Math.floor(expiresAt / 1000),
|
|
3700
|
+
};
|
|
3616
3701
|
}
|
|
3617
3702
|
if (op === "kv.put") {
|
|
3618
3703
|
// #58 — CF rejects expirationTtl under 60 seconds; matched here.
|
|
@@ -3626,8 +3711,8 @@ function __sbEmbeddedDispatch(msg) {
|
|
|
3626
3711
|
}
|
|
3627
3712
|
__sbSql(
|
|
3628
3713
|
store,
|
|
3629
|
-
"INSERT INTO kv (ns, key, value, expires_at, binary) VALUES (?1,?2,?3,?4,?5) ON CONFLICT (ns, key) DO UPDATE SET value = ?3, expires_at = ?4, binary = ?5",
|
|
3630
|
-
[msg.ns, msg.key, msg.value, expiresAt, msg.binary === true ? 1 : 0],
|
|
3714
|
+
"INSERT INTO kv (ns, key, value, expires_at, binary, metadata) VALUES (?1,?2,?3,?4,?5,?6) ON CONFLICT (ns, key) DO UPDATE SET value = ?3, expires_at = ?4, binary = ?5, metadata = ?6",
|
|
3715
|
+
[msg.ns, msg.key, msg.value, expiresAt, msg.binary === true ? 1 : 0, msg.metadata == null ? null : String(msg.metadata)],
|
|
3631
3716
|
);
|
|
3632
3717
|
return { ok: true };
|
|
3633
3718
|
}
|
|
@@ -3636,14 +3721,28 @@ function __sbEmbeddedDispatch(msg) {
|
|
|
3636
3721
|
return { ok: true };
|
|
3637
3722
|
}
|
|
3638
3723
|
if (op === "kv.list") {
|
|
3724
|
+
// The prefix matches exactly: LIKE ignored case and read "_" and "%" in it as wildcards.
|
|
3725
|
+
// With a limit (the shim from 2026-10-25), a page of entries after the cursor (the last
|
|
3726
|
+
// key of the previous page), with each key's expiration and metadata; without one,
|
|
3727
|
+
// every key, as before.
|
|
3728
|
+
const prefix = msg.prefix || "";
|
|
3729
|
+
const paged = msg.limit != null;
|
|
3730
|
+
const limit = paged ? Math.max(0, Math.floor(Number(msg.limit))) : -1;
|
|
3639
3731
|
const r = __sbSql(
|
|
3640
3732
|
store,
|
|
3641
|
-
"SELECT key FROM kv WHERE ns = ? AND key
|
|
3642
|
-
|
|
3733
|
+
"SELECT key, expires_at, metadata FROM kv WHERE ns = ?1 AND substr(key, 1, length(?2)) = ?2 AND key > ?3 " +
|
|
3734
|
+
"AND (expires_at IS NULL OR expires_at > ?4) ORDER BY key LIMIT ?5",
|
|
3735
|
+
[msg.ns, prefix, paged && msg.cursor ? String(msg.cursor) : "", Date.now(), paged ? limit + 1 : -1],
|
|
3643
3736
|
);
|
|
3644
3737
|
const keys = [];
|
|
3645
|
-
|
|
3646
|
-
|
|
3738
|
+
const entries = [];
|
|
3739
|
+
const count = paged ? Math.min(r.rows.length, limit) : r.rows.length;
|
|
3740
|
+
for (let i = 0; i < count; i++) {
|
|
3741
|
+
keys.push(r.rows[i][0]);
|
|
3742
|
+
entries.push({ name: r.rows[i][0], expiration: r.rows[i][1] == null ? null : Math.floor(r.rows[i][1] / 1000), metadata: r.rows[i][2] });
|
|
3743
|
+
}
|
|
3744
|
+
const complete = !paged || r.rows.length <= limit;
|
|
3745
|
+
return { ok: true, keys, entries, list_complete: complete, cursor: complete ? "" : keys[keys.length - 1] };
|
|
3647
3746
|
}
|
|
3648
3747
|
|
|
3649
3748
|
if (op === "assets.get") {
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
// LTX chain restore, the same independent checker as native-ltx.ts (copied, so that test
|
|
2
|
+
// stays as it is): every file's CRCs, checksums and TXIDs verified, pages applied in order.
|
|
3
|
+
import assert from "node:assert/strict";
|
|
4
|
+
import { readFile, readdir, writeFile } from "node:fs/promises";
|
|
5
|
+
import { join } from "node:path";
|
|
6
|
+
|
|
7
|
+
// Independent CRC baseline in Bun. This intentionally uses BigInt, unlike C.
|
|
8
|
+
const table: bigint[] = [];
|
|
9
|
+
for (let i = 0; i < 256; i++) {
|
|
10
|
+
let c = BigInt(i);
|
|
11
|
+
for (let j = 0; j < 8; j++) c = (c >> 1n) ^ (c & 1n ? 0xd800000000000000n : 0n);
|
|
12
|
+
table.push(c);
|
|
13
|
+
}
|
|
14
|
+
function crc(data: Uint8Array, init = 0xffffffffffffffffn) {
|
|
15
|
+
let c = init;
|
|
16
|
+
for (const b of data) c = table[Number((c ^ BigInt(b)) & 255n)] ^ (c >> 8n);
|
|
17
|
+
return c;
|
|
18
|
+
}
|
|
19
|
+
const flag = 0x8000000000000000n,
|
|
20
|
+
mask = 0xffffffffffffffffn;
|
|
21
|
+
function pageCrc(pgno: number, page: Uint8Array) {
|
|
22
|
+
const p = new Uint8Array(4);
|
|
23
|
+
new DataView(p.buffer).setUint32(0, pgno);
|
|
24
|
+
return flag | (crc(page, crc(p)) ^ mask);
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
export async function restore(epoch: string, output: string) {
|
|
28
|
+
const files = (await readdir(epoch)).filter((n) => n.endsWith(".ltx")).sort();
|
|
29
|
+
let db = new Uint8Array(0),
|
|
30
|
+
checksum = 0n,
|
|
31
|
+
txid = 0n;
|
|
32
|
+
for (const file of files) {
|
|
33
|
+
const b = new Uint8Array(await readFile(join(epoch, file)));
|
|
34
|
+
const v = new DataView(b.buffer, b.byteOffset, b.byteLength);
|
|
35
|
+
assert.equal(new TextDecoder().decode(b.subarray(0, 4)), "LTX1");
|
|
36
|
+
const pageSize = v.getUint32(8),
|
|
37
|
+
commit = v.getUint32(12);
|
|
38
|
+
assert.equal(v.getBigUint64(16), txid + 1n);
|
|
39
|
+
assert.equal(v.getBigUint64(40), checksum);
|
|
40
|
+
const next = new Uint8Array(commit * pageSize);
|
|
41
|
+
next.set(db.subarray(0, next.length));
|
|
42
|
+
db = next;
|
|
43
|
+
let offset = 100,
|
|
44
|
+
hash = crc(b.subarray(0, 100));
|
|
45
|
+
const entries = [];
|
|
46
|
+
for (;;) {
|
|
47
|
+
const begin = offset,
|
|
48
|
+
pgno = v.getUint32(offset),
|
|
49
|
+
flags = v.getUint16(offset + 4);
|
|
50
|
+
hash = crc(b.subarray(offset, offset + 6), hash);
|
|
51
|
+
offset += 6;
|
|
52
|
+
if (!pgno) {
|
|
53
|
+
assert.equal(flags, 0);
|
|
54
|
+
break;
|
|
55
|
+
}
|
|
56
|
+
assert.equal(flags, 1);
|
|
57
|
+
assert(pgno <= commit);
|
|
58
|
+
const size = v.getUint32(offset);
|
|
59
|
+
hash = crc(b.subarray(offset, offset + 4), hash);
|
|
60
|
+
offset += 4;
|
|
61
|
+
const stop = offset + size;
|
|
62
|
+
assert(stop <= b.length);
|
|
63
|
+
assert.equal(b[offset++], 0xf0);
|
|
64
|
+
let literals = 15,
|
|
65
|
+
byte;
|
|
66
|
+
do {
|
|
67
|
+
byte = b[offset++];
|
|
68
|
+
literals += byte;
|
|
69
|
+
} while (byte === 255);
|
|
70
|
+
assert.equal(literals, pageSize);
|
|
71
|
+
assert.equal(offset + pageSize, stop);
|
|
72
|
+
const page = b.subarray(offset, stop);
|
|
73
|
+
db.set(page, (pgno - 1) * pageSize);
|
|
74
|
+
hash = crc(page, hash);
|
|
75
|
+
offset = stop;
|
|
76
|
+
entries.push({ pgno, offset: begin, size: stop - begin });
|
|
77
|
+
}
|
|
78
|
+
const indexStart = offset;
|
|
79
|
+
function varint() {
|
|
80
|
+
let n = 0n,
|
|
81
|
+
s = 0n;
|
|
82
|
+
for (let i = 0; i < 10; i++) {
|
|
83
|
+
assert(offset < b.length);
|
|
84
|
+
const c = b[offset++];
|
|
85
|
+
n |= BigInt(c & 127) << s;
|
|
86
|
+
if (c < 128) return Number(n);
|
|
87
|
+
s += 7n;
|
|
88
|
+
}
|
|
89
|
+
throw new Error("varint overflow");
|
|
90
|
+
}
|
|
91
|
+
for (const entry of entries) {
|
|
92
|
+
assert.equal(varint(), entry.pgno);
|
|
93
|
+
assert.equal(varint(), entry.offset);
|
|
94
|
+
assert.equal(varint(), entry.size);
|
|
95
|
+
}
|
|
96
|
+
assert.equal(varint(), 0);
|
|
97
|
+
assert.equal(v.getBigUint64(offset), BigInt(offset - indexStart));
|
|
98
|
+
offset += 8;
|
|
99
|
+
hash = crc(b.subarray(indexStart, offset + 8), hash);
|
|
100
|
+
checksum = flag;
|
|
101
|
+
for (let p = 1; p <= commit; p++)
|
|
102
|
+
checksum = flag | (checksum ^ pageCrc(p, db.subarray((p - 1) * pageSize, p * pageSize)));
|
|
103
|
+
assert.equal(v.getBigUint64(offset), checksum, file);
|
|
104
|
+
assert.equal(v.getBigUint64(offset + 8), flag | (hash ^ mask), file);
|
|
105
|
+
assert.equal(offset + 16, b.length);
|
|
106
|
+
txid = v.getBigUint64(24);
|
|
107
|
+
}
|
|
108
|
+
await writeFile(output, db);
|
|
109
|
+
return { files: files.length, txid };
|
|
110
|
+
}
|
package/test/native-d1.ts
CHANGED
|
@@ -1,4 +1,6 @@
|
|
|
1
1
|
import assert from "node:assert/strict";
|
|
2
|
+
import { Database } from "bun:sqlite";
|
|
3
|
+
import { restore } from "./ltx-restore";
|
|
2
4
|
import { closeSync, openSync, readFileSync, readdirSync, readSync, writeSync } from "node:fs";
|
|
3
5
|
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
|
4
6
|
import { tmpdir } from "node:os";
|
|
@@ -47,7 +49,7 @@ int main(void) {
|
|
|
47
49
|
if (!strncmp(line, ":break ", 7)) { line[strcspn(line, "\\t")] = 0; char p[1024]; snprintf(p, sizeof(p), "%s.ltx/position.tmp", line + 7); mkdir(p, 0700); puts("{}"); fflush(stdout); continue; }
|
|
48
50
|
char* sql = strchr(line, '\\t'); if (!sql) return 2; *sql++ = 0;
|
|
49
51
|
char* params = strchr(sql, '\\t'); if (!params) return 2; *params++ = 0;
|
|
50
|
-
if (!strncmp(line, ":db ", 4)) { int r = sql[0] == 'p' ? sb_db_page_size(line + 4) : sb_db_quota_freeze(line + 4, sql[0] == 'f'); printf("{\\"n\\":%d}\\n", r); fflush(stdout); continue; }
|
|
52
|
+
if (!strncmp(line, ":db ", 4)) { int r = sql[0] == 'p' ? sb_db_page_size(line + 4) : sb_db_quota_freeze(line + 4, sql[0] == 'c' ? -1 : sql[0] == 'f'); printf("{\\"n\\":%d}\\n", r); fflush(stdout); continue; }
|
|
51
53
|
if (!strncmp(line, ":bytes ", 7)) { line[strcspn(line, "\\t")] = 0; printf("{\\"n\\":%lld}\\n", sb_dir_bytes(line + 7, 0)); fflush(stdout); continue; }
|
|
52
54
|
// ":script PATH" runs SQL as a script (sb_sql_script, what exec() uses).
|
|
53
55
|
char* out = !strncmp(line, ":script ", 8) ? sb_sql_script(line + 8, sql) : sb_sql_run(line, sql, params); puts(out); fflush(stdout); free(out);
|
|
@@ -149,7 +151,9 @@ int main(void) {
|
|
|
149
151
|
const toDriver = openSync(fifoIn, "w"),
|
|
150
152
|
fromDriver = openSync(fifoOut, "r");
|
|
151
153
|
let pending = "";
|
|
154
|
+
let driverClosed = false;
|
|
152
155
|
const sqlRaw = (path: string, sql: string, params: string) => {
|
|
156
|
+
if (driverClosed) throw new Error("original D1 driver is closed");
|
|
153
157
|
writeSync(toDriver, `${path}\t${sql}\t${params}\n`);
|
|
154
158
|
const chunk = Buffer.alloc(65536);
|
|
155
159
|
while (!pending.includes("\n")) {
|
|
@@ -185,6 +189,7 @@ int main(void) {
|
|
|
185
189
|
"__sbDirBytesRaw",
|
|
186
190
|
"__sbD1DbRaw",
|
|
187
191
|
"__sbSql",
|
|
192
|
+
"__sbSqlRaw",
|
|
188
193
|
`${transport.slice(quotaStart, quotaEnd)}\nreturn { __sbQuotaCheck, __sbD1Quota, __sbQuotaFull, expire() { __sbQuotaAt = -1; } };`,
|
|
189
194
|
)(
|
|
190
195
|
(name: string) => (name === "SB_STORAGE_MAX_BYTES" ? String(limit) : ""),
|
|
@@ -192,8 +197,70 @@ int main(void) {
|
|
|
192
197
|
(d: string) => String(JSON.parse(sqlRaw(":bytes " + d, "", "")).n),
|
|
193
198
|
(path: string, what: string) => String(JSON.parse(sqlRaw(":db " + path, what, "")).n),
|
|
194
199
|
sql,
|
|
200
|
+
sqlRaw,
|
|
195
201
|
);
|
|
196
202
|
const dirBytes = (d: string) => Number(JSON.parse(sqlRaw(":bytes " + d, "", "")).n);
|
|
203
|
+
// Another driver process (a restarted worker), with its own FIFOs and lifted blocks.
|
|
204
|
+
const connect = (tag: string) => {
|
|
205
|
+
const inPath = join(work, "in-" + tag),
|
|
206
|
+
outPath = join(work, "out-" + tag);
|
|
207
|
+
assert.equal(Bun.spawnSync(["mkfifo", inPath, outPath]).exitCode, 0);
|
|
208
|
+
const child = Bun.spawn(["sh", "-c", `exec "${binary}" < "${inPath}" > "${outPath}"`], {
|
|
209
|
+
env: { ...process.env, SB_LTX_ENABLE: "1", SB_LTX_DATABASE: "LTX" },
|
|
210
|
+
});
|
|
211
|
+
const to = openSync(inPath, "w"),
|
|
212
|
+
from = openSync(outPath, "r");
|
|
213
|
+
let buffered = "";
|
|
214
|
+
const raw = (path: string, text: string, params: string) => {
|
|
215
|
+
writeSync(to, `${path}\t${text}\t${params}\n`);
|
|
216
|
+
const chunk = Buffer.alloc(65536);
|
|
217
|
+
while (!buffered.includes("\n")) {
|
|
218
|
+
const n = readSync(from, chunk, 0, chunk.length, null);
|
|
219
|
+
if (n <= 0) throw new Error("driver exited");
|
|
220
|
+
buffered += chunk.toString("utf8", 0, n);
|
|
221
|
+
}
|
|
222
|
+
const line = buffered.slice(0, buffered.indexOf("\n"));
|
|
223
|
+
buffered = buffered.slice(line.length + 1);
|
|
224
|
+
return line;
|
|
225
|
+
};
|
|
226
|
+
const run = (path: string, text: string, params: unknown[]) => {
|
|
227
|
+
const r = JSON.parse(raw(path, text, JSON.stringify(params)));
|
|
228
|
+
if (r.ok === false) throw new Error("sqlite: " + r.error);
|
|
229
|
+
return r;
|
|
230
|
+
};
|
|
231
|
+
// oxlint-disable-next-line no-function-constructor -- lifting a block out of the text-only transport
|
|
232
|
+
const lifted = new Function(
|
|
233
|
+
"__sbSqlRaw",
|
|
234
|
+
"__sbSql",
|
|
235
|
+
"__sbSqlScriptRaw",
|
|
236
|
+
`${transport.slice(opsStart, opsEnd)}\nreturn { __sbD1Run, __sbD1Batch, __sbD1Open, __sbD1ExecLines };`,
|
|
237
|
+
)(raw, run, (path: string, text: string) => raw(":script " + path, text, "[]"));
|
|
238
|
+
const lift = (dir: string, limit: number) =>
|
|
239
|
+
// oxlint-disable-next-line no-function-constructor -- lifting a block out of the text-only transport
|
|
240
|
+
new Function(
|
|
241
|
+
"__sbEnv",
|
|
242
|
+
"__sbDir",
|
|
243
|
+
"__sbDirBytesRaw",
|
|
244
|
+
"__sbD1DbRaw",
|
|
245
|
+
"__sbSql",
|
|
246
|
+
"__sbSqlRaw",
|
|
247
|
+
`${transport.slice(quotaStart, quotaEnd)}\nreturn { __sbQuotaCheck, __sbD1Quota, __sbQuotaFull };`,
|
|
248
|
+
)(
|
|
249
|
+
(name: string) => (name === "SB_STORAGE_MAX_BYTES" ? String(limit) : ""),
|
|
250
|
+
() => dir,
|
|
251
|
+
(d: string) => String(JSON.parse(raw(":bytes " + d, "", "")).n),
|
|
252
|
+
(path: string, what: string) => String(JSON.parse(raw(":db " + path, what, "")).n),
|
|
253
|
+
run,
|
|
254
|
+
raw,
|
|
255
|
+
);
|
|
256
|
+
return {
|
|
257
|
+
sqlRaw: raw,
|
|
258
|
+
ops: lifted,
|
|
259
|
+
liftQuota: lift,
|
|
260
|
+
exited: child.exited,
|
|
261
|
+
close: () => (closeSync(to), closeSync(from)),
|
|
262
|
+
};
|
|
263
|
+
};
|
|
197
264
|
// exec() from 2026-10-24: line by line, all or nothing, the failing line named.
|
|
198
265
|
const ex = join(work, "data/d1/EXEC.sqlite");
|
|
199
266
|
ops.__sbD1ExecLines(ex, ["CREATE TABLE x (id INTEGER)", "INSERT INTO x VALUES (1); INSERT INTO x VALUES (2);"]);
|
|
@@ -282,8 +349,14 @@ int main(void) {
|
|
|
282
349
|
// With capture on, an authorization failure says so (it read "not an error" once capture
|
|
283
350
|
// had run), and the quota works: its page size comes from capture, not from the
|
|
284
351
|
// page_size pragma the capture authorizer refuses.
|
|
285
|
-
assert.throws(
|
|
286
|
-
|
|
352
|
+
assert.throws(
|
|
353
|
+
() => ops.__sbD1Run(ltx, "PRAGMA page_size", [], true),
|
|
354
|
+
(e: Error) => e.message === "D1_ERROR: not authorized: SQLITE_AUTH",
|
|
355
|
+
);
|
|
356
|
+
assert.throws(
|
|
357
|
+
() => ops.__sbD1ExecLines(ltx, ["PRAGMA journal_mode = DELETE"]),
|
|
358
|
+
(e: Error) => e.message === "D1_EXEC_ERROR: Error in line 1: PRAGMA journal_mode = DELETE: not authorized",
|
|
359
|
+
);
|
|
287
360
|
const data = join(work, "data");
|
|
288
361
|
const roomy = liftQuota(data, dirBytes(data) + (1 << 20));
|
|
289
362
|
roomy.__sbD1Quota(ltx);
|
|
@@ -295,12 +368,39 @@ int main(void) {
|
|
|
295
368
|
const tight = liftQuota(data, 1);
|
|
296
369
|
tight.__sbD1Quota(ltx);
|
|
297
370
|
const spool = ltxFiles();
|
|
298
|
-
const QUOTA =
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
371
|
+
const QUOTA =
|
|
372
|
+
"D1_ERROR: storage quota exceeded: writes are refused while this app's data, including its replication log, is over its quota: SQLITE_FULL";
|
|
373
|
+
for (const write of [
|
|
374
|
+
"INSERT INTO e VALUES (11, 'x')",
|
|
375
|
+
"UPDATE e SET v = 'y' WHERE id = 10",
|
|
376
|
+
"DELETE FROM e WHERE id = 10",
|
|
377
|
+
])
|
|
378
|
+
assert.throws(
|
|
379
|
+
() => ops.__sbD1Run(ltx, write, [], true),
|
|
380
|
+
(e: Error) => e.message === QUOTA,
|
|
381
|
+
write,
|
|
382
|
+
);
|
|
383
|
+
assert.throws(
|
|
384
|
+
() => ops.__sbD1ExecLines(ltx, ["UPDATE e SET v = 'z'"]),
|
|
385
|
+
(e: Error) => e.message.startsWith("D1_EXEC_ERROR: Error in line 1: UPDATE e SET v = 'z': storage quota exceeded"),
|
|
386
|
+
);
|
|
387
|
+
assert.throws(
|
|
388
|
+
() =>
|
|
389
|
+
ops.__sbD1Batch(
|
|
390
|
+
ltx,
|
|
391
|
+
[
|
|
392
|
+
{ sql: "SELECT 1", params: [] },
|
|
393
|
+
{ sql: "DELETE FROM e", params: [] },
|
|
394
|
+
],
|
|
395
|
+
true,
|
|
396
|
+
),
|
|
397
|
+
(e: Error) => e.message === QUOTA,
|
|
398
|
+
);
|
|
399
|
+
assert.equal(
|
|
400
|
+
ops.__sbD1Batch(ltx, [{ sql: "SELECT count(*) FROM e", params: [] }], true).length,
|
|
401
|
+
1,
|
|
402
|
+
"a read-only batch runs",
|
|
403
|
+
);
|
|
304
404
|
assert.equal(ops.__sbD1Run(ltx, "SELECT v FROM e WHERE id = 10", [], true).rows[0][0], "under the quota");
|
|
305
405
|
assert.equal(ltxFiles(), spool, "no LTX file while over the quota");
|
|
306
406
|
const again = liftQuota(data, dirBytes(data) + (1 << 20));
|
|
@@ -314,10 +414,14 @@ int main(void) {
|
|
|
314
414
|
"__sbSql",
|
|
315
415
|
"__sbSqlScriptRaw",
|
|
316
416
|
`${transport.slice(opsStart, opsEnd)}\nreturn { __sbD1Run, __sbD1Batch };`,
|
|
317
|
-
)(
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
417
|
+
)(
|
|
418
|
+
(path: string, text: string, params: string) => {
|
|
419
|
+
if (text === "RELEASE sb_d1_batch") sqlRaw(":break " + path, "", "");
|
|
420
|
+
return sqlRaw(path, text, params);
|
|
421
|
+
},
|
|
422
|
+
sql,
|
|
423
|
+
scriptRaw,
|
|
424
|
+
);
|
|
321
425
|
const LTX_FAILED =
|
|
322
426
|
"D1_ERROR: LTX capture failed: the transaction may be committed locally but is not durable in the replication log; writes are refused until restart: SQLITE_IOERR";
|
|
323
427
|
assert.throws(
|
|
@@ -336,7 +440,9 @@ int main(void) {
|
|
|
336
440
|
);
|
|
337
441
|
assert.throws(
|
|
338
442
|
() => ops.__sbD1ExecLines(ltx, ["INSERT INTO e VALUES (5, 'e')"]),
|
|
339
|
-
(e: Error) =>
|
|
443
|
+
(e: Error) =>
|
|
444
|
+
e.message ===
|
|
445
|
+
"D1_ERROR: LTX capture failed: the transaction may be committed locally but is not durable in the replication log; writes are refused until restart: SQLITE_IOERR",
|
|
340
446
|
"exec() after a capture failure",
|
|
341
447
|
);
|
|
342
448
|
const qdir = join(work, "quota");
|
|
@@ -349,25 +455,86 @@ int main(void) {
|
|
|
349
455
|
// SQLITE_FULL; then the directory is over, a KV write is refused, and reads and deletes
|
|
350
456
|
// keep working. VACUUM gives the space back and writes resume.
|
|
351
457
|
let rows = 0;
|
|
352
|
-
assert.throws(
|
|
353
|
-
|
|
354
|
-
|
|
458
|
+
assert.throws(
|
|
459
|
+
() => {
|
|
460
|
+
for (; rows < 64; rows++) ops.__sbD1Run(qdb, "INSERT INTO q VALUES (zeroblob(65536))", [], true);
|
|
461
|
+
},
|
|
462
|
+
(e: Error) => e.message === "D1_ERROR: database or disk is full: SQLITE_FULL",
|
|
463
|
+
);
|
|
355
464
|
assert(rows >= 2 && rows < 5, "filled to about 256 KiB: " + rows);
|
|
356
465
|
quota.expire();
|
|
357
466
|
assert.equal(quota.__sbQuotaFull(), true);
|
|
358
|
-
assert.throws(
|
|
467
|
+
assert.throws(
|
|
468
|
+
() => quota.__sbQuotaCheck("KV put()"),
|
|
469
|
+
/^Error: KV put\(\) failed: storage quota exceeded \(this app stores \d+ bytes; the limit is 262144\)$/,
|
|
470
|
+
);
|
|
359
471
|
quota.__sbD1Quota(qdb);
|
|
360
472
|
assert.equal(ops.__sbD1Run(qdb, "SELECT count(*) FROM q", [], true).rows[0][0], rows, "reads still work");
|
|
361
473
|
ops.__sbD1Run(qdb, "DELETE FROM q", [], true);
|
|
362
474
|
ops.__sbD1Run(qdb, "VACUUM", [], true);
|
|
363
|
-
|
|
475
|
+
// No checkpoint by the app: over the quota the runtime checkpoints and truncates the WAL
|
|
476
|
+
// (once a second) before measuring again, so DELETE and VACUUM alone make room.
|
|
477
|
+
await Bun.sleep(1000);
|
|
364
478
|
quota.expire();
|
|
365
479
|
quota.__sbD1Quota(qdb);
|
|
366
480
|
quota.__sbQuotaCheck("KV put()");
|
|
367
481
|
ops.__sbD1Run(qdb, "INSERT INTO q VALUES (zeroblob(65536))", [], true);
|
|
482
|
+
driverClosed = true;
|
|
368
483
|
closeSync(toDriver);
|
|
369
484
|
closeSync(fromDriver);
|
|
370
485
|
await driver.exited;
|
|
486
|
+
|
|
487
|
+
// A restart that finds the app over its quota, capture on: the first D1 call of the new
|
|
488
|
+
// worker (process) already freezes. Reads work, writes are refused, the spool and its
|
|
489
|
+
// position stay exactly as they were, and the chain restores the database byte for byte.
|
|
490
|
+
// A capture failure still reads as one, over the quota too.
|
|
491
|
+
await rm(ltx + ".ltx/position.tmp", { recursive: true, force: true });
|
|
492
|
+
const second = connect("restart");
|
|
493
|
+
const quota2 = second.liftQuota(data, 1);
|
|
494
|
+
quota2.__sbD1Quota(ltx);
|
|
495
|
+
const spoolBefore = readdirSync(generation()).sort();
|
|
496
|
+
const positionBefore = readFileSync(ltx + ".ltx/position", "utf8");
|
|
497
|
+
const count = second.ops.__sbD1Run(ltx, "SELECT count(*) FROM e", [], true).rows[0][0];
|
|
498
|
+
for (const write of ["INSERT INTO e VALUES (20, 'x')", "UPDATE e SET v = v", "DELETE FROM e"])
|
|
499
|
+
assert.throws(
|
|
500
|
+
() => second.ops.__sbD1Run(ltx, write, [], true),
|
|
501
|
+
(e: Error) => e.message === QUOTA,
|
|
502
|
+
"restart, over the quota: " + write,
|
|
503
|
+
);
|
|
504
|
+
assert.deepEqual(readdirSync(generation()).sort(), spoolBefore, "no LTX file after a clean resume over the quota");
|
|
505
|
+
assert.equal(readFileSync(ltx + ".ltx/position", "utf8"), positionBefore, "the position is unchanged");
|
|
506
|
+
second.close();
|
|
507
|
+
await second.exited;
|
|
508
|
+
{
|
|
509
|
+
const live = new Database(ltx);
|
|
510
|
+
live.exec("PRAGMA wal_checkpoint(TRUNCATE)");
|
|
511
|
+
assert.equal(live.query("SELECT count(*) AS n FROM e").get()?.n, count);
|
|
512
|
+
live.close();
|
|
513
|
+
const restored = join(work, "restart-restored.sqlite");
|
|
514
|
+
await restore(generation(), restored);
|
|
515
|
+
assert.deepEqual(
|
|
516
|
+
await readFile(restored),
|
|
517
|
+
await readFile(ltx),
|
|
518
|
+
"the retained chain restores the database byte for byte",
|
|
519
|
+
);
|
|
520
|
+
}
|
|
521
|
+
const third = connect("failure");
|
|
522
|
+
const thirdBytes = Number(JSON.parse(third.sqlRaw(":bytes " + data, "", "")).n);
|
|
523
|
+
third.liftQuota(data, thirdBytes + (1 << 20)).__sbD1Quota(ltx);
|
|
524
|
+
const breaking3 = (path: string, text: string, params: string) => {
|
|
525
|
+
if (text.startsWith("INSERT INTO e VALUES (30")) third.sqlRaw(":break " + path, "", "");
|
|
526
|
+
return third.sqlRaw(path, text, params);
|
|
527
|
+
};
|
|
528
|
+
const brokenWrite = JSON.parse(breaking3(ltx, "INSERT INTO e VALUES (30, 'x')", "[]"));
|
|
529
|
+
assert.equal(brokenWrite.code, 10, "a capture failure reports SQLITE_IOERR");
|
|
530
|
+
const failed3 = JSON.parse(third.sqlRaw(ltx, "INSERT INTO e VALUES (31, 'y')", "[]"));
|
|
531
|
+
assert.equal(failed3.code, 10, "after a capture failure, writes report it");
|
|
532
|
+
third.liftQuota(data, 1).__sbD1Quota(ltx);
|
|
533
|
+
const failedOver = JSON.parse(third.sqlRaw(ltx, "INSERT INTO e VALUES (32, 'z')", "[]"));
|
|
534
|
+
assert.equal(failedOver.code, 10, "over the quota too, the capture failure is what a write reports");
|
|
535
|
+
assert.match(failedOver.error, /^LTX capture failed/);
|
|
536
|
+
third.close();
|
|
537
|
+
await third.exited;
|
|
371
538
|
console.log(
|
|
372
539
|
"native d1: sb_sql_run and the D1 ops pass (blob binds, atomic batch, strict counts, foreign keys, error text)",
|
|
373
540
|
);
|
package/test/native-ltx.ts
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import assert from "node:assert/strict";
|
|
2
|
-
import { mkdtemp, readFile, readdir, rm, writeFile, stat } from "node:fs/promises";
|
|
2
|
+
import { mkdir, mkdtemp, readFile, readdir, rm, symlink, writeFile, stat } from "node:fs/promises";
|
|
3
3
|
import { tmpdir } from "node:os";
|
|
4
4
|
import { join } from "node:path";
|
|
5
5
|
import { Database } from "bun:sqlite";
|
|
@@ -24,12 +24,18 @@ const source = `
|
|
|
24
24
|
#include <sys/stat.h>
|
|
25
25
|
#include <time.h>
|
|
26
26
|
#include <signal.h>
|
|
27
|
+
#include <errno.h>
|
|
28
|
+
// Reproduce denial of global temporary-file access on every test platform.
|
|
29
|
+
// The sproutd sandbox qualification separately exercises real Linux Landlock.
|
|
30
|
+
static FILE *forbidden_tmpfile(void) { errno=EACCES; return NULL; }
|
|
31
|
+
#define tmpfile forbidden_tmpfile
|
|
27
32
|
#define SB_MAX_DB 16
|
|
28
33
|
${transport.slice(start, end)}
|
|
29
34
|
int main(int argc,char **argv) {
|
|
30
35
|
if(argc!=2)return 2;sb_ltx_compat=1;if(sb_ltx_register())return 3;
|
|
31
36
|
char lockpath[1024];snprintf(lockpath,sizeof(lockpath),"%s.ltx.lock",argv[1]);
|
|
32
37
|
int lock=open(lockpath,O_RDWR|O_CREAT,0600);if(lock<0||flock(lock,LOCK_EX|LOCK_NB))return 4;
|
|
38
|
+
if(getenv("SB_LTX_TEST_HOLD_LOCK")){puts("locked");fflush(stdout);getchar();close(lock);return 0;}
|
|
33
39
|
sqlite3 *db=0;if(sqlite3_open_v2(argv[1],&db,6,"sb-ltx-bounded"))return 5;
|
|
34
40
|
if(getenv("SB_LTX_TEST_AUTO_VACUUM"))sqlite3_exec(db,"PRAGMA auto_vacuum=INCREMENTAL;",0,0,0);
|
|
35
41
|
if(sqlite3_exec(db,"PRAGMA journal_mode=WAL;PRAGMA cache_size=-2000;",0,0,0)||sb_ltx_start(0,db,argv[1],lock))return 6;
|
|
@@ -153,6 +159,13 @@ async function restore(epoch: string, output: string) {
|
|
|
153
159
|
await writeFile(output, db);
|
|
154
160
|
return { files: files.length, txid };
|
|
155
161
|
}
|
|
162
|
+
async function noIndexScratch(path: string) {
|
|
163
|
+
const entries = await readdir(path + ".ltx", { withFileTypes: true });
|
|
164
|
+
assert(
|
|
165
|
+
!entries.some((entry) => entry.isFile() && /^index-[a-zA-Z0-9]{6}$/.test(entry.name)),
|
|
166
|
+
"index scratch file leaked",
|
|
167
|
+
);
|
|
168
|
+
}
|
|
156
169
|
async function run(path: string, sql: string, extra: Record<string, string> = {}) {
|
|
157
170
|
const child = Bun.spawn([binary, path], {
|
|
158
171
|
env: { ...process.env, ...extra },
|
|
@@ -169,6 +182,7 @@ async function run(path: string, sql: string, extra: Record<string, string> = {}
|
|
|
169
182
|
]);
|
|
170
183
|
assert.equal(code, 0, error);
|
|
171
184
|
assert(out.startsWith("ready\n"), out);
|
|
185
|
+
await noIndexScratch(path);
|
|
172
186
|
return out.split("\n").slice(1).filter(Boolean);
|
|
173
187
|
}
|
|
174
188
|
async function stoppedCommit(path: string, phase: string, count = 1) {
|
|
@@ -212,6 +226,10 @@ async function stoppedCommit(path: string, phase: string, count = 1) {
|
|
|
212
226
|
}
|
|
213
227
|
child.kill("SIGKILL");
|
|
214
228
|
await child.exited;
|
|
229
|
+
if (phase === "index-created") {
|
|
230
|
+
const entries = await readdir(path + ".ltx", { withFileTypes: true });
|
|
231
|
+
assert.equal(entries.filter((entry) => entry.isFile() && /^index-[a-zA-Z0-9]{6}$/.test(entry.name)).length, 1);
|
|
232
|
+
} else await noIndexScratch(path);
|
|
215
233
|
assert(stopped, "did not reach crash boundary " + phase);
|
|
216
234
|
return epoch;
|
|
217
235
|
}
|
|
@@ -241,6 +259,45 @@ try {
|
|
|
241
259
|
const cleanPosition = await readFile(root + "/position", "utf8");
|
|
242
260
|
assert(cleanPosition.split("\n")[2].startsWith("C "));
|
|
243
261
|
const finishedBefore = await readdir(join(root, epoch));
|
|
262
|
+
// SIGKILL can land between mkstemp and unlink. Only the next lock owner may
|
|
263
|
+
// discard that reserved scratch name; unrelated files and links stay intact.
|
|
264
|
+
const strayIndex = join(root, "index-abc123");
|
|
265
|
+
await writeFile(strayIndex, "stale index");
|
|
266
|
+
const unrelatedIndex = join(root, "index-not-scratch");
|
|
267
|
+
await writeFile(unrelatedIndex, "keep");
|
|
268
|
+
const indexDirectory = join(root, "index-DIR123");
|
|
269
|
+
await mkdir(indexDirectory);
|
|
270
|
+
await writeFile(join(indexDirectory, "keep"), "keep directory");
|
|
271
|
+
const outside = join(work, "outside-index-target");
|
|
272
|
+
await writeFile(outside, "keep target");
|
|
273
|
+
const indexLink = join(root, "index-LNK123");
|
|
274
|
+
await symlink(outside, indexLink);
|
|
275
|
+
const owner = Bun.spawn([binary, path], {
|
|
276
|
+
stdin: "pipe",
|
|
277
|
+
stdout: "pipe",
|
|
278
|
+
stderr: "pipe",
|
|
279
|
+
env: { ...process.env, SB_LTX_TEST_HOLD_LOCK: "1" },
|
|
280
|
+
});
|
|
281
|
+
const lockReader = owner.stdout.getReader();
|
|
282
|
+
try {
|
|
283
|
+
const ready = await lockReader.read();
|
|
284
|
+
assert.equal(new TextDecoder().decode(ready.value), "locked\n");
|
|
285
|
+
const blocked = Bun.spawnSync([binary, path], { stdin: "ignore" });
|
|
286
|
+
assert.equal(blocked.exitCode, 4, "non-owner cannot initialize capture");
|
|
287
|
+
assert.equal(await readFile(strayIndex, "utf8"), "stale index", "non-owner must not sweep indexes");
|
|
288
|
+
} finally {
|
|
289
|
+
owner.stdin.end();
|
|
290
|
+
await owner.exited;
|
|
291
|
+
lockReader.releaseLock();
|
|
292
|
+
}
|
|
293
|
+
await run(path, "SELECT 1;\n");
|
|
294
|
+
assert(!(await readdir(root)).includes("index-abc123"), "resume must reclaim stale index");
|
|
295
|
+
assert.equal(await readFile(unrelatedIndex, "utf8"), "keep");
|
|
296
|
+
assert.equal(await readFile(join(indexDirectory, "keep"), "utf8"), "keep directory");
|
|
297
|
+
assert.equal(await readFile(indexLink, "utf8"), "keep target");
|
|
298
|
+
assert.equal(await readFile(outside, "utf8"), "keep target");
|
|
299
|
+
assert.equal(await readFile(root + "/position", "utf8"), cleanPosition);
|
|
300
|
+
assert.deepEqual(await readdir(join(root, epoch)), finishedBefore);
|
|
244
301
|
for (let n = 0; n < 5; n++) {
|
|
245
302
|
await run(path, "SELECT 1;\n");
|
|
246
303
|
assert.equal(await readFile(root + "/position", "utf8"), cleanPosition, "idle restart must not rewrite position");
|
|
@@ -361,6 +418,7 @@ try {
|
|
|
361
418
|
failedDb.close();
|
|
362
419
|
const phases = [
|
|
363
420
|
"wal-committed",
|
|
421
|
+
"index-created",
|
|
364
422
|
"ltx-written",
|
|
365
423
|
"ltx-fsynced",
|
|
366
424
|
"ltx-published",
|
|
@@ -408,7 +466,7 @@ try {
|
|
|
408
466
|
assert.equal(pdb.query("PRAGMA integrity_check").get()?.integrity_check, "ok");
|
|
409
467
|
pdb.close();
|
|
410
468
|
console.log(
|
|
411
|
-
"PASS restart:
|
|
469
|
+
"PASS restart: 11 SIGKILL durability boundaries, matching-prefix resume, orphan/missing/reset fallback, pending committed WAL tail captured before serving",
|
|
412
470
|
);
|
|
413
471
|
console.log(
|
|
414
472
|
"PASS native LTX: 102-file chain, independent CRC/index checks, byte-exact restore, integrity, hard WAL ceiling and oversized transaction rollback",
|