@sproutboat/runtime 0.6.5 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +50 -0
- package/package.json +6 -1
- package/src/entry-crash.test.js +63 -0
- package/src/native-fetch-prelude.js +64 -15
- package/src/transport-broker.js +15 -1
- package/src/transport-embedded.js +147 -60
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,55 @@
|
|
|
1
1
|
# @sproutboat/runtime
|
|
2
2
|
|
|
3
|
+
## 0.7.0
|
|
4
|
+
|
|
5
|
+
### Minor Changes
|
|
6
|
+
|
|
7
|
+
- 6998a15: R2 object bytes no longer live inside SQLite, on either transport (baronunread/sproutboat#56). Every `put()` used to bind the whole object into a `body` column, and every `get()` read it back out — doubling the object through SQLite's own page cache/WAL machinery on top of the buffer it already arrived in, which was the real driver of #56's "large uploads make memory skyrocket" complaint (the JSON-escaping half of that was already fixed in #63).
|
|
8
|
+
|
|
9
|
+
Objects now live in their own file under `r2-blobs/`, named by a hash of bucket+key (so unicode keys and directory traversal are non-issues):
|
|
10
|
+
|
|
11
|
+
- **Embedded (standalone):** `<data-dir>/r2-blobs/` — `sb_r2_put_c`/`sb_r2_get_c` write/read the file directly via C `fopen`/`fwrite`/`fread`, no SQLite blob column at all. `sb_r2_delete_blob` removes the file on delete.
|
|
12
|
+
- **Broker:** colocated with whichever store file already holds that bucket's metadata — the main `db` for a bare-string binding, the resource's own file under `resourceDir` for an account-level one (#74) — so blobs persist across a redeploy exactly when that metadata does. `:memory:` stores (tests, local dev) fall back to an in-memory map.
|
|
13
|
+
|
|
14
|
+
No migration: this is a schema change for new writes going forward, not a converter for objects already stored inline from before.
|
|
15
|
+
|
|
16
|
+
Verified end-to-end against both transports (kitchen-sink's full conformance suite, 24/24 standalone + 27/27 broker) and with new unit tests for file-backed persistence, the `:memory:` fallback, and a real binary round-trip over the v1 wire protocol.
|
|
17
|
+
|
|
18
|
+
**Known limitation found while verifying this, not fixed here:** a binary `put()` body (bytes ≥ 0x80) is corrupted on the way *in*, before it ever reaches storage — `porf_native_fetch_read_value` (the function every inline-C string read in the runtime goes through) always UTF-8-encodes a Latin-1-range string on read, which is correct for genuine text but wrong for opaque bytes coming off a raw HTTP body. This predates this change (the old SQLite-blob path had the identical corruption) and is not something the storage backend can fix — it needs an input-side counterpart to `#176`'s `x-sb-raw-body` marker, or real `ArrayBuffer`/`Uint8Array` body support (the upstream ask already filed as Draft G in `patches/UPSTREAM.md`).
|
|
19
|
+
|
|
20
|
+
### Patch Changes
|
|
21
|
+
|
|
22
|
+
- 188d8e4: Fix (baronunread/sproutboat#184): a `put()` body (R2, both transports) with bytes ≥ 0x80 was corrupted on the way *in*, before storage — `porf_native_fetch_read_value` (the one function every inline-C string read in the runtime funnels through) always UTF-8-encodes a Latin-1-range value, correct for genuine text, wrong for opaque bytes off a raw HTTP upload. A 256-byte binary upload landed as 384 stored bytes.
|
|
23
|
+
|
|
24
|
+
`__sbR2PutRaw` (embedded) and `__sbCallBin` (broker, the only caller that ever passes it a real body) now try `porf_native_fetch_read_raw_bytes` first, falling back to the normal encoding path only for a genuine multi-byte JS string (which isn't a "bytestring" and so isn't eligible for the raw read anyway). This predates the #56 storage change — the old SQLite-blob `put()` called the identical `read_value`, so it was equally corrupted; the existing R2 conformance check never caught it because it only ever exercises a plain ASCII string.
|
|
25
|
+
|
|
26
|
+
Also fixes an adjacent, independently pre-existing bug found while verifying this: `R2Object.text()`/`.json()` returned the raw stored bytes without UTF-8-decoding them, producing mojibake for any object holding real text beyond ASCII — the same gap `__sbRawBodyResponse` already closed for assets/fetch/service-binding responses. Same fix here (lazy, memoized `__sbFromUtf8` decode).
|
|
27
|
+
|
|
28
|
+
Verified end-to-end against both transports with a real binary upload (`curl --data-binary`, not a synthetic string): stored bytes and `.body`/`.text()` now round-trip byte-for-byte/correctly for binary content, Latin-1 text, and multi-byte Unicode (CJK). Full kitchen-sink conformance stays green on both transports (24/24 standalone, 27/27 broker).
|
|
29
|
+
|
|
30
|
+
**Known remaining gap, not fixed here:** a genuine astral-plane character (an emoji, anything outside the Basic Multilingual Plane, encoded as a UTF-16 surrogate pair) still encodes to the wrong number of UTF-8 bytes on write — confirmed identical before and after this fix, so it's a separate, deeper bug in Porffor's own UTF-16→UTF-8 encoder (the `porf_native_fetch_read_value` "string" branch), not something in sproutboat's own code. Filing separately if it recurs; out of scope for this fix.
|
|
31
|
+
|
|
32
|
+
## 0.6.6
|
|
33
|
+
|
|
34
|
+
### Patch Changes
|
|
35
|
+
|
|
36
|
+
- c2b0359: Fix (baronunread/sproutboat#179): `__sbEntry` called `handlers.fetch()` (and
|
|
37
|
+
the `scheduled`/`queue`/`alarm` trigger paths) with no try/catch at all. Any
|
|
38
|
+
synchronous throw, or an async handler's rejected promise, propagated all the
|
|
39
|
+
way up and took down the entire process — not just the request that hit it,
|
|
40
|
+
but every other in-flight and future request on that sprout, until whatever
|
|
41
|
+
supervises it restarts the binary.
|
|
42
|
+
|
|
43
|
+
A throw from `fetch()` now returns a 500 instead. `scheduled()` and `alarm()`
|
|
44
|
+
still reply 204 either way (they had no failure signal before this either).
|
|
45
|
+
A throw from `queue()` falls through to the existing default-ack pass, the
|
|
46
|
+
same outcome as a handler that returns without calling `ack()`/`retry()` on
|
|
47
|
+
every message.
|
|
48
|
+
|
|
49
|
+
`entry-crash.test.js` covers the fetch path: a sync throw, an async
|
|
50
|
+
rejection, and that a throwing request doesn't stop the next one from being
|
|
51
|
+
served.
|
|
52
|
+
|
|
3
53
|
## 0.6.5
|
|
4
54
|
|
|
5
55
|
### Patch Changes
|
package/package.json
CHANGED
|
@@ -1,8 +1,13 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@sproutboat/runtime",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.7.0",
|
|
4
4
|
"license": "MIT",
|
|
5
5
|
"type": "module",
|
|
6
|
+
"repository": {
|
|
7
|
+
"type": "git",
|
|
8
|
+
"url": "git+https://github.com/baronunread/sproutboat-packages.git",
|
|
9
|
+
"directory": "packages/runtime"
|
|
10
|
+
},
|
|
6
11
|
"main": "./src/index.ts",
|
|
7
12
|
"types": "./src/index.ts",
|
|
8
13
|
"scripts": {
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
// #179 — an uncaught handler exception must turn into a 500 (or, for
|
|
2
|
+
// scheduled/queue/alarm, a swallowed failure), never a process-crashing
|
|
3
|
+
// throw/rejection out of __sbEntry. Same lift-the-prelude-and-eval approach
|
|
4
|
+
// as client-ip.test.js: the prelude is prepended as text and cannot import.
|
|
5
|
+
import { expect, test } from "bun:test";
|
|
6
|
+
import { readFileSync } from "node:fs";
|
|
7
|
+
import { fileURLToPath } from "node:url";
|
|
8
|
+
|
|
9
|
+
const src = readFileSync(fileURLToPath(new URL("./native-fetch-prelude.js", import.meta.url)), "utf8");
|
|
10
|
+
|
|
11
|
+
// oxlint-disable-next-line no-function-constructor -- lifting the whole text-only prelude for test
|
|
12
|
+
const load = new Function(
|
|
13
|
+
"Porffor",
|
|
14
|
+
"__envMap",
|
|
15
|
+
`${src}
|
|
16
|
+
return { __sbEntry, __sbRegisterDO };`,
|
|
17
|
+
);
|
|
18
|
+
// Every Porffor.c`...` block in the prelude only runs inside a function body
|
|
19
|
+
// (never at load time), so a no-op tag covers all of them: __sbCpuMs and
|
|
20
|
+
// __sbEnv just see an empty result, same as "native op returned nothing."
|
|
21
|
+
const H = load(
|
|
22
|
+
{ c: () => "" },
|
|
23
|
+
new Proxy({}, { get: () => undefined }),
|
|
24
|
+
);
|
|
25
|
+
|
|
26
|
+
const req = (overrides = {}) => ({
|
|
27
|
+
headers: { get: (k) => (overrides.headers && overrides.headers[k.toLowerCase()]) ?? null },
|
|
28
|
+
cf: undefined,
|
|
29
|
+
...overrides,
|
|
30
|
+
});
|
|
31
|
+
|
|
32
|
+
test("a synchronously-throwing fetch handler returns 500, not a throw", async () => {
|
|
33
|
+
const handlers = {
|
|
34
|
+
fetch() {
|
|
35
|
+
throw new Error("boom");
|
|
36
|
+
},
|
|
37
|
+
};
|
|
38
|
+
const res = await H.__sbEntry(handlers, req());
|
|
39
|
+
expect(res.status).toBe(500);
|
|
40
|
+
});
|
|
41
|
+
|
|
42
|
+
test("an async fetch handler whose promise rejects returns 500, not a rejection", async () => {
|
|
43
|
+
const handlers = {
|
|
44
|
+
async fetch() {
|
|
45
|
+
throw new Error("boom");
|
|
46
|
+
},
|
|
47
|
+
};
|
|
48
|
+
const res = await H.__sbEntry(handlers, req());
|
|
49
|
+
expect(res.status).toBe(500);
|
|
50
|
+
});
|
|
51
|
+
|
|
52
|
+
test("a throwing fetch handler doesn't stop the next request from being served", async () => {
|
|
53
|
+
const handlers = {
|
|
54
|
+
fetch(request) {
|
|
55
|
+
if (request.boom) throw new Error("boom");
|
|
56
|
+
return new Response("ok");
|
|
57
|
+
},
|
|
58
|
+
};
|
|
59
|
+
const first = await H.__sbEntry(handlers, req({ boom: true }));
|
|
60
|
+
expect(first.status).toBe(500);
|
|
61
|
+
const second = await H.__sbEntry(handlers, req({ boom: false }));
|
|
62
|
+
expect(await second.text()).toBe("ok");
|
|
63
|
+
});
|
|
@@ -1104,11 +1104,18 @@ function __sbR2Object(meta, body) {
|
|
|
1104
1104
|
};
|
|
1105
1105
|
if (body != null) {
|
|
1106
1106
|
obj.body = body;
|
|
1107
|
+
// #184 — `body` is raw bytes (a bytestring), not decoded text: `.text()`
|
|
1108
|
+
// returning it verbatim produced mojibake for any object holding real
|
|
1109
|
+
// UTF-8 content, the same gap __sbRawBodyResponse already closed for
|
|
1110
|
+
// assets/fetch/service-binding responses. Same fix here: decode lazily,
|
|
1111
|
+
// once, on first read.
|
|
1112
|
+
let decoded = null;
|
|
1107
1113
|
obj.text = function () {
|
|
1108
|
-
|
|
1114
|
+
if (decoded === null) decoded = __sbFromUtf8(body);
|
|
1115
|
+
return decoded;
|
|
1109
1116
|
};
|
|
1110
1117
|
obj.json = function () {
|
|
1111
|
-
return JSON.parse(
|
|
1118
|
+
return JSON.parse(obj.text());
|
|
1112
1119
|
};
|
|
1113
1120
|
}
|
|
1114
1121
|
return obj;
|
|
@@ -1703,15 +1710,31 @@ async function __sbDrainAll(pending) {
|
|
|
1703
1710
|
// one *this* async function creates, not a `.then()` derived from someone
|
|
1704
1711
|
// else's — see the note below on why that distinction matters here.
|
|
1705
1712
|
async function __sbFetchWithDrain(res, tasks) {
|
|
1706
|
-
|
|
1713
|
+
let r;
|
|
1714
|
+
try {
|
|
1715
|
+
r = res && __sbIsFn(res.then) ? await res : res;
|
|
1716
|
+
} catch {
|
|
1717
|
+
// #179 — a rejected handler promise must not take the process down with it.
|
|
1718
|
+
r = __sbErrorResponse();
|
|
1719
|
+
}
|
|
1707
1720
|
await __sbDrainWaitUntil(tasks);
|
|
1708
1721
|
return r;
|
|
1709
1722
|
}
|
|
1710
1723
|
|
|
1724
|
+
// #179 — turn an uncaught handler exception into a 500 instead of letting it
|
|
1725
|
+
// propagate past __sbEntry and crash the whole server.
|
|
1726
|
+
function __sbErrorResponse() {
|
|
1727
|
+
return new Response("Internal Server Error", { status: 500 });
|
|
1728
|
+
}
|
|
1729
|
+
|
|
1711
1730
|
// Shared by alarm and scheduled: both reply with an empty 204 once their
|
|
1712
1731
|
// handler (and anything it queued via `waitUntil`) has settled.
|
|
1713
1732
|
async function __sb204WithDrain(res, tasks) {
|
|
1714
|
-
|
|
1733
|
+
try {
|
|
1734
|
+
if (res && __sbIsFn(res.then)) await res;
|
|
1735
|
+
} catch {
|
|
1736
|
+
// #179 — same rule as fetch: a rejected scheduled/alarm promise must not crash the process.
|
|
1737
|
+
}
|
|
1715
1738
|
await __sbDrainWaitUntil(tasks);
|
|
1716
1739
|
return new Response("", { status: 204 });
|
|
1717
1740
|
}
|
|
@@ -1746,7 +1769,14 @@ globalThis.__sbEntry = function (handlers, request) {
|
|
|
1746
1769
|
if (p && __sbIsFn(p.then)) __sbTasks.push(p);
|
|
1747
1770
|
},
|
|
1748
1771
|
};
|
|
1749
|
-
|
|
1772
|
+
let __res;
|
|
1773
|
+
try {
|
|
1774
|
+
__res = handlers.fetch(request, ctx);
|
|
1775
|
+
} catch {
|
|
1776
|
+
// #179 — a synchronous throw anywhere in the handler must not take the
|
|
1777
|
+
// whole process (and every other in-flight request) down with it.
|
|
1778
|
+
return __sbErrorResponse();
|
|
1779
|
+
}
|
|
1750
1780
|
// Same promise-identity rule as above applies to `__sbFetchWithDrain`'s
|
|
1751
1781
|
// own return, which is why it's tail-called rather than chained on here.
|
|
1752
1782
|
if (__sbTasks.length || (__res && __sbIsFn(__res.then))) return __sbFetchWithDrain(__res, __sbTasks);
|
|
@@ -1766,10 +1796,16 @@ globalThis.__sbEntry = function (handlers, request) {
|
|
|
1766
1796
|
if (p && __sbIsFn(p.then)) __sbTasks.push(p);
|
|
1767
1797
|
},
|
|
1768
1798
|
};
|
|
1769
|
-
|
|
1770
|
-
|
|
1771
|
-
|
|
1772
|
-
|
|
1799
|
+
let __sres;
|
|
1800
|
+
try {
|
|
1801
|
+
__sres = handlers.scheduled(
|
|
1802
|
+
{ cron: body.cron || "", scheduledTime: body.scheduledTime || Date.now(), noRetry() {} },
|
|
1803
|
+
ctx,
|
|
1804
|
+
);
|
|
1805
|
+
} catch {
|
|
1806
|
+
// #179 — same rule as fetch: don't let a throw here crash the process.
|
|
1807
|
+
return new Response("", { status: 204 });
|
|
1808
|
+
}
|
|
1773
1809
|
if (__sbTasks.length || (__sres && __sbIsFn(__sres.then))) return __sb204WithDrain(__sres, __sbTasks);
|
|
1774
1810
|
return new Response("", { status: 204 });
|
|
1775
1811
|
}
|
|
@@ -1797,7 +1833,13 @@ globalThis.__sbEntry = function (handlers, request) {
|
|
|
1797
1833
|
const body = __sbReadJson(request);
|
|
1798
1834
|
const inst = __sbGetDOInstance(String(body.cls || ""), String(body.id || ""));
|
|
1799
1835
|
if (!__sbIsFn(inst.alarm)) return new Response("no alarm handler", { status: 404 });
|
|
1800
|
-
|
|
1836
|
+
let __ares;
|
|
1837
|
+
try {
|
|
1838
|
+
__ares = inst.alarm();
|
|
1839
|
+
} catch {
|
|
1840
|
+
// #179 — same rule as fetch: don't let a throw here crash the process.
|
|
1841
|
+
return new Response("", { status: 204 });
|
|
1842
|
+
}
|
|
1801
1843
|
// Was fire-and-forget before: an async `alarm()` and any `state.waitUntil()`
|
|
1802
1844
|
// it queued were both dropped the moment 204 went out. Await both.
|
|
1803
1845
|
if (inst.__sbTasks.length || (__ares && __sbIsFn(__ares.then))) return __sb204WithDrain(__ares, inst.__sbTasks);
|
|
@@ -1845,11 +1887,18 @@ async function __sbRunQueueBatch(handlers, body, ctx) {
|
|
|
1845
1887
|
for (let i = 0; i < messages.length; i++) messages[i].retry();
|
|
1846
1888
|
},
|
|
1847
1889
|
};
|
|
1848
|
-
|
|
1849
|
-
|
|
1850
|
-
|
|
1851
|
-
|
|
1852
|
-
|
|
1890
|
+
try {
|
|
1891
|
+
const __qres = handlers.queue(batch, ctx);
|
|
1892
|
+
// Was fire-and-forget: an async `queue()` had this default-ack pass run (and
|
|
1893
|
+
// the response go out) before its own `await`s did, so any ack()/retry()
|
|
1894
|
+
// past the first one never counted.
|
|
1895
|
+
if (__qres && __sbIsFn(__qres.then)) await __qres;
|
|
1896
|
+
} catch {
|
|
1897
|
+
// #179 — a throw here must not crash the process. Whatever ack()/retry()
|
|
1898
|
+
// calls the handler made before throwing still count; anything it never
|
|
1899
|
+
// touched falls through to the default-ack pass below, same as a handler
|
|
1900
|
+
// that returns without calling ack()/retry() on every message.
|
|
1901
|
+
}
|
|
1853
1902
|
// default: any message neither acked nor retried is treated as acked
|
|
1854
1903
|
for (let i = 0; i < messages.length; i++) {
|
|
1855
1904
|
if (acked.indexOf(messages[i].id) === -1 && retried.indexOf(messages[i].id) === -1) acked.push(messages[i].id);
|
package/src/transport-broker.js
CHANGED
|
@@ -19,6 +19,13 @@
|
|
|
19
19
|
|
|
20
20
|
// oxlint-disable-next-line no-unused-expressions -- Porffor.c`...` is inline C the compiler consumes, not a JS expression.
|
|
21
21
|
Porffor.c`
|
|
22
|
+
// baronunread/sproutboat#176 — defined in render.js, alongside
|
|
23
|
+
// porf_native_fetch_read_value (which every inline-C block below already
|
|
24
|
+
// calls with no forward declaration of its own — that one gets one from
|
|
25
|
+
// Porffor's own generated prototypes, this one doesn't, since it's a local
|
|
26
|
+
// patch addition, not something Porffor knows to prototype early).
|
|
27
|
+
extern int porf_native_fetch_read_raw_bytes(jsval value, const char** out_buf, size_t* out_len);
|
|
28
|
+
|
|
22
29
|
static int sb_io_all(int fd, unsigned char* buf, size_t len, int writing) {
|
|
23
30
|
size_t done = 0;
|
|
24
31
|
while (done < len) {
|
|
@@ -237,8 +244,15 @@ function __sbCallBin(reqJson, body) {
|
|
|
237
244
|
Porffor.c`
|
|
238
245
|
const char* __j; size_t __jl; char* __jo = 0;
|
|
239
246
|
porf_native_fetch_read_value(reqJson, &__j, &__jl, &__jo);
|
|
247
|
+
// baronunread/sproutboat#184 -- the only real (non-empty) body this ever
|
|
248
|
+
// carries today is an R2 put(); read raw, not through read_value's
|
|
249
|
+
// bytestring branch, which always UTF-8-encodes (right for text, wrong
|
|
250
|
+
// for opaque bytes off a raw HTTP upload). A genuine multi-byte JS string
|
|
251
|
+
// isn't a bytestring, so it falls back to the encoding path unaffected.
|
|
240
252
|
const char* __b; size_t __bl; char* __bo = 0;
|
|
241
|
-
|
|
253
|
+
if (porf_native_fetch_read_raw_bytes(body, &__b, &__bl) != 0) {
|
|
254
|
+
porf_native_fetch_read_value(body, &__b, &__bl, &__bo);
|
|
255
|
+
}
|
|
242
256
|
char* __resp = 0; size_t __resplen = 0;
|
|
243
257
|
int __rc = sb_broker_roundtrip_v1(__j, __jl, __b, __bl, &__resp, &__resplen);
|
|
244
258
|
if (__jo) free(__jo);
|
|
@@ -53,6 +53,13 @@ extern int64_t sqlite3_last_insert_rowid(sqlite3*);
|
|
|
53
53
|
extern const char* sqlite3_errmsg(sqlite3*);
|
|
54
54
|
extern int sqlite3_close(sqlite3*);
|
|
55
55
|
|
|
56
|
+
// baronunread/sproutboat#176 — defined in render.js, alongside
|
|
57
|
+
// porf_native_fetch_read_value (which every inline-C block below already
|
|
58
|
+
// calls with no forward declaration of its own — that one gets one from
|
|
59
|
+
// Porffor's own generated prototypes, this one doesn't, since it's a local
|
|
60
|
+
// patch addition, not something Porffor knows to prototype early).
|
|
61
|
+
extern int porf_native_fetch_read_raw_bytes(jsval value, const char** out_buf, size_t* out_len);
|
|
62
|
+
|
|
56
63
|
#define SB_SQLITE_ROW 100
|
|
57
64
|
#define SB_SQLITE_DONE 101
|
|
58
65
|
#define SB_SQLITE_TRANSIENT ((void*)-1)
|
|
@@ -240,11 +247,14 @@ static int sb_bind_params(sqlite3_stmt* st, const char* json) {
|
|
|
240
247
|
return 0;
|
|
241
248
|
}
|
|
242
249
|
|
|
243
|
-
// --- R2 bodies, without the JSON detour
|
|
244
|
-
// An object's bytes never enter a JSON frame
|
|
245
|
-
// parameter and
|
|
246
|
-
//
|
|
247
|
-
//
|
|
250
|
+
// --- R2 bodies, without the JSON detour or the SQLite detour ----------------
|
|
251
|
+
// An object's bytes never enter a JSON frame (they arrive as their own string
|
|
252
|
+
// parameter) and never enter SQLite either (baronunread/sproutboat#56): they
|
|
253
|
+
// go straight to their own file under "<data-dir>/r2-blobs/", named by a hash
|
|
254
|
+
// of bucket+key so unicode keys and directory traversal are both non-issues.
|
|
255
|
+
// Doubling every object through SQLite's own page cache on top of the buffer
|
|
256
|
+
// it already arrived in was the real cost behind #56's "large uploads make
|
|
257
|
+
// memory skyrocket" complaint, not just the JSON-escaping #63 already fixed.
|
|
248
258
|
|
|
249
259
|
static void sb_hex32(const unsigned char* in, char* out) {
|
|
250
260
|
static const char* d = "0123456789abcdef";
|
|
@@ -269,7 +279,24 @@ static void sb_iso_now(char* out, size_t cap) {
|
|
|
269
279
|
strftime(out, cap, "%Y-%m-%dT%H:%M:%S.000Z", &g);
|
|
270
280
|
}
|
|
271
281
|
|
|
272
|
-
//
|
|
282
|
+
// "<dir-of-store_path>/r2-blobs/<sha256(bucket 0x1e key)>.blob" — deterministic,
|
|
283
|
+
// so no extra column is needed to find an object's file back.
|
|
284
|
+
static void sb_r2_blob_path(char* out, size_t cap, const char* store_path, const char* bucket, const char* key) {
|
|
285
|
+
const char* slash = strrchr(store_path, '/');
|
|
286
|
+
int dirlen = slash ? (int)(slash - store_path) : 0;
|
|
287
|
+
br_sha256_context ctx;
|
|
288
|
+
br_sha256_init(&ctx);
|
|
289
|
+
br_sha256_update(&ctx, bucket, strlen(bucket));
|
|
290
|
+
br_sha256_update(&ctx, "\x1e", 1); // separator: a bucket/key split never collides across it
|
|
291
|
+
br_sha256_update(&ctx, key, strlen(key));
|
|
292
|
+
unsigned char digest[32];
|
|
293
|
+
br_sha256_out(&ctx, digest);
|
|
294
|
+
char hex[65];
|
|
295
|
+
sb_hex32(digest, hex);
|
|
296
|
+
snprintf(out, cap, "%.*s/r2-blobs/%s.blob", dirlen, store_path, hex);
|
|
297
|
+
}
|
|
298
|
+
|
|
299
|
+
// Returns malloc'd JSON metadata; the body itself is never serialised into it.
|
|
273
300
|
static char* sb_r2_put_c(const char* path, const char* bucket, const char* key, const char* body, size_t bodylen,
|
|
274
301
|
const char* http_json, const char* custom_json) {
|
|
275
302
|
sb_buf out = { 0, 0, 0 };
|
|
@@ -282,10 +309,19 @@ static char* sb_r2_put_c(const char* path, const char* bucket, const char* key,
|
|
|
282
309
|
char uploaded[40];
|
|
283
310
|
sb_iso_now(uploaded, sizeof(uploaded));
|
|
284
311
|
|
|
312
|
+
char blobpath[1024];
|
|
313
|
+
sb_r2_blob_path(blobpath, sizeof(blobpath), path, bucket, key);
|
|
314
|
+
sb_mkdirs(blobpath);
|
|
315
|
+
FILE* f = fopen(blobpath, "wb");
|
|
316
|
+
if (!f) { sb_puts(&out, "{\"ok\":false,\"error\":\"cannot write object\"}"); return out.p; }
|
|
317
|
+
size_t written = bodylen ? fwrite(body, 1, bodylen, f) : 0;
|
|
318
|
+
fclose(f);
|
|
319
|
+
if (written != bodylen) { sb_puts(&out, "{\"ok\":false,\"error\":\"short write\"}"); return out.p; }
|
|
320
|
+
|
|
285
321
|
sqlite3_stmt* st = 0;
|
|
286
322
|
const char* sql =
|
|
287
|
-
"INSERT INTO r2 (bucket, key,
|
|
288
|
-
"ON CONFLICT (bucket, key) DO UPDATE SET
|
|
323
|
+
"INSERT INTO r2 (bucket, key, size, etag, uploaded, http_json, custom_json) VALUES (?1,?2,?3,?4,?5,?6,?7) "
|
|
324
|
+
"ON CONFLICT (bucket, key) DO UPDATE SET size=?3, etag=?4, uploaded=?5, http_json=?6, custom_json=?7";
|
|
289
325
|
if (sqlite3_prepare_v2(db, sql, -1, &st, 0) != 0 || !st) {
|
|
290
326
|
sb_puts(&out, "{\"ok\":false,\"error\":");
|
|
291
327
|
const char* m = sqlite3_errmsg(db);
|
|
@@ -295,12 +331,11 @@ static char* sb_r2_put_c(const char* path, const char* bucket, const char* key,
|
|
|
295
331
|
}
|
|
296
332
|
sqlite3_bind_text(st, 1, bucket, -1, SB_SQLITE_TRANSIENT);
|
|
297
333
|
sqlite3_bind_text(st, 2, key, -1, SB_SQLITE_TRANSIENT);
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
sqlite3_bind_text(st, 5,
|
|
301
|
-
sqlite3_bind_text(st, 6,
|
|
302
|
-
sqlite3_bind_text(st, 7,
|
|
303
|
-
sqlite3_bind_text(st, 8, custom_json && *custom_json ? custom_json : "{}", -1, SB_SQLITE_TRANSIENT);
|
|
334
|
+
sqlite3_bind_int64(st, 3, (int64_t)bodylen);
|
|
335
|
+
sqlite3_bind_text(st, 4, etag, -1, SB_SQLITE_TRANSIENT);
|
|
336
|
+
sqlite3_bind_text(st, 5, uploaded, -1, SB_SQLITE_TRANSIENT);
|
|
337
|
+
sqlite3_bind_text(st, 6, http_json && *http_json ? http_json : "{}", -1, SB_SQLITE_TRANSIENT);
|
|
338
|
+
sqlite3_bind_text(st, 7, custom_json && *custom_json ? custom_json : "{}", -1, SB_SQLITE_TRANSIENT);
|
|
304
339
|
int rc = sqlite3_step(st);
|
|
305
340
|
sqlite3_finalize(st);
|
|
306
341
|
if (rc != SB_SQLITE_DONE) { sb_puts(&out, "{\"ok\":false,\"error\":\"r2 put failed\"}"); return out.p; }
|
|
@@ -311,30 +346,50 @@ static char* sb_r2_put_c(const char* path, const char* bucket, const char* key,
|
|
|
311
346
|
return out.p;
|
|
312
347
|
}
|
|
313
348
|
|
|
314
|
-
//
|
|
315
|
-
//
|
|
316
|
-
//
|
|
317
|
-
//
|
|
318
|
-
// one 8 MB allocation, not two. Returns the bytestring pointer, or 0.
|
|
349
|
+
// Reads the object's file into a Porffor bytestring. One malloc (the file
|
|
350
|
+
// read) plus the bytestring's own allocation — no SQLite page-cache copy on
|
|
351
|
+
// top, and no growing single database file holding every object ever put.
|
|
352
|
+
// Returns the bytestring pointer, or 0.
|
|
319
353
|
static u32 sb_r2_get_c(const char* path, const char* bucket, const char* key, int* found) {
|
|
320
354
|
*found = 0;
|
|
321
355
|
int idx = sb_db_for(path);
|
|
322
356
|
if (idx < 0) return 0;
|
|
323
357
|
sqlite3_stmt* st = 0;
|
|
324
|
-
if (sqlite3_prepare_v2(sb_dbs[idx], "SELECT
|
|
358
|
+
if (sqlite3_prepare_v2(sb_dbs[idx], "SELECT 1 FROM r2 WHERE bucket = ? AND key = ?", -1, &st, 0) != 0 || !st) return 0;
|
|
325
359
|
sqlite3_bind_text(st, 1, bucket, -1, SB_SQLITE_TRANSIENT);
|
|
326
360
|
sqlite3_bind_text(st, 2, key, -1, SB_SQLITE_TRANSIENT);
|
|
327
|
-
|
|
328
|
-
if (sqlite3_step(st) == SB_SQLITE_ROW) {
|
|
329
|
-
int n = sqlite3_column_bytes(st, 0);
|
|
330
|
-
const void* blob = sqlite3_column_blob(st, 0);
|
|
331
|
-
out = porf_native_fetch_alloc_bytestring((const char*)(blob ? blob : ""), (size_t)(n > 0 ? n : 0));
|
|
332
|
-
*found = 1;
|
|
333
|
-
}
|
|
361
|
+
int has_row = sqlite3_step(st) == SB_SQLITE_ROW;
|
|
334
362
|
sqlite3_finalize(st);
|
|
363
|
+
if (!has_row) return 0;
|
|
364
|
+
|
|
365
|
+
char blobpath[1024];
|
|
366
|
+
sb_r2_blob_path(blobpath, sizeof(blobpath), path, bucket, key);
|
|
367
|
+
FILE* f = fopen(blobpath, "rb");
|
|
368
|
+
// A metadata row with no blob file (deleted out from under it, disk issue)
|
|
369
|
+
// reads as not-found rather than crashing.
|
|
370
|
+
if (!f) return 0;
|
|
371
|
+
fseek(f, 0, SEEK_END);
|
|
372
|
+
long size = ftell(f);
|
|
373
|
+
rewind(f);
|
|
374
|
+
if (size < 0) { fclose(f); return 0; }
|
|
375
|
+
char* buf = (char*)malloc((size_t)size > 0 ? (size_t)size : 1);
|
|
376
|
+
size_t rd = size > 0 ? fread(buf, 1, (size_t)size, f) : 0;
|
|
377
|
+
fclose(f);
|
|
378
|
+
if (!buf || rd != (size_t)size) { free(buf); return 0; }
|
|
379
|
+
u32 out = porf_native_fetch_alloc_bytestring(buf, (size_t)size);
|
|
380
|
+
free(buf);
|
|
381
|
+
*found = 1;
|
|
335
382
|
return out;
|
|
336
383
|
}
|
|
337
384
|
|
|
385
|
+
// Best-effort: a metadata row deleted with no blob file (or vice versa) is
|
|
386
|
+
// still a correct end state, so a missing file here is not an error.
|
|
387
|
+
static void sb_r2_delete_blob(const char* path, const char* bucket, const char* key) {
|
|
388
|
+
char blobpath[1024];
|
|
389
|
+
sb_r2_blob_path(blobpath, sizeof(blobpath), path, bucket, key);
|
|
390
|
+
unlink(blobpath);
|
|
391
|
+
}
|
|
392
|
+
|
|
338
393
|
// Run a script: one or more statements separated by semicolons. sqlite3_exec
|
|
339
394
|
// handles the whole string, which prepare/step does not — it compiles the first
|
|
340
395
|
// statement and silently ignores the rest, so a schema built from one exec call
|
|
@@ -929,10 +984,18 @@ function __sbR2PutRaw(path, bucket, key, body, httpJson, customJson) {
|
|
|
929
984
|
char* __custom = (char*)malloc(__cl + 1); memcpy(__custom, __c, __cl); __custom[__cl] = 0;
|
|
930
985
|
if (__co) free(__co);
|
|
931
986
|
|
|
932
|
-
//
|
|
933
|
-
//
|
|
987
|
+
// baronunread/sproutboat#184 -- read raw, not through porf_native_fetch_
|
|
988
|
+
// read_value's bytestring branch, which always UTF-8-encodes (correct for
|
|
989
|
+
// genuine text, wrong for opaque bytes -- request.body off a raw HTTP
|
|
990
|
+
// upload is exactly that). R2 is object storage: put()/get() need to be
|
|
991
|
+
// byte-preserving, not text-preserving. Only a bytestring-typed value can
|
|
992
|
+
// be read raw; a genuine multi-byte JS string (real Unicode, "string" type
|
|
993
|
+
// rather than "bytestring") isn't one, so it falls back to the normal
|
|
994
|
+
// encoding path, which already handles that type correctly.
|
|
934
995
|
const char* __b; size_t __bl; char* __bo = 0;
|
|
935
|
-
|
|
996
|
+
if (porf_native_fetch_read_raw_bytes(body, &__b, &__bl) != 0) {
|
|
997
|
+
porf_native_fetch_read_value(body, &__b, &__bl, &__bo);
|
|
998
|
+
}
|
|
936
999
|
char* __out = sb_r2_put_c(__path, __bucket, __key, __b, __bl, __http, __custom);
|
|
937
1000
|
if (__bo) free(__bo);
|
|
938
1001
|
|
|
@@ -974,6 +1037,32 @@ function __sbR2GetRaw(path, bucket, key) {
|
|
|
974
1037
|
return res;
|
|
975
1038
|
}
|
|
976
1039
|
|
|
1040
|
+
// R2 delete: also removes the object's own file, not just the metadata row
|
|
1041
|
+
// (#56 — sb_r2_delete_blob is best-effort, a missing file is not an error).
|
|
1042
|
+
// oxlint-disable-next-line no-unused-vars -- read inside the RawC block below, not by JS.
|
|
1043
|
+
function __sbR2DeleteRaw(path, bucket, key) {
|
|
1044
|
+
// oxlint-disable-next-line no-unused-expressions -- Porffor.c`...` is inline C the compiler consumes, not a JS expression.
|
|
1045
|
+
Porffor.c`
|
|
1046
|
+
const char* __p; size_t __pl; char* __po = 0;
|
|
1047
|
+
porf_native_fetch_read_value(path, &__p, &__pl, &__po);
|
|
1048
|
+
char* __path = (char*)malloc(__pl + 1); memcpy(__path, __p, __pl); __path[__pl] = 0;
|
|
1049
|
+
if (__po) free(__po);
|
|
1050
|
+
|
|
1051
|
+
const char* __bk; size_t __bkl; char* __bko = 0;
|
|
1052
|
+
porf_native_fetch_read_value(bucket, &__bk, &__bkl, &__bko);
|
|
1053
|
+
char* __bucket = (char*)malloc(__bkl + 1); memcpy(__bucket, __bk, __bkl); __bucket[__bkl] = 0;
|
|
1054
|
+
if (__bko) free(__bko);
|
|
1055
|
+
|
|
1056
|
+
const char* __k; size_t __kl; char* __ko = 0;
|
|
1057
|
+
porf_native_fetch_read_value(key, &__k, &__kl, &__ko);
|
|
1058
|
+
char* __key = (char*)malloc(__kl + 1); memcpy(__key, __k, __kl); __key[__kl] = 0;
|
|
1059
|
+
if (__ko) free(__ko);
|
|
1060
|
+
|
|
1061
|
+
sb_r2_delete_blob(__path, __bucket, __key);
|
|
1062
|
+
free(__path); free(__bucket); free(__key);
|
|
1063
|
+
`;
|
|
1064
|
+
}
|
|
1065
|
+
|
|
977
1066
|
// Multi-statement exec. Same string-param pattern as __sbSqlRaw.
|
|
978
1067
|
// oxlint-disable-next-line no-unused-vars -- read inside the RawC block below, not by JS.
|
|
979
1068
|
function __sbSqlScriptRaw(path, sql) {
|
|
@@ -1158,9 +1247,11 @@ function __sbEnsureSchema() {
|
|
|
1158
1247
|
s,
|
|
1159
1248
|
"CREATE TABLE IF NOT EXISTS kv (ns TEXT NOT NULL, key TEXT NOT NULL, value TEXT NOT NULL, PRIMARY KEY (ns, key))",
|
|
1160
1249
|
);
|
|
1250
|
+
// #56 — no `body` column: an object's bytes live in their own file under
|
|
1251
|
+
// "<data-dir>/r2-blobs/" (sb_r2_put_c/sb_r2_get_c), never in the sqlite row.
|
|
1161
1252
|
__sbSql(
|
|
1162
1253
|
s,
|
|
1163
|
-
"CREATE TABLE IF NOT EXISTS r2 (bucket TEXT NOT NULL, key TEXT NOT NULL,
|
|
1254
|
+
"CREATE TABLE IF NOT EXISTS r2 (bucket TEXT NOT NULL, key TEXT NOT NULL, size INTEGER NOT NULL, " +
|
|
1164
1255
|
"etag TEXT NOT NULL, uploaded TEXT NOT NULL, http_json TEXT NOT NULL DEFAULT '{}', custom_json TEXT NOT NULL DEFAULT '{}', " +
|
|
1165
1256
|
"PRIMARY KEY (bucket, key))",
|
|
1166
1257
|
);
|
|
@@ -1350,38 +1441,33 @@ function __sbEmbeddedDispatch(msg) {
|
|
|
1350
1441
|
return { ok: true, path: dest, bytes: reply.bytes };
|
|
1351
1442
|
}
|
|
1352
1443
|
|
|
1444
|
+
// r2.put/get route through the same raw C path (__sbR2PutRaw/__sbR2GetRaw,
|
|
1445
|
+
// sb_r2_put_c/sb_r2_get_c) the R2 binding's own put()/get() use — reachable
|
|
1446
|
+
// separately only via a raw `__sbCall(op:"r2.put"/"r2.get")`, which nothing
|
|
1447
|
+
// in this codebase does today, but broker.ts's dispatch() answers the same
|
|
1448
|
+
// two ops, so the contract stays identical on both backends either way
|
|
1449
|
+
// (#56: neither one keeps object bytes in SQLite any more).
|
|
1353
1450
|
if (op === "r2.put") {
|
|
1354
|
-
const
|
|
1355
|
-
|
|
1356
|
-
|
|
1357
|
-
|
|
1358
|
-
|
|
1359
|
-
|
|
1360
|
-
[
|
|
1361
|
-
msg.bucket,
|
|
1362
|
-
msg.key,
|
|
1363
|
-
body,
|
|
1364
|
-
body.length,
|
|
1365
|
-
etag,
|
|
1366
|
-
new Date().toISOString(),
|
|
1451
|
+
const reply = JSON.parse(
|
|
1452
|
+
__sbR2PutRaw(
|
|
1453
|
+
store,
|
|
1454
|
+
String(msg.bucket),
|
|
1455
|
+
String(msg.key),
|
|
1456
|
+
String(msg.body == null ? "" : msg.body),
|
|
1367
1457
|
JSON.stringify(msg.httpMetadata || {}),
|
|
1368
1458
|
JSON.stringify(msg.customMetadata || {}),
|
|
1369
|
-
|
|
1459
|
+
),
|
|
1370
1460
|
);
|
|
1371
|
-
|
|
1372
|
-
|
|
1373
|
-
object: { key: msg.key, size: body.length, etag, uploaded: new Date().toISOString() },
|
|
1374
|
-
};
|
|
1461
|
+
if (reply.ok === false) throw new Error("r2 put: " + reply.error);
|
|
1462
|
+
return { ok: true, object: { key: msg.key, size: reply.size, etag: reply.etag, uploaded: reply.uploaded } };
|
|
1375
1463
|
}
|
|
1376
1464
|
if (op === "r2.get" || op === "r2.head") {
|
|
1377
|
-
// head must not
|
|
1378
|
-
//
|
|
1465
|
+
// head must not read the blob file: reading an 8 MB object only to drop
|
|
1466
|
+
// it costs the read and a full JSON escape for nothing.
|
|
1379
1467
|
const wantsBody = op === "r2.get";
|
|
1380
1468
|
const r = __sbSql(
|
|
1381
1469
|
store,
|
|
1382
|
-
|
|
1383
|
-
? "SELECT body, size, etag, uploaded, http_json, custom_json FROM r2 WHERE bucket = ? AND key = ?"
|
|
1384
|
-
: "SELECT '', size, etag, uploaded, http_json, custom_json FROM r2 WHERE bucket = ? AND key = ?",
|
|
1470
|
+
"SELECT size, etag, uploaded, http_json, custom_json FROM r2 WHERE bucket = ? AND key = ?",
|
|
1385
1471
|
[msg.bucket, msg.key],
|
|
1386
1472
|
);
|
|
1387
1473
|
if (!r.rows.length) return { ok: true, found: false };
|
|
@@ -1392,17 +1478,18 @@ function __sbEmbeddedDispatch(msg) {
|
|
|
1392
1478
|
found: true,
|
|
1393
1479
|
object: {
|
|
1394
1480
|
key: msg.key,
|
|
1395
|
-
size: Number(row[
|
|
1396
|
-
etag: row[
|
|
1397
|
-
uploaded: row[
|
|
1398
|
-
httpMetadata: JSON.parse(row[
|
|
1399
|
-
customMetadata: JSON.parse(row[
|
|
1481
|
+
size: Number(row[0]),
|
|
1482
|
+
etag: row[1],
|
|
1483
|
+
uploaded: row[2],
|
|
1484
|
+
httpMetadata: JSON.parse(row[3] || "{}"),
|
|
1485
|
+
customMetadata: JSON.parse(row[4] || "{}"),
|
|
1400
1486
|
},
|
|
1401
|
-
body: wantsBody ?
|
|
1487
|
+
body: wantsBody ? __sbR2GetRaw(store, String(msg.bucket), String(msg.key)) : undefined,
|
|
1402
1488
|
};
|
|
1403
1489
|
}
|
|
1404
1490
|
if (op === "r2.delete") {
|
|
1405
1491
|
__sbSql(store, "DELETE FROM r2 WHERE bucket = ? AND key = ?", [msg.bucket, msg.key]);
|
|
1492
|
+
__sbR2DeleteRaw(store, String(msg.bucket), String(msg.key));
|
|
1406
1493
|
return { ok: true };
|
|
1407
1494
|
}
|
|
1408
1495
|
if (op === "r2.list") {
|