@sproutboat/runtime 0.6.5 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,55 @@
1
1
  # @sproutboat/runtime
2
2
 
3
+ ## 0.7.0
4
+
5
+ ### Minor Changes
6
+
7
+ - 6998a15: R2 object bytes no longer live inside SQLite, on either transport (baronunread/sproutboat#56). Every `put()` used to bind the whole object into a `body` column, and every `get()` read it back out — doubling the object through SQLite's own page cache/WAL machinery on top of the buffer it already arrived in, which was the real driver of #56's "large uploads make memory skyrocket" complaint (the JSON-escaping half of that was already fixed in #63).
8
+
9
+ Objects now live in their own file under `r2-blobs/`, named by a hash of bucket+key (so unicode keys and directory traversal are non-issues):
10
+
11
+ - **Embedded (standalone):** `<data-dir>/r2-blobs/` — `sb_r2_put_c`/`sb_r2_get_c` write/read the file directly via C `fopen`/`fwrite`/`fread`, no SQLite blob column at all. `sb_r2_delete_blob` removes the file on delete.
12
+ - **Broker:** colocated with whichever store file already holds that bucket's metadata — the main `db` for a bare-string binding, the resource's own file under `resourceDir` for an account-level one (#74) — so blobs persist across a redeploy exactly when that metadata does. `:memory:` stores (tests, local dev) fall back to an in-memory map.
13
+
14
+ No migration: this is a schema change for new writes going forward, not a converter for objects already stored inline from before.
15
+
16
+ Verified end-to-end against both transports (kitchen-sink's full conformance suite, 24/24 standalone + 27/27 broker) and with new unit tests for file-backed persistence, the `:memory:` fallback, and a real binary round-trip over the v1 wire protocol.
17
+
18
+ **Known limitation found while verifying this, not fixed here:** a binary `put()` body (bytes ≥ 0x80) is corrupted on the way *in*, before it ever reaches storage — `porf_native_fetch_read_value` (the function every inline-C string read in the runtime goes through) always UTF-8-encodes a Latin-1-range string on read, which is correct for genuine text but wrong for opaque bytes coming off a raw HTTP body. This predates this change (the old SQLite-blob path had the identical corruption) and is not something the storage backend can fix — it needs an input-side counterpart to `#176`'s `x-sb-raw-body` marker, or real `ArrayBuffer`/`Uint8Array` body support (the upstream ask already filed as Draft G in `patches/UPSTREAM.md`).
19
+
20
+ ### Patch Changes
21
+
22
+ - 188d8e4: Fix (baronunread/sproutboat#184): a `put()` body (R2, both transports) with bytes ≥ 0x80 was corrupted on the way *in*, before storage — `porf_native_fetch_read_value` (the one function every inline-C string read in the runtime funnels through) always UTF-8-encodes a Latin-1-range value, correct for genuine text, wrong for opaque bytes off a raw HTTP upload. A 256-byte binary upload landed as 384 stored bytes.
23
+
24
+ `__sbR2PutRaw` (embedded) and `__sbCallBin` (broker, the only caller that ever passes it a real body) now try `porf_native_fetch_read_raw_bytes` first, falling back to the normal encoding path only for a genuine multi-byte JS string (which isn't a "bytestring" and so isn't eligible for the raw read anyway). This predates the #56 storage change — the old SQLite-blob `put()` called the identical `read_value`, so it was equally corrupted; the existing R2 conformance check never caught it because it only ever exercises a plain ASCII string.
25
+
26
+ Also fixes an adjacent, independently pre-existing bug found while verifying this: `R2Object.text()`/`.json()` returned the raw stored bytes without UTF-8-decoding them, producing mojibake for any object holding real text beyond ASCII — the same gap `__sbRawBodyResponse` already closed for assets/fetch/service-binding responses. Same fix here (lazy, memoized `__sbFromUtf8` decode).
27
+
28
+ Verified end-to-end against both transports with a real binary upload (`curl --data-binary`, not a synthetic string): stored bytes and `.body`/`.text()` now round-trip byte-for-byte/correctly for binary content, Latin-1 text, and multi-byte Unicode (CJK). Full kitchen-sink conformance stays green on both transports (24/24 standalone, 27/27 broker).
29
+
30
+ **Known remaining gap, not fixed here:** a genuine astral-plane character (an emoji, anything outside the Basic Multilingual Plane, encoded as a UTF-16 surrogate pair) still encodes to the wrong number of UTF-8 bytes on write — confirmed identical before and after this fix, so it's a separate, deeper bug in Porffor's own UTF-16→UTF-8 encoder (the `porf_native_fetch_read_value` "string" branch), not something in sproutboat's own code. Filing separately if it recurs; out of scope for this fix.
31
+
32
+ ## 0.6.6
33
+
34
+ ### Patch Changes
35
+
36
+ - c2b0359: Fix (baronunread/sproutboat#179): `__sbEntry` called `handlers.fetch()` (and
37
+ the `scheduled`/`queue`/`alarm` trigger paths) with no try/catch at all. Any
38
+ synchronous throw, or an async handler's rejected promise, propagated all the
39
+ way up and took down the entire process — not just the request that hit it,
40
+ but every other in-flight and future request on that sprout, until whatever
41
+ supervises it restarts the binary.
42
+
43
+ A throw from `fetch()` now returns a 500 instead. `scheduled()` and `alarm()`
44
+ still reply 204 either way (they had no failure signal before this either).
45
+ A throw from `queue()` falls through to the existing default-ack pass, the
46
+ same outcome as a handler that returns without calling `ack()`/`retry()` on
47
+ every message.
48
+
49
+ `entry-crash.test.js` covers the fetch path: a sync throw, an async
50
+ rejection, and that a throwing request doesn't stop the next one from being
51
+ served.
52
+
3
53
  ## 0.6.5
4
54
 
5
55
  ### Patch Changes
package/package.json CHANGED
@@ -1,8 +1,13 @@
1
1
  {
2
2
  "name": "@sproutboat/runtime",
3
- "version": "0.6.5",
3
+ "version": "0.7.0",
4
4
  "license": "MIT",
5
5
  "type": "module",
6
+ "repository": {
7
+ "type": "git",
8
+ "url": "git+https://github.com/baronunread/sproutboat-packages.git",
9
+ "directory": "packages/runtime"
10
+ },
6
11
  "main": "./src/index.ts",
7
12
  "types": "./src/index.ts",
8
13
  "scripts": {
@@ -0,0 +1,63 @@
1
+ // #179 — an uncaught handler exception must turn into a 500 (or, for
2
+ // scheduled/queue/alarm, a swallowed failure), never a process-crashing
3
+ // throw/rejection out of __sbEntry. Same lift-the-prelude-and-eval approach
4
+ // as client-ip.test.js: the prelude is prepended as text and cannot import.
5
+ import { expect, test } from "bun:test";
6
+ import { readFileSync } from "node:fs";
7
+ import { fileURLToPath } from "node:url";
8
+
9
+ const src = readFileSync(fileURLToPath(new URL("./native-fetch-prelude.js", import.meta.url)), "utf8");
10
+
11
+ // oxlint-disable-next-line no-function-constructor -- lifting the whole text-only prelude for test
12
+ const load = new Function(
13
+ "Porffor",
14
+ "__envMap",
15
+ `${src}
16
+ return { __sbEntry, __sbRegisterDO };`,
17
+ );
18
+ // Every Porffor.c`...` block in the prelude only runs inside a function body
19
+ // (never at load time), so a no-op tag covers all of them: __sbCpuMs and
20
+ // __sbEnv just see an empty result, same as "native op returned nothing."
21
+ const H = load(
22
+ { c: () => "" },
23
+ new Proxy({}, { get: () => undefined }),
24
+ );
25
+
26
+ const req = (overrides = {}) => ({
27
+ headers: { get: (k) => (overrides.headers && overrides.headers[k.toLowerCase()]) ?? null },
28
+ cf: undefined,
29
+ ...overrides,
30
+ });
31
+
32
+ test("a synchronously-throwing fetch handler returns 500, not a throw", async () => {
33
+ const handlers = {
34
+ fetch() {
35
+ throw new Error("boom");
36
+ },
37
+ };
38
+ const res = await H.__sbEntry(handlers, req());
39
+ expect(res.status).toBe(500);
40
+ });
41
+
42
+ test("an async fetch handler whose promise rejects returns 500, not a rejection", async () => {
43
+ const handlers = {
44
+ async fetch() {
45
+ throw new Error("boom");
46
+ },
47
+ };
48
+ const res = await H.__sbEntry(handlers, req());
49
+ expect(res.status).toBe(500);
50
+ });
51
+
52
+ test("a throwing fetch handler doesn't stop the next request from being served", async () => {
53
+ const handlers = {
54
+ fetch(request) {
55
+ if (request.boom) throw new Error("boom");
56
+ return new Response("ok");
57
+ },
58
+ };
59
+ const first = await H.__sbEntry(handlers, req({ boom: true }));
60
+ expect(first.status).toBe(500);
61
+ const second = await H.__sbEntry(handlers, req({ boom: false }));
62
+ expect(await second.text()).toBe("ok");
63
+ });
@@ -1104,11 +1104,18 @@ function __sbR2Object(meta, body) {
1104
1104
  };
1105
1105
  if (body != null) {
1106
1106
  obj.body = body;
1107
+ // #184 — `body` is raw bytes (a bytestring), not decoded text: `.text()`
1108
+ // returning it verbatim produced mojibake for any object holding real
1109
+ // UTF-8 content, the same gap __sbRawBodyResponse already closed for
1110
+ // assets/fetch/service-binding responses. Same fix here: decode lazily,
1111
+ // once, on first read.
1112
+ let decoded = null;
1107
1113
  obj.text = function () {
1108
- return body;
1114
+ if (decoded === null) decoded = __sbFromUtf8(body);
1115
+ return decoded;
1109
1116
  };
1110
1117
  obj.json = function () {
1111
- return JSON.parse(body);
1118
+ return JSON.parse(obj.text());
1112
1119
  };
1113
1120
  }
1114
1121
  return obj;
@@ -1703,15 +1710,31 @@ async function __sbDrainAll(pending) {
1703
1710
  // one *this* async function creates, not a `.then()` derived from someone
1704
1711
  // else's — see the note below on why that distinction matters here.
1705
1712
  async function __sbFetchWithDrain(res, tasks) {
1706
- const r = res && __sbIsFn(res.then) ? await res : res;
1713
+ let r;
1714
+ try {
1715
+ r = res && __sbIsFn(res.then) ? await res : res;
1716
+ } catch {
1717
+ // #179 — a rejected handler promise must not take the process down with it.
1718
+ r = __sbErrorResponse();
1719
+ }
1707
1720
  await __sbDrainWaitUntil(tasks);
1708
1721
  return r;
1709
1722
  }
1710
1723
 
1724
+ // #179 — turn an uncaught handler exception into a 500 instead of letting it
1725
+ // propagate past __sbEntry and crash the whole server.
1726
+ function __sbErrorResponse() {
1727
+ return new Response("Internal Server Error", { status: 500 });
1728
+ }
1729
+
1711
1730
  // Shared by alarm and scheduled: both reply with an empty 204 once their
1712
1731
  // handler (and anything it queued via `waitUntil`) has settled.
1713
1732
  async function __sb204WithDrain(res, tasks) {
1714
- if (res && __sbIsFn(res.then)) await res;
1733
+ try {
1734
+ if (res && __sbIsFn(res.then)) await res;
1735
+ } catch {
1736
+ // #179 — same rule as fetch: a rejected scheduled/alarm promise must not crash the process.
1737
+ }
1715
1738
  await __sbDrainWaitUntil(tasks);
1716
1739
  return new Response("", { status: 204 });
1717
1740
  }
@@ -1746,7 +1769,14 @@ globalThis.__sbEntry = function (handlers, request) {
1746
1769
  if (p && __sbIsFn(p.then)) __sbTasks.push(p);
1747
1770
  },
1748
1771
  };
1749
- const __res = handlers.fetch(request, ctx);
1772
+ let __res;
1773
+ try {
1774
+ __res = handlers.fetch(request, ctx);
1775
+ } catch {
1776
+ // #179 — a synchronous throw anywhere in the handler must not take the
1777
+ // whole process (and every other in-flight request) down with it.
1778
+ return __sbErrorResponse();
1779
+ }
1750
1780
  // Same promise-identity rule as above applies to `__sbFetchWithDrain`'s
1751
1781
  // own return, which is why it's tail-called rather than chained on here.
1752
1782
  if (__sbTasks.length || (__res && __sbIsFn(__res.then))) return __sbFetchWithDrain(__res, __sbTasks);
@@ -1766,10 +1796,16 @@ globalThis.__sbEntry = function (handlers, request) {
1766
1796
  if (p && __sbIsFn(p.then)) __sbTasks.push(p);
1767
1797
  },
1768
1798
  };
1769
- const __sres = handlers.scheduled(
1770
- { cron: body.cron || "", scheduledTime: body.scheduledTime || Date.now(), noRetry() {} },
1771
- ctx,
1772
- );
1799
+ let __sres;
1800
+ try {
1801
+ __sres = handlers.scheduled(
1802
+ { cron: body.cron || "", scheduledTime: body.scheduledTime || Date.now(), noRetry() {} },
1803
+ ctx,
1804
+ );
1805
+ } catch {
1806
+ // #179 — same rule as fetch: don't let a throw here crash the process.
1807
+ return new Response("", { status: 204 });
1808
+ }
1773
1809
  if (__sbTasks.length || (__sres && __sbIsFn(__sres.then))) return __sb204WithDrain(__sres, __sbTasks);
1774
1810
  return new Response("", { status: 204 });
1775
1811
  }
@@ -1797,7 +1833,13 @@ globalThis.__sbEntry = function (handlers, request) {
1797
1833
  const body = __sbReadJson(request);
1798
1834
  const inst = __sbGetDOInstance(String(body.cls || ""), String(body.id || ""));
1799
1835
  if (!__sbIsFn(inst.alarm)) return new Response("no alarm handler", { status: 404 });
1800
- const __ares = inst.alarm();
1836
+ let __ares;
1837
+ try {
1838
+ __ares = inst.alarm();
1839
+ } catch {
1840
+ // #179 — same rule as fetch: don't let a throw here crash the process.
1841
+ return new Response("", { status: 204 });
1842
+ }
1801
1843
  // Was fire-and-forget before: an async `alarm()` and any `state.waitUntil()`
1802
1844
  // it queued were both dropped the moment 204 went out. Await both.
1803
1845
  if (inst.__sbTasks.length || (__ares && __sbIsFn(__ares.then))) return __sb204WithDrain(__ares, inst.__sbTasks);
@@ -1845,11 +1887,18 @@ async function __sbRunQueueBatch(handlers, body, ctx) {
1845
1887
  for (let i = 0; i < messages.length; i++) messages[i].retry();
1846
1888
  },
1847
1889
  };
1848
- const __qres = handlers.queue(batch, ctx);
1849
- // Was fire-and-forget: an async `queue()` had this default-ack pass run (and
1850
- // the response go out) before its own `await`s did, so any ack()/retry()
1851
- // past the first one never counted.
1852
- if (__qres && __sbIsFn(__qres.then)) await __qres;
1890
+ try {
1891
+ const __qres = handlers.queue(batch, ctx);
1892
+ // Was fire-and-forget: an async `queue()` had this default-ack pass run (and
1893
+ // the response go out) before its own `await`s did, so any ack()/retry()
1894
+ // past the first one never counted.
1895
+ if (__qres && __sbIsFn(__qres.then)) await __qres;
1896
+ } catch {
1897
+ // #179 — a throw here must not crash the process. Whatever ack()/retry()
1898
+ // calls the handler made before throwing still count; anything it never
1899
+ // touched falls through to the default-ack pass below, same as a handler
1900
+ // that returns without calling ack()/retry() on every message.
1901
+ }
1853
1902
  // default: any message neither acked nor retried is treated as acked
1854
1903
  for (let i = 0; i < messages.length; i++) {
1855
1904
  if (acked.indexOf(messages[i].id) === -1 && retried.indexOf(messages[i].id) === -1) acked.push(messages[i].id);
@@ -19,6 +19,13 @@
19
19
 
20
20
  // oxlint-disable-next-line no-unused-expressions -- Porffor.c`...` is inline C the compiler consumes, not a JS expression.
21
21
  Porffor.c`
22
+ // baronunread/sproutboat#176 — defined in render.js, alongside
23
+ // porf_native_fetch_read_value (which every inline-C block below already
24
+ // calls with no forward declaration of its own — that one gets one from
25
+ // Porffor's own generated prototypes, this one doesn't, since it's a local
26
+ // patch addition, not something Porffor knows to prototype early).
27
+ extern int porf_native_fetch_read_raw_bytes(jsval value, const char** out_buf, size_t* out_len);
28
+
22
29
  static int sb_io_all(int fd, unsigned char* buf, size_t len, int writing) {
23
30
  size_t done = 0;
24
31
  while (done < len) {
@@ -237,8 +244,15 @@ function __sbCallBin(reqJson, body) {
237
244
  Porffor.c`
238
245
  const char* __j; size_t __jl; char* __jo = 0;
239
246
  porf_native_fetch_read_value(reqJson, &__j, &__jl, &__jo);
247
+ // baronunread/sproutboat#184 -- the only real (non-empty) body this ever
248
+ // carries today is an R2 put(); read raw, not through read_value's
249
+ // bytestring branch, which always UTF-8-encodes (right for text, wrong
250
+ // for opaque bytes off a raw HTTP upload). A genuine multi-byte JS string
251
+ // isn't a bytestring, so it falls back to the encoding path unaffected.
240
252
  const char* __b; size_t __bl; char* __bo = 0;
241
- porf_native_fetch_read_value(body, &__b, &__bl, &__bo);
253
+ if (porf_native_fetch_read_raw_bytes(body, &__b, &__bl) != 0) {
254
+ porf_native_fetch_read_value(body, &__b, &__bl, &__bo);
255
+ }
242
256
  char* __resp = 0; size_t __resplen = 0;
243
257
  int __rc = sb_broker_roundtrip_v1(__j, __jl, __b, __bl, &__resp, &__resplen);
244
258
  if (__jo) free(__jo);
@@ -53,6 +53,13 @@ extern int64_t sqlite3_last_insert_rowid(sqlite3*);
53
53
  extern const char* sqlite3_errmsg(sqlite3*);
54
54
  extern int sqlite3_close(sqlite3*);
55
55
 
56
+ // baronunread/sproutboat#176 — defined in render.js, alongside
57
+ // porf_native_fetch_read_value (which every inline-C block below already
58
+ // calls with no forward declaration of its own — that one gets one from
59
+ // Porffor's own generated prototypes, this one doesn't, since it's a local
60
+ // patch addition, not something Porffor knows to prototype early).
61
+ extern int porf_native_fetch_read_raw_bytes(jsval value, const char** out_buf, size_t* out_len);
62
+
56
63
  #define SB_SQLITE_ROW 100
57
64
  #define SB_SQLITE_DONE 101
58
65
  #define SB_SQLITE_TRANSIENT ((void*)-1)
@@ -240,11 +247,14 @@ static int sb_bind_params(sqlite3_stmt* st, const char* json) {
240
247
  return 0;
241
248
  }
242
249
 
243
- // --- R2 bodies, without the JSON detour -------------------------------------
244
- // An object's bytes never enter a JSON frame: they arrive as their own string
245
- // parameter and go straight into a BLOB column. That matters for size as much
246
- // as correctness — escaping a binary body inflates it several times over, and
247
- // the arena grows to fit the biggest thing it ever had to hold.
250
+ // --- R2 bodies, without the JSON detour or the SQLite detour ----------------
251
+ // An object's bytes never enter a JSON frame (they arrive as their own string
252
+ // parameter) and never enter SQLite either (baronunread/sproutboat#56): they
253
+ // go straight to their own file under "<data-dir>/r2-blobs/", named by a hash
254
+ // of bucket+key so unicode keys and directory traversal are both non-issues.
255
+ // Doubling every object through SQLite's own page cache on top of the buffer
256
+ // it already arrived in was the real cost behind #56's "large uploads make
257
+ // memory skyrocket" complaint, not just the JSON-escaping #63 already fixed.
248
258
 
249
259
  static void sb_hex32(const unsigned char* in, char* out) {
250
260
  static const char* d = "0123456789abcdef";
@@ -269,7 +279,24 @@ static void sb_iso_now(char* out, size_t cap) {
269
279
  strftime(out, cap, "%Y-%m-%dT%H:%M:%S.000Z", &g);
270
280
  }
271
281
 
272
- // Returns malloc'd JSON metadata; the body itself is never serialised.
282
+ // "<dir-of-store_path>/r2-blobs/<sha256(bucket 0x1e key)>.blob" — deterministic,
283
+ // so no extra column is needed to find an object's file back.
284
+ static void sb_r2_blob_path(char* out, size_t cap, const char* store_path, const char* bucket, const char* key) {
285
+ const char* slash = strrchr(store_path, '/');
286
+ int dirlen = slash ? (int)(slash - store_path) : 0;
287
+ br_sha256_context ctx;
288
+ br_sha256_init(&ctx);
289
+ br_sha256_update(&ctx, bucket, strlen(bucket));
290
+ br_sha256_update(&ctx, "\x1e", 1); // separator: a bucket/key split never collides across it
291
+ br_sha256_update(&ctx, key, strlen(key));
292
+ unsigned char digest[32];
293
+ br_sha256_out(&ctx, digest);
294
+ char hex[65];
295
+ sb_hex32(digest, hex);
296
+ snprintf(out, cap, "%.*s/r2-blobs/%s.blob", dirlen, store_path, hex);
297
+ }
298
+
299
+ // Returns malloc'd JSON metadata; the body itself is never serialised into it.
273
300
  static char* sb_r2_put_c(const char* path, const char* bucket, const char* key, const char* body, size_t bodylen,
274
301
  const char* http_json, const char* custom_json) {
275
302
  sb_buf out = { 0, 0, 0 };
@@ -282,10 +309,19 @@ static char* sb_r2_put_c(const char* path, const char* bucket, const char* key,
282
309
  char uploaded[40];
283
310
  sb_iso_now(uploaded, sizeof(uploaded));
284
311
 
312
+ char blobpath[1024];
313
+ sb_r2_blob_path(blobpath, sizeof(blobpath), path, bucket, key);
314
+ sb_mkdirs(blobpath);
315
+ FILE* f = fopen(blobpath, "wb");
316
+ if (!f) { sb_puts(&out, "{\"ok\":false,\"error\":\"cannot write object\"}"); return out.p; }
317
+ size_t written = bodylen ? fwrite(body, 1, bodylen, f) : 0;
318
+ fclose(f);
319
+ if (written != bodylen) { sb_puts(&out, "{\"ok\":false,\"error\":\"short write\"}"); return out.p; }
320
+
285
321
  sqlite3_stmt* st = 0;
286
322
  const char* sql =
287
- "INSERT INTO r2 (bucket, key, body, size, etag, uploaded, http_json, custom_json) VALUES (?1,?2,?3,?4,?5,?6,?7,?8) "
288
- "ON CONFLICT (bucket, key) DO UPDATE SET body=?3, size=?4, etag=?5, uploaded=?6, http_json=?7, custom_json=?8";
323
+ "INSERT INTO r2 (bucket, key, size, etag, uploaded, http_json, custom_json) VALUES (?1,?2,?3,?4,?5,?6,?7) "
324
+ "ON CONFLICT (bucket, key) DO UPDATE SET size=?3, etag=?4, uploaded=?5, http_json=?6, custom_json=?7";
289
325
  if (sqlite3_prepare_v2(db, sql, -1, &st, 0) != 0 || !st) {
290
326
  sb_puts(&out, "{\"ok\":false,\"error\":");
291
327
  const char* m = sqlite3_errmsg(db);
@@ -295,12 +331,11 @@ static char* sb_r2_put_c(const char* path, const char* bucket, const char* key,
295
331
  }
296
332
  sqlite3_bind_text(st, 1, bucket, -1, SB_SQLITE_TRANSIENT);
297
333
  sqlite3_bind_text(st, 2, key, -1, SB_SQLITE_TRANSIENT);
298
- sqlite3_bind_blob(st, 3, body, (int)bodylen, SB_SQLITE_STATIC);
299
- sqlite3_bind_int64(st, 4, (int64_t)bodylen);
300
- sqlite3_bind_text(st, 5, etag, -1, SB_SQLITE_TRANSIENT);
301
- sqlite3_bind_text(st, 6, uploaded, -1, SB_SQLITE_TRANSIENT);
302
- sqlite3_bind_text(st, 7, http_json && *http_json ? http_json : "{}", -1, SB_SQLITE_TRANSIENT);
303
- sqlite3_bind_text(st, 8, custom_json && *custom_json ? custom_json : "{}", -1, SB_SQLITE_TRANSIENT);
334
+ sqlite3_bind_int64(st, 3, (int64_t)bodylen);
335
+ sqlite3_bind_text(st, 4, etag, -1, SB_SQLITE_TRANSIENT);
336
+ sqlite3_bind_text(st, 5, uploaded, -1, SB_SQLITE_TRANSIENT);
337
+ sqlite3_bind_text(st, 6, http_json && *http_json ? http_json : "{}", -1, SB_SQLITE_TRANSIENT);
338
+ sqlite3_bind_text(st, 7, custom_json && *custom_json ? custom_json : "{}", -1, SB_SQLITE_TRANSIENT);
304
339
  int rc = sqlite3_step(st);
305
340
  sqlite3_finalize(st);
306
341
  if (rc != SB_SQLITE_DONE) { sb_puts(&out, "{\"ok\":false,\"error\":\"r2 put failed\"}"); return out.p; }
@@ -311,30 +346,50 @@ static char* sb_r2_put_c(const char* path, const char* bucket, const char* key,
311
346
  return out.p;
312
347
  }
313
348
 
314
- // Body bytes straight into a Porffor bytestring.
315
- //
316
- // The allocation happens while the statement is still open, so sqlite's own
317
- // buffer is the source and there is no intermediate copy: one 8 MB object means
318
- // one 8 MB allocation, not two. Returns the bytestring pointer, or 0.
349
+ // Reads the object's file into a Porffor bytestring. One malloc (the file
350
+ // read) plus the bytestring's own allocation — no SQLite page-cache copy on
351
+ // top, and no growing single database file holding every object ever put.
352
+ // Returns the bytestring pointer, or 0.
319
353
  static u32 sb_r2_get_c(const char* path, const char* bucket, const char* key, int* found) {
320
354
  *found = 0;
321
355
  int idx = sb_db_for(path);
322
356
  if (idx < 0) return 0;
323
357
  sqlite3_stmt* st = 0;
324
- if (sqlite3_prepare_v2(sb_dbs[idx], "SELECT body FROM r2 WHERE bucket = ? AND key = ?", -1, &st, 0) != 0 || !st) return 0;
358
+ if (sqlite3_prepare_v2(sb_dbs[idx], "SELECT 1 FROM r2 WHERE bucket = ? AND key = ?", -1, &st, 0) != 0 || !st) return 0;
325
359
  sqlite3_bind_text(st, 1, bucket, -1, SB_SQLITE_TRANSIENT);
326
360
  sqlite3_bind_text(st, 2, key, -1, SB_SQLITE_TRANSIENT);
327
- u32 out = 0;
328
- if (sqlite3_step(st) == SB_SQLITE_ROW) {
329
- int n = sqlite3_column_bytes(st, 0);
330
- const void* blob = sqlite3_column_blob(st, 0);
331
- out = porf_native_fetch_alloc_bytestring((const char*)(blob ? blob : ""), (size_t)(n > 0 ? n : 0));
332
- *found = 1;
333
- }
361
+ int has_row = sqlite3_step(st) == SB_SQLITE_ROW;
334
362
  sqlite3_finalize(st);
363
+ if (!has_row) return 0;
364
+
365
+ char blobpath[1024];
366
+ sb_r2_blob_path(blobpath, sizeof(blobpath), path, bucket, key);
367
+ FILE* f = fopen(blobpath, "rb");
368
+ // A metadata row with no blob file (deleted out from under it, disk issue)
369
+ // reads as not-found rather than crashing.
370
+ if (!f) return 0;
371
+ fseek(f, 0, SEEK_END);
372
+ long size = ftell(f);
373
+ rewind(f);
374
+ if (size < 0) { fclose(f); return 0; }
375
+ char* buf = (char*)malloc((size_t)size > 0 ? (size_t)size : 1);
376
+ size_t rd = size > 0 ? fread(buf, 1, (size_t)size, f) : 0;
377
+ fclose(f);
378
+ if (!buf || rd != (size_t)size) { free(buf); return 0; }
379
+ u32 out = porf_native_fetch_alloc_bytestring(buf, (size_t)size);
380
+ free(buf);
381
+ *found = 1;
335
382
  return out;
336
383
  }
337
384
 
385
+ // Best-effort: a metadata row deleted with no blob file (or vice versa) is
386
+ // still a correct end state, so a missing file here is not an error.
387
+ static void sb_r2_delete_blob(const char* path, const char* bucket, const char* key) {
388
+ char blobpath[1024];
389
+ sb_r2_blob_path(blobpath, sizeof(blobpath), path, bucket, key);
390
+ unlink(blobpath);
391
+ }
392
+
338
393
  // Run a script: one or more statements separated by semicolons. sqlite3_exec
339
394
  // handles the whole string, which prepare/step does not — it compiles the first
340
395
  // statement and silently ignores the rest, so a schema built from one exec call
@@ -929,10 +984,18 @@ function __sbR2PutRaw(path, bucket, key, body, httpJson, customJson) {
929
984
  char* __custom = (char*)malloc(__cl + 1); memcpy(__custom, __c, __cl); __custom[__cl] = 0;
930
985
  if (__co) free(__co);
931
986
 
932
- // The body is read in place and handed straight to sqlite3_bind_blob: no
933
- // copy beyond what the binding needs, and no escaping at all.
987
+ // baronunread/sproutboat#184 -- read raw, not through porf_native_fetch_
988
+ // read_value's bytestring branch, which always UTF-8-encodes (correct for
989
+ // genuine text, wrong for opaque bytes -- request.body off a raw HTTP
990
+ // upload is exactly that). R2 is object storage: put()/get() need to be
991
+ // byte-preserving, not text-preserving. Only a bytestring-typed value can
992
+ // be read raw; a genuine multi-byte JS string (real Unicode, "string" type
993
+ // rather than "bytestring") isn't one, so it falls back to the normal
994
+ // encoding path, which already handles that type correctly.
934
995
  const char* __b; size_t __bl; char* __bo = 0;
935
- porf_native_fetch_read_value(body, &__b, &__bl, &__bo);
996
+ if (porf_native_fetch_read_raw_bytes(body, &__b, &__bl) != 0) {
997
+ porf_native_fetch_read_value(body, &__b, &__bl, &__bo);
998
+ }
936
999
  char* __out = sb_r2_put_c(__path, __bucket, __key, __b, __bl, __http, __custom);
937
1000
  if (__bo) free(__bo);
938
1001
 
@@ -974,6 +1037,32 @@ function __sbR2GetRaw(path, bucket, key) {
974
1037
  return res;
975
1038
  }
976
1039
 
1040
+ // R2 delete: also removes the object's own file, not just the metadata row
1041
+ // (#56 — sb_r2_delete_blob is best-effort, a missing file is not an error).
1042
+ // oxlint-disable-next-line no-unused-vars -- read inside the RawC block below, not by JS.
1043
+ function __sbR2DeleteRaw(path, bucket, key) {
1044
+ // oxlint-disable-next-line no-unused-expressions -- Porffor.c`...` is inline C the compiler consumes, not a JS expression.
1045
+ Porffor.c`
1046
+ const char* __p; size_t __pl; char* __po = 0;
1047
+ porf_native_fetch_read_value(path, &__p, &__pl, &__po);
1048
+ char* __path = (char*)malloc(__pl + 1); memcpy(__path, __p, __pl); __path[__pl] = 0;
1049
+ if (__po) free(__po);
1050
+
1051
+ const char* __bk; size_t __bkl; char* __bko = 0;
1052
+ porf_native_fetch_read_value(bucket, &__bk, &__bkl, &__bko);
1053
+ char* __bucket = (char*)malloc(__bkl + 1); memcpy(__bucket, __bk, __bkl); __bucket[__bkl] = 0;
1054
+ if (__bko) free(__bko);
1055
+
1056
+ const char* __k; size_t __kl; char* __ko = 0;
1057
+ porf_native_fetch_read_value(key, &__k, &__kl, &__ko);
1058
+ char* __key = (char*)malloc(__kl + 1); memcpy(__key, __k, __kl); __key[__kl] = 0;
1059
+ if (__ko) free(__ko);
1060
+
1061
+ sb_r2_delete_blob(__path, __bucket, __key);
1062
+ free(__path); free(__bucket); free(__key);
1063
+ `;
1064
+ }
1065
+
977
1066
  // Multi-statement exec. Same string-param pattern as __sbSqlRaw.
978
1067
  // oxlint-disable-next-line no-unused-vars -- read inside the RawC block below, not by JS.
979
1068
  function __sbSqlScriptRaw(path, sql) {
@@ -1158,9 +1247,11 @@ function __sbEnsureSchema() {
1158
1247
  s,
1159
1248
  "CREATE TABLE IF NOT EXISTS kv (ns TEXT NOT NULL, key TEXT NOT NULL, value TEXT NOT NULL, PRIMARY KEY (ns, key))",
1160
1249
  );
1250
+ // #56 — no `body` column: an object's bytes live in their own file under
1251
+ // "<data-dir>/r2-blobs/" (sb_r2_put_c/sb_r2_get_c), never in the sqlite row.
1161
1252
  __sbSql(
1162
1253
  s,
1163
- "CREATE TABLE IF NOT EXISTS r2 (bucket TEXT NOT NULL, key TEXT NOT NULL, body TEXT NOT NULL, size INTEGER NOT NULL, " +
1254
+ "CREATE TABLE IF NOT EXISTS r2 (bucket TEXT NOT NULL, key TEXT NOT NULL, size INTEGER NOT NULL, " +
1164
1255
  "etag TEXT NOT NULL, uploaded TEXT NOT NULL, http_json TEXT NOT NULL DEFAULT '{}', custom_json TEXT NOT NULL DEFAULT '{}', " +
1165
1256
  "PRIMARY KEY (bucket, key))",
1166
1257
  );
@@ -1350,38 +1441,33 @@ function __sbEmbeddedDispatch(msg) {
1350
1441
  return { ok: true, path: dest, bytes: reply.bytes };
1351
1442
  }
1352
1443
 
1444
+ // r2.put/get route through the same raw C path (__sbR2PutRaw/__sbR2GetRaw,
1445
+ // sb_r2_put_c/sb_r2_get_c) the R2 binding's own put()/get() use — reachable
1446
+ // separately only via a raw `__sbCall(op:"r2.put"/"r2.get")`, which nothing
1447
+ // in this codebase does today, but broker.ts's dispatch() answers the same
1448
+ // two ops, so the contract stays identical on both backends either way
1449
+ // (#56: neither one keeps object bytes in SQLite any more).
1353
1450
  if (op === "r2.put") {
1354
- const body = String(msg.body == null ? "" : msg.body);
1355
- const etag = __sbHex(16);
1356
- __sbSql(
1357
- store,
1358
- "INSERT INTO r2 (bucket, key, body, size, etag, uploaded, http_json, custom_json) VALUES (?1,?2,?3,?4,?5,?6,?7,?8) " +
1359
- "ON CONFLICT (bucket, key) DO UPDATE SET body=?3, size=?4, etag=?5, uploaded=?6, http_json=?7, custom_json=?8",
1360
- [
1361
- msg.bucket,
1362
- msg.key,
1363
- body,
1364
- body.length,
1365
- etag,
1366
- new Date().toISOString(),
1451
+ const reply = JSON.parse(
1452
+ __sbR2PutRaw(
1453
+ store,
1454
+ String(msg.bucket),
1455
+ String(msg.key),
1456
+ String(msg.body == null ? "" : msg.body),
1367
1457
  JSON.stringify(msg.httpMetadata || {}),
1368
1458
  JSON.stringify(msg.customMetadata || {}),
1369
- ],
1459
+ ),
1370
1460
  );
1371
- return {
1372
- ok: true,
1373
- object: { key: msg.key, size: body.length, etag, uploaded: new Date().toISOString() },
1374
- };
1461
+ if (reply.ok === false) throw new Error("r2 put: " + reply.error);
1462
+ return { ok: true, object: { key: msg.key, size: reply.size, etag: reply.etag, uploaded: reply.uploaded } };
1375
1463
  }
1376
1464
  if (op === "r2.get" || op === "r2.head") {
1377
- // head must not select `body`: reading an 8 MB blob only to drop it costs
1378
- // the read, the text conversion, and a full JSON escape of the row.
1465
+ // head must not read the blob file: reading an 8 MB object only to drop
1466
+ // it costs the read and a full JSON escape for nothing.
1379
1467
  const wantsBody = op === "r2.get";
1380
1468
  const r = __sbSql(
1381
1469
  store,
1382
- wantsBody
1383
- ? "SELECT body, size, etag, uploaded, http_json, custom_json FROM r2 WHERE bucket = ? AND key = ?"
1384
- : "SELECT '', size, etag, uploaded, http_json, custom_json FROM r2 WHERE bucket = ? AND key = ?",
1470
+ "SELECT size, etag, uploaded, http_json, custom_json FROM r2 WHERE bucket = ? AND key = ?",
1385
1471
  [msg.bucket, msg.key],
1386
1472
  );
1387
1473
  if (!r.rows.length) return { ok: true, found: false };
@@ -1392,17 +1478,18 @@ function __sbEmbeddedDispatch(msg) {
1392
1478
  found: true,
1393
1479
  object: {
1394
1480
  key: msg.key,
1395
- size: Number(row[1]),
1396
- etag: row[2],
1397
- uploaded: row[3],
1398
- httpMetadata: JSON.parse(row[4] || "{}"),
1399
- customMetadata: JSON.parse(row[5] || "{}"),
1481
+ size: Number(row[0]),
1482
+ etag: row[1],
1483
+ uploaded: row[2],
1484
+ httpMetadata: JSON.parse(row[3] || "{}"),
1485
+ customMetadata: JSON.parse(row[4] || "{}"),
1400
1486
  },
1401
- body: wantsBody ? row[0] : undefined,
1487
+ body: wantsBody ? __sbR2GetRaw(store, String(msg.bucket), String(msg.key)) : undefined,
1402
1488
  };
1403
1489
  }
1404
1490
  if (op === "r2.delete") {
1405
1491
  __sbSql(store, "DELETE FROM r2 WHERE bucket = ? AND key = ?", [msg.bucket, msg.key]);
1492
+ __sbR2DeleteRaw(store, String(msg.bucket), String(msg.key));
1406
1493
  return { ok: true };
1407
1494
  }
1408
1495
  if (op === "r2.list") {