@sproutboat/runtime 0.6.6 → 0.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +40 -0
- package/package.json +6 -1
- package/src/native-fetch-prelude.js +24 -2
- package/src/r2-object.test.js +65 -0
- package/src/transport-broker.js +15 -1
- package/src/transport-embedded.js +147 -60
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,45 @@
|
|
|
1
1
|
# @sproutboat/runtime
|
|
2
2
|
|
|
3
|
+
## 0.7.1
|
|
4
|
+
|
|
5
|
+
### Patch Changes
|
|
6
|
+
|
|
7
|
+
- b0c5bbd: Fix #177: `new Response(obj.body)` on an `R2Object` UTF-8-re-encoded raw
|
|
8
|
+
bytes, corrupting any binary content (each byte 0x80-0xFF doubled on the
|
|
9
|
+
wire). Added `R2Object.toResponse(init?)`, which serves the bytes
|
|
10
|
+
unmodified via the same `x-sb-raw-body` marker `__sbRawBodyResponse` already
|
|
11
|
+
uses for assets/fetch/service-binding responses. Defaults `content-type`
|
|
12
|
+
from `httpMetadata` and `etag` from `httpEtag` when not set in `init`.
|
|
13
|
+
|
|
14
|
+
## 0.7.0
|
|
15
|
+
|
|
16
|
+
### Minor Changes
|
|
17
|
+
|
|
18
|
+
- 6998a15: R2 object bytes no longer live inside SQLite, on either transport (baronunread/sproutboat#56). Every `put()` used to bind the whole object into a `body` column, and every `get()` read it back out — doubling the object through SQLite's own page cache/WAL machinery on top of the buffer it already arrived in, which was the real driver of #56's "large uploads make memory skyrocket" complaint (the JSON-escaping half of that was already fixed in #63).
|
|
19
|
+
|
|
20
|
+
Objects now live in their own file under `r2-blobs/`, named by a hash of bucket+key (so unicode keys and directory traversal are non-issues):
|
|
21
|
+
|
|
22
|
+
- **Embedded (standalone):** `<data-dir>/r2-blobs/` — `sb_r2_put_c`/`sb_r2_get_c` write/read the file directly via C `fopen`/`fwrite`/`fread`, no SQLite blob column at all. `sb_r2_delete_blob` removes the file on delete.
|
|
23
|
+
- **Broker:** colocated with whichever store file already holds that bucket's metadata — the main `db` for a bare-string binding, the resource's own file under `resourceDir` for an account-level one (#74) — so blobs persist across a redeploy exactly when that metadata does. `:memory:` stores (tests, local dev) fall back to an in-memory map.
|
|
24
|
+
|
|
25
|
+
No migration: this is a schema change for new writes going forward, not a converter for objects already stored inline from before.
|
|
26
|
+
|
|
27
|
+
Verified end-to-end against both transports (kitchen-sink's full conformance suite, 24/24 standalone + 27/27 broker) and with new unit tests for file-backed persistence, the `:memory:` fallback, and a real binary round-trip over the v1 wire protocol.
|
|
28
|
+
|
|
29
|
+
**Known limitation found while verifying this, not fixed here:** a binary `put()` body (bytes ≥ 0x80) is corrupted on the way *in*, before it ever reaches storage — `porf_native_fetch_read_value` (the function every inline-C string read in the runtime goes through) always UTF-8-encodes a Latin-1-range string on read, which is correct for genuine text but wrong for opaque bytes coming off a raw HTTP body. This predates this change (the old SQLite-blob path had the identical corruption) and is not something the storage backend can fix — it needs an input-side counterpart to `#176`'s `x-sb-raw-body` marker, or real `ArrayBuffer`/`Uint8Array` body support (the upstream ask already filed as Draft G in `patches/UPSTREAM.md`).
|
|
30
|
+
|
|
31
|
+
### Patch Changes
|
|
32
|
+
|
|
33
|
+
- 188d8e4: Fix (baronunread/sproutboat#184): a `put()` body (R2, both transports) with bytes ≥ 0x80 was corrupted on the way *in*, before storage — `porf_native_fetch_read_value` (the one function every inline-C string read in the runtime funnels through) always UTF-8-encodes a Latin-1-range value, correct for genuine text, wrong for opaque bytes off a raw HTTP upload. A 256-byte binary upload landed as 384 stored bytes.
|
|
34
|
+
|
|
35
|
+
`__sbR2PutRaw` (embedded) and `__sbCallBin` (broker, the only caller that ever passes it a real body) now try `porf_native_fetch_read_raw_bytes` first, falling back to the normal encoding path only for a genuine multi-byte JS string (which isn't a "bytestring" and so isn't eligible for the raw read anyway). This predates the #56 storage change — the old SQLite-blob `put()` called the identical `read_value`, so it was equally corrupted; the existing R2 conformance check never caught it because it only ever exercises a plain ASCII string.
|
|
36
|
+
|
|
37
|
+
Also fixes an adjacent, independently pre-existing bug found while verifying this: `R2Object.text()`/`.json()` returned the raw stored bytes without UTF-8-decoding them, producing mojibake for any object holding real text beyond ASCII — the same gap `__sbRawBodyResponse` already closed for assets/fetch/service-binding responses. Same fix here (lazy, memoized `__sbFromUtf8` decode).
|
|
38
|
+
|
|
39
|
+
Verified end-to-end against both transports with a real binary upload (`curl --data-binary`, not a synthetic string): stored bytes and `.body`/`.text()` now round-trip byte-for-byte/correctly for binary content, Latin-1 text, and multi-byte Unicode (CJK). Full kitchen-sink conformance stays green on both transports (24/24 standalone, 27/27 broker).
|
|
40
|
+
|
|
41
|
+
**Known remaining gap, not fixed here:** a genuine astral-plane character (an emoji, anything outside the Basic Multilingual Plane, encoded as a UTF-16 surrogate pair) still encodes to the wrong number of UTF-8 bytes on write — confirmed identical before and after this fix, so it's a separate, deeper bug in Porffor's own UTF-16→UTF-8 encoder (the `porf_native_fetch_read_value` "string" branch), not something in sproutboat's own code. Filing separately if it recurs; out of scope for this fix.
|
|
42
|
+
|
|
3
43
|
## 0.6.6
|
|
4
44
|
|
|
5
45
|
### Patch Changes
|
package/package.json
CHANGED
|
@@ -1,8 +1,13 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@sproutboat/runtime",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.7.1",
|
|
4
4
|
"license": "MIT",
|
|
5
5
|
"type": "module",
|
|
6
|
+
"repository": {
|
|
7
|
+
"type": "git",
|
|
8
|
+
"url": "git+https://github.com/baronunread/sproutboat-packages.git",
|
|
9
|
+
"directory": "packages/runtime"
|
|
10
|
+
},
|
|
6
11
|
"main": "./src/index.ts",
|
|
7
12
|
"types": "./src/index.ts",
|
|
8
13
|
"scripts": {
|
|
@@ -1104,11 +1104,33 @@ function __sbR2Object(meta, body) {
|
|
|
1104
1104
|
};
|
|
1105
1105
|
if (body != null) {
|
|
1106
1106
|
obj.body = body;
|
|
1107
|
+
// #184 — `body` is raw bytes (a bytestring), not decoded text: `.text()`
|
|
1108
|
+
// returning it verbatim produced mojibake for any object holding real
|
|
1109
|
+
// UTF-8 content, the same gap __sbRawBodyResponse already closed for
|
|
1110
|
+
// assets/fetch/service-binding responses. Same fix here: decode lazily,
|
|
1111
|
+
// once, on first read.
|
|
1112
|
+
let decoded = null;
|
|
1107
1113
|
obj.text = function () {
|
|
1108
|
-
|
|
1114
|
+
if (decoded === null) decoded = __sbFromUtf8(body);
|
|
1115
|
+
return decoded;
|
|
1109
1116
|
};
|
|
1110
1117
|
obj.json = function () {
|
|
1111
|
-
return JSON.parse(
|
|
1118
|
+
return JSON.parse(obj.text());
|
|
1119
|
+
};
|
|
1120
|
+
// #177 — `new Response(obj.body)` UTF-8-re-encodes the raw bytes (they're
|
|
1121
|
+
// a bytestring, same ambiguity #172/#176 exist to resolve): any byte
|
|
1122
|
+
// 0x80-0xFF doubles into two bytes, corrupting binary content on the way
|
|
1123
|
+
// out. `toResponse()` is the fix already used for assets/fetch/service
|
|
1124
|
+
// bindings (`__sbRawBodyResponse`, below) — same reserved x-sb-raw-body
|
|
1125
|
+
// marker, applied here too.
|
|
1126
|
+
obj.toResponse = function (init) {
|
|
1127
|
+
const opts = init || {};
|
|
1128
|
+
const headers = new Headers(opts.headers || {});
|
|
1129
|
+
headers.set("x-sb-raw-body", "1");
|
|
1130
|
+
if (!headers.has("content-type") && obj.httpMetadata.contentType)
|
|
1131
|
+
headers.set("content-type", obj.httpMetadata.contentType);
|
|
1132
|
+
if (!headers.has("etag")) headers.set("etag", obj.httpEtag);
|
|
1133
|
+
return __sbRawBodyResponse(body, { status: opts.status, headers });
|
|
1112
1134
|
};
|
|
1113
1135
|
}
|
|
1114
1136
|
return obj;
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
// sproutboat #177 — `new Response(obj.body)` UTF-8-re-encodes an R2 object's
|
|
2
|
+
// raw bytes (they're a bytestring, indistinguishable from Latin-1 text at
|
|
3
|
+
// that layer). `obj.toResponse()` is the fix: same reserved x-sb-raw-body
|
|
4
|
+
// marker __sbRawBodyResponse already uses for assets/fetch/service bindings.
|
|
5
|
+
// The prelude is prepended as text and cannot be imported, so this test lifts
|
|
6
|
+
// the relevant functions out and exercises them directly, against real
|
|
7
|
+
// Response/Headers (both are Bun globals). If any of these move or their
|
|
8
|
+
// boundary markers change, this fails loudly rather than silently testing
|
|
9
|
+
// nothing.
|
|
10
|
+
import { expect, test } from "bun:test";
|
|
11
|
+
import { readFileSync } from "node:fs";
|
|
12
|
+
import { fileURLToPath } from "node:url";
|
|
13
|
+
|
|
14
|
+
const src = readFileSync(fileURLToPath(new URL("./native-fetch-prelude.js", import.meta.url)), "utf8");
|
|
15
|
+
const utf8Start = src.indexOf("function __sbFromUtf8(");
|
|
16
|
+
const rawStart = src.indexOf("function __sbRawBodyResponse(");
|
|
17
|
+
const utf8End = rawStart;
|
|
18
|
+
const rawEnd = src.indexOf("function __sbEqCt(");
|
|
19
|
+
const r2Start = src.indexOf("function __sbR2Object(");
|
|
20
|
+
const r2End = src.indexOf("// Installed only when the project declares bindings.");
|
|
21
|
+
if (utf8Start === -1 || utf8End === -1) throw new Error("__sbFromUtf8 block not found in prelude");
|
|
22
|
+
if (rawStart === -1 || rawEnd === -1) throw new Error("__sbRawBodyResponse block not found in prelude");
|
|
23
|
+
if (r2Start === -1 || r2End === -1 || r2End < r2Start) throw new Error("__sbR2Object block not found in prelude");
|
|
24
|
+
|
|
25
|
+
// oxlint-disable-next-line no-function-constructor -- lifting pure blocks out of the text-only prelude for test
|
|
26
|
+
const load = new Function(
|
|
27
|
+
`${src.slice(utf8Start, utf8End)}\n${src.slice(rawStart, rawEnd)}\n${src.slice(r2Start, r2End)}\nreturn { __sbR2Object };`,
|
|
28
|
+
);
|
|
29
|
+
const { __sbR2Object } = load();
|
|
30
|
+
|
|
31
|
+
const meta = { key: "f.bin", size: 3, etag: "abc123", uploaded: "2026-09-14T00:00:00.000Z" };
|
|
32
|
+
|
|
33
|
+
// The byte-for-byte round trip itself happens in the compiled runtime's
|
|
34
|
+
// write_response_value (#176's C-side scan, packages/toolchain/src/patch.ts) --
|
|
35
|
+
// a plain-JS Response, Bun's included, re-encodes a bytestring the same way
|
|
36
|
+
// Porffor's does, so this pure-JS test can only prove toResponse() sets the
|
|
37
|
+
// marker that steers that C-side read; the actual byte preservation was
|
|
38
|
+
// verified manually against a real compiled standalone binary (#177).
|
|
39
|
+
test("toResponse() tags the response with the runtime's raw-body marker", () => {
|
|
40
|
+
const bytes = [0x80, 0x81, 0xff].map((b) => String.fromCharCode(b)).join("");
|
|
41
|
+
const obj = __sbR2Object(meta, bytes);
|
|
42
|
+
const fixed = obj.toResponse();
|
|
43
|
+
expect(fixed.headers.get("x-sb-raw-body")).toBe("1");
|
|
44
|
+
});
|
|
45
|
+
|
|
46
|
+
test("toResponse() defaults content-type from httpMetadata and etag from httpEtag", () => {
|
|
47
|
+
const obj = __sbR2Object({ ...meta, httpMetadata: { contentType: "image/png" } }, "x");
|
|
48
|
+
const resp = obj.toResponse();
|
|
49
|
+
expect(resp.headers.get("content-type")).toBe("image/png");
|
|
50
|
+
expect(resp.headers.get("etag")).toBe(obj.httpEtag);
|
|
51
|
+
});
|
|
52
|
+
|
|
53
|
+
test("toResponse() lets caller-supplied headers/status override the defaults", () => {
|
|
54
|
+
const obj = __sbR2Object({ ...meta, httpMetadata: { contentType: "image/png" } }, "x");
|
|
55
|
+
const resp = obj.toResponse({ status: 206, headers: { "content-type": "application/octet-stream" } });
|
|
56
|
+
expect(resp.status).toBe(206);
|
|
57
|
+
expect(resp.headers.get("content-type")).toBe("application/octet-stream");
|
|
58
|
+
});
|
|
59
|
+
|
|
60
|
+
test("toResponse() on an empty body still returns bytes, not the string \"\"", async () => {
|
|
61
|
+
const obj = __sbR2Object(meta, "");
|
|
62
|
+
const resp = obj.toResponse();
|
|
63
|
+
const buf = new Uint8Array(await resp.arrayBuffer());
|
|
64
|
+
expect(buf.length).toBe(0);
|
|
65
|
+
});
|
package/src/transport-broker.js
CHANGED
|
@@ -19,6 +19,13 @@
|
|
|
19
19
|
|
|
20
20
|
// oxlint-disable-next-line no-unused-expressions -- Porffor.c`...` is inline C the compiler consumes, not a JS expression.
|
|
21
21
|
Porffor.c`
|
|
22
|
+
// baronunread/sproutboat#176 — defined in render.js, alongside
|
|
23
|
+
// porf_native_fetch_read_value (which every inline-C block below already
|
|
24
|
+
// calls with no forward declaration of its own — that one gets one from
|
|
25
|
+
// Porffor's own generated prototypes, this one doesn't, since it's a local
|
|
26
|
+
// patch addition, not something Porffor knows to prototype early).
|
|
27
|
+
extern int porf_native_fetch_read_raw_bytes(jsval value, const char** out_buf, size_t* out_len);
|
|
28
|
+
|
|
22
29
|
static int sb_io_all(int fd, unsigned char* buf, size_t len, int writing) {
|
|
23
30
|
size_t done = 0;
|
|
24
31
|
while (done < len) {
|
|
@@ -237,8 +244,15 @@ function __sbCallBin(reqJson, body) {
|
|
|
237
244
|
Porffor.c`
|
|
238
245
|
const char* __j; size_t __jl; char* __jo = 0;
|
|
239
246
|
porf_native_fetch_read_value(reqJson, &__j, &__jl, &__jo);
|
|
247
|
+
// baronunread/sproutboat#184 -- the only real (non-empty) body this ever
|
|
248
|
+
// carries today is an R2 put(); read raw, not through read_value's
|
|
249
|
+
// bytestring branch, which always UTF-8-encodes (right for text, wrong
|
|
250
|
+
// for opaque bytes off a raw HTTP upload). A genuine multi-byte JS string
|
|
251
|
+
// isn't a bytestring, so it falls back to the encoding path unaffected.
|
|
240
252
|
const char* __b; size_t __bl; char* __bo = 0;
|
|
241
|
-
|
|
253
|
+
if (porf_native_fetch_read_raw_bytes(body, &__b, &__bl) != 0) {
|
|
254
|
+
porf_native_fetch_read_value(body, &__b, &__bl, &__bo);
|
|
255
|
+
}
|
|
242
256
|
char* __resp = 0; size_t __resplen = 0;
|
|
243
257
|
int __rc = sb_broker_roundtrip_v1(__j, __jl, __b, __bl, &__resp, &__resplen);
|
|
244
258
|
if (__jo) free(__jo);
|
|
@@ -53,6 +53,13 @@ extern int64_t sqlite3_last_insert_rowid(sqlite3*);
|
|
|
53
53
|
extern const char* sqlite3_errmsg(sqlite3*);
|
|
54
54
|
extern int sqlite3_close(sqlite3*);
|
|
55
55
|
|
|
56
|
+
// baronunread/sproutboat#176 — defined in render.js, alongside
|
|
57
|
+
// porf_native_fetch_read_value (which every inline-C block below already
|
|
58
|
+
// calls with no forward declaration of its own — that one gets one from
|
|
59
|
+
// Porffor's own generated prototypes, this one doesn't, since it's a local
|
|
60
|
+
// patch addition, not something Porffor knows to prototype early).
|
|
61
|
+
extern int porf_native_fetch_read_raw_bytes(jsval value, const char** out_buf, size_t* out_len);
|
|
62
|
+
|
|
56
63
|
#define SB_SQLITE_ROW 100
|
|
57
64
|
#define SB_SQLITE_DONE 101
|
|
58
65
|
#define SB_SQLITE_TRANSIENT ((void*)-1)
|
|
@@ -240,11 +247,14 @@ static int sb_bind_params(sqlite3_stmt* st, const char* json) {
|
|
|
240
247
|
return 0;
|
|
241
248
|
}
|
|
242
249
|
|
|
243
|
-
// --- R2 bodies, without the JSON detour
|
|
244
|
-
// An object's bytes never enter a JSON frame
|
|
245
|
-
// parameter and
|
|
246
|
-
//
|
|
247
|
-
//
|
|
250
|
+
// --- R2 bodies, without the JSON detour or the SQLite detour ----------------
|
|
251
|
+
// An object's bytes never enter a JSON frame (they arrive as their own string
|
|
252
|
+
// parameter) and never enter SQLite either (baronunread/sproutboat#56): they
|
|
253
|
+
// go straight to their own file under "<data-dir>/r2-blobs/", named by a hash
|
|
254
|
+
// of bucket+key so unicode keys and directory traversal are both non-issues.
|
|
255
|
+
// Doubling every object through SQLite's own page cache on top of the buffer
|
|
256
|
+
// it already arrived in was the real cost behind #56's "large uploads make
|
|
257
|
+
// memory skyrocket" complaint, not just the JSON-escaping #63 already fixed.
|
|
248
258
|
|
|
249
259
|
static void sb_hex32(const unsigned char* in, char* out) {
|
|
250
260
|
static const char* d = "0123456789abcdef";
|
|
@@ -269,7 +279,24 @@ static void sb_iso_now(char* out, size_t cap) {
|
|
|
269
279
|
strftime(out, cap, "%Y-%m-%dT%H:%M:%S.000Z", &g);
|
|
270
280
|
}
|
|
271
281
|
|
|
272
|
-
//
|
|
282
|
+
// "<dir-of-store_path>/r2-blobs/<sha256(bucket 0x1e key)>.blob" — deterministic,
|
|
283
|
+
// so no extra column is needed to find an object's file back.
|
|
284
|
+
static void sb_r2_blob_path(char* out, size_t cap, const char* store_path, const char* bucket, const char* key) {
|
|
285
|
+
const char* slash = strrchr(store_path, '/');
|
|
286
|
+
int dirlen = slash ? (int)(slash - store_path) : 0;
|
|
287
|
+
br_sha256_context ctx;
|
|
288
|
+
br_sha256_init(&ctx);
|
|
289
|
+
br_sha256_update(&ctx, bucket, strlen(bucket));
|
|
290
|
+
br_sha256_update(&ctx, "\x1e", 1); // separator: a bucket/key split never collides across it
|
|
291
|
+
br_sha256_update(&ctx, key, strlen(key));
|
|
292
|
+
unsigned char digest[32];
|
|
293
|
+
br_sha256_out(&ctx, digest);
|
|
294
|
+
char hex[65];
|
|
295
|
+
sb_hex32(digest, hex);
|
|
296
|
+
snprintf(out, cap, "%.*s/r2-blobs/%s.blob", dirlen, store_path, hex);
|
|
297
|
+
}
|
|
298
|
+
|
|
299
|
+
// Returns malloc'd JSON metadata; the body itself is never serialised into it.
|
|
273
300
|
static char* sb_r2_put_c(const char* path, const char* bucket, const char* key, const char* body, size_t bodylen,
|
|
274
301
|
const char* http_json, const char* custom_json) {
|
|
275
302
|
sb_buf out = { 0, 0, 0 };
|
|
@@ -282,10 +309,19 @@ static char* sb_r2_put_c(const char* path, const char* bucket, const char* key,
|
|
|
282
309
|
char uploaded[40];
|
|
283
310
|
sb_iso_now(uploaded, sizeof(uploaded));
|
|
284
311
|
|
|
312
|
+
char blobpath[1024];
|
|
313
|
+
sb_r2_blob_path(blobpath, sizeof(blobpath), path, bucket, key);
|
|
314
|
+
sb_mkdirs(blobpath);
|
|
315
|
+
FILE* f = fopen(blobpath, "wb");
|
|
316
|
+
if (!f) { sb_puts(&out, "{\"ok\":false,\"error\":\"cannot write object\"}"); return out.p; }
|
|
317
|
+
size_t written = bodylen ? fwrite(body, 1, bodylen, f) : 0;
|
|
318
|
+
fclose(f);
|
|
319
|
+
if (written != bodylen) { sb_puts(&out, "{\"ok\":false,\"error\":\"short write\"}"); return out.p; }
|
|
320
|
+
|
|
285
321
|
sqlite3_stmt* st = 0;
|
|
286
322
|
const char* sql =
|
|
287
|
-
"INSERT INTO r2 (bucket, key,
|
|
288
|
-
"ON CONFLICT (bucket, key) DO UPDATE SET
|
|
323
|
+
"INSERT INTO r2 (bucket, key, size, etag, uploaded, http_json, custom_json) VALUES (?1,?2,?3,?4,?5,?6,?7) "
|
|
324
|
+
"ON CONFLICT (bucket, key) DO UPDATE SET size=?3, etag=?4, uploaded=?5, http_json=?6, custom_json=?7";
|
|
289
325
|
if (sqlite3_prepare_v2(db, sql, -1, &st, 0) != 0 || !st) {
|
|
290
326
|
sb_puts(&out, "{\"ok\":false,\"error\":");
|
|
291
327
|
const char* m = sqlite3_errmsg(db);
|
|
@@ -295,12 +331,11 @@ static char* sb_r2_put_c(const char* path, const char* bucket, const char* key,
|
|
|
295
331
|
}
|
|
296
332
|
sqlite3_bind_text(st, 1, bucket, -1, SB_SQLITE_TRANSIENT);
|
|
297
333
|
sqlite3_bind_text(st, 2, key, -1, SB_SQLITE_TRANSIENT);
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
sqlite3_bind_text(st, 5,
|
|
301
|
-
sqlite3_bind_text(st, 6,
|
|
302
|
-
sqlite3_bind_text(st, 7,
|
|
303
|
-
sqlite3_bind_text(st, 8, custom_json && *custom_json ? custom_json : "{}", -1, SB_SQLITE_TRANSIENT);
|
|
334
|
+
sqlite3_bind_int64(st, 3, (int64_t)bodylen);
|
|
335
|
+
sqlite3_bind_text(st, 4, etag, -1, SB_SQLITE_TRANSIENT);
|
|
336
|
+
sqlite3_bind_text(st, 5, uploaded, -1, SB_SQLITE_TRANSIENT);
|
|
337
|
+
sqlite3_bind_text(st, 6, http_json && *http_json ? http_json : "{}", -1, SB_SQLITE_TRANSIENT);
|
|
338
|
+
sqlite3_bind_text(st, 7, custom_json && *custom_json ? custom_json : "{}", -1, SB_SQLITE_TRANSIENT);
|
|
304
339
|
int rc = sqlite3_step(st);
|
|
305
340
|
sqlite3_finalize(st);
|
|
306
341
|
if (rc != SB_SQLITE_DONE) { sb_puts(&out, "{\"ok\":false,\"error\":\"r2 put failed\"}"); return out.p; }
|
|
@@ -311,30 +346,50 @@ static char* sb_r2_put_c(const char* path, const char* bucket, const char* key,
|
|
|
311
346
|
return out.p;
|
|
312
347
|
}
|
|
313
348
|
|
|
314
|
-
//
|
|
315
|
-
//
|
|
316
|
-
//
|
|
317
|
-
//
|
|
318
|
-
// one 8 MB allocation, not two. Returns the bytestring pointer, or 0.
|
|
349
|
+
// Reads the object's file into a Porffor bytestring. One malloc (the file
|
|
350
|
+
// read) plus the bytestring's own allocation — no SQLite page-cache copy on
|
|
351
|
+
// top, and no growing single database file holding every object ever put.
|
|
352
|
+
// Returns the bytestring pointer, or 0.
|
|
319
353
|
static u32 sb_r2_get_c(const char* path, const char* bucket, const char* key, int* found) {
|
|
320
354
|
*found = 0;
|
|
321
355
|
int idx = sb_db_for(path);
|
|
322
356
|
if (idx < 0) return 0;
|
|
323
357
|
sqlite3_stmt* st = 0;
|
|
324
|
-
if (sqlite3_prepare_v2(sb_dbs[idx], "SELECT
|
|
358
|
+
if (sqlite3_prepare_v2(sb_dbs[idx], "SELECT 1 FROM r2 WHERE bucket = ? AND key = ?", -1, &st, 0) != 0 || !st) return 0;
|
|
325
359
|
sqlite3_bind_text(st, 1, bucket, -1, SB_SQLITE_TRANSIENT);
|
|
326
360
|
sqlite3_bind_text(st, 2, key, -1, SB_SQLITE_TRANSIENT);
|
|
327
|
-
|
|
328
|
-
if (sqlite3_step(st) == SB_SQLITE_ROW) {
|
|
329
|
-
int n = sqlite3_column_bytes(st, 0);
|
|
330
|
-
const void* blob = sqlite3_column_blob(st, 0);
|
|
331
|
-
out = porf_native_fetch_alloc_bytestring((const char*)(blob ? blob : ""), (size_t)(n > 0 ? n : 0));
|
|
332
|
-
*found = 1;
|
|
333
|
-
}
|
|
361
|
+
int has_row = sqlite3_step(st) == SB_SQLITE_ROW;
|
|
334
362
|
sqlite3_finalize(st);
|
|
363
|
+
if (!has_row) return 0;
|
|
364
|
+
|
|
365
|
+
char blobpath[1024];
|
|
366
|
+
sb_r2_blob_path(blobpath, sizeof(blobpath), path, bucket, key);
|
|
367
|
+
FILE* f = fopen(blobpath, "rb");
|
|
368
|
+
// A metadata row with no blob file (deleted out from under it, disk issue)
|
|
369
|
+
// reads as not-found rather than crashing.
|
|
370
|
+
if (!f) return 0;
|
|
371
|
+
fseek(f, 0, SEEK_END);
|
|
372
|
+
long size = ftell(f);
|
|
373
|
+
rewind(f);
|
|
374
|
+
if (size < 0) { fclose(f); return 0; }
|
|
375
|
+
char* buf = (char*)malloc((size_t)size > 0 ? (size_t)size : 1);
|
|
376
|
+
size_t rd = size > 0 ? fread(buf, 1, (size_t)size, f) : 0;
|
|
377
|
+
fclose(f);
|
|
378
|
+
if (!buf || rd != (size_t)size) { free(buf); return 0; }
|
|
379
|
+
u32 out = porf_native_fetch_alloc_bytestring(buf, (size_t)size);
|
|
380
|
+
free(buf);
|
|
381
|
+
*found = 1;
|
|
335
382
|
return out;
|
|
336
383
|
}
|
|
337
384
|
|
|
385
|
+
// Best-effort: a metadata row deleted with no blob file (or vice versa) is
|
|
386
|
+
// still a correct end state, so a missing file here is not an error.
|
|
387
|
+
static void sb_r2_delete_blob(const char* path, const char* bucket, const char* key) {
|
|
388
|
+
char blobpath[1024];
|
|
389
|
+
sb_r2_blob_path(blobpath, sizeof(blobpath), path, bucket, key);
|
|
390
|
+
unlink(blobpath);
|
|
391
|
+
}
|
|
392
|
+
|
|
338
393
|
// Run a script: one or more statements separated by semicolons. sqlite3_exec
|
|
339
394
|
// handles the whole string, which prepare/step does not — it compiles the first
|
|
340
395
|
// statement and silently ignores the rest, so a schema built from one exec call
|
|
@@ -929,10 +984,18 @@ function __sbR2PutRaw(path, bucket, key, body, httpJson, customJson) {
|
|
|
929
984
|
char* __custom = (char*)malloc(__cl + 1); memcpy(__custom, __c, __cl); __custom[__cl] = 0;
|
|
930
985
|
if (__co) free(__co);
|
|
931
986
|
|
|
932
|
-
//
|
|
933
|
-
//
|
|
987
|
+
// baronunread/sproutboat#184 -- read raw, not through porf_native_fetch_
|
|
988
|
+
// read_value's bytestring branch, which always UTF-8-encodes (correct for
|
|
989
|
+
// genuine text, wrong for opaque bytes -- request.body off a raw HTTP
|
|
990
|
+
// upload is exactly that). R2 is object storage: put()/get() need to be
|
|
991
|
+
// byte-preserving, not text-preserving. Only a bytestring-typed value can
|
|
992
|
+
// be read raw; a genuine multi-byte JS string (real Unicode, "string" type
|
|
993
|
+
// rather than "bytestring") isn't one, so it falls back to the normal
|
|
994
|
+
// encoding path, which already handles that type correctly.
|
|
934
995
|
const char* __b; size_t __bl; char* __bo = 0;
|
|
935
|
-
|
|
996
|
+
if (porf_native_fetch_read_raw_bytes(body, &__b, &__bl) != 0) {
|
|
997
|
+
porf_native_fetch_read_value(body, &__b, &__bl, &__bo);
|
|
998
|
+
}
|
|
936
999
|
char* __out = sb_r2_put_c(__path, __bucket, __key, __b, __bl, __http, __custom);
|
|
937
1000
|
if (__bo) free(__bo);
|
|
938
1001
|
|
|
@@ -974,6 +1037,32 @@ function __sbR2GetRaw(path, bucket, key) {
|
|
|
974
1037
|
return res;
|
|
975
1038
|
}
|
|
976
1039
|
|
|
1040
|
+
// R2 delete: also removes the object's own file, not just the metadata row
|
|
1041
|
+
// (#56 — sb_r2_delete_blob is best-effort, a missing file is not an error).
|
|
1042
|
+
// oxlint-disable-next-line no-unused-vars -- read inside the RawC block below, not by JS.
|
|
1043
|
+
function __sbR2DeleteRaw(path, bucket, key) {
|
|
1044
|
+
// oxlint-disable-next-line no-unused-expressions -- Porffor.c`...` is inline C the compiler consumes, not a JS expression.
|
|
1045
|
+
Porffor.c`
|
|
1046
|
+
const char* __p; size_t __pl; char* __po = 0;
|
|
1047
|
+
porf_native_fetch_read_value(path, &__p, &__pl, &__po);
|
|
1048
|
+
char* __path = (char*)malloc(__pl + 1); memcpy(__path, __p, __pl); __path[__pl] = 0;
|
|
1049
|
+
if (__po) free(__po);
|
|
1050
|
+
|
|
1051
|
+
const char* __bk; size_t __bkl; char* __bko = 0;
|
|
1052
|
+
porf_native_fetch_read_value(bucket, &__bk, &__bkl, &__bko);
|
|
1053
|
+
char* __bucket = (char*)malloc(__bkl + 1); memcpy(__bucket, __bk, __bkl); __bucket[__bkl] = 0;
|
|
1054
|
+
if (__bko) free(__bko);
|
|
1055
|
+
|
|
1056
|
+
const char* __k; size_t __kl; char* __ko = 0;
|
|
1057
|
+
porf_native_fetch_read_value(key, &__k, &__kl, &__ko);
|
|
1058
|
+
char* __key = (char*)malloc(__kl + 1); memcpy(__key, __k, __kl); __key[__kl] = 0;
|
|
1059
|
+
if (__ko) free(__ko);
|
|
1060
|
+
|
|
1061
|
+
sb_r2_delete_blob(__path, __bucket, __key);
|
|
1062
|
+
free(__path); free(__bucket); free(__key);
|
|
1063
|
+
`;
|
|
1064
|
+
}
|
|
1065
|
+
|
|
977
1066
|
// Multi-statement exec. Same string-param pattern as __sbSqlRaw.
|
|
978
1067
|
// oxlint-disable-next-line no-unused-vars -- read inside the RawC block below, not by JS.
|
|
979
1068
|
function __sbSqlScriptRaw(path, sql) {
|
|
@@ -1158,9 +1247,11 @@ function __sbEnsureSchema() {
|
|
|
1158
1247
|
s,
|
|
1159
1248
|
"CREATE TABLE IF NOT EXISTS kv (ns TEXT NOT NULL, key TEXT NOT NULL, value TEXT NOT NULL, PRIMARY KEY (ns, key))",
|
|
1160
1249
|
);
|
|
1250
|
+
// #56 — no `body` column: an object's bytes live in their own file under
|
|
1251
|
+
// "<data-dir>/r2-blobs/" (sb_r2_put_c/sb_r2_get_c), never in the sqlite row.
|
|
1161
1252
|
__sbSql(
|
|
1162
1253
|
s,
|
|
1163
|
-
"CREATE TABLE IF NOT EXISTS r2 (bucket TEXT NOT NULL, key TEXT NOT NULL,
|
|
1254
|
+
"CREATE TABLE IF NOT EXISTS r2 (bucket TEXT NOT NULL, key TEXT NOT NULL, size INTEGER NOT NULL, " +
|
|
1164
1255
|
"etag TEXT NOT NULL, uploaded TEXT NOT NULL, http_json TEXT NOT NULL DEFAULT '{}', custom_json TEXT NOT NULL DEFAULT '{}', " +
|
|
1165
1256
|
"PRIMARY KEY (bucket, key))",
|
|
1166
1257
|
);
|
|
@@ -1350,38 +1441,33 @@ function __sbEmbeddedDispatch(msg) {
|
|
|
1350
1441
|
return { ok: true, path: dest, bytes: reply.bytes };
|
|
1351
1442
|
}
|
|
1352
1443
|
|
|
1444
|
+
// r2.put/get route through the same raw C path (__sbR2PutRaw/__sbR2GetRaw,
|
|
1445
|
+
// sb_r2_put_c/sb_r2_get_c) the R2 binding's own put()/get() use — reachable
|
|
1446
|
+
// separately only via a raw `__sbCall(op:"r2.put"/"r2.get")`, which nothing
|
|
1447
|
+
// in this codebase does today, but broker.ts's dispatch() answers the same
|
|
1448
|
+
// two ops, so the contract stays identical on both backends either way
|
|
1449
|
+
// (#56: neither one keeps object bytes in SQLite any more).
|
|
1353
1450
|
if (op === "r2.put") {
|
|
1354
|
-
const
|
|
1355
|
-
|
|
1356
|
-
|
|
1357
|
-
|
|
1358
|
-
|
|
1359
|
-
|
|
1360
|
-
[
|
|
1361
|
-
msg.bucket,
|
|
1362
|
-
msg.key,
|
|
1363
|
-
body,
|
|
1364
|
-
body.length,
|
|
1365
|
-
etag,
|
|
1366
|
-
new Date().toISOString(),
|
|
1451
|
+
const reply = JSON.parse(
|
|
1452
|
+
__sbR2PutRaw(
|
|
1453
|
+
store,
|
|
1454
|
+
String(msg.bucket),
|
|
1455
|
+
String(msg.key),
|
|
1456
|
+
String(msg.body == null ? "" : msg.body),
|
|
1367
1457
|
JSON.stringify(msg.httpMetadata || {}),
|
|
1368
1458
|
JSON.stringify(msg.customMetadata || {}),
|
|
1369
|
-
|
|
1459
|
+
),
|
|
1370
1460
|
);
|
|
1371
|
-
|
|
1372
|
-
|
|
1373
|
-
object: { key: msg.key, size: body.length, etag, uploaded: new Date().toISOString() },
|
|
1374
|
-
};
|
|
1461
|
+
if (reply.ok === false) throw new Error("r2 put: " + reply.error);
|
|
1462
|
+
return { ok: true, object: { key: msg.key, size: reply.size, etag: reply.etag, uploaded: reply.uploaded } };
|
|
1375
1463
|
}
|
|
1376
1464
|
if (op === "r2.get" || op === "r2.head") {
|
|
1377
|
-
// head must not
|
|
1378
|
-
//
|
|
1465
|
+
// head must not read the blob file: reading an 8 MB object only to drop
|
|
1466
|
+
// it costs the read and a full JSON escape for nothing.
|
|
1379
1467
|
const wantsBody = op === "r2.get";
|
|
1380
1468
|
const r = __sbSql(
|
|
1381
1469
|
store,
|
|
1382
|
-
|
|
1383
|
-
? "SELECT body, size, etag, uploaded, http_json, custom_json FROM r2 WHERE bucket = ? AND key = ?"
|
|
1384
|
-
: "SELECT '', size, etag, uploaded, http_json, custom_json FROM r2 WHERE bucket = ? AND key = ?",
|
|
1470
|
+
"SELECT size, etag, uploaded, http_json, custom_json FROM r2 WHERE bucket = ? AND key = ?",
|
|
1385
1471
|
[msg.bucket, msg.key],
|
|
1386
1472
|
);
|
|
1387
1473
|
if (!r.rows.length) return { ok: true, found: false };
|
|
@@ -1392,17 +1478,18 @@ function __sbEmbeddedDispatch(msg) {
|
|
|
1392
1478
|
found: true,
|
|
1393
1479
|
object: {
|
|
1394
1480
|
key: msg.key,
|
|
1395
|
-
size: Number(row[
|
|
1396
|
-
etag: row[
|
|
1397
|
-
uploaded: row[
|
|
1398
|
-
httpMetadata: JSON.parse(row[
|
|
1399
|
-
customMetadata: JSON.parse(row[
|
|
1481
|
+
size: Number(row[0]),
|
|
1482
|
+
etag: row[1],
|
|
1483
|
+
uploaded: row[2],
|
|
1484
|
+
httpMetadata: JSON.parse(row[3] || "{}"),
|
|
1485
|
+
customMetadata: JSON.parse(row[4] || "{}"),
|
|
1400
1486
|
},
|
|
1401
|
-
body: wantsBody ?
|
|
1487
|
+
body: wantsBody ? __sbR2GetRaw(store, String(msg.bucket), String(msg.key)) : undefined,
|
|
1402
1488
|
};
|
|
1403
1489
|
}
|
|
1404
1490
|
if (op === "r2.delete") {
|
|
1405
1491
|
__sbSql(store, "DELETE FROM r2 WHERE bucket = ? AND key = ?", [msg.bucket, msg.key]);
|
|
1492
|
+
__sbR2DeleteRaw(store, String(msg.bucket), String(msg.key));
|
|
1406
1493
|
return { ok: true };
|
|
1407
1494
|
}
|
|
1408
1495
|
if (op === "r2.list") {
|