@sproutboat/toolchain 0.4.10 → 0.4.12

package/CHANGELOG.md

@@ -1,5 +1,21 @@
 # @sproutboat/toolchain
 
+## 0.4.12
+
+### Patch Changes
+
+- ed9640a: Point Porffor patch diagnostics, runtime comments, and historical notes to the
+  shared upstream notes in the Sproutboat platform repository.
+- b6cc9e6: Pin Porffor alpha 9, including alpha 8's compiler and GC fixes and alpha 9's ESM module support. Adapt the native-fetch link and compile-flag patches to Porffor's split C-unit build. Preserve the published 0.4.11 `TypedArray.from` workaround, which alpha 9 still needs. Re-vendor the verified alpha 9 source archive. The uWebSockets commit is unchanged.
+
+## 0.4.11
+
+### Patch Changes
+
+- 6d72af7: Fix `TypedArray.from` with array-like inputs in native binaries and
+  regenerate Porffor's builtin table after the source patch, including when
+  the compiler is unpacked through a symlinked temporary path.
+
 ## 0.4.10
 
 ### Patch Changes
@@ -81,7 +97,7 @@
   prototype and stop once the "next" pointer stops advancing, reading a
   fixed-point chain as "no `.then` found" instead of spinning. Not specific to
   `Date.prototype.toISOString()` — an earlier note wrongly correlated the bug
-  with that one call; see `sproutboat-cli/patches/UPSTREAM.md`.
+  with that one call; see `https://github.com/baronunread/sproutboat/tree/main/patches/upstream`.
 
 ## 0.4.0
 

package/PORFFOR_ALPHA9.md

@@ -0,0 +1,70 @@
+# Porffor alpha 9 upgrade audit
+
+Pin: `de4eb588264885b3a1596f75010e371a2052033f`, tagged `alpha-9` on
+2026-09-24. The vendored codeload archive has SHA-256
+`1a62187a93356b36cb6ac703ce9770c8917e4fa65545fd7c33ee8a180ae5a4c9`.
+
+## Alpha 8 changes
+
+The [alpha 8 release](https://github.com/CanadaHonk/porffor/releases/tag/alpha-8)
+contains 27 commits. Every commit falls into one of these groups:
+
+| Area | Commits | Effect on Sproutboat |
+| --- | --- | --- |
+| Language and compiler correctness | `6063154a`, `8436b386`, `d3376a2d`, `8833624c`, `b92571c4`, `dfab4235`, `65a6bb3f`, `cb26d320`, `6a47f02c`, `acf6dda0`, `c54557ba`, `50855902`, `5e5db5a4`, `6ec00849`, `61d42690` | Better default parameters, spread calls, object values, closure parameters, uninitialized and `let` bindings, optional computed keys, `finally` and exception exits; the remaining edits remove redundant IR and code generation. These improve handler correctness without replacing a local patch. |
+| GC and memory | `83cd4902`, `b5eb0b98`, `9e283dd2`, `f687ac7c`, `08bf855d`, `135d2761`, `ae4002b1`, `a110867b` | Smaller memory commitment and less GC work. `a110867b` supplies a C macro used by the renderer. No Sproutboat GC patch exists to remove. |
+| String built-ins | `f483ba7e` | Better `substring` and `substr`; no corresponding local shim. |
+| Compiler progress | `b77750b0` | Shows compile progress on a TTY; no runtime contract change. |
+| Self-hosted tools and benchmarks | `370f1f75`, `7b12abd2` | Adds `stat.isFile` and a TypeScript compiler benchmark; neither is used by Sproutboat. |
+
+## Alpha 9 and cleanup decision
+
+The [alpha 9 release](https://github.com/CanadaHonk/porffor/releases/tag/alpha-9)
+contains `de4eb588`, adding ESM imports and exports. A direct native-fetch
+fixture importing a sibling module compiled and returned its imported value.
+Porffor now compiles modules as separate cached C units. The local `SB_EXTRA_LINK`
+and `SB_EXTRA_CFLAGS` patches had to move to the new link list and shared
+compile arguments. `UWS_COMMIT` remains `360c276d609d59af56ae6932adb95154ace9f15f`.
+
+No local patch is superseded by the alpha 8 or 9 changes. The patches provide
+Sproutboat-specific native-fetch behavior: `$PORT`, log routing, bytestring
+encoding, raw asset bodies, status and header handling, request size, R2
+transfers, and external link and include flags. The CLI still bundles before
+Porffor to validate the full dependency graph, apply its Node shims, and pass
+one controlled handler export to the runtime wrapper. Native ESM makes a future
+module-based wrapper possible, but removing Bun.build or export normalization
+now would also change capability validation and binding behavior. Porffor's
+old external esbuild requirement is gone in alpha 9, so the CLI can drop its
+esbuild dependency and bundled platform executable.
+
+A module-wrapper prototype compiled a separate handler module. Its normal
+`env.GREETING` reference returned HTTP 500; using `globalThis.env.GREETING`
+returned HTTP 200. The current export normalization keeps the existing
+unqualified `env` contract. The direct ESM import is now part of the CI native
+crypto vector, so a future pin cannot silently lose module support.
+
+Alpha 9 caches compiled C units by the absolute entry path. In one local
+`hello` handler-edit comparison, a stable path compiled in 549 ms versus
+1,505 ms with changing paths. The CLI's `dev` loop now uses a stable entry
+path per process while keeping candidate artifacts isolated. These are local
+single-run timings, not a published performance claim.
+
+The published `@sproutboat/toolchain@0.4.11` includes a `TypedArray.from`
+array-like input patch absent from this checkout's 0.4.10 base. Clean alpha 9
+still returns an empty array for `Uint8Array.from({ length: 4 }, mapFn)`. The
+0.4.11 patch was carried forward, its precompiled builtin table is regenerated,
+and a native crypto vector now exercises that array-like input.
+
+Regression evidence for this pin: toolchain tests (11), CLI tests (92), both
+kitchen-sink harnesses (27 broker checks, 29 standalone checks), and all 12 CLI
+examples passed. The direct ESM fixture returned HTTP 200 with the imported
+value. The package typecheck and lint passed.
+
+## Release order
+
+The registry's latest toolchain is 0.4.11 and still pins alpha 7. This checkout
+has been reconciled with that release's `TypedArray.from` patch; its pending
+changeset targets 0.4.12. Publish the alpha 9 toolchain first. Then update
+`sproutboat-cli`, the main `sproutboat` app, and `sproutboat-site/web` to require
+0.4.12 and refresh their lockfiles. The CLI can then ship without an esbuild
+executable. Until 0.4.12 is published, existing npm consumers still run alpha 7.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@sproutboat/toolchain",
-  "version": "0.4.10",
+  "version": "0.4.12",
   "license": "MIT",
   "type": "module",
   "repository": {

package/src/acquire.test.ts

@@ -28,7 +28,21 @@ async function fixture(): Promise<{ archive: string; sha256: string }> {
       "  }\n" +
       "  return -1;\n" +
       "}\n",
-    "compiler/index.js": "          '-xc', '-', '-c',\n          uSocketsArchive,\n          '-lm'\n",
+    "compiler/index.js": "      ...darwinReleaseCompileArgs,\n        uSocketsArchive,\n        '-lm'\n",
+    "compiler/precompile.js":
+      "const fs = require('node:fs');\n" +
+      "const path = require('node:path');\n" +
+      "fs.writeFileSync(path.join(__dirname, 'builtins_precompiled.js'), fs.readFileSync(path.join(__dirname, 'builtins/typedarray.js')));\n",
+    "compiler/builtins_precompiled.js": "fixture:precompiled\n",
+    "compiler/builtins/typedarray.js":
+      "export const __${name}_from = (arg: any, mapFn: any): ${name} => {\n" +
+      "  const arr: any[] = Porffor.array.new(4);\n" +
+      "  let len: i32 = 0;\n" +
+      "  if (Porffor.type(arg) == Porffor.TYPES.array) {\n" +
+      "    let i: i32 = 0;\n" +
+      "    for (const x of arg) arr[i++] = x;\n" +
+      "    len = i;\n" +
+      "  }\n\n  arr.length = len;\n\n  return new ${name}(arr);\n};\n",
     "compiler/uwebsockets.js":
       "static const size_t REQUEST_BODY_MAX_BYTES = 1024u * 1024u;\n" +
       "int porf_native_fetch_read_value(struct jsval value, const char** out_buf, size_t* out_len, char** out_owned);\n" +
@@ -202,6 +216,35 @@ test("a corrupted warm cache is replaced from the verified archive", async () =>
   expect(await readFile(join(root, "compiler/render.js"), "utf8")).toContain('getenv("PORT")');
 });
 
+test("a cache built by an older toolchain version is rebuilt, not silently reused", async () => {
+  const { archive, sha256 } = await fixture();
+  const cacheRoot = await mkdtemp(join(tmpdir(), "sb-porffor-stale-toolchain-"));
+  temporary.push(cacheRoot);
+  let requests = 0;
+  const options = {
+    cacheRoot,
+    url: "fixture",
+    expectedSha256: sha256,
+    fetcher: async () => {
+      requests += 1;
+      return new Response(Bun.file(archive));
+    },
+  };
+  const root = await ensurePorffor(options);
+  const manifestPath = join(root, ".sproutboat-complete");
+  const manifest = JSON.parse(await readFile(manifestPath, "utf8"));
+  expect(manifest.toolchainVersion).toBeTruthy();
+  // Simulate a cache left behind by an older @sproutboat/toolchain publish
+  // (baronunread/sproutboat#205): a patch.ts fix landing in a new version used
+  // to sit unused because the cache key only tracked the upstream commit.
+  await rm(manifestPath);
+  await writeFile(manifestPath, JSON.stringify({ ...manifest, toolchainVersion: "0.0.0-stale" }));
+  await ensurePorffor(options);
+  expect(requests).toBe(2);
+  const rebuilt = JSON.parse(await readFile(manifestPath, "utf8"));
+  expect(rebuilt.toolchainVersion).toBe(manifest.toolchainVersion);
+});
+
 test("an interrupted stale lock is recovered", async () => {
   const { archive, sha256 } = await fixture();
   const cacheRoot = await mkdtemp(join(tmpdir(), "sb-porffor-interrupted-"));

package/src/acquire.ts

@@ -5,9 +5,15 @@ import { homedir } from "node:os";
 import { resolve } from "node:path";
 // @ts-expect-error Bun's file loader supplies a path; see the pinned-commit
 // check below for why this is safe to trust unconditionally.
-import vendoredPorfforArchive from "../vendor/porffor-8f01541.tar.gz" with { type: "file" };
+import vendoredPorfforArchive from "../vendor/porffor-de4eb58.tar.gz" with { type: "file" };
 import { ensurePorfforPatched } from "./patch";
 import { PORFFOR_ARCHIVE_SHA256, PORFFOR_ARCHIVE_URL, PORFFOR_COMMIT_FULL } from "./pin";
+// Folded into the cache manifest below so a toolchain release that changes
+// patch.ts's output invalidates every machine's cache automatically
+// (baronunread/sproutboat#205 — a patch fix published as a new version used
+// to sit unused behind a still-"complete" cache keyed only on the upstream
+// Porffor commit).
+import { version as TOOLCHAIN_VERSION } from "../package.json" with { type: "json" };
 
 export class PorfforToolchainError extends Error {
   constructor(
@@ -31,6 +37,9 @@ const required = [
   "compiler/render.js",
   "compiler/index.js",
   "compiler/uwebsockets.js",
+  "compiler/precompile.js",
+  "compiler/builtins/typedarray.js",
+  "compiler/builtins_precompiled.js",
 ];
 async function digest(path: string): Promise<string> {
   return createHash("sha256")
@@ -45,9 +54,14 @@ async function complete(dir: string, expectedArchive = PORFFOR_ARCHIVE_SHA256):
     const manifest = JSON.parse(await readFile(resolve(dir, ".sproutboat-complete"), "utf8")) as {
       commit?: string;
       archiveSha256?: string;
+      toolchainVersion?: string;
       files?: Record<string, string>;
     };
-    if (manifest.commit !== PORFFOR_COMMIT_FULL || manifest.archiveSha256 !== expectedArchive)
+    if (
+      manifest.commit !== PORFFOR_COMMIT_FULL ||
+      manifest.archiveSha256 !== expectedArchive ||
+      manifest.toolchainVersion !== TOOLCHAIN_VERSION
+    )
       return false;
     for (const file of required)
       if (!manifest.files?.[file] || (await digest(resolve(dir, file))) !== manifest.files[file])
@@ -185,7 +199,7 @@ export async function ensurePorffor(options: AcquireOptions = {}): Promise<strin
     );
     await writeFile(
       resolve(stage, ".sproutboat-complete"),
-      JSON.stringify({ commit: PORFFOR_COMMIT_FULL, archiveSha256: actual, files }),
+      JSON.stringify({ commit: PORFFOR_COMMIT_FULL, archiveSha256: actual, toolchainVersion: TOOLCHAIN_VERSION, files }),
       { mode: 0o444 },
     );
     await rename(stage, dir);

package/src/patch.ts

@@ -1,15 +1,18 @@
 /**
  * Idempotent, marker-guarded in-place edits to Porffor's generated C
- * (`compiler/render.js`, `compiler/index.js`, `compiler/uwebsockets.js`). Run
+ * (`compiler/render.js`, `compiler/index.js`, `compiler/uwebsockets.js`,
+ * `compiler/builtins/typedarray.js`). Run
  * from the build path, not a `postinstall` hook: package managers block
  * dependency lifecycle scripts by default, so a published `postinstall` would
  * silently not run.
  *
  * Each edit is independent and re-applied on every build; a file patched by an
  * older version of this module still receives the newer edits. All of them are
- * tracked in patches/UPSTREAM.md and go away as Porffor closes the gaps.
+ * tracked in the platform's upstream notes:
+ * https://github.com/baronunread/sproutboat/tree/main/patches/upstream
+ * They go away as Porffor closes the gaps.
  */
-import { readFile, writeFile } from "node:fs/promises";
+import { readFile, realpath, writeFile } from "node:fs/promises";
 import { resolve } from "node:path";
 
 // --- compiler/render.js: the native-fetch server, rendered as C text ---
@@ -151,22 +154,45 @@ const RENDER_REPLACEMENTS = [
  * a musl (deploy) build overrides it outright. This splices one spread of
  * `SB_EXTRA_LINK` before `-lm`, inert unless the variable is set.
  */
-const LINK_ANCHOR = "          uSocketsArchive,\n          '-lm'\n";
+const LINK_ANCHOR = "        uSocketsArchive,\n        '-lm'\n";
 const LINK_INJECT =
-  "          ...(process.env.SB_EXTRA_LINK ? process.env.SB_EXTRA_LINK.split(' ').filter(Boolean) : []),\n";
+  "        ...(process.env.SB_EXTRA_LINK ? process.env.SB_EXTRA_LINK.split(' ').filter(Boolean) : []),\n";
 const LINK_MARKER = "SB_EXTRA_LINK";
 
 /**
  * #15 — and the same for the compile step, so the prelude's inline C can
- * `#include <bearssl.h>`. The link patch alone is not enough: Porffor compiles
- * the generated C from stdin with a fixed argument list, so there is otherwise
- * no way to add an include path.
+ * `#include <bearssl.h>`. The link patch alone is not enough: Porffor's
+ * module builds compile several C units with a fixed argument list, so there
+ * is otherwise no way to add an include path. Add the flag before the cache
+ * stamp is computed so a change invalidates compiled units.
  */
-const CFLAGS_ANCHOR = "          '-xc', '-', '-c',\n";
+const CFLAGS_ANCHOR = "      ...darwinReleaseCompileArgs,\n";
 const CFLAGS_INJECT =
-  "          ...(process.env.SB_EXTRA_CFLAGS ? process.env.SB_EXTRA_CFLAGS.split(' ').filter(Boolean) : []),\n";
+  "      ...(process.env.SB_EXTRA_CFLAGS ? process.env.SB_EXTRA_CFLAGS.split(' ').filter(Boolean) : []),\n";
 const CFLAGS_MARKER = "SB_EXTRA_CFLAGS";
 
+// Porffor's TypedArray.from only handles iterables. An array-like input such
+// as { length: 16 } silently becomes an empty typed array, including HMAC
+// keys made with Uint8Array.from({ length: 16 }, mapFn). The compiler uses a
+// precompiled builtin table, so the edited source must be precompiled again.
+const TYPED_ARRAY_FROM_ANCHOR =
+  "    len = i;\n  }\n\n  arr.length = len;\n\n  return new ${name}(arr);\n};";
+const TYPED_ARRAY_FROM_INJECT =
+  "    len = i;\n" +
+  "  } else {\n" +
+  "    // sproutboat: TypedArray.from accepts array-like objects.\n" +
+  "    let count = ecma262.ToIntegerOrInfinity(Porffor.object.get(arg, 'length'));\n" +
+  "    if (count < 0) count = 0;\n" +
+  "    if (count > 2147483643) throw new RangeError('Invalid TypedArray length (over maximum supported length)');\n" +
+  "    if (Porffor.type(mapFn) != Porffor.TYPES.undefined && Porffor.type(mapFn) != Porffor.TYPES.function)\n" +
+  "      throw new TypeError('Called TypedArray.from with a non-function mapFn');\n" +
+  "    for (let i: i32 = 0; i < count; i++) {\n" +
+  "      arr[i] = Porffor.type(mapFn) == Porffor.TYPES.undefined ? arg[i] : mapFn(arg[i], i);\n" +
+  "    }\n" +
+  "    len = count;\n" +
+  "  }\n\n  arr.length = len;\n\n  return new ${name}(arr);\n};";
+const TYPED_ARRAY_FROM_MARKER = "sproutboat: TypedArray.from accepts array-like objects";
+
 /**
  * #56 — make the inbound request-body limit configurable.
  *
@@ -794,7 +820,7 @@ export async function patchUwebsockets(root: string): Promise<void> {
     if (!src.includes(anchor)) {
       throw new Error(
         `could not patch Porffor's ${what}: anchor not found in ${file}. ` +
-          "Porffor's uWebSockets shim changed — check patches/UPSTREAM.md.",
+          "Porffor's uWebSockets shim changed. See https://github.com/baronunread/sproutboat/tree/main/patches/upstream.",
       );
     }
     src = src.replace(anchor, inject);
@@ -816,11 +842,11 @@ async function patchCompilerArgs(root: string): Promise<void> {
     if (at === -1) {
       throw new Error(
         `could not patch Porffor for ${what}: anchor not found in ${file}. ` +
-          "Porffor's native-fetch build changed — check patches/UPSTREAM.md.",
+          "Porffor's native-fetch build changed. See https://github.com/baronunread/sproutboat/tree/main/patches/upstream.",
       );
     }
-    // After the anchor for cflags (the args follow it), before it for the link
-    // line (the object list ends with it).
+    // Add flags to compileOnlyArgs so every generated C unit sees them and
+    // Porffor's module-build cache includes them in its compiler command.
     src =
       marker === CFLAGS_MARKER
         ? src.slice(0, at + anchor.length) + inject + src.slice(at + anchor.length)
@@ -830,6 +856,34 @@ async function patchCompilerArgs(root: string): Promise<void> {
   if (changed) await writeFile(file, src);
 }
 
+export async function patchTypedArrayFrom(root: string): Promise<void> {
+  const file = resolve(root, "compiler/builtins/typedarray.js");
+  const src = await readFile(file, "utf8");
+  if (!src.includes(TYPED_ARRAY_FROM_MARKER) && !src.includes(TYPED_ARRAY_FROM_ANCHOR)) {
+    throw new Error(`could not patch Porffor's TypedArray.from: anchor not found in ${file}`);
+  }
+  if (!src.includes(TYPED_ARRAY_FROM_MARKER))
+    await writeFile(file, src.replace(TYPED_ARRAY_FROM_ANCHOR, TYPED_ARRAY_FROM_INJECT));
+  const table = resolve(root, "compiler/builtins_precompiled.js");
+  const before = await readFile(table);
+  // Porffor only writes the table when import.meta.url matches argv[1].
+  // macOS's /var -> /private/var symlink makes a noncanonical argv[1] skip it.
+  const precompile = await realpath(resolve(root, "compiler/precompile.js"));
+  const child = Bun.spawn(["node", precompile], {
+    cwd: root,
+    stdout: "pipe",
+    stderr: "pipe",
+  });
+  const [code, stdout, stderr] = await Promise.all([
+    child.exited,
+    new Response(child.stdout).text(),
+    new Response(child.stderr).text(),
+  ]);
+  if (code !== 0) throw new Error(`could not precompile Porffor's TypedArray.from: ${stderr || stdout}`);
+  if (!src.includes(TYPED_ARRAY_FROM_MARKER) && before.equals(await readFile(table)))
+    throw new Error(`Porffor precompile did not update ${table}`);
+}
+
 /** The native-fetch server source: $PORT support + the #165 console sink. Exported for tests. */
 export async function patchRenderJs(root: string): Promise<void> {
   const file = resolve(root, "compiler/render.js");
@@ -841,7 +895,7 @@ export async function patchRenderJs(root: string): Promise<void> {
     if (at === -1) {
       throw new Error(
         `could not patch Porffor for ${what}: anchor not found in ${file}. ` +
-          "Porffor's native-fetch renderer changed — check patches/UPSTREAM.md.",
+          "Porffor's native-fetch renderer changed. See https://github.com/baronunread/sproutboat/tree/main/patches/upstream.",
       );
     }
     src = src.slice(0, at + anchor.length) + inject + src.slice(at + anchor.length);
@@ -855,7 +909,7 @@ export async function patchRenderJs(root: string): Promise<void> {
     if (!src.includes(anchor)) {
       throw new Error(
         `could not patch Porffor for ${what}: anchor not found in ${file}. ` +
-          "Porffor's native-fetch renderer changed — check patches/UPSTREAM.md.",
+          "Porffor's native-fetch renderer changed. See https://github.com/baronunread/sproutboat/tree/main/patches/upstream.",
       );
     }
     src = src.replace(anchor, inject);
@@ -876,5 +930,6 @@ export async function ensurePorfforPatched(root: string): Promise<void> {
   await patchCompilerArgs(root);
   await patchUwebsockets(root);
   await patchRenderJs(root);
+  await patchTypedArrayFrom(root);
   done.add(root);
 }

package/src/pin.ts

@@ -11,14 +11,14 @@
  * uWebSockets archive in `sproutboat-cli/vendor/` — then run both repos'
  * test + conformance suites. See sproutboat-cli/MIGRATION.md.
  */
-export const PORFFOR_CHANNEL = "alpha-7";
-export const PORFFOR_COMMIT_FULL = "8f01541498d6d61c0cbbf8a71be152330888be7e";
+export const PORFFOR_CHANNEL = "alpha-9";
+export const PORFFOR_COMMIT_FULL = "de4eb588264885b3a1596f75010e371a2052033f";
 export const PORFFOR_COMMIT = PORFFOR_COMMIT_FULL.slice(0, 7);
 export const PORFFOR_ARCHIVE_SHA256 =
-  "fae9cfb00c0a21e20e3aa79a9a8093bd24820062039985c6bde2b23e3e8bdb8c";
+  "1a62187a93356b36cb6ac703ce9770c8917e4fa65545fd7c33ee8a180ae5a4c9";
 export const PORFFOR_ARCHIVE_URL = `https://codeload.github.com/CanadaHonk/porffor/tar.gz/${PORFFOR_COMMIT_FULL}`;
 
-/** A compact identity string for a manifest / report (`alpha-7 (8f01541)`). */
+/** A compact identity string for a manifest / report (`alpha-9 (de4eb58)`). */
 export function porfforVersion(): string {
   return process.env.PORFFOR_VERSION || `${PORFFOR_CHANNEL} (${PORFFOR_COMMIT})`;
 }