castrum 0.9.3 → 0.9.6

package/docs/API.md

@@ -38,9 +38,10 @@ runtimes, and binary ops (`gzip*`, `brotli*`, `aead*`, `pbkdf2Sha256`,
 | `proven`, `PROVEN_SELECTION`, `provenImpl`, `provenStatus`, `isProven`, `provenSurface`, `provenSummary` | The full `rust.*` surface + the baked (pure-data) proven-selection registry of benchmark winners (`native` / `js` / `bun`). |
 | `opImpl`, `isNativeOp`, `opDecision` | Native-vs-JS selection hints (benchmark-driven). |
 | `loader`, `createLoader` | Higher-order loader over the batch surface. |
-| `encoder`, `decoder` | Codec-backed UTF-8 helpers (Bun: `Bun.ArrayBufferSink` / `CString`; Node: `TextEncoder`/`TextDecoder`). `encode`/`decode` accept the `Uint8Array | string` union and normalize. |
+| `encoder`, `decoder` | Codec-backed UTF-8 helpers (shared native `TextEncoder` / Bun: `CString`; Node: `TextEncoder`/`TextDecoder`). `encode`/`decode` accept the `Uint8Array | string` union and normalize. |
 | `uuidv7()` | UUIDv7 — `Bun.randomUUIDv7` on Bun, `crypto.randomUUID` on Node. |
 | `AdaptiveEstimate`, `AdaptiveEstimateOptions` | Bounded EWMA adaptive-estimate utility. |
+| `createTaskRuntime`, `TASK_OP`, `TASK_STATUS_OK` / `_ERROR` / `_CANCELLED`, `TaskRuntime`, `TaskStats`, `TaskRunOptions`, `Pbkdf2RunOptions`, `encodeGzipDecompressArgs`, `encodeGzipCompressHeader`, `encodeBrotliDecompressArgs`, `encodeArgon2VerifyArgs`, `encodePbkdf2Args`, `gzipCompressUpperBound`, `brotliCapacityGuess` | Off-thread task runtime ("castrum Tasks"): `await` native work on a Rust pool without blocking the event loop — `gzipDecompress`, `brotliDecompress`, `gzipCompress`, `pbkdf2Sha256`, `argon2Verify`, `AbortSignal`, `stats()`, zero-copy results *and* inputs. See [`RND-CONCURRENCY.md`](./RND-CONCURRENCY.md) §1b. |
 | `createMetrics`, `DEFAULT_BUCKETS` | Zero-dependency metrics registry (counters/gauges/histograms + Prometheus text). |
 | `packBatch`, `packPairs`, `readPairsPacked`, `readHttpPacked`, `pairsToObject`, `parseQueryString`, `parseCookieHeader`, `parseFormBody`, `unpackBitset`, `unpackByteResults`, `unpackI64ArrayAsBigInt`, `unpackU32Array` | Packed-wire + high-level parsing utilities. |
 | Ingress: `createIngress`, `createIngressSync`, `createIngressFast`, `createIngressHandler`, `createIngressServer`, `createIngressServerNode`, `createIngressRouter`, route factories, metrics/health/trace helpers | See [`INGRESS.md`](./INGRESS.md) and [`INGRESS-ROUTER.md`](./INGRESS-ROUTER.md). |

package/docs/ARCHITECTURE.md

@@ -31,8 +31,9 @@ The bridge between TypeScript and Rust has **two transports**, both loading the
 cdylib (built with napi-rs):
 
 - **`bun:ffi` (PRIMARY on Bun)** — `src/native/ffi.ts` `dlopen`s the addon's
-  `extern "C"` exports (`rust/ffi/`, 85 `castrum_*` symbols — 77 direct + 4
-  `validator_c_abi!` + 4 `compress_to_out!`, parity guarded by
+  `extern "C"` exports (`rust/ffi/`, 119 `castrum_*` symbols — 107 direct + 4
+  `validator_c_abi!` + 4 `validator_bytes_c_abi!` + 4 `compress_to_out!`,
+  parity guarded by
   `test/unit/native/ffi-symbol-parity.test.ts`) and Bun JIT-calls them
   directly (~10-20ns crossing vs ~100-350ns N-API). A bind-time self-test guards
   correctness: on ANY failure the layer is disabled and calls fall back to napi.

package/docs/CASE_STUDY.md

@@ -10,7 +10,9 @@ benchmarks actually showed, and the lessons that generalise to any
 All figures below come from the reports in `bench/results/` (generated on a
 release addon, baseline CPU, Bun 1.4.0). They're **representative**, not
 advertised as universal — re-run `bun run check` / `bun run bench:http` on your
-own hardware before trusting them for your numbers.
+own hardware before trusting them for your numbers. (The local toolchain is now
+Bun 1.4.2; the figures below still carry their 1.4.0 measurement date, and the
+newer `bench:task` / `bench:ffi:margin` rows name 1.4.2.)
 
 ---
 
@@ -42,8 +44,8 @@ as a pre-baked pipeline, and still be usable from both runtimes.
 ### 2.1 One cdylib, two transports
 
 The addon is built with napi-rs (so Node can load it), but it *also* exports
-`extern "C"` symbols (`rust/ffi/`, **85 `castrum_*` symbols** — 77 direct + 4
-`validator_c_abi!` + 4 `compress_to_out!`) so Bun can
+`extern "C"` symbols (`rust/ffi/`, **119 `castrum_*` symbols** — 107 direct + 4
+`validator_c_abi!` + 4 `validator_bytes_c_abi!` + 4 `compress_to_out!`) so Bun can
 `dlopen` it directly:
 
 - **`bun:ffi` (primary on Bun)** — Bun JIT-calls the C-ABI exports at

package/docs/ENVIRONMENT.md

@@ -33,6 +33,13 @@ for false; anything else is warned about and falls back to the default.
 | `CASTRUM_MAX_RAYON_THREADS` | int | `cores` | Hard cap on rayon threads (native). |
 | `CASTRUM_PIN_CORES` | (presence) | — | When set (Linux), pin rayon worker threads to distinct cores. |
 
+## Off-thread task pool (`rust/task/`)
+
+| Variable | Type | Default | Description |
+|----------|------|---------|-------------|
+| `CASTRUM_TASK_THREADS` | int | `max(1, cores-1)` | Worker threads for the off-thread task runtime (castrum Tasks). Read when the pool first starts — the pool is process-wide and cannot be resized afterwards (`tasks.stats().threads` reports the live count). |
+| `CASTRUM_TASK_PIN_CORES` | (presence) | — | When set (Linux), pin each task worker to its own core (core 0 is skipped so the JS event loop keeps it). `CASTRUM_PIN_CORES` enables it too. Off by default: pinning competes with the JS/rayon threads for the same box. |
+
 ## Native transport selection (`src/native/ffi.ts`)
 
 `bun:ffi` is the PRIMARY transport under Bun; NAPI is the fallback (Node,
@@ -66,11 +73,27 @@ whether the ffi transport is live.
 | `CASTRUM_FFI_LOAD_MS` | int | `400` | Per-window duration (ms) of the `ffi-all` load-run (`bench/ffi/ffi-all.ts`). |
 | `CASTRUM_FFI_LOAD_WINDOWS` | int | `4` | Number of load windows in `ffi-all` (`bench/ffi/ffi-all.ts`). |
 
-## Publishing (`scripts/prepublish.mjs`, `publish-manual.mjs`)
+## Publishing (`scripts/prepublish.mjs`, canonical `scripts/release.ts`)
+
+| Variable | Type | Default | Description |
+|----------|------|---------|-------------|
+| `CASTRUM_PUBLISH_ALLOW_PARTIAL` | bool | — | When `1`, `prepublishOnly` ships a tarball containing only the locally-built platforms instead of failing on the full `napi.targets` set. Set automatically by `bun run release:manual` (the local-publish path of the canonical release flow); **never** set it for a normal `npm publish`.
+
+## Postinstall fallback build (`scripts/postinstall.mjs`)
+
+The package ships prebuilt addons for every `napi.targets` platform (built by
+the CI "build" job on push/PR and staged into the tarball on a `v*` tag push).
+`postinstall` is a fast no-op when a matching prebuilt is present. When none
+ships for the host (an exotic platform / libc, or a partial tarball), it
+compiles the addon **from source** in the consumer's `node_modules` via
+`cargo build --release` (needs a Rust toolchain + crates.io access).
 
 | Variable | Type | Default | Description |
 |----------|------|---------|-------------|
-| `CASTRUM_PUBLISH_ALLOW_PARTIAL` | bool | — | When `1`, `prepublishOnly` ships a tarball containing only the locally-built platforms instead of failing on the full `napi.targets` set. Set automatically by `bun run publish:manual`; **never** set it for a normal `npm publish`.
+| `CASTRUM_SKIP_BUILD` | (presence) | — | Skip the source-build fallback entirely (warn and continue). Use when a Rust toolchain is intentionally unavailable; the runtime loader will then fail with a clear "build it" error if the addon is needed. |
+| `CASTRUM_REQUIRE_BUILD` | (presence) | — | Turn every postinstall "warn and skip" (missing toolchain, build failure, missing cdylib) into a hard install failure — for environments that must guarantee a loadable addon. |
+| `npm_config_build_from_source` | bool | — | Set by `npm install --build-from-source`; forces a rebuild even when a prebuilt artifact is present. |
+| `CI` | bool | — | GitHub Actions (and other CI) set this; the postinstall skips the fallback build because CI builds the addon explicitly (`bun run build`). |
 
 ## Security guidance
 

package/docs/FFI_BUN_GUIDE.md

@@ -7,10 +7,12 @@
 > code is **optimized for Bun** out of the box — correct on the first `dlopen`,
 > fast on the first call, and safe under burst load.
 >
-> **Version anchor**: behavior claims are verified against the **Rust-based Bun
-> 1.3.14** source (`/home/adeel/poc/bun`). `bun:ffi` is experimental upstream;
-> the bind-time self-test + napi fallback are what make it safe to use, and
-> they are mandatory — not optional.
+> **Version anchor**: behavior claims were distilled from the **Rust-based Bun
+> 1.3.14** source (`/home/adeel/poc/bun`) and RE-VERIFIED against the installed
+> **Bun 1.4.2** runtime (the local dev toolchain, matching the CI matrix). Where
+> a claim is runtime-specific the number is stated inline. `bun:ffi` is
+> experimental upstream; the bind-time self-test + napi fallback are what make
+> it safe to use, and they are mandatory — not optional.
 
 ---
 
@@ -99,14 +101,23 @@ the matching shape and nothing else:
    (no copy). **Benchmark-gated** (`bench:margin` `cstringArg` scenario):
    ~40–52 ns/call vs ~209–233 ns for `encodeUtf8` + `(ptr,len)` — a ~76–82% win,
    including non-ASCII inputs. It is a LOSS for BYTE inputs (decode + re-encode)
-   — keep `(ptr,len)` there. **Verified `bun:ffi` facts**: JS `null` → NULL
-   pointer, and JS `''` (empty string) → NULL too — a present-but-empty value is
-   indistinguishable from absent through a `cstring` arg (so SSE's `Option`
-   event/id semantics stay on `(ptr,len)`). Adopted by: the validators
-   (`castrum_validate_*`), `castrum_ws_accept_key`, `castrum_mime_from_extension`,
+   — keep `(ptr,len)` there (the validators ship BOTH shapes:
+   `castrum_validate_*` for strings, `castrum_validate_*_bytes` for bytes).
+   **Verified `bun:ffi` facts**: JS `null` → NULL pointer, and JS `''` (empty
+   string) → NULL too — a present-but-empty value is indistinguishable from
+   absent through a `cstring` arg UNLESS a presence flag crosses alongside it.
+   Adopted by: the validators (`castrum_validate_*`),
+   `castrum_ws_accept_key`, `castrum_mime_from_extension`,
    `castrum_password_verify_bcrypt` (phc), `castrum_rate_limiter_check` (key),
    and `castrum_ingress_handle_components` (url/ip).
 
+   > **CASE STUDY — a conversion we REVERTED (2026-08-23)**: SSE's event/id
+   > were converted to `cstring` ARGs (+ presence flags) and shipped briefly,
+   > then reverted. The conversion itself was sound; what failed was the dev
+   > loop — see §14. Every "bun:ffi is broken" signal during that episode
+   > traced back to a mismatched addon↔JS state, never to the engine. Do not
+   > cite this as an engine bug.
+
 Opaque instance handles are passed as a **bare `usize`** (`inner: usize`), and
 must be null-checked at the top of the function (return the `0` sentinel) —
 castrum's `castrum_ingress_handle_packed` / `castrum_*_matcher_matches` etc.
@@ -182,6 +193,66 @@ returns.
 **Rule**: use per-thread reused buffers for cstring returns; never return a
 pointer into JS-owned memory; treat `null` as the failure sentinel.
 
+### 6.1 `cstring` ARGS: the engine transcodes — and truncates at NUL
+
+The same doc section defines the **argument** side, which is a different trade
+from returns:
+
+> "When used in `args`, `FFIType.cstring` accepts everything `ptr` does and
+> additionally accepts a JavaScript string directly. The engine transcodes the
+> string to a **null-terminated** UTF-8 buffer that lives for the duration of
+> the call, so you don't need to encode it into a `Buffer` yourself."
+
+This is a real win — it removes a `TextEncoder` round trip per call, and it is
+why castrum binds the string surfaces (`validate*`, `metrics_*`,
+`regex_escape_str`, `ws_accept_key`, `mime_from_extension`, `session_*`, the JWT
+/route id strings) as `cstring` ARGs, with byte-input callers crossing as
+`(buffer, buffer_length)` instead. But "null-terminated" is load-bearing:
+
+**An embedded `U+0000` silently truncates the value native-side.** Where the
+callee answers with a VERDICT, that is a correctness hole, and two shipped bugs
+came from it (found 2026-09-11, both now fixed and pinned by
+`test/unit/native/cstring-nul.test.ts`):
+
+| call | was | now |
+|---|---|---|
+| `rust.validateEmail(bytes('a@b.com\0<script>'))` | `true` | `false` |
+| `rust.regexEscape('abc\0def')` | `'abc'` | `'abc\0def'` |
+
+The validator case was the worst shape: bytes were decoded to a JS string and
+handed to the `cstring` ARG, so a NUL **byte** made a non-address validate. The
+fix routes byte-input APIs through the `*_bytes` symbols — which is also
+**faster**, because the bytes then cross as an atomic `(buffer, buffer_length)`
+pair with no transcode. Measured through the PUBLIC API (`rust.validateEmail`,
+400k calls, Bun 1.4.2): email 236 → 110 ns, uuid 153 → 50 ns, ipv4 118 → 37 ns.
+Correctness and speed pointed the same way; the old comment claiming the decode
+route was faster predated the byte siblings.
+
+Two further byte-input sites were closed the same way, with one caveat worth
+recording. `mimeFromExtension` no longer resolves a NUL-suffixed extension as its
+prefix (`'.js\0evil'` resolved as `.js` — a Content-Type chosen from truncated
+input; it now takes the unknown sentinel). `wsAcceptKey` no longer computes the
+accept value for a prefix of the key. Both KEEP the `cstring` fast path and divert
+only the NUL case, because measuring the real call path reversed the raw-binding
+result: `rust.wsAcceptKey` is 367 ns via `cstring` versus 381 ns via the byte
+sibling (the extra `subarray` + decode outweighs the transcode saved), whereas
+measuring the bare binding suggested the opposite. **Measure through the function
+users call, not the binding** — the guard itself costs ~4 ns, so guarding is
+cheap and routing everything through bytes is not always a win.
+
+**Rule**: a `cstring` ARG is only correct when truncation cannot change the
+answer — i.e. the string is developer-supplied config or NUL-free by
+construction (a MIME extension, a base64 key, a header name). Anything
+reachable from user input must either
+
+1. cross as bytes (`buffer` + `buffer_length`) when the API is byte-shaped, or
+2. be guarded with `hasNul` (`src/native/ffi/build/util.ts`): short-circuit the
+   verdict (`false`), throw at declare time (metric names / label keys), or
+   route to the `(ptr, usize)` sibling for a byte-exact result (`regexEscape`,
+   batch hex validation).
+
+Never silently return a value derived from a truncated prefix.
+
 ---
 
 ## 7. `dlopen` lifetime — never `close()` while bound symbols live
@@ -286,6 +357,75 @@ napi-style unwind guard exists for raw `extern "C"`).
    the scalar builder, and `ffi-symbol-parity.test.ts`; a Rust unit test +
    self-test; cross-check FFI vs napi byte-for-byte in `ffi.test.ts`.
 
+### 11.1 Conformance with Bun's published FFI guidance (audit)
+
+The question "are we using Bun's FFI the way Bun recommends?" is answerable from
+this table rather than from memory. Verified 2026-09-11 against
+`https://bun.sh/docs/api/ffi` and the live binding on Bun 1.4.2
+(`ffiBufferMode() === 'buffer-pair'`).
+
+| Bun's guidance (docs) | castrum | Where |
+|---|---|---|
+| `buffer` + `buffer_length`: pass the **same view twice** for atomic ptr+len | **Followed and LIVE** — `abi()` rewrites every `(ptr,usize)` pair; `probeBufferLength()` verifies support at bind time with a `(ptr,usize)` fallback | §4, `src/native/ffi.ts` |
+| `cstring` **returns** are cloned at call time (NULL → null) | Followed — per-thread `CSTR_BUF`, clone is synchronous | §6, `rust/ffi/` |
+| `cstring` **args** accept a JS string; the engine transcodes it | Followed for the string surface; **byte-input APIs use `(buffer,buffer_length)` instead — measured 2.2–3.2× faster AND immune to NUL truncation** | §6.1 |
+| `u64_fast`/`i64_fast` for byte counts (avoid BigInt boxing) | Followed — `U64_FAST` on every size/count return | §5 |
+| `JSCallback`: pass `.ptr`, not the object ("slight performance boost") | Followed — `taskSetDoorbell(Number(cb.ptr))` | §10 |
+| `read.u8/u32/...` avoids a `DataView`/`ArrayBuffer` for pointer reads | Applied where it matters (hot paths use pooled JS scratch, not pointer reads); bind-time blob reads keep `DataView` | `instances.ts` |
+| `toArrayBuffer` with a deallocator for C-owned memory | **Deliberately NOT used** — Bun 1.4.2's deallocator path segfaults (`arity 1`, crash at `0x0`) while the docs describe 4/5-arg forms. Zero-copy instead keeps ownership in JS | `docs/RND-CONCURRENCY.md` §7a |
+| `dlopen` is never `close()`d while symbols live | Followed — bound once, held forever | §7 |
+| Async functions are not supported in `JSCallback` | Followed — the doorbell is a `void()` trampoline | §10 |
+
+Reading the table honestly: the `buffer`/`buffer_length` and `cstring` return
+practices were already conformant. The real gaps were on the **argument** side
+(§6.1) — and those turned out to be correctness bugs, not just style.
+
+### 11.2 Audit log — 2026-09-11 (Bun 1.4.2)
+
+The audit above was run because the practices were *questioned*, not because a
+failure was observed. Recording what it found, including the parts that were
+already right, so nobody has to re-derive it:
+
+**Already conformant (verified, do not "fix"):**
+
+- `buffer`/`buffer_length` is live — `ffiBufferMode() === 'buffer-pair'`, and
+  `abi()` rewrites every `(ptr, usize)` pair.
+- `cstring` for returns and for the string surface; `u64_fast` for byte counts;
+  `JSCallback.prototype.ptr`; `dlopen` never closed. All per the docs.
+
+**Found and fixed (all correctness, two of them security-grade):**
+
+| finding | evidence | fix |
+|---|---|---|
+| Validator verdicts could be fooled | `validateEmail(bytes('a@b.com\0<script>'))` → `true` | byte sibling (`buffer`/`buffer_length`), 2.2–3.2× faster too |
+| `regexEscape` truncated output | `'abc\0def'` → `'abc'` | route the NUL case to the `(ptr,len)` sibling |
+| Batch hex validation could pass a prefix | joined ids truncated at a NUL | same |
+| Query/cookie gates could validate a prefix | `cstring` ARG + user-supplied wire text | short-circuit `false` on NUL |
+| Metric label values silently merged series | truncation collapsed two label values into one | throw at declare time / reject at record |
+| `mimeFromExtension` resolved a NUL-suffixed extension as its prefix | `.js\0evil` → `text/javascript` | unknown sentinel |
+| `wsAcceptKey` hashed a prefix of the key | suffix silently ignored | byte path for the NUL case |
+
+**Methodological corrections (both were nearly written into the docs as fact):**
+
+1. **Raw-binding probes mislead.** `getBunFFI()!.op(...)` omits the memoized
+   `ffi()` hop and the caller's own decode/subarray. For `wsAcceptKey` it made the
+   byte path look 11% faster when through `rust.wsAcceptKey` it is 4% *slower*
+   (381 vs 367 ns) — so that call keeps `cstring` and diverts only the NUL case.
+2. **Closure-wrapped micro-benchmarks inflate small costs.** A guard that
+   measured 90 ns inside a benchmark closure is 3.6 ns isolated (`indexOf`). That
+   ~25× error would have justified adding a C-ABI symbol that earns nothing.
+
+**Also closed:** the sweep found no other non-conformance. `toArrayBuffer`
+appears once in `src/` (a comment explaining why it is avoided), the only
+`bun:ffi` import outside the native layer is a type import, `read.*` has no
+hot-path use (our pointer reads happen once at bind), and `rust.text.*` was
+already FFI-first with `cstring` (zero encode).
+
+**Follow-ups, deliberately not done:** `mimeFromExtension` still decodes for the
+`cstring` ARG because no byte sibling exists (a symbol that would earn ~40 ns on
+a non-hot path); `@types/bun` remains 1.3.14 (see §13 — the cache has no newer
+version, so the casts stay).
+
 ---
 
 ## 12. Bun's own Rust hygiene (borrowed rules)
@@ -309,11 +449,23 @@ that transfer directly to castrum's cdylib:
 
 ## 13. Version anchors & verification
 
-The behavioral claims above are verified against the Rust-based Bun source
+The behavioral claims above were distilled from the Rust-based Bun source
 checkout at `/home/adeel/poc/bun` (LATEST file: 1.3.14; `package.json` tracks
-the 1.4 development line). The decision matrix and ADR-0003 measure against
-the installed **Bun 1.4.0** runtime — both anchors describe the same
-Rust-based runtime family. Castrum's own measurements live in `bench/`.
+the 1.4 development line) and are **re-verified against the installed Bun 1.4.2
+runtime** that the code actually runs on (and that the CI matrix pins). The
+decision matrix and ADR-0003 measured on the installed **Bun 1.4.0** runtime —
+all anchors describe the same Rust-based runtime family. Castrum's own
+measurements live in `bench/`.
+
+**Known gap (2026-09-11): the type package lags the runtime.** `package.json`
+pins `@types/bun@^1.3.14` / `bun-types@1.3.14` while the runtime is 1.4.2, and the
+local package cache holds only 1.3.14 (so bumping it needs network access). The
+visible consequence is that 1.4-era `bun:ffi` declarations are absent from the
+types — `buffer_length` is not a valid `FFITypeOrString` literal there, so
+`src/native/ffi.ts` casts it (`as unknown as readonly FFITypeOrString[]`). Any
+new `bun:ffi` surface added in 1.4.x will need the same treatment until the
+types are upgraded. The casts are load-bearing, not sloppiness: dropping them is
+a type error, not a behavior change.
 
 | Claim | Source (Bun checkout `/home/adeel/poc/bun`) |
 |---|---|
@@ -328,8 +480,84 @@ Rust-based runtime family. Castrum's own measurements live in `bench/`.
 **Verify after any FFI change**:
 
 ```bash
-nm -D --defined-only <addon> | grep -c castrum_   # = 85
+nm -D --defined-only <addon> | grep -c castrum_   # = 109
 bun run bench:http:smoke                           # after touching decoders/handlers
 bun test test/unit/native/ffi.test.ts            # FFI↔napi parity + self-test
 bun test test/unit/native/ffi-symbol-parity.test.ts
 ```
+
+---
+
+## 14. Changing an existing symbol's ABI — the desync hazard (post-mortem 2026-08-23)
+
+Adding or re-signing a `castrum_*` symbol means touching **Rust and the JS
+dlopen map in lockstep**. `bun:ffi` performs ZERO runtime validation of the
+declared ABI: if the addon on disk and `src/native/ffi.ts` disagree — arity,
+arg order, or arg type — the call still crosses, the callee reads your scalar
+as a pointer (or vice versa), and the process **segfaults**. This is not a
+Bun bug; it is inherent to raw FFI, and it is why the bind-time self-test is
+mandatory.
+
+### The 2026-08-23 incident (recorded so nobody repeats it)
+
+Converting `castrum_sse_encode_into`'s event/id to `cstring` args triggered
+repeated SIGSEGVs that looked like an engine bug ("crash at address 0x5/0x6",
+flaky across runs, "raw bindings pass but the package crashes"). After a full
+root-cause pass, **every** crash traced to a development-loop desync — never
+to bun:ffi itself:
+
+1. **Half-updated trees**: rebuilding the Rust side while the JS map (or a
+   scratch probe) still declared the old signature — e.g. after `git stash`
+   / `git checkout` of one side. The callee then dereferenced a small integer
+   argument as a pointer: crash addresses literally equalled values passed in
+   earlier calls (`0x5`, `0x6`).
+2. **Probe methodology bugs**: shell probes treated ANY non-zero exit as a
+   crash — including clean `TypeError: Symbol "<key>" not found` failures
+   from a misnamed dlopen-map key. Several "crashes" were exit-code-1
+   TypeErrors.
+3. **A mid-session runtime auto-update** (Bun canary replaced under us),
+   invalidating comparisons across runs.
+
+The minimal C-dylib control experiment bound every suspect signature shape
+(`cstring` leading/middle/trailing, mixed with `buffer`/`buffer_length`,
+`u64_fast` returns) against trivial exports: all marshaled correctly on Bun
+1.4.0/1.4.1 canaries. **No engine bug exists to report.** (Re-run on the 1.4.2
+binding: `ffiBufferMode() === 'buffer-pair'`, i.e. the `buffer`/`buffer_length`
+pair is still live, and the full symbol set passes the bind-time self-test.)
+
+### Rules for any ABI change (signature, arg order, or type)
+
+1. **Treat Rust signature + JS dlopen-map entry + JS wrapper as ONE commit.**
+   Never run anything that binds (`bun test`, benches, `import castrum`)
+   between editing one side and the other.
+2. **Rebuild before you bind**: after ANY `rust/ffi/**` edit (including
+   reverting one via git), run `bun run build` before running tests. A stale
+   `castrum.*.node` next to fresh JS is a guaranteed segfault, not an error.
+3. **Fail loudly while debugging**: use `CASTRUM_FFI_MODE=ffi`. In `auto`
+   mode a failed bind/self-test silently falls back to napi — which masks
+   breakage as mysterious slowness or wrong-path results instead of an error.
+4. **Know the failure signatures**:
+   - `SIGSEGV at a tiny address (0x5–0xB)` right after a binding → almost
+     always an ABI mismatch; check `git status rust/ src/native/ffi.ts` and
+     rebuild before theorizing about the engine.
+   - `Symbol "<name>" not found` at dlopen → key/name drift (parity test's
+     domain).
+   - ffi silently off → self-test returned false; run with
+     `CASTRUM_FFI_MODE=ffi` to get the throw instead of the fallback.
+5. **Known guard gap**: `test/unit/native/ffi-symbol-parity.test.ts` checks
+   symbol NAMES (rust exports ⊆ dlopen map ⊆ self-test), NOT arities/types.
+   Arity drift surfaces only at call time — as a crash. When changing a
+   signature, update BOTH sides plus the self-test vectors in the same edit,
+   and let `bun test test/unit/native/ffi.test.ts` (FFI↔napi parity) catch
+   value divergence.
+6. **Probe discipline when investigating**: distinguish crash from failure
+   explicitly (exit code 139 + `Segmentation fault` line vs exit 1 +
+   stderr), verify `bun --version` before AND after long sessions (canary
+   auto-updates invalidate comparisons), and test hypotheses against a
+   minimal dylib before blaming the engine.
+7. **No repo-wide auto-fixers mid-investigation**: `bun run lint:fix`
+   rewrites import order across ~90 files; in this codebase import ORDER
+   carries side-effect semantics (eager-dlopen contract). Scope formatters
+   to your changed files, and treat mass diffs appearing during a debug
+   session as a variable to eliminate (revert them) before drawing
+   conclusions.

package/docs/HOW_TO_ADD_AN_OP.md

@@ -72,6 +72,16 @@ Rules (see `docs/FFI_BUN_GUIDE.md` for the full contract):
   pass as `usize`/number.
 - String ARGS come in as `*const c_char` (bun:ffi `cstring`); use
   `CStr::from_ptr(header).to_bytes()`.
+  
+  **NUL caveat**: the engine's transcode is NUL-terminated, so an embedded
+  `U+0000` TRUNCATES the value before Rust sees it. Only bind a `cstring` ARG
+  for developer-supplied config or input that cannot contain a NUL. If the op is
+  reachable from user input and answers with a verdict (validate/gate) or a
+  byte-exact value, provide a `*_bytes` sibling taking `(ptr,len)` (preferred —
+  exact length, no transcode, and measured faster for byte-shaped callers) or
+  guard the string form with `hasNul` (`src/shared/bytes.ts`).
+  See `docs/FFI_BUN_GUIDE.md` §6.1; regressions are caught by
+  `test/unit/native/cstring-nul.test.ts`.
 - String RETURNS use `cstring_return` (per-thread `CSTR_BUF`, cloned at call
   time — never return a pointer into JS-owned memory).
 - **Variable-size outputs**: return the EXACT required byte count on a

package/docs/REPO_MAP.md

@@ -53,7 +53,7 @@ Bun is the primary runtime; Node.js ≥20.3 is supported via a compiled ESM entr
 | Version consistency | `bun run check:version` | package.json ↔ Cargo.toml ↔ CHANGELOG |
 | JS dependency audit | `bun run audit` | `bun audit` (reports advisories) |
 | Cargo deny audit | `bun run deny` | `cargo deny check` |
-| Publish (manual, single-platform) | `bun run publish:manual[:dry]` | see §6 |
+| Publish (canonical) | `bun run release[:dry]` / `release:manual` | see §6 |
 
 ---
 
@@ -144,6 +144,11 @@ src/
     websocket.ts          createWebSocketUpgrade (RFC 6455 101 handshake + subprotocol).
     streaming.ts          sseResponse (SSE frames via rust.sseEncodeEvent).
   baseline/               Pure-JS baseline implementations (benchmark reference). Internal.
+  task/                   Off-thread task runtime ("castrum Tasks", Bun-first). See §RND-CONCURRENCY.
+    op.ts                 PURE op ids + packed-arg encoders (no addon import).
+    runtime.ts            createTaskRuntime() — process-wide singleton; submit → pool → batched
+                          doorbell → drained completion → promise; AbortSignal; zero-copy
+                          submitInto for ops with a size hint.
   bench/                  CPU benchmark framework (tasks, measure, report, checks, comparisons).
   data/                   json-rows.ts fixture generator (benchmark-only).
 ```
@@ -169,13 +174,17 @@ rust/                     ONE cdylib crate (Cargo [lib] → lib.rs).
                           native stack), options/time/packed, cors, proxy, ip_trust, rate_limit,
                           terminal, output.rs (single numeric layout source),
                           ingress_constants.rs (napi projection).
-  ffi/                    #[no_mangle] extern "C" exports (85 castrum_* symbols — 77 direct + 4
-                          validator_c_abi! + 4 compress_to_out!; parity guarded by
+  task/                   Off-thread task runtime (\"castrum Tasks\"): condvar pool (N = cores−1,
+                          CASTRUM_TASK_THREADS, batch dequeue + spin-then-park + optional
+                          CASTRUM_TASK_PIN_CORES), result ring + batched thread-safe doorbell,
+                          op dispatch + cancel + catch_unwind containment + `_into` ops.
+  ffi/                    #[no_mangle] extern "C" exports (119 castrum_* symbols — 107 direct + 4
+                          validator_c_abi! + 4 validator_bytes_c_abi! + 4 compress_to_out!; parity guarded by
                           test/unit/native/ffi-symbol-parity.test.ts) for Bun's bun:ffi
                           primary transport, incl. castrum_ingress_layout (the layout blob).
                           mod.rs (module map) + util.rs (panic_guard / HMAC cache / cstring helpers)
                           + per-domain wrappers (hashing/validators/crypto/jwt/http/payload/json/
-                          rate_limit/ingress/route/probe) + tests.rs.
+                          rate_limit/ingress/route/task/probe) + tests.rs.
   panic_safety.rs / test_support.rs   Cross-module fuzz + shared #[cfg(test)] helpers.
   proptest_suite.rs      Property-based adversarial-parser tests (dev-dep proptest).
 ```
@@ -284,8 +293,9 @@ consumed.
   (declared in package.json `napi.binaryName` + Cargo.toml — keep in sync).
 - **Multi-platform (recommended)**: push a `v*` tag → `.github/workflows/ci.yml`
   builds each platform addon, the `publish` job downloads them into
-  `./artifacts`, stages them, and runs `npm publish` (needs `NPM_TOKEN`).
-- **Manual single-platform**: `bun run publish:manual --increment minor`
+  `./artifacts`, stages them, and runs `npm publish` (npm Trusted Publishing /
+  OIDC — no token secret; needs the Trusted Publisher configured on npmjs.com).
+- **Manual single-platform**: `bun run release:manual`
   (syncs package.json ↔ Cargo.toml ↔ CHANGELOG, tags, builds, publishes with
   `CASTRUM_PUBLISH_ALLOW_PARTIAL=1`). `--dry-run` plans only.
 - `prepublishOnly` runs `build:js` + `scripts/prepublish.mjs` (stages
@@ -302,6 +312,7 @@ consumed.
 | [`API.md`](./API.md) | The complete public API reference |
 | [`CASE_STUDY.md`](./CASE_STUDY.md) | Data-driven case study: how castrum was built and measured |
 | [`ARCHITECTURE.md`](./ARCHITECTURE.md) | Deep-dive: modules, data flow, output layout, memory, concurrency |
+| [`RND-CONCURRENCY.md`](./RND-CONCURRENCY.md) | R&D + design plan: native (Rust) task offload for Bun — the "goroutine" gap, measured Bun concurrency costs, proposed `rust.tasks` runtime |
 | [`INGRESS.md`](./INGRESS.md) | The pre-baked ingress API + route factories + servers |
 | [`BENCHMARKS.md`](./BENCHMARKS.md) | Benchmark scenarios, report format, how to run |
 | [`ENVIRONMENT.md`](./ENVIRONMENT.md) | All `CASTRUM_*` env vars (+ legacy aliases) |