@nimbus-sh/fabric 0.8.0 → 0.9.0

package/README.md

@@ -22,9 +22,19 @@ npm install @nimbus-sh/fabric
 
 ## Requirements
 
-Set `compatibility_flags: ["nodejs_compat"]` in your Worker. The timer
-dispatcher needs `AsyncLocalStorage`, which workerd ships only under that
-flag. Without it the module fails to load at deploy time.
+Use a compatibility date of 2026-08-04 or later, or list `nodejs_compat` in
+`compatibility_flags`. The timer dispatcher needs `AsyncLocalStorage`, which
+workerd ships only under `nodejs_compat`, on by date from 2026-08-04. Without
+it the module fails to load at deploy time.
+
+`composeFabric` also needs `enhanced_error_serialization`: a compatibility
+date of 2026-04-21 or later, or the flag in `compatibility_flags` on an older
+date. A program's filesystem errors reach it across workerd RPC, and only
+that flag carries their `code`. On workerd without it `composeFabric` throws,
+naming both fixes, rather than every program seeing `EIO`. It throws where
+it is called. At module scope the Worker fails at startup. A library host
+that composes through `NimbusWorkspace.create({ fabric })` deploys and
+starts, and its first create throws.
 
 Import the root inside a Worker. Outside workerd, import subpaths such as
 `@nimbus-sh/fabric/timers.js`, which are typed against plain objects and run
@@ -224,13 +234,18 @@ Warm isolates are scoped to one session. A pool may opt into
 `cacheScope: 'global'` only if it takes no supervisor binding and keeps no
 user state.
 
-`Fanout` handles wider batches. One DO method can drive at most 4 concurrent
-loader fetches, so batches under 5 run in the coordinator and larger ones
-shard across up to 32 sibling objects, 4 at a time.
+`Fanout` handles wider batches. A Durable Object may have 10 distinct Dynamic
+Workers with in-flight requests at once (`DO_DYNAMIC_WORKER_LIMIT`), shared
+across every concurrent request to it; repeated requests to one Dynamic
+Worker count once. A batch that fits the coordinator's remaining headroom
+runs there, one Dynamic Worker per task; a wider one shards across up to 32
+sibling objects, 4 at a time, each spending its own headroom.
 
-Each keyed `loader.get(id)` permanently holds one of roughly 5–6
-dynamic-worker slots. `loaderLedgerStats(ctx)` reports what you have
-consumed, and a cap refusal names the IDs holding slots.
+The loader ledger counts what is in flight per DO: pool and one-shot calls,
+esbuild facet calls, git network ops, and every resident process for as long
+as it lives. `dynamicWorkerHeadroom(ctx)` is what is left,
+`claimDynamicWorkers(ctx, n)` reserves a width, `loaderLedgerStats(ctx)`
+reports it all, and a limit refusal names the workers in flight.
 
 ## Process fabric
 
@@ -370,8 +385,8 @@ or left to you.
 | Request-time `WebAssembly.compile`/`instantiate` CSP-blocked; wasm rides the loader modules map as `{ wasm: ArrayBuffer }`, compiled at module load | RPC of a compiled `Module` refused by structured clone; inlined bytes OOMed the supervisor |
 | Module scope bans I/O; `new Function` succeeds at module scope and throws at request time | code reaches a facet through the module map or not at all |
 | The facet start callback fires at most once | re-running it would re-execute the user's program |
-| ~5–6 concurrent dynamic workers per DO; at most 4 concurrent Loader fetches per DO method; loader-cache entries are never released | `IN_DO_THRESHOLD` = 5 sits under the fetch cap; every `loader.get(id)` permanently consumes a slot — counted per DO by the loader ledger, and a cap refusal names the ids holding them |
-| `ctx.facets.clone` is same-object only, absent from `@cloudflare/workers-types` and the pinned workerd, present in production | 18–31 ms / 45.7 MB, 34–54 ms / 1 GB; an unresolvable `src` silently EMPTIES the destination and reports success — `cloneStorage` enforces the both-ends validation |
+| 10 distinct Dynamic Workers with in-flight requests per DO, shared across its concurrent requests; repeated requests to one Dynamic Worker count once ([changelog, 2026-08-28](https://developers.cloudflare.com/changelog/post/2026-08-28-durable-objects-dynamic-workers-limit/)) | `DO_DYNAMIC_WORKER_LIMIT`; `Fanout` sizes in-DO batches to the live headroom, and a refusal names the workers in flight |
+| `ctx.facets.clone` is same-object only; declared by `@cloudflare/workers-types` 5 and present in workerd ≥ 1.20260926.1 and in production | 18–31 ms / 45.7 MB, 34–54 ms / 1 GB; an unresolvable `src` silently EMPTIES the destination and reports success — `cloneStorage` enforces the both-ends validation |
 | A DO dies at ~200 MiB of live wasm linear memory; reserved and written pages die at the same ceiling | lazy growth buys nothing; bound guest memory by rewriting the memory section |
 | A wasm stack suspended (JSPI) in one request cannot resume in another | 3 in-context resumes took 6 ms; the first cross-context one hit a 30 s timeout |
 

package/dist/budgets.d.ts

@@ -1,34 +1,39 @@
 /**
  * budgets.ts — per-DO accounting for the platform budgets the fabric spends:
- * the Worker Loader's two caps, the facet-ID lifetime budget, and the
- * dynamic-worker module-map ceiling.
+ * the Durable Object's Dynamic Worker concurrency limit, the facet-ID
+ * lifetime budget, and the dynamic-worker module-map ceiling.
  *
- * Measured on production workerd: a Durable Object admits ~5–6 concurrent
- * dynamic workers before the platform refuses with "Too many concurrent
- * dynamic workers", one DO method can drive at most 4 concurrent Loader
- * fetches, and loader-cache entries are never released — every DISTINCT
- * `loader.get(id)` permanently consumes one of the dynamic-worker slots for
- * the object's lifetime. Nimbus stays under the caps by construction
- * (`IN_DO_THRESHOLD` = 5 in the fanout pool), which until now meant the slots
- * were counted in prose. This ledger counts them at the fabric's loader call
- * sites instead — the loader pool's slots, a resident process's keyed worker,
- * a one-shot's load — so proximity is measurable and a cap failure can name
- * the ids actually holding slots.
+ * The Dynamic Worker model is Cloudflare's documented one
+ * ({@link DO_DYNAMIC_WORKER_LIMIT}): a Durable Object may have a fixed number
+ * of DISTINCT Dynamic Workers with in-flight requests at once, shared across
+ * every concurrent request to that object (one I/O context), and any number
+ * of in-flight requests to the same Dynamic Worker count as one. Only
+ * in-flight requests count: a loader id with nothing in flight holds nothing.
  *
- * Measurement only: no admission control. The caps are the platform's, they
- * are approximate ("~5–6"), and a gate on an approximate number would refuse
- * work the platform would have run.
+ * The ledger counts, per hosting actor, the distinct workers that are in
+ * flight right now, keyed by loader id (a fresh key per unkeyed `load`), plus
+ * the width fan-outs have claimed and not yet released. A fan-out spends only
+ * the {@link dynamicWorkerHeadroom} that leaves, so work a Durable Object
+ * already has in flight — a resident process, the esbuild facet, a git
+ * network op, another fan-out — keeps its slots.
  *
  * Keyed weakly off the hosting actor's `ctx`, like the facet slot books: the
- * caps are per Durable Object, and dynamic workers die with the isolate that
+ * limit is per Durable Object, and dynamic workers die with the isolate that
  * loaded them, so a ledger that goes away with its host describes nothing
  * that still exists.
  */
-/** Record a keyed `loader.get(id)` — a permanent slot if the id is new. */
-export declare function recordLoaderId(ctx: object, id: string): void;
 /**
- * Count one call into a dynamic worker as a live Loader fetch; the returned
- * function ends it (idempotently), from the caller's own `finally`.
+ * Distinct Dynamic Workers one Durable Object may have with in-flight
+ * requests at once, shared across all concurrent requests to that object;
+ * multiple in-flight requests to one Dynamic Worker count once.
+ * https://developers.cloudflare.com/changelog/post/2026-08-28-durable-objects-dynamic-workers-limit/
+ */
+export declare const DO_DYNAMIC_WORKER_LIMIT = 10;
+/**
+ * Hold the Dynamic Worker `workerKey` in flight on this actor's ledger; the
+ * returned function ends the hold (idempotently), from the caller's own
+ * `finally`. Holds on one key nest: the worker counts once until the last
+ * one ends, as the platform counts it.
  *
  * A begin/end pair rather than a wrapper on purpose, and the shape is
  * load-bearing: wrapping the stub call in a ledger-owned async frame
@@ -41,19 +46,35 @@ export declare function recordLoaderId(ctx: object, id: string): void;
  * workers: an RPC stub call must stay a direct property call awaited by the
  * frame that made it, so the ledger only brackets it.
  */
-export declare function beginLoaderFetch(ctx: object): () => void;
+export declare function beginLoaderFetch(ctx: object, workerKey: string): () => void;
+/**
+ * Distinct Dynamic Workers this actor may still put in flight: the limit
+ * less what is held and claimed right now. Never negative.
+ */
+export declare function dynamicWorkerHeadroom(ctx: object): number;
+/**
+ * Claim `width` distinct Dynamic Workers for one fan-out, or null when the
+ * headroom cannot hold it. The claim counts until `release` (idempotent), so
+ * a second fan-out sizing itself meanwhile sees it; the claimant's own
+ * dispatches are held as well while they run, which only ever over-counts
+ * toward sending that second fan-out elsewhere.
+ */
+export declare function claimDynamicWorkers(ctx: object, width: number): {
+    release(): void;
+} | null;
 /** Snapshot for the diag surface. Pure read; no I/O. */
 export declare function loaderLedgerStats(ctx: object): {
-    idsEverGotten: string[];
-    liveFetches: number;
-    peakLiveFetches: number;
+    limit: number;
+    inFlightWorkers: string[];
+    claimed: number;
+    headroom: number;
+    peak: number;
 };
 /**
- * Name the per-DO accounting on a "Too many concurrent dynamic workers"
+ * Name the per-DO accounting on a "Dynamic worker concurrency limit exceeded"
  * failure; hand every other error back untouched. The platform's message
- * says only that the cap was hit — which ids hold the slots, and that a
- * keyed id can never give one back, is what the operator needs to know to
- * shrink anything.
+ * says only that the limit was hit — which workers were in flight, and what
+ * fan-outs had claimed, is what the operator needs to know to shrink anything.
  */
 export declare function withDynamicWorkerCapNamed<E>(ctx: object, error: E): E | Error;
 /**

package/dist/budgets.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"budgets.d.ts","sourceRoot":"","sources":["../src/budgets.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;GAyBG;AAwBH,2EAA2E;AAC3E,wBAAgB,cAAc,CAAC,GAAG,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,GAAG,IAAI,CAE5D;AAED;;;;;;;;;;;;;;GAcG;AACH,wBAAgB,gBAAgB,CAAC,GAAG,EAAE,MAAM,GAAG,MAAM,IAAI,CAUxD;AAED,wDAAwD;AACxD,wBAAgB,iBAAiB,CAAC,GAAG,EAAE,MAAM,GAAG;IAC9C,aAAa,EAAE,MAAM,EAAE,CAAC;IACxB,WAAW,EAAE,MAAM,CAAC;IACpB,eAAe,EAAE,MAAM,CAAC;CACzB,CAOA;AAED;;;;;;GAMG;AACH,wBAAgB,yBAAyB,CAAC,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC,GAAG,CAAC,GAAG,KAAK,CAW7E;AAID;;;;;;;GAOG;AACH,eAAO,MAAM,+BAA+B,WAAa,CAAC;AAE1D;;;;;;;;;;;;;GAaG;AACH,wBAAgB,8BAA8B,CAAC,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,IAAI,CAoBrF;AAuBD;;;;;GAKG;AACH,eAAO,MAAM,wBAAwB,QAAS,CAAC;AAE/C,sEAAsE;AACtE,eAAO,MAAM,yBAAyB,iCAAiC,CAAC;AAExE,mEAAmE;AACnE,UAAU,sBAAsB;IAC9B,OAAO,EAAE;QACP,GAAG,CAAC,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC,GAAG,OAAO,CAAC;QAC7C,GAAG,CAAC,GAAG,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;KACjD,CAAC;CACH;AAqCD;;;;;GAKG;AACH,wBAAgB,qBAAqB,CAAC,GAAG,EAAE,sBAAsB,EAAE,KAAK,EAAE,MAAM,GAAG,IAAI,CAatF;AAED,4EAA4E;AAC5E,wBAAgB,cAAc,CAAC,GAAG,EAAE,sBAAsB,GAAG,MAAM,CAGlE;AAED,4EAA4E;AAC5E,wBAAsB,qBAAqB,CAAC,GAAG,EAAE,sBAAsB,GAAG,OAAO,CAAC,MAAM,CAAC,CAIxF;AAED;;;;;;;;GAQG;AACH,wBAAsB,aAAa,CACjC,GAAG,EAAE,sBAAsB,GAC1B,OAAO,CAAC;IAAE,QAAQ,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,MAAM,CAAA;CAAE,CAAC,CAK/C;AAED;;;;;;GAMG;AACH,wBAAgB,oBAAoB,CAAC,QAAQ,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,GAAG,OAAO,CAU9E"}
+{"version":3,"file":"budgets.d.ts","sourceRoot":"","sources":["../src/budgets.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;GAuBG;AAKH;;;;;GAKG;AACH,eAAO,MAAM,uBAAuB,KAAK,CAAC;AA0B1C;;;;;;;;;;;;;;;;GAgBG;AACH,wBAAgB,gBAAgB,CAAC,GAAG,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,GAAG,MAAM,IAAI,CAY3E;AAED;;;GAGG;AACH,wBAAgB,qBAAqB,CAAC,GAAG,EAAE,MAAM,GAAG,MAAM,CAEzD;AAED;;;;;;GAMG;AACH,wBAAgB,mBAAmB,CAAC,GAAG,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG;IAAE,OAAO,IAAI,IAAI,CAAA;CAAE,GAAG,IAAI,CAa1F;AAED,wDAAwD;AACxD,wBAAgB,iBAAiB,CAAC,GAAG,EAAE,MAAM,GAAG;IAC9C,KAAK,EAAE,MAAM,CAAC;IACd,eAAe,EAAE,MAAM,EAAE,CAAC;IAC1B,OAAO,EAAE,MAAM,CAAC;IAChB,QAAQ,EAAE,MAAM,CAAC;IACjB,IAAI,EAAE,MAAM,CAAC;CACd,CASA;AAED;;;;;GAKG;AACH,wBAAgB,yBAAyB,CAAC,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC,GAAG,CAAC,GAAG,KAAK,CAU7E;AAID;;;;;;;GAOG;AACH,eAAO,MAAM,+BAA+B,WAAa,CAAC;AAE1D;;;;;;;;;;;;;GAaG;AACH,wBAAgB,8BAA8B,CAAC,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,IAAI,CAoBrF;AA4BD;;;;;GAKG;AACH,eAAO,MAAM,wBAAwB,QAAS,CAAC;AAE/C,sEAAsE;AACtE,eAAO,MAAM,yBAAyB,iCAAiC,CAAC;AAExE,mEAAmE;AACnE,UAAU,sBAAsB;IAC9B,OAAO,EAAE;QACP,GAAG,CAAC,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC,GAAG,OAAO,CAAC;QAC7C,GAAG,CAAC,GAAG,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;KACjD,CAAC;CACH;AAqCD;;;;;GAKG;AACH,wBAAgB,qBAAqB,CAAC,GAAG,EAAE,sBAAsB,EAAE,KAAK,EAAE,MAAM,GAAG,IAAI,CAatF;AAED,4EAA4E;AAC5E,wBAAgB,cAAc,CAAC,GAAG,EAAE,sBAAsB,GAAG,MAAM,CAGlE;AAED,4EAA4E;AAC5E,wBAAsB,qBAAqB,CAAC,GAAG,EAAE,sBAAsB,GAAG,OAAO,CAAC,MAAM,CAAC,CAIxF;AAED;;;;;;;;GAQG;AACH,wBAAsB,aAAa,CACjC,GAAG,EAAE,sBAAsB,GAC1B,OAAO,CAAC;IAAE,QAAQ,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,MAAM,CAAA;CAAE,CAAC,CAK/C;AAED;;;;;;GAMG;AACH,wBAAgB,oBAAoB,CAAC,QAAQ,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,GAAG,OAAO,CAU9E"}

package/dist/budgets.js

@@ -1,46 +1,53 @@
 /**
  * budgets.ts — per-DO accounting for the platform budgets the fabric spends:
- * the Worker Loader's two caps, the facet-ID lifetime budget, and the
- * dynamic-worker module-map ceiling.
+ * the Durable Object's Dynamic Worker concurrency limit, the facet-ID
+ * lifetime budget, and the dynamic-worker module-map ceiling.
  *
- * Measured on production workerd: a Durable Object admits ~5–6 concurrent
- * dynamic workers before the platform refuses with "Too many concurrent
- * dynamic workers", one DO method can drive at most 4 concurrent Loader
- * fetches, and loader-cache entries are never released — every DISTINCT
- * `loader.get(id)` permanently consumes one of the dynamic-worker slots for
- * the object's lifetime. Nimbus stays under the caps by construction
- * (`IN_DO_THRESHOLD` = 5 in the fanout pool), which until now meant the slots
- * were counted in prose. This ledger counts them at the fabric's loader call
- * sites instead — the loader pool's slots, a resident process's keyed worker,
- * a one-shot's load — so proximity is measurable and a cap failure can name
- * the ids actually holding slots.
+ * The Dynamic Worker model is Cloudflare's documented one
+ * ({@link DO_DYNAMIC_WORKER_LIMIT}): a Durable Object may have a fixed number
+ * of DISTINCT Dynamic Workers with in-flight requests at once, shared across
+ * every concurrent request to that object (one I/O context), and any number
+ * of in-flight requests to the same Dynamic Worker count as one. Only
+ * in-flight requests count: a loader id with nothing in flight holds nothing.
  *
- * Measurement only: no admission control. The caps are the platform's, they
- * are approximate ("~5–6"), and a gate on an approximate number would refuse
- * work the platform would have run.
+ * The ledger counts, per hosting actor, the distinct workers that are in
+ * flight right now, keyed by loader id (a fresh key per unkeyed `load`), plus
+ * the width fan-outs have claimed and not yet released. A fan-out spends only
+ * the {@link dynamicWorkerHeadroom} that leaves, so work a Durable Object
+ * already has in flight — a resident process, the esbuild facet, a git
+ * network op, another fan-out — keeps its slots.
  *
  * Keyed weakly off the hosting actor's `ctx`, like the facet slot books: the
- * caps are per Durable Object, and dynamic workers die with the isolate that
+ * limit is per Durable Object, and dynamic workers die with the isolate that
  * loaded them, so a ledger that goes away with its host describes nothing
  * that still exists.
  */
 import { classifyError } from '@nimbus-sh/platform/oom-classify.js';
+import { hostWasmIdentity } from './host-wasm.js';
+/**
+ * Distinct Dynamic Workers one Durable Object may have with in-flight
+ * requests at once, shared across all concurrent requests to that object;
+ * multiple in-flight requests to one Dynamic Worker count once.
+ * https://developers.cloudflare.com/changelog/post/2026-08-28-durable-objects-dynamic-workers-limit/
+ */
+export const DO_DYNAMIC_WORKER_LIMIT = 10;
 const ledgers = new WeakMap();
 function ledger(ctx) {
     let entry = ledgers.get(ctx);
     if (!entry) {
-        entry = { ids: new Set(), liveFetches: 0, peakLiveFetches: 0 };
+        entry = { inFlight: new Map(), claimed: 0, peak: 0 };
         ledgers.set(ctx, entry);
     }
     return entry;
 }
-/** Record a keyed `loader.get(id)` — a permanent slot if the id is new. */
-export function recordLoaderId(ctx, id) {
-    ledger(ctx).ids.add(id);
+function inUse(entry) {
+    return entry.inFlight.size + entry.claimed;
 }
 /**
- * Count one call into a dynamic worker as a live Loader fetch; the returned
- * function ends it (idempotently), from the caller's own `finally`.
+ * Hold the Dynamic Worker `workerKey` in flight on this actor's ledger; the
+ * returned function ends the hold (idempotently), from the caller's own
+ * `finally`. Holds on one key nest: the worker counts once until the last
+ * one ends, as the platform counts it.
  *
  * A begin/end pair rather than a wrapper on purpose, and the shape is
  * load-bearing: wrapping the stub call in a ledger-owned async frame
@@ -53,43 +60,77 @@ export function recordLoaderId(ctx, id) {
  * workers: an RPC stub call must stay a direct property call awaited by the
  * frame that made it, so the ledger only brackets it.
  */
-export function beginLoaderFetch(ctx) {
+export function beginLoaderFetch(ctx, workerKey) {
     const entry = ledger(ctx);
-    entry.liveFetches++;
-    entry.peakLiveFetches = Math.max(entry.peakLiveFetches, entry.liveFetches);
+    entry.inFlight.set(workerKey, (entry.inFlight.get(workerKey) ?? 0) + 1);
+    entry.peak = Math.max(entry.peak, inUse(entry));
     let ended = false;
     return () => {
         if (ended)
             return;
         ended = true;
-        entry.liveFetches--;
+        const open = (entry.inFlight.get(workerKey) ?? 1) - 1;
+        if (open > 0)
+            entry.inFlight.set(workerKey, open);
+        else
+            entry.inFlight.delete(workerKey);
+    };
+}
+/**
+ * Distinct Dynamic Workers this actor may still put in flight: the limit
+ * less what is held and claimed right now. Never negative.
+ */
+export function dynamicWorkerHeadroom(ctx) {
+    return Math.max(0, DO_DYNAMIC_WORKER_LIMIT - inUse(ledger(ctx)));
+}
+/**
+ * Claim `width` distinct Dynamic Workers for one fan-out, or null when the
+ * headroom cannot hold it. The claim counts until `release` (idempotent), so
+ * a second fan-out sizing itself meanwhile sees it; the claimant's own
+ * dispatches are held as well while they run, which only ever over-counts
+ * toward sending that second fan-out elsewhere.
+ */
+export function claimDynamicWorkers(ctx, width) {
+    const entry = ledger(ctx);
+    if (width < 1 || width > DO_DYNAMIC_WORKER_LIMIT - inUse(entry))
+        return null;
+    entry.claimed += width;
+    entry.peak = Math.max(entry.peak, inUse(entry));
+    let released = false;
+    return {
+        release() {
+            if (released)
+                return;
+            released = true;
+            entry.claimed -= width;
+        },
     };
 }
 /** Snapshot for the diag surface. Pure read; no I/O. */
 export function loaderLedgerStats(ctx) {
     const entry = ledger(ctx);
     return {
-        idsEverGotten: [...entry.ids],
-        liveFetches: entry.liveFetches,
-        peakLiveFetches: entry.peakLiveFetches,
+        limit: DO_DYNAMIC_WORKER_LIMIT,
+        inFlightWorkers: [...entry.inFlight.keys()],
+        claimed: entry.claimed,
+        headroom: dynamicWorkerHeadroom(ctx),
+        peak: entry.peak,
     };
 }
 /**
- * Name the per-DO accounting on a "Too many concurrent dynamic workers"
+ * Name the per-DO accounting on a "Dynamic worker concurrency limit exceeded"
  * failure; hand every other error back untouched. The platform's message
- * says only that the cap was hit — which ids hold the slots, and that a
- * keyed id can never give one back, is what the operator needs to know to
- * shrink anything.
+ * says only that the limit was hit — which workers were in flight, and what
+ * fan-outs had claimed, is what the operator needs to know to shrink anything.
  */
 export function withDynamicWorkerCapNamed(ctx, error) {
     if (classifyError(error) !== 'dynamic_worker_cap')
         return error;
     const entry = ledger(ctx);
     const platform = error instanceof Error ? error.message : String(error);
-    return new Error(`${platform} — this Durable Object has ${entry.ids.size} loader id(s) permanently `
-        + `holding dynamic-worker slots (a loader.get id is never released): `
-        + `${[...entry.ids].join(', ') || '(none recorded)'}; live Loader fetches ${entry.liveFetches}, `
-        + `peak ${entry.peakLiveFetches}`, { cause: error });
+    return new Error(`${platform} — this Durable Object had ${entry.inFlight.size} distinct dynamic worker(s) in flight `
+        + `(${[...entry.inFlight.keys()].join(', ') || 'none recorded'}) and ${entry.claimed} claimed by fan-outs, `
+        + `against a limit of ${DO_DYNAMIC_WORKER_LIMIT}; peak ${entry.peak}`, { cause: error });
 }
 // ── Dynamic-worker module-map ceiling ───────────────────────────────────────
 /**
@@ -136,13 +177,18 @@ export function assertModuleMapWithinCodeLimit(modules) {
 }
 /**
  * Bytes one module-map member carries, across the loader's content kinds
- * (plain string, `{ js | cjs | py | text }`, `{ wasm | data }`). With an
- * encoder, text is measured exactly; without one, by code-unit length.
+ * (plain string, `{ js | cjs | py | text }`, `{ wasm | data }`, a bare
+ * WebAssembly.Module). With an encoder, text is measured exactly; without
+ * one, by code-unit length. A compiled module counts the wire size its host
+ * described (host-wasm.ts); one nobody described counts nothing here and is
+ * left to the platform's own refusal, as the text undercount is.
  */
 function memberBytes(content, encoder) {
     const textBytes = (text) => encoder ? encoder.encode(text).byteLength : text.length;
     if (typeof content === 'string')
         return textBytes(content);
+    if (content instanceof WebAssembly.Module)
+        return hostWasmIdentity(content)?.bytes ?? 0;
     if (content !== null && typeof content === 'object') {
         for (const value of Object.values(content)) {
             if (typeof value === 'string')
@@ -151,6 +197,8 @@ function memberBytes(content, encoder) {
                 return value.byteLength;
             if (ArrayBuffer.isView(value))
                 return value.byteLength;
+            if (value instanceof WebAssembly.Module)
+                return hostWasmIdentity(value)?.bytes ?? 0;
         }
     }
     return 0;

package/dist/do-calls.d.ts

@@ -40,6 +40,7 @@
  * namespace stubs, where the thunk shape is production-proven in Proteus.
  */
 import { type DoCallClass } from '@nimbus-sh/platform/oom-classify.js';
+import { type SpanRecorder } from '@nimbus-sh/platform/tracing.js';
 export interface DoCallRetryPolicy {
     maxAttempts?: number;
     baseDelayMs?: number;
@@ -78,7 +79,26 @@ export interface DoCallRetryPolicy {
      * its error.
      */
     onRetry?(info: DoCallRetryInfo): void;
+    /**
+     * Where the call's telemetry goes: its span's recorder. Each attempt lost
+     * to a transient or overloaded failure is recorded as an exception whose
+     * `code` is the failure's class, and once the call has settled it gets
+     * `do_call.attempts` (started), `do_call.hedges` (started by a hedge),
+     * `do_call.answered_by` (the attempt whose answer the call took, absent
+     * when none answered) and `do_call.outcome` ({@link DoCallOutcome}).
+     * Nothing recorded can change the call's answer. Records nothing when
+     * absent.
+     */
+    span?: SpanRecorder;
 }
+/**
+ * How an `idempotent` call ended: `answered` (an attempt succeeded),
+ * `callee_error` (the callee's own failure, which is an answer),
+ * `exhausted` (the last transient failure, no repeat left), `overloaded`
+ * (shed, nothing repeated after it), or `caller_error` (the resolver or
+ * `onRetry` threw).
+ */
+export type DoCallOutcome = 'answered' | 'callee_error' | 'exhausted' | 'overloaded' | 'caller_error';
 /** What one retry is answering: which call, which platform class, which
  *  attempt just failed out of how many. */
 export interface DoCallRetryInfo {

package/dist/do-calls.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"do-calls.d.ts","sourceRoot":"","sources":["../src/do-calls.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAwCG;AAEH,OAAO,EAAqC,KAAK,WAAW,EAAE,MAAM,qCAAqC,CAAC;AAU1G,MAAM,WAAW,iBAAiB;IAChC,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB;;;;;;OAMG;IACH,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB;;;;;;;;;;;;;;;OAeG;IACH,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB;;;;;;;;OAQG;IACH,OAAO,CAAC,CAAC,IAAI,EAAE,eAAe,GAAG,IAAI,CAAC;CACvC;AAED;2CAC2C;AAC3C,MAAM,WAAW,eAAe;IAC9B,SAAS,EAAE,MAAM,CAAC;IAClB,cAAc,EAAE,WAAW,CAAC;IAC5B;;;OAGG;IACH,OAAO,EAAE,MAAM,CAAC;IAChB,WAAW,EAAE,MAAM,CAAC;IACpB,KAAK,EAAE,OAAO,CAAC;CAChB;AAED;;;;;;;;;GASG;AACH,MAAM,MAAM,cAAc,CAAC,CAAC,IAAI,MAAM,CAAC,GAAG,OAAO,CAAC,CAAC,CAAC,CAAC;AAErD;;;;;GAKG;AACH,qBAAa,WAAY,SAAQ,KAAK;IAElC,QAAQ,CAAC,SAAS,EAAE,MAAM;IAC1B,QAAQ,CAAC,IAAI,EAAE,YAAY,GAAG,UAAU;IACxC,QAAQ,CAAC,cAAc,EAAE,WAAW;gBAF3B,SAAS,EAAE,MAAM,EACjB,IAAI,EAAE,YAAY,GAAG,UAAU,EAC/B,cAAc,EAAE,WAAW,EACpC,KAAK,EAAE,OAAO;CASjB;AAED;;;;;;;;;;;;;;GAcG;AACH,wBAAgB,UAAU,CAAC,CAAC,EAAE,CAAC,EAC7B,SAAS,EAAE,MAAM,EACjB,IAAI,EAAE,cAAc,CAAC,CAAC,CAAC,EACvB,IAAI,EAAE,CAAC,IAAI,EAAE,CAAC,KAAK,OAAO,CAAC,CAAC,CAAC,EAC7B,MAAM,GAAE,iBAAsB,GAC7B,OAAO,CAAC,CAAC,CAAC,CAmHZ;AAED;;;;;;;;;;;;;;;;;;;;GAoBG;AACH,wBAAsB,QAAQ,CAAC,CAAC,EAAE,CAAC,EACjC,SAAS,EAAE,MAAM,EACjB,IAAI,EAAE,cAAc,CAAC,CAAC,CAAC,EACvB,IAAI,EAAE,CAAC,IAAI,EAAE,CAAC,KAAK,OAAO,CAAC,CAAC,CAAC,GAC5B,OAAO,CAAC,CAAC,CAAC,CAUZ"}
+{"version":3,"file":"do-calls.d.ts","sourceRoot":"","sources":["../src/do-calls.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAwCG;AAEH,OAAO,EAAqC,KAAK,WAAW,EAAE,MAAM,qCAAqC,CAAC;AAE1G,OAAO,EAAY,KAAK,YAAY,EAAE,MAAM,gCAAgC,CAAC;AAS7E,MAAM,WAAW,iBAAiB;IAChC,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB;;;;;;OAMG;IACH,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB;;;;;;;;;;;;;;;OAeG;IACH,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB;;;;;;;;OAQG;IACH,OAAO,CAAC,CAAC,IAAI,EAAE,eAAe,GAAG,IAAI,CAAC;IACtC;;;;;;;;;OASG;IACH,IAAI,CAAC,EAAE,YAAY,CAAC;CACrB;AAED;;;;;;GAMG;AACH,MAAM,MAAM,aAAa,GAAG,UAAU,GAAG,cAAc,GAAG,WAAW,GAAG,YAAY,GAAG,cAAc,CAAC;AAEtG;2CAC2C;AAC3C,MAAM,WAAW,eAAe;IAC9B,SAAS,EAAE,MAAM,CAAC;IAClB,cAAc,EAAE,WAAW,CAAC;IAC5B;;;OAGG;IACH,OAAO,EAAE,MAAM,CAAC;IAChB,WAAW,EAAE,MAAM,CAAC;IACpB,KAAK,EAAE,OAAO,CAAC;CAChB;AAED;;;;;;;;;GASG;AACH,MAAM,MAAM,cAAc,CAAC,CAAC,IAAI,MAAM,CAAC,GAAG,OAAO,CAAC,CAAC,CAAC,CAAC;AAErD;;;;;GAKG;AACH,qBAAa,WAAY,SAAQ,KAAK;IAElC,QAAQ,CAAC,SAAS,EAAE,MAAM;IAC1B,QAAQ,CAAC,IAAI,EAAE,YAAY,GAAG,UAAU;IACxC,QAAQ,CAAC,cAAc,EAAE,WAAW;gBAF3B,SAAS,EAAE,MAAM,EACjB,IAAI,EAAE,YAAY,GAAG,UAAU,EAC/B,cAAc,EAAE,WAAW,EACpC,KAAK,EAAE,OAAO;CASjB;AAED;;;;;;;;;;;;;;GAcG;AACH,wBAAgB,UAAU,CAAC,CAAC,EAAE,CAAC,EAC7B,SAAS,EAAE,MAAM,EACjB,IAAI,EAAE,cAAc,CAAC,CAAC,CAAC,EACvB,IAAI,EAAE,CAAC,IAAI,EAAE,CAAC,KAAK,OAAO,CAAC,CAAC,CAAC,EAC7B,MAAM,GAAE,iBAAsB,GAC7B,OAAO,CAAC,CAAC,CAAC,CAgIZ;AAED;;;;;;;;;;;;;;;;;;;;GAoBG;AACH,wBAAsB,QAAQ,CAAC,CAAC,EAAE,CAAC,EACjC,SAAS,EAAE,MAAM,EACjB,IAAI,EAAE,cAAc,CAAC,CAAC,CAAC,EACvB,IAAI,EAAE,CAAC,IAAI,EAAE,CAAC,KAAK,OAAO,CAAC,CAAC,CAAC,GAC5B,OAAO,CAAC,CAAC,CAAC,CAUZ"}

package/dist/do-calls.js

@@ -41,6 +41,7 @@
  */
 import { classifyDoCall, isRetryableDoCall } from '@nimbus-sh/platform/oom-classify.js';
 import { disposeRpcResource } from '@nimbus-sh/platform/rpc-dispose.js';
+import { untraced } from '@nimbus-sh/platform/tracing.js';
 /** Total attempts. Two retries is what a dropped connection or a deploy
  *  bounce needs; beyond that the object is not coming back inside this
  *  request (the consumer's measured bound). */
@@ -87,7 +88,7 @@ export class DoCallError extends Error {
 export function idempotent(operation, stub, call, policy = {}) {
     const maxAttempts = policy.maxAttempts ?? MAX_ATTEMPTS;
     const baseDelayMs = policy.baseDelayMs ?? BASE_DELAY_MS;
-    const { hedgeAfterMs, retryWindowMs } = policy;
+    const { hedgeAfterMs, retryWindowMs, span = untraced } = policy;
     const startedAt = Date.now();
     // The executor form: fabric's library target predates Promise.withResolvers.
     return new Promise((resolve, reject) => {
@@ -99,7 +100,9 @@ export function idempotent(operation, stub, call, policy = {}) {
         // An attempt was shed as overloaded: nothing is repeated after it.
         let refused = false;
         let settled = false;
-        const settle = (answer) => {
+        // Attempts a hedge started rather than a retry or the first send.
+        let hedged = 0;
+        const settle = (outcome, answeredBy, answer) => {
             if (settled)
                 return;
             settled = true;
@@ -107,6 +110,12 @@ export function idempotent(operation, stub, call, policy = {}) {
                 clearTimeout(timer);
             hedges.clear();
             answer();
+            span.set({
+                'do_call.attempts': started,
+                'do_call.hedges': hedged,
+                'do_call.answered_by': answeredBy,
+                'do_call.outcome': outcome,
+            });
         };
         /** May another attempt start at `at`? */
         const canRepeat = (at) => !settled && !refused && started < maxAttempts
@@ -114,13 +123,20 @@ export function idempotent(operation, stub, call, policy = {}) {
         /** `error` ended an attempt that will not be repeated: the call's answer, once nothing else is live. */
         const exhausted = (error) => {
             if (live === 0)
-                settle(() => reject(error));
+                settle(refused ? 'overloaded' : 'exhausted', undefined, () => reject(error));
         };
         /** A failed attempt, numbered: repeat it after its backoff, or let it stand. */
         const failed = async (number, error) => {
             if (settled)
                 return;
             const classification = classifyDoCall(error);
+            if (!isRetryableDoCall(classification) && classification !== 'overloaded') {
+                // The call ran and its answer is this error — ENOENT is a read's answer as much as bytes are.
+                settle('callee_error', number, () => reject(error));
+                return;
+            }
+            // A lost attempt: the call's span says which, and why.
+            span.exception(error, classification, `attempt ${number} of ${maxAttempts}: `);
             if (classification === 'overloaded') {
                 // A shed call is no answer: nothing more is sent, and an attempt
                 // still in flight may yet answer.
@@ -128,11 +144,6 @@ export function idempotent(operation, stub, call, policy = {}) {
                 exhausted(error);
                 return;
             }
-            if (!isRetryableDoCall(classification)) {
-                // The call ran and its answer is this error — ENOENT is a read's answer as much as bytes are.
-                settle(() => reject(error));
-                return;
-            }
             const delayMs = Math.floor(Math.random() * 2 ** number * baseDelayMs);
             if (!canRepeat(Date.now() + delayMs)) {
                 exhausted(error);
@@ -163,8 +174,10 @@ export function idempotent(operation, stub, call, policy = {}) {
             const hedge = hedgeAfterMs === undefined ? undefined : setTimeout(() => {
                 if (hedge !== undefined)
                     hedges.delete(hedge);
-                if (canRepeat(Date.now()))
+                if (canRepeat(Date.now())) {
+                    hedged++;
                     attempt();
+                }
             }, hedgeAfterMs);
             if (hedge !== undefined)
                 hedges.add(hedge);
@@ -193,11 +206,11 @@ export function idempotent(operation, stub, call, policy = {}) {
                 disposeRpcResource(result);
                 return;
             }
-            settle(() => resolve(result));
+            settle('answered', number, () => resolve(result));
         };
         /** Start an attempt. Whatever it throws outside the call itself fails the call. */
         const attempt = () => {
-            run().catch((error) => settle(() => reject(error)));
+            run().catch((error) => settle('caller_error', undefined, () => reject(error)));
         };
         attempt();
     });