@intx/workflow-host 0.2.2 → 0.4.0

package/README.md

@@ -26,9 +26,14 @@ The package is organized along the abstract pieces it implements:
   startup recovery walk and a live `subscribeKind` loop so a
   `TimerSet` committed by an active workflow process fires without
   waiting for a process restart. `signal-channel.ts` funnels live
-  `SignalReceived` commits into the matching awaiter, with resume
-  rehydration consulting `unconsumedSignals` so a signal that
-  arrived while offline replays before live subscription begins.
+  `SignalReceived` commits into the matching awaiter. The channel
+  reads `unconsumedSignals` through its injected `readState`
+  reader, but every production call site passes an `emptyState`
+  reader, so that queue is always empty in production: a pre-await
+  signal resolves through the live `subscribeKind` tail, and
+  resume rehydration of a signal that arrived while the run was
+  offline is not wired. Plumbing the runtime body's own `RunState`
+  reader into the child is what that capability waits on.
 - `ipc/` — control and event channel implementations the
   supervisor wraps. Threat model lives at the top of
   `ipc/index.ts`; the supervisor uses these primitives directly.
@@ -95,7 +100,7 @@ The constructor argument shape:
   `require.resolve` / `import.meta.resolve` against the host's own
   package, `@intx/<host>`).
 - `substrateEnv`, `workflowRunRepoId`, `workflowRunRef`,
-  `deploymentId`, `deploymentMailAddress`, `readPrincipal`,
+  `anchorRunId`, `deploymentMailAddress`, `readPrincipal`,
   `deriveStepAddress`, `deriveStepRepoId?`, `ipcKeyPairFactory?` —
   per-deployment configuration the supervisor needs in its closure
   state.
@@ -144,9 +149,61 @@ signed shape as the operator and drain origins.
 `shutdown()` unregisters the mail address, kills the child, and
 disposes subscriptions.
 
-`drain` and `recycle` are stubs in this commit; the full
-implementations land with the drain controller and recycle paths
-respectively.
+`drain(opts)` sends the drain control mail and waits for in-flight
+runs to drain per each step's `drainBehavior`; on the drain-timeout it
+escalates to a signed `CancelRequested{origin: "supervisor-drain"}`.
+
+`recycle(opts)` tears the current child down and stands a fresh one up
+against the SAME deploy tree (same materialized source closure, same
+per-step credential repos). It is strictly orthogonal to redeploy,
+which mints a new deploy tree. Operator, supervisor-policy (max-uptime
+/ max-rss / grants-staleness), and workflow-process-self-initiated
+origins all funnel through the same path.
+
+### Respawn policy
+
+An UNEXPECTED child exit — a crash, OOM, panic, or signal, as opposed
+to a supervisor-initiated shutdown or recycle — is detected by watching
+the child process's `exited`, not the IPC channel: a clean process
+death ends the channel readers without a protocol-level crash callback,
+so `exited` is the only universal death signal. The supervisor
+classifies the exit by cohort generation and lifecycle phase — an exit
+of the current running cohort that no planned teardown owns is
+unexpected.
+
+On an unexpected exit the supervisor, with no external intervention:
+
+1. Replays any mail stranded mid-flight — entries the dead child's
+   in-flight dispatch left in the per-address `processing/` subtree —
+   back into `inbox/` under their original `<receivedAt>-<messageId>`
+   keys. Those keys sort ahead of any mail that arrived during the
+   kill/respawn gap, so the stranded entry is re-dispatched first and
+   FIFO ordering holds across the respawn boundary.
+2. Spawns a fresh workflow-process child against the same deploy tree
+   (reusing the recycle path's respawn machinery) and resumes dispatch.
+
+The respawn is bounded so a persistently-broken child cannot saturate
+the host. Every bound is operator-overridable via
+`WorkflowSupervisorBindings`; the defaults are:
+
+- **Exponential backoff.** Each respawn waits before spawning, starting
+  at `respawnBackoffInitialMs` (1s) and doubling to a
+  `respawnBackoffMaxMs` (30s) cap.
+- **Crash-loop guard.** If the child exits unexpectedly
+  `crashLoopMaxCount` (3) times within `crashLoopWindowMs` (60s), the
+  supervisor stops respawning and latches the deployment to a terminal
+  `crash-looping` state.
+- **Stable-run reset.** Once a crash-respawned child stays up for
+  `crashLoopStableResetMs` (60s), the crash counter and the backoff
+  reset, so a flap followed by stability does not permanently latch.
+
+`crash-looping` is an in-memory, per-process phase — no external reader
+observes it. The durable, externally-queryable signal is the run's
+status: on latch the supervisor (the sole writer of the workflow-run
+repo) commits a `RunFailed` for the deployment's stable run, flipping
+its `workflow_run.status` to `failed` through the same pack path every
+other terminal run uses. External automation that watches run status
+sees the crash-loop as a failed run.
 
 ### Host wiring
 
@@ -208,12 +265,6 @@ without reconstructing the env. The closure looks up the
 originating step's grants by `stepId` and delegates to a
 host-supplied `GrantEvaluator`.
 
-### Placeholders
-
-`DrainController` is a no-op placeholder in this commit; the real
-controller lands separately. `recycle` is a no-op pending the
-recycle path.
-
 ## Hosting the workflow-process child
 
 `@intx/workflow-host` ships the runtime body
@@ -241,7 +292,19 @@ The contract is intentionally narrow:
    factory returns `RunWorkflowChildBindings`: substrate `RepoStore`,
    principal, per-deployment repo ids, scheduler, step invoker, child
    spawner, grant evaluator. The factory consumes the typed struct,
-   never `NodeJS.ProcessEnv` directly.
+   never `NodeJS.ProcessEnv` directly. Each spawner declares
+   `hasUpstreamSignalResolver` on the runtime env it builds: `true`
+   for a run an answer can reach -- the deployment's own addressable
+   run, or a suspendable body whose container relays a decision back
+   down -- and `false` for a terminal `childWorkflow` child, which
+   carries no address and is run to its terminal rather than driven
+   across parks. The field is required, so an omission is a compile
+   error; a wrong `true` is not. Declared on a terminal-child seam it
+   type-checks and reinstates the hang the flag exists to remove. The
+   in-tree seams show both answers: `buildRuntimeEnv` declares `true`
+   for the deployment's own run, while in the sidecar's substrate
+   factory `createSidecarRunChild` declares `false` and
+   `createSidecarSpawnSuspendableChild` declares `true`.
 3. **The helper fails loudly.** A missing or malformed spawn-time
    env throws via `parseSpawnTimeEnv`; a substrate-config key the
    host listed but the supervisor did not populate throws before the

package/dist/adapters/mail-part-store.d.ts

@@ -0,0 +1,46 @@
+import type { Principal, RepoId, RepoStore as SubstrateRepoStore } from "@intx/hub-sessions/substrate";
+import type { Mail, MailPartReader, MessageHeaders, MessagePart } from "@intx/types/runtime";
+/**
+ * Thrown for a DETERMINISTIC, input-shaped rejection of an inbound mail -- a
+ * messageId that cannot form a usable path segment. Distinct from a transient
+ * substrate write failure so the caller drops the offending mail (replaying it
+ * would fail identically) rather than treating it as a retryable fault.
+ */
+export declare class InvalidMailError extends Error {
+    constructor(message: string, options?: {
+        cause?: unknown;
+    });
+}
+export type MailPartStoreOpts = {
+    substrate: SubstrateRepoStore;
+    repoId: RepoId;
+    principal: Principal;
+    runId: string;
+    ref: string;
+};
+/**
+ * Commit a decoded message's parts and assemble the JSON-safe `Mail`. Every
+ * part's bytes are written in ONE prefix-preserving commit under the message's
+ * directory (write-once, atomic), and each part becomes a `MailPart` descriptor
+ * carrying its metadata, an opaque `ref`, and -- for a small UTF-8 text part --
+ * its decoded `text` inline.
+ */
+export declare function commitMail(opts: MailPartStoreOpts, messageId: string, decoded: {
+    headers: MessageHeaders;
+    rawHeaders: Record<string, string[]>;
+    parts: MessagePart[];
+}): Promise<Mail>;
+export type MailPartReaderOpts = {
+    substrate: SubstrateRepoStore;
+    repoId: RepoId;
+    principal: Principal;
+    ref: string;
+};
+/**
+ * Construct the single mail-part reader for a deployment's workflow-run repo.
+ * `read` resolves any run's `MailPart.ref` to the committed bytes through a
+ * committed read pinned to the object store, so a cross-run read (a childflow
+ * or body step resolving a parent's part) never observes the lagging working
+ * tree.
+ */
+export declare function createMailPartReader(opts: MailPartReaderOpts): MailPartReader;

package/dist/adapters/mail-part-store.js

@@ -0,0 +1,251 @@
+// Durable store for a run's inbound-mail parts.
+//
+// The supervisor decodes an inbound MIME message into parts (via `decodeMail`)
+// and commits each part's decoded bytes here as a real file under
+// `runs/<runId>/parts/<urlEncoded(messageId)>/<index>-<name>`, returning the
+// JSON-safe `MailPart[]` descriptors that ride in the run's trigger/signal
+// payload. A `MailPartReader` resolves a descriptor's opaque `ref` back to the
+// committed bytes for any consumer -- an agent's content-block projection, a
+// workflow tool, a child run -- through the single, environment-agnostic
+// `MailPartReader` interface.
+//
+// Modeled on the sibling `blob-substrate` adapter: same per-run handles, same
+// substrate primitives (`writeTreePreservingPrefix` to write raw bytes;
+// `openCommittedReads` to read them back from a coherent object-store snapshot
+// rather than the lagging working tree). The write happens in one commit per
+// message (write-once, atomic). The kind handler validates the subtree shape;
+// this module sanitizes untrusted names to satisfy it and reuses the handler's
+// path-component byte cap.
+import { MAX_MAIL_PART_PATH_COMPONENT_BYTES, WORKFLOW_RUN_PARTS_DIR, WORKFLOW_RUN_RUNS_PREFIX, } from "@intx/hub-sessions/substrate";
+const REF_SCHEME = "mail-part:///";
+// Content types whose bytes are UTF-8 text and small enough to also inline as
+// `MailPart.text`, so a selector can read them without resolving the ref.
+const INLINE_TEXT_MAX_BYTES = 1024 * 1024;
+const CONTROL_CHAR_MAX = 0x1f;
+const DEL_CHAR = 0x7f;
+// Unicode line/paragraph separators. JavaScript's regex `.` does NOT match
+// these, so the kind handler's `<index>-<name>` check (whose name group is
+// `.+`) rejects a filename containing them. The sanitizer must strip them to
+// keep its "satisfies the handler by construction" contract.
+const LINE_SEPARATOR = 0x2028;
+const PARAGRAPH_SEPARATOR = 0x2029;
+const encoder = new TextEncoder();
+function byteLength(value) {
+    return encoder.encode(value).length;
+}
+/**
+ * Thrown for a DETERMINISTIC, input-shaped rejection of an inbound mail -- a
+ * messageId that cannot form a usable path segment. Distinct from a transient
+ * substrate write failure so the caller drops the offending mail (replaying it
+ * would fail identically) rather than treating it as a retryable fault.
+ */
+export class InvalidMailError extends Error {
+    constructor(message, options) {
+        super(message, options);
+        this.name = "InvalidMailError";
+    }
+}
+function encodeMessageSegment(messageId) {
+    const encoded = encodeURIComponent(messageId);
+    if (byteLength(encoded) > MAX_MAIL_PART_PATH_COMPONENT_BYTES) {
+        throw new InvalidMailError(`mail part store: messageId ${JSON.stringify(messageId)} url-encodes to ${String(byteLength(encoded))} bytes, over the ${String(MAX_MAIL_PART_PATH_COMPONENT_BYTES)}-byte path-component limit`);
+    }
+    // `encodeURIComponent` leaves `.` unescaped, so "." or ".." would form a
+    // traversal segment; reject it where the messageId -> segment constraint is
+    // owned. An empty segment is unreachable for a non-empty messageId but is
+    // refused for the same reason.
+    if (encoded.length === 0 || encoded === "." || encoded === "..") {
+        throw new InvalidMailError(`mail part store: messageId ${JSON.stringify(messageId)} url-encodes to ${JSON.stringify(encoded)}, which is not a usable path segment`);
+    }
+    return encoded;
+}
+/**
+ * Reduce an untrusted part name (a MIME filename, or a fallback) to one safe
+ * path segment: path separators, NUL, and control characters become `_`. The
+ * `<index>-` prefix guarantees per-message uniqueness, so a sanitization
+ * collision between two parts of one message is harmless.
+ */
+function sanitizePartName(name) {
+    let out = "";
+    for (const ch of name) {
+        const code = ch.codePointAt(0) ?? 0;
+        out +=
+            ch === "/" ||
+                ch === "\\" ||
+                code <= CONTROL_CHAR_MAX ||
+                code === DEL_CHAR ||
+                code === LINE_SEPARATOR ||
+                code === PARAGRAPH_SEPARATOR
+                ? "_"
+                : ch;
+    }
+    return out.length > 0 ? out : "part";
+}
+/** Truncate to at most `maxBytes` UTF-8 bytes on a codepoint boundary. */
+function truncateToBytes(value, maxBytes) {
+    if (byteLength(value) <= maxBytes)
+        return value;
+    let out = "";
+    let used = 0;
+    for (const ch of value) {
+        const chBytes = byteLength(ch);
+        if (used + chBytes > maxBytes)
+            break;
+        out += ch;
+        used += chBytes;
+    }
+    return out;
+}
+/**
+ * The on-disk filename for one part: `<index>-<name>`, sanitized and truncated
+ * to the handler's byte cap. Satisfies the handler's `<index>-<name>` shape by
+ * construction.
+ */
+function partFilename(index, part) {
+    const prefix = `${String(index)}-`;
+    const budget = MAX_MAIL_PART_PATH_COMPONENT_BYTES - byteLength(prefix);
+    const rawName = part.filename ?? defaultPartName(part.contentType);
+    const safeName = truncateToBytes(sanitizePartName(rawName), budget);
+    return `${prefix}${safeName.length > 0 ? safeName : "part"}`;
+}
+/** A stable fallback name for a part with no filename, derived from its type. */
+function defaultPartName(contentType) {
+    const slash = contentType.indexOf("/");
+    const subtype = slash === -1 ? contentType : contentType.slice(slash + 1);
+    const safeSubtype = subtype.replace(/[^a-z0-9]+/gi, "") || "bin";
+    return `part.${safeSubtype}`;
+}
+function isTextType(contentType) {
+    return (contentType.startsWith("text/") ||
+        contentType === "application/json" ||
+        contentType === "application/vnd.interchange+json");
+}
+function mailPartRef(runId, messageSegment, filename) {
+    return `${REF_SCHEME}${encodeURIComponent(runId)}/${messageSegment}/${encodeURIComponent(filename)}`;
+}
+/**
+ * Parse a `mail-part:///` ref into its run id, message segment, and filename.
+ * The ref is persisted in the event log and re-read on resume, so it is
+ * treated as untrusted: the scheme must match, it must be exactly three
+ * non-empty segments, and no segment may traverse.
+ */
+function parseMailPartRef(ref) {
+    if (!ref.startsWith(REF_SCHEME)) {
+        throw new Error(`mail part reader: unrecognized ref ${JSON.stringify(ref)}`);
+    }
+    const rest = ref.slice(REF_SCHEME.length);
+    const segments = rest.split("/");
+    const malformed = `mail part reader: malformed ref ${JSON.stringify(ref)}`;
+    if (segments.length !== 3 ||
+        segments.some((s) => s.length === 0) ||
+        rest.includes("\\")) {
+        throw new Error(malformed);
+    }
+    // `runId` and `filename` were percent-encoded into the ref; `messageSegment`
+    // is stored encoded and matches the on-disk directory name verbatim.
+    let runId;
+    let filename;
+    try {
+        runId = decodeURIComponent(segments[0] ?? "");
+        filename = decodeURIComponent(segments[2] ?? "");
+    }
+    catch (cause) {
+        throw new Error(malformed, { cause });
+    }
+    const messageSegment = segments[1] ?? "";
+    // Reject traversal on the DECODED values too: a ref could encode `..`
+    // (`%2e%2e`) or a path separator (`%2f`, `%5c`) that only reveals itself
+    // after decoding, forming a compound traversal segment like `../..`.
+    const traverses = (s) => s === "." || s === ".." || s.includes("/") || s.includes("\\");
+    if ([runId, messageSegment, filename].some(traverses)) {
+        throw new Error(malformed);
+    }
+    return { runId, messageSegment, filename };
+}
+/**
+ * Commit a decoded message's parts and assemble the JSON-safe `Mail`. Every
+ * part's bytes are written in ONE prefix-preserving commit under the message's
+ * directory (write-once, atomic), and each part becomes a `MailPart` descriptor
+ * carrying its metadata, an opaque `ref`, and -- for a small UTF-8 text part --
+ * its decoded `text` inline.
+ */
+export async function commitMail(opts, messageId, decoded) {
+    const messageSegment = encodeMessageSegment(messageId);
+    const messagePrefix = `${WORKFLOW_RUN_RUNS_PREFIX}/${opts.runId}/${WORKFLOW_RUN_PARTS_DIR}/${messageSegment}/`;
+    const fresh = {};
+    const mailParts = decoded.parts.map((part, index) => {
+        const filename = partFilename(index, part);
+        fresh[`${messagePrefix}${filename}`] = part.content;
+        const descriptor = {
+            contentType: part.contentType,
+            ref: mailPartRef(opts.runId, messageSegment, filename),
+        };
+        if (part.filename !== undefined)
+            descriptor.filename = part.filename;
+        if (part.disposition !== undefined)
+            descriptor.disposition = part.disposition;
+        if (isTextType(part.contentType) &&
+            part.content.byteLength <= INLINE_TEXT_MAX_BYTES) {
+            descriptor.text = new TextDecoder("utf-8", { fatal: false }).decode(part.content);
+        }
+        return descriptor;
+    });
+    if (Object.keys(fresh).length > 0) {
+        try {
+            await opts.substrate.writeTreePreservingPrefix(opts.principal, opts.repoId, opts.ref, {
+                preservePrefix: messagePrefix,
+                merge: async (existing) => {
+                    const files = {};
+                    for (const [k, v] of existing)
+                        files[k] = v;
+                    for (const [k, v] of Object.entries(fresh))
+                        files[k] = v;
+                    return files;
+                },
+                message: `commit ${String(decoded.parts.length)} mail part(s) for message ${messageId} of run ${opts.runId}`,
+            });
+        }
+        catch (cause) {
+            const message = cause instanceof Error ? cause.message : String(cause);
+            // A path_violation is a shape rejection of this message's own (already
+            // sanitized) content: it is deterministic, so replaying the same bytes
+            // fails identically. Surface it as InvalidMailError so the caller drops
+            // the mail rather than retrying it forever as a transient fault.
+            if (message.startsWith("path_violation: ")) {
+                throw new InvalidMailError(message.slice("path_violation: ".length), {
+                    cause,
+                });
+            }
+            throw cause;
+        }
+    }
+    return {
+        headers: decoded.headers,
+        rawHeaders: decoded.rawHeaders,
+        parts: mailParts,
+    };
+}
+/**
+ * Construct the single mail-part reader for a deployment's workflow-run repo.
+ * `read` resolves any run's `MailPart.ref` to the committed bytes through a
+ * committed read pinned to the object store, so a cross-run read (a childflow
+ * or body step resolving a parent's part) never observes the lagging working
+ * tree.
+ */
+export function createMailPartReader(opts) {
+    return {
+        async read(ref) {
+            const { runId, messageSegment, filename } = parseMailPartRef(ref);
+            const dir = `${WORKFLOW_RUN_RUNS_PREFIX}/${runId}/${WORKFLOW_RUN_PARTS_DIR}/${messageSegment}`;
+            const reads = await opts.substrate.openCommittedReads(opts.principal, opts.repoId, opts.ref);
+            if (reads === null) {
+                throw new Error(`mail part reader: repo ${opts.repoId.id} ref ${opts.ref} has no committed tree; cannot resolve ${ref}`);
+            }
+            const entry = (await reads.listDir(dir)).find((e) => e.name === filename && e.type === "blob");
+            if (entry === undefined) {
+                throw new Error(`mail part reader: no committed part at ${dir}/${filename}`);
+            }
+            return reads.readBlobByOid(entry.oid);
+        },
+    };
+}

package/dist/adapters/repo-store.d.ts

@@ -1,5 +1,5 @@
 import type { Principal, RepoId, RepoStore as SubstrateRepoStore } from "@intx/hub-sessions/substrate";
-import type { RepoStore } from "@intx/workflow";
+import type { RepoStore, WorkflowEvent } from "@intx/workflow";
 export type WorkflowRunRepoStoreOpts = {
     /**
      * Substrate handle the adapter reads from and writes to. The caller
@@ -23,6 +23,18 @@ export type WorkflowRunRepoStoreOpts = {
      * shape the production wiring supplies.
      */
     principal: Principal;
+    /**
+     * Principal used for a control-plane cancel append (a batch that is
+     * entirely `CancelRequested`). The workflow-run kind handler requires a
+     * `CancelRequested` be signed by a `supervisor` principal -- a
+     * `workflow-process` principal may write run-body events but not a cancel.
+     * An in-process child runs under real supervisor authority, so its host
+     * supplies a supervisor principal here while run-body events keep their
+     * `workflow-process` attribution. Absent when the writer issues no
+     * in-process cancel, in which case a cancel fails loud at the push boundary
+     * rather than being silently mis-attributed.
+     */
+    controlPlanePrincipal?: Principal;
     /**
      * Events ref the adapter reads from and writes to. The workflow-run
      * repo layout pins all `runs/<runId>/events/` blobs under a single
@@ -37,3 +49,12 @@ export type WorkflowRunRepoStoreOpts = {
  * routing live in closure.
  */
 export declare function createWorkflowRunRepoStore(opts: WorkflowRunRepoStoreOpts): RepoStore;
+/**
+ * Translate a state-machine `WorkflowEvent` (using `kind` as the
+ * discriminator) into the on-disk envelope shape (`{seq, type,
+ * ...rest}`) the workflow-run kind handler validates and the
+ * substrate's `subscribeKind` helper filters on. Exported so the
+ * supervisor's terminal-commit path encodes a supervisor-authored
+ * `RunFailed` through the same single source of the on-disk shape.
+ */
+export declare function workflowEventToOnDisk(event: WorkflowEvent, seq: number): Record<string, unknown>;

package/dist/adapters/repo-store.js

@@ -27,10 +27,9 @@
 //     prospective tree via `validatePush`. Translated into a thrown
 //     Error carrying the handler's `reason` text. No retries.
 import { type } from "arktype";
-import { subscribeKind, WORKFLOW_RUN_EVENTS_FILE, splitCombinedEventLog, } from "@intx/hub-sessions/substrate";
+import { parseEventSeq, subscribeKind, WORKFLOW_RUN_EVENTS_FILE, splitCombinedEventLog, } from "@intx/hub-sessions/substrate";
 const RUNS_PREFIX = "runs";
 const EVENTS_DIR = "events";
-const EVENT_FILENAME_RE = /^(0|[1-9][0-9]*)\.json$/;
 /**
  * On-disk envelope shape committed under
  * `runs/<runId>/events/<seq>.json`. Carries the seq cross-check the
@@ -61,6 +60,7 @@ const ALL_WORKFLOW_EVENT_TYPES = [
     "AttemptScheduled",
     "SignalAwaited",
     "SignalReceived",
+    "SignalAwaitAbandoned",
     "TimerSet",
     "TimerFired",
     "CancelRequested",
@@ -95,66 +95,56 @@ export function createWorkflowRunRepoStore(opts) {
     };
 }
 async function readAllEventsForRun(opts, runId) {
-    const fs = await import("node:fs/promises");
-    const path = await import("node:path");
-    const dir = opts.substrate.getRepoDir(opts.repoId);
-    const runDir = path.join(dir, RUNS_PREFIX, runId);
+    // Read the committed tree through the substrate, never the working
+    // checkout under `getRepoDir`. `openCommittedReads` pins the ref to its
+    // tip commit and serves every read from the git object store, so an
+    // enumerate-then-read sequence is a single coherent snapshot even while
+    // a concurrent append re-materializes the checkout. The prior
+    // implementation read the working tree directly (raw readdir/readFile)
+    // and raced that materialization: a blob `readdir` had just enumerated
+    // could vanish before `readFile` on a contended filesystem, surfacing a
+    // spurious ENOENT. Reading under the substrate's per-repo write lock was
+    // the alternative considered and rejected -- it would serialize every
+    // read behind the single writer and couple read latency to write
+    // contention, whereas the pinned committed tree is lock-free and already
+    // consistent because every append lands as exactly one commit.
+    const reads = await opts.substrate.openCommittedReads(opts.principal, opts.repoId, opts.ref);
+    // Null mirrors the prior readdir-ENOENT contract: an uninitialised repo
+    // or an unresolved ref holds no runs at all.
+    if (reads === null)
+        return [];
+    const runDir = `${RUNS_PREFIX}/${runId}`;
+    const runChildren = await reads.listDir(runDir);
+    const decoder = new TextDecoder();
     const entries = [];
     // A terminated run is sealed into a single combined `events.jsonl`; an
-    // in-flight run keeps per-event `events/<seq>.json` files. The combined
-    // file's presence selects the read path; the two forms are mutually
-    // exclusive in a run directory.
-    let combinedRaw;
-    try {
-        combinedRaw = await fs.readFile(path.join(runDir, WORKFLOW_RUN_EVENTS_FILE), "utf8");
-    }
-    catch (cause) {
-        if (!isErrnoNotFound(cause))
-            throw cause;
-        combinedRaw = null;
-    }
-    if (combinedRaw !== null) {
-        // The two forms are mutually exclusive; a run carrying both is a
-        // botched seal, and silently reading only the combined file would
-        // mask it, so surface it instead.
-        let perEventPresent = false;
-        try {
-            await fs.access(path.join(runDir, EVENTS_DIR));
-            perEventPresent = true;
-        }
-        catch (cause) {
-            if (!isErrnoNotFound(cause))
-                throw cause;
-        }
-        if (perEventPresent) {
+    // in-flight run keeps per-event `events/<seq>.json` files. The two forms
+    // are mutually exclusive; a run carrying both is a botched seal, and
+    // silently reading only the combined file would mask it, so surface it.
+    const combined = runChildren.find((e) => e.type === "blob" && e.name === WORKFLOW_RUN_EVENTS_FILE);
+    const perEventDir = runChildren.find((e) => e.type === "tree" && e.name === EVENTS_DIR);
+    if (combined !== undefined) {
+        if (perEventDir !== undefined) {
             throw new Error(`workflow-runtime: run ${runId} carries both a combined ${WORKFLOW_RUN_EVENTS_FILE} and a per-event ${EVENTS_DIR}/ directory`);
         }
+        const combinedRaw = decoder.decode(await reads.readBlobByOid(combined.oid));
         for (const line of splitCombinedEventLog(combinedRaw)) {
             entries.push(parseEventEnvelope(line, `${opts.repoId.id}/${runId}/${WORKFLOW_RUN_EVENTS_FILE}`));
         }
         entries.sort((a, b) => a.seq - b.seq);
         return entries.map((e) => e.event);
     }
-    const eventsDir = path.join(runDir, EVENTS_DIR);
-    let filenames;
-    try {
-        filenames = await fs.readdir(eventsDir);
-    }
-    catch (cause) {
-        if (isErrnoNotFound(cause))
-            return [];
-        throw cause;
-    }
-    for (const name of filenames) {
-        const match = EVENT_FILENAME_RE.exec(name);
-        if (match === null)
+    // Per-event form. `listDir` on an absent or non-tree path returns the
+    // empty array, so a run with no events reads as the empty log.
+    const eventBlobs = await reads.listDir(`${runDir}/${EVENTS_DIR}`);
+    for (const child of eventBlobs) {
+        if (child.type !== "blob")
             continue;
-        const seqStr = match[1];
-        if (seqStr === undefined)
+        const seqFromName = parseEventSeq(child.name);
+        if (seqFromName === null)
             continue;
-        const seqFromName = Number.parseInt(seqStr, 10);
-        const raw = await fs.readFile(path.join(eventsDir, name), "utf8");
-        const source = `${opts.repoId.id}/${runId}/${EVENTS_DIR}/${name}`;
+        const raw = decoder.decode(await reads.readBlobByOid(child.oid));
+        const source = `${opts.repoId.id}/${runId}/${EVENTS_DIR}/${child.name}`;
         const entry = parseEventEnvelope(raw, source);
         if (entry.seq !== seqFromName) {
             throw new Error(`workflow-runtime: read ${source} body.seq ${String(entry.seq)} does not match filename seq ${String(seqFromName)}`);
@@ -213,9 +203,11 @@ function onDiskToWorkflowEvent(envelope) {
  * Translate a state-machine `WorkflowEvent` (using `kind` as the
  * discriminator) into the on-disk envelope shape (`{seq, type,
  * ...rest}`) the workflow-run kind handler validates and the
- * substrate's `subscribeKind` helper filters on.
+ * substrate's `subscribeKind` helper filters on. Exported so the
+ * supervisor's terminal-commit path encodes a supervisor-authored
+ * `RunFailed` through the same single source of the on-disk shape.
  */
-function workflowEventToOnDisk(event, seq) {
+export function workflowEventToOnDisk(event, seq) {
     const { kind, seq: _eventSeq, ...rest } = event;
     return { seq, type: kind, ...rest };
 }
@@ -241,9 +233,18 @@ async function appendBatchEvents(opts, runId, events) {
     const lastEvent = events[events.length - 1];
     if (lastEvent === undefined)
         throw new Error("unreachable");
+    // A control-plane cancel append (an isolated `CancelRequested`, which the
+    // kind handler requires be signed by a supervisor principal) is written under
+    // `controlPlanePrincipal` when the host supplied one; every other batch --
+    // including run-body events -- keeps the workflow-process `principal`. A mixed
+    // batch is never a cancel, so it stays on `principal`.
+    const principal = opts.controlPlanePrincipal !== undefined &&
+        events.every((event) => event.kind === "CancelRequested")
+        ? opts.controlPlanePrincipal
+        : opts.principal;
     let seqConflict = null;
     try {
-        await opts.substrate.writeTreePreservingPrefix(opts.principal, opts.repoId, opts.ref, {
+        await opts.substrate.writeTreePreservingPrefix(principal, opts.repoId, opts.ref, {
             preservePrefix: prefix,
             merge: async (existing) => {
                 // The runtime body emits events at `state.lastSeq + 1` and
@@ -256,13 +257,9 @@ async function appendBatchEvents(opts, runId, events) {
                 let priorLastSeq = 0;
                 for (const filepath of existing.keys()) {
                     const name = filepath.slice(prefix.length);
-                    const match = EVENT_FILENAME_RE.exec(name);
-                    if (match === null)
-                        continue;
-                    const seqStr = match[1];
-                    if (seqStr === undefined)
+                    const seq = parseEventSeq(name);
+                    if (seq === null)
                         continue;
-                    const seq = Number.parseInt(seqStr, 10);
                     if (seq > priorLastSeq)
                         priorLastSeq = seq;
                 }
@@ -336,9 +333,3 @@ async function* subscribeRun(opts, runId, subOpts) {
         yield { seq: entry.event.seq, event };
     }
 }
-function isErrnoNotFound(cause) {
-    if (cause === null || typeof cause !== "object")
-        return false;
-    const code = cause.code;
-    return code === "ENOENT";
-}