@forwardimpact/libeval 0.1.44 → 0.1.45

package/README.md

@@ -7,12 +7,188 @@ reproducible evidence.
 
 <!-- END:description -->
 
-## Getting Started
+`libeval` provides the runtime and tool surface for multi-LLM
+coordination: an agent talks to a supervisor, a facilitator chairs a
+team meeting, or a lead drives an asynchronous discussion across a
+human channel. Every conversation produces a structured NDJSON trace
+for analysis.
+
+## Modes
+
+| Mode          | Lead          | Participants  | Terminal tool          |
+| ------------- | ------------- | ------------- | ---------------------- |
+| `run`         | (none)        | one agent     | task completion        |
+| `supervise`   | `supervisor`  | one `agent`   | `Conclude`             |
+| `facilitate` | `facilitator` | N named       | `Conclude`             |
+| `discuss`     | `lead`        | N named       | `Adjourn` or `Recess`  |
+| `judge`       | `judge`       | (none)        | `Conclude`             |
+
+Every mode except `run` and `judge` shares one orchestration loop
+(`OrchestrationLoop`) and one tool surface (`Ask` / `Answer` /
+`Announce` / `RollCall`, plus a mode-specific terminal tool). The
+loop fires the lead's LLM, fans messages out to participants over an
+in-memory bus, wakes them when something lands, and emits the
+universal `{source, seq, event}` NDJSON envelope for every line.
+
+## The Ask / Answer protocol
+
+Coordination uses one async request/reply pattern with one piece of
+state per question — the `askId`. Every Ask returns immediately; the
+reply arrives later on the asker's inbox.
+
+### Ask
+
+```text
+Ask({ question, to? })  →  { askIds: [N, …] }
+```
+
+The handler registers a pending entry per addressee, posts the
+question on the bus, and returns immediately. Each pending entry is
+keyed by a numeric `askId`. Two Asks to the same addressee each get
+their own id, so they coexist without overwriting.
+
+Broadcast: omit `to` on a multi-participant lead's Ask to fan out to
+every other participant — the result `askIds` array has one entry
+per addressee.
+
+### Answer
+
+```text
+Answer({ message, askId? })  →  routed to the asker
+```
+
+The reply lands in the asker's bus inbox as
+`[answer#N] <participant>: <text>` on a later turn. `askId` is
+optional and the handler is forgiving:
+
+- **Provided + matches an ask owed by the caller** → routes the reply
+  to that specific asker.
+- **Provided but unknown or wrong addressee** → `isError` with a
+  pointed message. The caller tried to specify; we tell them why.
+- **Omitted + exactly one ask is owed to the caller** → auto-picks
+  that ask. (Forcing an Announce when the only owed ask is obvious
+  would be pedantic.)
+- **Omitted + 0 or many asks owed** → broadcasts as Announce so the
+  message still reaches every participant.
+
+### Announce
+
+```text
+Announce({ message })  →  broadcast, no reply expected
+```
+
+Lands on every other participant's queue as `[shared] <from>: <text>`.
+
+### Inbox format
+
+Every line a participant reads on a resume is one bus message rendered
+with its tag:
+
+```text
+[ask#42]     facilitator: What is your current condition?
+[answer#41]  agent-1:     We're at 7 out of 10.
+[shared]     agent-2:     FYI I'm switching to Bun 1.2.
+[system]     @orchestrator: You have an unanswered ask from facilitator (askId=42)…
+```
+
+The `[ask#N]` tag is what the participant quotes back in Answer's
+`askId` field.
+
+### Why async
+
+The lead can issue Asks, end its turn, and use the gap between turns
+for planning, reflection, or follow-up Asks while participants work
+in parallel. Nothing blocks the LLM thread waiting on a reply. The
+orchestrator wakes the lead whenever the inbox has new content.
+
+## The orchestration loop
+
+`OrchestrationLoop` runs one outer pattern for both the lead and each
+participant:
+
+1. Drain the bus queue, or wait for the first message.
+2. Run (first turn) or resume (every subsequent turn) the LLM with the
+   drained messages formatted as tagged lines.
+3. If the participant ended a turn with an unanswered Ask owed to it,
+   inject one synthetic reminder and resume once more. If still
+   unanswered, emit a `protocol_violation` event and cancel the
+   pending entry with a synthetic null answer so the asker unblocks.
+
+The lead's first turn starts with the task as its initial prompt;
+participants' first runs are triggered by their first inbound message.
+
+Termination flips two flags:
+
+- `ctx.concluded` — explicit `Conclude` / `Adjourn` / `Recess`. The
+  handler also cancels any in-flight Asks with a synthetic null so
+  askers see why their question won't be answered.
+- `stopped` — broader: also true on a lead error, an agent crash, or
+  any abort path. Loops watch `stopped`; `ctx.concluded` is only used
+  for the summary's `success` / `verdict`.
+
+## Tool surface, by role
+
+| Role         | Ask | Answer | Announce | RollCall | Conclude | Other                                |
+| ------------ | --- | ------ | -------- | -------- | -------- | ------------------------------------ |
+| Facilitator  | ✓   | ✓      | ✓        | ✓        | ✓        |                                      |
+| Fac. agent   | ✓   | ✓      | ✓        | ✓        |          |                                      |
+| Supervisor   | ✓   | ✓      | ✓        | ✓        | ✓        |                                      |
+| Sup. agent   | ✓   | ✓      | ✓        | ✓        |          |                                      |
+| Discuss lead | ✓   | ✓      | ✓        | ✓        |          | `RequestForComment`, `Recess`, `Adjourn` |
+| Discuss agt  | ✓   | ✓      | ✓        | ✓        |          |                                      |
+| Judge        |     |        |          |          | ✓        |                                      |
+
+Ask's `to` accepts a participant name on multi-participant roles
+(facilitator, discuss lead, all participants); supervise's
+`supervisor` / `agent` pair don't accept `to` because there's only
+one possible target.
+
+## Minimal example: a two-participant facilitator
 
 ```js
-import { createTraceCollector, createTraceQuery, createAgentRunner } from '@forwardimpact/libeval';
+import { createFacilitator, createRedactor } from "@forwardimpact/libeval";
+import { query } from "@anthropic-ai/claude-agent-sdk";
+
+const facilitator = createFacilitator({
+  facilitatorCwd: process.cwd(),
+  agentConfigs: [
+    { name: "alice", role: "explorer", agentProfile: "alice" },
+    { name: "bob",   role: "tester",   agentProfile: "bob" },
+  ],
+  query,
+  output: process.stdout,
+  redactor: createRedactor(),
+  facilitatorProfile: "improvement-coach",
+});
+
+const result = await facilitator.run("Run a kata storyboard meeting.");
+// result.success / result.turns / NDJSON trace on process.stdout
 ```
 
+The facilitator's LLM, started with that task, has access to `Ask`,
+`Answer`, `Announce`, `RollCall`, and `Conclude`. Alice and Bob each
+get `Ask`, `Answer`, `Announce`, `RollCall`. Every tool call, every
+message routed through the bus, and every orchestrator event becomes a
+line in the trace.
+
+## Trace format
+
+Every line is one JSON object with three fields:
+
+```json
+{ "source": "facilitator", "seq": 42, "event": { … } }
+```
+
+- `source` — the participant whose LLM produced the line, or
+  `orchestrator` for loop-level events (`session_start`, `agent_start`,
+  `protocol_violation`, `lead_turn_limit`, `summary`).
+- `seq` — monotonically increasing across the whole trace; useful for
+  reconstructing the wall-clock order across concurrent participants.
+- `event` — the SDK event verbatim, or the orchestrator event payload.
+
+`fit-trace` consumes this format. See the trace analysis guide for the
+full schema.
+
 ## Trace redaction
 
 `fit-eval run`, `fit-eval supervise`, and `fit-eval facilitate` redact
@@ -21,14 +197,37 @@ secrets in trace artifacts before they reach disk. Two layers compose:
 - **Env-var allowlist**, defaulting to `ANTHROPIC_API_KEY`, `GH_TOKEN`,
   `GITHUB_TOKEN`. The runtime values of these vars are replaced with
   `[REDACTED:env:NAME]` wherever they appear in tool inputs, tool
-  outputs, assistant text, or orchestrator summaries. Override the list
-  with `LIBEVAL_REDACTION_ENV_VARS=NAME1,NAME2,…` (replaces, not extends).
-- **Credential-shape patterns**, covering Anthropic API keys (`sk-ant-`),
-  GitHub PATs (`ghp_`), installation tokens (`ghs_`), OAuth tokens
-  (`gho_`), and fine-grained PATs (`github_pat_`). Pattern hits become
-  `[REDACTED:pattern:KIND]`.
-
-Redaction is on by default. To disable, set `LIBEVAL_REDACTION_DISABLED=1`
-— a stderr warning fires once per run. Never set this in CI on a public
-repository: workflow artifacts there are downloadable through the
-retention window.
+  outputs, assistant text, or orchestrator summaries. Override the
+  list with `LIBEVAL_REDACTION_ENV_VARS=NAME1,NAME2,…` (replaces, not
+  extends).
+- **Credential-shape patterns**, covering Anthropic API keys
+  (`sk-ant-`), GitHub PATs (`ghp_`), installation tokens (`ghs_`),
+  OAuth tokens (`gho_`), and fine-grained PATs (`github_pat_`).
+  Pattern hits become `[REDACTED:pattern:KIND]`.
+
+Redaction is on by default. To disable, set
+`LIBEVAL_REDACTION_DISABLED=1` — a stderr warning fires once per run.
+Never set this in CI on a public repository: workflow artifacts there
+are downloadable through the retention window.
+
+## Module map
+
+| Module                       | Purpose                                                                 |
+| ---------------------------- | ----------------------------------------------------------------------- |
+| `agent-runner.js`            | One Claude Agent SDK session; emits NDJSON via the redactor.            |
+| `message-bus.js`             | In-memory per-participant queues + `waitForMessages` Promise wakeup.    |
+| `orchestration-toolkit.js`   | Shared Ask / Answer / Announce / Conclude / RollCall handlers + builders. |
+| `orchestration-loop.js`      | Unified lead+participant loop; reminder/violation handling.             |
+| `facilitator.js`             | `Facilitator` class + factory + system prompts.                         |
+| `supervisor.js`              | `Supervisor` class + factory + system prompts.                          |
+| `discuss-tools.js`           | Discuss-only RequestForComment / Recess / Adjourn handlers + tool servers. |
+| `discusser.js`               | `Discusser` class + factory + system prompt + resume hydration.         |
+| `judge.js`                   | One-shot post-hoc verdict via `Conclude`.                               |
+| `trace-collector.js` / `trace-query.js` / `trace-github.js` | Trace ingestion / querying / GitHub-attachment helpers. |
+| `redaction.js`               | Env-var allowlist + credential-shape pattern redaction.                 |
+
+## Documentation
+
+- [Agent Evaluations Guide](https://www.forwardimpact.team/docs/libraries/agent-evaluations/index.md) — how to run an eval and read its trace.
+- [Agent Collaboration Guide](https://www.forwardimpact.team/docs/libraries/agent-collaboration/index.md) — supervise / facilitate / discuss in depth.
+- [Trace Analysis Guide](https://www.forwardimpact.team/docs/libraries/trace-analysis/index.md) — analysing NDJSON traces with `fit-trace`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@forwardimpact/libeval",
-  "version": "0.1.44",
+  "version": "0.1.45",
   "description": "Agent evaluation framework — prove whether agent changes improved outcomes with reproducible evidence.",
   "keywords": [
     "eval",

package/src/agent-runner.js

@@ -1,7 +1,7 @@
 /**
- * AgentRunner — runs a single Claude Agent SDK session and emits raw NDJSON
- * events to an output stream. Building block for both `fit-eval run` and
- * `fit-eval supervise`.
+ * AgentRunner — runs a single Claude Agent SDK session and emits raw
+ * NDJSON events to an output stream. Building block for `fit-eval run`,
+ * `fit-eval supervise`, `fit-eval facilitate`, and `fit-eval discuss`.
  *
  * Follows OO+DI: constructor injection, factory function, tests bypass factory.
  */
@@ -13,25 +13,6 @@ const DEFAULT_ALLOWED_TOOLS = ["Bash", "Read", "Glob", "Grep", "Write", "Edit"];
 // overridable — so a future caller can't accidentally reduce permissions.
 const PERMISSION_MODE = "bypassPermissions";
 
-function applyDefaults(deps) {
-  return {
-    cwd: deps.cwd,
-    query: deps.query,
-    output: deps.output,
-    model: deps.model ?? "claude-opus-4-7[1m]",
-    maxTurns: deps.maxTurns ?? 50,
-    allowedTools: deps.allowedTools ?? DEFAULT_ALLOWED_TOOLS,
-    onLine: deps.onLine ?? null,
-    onBatch: deps.onBatch ?? null,
-    batchSize: deps.batchSize ?? 3,
-    settingSources: deps.settingSources ?? [],
-    systemPrompt: deps.systemPrompt ?? null,
-    disallowedTools: deps.disallowedTools ?? [],
-    mcpServers: deps.mcpServers ?? null,
-    taskAmend: deps.taskAmend ?? null,
-  };
-}
-
 /** Run a single Claude Agent SDK session and emit raw NDJSON events to an output stream. */
 export class AgentRunner {
   /**
@@ -43,29 +24,38 @@ export class AgentRunner {
    * @param {number} [deps.maxTurns] - Maximum agentic turns; 0 means unlimited
    * @param {string[]} [deps.allowedTools] - Tools the agent may use
    * @param {function} [deps.onLine] - Callback invoked with each NDJSON line as it's produced
-   * @param {function} [deps.onBatch] - Async callback invoked with a batch of NDJSON lines at flush boundaries: every `batchSize` assistant text blocks, the terminal `result` message, and — on iterator crash/abort — once more in a final flush carrying any lines that never reached a boundary. Receives `(lines, { abort })` where calling `abort()` stops the in-flight SDK session via the AbortController. Optional; assignable at runtime so the Supervisor can swap it per turn.
-   * @param {number} [deps.batchSize] - Assistant text-block messages to accumulate before firing onBatch. Tool-only assistant messages ride along without counting. Default 3: the supervisor reviews the agent every three text turns instead of every turn. The terminal `result` always flushes regardless of count.
    * @param {string[]} [deps.settingSources] - SDK setting sources (e.g. ['project'] to load CLAUDE.md)
    * @param {string|object} [deps.systemPrompt] - SDK system prompt (string replaces default; {type:'preset', preset:'claude_code', append} appends)
    * @param {string[]} [deps.disallowedTools] - Tools to explicitly remove from the model's context
    * @param {Record<string, object>} [deps.mcpServers] - MCP server configs to pass to the SDK query
+   * @param {object} deps.redactor
    */
   constructor(deps) {
     if (!deps.cwd) throw new Error("cwd is required");
     if (!deps.query) throw new Error("query is required");
     if (!deps.output) throw new Error("output is required");
     if (!deps.redactor) throw new Error("redactor is required");
-    Object.assign(this, applyDefaults(deps));
+    this.cwd = deps.cwd;
+    this.query = deps.query;
+    this.output = deps.output;
     this.redactor = deps.redactor;
+    this.model = deps.model ?? "claude-opus-4-7[1m]";
+    this.maxTurns = deps.maxTurns ?? 50;
+    this.allowedTools = deps.allowedTools ?? DEFAULT_ALLOWED_TOOLS;
+    this.onLine = deps.onLine ?? null;
+    this.settingSources = deps.settingSources ?? [];
+    this.systemPrompt = deps.systemPrompt ?? null;
+    this.disallowedTools = deps.disallowedTools ?? [];
+    this.mcpServers = deps.mcpServers ?? null;
+    this.taskAmend = deps.taskAmend ?? null;
     this.sessionId = null;
-    this.buffer = [];
     /** @type {AbortController|null} */
     this.currentAbortController = null;
   }
 
   /**
    * Run a new agent session with the given task.
-   * @param {string} task - The task prompt
+   * @param {string} task
    * @returns {Promise<{success: boolean, text: string, sessionId: string|null, error: Error|null, aborted: boolean}>}
    */
   async run(task) {
@@ -87,7 +77,7 @@ export class AgentRunner {
 
   /**
    * Resume an existing session with a follow-up prompt.
-   * @param {string} prompt - The follow-up prompt
+   * @param {string} prompt
    * @returns {Promise<{success: boolean, text: string, sessionId: string|null, error: Error|null, aborted: boolean}>}
    */
   async resume(prompt) {
@@ -108,17 +98,16 @@ export class AgentRunner {
   }
 
   /**
-   * Build the options passed to every SDK query() call. Shared by run() and
-   * resume() so the agent's configuration — cwd, tools, prompt, setting
-   * sources, turn budget — is identical across the session's lifetime. Only
-   * resume() layers `resume: this.sessionId` on top.
+   * Build the options passed to every SDK query() call. Shared by run()
+   * and resume() so the agent's configuration — cwd, tools, prompt,
+   * setting sources, turn budget — is identical across the session's
+   * lifetime. Only resume() layers `resume: this.sessionId` on top.
    *
-   * SDK options are call-attached, not session-attached: the resumed call
-   * loads the prior conversation but otherwise uses whatever options this
-   * call passes. Omitting tool/prompt/setting options on resume causes the
-   * agent to silently lose its restrictions and persona between turns.
-   * @param {AbortController} abortController
-   * @returns {object}
+   * SDK options are call-attached, not session-attached: the resumed
+   * call loads the prior conversation but otherwise uses whatever
+   * options this call passes. Omitting tool/prompt/setting options on
+   * resume causes the agent to silently lose its restrictions and
+   * persona between turns.
    */
   #callOptions(abortController) {
     return {
@@ -139,59 +128,28 @@ export class AgentRunner {
   }
 
   /**
-   * Shared consumer for both `run()` and `resume()`. Iterates the SDK query
-   * iterator, mirroring every line to the output stream / buffer / onLine
-   * callback, and — when `onBatch` is set — flushes accumulated lines to it
-   * at coarse boundaries: every `batchSize` assistant text-block messages,
-   * and the terminal `result` message. Tool-only assistant messages still
-   * accumulate in the pending batch and ride along in the next flush, so
-   * the supervisor always sees the tool calls that led up to each text
-   * block. Raising `batchSize` above 1 is the knob that makes the mid-turn
-   * supervisor review less chatty — with the default of 3, the supervisor
-   * sees the agent in chunks of three text turns instead of every turn.
-   *
-   * Corollary: a turn that is *entirely* tool_use with no text blocks and
-   * then hits `result` produces exactly one flush at `result` regardless
-   * of how many tools ran. That is deliberate — the supervisor only needs
-   * to weigh in when the agent surfaces something text-like to react to.
-   *
-   * INVARIANT: the `await this.onBatch(...)` call below is the ONLY
-   * suspension point in this loop. While it is pending, no further lines
-   * are pulled from the SDK generator. The Supervisor relies on this — its
-   * onBatch callback flips `currentSource` to "supervisor" for the duration
-   * of its mid-turn LLM call, and the invariant guarantees no agent line
-   * can arrive concurrently and be mis-tagged.
-   *
-   * If the supervisor calls `abort()` from inside the callback, the next
-   * iteration of the for-await loop will throw. We catch the throw, check
-   * `currentAbortController.signal.aborted` (avoiding fragility around
-   * AbortError vs DOMException shapes), and report `aborted: true` so the
-   * caller can distinguish "supervisor asked us to stop" from a real error.
+   * Iterate the SDK query iterator, mirroring every message to the
+   * output stream and the `onLine` callback. Captures `sessionId` from
+   * the SDK's `system/init` message and tracks Skill invocations into
+   * `LIBEVAL_SKILL` for downstream metrics.
    *
-   * If the iterator throws before a flush boundary, any lines still in the
-   * pending batch would otherwise vanish without the supervisor seeing
-   * them. The `finally` block emits a terminal batch so the supervisor can
-   * observe the partial state (e.g. note a crash or react to an external
-   * abort). A throw from that final flush becomes the returned `error`
-   * only if no earlier error was captured — the original failure wins.
-   * @param {AsyncIterable<object>} iterator
-   * @returns {Promise<{success: boolean, text: string, sessionId: string|null, error: Error|null, aborted: boolean}>}
+   * If the iterator throws and we triggered the abort ourselves
+   * (`currentAbortController.signal.aborted`), we report `aborted:
+   * true`; otherwise the error propagates as `error`.
    */
   async #consumeQuery(iterator) {
     let text = "";
     let stopReason = null;
     let error = null;
     let aborted = false;
-    const state = { pendingBatch: [], assistantTextCount: 0 };
 
     try {
       for await (const message of iterator) {
-        this.#recordLine(message, state);
+        this.#recordLine(message);
         if (message.type === "result") {
           text = message.result ?? "";
           stopReason = message.subtype;
         }
-        await this.#maybeFlushBatch(message, state);
       }
     } catch (err) {
       if (this.currentAbortController?.signal.aborted) {
@@ -201,118 +159,28 @@ export class AgentRunner {
       }
     }
 
-    const flushErr = await this.#terminalFlush(state, { error, aborted });
-    if (flushErr && !error) error = flushErr;
-
-    const success = stopReason === "success";
-    return { success, text, sessionId: this.sessionId, error, aborted };
+    return {
+      success: stopReason === "success",
+      text,
+      sessionId: this.sessionId,
+      error,
+      aborted,
+    };
   }
 
-  /**
-   * Mirror a single SDK message to the output stream, buffer, onLine
-   * callback, and (when set) the pending-batch state. Also handles
-   * session id capture and text-block counting so `#consumeQuery` can
-   * stay within the complexity budget.
-   * @param {object} message
-   * @param {{pendingBatch: string[], assistantTextCount: number}} state
-   */
-  #recordLine(message, state) {
+  #recordLine(message) {
     const redacted = this.redactor.redactValue(message);
     const line = JSON.stringify(redacted);
     this.output.write(line + "\n");
-    this.buffer.push(line);
     if (this.onLine) this.onLine(line);
-    if (this.onBatch) state.pendingBatch.push(line);
 
-    // Session-id / text-block tracking reads the ORIGINAL message —
-    // these fields are not secret carriers, and the trackers rely on
-    // shape, not string contents.
     if (message.type === "system" && message.subtype === "init") {
       this.sessionId = message.session_id;
     }
-    if (message.type === "assistant") {
-      if (hasTextBlock(message)) state.assistantTextCount++;
-      trackSkillInvocation(message);
-    }
-  }
-
-  /**
-   * Terminal flush — only fires on the abnormal-end paths (iterator
-   * threw or was aborted mid-stream). Delivers any pending lines so the
-   * supervisor sees the partial state instead of losing the tail of
-   * the run. A natural-end iterator that simply ran out of messages
-   * without a `result` marker is treated as an incomplete stub (the
-   * real SDK always terminates with `result`) and its pending batch is
-   * not re-flushed. Returns an error thrown by the flush callback, or
-   * `null` if the flush succeeded or did not fire.
-   * @param {{pendingBatch: string[], assistantTextCount: number}} state
-   * @param {{error: Error|null, aborted: boolean}} outcome
-   * @returns {Promise<Error|null>}
-   */
-  async #terminalFlush(state, { error, aborted }) {
-    const loopEndedAbnormally = Boolean(error || aborted);
-    if (!loopEndedAbnormally) return null;
-    if (!this.onBatch || state.pendingBatch.length === 0) return null;
-    try {
-      const batchLines = state.pendingBatch.splice(0);
-      await this.onBatch(batchLines, {
-        abort: () => this.currentAbortController?.abort(),
-      });
-      return null;
-    } catch (flushErr) {
-      return flushErr;
-    }
-  }
-
-  /**
-   * Flush the pending batch to `onBatch` if either the batchSize threshold
-   * has been reached or the current message is the terminal `result`.
-   * Extracted so that `#consumeQuery` stays within the project's complexity
-   * budget — the flush is one cohesive unit of logic in its own right.
-   * @param {object} message
-   * @param {{pendingBatch: string[], assistantTextCount: number}} state
-   */
-  async #maybeFlushBatch(message, state) {
-    if (!this.onBatch) return;
-    const shouldFlush =
-      message.type === "result" || state.assistantTextCount >= this.batchSize;
-    if (!shouldFlush) return;
-    state.assistantTextCount = 0;
-    const batchLines = state.pendingBatch.splice(0);
-    await this.onBatch(batchLines, {
-      abort: () => this.currentAbortController?.abort(),
-    });
-  }
-
-  /**
-   * Drain buffered output lines. Used by Supervisor to tag and re-emit lines.
-   * @returns {string[]}
-   */
-  drainOutput() {
-    const lines = [...this.buffer];
-    this.buffer = [];
-    return lines;
+    if (message.type === "assistant") trackSkillInvocation(message);
   }
 }
 
-/**
- * Whether an SDK assistant message contains at least one text block.
- * Only text-block messages count toward the `batchSize` threshold — tool-only
- * assistant messages accumulate silently into the pending batch and ride along
- * in the next flush, keeping supervisor LLM cost bounded. Exported so the mock
- * runner can mirror the real flush predicate without duplicating the logic.
- * @param {object} message
- * @returns {boolean}
- */
-export function hasTextBlock(message) {
-  const content = message.message?.content ?? message.content;
-  if (!Array.isArray(content)) return false;
-  for (const block of content) {
-    if (block.type === "text" && block.text) return true;
-  }
-  return false;
-}
-
 function trackSkillInvocation(message) {
   const content = message.message?.content ?? message.content;
   if (!Array.isArray(content)) return;
@@ -327,11 +195,7 @@ function trackSkillInvocation(message) {
   }
 }
 
-/**
- * Factory function — wires real dependencies.
- * @param {object} deps - Same as AgentRunner constructor
- * @returns {AgentRunner}
- */
+/** Factory function — wires real dependencies. */
 export function createAgentRunner(deps) {
   return new AgentRunner(deps);
 }

package/src/benchmark/runner.js

@@ -3,7 +3,7 @@
  *
  * Phases per (task, runIndex):
  *   1. WorkdirManager.start → seed CWD + run pre-flight probe
- *   2. Supervisor relay (agent + supervisor) → produce traces + submission
+ *   2. Supervisor session (agent + supervisor) → produce traces + submission
  *   3. Scorer.runScoring → exit-code-driven verdict via fd-3 NDJSON
  *   4. Judge.runJudge → Conclude-driven verdict mapped to pass/fail
  *   5. WorkdirManager.teardown → process-group cleanup
@@ -272,7 +272,7 @@ export class BenchmarkRunner {
   }
 
   /**
-   * Run the agent-under-test via a Supervisor relay. The supervisor writes
+   * Run the agent-under-test under a Supervisor. The supervisor writes
    * a combined tagged NDJSON trace; after the session we split it into
    * agent.ndjson and supervisor.ndjson and extract cost/turns/submission.
    */

package/src/commands/supervise.js

@@ -53,7 +53,9 @@ export function parseSuperviseOptions(values) {
 }
 
 /**
- * Supervise command — run two agents in a relay loop via the Claude Agent SDK.
+ * Supervise command — run one agent under a supervisor via the
+ * orchestration loop. The supervisor delegates work through Ask, sees
+ * each reply on its next turn, and ends with Conclude.
  *
  * Usage: fit-eval supervise [options]
  *