@forwardimpact/libeval 0.1.44 → 0.1.46

package/README.md

@@ -7,28 +7,193 @@ reproducible evidence.
 
 <!-- END:description -->
 
-## Getting Started
+`libeval` provides the runtime and tool surface for multi-LLM coordination —
+an agent talks to a supervisor, a facilitator chairs a meeting, a lead drives
+an asynchronous discussion — plus a CLI suite that runs evals, queries the
+traces they produce, and edits skill files under controlled conditions.
+
+## CLIs
+
+| CLI             | Purpose                                                                |
+| --------------- | ---------------------------------------------------------------------- |
+| `fit-eval`      | Run agents in `run`/`supervise`/`facilitate`/`discuss` subcommands.    |
+| `fit-trace`     | Download, query, and analyze NDJSON traces produced by `fit-eval`.     |
+| `fit-benchmark` | Run task families for N runs each and aggregate pass@k.                |
+| `fit-selfedit`  | Write stdin to `.claude/**` paths, gated by settings.json + branch.    |
+
+`fit-eval`'s subcommands share one orchestration loop and one async tool
+surface, below. The `judge` role is a profile passed to `supervise`.
+
+## Modes
+
+| Mode         | Lead          | Participants  | Terminal tool          |
+| ------------ | ------------- | ------------- | ---------------------- |
+| `run`        | (none)        | one agent     | task completion        |
+| `supervise`  | `supervisor`  | one `agent`   | `Conclude`             |
+| `facilitate` | `facilitator` | N named       | `Conclude`             |
+| `discuss`    | `lead`        | N named       | `Adjourn` or `Recess`  |
+| `judge`      | `judge`       | (none)        | `Conclude`             |
+
+`run` and `judge` are one-shot. The other three share `OrchestrationLoop`
+plus an async Ask/Answer/Announce/RollCall tool surface; the loop fans
+messages out over an in-memory bus and emits a `{source, seq, event}`
+NDJSON envelope for every line.
+
+## Async Ask / Answer / Announce
+
+```text
+Ask({ question, to? })       →  { askIds: [N, …] }
+Answer({ message, askId? })  →  routed to the asker
+Announce({ message })        →  broadcast, no reply expected
+```
+
+Every Ask returns immediately and registers a pending entry keyed by an
+`askId`. The reply arrives later on the asker's inbox as `[answer#N]
+<participant>: <text>`. Broadcast: omit `to` on a multi-participant
+lead. Answer's `askId` is optional — the handler is forgiving:
+
+- **Provided + matches an ask owed by the caller** → routes to that asker.
+- **Provided but unknown or wrong addressee** → `isError` with a pointed message.
+- **Omitted + exactly one ask owed to the caller** → auto-picks it.
+- **Omitted + 0 or many asks owed** → broadcasts as Announce.
+
+Inbox lines on resume:
+
+```text
+[ask#42]     facilitator: What is your current condition?
+[answer#41]  agent-1:     We're at 7 out of 10.
+[shared]     agent-2:     FYI I'm switching to Bun 1.2.
+[system]     @orchestrator: You have an unanswered ask from facilitator (askId=42)…
+```
+
+Async means the lead can issue Asks, end its turn, and plan in the gap
+while participants work in parallel — nothing blocks the LLM thread.
+
+## Orchestration loop
+
+Each participant drains the bus (or waits), runs/resumes the LLM with
+drained messages as tagged lines, and on an unanswered owed Ask injects
+one synthetic reminder before emitting `protocol_violation` and
+unblocking the asker with a synthetic null answer.
+
+Termination uses two flags. `ctx.concluded` is explicit
+`Conclude`/`Adjourn`/`Recess` — also cancels in-flight Asks so askers
+see why their question won't be answered. `stopped` is broader: lead
+error, agent crash, abort path. Loops watch `stopped`; `ctx.concluded`
+only feeds the summary's `success`/`verdict`.
+
+## Tool surface, by role
+
+| Role         | Ask | Answer | Announce | RollCall | Conclude | Other                                    |
+| ------------ | --- | ------ | -------- | -------- | -------- | ---------------------------------------- |
+| Facilitator  | ✓   | ✓      | ✓        | ✓        | ✓        |                                          |
+| Fac. agent   | ✓   | ✓      | ✓        | ✓        |          |                                          |
+| Supervisor   | ✓   | ✓      | ✓        | ✓        | ✓        |                                          |
+| Sup. agent   | ✓   | ✓      | ✓        | ✓        |          |                                          |
+| Discuss lead | ✓   | ✓      | ✓        | ✓        |          | `RequestForComment`, `Recess`, `Adjourn` |
+| Discuss agt  | ✓   | ✓      | ✓        | ✓        |          |                                          |
+| Judge        |     |        |          |          | ✓        |                                          |
+
+Ask's `to` accepts a participant name on multi-participant roles
+(facilitator, discuss lead, all participants). The supervise pair has
+only one possible target so `to` is rejected there.
+
+## Minimal example: two-participant facilitator
 
 ```js
-import { createTraceCollector, createTraceQuery, createAgentRunner } from '@forwardimpact/libeval';
+import { createFacilitator, createRedactor } from "@forwardimpact/libeval";
+import { query } from "@anthropic-ai/claude-agent-sdk";
+
+const facilitator = createFacilitator({
+  facilitatorCwd: process.cwd(),
+  agentConfigs: [
+    { name: "alice", role: "explorer", agentProfile: "alice" },
+    { name: "bob",   role: "tester",   agentProfile: "bob" },
+  ],
+  query,
+  output: process.stdout,
+  redactor: createRedactor(),
+  facilitatorProfile: "improvement-coach",
+});
+
+const result = await facilitator.run("Run a kata storyboard meeting.");
+// result.success / result.turns / NDJSON trace on process.stdout
+```
+
+The facilitator gets `Ask`/`Answer`/`Announce`/`RollCall`/`Conclude`;
+each agent gets the same minus `Conclude`. Every tool call, bus
+message, and orchestrator event becomes one trace line.
+
+## Trace format and redaction
+
+Each line is `{ "source": "<participant|orchestrator>", "seq": N, "event":
+{…} }`. `seq` is monotonic across the whole trace; `orchestrator` emits
+`session_start`, `agent_start`, `protocol_violation`, `lead_turn_limit`,
+and `summary`. `event` is the SDK event verbatim or the orchestrator
+payload. `fit-trace` consumes this format.
+
+Redaction is on by default for `fit-eval run`/`supervise`/`facilitate`
+and composes two layers:
+
+- **Env-var allowlist** — `ANTHROPIC_API_KEY`, `GH_TOKEN`, `GITHUB_TOKEN`
+  by default; override with `LIBEVAL_REDACTION_ENV_VARS=NAME1,…`
+  (replaces, not extends). Runtime values become `[REDACTED:env:NAME]`
+  everywhere they appear.
+- **Credential-shape patterns** — `sk-ant-`, `ghp_`, `ghs_`, `gho_`,
+  `github_pat_`. Hits become `[REDACTED:pattern:KIND]`.
+
+Set `LIBEVAL_REDACTION_DISABLED=1` to disable (one stderr warning per
+run). Never on CI for a public repo — workflow artifacts are
+downloadable through retention.
+
+## Module map
+
+| Module                                                      | Purpose                                                              |
+| ----------------------------------------------------------- | -------------------------------------------------------------------- |
+| `agent-runner.js`                                           | One Claude Agent SDK session; emits NDJSON via the redactor.         |
+| `message-bus.js`                                            | Per-participant queues + `waitForMessages` Promise wakeup.           |
+| `orchestration-toolkit.js`                                  | Shared Ask/Answer/Announce/Conclude/RollCall handlers + builders.    |
+| `orchestration-loop.js`                                     | Unified lead+participant loop; reminder/violation handling.          |
+| `facilitator.js` / `supervisor.js` / `discusser.js` / `judge.js` | Per-mode class + factory + system prompt.                       |
+| `discuss-tools.js`                                          | Discuss-only `RequestForComment`/`Recess`/`Adjourn`.                 |
+| `trace-collector.js` / `trace-query.js` / `trace-github.js` | Trace ingestion / querying / GitHub-attachment helpers.              |
+| `redaction.js`                                              | Env-var allowlist + credential-shape pattern redaction.              |
+
+## fit-selfedit
+
+A narrow, audited bypass for sessions where `Edit`/`Write` (and bash
+writes) are blocked against paths the project's own allowlist permits —
+see [#1162](https://github.com/forwardimpact/monorepo/issues/1162) and
+[#441](https://github.com/forwardimpact/monorepo/issues/441) for the
+original episodes. Reads stdin, writes the target, exits 0 / 2
+(safeguard violation) / 1 (I/O error).
+
+```sh
+echo "<content>" | bunx fit-selfedit <path>
 ```
 
-## Trace redaction
-
-`fit-eval run`, `fit-eval supervise`, and `fit-eval facilitate` redact
-secrets in trace artifacts before they reach disk. Two layers compose:
-
-- **Env-var allowlist**, defaulting to `ANTHROPIC_API_KEY`, `GH_TOKEN`,
-  `GITHUB_TOKEN`. The runtime values of these vars are replaced with
-  `[REDACTED:env:NAME]` wherever they appear in tool inputs, tool
-  outputs, assistant text, or orchestrator summaries. Override the list
-  with `LIBEVAL_REDACTION_ENV_VARS=NAME1,NAME2,…` (replaces, not extends).
-- **Credential-shape patterns**, covering Anthropic API keys (`sk-ant-`),
-  GitHub PATs (`ghp_`), installation tokens (`ghs_`), OAuth tokens
-  (`gho_`), and fine-grained PATs (`github_pat_`). Pattern hits become
-  `[REDACTED:pattern:KIND]`.
-
-Redaction is on by default. To disable, set `LIBEVAL_REDACTION_DISABLED=1`
-— a stderr warning fires once per run. Never set this in CI on a public
-repository: workflow artifacts there are downloadable through the
-retention window.
+Two safeguards, checked in order:
+
+1. **Settings-allow.** Walk upward from the target with
+   [`Finder.findUpward`](../libutil/src/finder.js) to find the nearest
+   `.claude/settings.json`. The target relative to its grandparent
+   directory must match at least one `Edit(<glob>)` rule in
+   `permissions.allow[]` (matched with
+   [`minimatch`](https://github.com/isaacs/minimatch), `dot: true`).
+   Settings.json is the single source of truth — widen the project
+   allowlist and the CLI follows. Traversal like `.claude/../README.md`
+   is rejected as a side effect: `path.resolve` collapses `..` first,
+   then the resolved path tests against the rules.
+
+2. **Branch scope.** `git rev-parse --abbrev-ref HEAD` must not be
+   `HEAD` (detached) or `main`. Edits ride a feature branch through
+   whatever merge gates the project has configured.
+
+Failure messages name the safeguard that rejected; safeguard 1 also
+lists the `Edit()` rules that were tried.
+
+## Documentation
+
+- [Agent Evaluations Guide](https://www.forwardimpact.team/docs/libraries/agent-evaluations/index.md) — how to run an eval and read its trace.
+- [Agent Collaboration Guide](https://www.forwardimpact.team/docs/libraries/agent-collaboration/index.md) — supervise / facilitate / discuss in depth.
+- [Trace Analysis Guide](https://www.forwardimpact.team/docs/libraries/trace-analysis/index.md) — analysing NDJSON traces with `fit-trace`.

package/bin/fit-selfedit.js

@@ -0,0 +1,162 @@
+#!/usr/bin/env node
+/**
+ * fit-selfedit — write stdin to a path that .claude/settings.json
+ * permits Edit on, while on a non-main git branch. See
+ * libraries/libeval/README.md § fit-selfedit for the full rationale.
+ */
+
+import { existsSync, readFileSync, writeFileSync } from "node:fs";
+import fsPromises from "node:fs/promises";
+import { parseArgs } from "node:util";
+import { resolve, relative, dirname } from "node:path";
+import { execFileSync } from "node:child_process";
+
+import { Finder } from "@forwardimpact/libutil";
+import { minimatch } from "minimatch";
+
+const HELP = `fit-selfedit — write stdin to a settings.json-allowed path on a non-main branch.
+
+Usage:
+  echo content | fit-selfedit <path>
+  fit-selfedit <path> < input.txt
+
+Safeguards (checked in order):
+  1. The nearest .claude/settings.json must contain an Edit(<glob>) rule
+     in permissions.allow[] that resolves to the target path.
+  2. HEAD must not be detached and the current branch must not be 'main'.
+
+Exit codes:
+  0  wrote the file
+  2  safeguard violation (no settings.json, no matching Edit rule, on
+     main, detached HEAD, missing parent directory, TTY stdin)
+  1  unexpected I/O error
+
+Why this exists:
+  Some session harnesses block Edit/Write (and interactive bash writes)
+  on .claude/skills/**, even when the project allowlist permits them.
+  This CLI is a narrow, audited bypass: a subprocess write that still
+  has to clear the project allowlist and the normal merge gates.
+`;
+
+function fail(message) {
+  process.stderr.write(`fit-selfedit: ${message}\n`);
+  process.exit(2);
+}
+
+const { values, positionals } = parseArgs({
+  options: {
+    help: { type: "boolean", short: "h" },
+    version: { type: "boolean" },
+  },
+  allowPositionals: true,
+});
+
+if (values.help) {
+  process.stdout.write(HELP);
+  process.exit(0);
+}
+
+if (values.version) {
+  const pkg = JSON.parse(
+    readFileSync(new URL("../package.json", import.meta.url), "utf8"),
+  );
+  process.stdout.write(`${pkg.version}\n`);
+  process.exit(0);
+}
+
+const [targetArg, ...extra] = positionals;
+if (!targetArg) fail("missing <path> (try --help)");
+if (extra.length > 0) fail(`unexpected extra arguments: ${extra.join(" ")}`);
+
+const absoluteTarget = resolve(process.cwd(), targetArg);
+
+// Safeguard 1: settings.json must grant Edit() on this path.
+const settingsPath = new Finder(fsPromises, { debug() {} }).findUpward(
+  dirname(absoluteTarget),
+  ".claude/settings.json",
+  20,
+);
+if (!settingsPath) {
+  fail(
+    `no .claude/settings.json found walking upward from ${dirname(absoluteTarget)}`,
+  );
+}
+
+const projectRoot = dirname(dirname(settingsPath));
+const relativeTarget = relative(projectRoot, absoluteTarget);
+
+let settings;
+try {
+  settings = JSON.parse(readFileSync(settingsPath, "utf8"));
+} catch (err) {
+  fail(`failed to parse ${settingsPath}: ${err.message}`);
+}
+
+const allowRules = settings?.permissions?.allow;
+if (!Array.isArray(allowRules)) {
+  fail(`${settingsPath} has no permissions.allow[] array`);
+}
+
+const editPatterns = allowRules
+  .filter((rule) => typeof rule === "string")
+  .map((rule) => rule.match(/^Edit\((.+)\)$/)?.[1])
+  .filter(Boolean);
+
+if (editPatterns.length === 0) {
+  fail(`${settingsPath} has no Edit() rules in permissions.allow[]`);
+}
+
+const matchedPattern = editPatterns.find((pattern) =>
+  minimatch(relativeTarget, pattern, { dot: true }),
+);
+if (!matchedPattern) {
+  fail(
+    `no Edit() rule in ${relative(projectRoot, settingsPath)} matches '${relativeTarget}' ` +
+      `(tried: ${editPatterns.map((p) => `Edit(${p})`).join(", ")})`,
+  );
+}
+
+// Safeguard 2: branch must not be main and HEAD must not be detached.
+let branch;
+try {
+  branch = execFileSync("git", ["rev-parse", "--abbrev-ref", "HEAD"], {
+    stdio: ["ignore", "pipe", "pipe"],
+    encoding: "utf8",
+  }).trim();
+} catch {
+  fail("failed to read current git branch (not inside a git repository?)");
+}
+
+if (branch === "HEAD") {
+  fail("HEAD is detached — refusing (check out a non-main branch first)");
+}
+if (branch === "main") {
+  fail("refusing to write while on branch 'main' — switch to a feature branch");
+}
+
+const parent = dirname(absoluteTarget);
+if (!existsSync(parent)) {
+  fail(`parent directory '${relative(projectRoot, parent)}' does not exist`);
+}
+
+if (process.stdin.isTTY) {
+  fail(
+    "stdin is a TTY — pipe content in (e.g. `echo … | fit-selfedit <path>`)",
+  );
+}
+
+const chunks = [];
+for await (const chunk of process.stdin) chunks.push(chunk);
+const content = Buffer.concat(chunks);
+
+try {
+  writeFileSync(absoluteTarget, content);
+} catch (err) {
+  process.stderr.write(`fit-selfedit: write failed: ${err.message}\n`);
+  process.exit(1);
+}
+
+process.stderr.write(
+  `fit-selfedit: wrote ${content.length} byte${content.length === 1 ? "" : "s"} to ${relativeTarget} ` +
+    `(matched Edit(${matchedPattern}), branch ${branch})\n`,
+);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@forwardimpact/libeval",
-  "version": "0.1.44",
+  "version": "0.1.46",
   "description": "Agent evaluation framework — prove whether agent changes improved outcomes with reproducible evidence.",
   "keywords": [
     "eval",
@@ -33,12 +33,14 @@
     ".": "./src/index.js",
     "./bin/fit-eval.js": "./bin/fit-eval.js",
     "./bin/fit-trace.js": "./bin/fit-trace.js",
-    "./bin/fit-benchmark.js": "./bin/fit-benchmark.js"
+    "./bin/fit-benchmark.js": "./bin/fit-benchmark.js",
+    "./bin/fit-selfedit.js": "./bin/fit-selfedit.js"
   },
   "bin": {
     "fit-eval": "./bin/fit-eval.js",
     "fit-trace": "./bin/fit-trace.js",
-    "fit-benchmark": "./bin/fit-benchmark.js"
+    "fit-benchmark": "./bin/fit-benchmark.js",
+    "fit-selfedit": "./bin/fit-selfedit.js"
   },
   "files": [
     "src/**/*.js",
@@ -53,7 +55,9 @@
     "@forwardimpact/libcli": "^0.1.0",
     "@forwardimpact/libconfig": "^0.1.0",
     "@forwardimpact/libtelemetry": "^0.1.22",
+    "@forwardimpact/libutil": "^0.1.0",
     "jmespath": "^0.16.0",
+    "minimatch": "^10.0.0",
     "zod": "^4.4.3"
   },
   "devDependencies": {