devlyn-cli 2.7.0 → 2.8.0

package/AGENTS.md

@@ -2,6 +2,10 @@
 
 devlyn-cli installs `/devlyn:ideate` (optional) and `/devlyn:resolve` (required) into Claude Code, plus the contract below. Codex CLI reads this file when invoked inside a project that has it. The principles are non-negotiable on every change.
 
+## North Star
+
+This contract serves one goal: any capable engine — Claude, GPT/Codex, or a future adapter-equipped model — takes a user's intent (prompt, spec, or queue entry) end-to-end to shipped, engineer-quality software, hands-free, with consistent quality across engines. The harness must measurably out-earn bare prompting, and every added layer (pair mode, probes, gates) must out-earn the layer below it. When rules below conflict or feel ambiguous in context, resolve toward this goal.
+
 ## Core principles
 
 Seven rules govern every change. Cite them by name when a decision touches one.
@@ -32,6 +36,10 @@ ideate (optional)  ->  resolve  ->  ship
 
 Each skill's `SKILL.md` is the source of truth for flags and workflow. Do not duplicate.
 
+Engine roles: orchestrator = whichever CLI the user opened (this contract is symmetric with CLAUDE.md; measured status 2026-07-05: Claude Code + omp run the full phase-gated pipeline; in the iter-0061 minimal-repo trivial-add shape on this machine, Codex CLI also ran it when this AGENTS.md was present in the project (ordinary invocation, 4/4); without project AGENTS.md, the same shape silently skipped the pipeline (iter-0040 F6, 4/4). If you are Codex orchestrating `/devlyn:resolve`, the phase machinery is mandatory regardless of task size; if you will not run it, say so explicitly and stop — do not silently degrade to ad-hoc execution); executor (PLAN/IMPLEMENT/CLEANUP) defaults to `claude`, overridable per run with `--engine <name>` or durably via machine-local `.devlyn/engines.json` `{"executor": "<name>", "pair_judge_priority": ["<name>", ...]}`; pair judge = first available OTHER engine. Pins fail closed (`BLOCKED:<engine>-unavailable` / `BLOCKED:invalid-engine-config`); new engines plug in by shipping `_shared/adapters/<name>.md`. `/devlyn:engines` (no args) shows the role table + detected engines; `executor <name>` / `pair <name>,...` / `clear` manage the pins.
+
+Conversational handoff + loop engineering (default entry): the orchestrating model — not the user — invokes the skills. `/devlyn:queue` fronts the intent queue (`docs/specs/queue.md`): no args = status, `add <intent>`, `drain` = serial unattended drain per the contract below. Large agreed work is written to `docs/specs/<id>/spec.md` (always a spec file; `--goal-file` is trivial/medium-only), summarized once for user review, then run hands-free via `--spec`. Outer loop per task: findings-backed verdicts (NEEDS_WORK, verify/build-gate exhaustion) get spec amendment + re-run, max 3 iterations; infrastructure / invalid-input / engine-availability / implement-empty BLOCKED verdicts surface immediately. Unattended queue drain (`docs/specs/queue.md`, strictly serial): assumptions may only take scope-narrowing, reversible, non-user-visible defaults — material ambiguity marks the item `[F] needs-review` and the drain continues.
+
 ## Subtractive-First Editing
 
 Before writing any change, answer in this order:
@@ -56,7 +64,7 @@ Default mode is execution toward the user's stated goal. Do not drift.
 Refuse these patterns:
 
 - Unrequested work ("while here, also fix..."). Pre-existing dead code → mention only, do NOT delete. Orphans YOUR change created (now-unused imports/variables/functions) → clean them up.
-- Tangential cleanup in files the task does not require. Match existing style even if you'd write it differently; do NOT touch comments, formatting, or code orthogonal to your real change — silent side-effects on neighboring lines are the most common Karpathy-observed regression class.
+- Tangential cleanup in files the task does not require. Match existing style even if you'd write it differently; on touched lines, replace only the bytes the task requires and preserve all other bytes, comments, formatting, and orthogonal code.
 - Speculative robustness for cases not observed in production, tests, findings, or the spec.
 - Mid-flight re-scoping without user approval.
 - Curiosity exploration that is not on the critical path.

package/CLAUDE.md

@@ -2,6 +2,10 @@
 
 devlyn-cli installs `/devlyn:ideate` (optional) and `/devlyn:resolve` (required) into Claude Code, plus the contract below. These principles are non-negotiable on every change — yours and any sub-agent's.
 
+## North Star
+
+This contract serves one goal: any capable engine — Claude, GPT/Codex, or a future adapter-equipped model — takes a user's intent (prompt, spec, or queue entry) end-to-end to shipped, engineer-quality software, hands-free, with consistent quality across engines. The harness must measurably out-earn bare prompting, and every added layer (pair mode, probes, gates) must out-earn the layer below it. When rules below conflict or feel ambiguous in context, resolve toward this goal.
+
 ## Core principles
 
 Seven rules govern every change. Cite them by name when a decision touches one.
@@ -24,22 +28,35 @@ The runtime sub-agent contract below (Subtractive-first / Goal-locked / No-worka
 
 ## Quick Start
 
-Two skills cover the full cycle. `/devlyn:ideate` is OPTIONAL; `/devlyn:resolve` is REQUIRED; `/devlyn:design-ui` is also REQUIRED as the creative UI exploration surface. **Both pipeline skills default to `--engine claude`** for PLAN/IMPLEMENT. Codex BUILD/IMPLEMENT and PLAN-pair remain research-only, but `/devlyn:resolve` VERIFY has conditional-default pair-JUDGE when its `SKILL.md` trigger policy fires. Pass `--engine codex` explicitly to opt into the broader research path. If a selected or conditionally required engine is unavailable, the run stops with `BLOCKED:<engine>-unavailable` and setup guidance.
+Two skills cover the full cycle. `/devlyn:ideate` is OPTIONAL; `/devlyn:resolve` is REQUIRED; `/devlyn:design-ui` is also REQUIRED as the creative UI exploration surface. Engine selection follows the role map below.
 
 1. `/devlyn:ideate` (optional) — unstructured idea → `docs/specs/<id>/spec.md` + `spec.expected.json`. Modes: default Q&A, `--quick` (autonomous-pipeline-safe), `--from-spec <path>`, `--project`.
 2. `/devlyn:resolve` — hands-free pipeline for any coding task. Free-form goal, `--spec <path>`, or `--verify-only <diff> --spec <path>`. Phases: PLAN → IMPLEMENT → BUILD_GATE → CLEANUP → VERIFY (fresh subagent, findings-only).
 
 Each skill's `SKILL.md` is the source of truth for its flags and workflow — don't duplicate them here.
 
-### When to use which
+### Engine roles — auto-detected, manually pinnable
+
+| Role | Default | Manual override |
+|---|---|---|
+| Orchestrator — conversation, handoff, loop driving | whichever CLI you open (contract is symmetric: CLAUDE.md ↔ AGENTS.md). Measured status 2026-07-05: Claude Code + omp run the full phase-gated pipeline; in the iter-0061 minimal-repo trivial-add shape on this machine, Codex CLI also ran it when the project carried the devlyn AGENTS.md (ordinary invocation, 4/4); without project AGENTS.md, the same shape silently skipped the pipeline (iter-0040 F6, 4/4) | switch CLIs; the file artifacts (spec/queue/state) carry over |
+| Executor — PLAN/IMPLEMENT/CLEANUP + primary VERIFY judge | `claude` | `--engine <name>` per run, or `/devlyn:engines executor <name>` (durable pin) |
+| Pair judge — VERIFY pair-JUDGE, risk probes | first available OTHER engine (claude↔codex) | `/devlyn:engines pair <name>,...`; `--no-pair` opts out |
+
+`/devlyn:engines` with no args shows the current role table, detected engines, and how to pin or clear — the pins live in `.devlyn/engines.json`.
+
+`.devlyn/engines.json` is machine-local — not committed, not archived. Pins are promises: a pinned unavailable engine stops with `BLOCKED:<engine>-unavailable`; a name without a `_shared/adapters/<name>.md` adapter stops with `BLOCKED:invalid-engine-config`. New engines (GLM, pi-agent backends) plug in by shipping an adapter file — no skill changes. Codex BUILD/IMPLEMENT and PLAN-pair remain research-only paths behind explicit `--engine codex`.
+
+### Conversational handoff + loop engineering — the default entry for all work
+
+The user does not invoke skills manually; the orchestrating model does. Small tasks: invoke `/devlyn:resolve "<goal>"` directly. Large tasks agreed in conversation:
+
+1. Write the agreed contract to `docs/specs/<id>/spec.md` (+ `spec.expected.json` when mechanical verifications exist). Always a spec file for large work — never `--goal-file`, which routes into `BLOCKED:large-needs-ideation`.
+2. Present a one-screen plan-contract summary — the user's single review checkpoint, BEFORE the pipeline starts.
+3. On go-ahead, invoke `/devlyn:resolve --spec <path>` hands-free to completion.
+4. **Per-task outer loop**: read the terminal verdict. PASS → done. Verdicts backed by spec/verification findings (NEEDS_WORK, verify/build-gate exhaustion) → adjudicate the findings, amend the spec (recorded in the spec file; spec stays read-only inside a run), re-invoke — at most 3 outer iterations, then surface with the findings trail. Infrastructure, invalid-input, engine-availability, and implement-empty BLOCKED verdicts are not spec-amendable: surface immediately. Every iteration re-enters through durable artifacts (spec, findings, run archive), never conversation memory.
 
-| Situation | Command |
-|-----------|---------|
-| New project / multi-feature plan / external spec to normalize | `/devlyn:ideate` (`--project`, default, or `--from-spec`) |
-| One-line goal you want turned into a spec autonomously | `/devlyn:ideate --quick` |
-| Spec already in hand | `/devlyn:resolve --spec <path>` |
-| Free-form fix / feature / refactor / debug / PR review | `/devlyn:resolve "<describe the work>"` |
-| Verify a diff or PR against a spec | `/devlyn:resolve --verify-only <ref> --spec <path>` |
+**Intent queue (unattended drain)**: the user stacks intents in `docs/specs/queue.md` (ordered checklist; `/devlyn:queue` is the front-end — no args shows status, `add <intent>` appends, `drain` starts the serial drain). Drain strictly serially — per item: spec it, run the outer loop, mark `[x]` done or `[F]` blocked with reason, continue; a blocked item never halts the queue. The queue entry is the user's go-ahead, so assume-and-log replaces the interactive checkpoint — but unattended assumptions may only take scope-narrowing, reversible, non-user-visible defaults. Material ambiguity (user-visible behavior, data/state semantics, new files/scripts/flags, implementation surface) → mark `[F] needs-review` and move on. End the drain with a per-item verdict + assumptions report.
 
 ### Subtractive-first editing — perfection = nothing left to remove
 <!-- runtime-principles:section=subtractive-first:begin -->
@@ -87,7 +104,7 @@ This rule exists because LLMs (including you) are trained to be helpful, compreh
 **The five drift patterns you must refuse to execute on:**
 
 1. **Unrequested work.** "While I'm here, I noticed X is broken/ugly/inefficient" → **stop**. The user did not ask for X. If X is a real defect, surface it as a finding, a follow-up suggestion, or an entry in a TODO list — do NOT fix it inside the current change. Mixing unrequested work with requested work is what makes diffs unreviewable and PRs eternal. **Pre-existing dead code → mention only, do NOT delete; orphans YOUR change created (now-unused imports, variables, functions) → clean them up.**
-2. **Tangential cleanup.** "This file looks messy, let me also tidy..." → **stop**. The current task is the only task. Unrelated cleanup is a separate change requiring its own justification, scope, and pre-flight 0 check. **Match existing style even if you'd write it differently; do NOT touch comments, formatting, or code orthogonal to your real change** — silent side-effects on neighboring lines are the most common Karpathy-observed regression class.
+2. **Tangential cleanup.** "This file looks messy, let me also tidy..." → **stop**. The current task is the only task. Unrelated cleanup is a separate change requiring its own justification, scope, and pre-flight 0 check. **Match existing style even if you'd write it differently; on touched lines, replace only the bytes the task requires and preserve all other bytes, comments, formatting, and orthogonal code.**
 3. **Speculative robustness.** "Just adding a check / fallback / handler for the case where..." → **stop**. If the case has not been observed (in production, in tests, in a finding), it does not belong in this change. Defensive code added for unobserved cases is the most common form of accretion debt — it never gets removed because nobody can prove the case never happens.
 4. **Re-scoping mid-flight.** "Actually, the better way to do this is to also restructure / rename / migrate..." → **stop**. If you discover the requested approach is wrong, surface that to the user with evidence and let them adjudicate. Do NOT silently expand scope. The user's explicit redirect is the only authorization to enlarge a task.
 5. **Curiosity detours.** "Let me also explore how Y works to understand this better..." → **stop**, unless Y is provably on the goal's critical path. Curiosity-driven exploration is creative-mode; default is execution-mode.
@@ -146,7 +163,7 @@ When `/devlyn:resolve` or `/devlyn:ideate` route a phase to Codex (`--engine cod
 
 ## Skill Boundary Policy
 
-The runtime pipeline surface is two skills — `/devlyn:resolve` and `/devlyn:ideate` — plus `/devlyn:design-ui` for creative UI exploration. `/devlyn:resolve` runs PLAN → IMPLEMENT → BUILD_GATE → CLEANUP → VERIFY inline; verification, cleanup, and security review (delegated to the native `security-review` Claude Code skill from BUILD_GATE) all live inside the pipeline. There are no standalone `/devlyn:review`, `/devlyn:evaluate`, or `/devlyn:team-resolve` surfaces. `/devlyn:design-ui` spawns a 5-specialist design team (Creative Director, Product Designer, Visual Designer, Interaction Designer, Accessibility Designer). `/devlyn:reap` is an optional user-invoked skill in `optional-skills/`; resolve never delegates to it.
+The runtime pipeline surface is two skills — `/devlyn:resolve` and `/devlyn:ideate` — plus `/devlyn:design-ui` for creative UI exploration and two utilities added on explicit user direction: `/devlyn:engines` (engine-role config, iter-0038) and `/devlyn:queue` (intent-queue status/add/drain, iter-0039). `/devlyn:resolve` runs PLAN → IMPLEMENT → BUILD_GATE → CLEANUP → VERIFY inline; verification, cleanup, and security review (delegated to the native `security-review` Claude Code skill from BUILD_GATE) all live inside the pipeline. There are no standalone `/devlyn:review`, `/devlyn:evaluate`, or `/devlyn:team-resolve` surfaces. `/devlyn:design-ui` spawns a 5-specialist design team (Creative Director, Product Designer, Visual Designer, Interaction Designer, Accessibility Designer). `/devlyn:reap` is an optional user-invoked skill in `optional-skills/`; resolve never delegates to it.
 
 Browser validation runs directly from BUILD_GATE using whichever toolchain is available (Chrome MCP, Playwright, or curl-tier fallback) — there is no separate `/devlyn:browser-validate` skill.
 

package/benchmark/auto-resolve/fixtures/F1-cli-trivial-flag/task.ko.txt

@@ -0,0 +1,8 @@
+bench-test-repo의 CLI(bin/cli.js)에 있는 `hello` 서브커맨드에 `--loud` 플래그를 추가해주세요. `--loud`가 전달되면 인사말이 대문자로 바뀌고 느낌표 두 개로 끝나야 합니다.
+
+예시:
+- `node bin/cli.js hello --loud` → `HELLO, WORLD!!`
+- `node bin/cli.js hello --loud --name alice` → `HELLO, ALICE!!`
+- `node bin/cli.js hello` → `Hello, world!` (기본값 변경 없음)
+
+기존 테스트가 계속 통과하는지 확인하고, `--loud` 경로에 대한 테스트를 최소 하나 추가해주세요. 관련 없는 파일은 건드리지 마세요 — `bin/cli.js`와 `tests/cli.test.js`만 수정하세요. 새로운 npm 의존성은 추가하지 마세요.

package/benchmark/auto-resolve/fixtures/retired/F34-cli-settle-inflight/NOTES.md

@@ -0,0 +1,28 @@
+# F34 CLI settle in-flight
+
+## Failure mode
+
+Detects implementations that couple authorization completion order to
+settlement order. The two cheap-but-wrong shapes: (a) sequential
+await-per-transfer — correct ledger, fails the visible overlap requirement
+(`high_water >= 3`); (b) apply-as-authorizations-resolve — meets overlap,
+corrupts file-order settlement (flips `t-60` accept and `t-40` reject in the
+verifier input). The correct shape starts authorizations concurrently and
+settles strictly in file order afterward — the decoupling is the measured
+skill (Tier-1 "H1" from memory `project_pair_fixture_tier1_2026_05_05`;
+keyword-hidden: spec never says race/lock/mutex/concurrency primitive).
+
+## Fairness
+
+Every verifier assertion quotes a visible spec bullet via `contract_refs`
+(lint-fixtures enforced). Gateway latency is deterministic (trailing digits
+of the id = ms), high-water threshold is stated in the spec with the exact
+sample size, and the flip case is derivable from the visible settlement rule
+alone. No hidden shape or tie-break exists outside spec.md.
+
+## Headroom prediction (pre-registered, unmeasured)
+
+bare ≤ 60 (expected to fail ordering-vs-completion or overlap), solo ≤ 80
+(hypothesis: misses the decoupling — settles in completion order). To be
+measured with `headroom-gate.py` before any pair-evidence claims; this
+fixture carries no measurement claim until then.

package/benchmark/auto-resolve/fixtures/retired/F34-cli-settle-inflight/RETIRED.md

@@ -0,0 +1,15 @@
+# F34 retired
+
+Retired from the active golden suite after headroom run
+`iter-0039-headroom-f34-f35`.
+
+Reason: bare scored 92, exceeding the headroom ceiling of 60, and
+`solo_claude` scored 88, exceeding the headroom ceiling of 80. Both arms
+solved the settle-inflight task competently (judge: bare 92 vs solo_claude 88,
+margin -4) and share the same minor scope slip (touching
+`data/gateway-stats.json` outside the expected diff); neither arm hit a hard
+disqualifier. The task was not hard enough for bare, let alone solo — there is
+no pair-lift headroom to measure.
+
+Future use: rework the visible contract or hidden verifiers so the task
+creates a fair pair-risk-probe gap without hiding requirements from the spec.

package/benchmark/auto-resolve/fixtures/retired/F34-cli-settle-inflight/expected.json

@@ -0,0 +1,57 @@
+{
+  "verification_commands": [
+    {
+      "cmd": "node --test tests/cli.test.js",
+      "exit_code": 0,
+      "stdout_contains": [],
+      "stdout_not_contains": ["not ok "]
+    },
+    {
+      "cmd": "node \"$BENCH_FIXTURE_DIR/verifiers/ordered-settlement.js\"",
+      "exit_code": 0,
+      "stdout_contains": ["\"ok\":true"],
+      "stdout_not_contains": [],
+      "contract_refs": [
+        "Every transfer is authorized by calling `authorize(transfer)` from `lib/gateway.js` exactly once per transfer.",
+        "Regardless of the order in which authorizations complete, transfers settle in file order, each exactly once.",
+        "Each rejected transfer produces exact row shape `{ \"id\": string, \"error\": \"insufficient_funds\", \"available\": number, \"requested\": number }`, where `available` is the sender's balance at that transfer's turn in file order.",
+        "The seeded gateway records the maximum number of authorizations in flight at once in `data/gateway-stats.json`; settling the six-transfer sample in `## Verification` must record `high_water` of at least `3`.",
+        "The command must not modify `data/accounts.json`."
+      ]
+    },
+    {
+      "cmd": "node \"$BENCH_FIXTURE_DIR/verifiers/reject-invalid.js\"",
+      "exit_code": 0,
+      "stdout_contains": ["\"ok\":true"],
+      "stdout_not_contains": [],
+      "contract_refs": [
+        "Validation happens before any authorization starts.",
+        "unknown account, or non-positive or non-integer `amount` exits `2` and writes exactly one JSON error object to stderr with shape `{ \"error\": \"invalid_input\", \"reason\": string }`, printing nothing to stdout."
+      ]
+    }
+  ],
+  "forbidden_patterns": [
+    {
+      "pattern": "catch\\s*\\([^)]*\\)\\s*\\{[^}]*return\\s+(null|undefined|''|\\{\\})",
+      "description": "silent catch returning fallback in settle path",
+      "files": ["bin/cli.js"],
+      "severity": "disqualifier"
+    },
+    {
+      "pattern": "catch\\s*\\([^)]*\\)\\s*\\{\\s*\\}",
+      "description": "empty catch block",
+      "files": ["bin/cli.js"],
+      "severity": "disqualifier"
+    },
+    {
+      "pattern": "/\\*\\s*eslint-disable",
+      "description": "eslint-disable without scoped justification",
+      "files": ["bin/cli.js"],
+      "severity": "disqualifier"
+    }
+  ],
+  "required_files": ["bin/cli.js", "tests/cli.test.js", "data/accounts.json", "lib/gateway.js"],
+  "forbidden_files": [],
+  "max_deps_added": 0,
+  "spec_output_files": ["bin/cli.js", "tests/cli.test.js"]
+}

package/benchmark/auto-resolve/fixtures/retired/F34-cli-settle-inflight/metadata.json

@@ -0,0 +1,10 @@
+{
+  "id": "F34-cli-settle-inflight",
+  "category": "high-risk",
+  "difficulty": "high",
+  "timeout_seconds": 1500,
+  "required_tools": ["node"],
+  "browser": false,
+  "deps_change_expected": false,
+  "intent": "Add a bench-cli settle command that authorizes transfers through a seeded async gateway with overlapping in-flight authorizations while settling balances strictly in file order, with exact accepted/rejected/balances JSON output and validation before any authorization."
+}

package/benchmark/auto-resolve/fixtures/retired/F34-cli-settle-inflight/setup.sh

@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+# F34 setup — seed accounts and the concurrency-tracking gateway.
+set -e
+
+mkdir -p data lib
+
+cat > data/accounts.json <<'JSON'
+{
+  "acct-a": 100,
+  "acct-b": 0,
+  "acct-c": 50,
+  "acct-d": 10
+}
+JSON
+
+cat > lib/gateway.js <<'JS'
+'use strict';
+// Seeded authorization gateway. DO NOT MODIFY (fixture contract).
+// Latency is deterministic per transfer id (trailing digits = milliseconds),
+// so authorization completion order can differ from file order. Tracks how
+// many authorizations are in flight at once and how many calls were made.
+const fs = require('fs');
+const path = require('path');
+
+const STATS_PATH = path.join(__dirname, '..', 'data', 'gateway-stats.json');
+let inFlight = 0;
+let highWater = 0;
+let calls = 0;
+
+function writeStats() {
+  fs.writeFileSync(STATS_PATH, JSON.stringify({ calls, high_water: highWater }) + '\n');
+}
+
+async function authorize(transfer) {
+  calls += 1;
+  inFlight += 1;
+  if (inFlight > highWater) highWater = inFlight;
+  writeStats();
+  const digits = String(transfer && transfer.id ? transfer.id : '').replace(/\D/g, '');
+  const ms = digits ? (Number(digits) % 100) || 5 : 5;
+  await new Promise((resolve) => setTimeout(resolve, ms));
+  inFlight -= 1;
+  writeStats();
+  return { authorized: true };
+}
+
+module.exports = { authorize };
+JS
+
+exit 0

package/benchmark/auto-resolve/fixtures/retired/F34-cli-settle-inflight/spec.md

@@ -0,0 +1,59 @@
+---
+id: "F34-cli-settle-inflight"
+title: "Settle command with in-flight authorizations"
+status: planned
+complexity: high
+depends-on: []
+---
+
+# F34 Settle command with in-flight authorizations
+
+## Context
+
+`bench-cli` gets a `settle` command that previews a batch of credit
+transfers between accounts. Every transfer must be authorized through the
+seeded gateway module before settlement. The gateway is slow, so
+authorizations for different transfers must be able to overlap — but the
+ledger math must stay exact and order-stable.
+
+## Requirements
+
+- [ ] `bench-cli settle --input <path>` reads JSON shaped as `{ "transfers": [{ "id": string, "from": string, "to": string, "amount": number }] }`.
+- [ ] Account starting balances come from `data/accounts.json`. The command must not modify `data/accounts.json`.
+- [ ] Every transfer is authorized by calling `authorize(transfer)` from `lib/gateway.js` exactly once per transfer. Do not modify, reimplement, or bypass `lib/gateway.js`.
+- [ ] Authorizations must be able to overlap: while one transfer's authorization is awaited, later transfers' authorizations must be able to start. The seeded gateway records the maximum number of authorizations in flight at once in `data/gateway-stats.json`; settling the six-transfer sample in `## Verification` must record `high_water` of at least `3`.
+- [ ] Regardless of the order in which authorizations complete, transfers settle in file order, each exactly once. A transfer is accepted when the sender's balance — after all earlier-in-file accepted transfers — covers the amount; otherwise it is rejected.
+- [ ] Each rejected transfer produces exact row shape `{ "id": string, "error": "insufficient_funds", "available": number, "requested": number }`, where `available` is the sender's balance at that transfer's turn in file order.
+- [ ] Validation happens before any authorization starts. Invalid JSON, missing or non-string `id`/`from`/`to`, unknown account, or non-positive or non-integer `amount` exits `2` and writes exactly one JSON error object to stderr with shape `{ "error": "invalid_input", "reason": string }`, printing nothing to stdout.
+- [ ] On success, write exactly one JSON object to stdout and no stderr. Keys: `accepted` (array of transfer ids in file order), `rejected` (array of rejected rows in file order), `balances` (object with the final balance of every account in `data/accounts.json`).
+- [ ] `tests/cli.test.js` is updated. Existing tests still pass AND at least two new tests cover `settle`: one where a later-in-file transfer depends on an earlier-in-file transfer's settlement, and one rejection.
+
+## Constraints
+
+- **No new npm dependencies.**
+- **No silent catches.** If parsing or file reading fails, emit a visible JSON error to stderr and exit `2`.
+- **No extra stdout/stderr text** on the success path; downstream tooling parses stdout as JSON.
+- Balances are integer credits; no floating-point money.
+
+## Out of Scope
+
+- Persisting settlement results (settle is a preview; `data/accounts.json` stays untouched).
+- Changing server routes, web UI, or `lib/gateway.js`.
+- Retry, timeout, or failure handling for the gateway beyond what it returns.
+- Touching `server/`, `web/`, or `tests/server.test.js`.
+
+## Verification
+
+- `node --test tests/cli.test.js` exits 0.
+- Settling a six-transfer file where authorizations complete in reverse file order still yields file-order settlement: exact `accepted`, `rejected`, and `balances` values.
+- `data/gateway-stats.json` shows `calls` equal to the number of transfers and `high_water` of at least `3`.
+- An unknown account exits `2`, prints one JSON error to stderr, and prints no stdout.
+- `git diff --stat` shows only `bin/cli.js` and `tests/cli.test.js` touched (accounts and gateway are seeded by setup, not the arm).
+
+## Solo-headroom hypothesis
+
+A capable solo_claude baseline is expected to miss the decoupling of
+authorization completion order from file-order settlement — applying
+balance changes as authorizations resolve instead of at each transfer's
+file-order turn; observable command
+`node "$BENCH_FIXTURE_DIR/verifiers/ordered-settlement.js"` exposes the miss.

package/benchmark/auto-resolve/fixtures/retired/F34-cli-settle-inflight/task.txt

@@ -0,0 +1,9 @@
+Add a `settle` command to `bench-cli` so users can run `bench-cli settle --input <path>` with a transfers JSON file. Starting balances come from `data/accounts.json` (read-only — settle is a preview and must not modify it). Every transfer must be authorized by calling `authorize(transfer)` from the seeded `lib/gateway.js` exactly once; do not modify or bypass that module.
+
+The gateway is slow, so authorizations for different transfers must be able to overlap — while one is awaited, later ones must be able to start. The seeded gateway records `calls` and the in-flight `high_water` mark in `data/gateway-stats.json`; a six-transfer batch must reach a high-water mark of at least 3.
+
+No matter which order authorizations finish in, transfers settle in file order, each exactly once: a transfer is accepted when the sender's balance after all earlier-in-file accepted transfers covers the amount, otherwise it is rejected with the exact row `{ "id", "error": "insufficient_funds", "available", "requested" }` where `available` is the balance at that transfer's file-order turn.
+
+Validate the whole file before any authorization starts: bad JSON, missing fields, unknown accounts, or non-positive/non-integer amounts exit `2` with exactly one JSON error object `{ "error": "invalid_input", "reason": string }` on stderr and nothing on stdout. On success print exactly one JSON object with `accepted` (ids in file order), `rejected` (rows in file order), and `balances` (every account) — and nothing on stderr.
+
+Update `tests/cli.test.js`: existing tests still pass, plus at least two new settle tests (one where a later transfer depends on an earlier transfer's settlement, one rejection). No new npm dependencies. Only touch `bin/cli.js` and `tests/cli.test.js`.

package/benchmark/auto-resolve/fixtures/retired/F34-cli-settle-inflight/verifiers/ordered-settlement.js

@@ -0,0 +1,88 @@
+'use strict';
+// F34 hidden verifier — file-order settlement under inverted completion order.
+// Transfer ids encode gateway latency (trailing digits = ms), so the FIRST
+// transfer in the file finishes authorization LAST. Applying balances in
+// completion order (or rejecting against pre-batch balances) flips t-60 and
+// t-40; only strict file-order settlement produces this exact output.
+
+const fs = require('node:fs');
+const os = require('node:os');
+const path = require('node:path');
+const { spawnSync } = require('node:child_process');
+
+// Structural equality: object key order is NOT part of the visible contract;
+// array order IS. Codex fairness-audit fix (stringify equality made JSON
+// property order a hidden oracle narrower than spec.md).
+function deepEqual(a, b) {
+  if (a === b) return true;
+  if (Array.isArray(a) && Array.isArray(b)) {
+    return a.length === b.length && a.every((v, i) => deepEqual(v, b[i]));
+  }
+  if (a && b && typeof a === 'object' && typeof b === 'object' && !Array.isArray(a) && !Array.isArray(b)) {
+    const ka = Object.keys(a).sort();
+    const kb = Object.keys(b).sort();
+    return deepEqual(ka, kb) && ka.every((k) => deepEqual(a[k], b[k]));
+  }
+  return false;
+}
+
+const inputPath = path.join(os.tmpdir(), `settle-input-${process.pid}.json`);
+fs.writeFileSync(inputPath, JSON.stringify({
+  transfers: [
+    { id: 't-80', from: 'acct-a', to: 'acct-b', amount: 70 },
+    { id: 't-60', from: 'acct-b', to: 'acct-c', amount: 50 },
+    { id: 't-40', from: 'acct-a', to: 'acct-d', amount: 40 },
+    { id: 't-20', from: 'acct-c', to: 'acct-a', amount: 100 },
+    { id: 't-10', from: 'acct-d', to: 'acct-b', amount: 5 },
+    { id: 't-5', from: 'acct-b', to: 'acct-a', amount: 25 }
+  ]
+}));
+
+const work = process.env.BENCH_WORKDIR;
+const cli = path.join(work, 'bin', 'cli.js');
+const result = spawnSync('node', [cli, 'settle', '--input', inputPath], {
+  cwd: work,
+  encoding: 'utf8'
+});
+
+let out;
+try {
+  out = JSON.parse(result.stdout);
+} catch {
+  out = null;
+}
+
+const expected = {
+  accepted: ['t-80', 't-60', 't-20', 't-10', 't-5'],
+  rejected: [
+    { id: 't-40', error: 'insufficient_funds', available: 30, requested: 40 }
+  ],
+  balances: { 'acct-a': 155, 'acct-b': 0, 'acct-c': 0, 'acct-d': 5 }
+};
+
+let stats;
+try {
+  stats = JSON.parse(fs.readFileSync(path.join(work, 'data', 'gateway-stats.json'), 'utf8'));
+} catch {
+  stats = null;
+}
+
+const accountsUntouched = (() => {
+  try {
+    const seeded = JSON.parse(fs.readFileSync(path.join(work, 'data', 'accounts.json'), 'utf8'));
+    return deepEqual(seeded, { 'acct-a': 100, 'acct-b': 0, 'acct-c': 50, 'acct-d': 10 });
+  } catch {
+    return false;
+  }
+})();
+
+const ok = result.status === 0
+  && result.stderr === ''
+  && deepEqual(out, expected)
+  && stats !== null
+  && stats.calls === 6
+  && stats.high_water >= 3
+  && accountsUntouched;
+
+console.log(JSON.stringify({ ok, status: result.status, stderr: result.stderr, out, stats, accountsUntouched }));
+process.exit(ok ? 0 : 1);

package/benchmark/auto-resolve/fixtures/retired/F34-cli-settle-inflight/verifiers/reject-invalid.js

@@ -0,0 +1,45 @@
+'use strict';
+// F34 hidden verifier — validation happens before any authorization: an
+// unknown account exits 2 with one JSON error object on stderr, no stdout,
+// and the gateway records zero calls for the run.
+
+const fs = require('node:fs');
+const os = require('node:os');
+const path = require('node:path');
+const { spawnSync } = require('node:child_process');
+
+const work = process.env.BENCH_WORKDIR;
+const statsPath = path.join(work, 'data', 'gateway-stats.json');
+fs.rmSync(statsPath, { force: true });
+
+const inputPath = path.join(os.tmpdir(), `settle-bad-${process.pid}.json`);
+fs.writeFileSync(inputPath, JSON.stringify({
+  transfers: [
+    { id: 't-30', from: 'acct-a', to: 'acct-zzz', amount: 10 }
+  ]
+}));
+
+const cli = path.join(work, 'bin', 'cli.js');
+const result = spawnSync('node', [cli, 'settle', '--input', inputPath], {
+  cwd: work,
+  encoding: 'utf8'
+});
+
+let err;
+try {
+  err = JSON.parse(result.stderr);
+} catch {
+  err = null;
+}
+
+const gatewayNeverCalled = !fs.existsSync(statsPath);
+
+const ok = result.status === 2
+  && result.stdout === ''
+  && err !== null
+  && err.error === 'invalid_input'
+  && typeof err.reason === 'string'
+  && gatewayNeverCalled;
+
+console.log(JSON.stringify({ ok, status: result.status, stdout: result.stdout, err, gatewayNeverCalled }));
+process.exit(ok ? 0 : 1);

package/benchmark/auto-resolve/fixtures/retired/F35-cli-apply-journal/NOTES.md

@@ -0,0 +1,30 @@
+# F35 CLI apply journal
+
+## Failure mode
+
+Detects implementations that flatten the composite ordering/replay/atomicity
+contract into the easy shape: apply-in-file-order against pre-batch stock
+with per-row persistence. Three discriminators: (a) `applied` order and
+journal append order must follow priority-desc with file-order ties (final
+quantities alone cannot catch this — addition commutes); (b) the failing
+op's `available` is measured at its turn AFTER higher-priority ops changed
+stock (rollback verifier: 15, not the pre-batch 10); (c) any failure leaves
+both files byte-for-byte unchanged. In-file duplicate `op_id` is a
+validation error while a journaled `op_id` is a per-op skip — the
+distinction is the replay axis (Tier-1 "A1" from memory
+`project_pair_fixture_tier1_2026_05_05`; keyword-hidden: spec never says
+atomic/transaction/idempotent/rollback).
+
+## Fairness
+
+Every verifier assertion quotes a visible spec bullet via `contract_refs`
+(lint-fixtures enforced). Verifiers reset the seeded files before running so
+command order cannot leak state between checks. No hidden shape, tie-break,
+or availability rule exists outside spec.md.
+
+## Headroom prediction (pre-registered, unmeasured)
+
+bare ≤ 60 (expected to miss at-its-turn availability or journal append
+order), solo ≤ 80 (hypothesis: evaluates availability against pre-batch or
+file-order state). To be measured with `headroom-gate.py` before any
+pair-evidence claims; this fixture carries no measurement claim until then.

package/benchmark/auto-resolve/fixtures/retired/F35-cli-apply-journal/RETIRED.md

@@ -0,0 +1,16 @@
+# F35 retired
+
+Retired from the active golden suite after headroom run
+`iter-0039-headroom-f34-f35`.
+
+Reason: `solo_claude` scored 97, exceeding the headroom ceiling of 80, while
+bare scored 50. `solo_claude` correctly implemented the `{ "ops": [...] }`
+input contract, priority ordering, rollback, and duplicate-op handling; bare
+failed only because it assumed a bare top-level array instead of the required
+object shape — an input-shape slip, not evidence the task is hard. The
+solo_claude arm also ran to the full 1500s timeout before finishing
+(elapsed_seconds 1501, INVOKE_EXIT=124), but its diff still verified clean, so
+the fixture is not pair-lift evidence.
+
+Future use: rework the visible contract or hidden verifiers so the task
+creates a fair pair-risk-probe gap without hiding requirements from the spec.

package/benchmark/auto-resolve/fixtures/retired/F35-cli-apply-journal/expected.json

@@ -0,0 +1,64 @@
+{
+  "verification_commands": [
+    {
+      "cmd": "node --test tests/cli.test.js",
+      "exit_code": 0,
+      "stdout_contains": [],
+      "stdout_not_contains": ["not ok "]
+    },
+    {
+      "cmd": "node \"$BENCH_FIXTURE_DIR/verifiers/priority-apply.js\"",
+      "exit_code": 0,
+      "stdout_contains": ["\"ok\":true"],
+      "stdout_not_contains": [],
+      "contract_refs": [
+        "An op whose `op_id` is already listed in the journal's `applied` array is recorded as skipped and changes nothing, regardless of its other fields.",
+        "Non-skipped ops apply in descending `priority`; ties apply in file order.",
+        "On success, persist the final quantities to `data/inventory.json`, append the applied `op_id`s to the journal's `applied` array in application order, and write exactly one JSON object to stdout with keys `applied` (op_ids in application order), `skipped` (op_ids in file order), and `inventory` (final quantities for every sku)."
+      ]
+    },
+    {
+      "cmd": "node \"$BENCH_FIXTURE_DIR/verifiers/rollback.js\"",
+      "exit_code": 0,
+      "stdout_contains": ["\"ok\":true"],
+      "stdout_not_contains": [],
+      "contract_refs": [
+        "An op fails when applying its `delta` would take the sku's quantity below zero, evaluated at its turn in that order, with exact error shape `{ \"error\": \"insufficient_stock\", \"op_id\": string, \"sku\": string, \"available\": number, \"requested\": number }` where `available` is the quantity at that op's turn and `requested` is the absolute value of its `delta`.",
+        "If any op fails, nothing persists: `data/inventory.json` and `data/journal.json` are byte-for-byte unchanged, the command exits `2`, and exactly one JSON error object is written to stderr with nothing on stdout."
+      ]
+    },
+    {
+      "cmd": "node \"$BENCH_FIXTURE_DIR/verifiers/dup-op.js\"",
+      "exit_code": 0,
+      "stdout_contains": ["\"ok\":true"],
+      "stdout_not_contains": [],
+      "contract_refs": [
+        "or an `op_id` that appears more than once in the input file exits `2` and writes exactly one JSON error object to stderr with shape `{ \"error\": \"invalid_input\", \"reason\": string }`, printing nothing to stdout and changing neither file."
+      ]
+    }
+  ],
+  "forbidden_patterns": [
+    {
+      "pattern": "catch\\s*\\([^)]*\\)\\s*\\{[^}]*return\\s+(null|undefined|''|\\{\\})",
+      "description": "silent catch returning fallback in apply path",
+      "files": ["bin/cli.js"],
+      "severity": "disqualifier"
+    },
+    {
+      "pattern": "catch\\s*\\([^)]*\\)\\s*\\{\\s*\\}",
+      "description": "empty catch block",
+      "files": ["bin/cli.js"],
+      "severity": "disqualifier"
+    },
+    {
+      "pattern": "/\\*\\s*eslint-disable",
+      "description": "eslint-disable without scoped justification",
+      "files": ["bin/cli.js"],
+      "severity": "disqualifier"
+    }
+  ],
+  "required_files": ["bin/cli.js", "tests/cli.test.js", "data/inventory.json", "data/journal.json"],
+  "forbidden_files": [],
+  "max_deps_added": 0,
+  "spec_output_files": ["bin/cli.js", "tests/cli.test.js"]
+}

package/benchmark/auto-resolve/fixtures/retired/F35-cli-apply-journal/metadata.json

@@ -0,0 +1,10 @@
+{
+  "id": "F35-cli-apply-journal",
+  "category": "high-risk",
+  "difficulty": "high",
+  "timeout_seconds": 1500,
+  "required_tools": ["node"],
+  "browser": false,
+  "deps_change_expected": false,
+  "intent": "Add a bench-cli apply command that applies inventory adjustments in priority order with journal-based replay skipping, at-its-turn stock checks, exact JSON output, and all-or-nothing persistence of inventory and journal."
+}