devlyn-cli 3.2.1 → 3.3.0

package/benchmark/auto-resolve/fixtures/F8-known-limit-ambiguous/expected.json

@@ -1,63 +0,0 @@
-{
-  "verification_commands": [
-    {
-      "cmd": "node bin/cli.js hello",
-      "exit_code": 0,
-      "stdout_contains": [
-        "Hello, world!"
-      ],
-      "stdout_not_contains": []
-    },
-    {
-      "cmd": "node bin/cli.js hello --name alice",
-      "exit_code": 0,
-      "stdout_contains": [
-        "Hello, alice!"
-      ],
-      "stdout_not_contains": []
-    },
-    {
-      "cmd": "node bin/cli.js version",
-      "exit_code": 0,
-      "stdout_contains": [
-        "0.1.0"
-      ],
-      "stdout_not_contains": []
-    },
-    {
-      "cmd": "node bin/cli.js --help",
-      "exit_code": 0,
-      "stdout_contains": [
-        "hello"
-      ],
-      "stdout_not_contains": []
-    },
-    {
-      "cmd": "node --test tests/cli.test.js",
-      "exit_code": 0,
-      "stdout_contains": [],
-      "stdout_not_contains": [
-        "not ok "
-      ]
-    }
-  ],
-  "forbidden_patterns": [
-    {
-      "pattern": "catch\\s*\\([^)]*\\)\\s*\\{\\s*\\}",
-      "description": "empty catch block",
-      "files": [
-        "bin/cli.js"
-      ],
-      "severity": "disqualifier"
-    }
-  ],
-  "required_files": [
-    "bin/cli.js"
-  ],
-  "forbidden_files": [],
-  "max_deps_added": 0,
-  "spec_output_files": [
-    "bin/cli.js",
-    "tests/**"
-  ]
-}

package/benchmark/auto-resolve/fixtures/F8-known-limit-ambiguous/metadata.json

@@ -1,10 +0,0 @@
-{
-  "id": "F8-known-limit-ambiguous",
-  "category": "edge",
-  "difficulty": "ambiguous",
-  "timeout_seconds": 900,
-  "required_tools": ["node"],
-  "browser": false,
-  "deps_change_expected": false,
-  "intent": "Deliberately under-specified feature request: 'make the CLI better'. Both arms must disambiguate before building. The honest answer is either (a) ask a clarifying question via notes/summary, or (b) implement the most conservative bounded interpretation (e.g., a --help improvement). Fixture documents where the harness may not beat bare."
-}

package/benchmark/auto-resolve/fixtures/F8-known-limit-ambiguous/setup.sh

@@ -1,4 +0,0 @@
-#!/usr/bin/env bash
-# F8 setup — no changes. The ambiguity IS the test.
-set -e
-exit 0

package/benchmark/auto-resolve/fixtures/F8-known-limit-ambiguous/spec.md

@@ -1,48 +0,0 @@
----
-id: "F8-known-limit-ambiguous"
-title: "Improve the CLI"
-status: planned
-complexity: medium
-depends-on: []
----
-
-# F8 Improve the CLI
-
-## Context
-
-> "Users have been saying the CLI could be better. Make it better."
-
-That is the entire ask. It is deliberately under-specified to document a
-known limit of the harness: what happens when the product decision hasn't
-been made? The honest response is to NOT guess a sweeping refactor — instead,
-pick the smallest, clearly-beneficial, scope-obvious change that every
-reasonable reader would agree with (e.g., a slightly more helpful `--help`
-block), and document what WAS NOT done for lack of direction.
-
-## Requirements
-
-- [ ] Make a small, defensible improvement to the CLI. Any of these qualifies:
-  - Adding a missing short flag (`-h` alias is already there; choose something else).
-  - Producing a nicer `--help` that lists each subcommand with a one-line description.
-  - Distinguishing unknown-command and no-command cases in error output.
-- [ ] Existing behavior is not regressed. `node bin/cli.js hello` / `version` / `--help` still exit 0 with their current semantics.
-- [ ] Add at least one test that asserts the new behavior.
-- [ ] Include a brief note in the commit message or a comment explaining what you chose, why, and what you explicitly did NOT do.
-
-## Constraints
-
-- **No new npm dependencies.**
-- **No sweeping refactors.** Do not rewrite the argument parser or invert the file's control flow.
-- **No breaking changes** to current subcommands.
-
-## Out of Scope
-
-- Adding new subcommands the user didn't ask for.
-- Restyling, renaming, or deleting existing subcommands.
-- Touching `server/` or `web/`.
-
-## Verification
-
-- Existing baseline commands behave identically.
-- At least one new assertion in `tests/` exercises the change.
-- `node bin/cli.js --help` (if changed) is valid UTF-8 and lists every real subcommand once.

package/benchmark/auto-resolve/fixtures/F8-known-limit-ambiguous/task.txt

@@ -1 +0,0 @@
-Users have been saying the CLI could be better. Make it better.

package/benchmark/auto-resolve/fixtures/F9-e2e-ideate-to-resolve/NOTES.md

@@ -1,105 +0,0 @@
-# F9 — Notes (2-skill contract, post iter-0033a)
-
-## Purpose
-
-**Load-bearing for the novice-user contract.** The suite ship-gate requires
-F9 to pass (variant arm margin ≥ +5) on every shipped version. If F9 fails,
-the "type `/devlyn:ideate` and ship worldclass software" promise is not being
-met.
-
-Renamed 2026-04-30 (iter-0033a) from the `-to-preflight` legacy id to match
-the shipped 2-skill product surface: `/devlyn:ideate` → `/devlyn:resolve --spec`.
-The pre-rename copy is preserved at `fixtures/retired/F9-e2e-ideate-to-preflight/`
-for recovery if the OLD 3-skill chain ever needs replay.
-
-## What the variant arm does (2-skill chain)
-
-A novice-simulating prompt (`task.txt` is identical to what the user typed)
-is delivered to a fresh Claude session. The session has the new 2-skill kit
-installed. The pipeline arm is expected to:
-
-1. Recognize this is a vague idea, not a spec → invoke `/devlyn:ideate`.
-2. Ideate produces `docs/specs/<id>-<slug>/spec.md` + `spec.expected.json`
-   and announces `spec ready — /devlyn:resolve --spec <emitted-path>`.
-3. Run `/devlyn:resolve --spec <emitted-path>` (PLAN → IMPLEMENT → BUILD_GATE
-   → CLEANUP → VERIFY in one skill). VERIFY is the fresh-subagent final
-   phase, replacing the standalone `/devlyn:preflight` skill from the
-   3-skill era.
-
-The variant prompt explicitly instructs this chain so the test isn't about
-Claude inventing the chain — it's about the new tools being usable end-to-end
-when invoked.
-
-## What the bare arm does
-
-Same raw task delivered as a direct prompt with anti-skill rules. Bare
-implements `gitstats` using its own judgment. Bare does NOT produce any
-`docs/specs/**` artifacts (and isn't expected to).
-
-## Why margin ≥ +5 is required (vs L0 / bare)
-
-The pipeline's whole value prop is that it trades some bare-case tokens for
-quality uplift on novice flows. If this fixture can't show ≥ +5 margin
-vs L0, we're paying pipeline cost without delivering on the novice promise.
-
-**OLD-vs-NEW comparison is NOT measured here.** OLD `/devlyn:ideate` was
-replaced in iter-0032 (the new ideate is the only ideate at HEAD). Calling
-the OLD F9 chain (`/devlyn:ideate` → `/devlyn:auto-resolve` → `/devlyn:preflight`)
-at HEAD would invoke NEW ideate against OLD auto-resolve — a broken hybrid.
-The harness refuses `--resolve-skill old` on F9 with a hard error.
-
-## Scoring notes
-
-- The variant's `docs/specs/<id>-<slug>/spec.md` + `spec.expected.json` ARE
-  part of the judge's evaluation. The judge sees the full product (code +
-  spec + tests), not just the diff to `bin/cli.js`.
-- Bare doesn't produce spec files, so bare's judge payload is code+test only.
-  This asymmetry is INTENTIONAL — the fixture tests total-output quality,
-  not per-file quality.
-
-## Skill-driven artifact check (out-of-band, NOT in expected.json)
-
-Per Codex R0.5 §B: `expected.json.verification_commands` apply to ALL arms
-(see `run-fixture.sh:472`). A `docs/specs/**` check in expected.json would
-punish the bare arm (which doesn't run ideate). Skill-driven artifact
-verification lives in `scripts/check-f9-artifacts.py`, which runs AFTER
-the per-fixture verification block and asserts every non-bare skill arm
-(`variant`, `solo_claude`, `l2_gated`, `l2_risk_probes`, `l2_forced`)
-produced:
-
-- `docs/specs/<id>-<slug>/spec.md` exists.
-- `docs/specs/<id>-<slug>/spec.expected.json` exists.
-- transcript contains `/devlyn:resolve --spec` exactly once.
-- transcript does NOT contain `/devlyn:auto-resolve` or `/devlyn:preflight`.
-
-## Failure modes detected
-
-- **Pipeline skips ideate.** Variant invokes `/devlyn:resolve` directly on
-  the raw idea → free-form classifier kicks in → spec quality is shallow.
-  Caught by `scripts/check-f9-artifacts.py`: `docs/specs/**` files missing.
-- **Bare over-engineers.** Without a skeleton, bare builds too much,
-  touches wrong files, adds deps. Caught by spec constraints (no new deps,
-  forbidden empty catch).
-- **Variant chains the OLD names.** If the variant transcript contains
-  `/devlyn:auto-resolve` or `/devlyn:preflight`, the prompt-following gate
-  fails. iter-0033a's harness change ensures the variant prompt names only
-  the 2 surviving skills.
-- **Spec emit path divergence.** If the new ideate refactors away from
-  `<spec-dir>/<id>-<slug>/spec.md`, the harness check fails (path-shape
-  regression smoke #4 of iter-0033a catches it before benchmark runs).
-
-## Rotation trigger
-
-F9 is the last fixture we rotate — it's the anchor. If it saturates
-(variant consistently > 95), the whole suite needs a harder novice-flow
-anchor before we retire this one.
-
-## Current pair-evidence status
-
-Rejected as pair-lift evidence until reworked. `20260512-f9-e2e-headroom`
-measured bare 60 / solo_claude 90 with a +30 solo-over-bare margin, and
-`check-f9-artifacts.py` passed for bare (exempt) and solo_claude. The headroom
-gate still failed because bare headroom was 0 < 5, solo_claude exceeded 80, and
-bare carried a judge disqualifier. Keep F9 as the novice-flow anchor, but do not
-spend pair arms on it as pair evidence until the fixture is reworked and clears
-a fresh headroom gate.

package/benchmark/auto-resolve/fixtures/F9-e2e-ideate-to-resolve/expected.json

@@ -1,74 +0,0 @@
-{
-  "verification_commands": [
-    {
-      "cmd": "node bin/cli.js gitstats",
-      "exit_code": 0,
-      "stdout_contains": [
-        "Commits:",
-        "Last commit:"
-      ],
-      "stdout_not_contains": [
-        "Error:"
-      ]
-    },
-    {
-      "cmd": "node bin/cli.js gitstats --json",
-      "exit_code": 0,
-      "stdout_contains": [
-        "{",
-        "commits",
-        "authors"
-      ],
-      "stdout_not_contains": []
-    },
-    {
-      "cmd": "cd /tmp && node -e 'const { spawnSync } = require(\"child_process\"); const p = process.env.BENCH_WORKDIR || process.cwd(); console.log(spawnSync(\"node\", [p + \"/bin/cli.js\", \"gitstats\"], { encoding: \"utf8\", cwd: \"/tmp\" }).status)'",
-      "exit_code": 0,
-      "stdout_contains": [
-        "2"
-      ],
-      "stdout_not_contains": [
-        "0"
-      ]
-    },
-    {
-      "cmd": "node bin/cli.js hello",
-      "exit_code": 0,
-      "stdout_contains": [
-        "Hello, world!"
-      ],
-      "stdout_not_contains": []
-    },
-    {
-      "cmd": "node --test tests/",
-      "exit_code": 0,
-      "stdout_contains": [],
-      "stdout_not_contains": []
-    }
-  ],
-  "forbidden_patterns": [
-    {
-      "pattern": "catch\\s*\\([^)]*\\)\\s*\\{\\s*\\}",
-      "description": "empty catch",
-      "files": [
-        "bin/cli.js"
-      ],
-      "severity": "disqualifier"
-    }
-  ],
-  "required_files": [
-    "bin/cli.js"
-  ],
-  "forbidden_files": [],
-  "max_deps_added": 0,
-  "tier_a_waivers": [
-    "docs/specs/**",
-    "docs/VISION.md",
-    "docs/ROADMAP.md",
-    "docs/roadmap/**"
-  ],
-  "spec_output_files": [
-    "bin/**",
-    "tests/**"
-  ]
-}

package/benchmark/auto-resolve/fixtures/F9-e2e-ideate-to-resolve/metadata.json

@@ -1,10 +0,0 @@
-{
-  "id": "F9-e2e-ideate-to-resolve",
-  "category": "e2e",
-  "difficulty": "high",
-  "timeout_seconds": 3600,
-  "required_tools": ["node"],
-  "browser": false,
-  "deps_change_expected": false,
-  "intent": "End-to-end novice flow (2-skill contract): from a vague idea ('git stats CLI for the current repo') the variant must run /devlyn:ideate → /devlyn:resolve --spec <emitted-path> to produce spec + implemented code + verified output. VERIFY is the fresh-subagent final phase of resolve (no separate preflight skill). The bare arm receives the same vague idea as a direct prompt. This fixture gates the novice-user contract."
-}

package/benchmark/auto-resolve/fixtures/F9-e2e-ideate-to-resolve/setup.sh

@@ -1,28 +0,0 @@
-#!/usr/bin/env bash
-# F9 setup — seed a few synthetic commits with different authors so the
-# `gitstats` subcommand's "top 3 authors by commit count" requirement is
-# meaningfully exercised. Without this, every commit author is the runner's
-# default and the ranking test is a no-op.
-set -e
-
-commit_as() {
-  local name="$1" email="$2" file="$3" message="$4"
-  echo "$(date +%s%N) $name" >> "$file"
-  git add "$file"
-  git -c user.name="$name" -c user.email="$email" commit -q -m "$message"
-}
-
-mkdir -p .bench-seed
-
-commit_as "Alpha Author"   "alpha@bench.test"   .bench-seed/log "seed: alpha 1"
-commit_as "Alpha Author"   "alpha@bench.test"   .bench-seed/log "seed: alpha 2"
-commit_as "Alpha Author"   "alpha@bench.test"   .bench-seed/log "seed: alpha 3"
-commit_as "Alpha Author"   "alpha@bench.test"   .bench-seed/log "seed: alpha 4"
-commit_as "Beta Author"    "beta@bench.test"    .bench-seed/log "seed: beta 1"
-commit_as "Beta Author"    "beta@bench.test"    .bench-seed/log "seed: beta 2"
-commit_as "Beta Author"    "beta@bench.test"    .bench-seed/log "seed: beta 3"
-commit_as "Gamma Author"   "gamma@bench.test"   .bench-seed/log "seed: gamma 1"
-commit_as "Gamma Author"   "gamma@bench.test"   .bench-seed/log "seed: gamma 2"
-commit_as "Delta Author"   "delta@bench.test"   .bench-seed/log "seed: delta 1"
-
-echo "F9 setup: seeded 10 commits across 4 authors (Alpha 4 / Beta 3 / Gamma 2 / Delta 1)"

package/benchmark/auto-resolve/fixtures/F9-e2e-ideate-to-resolve/spec.md

@@ -1,59 +0,0 @@
----
-id: "F9-e2e-ideate-to-resolve"
-title: "End-to-end: idea → shipped CLI feature (2-skill contract)"
-status: planned
-complexity: high
-depends-on: []
----
-
-# F9 End-to-End Novice Flow (2-skill chain)
-
-## Context
-
-A first-time user has a vague idea:
-
-> "I want a CLI subcommand that shows basic stats about the current git repo — commit count, last commit date, top 3 authors. Call it `gitstats`."
-
-The variant arm is expected to use the 2-skill chain:
-`/devlyn:ideate` → `/devlyn:resolve --spec <emitted-path>`. The bare arm
-receives the same idea as a direct prompt and implements it without the
-pipeline.
-
-This fixture is the suite's most important gate for the "novice user contract":
-a first-time user typing `/devlyn:ideate` should land at working,
-well-structured software. VERIFY runs as the fresh-subagent final phase
-inside `/devlyn:resolve` (no separate preflight skill in the 2-skill design).
-
-## Requirements
-
-- [ ] A new `gitstats` subcommand exists in `bin/cli.js`.
-- [ ] `node bin/cli.js gitstats` (run inside a git repo) prints:
-  - Line 1: commit count (e.g., `Commits: 42`).
-  - Line 2: last commit ISO date (e.g., `Last commit: 2026-04-23T12:00:00Z`).
-  - Lines 3-5: top 3 authors by commit count, format `<rank>. <name> <count>`.
-- [ ] Run outside a git repo → stderr message `Error: not a git repository` and exit 2.
-- [ ] `node bin/cli.js gitstats --json` emits valid JSON with the same data.
-- [ ] Existing subcommands (`hello`, `version`) unchanged.
-- [ ] Add at least one test.
-
-## Constraints
-
-- **No new npm dependencies.** Use `child_process` to shell out to `git`.
-- **No silent catches.**
-- **Non-git-repo handling.** Do not assume the user is always in a repo.
-
-
-## Out of Scope
-
-- Parsing commit messages, tags, branches.
-- Remote API calls.
-- Touching `server/` or `web/`.
-
-## Verification
-
-- Inside this worktree (which IS a git repo): `node bin/cli.js gitstats` exits 0 and prints at least 5 lines of summary.
-- `node bin/cli.js gitstats --json | node -e 'const d=JSON.parse(require("fs").readFileSync(0,"utf8")); console.log(typeof d.commits)'` prints `number`.
-- `cd /tmp && node <worktree>/bin/cli.js gitstats` (from outside a repo — use the worktree's absolute path) exits 2.
-- `node --test tests/` passes.
-
-(Skill-driven artifact checks — `docs/specs/<id>-<slug>/spec.md` + `spec.expected.json` existence, transcript fingerprint — live in `scripts/check-f9-artifacts.py`, NOT in the shared verification block above. Bare is exempt. See NOTES.md.)

package/benchmark/auto-resolve/fixtures/F9-e2e-ideate-to-resolve/task.txt

@@ -1,5 +0,0 @@
-I want a CLI subcommand that shows basic stats about the current git repo — commit count, last commit date, top 3 authors. Call it `gitstats`.
-
-Should work inside this repo when I run `node bin/cli.js gitstats`, and fail cleanly if I'm not in a git repo. A `--json` flag for machine-readable output would be useful too.
-
-Keep the existing `hello` and `version` subcommands working. Add a test. No new npm dependencies.

package/benchmark/auto-resolve/fixtures/SCHEMA.md

@@ -1,188 +0,0 @@
-# Fixture Schema
-
-Every fixture is a directory under `benchmark/auto-resolve/fixtures/F<N>-<slug>/` with these files. **All six files are required** (setup.sh may be empty when the starting `test-repo` copy needs no modification).
-
-## metadata.json
-
-```json
-{
-  "id": "F2-cli-medium-subcommand",
-  "category": "medium",
-  "difficulty": "medium",
-  "timeout_seconds": 1200,
-  "required_tools": ["node"],
-  "browser": false,
-  "deps_change_expected": false,
-  "intent": "One-sentence plain-language statement of the work, the SINGLE source of truth for spec.md and task.txt."
-}
-```
-
-- **id** — matches directory name, used across artifacts.
-- **category** — one of `trivial | medium | high-risk | stress | edge | e2e`. Drives which ship-gate rule applies.
-- **difficulty** — expected difficulty independent of category. Rubric uses this only for saturation detection (when `bare` and `solo_claude` both exceed 95 for two versions, flag fixture for rotation).
-- **timeout_seconds** — per-arm hard timeout. Runner kills the arm at this limit and marks result `TIMEOUT`.
-- **required_tools** — binaries the arm's environment must provide. Runner checks before invocation.
-- **browser** — true if arm must be able to run Playwright. Runner uses this to decide whether `test-repo`'s Playwright deps get installed before the arm starts.
-- **deps_change_expected** — true if the task involves modifying `package.json` / lockfiles. The pipeline arm's CRITIC security sub-pass is expected to trigger native `security-review` dep audit when true.
-- **intent** — **load-bearing**. A short plain-language statement shared by all arms. `spec.md` formalizes it into resolve-ready form; `task.txt` renders it as a direct prompt. A CI lint ensures both derive from this field and stay in sync.
-
-## spec.md
-
-Resolve-ready spec for the pipeline arm. Same format `/devlyn:ideate` produces:
-
-```markdown
----
-id: "<fixture-id>"
-title: "<short title>"
-status: planned
-complexity: medium
-depends-on: []
----
-
-# <fixture-id> <Title>
-
-## Context
-2-3 sentences describing WHY (not HOW). Must be traceable back to `metadata.intent`.
-
-## Requirements
-- [ ] Specific, testable, scoped.
-- [ ] ...
-
-## Constraints
-- Concrete, with reasoning for each (not bare).
-
-## Out of Scope
-- Explicit "must NOT build" list. Audited by resolve/JUDGE as anti-commitments.
-
-## Verification
-- Concrete commands whose expected behavior is named.
-```
-
-`complexity` is the resolve spec contract enum, not the benchmark difficulty
-label. Use `trivial`, `medium`, or `high` for new fixtures; `large` is accepted
-only for compatibility with external/legacy specs. Keep ambiguous calibration
-labels in `metadata.difficulty`, not spec frontmatter.
-
-## task.txt
-
-Bare-arm input. Plain English, same intent, but framed as a user request rather than a formal spec. Intentionally lacks the structured Requirements/Constraints/Out-of-Scope sections — bare must make those calls itself. Must not leak "use the devlyn skill" hints.
-
-## expected.json
-
-Machine-readable acceptance criteria used by both `run-fixture.sh` verification and the judge's rubric anchor.
-
-```json
-{
-  "verification_commands": [
-    {
-      "cmd": "node bin/cli.js doctor",
-      "exit_code": 0,
-      "stdout_contains": ["doctor: "],
-      "stdout_not_contains": ["undefined"],
-      "contract_refs": []
-    }
-  ],
-  "forbidden_patterns": [
-    {
-      "pattern": "catch\\s*\\(\\s*[a-zA-Z_]*\\s*\\)\\s*\\{\\s*return",
-      "description": "silent catch returning a fallback value — violates no-silent-catches policy",
-      "files": ["bin/cli.js"],
-      "severity": "disqualifier"
-    }
-  ],
-  "required_files": ["bin/cli.js"],
-  "forbidden_files": [],
-  "tier_a_waivers": [],
-  "spec_output_files": ["bin/cli.js"],
-  "max_deps_added": 0
-}
-```
-
-- **verification_commands** — runner executes each. Each command's pass/fail contributes to the arm's `verify_score`.
-  Commands run with `BENCH_WORKDIR` (fresh arm work tree) and
-  `BENCH_FIXTURE_DIR` (the fixture directory outside the arm work tree) in
-  the environment. Put discriminator/oracle scripts under the fixture
-  directory when the arm should not read the verifier source; any
-  `$BENCH_FIXTURE_DIR/...` file path referenced by a command must exist and
-  must not escape the fixture directory. Hidden oracle commands must reference
-  the verifier through an explicit `$BENCH_FIXTURE_DIR/...` path rather than
-  `cd "$BENCH_FIXTURE_DIR"` indirection.
-  Any command that references `BENCH_FIXTURE_DIR` is a hidden oracle and must
-  include `contract_refs`: exact substrings from `spec.md` proving the oracle
-  tests a visible contract rather than inventing a narrower one. Hidden oracle
-  commands must also assert `stdout_contains: ["\"ok\":true"]` so a verifier
-  cannot pass silently without emitting the success sentinel.
-- **forbidden_patterns** — regexes scanned across `diff.patch`. Match at `severity: "disqualifier"` is a hard-floor fail. Match at `severity: "warning"` goes into the judge's critical-findings report.
-- **required_files** — must exist after the arm runs.
-- **forbidden_files** — must NOT appear in the arm's diff.
-- **tier_a_waivers** — optional globs for files the spec explicitly authorizes even though Tier A scope oracle would normally flag them.
-- **spec_output_files** — files or globs that define the authorized output surface for Tier B scope tracing.
-- **max_deps_added** — count of new entries under `dependencies`/`devDependencies` in `package.json`. Exceeds → hard-floor fail.
-
-## high-risk metadata
-
-Fixtures with `metadata.json` `category: "high-risk"` must include at least
-one resolve risk-trigger term in `metadata.intent` or `spec.md`, matching the
-conditional risk-probe triggers used by `/devlyn:resolve`: security/auth,
-money/pricing/tax/ledger, persistence/data mutation, idempotency/replay,
-API/webhook/signature, allocation/scheduling/inventory/rollback/transaction,
-priority, or output/response-shape contracts. This keeps future pair-evidence
-candidates from relying on a label that would not actually activate the pair
-path.
-
-## NOTES.md
-
-Human-readable explanation of why this fixture exists. Must answer:
-
-1. What specific failure mode does this fixture detect?
-2. What pipeline phase(s) is this testing?
-3. Why can't another fixture cover this?
-4. When should this fixture be retired or replaced?
-
-Notes are read during suite design review, not during runs.
-
-If `NOTES.md` records that a fixture failed a headroom gate or was rejected as
-pair-lift evidence, add the fixture to
-`benchmark/auto-resolve/scripts/pair-rejected-fixtures.sh` in the same change.
-Rejected controls should remain replayable, but they must not be silently
-re-spent as fresh pair candidates.
-
-## Retired fixtures
-
-Move fixtures that are no longer valid active golden-suite evidence to
-`fixtures/retired/<fixture-id>/`. Retired fixtures are preserved for replay and
-historical diagnosis, but `run-suite.sh` does not auto-discover them. Each
-retired fixture must keep the six fixture files plus `RETIRED.md` explaining
-the run id or concrete reason it left the active suite.
-
-## setup.sh
-
-Deterministic starting state. Runs against a fresh copy of `benchmark/auto-resolve/fixtures/test-repo/` before either arm starts. Common uses:
-
-- Install extra deps (`npm install --prefix . something`).
-- Apply a `.patch` that introduces a bug to fix.
-- Create pre-existing files referenced by the spec.
-
-Script must be idempotent when re-applied. Empty file (just `#!/usr/bin/env bash\nset -e\n`) is valid when no setup needed.
-
----
-
-## Drift Prevention
-
-A CI lint step (`scripts/lint-fixtures.sh`) verifies:
-
-- All six files present per fixture.
-- `metadata.intent` substring appears in both `spec.md::Context` and `task.txt` (≥ 60% token overlap using simple tokenization).
-- `spec.md` frontmatter `id` matches directory name.
-- `expected.json` is valid JSON.
-- Active high-risk fixtures include a resolve risk-trigger term in
-  `metadata.intent` or `spec.md`.
-- Active fixtures whose `NOTES.md` records headroom-gate failure or pair-lift
-  rejection are covered by `pair-rejected-fixtures.sh`.
-- Active fixtures whose `NOTES.md` records `pair_evidence_passed` include an
-  actionable solo-headroom hypothesis in `spec.md`, using the same checker as
-  shadow candidates, and the hypothesis observable command must match a
-  `verification_commands[].cmd` entry in `expected.json`.
-- `setup.sh` is executable.
-- Retired fixtures under `fixtures/retired/F*/` keep `RETIRED.md`, preserve the
-  six fixture files, and are excluded from active suite discovery.

package/benchmark/auto-resolve/fixtures/retired/F27-cli-subscription-proration/NOTES.md

@@ -1,37 +0,0 @@
-# F27 CLI subscription proration
-
-## Failure mode
-
-This fixture detects billing implementations that look correct on one happy
-path but mishandle date boundaries, per-segment rounding, duplicate credits, or
-hardcode plan and tax rules instead of reading the seeded data file.
-
-## Pipeline phase target
-
-PLAN must separate input validation, period segmentation, per-segment proration,
-credit de-duplication, tax calculation, and output formatting. VERIFY should
-probe date boundary and data-source variants because a small example can pass
-while production invoices are off by one day or one cent.
-
-## Why existing fixtures do not cover it
-
-F25 covers cart promotions and F26 covers payout ledger events. This fixture
-adds subscription billing proration: effective-date segmentation, period-day
-denominators, credit idempotency, and tax after credits. It was intended to
-cover a pair-risk-probe gap, but the first real headroom smoke showed the
-visible contract was explicit enough for `solo_claude` to solve cleanly.
-
-## Retirement
-
-Retire or replace if both bare and solo consistently exceed the headroom
-thresholds, or if a later billing fixture provides the same proration and
-idempotent-credit signal with lower wall time.
-
-## Measurement notes
-
-- `20260511-f27-headroom-smoke-061401`: headroom FAIL. Judge scores were
-  bare 33 / solo_claude 94 (`solo_over_bare` +61). Bare passed 1 of 3
-  verification commands; solo passed 3 of 3 with terminal `PASS`.
-- Do not spend a pair arm on this fixture in its current shape. It needs either
-  a harder visible contract that solo misses without pair probes, or rotation
-  out of the pair-candidate set.

package/benchmark/auto-resolve/fixtures/retired/F27-cli-subscription-proration/RETIRED.md

@@ -1,13 +0,0 @@
-# Retired: F27 CLI subscription proration
-
-Retired from the active golden suite after headroom smoke
-`20260511-f27-headroom-smoke-061401`.
-
-Reason: `solo_claude` scored 94, exceeding the headroom ceiling of 80, while
-`bare` scored 33 and passed only 1 of 3 verification commands. The fixture is
-too explicit for current solo/pair lift measurement and too expensive to keep
-in the default suite.
-
-Future use: rework the visible contract so it creates a fair pair-risk-probe
-gap, or replace it with a different billing fixture. Do not count this fixture
-as pair evidence in its current form.