devlyn-cli 3.2.1 → 3.3.1

package/config/skills/devlyn:resolve/references/task-completion.md

@@ -7,22 +7,24 @@ authorized by the task scope; no additional approval ceremony is required.
 
 ## Allocate before work
 
-Use the installed `_shared/task-complete.py` from the retained base checkout,
+Use the installed `_shared/task-complete.py` from the anchor checkout,
 before committing owner inputs or starting direct/full work:
 
 ```sh
 python3 "$DEVLYN_SHARED_DIR/task-complete.py" allocate --repo . \
   --task '<task identity>' --branch '<absent task branch>' \
-  --repository '<owner/repo>' --remote origin --base main
+  --worktree '<absent path>' --repository '<owner/repo>' --remote origin --base main
 ```
 
-This creates and switches to a native task branch. Add `--worktree <absent path>`
-only when the owner explicitly chooses a linked checkout; allocation is optional
-for worktrees. Save the returned receipt path, outside the task checkout under
-the common Gitdir. Existing branches/trees cannot be adopted, even when their
-names look generated. An interrupted allocation stays blocked for inspection;
-do not delete its receipt and enroll the resulting branch. Pre-existing tasks
-remain owner-managed and cannot use this helper retroactively.
+Every task owns a linked worktree; `--worktree` is required. Its baseline is the
+exact fetched remote base, independent of the anchor's branch or dirty state;
+allocation leaves the anchor's HEAD, index and files untouched. Save the returned
+receipt path under the common Gitdir. `reconciled` reports earlier accepted,
+PR-delivered tasks whose merged resources were cleaned or retained; it is
+informational, so never resume or release a receipt you do not own. Existing
+branches/trees cannot be adopted, even when their names look generated. An
+interrupted allocation stays blocked for inspection; do not delete its receipt
+and enroll the resulting branch. Unreceipted tasks remain owner-managed.
 
 Allocation also returns a receipt-owned `scratch` directory. Put disposable
 build intermediates there (for example, set
@@ -48,8 +50,9 @@ history are never adopted or swept. A refused cleanup does not block source
 delivery; it stays visible in the handoff with its receipt and retry command
 (`CLEANUP_PENDING` after delivered source). No task is described as cleaned
 while owned disposable files remain. On resumption, finish any pending scratch
-cleanup before allocating more temporary storage. Rebuild from retained inputs
-when needed; cleanup is not an instruction to restart parked work.
+cleanup before allocating more temporary storage; the next allocation also
+retries it for completed deliveries with released writers. Rebuild from retained
+inputs when needed; cleanup is not an instruction to restart parked work.
 
 ## Accept a scoped commit
 
@@ -87,9 +90,14 @@ Complete once per item after that commit. Failed items are never published.
 
 ## Deliver and resume
 
+Wait for your children, stop your dev servers and task writers, then leave the
+task tree. Make the FIRST completion call from outside it with the installed
+helper and `--writers-stopped`; this releases the tree for cleanup after merge:
+
 ```sh
 python3 "$DEVLYN_SHARED_DIR/task-complete.py" complete \
-  --receipt '<returned receipt.json>' --acceptance .devlyn/acceptance.json
+  --receipt '<returned receipt.json>' \
+  --acceptance '<task worktree>/.devlyn/acceptance.json' --writers-stopped
 ```
 
 Project policy is `git config --local devlyn.completionMode auto|pr`; absent means
@@ -98,12 +106,14 @@ Use `complete --mode auto|pr` for one task, or `--local-only`/`--no-push` to hon
 the user's delivery restriction. These per-task choices persist in the receipt.
 
 `pr` pushes the exact accepted task ref and creates/reuses its exact repository,
-head and base PR, then reports `PR` with its URL and retains the checkout. `auto`
-also requests `gh pr merge --auto --merge --match-head-commit <accepted SHA>`.
-Repository checks/reviews and merge policy remain authoritative: no admin bypass,
-strategy fallback or repository setting changes. A repository that disallows
-merge commits fails visibly; disabled auto merge may also reject a pending
-request. Command success alone never proves merge.
+head and base PR. While OPEN, it cancels owned auto-merge, reports `PR` with its URL
+and retains the checkout. Once MERGED by any method, either mode cleans released
+resources in that call, on resume or at the next allocation. Default `auto` also requests
+`gh pr merge --auto --merge --match-head-commit <accepted SHA>`. Repository
+checks/reviews and merge policy remain authoritative: no admin bypass, strategy
+fallback or repository setting changes. When the repository disallows merge
+commits or refuses the request, delivery reports `PR` with `merge_refused` and
+the PR waits for a person. Command success alone never proves merge.
 
 `PENDING` retains the workspace and reports a receipt-based resume command.
 `BLOCKED` includes the cause and the same retry path; repair the reported condition
@@ -113,30 +123,29 @@ immutable product result; do not rewrite an archived PASS for a delivery error.
 
 ## Yield and retain recoverability
 
-After actual matching MERGED evidence, wait for the owner's actual children,
-stop/yield known task writers, and invoke completion with `--writers-stopped`.
-For a removable linked checkout, run from outside it using the installed helper
-outside that checkout. An OS process observation supplements the owner assertion;
-neither the receipt nor a scan guarantees exclusion of future writers. Unknown
-writers, inaccessible process state, caller cwd inside the tree, dirty/untracked
-files, changed refs/Gitdir/registration or locks retain the affected resources.
-On native Windows, locking and file durability are supported, but the helper cannot prove writer cessation and reports `writer observation unsupported on this platform; retain workspace`. Even `--writers-stopped` cannot authorize deletion without that observation: retain the workspace, refs and external receipt/custody, report delivery separately, and preserve the receipt-based resume command. Do not kill unknown processes or force worktree removal to make completion pass.
+Cleanup requires actual matching MERGED evidence; linked-tree removal also
+requires the owner's recorded release. OS observation supplements it; neither the receipt
+nor a scan guarantees exclusion of future writers. Processes with cwd or open
+files inside the tree, inaccessible process state (on Linux, other users'
+processes unless run as root), caller cwd inside the tree, dirty/untracked files,
+a nested repository or worktree, or changed refs/Gitdir/registration retain
+affected resources. Use `git worktree lock` to keep a tree.
+On native Windows, locking and file durability are supported, but the helper cannot prove writer cessation and reports `writer observation unsupported on this platform; retain workspace`. Even `--writers-stopped` cannot authorize deletion without that observation: retain the workspace, its task refs and external receipt/custody, report delivery separately, and preserve the receipt-based resume command. Do not kill unknown processes or force worktree removal to make completion pass.
 
 The receipt directory holds byte-verified `custody/`, `manifest.json` and a guarded
 `refs/devlyn/completed/<id>` recovery ref. Archive/check evidence is copied there
-before publication; removal rechecks it and source reachability through the merge.
-Ignored files in removable trees must be covered by that exact owned evidence
-manifest. Unknown ignored data requires explicit owner custody outside the helper;
-it never silently archives or deletes arbitrary user content. A partial/corrupt
-custody copy blocks cleanup and retains the original for repair.
-
-In-place completion retains the checkout and its ignored user data, switches
-safely to updated base, then compare-deletes only its exact owned task refs.
-Linked completion uses native non-force removal. Local ref deletion is guarded
-by the expected SHA; remote deletion uses an exact force-with-lease solely as
+before publication; removal rechecks it, the recovery ref and the merge commit on base.
+Ignored build output and other ignored files are disposable under native
+non-force worktree removal. The tree's `.devlyn/` files are byte/mode-verified
+and preserved as `<receipt dir>/records/.devlyn`; non-regular files or symlinks
+there retain the tree. Partial/corrupt custody or records retain it for repair.
+
+Legacy in-place receipts retire refs only once their task branch is no longer
+checked out; their checkout is never modified. Local ref deletion is guarded by
+the expected SHA; remote deletion uses an exact force-with-lease solely as
 compare-and-delete, never for product publication. A changed remote branch is
-retained. Retry after removal uses the external receipt and evidence, so the
-removed checkout is not needed. Recovery refs/evidence have no automatic expiry.
+retained. Retry uses the external receipt/evidence after removal. Recovery
+refs/evidence have no automatic expiry.
 
 Queue drains remain serial. Record source verdict and delivery status separately;
 retain pending/PR workspaces. Before advancing, use a separate owned branch from

package/{config/skills → optional-skills}/code-health-standards/SKILL.md

@@ -1,15 +1,11 @@
+---
+name: code-health-standards
+description: Maintainability standards for code you write or change — prevent dead code and orphaned references, justify every new dependency, keep functions, files, nesting and parameter counts within complexity thresholds, follow the established naming conventions, and keep debug logging, commented-out code, swallowed errors and hardcoded secrets out of production code. Use when writing, refactoring or deleting code, adding a dependency, or creating new files or modules.
+---
 # Code Health Standards
 
 Standards for keeping codebases lean and maintainable. Apply these thresholds during development — catching debt early is cheaper than cleaning it later.
 
-## Trigger
-
-- Writing new code or modifying existing code
-- Adding dependencies
-- Creating new files or modules
-- Refactoring or restructuring code
-- Any use of `/devlyn:clean`, `/devlyn:resolve`, or `/devlyn:review`
-
 ## Dead Code Prevention
 
 When writing code, check that you aren't creating orphans:
@@ -67,8 +63,3 @@ Code committed to production should not contain:
 - Empty catch blocks or swallowed errors
 - Hardcoded secrets, API keys, or environment-specific URLs
 - Type `any` (TypeScript) without a justifying comment
-
-## Routing
-
-- **Active cleanup**: Use `/devlyn:clean` to scan and remove accumulated debt
-- **Focused cleanup**: Use `/devlyn:clean [category]` for targeted sweeps (dead code, deps, tests, complexity, hygiene)

package/{config/skills → optional-skills}/code-review-standards/SKILL.md

@@ -1,14 +1,11 @@
+---
+name: code-review-standards
+description: Severity framework (CRITICAL security, HIGH code quality, MEDIUM best practice, LOW cleanup) and approval criteria for judging a code change. Use when reviewing, auditing or validating a diff, pull request or just-finished implementation, or when deciding whether a change is ready to approve.
+---
 # Code Review Standards
 
 Severity framework and quality bar for reviewing code changes. Apply this framework whenever reviewing, auditing, or validating code.
 
-## Trigger
-
-- Post-implementation review
-- Code review requests
-- PR review or diff analysis
-- Any use of `/devlyn:review` or `/devlyn:team-review`
-
 ## Severity Framework
 
 ### CRITICAL — Security (blocks approval)
@@ -57,8 +54,3 @@ Severity framework and quality bar for reviewing code changes. Apply this framew
 - MEDIUM issues are fixed or have concrete justification for deferral
 - Lint passes
 - Test suite passes
-
-## Routing
-
-- **Quick review** (few files, straightforward changes): Use `/devlyn:review`
-- **Thorough review** (many files, security-sensitive, user-facing): Use `/devlyn:team-review` for multi-perspective coverage

package/{config/skills → optional-skills}/root-cause-analysis/SKILL.md

@@ -1,15 +1,11 @@
+---
+name: root-cause-analysis
+description: Evidence-first debugging method — trace a symptom through successive why-steps to the actionable root cause, cite file:line for every claim, and fix the cause instead of masking the symptom. Use when diagnosing a bug, error, failing test, stack trace or unexpected behavior, or when asked why something happens.
+---
 # Root Cause Analysis
 
 Standard methodology for investigating bugs, issues, and unexpected behavior. Apply this framework whenever diagnosing a problem.
 
-## Trigger
-
-- User reports a bug or unexpected behavior
-- Error logs or stack traces need diagnosis
-- "Why does X happen?" or "What's causing X?" questions
-- Debugging sessions
-- Any use of `/devlyn:resolve`
-
 ## 5 Whys Protocol
 
 For every issue, trace from symptom to root cause:
@@ -59,7 +55,3 @@ Every fix MUST address the root cause. Stop immediately if you catch yourself:
 If the task is unreasonable or infeasible, or if any of the tests are incorrect, inform the user rather than working around them. The solution should be robust, maintainable, and extendable.
 
 If the real fix requires significant refactoring, present the scope to the user — never ship a workaround "for now".
-
-## Routing
-
-- Use `/devlyn:resolve` — handles both simple and multi-module investigations; VERIFY pair-judge fires conditionally on high-risk / complex specs.

package/{config/skills → optional-skills}/ui-implementation-standards/SKILL.md

@@ -1,15 +1,11 @@
+---
+name: ui-implementation-standards
+description: Quality bar for building or changing user interfaces — design-system token fidelity, WCAG 2.1 AA accessibility, loading/empty/error/success/disabled state coverage, purposeful animation, and mobile-first responsive layout. Use when implementing or modifying UI components, pages or styles, or when turning a design or design system into code.
+---
 # UI Implementation Standards
 
 Quality framework for building and improving UI from design systems. Apply these standards whenever implementing or modifying visual components.
 
-## Trigger
-
-- Building UI components or pages
-- Implementing designs from a design system
-- Improving or refactoring existing UI
-- Any use of `/devlyn:design-ui`
-- Frontend development tasks involving visual design
-
 ## Design System Fidelity
 
 When a design system exists at `docs/design-system.md`:
@@ -66,9 +62,3 @@ Every interactive component or data-dependent view needs all states:
 - No inline styles — use the token/theme system
 - Server components where possible (Next.js), client components only for interactivity
 - Keep component files focused — one component per file
-
-## Routing
-
-- **Build new UI or improve existing**: Use `/devlyn:implement-ui` for a full team approach
-- **Add features to existing UI**: Use `/devlyn:resolve` with the feature description
-- **Review UI code quality**: Use `/devlyn:team-review` for multi-perspective code review

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "devlyn-cli",
-  "version": "3.2.1",
+  "version": "3.3.1",
   "description": "Loop & harness engineering for AI coding agents (Claude Code, Codex, and any AGENTS.md-driven agent) — take intent to shipped, engineer-quality software hands-free with phase-gated pipelines and agent orchestration",
   "homepage": "https://github.com/fysoul17/devlyn-cli#readme",
   "bin": {
@@ -18,65 +18,9 @@
     "!config/skills/roadmap-archival-workspace",
     "!config/skills/roadmap-archival-workspace/**",
     "optional-skills",
-    "benchmark/auto-resolve/BENCHMARK-DESIGN.md",
-    "benchmark/auto-resolve/BENCHMARK-RESULTS.md",
-    "benchmark/auto-resolve/README.md",
-    "benchmark/auto-resolve/RUBRIC.md",
-    "benchmark/auto-resolve/run-real-benchmark.md",
-    "benchmark/auto-resolve/fixtures/SCHEMA.md",
-    "benchmark/auto-resolve/fixtures/F*/**",
-    "benchmark/auto-resolve/fixtures/retired/F*/**",
-    "benchmark/auto-resolve/shadow-fixtures/S*/**",
-    "benchmark/auto-resolve/fixtures/test-repo/**",
-    "!benchmark/auto-resolve/fixtures/test-repo/node_modules/**",
-    "benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/headroom-gate.json",
-    "benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/full-pipeline-pair-gate.md",
-    "benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/full-pipeline-pair-gate.json",
-    "benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/F16-cli-quote-tax-rules/l2_risk_probes/result.json",
-    "benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/F23-cli-fulfillment-wave/l2_risk_probes/result.json",
-    "benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/F25-cli-cart-promotion-rules/l2_risk_probes/result.json",
-    "benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/headroom-gate.json",
-    "benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/full-pipeline-pair-gate.md",
-    "benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/full-pipeline-pair-gate.json",
-    "benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/F21-cli-scheduler-priority/l2_risk_probes/result.json",
-    "benchmark/auto-resolve/results/20260507-f10-f11-tier1-full-pipeline/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260507-f10-f11-tier1-full-pipeline/headroom-gate.json",
-    "benchmark/auto-resolve/results/20260508-f22-exact-error-headroom/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260508-f22-exact-error-headroom/headroom-gate.json",
-    "benchmark/auto-resolve/results/20260508-f26-headroom/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260508-f26-headroom/headroom-gate.json",
-    "benchmark/auto-resolve/results/20260511-f3-http-error-headroom/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260511-f3-http-error-headroom/headroom-gate.json",
-    "benchmark/auto-resolve/results/20260511-f12-webhook-headroom/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260511-f12-webhook-headroom/headroom-gate.json",
-    "benchmark/auto-resolve/results/20260511-f15-concurrency-headroom/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260511-f15-concurrency-headroom/headroom-gate.json",
-    "benchmark/auto-resolve/results/20260512-f2-medium-headroom/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260512-f2-medium-headroom/headroom-gate.json",
-    "benchmark/auto-resolve/results/20260512-f4-web-headroom/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260512-f4-web-headroom/headroom-gate.json",
-    "benchmark/auto-resolve/results/20260512-f5-fixloop-headroom/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260512-f5-fixloop-headroom/headroom-gate.json",
-    "benchmark/auto-resolve/results/20260512-f6-checksum-headroom/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260512-f6-checksum-headroom/headroom-gate.json",
-    "benchmark/auto-resolve/results/20260512-f7-scope-headroom/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260512-f7-scope-headroom/headroom-gate.json",
-    "benchmark/auto-resolve/results/20260512-f9-e2e-headroom/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260512-f9-e2e-headroom/headroom-gate.json",
-    "benchmark/auto-resolve/results/20260512-f31-seat-rebalance-headroom/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260512-f31-seat-rebalance-headroom/headroom-gate.json",
-    "benchmark/auto-resolve/results/20260512-f32-subscription-renewal-headroom/headroom-gate.md",
-    "benchmark/auto-resolve/results/20260512-f32-subscription-renewal-headroom/headroom-gate.json",
-    "benchmark/auto-resolve/scripts/**",
     "!**/__pycache__",
     "!**/__pycache__/**",
     "!**/*.pyc",
-    "scripts/lint-fixtures.sh",
-    "scripts/lint-shadow-fixtures.sh",
-    "scripts/lint-skills.sh",
-    "scripts/test-owner-phases.py",
     "CLAUDE.md",
     "AGENTS.md"
   ],

package/benchmark/auto-resolve/BENCHMARK-DESIGN.md

@@ -1,289 +0,0 @@
-# Benchmark Suite Design — v1
-
-**Outer goal**: see [`autoresearch/NORTH-STAR.md`](../../autoresearch/NORTH-STAR.md) — the harness composes frontier LLMs into a hands-free pipeline that delivers engineer-quality software for users who do not know context engineering, with each composition layer (L0 bare → L1 solo harness → L2 pair harness) justifying its added cost on quality AND wall-time efficiency. This benchmark is the measurement instrument for that contract.
-
-**Purpose.** Replace ad-hoc harness benchmarking with a permanent, comprehensive,
-one-command suite that gates every future harness change with a ship/rollback
-decision. Any prompt edit, phase reorder, new native skill, or model upgrade
-can be validated by running the suite and reading the numbers.
-
-**Arm structure.** Current full-pipeline evidence uses three arms: `bare` (L0),
-`solo_claude` (L1 solo harness), and an L2 pair arm (`variant` in the smoke
-suite, or a focused pair arm such as `l2_risk_probes` in pair-candidate runs).
-Pair claims are headroom-gated: counted fixtures must leave room above solo
-(`bare <= 60`, `solo_claude <= 80`, default 5-point `bare`/`solo_claude` headroom margins),
-the pair arm must actually run, and blind judging must show pair above solo by
-the configured margin.
-
-**Non-goals.** Publishable-research statistical rigor. Not a regression test
-library for the product code — those live elsewhere. Not a substitute for
-production telemetry — just enough signal for ship decisions.
-
----
-
-## Principles
-
-1. **One command.** `npx devlyn-cli benchmark` runs everything and prints a
-   verdict. No manual fixture setup.
-2. **Novice-proof.** The suite exercises the same paths a first-time user
-   hits — including an end-to-end `ideate → resolve` fixture.
-3. **LLM-upgrade friendly.** Rubric, fixture semantics, and thresholds stay
-   stable; scores and margins float up as models improve. Nothing is
-   hardcoded to a specific model version.
-4. **Karpathy.** No fixture earns its place unless it tests a distinct
-   failure mode. Tooling stays boring. History plumbing is simple.
-5. **Ship gate is numbers, not vibes.** Concrete thresholds in RUBRIC.md.
-
----
-
-## Directory Layout
-
-```
-benchmark/auto-resolve/
-├── BENCHMARK-DESIGN.md       # this file
-├── README.md                 # how to run, interpret, extend
-├── RUBRIC.md                 # stable judge rubric + ship gates
-│
-├── fixtures/
-│   ├── SCHEMA.md             # fixture file format
-│   ├── test-repo/            # bootstrap Node project (shared base)
-│   │   ├── bin/cli.js
-│   │   ├── server/index.js
-│   │   ├── web/page.html
-│   │   ├── tests/
-│   │   ├── playwright.config.js
-│   │   └── package.json
-│   │
-│   ├── F1-cli-trivial-flag/
-│   ├── F2-cli-medium-subcommand/
-│   ├── F3-backend-contract-risk/
-│   ├── F4-web-browser-design/
-│   ├── F5-fix-loop-red-green/
-│   ├── F6-dep-audit-native-module/
-│   ├── F7-out-of-scope-trap/
-│   ├── F8-known-limit-ambiguous/
-│   ├── F9-e2e-ideate-to-resolve/
-│   └── F10+ extensions for headroom, full-pipeline pair, and frozen VERIFY
-│
-├── scripts/
-│   ├── run-suite.sh          # smoke entry — runs fixture arms + judge + report
-│   ├── run-fixture.sh        # one fixture, one arm
-│   ├── judge.sh              # Codex blind judge (model-agnostic)
-│   ├── compile-report.py     # aggregate into report.md + summary.json
-│   └── ship-gate.py          # apply thresholds, return ship/rollback verdict
-│
-├── results/                  # per-run artifacts (overwritten)
-│   └── <run-id>/
-│       ├── <fixture>/
-│       │   ├── bare/{input.md, transcript.txt, diff.patch, result.json}
-│       │   ├── solo_claude/{same}
-│       │   └── variant or l2_risk_probes/{same}
-│       ├── <fixture>/judge.json
-│       ├── report.md
-│       └── summary.json
-│
-└── history/
-    ├── runs/                 # append-only immutable records
-    │   └── 2026-04-23T120000Z-v3.6.json
-    ├── latest.json           # pointer to most recent run
-    └── baselines/
-        └── shipped.json      # last blessed version, used for regression check
-```
-
----
-
-## Fixture Schema
-
-Every fixture is a directory with these files (see `fixtures/SCHEMA.md`):
-
-| File | Purpose |
-|------|---------|
-| `metadata.json` | id, category, difficulty, timeout, required tools, intent block |
-| `spec.md` | pipeline-arm input (resolve-ready spec with Requirements/Constraints/Out-of-Scope/Verification) |
-| `task.txt` | bare-arm input (same intent, natural-language framing) |
-| `expected.json` | machine-readable acceptance criteria + forbidden patterns + verification commands |
-| `NOTES.md` | why this fixture exists, the specific failure mode it tests |
-| `setup.sh` | deterministic starting state — applies to a fresh copy of `test-repo/` |
-
-**Drift prevention**: `spec.md` and `task.txt` both derive from the same
-`intent` block in `metadata.json`. A lint step in CI verifies they stay
-consistent.
-
----
-
-## Core Fixtures And Extensions
-
-The original v3.6 matrix covered F1-F9. Later fixtures extend the same schema
-for headroom, full-pipeline pair, and frozen VERIFY evidence.
-
-Category coverage matrix for the original core set (rows = concerns, columns =
-fixtures):
-
-| Fixture | Trivial | Medium | High-risk | Stress | Edge | E2E |
-|---------|---------|--------|-----------|--------|------|-----|
-| F1-cli-trivial-flag | ✓ | | | | | |
-| F2-cli-medium-subcommand | | ✓ | | | | |
-| F3-backend-contract-risk | | | ✓ | | | |
-| F4-web-browser-design | | | | ✓ (browser-validate) | | |
-| F5-fix-loop-red-green | | | | ✓ (FIX LOOP) | | |
-| F6-dep-audit-native-module | | | | ✓ (CRITIC security dep audit) | | |
-| F7-out-of-scope-trap | | | | ✓ (scope discipline) | | |
-| F8-known-limit-ambiguous | | | | | ✓ (documents where pipeline may lose) | |
-| F9-e2e-ideate-to-resolve | | | | | | ✓ (novice full-flow) |
-
-**F9 is load-bearing** for the "novice user types `/devlyn:ideate`" promise.
-Input is a vague idea; the pipeline path turns it into a spec with ideate and
-then resolves that spec. Bare arm runs a direct prompt. Judge compares the final
-usable artifact set.
-
----
-
-## Single-Command Invocation
-
-### User experience
-
-```bash
-npx devlyn-cli benchmark            # n=1 smoke, all fixtures
-npx devlyn-cli benchmark F2 F5      # specific fixtures only
-npx devlyn-cli benchmark --judge-only --run-id <id>   # re-judge without re-running
-```
-
-Output on completion:
-
-```
-Benchmark Suite Run — 2026-04-23T12:00Z (v3.6)
-Judge: codex CLI flagship, xhigh, blind (model recorded in run history)
-
-Fixture                         variant (L2)  solo_claude (L1)  bare (L0)  variant-solo_claude  Verdict
-F1-cli-trivial-flag             95            92                88         +3                   PASS
-F2-cli-medium-subcommand        92            86                81         +6                   PASS
-F3-backend-contract-risk        89            80                72         +9                   PASS
-F4-web-browser-design           87            83                79         +4                   PASS
-F5-fix-loop-red-green           91            78                65         +13                  PASS
-F6-dep-audit-native-module      88            82                70         +6                   PASS
-F7-out-of-scope-trap            94            85                73         +9                   PASS
-F8-known-limit-ambiguous        78            79                79         -1                   EXPECTED (known-limit)
-F9-e2e-ideate-to-resolve        90            84                68         +6                   PASS
----------------------------------------------------------
-Suite average variant (L2) score:       89.3
-Suite average solo_claude (L1) score:   83.2
-Suite average bare (L0) score:          75.0
-Suite average variant-solo_claude margin: +6.1  (pair-evidence floor: +5 on eligible fixtures)
-Hard-floor violations:        0
-Regression vs shipped:       n/a (first run of v3.6)
-SHIP-GATE VERDICT: ✅ PASS
-```
-
-### Runner orchestration
-
-`run-suite.sh`:
-
-1. Generate run-id `<ISO>-<sha>-<branch>`
-2. For each fixture × each arm (`variant`/L2, `solo_claude`/L1, `bare`/L0): parallelizable via `xargs -P`
-   - `run-fixture.sh --fixture FX --arm variant` → writes `results/<run-id>/FX/variant/*`
-3. For each fixture: `judge.sh FX <run-id>` → writes `results/<run-id>/FX/judge.json`
-4. `compile-report.py <run-id>` → writes `report.md` + `summary.json`
-5. `ship-gate.py <run-id>` → exit 0 (PASS) / 1 (FAIL). Prints verdict to stdout.
-6. If PASS and `--bless` flag: copy `summary.json` → `history/baselines/shipped.json`
-7. Always: append `history/runs/<run-id>.json` + update `latest.json`
-
-### `run-fixture.sh` contract
-
-- Creates fresh temp copy of `test-repo/` at `/tmp/bench-<run-id>-<fixture>-<arm>/`
-- Applies `setup.sh` if present
-- Copies `spec.md` for `variant`/`solo_claude` or `task.txt` for `bare` as the prompt
-- Invokes `/devlyn:resolve --spec` for `variant`, `/devlyn:resolve --spec --engine claude --no-pair --no-risk-probes` for `solo_claude`, or bare Claude for `bare` via isolated Agent
-- Captures: `diff.patch`, `changed-files.txt`, `transcript.txt`, `timing.json`
-- Runs `expected.json::verification_commands`, writes pass/fail per command to `verify.json`
-- Writes `result.json` with aggregate: exit code, duration, files changed, verification score
-
-### `judge.sh` contract
-
-- Reads `results/<run-id>/<fixture>/{variant,solo_claude,bare}/{diff.patch,verify.json}` + fixture's `spec.md` + `expected.json`
-- Builds a blind prompt: labels arms A and B randomly per fixture (seed recorded)
-- Invokes isolated Codex (current flagship — no model hardcode) with RUBRIC.md
-- Writes `judge.json`: per-axis scores, winner, margin, critical findings, disqualifiers
-- Idempotent: re-running overwrites the same `judge.json`
-
----
-
-## LLM-Upgrade Resilience
-
-Three mechanisms:
-
-1. **No hardcoded models.** Judge invocation omits `-m`, so it inherits
-   whichever flagship the CLI currently ships. The blind judge is isolated from
-   user config/rules/hooks so local agent instructions cannot contaminate the
-   judgment. Same for agents — they run against whatever Claude Code
-   session-model the caller has. Model provenance is captured in `result.json`
-   per run.
-
-2. **Margin as primary signal, absolute score as secondary.** When models
-   improve, all arms tend to get better. Pairwise margins remain the stable
-   signal: `solo_claude`-`bare` (L1-L0) measures solo harness value,
-   pair-`solo_claude` (L2-L1) measures pair value on eligible fixtures, and
-   `variant`-`bare` (L2-L0) remains the legacy suite signal. Ship gates are
-   defined on margin (`>= +5`) and regression (`-3 or worse`), not absolute
-   score.
-
-3. **Fixture difficulty gradient.** F1 (trivial) is expected to saturate near
-   100 quickly as models improve — that's fine, it still catches catastrophic
-   regressions. F5/F9 (stress/E2E) have enough depth that even a near-perfect
-   model won't 100-zero bare. If any fixture saturates (all compared gated arms
-   > 95 for two consecutive versions), we replace it with a harder one and
-   document the swap in `history/runs/<ts>-fixture-rotation.json`.
-
----
-
-## Ship Gates (from RUBRIC.md)
-
-Hard floors (any single failure blocks ship):
-
-- **No silent-catch / fabricated verification / skipped required test in variant.** Judge flags this as disqualifier.
-- **Variant may not lose any fixture by more than −5** versus previous shipped version (per-fixture regression floor).
-- **At least 7 gated, headroom-available fixtures** must have margin ≥ +5
-  (suite coverage).
-- **F9 (E2E) must PASS** — novice-flow contract.
-
-Soft gates (trigger rollback discussion):
-
-- Suite average margin drop > 3 vs last shipped.
-- Any fixture with margin ≤ 0 that previously had margin > +5.
-- Critical-finding catch-rate decrease vs the last shipped comparable arm.
-
-Known-limit exception:
-
-- F8 is explicitly allowed to tie or lose (margin in [-3, +3]). Its job is to
-  document honesty, not to beat bare.
-
----
-
-## Karpathy Check
-
-Where over-engineering lurks:
-
-- ❌ **Automatic history mutation during development.** Add append-only
-  history AFTER the suite format stabilizes (one version after initial ship).
-- ❌ **Statistical tooling beyond mean/median/margin.** n=1-3 doesn't need
-  t-tests.
-- ❌ **Auto-generated fixture cards / dashboards.** Plain `report.md` is enough.
-- ✅ **Keep scripts under 100 lines each** unless they're doing concrete,
-  repeated work the user would do by hand.
-
-If the suite tooling grows past ~800 total lines, prune aggressively before
-adding anything.
-
----
-
-## Open Questions (to be answered before first full ship-gate run)
-
-1. Where does `benchmark` subcommand live? Inside `bin/devlyn.js` or as
-   standalone `benchmark/auto-resolve/scripts/run-suite.sh` invoked via `npm
-   run`? **Proposal**: both — `bin/devlyn.js benchmark` is the advertised
-   entry, which shells out to the script.
-2. Parallel run safety — can we run the selected fixture set × 3 arms concurrently without
-   rate-limit / lockfile conflicts? **Proposal**: default sequential with
-   `--parallel N` flag. Default `N=1` for safety; the user can opt in.
-3. Token accounting — Claude Code doesn't expose subagent totals reliably.
-   **Proposal**: capture wall time as primary efficiency metric; token
-   estimate as best-effort secondary. Do not gate ship on token math alone.