devlyn-cli 3.2.1 → 3.3.0

package/benchmark/auto-resolve/fixtures/retired/F9-e2e-ideate-to-preflight/NOTES.md

@@ -1,60 +0,0 @@
-# F9 — Notes
-
-## Purpose
-
-**Load-bearing for the novice-user contract.** The suite ship-gate requires
-F9 to pass (margin ≥ +5) on every shipped version. If F9 fails, the "type
-`/devlyn:ideate` and ship worldclass software" promise is not being met.
-
-## What the variant arm does
-
-A novice-simulating prompt (`task.txt` is identical to what the user typed)
-is delivered to a fresh Claude session. The session has our skills installed.
-The pipeline arm is expected to:
-
-1. Recognize this is a vague idea, not a spec → invoke `/devlyn:ideate`.
-2. Ideate produces `docs/VISION.md`, `docs/ROADMAP.md`, and
-   `docs/roadmap/phase-1/1.1-gitstats.md` (or similar).
-3. Run `/devlyn:auto-resolve` on the generated spec.
-4. Run `/devlyn:preflight` for verification.
-
-The variant arm's prompt explicitly instructs this chain so we're not
-relying on Claude to invent it. That's fair because the novice contract is
-about the TOOLS being available + discoverable; the user in this benchmark
-is already primed to use them.
-
-## What the bare arm does
-
-Same raw task delivered as a direct prompt. Bare implements `gitstats`
-using its own judgment. Bare does NOT produce VISION/ROADMAP documents
-(and isn't expected to).
-
-## Why margin ≥ +5 is required
-
-The pipeline's whole value prop is that it trades some bare-case tokens for
-quality uplift on novice flows. If this fixture can't show ≥ +5 margin,
-we're paying pipeline cost without delivering on the novice promise.
-
-## Scoring notes
-
-- The variant's `docs/VISION.md` + `ROADMAP.md` + spec files ARE part of
-  the judge's evaluation. The judge sees the full product (code + docs +
-  roadmap state), not just the diff to `bin/cli.js`.
-- Bare doesn't produce roadmap files, so bare's judge payload is
-  code+test only. This asymmetry is INTENTIONAL — the fixture tests
-  total-output quality, not per-file quality.
-
-## Failure modes detected
-
-- **Pipeline skips ideate.** Variant goes straight to auto-resolve with a
-  vague spec → downstream implementation is weak. Caught by judge:
-  `docs/roadmap/` files missing.
-- **Bare over-engineers.** Without a skeleton, bare builds too much,
-  touches wrong files, adds deps. Caught by spec constraints.
-- **Pipeline ships "done" but preflight was a no-op.** If `.devlyn/PREFLIGHT-REPORT.md` exists but shows no commitment audit, something is broken upstream.
-
-## Rotation trigger
-
-F9 is the last fixture we rotate — it's the anchor. If it saturates
-(variant consistently > 95), the whole suite needs a harder novice-flow
-anchor before we retire this one.

package/benchmark/auto-resolve/fixtures/retired/F9-e2e-ideate-to-preflight/RETIRED.md

@@ -1,29 +0,0 @@
-# RETIRED — F9-e2e-ideate-to-preflight
-
-**Retired**: 2026-04-30 (iter-0033a)
-**Replaced by**: `benchmark/auto-resolve/fixtures/F9-e2e-ideate-to-resolve/`
-**Source SHA**: `8d4d57f` (commit before the 2-skill-contract rename).
-
-## Why retired
-
-The 2-skill redesign (Phases 1-3, iter-0029 / 0031 / 0032) replaced
-`/devlyn:ideate` (greenfield) and folded `/devlyn:preflight` into
-`/devlyn:resolve`'s VERIFY phase. The OLD F9 fixture's contract assumed
-the 3-skill chain (`/devlyn:ideate` → `/devlyn:auto-resolve` →
-`/devlyn:preflight`), which is unobtainable at HEAD post-iter-0032
-because OLD ideate was deleted.
-
-iter-0033a redesigned F9 to match the shipped 2-skill contract.
-
-## When to consult this archive
-
-- Replaying a regression suspected from the OLD chain.
-- Migrating a pre-2026-04-30 historical run record back to readable shape.
-- Auditing what changed when the new fixture's measurements diverge from
-  pre-redesign baselines.
-
-## What lives here
-
-The exact file contents of the F9 fixture as of `8d4d57f` (the last commit
-before the rename). DO NOT use this directory as a live fixture — it is
-not picked up by `run-suite.sh`. Restore-and-run requires a manual copy.

package/benchmark/auto-resolve/fixtures/retired/F9-e2e-ideate-to-preflight/expected.json

@@ -1,73 +0,0 @@
-{
-  "verification_commands": [
-    {
-      "cmd": "node bin/cli.js gitstats",
-      "exit_code": 0,
-      "stdout_contains": [
-        "Commits:",
-        "Last commit:"
-      ],
-      "stdout_not_contains": [
-        "Error:"
-      ]
-    },
-    {
-      "cmd": "node bin/cli.js gitstats --json",
-      "exit_code": 0,
-      "stdout_contains": [
-        "{",
-        "commits",
-        "authors"
-      ],
-      "stdout_not_contains": []
-    },
-    {
-      "cmd": "cd /tmp && node -e 'const { spawnSync } = require(\"child_process\"); const p = process.env.BENCH_WORKDIR || process.cwd(); console.log(spawnSync(\"node\", [p + \"/bin/cli.js\", \"gitstats\"], { encoding: \"utf8\", cwd: \"/tmp\" }).status)'",
-      "exit_code": 0,
-      "stdout_contains": [
-        "2"
-      ],
-      "stdout_not_contains": [
-        "0"
-      ]
-    },
-    {
-      "cmd": "node bin/cli.js hello",
-      "exit_code": 0,
-      "stdout_contains": [
-        "Hello, world!"
-      ],
-      "stdout_not_contains": []
-    },
-    {
-      "cmd": "node --test tests/",
-      "exit_code": 0,
-      "stdout_contains": [],
-      "stdout_not_contains": []
-    }
-  ],
-  "forbidden_patterns": [
-    {
-      "pattern": "catch\\s*\\([^)]*\\)\\s*\\{\\s*\\}",
-      "description": "empty catch",
-      "files": [
-        "bin/cli.js"
-      ],
-      "severity": "disqualifier"
-    }
-  ],
-  "required_files": [
-    "bin/cli.js"
-  ],
-  "forbidden_files": [],
-  "max_deps_added": 0,
-  "tier_a_waivers": [
-    "docs/VISION.md",
-    "docs/ROADMAP.md",
-    "docs/roadmap/**"
-  ],
-  "spec_output_files": [
-    "bin/**",
-    "tests/**"
-  ]
-}

package/benchmark/auto-resolve/fixtures/retired/F9-e2e-ideate-to-preflight/metadata.json

@@ -1,10 +0,0 @@
-{
-  "id": "F9-e2e-ideate-to-preflight",
-  "category": "e2e",
-  "difficulty": "high",
-  "timeout_seconds": 3600,
-  "required_tools": ["node"],
-  "browser": false,
-  "deps_change_expected": false,
-  "intent": "End-to-end novice flow: from a vague idea ('git stats CLI for the current repo') the variant must run /devlyn:ideate → /devlyn:auto-resolve → /devlyn:preflight to produce Vision/Roadmap + implemented code + preflight sign-off. The bare arm receives the same vague idea as a direct prompt. This fixture gates the novice-user contract."
-}

package/benchmark/auto-resolve/fixtures/retired/F9-e2e-ideate-to-preflight/setup.sh

@@ -1,28 +0,0 @@
-#!/usr/bin/env bash
-# F9 setup — seed a few synthetic commits with different authors so the
-# `gitstats` subcommand's "top 3 authors by commit count" requirement is
-# meaningfully exercised. Without this, every commit author is the runner's
-# default and the ranking test is a no-op.
-set -e
-
-commit_as() {
-  local name="$1" email="$2" file="$3" message="$4"
-  echo "$(date +%s%N) $name" >> "$file"
-  git add "$file"
-  git -c user.name="$name" -c user.email="$email" commit -q -m "$message"
-}
-
-mkdir -p .bench-seed
-
-commit_as "Alpha Author"   "alpha@bench.test"   .bench-seed/log "seed: alpha 1"
-commit_as "Alpha Author"   "alpha@bench.test"   .bench-seed/log "seed: alpha 2"
-commit_as "Alpha Author"   "alpha@bench.test"   .bench-seed/log "seed: alpha 3"
-commit_as "Alpha Author"   "alpha@bench.test"   .bench-seed/log "seed: alpha 4"
-commit_as "Beta Author"    "beta@bench.test"    .bench-seed/log "seed: beta 1"
-commit_as "Beta Author"    "beta@bench.test"    .bench-seed/log "seed: beta 2"
-commit_as "Beta Author"    "beta@bench.test"    .bench-seed/log "seed: beta 3"
-commit_as "Gamma Author"   "gamma@bench.test"   .bench-seed/log "seed: gamma 1"
-commit_as "Gamma Author"   "gamma@bench.test"   .bench-seed/log "seed: gamma 2"
-commit_as "Delta Author"   "delta@bench.test"   .bench-seed/log "seed: delta 1"
-
-echo "F9 setup: seeded 10 commits across 4 authors (Alpha 4 / Beta 3 / Gamma 2 / Delta 1)"

package/benchmark/auto-resolve/fixtures/retired/F9-e2e-ideate-to-preflight/spec.md

@@ -1,58 +0,0 @@
----
-id: "F9-e2e-ideate-to-preflight"
-title: "End-to-end: idea → shipped CLI feature"
-status: planned
-complexity: high
-depends-on: []
----
-
-# F9 End-to-End Novice Flow
-
-## Context
-
-A first-time user has a vague idea:
-
-> "I want a CLI subcommand that shows basic stats about the current git repo — commit count, last commit date, top 3 authors. Call it `gitstats`."
-
-The variant arm is expected to use the pipeline: `/devlyn:ideate` to
-produce a VISION/ROADMAP + spec, then `/devlyn:auto-resolve` to implement
-per spec, then `/devlyn:preflight` to verify. The bare arm receives the
-same idea as a direct prompt and implements it without the pipeline.
-
-This fixture is the suite's most important gate for the "novice user contract":
-a first-time user typing `/devlyn:ideate` should land at working, well-structured software.
-
-## Requirements
-
-- [ ] A new `gitstats` subcommand exists in `bin/cli.js`.
-- [ ] `node bin/cli.js gitstats` (run inside a git repo) prints:
-  - Line 1: commit count (e.g., `Commits: 42`).
-  - Line 2: last commit ISO date (e.g., `Last commit: 2026-04-23T12:00:00Z`).
-  - Lines 3-5: top 3 authors by commit count, format `<rank>. <name> <count>`.
-- [ ] Run outside a git repo → stderr message `Error: not a git repository` and exit 2.
-- [ ] `node bin/cli.js gitstats --json` emits valid JSON with the same data.
-- [ ] Existing subcommands (`hello`, `version`) unchanged.
-- [ ] Add at least one test.
-- [ ] For variant: a `docs/VISION.md`, `docs/ROADMAP.md`, and a `docs/roadmap/phase-1/` spec file must exist after the run (evidence the ideate stage happened).
-
-## Constraints
-
-- **No new npm dependencies.** Use `child_process` to shell out to `git`.
-- **No silent catches.**
-- **Non-git-repo handling.** Do not assume the user is always in a repo.
-
-- **Lifecycle note.** The harness's DOCS phase flips this spec's frontmatter `status` after implementation completes — that is benchmark lifecycle bookkeeping, not a scope violation.
-
-## Out of Scope
-
-- Parsing commit messages, tags, branches.
-- Remote API calls.
-- Touching `server/` or `web/`.
-
-## Verification
-
-- Inside this worktree (which IS a git repo): `node bin/cli.js gitstats` exits 0 and prints at least 5 lines of summary.
-- `node bin/cli.js gitstats --json | node -e 'const d=JSON.parse(require("fs").readFileSync(0,"utf8")); console.log(typeof d.commits)'` prints `number`.
-- `cd /tmp && node <worktree>/bin/cli.js gitstats` (from outside a repo — use the worktree's absolute path) exits 2.
-- For variant: `test -f docs/VISION.md && test -f docs/ROADMAP.md && ls docs/roadmap/phase-1/*.md | head -1`.
-- `node --test tests/` passes.

package/benchmark/auto-resolve/fixtures/retired/F9-e2e-ideate-to-preflight/task.txt

@@ -1,5 +0,0 @@
-I want a CLI subcommand that shows basic stats about the current git repo — commit count, last commit date, top 3 authors. Call it `gitstats`.
-
-Should work inside this repo when I run `node bin/cli.js gitstats`, and fail cleanly if I'm not in a git repo. A `--json` flag for machine-readable output would be useful too.
-
-Keep the existing `hello` and `version` subcommands working. Add a test. No new npm dependencies.

package/benchmark/auto-resolve/fixtures/test-repo/README.md

@@ -1,12 +0,0 @@
-# harbor-tools
-
-A small Node.js toolkit with a command-line interface, an HTTP server, and a
-minimal browser page.
-
-## Commands
-
-- `npm test` runs the test suite.
-- `npm run cli -- --help` prints command-line help.
-- `npm start` starts the HTTP server.
-
-The project intentionally keeps each component small and independent.

package/benchmark/auto-resolve/fixtures/test-repo/bin/cli.js

@@ -1,63 +0,0 @@
-#!/usr/bin/env node
-// harbor-tools — small command-line utilities for local workflows.
-// Keep the core commands minimal and easy to understand.
-
-const fs = require('fs');
-const path = require('path');
-
-const USAGE = `Usage: bench-cli <command> [options]
-
-Commands:
-  hello [--name NAME]        Print a greeting (default name: "world")
-  version                    Print the CLI version from package.json
-  --help, -h                 Show this help
-
-Examples:
-  bench-cli hello
-  bench-cli hello --name alice
-  bench-cli version
-`;
-
-function readPackageVersion() {
-  const pkgPath = path.join(__dirname, '..', 'package.json');
-  const raw = fs.readFileSync(pkgPath, 'utf8');
-  return JSON.parse(raw).version;
-}
-
-function parseNameFlag(argv) {
-  const idx = argv.indexOf('--name');
-  if (idx === -1) return 'world';
-  const value = argv[idx + 1];
-  if (!value || value.startsWith('-')) {
-    console.error('--name requires a value');
-    process.exit(1);
-  }
-  return value;
-}
-
-function main(argv) {
-  const [command, ...rest] = argv;
-
-  if (!command || command === '--help' || command === '-h') {
-    process.stdout.write(USAGE);
-    return;
-  }
-
-  switch (command) {
-    case 'hello': {
-      const name = parseNameFlag(rest);
-      console.log(`Hello, ${name}!`);
-      return;
-    }
-    case 'version': {
-      console.log(readPackageVersion());
-      return;
-    }
-    default:
-      console.error(`Unknown command: ${command}`);
-      process.stderr.write(USAGE);
-      process.exit(1);
-  }
-}
-
-main(process.argv.slice(2));