devlyn-cli 1.15.0 → 2.1.0

package/benchmark/auto-resolve/fixtures/SCHEMA.md

@@ -0,0 +1,130 @@
+# Fixture Schema
+
+Every fixture is a directory under `benchmark/auto-resolve/fixtures/F<N>-<slug>/` with these files. **All six files are required** (setup.sh may be empty when the starting `test-repo` copy needs no modification).
+
+## metadata.json
+
+```json
+{
+  "id": "F2-cli-medium-subcommand",
+  "category": "medium",
+  "difficulty": "medium",
+  "timeout_seconds": 1200,
+  "required_tools": ["node"],
+  "browser": false,
+  "deps_change_expected": false,
+  "intent": "One-sentence plain-language statement of the work, the SINGLE source of truth for spec.md and task.txt."
+}
+```
+
+- **id** — matches directory name, used across artifacts.
+- **category** — one of `trivial | medium | high-risk | stress | edge | e2e`. Drives which ship-gate rule applies.
+- **difficulty** — expected difficulty independent of category. Rubric uses this only for saturation detection (when both arms > 95 for two versions, flag fixture for rotation).
+- **timeout_seconds** — per-arm hard timeout. Runner kills the arm at this limit and marks result `TIMEOUT`.
+- **required_tools** — binaries the arm's environment must provide. Runner checks before invocation.
+- **browser** — true if arm must be able to run Playwright. Runner uses this to decide whether `test-repo`'s Playwright deps get installed before the arm starts.
+- **deps_change_expected** — true if the task involves modifying `package.json` / lockfiles. Variant's CRITIC security sub-pass is expected to trigger native `security-review` dep audit when true.
+- **intent** — **load-bearing**. A short plain-language statement shared by both arms. `spec.md` formalizes it into auto-resolve-ready form; `task.txt` renders it as a direct prompt. A CI lint ensures both derive from this field and stay in sync.
+
+## spec.md
+
+Auto-resolve-ready spec for the pipeline arm. Same format `/devlyn:ideate` produces:
+
+```markdown
+---
+id: "<fixture-id>"
+title: "<short title>"
+status: planned
+complexity: medium
+depends-on: []
+---
+
+# <fixture-id> <Title>
+
+## Context
+2-3 sentences describing WHY (not HOW). Must be traceable back to `metadata.intent`.
+
+## Requirements
+- [ ] Specific, testable, scoped.
+- [ ] ...
+
+## Constraints
+- Concrete, with reasoning for each (not bare).
+
+## Out of Scope
+- Explicit "must NOT build" list. Audited by preflight as anti-commitments.
+
+## Verification
+- Concrete commands whose expected behavior is named.
+```
+
+## task.txt
+
+Bare-arm input. Plain English, same intent, but framed as a user request rather than a formal spec. Intentionally lacks the structured Requirements/Constraints/Out-of-Scope sections — bare must make those calls itself. Must not leak "use the devlyn skill" hints.
+
+## expected.json
+
+Machine-readable acceptance criteria used by both `run-fixture.sh` verification and the judge's rubric anchor.
+
+```json
+{
+  "verification_commands": [
+    {
+      "cmd": "node bin/cli.js doctor",
+      "exit_code": 0,
+      "stdout_contains": ["doctor: "],
+      "stdout_not_contains": ["undefined"]
+    }
+  ],
+  "forbidden_patterns": [
+    {
+      "pattern": "catch\\s*\\(\\s*[a-zA-Z_]*\\s*\\)\\s*\\{\\s*return",
+      "description": "silent catch returning a fallback value — violates no-silent-catches policy",
+      "files": ["bin/cli.js"],
+      "severity": "disqualifier"
+    }
+  ],
+  "required_files": ["bin/cli.js"],
+  "forbidden_files": [],
+  "max_deps_added": 0
+}
+```
+
+- **verification_commands** — runner executes each. Each command's pass/fail contributes to the arm's `verify_score`.
+- **forbidden_patterns** — regexes scanned across `diff.patch`. Match at `severity: "disqualifier"` is a hard-floor fail. Match at `severity: "warning"` goes into the judge's critical-findings report.
+- **required_files** — must exist after the arm runs.
+- **forbidden_files** — must NOT appear in the arm's diff.
+- **max_deps_added** — count of new entries under `dependencies`/`devDependencies` in `package.json`. Exceeds → hard-floor fail.
+
+## NOTES.md
+
+Human-readable explanation of why this fixture exists. Must answer:
+
+1. What specific failure mode does this fixture detect?
+2. What pipeline phase(s) is this testing?
+3. Why can't another fixture cover this?
+4. When should this fixture be retired or replaced?
+
+Notes are read during suite design review, not during runs.
+
+## setup.sh
+
+Deterministic starting state. Runs against a fresh copy of `benchmark/auto-resolve/fixtures/test-repo/` before either arm starts. Common uses:
+
+- Install extra deps (`npm install --prefix . something`).
+- Apply a `.patch` that introduces a bug to fix.
+- Create pre-existing files referenced by the spec.
+
+Script must be idempotent when re-applied. Empty file (just `#!/usr/bin/env bash\nset -e\n`) is valid when no setup needed.
+
+---
+
+## Drift Prevention
+
+A CI lint step (`scripts/lint-fixtures.sh`) verifies:
+
+- All six files present per fixture.
+- `metadata.intent` substring appears in both `spec.md::Context` and `task.txt` (≥ 60% token overlap using simple tokenization).
+- `spec.md` frontmatter `id` matches directory name.
+- `expected.json` is valid JSON.
+- `setup.sh` is executable.

package/benchmark/auto-resolve/fixtures/test-repo/README.md

@@ -0,0 +1,27 @@
+# bench-test-repo
+
+Deterministic base Node project used by every devlyn-cli auto-resolve
+benchmark fixture. Fixtures extend this skeleton via `setup.sh` patches.
+
+## What's in it
+
+- `bin/cli.js` — tiny CLI (`hello`, `version`)
+- `server/index.js` — tiny Express app (`/health`, `/items`, `/items/:id`)
+- `web/index.html` — minimal static page with a click interaction
+- `tests/cli.test.js`, `tests/server.test.js` — node:test fixtures
+- `playwright.config.js` — used by web/browser fixtures only
+- `package.json` — `express` dep, `engines: node >= 18`
+
+## How it's used
+
+`run-fixture.sh` copies this directory to a temp path per run, applies the
+fixture's `setup.sh`, then invokes the arm (variant or bare) against that
+copy. No fixture modifies this source tree — modifications happen only in
+the per-run temp copies.
+
+## Keep it minimal
+
+Adding features to `test-repo` enlarges the surface every fixture works
+against. Add only when an existing fixture can't express itself against the
+current baseline. Preferred path: push complexity into the fixture's
+`setup.sh`, not into this base.

package/benchmark/auto-resolve/fixtures/test-repo/bin/cli.js

@@ -0,0 +1,63 @@
+#!/usr/bin/env node
+// bench-test-repo — tiny CLI used as the deterministic base for benchmark fixtures.
+// Fixtures extend or modify this file; keep the baseline minimal and obvious.
+
+const fs = require('fs');
+const path = require('path');
+
+const USAGE = `Usage: bench-cli <command> [options]
+
+Commands:
+  hello [--name NAME]        Print a greeting (default name: "world")
+  version                    Print the CLI version from package.json
+  --help, -h                 Show this help
+
+Examples:
+  bench-cli hello
+  bench-cli hello --name alice
+  bench-cli version
+`;
+
+function readPackageVersion() {
+  const pkgPath = path.join(__dirname, '..', 'package.json');
+  const raw = fs.readFileSync(pkgPath, 'utf8');
+  return JSON.parse(raw).version;
+}
+
+function parseNameFlag(argv) {
+  const idx = argv.indexOf('--name');
+  if (idx === -1) return 'world';
+  const value = argv[idx + 1];
+  if (!value || value.startsWith('-')) {
+    console.error('--name requires a value');
+    process.exit(1);
+  }
+  return value;
+}
+
+function main(argv) {
+  const [command, ...rest] = argv;
+
+  if (!command || command === '--help' || command === '-h') {
+    process.stdout.write(USAGE);
+    return;
+  }
+
+  switch (command) {
+    case 'hello': {
+      const name = parseNameFlag(rest);
+      console.log(`Hello, ${name}!`);
+      return;
+    }
+    case 'version': {
+      console.log(readPackageVersion());
+      return;
+    }
+    default:
+      console.error(`Unknown command: ${command}`);
+      process.stderr.write(USAGE);
+      process.exit(1);
+  }
+}
+
+main(process.argv.slice(2));