@nurix/etna 0.20.36 → 0.20.38

package/data/harnesses/hr/skills/evaluator/SKILL.md

@@ -5,56 +5,60 @@ description: "Use when scoring a single software project for production-readines
 
 # Evaluator
 
-Single-project evaluator. One project in → one per-project workspace out. The caller supplies the output directory; the skill writes `<out_dir>/<project>/{README.md, scorecard.yaml, raw/<sub-criterion>-<artifact>.ext}` and returns that path.
-
-This skill scores **exactly one project**. It does not rank, normalize across projects, weight sub-criteria differently, or know anything about a batcher, a cohort, or other projects. Every band it assigns is **absolute** — computed from per-signal thresholds in each recipe — so the scorecard for a project is identical whether the project is scored alone or as one of fifty. Cross-project weighting and comparison, if wanted, is a separate concern layered on top by whoever calls this skill; the skill itself stays unaware of it.
+Scores exactly one project: one repo in, one workspace out at `<out_dir>/<project>/`, whose path you return. Every band is **absolute** — computed from the per-signal thresholds in each recipe — so a project's scorecard is identical whether it is scored alone or as one of fifty. The skill **never** ranks, normalizes across projects, weights sub-criteria unequally, or knows a batcher or cohort exists; cross-project comparison is a layer the caller adds on top.
 
 ## Preflight (run once per host, idempotent)
 
-This skill depends on external tools (`gitleaks`, `lizard`, `scc`, `semgrep`, `hadolint`, `ruff`, `biome`, `knip`, `deptry`, `shellcheck`, `golangci-lint`, `gh`, `jq`, `pip-audit`, `govulncheck`, `trivy`). Before scoring, run [`preflight.md`](preflight.md). It checks each tool, installs any that are missing (via `brew` / `npm -g` / `pipx` / `go install`), and fails loud if an install fails. **Do not silently fall back to grep-or-LLM-only substitutes** — that poisons the gate checks (VULN / SECRETS / AUTH / OBSERVABILITY) and makes the scorecard dishonest. If preflight exits non-zero, return a clear failure message naming the missing tools.
+Run [`preflight.md`](preflight.md) before scoring — it checks the external tools the recipes call, installs any that are missing, and exits non-zero if an install fails. If it exits non-zero, stop and return a failure message naming the missing tools. **Never fall back to a grep-or-LLM-only substitute for a missing tool** — the banding thresholds and the `VULN` / `SECRETS` / `AUTH` / `OBSERVABILITY` gates are calibrated against real tool output, so a substitute yields a scorecard that looks complete and is wrong exactly where it matters.
 
 ## Hard rule: every slow command gets a timeout
 
-Every command the skill runs that can hang or burst must be wrapped in a bounded wall-clock guard. Unbounded retries are how a single project eats 20+ minutes. Apply one of:
+Wrap every command that can hang or burst in a wall-clock bound, even where a recipe shows it naked — unbounded retries are how one project eats 20+ minutes.
 
-- `timeout 60 <cmd>` for any `gh api`, `semgrep`, `npm install`, `pip install`, `go build`, `go vet`, or `curl` — even when the recipe doesn't show it. If the recipe shows a naked command, wrap it yourself.
-- `--maxdepth N` and `-not -path '*/node_modules/*' -not -path '*/.venv/*' -not -path '*/.git/*'` on every `find`.
-- `git log -n 500` (or similar ceiling) instead of walking full history. Never `git log --all` without a limit.
-- `semgrep`: always pass `--timeout 30` (per-rule), `--disable-version-check`, `--metrics=off`. Avoid `--config p/code-quality` — that registry entry 404s in retry-storm; use per-language packs (`p/python`, `p/typescript`, `p/javascript`, `p/go`) or `p/default`.
-- Dep-vuln scans: `pip-audit --timeout 120`, `govulncheck` in a 180 s `timeout`, `trivy image --timeout 180s`. `npm audit --json` doesn't take a flag; wrap the whole call in `timeout 120`.
+| Command | Bound |
+| --- | --- |
+| `gh api`, `semgrep`, `npm install`, `pip install`, `go build`, `go vet`, `curl` | `timeout 60 <cmd>` |
+| every `find` | `-maxdepth N` and `-not -path '*/node_modules/*' -not -path '*/.venv/*' -not -path '*/.git/*'` |
+| `git log` | a ceiling such as `-n 500`; **never** `git log --all` without a limit |
+| `semgrep` | `--timeout 30` (per rule), `--disable-version-check`, `--metrics=off` |
+| `pip-audit` | `--timeout 120` |
+| `govulncheck`, `trivy fs` | inside `timeout 180` |
+| `npm audit --json` | takes no timeout flag — wrap the call in `timeout 120` |
 
-If a single mechanical command still exceeds ~2 min, something is wrong with the project (giant `node_modules/`, generated bundle, vendored deps). Skip it with `_NOT_RUN: true, reason: "timeout on <cmd>"` rather than retrying — a sparse signal is tolerable; a hang is not.
+**Never use `semgrep --config p/code-quality`** — that registry entry returns 404 in a retry storm. Use the per-language packs (`p/python`, `p/typescript`, `p/javascript`, `p/go`) or `p/default`.
 
-## Input
+If one mechanical command still exceeds ~2 min, the project carries something pathological (a giant `node_modules/`, a generated bundle, vendored deps). Skip it with `_NOT_RUN: true, reason: "timeout on <cmd>"` rather than retrying — a sparse signal is tolerable; a hang is not.
 
-A repo reference + an output directory:
+## Input
 
-- **Local clone path** — e.g. `/path/to/project/`. Used by filesystem reads (Semgrep, grep, file sampling, git analytics). This is the primary input.
-- **GitHub ref** (optional) — e.g. `<org>/<repo>`. Used by `gh api` calls for PR / code-review history when the project is hosted on GitHub. Absence just means those signals are skipped, not a failure.
-- **`out_dir`** — caller-supplied base directory. The skill writes into `<out_dir>/<project>/` and returns that path. The skill creates the folder; `raw/` is flat — one file per signal named `<sub-criterion>-<artifact>.ext`, no subdirectories.
+| Input | Required | Use |
+| --- | --- | --- |
+| Local clone path, e.g. `/path/to/project/` | yes — the primary input | Filesystem reads: Semgrep, grep, file sampling, git analytics |
+| GitHub ref, e.g. `<org>/<repo>` | no | `gh api` calls for PR and code-review history; absent ⇒ those signals are skipped, not failed |
+| `out_dir` | yes | Base directory; you create `<out_dir>/<project>/` under it |
 
-`<project>` is a stable slug for the project (its directory name is fine). Optional baseline stats (kLoC, commit count, contributor count) may be supplied to skip a pre-pass; otherwise derive them from the local clone.
+`<project>` is a stable slug — the directory name is fine. Baseline stats (kLoC, commit count, contributor count) may be supplied to skip a pre-pass; otherwise derive them from the clone.
 
 ## Output — per-project workspace
 
-Every invocation writes a folder at `<out_dir>/<project>/` with two parts plus raw dumps:
-
-```
-<out_dir>/<project>/
-├── README.md            # Human-readable narrative. Shape: § "README shape" below.
-├── scorecard.yaml       # Structured scorecard (below). Machine-readable.
-└── raw/                 # Flat directory — every file named `<sub-criterion>-<artifact>.ext`
-    ├── code-quality-*            # llm-sample.jsonl, lint-{ruff,biome,eslint,golangci}.json, semgrep-quality.json, cr-comments.jsonl
-    ├── testing-*                 # tests-inventory.jsonl, llm-sample.jsonl, semgrep-resilience.json, gha-runs.json, auth-scan-{http,ws,grpc,trpc}.json
-    ├── deployment-readiness-*    # artifacts.json, hadolint.json, localhost-refs.txt, observability-markers.json, llm-read.md
-    ├── maintainability-*         # gitleaks.json, lizard.csv, knip.json|deptry.json|gomod-diff.txt, file-sizes.txt, tech-debt.txt, dep-vuln-{pip-audit,npm-audit,govulncheck,trivy}.json, llm-read.md
-    ├── problem-substantiveness-* # llm-read.md, proxies.json
-    ├── ux-mechanics-*            # path-detection.json + path-specific files (ux-heuristics, routes, cli-probes, shellcheck, openapi-quality, api-surrogate), llm-read.md
-    ├── value-articulation-*      # llm-read.md, proxies.json
-    └── mechanical-ambition-*     # scc.json, stubs.json, git-analytics.json, tier-a-build.log, llm-read.md
-```
-
-`raw/` is a **flat directory** — no subfolders. Every file is named `<sub-criterion>-<artifact>.ext` (e.g. `code-quality-semgrep-quality.json`, `maintainability-deptry.json`). A signal that couldn't be extracted (tool absent, language not present, repo shape mismatch) produces **no file** — absence is the signal, and `scorecard.yaml` flags the gap. Per-recipe artifact lists are authoritative in each recipe's "Raw dumps" section.
+| Path under `<out_dir>/<project>/` | Contents |
+| --- | --- |
+| `README.md` | Human-readable narrative — § README shape |
+| `scorecard.yaml` | Machine-readable scorecard — § Scorecard shape |
+| `raw/` | Raw tool output, one file per signal |
+
+`raw/` is a **flat directory** — no subfolders; every file is named `<sub-criterion>-<artifact>.ext` (e.g. `code-quality-semgrep-quality.json`). A signal that cannot be extracted (tool absent, language not present, repo shape mismatch) writes **no file** — absence is the signal, and `scorecard.yaml` flags the gap. Each recipe's "Raw dumps" section is authoritative for its artifact list; in summary:
+
+| Prefix | Artifacts |
+| --- | --- |
+| `code-quality-` | `llm-sample.jsonl`, `lint-{ruff,biome,eslint,golangci}.json`, `semgrep-quality.json`, `cr-comments.jsonl` |
+| `testing-` | `tests-inventory.jsonl`, `llm-sample.jsonl`, `semgrep-resilience.json`, `gha-runs.json`, `auth-scan-{http,ws,grpc,trpc}.json` |
+| `deployment-readiness-` | `artifacts.json`, `hadolint.json`, `localhost-refs.txt`, `observability-markers.json`, `llm-read.md` |
+| `maintainability-` | `gitleaks.json`, `lizard.csv`, `knip.json` \| `deptry.json` \| `gomod-diff.txt`, `file-sizes.txt`, `tech-debt.txt`, `dep-vuln-{pip-audit,npm-audit,govulncheck,trivy}.json`, `llm-read.md` |
+| `problem-substantiveness-` | `llm-read.md`, `proxies.json` |
+| `ux-mechanics-` | `path-detection.json` + path-specific files (`ux-heuristics`, `routes`, `cli-probes`, `shellcheck`, `api-surrogate`), `llm-read.md` |
+| `value-articulation-` | `llm-read.md`, `proxies.json` |
+| `mechanical-ambition-` | `scc.json`, `stubs.json`, `git-analytics.json`, `tier-a-build.log`, `llm-read.md` |
 
 ### Scorecard shape
 
@@ -90,39 +94,37 @@ available: 100.0 # max points from currently-scored items (lower if any item is
 normalized: 66.25 # total ÷ available × 100
 ```
 
-**Each item scored 0–5** → points = `score × 2.5` (every item is weighted 12.5 pts, so the step is 2.5). `total` is the sum over scored items; `available` is the max points those scored items could have reached (12.5 × count); `normalized = total ÷ available × 100`. With all 8 items scored, `available == 100.0` and `total == normalized`. If a sub-criterion is genuinely unscoreable (`score: null`), it drops out of both `total` and `available` so the percentage stays honest.
-
-The `total` here is an **equal-weight** convenience score for reading one project on its own. The skill does not apply per-sub-criterion weights — it has no opinion on relative importance. A caller that wants weighted, cross-project-comparable scores reads the raw 0–5 `score` fields and applies its own weighting; that logic lives entirely outside this skill.
+The totals arithmetic is § Scoring model.
 
 ### Workspace-writing order
 
-Write the outputs in this order — each depends on the prior:
+Write the outputs in this order — each depends on the one before:
 
-1. **`raw/<sub-criterion>-<artifact>.ext`** — run each recipe (see table below), dump raw tool output verbatim to the flat `raw/` directory with per-recipe filename prefixes. This is the extraction pass.
-2. **`scorecard.yaml`** — fold each recipe's aggregates into the sub-criterion block, apply gates, compute `total` / `available` / `normalized`. Pin `commit_sha_pinned` (audit metadata) and `rubric_version`.
-3. **`README.md`** — render the README shape below with values drawn from `scorecard.yaml` and pointers to `raw/` artifacts. The raw-dump section is a pointer index, not a re-embedded dump.
+1. **`raw/<sub-criterion>-<artifact>.ext`** — run each recipe and dump raw tool output verbatim to the flat `raw/` directory. This is the extraction pass.
+2. **`scorecard.yaml`** — fold each recipe's aggregates into its sub-criterion block, apply gates, compute `total` / `available` / `normalized`, and pin `commit_sha_pinned` and `rubric_version`.
+3. **`README.md`** — render § README shape from `scorecard.yaml`, pointing at `raw/` artifacts rather than re-embedding them.
 
 ### README shape
 
-`README.md` is a self-contained human narrative — no external template. Render these sections from `scorecard.yaml`:
+`README.md` is a self-contained narrative with no external template. Render these sections from `scorecard.yaml`:
 
 1. **Title + verdict** — `# <project> — <normalized>/100` and a 2–3 sentence overall read.
 2. **Run metadata** — `rubric_version`, `scored_at`, `agent_model`, `commit_sha_pinned`, gates triggered, short-circuit flag.
 3. **Scorecard table** — one row per sub-criterion: `| Sub-criterion | Band (0–5) | Points / 12.5 | One-line evidence |`, split into a Code block and a Product block, with the `total / available / normalized` footer.
 4. **Per-sub-criterion detail** — for each item, the band, the full `evidence` string, and the key numbers from `data`.
-5. **Raw artifact index** — a pointer list of the `raw/<sub-criterion>-*` files that were produced (and a note on any that were skipped and why).
+5. **Raw artifact index** — a pointer list of the `raw/<sub-criterion>-*` files produced, plus a note on any skipped and why. Every `_NOT_RUN` signal is flagged here.
 
 ### Rubric version
 
-`rubric_version` is a simple integer marker recorded on each scorecard. Bump it whenever a recipe adds/drops a signal, a gate threshold moves, a banding table changes, or the scorecard shape changes. It exists so a reader can tell which rubric produced a scorecard and so a caller can decide whether a cached workspace is stale. It is **not** a cohort cache key and carries no cross-project semantics — one project, one rubric version, decided at extraction time.
+`rubric_version` is an integer marker on each scorecard. **Bump it whenever** a recipe adds or drops a signal, a gate threshold moves, a banding table changes, or the scorecard shape changes — a caller compares it to decide whether a cached workspace is stale. It is **not** a cohort cache key and carries no cross-project meaning.
 
 ## Scoring model
 
-- **100 points total, 50 per half:**
-  - **1.x Production-Grade Code** — 4 sub-criteria × 12.5 pts each = **50 pts**
-  - **2.x Product Thinking** — 4 sub-criteria × 12.5 pts each = **50 pts**
-- **Each item scored 0–5** → points = `score × 2.5`.
-- **Scorecard reports `total`, `available`, `normalized`.**
+- **100 points, 50 per half:** 1.x Production-Grade Code and 2.x Product Thinking, each 4 sub-criteria × 12.5 pts.
+- **Each item is scored 0–5**; points = `score × 2.5`.
+- **`total`** is the sum of points over scored items; **`available`** is 12.5 × the count of scored items; **`normalized`** = `total ÷ available × 100`. With all 8 scored, `available == 100.0` and `total == normalized`.
+- **A genuinely unscoreable item** (`score: null`) drops out of both `total` and `available`, so the percentage stays honest — a library with no UI is not punished for lacking one.
+- **`total` is equal-weight by design** — the skill has no opinion on relative importance. A caller wanting weighted, comparable scores reads the raw 0–5 `score` fields and applies its own weights outside the skill.
 
 ### Bands (absolute)
 
@@ -135,7 +137,7 @@ Write the outputs in this order — each depends on the prior:
 | **1** | Token effort — minimal execution, most criteria absent          |
 | **0** | Absent                                                          |
 
-Bands are **absolute, not relative to any other project**. Per-signal numeric thresholds live in each recipe's _Banding_ section. The rubric is single-project-invokable by design — there is no cohort pass, no quintile math, no re-banding step.
+Per-signal numeric thresholds live in each recipe's _Banding_ section. There is no cohort pass, no quintile math, no re-banding step.
 
 ### Gates (cap specific items; stack additively)
 
@@ -146,34 +148,32 @@ Bands are **absolute, not relative to any other project**. Per-signal numeric th
 | `UNTESTED` (hard) | `test_file_count == 0` | `testing ≤ 1` |
 | `TESTS_TRIVIAL` (soft) | Tests exist but `assertion_density < 1.0` per test file OR `trivial_body_ratio > 0.8` | `testing ≤ 2` |
 | `SECRETS` | `gitleaks_findings ≥ 1` — any secret in git history, even rotated out of HEAD | `maintainability ≤ 2` |
-| `VULN` | Dep-scan reports ≥ 1 CVE at CRITICAL OR ≥ 3 at HIGH across any dep (direct or transitive). Tools: `pip-audit` / `npm audit --json` / `govulncheck` / `trivy image`. Skipped if no recognized package manifest. | `maintainability ≤ 2` |
+| `VULN` | Dep-scan reports ≥ 1 CVE at CRITICAL OR ≥ 3 at HIGH across any dep (direct or transitive). Tools: `pip-audit` / `npm audit --json` / `govulncheck` / `trivy fs`. Skipped if no recognized package manifest. | `maintainability ≤ 2` |
 | `AUTH` | Network-routed handlers exist AND ≥ 1 mutating endpoint is not behind an auth middleware/dependency/decorator. Covers HTTP (FastAPI/Express/Flask/Gin/Django POST/PUT/PATCH/DELETE), WebSocket (ws/Socket.io/SignalR event handlers), gRPC (grpc/grpcio unary/streaming), tRPC (@trpc/server mutations). Skipped if no routed handlers detected. | `testing ≤ 2` |
-| `OBSERVABILITY` | Structured-logging library absent (no structlog / winston / pino / log-slog / zap AND no JSON log format detected in code) **AND** metrics/tracing library absent (no OpenTelemetry, Prometheus client, Datadog agent, statsd client, or custom `/metrics` endpoint). Both must be absent for the gate to fire. | `deployment_readiness ≤ 3` |
+| `OBSERVABILITY` | Structured-logging library absent (no structlog / loguru / winston / pino / log-slog / zap AND no JSON log format detected in code) **AND** metrics/tracing library absent (no OpenTelemetry, Prometheus client, Datadog agent, statsd client, or custom `/metrics` endpoint). Both must be absent for the gate to fire. | `deployment_readiness ≤ 3` |
 
-Gates cap the **band** of the named sub-criterion. When multiple gates cap the same item, the lowest cap wins. Gates are recorded in `gates_triggered`; a reader (or a downstream caller) can see exactly which caps applied.
+A gate caps the **band** of the named sub-criterion before totals are computed; when several cap one item, the lowest cap wins. Record every gate that fired in `gates_triggered`.
 
 ### Dead-project short-circuit
 
-If **3 or more gates** fire on one project, skip the interpretive LLM passes for all sub-criteria, emit a fixed floor scorecard, and set `short_circuited: true`:
+If **3 or more gates** fire on one project, skip the interpretive LLM passes, emit a fixed floor scorecard, and set `short_circuited: true` — the interpretive pass on a dead project costs compute and returns generous hallucinations.
 
 - **Non-gate-capped items**: `score: 1, evidence: "Floor — 3+ gates triggered; interpretive pass skipped"`.
 - **Gate-capped items**: take the cap value (e.g. `deployment_readiness: 1` when DEMO fires, `testing: 1` when UNTESTED fires, `maintainability: 2` when SECRETS or VULN fires, `deployment_readiness: 3` when OBSERVABILITY fires). If multiple gates cap the same item, use the lowest cap.
 - **Totals**: `total`, `available`, `normalized` computed normally over the floored items.
 
-Rationale: a project that's undeployable + untested + broken is clearly dead. Running the expensive interpretive pass produces generous LLM hallucinations on dead code. Short-circuit saves compute and keeps the score honest.
-
 ### CodeRabbit — calibrating modifier with null-data semantics
 
-If the project's GitHub history includes CodeRabbit review comments, they act as a **calibrating modifier on 1.1 (Code Quality) and 1.2 (Testing)**, never a primary signal.
+CodeRabbit (CR) review comments in the project's GitHub history are a **calibrating modifier on 1.1 (Code Quality) and 1.2 (Testing)**, never a primary signal.
 
-- When a project has CR data (`cr_pr_count > 0` **and** non-empty actionables): `cr_*_per_kloc` values feed the recipe's banding table as calibrating modifiers.
-- When a project has no CR data (no CR history, or walkthrough-only with `cr_actionable_total == 0`): every `cr_*_per_kloc` is emitted as `null`. The scorer treats `null` as a no-op; 1.1 / 1.2 bands are decided by the remaining signals.
+- **CR data present** (`cr_pr_count > 0` **and** non-empty actionables): the `cr_*_per_kloc` values feed the recipe's banding table as modifiers.
+- **No CR data** (no CR history, a project not on GitHub, or walkthrough-only with `cr_actionable_total == 0`): emit every `cr_*_per_kloc` as `null`; the modifier is a no-op and the remaining signals decide the band.
 
-Design principle: `null ≠ 0`. `null` means "no CR data, skip the modifier"; `0` means "CR reviewed the PR and found nothing" — a real signal pulling the band up. Projects not hosted on GitHub, or with no PRs, simply emit `null` and the modifier is a no-op.
+**`null ≠ 0`.** `null` means no CR data, skip the modifier; `0` means CR reviewed and found nothing — a real signal that pulls the band up. Never coerce one into the other.
 
 ## Sub-criterion extraction recipes
 
-One recipe file per sub-criterion. All 8 are AI-scored from repo evidence.
+One recipe per sub-criterion; all 8 are scored from repo evidence.
 
 | # | Item | Half | Contender | Recipe |
 | --- | --- | --- | --- | --- |
@@ -188,21 +188,19 @@ One recipe file per sub-criterion. All 8 are AI-scored from repo evidence.
 
 ## Workflow
 
-The skill scores one project end-to-end:
-
 1. **Preflight** — run [`preflight.md`](preflight.md). Halt with a clear error if it fails.
-2. **Mechanical extraction** — per recipe, run the shell commands under each recipe's "Extraction" section. Dump every tool output **verbatim** to the flat `<out_dir>/<project>/raw/` directory using the `<sub-criterion>-<artifact>.ext` naming convention. A signal that can't be extracted produces **no file** — absence is the signal.
-3. **LLM interpretive passes** — per recipe, run the LLM reads (sample files, score axes, README parse, shipped-vs-promised diff). Write `raw/<slug>-llm-sample.jsonl` or `raw/<slug>-llm-read.md` per sub-criterion. These are the load-bearing interpretive judgments.
-4. **Apply gates** — evaluate each gate (DEMO / BROKEN / UNTESTED / TESTS_TRIVIAL / SECRETS / VULN / AUTH / OBSERVABILITY) using the triggers above. If **≥3 gates** fire, emit the dead-project short-circuit floor scorecard (`short_circuited: true`) and skip remaining interpretive passes.
-5. **Emit `scorecard.yaml`** — apply each recipe's absolute-threshold band table directly. Record the run-metadata block (`commit_sha_pinned`, `rubric_version`, `scored_at` via `date -u +"%Y-%m-%dT%H:%M:%SZ"`, `agent_model`, `approx_tokens`). Compute `total` / `available` / `normalized`.
-6. **Render `README.md`** — fill the README shape above with values from `scorecard.yaml` and pointer references to `raw/` artifacts.
-7. **Return the workspace path** — `<out_dir>/<project>/`. Write the files and leave them uncommitted; whoever invoked the skill decides whether to commit.
+2. **Mechanical extraction** — for each recipe, run the commands in its "Extraction" section and dump every tool output **verbatim** to the flat `<out_dir>/<project>/raw/` directory. A signal that cannot be extracted writes **no file**. Then evaluate the six gates decidable from mechanical signals alone (`UNTESTED`, `TESTS_TRIVIAL`, `SECRETS`, `VULN`, `BROKEN`, `OBSERVABILITY`); if **≥3** fire, skip step 3 and emit the dead-project short-circuit floor scorecard at step 5.
+3. **LLM interpretive passes** — for each recipe, run its LLM reads (file samples, score axes, README parse, shipped-vs-promised diff) and write `raw/<slug>-llm-sample.jsonl` or `raw/<slug>-llm-read.md`. These are the load-bearing interpretive judgments.
+4. **Apply gates** — evaluate all eight gates (DEMO / BROKEN / UNTESTED / TESTS_TRIVIAL / SECRETS / VULN / AUTH / OBSERVABILITY) against § Gates. If **≥3** fire, emit the dead-project short-circuit floor scorecard (`short_circuited: true`).
+5. **Emit `scorecard.yaml`** — apply each recipe's absolute band table directly. Record the run metadata (`commit_sha_pinned`, `rubric_version`, `scored_at` via `date -u +"%Y-%m-%dT%H:%M:%SZ"`, `agent_model`, `approx_tokens`) and compute `total` / `available` / `normalized`.
+6. **Render `README.md`** — fill § README shape from `scorecard.yaml`, with pointers to `raw/` artifacts.
+7. **Return the workspace path** — `<out_dir>/<project>/`. Leave the files uncommitted; the caller decides whether to commit.
 
 ### Scorecard schema validator
 
-[`validate_scorecard.py`](validate_scorecard.py) — a Python script that asserts a `scorecard.yaml` has every required top-level field (including `total` / `available` / `normalized`), all 8 sub-criterion blocks, and `score` + `evidence` + `data` on each. Exits 0 on valid; exits 1 on first failure with a specific error on stderr. Run it after emitting a scorecard to catch shape errors early. **Extending:** if a scorecard-shape bump introduces a new required field, append it to `REQUIRED_TOP` in the validator.
+Run [`validate_scorecard.py`](validate_scorecard.py) on every emitted scorecard. It asserts every required top-level field (including `total` / `available` / `normalized`), all 8 sub-criterion blocks, and `score` + `evidence` + `data` on each; it exits 0 when valid and 1 on the first failure, with the error on stderr. When a scorecard-shape change adds a required top-level field, append it to `REQUIRED_TOP` in the validator.
 
 ## Not in scope
 
-- **Cross-project comparison, weighting, or ranking.** The skill emits raw 0–5 bands and an equal-weight `total`. Anything that compares projects, applies non-uniform weights, or normalizes a set against itself is a layer above this skill and must not leak into it.
-- **Runtime verification.** The skill reads the repo statically. Actually running `docker compose up` / `npm run dev` and clicking through the product is a separate manual pass a reviewer may add on top; the skill does not drive it.
+- **Cross-project comparison, weighting, or ranking.** The skill emits raw 0–5 bands and an equal-weight `total`; anything that compares projects, applies non-uniform weights, or normalizes a set against itself is a layer above this skill and **must not** leak into it.
+- **Runtime verification.** The skill reads the repo statically. Running `docker compose up` / `npm run dev` and clicking through the product is a separate manual pass a reviewer may add; the skill never drives it.

package/data/harnesses/hr/skills/evaluator/criteria/code-quality.md

@@ -1,21 +1,19 @@
 # Code Quality — Extraction Plan
 
-- **Sub-criterion:** 1.1 (AI-only)
-- **Weight:** 12.5 pts (4 Code AI × 12.5 = 50 pts half)
+- **Sub-criterion:** 1.1 (Production-Grade Code)
+- **Weight:** 12.5 pts
 - **Contenders:** LLM interpretive read (primary, load-bearing) + lint (`ruff` / `biome` / `golangci-lint`) + `lizard` complexity (**per-language CCN**: Python 10 / JS-TS 10 / Java 12 / Go 15) + `semgrep` quality packs + CodeRabbit scrape (**calibrating modifier**).
 - **Source quote:** _"Readable, modular, well-structured. Good naming, clean APIs, no code smells."_
 
-Bands are **absolute** — the _Banding_ table in this recipe declares per-signal numeric thresholds. `lizard_god_per_kloc` is computed against **per-language CCN thresholds**: Python / JS / TS functions are "god" at CCN > 10; Java at CCN > 12; Go at CCN > 15 — a per-language lookup keyed on file extension. `_NOT_RUN: true` is supported for tool-missing cases but not for language-not-present (emit `null` aggregates instead). 1.1 has no gates of its own; secret/vuln signals are owned by `maintainability.md`.
+Bands are **absolute** — the _Banding_ table declares per-signal numeric thresholds. Emit `_NOT_RUN: true` only for a missing tool; a language not present emits `null` aggregates instead. 1.1 owns no gates; secret and vuln signals belong to [maintainability.md](maintainability.md).
 
-> **Why CR is a modifier, not primary.** Null-data semantics and the plan-upgrade escape hatch are spelled out once in [`SKILL.md → CodeRabbit — calibrating modifier with null-data semantics`](../SKILL.md#coderabbit--calibrating-modifier-with-null-data-semantics). Summary: CR joins lint / complexity / file-size / semgrep as a **calibrating modifier**; contributes when a project has non-empty CR data, no-op (`null`) when empty.
-
-> **Source-quote is semantic.** "Readable", "good naming", "clean APIs", "no code smells" are all reader-experience judgments. No single mechanical tool captures them — so the LLM read is the load-bearing signal, with lint / complexity / file-size / semgrep / CR acting as calibrating modifiers.
+The source quote names reader-experience judgments no mechanical tool captures, so the LLM read is the load-bearing signal; lint, complexity, file size, semgrep and CodeRabbit (CR) are calibrating modifiers. CR's null-data semantics are [`SKILL.md → CodeRabbit — calibrating modifier with null-data semantics`](../SKILL.md#coderabbit--calibrating-modifier-with-null-data-semantics).
 
 ---
 
 ## Preflight
 
-Before running this recipe, execute [`../../preflight.md`](../../preflight.md). The recipe hard-requires:
+Before running this recipe, execute [`../preflight.md`](../preflight.md). The recipe hard-requires:
 
 - `ruff` (Python repos)
 - `biome` (TS/JS repos) — preferred over eslint for speed; eslint is a per-project fallback if biome rejects the repo's config shape
@@ -24,9 +22,7 @@ Before running this recipe, execute [`../../preflight.md`](../../preflight.md).
 - `semgrep` (polyglot static analysis)
 - `gh`, `jq` (CR scrape + utilities)
 
-If any of the above is missing **and preflight cannot install it**, the recipe must emit `{signal}_NOT_RUN: true` + `reason` and flag the shortfall loudly in `§6.1` of the analysis file. No silent grep-estimation substitutes.
-
-Tools are called only for languages present in the repo (see each signal below).
+If any of the above is missing **and preflight cannot install it**, emit `{signal}_NOT_RUN: true` + `reason` and flag the shortfall in the workspace README's raw artifact index. **Never** substitute a grep estimate. Call each tool only for languages present in the repo.
 
 ---
 
@@ -37,7 +33,7 @@ Tools are called only for languages present in the repo (see each signal below).
 Sample **6 source files** that give the reader a representative slice:
 
 1. **Largest source file** by LoC (one per primary language)
-2. **Highest-churn file** from `git log --numstat --all --no-merges | awk ...` (top by lines touched)
+2. **Highest-churn file** from `git log -n 500 --numstat --no-merges | awk ...` (top by lines touched)
 3. **Entrypoint / handler**: matches `*/main.{py,ts,js,go}`, `*/app.{py,ts}`, `*/handlers/*`, `*/routes/*`, `*/api/*`
 4. **3 random** across distinct directories, excluding `tests/`, `__tests__/`, generated, vendored, lockfiles. Prefer files > 50 LoC.
 
@@ -76,12 +72,9 @@ if echo "$LANGS" | grep -q '^py$'; then
   ruff check <repo> --output-format json > /tmp/ruff.json || true
 fi
 
-# TS / JS — prefer biome. Use `biome lint` (not `biome ci` or `biome check`):
-# `ci` / `check` bundle lint + format + organizeImports as diagnostics, which
-# inflates `lint_findings_per_kloc` with stylistic `category: "format"` noise
-# that the scorer doesn't want (one observed project showed 11.4/kLoC with
-# format, 5.1/kLoC without — crossing the band-4 threshold of 10). `biome lint`
-# is pure lint and matches the intent of the signal.
+# TS / JS — prefer biome. Use `biome lint`, never `biome ci` / `biome check`:
+# those add format + organizeImports diagnostics, and the format noise alone
+# can double `lint_findings_per_kloc` across the band-4 threshold of 10.
 if echo "$LANGS" | grep -qE '^(ts|tsx|js|jsx)$'; then
   cd <repo>
   biome lint . --reporter=json > /tmp/biome.json 2>/dev/null \
@@ -105,7 +98,7 @@ Parse (same shape across tools):
 
 ### Signal 3 — Complexity density (`lizard`; shared with 1.4)
 
-`lizard` is run once by 1.4 Maintainability — **read its output**, do not re-run. 1.4's Signal 2 applies per-language CCN thresholds (Python 10 / JS-TS 10 / Java 12 / Go 15 — see [maintainability.md](maintainability.md#signal-2--polyglot-complexity-lizard-per-language-ccn)); 1.1 reuses those aggregates verbatim.
+`lizard` runs once, in 1.4 Maintainability — **read its output**, do not re-run. 1.4's [Signal 2](maintainability.md#signal-2--polyglot-complexity-lizard-per-language-ccn) applies the per-language CCN thresholds; 1.1 reuses its aggregates verbatim. If 1.4 has not run yet for this repo, run lizard now with the same command and CSV location, and share the output.
 
 Reuse:
 
@@ -116,8 +109,6 @@ Reuse:
 - `lizard_god_ratio` — `god_functions ÷ total_functions`
 - `lizard_thresholds_applied` — `{py: 10, js: 10, ts: 10, java: 12, go: 15, default: 15}` (audit constant)
 
-If 1.4 has not been run yet for this repo, run lizard now (same command, same CSV location) and share the output.
-
 ### Signal 4 — File-size distribution (filesystem)
 
 ```bash
@@ -136,14 +127,13 @@ Capture:
 - `largest_file_loc` — largest source file
 - `mean_source_file_loc` — mean over all source files
 
-Shared with 1.4; avoid duplicate computation.
+Shared with 1.4 — compute once.
 
 ### Signal 5 — Semgrep quality rule-set
 
 ```bash
-# NOTE: `--config p/code-quality` is deprecated on Semgrep Registry and
-# returns HTTP 404 with retry-storm. Stick to the per-language packs that
-# resolve cleanly. If you need an umbrella, use `p/default` (lower signal).
+# Never `--config p/code-quality` — it 404s in a retry storm. Use the
+# per-language packs; `p/default` is the umbrella fallback (lower signal).
 timeout 180 semgrep --config p/python --config p/typescript \
         --config p/javascript --config p/go \
         --json --timeout 30 --disable-version-check --metrics=off \
@@ -160,13 +150,11 @@ Capture:
 
 ### Signal 6 — CodeRabbit scrape (**calibrating modifier**)
 
-Scrape CR comments from both endpoints CR posts to — inline review comments and issue-style walkthrough comments. Both are needed: walkthrough-only repos look empty on the pulls endpoint alone. CR's label text is authoritative; emoji is a fallback (CR has rotated emoji twice recently).
+Scrape both endpoints CR posts to — inline review comments and issue-style walkthrough comments; a walkthrough-only repo looks empty on the pulls endpoint alone. CR's label text is authoritative; emoji drifts and is the fallback.
 
 ```bash
-# Cap each endpoint at 90s so a CR-heavy repo can't stall the run. The
-# `--paginate-limit` flag was removed in gh 2.89+, so we rely on `--paginate`
-# (fetches all pages) wrapped in `timeout` for the actual cap. Typical comment
-# counts fit easily inside the 90s window.
+# Cap each endpoint at 90s so a CR-heavy repo can't stall the run. `gh api`
+# has no page-limit flag: `--paginate` fetches every page and `timeout` is the cap.
 (
   timeout 90 gh api "repos/<org>/<repo>/pulls/comments?per_page=100" \
     --paginate \
@@ -188,13 +176,13 @@ Parse each CR comment; prefer label text, fall back to emoji:
 | `Security`                 | `🔒`              | `security`    |
 | `Performance`              | `⚡`              | `performance` |
 
-**Walkthrough detection (positive-match, not silence-inferred).** CR posts one issue-level walkthrough per PR it reviews, regardless of plan tier. Detect walkthroughs positively via any of these markers in the body (robust across CR template drift):
+**Walkthrough detection (positive-match, never silence-inferred).** CR posts one issue-level walkthrough per PR it reviews, on every plan tier. Count a comment as a walkthrough when its body carries any of these markers:
 
-- HTML comment: `<!-- walkthrough_start -->` (current stable anchor)
+- HTML comment: `<!-- walkthrough_start -->` (the stable anchor)
 - Section heading: `## Walkthrough`
 - Collapsible summary: `<summary>📝 Walkthrough</summary>`
 
-Any one match ⇒ count the comment as a walkthrough and record `cr_walkthrough_count`. This is separate from `cr_pr_count` (unique PRs with ≥1 CR comment of any kind).
+Record the count as `cr_walkthrough_count` — distinct from `cr_pr_count` (unique PRs with ≥1 CR comment of any kind).
 
 **Actionable-count capture.** When CR has actionable inline feedback, the walkthrough summary carries a line of the form `Actionable comments posted: N` (optionally bold-wrapped). Match with a tolerant regex:
 
@@ -202,7 +190,7 @@ Any one match ⇒ count the comment as a walkthrough and record `cr_walkthrough_
 \*{0,2}Actionable comments posted:\s*(\d+)\*{0,2}
 ```
 
-Sum captured `N` across walkthroughs → `cr_actionable_total`. If no walkthrough contains this string **and** at least one walkthrough contains a Free-plan disclaimer (`Summarized by CodeRabbit Free` / `is on the Free plan`), set `cr_free_plan: true` — this distinguishes "zero actionables because Free-plan walkthrough-only" from "zero actionables because the regex is blind to a new template." Unmatched inline comments bucket to `uncategorized`; surface the count so loss is visible.
+Sum captured `N` across walkthroughs → `cr_actionable_total`. If no walkthrough contains this string **and** at least one carries a Free-plan disclaimer (`Summarized by CodeRabbit Free` / `is on the Free plan`), set `cr_free_plan: true` — it separates a Free-plan zero from a regex blind to a new template. Unmatched inline comments bucket to `uncategorized`; surface that count so loss is visible.
 
 Capture:
 
@@ -215,23 +203,23 @@ Capture:
 - `cr_severe_count` — `by_category.bug + by_category.security + by_category.warning`
 - `cr_severe_per_kloc` — `cr_severe_count ÷ (scc_code_loc ÷ 1000)` (lower-is-better; `null` when `cr_pr_count == 0`)
 
-**Banding rule:** CR is one of several calibrating modifiers (see _Banding_ below). When `cr_pr_count > 0`, `cr_actionable_per_kloc` and `cr_severe_per_kloc` join lint/lizard/semgrep as density signals that cap the band. When `cr_pr_count == 0` or all walkthroughs are actionable-free (the Free-plan default for many projects), both fields are `null` and the modifier is a no-op — the LLM + remaining density signals decide the band. Scorer cites the top 3 CR comments in the evidence string when `cr_actionable_total > 0`.
+**Banding rule:** when `cr_pr_count > 0`, `cr_actionable_per_kloc` and `cr_severe_per_kloc` join lint / lizard / semgrep as density caps in _Banding_. When `cr_pr_count == 0` or every walkthrough is actionable-free (the Free-plan default), both are `null` and the modifier is a no-op. When `cr_actionable_total > 0`, cite the top 3 CR comments in the evidence string.
 
 ---
 
 ## Raw dumps (flat — files under `raw/` named `code-quality-*`)
 
-Each signal writes its raw output verbatim to the flat `raw/` directory at the workspace root. Absent files = "not extracted" (tool missing / language not present). Aggregates land in `scorecard.yaml` under `data.`.
+Each signal writes its raw output verbatim to the flat `raw/` directory; an absent file means not extracted (tool missing / language not present). Aggregates land in `scorecard.yaml` under `data.`.
 
 | File | Source signal | Shape / note |
 | --- | --- | --- |
 | `raw/code-quality-llm-sample.jsonl` | Signal 1 | 6 lines, one per sampled file: `{path, naming, api_shape, abstraction, modularity, smells, one_sentence}` |
 | `raw/code-quality-lint-ruff.json` | Signal 2 | Raw `ruff check --output-format json` (Python only) |
-| `raw/code-quality-lint-biome.json` | Signal 2 | Raw `biome ci --reporter json` (TS/JS preferred) |
+| `raw/code-quality-lint-biome.json` | Signal 2 | Raw `biome lint --reporter=json` (TS/JS preferred) |
 | `raw/code-quality-lint-eslint.json` | Signal 2 | Raw `eslint --format json` (TS/JS fallback when biome rejects config) |
 | `raw/code-quality-lint-golangci.json` | Signal 2 | Raw `golangci-lint run --out-format json ./...` (Go only) |
 | `raw/code-quality-semgrep-quality.json` | Signal 5 | Post-filter semgrep output (resilience + security rules stripped) |
-| `raw/code-quality-cr-comments.jsonl` | Signal 6 | One line per CR comment: `{pr, file, line, category, label, body_excerpt}`. Empty file when `cr_pr_count == 0` — still emit it (absence of file would be ambiguous with "scrape not run"). |
+| `raw/code-quality-cr-comments.jsonl` | Signal 6 | One line per CR comment: `{pr, file, line, category, label, body_excerpt}`. Empty file when `cr_pr_count == 0` — still emit it, since a missing file reads as "scrape not run". |
 
 `raw/maintainability-lizard.csv` and `raw/maintainability-file-sizes.txt` are shared with 1.4 — read, don't duplicate.
 
@@ -250,8 +238,6 @@ Each signal writes its raw output verbatim to the flat `raw/` directory at the w
 - `cr_actionable_per_kloc` _(participates only when `cr_pr_count > 0`; null = no-op)_
 - `cr_severe_per_kloc` _(participates only when `cr_pr_count > 0`; null = no-op)_
 
-Absolute, per-signal thresholds. CR caps apply only when `cr_pr_count > 0` (null = no-op).
-
 | Band | `llm_quality_score` | Density caps |
 | --- | --- | --- |
 | **5** | ≥ 4.5 | AND `lint_findings_per_kloc` ≤ 5 AND `lizard_god_per_kloc` ≤ 2 AND `files_over_500_loc` ≤ 1 AND (when non-null) `cr_severe_per_kloc` ≤ 0.5 |
@@ -313,7 +299,7 @@ data:
   scoring_method: llm_primary_with_mechanical_modifiers
 ```
 
-When `cr_pr_count == 0` or all walkthroughs are actionable-free (the Free-plan default for most projects), emit:
+When `cr_pr_count == 0` or every walkthrough is actionable-free, emit:
 
 ```yaml
 cr_pr_count: 12
@@ -325,8 +311,6 @@ cr_by_category: {}
 signals_used: [llm, ruff, lizard, filesystem, semgrep, coderabbit_empty]
 ```
 
-`null` values signal the scorer to skip the CR modifier for this project — banding leans on the remaining density signals.
-
 If any required tool failed preflight:
 
 ```yaml
@@ -341,10 +325,7 @@ data:
 
 ## Caveats
 
-- **LLM sample size = 6 files.** Evidence must name every file sampled; the reader should be able to audit each axis score.
+- **LLM sample size = 6 files.** Evidence **must** name every file sampled, so each axis score is auditable.
 - **Lint tool coverage varies.** A TS repo without a `biome.json` or `.eslintrc` will emit different default rules than one with either — `lint_has_config_in_repo` surfaces the difference.
 - **Mixed-language repos.** Sum findings across tools; normalise by `scc_code_loc` (not per-language LoC) to avoid double-normalising.
-- **`lizard` CCN thresholds are language-idiomatic** — handled by per-language lookup (Py/JS/TS 10, Java 12, Go 15, other 15). See [maintainability.md → Signal 2](maintainability.md#signal-2--polyglot-complexity-lizard-per-language-ccn).
-- **CodeRabbit Free plan** sparsity → `cr_*_per_kloc: null`, modifier no-op. Full rationale + plan-upgrade escape hatch in [`SKILL.md → CodeRabbit — calibrating modifier with null-data semantics`](../SKILL.md#coderabbit--calibrating-modifier-with-null-data-semantics).
-- **`AI_AUTHORED_SUSPECTED` signal** (>50% added lines in commits > 500 LoC) remains useful — LLM reviewer should weigh LLM-generated code patterns in `llm_notes` (repetitive scaffolding, over-engineered edge-case branches, inconsistent naming within a single file, etc.).
-- **No silent fallback.** If a tool is absent despite preflight, the signal is emitted as `_NOT_RUN` with `reason`. Grep-based substitutes break the absolute band thresholds and are banned.
+- **AI-authored code** (`AI_AUTHORED_SUSPECTED`: >50% of added lines in commits > 500 LoC) — weigh LLM-generated patterns in `llm_notes`: repetitive scaffolding, over-engineered edge-case branches, inconsistent naming within one file.