@nurix/etna 0.18.4 → 0.18.5

package/data/harnesses/hr/skills/evaluator/criteria/mechanical-ambition.md

@@ -0,0 +1,300 @@
+# Mechanical Ambition — Extraction Plan
+
+- **Sub-criterion:** 2.4 (Product Thinking)
+- **Weight:** 12.5 pts
+- **Contenders:** scc + Semgrep stub rule + git analytics + Tier-A build check + LLM README-vs-shipped diff (layered)
+- **Source quote:** _"Genuinely shippable scope."_
+
+This recipe measures the **mechanical** evidence of ambition (LoC, commits, delivery concentration, stubs, shipped-vs-promised ratio, Tier-A build) against the project's apparent size, and whether the built scope holds together. Bands are **absolute**. Gate this recipe owns: **`BROKEN`** (Tier-A build check fails outside `DEPS_NOT_INSTALLED`; caps **`mechanical_ambition ≤ 2`**).
+
+---
+
+## Extraction recipe
+
+### Signal 1 — LoC + complexity + language mix (scc)
+
+```bash
+scc --format=json <local-repo-path> \
+  --exclude-dir node_modules,.venv,dist,build,.next,target \
+  > /tmp/scc.json
+```
+
+Parse:
+
+```bash
+# Total source LOC
+jq '[.[] | .Code] | add' /tmp/scc.json
+
+# Language breakdown (top 3)
+jq 'sort_by(-.Code) | .[0:3] | map({lang: .Name, code: .Code, files: .Count})' /tmp/scc.json
+
+# Total complexity + complexity-per-file distribution
+jq '[.[] | .Complexity] | add' /tmp/scc.json
+jq '[.[] | .ComplexityAvg] | add / length' /tmp/scc.json
+
+# COCOMO estimate (person-months)
+jq '.[] | .EstimatedScheduleMonths' /tmp/scc.json | head -1
+```
+
+Capture:
+
+- `total_loc`, `total_files`
+- `language_mix` (top 3 by LoC)
+- `total_complexity`, `mean_complexity_per_file`
+- `cocomo_estimated_months` (sanity check: is the built scope reasonable for the project's apparent size/duration?)
+
+### Signal 2 — Stub / placeholder detection (Semgrep)
+
+Create `/tmp/stubs.yml`:
+
+```yaml
+rules:
+  - id: stub-not-implemented
+    message: Stub / placeholder function body
+    severity: INFO
+    languages: [python, js, ts, go, java, rust]
+    pattern-either:
+      - pattern: raise NotImplementedError(...)
+      - pattern: throw new Error("not implemented")
+      - pattern: throw new Error("TODO")
+      - pattern: |
+          def $F(...):
+              pass
+      - pattern: |
+          def $F(...):
+              ...
+      - pattern: |
+          func $F(...) { }
+      - pattern: |
+          return null  // TODO
+```
+
+Run:
+
+```bash
+semgrep --config /tmp/stubs.yml --json <local-repo-path> > /tmp/stubs.json
+```
+
+Parse:
+
+```bash
+jq '[.results[] | {file: .path, line: .start.line, rule: .check_id}]' /tmp/stubs.json
+jq '.results | length' /tmp/stubs.json
+```
+
+Capture: `stub_count`, `stub_per_kloc`, `stub_files` (distinct files with stubs).
+
+### Signal 3 — Git analytics (cramming + scope)
+
+```bash
+cd <local-repo-path>
+
+# Commit count
+git rev-list --count HEAD
+
+# Contributor count
+git shortlog -sn --all | wc -l
+
+# Date range (days between first and last commit)
+first=$(git log --reverse --format=%ct | head -1)
+last=$(git log --format=%ct | head -1)
+echo "$(( (last - first) / 86400 )) days"
+
+# Last-day commit share (cramming signal)
+total=$(git log --format=%ct | wc -l)
+last_day_cutoff=$(( last - 86400 ))
+last_day=$(git log --format=%ct | awk -v c=$last_day_cutoff '$1 >= c' | wc -l)
+echo "Last-day commits: $last_day / $total"
+
+# Commit size distribution — mean lines added per commit
+git log --shortstat --pretty=format: | awk '/insertion/ {sum+=$4; n++} END {print (n?sum/n:0)}'
+```
+
+Capture:
+
+- `commit_count`
+- `contributor_count`
+- `project_duration_days` (first commit → last commit)
+- `last_day_commit_share` (ratio, 0–1)
+- `mean_lines_per_commit`
+- `cramming_suspected` (bool: `last_day_commit_share > 0.3`)
+
+### Signal 4 — Tier-A build check (BROKEN gate trigger)
+
+Dep-less syntax / parse / undefined-name checks. Runs whatever applies to the primary language:
+
+```bash
+cd <local-repo-path>
+
+# Python — syntax + undefined-name
+if find . -maxdepth 4 -name '*.py' -not -path '*/node_modules/*' -not -path '*/.venv/*' | head -1 > /dev/null; then
+  # Syntax
+  find . -name '*.py' -not -path '*/.venv/*' -not -path '*/node_modules/*' \
+    -exec python -m py_compile {} \; 2>&1 | head -20 > /tmp/py-syntax.txt
+  # Undefined names (ruff rule F821)
+  ruff check --select F --output-format json . > /tmp/ruff.json 2>/dev/null || true
+fi
+
+# TypeScript — noEmit parse check
+if [ -f tsconfig.json ]; then
+  npx --yes tsc --noEmit --skipLibCheck --pretty false 2>&1 | head -20 > /tmp/tsc.txt
+fi
+
+# Go — vet + gofmt
+if [ -f go.mod ]; then
+  go mod download 2>&1 | head -5
+  go vet ./... 2>&1 | head -20 > /tmp/govet.txt
+  gofmt -l . 2>&1 | head -20 > /tmp/gofmt.txt
+fi
+
+# JavaScript — node --check on entrypoints
+for f in $(jq -r '.main // empty' package.json 2>/dev/null) index.js src/index.js; do
+  [ -f "$f" ] && node --check "$f" 2>&1
+done
+```
+
+Aggregate `build_check_passed` bool — true if all applicable checks produced zero errors:
+
+- Python: `py_compile` exit 0 AND ruff F-errors == 0
+- TS: `tsc --noEmit --skipLibCheck` exit 0
+- Go: `go vet` exit 0 AND `gofmt -l` empty
+- JS: `node --check` exit 0
+
+Capture: `build_check_passed` (bool), `build_errors_summary` (string — first 20 lines for LLM context), `build_checks_applied` (list of checks that ran).
+
+**Escalation note:** if errors contain `Cannot find module` / `ModuleNotFoundError`, flag `DEPS_NOT_INSTALLED` — Tier-A can't distinguish "broken" from "needs install." Scorer may choose to escalate to Tier-B (shallow install) for such repos.
+
+### Signal 5 — LLM README-vs-shipped diff
+
+Assemble:
+
+1. `README.md` full contents
+2. `tree -L 2 <local-repo-path>` (top 2 levels)
+3. `scc` language-mix output
+4. `stub_count` and top 5 stub file paths
+
+Score:
+
+- Does the README promise N features? How many are visibly shipped?
+- Is the built scope reasonable for the project's apparent size/duration?
+- Are there obvious overreach signals (README talks about "microservices architecture" but repo is one file)?
+- Are there obvious underreach signals (README claims "MVP" but repo has 5000 LOC and 20 stubs)?
+
+Capture:
+
+- `llm_promises_count` (features named in README)
+- `llm_shipped_count` (features visibly implemented)
+- `llm_shipped_ratio` = `llm_shipped_count ÷ llm_promises_count`
+- `llm_scope_match_score` (0–5) — integrated "right-sized?" judgment
+- `llm_notes` (one sentence, e.g. "README promises 4 features; 2 shipped + 2 stubbed under feature flags")
+
+---
+
+## Raw dumps (flat — files under `raw/` named `mechanical-ambition-*`)
+
+Each signal writes its raw output to the flat `raw/` directory at the workspace root. Absent files = not extracted. Aggregates go to `scorecard.yaml` under `data.`.
+
+| File | Source signal | Shape / note |
+| --- | --- | --- |
+| `raw/mechanical-ambition-scc.json` | Signal 1 | Raw `scc --format=json` output (per-language LoC, complexity, COCOMO estimate) |
+| `raw/mechanical-ambition-stubs.json` | Signal 2 | Semgrep stub-rule findings: `[{file, line, rule, snippet}]`. Empty array when no stubs. |
+| `raw/mechanical-ambition-git-analytics.json` | Signal 3 | `{commit_count, contributor_count, project_duration_days, first_commit, last_commit, last_day_commit_share, mean_lines_per_commit, cramming_suspected, commits_by_day: [{date, count}]}` |
+| `raw/mechanical-ambition-tier-a-build.log` | Signal 4 | Combined output of checks that ran (py_compile, ruff -F, tsc --noEmit, go vet, gofmt, node --check). First 20 lines per tool. |
+| `raw/mechanical-ambition-llm-read.md` | Signal 5 | README-vs-shipped diff: `{promises: [...], shipped: [...], shipped_ratio, scope_match_score, notable_gaps, one_sentence}` |
+
+Additional short-lived intermediates: the `stubs.yml` semgrep rule config is a skill artifact — generate it under `$SKILL_DIR/criteria/mechanical-ambition-stubs.yml` (or inline on-the-fly), not in the repo workspace.
+
+---
+
+## Aggregation → 0–5 band
+
+Scorer integrates the 5 signals. Rough directionality:
+
+| Signal | Pulls score UP | Pulls score DOWN |
+| --- | --- | --- |
+| `build_check_passed` | true | false → `BROKEN` gate fires |
+| `commit_count` + `total_loc` | commits > 50, LoC > 1000 | both low → thin delivery, pulls score down |
+| `stub_per_kloc` | < 0.5 | > 5 |
+| `llm_shipped_ratio` | ≥ 0.8 | < 0.5 |
+| `cramming_suspected` | false | true (last-day share > 0.3) |
+| `cocomo_estimated_months` | matches apparent size/duration | wildly overshoots or undershoots |
+
+**Banding (absolute):**
+
+- **5** — Scope matches delivery (`llm_shipped_ratio ≥ 0.9`); `stub_per_kloc < 0.5`; build passes; delivery evenly spread; LLM ≥ 4
+- **4** — Strong delivery, one visible gap (e.g. 3/4 features shipped, minor delivery-concentration spike)
+- **3** — Most features shipped (0.6–0.8 ratio); 1–3 unfinished; stub density 0.5–2/kLoC; some delivery concentration
+- **2** — Overreached visibly; shipped ratio 0.4–0.6; stubs 2–5/kLoC; **`BROKEN` gate fires here (cap at 2)**.
+- **1** — README promises 4+ features, repo ships 1; stub density > 5/kLoC; delivery heavily concentrated; or gates firing AND low quality
+- **0** — Suppressed (gates enforce; use only when repo is truly empty or broken)
+
+---
+
+## Gates
+
+| Gate | Trigger | Cap |
+| --- | --- | --- |
+| `BROKEN` | `build_check_passed == false` (and not because of `DEPS_NOT_INSTALLED`) | `mechanical_ambition ≤ 2` |
+
+Affects this sub-criterion only. Canonical gate list in [../SKILL.md](../SKILL.md).
+
+---
+
+## Fallback — signals missing
+
+- scc doesn't support a language → LoC count may under-report; document in evidence.
+- Semgrep stub rule misses project-specific stub patterns → LLM catches what regex missed; increase weight of Signal 5.
+- Repo has < 5 commits → git analytics degrade (cramming, commit-rate meaningless); rely on Signals 1, 2, 5.
+- `DEPS_NOT_INSTALLED` flag on Tier-A → `BROKEN` gate does NOT fire on this alone; scorer should either: (a) escalate to Tier-B, or (b) defer build-status judgment to the LLM's read of repo shape.
+- README missing → Signal 5 is degraded; emit `llm_confidence: low`.
+
+Annotate evidence with `"Signals used: [...]"`.
+
+---
+
+## Output (what the scorer emits)
+
+```yaml
+sub_criterion: mechanical_ambition
+score: 3
+evidence: "1240 LOC TypeScript + 380 LOC Python; stub density 1.3/kLoC; git shows 42 commits over 6 days, last-day share 0.28 (borderline delivery concentration); Tier-A build passes; README promises 4 features, 2.5 shipped (dashboard + API done; scheduler + notifications stubbed). Overreached on scheduler."
+data:
+  total_loc: 1620
+  total_files: 47
+  language_mix:
+    [
+      { lang: TypeScript, code: 1240, files: 31 },
+      { lang: Python, code: 380, files: 16 },
+    ]
+  total_complexity: 142
+  mean_complexity_per_file: 3.0
+  cocomo_estimated_months: 0.4
+  stub_count: 9
+  stub_per_kloc: 1.3
+  stub_files: [src/scheduler.ts, src/notifications.ts, ...]
+  commit_count: 42
+  contributor_count: 3
+  project_duration_days: 6
+  last_day_commit_share: 0.28
+  mean_lines_per_commit: 38.5
+  cramming_suspected: false
+  build_check_passed: true
+  build_checks_applied: [tsc_noemit, ruff_F, py_compile]
+  build_errors_summary: ""
+  llm_promises_count: 4
+  llm_shipped_count: 2
+  llm_shipped_ratio: 0.5
+  llm_scope_match_score: 3
+  signals_used: [scc, semgrep_stubs, git_analytics, build_check, llm]
+  gates_triggered: []
+```
+
+---
+
+## Caveats
+
+- **Tier-A build check catches ~80% of breakage.** Import-resolution errors need deps installed. `DEPS_NOT_INSTALLED` flag suppresses the `BROKEN` gate when the check failed for install reasons.
+- **Stub detection via Semgrep misses project-specific patterns.** A function that returns hardcoded fake data without any stub marker looks "shipped" but isn't. LLM Signal 5 compensates.
+- **Delivery-concentration detection is heuristic.** Last-day share > 30% flags concentrated delivery but doesn't distinguish "final polish" from a single large drop. LLM evidence should adjudicate when the flag fires.
+- **`llm_shipped_ratio` depends on how the README frames features.** Terse READMEs may under-count promises; florid READMEs may over-count. Evidence should show the promise list.
+- **COCOMO is a sanity check, not a score driver.** `cocomo_estimated_months > 3` is a soft-overreach signal relative to the project's apparent size; `< 0.1` is a soft-underreach signal. Don't gate on it.

package/data/harnesses/hr/skills/evaluator/criteria/problem-substantiveness.md

@@ -0,0 +1,88 @@
+# Problem Substantiveness — Extraction Plan
+
+- **Sub-criterion:** 2.1 (Product Thinking)
+- **Weight:** 12.5 pts
+- **Source quote:** _"Is the problem clearly stated, non-trivial, with defined user/use-case?"_
+
+This recipe scores whether the problem is substantively framed — a crisp, non-trivial problem statement with a defined user and use-case, versus a toy-shaped placeholder.
+
+---
+
+## Extraction recipe
+
+LLM-primary. The evaluator reads the top of the repo (README, any PRD/SPEC doc, sample data, one or two handler files) and answers: **Is this a substantive, well-defined problem, or a toy-shaped placeholder?**
+
+### Signal 1 — Problem framing (LLM interpretive)
+
+Feed the LLM the following context:
+
+```bash
+# README (truncated to first ~500 lines)
+head -500 "$REPO/README.md" > /tmp/prob-sub-readme.md
+
+# PRD / SPEC / DESIGN docs if present
+find "$REPO" -maxdepth 3 -iname "PRD*.md" -o -iname "SPEC*.md" -o -iname "DESIGN*.md" \
+  -not -path '*/node_modules/*' -not -path '*/.venv/*' 2>/dev/null \
+  | xargs -I{} head -200 {} 2>/dev/null > /tmp/prob-sub-docs.md
+
+# Sample data files — check provenance
+find "$REPO" -maxdepth 4 \( -iname '*.csv' -o -iname '*.json' -o -iname 'sample*' -o -iname 'fixture*' -o -iname 'seed*' \) \
+  -not -path '*/node_modules/*' -not -path '*/.venv/*' 2>/dev/null | head -10 > /tmp/prob-sub-data-files.txt
+
+# Top-level folder structure (reveals user/use-case intent)
+tree -L 2 -I 'node_modules|.venv|.git|dist|build' "$REPO" 2>/dev/null | head -80 > /tmp/prob-sub-tree.txt
+```
+
+Write the LLM read to `raw/problem-substantiveness-llm-read.md`. The LLM emits:
+
+```yaml
+detected_domain: "internal dev tooling" | "customer-facing SaaS" | "infra automation" | "ops observability" | "data pipeline" | "agent/LLM tooling" | ...
+problem_clarity_score: 0-5          # 5 = crisp problem statement with user + constraint + outcome named; 0 = no statement
+target_user_inferred: "internal engineers" | "support agents" | "end customers" | "unclear" | ...
+sample_data_provenance: "synthetic" | "redacted-real" | "production" | "none" | "unclear"
+toy_pattern_detected: true | false  # hello-world, generic TODO app, calculator, tic-tac-toe, etc.
+one_sentence: "<LLM's one-line verdict on whether this is a substantive problem>"
+```
+
+### Signal 2 — Proxies JSON
+
+Aggregate the LLM output into `raw/problem-substantiveness-proxies.json`:
+
+```json
+{
+  "detected_domain": "...",
+  "problem_clarity_score": 0-5,
+  "target_user_inferred": "...",
+  "sample_data_provenance": "...",
+  "toy_pattern_detected": true|false
+}
+```
+
+---
+
+## Banding (absolute)
+
+| Band | Criteria |
+| --- | --- |
+| **5** | Problem statement is crisp: user + constraint + outcome named. Domain is non-trivial (enterprise ops / a non-trivial real-world domain / novel agent workflow). Sample data is production-shaped. `problem_clarity_score: 5`, `toy_pattern_detected: false`. |
+| **4** | Problem statement is substantive with one minor gap (user inferable but not named; outcome fuzzy). `problem_clarity_score: 4`, `toy_pattern_detected: false`. |
+| **3** | Problem is stated but under-specified: missing user definition OR missing success criterion. `problem_clarity_score: 3`. |
+| **2** | Problem statement is vague or generic. README is mostly boilerplate. `problem_clarity_score: 2`. |
+| **1** | No problem statement, or README is an unmodified starter/template README. `problem_clarity_score ≤ 1`. |
+| **0** | Empty README + no PRD/SPEC + toy pattern detected. |
+
+**Toy-pattern auto-floor:** if `toy_pattern_detected == true` (hello-world, generic TODO, calculator, tic-tac-toe, etc.), cap the band at **2** regardless of other signals — a clear problem statement doesn't rescue a toy.
+
+---
+
+## Raw dumps
+
+- `raw/problem-substantiveness-llm-read.md` — LLM's full interpretive read with axis scores.
+- `raw/problem-substantiveness-proxies.json` — aggregated JSON of signals (also embedded in `scorecard.yaml.problem_substantiveness.data`).
+
+---
+
+## Not in scope
+
+- Judging whether the problem matches a specific organization's real priorities needs operational context — out of scope; this scores whether the problem is substantively framed, not whether it's the right problem for any given org to be solving.
+- Judging the quality of the implementation — that's 1.1–1.4 + 2.4. This recipe only looks at framing.