@nurix/etna 0.20.35 → 0.20.37
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/data/catalog/rules/changelog.md +1 -1
- package/data/catalog/skills/nustack/SKILL.md +2 -2
- package/data/catalog/skills/nustack/references/cli-design.md +35 -35
- package/data/catalog/skills/nustack/references/monorepo-master-list.md +6 -6
- package/data/catalog/skills/nustack/references/monorepo-setup.md +13 -13
- package/data/catalog/skills/nustack/references/publishing.md +40 -39
- package/data/harness-index.json +1 -1
- package/data/harnesses/code/CLAUDE.md +4 -4
- package/data/harnesses/code/agents/brainstorm.md +2 -2
- package/data/harnesses/code/agents/lld-mode.md +2 -2
- package/data/harnesses/code/agents/maintenance.md +17 -17
- package/data/harnesses/code/codex/agents/brainstorm.toml +6 -2
- package/data/harnesses/code/codex/agents/lld-mode.toml +2 -2
- package/data/harnesses/code/codex/agents/maintenance.toml +18 -17
- package/data/harnesses/code/docs/README.md +9 -17
- package/data/harnesses/code/docs/history/CHANGELOG.md +19 -18
- package/data/harnesses/code/docs/structure.md +60 -45
- package/data/harnesses/code/harness.json +2 -2
- package/data/harnesses/code/hooks/compliance.mjs +8 -2
- package/data/harnesses/code/hooks/lib/sweep.mjs +3 -2
- package/data/harnesses/code/rules/changelog.md +1 -1
- package/data/harnesses/code/skills/boundary/SKILL.md +33 -22
- package/data/harnesses/code/skills/boundary/references/index.md +2 -4
- package/data/harnesses/code/skills/boundary/references/mechanisms.md +3 -10
- package/data/harnesses/code/skills/boundary/references/packs/frontend.md +17 -17
- package/data/harnesses/code/skills/boundary/references/packs/generic-tail.md +1 -1
- package/data/harnesses/code/skills/brainstorm/README.md +11 -13
- package/data/harnesses/code/skills/brainstorm/SKILL.md +149 -130
- package/data/harnesses/code/skills/brainstorm/references/doctrine.md +48 -51
- package/data/harnesses/code/skills/brainstorm/references/harvest.md +51 -57
- package/data/harnesses/code/skills/brainstorm/references/session-schema.md +48 -42
- package/data/harnesses/code/skills/design/SKILL.md +31 -35
- package/data/harnesses/code/skills/design/data/iconography.md +2 -2
- package/data/harnesses/code/skills/design/references/compose.md +14 -16
- package/data/harnesses/code/skills/design/references/foundations.md +5 -5
- package/data/harnesses/code/skills/design/references/intake.md +6 -6
- package/data/harnesses/code/skills/design/references/object-pass.md +1 -1
- package/data/harnesses/code/skills/design/references/ship-gate.md +5 -5
- package/data/harnesses/code/skills/design/references/ux-patterns/Readme.md +4 -4
- package/data/harnesses/code/skills/design/references/ux-patterns/ai-layer.md +1 -1
- package/data/harnesses/code/skills/design/references/ux-patterns/container-choice.md +1 -1
- package/data/harnesses/code/skills/design/references/ux-patterns/dashboards.md +1 -1
- package/data/harnesses/code/skills/design/references/ux-patterns/modes.md +1 -1
- package/data/harnesses/code/skills/design/references/ux-patterns/object-model.md +2 -2
- package/data/harnesses/code/skills/design/references/ux-patterns/promotion-demotion.md +1 -1
- package/data/harnesses/code/skills/design/references/ux-patterns/settings.md +1 -1
- package/data/harnesses/code/skills/design/references/ux-patterns/standard-pages.md +1 -1
- package/data/harnesses/code/skills/documentation/SKILL.md +92 -143
- package/data/harnesses/code/skills/documentation/references/examples.md +45 -72
- package/data/harnesses/code/skills/documentation/references/frontmatter.md +8 -8
- package/data/harnesses/code/skills/documentation/references/instruction-style.md +3 -3
- package/data/harnesses/code/skills/documentation/references/source-comments.md +19 -0
- package/data/harnesses/code/skills/documentation/references/style-guide.md +25 -11
- package/data/harnesses/code/skills/documentation/references/untemplated.md +11 -10
- package/data/harnesses/code/skills/documentation/scripts/doc-lint.mjs +181 -40
- package/data/harnesses/code/skills/documentation/templates/Readme.md +68 -41
- package/data/harnesses/code/skills/documentation/templates/acceptance-criteria.md +11 -19
- package/data/harnesses/code/skills/documentation/templates/api-reference.md +8 -3
- package/data/harnesses/code/skills/documentation/templates/architecture-decision.md +8 -12
- package/data/harnesses/code/skills/documentation/templates/backlog.md +2 -2
- package/data/harnesses/code/skills/documentation/templates/behavioral-checklist.md +10 -17
- package/data/harnesses/code/skills/documentation/templates/bookkeeping.md +2 -2
- package/data/harnesses/code/skills/documentation/templates/changelog-entry.md +2 -2
- package/data/harnesses/code/skills/documentation/templates/changelog-index.md +21 -29
- package/data/harnesses/code/skills/documentation/templates/docs-readme.md +4 -4
- package/data/harnesses/code/skills/documentation/templates/doctrine.md +3 -3
- package/data/harnesses/code/skills/documentation/templates/domain-model.md +10 -5
- package/data/harnesses/code/skills/documentation/templates/environment-registry.md +7 -2
- package/data/harnesses/code/skills/documentation/templates/feature.md +6 -6
- package/data/harnesses/code/skills/documentation/templates/glossary.md +6 -1
- package/data/harnesses/code/skills/documentation/templates/handover.md +17 -17
- package/data/harnesses/code/skills/documentation/templates/implementation-map.md +10 -5
- package/data/harnesses/code/skills/documentation/templates/journal.md +2 -2
- package/data/harnesses/code/skills/documentation/templates/memory-main.md +2 -2
- package/data/harnesses/code/skills/documentation/templates/migration-runbook.md +1 -1
- package/data/harnesses/code/skills/documentation/templates/non-goals.md +8 -3
- package/data/harnesses/code/skills/documentation/templates/operations-procedure.md +8 -3
- package/data/harnesses/code/skills/documentation/templates/package.md +5 -9
- package/data/harnesses/code/skills/documentation/templates/plan.md +5 -11
- package/data/harnesses/code/skills/documentation/templates/protected-files.md +7 -2
- package/data/harnesses/code/skills/documentation/templates/root-readme.md +5 -5
- package/data/harnesses/code/skills/documentation/templates/security-overview.md +9 -4
- package/data/harnesses/code/skills/documentation/templates/setup.md +6 -6
- package/data/harnesses/code/skills/documentation/templates/system-overview.md +12 -7
- package/data/harnesses/code/skills/documentation/templates/system.md +2 -2
- package/data/harnesses/code/skills/documentation/templates/testing-strategy.md +10 -5
- package/data/harnesses/code/skills/documentation/templates/ux-behavior.md +14 -5
- package/data/harnesses/code/skills/issue-bridge/SKILL.md +51 -16
- package/data/harnesses/code/skills/issue-bridge/references/trackers/github.md +5 -5
- package/data/harnesses/code/skills/issue-bridge/references/trackers/jira.md +11 -8
- package/data/harnesses/code/skills/issue-bridge/references/trackers/linear.md +16 -11
- package/data/harnesses/code/skills/nustack-graph/SKILL.md +38 -29
- package/data/harnesses/code/skills/polygraph/references/dispatch.md +31 -18
- package/data/harnesses/code/skills/polygraph/references/rows.md +11 -6
- package/data/harnesses/code/skills/schema-design/SKILL.md +39 -41
- package/data/harnesses/code/skills/test/SKILL.md +38 -42
- package/data/harnesses/code/skills/testing/SKILL.md +83 -73
- package/data/harnesses/code/skills/workflow-mapper/SKILL.md +156 -193
- package/data/harnesses/code/skills/workflow-mapper/references/patterns.md +15 -15
- package/data/harnesses/delivery/CLAUDE.md +1 -1
- package/data/harnesses/delivery/agents/brainstorm.md +2 -2
- package/data/harnesses/delivery/agents/lld-mode.md +2 -2
- package/data/harnesses/delivery/agents/maintenance.md +17 -17
- package/data/harnesses/delivery/codex/agents/brainstorm.toml +6 -2
- package/data/harnesses/delivery/codex/agents/lld-mode.toml +2 -2
- package/data/harnesses/delivery/codex/agents/maintenance.toml +18 -17
- package/data/harnesses/delivery/harness.json +2 -2
- package/data/harnesses/delivery/rules/changelog.md +1 -1
- package/data/harnesses/delivery/skills/brainstorm/README.md +11 -13
- package/data/harnesses/delivery/skills/brainstorm/SKILL.md +149 -130
- package/data/harnesses/delivery/skills/brainstorm/references/doctrine.md +48 -51
- package/data/harnesses/delivery/skills/brainstorm/references/harvest.md +51 -57
- package/data/harnesses/delivery/skills/brainstorm/references/session-schema.md +48 -42
- package/data/harnesses/delivery/skills/cost-report/SKILL.md +10 -10
- package/data/harnesses/delivery/skills/documentation/SKILL.md +92 -143
- package/data/harnesses/delivery/skills/documentation/references/examples.md +45 -72
- package/data/harnesses/delivery/skills/documentation/references/frontmatter.md +8 -8
- package/data/harnesses/delivery/skills/documentation/references/instruction-style.md +3 -3
- package/data/harnesses/delivery/skills/documentation/references/source-comments.md +19 -0
- package/data/harnesses/delivery/skills/documentation/references/style-guide.md +25 -11
- package/data/harnesses/delivery/skills/documentation/references/untemplated.md +11 -10
- package/data/harnesses/delivery/skills/documentation/scripts/doc-lint.mjs +181 -40
- package/data/harnesses/delivery/skills/documentation/templates/Readme.md +68 -41
- package/data/harnesses/delivery/skills/documentation/templates/acceptance-criteria.md +11 -19
- package/data/harnesses/delivery/skills/documentation/templates/api-reference.md +8 -3
- package/data/harnesses/delivery/skills/documentation/templates/architecture-decision.md +8 -12
- package/data/harnesses/delivery/skills/documentation/templates/backlog.md +2 -2
- package/data/harnesses/delivery/skills/documentation/templates/behavioral-checklist.md +10 -17
- package/data/harnesses/delivery/skills/documentation/templates/bookkeeping.md +2 -2
- package/data/harnesses/delivery/skills/documentation/templates/changelog-entry.md +2 -2
- package/data/harnesses/delivery/skills/documentation/templates/changelog-index.md +21 -29
- package/data/harnesses/delivery/skills/documentation/templates/docs-readme.md +4 -4
- package/data/harnesses/delivery/skills/documentation/templates/doctrine.md +3 -3
- package/data/harnesses/delivery/skills/documentation/templates/domain-model.md +10 -5
- package/data/harnesses/delivery/skills/documentation/templates/environment-registry.md +7 -2
- package/data/harnesses/delivery/skills/documentation/templates/feature.md +6 -6
- package/data/harnesses/delivery/skills/documentation/templates/glossary.md +6 -1
- package/data/harnesses/delivery/skills/documentation/templates/handover.md +17 -17
- package/data/harnesses/delivery/skills/documentation/templates/implementation-map.md +10 -5
- package/data/harnesses/delivery/skills/documentation/templates/journal.md +2 -2
- package/data/harnesses/delivery/skills/documentation/templates/memory-main.md +2 -2
- package/data/harnesses/delivery/skills/documentation/templates/migration-runbook.md +1 -1
- package/data/harnesses/delivery/skills/documentation/templates/non-goals.md +8 -3
- package/data/harnesses/delivery/skills/documentation/templates/operations-procedure.md +8 -3
- package/data/harnesses/delivery/skills/documentation/templates/package.md +5 -9
- package/data/harnesses/delivery/skills/documentation/templates/plan.md +5 -11
- package/data/harnesses/delivery/skills/documentation/templates/protected-files.md +7 -2
- package/data/harnesses/delivery/skills/documentation/templates/root-readme.md +5 -5
- package/data/harnesses/delivery/skills/documentation/templates/security-overview.md +9 -4
- package/data/harnesses/delivery/skills/documentation/templates/setup.md +6 -6
- package/data/harnesses/delivery/skills/documentation/templates/system-overview.md +12 -7
- package/data/harnesses/delivery/skills/documentation/templates/system.md +2 -2
- package/data/harnesses/delivery/skills/documentation/templates/testing-strategy.md +10 -5
- package/data/harnesses/delivery/skills/documentation/templates/ux-behavior.md +14 -5
- package/data/harnesses/delivery/skills/escalate/SKILL.md +11 -11
- package/data/harnesses/delivery/skills/prompt-eval/SKILL.md +8 -8
- package/data/harnesses/delivery/skills/prompt-library/SKILL.md +30 -9
- package/data/harnesses/delivery/skills/route/SKILL.md +8 -8
- package/data/harnesses/etna/CLAUDE.md +2 -2
- package/data/harnesses/etna/agents/brainstorm.md +2 -2
- package/data/harnesses/etna/agents/lld-mode.md +2 -2
- package/data/harnesses/etna/agents/maintenance.md +17 -17
- package/data/harnesses/etna/codex/agents/brainstorm.toml +6 -2
- package/data/harnesses/etna/codex/agents/lld-mode.toml +2 -2
- package/data/harnesses/etna/codex/agents/maintenance.toml +18 -17
- package/data/harnesses/etna/harness.json +2 -2
- package/data/harnesses/etna/rules/changelog.md +1 -1
- package/data/harnesses/etna/skills/boundary/SKILL.md +33 -22
- package/data/harnesses/etna/skills/boundary/references/index.md +2 -4
- package/data/harnesses/etna/skills/boundary/references/mechanisms.md +3 -10
- package/data/harnesses/etna/skills/boundary/references/packs/frontend.md +17 -17
- package/data/harnesses/etna/skills/boundary/references/packs/generic-tail.md +1 -1
- package/data/harnesses/etna/skills/brainstorm/README.md +11 -13
- package/data/harnesses/etna/skills/brainstorm/SKILL.md +149 -130
- package/data/harnesses/etna/skills/brainstorm/references/doctrine.md +48 -51
- package/data/harnesses/etna/skills/brainstorm/references/harvest.md +51 -57
- package/data/harnesses/etna/skills/brainstorm/references/session-schema.md +48 -42
- package/data/harnesses/etna/skills/documentation/SKILL.md +92 -143
- package/data/harnesses/etna/skills/documentation/references/examples.md +45 -72
- package/data/harnesses/etna/skills/documentation/references/frontmatter.md +8 -8
- package/data/harnesses/etna/skills/documentation/references/instruction-style.md +3 -3
- package/data/harnesses/etna/skills/documentation/references/source-comments.md +19 -0
- package/data/harnesses/etna/skills/documentation/references/style-guide.md +25 -11
- package/data/harnesses/etna/skills/documentation/references/untemplated.md +11 -10
- package/data/harnesses/etna/skills/documentation/scripts/doc-lint.mjs +181 -40
- package/data/harnesses/etna/skills/documentation/templates/Readme.md +68 -41
- package/data/harnesses/etna/skills/documentation/templates/acceptance-criteria.md +11 -19
- package/data/harnesses/etna/skills/documentation/templates/api-reference.md +8 -3
- package/data/harnesses/etna/skills/documentation/templates/architecture-decision.md +8 -12
- package/data/harnesses/etna/skills/documentation/templates/backlog.md +2 -2
- package/data/harnesses/etna/skills/documentation/templates/behavioral-checklist.md +10 -17
- package/data/harnesses/etna/skills/documentation/templates/bookkeeping.md +2 -2
- package/data/harnesses/etna/skills/documentation/templates/changelog-entry.md +2 -2
- package/data/harnesses/etna/skills/documentation/templates/changelog-index.md +21 -29
- package/data/harnesses/etna/skills/documentation/templates/docs-readme.md +4 -4
- package/data/harnesses/etna/skills/documentation/templates/doctrine.md +3 -3
- package/data/harnesses/etna/skills/documentation/templates/domain-model.md +10 -5
- package/data/harnesses/etna/skills/documentation/templates/environment-registry.md +7 -2
- package/data/harnesses/etna/skills/documentation/templates/feature.md +6 -6
- package/data/harnesses/etna/skills/documentation/templates/glossary.md +6 -1
- package/data/harnesses/etna/skills/documentation/templates/handover.md +17 -17
- package/data/harnesses/etna/skills/documentation/templates/implementation-map.md +10 -5
- package/data/harnesses/etna/skills/documentation/templates/journal.md +2 -2
- package/data/harnesses/etna/skills/documentation/templates/memory-main.md +2 -2
- package/data/harnesses/etna/skills/documentation/templates/migration-runbook.md +1 -1
- package/data/harnesses/etna/skills/documentation/templates/non-goals.md +8 -3
- package/data/harnesses/etna/skills/documentation/templates/operations-procedure.md +8 -3
- package/data/harnesses/etna/skills/documentation/templates/package.md +5 -9
- package/data/harnesses/etna/skills/documentation/templates/plan.md +5 -11
- package/data/harnesses/etna/skills/documentation/templates/protected-files.md +7 -2
- package/data/harnesses/etna/skills/documentation/templates/root-readme.md +5 -5
- package/data/harnesses/etna/skills/documentation/templates/security-overview.md +9 -4
- package/data/harnesses/etna/skills/documentation/templates/setup.md +6 -6
- package/data/harnesses/etna/skills/documentation/templates/system-overview.md +12 -7
- package/data/harnesses/etna/skills/documentation/templates/system.md +2 -2
- package/data/harnesses/etna/skills/documentation/templates/testing-strategy.md +10 -5
- package/data/harnesses/etna/skills/documentation/templates/ux-behavior.md +14 -5
- package/data/harnesses/etna/skills/workflow-mapper/SKILL.md +156 -193
- package/data/harnesses/etna/skills/workflow-mapper/references/patterns.md +15 -15
- package/data/harnesses/hr/harness.json +1 -1
- package/data/harnesses/hr/skills/evaluator/SKILL.md +71 -73
- package/data/harnesses/hr/skills/evaluator/criteria/code-quality.md +28 -47
- package/data/harnesses/hr/skills/evaluator/criteria/deployment-readiness.md +33 -43
- package/data/harnesses/hr/skills/evaluator/criteria/maintainability.md +34 -46
- package/data/harnesses/hr/skills/evaluator/criteria/mechanical-ambition.md +13 -13
- package/data/harnesses/hr/skills/evaluator/criteria/problem-substantiveness.md +7 -7
- package/data/harnesses/hr/skills/evaluator/criteria/testing.md +36 -41
- package/data/harnesses/hr/skills/evaluator/criteria/ux-mechanics.md +25 -11
- package/data/harnesses/hr/skills/evaluator/criteria/value-articulation.md +7 -7
- package/data/harnesses/hr/skills/evaluator/preflight.md +27 -46
- package/data/harnesses/hr/skills/interview-eval/SKILL.md +37 -42
- package/data/harnesses/hr/skills/interview-eval/rubric.md +21 -23
- package/package.json +2 -2
|
@@ -5,53 +5,56 @@ description: "Use when assessing how a human steered a Claude Code session — p
|
|
|
5
5
|
|
|
6
6
|
# interview-eval
|
|
7
7
|
|
|
8
|
-
Judge **how a human drove a Claude Code session
|
|
8
|
+
Judge **how a human drove a Claude Code session**: how they framed the problem, decomposed it, fed context, verified results, debugged, corrected course, and revealed engineering judgment through their prompts. Session transcripts in; per-session reports and scorecards out, plus a candidate rollup when there are several.
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
The sibling `evaluator` skill scores a repo's artifacts; this skill scores the conversation that produced them. They share no rubric.
|
|
11
11
|
|
|
12
12
|
## The one rule that governs every score
|
|
13
13
|
|
|
14
|
-
**You are scoring the human, not the model.** Claude's
|
|
14
|
+
**You are scoring the human, not the model.** Claude's clean code earns the candidate nothing and Claude's buggy code costs them nothing — **unless** the human's prompt caused it or the human failed to catch it. Reward the prompt that set Claude up to succeed, the verification that caught Claude's mistake, the redirect that pulled Claude off a bad path. Turn count, prompt length, and verbosity are **never** quality signals — precision, judgment, and verification are: three sharp prompts that let a capable agent run is good steering; forty vague nudges that rubber-stamp broken output is bad steering, even if something shipped.
|
|
15
15
|
|
|
16
16
|
## Input
|
|
17
17
|
|
|
18
|
-
- **
|
|
19
|
-
-
|
|
20
|
-
- **Optional context**
|
|
21
|
-
|
|
22
|
-
If the user just points at this folder, treat every `*.jsonl` as one candidate's sessions unless told otherwise.
|
|
18
|
+
- **Session transcripts** — Claude Code `.jsonl` files, as a list, a glob, or a directory (every `*.jsonl` inside). A bare directory is one candidate's sessions unless the user says otherwise.
|
|
19
|
+
- **`out_dir`** — default `./interview-eval-out/`. One subfolder per session, plus the rollup at the root when there is more than one session.
|
|
20
|
+
- **Optional context** — the candidate's seniority or role, the task brief, time limits, whether sessions are independent tasks or one continued task. Fold in what the user volunteers; **never** invent it.
|
|
23
21
|
|
|
24
22
|
## Process
|
|
25
23
|
|
|
26
24
|
### 0. Preflight (cheap)
|
|
27
25
|
|
|
28
|
-
|
|
26
|
+
The only dependency is `python3` (stdlib only) — no installs, no network. If `python3` is missing, say so and stop.
|
|
29
27
|
|
|
30
28
|
### 1. Extract an evidence pack per session (deterministic)
|
|
31
29
|
|
|
32
|
-
Run the extractor — it
|
|
30
|
+
Run the extractor — it surfaces signal and **never** scores:
|
|
33
31
|
|
|
34
32
|
```bash
|
|
35
33
|
python3 <skill_dir>/extract_session.py SESSION.jsonl --out <out_dir>/<session-slug>/ --json
|
|
36
34
|
```
|
|
37
35
|
|
|
38
|
-
It
|
|
39
|
-
|
|
40
|
-
- **Turn-by-turn**: each human prompt (IDE tags stripped) followed by a summary of what the agent did (tool counts + sample targets). This is your primary evidence — quote `U<n>` turn numbers in every finding.
|
|
41
|
-
- **Metrics**: prompt count, wall-clock & idle gaps, tool histogram, permission-mode transitions, plan-mode / subagent / AskUserQuestion / slash-command usage, interrupts, system/API errors.
|
|
42
|
-
- **Auto red-flag signals**: secrets pasted into prompts, destructive commands the agent ran (`rm -rf ~`, force-push, `DROP TABLE`, …). These are **leads to verify**, not verdicts.
|
|
36
|
+
It prints a markdown evidence pack on stdout and writes `<sessionId>.session.json` into `--out`. The pack holds:
|
|
43
37
|
|
|
44
|
-
|
|
38
|
+
| Section | Contents |
|
|
39
|
+
| --- | --- |
|
|
40
|
+
| Turn-by-turn | Each human prompt (IDE tags stripped) as `U<n>` with its timestamp and permission mode, then a summary of what the agent did (tool counts, sample targets). Your primary evidence — cite `U<n>` in every finding |
|
|
41
|
+
| Session metrics | Prompt count and length, wall-clock duration, tool histogram, permission-mode transitions, plan-mode / subagent / `AskUserQuestion` / slash-command usage, interrupts, system/API errors |
|
|
42
|
+
| Auto red-flag signals | Secrets pasted into prompts; destructive commands the agent ran (`rm -rf ~`, force-push, `DROP TABLE`, …). Leads to verify, **never** verdicts |
|
|
45
43
|
|
|
46
|
-
|
|
44
|
+
1. Read the **full** evidence pack before scoring. **Never** score from metrics alone — great metrics can hide terrible steering, and the reverse.
|
|
45
|
+
2. For a long session, open the raw `.jsonl` around every turn you intend to quote, to confirm its context.
|
|
46
|
+
3. Separate noise from signal:
|
|
47
|
+
- `system_errors` with `status: 529` or `ECONNRESET` are Anthropic-side overload — **never** penalize them. Neither are tool-permission prompts or model mistakes the candidate then caught.
|
|
48
|
+
- An interrupt is either a sharp early redirect (good) or thrash (bad) — read the surrounding turns to tell which.
|
|
49
|
+
- An idle gap (read from the turn timestamps) may be thinking, out-of-band testing, or absence — note it, never assume.
|
|
47
50
|
|
|
48
51
|
### 2. Classify the session type
|
|
49
52
|
|
|
50
|
-
Pick the dominant type
|
|
53
|
+
Pick the dominant type, noting a blend: `greenfield-build` · `feature-add` · `debugging` · `refactor` · `ops` · `research` · `review`. The type sets which dimensions are primary, secondary, or N/A — `rubric.md` → "Session-type matrix". A pure debugging session makes **E** the main event and may mark **B** N/A.
|
|
51
54
|
|
|
52
55
|
### 3. Score the 9 dimensions
|
|
53
56
|
|
|
54
|
-
|
|
57
|
+
Read **[`rubric.md`](rubric.md)** before every evaluation — the band anchors, flags, and gates live there; **never** score from memory.
|
|
55
58
|
|
|
56
59
|
| # | Dimension | Weight | The senior question it answers |
|
|
57
60
|
| --- | --- | :-: | --- |
|
|
@@ -65,13 +68,13 @@ Full band anchors (0–5, with concrete "looks like" descriptions and green/red
|
|
|
65
68
|
| **H** | Workflow & Tooling Leverage | 5 | Did they use the harness well (plan mode, subagents, git, slash/skills) and pick the right autonomy? |
|
|
66
69
|
| **I** | Outcome & Efficiency | 5 | Did the session reach a correct, working end state with economy proportional to scope? |
|
|
67
70
|
|
|
68
|
-
Each dimension gets an integer **0–5 band**, a one-paragraph **rationale**, and
|
|
71
|
+
Each dimension gets an integer **0–5 band**, a one-paragraph **rationale**, and **at least 2 quoted turn citations** (`U7: "i see the rows but they are empty"`). No citation, no score: write `INSUFFICIENT_EVIDENCE` and exclude the dimension from the total. Mark a genuinely inapplicable dimension `N/A` and renormalize over the remaining weight.
|
|
69
72
|
|
|
70
73
|
### 4. Apply gates (hard caps), then green flags
|
|
71
74
|
|
|
72
|
-
|
|
75
|
+
Gates — `SECRETS-LEAKED`, `DESTRUCTIVE-UNCHECKED`, `BLIND-SHIP`, `PROMPT-INJECTION-NAIVE` — cap the overall level regardless of dimension scores (`rubric.md` → "Gates"). They fire **only** on confirmed unsafe steering: treat each auto-detected red flag as a candidate, confirm it against the transcript, and record why before firing.
|
|
73
76
|
|
|
74
|
-
|
|
77
|
+
Green flags are evidence of strength and **never** push a dimension above its band's ceiling.
|
|
75
78
|
|
|
76
79
|
### 5. Compute totals and level
|
|
77
80
|
|
|
@@ -80,7 +83,7 @@ weighted = Σ(band_d / 5 × weight_d) over scored dimensions
|
|
|
80
83
|
normalized = weighted / Σ(weight_d of scored dimensions) × 100
|
|
81
84
|
```
|
|
82
85
|
|
|
83
|
-
Map `normalized` to a **steering level
|
|
86
|
+
Map `normalized` to a **steering level**, then apply the gate caps:
|
|
84
87
|
|
|
85
88
|
| Level | Normalized | Reads as |
|
|
86
89
|
| --- | :-: | --- |
|
|
@@ -90,30 +93,22 @@ Map `normalized` to a **steering level** (after gate caps):
|
|
|
90
93
|
| **Developing** | 40–54 | Works, but reactive/under-specified/under-verified; junior steering |
|
|
91
94
|
| **Weak** | 0–39 | Ineffective or unsafe steering |
|
|
92
95
|
|
|
93
|
-
The level is a **calibration aid, not a verdict** — the human reviewer decides hire
|
|
96
|
+
The level is a **calibration aid, not a verdict** — the human reviewer decides hire or no-hire. Present it with the evidence that earned it, and name the 1–2 dimensions that most moved it.
|
|
94
97
|
|
|
95
98
|
### 6. Write outputs (see [`TEMPLATE.md`](TEMPLATE.md))
|
|
96
99
|
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
- **`report.md`** — narrative: summary line, level, per-dimension table with citations, standout moments, weaknesses, gates/flags, and a "what a stronger operator would have done differently" section. Shape = TEMPLATE.md.
|
|
100
|
-
- **`scorecard.yaml`** — machine-readable: per-dimension band + rationale + citations, gates, green flags, totals, level.
|
|
101
|
-
|
|
102
|
-
When >1 session, also write `<out_dir>/candidate-rollup.md`:
|
|
100
|
+
Fill the shapes in `TEMPLATE.md` exactly:
|
|
103
101
|
|
|
104
|
-
- Per
|
|
105
|
-
-
|
|
106
|
-
- **Consistency** read — does the candidate steer the same way under pressure (debugging) as when greenfield?
|
|
107
|
-
- A single calibrated **overall steering level** with the 3 strongest and 3 weakest pieces of evidence across all sessions, and concrete interview follow-up questions to probe the gaps.
|
|
102
|
+
- Per session, in `<out_dir>/<session-slug>/`: `report.md` (the narrative) and `scorecard.yaml` (the machine-readable twin).
|
|
103
|
+
- When there is more than one session: `<out_dir>/candidate-rollup.md` — per-session levels, per-dimension trends, a consistency read (does the candidate steer the same way under debugging pressure as on greenfield?), and one calibrated overall level with the strongest and weakest evidence and interview follow-ups.
|
|
108
104
|
|
|
109
105
|
## Scoring discipline (read before every run)
|
|
110
106
|
|
|
111
107
|
- **Evidence or it didn't happen.** Every band cites real turns. "Seems thorough" is not a finding; `U7–U11 debugged the empty-rows bug by pasting the live API response (U9) and isolating restart behavior (U11)` is.
|
|
112
|
-
- **Absolute bands, not curve.**
|
|
113
|
-
- **
|
|
114
|
-
- **
|
|
115
|
-
- **
|
|
116
|
-
- **Surface the model's contribution honestly.** If Claude proposed the good architecture and the candidate merely said "yes", that is weak **G** with strong-agent luck — say so. If the candidate's prompt _specified_ the operational concern, that is strong **G**.
|
|
108
|
+
- **Absolute bands, not a curve.** Score each session against the anchors in `rubric.md`, never against other candidates. Across several sessions you may _report_ relative trends; each band stays absolute.
|
|
109
|
+
- **Ask what each turn accomplished.** Penalize wasted loops; reward turns that each move a verifiable step.
|
|
110
|
+
- **Charity with a spine.** Read terse prompts in the best plausible light (typos and brevity are not low skill), but **never** credit verification or judgment the transcript doesn't show.
|
|
111
|
+
- **Surface the model's contribution honestly.** Claude proposing the good architecture and the candidate saying "yes" is weak **G** with strong-agent luck — say so. The candidate's prompt _specifying_ the operational concern is strong **G**.
|
|
117
112
|
|
|
118
113
|
## What good and weak steering look like (calibration vignettes)
|
|
119
114
|
|
|
@@ -125,6 +120,6 @@ When >1 session, also write `<out_dir>/candidate-rollup.md`:
|
|
|
125
120
|
|
|
126
121
|
## Notes
|
|
127
122
|
|
|
128
|
-
-
|
|
129
|
-
-
|
|
130
|
-
-
|
|
123
|
+
- **Never mutate the source `.jsonl`** — the skill reads transcripts and writes **only** into `out_dir`.
|
|
124
|
+
- A corrupt or empty transcript, or one with zero human turns, gets a stub report saying so — **never** a fabricated score.
|
|
125
|
+
- The evidence bar: the candidate, shown the report, would recognize the session.
|
|
@@ -1,8 +1,6 @@
|
|
|
1
1
|
# interview-eval rubric — band anchors
|
|
2
2
|
|
|
3
|
-
Score each dimension **0–5** against the anchors below
|
|
4
|
-
|
|
5
|
-
Read the whole file before scoring. The general meaning of the 0–5 scale:
|
|
3
|
+
Score each dimension **0–5** against the anchors below; § Session-type matrix says which dimensions may be `N/A`. The general meaning of the scale:
|
|
6
4
|
|
|
7
5
|
| Band | Meaning |
|
|
8
6
|
| :-: | --- |
|
|
@@ -13,7 +11,7 @@ Read the whole file before scoring. The general meaning of the 0–5 scale:
|
|
|
13
11
|
| **1** | Weak. Counterproductive or near-absent. |
|
|
14
12
|
| **0** | None / actively harmful. |
|
|
15
13
|
|
|
16
|
-
|
|
14
|
+
Between two bands, pick the lower unless a green flag justifies the higher. **Never** use half-points.
|
|
17
15
|
|
|
18
16
|
---
|
|
19
17
|
|
|
@@ -70,11 +68,11 @@ _Did the human ground the agent in the real artifacts it needed instead of makin
|
|
|
70
68
|
|
|
71
69
|
## D — Verification & Skepticism (weight 18 — the strongest senior signal)
|
|
72
70
|
|
|
73
|
-
_Did the human treat the agent's output as a draft to be checked — running it, reading it, probing it — rather than rubber-stamping?_
|
|
71
|
+
_Did the human treat the agent's output as a draft to be checked — running it, reading it, probing it — rather than rubber-stamping?_
|
|
74
72
|
|
|
75
73
|
**Look for:** running the app/tests and reporting results; reading diffs and pushing back on questionable choices; inspecting actual behavior/output; catching subtle or superficial work the agent slipped in; confirming a fix actually fixed it before moving on; not declaring "done" on faith.
|
|
76
74
|
|
|
77
|
-
> Permission mode is **not** the signal. `acceptEdits` / bypass mode is a legitimate pragmatic choice — judge whether the human _verified behavior_
|
|
75
|
+
> Permission mode is **not** the signal. `acceptEdits` / bypass mode is a legitimate pragmatic choice — judge whether the human _verified behavior_ by any means (running, testing, reading output, inspecting the UI), not whether they approved each edit.
|
|
78
76
|
|
|
79
77
|
- **5** — Systematic verification throughout: runs the thing, inspects real output, catches agent errors the agent missed, confirms each fix before advancing, and pushes back on shaky choices. Skepticism is the default stance.
|
|
80
78
|
- **4** — Verifies most consequential output and catches real issues; a few steps taken on faith.
|
|
@@ -85,7 +83,7 @@ _Did the human treat the agent's output as a draft to be checked — running it,
|
|
|
85
83
|
|
|
86
84
|
**Green flags:** "restarting the server still shows old rows" (confirming a fix _didn't_ take); reading the actual response body rather than the diff; spotting that the agent stubbed/faked something. **Red flags:** "looks good" with no evidence; shipping immediately after a failed run; never once running the code in a build session.
|
|
87
85
|
|
|
88
|
-
**Gate interaction:**
|
|
86
|
+
**Gate interaction:** the human _shown_ broken output who ships or commits it anyway with no remediation fires `BLIND-SHIP` (§ Gates) — **D ≤ 2** and the overall level ≤ **Competent**.
|
|
89
87
|
|
|
90
88
|
---
|
|
91
89
|
|
|
@@ -93,7 +91,7 @@ _Did the human treat the agent's output as a draft to be checked — running it,
|
|
|
93
91
|
|
|
94
92
|
_When something broke, did the human drive toward root cause with high-signal diagnostics, or just nudge vaguely?_
|
|
95
93
|
|
|
96
|
-
**Look for:** precise reproduction; exact error/output pasted; expected-vs-actual framing; stated hypotheses; isolating variables one at a time; converging instead of flailing.
|
|
94
|
+
**Look for:** precise reproduction; exact error/output pasted; expected-vs-actual framing; stated hypotheses; isolating variables one at a time; converging instead of flailing. A pure debugging session makes this the main event; a clean build may make it minor or `N/A`.
|
|
97
95
|
|
|
98
96
|
- **5** — Drives debugging like an engineer: repro + verbatim evidence + hypothesis, isolates one variable at a time, narrows steadily to root cause, confirms the fix. Tight convergence.
|
|
99
97
|
- **4** — Good diagnostics and mostly systematic; occasional vague nudge but recovers.
|
|
@@ -110,7 +108,7 @@ _When something broke, did the human drive toward root cause with high-signal di
|
|
|
110
108
|
|
|
111
109
|
_Did the human catch the agent drifting and redirect surgically — and know when to refine vs. undo vs. restart?_
|
|
112
110
|
|
|
113
|
-
**Look for:** early detection of off-track work; minimal, targeted redirects that preserve good work; tightening constraints constructively ("run 10 at a time, spend 1s per task"); well-timed interrupts; choosing the cheapest correction (refine > undo > restart).
|
|
111
|
+
**Look for:** early detection of off-track work; minimal, targeted redirects that preserve good work; tightening constraints constructively ("run 10 at a time, spend 1s per task"); well-timed interrupts; choosing the cheapest correction (refine > undo > restart). **Against:** letting a wrong path run for many turns; chaotic yanking; re-sending an identical failing prompt; nuking good work to fix a small issue.
|
|
114
112
|
|
|
115
113
|
- **5** — Catches drift fast and corrects surgically; tightens scope constructively; interrupts are well-timed; always picks the cheapest effective correction.
|
|
116
114
|
- **4** — Generally redirects well and early; an occasional late catch or slightly heavy-handed correction.
|
|
@@ -125,11 +123,11 @@ _Did the human catch the agent drifting and redirect surgically — and know whe
|
|
|
125
123
|
|
|
126
124
|
## G — Technical & Architectural Judgment (weight 15)
|
|
127
125
|
|
|
128
|
-
_Do the prompts themselves reveal engineering maturity?_
|
|
126
|
+
_Do the prompts themselves reveal engineering maturity?_ Read the **content** of the human's instructions, independent of whether the agent executed well.
|
|
129
127
|
|
|
130
128
|
**Look for:** anticipating operational concerns unprompted (idempotency, reconciliation of stuck state, observability/logging, backpressure, failure/retry, race conditions, data consistency); sound trade-offs; right-sizing to context (descoping auth for a demo; not gold-plating); awareness of the system beyond the happy path; sensible tech choices for the stated constraints.
|
|
131
129
|
|
|
132
|
-
>
|
|
130
|
+
> Note who originated the judgment: _Claude_ proposing the mature design and the human assenting is weaker **G** than the human's prompt _specifying_ it.
|
|
133
131
|
|
|
134
132
|
- **5** — Prompts repeatedly surface senior-level concerns the agent wouldn't have without being told — operability, failure modes, consistency — and make sound, explicit trade-offs right-sized to the context.
|
|
135
133
|
- **4** — Clear engineering maturity in most prompts; raises real operational concerns; a few missed opportunities.
|
|
@@ -144,7 +142,7 @@ _Do the prompts themselves reveal engineering maturity?_ This reads the **conten
|
|
|
144
142
|
|
|
145
143
|
## H — Workflow & Tooling Leverage (weight 5)
|
|
146
144
|
|
|
147
|
-
_Did the human use the agent and harness well, and pick the right level of autonomy?_ Credit
|
|
145
|
+
_Did the human use the agent and harness well, and pick the right level of autonomy?_ Credit leverage where it applies; **never** penalize a small, well-handled task for not using advanced features — score it around 3.
|
|
148
146
|
|
|
149
147
|
**Look for:** plan mode for genuinely ambiguous/large work; subagents for parallel or large-scope fan-out; slash commands / skills; sane git hygiene (branch, review, meaningful commits); MCP/CI where useful; matching autonomy to risk (high autonomy on safe mechanical work, tighter control on risky/irreversible work); economy of interaction.
|
|
150
148
|
|
|
@@ -155,13 +153,13 @@ _Did the human use the agent and harness well, and pick the right level of auton
|
|
|
155
153
|
- **1** — Fights the tool; ignores obvious leverage; risky autonomy with no guardrails.
|
|
156
154
|
- **0** — Tooling actively works against the goal.
|
|
157
155
|
|
|
158
|
-
**Green flags:** plan mode before a big ambiguous build; subagents for an audit sweep; a clean commit/push flow with a written README. **Red flags:** bypass-all-permissions on destructive ops; no version control on substantial work.
|
|
156
|
+
**Green flags:** plan mode before a big ambiguous build; subagents for an audit sweep; a clean commit/push flow with a written README. **Red flags:** bypass-all-permissions on destructive ops; no version control on substantial work.
|
|
159
157
|
|
|
160
158
|
---
|
|
161
159
|
|
|
162
160
|
## I — Outcome & Efficiency (weight 5)
|
|
163
161
|
|
|
164
|
-
_Did the session reach a correct, working end state, with economy proportional to scope?_ A capstone
|
|
162
|
+
_Did the session reach a correct, working end state, with economy proportional to scope?_ A capstone, partly downstream of the other dimensions.
|
|
165
163
|
|
|
166
164
|
**Look for:** a working, correct result the human verified; effort (turns/time/tokens) proportional to the scope; little wasted looping; a clean stopping point (committed/pushed/documented).
|
|
167
165
|
|
|
@@ -172,13 +170,13 @@ _Did the session reach a correct, working end state, with economy proportional t
|
|
|
172
170
|
- **1** — Little working outcome; heavy waste.
|
|
173
171
|
- **0** — No working outcome.
|
|
174
172
|
|
|
175
|
-
**
|
|
173
|
+
**Scope-adjust:** a small task achieved crisply is a 5; a huge one half-finished with good reason may still be a 3–4. **Never** reward raw volume.
|
|
176
174
|
|
|
177
175
|
---
|
|
178
176
|
|
|
179
177
|
## Gates (hard caps on the overall level)
|
|
180
178
|
|
|
181
|
-
Gates fire only on
|
|
179
|
+
Gates fire **only** on behavior confirmed in the transcript — auto red flags are leads. When a gate fires, apply its cap, record the gate, and quote the turn. Multiple gates stack downward.
|
|
182
180
|
|
|
183
181
|
| Gate | Fires when | Cap |
|
|
184
182
|
| --- | --- | --- |
|
|
@@ -187,7 +185,7 @@ Gates fire only on **confirmed** behavior in the transcript (use auto red-flags
|
|
|
187
185
|
| **BLIND-SHIP** | The human is _shown_ output is broken (failed run, wrong result) and ships/commits/declares-done anyway, with no remediation. | **D ≤ 2** and overall ≤ **Competent** |
|
|
188
186
|
| **PROMPT-INJECTION-NAIVE** | The human pipes untrusted external content into the agent as instructions and acts on it without scrutiny in a context where that's dangerous. | ≤ **Competent** |
|
|
189
187
|
|
|
190
|
-
A gate is about _unsafe steering_, not mere mistakes
|
|
188
|
+
A gate is about _unsafe steering_, not mere mistakes: a bug the candidate then caught is **not** a gate — it is evidence _for_ D. Gates **must** have human direction or human ratification behind them; "the harness or model did X" never fires one.
|
|
191
189
|
|
|
192
190
|
---
|
|
193
191
|
|
|
@@ -204,16 +202,16 @@ A gate is about _unsafe steering_, not mere mistakes. A bug the candidate then c
|
|
|
204
202
|
|
|
205
203
|
## Session-type matrix — which dimensions are primary / secondary / N-A
|
|
206
204
|
|
|
207
|
-
Classify the dominant type
|
|
205
|
+
Classify the dominant type and note a blend. **Primary** = central to the read; expect strong evidence (the weights stay fixed). **Secondary** = score if present. **Often N/A** = exclude and renormalize, unless the session actually exercises it. The bold type is the `session_type` token in `scorecard.yaml`.
|
|
208
206
|
|
|
209
207
|
| Type | Primary | Secondary | Often N/A |
|
|
210
208
|
| --- | --- | --- | --- |
|
|
211
209
|
| **greenfield-build** | A, B, G, D | C, F, I | — |
|
|
212
210
|
| **feature-add** | A, D, G | B, C, F, I | — |
|
|
213
211
|
| **debugging** | E, D, C | F, G | B (little to decompose) |
|
|
214
|
-
| **refactor
|
|
215
|
-
| **ops
|
|
216
|
-
| **research
|
|
217
|
-
| **review
|
|
212
|
+
| **refactor** (cleanup) | D, G, B | C, F, I | E |
|
|
213
|
+
| **ops** (infra) | A, D, G, H | C, F, I | B, E |
|
|
214
|
+
| **research** (Q&A) | A, C, F | G, I | B, D (no code to verify), E, H |
|
|
215
|
+
| **review** (audit) | C, G, D, H | A, F | B, E, I |
|
|
218
216
|
|
|
219
|
-
|
|
217
|
+
A user-stated intent ("this was a time-boxed take-home") adjusts expectations — efficiency up, polish down — and **never** the band anchors.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@nurix/etna",
|
|
3
|
-
"version": "0.20.
|
|
3
|
+
"version": "0.20.37",
|
|
4
4
|
"description": "Bootstrap a project with a Nurix harness — writes the committed contract (CLAUDE.md, AGENTS.md, .claude/harness.json, settings.json, the mount hook) and mounts the harness itself, gitignored, from a bundled payload; sessions keep it current from the CDN.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -46,5 +46,5 @@
|
|
|
46
46
|
"agent",
|
|
47
47
|
"plugin"
|
|
48
48
|
],
|
|
49
|
-
"etnaFingerprint": "
|
|
49
|
+
"etnaFingerprint": "899bed2c91e6"
|
|
50
50
|
}
|