@nurix/etna 0.15.6 → 0.15.8

package/data/harnesses/hr/skills/interview-eval/SKILL.md

@@ -0,0 +1,118 @@
+---
+name: interview-eval
+description: Evaluate how well a person steered a Claude Code session — the human's prompting, decomposition, verification, debugging, course-correction, and engineering judgment — from one or more session transcripts (.jsonl). Built for interview/hiring assessment of engineers using AI coding agents. Takes 1+ Claude Code session `.jsonl` files (or a directory of them) plus an output directory; writes a per-session `report.md` + `scorecard.yaml`, and (for multiple sessions) a `candidate-rollup.md`. Scores 9 weighted dimensions on a 0–5 band scale against absolute anchors in `rubric.md`, normalizes to /100, maps to a steering level (Exemplary / Strong / Competent / Developing / Weak), applies hard-cap gates for unsafe behavior, and anchors every score to quoted turns. It judges the HUMAN'S steering, not Claude's code. Triggers on "evaluate this Claude Code session", "score how the candidate steered Claude", "judge these session transcripts", "how good is this person's prompting", "assess AI-pairing skill from a session log", or a `.jsonl` transcript handed over for review.
+---
+
+# interview-eval
+
+Judge **how a human drove a Claude Code session** — not how good Claude's output was. The unit of evaluation is the person's steering: how they framed the problem, decomposed it, fed context, verified results, debugged, corrected course, and revealed engineering judgment through their prompts. One or many session transcripts in → per-session reports + scorecards out, plus a candidate-level rollup when there are several.
+
+This is the **counterpart** to the sibling `evaluator` skill: `evaluator` scores a *repo's artifacts*; `interview-eval` scores the *conversation that produced them*. They share no rubric.
+
+## The one rule that governs every score
+
+**You are scoring the human, not the model.** Claude's code being clean does not earn the candidate points; Claude's code being buggy does not lose them points — *unless* the human's prompt caused it or the human failed to catch it. Reward the prompt that set Claude up to succeed, the verification that caught Claude's mistake, the redirect that pulled Claude off a bad path. A candidate who wrote three sharp prompts and let a capable agent run is steering *well*, even if the turn count is low. A candidate who typed forty vague nudges and rubber-stamped broken output is steering *badly*, even if something shipped. Turn count, prompt length, and verbosity are **not** quality signals — precision, judgment, and verification are.
+
+## Input
+
+- **One or more session transcripts** — Claude Code `.jsonl` files (e.g. `0fe33297-….jsonl`). Accept a list of files, a glob, or a directory (evaluate every `*.jsonl` inside).
+- **An output directory** — `out_dir` (default: `./interview-eval-out/`). One subfolder per session, plus a rollup at the root when >1 session.
+- **Optional context** the user may volunteer: the candidate's seniority/role, the task brief they were given, time limits, or whether sessions are independent tasks vs. one continued task. Fold it in; never invent it.
+
+If the user just points at this folder, treat every `*.jsonl` as one candidate's sessions unless told otherwise.
+
+## Process
+
+### 0. Preflight (cheap)
+Only dependency is `python3` (stdlib only). No installs, no network. If `python3` is missing, say so and stop.
+
+### 1. Extract an evidence pack per session (deterministic)
+Run the extractor — it never scores, it only surfaces signal:
+
+```bash
+python3 <skill_dir>/extract_session.py SESSION.jsonl --out <out_dir>/<session-slug>/ --json
+```
+
+It emits a markdown evidence pack on stdout and writes `<sessionId>.session.json`. The pack gives you:
+- **Turn-by-turn**: each human prompt (IDE tags stripped) followed by a summary of what the agent did (tool counts + sample targets). This is your primary evidence — quote `U<n>` turn numbers in every finding.
+- **Metrics**: prompt count, wall-clock & idle gaps, tool histogram, permission-mode transitions, plan-mode / subagent / AskUserQuestion / slash-command usage, interrupts, system/API errors.
+- **Auto red-flag signals**: secrets pasted into prompts, destructive commands the agent ran (`rm -rf ~`, force-push, `DROP TABLE`, …). These are **leads to verify**, not verdicts.
+
+**Read the full evidence pack before scoring.** For long sessions, also open the raw `.jsonl` around any turn you intend to quote, to confirm context. Do not score from metrics alone — a session can have great metrics and terrible steering, or vice-versa.
+
+**Separate noise from signal.** `system_errors` that are `status: 529` / `ECONNRESET` are Anthropic-side overload, not the candidate's fault — never penalize them. `interrupts` may be a sharp early redirect (good) or thrash (bad) — read the surrounding turns to tell which.
+
+### 2. Classify the session type
+Pick the dominant type (a session can be a blend — note it): **greenfield-build · feature-add · debugging · refactor · ops/infra · research/Q&A · review/audit**. The type sets which dimensions are *primary*, *secondary*, or *N/A* — see `rubric.md` → "Session-type matrix". A pure debugging session, for example, makes **E** the main event and may mark **B** N/A.
+
+### 3. Score the 9 dimensions
+Full band anchors (0–5, with concrete "looks like" descriptions and green/red flags) live in **[`rubric.md`](rubric.md)** — read it before every evaluation; do not score from memory. Summary:
+
+| # | Dimension | Weight | The senior question it answers |
+|---|-----------|:------:|--------------------------------|
+| **A** | Problem Framing & Specification | 15 | Did they tell the agent *what* and *why*, at the right altitude, with constraints + acceptance criteria? |
+| **B** | Decomposition & Sequencing | 12 | Did they break work into coherent, dependency-ordered, verifiable increments? |
+| **C** | Context Curation | 10 | Did they ground the agent in real artifacts (files, logs, outputs) instead of making it guess? |
+| **D** | Verification & Skepticism | 18 | Did they treat output as a draft to check — run it, read it, catch errors — rather than rubber-stamp? |
+| **E** | Debugging Collaboration | 10 | When it broke, did they give high-signal diagnostics (repro, exact error, expected-vs-actual, hypothesis)? |
+| **F** | Course Correction & Steering Control | 10 | Did they catch drift early and redirect surgically, instead of letting bad paths run or thrashing? |
+| **G** | Technical & Architectural Judgment | 15 | Do the prompts reveal engineering maturity — operability, failure modes, trade-offs, right-sizing? |
+| **H** | Workflow & Tooling Leverage | 5 | Did they use the harness well (plan mode, subagents, git, slash/skills) and pick the right autonomy? |
+| **I** | Outcome & Efficiency | 5 | Did the session reach a correct, working end state with economy proportional to scope? |
+
+Each dimension gets an integer **0–5 band**, a one-paragraph **rationale**, and **≥2 quoted turn citations** (`U7: "i see the rows but they are empty"`). No citation → no score; write `INSUFFICIENT_EVIDENCE` and exclude from the total. Mark genuinely inapplicable dimensions `N/A` and renormalize over the remaining weight.
+
+### 4. Apply gates (hard caps), then green flags
+**Gates** cap the overall band regardless of dimension scores — see `rubric.md` → "Gates". They fire only on *confirmed* unsafe steering (pasted live secrets, approved a destructive/irreversible op without checking, knowingly shipped broken output). Auto-detected red flags are candidates for a gate — confirm against the transcript before firing one, and record why.
+
+**Green flags** are noted as evidence of strength but never push a dimension above its band's ceiling.
+
+### 5. Compute totals and level
+```
+weighted = Σ(band_d / 5 × weight_d)  over scored dimensions
+normalized = weighted / Σ(weight_d of scored dimensions) × 100
+```
+Map `normalized` to a **steering level** (after gate caps):
+
+| Level | Normalized | Reads as |
+|-------|:----------:|----------|
+| **Exemplary** | 85–100 | Operates the agent as a true force-multiplier; senior+ signal |
+| **Strong** | 70–84 | Confident, effective, low-waste steering; clear senior signal |
+| **Competent** | 55–69 | Gets results; real gaps in verification or framing; mid-level |
+| **Developing** | 40–54 | Works, but reactive/under-specified/under-verified; junior steering |
+| **Weak** | 0–39 | Ineffective or unsafe steering |
+
+The level is a **calibration aid, not a verdict** — the human reviewer decides hire/no-hire. Present the level with the evidence that earned it, and call out the 1–2 dimensions that most moved it.
+
+### 6. Write outputs (see [`TEMPLATE.md`](TEMPLATE.md))
+Per session, write to `<out_dir>/<session-slug>/`:
+- **`report.md`** — narrative: summary line, level, per-dimension table with citations, standout moments, weaknesses, gates/flags, and a "what a stronger operator would have done differently" section. Shape = TEMPLATE.md.
+- **`scorecard.yaml`** — machine-readable: per-dimension band + rationale + citations, gates, green flags, totals, level.
+
+When >1 session, also write `<out_dir>/candidate-rollup.md`:
+- Per-session level + headline.
+- Per-dimension trend across sessions (is verification consistently weak? is framing consistently strong?).
+- **Consistency** read — does the candidate steer the same way under pressure (debugging) as when greenfield?
+- A single calibrated **overall steering level** with the 3 strongest and 3 weakest pieces of evidence across all sessions, and concrete interview follow-up questions to probe the gaps.
+
+## Scoring discipline (read before every run)
+
+- **Evidence or it didn't happen.** Every band cites real turns. "Seems thorough" is not a finding; `U7–U11 debugged the empty-rows bug by pasting the live API response (U9) and isolating restart behavior (U11)` is.
+- **Absolute bands, not curve.** A lone session is scored against the anchors in `rubric.md`, not against other candidates. With multiple sessions you may *report* relative trends, but each session's band is absolute.
+- **Don't confuse activity with skill.** A 40-turn session is not 40-turns-good. Ask what each turn *accomplished*. Penalize wasted loops; reward turns that each move a verifiable step.
+- **Don't penalize the harness.** API 529s, tool-permission prompts, and model mistakes the candidate then caught are not the candidate's failures. Idle gaps may be the candidate thinking, testing out-of-band, or away — note, don't assume.
+- **Charity with a spine.** Read terse prompts in the best plausible light (typos, brevity ≠ low skill), but do not invent verification or judgment the transcript doesn't show. If you can't see it, it isn't scored there.
+- **Surface the model's contribution honestly.** If Claude proposed the good architecture and the candidate merely said "yes", that is weak **G** with strong-agent luck — say so. If the candidate's prompt *specified* the operational concern, that is strong **G**.
+
+## What good and weak steering look like (calibration vignettes)
+
+- **Strong A+G+D:** Opens with a data model, the four API operations, and an explicit constraint ("use an in-memory DB"). Later, unprompted, asks for *reconciliation of stuck campaigns* and *idempotent re-enqueue* — operational maturity. Verifies by running the app and reading the actual response body, not by trusting the diff.
+- **Strong E:** On a bug, pastes the literal API response (`U9`), then narrows it — "I see the rows but they are empty" → "restarting the server still shows old rows" — feeding the agent expected-vs-actual until root cause. Converges in a few turns.
+- **Weak D:** Runs the whole session in an auto-accept mode and never once runs, reads, or questions the output; "make it work" → "still not working" → "ok ship it." No evidence the human ever confirmed correctness.
+- **Weak F:** Lets the agent build for many turns down a path the candidate didn't want, then restarts from scratch instead of redirecting; or re-sends the identical prompt three times when it fails instead of adding information.
+- **Gate fires:** Pastes a live API key into a prompt, or tells the agent to `git push --force` to a shared branch and approves it without looking — cap at **Developing** and flag, no matter how good the rest is.
+
+## Notes
+- The skill is read-only over the transcripts; it only writes into `out_dir`. Never mutate the source `.jsonl`.
+- If a transcript is corrupt/empty or has zero human turns, write a stub report saying so rather than fabricating a score.
+- Keep reports honest and specific enough that the candidate, if shown the report, would recognize the session. That is the bar for evidence quality.

package/data/harnesses/hr/skills/interview-eval/TEMPLATE.md

@@ -0,0 +1,124 @@
+# interview-eval output templates
+
+Two per-session artifacts (`report.md`, `scorecard.yaml`) and one rollup (`candidate-rollup.md`). Fill every `‹…›`. Keep citations to real `U<n>` turns with short verbatim quotes.
+
+---
+
+## `report.md` (per session)
+
+```markdown
+# Steering evaluation — ‹session title›
+`‹session-id›` · ‹session-type› · ‹N› human turns · ‹wall-clock› · cli ‹version›
+
+> **Steering level: ‹Exemplary | Strong | Competent | Developing | Weak›**  (‹normalized›/100)
+> ‹One-sentence verdict — what kind of operator this session shows, in plain language.›
+
+**Headline:** ‹2–3 sentences. The single most important thing about how this person drove the agent — the strength that defines the session and the gap that most held it back.›
+
+## Scores
+
+| Dim | Dimension | Band | Wt | Note |
+|-----|-----------|:----:|:--:|------|
+| A | Problem Framing & Specification | ‹0–5› | 15 | ‹≤10 words› |
+| B | Decomposition & Sequencing | ‹0–5› | 12 | ‹≤10 words› |
+| C | Context Curation | ‹0–5› | 10 | ‹≤10 words› |
+| D | Verification & Skepticism | ‹0–5› | 18 | ‹≤10 words› |
+| E | Debugging Collaboration | ‹0–5 / N/A› | 10 | ‹≤10 words› |
+| F | Course Correction & Steering | ‹0–5› | 10 | ‹≤10 words› |
+| G | Technical & Architectural Judgment | ‹0–5› | 15 | ‹≤10 words› |
+| H | Workflow & Tooling Leverage | ‹0–5› | 5 | ‹≤10 words› |
+| I | Outcome & Efficiency | ‹0–5› | 5 | ‹≤10 words› |
+| | **Weighted total** | | | **‹normalized›/100** |
+
+‹If any gate fired:› **Gate: ‹GATE-NAME›** — ‹what triggered it, with the turn cite›. Level capped at ‹cap›.
+
+## Per-dimension findings
+For each dimension, 2–4 sentences of rationale + the turn citations that justify the band.
+
+**A — Problem Framing (‹band›).** ‹rationale› _Evidence:_ ‹U1: "…"›, ‹U4: "…"›.
+
+**B — Decomposition (‹band›).** ‹rationale› _Evidence:_ ‹…›.
+
+‹…repeat C–I…›
+
+## Standout moments
+- ✅ ‹The 1–3 best steering moves, each with a turn cite and why it was good.›
+
+## Weaknesses
+- ⚠ ‹The 1–3 most costly gaps, each with a turn cite and the consequence.›
+
+## What a stronger operator would have done differently
+‹3–5 concrete, specific alternatives — the actual better prompt or move at a named turn. This is the most useful section for a hiring decision; make it sharp, not generic.›
+
+## Caveats
+‹Harness noise excluded from scoring (e.g. "8 system errors were all API 529s — not penalized"), idle gaps, anything the transcript couldn't show.›
+```
+
+---
+
+## `scorecard.yaml` (per session, machine-readable)
+
+```yaml
+session_id: ‹id›
+title: ‹ai-title›
+session_type: ‹greenfield-build | feature-add | debugging | refactor | ops | research | review›
+turns: ‹N›
+duration_human: ‹3h42m›
+
+dimensions:
+  framing:        { band: ‹0-5›, weight: 15, rationale: "‹…›", cites: ["U1", "U4"] }
+  decomposition:  { band: ‹0-5›, weight: 12, rationale: "‹…›", cites: ["U2", "U4"] }
+  context:        { band: ‹0-5›, weight: 10, rationale: "‹…›", cites: ["U9"] }
+  verification:   { band: ‹0-5›, weight: 18, rationale: "‹…›", cites: ["U7", "U11"] }
+  debugging:      { band: ‹0-5|null›, weight: 10, na: ‹false|true›, rationale: "‹…›", cites: ["U9"] }
+  correction:     { band: ‹0-5›, weight: 10, rationale: "‹…›", cites: ["U13"] }
+  judgment:       { band: ‹0-5›, weight: 15, rationale: "‹…›", cites: ["U23"] }
+  tooling:        { band: ‹0-5›, weight: 5,  rationale: "‹…›", cites: ["U17"] }
+  outcome:        { band: ‹0-5›, weight: 5,  rationale: "‹…›", cites: ["U26"] }
+
+gates_triggered: []          # e.g. [BLIND-SHIP]
+green_flags: []              # e.g. [empirical-debugging, unprompted-operability]
+
+weighted_total: ‹float›      # Σ(band/5 × weight) over scored dims
+available_weight: ‹int›      # Σ weight of scored (non-N/A) dims
+normalized: ‹float›          # weighted_total / available_weight × 100, after gate caps
+level: ‹Exemplary | Strong | Competent | Developing | Weak›
+
+evidence_excluded_from_scoring:
+  - "‹e.g. 8 API 529 errors — Anthropic-side overload, not candidate fault›"
+```
+
+---
+
+## `candidate-rollup.md` (only when >1 session)
+
+```markdown
+# Candidate steering rollup — ‹candidate / label›
+‹K› sessions · ‹total turns› · ‹combined wall-clock›
+
+> **Overall steering level: ‹level›**
+> ‹One-paragraph calibrated read of the candidate as an AI-pairing operator across all sessions.›
+
+## Sessions
+| Session | Type | Level | Norm. | Headline |
+|---------|------|-------|:-----:|----------|
+| ‹title› | ‹type› | ‹level› | ‹n› | ‹≤12 words› |
+
+## Dimension trend across sessions
+| Dim | S1 | S2 | … | Read |
+|-----|:--:|:--:|---|------|
+| A Framing | ‹b› | ‹b› | | ‹consistent? improving? context-dependent?› |
+| … | | | | |
+
+## Consistency
+‹Does the candidate steer the same under pressure (debugging) as greenfield? Where does quality drop — and is the drop in a dimension that matters for the role?›
+
+## Strongest evidence (top 3 across all sessions)
+1. ‹session · U‹n› · what it shows›
+
+## Weakest evidence (top 3 across all sessions)
+1. ‹session · U‹n› · what it shows›
+
+## Interview follow-ups to probe the gaps
+- ‹Concrete question that would confirm/deny the weakest dimension, e.g. "Walk me through how you'd have verified the worker actually drained the queue — what would you have checked?"›
+```

package/data/harnesses/hr/skills/interview-eval/extract_session.py

@@ -0,0 +1,418 @@
+#!/usr/bin/env python3
+"""
+extract_session.py — deterministic extractor for interview-eval.
+
+Turns a Claude Code session transcript (.jsonl) into a normalized "evidence pack"
+so the LLM judge scores from clean, quoted evidence instead of re-parsing raw JSONL.
+
+Usage:
+    python3 extract_session.py SESSION.jsonl [SESSION2.jsonl ...] [--json] [--out DIR]
+
+Default output: a human/LLM-readable markdown evidence pack per session on stdout.
+--json:   also emit the normalized structure as <out>/<sessionId>.session.json
+--out DIR: write artifacts to DIR (default: alongside the transcript)
+
+The extractor is purely mechanical. It NEVER scores. It surfaces signals
+(prompts, tool usage, modes, errors, auto-detected red flags) for the judge.
+"""
+import json
+import re
+import sys
+import os
+from collections import Counter, OrderedDict
+from datetime import datetime
+
+# ---------- transcript record helpers ----------
+
+IDE_TAG = re.compile(r"<ide_[a-z_]+>.*?</ide_[a-z_]+>", re.DOTALL)
+CMD_NAME = re.compile(r"<command-name>(.*?)</command-name>", re.DOTALL)
+CMD_ARGS = re.compile(r"<command-args>(.*?)</command-args>", re.DOTALL)
+
+# Auto-detected red-flag patterns. These are SIGNALS for the judge, not verdicts.
+SECRET_PATTERNS = [
+    (re.compile(r"sk-[A-Za-z0-9]{20,}"), "openai-style key"),
+    (re.compile(r"ghp_[A-Za-z0-9]{30,}"), "github PAT"),
+    (re.compile(r"AKIA[0-9A-Z]{16}"), "aws access key id"),
+    (re.compile(r"-----BEGIN (?:RSA |EC )?PRIVATE KEY-----"), "private key"),
+    (re.compile(r"(?i)\b(password|passwd|secret|api[_-]?key|token)\b\s*[:=]\s*['\"][^'\"]{6,}"), "inline secret"),
+]
+DANGER_CMD = [
+    (re.compile(r"\brm\s+-rf?\s+[/~]"), "rm -rf on root/home path"),
+    (re.compile(r"\bgit\s+push\b.*--force\b|\bgit\s+push\b.*\s-f\b"), "force push"),
+    (re.compile(r"\bgit\s+reset\s+--hard\b"), "hard reset"),
+    (re.compile(r"\bDROP\s+(TABLE|DATABASE)\b", re.I), "drop table/database"),
+    (re.compile(r"\bchmod\s+-R\s+777\b"), "chmod 777 -R"),
+    (re.compile(r":\s*\(\)\s*\{\s*:\|:&\s*\}\s*;:"), "fork bomb"),
+]
+SLASH_CMD = re.compile(r"^\s*/([A-Za-z][\w:-]*)")
+
+
+def load_records(path):
+    recs = []
+    with open(path, "r", errors="replace") as f:
+        for ln, line in enumerate(f, 1):
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                recs.append(json.loads(line))
+            except json.JSONDecodeError:
+                recs.append({"type": "_PARSE_ERROR", "_line": ln})
+    return recs
+
+
+def msg_role(rec):
+    return rec.get("message", {}).get("role")
+
+
+def extract_text(content):
+    """Return concatenated text blocks from a message content (str or list)."""
+    if isinstance(content, str):
+        return content
+    if isinstance(content, list):
+        parts = [b.get("text", "") for b in content
+                 if isinstance(b, dict) and b.get("type") == "text"]
+        return "\n".join(p for p in parts if p)
+    return ""
+
+
+def is_tool_result(content):
+    return isinstance(content, list) and any(
+        isinstance(b, dict) and b.get("type") == "tool_result" for b in content)
+
+
+def tool_uses(content):
+    out = []
+    if isinstance(content, list):
+        for b in content:
+            if isinstance(b, dict) and b.get("type") == "tool_use":
+                out.append((b.get("name", "?"), b.get("input", {}) or {}))
+    return out
+
+
+def tool_target(name, inp):
+    """One-line human-readable summary of a tool call's target."""
+    if not isinstance(inp, dict):
+        return ""
+    for k in ("file_path", "path", "pattern", "command", "url", "query",
+              "prompt", "skill", "notebook_path", "description"):
+        if k in inp and inp[k]:
+            v = str(inp[k]).replace("\n", " ")
+            return (v[:140] + "…") if len(v) > 140 else v
+    return ""
+
+
+def parse_ts(s):
+    if not s:
+        return None
+    try:
+        return datetime.fromisoformat(s.replace("Z", "+00:00"))
+    except Exception:
+        return None
+
+
+def clean_prompt(text):
+    """Strip IDE context tags; flag slash commands and local-command stdout."""
+    stripped = IDE_TAG.sub("", text).strip()
+    return stripped
+
+
+# ---------- core normalization ----------
+
+def normalize(path):
+    recs = load_records(path)
+    sid = None
+    ai_title = None
+    version = None
+    cwd = None
+    branch = None
+
+    turns = []           # ordered user prompts with following assistant actions
+    cur_actions = []     # assistant tool calls / text since last user prompt
+    pending_user = None  # the open user turn we are attaching actions to
+
+    tool_counts = Counter()
+    perm_modes = Counter()
+    perm_transitions = []
+    mode_records = []
+    interrupts = 0
+    sys_errors = []
+    parse_errors = 0
+    first_ts = None
+    last_ts = None
+    slash_commands = []
+    subagent_calls = []      # Task/Agent tool uses
+    askuser_calls = 0
+    plan_mode_used = False
+
+    redflags = []            # auto-detected; (kind, where, detail)
+    user_prompt_idx = 0
+
+    def close_turn():
+        nonlocal pending_user, cur_actions
+        if pending_user is not None:
+            pending_user["actions"] = cur_actions
+            turns.append(pending_user)
+        cur_actions = []
+        pending_user = None
+
+    for rec in recs:
+        t = rec.get("type")
+        if t == "_PARSE_ERROR":
+            parse_errors += 1
+            continue
+        ts = parse_ts(rec.get("timestamp"))
+        if ts:
+            if first_ts is None:
+                first_ts = ts
+            last_ts = ts
+
+        if t == "mode" or t == "permission-mode":
+            m = rec.get("mode")
+            if m:
+                mode_records.append(m)
+                if m in ("plan", "plan-mode"):
+                    plan_mode_used = True
+
+        if sid is None and rec.get("sessionId"):
+            sid = rec["sessionId"]
+        if rec.get("version"):
+            version = rec["version"]
+        if rec.get("cwd"):
+            cwd = rec["cwd"]
+        if rec.get("gitBranch"):
+            branch = rec["gitBranch"]
+
+        if t == "ai-title" and rec.get("aiTitle"):
+            ai_title = rec["aiTitle"]
+
+        if t == "system" and rec.get("level") == "error":
+            sys_errors.append({
+                "subtype": rec.get("subtype"),
+                "error": (str(rec.get("error", ""))[:200]),
+                "retryAttempt": rec.get("retryAttempt"),
+                "maxRetries": rec.get("maxRetries"),
+                "ts": rec.get("timestamp"),
+            })
+
+        if t == "user":
+            pm = rec.get("permissionMode")
+            if pm:
+                perm_modes[pm] += 1
+                if not perm_transitions or perm_transitions[-1] != pm:
+                    perm_transitions.append(pm)
+                if pm in ("plan", "plan-mode"):
+                    plan_mode_used = True
+            content = rec.get("message", {}).get("content")
+            if msg_role(rec) != "user":
+                continue
+            raw = extract_text(content)
+            # interrupt markers
+            if "interrupted by user" in (raw or ""):
+                interrupts += 1
+            if is_tool_result(content) and not (raw and raw.strip()):
+                continue  # pure tool result, not a human turn
+            if not (raw and raw.strip()):
+                continue
+            text = clean_prompt(raw)
+            if not text:
+                continue
+            # close prior turn, open a new one
+            close_turn()
+            user_prompt_idx += 1
+            # detect slash command
+            sm = SLASH_CMD.match(text)
+            if sm:
+                slash_commands.append(sm.group(1))
+            cm = CMD_NAME.search(text)
+            if cm:
+                slash_commands.append(cm.group(1).strip().lstrip("/"))
+            # red-flag scan on the human prompt text
+            for pat, label in SECRET_PATTERNS:
+                if pat.search(text):
+                    redflags.append(("secret-in-prompt", f"U{user_prompt_idx}", label))
+            had_ide = bool(IDE_TAG.search(raw))
+            pending_user = {
+                "idx": user_prompt_idx,
+                "ts": rec.get("timestamp"),
+                "text": text,
+                "len": len(text),
+                "had_ide_context": had_ide,
+                "permissionMode": pm,
+            }
+
+        elif t == "assistant":
+            content = rec.get("message", {}).get("content")
+            for name, inp in tool_uses(content):
+                tool_counts[name] += 1
+                tgt = tool_target(name, inp)
+                cur_actions.append({"tool": name, "target": tgt})
+                if name in ("Task", "Agent"):
+                    subagent_calls.append({"target": tgt,
+                                           "subagent": inp.get("subagent_type")})
+                if name == "AskUserQuestion":
+                    askuser_calls += 1
+                # danger scan on bash commands the agent ran
+                if name == "Bash":
+                    cmd = str(inp.get("command", ""))
+                    for pat, label in DANGER_CMD:
+                        if pat.search(cmd):
+                            redflags.append(("danger-cmd", f"after U{user_prompt_idx}", f"{label}: {cmd[:120]}"))
+            atext = extract_text(content)
+            if atext and atext.strip():
+                cur_actions.append({"assistant_text": atext.strip()[:400]})
+
+    close_turn()
+
+    duration_s = None
+    if first_ts and last_ts:
+        duration_s = int((last_ts - first_ts).total_seconds())
+
+    prompt_lens = [t["len"] for t in turns]
+    metrics = {
+        "user_prompts": len(turns),
+        "assistant_tool_calls": sum(tool_counts.values()),
+        "tool_counts": dict(tool_counts.most_common()),
+        "permission_modes": dict(perm_modes),
+        "permission_mode_transitions": perm_transitions,
+        "plan_mode_used": plan_mode_used,
+        "subagent_calls": len(subagent_calls),
+        "askuser_calls": askuser_calls,
+        "slash_commands": slash_commands,
+        "interrupts": interrupts,
+        "system_errors": len(sys_errors),
+        "parse_errors": parse_errors,
+        "duration_seconds": duration_s,
+        "duration_human": _human_dur(duration_s),
+        "avg_prompt_len": round(sum(prompt_lens) / len(prompt_lens), 1) if prompt_lens else 0,
+        "median_prompt_len": _median(prompt_lens),
+    }
+
+    return {
+        "session_id": sid,
+        "title": ai_title,
+        "version": version,
+        "cwd": cwd,
+        "git_branch": branch,
+        "start": first_ts.isoformat() if first_ts else None,
+        "end": last_ts.isoformat() if last_ts else None,
+        "metrics": metrics,
+        "turns": turns,
+        "subagent_calls": subagent_calls,
+        "system_errors": sys_errors,
+        "auto_redflags": redflags,
+        "source_path": os.path.abspath(path),
+    }
+
+
+def _human_dur(s):
+    if s is None:
+        return None
+    h, rem = divmod(s, 3600)
+    m, sec = divmod(rem, 60)
+    if h:
+        return f"{h}h{m}m"
+    if m:
+        return f"{m}m{sec}s"
+    return f"{sec}s"
+
+
+def _median(xs):
+    if not xs:
+        return 0
+    s = sorted(xs)
+    n = len(s)
+    return s[n // 2] if n % 2 else (s[n // 2 - 1] + s[n // 2]) / 2
+
+
+# ---------- rendering ----------
+
+def render_md(norm):
+    m = norm["metrics"]
+    L = []
+    L.append(f"# Evidence pack — session `{norm['session_id']}`")
+    if norm["title"]:
+        L.append(f"**Title:** {norm['title']}")
+    L.append("")
+    L.append("## Session metrics")
+    L.append(f"- User prompts (human turns): **{m['user_prompts']}**")
+    L.append(f"- Wall-clock: **{m['duration_human']}** ({m['duration_seconds']}s)  ·  cwd: `{norm['cwd']}`  ·  branch: `{norm['git_branch']}`  ·  cli: {norm['version']}")
+    L.append(f"- Tool calls: **{m['assistant_tool_calls']}** — {m['tool_counts']}")
+    L.append(f"- Permission modes: {m['permission_modes']}  ·  transitions: {' → '.join(m['permission_mode_transitions']) or '(none)'}")
+    L.append(f"- Plan mode used: **{m['plan_mode_used']}**  ·  Subagents: **{m['subagent_calls']}**  ·  AskUserQuestion: **{m['askuser_calls']}**  ·  Slash cmds: {m['slash_commands'] or '(none)'}")
+    L.append(f"- Interrupts (user stopped a tool): **{m['interrupts']}**  ·  System/API errors: **{m['system_errors']}**")
+    L.append(f"- Prompt length: avg **{m['avg_prompt_len']}** chars, median **{m['median_prompt_len']}**")
+    L.append("")
+    if norm["auto_redflags"]:
+        L.append("## ⚠ Auto-detected red-flag signals (verify before scoring)")
+        for kind, where, detail in norm["auto_redflags"]:
+            L.append(f"- [{kind}] {where}: {detail}")
+        L.append("")
+    if norm["system_errors"]:
+        L.append("## System/API errors (context noise, not always the candidate's fault)")
+        for e in norm["system_errors"][:8]:
+            L.append(f"- {e.get('subtype')} attempt {e.get('retryAttempt')}/{e.get('maxRetries')}: {e.get('error')}")
+        L.append("")
+    L.append("## Turn-by-turn (human prompt → agent actions)")
+    for t in norm["turns"]:
+        ide = "  ·  [had IDE file context]" if t["had_ide_context"] else ""
+        pm = f"  ·  mode={t['permissionMode']}" if t.get("permissionMode") else ""
+        L.append("")
+        L.append(f"### U{t['idx']}  ({(t['ts'] or '')[:19]}{pm}{ide})")
+        L.append("> " + t["text"].replace("\n", "\n> "))
+        acts = t.get("actions", [])
+        tool_line = _summarize_actions(acts)
+        if tool_line:
+            L.append("")
+            L.append(f"_agent did:_ {tool_line}")
+    L.append("")
+    return "\n".join(L)
+
+
+def _summarize_actions(acts):
+    bits = []
+    tc = Counter()
+    for a in acts:
+        if "tool" in a:
+            tc[a["tool"]] += 1
+    for name, n in tc.most_common():
+        bits.append(f"{name}×{n}")
+    # include first few concrete targets for color
+    targets = [a["target"] for a in acts if a.get("tool") in ("Bash", "Write", "Edit", "Read") and a.get("target")][:3]
+    s = ", ".join(bits)
+    if targets:
+        s += "  | e.g. " + " ; ".join(targets)
+    return s
+
+
+def main():
+    args = [a for a in sys.argv[1:] if not a.startswith("--")]
+    want_json = "--json" in sys.argv
+    out_dir = None
+    if "--out" in sys.argv:
+        i = sys.argv.index("--out")
+        if i + 1 < len(sys.argv):
+            out_dir = sys.argv[i + 1]
+            args = [a for a in args if a != out_dir]
+    if not args:
+        print(__doc__)
+        sys.exit(1)
+    for path in args:
+        if not os.path.exists(path):
+            print(f"!! not found: {path}", file=sys.stderr)
+            continue
+        norm = normalize(path)
+        md = render_md(norm)
+        print(md)
+        print("\n" + "=" * 80 + "\n")
+        base = out_dir or os.path.dirname(os.path.abspath(path))
+        if want_json or out_dir:
+            os.makedirs(base, exist_ok=True)
+            jp = os.path.join(base, f"{norm['session_id']}.session.json")
+            with open(jp, "w") as f:
+                json.dump(norm, f, indent=2)
+            print(f"[wrote {jp}]", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()