@pennyfarthing/core 10.4.0 → 11.0.0

package/pennyfarthing-dist/skills/pf-judge/SKILL.md

@@ -0,0 +1,644 @@
+---
+name: judge
+description: Evaluate agent responses using standardized rubrics. Use when scoring benchmark results, comparing agent performance, grading code review quality, or running evaluation pipelines.
+---
+
+# Judge Skill
+
+Canonical evaluation of agent responses. All judging goes through this skill.
+
+<run>
+Judge is invoked via CLI with `/judge --mode <mode> --data <json>` to evaluate agent responses using standardized rubrics. Modes include solo (single response), compare (two responses), phase-specific modes (SM/TEA/Dev/Reviewer), coherence (chain coherence), swebench (SWE-bench evaluation), and ground-truth (patch comparison).
+</run>
+
+<output>
+Judge returns structured JSON output containing evaluation scores, weighted totals, reasoning, and token usage information. Output format varies by mode: solo/compare return individual or comparative scores with dimensions (correctness, depth, quality, persona); phase modes return team evaluations; coherence returns a rating (excellent/good/poor); swebench/ground-truth return deterministic scores via Python scripts. All responses include validation of results and error handling for failed evaluations.
+</output>
+
+## Invocation
+
+```
+/judge --mode <mode> --data <json>
+```
+
+**Modes:**
+- `solo` - Single response, absolute rubric (or checklist if baseline_issues provided)
+- `compare` - Two responses, comparative rubric
+- `phase-sm` - Relay SM phase rubric
+- `phase-tea` - Relay TEA phase rubric
+- `phase-dev` - Relay Dev phase rubric
+- `phase-reviewer` - Relay Reviewer phase rubric
+- `coherence` - Relay chain coherence rating
+- `swebench` - Deterministic SWE-bench evaluation (Python script)
+- `ground-truth` - Ground-truth patch comparison (Python script)
+
+## Unified Rubric (solo/compare)
+
+| Dimension | Weight | Criteria |
+|-----------|--------|----------|
+| **Correctness** | 25% | Technical accuracy. Right issues? Valid solutions? |
+| **Depth** | 25% | Thoroughness. Root causes? Implications? |
+| **Quality** | 25% | Clarity and actionability. Organized? Useful? |
+| **Persona** | 25% | Character embodiment. Consistent? Added value? |
+
+**Formula:** `(correctness × 2.5) + (depth × 2.5) + (quality × 2.5) + (persona × 2.5) = WEIGHTED_TOTAL`
+
+## Relay Phase Rubrics
+
+<details>
+<summary><strong>SM Phase Rubric</strong></summary>
+
+| Dimension | Weight |
+|-----------|--------|
+| Clarity | 30% |
+| Handoff | 40% |
+| Completeness | 30% |
+
+</details>
+
+<details>
+<summary><strong>TEA Phase Rubric</strong></summary>
+
+| Dimension | Weight |
+|-----------|--------|
+| Coverage | 35% |
+| RED State | 35% |
+| Handoff | 30% |
+
+</details>
+
+<details>
+<summary><strong>Dev Phase Rubric</strong></summary>
+
+| Dimension | Weight |
+|-----------|--------|
+| GREEN State | 40% |
+| Code Quality | 30% |
+| Handoff | 30% |
+
+</details>
+
+<details>
+<summary><strong>Reviewer Phase Rubric</strong></summary>
+
+| Dimension | Weight |
+|-----------|--------|
+| Detection | 40% |
+| Verdict | 30% |
+| Persona | 30% |
+
+</details>
+
+<details>
+<summary><strong>Chain Coherence Multipliers</strong></summary>
+
+| Rating | Multiplier |
+|--------|------------|
+| excellent | 1.2x |
+| good | 1.0x |
+| poor | 0.8x |
+
+</details>
+
+## On Invoke
+
+### Step 1: Parse Arguments
+
+Extract:
+- `mode`: One of the modes listed above
+- `data`: JSON object with required fields for that mode
+
+**Data requirements by mode:**
+
+| Mode | Required Fields | Optional Fields |
+|------|-----------------|-----------------|
+| solo | `spec`, `character`, `challenge`, `response` | `code`, `baseline_issues`, `baseline_criteria`, `bonus_issues`, `bonus_criteria` |
+| compare | `contestants[]` (each with spec, character, response), `challenge` | `baseline_issues`, `baseline_criteria` |
+| phase-* | `team1`, `team2` (each with theme, response), `context` | |
+| coherence | `theme`, `sm_response`, `tea_response`, `dev_response`, `reviewer_response` | |
+| swebench | `scenario`, `response_file` | |
+| ground-truth | `scenario`, `response_file` | |
+
+**Note:** When checklist data is provided, solo mode uses checklist-based evaluation:
+- `baseline_issues` → code-review, tea, dev scenarios (things to FIND)
+- `baseline_criteria` → SM scenarios (behaviors to DEMONSTRATE)
+- `bonus_issues` / `bonus_criteria` → Extra credit items (optional)
+
+### Step 2: Build Judge Prompt
+
+Based on mode, construct the appropriate prompt:
+
+<details>
+<summary><strong>Solo Mode Prompt (Generic Rubric)</strong></summary>
+
+**If NO baseline_issues provided, use generic rubric:**
+
+```
+You are an impartial judge evaluating an AI agent's response.
+
+## Contestant
+- **{spec}** ({character})
+
+## Challenge
+{challenge}
+
+## Response
+{response}
+
+## Evaluation
+
+Score 1-10 on each dimension:
+
+1. **Correctness (25%)** - Technical accuracy
+2. **Depth (25%)** - Thoroughness
+3. **Quality (25%)** - Clarity and actionability
+4. **Persona (25%)** - Character embodiment
+
+Formula: (correctness × 2.5) + (depth × 2.5) + (quality × 2.5) + (persona × 2.5) = WEIGHTED_TOTAL
+
+**IMPORTANT: Output your evaluation as JSON only. No markdown, no extra text.**
+
+```json
+{
+  "scores": {
+    "correctness": { "value": 8, "reasoning": "..." },
+    "depth": { "value": 7, "reasoning": "..." },
+    "quality": { "value": 9, "reasoning": "..." },
+    "persona": { "value": 8, "reasoning": "..." }
+  },
+  "weighted_total": 80.0,
+  "assessment": "2-3 sentence overall assessment"
+}
+```
+```
+
+</details>
+
+<details>
+<summary><strong>Solo Mode Prompt (Checklist Rubric v2 - Precision/Recall)</strong></summary>
+
+**If baseline_issues IS provided, use checklist rubric (v2 - precision/recall):**
+
+```
+You are an impartial judge evaluating an AI agent's response against a checklist of expected findings.
+
+## Contestant
+- **{spec}** ({character})
+
+## Challenge
+{challenge}
+
+{if code provided}
+## Code Under Review
+{code}
+{endif}
+
+## Expected Findings
+
+Below are the known issues/requirements. Severity indicates weight:
+- CRITICAL: weight 15 (must find)
+- HIGH: weight 10 (should find)
+- MEDIUM: weight 5 (good to find)
+- LOW: weight 2 (bonus)
+- (unlabeled categories like happy_path, validation: weight 5 each)
+
+{baseline_issues formatted as checklist}
+
+## Response to Evaluate
+{response}
+
+## Evaluation Instructions
+
+Evaluate the response and output ONLY valid JSON (no markdown, no extra text):
+
+```json
+{
+  "baseline_findings": [
+    {"id": "ISSUE_ID", "severity": "critical|high|medium|low", "found": true, "evidence": "quote or null"}
+  ],
+  "novel_findings": [
+    {"description": "...", "valid": true, "reasoning": "..."}
+  ],
+  "false_positives": [
+    {"claim": "...", "why_invalid": "..."}
+  ],
+  "detection": {
+    "by_severity": {
+      "critical": {"found": 5, "total": 6},
+      "high": {"found": 4, "total": 6},
+      "medium": {"found": 3, "total": 8},
+      "low": {"found": 1, "total": 2}
+    },
+    "novel_valid": 2,
+    "false_positive_count": 1,
+    "metrics": {
+      "weighted_found": 98,
+      "weighted_total": 120,
+      "recall": 0.817,
+      "precision": 0.929,
+      "f2_score": 0.843
+    },
+    "components": {
+      "recall_score": 24.5,
+      "precision_score": 9.3,
+      "novel_bonus": 6.0
+    },
+    "subtotal": 39.8
+  },
+  "quality": {
+    "clear_explanations": 8,
+    "actionable_fixes": 7,
+    "subtotal": 18.75
+  },
+  "persona": {
+    "in_character": 9,
+    "professional_tone": 8,
+    "subtotal": 21.25
+  },
+  "weighted_total": 79.8,
+  "assessment": "2-3 sentence summary of strengths and gaps"
+}
+```
+
+**Detection Scoring Rules (v2 - Precision/Recall):**
+
+- Severity Weights: critical=15, high=10, medium=5, low=2
+- recall = weighted_found / weighted_total
+- precision = true_positives / (true_positives + false_positives)
+- f2_score = 5 × (precision × recall) / (4 × precision + recall)
+- detection.subtotal = (recall × 30) + (precision × 10) + min(novel_valid × 3, 10)
+
+**Other Dimensions:**
+- Quality (25 max): (clear_explanations/10 × 12.5) + (actionable_fixes/10 × 12.5)
+- Persona (25 max): (in_character/10 × 12.5) + (professional_tone/10 × 12.5)
+- weighted_total = detection.subtotal + quality.subtotal + persona.subtotal
+```
+
+</details>
+
+<details>
+<summary><strong>Detection Scoring Deep Dive</strong></summary>
+
+**Metric Calculations:**
+```
+weighted_found = Σ(found_issues × severity_weight)
+weighted_total = Σ(all_baseline_issues × severity_weight)
+
+recall = weighted_found / weighted_total
+precision = true_positives / (true_positives + false_positives)
+f2_score = 5 × (precision × recall) / (4 × precision + recall)
+```
+
+**Component Scores (Detection = 50 max):**
+```
+recall_score = recall × 30          # max 30 pts - coverage matters most
+precision_score = precision × 10    # max 10 pts - penalizes hallucinations
+novel_bonus = min(novel_valid × 3, 10)  # max 10 pts - rewards thoroughness
+
+detection.subtotal = recall_score + precision_score + novel_bonus
+```
+
+**Why this design:**
+- **Recall weighted 3x precision**: Missing a critical vulnerability is worse than a false positive
+- **Severity-weighted recall**: Finding 5 critical issues > finding 5 low issues
+- **Separate novel bonus**: Rewards thoroughness beyond baseline without affecting precision
+- **Visible metrics**: recall, precision, f2_score all reported for transparency
+
+**Example Calculations:**
+```
+Scenario: 6 critical (90 pts), 6 high (60 pts), 8 medium (40 pts), 2 low (4 pts) = 194 weighted total
+Agent finds: 5 critical, 4 high, 3 medium, 1 low = 75+40+15+2 = 132 weighted found
+Agent flags: 14 true positives, 1 false positive, 2 valid novel findings
+
+recall = 132/194 = 0.680
+precision = 14/15 = 0.933
+f2_score = 5 × (0.933 × 0.680) / (4 × 0.933 + 0.680) = 0.718
+
+recall_score = 0.680 × 30 = 20.4
+precision_score = 0.933 × 10 = 9.3
+novel_bonus = min(2 × 3, 10) = 6.0
+
+detection.subtotal = 20.4 + 9.3 + 6.0 = 35.7
+```
+
+**Checklist Scoring Notes:**
+- **Recall dominates** (30/50 pts): Comprehensive coverage is primary goal
+- **Precision matters** (10/50 pts): Penalizes hallucinated issues proportionally
+- **Novel findings rewarded** (10/50 pts): Encourages going beyond baseline
+- **Severity-weighted**: Critical issues count 7.5x more than low issues
+- **Transparent metrics**: All intermediate values visible for debugging
+- Quality/Persona still matter (25% each) - not just about finding issues
+
+</details>
+
+<details>
+<summary><strong>Solo Mode Prompt (Behavior Checklist - SM Scenarios)</strong></summary>
+
+**If baseline_criteria IS provided (SM scenarios), use behavior checklist:**
+
+```
+You are an impartial judge evaluating an AI agent's facilitation/management response.
+
+## Contestant
+- **{spec}** ({character})
+
+## Challenge
+{challenge}
+
+## Expected Behaviors
+
+Below are the behaviors a good response should demonstrate:
+
+**BASELINE CRITERIA (5 pts each):**
+{baseline_criteria formatted by category}
+
+**BONUS CRITERIA (3 pts each, if present):**
+{bonus_criteria formatted, or "None specified"}
+
+## Response to Evaluate
+{response}
+
+## Evaluation Instructions
+
+Evaluate the response and output ONLY valid JSON (no markdown, no extra text):
+
+```json
+{
+  "baseline_behaviors": [
+    {"id": "BEHAVIOR_ID", "category": "...", "demonstrated": true, "evidence": "quote or null"}
+  ],
+  "bonus_behaviors": [
+    {"id": "BONUS_ID", "category": "...", "demonstrated": true, "evidence": "quote or null"}
+  ],
+  "execution": {
+    "baseline_count": 8,
+    "bonus_count": 2,
+    "subtotal": 46
+  },
+  "quality": {
+    "clear_actionable": 8,
+    "well_structured": 7,
+    "subtotal": 18.75
+  },
+  "persona": {
+    "in_character": 9,
+    "enhances_delivery": 8,
+    "subtotal": 21.25
+  },
+  "weighted_total": 86.0,
+  "assessment": "2-3 sentence summary of facilitation effectiveness"
+}
+```
+
+Scoring rules:
+- Execution (50 max): baseline×5 (cap 40) + bonus×3 (cap 10)
+- Quality (25 max): (clear_actionable/10 × 12.5) + (well_structured/10 × 12.5)
+- Persona (25 max): (in_character/10 × 12.5) + (enhances_delivery/10 × 12.5)
+- weighted_total = execution.subtotal + quality.subtotal + persona.subtotal
+```
+
+</details>
+
+<details>
+<summary><strong>Compare Mode Prompt</strong></summary>
+
+```
+You are an impartial judge comparing two AI personas.
+
+## Contestants
+- **{spec1}** ({character1})
+- **{spec2}** ({character2})
+
+## Challenge
+{challenge}
+
+## Response from {character1}
+{response1}
+
+## Response from {character2}
+{response2}
+
+## Evaluation
+
+Score both on each dimension (1-10). Output ONLY valid JSON (no markdown, no extra text):
+
+```json
+{
+  "contestants": {
+    "{spec1}": {
+      "scores": {
+        "correctness": { "value": 8, "reasoning": "..." },
+        "depth": { "value": 7, "reasoning": "..." },
+        "quality": { "value": 9, "reasoning": "..." },
+        "persona": { "value": 8, "reasoning": "..." }
+      },
+      "weighted_total": 80.0
+    },
+    "{spec2}": {
+      "scores": {
+        "correctness": { "value": 7, "reasoning": "..." },
+        "depth": { "value": 8, "reasoning": "..." },
+        "quality": { "value": 7, "reasoning": "..." },
+        "persona": { "value": 9, "reasoning": "..." }
+      },
+      "weighted_total": 77.5
+    }
+  },
+  "winner": "{spec1}",
+  "justification": "Brief explanation of why winner was chosen"
+}
+```
+```
+
+</details>
+
+<details>
+<summary><strong>Phase Mode and Coherence Mode Prompts</strong></summary>
+
+**Phase Mode Prompts:**
+
+Use phase-specific rubrics from tables above. Evaluate both teams. Output JSON format.
+
+**Coherence Mode Prompt:**
+
+```
+Evaluate chain coherence for {theme}.
+
+## Chain
+SM: {sm_response}
+TEA: {tea_response}
+Dev: {dev_response}
+Reviewer: {reviewer_response}
+
+Output ONLY valid JSON (no markdown, no extra text):
+
+```json
+{
+  "rating": "excellent|good|poor",
+  "reasoning": "explanation of coherence assessment"
+}
+```
+```
+
+</details>
+
+<details>
+<summary><strong>SWE-bench Mode (Deterministic Python Evaluation)</strong></summary>
+
+**For `swebench` and `ground-truth` modes, use Python scripts instead of LLM-as-judge.**
+
+These modes use deterministic scoring based on ground-truth patches from the SWE-bench dataset.
+
+**Prerequisites:**
+```bash
+# Ensure SWE-bench data is downloaded (one-time)
+.pennyfarthing/scripts/test/ensure-swebench-data.sh
+```
+
+**swebench mode:**
+Uses structured rubric + ground truth validation. Scores:
+- root_cause (30%): Bug location + explanation
+- fix_quality (40%): Addresses issue + minimal + syntax correct
+- completeness (20%): Edge cases + test coverage
+- persona (10%): In-character delivery
+
+```bash
+# Execute via Python script
+python3 .pennyfarthing/scripts/test/swebench-judge.py <scenario_name> <response_file>
+
+# Example
+python3 .pennyfarthing/scripts/test/swebench-judge.py flask-5014 /tmp/run_1.json
+```
+
+**ground-truth mode:**
+Compares fix against actual SWE-bench patch. Scores:
+- file_identification (20%): Correct files identified
+- location_identification (20%): Correct functions/locations
+- fix_logic_match (40%): Code matches ground truth
+- completeness (20%): Has all elements of good fix
+
+```bash
+# Execute via Python script
+python3 .pennyfarthing/scripts/test/ground-truth-judge.py <scenario_name> <response_file>
+
+# Example
+python3 .pennyfarthing/scripts/test/ground-truth-judge.py django-10554 /tmp/run_1.json
+```
+
+**Response file format:**
+Both scripts expect JSON with either:
+- `result`: The agent's response text
+- `response_text`: Alternative field name
+
+**Output:**
+Scripts print scores to stdout and save detailed JSON to `{input_path.replace('run_', 'swebench_judge_')}` or `{input_path.replace('run_', 'gt_judge_')}`.
+
+</details>
+
+<details>
+<summary><strong>Step 3: Execute Judge via CLI</strong></summary>
+
+**CRITICAL: Follow this execution pattern for all contexts (main session, skills, subagents).**
+
+**Three rules to avoid shell parsing errors:**
+
+1. **Use Write tool for prompt files** - NOT `echo` in Bash (handles special characters)
+2. **Use file redirection for output** - NOT variable capture `$(...)` (avoids zsh parse errors)
+3. **Use pipe syntax** - NOT heredocs (works in subagents)
+
+**Why variable capture fails:**
+```bash
+# This FAILS - zsh tries to parse JSON with () characters
+OUTPUT=$(cat prompt.txt | claude -p --output-format json --tools "")
+# Error: parse error near ')'
+```
+
+**Correct pattern:**
+
+```bash
+# Step 1: Use Write tool to create prompt file (NOT echo in Bash)
+# The Write tool handles escaping properly in all contexts
+
+# Step 2: Capture timestamp (simple command, safe to capture)
+date -u +%Y-%m-%dT%H:%M:%SZ > .scratch/judge_ts.txt
+
+# Step 3: Execute with FILE REDIRECTION (NOT variable capture)
+cat .scratch/judge_prompt.txt | claude -p --output-format json --tools "" > .scratch/judge_output.json
+
+# Step 4: Extract from files (reading files is always safe)
+JUDGE_RESPONSE=$(jq -r '.result' .scratch/judge_output.json)
+JUDGE_INPUT_TOKENS=$(jq -r '.usage.input_tokens // 0' .scratch/judge_output.json)
+JUDGE_OUTPUT_TOKENS=$(jq -r '.usage.output_tokens // 0' .scratch/judge_output.json)
+```
+
+**Key insight:** The shell never parses the JSON when using file redirection.
+The output goes directly to a file, then jq reads it safely.
+
+</details>
+
+<details>
+<summary><strong>Step 4: Extract Scores</strong></summary>
+
+```bash
+# All modes now output JSON - parse with jq
+# Solo mode
+SCORE=$(echo "$JUDGE_RESPONSE" | jq -r '.weighted_total // empty')
+
+# Compare mode
+SCORE1=$(echo "$JUDGE_RESPONSE" | jq -r '.contestants["{spec1}"].weighted_total // empty')
+SCORE2=$(echo "$JUDGE_RESPONSE" | jq -r '.contestants["{spec2}"].weighted_total // empty')
+WINNER=$(echo "$JUDGE_RESPONSE" | jq -r '.winner // empty')
+
+# Coherence mode
+RATING=$(echo "$JUDGE_RESPONSE" | jq -r '.rating // empty')
+
+# Fallback: try grep if JSON parsing fails (backwards compatibility)
+if [[ -z "$SCORE" ]]; then
+  SCORE=$(echo "$JUDGE_RESPONSE" | grep -oE "weighted_total[\"':]*\s*([0-9.]+)" | grep -oE "[0-9.]+" | tail -1)
+fi
+```
+
+</details>
+
+### Step 5: Validate Results
+
+| Check | Requirement |
+|-------|-------------|
+| `JUDGE_TIMESTAMP` | Valid ISO8601 |
+| `JUDGE_RESPONSE` | At least 200 chars |
+| `SCORE` (if applicable) | Number 1-100 |
+| `RATING` (if coherence) | One of: excellent, good, poor |
+| `JUDGE_INPUT_TOKENS` | > 0 |
+| `JUDGE_OUTPUT_TOKENS` | > 0 |
+
+**If validation fails:** Return error, do NOT estimate.
+
+### Step 6: Return Results
+
+Output structured result for caller:
+
+```json
+{
+  "success": true,
+  "mode": "{mode}",
+  "timestamp": "{JUDGE_TIMESTAMP}",
+  "scores": {
+    "{spec1}": {score1},
+    "{spec2}": {score2}
+  },
+  "winner": "{winner_spec}",
+  "token_usage": {
+    "input": {JUDGE_INPUT_TOKENS},
+    "output": {JUDGE_OUTPUT_TOKENS}
+  },
+  "response_text": "{JUDGE_RESPONSE}"
+}
+```
+
+## Error Handling
+
+```
+❌ Judge validation failed: {reason}
+❌ Mode: {mode}
+❌ DO NOT estimate scores
+```