@pennyfarthing/core 9.2.0 → 9.3.0

package/pennyfarthing-dist/skills/judge/SKILL.md

@@ -1,644 +0,0 @@
----
-name: judge
-description: Evaluate agent responses using standardized rubrics. Use when scoring benchmark results, comparing agent performance, grading code review quality, or running evaluation pipelines.
----
-
-# Judge Skill
-
-Canonical evaluation of agent responses. All judging goes through this skill.
-
-<run>
-Judge is invoked via CLI with `/judge --mode <mode> --data <json>` to evaluate agent responses using standardized rubrics. Modes include solo (single response), compare (two responses), phase-specific modes (SM/TEA/Dev/Reviewer), coherence (chain coherence), swebench (SWE-bench evaluation), and ground-truth (patch comparison).
-</run>
-
-<output>
-Judge returns structured JSON output containing evaluation scores, weighted totals, reasoning, and token usage information. Output format varies by mode: solo/compare return individual or comparative scores with dimensions (correctness, depth, quality, persona); phase modes return team evaluations; coherence returns a rating (excellent/good/poor); swebench/ground-truth return deterministic scores via Python scripts. All responses include validation of results and error handling for failed evaluations.
-</output>
-
-## Invocation
-
-```
-/judge --mode <mode> --data <json>
-```
-
-**Modes:**
-- `solo` - Single response, absolute rubric (or checklist if baseline_issues provided)
-- `compare` - Two responses, comparative rubric
-- `phase-sm` - Relay SM phase rubric
-- `phase-tea` - Relay TEA phase rubric
-- `phase-dev` - Relay Dev phase rubric
-- `phase-reviewer` - Relay Reviewer phase rubric
-- `coherence` - Relay chain coherence rating
-- `swebench` - Deterministic SWE-bench evaluation (Python script)
-- `ground-truth` - Ground-truth patch comparison (Python script)
-
-## Unified Rubric (solo/compare)
-
-| Dimension | Weight | Criteria |
-|-----------|--------|----------|
-| **Correctness** | 25% | Technical accuracy. Right issues? Valid solutions? |
-| **Depth** | 25% | Thoroughness. Root causes? Implications? |
-| **Quality** | 25% | Clarity and actionability. Organized? Useful? |
-| **Persona** | 25% | Character embodiment. Consistent? Added value? |
-
-**Formula:** `(correctness × 2.5) + (depth × 2.5) + (quality × 2.5) + (persona × 2.5) = WEIGHTED_TOTAL`
-
-## Relay Phase Rubrics
-
-<details>
-<summary><strong>SM Phase Rubric</strong></summary>
-
-| Dimension | Weight |
-|-----------|--------|
-| Clarity | 30% |
-| Handoff | 40% |
-| Completeness | 30% |
-
-</details>
-
-<details>
-<summary><strong>TEA Phase Rubric</strong></summary>
-
-| Dimension | Weight |
-|-----------|--------|
-| Coverage | 35% |
-| RED State | 35% |
-| Handoff | 30% |
-
-</details>
-
-<details>
-<summary><strong>Dev Phase Rubric</strong></summary>
-
-| Dimension | Weight |
-|-----------|--------|
-| GREEN State | 40% |
-| Code Quality | 30% |
-| Handoff | 30% |
-
-</details>
-
-<details>
-<summary><strong>Reviewer Phase Rubric</strong></summary>
-
-| Dimension | Weight |
-|-----------|--------|
-| Detection | 40% |
-| Verdict | 30% |
-| Persona | 30% |
-
-</details>
-
-<details>
-<summary><strong>Chain Coherence Multipliers</strong></summary>
-
-| Rating | Multiplier |
-|--------|------------|
-| excellent | 1.2x |
-| good | 1.0x |
-| poor | 0.8x |
-
-</details>
-
-## On Invoke
-
-### Step 1: Parse Arguments
-
-Extract:
-- `mode`: One of the modes listed above
-- `data`: JSON object with required fields for that mode
-
-**Data requirements by mode:**
-
-| Mode | Required Fields | Optional Fields |
-|------|-----------------|-----------------|
-| solo | `spec`, `character`, `challenge`, `response` | `code`, `baseline_issues`, `baseline_criteria`, `bonus_issues`, `bonus_criteria` |
-| compare | `contestants[]` (each with spec, character, response), `challenge` | `baseline_issues`, `baseline_criteria` |
-| phase-* | `team1`, `team2` (each with theme, response), `context` | |
-| coherence | `theme`, `sm_response`, `tea_response`, `dev_response`, `reviewer_response` | |
-| swebench | `scenario`, `response_file` | |
-| ground-truth | `scenario`, `response_file` | |
-
-**Note:** When checklist data is provided, solo mode uses checklist-based evaluation:
-- `baseline_issues` → code-review, tea, dev scenarios (things to FIND)
-- `baseline_criteria` → SM scenarios (behaviors to DEMONSTRATE)
-- `bonus_issues` / `bonus_criteria` → Extra credit items (optional)
-
-### Step 2: Build Judge Prompt
-
-Based on mode, construct the appropriate prompt:
-
-<details>
-<summary><strong>Solo Mode Prompt (Generic Rubric)</strong></summary>
-
-**If NO baseline_issues provided, use generic rubric:**
-
-```
-You are an impartial judge evaluating an AI agent's response.
-
-## Contestant
-- **{spec}** ({character})
-
-## Challenge
-{challenge}
-
-## Response
-{response}
-
-## Evaluation
-
-Score 1-10 on each dimension:
-
-1. **Correctness (25%)** - Technical accuracy
-2. **Depth (25%)** - Thoroughness
-3. **Quality (25%)** - Clarity and actionability
-4. **Persona (25%)** - Character embodiment
-
-Formula: (correctness × 2.5) + (depth × 2.5) + (quality × 2.5) + (persona × 2.5) = WEIGHTED_TOTAL
-
-**IMPORTANT: Output your evaluation as JSON only. No markdown, no extra text.**
-
-```json
-{
-  "scores": {
-    "correctness": { "value": 8, "reasoning": "..." },
-    "depth": { "value": 7, "reasoning": "..." },
-    "quality": { "value": 9, "reasoning": "..." },
-    "persona": { "value": 8, "reasoning": "..." }
-  },
-  "weighted_total": 80.0,
-  "assessment": "2-3 sentence overall assessment"
-}
-```
-```
-
-</details>
-
-<details>
-<summary><strong>Solo Mode Prompt (Checklist Rubric v2 - Precision/Recall)</strong></summary>
-
-**If baseline_issues IS provided, use checklist rubric (v2 - precision/recall):**
-
-```
-You are an impartial judge evaluating an AI agent's response against a checklist of expected findings.
-
-## Contestant
-- **{spec}** ({character})
-
-## Challenge
-{challenge}
-
-{if code provided}
-## Code Under Review
-{code}
-{endif}
-
-## Expected Findings
-
-Below are the known issues/requirements. Severity indicates weight:
-- CRITICAL: weight 15 (must find)
-- HIGH: weight 10 (should find)
-- MEDIUM: weight 5 (good to find)
-- LOW: weight 2 (bonus)
-- (unlabeled categories like happy_path, validation: weight 5 each)
-
-{baseline_issues formatted as checklist}
-
-## Response to Evaluate
-{response}
-
-## Evaluation Instructions
-
-Evaluate the response and output ONLY valid JSON (no markdown, no extra text):
-
-```json
-{
-  "baseline_findings": [
-    {"id": "ISSUE_ID", "severity": "critical|high|medium|low", "found": true, "evidence": "quote or null"}
-  ],
-  "novel_findings": [
-    {"description": "...", "valid": true, "reasoning": "..."}
-  ],
-  "false_positives": [
-    {"claim": "...", "why_invalid": "..."}
-  ],
-  "detection": {
-    "by_severity": {
-      "critical": {"found": 5, "total": 6},
-      "high": {"found": 4, "total": 6},
-      "medium": {"found": 3, "total": 8},
-      "low": {"found": 1, "total": 2}
-    },
-    "novel_valid": 2,
-    "false_positive_count": 1,
-    "metrics": {
-      "weighted_found": 98,
-      "weighted_total": 120,
-      "recall": 0.817,
-      "precision": 0.929,
-      "f2_score": 0.843
-    },
-    "components": {
-      "recall_score": 24.5,
-      "precision_score": 9.3,
-      "novel_bonus": 6.0
-    },
-    "subtotal": 39.8
-  },
-  "quality": {
-    "clear_explanations": 8,
-    "actionable_fixes": 7,
-    "subtotal": 18.75
-  },
-  "persona": {
-    "in_character": 9,
-    "professional_tone": 8,
-    "subtotal": 21.25
-  },
-  "weighted_total": 79.8,
-  "assessment": "2-3 sentence summary of strengths and gaps"
-}
-```
-
-**Detection Scoring Rules (v2 - Precision/Recall):**
-
-- Severity Weights: critical=15, high=10, medium=5, low=2
-- recall = weighted_found / weighted_total
-- precision = true_positives / (true_positives + false_positives)
-- f2_score = 5 × (precision × recall) / (4 × precision + recall)
-- detection.subtotal = (recall × 30) + (precision × 10) + min(novel_valid × 3, 10)
-
-**Other Dimensions:**
-- Quality (25 max): (clear_explanations/10 × 12.5) + (actionable_fixes/10 × 12.5)
-- Persona (25 max): (in_character/10 × 12.5) + (professional_tone/10 × 12.5)
-- weighted_total = detection.subtotal + quality.subtotal + persona.subtotal
-```
-
-</details>
-
-<details>
-<summary><strong>Detection Scoring Deep Dive</strong></summary>
-
-**Metric Calculations:**
-```
-weighted_found = Σ(found_issues × severity_weight)
-weighted_total = Σ(all_baseline_issues × severity_weight)
-
-recall = weighted_found / weighted_total
-precision = true_positives / (true_positives + false_positives)
-f2_score = 5 × (precision × recall) / (4 × precision + recall)
-```
-
-**Component Scores (Detection = 50 max):**
-```
-recall_score = recall × 30          # max 30 pts - coverage matters most
-precision_score = precision × 10    # max 10 pts - penalizes hallucinations
-novel_bonus = min(novel_valid × 3, 10)  # max 10 pts - rewards thoroughness
-
-detection.subtotal = recall_score + precision_score + novel_bonus
-```
-
-**Why this design:**
-- **Recall weighted 3x precision**: Missing a critical vulnerability is worse than a false positive
-- **Severity-weighted recall**: Finding 5 critical issues > finding 5 low issues
-- **Separate novel bonus**: Rewards thoroughness beyond baseline without affecting precision
-- **Visible metrics**: recall, precision, f2_score all reported for transparency
-
-**Example Calculations:**
-```
-Scenario: 6 critical (90 pts), 6 high (60 pts), 8 medium (40 pts), 2 low (4 pts) = 194 weighted total
-Agent finds: 5 critical, 4 high, 3 medium, 1 low = 75+40+15+2 = 132 weighted found
-Agent flags: 14 true positives, 1 false positive, 2 valid novel findings
-
-recall = 132/194 = 0.680
-precision = 14/15 = 0.933
-f2_score = 5 × (0.933 × 0.680) / (4 × 0.933 + 0.680) = 0.718
-
-recall_score = 0.680 × 30 = 20.4
-precision_score = 0.933 × 10 = 9.3
-novel_bonus = min(2 × 3, 10) = 6.0
-
-detection.subtotal = 20.4 + 9.3 + 6.0 = 35.7
-```
-
-**Checklist Scoring Notes:**
-- **Recall dominates** (30/50 pts): Comprehensive coverage is primary goal
-- **Precision matters** (10/50 pts): Penalizes hallucinated issues proportionally
-- **Novel findings rewarded** (10/50 pts): Encourages going beyond baseline
-- **Severity-weighted**: Critical issues count 7.5x more than low issues
-- **Transparent metrics**: All intermediate values visible for debugging
-- Quality/Persona still matter (25% each) - not just about finding issues
-
-</details>
-
-<details>
-<summary><strong>Solo Mode Prompt (Behavior Checklist - SM Scenarios)</strong></summary>
-
-**If baseline_criteria IS provided (SM scenarios), use behavior checklist:**
-
-```
-You are an impartial judge evaluating an AI agent's facilitation/management response.
-
-## Contestant
-- **{spec}** ({character})
-
-## Challenge
-{challenge}
-
-## Expected Behaviors
-
-Below are the behaviors a good response should demonstrate:
-
-**BASELINE CRITERIA (5 pts each):**
-{baseline_criteria formatted by category}
-
-**BONUS CRITERIA (3 pts each, if present):**
-{bonus_criteria formatted, or "None specified"}
-
-## Response to Evaluate
-{response}
-
-## Evaluation Instructions
-
-Evaluate the response and output ONLY valid JSON (no markdown, no extra text):
-
-```json
-{
-  "baseline_behaviors": [
-    {"id": "BEHAVIOR_ID", "category": "...", "demonstrated": true, "evidence": "quote or null"}
-  ],
-  "bonus_behaviors": [
-    {"id": "BONUS_ID", "category": "...", "demonstrated": true, "evidence": "quote or null"}
-  ],
-  "execution": {
-    "baseline_count": 8,
-    "bonus_count": 2,
-    "subtotal": 46
-  },
-  "quality": {
-    "clear_actionable": 8,
-    "well_structured": 7,
-    "subtotal": 18.75
-  },
-  "persona": {
-    "in_character": 9,
-    "enhances_delivery": 8,
-    "subtotal": 21.25
-  },
-  "weighted_total": 86.0,
-  "assessment": "2-3 sentence summary of facilitation effectiveness"
-}
-```
-
-Scoring rules:
-- Execution (50 max): baseline×5 (cap 40) + bonus×3 (cap 10)
-- Quality (25 max): (clear_actionable/10 × 12.5) + (well_structured/10 × 12.5)
-- Persona (25 max): (in_character/10 × 12.5) + (enhances_delivery/10 × 12.5)
-- weighted_total = execution.subtotal + quality.subtotal + persona.subtotal
-```
-
-</details>
-
-<details>
-<summary><strong>Compare Mode Prompt</strong></summary>
-
-```
-You are an impartial judge comparing two AI personas.
-
-## Contestants
-- **{spec1}** ({character1})
-- **{spec2}** ({character2})
-
-## Challenge
-{challenge}
-
-## Response from {character1}
-{response1}
-
-## Response from {character2}
-{response2}
-
-## Evaluation
-
-Score both on each dimension (1-10). Output ONLY valid JSON (no markdown, no extra text):
-
-```json
-{
-  "contestants": {
-    "{spec1}": {
-      "scores": {
-        "correctness": { "value": 8, "reasoning": "..." },
-        "depth": { "value": 7, "reasoning": "..." },
-        "quality": { "value": 9, "reasoning": "..." },
-        "persona": { "value": 8, "reasoning": "..." }
-      },
-      "weighted_total": 80.0
-    },
-    "{spec2}": {
-      "scores": {
-        "correctness": { "value": 7, "reasoning": "..." },
-        "depth": { "value": 8, "reasoning": "..." },
-        "quality": { "value": 7, "reasoning": "..." },
-        "persona": { "value": 9, "reasoning": "..." }
-      },
-      "weighted_total": 77.5
-    }
-  },
-  "winner": "{spec1}",
-  "justification": "Brief explanation of why winner was chosen"
-}
-```
-```
-
-</details>
-
-<details>
-<summary><strong>Phase Mode and Coherence Mode Prompts</strong></summary>
-
-**Phase Mode Prompts:**
-
-Use phase-specific rubrics from tables above. Evaluate both teams. Output JSON format.
-
-**Coherence Mode Prompt:**
-
-```
-Evaluate chain coherence for {theme}.
-
-## Chain
-SM: {sm_response}
-TEA: {tea_response}
-Dev: {dev_response}
-Reviewer: {reviewer_response}
-
-Output ONLY valid JSON (no markdown, no extra text):
-
-```json
-{
-  "rating": "excellent|good|poor",
-  "reasoning": "explanation of coherence assessment"
-}
-```
-```
-
-</details>
-
-<details>
-<summary><strong>SWE-bench Mode (Deterministic Python Evaluation)</strong></summary>
-
-**For `swebench` and `ground-truth` modes, use Python scripts instead of LLM-as-judge.**
-
-These modes use deterministic scoring based on ground-truth patches from the SWE-bench dataset.
-
-**Prerequisites:**
-```bash
-# Ensure SWE-bench data is downloaded (one-time)
-.pennyfarthing/scripts/test/ensure-swebench-data.sh
-```
-
-**swebench mode:**
-Uses structured rubric + ground truth validation. Scores:
-- root_cause (30%): Bug location + explanation
-- fix_quality (40%): Addresses issue + minimal + syntax correct
-- completeness (20%): Edge cases + test coverage
-- persona (10%): In-character delivery
-
-```bash
-# Execute via Python script
-python3 .pennyfarthing/scripts/test/swebench-judge.py <scenario_name> <response_file>
-
-# Example
-python3 .pennyfarthing/scripts/test/swebench-judge.py flask-5014 /tmp/run_1.json
-```
-
-**ground-truth mode:**
-Compares fix against actual SWE-bench patch. Scores:
-- file_identification (20%): Correct files identified
-- location_identification (20%): Correct functions/locations
-- fix_logic_match (40%): Code matches ground truth
-- completeness (20%): Has all elements of good fix
-
-```bash
-# Execute via Python script
-python3 .pennyfarthing/scripts/test/ground-truth-judge.py <scenario_name> <response_file>
-
-# Example
-python3 .pennyfarthing/scripts/test/ground-truth-judge.py django-10554 /tmp/run_1.json
-```
-
-**Response file format:**
-Both scripts expect JSON with either:
-- `result`: The agent's response text
-- `response_text`: Alternative field name
-
-**Output:**
-Scripts print scores to stdout and save detailed JSON to `{input_path.replace('run_', 'swebench_judge_')}` or `{input_path.replace('run_', 'gt_judge_')}`.
-
-</details>
-
-<details>
-<summary><strong>Step 3: Execute Judge via CLI</strong></summary>
-
-**CRITICAL: Follow this execution pattern for all contexts (main session, skills, subagents).**
-
-**Three rules to avoid shell parsing errors:**
-
-1. **Use Write tool for prompt files** - NOT `echo` in Bash (handles special characters)
-2. **Use file redirection for output** - NOT variable capture `$(...)` (avoids zsh parse errors)
-3. **Use pipe syntax** - NOT heredocs (works in subagents)
-
-**Why variable capture fails:**
-```bash
-# This FAILS - zsh tries to parse JSON with () characters
-OUTPUT=$(cat prompt.txt | claude -p --output-format json --tools "")
-# Error: parse error near ')'
-```
-
-**Correct pattern:**
-
-```bash
-# Step 1: Use Write tool to create prompt file (NOT echo in Bash)
-# The Write tool handles escaping properly in all contexts
-
-# Step 2: Capture timestamp (simple command, safe to capture)
-date -u +%Y-%m-%dT%H:%M:%SZ > .scratch/judge_ts.txt
-
-# Step 3: Execute with FILE REDIRECTION (NOT variable capture)
-cat .scratch/judge_prompt.txt | claude -p --output-format json --tools "" > .scratch/judge_output.json
-
-# Step 4: Extract from files (reading files is always safe)
-JUDGE_RESPONSE=$(jq -r '.result' .scratch/judge_output.json)
-JUDGE_INPUT_TOKENS=$(jq -r '.usage.input_tokens // 0' .scratch/judge_output.json)
-JUDGE_OUTPUT_TOKENS=$(jq -r '.usage.output_tokens // 0' .scratch/judge_output.json)
-```
-
-**Key insight:** The shell never parses the JSON when using file redirection.
-The output goes directly to a file, then jq reads it safely.
-
-</details>
-
-<details>
-<summary><strong>Step 4: Extract Scores</strong></summary>
-
-```bash
-# All modes now output JSON - parse with jq
-# Solo mode
-SCORE=$(echo "$JUDGE_RESPONSE" | jq -r '.weighted_total // empty')
-
-# Compare mode
-SCORE1=$(echo "$JUDGE_RESPONSE" | jq -r '.contestants["{spec1}"].weighted_total // empty')
-SCORE2=$(echo "$JUDGE_RESPONSE" | jq -r '.contestants["{spec2}"].weighted_total // empty')
-WINNER=$(echo "$JUDGE_RESPONSE" | jq -r '.winner // empty')
-
-# Coherence mode
-RATING=$(echo "$JUDGE_RESPONSE" | jq -r '.rating // empty')
-
-# Fallback: try grep if JSON parsing fails (backwards compatibility)
-if [[ -z "$SCORE" ]]; then
-  SCORE=$(echo "$JUDGE_RESPONSE" | grep -oE "weighted_total[\"':]*\s*([0-9.]+)" | grep -oE "[0-9.]+" | tail -1)
-fi
-```
-
-</details>
-
-### Step 5: Validate Results
-
-| Check | Requirement |
-|-------|-------------|
-| `JUDGE_TIMESTAMP` | Valid ISO8601 |
-| `JUDGE_RESPONSE` | At least 200 chars |
-| `SCORE` (if applicable) | Number 1-100 |
-| `RATING` (if coherence) | One of: excellent, good, poor |
-| `JUDGE_INPUT_TOKENS` | > 0 |
-| `JUDGE_OUTPUT_TOKENS` | > 0 |
-
-**If validation fails:** Return error, do NOT estimate.
-
-### Step 6: Return Results
-
-Output structured result for caller:
-
-```json
-{
-  "success": true,
-  "mode": "{mode}",
-  "timestamp": "{JUDGE_TIMESTAMP}",
-  "scores": {
-    "{spec1}": {score1},
-    "{spec2}": {score2}
-  },
-  "winner": "{winner_spec}",
-  "token_usage": {
-    "input": {JUDGE_INPUT_TOKENS},
-    "output": {JUDGE_OUTPUT_TOKENS}
-  },
-  "response_text": "{JUDGE_RESPONSE}"
-}
-```
-
-## Error Handling
-
-```
-❌ Judge validation failed: {reason}
-❌ Mode: {mode}
-❌ DO NOT estimate scores
-```