@pennyfarthing/core 10.0.1 → 10.0.2

package/pennyfarthing-dist/guides/agent-template-tactical.md

@@ -0,0 +1,162 @@
+# Tactical Agent Template
+
+**For:** SM, TEA, Dev, Reviewer (TDD flow agents)
+
+This template defines the standard structure for tactical agents. All tactical agents should follow this section order for consistency and easy comparison.
+
+---
+
+## Section Order
+
+```
+1.  # Title
+2.  <persona>
+3.  <role>
+4.  <helpers>
+5.  <responsibilities>
+6.  <skills>
+7.  <context>
+8.  <reasoning-mode>
+9.  <on-activation>
+10. ## Workflows (markdown sections)
+11. ## Assessment Template
+12. ## Handoff
+13. <exit>
+```
+
+---
+
+## Template
+
+```markdown
+# {NAME} Agent - {Role Title}
+
+<persona>
+Auto-loaded by `agent-session.sh start` from theme config. See output above.
+
+**Fallback if not loaded:** {Brief personality description}
+</persona>
+
+<role>
+**Primary:** {When this agent is invoked, e.g., "via /tea for TDD test writing"}
+**Position:** {Where in TDD flow, e.g., "SM → **TEA** → Dev → Reviewer"}
+</role>
+
+<helpers>
+From theme config. Model: haiku. Tasks: {What helpers do for this agent}
+
+**Skills I Use:**
+- `/skill-name` - {description}
+</helpers>
+
+<responsibilities>
+- {Duty 1 - what this agent is responsible for}
+- {Duty 2}
+- {Duty 3}
+</responsibilities>
+
+<skills>
+**Skills I Use:**
+- `/sprint-context` - Sprint status and backlog
+- `/testing` - Test execution patterns
+</skills>
+
+<context>
+**See:** `.pennyfarthing/guides/agent-behavior.md` for shared tactical agent behavior.
+**See:** `.pennyfarthing/guides/agent-behavior.md` for project info and git strategy.
+</context>
+
+<reasoning-mode>
+**Default:** Quiet mode - follow ReAct pattern internally, show only key decisions
+
+**Toggle:** User says "verbose mode" to see explicit reasoning
+
+When verbose, show thought process:
+```
+THOUGHT: {what I'm considering}
+ACTION: {what I'm doing}
+OBSERVATION: {what I found}
+REFLECT: {what this means}
+```
+
+**{Agent}-Specific Reasoning:**
+- {When doing X: reason about Y}
+- {When doing Z: think through W}
+
+**REMINDER:** Delegate test runs to testing-runner subagent.
+</reasoning-mode>
+
+<on-activation>
+1. Run workflow status check (helper: `.pennyfarthing/agents/workflow-status-check.md`)
+2. Read session file if active work exists
+3. Verify actual state matches session file
+4. Check if handed off to me (Phase field)
+5. If handed off: offer to start immediately
+6. If not: show task menu
+</on-activation>
+
+## Primary Workflow: {Workflow Name}
+
+{Agent-specific workflow description}
+
+### Step 1: {Step Name}
+{Description}
+
+### Step 2: {Step Name}
+{Description}
+
+## {Agent} Assessment Template
+
+```markdown
+## {Agent} Assessment
+
+**{Key field 1}:** {value}
+**{Key field 2}:** {value}
+
+**Handoff:** To {next agent} for {next task}
+```
+
+## Handoff (Context-Aware Auto-Invoke)
+
+After completing work:
+
+1. Write assessment to session file
+2. Spawn handoff helper (`.pennyfarthing/agents/{agent}-handoff.md`)
+3. Check context: `eval $(.pennyfarthing/scripts/core/check-context.sh)`
+4. If `HANDOFF_MODE=auto`: Use Skill tool to invoke next agent
+5. If `HANDOFF_MODE=ask`: Tell user to start fresh session
+
+<exit>
+To exit: "Exit {Agent}" or "Switch to [other agent]"
+
+On exit, run: `.pennyfarthing/scripts/core/agent-session.sh stop`
+</exit>
+```
+
+---
+
+## Section Descriptions
+
+| Section | Purpose | Required |
+|---------|---------|----------|
+| `<persona>` | Character/personality from theme | Yes |
+| `<role>` | When invoked, position in flow | Yes |
+| `<helpers>` | What Haiku helpers do | Yes |
+| `<responsibilities>` | Duties this agent performs | Yes |
+| `<skills>` | /commands this agent uses | Yes |
+| `<context>` | Guide files to reference | Yes |
+| `<reasoning-mode>` | Verbose/quiet toggle | Yes |
+| `<on-activation>` | Startup checklist | Yes |
+| Workflows | Agent-specific procedures | Yes |
+| Assessment | Output format template | Yes |
+| Handoff | Auto-invoke instructions | Yes |
+| `<exit>` | How to leave agent mode | Yes |
+
+---
+
+## Notes
+
+- **Preserve existing workflow content** when aligning to this template
+- **Section order matters** for consistency across agents
+- **XML tags** are used for structured sections that benefit from clear boundaries
+- **Markdown headers** are used for longer narrative sections (workflows)

package/pennyfarthing-dist/guides/hooks.md

@@ -0,0 +1,230 @@
+# Hooks Configuration Guide
+
+Pennyfarthing uses Claude Code hooks to integrate with session lifecycle events and protect files from accidental modification.
+
+## Overview
+
+Hooks are shell scripts that Claude Code runs at specific events:
+
+| Hook Type | When It Runs | Purpose |
+|-----------|--------------|---------|
+| `SessionStart` | When Claude Code session begins | Initialize environment, set variables |
+| `PreToolUse` | Before a tool call executes | Validate/block operations |
+| `PostToolUse` | After a tool call completes | Log, cleanup, notifications |
+
+## Pennyfarthing Default Hooks
+
+### SessionStart: session-start.sh
+
+**Location:** `.claude/pennyfarthing/scripts/hooks/session-start.sh`
+
+Initializes the Pennyfarthing environment:
+- Creates `.session/` directory structure
+- Clears stale agent state from previous sessions
+- Sets `PROJECT_ROOT` and `SESSION_ID` environment variables
+
+### SessionStart: setup-env.sh
+
+**Location:** `.claude/project/hooks/setup-env.sh`
+
+Project-specific environment setup. Edit this file to:
+- Set custom environment variables
+- Configure project-specific paths
+- Initialize project dependencies
+
+### PreToolUse: pre-edit-check.sh
+
+**Location:** `.claude/pennyfarthing/scripts/hooks/pre-edit-check.sh`
+
+Protects sensitive files from accidental edits:
+- Blocks: `.env`, `.pem`, `.key`, credentials, secrets
+- Blocks: `.git/`, `node_modules/`, `vendor/`
+- Blocks: `.claude/pennyfarthing/*` (managed files)
+
+## Configuration Schema
+
+Hooks are configured in `.claude/settings.local.json`:
+
+```json
+{
+  "hooks": {
+    "SessionStart": [
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/pennyfarthing/scripts/hooks/session-start.sh"
+          }
+        ]
+      }
+    ],
+    "PreToolUse": [
+      {
+        "matcher": "Edit|Write",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/pennyfarthing/scripts/hooks/pre-edit-check.sh"
+          }
+        ]
+      }
+    ]
+  }
+}
+```
+
+### Hook Entry Structure
+
+```json
+{
+  "matcher": "ToolNameRegex",  // Optional: regex to filter by tool name
+  "hooks": [
+    {
+      "type": "command",
+      "command": "path/to/script.sh"
+    }
+  ]
+}
+```
+
+### Matcher Format
+
+The `matcher` field is a regex string that matches tool names:
+- `"Edit|Write"` - Match Edit or Write tools
+- `"Bash"` - Match Bash tool only
+- Omit matcher to run on all tool uses
+
+## Hook Script Contract
+
+### Input
+
+Hooks receive JSON via stdin:
+
+```json
+{
+  "session_id": "abc123",
+  "source": "vscode",
+  "tool_name": "Edit",
+  "tool_input": {
+    "file_path": "/path/to/file.ts",
+    "old_string": "...",
+    "new_string": "..."
+  }
+}
+```
+
+### Output
+
+| Exit Code | Meaning |
+|-----------|---------|
+| `0` | Allow (continue execution) |
+| `2` | Block (stderr shown to Claude as error) |
+
+### Environment Variables
+
+Available in hooks:
+- `CLAUDE_PROJECT_DIR` - Project root directory
+- `CLAUDE_ENV_FILE` - Path to write persistent env vars
+
+## Adding Custom Hooks
+
+### 1. Create Hook Script
+
+```bash
+#!/usr/bin/env zsh
+# .claude/project/hooks/my-custom-hook.sh
+set -euo pipefail
+
+input=$(cat)
+# Your logic here
+
+exit 0  # Allow, or exit 2 to block
+```
+
+### 2. Make Executable
+
+```bash
+chmod +x .claude/project/hooks/my-custom-hook.sh
+```
+
+### 3. Configure in settings.local.json
+
+```json
+{
+  "hooks": {
+    "PreToolUse": [
+      {
+        "matcher": "Bash",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/project/hooks/my-custom-hook.sh"
+          }
+        ]
+      }
+    ]
+  }
+}
+```
+
+## Examples
+
+### Block Specific File Patterns
+
+```bash
+#!/usr/bin/env zsh
+# Block edits to migration files
+input=$(cat)
+file_path=$(echo "$input" | jq -r '.tool_input.file_path // ""')
+
+if [[ "$file_path" == *"/migrations/"* ]]; then
+    echo "BLOCKED: Migration files are immutable" >&2
+    exit 2
+fi
+exit 0
+```
+
+### Log All Bash Commands
+
+```bash
+#!/usr/bin/env zsh
+# Log bash commands for audit
+input=$(cat)
+command=$(echo "$input" | jq -r '.tool_input.command // ""')
+
+echo "$(date -Iseconds) | $command" >> "$CLAUDE_PROJECT_DIR/.session/bash-log.txt"
+exit 0
+```
+
+### Environment Setup
+
+```bash
+#!/usr/bin/env zsh
+# .claude/project/hooks/setup-env.sh
+# Set project-specific environment
+
+if [ -n "${CLAUDE_ENV_FILE:-}" ]; then
+    cat >> "$CLAUDE_ENV_FILE" << EOF
+export API_URL="http://localhost:3000"
+export DATABASE_URL="postgres://localhost/myapp"
+EOF
+fi
+exit 0
+```
+
+## Debugging Hooks
+
+If a hook fails or behaves unexpectedly:
+
+1. **Check permissions:** `chmod +x your-hook.sh`
+2. **Test manually:** `echo '{}' | ./your-hook.sh`
+3. **Check stderr:** Hook errors appear in Claude Code output
+4. **Validate JSON:** Use `jq` to parse input correctly
+
+## File Locations
+
+| Type | Location | Editable |
+|------|----------|----------|
+| Managed hooks | `.claude/pennyfarthing/scripts/hooks/` | No (use pennyfarthing repo) |
+| Project hooks | `.claude/project/hooks/` | Yes |
+| Settings | `.claude/settings.local.json` | Yes |

package/pennyfarthing-dist/guides/measurement-framework.md

@@ -0,0 +1,210 @@
+# Measurement Framework for AI Evaluation
+
+> Based on: Wallach et al. (2025). "Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge." ICML 2025.
+>
+> Paper: https://arxiv.org/abs/2502.00561
+
+## Overview
+
+This guide establishes Pennyfarthing's approach to rigorous AI evaluation, grounded in social science measurement theory. The core insight: **evaluating AI systems is fundamentally a measurement challenge**, not merely a technical benchmarking exercise.
+
+Current AI evaluation practices—single-metric benchmarks, leaderboard rankings—fail to capture the complexity of what we're actually trying to measure. Proper evaluation requires the same methodological rigor social scientists apply to measuring abstract constructs like intelligence, fairness, or trust.
+
+## The Four-Level Measurement Framework
+
+All Pennyfarthing evaluation should distinguish between these four levels:
+
+```
+Level 1: Background Concept
+    ↓
+Level 2: Systematized Concept
+    ↓
+Level 3: Operationalization
+    ↓
+Level 4: Scores/Indicators
+```
+
+### Level 1: Background Concept
+
+The broad, often contested idea we care about.
+
+**Examples in Pennyfarthing:**
+- "Code review quality"
+- "Test effectiveness"
+- "Agent helpfulness"
+- "Persona consistency"
+
+**Characteristics:**
+- Often vague or contested
+- Multiple valid interpretations exist
+- May mean different things to different stakeholders
+
+### Level 2: Systematized Concept
+
+A more precise definition scoped to the evaluation context.
+
+**Example:** "Code review quality" → "The degree to which a code review identifies genuine issues, provides actionable feedback, and avoids false positives, within a TypeScript codebase context."
+
+**Requirements:**
+- Explicit scope boundaries
+- Stated assumptions
+- Clear relationship to Level 1 concept
+
+### Level 3: Operationalization
+
+The concrete procedure for measurement.
+
+**Examples:**
+- A benchmark scenario with known issues
+- A judge prompt with scoring rubric
+- A red-teaming protocol
+- Human annotation guidelines
+
+**Key questions:**
+- Does this operationalization actually capture the Level 2 concept?
+- What aspects of the concept does it miss?
+- What confounding factors might it introduce?
+
+### Level 4: Scores/Indicators
+
+The numeric outputs from applying the operationalization.
+
+**Examples:**
+- Detection score: 85/100
+- Precision: 0.89
+- Recall: 0.80
+- Krippendorff's Alpha: 0.72
+
+**Critical insight:** A score is only meaningful in relation to Levels 1-3. Without understanding what construct a benchmark measures, scores are uninterpretable.
+
+## Applying the Framework to Pennyfarthing
+
+### Judge Evaluation
+
+| Level | Current State | Target State |
+|-------|---------------|--------------|
+| 1. Background Concept | "Agent quality" (vague) | Explicit decomposition into sub-constructs |
+| 2. Systematized Concept | Implicit in rubric | Documented construct definitions |
+| 3. Operationalization | Checklist rubric | Anchored rubric with behavioral examples |
+| 4. Scores | 0-100 composite | Precision/recall + quality dimensions |
+
+### Benchmark Scenarios
+
+| Level | Question to Answer |
+|-------|-------------------|
+| 1 | What broad capability are we trying to measure? |
+| 2 | How do we scope that capability for this scenario type? |
+| 3 | Does our scenario design actually test that capability? |
+| 4 | What do the resulting scores tell us (and not tell us)? |
+
+## Validity Evidence
+
+A valid evaluation requires multiple forms of evidence:
+
+### Content Validity
+Does the operationalization cover the construct adequately?
+
+**For Pennyfarthing benchmarks:**
+- Do scenarios cover the range of situations the construct applies to?
+- Are there important aspects of the construct not represented?
+
+### Construct Validity
+Does the operationalization measure what it claims to measure?
+
+**Tests:**
+- Do scores correlate with other measures of the same construct?
+- Do scores NOT correlate with measures of different constructs?
+- Do known-good agents score higher than known-poor agents?
+
+### Criterion Validity
+Do scores predict real-world outcomes?
+
+**For Pennyfarthing:**
+- Do high code-review scores predict fewer bugs in production?
+- Do high test-writing scores predict better test coverage?
+
+### Reliability
+Does the measurement produce consistent results?
+
+**Metrics:**
+- Inter-rater reliability (Krippendorff's Alpha for multi-judge)
+- Test-retest reliability (same agent, same scenario, different runs)
+- Internal consistency (do related items correlate?)
+
+## Anti-Patterns to Avoid
+
+### 1. Jumping to Level 4
+**Problem:** Designing a benchmark without defining what it measures.
+**Symptom:** "We have a score, but we're not sure what it means."
+**Fix:** Start with Level 1-2 before designing operationalization.
+
+### 2. Conflating Operationalization with Construct
+**Problem:** Treating the benchmark as the definition of quality.
+**Symptom:** "A good agent is one that scores high on our benchmark."
+**Fix:** Acknowledge benchmarks are imperfect proxies. Use multiple operationalizations.
+
+### 3. Ignoring Annotator Disagreement
+**Problem:** Averaging away disagreement as "noise."
+**Symptom:** Low Krippendorff's Alpha treated as measurement error.
+**Fix:** Disagreement is signal about construct complexity. Investigate, don't suppress.
+
+### 4. Over-indexing on Single Metrics
+**Problem:** Optimizing for one number.
+**Symptom:** Agents that game benchmarks but fail in real use.
+**Fix:** Use multiple metrics, understand what each measures.
+
+## Implementation in Pennyfarthing Benchmarks
+
+### Scenario Design Checklist
+
+Before creating a new benchmark scenario:
+
+- [ ] **Level 1 defined:** What broad concept does this scenario test?
+- [ ] **Level 2 documented:** How is that concept scoped for this scenario?
+- [ ] **Validity argument:** Why does this scenario test the claimed construct?
+- [ ] **Known limitations:** What aspects of the construct does this NOT test?
+- [ ] **Baseline established:** What is expected performance range?
+
+### Judge Rubric Checklist
+
+Before using a scoring rubric:
+
+- [ ] **Constructs explicit:** What does each dimension measure?
+- [ ] **Anchors defined:** What behaviors correspond to each score level?
+- [ ] **Reliability tested:** What is the inter-judge agreement?
+- [ ] **Edge cases documented:** How should ambiguous situations be scored?
+
+### Results Interpretation Checklist
+
+Before reporting benchmark results:
+
+- [ ] **Context provided:** What construct was measured?
+- [ ] **Limitations stated:** What does this score NOT tell us?
+- [ ] **Confidence indicated:** How reliable is this measurement?
+- [ ] **Comparisons valid:** Are we comparing like with like?
+
+## Relation to Benchmark Reliability Epics
+
+The Benchmark Reliability initiative (epics 41-46) directly implements this framework:
+
+| Epic | Framework Alignment |
+|------|---------------------|
+| 41: Precision/Recall Detection | Level 4 improvement: separate metrics for distinct constructs |
+| 42: Anchored Rubric Criteria | Level 3 improvement: behavioral anchors reduce measurement variance |
+| 43: False Positive Traps | Level 3 improvement: test construct validity with red herrings |
+| 44: Multi-Judge Validation | Reliability evidence: measure inter-rater agreement |
+| 45: Gold Standard References | Level 3 improvement: calibration anchors for consistent scoring |
+| 46: Difficulty Profile | Level 2 improvement: multi-dimensional construct decomposition |
+
+## References
+
+- Wallach, H., et al. (2025). Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge. ICML 2025. https://arxiv.org/abs/2502.00561
+- Adcock, R., & Collier, D. (2001). Measurement validity: A shared standard for qualitative and quantitative research. APSR.
+- Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin.
+- Messick, S. (1995). Validity of psychological assessment. American Psychologist.
+- HELM: Holistic Evaluation of Language Models. Stanford CRFM.
+- ARC-AGI: A benchmark for measuring machine intelligence.
+
+## Changelog
+
+- 2026-01-23: Initial version based on Wallach et al. (2025) ICML paper