@pennyfarthing/core 7.9.2 → 7.9.5

package/pennyfarthing-dist/commands/run-ci.md

@@ -0,0 +1,116 @@
+---
+description: Detect and run CI locally
+---
+
+<purpose>
+Run CI locally by detecting the project's CI system and executing the appropriate commands. Thin wrapper that delegates to project configuration.
+</purpose>
+
+<when-to-use>
+- Before pushing to verify CI will pass
+- To reproduce CI failures locally
+- To run the full CI pipeline without pushing
+- When you want to know what CI system the project uses
+</when-to-use>
+
+<execution>
+
+## Running CI Locally
+
+Use the run-ci.sh script:
+
+```bash
+# Run CI locally (auto-detects CI system)
+$CLAUDE_PROJECT_DIR/.pennyfarthing/scripts/run-ci.sh
+
+# Show what CI system is detected without running
+$CLAUDE_PROJECT_DIR/.pennyfarthing/scripts/run-ci.sh --detect-only
+
+# Show what command would run without executing
+$CLAUDE_PROJECT_DIR/.pennyfarthing/scripts/run-ci.sh --dry-run
+
+# Show help
+$CLAUDE_PROJECT_DIR/.pennyfarthing/scripts/run-ci.sh --help
+```
+
+## Options
+
+| Option | Description |
+|--------|-------------|
+| `--help`, `-h` | Show help message |
+| `--detect-only` | Show detected CI system without running |
+| `--dry-run` | Show what would run without executing |
+
+</execution>
+
+<detection-order>
+
+## CI System Detection
+
+The script detects CI systems in this order:
+
+| Priority | System | Detection | Command |
+|----------|--------|-----------|---------|
+| 1 | Justfile | `just --list` shows `ci` recipe | `just ci` |
+| 2 | GitHub Actions | `.github/workflows/*.yml` exists | `act` |
+| 3 | GitLab CI | `.gitlab-ci.yml` exists | `gitlab-runner exec` |
+| 4 | npm fallback | `package.json` exists | `npm run build && npm test && npm run lint` |
+
+### Package Manager Detection
+
+For npm fallback, the script detects package manager:
+- `pnpm-lock.yaml` or `pnpm-workspace.yaml` → `pnpm`
+- `yarn.lock` → `yarn`
+- Otherwise → `npm`
+
+</detection-order>
+
+<output-format>
+
+## Output Examples
+
+### --detect-only
+```
+Detected CI system: github-actions
+Would run: act
+```
+
+### --dry-run
+```
+Detected: github-actions
+Would run: act
+```
+
+### Running CI
+```
+Running CI: github-actions
+Command: act
+
+[act output follows...]
+```
+
+Exit codes:
+- `0` - CI ran successfully
+- `1` - CI failed or no CI system detected
+
+</output-format>
+
+<requirements>
+
+## Tool Requirements
+
+| CI System | Required Tool | Installation |
+|-----------|--------------|--------------|
+| GitHub Actions | `act` | `brew install act` |
+| GitLab CI | `gitlab-runner` | See GitLab docs |
+| Justfile | `just` | `brew install just` |
+| npm fallback | `npm`/`pnpm`/`yarn` | Comes with Node.js |
+
+If a required tool is not installed, the script will show a warning.
+
+</requirements>
+
+<reference>
+- **Script:** `.pennyfarthing/scripts/run-ci.sh`
+- **Story:** 21-4 (Command & Skill Expansion)
+</reference>

package/pennyfarthing-dist/commands/set-theme.md

@@ -0,0 +1,56 @@
+---
+description: Set the active persona theme
+---
+
+# Set Theme
+
+Change the active persona theme for all agents.
+
+## Arguments
+
+- `<name>` - Theme name to activate (required)
+
+## Instructions
+
+1. If no theme name provided, list available themes:
+   ```bash
+   ls pennyfarthing-dist/personas/themes/*.yaml | xargs -I{} basename {} .yaml | sort
+   ```
+   Then ask the user which theme they want to use.
+
+2. Validate theme exists:
+   ```bash
+   ls pennyfarthing-dist/personas/themes/<name>.yaml
+   ```
+   If not found, show error and list available themes.
+
+3. Read the current config file to prepare for editing:
+   ```bash
+   cat .pennyfarthing/config.local.yaml 2>/dev/null || echo "FILE_NOT_FOUND"
+   ```
+
+4. Set the theme in `.pennyfarthing/config.local.yaml`:
+
+   **If the file exists and has a `theme:` line:**
+   - Use the Edit tool to replace `theme: <old>` with `theme: <new>`
+
+   **If the file doesn't exist or lacks the theme line:**
+   - Use the Write tool to create/overwrite the file:
+   ```yaml
+   # Pennyfarthing Local Configuration
+   # This file is gitignored - your personal preferences
+
+   theme: <name>
+   ```
+
+5. Verify the change was written:
+   ```bash
+   cat .pennyfarthing/config.local.yaml
+   ```
+   Confirm the theme line shows the new value.
+
+6. Refresh the current agent's persona to apply the new theme:
+   ```bash
+   d="$PWD"; while [[ ! -d "$d/.claude" ]] && [[ "$d" != "/" ]]; do d="$(dirname "$d")"; done; "$d/.pennyfarthing/scripts/core/run.sh" core/agent-session.sh refresh
+   ```
+   This outputs the updated persona. **Adopt the new character immediately** - do not continue using the old persona.

package/pennyfarthing-dist/commands/show-theme.md

@@ -0,0 +1,21 @@
+---
+description: Show details of a theme including all agent personas
+---
+
+# Show Theme
+
+Display full details of a theme including all agent character mappings.
+
+## Arguments
+
+- `[name]` - Theme name to show (optional, defaults to current theme)
+
+## Instructions
+
+Run the pennyfarthing CLI to show theme details:
+
+```bash
+pennyfarthing theme show [name]
+```
+
+Present the theme information clearly, showing each agent's character, style, and quote.

package/pennyfarthing-dist/commands/sm.md

@@ -0,0 +1,7 @@
+---
+description: Scrum Master - Story coordination and sprint management
+---
+
+```bash
+d="$PWD"; while [[ ! -d "$d/.claude" ]] && [[ "$d" != "/" ]]; do d="$(dirname "$d")"; done; "$d/.pennyfarthing/scripts/core/run.sh" core/phase-check-start.sh "sm"
+```

package/pennyfarthing-dist/commands/solo.md

@@ -0,0 +1,447 @@
+---
+description: Run a single agent on a scenario with absolute rubric scoring
+argument-hint: <theme:agent> --scenario <name> [--as <role>] [--runs N] [--no-judge]
+---
+
+# Solo Benchmark
+
+<purpose>
+Run a single agent on a scenario. This is the CANONICAL agent execution path.
+
+**Modes:**
+- **Full (default):** Agent runs → `/judge` evaluates → `/finalize-run` saves
+- **No-judge (`--no-judge`):** Agent runs only, returns raw response (for /duel, /relay)
+</purpose>
+
+<architecture>
+```
+/solo theme:agent --scenario X
+    │
+    ├──► Execute agent via CLI
+    │         └──► Response + tokens
+    │
+    ├──► /judge --mode solo (if not --no-judge)
+    │         └──► Score + verdict
+    │
+    └──► /finalize-run --type solo
+              └──► Validate + save
+```
+</architecture>
+
+<usage>
+```
+/solo <contestant> --scenario <name>
+/solo <contestant> --scenario <name> --runs 4
+/solo <contestant> --scenario <name> --no-judge
+/solo <contestant> --as <role> --scenario <name>
+```
+
+**Arguments:**
+- `contestant` - `theme:agent` format (e.g., `discworld:reviewer`) OR `theme:character` with `--as`
+- `--scenario` - Scenario from `scenarios/` directory
+- `--as <role>` - Override role (use character's persona for different role's task)
+- `--runs N` - Number of runs (default: 1, max: 20)
+- `--no-judge` - Skip judging, return raw response
+
+**Cross-Role Testing with `--as`:**
+
+The `--as` flag enables running any persona as any role, useful for research:
+
+```
+/solo shakespeare:prospero --as dev --scenario django-10554
+```
+
+This uses Prospero's persona traits (wise orchestrator, magic metaphors) but gives him a dev task.
+The scenario's role determines what the agent is asked to do; the character determines how they do it.
+</usage>
+
+<on-invoke>
+The user invoked this command with: $ARGUMENTS
+
+## Step 1: Parse Arguments
+
+Extract:
+- `contestant`: `theme:agent` spec (or `theme:character` if using `--as`)
+- `scenario_name`: After `--scenario`
+- `role_override`: After `--as` (optional - for cross-role testing)
+- `runs`: Number (default: 1)
+- `no_judge`: Boolean
+
+Validate spec contains `:`, scenario is required, runs is 1-20.
+
+**If `--as` is provided:**
+- The second part of the spec is a CHARACTER name, not a role
+- `role_override` becomes the effective role for scenario matching
+- Character persona is extracted by name lookup across all agents in theme
+
+## Step 2: Load Scenario
+
+```yaml
+Glob tool:
+  pattern: "scenarios/**/{scenario_name}.yaml"
+```
+
+Extract: `prompt`, `scenario_title`, `code_content` (if present)
+
+## Step 3: Load Persona
+
+Read: `pennyfarthing-dist/personas/themes/{theme}.yaml`
+
+**Standard mode (no `--as`):**
+- Look up `agents.{agent}` section
+- Extract: `character`, `style`, `expertise`, `catchphrases`, `emoji`
+- `effective_role` = agent name from spec
+
+**Cross-role mode (with `--as`):**
+- The spec contains `theme:character_name` (e.g., `shakespeare:prospero`)
+- Search ALL agent sections for one where `character` field matches (case-insensitive, partial match OK)
+- Extract persona traits from that agent's config
+- `effective_role` = the `--as` value (NOT the role the character normally fills)
+
+```python
+# Pseudocode for cross-role lookup
+if role_override:
+    character_query = spec.split(':')[1].lower()  # e.g., "prospero"
+    for agent_name, agent_config in theme['agents'].items():
+        char_name = agent_config.get('character', '').lower()
+        if character_query in char_name or char_name.startswith(character_query):
+            persona = agent_config
+            source_role = agent_name  # where character normally lives
+            break
+    effective_role = role_override  # what we're asking them to do
+else:
+    agent_name = spec.split(':')[1]  # e.g., "dev"
+    persona = theme['agents'][agent_name]
+    effective_role = agent_name
+    source_role = agent_name
+```
+
+This enables running Prospero (normally SM) as a dev, or Gus Fring (normally orchestrator) as a reviewer.
+
+## Step 3b: Build Agent Prompt
+
+Use the Write tool to create the prompt file with this template:
+
+```
+You are {character}.
+
+**Style:** {style}
+**Expertise:** {expertise}
+**Catchphrases:** {catchphrases}
+
+---
+
+## Challenge
+
+{scenario_prompt}
+
+{code_content if present}
+
+---
+
+Respond fully in character. Under 500 words.
+
+**IMPORTANT:** Provide your complete response directly. Do not attempt to use tools, read files, or make function calls.
+```
+
+**Cross-role note:** When using `--as`, the scenario prompt comes from the `effective_role` (e.g., dev tasks),
+but the character/style/expertise come from the character's original role config. This tests whether
+personality traits affect task performance independent of role-specific training.
+
+The final instruction is critical - without it, the model may output tool-call syntax even with `--tools ""`, resulting in incomplete responses.
+
+## Step 4: Execute Agent via CLI
+
+**RECOMMENDED: Use the shell script for reliable execution:**
+
+```bash
+./scripts/solo-runner.sh {theme}:{agent} {scenario} {output_dir}
+```
+
+The shell script handles all escaping, temp files, and JSON parsing correctly.
+Use inline commands only for simple cases or when the script isn't available.
+
+---
+
+**CRITICAL: The `--tools ""` flag is MANDATORY.**
+
+Without `--tools ""`, agents may use tools internally (Read, Write, Bash, etc.), causing:
+1. Multi-turn conversations (num_turns > 1)
+2. The `.result` field only captures the FINAL message (often just a summary)
+3. Full response content is LOST - judges only see truncated output
+4. Scores are INVALID because judges evaluate incomplete data
+
+**Evidence:** Miles Vorkosigan benchmark (2026-01-01) scored 76.69 with tools enabled vs Leo McGarry's 91.03 with `--tools ""`. Miles' runs had num_turns: 5-7 and judges only saw summaries, not full story breakdowns.
+
+**CRITICAL: Use PIPE syntax, NOT heredocs.**
+
+**NEVER USE HEREDOCS** - Heredoc syntax (`<<'EOF'`, `<<EOF`, `<<'PROMPT'`, etc.) FAILS in subagents.
+The permission system treats heredocs differently and they get auto-denied.
+
+**ALWAYS USE PIPE SYNTAX** - This works in both main sessions and subagents:
+- `echo "$PROMPT" | claude -p ...` - WORKS
+- `cat file.txt | claude -p ...` - WORKS
+- `printf '%s' "$PROMPT" | claude -p ...` - WORKS
+- `claude -p ... <<'EOF'` - **FAILS IN SUBAGENTS - DO NOT USE**
+
+**CRITICAL: Use FILE REDIRECTION, NOT variable capture.**
+
+**NEVER CAPTURE OUTPUT IN VARIABLES** - Command substitution with `$(...)` causes zsh parse errors
+when the JSON output contains parentheses or special characters:
+- `OUTPUT=$(cat file.txt | claude -p ...)` - **FAILS with `parse error near ')'`**
+
+**ALWAYS REDIRECT TO FILES** - This avoids shell parsing issues:
+- `cat file.txt | claude -p ... > output.json` - WORKS
+- Then read: `jq -r '.result' output.json` - WORKS
+
+```bash
+# Step 1: Capture timestamp to file (avoid variable capture issues)
+date -u +%Y-%m-%dT%H:%M:%SZ > /tmp/timestamp_$$.txt
+
+# Step 2: Write prompt to file using Write tool (avoids escaping issues)
+# Use the Write tool to create: /tmp/prompt_$$.txt
+
+# Step 3: Execute with file redirection (NOT variable capture)
+cat /tmp/prompt_$$.txt | claude -p --output-format json --tools "" > /tmp/output_$$.json
+
+# Step 4: Extract results from files
+TIMESTAMP=$(cat /tmp/timestamp_$$.txt)
+RESPONSE=$(jq -r '.result' /tmp/output_$$.json)
+INPUT_TOKENS=$(jq -r '.usage.input_tokens // 0' /tmp/output_$$.json)
+OUTPUT_TOKENS=$(jq -r '.usage.output_tokens // 0' /tmp/output_$$.json)
+
+# Step 5: Cleanup
+rm -f /tmp/timestamp_$$.txt /tmp/prompt_$$.txt /tmp/output_$$.json
+```
+
+**Why file redirection works:** The shell never tries to parse the JSON output.
+It goes directly to a file, then jq reads it safely.
+
+## Step 5: Check Mode
+
+**If `--no-judge`:** Return raw response and metadata, STOP.
+
+```markdown
+## Solo Agent Response
+
+**Contestant:** {spec} ({character})
+**Scenario:** {scenario_name}
+
+---
+
+{response}
+
+---
+
+```json
+{
+  "spec": "{spec}",
+  "character": "{character}",
+  "cli_timestamp": "{TIMESTAMP}",
+  "response_length": {length},
+  "input_tokens": {INPUT_TOKENS},
+  "output_tokens": {OUTPUT_TOKENS}
+}
+```
+```
+
+**If full mode:** Continue to Step 6.
+
+## Step 6: Invoke Judge Skill
+
+**Detect SWE-bench scenarios for deterministic evaluation:**
+
+Check if the scenario is from SWE-bench by looking at its path or category:
+```python
+is_swebench = (
+    'swe-bench' in scenario_path.lower() or
+    scenario.get('category') == 'swe-bench' or
+    scenario.get('source') == 'swe-bench'
+)
+```
+
+**If SWE-bench scenario:**
+
+Use deterministic Python-based evaluation instead of LLM-as-judge:
+
+```bash
+# Save response to temp file for Python judge
+echo '{"result": "{RESPONSE}"}' > /tmp/solo_response_$$.json
+
+# Run SWE-bench judge (deterministic scoring against ground truth)
+python3 .pennyfarthing/scripts/test/swebench-judge.py {scenario_name} /tmp/solo_response_$$.json
+```
+
+The Python script returns:
+- `total`: Score out of 100
+- `scores`: Breakdown by category (root_cause, fix_quality, completeness, persona)
+- `details`: Specific findings and matches
+
+**If standard scenario (non-SWE-bench):**
+
+Use LLM-as-judge:
+```
+/judge --mode solo --data {
+  "spec": "{spec}",
+  "character": "{character}",
+  "challenge": "{prompt}",
+  "response": "{RESPONSE}"
+}
+```
+
+Capture: `score`, `judge_timestamp`, `judge_response`, `judge_tokens`
+
+## Step 7: Invoke Finalize-Run Skill
+
+```
+/finalize-run --type solo --data {
+  "timestamp": "{ISO8601}",
+  "scenario": {"name": "{scenario_name}", "title": "{title}"},
+  "agents": [{
+    "spec": "{spec}",
+    "cli_timestamp": "{TIMESTAMP}",
+    "response_text": "{RESPONSE}",
+    "input_tokens": {INPUT_TOKENS},
+    "output_tokens": {OUTPUT_TOKENS}
+  }],
+  "judge": {
+    "cli_timestamp": "{judge_timestamp}",
+    "response_text": "{judge_response}",
+    "input_tokens": {judge_input},
+    "output_tokens": {judge_output}
+  },
+  "scores": {"{spec}": {score}},
+  "output_path": "internal/results/solo/{timestamp}-{theme}-{agent}.json"
+}
+```
+
+## Step 8: Display Results
+
+```markdown
+## Solo Evaluation
+
+{judge verdict}
+
+---
+
+## Efficiency
+
+| Metric | Value |
+|--------|-------|
+| Agent Tokens | {agent_total} |
+| Judge Tokens | {judge_total} |
+| Score | {score}/100 |
+| Tokens/Point | {tpp} |
+
+---
+
+✓ Saved to {output_path}
+```
+
+## Step 9: Multi-Run Mode (if runs > 1)
+
+1. Create output directory (see Step 10 for path logic)
+2. Repeat Steps 4-7 for each run
+3. Save each to `runs/run_{i}.json` and `runs/judge_{i}.json`
+4. Calculate statistics and save summary.yaml (Step 10)
+
+## Step 10: Save Summary (ALWAYS - even for n=1)
+
+**Output path logic:**
+
+```
+if theme == "control":
+  base_path = "internal/results/baselines/{scenario}/{effective_role}/"
+elif role_override:  # cross-role mode
+  # Include character name and effective role for clarity
+  base_path = "internal/results/benchmarks/{scenario}/{theme}-{character}-as-{effective_role}/"
+else:
+  base_path = "internal/results/benchmarks/{scenario}/{theme}-{effective_role}/"
+```
+
+**Cross-role example:** `/solo shakespeare:prospero --as dev --scenario django-10554`
+→ saves to `internal/results/benchmarks/django-10554/shakespeare-prospero-as-dev/`
+
+**For ALL runs (including n=1):**
+
+1. Create directory structure:
+   ```bash
+   mkdir -p "{base_path}/runs"
+   ```
+
+2. Save run files:
+   - `runs/run_{i}.json` - Agent response + tokens
+   - `runs/judge_{i}.json` - Judge evaluation
+
+3. Calculate statistics:
+   ```python
+   scores = [run.score for run in runs]
+   mean = sum(scores) / len(scores)
+   std_dev = sqrt(sum((s - mean)^2 for s in scores) / len(scores))
+   ```
+
+4. **ALWAYS save summary.yaml:**
+   ```yaml
+   # {theme}:{character} on {scenario} (as {effective_role})
+   # Generated: {ISO8601 timestamp}
+
+   agent:
+     theme: {theme}
+     character: {character_name}
+     effective_role: {effective_role}      # role being performed
+     source_role: {source_role}            # role where character normally lives
+     spec: {theme}:{character}             # original spec
+     cross_role: {true if role_override else false}
+
+   scenario:
+     name: {scenario_name}
+     category: {category}
+     difficulty: {difficulty}
+
+   statistics:
+     n: {run_count}
+     mean: {mean:.2f}
+     std_dev: {std_dev:.2f}
+     min: {min_score}
+     max: {max_score}
+     scores: [{score1}, {score2}, ...]
+
+   efficiency:
+     avg_input_tokens: {avg_in}
+     avg_output_tokens: {avg_out}
+     tokens_per_point: {tpp:.2f}
+
+   metadata:
+     created_at: {ISO8601 timestamp}
+     pennyfarthing_version: {version from package.json}  # REQUIRED
+     model: sonnet
+
+   # Include baseline comparison if baseline exists and theme != control
+   baseline_comparison:
+     control_mean: {baseline_mean}
+     control_stddev: {baseline_std}
+     delta: {mean - baseline_mean:+.2f}
+
+   runs:
+     - run_1.json
+     - run_2.json
+     # ...
+   ```
+
+5. Display:
+   ```
+   ✓ Saved {n} run(s) to {base_path}
+   ✓ Summary: {base_path}/summary.yaml
+   ```
+
+</on-invoke>
+
+<reference>
+- **Judge Skill:** `.claude/project/skills/judge/SKILL.md`
+- **Finalize-Run Skill:** `.claude/project/skills/finalize-run/SKILL.md`
+- **Themes:** `pennyfarthing-dist/personas/themes/*.yaml`
+- **Scenarios:** `scenarios/**/*.yaml`
+- **Baselines:** `internal/results/baselines/{scenario}/{role}/` (control theme)
+- **Benchmarks:** `internal/results/benchmarks/{scenario}/{theme}-{role}/` (all other themes)
+- **Results README:** `internal/results/README.md`
+</reference>