@pennyfarthing/core 9.2.0 → 9.4.0

package/pennyfarthing-dist/agents/workflow-status-check.md

@@ -1,96 +0,0 @@
----
-name: workflow-status-check
-description: Determine workflow state using sprint scripts
-tools: Bash, Read
-model: haiku
----
-
-<info>
-Universal entry point telling agents: what work exists, what phase, and whether to activate.
-Uses `/sprint` skill scripts for deterministic output.
-</info>
-
-<arguments>
-| Argument | Required | Description |
-|----------|----------|-------------|
-| `CALLING_AGENT` | Yes | Agent requesting status check (e.g., "SM", "Architect", "PM") |
-</arguments>
-
----
-
-## Execution
-
-Run the sprint status script and parse output:
-
-```bash
-.pennyfarthing/scripts/sprint/sprint-status.sh
-```
-
-Then check for active sessions:
-
-```bash
-if ls .session/*-session.md 1>/dev/null 2>&1; then
-  for f in .session/*-session.md; do head -30 "$f"; done
-else
-  echo "No active sessions"
-fi
-```
-
----
-
-## State Determination
-
-| State | Condition |
-|-------|-----------|
-| `FINISH_STATE` | Session exists with Phase=approved OR Status=approved |
-| `IN_PROGRESS_STATE` | Phased workflow session with active phase (setup/red/green/impl/review) |
-| `STEPPED_WORKFLOW_STATE` | Stepped workflow session (workflow type = stepped) |
-| `NEW_WORK_STATE` | No sessions AND sprint has backlog/ready stories |
-| `EMPTY_BACKLOG_STATE` | No sessions AND sprint has NO backlog/ready stories |
-
-**Detecting workflow type from session:**
-```bash
-# Read workflow name from session
-WORKFLOW=$(grep '^\*\*Workflow:\*\*' .session/*-session.md | head -1 | sed 's/.*: //')
-# Check if stepped
-.pennyfarthing/scripts/workflow/get-workflow-type.sh "$WORKFLOW"
-```
-
-**Important:** Sprints are fixed two-week periods (kanban-style). Never suggest closing a sprint early or starting sprint planning when backlog is empty. The correct response to `EMPTY_BACKLOG_STATE` is to suggest promoting stories from `future.yaml`.
-
----
-
-<output>
-## Output Format
-
-Return a `STATUS_CHECK_RESULT` block:
-
-### Success
-```
-STATUS_CHECK_RESULT:
-  status: success
-  state: {FINISH_STATE|IN_PROGRESS_STATE|NEW_WORK_STATE|EMPTY_BACKLOG_STATE}
-  story_id: {ID or null}
-  phase: {current phase or null}
-  phase_owner: {agent name or null}
-  sprint_number: {N}
-  backlog_count: {N}
-
-  next_steps:
-    - FINISH_STATE: "Proceed to Finish Flow - spawn sm-finish with PHASE=preflight"
-    - IN_PROGRESS_STATE: "Report phase owner '{phase_owner}' should continue. Run handoff-marker.sh {phase_owner}"
-    - STEPPED_WORKFLOW_STATE: "Stepped workflow in progress. Tell user to run /workflow resume or /workflow status"
-    - NEW_WORK_STATE: "Present available stories to user. Await selection, then spawn sm-setup MODE=setup"
-    - EMPTY_BACKLOG_STATE: "Report backlog empty. Suggest promoting from future.yaml"
-```
-
-### Active Session Details (if IN_PROGRESS_STATE)
-```
-  session:
-    story_id: {ID}
-    title: "{title}"
-    workflow: {workflow}
-    phase: {phase}
-    branch: {branch}
-```
-</output>

package/pennyfarthing-dist/commands/architect.md

@@ -1,9 +0,0 @@
----
-description: System Architect - Technical design and architecture
----
-
-```bash
-d="$PWD"; while [[ ! -d "$d/.pennyfarthing" ]] && [[ "$d" != "/" ]]; do d="$(dirname "$d")"; done
-export PYTHONPATH="$(dirname "$(dirname "$(cd "$d/.pennyfarthing/scripts" && pwd -P)")"):${PYTHONPATH:-}"
-python3 -m pennyfarthing_scripts.cli agent start "architect"
-```

package/pennyfarthing-dist/commands/benchmark-control.md

@@ -1,69 +0,0 @@
----
-description: Create control baseline for a scenario (shortcut for /benchmark control <agent>)
-argument-hint: <agent> [--scenario <name>] [--runs N]
----
-
-# Benchmark Control
-
-<purpose>
-Shortcut to run `/benchmark` with the `control` theme. Creates or extends a control baseline for comparing other personas against.
-
-This is equivalent to running:
-```
-/benchmark control <agent> [--scenario <name>] [--runs N]
-```
-
-Default: 10 runs for statistically meaningful baseline data.
-</purpose>
-
-<critical-integrity-requirements>
-**See `/benchmark` for full integrity requirements.**
-
-Baselines are saved to `internal/results/baselines/{scenario}/{agent}/` with:
-- Individual runs in `runs/*.json` with proof-of-work
-- Summary statistics in `summary.yaml` (mean, std_dev, CI)
-- Timestamp validation (runs must take 30+ seconds each)
-
-Control theme runs must include all proof fields. NO FABRICATION.
-</critical-integrity-requirements>
-
-<usage>
-```
-# Pick scenario interactively
-/benchmark-control sm
-/benchmark-control reviewer
-
-# Specify scenario directly
-/benchmark-control reviewer --scenario order-service
-/benchmark-control dev --scenario tdd-shopping-cart --runs 15
-```
-
-**Arguments:**
-- `agent` - The agent role (e.g., `sm`, `dev`, `reviewer`, `architect`)
-- `--scenario` - (Optional) Scenario name. If omitted, shows matching scenarios.
-- `--runs N` - Number of runs (default: 10 for baselines, max: 20)
-</usage>
-
-<on-invoke>
-The user invoked this command with: $ARGUMENTS
-
-**This is a shortcut.** Translate the arguments and invoke `/benchmark`:
-
-1. Prepend `control` as the theme
-2. Pass through all other arguments
-
-**Examples:**
-- `/benchmark-control sm` → `/benchmark control sm` (runs control:sm)
-- `/benchmark-control reviewer --scenario order-service` → `/benchmark control reviewer --scenario order-service` (runs control:reviewer)
-- `/benchmark-control dev --runs 15` → `/benchmark control dev --runs 15` (runs control:dev)
-
-**Default runs override:** If `--runs` is not specified, default to 10 (instead of 4) since baselines need more data.
-
-Now execute the equivalent `/benchmark` command with the translated arguments.
-</on-invoke>
-
-<reference>
-- Main command: `.claude/project/commands/benchmark.md`
-- Baselines location: `internal/results/baselines/{scenario}/{role}/`
-- Results README: `internal/results/README.md`
-</reference>

package/pennyfarthing-dist/commands/benchmark.md

@@ -1,485 +0,0 @@
----
-description: Compare an agent's performance against a stored baseline
-argument-hint: <theme> <agent> [--as <role>] [--scenario <name>] [--runs N]
----
-
-# Benchmark
-
-<purpose>
-Compare a persona agent's performance against the established control baseline. Runs the agent on the scenario and calculates statistical measures including effect size (Cohen's d) and significance.
-
-Default: 4 runs for comparison (balance between reliability and runtime). Runs execute in parallel for faster results.
-
-**Simplified Usage:** Just specify theme and agent role - you'll be presented with matching scenarios to choose from.
-</purpose>
-
-<critical-integrity-requirements>
-## DO NOT FABRICATE COMPARISON DATA
-
-Comparisons are only meaningful if BOTH the baseline AND the contestant runs are real.
-
-**Before comparing:**
-1. Validate baseline has proof-of-work (check runs have `proof.*` fields)
-2. Actually run `/solo` for the contestant with real Task tool calls
-3. Validate contestant runs have proof-of-work before calculating statistics
-
-**Baseline Validation:**
-Before using a baseline, spot-check at least one run file:
-- Read a run from `internal/results/baselines/{scenario}/{agent}/runs/*.json`
-- Verify it has `proof.agent_task_id`, `proof.agent_response_text`, `proof.judge_task_id`
-- Verify `proof.agent_response_text` is at least 200 characters
-- Verify `token_usage.input_tokens` > 0
-
-**If baseline validation fails:**
-```markdown
-Error: Baseline for '{scenario}' appears to be fabricated (missing proof-of-work).
-Run `/benchmark-control --scenario {scenario}` to create a real baseline.
-```
-
-**Contestant runs MUST include proof-of-work.** See `/solo` for requirements.
-</critical-integrity-requirements>
-
-<usage>
-```
-# Simple: Pick scenario interactively
-/benchmark the-expanse sm
-/benchmark discworld reviewer
-
-# Direct: Specify scenario explicitly
-/benchmark discworld reviewer --scenario order-service
-/benchmark ted-lasso dev --scenario tdd-shopping-cart --runs 8
-
-# Cross-role: Run any character as any role
-/benchmark shakespeare prospero --as dev --scenario django-10554
-/benchmark discworld granny --as dev --scenario tdd-shopping-cart
-```
-
-**Arguments:**
-- `theme` - The persona theme (e.g., `discworld`, `the-expanse`, `ted-lasso`)
-- `agent` - The agent role OR character name (if using `--as`)
-- `--as <role>` - (Optional) Override role for cross-role testing. Makes `agent` a character name lookup.
-- `--scenario` - (Optional) Scenario name. If omitted, shows matching scenarios to choose from.
-- `--runs N` - Number of evaluation runs (default: 4, max: 20)
-
-**Cross-Role Testing:**
-The `--as` flag enables running any character as any role:
-```
-/benchmark shakespeare prospero --as dev --scenario django-10554
-```
-This uses Prospero's persona traits (wise orchestrator) but gives him a dev task.
-The scenario's role determines what the agent is asked to do; the character determines HOW they do it.
-
-**Examples:**
-```
-# Let me pick from SM scenarios
-/benchmark the-expanse sm
-
-# Let me pick from code review scenarios
-/benchmark discworld reviewer
-
-# Run specific scenario directly
-/benchmark princess-bride reviewer --scenario order-service --runs 8
-
-# Cross-role: Prospero (SM) doing dev work
-/benchmark shakespeare prospero --as dev --scenario tdd-shopping-cart --runs 4
-```
-</usage>
-
-<on-invoke>
-The user invoked this command with: $ARGUMENTS
-
-## Step 1: Parse Arguments
-
-Parse the arguments to extract:
-- `theme`: First positional argument (e.g., `discworld`, `the-expanse`)
-- `agent_or_character`: Second positional argument (role name OR character name if `--as` is used)
-- `role_override`: Value after `--as` (OPTIONAL - enables cross-role mode)
-- `scenario_name`: Value after `--scenario` (OPTIONAL)
-- `runs`: Value after `--runs` (default: 4, max: 20)
-
-**Cross-Role Mode:**
-If `--as <role>` is provided:
-- `agent_or_character` is treated as a CHARACTER NAME (case-insensitive search)
-- `role_override` becomes the `effective_role` for scenario matching
-- Results save to `internal/results/benchmarks/{scenario}/{theme}-{character}-as-{role}/`
-
-**Legacy format support:** If first argument contains `:`, split it (e.g., `discworld:reviewer` → theme=discworld, agent_or_character=reviewer)
-
-**Validation:**
-- Theme must be a valid theme name
-- If `--as` is provided: validate `role_override` is one of: `sm`, `dev`, `reviewer`, `architect`, `tea`, `pm`
-- If `--as` is NOT provided: validate `agent_or_character` is one of: `sm`, `dev`, `reviewer`, `architect`, `tea`, `pm`
-- `--runs` must be a positive integer between 1 and 20
-
-**Determine effective_role:**
-```python
-if role_override:
-    effective_role = role_override  # e.g., "dev"
-    cross_role = True
-else:
-    effective_role = agent_or_character  # e.g., "dev"
-    cross_role = False
-```
-
-## Step 2: Scenario Discovery (if --scenario not provided)
-
-If `scenario_name` is NOT provided, discover matching scenarios.
-
-**Use `effective_role` (not `agent_or_character`) for scenario discovery.**
-Cross-role mode: Prospero --as dev should see dev scenarios, not SM scenarios.
-
-**Role-to-Category Mapping:**
-| effective_role | Scenario Categories |
-|----------------|---------------------|
-| sm | `sm` |
-| dev | `dev` (includes debug scenarios) |
-| reviewer | `code-review` |
-| architect | `architecture` |
-| tea | `tea` |
-
-**Time Estimates by Difficulty (parallel execution):**
-| Difficulty | Est. Time (4 runs) | Note |
-|------------|-------------------|------|
-| easy | ~1 min | Runs execute in parallel |
-| medium | ~2 min | Runs execute in parallel |
-| hard | ~4 min | Runs execute in parallel |
-| extreme | ~8 min | Runs execute in parallel |
-
-**Discover scenarios:**
-```bash
-# Use Bash to list matching scenarios
-ls scenarios/{category}/*.yaml | xargs -I {} yq -r '"{}|\(.name)|\(.difficulty)|\(.title)|\(.description)"' {}
-```
-
-**Present choices (Reflector-aware):**
-
-First output marker: `<!-- CYCLIST:CHOICES:scenario -->`
-
-Then use AskUserQuestion:
-```yaml
-AskUserQuestion:
-  questions:
-    - question: "Which scenario do you want to benchmark {theme}:{agent_type} on?"
-      header: "Scenario"
-      multiSelect: false
-      options:
-        - label: "{name} ({difficulty})"
-          description: "{title} - ~{time_estimate}"
-        # ... up to 4 options
-```
-
-If more than 4 scenarios exist, show the first 4 by difficulty (hardest first) and let user type "Other" for full list.
-
-**After user selects:** Set `scenario_name` to the selected scenario's name and continue.
-
-## Step 3: Control Theme Handling
-
-**If theme is `control`:** This is a baseline creation run.
-- Default `runs` to 10 (instead of 4) for statistical reliability
-- Results save to `internal/results/baselines/{scenario}/{agent}/` instead of comparison
-- Skip baseline validation (we're creating the baseline)
-- After running, calculate and save baseline statistics
-- Display baseline summary and exit
-
-**If theme is NOT `control`:** Continue to Step 4 for comparison workflow.
-
-## Step 4: Load and Validate Baseline
-
-**Baseline is based on `effective_role`, not the character's native role.**
-Cross-role tests compare against the effective role's baseline (e.g., prospero --as dev compares against control:dev).
-
-Check if baseline exists:
-
-```yaml
-Read tool:
-  file_path: "internal/results/baselines/{scenario_name}/{effective_role}/summary.yaml"
-```
-
-**If baseline does not exist:**
-```markdown
-Error: No baseline found for scenario '{scenario_name}' with agent type '{agent_type}'.
-
-To create a baseline, run:
-/benchmark control {agent_type} --scenario {scenario_name}
-
-Or use the shortcut:
-/benchmark-control {agent_type} --scenario {scenario_name}
-```
-
-**If baseline exists, VALIDATE IT:**
-
-1. Get list of run files:
-   ```yaml
-   Glob tool:
-     pattern: "internal/results/baselines/{scenario_name}/{agent_type}/runs/*.json"
-   ```
-
-2. Read at least one run file and validate proof-of-work:
-   ```yaml
-   Read tool:
-     file_path: "{first run file}"
-   ```
-
-3. **Check for proof-of-work fields:**
-   - Has `proof.agent_task_id`?
-   - Has `proof.agent_response_text` with length >= 200?
-   - Has `proof.judge_task_id`?
-   - Has `proof.judge_response_text`?
-   - Has `token_usage.input_tokens` > 0?
-   - Has `token_usage.output_tokens` > 0?
-
-4. **If validation fails:**
-   ```markdown
-   Error: Baseline for '{scenario_name}' is INVALID - missing proof-of-work.
-
-   The baseline data appears to be fabricated (no agent/judge response text,
-   no task IDs, or no token counts).
-
-   Delete the invalid baseline and create a real one:
-   rm -rf internal/results/baselines/{scenario_name}/{agent_type}
-   /benchmark-control --scenario {scenario_name}
-   ```
-
-**If baseline is valid:**
-- Extract `sample_size`, `statistics.total.mean`, `statistics.total.std_dev`
-- Display baseline info with validation confirmation
-
-**Sample size warning:**
-If baseline sample size < 5:
-```markdown
-**Warning:** Baseline sample size ({n}) is less than 5. Results may not be statistically reliable.
-Consider running `/benchmark-control --scenario {scenario_name} --runs 10` to add more data.
-```
-
-## Step 5: Run Contestant Evaluation (Parallel)
-
-For efficiency, spawn multiple runs in parallel using Task agents.
-
-**Batch Strategy:**
-- If runs ≤ 4: Spawn all in parallel (single message with N Task agents)
-- If runs > 4: Spawn in batches of 4 to avoid overwhelming the system
-
-**Build the /solo command:**
-```python
-if cross_role:
-    # Cross-role: agent_or_character is a character name
-    solo_cmd = f"/solo {theme}:{agent_or_character} --as {effective_role} --scenario {scenario_name}"
-else:
-    # Standard: agent_or_character is the role name
-    solo_cmd = f"/solo {theme}:{agent_or_character} --scenario {scenario_name}"
-```
-
-**For each run, spawn a Task agent:**
-```
-Task (run 1 of N):
-  subagent_type: general-purpose
-  prompt: |
-    Run {solo_cmd}
-    This is run 1 of N for baseline/benchmark.
-    Return the full result JSON including score and token_usage.
-```
-
-**Example commands:**
-- Standard: `/solo discworld:dev --scenario tdd-shopping-cart`
-- Cross-role: `/solo shakespeare:prospero --as dev --scenario tdd-shopping-cart`
-
-**Spawn all batch tasks in a SINGLE message for parallel execution.**
-
-Wait for all tasks to complete. Collect results:
-- Per-run scores (total, plus dimension breakdown if available)
-- Per-run token usage (input_tokens, output_tokens)
-- Per-run timestamps
-- Cross-role metadata (source_role, effective_role, cross_role flag)
-
-**If a run fails:** Note the failure, continue with successful runs. Warn if < 3 successful runs.
-
-## Step 6: Calculate Comparison Statistics
-
-**Contestant Statistics:**
-- `contestant_mean`: Average total score
-- `contestant_std_dev`: Standard deviation
-- `contestant_n`: Number of runs
-
-**Baseline Statistics (from summary.yaml):**
-- `baseline_mean`: statistics.total.mean
-- `baseline_std_dev`: statistics.total.std_dev
-- `baseline_n`: sample_size
-
-**Mean Difference:**
-```
-difference = contestant_mean - baseline_mean
-```
-
-**Cohen's d Effect Size:**
-```
-pooled_std_dev = sqrt((contestant_std_dev² + baseline_std_dev²) / 2)
-cohens_d = difference / pooled_std_dev
-```
-
-**Effect Size Interpretation:**
-| Cohen's d | Interpretation |
-|-----------|----------------|
-| < 0.2 | Negligible |
-| 0.2 - 0.5 | Small |
-| 0.5 - 0.8 | Medium |
-| > 0.8 | Large |
-
-**95% Confidence Interval for Difference:**
-```
-se_diff = sqrt(contestant_std_dev²/contestant_n + baseline_std_dev²/baseline_n)
-ci_lower = difference - 1.96 × se_diff
-ci_upper = difference + 1.96 × se_diff
-```
-
-**Statistical Significance:**
-If CI does not include 0, the difference is statistically significant at p < 0.05.
-
-## Step 7: Display Comparison Results
-
-```markdown
----
-
-## Baseline Comparison
-
-**Contestant:** {theme}:{agent_type} ({character_name})
-**Scenario:** {scenario_name}
-**Baseline:** control:{agent_type} (n={baseline_n})
-
-### Performance vs Baseline
-
-| Metric | Contestant | Baseline | Difference | Effect Size |
-|--------|------------|----------|------------|-------------|
-| Total Score | {c_mean} ± {c_std} | {b_mean} ± {b_std} | {diff:+.1f} | **{cohens_d:.1f}σ** ({interpretation}) |
-| Detection | {c_det} | {b_det} | {diff:+.1f} | {effect} |
-| Depth | {c_dep} | {b_dep} | {diff:+.1f} | {effect} |
-| Quality | {c_qual} | {b_qual} | {diff:+.1f} | {effect} |
-| Persona | {c_per} | {b_per} | {diff:+.1f} | {effect} |
-
-### Efficiency
-
-| Metric | Contestant | Baseline |
-|--------|------------|----------|
-| Tokens/Point | {c_tokens_per_point} | {b_tokens_per_point} |
-| Efficiency | {efficiency_pct}% of baseline | 100% |
-
-### Statistical Significance
-
-- **Effect Size (Cohen's d):** {cohens_d:.2f} ({interpretation})
-- **95% CI for difference:** [{ci_lower:+.1f}, {ci_upper:+.1f}]
-- **Significant:** {Yes/No} (p < 0.05)
-
-### Verdict
-
-{verdict based on effect size and significance}
-
----
-```
-
-**Verdict Logic:**
-- If not significant: "No statistically significant difference from baseline."
-- If significant and positive large effect: "Contestant **significantly outperforms** baseline with large effect size."
-- If significant and positive medium effect: "Contestant **outperforms** baseline with medium effect size."
-- If significant and positive small effect: "Contestant **slightly outperforms** baseline."
-- If significant and negative: "Contestant **underperforms** baseline."
-
-## Step 8: Save Results (ALWAYS)
-
-**Output path logic:**
-```python
-if theme == "control":
-    base_path = f"internal/results/baselines/{scenario_name}/{effective_role}/"
-elif cross_role:
-    # Cross-role: include character slug for clarity
-    character_slug = slugify(character_name)  # e.g., "prospero", "granny-weatherwax"
-    base_path = f"internal/results/benchmarks/{scenario_name}/{theme}-{character_slug}-as-{effective_role}/"
-else:
-    base_path = f"internal/results/benchmarks/{scenario_name}/{theme}-{effective_role}/"
-```
-
-**Cross-role examples:**
-- `/benchmark shakespeare prospero --as dev` → `internal/results/benchmarks/{scenario}/shakespeare-prospero-as-dev/`
-- `/benchmark discworld granny --as dev` → `internal/results/benchmarks/{scenario}/discworld-granny-weatherwax-as-dev/`
-
-**Save structure:**
-```
-{base_path}/
-├── runs/
-│   ├── run_1.json
-│   ├── judge_1.json
-│   └── ...
-└── summary.yaml
-```
-
-**summary.yaml format:** See `/solo` command Step 10. For cross-role runs, include:
-```yaml
-agent:
-  theme: {theme}
-  character: {character_name}
-  source_role: {source_role}      # where character normally lives (e.g., sm)
-  effective_role: {effective_role}  # what they're doing (e.g., dev)
-  cross_role: true
-```
-
-**REQUIRED: Capture Pennyfarthing version in metadata:**
-```bash
-# Get version from package.json
-version=$(node -p "require('./package.json').version")
-```
-
-Include in summary.yaml:
-```yaml
-metadata:
-  created_at: "{ISO timestamp}"
-  pennyfarthing_version: "{version}"  # REQUIRED for baseline staleness detection
-  model: sonnet
-```
-
-**ALWAYS save summary.yaml, even for n=1.** This ensures consistent data structure for analysis.
-
-Display:
-```
-✓ Saved {n} run(s) to {base_path}
-✓ Summary: {base_path}/summary.yaml
-```
-</on-invoke>
-
-<error-handling>
-**Baseline not found:**
-```markdown
-Error: No baseline found for scenario '{scenario_name}' with agent type '{agent_type}'.
-
-To create a baseline, run:
-/benchmark-control --scenario {scenario_name}
-```
-
-**Invalid contestant spec:**
-```markdown
-Error: Invalid contestant format. Expected 'theme:agent', got '{value}'.
-
-Examples:
-- discworld:reviewer
-- princess-bride:dev
-- control:sm
-```
-
-**Missing --scenario:**
-```markdown
-Error: --scenario is required.
-
-Usage: /benchmark <theme:agent> --scenario <name> [--runs N]
-```
-
-**Invalid runs value:**
-```markdown
-Error: --runs must be between 1 and 20. Got: {value}
-```
-</error-handling>
-
-<reference>
-- Solo Command: `.claude/project/commands/solo.md`
-- Establish Baseline: `.claude/project/commands/benchmark-control.md`
-- Effect Size: Cohen's d standard interpretation (0.2 small, 0.5 medium, 0.8 large)
-- Baselines: `internal/results/baselines/{scenario}/{role}/` (control theme)
-- Benchmarks: `internal/results/benchmarks/{scenario}/{theme}-{role}/` (all other themes)
-- Results README: `internal/results/README.md`
-</reference>