@pennyfarthing/core 10.0.1 → 10.0.3

package/pennyfarthing-dist/skills/dev-patterns/SKILL.md

@@ -0,0 +1,461 @@
+---
+name: dev-patterns
+description: Common development patterns, fixes, and gotchas. Use when implementing features, debugging issues, or avoiding known pitfalls.
+allowed_tools: [Read, Glob, Grep, Task]
+---
+
+<run>
+Reference this skill when implementing features, debugging issues, or avoiding known pitfalls in development. Use these patterns to guide:
+- Working directory management in bash commands
+- TypeScript type system decisions
+- Error handling in Go
+- Test structure and isolation
+- HTTP status code selection
+- Turn-efficient coding practices
+- Background task execution
+
+The patterns capture common gotchas and solutions across multiple languages and frameworks.
+</run>
+
+<output>
+Following these patterns produces:
+- Reliable bash commands that work regardless of current directory
+- TypeScript builds that pass verbatimModuleSyntax validation
+- Go code with proper error handling and validation
+- Well-isolated, maintainable test suites
+- Consistent HTTP API behavior
+- Faster task completion through turn-efficient techniques
+- Properly tracked background operations
+</output>
+
+# Dev Patterns Skill
+
+## Overview
+
+This skill captures common development patterns, fixes, and gotchas that apply across projects using the pennyfarthing agent framework.
+
+## Critical Patterns
+
+### Bash Tool Working Directory
+
+**Problem:** The Bash tool maintains a persistent working directory. Relative `cd` commands fail when already in a different directory.
+
+**Symptoms:**
+- `cd API && just test` fails with "no such file or directory"
+- Commands fail because assuming wrong directory
+
+**Solution:** Always use absolute paths with `$CLAUDE_PROJECT_DIR`:
+
+```bash
+# WRONG - relative cd fails if you're already somewhere else
+cd API && just test
+
+# CORRECT - absolute path always works
+cd $CLAUDE_PROJECT_DIR/API && just test
+```
+
+**Best Practice:** Explicit `cd` with absolute path in every Bash call:
+
+```bash
+# Single-repo: from anywhere - always works
+cd $CLAUDE_PROJECT_DIR && git push -u origin feat/branch
+cd $CLAUDE_PROJECT_DIR && just test
+
+# Multi-repo: use repo-utils.sh for dynamic lookup
+source $CLAUDE_PROJECT_DIR/scripts/repo-utils.sh
+cd $CLAUDE_PROJECT_DIR/$(get_repo_path "myrepo") && just test
+```
+
+### TypeScript Type Imports
+
+**Problem:** Build fails with `verbatimModuleSyntax` errors.
+
+**Solution:** Use `import type` for type-only imports:
+
+```typescript
+// WRONG - causes build errors
+import { User, Ticket } from '@/types';
+
+// CORRECT
+import type { User, Ticket } from '@/types';
+```
+
+### Go Error Handling
+
+**Problem:** Silent failures when errors aren't checked.
+
+**Solution:** Always handle errors explicitly:
+
+```go
+// WRONG - ignores error
+result, _ := doSomething()
+
+// CORRECT
+result, err := doSomething()
+if err != nil {
+    return fmt.Errorf("doSomething failed: %w", err)
+}
+```
+
+### UUID Parsing
+
+**Problem:** Invalid UUIDs cause panics or unexpected behavior.
+
+**Solution:** Always validate UUID parsing:
+
+```go
+// WRONG - panics on invalid UUID
+id := uuid.MustParse(rawID)
+
+// CORRECT
+id, err := uuid.Parse(rawID)
+if err != nil {
+    return fmt.Errorf("invalid UUID: %w", err)
+}
+```
+
+## HTTP Status Codes
+
+Use appropriate status codes consistently:
+
+| Code | When to Use |
+|------|-------------|
+| `200 OK` | Successful GET, PUT, PATCH |
+| `201 Created` | Successful POST creating new resource |
+| `204 No Content` | Successful DELETE |
+| `400 Bad Request` | Invalid input (validation errors) |
+| `401 Unauthorized` | Missing or invalid authentication |
+| `403 Forbidden` | Authenticated but not authorized |
+| `404 Not Found` | Resource doesn't exist |
+| `409 Conflict` | Resource state conflict (duplicate) |
+| `500 Internal Server Error` | Server-side error (always log it!) |
+
+<details>
+<summary><strong>Test Patterns</strong></summary>
+
+### Test Isolation
+
+Each test should:
+1. Set up its own state
+2. Clean up after itself
+3. Not depend on other tests' state
+
+```go
+func TestCreate(t *testing.T) {
+    // Setup
+    db := testutil.NewTestDB(t)
+    defer db.Cleanup()
+
+    // Test
+    result, err := Create(db, input)
+
+    // Assert
+    require.NoError(t, err)
+    assert.Equal(t, expected, result)
+}
+```
+
+### Table-Driven Tests
+
+```go
+func TestValidate(t *testing.T) {
+    tests := []struct {
+        name    string
+        input   string
+        wantErr bool
+    }{
+        {"valid input", "good", false},
+        {"empty input", "", true},
+        {"too long", strings.Repeat("x", 1000), true},
+    }
+
+    for _, tt := range tests {
+        t.Run(tt.name, func(t *testing.T) {
+            err := Validate(tt.input)
+            if tt.wantErr {
+                assert.Error(t, err)
+            } else {
+                assert.NoError(t, err)
+            }
+        })
+    }
+}
+```
+
+</details>
+
+## Common Gotchas
+
+1. **Working directory** - Always use `$CLAUDE_PROJECT_DIR` for absolute paths
+2. **Type imports** - Use `import type` for TypeScript types
+3. **Error handling** - Never ignore errors in Go
+4. **UUID parsing** - Always validate, never panic
+5. **Test isolation** - Each test manages its own state
+
+## Turn-Efficient Patterns
+
+Minimize API round-trips by batching operations and parallelizing where possible. Each tool call is a "turn" - fewer turns means faster completion.
+
+### Batch Bash Commands
+
+Chain related commands with `&&` instead of separate calls:
+
+```bash
+# INEFFICIENT - 3 turns
+git status
+git branch --show-current
+git log -1 --oneline
+
+# EFFICIENT - 1 turn
+git status && git branch --show-current && git log -1 --oneline
+```
+
+For git workflows, batch the entire operation:
+
+```bash
+# INEFFICIENT - 3 turns
+git add .
+git commit -m "feat: add feature"
+git push -u origin $(git branch --show-current)
+
+# EFFICIENT - 1 turn
+git add . && git commit -m "feat: add feature" && git push -u origin $(git branch --show-current)
+```
+
+### Parallel Tool Calls
+
+When operations are independent, invoke multiple tools in the same turn:
+
+```
+# INEFFICIENT - 3 sequential turns
+Turn 1: Read file A
+Turn 2: Read file B
+Turn 3: Read file C
+
+# EFFICIENT - 1 parallel turn
+Turn 1: Read file A, Read file B, Read file C (parallel)
+```
+
+**When to parallelize:**
+- Reading multiple independent files
+- Spawning subagents that don't depend on each other's results
+- Running Glob and Grep searches simultaneously
+- Fetching multiple web resources
+
+**When NOT to parallelize:**
+- Operations where later calls depend on earlier results
+- Writes that might conflict
+- Sequential logic (must read before edit)
+
+### Smart File Operations
+
+Use Grep's internal file reading instead of Read + manual search:
+
+```bash
+# INEFFICIENT - 2 turns
+Turn 1: Read entire file
+Turn 2: (mentally search for pattern)
+
+# EFFICIENT - 1 turn with context
+Grep pattern with -C 5 for context
+```
+
+Use Glob with brace expansion for multiple patterns:
+
+```bash
+# INEFFICIENT - 2 turns
+Glob: **/*.ts
+Glob: **/*.tsx
+
+# EFFICIENT - 1 turn
+Glob: **/*.{ts,tsx}
+```
+
+<details>
+<summary><strong>Compound Subagents</strong></summary>
+
+Design subagents to do multiple related steps in one spawn:
+
+```yaml
+# INEFFICIENT - 2 subagent spawns
+Spawn 1: workflow-status-check
+Spawn 2: sm-file-summary
+
+# EFFICIENT - 1 compound subagent
+Spawn 1: (combined status-check + file-summary subagent)
+```
+
+**Good candidates for combining:**
+- Preflight checks + handoff updates
+- Status gathering + session file updates
+- Test running + result parsing
+
+**Creating Compound Subagents:**
+
+When you notice a pattern of sequential subagent calls, consider creating a compound subagent:
+
+1. **Identify the pattern:** Look for repeated sequences like "always call A then B"
+2. **Check dependencies:** Can B start before A completes? If yes, parallelize instead.
+3. **Create combined prompt:** Merge both subagents' responsibilities into one
+
+**Example: Compound Preflight + Handoff**
+
+Instead of:
+```yaml
+# Turn 1: Preflight
+Task tool:
+  subagent_type: "reviewer-preflight"
+  prompt: |
+    STORY_ID: X-Y
+    ...
+
+# Turn 2: Wait for result, then handoff
+Task tool:
+  subagent_type: "reviewer-handoff-approve"
+  prompt: |
+    STORY_ID: X-Y
+    ...
+```
+
+Create a compound definition that does both:
+```yaml
+Task tool:
+  subagent_type: "general-purpose"
+  model: "haiku"
+  prompt: |
+    You are a combined preflight + handoff assistant.
+
+    ## Phase 1: Pre-flight checks
+    [Include preflight steps]
+
+    ## Phase 2: If all checks pass, complete handoff
+    [Include handoff steps]
+
+    ## Output
+    Report: preflight results + handoff status
+```
+
+**When NOT to combine:**
+- When steps need human review between them (e.g., reviewer assessment before handoff)
+- When the first step might fail often (better to fail fast)
+- When prompts are already long (context limits)
+
+</details>
+
+### Turn Budget Guidelines
+
+| Story Size | Target Turns |
+|------------|--------------|
+| 1 point (trivial) | 10-15 |
+| 2 points (small) | 15-25 |
+| 3 points (medium) | 25-35 |
+| 5 points (large) | 35-50 |
+
+**Techniques to reduce turns:**
+1. Batch bash commands with `&&`
+2. Read multiple files in parallel
+3. Use Grep with context instead of Read + search
+4. Combine related subagent work
+5. Use brace expansion in Glob patterns
+6. Spawn independent subagents in parallel
+7. Run tests in background while continuing work
+
+<details>
+<summary><strong>Background Subagent Execution</strong></summary>
+
+Run slow operations in background to continue working:
+
+```yaml
+# Spawn tests in background
+Task tool:
+  subagent_type: "testing-runner"
+  run_in_background: true
+  prompt: |
+    REPOS: all
+    CONTEXT: Background test run
+    RUN_ID: bg-001
+    SKIP_CACHE_WRITE: true
+```
+
+**Check status without blocking:**
+```yaml
+TaskOutput tool:
+  task_id: {task_id}
+  block: false
+  timeout: 1000
+```
+
+**When to use background execution:**
+- Test runs while writing more code
+- Multiple independent file searches
+- Parallel exploration of code paths
+- Long-running git operations
+
+**When NOT to use:**
+- Operations where next steps depend on the result
+- Operations that modify shared state
+- Before commit (need to verify GREEN)
+- During handoff (need synchronous verification)
+
+</details>
+
+<details>
+<summary><strong>Background Task Tracking</strong></summary>
+
+Use the background task tracking utilities to manage session file entries:
+
+```bash
+source $CLAUDE_PROJECT_DIR/.pennyfarthing/scripts/lib/background-tasks.sh
+SESSION_FILE="$CLAUDE_PROJECT_DIR/.session/${STORY_ID}-session.md"
+
+# After spawning background task, record it:
+bg_task_add "$SESSION_FILE" "$TASK_ID" "testing-runner" "Background test run"
+
+# After checking TaskOutput, update status:
+bg_task_update "$SESSION_FILE" "$TASK_ID" "completed"  # or "error"
+
+# Clean up finished tasks:
+bg_task_cleanup "$SESSION_FILE"
+```
+
+**Available functions:**
+| Function | Purpose |
+|----------|---------|
+| `bg_task_add` | Record new background task |
+| `bg_task_update` | Update task status (running/completed/error) |
+| `bg_task_cleanup` | Remove completed and errored tasks |
+| `bg_task_list` | Show all running tasks |
+| `bg_task_check` | Return 0 if any tasks running (for conditionals) |
+| `bg_task_summary` | Print counts by status |
+
+**Lifecycle:**
+1. Spawn with `run_in_background: true`
+2. Record: `bg_task_add "$SESSION_FILE" "$TASK_ID" "type" "description"`
+3. Continue other work
+4. Periodically check `TaskOutput` with `block: false`
+5. Update: `bg_task_update "$SESSION_FILE" "$TASK_ID" "completed"`
+6. Cleanup: `bg_task_cleanup "$SESSION_FILE"`
+
+</details>
+
+<details>
+<summary><strong>Example: Background Tests While Implementing</strong></summary>
+
+```
+Turn 1: Spawn testing-runner in background, continue editing
+Turn 2: Make code changes
+Turn 3: Check TaskOutput (still running), continue editing
+Turn 4: Make more changes
+Turn 5: Check TaskOutput (complete, GREEN), proceed to commit
+```
+
+This pattern saves turns by overlapping test execution with implementation work.
+
+</details>
+
+## Project Customization
+
+Projects should create their own dev-patterns skill in `.claude/project/skills/dev-patterns/` with:
+- Project-specific patterns and conventions
+- Known pitfalls for the tech stack
+- File location conventions

package/pennyfarthing-dist/skills/finalize-run/SKILL.md

@@ -0,0 +1,261 @@
+---
+name: finalize-run
+description: Validate and save benchmark run results. Use when completing a benchmark run, validating results before storage, or ensuring all runs pass through the single guardrail exit point.
+---
+
+# Finalize Run Skill
+
+<run>Validates and saves benchmark run results</run>
+<output>JSON with validation success status and saved file path</output>
+
+All runs MUST pass through this skill before saving. This is the guardrail.
+
+## Invocation
+
+```
+/finalize-run --type <type> --data <json>
+```
+
+**Types:**
+- `solo` - Single agent evaluation
+- `duel` - Two-agent comparison
+- `relay` - Team relay competition
+
+## Validation Rules
+
+### Agent Validation
+
+For EACH agent in the run:
+
+| Field | Rule | Action on Fail |
+|-------|------|----------------|
+| `cli_timestamp` | Valid ISO8601 | REJECT |
+| `response_text` | ≥ 200 characters | REJECT |
+| `input_tokens` | > 0 | REJECT |
+| `output_tokens` | > 0 | REJECT |
+
+### Judge Validation
+
+| Field | Rule | Action on Fail |
+|-------|------|----------------|
+| `cli_timestamp` | Valid ISO8601 | REJECT |
+| `response_text` | Contains "WEIGHTED_TOTAL" or "RATING:" | REJECT |
+| `response_text` | ≥ 100 characters | REJECT |
+
+### Score Validation
+
+| Field | Rule | Action on Fail |
+|-------|------|----------------|
+| `total` | Number 1-100 | REJECT |
+| Extracted from judge | Matches claimed score | REJECT |
+
+### Timestamp Sanity
+
+```
+elapsed = last_timestamp - first_timestamp
+minimum = 30 × number_of_agents
+
+if elapsed < minimum:
+  WARN: "Timestamps suspiciously close"
+```
+
+## On Invoke
+
+### Step 1: Parse Input
+
+Extract:
+- `type`: solo, duel, or relay
+- `data`: JSON with run data
+
+**Required data structure:**
+
+```json
+{
+  "type": "solo|duel|relay",
+  "timestamp": "ISO8601",
+  "scenario": {"name": "...", "title": "..."},
+  "agents": [
+    {
+      "spec": "theme:agent",
+      "cli_timestamp": "ISO8601",
+      "response_text": "full response",
+      "input_tokens": 1234,
+      "output_tokens": 5678
+    }
+  ],
+  "judge": {
+    "cli_timestamp": "ISO8601",
+    "response_text": "full verdict",
+    "input_tokens": 2345,
+    "output_tokens": 890
+  },
+  "scores": {"spec": score},
+  "output_path": "results/..."
+}
+```
+
+### Step 2: Validate Agents
+
+For each agent:
+
+```bash
+# Check timestamp format
+if ! [[ "$cli_timestamp" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}T ]]; then
+  REJECT "Invalid timestamp: $cli_timestamp"
+fi
+
+# Check response length
+response_len=${#response_text}
+if [[ $response_len -lt 200 ]]; then
+  REJECT "Response too short: $response_len chars (min 200)"
+fi
+
+# Check tokens
+if [[ $input_tokens -le 0 ]] || [[ $output_tokens -le 0 ]]; then
+  REJECT "Invalid tokens: in=$input_tokens out=$output_tokens"
+fi
+```
+
+### Step 3: Validate Judge
+
+```bash
+# Check for score marker
+if ! echo "$judge_response" | grep -qE "WEIGHTED_TOTAL|RATING:"; then
+  REJECT "Judge response missing score marker"
+fi
+
+# Check response length
+if [[ ${#judge_response} -lt 100 ]]; then
+  REJECT "Judge response too short"
+fi
+
+# Check tokens
+if [[ $judge_input_tokens -le 0 ]] || [[ $judge_output_tokens -le 0 ]]; then
+  REJECT "Invalid judge tokens"
+fi
+```
+
+### Step 4: Validate Scores
+
+```bash
+# Extract score from judge
+extracted=$(echo "$judge_response" | grep -oE "WEIGHTED_TOTAL[^0-9]*([0-9]+)" | grep -oE "[0-9]+" | tail -1)
+
+# Verify against claimed score
+if [[ "$extracted" != "$claimed_score" ]]; then
+  REJECT "Score mismatch: claimed=$claimed_score extracted=$extracted"
+fi
+
+# Check range
+if [[ $extracted -lt 1 ]] || [[ $extracted -gt 100 ]]; then
+  REJECT "Score out of range: $extracted"
+fi
+```
+
+### Step 5: Timestamp Sanity Check
+
+```bash
+# Calculate elapsed time
+first_ts=$(date -d "$first_agent_timestamp" +%s)
+last_ts=$(date -d "$judge_timestamp" +%s)
+elapsed=$((last_ts - first_ts))
+
+# Check minimum expected time
+num_agents=${#agents[@]}
+minimum=$((30 * num_agents))
+
+if [[ $elapsed -lt $minimum ]]; then
+  WARN "Elapsed time ${elapsed}s < expected ${minimum}s"
+fi
+```
+
+### Step 6: Display Validation Report
+
+```markdown
+### Finalize Run Validation
+
+**Type:** {type}
+**Scenario:** {scenario.name}
+
+#### Agent Validation
+| Agent | Timestamp | Response | Tokens | Status |
+|-------|-----------|----------|--------|--------|
+| {spec} | {ts} | {len} chars | in={in} out={out} | ✓ |
+
+#### Judge Validation
+| Field | Value | Status |
+|-------|-------|--------|
+| Timestamp | {ts} | ✓ |
+| Response | {len} chars | ✓ |
+| Score | {score}/100 | ✓ |
+
+#### Timestamp Sanity
+- Elapsed: {elapsed}s
+- Expected: ≥{minimum}s
+- Status: ✓ PASS
+
+---
+**VALIDATION: PASSED**
+```
+
+### Step 7: Save Results
+
+If ALL validations pass:
+
+```bash
+# Ensure directory exists
+mkdir -p "$(dirname "$output_path")"
+
+# Write result
+echo "$result_json" > "$output_path"
+```
+
+Display:
+```
+✓ Saved to {output_path}
+```
+
+### Step 8: Return Success
+
+```json
+{
+  "success": true,
+  "path": "{output_path}",
+  "validation": {
+    "agents_validated": {count},
+    "judge_validated": true,
+    "scores_verified": true,
+    "timestamp_sane": true
+  }
+}
+```
+
+## On Validation Failure
+
+```markdown
+### ❌ VALIDATION FAILED
+
+**Failed Check:** {which validation}
+**Reason:** {specific reason}
+**Value:** {what was provided}
+
+**This run will NOT be saved.**
+
+To fix:
+- {remediation steps}
+```
+
+Return:
+```json
+{
+  "success": false,
+  "error": "{reason}",
+  "failed_check": "{which}"
+}
+```
+
+## The Golden Rule
+
+**Real data or no data.**
+
+Never estimate. Never fabricate. If validation fails, the run did not happen.