@pennyfarthing/core 6.6.0

package/pennyfarthing-dist/skills/persona-benchmark/SKILL.md

@@ -0,0 +1,173 @@
+# Persona Benchmark Skill
+
+Run benchmarks to compare persona effectiveness.
+
+## Usage
+
+```
+/persona-benchmark <test-case-id> <persona>
+/persona-benchmark cr-001 discworld
+/persona-benchmark tw-001 literary-classics
+/persona-benchmark --suite              # Run all tests, all personas
+/persona-benchmark --analyze            # Analyze collected results
+```
+
+## Benchmark Execution Protocol
+
+### Step 1: Load Test Case
+
+Read the test case from `.claude/benchmarks/test-cases/{category}/{id}.yaml`
+
+Extract:
+- `instructions` - What to give the agent
+- `code` - The code/problem to analyze
+- `known_issues` or `known_edge_cases` or `known_considerations` - DO NOT reveal to agent
+
+### Step 2: Configure Persona
+
+Temporarily set persona in `.claude/persona-config.yaml`:
+```yaml
+theme: {persona}
+```
+
+### Step 3: Execute Task
+
+Invoke the appropriate agent:
+- `code-review` → `/reviewer`
+- `test-writing` → `/tea`
+- `architecture` → `/architect`
+
+Provide ONLY:
+- The instructions
+- The code/problem
+
+Do NOT reveal known issues list.
+
+### Step 4: Collect Results
+
+After agent completes, score the output:
+
+**Quantitative Scoring:**
+```yaml
+# For code review:
+issues_found: [list of issues detected]
+issues_matched: [map to known_issues ids]
+detection_rate: issues_matched / total_known_issues
+false_positives: issues not in known list
+
+# For test writing:
+edge_cases_found: [list of edge cases covered]
+edge_cases_matched: [map to known_edge_cases ids]
+coverage_rate: edge_cases_matched / total_known_edge_cases
+
+# For architecture:
+considerations_found: [list of considerations mentioned]
+considerations_matched: [map to known_considerations ids]
+completeness_rate: considerations_matched / total_known_considerations
+```
+
+**Qualitative Scoring (1-5):**
+- `persona_consistency`: Did the agent stay in character?
+- `explanation_quality`: How well did it explain its findings?
+- `actionability`: How usable is the output?
+- `engagement`: How enjoyable was the interaction?
+
+### Step 5: Save Results
+
+Write to `.claude/benchmarks/results/{timestamp}-{persona}-{test-case-id}.yaml`:
+
+```yaml
+benchmark:
+  test_case: cr-001
+  persona: discworld
+  agent: reviewer
+  character: Granny Weatherwax
+  timestamp: 2024-01-15T10:30:00Z
+
+quantitative:
+  items_found: 8
+  items_expected: 14
+  items_matched:
+    - SQL_INJECTION_1
+    - SQL_INJECTION_2
+    - PLAINTEXT_PASSWORD
+    - PASSWORD_EXPOSURE_1
+    - PASSWORD_EXPOSURE_2
+    - NO_AUTH_CHECK
+    - ASYNC_DELETE_NO_TX
+    - ROWS_NOT_CLOSED
+  detection_rate: 0.57
+  false_positives: 1
+  weighted_score: 15.5
+  max_weighted_score: 22.5
+  weighted_rate: 0.69
+
+qualitative:
+  persona_consistency: 5
+  explanation_quality: 4
+  actionability: 4
+  engagement: 5
+
+notes: |
+  Found both SQL injections immediately with strong language.
+  Missed the error handling issues.
+  Very much in character - "I aten't reviewing code that's already dead."
+
+raw_output: |
+  [Full agent output preserved here]
+```
+
+## Analysis Mode
+
+When run with `--analyze`:
+
+1. Load all results from `.claude/benchmarks/results/`
+
+2. Aggregate by persona:
+```
+| Persona          | Detection Rate | False Pos | Persona Score | Engagement |
+|------------------|----------------|-----------|---------------|------------|
+| discworld        | 0.71           | 1.2       | 4.8           | 4.9        |
+| star-trek        | 0.68           | 0.8       | 4.5           | 4.2        |
+| literary-classics| 0.73           | 1.5       | 4.2           | 4.0        |
+| minimalist       | 0.65           | 0.5       | N/A           | 3.2        |
+```
+
+3. Aggregate by test category:
+```
+| Category     | Best Persona      | Avg Detection |
+|--------------|-------------------|---------------|
+| code-review  | literary-classics | 0.71          |
+| test-writing | discworld         | 0.68          |
+| architecture | star-trek         | 0.75          |
+```
+
+4. Statistical significance:
+- Calculate standard deviation
+- Note if differences are significant
+
+5. Qualitative patterns:
+- Which personas stay in character best?
+- Which provide most actionable output?
+- User enjoyment patterns
+
+## Running a Full Suite
+
+```bash
+# This will take a while - runs each test case with each persona
+/persona-benchmark --suite
+```
+
+Executes:
+- All test cases in `test-cases/`
+- With each persona: discworld, star-trek, literary-classics, minimalist
+- Saves individual results
+- Produces summary comparison
+
+## Tips for Valid Benchmarks
+
+1. **Same evaluator**: Same person should score qualitative metrics
+2. **Blind evaluation**: Score output before checking which persona
+3. **Multiple runs**: Run each test 3+ times for reliability
+4. **Fresh context**: Start new session for each benchmark run
+5. **Control variables**: Same time of day, same evaluator state

package/pennyfarthing-dist/skills/skill-registry.schema.json

@@ -0,0 +1,102 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "$id": "https://pennyfarthing.dev/schemas/skill-registry.schema.json",
+  "title": "Pennyfarthing Skill Registry",
+  "description": "Schema for the centralized skill metadata registry",
+  "type": "object",
+  "properties": {
+    "version": {
+      "type": "string",
+      "pattern": "^\\d+\\.\\d+\\.\\d+$",
+      "description": "Semantic version of the registry format"
+    },
+    "skills": {
+      "type": "object",
+      "description": "Map of skill names to their metadata",
+      "additionalProperties": {
+        "$ref": "#/definitions/skill"
+      }
+    }
+  },
+  "required": ["version", "skills"],
+  "additionalProperties": false,
+  "definitions": {
+    "skill": {
+      "type": "object",
+      "properties": {
+        "name": {
+          "type": "string",
+          "description": "Canonical name of the skill (matches directory name)"
+        },
+        "description": {
+          "type": "string",
+          "description": "Brief description for discovery and help text"
+        },
+        "category": {
+          "type": "string",
+          "enum": [
+            "development",
+            "project-management",
+            "theming",
+            "tools",
+            "ai-llm",
+            "documentation",
+            "benchmarking"
+          ],
+          "description": "Primary category for grouping skills"
+        },
+        "tags": {
+          "type": "array",
+          "items": { "type": "string" },
+          "description": "Searchable tags for discovery"
+        },
+        "version": {
+          "type": "string",
+          "pattern": "^\\d+\\.\\d+\\.\\d+$",
+          "description": "Skill version (semantic versioning)"
+        },
+        "prerequisites": {
+          "type": "array",
+          "items": { "type": "string" },
+          "description": "Other skills or tools required before using this skill"
+        },
+        "examples": {
+          "type": "array",
+          "items": {
+            "type": "object",
+            "properties": {
+              "context": {
+                "type": "string",
+                "description": "When to use this invocation"
+              },
+              "invocation": {
+                "type": "string",
+                "description": "How to invoke the skill"
+              }
+            },
+            "required": ["context", "invocation"],
+            "additionalProperties": false
+          },
+          "description": "Example invocations with context"
+        },
+        "anti_patterns": {
+          "type": "array",
+          "items": { "type": "string" },
+          "description": "Common mistakes or misuses to avoid"
+        },
+        "related_skills": {
+          "type": "array",
+          "items": { "type": "string" },
+          "description": "Skills that complement or relate to this one"
+        },
+        "keywords": {
+          "type": "array",
+          "items": { "type": "string" },
+          "description": "Additional searchable terms for discovery"
+        }
+      },
+      "required": ["name", "description", "category", "tags"],
+      "additionalProperties": false
+    }
+  }
+}

package/pennyfarthing-dist/skills/skill-registry.yaml

@@ -0,0 +1,335 @@
+# Pennyfarthing Skill Registry
+# Central metadata for all built-in skills
+# Schema: skill-registry.schema.json
+
+version: "1.0.0"
+
+skills:
+  agentic-patterns:
+    name: agentic-patterns
+    description: Core reasoning patterns for building effective LLM agents
+    category: ai-llm
+    tags: [reasoning, patterns, agents, llm]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Designing agent behavior
+        invocation: /agentic-patterns
+      - context: Debugging agent failures
+        invocation: /agentic-patterns troubleshooting
+    anti_patterns:
+      - Don't apply patterns mechanically without understanding context
+    related_skills: [context-engineering]
+    keywords: [react, reflection, planning, tool-use, chain-of-thought]
+
+  changelog:
+    name: changelog
+    description: Maintain changelogs following Keep a Changelog format with conventional commits
+    category: documentation
+    tags: [changelog, releases, versioning]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Creating release notes
+        invocation: /changelog
+      - context: Auto-generating from commits
+        invocation: /changelog generate
+    anti_patterns:
+      - Don't manually edit CHANGELOG.md entries after generation
+    related_skills: []
+    keywords: [keepachangelog, conventional-commits, semver, release-notes]
+
+  code-review:
+    name: code-review
+    description: Code review checklists and patterns for quality assurance
+    category: development
+    tags: [review, quality, checklist]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Self-review before commit
+        invocation: /code-review
+      - context: Reviewing a PR
+        invocation: /code-review pr-checklist
+    anti_patterns:
+      - Don't skip security considerations in review
+      - Don't approve without running tests
+    related_skills: [testing, dev-patterns]
+    keywords: [pr, pull-request, quality, linting, security]
+
+  context-engineering:
+    name: context-engineering
+    description: Strategies for managing context windows in long-running agent sessions
+    category: ai-llm
+    tags: [context, optimization, agents]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Approaching context limits
+        invocation: /context-engineering
+      - context: Designing subagent prompts
+        invocation: /context-engineering subagent-design
+    anti_patterns:
+      - Don't load unnecessary context upfront
+      - Don't repeat large code blocks in prompts
+    related_skills: [agentic-patterns]
+    keywords: [context-window, tokens, summarization, lazy-loading]
+
+  cyclist:
+    name: cyclist
+    description: Launch Cyclist visual terminal for Claude Code monitoring
+    category: tools
+    tags: [visual, monitoring, terminal]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Starting visual monitor
+        invocation: /cyclist
+      - context: Debugging sessions
+        invocation: /cyclist debug
+    anti_patterns:
+      - Don't run Cyclist in headless environments
+    related_skills: []
+    keywords: [tui, terminal, visualization, dashboard]
+
+  dev-patterns:
+    name: dev-patterns
+    description: Common development patterns, fixes, and gotchas
+    category: development
+    tags: [patterns, debugging, implementation]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Implementing features
+        invocation: /dev-patterns
+      - context: Avoiding known pitfalls
+        invocation: /dev-patterns gotchas
+    anti_patterns:
+      - Don't apply patterns from other frameworks blindly
+    related_skills: [code-review, testing]
+    keywords: [implementation, debugging, fixes, best-practices]
+
+  finalize-run:
+    name: finalize-run
+    description: Validate and save benchmark run results - single exit point for all runs
+    category: benchmarking
+    tags: [benchmark, validation, results]
+    version: "1.0.0"
+    prerequisites: [judge]
+    examples:
+      - context: Completing a benchmark run
+        invocation: /finalize-run
+    anti_patterns:
+      - Never save results without passing through this skill
+    related_skills: [judge, persona-benchmark]
+    keywords: [benchmark, save, validate, guardrail]
+
+  jira:
+    name: jira
+    description: Jira CLI commands for sprint management
+    category: project-management
+    tags: [jira, issues, sprint]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Viewing sprint issues
+        invocation: /jira sprint
+      - context: Assigning issues
+        invocation: /jira assign ISSUE-123
+    anti_patterns:
+      - Don't bypass Jira for sprint tracking
+    related_skills: [sprint-context, story-management]
+    keywords: [atlassian, issues, tickets, backlog]
+
+  judge:
+    name: judge
+    description: Evaluate agent responses using standardized rubrics
+    category: benchmarking
+    tags: [evaluation, rubrics, scoring]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Evaluating agent response
+        invocation: /judge evaluate
+      - context: Running benchmark comparison
+        invocation: /judge compare
+    anti_patterns:
+      - Don't modify rubrics during a benchmark run
+    related_skills: [finalize-run, persona-benchmark]
+    keywords: [evaluation, scoring, metrics, grading]
+
+  just:
+    name: just
+    description: Run just recipes for project tasks like dev servers, tests, and databases
+    category: tools
+    tags: [tasks, runner, automation]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Starting dev servers
+        invocation: /just dev
+      - context: Running database tasks
+        invocation: /just db-reset
+    anti_patterns:
+      - Don't create complex recipes - keep them simple and composable
+    related_skills: [testing]
+    keywords: [justfile, make, tasks, automation, command-runner]
+
+  mermaid:
+    name: mermaid
+    description: Generate diagrams using Mermaid syntax for documentation
+    category: tools
+    tags: [diagrams, visualization, documentation]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Creating architecture diagrams
+        invocation: /mermaid architecture
+      - context: Sequence diagrams
+        invocation: /mermaid sequence
+    anti_patterns:
+      - Don't create overly complex diagrams - split into multiple if needed
+    related_skills: [changelog]
+    keywords: [flowchart, sequence, er-diagram, gantt, class-diagram]
+
+  permissions:
+    name: permissions
+    description: Manage runtime permission grants - list, grant, and revoke tool access
+    category: project-management
+    tags: [permissions, security, tools]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Viewing active grants
+        invocation: /permissions
+      - context: Granting WebFetch access
+        invocation: /permissions grant WebFetch "*.github.com"
+      - context: Revoking tool access
+        invocation: /permissions revoke Bash
+    anti_patterns:
+      - Don't manually edit settings.local.json permissions - use skill
+      - Don't grant overly broad scope patterns
+    related_skills: [sprint-context]
+    keywords: [grants, scopes, tools, runtime, security]
+
+  persona-benchmark:
+    name: persona-benchmark
+    description: Run benchmarks to compare persona effectiveness
+    category: benchmarking
+    tags: [benchmark, personas, comparison]
+    version: "1.0.0"
+    prerequisites: [judge]
+    examples:
+      - context: Testing persona on code review
+        invocation: /persona-benchmark cr-001 discworld
+      - context: Testing tech writer persona
+        invocation: /persona-benchmark tw-001 literary-classics
+    anti_patterns:
+      - Don't compare results across different rubric versions
+    related_skills: [judge, finalize-run, theme]
+    keywords: [benchmark, comparison, personas, testing]
+
+  sprint-context:
+    name: sprint-context
+    description: Sprint status, backlog, and story management context
+    category: project-management
+    tags: [sprint, status, backlog]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Checking sprint status
+        invocation: /sprint-context
+      - context: Finding available stories
+        invocation: /sprint-context available
+    anti_patterns:
+      - Don't manually edit sprint YAML - use skills
+    related_skills: [story-management, jira]
+    keywords: [sprint, backlog, velocity, kanban]
+
+  story-management:
+    name: story-management
+    description: Story creation, sizing, and sprint workflow patterns
+    category: project-management
+    tags: [stories, sizing, workflow]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Creating new stories
+        invocation: /story-management create
+      - context: Sizing stories
+        invocation: /story-management size
+    anti_patterns:
+      - Don't create stories without acceptance criteria
+    related_skills: [sprint-context, jira]
+    keywords: [user-stories, estimation, points, acceptance-criteria]
+
+  testing:
+    name: testing
+    description: Test commands and TDD workflow patterns
+    category: development
+    tags: [tdd, testing, quality]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Running project tests
+        invocation: /testing
+      - context: Debugging test failures
+        invocation: /testing troubleshoot
+    anti_patterns:
+      - Don't run tests directly - use testing-runner subagent
+      - Don't skip RED phase in TDD workflow
+    related_skills: [dev-patterns, code-review]
+    keywords: [jest, vitest, pytest, unit-test, integration, tdd, red-green]
+
+  theme:
+    name: theme
+    description: Manage persona themes - list, show, and set active theme
+    category: theming
+    tags: [personas, themes, customization]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Listing available themes
+        invocation: /list-themes
+      - context: Showing current theme
+        invocation: /show-theme
+      - context: Setting active theme
+        invocation: /set-theme discworld
+    anti_patterns:
+      - Don't edit config.local.yaml directly - use skill
+    related_skills: [theme-creation]
+    keywords: [personas, characters, discworld, literary-classics]
+
+  theme-creation:
+    name: theme-creation
+    description: Create custom persona themes with AI-driven generation
+    category: theming
+    tags: [personas, themes, creation]
+    version: "1.0.0"
+    prerequisites: [theme]
+    examples:
+      - context: Creating new theme
+        invocation: /theme-maker
+      - context: Interactive theme wizard
+        invocation: /create-theme
+    anti_patterns:
+      - Don't create incomplete themes missing required agents
+    related_skills: [theme]
+    keywords: [personas, custom, generation, wizard]
+
+  yq:
+    name: yq
+    description: YAML processor for reading, modifying, and querying YAML files
+    category: tools
+    tags: [yaml, processing, cli]
+    version: "1.0.0"
+    prerequisites: []
+    examples:
+      - context: Reading YAML values
+        invocation: /yq read
+      - context: Modifying YAML files
+        invocation: /yq modify
+    anti_patterns:
+      - Don't use complex expressions without testing first
+    related_skills: [just]
+    keywords: [yaml, jq, parsing, query, mikefarah]