@pennyfarthing/core 7.5.0 → 7.6.0

package/pennyfarthing-dist/skills/otel/skill.md

@@ -0,0 +1,223 @@
+---
+name: otel
+description: Claude Code OTEL telemetry format documentation. Use when working with OTEL span interception, enrichment, or correlation in Cyclist.
+allowed_tools: [Read, Glob, Grep, Task]
+---
+
+# OTEL Skill - Claude Code Telemetry
+
+## Purpose
+
+Document the actual OTEL data Claude Code emits. This is ground truth, not speculation.
+
+**WARNING:** Do NOT assume fields exist. Check this document first.
+
+## Quick Start - Enable Debug Logging
+
+```bash
+# Option 1: Via just command
+just cyclist-electron true
+
+# Option 2: Via environment variable
+OTEL_DEBUG=true npm run dev
+
+# Option 3: Runtime toggle (in code or devtools)
+import { setOtelDebug } from './otlp-receiver.js';
+setOtelDebug(true);
+```
+
+Captures go to:
+- Console: `[OTEL-CAPTURE]` prefix
+- File: `/tmp/otel-capture.jsonl`
+
+View captures:
+```bash
+cat /tmp/otel-capture.jsonl | jq .
+# Or tail live:
+tail -f /tmp/otel-capture.jsonl | jq .
+```
+
+## Status
+
+**WORKING** - Stories 36-9, 36-10, 36-2, and 36-3 have established working enrichment patterns.
+
+## Known Facts (Verified)
+
+### LogRecord Structure
+
+Claude Code exports OTEL logs (not traces/spans). The logRecord structure:
+
+```typescript
+interface LogRecord {
+  timeUnixNano: string;       // Nanoseconds since epoch
+  observedTimeUnixNano: string;
+  body: { stringValue: string };  // Event name (e.g., "claude_code.tool_result")
+  attributes: Attribute[];    // Key-value pairs
+  droppedAttributesCount: number;
+  // NOTE: traceId and spanId are NOT present at logRecord level
+}
+```
+
+### What We Know Is MISSING
+
+- `traceId` - NOT in logRecord (Story 36-9 confirmed)
+- `spanId` - NOT in logRecord (Story 36-9 confirmed)
+- `tool_use_id` - NOT in OTEL attributes; use Claude message stream `block.id` instead
+
+### Event Types Observed
+
+| Event Name | Description |
+|------------|-------------|
+| `claude_code.tool_result` | Tool execution completed |
+| `claude_code.user_prompt` | User prompt submitted |
+| `claude_code.api_request` | API request made |
+
+### Tool Result Attributes (VERIFIED)
+
+```
+tool_name: string           // e.g., "Read", "Edit", "Bash"
+tool_parameters: string     // JSON string with tool-specific input
+tool_output: string         // Tool output (truncated to ~2000 chars)
+duration_ms: string|number  // Execution time in milliseconds
+success: string             // "true" or "false" as string
+error?: string              // Error message if failed
+```
+
+### tool_parameters Format by Tool (VERIFIED)
+
+**Read:**
+```json
+{ "file_path": "/absolute/path/to/file.ts" }
+```
+
+**Edit:**
+```json
+{
+  "file_path": "/absolute/path/to/file.ts",
+  "old_string": "text to replace",
+  "new_string": "replacement text"
+}
+```
+
+**Bash:**
+```json
+{
+  "command": "git status",
+  "description": "Check git status",
+  "timeout": 120000,
+  "run_in_background": false
+}
+```
+
+## Correlation Strategy (WORKING)
+
+Story 36-10 solved the correlation problem:
+
+1. Claude message stream provides `tool_use` with `block.id`, `block.name`, `block.input`
+2. Store in pending queue via `storePendingToolInput(toolId, toolName, input)`
+3. When OTEL arrives, match via `consumePendingToolInput(toolName, parsedToolParams)`
+4. **Key fix:** For Read/Edit, match on `file_path` first for precise correlation
+5. Fall back to FIFO by toolName for tools without file_path (e.g., Bash)
+6. Use `block.id` (toolId) as synthetic spanId for correlation map
+
+**Why this works:**
+- Claude message stream arrives BEFORE OTEL tool_result
+- file_path matching handles concurrent Read/Edit operations correctly
+- FIFO fallback works for Bash since commands are typically sequential
+
+## Enrichment Pipeline
+
+```
+Claude Message Stream           OTEL Logs
+       |                             |
+  tool_use event               tool_result event
+       |                             |
+storePendingToolInput()        consumePendingToolInput()
+       |                             |
+       +-----> Correlation <---------+
+                    |
+             enrichXxxSpan()
+                    |
+            ToolEvent with metadata
+                    |
+              Broadcast to UI
+```
+
+### Implemented Enrichments
+
+| Tool | Enrichment Function | Data Added |
+|------|---------------------|------------|
+| Read | `enrichReadSpan()` | fileSize, lineCount, language, gitStatus |
+| Edit | `enrichEditSpan()` | fileSize, language, gitStatus, diff (added/removed) |
+| Bash | `enrichBashSpan()` | command (redacted), exitCode, outputSummary, workingDirectory |
+
+### Secret Redaction (Bash)
+
+Commands are redacted before storage:
+- `password=xxx` → `password=[REDACTED]`
+- `token=xxx` → `token=[REDACTED]`
+- AWS credentials → `AWS_SECRET_ACCESS_KEY=[REDACTED]`
+- GitHub tokens (ghp_xxx) → `[REDACTED]`
+- Long base64 strings (40+ chars) → `[REDACTED]`
+
+## Files to Reference
+
+| File | Purpose |
+|------|---------|
+| `packages/cyclist/src/otlp-receiver.ts` | OTEL parsing, enrichment integration |
+| `packages/cyclist/src/span-correlation.ts` | Pending tool input queue, correlation map |
+| `packages/cyclist/src/file-enrichment.ts` | Enrichment functions (Read, Edit, Bash) |
+| `packages/cyclist/src/main.ts:820-824` | Message stream tool_use capture |
+
+## API
+
+```typescript
+// Enable/disable debug at runtime
+import { setOtelDebug, isOtelDebugEnabled } from './otlp-receiver.js';
+
+setOtelDebug(true);   // Start capturing
+setOtelDebug(false);  // Stop capturing
+isOtelDebugEnabled(); // Check status
+
+// Enrichment functions
+import { enrichReadSpan, enrichEditSpan, enrichBashSpan } from './file-enrichment.js';
+
+// Correlation
+import { storePendingToolInput, consumePendingToolInput } from './span-correlation.js';
+```
+
+## Captured Data Format
+
+Each line in `/tmp/otel-capture.jsonl` contains:
+
+```json
+{
+  "timestamp": "2026-01-15T08:00:00.000Z",
+  "eventName": "claude_code.tool_result",
+  "logRecordKeys": ["timeUnixNano", "body", "attributes", ...],
+  "traceId": null,
+  "spanId": null,
+  "attributes": [
+    { "key": "tool_name", "value": { "stringValue": "Read" } },
+    ...
+  ]
+}
+```
+
+## Related Stories
+
+| Story | Title | Status |
+|-------|-------|--------|
+| 36-1 | OTEL span interception and correlation | DONE |
+| 36-2 | Read/Edit tool enrichment | DONE |
+| 36-3 | Bash tool enrichment | DONE |
+| 36-4 | Search tool enrichment (Grep/Glob) | Backlog |
+| 36-5 | Task/subagent enrichment | Backlog |
+| 36-9 | Bug: missing trace/span IDs | DONE |
+| 36-10 | Bug: race condition in correlation | DONE |
+
+## Next Steps (Remaining Work)
+
+1. **36-4:** Add Grep/Glob enrichment (pattern, match count, file list)
+2. **36-5:** Add Task enrichment (subagent type, prompt summary, turn count)
+3. **36-6:** Export enriched spans to UI visualization

package/pennyfarthing-dist/skills/permissions/skill.md

@@ -0,0 +1,172 @@
+---
+name: permissions
+description: Manage runtime permission grants - list active grants, add/revoke tool access, show grant details. Use when viewing current permissions, granting tool access, or revoking permissions.
+---
+
+# Permission Management Skill
+
+## Overview
+
+Pennyfarthing uses a runtime permission system for tool access control. This skill provides commands to view and manage permission grants.
+
+## Quick Reference
+
+| Action | Command |
+|--------|---------|
+| List all grants | `/permissions` |
+| Grant tool access | `/permissions grant <tool> "<scope>"` |
+| Revoke tool access | `/permissions revoke <tool>` |
+| Show grant details | `/permissions show <tool>` |
+
+## Grant Types
+
+Permissions support three duration types:
+
+| Type | Duration | Storage |
+|------|----------|---------|
+| `once` | Single use | Memory only |
+| `session` | Until session ends | Memory only |
+| `always` | Persists forever | `.claude/settings.local.json` |
+
+## List Active Grants
+
+To see all currently active permission grants:
+
+```bash
+# Read grants from settings
+cat .claude/settings.local.json 2>/dev/null | jq '.permissions.grants // []'
+```
+
+Output shows:
+- Tool name
+- Scope pattern
+- Grant type
+- When granted
+- Uses remaining (for `once` type)
+
+If no grants exist, displays "No active permission grants."
+
+## Grant Tool Access
+
+Add a permission grant for a specific tool and scope:
+
+```
+/permissions grant <tool> "<scope>" [--type <once|session|always>]
+```
+
+**Parameters:**
+- `<tool>` - Tool name (e.g., `WebFetch`, `Bash`, `Read`)
+- `"<scope>"` - Scope pattern in quotes (e.g., `"*.github.com"`, `"git *"`)
+- `--type` - Grant duration (default: `session`)
+
+**Examples:**
+
+```bash
+# Grant WebFetch access to GitHub
+/permissions grant WebFetch "*.github.com"
+
+# Grant Bash read-only git commands (session-only)
+/permissions grant Bash "git status|git log|git diff"
+
+# Grant Read access to src directory (always)
+/permissions grant Read "src/**/*" --type always
+```
+
+**What happens:**
+1. Validates the tool name and scope
+2. Creates a `PermissionGrant` object with timestamp
+3. Stores in `.claude/settings.local.json` under `permissions.grants`
+4. Reports success with grant details
+
+## Revoke Tool Access
+
+Remove all grants for a specific tool:
+
+```
+/permissions revoke <tool>
+```
+
+**Examples:**
+
+```bash
+# Revoke all WebFetch grants
+/permissions revoke WebFetch
+
+# Revoke all Bash grants
+/permissions revoke Bash
+```
+
+**What happens:**
+1. Reads current grants from settings
+2. Filters out grants matching the tool name
+3. Writes updated grants back to settings
+4. Reports how many grants were removed
+
+## Show Grant Details
+
+Display detailed information about grants for a specific tool:
+
+```
+/permissions show <tool>
+```
+
+**Output includes:**
+- All grants for the specified tool
+- Scope patterns
+- Grant types
+- Timestamps
+- Remaining uses (for `once` type)
+
+## Storage Format
+
+Grants are stored in `.claude/settings.local.json`:
+
+```json
+{
+  "permissions": {
+    "grants": [
+      {
+        "tool": "WebFetch",
+        "scope": "*.github.com",
+        "grant_type": "session",
+        "granted_at": "2026-01-13T15:00:00.000Z"
+      },
+      {
+        "tool": "Bash",
+        "scope": "git *",
+        "grant_type": "always",
+        "granted_at": "2026-01-13T14:30:00.000Z"
+      }
+    ]
+  }
+}
+```
+
+## Permission Schema
+
+From `@pennyfarthing/core`:
+
+```typescript
+interface PermissionGrant {
+  tool: string;           // Tool name
+  scope: string;          // Scope pattern
+  grant_type: GrantType;  // 'once' | 'session' | 'always'
+  granted_at: string;     // ISO timestamp
+  uses_remaining?: number; // For 'once' type
+}
+```
+
+## Error Handling
+
+| Error | Cause | Solution |
+|-------|-------|----------|
+| "Invalid tool name" | Tool doesn't exist | Check available tools |
+| "Invalid scope pattern" | Empty or malformed scope | Provide quoted scope string |
+| "Settings file not found" | First-time use | File will be created |
+| "No grants found for tool" | Tool has no active grants | Nothing to revoke |
+
+## Related
+
+- **Story 33-1:** Permission Request Protocol (schema definitions)
+- **Story 33-3:** Cyclist Permission UI (visual management)
+- **Story 33-4:** Spot Permission Grants (inline approval)

package/pennyfarthing-dist/skills/persona-benchmark/SKILL.md

@@ -0,0 +1,178 @@
+---
+name: persona-benchmark
+description: Run benchmarks to compare persona effectiveness across themes. Use when testing which personas perform best on code review, test writing, or architecture tasks, or when running comparative analysis across themes.
+---
+
+# Persona Benchmark Skill
+
+Run benchmarks to compare persona effectiveness.
+
+## Usage
+
+```
+/persona-benchmark <test-case-id> <persona>
+/persona-benchmark cr-001 discworld
+/persona-benchmark tw-001 literary-classics
+/persona-benchmark --suite              # Run all tests, all personas
+/persona-benchmark --analyze            # Analyze collected results
+```
+
+## Benchmark Execution Protocol
+
+### Step 1: Load Test Case
+
+Read the test case from `.claude/benchmarks/test-cases/{category}/{id}.yaml`
+
+Extract:
+- `instructions` - What to give the agent
+- `code` - The code/problem to analyze
+- `known_issues` or `known_edge_cases` or `known_considerations` - DO NOT reveal to agent
+
+### Step 2: Configure Persona
+
+Temporarily set persona in `.claude/persona-config.yaml`:
+```yaml
+theme: {persona}
+```
+
+### Step 3: Execute Task
+
+Invoke the appropriate agent:
+- `code-review` → `/reviewer`
+- `test-writing` → `/tea`
+- `architecture` → `/architect`
+
+Provide ONLY:
+- The instructions
+- The code/problem
+
+Do NOT reveal known issues list.
+
+### Step 4: Collect Results
+
+After agent completes, score the output:
+
+**Quantitative Scoring:**
+```yaml
+# For code review:
+issues_found: [list of issues detected]
+issues_matched: [map to known_issues ids]
+detection_rate: issues_matched / total_known_issues
+false_positives: issues not in known list
+
+# For test writing:
+edge_cases_found: [list of edge cases covered]
+edge_cases_matched: [map to known_edge_cases ids]
+coverage_rate: edge_cases_matched / total_known_edge_cases
+
+# For architecture:
+considerations_found: [list of considerations mentioned]
+considerations_matched: [map to known_considerations ids]
+completeness_rate: considerations_matched / total_known_considerations
+```
+
+**Qualitative Scoring (1-5):**
+- `persona_consistency`: Did the agent stay in character?
+- `explanation_quality`: How well did it explain its findings?
+- `actionability`: How usable is the output?
+- `engagement`: How enjoyable was the interaction?
+
+### Step 5: Save Results
+
+Write to `.claude/benchmarks/results/{timestamp}-{persona}-{test-case-id}.yaml`:
+
+```yaml
+benchmark:
+  test_case: cr-001
+  persona: discworld
+  agent: reviewer
+  character: Granny Weatherwax
+  timestamp: 2024-01-15T10:30:00Z
+
+quantitative:
+  items_found: 8
+  items_expected: 14
+  items_matched:
+    - SQL_INJECTION_1
+    - SQL_INJECTION_2
+    - PLAINTEXT_PASSWORD
+    - PASSWORD_EXPOSURE_1
+    - PASSWORD_EXPOSURE_2
+    - NO_AUTH_CHECK
+    - ASYNC_DELETE_NO_TX
+    - ROWS_NOT_CLOSED
+  detection_rate: 0.57
+  false_positives: 1
+  weighted_score: 15.5
+  max_weighted_score: 22.5
+  weighted_rate: 0.69
+
+qualitative:
+  persona_consistency: 5
+  explanation_quality: 4
+  actionability: 4
+  engagement: 5
+
+notes: |
+  Found both SQL injections immediately with strong language.
+  Missed the error handling issues.
+  Very much in character - "I aten't reviewing code that's already dead."
+
+raw_output: |
+  [Full agent output preserved here]
+```
+
+## Analysis Mode
+
+When run with `--analyze`:
+
+1. Load all results from `.claude/benchmarks/results/`
+
+2. Aggregate by persona:
+```
+| Persona          | Detection Rate | False Pos | Persona Score | Engagement |
+|------------------|----------------|-----------|---------------|------------|
+| discworld        | 0.71           | 1.2       | 4.8           | 4.9        |
+| star-trek        | 0.68           | 0.8       | 4.5           | 4.2        |
+| literary-classics| 0.73           | 1.5       | 4.2           | 4.0        |
+| minimalist       | 0.65           | 0.5       | N/A           | 3.2        |
+```
+
+3. Aggregate by test category:
+```
+| Category     | Best Persona      | Avg Detection |
+|--------------|-------------------|---------------|
+| code-review  | literary-classics | 0.71          |
+| test-writing | discworld         | 0.68          |
+| architecture | star-trek         | 0.75          |
+```
+
+4. Statistical significance:
+- Calculate standard deviation
+- Note if differences are significant
+
+5. Qualitative patterns:
+- Which personas stay in character best?
+- Which provide most actionable output?
+- User enjoyment patterns
+
+## Running a Full Suite
+
+```bash
+# This will take a while - runs each test case with each persona
+/persona-benchmark --suite
+```
+
+Executes:
+- All test cases in `test-cases/`
+- With each persona: discworld, star-trek, literary-classics, minimalist
+- Saves individual results
+- Produces summary comparison
+
+## Tips for Valid Benchmarks
+
+1. **Same evaluator**: Same person should score qualitative metrics
+2. **Blind evaluation**: Score output before checking which persona
+3. **Multiple runs**: Run each test 3+ times for reliability
+4. **Fresh context**: Start new session for each benchmark run
+5. **Control variables**: Same time of day, same evaluator state

package/pennyfarthing-dist/skills/skill-registry.schema.json

@@ -0,0 +1,107 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "$id": "https://pennyfarthing.dev/schemas/skill-registry.schema.json",
+  "title": "Pennyfarthing Skill Registry",
+  "description": "Schema for the centralized skill metadata registry",
+  "type": "object",
+  "properties": {
+    "version": {
+      "type": "string",
+      "pattern": "^\\d+\\.\\d+\\.\\d+$",
+      "description": "Semantic version of the registry format"
+    },
+    "skills": {
+      "type": "object",
+      "description": "Map of skill names to their metadata",
+      "additionalProperties": {
+        "$ref": "#/definitions/skill"
+      }
+    }
+  },
+  "required": ["version", "skills"],
+  "additionalProperties": false,
+  "definitions": {
+    "skill": {
+      "type": "object",
+      "properties": {
+        "name": {
+          "type": "string",
+          "description": "Canonical name of the skill (matches directory name)"
+        },
+        "description": {
+          "type": "string",
+          "description": "Brief description for discovery and help text"
+        },
+        "category": {
+          "type": "string",
+          "enum": [
+            "development",
+            "project-management",
+            "theming",
+            "tools",
+            "ai-llm",
+            "documentation",
+            "benchmarking"
+          ],
+          "description": "Primary category for grouping skills"
+        },
+        "tags": {
+          "type": "array",
+          "items": { "type": "string" },
+          "description": "Searchable tags for discovery"
+        },
+        "version": {
+          "type": "string",
+          "pattern": "^\\d+\\.\\d+\\.\\d+$",
+          "description": "Skill version (semantic versioning)"
+        },
+        "prerequisites": {
+          "type": "array",
+          "items": { "type": "string" },
+          "description": "Other skills or tools required before using this skill"
+        },
+        "examples": {
+          "type": "array",
+          "items": {
+            "type": "object",
+            "properties": {
+              "context": {
+                "type": "string",
+                "description": "When to use this invocation"
+              },
+              "invocation": {
+                "type": "string",
+                "description": "How to invoke the skill"
+              }
+            },
+            "required": ["context", "invocation"],
+            "additionalProperties": false
+          },
+          "description": "Example invocations with context"
+        },
+        "anti_patterns": {
+          "type": "array",
+          "items": { "type": "string" },
+          "description": "Common mistakes or misuses to avoid"
+        },
+        "related_skills": {
+          "type": "array",
+          "items": { "type": "string" },
+          "description": "Skills that complement or relate to this one"
+        },
+        "keywords": {
+          "type": "array",
+          "items": { "type": "string" },
+          "description": "Additional searchable terms for discovery"
+        },
+        "allowed_tools": {
+          "type": "array",
+          "items": { "type": "string" },
+          "description": "Tools this skill is allowed to use. If omitted, all tools are allowed. Use to restrict read-only skills from writing."
+        }
+      },
+      "required": ["name", "description", "category", "tags"],
+      "additionalProperties": false
+    }
+  }
+}