@heyai-rules/pilo-masterkit 1.2.2 → 2.1.0

package/.agent/skills/eval-harness/SKILL.md

@@ -0,0 +1,270 @@
+---
+name: eval-harness
+description: Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles
+origin: ECC
+tools: Read, Write, Edit, Bash, Grep, Glob
+---
+
+# Eval Harness Skill
+
+A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.
+
+## When to Activate
+
+- Setting up eval-driven development (EDD) for AI-assisted workflows
+- Defining pass/fail criteria for Claude Code task completion
+- Measuring agent reliability with pass@k metrics
+- Creating regression test suites for prompt or agent changes
+- Benchmarking agent performance across model versions
+
+## Philosophy
+
+Eval-Driven Development treats evals as the "unit tests of AI development":
+- Define expected behavior BEFORE implementation
+- Run evals continuously during development
+- Track regressions with each change
+- Use pass@k metrics for reliability measurement
+
+## Eval Types
+
+### Capability Evals
+Test if Claude can do something it couldn't before:
+```markdown
+[CAPABILITY EVAL: feature-name]
+Task: Description of what Claude should accomplish
+Success Criteria:
+  - [ ] Criterion 1
+  - [ ] Criterion 2
+  - [ ] Criterion 3
+Expected Output: Description of expected result
+```
+
+### Regression Evals
+Ensure changes don't break existing functionality:
+```markdown
+[REGRESSION EVAL: feature-name]
+Baseline: SHA or checkpoint name
+Tests:
+  - existing-test-1: PASS/FAIL
+  - existing-test-2: PASS/FAIL
+  - existing-test-3: PASS/FAIL
+Result: X/Y passed (previously Y/Y)
+```
+
+## Grader Types
+
+### 1. Code-Based Grader
+Deterministic checks using code:
+```bash
+# Check if file contains expected pattern
+grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"
+
+# Check if tests pass
+npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"
+
+# Check if build succeeds
+npm run build && echo "PASS" || echo "FAIL"
+```
+
+### 2. Model-Based Grader
+Use Claude to evaluate open-ended outputs:
+```markdown
+[MODEL GRADER PROMPT]
+Evaluate the following code change:
+1. Does it solve the stated problem?
+2. Is it well-structured?
+3. Are edge cases handled?
+4. Is error handling appropriate?
+
+Score: 1-5 (1=poor, 5=excellent)
+Reasoning: [explanation]
+```
+
+### 3. Human Grader
+Flag for manual review:
+```markdown
+[HUMAN REVIEW REQUIRED]
+Change: Description of what changed
+Reason: Why human review is needed
+Risk Level: LOW/MEDIUM/HIGH
+```
+
+## Metrics
+
+### pass@k
+"At least one success in k attempts"
+- pass@1: First attempt success rate
+- pass@3: Success within 3 attempts
+- Typical target: pass@3 > 90%
+
+### pass^k
+"All k trials succeed"
+- Higher bar for reliability
+- pass^3: 3 consecutive successes
+- Use for critical paths
+
+## Eval Workflow
+
+### 1. Define (Before Coding)
+```markdown
+## EVAL DEFINITION: feature-xyz
+
+### Capability Evals
+1. Can create new user account
+2. Can validate email format
+3. Can hash password securely
+
+### Regression Evals
+1. Existing login still works
+2. Session management unchanged
+3. Logout flow intact
+
+### Success Metrics
+- pass@3 > 90% for capability evals
+- pass^3 = 100% for regression evals
+```
+
+### 2. Implement
+Write code to pass the defined evals.
+
+### 3. Evaluate
+```bash
+# Run capability evals
+[Run each capability eval, record PASS/FAIL]
+
+# Run regression evals
+npm test -- --testPathPattern="existing"
+
+# Generate report
+```
+
+### 4. Report
+```markdown
+EVAL REPORT: feature-xyz
+========================
+
+Capability Evals:
+  create-user:     PASS (pass@1)
+  validate-email:  PASS (pass@2)
+  hash-password:   PASS (pass@1)
+  Overall:         3/3 passed
+
+Regression Evals:
+  login-flow:      PASS
+  session-mgmt:    PASS
+  logout-flow:     PASS
+  Overall:         3/3 passed
+
+Metrics:
+  pass@1: 67% (2/3)
+  pass@3: 100% (3/3)
+
+Status: READY FOR REVIEW
+```
+
+## Integration Patterns
+
+### Pre-Implementation
+```
+/eval define feature-name
+```
+Creates eval definition file at `.claude/evals/feature-name.md`
+
+### During Implementation
+```
+/eval check feature-name
+```
+Runs current evals and reports status
+
+### Post-Implementation
+```
+/eval report feature-name
+```
+Generates full eval report
+
+## Eval Storage
+
+Store evals in project:
+```
+.claude/
+  evals/
+    feature-xyz.md      # Eval definition
+    feature-xyz.log     # Eval run history
+    baseline.json       # Regression baselines
+```
+
+## Best Practices
+
+1. **Define evals BEFORE coding** - Forces clear thinking about success criteria
+2. **Run evals frequently** - Catch regressions early
+3. **Track pass@k over time** - Monitor reliability trends
+4. **Use code graders when possible** - Deterministic > probabilistic
+5. **Human review for security** - Never fully automate security checks
+6. **Keep evals fast** - Slow evals don't get run
+7. **Version evals with code** - Evals are first-class artifacts
+
+## Example: Adding Authentication
+
+```markdown
+## EVAL: add-authentication
+
+### Phase 1: Define (10 min)
+Capability Evals:
+- [ ] User can register with email/password
+- [ ] User can login with valid credentials
+- [ ] Invalid credentials rejected with proper error
+- [ ] Sessions persist across page reloads
+- [ ] Logout clears session
+
+Regression Evals:
+- [ ] Public routes still accessible
+- [ ] API responses unchanged
+- [ ] Database schema compatible
+
+### Phase 2: Implement (varies)
+[Write code]
+
+### Phase 3: Evaluate
+Run: /eval check add-authentication
+
+### Phase 4: Report
+EVAL REPORT: add-authentication
+==============================
+Capability: 5/5 passed (pass@3: 100%)
+Regression: 3/3 passed (pass^3: 100%)
+Status: SHIP IT
+```
+
+## Product Evals (v1.8)
+
+Use product evals when behavior quality cannot be captured by unit tests alone.
+
+### Grader Types
+
+1. Code grader (deterministic assertions)
+2. Rule grader (regex/schema constraints)
+3. Model grader (LLM-as-judge rubric)
+4. Human grader (manual adjudication for ambiguous outputs)
+
+### pass@k Guidance
+
+- `pass@1`: direct reliability
+- `pass@3`: practical reliability under controlled retries
+- `pass^3`: stability test (all 3 runs must pass)
+
+Recommended thresholds:
+- Capability evals: pass@3 >= 0.90
+- Regression evals: pass^3 = 1.00 for release-critical paths
+
+### Eval Anti-Patterns
+
+- Overfitting prompts to known eval examples
+- Measuring only happy-path outputs
+- Ignoring cost and latency drift while chasing pass rates
+- Allowing flaky graders in release gates
+
+### Minimal Eval Artifact Layout
+
+- `.claude/evals/<feature>.md` definition
+- `.claude/evals/<feature>.log` run history
+- `docs/releases/<version>/eval-summary.md` release snapshot

package/.agent/skills/exa-search/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: exa-search
+description: Neural search via Exa MCP for web, code, and company research. Use when the user needs web search, code examples, company intel, people lookup, or AI-powered deep research with Exa's neural search engine.
+origin: ECC
+---
+
+# Exa Search
+
+Neural search for web content, code, companies, and people via the Exa MCP server.
+
+## When to Activate
+
+- User needs current web information or news
+- Searching for code examples, API docs, or technical references
+- Researching companies, competitors, or market players
+- Finding professional profiles or people in a domain
+- Running background research for any development task
+- User says "search for", "look up", "find", or "what's the latest on"
+
+## MCP Requirement
+
+Exa MCP server must be configured. Add to `~/.claude.json`:
+
+```json
+"exa-web-search": {
+  "command": "npx",
+  "args": ["-y", "exa-mcp-server"],
+  "env": { "EXA_API_KEY": "YOUR_EXA_API_KEY_HERE" }
+}
+```
+
+Get an API key at [exa.ai](https://exa.ai).
+This repo's current Exa setup documents the tool surface exposed here: `web_search_exa` and `get_code_context_exa`.
+If your Exa server exposes additional tools, verify their exact names before depending on them in docs or prompts.
+
+## Core Tools
+
+### web_search_exa
+General web search for current information, news, or facts.
+
+```
+web_search_exa(query: "latest AI developments 2026", numResults: 5)
+```
+
+**Parameters:**
+
+| Param | Type | Default | Notes |
+|-------|------|---------|-------|
+| `query` | string | required | Search query |
+| `numResults` | number | 8 | Number of results |
+| `type` | string | `auto` | Search mode |
+| `livecrawl` | string | `fallback` | Prefer live crawling when needed |
+| `category` | string | none | Optional focus such as `company` or `research paper` |
+
+### get_code_context_exa
+Find code examples and documentation from GitHub, Stack Overflow, and docs sites.
+
+```
+get_code_context_exa(query: "Python asyncio patterns", tokensNum: 3000)
+```
+
+**Parameters:**
+
+| Param | Type | Default | Notes |
+|-------|------|---------|-------|
+| `query` | string | required | Code or API search query |
+| `tokensNum` | number | 5000 | Content tokens (1000-50000) |
+
+## Usage Patterns
+
+### Quick Lookup
+```
+web_search_exa(query: "Node.js 22 new features", numResults: 3)
+```
+
+### Code Research
+```
+get_code_context_exa(query: "Rust error handling patterns Result type", tokensNum: 3000)
+```
+
+### Company or People Research
+```
+web_search_exa(query: "Vercel funding valuation 2026", numResults: 3, category: "company")
+web_search_exa(query: "site:linkedin.com/in AI safety researchers Anthropic", numResults: 5)
+```
+
+### Technical Deep Dive
+```
+web_search_exa(query: "WebAssembly component model status and adoption", numResults: 5)
+get_code_context_exa(query: "WebAssembly component model examples", tokensNum: 4000)
+```
+
+## Tips
+
+- Use `web_search_exa` for current information, company lookups, and broad discovery
+- Use search operators like `site:`, quoted phrases, and `intitle:` to narrow results
+- Lower `tokensNum` (1000-2000) for focused code snippets, higher (5000+) for comprehensive context
+- Use `get_code_context_exa` when you need API usage or code examples rather than general web pages
+
+## Related Skills
+
+- `deep-research` — Full research workflow using firecrawl + exa together
+- `market-research` — Business-oriented research with decision frameworks

package/.agent/skills/fal-ai-media/SKILL.md

@@ -0,0 +1,284 @@
+---
+name: fal-ai-media
+description: Unified media generation via fal.ai MCP — image, video, and audio. Covers text-to-image (Nano Banana), text/image-to-video (Seedance, Kling, Veo 3), text-to-speech (CSM-1B), and video-to-audio (ThinkSound). Use when the user wants to generate images, videos, or audio with AI.
+origin: ECC
+---
+
+# fal.ai Media Generation
+
+Generate images, videos, and audio using fal.ai models via MCP.
+
+## When to Activate
+
+- User wants to generate images from text prompts
+- Creating videos from text or images
+- Generating speech, music, or sound effects
+- Any media generation task
+- User says "generate image", "create video", "text to speech", "make a thumbnail", or similar
+
+## MCP Requirement
+
+fal.ai MCP server must be configured. Add to `~/.claude.json`:
+
+```json
+"fal-ai": {
+  "command": "npx",
+  "args": ["-y", "fal-ai-mcp-server"],
+  "env": { "FAL_KEY": "YOUR_FAL_KEY_HERE" }
+}
+```
+
+Get an API key at [fal.ai](https://fal.ai).
+
+## MCP Tools
+
+The fal.ai MCP provides these tools:
+- `search` — Find available models by keyword
+- `find` — Get model details and parameters
+- `generate` — Run a model with parameters
+- `result` — Check async generation status
+- `status` — Check job status
+- `cancel` — Cancel a running job
+- `estimate_cost` — Estimate generation cost
+- `models` — List popular models
+- `upload` — Upload files for use as inputs
+
+---
+
+## Image Generation
+
+### Nano Banana 2 (Fast)
+Best for: quick iterations, drafts, text-to-image, image editing.
+
+```
+generate(
+  app_id: "fal-ai/nano-banana-2",
+  input_data: {
+    "prompt": "a futuristic cityscape at sunset, cyberpunk style",
+    "image_size": "landscape_16_9",
+    "num_images": 1,
+    "seed": 42
+  }
+)
+```
+
+### Nano Banana Pro (High Fidelity)
+Best for: production images, realism, typography, detailed prompts.
+
+```
+generate(
+  app_id: "fal-ai/nano-banana-pro",
+  input_data: {
+    "prompt": "professional product photo of wireless headphones on marble surface, studio lighting",
+    "image_size": "square",
+    "num_images": 1,
+    "guidance_scale": 7.5
+  }
+)
+```
+
+### Common Image Parameters
+
+| Param | Type | Options | Notes |
+|-------|------|---------|-------|
+| `prompt` | string | required | Describe what you want |
+| `image_size` | string | `square`, `portrait_4_3`, `landscape_16_9`, `portrait_16_9`, `landscape_4_3` | Aspect ratio |
+| `num_images` | number | 1-4 | How many to generate |
+| `seed` | number | any integer | Reproducibility |
+| `guidance_scale` | number | 1-20 | How closely to follow the prompt (higher = more literal) |
+
+### Image Editing
+Use Nano Banana 2 with an input image for inpainting, outpainting, or style transfer:
+
+```
+# First upload the source image
+upload(file_path: "/path/to/image.png")
+
+# Then generate with image input
+generate(
+  app_id: "fal-ai/nano-banana-2",
+  input_data: {
+    "prompt": "same scene but in watercolor style",
+    "image_url": "<uploaded_url>",
+    "image_size": "landscape_16_9"
+  }
+)
+```
+
+---
+
+## Video Generation
+
+### Seedance 1.0 Pro (ByteDance)
+Best for: text-to-video, image-to-video with high motion quality.
+
+```
+generate(
+  app_id: "fal-ai/seedance-1-0-pro",
+  input_data: {
+    "prompt": "a drone flyover of a mountain lake at golden hour, cinematic",
+    "duration": "5s",
+    "aspect_ratio": "16:9",
+    "seed": 42
+  }
+)
+```
+
+### Kling Video v3 Pro
+Best for: text/image-to-video with native audio generation.
+
+```
+generate(
+  app_id: "fal-ai/kling-video/v3/pro",
+  input_data: {
+    "prompt": "ocean waves crashing on a rocky coast, dramatic clouds",
+    "duration": "5s",
+    "aspect_ratio": "16:9"
+  }
+)
+```
+
+### Veo 3 (Google DeepMind)
+Best for: video with generated sound, high visual quality.
+
+```
+generate(
+  app_id: "fal-ai/veo-3",
+  input_data: {
+    "prompt": "a bustling Tokyo street market at night, neon signs, crowd noise",
+    "aspect_ratio": "16:9"
+  }
+)
+```
+
+### Image-to-Video
+Start from an existing image:
+
+```
+generate(
+  app_id: "fal-ai/seedance-1-0-pro",
+  input_data: {
+    "prompt": "camera slowly zooms out, gentle wind moves the trees",
+    "image_url": "<uploaded_image_url>",
+    "duration": "5s"
+  }
+)
+```
+
+### Video Parameters
+
+| Param | Type | Options | Notes |
+|-------|------|---------|-------|
+| `prompt` | string | required | Describe the video |
+| `duration` | string | `"5s"`, `"10s"` | Video length |
+| `aspect_ratio` | string | `"16:9"`, `"9:16"`, `"1:1"` | Frame ratio |
+| `seed` | number | any integer | Reproducibility |
+| `image_url` | string | URL | Source image for image-to-video |
+
+---
+
+## Audio Generation
+
+### CSM-1B (Conversational Speech)
+Text-to-speech with natural, conversational quality.
+
+```
+generate(
+  app_id: "fal-ai/csm-1b",
+  input_data: {
+    "text": "Hello, welcome to the demo. Let me show you how this works.",
+    "speaker_id": 0
+  }
+)
+```
+
+### ThinkSound (Video-to-Audio)
+Generate matching audio from video content.
+
+```
+generate(
+  app_id: "fal-ai/thinksound",
+  input_data: {
+    "video_url": "<video_url>",
+    "prompt": "ambient forest sounds with birds chirping"
+  }
+)
+```
+
+### ElevenLabs (via API, no MCP)
+For professional voice synthesis, use ElevenLabs directly:
+
+```python
+import os
+import requests
+
+resp = requests.post(
+    "https://api.elevenlabs.io/v1/text-to-speech/<voice_id>",
+    headers={
+        "xi-api-key": os.environ["ELEVENLABS_API_KEY"],
+        "Content-Type": "application/json"
+    },
+    json={
+        "text": "Your text here",
+        "model_id": "eleven_turbo_v2_5",
+        "voice_settings": {"stability": 0.5, "similarity_boost": 0.75}
+    }
+)
+with open("output.mp3", "wb") as f:
+    f.write(resp.content)
+```
+
+### VideoDB Generative Audio
+If VideoDB is configured, use its generative audio:
+
+```python
+# Voice generation
+audio = coll.generate_voice(text="Your narration here", voice="alloy")
+
+# Music generation
+music = coll.generate_music(prompt="upbeat electronic background music", duration=30)
+
+# Sound effects
+sfx = coll.generate_sound_effect(prompt="thunder crack followed by rain")
+```
+
+---
+
+## Cost Estimation
+
+Before generating, check estimated cost:
+
+```
+estimate_cost(
+  estimate_type: "unit_price",
+  endpoints: {
+    "fal-ai/nano-banana-pro": {
+      "unit_quantity": 1
+    }
+  }
+)
+```
+
+## Model Discovery
+
+Find models for specific tasks:
+
+```
+search(query: "text to video")
+find(endpoint_ids: ["fal-ai/seedance-1-0-pro"])
+models()
+```
+
+## Tips
+
+- Use `seed` for reproducible results when iterating on prompts
+- Start with lower-cost models (Nano Banana 2) for prompt iteration, then switch to Pro for finals
+- For video, keep prompts descriptive but concise — focus on motion and scene
+- Image-to-video produces more controlled results than pure text-to-video
+- Check `estimate_cost` before running expensive video generations
+
+## Related Skills
+
+- `videodb` — Video processing, editing, and streaming
+- `video-editing` — AI-powered video editing workflows
+- `content-engine` — Content creation for social platforms