@hailer/mcp 0.0.1

package/.claude/skills/skill-testing/README.md

@@ -0,0 +1,137 @@
+# Skill Testing
+
+Quick reference for using the skill-testing skill system.
+
+## What This Skill Provides
+
+This skill helps you test, validate, and ensure quality of skills in the Hailer MCP skill-based agent system. It provides:
+
+- **Testing Patterns**: Reusable test templates and scripts
+- **Validation Checklists**: Comprehensive quality checks
+- **Testing Guide**: Step-by-step testing procedures
+- **Best Practices**: Do's and don'ts for skill testing
+
+## When to Use This Skill
+
+Ask the agent about skill testing when you:
+- Create a new skill and need to test it
+- Want to validate skill quality before deployment
+- Need testing patterns or templates
+- Want to benchmark skill performance
+- Need guidance on testing best practices
+- Want to set up automated testing
+
+## Trigger Keywords
+
+This skill activates when you mention:
+- "test skill" / "test a skill"
+- "validate skill"
+- "skill testing"
+- "test pattern"
+- "benchmark"
+- "quality check"
+- "validation checklist"
+- "unit test" / "integration test"
+- "how to test"
+
+## What You'll Get
+
+When this skill is loaded, the agent receives:
+- Complete testing methodology (unit, integration, E2E, performance)
+- Copy-paste test script templates
+- Validation checklists for pre-deployment
+- Performance benchmarking patterns
+- CI/CD integration examples
+- Quality scoring matrices
+- Troubleshooting guides
+
+## Quick Start
+
+### Test a Single Skill
+
+```bash
+# Create quick test script
+npx tsx test-my-skill.ts
+
+# Or ask the agent:
+# "How do I test my new skill called 'my-skill'?"
+```
+
+### Test All Skills
+
+```bash
+# Run comprehensive test suite
+npx tsx test-all-skills.ts
+
+# Or ask the agent:
+# "Create a script to test all skills"
+```
+
+### Validate Before Deployment
+
+```bash
+# Run validation checks
+npx tsx validate-skill.ts my-skill
+
+# Or ask the agent:
+# "What validation checks should I run before deploying my-skill?"
+```
+
+## Files Included
+
+### Main Skill
+- **SKILL.md**: Overview, principles, quick start, patterns
+
+### References
+- **testing-guide.md**: Comprehensive step-by-step testing procedures
+- **test-patterns.md**: Reusable test templates and scripts
+- **validation-checklist.md**: Quality assurance checklists
+
+## Example Usage
+
+**User**: "I created a new skill called 'data-export'. How should I test it?"
+
+**Agent** (with skill-testing loaded):
+- Provides specific testing steps for the new skill
+- Suggests test patterns for the skill's functionality
+- Recommends validation checks
+- Offers performance benchmarking guidance
+- Provides copy-paste test scripts
+
+## Integration with Development Workflow
+
+1. **Create Skill** → Document functionality
+2. **Test Locally** → Use quick test scripts
+3. **Validate** → Run validation checklist
+4. **Benchmark** → Measure performance
+5. **Deploy** → With confidence!
+
+## Quality Standards
+
+Skills should meet:
+- ✅ All automated tests pass
+- ✅ Load time < 100ms (first load)
+- ✅ Cache time < 10ms
+- ✅ Content quality score ≥ 3.5/5
+- ✅ All validation checks pass
+
+## Getting Help
+
+Ask the agent:
+- "Show me test patterns for skills"
+- "What's in the validation checklist?"
+- "How do I benchmark skill performance?"
+- "Create a test script for my skill"
+- "What are the quality standards for skills?"
+
+## Tips
+
+- Start with quick test scripts for fast feedback
+- Use validation checklist before every deployment
+- Benchmark regularly to catch performance regressions
+- Integrate tests into CI/CD pipeline
+- Test with real user queries
+
+---
+
+*This skill was created using the agent-building skill patterns and is itself an example of a well-tested skill.*

package/.claude/skills/skill-testing/SKILL.md

@@ -0,0 +1,348 @@
+# Skill Testing
+
+A comprehensive guide for testing skills in the Hailer MCP skill-based agent system. Provides generic, reusable patterns for validating skill content, structure, integration, and performance.
+
+## Overview
+
+Testing skills ensures they:
+- Load correctly from disk
+- Match the right user requests (keyword matching works)
+- Provide useful content to the LLM
+- Don't break the system under various conditions
+- Perform efficiently (caching, load times)
+- Follow consistent structure and quality standards
+
+## Core Testing Principles
+
+### 1. Test Isolation
+Each skill should be testable in isolation without dependencies on:
+- Live MCP server
+- External APIs
+- Other skills
+- Specific LLM providers
+
+### 2. Test Coverage Levels
+```
+┌─────────────────────────────────────┐
+│  Unit Tests (Individual Skill)     │  <- Fast, isolated
+├─────────────────────────────────────┤
+│  Integration Tests (Skill System)  │  <- Real skill loading
+├─────────────────────────────────────┤
+│  E2E Tests (Full Agent Flow)       │  <- Complete workflow
+└─────────────────────────────────────┘
+```
+
+### 3. Testing Dimensions
+
+**Functional Tests:**
+- ✅ Skill loads without errors
+- ✅ Content is valid and complete
+- ✅ References load correctly
+- ✅ Keyword matching works as expected
+
+**Performance Tests:**
+- ⚡ First load time < 100ms
+- ⚡ Cached load time < 10ms
+- ⚡ Skill content size reasonable (< 100KB recommended)
+
+**Quality Tests:**
+- 📝 Content is clear and actionable
+- 📝 Examples are relevant and working
+- 📝 No broken references or links
+- 📝 Consistent formatting and structure
+
+**Integration Tests:**
+- 🔗 Works with SkillLoader
+- 🔗 Works with SkillManager
+- 🔗 Keyword mappings are accurate
+- 🔗 Graceful degradation on errors
+
+## Quick Start: Testing a New Skill
+
+```typescript
+// 1. Create test file: test-my-skill.ts
+import { SkillLoader } from './src/client/skill-loader';
+import { SkillManager } from './src/client/skill-manager';
+import * as path from 'path';
+
+async function testMySkill() {
+  const loader = new SkillLoader(path.join(process.cwd(), '.claude', 'skills'));
+  const manager = new SkillManager(loader);
+
+  // Test 1: Skill loads
+  const skill = await loader.load('my-skill');
+  console.assert(skill.content.length > 0, 'Skill content should not be empty');
+
+  // Test 2: Keyword matching
+  const guidance = await manager.analyzeRequest('my skill trigger phrase');
+  console.assert(guidance.skills.includes('my-skill'), 'Should match my-skill');
+
+  // Test 3: Performance
+  const start = Date.now();
+  await loader.load('my-skill'); // Should use cache
+  const duration = Date.now() - start;
+  console.assert(duration < 10, `Cached load should be fast (${duration}ms)`);
+}
+```
+
+## Test File Structure
+
+```
+server/
+├── tests/
+│   ├── skills/
+│   │   ├── skill-loader.test.ts      # Unit tests for SkillLoader
+│   │   ├── skill-manager.test.ts     # Unit tests for SkillManager
+│   │   ├── integration.test.ts       # Integration tests
+│   │   └── performance.test.ts       # Performance benchmarks
+│   └── fixtures/
+│       └── test-skill/               # Mock skill for testing
+│           └── SKILL.md
+```
+
+## Common Testing Patterns
+
+### Pattern 1: Test Skill Loading
+```typescript
+describe('SkillLoader', () => {
+  it('should load skill with content', async () => {
+    const loader = new SkillLoader(skillsPath);
+    const skill = await loader.load('my-skill');
+
+    expect(skill.name).toBe('my-skill');
+    expect(skill.content).toBeTruthy();
+    expect(skill.description).toBeTruthy();
+    expect(skill.loadedAt).toBeGreaterThan(0);
+  });
+});
+```
+
+### Pattern 2: Test Keyword Matching
+```typescript
+describe('SkillManager', () => {
+  it('should match skill by keywords', async () => {
+    const manager = new SkillManager(loader);
+    const testCases = [
+      { message: 'create a workflow', expectedSkill: 'mcp-tools' },
+      { message: 'list activities', expectedSkill: 'hailer-api' },
+      { message: 'build an agent', expectedSkill: 'agent-building' },
+    ];
+
+    for (const { message, expectedSkill } of testCases) {
+      const guidance = await manager.analyzeRequest(message);
+      expect(guidance.skills).toContain(expectedSkill);
+    }
+  });
+});
+```
+
+### Pattern 3: Test Caching
+```typescript
+describe('Skill Caching', () => {
+  it('should cache skills after first load', async () => {
+    const loader = new SkillLoader(skillsPath);
+
+    // First load (from disk)
+    const start1 = Date.now();
+    await loader.load('my-skill');
+    const firstLoad = Date.now() - start1;
+
+    // Second load (from cache)
+    const start2 = Date.now();
+    await loader.load('my-skill');
+    const cachedLoad = Date.now() - start2;
+
+    expect(cachedLoad).toBeLessThan(firstLoad / 5); // At least 5x faster
+  });
+});
+```
+
+### Pattern 4: Test Error Handling
+```typescript
+describe('Error Handling', () => {
+  it('should handle missing skills gracefully', async () => {
+    const loader = new SkillLoader(skillsPath);
+
+    await expect(
+      loader.load('non-existent-skill')
+    ).rejects.toThrow(/Failed to load skill/);
+  });
+
+  it('should return default guidance when no skill matches', async () => {
+    const manager = new SkillManager(loader);
+    const guidance = await manager.analyzeRequest('random gibberish xyz123');
+
+    expect(guidance.skills).toHaveLength(0);
+    expect(guidance.confidence).toBeLessThan(0.5);
+    expect(guidance.guidance).toContain('General assistance');
+  });
+});
+```
+
+## Testing Checklist
+
+Before deploying a new skill, verify:
+
+- [ ] **Structure**: SKILL.md exists and is valid markdown
+- [ ] **Content**: Contains clear, actionable information
+- [ ] **References**: All references/*.md files load correctly
+- [ ] **Keywords**: Added to SkillManager.buildSkillMappings()
+- [ ] **Loading**: Skill loads without errors
+- [ ] **Matching**: Keywords trigger the skill correctly
+- [ ] **Performance**: First load < 100ms, cached load < 10ms
+- [ ] **Size**: Total content < 100KB (or justified if larger)
+- [ ] **Examples**: All code examples are valid and tested
+- [ ] **Tools**: Recommended tools are accurate and available
+- [ ] **Integration**: Works with both OpenAI and Anthropic providers
+- [ ] **Graceful Failure**: Handles errors without crashing
+
+## Automated Testing
+
+### Quick Test Script
+Create `test-skill.ts` in server directory:
+
+```typescript
+import { SkillLoader } from './src/client/skill-loader';
+import { SkillManager } from './src/client/skill-manager';
+import * as path from 'path';
+import * as fs from 'fs/promises';
+
+async function validateSkill(skillName: string) {
+  console.log(`\n🧪 Testing skill: ${skillName}`);
+
+  const skillsPath = path.join(process.cwd(), '..', '.claude', 'skills');
+  const loader = new SkillLoader(skillsPath);
+  const manager = new SkillManager(loader);
+
+  try {
+    // 1. Structure validation
+    const skillPath = path.join(skillsPath, skillName);
+    const skillMdExists = await fs.stat(path.join(skillPath, 'SKILL.md'))
+      .then(() => true).catch(() => false);
+    console.log(`  ${skillMdExists ? '✅' : '❌'} SKILL.md exists`);
+
+    // 2. Loading test
+    const start = Date.now();
+    const skill = await loader.load(skillName);
+    const loadTime = Date.now() - start;
+    console.log(`  ${loadTime < 100 ? '✅' : '⚠️'} First load: ${loadTime}ms`);
+    console.log(`  ✅ Content loaded: ${skill.content.length} chars`);
+
+    // 3. Caching test
+    const start2 = Date.now();
+    await loader.load(skillName);
+    const cacheTime = Date.now() - start2;
+    console.log(`  ${cacheTime < 10 ? '✅' : '⚠️'} Cached load: ${cacheTime}ms`);
+
+    // 4. Size check
+    const sizeKB = skill.content.length / 1024;
+    console.log(`  ${sizeKB < 100 ? '✅' : '⚠️'} Size: ${sizeKB.toFixed(1)} KB`);
+
+    // 5. Content validation
+    const hasExamples = skill.content.includes('```');
+    const hasStructure = skill.content.includes('#');
+    console.log(`  ${hasExamples ? '✅' : '⚠️'} Contains code examples`);
+    console.log(`  ${hasStructure ? '✅' : '⚠️'} Has structured headings`);
+
+    console.log(`\n✅ ${skillName} passed validation!`);
+    return true;
+  } catch (error) {
+    console.error(`\n❌ ${skillName} failed:`, error);
+    return false;
+  }
+}
+
+// Test all skills
+async function testAllSkills() {
+  const skillsPath = path.join(process.cwd(), '..', '.claude', 'skills');
+  const loader = new SkillLoader(skillsPath);
+  const allSkills = await loader.getAvailableSkills();
+
+  console.log(`Found ${allSkills.length} skills to test\n`);
+
+  const results = await Promise.all(
+    allSkills.map(skill => validateSkill(skill))
+  );
+
+  const passed = results.filter(r => r).length;
+  console.log(`\n${'='.repeat(50)}`);
+  console.log(`Results: ${passed}/${allSkills.length} skills passed`);
+  console.log(`${'='.repeat(50)}\n`);
+}
+
+testAllSkills().catch(console.error);
+```
+
+Run with: `npx tsx test-skill.ts`
+
+## Best Practices
+
+### DO ✅
+- Write tests before creating the skill (TDD)
+- Test both success and failure cases
+- Validate skill content quality, not just structure
+- Test performance with realistic data sizes
+- Use fixtures for reproducible tests
+- Mock external dependencies
+- Test keyword matching with real user queries
+- Verify graceful degradation
+
+### DON'T ❌
+- Don't test against live APIs in unit tests
+- Don't skip error handling tests
+- Don't ignore performance regressions
+- Don't forget to test cache behavior
+- Don't assume keywords will match perfectly
+- Don't test implementation details
+- Don't create brittle tests that break on minor changes
+
+## Troubleshooting
+
+### Skill Doesn't Load
+1. Check file path: `skills/{skill-name}/SKILL.md`
+2. Verify file permissions
+3. Check for syntax errors in markdown
+4. Look at loader error logs
+
+### Keywords Don't Match
+1. Review keyword list in SkillManager
+2. Test with actual user messages
+3. Add more specific keywords
+4. Check for typos in keyword mappings
+
+### Performance Issues
+1. Check skill size (should be < 100KB)
+2. Verify caching is working
+3. Profile with real-world queries
+4. Consider splitting large skills
+
+### Content Quality Issues
+1. Have others review the content
+2. Test with the actual LLM
+3. Check if examples are clear
+4. Verify technical accuracy
+
+## Integration with CI/CD
+
+```yaml
+# .github/workflows/test-skills.yml
+name: Test Skills
+
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-node@v3
+      - run: npm install
+      - run: npx tsx test-skill.ts
+```
+
+## Further Reading
+
+See reference files for detailed guides:
+- `testing-guide.md` - Comprehensive testing procedures
+- `test-patterns.md` - Reusable test patterns and templates
+- `validation-checklist.md` - Quality assurance checklist