@hailer/mcp 0.0.1

package/.claude/skills/skill-testing/references/validation-checklist.md

@@ -0,0 +1,537 @@
+# Skill Validation Checklist
+
+Comprehensive quality checklist for validating skills before deployment.
+
+## Pre-Deployment Checklist
+
+Use this checklist for every new skill or major skill update:
+
+### 📁 Structure & Files
+
+```
+□ SKILL.md exists in root of skill directory
+□ SKILL.md contains valid markdown
+□ SKILL.md has clear title (# Heading)
+□ SKILL.md has overview/introduction section
+□ references/ directory exists (if needed)
+□ All reference files are .md format
+□ No broken file references
+□ File permissions are correct (readable)
+□ Total size < 100KB (or justified if larger)
+```
+
+### 📝 Content Quality
+
+```
+□ Title is clear and descriptive
+□ Overview explains skill purpose
+□ Content is technically accurate
+□ Examples are working and tested
+□ Code blocks use proper syntax highlighting
+□ Instructions are clear and actionable
+□ No typos or grammar errors
+□ Consistent formatting throughout
+□ Appropriate level of detail
+□ No outdated information
+```
+
+### 🔍 Keywords & Matching
+
+```
+□ Keywords added to SkillManager.buildSkillMappings()
+□ Primary keywords are specific and unique
+□ Secondary keywords provide coverage
+□ Keywords match actual user queries
+□ Tested with 5+ real user queries
+□ Confidence scores are appropriate
+□ No keyword conflicts with other skills
+□ Edge cases handled (typos, variations)
+```
+
+### 🔧 Tools & Integration
+
+```
+□ Recommended tools list is accurate
+□ All recommended tools actually exist
+□ Tools are appropriate for skill purpose
+□ Tool usage examples are provided
+□ Integration with MCP system verified
+□ Works with SkillLoader
+□ Works with SkillManager
+□ Error handling is graceful
+```
+
+### ⚡ Performance
+
+```
+□ First load time < 100ms
+□ Cached load time < 10ms
+□ Skill content is optimized
+□ No unnecessary large assets
+□ Caching works correctly
+□ No memory leaks
+□ Performance under load tested
+```
+
+### 🧪 Testing
+
+```
+□ Unit tests pass
+□ Integration tests pass
+□ Keyword matching tests pass
+□ Performance benchmarks pass
+□ Error handling tested
+□ Edge cases covered
+□ All test patterns applied
+```
+
+### 🤖 LLM Integration
+
+```
+□ Works with OpenAI provider
+□ Works with Anthropic provider
+□ System prompt enhancement works
+□ Skill content is helpful to LLM
+□ LLM can understand and use content
+□ Examples are LLM-friendly
+□ Guidance text is clear
+```
+
+### 📚 Documentation
+
+```
+□ Skill purpose is documented
+□ Usage examples provided
+□ Prerequisites listed (if any)
+□ Troubleshooting tips included
+□ References are cited
+□ Related skills mentioned
+□ Changelog/version info (if updated)
+```
+
+### 🔒 Security & Safety
+
+```
+□ No sensitive information in skill
+□ No hardcoded credentials
+□ No malicious code
+□ Safe examples (no destructive operations)
+□ Appropriate warnings for risky operations
+```
+
+### 🎯 User Experience
+
+```
+□ Skill solves a real user need
+□ Content is easy to understand
+□ Examples are relevant and practical
+□ Navigation is logical
+□ Skill complements existing skills
+□ No duplicate content with other skills
+```
+
+## Quality Scoring Matrix
+
+Score each dimension from 1-5:
+
+### Content Quality (Weight: 30%)
+```
+5 - Exceptional: Clear, comprehensive, accurate, well-organized
+4 - Good: Clear and accurate with minor improvements needed
+3 - Adequate: Usable but needs refinement
+2 - Poor: Confusing or inaccurate in places
+1 - Unacceptable: Major issues, not usable
+```
+
+### Technical Accuracy (Weight: 25%)
+```
+5 - Exceptional: 100% accurate, tested examples
+4 - Good: Accurate with minor issues
+3 - Adequate: Mostly accurate, some gaps
+2 - Poor: Several inaccuracies
+1 - Unacceptable: Fundamentally incorrect
+```
+
+### Usability (Weight: 20%)
+```
+5 - Exceptional: Intuitive, excellent examples, clear instructions
+4 - Good: Easy to use with minor confusion points
+3 - Adequate: Usable but requires effort
+2 - Poor: Confusing, difficult to apply
+1 - Unacceptable: Unusable
+```
+
+### Performance (Weight: 15%)
+```
+5 - Exceptional: Blazing fast, < 50ms first load
+4 - Good: Fast, < 100ms first load
+3 - Adequate: Acceptable, < 150ms first load
+2 - Poor: Slow, > 150ms first load
+1 - Unacceptable: Very slow, > 300ms first load
+```
+
+### Coverage (Weight: 10%)
+```
+5 - Exceptional: Comprehensive, covers all scenarios
+4 - Good: Covers main scenarios well
+3 - Adequate: Basic coverage
+2 - Poor: Significant gaps
+1 - Unacceptable: Minimal coverage
+```
+
+**Minimum acceptable score: 3.5/5 weighted average**
+
+## Automated Validation Script
+
+```typescript
+/**
+ * Automated skill validation
+ * Run: npx tsx validate-skill.ts <skill-name>
+ */
+
+import { SkillLoader } from './src/client/skill-loader';
+import { SkillManager } from './src/client/skill-manager';
+import * as path from 'path';
+import * as fs from 'fs/promises';
+
+interface ValidationResult {
+  category: string;
+  checks: Array<{ name: string; passed: boolean; message?: string }>;
+}
+
+async function validateSkill(skillName: string): Promise<boolean> {
+  console.log(`\n🔍 Validating Skill: ${skillName}\n`);
+
+  const skillsPath = path.join(process.cwd(), '..', '.claude', 'skills');
+  const skillPath = path.join(skillsPath, skillName);
+  const loader = new SkillLoader(skillsPath);
+  const manager = new SkillManager(loader);
+
+  const results: ValidationResult[] = [];
+  let totalPassed = 0;
+  let totalChecks = 0;
+
+  // Category 1: Structure & Files
+  const structureChecks: ValidationResult = {
+    category: 'Structure & Files',
+    checks: [],
+  };
+
+  try {
+    await fs.stat(path.join(skillPath, 'SKILL.md'));
+    structureChecks.checks.push({ name: 'SKILL.md exists', passed: true });
+  } catch {
+    structureChecks.checks.push({ name: 'SKILL.md exists', passed: false });
+  }
+
+  try {
+    const content = await fs.readFile(path.join(skillPath, 'SKILL.md'), 'utf-8');
+    structureChecks.checks.push({
+      name: 'Has title',
+      passed: /^#\s+/.test(content),
+    });
+    structureChecks.checks.push({
+      name: 'Has overview',
+      passed: /overview|introduction|about/i.test(content),
+    });
+    structureChecks.checks.push({
+      name: 'Size reasonable',
+      passed: content.length < 100000,
+      message: `${(content.length / 1024).toFixed(1)} KB`,
+    });
+  } catch {
+    structureChecks.checks.push({ name: 'Content readable', passed: false });
+  }
+
+  results.push(structureChecks);
+
+  // Category 2: Loading & Performance
+  const perfChecks: ValidationResult = {
+    category: 'Performance',
+    checks: [],
+  };
+
+  try {
+    loader.clearCache(skillName);
+    const start1 = Date.now();
+    await loader.load(skillName);
+    const loadTime = Date.now() - start1;
+
+    perfChecks.checks.push({
+      name: 'First load < 100ms',
+      passed: loadTime < 100,
+      message: `${loadTime}ms`,
+    });
+
+    const start2 = Date.now();
+    await loader.load(skillName);
+    const cacheTime = Date.now() - start2;
+
+    perfChecks.checks.push({
+      name: 'Cached load < 10ms',
+      passed: cacheTime < 10,
+      message: `${cacheTime}ms`,
+    });
+  } catch (error) {
+    perfChecks.checks.push({
+      name: 'Loads without errors',
+      passed: false,
+      message: String(error),
+    });
+  }
+
+  results.push(perfChecks);
+
+  // Category 3: Content Quality
+  const contentChecks: ValidationResult = {
+    category: 'Content Quality',
+    checks: [],
+  };
+
+  try {
+    const skill = await loader.load(skillName);
+    const content = skill.content;
+
+    contentChecks.checks.push({
+      name: 'Has examples',
+      passed: content.includes('```'),
+    });
+
+    contentChecks.checks.push({
+      name: 'Has structure (headings)',
+      passed: content.includes('##'),
+    });
+
+    contentChecks.checks.push({
+      name: 'Not too short',
+      passed: content.length > 500,
+      message: `${content.length} chars`,
+    });
+
+    contentChecks.checks.push({
+      name: 'Has actionable content',
+      passed: /how to|step|example|guide|use/i.test(content),
+    });
+  } catch (error) {
+    contentChecks.checks.push({
+      name: 'Content validation',
+      passed: false,
+      message: String(error),
+    });
+  }
+
+  results.push(contentChecks);
+
+  // Category 4: Keyword Matching
+  const keywordChecks: ValidationResult = {
+    category: 'Keyword Matching',
+    checks: [],
+  };
+
+  const mappings = manager.getSkillMappings();
+  const mapping = mappings.get(skillName);
+
+  if (mapping) {
+    keywordChecks.checks.push({
+      name: 'Keywords defined',
+      passed: mapping.keywords.length > 0,
+      message: `${mapping.keywords.length} keywords`,
+    });
+
+    keywordChecks.checks.push({
+      name: 'Tools defined',
+      passed: mapping.tools !== undefined,
+      message: `${mapping.tools?.length || 0} tools`,
+    });
+
+    // Test primary keyword
+    if (mapping.keywords.length > 0) {
+      const guidance = await manager.analyzeRequest(mapping.keywords[0]);
+      keywordChecks.checks.push({
+        name: 'Primary keyword matches',
+        passed: guidance.skills.includes(skillName),
+        message: `"${mapping.keywords[0]}"`,
+      });
+    }
+  } else {
+    keywordChecks.checks.push({
+      name: 'Registered in SkillManager',
+      passed: false,
+    });
+  }
+
+  results.push(keywordChecks);
+
+  // Print results
+  for (const result of results) {
+    console.log(`\n${result.category}`);
+    console.log('─'.repeat(50));
+
+    for (const check of result.checks) {
+      totalChecks++;
+      const icon = check.passed ? '✅' : '❌';
+      const msg = check.message ? ` (${check.message})` : '';
+      console.log(`${icon} ${check.name}${msg}`);
+
+      if (check.passed) totalPassed++;
+    }
+  }
+
+  // Summary
+  const passRate = (totalPassed / totalChecks * 100).toFixed(1);
+  console.log('\n' + '='.repeat(50));
+  console.log(`Results: ${totalPassed}/${totalChecks} checks passed (${passRate}%)`);
+  console.log('='.repeat(50) + '\n');
+
+  const passed = totalPassed === totalChecks;
+  if (passed) {
+    console.log(`✅ ${skillName} passed all validation checks!\n`);
+  } else {
+    console.log(`❌ ${skillName} has ${totalChecks - totalPassed} failing checks\n`);
+  }
+
+  return passed;
+}
+
+// CLI usage
+const skillName = process.argv[2];
+if (!skillName) {
+  console.error('Usage: npx tsx validate-skill.ts <skill-name>');
+  process.exit(1);
+}
+
+validateSkill(skillName)
+  .then(passed => process.exit(passed ? 0 : 1))
+  .catch(error => {
+    console.error('Validation error:', error);
+    process.exit(1);
+  });
+```
+
+## Manual Review Checklist
+
+After automated validation passes, perform manual review:
+
+### Content Review (10-15 minutes)
+```
+□ Read through entire skill content
+□ Verify examples are clear and useful
+□ Check for typos and grammar
+□ Ensure logical flow and organization
+□ Validate technical accuracy
+□ Confirm examples can be copied and used
+```
+
+### User Testing (15-20 minutes)
+```
+□ Test with 5 different user queries
+□ Verify skill triggers appropriately
+□ Check that content helps LLM respond well
+□ Test with both OpenAI and Anthropic (if available)
+□ Gather feedback from team members
+□ Validate recommended tools are correct
+```
+
+### Integration Testing (10 minutes)
+```
+□ Test in development environment
+□ Verify no errors in logs
+□ Check performance is acceptable
+□ Ensure graceful error handling
+□ Validate caching works
+□ Test with concurrent requests
+```
+
+## Post-Deployment Monitoring
+
+After deploying a skill to production:
+
+### Week 1 Monitoring
+```
+□ Monitor error rates
+□ Check load times
+□ Review skill matching accuracy
+□ Collect user feedback
+□ Track usage metrics
+□ Identify any issues
+```
+
+### Week 2-4 Iteration
+```
+□ Address reported issues
+□ Refine keywords based on usage
+□ Update content based on feedback
+□ Optimize performance if needed
+□ Improve examples
+□ Update documentation
+```
+
+### Monthly Review
+```
+□ Review usage statistics
+□ Assess content relevance
+□ Check for outdated information
+□ Compare with similar skills
+□ Identify improvement opportunities
+□ Plan updates if needed
+```
+
+## Quality Gates
+
+Define quality gates that must pass before deployment:
+
+### Gate 1: Automated Tests
+- All unit tests pass
+- All integration tests pass
+- Performance benchmarks met
+- No critical errors
+
+### Gate 2: Manual Review
+- Content reviewed by at least one other person
+- Examples tested manually
+- Technical accuracy verified
+- No major issues found
+
+### Gate 3: Staging Testing
+- Tested in staging environment
+- Works with all providers
+- No performance regressions
+- Logs are clean
+
+### Gate 4: Approval
+- Team lead approves
+- No blocking issues
+- Documentation complete
+- Rollback plan ready
+
+## Failure Response
+
+If validation fails:
+
+1. **Document the Issues**: Record all failing checks
+2. **Prioritize Fixes**: Address critical issues first
+3. **Fix and Retest**: Make corrections and re-run validation
+4. **Review Root Cause**: Understand why issues occurred
+5. **Update Process**: Improve process to prevent recurrence
+
+## Success Criteria
+
+A skill is ready for production when:
+
+- ✅ All automated checks pass
+- ✅ Manual review complete and approved
+- ✅ User testing shows positive results
+- ✅ Performance meets targets
+- ✅ No blocking issues identified
+- ✅ Team approval obtained
+- ✅ Monitoring plan in place
+
+## Continuous Improvement
+
+After deployment:
+
+1. **Monitor Metrics**: Track usage, performance, errors
+2. **Collect Feedback**: From users and LLM responses
+3. **Identify Issues**: Proactively find problems
+4. **Iterate Quickly**: Make small, frequent improvements
+5. **Share Learnings**: Update process and best practices