@diottodev/runic 1.0.0 → 1.1.0

package/engineering/skills/skill-tester/README.md

@@ -0,0 +1,229 @@
+# Skill Tester - Quality Assurance Meta-Skill
+
+A POWERFUL-tier skill that provides comprehensive validation, testing, and quality scoring for skills in the claude-skills ecosystem.
+
+## Overview
+
+The Skill Tester is a meta-skill that ensures quality and consistency across all skills in the repository through:
+
+- **Structure Validation** - Verifies directory structure, file presence, and documentation standards
+- **Script Testing** - Tests Python scripts for syntax, functionality, and compliance
+- **Quality Scoring** - Provides comprehensive quality assessment across multiple dimensions
+
+## Quick Start
+
+### Validate a Skill
+```bash
+# Basic validation
+python scripts/skill_validator.py engineering/my-skill
+
+# Validate against specific tier
+python scripts/skill_validator.py engineering/my-skill --tier POWERFUL --json
+```
+
+### Test Scripts
+```bash
+# Test all scripts in a skill
+python scripts/script_tester.py engineering/my-skill
+
+# Test with custom timeout
+python scripts/script_tester.py engineering/my-skill --timeout 60 --json
+```
+
+### Score Quality
+```bash
+# Get quality assessment
+python scripts/quality_scorer.py engineering/my-skill
+
+# Detailed scoring with improvement suggestions
+python scripts/quality_scorer.py engineering/my-skill --detailed --json
+```
+
+## Components
+
+### Scripts
+- **skill_validator.py** (700+ LOC) - Validates skill structure and compliance
+- **script_tester.py** (800+ LOC) - Tests script functionality and quality
+- **quality_scorer.py** (1100+ LOC) - Multi-dimensional quality assessment
+
+### Reference Documentation
+- **skill-structure-specification.md** - Complete structural requirements
+- **tier-requirements-matrix.md** - Tier-specific quality standards
+- **quality-scoring-rubric.md** - Detailed scoring methodology
+
+### Sample Assets
+- **sample-skill/** - Complete sample skill for testing the tester itself
+
+## Features
+
+### Validation Capabilities
+- SKILL.md format and content validation
+- Directory structure compliance checking
+- Python script syntax and import validation
+- Argparse implementation verification
+- Tier-specific requirement enforcement
+
+### Testing Framework
+- Syntax validation using AST parsing
+- Import analysis for external dependencies
+- Runtime execution testing with timeout protection
+- Help functionality verification
+- Sample data processing validation
+- Output format compliance checking
+
+### Quality Assessment
+- Documentation quality scoring (25%)
+- Code quality evaluation (25%)  
+- Completeness assessment (25%)
+- Usability analysis (25%)
+- Letter grade assignment (A+ to F)
+- Tier recommendation generation
+- Improvement roadmap creation
+
+## CI/CD Integration
+
+### GitHub Actions Example
+```yaml
+name: Skill Quality Gate
+on:
+  pull_request:
+    paths: ['engineering/**']
+    
+jobs:
+  validate-skills:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Setup Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.11'
+      - name: Validate Skills
+        run: |
+          for skill in $(git diff --name-only ${{ github.event.before }} | grep -E '^engineering/[^/]+/' | cut -d'/' -f1-2 | sort -u); do
+            python engineering/skill-tester/scripts/skill_validator.py $skill --json
+            python engineering/skill-tester/scripts/script_tester.py $skill
+            python engineering/skill-tester/scripts/quality_scorer.py $skill --minimum-score 75
+          done
+```
+
+### Pre-commit Hook
+```bash
+#!/bin/bash
+# .git/hooks/pre-commit
+python engineering/skill-tester/scripts/skill_validator.py engineering/my-skill --tier STANDARD
+if [ $? -ne 0 ]; then
+    echo "Skill validation failed. Commit blocked."
+    exit 1
+fi
+```
+
+## Quality Standards
+
+### All Scripts
+- **Zero External Dependencies** - Python standard library only
+- **Comprehensive Error Handling** - Meaningful error messages and recovery
+- **Dual Output Support** - Both JSON and human-readable formats
+- **Proper Documentation** - Comprehensive docstrings and comments
+- **CLI Best Practices** - Full argparse implementation with help text
+
+### Validation Accuracy
+- **Structure Checks** - 100% accurate directory and file validation
+- **Content Analysis** - Deep parsing of SKILL.md and documentation
+- **Code Analysis** - AST-based Python code validation
+- **Compliance Scoring** - Objective, repeatable quality assessment
+
+## Self-Testing
+
+The skill-tester can validate itself:
+
+```bash
+# Validate the skill-tester structure
+python scripts/skill_validator.py . --tier POWERFUL
+
+# Test the skill-tester scripts
+python scripts/script_tester.py .
+
+# Score the skill-tester quality
+python scripts/quality_scorer.py . --detailed
+```
+
+## Advanced Usage
+
+### Batch Validation
+```bash
+# Validate all skills in repository
+find engineering/ -maxdepth 1 -type d | while read skill; do
+  echo "Validating $skill..."
+  python engineering/skill-tester/scripts/skill_validator.py "$skill"
+done
+```
+
+### Quality Monitoring
+```bash
+# Generate quality report for all skills
+python engineering/skill-tester/scripts/quality_scorer.py engineering/ \
+  --batch --json > quality_report.json
+```
+
+### Custom Scoring Thresholds
+```bash
+# Enforce minimum quality scores
+python scripts/quality_scorer.py engineering/my-skill --minimum-score 80
+# Exit code 0 = passed, 1 = failed, 2 = needs improvement
+```
+
+## Error Handling
+
+All scripts provide comprehensive error handling:
+- **File System Errors** - Missing files, permission issues, invalid paths
+- **Content Errors** - Malformed YAML, invalid JSON, encoding issues  
+- **Execution Errors** - Script timeouts, runtime failures, import errors
+- **Validation Errors** - Standards violations, compliance failures
+
+## Output Formats
+
+### Human-Readable
+```
+=== SKILL VALIDATION REPORT ===
+Skill: engineering/my-skill
+Overall Score: 85.2/100 (B+)
+Tier Recommendation: STANDARD
+
+STRUCTURE VALIDATION:
+  ✓ PASS: SKILL.md found
+  ✓ PASS: README.md found
+  ✓ PASS: scripts/ directory found
+
+SUGGESTIONS:
+  • Add references/ directory
+  • Improve error handling in main.py
+```
+
+### JSON Format
+```json
+{
+  "skill_path": "engineering/my-skill",
+  "overall_score": 85.2,
+  "letter_grade": "B+",
+  "tier_recommendation": "STANDARD",
+  "dimensions": {
+    "Documentation": {"score": 88.5, "weight": 0.25},
+    "Code Quality": {"score": 82.0, "weight": 0.25},
+    "Completeness": {"score": 85.5, "weight": 0.25},
+    "Usability": {"score": 84.8, "weight": 0.25}
+  }
+}
+```
+
+## Requirements
+
+- **Python 3.7+** - No external dependencies required
+- **File System Access** - Read access to skill directories  
+- **Execution Permissions** - Ability to run Python scripts for testing
+
+## Contributing
+
+See [SKILL.md](SKILL.md) for comprehensive documentation and contribution guidelines.
+
+The skill-tester itself serves as a reference implementation of POWERFUL-tier quality standards.

package/engineering/skills/skill-tester/SKILL.md

@@ -0,0 +1,390 @@
+---
+name: "skill-tester"
+description: "Validate, test, and score the quality of skills within the claude-skills ecosystem. Comprehensive meta-skill: structure validation, Python script testing (syntax + imports + runtime + output format), multi-dimensional quality scoring with letter grades and tier classification (BASIC/STANDARD/POWERFUL). Use when authoring a new skill, auditing existing skills for tier promotion, setting up pre-commit hooks for skill quality, or integrating skill QA into CI."
+---
+
+# Skill Tester
+
+---
+
+**Name**: skill-tester
+**Tier**: POWERFUL
+**Category**: Engineering Quality Assurance
+**Dependencies**: None (Python Standard Library Only)
+**Author**: Claude Skills Engineering Team
+**Version**: 1.0.0
+**Last Updated**: 2026-02-16
+
+---
+
+## Description
+
+The Skill Tester is a comprehensive meta-skill designed to validate, test, and score the quality of skills within the claude-skills ecosystem. This powerful quality assurance tool ensures that all skills meet the rigorous standards required for BASIC, STANDARD, and POWERFUL tier classifications through automated validation, testing, and scoring mechanisms.
+
+As the gatekeeping system for skill quality, this meta-skill provides three core capabilities:
+1. **Structure Validation** - Ensures skills conform to required directory structures, file formats, and documentation standards
+2. **Script Testing** - Validates Python scripts for syntax, imports, functionality, and output format compliance  
+3. **Quality Scoring** - Provides comprehensive quality assessment across multiple dimensions with letter grades and improvement recommendations
+
+This skill is essential for maintaining ecosystem consistency, enabling automated CI/CD integration, and supporting both manual and automated quality assurance workflows. It serves as the foundation for pre-commit hooks, pull request validation, and continuous integration processes that maintain the high-quality standards of the claude-skills repository.
+
+## Core Features
+
+### Comprehensive Skill Validation
+- **Structure Compliance**: Validates directory structure, required files (SKILL.md, README.md, scripts/, references/, assets/, expected_outputs/)
+- **Documentation Standards**: Checks SKILL.md frontmatter, section completeness, minimum line counts per tier
+- **File Format Validation**: Ensures proper Markdown formatting, YAML frontmatter syntax, and file naming conventions
+
+### Advanced Script Testing
+- **Syntax Validation**: Compiles Python scripts to detect syntax errors before execution
+- **Import Analysis**: Enforces standard library only policy, identifies external dependencies
+- **Runtime Testing**: Executes scripts with sample data, validates argparse implementation, tests --help functionality
+- **Output Format Compliance**: Verifies dual output support (JSON + human-readable), proper error handling
+
+### Multi-Dimensional Quality Scoring
+- **Documentation Quality (25%)**: SKILL.md depth and completeness, README clarity, reference documentation quality
+- **Code Quality (25%)**: Script complexity, error handling robustness, output format consistency, maintainability
+- **Completeness (25%)**: Required directory presence, sample data adequacy, expected output verification
+- **Usability (25%)**: Example clarity, argparse help text quality, installation simplicity, user experience
+
+### Tier Classification System
+Automatically classifies skills based on complexity and functionality:
+
+#### BASIC Tier Requirements
+- Minimum 100 lines in SKILL.md
+- At least 1 Python script (100-300 LOC)
+- Basic argparse implementation
+- Simple input/output handling
+- Essential documentation coverage
+
+#### STANDARD Tier Requirements  
+- Minimum 200 lines in SKILL.md
+- 1-2 Python scripts (300-500 LOC each)
+- Advanced argparse with subcommands
+- JSON + text output formats
+- Comprehensive examples and references
+- Error handling and edge case management
+
+#### POWERFUL Tier Requirements
+- Minimum 300 lines in SKILL.md
+- 2-3 Python scripts (500-800 LOC each)
+- Complex argparse with multiple modes
+- Sophisticated output formatting and validation
+- Extensive documentation and reference materials
+- Advanced error handling and recovery mechanisms
+- CI/CD integration capabilities
+
+## Architecture & Design
+
+### Modular Design Philosophy
+The skill-tester follows a modular architecture where each component serves a specific validation purpose:
+
+- **skill_validator.py**: Core structural and documentation validation engine
+- **script_tester.py**: Runtime testing and execution validation framework  
+- **quality_scorer.py**: Multi-dimensional quality assessment and scoring system
+
+### Standards Enforcement
+All validation is performed against well-defined standards documented in the references/ directory:
+- **Skill Structure Specification**: Defines mandatory and optional components
+- **Tier Requirements Matrix**: Detailed requirements for each skill tier
+- **Quality Scoring Rubric**: Comprehensive scoring methodology and weightings
+
+### Integration Capabilities
+Designed for seamless integration into existing development workflows:
+- **Pre-commit Hooks**: Prevents substandard skills from being committed
+- **CI/CD Pipelines**: Automated quality gates in pull request workflows
+- **Manual Validation**: Interactive command-line tools for development-time validation
+- **Batch Processing**: Bulk validation and scoring of existing skill repositories
+
+## Implementation Details
+
+### skill_validator.py Core Functions
+```python
+# Primary validation workflow
+validate_skill_structure() -> ValidationReport
+check_skill_md_compliance() -> DocumentationReport  
+validate_python_scripts() -> ScriptReport
+generate_compliance_score() -> float
+```
+
+Key validation checks include:
+- SKILL.md frontmatter parsing and validation
+- Required section presence (Description, Features, Usage, etc.)
+- Minimum line count enforcement per tier
+- Python script argparse implementation verification
+- Standard library import enforcement
+- Directory structure compliance
+- README.md quality assessment
+
+### script_tester.py Testing Framework
+```python
+# Core testing functions
+syntax_validation() -> SyntaxReport
+import_validation() -> ImportReport
+runtime_testing() -> RuntimeReport
+output_format_validation() -> OutputReport
+```
+
+Testing capabilities encompass:
+- Python AST-based syntax validation
+- Import statement analysis and external dependency detection
+- Controlled script execution with timeout protection
+- Argparse --help functionality verification
+- Sample data processing and output validation
+- Expected output comparison and difference reporting
+
+### quality_scorer.py Scoring System
+```python
+# Multi-dimensional scoring
+score_documentation() -> float  # 25% weight
+score_code_quality() -> float   # 25% weight
+score_completeness() -> float   # 25% weight
+score_usability() -> float      # 25% weight
+calculate_overall_grade() -> str # A-F grade
+```
+
+Scoring dimensions include:
+- **Documentation**: Completeness, clarity, examples, reference quality
+- **Code Quality**: Complexity, maintainability, error handling, output consistency
+- **Completeness**: Required files, sample data, expected outputs, test coverage  
+- **Usability**: Help text quality, example clarity, installation simplicity
+
+## Usage Scenarios
+
+### Development Workflow Integration
+```bash
+# Pre-commit hook validation
+skill_validator.py path/to/skill --tier POWERFUL --json
+
+# Comprehensive skill testing
+script_tester.py path/to/skill --timeout 30 --sample-data
+
+# Quality assessment and scoring
+quality_scorer.py path/to/skill --detailed --recommendations
+```
+
+### CI/CD Pipeline Integration
+```yaml
+# GitHub Actions workflow example
+- name: "validate-skill-quality"
+  run: |
+    python skill_validator.py engineering/${{ matrix.skill }} --json | tee validation.json
+    python script_tester.py engineering/${{ matrix.skill }} | tee testing.json
+    python quality_scorer.py engineering/${{ matrix.skill }} --json | tee scoring.json
+```
+
+### Batch Repository Analysis
+```bash
+# Validate all skills in repository
+find engineering/ -type d -maxdepth 1 | xargs -I {} skill_validator.py {}
+
+# Generate repository quality report
+quality_scorer.py engineering/ --batch --output-format json > repo_quality.json
+```
+
+## Output Formats & Reporting
+
+### Dual Output Support
+All tools provide both human-readable and machine-parseable output:
+
+#### Human-Readable Format
+```
+=== SKILL VALIDATION REPORT ===
+Skill: engineering/example-skill
+Tier: STANDARD
+Overall Score: 85/100 (B)
+
+Structure Validation: ✓ PASS
+├─ SKILL.md: ✓ EXISTS (247 lines)
+├─ README.md: ✓ EXISTS  
+├─ scripts/: ✓ EXISTS (2 files)
+└─ references/: ⚠ MISSING (recommended)
+
+Documentation Quality: 22/25 (88%)
+Code Quality: 20/25 (80%)
+Completeness: 18/25 (72%)
+Usability: 21/25 (84%)
+
+Recommendations:
+• Add references/ directory with documentation
+• Improve error handling in main.py
+• Include more comprehensive examples
+```
+
+#### JSON Format
+```json
+{
+  "skill_path": "engineering/example-skill",
+  "timestamp": "2026-02-16T16:41:00Z",
+  "validation_results": {
+    "structure_compliance": {
+      "score": 0.95,
+      "checks": {
+        "skill_md_exists": true,
+        "readme_exists": true,
+        "scripts_directory": true,
+        "references_directory": false
+      }
+    },
+    "overall_score": 85,
+    "letter_grade": "B",
+    "tier_recommendation": "STANDARD",
+    "improvement_suggestions": [
+      "Add references/ directory",
+      "Improve error handling",
+      "Include comprehensive examples"
+    ]
+  }
+}
+```
+
+## Quality Assurance Standards
+
+### Code Quality Requirements
+- **Standard Library Only**: No external dependencies (pip packages)
+- **Error Handling**: Comprehensive exception handling with meaningful error messages
+- **Output Consistency**: Standardized JSON schema and human-readable formatting
+- **Performance**: Efficient validation algorithms with reasonable execution time
+- **Maintainability**: Clear code structure, comprehensive docstrings, type hints where appropriate
+
+### Testing Standards  
+- **Self-Testing**: The skill-tester validates itself (meta-validation)
+- **Sample Data Coverage**: Comprehensive test cases covering edge cases and error conditions
+- **Expected Output Verification**: All sample runs produce verifiable, reproducible outputs
+- **Timeout Protection**: Safe execution of potentially problematic scripts with timeout limits
+
+### Documentation Standards
+- **Comprehensive Coverage**: All functions, classes, and modules documented
+- **Usage Examples**: Clear, practical examples for all use cases
+- **Integration Guides**: Step-by-step CI/CD and workflow integration instructions
+- **Reference Materials**: Complete specification documents for standards and requirements
+
+## Integration Examples
+
+### Pre-Commit Hook Setup
+```bash
+#!/bin/bash
+# .git/hooks/pre-commit
+echo "Running skill validation..."
+python engineering/skill-tester/scripts/skill_validator.py engineering/new-skill --tier STANDARD
+if [ $? -ne 0 ]; then
+    echo "Skill validation failed. Commit blocked."
+    exit 1
+fi
+echo "Validation passed. Proceeding with commit."
+```
+
+### GitHub Actions Workflow
+```yaml
+name: "skill-quality-gate"
+on:
+  pull_request:
+    paths: ['engineering/**']
+
+jobs:
+  validate-skills:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: "setup-python"
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.11'
+      - name: "validate-changed-skills"
+        run: |
+          changed_skills=$(git diff --name-only ${{ github.event.before }} | grep -E '^engineering/[^/]+/' | cut -d'/' -f1-2 | sort -u)
+          for skill in $changed_skills; do
+            echo "Validating $skill..."
+            python engineering/skill-tester/scripts/skill_validator.py $skill --json
+            python engineering/skill-tester/scripts/script_tester.py $skill
+            python engineering/skill-tester/scripts/quality_scorer.py $skill --minimum-score 75
+          done
+```
+
+### Continuous Quality Monitoring
+```bash
+#!/bin/bash
+# Daily quality report generation
+echo "Generating daily skill quality report..."
+timestamp=$(date +"%Y-%m-%d")
+python engineering/skill-tester/scripts/quality_scorer.py engineering/ \
+  --batch --json > "reports/quality_report_${timestamp}.json"
+
+echo "Quality trends analysis..."
+python engineering/skill-tester/scripts/trend_analyzer.py reports/ \
+  --days 30 > "reports/quality_trends_${timestamp}.md"
+```
+
+## Performance & Scalability
+
+### Execution Performance
+- **Fast Validation**: Structure validation completes in <1 second per skill
+- **Efficient Testing**: Script testing with timeout protection (configurable, default 30s)
+- **Batch Processing**: Optimized for repository-wide analysis with parallel processing support
+- **Memory Efficiency**: Minimal memory footprint for large-scale repository analysis
+
+### Scalability Considerations
+- **Repository Size**: Designed to handle repositories with 100+ skills
+- **Concurrent Execution**: Thread-safe implementation supports parallel validation
+- **Resource Management**: Automatic cleanup of temporary files and subprocess resources
+- **Configuration Flexibility**: Configurable timeouts, memory limits, and validation strictness
+
+## Security & Safety
+
+### Safe Execution Environment
+- **Sandboxed Testing**: Scripts execute in controlled environment with timeout protection
+- **Resource Limits**: Memory and CPU usage monitoring to prevent resource exhaustion
+- **Input Validation**: All inputs sanitized and validated before processing
+- **No Network Access**: Offline operation ensures no external dependencies or network calls
+
+### Security Best Practices
+- **No Code Injection**: Static analysis only, no dynamic code generation
+- **Path Traversal Protection**: Secure file system access with path validation
+- **Minimal Privileges**: Operates with minimal required file system permissions
+- **Audit Logging**: Comprehensive logging for security monitoring and troubleshooting
+
+## Troubleshooting & Support
+
+### Common Issues & Solutions
+
+#### Validation Failures
+- **Missing Files**: Check directory structure against tier requirements
+- **Import Errors**: Ensure only standard library imports are used
+- **Documentation Issues**: Verify SKILL.md frontmatter and section completeness
+
+#### Script Testing Problems  
+- **Timeout Errors**: Increase timeout limit or optimize script performance
+- **Execution Failures**: Check script syntax and import statement validity
+- **Output Format Issues**: Ensure proper JSON formatting and dual output support
+
+#### Quality Scoring Discrepancies
+- **Low Scores**: Review scoring rubric and improvement recommendations
+- **Tier Misclassification**: Verify skill complexity against tier requirements
+- **Inconsistent Results**: Check for recent changes in quality standards or scoring weights
+
+### Debugging Support
+- **Verbose Mode**: Detailed logging and execution tracing available
+- **Dry Run Mode**: Validation without execution for debugging purposes
+- **Debug Output**: Comprehensive error reporting with file locations and suggestions
+
+## Future Enhancements
+
+### Planned Features
+- **Machine Learning Quality Prediction**: AI-powered quality assessment using historical data
+- **Performance Benchmarking**: Execution time and resource usage tracking across skills
+- **Dependency Analysis**: Automated detection and validation of skill interdependencies
+- **Quality Trend Analysis**: Historical quality tracking and regression detection
+
+### Integration Roadmap
+- **IDE Plugins**: Real-time validation in popular development environments
+- **Web Dashboard**: Centralized quality monitoring and reporting interface
+- **API Endpoints**: RESTful API for external integration and automation
+- **Notification Systems**: Automated alerts for quality degradation or validation failures
+
+## Conclusion
+
+The Skill Tester represents a critical infrastructure component for maintaining the high-quality standards of the claude-skills ecosystem. By providing comprehensive validation, testing, and scoring capabilities, it ensures that all skills meet or exceed the rigorous requirements for their respective tiers.
+
+This meta-skill not only serves as a quality gate but also as a development tool that guides skill authors toward best practices and helps maintain consistency across the entire repository. Through its integration capabilities and comprehensive reporting, it enables both manual and automated quality assurance workflows that scale with the growing claude-skills ecosystem.
+
+The combination of structural validation, runtime testing, and multi-dimensional quality scoring provides unparalleled visibility into skill quality while maintaining the flexibility needed for diverse skill types and complexity levels. As the claude-skills repository continues to grow, the Skill Tester will remain the cornerstone of quality assurance and ecosystem integrity.

package/engineering/skills/skill-tester/assets/sample-skill/README.md

@@ -0,0 +1,40 @@
+# Sample Text Processor
+
+A basic text processing skill that demonstrates BASIC tier requirements for the claude-skills ecosystem.
+
+## Quick Start
+
+```bash
+# Analyze a text file
+python scripts/text_processor.py analyze sample.txt
+
+# Get JSON output
+python scripts/text_processor.py analyze sample.txt --format json
+
+# Transform text to uppercase  
+python scripts/text_processor.py transform sample.txt --mode upper
+
+# Process multiple files
+python scripts/text_processor.py batch text_files/ --verbose
+```
+
+## Features
+
+- Word count and text statistics
+- Text transformations (upper, lower, title, reverse)
+- Batch file processing
+- JSON and human-readable output formats
+- Comprehensive error handling
+
+## Requirements
+
+- Python 3.7 or later
+- No external dependencies (standard library only)
+
+## Usage
+
+See [SKILL.md](SKILL.md) for comprehensive documentation and examples.
+
+## Testing
+
+Sample data files are provided in the `assets/` directory for testing the functionality.