@softspark/ai-toolkit 1.0.0

package/app/skills/evaluate/SKILL.md

@@ -0,0 +1,132 @@
+---
+name: evaluate
+description: "Evaluate skill quality and RAG retrieval accuracy"
+effort: medium
+disable-model-invocation: true
+argument-hint: "[--threshold N]"
+allowed-tools: Bash, Read
+---
+
+# RAG Evaluation
+
+Evaluate RAG quality using LLM-as-a-Judge methodology.
+
+## Usage
+
+```
+/evaluate [--threshold 0.7]
+```
+
+## Execution
+
+### Direct Execution (recommended for most projects)
+```bash
+# Run RAG evaluation
+python3 scripts/evaluate_rag.py
+
+# With custom thresholds
+python3 scripts/evaluate_rag.py \
+  --faithfulness 0.7 \
+  --relevancy 0.7 \
+  --context 0.6
+
+# Detect knowledge gaps
+python3 scripts/knowledge_gaps.py --detect
+
+# Generate gap report
+python3 scripts/knowledge_gaps.py --report
+```
+
+### Docker Execution (containerized projects)
+```bash
+# Replace {api-container} with your API server container name
+docker exec {api-container} python3 scripts/evaluate_rag.py
+
+# With custom thresholds
+docker exec {api-container} python3 scripts/evaluate_rag.py \
+  --faithfulness 0.7 \
+  --relevancy 0.7 \
+  --context 0.6
+
+# Detect knowledge gaps
+docker exec {api-container} python3 scripts/knowledge_gaps.py --detect
+
+# Generate gap report
+docker exec {api-container} python3 scripts/knowledge_gaps.py --report
+```
+
+## Metrics
+
+| Metric | Description | Target |
+|--------|-------------|--------|
+| **Faithfulness** | Is answer based on context? | >70% |
+| **Relevancy** | Does answer address question? | >70% |
+| **Context Precision** | Is found context accurate? | >60% |
+
+## Evaluation Process
+
+1. **Generate test queries** from golden dataset
+2. **Execute RAG pipeline** for each query
+3. **LLM judges** each response on metrics
+4. **Report** aggregate scores
+
+## Golden Dataset
+
+Located at: `scripts/golden_dataset.json` (or project-specific path)
+
+```json
+{
+  "queries": [
+    {
+      "query": "How to configure rate limiting?",
+      "expected_topics": ["nginx", "rate-limiting"],
+      "expected_sources": ["kb/nginx/howto/rate-limiting.md"]
+    }
+  ]
+}
+```
+
+## Output Example
+
+```
+RAG Evaluation Results
+======================
+Total Queries: 50
+Average Faithfulness: 0.82
+Average Relevancy: 0.78
+Average Context Precision: 0.71
+
+Quality: GOOD
+
+Failed Queries (faithfulness < 0.7):
+- Query: "How to backup PostgreSQL?"
+  Score: 0.45
+  Issue: No relevant documents found
+```
+
+## Knowledge Gaps
+
+After evaluation, check for gaps:
+
+```bash
+# Direct execution
+python3 scripts/knowledge_gaps.py --detect
+
+# Docker execution
+docker exec {api-container} python3 scripts/knowledge_gaps.py --detect
+```
+
+Output:
+```
+Knowledge Gaps Detected:
+1. PostgreSQL backup procedures (5 failed queries)
+2. Redis caching configuration (3 failed queries)
+3. Ollama model selection (2 failed queries)
+```
+
+## Quality Gates
+
+- [ ] Faithfulness >70%
+- [ ] Relevancy >70%
+- [ ] Context Precision >60%
+- [ ] No critical knowledge gaps

package/app/skills/evolve/SKILL.md

@@ -0,0 +1,27 @@
+---
+name: evolve
+description: "Evolve agent definitions via meta-architect"
+effort: medium
+disable-model-invocation: true
+context: fork
+agent: meta-architect
+allowed-tools: Read, Edit, Grep, Glob
+---
+
+# Evolve Command
+
+Triggers the Meta-Architect to improve the system.
+
+## Usage
+
+```bash
+/evolve [source]
+# Example: /evolve learnings (Analyze kb/learnings)
+# Example: /evolve last-failure (Analyze last error log)
+```
+
+## Protocol
+1. **Analyze**: Read input source for patterns of failure/inefficiency.
+2. **Design**: Draft changes to `.claude/agents/` or `.claude/skills/`.
+3. **Implement**: Apply changes.
+4. **Report**: Document what evolved.

package/app/skills/explain/SKILL.md

@@ -0,0 +1,54 @@
+---
+name: explain
+description: "Explain code, architecture, or concepts with diagrams"
+user-invocable: true
+effort: medium
+argument-hint: "[file or module path]"
+agent: code-archaeologist
+context: fork
+allowed-tools: Read, Grep, Glob
+---
+
+# Explain
+
+$ARGUMENTS
+
+Generates visual architecture explanations.
+
+## Output Format
+
+### 1. High-Level Role
+"This module handles [Responsibility]. It interacts with [Dependencies]."
+
+### 2. Dependency Graph (Mermaid)
+Generate a graph showing imports/exports.
+```mermaid
+graph TD
+    A[AuthService] -->|uses| B[UserRepo]
+    A -->|validates| C[Schema]
+    D[Controller] -->|calls| A
+```
+
+### 3. Key Flows (Sequence)
+If logical flows are detected:
+```mermaid
+sequenceDiagram
+    User->>Controller: Login
+    Controller->>Service: Validate
+    Service->>DB: Check Creds
+    DB-->>Service: Result
+    Service-->>Controller: Token
+```
+
+## Protocol
+1. **Scan**: Read file contents to identify classes and functions.
+2. **Link**: Identify imports to find collaborators.
+3. **Visualize**: Generate standard Mermaid syntax.
+
+## Automated Dependency Graph
+
+Run the bundled script to extract imports and generate a Mermaid diagram:
+
+```bash
+python3 ${CLAUDE_SKILL_DIR}/scripts/dependency-graph.py src/auth.py
+```

package/app/skills/explain/scripts/dependency-graph.py

@@ -0,0 +1,215 @@
+#!/usr/bin/env python3
+"""Extract import graph from source files and generate Mermaid syntax."""
+
+import json
+import os
+import re
+import sys
+from pathlib import Path
+
+IGNORE_DIRS = {'.git', 'node_modules', '__pycache__', '.venv', 'venv', 'dist', 'build', '.next', 'vendor', '.cache'}
+
+LANG_MAP = {
+    '.py': 'python',
+    '.js': 'javascript',
+    '.ts': 'typescript',
+    '.tsx': 'typescript',
+    '.jsx': 'javascript',
+    '.go': 'go',
+    '.php': 'php',
+}
+
+SOURCE_EXTS = set(LANG_MAP.keys())
+
+IMPORT_PATTERNS = {
+    'python': [
+        re.compile(r'^\s*import\s+([\w.]+)'),
+        re.compile(r'^\s*from\s+([\w.]+)\s+import'),
+    ],
+    'javascript': [
+        re.compile(r'''import\s+.*?from\s+['"]([^'"]+)['"]'''),
+        re.compile(r'''require\s*\(\s*['"]([^'"]+)['"]\s*\)'''),
+    ],
+    'typescript': [
+        re.compile(r'''import\s+.*?from\s+['"]([^'"]+)['"]'''),
+        re.compile(r'''require\s*\(\s*['"]([^'"]+)['"]\s*\)'''),
+    ],
+    'go': [
+        re.compile(r'^\s*"([^"]+)"'),
+        re.compile(r'^\s*import\s+"([^"]+)"'),
+    ],
+    'php': [
+        re.compile(r'^\s*use\s+([\w\\]+)'),
+    ],
+}
+
+FUNC_CLASS_PATTERNS = {
+    'python': {
+        'function': re.compile(r'^\s*def\s+(\w+)\s*\('),
+        'class': re.compile(r'^\s*class\s+(\w+)'),
+    },
+    'javascript': {
+        'function': re.compile(r'(?:^|\s)function\s+(\w+)|(?:const|let|var)\s+(\w+)\s*=\s*(?:async\s*)?\('),
+        'class': re.compile(r'^\s*class\s+(\w+)'),
+    },
+    'typescript': {
+        'function': re.compile(r'(?:^|\s)function\s+(\w+)|(?:const|let|var)\s+(\w+)\s*=\s*(?:async\s*)?\('),
+        'class': re.compile(r'^\s*(?:class|interface)\s+(\w+)'),
+    },
+    'go': {
+        'function': re.compile(r'^\s*func\s+(?:\([^)]*\)\s+)?(\w+)\s*\('),
+        'class': re.compile(r'^\s*type\s+(\w+)\s+(?:struct|interface)'),
+    },
+    'php': {
+        'function': re.compile(r'^\s*(?:public|private|protected|static|\s)*function\s+(\w+)'),
+        'class': re.compile(r'^\s*(?:abstract\s+)?class\s+(\w+)'),
+    },
+}
+
+
+def detect_language(filepath: str) -> str:
+    ext = Path(filepath).suffix
+    return LANG_MAP.get(ext, '')
+
+
+def extract_imports(filepath: str, language: str) -> list:
+    patterns = IMPORT_PATTERNS.get(language, [])
+    imports = []
+    try:
+        with open(filepath, 'r', errors='replace') as f:
+            for line in f:
+                for pat in patterns:
+                    m = pat.search(line)
+                    if m:
+                        # Use first non-None group
+                        val = next((g for g in m.groups() if g is not None), None)
+                        if val and val not in imports:
+                            imports.append(val)
+    except OSError:
+        pass
+    return imports
+
+
+def extract_symbols(filepath: str, language: str) -> tuple:
+    patterns = FUNC_CLASS_PATTERNS.get(language, {})
+    functions, classes = [], []
+    try:
+        with open(filepath, 'r', errors='replace') as f:
+            for line in f:
+                fp = patterns.get('function')
+                if fp:
+                    m = fp.search(line)
+                    if m:
+                        name = next((g for g in m.groups() if g is not None), None)
+                        if name and name not in functions:
+                            functions.append(name)
+                cp = patterns.get('class')
+                if cp:
+                    m = cp.search(line)
+                    if m:
+                        name = next((g for g in m.groups() if g is not None), None)
+                        if name and name not in classes:
+                            classes.append(name)
+    except OSError:
+        pass
+    return functions, classes
+
+
+def find_source_files(root: str) -> list:
+    result = []
+    for dirpath, dirnames, filenames in os.walk(root):
+        dirnames[:] = [d for d in dirnames if d not in IGNORE_DIRS]
+        for fn in filenames:
+            if Path(fn).suffix in SOURCE_EXTS:
+                result.append(os.path.join(dirpath, fn))
+    return result
+
+
+def build_reverse_map(project_root: str, target_stem: str) -> list:
+    imported_by = []
+    for src in find_source_files(project_root):
+        lang = detect_language(src)
+        if not lang:
+            continue
+        imps = extract_imports(src, lang)
+        for imp in imps:
+            if target_stem in imp or Path(target_stem).stem in imp:
+                rel = os.path.relpath(src, project_root)
+                if rel not in imported_by:
+                    imported_by.append(rel)
+                break
+    return imported_by
+
+
+def generate_mermaid(target_name: str, imports: list, imported_by: list) -> str:
+    lines = ['graph TD']
+    safe = re.sub(r'[^a-zA-Z0-9_]', '_', target_name)
+    lines.append(f'    {safe}["{target_name}"]')
+    for imp in imports:
+        imp_safe = re.sub(r'[^a-zA-Z0-9_]', '_', imp)
+        lines.append(f'    {safe} -->|imports| {imp_safe}["{imp}"]')
+    for dep in imported_by:
+        dep_safe = re.sub(r'[^a-zA-Z0-9_]', '_', dep)
+        lines.append(f'    {dep_safe}["{dep}"] -->|imports| {safe}')
+    return '\n'.join(lines)
+
+
+def main():
+    if len(sys.argv) < 2:
+        print(json.dumps({"error": "Usage: dependency-graph.py <file_or_directory>"}))
+        sys.exit(1)
+
+    target = os.path.abspath(sys.argv[1])
+    if not os.path.exists(target):
+        print(json.dumps({"error": f"Path not found: {target}"}))
+        sys.exit(1)
+
+    # Determine project root (walk up to find .git or use parent)
+    project_root = target if os.path.isdir(target) else os.path.dirname(target)
+    check = project_root
+    for _ in range(10):
+        if os.path.isdir(os.path.join(check, '.git')):
+            project_root = check
+            break
+        parent = os.path.dirname(check)
+        if parent == check:
+            break
+        check = parent
+
+    files = [target] if os.path.isfile(target) else find_source_files(target)
+
+    all_imports, all_functions, all_classes = [], [], []
+    language = ''
+
+    for fp in files:
+        lang = detect_language(fp)
+        if not lang:
+            continue
+        if not language:
+            language = lang
+        imps = extract_imports(fp, lang)
+        all_imports.extend(i for i in imps if i not in all_imports)
+        funcs, clss = extract_symbols(fp, lang)
+        all_functions.extend(f for f in funcs if f not in all_functions)
+        all_classes.extend(c for c in clss if c not in all_classes)
+
+    target_name = os.path.relpath(target, project_root)
+    imported_by = build_reverse_map(project_root, target_name)
+
+    mermaid = generate_mermaid(target_name, all_imports, imported_by)
+
+    result = {
+        "target": target_name,
+        "language": language,
+        "imports": all_imports,
+        "imported_by": imported_by,
+        "functions": all_functions,
+        "classes": all_classes,
+        "mermaid": mermaid,
+    }
+
+    print(json.dumps(result, indent=2))
+
+
+if __name__ == '__main__':
+    main()

package/app/skills/explore/SKILL.md

@@ -0,0 +1,112 @@
+---
+name: explore
+description: "Explore codebase structure, stack, and architecture"
+user-invocable: true
+effort: medium
+argument-hint: "[path or question]"
+allowed-tools: Read, Grep, Glob
+---
+
+# Codebase Exploration
+
+$ARGUMENTS
+
+Explore and understand a codebase structure.
+
+## Project context
+
+- Config files: !`find . -maxdepth 2 -type f -name "*.json" -o -name "*.toml" -o -name "*.yaml" -o -name "*.yml" 2>/dev/null | head -20`
+
+## Usage
+
+```
+/explore [path]
+```
+
+## What This Command Does
+
+1. **Maps** project structure
+2. **Identifies** technology stack
+3. **Finds** key files and patterns
+4. **Reports** architecture overview
+
+## Output Format
+
+```markdown
+## Codebase Analysis Report
+
+### Project Type
+- **Language**: [TypeScript/Python/PHP/etc.]
+- **Framework**: [Next.js/FastAPI/Laravel/etc.]
+- **Package Manager**: [npm/pnpm/pip/composer]
+
+### Directory Structure
+```
+project/
+├── src/           # Source code
+├── tests/         # Test files
+├── config/        # Configuration
+└── ...
+```
+
+### Key Files
+| File | Purpose |
+|------|---------|
+| `src/index.ts` | Entry point |
+| `src/api/` | API routes |
+
+### Dependencies
+- **Runtime**: [list]
+- **Dev**: [list]
+
+### Patterns Detected
+- [Pattern 1]
+- [Pattern 2]
+
+### Entry Points
+- [Entry 1]
+- [Entry 2]
+```
+
+## Technology Detection
+
+| Marker | Technology |
+|--------|------------|
+| `package.json` | Node.js |
+| `tsconfig.json` | TypeScript |
+| `next.config.*` | Next.js |
+| `nuxt.config.*` | Nuxt |
+| `pyproject.toml` | Python |
+| `composer.json` | PHP |
+| `pubspec.yaml` | Flutter/Dart |
+| `Cargo.toml` | Rust |
+| `go.mod` | Go |
+
+## When to Use
+
+- Starting work on unfamiliar codebase
+- Before planning major changes
+- Understanding dependencies
+- Finding specific code patterns
+
+## READ-ONLY
+
+This skill ONLY reads and analyzes.
+It does NOT write or modify any files.
+
+## Visual Output
+
+For an interactive HTML tree visualization of the codebase:
+
+```bash
+python ${CLAUDE_SKILL_DIR}/scripts/visualize.py .
+```
+
+This generates `codebase-map.html` with collapsible directories, file sizes, and type-colored indicators.
+
+## KB Integration
+
+```python
+smart_query("codebase analysis: {technology}")
+hybrid_search_kb("project structure {framework}")
+```

package/app/skills/explore/scripts/visualize.py

@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+"""Generate an interactive collapsible tree visualization of a codebase.
+
+Usage: python visualize.py [directory]
+Output: codebase-map.html (opened in browser)
+"""
+
+import json
+import sys
+import webbrowser
+from pathlib import Path
+from collections import Counter
+
+IGNORE = {'.git', 'node_modules', '__pycache__', '.venv', 'venv', 'dist', 'build', '.next', '.nuxt', '.cache', 'vendor', '.dart_tool'}
+
+def scan(path: Path, stats: dict) -> dict:
+    result = {"name": path.name, "children": [], "size": 0}
+    try:
+        for item in sorted(path.iterdir()):
+            if item.name in IGNORE or item.name.startswith('.'):
+                continue
+            if item.is_file():
+                size = item.stat().st_size
+                ext = item.suffix.lower() or '(no ext)'
+                result["children"].append({"name": item.name, "size": size, "ext": ext})
+                result["size"] += size
+                stats["files"] += 1
+                stats["extensions"][ext] += 1
+                stats["ext_sizes"][ext] += size
+            elif item.is_dir():
+                stats["dirs"] += 1
+                child = scan(item, stats)
+                if child["children"]:
+                    result["children"].append(child)
+                    result["size"] += child["size"]
+    except PermissionError:
+        pass
+    return result
+
+def fmt(b):
+    if b < 1024: return f"{b} B"
+    if b < 1048576: return f"{b/1024:.1f} KB"
+    return f"{b/1048576:.1f} MB"
+
+def generate_html(data: dict, stats: dict, output: Path) -> None:
+    ext_sizes = stats["ext_sizes"]
+    total_size = sum(ext_sizes.values()) or 1
+    sorted_exts = sorted(ext_sizes.items(), key=lambda x: -x[1])[:8]
+    colors = {
+        '.js': '#f7df1e', '.ts': '#3178c6', '.py': '#3776ab', '.go': '#00add8',
+        '.rs': '#dea584', '.rb': '#cc342d', '.css': '#264de4', '.html': '#e34c26',
+        '.json': '#6b7280', '.md': '#083fa1', '.yaml': '#cb171e', '.yml': '#cb171e',
+        '.tsx': '#3178c6', '.jsx': '#61dafb', '.sh': '#4eaa25', '.dart': '#0175c2',
+        '.php': '#777bb4', '.vue': '#42b883', '.svelte': '#ff3e00',
+    }
+    lang_bars = "".join(
+        f'<div class="bar-row"><span class="bar-label">{ext}</span>'
+        f'<div class="bar" style="width:{(size/total_size)*100}%;background:{colors.get(ext,"#6b7280")}"></div>'
+        f'<span class="bar-pct">{(size/total_size)*100:.1f}%</span></div>'
+        for ext, size in sorted_exts
+    )
+    html = f'''<!DOCTYPE html>
+<html><head><meta charset="utf-8"><title>Codebase Explorer</title>
+<style>
+body {{ font: 14px/1.5 system-ui, sans-serif; margin: 0; background: #1a1a2e; color: #eee; }}
+.container {{ display: flex; height: 100vh; }}
+.sidebar {{ width: 280px; background: #252542; padding: 20px; border-right: 1px solid #3d3d5c; overflow-y: auto; flex-shrink: 0; }}
+.main {{ flex: 1; padding: 20px; overflow-y: auto; }}
+h1 {{ margin: 0 0 10px; font-size: 18px; }} h2 {{ margin: 20px 0 10px; font-size: 14px; color: #888; text-transform: uppercase; }}
+.stat {{ display: flex; justify-content: space-between; padding: 8px 0; border-bottom: 1px solid #3d3d5c; }}
+.stat-value {{ font-weight: bold; }}
+.bar-row {{ display: flex; align-items: center; margin: 6px 0; }}
+.bar-label {{ width: 55px; font-size: 12px; color: #aaa; }}
+.bar {{ height: 18px; border-radius: 3px; }} .bar-pct {{ margin-left: 8px; font-size: 12px; color: #666; }}
+.tree {{ list-style: none; padding-left: 20px; }}
+details {{ cursor: pointer; }} summary {{ padding: 4px 8px; border-radius: 4px; }}
+summary:hover {{ background: #2d2d44; }} .folder {{ color: #ffd700; }}
+.file {{ display: flex; align-items: center; padding: 4px 8px; border-radius: 4px; }}
+.file:hover {{ background: #2d2d44; }} .size {{ color: #888; margin-left: auto; font-size: 12px; }}
+.dot {{ width: 8px; height: 8px; border-radius: 50%; margin-right: 8px; display: inline-block; }}
+</style></head><body>
+<div class="container">
+<div class="sidebar">
+<h1>Summary</h1>
+<div class="stat"><span>Files</span><span class="stat-value">{stats["files"]:,}</span></div>
+<div class="stat"><span>Directories</span><span class="stat-value">{stats["dirs"]:,}</span></div>
+<div class="stat"><span>Total size</span><span class="stat-value">{fmt(data["size"])}</span></div>
+<div class="stat"><span>File types</span><span class="stat-value">{len(stats["extensions"])}</span></div>
+<h2>By file type</h2>{lang_bars}
+</div>
+<div class="main"><h1>{data["name"]}</h1><ul class="tree" id="root"></ul></div>
+</div>
+<script>
+const data={json.dumps(data)};const colors={json.dumps(colors)};
+function fmt(b){{if(b<1024)return b+' B';if(b<1048576)return(b/1024).toFixed(1)+' KB';return(b/1048576).toFixed(1)+' MB';}}
+function render(n,p){{if(n.children){{const d=document.createElement('details');d.open=p===document.getElementById('root');
+d.innerHTML=`<summary><span class="folder">${{n.name}}</span><span class="size">${{fmt(n.size)}}</span></summary>`;
+const u=document.createElement('ul');u.className='tree';n.children.sort((a,b)=>(b.children?1:0)-(a.children?1:0)||a.name.localeCompare(b.name));
+n.children.forEach(c=>render(c,u));d.appendChild(u);const l=document.createElement('li');l.appendChild(d);p.appendChild(l);
+}}else{{const l=document.createElement('li');l.className='file';
+l.innerHTML=`<span class="dot" style="background:${{colors[n.ext]||'#6b7280'}}"></span>${{n.name}}<span class="size">${{fmt(n.size)}}</span>`;
+p.appendChild(l);}}}}
+data.children.forEach(c=>render(c,document.getElementById('root')));
+</script></body></html>'''
+    output.write_text(html)
+
+if __name__ == '__main__':
+    target = Path(sys.argv[1] if len(sys.argv) > 1 else '.').resolve()
+    stats = {"files": 0, "dirs": 0, "extensions": Counter(), "ext_sizes": Counter()}
+    data = scan(target, stats)
+    out = Path('codebase-map.html')
+    generate_html(data, stats, out)
+    print(f'Generated {out.absolute()}')
+    try:
+        webbrowser.open(f'file://{out.absolute()}')
+    except Exception:
+        pass