claude-code-swarm 0.3.3 → 0.3.5

package/references/skill-tree/docs/scraper/POC_PLAN.md

@@ -0,0 +1,420 @@
+# Proof of Concept Plan
+ 
+## Goal
+Build a minimal working prototype to validate the core concept: **AI agent-driven hierarchical skill indexing**
+ 
+## Scope (1-2 weeks)
+ 
+### What We'll Build
+1. Simple GitHub scraper that finds skills in a few known repos
+2. AI agent that categorizes skills into a tree structure
+3. Basic storage (JSON files to start, no DB yet)
+4. Simple CLI to browse the tree and search
+5. Proof that agent categorization works better than manual
+ 
+### What We'll Skip (for now)
+- Full database setup
+- Vector embeddings / semantic search
+- Web UI
+- Installation mechanism
+- Continuous scraping
+- Community features
+- Versioning
+ 
+## Technical Choices for POC
+ 
+### Stack
+- **Language**: Python 3.11
+- **GitHub API**: PyGithub
+- **AI**: Anthropic Claude API (Sonnet for quality)
+- **Storage**: JSON files (`skills.json`, `taxonomy.json`)
+- **CLI**: Typer (beautiful, simple)
+ 
+### Project Structure
+```
+skillindex-poc/
+├── src/
+│   ├── scraper.py          # GitHub scraping logic
+│   ├── agent.py            # AI categorization agent
+│   ├── storage.py          # JSON file operations
+│   ├── taxonomy.py         # Tree structure management
+│   └── cli.py              # CLI commands
+├── data/
+│   ├── skills.json         # Raw skill data
+│   └── taxonomy.json       # Tree structure
+├── tests/
+│   └── test_agent.py       # Basic tests
+├── requirements.txt
+├── README.md
+└── .env.example
+```
+ 
+## Implementation Steps
+ 
+### Step 1: Scraper (Day 1-2)
+```python
+# scraper.py - Pseudocode
+ 
+def scan_github_repo(repo_url):
+    """Scan a single GitHub repo for SKILL.md files"""
+    # Use PyGithub to:
+    # 1. Search for files named "SKILL.md"
+    # 2. Read each file
+    # 3. Parse YAML frontmatter
+    # 4. Extract metadata
+    # 5. Return list of skills
+    pass
+ 
+def scrape_known_repos():
+    """Scrape a hardcoded list of repos"""
+    repos = [
+        "anthropics/skills",
+        "numman-ali/openskills",
+        # Add more...
+    ]
+    all_skills = []
+    for repo in repos:
+        skills = scan_github_repo(repo)
+        all_skills.extend(skills)
+    return all_skills
+```
+ 
+**Output**: `data/skills.json`
+```json
+[
+  {
+    "id": "anthropics-skills-pdf",
+    "name": "pdf",
+    "description": "Extract text, fill forms, merge PDFs...",
+    "repo_url": "https://github.com/anthropics/skills",
+    "file_path": "skills/pdf/SKILL.md",
+    "content": "...",
+    "metadata": {
+      "allowed_tools": ["Read", "Bash"],
+      "model": null
+    }
+  }
+]
+```
+ 
+### Step 2: AI Agent Categorization (Day 3-5)
+```python
+# agent.py - Pseudocode
+ 
+CATEGORIZATION_PROMPT = """
+You are a skill taxonomy organizer. Your job is to place this skill
+in a hierarchical taxonomy tree.
+ 
+Skill:
+Name: {name}
+Description: {description}
+Content: {content}
+ 
+Current Taxonomy:
+{taxonomy_json}
+ 
+Tasks:
+1. Analyze the skill's purpose, domain, and use case
+2. Determine the best category path(s) in the taxonomy
+3. If no good category exists, suggest creating new ones
+4. Return JSON with:
+   - primary_path: ["Level1", "Level2", "Level3"]
+   - secondary_paths: [["Alt1", "Alt2"], ...]
+   - reasoning: "Why you chose this categorization"
+   - suggested_tags: ["tag1", "tag2"]
+   - relationships: [{"skill_id": "other-skill", "type": "depends_on"}]
+"""
+ 
+def categorize_skill(skill, current_taxonomy):
+    """Use Claude to categorize a skill"""
+    response = anthropic.messages.create(
+        model="claude-sonnet-4-5-20251101",
+        max_tokens=2000,
+        messages=[{
+            "role": "user",
+            "content": CATEGORIZATION_PROMPT.format(
+                name=skill["name"],
+                description=skill["description"],
+                content=skill["content"][:2000],  # truncate
+                taxonomy_json=json.dumps(current_taxonomy, indent=2)
+            )
+        }]
+    )
+    return parse_agent_response(response)
+ 
+def build_taxonomy(skills):
+    """Build taxonomy by processing skills one at a time"""
+    taxonomy = {"Root": {}}
+    placements = []
+ 
+    for skill in skills:
+        result = categorize_skill(skill, taxonomy)
+        # Update taxonomy with new categories if suggested
+        update_taxonomy(taxonomy, result["primary_path"])
+        for path in result["secondary_paths"]:
+            update_taxonomy(taxonomy, path)
+ 
+        placements.append({
+            "skill_id": skill["id"],
+            "primary_path": result["primary_path"],
+            "secondary_paths": result["secondary_paths"],
+            "reasoning": result["reasoning"],
+            "tags": result["suggested_tags"],
+            "relationships": result["relationships"]
+        })
+ 
+    return taxonomy, placements
+```
+ 
+**Output**: `data/taxonomy.json`
+```json
+{
+  "tree": {
+    "Development": {
+      "Languages": {
+        "Python": {
+          "Testing": {},
+          "Web-Frameworks": {}
+        },
+        "JavaScript": {
+          "React": {}
+        }
+      },
+      "Infrastructure": {
+        "Docker": {}
+      }
+    },
+    "Creative": {
+      "Design": {}
+    }
+  },
+  "placements": [
+    {
+      "skill_id": "anthropics-skills-pdf",
+      "primary_path": ["Creative", "Design"],
+      "secondary_paths": [["Development", "Document-Processing"]],
+      "reasoning": "PDF skill is primarily for document creation/editing...",
+      "tags": ["pdf", "documents", "forms"],
+      "relationships": []
+    }
+  ]
+}
+```
+ 
+### Step 3: CLI (Day 6-7)
+```python
+# cli.py - Pseudocode
+ 
+import typer
+from rich import print
+from rich.tree import Tree
+ 
+app = typer.Typer()
+ 
+@app.command()
+def scrape():
+    """Scrape GitHub repos and save skills"""
+    print("🔍 Scraping GitHub repositories...")
+    skills = scrape_known_repos()
+    save_skills(skills)
+    print(f"✅ Found {len(skills)} skills")
+ 
+@app.command()
+def index():
+    """Run AI agent to categorize skills"""
+    print("🤖 Running AI categorization agent...")
+    skills = load_skills()
+    taxonomy, placements = build_taxonomy(skills)
+    save_taxonomy(taxonomy, placements)
+    print("✅ Taxonomy built successfully")
+ 
+@app.command()
+def tree(path: str = "Root"):
+    """Display taxonomy tree"""
+    taxonomy = load_taxonomy()
+    display_tree(taxonomy, path)
+ 
+@app.command()
+def search(query: str):
+    """Search for skills (simple text match for POC)"""
+    skills = load_skills()
+    taxonomy = load_taxonomy()
+ 
+    results = []
+    for skill in skills:
+        if query.lower() in skill["name"].lower() or \
+           query.lower() in skill["description"].lower():
+            placement = find_placement(skill["id"], taxonomy)
+            results.append({
+                "skill": skill,
+                "path": placement["primary_path"]
+            })
+ 
+    for result in results:
+        print(f"📦 {result['skill']['name']}")
+        print(f"   Path: {' > '.join(result['path'])}")
+        print(f"   {result['skill']['description']}")
+        print()
+ 
+@app.command()
+def show(skill_name: str):
+    """Show detailed info about a skill"""
+    skills = load_skills()
+    taxonomy = load_taxonomy()
+ 
+    skill = find_skill_by_name(skill_name, skills)
+    placement = find_placement(skill["id"], taxonomy)
+ 
+    print(f"📦 {skill['name']}")
+    print(f"Primary Path: {' > '.join(placement['primary_path'])}")
+    if placement['secondary_paths']:
+        print(f"Also in: {placement['secondary_paths']}")
+    print(f"\nDescription:\n{skill['description']}")
+    print(f"\nReasoning:\n{placement['reasoning']}")
+    print(f"\nTags: {', '.join(placement['tags'])}")
+    print(f"\nSource: {skill['repo_url']}")
+ 
+if __name__ == "__main__":
+    app()
+```
+ 
+### Step 4: Testing & Validation (Day 8-10)
+ 
+**Manual Evaluation**:
+1. Run scraper on 3-5 known repos
+2. Categorize 20-50 skills with agent
+3. Manually review categorization quality
+4. Compare with flat list approaches
+5. Measure:
+   - Categorization accuracy (human eval)
+   - Taxonomy coherence (does it make sense?)
+   - Discoverability (can you find skills easily?)
+ 
+**Test Cases**:
+```python
+# tests/test_agent.py
+ 
+def test_agent_categorizes_pdf_skill():
+    """Agent should categorize PDF skill correctly"""
+    skill = load_test_skill("pdf")
+    taxonomy = load_empty_taxonomy()
+ 
+    result = categorize_skill(skill, taxonomy)
+ 
+    # Verify it's in a sensible category
+    assert "Creative" in result["primary_path"] or \
+           "Documents" in result["primary_path"]
+    assert result["reasoning"]  # Has reasoning
+    assert len(result["suggested_tags"]) > 0
+ 
+def test_agent_suggests_new_categories():
+    """Agent should create categories as needed"""
+    # Give it a skill in a novel domain
+    skill = load_test_skill("quantum-computing-helper")
+    taxonomy = load_basic_taxonomy()
+ 
+    result = categorize_skill(skill, taxonomy)
+ 
+    # Should suggest creating "Quantum Computing" category
+    assert "Quantum" in str(result["primary_path"])
+```
+ 
+## Success Criteria
+ 
+The POC is successful if:
+ 
+1. ✅ **Scraper works**: Can extract 20+ skills from GitHub
+2. ✅ **Agent categorizes coherently**: 80%+ of categorizations make sense to humans
+3. ✅ **Taxonomy is useful**: Browsing tree is more intuitive than flat list
+4. ✅ **Agent adapts**: Creates new categories when needed
+5. ✅ **CLI is usable**: Can scrape, index, browse, search in <5 commands
+6. ✅ **Performance is acceptable**: Categorizing 50 skills takes <5 minutes
+ 
+## Demo Script
+ 
+```bash
+# 1. Setup
+pip install -r requirements.txt
+cp .env.example .env
+# Add ANTHROPIC_API_KEY to .env
+ 
+# 2. Scrape GitHub
+python -m skillindex scrape
+# Output: ✅ Found 47 skills
+ 
+# 3. Run agent indexing
+python -m skillindex index
+# Output: 🤖 Categorizing skill 1/47: pdf
+#         🤖 Categorizing skill 2/47: docx
+#         ...
+#         ✅ Taxonomy built successfully
+ 
+# 4. Browse the tree
+python -m skillindex tree
+# Output: Interactive tree visualization
+ 
+# 5. Search
+python -m skillindex search "PDF"
+# Output: 📦 pdf
+#         Path: Creative > Design > Documents
+#         Extract text, fill forms, merge PDFs...
+ 
+# 6. Detailed view
+python -m skillindex show pdf
+# Output: Full skill details with categorization reasoning
+```
+ 
+## Evaluation Questions
+ 
+After building POC, answer:
+ 
+1. **Quality**: How accurate are the agent's categorizations?
+2. **Consistency**: Does agent categorize similar skills similarly?
+3. **Adaptability**: Does it create sensible new categories?
+4. **Discoverability**: Is browsing tree better than flat list?
+5. **Cost**: How much does it cost to categorize 50 skills?
+6. **Speed**: Is it fast enough for 1000+ skills?
+7. **Value**: Would users actually want this vs existing solutions?
+ 
+## Next Steps After POC
+ 
+If POC succeeds:
+1. **Add database**: Migrate from JSON to PostgreSQL
+2. **Add embeddings**: Semantic search with vector store
+3. **Continuous scraping**: Background jobs for updates
+4. **Installation**: Actually install skills to ~/.claude/skills
+5. **Web UI**: Build visual browser
+6. **Scale up**: Test with 500+ skills
+7. **Community**: Open source, gather feedback
+ 
+If POC fails or is mediocre:
+1. **Analyze failures**: Where did agent categorize poorly?
+2. **Improve prompts**: Refine categorization instructions
+3. **Try different approach**: Maybe hierarchical isn't the answer?
+4. **Pivot**: Focus on semantic search instead of tree?
+5. **Kill it**: If not working, cut losses and learn
+ 
+## Resources Needed
+ 
+- **API Costs**: ~$2-5 for 50 skills (Sonnet, ~1000 tokens/skill)
+- **Time**: 1-2 weeks part-time
+- **GitHub API**: Free tier (5000 req/hour) is plenty
+- **Compute**: Runs on laptop, no special infrastructure
+ 
+## Risk Mitigation
+ 
+**Risk**: Agent categorizes inconsistently
+- **Mitigation**: Use structured output, few-shot examples in prompt
+ 
+**Risk**: Too expensive at scale
+- **Mitigation**: Cache results, only re-categorize on changes, use cheaper model
+ 
+**Risk**: Taxonomy becomes unwieldy
+- **Mitigation**: Limit depth to 4 levels, agent suggests merging similar categories
+ 
+**Risk**: Skills don't fit hierarchy
+- **Mitigation**: Allow multi-path placements, use tags for cross-cutting
+ 
+---
+ 
+Ready to build? Let's start with setting up the project structure! 🚀

package/references/skill-tree/docs/scraper/README.md

@@ -0,0 +1,170 @@
+# GitHub Skills Indexer
+ 
+> An AI-powered system for continuously indexing, organizing, and distributing Agent Skills from GitHub
+ 
+## 📚 Documentation
+ 
+- **[BRAINSTORM.md](BRAINSTORM.md)** - Comprehensive architecture design, research findings, and technical specifications
+- **[POC_PLAN.md](POC_PLAN.md)** - Proof of concept implementation plan (1-2 weeks)
+ 
+## 🎯 Project Vision
+ 
+Build a system that:
+1. **Scrapes** GitHub continuously for Agent Skills (Anthropic format)
+2. **Indexes** skills using AI agents for intelligent hierarchical organization
+3. **Organizes** skills in a tree structure (not flat lists) for better discovery
+4. **Distributes** skills with easy installation and loading
+ 
+## ✨ Key Innovations
+ 
+### 🤖 AI-Managed Taxonomy
+Unlike existing solutions (OpenSkills, SkillsMP) that use flat lists or static categories, we use Claude to:
+- Semantically understand skill purposes
+- Dynamically organize skills into an evolving tree structure
+- Identify relationships (dependencies, alternatives, extensions)
+- Adapt to new skill domains automatically
+ 
+### 🌳 Hierarchical Organization
+```
+Development/
+├── Languages/
+│   ├── Python/
+│   │   ├── Testing/
+│   │   │   └── pytest-helper
+│   │   └── Web-Frameworks/
+│   │       └── fastapi-crud-generator
+│   └── JavaScript/
+│       └── React/
+│           └── component-builder
+└── Infrastructure/
+    └── Docker/
+        └── compose-manager
+```
+ 
+### 🔗 Multi-Dimensional Placement
+Skills can exist in multiple taxonomy paths:
+- `react-testing-helper` → Development/Frontend/React/Testing
+- `react-testing-helper` → Development/Testing/Unit-Testing
+- `react-testing-helper` → Languages/JavaScript/Testing-Tools
+ 
+## 🏗️ Architecture Overview
+ 
+```
+GitHub Repos → Scraper → AI Agent Indexer → Database → API/CLI/Web UI
+                           ↓
+                    Taxonomy Manager
+                    (rebalancing)
+```
+ 
+**Components:**
+1. **GitHub Scraper** - Discovers SKILL.md files across repos
+2. **AI Agent Indexer** - Categorizes skills using Claude
+3. **Data Layer** - PostgreSQL + pgvector for storage & search
+4. **API Service** - FastAPI with semantic search, taxonomy navigation
+5. **CLI Tool** - Browse tree, search, install skills
+6. **Web UI** - Visual browser with tree navigation (optional)
+ 
+## 📊 Comparison with Existing Solutions
+ 
+| Feature | OpenSkills | rscheiwe/open-skills | **Our Solution** |
+|---------|-----------|---------------------|------------------|
+| Organization | Flat directories | Flat with tags | **🌳 AI-managed tree** |
+| Discovery | Manual/keyword | Vector search | **Tree + semantic + vector** |
+| Installation | Git clone | API/library | **Smart dependency resolution** |
+| Versioning | ❌ | ✅ | ✅ |
+| Continuous Updates | ❌ | ❌ | **✅ Background scraping** |
+| Relationships | ❌ | ❌ | **✅ Dependency graph** |
+| Context-Aware | ❌ | ❌ | **✅ Project-based suggestions** |
+ 
+## 🚀 Getting Started (POC)
+ 
+See [POC_PLAN.md](POC_PLAN.md) for implementation details.
+ 
+Quick start:
+```bash
+# 1. Setup
+git clone <this-repo>
+cd skillindex-poc
+pip install -r requirements.txt
+cp .env.example .env
+# Add ANTHROPIC_API_KEY
+ 
+# 2. Scrape & Index
+python -m skillindex scrape  # Find skills on GitHub
+python -m skillindex index   # AI categorization
+ 
+# 3. Explore
+python -m skillindex tree                     # Browse hierarchy
+python -m skillindex search "testing tools"   # Semantic search
+python -m skillindex show pytest-helper       # Skill details
+```
+ 
+## 📖 Research Sources
+ 
+### Official Resources
+- [Anthropic Skills Repository](https://github.com/anthropics/skills)
+- [Agent Skills Specification](https://agentskills.io/home)
+- [Claude Code Skills Docs](https://code.claude.com/docs/en/skills)
+ 
+### Existing Implementations
+- [OpenSkills](https://github.com/numman-ali/openskills) - Universal CLI loader
+- [rscheiwe/open-skills](https://github.com/rscheiwe/open-skills) - Modular framework
+- [Awesome Claude Skills](https://github.com/travisvn/awesome-claude-skills) - Curated list
+ 
+### Articles & Analysis
+- [Anthropic Engineering Blog](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills)
+- [Simon Willison on Skills](https://simonwillison.net/2025/Oct/16/claude-skills/)
+- [Claude Skills Deep Dive](https://leehanchung.github.io/blogs/2025/10/26/claude-skills-deep-dive/)
+ 
+## 🗺️ Roadmap
+ 
+### Phase 1: POC (Weeks 1-2) ✅ Planning
+- [x] Research existing implementations
+- [x] Design architecture
+- [ ] Build basic scraper
+- [ ] Implement AI categorization
+- [ ] Create simple CLI
+- [ ] Validate with 20-50 skills
+ 
+### Phase 2: MVP (Weeks 3-6)
+- [ ] PostgreSQL + pgvector setup
+- [ ] Vector embeddings for semantic search
+- [ ] Background task queue
+- [ ] REST API
+- [ ] Enhanced CLI with install
+- [ ] Basic quality scoring
+ 
+### Phase 3: Advanced Features (Weeks 7-12)
+- [ ] Web UI with tree visualization
+- [ ] Dependency resolution
+- [ ] Context-aware recommendations
+- [ ] Community features (ratings)
+- [ ] Analytics dashboard
+ 
+### Phase 4: Scale & Launch (Weeks 13-16)
+- [ ] Performance optimization
+- [ ] Comprehensive testing
+- [ ] Documentation
+- [ ] Public launch
+- [ ] Community building
+ 
+## 🎓 Learning Outcomes
+ 
+This project explores:
+- **AI-assisted data organization** - Can agents maintain coherent taxonomies?
+- **Semantic understanding at scale** - Categorizing 1000+ diverse skills
+- **Progressive disclosure** - Loading skills on-demand
+- **Developer tools** - Building CLI/API for developer experience
+- **Open source ecosystems** - Indexing decentralized repositories
+ 
+## 🤝 Contributing
+ 
+(Coming soon after POC validation)
+ 
+## 📄 License
+ 
+MIT (tentative - to be decided)
+ 
+---
+ 
+**Status**: 📐 Design Phase - [See BRAINSTORM.md for full details](BRAINSTORM.md)