macro-agent 0.1.2 → 0.1.4

package/references/skill-tree/docs/scraper/POC_PLAN.md

@@ -1,420 +0,0 @@
-# Proof of Concept Plan
- 
-## Goal
-Build a minimal working prototype to validate the core concept: **AI agent-driven hierarchical skill indexing**
- 
-## Scope (1-2 weeks)
- 
-### What We'll Build
-1. Simple GitHub scraper that finds skills in a few known repos
-2. AI agent that categorizes skills into a tree structure
-3. Basic storage (JSON files to start, no DB yet)
-4. Simple CLI to browse the tree and search
-5. Proof that agent categorization works better than manual
- 
-### What We'll Skip (for now)
-- Full database setup
-- Vector embeddings / semantic search
-- Web UI
-- Installation mechanism
-- Continuous scraping
-- Community features
-- Versioning
- 
-## Technical Choices for POC
- 
-### Stack
-- **Language**: Python 3.11
-- **GitHub API**: PyGithub
-- **AI**: Anthropic Claude API (Sonnet for quality)
-- **Storage**: JSON files (`skills.json`, `taxonomy.json`)
-- **CLI**: Typer (beautiful, simple)
- 
-### Project Structure
-```
-skillindex-poc/
-├── src/
-│   ├── scraper.py          # GitHub scraping logic
-│   ├── agent.py            # AI categorization agent
-│   ├── storage.py          # JSON file operations
-│   ├── taxonomy.py         # Tree structure management
-│   └── cli.py              # CLI commands
-├── data/
-│   ├── skills.json         # Raw skill data
-│   └── taxonomy.json       # Tree structure
-├── tests/
-│   └── test_agent.py       # Basic tests
-├── requirements.txt
-├── README.md
-└── .env.example
-```
- 
-## Implementation Steps
- 
-### Step 1: Scraper (Day 1-2)
-```python
-# scraper.py - Pseudocode
- 
-def scan_github_repo(repo_url):
-    """Scan a single GitHub repo for SKILL.md files"""
-    # Use PyGithub to:
-    # 1. Search for files named "SKILL.md"
-    # 2. Read each file
-    # 3. Parse YAML frontmatter
-    # 4. Extract metadata
-    # 5. Return list of skills
-    pass
- 
-def scrape_known_repos():
-    """Scrape a hardcoded list of repos"""
-    repos = [
-        "anthropics/skills",
-        "numman-ali/openskills",
-        # Add more...
-    ]
-    all_skills = []
-    for repo in repos:
-        skills = scan_github_repo(repo)
-        all_skills.extend(skills)
-    return all_skills
-```
- 
-**Output**: `data/skills.json`
-```json
-[
-  {
-    "id": "anthropics-skills-pdf",
-    "name": "pdf",
-    "description": "Extract text, fill forms, merge PDFs...",
-    "repo_url": "https://github.com/anthropics/skills",
-    "file_path": "skills/pdf/SKILL.md",
-    "content": "...",
-    "metadata": {
-      "allowed_tools": ["Read", "Bash"],
-      "model": null
-    }
-  }
-]
-```
- 
-### Step 2: AI Agent Categorization (Day 3-5)
-```python
-# agent.py - Pseudocode
- 
-CATEGORIZATION_PROMPT = """
-You are a skill taxonomy organizer. Your job is to place this skill
-in a hierarchical taxonomy tree.
- 
-Skill:
-Name: {name}
-Description: {description}
-Content: {content}
- 
-Current Taxonomy:
-{taxonomy_json}
- 
-Tasks:
-1. Analyze the skill's purpose, domain, and use case
-2. Determine the best category path(s) in the taxonomy
-3. If no good category exists, suggest creating new ones
-4. Return JSON with:
-   - primary_path: ["Level1", "Level2", "Level3"]
-   - secondary_paths: [["Alt1", "Alt2"], ...]
-   - reasoning: "Why you chose this categorization"
-   - suggested_tags: ["tag1", "tag2"]
-   - relationships: [{"skill_id": "other-skill", "type": "depends_on"}]
-"""
- 
-def categorize_skill(skill, current_taxonomy):
-    """Use Claude to categorize a skill"""
-    response = anthropic.messages.create(
-        model="claude-sonnet-4-5-20251101",
-        max_tokens=2000,
-        messages=[{
-            "role": "user",
-            "content": CATEGORIZATION_PROMPT.format(
-                name=skill["name"],
-                description=skill["description"],
-                content=skill["content"][:2000],  # truncate
-                taxonomy_json=json.dumps(current_taxonomy, indent=2)
-            )
-        }]
-    )
-    return parse_agent_response(response)
- 
-def build_taxonomy(skills):
-    """Build taxonomy by processing skills one at a time"""
-    taxonomy = {"Root": {}}
-    placements = []
- 
-    for skill in skills:
-        result = categorize_skill(skill, taxonomy)
-        # Update taxonomy with new categories if suggested
-        update_taxonomy(taxonomy, result["primary_path"])
-        for path in result["secondary_paths"]:
-            update_taxonomy(taxonomy, path)
- 
-        placements.append({
-            "skill_id": skill["id"],
-            "primary_path": result["primary_path"],
-            "secondary_paths": result["secondary_paths"],
-            "reasoning": result["reasoning"],
-            "tags": result["suggested_tags"],
-            "relationships": result["relationships"]
-        })
- 
-    return taxonomy, placements
-```
- 
-**Output**: `data/taxonomy.json`
-```json
-{
-  "tree": {
-    "Development": {
-      "Languages": {
-        "Python": {
-          "Testing": {},
-          "Web-Frameworks": {}
-        },
-        "JavaScript": {
-          "React": {}
-        }
-      },
-      "Infrastructure": {
-        "Docker": {}
-      }
-    },
-    "Creative": {
-      "Design": {}
-    }
-  },
-  "placements": [
-    {
-      "skill_id": "anthropics-skills-pdf",
-      "primary_path": ["Creative", "Design"],
-      "secondary_paths": [["Development", "Document-Processing"]],
-      "reasoning": "PDF skill is primarily for document creation/editing...",
-      "tags": ["pdf", "documents", "forms"],
-      "relationships": []
-    }
-  ]
-}
-```
- 
-### Step 3: CLI (Day 6-7)
-```python
-# cli.py - Pseudocode
- 
-import typer
-from rich import print
-from rich.tree import Tree
- 
-app = typer.Typer()
- 
-@app.command()
-def scrape():
-    """Scrape GitHub repos and save skills"""
-    print("🔍 Scraping GitHub repositories...")
-    skills = scrape_known_repos()
-    save_skills(skills)
-    print(f"✅ Found {len(skills)} skills")
- 
-@app.command()
-def index():
-    """Run AI agent to categorize skills"""
-    print("🤖 Running AI categorization agent...")
-    skills = load_skills()
-    taxonomy, placements = build_taxonomy(skills)
-    save_taxonomy(taxonomy, placements)
-    print("✅ Taxonomy built successfully")
- 
-@app.command()
-def tree(path: str = "Root"):
-    """Display taxonomy tree"""
-    taxonomy = load_taxonomy()
-    display_tree(taxonomy, path)
- 
-@app.command()
-def search(query: str):
-    """Search for skills (simple text match for POC)"""
-    skills = load_skills()
-    taxonomy = load_taxonomy()
- 
-    results = []
-    for skill in skills:
-        if query.lower() in skill["name"].lower() or \
-           query.lower() in skill["description"].lower():
-            placement = find_placement(skill["id"], taxonomy)
-            results.append({
-                "skill": skill,
-                "path": placement["primary_path"]
-            })
- 
-    for result in results:
-        print(f"📦 {result['skill']['name']}")
-        print(f"   Path: {' > '.join(result['path'])}")
-        print(f"   {result['skill']['description']}")
-        print()
- 
-@app.command()
-def show(skill_name: str):
-    """Show detailed info about a skill"""
-    skills = load_skills()
-    taxonomy = load_taxonomy()
- 
-    skill = find_skill_by_name(skill_name, skills)
-    placement = find_placement(skill["id"], taxonomy)
- 
-    print(f"📦 {skill['name']}")
-    print(f"Primary Path: {' > '.join(placement['primary_path'])}")
-    if placement['secondary_paths']:
-        print(f"Also in: {placement['secondary_paths']}")
-    print(f"\nDescription:\n{skill['description']}")
-    print(f"\nReasoning:\n{placement['reasoning']}")
-    print(f"\nTags: {', '.join(placement['tags'])}")
-    print(f"\nSource: {skill['repo_url']}")
- 
-if __name__ == "__main__":
-    app()
-```
- 
-### Step 4: Testing & Validation (Day 8-10)
- 
-**Manual Evaluation**:
-1. Run scraper on 3-5 known repos
-2. Categorize 20-50 skills with agent
-3. Manually review categorization quality
-4. Compare with flat list approaches
-5. Measure:
-   - Categorization accuracy (human eval)
-   - Taxonomy coherence (does it make sense?)
-   - Discoverability (can you find skills easily?)
- 
-**Test Cases**:
-```python
-# tests/test_agent.py
- 
-def test_agent_categorizes_pdf_skill():
-    """Agent should categorize PDF skill correctly"""
-    skill = load_test_skill("pdf")
-    taxonomy = load_empty_taxonomy()
- 
-    result = categorize_skill(skill, taxonomy)
- 
-    # Verify it's in a sensible category
-    assert "Creative" in result["primary_path"] or \
-           "Documents" in result["primary_path"]
-    assert result["reasoning"]  # Has reasoning
-    assert len(result["suggested_tags"]) > 0
- 
-def test_agent_suggests_new_categories():
-    """Agent should create categories as needed"""
-    # Give it a skill in a novel domain
-    skill = load_test_skill("quantum-computing-helper")
-    taxonomy = load_basic_taxonomy()
- 
-    result = categorize_skill(skill, taxonomy)
- 
-    # Should suggest creating "Quantum Computing" category
-    assert "Quantum" in str(result["primary_path"])
-```
- 
-## Success Criteria
- 
-The POC is successful if:
- 
-1. ✅ **Scraper works**: Can extract 20+ skills from GitHub
-2. ✅ **Agent categorizes coherently**: 80%+ of categorizations make sense to humans
-3. ✅ **Taxonomy is useful**: Browsing tree is more intuitive than flat list
-4. ✅ **Agent adapts**: Creates new categories when needed
-5. ✅ **CLI is usable**: Can scrape, index, browse, search in <5 commands
-6. ✅ **Performance is acceptable**: Categorizing 50 skills takes <5 minutes
- 
-## Demo Script
- 
-```bash
-# 1. Setup
-pip install -r requirements.txt
-cp .env.example .env
-# Add ANTHROPIC_API_KEY to .env
- 
-# 2. Scrape GitHub
-python -m skillindex scrape
-# Output: ✅ Found 47 skills
- 
-# 3. Run agent indexing
-python -m skillindex index
-# Output: 🤖 Categorizing skill 1/47: pdf
-#         🤖 Categorizing skill 2/47: docx
-#         ...
-#         ✅ Taxonomy built successfully
- 
-# 4. Browse the tree
-python -m skillindex tree
-# Output: Interactive tree visualization
- 
-# 5. Search
-python -m skillindex search "PDF"
-# Output: 📦 pdf
-#         Path: Creative > Design > Documents
-#         Extract text, fill forms, merge PDFs...
- 
-# 6. Detailed view
-python -m skillindex show pdf
-# Output: Full skill details with categorization reasoning
-```
- 
-## Evaluation Questions
- 
-After building POC, answer:
- 
-1. **Quality**: How accurate are the agent's categorizations?
-2. **Consistency**: Does agent categorize similar skills similarly?
-3. **Adaptability**: Does it create sensible new categories?
-4. **Discoverability**: Is browsing tree better than flat list?
-5. **Cost**: How much does it cost to categorize 50 skills?
-6. **Speed**: Is it fast enough for 1000+ skills?
-7. **Value**: Would users actually want this vs existing solutions?
- 
-## Next Steps After POC
- 
-If POC succeeds:
-1. **Add database**: Migrate from JSON to PostgreSQL
-2. **Add embeddings**: Semantic search with vector store
-3. **Continuous scraping**: Background jobs for updates
-4. **Installation**: Actually install skills to ~/.claude/skills
-5. **Web UI**: Build visual browser
-6. **Scale up**: Test with 500+ skills
-7. **Community**: Open source, gather feedback
- 
-If POC fails or is mediocre:
-1. **Analyze failures**: Where did agent categorize poorly?
-2. **Improve prompts**: Refine categorization instructions
-3. **Try different approach**: Maybe hierarchical isn't the answer?
-4. **Pivot**: Focus on semantic search instead of tree?
-5. **Kill it**: If not working, cut losses and learn
- 
-## Resources Needed
- 
-- **API Costs**: ~$2-5 for 50 skills (Sonnet, ~1000 tokens/skill)
-- **Time**: 1-2 weeks part-time
-- **GitHub API**: Free tier (5000 req/hour) is plenty
-- **Compute**: Runs on laptop, no special infrastructure
- 
-## Risk Mitigation
- 
-**Risk**: Agent categorizes inconsistently
-- **Mitigation**: Use structured output, few-shot examples in prompt
- 
-**Risk**: Too expensive at scale
-- **Mitigation**: Cache results, only re-categorize on changes, use cheaper model
- 
-**Risk**: Taxonomy becomes unwieldy
-- **Mitigation**: Limit depth to 4 levels, agent suggests merging similar categories
- 
-**Risk**: Skills don't fit hierarchy
-- **Mitigation**: Allow multi-path placements, use tags for cross-cutting
- 
----
- 
-Ready to build? Let's start with setting up the project structure! 🚀

package/references/skill-tree/docs/scraper/README.md

@@ -1,170 +0,0 @@
-# GitHub Skills Indexer
- 
-> An AI-powered system for continuously indexing, organizing, and distributing Agent Skills from GitHub
- 
-## 📚 Documentation
- 
-- **[BRAINSTORM.md](BRAINSTORM.md)** - Comprehensive architecture design, research findings, and technical specifications
-- **[POC_PLAN.md](POC_PLAN.md)** - Proof of concept implementation plan (1-2 weeks)
- 
-## 🎯 Project Vision
- 
-Build a system that:
-1. **Scrapes** GitHub continuously for Agent Skills (Anthropic format)
-2. **Indexes** skills using AI agents for intelligent hierarchical organization
-3. **Organizes** skills in a tree structure (not flat lists) for better discovery
-4. **Distributes** skills with easy installation and loading
- 
-## ✨ Key Innovations
- 
-### 🤖 AI-Managed Taxonomy
-Unlike existing solutions (OpenSkills, SkillsMP) that use flat lists or static categories, we use Claude to:
-- Semantically understand skill purposes
-- Dynamically organize skills into an evolving tree structure
-- Identify relationships (dependencies, alternatives, extensions)
-- Adapt to new skill domains automatically
- 
-### 🌳 Hierarchical Organization
-```
-Development/
-├── Languages/
-│   ├── Python/
-│   │   ├── Testing/
-│   │   │   └── pytest-helper
-│   │   └── Web-Frameworks/
-│   │       └── fastapi-crud-generator
-│   └── JavaScript/
-│       └── React/
-│           └── component-builder
-└── Infrastructure/
-    └── Docker/
-        └── compose-manager
-```
- 
-### 🔗 Multi-Dimensional Placement
-Skills can exist in multiple taxonomy paths:
-- `react-testing-helper` → Development/Frontend/React/Testing
-- `react-testing-helper` → Development/Testing/Unit-Testing
-- `react-testing-helper` → Languages/JavaScript/Testing-Tools
- 
-## 🏗️ Architecture Overview
- 
-```
-GitHub Repos → Scraper → AI Agent Indexer → Database → API/CLI/Web UI
-                           ↓
-                    Taxonomy Manager
-                    (rebalancing)
-```
- 
-**Components:**
-1. **GitHub Scraper** - Discovers SKILL.md files across repos
-2. **AI Agent Indexer** - Categorizes skills using Claude
-3. **Data Layer** - PostgreSQL + pgvector for storage & search
-4. **API Service** - FastAPI with semantic search, taxonomy navigation
-5. **CLI Tool** - Browse tree, search, install skills
-6. **Web UI** - Visual browser with tree navigation (optional)
- 
-## 📊 Comparison with Existing Solutions
- 
-| Feature | OpenSkills | rscheiwe/open-skills | **Our Solution** |
-|---------|-----------|---------------------|------------------|
-| Organization | Flat directories | Flat with tags | **🌳 AI-managed tree** |
-| Discovery | Manual/keyword | Vector search | **Tree + semantic + vector** |
-| Installation | Git clone | API/library | **Smart dependency resolution** |
-| Versioning | ❌ | ✅ | ✅ |
-| Continuous Updates | ❌ | ❌ | **✅ Background scraping** |
-| Relationships | ❌ | ❌ | **✅ Dependency graph** |
-| Context-Aware | ❌ | ❌ | **✅ Project-based suggestions** |
- 
-## 🚀 Getting Started (POC)
- 
-See [POC_PLAN.md](POC_PLAN.md) for implementation details.
- 
-Quick start:
-```bash
-# 1. Setup
-git clone <this-repo>
-cd skillindex-poc
-pip install -r requirements.txt
-cp .env.example .env
-# Add ANTHROPIC_API_KEY
- 
-# 2. Scrape & Index
-python -m skillindex scrape  # Find skills on GitHub
-python -m skillindex index   # AI categorization
- 
-# 3. Explore
-python -m skillindex tree                     # Browse hierarchy
-python -m skillindex search "testing tools"   # Semantic search
-python -m skillindex show pytest-helper       # Skill details
-```
- 
-## 📖 Research Sources
- 
-### Official Resources
-- [Anthropic Skills Repository](https://github.com/anthropics/skills)
-- [Agent Skills Specification](https://agentskills.io/home)
-- [Claude Code Skills Docs](https://code.claude.com/docs/en/skills)
- 
-### Existing Implementations
-- [OpenSkills](https://github.com/numman-ali/openskills) - Universal CLI loader
-- [rscheiwe/open-skills](https://github.com/rscheiwe/open-skills) - Modular framework
-- [Awesome Claude Skills](https://github.com/travisvn/awesome-claude-skills) - Curated list
- 
-### Articles & Analysis
-- [Anthropic Engineering Blog](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills)
-- [Simon Willison on Skills](https://simonwillison.net/2025/Oct/16/claude-skills/)
-- [Claude Skills Deep Dive](https://leehanchung.github.io/blogs/2025/10/26/claude-skills-deep-dive/)
- 
-## 🗺️ Roadmap
- 
-### Phase 1: POC (Weeks 1-2) ✅ Planning
-- [x] Research existing implementations
-- [x] Design architecture
-- [ ] Build basic scraper
-- [ ] Implement AI categorization
-- [ ] Create simple CLI
-- [ ] Validate with 20-50 skills
- 
-### Phase 2: MVP (Weeks 3-6)
-- [ ] PostgreSQL + pgvector setup
-- [ ] Vector embeddings for semantic search
-- [ ] Background task queue
-- [ ] REST API
-- [ ] Enhanced CLI with install
-- [ ] Basic quality scoring
- 
-### Phase 3: Advanced Features (Weeks 7-12)
-- [ ] Web UI with tree visualization
-- [ ] Dependency resolution
-- [ ] Context-aware recommendations
-- [ ] Community features (ratings)
-- [ ] Analytics dashboard
- 
-### Phase 4: Scale & Launch (Weeks 13-16)
-- [ ] Performance optimization
-- [ ] Comprehensive testing
-- [ ] Documentation
-- [ ] Public launch
-- [ ] Community building
- 
-## 🎓 Learning Outcomes
- 
-This project explores:
-- **AI-assisted data organization** - Can agents maintain coherent taxonomies?
-- **Semantic understanding at scale** - Categorizing 1000+ diverse skills
-- **Progressive disclosure** - Loading skills on-demand
-- **Developer tools** - Building CLI/API for developer experience
-- **Open source ecosystems** - Indexing decentralized repositories
- 
-## 🤝 Contributing
- 
-(Coming soon after POC validation)
- 
-## 📄 License
- 
-MIT (tentative - to be decided)
- 
----
- 
-**Status**: 📐 Design Phase - [See BRAINSTORM.md for full details](BRAINSTORM.md)