claude-code-swarm 0.3.4 → 0.3.6

package/references/skill-tree/docs/scraper/BRAINSTORM.md

@@ -0,0 +1,583 @@
+# GitHub Skills Indexer - Architecture Brainstorm
+ 
+## Executive Summary
+ 
+Build a system that continuously scrapes GitHub for Agent Skills (Anthropic format), indexes them using AI agents for hierarchical organization, and provides easy installation/loading mechanisms.
+ 
+## Research Findings
+ 
+### Existing Implementations
+ 
+#### 1. **OpenSkills** (numman-ali/openskills)
+- **Architecture**: CLI tool with filesystem-based discovery
+- **Discovery**: Scans 4 priority locations (`./.agent/skills/`, `~/.agent/skills/`, `./.claude/skills/`, `~/.claude/skills/`)
+- **Installation**: Clones GitHub repos to skill directories
+- **Loading**: Progressive disclosure via `openskills sync` → generates XML block, then `openskills read <skill-name>` on-demand
+- **Organization**: Flat directory structure with deduplication by name
+- **Strengths**: Simple, works across multiple agent platforms
+- **Limitations**: No hierarchical organization, manual discovery
+ 
+#### 2. **rscheiwe/open-skills**
+- **Architecture**: FastAPI service + embedded library with PostgreSQL + pgvector
+- **Discovery**: Embedding-based semantic search using vector similarity
+- **Features**:
+  - Skill versioning (semantic versioning support)
+  - RESTful API for CRUD operations
+  - `.well-known/skills.json` manifest for tool discovery
+  - Supports both service and library modes
+- **Strengths**: Sophisticated search, versioning, API-first
+- **Limitations**: Still relatively flat organization, requires infrastructure
+ 
+#### 3. **Anthropic Official Skills Repo**
+- **Organization**: Simple categorical folders (Creative & Design, Development & Technical, Enterprise & Communication, Document Skills)
+- **Discovery**: Claude Code Marketplace integration, manual browsing
+- **Structure**: Each skill = folder with SKILL.md + optional references/, scripts/, assets/
+- **Strengths**: Official standard, clean examples
+- **Limitations**: No programmatic indexing, static categories
+ 
+#### 4. **SkillsMP Marketplace**
+- Aggregates skills from GitHub
+- Focus on human-friendly browsing
+- Limited technical details available
+ 
+### Agent Skills Standard Format
+ 
+```yaml
+---
+name: skill-identifier
+description: Complete description of what the skill does and when to use it
+allowed-tools: Tool1, Tool2  # Optional
+model: claude-model-name      # Optional
+---
+ 
+# Skill Instructions
+ 
+Markdown content with instructions, examples, guidelines...
+```
+ 
+**Directory Structure:**
+```
+skill-name/
+├── SKILL.md           # Required: metadata + instructions
+├── references/        # Optional: supporting docs
+├── scripts/          # Optional: helper utilities
+└── assets/           # Optional: templates, configs
+```
+ 
+**Key Characteristics:**
+- Skills load progressively (metadata first, full content on-demand)
+- Semantic matching on descriptions (not keywords)
+- Can bundle scripts and resources
+- Cross-platform standard (Anthropic, OpenAI adopted Dec 2025)
+ 
+## Proposed Architecture
+ 
+### Core Components
+ 
+#### 1. **GitHub Scraper Service**
+```
+┌─────────────────────────────────────┐
+│      GitHub Discovery Engine        │
+├─────────────────────────────────────┤
+│ - Search GitHub API for skills      │
+│ - Monitor known skill repos         │
+│ - Detect SKILL.md files             │
+│ - Clone/update repositories         │
+│ - Extract metadata & structure      │
+│ - Queue for processing              │
+└─────────────────────────────────────┘
+```
+ 
+**Discovery Strategies:**
+- GitHub Search API: `filename:SKILL.md`
+- Track known repositories (anthropics/skills, awesome lists)
+- GitHub trending/topics monitoring
+- User submissions
+- Recursive discovery (skills that reference other skills)
+ 
+**Technologies:**
+- Python + PyGithub or Octokit
+- Background task queue (Celery/RQ)
+- Rate limiting & caching
+ 
+#### 2. **AI Agent Indexer** (The Novel Part! 🌟)
+```
+┌─────────────────────────────────────────────────────────┐
+│           Hierarchical Indexing Agent                   │
+├─────────────────────────────────────────────────────────┤
+│  Agent reads skills and organizes them into a tree:     │
+│                                                          │
+│  Development/                                            │
+│  ├── Languages/                                          │
+│  │   ├── Python/                                         │
+│  │   │   ├── Testing/                                    │
+│  │   │   │   └── pytest-helper                           │
+│  │   │   └── Data-Science/                               │
+│  │   │       └── pandas-analyzer                         │
+│  │   └── JavaScript/                                     │
+│  │       └── react-component-builder                     │
+│  └── Infrastructure/                                     │
+│      ├── Docker/                                         │
+│      └── Kubernetes/                                     │
+│                                                          │
+│  Agent responsibilities:                                 │
+│  - Analyze skill content semantically                    │
+│  - Determine appropriate hierarchy placement             │
+│  - Identify relationships (dependencies, alternatives)   │
+│  - Suggest category creation/refinement                  │
+│  - Detect duplicates/similar skills                      │
+│  - Generate rich metadata (tags, use-cases, complexity)  │
+└─────────────────────────────────────────────────────────┘
+```
+ 
+**Agent Workflow:**
+1. **Analyze**: Read SKILL.md, understand purpose/domain
+2. **Classify**: Determine taxonomy path using semantic understanding
+3. **Position**: Find optimal placement in tree (may be multiple paths)
+4. **Enrich**: Extract/generate metadata (tags, difficulty, dependencies, alternatives)
+5. **Review**: Periodically rebalance tree, merge similar categories
+6. **Validate**: Check for quality, completeness, conflicts
+ 
+**Why Agent-Based?**
+- Better semantic understanding than rule-based categorization
+- Can handle novel/emerging skill types automatically
+- Adapts taxonomy as ecosystem evolves
+- Identifies nuanced relationships between skills
+- Human-like judgment for edge cases
+ 
+**Technologies:**
+- Claude API for classification agent
+- Prompt engineering for consistent categorization
+- Structured output (JSON) for taxonomy decisions
+- Vector embeddings for similarity analysis
+ 
+#### 3. **Data Layer**
+```
+┌──────────────────────────────────────┐
+│        PostgreSQL + pgvector         │
+├──────────────────────────────────────┤
+│ Tables:                              │
+│ - skills                             │
+│   - id, name, description, metadata  │
+│   - repo_url, install_path           │
+│   - embedding (vector)               │
+│   - quality_score, download_count    │
+│                                      │
+│ - taxonomy_nodes                     │
+│   - id, name, path, parent_id        │
+│   - description, agent_reasoning     │
+│                                      │
+│ - skill_placements                   │
+│   - skill_id, node_id                │
+│   - is_primary, relevance_score      │
+│                                      │
+│ - versions                           │
+│   - skill_id, version, commit_sha    │
+│   - changelog, release_date          │
+│                                      │
+│ - relationships                      │
+│   - from_skill_id, to_skill_id       │
+│   - relationship_type (depends_on,   │
+│     alternative_to, extends, etc.)   │
+└──────────────────────────────────────┘
+```
+ 
+**Key Features:**
+- Vector embeddings for semantic search
+- Multi-path taxonomy (skills can belong to multiple categories)
+- Relationship graph for dependencies/alternatives
+- Versioning support
+- Quality scoring (stars, usage stats, agent evaluation)
+ 
+#### 4. **Discovery & Search API**
+```
+┌─────────────────────────────────────────┐
+│          FastAPI/GraphQL Service        │
+├─────────────────────────────────────────┤
+│ Endpoints:                              │
+│                                         │
+│ GET  /api/skills                        │
+│      - List skills (paginated)          │
+│      - Filter by category, tags, etc.   │
+│                                         │
+│ GET  /api/skills/{id}                   │
+│      - Get full skill details           │
+│                                         │
+│ POST /api/skills/search                 │
+│      - Semantic search (embeddings)     │
+│      - Natural language queries         │
+│                                         │
+│ GET  /api/taxonomy                      │
+│      - Get tree structure               │
+│      - Navigate hierarchy               │
+│                                         │
+│ GET  /api/taxonomy/{path}/skills        │
+│      - Skills in category (recursive)   │
+│                                         │
+│ POST /api/install                       │
+│      - Install skill(s) to local system │
+│                                         │
+│ GET  /.well-known/skills.json           │
+│      - Tool manifest for discovery      │
+└─────────────────────────────────────────┘
+```
+ 
+#### 5. **CLI Tool** (Like openskills but tree-aware)
+```bash
+# Browse tree
+skillindex browse
+skillindex tree Development/Languages/Python
+ 
+# Semantic search
+skillindex search "help me test React components"
+ 
+# Install
+skillindex install react-testing-helper
+skillindex install Development/Languages/Python/*
+ 
+# Sync to local
+skillindex sync --global  # Updates ~/.claude/skills
+skillindex sync --project # Updates ./.claude/skills
+ 
+# List installed
+skillindex list --local
+skillindex list --tree
+ 
+# Agent management
+skillindex reindex      # Trigger agent re-categorization
+skillindex suggest      # Agent suggests skills based on project
+```
+ 
+#### 6. **Web UI** (Optional but nice!)
+```
+┌─────────────────────────────────────┐
+│      Interactive Skill Browser      │
+├─────────────────────────────────────┤
+│ - Tree navigation (collapsible)     │
+│ - Search with filters               │
+│ - Skill detail pages                │
+│ - Installation buttons              │
+│ - Community ratings/reviews         │
+│ - Usage statistics                  │
+│ - Relationship visualizations       │
+└─────────────────────────────────────┘
+```
+ 
+### Data Flow
+ 
+```
+┌──────────┐     ┌──────────┐     ┌────────────┐     ┌──────────┐
+│  GitHub  │────▶│ Scraper  │────▶│  AI Agent  │────▶│ Database │
+│   Repos  │     │ Service  │     │  Indexer   │     │ (PG+vec) │
+└──────────┘     └──────────┘     └────────────┘     └──────────┘
+                                         │                  │
+                                         │                  │
+                                         ▼                  ▼
+                                  ┌────────────┐     ┌──────────┐
+                                  │  Taxonomy  │     │   API    │
+                                  │  Rebalance │     │ Service  │
+                                  └────────────┘     └──────────┘
+                                                           │
+                                         ┌─────────────────┼─────────────┐
+                                         ▼                 ▼             ▼
+                                    ┌────────┐       ┌────────┐   ┌────────┐
+                                    │  CLI   │       │  Web   │   │ Direct │
+                                    │  Tool  │       │   UI   │   │  API   │
+                                    └────────┘       └────────┘   └────────┘
+```
+ 
+## Novel Features & Differentiators
+ 
+### 1. **AI-Managed Hierarchical Taxonomy** 🎯
+- **Current State**: All existing solutions use flat lists or simple hardcoded categories
+- **Our Approach**: AI agents continuously organize skills into an evolving tree structure
+- **Benefits**:
+  - Better discovery through logical grouping
+  - Automatic adaptation to new skill domains
+  - Semantic understanding of skill relationships
+  - Handles cross-cutting concerns (skill can be in multiple branches)
+ 
+### 2. **Multi-Dimensional Organization**
+Skills can exist in multiple taxonomy paths:
+```
+react-testing-helper
+├── Development/Frontend/React/Testing
+├── Development/Testing/Unit-Testing
+└── Languages/JavaScript/Testing-Tools
+```
+ 
+### 3. **Relationship Graph**
+Beyond hierarchy, capture:
+- Dependencies: "X requires Y to function"
+- Alternatives: "X and Y solve similar problems differently"
+- Extensions: "X builds upon Y"
+- Conflicts: "X and Y cannot coexist"
+ 
+### 4. **Smart Installation**
+```bash
+# Install with dependency resolution
+skillindex install react-testing-helper --with-deps
+ 
+# Install category (agent filters relevant ones)
+skillindex install Development/Python --suggest
+> Installing: pytest-helper, python-debugger, type-checker
+> Suggest also: pandas-analyzer (commonly used together)
+> Skip: advanced-ml-trainer (enterprise license)
+```
+ 
+### 5. **Context-Aware Recommendations**
+Agent analyzes current project and suggests skills:
+```bash
+cd my-python-project
+skillindex suggest
+> Detected: Python FastAPI project with pytest
+> Recommended skills:
+>   - api-testing-helper (for FastAPI testing)
+>   - pydantic-validator (you're using Pydantic models)
+>   - docker-compose-manager (found docker-compose.yml)
+```
+ 
+### 6. **Quality & Trust Signals**
+- GitHub stars, contributors, last updated
+- Agent-evaluated quality score (code quality, documentation, tests)
+- Community ratings
+- Verification badges (official, community-trusted)
+- Security scanning results
+ 
+### 7. **Continuous Learning**
+- Scraper runs periodically (daily/weekly)
+- Agent periodically rebalances taxonomy
+- New categories emerge automatically
+- Deprecated skills identified and flagged
+ 
+## Technical Stack Recommendations
+ 
+### Backend
+- **Language**: Python 3.11+
+- **Web Framework**: FastAPI (async, modern, OpenAPI docs)
+- **Database**: PostgreSQL 15+ with pgvector extension
+- **Task Queue**: Celery with Redis
+- **AI Integration**: Anthropic Claude API
+- **GitHub API**: PyGithub or octokit.py
+- **Vector Store**: pgvector or Qdrant
+ 
+### CLI
+- **Framework**: Click or Typer (Python)
+- **Alternative**: Rust CLI for performance (clap + tokio)
+ 
+### Web UI (Optional)
+- **Frontend**: Next.js/React with TailwindCSS
+- **Visualization**: D3.js for tree/graph rendering
+- **API Client**: React Query
+ 
+### Infrastructure
+- **Containerization**: Docker + Docker Compose
+- **Orchestration**: Kubernetes (for scale) or simple VPS
+- **CI/CD**: GitHub Actions
+- **Monitoring**: Prometheus + Grafana
+- **Logs**: Structured logging with Loki
+ 
+## Implementation Phases
+ 
+### Phase 1: MVP (Weeks 1-4)
+- [ ] Basic GitHub scraper for known repos
+- [ ] PostgreSQL schema with basic tables
+- [ ] Simple AI agent for classification (3-level taxonomy)
+- [ ] REST API with CRUD operations
+- [ ] Basic CLI for search/install
+- [ ] Local storage integration (~/.claude/skills)
+ 
+### Phase 2: Enhanced Indexing (Weeks 5-8)
+- [ ] Vector embeddings for semantic search
+- [ ] Multi-path taxonomy support
+- [ ] Relationship detection (dependencies)
+- [ ] Quality scoring system
+- [ ] Version tracking
+- [ ] Background task queue for continuous scraping
+ 
+### Phase 3: Advanced Features (Weeks 9-12)
+- [ ] Web UI with tree browser
+- [ ] Context-aware recommendations
+- [ ] Dependency resolution on install
+- [ ] Agent-driven taxonomy rebalancing
+- [ ] Community features (ratings, comments)
+- [ ] Analytics dashboard
+ 
+### Phase 4: Scale & Polish (Weeks 13-16)
+- [ ] Performance optimization
+- [ ] Comprehensive testing
+- [ ] Documentation & examples
+- [ ] Public API rate limiting
+- [ ] Security hardening
+- [ ] Marketing & community building
+ 
+## Example Taxonomy (Agent-Generated)
+ 
+```
+Root/
+├── Development/
+│   ├── Languages/
+│   │   ├── Python/
+│   │   │   ├── Testing/
+│   │   │   │   ├── pytest-helper
+│   │   │   │   ├── unittest-booster
+│   │   │   │   └── coverage-reporter
+│   │   │   ├── Web-Frameworks/
+│   │   │   │   ├── FastAPI/
+│   │   │   │   │   ├── fastapi-crud-generator
+│   │   │   │   │   └── fastapi-auth-helper
+│   │   │   │   └── Django/
+│   │   │   └── Data-Science/
+│   │   │       ├── pandas-analyzer
+│   │   │       └── matplotlib-helper
+│   │   ├── JavaScript/
+│   │   │   ├── React/
+│   │   │   │   ├── component-builder
+│   │   │   │   ├── testing-helper
+│   │   │   │   └── state-manager
+│   │   │   └── Node/
+│   │   └── Rust/
+│   ├── Infrastructure/
+│   │   ├── Docker/
+│   │   │   ├── compose-manager
+│   │   │   └── optimization-helper
+│   │   ├── Kubernetes/
+│   │   └── CI-CD/
+│   └── Databases/
+│       ├── SQL/
+│       ├── NoSQL/
+│       └── Migrations/
+├── Creative/
+│   ├── Design/
+│   │   ├── UI-UX/
+│   │   ├── Graphics/
+│   │   └── Brand/
+│   ├── Content/
+│   └── Media/
+├── Data/
+│   ├── Analysis/
+│   ├── Visualization/
+│   └── Machine-Learning/
+│       ├── Training/
+│       ├── Deployment/
+│       └── Evaluation/
+├── Communication/
+│   ├── Documentation/
+│   ├── Collaboration/
+│   └── Presentations/
+└── Security/
+    ├── Auditing/
+    ├── Compliance/
+    └── Penetration-Testing/
+```
+ 
+## Challenges & Solutions
+ 
+### Challenge 1: Taxonomy Drift
+**Problem**: Agent might categorize same skill differently over time
+**Solution**:
+- Store agent reasoning with each placement
+- Version taxonomy changes
+- Allow manual overrides
+- Periodic human review of major shifts
+ 
+### Challenge 2: GitHub API Rate Limits
+**Problem**: 5000 requests/hour (authenticated)
+**Solution**:
+- Incremental scanning (only new/updated repos)
+- Caching
+- Multiple API tokens rotation
+- Prioritize popular repos
+ 
+### Challenge 3: Skill Quality Variance
+**Problem**: Anyone can create a skill, quality varies
+**Solution**:
+- Multi-factor quality scoring
+- Agent-based code review
+- Community ratings
+- Verification badges for trusted sources
+- Quarantine period for new skills
+ 
+### Challenge 4: Installation Conflicts
+**Problem**: Skills might conflict or require specific environments
+**Solution**:
+- Dependency metadata in skills
+- Conflict detection before install
+- Virtual environment support
+- Sandboxing options
+ 
+### Challenge 5: Agent Classification Costs
+**Problem**: Running Claude API for every skill is expensive
+**Solution**:
+- Batch processing
+- Cache classifications
+- Re-classify only on significant content changes
+- Use cheaper models for initial pass, expensive for review
+- Incremental learning (fine-tuned model eventually)
+ 
+## Success Metrics
+ 
+### Technical Metrics
+- Skills indexed: Target 1000+ in first 6 months
+- Indexing accuracy: 90%+ correct categorization (human eval)
+- Search relevance: Top-3 results useful for 80%+ queries
+- Installation success rate: 95%+
+- API latency: <200ms p95
+ 
+### User Metrics
+- Monthly active users
+- Skills installed per user
+- Search-to-install conversion
+- User satisfaction (NPS)
+- Community contributions
+ 
+### Ecosystem Metrics
+- Number of skill creators
+- Skills updated/maintained (freshness)
+- Diversity of skill categories
+- Cross-platform compatibility
+ 
+## Open Questions for Discussion
+ 
+1. **Taxonomy Depth**: How many levels deep should the tree go? (3? 5? unlimited?)
+2. **Agent Model**: Which Claude model for indexing? (Haiku for speed vs Sonnet for accuracy)
+3. **Update Frequency**: How often to rescrape GitHub? (daily, weekly, on-demand?)
+4. **Hosting**: Self-hosted vs managed service vs hybrid?
+5. **Licensing**: How to handle skills with various licenses?
+6. **Monetization**: Free tier limits? Premium features?
+7. **Governance**: How to handle disputes over categorization?
+8. **Private Skills**: Support for enterprise private skill repos?
+ 
+## Next Steps
+ 
+1. **Validation**: Get feedback on architecture
+2. **Prototyping**: Build quick proof-of-concept for agent-based categorization
+3. **Scope**: Decide on MVP feature set
+4. **Tech Stack**: Finalize technology choices
+5. **Setup**: Initialize project structure, repo, dev environment
+6. **Build**: Start with Phase 1 implementation
+ 
+---
+ 
+## Resources & References
+ 
+### Official Documentation
+- [Anthropic Skills Repository](https://github.com/anthropics/skills)
+- [Agent Skills Specification](https://agentskills.io/home)
+- [Claude Code Skills Docs](https://code.claude.com/docs/en/skills)
+- [Agent Skills API Docs](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview)
+ 
+### Existing Implementations
+- [OpenSkills](https://github.com/numman-ali/openskills) - Universal CLI skills loader
+- [rscheiwe/open-skills](https://github.com/rscheiwe/open-skills) - Modular framework with API
+- [BandarLabs/open-skills](https://github.com/BandarLabs/open-skills) - Run skills with any LLM
+ 
+### Community Resources
+- [Awesome Claude Skills](https://github.com/travisvn/awesome-claude-skills) - Curated list
+- [SkillsMP Marketplace](https://skillsmp.com/) - Community marketplace
+- [Skills Registries Collection](https://medium.com/@frulouis/25-top-claude-agent-skills-registries-community-collections-you-should-know-2025-52aab45c877d)
+ 
+### Technical Inspiration
+- [Anthropic Engineering Blog](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills)
+- [Simon Willison's Take on Skills](https://simonwillison.net/2025/Oct/16/claude-skills/)
+- [Claude Skills Deep Dive](https://leehanchung.github.io/blogs/2025/10/26/claude-skills-deep-dive/)