macro-agent 0.1.2 → 0.1.3

package/references/skill-tree/docs/scraper/BRAINSTORM.md

@@ -1,583 +0,0 @@
-# GitHub Skills Indexer - Architecture Brainstorm
- 
-## Executive Summary
- 
-Build a system that continuously scrapes GitHub for Agent Skills (Anthropic format), indexes them using AI agents for hierarchical organization, and provides easy installation/loading mechanisms.
- 
-## Research Findings
- 
-### Existing Implementations
- 
-#### 1. **OpenSkills** (numman-ali/openskills)
-- **Architecture**: CLI tool with filesystem-based discovery
-- **Discovery**: Scans 4 priority locations (`./.agent/skills/`, `~/.agent/skills/`, `./.claude/skills/`, `~/.claude/skills/`)
-- **Installation**: Clones GitHub repos to skill directories
-- **Loading**: Progressive disclosure via `openskills sync` → generates XML block, then `openskills read <skill-name>` on-demand
-- **Organization**: Flat directory structure with deduplication by name
-- **Strengths**: Simple, works across multiple agent platforms
-- **Limitations**: No hierarchical organization, manual discovery
- 
-#### 2. **rscheiwe/open-skills**
-- **Architecture**: FastAPI service + embedded library with PostgreSQL + pgvector
-- **Discovery**: Embedding-based semantic search using vector similarity
-- **Features**:
-  - Skill versioning (semantic versioning support)
-  - RESTful API for CRUD operations
-  - `.well-known/skills.json` manifest for tool discovery
-  - Supports both service and library modes
-- **Strengths**: Sophisticated search, versioning, API-first
-- **Limitations**: Still relatively flat organization, requires infrastructure
- 
-#### 3. **Anthropic Official Skills Repo**
-- **Organization**: Simple categorical folders (Creative & Design, Development & Technical, Enterprise & Communication, Document Skills)
-- **Discovery**: Claude Code Marketplace integration, manual browsing
-- **Structure**: Each skill = folder with SKILL.md + optional references/, scripts/, assets/
-- **Strengths**: Official standard, clean examples
-- **Limitations**: No programmatic indexing, static categories
- 
-#### 4. **SkillsMP Marketplace**
-- Aggregates skills from GitHub
-- Focus on human-friendly browsing
-- Limited technical details available
- 
-### Agent Skills Standard Format
- 
-```yaml
----
-name: skill-identifier
-description: Complete description of what the skill does and when to use it
-allowed-tools: Tool1, Tool2  # Optional
-model: claude-model-name      # Optional
----
- 
-# Skill Instructions
- 
-Markdown content with instructions, examples, guidelines...
-```
- 
-**Directory Structure:**
-```
-skill-name/
-├── SKILL.md           # Required: metadata + instructions
-├── references/        # Optional: supporting docs
-├── scripts/          # Optional: helper utilities
-└── assets/           # Optional: templates, configs
-```
- 
-**Key Characteristics:**
-- Skills load progressively (metadata first, full content on-demand)
-- Semantic matching on descriptions (not keywords)
-- Can bundle scripts and resources
-- Cross-platform standard (Anthropic, OpenAI adopted Dec 2025)
- 
-## Proposed Architecture
- 
-### Core Components
- 
-#### 1. **GitHub Scraper Service**
-```
-┌─────────────────────────────────────┐
-│      GitHub Discovery Engine        │
-├─────────────────────────────────────┤
-│ - Search GitHub API for skills      │
-│ - Monitor known skill repos         │
-│ - Detect SKILL.md files             │
-│ - Clone/update repositories         │
-│ - Extract metadata & structure      │
-│ - Queue for processing              │
-└─────────────────────────────────────┘
-```
- 
-**Discovery Strategies:**
-- GitHub Search API: `filename:SKILL.md`
-- Track known repositories (anthropics/skills, awesome lists)
-- GitHub trending/topics monitoring
-- User submissions
-- Recursive discovery (skills that reference other skills)
- 
-**Technologies:**
-- Python + PyGithub or Octokit
-- Background task queue (Celery/RQ)
-- Rate limiting & caching
- 
-#### 2. **AI Agent Indexer** (The Novel Part! 🌟)
-```
-┌─────────────────────────────────────────────────────────┐
-│           Hierarchical Indexing Agent                   │
-├─────────────────────────────────────────────────────────┤
-│  Agent reads skills and organizes them into a tree:     │
-│                                                          │
-│  Development/                                            │
-│  ├── Languages/                                          │
-│  │   ├── Python/                                         │
-│  │   │   ├── Testing/                                    │
-│  │   │   │   └── pytest-helper                           │
-│  │   │   └── Data-Science/                               │
-│  │   │       └── pandas-analyzer                         │
-│  │   └── JavaScript/                                     │
-│  │       └── react-component-builder                     │
-│  └── Infrastructure/                                     │
-│      ├── Docker/                                         │
-│      └── Kubernetes/                                     │
-│                                                          │
-│  Agent responsibilities:                                 │
-│  - Analyze skill content semantically                    │
-│  - Determine appropriate hierarchy placement             │
-│  - Identify relationships (dependencies, alternatives)   │
-│  - Suggest category creation/refinement                  │
-│  - Detect duplicates/similar skills                      │
-│  - Generate rich metadata (tags, use-cases, complexity)  │
-└─────────────────────────────────────────────────────────┘
-```
- 
-**Agent Workflow:**
-1. **Analyze**: Read SKILL.md, understand purpose/domain
-2. **Classify**: Determine taxonomy path using semantic understanding
-3. **Position**: Find optimal placement in tree (may be multiple paths)
-4. **Enrich**: Extract/generate metadata (tags, difficulty, dependencies, alternatives)
-5. **Review**: Periodically rebalance tree, merge similar categories
-6. **Validate**: Check for quality, completeness, conflicts
- 
-**Why Agent-Based?**
-- Better semantic understanding than rule-based categorization
-- Can handle novel/emerging skill types automatically
-- Adapts taxonomy as ecosystem evolves
-- Identifies nuanced relationships between skills
-- Human-like judgment for edge cases
- 
-**Technologies:**
-- Claude API for classification agent
-- Prompt engineering for consistent categorization
-- Structured output (JSON) for taxonomy decisions
-- Vector embeddings for similarity analysis
- 
-#### 3. **Data Layer**
-```
-┌──────────────────────────────────────┐
-│        PostgreSQL + pgvector         │
-├──────────────────────────────────────┤
-│ Tables:                              │
-│ - skills                             │
-│   - id, name, description, metadata  │
-│   - repo_url, install_path           │
-│   - embedding (vector)               │
-│   - quality_score, download_count    │
-│                                      │
-│ - taxonomy_nodes                     │
-│   - id, name, path, parent_id        │
-│   - description, agent_reasoning     │
-│                                      │
-│ - skill_placements                   │
-│   - skill_id, node_id                │
-│   - is_primary, relevance_score      │
-│                                      │
-│ - versions                           │
-│   - skill_id, version, commit_sha    │
-│   - changelog, release_date          │
-│                                      │
-│ - relationships                      │
-│   - from_skill_id, to_skill_id       │
-│   - relationship_type (depends_on,   │
-│     alternative_to, extends, etc.)   │
-└──────────────────────────────────────┘
-```
- 
-**Key Features:**
-- Vector embeddings for semantic search
-- Multi-path taxonomy (skills can belong to multiple categories)
-- Relationship graph for dependencies/alternatives
-- Versioning support
-- Quality scoring (stars, usage stats, agent evaluation)
- 
-#### 4. **Discovery & Search API**
-```
-┌─────────────────────────────────────────┐
-│          FastAPI/GraphQL Service        │
-├─────────────────────────────────────────┤
-│ Endpoints:                              │
-│                                         │
-│ GET  /api/skills                        │
-│      - List skills (paginated)          │
-│      - Filter by category, tags, etc.   │
-│                                         │
-│ GET  /api/skills/{id}                   │
-│      - Get full skill details           │
-│                                         │
-│ POST /api/skills/search                 │
-│      - Semantic search (embeddings)     │
-│      - Natural language queries         │
-│                                         │
-│ GET  /api/taxonomy                      │
-│      - Get tree structure               │
-│      - Navigate hierarchy               │
-│                                         │
-│ GET  /api/taxonomy/{path}/skills        │
-│      - Skills in category (recursive)   │
-│                                         │
-│ POST /api/install                       │
-│      - Install skill(s) to local system │
-│                                         │
-│ GET  /.well-known/skills.json           │
-│      - Tool manifest for discovery      │
-└─────────────────────────────────────────┘
-```
- 
-#### 5. **CLI Tool** (Like openskills but tree-aware)
-```bash
-# Browse tree
-skillindex browse
-skillindex tree Development/Languages/Python
- 
-# Semantic search
-skillindex search "help me test React components"
- 
-# Install
-skillindex install react-testing-helper
-skillindex install Development/Languages/Python/*
- 
-# Sync to local
-skillindex sync --global  # Updates ~/.claude/skills
-skillindex sync --project # Updates ./.claude/skills
- 
-# List installed
-skillindex list --local
-skillindex list --tree
- 
-# Agent management
-skillindex reindex      # Trigger agent re-categorization
-skillindex suggest      # Agent suggests skills based on project
-```
- 
-#### 6. **Web UI** (Optional but nice!)
-```
-┌─────────────────────────────────────┐
-│      Interactive Skill Browser      │
-├─────────────────────────────────────┤
-│ - Tree navigation (collapsible)     │
-│ - Search with filters               │
-│ - Skill detail pages                │
-│ - Installation buttons              │
-│ - Community ratings/reviews         │
-│ - Usage statistics                  │
-│ - Relationship visualizations       │
-└─────────────────────────────────────┘
-```
- 
-### Data Flow
- 
-```
-┌──────────┐     ┌──────────┐     ┌────────────┐     ┌──────────┐
-│  GitHub  │────▶│ Scraper  │────▶│  AI Agent  │────▶│ Database │
-│   Repos  │     │ Service  │     │  Indexer   │     │ (PG+vec) │
-└──────────┘     └──────────┘     └────────────┘     └──────────┘
-                                         │                  │
-                                         │                  │
-                                         ▼                  ▼
-                                  ┌────────────┐     ┌──────────┐
-                                  │  Taxonomy  │     │   API    │
-                                  │  Rebalance │     │ Service  │
-                                  └────────────┘     └──────────┘
-                                                           │
-                                         ┌─────────────────┼─────────────┐
-                                         ▼                 ▼             ▼
-                                    ┌────────┐       ┌────────┐   ┌────────┐
-                                    │  CLI   │       │  Web   │   │ Direct │
-                                    │  Tool  │       │   UI   │   │  API   │
-                                    └────────┘       └────────┘   └────────┘
-```
- 
-## Novel Features & Differentiators
- 
-### 1. **AI-Managed Hierarchical Taxonomy** 🎯
-- **Current State**: All existing solutions use flat lists or simple hardcoded categories
-- **Our Approach**: AI agents continuously organize skills into an evolving tree structure
-- **Benefits**:
-  - Better discovery through logical grouping
-  - Automatic adaptation to new skill domains
-  - Semantic understanding of skill relationships
-  - Handles cross-cutting concerns (skill can be in multiple branches)
- 
-### 2. **Multi-Dimensional Organization**
-Skills can exist in multiple taxonomy paths:
-```
-react-testing-helper
-├── Development/Frontend/React/Testing
-├── Development/Testing/Unit-Testing
-└── Languages/JavaScript/Testing-Tools
-```
- 
-### 3. **Relationship Graph**
-Beyond hierarchy, capture:
-- Dependencies: "X requires Y to function"
-- Alternatives: "X and Y solve similar problems differently"
-- Extensions: "X builds upon Y"
-- Conflicts: "X and Y cannot coexist"
- 
-### 4. **Smart Installation**
-```bash
-# Install with dependency resolution
-skillindex install react-testing-helper --with-deps
- 
-# Install category (agent filters relevant ones)
-skillindex install Development/Python --suggest
-> Installing: pytest-helper, python-debugger, type-checker
-> Suggest also: pandas-analyzer (commonly used together)
-> Skip: advanced-ml-trainer (enterprise license)
-```
- 
-### 5. **Context-Aware Recommendations**
-Agent analyzes current project and suggests skills:
-```bash
-cd my-python-project
-skillindex suggest
-> Detected: Python FastAPI project with pytest
-> Recommended skills:
->   - api-testing-helper (for FastAPI testing)
->   - pydantic-validator (you're using Pydantic models)
->   - docker-compose-manager (found docker-compose.yml)
-```
- 
-### 6. **Quality & Trust Signals**
-- GitHub stars, contributors, last updated
-- Agent-evaluated quality score (code quality, documentation, tests)
-- Community ratings
-- Verification badges (official, community-trusted)
-- Security scanning results
- 
-### 7. **Continuous Learning**
-- Scraper runs periodically (daily/weekly)
-- Agent periodically rebalances taxonomy
-- New categories emerge automatically
-- Deprecated skills identified and flagged
- 
-## Technical Stack Recommendations
- 
-### Backend
-- **Language**: Python 3.11+
-- **Web Framework**: FastAPI (async, modern, OpenAPI docs)
-- **Database**: PostgreSQL 15+ with pgvector extension
-- **Task Queue**: Celery with Redis
-- **AI Integration**: Anthropic Claude API
-- **GitHub API**: PyGithub or octokit.py
-- **Vector Store**: pgvector or Qdrant
- 
-### CLI
-- **Framework**: Click or Typer (Python)
-- **Alternative**: Rust CLI for performance (clap + tokio)
- 
-### Web UI (Optional)
-- **Frontend**: Next.js/React with TailwindCSS
-- **Visualization**: D3.js for tree/graph rendering
-- **API Client**: React Query
- 
-### Infrastructure
-- **Containerization**: Docker + Docker Compose
-- **Orchestration**: Kubernetes (for scale) or simple VPS
-- **CI/CD**: GitHub Actions
-- **Monitoring**: Prometheus + Grafana
-- **Logs**: Structured logging with Loki
- 
-## Implementation Phases
- 
-### Phase 1: MVP (Weeks 1-4)
-- [ ] Basic GitHub scraper for known repos
-- [ ] PostgreSQL schema with basic tables
-- [ ] Simple AI agent for classification (3-level taxonomy)
-- [ ] REST API with CRUD operations
-- [ ] Basic CLI for search/install
-- [ ] Local storage integration (~/.claude/skills)
- 
-### Phase 2: Enhanced Indexing (Weeks 5-8)
-- [ ] Vector embeddings for semantic search
-- [ ] Multi-path taxonomy support
-- [ ] Relationship detection (dependencies)
-- [ ] Quality scoring system
-- [ ] Version tracking
-- [ ] Background task queue for continuous scraping
- 
-### Phase 3: Advanced Features (Weeks 9-12)
-- [ ] Web UI with tree browser
-- [ ] Context-aware recommendations
-- [ ] Dependency resolution on install
-- [ ] Agent-driven taxonomy rebalancing
-- [ ] Community features (ratings, comments)
-- [ ] Analytics dashboard
- 
-### Phase 4: Scale & Polish (Weeks 13-16)
-- [ ] Performance optimization
-- [ ] Comprehensive testing
-- [ ] Documentation & examples
-- [ ] Public API rate limiting
-- [ ] Security hardening
-- [ ] Marketing & community building
- 
-## Example Taxonomy (Agent-Generated)
- 
-```
-Root/
-├── Development/
-│   ├── Languages/
-│   │   ├── Python/
-│   │   │   ├── Testing/
-│   │   │   │   ├── pytest-helper
-│   │   │   │   ├── unittest-booster
-│   │   │   │   └── coverage-reporter
-│   │   │   ├── Web-Frameworks/
-│   │   │   │   ├── FastAPI/
-│   │   │   │   │   ├── fastapi-crud-generator
-│   │   │   │   │   └── fastapi-auth-helper
-│   │   │   │   └── Django/
-│   │   │   └── Data-Science/
-│   │   │       ├── pandas-analyzer
-│   │   │       └── matplotlib-helper
-│   │   ├── JavaScript/
-│   │   │   ├── React/
-│   │   │   │   ├── component-builder
-│   │   │   │   ├── testing-helper
-│   │   │   │   └── state-manager
-│   │   │   └── Node/
-│   │   └── Rust/
-│   ├── Infrastructure/
-│   │   ├── Docker/
-│   │   │   ├── compose-manager
-│   │   │   └── optimization-helper
-│   │   ├── Kubernetes/
-│   │   └── CI-CD/
-│   └── Databases/
-│       ├── SQL/
-│       ├── NoSQL/
-│       └── Migrations/
-├── Creative/
-│   ├── Design/
-│   │   ├── UI-UX/
-│   │   ├── Graphics/
-│   │   └── Brand/
-│   ├── Content/
-│   └── Media/
-├── Data/
-│   ├── Analysis/
-│   ├── Visualization/
-│   └── Machine-Learning/
-│       ├── Training/
-│       ├── Deployment/
-│       └── Evaluation/
-├── Communication/
-│   ├── Documentation/
-│   ├── Collaboration/
-│   └── Presentations/
-└── Security/
-    ├── Auditing/
-    ├── Compliance/
-    └── Penetration-Testing/
-```
- 
-## Challenges & Solutions
- 
-### Challenge 1: Taxonomy Drift
-**Problem**: Agent might categorize same skill differently over time
-**Solution**:
-- Store agent reasoning with each placement
-- Version taxonomy changes
-- Allow manual overrides
-- Periodic human review of major shifts
- 
-### Challenge 2: GitHub API Rate Limits
-**Problem**: 5000 requests/hour (authenticated)
-**Solution**:
-- Incremental scanning (only new/updated repos)
-- Caching
-- Multiple API tokens rotation
-- Prioritize popular repos
- 
-### Challenge 3: Skill Quality Variance
-**Problem**: Anyone can create a skill, quality varies
-**Solution**:
-- Multi-factor quality scoring
-- Agent-based code review
-- Community ratings
-- Verification badges for trusted sources
-- Quarantine period for new skills
- 
-### Challenge 4: Installation Conflicts
-**Problem**: Skills might conflict or require specific environments
-**Solution**:
-- Dependency metadata in skills
-- Conflict detection before install
-- Virtual environment support
-- Sandboxing options
- 
-### Challenge 5: Agent Classification Costs
-**Problem**: Running Claude API for every skill is expensive
-**Solution**:
-- Batch processing
-- Cache classifications
-- Re-classify only on significant content changes
-- Use cheaper models for initial pass, expensive for review
-- Incremental learning (fine-tuned model eventually)
- 
-## Success Metrics
- 
-### Technical Metrics
-- Skills indexed: Target 1000+ in first 6 months
-- Indexing accuracy: 90%+ correct categorization (human eval)
-- Search relevance: Top-3 results useful for 80%+ queries
-- Installation success rate: 95%+
-- API latency: <200ms p95
- 
-### User Metrics
-- Monthly active users
-- Skills installed per user
-- Search-to-install conversion
-- User satisfaction (NPS)
-- Community contributions
- 
-### Ecosystem Metrics
-- Number of skill creators
-- Skills updated/maintained (freshness)
-- Diversity of skill categories
-- Cross-platform compatibility
- 
-## Open Questions for Discussion
- 
-1. **Taxonomy Depth**: How many levels deep should the tree go? (3? 5? unlimited?)
-2. **Agent Model**: Which Claude model for indexing? (Haiku for speed vs Sonnet for accuracy)
-3. **Update Frequency**: How often to rescrape GitHub? (daily, weekly, on-demand?)
-4. **Hosting**: Self-hosted vs managed service vs hybrid?
-5. **Licensing**: How to handle skills with various licenses?
-6. **Monetization**: Free tier limits? Premium features?
-7. **Governance**: How to handle disputes over categorization?
-8. **Private Skills**: Support for enterprise private skill repos?
- 
-## Next Steps
- 
-1. **Validation**: Get feedback on architecture
-2. **Prototyping**: Build quick proof-of-concept for agent-based categorization
-3. **Scope**: Decide on MVP feature set
-4. **Tech Stack**: Finalize technology choices
-5. **Setup**: Initialize project structure, repo, dev environment
-6. **Build**: Start with Phase 1 implementation
- 
----
- 
-## Resources & References
- 
-### Official Documentation
-- [Anthropic Skills Repository](https://github.com/anthropics/skills)
-- [Agent Skills Specification](https://agentskills.io/home)
-- [Claude Code Skills Docs](https://code.claude.com/docs/en/skills)
-- [Agent Skills API Docs](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview)
- 
-### Existing Implementations
-- [OpenSkills](https://github.com/numman-ali/openskills) - Universal CLI skills loader
-- [rscheiwe/open-skills](https://github.com/rscheiwe/open-skills) - Modular framework with API
-- [BandarLabs/open-skills](https://github.com/BandarLabs/open-skills) - Run skills with any LLM
- 
-### Community Resources
-- [Awesome Claude Skills](https://github.com/travisvn/awesome-claude-skills) - Curated list
-- [SkillsMP Marketplace](https://skillsmp.com/) - Community marketplace
-- [Skills Registries Collection](https://medium.com/@frulouis/25-top-claude-agent-skills-registries-community-collections-you-should-know-2025-52aab45c877d)
- 
-### Technical Inspiration
-- [Anthropic Engineering Blog](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills)
-- [Simon Willison's Take on Skills](https://simonwillison.net/2025/Oct/16/claude-skills/)
-- [Claude Skills Deep Dive](https://leehanchung.github.io/blogs/2025/10/26/claude-skills-deep-dive/)