macro-agent 0.1.1 → 0.1.2

package/references/skill-tree/scraper/docs/PLAN.md

@@ -0,0 +1,336 @@
+# Skills Indexer Implementation Plan
+
+## Overview
+
+This document outlines the implementation plan for the GitHub Skills Indexer - an AI-powered system that discovers, indexes, and organizes Agent Skills from GitHub repositories into a hierarchical taxonomy.
+
+## Technology Stack
+
+| Component | Choice | Rationale |
+|-----------|--------|-----------|
+| **Runtime** | Node.js 20+ | Async-first, excellent TypeScript support |
+| **Language** | TypeScript 5.x | Type safety, IDE support, matches parent project |
+| **CLI** | Commander.js | Lightweight, well-documented |
+| **HTTP** | undici / fetch | Native, fast, async |
+| **Database** | better-sqlite3 + Drizzle ORM | Sync SQLite with async wrapper, easy PostgreSQL migration |
+| **AI** | Anthropic SDK | Direct Claude integration for classification |
+| **Agents** | Claude SDK spawn pattern | Spawn specialized agents for parallel work |
+| **Config** | dotenv + zod | Type-safe environment configuration |
+
+## Seed Repositories
+
+Initial skill sources for indexing:
+
+1. **VoltAgent/awesome-agent-skills** - Curated list with ~172+ skills
+2. **VoltAgent/awesome-openclaw-skills** - ~700+ community skills
+3. **openclaw/skills** (linked from above) - ~1,874 skills from 777 authors
+
+## Implementation Phases
+
+### Phase 1: Foundation (Days 1-2)
+
+**Goal**: Project setup and core infrastructure
+
+#### Tasks
+
+1. **Project Initialization**
+   - Initialize npm project with TypeScript
+   - Configure tsconfig.json for ES modules
+   - Set up build tooling (tsup)
+   - Configure ESLint + Prettier
+
+2. **Core Types & Models**
+   - Define `Skill` interface (SKILL.md format)
+   - Define `TaxonomyNode` for hierarchy
+   - Define `SkillPlacement` for multi-path mapping
+   - Define database schemas with Drizzle
+
+3. **Database Layer**
+   - Implement `DatabaseAdapter` interface
+   - Create `SQLiteAdapter` implementation
+   - Set up migrations system
+   - Implement basic CRUD operations
+
+4. **Configuration**
+   - Environment variable schema (zod)
+   - GitHub token handling
+   - Anthropic API key handling
+   - Database path configuration
+
+**Deliverables**:
+- Working project structure
+- Database with migrations
+- Type definitions
+- Configuration system
+
+---
+
+### Phase 2: GitHub Scraper (Days 3-4)
+
+**Goal**: Discover and fetch skills from GitHub repositories
+
+#### Tasks
+
+1. **GitHub Client**
+   - Implement authenticated GitHub API client
+   - Handle rate limiting with exponential backoff
+   - Implement pagination for large result sets
+   - Add response caching (file-based)
+
+2. **Repository Discovery**
+   - Parse awesome-list README.md for repo links
+   - Clone/fetch repository contents via API
+   - Traverse directory structure
+   - Identify SKILL.md files
+
+3. **Skill Parser**
+   - Parse YAML frontmatter from SKILL.md
+   - Extract markdown content sections
+   - Parse _meta.json for version info
+   - Handle SKILL.json format (edge case)
+
+4. **Scraper Agent**
+   - Spawn agent to orchestrate scraping
+   - Queue discovered skills for processing
+   - Track scrape progress and state
+   - Handle incremental updates
+
+**Deliverables**:
+- GitHub API integration
+- Skill discovery from awesome-lists
+- SKILL.md/json parsing
+- Raw skills in database
+
+---
+
+### Phase 3: AI Indexer (Days 5-7)
+
+**Goal**: Classify skills into hierarchical taxonomy using Claude
+
+#### Tasks
+
+1. **Classification Agent**
+   - Design classification prompt
+   - Implement Claude API integration
+   - Parse structured JSON responses
+   - Handle classification errors gracefully
+
+2. **Taxonomy Management**
+   - Create/update taxonomy nodes
+   - Handle multi-path placement
+   - Store agent reasoning with decisions
+   - Implement taxonomy tree operations
+
+3. **Relationship Detection**
+   - Identify skill dependencies
+   - Detect similar/alternative skills
+   - Find extension relationships
+   - Build relationship graph
+
+4. **Batch Processing**
+   - Implement async job queue
+   - Process skills in parallel batches
+   - Track indexing progress
+   - Support resume on failure
+
+**Deliverables**:
+- Working AI classification
+- Hierarchical taxonomy
+- Skill relationships
+- Indexed skills in database
+
+---
+
+### Phase 4: CLI Interface (Days 8-9)
+
+**Goal**: User-facing commands for interacting with the index
+
+#### Tasks
+
+1. **Core Commands**
+   ```bash
+   skillindexer scrape <repo-url>     # Scrape specific repo
+   skillindexer scrape --discover     # Scrape from awesome-lists
+   skillindexer index                 # Index unclassified skills
+   skillindexer index --skill <id>    # Re-index specific skill
+   ```
+
+2. **Browse Commands**
+   ```bash
+   skillindexer tree                  # Show full taxonomy
+   skillindexer tree <path>           # Show subtree
+   skillindexer list                  # List all skills
+   skillindexer show <skill-id>       # Show skill details
+   ```
+
+3. **Search Commands**
+   ```bash
+   skillindexer search <query>        # Keyword search
+   skillindexer find --tag <tag>      # Filter by tag
+   skillindexer find --author <name>  # Filter by author
+   ```
+
+4. **Maintenance Commands**
+   ```bash
+   skillindexer stats                 # Show statistics
+   skillindexer export --json         # Export database
+   skillindexer reset                 # Clear database
+   ```
+
+**Deliverables**:
+- Complete CLI tool
+- Interactive tree browser
+- Search functionality
+- Export capabilities
+
+---
+
+### Phase 5: Polish & Integration (Days 10-12)
+
+**Goal**: Production readiness and integration with skill-tree
+
+#### Tasks
+
+1. **Error Handling**
+   - Comprehensive error types
+   - User-friendly error messages
+   - Retry logic for transient failures
+   - Graceful degradation
+
+2. **Logging & Progress**
+   - Structured logging
+   - Progress bars for long operations
+   - Verbose mode for debugging
+   - Quiet mode for scripts
+
+3. **Testing**
+   - Unit tests for parsers
+   - Integration tests for scraper
+   - Mock Claude responses for indexer tests
+   - CLI command tests
+
+4. **Documentation**
+   - README with setup instructions
+   - Configuration reference
+   - CLI command documentation
+   - Contributing guide
+
+5. **Integration**
+   - Export format compatible with skill-tree
+   - Shared types between projects
+   - Import command for skill-tree
+
+**Deliverables**:
+- Production-ready tool
+- Test coverage
+- Documentation
+- skill-tree integration
+
+---
+
+## Directory Structure
+
+```
+/scraper
+├── package.json
+├── tsconfig.json
+├── .env.example
+├── README.md
+│
+├── docs/
+│   ├── PLAN.md              # This file
+│   └── DESIGN.md            # Architecture details
+│
+├── src/
+│   ├── index.ts             # Main entry point
+│   ├── cli.ts               # CLI commands
+│   ├── config.ts            # Configuration
+│   │
+│   ├── types/
+│   │   ├── index.ts         # Re-exports
+│   │   ├── skill.ts         # Skill types
+│   │   ├── taxonomy.ts      # Taxonomy types
+│   │   └── database.ts      # DB schema types
+│   │
+│   ├── database/
+│   │   ├── index.ts         # DB exports
+│   │   ├── adapter.ts       # Adapter interface
+│   │   ├── sqlite.ts        # SQLite implementation
+│   │   ├── schema.ts        # Drizzle schema
+│   │   └── migrations/      # DB migrations
+│   │
+│   ├── scraper/
+│   │   ├── index.ts         # Scraper exports
+│   │   ├── github.ts        # GitHub API client
+│   │   ├── parser.ts        # SKILL.md parser
+│   │   ├── discovery.ts     # Repo discovery
+│   │   └── agent.ts         # Scraper agent
+│   │
+│   ├── indexer/
+│   │   ├── index.ts         # Indexer exports
+│   │   ├── classifier.ts    # AI classification
+│   │   ├── taxonomy.ts      # Taxonomy operations
+│   │   ├── relationships.ts # Relationship detection
+│   │   └── agent.ts         # Indexer agent
+│   │
+│   └── utils/
+│       ├── index.ts         # Utility exports
+│       ├── logger.ts        # Logging
+│       ├── cache.ts         # File cache
+│       └── retry.ts         # Retry logic
+│
+├── data/                    # Runtime data (gitignored)
+│   ├── skills.db            # SQLite database
+│   └── cache/               # GitHub cache
+│
+└── test/
+    ├── fixtures/            # Test fixtures
+    ├── scraper.test.ts
+    ├── parser.test.ts
+    ├── indexer.test.ts
+    └── cli.test.ts
+```
+
+## Success Metrics
+
+### Phase 1 Complete When:
+- [ ] `npm run build` succeeds
+- [ ] Database migrations run
+- [ ] Types compile without errors
+
+### Phase 2 Complete When:
+- [ ] Can scrape skills from a single repo
+- [ ] Skills stored in database with raw content
+- [ ] Incremental scraping works
+
+### Phase 3 Complete When:
+- [ ] Skills classified into taxonomy
+- [ ] Multi-path placement works
+- [ ] Relationships detected
+
+### Phase 4 Complete When:
+- [ ] All CLI commands functional
+- [ ] Tree browsing works
+- [ ] Search returns relevant results
+
+### Phase 5 Complete When:
+- [ ] Tests pass
+- [ ] Documentation complete
+- [ ] Export works with skill-tree
+
+## Risk Mitigation
+
+| Risk | Impact | Mitigation |
+|------|--------|------------|
+| GitHub rate limits | Scraping blocked | Caching, incremental updates, token rotation |
+| Claude API costs | Budget exceeded | Batch processing, cache classifications, use Haiku for initial pass |
+| Large skill repos | Memory issues | Streaming, pagination, incremental processing |
+| Classification drift | Inconsistent taxonomy | Store reasoning, version taxonomy, manual overrides |
+| SKILL.md variations | Parser failures | Graceful fallbacks, log unparseable skills |
+
+## Next Steps
+
+1. Review and approve this plan
+2. Create DESIGN.md with architecture details
+3. Initialize project structure
+4. Begin Phase 1 implementation

package/references/skill-tree/scraper/drizzle.config.ts

@@ -0,0 +1,10 @@
+import { defineConfig } from 'drizzle-kit';
+
+export default defineConfig({
+  schema: './src/database/schema.ts',
+  out: './src/database/migrations',
+  dialect: 'sqlite',
+  dbCredentials: {
+    url: process.env.SKILLINDEXER_DB_PATH || './data/skills.db',
+  },
+});