macro-agent 0.1.2 → 0.1.3

package/references/skill-tree/scraper/docs/PLAN.md

@@ -1,336 +0,0 @@
-# Skills Indexer Implementation Plan
-
-## Overview
-
-This document outlines the implementation plan for the GitHub Skills Indexer - an AI-powered system that discovers, indexes, and organizes Agent Skills from GitHub repositories into a hierarchical taxonomy.
-
-## Technology Stack
-
-| Component | Choice | Rationale |
-|-----------|--------|-----------|
-| **Runtime** | Node.js 20+ | Async-first, excellent TypeScript support |
-| **Language** | TypeScript 5.x | Type safety, IDE support, matches parent project |
-| **CLI** | Commander.js | Lightweight, well-documented |
-| **HTTP** | undici / fetch | Native, fast, async |
-| **Database** | better-sqlite3 + Drizzle ORM | Sync SQLite with async wrapper, easy PostgreSQL migration |
-| **AI** | Anthropic SDK | Direct Claude integration for classification |
-| **Agents** | Claude SDK spawn pattern | Spawn specialized agents for parallel work |
-| **Config** | dotenv + zod | Type-safe environment configuration |
-
-## Seed Repositories
-
-Initial skill sources for indexing:
-
-1. **VoltAgent/awesome-agent-skills** - Curated list with ~172+ skills
-2. **VoltAgent/awesome-openclaw-skills** - ~700+ community skills
-3. **openclaw/skills** (linked from above) - ~1,874 skills from 777 authors
-
-## Implementation Phases
-
-### Phase 1: Foundation (Days 1-2)
-
-**Goal**: Project setup and core infrastructure
-
-#### Tasks
-
-1. **Project Initialization**
-   - Initialize npm project with TypeScript
-   - Configure tsconfig.json for ES modules
-   - Set up build tooling (tsup)
-   - Configure ESLint + Prettier
-
-2. **Core Types & Models**
-   - Define `Skill` interface (SKILL.md format)
-   - Define `TaxonomyNode` for hierarchy
-   - Define `SkillPlacement` for multi-path mapping
-   - Define database schemas with Drizzle
-
-3. **Database Layer**
-   - Implement `DatabaseAdapter` interface
-   - Create `SQLiteAdapter` implementation
-   - Set up migrations system
-   - Implement basic CRUD operations
-
-4. **Configuration**
-   - Environment variable schema (zod)
-   - GitHub token handling
-   - Anthropic API key handling
-   - Database path configuration
-
-**Deliverables**:
-- Working project structure
-- Database with migrations
-- Type definitions
-- Configuration system
-
----
-
-### Phase 2: GitHub Scraper (Days 3-4)
-
-**Goal**: Discover and fetch skills from GitHub repositories
-
-#### Tasks
-
-1. **GitHub Client**
-   - Implement authenticated GitHub API client
-   - Handle rate limiting with exponential backoff
-   - Implement pagination for large result sets
-   - Add response caching (file-based)
-
-2. **Repository Discovery**
-   - Parse awesome-list README.md for repo links
-   - Clone/fetch repository contents via API
-   - Traverse directory structure
-   - Identify SKILL.md files
-
-3. **Skill Parser**
-   - Parse YAML frontmatter from SKILL.md
-   - Extract markdown content sections
-   - Parse _meta.json for version info
-   - Handle SKILL.json format (edge case)
-
-4. **Scraper Agent**
-   - Spawn agent to orchestrate scraping
-   - Queue discovered skills for processing
-   - Track scrape progress and state
-   - Handle incremental updates
-
-**Deliverables**:
-- GitHub API integration
-- Skill discovery from awesome-lists
-- SKILL.md/json parsing
-- Raw skills in database
-
----
-
-### Phase 3: AI Indexer (Days 5-7)
-
-**Goal**: Classify skills into hierarchical taxonomy using Claude
-
-#### Tasks
-
-1. **Classification Agent**
-   - Design classification prompt
-   - Implement Claude API integration
-   - Parse structured JSON responses
-   - Handle classification errors gracefully
-
-2. **Taxonomy Management**
-   - Create/update taxonomy nodes
-   - Handle multi-path placement
-   - Store agent reasoning with decisions
-   - Implement taxonomy tree operations
-
-3. **Relationship Detection**
-   - Identify skill dependencies
-   - Detect similar/alternative skills
-   - Find extension relationships
-   - Build relationship graph
-
-4. **Batch Processing**
-   - Implement async job queue
-   - Process skills in parallel batches
-   - Track indexing progress
-   - Support resume on failure
-
-**Deliverables**:
-- Working AI classification
-- Hierarchical taxonomy
-- Skill relationships
-- Indexed skills in database
-
----
-
-### Phase 4: CLI Interface (Days 8-9)
-
-**Goal**: User-facing commands for interacting with the index
-
-#### Tasks
-
-1. **Core Commands**
-   ```bash
-   skillindexer scrape <repo-url>     # Scrape specific repo
-   skillindexer scrape --discover     # Scrape from awesome-lists
-   skillindexer index                 # Index unclassified skills
-   skillindexer index --skill <id>    # Re-index specific skill
-   ```
-
-2. **Browse Commands**
-   ```bash
-   skillindexer tree                  # Show full taxonomy
-   skillindexer tree <path>           # Show subtree
-   skillindexer list                  # List all skills
-   skillindexer show <skill-id>       # Show skill details
-   ```
-
-3. **Search Commands**
-   ```bash
-   skillindexer search <query>        # Keyword search
-   skillindexer find --tag <tag>      # Filter by tag
-   skillindexer find --author <name>  # Filter by author
-   ```
-
-4. **Maintenance Commands**
-   ```bash
-   skillindexer stats                 # Show statistics
-   skillindexer export --json         # Export database
-   skillindexer reset                 # Clear database
-   ```
-
-**Deliverables**:
-- Complete CLI tool
-- Interactive tree browser
-- Search functionality
-- Export capabilities
-
----
-
-### Phase 5: Polish & Integration (Days 10-12)
-
-**Goal**: Production readiness and integration with skill-tree
-
-#### Tasks
-
-1. **Error Handling**
-   - Comprehensive error types
-   - User-friendly error messages
-   - Retry logic for transient failures
-   - Graceful degradation
-
-2. **Logging & Progress**
-   - Structured logging
-   - Progress bars for long operations
-   - Verbose mode for debugging
-   - Quiet mode for scripts
-
-3. **Testing**
-   - Unit tests for parsers
-   - Integration tests for scraper
-   - Mock Claude responses for indexer tests
-   - CLI command tests
-
-4. **Documentation**
-   - README with setup instructions
-   - Configuration reference
-   - CLI command documentation
-   - Contributing guide
-
-5. **Integration**
-   - Export format compatible with skill-tree
-   - Shared types between projects
-   - Import command for skill-tree
-
-**Deliverables**:
-- Production-ready tool
-- Test coverage
-- Documentation
-- skill-tree integration
-
----
-
-## Directory Structure
-
-```
-/scraper
-├── package.json
-├── tsconfig.json
-├── .env.example
-├── README.md
-│
-├── docs/
-│   ├── PLAN.md              # This file
-│   └── DESIGN.md            # Architecture details
-│
-├── src/
-│   ├── index.ts             # Main entry point
-│   ├── cli.ts               # CLI commands
-│   ├── config.ts            # Configuration
-│   │
-│   ├── types/
-│   │   ├── index.ts         # Re-exports
-│   │   ├── skill.ts         # Skill types
-│   │   ├── taxonomy.ts      # Taxonomy types
-│   │   └── database.ts      # DB schema types
-│   │
-│   ├── database/
-│   │   ├── index.ts         # DB exports
-│   │   ├── adapter.ts       # Adapter interface
-│   │   ├── sqlite.ts        # SQLite implementation
-│   │   ├── schema.ts        # Drizzle schema
-│   │   └── migrations/      # DB migrations
-│   │
-│   ├── scraper/
-│   │   ├── index.ts         # Scraper exports
-│   │   ├── github.ts        # GitHub API client
-│   │   ├── parser.ts        # SKILL.md parser
-│   │   ├── discovery.ts     # Repo discovery
-│   │   └── agent.ts         # Scraper agent
-│   │
-│   ├── indexer/
-│   │   ├── index.ts         # Indexer exports
-│   │   ├── classifier.ts    # AI classification
-│   │   ├── taxonomy.ts      # Taxonomy operations
-│   │   ├── relationships.ts # Relationship detection
-│   │   └── agent.ts         # Indexer agent
-│   │
-│   └── utils/
-│       ├── index.ts         # Utility exports
-│       ├── logger.ts        # Logging
-│       ├── cache.ts         # File cache
-│       └── retry.ts         # Retry logic
-│
-├── data/                    # Runtime data (gitignored)
-│   ├── skills.db            # SQLite database
-│   └── cache/               # GitHub cache
-│
-└── test/
-    ├── fixtures/            # Test fixtures
-    ├── scraper.test.ts
-    ├── parser.test.ts
-    ├── indexer.test.ts
-    └── cli.test.ts
-```
-
-## Success Metrics
-
-### Phase 1 Complete When:
-- [ ] `npm run build` succeeds
-- [ ] Database migrations run
-- [ ] Types compile without errors
-
-### Phase 2 Complete When:
-- [ ] Can scrape skills from a single repo
-- [ ] Skills stored in database with raw content
-- [ ] Incremental scraping works
-
-### Phase 3 Complete When:
-- [ ] Skills classified into taxonomy
-- [ ] Multi-path placement works
-- [ ] Relationships detected
-
-### Phase 4 Complete When:
-- [ ] All CLI commands functional
-- [ ] Tree browsing works
-- [ ] Search returns relevant results
-
-### Phase 5 Complete When:
-- [ ] Tests pass
-- [ ] Documentation complete
-- [ ] Export works with skill-tree
-
-## Risk Mitigation
-
-| Risk | Impact | Mitigation |
-|------|--------|------------|
-| GitHub rate limits | Scraping blocked | Caching, incremental updates, token rotation |
-| Claude API costs | Budget exceeded | Batch processing, cache classifications, use Haiku for initial pass |
-| Large skill repos | Memory issues | Streaming, pagination, incremental processing |
-| Classification drift | Inconsistent taxonomy | Store reasoning, version taxonomy, manual overrides |
-| SKILL.md variations | Parser failures | Graceful fallbacks, log unparseable skills |
-
-## Next Steps
-
-1. Review and approve this plan
-2. Create DESIGN.md with architecture details
-3. Initialize project structure
-4. Begin Phase 1 implementation

package/references/skill-tree/scraper/drizzle.config.ts

@@ -1,10 +0,0 @@
-import { defineConfig } from 'drizzle-kit';
-
-export default defineConfig({
-  schema: './src/database/schema.ts',
-  out: './src/database/migrations',
-  dialect: 'sqlite',
-  dbCredentials: {
-    url: process.env.SKILLINDEXER_DB_PATH || './data/skills.db',
-  },
-});