macro-agent 0.1.1 → 0.1.2

package/references/skill-tree/docs/INTEGRATION_PLAN.md

@@ -0,0 +1,467 @@
+# Skill Indexer Integration Plan
+
+## Overview
+
+This document outlines the plan to integrate the **skill indexer** (`scraper/`) with the main **skill-tree** CLI and SkillBank. The goal is to create a cohesive system where externally discovered skills from GitHub can seamlessly flow into the skill-tree data structure.
+
+## Current Architecture
+
+```
+┌─────────────────────────────────────┐     ┌─────────────────────────────────────┐
+│           SKILL INDEXER             │     │            SKILL-TREE               │
+│           (scraper/)                │     │              (src/)                 │
+├─────────────────────────────────────┤     ├─────────────────────────────────────┤
+│  • GitHub skill discovery           │     │  • Agent trajectory extraction      │
+│  • AI-based classification          │     │  • Skill versioning & lineage       │
+│  • Taxonomy management              │     │  • Semantic matching                │
+│  • Relationship detection           │     │  • Storage adapters                 │
+│  • SQLite (scraper/data/)           │     │  • SQLite/Filesystem/Memory         │
+├─────────────────────────────────────┤     ├─────────────────────────────────────┤
+│  CLI: skillindexer                  │     │  CLI: skill-tree                    │
+│  • scrape, index, tree, etc.        │     │  • list, show, import, export, etc. │
+└─────────────────────────────────────┘     └─────────────────────────────────────┘
+                    │                                         │
+                    └──────────── GAP ────────────────────────┘
+                         (no direct connection)
+```
+
+## Target Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────────────┐
+│                            UNIFIED SKILL-TREE                                   │
+├─────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                 │
+│  ┌─────────────────────┐    ┌─────────────────────┐    ┌──────────────────────┐ │
+│  │   Skill Sources     │    │    Core Engine      │    │     Storage          │ │
+│  ├─────────────────────┤    ├─────────────────────┤    ├──────────────────────┤ │
+│  │ • GitHub Indexer    │───▶│ • SkillBank         │◀──▶│ • Unified SQLite     │ │
+│  │ • Trajectory Extract│───▶│ • Versioning        │    │ • Filesystem backup  │ │
+│  │ • Manual Import     │───▶│ • Lineage Tracking  │    │ • Taxonomy tables    │ │
+│  │ • Composed Skills   │───▶│ • Semantic Match    │    │ • Relationship tables│ │
+│  └─────────────────────┘    └─────────────────────┘    └──────────────────────┘ │
+│                                                                                 │
+│  ┌─────────────────────────────────────────────────────────────────────────────┐│
+│  │                           Unified CLI: skill-tree                           ││
+│  ├─────────────────────────────────────────────────────────────────────────────┤│
+│  │ Core:    list | show | search | stats | versions | diff | export | import   ││
+│  │ Index:   scrape | index | taxonomy | relationships                          ││
+│  │ Evolve:  rollback | fork | deprecate | activate | delete                    ││
+│  └─────────────────────────────────────────────────────────────────────────────┘│
+└─────────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Phase 1: Import Pipeline (Quick Win)
+
+**Goal**: Enable `skill-tree import` to accept indexer output directly.
+
+### 1.1 Add `--from-indexer` Flag to Import Command
+
+Modify `src/cli/commands/import.ts` to detect and convert indexer format:
+
+```typescript
+// New option
+.option('--from-indexer', 'Import from skill-indexer export format')
+
+// Detection logic
+if (options.fromIndexer || detectIndexerFormat(content)) {
+  skills = convertFromIndexer(content);
+}
+```
+
+### 1.2 Integrate Converter
+
+Move or re-export the converter from `scraper/src/compat/converter.ts`:
+
+```
+src/import/
+├── index.ts
+├── converter.ts       # Adapted from scraper/src/compat/converter.ts
+└── detect.ts          # Format auto-detection
+```
+
+### 1.3 Workflow
+
+```bash
+# Current (two-step)
+skillindexer export-skilltree -o indexed-skills.json
+skill-tree import indexed-skills.json
+
+# After Phase 1 (still two-step, but seamless)
+skillindexer export -o indexed-skills.json
+skill-tree import --from-indexer indexed-skills.json
+```
+
+**Effort**: ~2-3 hours
+**Files**: `src/cli/commands/import.ts`, new `src/import/` directory
+
+---
+
+## Phase 2: CLI Unification
+
+**Goal**: Single CLI with all commands.
+
+### 2.1 New Command Group: `skill-tree index`
+
+Add indexer commands under a subcommand group:
+
+```
+skill-tree index scrape [url]         # Discover skills from GitHub
+skill-tree index classify             # AI-powered classification
+skill-tree index taxonomy [path]      # Browse taxonomy tree
+skill-tree index relationships        # Detect skill relationships
+skill-tree index stats                # Indexer statistics
+```
+
+### 2.2 Implementation Structure
+
+```
+src/cli/
+├── index.ts                    # Main entry (existing)
+├── commands/
+│   ├── list.ts                 # Existing commands
+│   ├── show.ts
+│   ├── import.ts
+│   ├── ...
+│   └── index/                  # NEW: indexer subcommands
+│       ├── index.ts            # Parent command group
+│       ├── scrape.ts
+│       ├── classify.ts
+│       ├── taxonomy.ts
+│       └── relationships.ts
+```
+
+### 2.3 Shared Services
+
+Create a service layer accessible by both command sets:
+
+```typescript
+// src/services/indexer.ts
+export class IndexerService {
+  constructor(
+    private skillBank: SkillBank,
+    private config: IndexerConfig
+  ) {}
+
+  async scrape(sources: SkillSource[]): Promise<ScrapeResult> { ... }
+  async classify(): Promise<ClassifyResult> { ... }
+  async detectRelationships(): Promise<RelationshipResult> { ... }
+}
+```
+
+### 2.4 Configuration Merge
+
+Unify configuration under a single config file:
+
+```yaml
+# ~/.skill-tree/config.yaml
+storage:
+  type: sqlite
+  path: ~/.skill-tree/skills.db
+
+indexer:
+  github_token: ${GITHUB_TOKEN}
+  anthropic_key: ${ANTHROPIC_API_KEY}
+  sources:
+    - https://github.com/VoltAgent/awesome-agent-skills
+    - https://github.com/VoltAgent/awesome-openclaw-skills
+  batch_size: 10
+  min_confidence: 0.7
+
+matching:
+  embedding_model: text-embedding-3-small
+```
+
+**Effort**: ~6-8 hours
+**Files**: New `src/cli/commands/index/`, `src/services/indexer.ts`
+
+---
+
+## Phase 3: Unified Storage
+
+**Goal**: Single database with shared schema for both indexed and extracted skills.
+
+### 3.1 Schema Extension
+
+Extend the existing SQLite schema to include indexer tables:
+
+```sql
+-- Existing skill-tree tables
+CREATE TABLE skills (...);
+CREATE TABLE skill_versions (...);
+
+-- New indexer tables (merged)
+CREATE TABLE taxonomy_nodes (
+  id TEXT PRIMARY KEY,
+  name TEXT NOT NULL,
+  description TEXT,
+  parent_id TEXT REFERENCES taxonomy_nodes(id),
+  path TEXT NOT NULL,           -- Materialized path: "Development/Python/Testing"
+  skill_count INTEGER DEFAULT 0,
+  created_at DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+
+CREATE TABLE skill_taxonomy_placements (
+  skill_id TEXT REFERENCES skills(id),
+  node_id TEXT REFERENCES taxonomy_nodes(id),
+  is_primary BOOLEAN DEFAULT FALSE,
+  confidence REAL,
+  reasoning TEXT,
+  PRIMARY KEY (skill_id, node_id)
+);
+
+CREATE TABLE skill_relationships (
+  source_skill_id TEXT REFERENCES skills(id),
+  target_skill_id TEXT REFERENCES skills(id),
+  type TEXT NOT NULL,           -- 'depends_on', 'extends', 'alternative', 'related'
+  confidence REAL,
+  reasoning TEXT,
+  created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+  PRIMARY KEY (source_skill_id, target_skill_id, type)
+);
+
+-- Track external sources
+CREATE TABLE skill_sources (
+  id TEXT PRIMARY KEY,
+  type TEXT NOT NULL,           -- 'github-repo', 'awesome-list', 'trajectory', 'manual'
+  url TEXT,
+  last_scraped DATETIME,
+  etag TEXT,
+  skill_count INTEGER DEFAULT 0
+);
+```
+
+### 3.2 Extended Skill Type
+
+Add optional indexer fields to the core Skill type:
+
+```typescript
+// src/types.ts
+export interface Skill {
+  // ... existing fields ...
+
+  // Indexer extensions (optional)
+  taxonomy?: {
+    primaryPath: string[];
+    secondaryPaths?: string[][];
+    confidence?: number;
+  };
+  relationships?: SkillRelationship[];
+  externalSource?: {
+    url: string;
+    repo: string;
+    scrapedAt: Date;
+    etag?: string;
+  };
+}
+
+export interface SkillRelationship {
+  targetSkillId: string;
+  type: 'depends_on' | 'extends' | 'alternative' | 'related';
+  confidence: number;
+  reasoning?: string;
+}
+```
+
+### 3.3 Storage Adapter Extension
+
+Extend `SqliteStorageAdapter` to handle taxonomy and relationships:
+
+```typescript
+// src/storage/sqlite.ts
+export class SqliteStorageAdapter implements StorageAdapter {
+  // ... existing methods ...
+
+  // Taxonomy methods
+  async getTaxonomyTree(rootPath?: string[]): Promise<TaxonomyTree>;
+  async placeInTaxonomy(skillId: string, placement: TaxonomyPlacement): Promise<void>;
+
+  // Relationship methods
+  async addRelationship(rel: SkillRelationship): Promise<void>;
+  async getRelationships(skillId: string): Promise<SkillRelationship[]>;
+  async getSkillsInNode(nodeId: string): Promise<Skill[]>;
+}
+```
+
+**Effort**: ~8-10 hours
+**Files**: `src/storage/sqlite.ts`, `src/types.ts`, migration scripts
+
+---
+
+## Phase 4: Direct Integration
+
+**Goal**: Scraper writes directly to SkillBank, bypassing export/import.
+
+### 4.1 Indexer as SkillBank Client
+
+Modify indexer to use SkillBank as its storage backend:
+
+```typescript
+// src/services/indexer.ts
+export class IndexerService {
+  constructor(
+    private skillBank: SkillBank,
+    private classifier: SkillClassifier,
+    private config: IndexerConfig
+  ) {}
+
+  async scrapeAndIndex(sources: SkillSource[]): Promise<IndexResult> {
+    const rawSkills = await this.scraper.scrape(sources);
+
+    for (const raw of rawSkills) {
+      // Convert to skill-tree format
+      const skill = this.converter.convert(raw);
+
+      // Classify
+      const classification = await this.classifier.classify(skill);
+      skill.taxonomy = classification;
+
+      // Store directly in SkillBank
+      await this.skillBank.addSkill(skill, {
+        source: 'indexed',
+        sourceUrl: raw.sourceUrl,
+      });
+    }
+
+    return { indexed: rawSkills.length };
+  }
+}
+```
+
+### 4.2 Event-Driven Updates
+
+Use SkillBank events to trigger relationship detection:
+
+```typescript
+skillBank.on('skill:created', async (event) => {
+  if (event.skill.source?.type === 'imported') {
+    await relationshipDetector.detectForSkill(event.skill);
+  }
+});
+```
+
+### 4.3 Workflow
+
+```bash
+# Direct flow (no export/import step)
+skill-tree index scrape https://github.com/VoltAgent/awesome-agent-skills
+# Skills are now in the main skill-tree database with taxonomy and relationships
+```
+
+**Effort**: ~6-8 hours
+**Files**: `src/services/indexer.ts`, `src/skill-bank.ts` event handlers
+
+---
+
+## Phase 5: Bidirectional Sync
+
+**Goal**: Keep indexed skills updated and track changes.
+
+### 5.1 Change Detection
+
+```typescript
+interface SkillSyncState {
+  skillId: string;
+  sourceUrl: string;
+  sourceEtag: string;
+  lastSynced: Date;
+  localVersion: string;
+  remoteVersion: string;
+  status: 'synced' | 'local-modified' | 'remote-modified' | 'conflict';
+}
+```
+
+### 5.2 Sync Commands
+
+```bash
+skill-tree index sync              # Check for updates
+skill-tree index sync --pull       # Pull remote changes
+skill-tree index sync --push       # Push local changes (to forks?)
+skill-tree index sync --status     # Show sync status
+```
+
+### 5.3 Conflict Resolution
+
+When both local and remote have changed:
+
+1. Create a new version with merged changes
+2. Keep lineage tracking intact
+3. Mark conflicts for manual review
+
+**Effort**: ~10-12 hours
+**Files**: `src/services/sync.ts`, `src/cli/commands/index/sync.ts`
+
+---
+
+## Implementation Order
+
+| Phase | Description | Priority | Effort | Dependencies |
+|-------|-------------|----------|--------|--------------|
+| 1 | Import Pipeline | High | 2-3 hrs | None |
+| 2 | CLI Unification | High | 6-8 hrs | Phase 1 |
+| 3 | Unified Storage | Medium | 8-10 hrs | Phase 2 |
+| 4 | Direct Integration | Medium | 6-8 hrs | Phase 3 |
+| 5 | Bidirectional Sync | Low | 10-12 hrs | Phase 4 |
+
+**Recommended starting point**: Phase 1 (immediate value) → Phase 2 (user experience) → Phase 3 (foundation for advanced features)
+
+---
+
+## Migration Strategy
+
+### For Existing Indexer Users
+
+```bash
+# Export existing indexed skills
+skillindexer export -o backup.json
+
+# After integration, import into unified database
+skill-tree import --from-indexer backup.json
+
+# Verify
+skill-tree list --source indexed
+skill-tree index taxonomy
+```
+
+### Database Migration
+
+Create migration script for existing SQLite databases:
+
+```typescript
+// scripts/migrate-indexer-db.ts
+async function migrateIndexerDatabase(
+  indexerDbPath: string,
+  skillTreeDbPath: string
+): Promise<MigrationResult> {
+  // 1. Read all skills from indexer DB
+  // 2. Convert using existing converter
+  // 3. Insert into skill-tree DB with taxonomy preservation
+  // 4. Migrate taxonomy_nodes table
+  // 5. Migrate relationships table
+}
+```
+
+---
+
+## Open Questions
+
+1. **Taxonomy ownership**: Should indexed skills always preserve their original taxonomy, or should they be re-classified by skill-tree's own taxonomy?
+
+2. **Deduplication**: How to handle the same skill discovered from multiple sources?
+
+3. **Version conflicts**: When an indexed skill is updated externally and locally modified, which takes precedence?
+
+4. **Namespace separation**: Should indexed skills have a separate namespace (e.g., `indexed:skill-name`) or be fully merged?
+
+---
+
+## Success Criteria
+
+- [ ] Single CLI (`skill-tree`) for all operations
+- [ ] Single database with skills from all sources
+- [ ] Taxonomy browsing works for all skills
+- [ ] Relationships detected across indexed and extracted skills
+- [ ] Clear provenance tracking (source: extracted vs indexed)
+- [ ] Seamless workflow from discovery to usage

package/references/skill-tree/docs/TODOS.md

@@ -0,0 +1,91 @@
+# skill-tree TODOs
+
+Tracking remaining features and improvements for the skill-tree library.
+
+## Completed ✅
+
+- [x] **Core Types** - Skill, Trajectory, Extraction, Storage interfaces
+- [x] **Session Adapters** - Claude Code JSONL parser, extensible registry
+- [x] **Manual Extraction** - User-guided extraction with quality gates
+- [x] **Automatic Extraction** - LLM-powered skill identification
+- [x] **Quality Gates** - Reusability, non-trivial, specific triggers, verified
+- [x] **Memory Storage** - In-memory adapter for testing
+- [x] **Filesystem Storage** - OpenSkills-compatible (YAML frontmatter + Markdown)
+- [x] **Semantic Versioning** - Parse, compare, bump, range satisfaction
+- [x] **Lineage Tracking** - Version history, fork, merge, rollback
+- [x] **SkillBank Orchestrator** - Unified API, events, CRUD, import/export
+- [x] **Test Suite** - 352 tests across all modules
+- [x] **Semantic Matching** - Embedding providers, similarity search, context matching, caching
+- [x] **Batch Processing** - Multi-session extraction, deduplication, parallel processing, progress reporting
+- [x] **Skill Composer** - Compound skills, dependency graph, composition suggestions, conflict detection
+- [x] **Additional Session Adapters** - OpenAI, Anthropic, Generic JSON with configurable field mapping
+- [x] **Skill Merging Improvements** - SkillMerger with conflict detection, preview, multiple strategies, merge suggestions
+- [x] **Hooks & Activation System** - HookRegistry, ActivationManager, built-in hooks, auto-trigger extraction
+- [x] **CLI Tool** - Management CLI with list, show, search, stats, versions, diff, rollback, fork, deprecate, activate, delete, export, import
+- [x] **SQLite Storage** - SQLite adapter with FTS5 full-text search, WAL mode, version history
+- [x] **Metrics & Analytics** - MetricsTracker with usage tracking, feedback, rankings, trends, reports, deprecation suggestions
+- [x] **AGENTS.md Integration** - Generator, Parser, Sync for Claude Code's AGENTS.md format (XML, Markdown, JSON)
+- [x] **Skill Validation** - SkillValidator with test cases, version compatibility checking, deprecation suggestions, scheduling
+- [x] **Test Suite** - 459 tests across all modules
+
+---
+
+## High Priority
+
+(All high priority items completed!)
+
+---
+
+## Medium Priority
+
+(All medium priority items completed!)
+
+---
+
+## Low Priority
+
+### Database Storage Adapters
+- [x] SQLite adapter for local persistence
+- [ ] PostgreSQL adapter for team/production use
+- [ ] Migration support between storage backends
+- [x] Full-text search integration (FTS5)
+
+**Why**: Filesystem works well for individual use; DB needed for larger deployments.
+
+### Skill Validation
+- [x] Runtime verification that skills still work
+- [x] Periodic re-testing of skill solutions
+- [x] Deprecation suggestions for failing skills
+- [x] Version compatibility checking
+
+**Why**: Skills can become stale as codebases/tools evolve.
+
+### AGENTS.md Integration
+- [x] Generate AGENTS.md from skill bank (like OpenSkills)
+- [x] Sync skills to/from AGENTS.md
+- [x] XML format for skill descriptions
+- [x] Selective skill exposure
+
+**Why**: Integration with Claude Code's native skill loading.
+
+---
+
+## Ideas / Future Exploration
+
+- [ ] **Tiered Skill Loading** - Skills can have links to other skills for progressive discovery/loading for large skill libraries
+- [ ] **Skill Templates** - Predefined templates for common skill types
+- [ ] **Skill Marketplace** - Share skills between users/teams
+- [ ] **A/B Testing** - Compare skill versions in practice
+- [ ] **Auto-improvement** - Use feedback to refine skills automatically
+- [ ] **Cross-language Support** - Skills that work across programming languages
+- [ ] **Skill Dependencies** - Declare when one skill requires another
+- [ ] **Context Injection** - Automatically inject relevant skills into agent context
+- [ ] **Skill Clustering** - Group similar skills automatically
+
+---
+
+## References
+
+- [arXiv:2512.17102](https://arxiv.org/abs/2512.17102) - Skill Libraries via RL
+- [Claudeception](https://github.com/blader/Claudeception) - Quality gates, extraction hooks
+- [OpenSkills](https://github.com/numman-ali/openskills) - Skill format, discovery mechanism