macro-agent 0.1.2 → 0.1.3

package/references/skill-tree/docs/INTEGRATION_PLAN.md

@@ -1,467 +0,0 @@
-# Skill Indexer Integration Plan
-
-## Overview
-
-This document outlines the plan to integrate the **skill indexer** (`scraper/`) with the main **skill-tree** CLI and SkillBank. The goal is to create a cohesive system where externally discovered skills from GitHub can seamlessly flow into the skill-tree data structure.
-
-## Current Architecture
-
-```
-┌─────────────────────────────────────┐     ┌─────────────────────────────────────┐
-│           SKILL INDEXER             │     │            SKILL-TREE               │
-│           (scraper/)                │     │              (src/)                 │
-├─────────────────────────────────────┤     ├─────────────────────────────────────┤
-│  • GitHub skill discovery           │     │  • Agent trajectory extraction      │
-│  • AI-based classification          │     │  • Skill versioning & lineage       │
-│  • Taxonomy management              │     │  • Semantic matching                │
-│  • Relationship detection           │     │  • Storage adapters                 │
-│  • SQLite (scraper/data/)           │     │  • SQLite/Filesystem/Memory         │
-├─────────────────────────────────────┤     ├─────────────────────────────────────┤
-│  CLI: skillindexer                  │     │  CLI: skill-tree                    │
-│  • scrape, index, tree, etc.        │     │  • list, show, import, export, etc. │
-└─────────────────────────────────────┘     └─────────────────────────────────────┘
-                    │                                         │
-                    └──────────── GAP ────────────────────────┘
-                         (no direct connection)
-```
-
-## Target Architecture
-
-```
-┌─────────────────────────────────────────────────────────────────────────────────┐
-│                            UNIFIED SKILL-TREE                                   │
-├─────────────────────────────────────────────────────────────────────────────────┤
-│                                                                                 │
-│  ┌─────────────────────┐    ┌─────────────────────┐    ┌──────────────────────┐ │
-│  │   Skill Sources     │    │    Core Engine      │    │     Storage          │ │
-│  ├─────────────────────┤    ├─────────────────────┤    ├──────────────────────┤ │
-│  │ • GitHub Indexer    │───▶│ • SkillBank         │◀──▶│ • Unified SQLite     │ │
-│  │ • Trajectory Extract│───▶│ • Versioning        │    │ • Filesystem backup  │ │
-│  │ • Manual Import     │───▶│ • Lineage Tracking  │    │ • Taxonomy tables    │ │
-│  │ • Composed Skills   │───▶│ • Semantic Match    │    │ • Relationship tables│ │
-│  └─────────────────────┘    └─────────────────────┘    └──────────────────────┘ │
-│                                                                                 │
-│  ┌─────────────────────────────────────────────────────────────────────────────┐│
-│  │                           Unified CLI: skill-tree                           ││
-│  ├─────────────────────────────────────────────────────────────────────────────┤│
-│  │ Core:    list | show | search | stats | versions | diff | export | import   ││
-│  │ Index:   scrape | index | taxonomy | relationships                          ││
-│  │ Evolve:  rollback | fork | deprecate | activate | delete                    ││
-│  └─────────────────────────────────────────────────────────────────────────────┘│
-└─────────────────────────────────────────────────────────────────────────────────┘
-```
-
----
-
-## Phase 1: Import Pipeline (Quick Win)
-
-**Goal**: Enable `skill-tree import` to accept indexer output directly.
-
-### 1.1 Add `--from-indexer` Flag to Import Command
-
-Modify `src/cli/commands/import.ts` to detect and convert indexer format:
-
-```typescript
-// New option
-.option('--from-indexer', 'Import from skill-indexer export format')
-
-// Detection logic
-if (options.fromIndexer || detectIndexerFormat(content)) {
-  skills = convertFromIndexer(content);
-}
-```
-
-### 1.2 Integrate Converter
-
-Move or re-export the converter from `scraper/src/compat/converter.ts`:
-
-```
-src/import/
-├── index.ts
-├── converter.ts       # Adapted from scraper/src/compat/converter.ts
-└── detect.ts          # Format auto-detection
-```
-
-### 1.3 Workflow
-
-```bash
-# Current (two-step)
-skillindexer export-skilltree -o indexed-skills.json
-skill-tree import indexed-skills.json
-
-# After Phase 1 (still two-step, but seamless)
-skillindexer export -o indexed-skills.json
-skill-tree import --from-indexer indexed-skills.json
-```
-
-**Effort**: ~2-3 hours
-**Files**: `src/cli/commands/import.ts`, new `src/import/` directory
-
----
-
-## Phase 2: CLI Unification
-
-**Goal**: Single CLI with all commands.
-
-### 2.1 New Command Group: `skill-tree index`
-
-Add indexer commands under a subcommand group:
-
-```
-skill-tree index scrape [url]         # Discover skills from GitHub
-skill-tree index classify             # AI-powered classification
-skill-tree index taxonomy [path]      # Browse taxonomy tree
-skill-tree index relationships        # Detect skill relationships
-skill-tree index stats                # Indexer statistics
-```
-
-### 2.2 Implementation Structure
-
-```
-src/cli/
-├── index.ts                    # Main entry (existing)
-├── commands/
-│   ├── list.ts                 # Existing commands
-│   ├── show.ts
-│   ├── import.ts
-│   ├── ...
-│   └── index/                  # NEW: indexer subcommands
-│       ├── index.ts            # Parent command group
-│       ├── scrape.ts
-│       ├── classify.ts
-│       ├── taxonomy.ts
-│       └── relationships.ts
-```
-
-### 2.3 Shared Services
-
-Create a service layer accessible by both command sets:
-
-```typescript
-// src/services/indexer.ts
-export class IndexerService {
-  constructor(
-    private skillBank: SkillBank,
-    private config: IndexerConfig
-  ) {}
-
-  async scrape(sources: SkillSource[]): Promise<ScrapeResult> { ... }
-  async classify(): Promise<ClassifyResult> { ... }
-  async detectRelationships(): Promise<RelationshipResult> { ... }
-}
-```
-
-### 2.4 Configuration Merge
-
-Unify configuration under a single config file:
-
-```yaml
-# ~/.skill-tree/config.yaml
-storage:
-  type: sqlite
-  path: ~/.skill-tree/skills.db
-
-indexer:
-  github_token: ${GITHUB_TOKEN}
-  anthropic_key: ${ANTHROPIC_API_KEY}
-  sources:
-    - https://github.com/VoltAgent/awesome-agent-skills
-    - https://github.com/VoltAgent/awesome-openclaw-skills
-  batch_size: 10
-  min_confidence: 0.7
-
-matching:
-  embedding_model: text-embedding-3-small
-```
-
-**Effort**: ~6-8 hours
-**Files**: New `src/cli/commands/index/`, `src/services/indexer.ts`
-
----
-
-## Phase 3: Unified Storage
-
-**Goal**: Single database with shared schema for both indexed and extracted skills.
-
-### 3.1 Schema Extension
-
-Extend the existing SQLite schema to include indexer tables:
-
-```sql
--- Existing skill-tree tables
-CREATE TABLE skills (...);
-CREATE TABLE skill_versions (...);
-
--- New indexer tables (merged)
-CREATE TABLE taxonomy_nodes (
-  id TEXT PRIMARY KEY,
-  name TEXT NOT NULL,
-  description TEXT,
-  parent_id TEXT REFERENCES taxonomy_nodes(id),
-  path TEXT NOT NULL,           -- Materialized path: "Development/Python/Testing"
-  skill_count INTEGER DEFAULT 0,
-  created_at DATETIME DEFAULT CURRENT_TIMESTAMP
-);
-
-CREATE TABLE skill_taxonomy_placements (
-  skill_id TEXT REFERENCES skills(id),
-  node_id TEXT REFERENCES taxonomy_nodes(id),
-  is_primary BOOLEAN DEFAULT FALSE,
-  confidence REAL,
-  reasoning TEXT,
-  PRIMARY KEY (skill_id, node_id)
-);
-
-CREATE TABLE skill_relationships (
-  source_skill_id TEXT REFERENCES skills(id),
-  target_skill_id TEXT REFERENCES skills(id),
-  type TEXT NOT NULL,           -- 'depends_on', 'extends', 'alternative', 'related'
-  confidence REAL,
-  reasoning TEXT,
-  created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
-  PRIMARY KEY (source_skill_id, target_skill_id, type)
-);
-
--- Track external sources
-CREATE TABLE skill_sources (
-  id TEXT PRIMARY KEY,
-  type TEXT NOT NULL,           -- 'github-repo', 'awesome-list', 'trajectory', 'manual'
-  url TEXT,
-  last_scraped DATETIME,
-  etag TEXT,
-  skill_count INTEGER DEFAULT 0
-);
-```
-
-### 3.2 Extended Skill Type
-
-Add optional indexer fields to the core Skill type:
-
-```typescript
-// src/types.ts
-export interface Skill {
-  // ... existing fields ...
-
-  // Indexer extensions (optional)
-  taxonomy?: {
-    primaryPath: string[];
-    secondaryPaths?: string[][];
-    confidence?: number;
-  };
-  relationships?: SkillRelationship[];
-  externalSource?: {
-    url: string;
-    repo: string;
-    scrapedAt: Date;
-    etag?: string;
-  };
-}
-
-export interface SkillRelationship {
-  targetSkillId: string;
-  type: 'depends_on' | 'extends' | 'alternative' | 'related';
-  confidence: number;
-  reasoning?: string;
-}
-```
-
-### 3.3 Storage Adapter Extension
-
-Extend `SqliteStorageAdapter` to handle taxonomy and relationships:
-
-```typescript
-// src/storage/sqlite.ts
-export class SqliteStorageAdapter implements StorageAdapter {
-  // ... existing methods ...
-
-  // Taxonomy methods
-  async getTaxonomyTree(rootPath?: string[]): Promise<TaxonomyTree>;
-  async placeInTaxonomy(skillId: string, placement: TaxonomyPlacement): Promise<void>;
-
-  // Relationship methods
-  async addRelationship(rel: SkillRelationship): Promise<void>;
-  async getRelationships(skillId: string): Promise<SkillRelationship[]>;
-  async getSkillsInNode(nodeId: string): Promise<Skill[]>;
-}
-```
-
-**Effort**: ~8-10 hours
-**Files**: `src/storage/sqlite.ts`, `src/types.ts`, migration scripts
-
----
-
-## Phase 4: Direct Integration
-
-**Goal**: Scraper writes directly to SkillBank, bypassing export/import.
-
-### 4.1 Indexer as SkillBank Client
-
-Modify indexer to use SkillBank as its storage backend:
-
-```typescript
-// src/services/indexer.ts
-export class IndexerService {
-  constructor(
-    private skillBank: SkillBank,
-    private classifier: SkillClassifier,
-    private config: IndexerConfig
-  ) {}
-
-  async scrapeAndIndex(sources: SkillSource[]): Promise<IndexResult> {
-    const rawSkills = await this.scraper.scrape(sources);
-
-    for (const raw of rawSkills) {
-      // Convert to skill-tree format
-      const skill = this.converter.convert(raw);
-
-      // Classify
-      const classification = await this.classifier.classify(skill);
-      skill.taxonomy = classification;
-
-      // Store directly in SkillBank
-      await this.skillBank.addSkill(skill, {
-        source: 'indexed',
-        sourceUrl: raw.sourceUrl,
-      });
-    }
-
-    return { indexed: rawSkills.length };
-  }
-}
-```
-
-### 4.2 Event-Driven Updates
-
-Use SkillBank events to trigger relationship detection:
-
-```typescript
-skillBank.on('skill:created', async (event) => {
-  if (event.skill.source?.type === 'imported') {
-    await relationshipDetector.detectForSkill(event.skill);
-  }
-});
-```
-
-### 4.3 Workflow
-
-```bash
-# Direct flow (no export/import step)
-skill-tree index scrape https://github.com/VoltAgent/awesome-agent-skills
-# Skills are now in the main skill-tree database with taxonomy and relationships
-```
-
-**Effort**: ~6-8 hours
-**Files**: `src/services/indexer.ts`, `src/skill-bank.ts` event handlers
-
----
-
-## Phase 5: Bidirectional Sync
-
-**Goal**: Keep indexed skills updated and track changes.
-
-### 5.1 Change Detection
-
-```typescript
-interface SkillSyncState {
-  skillId: string;
-  sourceUrl: string;
-  sourceEtag: string;
-  lastSynced: Date;
-  localVersion: string;
-  remoteVersion: string;
-  status: 'synced' | 'local-modified' | 'remote-modified' | 'conflict';
-}
-```
-
-### 5.2 Sync Commands
-
-```bash
-skill-tree index sync              # Check for updates
-skill-tree index sync --pull       # Pull remote changes
-skill-tree index sync --push       # Push local changes (to forks?)
-skill-tree index sync --status     # Show sync status
-```
-
-### 5.3 Conflict Resolution
-
-When both local and remote have changed:
-
-1. Create a new version with merged changes
-2. Keep lineage tracking intact
-3. Mark conflicts for manual review
-
-**Effort**: ~10-12 hours
-**Files**: `src/services/sync.ts`, `src/cli/commands/index/sync.ts`
-
----
-
-## Implementation Order
-
-| Phase | Description | Priority | Effort | Dependencies |
-|-------|-------------|----------|--------|--------------|
-| 1 | Import Pipeline | High | 2-3 hrs | None |
-| 2 | CLI Unification | High | 6-8 hrs | Phase 1 |
-| 3 | Unified Storage | Medium | 8-10 hrs | Phase 2 |
-| 4 | Direct Integration | Medium | 6-8 hrs | Phase 3 |
-| 5 | Bidirectional Sync | Low | 10-12 hrs | Phase 4 |
-
-**Recommended starting point**: Phase 1 (immediate value) → Phase 2 (user experience) → Phase 3 (foundation for advanced features)
-
----
-
-## Migration Strategy
-
-### For Existing Indexer Users
-
-```bash
-# Export existing indexed skills
-skillindexer export -o backup.json
-
-# After integration, import into unified database
-skill-tree import --from-indexer backup.json
-
-# Verify
-skill-tree list --source indexed
-skill-tree index taxonomy
-```
-
-### Database Migration
-
-Create migration script for existing SQLite databases:
-
-```typescript
-// scripts/migrate-indexer-db.ts
-async function migrateIndexerDatabase(
-  indexerDbPath: string,
-  skillTreeDbPath: string
-): Promise<MigrationResult> {
-  // 1. Read all skills from indexer DB
-  // 2. Convert using existing converter
-  // 3. Insert into skill-tree DB with taxonomy preservation
-  // 4. Migrate taxonomy_nodes table
-  // 5. Migrate relationships table
-}
-```
-
----
-
-## Open Questions
-
-1. **Taxonomy ownership**: Should indexed skills always preserve their original taxonomy, or should they be re-classified by skill-tree's own taxonomy?
-
-2. **Deduplication**: How to handle the same skill discovered from multiple sources?
-
-3. **Version conflicts**: When an indexed skill is updated externally and locally modified, which takes precedence?
-
-4. **Namespace separation**: Should indexed skills have a separate namespace (e.g., `indexed:skill-name`) or be fully merged?
-
----
-
-## Success Criteria
-
-- [ ] Single CLI (`skill-tree`) for all operations
-- [ ] Single database with skills from all sources
-- [ ] Taxonomy browsing works for all skills
-- [ ] Relationships detected across indexed and extracted skills
-- [ ] Clear provenance tracking (source: extracted vs indexed)
-- [ ] Seamless workflow from discovery to usage

package/references/skill-tree/docs/TODOS.md

@@ -1,91 +0,0 @@
-# skill-tree TODOs
-
-Tracking remaining features and improvements for the skill-tree library.
-
-## Completed ✅
-
-- [x] **Core Types** - Skill, Trajectory, Extraction, Storage interfaces
-- [x] **Session Adapters** - Claude Code JSONL parser, extensible registry
-- [x] **Manual Extraction** - User-guided extraction with quality gates
-- [x] **Automatic Extraction** - LLM-powered skill identification
-- [x] **Quality Gates** - Reusability, non-trivial, specific triggers, verified
-- [x] **Memory Storage** - In-memory adapter for testing
-- [x] **Filesystem Storage** - OpenSkills-compatible (YAML frontmatter + Markdown)
-- [x] **Semantic Versioning** - Parse, compare, bump, range satisfaction
-- [x] **Lineage Tracking** - Version history, fork, merge, rollback
-- [x] **SkillBank Orchestrator** - Unified API, events, CRUD, import/export
-- [x] **Test Suite** - 352 tests across all modules
-- [x] **Semantic Matching** - Embedding providers, similarity search, context matching, caching
-- [x] **Batch Processing** - Multi-session extraction, deduplication, parallel processing, progress reporting
-- [x] **Skill Composer** - Compound skills, dependency graph, composition suggestions, conflict detection
-- [x] **Additional Session Adapters** - OpenAI, Anthropic, Generic JSON with configurable field mapping
-- [x] **Skill Merging Improvements** - SkillMerger with conflict detection, preview, multiple strategies, merge suggestions
-- [x] **Hooks & Activation System** - HookRegistry, ActivationManager, built-in hooks, auto-trigger extraction
-- [x] **CLI Tool** - Management CLI with list, show, search, stats, versions, diff, rollback, fork, deprecate, activate, delete, export, import
-- [x] **SQLite Storage** - SQLite adapter with FTS5 full-text search, WAL mode, version history
-- [x] **Metrics & Analytics** - MetricsTracker with usage tracking, feedback, rankings, trends, reports, deprecation suggestions
-- [x] **AGENTS.md Integration** - Generator, Parser, Sync for Claude Code's AGENTS.md format (XML, Markdown, JSON)
-- [x] **Skill Validation** - SkillValidator with test cases, version compatibility checking, deprecation suggestions, scheduling
-- [x] **Test Suite** - 459 tests across all modules
-
----
-
-## High Priority
-
-(All high priority items completed!)
-
----
-
-## Medium Priority
-
-(All medium priority items completed!)
-
----
-
-## Low Priority
-
-### Database Storage Adapters
-- [x] SQLite adapter for local persistence
-- [ ] PostgreSQL adapter for team/production use
-- [ ] Migration support between storage backends
-- [x] Full-text search integration (FTS5)
-
-**Why**: Filesystem works well for individual use; DB needed for larger deployments.
-
-### Skill Validation
-- [x] Runtime verification that skills still work
-- [x] Periodic re-testing of skill solutions
-- [x] Deprecation suggestions for failing skills
-- [x] Version compatibility checking
-
-**Why**: Skills can become stale as codebases/tools evolve.
-
-### AGENTS.md Integration
-- [x] Generate AGENTS.md from skill bank (like OpenSkills)
-- [x] Sync skills to/from AGENTS.md
-- [x] XML format for skill descriptions
-- [x] Selective skill exposure
-
-**Why**: Integration with Claude Code's native skill loading.
-
----
-
-## Ideas / Future Exploration
-
-- [ ] **Tiered Skill Loading** - Skills can have links to other skills for progressive discovery/loading for large skill libraries
-- [ ] **Skill Templates** - Predefined templates for common skill types
-- [ ] **Skill Marketplace** - Share skills between users/teams
-- [ ] **A/B Testing** - Compare skill versions in practice
-- [ ] **Auto-improvement** - Use feedback to refine skills automatically
-- [ ] **Cross-language Support** - Skills that work across programming languages
-- [ ] **Skill Dependencies** - Declare when one skill requires another
-- [ ] **Context Injection** - Automatically inject relevant skills into agent context
-- [ ] **Skill Clustering** - Group similar skills automatically
-
----
-
-## References
-
-- [arXiv:2512.17102](https://arxiv.org/abs/2512.17102) - Skill Libraries via RL
-- [Claudeception](https://github.com/blader/Claudeception) - Quality gates, extraction hooks
-- [OpenSkills](https://github.com/numman-ali/openskills) - Skill format, discovery mechanism