cognitive-core 0.0.2 → 0.1.2

package/references/skill-tree/docs/GAPS_v1.md

@@ -0,0 +1,221 @@
+# Integration Gaps Tracking - v1
+
+This document tracks the remaining work to complete the skill indexer integration as defined in [INTEGRATION_PLAN.md](./INTEGRATION_PLAN.md).
+
+## Status Overview
+
+| Phase | Status | Completion |
+|-------|--------|------------|
+| Phase 1: Import Pipeline | ✅ Complete | 100% |
+| Phase 2: CLI Unification | ✅ Complete | 100% |
+| Phase 3: Unified Storage | ✅ Complete | 100% |
+| Phase 4: Direct Integration | ✅ Complete | 100% |
+| Phase 5: Bidirectional Sync | ✅ Complete | 100% |
+
+---
+
+## High Priority Items
+
+### 1. Unified Configuration System
+
+**Status**: ✅ Complete
+
+**Goal**: Single config file for both skill-tree and indexer settings.
+
+**Location**: `~/.skill-tree/config.yaml`
+
+**Tasks**:
+- [x] Create `src/config/` module
+- [x] Define `SkillTreeConfig` interface
+- [x] Implement config file loading with defaults
+- [x] Support environment variable substitution
+- [x] Integrate with CLI global options
+- [x] Integrate with IndexerService
+
+**Files Created**:
+- `src/config/types.ts` - Configuration type definitions
+- `src/config/loader.ts` - Config loading and merging logic
+- `src/config/index.ts` - Module exports
+- `src/cli/commands/config.ts` - CLI config command
+
+**Config Schema**:
+```yaml
+storage:
+  type: sqlite | filesystem | memory
+  path: ~/.skill-tree/skills.db
+
+indexer:
+  github_token: ${GITHUB_TOKEN}
+  anthropic_key: ${ANTHROPIC_API_KEY}
+  sources:
+    - https://github.com/VoltAgent/awesome-agent-skills
+  batch_size: 10
+  min_confidence: 0.7
+
+matching:
+  embedding_model: text-embedding-3-small
+  similarity_threshold: 0.8
+
+sync:
+  auto_check: false
+  check_interval_hours: 24
+  conflict_resolution: prompt
+
+cli:
+  output_format: table
+  color: true
+  quiet: false
+```
+
+---
+
+### 2. IndexerService Direct Execution
+
+**Status**: ✅ Complete
+
+**Goal**: CLI commands execute directly instead of delegating to subprocess.
+
+**Tasks**:
+- [x] Update `IndexerService.initialize()` to properly load scraper modules
+- [x] Implement `scrape()` method with actual scraper integration
+- [x] Implement `classify()` method with actual indexer integration
+- [x] Implement `detectRelationships()` method
+- [x] Implement `getTaxonomyTree()` using storage adapter
+- [x] Implement `getStats()` method
+- [x] Update CLI commands to use IndexerService directly
+- [x] Add fallback to subprocess if modules unavailable
+
+**Files Modified**:
+- `src/services/indexer.ts` - Full IndexerService implementation
+- `src/cli/commands/indexer/scrape.ts` - Direct execution with fallback
+- `src/cli/commands/indexer/classify.ts` - Direct execution with fallback
+- `src/cli/commands/indexer/taxonomy.ts` - Direct execution with fallback
+- `src/cli/commands/indexer/relationships.ts` - Direct execution with fallback
+- `src/cli/commands/indexer/stats.ts` - Direct execution with fallback
+
+---
+
+## Medium Priority Items
+
+### 3. Direct SkillBank Integration
+
+**Status**: ✅ Complete
+
+**Goal**: IndexerService writes directly to SkillBank, no export/import step.
+
+**Tasks**:
+- [x] Implement `IndexerService.scrapeAndIndex()` method
+- [x] Convert scraped skills on-the-fly using converter
+- [x] Store directly via SkillBank.addSkill()
+- [x] Auto-place in taxonomy after save
+- [x] Update `skill-tree index scrape` to use direct integration
+
+**Files Modified**:
+- `src/services/indexer.ts` - Added `scrapeAndIndex()` method
+- `src/skill-bank.ts` - Added `addSkill()` method with taxonomy placement
+- `src/cli/commands/indexer/scrape.ts` - Added `--import` flag for auto-import
+
+**Workflow After**:
+```bash
+# One command does everything
+skill-tree index scrape https://github.com/org/repo
+# Skills immediately available in skill-tree
+skill-tree list
+```
+
+---
+
+### 4. SkillBank Event System
+
+**Status**: ✅ Complete
+
+**Goal**: Event-driven hooks for skill lifecycle.
+
+**Tasks**:
+- [x] Add EventEmitter to SkillBank (already existed via `on()`)
+- [x] Define event types: `skill:created`, `skill:updated`, `skill:deleted`
+- [x] Emit events from `addSkill()`, `updateSkill()`, `deleteSkill()`
+- [x] Hook relationship detection to `skill:created` for imported skills
+- [x] Add `on()` and `off()` methods to SkillBank API
+- [x] Add `onEvent()` for typed event subscriptions
+- [x] Add `getStorage()` for advanced integration
+
+**Files Modified**:
+- `src/skill-bank.ts` - Enhanced event system with `off()`, `onEvent()`, `getStorage()`
+
+**Event Interface**:
+```typescript
+interface SkillEvent {
+  type: 'skill:created' | 'skill:updated' | 'skill:deleted' | 'skill:deprecated';
+  skill: Skill;
+  timestamp: Date;
+  source?: 'manual' | 'imported' | 'extracted';
+}
+```
+
+---
+
+## Lower Priority Items
+
+### 5. Bidirectional Sync
+
+**Status**: ✅ Complete
+
+**Goal**: Keep skills in sync with remote sources.
+
+**Tasks**:
+- [x] Create `src/services/sync.ts`
+- [x] Define `SkillSyncState` type
+- [x] Implement `checkForUpdates()` - compare ETags/versions
+- [x] Implement `pullRemoteChanges()` - fetch and merge
+- [x] Implement `detectConflicts()` - local vs remote changes
+- [x] Add `skill-tree index sync` command with options:
+  - [x] `--status` - show sync state
+  - [x] `--pull` - pull remote changes
+  - [x] `--dry-run` - preview changes
+
+**Files Created**:
+- `src/services/sync.ts` - Full SyncService implementation
+
+**Files Modified**:
+- `src/cli/commands/indexer/sync.ts` - Updated with sync status and pull commands
+
+**Note**: SQLite sync_state table not yet added to schema (sync states stored in memory for now). This can be added in a future iteration.
+
+---
+
+## Open Questions (Decisions Made)
+
+| Question | Status | Decision |
+|----------|--------|----------|
+| Taxonomy ownership | ✅ Resolved | Preserve original, allow re-classification on demand |
+| Deduplication strategy | ✅ Resolved | Use `externalSource.url` as dedup key |
+| Version conflict resolution | ✅ Resolved | Default to manual, with `--resolve` flag for automation |
+| Namespace separation | ✅ Resolved | Use `source.type` field, no namespace prefix |
+
+---
+
+## Success Criteria Checklist
+
+- [x] Single CLI (`skill-tree`) for all operations
+- [x] Single database with skills from all sources
+- [x] Taxonomy browsing works for all skills
+- [x] Relationships detected across indexed and extracted skills
+- [x] Clear provenance tracking (source: extracted vs indexed)
+- [x] Seamless workflow from discovery to usage
+
+---
+
+## Change Log
+
+| Date | Change |
+|------|--------|
+| 2025-01-XX | Initial gap analysis created |
+| 2025-01-XX | Phases 1-3 implemented |
+| 2025-01-XX | Tests added (56 tests) |
+| 2025-02-01 | Unified configuration system implemented |
+| 2025-02-01 | IndexerService direct execution implemented |
+| 2025-02-01 | Direct SkillBank integration implemented |
+| 2025-02-01 | SkillBank event system enhanced |
+| 2025-02-01 | Bidirectional sync service implemented |
+| 2025-02-01 | All phases complete |

package/references/skill-tree/docs/INTEGRATION_PLAN.md

@@ -0,0 +1,467 @@
+# Skill Indexer Integration Plan
+
+## Overview
+
+This document outlines the plan to integrate the **skill indexer** (`scraper/`) with the main **skill-tree** CLI and SkillBank. The goal is to create a cohesive system where externally discovered skills from GitHub can seamlessly flow into the skill-tree data structure.
+
+## Current Architecture
+
+```
+┌─────────────────────────────────────┐     ┌─────────────────────────────────────┐
+│           SKILL INDEXER             │     │            SKILL-TREE               │
+│           (scraper/)                │     │              (src/)                 │
+├─────────────────────────────────────┤     ├─────────────────────────────────────┤
+│  • GitHub skill discovery           │     │  • Agent trajectory extraction      │
+│  • AI-based classification          │     │  • Skill versioning & lineage       │
+│  • Taxonomy management              │     │  • Semantic matching                │
+│  • Relationship detection           │     │  • Storage adapters                 │
+│  • SQLite (scraper/data/)           │     │  • SQLite/Filesystem/Memory         │
+├─────────────────────────────────────┤     ├─────────────────────────────────────┤
+│  CLI: skillindexer                  │     │  CLI: skill-tree                    │
+│  • scrape, index, tree, etc.        │     │  • list, show, import, export, etc. │
+└─────────────────────────────────────┘     └─────────────────────────────────────┘
+                    │                                         │
+                    └──────────── GAP ────────────────────────┘
+                         (no direct connection)
+```
+
+## Target Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────────────┐
+│                            UNIFIED SKILL-TREE                                   │
+├─────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                 │
+│  ┌─────────────────────┐    ┌─────────────────────┐    ┌──────────────────────┐ │
+│  │   Skill Sources     │    │    Core Engine      │    │     Storage          │ │
+│  ├─────────────────────┤    ├─────────────────────┤    ├──────────────────────┤ │
+│  │ • GitHub Indexer    │───▶│ • SkillBank         │◀──▶│ • Unified SQLite     │ │
+│  │ • Trajectory Extract│───▶│ • Versioning        │    │ • Filesystem backup  │ │
+│  │ • Manual Import     │───▶│ • Lineage Tracking  │    │ • Taxonomy tables    │ │
+│  │ • Composed Skills   │───▶│ • Semantic Match    │    │ • Relationship tables│ │
+│  └─────────────────────┘    └─────────────────────┘    └──────────────────────┘ │
+│                                                                                 │
+│  ┌─────────────────────────────────────────────────────────────────────────────┐│
+│  │                           Unified CLI: skill-tree                           ││
+│  ├─────────────────────────────────────────────────────────────────────────────┤│
+│  │ Core:    list | show | search | stats | versions | diff | export | import   ││
+│  │ Index:   scrape | index | taxonomy | relationships                          ││
+│  │ Evolve:  rollback | fork | deprecate | activate | delete                    ││
+│  └─────────────────────────────────────────────────────────────────────────────┘│
+└─────────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Phase 1: Import Pipeline (Quick Win)
+
+**Goal**: Enable `skill-tree import` to accept indexer output directly.
+
+### 1.1 Add `--from-indexer` Flag to Import Command
+
+Modify `src/cli/commands/import.ts` to detect and convert indexer format:
+
+```typescript
+// New option
+.option('--from-indexer', 'Import from skill-indexer export format')
+
+// Detection logic
+if (options.fromIndexer || detectIndexerFormat(content)) {
+  skills = convertFromIndexer(content);
+}
+```
+
+### 1.2 Integrate Converter
+
+Move or re-export the converter from `scraper/src/compat/converter.ts`:
+
+```
+src/import/
+├── index.ts
+├── converter.ts       # Adapted from scraper/src/compat/converter.ts
+└── detect.ts          # Format auto-detection
+```
+
+### 1.3 Workflow
+
+```bash
+# Current (two-step)
+skillindexer export-skilltree -o indexed-skills.json
+skill-tree import indexed-skills.json
+
+# After Phase 1 (still two-step, but seamless)
+skillindexer export -o indexed-skills.json
+skill-tree import --from-indexer indexed-skills.json
+```
+
+**Effort**: ~2-3 hours
+**Files**: `src/cli/commands/import.ts`, new `src/import/` directory
+
+---
+
+## Phase 2: CLI Unification
+
+**Goal**: Single CLI with all commands.
+
+### 2.1 New Command Group: `skill-tree index`
+
+Add indexer commands under a subcommand group:
+
+```
+skill-tree index scrape [url]         # Discover skills from GitHub
+skill-tree index classify             # AI-powered classification
+skill-tree index taxonomy [path]      # Browse taxonomy tree
+skill-tree index relationships        # Detect skill relationships
+skill-tree index stats                # Indexer statistics
+```
+
+### 2.2 Implementation Structure
+
+```
+src/cli/
+├── index.ts                    # Main entry (existing)
+├── commands/
+│   ├── list.ts                 # Existing commands
+│   ├── show.ts
+│   ├── import.ts
+│   ├── ...
+│   └── index/                  # NEW: indexer subcommands
+│       ├── index.ts            # Parent command group
+│       ├── scrape.ts
+│       ├── classify.ts
+│       ├── taxonomy.ts
+│       └── relationships.ts
+```
+
+### 2.3 Shared Services
+
+Create a service layer accessible by both command sets:
+
+```typescript
+// src/services/indexer.ts
+export class IndexerService {
+  constructor(
+    private skillBank: SkillBank,
+    private config: IndexerConfig
+  ) {}
+
+  async scrape(sources: SkillSource[]): Promise<ScrapeResult> { ... }
+  async classify(): Promise<ClassifyResult> { ... }
+  async detectRelationships(): Promise<RelationshipResult> { ... }
+}
+```
+
+### 2.4 Configuration Merge
+
+Unify configuration under a single config file:
+
+```yaml
+# ~/.skill-tree/config.yaml
+storage:
+  type: sqlite
+  path: ~/.skill-tree/skills.db
+
+indexer:
+  github_token: ${GITHUB_TOKEN}
+  anthropic_key: ${ANTHROPIC_API_KEY}
+  sources:
+    - https://github.com/VoltAgent/awesome-agent-skills
+    - https://github.com/VoltAgent/awesome-openclaw-skills
+  batch_size: 10
+  min_confidence: 0.7
+
+matching:
+  embedding_model: text-embedding-3-small
+```
+
+**Effort**: ~6-8 hours
+**Files**: New `src/cli/commands/index/`, `src/services/indexer.ts`
+
+---
+
+## Phase 3: Unified Storage
+
+**Goal**: Single database with shared schema for both indexed and extracted skills.
+
+### 3.1 Schema Extension
+
+Extend the existing SQLite schema to include indexer tables:
+
+```sql
+-- Existing skill-tree tables
+CREATE TABLE skills (...);
+CREATE TABLE skill_versions (...);
+
+-- New indexer tables (merged)
+CREATE TABLE taxonomy_nodes (
+  id TEXT PRIMARY KEY,
+  name TEXT NOT NULL,
+  description TEXT,
+  parent_id TEXT REFERENCES taxonomy_nodes(id),
+  path TEXT NOT NULL,           -- Materialized path: "Development/Python/Testing"
+  skill_count INTEGER DEFAULT 0,
+  created_at DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+
+CREATE TABLE skill_taxonomy_placements (
+  skill_id TEXT REFERENCES skills(id),
+  node_id TEXT REFERENCES taxonomy_nodes(id),
+  is_primary BOOLEAN DEFAULT FALSE,
+  confidence REAL,
+  reasoning TEXT,
+  PRIMARY KEY (skill_id, node_id)
+);
+
+CREATE TABLE skill_relationships (
+  source_skill_id TEXT REFERENCES skills(id),
+  target_skill_id TEXT REFERENCES skills(id),
+  type TEXT NOT NULL,           -- 'depends_on', 'extends', 'alternative', 'related'
+  confidence REAL,
+  reasoning TEXT,
+  created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+  PRIMARY KEY (source_skill_id, target_skill_id, type)
+);
+
+-- Track external sources
+CREATE TABLE skill_sources (
+  id TEXT PRIMARY KEY,
+  type TEXT NOT NULL,           -- 'github-repo', 'awesome-list', 'trajectory', 'manual'
+  url TEXT,
+  last_scraped DATETIME,
+  etag TEXT,
+  skill_count INTEGER DEFAULT 0
+);
+```
+
+### 3.2 Extended Skill Type
+
+Add optional indexer fields to the core Skill type:
+
+```typescript
+// src/types.ts
+export interface Skill {
+  // ... existing fields ...
+
+  // Indexer extensions (optional)
+  taxonomy?: {
+    primaryPath: string[];
+    secondaryPaths?: string[][];
+    confidence?: number;
+  };
+  relationships?: SkillRelationship[];
+  externalSource?: {
+    url: string;
+    repo: string;
+    scrapedAt: Date;
+    etag?: string;
+  };
+}
+
+export interface SkillRelationship {
+  targetSkillId: string;
+  type: 'depends_on' | 'extends' | 'alternative' | 'related';
+  confidence: number;
+  reasoning?: string;
+}
+```
+
+### 3.3 Storage Adapter Extension
+
+Extend `SqliteStorageAdapter` to handle taxonomy and relationships:
+
+```typescript
+// src/storage/sqlite.ts
+export class SqliteStorageAdapter implements StorageAdapter {
+  // ... existing methods ...
+
+  // Taxonomy methods
+  async getTaxonomyTree(rootPath?: string[]): Promise<TaxonomyTree>;
+  async placeInTaxonomy(skillId: string, placement: TaxonomyPlacement): Promise<void>;
+
+  // Relationship methods
+  async addRelationship(rel: SkillRelationship): Promise<void>;
+  async getRelationships(skillId: string): Promise<SkillRelationship[]>;
+  async getSkillsInNode(nodeId: string): Promise<Skill[]>;
+}
+```
+
+**Effort**: ~8-10 hours
+**Files**: `src/storage/sqlite.ts`, `src/types.ts`, migration scripts
+
+---
+
+## Phase 4: Direct Integration
+
+**Goal**: Scraper writes directly to SkillBank, bypassing export/import.
+
+### 4.1 Indexer as SkillBank Client
+
+Modify indexer to use SkillBank as its storage backend:
+
+```typescript
+// src/services/indexer.ts
+export class IndexerService {
+  constructor(
+    private skillBank: SkillBank,
+    private classifier: SkillClassifier,
+    private config: IndexerConfig
+  ) {}
+
+  async scrapeAndIndex(sources: SkillSource[]): Promise<IndexResult> {
+    const rawSkills = await this.scraper.scrape(sources);
+
+    for (const raw of rawSkills) {
+      // Convert to skill-tree format
+      const skill = this.converter.convert(raw);
+
+      // Classify
+      const classification = await this.classifier.classify(skill);
+      skill.taxonomy = classification;
+
+      // Store directly in SkillBank
+      await this.skillBank.addSkill(skill, {
+        source: 'indexed',
+        sourceUrl: raw.sourceUrl,
+      });
+    }
+
+    return { indexed: rawSkills.length };
+  }
+}
+```
+
+### 4.2 Event-Driven Updates
+
+Use SkillBank events to trigger relationship detection:
+
+```typescript
+skillBank.on('skill:created', async (event) => {
+  if (event.skill.source?.type === 'imported') {
+    await relationshipDetector.detectForSkill(event.skill);
+  }
+});
+```
+
+### 4.3 Workflow
+
+```bash
+# Direct flow (no export/import step)
+skill-tree index scrape https://github.com/VoltAgent/awesome-agent-skills
+# Skills are now in the main skill-tree database with taxonomy and relationships
+```
+
+**Effort**: ~6-8 hours
+**Files**: `src/services/indexer.ts`, `src/skill-bank.ts` event handlers
+
+---
+
+## Phase 5: Bidirectional Sync
+
+**Goal**: Keep indexed skills updated and track changes.
+
+### 5.1 Change Detection
+
+```typescript
+interface SkillSyncState {
+  skillId: string;
+  sourceUrl: string;
+  sourceEtag: string;
+  lastSynced: Date;
+  localVersion: string;
+  remoteVersion: string;
+  status: 'synced' | 'local-modified' | 'remote-modified' | 'conflict';
+}
+```
+
+### 5.2 Sync Commands
+
+```bash
+skill-tree index sync              # Check for updates
+skill-tree index sync --pull       # Pull remote changes
+skill-tree index sync --push       # Push local changes (to forks?)
+skill-tree index sync --status     # Show sync status
+```
+
+### 5.3 Conflict Resolution
+
+When both local and remote have changed:
+
+1. Create a new version with merged changes
+2. Keep lineage tracking intact
+3. Mark conflicts for manual review
+
+**Effort**: ~10-12 hours
+**Files**: `src/services/sync.ts`, `src/cli/commands/index/sync.ts`
+
+---
+
+## Implementation Order
+
+| Phase | Description | Priority | Effort | Dependencies |
+|-------|-------------|----------|--------|--------------|
+| 1 | Import Pipeline | High | 2-3 hrs | None |
+| 2 | CLI Unification | High | 6-8 hrs | Phase 1 |
+| 3 | Unified Storage | Medium | 8-10 hrs | Phase 2 |
+| 4 | Direct Integration | Medium | 6-8 hrs | Phase 3 |
+| 5 | Bidirectional Sync | Low | 10-12 hrs | Phase 4 |
+
+**Recommended starting point**: Phase 1 (immediate value) → Phase 2 (user experience) → Phase 3 (foundation for advanced features)
+
+---
+
+## Migration Strategy
+
+### For Existing Indexer Users
+
+```bash
+# Export existing indexed skills
+skillindexer export -o backup.json
+
+# After integration, import into unified database
+skill-tree import --from-indexer backup.json
+
+# Verify
+skill-tree list --source indexed
+skill-tree index taxonomy
+```
+
+### Database Migration
+
+Create migration script for existing SQLite databases:
+
+```typescript
+// scripts/migrate-indexer-db.ts
+async function migrateIndexerDatabase(
+  indexerDbPath: string,
+  skillTreeDbPath: string
+): Promise<MigrationResult> {
+  // 1. Read all skills from indexer DB
+  // 2. Convert using existing converter
+  // 3. Insert into skill-tree DB with taxonomy preservation
+  // 4. Migrate taxonomy_nodes table
+  // 5. Migrate relationships table
+}
+```
+
+---
+
+## Open Questions
+
+1. **Taxonomy ownership**: Should indexed skills always preserve their original taxonomy, or should they be re-classified by skill-tree's own taxonomy?
+
+2. **Deduplication**: How to handle the same skill discovered from multiple sources?
+
+3. **Version conflicts**: When an indexed skill is updated externally and locally modified, which takes precedence?
+
+4. **Namespace separation**: Should indexed skills have a separate namespace (e.g., `indexed:skill-name`) or be fully merged?
+
+---
+
+## Success Criteria
+
+- [ ] Single CLI (`skill-tree`) for all operations
+- [ ] Single database with skills from all sources
+- [ ] Taxonomy browsing works for all skills
+- [ ] Relationships detected across indexed and extracted skills
+- [ ] Clear provenance tracking (source: extracted vs indexed)
+- [ ] Seamless workflow from discovery to usage