cognitive-core 0.2.1 → 0.2.3

package/references/skill-tree/docs/design/multi-agent-sync.md

@@ -1,759 +0,0 @@
-# Multi-Agent Skill Synchronization Design
-
-## Overview
-
-This document explores how skill-tree can support synchronization across multiple agents and compute environments, enabling collaborative skill sharing and evolution.
-
----
-
-## Design Decisions
-
-The following decisions have been made based on design discussions:
-
-| Decision | Choice | Rationale |
-|----------|--------|-----------|
-| **Sync Model** | Shared Main (Model A) | Simple to start, can evolve to Model C later |
-| **Transport** | Git-first, HTTP later | Git gives versioning for free, hybrid approach |
-| **Sync Frequency** | Periodic | Simpler than real-time, sufficient for most cases |
-| **Conflict Resolution** | Auto-resolve when possible | `prefer-newer` + `prefer-local`, use `SkillMerger` for complex |
-| **Agent Identity** | User-provided slug | Simple, human-readable, optional UUID later |
-| **Version Storage** | Git history in git mode | `.versions/` folder is mirror only, not source of truth |
-| **MVP Scope** | Pull, push, conflict workflow | Minimum viable for multi-agent collaboration |
-
-### Version Storage Clarification
-
-In **git mode**, the `.versions/` folder behavior changes:
-- Git commit history is the **source of truth** for version history
-- `.versions/` folder is **optional mirror** for non-git access patterns
-- `getVersionHistory()` reads from git log, not filesystem
-- Local storage (SQLite/Memory) still uses `.versions/` as primary
-
-## Current Architecture (What We Have)
-
-### Existing Strengths
-- **StorageAdapter interface** - Clean abstraction (Memory, Filesystem, SQLite)
-- **FilesystemStorageAdapter** - YAML frontmatter + Markdown (git-friendly)
-- **SkillMerger** - Field-level conflict detection with multiple strategies
-- **LineageTracker** - Version history, forks, parentVersion, derivedFrom
-- **Content hashing** - Change detection via contentHash
-- **SyncService** - Basic GitHub pull-based sync with ETag tracking
-
-### Current Limitations
-- No push capability (pull-only from GitHub)
-- No agent-to-agent sync
-- No distributed conflict resolution
-- No agent identity/provenance tracking
-- Version history stored in `.versions/` folders (redundant with git)
-
----
-
-## Proposed Architecture
-
-### Repository Structure
-
-```
-skills-repo/
-├── .git/
-├── skills/
-│   ├── error-handling-python/
-│   │   └── SKILL.md              # YAML frontmatter + markdown
-│   ├── api-rate-limiting/
-│   │   └── SKILL.md
-│   └── ...
-├── .skillbank/
-│   ├── config.yaml               # Sync settings, remote URLs
-│   ├── agents/                   # Agent manifests
-│   │   ├── agent-alpha.yaml
-│   │   └── agent-beta.yaml
-│   ├── sync-state.yaml           # Track what's synced where
-│   └── conflicts/                # Unresolved conflicts
-│       └── {skillId}-{timestamp}.yaml
-├── taxonomy/
-│   └── tree.yaml                 # Shared taxonomy
-└── README.md
-```
-
-**Key insight**: Git history IS version history. No need for `.versions/` folders—each skill version is a commit.
-
----
-
-## Core Interfaces
-
-### SyncConfig
-
-```typescript
-interface SyncConfig {
-  // Remote repository configuration
-  remote: {
-    type: 'git' | 'http' | 'custom';
-    url: string;                    // git@github.com:org/skills.git
-    branch?: string;                // default: 'main'
-    skillsPath?: string;            // default: 'skills/'
-    auth?: {
-      type: 'ssh' | 'token' | 'oauth';
-      // ... credentials
-    };
-  };
-
-  // Agent identity
-  agent: {
-    id: string;                     // unique identifier (uuid or slug)
-    name?: string;                  // human-readable
-    environment?: string;           // 'production', 'dev', 'agent-cluster-1'
-    capabilities?: string[];        // what this agent specializes in
-  };
-
-  // Sync behavior
-  sync: {
-    mode: 'manual' | 'on-change' | 'periodic';
-    intervalMs?: number;            // for periodic mode
-    pushStrategy: 'immediate' | 'batch' | 'validated-only';
-    pullStrategy: 'auto-merge' | 'manual' | 'preview-first';
-    branchStrategy: 'shared-main' | 'agent-branches' | 'promotion';
-  };
-
-  // Conflict resolution
-  conflicts: {
-    defaultStrategy: ConflictStrategy;
-    fieldStrategies?: Record<keyof Skill, ConflictStrategy>;
-    autoResolveThreshold?: number;  // confidence level for auto-resolve
-  };
-}
-
-type ConflictStrategy =
-  | 'prefer-local'    // keep local version
-  | 'prefer-remote'   // accept remote version
-  | 'prefer-newer'    // compare updatedAt timestamps
-  | 'union'           // for arrays: combine unique items
-  | 'combine'         // for strings: concatenate
-  | 'manual';         // delegate to SkillMerger for resolution
-```
-
-### SyncService Interface
-
-```typescript
-interface SyncService {
-  // Initialize sync (clone or configure existing repo)
-  initialize(config: SyncConfig): Promise<void>;
-
-  // Fetch remote state without applying
-  fetch(): Promise<RemoteState>;
-
-  // Pull and merge remote changes
-  pull(options?: PullOptions): Promise<SyncResult>;
-
-  // Push local changes to remote
-  push(options?: PushOptions): Promise<SyncResult>;
-
-  // Get current sync status
-  status(): Promise<SyncStatus>;
-
-  // Resolve a pending conflict
-  resolveConflict(skillId: string, resolution: ConflictResolution): Promise<void>;
-
-  // List pending conflicts
-  listConflicts(): Promise<SkillConflict[]>;
-}
-
-interface PullOptions {
-  skillIds?: string[];          // specific skills, or all
-  force?: boolean;              // overwrite local changes
-  dryRun?: boolean;             // preview only
-}
-
-interface PushOptions {
-  skillIds?: string[];          // specific skills, or all
-  message?: string;             // commit message
-  force?: boolean;              // overwrite remote (dangerous)
-  minValidationScore?: number;  // only push validated skills
-}
-
-interface SyncResult {
-  pulled: SkillChange[];
-  pushed: SkillChange[];
-  conflicts: SkillConflict[];
-  autoMerged: SkillMerge[];
-  errors: SyncError[];
-}
-
-interface SyncStatus {
-  lastSync: Date;
-  localAhead: number;           // commits ahead of remote
-  localBehind: number;          // commits behind remote
-  pendingConflicts: number;
-  modifiedSkills: string[];     // locally modified
-  stagedSkills: string[];       // ready to push
-}
-```
-
-### Skill Conflict Model
-
-```typescript
-interface SkillConflict {
-  skillId: string;
-  localVersion: Skill;
-  remoteVersion: Skill;
-  baseVersion?: Skill;          // common ancestor (3-way merge)
-  conflictingFields: (keyof Skill)[];
-  suggestedResolution?: Skill;  // auto-merged where possible
-  confidence: number;           // 0-1, how confident in suggestion
-}
-
-interface ConflictResolution {
-  strategy: 'accept-local' | 'accept-remote' | 'accept-merged' | 'custom';
-  customSkill?: Skill;          // for 'custom' strategy
-  fieldOverrides?: Partial<Record<keyof Skill, 'local' | 'remote'>>;
-}
-```
-
-### Agent Provenance
-
-```typescript
-// Added to Skill interface
-interface SkillProvenance {
-  createdBy: AgentRef;
-  modifiedBy: AgentRef[];
-  validatedBy: AgentValidation[];
-}
-
-interface AgentRef {
-  agentId: string;
-  environment?: string;
-  timestamp: Date;
-}
-
-interface AgentValidation {
-  agentId: string;
-  timestamp: Date;
-  outcome: 'success' | 'failure' | 'partial';
-  context?: string;             // what task was this used for
-  score?: number;               // 0-1 effectiveness rating
-}
-```
-
----
-
-## Sync Models
-
-### Model A: Shared Main Branch
-
-```
-main ─────●─────●─────●─────●─────●
-          ↑     ↑     ↑     ↑     ↑
-        agent1 agent2 agent1 agent3 agent2
-```
-
-**How it works:**
-- All agents push directly to main
-- Conflicts resolved at push time
-- First to push wins; others must pull and merge
-
-**Pros:**
-- Simple mental model
-- Always up-to-date
-- Single source of truth
-
-**Cons:**
-- Contention with many agents
-- Fast-moving skills may conflict often
-- Requires good auto-merge or frequent manual resolution
-
-**Best for:** Small teams, low-frequency updates, high-trust environments
-
----
-
-### Model B: Agent Branches + Periodic Merge
-
-```
-main ────────●───────────────●───────────────●
-             │               │               │
-agent-alpha ─┴──●──●──●──────┴──●──●──●──────┴──
-             │
-agent-beta ──┴──●──●─────●──────●──●─────●─────
-```
-
-**How it works:**
-- Each agent works on their own branch
-- Periodic sync to/from main (manual or scheduled)
-- Agents can cherry-pick from each other
-
-**Pros:**
-- Isolation—agents don't block each other
-- Can work offline for extended periods
-- Review before merging to main
-
-**Cons:**
-- Merge complexity increases over time
-- Divergence risk
-- More branches to manage
-
-**Best for:** Distributed teams, agents with different specializations, environments needing review gates
-
----
-
-### Model C: Validation-Based Promotion
-
-```
-                                    ┌─────────────┐
-                                    │   active    │  (validated by 3+ agents)
-                                    └──────▲──────┘
-                                           │ promote
-                                    ┌──────┴──────┐
-                                    │ experimental │  (validated by 1+ agent)
-                                    └──────▲──────┘
-                                           │ promote
-┌─────────┐  ┌─────────┐  ┌─────────┐      │
-│ Agent 1 │  │ Agent 2 │  │ Agent 3 │──────┘
-│ (draft) │  │ (draft) │  │ (draft) │
-└─────────┘  └─────────┘  └─────────┘
-```
-
-**How it works:**
-- Skills created locally as `draft`
-- After N successful uses → pushed as `experimental`
-- After M validations across agents → promoted to `active`
-- Conflicts resolved by validation score
-
-**Promotion rules:**
-```typescript
-interface PromotionRules {
-  draftToExperimental: {
-    minLocalSuccesses: number;      // e.g., 3
-    minSuccessRate: number;         // e.g., 0.7
-  };
-  experimentalToActive: {
-    minAgentValidations: number;    // e.g., 3 different agents
-    minTotalSuccesses: number;      // e.g., 10
-    minSuccessRate: number;         // e.g., 0.8
-  };
-  deprecation: {
-    maxConsecutiveFailures: number; // e.g., 5
-    minFailureRate: number;         // e.g., 0.5 over last 20 uses
-  };
-}
-```
-
-**Pros:**
-- Quality gate—only validated skills propagate
-- Conflict resolution is data-driven
-- Natural curation without human review
-
-**Cons:**
-- Slower propagation of new skills
-- Cold-start problem for new skills
-- Needs robust validation tracking
-
-**Best for:** Production systems, quality-critical environments, large agent fleets
-
----
-
-## Field-Level Merge Strategies
-
-When two agents modify the same skill, use semantic merging instead of line-by-line:
-
-| Field | Default Strategy | Rationale |
-|-------|------------------|-----------|
-| `triggerConditions` | `union` | Both agents discovered valid triggers |
-| `examples` | `union` | More examples = better coverage |
-| `tags` | `union` | Additive metadata |
-| `solution` | `combine` or `manual` | Core content, may need review |
-| `verification` | `prefer-newer` | Usually improvements |
-| `metrics` | `merge-stats` | Combine usage statistics |
-| `notes` | `combine` | Accumulate learnings |
-| `status` | `prefer-higher` | draft < experimental < active |
-| `version` | `bump` | Auto-increment on merge |
-
-```typescript
-const defaultFieldStrategies: Record<keyof Skill, ConflictStrategy> = {
-  id: 'error',                    // should never conflict
-  version: 'bump',                // auto-increment
-  problem: 'prefer-longer',       // more detailed is better
-  triggerConditions: 'union',
-  solution: 'manual',             // too important to auto-merge
-  verification: 'prefer-newer',
-  examples: 'union',
-  notes: 'combine',
-  tags: 'union',
-  status: 'prefer-higher',
-  metrics: 'merge-stats',
-  // ...
-};
-```
-
----
-
-## Implementation Phases
-
-### Phase 1: Git Sync Foundation (MVP)
-
-**Goal**: Enable many agents on one shared repo with periodic sync.
-
-#### 1.1 Core Infrastructure
-- [ ] `GitSyncAdapter` class wrapping `simple-git`
-- [ ] Extends/composes with `FilesystemStorageAdapter`
-- [ ] Agent identity configuration (`SyncConfig.agent`)
-- [ ] Remote repository configuration (`SyncConfig.remote`)
-
-#### 1.2 Pull Operation
-```typescript
-async pull(options?: PullOptions): Promise<SyncResult>
-```
-- [ ] `git fetch origin main`
-- [ ] Detect changed skill files via `git diff`
-- [ ] For each changed skill:
-  - New remote skill → import to local storage
-  - Updated remote skill (no local changes) → update local
-  - Both changed → detect conflict
-- [ ] Return `SyncResult` with pulled/conflicts/errors
-
-#### 1.3 Push Operation
-```typescript
-async push(options?: PushOptions): Promise<SyncResult>
-```
-- [ ] Export locally modified skills to filesystem
-- [ ] `git add` + `git commit` with agent attribution
-- [ ] `git push origin main`
-- [ ] Handle non-fast-forward (pull first, then retry)
-- [ ] Return `SyncResult` with pushed/conflicts/errors
-
-#### 1.4 Conflict Workflow
-- [ ] Detect conflicts: local and remote both modified since last sync
-- [ ] Store conflicts in `.skillbank/conflicts/`
-- [ ] `listConflicts()` → list pending conflicts
-- [ ] `resolveConflict(skillId, resolution)` → apply resolution
-- [ ] Integration with `SkillMerger` for field-level merge suggestions
-
-#### 1.5 CLI Commands
-```bash
-skill-tree sync status          # Show sync status
-skill-tree sync pull            # Pull remote changes
-skill-tree sync push            # Push local changes
-skill-tree sync conflicts       # List pending conflicts
-skill-tree sync resolve <id>    # Resolve a conflict
-```
-
-### Phase 2: Enhanced Conflict Resolution
-
-- [ ] 3-way merge using git merge-base
-- [ ] Field-specific auto-resolution strategies
-- [ ] Conflict preview with suggested merge
-- [ ] Interactive conflict resolution CLI
-
-### Phase 3: Multi-Agent Coordination (Future)
-
-- [ ] Agent branch support (Model B)
-- [ ] Validation tracking and provenance
-- [ ] Promotion rules (Model C)
-- [ ] Cross-agent skill discovery
-
-### Phase 4: Advanced Features (Future)
-
-- [ ] HTTP API sync adapter
-- [ ] Real-time sync via webhooks
-- [ ] Peer-to-peer sync
-- [ ] Skill request/broadcast protocol
-
----
-
-## Conflict Resolution Flow
-
-```
-┌─────────────────────────────────────────────────────────────────┐
-│                        On Pull                                   │
-├─────────────────────────────────────────────────────────────────┤
-│                                                                  │
-│   Remote changed?  ──No──►  No action                           │
-│        │                                                         │
-│       Yes                                                        │
-│        │                                                         │
-│        ▼                                                         │
-│   Local changed?  ──No──►  Accept remote (fast-forward)         │
-│        │                                                         │
-│       Yes                                                        │
-│        │                                                         │
-│        ▼                                                         │
-│   Can auto-merge?  ──Yes──►  Apply merge, record in SyncResult  │
-│        │                                                         │
-│       No                                                         │
-│        │                                                         │
-│        ▼                                                         │
-│   Store conflict in .skillbank/conflicts/                       │
-│   Return in SyncResult.conflicts                                │
-│                                                                  │
-└─────────────────────────────────────────────────────────────────┘
-```
-
-**Auto-merge conditions** (using existing SkillMerger):
-1. Only non-overlapping fields changed → merge both
-2. Same field changed to same value → no conflict
-3. Array fields (tags, examples, triggers) → union
-4. Otherwise → manual resolution required
-
----
-
-## GitSyncAdapter Interface (Detailed)
-
-The `GitSyncAdapter` is the core component for git-based synchronization. It wraps the `FilesystemStorageAdapter` and adds git operations.
-
-### Class Design
-
-```typescript
-import { SimpleGit, simpleGit } from 'simple-git';
-import { FilesystemStorageAdapter } from '../storage/filesystem.js';
-import { SkillMerger } from '../versioning/merge.js';
-
-/**
- * Git-aware storage adapter for multi-agent synchronization
- *
- * Extends FilesystemStorageAdapter with:
- * - Git operations (fetch, pull, push, commit)
- * - Agent identity tracking
- * - Conflict detection and resolution workflow
- */
-export class GitSyncAdapter {
-  private git: SimpleGit;
-  private storage: FilesystemStorageAdapter;
-  private merger: SkillMerger;
-  private config: SyncConfig;
-  private conflictStore: ConflictStore;
-
-  constructor(options: GitSyncAdapterOptions) {
-    this.config = options.config;
-    this.git = simpleGit(options.repoPath);
-    this.storage = new FilesystemStorageAdapter(options.repoPath);
-    this.merger = new SkillMerger();
-    this.conflictStore = new ConflictStore(
-      path.join(options.repoPath, '.skillbank', 'conflicts')
-    );
-  }
-
-  /**
-   * Initialize the adapter
-   * - Ensures git repo exists
-   * - Configures remote if needed
-   * - Loads existing conflict state
-   */
-  async initialize(): Promise<void>;
-
-  /**
-   * Fetch remote changes without applying
-   * Returns summary of what would change
-   */
-  async fetch(): Promise<FetchResult>;
-
-  /**
-   * Pull remote changes and merge into local
-   *
-   * Flow:
-   * 1. git fetch origin main
-   * 2. Diff local vs remote skill files
-   * 3. For each changed skill:
-   *    - No local changes → accept remote
-   *    - Local changes, no remote → keep local
-   *    - Both changed → attempt auto-merge or create conflict
-   * 4. Update local storage
-   * 5. Return result with conflicts
-   */
-  async pull(options?: PullOptions): Promise<SyncResult>;
-
-  /**
-   * Push local changes to remote
-   *
-   * Flow:
-   * 1. Get locally modified skills (vs last sync point)
-   * 2. Export skills to filesystem (already done by storage)
-   * 3. git add skills/
-   * 4. git commit with agent attribution
-   * 5. git push origin main
-   * 6. Handle non-fast-forward: pull, merge, retry
-   */
-  async push(options?: PushOptions): Promise<SyncResult>;
-
-  /**
-   * Get current sync status
-   */
-  async status(): Promise<SyncStatus>;
-
-  /**
-   * List pending conflicts
-   */
-  async listConflicts(): Promise<SkillConflict[]>;
-
-  /**
-   * Resolve a conflict
-   *
-   * Strategies:
-   * - 'accept-local': Keep local version, discard remote
-   * - 'accept-remote': Accept remote version, discard local changes
-   * - 'accept-merged': Apply auto-merged suggestion
-   * - 'custom': Apply user-provided merged skill
-   */
-  async resolveConflict(
-    skillId: string,
-    resolution: ConflictResolution
-  ): Promise<void>;
-
-  // === Internal helpers ===
-
-  /**
-   * Get skills modified locally since last sync
-   */
-  private async getLocallyModifiedSkills(): Promise<Skill[]>;
-
-  /**
-   * Get skills modified on remote since last sync
-   */
-  private async getRemoteChangedSkills(): Promise<RemoteSkillChange[]>;
-
-  /**
-   * Attempt to auto-merge two skill versions
-   * Uses SkillMerger with configured field strategies
-   */
-  private async attemptAutoMerge(
-    local: Skill,
-    remote: Skill,
-    base?: Skill
-  ): Promise<MergeAttempt>;
-
-  /**
-   * Get the common ancestor (base) version for 3-way merge
-   */
-  private async getBaseVersion(skillId: string): Promise<Skill | null>;
-
-  /**
-   * Track sync point (commit hash) for change detection
-   */
-  private async recordSyncPoint(): Promise<void>;
-}
-```
-
-### Supporting Types
-
-```typescript
-interface GitSyncAdapterOptions {
-  repoPath: string;           // Local git repository path
-  config: SyncConfig;         // Sync configuration
-}
-
-interface FetchResult {
-  ahead: number;              // Local commits not on remote
-  behind: number;             // Remote commits not local
-  changedSkills: string[];    // Skill IDs with changes
-  newSkills: string[];        // New skills on remote
-  deletedSkills: string[];    // Skills deleted on remote
-}
-
-interface RemoteSkillChange {
-  skillId: string;
-  changeType: 'added' | 'modified' | 'deleted';
-  remoteSkill?: Skill;
-  commitHash: string;
-  author: string;
-  timestamp: Date;
-}
-
-interface MergeAttempt {
-  success: boolean;
-  merged?: Skill;             // Merged result if successful
-  conflicts?: MergeConflict[]; // Field-level conflicts if failed
-  confidence: number;          // 0-1, how confident in merge
-}
-
-interface ConflictStore {
-  save(conflict: SkillConflict): Promise<void>;
-  get(skillId: string): Promise<SkillConflict | null>;
-  list(): Promise<SkillConflict[]>;
-  remove(skillId: string): Promise<void>;
-}
-```
-
-### Commit Message Format
-
-Agent attribution in commit messages:
-
-```
-[agent:claude-prod-1] Update skill: api-rate-limiting
-
-Modified fields: solution, examples
-Auto-merged: triggerConditions (union)
-
----
-Agent: claude-prod-1
-Environment: production
-Sync-Version: 1.0.0
-```
-
-### Sync State Tracking
-
-Track last sync point to detect changes:
-
-```yaml
-# .skillbank/sync-state.yaml
-lastSyncCommit: "abc123def456"
-lastSyncTime: "2024-01-15T10:30:00Z"
-agent:
-  id: "claude-prod-1"
-  environment: "production"
-skills:
-  api-rate-limiting:
-    localHash: "hash1"
-    syncedAt: "2024-01-15T10:30:00Z"
-    syncedVersion: "1.2.0"
-  error-handling-python:
-    localHash: "hash2"
-    syncedAt: "2024-01-15T10:30:00Z"
-    syncedVersion: "2.0.0"
-```
-
-### Integration with Existing SyncService
-
-The existing `SyncService` in `src/services/sync.ts` handles GitHub-based imports. The new `GitSyncAdapter` is complementary:
-
-| Component | Purpose |
-|-----------|---------|
-| `SyncService` | Pull-only sync from GitHub URLs (imports) |
-| `GitSyncAdapter` | Bidirectional sync with git repo (multi-agent) |
-
-They can coexist:
-- Skills imported via `SyncService` can be pushed to the shared repo
-- Skills from the shared repo can be imported to agents that don't have direct git access
-
----
-
-## Key Files to Create/Modify
-
-### New Files
-```
-src/sync/
-├── index.ts                    # Public exports
-├── types.ts                    # SyncConfig, SyncResult, etc.
-├── git-sync-adapter.ts         # GitSyncAdapter class
-├── conflict-store.ts           # Conflict persistence
-└── sync-service.ts             # High-level sync orchestration
-
-src/cli/commands/
-└── sync.ts                     # CLI commands for sync
-```
-
-### Modified Files
-```
-src/types.ts                    # Add SkillProvenance to Skill
-src/storage/filesystem.ts       # Add git-aware mode flag
-src/versioning/merge.ts         # Enhance for sync use cases
-```
-
----
-
-## Open Questions (Remaining)
-
-1. **Offline Duration**: How long can agents work offline before divergence becomes problematic?
-   - Recommendation: No hard limit, but warn on large divergence
-
-2. **Conflict Expiration**: Should unresolved conflicts expire?
-   - Recommendation: No, but surface them prominently in status
-
-3. **Partial Sync**: Should we support syncing only specific skills?
-   - Recommendation: Yes, via `--skills` flag in CLI
-
-4. **Authentication**: How to handle git auth in different environments?
-   - Recommendation: Rely on system git config (SSH keys, credential helpers)