trellis 1.0.8 → 2.0.6

package/README.md

@@ -1,116 +1,597 @@
-# Trellis Kernel
+<p align="center">
+  <img src="logo.png" alt="Trellis" width="128" />
+</p>
 
-Trellis is a **semantic operating system for knowledge work** that treats your workspace as a declarative configuration. This repository contains the core Trellis kernel: a high-performance, durable, and extensible semantic engine.
+# Trellis
 
-The kernel provides a unified interface for data ingestion, querying (EQL-S, Datalog, Natural Language), and automated reasoning, all backed by an append-only operation log for full reproducibility and sync.
+> **Graph-native, code-first version control.**
+> Every file save is an op. Every op is a node. History is a causal graph, not a linear diff.
+
+Trellis is a from-scratch VCS built on five pillars:
+
+1. **Causal Stream** — an immutable, content-addressed log of ops where every change is causally chained to its predecessor.
+2. **Semantic Patching** — changes are recorded as structured AST patches (`symbolRename`, `symbolModify`, …) rather than line diffs, enabling conflict-free merges.
+3. **Narrative Milestones** — you never "commit". Instead you carve checkpoints and milestones out of a continuous stream of work.
+4. **Governance Subgraph** — op signing, identity management, and policy rules enforced before ops enter the stream.
+5. **Idea Garden** — abandoned work clusters are automatically detected and surfaced for later revival.
+
+---
+
+## Table of Contents
+
+- [Trellis Overview](#trellis-overview)
+- [Project Surfaces](#project-surfaces)
+- [Quick Start](#quick-start)
+- [CLI Overview](#cli-overview)
+- [VS Code Extension](#vs-code-extension)
+- [Module & Subpath Guide](#module--subpath-guide)
+- [Architecture Overview](#architecture-overview)
+- [Development & Releases](#development--releases)
+- [Roadmap](#roadmap)
+
+---
+
+## Trellis Overview
+
+Trellis is an umbrella project with a few distinct surfaces that all sit on the same underlying model:
+
+- **`trellis` npm package** — the published package for engine APIs, subpath exports, and reusable modules.
+- **`trellis` CLI** — the command-line interface for repo setup, history, milestones, semantic diffing, and automation.
+- **VS Code extension** — the visual shell for status, issues, timeline, kanban, and wiki-link navigation.
+- **Core modules** — reusable building blocks for graph storage, semantic analysis, sync, links, embeddings, and decision traces.
+
+The core idea stays the same across all of them:
+
+1. **Causal Stream** — immutable, content-addressed ops linked by causal history.
+2. **Semantic Patching** — AST-aware changes instead of line-only diffs.
+3. **Narrative Milestones** — human checkpoints over a continuous stream of work.
+4. **Governance Subgraph** — signing, identities, and policy enforcement.
+5. **Idea Garden** — abandoned work detection and revival.
+
+## Project Surfaces
+
+| Surface               | Location                                                      | Purpose                                                            |
+| :-------------------- | :------------------------------------------------------------ | :----------------------------------------------------------------- |
+| **npm package**       | `package.json`, `src/`, `dist/`                               | Published as `trellis`; exposes engine APIs and subpath exports    |
+| **CLI**               | `src/cli/index.ts`                                            | Repository lifecycle, branch/milestone workflows, semantic tooling |
+| **VS Code extension** | `vscode-extension/`                                           | Timeline, kanban, issue browser, wiki-link UX                      |
+| **Core store**        | `src/core/`                                                   | Inlined EAV store, kernel persistence, middleware types            |
+| **Platform modules**  | `src/vcs/`, `src/links/`, `src/embeddings/`, `src/decisions/` | Reusable building blocks behind the product surfaces               |
+
+---
 
 ## Quick Start
 
+### Use the npm package
+
+```bash
+bun add trellis
+```
+
+```typescript
+import { TrellisVcsEngine } from 'trellis';
+
+const engine = new TrellisVcsEngine({ rootPath: '/my/project' });
+engine.open();
+```
+
+### Work on the repo locally
+
 ```bash
 # Install dependencies
 bun install
 
-# Run the CLI on bundled sample data
-bun run tql -d data/posts.json -q "FIND post AS ?p RETURN ?p.id, ?p.title"
+# Initialize a new Trellis repo in the current directory
+bun run src/cli/index.ts init
 
-# Run tests
-bun run test
-```
+# Watch for file changes (continuous op stream)
+bun run src/cli/index.ts watch
+
+# Check status
+bun run src/cli/index.ts status
 
-## Key Capabilities
+# Import an existing Git repo
+bun run src/cli/index.ts import --from /path/to/git-repo
+```
 
-- **Semantic Storage** – Durable EAV (Entity-Attribute-Value) store backed by SQLite with in-memory indexing for query performance.
-- **Event Sourcing** – Content-addressed operation log with causality tracking (hashing, agent metadata, linked chains).
-- **Multi-Modal Querying** –
-  - **EQL-S**: A strict SQL-like DSL for entities and relationships.
-  - **Datalog**: Direct access to the semi-naive evaluator for recursive and complex logic.
-  - **Natural Language**: AI-powered translation of human intent into structured queries.
-- **Declarative Workspaces** – Boot entire environments from `.trellis` files containing ontologies (schemas), projections (views), and graph data.
-- **Middleware System** – Async-aware hooks for security (capability-based), schema enforcement, and logic (formulas, rollups).
-- **Time-Travel** – Query the state of your graph at any specific operation hash or historical timestamp.
+---
 
-## Project Structure
+## Architecture Overview
 
 ```
+trellis/
 ├── src/
-│   ├── store/           # Core EAV engine, indexing, and catalog
-│   ├── persist/         # SQLite backend and operation log
-│   ├── query/           # EQL-S parser/compiler and Datalog engine
-│   ├── kernel/          # Kernel API, Middleware, and AI Interop
-│   ├── ai/              # NL providers and orchestrator
-│   ├── graph/           # Agent graph runtime (optional)
-│   └── workflows/       # Workflow engine (optional)
-├── docs/                # Architectural guides and plan
-└── test/                # Comprehensive Vitest suites
+│   ├── core/              # EAV store + kernel types exposed as trellis/core
+│   ├── vcs/               # Ops, branches, milestones, checkpoints, diff, merge
+│   ├── git/               # Git import/export bridge
+│   ├── links/             # Wiki-link parsing, resolution, backlink index
+│   ├── embeddings/        # Semantic indexing and vector search
+│   ├── decisions/         # Decision trace capture and querying
+│   ├── semantic/          # AST parsers, semantic diff, semantic merge
+│   ├── sync/              # Peer sync and reconciler
+│   ├── watcher/           # File watching + ingestion pipeline
+│   ├── cli/               # CLI entrypoint
+│   ├── engine.ts          # Composition root
+│   └── index.ts           # Main package entrypoint
+├── vscode-extension/      # VS Code extension surface
+├── test/                  # Phase and subsystem tests
+├── DESIGN.md              # Detailed architecture spec
+└── justfile               # Local build/release recipes
 ```
 
-## Usage
+### The Op Stream
 
-### Programmatic Example
+Every action in Trellis is an immutable `VcsOp`:
 
 ```typescript
-import { TrellisKernel } from 'trellis/kernel';
-import { SqliteKernelBackend } from 'trellis/kernel/sqlite';
-
-// Initialize with persistence
-const backend = new SqliteKernelBackend({ filename: 'workspace.sqlite' });
-const kernel = new TrellisKernel({ backend });
-
-// Boot with data
-await kernel.boot(
-  [
-    { id: 1, title: 'Build Trellis', status: 'active' },
-    { id: 2, title: 'Release Kernel', status: 'planning' },
-  ],
-  { entityType: 'Task' },
-);
-
-// Query with EQL-S
-const res = await kernel.query(
-  'FIND Task AS ?t WHERE ?t.status = "active" RETURN ?t.title',
-);
-
-console.log(res.rows); // [{ "?t.title": "Build Trellis" }]
-```
-
-### Declarative Workspace (.trellis)
-
-```json
-{
-  "workspace": {
-    "name": "My Projects",
-    "ontologies": {
-      "trellis:schema/project": {
-        /* schema def */
-      }
-    },
-    "graph": {
-      "nodes": [{ "@id": "p1", "@type": "Project", "name": "Web App" }]
-    },
-    "projections": {
-      "active-view": { "name": "Active", "query": "FIND Project AS ?p ..." }
-    }
-  }
+interface VcsOp {
+  hash: string; // trellis:op:<sha256> — content-addressed
+  kind: VcsOpKind; // e.g. 'vcs:fileModify'
+  timestamp: string; // ISO 8601
+  agentId: string; // Author identity (DID)
+  previousHash?: string; // Causal chain link
+  vcs: VcsPayload; // Op-specific data (filePath, contentHash, …)
+  signature?: string; // Ed25519 signature (P4+)
 }
 ```
 
+Ops are written to `.trellis/ops.json` and replayed into an in-memory EAV graph on startup. They are **never rewritten or deleted**.
+
+### Op Tiers
+
+| Tier  | Kinds                                                       | Description                       |
+| :---- | :---------------------------------------------------------- | :-------------------------------- |
+| **0** | `fileAdd`, `fileModify`, `fileDelete`, `fileRename`         | File-level mutations from watcher |
+| **1** | `dirAdd`, `dirDelete`, `branchCreate`, `milestoneCreate`, … | Structural VCS control ops        |
+| **2** | `symbolRename`, `symbolMove`, `symbolExtract`               | AST-level semantic patches        |
+
+---
+
+## CLI Overview
+
+The CLI is the operational surface for Trellis repositories. It is the fastest way to initialize repos, inspect history, create milestones, diff/merge work, and script automation around the op stream.
+
+### Repository Setup
+
+```bash
+trellis init [--path <dir>]            # Initialize a new repo
+trellis import --from <git-repo>       # Import Git history as milestones
+trellis export --to <dir>              # Export milestones to Git commits
+```
+
+### Working
+
+```bash
+trellis status                          # Current branch, op count, pending changes
+trellis log [--limit N] [--branch b]    # Op stream history
+trellis files [--deleted]               # All tracked files
+trellis watch                           # Start continuous file watching
+```
+
+### Branches
+
+```bash
+trellis branch                          # List branches
+trellis branch <name>                   # Create a branch
+trellis branch -d <name>                # Delete a branch
+```
+
+### Milestones & Checkpoints
+
+```bash
+trellis milestone create -m "message"                          # Create milestone
+trellis milestone create -m "msg" --from <hash> --to <hash>   # Over a specific range
+trellis milestone list                                          # List milestones
+trellis checkpoint create                                       # Manual checkpoint
+trellis checkpoint list                                         # List checkpoints
+```
+
+### Diff & Merge
+
+```bash
+trellis diff [--from <hash>] [--to <hash>]  # File-level diff
+trellis merge --branch <name> [--dry-run]   # Three-way merge
+trellis parse <file>                        # Semantic parse (declarations, imports)
+trellis sdiff <fileA> <fileB>               # Semantic diff between two versions
+```
+
+### Identity & Governance
+
+```bash
+trellis identity [show|create|list]         # Manage identities
+trellis governance [list|add|remove]        # Manage policy rules
+```
+
+### Idea Garden
+
+```bash
+trellis garden                                        # List abandoned clusters
+trellis garden list [--status <s>] [--file <f>]      # Filter clusters
+trellis garden show <cluster-id>                      # Inspect a cluster
+trellis garden search --keyword <term>                # Search by keyword
+trellis garden revive <cluster-id>                    # Revive cluster as branch
+trellis garden stats                                  # Garden statistics
+```
+
+### Sync
+
+```bash
+trellis sync status                       # Local sync state
+trellis sync reconcile --remote <path>    # Reconcile with another local repo
+```
+
+---
+
+## VS Code Extension
+
+The VS Code extension is the visual surface for Trellis.
+
+- **Status view** — current branch, op count, tracked files, recent activity
+- **Causal stream** — browse ops as a readable event timeline
+- **Issues view** — inspect and manage issue state directly in the editor
+- **Kanban board** — move work across lifecycle stages visually
+- **Timeline panel** — inspect history as a richer interactive visualization
+- **Wiki-link UX** — click, hover, validate, and autocomplete `[[wiki-links]]`
+
+The extension lives in [`vscode-extension/`](./vscode-extension) and publishes separately from the npm package.
+
+---
+
+## Module & Subpath Guide
+
+The `trellis` package is intentionally split into subpaths so consumers can depend on the surface they actually need.
+
+### Published Package Surface
+
+```bash
+bun add trellis
+```
+
+```typescript
+// Main entry — engine + top-level API
+import { TrellisVcsEngine } from 'trellis';
+
+// Core store and kernel types
+import { EAVStore } from 'trellis/core';
+import type { Fact, Link } from 'trellis/core';
+
+// VCS primitives
+import type { VcsOp, VcsOpKind } from 'trellis/vcs';
+
+// Semantic search
+import { EmbeddingManager } from 'trellis/ai';
+
+// Wiki-linking
+import { parseFileRefs, resolveRef, RefIndex } from 'trellis/links';
+
+// Decision traces
+import { recordDecision, queryDecisions } from 'trellis/decisions';
+```
+
+### `trellis` — Engine Entry Point (`TrellisVcsEngine`)
+
+The main entry point. Ties together the kernel, file watcher, op log, branches, milestones, identity, garden, semantic parser, and sync.
+
+```typescript
+import { TrellisVcsEngine } from 'trellis';
+
+const engine = new TrellisVcsEngine({ rootPath: '/my/project' });
+
+// New repo
+await engine.initRepo();
+
+// Existing repo
+engine.open();
+
+// Core queries
+engine.getOps(); // All ops
+engine.status(); // Branch, op count, files
+engine.log(); // Formatted op history
+engine.getFiles(); // Currently tracked files
+
+// Branches
+await engine.createBranch('feature/x');
+engine.switchBranch('feature/x');
+engine.listBranches();
+await engine.deleteBranch('feature/x');
+
+// Milestones
+await engine.createMilestone('Implement auth', { fromOpHash, toOpHash });
+engine.listMilestones();
+
+// Semantic parsing
+engine.parseFile(content, 'src/auth.ts');
+engine.semanticDiff(oldContent, newContent, 'src/auth.ts');
+
+// Idea Garden
+const garden = engine.garden();
+garden.listClusters();
+garden.search({ keyword: 'auth' });
+garden.revive(clusterId);
+```
+
+### `trellis/core` — Core Store
+
+The inlined core store layer exposes the EAV primitives and kernel-adjacent types that the rest of Trellis is built on.
+
+```typescript
+import { EAVStore } from 'trellis/core';
+import type { Fact, Link, KernelOp } from 'trellis/core';
+
+const store = new EAVStore();
+```
+
+### `trellis/vcs` — VCS Model
+
+The VCS subpath exposes operation types and domain-level building blocks for branches, milestones, checkpoints, issues, diffing, merging, and blob storage.
+
+Use this subpath when you want Trellis domain types and behavior without importing the full engine surface.
+
+### `trellis/links` — Wiki-Link References
+
+Parse `[[wiki-links]]` from markdown, doc-comments, and source files. Resolve references to issues, files, symbols, milestones, and decisions. Build a bidirectional reference index.
+
+```typescript
+import { parseFileRefs, resolveRef, RefIndex } from 'trellis/links';
+
+const refs = parseFileRefs('src/engine.ts', source);
+const resolved = resolveRef(ref, context);
+const index = new RefIndex();
+index.indexFile('README.md', refs, context);
+index.getIncoming('issue:TRL-5');
+```
+
+### `trellis/ai` — Embeddings & Semantic Search
+
+Embed issues, milestones, files, code entities, and decisions into a vector store for semantic search.
+
+```typescript
+import { EmbeddingManager } from 'trellis/ai';
+
+const manager = new EmbeddingManager(engine, { dbPath: 'embeddings.db' });
+await manager.reindex();
+const results = await manager.search('auth flow');
+```
+
+### `trellis/decisions` — Decision Traces
+
+Automatically capture tool invocations as auditable decision records. Enrich them with rationale via hooks and query them later by tool, entity, or chain.
+
+```typescript
+import { recordDecision, queryDecisions } from 'trellis/decisions';
+
+const dec = await recordDecision(ctx, {
+  toolName: 'trellis_issue_create',
+  input: { title: 'Add parser' },
+  output: { id: 'TRL-5' },
+});
+
+const decs = queryDecisions(ctx, { tool: 'trellis_issue_*' });
+```
+
+### `src/garden/` — Idea Garden
+
+Detects abandoned work using three heuristics, then exposes a query API.
+
 ```typescript
-await kernel.boot(trellisConfig);
-const view = await kernel.executeProjection('active-view');
+import { detectClusters, IdeaGarden } from './src/garden/index.js';
+
+const clusters = detectClusters(ops, milestonedHashes);
+const garden = new IdeaGarden(ops, milestones);
+
+garden.search({ file: 'auth.ts', status: 'abandoned', limit: 10 });
+garden.stats(); // { total, abandoned, draft, revived, totalOps, totalFiles }
+garden.revive('cluster:abc');
+```
+
+**Detection heuristics:**
+
+| Heuristic               | Trigger             | Signal                                                            |
+| :---------------------- | :------------------ | :---------------------------------------------------------------- |
+| `contextSwitchDetector` | File-set pivot      | Group of ops in unrelated dirs followed by a context switch away  |
+| `revertDetector`        | Complementary ops   | Ops undone by a subsequent inverse op (add→delete, modify→revert) |
+| `staleBranchDetector`   | Time + no milestone | Ops on non-`main` branches untouched >7 days without a milestone  |
+
+### `src/semantic/` — Semantic Patching
+
+Parses TypeScript/JavaScript into structural entities and computes semantic diffs.
+
+```typescript
+import { typescriptParser, semanticMerge } from './src/semantic/index.js';
+
+// Parse
+const result = typescriptParser.parse(source, 'file.ts');
+result.declarations; // ASTEntity[] — functions, classes, interfaces, enums, …
+result.imports; // ImportRelation[]
+result.exports; // ExportRelation[]
+
+// Diff
+const patches = typescriptParser.diff(oldResult, newResult);
+// SemanticPatch[] — symbolAdd | symbolRemove | symbolModify |
+//                   symbolRename | importAdd | importRemove | …
+
+// Merge (patch commutativity per DESIGN.md §4.4)
+const merged = semanticMerge(ourPatches, theirPatches, 'file.ts');
+merged.clean; // true if no conflicts
+merged.patches; // Merged patch list
+merged.conflicts; // SemanticMergeConflict[] — entity-level, with suggestion
+```
+
+### `src/sync/` — Peer Sync
+
+Implements the have→want→ops→ack protocol with linear and CRDT branch modes.
+
+```typescript
+import { SyncEngine, MemoryTransport, reconcile } from './src/sync/index.js';
+
+// Standalone CRDT reconciler
+const result = reconcile(localOps, remoteOps);
+result.merged; // Interleaved by timestamp
+result.forkPoint; // Last common op hash
+result.conflicts; // File-level conflicts
+
+// Full sync engine
+const transport = new MemoryTransport('peer-a', 'Alice');
+const engine = new SyncEngine({
+  localPeerId: 'peer-a',
+  transport,
+  getLocalOps: () => ops,
+  onOpsReceived: (newOps) => {
+    /* integrate */
+  },
+  branchPolicy: { linear: false }, // CRDT mode
+});
+
+await engine.pushTo('peer-b');
+await engine.pullFrom('peer-b');
+engine.reconcileWith(remoteOps);
+```
+
+These `src/*` sections are internal subsystem guides rather than published npm subpaths. They are useful if you are contributing to Trellis itself or navigating the codebase.
+
+---
+
+## How It Works
+
+### Init Flow
+
+```
+trellis init
+  → mkdirSync .trellis/
+  → write config.json (agentId, defaultBranch, ignorePatterns)
+  → create vcs:branchCreate op for "main"
+  → scan filesystem → create vcs:fileAdd op per file
+  → flush ops to .trellis/ops.json
+```
+
+### Watch Flow
+
+```
+trellis watch
+  → FileWatcher.scan() populates known-files map
+  → FileWatcher.start() sets up Bun fs.watch
+  → on change → debounce → SHA-256 hash → emit FileChangeEvent
+  → Ingestion.process(event) → createVcsOp(kind, payload)
+  → engine.applyOp(op) → EAV store + op log
+  → auto-checkpoint if threshold crossed
+```
+
+### Milestone Flow
+
+```
+trellis milestone create -m "Add auth"
+  → auto-detect fromOpHash (last milestone's toOpHash + 1)
+  → collect affected files from ops in range
+  → createVcsOp('vcs:milestoneCreate', { message, fromOpHash, toOpHash, affectedFiles })
+  → optional: trellis export → Git commit
+```
+
+### Semantic Diff Flow
+
+```
+trellis sdiff old.ts new.ts
+  → typescriptParser.parse(old) → ParseResult { declarations, imports, exports }
+  → typescriptParser.parse(new) → ParseResult
+  → typescriptParser.diff(old, new) → SemanticPatch[]
+    - symbolAdd / symbolRemove / symbolModify
+    - symbolRename (detected via signature similarity)
+    - importAdd / importRemove / importModify
+```
+
+### Sync / Reconcile Flow
+
+```
+Engine A sends 'have' { heads: { main: 'h42' }, opCount: 42 }
+  → Engine B compares to its own heads
+  → B sends 'want' { afterHash: 'h38' }
+  → A sends 'ops' [ op39, op40, op41, op42 ]
+  → B integrates new ops (linear: filter dupes; CRDT: reconcile())
+  → B sends 'ack' { integrated: ['h39', …] }
+```
+
+---
+
+## Design Doc
+
+See [`DESIGN.md`](./DESIGN.md) for the full architecture specification, including:
+
+- §3 — Causal stream and branch concurrency model
+- §4 — Semantic patching: parser adapter interface, patch types, commutativity table
+- §5 — Milestone narrative model
+- §6 — Identity, signing, and governance
+- §7 — Idea Garden cluster detection heuristics
+- §10 — Open questions and architectural trade-offs
+
+---
+
+## Development & Releases
+
+### Prerequisites
+
+- [Bun](https://bun.sh) ≥ 1.0
+- No other native dependencies
+
+### Install
+
+```bash
+bun install
 ```
 
-## Documentation
+### Test
+
+```bash
+just test-core
+# or run the full suite when working on those areas:
+bun test
+```
 
-- [Kernel Alignment Plan](docs/KERNEL-ALIGNMENT-PLAN.md) – The roadmap followed for the current architecture.
-- [EAV Engine Guide](docs/EAV-README.md) – Deep dive into the storage layer.
-- [Workflows & Agent Graphs](docs/WORKFLOWS.md) – High-level automation.
+**The release flow currently validates against the targeted passing core suite.**
 
-## Development
+### Typecheck
 
 ```bash
-bun run typecheck  # Type checking
-bun run test       # Run Vitest suites
-bun run clean      # Clean build artifacts
+bun run typecheck
 ```
 
-Built with [Bun](https://bun.sh) for fast TypeScript execution.
+### Build
+
+```bash
+# Build for npm (bun build → dist/)
+bun run build
+
+# Run CLI directly during development
+bun run src/cli/index.ts <command>
+```
+
+---
+
+### Release Workflow
+
+```bash
+# Validate locally without publishing
+just publish-dry-run
+
+# Publish to npm locally, then tag/push/create release metadata
+just publish
+```
+
+> **Requires [Bun](https://bun.sh) ≥ 1.0** — Trellis uses `bun:sqlite` for the vector store and Bun's native TS support.
+
+---
+
+## Roadmap
+
+| Phase | Deliverable                                     | Status | Commit    |
+| :---- | :---------------------------------------------- | :----- | :-------- |
+| P0    | Causal stream + CLI                             | ✅     | `51475d3` |
+| P0.5  | VS Code extension / visual timeline             | ✅     | `947d5a1` |
+| P1    | Git import bridge                               | ✅     | `f4cc4a6` |
+| P2    | Branches, milestones, checkpoints               | ✅     | `3f91e9a` |
+| P2.5  | Blob store, engine modularization, git exporter | ✅     | `5c43a31` |
+| P3    | File-level diff + text-based merge              | ✅     | `c953654` |
+| P4    | Identity + op signing + governance              | ✅     | `3acddda` |
+| P5    | Idea Garden — cluster detection + query         | ✅     | `105a207` |
+| P6    | Semantic patching — parser adapter + diff/merge | ✅     | `22192ae` |
+| P7    | Peer sync + CRDT reconciler                     | ✅     | `d02f3f7` |
+| P8    | Wiki-links, embeddings, decision traces         | ✅     | `b9cd5b7` |
+| P9    | npm package + VSCode extension publish          | 🚧     |           |

package/bin/trellis.mjs

@@ -0,0 +1,2 @@
+#!/usr/bin/env bun
+import '../dist/cli/index.js';