compend 0.0.1 → 1.0.0

package/CHANGELOG.md

@@ -0,0 +1,34 @@
+# Changelog
+
+All notable changes to Compend will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.0.0] - 2026-06-27
+
+### Added
+- Initial release
+- 5 MCP tools: `compend_index`, `compend_deindex`, `compend_search`, `compend_get`, `compend_list`
+- `compend_index` mirrors filesystem into index — accepts file path, directory path, or no args (all configured paths)
+- `compend_deindex` removes concepts by slug or path prefix (files never touched)
+- `compend_search` with hybrid FTS5 + vec0 weighted scoring (MurmurHash3 256-dim, alpha 0-1)
+- `compend_get` returns full concept: frontmatter JSON, markdown body, children (references), and dependencies
+- `compend_list` browse with type, tag, and status filters + pagination
+- Dashboard on port 3457: read-only, dark/light theme, SSE real-time updates, type/tag/status filters, rendered markdown viewer
+- CLI: `compend` (start dashboard), `compend stop`, `compend restart`
+- 7 extensible concept types: skill, agent, instruction, prompt, workflow, reference, knowledge
+- Type schemas in config.js with per-type status validation (matching Hemisphere's kind schema pattern)
+- OKF frontmatter parsing (YAML + markdown body) with automatic type inference
+- Auto-index on first run from opencode.json `skills.paths` + `instructions`
+- Parent/child concept resolution via slug hierarchy (file path convention)
+- Dependency resolution from OKF frontmatter `dependencies` field
+- `~/.compend/config.json` config with env var overrides (`COMPEND_PORT`, `COMPEND_DB_PATH`)
+- Index paths from opencode.json auto-discovery + `~/.compend/config.json` overrides
+
+## [0.0.1] - 2026-06-27
+
+### Added
+- Placeholder release to secure npm namespace
+- Logo SVG (48x48 book icon, currentColor)
+- GPL-3.0-only license

package/README.md

@@ -2,8 +2,383 @@
 
 # Compend
 
+[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0) [![Node](https://img.shields.io/badge/node-%3E%3D18-brightgreen)](https://nodejs.org)
+
+**Author:** [Hector Jarquin](https://hectorjarquin.com)
+
 A reference engine for AI agents. Index your Markdown skills, prompts, and knowledge bases — and serve them as just-in-time grounding with hybrid search and dynamic context. No retraining. No keys. No Python.
 
-## Coming soon
+## Quick Start
+
+```bash
+npm install -g compend
+```
+
+Once installed, run the dashboard:
+
+```bash
+compend
+# → http://localhost:3457
+```
+
+### CLI commands
+
+```bash
+compend              # Start the dashboard
+compend stop         # Stop a running instance
+compend restart      # Stop then restart
+```
+
+### MCP server
+
+Add to `opencode.json`:
+
+```json
+{
+  "mcp": {
+    "compend": {
+      "type": "local",
+      "command": ["node", "/usr/lib/node_modules/compend/index.js"],
+      "enabled": true
+    }
+  }
+}
+```
+
+Find your exact path with `npm root -g` — append `/compend/index.js`. For nvm users the path is typically `~/.nvm/versions/node/vX/lib/node_modules/compend/index.js`.
+
+### Dashboard
+
+Browse, search and read knowledge concepts visually at `http://localhost:3457`.
+
+Port configurable via `~/.compend/config.json` or `COMPEND_PORT`.
+
+**Features:** dark/light theme toggle with SVG icon, WCAG 2.1 AA accessibility (keyboard navigation, ARIA labels, focus-visible outlines), real-time SSE updates (no polling), toast notifications, skeleton loading, search with debounce, type, tag, and status filters, expandable detail rows with rendered markdown body.
+
+## Installation
+
+### Prerequisites
+
+- Node.js 18+
+- C++ build tools (`build-essential` on Debian/Ubuntu, Xcode CLI tools on macOS) — required to compile `better-sqlite3`
+
+### From npm (recommended)
+
+```bash
+npm install -g compend
+```
+
+The `compend` CLI is now available globally. The MCP server runs via your AI client — see [Configuration](#configuration).
+
+### From git
+
+```bash
+git clone https://github.com/hectorjarquin/compend.git ~/compend
+cd ~/compend
+npm install
+npm link  # creates global compend command
+```
+
+### Configuration
+
+```json
+{
+  "mcp": {
+    "compend": {
+      "type": "local",
+      "command": ["node", "/usr/lib/node_modules/compend/index.js"],
+      "enabled": true
+    }
+  }
+}
+```
+
+Find your exact path with `npm root -g` — append `/compend/index.js`.
+
+Concepts are discovered automatically from your opencode.json `skills.paths` and `instructions` directories. Add custom project knowledge bundles via `~/.compend/config.json`.
+
+## Updating
+
+### npm
+
+```bash
+npm install -g compend@latest
+```
+
+Then restart your MCP client to pick up the new tools, and run `compend restart` to refresh the dashboard.
+
+Database migrations run automatically on first launch — no manual steps required.
+
+### From git
+
+```bash
+cd ~/compend
+git pull
+npm install
+npm link
+compend restart
+```
+
+## MCP Tools (Agent-Facing)
+
+### `compend_index`
+
+Mirror the filesystem source of truth into the index. No args scans all configured paths. Pass `{ path }` to index a single `.md` file or a directory (recursively). Uses SHA-256 hash diffing — unchanged files are skipped. Files missing from disk are removed from the index.
+
+| Parameter | Type | Required | Default | Description |
+|-----------|------|----------|---------|-------------|
+| `path` | string | no | — | Absolute path to a `.md` file or directory to index. Omit to scan all configured paths. |
+
+Returns `{ added: [...slugs], updated: [...slugs], removed: [...slugs], total: N }`.
+
+### `compend_deindex`
+
+Remove concepts from the index. Pass `{ slug }` to remove one concept, or `{ path }` to remove all concepts under a directory path. Files on disk are never touched — deindex is index-only. If the file still exists, the next `compend_index` will re-index it.
+
+| Parameter | Type | Required | Default | Description |
+|-----------|------|----------|---------|-------------|
+| `slug` | string | no | — | Concept slug to remove |
+| `path` | string | no | — | Directory or file path — all concepts whose `file_path` starts with this are removed |
+
+Returns `{ removed: [...slugs], total: N }`.
+
+### `compend_search`
+
+Hybrid FTS + vector search across indexed concepts. Returns metadata with snippet and relevance score.
+
+| Parameter | Type | Required | Default | Description |
+|-----------|------|----------|---------|-------------|
+| `query` | string | yes | — | Search query text |
+| `type` | string | no | — | Filter by concept type (skill, agent, instruction, etc.) |
+| `tags` | string[] | no | — | Filter by tags (AND match) |
+| `limit` | number | no | `10` | Max results |
+| `alpha` | number | no | `0.3` | Vector weight. `0` = FTS-only, `1` = vector-only |
+
+Returns concepts sorted by relevance (score 0–1) with `id`, `slug`, `type`, `title`, `description`, `tags`, `status`, `source`, `score`, and `snippet`.
+
+### `compend_get`
+
+Retrieve a full concept by slug. Includes frontmatter JSON, markdown body, child references (concepts whose slug starts with `{slug}/`), and dependencies (from the OKF `dependencies` frontmatter field).
+
+| Parameter | Type | Required | Default | Description |
+|-----------|------|----------|---------|-------------|
+| `slug` | string | yes | — | Concept slug (e.g. `"wp-image-to-blocks"`) |
+
+Returns `{ id, slug, type, title, description, tags, status, frontmatter, body, references: [...], dependencies: [...] }`.
+
+### `compend_list`
+
+List concepts with optional filters. Returns compact metadata — no body.
+
+| Parameter | Type | Required | Default | Description |
+|-----------|------|----------|---------|-------------|
+| `type` | string | no | — | Filter by concept type |
+| `tags` | string[] | no | — | Filter by tags (AND match) |
+| `status` | string | no | — | Filter by status (stable, draft, deprecated) |
+| `limit` | number | no | `50` | Max results |
+| `offset` | number | no | `0` | Pagination offset |
+
+Returns `{ concepts: [...], total, limit, offset }`.
+
+## HTTP API (Dashboard-Facing)
+
+The dashboard exposes REST endpoints:
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/api/stats` | Get total concept count and counts by type |
+| `GET` | `/api/tags?type=` | List tags with concept counts, optional type filter |
+| `GET` | `/api/concepts?type=&status=&tags=&search=&limit=&offset=` | Paginated concept list. `search=` triggers FTS+vector. |
+| `GET` | `/api/concepts/{slug}` | Full concept including frontmatter, body, references, and dependencies |
+| `POST` | `/api/notify` | SSE event relay from MCP server (internal — called by `notifyDash()`) |
+| `GET` | `/api/events` | SSE (Server-Sent Events) stream for real-time dashboard updates |
+
+## Concept Types
+
+Seven types are built into the default schema, extensible via `~/.compend/config.json`:
+
+| Type | Statuses | Default | Description |
+|------|----------|---------|-------------|
+| `skill` | `stable`, `draft`, `deprecated` | `stable` | Agent skill definition (SKILL.md) |
+| `agent` | `stable`, `draft`, `deprecated` | `stable` | Agent/subagent definition |
+| `instruction` | `stable`, `draft`, `deprecated` | `stable` | Instruction files injected into system prompt |
+| `prompt` | `stable`, `draft`, `deprecated` | `stable` | Prompt templates |
+| `workflow` | `stable`, `draft`, `deprecated` | `stable` | Multi-step pipeline definitions |
+| `reference` | — (no validation) | — | Reference documents, examples, templates |
+| `knowledge` | — (no validation) | — | Project domain knowledge (OKF bundles) |
+
+Type is inferred from OKF frontmatter, file path, or directory location. Unknown types pass through without validation — existing data is never affected. Custom types can be added via `~/.compend/config.json`:
+
+```json
+{
+  "schemas": {
+    "default": {
+      "types": {
+        "template": { "statuses": ["draft","published","retired"], "defaults": { "status": "draft" } }
+      }
+    }
+  }
+}
+```
+
+## Configuration
+
+### Config File (`~/.compend/config.json`)
+
+Create an optional JSON config file to customize operational settings. All keys are optional — missing keys use the code defaults.
+
+**Priority chain** (highest wins): code defaults < config file < environment variables.
+
+```json
+{
+  "port": 3457,
+  "dbPath": "~/.compend/concepts.db",
+  "search": {
+    "limit": 10,
+    "alpha": 0.3
+  },
+  "dashboard": {
+    "paginationLimit": 50,
+    "maxLimit": 200
+  },
+  "schemas": {
+    "default": {
+      "types": {}
+    }
+  },
+  "index": {
+    "paths": []
+  }
+}
+```
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `port` | number | `3457` | Dashboard HTTP server port |
+| `dbPath` | string | `~/.compend/concepts.db` | SQLite database file path (supports `~`) |
+| `search.limit` | number | `10` | Default search result count |
+| `search.alpha` | number | `0.3` | Vector weight in hybrid search (0–1) |
+| `dashboard.paginationLimit` | number | `50` | Default page size for dashboard API |
+| `dashboard.maxLimit` | number | `200` | Hard cap on API page size |
+| `schemas.default.types` | object | built-in set | Type definitions with `statuses` and `defaults` |
+| `index.paths` | string[] | `[]` | Additional paths to scan for `.md` files (appended to opencode auto-discovered paths) |
+
+### Environment Variables
+
+| Variable | Equivalent Config Key | Default |
+|----------|----------------------|---------|
+| `COMPEND_PORT` | `port` | `3457` |
+| `COMPEND_DB_PATH` | `dbPath` | `~/.compend/concepts.db` |
+
+### Path Discovery
+
+Compend auto-discovers filesystem paths from your opencode.json:
+
+1. `skills.paths` — skill directories (scanned recursively for `*.md`)
+2. `instructions` — instruction files (indexed directly)
+
+Additional paths can be added via `index.paths` in `~/.compend/config.json`. This is where you add project OKF knowledge bundles:
+
+```json
+{
+  "index": {
+    "paths": ["/home/user/projects/my-project/knowledge"]
+  }
+}
+```
+
+## Architecture
+
+### Embedding
+
+MurmurHash3 (32-bit x86) applied to word unigrams and bigrams, accumulated into a 256-bin `Float32Array`, then L2-normalized. Runs in ~2µs per text with no external dependencies. Collisions are inherent to feature hashing and don't materially affect retrieval quality at this dimension count.
+
+### Hybrid Search
+
+Three indexing layers work together:
+
+1. **FTS5** — SQLite full-text search with BM25 ranking (`unicode61` tokenizer)
+2. **Vector** — `sqlite-vec` virtual table with 256-dim float vectors, cosine distance
+3. **Merge** — Weighted combination: `score = alpha * vec + (1 - alpha) * fts`. Both sides are normalized to 0–1 before merging.
+
+### Database
+
+- WAL journal mode for concurrent reads, `busy_timeout=5000ms` for write-contention safety across processes
+- `concepts` table (slug, type, title, description, tags, status, frontmatter, body, file_path, file_hash, source, created_at, updated_at, last_synced_at)
+- `concepts_fts` — FTS5 external content table on title, description, tags, body
+- `concepts_vec` — `vec0` table with `float[256]`
+
+### Index vs Source of Truth
+
+Compend does not own content. It mirrors a source of truth — the local filesystem or a remote CMS (`compend_sync` in v2). There is no CRUD lifecycle (no create/edit/delete of concepts from the index). The source is always authoritative. The index is a mirror — always rebuildable from source.
+
+`compend_index` uses SHA-256 hash diffing: unchanged files are skipped, changed files are updated, missing files are removed from the index. `compend_deindex` removes concepts without touching files — the next index run restores them.
+
+### SSE Real-Time Updates
+
+The dashboard uses Server-Sent Events for live updates — no polling. Events flow through a three-hop chain:
+
+MCP server (index.js) → `notifyDash()` → dashboard `/api/notify` → `broadcast()` → SSE clients → `app.js` listeners
+
+| Event | Source | Frontend Behavior |
+|---|---|---|
+| `index_complete` | `compend_index`, `compend_deindex` | Toast with count summary + reload list |
+
+If the SSE connection drops, a 30-second fallback poll resumes.
+
+## Project Structure
+
+```
+compend/
+├── index.js                MCP stdio server (5 tool handlers)
+├── dashboard.js            HTTP dashboard + SSE broadcast server
+├── dashboard/
+│   ├── api-handler.js      Dashboard REST API routes
+│   └── public/
+│       ├── index.html      Dashboard HTML + ARIA structure
+│       ├── style.css       Full theme (dark/light), toast, skeleton
+│       ├── app.js          Frontend SSE client, keyboard nav, WCAG 2.1 AA
+│       └── logo.svg        Book icon (48x48, currentColor)
+├── db.js                   SQLite init, CRUD, hybrid FTS+vec search
+├── embedding.js            MurmurHash3 → 256-dim float vector
+├── config.js               Config loader (defaults ← config.json ← env vars)
+├── package.json
+├── CHANGELOG.md
+└── README.md
+```
+
+## Development
+
+### Scripts
+
+For local development (after `git clone`):
+
+```bash
+npm start           # Start dashboard server (same as compend)
+npm run stop        # Stop running instance (same as compend stop)
+npm run restart     # Stop then start
+```
+
+For installed users, the global `compend` CLI handles these — see [Quick Start](#quick-start).
+
+### Testing the MCP server
+
+Test with the MCP Inspector:
+
+```bash
+npx @modelcontextprotocol/inspector node index.js
+```
+
+## Roadmap
+
+- **Remote sync** — `compend_sync` for REST-based reconciliation with distributed teams
+- **Import / export** — portable OKF bundle archives across Compend instances
+- **Auth / access control** — API keys for multi-client deployments
+
+## Contributing
+
+Open an issue or PR at [github.com/hectorjarquin/compend](https://github.com/hectorjarquin/compend).
+
+## License
 
-Compend is an MCP server that indexes OKF-formatted knowledge documents — skills, agents, instructions, prompts, workflows, and references — with FTS + vector search and REST-based reconciliation for distributed teams.
+GNU General Public License v3.0 or later — see [LICENSE](LICENSE) for details.

package/config.js

@@ -0,0 +1,147 @@
+import { homedir } from 'node:os';
+import { join, dirname } from 'node:path';
+import { existsSync, mkdirSync, readFileSync } from 'node:fs';
+
+const DEFAULTS = {
+  port: 3457,
+  dbPath: join(homedir(), '.compend', 'concepts.db'),
+  search: {
+    limit: 10,
+    alpha: 0.3
+  },
+  dashboard: {
+    paginationLimit: 50,
+    maxLimit: 200
+  },
+  schemas: {
+    default: {
+      types: {
+        skill:       { statuses: ['stable','draft','deprecated'], defaults: { status: 'stable' } },
+        agent:       { statuses: ['stable','draft','deprecated'], defaults: { status: 'stable' } },
+        instruction: { statuses: ['stable','draft','deprecated'], defaults: { status: 'stable' } },
+        prompt:      { statuses: ['stable','draft','deprecated'], defaults: { status: 'stable' } },
+        workflow:    { statuses: ['stable','draft','deprecated'], defaults: { status: 'stable' } },
+        reference:   { statuses: [], defaults: {} },
+        knowledge:   { statuses: [], defaults: {} }
+      }
+    }
+  }
+};
+
+const isObject = (v) => v !== null && typeof v === 'object' && !Array.isArray(v);
+
+function deepMerge(target, source) {
+  for (const key of Object.keys(source)) {
+    if (!Object.hasOwn(target, key)) {
+      target[key] = JSON.parse(JSON.stringify(source[key]));
+      continue;
+    }
+    const tv = target[key];
+    const sv = source[key];
+    if (isObject(tv) && isObject(sv)) {
+      deepMerge(tv, sv);
+    } else {
+      target[key] = sv;
+    }
+  }
+  return target;
+}
+
+function readConfigFile() {
+  try {
+    const filePath = join(homedir(), '.compend', 'config.json');
+    if (!existsSync(filePath)) return {};
+    const raw = readFileSync(filePath, 'utf-8').trim();
+    if (!raw) return {};
+    return JSON.parse(raw);
+  } catch (e) {
+    console.warn('Compend: config.json parse error — using defaults:', e.message);
+    return {};
+  }
+}
+
+function resolveTilde(p) {
+  if (typeof p === 'string' && p.startsWith('~')) {
+    return join(homedir(), p.slice(1));
+  }
+  return p;
+}
+
+function applyEnvOverrides(cfg) {
+  if (process.env.COMPEND_PORT && process.env.COMPEND_PORT.trim()) {
+    cfg.port = parseInt(process.env.COMPEND_PORT.trim(), 10) || cfg.port;
+  }
+  if (process.env.COMPEND_DB_PATH && process.env.COMPEND_DB_PATH.trim()) {
+    cfg.dbPath = resolveTilde(process.env.COMPEND_DB_PATH.trim());
+  }
+}
+
+let _cfg = null;
+
+export function getConfig() {
+  if (_cfg) return _cfg;
+  const defaults = JSON.parse(JSON.stringify(DEFAULTS));
+  _cfg = deepMerge(defaults, readConfigFile());
+  applyEnvOverrides(_cfg);
+  _cfg.dbPath = resolveTilde(_cfg.dbPath);
+  return _cfg;
+}
+
+export function getDbPath() {
+  const path = getConfig().dbPath;
+  const dir = dirname(path);
+  if (!existsSync(dir)) {
+    mkdirSync(dir, { recursive: true });
+  }
+  return path;
+}
+
+export function resolveTypeSchema(type) {
+  const schemas = getConfig().schemas || {};
+  const projectSchema = schemas.default || {};
+  const types = projectSchema.types || {};
+  return types[type] || null;
+}
+
+export function getIndexPaths() {
+  const paths = [];
+
+  try {
+    const opencodePaths = [
+      join(homedir(), '.config', 'opencode', 'opencode.json'),
+    ];
+    for (const p of opencodePaths) {
+      if (existsSync(p)) {
+        const raw = readFileSync(p, 'utf-8').trim();
+        if (raw) {
+          const cfg = JSON.parse(raw);
+          if (cfg.skills && Array.isArray(cfg.skills.paths)) {
+            for (const sp of cfg.skills.paths) {
+              paths.push(resolveTilde(sp));
+            }
+          }
+          if (cfg.instructions && Array.isArray(cfg.instructions)) {
+            for (const ip of cfg.instructions) {
+              paths.push(resolveTilde(ip));
+            }
+          }
+        }
+      }
+    }
+  } catch (e) {
+    console.warn('Compend: could not read opencode.json:', e.message);
+  }
+
+  try {
+    const compendCfg = readConfigFile();
+    if (compendCfg.index && Array.isArray(compendCfg.index.paths)) {
+      for (const p of compendCfg.index.paths) {
+        paths.push(resolveTilde(p));
+      }
+    }
+  } catch (e) {
+    console.warn('Compend: could not read compend config paths:', e.message);
+  }
+
+  return paths;
+}

package/dashboard/api-handler.js

@@ -0,0 +1,77 @@
+import { searchHybrid, getConcept, listConcepts, getTags, getChanges, getDeps } from '../db.js';
+import { getConfig } from '../config.js';
+
+export function json(data, status = 200) {
+  return { status, headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(data) };
+}
+
+export function err(msg, status = 500) {
+  return json({ error: msg }, status);
+}
+
+export function createApiHandler(db) {
+  return function handleApi(path, method, params) {
+    if (path === '/api/stats' && method === 'GET') {
+      const total = db.prepare('SELECT COUNT(*) as c FROM concepts').get().c;
+      const types = db.prepare('SELECT type, COUNT(*) as c FROM concepts GROUP BY type ORDER BY c DESC').all().map(r => ({ type: r.type, count: r.c }));
+      return json({ total, types });
+    }
+
+    if (path === '/api/tags' && method === 'GET') {
+      const type = params.get('type') || '';
+      const results = getTags(type || undefined);
+      return json({ tags: results });
+    }
+
+    if (path === '/api/concepts' && method === 'GET') {
+      const type = params.get('type') || '';
+      const status = params.get('status') || '';
+      const search = params.get('search') || '';
+      const tagsRaw = params.get('tags') || '';
+      const maxLimit = getConfig().dashboard.maxLimit;
+      const limit = Math.min(parseInt(params.get('limit') || String(getConfig().dashboard.paginationLimit), 10), maxLimit);
+      const offset = Math.max(parseInt(params.get('offset') || '0', 10), 0);
+
+      if (search.trim()) {
+        try {
+          const rows = searchHybrid({ query: search, type: type || undefined, limit, alpha: getConfig().search.alpha });
+          return json({ concepts: rows, total: rows.length, limit, offset });
+        } catch (e) {
+          return err('Search error', 400);
+        }
+      }
+
+      const result = listConcepts({
+        type: type || undefined,
+        status: status || undefined,
+        tags: tagsRaw ? tagsRaw.split(',').filter(Boolean) : undefined,
+        limit,
+        offset
+      });
+      return json(result);
+    }
+
+    const conceptMatch = path.match(/^\/api\/concepts\/(.+)$/);
+    if (conceptMatch && method === 'GET') {
+      const slug = decodeURIComponent(conceptMatch[1]);
+      const concept = getConcept(slug);
+      if (!concept) return err('Concept not found', 404);
+      return json(concept);
+    }
+
+    if (path === '/api/changes' && method === 'GET') {
+      const since = params.get('since') || '0';
+      const results = getChanges(parseInt(since, 10) || 0);
+      return json({ changes: results });
+    }
+
+    const depsMatch = path.match(/^\/api\/deps\/(.+)$/);
+    if (depsMatch && method === 'GET') {
+      const slug = decodeURIComponent(depsMatch[1]);
+      const results = getDeps(slug);
+      return json(results);
+    }
+
+    return null;
+  };
+}