harper-knowledge 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Nathan Heskew
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,276 @@
+# harper-knowledge
+
+Knowledge base for [Harper](https://harper.fast/), built on Harper, with MCP server integration.
+
+A Harper sub-component plugin that provides searchable, scoped knowledge entries with vector embeddings for semantic search. Exposes a REST API, MCP endpoint, and web UI.
+
+## Consumers
+
+- **Support team** — finding solutions, patterns, gotchas, customer edge cases
+- **DX lab "Harper expert"** — backing knowledge for the AI expert role in Gas Town labs
+- **Claude Code / IDE assistants** — Harper context via MCP without per-project CLAUDE.md files
+- **Any MCP client** — Cursor, VS Code + Copilot, JetBrains, ChatGPT, Gemini, etc.
+
+## Quick Start
+
+### Prerequisites
+
+- [Harper](https://harper.fast/) >= 4.7.0
+- Node.js >= 22
+
+### Install
+
+```bash
+npm install harper-knowledge
+```
+
+### Configure
+
+Add to your parent application's `config.yaml`:
+
+```yaml
+"harper-knowledge":
+  package: "harper-knowledge"
+  embeddingModel: nomic-embed-text # default
+```
+
+### Download the Embedding Model
+
+The plugin uses [nomic-embed-text](https://huggingface.co/nomic-ai) for vector embeddings, run locally via [node-llama-cpp](https://github.com/withcatai/node-llama-cpp). The model is downloaded to `~/hdb/models/` on first startup, but you can pre-download it:
+
+```bash
+# Download the default model
+npm run model:download
+
+# Download and verify with a test embedding
+npm run model:test
+```
+
+### Run
+
+```bash
+harperdb dev .
+```
+
+## Embedding Models
+
+Two [Nomic](https://huggingface.co/nomic-ai) embedding models are supported, both run entirely on CPU with no cloud dependency via [node-llama-cpp](https://github.com/withcatai/node-llama-cpp). On first plugin startup (or `npm run model:download`), the configured model is downloaded from Hugging Face to `~/hdb/models/`. A file lock prevents multiple Harper worker threads from downloading simultaneously.
+
+### nomic-embed-text v1.5 (default)
+
+|                  |                                                                                                   |
+| ---------------- | ------------------------------------------------------------------------------------------------- |
+| **Model**        | [nomic-ai/nomic-embed-text-v1.5-GGUF](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF) |
+| **Config key**   | `nomic-embed-text`                                                                                |
+| **Parameters**   | 137M                                                                                              |
+| **Dimensions**   | 768                                                                                               |
+| **Quantization** | Q4_K_M (~135 MB)                                                                                  |
+| **Context**      | 8192 tokens                                                                                       |
+| **License**      | Apache 2.0                                                                                        |
+
+### nomic-embed-text v2 MoE
+
+|                  |                                                                                                       |
+| ---------------- | ----------------------------------------------------------------------------------------------------- |
+| **Model**        | [nomic-ai/nomic-embed-text-v2-moe-GGUF](https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe-GGUF) |
+| **Config key**   | `nomic-embed-text-v2-moe`                                                                             |
+| **Parameters**   | 475M (Mixture of Experts)                                                                             |
+| **Dimensions**   | 768                                                                                                   |
+| **Quantization** | Q4_K_M                                                                                                |
+| **Context**      | 8192 tokens                                                                                           |
+| **License**      | Apache 2.0                                                                                            |
+
+The v2 MoE model is larger but produces higher-quality embeddings, especially for longer and more nuanced content.
+
+### Switching models
+
+```yaml
+"harper-knowledge":
+  package: "harper-knowledge"
+  embeddingModel: nomic-embed-text # v1.5 (default)
+  # embeddingModel: nomic-embed-text-v2-moe # v2 MoE
+```
+
+## Architecture
+
+```
+harper-knowledge
+├── src/
+│   ├── index.ts              ← plugin entry: handleApplication()
+│   ├── core/                 ← shared logic
+│   │   ├── embeddings.ts     ← model download, init, vector generation
+│   │   ├── entries.ts        ← CRUD + relationship management
+│   │   ├── history.ts        ← edit history audit log
+│   │   ├── search.ts         ← keyword / semantic / hybrid search
+│   │   ├── tags.ts           ← tag registry with counts
+│   │   └── triage.ts         ← webhook intake queue
+│   ├── resources/            ← REST Resource classes
+│   │   ├── KnowledgeEntryResource.ts
+│   │   ├── TriageResource.ts
+│   │   ├── TagResource.ts
+│   │   ├── QueryLogResource.ts
+│   │   ├── ServiceKeyResource.ts
+│   │   └── HistoryResource.ts
+│   ├── mcp/                  ← MCP server (Streamable HTTP)
+│   │   ├── server.ts
+│   │   └── tools.ts
+│   ├── oauth/                ← OAuth 2.1 authorization server
+│   │   ├── authorize.ts
+│   │   ├── keys.ts
+│   │   ├── metadata.ts
+│   │   ├── middleware.ts
+│   │   ├── register.ts
+│   │   ├── token.ts
+│   │   └── validate.ts
+│   ├── webhooks/             ← webhook intake (GitHub, Datadog)
+│   │   ├── middleware.ts
+│   │   ├── github.ts
+│   │   └── datadog.ts
+│   └── types.ts
+├── schema/
+│   └── knowledge.graphql     ← table definitions (database: "kb")
+├── web/                      ← static web UI
+├── scripts/
+│   └── download-model.js     ← standalone model download/test
+├── config.yaml
+├── package.json
+└── test/
+```
+
+Both REST and MCP run in the Harper process, both call the same core functions with zero overhead.
+
+## REST API
+
+| Endpoint                | Method          | Auth       | Description               |
+| ----------------------- | --------------- | ---------- | ------------------------- |
+| `/Knowledge/<id>`       | GET             | Public     | Get entry by ID           |
+| `/Knowledge/?query=...` | GET             | Public     | Search entries            |
+| `/Knowledge/`           | POST            | Required   | Create entry              |
+| `/Knowledge/<id>`       | PUT             | Required   | Update entry              |
+| `/Knowledge/<id>`       | DELETE          | Team       | Deprecate entry           |
+| `/KnowledgeTag/`        | GET             | Public     | List all tags             |
+| `/Triage/`              | GET             | Team       | List pending triage items |
+| `/Triage/`              | POST            | Service/AI | Submit triage item        |
+| `/Triage/<id>`          | PUT             | Team       | Process triage item       |
+| `/QueryLog/`            | GET             | Team       | Search analytics          |
+| `/ServiceKey/`          | GET/POST/DELETE | Team       | API key management        |
+| `/History/<entryId>`    | GET             | Public     | Edit history for an entry |
+
+### Search Parameters
+
+```
+GET /Knowledge/?query=MQTT+auth&tags=mqtt,config&limit=10&mode=keyword&context={"harper":"5.0","storageEngine":"lmdb"}
+```
+
+- `query` — search text (required)
+- `tags` — comma-separated tag filter
+- `limit` — max results (default 10)
+- `mode` — `keyword`, `semantic`, or `hybrid` (default)
+- `context` — JSON applicability context for result boosting
+
+## MCP Endpoint
+
+Connect any MCP-compatible client to `/mcp`:
+
+```json
+{
+  "mcpServers": {
+    "harper-kb": {
+      "url": "https://kb.harper.fast:9926/mcp"
+    }
+  }
+}
+```
+
+### Tools
+
+| Tool                  | Description                                                       |
+| --------------------- | ----------------------------------------------------------------- |
+| `knowledge_search`    | Search with keyword/semantic/hybrid modes + applicability context |
+| `knowledge_add`       | Add a new entry (auto-tagged `ai-generated`)                      |
+| `knowledge_get`       | Get entry by ID with full relationship chain                      |
+| `knowledge_update`    | Update an entry with edit history tracking                        |
+| `knowledge_related`   | Find related entries (explicit + semantic similarity)             |
+| `knowledge_list_tags` | List all tags with counts                                         |
+| `knowledge_triage`    | Submit to triage queue for review                                 |
+| `knowledge_history`   | Get edit history for an entry (who changed what, when, why)       |
+
+## Schema
+
+Tables in the `kb` database:
+
+- **KnowledgeEntry** — core entries with HNSW vector index, `@relationship` directives for supersession/siblings/related, `@createdTime`/`@updatedTime`
+- **KnowledgeEntryEdit** — append-only edit history audit log
+- **TriageItem** — webhook intake queue (7-day TTL)
+- **KnowledgeTag** — tag name as primary key with entry counts
+- **QueryLog** — search analytics (30-day TTL)
+- **ServiceKey** — API keys with scrypt-hashed secrets
+- **OAuthClient** — dynamic client registrations (RFC 7591)
+- **OAuthCode** — authorization codes (5-minute TTL)
+- **OAuthRefreshToken** — refresh tokens (30-day TTL)
+- **OAuthSigningKey** — RSA key pair for JWT signing
+
+### Applicability Scoping
+
+Entries carry an `appliesTo` scope:
+
+```json
+{
+  "harper": ">=4.0 <5.0",
+  "storageEngine": "lmdb",
+  "node": ">=22",
+  "platform": "linux"
+}
+```
+
+Search results are boosted or demoted (never hidden) based on the caller's context.
+
+### Entry Relationships
+
+- **Supersedes** — "This replaces that for newer versions"
+- **Siblings** — "Same topic, different config" (e.g., LMDB vs RocksDB behavior)
+- **Related** — loose "see also" association
+
+## Auth Model
+
+| Role              | Read | Write                        | Review | Manage |
+| ----------------- | ---- | ---------------------------- | ------ | ------ |
+| `team`            | Yes  | Yes                          | Yes    | Yes    |
+| `ai_agent`        | Yes  | Yes (flagged `ai-generated`) | No     | No     |
+| `service_account` | Yes  | Triage queue only            | No     | No     |
+
+MCP uses OAuth 2.1 with PKCE for authentication. MCP clients discover auth requirements via `/.well-known/oauth-protected-resource`, register dynamically, and authenticate through a browser-based login flow (GitHub OAuth primary, Harper credentials fallback). The web UI uses GitHub OAuth via `@harperfast/oauth` with Harper credentials as fallback.
+
+## Development
+
+```bash
+# Build
+npm run build
+
+# Run tests (202 tests)
+npm test
+
+# Test with coverage
+npm run test:coverage
+
+# Download embedding model
+npm run model:download
+
+# Download + verify embedding model
+npm run model:test
+
+# Watch mode
+npm run dev
+```
+
+### Testing
+
+Tests use Node.js built-in test runner (`node:test`) with mock Harper globals (in-memory tables). Tests run against compiled output in `dist/`.
+
+```bash
+npm test
+```
+
+## License
+
+MIT

package/config.yaml

@@ -0,0 +1,17 @@
+# Knowledge Base Plugin Configuration
+# This file defines the plugin entry point and default settings
+# All settings can be overridden in your application's config.yaml
+
+# Plugin entry point (required by Harper)
+pluginModule: "dist/index.js"
+
+# GraphQL schema for knowledge base tables
+graphqlSchema:
+  files: "schema/knowledge.graphql"
+
+# Static web UI files
+static:
+  files: "web/**"
+
+# Default settings (optional - these are used if not specified in app config)
+embeddingModel: "nomic-embed-text"

package/dist/core/embeddings.d.ts

@@ -0,0 +1,29 @@
+/**
+ * Embedding Model Management
+ *
+ * Downloads and initializes the nomic-embed-text model for generating
+ * vector embeddings. Uses file-based locking to prevent concurrent
+ * downloads across worker threads.
+ */
+/**
+ * Initialize the embedding model.
+ * Downloads the model to ~/hdb/models/ if not present.
+ * Uses file-based locking so only one thread downloads.
+ *
+ * @param config - Plugin configuration with embeddingModel name
+ */
+export declare function initEmbeddingModel(config: {
+    embeddingModel: string;
+}): Promise<void>;
+/**
+ * Generate an embedding vector for the given text.
+ *
+ * @param text - Text to generate embedding for
+ * @returns Embedding vector as array of numbers
+ * @throws Error if the model has not been initialized
+ */
+export declare function generateEmbedding(text: string): Promise<number[]>;
+/**
+ * Clean up embedding model resources.
+ */
+export declare function dispose(): Promise<void>;

package/dist/core/embeddings.js

@@ -0,0 +1,199 @@
+/**
+ * Embedding Model Management
+ *
+ * Downloads and initializes the nomic-embed-text model for generating
+ * vector embeddings. Uses file-based locking to prevent concurrent
+ * downloads across worker threads.
+ */
+import { writeFile, readFile, unlink, mkdir } from "node:fs/promises";
+import { existsSync } from "node:fs";
+import path from "node:path";
+import os from "node:os";
+// Module-level state for the loaded model and context
+let llama = null;
+let embeddingModel = null;
+let embeddingContext = null;
+// Model configuration: Hugging Face model identifiers for node-llama-cpp
+const MODEL_CONFIGS = {
+    "nomic-embed-text": {
+        repo: "nomic-ai/nomic-embed-text-v1.5-GGUF",
+        file: "nomic-embed-text-v1.5.Q4_K_M.gguf",
+    },
+    "nomic-embed-text-v2-moe": {
+        repo: "nomic-ai/nomic-embed-text-v2-moe-GGUF",
+        file: "nomic-embed-text-v2-moe.Q4_K_M.gguf",
+    },
+};
+/**
+ * Get the directory where models are stored.
+ */
+function getModelsDir() {
+    return path.join(os.homedir(), "hdb", "models");
+}
+/**
+ * Get the lock file path for download synchronization.
+ */
+function getLockFilePath(modelName) {
+    return path.join(getModelsDir(), `${modelName}.lock`);
+}
+/**
+ * Get the model URI for node-llama-cpp downloads.
+ */
+function getModelUri(modelName) {
+    const config = MODEL_CONFIGS[modelName];
+    if (!config) {
+        throw new Error(`Unknown embedding model: ${modelName}. Supported: ${Object.keys(MODEL_CONFIGS).join(", ")}`);
+    }
+    return `hf:${config.repo}/${config.file}`;
+}
+/**
+ * Acquire a file-based lock for model download.
+ * Returns true if lock was acquired, false if another thread holds it.
+ */
+async function acquireDownloadLock(modelName) {
+    const lockPath = getLockFilePath(modelName);
+    try {
+        if (existsSync(lockPath)) {
+            // Check if lock is stale (older than 10 minutes)
+            const lockContent = await readFile(lockPath, "utf-8");
+            const lockTime = parseInt(lockContent, 10);
+            if (!isNaN(lockTime) && Date.now() - lockTime < 10 * 60 * 1000) {
+                return false; // Lock is held and not stale
+            }
+            // Stale lock — remove and re-acquire
+        }
+        await writeFile(lockPath, String(Date.now()), { flag: "wx" }).catch(async () => {
+            // wx flag failed (file exists), try overwriting stale lock
+            await writeFile(lockPath, String(Date.now()));
+        });
+        return true;
+    }
+    catch {
+        return false;
+    }
+}
+/**
+ * Release the download lock.
+ */
+async function releaseDownloadLock(modelName) {
+    const lockPath = getLockFilePath(modelName);
+    try {
+        await unlink(lockPath);
+    }
+    catch {
+        // Lock file already removed — safe to ignore
+    }
+}
+/**
+ * Wait for another thread's download to complete.
+ * Polls the lock file until it disappears or the model file appears.
+ */
+async function waitForDownload(modelName, modelPath) {
+    const lockPath = getLockFilePath(modelName);
+    const maxWait = 10 * 60 * 1000; // 10 minutes
+    const pollInterval = 2000; // 2 seconds
+    const start = Date.now();
+    while (Date.now() - start < maxWait) {
+        if (existsSync(modelPath) && !existsSync(lockPath)) {
+            return; // Download completed
+        }
+        await new Promise((resolve) => setTimeout(resolve, pollInterval));
+    }
+    throw new Error(`Timed out waiting for model download: ${modelName}`);
+}
+/**
+ * Download the embedding model if not already present.
+ * Uses file-based locking so only one worker thread downloads.
+ */
+async function downloadModelIfNeeded(modelName) {
+    const modelUri = getModelUri(modelName);
+    const modelsDir = getModelsDir();
+    // Ensure models directory exists
+    await mkdir(modelsDir, { recursive: true });
+    // Use node-llama-cpp to resolve the actual file path and download if needed.
+    // node-llama-cpp prefixes filenames (e.g., hf_nomic-ai_<file>.gguf),
+    // so we use its entrypointFilePath rather than guessing the name.
+    const { createModelDownloader } = (await import("node-llama-cpp"));
+    const downloader = await createModelDownloader({
+        modelUri,
+        dirPath: modelsDir,
+        skipExisting: true,
+    });
+    const modelPath = downloader.entrypointFilePath;
+    // Already downloaded
+    if (existsSync(modelPath)) {
+        return modelPath;
+    }
+    // Try to acquire download lock
+    const acquired = await acquireDownloadLock(modelName);
+    if (!acquired) {
+        // Another thread is downloading — wait for it
+        logger?.info?.(`Another thread is downloading ${modelName}, waiting...`);
+        await waitForDownload(modelName, modelPath);
+        return modelPath;
+    }
+    try {
+        logger?.info?.(`Downloading embedding model: ${modelName} from ${modelUri}`);
+        const resultPath = await downloader.download();
+        logger?.info?.(`Model ${modelName} downloaded successfully to ${resultPath}`);
+        return resultPath;
+    }
+    finally {
+        await releaseDownloadLock(modelName);
+    }
+}
+/**
+ * Initialize the embedding model.
+ * Downloads the model to ~/hdb/models/ if not present.
+ * Uses file-based locking so only one thread downloads.
+ *
+ * @param config - Plugin configuration with embeddingModel name
+ */
+export async function initEmbeddingModel(config) {
+    const modelName = config.embeddingModel || "nomic-embed-text";
+    if (embeddingModel) {
+        logger?.debug?.("Embedding model already initialized");
+        return;
+    }
+    const modelPath = await downloadModelIfNeeded(modelName);
+    // Load the model with node-llama-cpp
+    const { getLlama } = (await import("node-llama-cpp"));
+    llama = await getLlama({ progressLogs: false });
+    embeddingModel = await llama.loadModel({ modelPath });
+    embeddingContext = await embeddingModel.createEmbeddingContext({
+        contextSize: "auto",
+    });
+    logger?.info?.(`Embedding model ${modelName} loaded successfully`);
+}
+/**
+ * Generate an embedding vector for the given text.
+ *
+ * @param text - Text to generate embedding for
+ * @returns Embedding vector as array of numbers
+ * @throws Error if the model has not been initialized
+ */
+export async function generateEmbedding(text) {
+    if (!embeddingContext) {
+        throw new Error("Embedding model not initialized. Call initEmbeddingModel() first.");
+    }
+    const result = await embeddingContext.getEmbeddingFor(text);
+    return Array.from(result.vector);
+}
+/**
+ * Clean up embedding model resources.
+ */
+export async function dispose() {
+    if (embeddingContext && !embeddingContext.disposed) {
+        await embeddingContext.dispose();
+    }
+    embeddingContext = null;
+    if (embeddingModel) {
+        await embeddingModel.dispose();
+    }
+    embeddingModel = null;
+    if (llama) {
+        await llama.dispose();
+    }
+    llama = null;
+    logger?.info?.("Embedding model disposed");
+}

package/dist/core/entries.d.ts

@@ -0,0 +1,85 @@
+/**
+ * Knowledge Entry Management
+ *
+ * CRUD operations for knowledge base entries. Handles embedding generation,
+ * tag synchronization, and relationship management.
+ */
+import type { KnowledgeEntry, KnowledgeEntryInput, KnowledgeEntryUpdate } from "../types.ts";
+/**
+ * Strip embedding vectors from an entry to keep responses compact.
+ * Embeddings are large float arrays not useful in API responses.
+ */
+export declare function stripEmbedding<T extends {
+    embedding?: number[];
+}>(entry: T): Omit<T, "embedding">;
+/**
+ * Create a new knowledge entry.
+ *
+ * Generates an embedding from title + content, synchronizes tags,
+ * and stores the entry. A UUID is generated if no id is provided.
+ *
+ * @param data - Entry data to create
+ * @returns The created knowledge entry
+ */
+export declare function createEntry(data: KnowledgeEntryInput): Promise<KnowledgeEntry>;
+/**
+ * Get a knowledge entry by ID.
+ *
+ * @param id - Entry ID
+ * @returns The entry, or null if not found
+ */
+export declare function getEntry(id: string): Promise<KnowledgeEntry | null>;
+/**
+ * Update an existing knowledge entry.
+ *
+ * Merges the update data with the existing entry. If title or content changed,
+ * regenerates the embedding. Synchronizes tag counts if tags changed.
+ * Optionally logs the edit to the history table.
+ *
+ * @param id - ID of the entry to update
+ * @param data - Fields to update
+ * @param options - Optional edit tracking metadata
+ * @returns The updated entry
+ * @throws Error if the entry does not exist
+ */
+export declare function updateEntry(id: string, data: KnowledgeEntryUpdate, options?: {
+    editedBy?: string;
+    editSummary?: string;
+}): Promise<KnowledgeEntry>;
+/**
+ * Mark an entry as deprecated.
+ *
+ * @param id - ID of the entry to deprecate
+ * @throws Error if the entry does not exist
+ */
+export declare function deprecateEntry(id: string): Promise<void>;
+/**
+ * Link a new entry as superseding an old entry.
+ *
+ * Sets newEntry.supersedesId = oldId and oldEntry.supersededById = newId.
+ *
+ * @param newId - ID of the new (superseding) entry
+ * @param oldId - ID of the old (superseded) entry
+ * @throws Error if either entry does not exist
+ */
+export declare function linkSupersedes(newId: string, oldId: string): Promise<void>;
+/**
+ * Link multiple entries as siblings.
+ *
+ * For each entry, adds all other entry IDs to its siblingIds array (deduplicated).
+ *
+ * @param ids - IDs of entries to link as siblings
+ * @throws Error if any entry does not exist
+ */
+export declare function linkSiblings(ids: string[]): Promise<void>;
+/**
+ * Link two entries as related.
+ *
+ * Adds relatedId to the entry's relatedIds array (deduplicated).
+ * This is a one-directional link; call twice for bidirectional.
+ *
+ * @param id - ID of the entry to add a related link to
+ * @param relatedId - ID of the related entry
+ * @throws Error if the entry does not exist
+ */
+export declare function linkRelated(id: string, relatedId: string): Promise<void>;