opencode-lore 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 BYK
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,123 @@
+# opencode-lore
+
+> **Experimental** — This plugin is under active development. APIs, storage format, and behavior may change.
+
+An implementation of [Sanity's Nuum](https://www.sanity.io/blog/how-we-solved-the-agent-memory-problem) memory architecture and [Mastra's Observational Memory](https://mastra.ai/research/observational-memory) system as a plugin for [OpenCode](https://opencode.ai). Both projects pioneered the idea that coding agents need **distillation, not summarization** — preserving operational intelligence (file paths, error messages, exact decisions) rather than narrative summaries that lose the details agents need to keep working. This plugin brings those ideas to OpenCode.
+
+## Why
+
+Coding agents forget. Once a conversation exceeds the context window, earlier decisions, bug fixes, and architectural choices vanish. The default approach — summarize-and-compact — loses exactly the operational details agents need. After a few compaction passes, the agent knows you "discussed authentication" but can't actually continue the work.
+
+## How it works
+
+Lore uses a three-tier memory architecture (following [Nuum's design](https://www.sanity.io/blog/how-we-solved-the-agent-memory-problem)):
+
+1. **Temporal storage** — every message is stored in a local SQLite FTS5 database, searchable on demand via the `recall` tool.
+
+2. **Distillation** — messages are incrementally distilled into an observation log (dated, timestamped, priority-tagged entries), following [Mastra's observer/reflector pattern](https://mastra.ai/research/observational-memory). When segments accumulate, older distillations are recursively merged to prevent unbounded growth. The observer prompt is tuned to preserve exact numbers, bug fixes, file paths, and assistant-generated content.
+
+3. **Long-term knowledge** — a curated knowledge base of facts, patterns, decisions, and gotchas that matter across projects, maintained by a background curator agent.
+
+A **gradient context manager** decides how much of each tier to include in each turn, using a 4-layer safety system that calibrates overhead dynamically from real API token counts. This handles the unpredictable context consumption of coding agents (large tool outputs, system prompts, injected instructions) better than a fixed-budget approach.
+
+## Benchmarks
+
+> Scores below are on Claude Sonnet 4 (claude-sonnet-4-6). Results may vary with other models.
+
+### General memory recall
+
+500-question evaluation using the [LongMemEval](https://github.com/xiaowu0162/LongMemEval) benchmark (ICLR 2025), tested in oracle mode (full message history provided as conversation context).
+
+| Category                  | No plugin | Lore    |
+|---------------------------|-----------|---------|
+| Single-session (user)     | 71.9%     | 93.8%   |
+| Single-session (prefs)    | 46.7%     | 86.7%   |
+| Single-session (assistant)| 91.1%     | 96.4%   |
+| Multi-session             | 76.9%     | 85.1%   |
+| Knowledge updates         | 84.7%     | 93.1%   |
+| Temporal reasoning        | 64.6%     | 81.9%   |
+| Abstention                | 53.3%     | 86.7%   |
+| **Overall**               | **72.6%** | **88.0%** |
+
+### Coding session recall
+
+15 questions across 3 real coding sessions, each asking about a specific fact from the conversation. Compared against OpenCode's default behavior (last ~80K tokens of context).
+
+| Metric         | Default | Lore         |
+|----------------|---------|--------------|
+| Score          | 10/15   | **14/15**    |
+| Accuracy       | 66.7%   | **93.3%**    |
+
+Lore's advantage is largest on early/mid-session details that fall outside the recent-context window — facts like which PR was being tested, why an endpoint was changed, how many rows were updated, or what a specific bug's root cause was. The `recall` tool covers gaps where the distilled observations lack fine-grained detail.
+
+## How we got here
+
+This plugin was built in a few intense sessions. Some highlights:
+
+**v1 — structured distillation.** The initial version used Nuum's `{ narrative, facts }` JSON format. It worked well for single-session preference recall (+40pp over baseline) but *regressed* on multi-session and temporal reasoning — the structured format was too rigid and lost temporal context.
+
+**Markdown injection.** Property-based testing with fast-check revealed that user-generated content in facts (code fences, heading markers, thematic breaks) could break the markdown structure of the injected context, confusing the model.
+
+**v2 — observation logs.** Switching to Mastra's observer/reflector architecture with plain-text timestamped observation logs was the breakthrough — LongMemEval jumped from 73.8% to 88.0%. The key insight: dated event logs preserve temporal relationships that structured JSON destroys.
+
+**Prompt refinements.** The final push from 80% to 93.3% on coding recall came from two observer prompt additions: "EXACT NUMBERS — NEVER APPROXIMATE" (the observer was rounding counts) and "BUG FIXES — ALWAYS RECORD" (early-session fixes were being compressed away during reflection).
+
+## Installation
+
+### Prerequisites
+
+- [OpenCode](https://opencode.ai)
+
+### Setup
+
+Add `opencode-lore` to the `plugin` array in your project's `opencode.json`:
+
+```json
+{
+  "plugin": [
+    "opencode-lore"
+  ]
+}
+```
+
+Restart OpenCode and the plugin will be installed automatically.
+
+#### Development setup
+
+To use a local clone instead of the published package:
+
+```json
+{
+  "plugin": [
+    "file:///absolute/path/to/opencode-lore"
+  ]
+}
+```
+
+## What gets stored
+
+All data lives locally in `~/.local/share/opencode-lore/lore.db`:
+
+- **Session observations** — timestamped event log of each conversation: what was asked, what was done, decisions made, errors found
+- **Long-term knowledge** — patterns, gotchas, and architectural decisions curated across sessions and projects
+- **Raw messages** — full message history in FTS5-indexed SQLite for the `recall` tool
+
+## The `recall` tool
+
+The assistant gets a `recall` tool that searches across stored messages and knowledge. It's used automatically when the distilled context doesn't have enough detail:
+
+- "What did we decide about auth last week?"
+- "What was the error from the migration?"
+- "What's my database schema convention?"
+
+## Standing on the shoulders of
+
+- [How we solved the agent memory problem](https://www.sanity.io/blog/how-we-solved-the-agent-memory-problem) — Simen Svale at Sanity on the Nuum memory architecture: three-tier storage, distillation not summarization, recursive compression. The foundation this plugin is built on.
+- [Mastra Observational Memory](https://mastra.ai/research/observational-memory) — the observer/reflector architecture and the switch from structured JSON to timestamped observation logs that made v2 work.
+- [Mastra Memory source](https://github.com/mastra-ai/mastra/tree/main/packages/memory) — reference implementation.
+- [LongMemEval](https://arxiv.org/abs/2410.10813) — the evaluation benchmark (ICLR 2025) we used to measure progress.
+- [OpenCode](https://opencode.ai) — the coding agent this plugin extends.
+
+## License
+
+MIT

package/package.json

@@ -0,0 +1,47 @@
+{
+  "name": "opencode-lore",
+  "version": "0.1.0",
+  "type": "module",
+  "license": "MIT",
+  "description": "Three-tier memory architecture for OpenCode — distillation, not summarization",
+  "main": "src/index.ts",
+  "exports": {
+    ".": "./src/index.ts"
+  },
+  "scripts": {
+    "typecheck": "bun run tsc --noEmit",
+    "test": "bun test"
+  },
+  "peerDependencies": {
+    "@opencode-ai/plugin": ">=1.1.0"
+  },
+  "dependencies": {
+    "remark": "^15.0.1",
+    "zod": "^3.25.0"
+  },
+  "devDependencies": {
+    "@opencode-ai/plugin": "^1.1.39",
+    "@opencode-ai/sdk": "^1.1.39",
+    "@types/bun": "^1.2.0",
+    "fast-check": "^4.5.3",
+    "typescript": "^5.8.0"
+  },
+  "files": [
+    "src/",
+    "README.md",
+    "LICENSE"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/BYK/opencode-lore.git"
+  },
+  "keywords": [
+    "opencode",
+    "plugin",
+    "memory",
+    "agent",
+    "distillation",
+    "llm"
+  ],
+  "author": "BYK"
+}

package/src/config.ts

@@ -0,0 +1,54 @@
+import { z } from "zod";
+
+export const LoreConfig = z.object({
+  model: z
+    .object({
+      providerID: z.string(),
+      modelID: z.string(),
+    })
+    .optional(),
+  budget: z
+    .object({
+      distilled: z.number().min(0.05).max(0.5).default(0.25),
+      raw: z.number().min(0.1).max(0.7).default(0.4),
+      output: z.number().min(0.1).max(0.5).default(0.25),
+    })
+    .default({}),
+  distillation: z
+    .object({
+      minMessages: z.number().min(3).default(8),
+      maxSegment: z.number().min(5).default(50),
+      metaThreshold: z.number().min(3).default(10),
+    })
+    .default({}),
+  curator: z
+    .object({
+      enabled: z.boolean().default(true),
+      onIdle: z.boolean().default(true),
+      afterTurns: z.number().min(1).default(10),
+    })
+    .default({}),
+  crossProject: z.boolean().default(true),
+});
+
+export type LoreConfig = z.infer<typeof LoreConfig>;
+
+let current: LoreConfig = LoreConfig.parse({});
+
+export function config(): LoreConfig {
+  return current;
+}
+
+export async function load(directory: string): Promise<LoreConfig> {
+  const paths = [`${directory}/.opencode/lore.json`, `${directory}/lore.json`];
+  for (const path of paths) {
+    const file = Bun.file(path);
+    if (await file.exists()) {
+      const raw = await file.json();
+      current = LoreConfig.parse(raw);
+      return current;
+    }
+  }
+  current = LoreConfig.parse({});
+  return current;
+}

package/src/curator.ts

@@ -0,0 +1,154 @@
+import type { createOpencodeClient } from "@opencode-ai/sdk";
+import { config } from "./config";
+import * as temporal from "./temporal";
+import * as ltm from "./ltm";
+import { CURATOR_SYSTEM, curatorUser } from "./prompt";
+import { workerSessionIDs } from "./distillation";
+
+type Client = ReturnType<typeof createOpencodeClient>;
+
+const workerSessions = new Map<string, string>();
+
+async function ensureWorkerSession(
+  client: Client,
+  parentID: string,
+): Promise<string> {
+  const existing = workerSessions.get(parentID);
+  if (existing) return existing;
+  const session = await client.session.create({
+    body: { parentID, title: "lore curator" },
+  });
+  const id = session.data!.id;
+  workerSessions.set(parentID, id);
+  workerSessionIDs.add(id);
+  return id;
+}
+
+type CuratorOp =
+  | {
+      op: "create";
+      category: string;
+      title: string;
+      content: string;
+      scope: "project" | "global";
+      crossProject?: boolean;
+    }
+  | { op: "update"; id: string; content?: string; confidence?: number }
+  | { op: "delete"; id: string; reason: string };
+
+function parseOps(text: string): CuratorOp[] {
+  const cleaned = text
+    .trim()
+    .replace(/^```json?\s*/i, "")
+    .replace(/\s*```$/i, "");
+  try {
+    const parsed = JSON.parse(cleaned);
+    if (!Array.isArray(parsed)) return [];
+    return parsed.filter(
+      (op: unknown) =>
+        typeof op === "object" &&
+        op !== null &&
+        "op" in op &&
+        typeof (op as Record<string, unknown>).op === "string",
+    ) as CuratorOp[];
+  } catch {
+    return [];
+  }
+}
+
+// Track which messages we've already curated
+let lastCuratedAt = 0;
+
+export async function run(input: {
+  client: Client;
+  projectPath: string;
+  sessionID: string;
+  model?: { providerID: string; modelID: string };
+}): Promise<{ created: number; updated: number; deleted: number }> {
+  const cfg = config();
+  if (!cfg.curator.enabled) return { created: 0, updated: 0, deleted: 0 };
+
+  // Get recent messages since last curation
+  const all = temporal.bySession(input.projectPath, input.sessionID);
+  const recent = all.filter((m) => m.created_at > lastCuratedAt);
+  if (recent.length < 3) return { created: 0, updated: 0, deleted: 0 };
+
+  const text = recent.map((m) => `[${m.role}] ${m.content}`).join("\n\n");
+  const existing = ltm.forProject(input.projectPath, cfg.crossProject);
+  const existingForPrompt = existing.map((e) => ({
+    id: e.id,
+    category: e.category,
+    title: e.title,
+    content: e.content,
+  }));
+
+  const userContent = curatorUser({
+    messages: text,
+    existing: existingForPrompt,
+  });
+  const workerID = await ensureWorkerSession(input.client, input.sessionID);
+  const model = input.model ?? cfg.model;
+  const parts = [
+    { type: "text" as const, text: `${CURATOR_SYSTEM}\n\n${userContent}` },
+  ];
+
+  await input.client.session.prompt({
+    path: { id: workerID },
+    body: {
+      parts,
+      agent: "lore-curator",
+      ...(model ? { model } : {}),
+    },
+  });
+
+  const msgs = await input.client.session.messages({
+    path: { id: workerID },
+    query: { limit: 2 },
+  });
+  const last = msgs.data?.at(-1);
+  if (!last || last.info.role !== "assistant")
+    return { created: 0, updated: 0, deleted: 0 };
+
+  const responsePart = last.parts.find((p) => p.type === "text");
+  if (!responsePart || responsePart.type !== "text")
+    return { created: 0, updated: 0, deleted: 0 };
+
+  const ops = parseOps(responsePart.text);
+  let created = 0;
+  let updated = 0;
+  let deleted = 0;
+
+  for (const op of ops) {
+    if (op.op === "create") {
+      ltm.create({
+        projectPath: op.scope === "project" ? input.projectPath : undefined,
+        category: op.category,
+        title: op.title,
+        content: op.content,
+        session: input.sessionID,
+        scope: op.scope,
+        crossProject: op.crossProject ?? true,
+      });
+      created++;
+    } else if (op.op === "update") {
+      const entry = ltm.get(op.id);
+      if (entry) {
+        ltm.update(op.id, { content: op.content, confidence: op.confidence });
+        updated++;
+      }
+    } else if (op.op === "delete") {
+      const entry = ltm.get(op.id);
+      if (entry) {
+        ltm.remove(op.id);
+        deleted++;
+      }
+    }
+  }
+
+  lastCuratedAt = Date.now();
+  return { created, updated, deleted };
+}
+
+export function resetCurationTracker() {
+  lastCuratedAt = 0;
+}

package/src/db.ts

@@ -0,0 +1,198 @@
+import { Database } from "bun:sqlite";
+import { join } from "path";
+import { mkdirSync } from "fs";
+
+const SCHEMA_VERSION = 2;
+
+const MIGRATIONS: string[] = [
+  `
+  -- Version 1: Initial schema
+
+  CREATE TABLE IF NOT EXISTS projects (
+    id TEXT PRIMARY KEY,
+    path TEXT NOT NULL UNIQUE,
+    name TEXT,
+    created_at INTEGER NOT NULL
+  );
+
+  CREATE TABLE IF NOT EXISTS temporal_messages (
+    id TEXT PRIMARY KEY,
+    project_id TEXT NOT NULL REFERENCES projects(id),
+    session_id TEXT NOT NULL,
+    role TEXT NOT NULL,
+    content TEXT NOT NULL,
+    tokens INTEGER DEFAULT 0,
+    distilled INTEGER DEFAULT 0,
+    created_at INTEGER NOT NULL,
+    metadata TEXT
+  );
+
+  CREATE VIRTUAL TABLE IF NOT EXISTS temporal_fts USING fts5(
+    content,
+    content=temporal_messages,
+    content_rowid=rowid,
+    tokenize='porter unicode61'
+  );
+
+  -- Triggers to keep FTS in sync
+  CREATE TRIGGER IF NOT EXISTS temporal_fts_insert AFTER INSERT ON temporal_messages BEGIN
+    INSERT INTO temporal_fts(rowid, content) VALUES (new.rowid, new.content);
+  END;
+
+  CREATE TRIGGER IF NOT EXISTS temporal_fts_delete AFTER DELETE ON temporal_messages BEGIN
+    INSERT INTO temporal_fts(temporal_fts, rowid, content) VALUES('delete', old.rowid, old.content);
+  END;
+
+  CREATE TRIGGER IF NOT EXISTS temporal_fts_update AFTER UPDATE ON temporal_messages BEGIN
+    INSERT INTO temporal_fts(temporal_fts, rowid, content) VALUES('delete', old.rowid, old.content);
+    INSERT INTO temporal_fts(rowid, content) VALUES (new.rowid, new.content);
+  END;
+
+  CREATE INDEX IF NOT EXISTS idx_temporal_session ON temporal_messages(session_id);
+  CREATE INDEX IF NOT EXISTS idx_temporal_project ON temporal_messages(project_id);
+  CREATE INDEX IF NOT EXISTS idx_temporal_distilled ON temporal_messages(distilled);
+  CREATE INDEX IF NOT EXISTS idx_temporal_created ON temporal_messages(created_at);
+
+  CREATE TABLE IF NOT EXISTS distillations (
+    id TEXT PRIMARY KEY,
+    project_id TEXT NOT NULL REFERENCES projects(id),
+    session_id TEXT NOT NULL,
+    narrative TEXT NOT NULL,
+    facts TEXT NOT NULL,
+    source_ids TEXT NOT NULL,
+    generation INTEGER DEFAULT 0,
+    token_count INTEGER DEFAULT 0,
+    created_at INTEGER NOT NULL
+  );
+
+  CREATE INDEX IF NOT EXISTS idx_distillation_session ON distillations(session_id);
+  CREATE INDEX IF NOT EXISTS idx_distillation_project ON distillations(project_id);
+  CREATE INDEX IF NOT EXISTS idx_distillation_generation ON distillations(generation);
+  CREATE INDEX IF NOT EXISTS idx_distillation_created ON distillations(created_at);
+
+  CREATE TABLE IF NOT EXISTS knowledge (
+    id TEXT PRIMARY KEY,
+    project_id TEXT,
+    category TEXT NOT NULL,
+    title TEXT NOT NULL,
+    content TEXT NOT NULL,
+    source_session TEXT,
+    cross_project INTEGER DEFAULT 0,
+    confidence REAL DEFAULT 1.0,
+    created_at INTEGER NOT NULL,
+    updated_at INTEGER NOT NULL,
+    metadata TEXT
+  );
+
+  CREATE VIRTUAL TABLE IF NOT EXISTS knowledge_fts USING fts5(
+    title,
+    content,
+    category,
+    content=knowledge,
+    content_rowid=rowid,
+    tokenize='porter unicode61'
+  );
+
+  CREATE TRIGGER IF NOT EXISTS knowledge_fts_insert AFTER INSERT ON knowledge BEGIN
+    INSERT INTO knowledge_fts(rowid, title, content, category)
+    VALUES (new.rowid, new.title, new.content, new.category);
+  END;
+
+  CREATE TRIGGER IF NOT EXISTS knowledge_fts_delete AFTER DELETE ON knowledge BEGIN
+    INSERT INTO knowledge_fts(knowledge_fts, rowid, title, content, category)
+    VALUES('delete', old.rowid, old.title, old.content, old.category);
+  END;
+
+  CREATE TRIGGER IF NOT EXISTS knowledge_fts_update AFTER UPDATE ON knowledge BEGIN
+    INSERT INTO knowledge_fts(knowledge_fts, rowid, title, content, category)
+    VALUES('delete', old.rowid, old.title, old.content, old.category);
+    INSERT INTO knowledge_fts(rowid, title, content, category)
+    VALUES (new.rowid, new.title, new.content, new.category);
+  END;
+
+  CREATE INDEX IF NOT EXISTS idx_knowledge_project ON knowledge(project_id);
+  CREATE INDEX IF NOT EXISTS idx_knowledge_category ON knowledge(category);
+  CREATE INDEX IF NOT EXISTS idx_knowledge_cross ON knowledge(cross_project);
+
+  CREATE TABLE IF NOT EXISTS schema_version (
+    version INTEGER NOT NULL
+  );
+
+  INSERT INTO schema_version (version) VALUES (1);
+  `,
+  `
+  -- Version 2: Replace narrative+facts with observations text
+  ALTER TABLE distillations ADD COLUMN observations TEXT NOT NULL DEFAULT '';
+  `,
+];
+
+function dataDir() {
+  const xdg = process.env.XDG_DATA_HOME;
+  const base = xdg || join(process.env.HOME || "~", ".local", "share");
+  return join(base, "opencode-lore");
+}
+
+let instance: Database | undefined;
+
+export function db(): Database {
+  if (instance) return instance;
+  const dir = dataDir();
+  mkdirSync(dir, { recursive: true });
+  const path = join(dir, "lore.db");
+  instance = new Database(path, { create: true });
+  instance.exec("PRAGMA journal_mode = WAL");
+  instance.exec("PRAGMA foreign_keys = ON");
+  migrate(instance);
+  return instance;
+}
+
+function migrate(database: Database) {
+  const row = database
+    .query(
+      "SELECT name FROM sqlite_master WHERE type='table' AND name='schema_version'",
+    )
+    .get() as { name: string } | null;
+  const current = row
+    ? ((
+        database.query("SELECT version FROM schema_version").get() as {
+          version: number;
+        }
+      )?.version ?? 0)
+    : 0;
+  if (current >= MIGRATIONS.length) return;
+  for (let i = current; i < MIGRATIONS.length; i++) {
+    database.exec(MIGRATIONS[i]);
+  }
+  // Update version to latest. Migration 0 inserts version=1 via its own INSERT,
+  // but subsequent migrations don't update it, so always normalize to MIGRATIONS.length.
+  database.exec(`UPDATE schema_version SET version = ${MIGRATIONS.length}`);
+}
+
+export function close() {
+  if (instance) {
+    instance.close();
+    instance = undefined;
+  }
+}
+
+// Project management
+export function ensureProject(path: string, name?: string): string {
+  const existing = db()
+    .query("SELECT id FROM projects WHERE path = ?")
+    .get(path) as { id: string } | null;
+  if (existing) return existing.id;
+  const id = crypto.randomUUID();
+  db()
+    .query(
+      "INSERT INTO projects (id, path, name, created_at) VALUES (?, ?, ?, ?)",
+    )
+    .run(id, path, name ?? path.split("/").pop() ?? "unknown", Date.now());
+  return id;
+}
+
+export function projectId(path: string): string | undefined {
+  const row = db()
+    .query("SELECT id FROM projects WHERE path = ?")
+    .get(path) as { id: string } | null;
+  return row?.id;
+}