opencode-lore 0.2.9 → 0.3.0

package/README.md

@@ -24,31 +24,45 @@ A **gradient context manager** decides how much of each tier to include in each
 
 > Scores below are on Claude Sonnet 4 (claude-sonnet-4-6). Results may vary with other models.
 
-### General memory recall
+### Coding session recall
 
-500-question evaluation using the [LongMemEval](https://github.com/xiaowu0162/LongMemEval) benchmark (ICLR 2025), tested in oracle mode (full message history provided as conversation context).
+20 questions across 2 real coding sessions (113K and 353K tokens), targeting specific facts at varying depths. Default mode simulates OpenCode's actual behavior: compaction of early messages + 80K-token tail window. Lore mode uses on-the-fly distillation + the `recall` tool for searching raw message history.
 
-| Category                  | No plugin | Lore    |
-|---------------------------|-----------|---------|
-| Single-session (user)     | 71.9%     | 93.8%   |
-| Single-session (prefs)    | 46.7%     | 86.7%   |
-| Single-session (assistant)| 91.1%     | 96.4%   |
-| Multi-session             | 76.9%     | 85.1%   |
-| Knowledge updates         | 84.7%     | 93.1%   |
-| Temporal reasoning        | 64.6%     | 81.9%   |
-| Abstention                | 53.3%     | 86.7%   |
-| **Overall**               | **72.6%** | **88.0%** |
+**Accuracy:**
 
-### Coding session recall
+| Mode    | Score     | Accuracy   |
+|---------|-----------|------------|
+| Default | 10/20     | 50.0%      |
+| Lore    | **17/20** | **85.0%**  |
+
+**By question depth** (where in the session the answer lives):
+
+| Depth        | Default  | Lore        | Gap     |
+|--------------|----------|-------------|---------|
+| Early detail | 1/7      | **6/7**     | +71pp   |
+| Mid detail   | 3/5      | **5/5**     | +40pp   |
+| Late detail  | 6/7      | 6/7         | tied    |
+
+Early and mid details — specific numbers, file paths, design decisions, error messages — are what compaction loses and distillation preserves. Late details are in both modes' context windows, so they tie.
 
-15 questions across 3 real coding sessions, each asking about a specific fact from the conversation. Compared against OpenCode's default behavior (last ~80K tokens of context).
+**Cost:**
 
-| Metric         | Default | Lore         |
-|----------------|---------|--------------|
-| Score          | 10/15   | **14/15**    |
-| Accuracy       | 66.7%   | **93.3%**    |
+| Metric             | Default    | Lore       | Factor       |
+|--------------------|------------|------------|--------------|
+| Avg input/question | 126K tok   | 50K tok    | 2.5x less    |
+| Total cost         | $8.14      | $1.87      | 4.4x cheaper |
+| Cost/correct       | $0.81      | **$0.11**  | 7.4x cheaper |
 
-Lore's advantage is largest on early/mid-session details that fall outside the recent-context window — facts like which PR was being tested, why an endpoint was changed, how many rows were updated, or what a specific bug's root cause was. The `recall` tool covers gaps where the distilled observations lack fine-grained detail.
+Lore's distilled context is smaller and more cacheable than raw tail windows, making it both more accurate and cheaper per correct answer.
+
+**Distillation compression:**
+
+| Session            | Messages | Tokens | Distilled to     | Compression |
+|--------------------|----------|--------|------------------|-------------|
+| cli-sentry-issue   | 318      | 113K   | ~6K tokens       | 19x         |
+| cli-nightly        | 898      | 353K   | ~19K tokens      | 19x         |
+
+The eval is self-contained and reproducible: session transcripts are stored as JSON files with no database dependency. See [`eval/`](eval/) for the harness and data.
 
 ## How we got here
 
@@ -58,9 +72,11 @@ This plugin was built in a few intense sessions. Some highlights:
 
 **Markdown injection.** Property-based testing with fast-check revealed that user-generated content in facts (code fences, heading markers, thematic breaks) could break the markdown structure of the injected context, confusing the model.
 
-**v2 — observation logs.** Switching to Mastra's observer/reflector architecture with plain-text timestamped observation logs was the breakthrough — LongMemEval jumped from 73.8% to 88.0%. The key insight: dated event logs preserve temporal relationships that structured JSON destroys.
+**v2 — observation logs.** Switching to Mastra's observer/reflector architecture with plain-text timestamped observation logs was the breakthrough. The key insight: dated event logs preserve temporal relationships that structured JSON destroys.
+
+**Prompt refinements.** The push from 80% to 93.3% on the initial coding recall eval came from two observer prompt additions: "EXACT NUMBERS — NEVER APPROXIMATE" (the observer was rounding counts) and "BUG FIXES — ALWAYS RECORD" (early-session fixes were being compressed away during reflection).
 
-**Prompt refinements.** The final push from 80% to 93.3% on coding recall came from two observer prompt additions: "EXACT NUMBERS — NEVER APPROXIMATE" (the observer was rounding counts) and "BUG FIXES — ALWAYS RECORD" (early-session fixes were being compressed away during reflection).
+**v3 — gradient fixes, caching, and proper eval.** A month of fixes (per-session gradient state, current-turn protection, cache.write calibration, prefix caching, LTM relevance scoring) shipped alongside a new self-contained eval harness. The old coding eval used DB-resident sessions that degraded over time as temporal pruning deleted messages. The new eval extracts full session transcripts into portable JSON files, distills on the fly with the current production prompt, seeds the DB for recall tool access, and compares against OpenCode's actual compaction behavior. This moved the coding eval from 15 questions on degraded data to 20 questions on clean 113K-353K token sessions — and confirmed the +35pp accuracy gap and 7x cost efficiency advantage.
 
 ## Installation
 
@@ -126,7 +142,6 @@ The assistant gets a `recall` tool that searches across stored messages and know
 - [How we solved the agent memory problem](https://www.sanity.io/blog/how-we-solved-the-agent-memory-problem) — Simen Svale at Sanity on the Nuum memory architecture: three-tier storage, distillation not summarization, recursive compression. The foundation this plugin is built on.
 - [Mastra Observational Memory](https://mastra.ai/research/observational-memory) — the observer/reflector architecture and the switch from structured JSON to timestamped observation logs that made v2 work.
 - [Mastra Memory source](https://github.com/mastra-ai/mastra/tree/main/packages/memory) — reference implementation.
-- [LongMemEval](https://arxiv.org/abs/2410.10813) — the evaluation benchmark (ICLR 2025) we used to measure progress.
 - [OpenCode](https://opencode.ai) — the coding agent this plugin extends.
 
 ## License

package/package.json

@@ -1,9 +1,9 @@
 {
   "name": "opencode-lore",
-  "version": "0.2.9",
+  "version": "0.3.0",
   "type": "module",
   "license": "MIT",
-  "description": "Three-tier memory architecture for OpenCode — distillation, not summarization",
+  "description": "Three-tier memory architecture for OpenCode \u2014 distillation, not summarization",
   "main": "src/index.ts",
   "exports": {
     ".": "./src/index.ts"

package/src/agents-file.ts

@@ -175,7 +175,7 @@ function buildSection(projectPath: string): string {
   // Section heading
   out.push("## Long-term Knowledge");
 
-  for (const [category, items] of grouped) {
+  for (const [category, items] of [...grouped.entries()].sort((a, b) => a[0].localeCompare(b[0]))) {
     out.push("");
     out.push(`### ${category.charAt(0).toUpperCase() + category.slice(1)}`);
     out.push("");

package/src/index.ts

@@ -234,6 +234,12 @@ export const LorePlugin: Plugin = async (ctx) => {
       }
 
       if (event.type === "session.error") {
+        // Skip eval/worker child sessions — only handle errors for real user sessions.
+        const errorSessionID = (event.properties as Record<string, unknown>).sessionID as
+          | string
+          | undefined;
+        if (errorSessionID && await shouldSkip(errorSessionID)) return;
+
         // Detect "prompt is too long" API errors and auto-recover:
         // 1. Force the gradient transform to escalate on the next call (skip layer 0/1)
         // 2. Force distillation to capture all temporal data before compaction
@@ -260,22 +266,19 @@ export const LorePlugin: Plugin = async (ctx) => {
         );
 
         if (isPromptTooLong) {
-          const sessionID = (event.properties as Record<string, unknown>).sessionID as
-            | string
-            | undefined;
           console.error(
-            `[lore] detected 'prompt too long' error — forcing distillation + layer escalation (session: ${sessionID?.substring(0, 16)})`,
+            `[lore] detected 'prompt too long' error — forcing distillation + layer escalation (session: ${errorSessionID?.substring(0, 16)})`,
           );
           // Force layer 2 on next transform — layers 0 and 1 were already too large.
           // The gradient at layers 2-4 will compress the context enough for the next turn.
           // Do NOT call session.summarize() here — it sends all messages to the model,
           // which would overflow again and create a stuck compaction loop.
-          setForceMinLayer(2, sessionID);
+          setForceMinLayer(2, errorSessionID);
 
-          if (sessionID) {
+          if (errorSessionID) {
             // Force distillation to capture all undistilled messages into the temporal
             // store so they're preserved even if the session is later compacted manually.
-            await backgroundDistill(sessionID, true);
+            await backgroundDistill(errorSessionID, true);
           }
         }
       }