@psiclawops/hypermem 0.5.0 → 0.5.2

package/ARCHITECTURE.md

@@ -161,7 +161,7 @@ This means:
 
 ## Context Engine Plugin
 
-`plugin/src/index.ts` — OpenClaw context engine plugin (replaces basic hook integration):
+`plugin/src/index.ts` — OpenClaw context engine plugin (`hypercompositor`, fills `contextEngine` slot):
 
 ```
 gateway:startup     → Init hypermem, auto-rotate DBs, hydrate fleet cache
@@ -171,7 +171,15 @@ agent:afterTurn     → Ingest new messages to SQLite + Redis, trigger backgroun
 ```
 
 Registers with `ownsCompaction: true` — runtime skips legacy compaction entirely.
-Deployed as managed hook at `~/.openclaw/hooks/hypermem-core/handler.js`.
+
+## Memory Plugin
+
+`memory-plugin/src/index.ts` — Lightweight memory provider (`hypermem`, fills `memory` slot):
+
+- Registers `MemoryPluginCapability` with a `MemorySearchManager` backed by HyperMem's hybrid FTS5 + KNN retrieval
+- Provides the `memory_search` tool through the official memory slot interface
+- Public artifacts provider lists `MEMORY.md` and `memory/*.md` for all configured agents
+- Stateless wrapper: lifecycle is owned by the context engine plugin
 
 ### Plugin Data Flow
 
@@ -269,7 +277,8 @@ Incident history: `specs/HYPERMEM_INCIDENT_HISTORY.md`
 | `episode-store.ts` | ~180 | L4 | Significant event tracking |
 | `preference-store.ts` | ~170 | L4 | Operator behavioral patterns |
 | `topic-store.ts` | ~160 | L4 | Cross-session thread tracking |
-| `plugin/src/index.ts` | ~590 | - | OpenClaw context engine plugin + window invalidation |
+| `plugin/src/index.ts` | ~590 | - | `hypercompositor` context engine plugin + window invalidation |
+| `memory-plugin/src/index.ts` | ~290 | - | `hypermem` memory slot plugin (memory_search via hybrid retrieval) |
 
 ## Test Coverage (105 assertions, 11 suites)
 

package/README.md

@@ -140,7 +140,7 @@ High-signal turns are marked as keystones and survive pressure trimming ahead of
 
 ## hyperform
 
-Raw model output has two problems. It drifts from your standards (sycophancy, hedging, pagination, formatting) and it drifts from your facts (confabulation, contradiction, stale claims). hyperform handles both.
+Raw model output has two problems. It drifts from your standards (sycophancy, hedging, pagination, formatting) and it drifts from your facts (confabulation, contradiction, stale claims). hyperform handles both: normalization enforces consistency, confabulation resistance checks output against what's actually stored.
 
 **Normalization** shapes output to match a profile you define. Three presets ship with hypermem:
 
@@ -175,7 +175,7 @@ tool context, and leave ~30k as allocator reserve. hypermem handles slot competi
 automatically -- set contextWindowReserve to your preferred floor and let the compositor fill.
 ```
 
-**Verification** checks claims against the fact corpus before they're recorded. No LLM call. Pattern matching against stored facts, with confidence scoring and contradiction detection. Unsupported claims are flagged, contradictions surface in diagnostics, and a confabulation risk score is attached to the stored episode.
+**Confabulation resistance** checks output against stored facts before claims are recorded. No LLM call. Pattern matching against the fact corpus, with confidence scoring and contradiction detection. Unsupported claims are flagged, contradictions surface in diagnostics, and a confabulation risk score is attached to the stored episode.
 
 ---
 
@@ -256,7 +256,16 @@ L1 and L4 structured retrieval are sub-millisecond. Vector embeddings are comput
 
 ## Architecture
 
-hypermem plugs into OpenClaw as a context engine and owns the full prompt composition lifecycle. It registers as both `contextEngine` and `memory`, providing the standard memory slot interface alongside full prompt composition: `memory_search` routes through the official slot and shows correctly in `openclaw plugins list`.
+hypermem plugs into OpenClaw via two plugins that fill both composition slots:
+
+| Plugin | ID | Slot | What it does |
+|---|---|---|---|
+| `@psiclawops/hypercompositor` | `hypercompositor` | `contextEngine` | Owns session lifecycle, ingest, compose, afterTurn indexing, tool compression, hyperform |
+| `@psiclawops/hypermem-memory` | `hypermem` | `memory` | Provides `memory_search` tool backed by hybrid FTS5 + KNN retrieval against library.db |
+
+Both load from the same repo and share a single HyperMem core singleton. The context engine plugin (`hypercompositor`) is the heavy one: session warming, compositor, tool gradient, hyperform. The memory plugin (`hypermem`) is a thin wrapper that exposes HyperMem's hybrid retrieval as OpenClaw's standard `MemoryPluginCapability`, so `memory_search` routes through the official memory slot and shows correctly in `openclaw plugins list`.
+
+The Plugin column is the npm package name. The ID column is what goes in `plugins.allow` and `plugins.slots.*`. Don't put the package name in a slot config.
 
 **L1: SQLite in-memory.** Sub-millisecond hot reads, no network dependency, no daemon, no retry logic. Identity, compressed session history, cached embeddings, topic-scoped session and recall state, and agent registry data. The compositor hits this first on every turn.
 
@@ -276,7 +285,7 @@ FTS5 queries use compound indexes on `agentId + sort key` and prefix optimizatio
 
 | Collection | What it holds |
 |---|---|
-| Facts | Verifiable claims with confidence, domain, expiry, supersedes chains |
+| Facts | Claims with confidence scoring, domain, expiry, supersedes chains |
 | Knowledge | Domain/key/value structured data with full-text search |
 | Episodes | Significant events with impact scores and participant tracking |
 | Topics | Cross-session thread tracking and synthesized wiki pages |
@@ -371,10 +380,12 @@ git clone https://github.com/PsiClawOps/hypermem.git ~/.openclaw/workspace/repo/
 cd ~/.openclaw/workspace/repo/hypermem
 npm install && npm run build
 npm --prefix plugin install && npm --prefix plugin run build
+npm --prefix memory-plugin install && npm --prefix memory-plugin run build
 
-openclaw config set plugins.slots.contextEngine hypermem
+openclaw config set plugins.slots.contextEngine hypercompositor
 openclaw config set plugins.slots.memory hypermem
-openclaw config set plugins.load.paths '["~/.openclaw/workspace/repo/hypermem/plugin"]' --strict-json
+openclaw config set plugins.load.paths '["~/.openclaw/workspace/repo/hypermem/plugin","~/.openclaw/workspace/repo/hypermem/memory-plugin"]' --strict-json
+openclaw config set plugins.allow '["hypercompositor","hypermem"]' --strict-json
 openclaw gateway restart
 ```
 
@@ -486,6 +497,19 @@ const spawn = await buildSpawnContext(
 
 ---
 
+## CLI
+
+`bin/hypermem-status.mjs` provides health checks and metrics from the command line:
+
+```bash
+node bin/hypermem-status.mjs              # full dashboard
+node bin/hypermem-status.mjs --agent forge   # scoped to one agent
+node bin/hypermem-status.mjs --json          # machine-readable output
+node bin/hypermem-status.mjs --health        # health checks only (exit 1 on failure)
+```
+
+---
+
 ## Data directory
 
 ```text

package/bin/hypermem-status.mjs

@@ -0,0 +1,166 @@
+#!/usr/bin/env node
+/**
+ * hypermem status — health check and metrics dashboard CLI
+ *
+ * Usage:
+ *   node bin/hypermem-status.mjs              # full dashboard
+ *   node bin/hypermem-status.mjs --agent forge # scoped to one agent
+ *   node bin/hypermem-status.mjs --json        # machine-readable output
+ *   node bin/hypermem-status.mjs --health      # health checks only (exit 1 on failure)
+ *
+ * Requires: compiled dist/ (run `npm run build` first)
+ */
+
+import { existsSync } from 'node:fs';
+import { resolve, join, dirname } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import os from 'node:os';
+
+const __dir = dirname(fileURLToPath(import.meta.url));
+const root = resolve(__dir, '..');
+
+// ── Arg parsing ──────────────────────────────────────────────────
+const args = process.argv.slice(2);
+const flags = {
+  json: args.includes('--json'),
+  health: args.includes('--health'),
+  help: args.includes('--help') || args.includes('-h'),
+  agent: null,
+};
+
+const agentIdx = args.indexOf('--agent');
+if (agentIdx !== -1 && args[agentIdx + 1]) {
+  flags.agent = args[agentIdx + 1];
+}
+
+if (flags.help) {
+  console.log(`
+hypermem status — health check and metrics dashboard
+
+Usage:
+  hypermem-status.mjs [options]
+
+Options:
+  --agent <id>   Scope metrics to a specific agent
+  --json         Output raw JSON instead of formatted summary
+  --health       Health checks only (exits 1 if any check fails)
+  -h, --help     Show this help
+`);
+  process.exit(0);
+}
+
+// ── Resolve data directory ───────────────────────────────────────
+const dataDir = process.env.HYPERMEM_DATA_DIR
+  || join(process.env.HOME || os.homedir(), '.openclaw', 'hypermem');
+
+if (!existsSync(dataDir)) {
+  console.error(`Error: data directory not found: ${dataDir}`);
+  console.error('Is HyperMem installed? Set HYPERMEM_DATA_DIR if using a custom path.');
+  process.exit(1);
+}
+
+// ── Open DBs ─────────────────────────────────────────────────────
+import { DatabaseSync } from 'node:sqlite';
+
+function openDb(filePath, label) {
+  if (!existsSync(filePath)) {
+    console.error(`Error: ${label} not found: ${filePath}`);
+    process.exit(1);
+  }
+  try {
+    const db = new DatabaseSync(filePath, { open: true });
+    db.exec('PRAGMA journal_mode = WAL');
+    db.exec('PRAGMA busy_timeout = 3000');
+    return db;
+  } catch (err) {
+    console.error(`Error opening ${label}: ${err.message}`);
+    process.exit(1);
+  }
+}
+
+// Main DB: pick any agent's messages.db for composition stats, or fall back
+// For fleet-wide, we need at least one agent's message db.
+// The metrics dashboard expects a "main" db — find one.
+let mainDbPath;
+
+if (flags.agent) {
+  mainDbPath = join(dataDir, 'agents', flags.agent, 'messages.db');
+} else {
+  // Find first available agent messages.db
+  const agentsDir = join(dataDir, 'agents');
+  if (existsSync(agentsDir)) {
+    const { readdirSync } = await import('node:fs');
+    const agents = readdirSync(agentsDir, { withFileTypes: true })
+      .filter(d => d.isDirectory())
+      .map(d => d.name);
+    for (const a of agents) {
+      const candidate = join(agentsDir, a, 'messages.db');
+      if (existsSync(candidate)) {
+        mainDbPath = candidate;
+        break;
+      }
+    }
+  }
+}
+
+if (!mainDbPath || !existsSync(mainDbPath)) {
+  console.error('Error: no agent messages.db found. Has HyperMem ingested any sessions?');
+  process.exit(1);
+}
+
+const libraryDbPath = join(dataDir, 'library.db');
+
+const mainDb = openDb(mainDbPath, 'messages.db');
+const libraryDb = openDb(libraryDbPath, 'library.db');
+
+// ── Import metrics functions ─────────────────────────────────────
+const distPath = join(root, 'dist', 'metrics-dashboard.js');
+if (!existsSync(distPath)) {
+  console.error('Error: dist/metrics-dashboard.js not found. Run `npm run build` first.');
+  process.exit(1);
+}
+
+const { collectMetrics, formatMetricsSummary } = await import(distPath);
+
+// ── Collect and output ───────────────────────────────────────────
+const opts = {};
+if (flags.agent) {
+  opts.agentIds = [flags.agent];
+}
+
+try {
+  const metrics = await collectMetrics(mainDb, libraryDb, opts);
+
+  if (flags.health) {
+    // Health-only mode: check and exit
+    const h = metrics.health;
+    const ok = h.mainDbOk && h.libraryDbOk && (h.cacheOk === null || h.cacheOk);
+
+    if (flags.json) {
+      console.log(JSON.stringify(h, null, 2));
+    } else {
+      console.log(`hypermem ${h.packageVersion} health check`);
+      console.log(`  main db:    ${h.mainDbOk ? '✅' : '❌'}${h.mainSchemaVersion !== null ? ` (schema v${h.mainSchemaVersion})` : ''}`);
+      console.log(`  library db: ${h.libraryDbOk ? '✅' : '❌'}${h.librarySchemaVersion !== null ? ` (schema v${h.librarySchemaVersion})` : ''}`);
+      if (h.cacheOk !== null) {
+        console.log(`  cache:      ${h.cacheOk ? '✅' : '❌'}`);
+      }
+      console.log(`  status:     ${ok ? '✅ healthy' : '❌ degraded'}`);
+    }
+
+    process.exit(ok ? 0 : 1);
+  }
+
+  if (flags.json) {
+    console.log(JSON.stringify(metrics, null, 2));
+  } else {
+    console.log(formatMetricsSummary(metrics));
+  }
+} catch (err) {
+  console.error(`Error collecting metrics: ${err.message}`);
+  if (err.stack) console.error(err.stack);
+  process.exit(1);
+} finally {
+  try { mainDb.close(); } catch {}
+  try { libraryDb.close(); } catch {}
+}

package/dist/background-indexer.d.ts

@@ -0,0 +1,132 @@
+/**
+ * hypermem Background Indexer
+ *
+ * Processes message history to extract structured knowledge:
+ *   - Facts: atomic pieces of learned information
+ *   - Episodes: significant events worth remembering
+ *   - Topics: conversation threads and their lifecycle
+ *   - Knowledge: durable structured entries (domain + key)
+ *
+ * Runs as a periodic background task, processing unindexed messages
+ * in batches. Each batch is scored, classified, and stored in L4 (library.db).
+ *
+ * Design principles:
+ *   - No LLM dependency: extraction uses pattern matching + heuristics
+ *   - Idempotent: tracks watermarks per agent to avoid reprocessing
+ *   - Bounded: processes N messages per tick to avoid blocking
+ *   - Observable: logs extraction stats for monitoring
+ */
+import type { DatabaseSync } from 'node:sqlite';
+import type { IndexerConfig, SessionCursor } from './types.js';
+import { type DreamerConfig } from './dreaming-promoter.js';
+import type { VectorStore } from './vector-store.js';
+export interface IndexerStats {
+    agentId: string;
+    messagesProcessed: number;
+    factsExtracted: number;
+    episodesRecorded: number;
+    topicsUpdated: number;
+    knowledgeUpserted: number;
+    /** Number of superseded fact vectors tombstoned from the vector index this tick. */
+    tombstoned: number;
+    elapsedMs: number;
+    /** Number of messages that were post-cursor (unseen by model, high-signal priority). */
+    postCursorMessages: number;
+}
+/**
+ * Optional callback to fetch the session cursor for an agent+session.
+ * When provided, the indexer uses the cursor to prioritize unseen messages.
+ * The cursor boundary separates "model has seen this" from "new since last compose".
+ */
+export type CursorFetcher = (agentId: string, sessionKey: string) => Promise<SessionCursor | null>;
+export interface WatermarkState {
+    agentId: string;
+    lastMessageId: number;
+    lastRunAt: string;
+}
+export declare class BackgroundIndexer {
+    private getMessageDb?;
+    private getLibraryDb?;
+    private listAgents?;
+    private getCursor?;
+    private readonly config;
+    private readonly dreamerConfig;
+    private intervalHandle;
+    private running;
+    private vectorStore;
+    private synthesizer;
+    private tickCount;
+    constructor(config?: Partial<IndexerConfig>, getMessageDb?: ((agentId: string) => DatabaseSync) | undefined, getLibraryDb?: (() => DatabaseSync) | undefined, listAgents?: (() => string[]) | undefined, getCursor?: CursorFetcher | undefined, dreamerConfig?: Partial<DreamerConfig>);
+    /**
+     * Set the vector store for embedding new facts/episodes at index time.
+     * Optional — if not set, indexer runs without embedding (FTS5-only mode).
+     */
+    setVectorStore(vs: VectorStore): void;
+    /**
+     * Start periodic indexing.
+     */
+    start(): void;
+    /**
+     * Stop periodic indexing.
+     */
+    stop(): void;
+    /**
+     * Run one indexing pass across all agents.
+     */
+    tick(): Promise<IndexerStats[]>;
+    /**
+     * Process a single agent's unindexed messages.
+     *
+     * When a cursor fetcher is available, messages are split into two tiers:
+     *   - Post-cursor (id > cursor.lastSentId): "unseen" by the model, high-signal priority
+     *   - Pre-cursor (id <= cursor.lastSentId): already in the model's context window, lower priority
+     * Post-cursor messages are processed first. This ensures the indexer prioritizes
+     * content the model hasn't seen yet — decisions, incidents, and discoveries that
+     * happened between context windows.
+     */
+    private processAgent;
+    /**
+     * Fetch unindexed messages for an agent.
+     */
+    private getUnindexedMessages;
+    /**
+     * Get the session key for a conversation ID.
+     */
+    private getSessionKeyForMessage;
+    /**
+     * Get the indexing watermark for an agent.
+     */
+    private getWatermark;
+    /**
+     * Set the indexing watermark for an agent.
+     */
+    private setWatermark;
+    /**
+     * Apply time-based decay to facts.
+     * Increases decay_score for older facts, making them less relevant.
+     */
+    private applyDecay;
+    /**
+     * Parse a duration string like "24h", "7d" into seconds.
+     */
+    private parseDuration;
+    /**
+     * One-time backfill: embed episodes with sig>=0.5 that were missed by the
+     * old >=0.7 vectorization threshold.
+     *
+     * Gated by a system_state flag 'indexer:episode_backfill_v1' so it runs
+     * exactly once even across gateway restarts. Safe to re-run manually
+     * (delete the flag row first) if re-backfill is ever needed.
+     */
+    backfillEpisodeVectors(): Promise<void>;
+    /**
+     * Get current watermarks for all agents.
+     */
+    getWatermarks(libraryDb: DatabaseSync): WatermarkState[];
+}
+/**
+ * Create and start a background indexer connected to hypermem databases.
+ * Used by the hook or a standalone daemon.
+ */
+export declare function createIndexer(getMessageDb: (agentId: string) => DatabaseSync, getLibraryDb: () => DatabaseSync, listAgents: () => string[], config?: Partial<IndexerConfig>, getCursor?: CursorFetcher, vectorStore?: VectorStore, dreamerConfig?: Partial<DreamerConfig>): BackgroundIndexer;
+//# sourceMappingURL=background-indexer.d.ts.map

package/dist/background-indexer.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"background-indexer.d.ts","sourceRoot":"","sources":["../src/background-indexer.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;GAiBG;AAEH,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAChD,OAAO,KAAK,EAAiB,aAAa,EAAe,aAAa,EAAE,MAAM,YAAY,CAAC;AAK3F,OAAO,EAA2B,KAAK,aAAa,EAAE,MAAM,wBAAwB,CAAC;AAOrF,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,mBAAmB,CAAC;AAuCrD,MAAM,WAAW,YAAY;IAC3B,OAAO,EAAE,MAAM,CAAC;IAChB,iBAAiB,EAAE,MAAM,CAAC;IAC1B,cAAc,EAAE,MAAM,CAAC;IACvB,gBAAgB,EAAE,MAAM,CAAC;IACzB,aAAa,EAAE,MAAM,CAAC;IACtB,iBAAiB,EAAE,MAAM,CAAC;IAC1B,oFAAoF;IACpF,UAAU,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,MAAM,CAAC;IAClB,wFAAwF;IACxF,kBAAkB,EAAE,MAAM,CAAC;CAC5B;AAED;;;;GAIG;AACH,MAAM,MAAM,aAAa,GAAG,CAAC,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,KAAK,OAAO,CAAC,aAAa,GAAG,IAAI,CAAC,CAAC;AAEnG,MAAM,WAAW,cAAc;IAC7B,OAAO,EAAE,MAAM,CAAC;IAChB,aAAa,EAAE,MAAM,CAAC;IACtB,SAAS,EAAE,MAAM,CAAC;CACnB;AA+XD,qBAAa,iBAAiB;IAW1B,OAAO,CAAC,YAAY,CAAC;IACrB,OAAO,CAAC,YAAY,CAAC;IACrB,OAAO,CAAC,UAAU,CAAC;IACnB,OAAO,CAAC,SAAS,CAAC;IAbpB,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAgB;IACvC,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAyB;IACvD,OAAO,CAAC,cAAc,CAA+C;IACrE,OAAO,CAAC,OAAO,CAAS;IACxB,OAAO,CAAC,WAAW,CAA4B;IAC/C,OAAO,CAAC,WAAW,CAAiC;IACpD,OAAO,CAAC,SAAS,CAAa;gBAG5B,MAAM,CAAC,EAAE,OAAO,CAAC,aAAa,CAAC,EACvB,YAAY,CAAC,GAAE,CAAC,OAAO,EAAE,MAAM,KAAK,YAAY,aAAA,EAChD,YAAY,CAAC,GAAE,MAAM,YAAY,aAAA,EACjC,UAAU,CAAC,GAAE,MAAM,MAAM,EAAE,aAAA,EAC3B,SAAS,CAAC,EAAE,aAAa,YAAA,EACjC,aAAa,CAAC,EAAE,OAAO,CAAC,aAAa,CAAC;IA8BxC;;;OAGG;IACH,cAAc,CAAC,EAAE,EAAE,WAAW,GAAG,IAAI;IAIrC;;OAEG;IACH,KAAK,IAAI,IAAI;IA0Bb;;OAEG;IACH,IAAI,IAAI,IAAI;IAOZ;;OAEG;IACG,IAAI,IAAI,OAAO,CAAC,YAAY,EAAE,CAAC;IAsIrC;;;;;;;;;OASG;YACW,YAAY;IA4M1B;;OAEG;IACH,OAAO,CAAC,oBAAoB;IA+B5B;;OAEG;IACH,OAAO,CAAC,uBAAuB;IAK/B;;OAEG;IACH,OAAO,CAAC,YAAY;IAsBpB;;OAEG;IACH,OAAO,CAAC,YAAY;IAWpB;;;OAGG;IACH,OAAO,CAAC,UAAU;IA8ClB;;OAEG;IACH,OAAO,CAAC,aAAa;IAarB;;;;;;;OAOG;IACG,sBAAsB,IAAI,OAAO,CAAC,IAAI,CAAC;IAgF7C;;OAEG;IACH,aAAa,CAAC,SAAS,EAAE,YAAY,GAAG,cAAc,EAAE;CAezD;AAID;;;GAGG;AACH,wBAAgB,aAAa,CAC3B,YAAY,EAAE,CAAC,OAAO,EAAE,MAAM,KAAK,YAAY,EAC/C,YAAY,EAAE,MAAM,YAAY,EAChC,UAAU,EAAE,MAAM,MAAM,EAAE,EAC1B,MAAM,CAAC,EAAE,OAAO,CAAC,aAAa,CAAC,EAC/B,SAAS,CAAC,EAAE,aAAa,EACzB,WAAW,CAAC,EAAE,WAAW,EACzB,aAAa,CAAC,EAAE,OAAO,CAAC,aAAa,CAAC,GACrC,iBAAiB,CAInB"}