@modusensus/dsh-mneme 0.3.8 → 0.4.1

package/README.md

@@ -5,7 +5,7 @@
 [![npm version](https://img.shields.io/npm/v/@modusensus/dsh-mneme?color=blue&label=npm)](https://www.npmjs.com/package/@modusensus/dsh-mneme)
 [![license](https://img.shields.io/badge/license-MIT-green)](LICENSE)
 [![Awesome](https://awesome-dsh-plugin.com/badge.svg)](https://github.com/awesome-dsh-plugin/awesome-dsh-plugin)
-[![tests](https://img.shields.io/badge/tests-443%20passed-success)](https://github.com/modusensus/dsh-mneme)
+[![tests](https://img.shields.io/badge/tests-447%20passed-success)](https://github.com/modusensus/dsh-mneme)
 
 > 给 DeepSeek Harness 的跨会话记忆插件：让 Agent 记住你、记住项目、自动整理记忆。**Mneme**（Μνήμη）——希腊记忆女神 Mnemosyne 之名，掌管记忆与梦境，正如 autoDream 在后台巩固记忆。
 
@@ -18,6 +18,11 @@
 - **SQLite 主存储**：`~/.dsh/memory/memory.db`，`node:sqlite` 内置，零原生依赖
 - **Markdown 镜像**：`preferences.md` / `projects.md` / `decisions.md` / `history.md` / `summary.md`，人类可读、可手工编辑（**人工修改优先**合并回库）
 - **4+1 种记忆类型**：`preference`（偏好）/ `project`（项目）/ `decision`（决策）/ `history`（历史）/ `summary`（总览）
+- **镜像同步状态机（v0.3.6+）**：mirror 与主库强一致，用 `generation`（期望轮次）/ `applied_generation`（已应用轮次）建模同步债务
+  - 业务写操作在**自身事务内原子递增** desired generation——崩溃在 COMMIT 后、渲染前，重启也能凭 durable 债务恢复，绝不静默跳过（v0.3.8）
+  - `generation` 用 SQLite 原子语句递增，多进程并发零丢失；带 `CHECK` 上界，负数/溢出拒绝
+  - 逐 type 记录 `committed / failed / pending` 回执，健康端点区分 `ok / degraded / unknown`
+  - 状态写失败不静默：同步失败落日志并留债务，重启自动收敛
 
 ### 模型工具（7 个）
 
@@ -106,6 +111,28 @@ v0.3.0 起新增**记忆基因**层：从记忆里抽取**命名实体**、**带
 
 > 📖 详见 [实体结构化记忆设计](docs/ENTITIES.md) · [语义增强架构](docs/SEMANTIC.md) · [本地模型部署指南](docs/LOCAL_MODEL.md) · [从 v0.1 升级说明](docs/MIGRATION.md)
 
+## 🆕 最近版本亮点
+
+| 版本 | 亮点 |
+|------|------|
+| **v0.3.8** | audit peer 复验 6 项运行时阻断全部修复：desired generation 同事务原子递增（崩溃窗口不再静默跳过）、同步失败不静默、原子 generation 增量（多进程零丢失）、逐 type committed/failed/pending 回执、读取失败显式 unknown、generation 上界/负数 CHECK |
+| **v0.3.7** | 启动竞态修复：人工编辑 md 镜像后重启向量重建失败（回灌移入 init 就绪后 + scheduleEmbed 就绪门） |
+| **v0.3.6** | mirror 同步状态机：generation/applied_generation 债务建模、F-NEW-03 mirror 健康状态、持久 dirty + 启动 recoverMirror |
+| **v0.3.0** | 记忆基因：实体/属性/关系三表 + 时间轴 + 实体搜索 + autoDream supersedes |
+
+## 🗺️ 进化路线图
+
+| 版本 | 状态 | 主题 | 说明 |
+|------|------|------|------|
+| v0.2.x | ✅ 完成 | 语义增强 + 反思更新 | 本地 Embedding/Rerank/聚类、`failure_memories` 失败追踪 |
+| v0.3.0 | ✅ 完成 | 记忆基因 | entities/attrs/relations 三表 + 时间轴 + 实体搜索 |
+| v0.3.6–0.3.8 | ✅ 完成 | 镜像一致性 + 审计加固 | generation 同步状态机、audit peer 6 项运行时阻断修复、447 测试全绿 |
+| **v0.4.0** | ⏳ 规划 | 反思性成长 | 纠错双向回流（改记忆同时反思"为什么记错/召回错"）+ 规则演进（从 failure 提炼规律注入系统提示）+ 自适应参数 |
+| **v0.4.1** | ✅ 完成 | 系统级睡眠 | Sleep 调度器（空闲检测 + 最小间隔，无 cron）+ 三层深度整理（冲突消解 / 归档降级 / 模式发现）+ create 决策 + 访问 touch 免生成号，471 测试全绿 |
+| **v0.5.0+** | 🚀 远期 | 自进化记忆 | 兴趣漂移跟踪 + 跨 workspace 记忆共享（等 DSH 支持） |
+
+> 新能力一律做成**可开关的功能**（配置启用/关闭），默认保守开启、不破坏现有行为。`failure_memories` 表与 autoDream 决策引擎已为 v0.4.0 铺好路。
+
 ## 📦 安装
 
 ### 前置条件
@@ -243,7 +270,7 @@ src/
 lib/
 ├── client.js         # Web 面板（手写 ModuleLoader bundle）
 └── *.js              # src 的同步分发产物
-test/                 # 443 个 node:test 测试（含审计与三轴线压测不变量）
+test/                 # 447 个 node:test 测试（含审计与三轴线压测不变量）
 scripts/              # e2e-dsh.js 端到端演示 · stress-dsh.js 三轴线压测 · sync-lib.js 同步
 ```
 
@@ -252,7 +279,7 @@ scripts/              # e2e-dsh.js 端到端演示 · stress-dsh.js 三轴线压
 ```bash
 cd dsh-mneme
 npm install        # 安装 peer 依赖（以 devDependencies 形式，用于本地测试）
-npm test           # 运行 443 个测试
+npm test           # 运行 447 个测试
 npm run stress     # 三轴线压测：长会话检索 / 冲突仲裁 / 多 Agent 并发（离线 mock LLM）
 npm run sync       # 把 src/ 同步到 lib/（发布时由 prepack 钩子自动执行）
 ```

package/lib/config.js

@@ -91,4 +91,32 @@ export const Config = z.object({
   entityExtractionMaxAttrs: z.natural().min(1).max(50).default(20),
   // Prefix/semantic search over entity names (used by recall).
   entitySearchEnabled: z.boolean().default(true),
+
+  // --- system-level sleep (v0.4.1) -----------------------------------------
+  // Opt-in: when false (default) the plugin never runs a sleep cycle, so the
+  // access-touch bookkeeping on recall/inject paths stays off too. Sleep is
+  // three phases: conflict resolution (reuses the dream conflict machinery),
+  // archival demotion (unrecalled memories tier down to summary then archive),
+  // and pattern discovery (LLM scans recent memories and mints type=pattern
+  // entries with evidence references).
+  sleepEnabled: z.boolean().default(false),
+  // A sleep run only fires when the store has been idle for this long and the
+  // last run is older than sleepMinIntervalHours. Idle detection replaces a
+  // cron-like schedule (DSH plugins have no resident crontab).
+  sleepIdleMinutes: z.natural().min(1).max(1440).default(30),
+  sleepMinIntervalHours: z.natural().min(1).max(168).default(8),
+  // Unrecalled (COALESCE(last_accessed_at, updated_at, created_at)) beyond
+  // sleepArchiveDays → demote: full body moves to _full_content, content
+  // becomes a one-line summary. Beyond sleepDeepArchiveDays → archive.
+  sleepArchiveDays: z.natural().min(1).max(365).default(30),
+  sleepDeepArchiveDays: z.natural().min(1).max(3650).default(90),
+  // How many most-recent memories the pattern-discovery pass scans.
+  sleepPatternScanCount: z.natural().min(10).max(500).default(100),
+  // Max patterns minted per sleep run (mirrors decisions maxCreatePerRun).
+  sleepMaxPatterns: z.natural().min(1).max(20).default(5),
+  // Optional LLM route override; empty = fall back to agentDefaultModel then
+  // the dream route. Distinct from dreamProvider/dreamModel so the sleep pass
+  // can pin a cheaper model for its bulk summarization.
+  sleepProvider: z.string().default(""),
+  sleepModel: z.string().default(""),
 });

package/lib/dream/decisions.js

@@ -1,4 +1,8 @@
-const ACTIONS = new Set(["keep", "merge", "archive", "conflict", "update"]);
+const ACTIONS = new Set(["keep", "merge", "archive", "conflict", "update", "create"]);
+
+// create is used by sleep pattern discovery (v0.4.1). It fabricates a new
+// memory of any known type (default pattern) rather than touching existing ids.
+const CREATE_TYPES = new Set(["pattern", "preference", "project", "decision", "history", "summary"]);
 
 /**
  * Validate a dream decision list against a snapshot of eligible memories.
@@ -9,6 +13,7 @@ const ACTIONS = new Set(["keep", "merge", "archive", "conflict", "update"]);
 export function validateDecisions(decisions, snapshot, options = {}) {
   const errors = [];
   const maxUpdatePerRun = options.maxUpdatePerRun ?? 2;
+  const maxCreatePerRun = options.maxCreatePerRun ?? 5;
   const minAgeHours = options.minAgeHours ?? 24;
   if (!Array.isArray(decisions) || decisions.length === 0) {
     return { ok: false, errors: ["decision list must be a non-empty array"] };
@@ -20,6 +25,35 @@ export function validateDecisions(decisions, snapshot, options = {}) {
       errors.push(`${at}: invalid action ${JSON.stringify(d?.action)}`);
       continue;
     }
+    // create claims no existing id: it fabricates a new memory, so it runs its
+    // own field validation and skips the ids-required check + the claimed set.
+    if (d.action === "create") {
+      if (typeof d.title !== "string" || !d.title.trim()) {
+        errors.push(`${at}: create needs non-empty title`);
+        continue;
+      }
+      if (typeof d.content !== "string" || !d.content.trim()) {
+        errors.push(`${at}: create needs non-empty content`);
+        continue;
+      }
+      if (d.importance !== undefined && (!Number.isInteger(d.importance) || d.importance < 1 || d.importance > 5)) {
+        errors.push(`${at}: create importance must be an integer 1-5 when provided`);
+        continue;
+      }
+      if (d.type !== undefined && (typeof d.type !== "string" || !CREATE_TYPES.has(d.type))) {
+        errors.push(`${at}: create type must be one of ${[...CREATE_TYPES].join(", ")}`);
+        continue;
+      }
+      if (d.evidence !== undefined && !Array.isArray(d.evidence)) {
+        errors.push(`${at}: create evidence must be an array of memory ids`);
+        continue;
+      }
+      if (d.tags !== undefined && !Array.isArray(d.tags)) {
+        errors.push(`${at}: create tags must be an array`);
+        continue;
+      }
+      continue;
+    }
     const ids = d.action === "conflict" ? [d.winner, d.loser] : (d.ids ?? []);
     if (d.action === "conflict") {
       if (!d.winner || !d.loser || d.winner === d.loser) {
@@ -95,6 +129,12 @@ export function validateDecisions(decisions, snapshot, options = {}) {
   if (updateCount > maxUpdatePerRun) {
     errors.push(`too many update decisions: ${updateCount} > ${maxUpdatePerRun}`);
   }
+  // Cap create churn: a pattern-discovery loop fabricating endless new memories
+  // would bloat the store, so a run can mint at most maxCreatePerRun.
+  const createCount = decisions.filter((d) => d.action === "create").length;
+  if (createCount > maxCreatePerRun) {
+    errors.push(`too many create decisions: ${createCount} > ${maxCreatePerRun}`);
+  }
   // Every snapshot id must appear in at least one decision
   for (const id of snapshot.keys()) {
     if (!claimed.has(id)) errors.push(`memory ${JSON.stringify(id)} missing from decisions`);
@@ -202,10 +242,48 @@ function applyOne(d, service, snapshot, config = {}) {
     case "archive": return applyArchive(d, service, snapshot);
     case "merge": return applyMerge(d, service, snapshot, config);
     case "conflict": return applyConflict(d, service, snapshot);
+    case "create": return applyCreate(d, service, snapshot);
     default: return applyUpdate(d, service, snapshot, config);
   }
 }
 
+/**
+ * Mint a new memory (sleep pattern discovery). saveWithDedupe dedupes by
+ * (type, title) so a replayed create merges instead of duplicating — the
+ * idempotency guard. Evidence ids are folded into tags as `ev:<id>` so a
+ * pattern's provenance stays queryable after creation.
+ */
+function applyCreate(d, service, snapshot) {
+  const evidence = Array.isArray(d.evidence) ? d.evidence : [];
+  const tags = [
+    ...(Array.isArray(d.tags) ? d.tags : []),
+    ...evidence.map((id) => `ev:${id}`)
+  ];
+  const result = service.saveWithDedupe({
+    type: d.type ?? "pattern",
+    title: d.title,
+    content: d.content,
+    importance: d.importance ?? 3,
+    tags,
+    source: "dream-create"
+  });
+  if (!result?.memory) return "skipped";
+  return {
+    applied: 1,
+    committed: {
+      action: "create",
+      id: result.memory.id,
+      type: d.type ?? "pattern",
+      title: d.title,
+      content: d.content,
+      importance: d.importance ?? 3,
+      evidence,
+      count_before: 0,
+      count_after: 1
+    }
+  };
+}
+
 function applyArchive(d, service, snapshot) {
   const targets = d.ids.filter((id) => {
     const mem = service.getById(id);

package/lib/index.js

@@ -5,6 +5,7 @@ import { createTools } from "./tools.js";
 import { createInjector } from "./inject.js";
 import { createSummarizer } from "./summarize.js";
 import { createDreamScheduler } from "./dream.js";
+import { createSleepScheduler, runSleep } from "./sleep.js";
 import { createApi } from "./api.js";
 import { createSettings } from "./settings.js";
 import { createCommandManager } from "./commands.js";
@@ -181,6 +182,25 @@ export const apply = (ctx, config) => {
     service.setDreamHook(() => dream.maybeSchedule(service));
   }
 
+  // Sleep scheduler (v0.4.1): idle-triggered deep pass (conflict resolution +
+  // archival demotion + pattern discovery). opt-in via sleepEnabled; writes
+  // through the service reset the idle clock (setSleepHook), and when the store
+  // stays quiet for sleepIdleMinutes past the sleepMinIntervalHours gate, the
+  // scheduler fires one cycle. The onRun closure reuses the same semantic
+  // pipeline as dream for the conflict phase.
+  let sleep = null;
+  if (cfg.sleepEnabled) {
+    sleep = createSleepScheduler({
+      service,
+      config: cfg,
+      logger: ctx.logger,
+      onRun: () => (sleep
+        ? runSleep(ctx, service, cfg, ctx.logger, { embedder, vectorIndex })
+        : Promise.resolve({ ok: true, skipped: true }))
+    });
+    service.setSleepHook(() => sleep.noteWrite());
+  }
+
   // Entity gene extraction (v0.3.0): wire the extractor into the service as a
   // hook so saveWithDedupe can fire-and-forget an extraction pass on fresh
   // writes. The service never sees ctx.llm — index.js adapts it here into the
@@ -251,6 +271,7 @@ export const apply = (ctx, config) => {
     }
     commands?.dispose();
     if (dream) await dream.dispose();
+    if (sleep) await sleep.dispose();
     store.close();
   };
 };

package/lib/service.js

@@ -9,6 +9,11 @@ export function createService({ store, mirror, config, onWrite, logger }) {
   // passed in the constructor). Fired on the same write events as onWrite.
   let dreamHook = null;
 
+  // Optional sleep scheduler hook (v0.4.1), installed via setSleepHook after
+  // creation. Fired on the same write events: it tells the sleep scheduler the
+  // store just changed so the idle-detection clock resets.
+  let sleepHook = null;
+
   // Optional vector embedder, installed via setEmbedder after creation. After
   // any content write it fire-and-forgets a re-embed of the row so vector
   // search stays in sync; failures are swallowed inside the embedder.
@@ -37,6 +42,19 @@ export function createService({ store, mirror, config, onWrite, logger }) {
   // replays them exactly once against the committed state.
   let txDepth = 0;
 
+  // Serial task queue (sleep v0.4.1). Long-running background passes — dream
+  // consolidation, sleep cycles — must never overlap: two sleep runs racing
+  // would double-demote or double-mint patterns. enqueue chains the task onto
+  // a promise tail so N callers can queue work that runs strictly one at a
+  // time. A task that rejects doesn't poison the queue (the tail swallows the
+  // rejection) but the rejection still propagates to that caller.
+  let queueTail = Promise.resolve();
+  function enqueue(fn) {
+    const next = queueTail.then(fn, fn);
+    queueTail = next.catch(() => {});
+    return next;
+  }
+
   // issue #6 (part 2): startup race defense. A local embedder (LocalEmbedder /
   // Ollama) exposes an async init(), so between `setEmbedder` and init()
   // resolving there is a window where embedSingle would throw "not initialized"
@@ -243,11 +261,15 @@ export function createService({ store, mirror, config, onWrite, logger }) {
     // entity:/attr: 前缀路由（v0.3.0 Phase 3）。entitySearchEnabled 关闭时走原逻辑。
     if (config?.entitySearchEnabled) {
       if (q.startsWith("entity:")) {
-        return searchByEntity(q.slice(7).trim(), options);
+        const hits = searchByEntity(q.slice(7).trim(), options);
+        touchRecalled(hits);
+        return hits;
       }
       if (q.startsWith("attr:")) {
         const [key, value] = q.slice(5).split("=");
-        return searchByAttr(key, value, options);
+        const hits = searchByAttr(key, value, options);
+        touchRecalled(hits);
+        return hits;
       }
     }
 
@@ -345,6 +367,7 @@ export function createService({ store, mirror, config, onWrite, logger }) {
         });
       } catch { /* recall receipt is best effort */ }
     }
+    touchRecalled(result);
     return result;
   }
 
@@ -362,6 +385,9 @@ export function createService({ store, mirror, config, onWrite, logger }) {
     if (dreamHook) {
       try { dreamHook(); } catch { /* ignore */ }
     }
+    if (sleepHook) {
+      try { sleepHook(); } catch { /* ignore */ }
+    }
   }
 
   /**
@@ -430,6 +456,23 @@ export function createService({ store, mirror, config, onWrite, logger }) {
     return { action: "created", memory: created };
   }
 
+  /**
+   * Sleep touch (v0.4.1): when sleep is enabled, any memory surfaced by recall
+   * or auto-injection gets its last_accessed_at bumped, so the "unrecalled N
+   * days → demote/archive" tiering counts real access. Best-effort and gated on
+   * config.sleepEnabled — when sleep is off this is a complete no-op (no writes
+   * on the hot recall path). A touch failure must never break search/inject.
+   */
+  function touchRecalled(memories) {
+    if (config?.sleepEnabled !== true || !Array.isArray(memories) || memories.length === 0) return;
+    for (const m of memories) {
+      if (!m?.id) continue;
+      try {
+        store.touchAccess(m.id);
+      } catch { /* touch is best effort */ }
+    }
+  }
+
   /**
    * Candidate memories for automatic context injection:
    * summaries first, then all preferences, then non-forgotten items with
@@ -446,7 +489,9 @@ export function createService({ store, mirror, config, onWrite, logger }) {
         const pb = b.type === "summary" ? 0 : b.type === "preference" ? 1 : 2;
         return pa - pb || b.importance - a.importance;
       });
-    return items.slice(0, maxItems);
+    const selected = items.slice(0, maxItems);
+    touchRecalled(selected);
+    return selected;
   }
 
   /**
@@ -758,7 +803,9 @@ export function createService({ store, mirror, config, onWrite, logger }) {
     mergeHumanEdits,
     toApiList,
     transaction,
+    enqueue,
     setDreamHook(fn) { dreamHook = fn; },
+    setSleepHook(fn) { sleepHook = fn; },
     setEmbedder(emb) {
       embedder = emb;
       if (!emb) {
@@ -870,6 +917,11 @@ export function createService({ store, mirror, config, onWrite, logger }) {
       afterSync("write");
       return updated;
     },
+    demoteToSummary: (id, summary, opts) => {
+      const updated = store.demoteToSummary(id, summary, opts);
+      afterSync("write");
+      return updated;
+    },
     // autoDream audit trail: passthroughs deliberately bypass write hooks —
     // an audit write is bookkeeping, and notifyWrite would loop back into the
     // dream scheduler that just recorded the run.