recallnest 1.4.1 → 2.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (233) hide show
  1. package/.env +4 -0
  2. package/CLAUDE.md +7 -3
  3. package/README.md +202 -130
  4. package/README_CN.md +263 -190
  5. package/bin/recallnest.mjs +62 -0
  6. package/config.json +55 -0
  7. package/daily-recall.sh +23 -0
  8. package/integrations/setup-all.sh +31 -0
  9. package/logs/daily-recall.error.log +0 -0
  10. package/logs/daily-recall.log +30 -0
  11. package/logs/ingest-2026-04-11.log +110 -0
  12. package/logs/ingest-2026-04-12.log +90 -0
  13. package/logs/launchagent-stderr.log +0 -0
  14. package/logs/launchagent-stdout.log +0 -0
  15. package/package.json +10 -3
  16. package/src/access-tracker.ts +3 -2
  17. package/src/activity-counter.ts +109 -0
  18. package/src/adaptive-retrieval.ts +6 -5
  19. package/src/admission-control.ts +163 -0
  20. package/src/anchor-generator.ts +80 -0
  21. package/src/api-server.ts +10 -0
  22. package/src/audit-log.ts +122 -0
  23. package/src/auto-gc.ts +98 -29
  24. package/src/auto-recall.ts +33 -4
  25. package/src/capture-engine.ts +372 -23
  26. package/src/cascade-forget.ts +3 -2
  27. package/src/cli.ts +107 -5
  28. package/src/compat.ts +22 -0
  29. package/src/confidence-tracker.ts +210 -47
  30. package/src/conflict-store.ts +12 -3
  31. package/src/consolidation-engine.ts +296 -8
  32. package/src/context-collapse-renderer.ts +222 -0
  33. package/src/context-composer-stable-selection.ts +5 -1
  34. package/src/context-composer.ts +239 -4
  35. package/src/context-reconstructor.ts +404 -0
  36. package/src/conversation-importer.ts +441 -0
  37. package/src/data-checkup.ts +232 -0
  38. package/src/decay-engine.ts +84 -0
  39. package/src/distill-lock.ts +122 -0
  40. package/src/doctor.ts +8 -10
  41. package/src/dream-pipeline.ts +227 -0
  42. package/src/emotion-detector.ts +90 -0
  43. package/src/eval.ts +3 -2
  44. package/src/forget-engine.ts +269 -0
  45. package/src/frequency-tracker.ts +175 -0
  46. package/src/graph-export.ts +644 -0
  47. package/src/ingest.ts +36 -6
  48. package/src/interference-detector.ts +187 -0
  49. package/src/kg-extractor.ts +15 -24
  50. package/src/kg-store.ts +2 -1
  51. package/src/language-hook.ts +67 -0
  52. package/src/llm-client.ts +289 -6
  53. package/src/mcp-server.ts +707 -80
  54. package/src/memory-assets.ts +22 -5
  55. package/src/memory-evolution.ts +365 -0
  56. package/src/memory-lint.ts +429 -0
  57. package/src/memory-output.ts +69 -0
  58. package/src/memory-schema.ts +74 -2
  59. package/src/narrative-schema.ts +79 -0
  60. package/src/narrative-tagger.ts +164 -0
  61. package/src/pii-detector.ts +98 -0
  62. package/src/post-retrieval-filter.ts +76 -0
  63. package/src/prediction-engine.ts +253 -0
  64. package/src/preference-slots.ts +65 -2
  65. package/src/prospective-memory.ts +215 -0
  66. package/src/query-expander.ts +67 -0
  67. package/src/recall-governor.ts +183 -0
  68. package/src/retention-policy.ts +90 -0
  69. package/src/retriever.ts +828 -55
  70. package/src/rif.ts +34 -8
  71. package/src/runtime-config.ts +23 -8
  72. package/src/scope-inventory.ts +4 -3
  73. package/src/scope-policy.ts +4 -0
  74. package/src/session-distiller.ts +307 -0
  75. package/src/session-output.ts +40 -0
  76. package/src/session-schema.ts +37 -0
  77. package/src/session-store.ts +7 -2
  78. package/src/skill-engine.ts +208 -0
  79. package/src/skill-promotion.ts +293 -0
  80. package/src/skill-schema.ts +39 -0
  81. package/src/store.ts +96 -10
  82. package/src/topic-tag.ts +66 -0
  83. package/src/tracker.ts +3 -1
  84. package/src/ui-server.ts +80 -8
  85. package/src/workflow-observation-scope-review.ts +2 -1
  86. package/src/workflow-observation-store.ts +2 -1
  87. package/src/write-verifier.ts +126 -0
  88. package/.claude-plugin/CLAUDE.md +0 -29
  89. package/.claude-plugin/marketplace.json +0 -18
  90. package/.github/workflows/ci.yml +0 -28
  91. package/.mcp.json +0 -9
  92. package/CHANGELOG.md +0 -51
  93. package/DEMO.md +0 -94
  94. package/EVAL.md +0 -89
  95. package/FAILURES.md +0 -75
  96. package/OPERATIONS.md +0 -59
  97. package/ROADMAP.md +0 -198
  98. package/assets/recallnest-workbench.png +0 -0
  99. package/assets/ui/app.js +0 -807
  100. package/assets/ui/index.html +0 -154
  101. package/assets/ui/styles.css +0 -605
  102. package/assets/wechat-qr.jpg +0 -0
  103. package/daily-reflections/2026-03-05.md +0 -15
  104. package/daily-reflections/2026-03-06.md +0 -15
  105. package/daily-reflections/2026-03-07.md +0 -7
  106. package/daily-reflections/2026-03-08.md +0 -34
  107. package/daily-reflections/2026-03-09.md +0 -35
  108. package/daily-reflections/2026-03-10.md +0 -36
  109. package/daily-reflections/2026-03-11.md +0 -35
  110. package/daily-reflections/2026-03-12.md +0 -36
  111. package/daily-reflections/2026-03-13.md +0 -35
  112. package/daily-reflections/2026-03-14.md +0 -35
  113. package/daily-reflections/2026-03-15.md +0 -35
  114. package/daily-reflections/2026-03-16.md +0 -41
  115. package/daily-reflections/2026-03-17.md +0 -10
  116. package/daily-reflections/2026-03-18.md +0 -10
  117. package/daily-reflections/2026-03-19.md +0 -10
  118. package/daily-reflections/2026-03-20.md +0 -10
  119. package/daily-reflections/2026-03-21.md +0 -10
  120. package/daily-reflections/2026-03-22.md +0 -10
  121. package/daily-reflections/2026-03-23.md +0 -10
  122. package/daily-reflections/2026-03-24.md +0 -10
  123. package/daily-reflections/2026-03-25.md +0 -32
  124. package/daily-reflections/2026-03-26.md +0 -37
  125. package/daily-reflections/2026-03-27.md +0 -35
  126. package/daily-reflections/2026-03-28.md +0 -35
  127. package/daily-reflections/2026-03-29.md +0 -35
  128. package/daily-reflections/2026-03-30.md +0 -36
  129. package/daily-reflections/2026-03-31.md +0 -35
  130. package/daily-reflections/2026-04-01.md +0 -35
  131. package/docs/api-reference.md +0 -1291
  132. package/docs/architecture.md +0 -168
  133. package/docs/memory-boundary-contract.md +0 -226
  134. package/docs/memory-categories.md +0 -119
  135. package/docs/pipeline-audit-ultramemory.md +0 -177
  136. package/docs/plans/2026-03-08-query-expansion.md +0 -270
  137. package/docs/plans/2026-03-16-continuity-layer.md +0 -196
  138. package/docs/plans/mcp-tiering-plan.md +0 -126
  139. package/docs/research/recall-quality-audit-2026-03-22.md +0 -153
  140. package/docs/self-evolution.md +0 -198
  141. package/docs/superpowers/plans/2026-03-19-ingest-pipeline-upgrade.md +0 -782
  142. package/docs/superpowers/specs/2026-03-19-ingest-pipeline-upgrade-design.md +0 -77
  143. package/docs/superpowers/specs/2026-03-23-pull-claude-desktop-design.md +0 -95
  144. package/eval/continuity/results-2026-03-16.md +0 -117
  145. package/eval/continuity/results-2026-03-19.md +0 -453
  146. package/eval/continuity/runbook.md +0 -110
  147. package/eval/continuity/seed-cases.md +0 -54
  148. package/eval/reports/2026-03-06-abstract-query-baseline.md +0 -96
  149. package/eval/reports/2026-03-06-aws-ssh-baseline.md +0 -82
  150. package/eval/reports/2026-03-06-baseline.md +0 -82
  151. package/eval/reports/query-expansion.md +0 -138
  152. package/lm +0 -19
  153. package/plan-v1.2-distributable.md +0 -105
  154. package/scripts/archive-distilled-junk.ts +0 -116
  155. package/scripts/backfill-l0l1l2.ts +0 -98
  156. package/scripts/blind-scan.ts +0 -588
  157. package/scripts/cleanup-step1-exact-dups.ts +0 -117
  158. package/scripts/cleanup-step2-template-dups.ts +0 -113
  159. package/scripts/cleanup-step3-stale-content.ts +0 -149
  160. package/scripts/cleanup-step4-orphan-scopes.ts +0 -131
  161. package/scripts/delete-distilled-junk.ts +0 -88
  162. package/scripts/distill-facts.ts +0 -305
  163. package/scripts/health-check.ts +0 -343
  164. package/scripts/incremental-ingest.sh +0 -71
  165. package/scripts/mlp-health-check.ts +0 -382
  166. package/scripts/pull-claude-desktop.ts +0 -515
  167. package/scripts/re-ingest-cc.sh +0 -66
  168. package/scripts/rebalance-memory-health.ts +0 -181
  169. package/scripts/run-desktop-ingest.sh +0 -11
  170. package/scripts/start-server.sh +0 -12
  171. package/scripts/viz-dashboard.html +0 -423
  172. package/scripts/viz-export.ts +0 -703
  173. package/src/__tests__/access-tracker-novelty.test.ts +0 -228
  174. package/src/__tests__/auto-consolidation.test.ts +0 -168
  175. package/src/__tests__/auto-recall.test.ts +0 -131
  176. package/src/__tests__/batch-dedup.test.ts +0 -65
  177. package/src/__tests__/capture-engine.test.ts +0 -1142
  178. package/src/__tests__/capture-heuristic.test.ts +0 -171
  179. package/src/__tests__/cascade-forget.test.ts +0 -210
  180. package/src/__tests__/category-min-scores.test.ts +0 -33
  181. package/src/__tests__/confidence-tracker.test.ts +0 -196
  182. package/src/__tests__/conflict-advisor.test.ts +0 -206
  183. package/src/__tests__/conflict-escalation.test.ts +0 -149
  184. package/src/__tests__/conflict-lifecycle.test.ts +0 -83
  185. package/src/__tests__/conflict-output.test.ts +0 -85
  186. package/src/__tests__/conflict-store.test.ts +0 -374
  187. package/src/__tests__/consolidation-engine.test.ts +0 -181
  188. package/src/__tests__/context-composer-pins.test.ts +0 -64
  189. package/src/__tests__/context-composer-task-selection.test.ts +0 -24
  190. package/src/__tests__/context-renderer.test.ts +0 -91
  191. package/src/__tests__/continuity-eval.test.ts +0 -275
  192. package/src/__tests__/continuity-schema.test.ts +0 -97
  193. package/src/__tests__/core-recall-eval.test.ts +0 -235
  194. package/src/__tests__/core-summary.test.ts +0 -109
  195. package/src/__tests__/dedup-actions.test.ts +0 -58
  196. package/src/__tests__/deterministic-id.test.ts +0 -39
  197. package/src/__tests__/doctor.test.ts +0 -303
  198. package/src/__tests__/drill-down.test.ts +0 -108
  199. package/src/__tests__/embedder.test.ts +0 -122
  200. package/src/__tests__/entity-resolver.test.ts +0 -58
  201. package/src/__tests__/event-segmenter.test.ts +0 -153
  202. package/src/__tests__/hotness-blend.test.ts +0 -76
  203. package/src/__tests__/ingest.test.ts +0 -360
  204. package/src/__tests__/kg-extractor.test.ts +0 -184
  205. package/src/__tests__/kg-store.test.ts +0 -198
  206. package/src/__tests__/llm-consolidation.test.ts +0 -226
  207. package/src/__tests__/memory-boundaries.test.ts +0 -237
  208. package/src/__tests__/memory-health-rebalance.test.ts +0 -144
  209. package/src/__tests__/memory-output.test.ts +0 -215
  210. package/src/__tests__/multi-vector.test.ts +0 -242
  211. package/src/__tests__/ppr-traversal.test.ts +0 -256
  212. package/src/__tests__/preference-guard-smoke.test.ts +0 -211
  213. package/src/__tests__/preference-matcher.test.ts +0 -230
  214. package/src/__tests__/preference-slots.test.ts +0 -106
  215. package/src/__tests__/prospective-memory.test.ts +0 -283
  216. package/src/__tests__/query-entity-detector.test.ts +0 -171
  217. package/src/__tests__/query-expander.test.ts +0 -34
  218. package/src/__tests__/recall-mode.test.ts +0 -64
  219. package/src/__tests__/result-synthesizer.test.ts +0 -198
  220. package/src/__tests__/resume-context.test.ts +0 -4743
  221. package/src/__tests__/retrieval-trace.test.ts +0 -213
  222. package/src/__tests__/retriever-boundary-weight.test.ts +0 -136
  223. package/src/__tests__/retroactive-boost.test.ts +0 -223
  224. package/src/__tests__/scope-inventory.test.ts +0 -200
  225. package/src/__tests__/scope-policy.test.ts +0 -69
  226. package/src/__tests__/session-checkpoint.test.ts +0 -259
  227. package/src/__tests__/session-gc.test.ts +0 -215
  228. package/src/__tests__/store-scope-audit.test.ts +0 -81
  229. package/src/__tests__/temporal-parser.test.ts +0 -241
  230. package/src/__tests__/term-registry.test.ts +0 -121
  231. package/src/__tests__/tool-output-compressor.test.ts +0 -133
  232. package/src/__tests__/version-manager.test.ts +0 -213
  233. package/src/__tests__/workflow-observation.test.ts +0 -400
@@ -1,177 +0,0 @@
1
- # Pipeline Audit: RecallNest vs UltraMemory
2
-
3
- > Auditor: CC (Claude Code Opus 4.6)
4
- > Date: 2026-03-28
5
- > Reference: trihippo/UltraMemory v1.1.0-beta.10 (private)
6
- > Target: AliceLJY/recallnest (~/recallnest)
7
- > Method: Three-Lens Pipeline Audit (pipeline-borrower skill)
8
-
9
- ---
10
-
11
- ## Lens 1: User Lens (Daily Experience)
12
-
13
- | Pain Point | Stage | Severity |
14
- |---|---|---|
15
- | No real-time capture from live conversations — must run `ingest` manually | CAPTURE | High |
16
- | Same memory stored twice gets different UUIDs — canonical key helps but ID-level dedup missing | STORE | Medium |
17
- | Recall results not reordered by task context — same ranking for "fix a bug" vs "write an article" | RETRIEVE | Medium |
18
- | No one-click memory cleanup — conflict resolution is manual, no automated clustering | DECAY | Medium |
19
- | "TS" and "TypeScript" treated as different entities in search | RETRIEVE | Low |
20
-
21
- ## Lens 2: Developer Lens (Pipeline Gaps)
22
-
23
- ### Stage 1: CAPTURE
24
-
25
- **Gap: No heuristic pre-capture for live conversations**
26
-
27
- - **Our code:** `capture-engine.ts:87-105` — Only accepts explicit `CaptureMemoryInput` from tool calls
28
- - **Their solution:** `packages/core/src/auto-capture.ts:1-60` — `shouldCapture(text)` rejects greetings/noise/short text; `extractHeuristic(text)` extracts preference/identity/decision/correction signals with zero LLM calls
29
- - **Portable?** Yes — pure regex/heuristic, no dependencies
30
- - **Proposed fix:**
31
- - Create `src/capture-heuristic.ts` with `shouldCapture()` + `extractHeuristic()`
32
- - Hook into MCP `store_memory` tool as optional pre-filter
33
- - Add new MCP tool `auto_capture` for live conversation analysis
34
- - Estimated: ~120 LOC
35
- - **Priority: High** — biggest user pain point, lowest effort
36
-
37
- ### Stage 4: STORE
38
-
39
- **Gap: Random UUIDs allow duplicate entries with different IDs**
40
-
41
- - **Our code:** `store.ts:327` — `randomUUID()` for every new memory
42
- - **Their solution:** `packages/core/src/utils/deterministic-id.ts:1-20` — UUID v5 from `scope + text` → same content always gets same ID
43
- - **Portable?** Yes — single function, `uuid` package already available
44
- - **Proposed fix:**
45
- - Add `deterministicId(scope, text)` to `store.ts`
46
- - Use for durable memories when canonical key exists; keep random UUID for evidence/ephemeral
47
- - Estimated: ~25 LOC
48
- - **Priority: High** — prevents data pollution, trivial to implement
49
-
50
- ### Stage 5: RETRIEVE
51
-
52
- **Gap 5a: Single vector search — no multi-granularity retrieval**
53
-
54
- - **Our code:** `store.ts:44` — Single `vector` column; `retriever.ts:350` — searches only one vector
55
- - **Their solution:** `packages/core/src/store.ts:297-309` — 3 vector columns (vector, vector_l0, vector_l1, vector_l2) with auto-migration; `packages/core/src/retriever.ts:143-147` — weighted blend (vector:0.35, bm25:0.30, l0:0.15, l1:0.10, l2:0.10)
56
- - **Portable?** Medium — requires LanceDB schema migration + embedding pipeline change
57
- - **Proposed fix:**
58
- - Add `vector_l0`, `vector_l1`, `vector_l2` columns to LanceDB schema
59
- - Generate L0 (1-sentence abstract), L1 (bullet), L2 (full) at embed time
60
- - Blend scores in retriever with configurable weights
61
- - Estimated: ~200 LOC (store migration + retriever weights + embedder pipeline)
62
- - **Priority: Medium** — high value but high effort, RecallNest's metadata already stores l0/l1/l2 text but not as separate vectors
63
- - **Dependency:** Needs embedding cost analysis (3x API calls per memory)
64
-
65
- **Gap 5b: No per-category score thresholds**
66
-
67
- - **Our code:** `retriever.ts:59,109-121` — Single global `hardMinScore` (0.35)
68
- - **Their solution:** `packages/core/src/retriever.ts:97-108` — `categoryScoreThresholds` map with per-category overrides
69
- - **Portable?** Yes — optional config, backward-compatible
70
- - **Proposed fix:**
71
- - Add `categoryMinScores?: Record<string, number>` to `RetrievalConfig`
72
- - In scoring pipeline, check category-specific threshold before global
73
- - Default: preferences=0.25, entities=0.30, events=0.35, cases=0.40, patterns=0.45, profile=0.25
74
- - Estimated: ~30 LOC
75
- - **Priority: Medium** — improves recall precision per category
76
-
77
- **Gap 5c: No entity resolution in search**
78
-
79
- - **Our code:** `query-expander.ts` — hardcoded synonym dict (~20 groups), no entity normalization
80
- - **Their solution:** `packages/core/src/entity-resolver.ts:1-62` — `EntityResolver` class with builtin + user aliases, resolves TS→typescript, React.js→react, DB aliases
81
- - **Portable?** Yes — pure mapping logic
82
- - **Proposed fix:**
83
- - Create `src/entity-resolver.ts` with tech entity mappings
84
- - Hook into `query-expander.ts` before synonym expansion
85
- - Also normalize entities at capture time (before canonical key generation)
86
- - Estimated: ~80 LOC
87
- - **Priority: Low** — nice-to-have, query-expander partially covers this
88
-
89
- ### Stage 6: INJECT
90
-
91
- **Gap: No contextual reordering of recall results**
92
-
93
- - **Our code:** `context-composer.ts:41-100` — scope-based prioritization only; `retriever.ts:346` — returns results in score order
94
- - **Their solution:** `packages/core/src/context-renderer.ts:1-98` — `renderMemories()` reorders by 60% vector score + 40% term overlap, with task context awareness
95
- - **Portable?** Yes — pure function, no dependencies
96
- - **Proposed fix:**
97
- - Create `src/context-renderer.ts` with `renderMemories()`
98
- - Hook into `context-composer.ts` after retrieval, before budget cutting
99
- - Also hook into `search_memory` tool result formatting
100
- - Estimated: ~100 LOC
101
- - **Priority: Medium** — improves relevance of injected context
102
-
103
- ### Stage 7: DECAY
104
-
105
- **Gap: No automated semantic consolidation**
106
-
107
- - **Our code:** `memory-health-rebalance.ts:145-197` — tier rebalancing only; `conflict-engine.ts` — passive conflict detection on write
108
- - **Their solution:** `packages/core/src/consolidation-engine.ts:1-388` — `ConsolidationEngine.run(scope)` scans all entries, clusters by vector similarity (>0.82), merges near-duplicates (>0.92), detects contradictions, adds relation links
109
- - **Portable?** Medium — needs store's vectorSearch API; RecallNest already has it
110
- - **Proposed fix:**
111
- - Create `src/consolidation-engine.ts` adapting UltraMemory's approach
112
- - Reuse RecallNest's `store.vectorSearch()` and `cosineSimilarity()`
113
- - Add MCP tool `consolidate_memories` (scope, threshold params)
114
- - Integrate with existing `conflict-engine.ts` for conflict creation
115
- - Estimated: ~250 LOC
116
- - **Priority: Medium** — automates manual cleanup; high value at scale
117
-
118
- ## Lens 3: Evidence Table
119
-
120
- | Feature | RecallNest | UltraMemory | Status |
121
- |---|---|---|---|
122
- | Hybrid retrieval (vector + BM25 + RRF) | `retriever.ts:350` | `retriever.ts:250` | **both have** |
123
- | Cross-encoder reranking | `retriever.ts:400+` (Jina/Voyage/Pinecone/vLLM) | `retriever.ts:500+` (Jina) | **our advantage** (more providers) |
124
- | Weibull decay + tier system | `decay-engine.ts:145-159` (β per tier) | `decay-engine.ts:111` (retention presets) | **our advantage** (more sophisticated) |
125
- | Conflict detection + lifecycle | `conflict-engine.ts` + `conflict-store.ts` + `conflict-lifecycle.ts` | `conflict-detector.ts` (detect only, no lifecycle) | **our advantage** |
126
- | Canonical key dedup | `memory-boundaries.ts:125-148` | None | **our advantage** |
127
- | Workflow observation | `workflow-observation-engine.ts` (7 files) | None | **our advantage** |
128
- | Checkpoint sanitization | `session-engine.ts` (strips repo state) | None | **our advantage** |
129
- | Retrieval profiles (4 presets) | `retrieval-profiles.ts` (writing/debug/fact-check/default) | None (single config) | **our advantage** |
130
- | Provenance chain (20 levels) | `memory-boundaries.ts` (provenanceHistory) | None | **our advantage** |
131
- | MCP tool tier control | `mcp-server.ts` (core/advanced/full) | None (all exposed) | **our advantage** |
132
- | Multi-vector L0/L1/L2 embedding | Metadata storage only (`mcp-server.ts:1083`) | Full pipeline (`store.ts:297`, `retriever.ts:143`) | **their advantage** |
133
- | Auto-capture heuristic | None | `auto-capture.ts:1-60` (zero LLM) | **their advantage** |
134
- | Semantic consolidation | Rebalancing only (`memory-health-rebalance.ts`) | Full engine (`consolidation-engine.ts:1-388`) | **their advantage** |
135
- | Context renderer | Scope-based only (`context-composer.ts`) | Intent-based (`context-renderer.ts:1-98`) | **their advantage** |
136
- | Deterministic IDs | Random UUID (`store.ts:327`) | UUID v5 (`deterministic-id.ts`) | **their advantage** |
137
- | Entity resolver | None | `entity-resolver.ts:1-62` | **their advantage** |
138
- | Per-category score thresholds | Profile-based only (`retrieval-profiles.ts`) | Category-based (`retriever.ts:97-108`) | **their advantage** |
139
-
140
- **Score: RecallNest 10 advantages, UltraMemory 7 advantages.** RecallNest's strengths are in governance/lifecycle; UltraMemory's are in retrieval precision.
141
-
142
- ---
143
-
144
- ## Phase 4: Implementation Plan
145
-
146
- ### Phase 1 (No deps, can start immediately)
147
-
148
- | # | Gap | File | LOC | Priority |
149
- |---|---|---|---|---|
150
- | 1 | Auto-capture heuristic | new `capture-heuristic.ts` | ~120 | **High** |
151
- | 2 | Deterministic IDs | `store.ts:327` | ~25 | **High** |
152
- | 3 | Per-category score thresholds | `retriever.ts:59` | ~30 | Medium |
153
-
154
- ### Phase 2 (Depends on Phase 1)
155
-
156
- | # | Gap | File | LOC | Priority |
157
- |---|---|---|---|---|
158
- | 4 | Entity resolver | new `entity-resolver.ts` → hooks into capture + query | ~80 | Low |
159
- | 5 | Context renderer | new `context-renderer.ts` → hooks into composer | ~100 | Medium |
160
-
161
- ### Phase 3 (Depends on vectorSearch stability)
162
-
163
- | # | Gap | File | LOC | Priority |
164
- |---|---|---|---|---|
165
- | 6 | Consolidation engine | new `consolidation-engine.ts` + MCP tool | ~250 | Medium |
166
- | 7 | Multi-vector L0/L1/L2 | `store.ts` + `retriever.ts` + `embedder.ts` | ~200 | Medium |
167
-
168
- ### Total estimated: ~805 LOC across 7 proposals
169
-
170
- **Recommended order:** 1 → 2 → 3 → 5 → 6 → 4 → 7
171
-
172
- Rationale: Start with biggest pain point (auto-capture) and easiest win (deterministic IDs). Multi-vector last because it's highest effort and needs embedding cost analysis (3x API calls).
173
-
174
- ---
175
-
176
- *Pipeline audit complete. Methodology: Three-Lens (User/Developer/Evidence).*
177
- *Key lesson applied: Be Team B (pipeline gaps), not Team A (feature envy).*
@@ -1,270 +0,0 @@
1
- # Query Expansion for Fuzzy Chinese Search
2
-
3
- > **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
4
-
5
- **Goal:** Add a lightweight Chinese synonym/colloquial-to-technical query expansion layer before retrieval, improving fuzzy search hit rate without adding API dependencies.
6
-
7
- **Architecture:** New `query-expander.ts` module with a static synonym dictionary. Called in `retriever.ts` before BM25 search — expands the original query into additional BM25 search terms. Vector search stays unchanged (Jina embeddings already handle semantic similarity). The expansion only boosts BM25 recall for terms that the user says differently from how they're stored.
8
-
9
- **Tech Stack:** Pure TypeScript, no new dependencies. Static dictionary file.
10
-
11
- ---
12
-
13
- ### Task 1: Create synonym dictionary + expander module
14
-
15
- **Files:**
16
- - Create: `src/query-expander.ts`
17
- - Test: `src/__tests__/query-expander.test.ts`
18
-
19
- **Step 1: Write the failing test**
20
-
21
- ```typescript
22
- // src/__tests__/query-expander.test.ts
23
- import { describe, it, expect } from "bun:test";
24
- import { expandQuery } from "../query-expander.js";
25
-
26
- describe("expandQuery", () => {
27
- it("expands colloquial Chinese to technical terms", () => {
28
- const result = expandQuery("bot 突然挂了");
29
- expect(result).toContain("崩溃");
30
- expect(result).toContain("crash");
31
- expect(result).toContain("挂了");
32
- });
33
-
34
- it("expands fuzzy feeling queries", () => {
35
- const result = expandQuery("AI 到底有没有感受");
36
- expect(result).toContain("意识");
37
- expect(result).toContain("consciousness");
38
- expect(result).toContain("感受");
39
- });
40
-
41
- it("preserves original query terms", () => {
42
- const result = expandQuery("配图风格");
43
- expect(result).toContain("配图");
44
- expect(result).toContain("风格");
45
- });
46
-
47
- it("returns original for already-precise queries", () => {
48
- const result = expandQuery("JINA_API_KEY");
49
- expect(result).toBe("JINA_API_KEY");
50
- });
51
-
52
- it("handles empty/short queries", () => {
53
- expect(expandQuery("")).toBe("");
54
- expect(expandQuery("hi")).toBe("hi");
55
- });
56
- });
57
- ```
58
-
59
- **Step 2: Run test to verify it fails**
60
-
61
- Run: `cd ~/local-memory && bun test src/__tests__/query-expander.test.ts`
62
- Expected: FAIL — module not found
63
-
64
- **Step 3: Write the implementation**
65
-
66
- ```typescript
67
- // src/query-expander.ts
68
- /**
69
- * Lightweight Chinese query expansion via static synonym dictionary.
70
- * Expands colloquial/fuzzy terms into technical equivalents for BM25 boost.
71
- * No API calls — pure local dictionary lookup.
72
- */
73
-
74
- // Each entry: [trigger patterns, expansion terms]
75
- // Trigger: if any pattern matches (substring), add all expansion terms to query
76
- const SYNONYM_MAP: Array<[string[], string[]]> = [
77
- // --- Status / Failure ---
78
- [["挂了", "挂掉", "宕机", "down"], ["崩溃", "crash", "error", "报错", "挂了", "宕机", "失败"]],
79
- [["卡住", "卡死", "没反应"], ["hang", "timeout", "超时", "卡住", "无响应", "stuck"]],
80
- [["炸了", "爆了"], ["崩溃", "crash", "OOM", "内存溢出", "error"]],
81
-
82
- // --- AI / Consciousness ---
83
- [["感受", "感觉", "情感"], ["意识", "consciousness", "experiencing", "感受", "情感", "qualia"]],
84
- [["有没有意识", "是否有意识"], ["consciousness", "意识", "sentience", "感知", "自我意识"]],
85
- [["自由意志"], ["free will", "自由意志", "决定论", "determinism"]],
86
-
87
- // --- Config / Deploy ---
88
- [["配置", "设置", "config"], ["配置", "config", "configuration", "settings", "设置"]],
89
- [["部署", "上线"], ["deploy", "部署", "上线", "发布", "release"]],
90
- [["容器", "docker"], ["Docker", "容器", "container", "docker-compose"]],
91
-
92
- // --- Code / Debug ---
93
- [["报错", "出错", "错误"], ["error", "报错", "exception", "错误", "失败", "bug"]],
94
- [["修复", "修了", "修好"], ["fix", "修复", "patch", "修了", "解决"]],
95
- [["踩坑", "坑"], ["踩坑", "bug", "问题", "教训", "排查", "troubleshoot"]],
96
-
97
- // --- Writing / Content ---
98
- [["配图", "插图"], ["配图", "封面", "style-catalog", "风格", "图片", "image"]],
99
- [["排版", "版式"], ["排版", "layout", "主题", "theme", "样式"]],
100
- [["风格"], ["风格", "style", "轮换", "catalog"]],
101
- [["写作", "写文章"], ["写作", "writing", "文章", "公众号", "content-alchemy"]],
102
-
103
- // --- Infrastructure ---
104
- [["bot", "机器人"], ["bot", "机器人", "OpenClaw", "agent", "gateway"]],
105
- [["推送", "push"], ["push", "推送", "git push", "commit"]],
106
- [["记忆", "memory"], ["记忆", "memory", "记忆系统", "LanceDB", "索引"]],
107
- [["搜索", "查找", "找"], ["搜索", "search", "retrieval", "检索", "查找"]],
108
- ];
109
-
110
- /**
111
- * Expand a query by appending synonym terms from the dictionary.
112
- * Returns the original query with additional terms appended.
113
- * Idempotent — already-precise queries pass through unchanged.
114
- */
115
- export function expandQuery(query: string): string {
116
- if (!query || query.trim().length < 2) return query;
117
-
118
- const lower = query.toLowerCase();
119
- const additions = new Set<string>();
120
-
121
- for (const [triggers, expansions] of SYNONYM_MAP) {
122
- if (triggers.some(t => lower.includes(t.toLowerCase()))) {
123
- for (const exp of expansions) {
124
- // Don't add terms already in the query
125
- if (!lower.includes(exp.toLowerCase())) {
126
- additions.add(exp);
127
- }
128
- }
129
- }
130
- }
131
-
132
- if (additions.size === 0) return query;
133
- return `${query} ${[...additions].join(" ")}`;
134
- }
135
- ```
136
-
137
- **Step 4: Run tests to verify they pass**
138
-
139
- Run: `cd ~/local-memory && bun test src/__tests__/query-expander.test.ts`
140
- Expected: All 5 tests PASS
141
-
142
- **Step 5: Commit**
143
-
144
- ```bash
145
- cd ~/local-memory
146
- git add src/query-expander.ts src/__tests__/query-expander.test.ts
147
- git commit -m "feat: add Chinese synonym query expansion dictionary"
148
- ```
149
-
150
- ---
151
-
152
- ### Task 2: Wire expander into BM25 search path
153
-
154
- **Files:**
155
- - Modify: `src/retriever.ts` (lines 334-349, hybridRetrieval method)
156
-
157
- **Step 1: Write the integration**
158
-
159
- In `retriever.ts`, import the expander at the top:
160
-
161
- ```typescript
162
- import { expandQuery } from "./query-expander.js";
163
- ```
164
-
165
- Then in `hybridRetrieval()` (line 348), change the BM25 call to use expanded query:
166
-
167
- ```typescript
168
- // Before:
169
- this.runBM25Search(query, candidatePoolSize, scopeFilter, category),
170
-
171
- // After:
172
- this.runBM25Search(expandQuery(query), candidatePoolSize, scopeFilter, category),
173
- ```
174
-
175
- Note: Vector search (line 347) stays unchanged — Jina embeddings handle semantics.
176
- Note: The reranker (line 359) uses the original `query`, not expanded — reranking should judge against user intent.
177
-
178
- **Step 2: Run existing cases to verify no regression**
179
-
180
- Run: `cd ~/local-memory && bun run src/eval.ts`
181
- Expected: All 6 existing cases still PASS (or same score as before)
182
-
183
- **Step 3: Commit**
184
-
185
- ```bash
186
- cd ~/local-memory
187
- git add src/retriever.ts
188
- git commit -m "feat: wire query expansion into BM25 retrieval path"
189
- ```
190
-
191
- ---
192
-
193
- ### Task 3: Add fuzzy test cases and run before/after comparison
194
-
195
- **Files:**
196
- - Modify: `eval/cases.json`
197
-
198
- **Step 1: Add fuzzy test cases**
199
-
200
- Append these cases to `eval/cases.json`:
201
-
202
- ```json
203
- {
204
- "name": "fuzzy_bot_crash",
205
- "query": "上次那个 bot 突然挂了是怎么回事来着",
206
- "profile": "debug",
207
- "limit": 5,
208
- "expectAny": ["崩溃", "crash", "error", "报错", "排查", "修复", "docker"],
209
- "expectScopePrefixes": ["cc"],
210
- "notes": "Fuzzy colloquial query about bot crash should find debugging conversations."
211
- },
212
- {
213
- "name": "fuzzy_ai_feelings",
214
- "query": "之前聊过一个关于 AI 到底有没有感受的话题",
215
- "profile": "writing",
216
- "limit": 5,
217
- "expectAny": ["意识", "consciousness", "experiencing", "感受", "自由意志", "sentience"],
218
- "expectScopePrefixes": ["cc"],
219
- "notes": "Vague conversational query about AI consciousness should recall philosophical discussions."
220
- },
221
- {
222
- "name": "fuzzy_image_style",
223
- "query": "那个配图风格轮换的逻辑是怎么搞的",
224
- "profile": "default",
225
- "limit": 5,
226
- "expectAny": ["风格", "style", "轮换", "catalog", "配图"],
227
- "expectScopePrefixes": ["cc"],
228
- "notes": "Casual question about image style rotation should find implementation discussions."
229
- }
230
- ```
231
-
232
- **Step 2: Run full suite and save report**
233
-
234
- Run: `cd ~/local-memory && bun run src/eval.ts --output eval/reports/query-expansion.md`
235
- Expected: New fuzzy cases should score higher than without expansion
236
-
237
- **Step 3: Commit**
238
-
239
- ```bash
240
- cd ~/local-memory
241
- git add eval/cases.json eval/reports/query-expansion.md
242
- git commit -m "test: add fuzzy Chinese query cases and expansion baseline report"
243
- ```
244
-
245
- ---
246
-
247
- ### Task 4: Manual smoke test with real queries
248
-
249
- **Not code — manual verification via CLI**
250
-
251
- Run these and visually check relevance:
252
-
253
- ```bash
254
- cd ~/local-memory
255
- ./lm search "上次那个 bot 突然挂了是怎么回事来着" --profile debug
256
- ./lm search "之前聊过一个关于 AI 到底有没有感受的话题" --profile writing
257
- ./lm search "那个配图风格轮换的逻辑是怎么搞的"
258
- ./lm search "怎么给秦超老师的项目提 PR 来着"
259
- ./lm search "公众号发布流程是什么"
260
- ```
261
-
262
- Compare against the results we got earlier (before expansion) to confirm improvement.
263
-
264
- **Commit (if any dictionary tuning needed):**
265
-
266
- ```bash
267
- cd ~/local-memory
268
- git add src/query-expander.ts
269
- git commit -m "tune: adjust synonym dictionary based on smoke test"
270
- ```
@@ -1,196 +0,0 @@
1
- # Continuity Layer Plan
2
-
3
- > Phase 1 design for turning RecallNest from searchable history into a continuity layer.
4
-
5
- ## Goal
6
-
7
- RecallNest already provides shared memory across Claude Code, Codex, and Gemini CLI. The next step is not "more search." The next step is continuity:
8
-
9
- - stable context should survive across windows
10
- - active work state should not be mixed into long-term memory
11
- - startup context should be composed deliberately, not copied from raw search results
12
-
13
- ## The Three Layers
14
-
15
- ### 1. Durable Memory
16
-
17
- Purpose:
18
-
19
- - long-lived context worth keeping across sessions and terminals
20
-
21
- Categories:
22
-
23
- - `profile`
24
- - `preferences`
25
- - `entities`
26
- - `events`
27
- - `cases`
28
- - `patterns`
29
-
30
- Characteristics:
31
-
32
- - stored in the main memory index
33
- - survives across windows
34
- - retrieved through the normal retrieval stack
35
-
36
- Primary write interface:
37
-
38
- - `store_memory`
39
-
40
- ### 2. Session Checkpoint
41
-
42
- Purpose:
43
-
44
- - preserve current work state without polluting long-term memory
45
-
46
- Typical contents:
47
-
48
- - current task summary
49
- - decisions made in the session
50
- - open loops
51
- - next actions
52
- - active entities
53
- - relevant files
54
-
55
- Characteristics:
56
-
57
- - scoped to the current session or work thread
58
- - short-lived compared with durable memory
59
- - should be stored separately from the main durable memory table
60
-
61
- Primary write interface:
62
-
63
- - `checkpoint_session`
64
-
65
- ### 3. Resume Context
66
-
67
- Purpose:
68
-
69
- - provide a fresh window with the minimum context needed to work effectively
70
-
71
- Sources:
72
-
73
- - stable durable memories
74
- - pinned assets
75
- - recent cases and patterns
76
- - latest session checkpoint
77
-
78
- Characteristics:
79
-
80
- - composed output, not raw search results
81
- - optimized for prompt injection or startup display
82
- - should bias toward stable background, not "resume the last conversation verbatim"
83
-
84
- Primary read interface:
85
-
86
- - `resume_context`
87
-
88
- ## Interface Boundaries
89
-
90
- ### `store_memory`
91
-
92
- Use when:
93
-
94
- - a fact, preference, entity, case, or reusable pattern should survive long term
95
-
96
- Do not use when:
97
-
98
- - the content is only about the current unfinished work state
99
-
100
- Schema:
101
-
102
- - code source: [`memory-schema.ts`](/Users/anxianjingya/recallnest/src/memory-schema.ts)
103
-
104
- Key fields:
105
-
106
- - `text`
107
- - `category`
108
- - `importance`
109
- - `scope`
110
- - `source`
111
- - `tags`
112
-
113
- ### `checkpoint_session`
114
-
115
- Use when:
116
-
117
- - a session needs a compact representation of current progress
118
-
119
- Do not use when:
120
-
121
- - the information is already a durable user/project fact
122
-
123
- Schema:
124
-
125
- - code source: [`session-schema.ts`](/Users/anxianjingya/recallnest/src/session-schema.ts)
126
-
127
- Key fields:
128
-
129
- - `sessionId`
130
- - `summary`
131
- - `decisions`
132
- - `openLoops`
133
- - `nextActions`
134
- - `entities`
135
- - `files`
136
-
137
- ### `resume_context`
138
-
139
- Use when:
140
-
141
- - a new window or task start needs continuity
142
-
143
- The response should answer:
144
-
145
- - who is this user
146
- - what stable preferences matter
147
- - what project entities are relevant
148
- - what reusable patterns or recent cases matter
149
- - what is the latest active checkpoint
150
-
151
- Schema:
152
-
153
- - code source: [`session-schema.ts`](/Users/anxianjingya/recallnest/src/session-schema.ts)
154
-
155
- Key fields:
156
-
157
- - `summary`
158
- - `stableContext`
159
- - `relevantPatterns`
160
- - `recentCases`
161
- - `latestCheckpoint`
162
-
163
- ## Storage Direction
164
-
165
- ### Durable memory
166
-
167
- - stays in the main `memories` index
168
- - uses existing retrieval, decay, and tiering
169
-
170
- ### Session checkpoints
171
-
172
- - should move to a dedicated store or table
173
- - should not be mixed with durable memory rows by default
174
-
175
- ## Why This Split Matters
176
-
177
- Without this split, RecallNest tends to accumulate two incompatible things in one place:
178
-
179
- - stable memory that should be reused for months
180
- - transient working notes that are only useful during the current task
181
-
182
- That creates noisy recall and weak startup continuity.
183
-
184
- ## Implementation Order
185
-
186
- 1. Define schema and contracts
187
- 2. Add structured durable memory writes
188
- 3. Add session checkpoint persistence
189
- 4. Add resume context composition
190
- 5. Add continuity-focused evaluation
191
-
192
- ## Done in Phase 1
193
-
194
- - Added durable memory schemas in [`memory-schema.ts`](/Users/anxianjingya/recallnest/src/memory-schema.ts)
195
- - Added checkpoint and resume schemas in [`session-schema.ts`](/Users/anxianjingya/recallnest/src/session-schema.ts)
196
- - Wired existing category validation to the new durable memory schema in [`api-server.ts`](/Users/anxianjingya/recallnest/src/api-server.ts) and [`mcp-server.ts`](/Users/anxianjingya/recallnest/src/mcp-server.ts)