nlm-memory 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (472) hide show
  1. package/.agents/plugins/marketplace.json +20 -0
  2. package/.github/workflows/ci.yml +30 -0
  3. package/LICENSE +151 -0
  4. package/README.md +119 -0
  5. package/dist/cli/classify-parity.d.ts +48 -0
  6. package/dist/cli/classify-parity.js +182 -0
  7. package/dist/cli/classify-parity.js.map +1 -0
  8. package/dist/cli/launchctl-helpers.d.ts +26 -0
  9. package/dist/cli/launchctl-helpers.js +42 -0
  10. package/dist/cli/launchctl-helpers.js.map +1 -0
  11. package/dist/cli/nlm.d.ts +25 -0
  12. package/dist/cli/nlm.js +832 -0
  13. package/dist/cli/nlm.js.map +1 -0
  14. package/dist/core/actions/actions-log.d.ts +40 -0
  15. package/dist/core/actions/actions-log.js +72 -0
  16. package/dist/core/actions/actions-log.js.map +1 -0
  17. package/dist/core/actions/overlay.d.ts +30 -0
  18. package/dist/core/actions/overlay.js +101 -0
  19. package/dist/core/actions/overlay.js.map +1 -0
  20. package/dist/core/adapters/aider.d.ts +33 -0
  21. package/dist/core/adapters/aider.js +167 -0
  22. package/dist/core/adapters/aider.js.map +1 -0
  23. package/dist/core/adapters/claude-code.d.ts +32 -0
  24. package/dist/core/adapters/claude-code.js +270 -0
  25. package/dist/core/adapters/claude-code.js.map +1 -0
  26. package/dist/core/adapters/common.d.ts +20 -0
  27. package/dist/core/adapters/common.js +60 -0
  28. package/dist/core/adapters/common.js.map +1 -0
  29. package/dist/core/adapters/from-source.d.ts +11 -0
  30. package/dist/core/adapters/from-source.js +55 -0
  31. package/dist/core/adapters/from-source.js.map +1 -0
  32. package/dist/core/adapters/hermes-agent.d.ts +34 -0
  33. package/dist/core/adapters/hermes-agent.js +192 -0
  34. package/dist/core/adapters/hermes-agent.js.map +1 -0
  35. package/dist/core/adapters/hermes.d.ts +31 -0
  36. package/dist/core/adapters/hermes.js +247 -0
  37. package/dist/core/adapters/hermes.js.map +1 -0
  38. package/dist/core/adapters/jsonl-generic.d.ts +56 -0
  39. package/dist/core/adapters/jsonl-generic.js +185 -0
  40. package/dist/core/adapters/jsonl-generic.js.map +1 -0
  41. package/dist/core/adapters/opencode.d.ts +36 -0
  42. package/dist/core/adapters/opencode.js +213 -0
  43. package/dist/core/adapters/opencode.js.map +1 -0
  44. package/dist/core/adapters/pi.d.ts +32 -0
  45. package/dist/core/adapters/pi.js +233 -0
  46. package/dist/core/adapters/pi.js.map +1 -0
  47. package/dist/core/classifier/prompt.d.ts +60 -0
  48. package/dist/core/classifier/prompt.js +178 -0
  49. package/dist/core/classifier/prompt.js.map +1 -0
  50. package/dist/core/dataset/build-dataset.d.ts +87 -0
  51. package/dist/core/dataset/build-dataset.js +335 -0
  52. package/dist/core/dataset/build-dataset.js.map +1 -0
  53. package/dist/core/embedding/chunk-body.d.ts +30 -0
  54. package/dist/core/embedding/chunk-body.js +60 -0
  55. package/dist/core/embedding/chunk-body.js.map +1 -0
  56. package/dist/core/embedding/embed-backfill.d.ts +36 -0
  57. package/dist/core/embedding/embed-backfill.js +168 -0
  58. package/dist/core/embedding/embed-backfill.js.map +1 -0
  59. package/dist/core/embedding/embed-normalize.d.ts +28 -0
  60. package/dist/core/embedding/embed-normalize.js +98 -0
  61. package/dist/core/embedding/embed-normalize.js.map +1 -0
  62. package/dist/core/facts/backfill-facts.d.ts +58 -0
  63. package/dist/core/facts/backfill-facts.js +169 -0
  64. package/dist/core/facts/backfill-facts.js.map +1 -0
  65. package/dist/core/facts/extract-facts.d.ts +20 -0
  66. package/dist/core/facts/extract-facts.js +37 -0
  67. package/dist/core/facts/extract-facts.js.map +1 -0
  68. package/dist/core/hook/citation-detect.d.ts +32 -0
  69. package/dist/core/hook/citation-detect.js +105 -0
  70. package/dist/core/hook/citation-detect.js.map +1 -0
  71. package/dist/core/hook/cite-memo.d.ts +20 -0
  72. package/dist/core/hook/cite-memo.js +68 -0
  73. package/dist/core/hook/cite-memo.js.map +1 -0
  74. package/dist/core/hook/claude-settings.d.ts +34 -0
  75. package/dist/core/hook/claude-settings.js +117 -0
  76. package/dist/core/hook/claude-settings.js.map +1 -0
  77. package/dist/core/hook/gate.d.ts +11 -0
  78. package/dist/core/hook/gate.js +19 -0
  79. package/dist/core/hook/gate.js.map +1 -0
  80. package/dist/core/hook/hook-log.d.ts +25 -0
  81. package/dist/core/hook/hook-log.js +28 -0
  82. package/dist/core/hook/hook-log.js.map +1 -0
  83. package/dist/core/hook/memo-sweep.d.ts +55 -0
  84. package/dist/core/hook/memo-sweep.js +134 -0
  85. package/dist/core/hook/memo-sweep.js.map +1 -0
  86. package/dist/core/hook/memo.d.ts +20 -0
  87. package/dist/core/hook/memo.js +66 -0
  88. package/dist/core/hook/memo.js.map +1 -0
  89. package/dist/core/hook/pointer-block.d.ts +14 -0
  90. package/dist/core/hook/pointer-block.js +19 -0
  91. package/dist/core/hook/pointer-block.js.map +1 -0
  92. package/dist/core/hook/select.d.ts +21 -0
  93. package/dist/core/hook/select.js +15 -0
  94. package/dist/core/hook/select.js.map +1 -0
  95. package/dist/core/hook/transcript.d.ts +31 -0
  96. package/dist/core/hook/transcript.js +103 -0
  97. package/dist/core/hook/transcript.js.map +1 -0
  98. package/dist/core/ingest/ingest-session.d.ts +40 -0
  99. package/dist/core/ingest/ingest-session.js +71 -0
  100. package/dist/core/ingest/ingest-session.js.map +1 -0
  101. package/dist/core/providers/provider-models.d.ts +24 -0
  102. package/dist/core/providers/provider-models.js +72 -0
  103. package/dist/core/providers/provider-models.js.map +1 -0
  104. package/dist/core/providers/provider-registry.d.ts +62 -0
  105. package/dist/core/providers/provider-registry.js +143 -0
  106. package/dist/core/providers/provider-registry.js.map +1 -0
  107. package/dist/core/recall/citation-log.d.ts +28 -0
  108. package/dist/core/recall/citation-log.js +90 -0
  109. package/dist/core/recall/citation-log.js.map +1 -0
  110. package/dist/core/recall/filter.d.ts +11 -0
  111. package/dist/core/recall/filter.js +20 -0
  112. package/dist/core/recall/filter.js.map +1 -0
  113. package/dist/core/recall/index.d.ts +6 -0
  114. package/dist/core/recall/index.js +5 -0
  115. package/dist/core/recall/index.js.map +1 -0
  116. package/dist/core/recall/match-fields.d.ts +10 -0
  117. package/dist/core/recall/match-fields.js +37 -0
  118. package/dist/core/recall/match-fields.js.map +1 -0
  119. package/dist/core/recall/query-log.d.ts +36 -0
  120. package/dist/core/recall/query-log.js +112 -0
  121. package/dist/core/recall/query-log.js.map +1 -0
  122. package/dist/core/recall/query-shape.d.ts +22 -0
  123. package/dist/core/recall/query-shape.js +64 -0
  124. package/dist/core/recall/query-shape.js.map +1 -0
  125. package/dist/core/recall/recall-service.d.ts +19 -0
  126. package/dist/core/recall/recall-service.js +252 -0
  127. package/dist/core/recall/recall-service.js.map +1 -0
  128. package/dist/core/recall/recent-log.d.ts +16 -0
  129. package/dist/core/recall/recent-log.js +46 -0
  130. package/dist/core/recall/recent-log.js.map +1 -0
  131. package/dist/core/recall/tokenize.d.ts +7 -0
  132. package/dist/core/recall/tokenize.js +18 -0
  133. package/dist/core/recall/tokenize.js.map +1 -0
  134. package/dist/core/recall/useful-scan.d.ts +52 -0
  135. package/dist/core/recall/useful-scan.js +300 -0
  136. package/dist/core/recall/useful-scan.js.map +1 -0
  137. package/dist/core/recall-facts/fact-query-log.d.ts +42 -0
  138. package/dist/core/recall-facts/fact-query-log.js +115 -0
  139. package/dist/core/recall-facts/fact-query-log.js.map +1 -0
  140. package/dist/core/recall-facts/fact-recall-service.d.ts +34 -0
  141. package/dist/core/recall-facts/fact-recall-service.js +246 -0
  142. package/dist/core/recall-facts/fact-recall-service.js.map +1 -0
  143. package/dist/core/scheduler/scan-once.d.ts +32 -0
  144. package/dist/core/scheduler/scan-once.js +100 -0
  145. package/dist/core/scheduler/scan-once.js.map +1 -0
  146. package/dist/core/scheduler/scheduler.d.ts +59 -0
  147. package/dist/core/scheduler/scheduler.js +158 -0
  148. package/dist/core/scheduler/scheduler.js.map +1 -0
  149. package/dist/core/sources/source-registry.d.ts +68 -0
  150. package/dist/core/sources/source-registry.js +208 -0
  151. package/dist/core/sources/source-registry.js.map +1 -0
  152. package/dist/core/storage/db-restore.d.ts +53 -0
  153. package/dist/core/storage/db-restore.js +113 -0
  154. package/dist/core/storage/db-restore.js.map +1 -0
  155. package/dist/core/storage/live-status.d.ts +15 -0
  156. package/dist/core/storage/live-status.js +43 -0
  157. package/dist/core/storage/live-status.js.map +1 -0
  158. package/dist/core/storage/migrate.d.ts +14 -0
  159. package/dist/core/storage/migrate.js +52 -0
  160. package/dist/core/storage/migrate.js.map +1 -0
  161. package/dist/core/storage/sqlite-fact-store.d.ts +50 -0
  162. package/dist/core/storage/sqlite-fact-store.js +256 -0
  163. package/dist/core/storage/sqlite-fact-store.js.map +1 -0
  164. package/dist/core/storage/sqlite-session-store.d.ts +152 -0
  165. package/dist/core/storage/sqlite-session-store.js +587 -0
  166. package/dist/core/storage/sqlite-session-store.js.map +1 -0
  167. package/dist/hook/pre-compact-hook.d.ts +26 -0
  168. package/dist/hook/pre-compact-hook.js +94 -0
  169. package/dist/hook/pre-compact-hook.js.map +1 -0
  170. package/dist/hook/prompt-recall-hook.d.ts +23 -0
  171. package/dist/hook/prompt-recall-hook.js +141 -0
  172. package/dist/hook/prompt-recall-hook.js.map +1 -0
  173. package/dist/hook/session-end-hook.d.ts +18 -0
  174. package/dist/hook/session-end-hook.js +67 -0
  175. package/dist/hook/session-end-hook.js.map +1 -0
  176. package/dist/hook/session-start-hook.d.ts +25 -0
  177. package/dist/hook/session-start-hook.js +129 -0
  178. package/dist/hook/session-start-hook.js.map +1 -0
  179. package/dist/hook/stop-hook.d.ts +38 -0
  180. package/dist/hook/stop-hook.js +171 -0
  181. package/dist/hook/stop-hook.js.map +1 -0
  182. package/dist/hook/subagent-start-hook.d.ts +30 -0
  183. package/dist/hook/subagent-start-hook.js +108 -0
  184. package/dist/hook/subagent-start-hook.js.map +1 -0
  185. package/dist/http/app.d.ts +65 -0
  186. package/dist/http/app.js +1009 -0
  187. package/dist/http/app.js.map +1 -0
  188. package/dist/install/claude-code.d.ts +57 -0
  189. package/dist/install/claude-code.js +76 -0
  190. package/dist/install/claude-code.js.map +1 -0
  191. package/dist/install/codex.d.ts +82 -0
  192. package/dist/install/codex.js +277 -0
  193. package/dist/install/codex.js.map +1 -0
  194. package/dist/install/hermes-agent.d.ts +35 -0
  195. package/dist/install/hermes-agent.js +48 -0
  196. package/dist/install/hermes-agent.js.map +1 -0
  197. package/dist/install/hermes.d.ts +29 -0
  198. package/dist/install/hermes.js +52 -0
  199. package/dist/install/hermes.js.map +1 -0
  200. package/dist/install/ollama.d.ts +54 -0
  201. package/dist/install/ollama.js +178 -0
  202. package/dist/install/ollama.js.map +1 -0
  203. package/dist/install/setup.d.ts +37 -0
  204. package/dist/install/setup.js +339 -0
  205. package/dist/install/setup.js.map +1 -0
  206. package/dist/llm/classifier-box.d.ts +29 -0
  207. package/dist/llm/classifier-box.js +43 -0
  208. package/dist/llm/classifier-box.js.map +1 -0
  209. package/dist/llm/deepseek-client.d.ts +40 -0
  210. package/dist/llm/deepseek-client.js +114 -0
  211. package/dist/llm/deepseek-client.js.map +1 -0
  212. package/dist/llm/env-autoload.d.ts +8 -0
  213. package/dist/llm/env-autoload.js +54 -0
  214. package/dist/llm/env-autoload.js.map +1 -0
  215. package/dist/llm/ollama-client.d.ts +47 -0
  216. package/dist/llm/ollama-client.js +156 -0
  217. package/dist/llm/ollama-client.js.map +1 -0
  218. package/dist/mcp/server.d.ts +64 -0
  219. package/dist/mcp/server.js +430 -0
  220. package/dist/mcp/server.js.map +1 -0
  221. package/dist/ports/fact-store.d.ts +82 -0
  222. package/dist/ports/fact-store.js +16 -0
  223. package/dist/ports/fact-store.js.map +1 -0
  224. package/dist/ports/llm-client.d.ts +42 -0
  225. package/dist/ports/llm-client.js +14 -0
  226. package/dist/ports/llm-client.js.map +1 -0
  227. package/dist/ports/logger.d.ts +13 -0
  228. package/dist/ports/logger.js +8 -0
  229. package/dist/ports/logger.js.map +1 -0
  230. package/dist/ports/session-store.d.ts +29 -0
  231. package/dist/ports/session-store.js +9 -0
  232. package/dist/ports/session-store.js.map +1 -0
  233. package/dist/ports/transcript-adapter.d.ts +48 -0
  234. package/dist/ports/transcript-adapter.js +15 -0
  235. package/dist/ports/transcript-adapter.js.map +1 -0
  236. package/dist/shared/types.d.ts +129 -0
  237. package/dist/shared/types.js +6 -0
  238. package/dist/shared/types.js.map +1 -0
  239. package/dist/ui/assets/index-BA6IpU8g.css +1 -0
  240. package/dist/ui/assets/index-B_qIVV0k.js +69 -0
  241. package/dist/ui/index.html +13 -0
  242. package/docs/methodology/re-derivation-rate.md +112 -0
  243. package/docs/methodology/useful-hit-rate.md +79 -0
  244. package/docs/plans/2026-05-20-fts5-lexical-recall.md +1088 -0
  245. package/docs/plans/2026-05-20-recall-daemon-wedge-fix.md +662 -0
  246. package/docs/plans/2026-05-20-recall-hook-design.md +131 -0
  247. package/docs/plans/2026-05-20-recall-hook-implementation.md +1222 -0
  248. package/docs/plans/desktop-product.md +69 -0
  249. package/docs/plans/factstore-design.md +236 -0
  250. package/logs/CHANGELOG/CHANGELOG-2026.md +1389 -0
  251. package/logs/CHANGELOG/CHANGELOG.md +320 -0
  252. package/migrations/000_initial_schema.sql +174 -0
  253. package/migrations/001_entity_type_rename.sql +17 -0
  254. package/migrations/002_adapter_state_extend.sql +12 -0
  255. package/migrations/003_session_embeddings.sql +11 -0
  256. package/migrations/004_facts.sql +46 -0
  257. package/migrations/005_sources.sql +31 -0
  258. package/migrations/006_providers.sql +33 -0
  259. package/migrations/007_source_tokens.sql +17 -0
  260. package/migrations/008_fts_rebuild.sql +9 -0
  261. package/migrations/009_session_embedding_chunks.sql +46 -0
  262. package/migrations/010_sources_opencode.sql +30 -0
  263. package/migrations/011_sources_hermes_agent.sql +30 -0
  264. package/migrations/012_sources_aider.sql +30 -0
  265. package/migrations/013_adapter_state_failure_count.sql +12 -0
  266. package/package.json +56 -0
  267. package/plugin/.codex-plugin/plugin.json +22 -0
  268. package/plugin/.mcp.json +8 -0
  269. package/plugin/README.md +51 -0
  270. package/plugin/hooks/hooks.json +25 -0
  271. package/plugin/scripts/prompt-recall-hook.mjs +202 -0
  272. package/plugin/scripts/stop-hook.mjs +306 -0
  273. package/plugin-hermes-agent/README.md +49 -0
  274. package/plugin-hermes-agent/__init__.py +75 -0
  275. package/plugin-hermes-agent/plugin.yaml +15 -0
  276. package/scripts/backfill-citations.mjs +0 -0
  277. package/scripts/build-codex-plugin.mjs +61 -0
  278. package/scripts/deepseek-probe.mjs +67 -0
  279. package/scripts/extract-triples.mjs +207 -0
  280. package/scripts/longmemeval/embedding-cache.ts +77 -0
  281. package/scripts/longmemeval/fetch-dataset.sh +25 -0
  282. package/scripts/longmemeval/run-harness.ts +315 -0
  283. package/scripts/longmemeval/scorer.ts +99 -0
  284. package/scripts/longmemeval/tsconfig.json +9 -0
  285. package/scripts/longmemeval/types.ts +35 -0
  286. package/scripts/nlm-daily-digest.py +239 -0
  287. package/scripts/nlm-daily-digest.sh +28 -0
  288. package/src/cli/classify-parity.ts +257 -0
  289. package/src/cli/launchctl-helpers.ts +49 -0
  290. package/src/cli/nlm.ts +885 -0
  291. package/src/core/actions/actions-log.ts +118 -0
  292. package/src/core/actions/overlay.ts +117 -0
  293. package/src/core/adapters/aider.ts +205 -0
  294. package/src/core/adapters/claude-code.ts +293 -0
  295. package/src/core/adapters/common.ts +54 -0
  296. package/src/core/adapters/from-source.ts +57 -0
  297. package/src/core/adapters/hermes-agent.ts +240 -0
  298. package/src/core/adapters/hermes.ts +277 -0
  299. package/src/core/adapters/jsonl-generic.ts +208 -0
  300. package/src/core/adapters/opencode.ts +281 -0
  301. package/src/core/adapters/pi.ts +264 -0
  302. package/src/core/classifier/prompt.ts +200 -0
  303. package/src/core/dataset/build-dataset.ts +463 -0
  304. package/src/core/embedding/chunk-body.ts +76 -0
  305. package/src/core/embedding/embed-backfill.ts +210 -0
  306. package/src/core/embedding/embed-normalize.ts +135 -0
  307. package/src/core/facts/backfill-facts.ts +254 -0
  308. package/src/core/facts/extract-facts.ts +50 -0
  309. package/src/core/hook/citation-detect.ts +124 -0
  310. package/src/core/hook/cite-memo.ts +68 -0
  311. package/src/core/hook/claude-settings.ts +166 -0
  312. package/src/core/hook/gate.ts +25 -0
  313. package/src/core/hook/hook-log.ts +41 -0
  314. package/src/core/hook/memo-sweep.ts +164 -0
  315. package/src/core/hook/memo.ts +67 -0
  316. package/src/core/hook/pointer-block.ts +26 -0
  317. package/src/core/hook/select.ts +32 -0
  318. package/src/core/hook/transcript.ts +121 -0
  319. package/src/core/ingest/ingest-session.ts +111 -0
  320. package/src/core/providers/provider-models.ts +100 -0
  321. package/src/core/providers/provider-registry.ts +196 -0
  322. package/src/core/recall/citation-log.ts +108 -0
  323. package/src/core/recall/filter.ts +27 -0
  324. package/src/core/recall/index.ts +6 -0
  325. package/src/core/recall/match-fields.ts +40 -0
  326. package/src/core/recall/query-log.ts +149 -0
  327. package/src/core/recall/query-shape.ts +66 -0
  328. package/src/core/recall/recall-service.ts +320 -0
  329. package/src/core/recall/recent-log.ts +59 -0
  330. package/src/core/recall/tokenize.ts +18 -0
  331. package/src/core/recall/useful-scan.ts +336 -0
  332. package/src/core/recall-facts/fact-query-log.ts +150 -0
  333. package/src/core/recall-facts/fact-recall-service.ts +327 -0
  334. package/src/core/scheduler/scan-once.ts +142 -0
  335. package/src/core/scheduler/scheduler.ts +225 -0
  336. package/src/core/sources/source-registry.ts +260 -0
  337. package/src/core/storage/db-restore.ts +133 -0
  338. package/src/core/storage/live-status.ts +45 -0
  339. package/src/core/storage/migrate.ts +72 -0
  340. package/src/core/storage/sqlite-fact-store.ts +304 -0
  341. package/src/core/storage/sqlite-session-store.ts +765 -0
  342. package/src/hook/prompt-recall-hook.ts +174 -0
  343. package/src/hook/session-end-hook.ts +81 -0
  344. package/src/hook/session-start-hook.ts +165 -0
  345. package/src/hook/stop-hook.ts +236 -0
  346. package/src/http/app.ts +1114 -0
  347. package/src/install/claude-code.ts +128 -0
  348. package/src/install/codex.ts +367 -0
  349. package/src/install/hermes-agent.ts +76 -0
  350. package/src/install/hermes.ts +78 -0
  351. package/src/install/ollama.ts +208 -0
  352. package/src/install/setup.ts +368 -0
  353. package/src/llm/classifier-box.ts +64 -0
  354. package/src/llm/deepseek-client.ts +150 -0
  355. package/src/llm/env-autoload.ts +55 -0
  356. package/src/llm/ollama-client.ts +189 -0
  357. package/src/mcp/server.ts +534 -0
  358. package/src/ports/fact-store.ts +102 -0
  359. package/src/ports/llm-client.ts +52 -0
  360. package/src/ports/logger.ts +16 -0
  361. package/src/ports/session-store.ts +45 -0
  362. package/src/ports/transcript-adapter.ts +55 -0
  363. package/src/shared/types.ts +145 -0
  364. package/src/ui/App.tsx +58 -0
  365. package/src/ui/components/PromoteOpenButton.tsx +65 -0
  366. package/src/ui/components/SessionDrawer.tsx +136 -0
  367. package/src/ui/components/SideNav.tsx +162 -0
  368. package/src/ui/components/Skeleton.tsx +107 -0
  369. package/src/ui/index.html +13 -0
  370. package/src/ui/lib/actions.ts +30 -0
  371. package/src/ui/lib/api.ts +92 -0
  372. package/src/ui/lib/dataset.ts +141 -0
  373. package/src/ui/lib/registries.ts +155 -0
  374. package/src/ui/lib/view-settings.ts +41 -0
  375. package/src/ui/main.tsx +15 -0
  376. package/src/ui/pages/Live.tsx +229 -0
  377. package/src/ui/pages/Pulse.tsx +415 -0
  378. package/src/ui/pages/Recall.tsx +190 -0
  379. package/src/ui/pages/River.tsx +308 -0
  380. package/src/ui/pages/Search.tsx +93 -0
  381. package/src/ui/pages/Stub.tsx +9 -0
  382. package/src/ui/pages/Thread.tsx +262 -0
  383. package/src/ui/pages/settings/Classifier.tsx +227 -0
  384. package/src/ui/pages/settings/Data.tsx +190 -0
  385. package/src/ui/pages/settings/Index.tsx +65 -0
  386. package/src/ui/pages/settings/Labels.tsx +224 -0
  387. package/src/ui/pages/settings/Providers.tsx +305 -0
  388. package/src/ui/pages/settings/SettingsSubnav.tsx +28 -0
  389. package/src/ui/pages/settings/Sources.tsx +326 -0
  390. package/src/ui/pages/settings/Views.tsx +96 -0
  391. package/src/ui/styles.css +1766 -0
  392. package/src/ui/tsconfig.json +21 -0
  393. package/src/ui/vite.config.ts +19 -0
  394. package/tests/fixtures/claude_code/short_session.jsonl +2 -0
  395. package/tests/fixtures/claude_code/standard_iso.jsonl +4 -0
  396. package/tests/fixtures/claude_code/tool_heavy.jsonl +8 -0
  397. package/tests/fixtures/claude_code/with_subagent.jsonl +7 -0
  398. package/tests/fixtures/facts.ts +17 -0
  399. package/tests/fixtures/golden-corpus.ts +85 -0
  400. package/tests/fixtures/hermes/paired_request_dump.json +24 -0
  401. package/tests/fixtures/hermes/paired_session.json +23 -0
  402. package/tests/fixtures/hermes/request_dump.json +28 -0
  403. package/tests/fixtures/hermes/session_iso.json +38 -0
  404. package/tests/fixtures/hermes/session_unix.json +38 -0
  405. package/tests/fixtures/hermes/system_only.json +18 -0
  406. package/tests/fixtures/pi/error-connection-abort.jsonl +8 -0
  407. package/tests/fixtures/pi/short-successful.jsonl +5 -0
  408. package/tests/fixtures/pi/with-custom-message.jsonl +6 -0
  409. package/tests/fixtures/sessions.ts +22 -0
  410. package/tests/integration/backfill-facts.test.ts +362 -0
  411. package/tests/integration/citation-explicit.test.ts +111 -0
  412. package/tests/integration/cite-event.test.ts +169 -0
  413. package/tests/integration/cite-memo.test.ts +87 -0
  414. package/tests/integration/db-restore.test.ts +153 -0
  415. package/tests/integration/embed-backfill.test.ts +176 -0
  416. package/tests/integration/fact-supersedence.test.ts +313 -0
  417. package/tests/integration/fts-index.test.ts +60 -0
  418. package/tests/integration/getbyids-sqlite.test.ts +60 -0
  419. package/tests/integration/hermes-agent-hooks.test.ts +248 -0
  420. package/tests/integration/hook-claude-settings.test.ts +205 -0
  421. package/tests/integration/hook-log.test.ts +54 -0
  422. package/tests/integration/hook-memo.test.ts +68 -0
  423. package/tests/integration/hook-pre-compact.test.ts +105 -0
  424. package/tests/integration/hook-subagent-start.test.ts +102 -0
  425. package/tests/integration/http.test.ts +401 -0
  426. package/tests/integration/keyword-search-fts.test.ts +66 -0
  427. package/tests/integration/mcp-recall-logging.test.ts +88 -0
  428. package/tests/integration/mcp.test.ts +248 -0
  429. package/tests/integration/memo-sweep.test.ts +91 -0
  430. package/tests/integration/prompt-recall-hook.test.ts +88 -0
  431. package/tests/integration/provider-registry.test.ts +107 -0
  432. package/tests/integration/recall-golden.test.ts +59 -0
  433. package/tests/integration/recall-sqlite.test.ts +169 -0
  434. package/tests/integration/scheduler.test.ts +391 -0
  435. package/tests/integration/session-end-hook.test.ts +48 -0
  436. package/tests/integration/session-start-hook.test.ts +126 -0
  437. package/tests/integration/source-registry.test.ts +120 -0
  438. package/tests/integration/sqlite-fact-store.test.ts +346 -0
  439. package/tests/integration/stop-hook.test.ts +560 -0
  440. package/tests/integration/wal-checkpoint.test.ts +49 -0
  441. package/tests/unit/cli/launchctl-helpers.test.ts +60 -0
  442. package/tests/unit/core/adapters/aider.test.ts +230 -0
  443. package/tests/unit/core/adapters/claude-code.test.ts +118 -0
  444. package/tests/unit/core/adapters/hermes-agent.test.ts +329 -0
  445. package/tests/unit/core/adapters/hermes.test.ts +81 -0
  446. package/tests/unit/core/adapters/jsonl-generic.test.ts +142 -0
  447. package/tests/unit/core/adapters/opencode.test.ts +354 -0
  448. package/tests/unit/core/adapters/pi.test.ts +110 -0
  449. package/tests/unit/core/classifier/prompt.test.ts +126 -0
  450. package/tests/unit/core/embedding/chunk-body.test.ts +100 -0
  451. package/tests/unit/core/facts/extract-facts.test.ts +117 -0
  452. package/tests/unit/core/filter.test.ts +40 -0
  453. package/tests/unit/core/hook/citation-detect-cite-session.test.ts +96 -0
  454. package/tests/unit/core/hook/citation-detect.test.ts +124 -0
  455. package/tests/unit/core/hook/gate.test.ts +29 -0
  456. package/tests/unit/core/hook/pointer-block.test.ts +22 -0
  457. package/tests/unit/core/hook/select.test.ts +66 -0
  458. package/tests/unit/core/match-fields.test.ts +39 -0
  459. package/tests/unit/core/mcp-cite-session.test.ts +51 -0
  460. package/tests/unit/core/providers/provider-models.test.ts +101 -0
  461. package/tests/unit/core/query-shape.test.ts +92 -0
  462. package/tests/unit/core/recall-facts/fact-recall-service.test.ts +258 -0
  463. package/tests/unit/core/recall-service.test.ts +200 -0
  464. package/tests/unit/core/storage/live-status.test.ts +54 -0
  465. package/tests/unit/core/tokenize.test.ts +32 -0
  466. package/tests/unit/core/useful-scan.test.ts +537 -0
  467. package/tests/unit/llm/embed.test.ts +93 -0
  468. package/tests/unit/llm/ollama-client.test.ts +124 -0
  469. package/tests/unit/scripts/longmemeval-scorer.test.ts +114 -0
  470. package/tsconfig.json +31 -0
  471. package/tsconfig.test.json +11 -0
  472. package/vitest.config.ts +22 -0
package/dist/ui/index.html ADDED
@@ -0,0 +1,13 @@
1
+ <!doctype html>
2
+ <html lang="en">
3
+ <head>
4
+ <meta charset="UTF-8" />
5
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
6
+ <title>nlm memory</title>
7
+ <script type="module" crossorigin src="/ui/assets/index-B_qIVV0k.js"></script>
8
+ <link rel="stylesheet" crossorigin href="/ui/assets/index-BA6IpU8g.css">
9
+ </head>
10
+ <body>
11
+ <div id="root"></div>
12
+ </body>
13
+ </html>
package/docs/methodology/re-derivation-rate.md ADDED
@@ -0,0 +1,112 @@
1
+ # re-derivation_rate — design
2
+
3
+ ## Why
4
+
5
+ `re_derivation_rate` is NLM's strategic metric — the operator-outcome number that competitors (mem0, agentmemory, Letta) cannot match because their destructive lifecycle (decay, auto-forget) erases the data needed to compute it. It is the headline number for Pulse, the cron digest, and any public marketing scorecard. Detection rule, methodology, and a reproducible script live here so the metric is auditable.
6
+
7
+ ## Plain-language definition
8
+
9
+ A *re-derivation* is when an operator (you, in any AI runtime) solves the same problem twice across multiple sessions without recall of the prior solution. It is the tax NLM exists to eliminate: every re-derivation is a session where memory could have helped but didn't.
10
+
11
+ `re_derivation_rate` over a window = (re-derivation events) / (decision events) in that window.
12
+
13
+ `re_derivations_prevented` = recall events whose `useful_hit_rate` is true AND whose returned session contained the matching decision. Inverse of re-derivation: the events where memory *did* help.
14
+
15
+ ## Detection rule (V1)
16
+
17
+ A pair of sessions `(A, B)` is a re-derivation iff all of the following hold:
18
+
19
+ 1. **Same entity.** A and B share at least one entity in their respective `entities` arrays.
20
+ 2. **Same decision normalized.** A `decision` marker in A and a `decision` marker in B normalize to overlapping content. Normalization: lowercase, strip stopwords, tokenize, Jaccard similarity ≥ 0.6.
21
+ 3. **Temporal gap.** `B.started_at - A.started_at >= 7 days`.
22
+ 4. **No supersedence link.** No `session_edges` row of kind `supersedes` connects A and B in either direction.
23
+ 5. **No continues link.** No `session_edges` row of kind `continues` connects A and B.
24
+ 6. **No intervening recall.** Between A.started_at and B.started_at, no recall event in `query-log.jsonl` or `hook-log.jsonl` returned A's id (would mean B's operator was aware of A and chose not to link).
25
+
26
+ When all six are true, `B` is a re-derivation of `A`. Count B (not A) — the metric measures fresh re-derivations, not the original.
27
+
28
+ ## Edge cases and resolutions
29
+
30
+ - **Three sessions A, B, C** where B re-derives A and C re-derives B: count B and C, not A.
31
+ - **Trivial decisions.** Decisions under N tokens (default 6) are excluded — "yes ship it" is not a meaningful decision to track.
32
+ - **High-frequency entities.** If an entity has >50 sessions in the window, scale the Jaccard threshold up to 0.75 to reduce false positives (common topics will inevitably overlap in keyword-trivial ways).
33
+ - **Probe / test entities.** Sessions whose label matches probe patterns (see useful-hit-rate.md) are excluded from both sides.
34
+
35
+ ## Computation algorithm
36
+
37
+ ```python
38
+ def find_re_derivations(sessions, edges, recalls, window_days):
39
+ pairs = []
40
+ decisions = collect_decisions(sessions) # one row per (session_id, normalized_decision_tokens, entities)
41
+ for ent in distinct_entities(decisions):
42
+ ent_decisions = sorted(by_session_start([d for d in decisions if ent in d.entities]))
43
+ for i, a in enumerate(ent_decisions):
44
+ for b in ent_decisions[i+1:]:
45
+ if days_between(a, b) < 7: continue
46
+ if days_between(a, b) > window_days: break
47
+ if jaccard(a.tokens, b.tokens) < threshold(ent): continue
48
+ if has_edge(edges, a, b, ("supersedes", "continues")): continue
49
+ if recall_returned_a_between(recalls, a, b): continue
50
+ pairs.append((a, b))
51
+ return pairs
52
+ ```
53
+
54
+ Runs over the existing canonical sqlite (sessions + session_edges) and the recall log jsonl files. No new schema, no migration. Computed in a single pass; results cached by `(window_start, window_end)` in a new `re_derivation_log` table.
55
+
56
+ ## Storage
57
+
58
+ - New table `re_derivation_log`: `(window_start, window_end, computed_at, session_a_id, session_b_id, entity, jaccard, decision_a, decision_b)`. One row per detected pair. Re-computable; deletable; not source of truth.
59
+ - New endpoint field on `/api/recall/stats`: `re_derivation_count_7d`, `re_derivations_prevented_7d`.
60
+ - Pulse: new headline tile showing both numbers and the weekly trend.
61
+
62
+ ## CLI
63
+
64
+ - `nlm re-derivation scan` — recomputes the log for a window. Default last 30 days.
65
+ - `nlm re-derivation list --since 7d` — lists detected pairs with the matched decisions for human review (false-positive triage).
66
+ - `nlm re-derivation explain <session-b-id>` — for one B, show why it was flagged (matched A, decision overlap, why no recall covered it).
67
+
68
+ ## Calibration loop
69
+
70
+ Re-derivation detection is heuristic. False positives waste reader trust; false negatives undersell the metric. Calibration weekly for the first month after V1:
71
+
72
+ 1. Run `nlm re-derivation list --since 7d`
73
+ 2. Edward reviews each flagged pair
74
+ 3. Mark `true_re_derivation: true|false` in a `re_derivation_feedback` table
75
+ 4. Adjust Jaccard threshold + minimum decision length until precision/recall both > 70% on Edward's review
76
+
77
+ After 4 weeks of calibration, freeze the parameters and publish them in `docs/methodology/re-derivation-rate.md` for external use.
78
+
79
+ ## Public scorecard format
80
+
81
+ For external publication (gated on the marketing-readiness checklist):
82
+
83
+ ```
84
+ Edward's corpus, week of YYYY-MM-DD:
85
+ Sessions in window: N
86
+ Decisions in window: M
87
+ Re-derivations detected: X
88
+ Re-derivations prevented: Y (recall returned the matching prior session)
89
+ Re-derivation rate: X / M = Z.Z%
90
+ Methodology: docs/methodology/re-derivation-rate.md
91
+ Calibration set: docs/calibration/re-derivation-2026-MM.md
92
+ ```
93
+
94
+ Publish weekly to the repo. The trend (rate falling over time as NLM gets more useful) is the marketing story.
95
+
96
+ ## Why competitors cannot match this
97
+
98
+ agentmemory's 4-tier lifecycle decays old observations and auto-forgets stale facts. Without the historical session record intact, there is no Session A to detect a re-derivation against — the data is gone. mem0 uses passive extraction and accretion, with no native concept of session identity that would let you pair A and B. Letta's core memory is in-context, not historical.
99
+
100
+ NLM's supersedence + full-session retention is the prerequisite for this metric. It is the strategic moat made measurable.
101
+
102
+ ## Out of scope (V1)
103
+
104
+ - Cross-runtime re-derivation (decision in Claude Code, re-derived in Hermes). Requires reliable entity normalization across adapters; defer to V2.
105
+ - Semantic similarity instead of Jaccard (would catch paraphrased decisions but requires embedding every decision). Defer.
106
+ - Automatic supersedence link suggestion from detected re-derivations. The metric should measure, not act, until calibrated.
107
+
108
+ ## Implementation phasing
109
+
110
+ 1. **Phase 1 (after #152, #153, #154 ship):** implement detection algorithm + CLI + scan command. No UI changes. Validate on Edward's corpus.
111
+ 2. **Phase 2 (after 2 weeks of calibration):** wire `re_derivation_count_7d` into `/api/recall/stats` and the daily digest. Pulse tile.
112
+ 3. **Phase 3 (gated on marketing readiness):** publish first weekly scorecard publicly. Repo README. Landing site.
package/docs/methodology/useful-hit-rate.md ADDED
@@ -0,0 +1,79 @@
1
+ # useful_hit_rate — design
2
+
3
+ ## Why
4
+
5
+ `hit_rate` reports the fraction of recall calls that returned ≥1 row. With the MCP default now hybrid, that number is structurally close to 100% — semantic always returns *something*. `hit_rate` no longer separates "found stuff" from "found stuff that mattered." `useful_hit_rate` is the metric we actually want: the fraction of recall calls whose returned results were referenced in the next assistant turn.
6
+
7
+ This is the signal that lets us answer "is NLM serving its intended purpose" with evidence instead of opinion, and it's an input to the headline re-derivation rate metric (see [re-derivation-rate.md](re-derivation-rate.md) — pending).
8
+
9
+ ## Definitions
10
+
11
+ **A recall event** is one of:
12
+ - A hook fire (logged in `~/.nlm/hook-log.jsonl` with `wouldInject` ids)
13
+ - An MCP `recall_sessions` / `recall_facts` call (logged in `~/.nlm/query-log.jsonl`)
14
+ - An HTTP `/api/recall` call (logged in `~/.nlm/query-log.jsonl`)
15
+
16
+ **A useful recall** is a recall event where:
17
+ - At least one of the returned session ids OR session labels appears in the next assistant message in the same conversation transcript, AND
18
+ - The match occurs within 3 assistant turns of the recall, AND
19
+ - The recall is not a probe (excluded query patterns: `concurrency probe`, `test probe`, `path test`, `recall test`, smoke/cutover patterns)
20
+
21
+ **`useful_hit_rate`** = (useful recalls) / (real recalls) over the reporting window.
22
+
23
+ ## Detection algorithm
24
+
25
+ ```
26
+ for each real recall event in window:
27
+ transcript = find_transcript(event.conversationId)
28
+ if transcript is None:
29
+ mark useful = null (unmeasurable)
30
+ continue
31
+ next_assistant_msgs = transcript.messages_after(event.ts, role="assistant", limit=3)
32
+ haystack = " ".join(m.content for m in next_assistant_msgs)
33
+ for hit_id in event.returnedIds:
34
+ if hit_id in haystack or session_label(hit_id) in haystack:
35
+ mark useful = true; break
36
+ else:
37
+ mark useful = false
38
+ ```
39
+
40
+ ## Data flow
41
+
42
+ 1. **Hook recalls** have `conversationId` directly. Transcript path: `~/.claude/projects/<sanitized-project>/<conversationId>.jsonl`.
43
+ 2. **MCP recalls** currently have no conversation context in `query-log.jsonl`. Adding `x-claude-session-id` capture to the MCP server is a prerequisite for measuring MCP useful_hit_rate.
44
+ 3. **HTTP recalls** are operator-driven (UI browsing) and excluded from this metric — `useful_hit_rate` measures agent recall usefulness, not UI search satisfaction.
45
+
46
+ ## Storage
47
+
48
+ - New log file `~/.nlm/useful-hit-log.jsonl`, one entry per scanned recall:
49
+ ```json
50
+ {"ts": "...", "source": "hook|mcp", "conversationId": "...", "returnedIds": [...], "useful": true|false|null, "matchedId": "...", "scannedAt": "..."}
51
+ ```
52
+ - New CLI: `nlm useful-scan` — scans the last 24h of recalls, joins against transcripts, appends to the log
53
+ - New endpoint field: `/api/recall/stats` includes `useful_hit_rate` and `useful_hit_count` over the same window as `hit_rate`
54
+
55
+ ## Out of scope (V1)
56
+
57
+ - MCP useful_hit_rate (blocked on conversation-id capture; track as follow-up)
58
+ - Real-time useful-hit detection (V1 is batch-scan, run on the daily digest cron)
59
+ - Distinguishing "agent quoted the recall" vs "agent acted on it" (the former is a proxy for the latter; V2 could refine)
60
+ - HTTP UI click-through (different metric — would live under a separate `ui_click_rate`)
61
+
62
+ ## V1 scope (shipping now)
63
+
64
+ - Ship the daily digest cron consuming existing `hit_rate` (this doc justifies the upgrade path)
65
+ - Add stub field `useful_hit_rate: null` to `/api/recall/stats` so the digest schema is forward-compatible
66
+ - Implement the scanner + CLI in a follow-up commit (target: within 7 days)
67
+
68
+ ## Why batch-scan vs hook-vs-hook real-time
69
+
70
+ A second Claude Code hook (`Stop` or `PostToolUse`) could compute usefulness in real time. Rejected because:
71
+ - Doubles installation surface (two hooks per agent runtime)
72
+ - Adds per-turn latency for a metric the user reads once/day
73
+ - Doesn't generalize to Hermes, pi, Codex, Gemini, Aider (no equivalent post-turn hook on most)
74
+ - Batch-scan reads the same transcript files the daemon already polls
75
+
76
+ ## Open questions
77
+
78
+ - Hit-label heuristic: substring match is cheap but noisy. Worth fuzzy matching session label tokens? Defer until V1 data shows the false-positive rate.
79
+ - Window for scan: hour-bucket vs day-bucket? Daily-bucket for now to match the digest cadence; revisit if cron interval changes.