nlm-memory 0.4.2 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +72 -34
- package/dist/cli/nlm.js +223 -33
- package/dist/cli/nlm.js.map +1 -1
- package/dist/core/adapters/cursor.d.ts +45 -0
- package/dist/core/adapters/cursor.js +397 -0
- package/dist/core/adapters/cursor.js.map +1 -0
- package/dist/core/adapters/from-source.js +10 -0
- package/dist/core/adapters/from-source.js.map +1 -1
- package/dist/core/adapters/windsurf.d.ts +44 -0
- package/dist/core/adapters/windsurf.js +299 -0
- package/dist/core/adapters/windsurf.js.map +1 -0
- package/dist/core/hook/claude-settings.d.ts +12 -5
- package/dist/core/hook/claude-settings.js +21 -6
- package/dist/core/hook/claude-settings.js.map +1 -1
- package/dist/core/sources/source-registry.d.ts +1 -1
- package/dist/core/sources/source-registry.js +18 -0
- package/dist/core/sources/source-registry.js.map +1 -1
- package/dist/core/storage/sqlite-session-store.d.ts +2 -0
- package/dist/core/storage/sqlite-session-store.js +38 -2
- package/dist/core/storage/sqlite-session-store.js.map +1 -1
- package/dist/hook/hook-auth.d.ts +13 -0
- package/dist/hook/hook-auth.js +19 -0
- package/dist/hook/hook-auth.js.map +1 -0
- package/dist/hook/prompt-recall-hook.js +7 -1
- package/dist/hook/prompt-recall-hook.js.map +1 -1
- package/dist/hook/session-start-hook.js +4 -1
- package/dist/hook/session-start-hook.js.map +1 -1
- package/dist/hook/stop-hook.js +4 -1
- package/dist/hook/stop-hook.js.map +1 -1
- package/dist/http/app.d.ts +2 -0
- package/dist/http/app.js +76 -1
- package/dist/http/app.js.map +1 -1
- package/dist/install/claude-code.js +1 -1
- package/dist/install/claude-code.js.map +1 -1
- package/dist/install/cursor.d.ts +25 -0
- package/dist/install/cursor.js +43 -0
- package/dist/install/cursor.js.map +1 -0
- package/dist/install/nlm-dir-perms.d.ts +19 -0
- package/dist/install/nlm-dir-perms.js +43 -0
- package/dist/install/nlm-dir-perms.js.map +1 -0
- package/dist/install/ollama.d.ts +18 -1
- package/dist/install/ollama.js +62 -7
- package/dist/install/ollama.js.map +1 -1
- package/dist/install/setup.d.ts +4 -0
- package/dist/install/setup.js +141 -18
- package/dist/install/setup.js.map +1 -1
- package/dist/install/windsurf.d.ts +25 -0
- package/dist/install/windsurf.js +43 -0
- package/dist/install/windsurf.js.map +1 -0
- package/dist/mcp/server.js +20 -1
- package/dist/mcp/server.js.map +1 -1
- package/dist/shared/types.d.ts +4 -0
- package/dist/ui/assets/{index-BA6IpU8g.css → index-Beo8psd-.css} +1 -1
- package/dist/ui/assets/index-CSPTTeeM.js +69 -0
- package/dist/ui/index.html +2 -2
- package/package.json +26 -1
- package/plugin/scripts/prompt-recall-hook.mjs +55 -4
- package/plugin/scripts/stop-hook.mjs +57 -6
- package/.agents/plugins/marketplace.json +0 -20
- package/.github/workflows/ci.yml +0 -30
- package/dist/ui/assets/index-B_qIVV0k.js +0 -69
- package/docs/methodology/re-derivation-rate.md +0 -112
- package/docs/methodology/useful-hit-rate.md +0 -79
- package/docs/plans/2026-05-20-fts5-lexical-recall.md +0 -1088
- package/docs/plans/2026-05-20-recall-daemon-wedge-fix.md +0 -662
- package/docs/plans/2026-05-20-recall-hook-design.md +0 -131
- package/docs/plans/2026-05-20-recall-hook-implementation.md +0 -1222
- package/docs/plans/desktop-product.md +0 -69
- package/docs/plans/factstore-design.md +0 -236
- package/logs/CHANGELOG/CHANGELOG-2026.md +0 -1389
- package/logs/CHANGELOG/CHANGELOG.md +0 -337
- package/migrations/000_initial_schema.sql +0 -174
- package/migrations/001_entity_type_rename.sql +0 -17
- package/migrations/002_adapter_state_extend.sql +0 -12
- package/migrations/003_session_embeddings.sql +0 -11
- package/migrations/004_facts.sql +0 -46
- package/migrations/005_sources.sql +0 -31
- package/migrations/006_providers.sql +0 -33
- package/migrations/007_source_tokens.sql +0 -17
- package/migrations/008_fts_rebuild.sql +0 -9
- package/migrations/009_session_embedding_chunks.sql +0 -46
- package/migrations/010_sources_opencode.sql +0 -30
- package/migrations/011_sources_hermes_agent.sql +0 -30
- package/migrations/012_sources_aider.sql +0 -30
- package/migrations/013_adapter_state_failure_count.sql +0 -12
- package/plugin-hermes-agent/README.md +0 -49
- package/plugin-hermes-agent/__init__.py +0 -75
- package/plugin-hermes-agent/plugin.yaml +0 -15
- package/scripts/backfill-citations.mjs +0 -0
- package/scripts/build-codex-plugin.mjs +0 -61
- package/scripts/deepseek-probe.mjs +0 -67
- package/scripts/extract-triples.mjs +0 -207
- package/scripts/longmemeval/embedding-cache.ts +0 -77
- package/scripts/longmemeval/fetch-dataset.sh +0 -25
- package/scripts/longmemeval/run-harness.ts +0 -315
- package/scripts/longmemeval/scorer.ts +0 -99
- package/scripts/longmemeval/tsconfig.json +0 -9
- package/scripts/longmemeval/types.ts +0 -35
- package/scripts/nlm-daily-digest.py +0 -239
- package/scripts/nlm-daily-digest.sh +0 -28
- package/src/cli/classify-parity.ts +0 -257
- package/src/cli/launchctl-helpers.ts +0 -49
- package/src/cli/nlm.ts +0 -885
- package/src/core/actions/actions-log.ts +0 -118
- package/src/core/actions/overlay.ts +0 -117
- package/src/core/adapters/aider.ts +0 -205
- package/src/core/adapters/claude-code.ts +0 -293
- package/src/core/adapters/common.ts +0 -54
- package/src/core/adapters/from-source.ts +0 -57
- package/src/core/adapters/hermes-agent.ts +0 -240
- package/src/core/adapters/hermes.ts +0 -277
- package/src/core/adapters/jsonl-generic.ts +0 -208
- package/src/core/adapters/opencode.ts +0 -281
- package/src/core/adapters/pi.ts +0 -264
- package/src/core/classifier/prompt.ts +0 -200
- package/src/core/dataset/build-dataset.ts +0 -463
- package/src/core/embedding/chunk-body.ts +0 -76
- package/src/core/embedding/embed-backfill.ts +0 -210
- package/src/core/embedding/embed-normalize.ts +0 -135
- package/src/core/facts/backfill-facts.ts +0 -254
- package/src/core/facts/extract-facts.ts +0 -50
- package/src/core/hook/citation-detect.ts +0 -124
- package/src/core/hook/cite-memo.ts +0 -68
- package/src/core/hook/claude-settings.ts +0 -166
- package/src/core/hook/gate.ts +0 -25
- package/src/core/hook/hook-log.ts +0 -41
- package/src/core/hook/memo-sweep.ts +0 -164
- package/src/core/hook/memo.ts +0 -67
- package/src/core/hook/pointer-block.ts +0 -26
- package/src/core/hook/select.ts +0 -32
- package/src/core/hook/transcript.ts +0 -121
- package/src/core/ingest/ingest-session.ts +0 -111
- package/src/core/providers/provider-models.ts +0 -100
- package/src/core/providers/provider-registry.ts +0 -196
- package/src/core/recall/citation-log.ts +0 -108
- package/src/core/recall/filter.ts +0 -27
- package/src/core/recall/index.ts +0 -6
- package/src/core/recall/match-fields.ts +0 -40
- package/src/core/recall/query-log.ts +0 -149
- package/src/core/recall/query-shape.ts +0 -66
- package/src/core/recall/recall-service.ts +0 -320
- package/src/core/recall/recent-log.ts +0 -59
- package/src/core/recall/tokenize.ts +0 -18
- package/src/core/recall/useful-scan.ts +0 -336
- package/src/core/recall-facts/fact-query-log.ts +0 -150
- package/src/core/recall-facts/fact-recall-service.ts +0 -327
- package/src/core/scheduler/scan-once.ts +0 -142
- package/src/core/scheduler/scheduler.ts +0 -225
- package/src/core/sources/source-registry.ts +0 -260
- package/src/core/storage/db-restore.ts +0 -133
- package/src/core/storage/live-status.ts +0 -45
- package/src/core/storage/migrate.ts +0 -72
- package/src/core/storage/sqlite-fact-store.ts +0 -304
- package/src/core/storage/sqlite-session-store.ts +0 -765
- package/src/hook/prompt-recall-hook.ts +0 -174
- package/src/hook/session-end-hook.ts +0 -81
- package/src/hook/session-start-hook.ts +0 -165
- package/src/hook/stop-hook.ts +0 -236
- package/src/http/app.ts +0 -1137
- package/src/install/claude-code.ts +0 -128
- package/src/install/codex.ts +0 -367
- package/src/install/hermes-agent.ts +0 -76
- package/src/install/hermes.ts +0 -78
- package/src/install/ollama.ts +0 -211
- package/src/install/setup.ts +0 -368
- package/src/llm/classifier-box.ts +0 -64
- package/src/llm/deepseek-client.ts +0 -150
- package/src/llm/env-autoload.ts +0 -55
- package/src/llm/ollama-client.ts +0 -189
- package/src/mcp/server.ts +0 -534
- package/src/ports/fact-store.ts +0 -102
- package/src/ports/llm-client.ts +0 -52
- package/src/ports/logger.ts +0 -16
- package/src/ports/session-store.ts +0 -45
- package/src/ports/transcript-adapter.ts +0 -55
- package/src/shared/types.ts +0 -145
- package/src/ui/App.tsx +0 -58
- package/src/ui/components/PromoteOpenButton.tsx +0 -65
- package/src/ui/components/SessionDrawer.tsx +0 -136
- package/src/ui/components/SideNav.tsx +0 -162
- package/src/ui/components/Skeleton.tsx +0 -107
- package/src/ui/index.html +0 -13
- package/src/ui/lib/actions.ts +0 -30
- package/src/ui/lib/api.ts +0 -92
- package/src/ui/lib/dataset.ts +0 -141
- package/src/ui/lib/registries.ts +0 -155
- package/src/ui/lib/view-settings.ts +0 -41
- package/src/ui/main.tsx +0 -15
- package/src/ui/pages/Live.tsx +0 -229
- package/src/ui/pages/Pulse.tsx +0 -415
- package/src/ui/pages/Recall.tsx +0 -190
- package/src/ui/pages/River.tsx +0 -308
- package/src/ui/pages/Search.tsx +0 -93
- package/src/ui/pages/Stub.tsx +0 -9
- package/src/ui/pages/Thread.tsx +0 -262
- package/src/ui/pages/settings/Classifier.tsx +0 -227
- package/src/ui/pages/settings/Data.tsx +0 -190
- package/src/ui/pages/settings/Index.tsx +0 -65
- package/src/ui/pages/settings/Labels.tsx +0 -224
- package/src/ui/pages/settings/Providers.tsx +0 -305
- package/src/ui/pages/settings/SettingsSubnav.tsx +0 -28
- package/src/ui/pages/settings/Sources.tsx +0 -326
- package/src/ui/pages/settings/Views.tsx +0 -96
- package/src/ui/styles.css +0 -1766
- package/src/ui/tsconfig.json +0 -21
- package/src/ui/vite.config.ts +0 -19
- package/tests/fixtures/claude_code/short_session.jsonl +0 -2
- package/tests/fixtures/claude_code/standard_iso.jsonl +0 -4
- package/tests/fixtures/claude_code/tool_heavy.jsonl +0 -8
- package/tests/fixtures/claude_code/with_subagent.jsonl +0 -7
- package/tests/fixtures/facts.ts +0 -17
- package/tests/fixtures/golden-corpus.ts +0 -85
- package/tests/fixtures/hermes/paired_request_dump.json +0 -24
- package/tests/fixtures/hermes/paired_session.json +0 -23
- package/tests/fixtures/hermes/request_dump.json +0 -28
- package/tests/fixtures/hermes/session_iso.json +0 -38
- package/tests/fixtures/hermes/session_unix.json +0 -38
- package/tests/fixtures/hermes/system_only.json +0 -18
- package/tests/fixtures/pi/error-connection-abort.jsonl +0 -8
- package/tests/fixtures/pi/short-successful.jsonl +0 -5
- package/tests/fixtures/pi/with-custom-message.jsonl +0 -6
- package/tests/fixtures/sessions.ts +0 -22
- package/tests/integration/backfill-facts.test.ts +0 -362
- package/tests/integration/citation-explicit.test.ts +0 -111
- package/tests/integration/cite-event.test.ts +0 -169
- package/tests/integration/cite-memo.test.ts +0 -87
- package/tests/integration/db-restore.test.ts +0 -153
- package/tests/integration/embed-backfill.test.ts +0 -176
- package/tests/integration/fact-supersedence.test.ts +0 -313
- package/tests/integration/fts-index.test.ts +0 -60
- package/tests/integration/getbyids-sqlite.test.ts +0 -60
- package/tests/integration/hermes-agent-hooks.test.ts +0 -248
- package/tests/integration/hook-claude-settings.test.ts +0 -205
- package/tests/integration/hook-log.test.ts +0 -54
- package/tests/integration/hook-memo.test.ts +0 -68
- package/tests/integration/hook-pre-compact.test.ts +0 -105
- package/tests/integration/hook-subagent-start.test.ts +0 -102
- package/tests/integration/http.test.ts +0 -401
- package/tests/integration/keyword-search-fts.test.ts +0 -66
- package/tests/integration/mcp-recall-logging.test.ts +0 -88
- package/tests/integration/mcp.test.ts +0 -248
- package/tests/integration/memo-sweep.test.ts +0 -91
- package/tests/integration/prompt-recall-hook.test.ts +0 -88
- package/tests/integration/provider-registry.test.ts +0 -107
- package/tests/integration/recall-golden.test.ts +0 -59
- package/tests/integration/recall-sqlite.test.ts +0 -169
- package/tests/integration/scheduler.test.ts +0 -391
- package/tests/integration/session-end-hook.test.ts +0 -48
- package/tests/integration/session-start-hook.test.ts +0 -126
- package/tests/integration/source-registry.test.ts +0 -120
- package/tests/integration/sqlite-fact-store.test.ts +0 -346
- package/tests/integration/stop-hook.test.ts +0 -560
- package/tests/integration/wal-checkpoint.test.ts +0 -49
- package/tests/unit/cli/launchctl-helpers.test.ts +0 -60
- package/tests/unit/core/adapters/aider.test.ts +0 -230
- package/tests/unit/core/adapters/claude-code.test.ts +0 -118
- package/tests/unit/core/adapters/hermes-agent.test.ts +0 -329
- package/tests/unit/core/adapters/hermes.test.ts +0 -81
- package/tests/unit/core/adapters/jsonl-generic.test.ts +0 -142
- package/tests/unit/core/adapters/opencode.test.ts +0 -354
- package/tests/unit/core/adapters/pi.test.ts +0 -110
- package/tests/unit/core/classifier/prompt.test.ts +0 -126
- package/tests/unit/core/embedding/chunk-body.test.ts +0 -100
- package/tests/unit/core/facts/extract-facts.test.ts +0 -117
- package/tests/unit/core/filter.test.ts +0 -40
- package/tests/unit/core/hook/citation-detect-cite-session.test.ts +0 -96
- package/tests/unit/core/hook/citation-detect.test.ts +0 -124
- package/tests/unit/core/hook/gate.test.ts +0 -29
- package/tests/unit/core/hook/pointer-block.test.ts +0 -22
- package/tests/unit/core/hook/select.test.ts +0 -66
- package/tests/unit/core/match-fields.test.ts +0 -39
- package/tests/unit/core/mcp-cite-session.test.ts +0 -51
- package/tests/unit/core/providers/provider-models.test.ts +0 -101
- package/tests/unit/core/query-shape.test.ts +0 -92
- package/tests/unit/core/recall-facts/fact-recall-service.test.ts +0 -258
- package/tests/unit/core/recall-service.test.ts +0 -200
- package/tests/unit/core/storage/live-status.test.ts +0 -54
- package/tests/unit/core/tokenize.test.ts +0 -32
- package/tests/unit/core/useful-scan.test.ts +0 -537
- package/tests/unit/llm/embed.test.ts +0 -93
- package/tests/unit/llm/ollama-client.test.ts +0 -124
- package/tests/unit/scripts/longmemeval-scorer.test.ts +0 -114
- package/tsconfig.json +0 -31
- package/tsconfig.test.json +0 -11
- package/vitest.config.ts +0 -22
|
@@ -1,662 +0,0 @@
|
|
|
1
|
-
# Recall Daemon Wedge Fix — Implementation Plan
|
|
2
|
-
|
|
3
|
-
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
4
|
-
|
|
5
|
-
**Goal:** Stop the NLM daemon's `/api/recall` from wedging by making recall fetch only the hit sessions (not the whole 99 MB corpus), and bound the SQLite WAL with checkpoint management.
|
|
6
|
-
|
|
7
|
-
**Architecture:** `RecallService.search()` currently calls `SqliteSessionStore.list()` on every request, which `SELECT`s the `body` column — 99 MB of session markdown across 2,097 rows — synchronously on the Node event loop (measured 239ms vs 35ms without `body`). better-sqlite3 is synchronous and single-threaded, so concurrent recalls serialize into multi-second head-of-line blocking. The fix: the FTS5 `keywordSearch` / sqlite-vec `semanticSearch` legs already return ranked session IDs — resolve only those (~15) sessions via a new `getByIds` store method that omits `body`, and apply the entity/kind filter after the fetch. Separately, add a periodic `wal_checkpoint(TRUNCATE)` to the daemon so the WAL (currently 38 MB, never drained) stays bounded.
|
|
8
|
-
|
|
9
|
-
**Tech Stack:** TypeScript (NodeNext, strict), better-sqlite3 11, vitest, Node 22. Hexagonal — `RecallService` depends on the `SessionStore` port; `SqliteSessionStore` is the adapter.
|
|
10
|
-
|
|
11
|
-
**Root cause** (confirmed by profiling): `sample` of the daemon during a wedge showed ~50% of the window in one synchronous better-sqlite3 query, 85% of that in `vdbeColumnFromOverflow` → `pread` (reading `body` overflow pages). A `/api/health` call measured 8.2s during recall load — the event loop is blocked.
|
|
12
|
-
|
|
13
|
-
**Branch:** Create and work on `fix/recall-daemon-wedge` off `main`.
|
|
14
|
-
|
|
15
|
-
**Must stay green:** `tests/integration/recall-golden.test.ts` (the recall-correctness regression gate) and `tests/integration/recall-sqlite.test.ts`.
|
|
16
|
-
|
|
17
|
-
**Out of scope:** Reducing the leaked `nlm mcp` processes; rearchitecting better-sqlite3 off the main thread. Do not do these.
|
|
18
|
-
|
|
19
|
-
---
|
|
20
|
-
|
|
21
|
-
## File Structure
|
|
22
|
-
|
|
23
|
-
| File | Change |
|
|
24
|
-
|---|---|
|
|
25
|
-
| `src/ports/session-store.ts` | Add `getByIds` to the `SessionStore` interface. |
|
|
26
|
-
| `src/core/storage/sqlite-session-store.ts` | Implement `getByIds` (no `body` column); add `checkpoint()`. |
|
|
27
|
-
| `src/core/recall/recall-service.ts` | `search()` fetches only hit sessions; delete `runKeyword`/`runSemantic`; add `uniqueIds`. |
|
|
28
|
-
| `src/cli/nlm.ts` | Wire a periodic + boot `wal_checkpoint` into `nlm start`. |
|
|
29
|
-
| `tests/unit/core/recall-service.test.ts` | `InMemoryStore` fake gains `getByIds` + call counters; add root-cause test. |
|
|
30
|
-
| `tests/integration/getbyids-sqlite.test.ts` | New — covers `SqliteSessionStore.getByIds`. |
|
|
31
|
-
| `tests/integration/wal-checkpoint.test.ts` | New — covers `SqliteSessionStore.checkpoint()`. |
|
|
32
|
-
|
|
33
|
-
---
|
|
34
|
-
|
|
35
|
-
## Task 1: `getByIds` on the SessionStore port
|
|
36
|
-
|
|
37
|
-
Add a batched, body-free session fetch. The recall path needs id/label/summary/entities/decisions/open/status for a handful of sessions — never `body`. Omitting `body` is the core of the fix (it is the 99 MB / 239ms cost).
|
|
38
|
-
|
|
39
|
-
**Files:**
|
|
40
|
-
- Modify: `src/ports/session-store.ts`
|
|
41
|
-
- Modify: `src/core/storage/sqlite-session-store.ts`
|
|
42
|
-
- Modify: `tests/unit/core/recall-service.test.ts` (add `getByIds` to the `InMemoryStore` fake — keeps typecheck green when the port changes)
|
|
43
|
-
- Test: `tests/integration/getbyids-sqlite.test.ts`
|
|
44
|
-
|
|
45
|
-
- [ ] **Step 1: Write the failing test**
|
|
46
|
-
|
|
47
|
-
Create `tests/integration/getbyids-sqlite.test.ts`:
|
|
48
|
-
|
|
49
|
-
```typescript
|
|
50
|
-
/**
|
|
51
|
-
* SqliteSessionStore.getByIds — batched, body-free session fetch used by
|
|
52
|
-
* the recall path so it never loads the full corpus.
|
|
53
|
-
*/
|
|
54
|
-
|
|
55
|
-
import { mkdtempSync, rmSync } from "node:fs";
|
|
56
|
-
import { tmpdir } from "node:os";
|
|
57
|
-
import { join, resolve } from "node:path";
|
|
58
|
-
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
|
59
|
-
import { SqliteSessionStore } from "../../src/core/storage/sqlite-session-store.js";
|
|
60
|
-
import { makeSession } from "../fixtures/sessions.js";
|
|
61
|
-
|
|
62
|
-
const MIGRATIONS_DIR = resolve(__dirname, "../../migrations");
|
|
63
|
-
|
|
64
|
-
describe("SqliteSessionStore.getByIds", () => {
|
|
65
|
-
let tmp: string;
|
|
66
|
-
let store: SqliteSessionStore;
|
|
67
|
-
|
|
68
|
-
beforeEach(() => {
|
|
69
|
-
tmp = mkdtempSync(join(tmpdir(), "nlm-getbyids-"));
|
|
70
|
-
store = new SqliteSessionStore({
|
|
71
|
-
dbPath: join(tmp, "canonical.sqlite"),
|
|
72
|
-
migrationsDir: MIGRATIONS_DIR,
|
|
73
|
-
});
|
|
74
|
-
store.insertSessionForTest(
|
|
75
|
-
makeSession({ id: "s1", label: "alpha", body: "BODY ONE", entities: ["NLM"], decisions: ["d1"] }),
|
|
76
|
-
);
|
|
77
|
-
store.insertSessionForTest(
|
|
78
|
-
makeSession({ id: "s2", label: "beta", body: "BODY TWO", open: ["q1"] }),
|
|
79
|
-
);
|
|
80
|
-
store.insertSessionForTest(makeSession({ id: "s3", label: "gamma" }));
|
|
81
|
-
});
|
|
82
|
-
|
|
83
|
-
afterEach(() => {
|
|
84
|
-
store.close();
|
|
85
|
-
rmSync(tmp, { recursive: true, force: true });
|
|
86
|
-
});
|
|
87
|
-
|
|
88
|
-
it("returns only the requested sessions", async () => {
|
|
89
|
-
const got = await store.getByIds(["s1", "s3"]);
|
|
90
|
-
expect(got.map((s) => s.id).sort()).toEqual(["s1", "s3"]);
|
|
91
|
-
});
|
|
92
|
-
|
|
93
|
-
it("returns an empty array for an empty id list", async () => {
|
|
94
|
-
expect(await store.getByIds([])).toEqual([]);
|
|
95
|
-
});
|
|
96
|
-
|
|
97
|
-
it("ignores ids that do not exist", async () => {
|
|
98
|
-
const got = await store.getByIds(["s2", "missing"]);
|
|
99
|
-
expect(got.map((s) => s.id)).toEqual(["s2"]);
|
|
100
|
-
});
|
|
101
|
-
|
|
102
|
-
it("populates entities and markers but omits body (body is empty)", async () => {
|
|
103
|
-
const got = await store.getByIds(["s1"]);
|
|
104
|
-
const s1 = got[0];
|
|
105
|
-
expect(s1?.entities).toEqual(["NLM"]);
|
|
106
|
-
expect(s1?.decisions).toEqual(["d1"]);
|
|
107
|
-
expect(s1?.body).toBe("");
|
|
108
|
-
});
|
|
109
|
-
});
|
|
110
|
-
```
|
|
111
|
-
|
|
112
|
-
- [ ] **Step 2: Run the test to verify it fails**
|
|
113
|
-
|
|
114
|
-
Run: `npm test -- tests/integration/getbyids-sqlite.test.ts`
|
|
115
|
-
Expected: FAIL — `store.getByIds is not a function`.
|
|
116
|
-
|
|
117
|
-
- [ ] **Step 3: Add `getByIds` to the port**
|
|
118
|
-
|
|
119
|
-
In `src/ports/session-store.ts`, add this method to the `SessionStore` interface, immediately after the `getById` declaration:
|
|
120
|
-
|
|
121
|
-
```typescript
|
|
122
|
-
getByIds(ids: ReadonlyArray<string>): Promise<ReadonlyArray<Session>>;
|
|
123
|
-
```
|
|
124
|
-
|
|
125
|
-
- [ ] **Step 4: Implement `getByIds` in `SqliteSessionStore`**
|
|
126
|
-
|
|
127
|
-
First read `src/core/storage/sqlite-session-store.ts` to confirm the exact shapes of `SessionRow` (a type with `id, runtime, runtime_session_id, started_at, ended_at, duration_min, label, summary, status, transcript_kind, transcript_path, body`), the private helpers `loadEntities(ids)` / `loadMarkers(ids)`, the free function `loadActionOverlay(db)`, and `rowToSession(row, entitiesById, markersById, overlay)` (which sets `body: row.body ?? ""`).
|
|
128
|
-
|
|
129
|
-
Add this method immediately after `getById` (around line 441):
|
|
130
|
-
|
|
131
|
-
```typescript
|
|
132
|
-
/**
|
|
133
|
-
* Batched session fetch for the recall path. Deliberately omits the
|
|
134
|
-
* `body` column — body is ~48KB/row of session markdown that recall
|
|
135
|
-
* never reads, and SELECTing it for the corpus is what wedged the
|
|
136
|
-
* daemon. Resolved sessions carry `body: ""`.
|
|
137
|
-
*/
|
|
138
|
-
async getByIds(ids: ReadonlyArray<string>): Promise<ReadonlyArray<Session>> {
|
|
139
|
-
if (ids.length === 0) return [];
|
|
140
|
-
const placeholders = ids.map(() => "?").join(",");
|
|
141
|
-
const rows = this.db
|
|
142
|
-
.prepare<string[], Omit<SessionRow, "body">>(`
|
|
143
|
-
SELECT id, runtime, runtime_session_id, started_at, ended_at, duration_min,
|
|
144
|
-
label, summary, status, transcript_kind, transcript_path
|
|
145
|
-
FROM sessions
|
|
146
|
-
WHERE id IN (${placeholders})
|
|
147
|
-
`)
|
|
148
|
-
.all(...ids);
|
|
149
|
-
|
|
150
|
-
if (rows.length === 0) return [];
|
|
151
|
-
const foundIds = rows.map((r) => r.id);
|
|
152
|
-
const entitiesByIdMap = this.loadEntities(foundIds);
|
|
153
|
-
const markersByIdMap = this.loadMarkers(foundIds);
|
|
154
|
-
const overlay = loadActionOverlay(this.db);
|
|
155
|
-
return rows.map((r) =>
|
|
156
|
-
this.rowToSession({ ...r, body: null }, entitiesByIdMap, markersByIdMap, overlay),
|
|
157
|
-
);
|
|
158
|
-
}
|
|
159
|
-
```
|
|
160
|
-
|
|
161
|
-
Note: `{ ...r, body: null }` reconstitutes a full `SessionRow` so `rowToSession` is reused unchanged — `rowToSession` does `body: row.body ?? ""`, so `null` yields `""`. Do not modify `rowToSession`, `list`, or `getById`.
|
|
162
|
-
|
|
163
|
-
- [ ] **Step 5: Add `getByIds` to the `InMemoryStore` test fake**
|
|
164
|
-
|
|
165
|
-
In `tests/unit/core/recall-service.test.ts`, the `InMemoryStore` class `implements SessionStore` and will not compile without the new method. Add this method to `InMemoryStore`, immediately after its `getById` method:
|
|
166
|
-
|
|
167
|
-
```typescript
|
|
168
|
-
async getByIds(ids: ReadonlyArray<string>): Promise<ReadonlyArray<Session>> {
|
|
169
|
-
return this.sessions.filter((s) => ids.includes(s.id));
|
|
170
|
-
}
|
|
171
|
-
```
|
|
172
|
-
|
|
173
|
-
- [ ] **Step 6: Run the tests**
|
|
174
|
-
|
|
175
|
-
Run: `npm test -- tests/integration/getbyids-sqlite.test.ts && npm run typecheck`
|
|
176
|
-
Expected: PASS — all 4 `getByIds` cases; typecheck clean (the port change is satisfied by both `SqliteSessionStore` and the `InMemoryStore` fake).
|
|
177
|
-
|
|
178
|
-
Run: `npm test -- tests/integration/recall-golden.test.ts`
|
|
179
|
-
Expected: PASS — golden gate still green (`RecallService` unchanged this task).
|
|
180
|
-
|
|
181
|
-
- [ ] **Step 7: Commit**
|
|
182
|
-
|
|
183
|
-
```bash
|
|
184
|
-
git checkout -b fix/recall-daemon-wedge
|
|
185
|
-
git add src/ports/session-store.ts src/core/storage/sqlite-session-store.ts tests/unit/core/recall-service.test.ts tests/integration/getbyids-sqlite.test.ts
|
|
186
|
-
git commit -m "feat: add body-free getByIds batch fetch to the SessionStore port"
|
|
187
|
-
```
|
|
188
|
-
|
|
189
|
-
---
|
|
190
|
-
|
|
191
|
-
## Task 2: Recall fetches only the hits, not the whole corpus
|
|
192
|
-
|
|
193
|
-
Refactor `RecallService.search()` so it never calls `store.list()`. The search legs already return ranked IDs — fetch only those sessions via `getByIds`, then apply the entity/kind filter post-fetch. This is the root-cause fix.
|
|
194
|
-
|
|
195
|
-
**Files:**
|
|
196
|
-
- Modify: `src/core/recall/recall-service.ts`
|
|
197
|
-
- Modify: `tests/unit/core/recall-service.test.ts`
|
|
198
|
-
|
|
199
|
-
- [ ] **Step 1: Write the failing root-cause test**
|
|
200
|
-
|
|
201
|
-
In `tests/unit/core/recall-service.test.ts`, add call counters to the `InMemoryStore` fake. Change the class so it has two public counter fields and increments them. The fake currently looks like:
|
|
202
|
-
|
|
203
|
-
```typescript
|
|
204
|
-
class InMemoryStore implements SessionStore {
|
|
205
|
-
constructor(
|
|
206
|
-
private readonly sessions: Session[],
|
|
207
|
-
private readonly neighbors: SemanticNeighbor[] = [],
|
|
208
|
-
private readonly keywordHits: KeywordNeighbor[] = [],
|
|
209
|
-
) {}
|
|
210
|
-
async list(): Promise<ReadonlyArray<Session>> {
|
|
211
|
-
return this.sessions;
|
|
212
|
-
}
|
|
213
|
-
async getById(id: string): Promise<Session | null> {
|
|
214
|
-
return this.sessions.find((s) => s.id === id) ?? null;
|
|
215
|
-
}
|
|
216
|
-
async getByIds(ids: ReadonlyArray<string>): Promise<ReadonlyArray<Session>> {
|
|
217
|
-
return this.sessions.filter((s) => ids.includes(s.id));
|
|
218
|
-
}
|
|
219
|
-
async semanticSearch(): Promise<ReadonlyArray<SemanticNeighbor>> {
|
|
220
|
-
return this.neighbors;
|
|
221
|
-
}
|
|
222
|
-
async keywordSearch(): Promise<ReadonlyArray<KeywordNeighbor>> {
|
|
223
|
-
return this.keywordHits;
|
|
224
|
-
}
|
|
225
|
-
async updateStatus(): Promise<void> {}
|
|
226
|
-
}
|
|
227
|
-
```
|
|
228
|
-
|
|
229
|
-
Replace it with:
|
|
230
|
-
|
|
231
|
-
```typescript
|
|
232
|
-
class InMemoryStore implements SessionStore {
|
|
233
|
-
listCalls = 0;
|
|
234
|
-
getByIdsCalls = 0;
|
|
235
|
-
constructor(
|
|
236
|
-
private readonly sessions: Session[],
|
|
237
|
-
private readonly neighbors: SemanticNeighbor[] = [],
|
|
238
|
-
private readonly keywordHits: KeywordNeighbor[] = [],
|
|
239
|
-
) {}
|
|
240
|
-
async list(): Promise<ReadonlyArray<Session>> {
|
|
241
|
-
this.listCalls += 1;
|
|
242
|
-
return this.sessions;
|
|
243
|
-
}
|
|
244
|
-
async getById(id: string): Promise<Session | null> {
|
|
245
|
-
return this.sessions.find((s) => s.id === id) ?? null;
|
|
246
|
-
}
|
|
247
|
-
async getByIds(ids: ReadonlyArray<string>): Promise<ReadonlyArray<Session>> {
|
|
248
|
-
this.getByIdsCalls += 1;
|
|
249
|
-
return this.sessions.filter((s) => ids.includes(s.id));
|
|
250
|
-
}
|
|
251
|
-
async semanticSearch(): Promise<ReadonlyArray<SemanticNeighbor>> {
|
|
252
|
-
return this.neighbors;
|
|
253
|
-
}
|
|
254
|
-
async keywordSearch(): Promise<ReadonlyArray<KeywordNeighbor>> {
|
|
255
|
-
return this.keywordHits;
|
|
256
|
-
}
|
|
257
|
-
async updateStatus(): Promise<void> {}
|
|
258
|
-
}
|
|
259
|
-
```
|
|
260
|
-
|
|
261
|
-
Then add this test inside the `describe("RecallService.search", ...)` block:
|
|
262
|
-
|
|
263
|
-
```typescript
|
|
264
|
-
it("resolves only the hit sessions and never loads the full corpus", async () => {
|
|
265
|
-
const big: Session[] = Array.from({ length: 100 }, (_, i) =>
|
|
266
|
-
makeSession({ id: `s${i}`, label: `session ${i}` }),
|
|
267
|
-
);
|
|
268
|
-
const store = new InMemoryStore(big, [], [
|
|
269
|
-
{ sessionId: "s7", score: 9 },
|
|
270
|
-
{ sessionId: "s42", score: 8 },
|
|
271
|
-
]);
|
|
272
|
-
const svc = new RecallService({ store, llm: new StubEmbedder() });
|
|
273
|
-
const result = await svc.search({ query: "anything", mode: "keyword" });
|
|
274
|
-
expect(result.results.map((r) => r.id)).toEqual(["s7", "s42"]);
|
|
275
|
-
expect(store.listCalls).toBe(0);
|
|
276
|
-
expect(store.getByIdsCalls).toBe(1);
|
|
277
|
-
});
|
|
278
|
-
```
|
|
279
|
-
|
|
280
|
-
- [ ] **Step 2: Run the test to verify it fails**
|
|
281
|
-
|
|
282
|
-
Run: `npm test -- tests/unit/core/recall-service.test.ts -t "resolves only the hit sessions"`
|
|
283
|
-
Expected: FAIL — `store.listCalls` is `1` (the current `search()` calls `list()`), not `0`.
|
|
284
|
-
|
|
285
|
-
- [ ] **Step 3: Refactor `RecallService.search`**
|
|
286
|
-
|
|
287
|
-
In `src/core/recall/recall-service.ts`:
|
|
288
|
-
|
|
289
|
-
(a) Change the port import to add the neighbor types. Replace:
|
|
290
|
-
|
|
291
|
-
```typescript
|
|
292
|
-
import type { SessionStore } from "@ports/session-store.js";
|
|
293
|
-
```
|
|
294
|
-
|
|
295
|
-
with:
|
|
296
|
-
|
|
297
|
-
```typescript
|
|
298
|
-
import type {
|
|
299
|
-
KeywordNeighbor,
|
|
300
|
-
SemanticNeighbor,
|
|
301
|
-
SessionStore,
|
|
302
|
-
} from "@ports/session-store.js";
|
|
303
|
-
```
|
|
304
|
-
|
|
305
|
-
(b) Replace the entire `search` method (currently lines 39–106, from `async search(` through its closing `}`) with:
|
|
306
|
-
|
|
307
|
-
```typescript
|
|
308
|
-
async search(input: RecallQuery): Promise<RecallResult> {
|
|
309
|
-
const mode: RecallMode = input.mode ?? "keyword";
|
|
310
|
-
const limit = clampLimit(input.limit);
|
|
311
|
-
const entity = input.entity ?? null;
|
|
312
|
-
const kind = input.kind ?? null;
|
|
313
|
-
|
|
314
|
-
const empty: RecallResult = {
|
|
315
|
-
query: input.query,
|
|
316
|
-
entity,
|
|
317
|
-
kind,
|
|
318
|
-
mode,
|
|
319
|
-
limit,
|
|
320
|
-
total: 0,
|
|
321
|
-
results: [],
|
|
322
|
-
};
|
|
323
|
-
|
|
324
|
-
if (!input.query && !entity && !kind) return empty;
|
|
325
|
-
|
|
326
|
-
// 1. Search legs — ranked neighbor IDs only. No session bodies loaded.
|
|
327
|
-
const kwNeighbors: ReadonlyArray<KeywordNeighbor> =
|
|
328
|
-
(mode === "keyword" || mode === "hybrid") && input.query
|
|
329
|
-
? await this.deps.store.keywordSearch(input.query, limit * KEYWORD_OVERFETCH)
|
|
330
|
-
: [];
|
|
331
|
-
|
|
332
|
-
let semNeighbors: ReadonlyArray<SemanticNeighbor> = [];
|
|
333
|
-
let semError: "ollama_unreachable" | null = null;
|
|
334
|
-
if ((mode === "semantic" || mode === "hybrid") && input.query) {
|
|
335
|
-
try {
|
|
336
|
-
const embedding = await this.deps.llm.embed(input.query, "query");
|
|
337
|
-
semNeighbors = await this.deps.store.semanticSearch(
|
|
338
|
-
embedding.vector,
|
|
339
|
-
limit * SEMANTIC_OVERFETCH,
|
|
340
|
-
);
|
|
341
|
-
} catch (err) {
|
|
342
|
-
if (err instanceof LLMUnreachableError) {
|
|
343
|
-
semError = "ollama_unreachable";
|
|
344
|
-
} else {
|
|
345
|
-
throw err;
|
|
346
|
-
}
|
|
347
|
-
}
|
|
348
|
-
}
|
|
349
|
-
|
|
350
|
-
if (mode === "semantic" && semError) {
|
|
351
|
-
return { ...empty, modeUnavailable: semError };
|
|
352
|
-
}
|
|
353
|
-
|
|
354
|
-
// 2. Resolve ONLY the hit sessions — never the whole corpus. The
|
|
355
|
-
// entity/kind filter is applied to the fetched hits; a filtered-out
|
|
356
|
-
// session is absent from byId and is skipped during resolution.
|
|
357
|
-
const hitIds = uniqueIds(kwNeighbors, semNeighbors);
|
|
358
|
-
const hitSessions = await this.deps.store.getByIds(hitIds);
|
|
359
|
-
const filterArgs: { entity?: string; kind?: typeof input.kind } = {};
|
|
360
|
-
if (input.entity !== undefined) filterArgs.entity = input.entity;
|
|
361
|
-
if (input.kind !== undefined) filterArgs.kind = input.kind;
|
|
362
|
-
const byId = new Map<string, Session>(
|
|
363
|
-
applyFilter(hitSessions, filterArgs).map((s) => [s.id, s]),
|
|
364
|
-
);
|
|
365
|
-
|
|
366
|
-
// 3. Build hits from the resolved sessions, preserving leg rank order.
|
|
367
|
-
const queryTokens = input.query
|
|
368
|
-
? new Set(tokenSet(input.query))
|
|
369
|
-
: new Set<string>();
|
|
370
|
-
|
|
371
|
-
const kwHits: KeywordHit[] = [];
|
|
372
|
-
for (const n of kwNeighbors) {
|
|
373
|
-
const session = byId.get(n.sessionId);
|
|
374
|
-
if (!session) continue;
|
|
375
|
-
kwHits.push({
|
|
376
|
-
session,
|
|
377
|
-
score: n.score,
|
|
378
|
-
matchedIn: keywordMatchFields(session, queryTokens),
|
|
379
|
-
});
|
|
380
|
-
}
|
|
381
|
-
|
|
382
|
-
const semHits: SemanticHit[] = [];
|
|
383
|
-
for (const n of semNeighbors) {
|
|
384
|
-
const session = byId.get(n.sessionId);
|
|
385
|
-
if (!session) continue;
|
|
386
|
-
semHits.push({ session, similarity: cosineFromL2(n.distance) });
|
|
387
|
-
}
|
|
388
|
-
|
|
389
|
-
// 4. Finalize per mode.
|
|
390
|
-
if (mode === "keyword") {
|
|
391
|
-
return finalize(input.query, entity, kind, mode, limit, kwHits.map(toKeywordHit));
|
|
392
|
-
}
|
|
393
|
-
if (mode === "semantic") {
|
|
394
|
-
return finalize(input.query, entity, kind, mode, limit, semHits.map(toSemanticHit));
|
|
395
|
-
}
|
|
396
|
-
const merged = mergeHybrid(kwHits, semHits, byId);
|
|
397
|
-
const result = finalize(input.query, entity, kind, mode, limit, merged);
|
|
398
|
-
return semError ? { ...result, modeUnavailable: semError } : result;
|
|
399
|
-
}
|
|
400
|
-
```
|
|
401
|
-
|
|
402
|
-
(c) Delete the `runSemantic` and `runKeyword` private methods entirely (currently lines 108–142). After this, the `RecallService` class body is just the constructor and `search`.
|
|
403
|
-
|
|
404
|
-
(d) Add this module-level helper immediately after the `RecallService` class closing brace (before the `interface KeywordHit` declaration):
|
|
405
|
-
|
|
406
|
-
```typescript
|
|
407
|
-
function uniqueIds(
|
|
408
|
-
kw: ReadonlyArray<KeywordNeighbor>,
|
|
409
|
-
sem: ReadonlyArray<SemanticNeighbor>,
|
|
410
|
-
): ReadonlyArray<string> {
|
|
411
|
-
const ids = new Set<string>();
|
|
412
|
-
for (const n of kw) ids.add(n.sessionId);
|
|
413
|
-
for (const n of sem) ids.add(n.sessionId);
|
|
414
|
-
return [...ids];
|
|
415
|
-
}
|
|
416
|
-
```
|
|
417
|
-
|
|
418
|
-
Leave `KeywordHit`, `SemanticHit`, `mergeHybrid`, `toKeywordHit`, `toSemanticHit`, `sessionHitFields`, `finalize`, `clampLimit`, `cosineFromL2`, `round4`, `uniqueFields` unchanged. `applyFilter`, `keywordMatchFields`, `tokenSet` imports stay.
|
|
419
|
-
|
|
420
|
-
- [ ] **Step 4: Run the root-cause test**
|
|
421
|
-
|
|
422
|
-
Run: `npm test -- tests/unit/core/recall-service.test.ts`
|
|
423
|
-
Expected: PASS — the new root-cause test and all pre-existing `RecallService.search` tests. The pre-existing tests feed `neighbors`/`keywordHits` to the fake and assert ranking/filter/limit/hybrid behavior; the refactor preserves all of it (the fake's `getByIds` returns the corpus sessions matching the hit ids, `applyFilter` drops entity/kind mismatches exactly as before).
|
|
424
|
-
|
|
425
|
-
If a pre-existing test fails, do NOT weaken it — the refactor has a behavior bug; fix the refactor. The most likely culprit is the entity-filter test: confirm `applyFilter(hitSessions, filterArgs)` runs on the fetched hits and that filtered-out sessions are correctly absent from `byId`.
|
|
426
|
-
|
|
427
|
-
- [ ] **Step 5: Run the integration + golden suites**
|
|
428
|
-
|
|
429
|
-
Run: `npm test -- tests/integration/recall-sqlite.test.ts tests/integration/recall-golden.test.ts && npm run typecheck`
|
|
430
|
-
Expected: PASS — `recall-golden.test.ts` (the recall-correctness regression gate) green proves the fetch-only-hits refactor did not regress recall quality; typecheck clean.
|
|
431
|
-
|
|
432
|
-
- [ ] **Step 6: Run the full suite**
|
|
433
|
-
|
|
434
|
-
Run: `npm test`
|
|
435
|
-
Expected: PASS — whole suite green. If anything outside the recall files broke, STOP and report it.
|
|
436
|
-
|
|
437
|
-
- [ ] **Step 7: Commit**
|
|
438
|
-
|
|
439
|
-
```bash
|
|
440
|
-
git add src/core/recall/recall-service.ts tests/unit/core/recall-service.test.ts
|
|
441
|
-
git commit -m "fix: recall resolves only hit sessions, never loads the full corpus"
|
|
442
|
-
```
|
|
443
|
-
|
|
444
|
-
---
|
|
445
|
-
|
|
446
|
-
## Task 3: WAL checkpoint management
|
|
447
|
-
|
|
448
|
-
The codebase has no checkpoint management — the live WAL has grown to 38 MB and never drains. Add a `checkpoint()` method to the store and a periodic + boot checkpoint to the daemon.
|
|
449
|
-
|
|
450
|
-
**Files:**
|
|
451
|
-
- Modify: `src/core/storage/sqlite-session-store.ts`
|
|
452
|
-
- Modify: `src/cli/nlm.ts`
|
|
453
|
-
- Test: `tests/integration/wal-checkpoint.test.ts`
|
|
454
|
-
|
|
455
|
-
- [ ] **Step 1: Write the failing test**
|
|
456
|
-
|
|
457
|
-
Create `tests/integration/wal-checkpoint.test.ts`:
|
|
458
|
-
|
|
459
|
-
```typescript
|
|
460
|
-
/**
|
|
461
|
-
* SqliteSessionStore.checkpoint — drains the WAL into the main DB and
|
|
462
|
-
* truncates the -wal file, so it cannot grow unbounded.
|
|
463
|
-
*/
|
|
464
|
-
|
|
465
|
-
import { mkdtempSync, rmSync, statSync } from "node:fs";
|
|
466
|
-
import { tmpdir } from "node:os";
|
|
467
|
-
import { join, resolve } from "node:path";
|
|
468
|
-
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
|
469
|
-
import { SqliteSessionStore } from "../../src/core/storage/sqlite-session-store.js";
|
|
470
|
-
import { makeSession } from "../fixtures/sessions.js";
|
|
471
|
-
|
|
472
|
-
const MIGRATIONS_DIR = resolve(__dirname, "../../migrations");
|
|
473
|
-
|
|
474
|
-
describe("SqliteSessionStore.checkpoint", () => {
|
|
475
|
-
let tmp: string;
|
|
476
|
-
let dbPath: string;
|
|
477
|
-
let store: SqliteSessionStore;
|
|
478
|
-
|
|
479
|
-
beforeEach(() => {
|
|
480
|
-
tmp = mkdtempSync(join(tmpdir(), "nlm-wal-"));
|
|
481
|
-
dbPath = join(tmp, "canonical.sqlite");
|
|
482
|
-
store = new SqliteSessionStore({ dbPath, migrationsDir: MIGRATIONS_DIR });
|
|
483
|
-
});
|
|
484
|
-
|
|
485
|
-
afterEach(() => {
|
|
486
|
-
store.close();
|
|
487
|
-
rmSync(tmp, { recursive: true, force: true });
|
|
488
|
-
});
|
|
489
|
-
|
|
490
|
-
it("truncates the -wal file after checkpoint", () => {
|
|
491
|
-
for (let i = 0; i < 30; i++) {
|
|
492
|
-
store.insertSessionForTest(
|
|
493
|
-
makeSession({ id: `s${i}`, label: `session ${i}`, body: "x".repeat(5000) }),
|
|
494
|
-
);
|
|
495
|
-
}
|
|
496
|
-
const walBefore = statSync(`${dbPath}-wal`).size;
|
|
497
|
-
expect(walBefore).toBeGreaterThan(0);
|
|
498
|
-
|
|
499
|
-
store.checkpoint();
|
|
500
|
-
|
|
501
|
-
const walAfter = statSync(`${dbPath}-wal`).size;
|
|
502
|
-
expect(walAfter).toBe(0);
|
|
503
|
-
});
|
|
504
|
-
|
|
505
|
-
it("is safe to call when the WAL is already empty", () => {
|
|
506
|
-
expect(() => store.checkpoint()).not.toThrow();
|
|
507
|
-
});
|
|
508
|
-
});
|
|
509
|
-
```
|
|
510
|
-
|
|
511
|
-
- [ ] **Step 2: Run the test to verify it fails**
|
|
512
|
-
|
|
513
|
-
Run: `npm test -- tests/integration/wal-checkpoint.test.ts`
|
|
514
|
-
Expected: FAIL — `store.checkpoint is not a function`.
|
|
515
|
-
|
|
516
|
-
- [ ] **Step 3: Add `checkpoint()` to `SqliteSessionStore`**
|
|
517
|
-
|
|
518
|
-
In `src/core/storage/sqlite-session-store.ts`, add this method immediately after the `close()` method (near the top of the class, around line 116):
|
|
519
|
-
|
|
520
|
-
```typescript
|
|
521
|
-
/**
|
|
522
|
-
* Drains the WAL into the main database and truncates the -wal file.
|
|
523
|
-
* WAL mode is on but nothing else checkpoints, so the file grows
|
|
524
|
-
* unbounded under continuous readers. The daemon calls this on an
|
|
525
|
-
* interval. Synchronous — keep the WAL small so each call is cheap.
|
|
526
|
-
*/
|
|
527
|
-
checkpoint(): void {
|
|
528
|
-
this.db.pragma("wal_checkpoint(TRUNCATE)");
|
|
529
|
-
}
|
|
530
|
-
```
|
|
531
|
-
|
|
532
|
-
- [ ] **Step 4: Run the test to verify it passes**
|
|
533
|
-
|
|
534
|
-
Run: `npm test -- tests/integration/wal-checkpoint.test.ts`
|
|
535
|
-
Expected: PASS — both cases.
|
|
536
|
-
|
|
537
|
-
- [ ] **Step 5: Wire periodic + boot checkpoint into the daemon**
|
|
538
|
-
|
|
539
|
-
In `src/cli/nlm.ts`, find the `start` command's `.action(async (opts) => { ... })` (around line 148). After the `serve({ fetch: app.fetch, port: p }, ...)` call and before the `if (opts.scheduler !== false)` block, insert:
|
|
540
|
-
|
|
541
|
-
```typescript
|
|
542
|
-
// Keep the SQLite WAL bounded. WAL mode is on but nothing else
|
|
543
|
-
// checkpoints it; under continuous readers it grows without limit
|
|
544
|
-
// (it had reached 38 MB), which slows every read. Drain once at boot,
|
|
545
|
-
// then every 5 minutes.
|
|
546
|
-
const WAL_CHECKPOINT_INTERVAL_MS = 5 * 60_000;
|
|
547
|
-
try {
|
|
548
|
-
store.checkpoint();
|
|
549
|
-
} catch {
|
|
550
|
-
// Boot checkpoint can lose a race with readers — the interval retries.
|
|
551
|
-
}
|
|
552
|
-
const checkpointTimer = setInterval(() => {
|
|
553
|
-
try {
|
|
554
|
-
store.checkpoint();
|
|
555
|
-
} catch {
|
|
556
|
-
// Checkpoint contention — the next tick retries.
|
|
557
|
-
}
|
|
558
|
-
}, WAL_CHECKPOINT_INTERVAL_MS);
|
|
559
|
-
checkpointTimer.unref();
|
|
560
|
-
```
|
|
561
|
-
|
|
562
|
-
Then, in the existing `shutdown` function inside the `if (opts.scheduler !== false)` block (currently `const shutdown = () => { scheduler.stop(); store.close(); process.exit(0); };`), add a `clearInterval` call so it becomes:
|
|
563
|
-
|
|
564
|
-
```typescript
|
|
565
|
-
const shutdown = () => {
|
|
566
|
-
clearInterval(checkpointTimer);
|
|
567
|
-
scheduler.stop();
|
|
568
|
-
store.close();
|
|
569
|
-
process.exit(0);
|
|
570
|
-
};
|
|
571
|
-
```
|
|
572
|
-
|
|
573
|
-
Note: `store` is the concrete `SqliteSessionStore` from `buildStack()`, so `store.checkpoint()` is in scope. `checkpointTimer` is declared in the `.action` closure, so `shutdown` (also in that closure) can reference it.
|
|
574
|
-
|
|
575
|
-
- [ ] **Step 6: Verify typecheck and the full suite**
|
|
576
|
-
|
|
577
|
-
Run: `npm run typecheck && npm test`
|
|
578
|
-
Expected: PASS — typecheck clean, whole suite green.
|
|
579
|
-
|
|
580
|
-
- [ ] **Step 7: Commit**
|
|
581
|
-
|
|
582
|
-
```bash
|
|
583
|
-
git add src/core/storage/sqlite-session-store.ts src/cli/nlm.ts tests/integration/wal-checkpoint.test.ts
|
|
584
|
-
git commit -m "fix: bound the SQLite WAL with periodic checkpoint management"
|
|
585
|
-
```
|
|
586
|
-
|
|
587
|
-
---
|
|
588
|
-
|
|
589
|
-
## Task 4: Rebuild `dist/` and update the CHANGELOG
|
|
590
|
-
|
|
591
|
-
**Files:**
|
|
592
|
-
- Modify: `dist/` (regenerated)
|
|
593
|
-
- Modify: `logs/CHANGELOG/CHANGELOG.md`
|
|
594
|
-
|
|
595
|
-
- [ ] **Step 1: Rebuild `dist/`**
|
|
596
|
-
|
|
597
|
-
Run: `npm run build`
|
|
598
|
-
Expected: `build:server` and `build:ui` both succeed. If the build fails, STOP and report the error.
|
|
599
|
-
|
|
600
|
-
- [ ] **Step 2: Append the CHANGELOG entry**
|
|
601
|
-
|
|
602
|
-
Insert this as the newest (first) dated entry in `logs/CHANGELOG/CHANGELOG.md`, immediately below the title/intro block:
|
|
603
|
-
|
|
604
|
-
```markdown
|
|
605
|
-
## 2026-05-20 — Fix: recall daemon wedge (corpus-load + WAL bloat)
|
|
606
|
-
|
|
607
|
-
`/api/recall` intermittently wedged for 10-25s, starving the whole HTTP server (a health check measured 8.2s during recall load).
|
|
608
|
-
|
|
609
|
-
**Root cause** — `RecallService.search()` called `SqliteSessionStore.list()` on every request, which `SELECT`ed the `body` column: 99 MB of session markdown across 2,097 rows, loaded synchronously on the Node event loop (239ms with `body` vs 35ms without). better-sqlite3 is synchronous, so concurrent recalls serialized into multi-second head-of-line blocking. A `sample` confirmed ~50% of a wedge window in one synchronous query, 85% of that reading `body` overflow pages. The recall path never uses `body`.
|
|
610
|
-
|
|
611
|
-
**Changes**
|
|
612
|
-
- `SessionStore.getByIds(ids)` — batched session fetch that omits the `body` column.
|
|
613
|
-
- `RecallService.search()` no longer calls `list()`. The FTS5 / sqlite-vec legs already return ranked IDs; recall now resolves only those (~15) sessions via `getByIds` and applies the entity/kind filter post-fetch. Per-query cost is O(hits), not O(corpus).
|
|
614
|
-
- `SqliteSessionStore.checkpoint()` + a 5-minute (and boot) `wal_checkpoint(TRUNCATE)` in `nlm start` — the WAL had grown to 38 MB with no checkpoint management and never drained.
|
|
615
|
-
|
|
616
|
-
**State:** v0.3.0. Recall is O(hits); the WAL stays bounded.
|
|
617
|
-
```
|
|
618
|
-
|
|
619
|
-
If `CHANGELOG.md` now exceeds 10 `## ` dated headings, move the oldest beyond 10 into `logs/CHANGELOG/CHANGELOG-2026.md` (prepend) and ensure the `_Older entries archived in CHANGELOG-2026.md_` pointer line is present at the bottom of `CHANGELOG.md`.
|
|
620
|
-
|
|
621
|
-
- [ ] **Step 3: Final verification**
|
|
622
|
-
|
|
623
|
-
Run: `npm test && npm run typecheck`
|
|
624
|
-
Expected: PASS — full suite green (~290 tests), typecheck clean.
|
|
625
|
-
|
|
626
|
-
- [ ] **Step 4: Commit**
|
|
627
|
-
|
|
628
|
-
```bash
|
|
629
|
-
git add dist logs/CHANGELOG/CHANGELOG.md
|
|
630
|
-
git commit -m "build: rebuild dist for the recall wedge fix + CHANGELOG"
|
|
631
|
-
```
|
|
632
|
-
|
|
633
|
-
If Step 2 created/modified `CHANGELOG-2026.md`, `git add logs/CHANGELOG/CHANGELOG-2026.md` before committing.
|
|
634
|
-
|
|
635
|
-
---
|
|
636
|
-
|
|
637
|
-
## Self-Review
|
|
638
|
-
|
|
639
|
-
**Spec coverage:**
|
|
640
|
-
- Recall stops loading the whole corpus / `body` → Task 1 (`getByIds` omits `body`) + Task 2 (`search` uses `getByIds`, not `list`). ✓
|
|
641
|
-
- Entity/kind filter moves post-fetch → Task 2 Step 3(b), `applyFilter(hitSessions, filterArgs)`. ✓
|
|
642
|
-
- WAL checkpoint management → Task 3 (`checkpoint()` + daemon interval + boot drain). ✓
|
|
643
|
-
- One-time drain of the current 38 MB WAL → Task 3 Step 5, the boot `store.checkpoint()` runs when the rebuilt daemon next starts. ✓
|
|
644
|
-
- Failing test reproducing the root cause → Task 2 Step 1 (`listCalls`/`getByIdsCalls` assertion). ✓
|
|
645
|
-
- `dist/` rebuilt, CHANGELOG → Task 4. ✓
|
|
646
|
-
- Golden + `recall-sqlite` integration stay green → verified in Task 2 Step 5. ✓
|
|
647
|
-
|
|
648
|
-
**Placeholder scan:** No TBDs; every code step has complete code; every command has an expected result.
|
|
649
|
-
|
|
650
|
-
**Type consistency:** `getByIds(ids: ReadonlyArray<string>): Promise<ReadonlyArray<Session>>` — identical signature in the port (Task 1 Step 3), `SqliteSessionStore` (Step 4), and the `InMemoryStore` fake (Step 5, Task 2 Step 1). `KeywordNeighbor`/`SemanticNeighbor` imported in `recall-service.ts` (Task 2 Step 3a) and used for `kwNeighbors`/`semNeighbors`. `uniqueIds` defined in Task 2 Step 3d, called in the new `search`. `checkpoint(): void` — defined in Task 3 Step 3, called in Task 3 Step 5. `KeywordHit`/`SemanticHit` unchanged and still consumed by `mergeHybrid`/`toKeywordHit`/`toSemanticHit`. ✓
|
|
651
|
-
|
|
652
|
-
---
|
|
653
|
-
|
|
654
|
-
## Execution Handoff
|
|
655
|
-
|
|
656
|
-
**Plan complete and saved to `docs/plans/2026-05-20-recall-daemon-wedge-fix.md`. Two execution options:**
|
|
657
|
-
|
|
658
|
-
**1. Subagent-Driven (recommended)** — I dispatch a fresh subagent per task, review between tasks, fast iteration.
|
|
659
|
-
|
|
660
|
-
**2. Inline Execution** — execute tasks in this session using executing-plans, batch execution with checkpoints.
|
|
661
|
-
|
|
662
|
-
**Which approach?**
|