sessionmem 1.4.0 → 1.5.0

package/docs/architecture.md

@@ -30,7 +30,7 @@
 
 ### Core engine
 
-The core engine captures session events, summarizes them into stored memories at session end, ranks those memories, and selects which to inject. It uses the adapter layer for all communication with specific tools. Retrieval combines semantic similarity, recency, and importance into a relevance score.
+The core engine captures session events, summarizes them into stored memories at session end, ranks those memories, and selects which to inject. It uses the adapter layer for all communication with specific tools. Retrieval combines BM25 lexical relevance, entity overlap, recency, and importance into a relevance score.
 
 ### Adapters
 
@@ -42,11 +42,35 @@ The CLI (`sessionmem`) registers the server with a host and runs the MCP server.
 
 ### SQLite storage
 
-SQLite stores memories, embeddings for vector search, raw session events, and feedback history in one local file. Database access uses `better-sqlite3`, with forward-only [migrations](migration.md). No hosted database is required.
+SQLite stores memories, embeddings for vector search, raw session events, extracted entities and relations, and feedback history in one local file. Database access uses `better-sqlite3`, with forward-only [migrations](migration.md). No hosted database is required.
+
+## Entity and relation extraction
+
+Capture, batch writes, imports, team pulls, and content rewrites derive a graph locally. Session-end summarization (including scan/watch) reads the same event text as the existing summarizer before summary line/token caps. Extraction always applies the built-in redaction rules, including when indexing legacy text. It never invokes a model or an external API.
+
+The rules recognize labeled speakers, explicit `person`, `project`, `tool`, `service`, and `file` names, common tools/services, and file paths with supported extensions. Short affirmative clauses such as `Alice works on project Atlas`, `Alice uses SQLite`, `Alice decided to use SQLite`, `Alice prefers Python`, and `Project Atlas is blocked by GitHub` yield `works_on`, `uses`, `decided`, `prefers`, and `blocked_by` edges. Decision clauses become decision entities. `Ran:` tool events yield `uses` for recognized executables; git commits also yield decisions, matching the existing local summarizer. Unlabeled first-person/imperative clauses use `user` (or `team` for `we`); labeled speakers resolve `I`. Questions, negation, hypothetical clauses, and unresolved third-person pronouns are skipped. These conservative rules are not general natural-language understanding and can miss or misclassify unfamiliar phrasing.
+
+Names use NFKC, lowercase, stripped quotes/possessives, and collapsed whitespace. This also folds file-path case. `memory_entities` stores unique `(memory_id, kind, name)` rows; `memory_relations` references both endpoints in the same memory. Updates replace derived rows atomically, and foreign keys cascade memory deletion and summary ID changes. Migration 014 backfills existing memories, using full events for summaries when available. Expired memories are excluded from graph reads and ranking. Export/import request and response formats are unchanged; graphs are rebuilt from imported content, so event-only details absent from an exported summary do not round-trip.
+
+`sessionmem entities --project <id>` lists entities and relations with source memory IDs; `--json` provides structured output. Omitting `--project` selects the current project.
+
+## Contradiction and supersession
+
+Migration 015 adds `memory_claims` and `memory_supersessions`. Claims use migration 014's canonical entity and typed-relation extractor, with a small grammar for `does not`, `no longer`, `stopped using`, `switched to/from`, and `replaced ... with`. A comparison requires the same entity and relation. Opposite polarity on the same value is a contradiction. Different positive values supersede one another when their lexical purpose overlaps (at least two words and Jaccard overlap 0.6, or an exact purpose), their tool category matches (editor, language, database, package manager, test runner), or a replacement explicitly names the old value. `now works on` also replaces a prior project. Different purposes, entities, or relation types remain independent. Questions, hypothetical statements, reported quotations, and unresolved pronouns are skipped. Rules are conservative heuristics and do not resolve arbitrary language or coreference.
+
+Each link records the old memory in `supersedes`, the replacement in `superseded_by`, both claim keys, the reason, the replacement assertion's `superseded_at`, and the wall-clock `detected_at`. Memory bodies, IDs, and importance are preserved. A derived `superseded_memories` view marks a memory fully superseded only when every extracted claim has a replacement. Independent claims in a mixed summary keep it searchable; its content can still contain stale claims, so inspect partial supersessions in history when auditing such summaries. No old memory is deleted by supersession; explicit deletion and retention keep their existing behavior.
+
+Writes reconcile only affected entity/relation groups within the same project, atomically with memory and graph updates. Assertions are ordered by their stored timestamp, then memory insertion rowid for equal timestamps. This gives pinned-clock captures a deterministic arrival order. The earliest later contradictory assertion supplies each link, permitting chains and later reaffirmations without cycles. Identical replay preserves links and detection timestamps; imports, content rewrites, summary ID changes, and deletion reconnect affected chains. Migration 015 backfills existing memories once, using full session events for summaries. Scan preserves valid original transcript timestamps, so filename order does not determine freshness. Without original timestamps, stored capture order is the fallback. An unrelated appended event does not change an earlier assertion's time.
+
+Recall filters fully superseded memories before the FTS candidate limit and fallback scan, giving current facts priority. The optional `includeHistory: true` request field, exposed as `sessionmem search --include-history`, restores normal relevance ranking across current and superseded memories. Existing requests and response shapes remain compatible. `getMemory`, list and export still expose earlier memories. `sessionmem history [--project <id>] [--json]` lists links with old/new values, timestamps, reasons, source IDs, and partial/full status. Export formats remain unchanged: importing rebuilds links from exported bodies and timestamps; event-only claims absent from a summary cannot round-trip.
 
 ## Retrieval flow
 
-When an agent or `sessionmem search` submits a query, the core engine embeds it and runs a vector search over the `memories` table. It ranks candidates by semantic similarity, recency, and importance, then returns the memories with their scores.
+When an agent or `sessionmem search` submits a query, the core engine searches the SQLite FTS5 index over stored memory content, including summaries. It removes common English question words and matches any remaining term. SQLite's built-in Porter tokenizer matches English inflections such as “hike” and “hiking”. BM25 accounts for term rarity, frequency, and memory length; at most 50 lexical candidates are returned, ordered by BM25 and then ID.
+
+The ranker divides each positive BM25 score by the best score for that query. If the query contains known entity names in this project, relevance is `0.90 * base relevance + 0.10 * entity overlap`. Overlap is the fraction of distinct matched query names linked to the candidate memory; names are matched as literal token sequences (up to eight tokens), including lowercase questions. With no known query entities, relevance is unchanged. This reranks the existing candidates; it does not expand the FTS candidate pool or traverse relations. The outer relevance, recency, and importance weights stay 0.60, 0.25, and 0.15. Sparse lexical matches retain the recency/importance fallback union. If there are no lexical matches, base relevance uses the previous hash-vector ranking. Project scope and expiry checks apply to both paths. Automatic migrations preserve memory bodies and IDs.
+
+Existing public request fields and all response fields are unchanged; `includeHistory` is an optional addition. The `semantic` field still reports hash cosine similarity. The legacy `score.raw.semantic` and `score.weighted.semantic` fields carry the relevance component used for ranking, including entity overlap when query entities match. Scores are relative to the current query's candidates, not calibrated probabilities. Retrieval needs no API keys, downloaded models, or new dependencies.
 
 ## Injection flow
 

package/docs/benchmark.md

@@ -1,50 +1,122 @@
-# Benchmark
+# Conversational retrieval benchmark
 
-`npm run benchmark` (`scripts/benchmark.mjs`) generates this report using the production functions `retrieveMemories`, `formatStartupInjection`, and `countTokens`. It measures token reduction against full session history and retrieval relevance using curated, synthetic fixtures. It runs offline with a hash-based embedding function, `deterministicEmbed`, and a fixed clock, so repeated runs produce identical output.
+`npm run benchmark` runs the committed LoCoMo corpus through the public
+`storeMemory` capture and `retrieveMemories` recall APIs. See
+[benchmarks/README.md](../benchmarks/README.md) for the corpus license, selection
+rules, metric definitions, and detailed reproduction instructions.
 
-Startup injection used **~85.6% fewer tokens** than full session history. Retrieval had a **100.0% hit-rate** across 10 curated queries.
+## Entity overlap comparison
 
-## Token reduction
+The baseline was run first on current main at `81f987c`. A fixed 10% entity
+contribution inside the existing relevance component was then evaluated without
+a weight sweep. Both runs used Node v24.19.0, Linux x64, sessionmem 1.4.0, the
+same installed lockfile dependencies, top-5, fixed clocks, and the unchanged
+LoCoMo corpus and scoring rules. Each run scored 1,527 questions after capturing
+5,882 turns; both recorded zero network attempts.
 
-Token counts use the production `countTokens` function (js-tiktoken `o200k_base`). The table compares a single startup injection and average per-query context with the same full-history baseline. The result above uses startup injection.
+| Macro retrieval metric @5 | Main (BM25) | BM25 + entity overlap |
+| --- | ---: | ---: |
+| Hit rate | 61.95% | 63.92% |
+| Precision | 13.45% | 14.05% |
+| Recall | 55.96% | 57.45% |
+| All-evidence rate | 51.15% | 52.26% |
 
-| Framing | Baseline tokens (full history) | Injected tokens | Reduction |
-|---------|-------------------------------|-----------------|-----------|
-| (a) Top-3 startup injection vs full history | 1587 | 228 | 85.6% |
-| (b) Per-query injected context (avg) vs full history | 1587 | 232 | 85.4% |
+| Recall latency | Main | Entity overlap |
+| --- | ---: | ---: |
+| Mean | 8.57 ms | 8.05 ms |
+| Median | 7.27 ms | 7.58 ms |
+| p95 | 21.04 ms | 14.94 ms |
+| Maximum | 35.81 ms | 28.38 ms |
 
-Row (a) measures one startup injection of the top-ranked memories. Row (b) averages the injected context across the curated queries to model per-turn retrieval.
+Entity overlap gained hits on 34 questions and lost hits on four, a net gain
+of 30. Precision and recall increased on 48 questions and decreased on five.
+The signal measurably improved retrieval on this corpus, so it is retained.
+Timing is one call per question without warmup and varies with load; the lower
+mean/p95 is not evidence that the added lookup makes retrieval faster. This
+does not establish generalization to other corpora or measure extraction accuracy.
+The rule vocabulary and fixed weight were not tuned against these results.
 
-## Retrieval relevance
+[Recorded aggregate results](../benchmarks/entity-overlap-results.json) include
+the corpus checksum, shared configuration, runtime, category metrics, and paired
+question counts. To reproduce, run the commands below on `81f987c` with output
+`benchmarks/results/locomo-main.json`, then on this change with output
+`benchmarks/results/locomo-entities.json`. Full per-question reports remain ignored
+local artifacts.
 
-Each query runs through `retrieveMemories` with a limit of 3 results. The returned ids are compared with the expected memory ids. A query counts as a hit if at least one expected memory is returned.
+## Earlier BM25 comparison
 
-- **Hit-rate:** 100.0% (10 queries)
-- **Overall recall:** 100.0%
-- **Overall precision:** 33.3%
+The baseline was run first on main at `c06857d`, before changing retrieval.
+Both runs used sessionmem 1.4.0, Node v24.19.0, Linux x64, top-5,
+and the same lockfile dependencies and fixed capture/retrieval clock.
+They captured 5,882 turns across 272 sessions in 10 conversations and scored
+1,527 questions. The same 446 adversarial, four missing-evidence, and nine
+invalid-evidence questions were excluded by fixed annotation rules.
 
-| Query (memory content) | Expected memory | Recall | Precision |
-|------------------------|-----------------|--------|-----------|
-| We chose JWT access tokens with a 15-minute expiry and ro... | mem-auth-decision | 100.0% | 33.3% |
-| The persistence layer uses better-sqlite3 with WAL mode; ... | mem-db-decision | 100.0% | 33.3% |
-| Embeddings are computed offline with a deterministic sha2... | mem-embed-fact | 100.0% | 33.3% |
-| Retrieval ranks candidates by a weighted blend of semanti... | mem-retrieval-fact | 100.0% | 33.3% |
-| Startup injection formats the top-ranked memories under a... | mem-injection-fact | 100.0% | 33.3% |
-| Never edit an already-shipped migration file in place; ad... | mem-migration-warning | 100.0% | 33.3% |
-| Redaction must run on every memory-write path before stor... | mem-secret-warning | 100.0% | 33.3% |
-| The CLI exposes remember, retention prune, redact-scan, a... | mem-cli-fact | 100.0% | 33.3% |
-| Tests use vitest; integration specs seed an in-memory dat... | mem-testing-fact | 100.0% | 33.3% |
-| The MCP server runs over stdio and exposes memory retriev... | mem-mcp-decision | 100.0% | 33.3% |
+| Macro retrieval metric @5 | Main (hash ranking) | BM25 + stop words + stemming |
+| --- | ---: | ---: |
+| Hit rate | 1.44% | 61.95% |
+| Precision | 0.29% | 13.45% |
+| Recall | 0.88% | 55.96% |
+| All-evidence rate | 0.65% | 51.15% |
 
-### Relevance interpretation
+| Recall latency | Main | Selected change |
+| --- | ---: | ---: |
+| Mean | 16.33 ms | 14.93 ms |
+| Median | 15.22 ms | 13.52 ms |
+| p95 | 28.80 ms | 28.81 ms |
+| Maximum | 133.49 ms | 56.73 ms |
 
-`deterministicEmbed` uses a hash-based fingerprint. Identical text has a cosine similarity of 1.0; wording changes produce noisy scores. It supports exact or near-exact recall rather than semantic matching of paraphrases. These queries contain the target memory's content, so the hit-rate measures whether ranking retrieves that memory in that case. Matching paraphrases requires a semantic embedding model. Replacing the embedding model leaves the retrieval and scoring code unchanged, so the benchmark can still exercise those paths.
+Latency includes the complete public service response, including startup-injection
+formatting, and excludes capture, process startup, and corpus loading. Each query
+is timed once without warmup. Timing depends on hardware and load.
 
-## Reproducing
+| Question category | Questions | Hit rate | Precision | Recall |
+| --- | ---: | ---: | ---: | ---: |
+| Multi-hop | 278 | 46.76% | 13.60% | 25.06% |
+| Temporal | 320 | 69.38% | 14.38% | 66.22% |
+| Open-domain | 89 | 38.20% | 8.09% | 28.25% |
+| Single-hop | 840 | 66.67% | 13.62% | 65.22% |
+
+## Candidate comparisons
+
+Each candidate was run with the existing offline evaluator on the same questions.
+Only additions that increased hit rate, precision, and recall were retained.
+
+| Candidate @5 | Hit rate | Precision | Recall |
+| --- | ---: | ---: | ---: |
+| BM25 with OR terms | 52.78% | 11.15% | 47.69% |
+| + English stop-word filtering | 57.69% | 12.39% | 52.62% |
+| + Porter stemming (selected) | 61.95% | 13.45% | 55.96% |
+| 90% BM25 / 10% hash cosine (discarded) | 61.76% | 13.43% | 55.78% |
+
+BM25 preserves lexical relevance instead of replacing it with hash-vector noise.
+Common question words no longer require a literal match, and SQLite's built-in
+Porter tokenizer matches English inflections. The existing recency, importance,
+expiry, project scope, and no-match vector fallback remain in place. There are
+no new dependencies, API keys, or model downloads. Migration 013 rebuilds the
+existing FTS index; memory content and IDs are preserved.
+
+These comparisons use the same corpus used to select the ranker; they are not
+a held-out generalization estimate. BM25 still needs lexical overlap, and a hit
+can leave evidence missing for multi-hop questions. These are annotated-evidence
+retrieval scores, not answer-generation accuracy or summarization quality.
+
+## Reproduce
+
+After installing the lockfile dependencies, run on main to record the baseline,
+then repeat on the change:
 
 ```bash
-npm run build      # benchmark imports built functions from dist/
-npm run benchmark  # regenerates this file deterministically
+npm run build
+npm run benchmark -- --output benchmarks/results/locomo.json
 ```
 
-_Fixtures are synthetic; the report contains only aggregate metrics and synthetic example content. Generated by `scripts/benchmark.mjs`._
+The corpus is pinned to upstream commit
+`3eb6f2c585f5e1699204e3c3bdf7adc5c28cb376`, SHA-256
+`79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4`.
+The runner verifies this checksum and blocks network calls. No API keys are
+required. A second full run with an empty environment produced identical
+retrieved IDs and quality metrics for all 1,527 questions. JSON output includes
+every scored question, answer, evidence list, returned IDs, and exclusion reason
+so failures can be inspected. The report above is a recorded result; the command
+prints current results and writes JSON only when requested.

package/docs/migration.md

@@ -19,6 +19,8 @@ src/core/schema/migrations/
 
 At startup, the runner creates the `_migrations` tracking table if needed and sorts the `.sql` files by name. It applies each migration in its own transaction, skipping any already recorded in `_migrations`. Starting sessionmem again does not reapply completed migrations. There are no down-migrations.
 
+Migration `015_memory_supersessions.sql` adds derived fact claims, timestamped replacement links, and a view identifying fully superseded memories. It backfills existing records locally in the migration transaction, using full session events for summaries where available. Stored memory content and IDs are preserved. Recall then defaults to current facts; use `includeHistory: true` or `sessionmem search --include-history` to retrieve superseded facts, and `sessionmem history` to inspect their chains.
+
 ### Build-time copy: `copy-migrations`
 
 The TypeScript compiler does not copy `.sql` files. The build step `copy-migrations` runs `scripts/copy-migrations.mjs` to copy them from `src/core/schema/migrations/` to `dist/core/schema/migrations/`. This includes the migrations in the published package alongside the compiled code.

package/docs/troubleshooting.md

@@ -2,6 +2,41 @@
 
 This guide covers install failures, adapter (MCP host) issues, and `better-sqlite3` native-build failures. Configuration lives in `~/.sessionmem/config.json`.
 
+## Run offline diagnostics
+
+```bash
+sessionmem doctor
+sessionmem doctor --adapter claude-code  # select one host explicitly
+```
+
+Doctor prints PASS or FAIL and a fix hint for each problem. Any failed check
+returns a nonzero exit status. It checks:
+
+- **Memory store:** the selected database exists, opens, and passes SQLite's quick integrity check. It does not create a missing database.
+- **Schema:** all bundled migrations have been applied. It reports pending or unrecognized migrations without changing the schema.
+- **Hooks, per host:** Claude Code's SessionStart, SessionEnd, and PostToolUse commands, types, matchers, and enabled state match the current version's definitions. Other hosts report that lifecycle hooks are not required.
+- **Adapter wiring, per host:** the installed MCP entry matches the installer, its local command starts, the server version matches, and the expected tools and stats response work. This tests the configured server process without starting the coding agent itself.
+- **Recall:** the real memory service writes and retrieves a canary in a separate temporary project. The transaction is rolled back on success or failure, including index and access-count changes.
+
+For Codex, the wiring check supports the installer-generated TOML command and
+args. Custom Codex launch settings require manual verification.
+
+All checks work offline, including when cloud summarization is enabled. Doctor
+does not summarize sessions or sync shared memories. It honors `SESSIONMEM_DB_PATH`
+and the same host overrides as install. Keep those environment settings when
+running a suggested repair.
+
+Safe repair commands printed by doctor include:
+
+```bash
+sessionmem ping                         # initialize a missing store / apply pending migrations
+sessionmem install --adapter claude-code # repair MCP registration and hook definitions
+```
+
+Re-run doctor after repair. A corrupt store, a database from a newer version,
+globally disabled hooks, or missing filesystem permissions needs the indicated
+manual repair first; doctor does not delete data or downgrade a schema.
+
 ## Install failures
 
 ### `config.json` not written
@@ -10,7 +45,17 @@ This guide covers install failures, adapter (MCP host) issues, and `better-sqlit
 
 ### Adapter not detected
 
-`install` registers sessionmem with the MCP host in the current directory. If install reports that it could not find a host, you are likely running it outside a project the host recognizes. `cd` into your project root (where the host's config lives) and re-run `sessionmem install`. Use `sessionmem install --adapter <name>` from a plain terminal (for example `--adapter cline` or `--adapter windsurf`, which cannot be reliably auto-detected). Run `sessionmem ping` afterward to check local database access; it does not probe the host's MCP connection.
+`install` discovers installed hosts from executables on the machine, including
+from a plain terminal. Automatic discovery currently supports Claude Code on
+`PATH`, `~/.local/bin/claude`, or `~/.claude/local/claude` (including Windows
+executable extensions). It does not rely on an active agent session or treat a
+leftover config directory as proof that the host is installed.
+
+If no supported host is found, install exits nonzero before creating files. Put
+Claude Code on `PATH` or select a host with `sessionmem install --adapter <name>`.
+Use the flag for other adapters, such as `--adapter cline` or `--adapter windsurf`,
+and for automation. Run `sessionmem doctor --adapter <name>` afterward. The
+`SESSIONMEM_ADAPTER` environment variable is also supported; the flag wins.
 
 ### Command not found
 
@@ -20,11 +65,14 @@ If `sessionmem` is not on your `PATH` after `npm install -g sessionmem`, your np
 
 Each MCP host uses its own adapter. If the server starts but memories are missing, check the host's configuration:
 
-- **Claude Code**: if memories are not injected at session start, confirm `sessionmem install` was run inside the project and that the MCP server entry exists in the host config. Use `sessionmem ping` to verify local database access, then check MCP connectivity in the host.
-- **Cursor**: Cursor must be pointed at the registered MCP server. Re-run `install` from the project root and restart Cursor so it reloads MCP servers.
+- **Claude Code**: if memories are not injected at session start, run `sessionmem doctor --adapter claude-code`. Re-run `sessionmem install --adapter claude-code` to refresh the registration in `~/.claude.json` and all three hooks in `~/.claude/settings.json`, then restart the agent.
+- **Cursor**: run `sessionmem doctor --adapter cursor`, repair registration with `sessionmem install --adapter cursor`, and restart Cursor so it reloads MCP servers.
 - **Generic MCP host**: for any other MCP-compatible host, ensure the host is configured to launch `sessionmem run` as the MCP server command. If the host starts but no memory appears, check that the working directory matches the project whose memories you expect.
 
-If commands hang after the host loads, run `sessionmem ping` to check local database access, then check the host's MCP connection.
+If commands hang after the host loads, run `sessionmem doctor --adapter <name>`
+to check the configured MCP command as well as local database access. A missing
+`sessionmem` executable, stale server version, or failed MCP response fails the
+adapter wiring check. Fix `PATH` for the environment that starts your host too.
 
 ## "0 sessions" / no session data recorded
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sessionmem",
-  "version": "1.4.0",
+  "version": "1.5.0",
   "private": false,
   "type": "module",
   "description": "Local-first MCP memory layer for coding agents across Claude Code, Codex, Cursor, Cline, Windsurf, and other MCP-compatible hosts.",