sessionmem 1.3.0 → 1.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/sessionmem-logo.svg +57 -0
- package/CHANGELOG.md +128 -0
- package/CONTRIBUTING.md +130 -0
- package/README.md +292 -108
- package/SECURITY.md +33 -0
- package/dist/adapters/diagnostics.js +109 -0
- package/dist/adapters/generic.js +6 -2
- package/dist/adapters/global/antigravity.js +4 -1
- package/dist/adapters/global/claudeCode.js +19 -4
- package/dist/adapters/global/codex.js +7 -6
- package/dist/adapters/global/qcoder.js +4 -1
- package/dist/adapters/ide/cline.js +3 -0
- package/dist/adapters/ide/cursor.js +3 -0
- package/dist/adapters/ide/installer.js +17 -5
- package/dist/adapters/ide/windsurf.js +3 -0
- package/dist/adapters/installedHosts.js +51 -0
- package/dist/cli/commands/dedupe.js +1 -1
- package/dist/cli/commands/doctor.js +146 -0
- package/dist/cli/commands/entities.js +27 -0
- package/dist/cli/commands/history.js +23 -0
- package/dist/cli/commands/import.js +15 -44
- package/dist/cli/commands/ingestHook.js +4 -78
- package/dist/cli/commands/install.js +37 -59
- package/dist/cli/commands/list.js +3 -0
- package/dist/cli/commands/run.js +10 -12
- package/dist/cli/commands/scan.js +16 -0
- package/dist/cli/commands/search.js +9 -4
- package/dist/cli/commands/team.js +7 -3
- package/dist/cli/commands/ui.js +67 -0
- package/dist/cli/commands/uninstall.js +2 -1
- package/dist/cli/commands/watch.js +48 -0
- package/dist/cli/context.js +8 -6
- package/dist/cli/index.js +48 -7
- package/dist/cli/projectId.js +1 -2
- package/dist/cli/transcriptSweep.js +182 -0
- package/dist/core/api/contracts.js +1 -0
- package/dist/core/api/memoryCoreService.js +8 -0
- package/dist/core/api/sessionLifecycleService.js +13 -1
- package/dist/core/capture/toolEvent.js +82 -0
- package/dist/core/config/policyConfig.js +5 -1
- package/dist/core/extract/factClaims.js +97 -0
- package/dist/core/extract/knowledge.js +127 -0
- package/dist/core/injection/tokenBudget.js +10 -2
- package/dist/core/retrieve/retrieveMemories.js +20 -4
- package/dist/core/schema/migrations/012_transcript_checkpoints.sql +4 -0
- package/dist/core/schema/migrations/013_fts_porter.sql +32 -0
- package/dist/core/schema/migrations/014_memory_entities.sql +19 -0
- package/dist/core/schema/migrations/015_memory_supersessions.sql +41 -0
- package/dist/core/schema/runMigrations.js +55 -5
- package/dist/core/storage/db.js +31 -9
- package/dist/core/storage/knowledgeRepo.js +67 -0
- package/dist/core/storage/memoryRepo.js +45 -12
- package/dist/core/storage/memorySearchRepo.js +30 -20
- package/dist/core/storage/supersessionRepo.js +101 -0
- package/dist/core/summarize/cloudSummarizer.js +3 -8
- package/dist/core/summarize/summaryShape.js +1 -1
- package/dist/ui/assets.js +181 -0
- package/dist/ui/server.js +147 -0
- package/docs/REGISTRY_LISTING.md +88 -0
- package/docs/architecture.md +77 -0
- package/docs/benchmark.md +122 -0
- package/docs/cloud-summarization.md +57 -0
- package/docs/migration.md +44 -0
- package/docs/privacy-and-retention.md +139 -0
- package/docs/team-mode.md +117 -0
- package/docs/troubleshooting.md +126 -0
- package/docs/typescript-upgrade.md +48 -0
- package/package.json +25 -13
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
# Registry listing — draft for review
|
|
2
|
+
|
|
3
|
+
**Status: DRAFT FOR REVIEW. This is proposed listing text only; it has not been
|
|
4
|
+
submitted to any MCP or tool registry.**
|
|
5
|
+
|
|
6
|
+
## Name
|
|
7
|
+
|
|
8
|
+
sessionmem
|
|
9
|
+
|
|
10
|
+
MCP identifier: `io.github.catfish-1234/sessionmem`
|
|
11
|
+
|
|
12
|
+
## One-line description
|
|
13
|
+
|
|
14
|
+
An offline MCP memory server that keeps coding-agent decisions, warnings, and
|
|
15
|
+
project context in local SQLite across sessions.
|
|
16
|
+
|
|
17
|
+
## Features
|
|
18
|
+
|
|
19
|
+
- Persistent project memory through MCP tools for storing, retrieving, listing,
|
|
20
|
+
inspecting, and forgetting memories.
|
|
21
|
+
- Claude Code hooks capture supported tool events, summarize sessions locally,
|
|
22
|
+
and load relevant prior context automatically. Other MCP hosts use memory
|
|
23
|
+
tools and installed guidance; automatic lifecycle capture depends on the host.
|
|
24
|
+
- Adapters for Claude Code, Codex, Cursor, Cline, Windsurf, Antigravity, QCoder,
|
|
25
|
+
and generic MCP hosts.
|
|
26
|
+
- Offline BM25 retrieval with English stemming, entity overlap, recency, and
|
|
27
|
+
importance; startup context is trimmed to a token budget.
|
|
28
|
+
- Deterministic entity/relation extraction and supersession history. Fully
|
|
29
|
+
replaced memories are excluded from normal recall, with explicit history
|
|
30
|
+
lookup available. These language rules have limited coverage.
|
|
31
|
+
- Offline `scan`/`watch` commands for existing and new Claude Code transcripts.
|
|
32
|
+
- Local SQLite storage, default secret-pattern redaction, configurable retention,
|
|
33
|
+
JSON export/import, and offline installation diagnostics with `sessionmem doctor`.
|
|
34
|
+
- A committed LoCoMo corpus and reproducible offline evidence-retrieval runner;
|
|
35
|
+
see the [comparison and evaluation limits](../COMPARISON.md#evaluation-and-measured-results).
|
|
36
|
+
|
|
37
|
+
## Install and connect
|
|
38
|
+
|
|
39
|
+
Requires Node.js `^20.19.0 || >=22.13.0`.
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
npm install -g sessionmem
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
From the project directory, register it with a configured host:
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
sessionmem install
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Automatic discovery currently detects Claude Code. Select other adapters
|
|
52
|
+
explicitly, for example:
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
sessionmem install --adapter codex
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Restart the host after registration. The host launches the stdio server with
|
|
59
|
+
`sessionmem run`; this is the server command for registries that request one.
|
|
60
|
+
See the [quickstart](../README.md#quickstart) for host-specific setup.
|
|
61
|
+
|
|
62
|
+
## Offline guarantee
|
|
63
|
+
|
|
64
|
+
**The local memory pipeline requires no API keys and makes no network calls.**
|
|
65
|
+
Capture, local summarization, entity/relation extraction, supersession, and
|
|
66
|
+
retrieval run on the user's machine without downloading models. Memories are
|
|
67
|
+
stored by default in `~/.sessionmem/memories.db`. No hosted memory account or
|
|
68
|
+
service is needed. The coding agent's own model connection is separate.
|
|
69
|
+
|
|
70
|
+
Package installation may require network access or an offline package cache.
|
|
71
|
+
Explicitly enabling [cloud summarization](cloud-summarization.md) with both its
|
|
72
|
+
configuration flag and an Anthropic API key sends a redacted transcript to
|
|
73
|
+
Anthropic. Optional [team sharing](team-mode.md) uses a user-selected shared
|
|
74
|
+
folder. These opt-in paths are outside the local-only guarantee. The optional
|
|
75
|
+
read-only viewer uses loopback HTTP with locally bundled assets and no external
|
|
76
|
+
services or CDNs.
|
|
77
|
+
|
|
78
|
+
## Links and license
|
|
79
|
+
|
|
80
|
+
- [Source and setup](https://github.com/catfish-1234/sessionmem)
|
|
81
|
+
- [npm package](https://www.npmjs.com/package/sessionmem)
|
|
82
|
+
- [Comparison](../COMPARISON.md)
|
|
83
|
+
- [Privacy and retention](privacy-and-retention.md)
|
|
84
|
+
- [MIT license](../LICENSE); the bundled LoCoMo dataset has a separate
|
|
85
|
+
[CC BY-NC 4.0 license](../benchmarks/data/LICENSE.txt).
|
|
86
|
+
|
|
87
|
+
Review the wording and destination-specific registry fields before any future
|
|
88
|
+
submission. This draft does not claim registry acceptance or verification.
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Architecture
|
|
2
|
+
|
|
3
|
+
`sessionmem` connects a coding agent's MCP host to a local SQLite memory database. This overview covers its four subsystems and the retrieval and injection flows. It runs locally by default, with settings in `~/.sessionmem/config.json`.
|
|
4
|
+
|
|
5
|
+
## Subsystems
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
┌──────────────────────────────────────────────┐
|
|
9
|
+
│ MCP host │
|
|
10
|
+
│ (Claude Code, Cursor, Codex, Cline, ...) │
|
|
11
|
+
└───────────────────────┬──────────────────────┘
|
|
12
|
+
│ MCP / hooks
|
|
13
|
+
┌───────────▼───────────┐ ┌──────────────┐
|
|
14
|
+
│ adapters │ │ CLI │
|
|
15
|
+
│ (host-specific glue) │ │ (sessionmem) │
|
|
16
|
+
└───────────┬───────────┘ └──────┬───────┘
|
|
17
|
+
│ │
|
|
18
|
+
▼ ▼
|
|
19
|
+
┌──────────────────────────────────────────────┐
|
|
20
|
+
│ core engine │
|
|
21
|
+
│ capture · summarize · retrieval · injection │
|
|
22
|
+
└───────────────────────┬──────────────────────┘
|
|
23
|
+
│
|
|
24
|
+
▼
|
|
25
|
+
┌──────────────────────────────────────────────┐
|
|
26
|
+
│ SQLite storage (local) │
|
|
27
|
+
│ memories · session events · feedback │
|
|
28
|
+
└──────────────────────────────────────────────┘
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
### Core engine
|
|
32
|
+
|
|
33
|
+
The core engine captures session events, summarizes them into stored memories at session end, ranks those memories, and selects which to inject. It uses the adapter layer for all communication with specific tools. Retrieval combines BM25 lexical relevance, entity overlap, recency, and importance into a relevance score.
|
|
34
|
+
|
|
35
|
+
### Adapters
|
|
36
|
+
|
|
37
|
+
Adapters translate between each MCP host's hooks or runtime and the core engine's API. There are adapters for Claude Code, Cursor, Codex, Cline, Windsurf, Antigravity, QCoder, and generic MCP hosts. They connect host lifecycle events to the engine's capture, summarize, and inject calls. A new platform needs an adapter; the core engine stays the same.
|
|
38
|
+
|
|
39
|
+
### CLI
|
|
40
|
+
|
|
41
|
+
The CLI (`sessionmem`) registers the server with a host and runs the MCP server. It also has commands to search, list, show, forget, export, import, prune, and configure memories. These commands call the same core engine as the adapters.
|
|
42
|
+
|
|
43
|
+
### SQLite storage
|
|
44
|
+
|
|
45
|
+
SQLite stores memories, embeddings for vector search, raw session events, extracted entities and relations, and feedback history in one local file. Database access uses `better-sqlite3`, with forward-only [migrations](migration.md). No hosted database is required.
|
|
46
|
+
|
|
47
|
+
## Entity and relation extraction
|
|
48
|
+
|
|
49
|
+
Capture, batch writes, imports, team pulls, and content rewrites derive a graph locally. Session-end summarization (including scan/watch) reads the same event text as the existing summarizer before summary line/token caps. Extraction always applies the built-in redaction rules, including when indexing legacy text. It never invokes a model or an external API.
|
|
50
|
+
|
|
51
|
+
The rules recognize labeled speakers, explicit `person`, `project`, `tool`, `service`, and `file` names, common tools/services, and file paths with supported extensions. Short affirmative clauses such as `Alice works on project Atlas`, `Alice uses SQLite`, `Alice decided to use SQLite`, `Alice prefers Python`, and `Project Atlas is blocked by GitHub` yield `works_on`, `uses`, `decided`, `prefers`, and `blocked_by` edges. Decision clauses become decision entities. `Ran:` tool events yield `uses` for recognized executables; git commits also yield decisions, matching the existing local summarizer. Unlabeled first-person/imperative clauses use `user` (or `team` for `we`); labeled speakers resolve `I`. Questions, negation, hypothetical clauses, and unresolved third-person pronouns are skipped. These conservative rules are not general natural-language understanding and can miss or misclassify unfamiliar phrasing.
|
|
52
|
+
|
|
53
|
+
Names use NFKC, lowercase, stripped quotes/possessives, and collapsed whitespace. This also folds file-path case. `memory_entities` stores unique `(memory_id, kind, name)` rows; `memory_relations` references both endpoints in the same memory. Updates replace derived rows atomically, and foreign keys cascade memory deletion and summary ID changes. Migration 014 backfills existing memories, using full events for summaries when available. Expired memories are excluded from graph reads and ranking. Export/import request and response formats are unchanged; graphs are rebuilt from imported content, so event-only details absent from an exported summary do not round-trip.
|
|
54
|
+
|
|
55
|
+
`sessionmem entities --project <id>` lists entities and relations with source memory IDs; `--json` provides structured output. Omitting `--project` selects the current project.
|
|
56
|
+
|
|
57
|
+
## Contradiction and supersession
|
|
58
|
+
|
|
59
|
+
Migration 015 adds `memory_claims` and `memory_supersessions`. Claims use migration 014's canonical entity and typed-relation extractor, with a small grammar for `does not`, `no longer`, `stopped using`, `switched to/from`, and `replaced ... with`. A comparison requires the same entity and relation. Opposite polarity on the same value is a contradiction. Different positive values supersede one another when their lexical purpose overlaps (at least two words and Jaccard overlap 0.6, or an exact purpose), their tool category matches (editor, language, database, package manager, test runner), or a replacement explicitly names the old value. `now works on` also replaces a prior project. Different purposes, entities, or relation types remain independent. Questions, hypothetical statements, reported quotations, and unresolved pronouns are skipped. Rules are conservative heuristics and do not resolve arbitrary language or coreference.
|
|
60
|
+
|
|
61
|
+
Each link records the old memory in `supersedes`, the replacement in `superseded_by`, both claim keys, the reason, the replacement assertion's `superseded_at`, and the wall-clock `detected_at`. Memory bodies, IDs, and importance are preserved. A derived `superseded_memories` view marks a memory fully superseded only when every extracted claim has a replacement. Independent claims in a mixed summary keep it searchable; its content can still contain stale claims, so inspect partial supersessions in history when auditing such summaries. No old memory is deleted by supersession; explicit deletion and retention keep their existing behavior.
|
|
62
|
+
|
|
63
|
+
Writes reconcile only affected entity/relation groups within the same project, atomically with memory and graph updates. Assertions are ordered by their stored timestamp, then memory insertion rowid for equal timestamps. This gives pinned-clock captures a deterministic arrival order. The earliest later contradictory assertion supplies each link, permitting chains and later reaffirmations without cycles. Identical replay preserves links and detection timestamps; imports, content rewrites, summary ID changes, and deletion reconnect affected chains. Migration 015 backfills existing memories once, using full session events for summaries. Scan preserves valid original transcript timestamps, so filename order does not determine freshness. Without original timestamps, stored capture order is the fallback. An unrelated appended event does not change an earlier assertion's time.
|
|
64
|
+
|
|
65
|
+
Recall filters fully superseded memories before the FTS candidate limit and fallback scan, giving current facts priority. The optional `includeHistory: true` request field, exposed as `sessionmem search --include-history`, restores normal relevance ranking across current and superseded memories. Existing requests and response shapes remain compatible. `getMemory`, list and export still expose earlier memories. `sessionmem history [--project <id>] [--json]` lists links with old/new values, timestamps, reasons, source IDs, and partial/full status. Export formats remain unchanged: importing rebuilds links from exported bodies and timestamps; event-only claims absent from a summary cannot round-trip.
|
|
66
|
+
|
|
67
|
+
## Retrieval flow
|
|
68
|
+
|
|
69
|
+
When an agent or `sessionmem search` submits a query, the core engine searches the SQLite FTS5 index over stored memory content, including summaries. It removes common English question words and matches any remaining term. SQLite's built-in Porter tokenizer matches English inflections such as “hike” and “hiking”. BM25 accounts for term rarity, frequency, and memory length; at most 50 lexical candidates are returned, ordered by BM25 and then ID.
|
|
70
|
+
|
|
71
|
+
The ranker divides each positive BM25 score by the best score for that query. If the query contains known entity names in this project, relevance is `0.90 * base relevance + 0.10 * entity overlap`. Overlap is the fraction of distinct matched query names linked to the candidate memory; names are matched as literal token sequences (up to eight tokens), including lowercase questions. With no known query entities, relevance is unchanged. This reranks the existing candidates; it does not expand the FTS candidate pool or traverse relations. The outer relevance, recency, and importance weights stay 0.60, 0.25, and 0.15. Sparse lexical matches retain the recency/importance fallback union. If there are no lexical matches, base relevance uses the previous hash-vector ranking. Project scope and expiry checks apply to both paths. Automatic migrations preserve memory bodies and IDs.
|
|
72
|
+
|
|
73
|
+
Existing public request fields and all response fields are unchanged; `includeHistory` is an optional addition. The `semantic` field still reports hash cosine similarity. The legacy `score.raw.semantic` and `score.weighted.semantic` fields carry the relevance component used for ranking, including entity overlap when query entities match. Scores are relative to the current query's candidates, not calibrated probabilities. Retrieval needs no API keys, downloaded models, or new dependencies.
|
|
74
|
+
|
|
75
|
+
## Injection flow
|
|
76
|
+
|
|
77
|
+
At session start, the engine retrieves relevant memories and trims them to a token budget, a few hundred tokens by default. It keeps critical warnings and high-importance decisions first, dropping lower-priority content as needed. The adapter adds the result to the host's context. Claude Code's installed SessionStart hook does this automatically. Other hosts need an agent or lifecycle integration to call the corresponding MCP tools; MCP registration alone does not install hooks in those hosts.
|
|
@@ -0,0 +1,122 @@
|
|
|
1
|
+
# Conversational retrieval benchmark
|
|
2
|
+
|
|
3
|
+
`npm run benchmark` runs the committed LoCoMo corpus through the public
|
|
4
|
+
`storeMemory` capture and `retrieveMemories` recall APIs. See
|
|
5
|
+
[benchmarks/README.md](../benchmarks/README.md) for the corpus license, selection
|
|
6
|
+
rules, metric definitions, and detailed reproduction instructions.
|
|
7
|
+
|
|
8
|
+
## Entity overlap comparison
|
|
9
|
+
|
|
10
|
+
The baseline was run first on current main at `81f987c`. A fixed 10% entity
|
|
11
|
+
contribution inside the existing relevance component was then evaluated without
|
|
12
|
+
a weight sweep. Both runs used Node v24.19.0, Linux x64, sessionmem 1.4.0, the
|
|
13
|
+
same installed lockfile dependencies, top-5, fixed clocks, and the unchanged
|
|
14
|
+
LoCoMo corpus and scoring rules. Each run scored 1,527 questions after capturing
|
|
15
|
+
5,882 turns; both recorded zero network attempts.
|
|
16
|
+
|
|
17
|
+
| Macro retrieval metric @5 | Main (BM25) | BM25 + entity overlap |
|
|
18
|
+
| --- | ---: | ---: |
|
|
19
|
+
| Hit rate | 61.95% | 63.92% |
|
|
20
|
+
| Precision | 13.45% | 14.05% |
|
|
21
|
+
| Recall | 55.96% | 57.45% |
|
|
22
|
+
| All-evidence rate | 51.15% | 52.26% |
|
|
23
|
+
|
|
24
|
+
| Recall latency | Main | Entity overlap |
|
|
25
|
+
| --- | ---: | ---: |
|
|
26
|
+
| Mean | 8.57 ms | 8.05 ms |
|
|
27
|
+
| Median | 7.27 ms | 7.58 ms |
|
|
28
|
+
| p95 | 21.04 ms | 14.94 ms |
|
|
29
|
+
| Maximum | 35.81 ms | 28.38 ms |
|
|
30
|
+
|
|
31
|
+
Entity overlap gained hits on 34 questions and lost hits on four, a net gain
|
|
32
|
+
of 30. Precision and recall increased on 48 questions and decreased on five.
|
|
33
|
+
The signal measurably improved retrieval on this corpus, so it is retained.
|
|
34
|
+
Timing is one call per question without warmup and varies with load; the lower
|
|
35
|
+
mean/p95 is not evidence that the added lookup makes retrieval faster. This
|
|
36
|
+
does not establish generalization to other corpora or measure extraction accuracy.
|
|
37
|
+
The rule vocabulary and fixed weight were not tuned against these results.
|
|
38
|
+
|
|
39
|
+
[Recorded aggregate results](../benchmarks/entity-overlap-results.json) include
|
|
40
|
+
the corpus checksum, shared configuration, runtime, category metrics, and paired
|
|
41
|
+
question counts. To reproduce, run the commands below on `81f987c` with output
|
|
42
|
+
`benchmarks/results/locomo-main.json`, then on this change with output
|
|
43
|
+
`benchmarks/results/locomo-entities.json`. Full per-question reports remain ignored
|
|
44
|
+
local artifacts.
|
|
45
|
+
|
|
46
|
+
## Earlier BM25 comparison
|
|
47
|
+
|
|
48
|
+
The baseline was run first on main at `c06857d`, before changing retrieval.
|
|
49
|
+
Both runs used sessionmem 1.4.0, Node v24.19.0, Linux x64, top-5,
|
|
50
|
+
and the same lockfile dependencies and fixed capture/retrieval clock.
|
|
51
|
+
They captured 5,882 turns across 272 sessions in 10 conversations and scored
|
|
52
|
+
1,527 questions. The same 446 adversarial, four missing-evidence, and nine
|
|
53
|
+
invalid-evidence questions were excluded by fixed annotation rules.
|
|
54
|
+
|
|
55
|
+
| Macro retrieval metric @5 | Main (hash ranking) | BM25 + stop words + stemming |
|
|
56
|
+
| --- | ---: | ---: |
|
|
57
|
+
| Hit rate | 1.44% | 61.95% |
|
|
58
|
+
| Precision | 0.29% | 13.45% |
|
|
59
|
+
| Recall | 0.88% | 55.96% |
|
|
60
|
+
| All-evidence rate | 0.65% | 51.15% |
|
|
61
|
+
|
|
62
|
+
| Recall latency | Main | Selected change |
|
|
63
|
+
| --- | ---: | ---: |
|
|
64
|
+
| Mean | 16.33 ms | 14.93 ms |
|
|
65
|
+
| Median | 15.22 ms | 13.52 ms |
|
|
66
|
+
| p95 | 28.80 ms | 28.81 ms |
|
|
67
|
+
| Maximum | 133.49 ms | 56.73 ms |
|
|
68
|
+
|
|
69
|
+
Latency includes the complete public service response, including startup-injection
|
|
70
|
+
formatting, and excludes capture, process startup, and corpus loading. Each query
|
|
71
|
+
is timed once without warmup. Timing depends on hardware and load.
|
|
72
|
+
|
|
73
|
+
| Question category | Questions | Hit rate | Precision | Recall |
|
|
74
|
+
| --- | ---: | ---: | ---: | ---: |
|
|
75
|
+
| Multi-hop | 278 | 46.76% | 13.60% | 25.06% |
|
|
76
|
+
| Temporal | 320 | 69.38% | 14.38% | 66.22% |
|
|
77
|
+
| Open-domain | 89 | 38.20% | 8.09% | 28.25% |
|
|
78
|
+
| Single-hop | 840 | 66.67% | 13.62% | 65.22% |
|
|
79
|
+
|
|
80
|
+
## Candidate comparisons
|
|
81
|
+
|
|
82
|
+
Each candidate was run with the existing offline evaluator on the same questions.
|
|
83
|
+
Only additions that increased hit rate, precision, and recall were retained.
|
|
84
|
+
|
|
85
|
+
| Candidate @5 | Hit rate | Precision | Recall |
|
|
86
|
+
| --- | ---: | ---: | ---: |
|
|
87
|
+
| BM25 with OR terms | 52.78% | 11.15% | 47.69% |
|
|
88
|
+
| + English stop-word filtering | 57.69% | 12.39% | 52.62% |
|
|
89
|
+
| + Porter stemming (selected) | 61.95% | 13.45% | 55.96% |
|
|
90
|
+
| 90% BM25 / 10% hash cosine (discarded) | 61.76% | 13.43% | 55.78% |
|
|
91
|
+
|
|
92
|
+
BM25 preserves lexical relevance instead of replacing it with hash-vector noise.
|
|
93
|
+
Common question words no longer require a literal match, and SQLite's built-in
|
|
94
|
+
Porter tokenizer matches English inflections. The existing recency, importance,
|
|
95
|
+
expiry, project scope, and no-match vector fallback remain in place. There are
|
|
96
|
+
no new dependencies, API keys, or model downloads. Migration 013 rebuilds the
|
|
97
|
+
existing FTS index; memory content and IDs are preserved.
|
|
98
|
+
|
|
99
|
+
These comparisons use the same corpus used to select the ranker; they are not
|
|
100
|
+
a held-out generalization estimate. BM25 still needs lexical overlap, and a hit
|
|
101
|
+
can leave evidence missing for multi-hop questions. These are annotated-evidence
|
|
102
|
+
retrieval scores, not answer-generation accuracy or summarization quality.
|
|
103
|
+
|
|
104
|
+
## Reproduce
|
|
105
|
+
|
|
106
|
+
After installing the lockfile dependencies, run on main to record the baseline,
|
|
107
|
+
then repeat on the change:
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
npm run build
|
|
111
|
+
npm run benchmark -- --output benchmarks/results/locomo.json
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
The corpus is pinned to upstream commit
|
|
115
|
+
`3eb6f2c585f5e1699204e3c3bdf7adc5c28cb376`, SHA-256
|
|
116
|
+
`79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4`.
|
|
117
|
+
The runner verifies this checksum and blocks network calls. No API keys are
|
|
118
|
+
required. A second full run with an empty environment produced identical
|
|
119
|
+
retrieved IDs and quality metrics for all 1,527 questions. JSON output includes
|
|
120
|
+
every scored question, answer, evidence list, returned IDs, and exclusion reason
|
|
121
|
+
so failures can be inspected. The report above is a recorded result; the command
|
|
122
|
+
prints current results and writes JSON only when requested.
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
# Cloud Summarization
|
|
2
|
+
|
|
3
|
+
## Default Mode
|
|
4
|
+
|
|
5
|
+
`sessionmem` defaults to local summarization. Local mode runs without external API calls.
|
|
6
|
+
|
|
7
|
+
## How to Enable Cloud Mode
|
|
8
|
+
|
|
9
|
+
Cloud mode activates only when both conditions are true:
|
|
10
|
+
|
|
11
|
+
1. `allowCloudSummarization=true`
|
|
12
|
+
2. `ANTHROPIC_API_KEY` is present (mapped to `anthropicApiKey` in runtime config)
|
|
13
|
+
|
|
14
|
+
If either condition is missing, summarization stays local.
|
|
15
|
+
|
|
16
|
+
## How It Works
|
|
17
|
+
|
|
18
|
+
Cloud summarization runs in two steps:
|
|
19
|
+
|
|
20
|
+
1. **Local preprocessing** - the local summarizer extracts and structures session events
|
|
21
|
+
into a short transcript, with secret redaction applied before it leaves the machine.
|
|
22
|
+
|
|
23
|
+
2. **Cloud compression** - the transcript is sent to the Anthropic Messages API with
|
|
24
|
+
instructions to compress it into a short list of facts, decisions, and context.
|
|
25
|
+
|
|
26
|
+
### Model Configuration
|
|
27
|
+
|
|
28
|
+
The default model is `claude-sonnet-4-6`, defined as `DEFAULT_SUMMARIZER_MODEL` in
|
|
29
|
+
`src/core/config/policyConfig.ts`. Callers can override the model via the `model` field
|
|
30
|
+
in `CloudSummarizeInput`.
|
|
31
|
+
|
|
32
|
+
### Token Limits
|
|
33
|
+
|
|
34
|
+
The API's `max_tokens` parameter is `Math.min(Math.floor((summaryTokenCap ?? 4000) * 1.5), 8192)`.
|
|
35
|
+
The response is then trimmed to `summaryTokenCap`.
|
|
36
|
+
|
|
37
|
+
## Runtime Warning Signals
|
|
38
|
+
|
|
39
|
+
When cloud mode is active, lifecycle responses include:
|
|
40
|
+
|
|
41
|
+
- warning code: `cloud_summarization_enabled`
|
|
42
|
+
- warning message: `Cloud summarization active: allowCloudSummarization=true and ANTHROPIC_API_KEY present`
|
|
43
|
+
|
|
44
|
+
Use these warnings to check whether cloud summarization is active.
|
|
45
|
+
|
|
46
|
+
## Failure Fallback Behavior
|
|
47
|
+
|
|
48
|
+
Cloud summarization retries 2 times. If the retries fail, it falls back to local summarization automatically.
|
|
49
|
+
|
|
50
|
+
Cloud failures are saved in `summarization_failures`. If local fallback succeeds,
|
|
51
|
+
the response includes a warning about the cloud failure. If fallback also fails,
|
|
52
|
+
an additional failure record is saved.
|
|
53
|
+
|
|
54
|
+
## Manual Summarization When Auto Is Disabled
|
|
55
|
+
|
|
56
|
+
Setting `autoSummarize=false` disables automatic `session_end` summarization only.
|
|
57
|
+
You can still call `summarizeSessionToMemory` manually.
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# Migration and Upgrade Policy
|
|
2
|
+
|
|
3
|
+
This guide explains SQLite migrations and what to expect when upgrading sessionmem. Configuration and the default database live under `~/.sessionmem/`.
|
|
4
|
+
|
|
5
|
+
## Migration system
|
|
6
|
+
|
|
7
|
+
`sessionmem` updates its SQLite schema with forward-only migrations. Each migration is a `.sql` file under `src/core/schema/migrations/`. Zero-padded numeric prefixes determine the order:
|
|
8
|
+
|
|
9
|
+
```
|
|
10
|
+
src/core/schema/migrations/
|
|
11
|
+
001_initial.sql
|
|
12
|
+
002_indexes.sql
|
|
13
|
+
003_summarization_failures.sql
|
|
14
|
+
004_memory_feedback.sql
|
|
15
|
+
005_team_provenance.sql
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
### How migrations run
|
|
19
|
+
|
|
20
|
+
At startup, the runner creates the `_migrations` tracking table if needed and sorts the `.sql` files by name. It applies each migration in its own transaction, skipping any already recorded in `_migrations`. Starting sessionmem again does not reapply completed migrations. There are no down-migrations.
|
|
21
|
+
|
|
22
|
+
Migration `015_memory_supersessions.sql` adds derived fact claims, timestamped replacement links, and a view identifying fully superseded memories. It backfills existing records locally in the migration transaction, using full session events for summaries where available. Stored memory content and IDs are preserved. Recall then defaults to current facts; use `includeHistory: true` or `sessionmem search --include-history` to retrieve superseded facts, and `sessionmem history` to inspect their chains.
|
|
23
|
+
|
|
24
|
+
### Build-time copy: `copy-migrations`
|
|
25
|
+
|
|
26
|
+
The TypeScript compiler does not copy `.sql` files. The build step `copy-migrations` runs `scripts/copy-migrations.mjs` to copy them from `src/core/schema/migrations/` to `dist/core/schema/migrations/`. This includes the migrations in the published package alongside the compiled code.
|
|
27
|
+
|
|
28
|
+
At runtime, sessionmem loads migrations relative to the installed module, using the packaged `dist` copy. It does not search the working directory, where an attacker could place a malicious `migrations/` folder (Phase 5 Decision #15). After adding or editing a migration, re-run the build so `copy-migrations` updates `dist`.
|
|
29
|
+
|
|
30
|
+
### Adding a migration (contributors)
|
|
31
|
+
|
|
32
|
+
1. Add a new `NNN_description.sql` file with the next ordinal in `src/core/schema/migrations/`.
|
|
33
|
+
2. Keep it forward-only and idempotent where practical.
|
|
34
|
+
3. Re-run the build so `copy-migrations` copies it into `dist/core/schema/migrations/`.
|
|
35
|
+
|
|
36
|
+
## Version-upgrade policy
|
|
37
|
+
|
|
38
|
+
`sessionmem` follows **semantic versioning** (semver): `MAJOR.MINOR.PATCH`.
|
|
39
|
+
|
|
40
|
+
- **Patch / minor upgrades** work with your existing database. Additive, forward-only migrations run automatically when the upgraded version first opens it. Stored memories are preserved and no manual migration is needed.
|
|
41
|
+
- **Major-version upgrades** may introduce breaking changes. The release notes will describe behavior changes and any manual migration steps.
|
|
42
|
+
- **Downgrades after migration are unsupported.** Before a major upgrade, run `sessionmem export` to keep a portable JSON copy of your memories that you can re-import if needed.
|
|
43
|
+
|
|
44
|
+
This policy applies to upgrades between published v1 releases. Each release opens existing databases through the same forward-only migration runner.
|
|
@@ -0,0 +1,139 @@
|
|
|
1
|
+
# Privacy and Retention
|
|
2
|
+
|
|
3
|
+
`sessionmem` stores memories locally, redacts secrets before saving, and prunes old memories according to a retention policy. Configure both in `~/.sessionmem/config.json` or with the CLI commands below.
|
|
4
|
+
|
|
5
|
+
## Secret Redaction
|
|
6
|
+
|
|
7
|
+
Redaction replaces common secret patterns with `REDACTED` before saving memory content. It runs during:
|
|
8
|
+
|
|
9
|
+
- automatic summarization at session end,
|
|
10
|
+
- manual memory storage (`storeMemory` or `batchStoreMemory`),
|
|
11
|
+
- import.
|
|
12
|
+
|
|
13
|
+
The `redactionEnabled` flag controls all of these operations and defaults to `true`.
|
|
14
|
+
|
|
15
|
+
### Redacted categories
|
|
16
|
+
|
|
17
|
+
| Category | Examples |
|
|
18
|
+
|----------|----------|
|
|
19
|
+
| Email addresses | `user@example.com` |
|
|
20
|
+
| `sk-` API keys | OpenAI-style `sk-...` keys |
|
|
21
|
+
| AWS access keys | `AKIA...` |
|
|
22
|
+
| GitHub tokens | `ghp_...`, `gho_...`, and related prefixes |
|
|
23
|
+
| Bearer tokens | `Authorization: Bearer ...` |
|
|
24
|
+
| Private key blocks | `-----BEGIN ... PRIVATE KEY-----` private key blocks |
|
|
25
|
+
| Connection-string assignments | `password=...`, `secret=...` |
|
|
26
|
+
| JWTs | three-part `xxxxx.yyyyy.zzzzz` tokens |
|
|
27
|
+
|
|
28
|
+
The project's CI secret scanning checks for similar categories.
|
|
29
|
+
|
|
30
|
+
### Partial-failure warnings
|
|
31
|
+
|
|
32
|
+
If redaction partly fails, the response includes a `redaction_partial_failure` warning through the usual lifecycle warning mechanism. The write still completes. Review the affected memory when you see this warning.
|
|
33
|
+
|
|
34
|
+
### Disabling redaction
|
|
35
|
+
|
|
36
|
+
Setting `redactionEnabled` to `false` disables redaction for all memory writes. Leave it enabled to keep these checks.
|
|
37
|
+
|
|
38
|
+
## Retention Policy
|
|
39
|
+
|
|
40
|
+
Retention pruning limits growth of the local store by deleting memories older than a configured age.
|
|
41
|
+
|
|
42
|
+
- **Scope:** the `memories` table only. Session events and feedback history are never touched.
|
|
43
|
+
- **Age basis:** a memory's `created_at` timestamp.
|
|
44
|
+
- **Default window:** `90` days (`retentionDays`).
|
|
45
|
+
- **Disabling:** set `retentionDays` to `0` to disable pruning. Negative values are rejected by the config schema.
|
|
46
|
+
- **Hard delete:** pruned memories are permanently deleted from SQLite. There is no archive or soft-delete.
|
|
47
|
+
|
|
48
|
+
> Pruning is permanent. Run `sessionmem export` first if you want to keep a copy of the memories.
|
|
49
|
+
|
|
50
|
+
## Automatic vs Manual Pruning
|
|
51
|
+
|
|
52
|
+
**Automatic (session-end):** a lightweight prune check runs alongside summarization without blocking or delaying it.
|
|
53
|
+
|
|
54
|
+
**Manual:** the prune command previews deletions by default. It only deletes memories when you pass `--force`:
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
sessionmem retention prune
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Dry-run output:
|
|
61
|
+
|
|
62
|
+
```
|
|
63
|
+
Would delete N memories older than {retentionDays} days. Pass --force to confirm.
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
To delete eligible memories:
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
sessionmem retention prune --force
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Use `--days <n>` to set the retention window for one run, overriding `config.json` and the built-in default:
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
sessionmem retention prune --days 30
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
## One-time Redaction Scrub
|
|
79
|
+
|
|
80
|
+
Normal memory writes do not recheck older memories. To check memories saved before redaction coverage was expanded, run `redact-scan`. It previews changes by default:
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
sessionmem redact-scan
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
The preview shows matching memories and a count. To redact and update them in place:
|
|
87
|
+
|
|
88
|
+
```bash
|
|
89
|
+
sessionmem redact-scan --apply
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
`--apply` rewrites matching memories with `REDACTED` placeholders and prints a summary count.
|
|
93
|
+
|
|
94
|
+
## Policy Config
|
|
95
|
+
|
|
96
|
+
Policy settings live in `~/.sessionmem/config.json`:
|
|
97
|
+
|
|
98
|
+
```json
|
|
99
|
+
{
|
|
100
|
+
"retentionDays": 90,
|
|
101
|
+
"redactionEnabled": true
|
|
102
|
+
}
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
### Precedence
|
|
106
|
+
|
|
107
|
+
Settings take precedence in this order, highest first:
|
|
108
|
+
|
|
109
|
+
1. **CLI flag** (e.g. `retention prune --days 30`)
|
|
110
|
+
2. **`config.json`** value
|
|
111
|
+
3. **built-in default**
|
|
112
|
+
|
|
113
|
+
If `config.json` is missing or invalid, sessionmem uses the built-in defaults unless a CLI flag overrides them.
|
|
114
|
+
|
|
115
|
+
### Reading and writing config
|
|
116
|
+
|
|
117
|
+
Use `config get` and `config set` to read and change settings:
|
|
118
|
+
|
|
119
|
+
```bash
|
|
120
|
+
sessionmem config get retentionDays
|
|
121
|
+
sessionmem config set retention.days 30
|
|
122
|
+
sessionmem config set redactionEnabled false
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
You can use either the dotted key `retention.days` or the field name `retentionDays`. Invalid values, such as a non-integer for `retentionDays` or a value other than `true`/`false` for `redactionEnabled`, produce an error without changing the file.
|
|
126
|
+
|
|
127
|
+
### Install and uninstall behavior
|
|
128
|
+
|
|
129
|
+
- `sessionmem install` creates `config.json` with defaults (`retentionDays: 90`, `redactionEnabled: true`) if the file is missing. It leaves an existing file unchanged.
|
|
130
|
+
- `sessionmem uninstall` keeps `config.json` and the database unless you pass `--purge`.
|
|
131
|
+
|
|
132
|
+
## Stats Visibility
|
|
133
|
+
|
|
134
|
+
`sessionmem stats` shows the current retention and redaction settings, for example:
|
|
135
|
+
|
|
136
|
+
```
|
|
137
|
+
Retention: 90 days (12 memories eligible for pruning)
|
|
138
|
+
Redaction: enabled
|
|
139
|
+
```
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
# Team Mode (Shared Memory)
|
|
2
|
+
|
|
3
|
+
Team mode shares `sessionmem` memories through a directory everyone on the team can read and write, such as a network drive or synced folder. With team mode enabled, `sessionmem sync` writes your local memories there and reads teammates' memories into your database. Your agent can then use decisions, warnings, and facts recorded by the team.
|
|
4
|
+
|
|
5
|
+
Team mode is off by default. Enable it to start sharing.
|
|
6
|
+
|
|
7
|
+
## Setup
|
|
8
|
+
|
|
9
|
+
Enable team mode by pointing it at a shared directory:
|
|
10
|
+
|
|
11
|
+
```
|
|
12
|
+
sessionmem team enable <shared-path>
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
`<shared-path>` must be a directory that all teammates can read and write, typically a network share or a folder synced across machines. The path is recorded in `~/.sessionmem/config.json` under the `team` section (`team.enabled` / `team.sharedPath`).
|
|
16
|
+
|
|
17
|
+
Inside the shared path, `sessionmem` uses a per-project, per-user file layout:
|
|
18
|
+
|
|
19
|
+
```
|
|
20
|
+
{sharedPath}/{project_id}/{username}.json
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
Each teammate writes one snapshot file named after their OS username. Each project has its own subdirectory to keep snapshots separate.
|
|
24
|
+
|
|
25
|
+
To check the current state:
|
|
26
|
+
|
|
27
|
+
```
|
|
28
|
+
sessionmem team status
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
This reports whether team mode is enabled and whether the shared path is reachable.
|
|
32
|
+
|
|
33
|
+
## Usage: syncing
|
|
34
|
+
|
|
35
|
+
Once team mode is enabled, run:
|
|
36
|
+
|
|
37
|
+
```
|
|
38
|
+
sessionmem sync
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Each `sync` runs two steps:
|
|
42
|
+
|
|
43
|
+
1. **Push**: writes a full snapshot of this project's local memories to
|
|
44
|
+
`{sharedPath}/{project_id}/{username}.json`. The write is atomic (a temp file
|
|
45
|
+
is written in the same directory and then renamed) so teammates don't read a
|
|
46
|
+
partly written snapshot.
|
|
47
|
+
2. **Pull**: reads every other `*.json` snapshot in the project directory and
|
|
48
|
+
merges those memories into your local database.
|
|
49
|
+
|
|
50
|
+
When it finishes, `sync` prints a summary:
|
|
51
|
+
|
|
52
|
+
```
|
|
53
|
+
Pushed N memories, pulled M new + updated K from teammates.
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
`N` is the number of local memories shared, `M` is the number of new teammate memories added, and `K` is the number of existing memories updated from teammate snapshots.
|
|
57
|
+
|
|
58
|
+
If team mode is disabled, `sync` makes no changes and tells you to run `team enable` first.
|
|
59
|
+
|
|
60
|
+
## Provenance and author annotation
|
|
61
|
+
|
|
62
|
+
Each memory records its source:
|
|
63
|
+
|
|
64
|
+
- **`author`**: the OS username of whoever created the memory.
|
|
65
|
+
- **`created_at`**: when it was first created.
|
|
66
|
+
|
|
67
|
+
In the agent's startup context, teammate memories have an `author:` prefix showing their recorded author, for example:
|
|
68
|
+
|
|
69
|
+
```
|
|
70
|
+
- [decision] alice: decided to use X (...)
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
Your own memories have no prefix. Older memories with an empty author also have no prefix. The prefix appears when the recorded author differs from your local username.
|
|
74
|
+
|
|
75
|
+
## Conflict behavior
|
|
76
|
+
|
|
77
|
+
When the same memory id exists locally and in a teammate's snapshot, `sync` resolves it as follows:
|
|
78
|
+
|
|
79
|
+
- **Content**: sync uses last-write-wins by id, keeping the most recent content.
|
|
80
|
+
- **Importance**: a teammate's snapshot cannot lower your local copy's importance score.
|
|
81
|
+
- **Cross-project id collisions**: records whose ids already belong to a different project are skipped.
|
|
82
|
+
- **Redaction**: every pulled record is checked again using your secret redaction policy.
|
|
83
|
+
|
|
84
|
+
## Disable and recovery
|
|
85
|
+
|
|
86
|
+
To stop syncing and keep memories already pulled from teammates:
|
|
87
|
+
|
|
88
|
+
```
|
|
89
|
+
sessionmem team disable
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
To also delete teammate-authored memories:
|
|
93
|
+
|
|
94
|
+
```
|
|
95
|
+
sessionmem team disable --remove-team-memories
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
This deletes the teammate-authored memories for this project, leaving only the memories you authored locally.
|
|
99
|
+
|
|
100
|
+
## Trust boundary and security
|
|
101
|
+
|
|
102
|
+
Anyone with write access to the shared path can add memories that your agent may use. Control access through the shared directory's filesystem permissions:
|
|
103
|
+
|
|
104
|
+
- Restrict write access with OS filesystem permissions (ACLs) to the people you intend to share memories with.
|
|
105
|
+
- The `author` field is advisory. A person with write access can change it, so the `author:` prefix does not verify identity.
|
|
106
|
+
- `sessionmem` does not authenticate teammates, sign snapshots, or verify authorship. Enforce any additional access controls through the filesystem or file share.
|
|
107
|
+
|
|
108
|
+
## Failure recovery
|
|
109
|
+
|
|
110
|
+
When `sync` encounters filesystem or snapshot errors:
|
|
111
|
+
|
|
112
|
+
- **Missing or unwritable shared path**: `sync` reports the error to stderr and
|
|
113
|
+
exits with a non-zero status. Your local memories are untouched.
|
|
114
|
+
- **A corrupt or truncated teammate file**: the file is skipped with a warning,
|
|
115
|
+
and the rest of the pull continues.
|
|
116
|
+
- **An invalid record inside an otherwise-valid file**: the record is skipped
|
|
117
|
+
with a warning; the remaining records still merge.
|