dsh-hypatia-auto-memory 0.1.1 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -24,7 +24,12 @@ ships here: auto-approval for the agent's own bash `hypatia` calls, and its
24
24
 
25
25
  Prerequisite: `hypatia` CLI on PATH (or add its basename to `binaries`). A
26
26
  build with the `mcp` subcommand (hypatia #20) is used over MCP; an older one
27
- works through the CLI instead, with one warning at the first call.
27
+ works through the CLI instead, with one warning at the first call. Two later
28
+ additions are used when the binary has them and skipped when it does not:
29
+ `--no-embed` (hypatia #26) keeps the log layer out of the vector index, and
30
+ `similar --exclude-tags` (#35) keeps it out of the work-unit candidate search.
31
+ The first call logs which the binary offers
32
+ (`hypatia over mcp: no-embed yes, similar filters yes`).
28
33
 
29
34
  ```bash
30
35
  # from a local checkout (development)
@@ -129,6 +134,19 @@ agent/session-start ──▶ rules/taboos inject()
129
134
  `(tool calls only)`. Plugin-sourced
130
135
  messages are skipped — no feedback loops.
131
136
 
137
+ - **The log layer is not embedded.** `msg-*` and `session-*` entries and the
138
+ `belongTo` / `summary` edges are written `embed: false` (hypatia #26): stored,
139
+ full-text indexed and reachable by JSE, but given no vector. The protocol has
140
+ precise recall drill down from a summary along `summary` edges rather than
141
+ hit raw messages by meaning, and embedded, a raw message outranks the
142
+ knowledge distilled from it because it holds the very words. On the shelf
143
+ this was measured on the layer was 85% of entries and 77% of statements, each
144
+ costing a forward pass on the conversation's hot path. Summaries and work
145
+ units are embedded as before. Statements have no tags, so a shelf's
146
+ `embedding.skip_tags` could not have reached the edges; the plugin says so per
147
+ write. On a binary without the flag the transport drops it and everything is
148
+ embedded as it always was.
149
+
132
150
  Spans are cut at `turn/end`, and — for a turn that outlasts `flushWindowMs` —
133
151
  at `step/end`, never at an arbitrary moment. DSH appends `assistant/message`
134
152
  *before* running the tools it requested and `step/end` only after they have
@@ -164,8 +182,10 @@ agent/session-start ──▶ rules/taboos inject()
164
182
  previous tier already distilled.
165
183
 
166
184
  - **Work units** are adjudicated, not guessed. Candidates come from `similar`
167
- (the only search that reports a distance), filtered by a distance ceiling and
168
- by an exclusion list for the operational layer, then a small model call
185
+ (the only search that reports a distance), with the operational layer left
186
+ out — in the query, by `--exclude-tags`, on a binary that has it (hypatia
187
+ #35); by fetching four times as many rows and dropping them on one that does
188
+ not — and filtered by a distance ceiling, then a small model call
169
189
  decides `duplicate | refines | extends | supersedes | contradicts | unrelated`.
170
190
  A contradiction keeps **both** entries and records `supersedes` — a memory
171
191
  system must not quietly forget what it once believed.
@@ -335,7 +355,12 @@ changes. No card claims it, so it renders nowhere; nothing ever writes to it.
335
355
  (get-before-create); uncovered ranges are re-enqueued from watermarks.
336
356
  4. To check the route warning: consolidation with no selected `models`
337
357
  logs a one-time warning and otherwise stays silent.
338
- 5. Uninstall: `dsh plugin --profile web remove dsh-hypatia-auto-memory` —
358
+ 5. `hypatia backfill --status` reports the shelf's embedding debt. With a
359
+ current hypatia, logging a turn adds nothing to it; only summaries and work
360
+ units are pending until the next flush. `hypatia scope list --count` shows
361
+ the scopes in use — the one this project's `msg-*` entries carry is the one
362
+ the session seed reads.
363
+ 6. Uninstall: `dsh plugin --profile web remove dsh-hypatia-auto-memory` —
339
364
  hypatia entries themselves are left in `~/.hypatia/`.
340
365
 
341
366
  ## Failure modes
@@ -347,7 +372,8 @@ changes. No card claims it, so it renders nowhere; nothing ever writes to it.
347
372
  | Entries appear only after a turn finishes | By design: spans are cut at `turn/end`, or at the first `step/end` once a turn has run longer than `flushWindowMs`, so an entry never lacks its own tool results |
348
373
  | Logging works, no summaries | `consolidation.models` is empty or invalid — one warning at first trigger |
349
374
  | Summaries but no `sum2-*` | Fewer than `cascade.batchSize` unarchived tier-1 summaries in that project yet |
350
- | Work units have no relationships | No embedding model on the shelf (`similar` fails), or every candidate was beyond `dedupMaxDistance` |
375
+ | Work units have no relationships | No embedding model on the shelf (`similar` fails), or every candidate was beyond `dedupMaxDistance`. Since hypatia #19 the local model is looked up in `~/.hypatia/models/<org>/<name>` (or the Hugging Face cache), no longer beside the shelf: a shelf that used to answer `similar` and now says `is not installed` needs `hypatia model install <model>`, or `hypatia model register <model> <dir>` pointing at the files it already has |
376
+ | `similar` still returns `msg-*` rows | They were written before this plugin opted the log layer out of embedding, or by a hypatia without `--no-embed`. Retract the knowledge vectors once with `embedding.skip_tags = ["message", "session"]` in the shelf's `shelf.toml` followed by `hypatia backfill` (a binary older than the key refuses to open the shelf, so upgrade every binary sharing it first). Statements have no tags and no update command, so `belongTo` / `summary` edges written before keep their vectors |
351
377
  | Task in `deferred` state | Neither the live store nor persistence could supply its session. Not an error: it spends no attempts and runs as soon as one of them can. Normally storage answers immediately — the state persists only when `sessionPersistence` is absent from the composition, or its read failed (look for `could not be read` in the log) |
352
378
  | Task in `failed` state | A hypatia or model error persisted after `maxAttempts`. Failed `log-message` records are pruned at the next startup, since the watermark re-derives their range; other kinds are kept for inspection — delete one to let the next trigger re-create it |
353
379
  | Watermark says logged, but the shelf has no entries | The shelf was reset or switched after logging. Handled at startup: `housekeeping.reconcileOnStartup` resets any row whose session has no `msg-*` left, and that session is re-logged — and re-consolidated — from the start on its next activity. A shelf query that fails leaves the row untouched. A session whose messages were all deleted on purpose is indistinguishable and is logged again; turn the switch off if that matters |
@@ -359,7 +385,7 @@ changes. No card claims it, so it renders nowhere; nothing ever writes to it.
359
385
  | The agent's own `hypatia` write still asks for approval | `autoApprove: false` (needs a profile reload to change), the command pipes/redirects/chains outside quotes, or its first word is not one of `binaries` — only plain calls are answered, by design. Reads never reach approval at all, so nothing to fix there |
360
386
  | The agent got a memory protocol that tells it to log messages by hand | Another plugin registered `hypatia-memory` first (`dsh-hypatia` still in the profile), or a `hypatia-memory` exists on disk — typically `~/.agents/skills/hypatia-memory`, written by `hypatia skill install --agent codex`. Agent presets load disk skills in a layer nearer than plugins, so it wins in sessions even though the skill center may list this plugin. Remove the copy the startup warning names |
361
387
  | Startup warnings never appear in the terminal | `dsh web` mounts no log exporter, so plugin log lines of any level go nowhere; the console exporter's default threshold would also drop warnings (warn is level 2, above info's 1). Mount a logger such as `dsh-logbook` or `dsh-boot-doctor` temporarily to read them |
362
- | Project scope looks wrong | Scope = basename of the session cwd's git top level (`git rev-parse --show-toplevel`), or of the cwd itself when git finds no work tree or cannot answer (not installed, a repo it refuses as unsafe). A `rev-parse` slower than 3 s leaves that one session on the cwd's own name. Two same-named checkouts share a scope by design; a linked worktree is scoped by its own directory, not the main checkout's; git resolves symlinks, so a checkout opened through a link named differently from its target gets the target's name. A name hypatia would rewrite is normalized first: a session at `/` is scoped `/`, commas become `_`, surrounding whitespace goes. Entries written before these fixes stay where they were stored: a subdirectory session's under that directory's name (`repo/src` wrote under `src` — the git root was never read), a session at `/` with no scope, `a,b` under both `a` and `b`, `foo,` under `foo` and global. Padded names were stored trimmed, which is what the query now asks for. Nothing migrates them; `knowledge-update --scopes` moves one entry and keeps its `created_at` |
388
+ | Project scope looks wrong | Scope = basename of the session cwd's git top level (`git rev-parse --show-toplevel`), or of the cwd itself when git finds no work tree or cannot answer (not installed, a repo it refuses as unsafe). A `rev-parse` slower than 3 s leaves that one session on the cwd's own name. Two same-named checkouts share a scope by design; a linked worktree is scoped by its own directory, not the main checkout's; git resolves symlinks, so a checkout opened through a link named differently from its target gets the target's name. A name hypatia would rewrite is normalized first: a session at `/` is scoped `/`, commas become `_`, surrounding whitespace goes. Entries written before these fixes stay where they were stored: a subdirectory session's under that directory's name (`repo/src` wrote under `src` — the git root was never read), a session at `/` with no scope, `a,b` under both `a` and `b`, `foo,` under `foo` and global. Padded names were stored trimmed, which is what the query now asks for. Nothing migrates them; `knowledge-update --scopes` moves one entry and keeps its `created_at`. `hypatia scope list --count` (hypatia #30) shows every spelling in use and how many entries each holds |
363
389
 
364
390
  ## Known limitations
365
391
 
@@ -392,6 +418,12 @@ Deliberate, and worth knowing before you rely on them:
392
418
  exits 0 with `Statement already exists`); on an older binary the same
393
419
  error-text path covers it. The matching is confined to one function, and
394
420
  `test/integration` accepts either statement behaviour.
421
+ - **What was embedded stays embedded.** The log layer opts out of the vector
422
+ index at write time, so entries and edges written before this version, or by
423
+ a hypatia without `--no-embed`, keep their vectors and keep surfacing in
424
+ `similar`. `skip_tags` + `backfill` retracts the knowledge entries' (see
425
+ *Failure modes*); the edges have no retraction path short of deleting and
426
+ recreating them, which the plugin does not do.
395
427
  - **Adjudication and dedup need an embedding model.** On a shelf without one,
396
428
  `similar` fails outright and work units are stored with no relationship —
397
429
  never dropped.
package/lib/client.js CHANGED
@@ -108,46 +108,46 @@ window.__ModuleLoader__.load({
108
108
  document.head.appendChild(tag);
109
109
  }
110
110
  var SettingsCard_module_css_default = {
111
- "headText": "YVNYya_headText",
112
- "chevronOpen": "YVNYya_chevronOpen",
113
- "chevron": "YVNYya_chevron",
114
- "model": "YVNYya_model",
115
- "catalogError": "YVNYya_catalogError",
116
- "groupTitle": "YVNYya_groupTitle",
117
- "input": "YVNYya_input",
118
- "header": "YVNYya_header",
119
- "group": "YVNYya_group",
120
- "advancedHint": "YVNYya_advancedHint",
121
111
  "hint": "YVNYya_hint",
122
- "modelSelection": "YVNYya_modelSelection",
123
- "switch": "YVNYya_switch",
112
+ "field": "YVNYya_field",
124
113
  "readOnly": "YVNYya_readOnly",
125
- "fieldHead": "YVNYya_fieldHead",
126
- "switchOn": "YVNYya_switchOn",
127
- "reset": "YVNYya_reset",
128
114
  "select": "YVNYya_select",
129
- "card": "YVNYya_card",
130
- "body": "YVNYya_body",
131
- "cardOpen": "YVNYya_cardOpen",
132
- "modelName": "YVNYya_modelName",
133
- "limitsTitle": "YVNYya_limitsTitle",
134
- "label": "YVNYya_label",
135
- "pairedFields": "YVNYya_pairedFields",
136
115
  "switchThumb": "YVNYya_switchThumb",
116
+ "pairedFields": "YVNYya_pairedFields",
117
+ "fieldHead": "YVNYya_fieldHead",
118
+ "modelName": "YVNYya_modelName",
137
119
  "status": "YVNYya_status",
120
+ "switchOn": "YVNYya_switchOn",
121
+ "headText": "YVNYya_headText",
122
+ "chevron": "YVNYya_chevron",
123
+ "footer": "YVNYya_footer",
124
+ "unavailable": "YVNYya_unavailable",
125
+ "modelSelection": "YVNYya_modelSelection",
126
+ "chevronOpen": "YVNYya_chevronOpen",
127
+ "body": "YVNYya_body",
128
+ "error": "YVNYya_error",
129
+ "modelGroup": "YVNYya_modelGroup",
130
+ "card": "YVNYya_card",
131
+ "warning": "YVNYya_warning",
132
+ "switch": "YVNYya_switch",
138
133
  "notice": "YVNYya_notice",
139
- "field": "YVNYya_field",
134
+ "limitsTitle": "YVNYya_limitsTitle",
135
+ "reset": "YVNYya_reset",
136
+ "description": "YVNYya_description",
137
+ "group": "YVNYya_group",
138
+ "name": "YVNYya_name",
139
+ "advancedHint": "YVNYya_advancedHint",
140
140
  "models": "YVNYya_models",
141
+ "catalogError": "YVNYya_catalogError",
142
+ "label": "YVNYya_label",
141
143
  "providerName": "YVNYya_providerName",
142
- "name": "YVNYya_name",
143
- "description": "YVNYya_description",
144
- "error": "YVNYya_error",
145
- "warning": "YVNYya_warning",
146
- "route": "YVNYya_route",
147
- "unavailable": "YVNYya_unavailable",
148
- "footer": "YVNYya_footer",
144
+ "cardOpen": "YVNYya_cardOpen",
149
145
  "pending": "YVNYya_pending",
150
- "modelGroup": "YVNYya_modelGroup"
146
+ "route": "YVNYya_route",
147
+ "groupTitle": "YVNYya_groupTitle",
148
+ "model": "YVNYya_model",
149
+ "input": "YVNYya_input",
150
+ "header": "YVNYya_header"
151
151
  };
152
152
  //#endregion
153
153
  //#region src/client/locales.ts
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "dsh-hypatia-auto-memory",
3
- "version": "0.1.1",
3
+ "version": "0.2.0",
4
4
  "description": "Event-driven Hypatia memory for DSH: automatic conversation logging via session events, queued background consolidation (span summaries + work-unit extraction) with a dedicated model route, and recall injection — zero per-call model work in the main session.",
5
5
  "type": "module",
6
6
  "main": "src/index.js",
@@ -45,13 +45,13 @@
45
45
  "node": "^22.19.0 || >=24.0.0"
46
46
  },
47
47
  "dependencies": {
48
- "@deepseek-ai/dsh-llm": "^0.1.2-rc.1",
49
- "@deepseek-ai/dsh-storage-domain": "^0.1.2-rc.1",
50
48
  "@deepseek-ai/schemastery": "^3.18.2",
51
49
  "zod": "^4.4.3"
52
50
  },
53
51
  "peerDependencies": {
54
52
  "@deepseek-ai/cordis": "^4.0.2",
53
+ "@deepseek-ai/dsh-llm": "^0.1.2-rc.1",
54
+ "@deepseek-ai/dsh-storage-domain": "^0.1.2-rc.1",
55
55
  "react": "^18.2.0"
56
56
  },
57
57
  "devDependencies": {
@@ -64,6 +64,8 @@
64
64
  "@deepseek-ai/dsh-client-ui-settings": "^0.1.2-rc.1",
65
65
  "@deepseek-ai/dsh-client-ui-settings-plugins": "^0.1.2-rc.1",
66
66
  "@deepseek-ai/dsh-client-ui-slots": "^0.1.2-rc.1",
67
+ "@deepseek-ai/dsh-llm": "^0.1.2-rc.1",
68
+ "@deepseek-ai/dsh-storage-domain": "^0.1.2-rc.1",
67
69
  "@types/react": "^18.2.0",
68
70
  "lightningcss": "^1.32.0",
69
71
  "react": "^18.2.0",
@@ -68,6 +68,38 @@ hypatia knowledge-create "prefer immutable" -d "always create new objects" --tag
68
68
  hypatia knowledge-create "no mock DB" -d "never mock database in tests" --tags "taboo" --scopes "my-project,"
69
69
  ```
70
70
 
71
+ ### Listing the scopes and tags already in use
72
+
73
+ A scope or tag you invent is not an error — the entry is stored, and every later
74
+ lookup by the value you meant misses it. Check before you write:
75
+
76
+ ```bash
77
+ # What this shelf already uses, with how many entries carry each
78
+ hypatia scope list --count
79
+ hypatia tag list --count
80
+
81
+ # Exact values, when you are going to write one back
82
+ hypatia scope list --json # [{"value":"","entries":12},{"value":"my-project",…}]
83
+
84
+ # Confirm one spelling; exit 0 if in use, 1 if not
85
+ hypatia tag exists rule
86
+ hypatia scope exists my-project
87
+ hypatia scope exists "" # the global scope
88
+ ```
89
+
90
+ The plain listing prints the global scope as `(global)`. That is a label for the
91
+ terminal, not the value — the value is the empty string, and `--json` and
92
+ `exists` both give it back verbatim. Never copy `(global)` into `--scopes`.
93
+
94
+ Both cover knowledge and statements. An entry that declares no scopes at all is
95
+ not listed as global.
96
+
97
+ | User says | Command |
98
+ |---|---|
99
+ | "what scopes/projects are in here?" | `hypatia scope list --count` |
100
+ | "what tags exist?" / "which labels are used?" | `hypatia tag list --count` |
101
+ | "is there already a `rule` tag?" | `hypatia tag exists rule` |
102
+
71
103
  ## Knowledge CRUD
72
104
 
73
105
  Knowledge entries are independent information points with a name, content, and tags.
@@ -75,14 +107,17 @@ Knowledge entries are independent information points with a name, content, and t
75
107
  ### Create
76
108
 
77
109
  ```
78
- hypatia knowledge-create <name> -d "<data>" -t "<tag1,tag2>" --figures "archive://path/to/file"
110
+ hypatia knowledge-create <name> -d "<data>" -t "<tag1,tag2>" --figures "archive://path/to/file" [--no-embed]
79
111
  ```
80
112
 
113
+ `--no-embed` keeps the entry out of the vector index: it is still stored and still found by `search` and `query`, but it is never embedded, so it costs no model call on the way in and is not a `similar` result. Use it for raw session logs and other bulk that carries no distilled knowledge. A shelf can skip whole tags instead, with `embedding.skip_tags` in `shelf.toml`.
114
+
81
115
  | User says | Command |
82
116
  |---|---|
83
117
  | "remember Rust as a systems programming language" | `hypatia knowledge-create "Rust" -d "systems programming language" -t "language,compiled"` |
84
118
  | "save knowledge about Go with tags language and compiled" | `hypatia knowledge-create "Go" -d "" -t "language,compiled"` |
85
119
  | "store that Python is a scripting language, tag it as dynamic" | `hypatia knowledge-create "Python" -d "scripting language" -t "dynamic"` |
120
+ | "log this turn but don't make it searchable by meaning" | `hypatia knowledge-create "msg-42" -d "<turn>" -t "message" --no-embed` |
86
121
 
87
122
  ### Read
88
123
 
@@ -97,9 +132,11 @@ hypatia knowledge-get <name>
97
132
  ### Update
98
133
 
99
134
  ```
100
- hypatia knowledge-update <name> [-d "<data>"] [-t "<tags>"] [--synonyms "<csv>"] [--figures "<refs>"] [--scopes "<scopes>"]
135
+ hypatia knowledge-update <name> [-d "<data>"] [-t "<tags>"] [--synonyms "<csv>"] [--figures "<refs>"] [--scopes "<scopes>"] [--no-embed | --embed]
101
136
  ```
102
137
 
138
+ `--no-embed` takes the entry out of the vector index and discards the vector it had; `--embed` puts it back, and it is embedded on the next flush. The two are mutually exclusive. `--embed` does not override the shelf's `embedding.skip_tags`: an entry can ask for less indexing than its shelf, never more.
139
+
103
140
  Only the fields you pass change. An omitted field keeps its stored value, and exactly `""` clears a field, such as `-t ""`; for tags, `","` or `" "` would store blank tags instead. `--scopes` replaces the stored scopes and is parsed as on create: end the list with a comma, such as `"q,"`, to include the global scope, or the entry drops out of global lookups. The entry keeps its `created_at`. Its old vector is discarded, and a new one is generated on the next flush, as after `knowledge-create`. Updating an entry that does not exist is an error. An update that changes nothing prints `Knowledge unchanged: <name>` and writes nothing.
104
141
 
105
142
  | User says | Command |
@@ -129,9 +166,11 @@ When the user asks to store or remember information, **always create statements
129
166
  **Pattern**: After creating a knowledge entry, identify entities mentioned in the content and create `$triple` relationships between them. At minimum, create one `is_a` statement for every new knowledge entry.
130
167
 
131
168
  ```
132
- hypatia statement-create <head> <relation> <tail> -d "<data>"
169
+ hypatia statement-create <head> <relation> <tail> -d "<data>" [--no-embed]
133
170
  ```
134
171
 
172
+ `--no-embed` works as it does on `knowledge-create`: the triple is still stored and still traversable, but gets no vector. Two differences: statements carry no tags, so a shelf's `embedding.skip_tags` cannot reach them and bulk links must say `--no-embed` themselves; and there is no `statement-update`, so the choice is made once at creation.
173
+
135
174
  ### Triple Extraction Patterns
136
175
 
137
176
  | User says | head | relation | tail |
@@ -341,19 +380,25 @@ hypatia search <query> [-c <catalog>] [--limit N] [--offset N]
341
380
  Semantic search using embedding vectors. Finds entries with similar meaning, even when keywords don't match.
342
381
 
343
382
  ```
344
- hypatia similar <query> [--limit N]
383
+ hypatia similar <query> [--limit N] [-t knowledge|statement|both] [--tags <a,b>] [--exclude-tags <a,b>] [--where '<JSE condition>']
345
384
  ```
346
385
 
386
+ `--tags` keeps entries carrying at least one of the tags, `--exclude-tags` drops entries carrying any of them, and `--where` takes a JSE condition such as `["$contains", "scopes", "project-a"]` (`$eq`, `$like`, `$contains`, `$and`/`$or`/`$not` and the other filters `$knowledge` takes, but not `$search`, `$similar` or `$k-hop`). They narrow the entries before ranking, so `--limit N` still returns N entries whenever that many qualify. `--exclude-tags message,summary,session` keeps the session-log layer from crowding out the knowledge distilled from it. With the default target `both` the filter applies to statements too, so a condition on `name` needs `-t knowledge`.
387
+
347
388
  Requires an embedding model configured in `shelf.toml` (default: BAAI/bge-m3).
348
389
 
349
390
  Entries are embedded in batches after they are written or updated. With a remote embedding API, `similar` can miss entries changed in the last minute; `search` finds them at once.
350
391
 
392
+ Entries written with `--no-embed`, and entries carrying a tag the shelf lists in `embedding.skip_tags`, are never embedded and so are never `similar` results. `search` and `query` still find them. Each entry keeps the answer it was written with, so after changing `skip_tags` run `hypatia backfill` once to settle what is already stored.
393
+
351
394
  ### Examples
352
395
 
353
396
  | User says | Command |
354
397
  |---|---|
355
398
  | "find similar to distributed systems" | `hypatia similar "distributed systems"` |
356
399
  | "semantic search for memory management" | `hypatia similar "memory management" --limit 5` |
400
+ | "what do we know about auth, not the chat logs" | `hypatia similar "auth" --exclude-tags message,summary,session` |
401
+ | "rules like this one for project-a" | `hypatia similar "<rule text>" -t knowledge --tags rule --where '["$contains", "scopes", "project-a"]'` |
357
402
 
358
403
  ## JSE Query Translation
359
404
 
@@ -559,15 +604,17 @@ even read-only SQLite needs to create WAL/journal temp files next to the databas
559
604
  go straight to the immutable read-only URI, which needs no journal and no directory writes:
560
605
 
561
606
  ```bash
562
- sqlite3 "file:$HOME/.hypatia/default/hypatia.sqlite?mode=ro&immutable=1" "SELECT count(*) FROM knowledge;"
607
+ sqlite3 "file:<shelf-dir>/hypatia.sqlite?mode=ro&immutable=1" "SELECT count(*) FROM knowledge;"
563
608
  ```
564
609
 
565
610
  - Use only for **reads** (SELECT). Never write through this path — `immutable=1` tells SQLite the
566
611
  file never changes, so writes would corrupt state or be lost.
567
- - Adjust the shelf path for non-default shelves (`~/.hypatia/<shelf>/<db>.sqlite`).
568
- - First, locate the actual DB file: `ls ~/.hypatia/*/` (also check `-wal`/`-shm` siblings; if a
569
- WAL exists and is non-empty, `immutable=1` may miss recent un-checkpointed rows — in that case
570
- prefer requesting wider sandbox permissions for a normal read instead).
612
+ - First, get `<shelf-dir>` from `hypatia list`, which prints the directory each shelf is open at.
613
+ Do not assume `~/.hypatia/<shelf>/`: any shelf, `default` included, can be registered elsewhere.
614
+ If the CLI itself cannot run, read the paths from `~/.hypatia/shelves.json`.
615
+ - Then check the `-wal`/`-shm` siblings; if a WAL exists and is non-empty, `immutable=1` may miss
616
+ recent un-checkpointed rows — in that case prefer requesting wider sandbox permissions for a
617
+ normal read instead.
571
618
  - Schema hints: tables include `knowledge` and `statement`; statement triples are stored in
572
619
  `head` / `relation` / `tail` columns.
573
620
 
@@ -25,6 +25,11 @@ Do not do these by hand — you would only create duplicates:
25
25
  - **Session seed.** Project and global entries tagged `rule` or `taboo` are
26
26
  injected into your context at session start. Nothing else is pushed to you.
27
27
 
28
+ The log layer — `msg-*`, `session-*`, and the `belongTo` / `summary` edges —
29
+ is written **without vectors** (`--no-embed`) on a hypatia that has the flag.
30
+ It is stored and full-text searchable, but `similar` does not rank it. Entries
31
+ logged before that, or by an older hypatia, still carry vectors.
32
+
28
33
  ## Which shelf
29
34
 
30
35
  The plugin writes to one hypatia shelf, chosen in its settings. When that is
@@ -47,7 +52,8 @@ costs a wrong answer that contradicts a decision already made.
47
52
  ### Semantic search — the default
48
53
 
49
54
  ```bash
50
- hypatia similar "<what you actually want to know>" -t knowledge --limit 20
55
+ hypatia similar "<what you actually want to know>" -t knowledge --limit 5 \
56
+ --exclude-tags message,summary,session,hypatia-dream-run
51
57
  ```
52
58
 
53
59
  Write the query as the *idea* you are looking for, not the user's words verbatim.
@@ -55,8 +61,11 @@ Rows come back as `{"name", "content" (object), "distance"}`; lower `distance` i
55
61
  closer. If it exits non-zero with `model unavailable` / `no embedding provider`,
56
62
  this shelf has no embedding model — fall back to `search` and carry on.
57
63
 
58
- The limit is deliberately about four times the rows you want: most of what comes
59
- back will be the operational layer, which you then drop (see below).
64
+ `--exclude-tags` leaves the operational layer out *before* ranking, so five rows
65
+ asked for are five rows of knowledge. Add `--where '["$contains","scopes","<PROJECT>"]'`
66
+ to stay inside this project. If the flag is rejected as an `unexpected argument`,
67
+ the hypatia is older than #35: drop it, ask for `--limit 20`, and discard the
68
+ operational rows yourself (see below).
60
69
 
61
70
  ### Keyword search — for exact identifiers
62
71
 
@@ -79,10 +88,21 @@ hypatia query '["$knowledge", ["$contains","tags","rule"],
79
88
  hypatia query '["$statement", ["$k-hop", "<entry-name>", "$*", 2]]'
80
89
  ```
81
90
 
82
- `<PROJECT>` is the basename of the workspace's git root (of the workspace
83
- itself outside git), written as the plugin writes it: commas become `_`,
84
- surrounding whitespace is dropped, and the filesystem root is `/`. Written raw,
85
- a name with a comma would be split into two scopes.
91
+ `<PROJECT>` is the scope the plugin writes this workspace under: the basename
92
+ of its git root (of the workspace itself outside git), with commas turned into
93
+ `_`, surrounding whitespace dropped, and the filesystem root written `/`. You
94
+ do not have to derive it — the plugin has already logged this session under it,
95
+ so it is on the shelf:
96
+
97
+ ```bash
98
+ hypatia scope list --count # every scope in use, with how many entries carry it
99
+ hypatia scope exists "<PROJECT>" # exit 0 if in use, 1 if not
100
+ ```
101
+
102
+ The plain listing prints the global scope as `(global)`; that is a label, the
103
+ value is `""`. Use `--json` when you will write a value back. Both need hypatia
104
+ #30; an older one answers `unrecognized subcommand`, and then the rule above is
105
+ all there is.
86
106
 
87
107
  ### Always exclude the operational layer
88
108
 
@@ -92,13 +112,14 @@ name starts with `msg-`, `sum`, `session-`, or `hypatia-dream-run-`, and rows
92
112
  tagged `message`, `session`, `summary`, or `hypatia-dream-run`. Read `wu-*`,
93
113
  `rule`, `taboo`, and ordinary named entries.
94
114
 
95
- Neither `similar` nor `search` can filter by name or tag, and that layer is most
96
- of the shelf — every message is logged. It also matches well: a raw `msg-*`
97
- holds the very words of the conversation you are asking about. Measured on a
98
- live shelf, 7 of the 10 nearest `similar` rows were operational, so a
99
- `--limit 5` search filtered afterwards leaves one row or none. Hence `--limit 20`. If too few rows
100
- survive the filter, search again with a larger limit or a different wording
101
- before concluding that nothing is stored.
115
+ That layer is most of the shelf — every message is logged — and it matches
116
+ well: a raw `msg-*` holds the very words of the conversation you are asking
117
+ about. Measured on a live shelf, 7 of the 10 nearest `similar` rows were
118
+ operational. `similar --exclude-tags` handles it; `search` has no such flag, and
119
+ neither can see an operational entry that lost its tags, so still check names.
120
+ Whenever you filter after the fact, ask for `--limit 20`, and if too few rows
121
+ survive, search again with a larger limit or a different wording before
122
+ concluding that nothing is stored.
102
123
 
103
124
  Reach into `msg-*` only when the user asks what was literally said, and prefer
104
125
  reaching it by walking `summary` statements down from a `sum-*` entry.
@@ -119,8 +140,14 @@ hypatia knowledge-create "<short-kebab-name>" \
119
140
  --scopes "<PROJECT>"
120
141
  ```
121
142
 
143
+ - **Write the scope the plugin uses**, exactly. The session seed loads rules by
144
+ that spelling; a rule under `my-app` is invisible to a plugin that logs under
145
+ `my_app`. `hypatia scope list --count` shows it — the scope this project's
146
+ `msg-*` entries carry — and `hypatia scope exists "<PROJECT>"` confirms it
147
+ before you write.
122
148
  - `--tags rule` for conventions to follow, `taboo` for never-do items,
123
- `memory` for durable facts.
149
+ `memory` for durable facts. Beyond those three, reuse what `hypatia tag list`
150
+ already shows before inventing a label.
124
151
  - **Scope syntax matters.** `--scopes "<PROJECT>"` is project-only.
125
152
  `--scopes "<PROJECT>,"` — with a *trailing comma* — also marks it global.
126
153
  `--scopes ""` writes no scope at all, which is not the same as global and will
package/src/cascade.js CHANGED
@@ -179,8 +179,12 @@ export function createCascade({ cli, llm, selectRoute, getConfig, status }) {
179
179
  // under a different name and the first entry keeps only its partial set.
180
180
  // Nothing is lost — every member still ends up archived exactly once — but
181
181
  // the grouping boundary is not the one the first attempt intended.
182
+ // The archive entry is embedded — it is distilled knowledge, and the entry
183
+ // point for drilling down. Its edges are not: they are for the traversal
184
+ // and for `$not-summaried`, and nobody looks for a `summary` edge by
185
+ // meaning (see writer.js `writeSummary`).
182
186
  for (const member of members) {
183
- await cli.statementCreate(name, 'summary', member, { scopes: [task.project] })
187
+ await cli.statementCreate(name, 'summary', member, { scopes: [task.project], embed: false })
184
188
  }
185
189
  status.info(`cascade: tier ${level} archived ${members.length} entries as ${name}`)
186
190
  }
@@ -172,6 +172,56 @@ export function createHypatiaCli(ctx, config) {
172
172
  }
173
173
  }
174
174
 
175
+ /**
176
+ * What this binary can do beyond the baseline every call relies on.
177
+ *
178
+ * Read off `--help`, the one place clap states a flag's existence: an older
179
+ * binary answers an unknown flag with a usage error (exit 2) *instead of*
180
+ * running the command, so a write that guessed at `--no-embed` would fail
181
+ * outright on a binary from before hypatia #26, and a `similar` guessing at
182
+ * `--exclude-tags` on one from before #35. Both write commands are asked
183
+ * about `--no-embed` — they gained it together, but the answer is only worth
184
+ * having if it is true of each command it gates.
185
+ *
186
+ * @returns {Promise<{noEmbed: boolean, similarFilters: boolean}>}
187
+ */
188
+ async function probeFeatures() {
189
+ const [knowledge, statement, similar] = await Promise.all([
190
+ run(['knowledge-create', '--help']),
191
+ run(['statement-create', '--help']),
192
+ run(['similar', '--help']),
193
+ ])
194
+ for (const result of [knowledge, statement, similar]) {
195
+ if (result.exitCode !== 0) {
196
+ throw new HypatiaCliError(`hypatia --help exited ${result.exitCode}`, {
197
+ code: 'NONZERO_EXIT',
198
+ stderr: result.stderr.trim().slice(0, 500),
199
+ exitCode: result.exitCode,
200
+ })
201
+ }
202
+ }
203
+ return {
204
+ noEmbed: knowledge.stdout.includes('--no-embed') && statement.stdout.includes('--no-embed'),
205
+ similarFilters: similar.stdout.includes('--exclude-tags'),
206
+ }
207
+ }
208
+
209
+ /** The probe's answer, kept for the life of the client; a probe that failed is asked again. */
210
+ let featuresProbe
211
+
212
+ function features() {
213
+ if (featuresProbe === undefined) {
214
+ featuresProbe = probeFeatures().catch(() => {
215
+ // A binary that cannot even print its help fails the next real call
216
+ // with its own error; the answer here only decides which flags that
217
+ // call carries, and the conservative answer is none.
218
+ featuresProbe = undefined
219
+ return { noEmbed: false, similarFilters: false }
220
+ })
221
+ }
222
+ return featuresProbe
223
+ }
224
+
175
225
  /**
176
226
  * Whether stdout is hypatia's empty-result sentinel.
177
227
  *
@@ -196,6 +246,7 @@ export function createHypatiaCli(ctx, config) {
196
246
  return {
197
247
  run,
198
248
  runOk,
249
+ features,
199
250
 
200
251
  /**
201
252
  * Read one knowledge entry. Missing entries are a normal result, not an
@@ -211,7 +262,11 @@ export function createHypatiaCli(ctx, config) {
211
262
  },
212
263
 
213
264
  /**
214
- * @param {{data: string, tags?: string[], scopes?: string[], shelf?: string}} entry
265
+ * `embed: false` keeps the entry out of the vector index (hypatia #26); on
266
+ * a binary without the flag it is dropped, and the entry is embedded as it
267
+ * always was.
268
+ *
269
+ * @param {{data: string, tags?: string[], scopes?: string[], embed?: boolean, shelf?: string}} entry
215
270
  * @returns {Promise<boolean>} true when this call created the entry.
216
271
  */
217
272
  async knowledgeCreate(name, entry) {
@@ -222,11 +277,15 @@ export function createHypatiaCli(ctx, config) {
222
277
  if (entry.data) argv.push(`--data=${fitDataArgument(entry.data)}`)
223
278
  if (entry.tags && entry.tags.length > 0) argv.push('--tags', entry.tags.join(','))
224
279
  if (entry.scopes && entry.scopes.length > 0) argv.push('--scopes', entry.scopes.join(','))
280
+ if (entry.embed === false && (await features()).noEmbed) argv.push('--no-embed')
225
281
  return runCreate(argv)
226
282
  },
227
283
 
228
284
  /**
229
- * @param {{data?: string, scopes?: string[], shelf?: string}} [entry]
285
+ * `embed: false` as on `knowledgeCreate`. A statement has no update
286
+ * command, so the choice is made once, here.
287
+ *
288
+ * @param {{data?: string, scopes?: string[], embed?: boolean, shelf?: string}} [entry]
230
289
  * @returns {Promise<boolean>} true when this call created the triple.
231
290
  */
232
291
  async statementCreate(head, relation, tail, entry = {}) {
@@ -239,6 +298,7 @@ export function createHypatiaCli(ctx, config) {
239
298
  // expose to clap's flag parsing.
240
299
  if (entry.data) argv.push(`--data=${fitDataArgument(entry.data)}`)
241
300
  if (entry.scopes && entry.scopes.length > 0) argv.push('--scopes', entry.scopes.join(','))
301
+ if (entry.embed === false && (await features()).noEmbed) argv.push('--no-embed')
242
302
  return runCreate(argv)
243
303
  },
244
304
 
@@ -258,10 +318,18 @@ export function createHypatiaCli(ctx, config) {
258
318
  /**
259
319
  * Vector-similarity search — the recall path's primary ranking tool.
260
320
  * NOTE the flag is `-t/--target` (unlike keyword search's `-c`).
321
+ *
322
+ * `excludeTags` leaves entries carrying any of those tags out BEFORE the
323
+ * ranking (hypatia #35), so `limit` rows come back whenever that many
324
+ * qualify. A binary without the flag ranks everything; the caller reads
325
+ * `features()` to know whether it must over-fetch and filter instead.
326
+ *
261
327
  * @returns {Promise<any[]>}
262
328
  */
263
- async similar(query, { target = 'knowledge', limit = 5, shelf = shelfOf() } = {}) {
264
- const { stdout } = await runOk(['similar', query, '-t', target, '--limit', String(limit), '--shelf', shelf])
329
+ async similar(query, { target = 'knowledge', limit = 5, shelf = shelfOf(), excludeTags = [] } = {}) {
330
+ const argv = ['similar', query, '-t', target, '--limit', String(limit), '--shelf', shelf]
331
+ if (excludeTags.length > 0 && (await features()).similarFilters) argv.push('--exclude-tags', excludeTags.join(','))
332
+ const { stdout } = await runOk(argv)
265
333
  if (isEmptyResult(stdout)) return []
266
334
  const parsed = parseJson(stdout, 'similar')
267
335
  return Array.isArray(parsed) ? parsed : []
@@ -54,7 +54,21 @@ export function createHypatiaClient(ctx, config, { status, connection }) {
54
54
  }
55
55
  }
56
56
 
57
+ /** Logged once, so the profile log says which flags this run's writes carry. */
58
+ let announced = false
59
+ async function features() {
60
+ const found = await via('features', [])
61
+ if (!announced) {
62
+ announced = true
63
+ const yes = (flag) => (flag ? 'yes' : 'no')
64
+ status.info?.(`hypatia over ${usesMcp() ? 'mcp' : 'cli'}: no-embed ${yes(found.noEmbed)}, similar filters ${yes(found.similarFilters)}`)
65
+ }
66
+ return found
67
+ }
68
+
57
69
  return {
70
+ /** What the binary can do beyond the baseline (see hypatia-cli.js `features`). */
71
+ features,
58
72
  knowledgeGet: (...args) => via('knowledgeGet', args),
59
73
  knowledgeCreate: (...args) => via('knowledgeCreate', args),
60
74
  statementCreate: (...args) => via('statementCreate', args),
@@ -50,6 +50,27 @@ export const REQUIRED_TOOLS = Object.freeze([
50
50
  'query',
51
51
  ])
52
52
 
53
+ /**
54
+ * What a server can do beyond REQUIRED_TOOLS, read off the schemas in its
55
+ * `tools/list`. Every tool declares `additionalProperties: false`, so an
56
+ * argument a server does not list fails the call as a validation error rather
57
+ * than being ignored — which is why the plugin asks first instead of trying.
58
+ * The names mirror hypatia-cli.js's `features()`.
59
+ *
60
+ * @param {Array<{name?: string, inputSchema?: {properties?: object}}>} tools
61
+ * @returns {{noEmbed: boolean, similarFilters: boolean}}
62
+ */
63
+ export function featuresOf(tools) {
64
+ const accepts = (name, argument) => Object.hasOwn(
65
+ tools.find((tool) => tool?.name === name)?.inputSchema?.properties ?? {},
66
+ argument,
67
+ )
68
+ return {
69
+ noEmbed: accepts('knowledge_create', 'embed') && accepts('statement_create', 'embed'),
70
+ similarFilters: accepts('similar', 'exclude_tags'),
71
+ }
72
+ }
73
+
53
74
  /**
54
75
  * One request line. A lone UTF-16 surrogate — what a `.slice()` through an
55
76
  * emoji leaves behind — is written by `JSON.stringify` as a bare `\ud83d`
@@ -255,11 +276,13 @@ export function createMcpConnection(ctx, options) {
255
276
  }, handshakeTimeoutMs)
256
277
  conn.handle.stdin.write(encode({ jsonrpc: '2.0', method: 'notifications/initialized' }))
257
278
  const listed = await request(conn, 'tools/list', {}, handshakeTimeoutMs)
258
- const names = new Set((listed?.tools ?? []).map((tool) => tool?.name))
279
+ const tools = listed?.tools ?? []
280
+ const names = new Set(tools.map((tool) => tool?.name))
259
281
  const missing = REQUIRED_TOOLS.filter((name) => !names.has(name))
260
282
  if (missing.length > 0) {
261
283
  throw new HypatiaCliError(`hypatia mcp lacks ${missing.join(', ')}`, { code: 'MCP_UNAVAILABLE' })
262
284
  }
285
+ conn.features = featuresOf(tools)
263
286
  } catch (error) {
264
287
  kill(conn, error)
265
288
  if (!isUnavailable(error)) throw error
@@ -282,6 +305,37 @@ export function createMcpConnection(ctx, options) {
282
305
  idleTimer.unref?.()
283
306
  }
284
307
 
308
+ /**
309
+ * Run `step` against a live, handshaken process, after the steps before it.
310
+ * @template T
311
+ * @param {(conn: object) => Promise<T>} step
312
+ * @returns {Promise<T>}
313
+ */
314
+ function withProcess(step) {
315
+ const run = async () => {
316
+ if (disposed) throw new HypatiaCliError('hypatia mcp connection disposed', { code: 'MCP_CLOSED' })
317
+ if (unavailable !== undefined) throw unavailable
318
+ clearTimeout(idleTimer)
319
+ try {
320
+ if (current === undefined || current.closed) {
321
+ const conn = await open()
322
+ // Disposed during the handshake: nothing else would close it.
323
+ if (disposed) {
324
+ close(conn)
325
+ throw new HypatiaCliError('hypatia mcp connection disposed', { code: 'MCP_CLOSED' })
326
+ }
327
+ current = conn
328
+ }
329
+ return await step(current)
330
+ } finally {
331
+ if (!disposed) armIdle()
332
+ }
333
+ }
334
+ const result = chain.then(run, run)
335
+ chain = result.catch(() => {})
336
+ return result
337
+ }
338
+
285
339
  return {
286
340
  /**
287
341
  * One `tools/call`, after the calls before it.
@@ -290,31 +344,21 @@ export function createMcpConnection(ctx, options) {
290
344
  * @returns {Promise<any>} the MCP `CallToolResult`, `isError` included.
291
345
  */
292
346
  call(name, args) {
293
- const run = async () => {
294
- if (disposed) throw new HypatiaCliError('hypatia mcp connection disposed', { code: 'MCP_CLOSED' })
295
- if (unavailable !== undefined) throw unavailable
296
- clearTimeout(idleTimer)
297
- try {
298
- if (current === undefined || current.closed) {
299
- const conn = await open()
300
- // Disposed during the handshake: nothing else would close it.
301
- if (disposed) {
302
- close(conn)
303
- throw new HypatiaCliError('hypatia mcp connection disposed', { code: 'MCP_CLOSED' })
304
- }
305
- current = conn
306
- }
307
- // An error quotes what this call made the process write, not what
308
- // earlier calls did. The handshake's errors quote everything.
309
- current.stderrMark = current.handle.collected.stderr?.readFrom(0).nextOffset ?? 0
310
- return await request(current, 'tools/call', { name, arguments: args }, timeoutMs)
311
- } finally {
312
- if (!disposed) armIdle()
313
- }
314
- }
315
- const result = chain.then(run, run)
316
- chain = result.catch(() => {})
317
- return result
347
+ return withProcess((conn) => {
348
+ // An error quotes what this call made the process write, not what
349
+ // earlier calls did. The handshake's errors quote everything.
350
+ conn.stderrMark = conn.handle.collected.stderr?.readFrom(0).nextOffset ?? 0
351
+ return request(conn, 'tools/call', { name, arguments: args }, timeoutMs)
352
+ })
353
+ },
354
+
355
+ /**
356
+ * What the server can do beyond REQUIRED_TOOLS (see `featuresOf`), from
357
+ * the handshake — starting the process if none is running.
358
+ * @returns {Promise<{noEmbed: boolean, similarFilters: boolean}>}
359
+ */
360
+ features() {
361
+ return withProcess((conn) => Promise.resolve(conn.features))
318
362
  },
319
363
 
320
364
  /** Drop the process; the next call starts a fresh one. */
@@ -438,23 +482,32 @@ export function createHypatiaMcp(connection, config) {
438
482
  }
439
483
  },
440
484
 
485
+ features: () => connection.features(),
486
+
441
487
  /**
442
- * @param {{data: string, tags?: string[], scopes?: string[], shelf?: string}} entry
488
+ * `embed: false` keeps the entry out of the vector index (hypatia #26); a
489
+ * server whose `knowledge_create` does not list the argument would reject
490
+ * it, so there it is dropped and the entry is embedded as before.
491
+ *
492
+ * @param {{data: string, tags?: string[], scopes?: string[], embed?: boolean, shelf?: string}} entry
443
493
  * @returns {Promise<boolean>} true when this call created the entry.
444
494
  */
445
495
  async knowledgeCreate(name, entry) {
446
496
  const args = { name, shelf: entry.shelf ?? shelfOf() }
447
497
  if (entry.data) args.data = entry.data
498
+ if (entry.embed === false && (await connection.features()).noEmbed) args.embed = false
448
499
  return (await create('knowledge_create', lists(args, entry))) !== undefined
449
500
  },
450
501
 
451
502
  /**
452
- * @param {{data?: string, scopes?: string[], shelf?: string}} [entry]
503
+ * `embed: false` as on `knowledgeCreate`.
504
+ * @param {{data?: string, scopes?: string[], embed?: boolean, shelf?: string}} [entry]
453
505
  * @returns {Promise<boolean>} true when this call created the triple.
454
506
  */
455
507
  async statementCreate(head, relation, tail, entry = {}) {
456
508
  const args = { head, relation, tail, shelf: entry.shelf ?? shelfOf() }
457
509
  if (entry.data) args.data = entry.data
510
+ if (entry.embed === false && (await connection.features()).noEmbed) args.embed = false
458
511
  const outcome = await create('statement_create', lists(args, { scopes: entry.scopes }))
459
512
  return outcome !== undefined && outcome.created !== false
460
513
  },
@@ -464,9 +517,17 @@ export function createHypatiaMcp(connection, config) {
464
517
  return rowsOf(await invoke('search', { query, catalog, limit, shelf }))
465
518
  },
466
519
 
467
- /** Vector-similarity search; rows as the CLI prints them. */
468
- async similar(query, { target = 'knowledge', limit = 5, shelf = shelfOf() } = {}) {
469
- return rowsOf(await invoke('similar', { query, target, limit, shelf }))
520
+ /**
521
+ * Vector-similarity search; rows as the CLI prints them. `excludeTags`
522
+ * narrows the entries before they are ranked (hypatia #35) and is dropped
523
+ * on a server that does not list it, as on the CLI.
524
+ */
525
+ async similar(query, { target = 'knowledge', limit = 5, shelf = shelfOf(), excludeTags = [] } = {}) {
526
+ const args = { query, target, limit, shelf }
527
+ if (excludeTags.length > 0 && (await connection.features()).similarFilters) {
528
+ args.exclude_tags = asCliList(excludeTags)
529
+ }
530
+ return rowsOf(await invoke('similar', args))
470
531
  },
471
532
 
472
533
  /** Raw JSE query; `jse` is the JSON text the CLI would take as its argument. */
package/src/writer.js CHANGED
@@ -29,11 +29,19 @@ const OPERATIONAL_PREFIXES = ['msg-', 'sum', 'session-', 'hypatia-dream-run-']
29
29
  const OPERATIONAL_TAGS = new Set(['message', 'session', 'hypatia-dream-run'])
30
30
 
31
31
  /**
32
- * How many rows to ask `similar` for per candidate actually wanted.
32
+ * The same layer as tags `similar` can be told to leave out (hypatia #35).
33
+ * `summary` alone covers every summary tier: each carries it beside its
34
+ * `summary <N>` tag.
35
+ */
36
+ export const EXCLUDED_TAGS = Object.freeze([...OPERATIONAL_TAGS, 'summary'])
37
+
38
+ /**
39
+ * How many rows to ask `similar` for per candidate actually wanted, on a
40
+ * binary whose `similar` cannot exclude tags.
33
41
  *
34
- * The operational layer cannot be excluded in the query (see `findCandidates`),
35
- * outnumbers knowledge on an active shelf, and ranks nearer, so the fetch has
36
- * to absorb it. Four covers the measured 7-in-10 ratio with room to spare.
42
+ * There the operational layer is ranked along with everything else, outnumbers
43
+ * knowledge on an active shelf, and ranks nearer, so the fetch has to absorb
44
+ * it. Four covers the measured 7-in-10 ratio with room to spare.
37
45
  */
38
46
  const CANDIDATE_OVERSAMPLE = 4
39
47
 
@@ -113,6 +121,13 @@ export function createWriter(cli, { status, adjudicate }) {
113
121
  * from a replayed task) are counted and skipped — the watermark must still
114
122
  * advance past them, which the queue does based on the resolved range.
115
123
  *
124
+ * Written with `embed: false`: the log layer is not on the retrieval hot
125
+ * path — `docs/memory-nolinear.md` has precise recall drill down from a
126
+ * summary along `summary` edges — and embedded, a raw message outranks the
127
+ * knowledge distilled from it, because it holds the very words. It stays
128
+ * stored and full-text searchable; only the vector is not made. On a binary
129
+ * without `--no-embed` the flag is dropped by the transport.
130
+ *
116
131
  * @param {{sessionId: string, index: number, markdown: string, project: string}} entry
117
132
  */
118
133
  async function writeMessage({ sessionId, index, markdown, project }) {
@@ -122,7 +137,7 @@ export function createWriter(cli, { status, adjudicate }) {
122
137
  status.count('duplicate')
123
138
  return { name, written: false }
124
139
  }
125
- await cli.knowledgeCreate(name, { data: markdown, tags: ['message'], scopes: [project] })
140
+ await cli.knowledgeCreate(name, { data: markdown, tags: ['message'], scopes: [project], embed: false })
126
141
  status.count('written')
127
142
  return { name, written: true }
128
143
  }
@@ -147,6 +162,11 @@ export function createWriter(cli, { status, adjudicate }) {
147
162
  * permanent. `statementCreate` swallows primary-key collisions, so
148
163
  * re-asserting an existing edge is free.
149
164
  *
165
+ * The summary itself is embedded — it is distilled knowledge and the entry
166
+ * point for drilling down. Its edges are written `embed: false`: they exist
167
+ * for the traversal and for `$not-summaried`, and statements carry no tags,
168
+ * so a shelf's `embedding.skip_tags` could not reach them.
169
+ *
150
170
  * @param {{sessionId: string, fromSeq: number, toSeq: number, markdown: string,
151
171
  * project: string, items?: readonly string[], level?: number}} entry
152
172
  * `level` is the archive tier this summary sits at (1 = summarises messages).
@@ -168,7 +188,7 @@ export function createWriter(cli, { status, adjudicate }) {
168
188
  status.count('written')
169
189
  }
170
190
  for (const item of items) {
171
- await cli.statementCreate(name, 'summary', item, { scopes: [project] })
191
+ await cli.statementCreate(name, 'summary', item, { scopes: [project], embed: false })
172
192
  }
173
193
  return { name, written: existing.found === false, links: items.length }
174
194
  }
@@ -184,6 +204,10 @@ export function createWriter(cli, { status, adjudicate }) {
184
204
  * hypatia has no `knowledge-update`, so a second title cannot replace the
185
205
  * first; the original stands and the caller is told nothing was written.
186
206
  *
207
+ * Node and `belongTo` edges are written `embed: false`, as the messages are:
208
+ * the node is bookkeeping for the traversal, and its title is what
209
+ * `session/title` already gave DSH.
210
+ *
187
211
  * @param {{sessionId: string, markdown: string, project: string,
188
212
  * linkFrom: number, linkTo: number}} entry - `[linkFrom, linkTo)` is
189
213
  * the half-open range of message ordinals still needing a `belongTo` edge.
@@ -200,11 +224,11 @@ export function createWriter(cli, { status, adjudicate }) {
200
224
  // it absent and drop the edges with it.
201
225
  return { name, written: false, links: 0 }
202
226
  } else {
203
- await cli.knowledgeCreate(name, { data: markdown, tags: ['session'], scopes: [project] })
227
+ await cli.knowledgeCreate(name, { data: markdown, tags: ['session'], scopes: [project], embed: false })
204
228
  status.count('written')
205
229
  }
206
230
  for (let index = linkFrom; index < linkTo; index += 1) {
207
- await cli.statementCreate(messageName(sessionId, index), 'belongTo', name, { scopes: [project] })
231
+ await cli.statementCreate(messageName(sessionId, index), 'belongTo', name, { scopes: [project], embed: false })
208
232
  }
209
233
  return { name, written: existing.found === false, links: Math.max(0, linkTo - linkFrom) }
210
234
  }
@@ -220,19 +244,24 @@ export function createWriter(cli, { status, adjudicate }) {
220
244
  */
221
245
  async function findCandidates(unit, name, { maxDistance, limit }) {
222
246
  try {
223
- // OVERSAMPLED, because the exclusion below cannot be pushed into the
224
- // query: `hypatia similar` takes only target/limit/shelf, no tag or scope
225
- // filter. The operational layer is the majority of an active shelf (101
226
- // of 183 knowledge entries on the profile this was measured on) and it
227
- // scores CLOSER than real knowledge — a raw `msg-*` holds the very words
228
- // of the conversation the unit was extracted from. Measured: 7 of the
229
- // nearest 10 rows were operational, so asking for 5 and filtering
230
- // afterwards left one candidate, often none, and adjudication silently
231
- // degraded to "no relationship" on every write.
247
+ // The operational layer is excluded IN the query where the binary allows
248
+ // it (hypatia #35), and over-fetched around where it does not. It is the
249
+ // majority of an active shelf (101 of 183 knowledge entries on the
250
+ // profile this was measured on) and it scores CLOSER than real knowledge
251
+ // — a raw `msg-*` holds the very words of the conversation the unit was
252
+ // extracted from. Measured: 7 of the nearest 10 rows were operational, so
253
+ // asking for 5 and filtering afterwards left one candidate, often none,
254
+ // and adjudication silently degraded to "no relationship" on every write.
255
+ // Entries this plugin now writes `embed: false` are not ranked at all,
256
+ // but a shelf holds what earlier versions and other writers embedded.
257
+ const { similarFilters } = await cli.features()
232
258
  const rows = await cli.similar(`${unit.title}\n${unit.content}`.slice(0, 500), {
233
259
  target: 'knowledge',
234
- limit: Math.min(limit * CANDIDATE_OVERSAMPLE, CANDIDATE_FETCH_CEILING),
260
+ limit: similarFilters ? limit : Math.min(limit * CANDIDATE_OVERSAMPLE, CANDIDATE_FETCH_CEILING),
261
+ excludeTags: EXCLUDED_TAGS,
235
262
  })
263
+ // The name-prefix half of `isOperationalRow` still applies: a tag filter
264
+ // cannot see an operational entry that lost its tags.
236
265
  return rows
237
266
  .filter((row) => rowName(row) !== name)
238
267
  .filter((row) => !isOperationalRow(row))