dsh-hypatia-auto-memory 0.1.1 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +38 -6
- package/lib/client.js +31 -31
- package/package.json +5 -3
- package/skills/hypatia/SKILL.md +56 -9
- package/skills/hypatia-memory/SKILL.md +42 -15
- package/src/cascade.js +5 -1
- package/src/hypatia-cli.js +72 -4
- package/src/hypatia-client.js +14 -0
- package/src/hypatia-mcp.js +92 -31
- package/src/writer.js +47 -18
package/README.md
CHANGED
|
@@ -24,7 +24,12 @@ ships here: auto-approval for the agent's own bash `hypatia` calls, and its
|
|
|
24
24
|
|
|
25
25
|
Prerequisite: `hypatia` CLI on PATH (or add its basename to `binaries`). A
|
|
26
26
|
build with the `mcp` subcommand (hypatia #20) is used over MCP; an older one
|
|
27
|
-
works through the CLI instead, with one warning at the first call.
|
|
27
|
+
works through the CLI instead, with one warning at the first call. Two later
|
|
28
|
+
additions are used when the binary has them and skipped when it does not:
|
|
29
|
+
`--no-embed` (hypatia #26) keeps the log layer out of the vector index, and
|
|
30
|
+
`similar --exclude-tags` (#35) keeps it out of the work-unit candidate search.
|
|
31
|
+
The first call logs which the binary offers
|
|
32
|
+
(`hypatia over mcp: no-embed yes, similar filters yes`).
|
|
28
33
|
|
|
29
34
|
```bash
|
|
30
35
|
# from a local checkout (development)
|
|
@@ -129,6 +134,19 @@ agent/session-start ──▶ rules/taboos inject()
|
|
|
129
134
|
`(tool calls only)`. Plugin-sourced
|
|
130
135
|
messages are skipped — no feedback loops.
|
|
131
136
|
|
|
137
|
+
- **The log layer is not embedded.** `msg-*` and `session-*` entries and the
|
|
138
|
+
`belongTo` / `summary` edges are written `embed: false` (hypatia #26): stored,
|
|
139
|
+
full-text indexed and reachable by JSE, but given no vector. The protocol has
|
|
140
|
+
precise recall drill down from a summary along `summary` edges rather than
|
|
141
|
+
hit raw messages by meaning, and embedded, a raw message outranks the
|
|
142
|
+
knowledge distilled from it because it holds the very words. On the shelf
|
|
143
|
+
this was measured on the layer was 85% of entries and 77% of statements, each
|
|
144
|
+
costing a forward pass on the conversation's hot path. Summaries and work
|
|
145
|
+
units are embedded as before. Statements have no tags, so a shelf's
|
|
146
|
+
`embedding.skip_tags` could not have reached the edges; the plugin says so per
|
|
147
|
+
write. On a binary without the flag the transport drops it and everything is
|
|
148
|
+
embedded as it always was.
|
|
149
|
+
|
|
132
150
|
Spans are cut at `turn/end`, and — for a turn that outlasts `flushWindowMs` —
|
|
133
151
|
at `step/end`, never at an arbitrary moment. DSH appends `assistant/message`
|
|
134
152
|
*before* running the tools it requested and `step/end` only after they have
|
|
@@ -164,8 +182,10 @@ agent/session-start ──▶ rules/taboos inject()
|
|
|
164
182
|
previous tier already distilled.
|
|
165
183
|
|
|
166
184
|
- **Work units** are adjudicated, not guessed. Candidates come from `similar`
|
|
167
|
-
(the only search that reports a distance),
|
|
168
|
-
|
|
185
|
+
(the only search that reports a distance), with the operational layer left
|
|
186
|
+
out — in the query, by `--exclude-tags`, on a binary that has it (hypatia
|
|
187
|
+
#35); by fetching four times as many rows and dropping them on one that does
|
|
188
|
+
not — and filtered by a distance ceiling, then a small model call
|
|
169
189
|
decides `duplicate | refines | extends | supersedes | contradicts | unrelated`.
|
|
170
190
|
A contradiction keeps **both** entries and records `supersedes` — a memory
|
|
171
191
|
system must not quietly forget what it once believed.
|
|
@@ -335,7 +355,12 @@ changes. No card claims it, so it renders nowhere; nothing ever writes to it.
|
|
|
335
355
|
(get-before-create); uncovered ranges are re-enqueued from watermarks.
|
|
336
356
|
4. To check the route warning: consolidation with no selected `models`
|
|
337
357
|
logs a one-time warning and otherwise stays silent.
|
|
338
|
-
5.
|
|
358
|
+
5. `hypatia backfill --status` reports the shelf's embedding debt. With a
|
|
359
|
+
current hypatia, logging a turn adds nothing to it; only summaries and work
|
|
360
|
+
units are pending until the next flush. `hypatia scope list --count` shows
|
|
361
|
+
the scopes in use — the one this project's `msg-*` entries carry is the one
|
|
362
|
+
the session seed reads.
|
|
363
|
+
6. Uninstall: `dsh plugin --profile web remove dsh-hypatia-auto-memory` —
|
|
339
364
|
hypatia entries themselves are left in `~/.hypatia/`.
|
|
340
365
|
|
|
341
366
|
## Failure modes
|
|
@@ -347,7 +372,8 @@ changes. No card claims it, so it renders nowhere; nothing ever writes to it.
|
|
|
347
372
|
| Entries appear only after a turn finishes | By design: spans are cut at `turn/end`, or at the first `step/end` once a turn has run longer than `flushWindowMs`, so an entry never lacks its own tool results |
|
|
348
373
|
| Logging works, no summaries | `consolidation.models` is empty or invalid — one warning at first trigger |
|
|
349
374
|
| Summaries but no `sum2-*` | Fewer than `cascade.batchSize` unarchived tier-1 summaries in that project yet |
|
|
350
|
-
| Work units have no relationships | No embedding model on the shelf (`similar` fails), or every candidate was beyond `dedupMaxDistance` |
|
|
375
|
+
| Work units have no relationships | No embedding model on the shelf (`similar` fails), or every candidate was beyond `dedupMaxDistance`. Since hypatia #19 the local model is looked up in `~/.hypatia/models/<org>/<name>` (or the Hugging Face cache), no longer beside the shelf: a shelf that used to answer `similar` and now says `is not installed` needs `hypatia model install <model>`, or `hypatia model register <model> <dir>` pointing at the files it already has |
|
|
376
|
+
| `similar` still returns `msg-*` rows | They were written before this plugin opted the log layer out of embedding, or by a hypatia without `--no-embed`. Retract the knowledge vectors once with `embedding.skip_tags = ["message", "session"]` in the shelf's `shelf.toml` followed by `hypatia backfill` (a binary older than the key refuses to open the shelf, so upgrade every binary sharing it first). Statements have no tags and no update command, so `belongTo` / `summary` edges written before keep their vectors |
|
|
351
377
|
| Task in `deferred` state | Neither the live store nor persistence could supply its session. Not an error: it spends no attempts and runs as soon as one of them can. Normally storage answers immediately — the state persists only when `sessionPersistence` is absent from the composition, or its read failed (look for `could not be read` in the log) |
|
|
352
378
|
| Task in `failed` state | A hypatia or model error persisted after `maxAttempts`. Failed `log-message` records are pruned at the next startup, since the watermark re-derives their range; other kinds are kept for inspection — delete one to let the next trigger re-create it |
|
|
353
379
|
| Watermark says logged, but the shelf has no entries | The shelf was reset or switched after logging. Handled at startup: `housekeeping.reconcileOnStartup` resets any row whose session has no `msg-*` left, and that session is re-logged — and re-consolidated — from the start on its next activity. A shelf query that fails leaves the row untouched. A session whose messages were all deleted on purpose is indistinguishable and is logged again; turn the switch off if that matters |
|
|
@@ -359,7 +385,7 @@ changes. No card claims it, so it renders nowhere; nothing ever writes to it.
|
|
|
359
385
|
| The agent's own `hypatia` write still asks for approval | `autoApprove: false` (needs a profile reload to change), the command pipes/redirects/chains outside quotes, or its first word is not one of `binaries` — only plain calls are answered, by design. Reads never reach approval at all, so nothing to fix there |
|
|
360
386
|
| The agent got a memory protocol that tells it to log messages by hand | Another plugin registered `hypatia-memory` first (`dsh-hypatia` still in the profile), or a `hypatia-memory` exists on disk — typically `~/.agents/skills/hypatia-memory`, written by `hypatia skill install --agent codex`. Agent presets load disk skills in a layer nearer than plugins, so it wins in sessions even though the skill center may list this plugin. Remove the copy the startup warning names |
|
|
361
387
|
| Startup warnings never appear in the terminal | `dsh web` mounts no log exporter, so plugin log lines of any level go nowhere; the console exporter's default threshold would also drop warnings (warn is level 2, above info's 1). Mount a logger such as `dsh-logbook` or `dsh-boot-doctor` temporarily to read them |
|
|
362
|
-
| Project scope looks wrong | Scope = basename of the session cwd's git top level (`git rev-parse --show-toplevel`), or of the cwd itself when git finds no work tree or cannot answer (not installed, a repo it refuses as unsafe). A `rev-parse` slower than 3 s leaves that one session on the cwd's own name. Two same-named checkouts share a scope by design; a linked worktree is scoped by its own directory, not the main checkout's; git resolves symlinks, so a checkout opened through a link named differently from its target gets the target's name. A name hypatia would rewrite is normalized first: a session at `/` is scoped `/`, commas become `_`, surrounding whitespace goes. Entries written before these fixes stay where they were stored: a subdirectory session's under that directory's name (`repo/src` wrote under `src` — the git root was never read), a session at `/` with no scope, `a,b` under both `a` and `b`, `foo,` under `foo` and global. Padded names were stored trimmed, which is what the query now asks for. Nothing migrates them; `knowledge-update --scopes` moves one entry and keeps its `created_at` |
|
|
388
|
+
| Project scope looks wrong | Scope = basename of the session cwd's git top level (`git rev-parse --show-toplevel`), or of the cwd itself when git finds no work tree or cannot answer (not installed, a repo it refuses as unsafe). A `rev-parse` slower than 3 s leaves that one session on the cwd's own name. Two same-named checkouts share a scope by design; a linked worktree is scoped by its own directory, not the main checkout's; git resolves symlinks, so a checkout opened through a link named differently from its target gets the target's name. A name hypatia would rewrite is normalized first: a session at `/` is scoped `/`, commas become `_`, surrounding whitespace goes. Entries written before these fixes stay where they were stored: a subdirectory session's under that directory's name (`repo/src` wrote under `src` — the git root was never read), a session at `/` with no scope, `a,b` under both `a` and `b`, `foo,` under `foo` and global. Padded names were stored trimmed, which is what the query now asks for. Nothing migrates them; `knowledge-update --scopes` moves one entry and keeps its `created_at`. `hypatia scope list --count` (hypatia #30) shows every spelling in use and how many entries each holds |
|
|
363
389
|
|
|
364
390
|
## Known limitations
|
|
365
391
|
|
|
@@ -392,6 +418,12 @@ Deliberate, and worth knowing before you rely on them:
|
|
|
392
418
|
exits 0 with `Statement already exists`); on an older binary the same
|
|
393
419
|
error-text path covers it. The matching is confined to one function, and
|
|
394
420
|
`test/integration` accepts either statement behaviour.
|
|
421
|
+
- **What was embedded stays embedded.** The log layer opts out of the vector
|
|
422
|
+
index at write time, so entries and edges written before this version, or by
|
|
423
|
+
a hypatia without `--no-embed`, keep their vectors and keep surfacing in
|
|
424
|
+
`similar`. `skip_tags` + `backfill` retracts the knowledge entries' (see
|
|
425
|
+
*Failure modes*); the edges have no retraction path short of deleting and
|
|
426
|
+
recreating them, which the plugin does not do.
|
|
395
427
|
- **Adjudication and dedup need an embedding model.** On a shelf without one,
|
|
396
428
|
`similar` fails outright and work units are stored with no relationship —
|
|
397
429
|
never dropped.
|
package/lib/client.js
CHANGED
|
@@ -108,46 +108,46 @@ window.__ModuleLoader__.load({
|
|
|
108
108
|
document.head.appendChild(tag);
|
|
109
109
|
}
|
|
110
110
|
var SettingsCard_module_css_default = {
|
|
111
|
-
"headText": "YVNYya_headText",
|
|
112
|
-
"chevronOpen": "YVNYya_chevronOpen",
|
|
113
|
-
"chevron": "YVNYya_chevron",
|
|
114
|
-
"model": "YVNYya_model",
|
|
115
|
-
"catalogError": "YVNYya_catalogError",
|
|
116
|
-
"groupTitle": "YVNYya_groupTitle",
|
|
117
|
-
"input": "YVNYya_input",
|
|
118
|
-
"header": "YVNYya_header",
|
|
119
|
-
"group": "YVNYya_group",
|
|
120
|
-
"advancedHint": "YVNYya_advancedHint",
|
|
121
111
|
"hint": "YVNYya_hint",
|
|
122
|
-
"
|
|
123
|
-
"switch": "YVNYya_switch",
|
|
112
|
+
"field": "YVNYya_field",
|
|
124
113
|
"readOnly": "YVNYya_readOnly",
|
|
125
|
-
"fieldHead": "YVNYya_fieldHead",
|
|
126
|
-
"switchOn": "YVNYya_switchOn",
|
|
127
|
-
"reset": "YVNYya_reset",
|
|
128
114
|
"select": "YVNYya_select",
|
|
129
|
-
"card": "YVNYya_card",
|
|
130
|
-
"body": "YVNYya_body",
|
|
131
|
-
"cardOpen": "YVNYya_cardOpen",
|
|
132
|
-
"modelName": "YVNYya_modelName",
|
|
133
|
-
"limitsTitle": "YVNYya_limitsTitle",
|
|
134
|
-
"label": "YVNYya_label",
|
|
135
|
-
"pairedFields": "YVNYya_pairedFields",
|
|
136
115
|
"switchThumb": "YVNYya_switchThumb",
|
|
116
|
+
"pairedFields": "YVNYya_pairedFields",
|
|
117
|
+
"fieldHead": "YVNYya_fieldHead",
|
|
118
|
+
"modelName": "YVNYya_modelName",
|
|
137
119
|
"status": "YVNYya_status",
|
|
120
|
+
"switchOn": "YVNYya_switchOn",
|
|
121
|
+
"headText": "YVNYya_headText",
|
|
122
|
+
"chevron": "YVNYya_chevron",
|
|
123
|
+
"footer": "YVNYya_footer",
|
|
124
|
+
"unavailable": "YVNYya_unavailable",
|
|
125
|
+
"modelSelection": "YVNYya_modelSelection",
|
|
126
|
+
"chevronOpen": "YVNYya_chevronOpen",
|
|
127
|
+
"body": "YVNYya_body",
|
|
128
|
+
"error": "YVNYya_error",
|
|
129
|
+
"modelGroup": "YVNYya_modelGroup",
|
|
130
|
+
"card": "YVNYya_card",
|
|
131
|
+
"warning": "YVNYya_warning",
|
|
132
|
+
"switch": "YVNYya_switch",
|
|
138
133
|
"notice": "YVNYya_notice",
|
|
139
|
-
"
|
|
134
|
+
"limitsTitle": "YVNYya_limitsTitle",
|
|
135
|
+
"reset": "YVNYya_reset",
|
|
136
|
+
"description": "YVNYya_description",
|
|
137
|
+
"group": "YVNYya_group",
|
|
138
|
+
"name": "YVNYya_name",
|
|
139
|
+
"advancedHint": "YVNYya_advancedHint",
|
|
140
140
|
"models": "YVNYya_models",
|
|
141
|
+
"catalogError": "YVNYya_catalogError",
|
|
142
|
+
"label": "YVNYya_label",
|
|
141
143
|
"providerName": "YVNYya_providerName",
|
|
142
|
-
"
|
|
143
|
-
"description": "YVNYya_description",
|
|
144
|
-
"error": "YVNYya_error",
|
|
145
|
-
"warning": "YVNYya_warning",
|
|
146
|
-
"route": "YVNYya_route",
|
|
147
|
-
"unavailable": "YVNYya_unavailable",
|
|
148
|
-
"footer": "YVNYya_footer",
|
|
144
|
+
"cardOpen": "YVNYya_cardOpen",
|
|
149
145
|
"pending": "YVNYya_pending",
|
|
150
|
-
"
|
|
146
|
+
"route": "YVNYya_route",
|
|
147
|
+
"groupTitle": "YVNYya_groupTitle",
|
|
148
|
+
"model": "YVNYya_model",
|
|
149
|
+
"input": "YVNYya_input",
|
|
150
|
+
"header": "YVNYya_header"
|
|
151
151
|
};
|
|
152
152
|
//#endregion
|
|
153
153
|
//#region src/client/locales.ts
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-hypatia-auto-memory",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.2.0",
|
|
4
4
|
"description": "Event-driven Hypatia memory for DSH: automatic conversation logging via session events, queued background consolidation (span summaries + work-unit extraction) with a dedicated model route, and recall injection — zero per-call model work in the main session.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
|
@@ -45,13 +45,13 @@
|
|
|
45
45
|
"node": "^22.19.0 || >=24.0.0"
|
|
46
46
|
},
|
|
47
47
|
"dependencies": {
|
|
48
|
-
"@deepseek-ai/dsh-llm": "^0.1.2-rc.1",
|
|
49
|
-
"@deepseek-ai/dsh-storage-domain": "^0.1.2-rc.1",
|
|
50
48
|
"@deepseek-ai/schemastery": "^3.18.2",
|
|
51
49
|
"zod": "^4.4.3"
|
|
52
50
|
},
|
|
53
51
|
"peerDependencies": {
|
|
54
52
|
"@deepseek-ai/cordis": "^4.0.2",
|
|
53
|
+
"@deepseek-ai/dsh-llm": "^0.1.2-rc.1",
|
|
54
|
+
"@deepseek-ai/dsh-storage-domain": "^0.1.2-rc.1",
|
|
55
55
|
"react": "^18.2.0"
|
|
56
56
|
},
|
|
57
57
|
"devDependencies": {
|
|
@@ -64,6 +64,8 @@
|
|
|
64
64
|
"@deepseek-ai/dsh-client-ui-settings": "^0.1.2-rc.1",
|
|
65
65
|
"@deepseek-ai/dsh-client-ui-settings-plugins": "^0.1.2-rc.1",
|
|
66
66
|
"@deepseek-ai/dsh-client-ui-slots": "^0.1.2-rc.1",
|
|
67
|
+
"@deepseek-ai/dsh-llm": "^0.1.2-rc.1",
|
|
68
|
+
"@deepseek-ai/dsh-storage-domain": "^0.1.2-rc.1",
|
|
67
69
|
"@types/react": "^18.2.0",
|
|
68
70
|
"lightningcss": "^1.32.0",
|
|
69
71
|
"react": "^18.2.0",
|
package/skills/hypatia/SKILL.md
CHANGED
|
@@ -68,6 +68,38 @@ hypatia knowledge-create "prefer immutable" -d "always create new objects" --tag
|
|
|
68
68
|
hypatia knowledge-create "no mock DB" -d "never mock database in tests" --tags "taboo" --scopes "my-project,"
|
|
69
69
|
```
|
|
70
70
|
|
|
71
|
+
### Listing the scopes and tags already in use
|
|
72
|
+
|
|
73
|
+
A scope or tag you invent is not an error — the entry is stored, and every later
|
|
74
|
+
lookup by the value you meant misses it. Check before you write:
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
# What this shelf already uses, with how many entries carry each
|
|
78
|
+
hypatia scope list --count
|
|
79
|
+
hypatia tag list --count
|
|
80
|
+
|
|
81
|
+
# Exact values, when you are going to write one back
|
|
82
|
+
hypatia scope list --json # [{"value":"","entries":12},{"value":"my-project",…}]
|
|
83
|
+
|
|
84
|
+
# Confirm one spelling; exit 0 if in use, 1 if not
|
|
85
|
+
hypatia tag exists rule
|
|
86
|
+
hypatia scope exists my-project
|
|
87
|
+
hypatia scope exists "" # the global scope
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
The plain listing prints the global scope as `(global)`. That is a label for the
|
|
91
|
+
terminal, not the value — the value is the empty string, and `--json` and
|
|
92
|
+
`exists` both give it back verbatim. Never copy `(global)` into `--scopes`.
|
|
93
|
+
|
|
94
|
+
Both cover knowledge and statements. An entry that declares no scopes at all is
|
|
95
|
+
not listed as global.
|
|
96
|
+
|
|
97
|
+
| User says | Command |
|
|
98
|
+
|---|---|
|
|
99
|
+
| "what scopes/projects are in here?" | `hypatia scope list --count` |
|
|
100
|
+
| "what tags exist?" / "which labels are used?" | `hypatia tag list --count` |
|
|
101
|
+
| "is there already a `rule` tag?" | `hypatia tag exists rule` |
|
|
102
|
+
|
|
71
103
|
## Knowledge CRUD
|
|
72
104
|
|
|
73
105
|
Knowledge entries are independent information points with a name, content, and tags.
|
|
@@ -75,14 +107,17 @@ Knowledge entries are independent information points with a name, content, and t
|
|
|
75
107
|
### Create
|
|
76
108
|
|
|
77
109
|
```
|
|
78
|
-
hypatia knowledge-create <name> -d "<data>" -t "<tag1,tag2>" --figures "archive://path/to/file"
|
|
110
|
+
hypatia knowledge-create <name> -d "<data>" -t "<tag1,tag2>" --figures "archive://path/to/file" [--no-embed]
|
|
79
111
|
```
|
|
80
112
|
|
|
113
|
+
`--no-embed` keeps the entry out of the vector index: it is still stored and still found by `search` and `query`, but it is never embedded, so it costs no model call on the way in and is not a `similar` result. Use it for raw session logs and other bulk that carries no distilled knowledge. A shelf can skip whole tags instead, with `embedding.skip_tags` in `shelf.toml`.
|
|
114
|
+
|
|
81
115
|
| User says | Command |
|
|
82
116
|
|---|---|
|
|
83
117
|
| "remember Rust as a systems programming language" | `hypatia knowledge-create "Rust" -d "systems programming language" -t "language,compiled"` |
|
|
84
118
|
| "save knowledge about Go with tags language and compiled" | `hypatia knowledge-create "Go" -d "" -t "language,compiled"` |
|
|
85
119
|
| "store that Python is a scripting language, tag it as dynamic" | `hypatia knowledge-create "Python" -d "scripting language" -t "dynamic"` |
|
|
120
|
+
| "log this turn but don't make it searchable by meaning" | `hypatia knowledge-create "msg-42" -d "<turn>" -t "message" --no-embed` |
|
|
86
121
|
|
|
87
122
|
### Read
|
|
88
123
|
|
|
@@ -97,9 +132,11 @@ hypatia knowledge-get <name>
|
|
|
97
132
|
### Update
|
|
98
133
|
|
|
99
134
|
```
|
|
100
|
-
hypatia knowledge-update <name> [-d "<data>"] [-t "<tags>"] [--synonyms "<csv>"] [--figures "<refs>"] [--scopes "<scopes>"]
|
|
135
|
+
hypatia knowledge-update <name> [-d "<data>"] [-t "<tags>"] [--synonyms "<csv>"] [--figures "<refs>"] [--scopes "<scopes>"] [--no-embed | --embed]
|
|
101
136
|
```
|
|
102
137
|
|
|
138
|
+
`--no-embed` takes the entry out of the vector index and discards the vector it had; `--embed` puts it back, and it is embedded on the next flush. The two are mutually exclusive. `--embed` does not override the shelf's `embedding.skip_tags`: an entry can ask for less indexing than its shelf, never more.
|
|
139
|
+
|
|
103
140
|
Only the fields you pass change. An omitted field keeps its stored value, and exactly `""` clears a field, such as `-t ""`; for tags, `","` or `" "` would store blank tags instead. `--scopes` replaces the stored scopes and is parsed as on create: end the list with a comma, such as `"q,"`, to include the global scope, or the entry drops out of global lookups. The entry keeps its `created_at`. Its old vector is discarded, and a new one is generated on the next flush, as after `knowledge-create`. Updating an entry that does not exist is an error. An update that changes nothing prints `Knowledge unchanged: <name>` and writes nothing.
|
|
104
141
|
|
|
105
142
|
| User says | Command |
|
|
@@ -129,9 +166,11 @@ When the user asks to store or remember information, **always create statements
|
|
|
129
166
|
**Pattern**: After creating a knowledge entry, identify entities mentioned in the content and create `$triple` relationships between them. At minimum, create one `is_a` statement for every new knowledge entry.
|
|
130
167
|
|
|
131
168
|
```
|
|
132
|
-
hypatia statement-create <head> <relation> <tail> -d "<data>"
|
|
169
|
+
hypatia statement-create <head> <relation> <tail> -d "<data>" [--no-embed]
|
|
133
170
|
```
|
|
134
171
|
|
|
172
|
+
`--no-embed` works as it does on `knowledge-create`: the triple is still stored and still traversable, but gets no vector. Two differences: statements carry no tags, so a shelf's `embedding.skip_tags` cannot reach them and bulk links must say `--no-embed` themselves; and there is no `statement-update`, so the choice is made once at creation.
|
|
173
|
+
|
|
135
174
|
### Triple Extraction Patterns
|
|
136
175
|
|
|
137
176
|
| User says | head | relation | tail |
|
|
@@ -341,19 +380,25 @@ hypatia search <query> [-c <catalog>] [--limit N] [--offset N]
|
|
|
341
380
|
Semantic search using embedding vectors. Finds entries with similar meaning, even when keywords don't match.
|
|
342
381
|
|
|
343
382
|
```
|
|
344
|
-
hypatia similar <query> [--limit N]
|
|
383
|
+
hypatia similar <query> [--limit N] [-t knowledge|statement|both] [--tags <a,b>] [--exclude-tags <a,b>] [--where '<JSE condition>']
|
|
345
384
|
```
|
|
346
385
|
|
|
386
|
+
`--tags` keeps entries carrying at least one of the tags, `--exclude-tags` drops entries carrying any of them, and `--where` takes a JSE condition such as `["$contains", "scopes", "project-a"]` (`$eq`, `$like`, `$contains`, `$and`/`$or`/`$not` and the other filters `$knowledge` takes, but not `$search`, `$similar` or `$k-hop`). They narrow the entries before ranking, so `--limit N` still returns N entries whenever that many qualify. `--exclude-tags message,summary,session` keeps the session-log layer from crowding out the knowledge distilled from it. With the default target `both` the filter applies to statements too, so a condition on `name` needs `-t knowledge`.
|
|
387
|
+
|
|
347
388
|
Requires an embedding model configured in `shelf.toml` (default: BAAI/bge-m3).
|
|
348
389
|
|
|
349
390
|
Entries are embedded in batches after they are written or updated. With a remote embedding API, `similar` can miss entries changed in the last minute; `search` finds them at once.
|
|
350
391
|
|
|
392
|
+
Entries written with `--no-embed`, and entries carrying a tag the shelf lists in `embedding.skip_tags`, are never embedded and so are never `similar` results. `search` and `query` still find them. Each entry keeps the answer it was written with, so after changing `skip_tags` run `hypatia backfill` once to settle what is already stored.
|
|
393
|
+
|
|
351
394
|
### Examples
|
|
352
395
|
|
|
353
396
|
| User says | Command |
|
|
354
397
|
|---|---|
|
|
355
398
|
| "find similar to distributed systems" | `hypatia similar "distributed systems"` |
|
|
356
399
|
| "semantic search for memory management" | `hypatia similar "memory management" --limit 5` |
|
|
400
|
+
| "what do we know about auth, not the chat logs" | `hypatia similar "auth" --exclude-tags message,summary,session` |
|
|
401
|
+
| "rules like this one for project-a" | `hypatia similar "<rule text>" -t knowledge --tags rule --where '["$contains", "scopes", "project-a"]'` |
|
|
357
402
|
|
|
358
403
|
## JSE Query Translation
|
|
359
404
|
|
|
@@ -559,15 +604,17 @@ even read-only SQLite needs to create WAL/journal temp files next to the databas
|
|
|
559
604
|
go straight to the immutable read-only URI, which needs no journal and no directory writes:
|
|
560
605
|
|
|
561
606
|
```bash
|
|
562
|
-
sqlite3 "file
|
|
607
|
+
sqlite3 "file:<shelf-dir>/hypatia.sqlite?mode=ro&immutable=1" "SELECT count(*) FROM knowledge;"
|
|
563
608
|
```
|
|
564
609
|
|
|
565
610
|
- Use only for **reads** (SELECT). Never write through this path — `immutable=1` tells SQLite the
|
|
566
611
|
file never changes, so writes would corrupt state or be lost.
|
|
567
|
-
-
|
|
568
|
-
|
|
569
|
-
|
|
570
|
-
|
|
612
|
+
- First, get `<shelf-dir>` from `hypatia list`, which prints the directory each shelf is open at.
|
|
613
|
+
Do not assume `~/.hypatia/<shelf>/`: any shelf, `default` included, can be registered elsewhere.
|
|
614
|
+
If the CLI itself cannot run, read the paths from `~/.hypatia/shelves.json`.
|
|
615
|
+
- Then check the `-wal`/`-shm` siblings; if a WAL exists and is non-empty, `immutable=1` may miss
|
|
616
|
+
recent un-checkpointed rows — in that case prefer requesting wider sandbox permissions for a
|
|
617
|
+
normal read instead.
|
|
571
618
|
- Schema hints: tables include `knowledge` and `statement`; statement triples are stored in
|
|
572
619
|
`head` / `relation` / `tail` columns.
|
|
573
620
|
|
|
@@ -25,6 +25,11 @@ Do not do these by hand — you would only create duplicates:
|
|
|
25
25
|
- **Session seed.** Project and global entries tagged `rule` or `taboo` are
|
|
26
26
|
injected into your context at session start. Nothing else is pushed to you.
|
|
27
27
|
|
|
28
|
+
The log layer — `msg-*`, `session-*`, and the `belongTo` / `summary` edges —
|
|
29
|
+
is written **without vectors** (`--no-embed`) on a hypatia that has the flag.
|
|
30
|
+
It is stored and full-text searchable, but `similar` does not rank it. Entries
|
|
31
|
+
logged before that, or by an older hypatia, still carry vectors.
|
|
32
|
+
|
|
28
33
|
## Which shelf
|
|
29
34
|
|
|
30
35
|
The plugin writes to one hypatia shelf, chosen in its settings. When that is
|
|
@@ -47,7 +52,8 @@ costs a wrong answer that contradicts a decision already made.
|
|
|
47
52
|
### Semantic search — the default
|
|
48
53
|
|
|
49
54
|
```bash
|
|
50
|
-
hypatia similar "<what you actually want to know>" -t knowledge --limit
|
|
55
|
+
hypatia similar "<what you actually want to know>" -t knowledge --limit 5 \
|
|
56
|
+
--exclude-tags message,summary,session,hypatia-dream-run
|
|
51
57
|
```
|
|
52
58
|
|
|
53
59
|
Write the query as the *idea* you are looking for, not the user's words verbatim.
|
|
@@ -55,8 +61,11 @@ Rows come back as `{"name", "content" (object), "distance"}`; lower `distance` i
|
|
|
55
61
|
closer. If it exits non-zero with `model unavailable` / `no embedding provider`,
|
|
56
62
|
this shelf has no embedding model — fall back to `search` and carry on.
|
|
57
63
|
|
|
58
|
-
|
|
59
|
-
|
|
64
|
+
`--exclude-tags` leaves the operational layer out *before* ranking, so five rows
|
|
65
|
+
asked for are five rows of knowledge. Add `--where '["$contains","scopes","<PROJECT>"]'`
|
|
66
|
+
to stay inside this project. If the flag is rejected as an `unexpected argument`,
|
|
67
|
+
the hypatia is older than #35: drop it, ask for `--limit 20`, and discard the
|
|
68
|
+
operational rows yourself (see below).
|
|
60
69
|
|
|
61
70
|
### Keyword search — for exact identifiers
|
|
62
71
|
|
|
@@ -79,10 +88,21 @@ hypatia query '["$knowledge", ["$contains","tags","rule"],
|
|
|
79
88
|
hypatia query '["$statement", ["$k-hop", "<entry-name>", "$*", 2]]'
|
|
80
89
|
```
|
|
81
90
|
|
|
82
|
-
`<PROJECT>` is the
|
|
83
|
-
|
|
84
|
-
surrounding whitespace
|
|
85
|
-
|
|
91
|
+
`<PROJECT>` is the scope the plugin writes this workspace under: the basename
|
|
92
|
+
of its git root (of the workspace itself outside git), with commas turned into
|
|
93
|
+
`_`, surrounding whitespace dropped, and the filesystem root written `/`. You
|
|
94
|
+
do not have to derive it — the plugin has already logged this session under it,
|
|
95
|
+
so it is on the shelf:
|
|
96
|
+
|
|
97
|
+
```bash
|
|
98
|
+
hypatia scope list --count # every scope in use, with how many entries carry it
|
|
99
|
+
hypatia scope exists "<PROJECT>" # exit 0 if in use, 1 if not
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
The plain listing prints the global scope as `(global)`; that is a label, the
|
|
103
|
+
value is `""`. Use `--json` when you will write a value back. Both need hypatia
|
|
104
|
+
#30; an older one answers `unrecognized subcommand`, and then the rule above is
|
|
105
|
+
all there is.
|
|
86
106
|
|
|
87
107
|
### Always exclude the operational layer
|
|
88
108
|
|
|
@@ -92,13 +112,14 @@ name starts with `msg-`, `sum`, `session-`, or `hypatia-dream-run-`, and rows
|
|
|
92
112
|
tagged `message`, `session`, `summary`, or `hypatia-dream-run`. Read `wu-*`,
|
|
93
113
|
`rule`, `taboo`, and ordinary named entries.
|
|
94
114
|
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
115
|
+
That layer is most of the shelf — every message is logged — and it matches
|
|
116
|
+
well: a raw `msg-*` holds the very words of the conversation you are asking
|
|
117
|
+
about. Measured on a live shelf, 7 of the 10 nearest `similar` rows were
|
|
118
|
+
operational. `similar --exclude-tags` handles it; `search` has no such flag, and
|
|
119
|
+
neither can see an operational entry that lost its tags, so still check names.
|
|
120
|
+
Whenever you filter after the fact, ask for `--limit 20`, and if too few rows
|
|
121
|
+
survive, search again with a larger limit or a different wording before
|
|
122
|
+
concluding that nothing is stored.
|
|
102
123
|
|
|
103
124
|
Reach into `msg-*` only when the user asks what was literally said, and prefer
|
|
104
125
|
reaching it by walking `summary` statements down from a `sum-*` entry.
|
|
@@ -119,8 +140,14 @@ hypatia knowledge-create "<short-kebab-name>" \
|
|
|
119
140
|
--scopes "<PROJECT>"
|
|
120
141
|
```
|
|
121
142
|
|
|
143
|
+
- **Write the scope the plugin uses**, exactly. The session seed loads rules by
|
|
144
|
+
that spelling; a rule under `my-app` is invisible to a plugin that logs under
|
|
145
|
+
`my_app`. `hypatia scope list --count` shows it — the scope this project's
|
|
146
|
+
`msg-*` entries carry — and `hypatia scope exists "<PROJECT>"` confirms it
|
|
147
|
+
before you write.
|
|
122
148
|
- `--tags rule` for conventions to follow, `taboo` for never-do items,
|
|
123
|
-
`memory` for durable facts.
|
|
149
|
+
`memory` for durable facts. Beyond those three, reuse what `hypatia tag list`
|
|
150
|
+
already shows before inventing a label.
|
|
124
151
|
- **Scope syntax matters.** `--scopes "<PROJECT>"` is project-only.
|
|
125
152
|
`--scopes "<PROJECT>,"` — with a *trailing comma* — also marks it global.
|
|
126
153
|
`--scopes ""` writes no scope at all, which is not the same as global and will
|
package/src/cascade.js
CHANGED
|
@@ -179,8 +179,12 @@ export function createCascade({ cli, llm, selectRoute, getConfig, status }) {
|
|
|
179
179
|
// under a different name and the first entry keeps only its partial set.
|
|
180
180
|
// Nothing is lost — every member still ends up archived exactly once — but
|
|
181
181
|
// the grouping boundary is not the one the first attempt intended.
|
|
182
|
+
// The archive entry is embedded — it is distilled knowledge, and the entry
|
|
183
|
+
// point for drilling down. Its edges are not: they are for the traversal
|
|
184
|
+
// and for `$not-summaried`, and nobody looks for a `summary` edge by
|
|
185
|
+
// meaning (see writer.js `writeSummary`).
|
|
182
186
|
for (const member of members) {
|
|
183
|
-
await cli.statementCreate(name, 'summary', member, { scopes: [task.project] })
|
|
187
|
+
await cli.statementCreate(name, 'summary', member, { scopes: [task.project], embed: false })
|
|
184
188
|
}
|
|
185
189
|
status.info(`cascade: tier ${level} archived ${members.length} entries as ${name}`)
|
|
186
190
|
}
|
package/src/hypatia-cli.js
CHANGED
|
@@ -172,6 +172,56 @@ export function createHypatiaCli(ctx, config) {
|
|
|
172
172
|
}
|
|
173
173
|
}
|
|
174
174
|
|
|
175
|
+
/**
|
|
176
|
+
* What this binary can do beyond the baseline every call relies on.
|
|
177
|
+
*
|
|
178
|
+
* Read off `--help`, the one place clap states a flag's existence: an older
|
|
179
|
+
* binary answers an unknown flag with a usage error (exit 2) *instead of*
|
|
180
|
+
* running the command, so a write that guessed at `--no-embed` would fail
|
|
181
|
+
* outright on a binary from before hypatia #26, and a `similar` guessing at
|
|
182
|
+
* `--exclude-tags` on one from before #35. Both write commands are asked
|
|
183
|
+
* about `--no-embed` — they gained it together, but the answer is only worth
|
|
184
|
+
* having if it is true of each command it gates.
|
|
185
|
+
*
|
|
186
|
+
* @returns {Promise<{noEmbed: boolean, similarFilters: boolean}>}
|
|
187
|
+
*/
|
|
188
|
+
async function probeFeatures() {
|
|
189
|
+
const [knowledge, statement, similar] = await Promise.all([
|
|
190
|
+
run(['knowledge-create', '--help']),
|
|
191
|
+
run(['statement-create', '--help']),
|
|
192
|
+
run(['similar', '--help']),
|
|
193
|
+
])
|
|
194
|
+
for (const result of [knowledge, statement, similar]) {
|
|
195
|
+
if (result.exitCode !== 0) {
|
|
196
|
+
throw new HypatiaCliError(`hypatia --help exited ${result.exitCode}`, {
|
|
197
|
+
code: 'NONZERO_EXIT',
|
|
198
|
+
stderr: result.stderr.trim().slice(0, 500),
|
|
199
|
+
exitCode: result.exitCode,
|
|
200
|
+
})
|
|
201
|
+
}
|
|
202
|
+
}
|
|
203
|
+
return {
|
|
204
|
+
noEmbed: knowledge.stdout.includes('--no-embed') && statement.stdout.includes('--no-embed'),
|
|
205
|
+
similarFilters: similar.stdout.includes('--exclude-tags'),
|
|
206
|
+
}
|
|
207
|
+
}
|
|
208
|
+
|
|
209
|
+
/** The probe's answer, kept for the life of the client; a probe that failed is asked again. */
|
|
210
|
+
let featuresProbe
|
|
211
|
+
|
|
212
|
+
function features() {
|
|
213
|
+
if (featuresProbe === undefined) {
|
|
214
|
+
featuresProbe = probeFeatures().catch(() => {
|
|
215
|
+
// A binary that cannot even print its help fails the next real call
|
|
216
|
+
// with its own error; the answer here only decides which flags that
|
|
217
|
+
// call carries, and the conservative answer is none.
|
|
218
|
+
featuresProbe = undefined
|
|
219
|
+
return { noEmbed: false, similarFilters: false }
|
|
220
|
+
})
|
|
221
|
+
}
|
|
222
|
+
return featuresProbe
|
|
223
|
+
}
|
|
224
|
+
|
|
175
225
|
/**
|
|
176
226
|
* Whether stdout is hypatia's empty-result sentinel.
|
|
177
227
|
*
|
|
@@ -196,6 +246,7 @@ export function createHypatiaCli(ctx, config) {
|
|
|
196
246
|
return {
|
|
197
247
|
run,
|
|
198
248
|
runOk,
|
|
249
|
+
features,
|
|
199
250
|
|
|
200
251
|
/**
|
|
201
252
|
* Read one knowledge entry. Missing entries are a normal result, not an
|
|
@@ -211,7 +262,11 @@ export function createHypatiaCli(ctx, config) {
|
|
|
211
262
|
},
|
|
212
263
|
|
|
213
264
|
/**
|
|
214
|
-
*
|
|
265
|
+
* `embed: false` keeps the entry out of the vector index (hypatia #26); on
|
|
266
|
+
* a binary without the flag it is dropped, and the entry is embedded as it
|
|
267
|
+
* always was.
|
|
268
|
+
*
|
|
269
|
+
* @param {{data: string, tags?: string[], scopes?: string[], embed?: boolean, shelf?: string}} entry
|
|
215
270
|
* @returns {Promise<boolean>} true when this call created the entry.
|
|
216
271
|
*/
|
|
217
272
|
async knowledgeCreate(name, entry) {
|
|
@@ -222,11 +277,15 @@ export function createHypatiaCli(ctx, config) {
|
|
|
222
277
|
if (entry.data) argv.push(`--data=${fitDataArgument(entry.data)}`)
|
|
223
278
|
if (entry.tags && entry.tags.length > 0) argv.push('--tags', entry.tags.join(','))
|
|
224
279
|
if (entry.scopes && entry.scopes.length > 0) argv.push('--scopes', entry.scopes.join(','))
|
|
280
|
+
if (entry.embed === false && (await features()).noEmbed) argv.push('--no-embed')
|
|
225
281
|
return runCreate(argv)
|
|
226
282
|
},
|
|
227
283
|
|
|
228
284
|
/**
|
|
229
|
-
*
|
|
285
|
+
* `embed: false` as on `knowledgeCreate`. A statement has no update
|
|
286
|
+
* command, so the choice is made once, here.
|
|
287
|
+
*
|
|
288
|
+
* @param {{data?: string, scopes?: string[], embed?: boolean, shelf?: string}} [entry]
|
|
230
289
|
* @returns {Promise<boolean>} true when this call created the triple.
|
|
231
290
|
*/
|
|
232
291
|
async statementCreate(head, relation, tail, entry = {}) {
|
|
@@ -239,6 +298,7 @@ export function createHypatiaCli(ctx, config) {
|
|
|
239
298
|
// expose to clap's flag parsing.
|
|
240
299
|
if (entry.data) argv.push(`--data=${fitDataArgument(entry.data)}`)
|
|
241
300
|
if (entry.scopes && entry.scopes.length > 0) argv.push('--scopes', entry.scopes.join(','))
|
|
301
|
+
if (entry.embed === false && (await features()).noEmbed) argv.push('--no-embed')
|
|
242
302
|
return runCreate(argv)
|
|
243
303
|
},
|
|
244
304
|
|
|
@@ -258,10 +318,18 @@ export function createHypatiaCli(ctx, config) {
|
|
|
258
318
|
/**
|
|
259
319
|
* Vector-similarity search — the recall path's primary ranking tool.
|
|
260
320
|
* NOTE the flag is `-t/--target` (unlike keyword search's `-c`).
|
|
321
|
+
*
|
|
322
|
+
* `excludeTags` leaves entries carrying any of those tags out BEFORE the
|
|
323
|
+
* ranking (hypatia #35), so `limit` rows come back whenever that many
|
|
324
|
+
* qualify. A binary without the flag ranks everything; the caller reads
|
|
325
|
+
* `features()` to know whether it must over-fetch and filter instead.
|
|
326
|
+
*
|
|
261
327
|
* @returns {Promise<any[]>}
|
|
262
328
|
*/
|
|
263
|
-
async similar(query, { target = 'knowledge', limit = 5, shelf = shelfOf() } = {}) {
|
|
264
|
-
const
|
|
329
|
+
async similar(query, { target = 'knowledge', limit = 5, shelf = shelfOf(), excludeTags = [] } = {}) {
|
|
330
|
+
const argv = ['similar', query, '-t', target, '--limit', String(limit), '--shelf', shelf]
|
|
331
|
+
if (excludeTags.length > 0 && (await features()).similarFilters) argv.push('--exclude-tags', excludeTags.join(','))
|
|
332
|
+
const { stdout } = await runOk(argv)
|
|
265
333
|
if (isEmptyResult(stdout)) return []
|
|
266
334
|
const parsed = parseJson(stdout, 'similar')
|
|
267
335
|
return Array.isArray(parsed) ? parsed : []
|
package/src/hypatia-client.js
CHANGED
|
@@ -54,7 +54,21 @@ export function createHypatiaClient(ctx, config, { status, connection }) {
|
|
|
54
54
|
}
|
|
55
55
|
}
|
|
56
56
|
|
|
57
|
+
/** Logged once, so the profile log says which flags this run's writes carry. */
|
|
58
|
+
let announced = false
|
|
59
|
+
async function features() {
|
|
60
|
+
const found = await via('features', [])
|
|
61
|
+
if (!announced) {
|
|
62
|
+
announced = true
|
|
63
|
+
const yes = (flag) => (flag ? 'yes' : 'no')
|
|
64
|
+
status.info?.(`hypatia over ${usesMcp() ? 'mcp' : 'cli'}: no-embed ${yes(found.noEmbed)}, similar filters ${yes(found.similarFilters)}`)
|
|
65
|
+
}
|
|
66
|
+
return found
|
|
67
|
+
}
|
|
68
|
+
|
|
57
69
|
return {
|
|
70
|
+
/** What the binary can do beyond the baseline (see hypatia-cli.js `features`). */
|
|
71
|
+
features,
|
|
58
72
|
knowledgeGet: (...args) => via('knowledgeGet', args),
|
|
59
73
|
knowledgeCreate: (...args) => via('knowledgeCreate', args),
|
|
60
74
|
statementCreate: (...args) => via('statementCreate', args),
|
package/src/hypatia-mcp.js
CHANGED
|
@@ -50,6 +50,27 @@ export const REQUIRED_TOOLS = Object.freeze([
|
|
|
50
50
|
'query',
|
|
51
51
|
])
|
|
52
52
|
|
|
53
|
+
/**
|
|
54
|
+
* What a server can do beyond REQUIRED_TOOLS, read off the schemas in its
|
|
55
|
+
* `tools/list`. Every tool declares `additionalProperties: false`, so an
|
|
56
|
+
* argument a server does not list fails the call as a validation error rather
|
|
57
|
+
* than being ignored — which is why the plugin asks first instead of trying.
|
|
58
|
+
* The names mirror hypatia-cli.js's `features()`.
|
|
59
|
+
*
|
|
60
|
+
* @param {Array<{name?: string, inputSchema?: {properties?: object}}>} tools
|
|
61
|
+
* @returns {{noEmbed: boolean, similarFilters: boolean}}
|
|
62
|
+
*/
|
|
63
|
+
export function featuresOf(tools) {
|
|
64
|
+
const accepts = (name, argument) => Object.hasOwn(
|
|
65
|
+
tools.find((tool) => tool?.name === name)?.inputSchema?.properties ?? {},
|
|
66
|
+
argument,
|
|
67
|
+
)
|
|
68
|
+
return {
|
|
69
|
+
noEmbed: accepts('knowledge_create', 'embed') && accepts('statement_create', 'embed'),
|
|
70
|
+
similarFilters: accepts('similar', 'exclude_tags'),
|
|
71
|
+
}
|
|
72
|
+
}
|
|
73
|
+
|
|
53
74
|
/**
|
|
54
75
|
* One request line. A lone UTF-16 surrogate — what a `.slice()` through an
|
|
55
76
|
* emoji leaves behind — is written by `JSON.stringify` as a bare `\ud83d`
|
|
@@ -255,11 +276,13 @@ export function createMcpConnection(ctx, options) {
|
|
|
255
276
|
}, handshakeTimeoutMs)
|
|
256
277
|
conn.handle.stdin.write(encode({ jsonrpc: '2.0', method: 'notifications/initialized' }))
|
|
257
278
|
const listed = await request(conn, 'tools/list', {}, handshakeTimeoutMs)
|
|
258
|
-
const
|
|
279
|
+
const tools = listed?.tools ?? []
|
|
280
|
+
const names = new Set(tools.map((tool) => tool?.name))
|
|
259
281
|
const missing = REQUIRED_TOOLS.filter((name) => !names.has(name))
|
|
260
282
|
if (missing.length > 0) {
|
|
261
283
|
throw new HypatiaCliError(`hypatia mcp lacks ${missing.join(', ')}`, { code: 'MCP_UNAVAILABLE' })
|
|
262
284
|
}
|
|
285
|
+
conn.features = featuresOf(tools)
|
|
263
286
|
} catch (error) {
|
|
264
287
|
kill(conn, error)
|
|
265
288
|
if (!isUnavailable(error)) throw error
|
|
@@ -282,6 +305,37 @@ export function createMcpConnection(ctx, options) {
|
|
|
282
305
|
idleTimer.unref?.()
|
|
283
306
|
}
|
|
284
307
|
|
|
308
|
+
/**
|
|
309
|
+
* Run `step` against a live, handshaken process, after the steps before it.
|
|
310
|
+
* @template T
|
|
311
|
+
* @param {(conn: object) => Promise<T>} step
|
|
312
|
+
* @returns {Promise<T>}
|
|
313
|
+
*/
|
|
314
|
+
function withProcess(step) {
|
|
315
|
+
const run = async () => {
|
|
316
|
+
if (disposed) throw new HypatiaCliError('hypatia mcp connection disposed', { code: 'MCP_CLOSED' })
|
|
317
|
+
if (unavailable !== undefined) throw unavailable
|
|
318
|
+
clearTimeout(idleTimer)
|
|
319
|
+
try {
|
|
320
|
+
if (current === undefined || current.closed) {
|
|
321
|
+
const conn = await open()
|
|
322
|
+
// Disposed during the handshake: nothing else would close it.
|
|
323
|
+
if (disposed) {
|
|
324
|
+
close(conn)
|
|
325
|
+
throw new HypatiaCliError('hypatia mcp connection disposed', { code: 'MCP_CLOSED' })
|
|
326
|
+
}
|
|
327
|
+
current = conn
|
|
328
|
+
}
|
|
329
|
+
return await step(current)
|
|
330
|
+
} finally {
|
|
331
|
+
if (!disposed) armIdle()
|
|
332
|
+
}
|
|
333
|
+
}
|
|
334
|
+
const result = chain.then(run, run)
|
|
335
|
+
chain = result.catch(() => {})
|
|
336
|
+
return result
|
|
337
|
+
}
|
|
338
|
+
|
|
285
339
|
return {
|
|
286
340
|
/**
|
|
287
341
|
* One `tools/call`, after the calls before it.
|
|
@@ -290,31 +344,21 @@ export function createMcpConnection(ctx, options) {
|
|
|
290
344
|
* @returns {Promise<any>} the MCP `CallToolResult`, `isError` included.
|
|
291
345
|
*/
|
|
292
346
|
call(name, args) {
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
// earlier calls did. The handshake's errors quote everything.
|
|
309
|
-
current.stderrMark = current.handle.collected.stderr?.readFrom(0).nextOffset ?? 0
|
|
310
|
-
return await request(current, 'tools/call', { name, arguments: args }, timeoutMs)
|
|
311
|
-
} finally {
|
|
312
|
-
if (!disposed) armIdle()
|
|
313
|
-
}
|
|
314
|
-
}
|
|
315
|
-
const result = chain.then(run, run)
|
|
316
|
-
chain = result.catch(() => {})
|
|
317
|
-
return result
|
|
347
|
+
return withProcess((conn) => {
|
|
348
|
+
// An error quotes what this call made the process write, not what
|
|
349
|
+
// earlier calls did. The handshake's errors quote everything.
|
|
350
|
+
conn.stderrMark = conn.handle.collected.stderr?.readFrom(0).nextOffset ?? 0
|
|
351
|
+
return request(conn, 'tools/call', { name, arguments: args }, timeoutMs)
|
|
352
|
+
})
|
|
353
|
+
},
|
|
354
|
+
|
|
355
|
+
/**
|
|
356
|
+
* What the server can do beyond REQUIRED_TOOLS (see `featuresOf`), from
|
|
357
|
+
* the handshake — starting the process if none is running.
|
|
358
|
+
* @returns {Promise<{noEmbed: boolean, similarFilters: boolean}>}
|
|
359
|
+
*/
|
|
360
|
+
features() {
|
|
361
|
+
return withProcess((conn) => Promise.resolve(conn.features))
|
|
318
362
|
},
|
|
319
363
|
|
|
320
364
|
/** Drop the process; the next call starts a fresh one. */
|
|
@@ -438,23 +482,32 @@ export function createHypatiaMcp(connection, config) {
|
|
|
438
482
|
}
|
|
439
483
|
},
|
|
440
484
|
|
|
485
|
+
features: () => connection.features(),
|
|
486
|
+
|
|
441
487
|
/**
|
|
442
|
-
*
|
|
488
|
+
* `embed: false` keeps the entry out of the vector index (hypatia #26); a
|
|
489
|
+
* server whose `knowledge_create` does not list the argument would reject
|
|
490
|
+
* it, so there it is dropped and the entry is embedded as before.
|
|
491
|
+
*
|
|
492
|
+
* @param {{data: string, tags?: string[], scopes?: string[], embed?: boolean, shelf?: string}} entry
|
|
443
493
|
* @returns {Promise<boolean>} true when this call created the entry.
|
|
444
494
|
*/
|
|
445
495
|
async knowledgeCreate(name, entry) {
|
|
446
496
|
const args = { name, shelf: entry.shelf ?? shelfOf() }
|
|
447
497
|
if (entry.data) args.data = entry.data
|
|
498
|
+
if (entry.embed === false && (await connection.features()).noEmbed) args.embed = false
|
|
448
499
|
return (await create('knowledge_create', lists(args, entry))) !== undefined
|
|
449
500
|
},
|
|
450
501
|
|
|
451
502
|
/**
|
|
452
|
-
*
|
|
503
|
+
* `embed: false` as on `knowledgeCreate`.
|
|
504
|
+
* @param {{data?: string, scopes?: string[], embed?: boolean, shelf?: string}} [entry]
|
|
453
505
|
* @returns {Promise<boolean>} true when this call created the triple.
|
|
454
506
|
*/
|
|
455
507
|
async statementCreate(head, relation, tail, entry = {}) {
|
|
456
508
|
const args = { head, relation, tail, shelf: entry.shelf ?? shelfOf() }
|
|
457
509
|
if (entry.data) args.data = entry.data
|
|
510
|
+
if (entry.embed === false && (await connection.features()).noEmbed) args.embed = false
|
|
458
511
|
const outcome = await create('statement_create', lists(args, { scopes: entry.scopes }))
|
|
459
512
|
return outcome !== undefined && outcome.created !== false
|
|
460
513
|
},
|
|
@@ -464,9 +517,17 @@ export function createHypatiaMcp(connection, config) {
|
|
|
464
517
|
return rowsOf(await invoke('search', { query, catalog, limit, shelf }))
|
|
465
518
|
},
|
|
466
519
|
|
|
467
|
-
/**
|
|
468
|
-
|
|
469
|
-
|
|
520
|
+
/**
|
|
521
|
+
* Vector-similarity search; rows as the CLI prints them. `excludeTags`
|
|
522
|
+
* narrows the entries before they are ranked (hypatia #35) and is dropped
|
|
523
|
+
* on a server that does not list it, as on the CLI.
|
|
524
|
+
*/
|
|
525
|
+
async similar(query, { target = 'knowledge', limit = 5, shelf = shelfOf(), excludeTags = [] } = {}) {
|
|
526
|
+
const args = { query, target, limit, shelf }
|
|
527
|
+
if (excludeTags.length > 0 && (await connection.features()).similarFilters) {
|
|
528
|
+
args.exclude_tags = asCliList(excludeTags)
|
|
529
|
+
}
|
|
530
|
+
return rowsOf(await invoke('similar', args))
|
|
470
531
|
},
|
|
471
532
|
|
|
472
533
|
/** Raw JSE query; `jse` is the JSON text the CLI would take as its argument. */
|
package/src/writer.js
CHANGED
|
@@ -29,11 +29,19 @@ const OPERATIONAL_PREFIXES = ['msg-', 'sum', 'session-', 'hypatia-dream-run-']
|
|
|
29
29
|
const OPERATIONAL_TAGS = new Set(['message', 'session', 'hypatia-dream-run'])
|
|
30
30
|
|
|
31
31
|
/**
|
|
32
|
-
*
|
|
32
|
+
* The same layer as tags `similar` can be told to leave out (hypatia #35).
|
|
33
|
+
* `summary` alone covers every summary tier: each carries it beside its
|
|
34
|
+
* `summary <N>` tag.
|
|
35
|
+
*/
|
|
36
|
+
export const EXCLUDED_TAGS = Object.freeze([...OPERATIONAL_TAGS, 'summary'])
|
|
37
|
+
|
|
38
|
+
/**
|
|
39
|
+
* How many rows to ask `similar` for per candidate actually wanted, on a
|
|
40
|
+
* binary whose `similar` cannot exclude tags.
|
|
33
41
|
*
|
|
34
|
-
*
|
|
35
|
-
*
|
|
36
|
-
*
|
|
42
|
+
* There the operational layer is ranked along with everything else, outnumbers
|
|
43
|
+
* knowledge on an active shelf, and ranks nearer, so the fetch has to absorb
|
|
44
|
+
* it. Four covers the measured 7-in-10 ratio with room to spare.
|
|
37
45
|
*/
|
|
38
46
|
const CANDIDATE_OVERSAMPLE = 4
|
|
39
47
|
|
|
@@ -113,6 +121,13 @@ export function createWriter(cli, { status, adjudicate }) {
|
|
|
113
121
|
* from a replayed task) are counted and skipped — the watermark must still
|
|
114
122
|
* advance past them, which the queue does based on the resolved range.
|
|
115
123
|
*
|
|
124
|
+
* Written with `embed: false`: the log layer is not on the retrieval hot
|
|
125
|
+
* path — `docs/memory-nolinear.md` has precise recall drill down from a
|
|
126
|
+
* summary along `summary` edges — and embedded, a raw message outranks the
|
|
127
|
+
* knowledge distilled from it, because it holds the very words. It stays
|
|
128
|
+
* stored and full-text searchable; only the vector is not made. On a binary
|
|
129
|
+
* without `--no-embed` the flag is dropped by the transport.
|
|
130
|
+
*
|
|
116
131
|
* @param {{sessionId: string, index: number, markdown: string, project: string}} entry
|
|
117
132
|
*/
|
|
118
133
|
async function writeMessage({ sessionId, index, markdown, project }) {
|
|
@@ -122,7 +137,7 @@ export function createWriter(cli, { status, adjudicate }) {
|
|
|
122
137
|
status.count('duplicate')
|
|
123
138
|
return { name, written: false }
|
|
124
139
|
}
|
|
125
|
-
await cli.knowledgeCreate(name, { data: markdown, tags: ['message'], scopes: [project] })
|
|
140
|
+
await cli.knowledgeCreate(name, { data: markdown, tags: ['message'], scopes: [project], embed: false })
|
|
126
141
|
status.count('written')
|
|
127
142
|
return { name, written: true }
|
|
128
143
|
}
|
|
@@ -147,6 +162,11 @@ export function createWriter(cli, { status, adjudicate }) {
|
|
|
147
162
|
* permanent. `statementCreate` swallows primary-key collisions, so
|
|
148
163
|
* re-asserting an existing edge is free.
|
|
149
164
|
*
|
|
165
|
+
* The summary itself is embedded — it is distilled knowledge and the entry
|
|
166
|
+
* point for drilling down. Its edges are written `embed: false`: they exist
|
|
167
|
+
* for the traversal and for `$not-summaried`, and statements carry no tags,
|
|
168
|
+
* so a shelf's `embedding.skip_tags` could not reach them.
|
|
169
|
+
*
|
|
150
170
|
* @param {{sessionId: string, fromSeq: number, toSeq: number, markdown: string,
|
|
151
171
|
* project: string, items?: readonly string[], level?: number}} entry
|
|
152
172
|
* `level` is the archive tier this summary sits at (1 = summarises messages).
|
|
@@ -168,7 +188,7 @@ export function createWriter(cli, { status, adjudicate }) {
|
|
|
168
188
|
status.count('written')
|
|
169
189
|
}
|
|
170
190
|
for (const item of items) {
|
|
171
|
-
await cli.statementCreate(name, 'summary', item, { scopes: [project] })
|
|
191
|
+
await cli.statementCreate(name, 'summary', item, { scopes: [project], embed: false })
|
|
172
192
|
}
|
|
173
193
|
return { name, written: existing.found === false, links: items.length }
|
|
174
194
|
}
|
|
@@ -184,6 +204,10 @@ export function createWriter(cli, { status, adjudicate }) {
|
|
|
184
204
|
* hypatia has no `knowledge-update`, so a second title cannot replace the
|
|
185
205
|
* first; the original stands and the caller is told nothing was written.
|
|
186
206
|
*
|
|
207
|
+
* Node and `belongTo` edges are written `embed: false`, as the messages are:
|
|
208
|
+
* the node is bookkeeping for the traversal, and its title is what
|
|
209
|
+
* `session/title` already gave DSH.
|
|
210
|
+
*
|
|
187
211
|
* @param {{sessionId: string, markdown: string, project: string,
|
|
188
212
|
* linkFrom: number, linkTo: number}} entry - `[linkFrom, linkTo)` is
|
|
189
213
|
* the half-open range of message ordinals still needing a `belongTo` edge.
|
|
@@ -200,11 +224,11 @@ export function createWriter(cli, { status, adjudicate }) {
|
|
|
200
224
|
// it absent and drop the edges with it.
|
|
201
225
|
return { name, written: false, links: 0 }
|
|
202
226
|
} else {
|
|
203
|
-
await cli.knowledgeCreate(name, { data: markdown, tags: ['session'], scopes: [project] })
|
|
227
|
+
await cli.knowledgeCreate(name, { data: markdown, tags: ['session'], scopes: [project], embed: false })
|
|
204
228
|
status.count('written')
|
|
205
229
|
}
|
|
206
230
|
for (let index = linkFrom; index < linkTo; index += 1) {
|
|
207
|
-
await cli.statementCreate(messageName(sessionId, index), 'belongTo', name, { scopes: [project] })
|
|
231
|
+
await cli.statementCreate(messageName(sessionId, index), 'belongTo', name, { scopes: [project], embed: false })
|
|
208
232
|
}
|
|
209
233
|
return { name, written: existing.found === false, links: Math.max(0, linkTo - linkFrom) }
|
|
210
234
|
}
|
|
@@ -220,19 +244,24 @@ export function createWriter(cli, { status, adjudicate }) {
|
|
|
220
244
|
*/
|
|
221
245
|
async function findCandidates(unit, name, { maxDistance, limit }) {
|
|
222
246
|
try {
|
|
223
|
-
//
|
|
224
|
-
//
|
|
225
|
-
//
|
|
226
|
-
//
|
|
227
|
-
//
|
|
228
|
-
//
|
|
229
|
-
//
|
|
230
|
-
//
|
|
231
|
-
//
|
|
247
|
+
// The operational layer is excluded IN the query where the binary allows
|
|
248
|
+
// it (hypatia #35), and over-fetched around where it does not. It is the
|
|
249
|
+
// majority of an active shelf (101 of 183 knowledge entries on the
|
|
250
|
+
// profile this was measured on) and it scores CLOSER than real knowledge
|
|
251
|
+
// — a raw `msg-*` holds the very words of the conversation the unit was
|
|
252
|
+
// extracted from. Measured: 7 of the nearest 10 rows were operational, so
|
|
253
|
+
// asking for 5 and filtering afterwards left one candidate, often none,
|
|
254
|
+
// and adjudication silently degraded to "no relationship" on every write.
|
|
255
|
+
// Entries this plugin now writes `embed: false` are not ranked at all,
|
|
256
|
+
// but a shelf holds what earlier versions and other writers embedded.
|
|
257
|
+
const { similarFilters } = await cli.features()
|
|
232
258
|
const rows = await cli.similar(`${unit.title}\n${unit.content}`.slice(0, 500), {
|
|
233
259
|
target: 'knowledge',
|
|
234
|
-
limit: Math.min(limit * CANDIDATE_OVERSAMPLE, CANDIDATE_FETCH_CEILING),
|
|
260
|
+
limit: similarFilters ? limit : Math.min(limit * CANDIDATE_OVERSAMPLE, CANDIDATE_FETCH_CEILING),
|
|
261
|
+
excludeTags: EXCLUDED_TAGS,
|
|
235
262
|
})
|
|
263
|
+
// The name-prefix half of `isOperationalRow` still applies: a tag filter
|
|
264
|
+
// cannot see an operational entry that lost its tags.
|
|
236
265
|
return rows
|
|
237
266
|
.filter((row) => rowName(row) !== name)
|
|
238
267
|
.filter((row) => !isOperationalRow(row))
|