@etzelm/openclaw-memory-hindsight 0.1.2 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,141 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.3.0 (2026-11-08)
4
+
5
+ ### Upgrade notes
6
+
7
+ - No config change is needed. The new `knowledgeReadTools` key is off by default, so the
8
+ registered tools are the same as in 0.2.0 unless you turn it on.
9
+ - The retain queue file format is unchanged. See Downgrade notes before going back to 0.2.0.
10
+ - Recalled memories that contain text shaped like the `<hindsight_memories>` wrapper or
11
+ OpenClaw's internal-context delimiters are now escaped before they reach the model.
12
+
13
+ ### Security
14
+
15
+ - Recall wrapper: `<hindsight_memories>` and `</hindsight_memories>` tags inside recalled text
16
+ are neutralized in more forms than the exact literal: any case, whitespace inside the tag,
17
+ attributes, self-closing, a missing `>`, `-` for `_`, invisible characters, accents, common
18
+ Cyrillic/Greek lookalike letters, fullwidth letters and `<` lookalikes (everything Unicode
19
+ confusables.txt maps to `<`, plus angle brackets).
20
+ - OpenClaw prompt delimiters in recalled text are escaped: `<<<BEGIN_OPENCLAW_INTERNAL_CONTEXT>>>`
21
+ and `<<<END_OPENCLAW_INTERNAL_CONTEXT>>>` (variants in case, spacing or invisible characters
22
+ too), `OpenClaw runtime context:`, `End OpenClaw runtime context.`,
23
+ `OpenClaw runtime context (internal):`, the runtime-context notice line and the legacy
24
+ carrier prefaces that precede it. Prose that mentions OpenClaw runtime context is unchanged.
25
+
26
+ ### Added
27
+
28
+ - `scripts/compat-matrix.sh`: host keyword `beta` (npm dist-tag `beta`), comma-separated
29
+ `--hosts` and `--runtimes`, and matrix rows in the requested host order.
30
+ - `knowledgeReadTools` (default off): opt-in read-only knowledge-base tools
31
+ `hindsight_get_knowledge_base_tree`, `hindsight_search_knowledge_base` and
32
+ `hindsight_get_knowledge_page`, with the vendor plugin's names. They read the same
33
+ audience-routed bank as `hindsight_recall`; a missing bank reads as empty. Off, the
34
+ registered tool set is unchanged; the manifest's static `contracts.tools` still lists the
35
+ three names. Output is capped (page 32000 characters, tree 500 entries, search 50 hits)
36
+ and a knowledge-base response body over 1 MiB fails.
37
+
38
+ ### Changed
39
+
40
+ - CI enforces a vitest coverage floor (`npm run test:coverage`, `@vitest/coverage-v8`) at
41
+ lines 94%, branches 86%, functions 95%, statements 93% (develop's measured coverage minus a
42
+ margin). GitLab shows the percentage on MRs and keeps the report as an artifact. Both pack
43
+ checks also reject `coverage/` (the `files` allowlist already kept it out of the pack).
44
+ - `scripts/launchd/hs-compat-nightly.plist.example`: `LimitLoadToSessionType` Aqua and
45
+ Background, so it also loads in the `user/<uid>` domain; the docs cover installing it for an
46
+ account without a GUI session.
47
+
48
+ ### Fixes
49
+
50
+ - `scripts/live-proof.sh`: X2 now searches every Claude Code project dir that appeared during
51
+ the run, and every file under the projects dir modified during it, for the code word, not
52
+ only names with `hs-live-proof`. Dirs it did not predict are counted and never deleted;
53
+ symlinks there are not followed.
54
+ - `scripts/compat-matrix.sh`: an empty `--hosts` or `--runtimes` list is refused (exit 2).
55
+ - `scripts/compat-nightly.sh`: one run at a time per `WORK_ROOT`; a lock is stale when its pid
56
+ does not run the same script path or it is older than 12 h.
57
+ - Config: `hindsightApiUrl` with a bare trailing `?` or `#` is rejected like a non-empty query
58
+ string or fragment.
59
+ - Retain queue: a retain is no longer lost when the file cannot be rewritten but can still be
60
+ appended to. The new copy is appended instead (load keeps the newest copy per document), and the
61
+ 32 MiB bound still applies to appends, so the file cannot grow without bound. A failed replay
62
+ save is retried by the next one.
63
+ - Retain queue: after a crash between replay's batched writes, load keeps only the newest copy
64
+ per bank and document id, and replay always sends `update_mode: replace` with the item's
65
+ document id.
66
+ - Auto-retain no longer loses a turn when the host continues a run after a transient model
67
+ retry. The second `agent_end` for the same run id now re-sends the whole run from its first
68
+ start instead of only the continuation, so the run's document is replaced with complete
69
+ content. The run start is kept per session and run id; a new run id, `session_end` or a stop
70
+ drops it. Host maintenance turns no longer touch it.
71
+
72
+ ### Downgrade notes
73
+
74
+ - The queue file format is unchanged. 0.2.0 keeps duplicate copies and replays them oldest first
75
+ (same document id, replace), so a re-retain of that document before replay finishes can leave
76
+ older text. A build that ignores tombstone lines (unreleased queue builds before 0.2.0) may
77
+ replay an out-of-date copy over a newer one that was sent live. Let the queue drain, or delete
78
+ `retain-queue.jsonl`, before downgrading.
79
+
80
+ ## 0.2.0 (2026-11-07)
81
+
82
+ ### Upgrade notes
83
+
84
+ - OpenClaw `>=2026.10.10` is now required (was `>=2026.10.1`). Upgrade OpenClaw first; older
85
+ hosts refuse the install. See Fixes below for why.
86
+ - The offline retain queue is on by default (`retainQueueEnabled`). Auto-retains that fail
87
+ because Hindsight is unreachable are written to
88
+ `<state dir>/hindsight-native/retain-queue.jsonl` and sent later. The file holds conversation
89
+ text (mode 600). Set `retainQueueEnabled: false` to drop them as before.
90
+
91
+ ### Added
92
+
93
+ - `retainTags`: tags added to every retained memory, with the same key and meaning as
94
+ `@vectorize-io/hindsight-openclaw`. They apply to auto-retain, `hindsight_retain` and the
95
+ pre-compaction flush, merged before inline `<retain_tags>` and per-call tags without
96
+ duplicates. A vendor config that sets it no longer fails with "unknown keys: retainTags".
97
+ - `scripts/live-proof.sh` is now a reusable harness: `--openclaw` (npm version, git ref or
98
+ path), `--runtime embedded|claude-cli`, `--plugin` (checkout, tarball or install spec) and
99
+ `--phase`, with a PASS/FAIL/SKIP table and `results.json`. It replaces the earlier
100
+ single-host script.
101
+ - `scripts/compat-matrix.sh` runs the harness for the minimum host version, the latest release
102
+ and `main` on both runtimes and writes `MATRIX.md`. A nightly launchd example is documented,
103
+ not installed.
104
+ - Offline retain queue, on by default (`retainQueueEnabled`). An auto-retain that fails because
105
+ Hindsight is unreachable (network error, timeout, HTTP 5xx) is saved to
106
+ `<state dir>/hindsight-native/retain-queue.jsonl` (mode 600) and sent later, instead of being
107
+ dropped. A 4xx answer is still logged and dropped.
108
+ - Each queued item keeps its bank and document id. Replay never routes it to another bank, and a
109
+ second retain of the same document replaces the queued copy. No URL or token is stored.
110
+ - Replay runs at service start, on a backoff timer (30 s doubling to 15 min) and after the next
111
+ successful retain, oldest first, so each bank sees its writes in order.
112
+ - `retainQueueMaxItems` (default 500), `retainQueueMaxAgeMs` (default 7 days) and a fixed 32 MiB
113
+ total (UTF-8 bytes) bound the queue. A single retain over the bound is not queued. Dropped
114
+ items log one warning until the queue drains.
115
+ - Queued items keep the tags resolved when the run ended (`retainTags` plus inline
116
+ `<retain_tags>`). A queued retain whose bank no longer exists (Hindsight 404 `Bank not found`)
117
+ is dropped with a warning on replay, like other 4xx answers.
118
+ - Replay drops an item only on a Hindsight HTTP 4xx about that item. A refused redirect, an
119
+ ambiguous 404, auth and rate-limit answers, a 4xx from bank setup and unexpected errors keep it
120
+ and back off. If a bank already has queued items and the queue file cannot be written, the
121
+ retain is sent live instead.
122
+
123
+ ### Fixes
124
+
125
+ - A first recall on a bank nothing was retained to yet no longer logs a warning
126
+ (`auto-recall skipped: Hindsight recall failed: HTTP 404`). Hindsight's
127
+ `Bank '<id>' not found` 404 now reads as "no memories" in auto-recall, memory search and the
128
+ `hindsight_recall` / `hindsight_reflect` tools, with a debug line at most. A 404 from a route
129
+ that does not exist (wrong `hindsightApiUrl` path) and other HTTP errors stay loud.
130
+ - No auto-recall and no auto-retain on host maintenance turns: the pre-compaction flush turn
131
+ (trigger `memory`) and Active Memory's recall child (`<session>:active-memory:<id>`). Before,
132
+ the flush turn recalled with the host's flush prompt as the query, and the recall child could
133
+ be retained as if it were a conversation. The `hindsight_*` tools stay available to both.
134
+ - `openclaw.install.minHostVersion` and the `openclaw` peer dependency are now `>=2026.10.10`
135
+ (was `>=2026.10.1`). OpenClaw 2026.10.1 through 2026.10.9 give turns in a dashboard session
136
+ created by `sessions.create` no memory audience, so the plugin could not recall or retain
137
+ there. 2026.10.10 records those sessions as direct chats (openclaw#181359).
138
+
3
139
  ## 0.1.2 (2026-11-06)
4
140
 
5
141
  ### Security
package/README.md CHANGED
@@ -8,8 +8,11 @@ contracts instead of only hooking prompts:
8
8
  - **Provider runtime**: `memory.status`, `memory.search` (v2) and `memory.get` in the Gateway,
9
9
  CLI and Control UI read Hindsight through the host's audience-checked provider API.
10
10
  - **Auto-recall** before each prompt and **auto-retain** of each finished run (one document per
11
- run, replaced on re-delivery).
12
- - **Agent tools** `hindsight_retain`, `hindsight_recall`, `hindsight_reflect`.
11
+ run, replaced on re-delivery). Host maintenance turns (the pre-compaction flush and Active
12
+ Memory's recall child) get neither; they use the tools. A bank nothing was retained to yet
13
+ reads as "no memories", without a warning.
14
+ - **Agent tools** `hindsight_retain`, `hindsight_recall`, `hindsight_reflect`, plus opt-in
15
+ read-only knowledge-base tools (see [Knowledge-base read tools](#knowledge-base-read-tools)).
13
16
  - **Recall tool declaration** (`recallToolNames`): Active Memory uses `hindsight_recall` and
14
17
  `hindsight_reflect` without a `toolsAllow` list.
15
18
  - **Pre-compaction flush, tools arm**: before compaction the host runs a silent turn that saves
@@ -31,7 +34,7 @@ It only imports `openclaw/plugin-sdk/*`. The single runtime dependency is `typeb
31
34
 
32
35
  ## Install
33
36
 
34
- Needs OpenClaw 2026.10.1 or newer and a reachable Hindsight API.
37
+ Needs OpenClaw 2026.10.10 or newer and a reachable Hindsight API.
35
38
 
36
39
  From npm:
37
40
 
@@ -56,7 +59,7 @@ tarball and install that:
56
59
  git clone https://github.com/etzelm/openclaw-memory-hindsight.git
57
60
  cd openclaw-memory-hindsight
58
61
  npm ci && npm pack
59
- openclaw plugins install npm-pack:./etzelm-openclaw-memory-hindsight-0.1.1.tgz --force --accept-capabilities
62
+ openclaw plugins install npm-pack:./etzelm-openclaw-memory-hindsight-0.3.0.tgz --force --accept-capabilities
60
63
  ```
61
64
 
62
65
  `--force` confirms the "outside ClawHub review" warning for a local archive. Every install asks for
@@ -101,40 +104,147 @@ tool conflict.
101
104
 
102
105
  ## Configuration
103
106
 
104
- | Key | Default | Meaning |
105
- | --------------------------- | ----------------- | ----------------------------------------------------------------------------------------------------- |
106
- | `hindsightApiUrl` | required | Base URL of the Hindsight API |
107
- | `hindsightApiToken` | none | Bearer token; use a SecretRef |
108
- | `memoryMode` | `hybrid` | `hybrid`: auto-recall and tools; `context`: auto-recall only, no tools, no flush; `tools`: tools only |
109
- | `autoRecall` / `autoRetain` | `true` | Turn the hooks on or off |
110
- | `recallBudget` | `mid` | Hindsight recall budget: `low`, `mid`, `high` |
111
- | `recallTypes` | all | Subset of `world`, `experience`, `observation` |
112
- | `preferObservations` | Hindsight default | Prefer consolidated observations in recall |
113
- | `recallPromptPreamble` | built in | Text above the injected `<hindsight_memories>` block |
114
- | `retainMission` | none | Applied to each bank once before its first write |
115
- | `retainContext` | built in | Context string sent with each retain |
116
- | `excludeProviders` | `[]` | Message providers that never recall or retain (for example `heartbeat`) |
117
- | `skipOperationalSessions` | `true` | Skip cron, heartbeat and subagent sessions |
118
- | `agentBankMap` | `{}` | Agent id to bank id, so several agents share one owner bank |
119
- | `allowInsecureHttp` | `false` | Allow plain http to a non-loopback `hindsightApiUrl` (token and conversation text unencrypted) |
107
+ | Key | Default | Meaning |
108
+ | --------------------------- | -------------------- | ----------------------------------------------------------------------------------------------------- |
109
+ | `hindsightApiUrl` | required | Base URL of the Hindsight API |
110
+ | `hindsightApiToken` | none | Bearer token; use a SecretRef |
111
+ | `memoryMode` | `hybrid` | `hybrid`: auto-recall and tools; `context`: auto-recall only, no tools, no flush; `tools`: tools only |
112
+ | `autoRecall` / `autoRetain` | `true` | Turn the hooks on or off |
113
+ | `recallBudget` | `mid` | Hindsight recall budget: `low`, `mid`, `high` |
114
+ | `recallTypes` | all | Subset of `world`, `experience`, `observation` |
115
+ | `preferObservations` | Hindsight default | Prefer consolidated observations in recall |
116
+ | `recallPromptPreamble` | built in | Text above the injected `<hindsight_memories>` block |
117
+ | `retainMission` | none | Applied to each bank once before its first write |
118
+ | `retainContext` | built in | Context string sent with each retain |
119
+ | `retainTags` | `[]` | Tags added to every retained memory (auto-retain, `hindsight_retain`, pre-compaction flush) |
120
+ | `excludeProviders` | `[]` | Message providers that never recall or retain (for example `heartbeat`) |
121
+ | `skipOperationalSessions` | `true` | Skip cron, heartbeat and subagent sessions |
122
+ | `agentBankMap` | `{}` | Agent id to bank id, so several agents share one owner bank |
123
+ | `allowInsecureHttp` | `false` | Allow plain http to a non-loopback `hindsightApiUrl` (token and conversation text unencrypted) |
124
+ | `retainQueueEnabled` | `true` | Queue auto-retains that fail because Hindsight is unreachable and send them later (see below) |
125
+ | `retainQueueMaxItems` | `500` | Most queued retains kept (1 to 10000); the oldest are dropped first |
126
+ | `retainQueueMaxAgeMs` | `604800000` (7 days) | Queued retains older than this are dropped |
127
+ | `knowledgeReadTools` | `false` | Offer the read-only knowledge-base tools (see below); turning it on needs a gateway restart |
120
128
 
121
129
  Fixed values (not configurable): recall max tokens 1024, query cap 800 characters, recall
122
- timeout 10 s, retain and reflect timeout 30 s, memory get timeout 10 s (each combined with the
123
- host's abort signal when there is one), retained roles user and assistant, JSON transcript format, recall injected before
124
- the user message (outside the system prompt, so the cached prefix stays stable).
130
+ timeout 10 s, retain and reflect timeout 30 s, memory get and knowledge-base tool timeout 10 s
131
+ (each combined with the host's abort signal when there is one), retained roles user and
132
+ assistant, JSON transcript format, recall injected before the user message (outside the system
133
+ prompt, so the cached prefix stays stable). Knowledge-base tool caps: a response body over
134
+ 1 MiB fails; a page shows its first 32000 characters, then `(truncated, N more characters)`;
135
+ the tree shows 500 entries, then `(truncated, N more entries)`; search shows at most 50 hits
136
+ with snippets cut at 500 characters and names and ids at 200.
125
137
 
126
138
  Config changes apply without a restart: the plugin re-reads its entry from the host's current
127
139
  config snapshot (`api.runtime.config.current()`), falling back to the register-time
128
- `api.pluginConfig`.
140
+ `api.pluginConfig`. One exception: `knowledgeReadTools` registers its tools at startup, so turning
141
+ it on takes a gateway restart (turning it off applies at once).
142
+
143
+ ### Offline retain queue
144
+
145
+ When an auto-retain cannot reach Hindsight (network error, timeout or HTTP 5xx), the plugin
146
+ writes it to `<state dir>/hindsight-native/retain-queue.jsonl` and logs one warning per failed
147
+ retain. The state dir is the one the host passes to plugin services (`~/.openclaw` by default,
148
+ or `OPENCLAW_STATE_DIR`). The file is mode 600 in a mode 700 directory, one JSON line per item.
149
+ It holds the bank id, document id and the retain payload (transcript, context, tags, metadata),
150
+ and never the API URL or token. The transcript is conversation text, so treat the file like
151
+ session transcripts.
152
+
153
+ - **Same bank, always.** Each item keeps the bank chosen from the run's memory audience when the
154
+ run ended. Replay writes to that bank only, even if `agentBankMap` changed in between. Items do
155
+ not record the server: if `hindsightApiUrl` points at another Hindsight by then, they go there
156
+ under the same bank id.
157
+ - **No duplicates.** Each item keeps its document id and is sent with `update_mode: replace`. A
158
+ second retain of the same document replaces the queued copy, and a replay that the server had
159
+ already received overwrites instead of adding a document.
160
+ - **Replay.** At service start, on a timer after a failure, and after the next successful retain.
161
+ The first retry comes 30 s after a failed pass, and the wait doubles after each further failed
162
+ pass (a turn for a bank with queued items also starts a pass), up to 15 min. Items go out oldest
163
+ first and a pass stops at the first item it cannot deliver, so each bank sees its writes in order.
164
+ While a bank has queued items, a new retain for that bank queues behind them. Two retains for one
165
+ bank that are in flight at the same moment can still land out of order if the earlier one fails
166
+ and is queued.
167
+ - **Not queued.** A 4xx answer (bad request, unknown bank, auth) and a refused redirect are logged
168
+ and dropped as before. On replay, only a Hindsight HTTP 4xx about the item drops it, with a
169
+ warning: any 4xx except 401, 403, 429 and an ambiguous 404 (a wrong `hindsightApiUrl` path, an
170
+ HTML or empty body). Hindsight creates a bank on its first retain, so its explicit `Bank '<id>'
171
+ not found` 404 is about the item and drops it. Everything else keeps the item, ends the pass and
172
+ backs off: network errors, timeouts, 5xx, 401, 403, 429, an ambiguous 404, a 4xx from bank setup
173
+ (applying `retainMission`), a refused redirect and any unexpected error. Kept items hold back
174
+ items for other banks until the next attempt; anything other than a network error, timeout or 5xx
175
+ logs one warning until something is delivered again.
176
+ - **Tags.** An item keeps the tags resolved when the run ended (`retainTags` plus inline
177
+ `<retain_tags>`); a later `retainTags` change does not touch queued items.
178
+ - **Limits.** Items beyond `retainQueueMaxItems` or a fixed 32 MiB total (UTF-8 bytes on disk,
179
+ oldest first), or older than `retainQueueMaxAgeMs`, are dropped with one warning until the queue
180
+ is empty again. A single retain larger than 32 MiB is not queued: it is sent live without waiting
181
+ for its bank's queue, and its failure is logged as before.
182
+ - **Disk writes.** A new item is appended, on a new line even after a torn last line. Replacing,
183
+ pruning and replay rewrite the file (atomic rename); replay writes once per 25 delivered items and
184
+ at the end of each pass, and a failed write is retried by the next one. If a rewrite fails, the
185
+ new or newer copy is appended instead, with a tombstone for any item the limits drop; the older
186
+ copy stays in the file until the next successful rewrite, and load keeps only the newest line per
187
+ bank and document id. An append that would take
188
+ the file past 32 MiB rewrites it instead. Only when that rewrite fails too is the item not queued:
189
+ an out-of-date copy it replaced gets a tombstone line so a restart does not replay it, and the
190
+ retain is sent live, or, when its live write already failed, logged and not kept. Until the next
191
+ successful rewrite, the file stays within 32 MiB plus at most one short tombstone line per item
192
+ removed from the queue meanwhile. A crash between replay's writes can also leave an already delivered copy before
193
+ a newer one; load keeps the newer one, and replay always sends `update_mode: replace` with the
194
+ item's document id, so a repeat is harmless. One queue per file per process, shared across
195
+ service restarts.
196
+ - **Downgrade.** The file format is unchanged. 0.2.0 reads tombstones but keeps every copy of a
197
+ document (left by a crash during replay or by a replace that could not rewrite the file) and
198
+ sends them oldest first, same document id with replace, so the newest text wins unless the same
199
+ document is retained again before that replay finishes. 0.1.x has no queue and ignores the file.
200
+ A build that ignores tombstone lines (only unreleased queue builds before 0.2.0) can replay an
201
+ out-of-date copy a tombstone hid; if the newer copy was then sent live and never queued, the stale
202
+ text overwrites it on the server. Let the queue drain (the file is deleted when it is empty), or
203
+ delete the file, before downgrading.
204
+ - Only auto-retain is queued. `hindsight_retain` and the pre-compaction flush return the error to
205
+ the agent, as before. With `retainQueueEnabled: false` the plugin neither reads nor writes the
206
+ file; delete it, or turn the queue back on to drain it. `retainQueueEnabled` is read at service
207
+ start (restart the gateway after changing it); the two limits apply live. One gateway process per
208
+ state dir: the file is not locked.
209
+
210
+ ### Knowledge-base read tools
211
+
212
+ With `knowledgeReadTools: true` the agent also gets three read-only tools over Hindsight's
213
+ knowledge pages (documents Hindsight writes and refreshes from memories), under the same names as
214
+ the vendor plugin:
215
+
216
+ - `hindsight_get_knowledge_base_tree`: folders and pages with page ids and a stale mark
217
+ (`GET .../knowledge-base/tree`).
218
+ - `hindsight_search_knowledge_base`: pages matching a query, with a snippet (`query`, optional
219
+ `limit` clamped to 1-50; `GET .../knowledge-base/search`).
220
+ - `hindsight_get_knowledge_page`: one page as markdown by `page_id`
221
+ (`GET .../knowledge-base/pages/<id>`).
222
+
223
+ They follow the recall tools' rules: same bank from the host-minted memory audience (an
224
+ owner-private session reads the owner bank, a conversation reads only its own
225
+ `<bank>::conv::<hash>` bank), same session filters, none in `context` mode or without an
226
+ audience. Hindsight scopes page ids to the bank in the URL, so a page id from the owner bank is
227
+ "not found" through a conversation bank. A bank nothing was written to reads as "no knowledge
228
+ pages yet"; as with recall, a 404 with an empty body counts as a missing bank too. A route 404
229
+ (`{"detail":"Not Found"}`: wrong URL or a Hindsight version without these endpoints) stays an
230
+ error. `page_id` must look like a Hindsight id (letters, digits, `_ . : -`, starting with a
231
+ letter or digit, at most 200 characters) and is refused otherwise, before any request. Output
232
+ is marked as untrusted data, like recall, and capped (see Fixed values). The manifest's
233
+ `contracts.tools` always lists the three names (it is static), so they appear in plugin status
234
+ and doctor tool-owner output even with the key off; they are registered only when it is on.
235
+ Host tool listings can therefore show them while the key is off: the manifest's `configSignals`
236
+ only test that a key is present, not that it is `true`, so they cannot express this key. The
237
+ write tools (create folder, create page, update, delete) are not ported.
129
238
 
130
239
  ### Mapping from `@vectorize-io/hindsight-openclaw`
131
240
 
132
- | Vendor key | Here |
133
- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
134
- | `hindsightApiUrl`, `hindsightApiToken`, `memoryMode`, `autoRecall`, `autoRetain`, `recallBudget`, `recallTypes`, `preferObservations`, `recallPromptPreamble`, `retainMission`, `retainContext`, `excludeProviders`, `skipOperationalSessions`, `agentBankMap` | Same key, same meaning |
135
- | `dynamicBankId`, `dynamicBankGranularity`, `bankIdPrefix` | Replaced by audience routing: owner bank = agent id (or `agentBankMap`), conversations get their own bank |
136
- | `retainRoles`, `recallRoles`, `retainFormat`, `recallMaxTokens`, `recallMaxQueryChars`, `recallTimeoutMs`, `recallInjectionPosition`, `retainSource` | Fixed values (see above) |
137
- | `retainToolCalls`, `retainEveryNTurns`, `retainOverlapTurns`, `recallContextTurns`, `retainTags`, `retainQueue*`, `enableKnowledgeTools`, `allowKnowledgeDelete`, daemon keys (`apiPort`, `embedPort`, `embedVersion`, `daemonIdleTimeout`), `llm*`, `debug*`, `log*`, `skipStatelessSessions` | Not supported; remove them (unknown keys are rejected) |
241
+ | Vendor key | Here |
242
+ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
243
+ | `hindsightApiUrl`, `hindsightApiToken`, `memoryMode`, `autoRecall`, `autoRetain`, `recallBudget`, `recallTypes`, `preferObservations`, `recallPromptPreamble`, `retainMission`, `retainContext`, `retainTags`, `excludeProviders`, `skipOperationalSessions`, `agentBankMap` | Same key, same meaning |
244
+ | `dynamicBankId`, `dynamicBankGranularity`, `bankIdPrefix` | Replaced by audience routing: owner bank = agent id (or `agentBankMap`), conversations get their own bank |
245
+ | `retainRoles`, `recallRoles`, `retainFormat`, `recallMaxTokens`, `recallMaxQueryChars`, `recallTimeoutMs`, `recallInjectionPosition`, `retainSource` | Fixed values (see above) |
246
+ | `retainQueueMaxAgeMs` | Same key; must be a positive number of ms (the vendor's `-1` for no limit is rejected) |
247
+ | `retainToolCalls`, `retainEveryNTurns`, `retainOverlapTurns`, `recallContextTurns`, `retainQueuePath`, `retainQueueFlushIntervalMs`, `enableKnowledgeTools`, `allowKnowledgeDelete`, daemon keys (`apiPort`, `embedPort`, `embedVersion`, `daemonIdleTimeout`), `llm*`, `debug*`, `log*`, `skipStatelessSessions` | Not supported; remove them (unknown keys are rejected) |
138
248
 
139
249
  ## Compared with `@vectorize-io/hindsight-openclaw`
140
250
 
@@ -168,11 +278,32 @@ or a loopback tunnel. To accept the risk on a trusted network, set `allowInsecur
168
278
  plugin then warns at service start that the token and conversation text travel unencrypted.
169
279
  Redirects are refused, so an https endpoint cannot forward a request body to plain http.
170
280
 
281
+ Report vulnerabilities privately; see [SECURITY.md](SECURITY.md) for how and for supported
282
+ versions.
283
+
171
284
  ## Compatibility
172
285
 
173
- `openclaw.install.minHostVersion` and `openclaw.compat.pluginApi` are `>=2026.10.1`. Evidence
174
- (published `openclaw` npm tarballs and release tags, checked 2026-11-03):
286
+ ### Supported OpenClaw versions
287
+
288
+ OpenClaw `>=2026.10.10` is supported. Older hosts lack APIs or session behavior the plugin needs
289
+ (see [Host API evidence](#host-api-evidence)). Memory needs the embedded agent runtime; CLI
290
+ runtimes such as `claude-cli` get none yet (see [Limitations](#limitations)).
291
+
292
+ `scripts/compat-matrix.sh` runs the live proof on the floor host (2026.10.10), the latest
293
+ OpenClaw release and OpenClaw `main`, on the embedded and `claude-cli` runtimes. See
294
+ [docs/compat-matrix.md](docs/compat-matrix.md) for how to run it and the expected cells.
295
+
296
+ ### Host API evidence
175
297
 
298
+ `openclaw.install.minHostVersion` is `>=2026.10.10`; `openclaw.compat.pluginApi` is
299
+ `>=2026.10.1`. Evidence (published `openclaw` npm tarballs and release tags, checked 2026-11-03
300
+ and 2026-11-07):
301
+
302
+ - 2026.10.10 is the first release that records a dashboard session created by
303
+ `sessions.create` as a direct chat (openclaw#181359). On 2026.10.1 through 2026.10.9 the host
304
+ gives turns in such a session no memory audience ("root session ... has no durable chat
305
+ type"), so the plugin neither recalls nor retains there: no memory is read or written in those
306
+ sessions, including `openclaw agent --session-key` runs in them.
176
307
  - 2026.9.36 has no tools-arm flush plan (`persistenceToolNames`) and no
177
308
  `assertMemoryAudienceCurrent`. 2026.10.1 is the first release with `recallToolNames`, the
178
309
  tools-arm `providerFlushPlanResolver` result and `memoryAudience` /
@@ -187,6 +318,15 @@ Redirects are refused, so an https endpoint cannot forward a request body to pla
187
318
  is on `main` after 2026.10.10. On older hosts Memory Core's tools can appear next to ours when
188
319
  dreaming keeps it loaded.
189
320
 
321
+ ## Releases and versioning
322
+
323
+ The plugin follows [semver](https://semver.org/). Before 1.0, a patch release (0.1.x) carries
324
+ fixes and security fixes only, and a minor release (0.x.0) may change config or behavior, with an
325
+ upgrade note in [CHANGELOG.md](CHANGELOG.md). Raising the minimum OpenClaw version is at least a
326
+ minor release. Each release has a CHANGELOG entry, a `vX.Y.Z` tag on GitHub and the same version
327
+ on npm and ClawHub. Only the latest release gets fixes. A version with a known security problem is
328
+ deprecated on npm once the fix ships, as 0.1.1 is.
329
+
190
330
  ## Limitations
191
331
 
192
332
  - **No memory on CLI-backed runtimes.** OpenClaw's CLI runner (for example `claude-cli`)
@@ -194,7 +334,10 @@ Redirects are refused, so an https endpoint cannot forward a request body to pla
194
334
  as of 2026-11-07). On those turns auto-recall, auto-retain and the `hindsight_*` tools are
195
335
  skipped with "this session has no memory audience". Use an embedded model route for agents
196
336
  that need memory, or keep `@vectorize-io/hindsight-openclaw` for them until the host passes
197
- the audience on CLI turns.
337
+ the audience on CLI turns. The host bug is
338
+ [openclaw/openclaw#194453](https://github.com/openclaw/openclaw/issues/194453); the fix,
339
+ [openclaw/openclaw#194454](https://github.com/openclaw/openclaw/pull/194454), is in review and
340
+ not merged as of 2026-11-07.
198
341
  - **Sessions without a memory audience get no memory.** Recall, retain and the tools need a
199
342
  host-minted audience. The host mints one only for sessions whose root has a durable chat
200
343
  type: owner direct chats get the owner-private bank, groups and other senders get a
@@ -203,6 +346,12 @@ Redirects are refused, so an https endpoint cannot forward a request body to pla
203
346
  (the vendor plugin fell back to the agent bank). This is intended per the SDK contract ("a
204
347
  session authority without an audience has no private or conversation grant"). Run
205
348
  `openclaw doctor --fix` to stamp older dashboard roots.
349
+ - **Workspace memory files are outside the bank split.** The audience split covers Hindsight
350
+ banks only. An agent with file or shell tools can still write a fact to its workspace (for
351
+ example the daily note `memory/YYYY-MM-DD.md` that the default `AGENTS.md` suggests for
352
+ "remember this"), and any later session of that agent with the same tools can read it,
353
+ group chats included. To keep owner facts out of group sessions, sandbox non-main sessions
354
+ or deny `group:fs` and `group:runtime` for them in OpenClaw's tool policy.
206
355
  - **Auto-retain does not re-check audience currency.** The host starts `agent_end` handlers
207
356
  without awaiting them and releases the run's audience when the attempt ends, so a currency
208
357
  check there would race and drop retains. The bank still comes from the host-minted audience of
@@ -210,17 +359,18 @@ Redirects are refused, so an https endpoint cannot forward a request body to pla
210
359
  - **`memory.search` through the provider runtime skips the plugin filters.** It is gated by the
211
360
  caller's audience (owner-private or conversation), not by `memoryMode`, `excludeProviders` or
212
361
  incognito session keys. Those filters apply to auto-recall, auto-retain and the tools only.
213
- - `hindsightApiUrl` must not contain credentials, a query string or a fragment (the parser
214
- rejects it). An unresolved SecretRef token, or a token sent over plain `http` to a
215
- non-loopback host, logs one warning at service start.
362
+ - `hindsightApiUrl` must not contain credentials, a query string or a fragment, even an empty
363
+ one such as a trailing `?` or `#` (the parser rejects it). An unresolved SecretRef token, or a
364
+ token sent over plain `http` to a non-loopback host, logs one warning at service start.
216
365
  - Flush dedupe is per process; a restart mid-cycle can repeat a flush write, which the stable
217
366
  document id plus `update_mode: replace` turns into an overwrite.
218
367
  - Conversation banks are never pruned by the plugin, and `openclaw memory forget` does not reach
219
368
  slot owners (no host seam yet).
220
369
 
221
- Not ported from the vendor plugin: knowledge-base tools, tool-call retention
222
- (`retainToolCalls`), `retainEveryNTurns` windows, the offline retain queue, the local Hindsight
223
- daemon, and per-user/per-channel bank granularity (replaced by audience routing).
370
+ Not ported from the vendor plugin: knowledge-base write tools (create, update, delete; the
371
+ read tools are opt-in, see above), tool-call retention
372
+ (`retainToolCalls`), `retainEveryNTurns` windows, the local Hindsight daemon, and
373
+ per-user/per-channel bank granularity (replaced by audience routing).
224
374
 
225
375
  ## Rollback
226
376
 
@@ -243,5 +393,8 @@ npm run typecheck && npm run build && npm test && npm run lint && npm run format
243
393
  npm pack --dry-run # dist/*.js, openclaw.plugin.json, README, CHANGELOG, LICENSE only
244
394
  ```
245
395
 
246
- `scripts/live-proof.sh` runs the end-to-end proof against a throwaway Gateway; see
247
- [docs/proof/2026-11-03.md](docs/proof/2026-11-03.md).
396
+ `scripts/live-proof.sh` runs one OpenClaw host (npm version, git ref or path) on the embedded
397
+ or `claude-cli` runtime through a throwaway Gateway and real turns, and prints a PASS/FAIL
398
+ table. `scripts/compat-matrix.sh` runs it for the floor host, the latest release and `main`.
399
+ See [docs/compat-matrix.md](docs/compat-matrix.md), including the expected `claude-cli` cells
400
+ and an optional nightly launchd schedule.
package/dist/client.js CHANGED
@@ -1,14 +1,88 @@
1
1
  /** HTTP status from the Hindsight API; 404 on a bank nothing was retained to yet. */
2
2
  export class HindsightHttpError extends Error {
3
3
  status;
4
- constructor(message, status) {
4
+ bankMissing;
5
+ bankMissingExplicit;
6
+ pageMissing;
7
+ constructor(message, status,
8
+ /** True only for Hindsight's "Bank '<id>' not found" 404, not for a route that does not exist. */
9
+ bankMissing = false,
10
+ /** True only when the body said so; an empty 404 body sets `bankMissing` but not this. */
11
+ bankMissingExplicit = false,
12
+ /** True only for Hindsight's "Knowledge page '<id>' not found" 404. */
13
+ pageMissing = false) {
5
14
  super(message);
6
15
  this.status = status;
16
+ this.bankMissing = bankMissing;
17
+ this.bankMissingExplicit = bankMissingExplicit;
18
+ this.pageMissing = pageMissing;
7
19
  this.name = "HindsightHttpError";
8
20
  }
9
21
  }
10
22
  export function isBankNotFound(error) {
11
- return error instanceof HindsightHttpError && error.status === 404;
23
+ return error instanceof HindsightHttpError && error.bankMissing;
24
+ }
25
+ /**
26
+ * Hindsight answers a missing bank with 404 `{"detail":"Bank '<id>' not found"}` and an unknown
27
+ * route (wrong hindsightApiUrl path, proxy prefix) with 404 `{"detail":"Not Found"}`. Only the
28
+ * first means "no memories yet"; the second stays a loud error. An empty 404 body counts as a
29
+ * missing bank, as before. The body is only inspected, never copied into the error.
30
+ */
31
+ export function isBankMissingBody(text) {
32
+ return !text.trim() || isExplicitBankMissingBody(text);
33
+ }
34
+ /** Only Hindsight's `Bank '<id>' not found` detail, without the empty-body fallback. */
35
+ export function isExplicitBankMissingBody(text) {
36
+ try {
37
+ const detail = JSON.parse(text)?.detail;
38
+ return typeof detail === "string" && /\bbank\b.*\bnot found\b/i.test(detail);
39
+ }
40
+ catch {
41
+ return false;
42
+ }
43
+ }
44
+ /** Only Hindsight's `Knowledge page '<id>' not found` detail; a route "Not Found" does not count. */
45
+ export function isExplicitPageMissingBody(text) {
46
+ try {
47
+ const detail = JSON.parse(text)?.detail;
48
+ return typeof detail === "string" && /\bknowledge page\b.*\bnot found\b/i.test(detail);
49
+ }
50
+ catch {
51
+ return false;
52
+ }
53
+ }
54
+ /** Largest knowledge-base response body read; a bigger one fails instead of filling memory. */
55
+ export const KB_MAX_RESPONSE_BYTES = 1024 * 1024;
56
+ // Error bodies are only classified, never copied, so a small prefix is enough.
57
+ const ERROR_BODY_MAX_BYTES = 64 * 1024;
58
+ class BodyTooLargeError extends Error {
59
+ }
60
+ /** Reads at most `maxBytes` of the body; throws BodyTooLargeError past it and stops reading. */
61
+ async function readBoundedText(response, maxBytes) {
62
+ const declared = Number(response.headers.get("content-length"));
63
+ if (Number.isFinite(declared) && declared > maxBytes) {
64
+ await response.body?.cancel().catch(() => undefined);
65
+ throw new BodyTooLargeError();
66
+ }
67
+ if (!response.body) {
68
+ return "";
69
+ }
70
+ const reader = response.body.getReader();
71
+ const chunks = [];
72
+ let total = 0;
73
+ for (;;) {
74
+ const { done, value } = await reader.read();
75
+ if (done) {
76
+ break;
77
+ }
78
+ total += value.byteLength;
79
+ if (total > maxBytes) {
80
+ await reader.cancel().catch(() => undefined);
81
+ throw new BodyTooLargeError();
82
+ }
83
+ chunks.push(value);
84
+ }
85
+ return new TextDecoder().decode(Buffer.concat(chunks));
12
86
  }
13
87
  const HEALTH_TIMEOUT_MS = 5_000;
14
88
  /** A fixed timeout, combined with the host signal when there is one (host abort still wins). */
@@ -46,10 +120,31 @@ export class HindsightClient {
46
120
  ...(init.signal ? { signal: init.signal } : {}),
47
121
  });
48
122
  if (!response.ok) {
123
+ const body = response.status === 404
124
+ ? // An oversized body is not "empty", so it never reads as a missing bank.
125
+ await readBoundedText(response, ERROR_BODY_MAX_BYTES).catch((error) => error instanceof BodyTooLargeError ? "<oversized>" : "")
126
+ : "";
127
+ const bankMissing = response.status === 404 && isBankMissingBody(body);
128
+ const bankMissingExplicit = response.status === 404 && isExplicitBankMissingBody(body);
129
+ const pageMissing = response.status === 404 && isExplicitPageMissingBody(body);
49
130
  // Server error bodies can echo request content; keep only the status in the message.
50
- throw new HindsightHttpError(`Hindsight ${operation} failed: HTTP ${response.status}`, response.status);
131
+ throw new HindsightHttpError(`Hindsight ${operation} failed: HTTP ${response.status}`, response.status, bankMissing, bankMissingExplicit, pageMissing);
132
+ }
133
+ let text;
134
+ if (init.maxBodyBytes === undefined) {
135
+ text = await response.text();
136
+ }
137
+ else {
138
+ try {
139
+ text = await readBoundedText(response, init.maxBodyBytes);
140
+ }
141
+ catch (error) {
142
+ if (error instanceof BodyTooLargeError) {
143
+ throw new Error(`Hindsight ${operation} failed: response larger than ${init.maxBodyBytes} bytes`, { cause: error });
144
+ }
145
+ throw error;
146
+ }
51
147
  }
52
- const text = await response.text();
53
148
  // SAFETY: T only names optional fields; every caller type-checks each field before use.
54
149
  return (text ? JSON.parse(text) : {});
55
150
  }
@@ -88,7 +183,43 @@ export class HindsightClient {
88
183
  return typeof body.text === "string" ? { id: memoryId, text: body.text } : null;
89
184
  }
90
185
  catch (error) {
91
- if (isBankNotFound(error)) {
186
+ // Any 404 here: "Bank '<id>' not found" and "Memory unit '<id>' not found" both mean absent.
187
+ if (error instanceof HindsightHttpError && error.status === 404) {
188
+ return null;
189
+ }
190
+ throw error;
191
+ }
192
+ }
193
+ async getKnowledgeBaseTree(bankId, signal) {
194
+ const response = await this.request("knowledge tree", this.bankUrl(bankId, "/knowledge-base/tree"), { method: "GET", signal, maxBodyBytes: KB_MAX_RESPONSE_BYTES });
195
+ return Array.isArray(response.roots) ? response.roots : [];
196
+ }
197
+ async searchKnowledgeBase(bankId, query, options = {}) {
198
+ const params = new URLSearchParams({ q: query });
199
+ if (options.limit !== undefined) {
200
+ params.set("limit", String(options.limit));
201
+ }
202
+ const response = await this.request("knowledge search", this.bankUrl(bankId, `/knowledge-base/search?${params.toString()}`), { method: "GET", signal: options.signal, maxBodyBytes: KB_MAX_RESPONSE_BYTES });
203
+ return Array.isArray(response.results) ? response.results : [];
204
+ }
205
+ /**
206
+ * Returns null when the bank or page does not exist. Hindsight scopes page ids to the bank in
207
+ * the URL, so a page id from another bank is also null. A route 404 stays an error.
208
+ */
209
+ async getKnowledgePage(bankId, pageId, signal) {
210
+ try {
211
+ const body = await this.request("knowledge page", this.bankUrl(bankId, `/knowledge-base/pages/${encodeURIComponent(pageId)}`), { method: "GET", signal, maxBodyBytes: KB_MAX_RESPONSE_BYTES });
212
+ if (typeof body.markdown !== "string") {
213
+ throw new Error("Hindsight knowledge page failed: unexpected response");
214
+ }
215
+ return {
216
+ id: pageId,
217
+ name: typeof body.name === "string" ? body.name : "",
218
+ markdown: body.markdown,
219
+ };
220
+ }
221
+ catch (error) {
222
+ if (error instanceof HindsightHttpError && (error.bankMissing || error.pageMissing)) {
92
223
  return null;
93
224
  }
94
225
  throw error;