@etzelm/openclaw-memory-hindsight 0.3.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,57 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.5.0 (2026-11-10)
4
+
5
+ ### Upgrade notes
6
+
7
+ - No config change is needed. All new keys default off.
8
+ - The auto-recall query strip is always on. Recall queries change slightly, because host
9
+ boilerplate is no longer sent.
10
+ - The config schema rejects unknown keys. A config that uses `listToolsWithoutAudience`,
11
+ `ignoreSessionPatterns`, `recallOnlySessionPatterns`, `recallCacheTtlMs` or
12
+ `recallFollowupMaxWords` fails validation on 0.3.0 and 0.4.0. Remove those keys before a
13
+ downgrade.
14
+
15
+ ### Added
16
+
17
+ - `listToolsWithoutAudience` (default off): list the `hindsight_*` tools for agent sessions whose
18
+ tool list the host builds without a memory audience. OpenClaw 2026.11.x does this for claude-cli
19
+ turns (MCP tool projection), so those sessions had no Hindsight tools. A call still resolves its
20
+ bank only from the host-minted audience and fails without one.
21
+ - `ignoreSessionPatterns` and `recallOnlySessionPatterns` (session-key globs, both default `[]`).
22
+ An ignored session gets no auto-recall, no auto-retain and no `hindsight_*` tools. A
23
+ recall-only session gets auto-recall and the read tools (`hindsight_recall`,
24
+ `hindsight_reflect`, and the knowledge-base read tools when `knowledgeReadTools` is on) even if
25
+ `skipOperationalSessions` would skip it, but never retains. Ignore wins over recall-only;
26
+ `excludeProviders`, `temp:` and incognito sessions still win over both. Patterns never select
27
+ a bank. See README, Configuration.
28
+ - Auto-recall drops host boilerplate from the query (inter-session header, routed-by line,
29
+ messages-recorded banner, subagent context lines, cron tag, CLI-resumed and automation-repair
30
+ notices); a message that is only boilerplate no longer triggers a recall.
31
+ - `recallCacheTtlMs` (default `0`, off): per-session cache of auto-recall results, keyed by
32
+ session, bank and query. Never shared across sessions or banks.
33
+ - `recallFollowupMaxWords` (default `0`, off): short follow-ups skip auto-recall while the session
34
+ has a recent recall that returned memories, and otherwise recall with the previous user message
35
+ added to the query. An empty recall never enables the skip. Debug log tags each auto-recall
36
+ `fresh`, `cached`, `followup-skip` or `followup-expanded`.
37
+
38
+ ## 0.4.0 (2026-11-09)
39
+
40
+ ### Upgrade notes
41
+
42
+ - No config change is needed. The plugin now serves Prometheus metrics at
43
+ `GET /plugins/hindsight-native/metrics` on the Gateway port. To collect them, add a scrape job
44
+ for that path that sends a Gateway token with `operator.read`. Without a scrape job nothing
45
+ changes.
46
+
47
+ ### Added
48
+
49
+ - Gateway metrics: Prometheus text at `GET /plugins/hindsight-native/metrics` on the Gateway
50
+ port (Gateway auth, `operator.read`). Auto-recall and auto-retain counts by outcome with
51
+ latency histograms, and retain queue depth, replays and drops. Labels are `agent` and `bank`
52
+ plus fixed outcome and reason values; conversation banks share one label per owner bank.
53
+ See README, Gateway metrics.
54
+
3
55
  ## 0.3.0 (2026-11-08)
4
56
 
5
57
  ### Upgrade notes
package/README.md CHANGED
@@ -119,12 +119,26 @@ tool conflict.
119
119
  | `retainTags` | `[]` | Tags added to every retained memory (auto-retain, `hindsight_retain`, pre-compaction flush) |
120
120
  | `excludeProviders` | `[]` | Message providers that never recall or retain (for example `heartbeat`) |
121
121
  | `skipOperationalSessions` | `true` | Skip cron, heartbeat and subagent sessions |
122
+ | `ignoreSessionPatterns` | `[]` | Session-key globs with no recall, retain or `hindsight_*` tools (see below) |
123
+ | `recallOnlySessionPatterns` | `[]` | Session-key globs that recall and read but never retain (see below) |
122
124
  | `agentBankMap` | `{}` | Agent id to bank id, so several agents share one owner bank |
123
125
  | `allowInsecureHttp` | `false` | Allow plain http to a non-loopback `hindsightApiUrl` (token and conversation text unencrypted) |
124
126
  | `retainQueueEnabled` | `true` | Queue auto-retains that fail because Hindsight is unreachable and send them later (see below) |
125
127
  | `retainQueueMaxItems` | `500` | Most queued retains kept (1 to 10000); the oldest are dropped first |
126
128
  | `retainQueueMaxAgeMs` | `604800000` (7 days) | Queued retains older than this are dropped |
129
+ | `recallCacheTtlMs` | `0` (off) | Auto-recall cache per session and bank, 0 to 3600000 ms; a repeated query skips the API call |
130
+ | `recallFollowupMaxWords` | `0` (off) | At most N words, no URL or path: skip after a recent recall with hits, else add the prior user turn |
127
131
  | `knowledgeReadTools` | `false` | Offer the read-only knowledge-base tools (see below); turning it on needs a gateway restart |
132
+ | `listToolsWithoutAudience` | `false` | List the `hindsight_*` tools when the host builds the tool list without an audience (see below) |
133
+
134
+ Session patterns: `*` matches any run of characters except `:`, `**` matches anything including
135
+ `:`, everything else is literal. Patterns match the trimmed session key. `ignoreSessionPatterns`
136
+ sessions get no auto-recall, no auto-retain and no `hindsight_*` tools. `recallOnlySessionPatterns`
137
+ sessions (for example `agent:dev:subagent:**`) get auto-recall, `hindsight_recall`,
138
+ `hindsight_reflect` and, with `knowledgeReadTools`, the knowledge-base read tools, even when
139
+ `skipOperationalSessions` would skip them, but never auto-retain and no `hindsight_retain`.
140
+ `excludeProviders`, `temp:` and incognito sessions win over both lists, and ignore wins over
141
+ recall-only. A pattern never selects a bank; the bank still comes from the memory audience.
128
142
 
129
143
  Fixed values (not configurable): recall max tokens 1024, query cap 800 characters, recall
130
144
  timeout 10 s, retain and reflect timeout 30 s, memory get and knowledge-base tool timeout 10 s
@@ -135,6 +149,15 @@ prompt, so the cached prefix stays stable). Knowledge-base tool caps: a response
135
149
  the tree shows 500 entries, then `(truncated, N more entries)`; search shows at most 50 hits
136
150
  with snippets cut at 500 characters and names and ids at 200.
137
151
 
152
+ `listToolsWithoutAudience`: the claude-cli runtime on OpenClaw 2026.11.x builds a turn's MCP tool
153
+ list before it attaches the turn's memory audience, so every tool factory sees no audience, returns
154
+ null, and `hindsight_*` never reach claude-cli sessions (auto-recall still works; it reads the
155
+ audience from the hook context). With this key on, an agent session with a session key gets the
156
+ tools listed anyway. The host then builds the callable copy with the audience from the run's grant,
157
+ and that copy alone reaches Hindsight; a call that still has no audience fails with an error before
158
+ any request. The session filters (`memoryMode`, incognito, `excludeProviders`,
159
+ `skipOperationalSessions`) apply as before, and an audience for another agent is still refused.
160
+
138
161
  Config changes apply without a restart: the plugin re-reads its entry from the host's current
139
162
  config snapshot (`api.runtime.config.current()`), falling back to the register-time
140
163
  `api.pluginConfig`. One exception: `knowledgeReadTools` registers its tools at startup, so turning
@@ -236,6 +259,40 @@ Host tool listings can therefore show them while the key is off: the manifest's
236
259
  only test that a key is present, not that it is `true`, so they cannot express this key. The
237
260
  write tools (create folder, create page, update, delete) are not ported.
238
261
 
262
+ ### Gateway metrics
263
+
264
+ The plugin serves Prometheus text at `GET /plugins/hindsight-native/metrics` on the Gateway
265
+ port. The route uses Gateway auth and needs `operator.read` (or `operator.write` /
266
+ `operator.admin`), the same as `diagnostics-prometheus`, so scrape it with the token you use
267
+ for `/api/diagnostics/prometheus`. `diagnostics-prometheus` renders only core diagnostic
268
+ events and has no plugin metric surface, which is why this is a separate path. There is no
269
+ config key; counters live in memory and reset when the Gateway restarts.
270
+
271
+ | Series (`openclaw_hindsight_` prefix) | Type | Labels |
272
+ | ------------------------------------- | --------- | -------------------------- |
273
+ | `recall_total` | counter | `agent`, `bank`, `outcome` |
274
+ | `recall_duration_seconds` | histogram | `agent`, `bank` |
275
+ | `retain_total` | counter | `agent`, `bank`, `outcome` |
276
+ | `retain_duration_seconds` | histogram | `agent`, `bank` |
277
+ | `retain_queue_depth` | gauge | `bank` |
278
+ | `retain_queue_replays_total` | counter | `bank`, `outcome` |
279
+ | `retain_queue_drops_total` | counter | `bank`, `reason` |
280
+
281
+ - Recall counts the auto-recall request only: `injected`, `empty` (no results or no bank yet),
282
+ `timeout`, `error`. A recall the host cancels (audience changed, hook invocation stale) is
283
+ not counted. Retain counts auto-retain only: `sent`, `queued`, `timeout`, `error`. Tool
284
+ calls are not counted.
285
+ - The latency histograms cover every outcome, so timeouts and fast `queued` retains are in the
286
+ same quantiles as successful calls.
287
+ - Replays: `delivered`, or `deferred` (Hindsight unreachable, item kept). Drops count retains
288
+ that are lost: `limit` (`retainQueueMaxItems`, `retainQueueMaxAgeMs` or the size bound),
289
+ `too_large` (the live write failed and the item is over the queue's size bound), `rejected`
290
+ (a 4xx on replay).
291
+ - `agent` is the normalized agent id. `bank` is the owner bank; every conversation bank of
292
+ that owner shares one `<bank>::conv` label. Values that are not label-safe read `unknown`.
293
+ Past 200 label sets per metric family, new `agent` and `bank` values fold into `other`;
294
+ `outcome` and `reason` keep their value. No session ids, document ids or text.
295
+
239
296
  ### Mapping from `@vectorize-io/hindsight-openclaw`
240
297
 
241
298
  | Vendor key | Here |
package/dist/config.js CHANGED
@@ -1,6 +1,8 @@
1
1
  export const RECALL_BUDGETS = ["low", "mid", "high"];
2
2
  export const RECALL_TYPES = ["world", "experience", "observation"];
3
3
  export const MEMORY_MODES = ["hybrid", "context", "tools"];
4
+ export const MAX_RECALL_CACHE_TTL_MS = 3_600_000;
5
+ export const MAX_RECALL_FOLLOWUP_WORDS = 100;
4
6
  export const DEFAULT_RETAIN_QUEUE_MAX_ITEMS = 500;
5
7
  export const MAX_RETAIN_QUEUE_MAX_ITEMS = 10_000;
6
8
  export const DEFAULT_RETAIN_QUEUE_MAX_AGE_MS = 7 * 24 * 60 * 60 * 1000;
@@ -30,12 +32,17 @@ const CONFIG_KEYS = [
30
32
  "retainTags",
31
33
  "excludeProviders",
32
34
  "skipOperationalSessions",
35
+ "ignoreSessionPatterns",
36
+ "recallOnlySessionPatterns",
33
37
  "agentBankMap",
34
38
  "allowInsecureHttp",
35
39
  "retainQueueEnabled",
36
40
  "retainQueueMaxItems",
37
41
  "retainQueueMaxAgeMs",
38
42
  "knowledgeReadTools",
43
+ "listToolsWithoutAudience",
44
+ "recallCacheTtlMs",
45
+ "recallFollowupMaxWords",
39
46
  ];
40
47
  function isRecord(value) {
41
48
  return typeof value === "object" && value !== null && !Array.isArray(value);
@@ -93,6 +100,17 @@ function readPositiveInt(cfg, key, fallback, max = Number.MAX_SAFE_INTEGER) {
93
100
  }
94
101
  return value;
95
102
  }
103
+ /** Integer from 0 to max; 0 means off. */
104
+ function readNonNegativeInt(cfg, key, fallback, max) {
105
+ const value = cfg[key];
106
+ if (value === undefined) {
107
+ return fallback;
108
+ }
109
+ if (typeof value !== "number" || !Number.isInteger(value) || value < 0 || value > max) {
110
+ throw new Error(`${key} must be an integer from 0 to ${max}`);
111
+ }
112
+ return value;
113
+ }
96
114
  function readBankMap(cfg) {
97
115
  const value = cfg.agentBankMap;
98
116
  if (value === undefined) {
@@ -201,12 +219,17 @@ export const hindsightConfigSchema = {
201
219
  retainTags: readStringList(value, "retainTags") ?? [],
202
220
  excludeProviders: readStringList(value, "excludeProviders") ?? [],
203
221
  skipOperationalSessions: readBool(value, "skipOperationalSessions", true),
222
+ ignoreSessionPatterns: readStringList(value, "ignoreSessionPatterns") ?? [],
223
+ recallOnlySessionPatterns: readStringList(value, "recallOnlySessionPatterns") ?? [],
204
224
  agentBankMap: readBankMap(value),
205
225
  allowInsecureHttp,
206
226
  retainQueueEnabled: readBool(value, "retainQueueEnabled", true),
207
227
  retainQueueMaxItems: readPositiveInt(value, "retainQueueMaxItems", DEFAULT_RETAIN_QUEUE_MAX_ITEMS, MAX_RETAIN_QUEUE_MAX_ITEMS),
208
228
  retainQueueMaxAgeMs: readPositiveInt(value, "retainQueueMaxAgeMs", DEFAULT_RETAIN_QUEUE_MAX_AGE_MS),
209
229
  knowledgeReadTools: readBool(value, "knowledgeReadTools", false),
230
+ listToolsWithoutAudience: readBool(value, "listToolsWithoutAudience", false),
231
+ recallCacheTtlMs: readNonNegativeInt(value, "recallCacheTtlMs", 0, MAX_RECALL_CACHE_TTL_MS),
232
+ recallFollowupMaxWords: readNonNegativeInt(value, "recallFollowupMaxWords", 0, MAX_RECALL_FOLLOWUP_WORDS),
210
233
  };
211
234
  },
212
235
  };
package/dist/index.js CHANGED
@@ -6,15 +6,21 @@ import { asOptionalRecord } from "openclaw/plugin-sdk/string-coerce-runtime";
6
6
  import { isBankNotFound } from "./client.js";
7
7
  import { hasUnresolvedToken, hindsightConfigSchema, isPlainHttpRemote, RECALL_MAX_TOKENS, RECALL_TIMEOUT_MS, RETAIN_SOURCE, RETAIN_TIMEOUT_MS, } from "./config.js";
8
8
  import { createLearnContribution, createLegacyLearnFields, createProviderFlushPlanResolver, RECALL_TOOL_NAMES, } from "./memory-capability.js";
9
+ import { createMetricsRouteHandler, HindsightMetrics, isTimeoutError, METRICS_ROUTE_PATH, } from "./metrics.js";
9
10
  import { createHindsightProviderRuntime } from "./provider-runtime.js";
10
11
  import { isMaintenanceTurn, routeAudience, sessionUsesMemory } from "./routing.js";
11
12
  import { resolveLearnRegistrar } from "./sdk-types.js";
12
13
  import { createHindsightService } from "./service.js";
13
14
  import { createFlushDeduper, createKnowledgePageTool, createKnowledgeSearchTool, createKnowledgeTreeTool, createRecallTool, createReflectTool, createRetainTool, KB_PAGE_TOOL, KB_SEARCH_TOOL, KB_TREE_TOOL, RECALL_TOOL, REFLECT_TOOL, RETAIN_TOOL, } from "./tools.js";
14
- import { buildRecallContext, extractRecallQuery, isEphemeralOperationalText, prepareRetentionTranscript, retainStartIndex, runStartMarker, } from "./transcript.js";
15
+ import { buildRecallContext, expandFollowupQuery, extractRecallQuery, isEphemeralOperationalText, isShortFollowup, prepareRetentionTranscript, previousUserQuery, retainStartIndex, runStartMarker, } from "./transcript.js";
15
16
  export const PLUGIN_ID = "hindsight-native";
16
17
  // Bounds per-session run markers; one entry per active session is enough.
17
18
  const MAX_RUN_MARKERS = 1_000;
19
+ // Bounds the auto-recall cache; one entry per active session, oldest dropped first.
20
+ const MAX_RECALL_CACHE_SESSIONS = 512;
21
+ function queryDigest(query) {
22
+ return createHash("sha256").update(query).digest("hex");
23
+ }
18
24
  /**
19
25
  * Reads `plugins.entries[<id>].config` from a runtime config snapshot. Returns undefined when the
20
26
  * snapshot has no entry, so callers fall back to the register-time `api.pluginConfig`.
@@ -80,14 +86,31 @@ export default definePluginEntry({
80
86
  return startupConfig;
81
87
  }
82
88
  };
89
+ const metrics = new HindsightMetrics();
83
90
  const service = createHindsightService({
84
91
  config: currentConfig,
85
92
  logger: api.logger,
93
+ metrics,
94
+ });
95
+ // Gateway-side metrics on the gateway port, behind gateway auth (see metrics.ts for why this
96
+ // is a plugin route and not diagnostics-prometheus). The floor host has the method; the check
97
+ // only keeps a host or test double without it from failing registration.
98
+ if (typeof api.registerHttpRoute !== "function") {
99
+ api.logger.debug?.("memory-hindsight: host has no registerHttpRoute; metrics route not served");
100
+ }
101
+ api.registerHttpRoute?.({
102
+ path: METRICS_ROUTE_PATH,
103
+ auth: "gateway",
104
+ match: "exact",
105
+ gatewayRuntimeScopeSurface: "trusted-operator",
106
+ handler: createMetricsRouteHandler(metrics, () => service.retainQueueDepth()),
86
107
  });
87
108
  const flushDeduper = createFlushDeduper();
88
109
  // Per session: where the current run started, kept for the whole run (by runId) so a second
89
110
  // agent_end for the run (host continuation after a retry, or a re-delivery) re-sends all of it.
90
111
  const runStartBySession = new Map();
112
+ // Last auto-recall per session (never shared across sessions; the bank is checked on read).
113
+ const recallCache = new Map();
91
114
  const capability = {
92
115
  providerRuntime: createHindsightProviderRuntime({
93
116
  providerId: PLUGIN_ID,
@@ -152,6 +175,7 @@ export default definePluginEntry({
152
175
  !sessionUsesMemory(config, {
153
176
  sessionKey,
154
177
  messageProvider: ctx.messageProvider,
178
+ purpose: "recall",
155
179
  })) {
156
180
  return undefined;
157
181
  }
@@ -166,17 +190,71 @@ export default definePluginEntry({
166
190
  }
167
191
  try {
168
192
  ctx.assertMemoryAudienceCurrent?.();
169
- const results = await service.client.recall(route.bankId, query, {
170
- budget: config.recallBudget,
171
- maxTokens: RECALL_MAX_TOKENS,
172
- ...(config.recallTypes ? { types: config.recallTypes } : {}),
173
- ...(config.preferObservations === undefined
174
- ? {}
175
- : { preferObservations: config.preferObservations }),
176
- signal: AbortSignal.timeout(RECALL_TIMEOUT_MS),
177
- });
178
- ctx.assertMemoryAudienceCurrent?.();
179
- ctx.hookInvocation?.assertActive();
193
+ const ttl = config.recallCacheTtlMs;
194
+ const now = Date.now();
195
+ const cached = ttl > 0 && sessionKey ? recallCache.get(sessionKey) : undefined;
196
+ const live = cached && cached.bankId === route.bankId && now - cached.at < ttl;
197
+ const followup = isShortFollowup(query, config.recallFollowupMaxWords);
198
+ if (followup && live && cached.results.length > 0) {
199
+ api.logger.debug?.("memory-hindsight: auto-recall source=followup-skip");
200
+ return undefined;
201
+ }
202
+ let recallQuery = query;
203
+ let source = "fresh";
204
+ if (followup) {
205
+ const previous = previousUserQuery(event.messages, query);
206
+ if (previous) {
207
+ recallQuery = expandFollowupQuery(previous, query);
208
+ source = "followup-expanded";
209
+ }
210
+ }
211
+ const queryHash = queryDigest(recallQuery);
212
+ let results;
213
+ if (live && cached.queryHash === queryHash) {
214
+ results = cached.results;
215
+ source = "cached";
216
+ }
217
+ else {
218
+ // Metrics count the Hindsight request only. A host assert that throws (audience changed,
219
+ // hook invocation stale) is host lifecycle, not a recall outcome, so it records nothing.
220
+ const recallStarted = performance.now();
221
+ try {
222
+ results = await service.client.recall(route.bankId, recallQuery, {
223
+ budget: config.recallBudget,
224
+ maxTokens: RECALL_MAX_TOKENS,
225
+ ...(config.recallTypes ? { types: config.recallTypes } : {}),
226
+ ...(config.preferObservations === undefined
227
+ ? {}
228
+ : { preferObservations: config.preferObservations }),
229
+ signal: AbortSignal.timeout(RECALL_TIMEOUT_MS),
230
+ });
231
+ }
232
+ catch (error) {
233
+ metrics.recordRecall(ctx.agentId, route.bankId, isBankNotFound(error) ? "empty" : isTimeoutError(error) ? "timeout" : "error", performance.now() - recallStarted);
234
+ throw error;
235
+ }
236
+ const recallMs = performance.now() - recallStarted;
237
+ ctx.assertMemoryAudienceCurrent?.();
238
+ ctx.hookInvocation?.assertActive();
239
+ metrics.recordRecall(ctx.agentId, route.bankId, results.length === 0 ? "empty" : "injected", recallMs);
240
+ if (ttl > 0 && sessionKey) {
241
+ recallCache.delete(sessionKey);
242
+ recallCache.set(sessionKey, {
243
+ bankId: route.bankId,
244
+ queryHash,
245
+ results,
246
+ at: Date.now(),
247
+ });
248
+ if (recallCache.size > MAX_RECALL_CACHE_SESSIONS) {
249
+ recallCache.delete(recallCache.keys().next().value);
250
+ }
251
+ }
252
+ }
253
+ api.logger.debug?.(`memory-hindsight: auto-recall source=${source}`);
254
+ if (source === "cached") {
255
+ ctx.assertMemoryAudienceCurrent?.();
256
+ ctx.hookInvocation?.assertActive();
257
+ }
180
258
  if (results.length === 0) {
181
259
  return undefined;
182
260
  }
@@ -221,6 +299,7 @@ export default definePluginEntry({
221
299
  !sessionUsesMemory(config, {
222
300
  sessionKey,
223
301
  messageProvider: ctx.messageProvider,
302
+ purpose: "retain",
224
303
  })) {
225
304
  return;
226
305
  }
@@ -238,6 +317,7 @@ export default definePluginEntry({
238
317
  // so a continuation or a re-delivery replaces the document with the whole run.
239
318
  const runPart = sanitizeDocumentIdPart(runId, `${Date.now()}`);
240
319
  const documentId = `openclaw:${sanitizeDocumentIdPart(sessionKey, "session")}-${sessionKeyDigest(sessionKey)}:run:${runPart}`;
320
+ const retainStarted = performance.now();
241
321
  try {
242
322
  // No ctx.assertMemoryAudienceCurrent() here. The host starts agent_end handlers without
243
323
  // awaiting them (runAgentEndSideEffectsAsync -> void runAgentEnd, after awaiting the
@@ -264,9 +344,11 @@ export default definePluginEntry({
264
344
  ...(ctx.channelId ? { channel_id: ctx.channelId } : {}),
265
345
  },
266
346
  }, signal);
347
+ metrics.recordRetain(ctx.agentId, route.bankId, outcome, performance.now() - retainStarted);
267
348
  api.logger.debug?.(`memory-hindsight: ${outcome === "sent" ? "retained" : "queued"} ${retention.messageCount} messages (${route.audience.kind})`);
268
349
  }
269
350
  catch (error) {
351
+ metrics.recordRetain(ctx.agentId, route.bankId, isTimeoutError(error) ? "timeout" : "error", performance.now() - retainStarted);
270
352
  api.logger.warn?.(`memory-hindsight: auto-retain failed: ${formatErrorMessage(error)}`);
271
353
  }
272
354
  });
@@ -278,6 +360,7 @@ export default definePluginEntry({
278
360
  const key = ctx.sessionKey ?? event.sessionKey;
279
361
  if (key) {
280
362
  runStartBySession.delete(key);
363
+ recallCache.delete(key);
281
364
  }
282
365
  });
283
366
  api.registerService({
@@ -299,6 +382,7 @@ export default definePluginEntry({
299
382
  stop: () => {
300
383
  service.stop();
301
384
  runStartBySession.clear();
385
+ recallCache.clear();
302
386
  api.logger.info("memory-hindsight: stopped");
303
387
  },
304
388
  });
@@ -0,0 +1,197 @@
1
+ import { normalizeDiagnosticValue } from "openclaw/plugin-sdk/diagnostic-runtime";
2
+ import { getPluginRuntimeGatewayRequestScope } from "openclaw/plugin-sdk/plugin-runtime";
3
+ import { normalizeAgentIdStrict } from "openclaw/plugin-sdk/routing";
4
+ /**
5
+ * Gateway-side metrics for auto-recall, auto-retain and the retain queue, rendered as Prometheus
6
+ * text on a gateway-auth plugin route. diagnostics-prometheus renders only core diagnostic events
7
+ * and has no plugin metric surface, so the plugin serves its own series on the gateway port.
8
+ * Labels are agent and bank plus fixed enums; never session ids, document ids or text.
9
+ */
10
+ export const METRICS_ROUTE_PATH = "/plugins/hindsight-native/metrics";
11
+ const PREFIX = "openclaw_hindsight_";
12
+ // Seconds. Recall times out at 10 s, retain at 30 s.
13
+ export const LATENCY_BUCKETS = [0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30];
14
+ // Label sets per family before new agent and bank values fold into "other"; agents and banks come
15
+ // from config. Outcome and reason keep their value, so the bound grows by the enum size at most.
16
+ export const MAX_SERIES_PER_FAMILY = 200;
17
+ const OTHER = "other";
18
+ const FOLDED_LABELS = new Set(["agent", "bank"]);
19
+ /** Agent label: the normalized agent id, or "unknown" when it is missing or not label-safe. */
20
+ export function agentLabel(agentId) {
21
+ // The guard runs first: normalizing would turn a session-shaped value into a safe-looking id.
22
+ // The strict form never falls back to the default agent, so an odd id reads unknown, not main.
23
+ const safe = normalizeDiagnosticValue(agentId);
24
+ if (safe === "unknown") {
25
+ return safe;
26
+ }
27
+ const normalized = normalizeAgentIdStrict(safe);
28
+ return normalized.ok ? normalized.value : "unknown";
29
+ }
30
+ /**
31
+ * Bank label. A conversation bank (`<owner>::conv::<digest>`, one per root session) folds into
32
+ * `<owner>::conv`, so the label set stays as small as the configured agents and bank map.
33
+ */
34
+ export function bankLabel(bankId) {
35
+ if (!bankId) {
36
+ return "unknown";
37
+ }
38
+ const conv = bankId.indexOf("::conv::");
39
+ return normalizeDiagnosticValue(conv >= 0 ? `${bankId.slice(0, conv)}::conv` : bankId);
40
+ }
41
+ /** True for the error an `AbortSignal.timeout` fetch rejects with. */
42
+ export function isTimeoutError(error) {
43
+ return error?.name === "TimeoutError";
44
+ }
45
+ function labelKey(labels) {
46
+ return JSON.stringify(Object.entries(labels));
47
+ }
48
+ function escapeLabelValue(value) {
49
+ return value.replace(/\\/g, "\\\\").replace(/"/g, '\\"').replace(/\n/g, "\\n");
50
+ }
51
+ function formatLabels(labels, extra) {
52
+ const pairs = Object.entries(labels);
53
+ if (extra) {
54
+ pairs.push(extra);
55
+ }
56
+ if (pairs.length === 0) {
57
+ return "";
58
+ }
59
+ return `{${pairs.map(([k, v]) => `${k}="${escapeLabelValue(v)}"`).join(",")}}`;
60
+ }
61
+ /** Bounded map of label set to state; past the bound every new set lands on the "other" set. */
62
+ class Series {
63
+ create;
64
+ entries = new Map();
65
+ constructor(create) {
66
+ this.create = create;
67
+ }
68
+ get(labels) {
69
+ let key = labelKey(labels);
70
+ let entry = this.entries.get(key);
71
+ if (!entry) {
72
+ if (this.entries.size >= MAX_SERIES_PER_FAMILY) {
73
+ labels = Object.fromEntries(Object.entries(labels).map(([k, v]) => [k, FOLDED_LABELS.has(k) ? OTHER : v]));
74
+ key = labelKey(labels);
75
+ entry = this.entries.get(key);
76
+ }
77
+ if (!entry) {
78
+ entry = { labels, value: this.create() };
79
+ this.entries.set(key, entry);
80
+ }
81
+ }
82
+ return entry.value;
83
+ }
84
+ values() {
85
+ return [...this.entries.values()];
86
+ }
87
+ }
88
+ function newHistogram() {
89
+ return { buckets: LATENCY_BUCKETS.map(() => 0), sum: 0, count: 0 };
90
+ }
91
+ function formatNumber(value) {
92
+ return Number.isFinite(value) ? String(value) : value > 0 ? "+Inf" : "-Inf";
93
+ }
94
+ export class HindsightMetrics {
95
+ recalls = new Series(() => ({ n: 0 }));
96
+ recallLatency = new Series(newHistogram);
97
+ retains = new Series(() => ({ n: 0 }));
98
+ retainLatency = new Series(newHistogram);
99
+ replays = new Series(() => ({ n: 0 }));
100
+ drops = new Series(() => ({ n: 0 }));
101
+ // Kept across scrapes, so a bank whose queue drained reads 0 instead of disappearing.
102
+ depth = new Series(() => ({ n: 0 }));
103
+ recordRecall(agentId, bankId, outcome, ms) {
104
+ const base = { agent: agentLabel(agentId), bank: bankLabel(bankId) };
105
+ this.recalls.get({ ...base, outcome }).n += 1;
106
+ observe(this.recallLatency.get(base), ms);
107
+ }
108
+ recordRetain(agentId, bankId, outcome, ms) {
109
+ const base = { agent: agentLabel(agentId), bank: bankLabel(bankId) };
110
+ this.retains.get({ ...base, outcome }).n += 1;
111
+ observe(this.retainLatency.get(base), ms);
112
+ }
113
+ recordReplay(bankId, outcome) {
114
+ this.replays.get({ bank: bankLabel(bankId), outcome }).n += 1;
115
+ }
116
+ recordDrop(bankId, reason, count = 1) {
117
+ if (count > 0) {
118
+ this.drops.get({ bank: bankLabel(bankId), reason }).n += count;
119
+ }
120
+ }
121
+ /** Prometheus text exposition (version 0.0.4). Families with no samples print only HELP/TYPE. */
122
+ render(queueDepth) {
123
+ const lines = [];
124
+ const counter = (name, help, series) => {
125
+ lines.push(`# HELP ${PREFIX}${name} ${help}`, `# TYPE ${PREFIX}${name} counter`);
126
+ for (const { labels, value } of series.values()) {
127
+ lines.push(`${PREFIX}${name}${formatLabels(labels)} ${value.n}`);
128
+ }
129
+ };
130
+ const histogram = (name, help, series) => {
131
+ lines.push(`# HELP ${PREFIX}${name} ${help}`, `# TYPE ${PREFIX}${name} histogram`);
132
+ for (const { labels, value } of series.values()) {
133
+ LATENCY_BUCKETS.forEach((le, i) => {
134
+ lines.push(`${PREFIX}${name}_bucket${formatLabels(labels, ["le", String(le)])} ${value.buckets[i]}`);
135
+ });
136
+ lines.push(`${PREFIX}${name}_bucket${formatLabels(labels, ["le", "+Inf"])} ${value.count}`, `${PREFIX}${name}_sum${formatLabels(labels)} ${formatNumber(value.sum)}`, `${PREFIX}${name}_count${formatLabels(labels)} ${value.count}`);
137
+ }
138
+ };
139
+ counter("recall_total", "Auto-recall attempts by outcome.", this.recalls);
140
+ histogram("recall_duration_seconds", "Auto-recall latency.", this.recallLatency);
141
+ counter("retain_total", "Auto-retain attempts by outcome.", this.retains);
142
+ histogram("retain_duration_seconds", "Auto-retain latency.", this.retainLatency);
143
+ lines.push(`# HELP ${PREFIX}retain_queue_depth Retains waiting in the offline queue.`, `# TYPE ${PREFIX}retain_queue_depth gauge`);
144
+ for (const { value } of this.depth.values()) {
145
+ value.n = 0;
146
+ }
147
+ for (const [bankId, n] of queueDepth?.() ?? []) {
148
+ this.depth.get({ bank: bankLabel(bankId) }).n += n;
149
+ }
150
+ for (const { labels, value } of this.depth.values()) {
151
+ lines.push(`${PREFIX}retain_queue_depth${formatLabels(labels)} ${value.n}`);
152
+ }
153
+ counter("retain_queue_replays_total", "Queued retain replay attempts by outcome.", this.replays);
154
+ counter("retain_queue_drops_total", "Retains dropped from the queue by reason.", this.drops);
155
+ return `${lines.join("\n")}\n`;
156
+ }
157
+ }
158
+ function observe(h, ms) {
159
+ const seconds = Math.max(0, ms) / 1000;
160
+ LATENCY_BUCKETS.forEach((le, i) => {
161
+ if (seconds <= le) {
162
+ h.buckets[i] += 1;
163
+ }
164
+ });
165
+ h.sum += seconds;
166
+ h.count += 1;
167
+ }
168
+ // Mirrors diagnostics-prometheus: the route is gateway-auth, and read access is checked here so a
169
+ // scope-narrowed operator identity cannot read it.
170
+ const READ_SCOPES = ["operator.read", "operator.write", "operator.admin"];
171
+ function hasReadScope() {
172
+ const scopes = getPluginRuntimeGatewayRequestScope()?.client?.connect?.scopes;
173
+ return Array.isArray(scopes) && READ_SCOPES.some((scope) => scopes.includes(scope));
174
+ }
175
+ export function createMetricsRouteHandler(metrics, queueDepth, readScope = hasReadScope) {
176
+ return (req, res) => {
177
+ res.setHeader("Cache-Control", "no-store");
178
+ if (!readScope()) {
179
+ res.statusCode = 403;
180
+ res.setHeader("Content-Type", "text/plain; charset=utf-8");
181
+ res.end("missing scope: operator.read");
182
+ return true;
183
+ }
184
+ if (req.method !== "GET" && req.method !== "HEAD") {
185
+ res.statusCode = 405;
186
+ res.setHeader("Allow", "GET, HEAD");
187
+ res.end("Method Not Allowed");
188
+ return true;
189
+ }
190
+ const body = metrics.render(queueDepth);
191
+ res.statusCode = 200;
192
+ res.setHeader("Content-Type", "text/plain; version=0.0.4; charset=utf-8");
193
+ res.setHeader("Content-Length", String(Buffer.byteLength(body, "utf8")));
194
+ res.end(req.method === "HEAD" ? undefined : body);
195
+ return true;
196
+ };
197
+ }
@@ -131,10 +131,12 @@ export class RetainQueue {
131
131
  fileBytes = 0;
132
132
  limits;
133
133
  logger;
134
+ onDrop;
134
135
  constructor(options) {
135
136
  this.options = options;
136
137
  this.limits = options.limits;
137
138
  this.logger = options.logger;
139
+ this.onDrop = options.onDrop;
138
140
  }
139
141
  get filePath() {
140
142
  return this.options.filePath;
@@ -143,6 +145,7 @@ export class RetainQueue {
143
145
  configure(params) {
144
146
  this.limits = params.limits;
145
147
  this.logger = params.logger;
148
+ this.onDrop = params.onDrop;
146
149
  }
147
150
  now() {
148
151
  return this.options.now?.() ?? Date.now();
@@ -224,6 +227,21 @@ export class RetainQueue {
224
227
  peek() {
225
228
  return this.stored[0]?.entry;
226
229
  }
230
+ /**
231
+ * True when the item alone is over the size bound, so `enqueue` refuses it. Not reported as a
232
+ * drop here: the caller may still send it live, and reports it only when it is lost.
233
+ */
234
+ isTooLarge(entry) {
235
+ return stored({ ...entry, queuedAt: this.now() }).size > this.maxBytes();
236
+ }
237
+ /** Queued items per bank id. */
238
+ depthByBank() {
239
+ const depth = new Map();
240
+ for (const { entry } of this.stored) {
241
+ depth.set(entry.bankId, (depth.get(entry.bankId) ?? 0) + 1);
242
+ }
243
+ return depth;
244
+ }
227
245
  /** Bank and document id of every queued item, oldest first. */
228
246
  list() {
229
247
  return this.stored.map(({ entry }) => ({ bankId: entry.bankId, documentId: entry.documentId }));
@@ -264,7 +282,10 @@ export class RetainQueue {
264
282
  }
265
283
  let rewriteError;
266
284
  try {
267
- if (this.prune(false) === 0 &&
285
+ // Drops are reported only once the write that makes them stick succeeds; the fallback below
286
+ // restores memory and prunes again, and reports then.
287
+ const pruned = this.pruneStored();
288
+ if (pruned.length === 0 &&
268
289
  index < 0 &&
269
290
  existsSync(this.filePath) &&
270
291
  this.canAppend(next.size)) {
@@ -272,6 +293,7 @@ export class RetainQueue {
272
293
  return true;
273
294
  }
274
295
  this.persist();
296
+ this.reportDrops(pruned);
275
297
  // False when the bound dropped the item just added or replaced.
276
298
  return this.stored.includes(next);
277
299
  }
@@ -285,7 +307,8 @@ export class RetainQueue {
285
307
  // older one, and a tombstone for every item the bounds drop, so no dropped line can come back
286
308
  // on a restart (say, after a later tombstone brings the count under maxItems).
287
309
  this.stored = [...kept, next];
288
- this.prune(false);
310
+ // Reported only if the append below sticks; otherwise memory goes back to `kept`.
311
+ const prunedAgain = this.pruneStored();
289
312
  const keepsNext = this.stored.includes(next);
290
313
  const dead = kept.filter((s) => !this.stored.includes(s)).map((s) => s.entry);
291
314
  if (!keepsNext && index >= 0) {
@@ -302,6 +325,7 @@ export class RetainQueue {
302
325
  if (text) {
303
326
  this.append(text);
304
327
  }
328
+ this.reportDrops(prunedAgain);
305
329
  return keepsNext;
306
330
  }
307
331
  catch {
@@ -345,9 +369,17 @@ export class RetainQueue {
345
369
  * stays quiet until the queue is empty again.
346
370
  */
347
371
  prune(persist = true) {
372
+ const dropped = this.pruneStored();
373
+ this.reportDrops(dropped);
374
+ if (dropped.length > 0 && persist) {
375
+ this.persist();
376
+ }
377
+ return dropped.length;
378
+ }
379
+ /** The bounds step of `prune()`: updates memory and warns, but neither reports nor writes. */
380
+ pruneStored() {
348
381
  const { maxItems, maxAgeMs } = this.limits();
349
382
  const cutoff = this.now() - maxAgeMs;
350
- const before = this.stored.length;
351
383
  let kept = this.stored.filter((s) => s.entry.queuedAt >= cutoff);
352
384
  if (kept.length > maxItems) {
353
385
  kept = kept.slice(kept.length - maxItems);
@@ -356,19 +388,20 @@ export class RetainQueue {
356
388
  while (kept.length > 0 && bytes > this.maxBytes()) {
357
389
  bytes -= kept.shift().size;
358
390
  }
391
+ const keptSet = new Set(kept);
392
+ const dropped = this.stored.filter((s) => !keptSet.has(s));
359
393
  this.stored = kept;
360
- const dropped = before - kept.length;
361
- if (dropped > 0) {
362
- if (!this.dropWarned) {
363
- this.dropWarned = true;
364
- this.logger.warn?.(`memory-hindsight: retain queue dropped ${dropped} item(s) over retainQueueMaxItems (${maxItems}), retainQueueMaxAgeMs (${maxAgeMs}) or the ${this.maxBytes()}-byte size bound; further drops are not logged until the queue is empty`);
365
- }
366
- if (persist) {
367
- this.persist();
368
- }
394
+ if (dropped.length > 0 && !this.dropWarned) {
395
+ this.dropWarned = true;
396
+ this.logger.warn?.(`memory-hindsight: retain queue dropped ${dropped.length} item(s) over retainQueueMaxItems (${maxItems}), retainQueueMaxAgeMs (${maxAgeMs}) or the ${this.maxBytes()}-byte size bound; further drops are not logged until the queue is empty`);
369
397
  }
370
398
  return dropped;
371
399
  }
400
+ reportDrops(dropped) {
401
+ for (const s of dropped) {
402
+ this.onDrop?.(s.entry.bankId, "limit");
403
+ }
404
+ }
372
405
  ensureDir() {
373
406
  const dir = dirname(this.filePath);
374
407
  mkdirSync(dir, { recursive: true, mode: 0o700 });
@@ -526,6 +559,7 @@ export class RetainQueueRunner {
526
559
  return delivered;
527
560
  }
528
561
  if (!isDroppingRejection(error)) {
562
+ this.options.onReplay?.(entry.bankId, "deferred");
529
563
  // Not an answer about this item (outage, auth, rate limit, ambiguous 404, refused
530
564
  // redirect, or an unexpected error): keep it and back off.
531
565
  if (!isTransportFailure(error) && !this.rejectionWarned) {
@@ -540,6 +574,7 @@ export class RetainQueueRunner {
540
574
  }
541
575
  // Hindsight answered 4xx about this item: retrying cannot help, so it leaves the queue.
542
576
  this.queue.remove(entry, { persist: false });
577
+ this.options.onDrop?.(entry.bankId, "rejected");
543
578
  logger.warn?.(`memory-hindsight: queued retain rejected and dropped: ${formatErrorMessage(error)}`);
544
579
  continue;
545
580
  }
@@ -548,6 +583,7 @@ export class RetainQueueRunner {
548
583
  return delivered;
549
584
  }
550
585
  this.queue.remove(entry, { persist: false });
586
+ this.options.onReplay?.(entry.bankId, "delivered");
551
587
  this.rejectionWarned = false;
552
588
  delivered += 1;
553
589
  if (delivered % SAVE_EVERY === 0) {
package/dist/routing.js CHANGED
@@ -43,15 +43,60 @@ export function routeAudience(agentId, audience, bankMap) {
43
43
  }
44
44
  const OPERATIONAL_SESSION_RE = /^agent:[^:]+:(cron|heartbeat|subagent):/;
45
45
  const OPERATIONAL_PROVIDERS = new Set(["cron", "heartbeat", "subagent"]);
46
- /** Session filter shared by auto-recall, auto-retain and tool registration. */
47
- export function sessionUsesMemory(config, params) {
48
- const { sessionKey, messageProvider } = params;
46
+ const globCache = new Map();
47
+ /**
48
+ * Session-key glob: `*` matches any run of characters except ":", `**` matches anything
49
+ * including ":", everything else is literal.
50
+ */
51
+ function sessionGlobRegExp(glob) {
52
+ let re = globCache.get(glob);
53
+ if (!re) {
54
+ let source = "";
55
+ for (let i = 0; i < glob.length; i += 1) {
56
+ const ch = glob[i];
57
+ if (ch === "*") {
58
+ if (glob[i + 1] === "*") {
59
+ source += ".*";
60
+ i += 1;
61
+ }
62
+ else {
63
+ source += "[^:]*";
64
+ }
65
+ }
66
+ else {
67
+ source += ch.replace(/[.+?^${}()|[\]\\]/g, "\\$&");
68
+ }
69
+ }
70
+ re = new RegExp(`^${source}$`, "s");
71
+ globCache.set(glob, re);
72
+ }
73
+ return re;
74
+ }
75
+ /** True when the trimmed session key matches any glob; a missing key matches nothing. */
76
+ export function matchesSessionPattern(sessionKey, globs) {
77
+ const key = sessionKey?.trim();
78
+ return Boolean(key) && globs.some((glob) => sessionGlobRegExp(glob).test(key));
79
+ }
80
+ /**
81
+ * Session filter shared by auto-recall, auto-retain and tool registration. Patterns only
82
+ * decide whether a session uses memory; they never select a bank.
83
+ */
84
+ export function sessionUsesMemory(config,
85
+ // A missing purpose is treated as "retain", the stricter side.
86
+ params) {
87
+ const { sessionKey, messageProvider, purpose } = params;
49
88
  if (messageProvider && config.excludeProviders.includes(messageProvider)) {
50
89
  return false;
51
90
  }
52
91
  if (sessionKey?.startsWith("temp:")) {
53
92
  return false;
54
93
  }
94
+ if (matchesSessionPattern(sessionKey, config.ignoreSessionPatterns ?? [])) {
95
+ return false;
96
+ }
97
+ if (matchesSessionPattern(sessionKey, config.recallOnlySessionPatterns ?? [])) {
98
+ return purpose === "recall";
99
+ }
55
100
  if (!config.skipOperationalSessions) {
56
101
  return true;
57
102
  }
package/dist/service.js CHANGED
@@ -3,6 +3,12 @@ import { boundedSignal, HindsightClient } from "./client.js";
3
3
  import { RETAIN_TIMEOUT_MS } from "./config.js";
4
4
  import { BankSetupError, isDroppingRejection, isTransportFailure, RetainQueueRunner, retainQueuePath, sharedRetainQueue, } from "./retain-queue.js";
5
5
  export function createHindsightService(params) {
6
+ const { metrics } = params;
7
+ // Queue and replay observers feed the gateway metrics; none when the caller passes no metrics.
8
+ const queueObserver = metrics ? { onDrop: metrics.recordDrop.bind(metrics) } : {};
9
+ const runnerObserver = metrics
10
+ ? { onDrop: metrics.recordDrop.bind(metrics), onReplay: metrics.recordReplay.bind(metrics) }
11
+ : {};
6
12
  const preparedBanks = new Map();
7
13
  let stopped = false;
8
14
  let runner;
@@ -71,6 +77,10 @@ export function createHindsightService(params) {
71
77
  // Read again after the await: the service may have stopped or restarted meanwhile.
72
78
  const target = queue;
73
79
  if (!target || refused || !isTransportFailure(error)) {
80
+ // Queue-first path: the queue refused the item and the live write failed, so it is lost.
81
+ if (target && refused && isTransportFailure(error) && target.isTooLarge(entry)) {
82
+ metrics?.recordDrop(bankId, "too_large");
83
+ }
74
84
  throw error;
75
85
  }
76
86
  let queued;
@@ -82,6 +92,11 @@ export function createHindsightService(params) {
82
92
  throw error;
83
93
  }
84
94
  if (!queued) {
95
+ // Lost: the live write failed and the queue refused it. A bound that dropped the new item
96
+ // is already counted as `limit` by the queue.
97
+ if (target.isTooLarge(entry)) {
98
+ metrics?.recordDrop(bankId, "too_large");
99
+ }
85
100
  throw error;
86
101
  }
87
102
  params.logger.warn?.(`memory-hindsight: Hindsight unreachable, retain queued (${target.size()} pending): ${formatErrorMessage(error)}`);
@@ -109,7 +124,11 @@ export function createHindsightService(params) {
109
124
  return { maxItems: config.retainQueueMaxItems, maxAgeMs: config.retainQueueMaxAgeMs };
110
125
  };
111
126
  try {
112
- queue = sharedRetainQueue(retainQueuePath(stateDir), { limits, logger: params.logger });
127
+ queue = sharedRetainQueue(retainQueuePath(stateDir), {
128
+ limits,
129
+ logger: params.logger,
130
+ ...queueObserver,
131
+ });
113
132
  }
114
133
  catch (error) {
115
134
  queue = undefined;
@@ -131,6 +150,7 @@ export function createHindsightService(params) {
131
150
  await client.retain(entry.bankId, { ...entry.item, document_id: entry.documentId, update_mode: "replace" }, { signal });
132
151
  },
133
152
  logger: params.logger,
153
+ ...runnerObserver,
134
154
  ...params.retainQueueRetry,
135
155
  });
136
156
  if (queue.size() > 0) {
@@ -139,6 +159,7 @@ export function createHindsightService(params) {
139
159
  }
140
160
  },
141
161
  retainQueue: () => runner,
162
+ retainQueueDepth: () => queue?.depthByBank(),
142
163
  stopped: () => stopped,
143
164
  stop() {
144
165
  stopped = true;
package/dist/tools.js CHANGED
@@ -24,17 +24,43 @@ function requiredString(params, name) {
24
24
  }
25
25
  return value.trim();
26
26
  }
27
+ export const NO_AUDIENCE_AT_CALL = "This session has no host memory audience for this call, so Hindsight memory is unavailable.";
28
+ /**
29
+ * A tool listed for a session whose tool list the host built without a memory audience. Reading
30
+ * `bankId` throws, so `execute` fails before any request; the host builds the callable tool again
31
+ * with the session's audience, and that copy gets a real bank.
32
+ */
33
+ function deferredBank() {
34
+ return {
35
+ get bankId() {
36
+ throw new Error(NO_AUDIENCE_AT_CALL);
37
+ },
38
+ };
39
+ }
27
40
  /**
28
41
  * Resolves the bank for one tool context, or null when the session must not get the tool.
29
42
  * The bank comes only from the host-minted audience; the model never names a bank.
30
43
  */
31
- function resolveToolBank(service, ctx) {
44
+ function resolveToolBank(service, ctx, purpose) {
32
45
  const config = service.config();
33
46
  if (config.memoryMode === "context" ||
34
47
  isIncognitoSessionKey(ctx.sessionKey) ||
35
- !sessionUsesMemory(config, { sessionKey: ctx.sessionKey, messageProvider: ctx.messageChannel })) {
48
+ !sessionUsesMemory(config, {
49
+ sessionKey: ctx.sessionKey,
50
+ messageProvider: ctx.messageChannel,
51
+ purpose,
52
+ })) {
36
53
  return null;
37
54
  }
55
+ // Some hosts list a session's tools before they attach its audience (claude-cli MCP tool
56
+ // projection on OpenClaw 2026.11.x). Opt-in: list the tool for an agent session with a key, and
57
+ // let only the audience-bearing copy built for the call reach Hindsight.
58
+ if (config.listToolsWithoutAudience &&
59
+ !ctx.memoryAudience &&
60
+ ctx.agentId?.trim() &&
61
+ ctx.sessionKey?.trim()) {
62
+ return deferredBank();
63
+ }
38
64
  const route = routeAudience(ctx.agentId, ctx.memoryAudience, config.agentBankMap);
39
65
  return route.ok ? { bankId: route.bankId } : null;
40
66
  }
@@ -69,7 +95,7 @@ export function flushDocumentId(flushId, content) {
69
95
  return `openclaw-flush:${digest.slice(0, 32)}`;
70
96
  }
71
97
  export function createRetainTool(service, deduper, ctx) {
72
- const bank = resolveToolBank(service, ctx);
98
+ const bank = resolveToolBank(service, ctx, "retain");
73
99
  if (!bank) {
74
100
  return null;
75
101
  }
@@ -134,7 +160,7 @@ export function createRetainTool(service, deduper, ctx) {
134
160
  };
135
161
  }
136
162
  export function createRecallTool(service, ctx) {
137
- const bank = resolveToolBank(service, ctx);
163
+ const bank = resolveToolBank(service, ctx, "recall");
138
164
  if (!bank) {
139
165
  return null;
140
166
  }
@@ -176,7 +202,7 @@ export function createRecallTool(service, ctx) {
176
202
  };
177
203
  }
178
204
  export function createReflectTool(service, ctx) {
179
- const bank = resolveToolBank(service, ctx);
205
+ const bank = resolveToolBank(service, ctx, "recall");
180
206
  if (!bank) {
181
207
  return null;
182
208
  }
@@ -220,7 +246,7 @@ export const KB_NAME_MAX_CHARS = 200;
220
246
  const KB_TREE_MAX_DEPTH = 32;
221
247
  /** The knowledge tools are opt-in and otherwise follow the recall tools' bank routing. */
222
248
  function resolveKnowledgeBank(service, ctx) {
223
- return service.config().knowledgeReadTools ? resolveToolBank(service, ctx) : null;
249
+ return service.config().knowledgeReadTools ? resolveToolBank(service, ctx, "recall") : null;
224
250
  }
225
251
  /** Server text as one line, so a name or snippet cannot forge extra list entries. */
226
252
  function oneLine(value, maxChars = Number.MAX_SAFE_INTEGER) {
@@ -116,22 +116,89 @@ export function isEphemeralOperationalText(text) {
116
116
  return OPERATIONAL_TEXT_PATTERNS.some((pattern) => pattern.test(normalized));
117
117
  }
118
118
  const MIN_QUERY_CHARS = 5;
119
+ // Optional host timestamp prefix some of these lines carry, as in `[Wed 2026-10-28 19:24 PDT] `.
120
+ const BOILERPLATE_TS = "(?:\\[(?:Mon|Tue|Wed|Thu|Fri|Sat|Sun)[^\\]\\n]*\\]\\s*)?";
121
+ const BOILERPLATE_PATTERNS = [
122
+ // Inter-session header, routed-by line, subagent tags, banners and scheduler notices.
123
+ /^\[Inter-session message\][^\n]*(?:\n|$)/gm,
124
+ /^This content was routed by OpenClaw[^\n]*(?:\n|$)/gm,
125
+ /^\[OpenClaw: \d+ (?:messages recorded|earlier messages|messages occurred)[^\n]*\](?:\n|$)/gm,
126
+ /^\[Subagent Context\][^\n]*(?:\n|$)/gm,
127
+ /^Every subagent in this batch[^\n]*(?:\n|$)/gm,
128
+ // Only the tag goes; the reminder text after it stays.
129
+ /^\[cron:[0-9a-f-]{8,} [^\]\n]*\][ \t]*/gm,
130
+ new RegExp(`^${BOILERPLATE_TS}OpenClaw resumed this CLI session[^\\n]*(?:\\n|$)`, "gm"),
131
+ new RegExp(`^${BOILERPLATE_TS}Automation repair request from the scheduler, not a user message\\.[^\\n]*(?:\\n|$)`, "gm"),
132
+ ];
133
+ /** Removes host-generated boilerplate lines that carry no user intent. */
134
+ export function stripRecallBoilerplate(text) {
135
+ let out = text;
136
+ for (const pattern of BOILERPLATE_PATTERNS) {
137
+ out = out.replace(pattern, "");
138
+ }
139
+ return out.trim();
140
+ }
141
+ /** Cleaned recall text for one message, or null when too short or not user text. */
142
+ function cleanRecallCandidate(candidate) {
143
+ let cleaned = stripRecallBoilerplate(cleanMessageText(candidate));
144
+ cleaned = cleaned.replace(/^(?:System:.*\n)+\n?/, "");
145
+ cleaned = cleaned.replace(/\n\[from:[^\]]*\]\s*$/, "").trim();
146
+ if (cleaned.length >= MIN_QUERY_CHARS && !/^\s*system:/i.test(cleaned)) {
147
+ return cleaned.slice(0, RECALL_MAX_QUERY_CHARS);
148
+ }
149
+ return null;
150
+ }
119
151
  /** The user's current request with host envelopes removed and capped, or null when too short. */
120
152
  export function extractRecallQuery(currentUserMessage, prompt) {
121
- const candidates = [currentUserMessage, prompt];
122
- for (const candidate of candidates) {
153
+ for (const candidate of [currentUserMessage, prompt]) {
123
154
  if (!candidate) {
124
155
  continue;
125
156
  }
126
- let cleaned = cleanMessageText(candidate);
127
- cleaned = cleaned.replace(/^(?:System:.*\n)+\n?/, "");
128
- cleaned = cleaned.replace(/\n\[from:[^\]]*\]\s*$/, "").trim();
129
- if (cleaned.length >= MIN_QUERY_CHARS && !/^\s*system:/i.test(cleaned)) {
130
- return cleaned.slice(0, RECALL_MAX_QUERY_CHARS);
157
+ const cleaned = cleanRecallCandidate(candidate);
158
+ if (cleaned) {
159
+ return cleaned;
131
160
  }
132
161
  }
133
162
  return null;
134
163
  }
164
+ /** True for a short message with no URL or path: it leans on the turn before it. */
165
+ export function isShortFollowup(query, maxWords) {
166
+ if (maxWords < 1 || /\/|~\/|http/i.test(query)) {
167
+ return false;
168
+ }
169
+ return query.split(/\s+/).filter(Boolean).length <= maxWords;
170
+ }
171
+ /**
172
+ * The user message before the current one, cleaned like the query. The host may or may not list
173
+ * the current message last, so a trailing user message equal to the query is skipped.
174
+ */
175
+ export function previousUserQuery(messages, query) {
176
+ if (!Array.isArray(messages)) {
177
+ return null;
178
+ }
179
+ let skippedCurrent = false;
180
+ for (let i = messages.length - 1; i >= 0; i--) {
181
+ const message = messages[i];
182
+ if (asMessage(message)?.role !== "user") {
183
+ continue;
184
+ }
185
+ const cleaned = cleanRecallCandidate(messageText(message));
186
+ if (!cleaned) {
187
+ continue;
188
+ }
189
+ if (!skippedCurrent && cleaned === query) {
190
+ skippedCurrent = true;
191
+ continue;
192
+ }
193
+ return isEphemeralOperationalText(cleaned) ? null : cleaned;
194
+ }
195
+ return null;
196
+ }
197
+ /** Previous user message plus the follow-up, capped with the follow-up kept whole. */
198
+ export function expandFollowupQuery(previous, followup) {
199
+ const room = RECALL_MAX_QUERY_CHARS - followup.length - 1;
200
+ return room > 0 ? `${previous.slice(0, room)}\n${followup}` : followup;
201
+ }
135
202
  function formatOccurred(start, end) {
136
203
  if (start && end) {
137
204
  return start === end ? ` [occurred: ${start}]` : ` [occurred: ${start} -> ${end}]`;
@@ -72,6 +72,16 @@
72
72
  "help": "Queued retains older than this are dropped (default 7 days)",
73
73
  "advanced": true
74
74
  },
75
+ "recallCacheTtlMs": {
76
+ "label": "Recall Cache TTL (ms)",
77
+ "help": "Serve a repeated auto-recall query from a per-session cache for this long; 0 turns the cache off",
78
+ "advanced": true
79
+ },
80
+ "recallFollowupMaxWords": {
81
+ "label": "Recall Follow-up Max Words",
82
+ "help": "Messages this short (no URL or path) skip auto-recall while the session has a recent non-empty recall, else recall with the previous user message added; 0 turns it off",
83
+ "advanced": true
84
+ },
75
85
  "retainMission": {
76
86
  "label": "Retain Mission",
77
87
  "help": "Applied to each bank before its first write: what Hindsight should extract",
@@ -86,6 +96,11 @@
86
96
  "label": "Knowledge Base Read Tools",
87
97
  "help": "Offer hindsight_get_knowledge_base_tree, hindsight_search_knowledge_base and hindsight_get_knowledge_page on the same bank as hindsight_recall",
88
98
  "advanced": true
99
+ },
100
+ "listToolsWithoutAudience": {
101
+ "label": "List Tools Without Audience",
102
+ "help": "List the hindsight_* tools when the host builds a session's tool list before it attaches the memory audience (claude-cli sessions on OpenClaw 2026.11.x). Calls still need the audience",
103
+ "advanced": true
89
104
  }
90
105
  },
91
106
  "configSchema": {
@@ -152,6 +167,22 @@
152
167
  "skipOperationalSessions": {
153
168
  "type": "boolean"
154
169
  },
170
+ "ignoreSessionPatterns": {
171
+ "type": "array",
172
+ "items": {
173
+ "type": "string"
174
+ },
175
+ "default": [],
176
+ "description": "Session-key globs that get no auto-recall, no auto-retain and no hindsight_* tools. `*` matches any run of characters except ':', `**` matches anything including ':'. Checked after excludeProviders, temp: and incognito, and before recallOnlySessionPatterns. Never selects a bank"
177
+ },
178
+ "recallOnlySessionPatterns": {
179
+ "type": "array",
180
+ "items": {
181
+ "type": "string"
182
+ },
183
+ "default": [],
184
+ "description": "Session-key globs that get auto-recall and the read tools (hindsight_recall, hindsight_reflect, knowledge-base read tools) even when skipOperationalSessions would skip them, but never auto-retain and no hindsight_retain tool. Same glob syntax as ignoreSessionPatterns. Never selects a bank"
185
+ },
155
186
  "retainQueueEnabled": {
156
187
  "type": "boolean",
157
188
  "default": true,
@@ -170,11 +201,30 @@
170
201
  "default": 604800000,
171
202
  "description": "Queued retains older than this many milliseconds are dropped (default 7 days)"
172
203
  },
204
+ "recallCacheTtlMs": {
205
+ "type": "integer",
206
+ "minimum": 0,
207
+ "maximum": 3600000,
208
+ "default": 0,
209
+ "description": "Auto-recall cache lifetime in milliseconds, per session and bank (same query within this time is served without an API call); 0 = off. Also bounds how long a recall counts for recallFollowupMaxWords"
210
+ },
211
+ "recallFollowupMaxWords": {
212
+ "type": "integer",
213
+ "minimum": 0,
214
+ "maximum": 100,
215
+ "default": 0,
216
+ "description": "A message with at most this many words and no URL or path is a short follow-up. Within recallCacheTtlMs of a recall that returned memories it skips auto-recall; otherwise the query is the previous user message plus the follow-up. Needs recallCacheTtlMs > 0 to skip; 0 = off"
217
+ },
173
218
  "knowledgeReadTools": {
174
219
  "type": "boolean",
175
220
  "default": false,
176
221
  "description": "Offer the read-only knowledge-base tools (tree, search, page) on the session's own memory bank"
177
222
  },
223
+ "listToolsWithoutAudience": {
224
+ "type": "boolean",
225
+ "default": false,
226
+ "description": "List the hindsight_* tools for an agent session whose tool list the host builds without its memory audience (claude-cli MCP tool projection on OpenClaw 2026.11.x). The bank still comes only from the audience: a call without one fails before any request"
227
+ },
178
228
  "agentBankMap": {
179
229
  "type": "object",
180
230
  "additionalProperties": {
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@etzelm/openclaw-memory-hindsight",
3
- "version": "0.3.0",
3
+ "version": "0.5.0",
4
4
  "description": "Native Hindsight memory engine for OpenClaw: memory slot provider runtime, auto-recall, auto-retain, pre-compaction flush and hindsight_* agent tools.",
5
5
  "keywords": [
6
6
  "hindsight",