@etzelm/openclaw-memory-hindsight 0.2.0 → 0.4.0

package/CHANGELOG.md

@@ -1,5 +1,99 @@
 # Changelog
 
+## 0.4.0 (2026-11-09)
+
+### Upgrade notes
+
+- No config change is needed. The plugin now serves Prometheus metrics at
+  `GET /plugins/hindsight-native/metrics` on the Gateway port. To collect them, add a scrape job
+  for that path that sends a Gateway token with `operator.read`. Without a scrape job nothing
+  changes.
+
+### Added
+
+- Gateway metrics: Prometheus text at `GET /plugins/hindsight-native/metrics` on the Gateway
+  port (Gateway auth, `operator.read`). Auto-recall and auto-retain counts by outcome with
+  latency histograms, and retain queue depth, replays and drops. Labels are `agent` and `bank`
+  plus fixed outcome and reason values; conversation banks share one label per owner bank.
+  See README, Gateway metrics.
+
+## 0.3.0 (2026-11-08)
+
+### Upgrade notes
+
+- No config change is needed. The new `knowledgeReadTools` key is off by default, so the
+  registered tools are the same as in 0.2.0 unless you turn it on.
+- The retain queue file format is unchanged. See Downgrade notes before going back to 0.2.0.
+- Recalled memories that contain text shaped like the `<hindsight_memories>` wrapper or
+  OpenClaw's internal-context delimiters are now escaped before they reach the model.
+
+### Security
+
+- Recall wrapper: `<hindsight_memories>` and `</hindsight_memories>` tags inside recalled text
+  are neutralized in more forms than the exact literal: any case, whitespace inside the tag,
+  attributes, self-closing, a missing `>`, `-` for `_`, invisible characters, accents, common
+  Cyrillic/Greek lookalike letters, fullwidth letters and `<` lookalikes (everything Unicode
+  confusables.txt maps to `<`, plus angle brackets).
+- OpenClaw prompt delimiters in recalled text are escaped: `<<<BEGIN_OPENCLAW_INTERNAL_CONTEXT>>>`
+  and `<<<END_OPENCLAW_INTERNAL_CONTEXT>>>` (variants in case, spacing or invisible characters
+  too), `OpenClaw runtime context:`, `End OpenClaw runtime context.`,
+  `OpenClaw runtime context (internal):`, the runtime-context notice line and the legacy
+  carrier prefaces that precede it. Prose that mentions OpenClaw runtime context is unchanged.
+
+### Added
+
+- `scripts/compat-matrix.sh`: host keyword `beta` (npm dist-tag `beta`), comma-separated
+  `--hosts` and `--runtimes`, and matrix rows in the requested host order.
+- `knowledgeReadTools` (default off): opt-in read-only knowledge-base tools
+  `hindsight_get_knowledge_base_tree`, `hindsight_search_knowledge_base` and
+  `hindsight_get_knowledge_page`, with the vendor plugin's names. They read the same
+  audience-routed bank as `hindsight_recall`; a missing bank reads as empty. Off, the
+  registered tool set is unchanged; the manifest's static `contracts.tools` still lists the
+  three names. Output is capped (page 32000 characters, tree 500 entries, search 50 hits)
+  and a knowledge-base response body over 1 MiB fails.
+
+### Changed
+
+- CI enforces a vitest coverage floor (`npm run test:coverage`, `@vitest/coverage-v8`) at
+  lines 94%, branches 86%, functions 95%, statements 93% (develop's measured coverage minus a
+  margin). GitLab shows the percentage on MRs and keeps the report as an artifact. Both pack
+  checks also reject `coverage/` (the `files` allowlist already kept it out of the pack).
+- `scripts/launchd/hs-compat-nightly.plist.example`: `LimitLoadToSessionType` Aqua and
+  Background, so it also loads in the `user/<uid>` domain; the docs cover installing it for an
+  account without a GUI session.
+
+### Fixes
+
+- `scripts/live-proof.sh`: X2 now searches every Claude Code project dir that appeared during
+  the run, and every file under the projects dir modified during it, for the code word, not
+  only names with `hs-live-proof`. Dirs it did not predict are counted and never deleted;
+  symlinks there are not followed.
+- `scripts/compat-matrix.sh`: an empty `--hosts` or `--runtimes` list is refused (exit 2).
+- `scripts/compat-nightly.sh`: one run at a time per `WORK_ROOT`; a lock is stale when its pid
+  does not run the same script path or it is older than 12 h.
+- Config: `hindsightApiUrl` with a bare trailing `?` or `#` is rejected like a non-empty query
+  string or fragment.
+- Retain queue: a retain is no longer lost when the file cannot be rewritten but can still be
+  appended to. The new copy is appended instead (load keeps the newest copy per document), and the
+  32 MiB bound still applies to appends, so the file cannot grow without bound. A failed replay
+  save is retried by the next one.
+- Retain queue: after a crash between replay's batched writes, load keeps only the newest copy
+  per bank and document id, and replay always sends `update_mode: replace` with the item's
+  document id.
+- Auto-retain no longer loses a turn when the host continues a run after a transient model
+  retry. The second `agent_end` for the same run id now re-sends the whole run from its first
+  start instead of only the continuation, so the run's document is replaced with complete
+  content. The run start is kept per session and run id; a new run id, `session_end` or a stop
+  drops it. Host maintenance turns no longer touch it.
+
+### Downgrade notes
+
+- The queue file format is unchanged. 0.2.0 keeps duplicate copies and replays them oldest first
+  (same document id, replace), so a re-retain of that document before replay finishes can leave
+  older text. A build that ignores tombstone lines (unreleased queue builds before 0.2.0) may
+  replay an out-of-date copy over a newer one that was sent live. Let the queue drain, or delete
+  `retain-queue.jsonl`, before downgrading.
+
 ## 0.2.0 (2026-11-07)
 
 ### Upgrade notes

package/README.md

@@ -11,7 +11,8 @@ contracts instead of only hooking prompts:
   run, replaced on re-delivery). Host maintenance turns (the pre-compaction flush and Active
   Memory's recall child) get neither; they use the tools. A bank nothing was retained to yet
   reads as "no memories", without a warning.
-- **Agent tools** `hindsight_retain`, `hindsight_recall`, `hindsight_reflect`.
+- **Agent tools** `hindsight_retain`, `hindsight_recall`, `hindsight_reflect`, plus opt-in
+  read-only knowledge-base tools (see [Knowledge-base read tools](#knowledge-base-read-tools)).
 - **Recall tool declaration** (`recallToolNames`): Active Memory uses `hindsight_recall` and
   `hindsight_reflect` without a `toolsAllow` list.
 - **Pre-compaction flush, tools arm**: before compaction the host runs a silent turn that saves
@@ -58,7 +59,7 @@ tarball and install that:
 git clone https://github.com/etzelm/openclaw-memory-hindsight.git
 cd openclaw-memory-hindsight
 npm ci && npm pack
-openclaw plugins install npm-pack:./etzelm-openclaw-memory-hindsight-0.2.0.tgz --force --accept-capabilities
+openclaw plugins install npm-pack:./etzelm-openclaw-memory-hindsight-0.3.0.tgz --force --accept-capabilities
 ```
 
 `--force` confirms the "outside ClawHub review" warning for a local archive. Every install asks for
@@ -123,15 +124,21 @@ tool conflict.
 | `retainQueueEnabled`        | `true`               | Queue auto-retains that fail because Hindsight is unreachable and send them later (see below)         |
 | `retainQueueMaxItems`       | `500`                | Most queued retains kept (1 to 10000); the oldest are dropped first                                   |
 | `retainQueueMaxAgeMs`       | `604800000` (7 days) | Queued retains older than this are dropped                                                            |
+| `knowledgeReadTools`        | `false`              | Offer the read-only knowledge-base tools (see below); turning it on needs a gateway restart           |
 
 Fixed values (not configurable): recall max tokens 1024, query cap 800 characters, recall
-timeout 10 s, retain and reflect timeout 30 s, memory get timeout 10 s (each combined with the
-host's abort signal when there is one), retained roles user and assistant, JSON transcript format, recall injected before
-the user message (outside the system prompt, so the cached prefix stays stable).
+timeout 10 s, retain and reflect timeout 30 s, memory get and knowledge-base tool timeout 10 s
+(each combined with the host's abort signal when there is one), retained roles user and
+assistant, JSON transcript format, recall injected before the user message (outside the system
+prompt, so the cached prefix stays stable). Knowledge-base tool caps: a response body over
+1 MiB fails; a page shows its first 32000 characters, then `(truncated, N more characters)`;
+the tree shows 500 entries, then `(truncated, N more entries)`; search shows at most 50 hits
+with snippets cut at 500 characters and names and ids at 200.
 
 Config changes apply without a restart: the plugin re-reads its entry from the host's current
 config snapshot (`api.runtime.config.current()`), falling back to the register-time
-`api.pluginConfig`.
+`api.pluginConfig`. One exception: `knowledgeReadTools` registers its tools at startup, so turning
+it on takes a gateway restart (turning it off applies at once).
 
 ### Offline retain queue
 
@@ -174,17 +181,95 @@ session transcripts.
   for its bank's queue, and its failure is logged as before.
 - **Disk writes.** A new item is appended, on a new line even after a torn last line. Replacing,
   pruning and replay rewrite the file (atomic rename); replay writes once per 25 delivered items and
-  at the end of each pass. If a rewrite that should drop an out-of-date copy fails, a tombstone line
-  is appended so a restart does not replay that copy. One queue per file per process, shared across
-  service restarts. If a bank already has queued items and the queue file cannot be written (disk
-  full, permissions), the retain is sent live instead. If the live write already failed, the queue
-  error is logged and the retain is not kept.
+  at the end of each pass, and a failed write is retried by the next one. If a rewrite fails, the
+  new or newer copy is appended instead, with a tombstone for any item the limits drop; the older
+  copy stays in the file until the next successful rewrite, and load keeps only the newest line per
+  bank and document id. An append that would take
+  the file past 32 MiB rewrites it instead. Only when that rewrite fails too is the item not queued:
+  an out-of-date copy it replaced gets a tombstone line so a restart does not replay it, and the
+  retain is sent live, or, when its live write already failed, logged and not kept. Until the next
+  successful rewrite, the file stays within 32 MiB plus at most one short tombstone line per item
+  removed from the queue meanwhile. A crash between replay's writes can also leave an already delivered copy before
+  a newer one; load keeps the newer one, and replay always sends `update_mode: replace` with the
+  item's document id, so a repeat is harmless. One queue per file per process, shared across
+  service restarts.
+- **Downgrade.** The file format is unchanged. 0.2.0 reads tombstones but keeps every copy of a
+  document (left by a crash during replay or by a replace that could not rewrite the file) and
+  sends them oldest first, same document id with replace, so the newest text wins unless the same
+  document is retained again before that replay finishes. 0.1.x has no queue and ignores the file.
+  A build that ignores tombstone lines (only unreleased queue builds before 0.2.0) can replay an
+  out-of-date copy a tombstone hid; if the newer copy was then sent live and never queued, the stale
+  text overwrites it on the server. Let the queue drain (the file is deleted when it is empty), or
+  delete the file, before downgrading.
 - Only auto-retain is queued. `hindsight_retain` and the pre-compaction flush return the error to
   the agent, as before. With `retainQueueEnabled: false` the plugin neither reads nor writes the
   file; delete it, or turn the queue back on to drain it. `retainQueueEnabled` is read at service
   start (restart the gateway after changing it); the two limits apply live. One gateway process per
   state dir: the file is not locked.
 
+### Knowledge-base read tools
+
+With `knowledgeReadTools: true` the agent also gets three read-only tools over Hindsight's
+knowledge pages (documents Hindsight writes and refreshes from memories), under the same names as
+the vendor plugin:
+
+- `hindsight_get_knowledge_base_tree`: folders and pages with page ids and a stale mark
+  (`GET .../knowledge-base/tree`).
+- `hindsight_search_knowledge_base`: pages matching a query, with a snippet (`query`, optional
+  `limit` clamped to 1-50; `GET .../knowledge-base/search`).
+- `hindsight_get_knowledge_page`: one page as markdown by `page_id`
+  (`GET .../knowledge-base/pages/<id>`).
+
+They follow the recall tools' rules: same bank from the host-minted memory audience (an
+owner-private session reads the owner bank, a conversation reads only its own
+`<bank>::conv::<hash>` bank), same session filters, none in `context` mode or without an
+audience. Hindsight scopes page ids to the bank in the URL, so a page id from the owner bank is
+"not found" through a conversation bank. A bank nothing was written to reads as "no knowledge
+pages yet"; as with recall, a 404 with an empty body counts as a missing bank too. A route 404
+(`{"detail":"Not Found"}`: wrong URL or a Hindsight version without these endpoints) stays an
+error. `page_id` must look like a Hindsight id (letters, digits, `_ . : -`, starting with a
+letter or digit, at most 200 characters) and is refused otherwise, before any request. Output
+is marked as untrusted data, like recall, and capped (see Fixed values). The manifest's
+`contracts.tools` always lists the three names (it is static), so they appear in plugin status
+and doctor tool-owner output even with the key off; they are registered only when it is on.
+Host tool listings can therefore show them while the key is off: the manifest's `configSignals`
+only test that a key is present, not that it is `true`, so they cannot express this key. The
+write tools (create folder, create page, update, delete) are not ported.
+
+### Gateway metrics
+
+The plugin serves Prometheus text at `GET /plugins/hindsight-native/metrics` on the Gateway
+port. The route uses Gateway auth and needs `operator.read` (or `operator.write` /
+`operator.admin`), the same as `diagnostics-prometheus`, so scrape it with the token you use
+for `/api/diagnostics/prometheus`. `diagnostics-prometheus` renders only core diagnostic
+events and has no plugin metric surface, which is why this is a separate path. There is no
+config key; counters live in memory and reset when the Gateway restarts.
+
+| Series (`openclaw_hindsight_` prefix) | Type      | Labels                     |
+| ------------------------------------- | --------- | -------------------------- |
+| `recall_total`                        | counter   | `agent`, `bank`, `outcome` |
+| `recall_duration_seconds`             | histogram | `agent`, `bank`            |
+| `retain_total`                        | counter   | `agent`, `bank`, `outcome` |
+| `retain_duration_seconds`             | histogram | `agent`, `bank`            |
+| `retain_queue_depth`                  | gauge     | `bank`                     |
+| `retain_queue_replays_total`          | counter   | `bank`, `outcome`          |
+| `retain_queue_drops_total`            | counter   | `bank`, `reason`           |
+
+- Recall counts the auto-recall request only: `injected`, `empty` (no results or no bank yet),
+  `timeout`, `error`. A recall the host cancels (audience changed, hook invocation stale) is
+  not counted. Retain counts auto-retain only: `sent`, `queued`, `timeout`, `error`. Tool
+  calls are not counted.
+- The latency histograms cover every outcome, so timeouts and fast `queued` retains are in the
+  same quantiles as successful calls.
+- Replays: `delivered`, or `deferred` (Hindsight unreachable, item kept). Drops count retains
+  that are lost: `limit` (`retainQueueMaxItems`, `retainQueueMaxAgeMs` or the size bound),
+  `too_large` (the live write failed and the item is over the queue's size bound), `rejected`
+  (a 4xx on replay).
+- `agent` is the normalized agent id. `bank` is the owner bank; every conversation bank of
+  that owner shares one `<bank>::conv` label. Values that are not label-safe read `unknown`.
+  Past 200 label sets per metric family, new `agent` and `bank` values fold into `other`;
+  `outcome` and `reason` keep their value. No session ids, document ids or text.
+
 ### Mapping from `@vectorize-io/hindsight-openclaw`
 
 | Vendor key                                                                                                                                                                                                                                                                                                        | Here                                                                                                      |
@@ -251,8 +336,8 @@ and 2026-11-07):
 - 2026.10.10 is the first release that records a dashboard session created by
   `sessions.create` as a direct chat (openclaw#181359). On 2026.10.1 through 2026.10.9 the host
   gives turns in such a session no memory audience ("root session ... has no durable chat
-  type"), so the plugin neither recalls nor retains there. Nothing leaks; memory just does not
-  work for those sessions, including `openclaw agent --session-key` runs in them.
+  type"), so the plugin neither recalls nor retains there: no memory is read or written in those
+  sessions, including `openclaw agent --session-key` runs in them.
 - 2026.9.36 has no tools-arm flush plan (`persistenceToolNames`) and no
   `assertMemoryAudienceCurrent`. 2026.10.1 is the first release with `recallToolNames`, the
   tools-arm `providerFlushPlanResolver` result and `memoryAudience` /
@@ -308,15 +393,16 @@ deprecated on npm once the fix ships, as 0.1.1 is.
 - **`memory.search` through the provider runtime skips the plugin filters.** It is gated by the
   caller's audience (owner-private or conversation), not by `memoryMode`, `excludeProviders` or
   incognito session keys. Those filters apply to auto-recall, auto-retain and the tools only.
-- `hindsightApiUrl` must not contain credentials, a query string or a fragment (the parser
-  rejects it). An unresolved SecretRef token, or a token sent over plain `http` to a
-  non-loopback host, logs one warning at service start.
+- `hindsightApiUrl` must not contain credentials, a query string or a fragment, even an empty
+  one such as a trailing `?` or `#` (the parser rejects it). An unresolved SecretRef token, or a
+  token sent over plain `http` to a non-loopback host, logs one warning at service start.
 - Flush dedupe is per process; a restart mid-cycle can repeat a flush write, which the stable
   document id plus `update_mode: replace` turns into an overwrite.
 - Conversation banks are never pruned by the plugin, and `openclaw memory forget` does not reach
   slot owners (no host seam yet).
 
-Not ported from the vendor plugin: knowledge-base tools, tool-call retention
+Not ported from the vendor plugin: knowledge-base write tools (create, update, delete; the
+read tools are opt-in, see above), tool-call retention
 (`retainToolCalls`), `retainEveryNTurns` windows, the local Hindsight daemon, and
 per-user/per-channel bank granularity (replaced by audience routing).
 

package/dist/client.js

@@ -3,15 +3,19 @@ export class HindsightHttpError extends Error {
     status;
     bankMissing;
     bankMissingExplicit;
+    pageMissing;
     constructor(message, status, 
     /** True only for Hindsight's "Bank '<id>' not found" 404, not for a route that does not exist. */
     bankMissing = false, 
     /** True only when the body said so; an empty 404 body sets `bankMissing` but not this. */
-    bankMissingExplicit = false) {
+    bankMissingExplicit = false, 
+    /** True only for Hindsight's "Knowledge page '<id>' not found" 404. */
+    pageMissing = false) {
         super(message);
         this.status = status;
         this.bankMissing = bankMissing;
         this.bankMissingExplicit = bankMissingExplicit;
+        this.pageMissing = pageMissing;
         this.name = "HindsightHttpError";
     }
 }
@@ -37,6 +41,49 @@ export function isExplicitBankMissingBody(text) {
         return false;
     }
 }
+/** Only Hindsight's `Knowledge page '<id>' not found` detail; a route "Not Found" does not count. */
+export function isExplicitPageMissingBody(text) {
+    try {
+        const detail = JSON.parse(text)?.detail;
+        return typeof detail === "string" && /\bknowledge page\b.*\bnot found\b/i.test(detail);
+    }
+    catch {
+        return false;
+    }
+}
+/** Largest knowledge-base response body read; a bigger one fails instead of filling memory. */
+export const KB_MAX_RESPONSE_BYTES = 1024 * 1024;
+// Error bodies are only classified, never copied, so a small prefix is enough.
+const ERROR_BODY_MAX_BYTES = 64 * 1024;
+class BodyTooLargeError extends Error {
+}
+/** Reads at most `maxBytes` of the body; throws BodyTooLargeError past it and stops reading. */
+async function readBoundedText(response, maxBytes) {
+    const declared = Number(response.headers.get("content-length"));
+    if (Number.isFinite(declared) && declared > maxBytes) {
+        await response.body?.cancel().catch(() => undefined);
+        throw new BodyTooLargeError();
+    }
+    if (!response.body) {
+        return "";
+    }
+    const reader = response.body.getReader();
+    const chunks = [];
+    let total = 0;
+    for (;;) {
+        const { done, value } = await reader.read();
+        if (done) {
+            break;
+        }
+        total += value.byteLength;
+        if (total > maxBytes) {
+            await reader.cancel().catch(() => undefined);
+            throw new BodyTooLargeError();
+        }
+        chunks.push(value);
+    }
+    return new TextDecoder().decode(Buffer.concat(chunks));
+}
 const HEALTH_TIMEOUT_MS = 5_000;
 /** A fixed timeout, combined with the host signal when there is one (host abort still wins). */
 export function boundedSignal(timeoutMs, hostSignal) {
@@ -73,13 +120,31 @@ export class HindsightClient {
             ...(init.signal ? { signal: init.signal } : {}),
         });
         if (!response.ok) {
-            const body = response.status === 404 ? await response.text().catch(() => "") : "";
+            const body = response.status === 404
+                ? // An oversized body is not "empty", so it never reads as a missing bank.
+                    await readBoundedText(response, ERROR_BODY_MAX_BYTES).catch((error) => error instanceof BodyTooLargeError ? "<oversized>" : "")
+                : "";
             const bankMissing = response.status === 404 && isBankMissingBody(body);
             const bankMissingExplicit = response.status === 404 && isExplicitBankMissingBody(body);
+            const pageMissing = response.status === 404 && isExplicitPageMissingBody(body);
             // Server error bodies can echo request content; keep only the status in the message.
-            throw new HindsightHttpError(`Hindsight ${operation} failed: HTTP ${response.status}`, response.status, bankMissing, bankMissingExplicit);
+            throw new HindsightHttpError(`Hindsight ${operation} failed: HTTP ${response.status}`, response.status, bankMissing, bankMissingExplicit, pageMissing);
+        }
+        let text;
+        if (init.maxBodyBytes === undefined) {
+            text = await response.text();
+        }
+        else {
+            try {
+                text = await readBoundedText(response, init.maxBodyBytes);
+            }
+            catch (error) {
+                if (error instanceof BodyTooLargeError) {
+                    throw new Error(`Hindsight ${operation} failed: response larger than ${init.maxBodyBytes} bytes`, { cause: error });
+                }
+                throw error;
+            }
         }
-        const text = await response.text();
         // SAFETY: T only names optional fields; every caller type-checks each field before use.
         return (text ? JSON.parse(text) : {});
     }
@@ -125,6 +190,41 @@ export class HindsightClient {
             throw error;
         }
     }
+    async getKnowledgeBaseTree(bankId, signal) {
+        const response = await this.request("knowledge tree", this.bankUrl(bankId, "/knowledge-base/tree"), { method: "GET", signal, maxBodyBytes: KB_MAX_RESPONSE_BYTES });
+        return Array.isArray(response.roots) ? response.roots : [];
+    }
+    async searchKnowledgeBase(bankId, query, options = {}) {
+        const params = new URLSearchParams({ q: query });
+        if (options.limit !== undefined) {
+            params.set("limit", String(options.limit));
+        }
+        const response = await this.request("knowledge search", this.bankUrl(bankId, `/knowledge-base/search?${params.toString()}`), { method: "GET", signal: options.signal, maxBodyBytes: KB_MAX_RESPONSE_BYTES });
+        return Array.isArray(response.results) ? response.results : [];
+    }
+    /**
+     * Returns null when the bank or page does not exist. Hindsight scopes page ids to the bank in
+     * the URL, so a page id from another bank is also null. A route 404 stays an error.
+     */
+    async getKnowledgePage(bankId, pageId, signal) {
+        try {
+            const body = await this.request("knowledge page", this.bankUrl(bankId, `/knowledge-base/pages/${encodeURIComponent(pageId)}`), { method: "GET", signal, maxBodyBytes: KB_MAX_RESPONSE_BYTES });
+            if (typeof body.markdown !== "string") {
+                throw new Error("Hindsight knowledge page failed: unexpected response");
+            }
+            return {
+                id: pageId,
+                name: typeof body.name === "string" ? body.name : "",
+                markdown: body.markdown,
+            };
+        }
+        catch (error) {
+            if (error instanceof HindsightHttpError && (error.bankMissing || error.pageMissing)) {
+                return null;
+            }
+            throw error;
+        }
+    }
     /** Creates the bank or updates its missions; idempotent on the server. */
     async applyBankDefaults(bankId, defaults, signal) {
         const body = {};

package/dist/config.js

@@ -35,6 +35,7 @@ const CONFIG_KEYS = [
     "retainQueueEnabled",
     "retainQueueMaxItems",
     "retainQueueMaxAgeMs",
+    "knowledgeReadTools",
 ];
 function isRecord(value) {
     return typeof value === "object" && value !== null && !Array.isArray(value);
@@ -126,7 +127,9 @@ function normalizeApiUrl(raw) {
     if (url.username || url.password) {
         throw new Error("hindsightApiUrl must not contain credentials (user:password@); use hindsightApiToken");
     }
-    if (url.search || url.hash) {
+    // An empty query or fragment (a bare trailing `?` or `#`) leaves search/hash empty but stays
+    // in href; `?` and `#` cannot appear in an http(s) href except as those delimiters.
+    if (/[?#]/.test(url.href)) {
         throw new Error("hindsightApiUrl must not contain a query string or fragment");
     }
     return raw.replace(/\/+$/, "");
@@ -203,6 +206,7 @@ export const hindsightConfigSchema = {
             retainQueueEnabled: readBool(value, "retainQueueEnabled", true),
             retainQueueMaxItems: readPositiveInt(value, "retainQueueMaxItems", DEFAULT_RETAIN_QUEUE_MAX_ITEMS, MAX_RETAIN_QUEUE_MAX_ITEMS),
             retainQueueMaxAgeMs: readPositiveInt(value, "retainQueueMaxAgeMs", DEFAULT_RETAIN_QUEUE_MAX_AGE_MS),
+            knowledgeReadTools: readBool(value, "knowledgeReadTools", false),
         };
     },
 };

package/dist/index.js

@@ -6,11 +6,12 @@ import { asOptionalRecord } from "openclaw/plugin-sdk/string-coerce-runtime";
 import { isBankNotFound } from "./client.js";
 import { hasUnresolvedToken, hindsightConfigSchema, isPlainHttpRemote, RECALL_MAX_TOKENS, RECALL_TIMEOUT_MS, RETAIN_SOURCE, RETAIN_TIMEOUT_MS, } from "./config.js";
 import { createLearnContribution, createLegacyLearnFields, createProviderFlushPlanResolver, RECALL_TOOL_NAMES, } from "./memory-capability.js";
+import { createMetricsRouteHandler, HindsightMetrics, isTimeoutError, METRICS_ROUTE_PATH, } from "./metrics.js";
 import { createHindsightProviderRuntime } from "./provider-runtime.js";
 import { isMaintenanceTurn, routeAudience, sessionUsesMemory } from "./routing.js";
 import { resolveLearnRegistrar } from "./sdk-types.js";
 import { createHindsightService } from "./service.js";
-import { createFlushDeduper, createRecallTool, createReflectTool, createRetainTool, RECALL_TOOL, REFLECT_TOOL, RETAIN_TOOL, } from "./tools.js";
+import { createFlushDeduper, createKnowledgePageTool, createKnowledgeSearchTool, createKnowledgeTreeTool, createRecallTool, createReflectTool, createRetainTool, KB_PAGE_TOOL, KB_SEARCH_TOOL, KB_TREE_TOOL, RECALL_TOOL, REFLECT_TOOL, RETAIN_TOOL, } from "./tools.js";
 import { buildRecallContext, extractRecallQuery, isEphemeralOperationalText, prepareRetentionTranscript, retainStartIndex, runStartMarker, } from "./transcript.js";
 export const PLUGIN_ID = "hindsight-native";
 // Bounds per-session run markers; one entry per active session is enough.
@@ -80,11 +81,28 @@ export default definePluginEntry({
                 return startupConfig;
             }
         };
+        const metrics = new HindsightMetrics();
         const service = createHindsightService({
             config: currentConfig,
             logger: api.logger,
+            metrics,
+        });
+        // Gateway-side metrics on the gateway port, behind gateway auth (see metrics.ts for why this
+        // is a plugin route and not diagnostics-prometheus). The floor host has the method; the check
+        // only keeps a host or test double without it from failing registration.
+        if (typeof api.registerHttpRoute !== "function") {
+            api.logger.debug?.("memory-hindsight: host has no registerHttpRoute; metrics route not served");
+        }
+        api.registerHttpRoute?.({
+            path: METRICS_ROUTE_PATH,
+            auth: "gateway",
+            match: "exact",
+            gatewayRuntimeScopeSurface: "trusted-operator",
+            handler: createMetricsRouteHandler(metrics, () => service.retainQueueDepth()),
         });
         const flushDeduper = createFlushDeduper();
+        // Per session: where the current run started, kept for the whole run (by runId) so a second
+        // agent_end for the run (host continuation after a retry, or a re-delivery) re-sends all of it.
         const runStartBySession = new Map();
         const capability = {
             providerRuntime: createHindsightProviderRuntime({
@@ -112,12 +130,32 @@ export default definePluginEntry({
         registerTool(RETAIN_TOOL, (ctx) => createRetainTool(service, flushDeduper, ctx));
         registerTool(RECALL_TOOL, (ctx) => createRecallTool(service, ctx));
         registerTool(REFLECT_TOOL, (ctx) => createReflectTool(service, ctx));
+        // Opt-in: registered only when knowledgeReadTools is on at startup, so the default tool set is
+        // unchanged. Each factory also reads the live key and returns null once it is turned off.
+        if (startupConfig.knowledgeReadTools) {
+            registerTool(KB_TREE_TOOL, (ctx) => createKnowledgeTreeTool(service, ctx));
+            registerTool(KB_SEARCH_TOOL, (ctx) => createKnowledgeSearchTool(service, ctx));
+            registerTool(KB_PAGE_TOOL, (ctx) => createKnowledgePageTool(service, ctx));
+        }
         api.on("before_prompt_build", async (event, ctx) => {
             const config = currentConfig();
             const sessionKey = ctx.sessionKey;
-            // Marks the run start for agent_end even when recall is skipped.
-            if (sessionKey && !runStartBySession.has(sessionKey) && Array.isArray(event.messages)) {
-                runStartBySession.set(sessionKey, runStartMarker(event.messages, event.currentUserMessage));
+            const maintenance = isMaintenanceTurn({
+                trigger: ctx.trigger,
+                sessionKey,
+            });
+            // Marks the run start for agent_end even when recall is skipped. A rebuild or continuation
+            // of the same run keeps the first start; maintenance turns never touch the markers.
+            const prior = sessionKey ? runStartBySession.get(sessionKey) : undefined;
+            if (sessionKey &&
+                !maintenance &&
+                Array.isArray(event.messages) &&
+                (!prior || prior.runId !== ctx.runId)) {
+                runStartBySession.delete(sessionKey);
+                runStartBySession.set(sessionKey, {
+                    runId: ctx.runId,
+                    marker: runStartMarker(event.messages, event.currentUserMessage),
+                });
                 if (runStartBySession.size > MAX_RUN_MARKERS) {
                     runStartBySession.delete(runStartBySession.keys().next().value);
                 }
@@ -125,7 +163,7 @@ export default definePluginEntry({
             if (service.stopped() ||
                 !config.autoRecall ||
                 config.memoryMode === "tools" ||
-                isMaintenanceTurn({ trigger: ctx.trigger, sessionKey }) ||
+                maintenance ||
                 isIncognitoSessionKey(sessionKey) ||
                 !sessionUsesMemory(config, {
                     sessionKey,
@@ -144,17 +182,29 @@ export default definePluginEntry({
             }
             try {
                 ctx.assertMemoryAudienceCurrent?.();
-                const results = await service.client.recall(route.bankId, query, {
-                    budget: config.recallBudget,
-                    maxTokens: RECALL_MAX_TOKENS,
-                    ...(config.recallTypes ? { types: config.recallTypes } : {}),
-                    ...(config.preferObservations === undefined
-                        ? {}
-                        : { preferObservations: config.preferObservations }),
-                    signal: AbortSignal.timeout(RECALL_TIMEOUT_MS),
-                });
+                // Metrics count the Hindsight request only. A host assert that throws (audience changed,
+                // hook invocation stale) is host lifecycle, not a recall outcome, so it records nothing.
+                const recallStarted = performance.now();
+                let results;
+                try {
+                    results = await service.client.recall(route.bankId, query, {
+                        budget: config.recallBudget,
+                        maxTokens: RECALL_MAX_TOKENS,
+                        ...(config.recallTypes ? { types: config.recallTypes } : {}),
+                        ...(config.preferObservations === undefined
+                            ? {}
+                            : { preferObservations: config.preferObservations }),
+                        signal: AbortSignal.timeout(RECALL_TIMEOUT_MS),
+                    });
+                }
+                catch (error) {
+                    metrics.recordRecall(ctx.agentId, route.bankId, isBankNotFound(error) ? "empty" : isTimeoutError(error) ? "timeout" : "error", performance.now() - recallStarted);
+                    throw error;
+                }
+                const recallMs = performance.now() - recallStarted;
                 ctx.assertMemoryAudienceCurrent?.();
                 ctx.hookInvocation?.assertActive();
+                metrics.recordRecall(ctx.agentId, route.bankId, results.length === 0 ? "empty" : "injected", recallMs);
                 if (results.length === 0) {
                     return undefined;
                 }
@@ -176,9 +226,16 @@ export default definePluginEntry({
         });
         api.on("agent_end", async (event, ctx) => {
             const sessionKey = ctx.sessionKey;
-            // Taken before any skip so the next run marks its own start.
-            const runStart = sessionKey ? runStartBySession.get(sessionKey) : undefined;
-            if (sessionKey) {
+            const runId = event.runId ?? ctx.runId;
+            const maintenance = isMaintenanceTurn({
+                trigger: ctx.trigger,
+                sessionKey,
+            });
+            const start = sessionKey && !maintenance ? runStartBySession.get(sessionKey) : undefined;
+            const runStart = start?.marker;
+            // Taken before any skip. The marker stays while the run can still continue; without a
+            // runId it is dropped so the next run marks its own start.
+            if (sessionKey && start && (runId === undefined || start.runId !== runId)) {
                 runStartBySession.delete(sessionKey);
             }
             const config = currentConfig();
@@ -187,7 +244,7 @@ export default definePluginEntry({
                 !event.success ||
                 !Array.isArray(event.messages) ||
                 event.messages.length === 0 ||
-                isMaintenanceTurn({ trigger: ctx.trigger, sessionKey }) ||
+                maintenance ||
                 isIncognitoSessionKey(sessionKey) ||
                 !sessionUsesMemory(config, {
                     sessionKey,
@@ -205,9 +262,11 @@ export default definePluginEntry({
                 return;
             }
             const tags = [...new Set([...config.retainTags, ...retention.tags])];
-            // One document per run: a re-delivered agent_end for the same run replaces, not duplicates.
-            const runPart = sanitizeDocumentIdPart(event.runId ?? ctx.runId, `${Date.now()}`);
+            // One document per run. Every agent_end for the run carries the run from its first start,
+            // so a continuation or a re-delivery replaces the document with the whole run.
+            const runPart = sanitizeDocumentIdPart(runId, `${Date.now()}`);
             const documentId = `openclaw:${sanitizeDocumentIdPart(sessionKey, "session")}-${sessionKeyDigest(sessionKey)}:run:${runPart}`;
+            const retainStarted = performance.now();
             try {
                 // No ctx.assertMemoryAudienceCurrent() here. The host starts agent_end handlers without
                 // awaiting them (runAgentEndSideEffectsAsync -> void runAgentEnd, after awaiting the
@@ -234,9 +293,11 @@ export default definePluginEntry({
                         ...(ctx.channelId ? { channel_id: ctx.channelId } : {}),
                     },
                 }, signal);
+                metrics.recordRetain(ctx.agentId, route.bankId, outcome, performance.now() - retainStarted);
                 api.logger.debug?.(`memory-hindsight: ${outcome === "sent" ? "retained" : "queued"} ${retention.messageCount} messages (${route.audience.kind})`);
             }
             catch (error) {
+                metrics.recordRetain(ctx.agentId, route.bankId, isTimeoutError(error) ? "timeout" : "error", performance.now() - retainStarted);
                 api.logger.warn?.(`memory-hindsight: auto-retain failed: ${formatErrorMessage(error)}`);
             }
         });