@lacneu/openclaw-knowledge 3.1.2 → 3.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (44) hide show
  1. package/CHANGELOG.md +368 -1
  2. package/README.md +131 -0
  3. package/dist/config.d.ts +4 -0
  4. package/dist/config.js +26 -0
  5. package/dist/config.js.map +1 -1
  6. package/dist/index.d.ts +61 -4
  7. package/dist/index.js +463 -50
  8. package/dist/index.js.map +1 -1
  9. package/dist/jina/classifier.d.ts +55 -0
  10. package/dist/jina/classifier.js +170 -0
  11. package/dist/jina/classifier.js.map +1 -0
  12. package/dist/jina/client.d.ts +30 -0
  13. package/dist/jina/client.js +131 -0
  14. package/dist/jina/client.js.map +1 -0
  15. package/dist/jina/errors.d.ts +42 -0
  16. package/dist/jina/errors.js +113 -0
  17. package/dist/jina/errors.js.map +1 -0
  18. package/dist/jina/reranker.d.ts +34 -0
  19. package/dist/jina/reranker.js +95 -0
  20. package/dist/jina/reranker.js.map +1 -0
  21. package/dist/jina/types.d.ts +78 -0
  22. package/dist/jina/types.js +12 -0
  23. package/dist/jina/types.js.map +1 -0
  24. package/dist/pgvector.d.ts +29 -0
  25. package/dist/pgvector.js +68 -0
  26. package/dist/pgvector.js.map +1 -1
  27. package/dist/router/heuristic.d.ts +29 -0
  28. package/dist/router/heuristic.js +104 -0
  29. package/dist/router/heuristic.js.map +1 -0
  30. package/dist/router/index.d.ts +33 -0
  31. package/dist/router/index.js +94 -0
  32. package/dist/router/index.js.map +1 -0
  33. package/dist/router/labels.d.ts +33 -0
  34. package/dist/router/labels.js +67 -0
  35. package/dist/router/labels.js.map +1 -0
  36. package/dist/router/types.d.ts +23 -0
  37. package/dist/router/types.js +7 -0
  38. package/dist/router/types.js.map +1 -0
  39. package/dist/tracing/events.d.ts +83 -0
  40. package/dist/tracing/events.js +86 -0
  41. package/dist/tracing/events.js.map +1 -0
  42. package/dist/types.d.ts +61 -1
  43. package/openclaw.plugin.json +97 -4
  44. package/package.json +3 -3
package/CHANGELOG.md CHANGED
@@ -7,6 +7,371 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [3.2.1] - 2026-05-23
11
+
12
+ ### Fixed — router and reranker received the aggregated context, not the user query
13
+
14
+ In 3.2.0, the hook handler called `extractQueryFromMessages(event.messages)`
15
+ to obtain the user query that drove routing decisions and reranker calls.
16
+ Empirical observation in production (Opik trace
17
+ `019e565a-806b-...`) showed that on a real CLI turn with a 146-char user
18
+ prompt (`"Sender (untrusted metadata)...\n[Sat 2026-05-23 15:40 EDT] Quel
19
+ est la version du plugin knowledge ?"`), the `messages[last].content`
20
+ slot reached **23 893 chars** — OpenClaw 2026.5.x aggregates the
21
+ conversation window plus framing in there for LLM consumption.
22
+
23
+ Effect: the heuristic router (keyword matching, CLI-trivial regex,
24
+ meta-agent regex) ran against ~24 KB of accumulated context instead of
25
+ the 50-char user utterance. Matches fired effectively at random and the
26
+ router decisions became uncorrelated with the actual question. The
27
+ Jina classifier, when active, embedded a 24 KB blob for every turn —
28
+ needlessly expensive and noisy.
29
+
30
+ The SDK exposes the raw user prompt directly via `event.prompt`
31
+ (`PluginHookBeforePromptBuildEvent.prompt`, SDK >= 2026.5.0). The fix
32
+ reads `event.prompt` first, strips the OpenClaw envelope (sender
33
+ metadata + `[Day YYYY-MM-DD HH:MM TZ]` marker) via the new
34
+ `stripOpenClawHeaders()` helper, and falls back to the legacy
35
+ `extractQueryFromMessages` path only when `event.prompt` is absent.
36
+
37
+ Two new exported helpers (`extractUserQuery`, `stripOpenClawHeaders`)
38
+ cover the extraction logic and are unit-tested with 13 cases including
39
+ the exact 24 KB regression scenario.
40
+
41
+ ### Migration
42
+
43
+ Drop-in patch — no config change required. `update` the plugin and
44
+ restart the gateway:
45
+
46
+ ```bash
47
+ sudo docker exec openclaw-jerome openclaw plugins update @lacneu/openclaw-knowledge
48
+ sudo docker exec openclaw-olivier openclaw plugins update @lacneu/openclaw-knowledge
49
+ sudo docker restart openclaw-jerome openclaw-olivier
50
+ ```
51
+
52
+ After the restart, `[knowledge.event]` logs should show `queryLength`
53
+ in the tens-to-hundreds range on normal CLI turns (down from
54
+ ~24 000 in 3.2.0).
55
+
56
+ ### Envelope-stripping contract
57
+
58
+ The OpenClaw envelope produced by `event.prompt` follows this grammar:
59
+
60
+ ```
61
+ ( <header> "(" ("untrusted"|"metadata") <variant> "):" \n ``` <body> ``` \n+ ){0,8}
62
+ ( [ <day> YYYY-MM-DD HH:MM[:SS] <tz> ] )?
63
+ <user utterance>
64
+ ( <header> "(" ("untrusted"|"metadata") <variant> "):" <anything> EOF )?
65
+ ```
66
+
67
+ The trailing suffix's body can be a fenced code block OR raw lines
68
+ (e.g. `<<<EXTERNAL_UNTRUSTED_CONTENT` / `Source:` markers). We anchor
69
+ on the header line only and drop everything after it.
70
+
71
+ `stripOpenClawHeaders` matches this shape anchored at the start:
72
+
73
+ - ZERO to EIGHT inbound-context blocks whose header contains any
74
+ `(untrusted …)` sentinel — covers the six SDK-known sentinels
75
+ (`Sender`, `Conversation info`, `Thread starter`, `Replied message`,
76
+ `Forwarded message context`, `Chat history since last reply`) plus
77
+ headroom for future additions.
78
+ - An OPTIONAL timestamp marker that can appear EITHER before OR after
79
+ the metadata blocks. Both `block+ ts?` (legacy CLI path) and
80
+ `ts blocks+` (timestamp-first injection path) are accepted because
81
+ production traces show the SDK emits both orderings. CLI turns
82
+ include the marker; webchat / Telegram channels can embed the
83
+ timestamp inside the `Conversation info` JSON instead. When present,
84
+ the TZ suffix is permissive (`[^\]\n]+`) so it accepts named
85
+ abbreviations (`EDT`, `UTC`, …) AND `Intl.DateTimeFormat` offsets
86
+ (`GMT+2`, `GMT+5:30`, `UTC-5`).
87
+ - The leading anchor preserves user content that itself contains
88
+ timestamp-shaped substrings (e.g. a pasted log excerpt).
89
+ - The hard cap of `MAX_ENVELOPE_BLOCKS + 2` iterations bounds the
90
+ regex engine's worst-case cost on malformed input.
91
+
92
+ `extractUserQuery` is authoritative on `event.prompt`: when the SDK
93
+ supplies it, the result of `stripOpenClawHeaders` is returned as-is,
94
+ even when empty. The legacy `extractQueryFromMessages(event.messages)`
95
+ fallback is only taken when `event.prompt` is `undefined` (older SDK).
96
+ Downstream `MIN_QUERY_LENGTH` drops empty results, so the
97
+ present-but-empty case is safe.
98
+
99
+ ### Test coverage
100
+
101
+ - Total: 223 tests, all green.
102
+ - 34 tests in `test/extract-query.test.ts` covering both helpers,
103
+ including the v3.2.0 regression scenario (146-char prompt vs 24 KB
104
+ messages aggregate), the Codex pass #7 regression (empty-stripped
105
+ prompt MUST NOT fall back to the messages aggregate), the Codex
106
+ pass #8 regression (inner timestamp in user content MUST NOT trigger
107
+ stripping), the Codex pass #9 regression (multiple stacked metadata
108
+ blocks before the marker MUST all be stripped), the Codex pass #10
109
+ regression (`(untrusted, for context)` sentinel + GMT/UTC offset TZ
110
+ formats), and the Codex pass #24 regression (timestamp-first envelope
111
+ ordering: `[Sat …] Sender (untrusted metadata): … user query`).
112
+
113
+ ## [3.2.0] - 2026-05-23
114
+
115
+ ### Added — Jina-powered router (`jina.router.*`)
116
+
117
+ The plugin can now skip retrieval entirely when the user turn is clearly
118
+ not a knowledge-base question. Two operational sources of waste are
119
+ eliminated:
120
+
121
+ - **Heartbeats**, **cron**, and **memory** triggers (from
122
+ `PluginHookAgentContext.trigger`) — gated deterministically, zero
123
+ Jina cost, zero ambiguity. In observed traces these accounted for the
124
+ majority of pgvector / LightRAG / Jina rerank calls.
125
+ - **Meta-agent questions** ("what is your session id", "combien d'agents
126
+ dans cette instance") that the knowledge base can never answer.
127
+ - **CLI test pings** (`isCli && /^test|ping|hello|salut.*$/i`) when the
128
+ sender is the local CLI harness.
129
+
130
+ Two modes:
131
+
132
+ - `heuristic` (default) — zero-cost regex + trigger rules only. Safe to
133
+ enable as a first step; never crashes, never consumes Jina tokens.
134
+ - `jina-classifier` — same heuristics first, then Jina `/v1/classify`
135
+ for ambiguous queries. Supports both **zero-shot** (built-in labels)
136
+ and **few-shot** (operator-trained `classifierId`). The plugin does
137
+ NOT implement `/v1/train` — training is an out-of-band step.
138
+
139
+ The router is **fail-open** by contract: any Jina outage falls back to
140
+ `ALL` (the pre-3.2.0 behavior) and never blocks the agent.
141
+
142
+ ### Added — Jina-powered pgvector reranker (`jina.pgvectorReranker.*`)
143
+
144
+ After the cosine-similarity recall stage, results may optionally be
145
+ re-ordered by a Jina cross-encoder. This dramatically improves precision
146
+ on noisy candidate sets (the 0.36–0.41 cosine scores we observed in
147
+ production were below the practical relevance floor).
148
+
149
+ - Default model: `jina-reranker-v2-base-multilingual` (recommended for
150
+ French content; v3 is English-biased).
151
+ - Hard-coded `return_documents: false` for token economy (the plugin
152
+ already owns the source rows; only `(index, score)` is needed back).
153
+ - Hard-coded `truncate: true` so over-long chunks get clipped rather
154
+ than failing the whole batch.
155
+ - Independent cooldown counter — a Jina rerank outage does NOT trip the
156
+ router cooldown, and vice versa.
157
+ - At init, warns if `topK < pgvectorRerankerTopN × 2` (not enough recall
158
+ for the reranker to meaningfully change ordering).
159
+
160
+ ### Added — structured event tracing (`[knowledge.event]`)
161
+
162
+ Every router decision, source execution, reranker run, and cooldown
163
+ transition emits a single-line JSON event through `logger.info`, prefixed
164
+ with `[knowledge.event] `. Operators can scrape these lines into Opik,
165
+ LangFuse, or any OTLP collector without the plugin needing to depend on
166
+ a specific tracing SDK (preserves the single-`pg`-dep promise).
167
+
168
+ ### Added — modular Jina client (`src/jina/`)
169
+
170
+ A small, dependency-free HTTP client (`client.ts`) used by the classifier
171
+ and reranker. Features:
172
+
173
+ - Bearer-only auth — the API key never appears in the URL or in error
174
+ messages.
175
+ - AbortController-based 8 s timeout per request, composable with a
176
+ caller-supplied `AbortSignal`.
177
+ - Defensive JSON parsing — CDN HTML 5xx pages don't crash the plugin.
178
+ - Typed errors: `JinaAuthError` / `JinaRateLimitError` / `JinaApiError`
179
+ / `JinaNetworkError` (all extending `JinaError`).
180
+ - Error bodies truncated to 200 chars in messages, matching the existing
181
+ pattern in `embeddings.ts` / `lightrag.ts`.
182
+
183
+ ### Changed — hook handler now reads `PluginHookAgentContext`
184
+
185
+ `createBeforePromptBuildHandler` returns a handler with the canonical
186
+ SDK signature `(event, ctx?)`. `ctx.trigger` is consumed by the router
187
+ gate; other ctx fields are ignored. The change is backward-compatible —
188
+ calling the handler with no `ctx` argument keeps the pre-3.2.0 behavior.
189
+
190
+ ### Changed — three independent cooldown counters
191
+
192
+ The pre-existing 3-errors → 5-min cooldown remains shared between
193
+ pgvector and LightRAG (the "global" scope). Router and pgvector reranker
194
+ each get their own counter so a Jina outage on one path cannot stop the
195
+ other. All three are reset to zero on the first success after expiry.
196
+
197
+ ### Migration
198
+
199
+ Pre-3.2.0 configs continue to work identically — every new feature
200
+ defaults to OFF and requires explicit opt-in via the `jina.*` block.
201
+ The `jina` block as a whole is optional; omit it to keep current
202
+ behavior.
203
+
204
+ To enable the router (recommended starting point):
205
+
206
+ ```yaml
207
+ plugins:
208
+ openclaw-knowledge:
209
+ config:
210
+ jina:
211
+ apiKey: ${JINA_API_KEY}
212
+ router:
213
+ enabled: true
214
+ mode: heuristic # safe default, no Jina calls
215
+ ```
216
+
217
+ > **Operational note — `isCli` heuristic.** The CLI-trivial skip rule
218
+ > fires only when `ctx.messageProvider === "cli"`. The exact value the
219
+ > OpenClaw SDK populates depends on the channel that initiated the
220
+ > turn. Before enabling the router in production, log `ctx.messageProvider`
221
+ > for one CLI turn and verify it matches `"cli"`. If your gateway uses
222
+ > a different identifier, the safest path is to leave this branch
223
+ > dormant — meta-agent regex and trigger gating already cover the most
224
+ > wasteful traffic (heartbeats, "what is your session id?").
225
+
226
+ To enable the pgvector reranker:
227
+
228
+ ```yaml
229
+ jina:
230
+ apiKey: ${JINA_API_KEY}
231
+ pgvectorReranker:
232
+ enabled: true
233
+ # model: jina-reranker-v2-base-multilingual (default)
234
+ # topN: 5 (default)
235
+ topK: 20 # recommended ≥ rerankerTopN × 2
236
+ ```
237
+
238
+ ### Fixed during code review
239
+
240
+ Eleven pre-release issues flagged across six Codex adversarial review
241
+ passes (2026-05-23):
242
+
243
+ - **Label/Route name mismatch.** The classifier labels used the literal
244
+ `"NO_RETRIEVAL"` while the `Route` type and `isKnownRoute()` only
245
+ accepted `"NONE"`. Effect: every no-retrieval prediction silently fell
246
+ back to `ALL`, defeating the whole point of the router for the most
247
+ important class. Few-shot classifiers trained against `"NONE"` were
248
+ also unreachable. Renamed `ROUTE_NO_RETRIEVAL` → `ROUTE_NONE` with the
249
+ literal value `"NONE"`. Operators training a few-shot classifier must
250
+ use the four canonical names `NONE`, `PGVECTOR_ONLY`, `LIGHTRAG_ONLY`,
251
+ `ALL`. Pinned by a regression test.
252
+
253
+ - **Exclusive route on single-source deployment dropped retrieval.**
254
+ In a pgvector-only deployment (no LightRAG), a router decision of
255
+ `LIGHTRAG_ONLY` produced zero tasks and a silent context drop, even
256
+ though pgvector was available. Added `projectRouteOnEnabledSources()`
257
+ that falls back from `PGVECTOR_ONLY`/`LIGHTRAG_ONLY` to the available
258
+ source when the target is disabled (and to `NONE` when neither is
259
+ available). `ALL` and `NONE` remain pass-through. The router event
260
+ emitted to the log now reflects the EFFECTIVE (projected) route, not
261
+ the abstract decision. Covered by 7 unit tests + 1 e2e regression
262
+ test.
263
+
264
+ - **Jina client timeout did not cover the body read.** The internal
265
+ `clearTimeout` ran in a `finally` immediately after `fetch()` returned,
266
+ BEFORE `resp.text()`. If an upstream proxy delivered headers fast but
267
+ stalled the body stream, the response could hang indefinitely (only
268
+ the SDK's outer timeout would eventually catch it — far too long for
269
+ `before_prompt_build`). Refactored `postJson` so the single
270
+ `clearTimeout` + signal-detach happen in an outer `finally` that wraps
271
+ the entire request-AND-body cycle. Added a dedicated regression test
272
+ using a `ReadableStream` body that only completes on abort.
273
+
274
+ - **README documented the wrong no-retrieval label.** The "Adaptive
275
+ router" section listed `NO_RETRIEVAL` as one of the four routes, but
276
+ the code accepts only `NONE`. Operators following the doc to train a
277
+ few-shot classifier would have produced an unusable classifier.
278
+ Corrected to `NONE` and added an explicit reminder that few-shot
279
+ classifiers MUST be trained against the four canonical names.
280
+
281
+ - **Privacy: query preview removed from debug logs.** The original
282
+ `emitQueryPreview` logged the first 80 chars of every user query when
283
+ `logger.debug` was active. In Ataraxis-style deployments those queries
284
+ routinely carry PHI / client content / occasionally secrets, so even a
285
+ truncated preview was a leak vector. Replaced by `emitQueryFingerprint`
286
+ which logs a non-reversible SHA-256 prefix (`fp=<12 hex chars>`) and
287
+ the integer length only. Operators still get turn-level correlation
288
+ across the router event, source events, and downstream Opik traces,
289
+ without any portion of the underlying text appearing in logs.
290
+ Regression test asserts that **every word of a sensitive query is
291
+ absent** from the emitted log line.
292
+
293
+ - **Router heuristic falsely classified business questions as meta.**
294
+ The "status" trigger pattern matched any prompt ending with the word
295
+ "status", so `what is the ACME project status?` was being routed to
296
+ `NONE` (no retrieval). Anchored the relevant patterns with `^`/`$`
297
+ so only whole-prompt pings (`status?`, `system status?`, `are you
298
+ there?`) trigger the meta-skip. Business questions that happen to
299
+ mention the words remain on the retrieval path. Pinned by 4 new
300
+ regression tests.
301
+
302
+ - **Privacy: Jina error bodies could echo PHI into error logs.**
303
+ `JinaApiError.message` includes the first 200 chars of the upstream
304
+ response body. On `/v1/rerank` failures, that body can echo the user
305
+ query or a document chunk back. The previous
306
+ `logger.error(\`...— \${err.message}\`)` re-published this content in
307
+ plain text. Added `summarizeJinaError()` that returns only the error
308
+ class and HTTP status code (e.g. `JinaApiError(status=503)`), and
309
+ used it everywhere the hook handler logs an error. Body content
310
+ never reaches the log. Covered by 7 new unit tests.
311
+
312
+ - **Pgvector reranker lost the first turn after cooldown expiry.**
313
+ `maybeResetCooldown` was called AFTER the `rerankerActive` check, so
314
+ the first turn after the 5-min window expired still ran cosine-only,
315
+ even though the operator's log said `resuming`. Moved the reset
316
+ before the check. The behavior is now documented inline and the
317
+ ordering invariant is preserved by source comments — full e2e
318
+ coverage requires a pg.Pool seam that doesn't exist yet.
319
+
320
+ - **Router cooldown re-enabled retrieval for heartbeats during a Jina
321
+ outage.** `runRouterWithCooldown` used to short-circuit straight to
322
+ `ALL` once the classifier circuit opened, bypassing the zero-cost
323
+ heuristic layer entirely. Result: during a 5-minute Jina outage,
324
+ every heartbeat / cron / meta-question would resume calling
325
+ pgvector + LightRAG — the exact waste the router was designed to
326
+ block. Fixed by DOWNGRADING the router mode to `"heuristic"` during
327
+ cooldown rather than short-circuiting, so heartbeat / trigger /
328
+ meta-regex / CLI rules still apply. Pinned by an end-to-end
329
+ regression test that: trips 3 classifier errors, then sends a
330
+ heartbeat turn and asserts ZERO fetch calls (no Gemini embed, no
331
+ LightRAG, no Jina). The error counter is also no longer reset by
332
+ successful heuristic-only turns during cooldown — that would have
333
+ prematurely declared the classifier healthy.
334
+
335
+ - **Privacy: query fingerprint hash was dictionary-recoverable on
336
+ short prompts.** `emitQueryFingerprint` emitted the first 12 hex
337
+ chars of `SHA-256(query)` as a debug-only "non-reversible"
338
+ correlation key. On low-entropy prompts (the hook accepts queries as
339
+ short as 3 chars), the hash is brute-forceable offline against a
340
+ dictionary of likely prompts — so the "privacy invariant" was leaky
341
+ in exactly the deployments where it mattered most. Removed the hash
342
+ entirely. Replaced by `emitTurnMetadata(logger, ctx.runId, query.length)`
343
+ which emits the SDK's non-query-derived `runId` and a length count
344
+ only. Pinned by a regression test that asserts no query word AND no
345
+ long hex token appears in the debug line. Operators who want
346
+ cross-turn content correlation must instrument at the SDK layer with
347
+ their own keyed scheme (HMAC + deployment secret); the plugin will
348
+ not do it.
349
+
350
+ - **Telemetry: `rawCount` was post-rerank, hiding recall vs pruning.**
351
+ When the pgvector reranker was active, `PgvectorEvent.rawCount`
352
+ reflected the post-rerank truncated size (`topN`), not the number of
353
+ candidates pgvector actually returned. Operators relying on the
354
+ event to monitor recall would see the wrong value. Fixed by carrying
355
+ the pre-rerank count as a dedicated `rawCount: number` field on the
356
+ internal `PgvectorSourceResult`, captured BEFORE `rerankPgvectorResults`
357
+ runs. `rerankedCount` continues to reflect the post-truncation final
358
+ size, so operators can compute pruning = `rawCount − rerankedCount`.
359
+
360
+ ### Test coverage
361
+
362
+ - 56 pre-existing tests preserved (no behavioral regression on legacy
363
+ paths).
364
+ - 133 new tests covering: Jina client error mapping (incl. body-stall
365
+ abort regression), `summarizeJinaError` privacy contract, classifier
366
+ defensive parsing across 4 known response shapes, reranker defensive
367
+ parsing, router heuristics across triggers / meta-regex / CLI / keyword
368
+ fast-paths (incl. business-status false-positive regression), router
369
+ orchestration in all fail-open scenarios incl. heuristic preservation
370
+ during classifier cooldown, route projection onto enabled sources,
371
+ pgvector reranker integration, turn-metadata tracing via SDK runId
372
+ (regression-pinned: no query content, no hash, in logs).
373
+ - Total: 189 tests, all green.
374
+
10
375
  ### TODO — full migration to `kind: "context-engine"` (deferred)
11
376
 
12
377
  The OpenClaw doctor classifies the current `before_prompt_build`-only
@@ -208,7 +573,9 @@ For instance owners on `@lacneu/openclaw-knowledge@3.1.0` or `3.1.1`:
208
573
  - Release workflow: creates GitHub Release with tarball on tag push.
209
574
  - Architecture, lifecycle, and sequence diagrams in `schemas/`.
210
575
 
211
- [Unreleased]: https://github.com/OlivierNeu/openclaw-knowledge-plugin/compare/v3.1.0...HEAD
576
+ [Unreleased]: https://github.com/OlivierNeu/openclaw-knowledge-plugin/compare/v3.2.1...HEAD
577
+ [3.2.1]: https://github.com/OlivierNeu/openclaw-knowledge-plugin/compare/v3.2.0...v3.2.1
578
+ [3.2.0]: https://github.com/OlivierNeu/openclaw-knowledge-plugin/compare/v3.1.2...v3.2.0
212
579
  [3.1.0]: https://github.com/OlivierNeu/openclaw-knowledge-plugin/compare/v1.2.0...v3.1.0
213
580
  [1.2.0]: https://github.com/OlivierNeu/openclaw-knowledge-plugin/compare/v1.1.2...v1.2.0
214
581
  [1.1.2]: https://github.com/OlivierNeu/openclaw-knowledge-plugin/compare/v1.1.1...v1.1.2
package/README.md CHANGED
@@ -183,6 +183,14 @@ openclaw gateway restart
183
183
  | `lightragQueryMode` | string | `"hybrid"` | Query mode: `naive`, `local`, `global`, `hybrid` |
184
184
  | `lightragMaxChars` | number | `4000` | Character budget for LightRAG context |
185
185
  | `lightragEnabled` | boolean | `true` if `lightragUrl` set | Disable LightRAG while keeping pgvector |
186
+ | **Jina integration (optional, v3.2.0+)** | | | |
187
+ | `jina.apiKey` | string | — | Jina API key shared by router & reranker (supports `${ENV_VAR}`) |
188
+ | `jina.router.enabled` | boolean | `false` | Adaptive routing (skip irrelevant retrievals) |
189
+ | `jina.router.mode` | string | `"heuristic"` | `heuristic` (zero-cost) or `jina-classifier` (heuristic + Jina fallback) |
190
+ | `jina.router.classifierId` | string | — | Optional pre-trained few-shot classifier ID |
191
+ | `jina.pgvectorReranker.enabled` | boolean | `false` | Cross-encoder re-ordering of pgvector results |
192
+ | `jina.pgvectorReranker.model` | string | `"jina-reranker-v2-base-multilingual"` | Reranker model |
193
+ | `jina.pgvectorReranker.topN` | number | `5` | Max results returned after rerank |
186
194
 
187
195
  ### LightRAG query modes
188
196
 
@@ -195,6 +203,129 @@ openclaw gateway restart
195
203
 
196
204
  ---
197
205
 
206
+ ## Jina integration (v3.2.0+)
207
+
208
+ The plugin can optionally call the [Jina AI](https://jina.ai/) cloud API
209
+ to make two improvements:
210
+
211
+ ### Adaptive router — skip retrieval when it can't help
212
+
213
+ By default the plugin queries every configured source on every turn.
214
+ That's wasteful on heartbeats, cron-driven turns, and meta-questions
215
+ ("what is your session id?") that no knowledge base can answer.
216
+
217
+ Enabling the router introduces a gating step before the sources are
218
+ called:
219
+
220
+ 1. **Zero-cost heuristics first.** Skips on `PluginHookAgentContext.trigger ∈ {heartbeat, cron, memory}`, on
221
+ meta-agent regex matches, and on CLI test pings from
222
+ `messageProvider="cli"`. Keyword fast-paths route obvious factual
223
+ lookups to pgvector and obvious multi-hop questions to LightRAG.
224
+ 2. **Jina Classifier fallback** (only in `mode: "jina-classifier"`).
225
+ When the heuristics are ambiguous, the plugin calls
226
+ `POST /v1/classify` to pick one of four routes:
227
+ `NONE`, `PGVECTOR_ONLY`, `LIGHTRAG_ONLY`, or `ALL`. **Few-shot
228
+ classifiers MUST be trained against these exact canonical names** —
229
+ any other label is silently rejected and falls back to `ALL`.
230
+ Supports **zero-shot** (built-in labels, no training required) and
231
+ **few-shot** (pre-trained classifier_id, ~50 tokens per call vs ~200
232
+ for zero-shot).
233
+ 3. **Fail-open.** Any Jina outage degrades silently to `ALL` — the
234
+ pre-3.2.0 behavior. Routing never blocks the agent.
235
+
236
+ Enable in `openclaw.json`:
237
+
238
+ ```json
239
+ "jina": {
240
+ "apiKey": "${JINA_API_KEY}",
241
+ "router": {
242
+ "enabled": true,
243
+ "mode": "heuristic"
244
+ }
245
+ }
246
+ ```
247
+
248
+ Then switch to `mode: "jina-classifier"` once you're comfortable.
249
+
250
+ ### Pgvector reranker — re-order vector results by relevance
251
+
252
+ Vector cosine similarity is great recall but mediocre precision: the
253
+ top-K candidates are often noisy. A cross-encoder reranker re-scores
254
+ each (query, candidate) pair as a pair, which is much more accurate
255
+ than independent embeddings — at the cost of one Jina call per turn.
256
+
257
+ Enable in `openclaw.json`:
258
+
259
+ ```json
260
+ "jina": {
261
+ "apiKey": "${JINA_API_KEY}",
262
+ "pgvectorReranker": {
263
+ "enabled": true
264
+ }
265
+ },
266
+ "topK": 20
267
+ ```
268
+
269
+ Recommended `topK ≥ topN × 2` so the reranker has room to re-order.
270
+ The plugin warns at init if the ratio is too tight.
271
+
272
+ **Model default:** `jina-reranker-v2-base-multilingual` — best for
273
+ French content. v3 is larger (131K context) but English-biased. Switch
274
+ via `jina.pgvectorReranker.model`.
275
+
276
+ ### Observability
277
+
278
+ Every router decision, source execution, and cooldown transition emits
279
+ a structured event line:
280
+
281
+ ```
282
+ [knowledge.event] {"type":"router","route":"PGVECTOR_ONLY","reason":"heuristic_keyword","score":null,"queryLength":42,"trigger":"user"}
283
+ [knowledge.event] {"type":"pgvector","collections":["knowledge_olivier"],"rawCount":5,"rerankedCount":5,"topScore":0.78,"durationMs":124}
284
+ [knowledge.event] {"type":"cooldown","scope":"router","consecutiveErrors":3}
285
+ ```
286
+
287
+ These lines can be scraped by Opik, LangFuse, or any OTLP collector
288
+ without the plugin depending on a specific tracing SDK.
289
+
290
+ #### Privacy invariant
291
+
292
+ The plugin **never** logs any portion of the raw user query, any
293
+ retrieved chunk text, any hash of them, or any other potentially-PII
294
+ payload. When `logger.debug` is enabled, an extra correlation line
295
+ carries the SDK-provided turn identifier only:
296
+
297
+ ```
298
+ [knowledge.event] turn.metadata runId=01HF... qlen=42
299
+ ```
300
+
301
+ `runId` comes from `PluginHookAgentContext.runId` (the OpenClaw SDK's
302
+ non-query-derived turn identifier). `qlen` is just a character count.
303
+ The plugin deliberately does NOT publish any hash of the query — a
304
+ deterministic SHA-256 prefix of a 3–10 character prompt is
305
+ dictionary-recoverable offline, which would defeat the invariant on
306
+ exactly the deployments where it matters most (PHI / regulated
307
+ content).
308
+
309
+ Operators who need CONTENT correlation across turns (e.g. "this user
310
+ asked the same question twice") must instrument at the SDK layer with
311
+ a keyed HMAC and a deployment-side secret; the plugin will not do it
312
+ for them.
313
+
314
+ ### Cooldown isolation
315
+
316
+ The pre-existing 3-errors → 5-min cooldown is now split into three
317
+ independent counters:
318
+
319
+ | Scope | Triggers cooldown |
320
+ |-------|-------------------|
321
+ | `global` | Both pgvector AND LightRAG fail in the same turn |
322
+ | `router` | Repeated Jina classifier errors |
323
+ | `pgvector_reranker` | Repeated Jina rerank errors |
324
+
325
+ A Jina outage on one path no longer affects the others.
326
+
327
+ ---
328
+
198
329
  ## Data model
199
330
 
200
331
  ### pgvector: `knowledge_vectors` table
package/dist/config.d.ts CHANGED
@@ -10,5 +10,9 @@ export declare function resolveEnv<T>(value: T): T;
10
10
  * Apply defaults and env substitution to the raw plugin config. A source is
11
11
  * enabled when its credentials are present, unless the user explicitly toggles
12
12
  * `pgvectorEnabled`/`lightragEnabled` off.
13
+ *
14
+ * Jina-derived features (router + pgvector reranker) follow the same
15
+ * "default to off" discipline: nothing activates without an explicit opt-in,
16
+ * so pre-3.2.0 configs continue to work identically.
13
17
  */
14
18
  export declare function resolveConfig(cfg?: KnowledgePluginConfig): ResolvedKnowledgeConfig;
package/dist/config.js CHANGED
@@ -22,16 +22,28 @@ const DEFAULT_SCORE_THRESHOLD = 0.3;
22
22
  const DEFAULT_MAX_INJECT_CHARS = 4000;
23
23
  const DEFAULT_LIGHTRAG_MODE = "hybrid";
24
24
  const DEFAULT_LIGHTRAG_MAX_CHARS = 4000;
25
+ const DEFAULT_ROUTER_MODE = "heuristic";
26
+ const DEFAULT_RERANKER_MODEL = "jina-reranker-v2-base-multilingual";
27
+ const DEFAULT_RERANKER_TOP_N = 5;
25
28
  /**
26
29
  * Apply defaults and env substitution to the raw plugin config. A source is
27
30
  * enabled when its credentials are present, unless the user explicitly toggles
28
31
  * `pgvectorEnabled`/`lightragEnabled` off.
32
+ *
33
+ * Jina-derived features (router + pgvector reranker) follow the same
34
+ * "default to off" discipline: nothing activates without an explicit opt-in,
35
+ * so pre-3.2.0 configs continue to work identically.
29
36
  */
30
37
  export function resolveConfig(cfg = {}) {
31
38
  const geminiApiKey = resolveEnv(cfg.geminiApiKey ?? "");
32
39
  const postgresUrl = resolveEnv(cfg.postgresUrl ?? DEFAULT_POSTGRES_URL);
33
40
  const lightragUrl = resolveEnv(cfg.lightragUrl ?? "");
34
41
  const lightragApiKey = resolveEnv(cfg.lightragApiKey ?? "");
42
+ const jina = (cfg.jina ?? {});
43
+ const router = (jina.router ?? {});
44
+ const reranker = (jina.pgvectorReranker ?? {});
45
+ const jinaApiKey = resolveEnv(jina.apiKey ?? "");
46
+ const routerClassifierId = resolveEnv(router.classifierId ?? "");
35
47
  return {
36
48
  enabled: cfg.enabled !== false,
37
49
  geminiApiKey,
@@ -46,6 +58,20 @@ export function resolveConfig(cfg = {}) {
46
58
  lightragQueryMode: cfg.lightragQueryMode ?? DEFAULT_LIGHTRAG_MODE,
47
59
  lightragMaxChars: cfg.lightragMaxChars ?? DEFAULT_LIGHTRAG_MAX_CHARS,
48
60
  lightragEnabled: cfg.lightragEnabled !== false && Boolean(lightragUrl),
61
+ // Jina shared key (used by router and/or reranker)
62
+ jinaApiKey,
63
+ // Router — disabled by default, even with a Jina key present, so
64
+ // operators must opt in explicitly. "heuristic" mode is the safest
65
+ // entry point: zero cost, deterministic.
66
+ routerEnabled: router.enabled === true,
67
+ routerMode: router.mode ?? DEFAULT_ROUTER_MODE,
68
+ routerClassifierId,
69
+ // Pgvector reranker — disabled by default. Requires both the toggle
70
+ // and a Jina key to actually activate at runtime (the handler checks
71
+ // this combination before calling).
72
+ pgvectorRerankerEnabled: reranker.enabled === true && Boolean(jinaApiKey),
73
+ pgvectorRerankerModel: reranker.model ?? DEFAULT_RERANKER_MODEL,
74
+ pgvectorRerankerTopN: reranker.topN ?? DEFAULT_RERANKER_TOP_N,
49
75
  };
50
76
  }
51
77
  //# sourceMappingURL=config.js.map
package/dist/config.js.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"config.js","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,gCAAgC;AAChC,EAAE;AACF,2EAA2E;AAC3E,6DAA6D;AAQ7D;;;;;GAKG;AACH,MAAM,UAAU,UAAU,CAAI,KAAQ;IACpC,IAAI,OAAO,KAAK,KAAK,QAAQ;QAAE,OAAO,KAAK,CAAC;IAC5C,OAAO,KAAK,CAAC,OAAO,CAAC,cAAc,EAAE,CAAC,CAAC,EAAE,IAAY,EAAE,EAAE;QACvD,OAAO,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC;IACjC,CAAC,CAAiB,CAAC;AACrB,CAAC;AAED,MAAM,oBAAoB,GAAG,kDAAkD,CAAC;AAChF,MAAM,mBAAmB,GAAG,CAAC,mBAAmB,CAAC,CAAC;AAClD,MAAM,aAAa,GAAG,CAAC,CAAC;AACxB,MAAM,uBAAuB,GAAG,GAAG,CAAC;AACpC,MAAM,wBAAwB,GAAG,IAAI,CAAC;AACtC,MAAM,qBAAqB,GAAsB,QAAQ,CAAC;AAC1D,MAAM,0BAA0B,GAAG,IAAI,CAAC;AAExC;;;;GAIG;AACH,MAAM,UAAU,aAAa,CAC3B,MAA6B,EAAE;IAE/B,MAAM,YAAY,GAAG,UAAU,CAAC,GAAG,CAAC,YAAY,IAAI,EAAE,CAAC,CAAC;IACxD,MAAM,WAAW,GAAG,UAAU,CAAC,GAAG,CAAC,WAAW,IAAI,oBAAoB,CAAC,CAAC;IACxE,MAAM,WAAW,GAAG,UAAU,CAAC,GAAG,CAAC,WAAW,IAAI,EAAE,CAAC,CAAC;IACtD,MAAM,cAAc,GAAG,UAAU,CAAC,GAAG,CAAC,cAAc,IAAI,EAAE,CAAC,CAAC;IAE5D,OAAO;QACL,OAAO,EAAE,GAAG,CAAC,OAAO,KAAK,KAAK;QAC9B,YAAY;QACZ,WAAW;QACX,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,mBAAmB;QACnD,IAAI,EAAE,GAAG,CAAC,IAAI,IAAI,aAAa;QAC/B,cAAc,EAAE,GAAG,CAAC,cAAc,IAAI,uBAAuB;QAC7D,cAAc,EAAE,GAAG,CAAC,cAAc,IAAI,wBAAwB;QAC9D,eAAe,EAAE,GAAG,CAAC,eAAe,KAAK,KAAK,IAAI,OAAO,CAAC,YAAY,CAAC;QACvE,WAAW;QACX,cAAc;QACd,iBAAiB,EAAE,GAAG,CAAC,iBAAiB,IAAI,qBAAqB;QACjE,gBAAgB,EAAE,GAAG,CAAC,gBAAgB,IAAI,0BAA0B;QACpE,eAAe,EAAE,GAAG,CAAC,eAAe,KAAK,KAAK,IAAI,OAAO,CAAC,WAAW,CAAC;KACvE,CAAC;AACJ,CAAC"}
1
+ {"version":3,"file":"config.js","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,gCAAgC;AAChC,EAAE;AACF,2EAA2E;AAC3E,6DAA6D;AAY7D;;;;;GAKG;AACH,MAAM,UAAU,UAAU,CAAI,KAAQ;IACpC,IAAI,OAAO,KAAK,KAAK,QAAQ;QAAE,OAAO,KAAK,CAAC;IAC5C,OAAO,KAAK,CAAC,OAAO,CAAC,cAAc,EAAE,CAAC,CAAC,EAAE,IAAY,EAAE,EAAE;QACvD,OAAO,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC;IACjC,CAAC,CAAiB,CAAC;AACrB,CAAC;AAED,MAAM,oBAAoB,GAAG,kDAAkD,CAAC;AAChF,MAAM,mBAAmB,GAAG,CAAC,mBAAmB,CAAC,CAAC;AAClD,MAAM,aAAa,GAAG,CAAC,CAAC;AACxB,MAAM,uBAAuB,GAAG,GAAG,CAAC;AACpC,MAAM,wBAAwB,GAAG,IAAI,CAAC;AACtC,MAAM,qBAAqB,GAAsB,QAAQ,CAAC;AAC1D,MAAM,0BAA0B,GAAG,IAAI,CAAC;AAExC,MAAM,mBAAmB,GAAoC,WAAW,CAAC;AACzE,MAAM,sBAAsB,GAAkB,oCAAoC,CAAC;AACnF,MAAM,sBAAsB,GAAG,CAAC,CAAC;AAEjC;;;;;;;;GAQG;AACH,MAAM,UAAU,aAAa,CAC3B,MAA6B,EAAE;IAE/B,MAAM,YAAY,GAAG,UAAU,CAAC,GAAG,CAAC,YAAY,IAAI,EAAE,CAAC,CAAC;IACxD,MAAM,WAAW,GAAG,UAAU,CAAC,GAAG,CAAC,WAAW,IAAI,oBAAoB,CAAC,CAAC;IACxE,MAAM,WAAW,GAAG,UAAU,CAAC,GAAG,CAAC,WAAW,IAAI,EAAE,CAAC,CAAC;IACtD,MAAM,cAAc,GAAG,UAAU,CAAC,GAAG,CAAC,cAAc,IAAI,EAAE,CAAC,CAAC;IAE5D,MAAM,IAAI,GAAG,CAAC,GAAG,CAAC,IAAI,IAAI,EAAE,CAAqB,CAAC;IAClD,MAAM,MAAM,GAAG,CAAC,IAAI,CAAC,MAAM,IAAI,EAAE,CAAuB,CAAC;IACzD,MAAM,QAAQ,GAAG,CAAC,IAAI,CAAC,gBAAgB,IAAI,EAAE,CAAiC,CAAC;IAC/E,MAAM,UAAU,GAAG,UAAU,CAAC,IAAI,CAAC,MAAM,IAAI,EAAE,CAAC,CAAC;IACjD,MAAM,kBAAkB,GAAG,UAAU,CAAC,MAAM,CAAC,YAAY,IAAI,EAAE,CAAC,CAAC;IAEjE,OAAO;QACL,OAAO,EAAE,GAAG,CAAC,OAAO,KAAK,KAAK;QAC9B,YAAY;QACZ,WAAW;QACX,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,mBAAmB;QACnD,IAAI,EAAE,GAAG,CAAC,IAAI,IAAI,aAAa;QAC/B,cAAc,EAAE,GAAG,CAAC,cAAc,IAAI,uBAAuB;QAC7D,cAAc,EAAE,GAAG,CAAC,cAAc,IAAI,wBAAwB;QAC9D,eAAe,EAAE,GAAG,CAAC,eAAe,KAAK,KAAK,IAAI,OAAO,CAAC,YAAY,CAAC;QACvE,WAAW;QACX,cAAc;QACd,iBAAiB,EAAE,GAAG,CAAC,iBAAiB,IAAI,qBAAqB;QACjE,gBAAgB,EAAE,GAAG,CAAC,gBAAgB,IAAI,0BAA0B;QACpE,eAAe,EAAE,GAAG,CAAC,eAAe,KAAK,KAAK,IAAI,OAAO,CAAC,WAAW,CAAC;QAEtE,mDAAmD;QACnD,UAAU;QAEV,iEAAiE;QACjE,mEAAmE;QACnE,yCAAyC;QACzC,aAAa,EAAE,MAAM,CAAC,OAAO,KAAK,IAAI;QACtC,UAAU,EAAE,MAAM,CAAC,IAAI,IAAI,mBAAmB;QAC9C,kBAAkB;QAElB,oEAAoE;QACpE,qEAAqE;QACrE,oCAAoC;QACpC,uBAAuB,EAAE,QAAQ,CAAC,OAAO,KAAK,IAAI,IAAI,OAAO,CAAC,UAAU,CAAC;QACzE,qBAAqB,EAAE,QAAQ,CAAC,KAAK,IAAI,sBAAsB;QAC/D,oBAAoB,EAAE,QAAQ,CAAC,IAAI,IAAI,sBAAsB;KAC9D,CAAC;AACJ,CAAC"}