@xdarkicex/openclaw-memory-libravdb 1.3.5

package/README.md

@@ -0,0 +1,46 @@
+# LibraVDB Memory
+
+## Install
+
+```bash
+openclaw plugins install @xdarkicex/openclaw-memory-libravdb
+```
+
+The installer builds the Go sidecar, provisions the bundled embedding/runtime assets, optionally provisions the T5 summarizer, and fails fast if the sidecar cannot pass its startup health check.
+
+Minimum host version:
+
+- OpenClaw `>= 2026.3.22`
+
+Security note:
+
+- `scripts/setup.ts` verifies SHA-256 checksums for downloaded sidecar/runtime/model assets
+- the sidecar installer downloads prebuilt sidecar release assets only from `github.com/xDarkicex/openclaw-memory-libravdb` releases
+- after install, the plugin makes no required network calls for embedding or extractive compaction
+- the only optional runtime network path is an explicitly configured remote summarizer endpoint such as `ollama-local`
+
+## Activate
+
+Add this to `~/.openclaw/openclaw.json`:
+
+```json
+{
+  "plugins": {
+    "slots": {
+      "memory": "libravdb-memory"
+    }
+  }
+}
+```
+
+Without the `plugins.slots.memory` entry, OpenClaw's default memory continues to run in parallel and this plugin does not take over the exclusive memory slot.
+
+## Verify
+
+Run:
+
+```bash
+openclaw memory status
+```
+
+Expected output includes a readable status table showing the sidecar is running, stored turn/memory counts, the active ingestion gate threshold, and whether the abstractive summarizer is provisioned.

package/docs/README.md

@@ -0,0 +1,14 @@
+# Documentation Index
+
+- [installation.md](./installation.md) - Complete install, activation, verification, and troubleshooting reference.
+- [architecture.md](./architecture.md) - End-to-end component model, turn lifecycle, compaction flow, and degraded behavior.
+- [problem.md](./problem.md) - Technical argument for replacing the stock OpenClaw memory lifecycle in this use case.
+- [mathematics.md](./mathematics.md) - Formal reference for hybrid scoring, decay, token budgeting, Matryoshka retrieval, and compaction.
+- [gating.md](./gating.md) - Full derivation and calibration guide for the domain-adaptive gating scalar.
+- [implementation.md](./implementation.md) - Non-obvious implementation decisions and their rationale.
+- [dependencies.md](./dependencies.md) - Why LibraVDB and slab-based storage were chosen for this plugin.
+- [models.md](./models.md) - ONNX model strategy, latency trade-offs, and shipped model roles.
+- [security.md](./security.md) - Security model, untrusted-memory framing, isolation guarantees, and deletion boundaries.
+- [contributing.md](./contributing.md) - Contributor workflow, prerequisites, and invariant test expectations.
+- [architecture-decisions/README.md](./architecture-decisions/README.md) - Index of the repository ADRs.
+- [embedding-profiles.md](./embedding-profiles.md) - Shipped embedding profile baseline and current profile metadata.

package/docs/architecture-decisions/README.md

@@ -0,0 +1,6 @@
+# Architecture Decisions
+
+- [adr-001-onnx-over-ollama.md](./adr-001-onnx-over-ollama.md)
+- [adr-002-libravdb-over-lancedb.md](./adr-002-libravdb-over-lancedb.md)
+- [adr-003-convex-gating-over-threshold.md](./adr-003-convex-gating-over-threshold.md)
+- [adr-004-sidecar-over-native-ts.md](./adr-004-sidecar-over-native-ts.md)

package/docs/architecture-decisions/adr-001-onnx-over-ollama.md

@@ -0,0 +1,21 @@
+# ADR-001: ONNX Over Ollama
+
+## Context
+
+The plugin needs local embedding inference on the prompt-assembly critical path and optional local summarization for compaction.
+
+## Decision
+
+Use ONNX-first local inference for embedding and optional summarization. Treat Ollama as an optional external backend, not the primary dependency.
+
+## Alternatives Considered
+
+- Ollama for both embedding and summarization
+- remote inference APIs
+
+## Consequences
+
+- predictable latency
+- deterministic embeddings
+- offline operation
+- larger local artifact footprint

package/docs/architecture-decisions/adr-002-libravdb-over-lancedb.md

@@ -0,0 +1,19 @@
+# ADR-002: LibraVDB Over LanceDB
+
+## Context
+
+The plugin needs multi-scope namespacing, delete-heavy compaction flows, and local-first operation without a Python dependency chain.
+
+## Decision
+
+Use LibraVDB as the vector store.
+
+## Alternatives Considered
+
+- LanceDB
+
+## Consequences
+
+- better fit for collection-scoped lifecycle management
+- more control over local operational behavior
+- deeper ownership of vector store behavior and tuning

package/docs/architecture-decisions/adr-003-convex-gating-over-threshold.md

@@ -0,0 +1,27 @@
+# ADR-003: Convex Gating Over Per-Domain Thresholds
+
+## Context
+
+A single conversational gating scalar suppressed useful technical workflow memory because conversational redundancy and technical redundancy mean different things.
+
+## Decision
+
+Use a convex mixture:
+
+$$
+G(t) = (1 - T(t))G_{\mathrm{conv}}(t) + T(t)G_{\mathrm{tech}}(t)
+$$
+
+instead of per-domain thresholds or user classification flags.
+
+## Alternatives Considered
+
+- separate thresholds for technical vs conversational users
+- explicit user-level mode flags
+- a larger conversational heuristic rule set
+
+## Consequences
+
+- one threshold instead of multiple user modes
+- continuous behavior on mixed content
+- greater observability through decomposed signals

package/docs/architecture-decisions/adr-004-sidecar-over-native-ts.md

@@ -0,0 +1,21 @@
+# ADR-004: Sidecar Over Native TypeScript
+
+## Context
+
+The plugin requires local vector storage, ONNX inference, transport isolation, and bounded failure semantics that should not crash the host chat session.
+
+## Decision
+
+Implement the memory engine as a Go sidecar with a narrow JSON-RPC transport boundary.
+
+## Alternatives Considered
+
+- native TypeScript implementation
+- WASM-only embedding and storage path
+
+## Consequences
+
+- strong process isolation
+- efficient local inference and storage integration
+- extra packaging complexity
+- a separate binary distribution story

package/docs/architecture.md

@@ -0,0 +1,188 @@
+# System Architecture
+
+This document describes the current implemented architecture, not just the
+design intent. Every component and data flow here maps to code in the
+repository as of the current `main` branch.
+
+## 1. Component Map
+
+```mermaid
+flowchart LR
+  Host["OpenClaw host process\n(TypeScript plugin shell)"]
+  CE["Context engine factory\nbootstrap / ingest / assemble / compact"]
+  MPS["memoryPromptSection\nuser+global recall"]
+  Runtime["Plugin runtime\nlazy sidecar startup + RPC client"]
+  Sidecar["Go sidecar process"]
+  RPC["JSON-RPC over newline-delimited frames\nUnix socket or TCP loopback on Windows"]
+  Store["LibraVDB store on disk"]
+  Session["session:<sessionId>"]
+  Turns["turns:<userId>"]
+  User["user:<userId>"]
+  Global["global"]
+  Dirty["_tier_dirty"]
+  Embed["ONNX embedding engine"]
+  Extractive["Extractive summarizer"]
+  T5["Optional ONNX T5 summarizer"]
+  Ollama["Optional Ollama summarizer endpoint"]
+
+  Host --> CE
+  Host --> MPS
+  CE --> Runtime
+  MPS --> Runtime
+  Runtime --> RPC
+  RPC --> Sidecar
+  Sidecar --> Embed
+  Sidecar --> Extractive
+  Sidecar --> T5
+  Sidecar --> Ollama
+  Sidecar --> Store
+  Store --> Session
+  Store --> Turns
+  Store --> User
+  Store --> Global
+  Store --> Dirty
+```
+
+Implementation anchors:
+
+- plugin entry: [`src/index.ts`](../src/index.ts)
+- lazy runtime startup: [`src/plugin-runtime.ts`](../src/plugin-runtime.ts)
+- sidecar supervision and endpoint discovery: [`src/sidecar.ts`](../src/sidecar.ts)
+- transport listener: [`sidecar/server/transport.go`](../sidecar/server/transport.go)
+- RPC method table: [`sidecar/server/rpc.go`](../sidecar/server/rpc.go)
+- store: [`sidecar/store/libravdb.go`](../sidecar/store/libravdb.go)
+
+## 2. Single-Turn Data Flow
+
+### 2.1 `ingest`
+
+Implemented in [`src/context-engine.ts`](../src/context-engine.ts).
+
+For every non-heartbeat message:
+
+1. The host gets an RPC client from the plugin runtime. This lazily starts the
+   sidecar if it is not already running.
+2. The message is written to `session:<sessionId>` with `type: "turn"`.
+3. If `message.role === "user"`, the same text is written to `turns:<userId>`.
+4. The host calls `gating_scalar` with `{ userId, text }`.
+5. If `g >= ingestionGateThreshold`, the turn is promoted into
+   `user:<userId>` with the full gating decomposition in metadata.
+
+Important constraints from the current implementation:
+
+- session insertion is fire-and-forget
+- durable promotion is best-effort
+- gating failure does not fail the user turn
+- assistant turns are stored in session memory but are not promoted into
+  durable user memory
+
+### 2.2 `memoryPromptSection`
+
+Implemented in [`src/memory-provider.ts`](../src/memory-provider.ts).
+
+Before the main assembly path runs, the plugin builds a lightweight recall
+section:
+
+1. search `user:<userId>`
+2. search `global`
+3. hybrid-rank the combined hits
+4. fit them to a fixed prompt budget of `800` estimated tokens
+5. return a textual header fragment for the host prompt
+
+This path does not search session memory. Its job is durable context recall, not
+active-turn recall.
+
+### 2.3 `assemble`
+
+Implemented in [`src/context-engine.ts`](../src/context-engine.ts).
+
+For the current query text (last message content), the host:
+
+1. builds an exclusion set from the most recent message ids
+2. searches `session:<sessionId>`, `user:<userId>`, and `global` in parallel
+3. hybrid-ranks the combined results using host-side scoring
+4. fits the ranked set to `tokenBudget * tokenBudgetFraction`
+5. prepends the selected memories as synthetic `system` messages
+6. returns both the expanded message array and a `systemPromptAddition`
+
+Current implementation details that matter:
+
+- user/global hits may be reused from the earlier prompt-section cache
+- `assemble` falls back to the unmodified message list on RPC failure
+- `assemble` does not mutate the original `messages` array in place; it returns
+  a new array
+
+## 3. Compaction Data Flow
+
+Implemented primarily in [`src/context-engine.ts`](../src/context-engine.ts)
+and [`sidecar/compact/summarize.go`](../sidecar/compact/summarize.go).
+
+When compaction is triggered:
+
+1. the host calls `compact_session` with `{ sessionId, force, targetSize }`
+2. the sidecar loads eligible non-summary turns from `session:<sessionId>`
+3. turns are sorted by `(ts, id)` and partitioned into deterministic
+   chronological clusters
+4. each cluster is routed to:
+   - extractive summarization by default
+   - optional abstractive summarization if `mean(gating_score) >= 0.60` and an
+     abstractive summarizer is ready
+5. the summary record is inserted back into the same session collection
+6. source turns are deleted only after summary insertion succeeds
+
+Current implementation facts:
+
+- compaction only touches `session:<sessionId>`
+- raw source turns are preserved if summary insertion fails
+- delete failure logs and leaves the inserted summary in place
+- compaction logs `cluster_id`, `mean_gating_score`, and `summarizer_used`
+
+## 4. Failure Modes and Degraded Behavior
+
+The table below reflects current code behavior, with notes where it diverges
+from the original spec phrasing.
+
+| Failure | Current behavior | User impact |
+|---|---|---|
+| Sidecar unavailable on first RPC use | `getRpc()` rejects when lazy startup or health check fails | That hook fails or falls back, but plugin registration itself does not crash eagerly |
+| Sidecar connection closes mid-session | `SidecarSupervisor` retries with exponential backoff until retry budget is exhausted, then enters degraded mode | Memory becomes unavailable until restart succeeds |
+| `memoryPromptSection` RPC failure | individual searches are caught and replaced with empty result sets | Prompt section becomes empty rather than crashing the run |
+| `assemble` RPC failure | returns original messages, original token count, and empty `systemPromptAddition` | That turn gets no recall augmentation |
+| `ingest` gating or durable insert failure | session write already happened; durable promotion is skipped | Session memory survives, durable memory may miss that turn |
+| Compaction summarizer unavailable | extractive summarizer remains required; optional abstractive path is skipped | Compaction still runs extractively when extractive is healthy |
+| Disk full or insert error | Go RPC returns an error; TypeScript caller logs or degrades | New records are not stored, but chat continues |
+| Empty lower Matryoshka tiers | cascade search naturally falls through because empty tiers return `best = 0.0` | Retrieval degrades to higher tiers without returning false confident exits |
+
+Relevant code:
+
+- retry/degraded behavior: [`src/sidecar.ts`](../src/sidecar.ts)
+- lazy startup and health gate: [`src/plugin-runtime.ts`](../src/plugin-runtime.ts)
+- compaction routing and insert/delete ordering:
+  [`sidecar/compact/summarize.go`](../sidecar/compact/summarize.go)
+
+## 5. Gating Decision Path
+
+The gating decision spans both layers:
+
+1. `ingest` writes the user turn to `turns:<userId>`
+2. the host calls `gating_scalar`
+3. the Go sidecar performs exactly two searches:
+   - `SearchText("turns:<userId>", text, 10, nil)`
+   - `SearchText("user:<userId>", text, 5, nil)`
+4. the sidecar computes `GatingSignals` with [`compact.ComputeGating`](../sidecar/compact/gate.go)
+5. the host compares `g` to `ingestionGateThreshold`
+6. on pass, the host writes the turn into `user:<userId>` with all gating
+   metadata fields
+7. later, compaction computes the mean `gating_score` of a cluster and may route
+   high-value clusters to the abstractive summarizer
+
+If the gate fails:
+
+- the turn still exists in `session:<sessionId>`
+- the turn still exists in `turns:<userId>`
+- the turn is not promoted into `user:<userId>`
+- downstream durable recall and compaction routing cannot use that turn's
+  gating metadata because it was never promoted
+
+That makes the gate a durable-memory admission control, not a full-ingestion
+blocker.

package/docs/contributing.md

@@ -0,0 +1,76 @@
+# Contributing
+
+## Prerequisites
+
+- Node.js `>= 22`
+- Go `>= 1.22` for development and local fallback builds
+- `pnpm`
+- OpenClaw CLI for end-to-end plugin testing
+
+## Core Validation Commands
+
+TypeScript and unit checks:
+
+```bash
+pnpm check
+```
+
+Integration tests:
+
+```bash
+npm run test:integration
+```
+
+Go sidecar tests:
+
+```bash
+cd sidecar
+env GOCACHE=/tmp/openclaw-memory-libravdb-gocache go test ./...
+env GOCACHE=/tmp/openclaw-memory-libravdb-gocache go test -race ./...
+```
+
+## Local Sidecar Build
+
+```bash
+bash scripts/build-sidecar.sh
+```
+
+This creates `.sidecar-bin/libravdb-sidecar` and copies locally available bundled assets into `.sidecar-bin/`.
+
+## Gating Invariants
+
+Do not weaken the gate invariants casually. The tests in `sidecar/compact/gate_test.go` check structural properties:
+
+- empty-memory novelty
+- saturation veto
+- convex boundedness
+- conversational collapse at `T = 0`
+- technical collapse at `T = 1`
+- non-overfiring conversational structure on code
+
+If you add a new signal, it must preserve those invariants.
+
+## Calibration Coverage
+
+There is not yet a dedicated `gate_calibration_test.go` golden set in the
+repository. Current gating correctness is enforced by the invariant suite in
+[`sidecar/compact/gate_test.go`](../sidecar/compact/gate_test.go).
+
+If you introduce new signals or change weighting behavior, do not only update
+the implementation. Add one of:
+
+- a new invariant if the change alters a structural property of the gate
+- a dedicated calibration/golden test file if the change adds new labeled
+  examples or expected decompositions
+
+Do not rewrite expectations just to make regressions disappear.
+
+## PR Expectations
+
+Before opening a PR:
+
+- `pnpm check` must pass
+- `go test -race ./...` from `sidecar/` must pass
+- any new gating signal must come with calibration or invariant coverage
+- any retrieval math change must be reflected in [mathematics.md](./mathematics.md)
+- any gating change must be reflected in [gating.md](./gating.md)

package/docs/dependencies.md

@@ -0,0 +1,38 @@
+# Dependency Rationale
+
+## LibraVDB over LanceDB
+
+LibraVDB was chosen as the vector store because the plugin needs more than a single-table embedding lookup.
+
+Key reasons:
+
+- collection-level namespacing for:
+  - `session:*`
+  - `turns:*`
+  - `user:*`
+  - `global`
+- delete and batch-delete operations used by compaction
+- local-first Go-native operation with no Python bridge or remote service dependency
+- retrieval infrastructure compatible with HNSW and future IVF/PQ-oriented layering
+
+LanceDB was the natural alternative. It is a solid choice for straightforward durable vector retrieval, but using it here would still have required additional machinery around:
+
+- scope isolation
+- delete-heavy compaction flows
+- local-first lifecycle management around a multi-scope memory design
+
+The decision was therefore about operational fit, not abstract preference.
+
+## Slabby
+
+The LibraVDB profiling work showed that this workload is allocation-sensitive, especially in repeated insert/search paths over vector-heavy collections.
+
+Slab-style raw-vector storage was selected because:
+
+- vectors are fixed-size payloads
+- collections grow in bursty append patterns
+- compaction and search create pressure on allocation churn
+
+The measured conclusion from the internal profiling pass was that slab-backed raw-vector storage was performance-competitive with the plain in-memory backend while making allocation behavior more predictable. The main trade-off is reserved-but-unused capacity, which is acceptable for this local sidecar workload.
+
+The dependency is therefore justified by workload shape, not by novelty.

package/docs/embedding-profiles.md

@@ -0,0 +1,42 @@
+# Embedding Profiles
+
+The plugin now supports a lightweight `embeddingProfile` setting for named local model metadata defaults.
+
+Default selection baseline as of `2026-03-28`:
+
+- default embedding profile: `nomic-embed-text-v1.5`
+- bundled fallback profile: `all-minilm-l6-v2`
+
+Why:
+
+- MiniLM and Nomic are equivalent on the current lexical and paraphrase baseline.
+- Nomic materially outperforms MiniLM on cross-domain ranking quality.
+- Nomic is the only profile that clears the long-context baseline once sliding-window document embedding is applied.
+- Adversarial lexical traps remain reranker-window cases, but Nomic still narrows the relevant-vs-distractor margin materially.
+
+Current shipped profile names:
+
+- `all-minilm-l6-v2`
+  - family: `all-minilm-l6-v2`
+  - dimensions: `384`
+  - normalize: `true`
+  - max context tokens: `128`
+
+- `nomic-embed-text-v1.5`
+  - family: `nomic-embed-text-v1.5`
+  - dimensions: `768`
+  - normalize: `true`
+  - max context tokens: `8192`
+
+How it works:
+
+- `embeddingProfile` supplies metadata defaults like family, dimensions, and normalize behavior.
+- `onnx-local` still requires local model assets through `embeddingModelPath`, typically a directory containing `embedding.json`.
+- The manifest may override or refine the profile, but explicit dimension mismatches fail closed.
+- The sidecar store persists an embedding fingerprint, so reopening an existing store with a different effective model profile will fail instead of silently mixing vector spaces.
+
+Recommended usage:
+
+- `bundled` for the shipped default path, which now prefers Nomic and falls back to MiniLM if the primary profile is unavailable.
+- `onnx-local` plus `embeddingProfile` when a power user wants a known model family like Nomic with local assets.
+- treat remote/Ollama providers as future separate backend types, not as overloads of `custom-local`.