@xdarkicex/openclaw-memory-libravdb 1.3.12 → 1.3.17

package/README.md

@@ -1,84 +1,283 @@
-# LibraVDB Memory
+# LibraVDB Memory for OpenClaw
 
-## Install
+[![Go](https://img.shields.io/badge/Go-1.25%2B-00ADD8?logo=go&logoColor=white)](./sidecar/go.mod)
+[![TypeScript](https://img.shields.io/badge/TypeScript-5.x-3178C6?logo=typescript&logoColor=white)](./package.json)
+[![OpenClaw](https://img.shields.io/badge/OpenClaw-memory%20plugin-111827)](./openclaw.plugin.json)
 
-Recommended on macOS:
+`@xdarkicex/openclaw-memory-libravdb` is a local-first OpenClaw memory system
+for people who want more than "top-k vectors plus a prompt footer."
 
-```bash
-brew tap xDarkicex/openclaw-libravdb-memory
-brew install libravdbd
-brew services start libravdbd
-openclaw plugins install @xdarkicex/openclaw-memory-libravdb
-```
+It replaces the default lightweight memory path with a full context lifecycle:
 
-Then activate the plugin in `~/.openclaw/openclaw.json`.
+- active session memory
+- durable per-user memory
+- shared global memory
+- continuity-aware compaction
+- authored context partitioning
+- hybrid scoring across scope, recency, and similarity
 
-Manual plugin install:
+This repository pairs a TypeScript OpenClaw plugin with a Go daemon backed by
+`libraVDB`. The plugin owns both the `memory` and `contextEngine` slots, while
+the daemon handles embeddings, retrieval, storage, and compaction.
+On newer OpenClaw builds, it also bridges the built-in `memory_search` runtime
+to the same libraVDB sidecar instead of leaving that tool inert.
 
-```bash
-openclaw plugins install @xdarkicex/openclaw-memory-libravdb
-```
+## Why This Exists
 
-The published plugin is connect-only. It does not spawn a local binary during install or at runtime. For durable memory, run a local `libravdbd` daemon separately and point the plugin at its endpoint.
+The stock "single memory bucket" pattern is good for simple persistence, but it
+starts to break down when you care about:
 
-Minimum host version:
+- keeping the newest working context raw and intact
+- separating ephemeral session state from durable memory
+- avoiding long-session prompt collapse
+- preserving authored instructions differently from recalled user content
+- treating memory retrieval as a ranked assembly problem instead of plain
+  nearest-neighbor lookup
 
-- OpenClaw `>= 2026.3.22`
+LibraVDB Memory exists for that harder class of memory problem.
 
-Security note:
+## What Makes It Different
 
-- the published plugin package contains no `postinstall`, no `openclaw.setup`, and no direct `child_process` usage
-- the plugin only connects to a local `libravdbd` endpoint such as `unix:/Users/<you>/.clawdb/run/libravdb.sock` or `tcp:127.0.0.1:37421`
-- after install, the plugin makes no required network calls for embedding or extractive compaction
-- the only optional runtime network path is an explicitly configured remote summarizer endpoint such as `ollama-local`
+These are the core differentiators the project is built around:
 
-## Daemon
+- Dual slot ownership: the plugin owns both memory prompt injection and the
+  full context lifecycle.
+- Built-in `memory_search` bridge: newer OpenClaw memory runtime calls are
+  routed into the same sidecar-backed retrieval path.
+- Lifecycle hint adoption: `before_reset` and `session_end` are used as
+  advisory signals into the sidecar without giving OpenClaw control of ingest
+  or compaction.
+- Sidecar-owned lifecycle journal: reset/end hints are recorded internally for
+  debugging and auditing without entering normal memory retrieval.
+  The journal is bounded by a sidecar retention cap so it does not grow
+  forever.
+- Local-first runtime: the core path does not depend on external embedding
+  services.
+- Three-tier memory: session, durable user, and global memory stay distinct.
+- Hybrid scoring: retrieval is ranked by semantic similarity, recency, scope,
+  and summary quality instead of cosine alone.
+- Automatic compaction: long sessions compact behind a protected recent tail.
+- Crash-resilient IPC: the host talks to a sidecar over a stable local socket
+  or loopback TCP endpoint with degraded-mode fallback.
 
-Install and start `libravdbd` separately, then point the plugin at the running daemon if you do not want the default endpoint.
+## Quick Start
 
-Default endpoints:
+The supported install flow is:
 
-- macOS/Linux: `unix:$HOME/.clawdb/run/libravdb.sock`
-- Windows: `tcp:127.0.0.1:37421`
-
-Phase 2 packaging assets now live under [`packaging/`](./packaging):
-
-- `packaging/systemd/libravdbd.service` for Linux user services
-- `packaging/launchd/com.xdarkicex.libravdbd.plist` for macOS LaunchAgents
-- `packaging/homebrew/libravdbd.rb.tmpl` as the source template for a generated Homebrew formula
-
-Recommended service startup commands:
-
-- macOS: `brew services start libravdbd`
-- Linux: `systemctl --user enable --now libravdbd.service`
+```bash
+brew tap xDarkicex/openclaw-libravdb-memory
+brew install libravdbd
+brew services start libravdbd
+openclaw plugins install @xdarkicex/openclaw-memory-libravdb
+```
 
-## Activate
+The Homebrew formula installs the daemon plus the bundled ONNX Runtime, embedding assets, and T5 summarizer assets it needs to boot cleanly on supported platforms.
 
-Add this to `~/.openclaw/openclaw.json`:
+Then assign the plugin to both required OpenClaw slots in
+`~/.openclaw/openclaw.json`:
 
 ```json
 {
   "plugins": {
     "slots": {
-      "memory": "libravdb-memory"
+      "memory": "libravdb-memory",
+      "contextEngine": "libravdb-memory"
     },
     "configs": {
       "libravdb-memory": {
-        "sidecarPath": "unix:/Users/<you>/.clawdb/run/libravdb.sock"
+        "sidecarPath": "auto"
       }
     }
   }
 }
 ```
 
-Without the `plugins.slots.memory` entry, OpenClaw's default memory continues to run in parallel and this plugin does not take over the exclusive memory slot.
-
-## Verify
-
-Run:
+Verify the setup:
 
 ```bash
 openclaw memory status
 ```
 
-Expected output includes a readable status table showing the daemon is reachable, stored turn/memory counts, the active ingestion gate threshold, and whether the abstractive summarizer is provisioned.
+Expected healthy state:
+
+- the daemon is reachable
+- the plugin is active as the memory provider
+- the runtime can report stored counts and model readiness
+
+## Install Model
+
+This plugin is intentionally **connect-only** at install time.
+
+It does not compile Go code during plugin installation, and it does not manage
+daemon lifecycle automatically from the npm package. That is deliberate: some
+OpenClaw environments are strict about postinstall behavior, daemon spawning,
+and anything that looks like binary bootstrap or process management.
+
+Current model:
+
+- npm/OpenClaw package: plugin code and docs
+- `libravdbd`: installed and managed separately
+- default daemon endpoint on macOS/Linux:
+  `unix:$HOME/.clawdb/run/libravdb.sock`
+- default daemon endpoint on Windows:
+  `tcp:127.0.0.1:37421`
+
+If your daemon runs elsewhere, set an explicit `sidecarPath`, for example:
+
+- `unix:/custom/path/libravdb.sock`
+- `tcp:127.0.0.1:9999`
+
+## Architecture At A Glance
+
+```text
+OpenClaw host
+  -> memoryPromptSection (durable user/global recall)
+  -> memory runtime bridge (built-in memory_search)
+  -> context engine (bootstrap / ingest / assemble / compact)
+  -> plugin runtime
+  -> JSON-RPC
+  -> libravdbd
+  -> libraVDB + local embedding/summarization stack
+```
+
+The main runtime split is:
+
+- TypeScript host layer:
+  - OpenClaw plugin registration
+  - prompt assembly
+  - hybrid ranking
+  - continuity-aware token budgeting
+  - degraded-mode behavior
+- Go daemon layer:
+  - vector storage
+  - embeddings
+  - search RPCs
+  - compaction and summarization
+  - stable local IPC endpoint
+
+For the implemented architecture map, read
+[docs/architecture.md](./docs/architecture.md).
+
+## Retrieval Model
+
+The assembly path is not "just search some vectors and paste the top hits."
+
+It combines:
+
+- session search for current-work relevance
+- durable user recall for long-lived personal context
+- global recall for shared facts
+- authored invariant and variant context
+- continuity-preserving recent-tail injection
+- token-budgeted fitting
+
+The ranking model currently blends:
+
+- semantic similarity
+- scope weighting
+- recency decay
+- summary quality attenuation
+
+The formal math lives in:
+
+- [docs/mathematics-v2.md](./docs/mathematics-v2.md)
+- [docs/continuity.md](./docs/continuity.md)
+- [docs/ast-v2.md](./docs/ast-v2.md)
+- [docs/elevated-guidance.md](./docs/elevated-guidance.md)
+
+## Compaction Model
+
+This system does not treat long chats as append-only forever.
+
+Older session turns compact behind a protected recent tail, so the plugin can:
+
+- keep the newest working context raw
+- preserve adjacency-sensitive continuity near the boundary
+- promote older material into summaries
+- avoid letting long sessions drown their own prompt budget
+
+Compaction is designed as part of the memory system itself, not as a separate
+maintenance convenience.
+
+## For Power Users
+
+If you are evaluating this as an operator or advanced OpenClaw user, the key
+practical points are:
+
+- This plugin should own both `memory` and `contextEngine`. Partial slot
+  assignment is a misconfiguration.
+- On hosts that expose `registerMemoryRuntime`, the built-in `memory_search`
+  tool now searches the same libraVDB-backed memory stores.
+- The daemon is a separate operational unit. Treat plugin lifecycle and daemon
+  lifecycle as different concerns.
+- The system is local-first by design. The critical retrieval path does not
+  require a remote embedding service.
+- The sidecar transport is stable and explicit, which makes it service-manager
+  friendly on macOS, Linux, and Windows.
+
+Good entry points:
+
+- [docs/install.md](./docs/install.md)
+- [docs/installation.md](./docs/installation.md)
+- [docs/uninstall.md](./docs/uninstall.md)
+- [docs/implementation.md](./docs/implementation.md)
+
+## For Researchers And Builders
+
+If you are studying retrieval, memory systems, or agent architecture, the
+interesting parts of this repo are:
+
+- continuity-aware assembly:
+  `C_total(q) = I union T_recent union Proj(V_rest, q)`
+- hybrid ranking instead of pure cosine retrieval
+- separation of authored invariants from searchable authored lore
+- durable-memory admission via domain-adaptive gating
+- local daemon architecture rather than in-process TS vector plumbing
+- compaction that preserves recent working context instead of flattening the
+  whole transcript
+
+Start here:
+
+- [docs/problem.md](./docs/problem.md)
+- [docs/architecture.md](./docs/architecture.md)
+- [docs/mathematics-v2.md](./docs/mathematics-v2.md)
+- [docs/gating.md](./docs/gating.md)
+- [docs/continuity.md](./docs/continuity.md)
+
+## Runtime Facts
+
+- npm package: `@xdarkicex/openclaw-memory-libravdb`
+- OpenClaw plugin id: `libravdb-memory`
+- minimum host version: `openclaw >= 2026.3.22`
+- default daemon data path: `$HOME/.clawdb/data.libravdb`
+- default daemon endpoint on macOS/Linux:
+  `unix:$HOME/.clawdb/run/libravdb.sock`
+- default daemon endpoint on Windows:
+  `tcp:127.0.0.1:37421`
+
+## Repository Guide
+
+- [docs/install.md](./docs/install.md): quick install and lifecycle guide
+- [docs/installation.md](./docs/installation.md): full installation and
+  packaging reference
+- [docs/uninstall.md](./docs/uninstall.md): clean shutdown and removal
+- [docs/architecture.md](./docs/architecture.md): current implemented system
+  architecture
+- [docs/implementation.md](./docs/implementation.md): important implementation
+  contracts
+- [docs/mathematics-v2.md](./docs/mathematics-v2.md): formal scoring and
+  optimization reference
+
+## Current Constraint
+
+Because OpenClaw environments can be strict about postinstall downloads,
+daemon spawning, and scanner-visible binary bootstrap behavior, the cleanest
+supported user path today is:
+
+- install plugin
+- install daemon
+- assign both slots
+- let the plugin connect to a stable local endpoint
+
+That tradeoff is intentional. It keeps the plugin installation surface simple
+and auditable while preserving the full local memory engine at runtime.

package/docs/README.md

@@ -5,12 +5,15 @@ legacy non-versioned predecessor also exists. Older non-versioned docs are kept
 to preserve project history and design evolution.
 
 - [installation.md](./installation.md) - Complete install, activation, verification, and troubleshooting reference.
+- [install.md](./install.md) - Practical lifecycle guide for Homebrew, the OpenClaw plugin, and manual daemon management.
+- [uninstall.md](./uninstall.md) - Clean shutdown and removal guide for the plugin, daemon, and optional local data.
 - [architecture.md](./architecture.md) - End-to-end component model, turn lifecycle, compaction flow, and degraded behavior.
 - [problem.md](./problem.md) - Technical argument for replacing the stock OpenClaw memory lifecycle in this use case.
 - [mathematics-v2.md](./mathematics-v2.md) - Formal reference for hybrid scoring, decay, token budgeting, Matryoshka retrieval, compaction, and planned two-pass retrieval.
 - [compaction-evaluation.md](./compaction-evaluation.md) - Real-model benchmark notes for T5 summary confidence, Nomic-space preservation, and the hard preservation gate.
 - [continuity.md](./continuity.md) - Continuity model for invariant context, preserved recent raw session tail, and retrieved older memory.
 - [ast-v2.md](./ast-v2.md) - Reviewed authoritative AST partitioning reference for authored Markdown hard invariants, soft invariants, and variant lore.
+- [elevated-guidance.md](./elevated-guidance.md) - Tier 1.5 protected-shard and elevated-guidance model for preserving shadow rules through compaction.
 - [ast.md](./ast.md) - Historical predecessor to `ast-v2.md`, kept to show design evolution and earlier bugs.
 - [gating.md](./gating.md) - Full derivation and calibration guide for the domain-adaptive gating scalar.
 - [implementation.md](./implementation.md) - Non-obvious implementation decisions and their rationale.

package/docs/ast-v2.md

@@ -33,6 +33,27 @@ We formalize this as a binary promotion scalar \(\sigma: N_d \to \{0,1\}\). This
 \end{cases}
 \]
 
+To reason about tuning noise in the bigram set \(W_{\mathrm{deontic}}\), we
+also define the paragraph classifier error rates:
+\[
+P_{\mathrm{fp}} = P(\sigma(n) = 1 \mid n \text{ is narrative lore})
+\]
+\[
+P_{\mathrm{fn}} = P(\sigma(n) = 0 \mid n \text{ is behavioral rule})
+\]
+
+For authored documents whose lore paragraphs would otherwise remain in
+\(\mathcal{V}_d\), the expected Tier-2 waste introduced by false positives is:
+\[
+\mathbb{E}[\mathrm{wasted\ toks\ in\ }\mathcal{I}_2]
+=
+P_{\mathrm{fp}} \cdot |\mathcal{V}_{d,\mathrm{paragraphs}}| \cdot \mathbb{E}[\mathrm{toks}(n)]
+\]
+
+This gives the parser a concrete quantity to minimize when adjusting
+\(W_{\mathrm{deontic}}\), while \(P_{\mathrm{fn}}\) measures the risk of leaving
+true behavioral rules behind in \(\mathcal{V}_d\).
+
 *Implemented via `NewDeonticFrame` and `EvaluateText` in the zero-allocation byte lexer.*
 
 ## 3. The Three-Tier Structural Indicator Function \(\iota\)
@@ -59,12 +80,27 @@ We define the structural indicator function \(\iota: N_d \to \{0,1,2\}\) mapping
 ## 4. Corpus Decomposition and Set Integration
 
 For any document \(d \in \mathbf{D}_{\text{agents}} \cup \mathbf{D}_{\text{souls}}\), the node set \(N_d\) is partitioned cleanly into three sets:
-- **Hard Directives:** \(\mathcal{I}_{1d} = \{ n \in N_d \mid \iota(n) = 1 \}\)
+- **Hard Directives:** \(\mathcal{I}_{1d} = \langle n \in N_d \mid \iota(n) = 1 \rangle\), ordered by \(\mathrm{position}(n)\) ascending, where \(\mathrm{position}(n)\) is the byte offset of node \(n\) in \(d_{\mathrm{raw}}\)
 - **Soft Directives:** \(\mathcal{I}_{2d} = \{ n \in N_d \mid \iota(n) = 2 \}\)
 - **Contextual Lore:** \(\mathcal{V}_d = \{ n \in N_d \mid \iota(n) = 0 \}\)
 
 *Partition Completeness:* Because \(\iota(n)\) maps every node to exactly one integer in \(\{0, 1, 2\}\), the resulting sets are mutually exclusive and collectively exhaustive:
-\[ \mathcal{I}_{1d} \cup \mathcal{I}_{2d} \cup \mathcal{V}_d = N_d \quad \text{and} \quad \mathcal{I}_{1d} \cap \mathcal{I}_{2d} \cap \mathcal{V}_d = \emptyset \]
+\[
+\mathcal{I}_{1d} \cup \mathcal{I}_{2d} \cup \mathcal{V}_d = N_d
+\]
+\[
+\mathcal{I}_{1d} \cap \mathcal{I}_{2d} = \emptyset
+\]
+\[
+\mathcal{I}_{1d} \cap \mathcal{V}_d = \emptyset
+\]
+\[
+\mathcal{I}_{2d} \cap \mathcal{V}_d = \emptyset
+\]
+
+These pairwise disjointness statements follow directly from \(\iota\) being a
+single-valued total function into \(\{0,1,2\}\): no node can be assigned to
+more than one tier simultaneously.
 
 These sets integrate into the global corpus. Let \(\mathbf{D}_{\text{standard}}\) be the set of standard memory documents (non-core files). We formally define the standard variant node set as \(\mathcal{V}_{\text{standard}} = \bigcup_{d \in \mathbf{D}_{\text{standard}}} E(d)\). The global corpus is then:
 \[ \mathcal{I}_1 = \bigcup_{d} \mathcal{I}_{1d} \qquad \mathcal{I}_2 = \bigcup_{d} \mathcal{I}_{2d} \qquad \mathcal{V} = \mathcal{V}_{\text{standard}} \cup \left( \bigcup_{d} \mathcal{V}_d \right) \]
@@ -86,7 +122,7 @@ For Hard Invariants (\(\alpha_1\)):
 \[ \sum_{n \in \mathcal{I}_{1d}} \mathrm{toks}(n) \le \alpha_1 \tau \implies \text{fast-fail and reject agent load if exceeded} \]
 
 For Soft Invariants (\(\alpha_2\)):
-\[ \sum_{n \in \mathcal{I}_{2d}} \mathrm{toks}(n) \le \alpha_2 \tau \implies \text{truncate by position if exceeded} \]
+\[ \sum_{n \in \mathcal{I}_{2d}} \mathrm{toks}(n) \le \alpha_2 \tau \implies \text{truncate by source position if exceeded} \]
 
 *Cumulative Verification Proof:* Let the total reserved invariant budget fraction be \(\alpha\), where \(\alpha_1 + \alpha_2 \le \alpha\). If both independent enforcement bounds are satisfied, then:
 \[ \sum_{n \in \mathcal{I}_{1d}} \mathrm{toks}(n) + \sum_{n \in \mathcal{I}_{2d}} \mathrm{toks}(n) \le \alpha_1 \tau + \alpha_2 \tau = (\alpha_1 + \alpha_2)\tau \le \alpha \tau \]
@@ -102,14 +138,20 @@ therefore treats the tiers with the following precedence:
 3. **Tier 2 / Soft invariants** are injected by longest-prefix truncation under the effective budget
    \[
    \tau_{\mathcal{I}_2}^{\mathrm{eff}}=
+   \max\!\left(0,\,
    \min\!\left(\alpha_2\tau,\,
-   \tau-\tau_{\mathcal{I}_1}-\mathrm{toks}(T_{\mathrm{base}})\right)
+   \tau-\tau_{\mathcal{I}_1}-\mathrm{toks}(T_{\mathrm{base}})\right)\right)
    \]
 4. **Variant lore** competes only for the final residual budget after Tier 1,
    the admitted Tier 2 prefix, and the exact recent tail are accounted for.
 
 This makes \(\mathcal{I}_1\) and the minimum continuity suffix hard
 constraints, while keeping \(\mathcal{I}_2\) order-preserving but elastic.
+Equivalently, the runtime safety invariant is:
+\[
+\tau_{\mathcal{I}_1} + \mathrm{toks}(T_{\mathrm{base}}) \le \tau
+\quad \text{must hold at runtime or Tier 2 is fully evicted}
+\]
 
 ## 7. The Document-Addressed Cache (\(\Psi\)) and Runtime Implications
 
@@ -121,5 +163,5 @@ Because the token estimator function \(\lceil \frac{|t|}{\chi(t)} \rceil\) depen
 
 At runtime:
 1. **Tier 1 (\(\mathcal{I}_{1d}\))** is injected via an \(O(1)\) memory copy.
-2. **Tier 2 (\(\mathcal{I}_{2d}\))** is evaluated via an \(O(|\mathcal{I}_{2d}|)\) prefix sum to enforce position truncation under \(\tau_{\mathcal{I}_2}^{\mathrm{eff}}\).
+2. **Tier 2 (\(\mathcal{I}_{2d}\))** is evaluated via an \(O(|\mathcal{I}_{2d}|)\) prefix sum to enforce source-order truncation under \(\tau_{\mathcal{I}_2}^{\mathrm{eff}}\).
 3. **Tier 0 (\(\mathcal{V}_d\))** bypasses re-parsing and feeds into the semantic Pass 1 vector retrieval only after the continuity layer removes the exact recent tail into \(T_{\mathrm{recent}}\), leaving \(\mathcal{V}_{\mathrm{rest}}\).