@xdarkicex/openclaw-memory-libravdb 1.3.13 → 1.3.17

package/README.md

@@ -4,30 +4,84 @@
 [![TypeScript](https://img.shields.io/badge/TypeScript-5.x-3178C6?logo=typescript&logoColor=white)](./package.json)
 [![OpenClaw](https://img.shields.io/badge/OpenClaw-memory%20plugin-111827)](./openclaw.plugin.json)
 
-Local-first memory for OpenClaw that pairs a TypeScript plugin with a Go
-daemon, keeps recent work intact, and promotes durable memory only when the
-signal is strong enough to matter.
+`@xdarkicex/openclaw-memory-libravdb` is a local-first OpenClaw memory system
+for people who want more than "top-k vectors plus a prompt footer."
 
-## Install and Lifecycle
+It replaces the default lightweight memory path with a full context lifecycle:
 
-- [Install guide](./docs/install.md) for Homebrew, OpenClaw / OpenClaw.ai plugin setup, and manual daemon lifecycle.
-- [Uninstall guide](./docs/uninstall.md) for clean plugin removal, daemon shutdown, and optional data cleanup.
-- [Full installation reference](./docs/installation.md) for deeper operational detail, troubleshooting, and packaging notes.
+- active session memory
+- durable per-user memory
+- shared global memory
+- continuity-aware compaction
+- authored context partitioning
+- hybrid scoring across scope, recency, and similarity
 
-Start with the [install guide](./docs/install.md) for the supported daemon
-setup paths and activation flow. The short version is:
+This repository pairs a TypeScript OpenClaw plugin with a Go daemon backed by
+`libraVDB`. The plugin owns both the `memory` and `contextEngine` slots, while
+the daemon handles embeddings, retrieval, storage, and compaction.
+On newer OpenClaw builds, it also bridges the built-in `memory_search` runtime
+to the same libraVDB sidecar instead of leaving that tool inert.
 
-- install the plugin with `openclaw plugins install @xdarkicex/openclaw-memory-libravdb`
-- install and start `libravdbd` separately
-- assign `libravdb-memory` to the OpenClaw `memory` slot
+## Why This Exists
 
-Then activate the plugin in `~/.openclaw/openclaw.json`:
+The stock "single memory bucket" pattern is good for simple persistence, but it
+starts to break down when you care about:
+
+- keeping the newest working context raw and intact
+- separating ephemeral session state from durable memory
+- avoiding long-session prompt collapse
+- preserving authored instructions differently from recalled user content
+- treating memory retrieval as a ranked assembly problem instead of plain
+  nearest-neighbor lookup
+
+LibraVDB Memory exists for that harder class of memory problem.
+
+## What Makes It Different
+
+These are the core differentiators the project is built around:
+
+- Dual slot ownership: the plugin owns both memory prompt injection and the
+  full context lifecycle.
+- Built-in `memory_search` bridge: newer OpenClaw memory runtime calls are
+  routed into the same sidecar-backed retrieval path.
+- Lifecycle hint adoption: `before_reset` and `session_end` are used as
+  advisory signals into the sidecar without giving OpenClaw control of ingest
+  or compaction.
+- Sidecar-owned lifecycle journal: reset/end hints are recorded internally for
+  debugging and auditing without entering normal memory retrieval.
+  The journal is bounded by a sidecar retention cap so it does not grow
+  forever.
+- Local-first runtime: the core path does not depend on external embedding
+  services.
+- Three-tier memory: session, durable user, and global memory stay distinct.
+- Hybrid scoring: retrieval is ranked by semantic similarity, recency, scope,
+  and summary quality instead of cosine alone.
+- Automatic compaction: long sessions compact behind a protected recent tail.
+- Crash-resilient IPC: the host talks to a sidecar over a stable local socket
+  or loopback TCP endpoint with degraded-mode fallback.
+
+## Quick Start
+
+The supported install flow is:
+
+```bash
+brew tap xDarkicex/openclaw-libravdb-memory
+brew install libravdbd
+brew services start libravdbd
+openclaw plugins install @xdarkicex/openclaw-memory-libravdb
+```
+
+The Homebrew formula installs the daemon plus the bundled ONNX Runtime, embedding assets, and T5 summarizer assets it needs to boot cleanly on supported platforms.
+
+Then assign the plugin to both required OpenClaw slots in
+`~/.openclaw/openclaw.json`:
 
 ```json
 {
   "plugins": {
     "slots": {
-      "memory": "libravdb-memory"
+      "memory": "libravdb-memory",
+      "contextEngine": "libravdb-memory"
     },
     "configs": {
       "libravdb-memory": {
@@ -38,45 +92,192 @@ Then activate the plugin in `~/.openclaw/openclaw.json`:
 }
 ```
 
-The published plugin is connect-only. It does not compile or spawn a local Go
-binary during install. The `libravdbd` daemon is managed separately and the
-plugin connects to an endpoint such as `unix:$HOME/.clawdb/run/libravdb.sock`
-or `tcp:127.0.0.1:37421`.
+Verify the setup:
 
-Use `sidecarPath: "auto"` or omit the field to use the platform default
-endpoint. If your daemon listens elsewhere, set an explicit endpoint such as
-`unix:/custom/path/libravdb.sock` or `tcp:127.0.0.1:9999`.
+```bash
+openclaw memory status
+```
 
-## How It Works
+Expected healthy state:
 
-- [Hybrid retrieval and prompt assembly](./docs/mathematics-v2.md): combines semantic similarity, recency, memory scope, and budget-aware packing so the prompt keeps the most useful memory instead of only the nearest vectors.
-- [Authored context partitioning](./docs/ast-v2.md): splits authored Markdown into hard directives, soft directives, and searchable lore so critical instructions are always preserved while narrative context still competes through retrieval.
-- [Domain-Adaptive Gating](./docs/gating.md): decides which turns deserve promotion into durable memory by blending conversational and technical signals rather than treating all chats like generic prose.
-- [Continuity preservation](./docs/continuity.md): protects a recent raw session tail and lets older history compact behind it, preventing summaries from erasing the newest working context.
+- the daemon is reachable
+- the plugin is active as the memory provider
+- the runtime can report stored counts and model readiness
 
-Three practical ideas shape the runtime:
+## Install Model
 
-- Hybrid ranking keeps session turns, durable user memory, and global memory on the same scoreboard while still respecting recency.
-- Two-pass, in-place compaction preserves continuity by refusing destructive rewrites of the newest working tail.
-- Domain-adaptive ingestion avoids over-saving noisy chatter while still retaining technical decisions, file paths, error signatures, and workflow milestones.
+This plugin is intentionally **connect-only** at install time.
 
-## Runtime Model
+It does not compile Go code during plugin installation, and it does not manage
+daemon lifecycle automatically from the npm package. That is deliberate: some
+OpenClaw environments are strict about postinstall behavior, daemon spawning,
+and anything that looks like binary bootstrap or process management.
 
-- Plugin package: `@xdarkicex/openclaw-memory-libravdb`
-- OpenClaw plugin id: `libravdb-memory`
-- Minimum host version: `openclaw >= 2026.3.22`
-- Default daemon endpoint on macOS/Linux: `unix:$HOME/.clawdb/run/libravdb.sock`
-- Default daemon endpoint on Windows: `tcp:127.0.0.1:37421`
-- Default daemon data path: `$HOME/.clawdb/data.libravdb`
+Current model:
 
-## Verify
+- npm/OpenClaw package: plugin code and docs
+- `libravdbd`: installed and managed separately
+- default daemon endpoint on macOS/Linux:
+  `unix:$HOME/.clawdb/run/libravdb.sock`
+- default daemon endpoint on Windows:
+  `tcp:127.0.0.1:37421`
 
-Run:
+If your daemon runs elsewhere, set an explicit `sidecarPath`, for example:
 
-```bash
-openclaw memory status
+- `unix:/custom/path/libravdb.sock`
+- `tcp:127.0.0.1:9999`
+
+## Architecture At A Glance
+
+```text
+OpenClaw host
+  -> memoryPromptSection (durable user/global recall)
+  -> memory runtime bridge (built-in memory_search)
+  -> context engine (bootstrap / ingest / assemble / compact)
+  -> plugin runtime
+  -> JSON-RPC
+  -> libravdbd
+  -> libraVDB + local embedding/summarization stack
 ```
 
-Expected output includes a readable status table showing whether the daemon is
-reachable, how much memory is stored, and whether the local summarization path
-is provisioned.
+The main runtime split is:
+
+- TypeScript host layer:
+  - OpenClaw plugin registration
+  - prompt assembly
+  - hybrid ranking
+  - continuity-aware token budgeting
+  - degraded-mode behavior
+- Go daemon layer:
+  - vector storage
+  - embeddings
+  - search RPCs
+  - compaction and summarization
+  - stable local IPC endpoint
+
+For the implemented architecture map, read
+[docs/architecture.md](./docs/architecture.md).
+
+## Retrieval Model
+
+The assembly path is not "just search some vectors and paste the top hits."
+
+It combines:
+
+- session search for current-work relevance
+- durable user recall for long-lived personal context
+- global recall for shared facts
+- authored invariant and variant context
+- continuity-preserving recent-tail injection
+- token-budgeted fitting
+
+The ranking model currently blends:
+
+- semantic similarity
+- scope weighting
+- recency decay
+- summary quality attenuation
+
+The formal math lives in:
+
+- [docs/mathematics-v2.md](./docs/mathematics-v2.md)
+- [docs/continuity.md](./docs/continuity.md)
+- [docs/ast-v2.md](./docs/ast-v2.md)
+- [docs/elevated-guidance.md](./docs/elevated-guidance.md)
+
+## Compaction Model
+
+This system does not treat long chats as append-only forever.
+
+Older session turns compact behind a protected recent tail, so the plugin can:
+
+- keep the newest working context raw
+- preserve adjacency-sensitive continuity near the boundary
+- promote older material into summaries
+- avoid letting long sessions drown their own prompt budget
+
+Compaction is designed as part of the memory system itself, not as a separate
+maintenance convenience.
+
+## For Power Users
+
+If you are evaluating this as an operator or advanced OpenClaw user, the key
+practical points are:
+
+- This plugin should own both `memory` and `contextEngine`. Partial slot
+  assignment is a misconfiguration.
+- On hosts that expose `registerMemoryRuntime`, the built-in `memory_search`
+  tool now searches the same libraVDB-backed memory stores.
+- The daemon is a separate operational unit. Treat plugin lifecycle and daemon
+  lifecycle as different concerns.
+- The system is local-first by design. The critical retrieval path does not
+  require a remote embedding service.
+- The sidecar transport is stable and explicit, which makes it service-manager
+  friendly on macOS, Linux, and Windows.
+
+Good entry points:
+
+- [docs/install.md](./docs/install.md)
+- [docs/installation.md](./docs/installation.md)
+- [docs/uninstall.md](./docs/uninstall.md)
+- [docs/implementation.md](./docs/implementation.md)
+
+## For Researchers And Builders
+
+If you are studying retrieval, memory systems, or agent architecture, the
+interesting parts of this repo are:
+
+- continuity-aware assembly:
+  `C_total(q) = I union T_recent union Proj(V_rest, q)`
+- hybrid ranking instead of pure cosine retrieval
+- separation of authored invariants from searchable authored lore
+- durable-memory admission via domain-adaptive gating
+- local daemon architecture rather than in-process TS vector plumbing
+- compaction that preserves recent working context instead of flattening the
+  whole transcript
+
+Start here:
+
+- [docs/problem.md](./docs/problem.md)
+- [docs/architecture.md](./docs/architecture.md)
+- [docs/mathematics-v2.md](./docs/mathematics-v2.md)
+- [docs/gating.md](./docs/gating.md)
+- [docs/continuity.md](./docs/continuity.md)
+
+## Runtime Facts
+
+- npm package: `@xdarkicex/openclaw-memory-libravdb`
+- OpenClaw plugin id: `libravdb-memory`
+- minimum host version: `openclaw >= 2026.3.22`
+- default daemon data path: `$HOME/.clawdb/data.libravdb`
+- default daemon endpoint on macOS/Linux:
+  `unix:$HOME/.clawdb/run/libravdb.sock`
+- default daemon endpoint on Windows:
+  `tcp:127.0.0.1:37421`
+
+## Repository Guide
+
+- [docs/install.md](./docs/install.md): quick install and lifecycle guide
+- [docs/installation.md](./docs/installation.md): full installation and
+  packaging reference
+- [docs/uninstall.md](./docs/uninstall.md): clean shutdown and removal
+- [docs/architecture.md](./docs/architecture.md): current implemented system
+  architecture
+- [docs/implementation.md](./docs/implementation.md): important implementation
+  contracts
+- [docs/mathematics-v2.md](./docs/mathematics-v2.md): formal scoring and
+  optimization reference
+
+## Current Constraint
+
+Because OpenClaw environments can be strict about postinstall downloads,
+daemon spawning, and scanner-visible binary bootstrap behavior, the cleanest
+supported user path today is:
+
+- install plugin
+- install daemon
+- assign both slots
+- let the plugin connect to a stable local endpoint
+
+That tradeoff is intentional. It keeps the plugin installation surface simple
+and auditable while preserving the full local memory engine at runtime.

package/docs/README.md

@@ -13,6 +13,7 @@ to preserve project history and design evolution.
 - [compaction-evaluation.md](./compaction-evaluation.md) - Real-model benchmark notes for T5 summary confidence, Nomic-space preservation, and the hard preservation gate.
 - [continuity.md](./continuity.md) - Continuity model for invariant context, preserved recent raw session tail, and retrieved older memory.
 - [ast-v2.md](./ast-v2.md) - Reviewed authoritative AST partitioning reference for authored Markdown hard invariants, soft invariants, and variant lore.
+- [elevated-guidance.md](./elevated-guidance.md) - Tier 1.5 protected-shard and elevated-guidance model for preserving shadow rules through compaction.
 - [ast.md](./ast.md) - Historical predecessor to `ast-v2.md`, kept to show design evolution and earlier bugs.
 - [gating.md](./gating.md) - Full derivation and calibration guide for the domain-adaptive gating scalar.
 - [implementation.md](./implementation.md) - Non-obvious implementation decisions and their rationale.

package/docs/ast-v2.md

@@ -33,6 +33,27 @@ We formalize this as a binary promotion scalar \(\sigma: N_d \to \{0,1\}\). This
 \end{cases}
 \]
 
+To reason about tuning noise in the bigram set \(W_{\mathrm{deontic}}\), we
+also define the paragraph classifier error rates:
+\[
+P_{\mathrm{fp}} = P(\sigma(n) = 1 \mid n \text{ is narrative lore})
+\]
+\[
+P_{\mathrm{fn}} = P(\sigma(n) = 0 \mid n \text{ is behavioral rule})
+\]
+
+For authored documents whose lore paragraphs would otherwise remain in
+\(\mathcal{V}_d\), the expected Tier-2 waste introduced by false positives is:
+\[
+\mathbb{E}[\mathrm{wasted\ toks\ in\ }\mathcal{I}_2]
+=
+P_{\mathrm{fp}} \cdot |\mathcal{V}_{d,\mathrm{paragraphs}}| \cdot \mathbb{E}[\mathrm{toks}(n)]
+\]
+
+This gives the parser a concrete quantity to minimize when adjusting
+\(W_{\mathrm{deontic}}\), while \(P_{\mathrm{fn}}\) measures the risk of leaving
+true behavioral rules behind in \(\mathcal{V}_d\).
+
 *Implemented via `NewDeonticFrame` and `EvaluateText` in the zero-allocation byte lexer.*
 
 ## 3. The Three-Tier Structural Indicator Function \(\iota\)
@@ -59,12 +80,27 @@ We define the structural indicator function \(\iota: N_d \to \{0,1,2\}\) mapping
 ## 4. Corpus Decomposition and Set Integration
 
 For any document \(d \in \mathbf{D}_{\text{agents}} \cup \mathbf{D}_{\text{souls}}\), the node set \(N_d\) is partitioned cleanly into three sets:
-- **Hard Directives:** \(\mathcal{I}_{1d} = \{ n \in N_d \mid \iota(n) = 1 \}\)
+- **Hard Directives:** \(\mathcal{I}_{1d} = \langle n \in N_d \mid \iota(n) = 1 \rangle\), ordered by \(\mathrm{position}(n)\) ascending, where \(\mathrm{position}(n)\) is the byte offset of node \(n\) in \(d_{\mathrm{raw}}\)
 - **Soft Directives:** \(\mathcal{I}_{2d} = \{ n \in N_d \mid \iota(n) = 2 \}\)
 - **Contextual Lore:** \(\mathcal{V}_d = \{ n \in N_d \mid \iota(n) = 0 \}\)
 
 *Partition Completeness:* Because \(\iota(n)\) maps every node to exactly one integer in \(\{0, 1, 2\}\), the resulting sets are mutually exclusive and collectively exhaustive:
-\[ \mathcal{I}_{1d} \cup \mathcal{I}_{2d} \cup \mathcal{V}_d = N_d \quad \text{and} \quad \mathcal{I}_{1d} \cap \mathcal{I}_{2d} \cap \mathcal{V}_d = \emptyset \]
+\[
+\mathcal{I}_{1d} \cup \mathcal{I}_{2d} \cup \mathcal{V}_d = N_d
+\]
+\[
+\mathcal{I}_{1d} \cap \mathcal{I}_{2d} = \emptyset
+\]
+\[
+\mathcal{I}_{1d} \cap \mathcal{V}_d = \emptyset
+\]
+\[
+\mathcal{I}_{2d} \cap \mathcal{V}_d = \emptyset
+\]
+
+These pairwise disjointness statements follow directly from \(\iota\) being a
+single-valued total function into \(\{0,1,2\}\): no node can be assigned to
+more than one tier simultaneously.
 
 These sets integrate into the global corpus. Let \(\mathbf{D}_{\text{standard}}\) be the set of standard memory documents (non-core files). We formally define the standard variant node set as \(\mathcal{V}_{\text{standard}} = \bigcup_{d \in \mathbf{D}_{\text{standard}}} E(d)\). The global corpus is then:
 \[ \mathcal{I}_1 = \bigcup_{d} \mathcal{I}_{1d} \qquad \mathcal{I}_2 = \bigcup_{d} \mathcal{I}_{2d} \qquad \mathcal{V} = \mathcal{V}_{\text{standard}} \cup \left( \bigcup_{d} \mathcal{V}_d \right) \]
@@ -86,7 +122,7 @@ For Hard Invariants (\(\alpha_1\)):
 \[ \sum_{n \in \mathcal{I}_{1d}} \mathrm{toks}(n) \le \alpha_1 \tau \implies \text{fast-fail and reject agent load if exceeded} \]
 
 For Soft Invariants (\(\alpha_2\)):
-\[ \sum_{n \in \mathcal{I}_{2d}} \mathrm{toks}(n) \le \alpha_2 \tau \implies \text{truncate by position if exceeded} \]
+\[ \sum_{n \in \mathcal{I}_{2d}} \mathrm{toks}(n) \le \alpha_2 \tau \implies \text{truncate by source position if exceeded} \]
 
 *Cumulative Verification Proof:* Let the total reserved invariant budget fraction be \(\alpha\), where \(\alpha_1 + \alpha_2 \le \alpha\). If both independent enforcement bounds are satisfied, then:
 \[ \sum_{n \in \mathcal{I}_{1d}} \mathrm{toks}(n) + \sum_{n \in \mathcal{I}_{2d}} \mathrm{toks}(n) \le \alpha_1 \tau + \alpha_2 \tau = (\alpha_1 + \alpha_2)\tau \le \alpha \tau \]
@@ -102,14 +138,20 @@ therefore treats the tiers with the following precedence:
 3. **Tier 2 / Soft invariants** are injected by longest-prefix truncation under the effective budget
    \[
    \tau_{\mathcal{I}_2}^{\mathrm{eff}}=
+   \max\!\left(0,\,
    \min\!\left(\alpha_2\tau,\,
-   \tau-\tau_{\mathcal{I}_1}-\mathrm{toks}(T_{\mathrm{base}})\right)
+   \tau-\tau_{\mathcal{I}_1}-\mathrm{toks}(T_{\mathrm{base}})\right)\right)
    \]
 4. **Variant lore** competes only for the final residual budget after Tier 1,
    the admitted Tier 2 prefix, and the exact recent tail are accounted for.
 
 This makes \(\mathcal{I}_1\) and the minimum continuity suffix hard
 constraints, while keeping \(\mathcal{I}_2\) order-preserving but elastic.
+Equivalently, the runtime safety invariant is:
+\[
+\tau_{\mathcal{I}_1} + \mathrm{toks}(T_{\mathrm{base}}) \le \tau
+\quad \text{must hold at runtime or Tier 2 is fully evicted}
+\]
 
 ## 7. The Document-Addressed Cache (\(\Psi\)) and Runtime Implications
 
@@ -121,5 +163,5 @@ Because the token estimator function \(\lceil \frac{|t|}{\chi(t)} \rceil\) depen
 
 At runtime:
 1. **Tier 1 (\(\mathcal{I}_{1d}\))** is injected via an \(O(1)\) memory copy.
-2. **Tier 2 (\(\mathcal{I}_{2d}\))** is evaluated via an \(O(|\mathcal{I}_{2d}|)\) prefix sum to enforce position truncation under \(\tau_{\mathcal{I}_2}^{\mathrm{eff}}\).
+2. **Tier 2 (\(\mathcal{I}_{2d}\))** is evaluated via an \(O(|\mathcal{I}_{2d}|)\) prefix sum to enforce source-order truncation under \(\tau_{\mathcal{I}_2}^{\mathrm{eff}}\).
 3. **Tier 0 (\(\mathcal{V}_d\))** bypasses re-parsing and feeds into the semantic Pass 1 vector retrieval only after the continuity layer removes the exact recent tail into \(T_{\mathrm{recent}}\), leaving \(\mathcal{V}_{\mathrm{rest}}\).