@xdarkicex/openclaw-memory-libravdb 1.3.12 → 1.3.17

package/docs/continuity.md

@@ -10,6 +10,14 @@ $$
 \text{continuity} \neq \text{semantic summary alone}
 $$
 
+This document also defines a proposed lossless extension to the current model.
+That extension is inspired by the immutable-store and expandable-summary
+architecture in the LCM paper, "Lossless Context Management"
+([Ehrlich and Blackman, 2026](https://papers.voltropy.com/LCM)). Where this
+document adopts that idea directly, it cites the paper explicitly. The
+mathematical notation below is adapted to this repository's existing
+invariant/tail/retrieval decomposition rather than copied from the paper.
+
 Instead, continuity is modeled as the composition of:
 
 $$
@@ -157,6 +165,10 @@ means the runtime may extend $T_{\mathrm{recent}}$ slightly backward to keep a
 recent cause/effect pair, request/response pair, or equivalent tightly coupled
 artifact bundle intact.
 
+**Policy note.** Bundle coupling is a heuristic policy layer, not a formal
+theorem term. It is listed in Section 13.4 as a heuristic and is not part of
+the core $C_{\mathrm{total}}(q)$ assembly theorem.
+
 ## 4. Budget Partition
 
 Let the total prompt budget be $\tau$. Then the continuity-aware allocation is:
@@ -267,6 +279,7 @@ The boundary must also be bundle-safe. If a cluster candidate would split a
 tightly coupled local unit across the tail boundary, the runtime should move the
 boundary backward so that the unit stays entirely in $T_{\mathrm{recent}}$ or
 entirely in $\mathcal{V}_{\mathrm{rest}}$.
+*(This is a heuristic policy; see Section 13.4.)*
 
 ## 6. Compaction Progress Guarantee
 
@@ -309,6 +322,12 @@ $$
 
 whenever a cluster is actually replaced.
 
+**Edge case — singleton clusters.** If a cluster contains only a single turn
+($|C_j| = 1$), the clustering algorithm produces a `trivial`-tagged summary that
+does not represent meaningful compaction progress. The $\Delta_{\mathrm{compact}} > 0$
+guarantee applies only to clusters with $|C_j| \ge 2$ that are meaningfully replaced;
+trivial singletons are boundary cases excluded from the progress invariant.
+
 ## 7. Summary Lineage And Recoverability
 
 Continuity improves when summary nodes are not opaque replacements but
@@ -347,6 +366,96 @@ $$
 This does not replace retrieval scoring. It guarantees that compressed history
 remains inspectable and attributable.
 
+## 7.5 Lossless Recoverability Extension
+
+The current implementation stores lineage metadata for summaries, but it does
+not yet preserve a fully immutable raw session store after compaction. A
+stronger continuity contract is to treat compaction summaries as derived views
+over immutable raw history rather than destructive replacements. This is the
+main architectural idea adopted from the LCM paper's immutable store, summary
+DAG, and bounded expansion model
+([Ehrlich and Blackman, 2026](https://papers.voltropy.com/LCM)).
+
+Let the raw session history be:
+
+$$
+\mathcal{R}_{\mathrm{session}}=\langle r_1,r_2,\dots,r_n\rangle
+$$
+
+where each $r_i$ is a raw persisted turn and raw-history persistence is
+append-only:
+
+$$
+\mathrm{Compact}(\mathcal{R}_{\mathrm{session}})=\mathcal{R}_{\mathrm{session}}
+$$
+
+Compaction instead constructs a summary-node set:
+
+$$
+\mathbf{S}=\{s_1,s_2,\dots\}
+$$
+
+and a parent relation:
+
+$$
+E_{\triangleleft}\subseteq (\mathbf{S}\times\mathbf{S})\cup(\mathbf{S}\times\mathcal{R}_{\mathrm{session}})
+$$
+
+where an edge $(s,x)\in E_{\triangleleft}$ means summary node $s$ directly
+covers child node $x$, with $x$ either a raw turn or a lower-order summary.
+
+The resulting continuity graph is:
+
+$$
+\mathcal{G}_{\mathrm{cont}}=(\mathbf{S}\cup\mathcal{R}_{\mathrm{session}}, E_{\triangleleft})
+$$
+
+with the intended acyclicity invariant:
+
+$$
+\mathcal{G}_{\mathrm{cont}} \text{ is a DAG}
+$$
+
+Define recursive expansion to leaf raw turns:
+
+$$
+\mathrm{Expand}^{*}(x)=
+\begin{cases}
+\{x\} & \text{if } x\in\mathcal{R}_{\mathrm{session}} \\
+\bigcup_{y:(x,y)\in E_{\triangleleft}} \mathrm{Expand}^{*}(y) & \text{if } x\in\mathbf{S}
+\end{cases}
+$$
+
+Then lossless recoverability means:
+
+$$
+\forall s\in\mathbf{S},\ \mathrm{Expand}^{*}(s)\neq\emptyset
+$$
+
+and:
+
+$$
+\forall r\in\mathcal{R}_{\mathrm{session}},\ \exists x\in \mathbf{S}\cup T_{\mathrm{recent}} \text{ such that } r\in \mathrm{Expand}^{*}(x)
+$$
+
+Operationally, this means compaction may change which nodes are injected or
+searched first, but it must not erase the ability to navigate back to the raw
+turns covered by a summary.
+
+The current repository should treat this as a proposed extension, not as a
+claim about present behavior. Today the compactor inserts summaries with
+structured lineage metadata, then deletes the covered source turns from the
+session collection after successful replacement. A future lossless
+implementation should separate:
+
+- immutable raw turn storage
+- active/searchable summary views
+- bounded expansion and search over compacted history
+
+The corresponding data-model change is to add a raw immutable session layer and
+store summary coverage edges explicitly instead of using lineage metadata alone
+as the recoverability surface.
+
 ## 8. Continuity-Aware Summarization Input
 
 Compaction input should be continuity-safe before it reaches the summarizer.
@@ -469,6 +578,19 @@ $$
 No continuity-critical local bundle may be split across the recent-tail and
 compaction boundary.
 
+9. Lossless recoverability when the extension is enabled:
+
+$$
+\forall s\in\mathbf{S},\ \mathrm{Expand}^{*}(s)\subseteq\mathcal{R}_{\mathrm{session}}
+\qquad\text{and}\qquad
+\mathrm{Expand}^{*}(s)\neq\emptyset
+$$
+
+10. Raw-history immutability when the extension is enabled:
+
+Compaction may add summary nodes and coverage edges, but it must not delete
+raw turns from $\mathcal{R}_{\mathrm{session}}$.
+
 ## 12. Practical Interpretation
 
 In practical terms, continuity for this system is:
@@ -486,3 +608,101 @@ This avoids the failure mode where continuity depends entirely on a semantic
 summary being perfect. It also means compaction is not merely a storage
 optimization. It is a constrained transformation that must preserve exact
 recent state, recoverable lineage, and monotone progress.
+
+## 13. Layer Separation And Review Guidance
+
+The strongest follow-on review result for this document is that the continuity
+theory is healthiest when it keeps three layers separate:
+
+1. storage axioms
+2. core retrieval and assembly math
+3. recoverability policy
+
+The authoritative continuity contract in this document should therefore be read
+as follows.
+
+### 13.1 Storage Axioms
+
+When the lossless extension is enabled, raw-history immutability is a storage
+axiom:
+
+$$
+\mathrm{Compact}(\mathcal{R}_{\mathrm{session}})=\mathcal{R}_{\mathrm{session}}
+$$
+
+That statement is unconditional. It does not depend on query relevance,
+summary confidence, or token budget. It is stronger than lineage metadata or
+query-time expansion. It simply means compaction does not delete raw source
+turns from the immutable raw layer.
+
+### 13.2 Recoverability Theorem
+
+The summary-coverage DAG and $\mathrm{Expand}^{*}$ belong to recoverability,
+not to the primary retrieval theorem. Their job is to guarantee that compacted
+history remains navigable back to raw source turns:
+
+$$
+\forall s\in\mathbf{S},\ \mathrm{Expand}^{*}(s)\subseteq\mathcal{R}_{\mathrm{session}}
+\qquad\text{and}\qquad
+\mathrm{Expand}^{*}(s)\neq\emptyset
+$$
+
+This is a structural property of the continuity graph. It is not by itself a
+claim that every query should traverse that graph during normal assembly.
+
+### 13.3 Retrieval Boundary
+
+The core continuity theorem remains:
+
+$$
+C_{\mathrm{total}}(q)=\mathcal{I}_1\cup \mathcal{I}_2^{*}\cup T_{\mathrm{recent}}\cup \mathrm{Proj}(\mathcal{V}_{\mathrm{rest}}, q)
+$$
+
+This document treats that expression as the primary assembly law. A runtime may
+experiment with query-time summary expansion, but such expansion should be
+treated as a bounded policy layer wrapped around the core theorem unless it is
+formally re-derived inside the governing retrieval math.
+
+In particular, policy knobs such as:
+
+- summary-expansion confidence thresholds
+- expansion token budgets
+- depth limits
+- expansion penalties or attenuations
+
+are not themselves continuity axioms. They are deployment and retrieval-policy
+choices layered on top of the structural guarantees above.
+
+### 13.4 Heuristic vs. Theorem Boundary
+
+The following ideas remain useful, but should be read as heuristics unless
+their mathematics is defined explicitly elsewhere:
+
+- **bundle-safe boundary extension** (Section 3): the runtime may extend
+  $T_{\mathrm{recent}}$ backward to avoid splitting a coupled local bundle;
+  this is a heuristic policy, not a formal tail selector term
+- specific escalation ladders for compaction fallback
+- **confidence-triggered automatic expansion**: query-time summary expansion is
+  explicit recovery/audit only; it was removed from the hot retrieval path and
+  is not the default behavior — see Section 13.3 and memory 283
+- any fixed expansion penalty not derived from the governing score equations
+
+This distinction matters because continuity should stay theorem-safe even when
+those policies are tuned, replaced, or disabled.
+
+### 13.5 Future Theory Direction
+
+Several mathematically interesting review suggestions are worth preserving for
+future refinement, but they are not part of the current authoritative theorem:
+
+- information-theoretic or rate-distortion views of compaction quality
+- hot-spot preservation tiers based on access concentration
+- causal-centrality-aware compaction vetoes
+- entropy-driven tail selection instead of fixed turn-count rules
+- explicit recovery-state machines triggered by retrieval failure (the vNext
+  retrieval-failure signals S1/S2/S3 are defined separately in the vNext spec
+  slice; they are not part of the current $C_{\mathrm{total}}$ theorem)
+
+These are promising research directions for later versions. The current
+document keeps the simpler invariant-first continuity model as the normative
+contract until one of those stronger formulations is deliberately adopted.

package/docs/contributing.md

@@ -36,6 +36,7 @@ bash scripts/build-daemon.sh
 ```
 
 This creates `.daemon-bin/libravdbd` and copies locally available bundled assets into `.daemon-bin/`.
+That includes the embedding models, ONNX Runtime, and the bundled T5 summarizer assets when they are present under `.models/`.
 
 ## Gating Invariants
 

package/docs/elevated-guidance.md

@@ -0,0 +1,258 @@
+# Elevated Guidance Model
+
+This document defines the Tier 1.5 elevated-guidance path that sits between
+authored invariants and ordinary recalled memory. Its purpose is to preserve
+high-value "shadow rules" that are too weakly structured for AST promotion but
+too directive to be allowed to decay into lossy summaries or low-trust recalled
+memory.
+
+The design goal is:
+
+$$
+\text{preserve high-value guidance without promoting it to Tier 0 invariants}
+$$
+
+The elevated-guidance path is therefore:
+
+- stronger than ordinary semantic recall
+- weaker than authored hard or soft invariants
+- assembled separately from `<recalled_memories>`
+- bounded by its own token reservation so it cannot starve continuity or Tier 0
+
+## 1. Protected Summarization
+
+During compaction, let a chronological cluster be:
+
+$$
+C_j = \{ t_1, t_2, \dots, t_m \}
+$$
+
+Define a deterministic deontic indicator:
+
+$$
+\delta(t_i) \in \{0,1\}
+$$
+
+where $\delta(t_i)=1$ means the turn contains guidance-like imperative or
+prohibitive surface forms detectable by the local deontic frame.
+
+Let $a_{t_i}\in[0,1]$ be the authored stability weight for a turn. Stable
+authored sources may set $a_{t_i}=1$, while ordinary session text defaults
+lower. The ideal shard-protection predicate is:
+
+$$
+P_{\mathrm{shard}}(t_i)=
+\begin{cases}
+1 & \text{if } \delta(t_i)=1 \land a_{t_i}\ge\tau_{\mathrm{stable}} \\
+0 & \text{otherwise}
+\end{cases}
+$$
+
+For the current first implementation, the runtime uses a conservative
+deterministic approximation that protects deontic-like turns directly and gates
+them by a stored stability weight rather than depending on a local model to
+decide whether preservation should happen.
+
+The cluster is partitioned into protected shards and compressible turns:
+
+$$
+C_j^{\mathrm{protected}}=\{t_i\in C_j \mid P_{\mathrm{shard}}(t_i)=1\}
+$$
+
+$$
+C_j^{\mathrm{compress}}=C_j \setminus C_j^{\mathrm{protected}}
+$$
+
+Compaction then becomes:
+
+$$
+\mathrm{Compaction}(C_j)=
+\left\{s_{\mathrm{abstractive}}(C_j^{\mathrm{compress}})\right\}
+\cup C_j^{\mathrm{protected}}
+$$
+
+where the protected shard members survive verbatim as elevated-guidance records
+instead of being melted into the cluster summary.
+
+In the current implementation, protected records are persisted outside the live
+session collection into durable elevated-guidance namespaces such as:
+
+- `elevated:user:<userId>` when user provenance is available
+- `elevated:session:<sessionId>` as a fallback
+
+## 2. Tier 1.5 Admission Gate
+
+At retrieval time, let $s$ range over the protected-shard records produced by
+compaction. Elevated guidance is admitted only when both conditions hold:
+
+1. the record was structurally protected during compaction
+2. the current query is semantically relevant to it
+
+Formally:
+
+$$
+G_{\mathrm{elevated}}(q,s)=
+\begin{cases}
+1 & \text{if } \mathrm{sim}(q,s)>\theta_1 \land s\in\bigcup_j C_j^{\mathrm{protected}} \\
+0 & \text{otherwise}
+\end{cases}
+$$
+
+The elevated buffer for query $q$ is:
+
+$$
+E(q)=\{s \mid G_{\mathrm{elevated}}(q,s)=1\}
+$$
+
+This set is assembled separately from `<recalled_memories>` so it can outrank
+ordinary semantic recall without claiming the full normative force of authored
+context.
+
+## 3. Assembly Order and Budget
+
+Let $\tau$ be the total memory prompt budget. The continuity-aware assembly with
+Tier 1.5 becomes:
+
+$$
+C_{\mathrm{total}}(q)=
+\mathcal{I}_1
+\cup T_{\mathrm{recent}}
+\cup \mathcal{I}_2^{*}
+\cup E^{*}(q)
+\cup \mathrm{Proj}(\mathcal{V}_{\mathrm{rest}}, q)
+$$
+
+where:
+
+- $\mathcal{I}_1$ is hard authored context
+- $T_{\mathrm{recent}}$ is the exact preserved raw recent tail
+- $\mathcal{I}_2^{*}$ is the admitted soft-invariant prefix
+- $E^{*}(q)$ is the budget-truncated elevated-guidance set
+- $\mathrm{Proj}(\mathcal{V}_{\mathrm{rest}}, q)$ is ordinary residual semantic recall
+
+Let $\rho_E\in(0,1)$ reserve a fraction of the prompt for elevated guidance.
+The effective elevated-guidance token mass is:
+
+$$
+\tau_E^{\mathrm{eff}}=
+\min\!\left(
+\sum_{s\in E(q)}\mathrm{toks}(s),\,
+\rho_E\tau
+\right)
+$$
+
+The residual variant budget becomes:
+
+$$
+\tau_{\mathcal{V}}=
+\tau
+-\tau_{\mathcal{I}_1}
+-\mathrm{toks}(T_{\mathrm{recent}})
+-\tau_{\mathcal{I}_2}^{*}
+-\tau_E^{\mathrm{eff}}
+$$
+
+If $\tau_{\mathcal{V}}\le 0$, ordinary semantic recall is intentionally starved
+before elevated guidance is displaced.
+
+## 4. Trust Boundary
+
+Tier 1.5 is not a replacement for authored invariants. It is an elevated
+advisory enclave:
+
+- authored context still wins on conflict
+- elevated guidance outranks ordinary semantic recall
+- ordinary recalled memory remains untrusted historical context
+
+The intended prompt precedence is:
+
+1. authored context
+2. recent raw tail
+3. elevated guidance
+4. recalled memories
+
+This preserves the Section 11 safety rule that recalled memory must not be
+followed as instructions while still giving preserved shadow rules more weight
+than generic historical recall.
+
+## 5. Failure Policy
+
+Protected summarization is deterministic-first and model-optional.
+
+If a local abstractive model is unavailable, slow, or times out, the system
+must not fail open to deleting potential shadow rules. The safety rule is:
+
+$$
+\text{model failure} \Rightarrow \text{keep deterministic protected shards}
+$$
+
+In practical terms:
+
+- destructive compaction may proceed only after protected shards are persisted
+- model timeouts may reduce summary quality, but they must not erase the shard set
+- when in doubt, preserve guidance verbatim rather than compressing it away
+
+## 6. Current Runtime Approximation
+
+The fully general model allows provenance weighting $a_{t_i}$ to distinguish
+stable authored sources from ordinary session text. The current implementation
+approximates this with explicit ingest-time metadata:
+
+- session turns receive a `provenance_class`
+- session turns receive a `stability_weight`
+- compaction protects only turns with deontic surface signals and
+  `stability_weight \ge \tau_{\mathrm{stable}}`
+
+This is enough to make Tier 1.5 durable and provenance-weighted without yet
+requiring a local model in the admission path.
+
+## 7. Additive Local-Model Booster
+
+The final admission stage may use a local model only as an additive booster.
+The current implementation reuses the canonical local embedder exposed by the
+extractive summarizer.
+
+Let $b_{\mathrm{sem}}(t)\in[0,1]$ be the maximum cosine similarity between turn
+$t$ and a small fixed set of guidance prototypes:
+
+$$
+b_{\mathrm{sem}}(t)=\max_{p\in\mathcal{P}_{\mathrm{guide}}}\cos(\varphi(t),\varphi(p))
+$$
+
+This signal is only considered for turns that already satisfy:
+
+- sufficient stability weight
+- a lightweight guidance surface hint
+- failure to pass the strict deterministic deontic gate
+
+The current rescue condition is therefore:
+
+$$
+P_{\mathrm{boost}}(t)=
+\mathbf{1}\!\left[
+a_t\ge\tau_{\mathrm{stable}}
+\land H_{\mathrm{surface}}(t)=1
+\land \delta(t)=0
+\land b_{\mathrm{sem}}(t)\ge\tau_{\mathrm{boost}}
+\right]
+$$
+
+and final protection becomes:
+
+$$
+P_{\mathrm{final}}(t)=
+\mathbf{1}\!\left[
+P_{\mathrm{shard}}(t)=1
+\;\lor\;
+P_{\mathrm{boost}}(t)=1
+\right]
+$$
+
+This preserves the key safety invariant:
+
+$$
+\text{model assistance may raise borderline candidates, but it is never the sole deletion-safety gate}
+$$
+
+If embedding fails or times out, the booster contributes zero and the
+deterministic path remains authoritative.

package/docs/implementation.md

@@ -5,13 +5,14 @@ reading the code piecemeal.
 
 ## Memory Kind Plus Explicit Context Engine Registration
 
-The plugin declares `kind: "memory"` in
+The plugin declares `kind: ["memory", "context-engine"]` in
 [`openclaw.plugin.json`](../openclaw.plugin.json), but still registers both a
 context engine and a memory prompt section in [`src/index.ts`](../src/index.ts).
 
 Why:
 
-- the exclusive slot takeover happens through the `memory` kind
+- the intended runtime contract is that `libravdb-memory` owns both the
+  `memory` and `contextEngine` slots together
 - the runtime behavior still needs explicit lifecycle hooks for:
   - `bootstrap`
   - `ingest`
@@ -23,6 +24,63 @@ Why:
 This is why the code registers both `registerContextEngine("libravdb-memory", …)`
 and `registerMemoryPromptSection(...)` instead of relying on only one hook.
 
+On newer OpenClaw hosts, [`src/index.ts`](../src/index.ts) also registers
+`registerMemoryRuntime(...)` as an additive bridge for the built-in
+`memory_search` tool. That bridge reuses the same sidecar-backed retrieval path
+instead of introducing a second memory backend.
+
+## Why `registerMemoryRuntime` Is Additive
+
+Implemented in [`src/memory-runtime.ts`](../src/memory-runtime.ts).
+
+The newer OpenClaw memory runtime seam is useful, but it does not replace the
+spec-driven architecture in this repository.
+
+What the runtime bridge does:
+
+- exposes a search manager for the built-in `memory_search` tool
+- routes search into the same libraVDB collections already used by the plugin
+- reports sidecar status through the existing JSON-RPC `status` method
+
+What it intentionally does not do yet:
+
+- it does not replace context-engine ingest
+- it does not replace context-engine compaction
+- it does not register a host flush plan that could duplicate transcript ingest
+
+That split is deliberate. The plugin already owns ingest and compaction through
+the context engine and sidecar, so `registerMemoryRuntime` is safe as a search
+bridge while `registerMemoryFlushPlan` remains deferred until it can be mapped
+cleanly onto the existing lifecycle.
+
+## Why `before_reset` and `session_end` Stay Advisory
+
+Implemented in [`src/lifecycle-hooks.ts`](../src/lifecycle-hooks.ts) and
+[`src/plugin-runtime.ts`](../src/plugin-runtime.ts).
+
+Newer OpenClaw hosts expose `before_reset` and `session_end` plugin hooks.
+This plugin uses them, but only as hints into the sidecar.
+
+Current behavior:
+
+- `before_reset` forwards session identifiers, reset reason, and observed
+  message count
+- `session_end` forwards end reason, archive linkage, and follow-on session
+  metadata
+- the sidecar appends the hint to an internal lifecycle journal and performs a
+  best-effort flush/ack
+
+Important boundary:
+
+- these hooks are not the source of truth for memory correctness
+- failure to deliver them must not break the session
+- ingest, retrieval, and compaction still belong to the context engine and
+  sidecar runtime we control
+- lifecycle journal entries live in an internal collection and are only visible
+  through explicit status/debug surfaces such as `openclaw memory journal`
+- lifecycle retention is bounded and enforced on append by pruning the oldest
+  journal entries first
+
 ## Why Ingest Is Fire-and-Forget
 
 Implemented in [`src/context-engine.ts`](../src/context-engine.ts).