@xdarkicex/openclaw-memory-libravdb 1.4.3 → 1.4.4

package/docs/elevated-guidance.md

@@ -1,258 +0,0 @@
-# Elevated Guidance Model
-
-This document defines the Tier 1.5 elevated-guidance path that sits between
-authored invariants and ordinary recalled memory. Its purpose is to preserve
-high-value "shadow rules" that are too weakly structured for AST promotion but
-too directive to be allowed to decay into lossy summaries or low-trust recalled
-memory.
-
-The design goal is:
-
-$$
-\text{preserve high-value guidance without promoting it to Tier 0 invariants}
-$$
-
-The elevated-guidance path is therefore:
-
-- stronger than ordinary semantic recall
-- weaker than authored hard or soft invariants
-- assembled separately from `<recalled_memories>`
-- bounded by its own token reservation so it cannot starve continuity or Tier 0
-
-## 1. Protected Summarization
-
-During compaction, let a chronological cluster be:
-
-$$
-C_j = \{ t_1, t_2, \dots, t_m \}
-$$
-
-Define a deterministic deontic indicator:
-
-$$
-\delta(t_i) \in \{0,1\}
-$$
-
-where $\delta(t_i)=1$ means the turn contains guidance-like imperative or
-prohibitive surface forms detectable by the local deontic frame.
-
-Let $a_{t_i}\in[0,1]$ be the authored stability weight for a turn. Stable
-authored sources may set $a_{t_i}=1$, while ordinary session text defaults
-lower. The ideal shard-protection predicate is:
-
-$$
-P_{\mathrm{shard}}(t_i)=
-\begin{cases}
-1 & \text{if } \delta(t_i)=1 \land a_{t_i}\ge\tau_{\mathrm{stable}} \\
-0 & \text{otherwise}
-\end{cases}
-$$
-
-For the current first implementation, the runtime uses a conservative
-deterministic approximation that protects deontic-like turns directly and gates
-them by a stored stability weight rather than depending on a local model to
-decide whether preservation should happen.
-
-The cluster is partitioned into protected shards and compressible turns:
-
-$$
-C_j^{\mathrm{protected}}=\{t_i\in C_j \mid P_{\mathrm{shard}}(t_i)=1\}
-$$
-
-$$
-C_j^{\mathrm{compress}}=C_j \setminus C_j^{\mathrm{protected}}
-$$
-
-Compaction then becomes:
-
-$$
-\mathrm{Compaction}(C_j)=
-\left\{s_{\mathrm{abstractive}}(C_j^{\mathrm{compress}})\right\}
-\cup C_j^{\mathrm{protected}}
-$$
-
-where the protected shard members survive verbatim as elevated-guidance records
-instead of being melted into the cluster summary.
-
-In the current implementation, protected records are persisted outside the live
-session collection into durable elevated-guidance namespaces such as:
-
-- `elevated:user:<userId>` when user provenance is available
-- `elevated:session:<sessionId>` as a fallback
-
-## 2. Tier 1.5 Admission Gate
-
-At retrieval time, let $s$ range over the protected-shard records produced by
-compaction. Elevated guidance is admitted only when both conditions hold:
-
-1. the record was structurally protected during compaction
-2. the current query is semantically relevant to it
-
-Formally:
-
-$$
-G_{\mathrm{elevated}}(q,s)=
-\begin{cases}
-1 & \text{if } \mathrm{sim}(q,s)>\theta_1 \land s\in\bigcup_j C_j^{\mathrm{protected}} \\
-0 & \text{otherwise}
-\end{cases}
-$$
-
-The elevated buffer for query $q$ is:
-
-$$
-E(q)=\{s \mid G_{\mathrm{elevated}}(q,s)=1\}
-$$
-
-This set is assembled separately from `<recalled_memories>` so it can outrank
-ordinary semantic recall without claiming the full normative force of authored
-context.
-
-## 3. Assembly Order and Budget
-
-Let $\tau$ be the total memory prompt budget. The continuity-aware assembly with
-Tier 1.5 becomes:
-
-$$
-C_{\mathrm{total}}(q)=
-\mathcal{I}_1
-\cup T_{\mathrm{recent}}
-\cup \mathcal{I}_2^{*}
-\cup E^{*}(q)
-\cup \mathrm{Proj}(\mathcal{V}_{\mathrm{rest}}, q)
-$$
-
-where:
-
-- $\mathcal{I}_1$ is hard authored context
-- $T_{\mathrm{recent}}$ is the exact preserved raw recent tail
-- $\mathcal{I}_2^{*}$ is the admitted soft-invariant prefix
-- $E^{*}(q)$ is the budget-truncated elevated-guidance set
-- $\mathrm{Proj}(\mathcal{V}_{\mathrm{rest}}, q)$ is ordinary residual semantic recall
-
-Let $\rho_E\in(0,1)$ reserve a fraction of the prompt for elevated guidance.
-The effective elevated-guidance token mass is:
-
-$$
-\tau_E^{\mathrm{eff}}=
-\min\!\left(
-\sum_{s\in E(q)}\mathrm{toks}(s),\,
-\rho_E\tau
-\right)
-$$
-
-The residual variant budget becomes:
-
-$$
-\tau_{\mathcal{V}}=
-\tau
--\tau_{\mathcal{I}_1}
--\mathrm{toks}(T_{\mathrm{recent}})
--\tau_{\mathcal{I}_2}^{*}
--\tau_E^{\mathrm{eff}}
-$$
-
-If $\tau_{\mathcal{V}}\le 0$, ordinary semantic recall is intentionally starved
-before elevated guidance is displaced.
-
-## 4. Trust Boundary
-
-Tier 1.5 is not a replacement for authored invariants. It is an elevated
-advisory enclave:
-
-- authored context still wins on conflict
-- elevated guidance outranks ordinary semantic recall
-- ordinary recalled memory remains untrusted historical context
-
-The intended prompt precedence is:
-
-1. authored context
-2. recent raw tail
-3. elevated guidance
-4. recalled memories
-
-This preserves the Section 11 safety rule that recalled memory must not be
-followed as instructions while still giving preserved shadow rules more weight
-than generic historical recall.
-
-## 5. Failure Policy
-
-Protected summarization is deterministic-first and model-optional.
-
-If a local abstractive model is unavailable, slow, or times out, the system
-must not fail open to deleting potential shadow rules. The safety rule is:
-
-$$
-\text{model failure} \Rightarrow \text{keep deterministic protected shards}
-$$
-
-In practical terms:
-
-- destructive compaction may proceed only after protected shards are persisted
-- model timeouts may reduce summary quality, but they must not erase the shard set
-- when in doubt, preserve guidance verbatim rather than compressing it away
-
-## 6. Current Runtime Approximation
-
-The fully general model allows provenance weighting $a_{t_i}$ to distinguish
-stable authored sources from ordinary session text. The current implementation
-approximates this with explicit ingest-time metadata:
-
-- session turns receive a `provenance_class`
-- session turns receive a `stability_weight`
-- compaction protects only turns with deontic surface signals and
-  `stability_weight \ge \tau_{\mathrm{stable}}`
-
-This is enough to make Tier 1.5 durable and provenance-weighted without yet
-requiring a local model in the admission path.
-
-## 7. Additive Local-Model Booster
-
-The final admission stage may use a local model only as an additive booster.
-The current implementation reuses the canonical local embedder exposed by the
-extractive summarizer.
-
-Let $b_{\mathrm{sem}}(t)\in[0,1]$ be the maximum cosine similarity between turn
-$t$ and a small fixed set of guidance prototypes:
-
-$$
-b_{\mathrm{sem}}(t)=\max_{p\in\mathcal{P}_{\mathrm{guide}}}\cos(\varphi(t),\varphi(p))
-$$
-
-This signal is only considered for turns that already satisfy:
-
-- sufficient stability weight
-- a lightweight guidance surface hint
-- failure to pass the strict deterministic deontic gate
-
-The current rescue condition is therefore:
-
-$$
-P_{\mathrm{boost}}(t)=
-\mathbf{1}\!\left[
-a_t\ge\tau_{\mathrm{stable}}
-\land H_{\mathrm{surface}}(t)=1
-\land \delta(t)=0
-\land b_{\mathrm{sem}}(t)\ge\tau_{\mathrm{boost}}
-\right]
-$$
-
-and final protection becomes:
-
-$$
-P_{\mathrm{final}}(t)=
-\mathbf{1}\!\left[
-P_{\mathrm{shard}}(t)=1
-\;\lor\;
-P_{\mathrm{boost}}(t)=1
-\right]
-$$
-
-This preserves the key safety invariant:
-
-$$
-\text{model assistance may raise borderline candidates, but it is never the sole deletion-safety gate}
-$$
-
-If embedding fails or times out, the booster contributes zero and the
-deterministic path remains authoritative.

package/docs/gating.md

@@ -1,134 +0,0 @@
-# Domain-Adaptive Gating Scalar
-
-This document describes the ingestion gate used to decide whether a user turn should be promoted into durable `user:` memory. It is the most novel scoring component in the repository.
-
-Implemented in:
-- `sidecar/compact/gate.go`
-- `sidecar/compact/tokens.go`
-- `sidecar/compact/summarize.go` for the downstream abstractive-routing threshold
-
-## 1. Why the Original Scalar Failed
-
-The original scalar assumed conversational memory semantics:
-- low novelty meant "already known"
-- repetition meant "probably redundant"
-- low natural-language structure meant "probably noise"
-
-That logic breaks for technical sessions. Repeated workflow context is often exactly what should be remembered: file paths, APIs, failure signatures, configuration changes, and architectural decisions. In technical work, repetition can indicate persistent work context rather than low value.
-
-## 2. The Convex Mixture
-
-The corrected gate is:
-\[ G(t) = (1 - T(t)) \cdot G_{\mathrm{conv}}(t) + T(t) \cdot G_{\mathrm{tech}}(t) \]
-
-where:
-\[ G_{\mathrm{conv}}(t) = w_1^c H(t) + w_2^c R(t) + w_3^c D_{nl}(t) \]
-\[ G_{\mathrm{tech}}(t) = w_1^t P(t) + w_2^t A(t) + w_3^t D_{\mathrm{tech}}(t) \]
-
-and the domain indicator is bounded:
-\[ T(t) \in [0,1] \]
-
-### Weight Invariants
-To guarantee that the sub-branch scores remain strictly bounded to $[0,1]$, the configuration must satisfy:
-\[ \sum_{i=1}^3 w_i^c = 1 \quad \text{and} \quad \sum_{i=1}^3 w_i^t = 1 \]
-
-Current default weights from `DefaultGatingConfig()`:
-- conversational branch: $w_1^c = 0.35$, $w_2^c = 0.40$, $w_3^c = 0.25$
-- technical branch: $w_1^t = 0.40$, $w_2^t = 0.35$, $w_3^t = 0.25$
-
-### Boundedness and Continuity
-Because $T(t) \in [0,1]$, $G_{\mathrm{conv}}(t) \in [0,1]$, and $G_{\mathrm{tech}}(t) \in [0,1]$, $G(t)$ is a true convex combination bounded to $[0,1]$.
-
-The gate is continuous in $T$:
-\[ \frac{\partial G}{\partial T} = G_{\mathrm{tech}} - G_{\mathrm{conv}} \]
-There is no discontinuous jump at a domain boundary. A mixed technical/conversational turn interpolates smoothly.
-
-## 3. Domain Detection $T(t)$
-
-Technical density is a weighted sum of technical patterns with saturation:
-\[ T(t) = \min\left(\frac{\sum_i s_i \cdot \mathbf{1}[\mathrm{pattern}_i(t)]}{\theta_{\mathrm{norm}}}, 1\right) \]
-
-The shipped patterns include code fences, file paths, function definitions, shell commands, URLs, stack traces, and hashes.
-
-Default normalization is $\theta_{\mathrm{norm}} = 1.5$. This means two strong technical signals are enough to saturate the branch weight. Saturation at `1.0` is correct because the gate only needs the branch mixture weight, not a unbounded "technical magnitude."
-
-## 4. Conversational Branch
-
-### Novelty $H(t)$
-
-In the live implementation (`sidecar/compact/gate.go`), retrieval scores reaching the gate use the public higher-is-better cosine-style relevance contract from the retrieval layer, spanning $[-1, 1]$ for cosine collections. To ensure the novelty term remains in $[0,1]$ for the convex mixture, the mathematical model applies a zero-clamp:
-
-\[ H(t) = \begin{cases} 
-1.0 & \text{if } |K| = 0 \\ 
-1 - \frac{1}{|K|} \sum_{k \in K} \max(0, \cos(\vec{v}_t, \vec{v}_k)) & \text{otherwise}
-\end{cases} \]
-
-where $K$ is the retrieved nearest-neighbor set from durable `user:` memory.
-
-Properties:
-- An empty memory (cold start) safely returns $H=1.0$ instead of a division-by-zero.
-- Highly similar existing memories ($\cos \to 1$) drive $H \to 0$.
-- Negative or orthogonal neighbors are clamped to prevent $H(t) > 1$.
-
-### Repetition Gate $R(t)$
-
-The repetition term is a product, not a sum:
-\[ R(t) = F(t) \cdot (1 - S(t)) \]
-
-with:
-\[ F(t) = \min\left(\frac{\mathrm{hitsAbove}(\mathrm{turns:u}, 0.80, k=10)}{5}, 1\right) \]
-\[ S(t) = \min\left(\frac{\mathrm{hitsAbove}(\mathrm{user:u}, 0.85, k=5)}{3}, 1\right) \]
-
-where $u$ is the resolved durable namespace used by the host boundary. The
-resolver chooses $u$ in this order: explicit `userId`, then the
-session-key-derived namespace, then `agentId`, and finally the resolver
-fallback/default when no host identity is available. When the host does not
-provide a `userId`, the gate still measures repetition and saturation against a
-stable durable scope.
-
-Why a product? High input frequency should help only if durable memory is not already saturated. High saturation must veto the repetition term regardless of frequency. The veto property is structural: $S(t) = 1 \Rightarrow R(t) = 0$.
-
-### Natural-Language Structural Load $D_{nl}(t)$
-Detects heuristics like preferences, human-name references, dates, and fact assertions.
-
-## 5. Technical Branch
-
-### Specificity $P(t)$
-
-Specificity measures concrete artifact density normalized by turn length:
-
-\[ P(t) = \min\left( \frac{\sum_j p_j \cdot \mathrm{count}_j(t)}{\max(L(t)/100.0, 1.0)}, 1 \right) \]
-
-The numerator counts things like file paths, error codes, and API endpoints.
-The normalization denominator is the token estimator used by the gating subsystem (`sidecar/compact/tokens.go`):
-\[ L(t) = \max\left(\left\lfloor \frac{\mathrm{RuneCount}(t)}{4} \right\rfloor, 1\right) \]
-
-Length normalization matters. Without it, any long technical turn would score high simply because it contains more surface area, not because it is more memory-worthy.
-
-### Actionability $A(t)$
-Captures architectural decisions, fixes, merge milestones, and configuration changes.
-
-### Technical Structural Load $D_{\mathrm{tech}}(t)$
-Detects function definitions, dependencies, and tests. It is the technical analogue to $D_{nl}$.
-
-## 6. Calibration
-
-For threshold tuning, isotonic regression is the correct calibration method once usefulness labels exist:
-\[ P(\mathrm{useful} \mid G) = \mathrm{IsotonicRegression}(G, y) \]
-
-Current thresholds implemented in code:
-- durable promotion: `DefaultGatingConfig().Threshold = 0.35`
-- abstractive routing: `AbstractiveRoutingThreshold = 0.60`
-
-## 7. Invariants
-
-The gate preserves six mathematical invariants mapped to `gate_test.go`:
-
-1. **Empty memory implies full novelty:** $\mathrm{memHits} = \emptyset \Rightarrow H = 1.0$
-2. **Saturation vetoes repetition:** $\mathrm{MemSaturation} = 1 \Rightarrow R = 0$
-3. **The convex blend stays in bounds:** $G \in [0,1]$
-4. **Monotonic Interpolation:** $G \in [\min(G_{\mathrm{conv}}, G_{\mathrm{tech}}), \max(G_{\mathrm{conv}}, G_{\mathrm{tech}})]$
-5. **Purely conversational turns collapse:** $T = 0 \Rightarrow G = G_{\mathrm{conv}}$
-6. **Purely technical turns collapse:** $T = 1 \Rightarrow G = G_{\mathrm{tech}}$
-
-Conversational structure must not overfire on pure code. Together these invariants make the scalar interpretable, stable, and safe to tune.