@xdarkicex/openclaw-memory-libravdb 1.3.5

package/docs/gating.md

@@ -0,0 +1,329 @@
+# Domain-Adaptive Gating Scalar
+
+This document describes the ingestion gate used to decide whether a user turn should be promoted into durable `user:` memory. It is the most novel scoring component in the repository.
+
+Implemented in:
+
+- [`sidecar/compact/gate.go`](../sidecar/compact/gate.go)
+- [`sidecar/compact/tokens.go`](../sidecar/compact/tokens.go)
+- [`sidecar/compact/summarize.go`](../sidecar/compact/summarize.go) for the
+  downstream abstractive-routing threshold
+
+## 1. Why the Original Scalar Failed
+
+The original scalar assumed conversational memory semantics:
+
+- low novelty meant "already known"
+- repetition meant "probably redundant"
+- low natural-language structure meant "probably noise"
+
+That logic breaks for technical sessions.
+
+Repeated workflow context is often exactly what should be remembered:
+
+- file paths
+- APIs
+- failure signatures
+- configuration changes
+- architectural decisions
+
+In technical work, repetition can indicate persistent work context rather than low value.
+
+## 2. The Convex Mixture
+
+The corrected gate is:
+
+$$
+G(t) = (1 - T(t)) \cdot G_{\mathrm{conv}}(t) + T(t) \cdot G_{\mathrm{tech}}(t)
+$$
+
+where:
+
+$$
+G_{\mathrm{conv}}(t) = w_1^c H(t) + w_2^c R(t) + w_3^c D_{nl}(t)
+$$
+
+$$
+G_{\mathrm{tech}}(t) = w_1^t P(t) + w_2^t A(t) + w_3^t D_{\mathrm{tech}}(t)
+$$
+
+and:
+
+$$
+T(t) \in [0,1]
+$$
+
+is the technical-density signal.
+
+Current default weights from
+[`DefaultGatingConfig()`](../sidecar/compact/gate.go):
+
+- conversational branch: $w_1^c = 0.35$, $w_2^c = 0.40$, $w_3^c = 0.25$
+- technical branch: $w_1^t = 0.40$, $w_2^t = 0.35$, $w_3^t = 0.25$
+
+### Boundedness
+
+If:
+
+- $T(t) \in [0,1]$
+- $G_{\mathrm{conv}}(t) \in [0,1]$
+- $G_{\mathrm{tech}}(t) \in [0,1]$
+
+then:
+
+$$
+G(t) \in [0,1]
+$$
+
+because $G$ is a convex combination of two values in $[0,1]$.
+
+### Continuity
+
+The gate is continuous in $T$:
+
+$$
+\frac{\partial G}{\partial T} = G_{\mathrm{tech}} - G_{\mathrm{conv}}
+$$
+
+There is no discontinuous jump at a domain boundary. A mixed technical/conversational turn interpolates smoothly between the two sub-formulas.
+
+## 3. Domain Detection $T(t)$
+
+Technical density is a weighted sum of technical patterns with saturation:
+
+$$
+T(t) = \min\left(\frac{\sum_i s_i \cdot \mathbf{1}[\mathrm{pattern}_i(t)]}{\theta_{\mathrm{norm}}}, 1\right)
+$$
+
+The shipped patterns include:
+
+- code fences
+- file paths
+- function definitions
+- shell commands
+- URLs or endpoints
+- stack traces
+- hashes or hex identifiers
+
+Default normalization:
+
+$$
+\theta_{\mathrm{norm}} = 1.5
+$$
+
+This means two strong technical signals are enough to saturate the branch weight.
+
+Saturation at `1.0` is correct because the gate does not need "how technical beyond fully technical"; it only needs the branch mixture weight.
+
+## 4. Conversational Branch
+
+### Novelty $H(t)$
+
+Novelty is:
+
+$$
+H(t) = 1 - \frac{1}{|K|} \sum_{k \in K} \cos(\vec{v}_t, \vec{v}_k)
+$$
+
+where $K$ is the retrieved nearest-neighbor set from durable `user:` memory.
+
+Properties:
+
+- empty memory gives $H=1.0$
+- highly similar existing memories drive $H$ toward `0`
+
+The implementation deliberately uses top-k mean similarity rather than centroid distance because user memory is often multimodal.
+
+### Repetition Gate $R(t)$
+
+The repetition term is:
+
+$$
+R(t) = F(t) \cdot (1 - S(t))
+$$
+
+with:
+
+$$
+F(t) = \min\left(\frac{\mathrm{hitsAbove}(\mathrm{turns:userId}, 0.80, k=10)}{5}, 1\right)
+$$
+
+$$
+S(t) = \min\left(\frac{\mathrm{hitsAbove}(\mathrm{user:userId}, 0.85, k=5)}{3}, 1\right)
+$$
+
+This is intentionally a product, not a sum.
+
+Why:
+
+- high input frequency should help only if durable memory is not already saturated
+- high saturation should veto the repetition term regardless of frequency
+
+The veto property is structural:
+
+$$
+S(t) = 1 \Rightarrow R(t) = 0
+$$
+
+### Natural-Language Structural Load $D_{nl}(t)$
+
+The conversational branch adds heuristic structure for turns that look like:
+
+- preferences
+- human-name references
+- dates
+- quantities
+- fact assertions
+
+This is intentionally narrow. It excludes general proper-noun detection so technical identifiers do not inflate the conversational signal.
+
+## 5. Technical Branch
+
+### Specificity $P(t)$
+
+Specificity measures concrete artifact density:
+
+$$
+P(t) = \min\left(
+\frac{
+\sum_j p_j \cdot \mathrm{count}_j(t)
+}{
+\max(\mathrm{EstimateTokens}(t)/100, 1)
+},
+1
+\right)
+$$
+
+The numerator counts things like:
+
+- file paths
+- function references
+- error codes
+- git references
+- API endpoints
+
+The normalization denominator is implemented in
+[`sidecar/compact/tokens.go`](../sidecar/compact/tokens.go):
+
+$$
+L(t)=\max\left(\left\lfloor \frac{\mathrm{len}(t)}{4} \right\rfloor, 1\right)
+$$
+
+This bytes-per-token heuristic is the token estimator used by the gating
+subsystem. It is intentionally cheap and deterministic. It is not the same as
+the separate host-side prompt-budget estimator in [`src/tokens.ts`](../src/tokens.ts).
+
+Length normalization matters. Without it, any long technical turn would score
+high simply because it contains more surface area, not because it is more
+memory-worthy.
+
+### Actionability $A(t)$
+
+Actionability captures decision and outcome content:
+
+- architectural decisions
+- fixes or resolutions
+- deployment or merge milestones
+- configuration changes
+
+These are the kinds of technical turns that are expensive to reconstruct later and therefore worth persisting.
+
+### Technical Structural Load $D_{\mathrm{tech}}(t)$
+
+This branch detects structural technical content such as:
+
+- function definitions
+- data structures
+- dependencies
+- tests
+- documentation comments
+
+It is the technical analogue to $D_{nl}$, not a replacement for it.
+
+## 6. Calibration
+
+Stored metadata includes:
+
+- `gating_score`
+- `gating_t`
+- `gating_h`
+- `gating_r`
+- `gating_d`
+- `gating_p`
+- `gating_a`
+- `gating_dtech`
+- `gating_gconv`
+- `gating_gtech`
+
+The first calibration pass should inspect the empirical score distribution after real traffic arrives.
+
+What to look for:
+
+- bimodality in `gating_score`
+- sensible spread in `gating_t`
+- non-degenerate contributions from both `gconv` and `gtech`
+
+For threshold tuning, isotonic regression is the correct calibration method once usefulness labels exist:
+
+$$
+P(\mathrm{useful} \mid G) = \mathrm{IsotonicRegression}(G, y)
+$$
+
+It preserves the monotonic design of the gate without assuming a sigmoid link function.
+
+Current thresholds implemented in code:
+
+- durable promotion threshold:
+  [`DefaultGatingConfig().Threshold = 0.35`](../sidecar/compact/gate.go)
+- abstractive compaction routing threshold:
+  [`AbstractiveRoutingThreshold = 0.60`](../sidecar/compact/summarize.go)
+
+## 7. Invariants
+
+The gate has six mathematical invariants in `gate_test.go`.
+
+### 1. Empty memory implies full novelty
+
+$$
+\mathrm{memHits} = \emptyset \Rightarrow H = 1.0
+$$
+
+This prevents a cold start from suppressing every first durable insertion.
+
+### 2. Saturation vetoes repetition
+
+$$
+\mathrm{MemSaturation} = 1 \Rightarrow R = 0
+$$
+
+This is what makes the repetition term a true gate instead of an accumulation bonus.
+
+### 3. The convex blend stays in bounds
+
+$$
+G \in [0,1]
+$$
+
+and:
+
+$$
+G \in [\min(G_{\mathrm{conv}}, G_{\mathrm{tech}}), \max(G_{\mathrm{conv}}, G_{\mathrm{tech}})]
+$$
+
+### 4. Purely conversational turns collapse to the conversational branch
+
+$$
+T = 0 \Rightarrow G = G_{\mathrm{conv}}
+$$
+
+### 5. Purely technical turns collapse to the technical branch
+
+$$
+T = 1 \Rightarrow G = G_{\mathrm{tech}}
+$$
+
+### 6. Conversational structure should not overfire on pure code
+
+This guards against a common failure mode where technical identifiers masquerade as conversational entities.
+
+Together these invariants make the scalar interpretable, stable, and safe to tune later from real traffic rather than intuition.

package/docs/implementation.md

@@ -0,0 +1,381 @@
+# Implementation Notes and Interfaces
+
+This document explains the implemented contracts that are easy to miss when
+reading the code piecemeal.
+
+## Memory Kind Plus Explicit Context Engine Registration
+
+The plugin declares `kind: "memory"` in
+[`openclaw.plugin.json`](../openclaw.plugin.json), but still registers both a
+context engine and a memory prompt section in [`src/index.ts`](../src/index.ts).
+
+Why:
+
+- the exclusive slot takeover happens through the `memory` kind
+- the runtime behavior still needs explicit lifecycle hooks for:
+  - `bootstrap`
+  - `ingest`
+  - `assemble`
+  - `compact`
+- the lightweight memory prompt section remains useful as a separate early
+  durable-recall pass
+
+This is why the code registers both `registerContextEngine("libravdb-memory", …)`
+and `registerMemoryPromptSection(...)` instead of relying on only one hook.
+
+## Why Ingest Is Fire-and-Forget
+
+Implemented in [`src/context-engine.ts`](../src/context-engine.ts).
+
+Session insertion is intentionally fire-and-forget:
+
+- the active conversation should not block on persistence
+- session memory is useful immediately, but not allowed to become a hard
+  dependency for response generation
+
+The current code writes to `session:<sessionId>` asynchronously and only then
+attempts the more expensive durable-promotion path for user turns.
+
+## Why Gating Uses Exactly Two Searches
+
+Implemented in [`sidecar/server/rpc.go`](../sidecar/server/rpc.go).
+
+`gating_scalar` performs exactly:
+
+1. one search against `turns:<userId>`
+2. one search against `user:<userId>`
+
+Novelty and durable-memory saturation reuse the same `user:` hit set. This keeps
+the RPC bounded and predictable. There is no third store query for novelty.
+
+## Why the Token Estimator Uses Bytes/4 in the Gate
+
+Implemented in [`sidecar/compact/tokens.go`](../sidecar/compact/tokens.go).
+
+The gate's specificity term uses:
+
+$$
+\mathrm{EstimateTokens}(t)=\max(\lfloor \mathrm{len}(t)/4 \rfloor, 1)
+$$
+
+Why not word count:
+
+- word count behaves badly on code
+- file paths, stack traces, and identifiers are token-dense but word-sparse
+- bytes/4 is cheap and stable across prose, code, and mixed technical content
+
+Important boundary:
+
+- this is the gating estimator
+- prompt-budget fitting uses a separate host-side chars-per-token heuristic in
+  [`src/tokens.ts`](../src/tokens.ts)
+
+## Why the Sidecar Emits Its Endpoint on stdout
+
+Implemented in [`sidecar/main.go`](../sidecar/main.go) and
+[`src/sidecar.ts`](../src/sidecar.ts).
+
+The sidecar prints its runtime endpoint to stdout on startup instead of binding
+to a fixed path known in advance.
+
+Why:
+
+- fixed Unix socket paths create collision risk across concurrent runs
+- temporary per-process endpoints avoid stale socket cleanup problems
+- Windows already requires a dynamic TCP fallback
+
+The host watches stdout, captures the endpoint, and then establishes the
+JSON-RPC transport.
+
+## Why Degraded Mode Continues the Session
+
+Implemented in [`src/sidecar.ts`](../src/sidecar.ts) and
+[`src/context-engine.ts`](../src/context-engine.ts).
+
+If the sidecar fails repeatedly, the plugin enters degraded mode instead of
+failing the chat session.
+
+Why:
+
+- memory augmentation is valuable, but it is not allowed to become a hard
+  dependency for the core conversation path
+- the safe fallback is "continue without memory augmentation" rather than
+  "reject the entire turn"
+
+This is deliberate fault containment.
+
+## `ownsCompaction: true`
+
+Implemented in [`src/context-engine.ts`](../src/context-engine.ts).
+
+The context engine factory returns `ownsCompaction: true`.
+
+This tells the host that compaction belongs to the memory engine lifecycle
+itself. In this plugin, compaction is not an optional helper or an external
+maintenance job; it is part of the actual memory system contract.
+
+## Interface: `GatingConfig`
+
+Defined in [`sidecar/compact/gate.go`](../sidecar/compact/gate.go).
+
+```go
+type GatingConfig struct {
+    W1c float64
+    W2c float64
+    W3c float64
+    W1t float64
+    W2t float64
+    W3t float64
+    TechNorm  float64
+    Threshold float64
+}
+```
+
+Field meanings:
+
+- `W1c`, `W2c`, `W3c`: conversational-branch weights for novelty `H`,
+  repetition gate `R`, and conversational structure `D`
+- `W1t`, `W2t`, `W3t`: technical-branch weights for specificity `P`,
+  actionability `A`, and technical structure `Dtech`
+- `TechNorm`: normalization constant for technical-density saturation
+- `Threshold`: durable-promotion cutoff used by the host
+
+Contract:
+
+- all weights are intended to be in `[0,1]`
+- each branch should sum to `1.0` by convention
+- `TechNorm <= 0` is normalized back to the default inside `computeT`
+- zero values are not generally meaningful outside tests; callers should use
+  `DefaultGatingConfig()` or config-derived values from [`sidecar/main.go`](../sidecar/main.go)
+
+## Interface: `GatingSignals`
+
+Defined in [`sidecar/compact/gate.go`](../sidecar/compact/gate.go).
+
+```go
+type GatingSignals struct {
+    G float64
+    T float64
+    H float64
+    R float64
+    D float64
+    InputFreq     float64
+    MemSaturation float64
+    P     float64
+    A     float64
+    Dtech float64
+    Gconv float64
+    Gtech float64
+}
+```
+
+Field meanings:
+
+- `G`: final gate score in `[0,1]`
+- `T`: technical-density branch weight in `[0,1]`
+- `H`: novelty score in `[0,1]`
+- `R`: repetition product gate in `[0,1]`
+- `D`: conversational structural load in `[0,1]`
+- `InputFreq`: normalized repeated-mention signal in `[0,1]`
+- `MemSaturation`: normalized durable-memory saturation signal in `[0,1]`
+- `P`: technical specificity score in `[0,1]`
+- `A`: technical actionability score in `[0,1]`
+- `Dtech`: technical structural load in `[0,1]`
+- `Gconv`: weighted conversational branch score
+- `Gtech`: weighted technical branch score
+
+Zero-value behavior:
+
+- an all-zero `GatingSignals` struct does not mean "valid low-confidence turn";
+  it usually means "not computed yet"
+- missing metadata readers should treat absent values as `0.0` and not panic
+
+Inputs vs outputs:
+
+- `GatingConfig` is input
+- `turnHits`, `memHits`, and `text` are inputs to `ComputeGating`
+- `GatingSignals` is output and is intended to be persisted in metadata
+
+## Interface: JSON-RPC Surface
+
+Implemented in [`sidecar/server/rpc.go`](../sidecar/server/rpc.go).
+
+Method names are snake_case in the actual protocol.
+
+### `health`
+
+- request: `{}`
+- response: `{ ok: boolean, message: string }`
+- errors: none expected unless transport fails
+
+### `status`
+
+- request: `{}`
+- response:
+  `{ ok, message, turnCount, memoryCount, gatingThreshold, abstractiveReady, embeddingProfile }`
+- errors: none expected unless transport fails
+
+### `ensure_collections`
+
+- request: `{ collections: string[] }`
+- response: `{ ok: true }`
+- errors:
+  - collection creation failure in the Go store
+
+### `insert_text`
+
+- request:
+  `{ collection: string, id: string, text: string, metadata: object }`
+- response: `{ ok: true }`
+- errors:
+  - embedding failure
+  - record validation failure
+  - store insertion failure
+
+### `gating_scalar`
+
+- request: `{ userId: string, text: string }`
+- response: full `GatingSignals` JSON payload
+- errors:
+  - search failure on `turns:<userId>` or `user:<userId>`
+
+### `search_text`
+
+- request:
+  `{ collection: string, text: string, k: number, excludeIds?: string[] }`
+- response: `{ results: SearchResult[] }`
+- errors:
+  - query embedding failure
+  - store search failure
+
+### `list_by_meta`
+
+- request: `{ collection: string, key: string, value: string }`
+- response: `{ results: SearchResult[] }`
+- errors:
+  - store listing failure
+
+### `export_memory`
+
+- request: `{ userId?: string }`
+- response:
+  `{ records: Array<{ collection, id, text, metadata }> }`
+- errors:
+  - collection listing failure
+
+### `flush_namespace`
+
+- request: `{ userId: string }`
+- response: `{ ok: true }`
+- errors:
+  - missing `userId`
+  - delete-by-prefix failure
+
+### `delete`
+
+- request: `{ collection: string, id: string }`
+- response: `{ ok: true }`
+- errors:
+  - delete failure
+
+### `delete_batch`
+
+- request: `{ collection: string, ids: string[] }`
+- response: `{ ok: true }`
+- errors:
+  - batch delete failure
+
+### `compact_session`
+
+- request: `{ sessionId: string, force: boolean, targetSize?: number }`
+- response:
+  `{ didCompact, clustersFormed, turnsRemoved, summaryMethod, meanConfidence }`
+- errors:
+  - missing session id
+  - extractive summarizer unavailable
+  - summary insertion failure
+  - summarization failure
+
+### `flush`
+
+- request: `{}`
+- response: `{ ok: true }`
+- errors:
+  - store flush failure
+
+## Interface: Context Engine Lifecycle
+
+Implemented in [`src/context-engine.ts`](../src/context-engine.ts).
+
+The factory returns an object with this effective shape:
+
+```ts
+{
+  ownsCompaction: true,
+  bootstrap(args: ContextBootstrapArgs): Promise<{ ok: true }>,
+  ingest(args: ContextIngestArgs): Promise<{ ingested: boolean }>,
+  assemble(args: ContextAssembleArgs): Promise<{
+    messages: MemoryMessage[],
+    estimatedTokens: number,
+    systemPromptAddition: string,
+  }>,
+  compact(args: ContextCompactArgs): Promise<{ ok: true, compacted: boolean }>,
+}
+```
+
+### `bootstrap`
+
+Input:
+
+- `sessionId`
+- `userId`
+
+Behavior:
+
+- ensures `session:`, `turns:`, `user:`, and `global` collections exist
+
+### `ingest`
+
+Input:
+
+- `sessionId`
+- `userId`
+- `message`
+- `isHeartbeat?`
+
+Behavior:
+
+- must not block the session on best-effort persistence
+- writes all non-heartbeat messages to session memory
+- only user turns go through the durable gating path
+
+### `assemble`
+
+Input:
+
+- `sessionId`
+- `userId`
+- `messages`
+- `tokenBudget`
+
+Behavior:
+
+- must not mutate the incoming `messages` array in place
+- searches three scopes in parallel
+- hybrid-ranks and budget-fits the result set
+- prepends selected memories as synthetic system messages
+- falls back cleanly to the original message list on failure
+
+### `compact`
+
+Input:
+
+- `sessionId`
+- `force?`
+- `targetSize?`
+
+Behavior:
+
+- delegates to the sidecar `compact_session` RPC
+- returns `{ ok: true, compacted }`
+- treats compaction failure as non-fatal to the active session