@xdarkicex/openclaw-memory-libravdb 1.3.5

package/docs/models.md

@@ -0,0 +1,63 @@
+# Model Strategy
+
+## Why ONNX Over Ollama
+
+The plugin uses ONNX-first local inference for embedding and optional abstractive summarization.
+
+### Latency
+
+`assemble` is on the critical path before every response build. An embedding request that crosses process and HTTP boundaries adds avoidable tail latency. Local ONNX inference inside the sidecar keeps the retrieval path in the low-millisecond range on the target hardware profile.
+`assemble` is on the critical path before every response build. An embedding
+request that crosses process and HTTP boundaries adds avoidable tail latency.
+Local ONNX inference inside the sidecar keeps the retrieval path local and
+predictable. On the current Apple M2 development machine, the repository's own
+benchmark harness measures roughly `16-23 ms/op` for MiniLM query embeddings and
+about `44 ms/op` for Nomic in the steady-state Go benchmark path.
+
+### Offline Operation
+
+The plugin is designed to be local-first. Requiring a running Ollama server would break that guarantee. ONNX assets can be provisioned once and reused without network or daemon availability.
+
+### Determinism
+
+ONNX inference is deterministic given fixed weights and input. Deterministic embeddings give stable similarity ordering and reproducible retrieval behavior.
+
+### Binary Size Trade-Off
+
+Local models increase the artifact footprint. That is an explicit trade-off accepted by the architecture because predictable latency and offline operation are more important for this plugin than minimal package size.
+
+## Why `nomic-embed-text-v1.5`
+
+This is the default embedding profile because it earned the role on two axes:
+
+- long-context document support
+- Matryoshka structure for tiered retrieval
+
+The model’s Matryoshka training is what makes the `64d -> 256d -> 768d` cascade principled rather than arbitrary truncation.
+
+## Why `all-minilm-l6-v2` Still Exists
+
+MiniLM remains the lightweight fallback profile. It is useful when:
+
+- the full Nomic profile is unavailable
+- a smaller bundled footprint matters more than long-context or Matryoshka behavior
+
+It is no longer the quality-first default.
+
+## Why T5-small for Summarization
+
+The abstractive summarization path is optional and must remain CPU-feasible on local machines. T5-small fits that constraint better than larger generative models:
+
+- small enough to run locally
+- expressive enough for session-cluster summarization
+- does not require a remote server
+
+The plugin still degrades gracefully to extractive compaction when the T5 assets are not provisioned.
+
+## Model Roles in the System
+
+- Nomic embedder: quality-first retrieval path, Matryoshka tiers
+- MiniLM: fallback embedder
+- T5-small: optional higher-quality compaction summarizer
+
+The model strategy is therefore not “use ONNX everywhere because ONNX is fashionable.” It is “use ONNX where local deterministic inference is part of the product contract.”

package/docs/problem.md

@@ -0,0 +1,64 @@
+# The Problem This Plugin Solves
+
+This plugin exists because the stock OpenClaw memory path is optimized for lightweight persistence, not for a full context lifecycle with scope separation, compaction, and bounded retrieval.
+
+## What the Stock Plugin Is Good At
+
+The default `memory-lancedb` style plugin is still a good fit when you want:
+
+- a simple top-k semantic lookup over durable notes
+- a low-complexity setup
+- a heuristic capture model where the agent decides what to save
+- no additional sidecar process
+
+For short sessions and light persistent memory, that is often the right answer.
+
+## Where the Stock Model Breaks Down
+
+The workload this plugin targets is different:
+
+- long technical sessions
+- repeated workflow state that must not be lost
+- user/session/global scopes that behave differently
+- compaction instead of unbounded append-only growth
+- retrieval that must fit within a prompt token budget
+
+The failure modes are structural, not cosmetic.
+
+### Context collapse in long sessions
+
+A single-table top-k memory system has no first-class notion of ephemeral session state versus durable user memory. A long active session accumulates turns that are highly relevant right now but should not pollute long-term recall forever.
+
+### No scope separation
+
+This plugin uses distinct namespaces for:
+
+- `session:<sessionId>`
+- `turns:<userId>`
+- `user:<userId>`
+- `global`
+
+Those scopes are not interchangeable. Session state needs different retention, different recency behavior, and different retrieval weighting than durable user memory.
+
+### No token budget management
+
+Retrieval is only half the problem. The selected memories must still fit inside a bounded prompt budget. This plugin treats ranking and packing as one pipeline, not as disconnected concerns.
+
+### No automatic compaction
+
+Raw turns are not the right long-term storage format. The plugin clusters session turns, summarizes them, removes source turns after confirmed summary insertion, and lets summary confidence affect later retrieval. Without compaction, memory becomes an append-only dump that eventually competes with itself.
+
+## Why This Is a Lifecycle Replacement
+
+The solution is not a different LanceDB parameter or a larger `topK`.
+
+The plugin replaces the whole context lifecycle:
+
+1. ingestion
+2. gating and promotion
+3. retrieval and ranking
+4. token-budget fitting
+5. compaction and summarization
+6. quality feedback into future retrieval
+
+That is why the plugin is a slot-level memory replacement instead of a helper library layered beside the stock plugin.

package/docs/security.md

@@ -0,0 +1,86 @@
+# Security Model
+
+This plugin treats recalled memory as untrusted historical context. That is a structural design rule, not a prompt-style suggestion.
+
+## Ingestion Security Layers
+
+The practical pipeline is:
+
+1. collection scoping
+2. metadata tagging
+3. bounded retrieval
+4. untrusted-memory framing
+5. host fallback when memory is unavailable
+
+The system is designed so a failure in one layer does not automatically collapse the others.
+
+## Supply Chain and Installer Trust Boundary
+
+This repository uses both `postinstall` and `openclaw.setup`. That is a real
+security-sensitive surface in the OpenClaw ecosystem and should be evaluated
+explicitly rather than hand-waved away.
+
+Current implementation facts:
+
+- [`scripts/postinstall.js`](../scripts/postinstall.js) installs the sidecar
+  binary using a prebuilt-first strategy with a local Go fallback
+- [`scripts/setup.ts`](../scripts/setup.ts) provisions model/runtime assets and
+  verifies them before they are accepted
+- required downloaded assets are SHA-256 checked before use
+- an asset that exists but fails verification is deleted and re-downloaded
+
+The current installer fetches from these classes of sources only:
+
+- GitHub release assets for prebuilt sidecar binaries
+- ONNX Runtime release assets
+- model artifacts explicitly referenced in `setup.ts`
+
+After installation, the plugin is local-first:
+
+- no required network calls are made for embedding
+- no required network calls are made for extractive compaction
+- the only optional runtime network path is an explicitly configured external
+  summarizer endpoint, such as an Ollama server
+
+That trust boundary matters because it is exactly the area security-conscious
+users will inspect first.
+
+## Untrusted-Memory Framing
+
+Retrieved memory is injected with framing that explicitly tells the downstream model to treat it as untrusted historical context only.
+
+This matters because memory is persistent user-controlled content. Without structural framing, a stored memory can become an unintended prompt injection surface.
+
+The framing is implemented in the host-side memory header builder and is applied consistently at assembly time.
+
+## Collection Isolation
+
+The plugin structurally separates:
+
+- session memory
+- raw turn history
+- durable user memory
+- global memory
+
+Cross-user leakage is prevented by collection naming and lookup boundaries. The gate, compaction, and retrieval code all operate on explicit scope-qualified collection names rather than a shared unscoped table.
+
+## What the Plugin Cannot Protect Against
+
+The plugin does not claim to protect against:
+
+- a compromised host process
+- a compromised local machine
+- a downstream model that ignores the untrusted-memory framing instruction
+- intentionally malicious content stored by an already-authorized local actor
+
+It reduces risk; it does not create a trusted execution environment.
+
+## Deletion and Data Protection
+
+The sidecar exposes deletion and flush primitives. That matters operationally for:
+
+- user-requested memory removal
+- namespace cleanup
+- compaction source-turn deletion
+
+The GDPR-relevant boundary is simple: local stored memory can be deleted by namespace. The plugin does not by itself guarantee remote erasure from any external system because the architecture is intentionally local-first.

package/openclaw.plugin.json

@@ -0,0 +1,84 @@
+{
+  "id": "libravdb-memory",
+  "name": "LibraVDB Memory",
+  "description": "Persistent vector memory with three-tier hybrid scoring",
+  "version": "1.3.3",
+  "kind": "memory",
+  "configSchema": {
+    "type": "object",
+    "additionalProperties": false,
+    "properties": {
+      "dbPath": { "type": "string" },
+      "sidecarPath": { "type": "string" },
+      "embeddingRuntimePath": { "type": "string" },
+      "embeddingBackend": { "type": "string", "enum": ["bundled", "onnx-local", "custom-local"] },
+      "embeddingProfile": {
+        "type": "string",
+        "default": "nomic-embed-text-v1.5"
+      },
+      "fallbackProfile": {
+        "type": "string",
+        "default": "all-minilm-l6-v2"
+      },
+      "embeddingModelPath": { "type": "string" },
+      "embeddingTokenizerPath": { "type": "string" },
+      "embeddingDimensions": { "type": "number" },
+      "embeddingNormalize": { "type": "boolean" },
+      "summarizerBackend": { "type": "string", "enum": ["bundled", "onnx-local", "ollama-local", "custom-local"] },
+      "summarizerProfile": { "type": "string" },
+      "summarizerRuntimePath": { "type": "string" },
+      "summarizerModelPath": { "type": "string" },
+      "summarizerTokenizerPath": { "type": "string" },
+      "summarizerModel": { "type": "string" },
+      "summarizerEndpoint": { "type": "string" },
+      "sessionTTL": { "type": "number" },
+      "topK": { "type": "number" },
+      "alpha": { "type": "number" },
+      "beta": { "type": "number" },
+      "gamma": { "type": "number" },
+      "ingestionGateThreshold": {
+        "type": "number",
+        "default": 0.35
+      },
+      "gatingWeights": {
+        "type": "object",
+        "additionalProperties": false,
+        "default": {
+          "w1c": 0.35, "w2c": 0.40, "w3c": 0.25,
+          "w1t": 0.40, "w2t": 0.35, "w3t": 0.25
+        },
+        "properties": {
+          "w1c": { "type": "number", "default": 0.35 },
+          "w2c": { "type": "number", "default": 0.40 },
+          "w3c": { "type": "number", "default": 0.25 },
+          "w1t": { "type": "number", "default": 0.40 },
+          "w2t": { "type": "number", "default": 0.35 },
+          "w3t": { "type": "number", "default": 0.25 }
+        }
+      },
+      "gatingTechNorm": {
+        "type": "number",
+        "default": 1.5
+      },
+      "gatingCentroidK": {
+        "type": "number",
+        "default": 10
+      },
+      "compactionQualityWeight": {
+        "type": "number",
+        "default": 0.5,
+        "description": "Controls how much summary confidence affects retrieval score. 0 ignores summary quality and 1 fully suppresses zero-confidence summaries."
+      },
+      "recencyLambdaSession": { "type": "number" },
+      "recencyLambdaUser": { "type": "number" },
+      "recencyLambdaGlobal": { "type": "number" },
+      "tokenBudgetFraction": { "type": "number" },
+      "compactThreshold": { "type": "number" },
+      "ollamaUrl": { "type": "string" },
+      "compactModel": { "type": "string" },
+      "rpcTimeoutMs": { "type": "number", "default": 30000 },
+      "maxRetries": { "type": "number" },
+      "logLevel": { "type": "string" }
+    }
+  }
+}

package/package.json

@@ -0,0 +1,41 @@
+{
+  "name": "@xdarkicex/openclaw-memory-libravdb",
+  "version": "1.3.5",
+  "type": "module",
+  "publishConfig": {
+    "access": "public"
+  },
+  "files": [
+    "README.md",
+    "openclaw.plugin.json",
+    "package.json",
+    "src/",
+    "scripts/",
+    "sidecar/",
+    "docs/",
+    "tsconfig.json",
+    "tsconfig.tests.json"
+  ],
+  "engines": {
+    "node": ">=22"
+  },
+  "openclaw": {
+    "extensions": [
+      "./src/index.ts"
+    ],
+    "setup": "./scripts/setup.ts"
+  },
+  "scripts": {
+    "check": "./.ts-toolchain/node_modules/.bin/tsc --noEmit && pnpm run test:ts",
+    "test:ts": "./.ts-toolchain/node_modules/.bin/tsc -p tsconfig.tests.json && node --test .ts-build/test/unit/*.test.js",
+    "test:integration": "./.ts-toolchain/node_modules/.bin/tsc -p tsconfig.tests.json && node --test .ts-build/test/integration/checklist-validation.test.js .ts-build/test/integration/host-flow.test.js .ts-build/test/integration/sidecar-lifecycle.test.js",
+    "postinstall": "node scripts/postinstall.js",
+    "build:sidecar": "bash scripts/build-sidecar.sh"
+  },
+  "dependencies": {
+    "openclaw": "*"
+  },
+  "peerDependencies": {
+    "openclaw": ">=2026.3.22"
+  }
+}

package/scripts/build-sidecar.sh

@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT_DIR="$(cd "$(dirname "$0")/.." && pwd)"
+OUT_DIR="$ROOT_DIR/.sidecar-bin"
+BIN_NAME="libravdb-sidecar"
+MODELS_DIR="$ROOT_DIR/.models"
+OUT_MODELS_DIR="$OUT_DIR/models"
+OUT_RUNTIME_DIR="$OUT_DIR/onnxruntime"
+
+if [[ "${OS:-}" == "Windows_NT" ]]; then
+  BIN_NAME="libravdb-sidecar.exe"
+fi
+
+mkdir -p "$OUT_DIR"
+cd "$ROOT_DIR/sidecar"
+GOCACHE="${GOCACHE:-/tmp/openclaw-memory-libravdb-gocache}" go build -o "$OUT_DIR/$BIN_NAME" .
+rm -rf "$OUT_MODELS_DIR" "$OUT_RUNTIME_DIR"
+mkdir -p "$OUT_MODELS_DIR"
+if [[ -d "$MODELS_DIR/all-minilm-l6-v2" ]]; then
+  cp -R "$MODELS_DIR/all-minilm-l6-v2" "$OUT_MODELS_DIR/all-minilm-l6-v2"
+fi
+if [[ -d "$MODELS_DIR/nomic-embed-text-v1.5" ]]; then
+  cp -R "$MODELS_DIR/nomic-embed-text-v1.5" "$OUT_MODELS_DIR/nomic-embed-text-v1.5"
+fi
+if [[ -d "$MODELS_DIR/onnxruntime" ]]; then
+  mkdir -p "$OUT_RUNTIME_DIR"
+  cp -R "$MODELS_DIR/onnxruntime/." "$OUT_RUNTIME_DIR/"
+fi
+echo "built sidecar: $OUT_DIR/$BIN_NAME"

package/scripts/postinstall.js

@@ -0,0 +1,169 @@
+#!/usr/bin/env node
+
+import { chmodSync, cpSync, createWriteStream, existsSync, mkdirSync, readFileSync, renameSync, rmSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { createHash } from "node:crypto";
+import { spawnSync } from "node:child_process";
+import { Readable } from "node:stream";
+import { pipeline } from "node:stream/promises";
+import { fileURLToPath } from "node:url";
+import { buildSidecarReleaseAssetURL, detectSidecarReleaseTarget } from "./sidecar-release.js";
+
+const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
+const sidecarDir = path.join(root, "sidecar");
+const outDir = path.join(root, ".sidecar-bin");
+const binary = process.platform === "win32" ? "libravdb-sidecar.exe" : "libravdb-sidecar";
+const modelsDir = path.join(root, ".models");
+const outModelsDir = path.join(outDir, "models");
+const outRuntimeDir = path.join(outDir, "onnxruntime");
+const pkg = JSON.parse(readFileSync(path.join(root, "package.json"), "utf8"));
+
+mkdirSync(outDir, { recursive: true });
+
+const installed = await installSidecar(pkg.version);
+if (!installed) {
+  console.error("[openclaw-memory-libravdb] FATAL: sidecar binary could not be installed.");
+  console.error("  1. Check your internet connection (prebuilt download failed)");
+  console.error("  2. Or install Go >= 1.22 and retry: https://go.dev/dl/");
+  process.exit(1);
+}
+
+rmSync(outModelsDir, { recursive: true, force: true });
+rmSync(outRuntimeDir, { recursive: true, force: true });
+mkdirSync(outModelsDir, { recursive: true });
+
+const bundledMiniLM = path.join(modelsDir, "all-minilm-l6-v2");
+if (existsSync(bundledMiniLM)) {
+  cpSync(bundledMiniLM, path.join(outModelsDir, "all-minilm-l6-v2"), { recursive: true });
+}
+
+const bundledNomic = path.join(modelsDir, "nomic-embed-text-v1.5");
+if (existsSync(bundledNomic)) {
+  cpSync(bundledNomic, path.join(outModelsDir, "nomic-embed-text-v1.5"), { recursive: true });
+}
+
+const runtimeBundle = path.join(modelsDir, "onnxruntime");
+if (existsSync(runtimeBundle)) {
+  cpSync(runtimeBundle, outRuntimeDir, { recursive: true });
+}
+
+async function installSidecar(version) {
+  const target = detectSidecarReleaseTarget();
+  if (target) {
+    const assetUrl = buildSidecarReleaseAssetURL(version, target);
+    const checksumUrl = `${assetUrl}.sha256`;
+    const downloaded = await tryDownloadPrebuilt(assetUrl, checksumUrl, path.join(outDir, binary));
+    if (downloaded) {
+      console.log(`[openclaw-memory-libravdb] Sidecar installed (prebuilt ${target})`);
+      return true;
+    }
+    console.warn("[openclaw-memory-libravdb] Prebuilt binary unavailable or failed verification; attempting local go build.");
+  } else {
+    console.warn(`[openclaw-memory-libravdb] No prebuilt target for ${process.platform}-${process.arch}; attempting local go build.`);
+  }
+
+  return tryGoBuild(path.join(outDir, binary));
+}
+
+async function tryDownloadPrebuilt(assetUrl, checksumUrl, dest) {
+  try {
+    const checksum = await fetchChecksum(checksumUrl);
+    if (!checksum) {
+      return false;
+    }
+    await downloadToFile(assetUrl, dest);
+    const actual = sha256File(dest);
+    if (actual !== checksum) {
+      rmSync(dest, { force: true });
+      console.warn(`[openclaw-memory-libravdb] Prebuilt sidecar checksum mismatch for ${assetUrl}.`);
+      return false;
+    }
+    if (process.platform !== "win32") {
+      chmodSync(dest, 0o755);
+    }
+    return true;
+  } catch (error) {
+    rmSync(dest, { force: true });
+    console.warn(`[openclaw-memory-libravdb] Prebuilt sidecar download failed: ${formatError(error)}`);
+    return false;
+  }
+}
+
+function tryGoBuild(dest) {
+  const goCheck = spawnSync("go", ["version"], {
+    stdio: "pipe",
+    env: process.env,
+  });
+
+  if (goCheck.error && goCheck.error.code === "ENOENT") {
+    console.error("FATAL: Go toolchain not found on PATH. The LibraVDB sidecar cannot be built locally.");
+    return false;
+  }
+
+  if (goCheck.status !== 0) {
+    console.error("FATAL: Go toolchain check failed. The LibraVDB sidecar cannot be built locally.");
+    if (goCheck.stderr?.length) {
+      process.stderr.write(goCheck.stderr);
+    }
+    return false;
+  }
+
+  const result = spawnSync("go", ["build", "-o", dest, "."], {
+    cwd: sidecarDir,
+    stdio: "inherit",
+    env: {
+      ...process.env,
+      GOCACHE: process.env.GOCACHE ?? path.join(os.tmpdir(), "openclaw-memory-libravdb-gocache"),
+    },
+  });
+
+  if (result.error && result.error.code === "ENOENT") {
+    console.error("FATAL: Go toolchain disappeared while building the LibraVDB sidecar.");
+    return false;
+  }
+
+  if (result.status !== 0) {
+    console.error("FATAL: go build failed. The LibraVDB sidecar will not start.");
+    return false;
+  }
+
+  console.log("[openclaw-memory-libravdb] Sidecar installed (local build)");
+  return true;
+}
+
+async function fetchChecksum(url) {
+  try {
+    const response = await fetch(url);
+    if (!response.ok) {
+      return null;
+    }
+    const text = (await response.text()).trim();
+    const match = text.match(/[a-f0-9]{64}/i);
+    return match ? match[0].toLowerCase() : null;
+  } catch {
+    return null;
+  }
+}
+
+async function downloadToFile(url, dest) {
+  const response = await fetch(url);
+  if (!response.ok || !response.body) {
+    throw new Error(`download failed: ${url} (${response.status})`);
+  }
+  const tempPath = `${dest}.tmp`;
+  await pipeline(Readable.fromWeb(response.body), createWriteStream(tempPath));
+  rmSync(dest, { force: true });
+  renameSync(tempPath, dest);
+}
+
+function sha256File(filePath) {
+  return createHash("sha256").update(readFileSync(filePath)).digest("hex");
+}
+
+function formatError(error) {
+  if (error instanceof Error) {
+    return error.message;
+  }
+  return String(error);
+}

package/scripts/setup.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT_DIR="$(cd "$(dirname "$0")/.." && pwd)"
+MODELS_DIR="$ROOT_DIR/.models"
+ORT_ARCHIVE="$MODELS_DIR/onnxruntime/onnxruntime-osx-arm64-1.23.0.tgz"
+ORT_LIB="$MODELS_DIR/onnxruntime/onnxruntime-osx-arm64-1.23.0/lib/libonnxruntime.dylib"
+
+if [[ ! -f "$ORT_ARCHIVE" ]]; then
+  echo "ONNX Runtime archive not found: $ORT_ARCHIVE" >&2
+  exit 1
+fi
+
+if [[ ! -f "$ORT_LIB" ]]; then
+  echo "Unpacking ONNX Runtime..."
+  tar -xzf "$ORT_ARCHIVE" -C "$MODELS_DIR/onnxruntime/"
+  echo "Done: $ORT_LIB"
+else
+  echo "ONNX Runtime already unpacked."
+fi