@psiclawops/hypermem 0.8.4 → 0.9.0

package/docs/DIAGNOSTICS.md

@@ -0,0 +1,205 @@
+# HyperMem Diagnostics
+
+HyperMem diagnostics are split into operator CLIs, validation scripts, and runtime logs. Use this page to decide which surface proves which part of the system.
+
+## Diagnostic surfaces
+
+| Surface | Command | Proves |
+|---|---|---|
+| Health summary | `hypermem-status --health` | data dir, SQLite health, basic runtime state |
+| Operator dashboard | `hypermem-status --master` | concise fleet-facing status summary |
+| JSON status | `hypermem-status --json` | machine-readable database and runtime state |
+| Model audit | `hypermem-model-audit --strict` | active models have known context-window behavior or overrides |
+| Memory benchmark | `hypermem-bench --iterations 1000 --warmup 50 --agent <agent>` | local dataset access latency with p50/p95/p99 timings |
+| Compose report | `node scripts/compose-report.mjs` | direct compositor slot selection, budget decisions, diagnostics fields |
+| Trim report | `node scripts/trim-report.mjs` | trim events, cache invalidation, pressure behavior |
+| Version parity | `npm run validate:version-parity` | package versions and plugin dependencies are release-aligned |
+| Release path | `npm run validate:release-path` | build plus plugin pipeline gateway path |
+| Docs/config validation | `npm run validate:docs && npm run validate:config` | documented commands and config surfaces match code |
+
+## `hypermem-status`
+
+Primary uses:
+
+- confirm data directory exists
+- confirm SQLite databases are readable
+- confirm vectors and message stores are not obviously corrupt
+- distinguish healthy-empty installs from broken installs
+
+Examples:
+
+```bash
+hypermem-status --health
+hypermem-status --master
+hypermem-status --agent forge --master
+hypermem-status --json
+```
+
+Healthy fresh install behavior may include empty counts or `no sessions ingested`. That is not a failure. A fresh install becomes active only after the gateway loads the plugins and an agent turn runs.
+
+Failure signals:
+
+| Signal | Meaning | Next check |
+|---|---|---|
+| missing data dir | HyperMem has not initialized or `HYPERMEM_DATA_DIR` points elsewhere | verify plugin load and gateway logs |
+| SQLite open failure | permissions, corruption, or incompatible data dir | check filesystem owner and run SQLite integrity check |
+| vector dimension mismatch | embedding provider changed without rebuild or migration | inspect provider/model/dimension config |
+| stale index time | background indexing not running or no new data | check gateway logs for embed/index messages |
+
+## `hypermem-model-audit`
+
+Model audit checks whether HyperMem can size the context budget for active models.
+
+```bash
+hypermem-model-audit
+hypermem-model-audit --strict
+hypermem-model-audit --models openai-codex/gpt-5.4,ollama/llama-3.3-70b
+```
+
+Use strict mode in release validation. A strict failure means the model should get an explicit `contextWindowOverrides` entry before treating pressure diagnostics as authoritative.
+
+## Memory access benchmark
+
+`hypermem-bench` measures storage access speed against the operator's local HyperMem dataset, normally `~/.openclaw/hypermem`.
+
+```bash
+hypermem-bench --iterations 1000 --warmup 50 --agent main
+hypermem-bench --iterations 1000 --warmup 50 --data-dir /path/to/hypermem
+```
+
+It reports min, average, p50, p95, p99, and max latency for the data paths present in that install, including message hot-path lookups, session/conversation lookup, message FTS, facts, episodes, topics, fleet records, and doc chunks.
+
+Use this for local validation of README speed claims. Results depend on hardware, database size, selected agent, and which optional surfaces are enabled. Vector embedding generation and hosted reranker latency are not part of these SQLite access timings; they are configured separately and should be measured against the chosen provider.
+
+## Runtime logs
+
+After wiring and restart:
+
+```bash
+openclaw logs --limit 100 | grep -E 'hypermem|context-engine|falling back'
+```
+
+Healthy startup and compose lines include:
+
+```text
+[hypermem] hypermem initialized
+[hypermem] Embedding provider: none - semantic search disabled, using FTS5 fallback
+[hypermem:compose]
+```
+
+Hard failure lines include:
+
+```text
+falling back to default engine "legacy"
+Cannot find module 'zod'
+Cannot find module 'sqlite-vec'
+```
+
+If OpenClaw falls back to `legacy`, HyperMem is not composing prompts. Treat that as install failure, not degraded success.
+
+## Compose diagnostics
+
+Compose diagnostics prove that the compositor is selecting, budgeting, and exposing runtime fields.
+
+Key fields to inspect:
+
+| Field family | What it proves |
+|---|---|
+| token budget fields | model-aware context window sizing and reserve math |
+| `sessionPressureFraction` and `pressureSource` | unified pressure signal is present |
+| history depth fields | pre-compose history depth estimation is active |
+| prompt placement fields | query-shaped tail and slot placement behavior |
+| adaptive lifecycle fields | lifecycle band and breadcrumb decisions are visible |
+| warm restore fields | snapshot restore, repair, parity, and rollout gates are visible |
+| tool/artifact fields | tool chain preservation and oversize degradation are working |
+
+Direct check:
+
+```bash
+node scripts/compose-report.mjs
+```
+
+Release gate coverage:
+
+```bash
+node test/unified-pressure-signal.mjs
+node test/history-depth-estimator.mjs
+node test/sprint4-prompt-placement.mjs
+node test/adaptive-lifecycle.mjs
+node --test test/composition-snapshot-integrity.test.mjs
+node --test test/composition-snapshot-store.test.mjs
+```
+
+## Warm restore diagnostics
+
+Warm restore validation must prove 4 things:
+
+1. snapshots are written with integrity metadata
+2. restored snapshots preserve required system repair notices
+3. repaired snapshots are capped and cannot become restore sources
+4. rollout gates fall back to cold rewarm when parity or provider checks fail
+
+Release gate coverage:
+
+```bash
+node --test test/composition-snapshot-integrity.test.mjs
+node --test test/composition-snapshot-store.test.mjs
+node test/repair-tool-pairs.mjs
+```
+
+## Adaptive lifecycle diagnostics
+
+0.9.0 ships the adaptive lifecycle behavior set: shared pressure-band policy, adaptive recall breadth, adaptive eviction ordering, lifecycle telemetry, and metadata-only topic-signal reporting.
+
+Validate:
+
+```bash
+node test/adaptive-lifecycle.mjs
+node test/unified-pressure-signal.mjs
+```
+
+Expected proof:
+
+- lifecycle band is computed
+- compose diagnostics expose lifecycle fields
+- telemetry JSONL includes `lifecycle-policy` events for `compose.preRecall` and `compose.eviction`
+- `node scripts/trim-report.mjs <telemetry.jsonl>` reports lifecycle policy counts, band counts, and divergence turns
+- `trim-report.mjs` and `compose-report.mjs` classify compose topic signal with metadata-only fields:
+  - `present`
+  - `absent-no-active-topic`
+  - `absent-stamping-incomplete`
+  - `intentionally-suppressed`
+  - `unknown`
+- topic-signal reports use booleans, enums, counts, percentages, and reason codes only; they must not emit topic names, prompt text, document text, or user content
+- afterTurn gradient cap limits pressure spikes
+- threshold tuning remains deferred unless a populated telemetry baseline shows a specific threshold or behavior defect
+
+Topic-signal interpretation:
+
+- `present` means an active topic was resolved and stamped history was available.
+- `absent-no-active-topic` means compose had no active topic source to use.
+- `absent-stamping-incomplete` means an active topic existed but stamped inputs were missing or insufficient.
+- `intentionally-suppressed` means schema or privacy policy intentionally omitted topic telemetry.
+
+This report path closes the ambiguity from a baseline with no usable assemble topic fields. The 0.9.0 release gate is covered by deterministic metadata-only topic evidence; safe live topic-bearing compose samples remain future tuning evidence before topic-aware threshold changes.
+
+## Runtime diagnostics API allowlist
+
+Verified 2026-04-24 against the installed OpenClaw runtime: `openclaw doctor --non-interactive` no longer reports the bundled public-surface allowlist blocker, and the memory-core runtime facade can load `memory-core/runtime-api.js`.
+
+This is an upstream OpenClaw diagnostics surface, not HyperMem-owned runtime behavior. If the blocker returns, do not patch OpenClaw from this repo. Record the exact failing public-surface path and classify it as `upstream-required` unless the failing surface is HyperMem's own memory plugin diagnostics.
+
+## Release diagnostics checklist
+
+Run this checklist before tagging:
+
+```bash
+npm run validate:version-parity
+npm run validate:docs
+npm run validate:config
+npm run validate:release-path
+npm test
+hypermem-model-audit --strict || true
+```
+
+The local repo may not be the active OpenClaw runtime, so `hypermem-status` and `hypermem-model-audit` should be interpreted against the configured deployment. For package release proof, the required gates are version parity, docs/config validation, release path, tests, and pack dry runs.

package/docs/INTEGRATION_VALIDATION.md

@@ -0,0 +1,186 @@
+# HyperMem Integration Validation
+
+This page is the operator validation contract for HyperMem releases. It describes how the runtime pieces fit together and how to prove each one works before publishing or upgrading a deployment.
+
+## Runtime pieces
+
+| Piece | Package or path | Purpose | Verification |
+|---|---|---|---|
+| Core library | `@psiclawops/hypermem` | SQLite stores, compositor primitives, retrieval, lifecycle policy, diagnostics helpers | `node -e "import('@psiclawops/hypermem')"` from an installed package |
+| Context engine plugin | `~/.openclaw/plugins/hypermem/plugin` | OpenClaw `contextEngine` slot, prompt composition, warming, trimming, adaptive lifecycle diagnostics | `openclaw plugins list` shows `hypercompositor` loaded |
+| Memory plugin | `~/.openclaw/plugins/hypermem/memory-plugin` | OpenClaw memory slot integration for `memory_search` and retrieval surfaces | `openclaw plugins list` shows `hypermem` loaded |
+| Runtime staging tool | `hypermem-install` | Copies package runtime into `~/.openclaw/plugins/hypermem` | staged directory contains `dist`, `plugin`, `memory-plugin`, `bin` |
+| Status CLI | `hypermem-status` | Health, database, vector, and runtime summary | `hypermem-status --health` exits cleanly or reports only healthy-empty state |
+| Model audit CLI | `hypermem-model-audit` | Checks model context-window detection and overrides | `hypermem-model-audit --strict` reports no risky unknown models, or known required overrides |
+
+## Install state machine
+
+Do not treat installation as complete until all 5 states pass.
+
+| State | Meaning | Required proof |
+|---|---|---|
+| 1. Package installed | npm package is available | `npm install @psiclawops/hypermem` succeeds |
+| 2. Runtime staged | package payload is copied into OpenClaw plugin runtime dir | `ls ~/.openclaw/plugins/hypermem` shows `dist`, `plugin`, `memory-plugin`, `bin` |
+| 3. OpenClaw wired | OpenClaw config points at staged plugins | `plugins.load.paths` includes both staged plugin dirs, slots are set |
+| 4. Runtime loaded | Gateway has loaded both plugins | `openclaw plugins list` shows `hypercompositor` and `hypermem` loaded |
+| 5. Runtime active | HyperMem is composing live turns | logs show `[hypermem] hypermem initialized` and `[hypermem:compose]` |
+
+A successful `hypermem-install` proves only state 2. It does not modify OpenClaw config and does not restart the gateway.
+
+## Fresh install validation
+
+Run from a clean shell with OpenClaw already onboarded.
+
+```bash
+node --version                # v22+
+openclaw gateway status       # running or ready
+openclaw config get gateway   # returns config, not an onboarding error
+npm install @psiclawops/hypermem
+npx hypermem-install
+```
+
+Create the lightweight starter config before the first restart:
+
+```bash
+mkdir -p ~/.openclaw/hypermem
+cat > ~/.openclaw/hypermem/config.json <<'JSON'
+{
+  "embedding": { "provider": "none" },
+  "compositor": {
+    "budgetFraction": 0.55,
+    "contextWindowReserve": 0.25,
+    "targetBudgetFraction": 0.50,
+    "warmHistoryBudgetFraction": 0.27,
+    "maxFacts": 25,
+    "maxHistoryMessages": 500,
+    "maxCrossSessionContext": 4000,
+    "maxRecentToolPairs": 3,
+    "maxProseToolPairs": 10,
+    "keystoneHistoryFraction": 0.15,
+    "keystoneMaxMessages": 12,
+    "wikiTokenCap": 500
+  }
+}
+JSON
+```
+
+Wire OpenClaw using merge-safe config changes:
+
+```bash
+openclaw config get plugins.load.paths
+openclaw config get plugins.allow
+
+HYPERMEM_PATHS="["${HOME}/.openclaw/plugins/hypermem/plugin","${HOME}/.openclaw/plugins/hypermem/memory-plugin"]"
+openclaw config set plugins.load.paths "$HYPERMEM_PATHS" --strict-json
+openclaw config set plugins.slots.contextEngine hypercompositor
+openclaw config set plugins.slots.memory hypermem
+
+# Only if plugins.allow already contains an array, append the two ids to that existing array.
+# Do not create a new allowlist just for HyperMem.
+openclaw config set plugins.allow '["existing-plugin","hypercompositor","hypermem"]' --strict-json
+
+openclaw gateway restart
+```
+
+Verify activation:
+
+```bash
+openclaw plugins list
+openclaw logs --limit 100 | grep -E 'hypermem|context-engine'
+hypermem-status --health
+hypermem-model-audit --strict
+```
+
+Expected lightweight signals:
+
+```text
+[hypermem] hypermem initialized
+[hypermem] Embedding provider: none - semantic search disabled, using FTS5 fallback
+[hypermem:compose]
+```
+
+## Upgrade validation
+
+An upgrade must preserve data and config.
+
+```bash
+cp -a ~/.openclaw/plugins/hypermem ~/.openclaw/plugins/hypermem.backup.$(date +%Y%m%d-%H%M%S) 2>/dev/null || true
+npm install @psiclawops/hypermem@latest
+npx hypermem-install
+openclaw gateway restart
+```
+
+Validate after restart:
+
+```bash
+openclaw plugins list
+openclaw logs --limit 100 | grep -E 'hypermem|context-engine|falling back'
+hypermem-status --health
+hypermem-model-audit --strict
+```
+
+Pass criteria:
+
+- existing `~/.openclaw/hypermem/config.json` is unchanged unless the operator edits it intentionally
+- existing `~/.openclaw/hypermem/agents/*/messages.db` files remain present
+- `openclaw plugins list` shows both plugins loaded
+- logs do not show `falling back to default engine "legacy"`
+- health output is clean, or reports only healthy-empty state on unused installs
+
+Rollback:
+
+```bash
+openclaw config set plugins.slots.contextEngine legacy
+openclaw config set plugins.slots.memory none
+openclaw gateway restart
+```
+
+Data under `~/.openclaw/hypermem` is not removed by rollback.
+
+## Release package validation
+
+Before publishing a release, run:
+
+```bash
+npm run validate:version-parity
+npm run validate:docs
+npm run validate:config
+npm run validate:release-path
+npm test
+npm --prefix plugin run build
+npm --prefix memory-plugin run build
+npm pack --dry-run
+npm pack ./plugin --dry-run
+npm pack ./memory-plugin --dry-run
+```
+
+Then test the packed artifact from a temp directory:
+
+```bash
+TMP=$(mktemp -d)
+npm --prefix "$TMP" init -y
+npm --prefix "$TMP" install ./psiclawops-hypermem-*.tgz
+node "$TMP/node_modules/@psiclawops/hypermem/scripts/install-runtime.mjs" "$TMP/runtime"
+find "$TMP/runtime" -maxdepth 3 -type f | sort
+```
+
+The staged runtime must contain:
+
+- `dist/`
+- `plugin/dist/`
+- `memory-plugin/dist/`
+- `bin/hypermem-status.mjs`
+- `bin/hypermem-model-audit.mjs`
+- `node_modules/sqlite-vec` plus native sqlite-vec packages when present
+- `node_modules/zod`
+
+## Integration failure signatures
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| `falling back to default engine "legacy"` | Plugin not loaded, wrong slot, stale runtime, or allowlist collision | verify load paths, slots, allowlist merge, and staged `dist` files |
+| `Cannot find module 'zod'` | runtime staging missed plugin dependency | rerun `hypermem-install`, verify `node_modules/zod` staged |
+| sqlite-vec native load error | native sqlite-vec package missing from staged runtime | rerun staging from a package with sqlite-vec native payload included |
+| `memory_search` bypasses HyperMem | memory slot not set or memory plugin not loaded | set `plugins.slots.memory hypermem`, restart, verify plugins list |
+| status tool reports missing data dir | gateway has not initialized HyperMem yet | restart after wiring and send one agent message |
+| model audit flags unknown context window | active model needs explicit override | add `contextWindowOverrides` in HyperMem config |

package/docs/MIGRATION.md

@@ -11,9 +11,9 @@ All migration examples default to **dry-run** where shown. Add `--apply` only wh
 
 ## Multi-operator deployments (breaking change warning)
 
-> ⚠️ **KL-01: No global-scope write gate (still open as of 0.8.0)**
+> ⚠️ **KL-01: No global-scope write gate (still open as of 0.9.0)**
 >
-> hypermem 0.5.0 ships without a write gate for `scope='global'` facts. In a
+> hypermem 0.9.0 ships without a write gate for `scope='global'` facts. In a
 > **single-operator deployment** (one user, one fleet sharing `library.db`), this
 > is acceptable — agents share context intentionally.
 >
@@ -21,7 +21,7 @@ All migration examples default to **dry-run** where shown. Add `--apply` only wh
 > `library.db`), this is a **breaking isolation risk**: a global-scope write from
 > one operator's agent propagates to all other operators' agents.
 >
-> **Do not run hypermem 0.5.0 in a multi-operator deployment without an external
+> **Do not run hypermem in a multi-operator deployment without an external
 > write gate at the application layer.** The write gate is deferred to 1.0.0
 > where it will be enforced at the library DB level.
 >
@@ -37,6 +37,7 @@ Older versions below 0.5.0 used Redis, so the table keeps that history explicit.
 
 | hypermem version | Main DB schema | Library DB schema | Min Node | External cache |
 |---|---|---|---|---|
+| 0.9.0 | v11 | v19 | 22.0.0 | none, SQLite `:memory:` hot cache |
 | 0.8.0 | v10 | v19 | 22.0.0 | none, SQLite `:memory:` hot cache |
 | 0.7.0 | v7 | v13 | 22.0.0 | none, SQLite `:memory:` hot cache |
 | 0.6.0 | v6 | v12 | 22.0.0 | none, SQLite `:memory:` hot cache |
@@ -46,11 +47,13 @@ Older versions below 0.5.0 used Redis, so the table keeps that history explicit.
 Schema versions are importable for programmatic checks:
 ```ts
 import { SCHEMA_COMPAT, HYPERMEM_COMPAT_VERSION } from 'hypermem';
-// HYPERMEM_COMPAT_VERSION = '0.8.0'
-// SCHEMA_COMPAT = { compatVersion: '0.8.0', mainSchema: 10, librarySchema: 19 }
+// HYPERMEM_COMPAT_VERSION = ENGINE_VERSION, for example '0.9.0'
+// SCHEMA_COMPAT = { compatVersion: ENGINE_VERSION, mainSchema: 11, librarySchema: 19 }
 ```
 
 If your DBs are behind, run:
 ```bash
-node scripts/migrate-legacy-sessions.mjs --apply
+# Current releases migrate schemas on open. Back up the data dir first.
+cp -a ~/.openclaw/hypermem ~/.openclaw/hypermem.pre-upgrade.$(date +%Y%m%d-%H%M%S)
+openclaw gateway restart
 ```