@psiclawops/hypermem 0.8.4 → 0.9.0

package/CHANGELOG.md

@@ -2,6 +2,39 @@
 
 All notable changes to hypermem are documented here.
 
+## 0.9.0 - adaptive context lifecycle
+
+- **Adaptive lifecycle is now production behavior.** Compose, afterTurn, recall, trim, compaction, and eviction share the same pressure-band policy across bootstrap, warmup, steady, elevated, high, and critical states.
+- **Smart recall and adaptive eviction landed.** `/new` and confident topic shifts widen recall, high-pressure turns gate recall down, and topic-centroid-guided eviction activates only at elevated pressure or worse.
+- **Lifecycle telemetry is release-gated.** Trim and compose reports classify lifecycle bands, divergence, and metadata-only topic signal without exposing topic names, prompt text, document text, or user content.
+- **Deterministic topic evidence replaces live-sample gating.** The 0.9.0 topic-bearing compose gate is covered by deterministic fixtures and report tests, while live topic-bearing samples remain future tuning evidence only.
+- **Forked-context integration is wired.** Forked subagent children inherit bounded parent hot-window context and start warmup or steady instead of cold bootstrap unless `/new` is explicit.
+- **Vector coverage is repaired.** Active facts, knowledge, and eligible episodes reached 100% vector coverage before the release candidate validation pass.
+
+## 0.8.8 - release hardening, diagnostics, lifecycle visibility
+
+- **Release packaging aligned across packages.** Core, hypercompositor, and memory plugin versions align at 0.8.8, with version parity validation and bump-script hardening to prevent stale plugin dependencies or lockfile drift.
+- **Installer path simplified.** The shell installer now follows the npm-first path, stages the runtime with `hypermem-install`, preserves existing config/data, backs up existing staged runtime when confirmed, and prints merge-safe OpenClaw activation commands.
+- **Integration validation documented.** `docs/INTEGRATION_VALIDATION.md` defines the install state machine, fresh install checks, upgrade checks, package dry-run validation, and common integration failure signatures.
+- **Diagnostics documented.** `docs/DIAGNOSTICS.md` covers `hypermem-status`, `hypermem-model-audit`, compose/trim reports, version parity, release-path checks, runtime logs, warm restore diagnostics, adaptive lifecycle diagnostics, and the runtime diagnostics API allowlist blocker.
+- **Adaptive lifecycle visibility landed.** The pure lifecycle policy kernel, compose diagnostics, and afterTurn gradient cap are wired while leaving recall breadth, eviction tuning, and telemetry tuning deferred to 0.9.0.
+- **Warm restore hardening is included.** Snapshot integrity, repaired restore paths, provider/parity gates, repair notices, and repair-depth caps are covered by validation guidance and release tests.
+- **Reranker and embedding fixes are included.** Reranker wiring, ZeroEntropy endpoint handling, sqlite-vec native runtime packaging, and provider/model diagnostics are part of the 0.8.8 operational release train.
+
+## 0.8.6 — docs cleanup, model audit config parsing, validator fix
+
+- **`hypermem-model-audit` now understands object-shaped OpenClaw model config.** It correctly reads `model.primary` plus fallback arrays from modern agent config instead of reporting an empty model set.
+- **Docs release notes no longer drift on package version text.** README and INSTALL now point operators to the next npm release instead of hard-coding conflicting minimum versions.
+- **README and INSTALL roles are clearer.** INSTALL is explicitly the canonical operator bring-up guide, while README stays as the shorter install-state overview.
+- **Docs validator false positives fixed.** `scripts/validate-docs.mjs` now checks only `openclaw config set plugins.slots.* ...` lines, so `config get` examples no longer trigger bogus plugin-id mismatch warnings.
+
+## 0.8.5 — provider/model-aware failover detection, release parity
+
+- **Provider + model identity is now tracked explicitly in HyperCompositor model state.** Mid-session routing changes are detected on the full `provider/model` key, not budget alone, so `github-copilot/claude-sonnet-4-6` and `anthropic/claude-sonnet-4-6` are treated as different operational envelopes.
+- **Downshift detection now keys off provider/model-aware state.** Budget-downshift handling still stays conservative, but verbose logs now surface provider swaps, model swaps, and budget deltas clearly during `context:assemble`.
+- **Install docs now declare the full operator path.** README, INSTALL.md, and TUNING.md now separate staging from activation, document install states explicitly, add merge-safe wiring guidance, and clarify what healthy-but-empty looks like on a first run.
+- **0.8.5 release parity.** Package versions are aligned for the next publish while preserving the npm-first installer and merge-safe config guidance landed from Pylon's install review.
+
 ## 0.8.4 — compaction fence fix, install-path fixes, zod runtime packaging
 
 - **Compaction fence tail preservation fixed.** The recent-tail preservation fix is included so compaction no longer drops the protected recent tail during fence advancement.

package/INSTALL.md

@@ -1,5 +1,7 @@
 # hypermem — Installation Guide
 
+This is the canonical install procedure. Keep README shorter and point operators here for the full bring-up path.
+
 ## Prerequisites
 
 - **Node.js 22+** (uses built-in `node:sqlite`)
@@ -15,22 +17,67 @@ openclaw config get gateway # should return gateway config, not an error
 
 If `gateway status` shows "disabled" or "not configured", complete OpenClaw onboarding first. `openclaw gateway restart` only works when the gateway service is already set up. On a brand-new OpenClaw install that has never been started, you need `openclaw gateway start` (or the full onboarding flow) before installing plugins.
 
+## Non-OpenClaw usage
+
+HyperMem can also be used as a normal Node.js library without OpenClaw plugins. This mode is useful for tests, custom agents, migration tooling, and experiments with the memory/composition API.
+
+```bash
+npm install @psiclawops/hypermem
+```
+
+```typescript
+import { HyperMem } from '@psiclawops/hypermem';
+import { join } from 'node:path';
+import { homedir } from 'node:os';
+
+const hm = await HyperMem.create({
+  dataDir: join(homedir(), '.openclaw', 'hypermem'),
+  embedding: { provider: 'none' },
+});
+
+await hm.recordUserMessage('my-agent', 'session-1', 'Hello');
+const composed = await hm.compose({
+  agentId: 'my-agent',
+  sessionKey: 'session-1',
+  prompt: 'Hello',
+  tokenBudget: 4000,
+  provider: 'anthropic',
+});
+```
+
+No gateway, plugin load path, or OpenClaw config is required in library mode. OpenClaw-specific setup starts below.
+
 ## Quick Start
 
-> **Release note:** if the npm package you installed does not contain `hypermem-install` or `install:runtime`, you are on an older public release. Use the source-clone path in this guide or wait for `0.8.4+`.
+This guide is deliberately declarative. Follow the steps in order and verify each install state before moving on.
+
+> **Release note:** if the npm package you installed does not contain `hypermem-install`, `install:runtime`, and `hypermem-model-audit`, you are on an older public release. Use the source-clone path in this guide or wait for the next npm release.
 
 ```bash
 npm install @psiclawops/hypermem
 npx hypermem-install
 ```
 
-`hypermem-install` stages the plugin runtime into `~/.openclaw/plugins/hypermem`. It does **not** modify your OpenClaw config and does **not** restart the gateway.
+`hypermem-install` stages the plugin runtime into `~/.openclaw/plugins/hypermem`. It does **not** modify your OpenClaw config and does **not** restart the gateway. That means a successful `npx hypermem-install` is **not** a completed install. It is only a completed staging step.
+
+The shell installer is now a thin npm-first wrapper around this same path:
+
+```bash
+curl -fsSL https://raw.githubusercontent.com/PsiClawOps/hypermem/main/install.sh | bash
+```
+
+It installs the npm package into `~/.hypermem`, backs up any existing staged runtime when confirmed, stages the runtime with `hypermem-install`, writes a lightweight starter config only if no config exists, and prints merge-safe OpenClaw activation commands. It does not edit OpenClaw config and does not restart the gateway.
+
+Release validation details live in [docs/INTEGRATION_VALIDATION.md](./docs/INTEGRATION_VALIDATION.md). Diagnostic surfaces live in [docs/DIAGNOSTICS.md](./docs/DIAGNOSTICS.md).
 
 > **Prerequisites:** OpenClaw must be installed and onboarded. Run `openclaw gateway status` to confirm. If the gateway is not configured, complete OpenClaw setup first.
 >
 > **Config merge warning:** if you already have values in `plugins.load.paths` or `plugins.allow`, merge them instead of overwriting blindly.
 
-Create the config directory and set the embedding provider:
+Create the config directory and write the current recommended fresh-install starter config. This does 2 things:
+
+1. sets `embedding.provider` to `none` so a clean install does not try to use Ollama by default
+2. applies the current recommended lean compositor settings so fresh installs start from the same OpenClaw and HyperMem guidance we want operators to use
 
 ```bash
 mkdir -p ~/.openclaw/hypermem
@@ -38,11 +85,41 @@ cat > ~/.openclaw/hypermem/config.json <<'JSON'
 {
   "embedding": {
     "provider": "none"
+  },
+  "compositor": {
+    "budgetFraction": 0.55,
+    "contextWindowReserve": 0.25,
+    "targetBudgetFraction": 0.50,
+    "warmHistoryBudgetFraction": 0.27,
+    "maxFacts": 25,
+    "maxHistoryMessages": 500,
+    "maxCrossSessionContext": 4000,
+    "maxRecentToolPairs": 3,
+    "maxProseToolPairs": 10,
+    "keystoneHistoryFraction": 0.15,
+    "keystoneMaxMessages": 12,
+    "wikiTokenCap": 500
   }
 }
 JSON
 ```
 
+If you want a lighter or richer memory profile later, adjust from this baseline using the tuning guidance below instead of starting from the older code defaults.
+
+### Install states
+
+Treat the install as 5 explicit states:
+
+| State | Meaning | Success signal |
+|---|---|---|
+| 1. Package installed | npm package is present | `npm install` succeeds |
+| 2. Runtime staged | plugin payload copied into `~/.openclaw/plugins/hypermem` | `npx hypermem-install` succeeds and files exist |
+| 3. OpenClaw wired | config points at HyperMem plugins | `plugins.load.paths` and `plugins.slots.*` are set correctly |
+| 4. Runtime loaded | gateway restarted and loaded both plugins | `openclaw plugins list` shows both plugins loaded |
+| 5. Runtime active | HyperMem is actually composing turns | logs show `hypermem initialized` and compose activity |
+
+Do not stop at state 2 and call the install done.
+
 Wire both plugins into OpenClaw:
 
 **Step 1: Check your existing config (mandatory — do this before wiring).**
@@ -52,7 +129,7 @@ openclaw config get plugins.load.paths
 openclaw config get plugins.allow
 ```
 
-Note any existing values. You must include them in the commands below.
+Note any existing values. You must include them in the commands below. If `plugins.allow` is null, unset, or empty, skip the allowlist step entirely.
 
 **Step 2: Wire plugins.**
 
@@ -83,30 +160,104 @@ OpenClaw loads the plugin runtime from `~/.openclaw/plugins/hypermem/`.
 
 ### Verification checkpoints
 
-1. **Build verified**
-   - root build succeeds
-   - `plugin` build succeeds
-   - `memory-plugin` build succeeds
+Run the installed-system doctor first:
+
+```bash
+hypermem-doctor --fix-plan
+```
+
+`hypermem-doctor` is read-only. It inspects OpenClaw config, HyperMem config, plugin wiring, recommended OpenClaw runtime settings, data directory shape, runtime plugin load state, and active model context-window risk. It prints exact `openclaw config set ...` commands when something needs review. It does not edit config or restart the gateway.
+
+Expected result after a complete install: no required failures. Recommendation warnings should be reviewed before production use, especially context-window warnings for GPT/OpenAI-compatible/local gateways.
+
+Walk the install state machine explicitly if you need a manual check:
+
+1. **Runtime staged**
+   ```bash
+   ls ~/.openclaw/plugins/hypermem
+   ```
+   Expected: `dist`, `plugin`, `memory-plugin`, and package metadata exist.
 
 2. **Wiring verified**
-   - OpenClaw accepts `plugins.load.paths`
-   - slots are set to `hypercompositor` and `hypermem`
-   - gateway restart succeeds
+   ```bash
+   openclaw config get plugins.load.paths
+   openclaw config get plugins.slots.contextEngine
+   openclaw config get plugins.slots.memory
+   openclaw config get plugins.allow
+   ```
+   Expected: HyperMem paths are present, slots point to `hypercompositor` and `hypermem`, and allowlist entries were merged only if the install already used an allowlist.
+
+3. **Runtime loaded**
+   ```bash
+   openclaw plugins list
+   ```
+   Expected: both `hypercompositor` and `hypermem` show as loaded.
+
+4. **Runtime healthy**
+   ```bash
+   hypermem-status --health
+   ```
+   Expected on fresh installs: the plugin may report `no sessions ingested` or empty counts. That means healthy but unused, not broken.
+
+5. **Model budget audited**
+   ```bash
+   hypermem-model-audit --strict
+   ```
+   Expected: every configured model either matches a known context-window pattern or has explicit `contextWindowOverrides`. For GPT/OpenAI-compatible/local gateways, prefer explicit overrides unless logs prove OpenClaw passes a correct runtime `tokenBudget`.
+
+6. **Runtime active**
+
+   Send a message to any agent, then verify:
+
+   ```bash
+   openclaw logs --limit 100 | grep -E 'hypermem|context-engine|budget source'
+   ```
+
+   Expected lightweight-mode lines:
+   - `[hypermem] hypermem initialized`
+   - `[hypermem] Embedding provider: none — semantic search disabled, using FTS5 fallback`
+   - `[hypermem:compose]`
+   - `budget source: runtime tokenBudget=...` or `budget source: contextWindowOverrides[...]` for the active model
+
+If you see a fallback like `falling back to default engine "legacy"`, the install is **not** fully active yet even if staging and wiring succeeded.
+
+---
 
-3. **Runtime verified active**
+## Upgrade Path
 
-Send a message to any agent, then verify:
+Upgrades preserve the HyperMem data directory and existing config. The runtime staging directory is replaceable.
 
 ```bash
-openclaw logs --limit 100 | grep -E 'hypermem|context-engine'
+cp -a ~/.openclaw/plugins/hypermem ~/.openclaw/plugins/hypermem.backup.$(date +%Y%m%d-%H%M%S) 2>/dev/null || true
+npm install @psiclawops/hypermem@latest
+npx hypermem-install
+openclaw gateway restart
 ```
 
-Expected lightweight-mode lines:
-- `[hypermem] hypermem initialized`
-- `[hypermem] Embedding provider: none — semantic search disabled, using FTS5 fallback`
-- `[hypermem:compose]`
+Then validate:
+
+```bash
+openclaw plugins list
+openclaw logs --limit 100 | grep -E 'hypermem|context-engine|falling back'
+hypermem-status --health
+hypermem-model-audit --strict
+```
 
-If you see a fallback like `falling back to default engine "legacy"`, the install is **not** fully active yet even if the build and wiring steps succeeded.
+Pass criteria:
+
+- `~/.openclaw/hypermem/config.json` is preserved unless the operator edits it intentionally.
+- existing `~/.openclaw/hypermem/agents/*/messages.db` files remain present.
+- `openclaw plugins list` shows both `hypercompositor` and `hypermem`.
+- logs do not show `falling back to default engine "legacy"`.
+- health output is clean, or reports only healthy-empty state on unused installs.
+
+Rollback disables HyperMem without deleting data:
+
+```bash
+openclaw config set plugins.slots.contextEngine legacy
+openclaw config set plugins.slots.memory none
+openclaw gateway restart
+```
 
 ---
 
@@ -139,7 +290,7 @@ Everything runs in-process on SQLite memory databases. No external database serv
 
 > **Package versions:** the root package (`hypermem`) and the two plugins (`hypercompositor`, `hypermem-memory`) are versioned independently. Plugin versions trail the core by one minor version when no plugin-facing API changes ship in a release — this is expected.
 
-The **embedding layer** (L3 semantic search) requires a configured provider. Without one, hypermem falls back to FTS5 keyword matching. This is functional but degrades recall quality. See [Setup Styles](#setup-styles) below.
+The **embedding layer** (L3 semantic search) requires a configured provider. Without one, hypermem falls back to FTS5 keyword matching. This is functional but degrades recall quality. Fresh installs should start with `provider: "none"` explicitly so the runtime behavior is intentional and easy to verify, then upgrade later. See [Setup Styles](#setup-styles) below.
 
 ---
 
@@ -428,6 +579,8 @@ See [Embedding Providers](#embedding-providers) above.
 
 ### Step 4 — Restart and verify
 
+Do not start tuning before this section passes. If HyperMem is not loaded and composing, the next problem is installation, not tuning.
+
 ```bash
 openclaw gateway restart
 ```
@@ -659,6 +812,16 @@ Solo installs can skip this.
 
 ## Token Budget Tuning
 
+**Lookup paths for operators and agents.** To inspect the active config at any time:
+
+```bash
+cat ~/.openclaw/hypermem/config.json
+openclaw config get plugins.entries.hypercompositor.config
+openclaw config get plugins.slots.contextEngine
+```
+
+Resolution order is: (1) `plugins.entries.hypercompositor.config` in `openclaw.json`, (2) `~/.openclaw/hypermem/config.json`, (3) code defaults. See [docs/TUNING.md](./docs/TUNING.md) for the full tuning reference.
+
 These settings live in `~/.openclaw/hypermem/config.json` under the `compositor` key. All fields are optional — omit any knob to get the code-level default. Gateway restart required after changes.
 
 The recommended starting config for a standard single-agent deployment is intentionally lean on turn-1 warming. Semantic recall and fact triggers fire against each incoming message, so topic-relevant context surfaces as the conversation takes shape. This produces a steadier pressure profile than aggressive pre-loading and avoids the warm→trim→compact cycling you see when every session starts near the top of the budget.
@@ -728,6 +891,10 @@ HyperMem sizes the token budget from the model string using an internal pattern
 - `warmHistoryBudgetFraction` × *wrong budget* → wrong warm load on first turn
 - Trim tiers and compaction thresholds fire against the wrong ceiling
 
+**Treat this as mandatory tuning, not optional polish.** If the runtime does not pass the real model budget, you must supply the correct window yourself. In practice that means OpenAI-compatible surfaces, Codex/OpenRouter-style providers, custom provider prefixes, and local model gateways often need an explicit override because the model string alone is not a trustworthy source of truth.
+
+**When you know both numbers, set both.** Each override entry accepts `contextTokens` and `contextWindow`. HyperMem resolves from `contextTokens` first, then `contextWindow`, and the validator enforces `contextTokens <= contextWindow`. Setting both makes your intended usable budget explicit and documents the full advertised window for future operators.
+
 The two symptoms that indicate window-detection failure:
 
 1. **Undersized window detected** (you have a 200k model, HyperMem thinks it's 90k): every turn warms near the top of the misdetected budget, trim fires constantly, semantic recall and facts get starved. You see continuous `warm→trim→compact` cycling even on short sessions.
@@ -743,6 +910,16 @@ The two symptoms that indicate window-detection failure:
 
 If you see `fallback contextWindowSize` for your model, detection failed and you need an override.
 
+**Packaged audit helper.** HyperMem now ships `hypermem-model-audit`, which inspects your configured models plus any existing `contextWindowOverrides` and flags models that are running on risky autodetect paths:
+
+```bash
+hypermem-model-audit
+hypermem-model-audit --strict
+hypermem-model-audit --models openai-codex/gpt-5.4,ollama/llama-3.3-70b
+```
+
+Use it during install and after model changes. `--strict` exits non-zero if a model is missing explicit metadata or is only partially overridden.
+
 **Apply an override.** Add a `contextWindowOverrides` block to `~/.openclaw/hypermem/config.json`. The key is `"provider/model"` as it appears in your agent's model string (lowercase, exact match):
 
 ```json
@@ -752,9 +929,10 @@ If you see `fallback contextWindowSize` for your model, detection failed and you
     "contextWindowReserve": 0.25,
     "warmHistoryBudgetFraction": 0.27,
     "contextWindowOverrides": {
-      "ollama/llama-3.3-70b":      { "contextTokens": 131072 },
-      "copilot-local/custom-sft":  { "contextTokens": 32768 },
-      "vllm/qwen3-coder-ft":       { "contextTokens": 262144 }
+      "ollama/llama-3.3-70b":      { "contextTokens": 131072, "contextWindow": 131072 },
+      "openai-codex/gpt-5.4":      { "contextTokens": 200000, "contextWindow": 200000 },
+      "copilot-local/custom-sft":  { "contextTokens": 32768,  "contextWindow": 32768 },
+      "vllm/qwen3-coder-ft":       { "contextTokens": 262144, "contextWindow": 262144 }
     }
   }
 }
@@ -769,6 +947,8 @@ Resolution order, highest-to-lowest priority:
 
 Gateway restart required after editing overrides. Invalid override entries (malformed keys, impossible ranges, empty values) are dropped on load with a warning; the sanitizer will not let a bad override poison the resolver.
 
+**OpenAI-family warning.** Do not assume `gpt-*`, `openai/*`, `openai-codex/*`, or OpenAI-compatible hosted endpoints will always arrive with correct runtime budget metadata. If you cannot prove the runtime is logging `runtime tokenBudget=...` for that exact model, add an explicit override and verify it in logs before tuning anything else.
+
 **Interaction with warming and trimming.** Once the correct window is in place:
 
 - First-turn warm load = `detectedWindow × budgetFraction × (1 - contextWindowReserve) × warmHistoryBudgetFraction`