@psiclawops/hypermem 0.9.3 → 0.9.5

package/docs/TUNING.md

@@ -1,6 +1,6 @@
 # hypermem Tuning Guide
 
-Configuration reference for operators and agents. All settings are optional. The recommended install path writes a starter `config.json` with an explicit lightweight embedding choice and a declarative baseline compositor profile. Tune from that verified baseline, not from guesswork.
+Configuration reference for operators and agents. All settings are optional. The recommended install path writes a starter `config.json` with the 0.9.4 recall-friendly standard context profile and `embedding.provider: "none"` so first install has no external service dependency. Tune from that verified baseline, then opt into Ollama or a hosted embedding provider when you want semantic vector recall.
 
 Config lives in `~/.openclaw/hypermem/config.json` (takes effect on gateway restart) or is passed programmatically via `HyperMem.create()`:
 
@@ -56,7 +56,7 @@ That said, not every deployment needs full context richness. If you're running a
 
 | Situation | Preset | Reason |
 |---|---|---|
-| First install, trying it out | `light` | Minimal overhead, easy to reason about |
+| First install, trying it out | Installed baseline | FTS5/no-embedding mode verifies cleanly before provider tuning |
 | Single conversational agent | `standard` | Richer memory without fleet overhead |
 | Multi-agent fleet, long-running sessions | `full` | Full continuity, keystone recall, cross-session context |
 | CI pipelines, one-shot tasks | Custom light | Disable indexer, keep only history |
@@ -114,13 +114,38 @@ Estimated context per turn: **12–35k tokens** on a 200k model (lower on smalle
 
 ### Standard setup
 
-The default out-of-the-box configuration. All memory layers active with balanced caps. Good for most single-agent deployments.
+The standard semantic configuration. All memory layers are active with 0.9.4 recall preservation turned on. Good for most single-agent deployments after the baseline FTS5 install is verified.
 
 ```json
 {
+  "embedding": {
+    "provider": "ollama",
+    "model": "nomic-embed-text",
+    "dims": 768,
+    "dimensions": 768,
+    "ollamaUrl": "http://localhost:11434"
+  },
   "compositor": {
-    "budgetFraction": 0.703,
+    "turnBudget": {
+      "budgetFraction": 0.6,
+      "minContextFraction": 0.18
+    },
+    "warming": {
+      "protectedFloorEnabled": true,
+      "shapedWarmupDecay": true
+    },
+    "adjacency": {
+      "enabled": true,
+      "boostMultiplier": 1.3,
+      "maxLookback": 5,
+      "maxClockDeltaMin": 10,
+      "evictionGuardMessages": 3,
+      "evictionGuardTokenCap": 4000
+    },
+    "budgetFraction": 0.6,
     "contextWindowReserve": 0.25,
+    "targetBudgetFraction": 0.5,
+    "warmHistoryBudgetFraction": 0.45,
     "maxFacts": 28,
     "maxHistoryMessages": 250,
     "maxCrossSessionContext": 0,
@@ -129,13 +154,29 @@ The default out-of-the-box configuration. All memory layers active with balanced
     "hyperformProfile": "standard"
   },
   "indexer": {
+    "enabled": true,
     "factExtractionMode": "tiered",
     "periodicInterval": 300000
   }
 }
 ```
 
-Estimated context per turn: **35–80k tokens** on a 200k model. This is what ships if you don't touch the config.
+Estimated context per turn: **35–80k tokens** on a 200k model. `hypermem-install` writes these context and recall-preservation knobs when no existing config is present, but leaves embeddings disabled by default for installation safety. Existing config files are preserved, so run `hypermem-doctor --fix-plan` after upgrades to see whether older installs are missing the 0.9.4 recall-surface knobs.
+
+### 0.9.4 recall preservation knobs
+
+These settings prevent the regression class that motivated 0.9.4: the direct antecedent or topic-bearing warm context being trimmed away while the system still had enough usable budget.
+
+| Knob | Default | What it protects | Health signal |
+|---|---:|---|---|
+| `compositor.turnBudget.budgetFraction` | `0.6` | Keeps the active turn budget below the old over-warming posture | `hypermem-status --master` recall surface line |
+| `compositor.turnBudget.minContextFraction` | `0.18` | Guarantees a minimum context allocation on constrained windows | `hypermem-doctor --fix-plan` |
+| `compositor.warming.protectedFloorEnabled` | `true` | Prevents afterTurn trim from evicting bootstrap/warmup floor content before elevated pressure | `node test/afterturn-protected-floor.mjs` |
+| `compositor.warming.shapedWarmupDecay` | `true` | Uses topic-bearing turn count instead of blind fixed-turn decay | `node test/adaptive-lifecycle.mjs` |
+| `compositor.adjacency.enabled` | `true` | Boosts recent antecedents during fused retrieval | `node test/hybrid-retrieval.mjs` |
+| `compositor.adjacency.evictionGuardTokenCap` | `4000` | Caps the literal antecedent guard so it cannot consume the whole budget | `node test/compositor.mjs` |
+
+Do not tune these off to save tokens until you have evidence. Lower `maxFacts`, `maxCrossSessionContext`, or wiki caps first. Disabling adjacency or protected warming is a correctness trade, not a cost preset.
 
 ### Full performance setup
 
@@ -208,7 +249,7 @@ const custom = mergeProfile('standard', {
 });
 ```
 
-Start with `light`. Move up when you need richer context and have the headroom.
+Start from the installed baseline. Move to `light` only when you need lower first-turn cost, and move to `standard` or `full` when richer context is worth the token spend.
 
 ---
 

package/install.sh

@@ -166,7 +166,7 @@ stage_runtime() {
 write_minimal_config_if_missing() {
   echo -e "\n${BOLD}  Config check${NC}"
   if $DRY_RUN; then
-    dryrun "would preserve existing config or write lightweight starter config: $CONFIG_FILE"
+    dryrun "would preserve existing config or write FTS5 starter config: $CONFIG_FILE"
     return
   fi
 
@@ -175,66 +175,11 @@ write_minimal_config_if_missing() {
     return
   fi
 
+  local default_config="$INSTALL_DIR/node_modules/@psiclawops/hypermem/assets/default-config.json"
+  [[ -f "$default_config" ]] || die "missing packaged default config: $default_config"
   mkdir -p "$(dirname "$CONFIG_FILE")"
-  cat > "$CONFIG_FILE" <<'JSON'
-{
-  "contextWindowSize": 128000,
-  "contextWindowReserve": 0.25,
-  "deferToolPruning": false,
-  "verboseLogging": false,
-  "contextWindowOverrides": {},
-  "warmCacheReplayThresholdMs": 120000,
-  "subagentWarming": "light",
-  "embedding": {
-    "provider": "none"
-  },
-  "compositor": {
-    "budgetFraction": 0.55,
-    "reserveFraction": 0.25,
-    "historyFraction": 0.4,
-    "memoryFraction": 0.4,
-    "defaultTokenBudget": 90000,
-    "maxHistoryMessages": 500,
-    "maxFacts": 25,
-    "maxExpertisePatterns": 6,
-    "maxCrossSessionContext": 4000,
-    "maxTotalTriggerTokens": 4000,
-    "maxRecentToolPairs": 3,
-    "maxProseToolPairs": 10,
-    "warmHistoryBudgetFraction": 0.27,
-    "contextWindowReserve": 0.25,
-    "dynamicReserveTurnHorizon": 5,
-    "dynamicReserveMax": 0.5,
-    "dynamicReserveEnabled": true,
-    "keystoneHistoryFraction": 0.15,
-    "keystoneMaxMessages": 12,
-    "keystoneMinSignificance": 0.5,
-    "targetBudgetFraction": 0.50,
-    "enableFOS": true,
-    "enableMOD": true,
-    "hyperformProfile": "standard",
-    "wikiTokenCap": 500,
-    "zigzagOrdering": true
-  },
-  "eviction": {
-    "enabled": true,
-    "imageAgeTurns": 2,
-    "toolResultAgeTurns": 4,
-    "minTokensToEvict": 200,
-    "keepPreviewChars": 120
-  },
-  "maintenance": {
-    "periodicInterval": 300000,
-    "maxActiveConversations": 5,
-    "recentConversationCooldownMs": 30000,
-    "maxCandidatesPerPass": 200
-  },
-  "vectorStore": {
-    "enabled": false
-  }
-}
-JSON
-  success "lightweight starter config written: $CONFIG_FILE"
+  cp "$default_config" "$CONFIG_FILE"
+  success "FTS5 starter config written: $CONFIG_FILE"
 }
 
 verify_stage() {

package/memory-plugin/dist/index.js

@@ -18,6 +18,7 @@ import { definePluginEntry, emptyPluginConfigSchema } from 'openclaw/plugin-sdk/
 import { matchTriggers, TRIGGER_REGISTRY } from '@psiclawops/hypermem';
 import path from 'path';
 import fs from 'fs/promises';
+import fsSync from 'fs';
 import os from 'os';
 import { fileURLToPath } from 'url';
 // ─── HyperMem singleton ────────────────────────────────────────
@@ -283,6 +284,196 @@ function createMemorySearchManager(hm, agentId, workspaceDir) {
         },
     };
 }
+// ─── history.query agent tool ───────────────────────────────────
+const HISTORY_QUERY_MODES = [
+    'runtime_chain',
+    'transcript_tail',
+    'tool_events',
+    'by_topic',
+    'by_context',
+    'cross_session',
+];
+const HISTORY_QUERY_TOOL_PARAMETERS = {
+    type: 'object',
+    additionalProperties: false,
+    properties: {
+        mode: {
+            type: 'string',
+            enum: HISTORY_QUERY_MODES,
+            description: 'History query mode. cross_session is scoped to the current agent by default.',
+        },
+        sessionKey: {
+            type: 'string',
+            description: 'Optional session key. Defaults to the active session when available.',
+        },
+        conversationId: {
+            type: 'number',
+            description: 'Optional direct conversation id. Must belong to the current agent.',
+        },
+        contextId: {
+            type: 'number',
+            description: 'Required for by_context mode. Must belong to the current agent.',
+        },
+        topicId: {
+            type: 'string',
+            description: 'Required for by_topic mode.',
+        },
+        limit: {
+            type: 'number',
+            description: 'Optional result limit. HyperMem clamps this to the hard cap for the selected mode.',
+        },
+        minMessageId: {
+            type: 'number',
+            description: 'Optional lower message id bound for supported modes.',
+        },
+        since: {
+            type: 'string',
+            description: 'Optional ISO timestamp lower bound for cross_session mode.',
+        },
+        includeArchived: {
+            type: 'boolean',
+            description: 'Allow archived/forked contexts for by_context mode.',
+        },
+        includeToolPayloads: {
+            type: 'boolean',
+            description: 'Return raw tool payloads for tool_events. Requires owner context; default is redacted.',
+        },
+    },
+    required: ['mode'],
+};
+function asRecord(value) {
+    return value && typeof value === 'object' && !Array.isArray(value)
+        ? value
+        : {};
+}
+function optionalString(params, key) {
+    const value = params[key];
+    return typeof value === 'string' && value.trim() ? value.trim() : undefined;
+}
+function optionalNumber(params, key) {
+    const value = params[key];
+    if (typeof value !== 'number' || !Number.isFinite(value))
+        return undefined;
+    return value;
+}
+function optionalBoolean(params, key) {
+    return typeof params[key] === 'boolean' ? params[key] : undefined;
+}
+function isHistoryQueryMode(value) {
+    return typeof value === 'string' && HISTORY_QUERY_MODES.includes(value);
+}
+function historyQueryTelemetryEnabled() {
+    return process.env.HYPERMEM_TELEMETRY === '1';
+}
+function emitHistoryQueryTelemetry(event) {
+    if (!historyQueryTelemetryEnabled())
+        return;
+    const telemetryPath = process.env.HYPERMEM_TELEMETRY_PATH || './hypermem-telemetry.jsonl';
+    try {
+        fsSync.appendFileSync(telemetryPath, `${JSON.stringify(event)}\n`, 'utf8');
+    }
+    catch {
+        // Telemetry must never break the tool path.
+    }
+}
+function createHistoryQueryTool(ctx) {
+    return {
+        name: 'history_query',
+        label: 'history.query',
+        description: [
+            'Query HyperMem SQLite-backed message history for the current agent.',
+            'Use this when exact conversation state is needed instead of semantic recall.',
+            'Modes: runtime_chain, transcript_tail, tool_events, by_topic, by_context, cross_session.',
+            'Tool payloads are redacted by default; raw payloads require owner context.',
+        ].join(' '),
+        parameters: HISTORY_QUERY_TOOL_PARAMETERS,
+        displaySummary: 'Query HyperMem message history',
+        async execute(_toolCallId, rawParams) {
+            const started = Date.now();
+            const params = asRecord(rawParams);
+            const mode = params.mode;
+            const agentId = ctx.agentId || 'main';
+            const includeToolPayloads = optionalBoolean(params, 'includeToolPayloads') === true;
+            const baseTelemetry = {
+                ts: new Date().toISOString(),
+                agentId,
+                hasSessionKey: Boolean(optionalString(params, 'sessionKey') ?? ctx.sessionKey),
+                hasConversationId: optionalNumber(params, 'conversationId') !== undefined,
+                includeToolPayloads,
+            };
+            if (!isHistoryQueryMode(mode)) {
+                emitHistoryQueryTelemetry({
+                    event: 'history-query',
+                    status: 'error',
+                    ...baseTelemetry,
+                    durationMs: Date.now() - started,
+                    errorCode: 'invalid-mode',
+                });
+                throw new Error(`history.query: mode must be one of ${HISTORY_QUERY_MODES.join(', ')}`);
+            }
+            if (includeToolPayloads && !ctx.senderIsOwner) {
+                emitHistoryQueryTelemetry({
+                    event: 'history-query',
+                    status: 'error',
+                    mode,
+                    ...baseTelemetry,
+                    durationMs: Date.now() - started,
+                    errorCode: 'owner-required',
+                });
+                throw new Error('history.query: includeToolPayloads requires owner context');
+            }
+            const query = {
+                agentId,
+                mode,
+                sessionKey: optionalString(params, 'sessionKey') ?? ctx.sessionKey,
+                conversationId: optionalNumber(params, 'conversationId'),
+                contextId: optionalNumber(params, 'contextId'),
+                topicId: optionalString(params, 'topicId'),
+                limit: optionalNumber(params, 'limit'),
+                minMessageId: optionalNumber(params, 'minMessageId'),
+                since: optionalString(params, 'since'),
+                includeArchived: optionalBoolean(params, 'includeArchived'),
+                includeToolPayloads,
+            };
+            for (const [key, value] of Object.entries(query)) {
+                if (value === undefined)
+                    delete query[key];
+            }
+            try {
+                const hm = await getHyperMem();
+                const result = hm.queryHistory(query);
+                emitHistoryQueryTelemetry({
+                    event: 'history-query',
+                    status: 'ok',
+                    mode,
+                    ...baseTelemetry,
+                    messageCount: result.messages.length,
+                    truncated: Boolean(result.truncated),
+                    redacted: Boolean(result.redacted),
+                    durationMs: Date.now() - started,
+                });
+                const summary = `history.query ${mode}: ${result.messages.length} message(s)`
+                    + (result.truncated ? ' (truncated)' : '')
+                    + (result.redacted ? ' (redacted)' : '');
+                return {
+                    content: [{ type: 'text', text: JSON.stringify(result, null, 2) }],
+                    details: { status: 'ok', summary, result },
+                };
+            }
+            catch (err) {
+                emitHistoryQueryTelemetry({
+                    event: 'history-query',
+                    status: 'error',
+                    mode,
+                    ...baseTelemetry,
+                    durationMs: Date.now() - started,
+                    errorCode: 'query-failed',
+                });
+                throw err;
+            }
+        },
+    };
+}
 // ─── Manager cache ──────────────────────────────────────────────
 // One manager per agentId; closed on plugin dispose.
 const _managers = new Map();
@@ -294,6 +485,7 @@ export default definePluginEntry({
     kind: 'memory',
     configSchema: emptyPluginConfigSchema(),
     register(api) {
+        api.registerTool((ctx) => createHistoryQueryTool(ctx), { name: 'history_query', optional: true });
         api.registerMemoryCapability({
             runtime: {
                 async getMemorySearchManager(params) {

package/memory-plugin/openclaw.plugin.json

@@ -3,11 +3,208 @@
   "enabledByDefault": false,
   "kind": "memory",
   "activation": {
-    "onCapabilities": ["memory"]
+    "onCapabilities": [
+      "memory"
+    ]
   },
   "configSchema": {
     "type": "object",
     "additionalProperties": false,
-    "properties": {}
+    "properties": {
+      "hyperMemPath": {
+        "type": "string"
+      },
+      "dataDir": {
+        "type": "string"
+      },
+      "contextWindowSize": {
+        "type": "integer",
+        "minimum": 1
+      },
+      "contextWindowReserve": {
+        "type": "number",
+        "minimum": 0,
+        "maximum": 0.5
+      },
+      "deferToolPruning": {
+        "type": "boolean"
+      },
+      "verboseLogging": {
+        "type": "boolean"
+      },
+      "warmCacheReplayThresholdMs": {
+        "type": "integer",
+        "minimum": 0
+      },
+      "subagentWarming": {
+        "type": "string",
+        "enum": [
+          "full",
+          "light",
+          "off"
+        ]
+      },
+      "contextWindowOverrides": {
+        "type": "object",
+        "additionalProperties": {
+          "type": "object",
+          "additionalProperties": false,
+          "properties": {
+            "contextTokens": {
+              "type": "integer",
+              "minimum": 1
+            },
+            "contextWindow": {
+              "type": "integer",
+              "minimum": 1
+            }
+          }
+        }
+      },
+      "compositor": {
+        "type": "object",
+        "additionalProperties": true
+      },
+      "eviction": {
+        "type": "object",
+        "additionalProperties": false,
+        "properties": {
+          "enabled": {
+            "type": "boolean"
+          },
+          "imageAgeTurns": {
+            "type": "integer",
+            "minimum": 0
+          },
+          "toolResultAgeTurns": {
+            "type": "integer",
+            "minimum": 0
+          },
+          "minTokensToEvict": {
+            "type": "integer",
+            "minimum": 0
+          },
+          "keepPreviewChars": {
+            "type": "integer",
+            "minimum": 0
+          }
+        }
+      },
+      "embedding": {
+        "type": "object",
+        "additionalProperties": false,
+        "properties": {
+          "provider": {
+            "type": "string",
+            "enum": [
+              "none",
+              "ollama",
+              "openai",
+              "gemini"
+            ]
+          },
+          "ollamaUrl": {
+            "type": "string"
+          },
+          "openaiApiKey": {
+            "type": "string"
+          },
+          "openaiBaseUrl": {
+            "type": "string"
+          },
+          "geminiApiKey": {
+            "type": "string"
+          },
+          "geminiBaseUrl": {
+            "type": "string"
+          },
+          "geminiIndexTaskType": {
+            "type": "string"
+          },
+          "geminiQueryTaskType": {
+            "type": "string"
+          },
+          "queryInputType": {
+            "type": "string"
+          },
+          "documentInputType": {
+            "type": "string"
+          },
+          "queryPrefix": {
+            "type": "string"
+          },
+          "documentPrefix": {
+            "type": "string"
+          },
+          "model": {
+            "type": "string"
+          },
+          "dimensions": {
+            "type": "integer",
+            "minimum": 1
+          },
+          "timeout": {
+            "type": "integer",
+            "minimum": 1
+          },
+          "batchSize": {
+            "type": "integer",
+            "minimum": 1
+          }
+        }
+      },
+      "reranker": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": [
+          "provider"
+        ],
+        "properties": {
+          "provider": {
+            "type": "string",
+            "enum": [
+              "zeroentropy",
+              "openrouter",
+              "local",
+              "none"
+            ]
+          },
+          "minCandidates": {
+            "type": "integer",
+            "minimum": 0
+          },
+          "maxDocuments": {
+            "type": "integer",
+            "minimum": 1
+          },
+          "topK": {
+            "type": "integer",
+            "minimum": 1
+          },
+          "timeoutMs": {
+            "type": "integer",
+            "minimum": 1
+          },
+          "zeroEntropyApiKey": {
+            "type": "string"
+          },
+          "zeroEntropyModel": {
+            "type": "string"
+          },
+          "openrouterApiKey": {
+            "type": "string"
+          },
+          "openrouterModel": {
+            "type": "string"
+          },
+          "ollamaUrl": {
+            "type": "string"
+          },
+          "ollamaModel": {
+            "type": "string"
+          }
+        }
+      }
+    }
   }
 }

package/memory-plugin/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@psiclawops/hypermem-memory",
-  "version": "0.9.3",
+  "version": "0.9.5",
   "description": "HyperMem memory plugin for OpenClaw \u2014 bridges HyperMem retrieval into the memory slot",
   "type": "module",
   "main": "dist/index.js",
@@ -40,7 +40,7 @@
     "typecheck": "tsc --noEmit"
   },
   "dependencies": {
-    "@psiclawops/hypermem": "0.9.2"
+    "@psiclawops/hypermem": "0.9.5"
   },
   "devDependencies": {
     "openclaw": "*",