hippo-memory 1.8.1 → 1.10.0

package/README.md

@@ -85,6 +85,38 @@ hippo recall "data pipeline issues" --budget 2000
 
 ---
 
+### What's new in v1.10.0
+
+- **Server and lifecycle hardening.** Closes the `TODOS.md` "server / lifecycle hardening" cluster (deferred follow-ups from the v0.37 server-mode work, the v0.40 security pass, and the A3 envelope review), six items in all. `detectServer` is now async and confirms a recorded server is genuinely this hippo process by matching a `/health` `started_at` before the CLI routes to it (H1). The pidfile carries a `schema` version (L3). `hippo serve` refuses to start when a live peer already serves the hippoRoot (H3). The 413 over-cap-body path closes the socket instead of draining the rest (M3). A `HIPPO_REQUIRE_SERVER` env knob turns a missing server into a loud error instead of a silent direct-mode fallback that discards `HIPPO_API_KEY` (H2). And `hippo forget --archive --reason` gives raw, append-only memories a real removal path via `archiveRaw` instead of a misleading "not found" (A3).
+- **Reviewed via the dev-framework chain.** self-review, an independent code review, a cross-model codex (gpt-5.5) pass, and a security pass. Four findings were fixed before ship: the `forget --archive` server-routing bypass, an unbounded `/health` response parse, a timeout that wrongly unlinked a busy server's pidfile, and pidfile-url validation against off-box redirection. Full suite green: 216 files, 1557 tests.
+
+### What's new in v1.9.3
+
+- **Reranker review-tail patch.** Closes the three follow-ups raised on PR #25: `src/rerankers/llm.ts` now wires `AbortController` + `setTimeout` around the fetch (default 30 s, overridable via `HIPPO_LLM_RERANKER_TIMEOUT_MS`) so recall never hangs on a wedged endpoint; `src/rerankers/cross-encoder.ts` emits a single `console.warn` on first identity-fallback per process so silent fallback no longer masquerades as a working reranker; the orphan `RerankSignals` type (sole consumer retracted in v1.9.1) is removed at both the re-export and the definition.
+- **Version alignment.** `package.json` bumped 1.8.1 → 1.9.3. v1.9.0 / v1.9.1 / v1.9.2 were on-master research milestones never published to npm; v1.9.3 is the first published `1.9.x` release and carries the cumulative scope from F6 (rerankers) through F13 (chunk-per-turn) plus the F10 HARD RETRACTION.
+- **Mechanism cumulative-null status unaffected.** Per `docs/RETRACTION.md:94-113`. No `src/` change in this patch touches the dlPFC goal-stack mechanism. **This release does not re-assert the retracted −10pp magnitude.**
+
+### What's new in v1.9.2
+
+- **F13 chunk-per-turn LongMemEval R@5 = 86.8 on oracle (Gate-B PASS).** Plan F13 (`docs/evals/2026-05-12-r5-track6-chunk-per-turn-prereg.md`) addresses the structural pathology that limited every prior LongMemEval track (F8–F12): sessions in `data/longmemeval_oracle.json` are ~14k chars median (~3,500 tokens), but the embedders we can reach (MiniLM, BGE-base, multilingual-e5-large) cap at 512–514 tokens. Every prior track embedded only the first ~2 turns of each 12-turn session and truncated the rest. F13 replaces session-level embedding with turn-level embedding (10,866 turns over the 940 oracle sessions, max-pool by `session_id` at retrieval). Gate-A PASS (10,866 turns, all 940 sessions covered, 768-dim normalized). **Gate-B PASS:** F13 + F9 sub-agent rerank stack R@5 = 86.8 on `data/longmemeval_oracle.json` (threshold ≥ 83.2 = F11+F9 deployable best 78.2 + 5pp; margin 3.6). R@1 = 70.8, R@10 = 90.2, R@20 = 93.4.
+- **Roadmap target met (oracle split).** R@5 ≥ 85% was NON-binding per every prior prereg; observed 86.8 on `data/longmemeval_oracle.json` as of this release. Descriptive characterisation; not a re-assertion of any retracted magnitude.
+- **Split-mismatch with gbrain (unchanged).** `longmemeval_oracle` carries 3 sessions per haystack; gbrain v0.28.8's 97.60 figure is on `longmemeval_s_cleaned` (~40 sessions per haystack) with OpenAI `text-embedding-3-large@1536`. Both HF Hub and OpenAI API are host-blocked from this sandbox (verified 2026-05-12). F13's 86.8 is NOT directly comparable to gbrain's 97.60.
+- **F12 retracted.** Plan F12 (`docs/evals/2026-05-11-r5-track5-e5-large-top100-prereg.md`) vendored `intfloat/multilingual-e5-large` and widened the candidate pool to top-100. Gate-A PASS; Gate-B FAIL with best variant R@5 = 78.8 (threshold 83.2). HARD RETRACTION executed: `hippo_store2/` reverted to BGE-base; the `prefixFor` / `preferredBackend` dispatch helpers stay in `src/embeddings.ts` per the dispatch-shape carve-out (they return the legacy behaviour for non-e5 models).
+- **No `src/` changes in v1.9.2.** F13 is implemented as `benchmarks/longmemeval/chunk_per_turn_{embed,retrieve}.mjs` and reuses F11/F12's existing dispatch helpers. The cumulative-null status of the dlPFC goal-stack mechanism (`docs/RETRACTION.md:94-113`) is unaffected. **This release does not re-assert the retracted −10pp magnitude.**
+
+### What's new in v1.9.1
+
+- **F10 features-reranker retraction.** Plan F10 (`docs/plans/2026-05-11-r5-track3-richer-ingest.md`) tested whether populating entry-level signals via 19 Claude-sub-agent invocations would let the features reranker move R@5 above features-default + 5pp on LongMemEval. Observed: features-enriched R@5 = 59.2 vs features-default R@5 = 75.8 (same bge-base embedding model), a 21.6pp shortfall against the binding gate. Per the prereg's HARD RETRACTION clause, `src/rerankers/features.ts` + its test + its micro-fixture + its dispatcher case are removed in v1.9.1. The Track 2 cross-encoder and Track 3 LLM-rerank skeletons are preserved. **This release does not re-assert the retracted −10pp magnitude.** Per `docs/RETRACTION.md`.
+- **F11 embedding upgrade tested and documented (not shipped as default).** Plan F11 (`docs/plans/2026-05-11-r5-track4-embedding-upgrade.md`) swapped `Xenova/all-MiniLM-L6-v2` for `BAAI/bge-base-en-v1.5` (768-dim, CLS pooling). Gate-A PASS; Gate-B FAIL (R@5 = 77.0% vs threshold 81.8%). The `poolingFor` per-model dispatch in `src/embeddings.ts` and the `--model` flag in `scripts/fetch_embedding_model.mjs` ship; MiniLM remains the project default.
+- **Cross-track R@5 status (as of v1.9.1):** F8 hybrid tuning (MiniLM) 76.8, F9 v2 sub-agent LLM rerank (MiniLM) 78.0, F11 bge-base baseline 77.0, F11+F9 stack (BGE-base + sub-agent rerank) 78.2 — cross-track best at v1.9.1 — F10 features-enriched (retracted) 59.2. Roadmap target R@5 ≥ 85% was NOT MET at v1.9.1. NON-binding per each prereg. *(Superseded in v1.9.2 by F13 + F9 stack R@5 = 86.8 on oracle.)*
+
+### What's new in v1.9.0
+
+- **F6 reranker hardening shipped.** New `RerankerFn` seam in `hybridSearch` with three reranker tracks: Track 1 features (`MemoryEntry`-level signals, no external deps), Track 2 cross-encoder (MS-MARCO MiniLM via optional `@xenova/transformers`, identity-fallback on load failure), Track 3 LLM (env-gated skeleton against an OpenAI-compatible endpoint). Opt in via `hippo recall --reranker <name>`.
+- **Workload-validity verdicts on the LongMemEval sweep** (`docs/evals/2026-05-10-f6-reranker-result.md`, prereg `docs/evals/2026-05-10-f6-reranker-prereg.md`): Gate-A (firing rate, binding) PASS for the features track, PASS-with-caveat for cross-encoder (500/500 invocations all took the identity-fallback branch — HF model download was blocked in the test environment, so this is NOT a real cross-encoder evaluation). Gate-B (hyperparameter discrimination, binding) FAIL — features_topk{20,50,100} produced byte-identical R@K, so no per-hyperparameter R@5 effect is claimed.
+- **Roadmap R@5 ≥ 85% target NOT met on the workload tested.** Observed R@5 = 75.4% (features, all three top-K settings) and 75.6% (baseline). Per the prereg this is descriptive characterisation, not a binding gate; the mechanism ships, and a real attempt at the target requires either a real cross-encoder evaluation (HF access) or a richer ingest path that populates entry-level reranker signals.
+- **This release does not re-assert the retracted −10pp magnitude.** Per `docs/RETRACTION.md`. The dlPFC goal-stack cumulative-null status (`docs/RETRACTION.md:94-113`) is independent of this release.
+
 ### What's new in v1.8.1
 
 - **v1.8 prereg's v1.9 LongMemEval cross-validation pre-commitment RETRACTED.** Outside-voice review on two iterations of the v1.9 plan found six structural barriers (canonical harness bypasses the boost path; ingest tag namespace excludes content-derived stems; pushGoal API field mismatch; depth-cap suspension; trigger AND clause unreachable; workload-validity gate ceremonial). Per Root Cause Over Patches, public retraction over re-architecture. **This release does not re-assert the retracted −10pp magnitude.** Per `docs/RETRACTION.md`.

package/bin/hippo.js

@@ -1,2 +1,2 @@
-#!/usr/bin/env node
-import '../dist/cli.js';
+#!/usr/bin/env node
+import '../dist/cli.js';

package/dist/cli.d.ts

@@ -14,7 +14,7 @@
  *   hippo session <log|show|latest|resume|complete>
  *   hippo handoff <create|latest|show>
  *   hippo current <show>
- *   hippo forget <id>
+ *   hippo forget <id> [--archive --reason "<why>"]
  *   hippo inspect <id>
  *   hippo embed [--status]
  *   hippo watch "<command>"

package/dist/cli.js

@@ -14,7 +14,7 @@
  *   hippo session <log|show|latest|resume|complete>
  *   hippo handoff <create|latest|show>
  *   hippo current <show>
- *   hippo forget <id>
+ *   hippo forget <id> [--archive --reason "<why>"]
  *   hippo inspect <id>
  *   hippo embed [--status]
  *   hippo watch "<command>"
@@ -64,6 +64,7 @@ import { runFeatureEval, formatResult, resultToBaseline, detectRegressions } fro
 import { refineStore } from './refine-llm.js';
 import { wmPush, wmRead, wmClear, wmFlush } from './working-memory.js';
 import { multihopSearch } from './multihop.js';
+import { getReranker } from './rerankers/index.js';
 import { computeSalience } from './salience.js';
 import { computeAmbientState, renderAmbientSummary } from './ambient.js';
 import { listDlq, replayDlqEntry } from './connectors/slack/dlq.js';
@@ -116,6 +117,19 @@ function requireInit(hippoRoot) {
         process.exit(1);
     }
 }
+/**
+ * H2: when HIPPO_REQUIRE_SERVER is set, the CLI must not silently fall back to
+ * direct DB mode — a missing server then masks a real misconfiguration (the
+ * configured HIPPO_API_KEY is also silently discarded on fallback). Throws a
+ * clear, actionable error in that case; a no-op when the knob is unset, so
+ * default behaviour is unchanged.
+ */
+function failIfServerRequired(reason) {
+    if (process.env['HIPPO_REQUIRE_SERVER']) {
+        throw new Error(`hippo: HIPPO_REQUIRE_SERVER is set but ${reason}. ` +
+            `Start \`hippo serve\`, or unset HIPPO_REQUIRE_SERVER to allow direct-mode fallback.`);
+    }
+}
 /**
  * Run an HTTP-routed command if a `hippo serve` instance is detected for
  * `hippoRoot`. Returns:
@@ -126,11 +140,16 @@ function requireInit(hippoRoot) {
  *           and the caller should fall back to the direct path.
  *
  * Per the A1 plan footgun #1: stale pidfiles must self-heal, not crash.
+ * H2: when HIPPO_REQUIRE_SERVER is set, both fallback paths throw instead of
+ * returning false, so a missing server fails loudly rather than silently
+ * degrading to direct mode.
  */
 async function runViaServerIfAvailable(hippoRoot, httpFn) {
-    const info = detectServer(hippoRoot);
-    if (!info)
+    const info = await detectServer(hippoRoot);
+    if (!info) {
+        failIfServerRequired('no running server was detected for this hippoRoot');
         return false;
+    }
     const apiKey = process.env['HIPPO_API_KEY'];
     try {
         await httpFn(info, apiKey);
@@ -138,6 +157,7 @@ async function runViaServerIfAvailable(hippoRoot, httpFn) {
     }
     catch (err) {
         if (client.isConnectionRefused(err)) {
+            failIfServerRequired('the server pidfile was stale (connection refused)');
             console.error('hippo: stale server pidfile detected, falling back to direct mode');
             removePidfile(hippoRoot);
             return false;
@@ -701,17 +721,21 @@ async function cmdRecall(hippoRoot, query, flags) {
             physicsConfig: config.physics,
             minResults,
             scope: recallActiveScope,
+            includeSuperseded,
+            asOf,
         });
     }
     else if (hasGlobal) {
         // Use searchBothHybrid for merged results with embedding support
         results = await searchBothHybrid(query, hippoRoot, globalRoot, {
             budget, mmr: mmrEnabled, mmrLambda, localBump, minResults, scope: recallActiveScope, tenantId,
+            includeSuperseded, asOf,
         });
     }
     else {
         results = await hybridSearch(query, localEntries, {
             budget, hippoRoot, mmr: mmrEnabled, mmrLambda, minResults, scope: recallActiveScope,
+            includeSuperseded, asOf,
         });
     }
     // ACC EVC-adaptive recall (RESEARCH.md §PFC.ACC). When the initial top-K is
@@ -818,6 +842,39 @@ async function cmdRecall(hippoRoot, query, flags) {
         })
             .sort((a, b) => b.score - a.score);
     }
+    // F6 reranker pass (docs/plans/2026-05-10-f6-reranker-hardening.md). When
+    // --reranker <name> is set, look up the reranker fn from the registry
+    // (src/rerankers/index.ts) and apply it to the top-K candidates. The
+    // reranker reorders (and may rescale) results; the post-budget set is
+    // returned. Default off; opt-in via --reranker <cross-encoder|llm>. The
+    // structurally similar --rerank-utility block above is the OFC MVP and is
+    // independent — both can run in the same recall, with --rerank-utility
+    // applied first. Available rerankers: cross-encoder, llm (see
+    // src/rerankers/index.ts). The Track 1 `features` reranker was removed in
+    // v1.9.1 per the F10 HARD RETRACTION; it is no longer a valid value.
+    const rerankerName = flags['reranker'] !== undefined ? String(flags['reranker']).trim() : '';
+    if (rerankerName) {
+        const rerankerFn = getReranker(rerankerName);
+        if (rerankerFn) {
+            const topK = flags['reranker-top-k'] !== undefined
+                ? parseInt(String(flags['reranker-top-k']), 10)
+                : 50;
+            const head = results.slice(0, topK);
+            const tail = results.slice(topK);
+            const rerankInput = head.map((r, i) => ({ ...r, preRerankRank: i + 1 }));
+            const reranked = await rerankerFn(query, rerankInput, { topK });
+            // Copy rerankScore into score so downstream blocks (--goal, goal-stack,
+            // salience) that sort by `r.score` honor the reranker's order rather
+            // than unwinding it. Original score is preserved on rerankScore's
+            // input, but downstream sorters key on `score`.
+            const withPostRank = reranked.map((r, i) => ({
+                ...r,
+                score: r.rerankScore,
+                postRerankRank: i + 1,
+            }));
+            results = [...withPostRank, ...tail];
+        }
+    }
     // dlPFC goal-conditioned recall MVP (RESEARCH.md §PFC.dlPFC). When --goal
     // <tag> is set, memories whose `tags` array contains the goal tag receive
     // a 1.5x score boost and results are re-sorted. The full dlPFC spec
@@ -2325,20 +2382,49 @@ function cmdOutcome(hippoRoot, flags) {
     }
     console.log(`Applied ${good ? 'positive' : 'negative'} outcome to ${updated} memor${updated === 1 ? 'y' : 'ies'}`);
 }
-function cmdForget(hippoRoot, id) {
+function cmdForget(hippoRoot, id, flags) {
     requireInit(hippoRoot);
     const ctx = {
         hippoRoot,
         tenantId: resolveTenantId({}),
         actor: 'cli',
     };
+    // A3: raw memories (Slack / GitHub connector ingestion) are append-only — a
+    // BEFORE-DELETE trigger aborts any delete. archiveRaw is the sanctioned
+    // removal path; it records ctx.actor as the archiver for provenance.
+    if (flags['archive'] === true) {
+        const reason = typeof flags['reason'] === 'string' ? flags['reason'] : null;
+        if (!reason) {
+            console.error('hippo forget --archive requires --reason "<why>" (recorded on the archive).');
+            process.exit(1);
+        }
+        try {
+            api.archiveRaw(ctx, id, reason);
+            updateStats(hippoRoot, { forgotten: 1 });
+            console.log(`Archived ${id}`);
+        }
+        catch (err) {
+            console.error(`Could not archive ${id}: ${err instanceof Error ? err.message : String(err)}`);
+            process.exit(1);
+        }
+        return;
+    }
     try {
         api.forget(ctx, id);
         updateStats(hippoRoot, { forgotten: 1 });
         console.log(`Forgot ${id}`);
     }
-    catch {
-        console.error(`Memory not found: ${id}`);
+    catch (err) {
+        const msg = err instanceof Error ? err.message : String(err);
+        if (/append-only/i.test(msg)) {
+            // The delete was refused by the append-only trigger — this is a raw
+            // memory, not a missing one. Point the user at the archive path.
+            console.error(`Cannot forget ${id}: it is a raw, append-only memory. ` +
+                `Archive it instead: hippo forget ${id} --archive --reason "<why>"`);
+        }
+        else {
+            console.error(`Memory not found: ${id}`);
+        }
         process.exit(1);
     }
 }
@@ -4649,6 +4735,12 @@ Commands:
                            = score * (0.5 + 0.5 * strength) * (1 - cost_factor)
                            where cost_factor = min(0.3, tokens / 10000). Re-sorts
                            results by utility. Default off. RESEARCH.md §PFC.OFC.
+    --reranker <name>      Apply a reranker pass after retrieval
+                           (cross-encoder|llm). Looks up the named
+                           reranker from src/rerankers/index.ts and re-orders
+                           the top-K candidates. Default unset (no reranker).
+                           See docs/plans/2026-05-10-f6-reranker-hardening.md.
+    --reranker-top-k <n>   Cap candidates passed to the reranker (default 50).
     --goal <tag>           dlPFC goal-conditioned recall: memories tagged with
                            the goal tag get a 1.5x score boost and results are
                            re-sorted. Default off. RESEARCH.md §PFC.dlPFC.
@@ -4797,6 +4889,8 @@ Commands:
     current show           Active task + recent session events (default)
       --json               Output as JSON
   forget <id>              Force remove a memory
+    --archive              Archive a raw (append-only) memory instead of deleting
+    --reason "<why>"       Reason recorded on the archive (required with --archive)
   inspect <id>             Show full memory detail
   embed                    Embed all memories for semantic search
     --status               Show embedding coverage
@@ -5239,19 +5333,24 @@ async function main() {
                 console.error('Please provide a memory ID.');
                 process.exit(1);
             }
-            const routed = await runViaServerIfAvailable(hippoRoot, async (info, apiKey) => {
-                try {
-                    await client.forget(info.url, apiKey, id);
-                    console.log(`Forgot ${id}`);
-                }
-                catch (err) {
-                    console.error(err.message);
-                    process.exit(1);
-                }
-            });
-            if (routed)
-                break;
-            cmdForget(hippoRoot, id);
+            // A3: --archive is a direct-DB operation (raw archival via archiveRaw);
+            // the HTTP forget route does not carry it, so archive requests always
+            // take the direct path rather than silently no-op'ing over a server.
+            if (flags['archive'] !== true) {
+                const routed = await runViaServerIfAvailable(hippoRoot, async (info, apiKey) => {
+                    try {
+                        await client.forget(info.url, apiKey, id);
+                        console.log(`Forgot ${id}`);
+                    }
+                    catch (err) {
+                        console.error(err.message);
+                        process.exit(1);
+                    }
+                });
+                if (routed)
+                    break;
+            }
+            cmdForget(hippoRoot, id, flags);
             break;
         }
         case 'inspect': {