hippocamp 0.2.2 → 0.2.4

package/AGENTS.md

@@ -22,12 +22,14 @@ This repository is a local-first MCP package for Git-backed agent memory. Keep c
 
 ## Repository Map
 
-- `scripts/hippocamp.cjs`: small CLI wrapper for `dream`, `mcp`, `install-codex`, `install-claude`, and `install-grok`.
+- `scripts/hippocamp.cjs`: small CLI wrapper for `install` (every agent CLI found), `dream`, `mcp`, `install-codex`, `install-claude`, and `install-grok`.
 - `scripts/hippocamp-dream.cjs`: offline Dream CLI for scheduled compaction of curated project wake-up files.
 - `scripts/hippocamp-memory.cjs`: local filesystem-backed Hippocamp memory helpers for MCP use.
 - `scripts/hippocamp-mcp.cjs`: local stdio MCP server for Hippocamp memory tools.
 - `scripts/eval-lagoon.cjs`: read-only retrieval check against real Lagoon cases in `<Lagoon root>/evals/retrieval-cases.json`.
+- `scripts/eval-qa.cjs`: LongMemEval-S answer-accuracy check (reader and grader models) for Hippocamp snippets against the labeled answer sessions.
 - `scripts/eval-retrieval.cjs`: retrieval-only LongMemEval-S check for `searchMemory` against a BM25 reference.
+- `scripts/install-common.cjs`: shared installer helpers: the MCP launch command and Lagoon repo preparation.
 - `scripts/install-claude.cjs`: one-step installer for Claude Code MCP setup and user-level `CLAUDE.md` guidance.
 - `scripts/install-codex.cjs`: one-step installer for the Hippocamp skill and MCP server in Codex.
 - `scripts/install-grok.cjs`: one-step installer for the Hippocamp skill, Grok home rules, and MCP server in Grok Build.
@@ -74,10 +76,11 @@ When changing code, use the smallest verification that proves the change:
 - Run `npm run mcp:smoke` when changing memory reads, root resolution, or wake-up behavior.
 - Run `npm run dream -- --help` when changing Dream CLI arguments.
 - Run `npm run eval:retrieval -- --limit 100` when changing search ranking or snippets; report Recall@5 against the BM25 row and snippet evidence. Also run `npm run eval:lagoon` when a Lagoon is available.
+- Run `npm run eval:qa -- --limit 3 --dry-run` when changing `scripts/eval-qa.cjs`; a full run calls paid models, so run it only when asked.
 
 ## Notes For Future Agents
 
 - The cloud/API version is intentionally deferred.
 - Do not reintroduce Next.js, GitHub API token auth, or GitHub App auth for the local MVP.
 - Keep Dream minimal: scheduled coherent compaction, current/open curated files plus bounded thread evidence, one PR per project.
-- The local install story should stay: install MCP, point at Lagoon, rely on normal Git auth.
+- The local install story should stay: `npx hippocamp@latest install`, a local Lagoon repo created if missing, normal Git auth. Installers never create remotes or push.

package/README.md

@@ -6,13 +6,77 @@
   <img src="./assets/brand/hippocamp-mascot.png" alt="Hippocamp mascot" width="220" />
 </p>
 
-Hippocamp is local Git-backed memory for AI coding agents.
+Agent memory you can `git log`.
 
-Agents can edit large codebases, run tests, and push commits, but every new session still starts with amnesia. Project decisions, user preferences, open threads, and "why we did this" context get scattered across chat history, scratch notes, and PR comments. The result is repeated explanations, stale assumptions, and agents rediscovering the same facts instead of continuing the work.
+Hippocamp gives Claude Code, Codex, and Grok Build one shared memory: plain Markdown in a private Git repo you own. Every session starts by waking up from it, and every decision an agent records is a commit you can read, diff, and revert.
 
-Hippocamp gives Codex, Claude Code, and other MCP clients a small shared memory surface they can wake up from, update, and sync. The memory is plain Markdown in a private Git repo you own.
+Open source (MIT). No account, no API key, no cloud service: it runs on your machine and works the moment it is installed. Your memory never leaves your computer except to a Git remote you choose. No database, no vector store, nothing to pay for.
 
-No database. No hosted memory service. No vector store. No separate token broker.
+```bash
+npx hippocamp@latest install
+```
+
+This installs Hippocamp into every supported agent CLI it finds, creates `~/.lagoon` as a local Git repo if it does not exist, and tells you how to add a private remote. Restart your agent, and it calls `wake_up` at the start of each task.
+
+## Why Not Built-In Memory?
+
+Claude Code and Codex now ship their own memory. Use Hippocamp when you want:
+
+- **One memory for every agent.** Claude Code, Codex, and Grok read and write the same repo. Built-in memory stays inside one tool.
+- **History you can audit.** Every memory change is a Git commit with an author, a diff, and a timestamp. Roll back a bad memory like bad code.
+- **Files you own.** Markdown in your own private repo, readable without any UI, portable across machines through your normal Git credentials.
+- **Nothing to sign up for.** Open source, local, free. No account, no API key, no vendor that can change terms or shut down.
+- **Reviewed compaction.** Optional Dream mode summarizes memory offline and proposes the result as a pull request.
+
+## Results
+
+Every number below comes from a script in this repo, so you can rerun it. We report where Hippocamp falls short too.
+
+### Finding the right memory
+
+LongMemEval-S, all 470 answerable questions, top 5 results, no model involved (`npm run eval:retrieval`).
+
+| Measure | Hippocamp | BM25 reference |
+| --- | ---: | ---: |
+| Right session in top 5 (Recall@5) | 90.5% | 91.4% |
+| At least one right session in top 5 (Hit@5) | 96.0% | 96.8% |
+| Returned text contains the answer turn | 87.9% | n/a |
+| Text returned per query | 5.9k chars | 65.5k chars |
+| Search latency, cold index | 42 ms | n/a |
+
+BM25 returns whole sessions; Hippocamp returns the matching paragraphs of each result.
+
+### Answering with it
+
+100 LongMemEval-S questions, reader Claude Sonnet 5, grader gpt-4o with the official LongMemEval grading prompts (`npm run eval:qa`).
+
+| What the reader gets | Text per question | Answer accuracy |
+| --- | ---: | ---: |
+| Hippocamp, top 5 | 6.7k chars | 71% |
+| Hippocamp, top 10 | 12.9k chars | 77% |
+| The correct sessions, given directly | 28.8k chars | 90% |
+
+The same reader reaches 90% with the correct sessions, so the remaining gap is retrieval, not the reader.
+
+### Real coding questions
+
+53 questions about real work in a personal Lagoon across 13 projects (`npm run eval:lagoon`). This is our own test set, kept in the private Lagoon.
+
+| Measure | Result |
+| --- | ---: |
+| Right event in top 5, 43 answerable questions | 42 / 43 |
+| No results for 10 questions no memory is about | 10 / 10 |
+
+### Why less text matters
+
+Memory results share the agent's context with code, diffs, and tool output, and agents search several times per task. About 6k characters per search is roughly 1.5k tokens; whole sessions would be about 16k. Fewer tokens make each turn cheaper and faster, and keep the relevant lines from being buried. Less text is only useful when the answer is still in it, which is why we report answer accuracy next to text size.
+
+### Limits
+
+- Recall@5 is just under the BM25 reference, by less than 1 point, with 11 times less text.
+- No embeddings or semantic search: a memory written in other words can be missed. On LongMemEval-S, 9 of 470 questions miss with no word in common with the answer.
+- Answer accuracy is measured on 100 questions, about plus or minus 4 points.
+- Self-reported scores from other memory systems use their own readers and graders, so they are not directly comparable with these.
 
 ## What It Stores
 
@@ -33,65 +97,28 @@ Git is a practical default for agent memory:
 
 ## Install
 
-Clone this repo, install dependencies, then install the MCP server for your agent:
-
 ```bash
-npm install
-npm run install:codex
+npx hippocamp@latest install
 ```
 
-For Claude Code:
+To install into one agent only:
 
 ```bash
-npm install
-npm run install:claude
+npx hippocamp@latest install-claude
+npx hippocamp@latest install-codex
+npx hippocamp@latest install-grok
 ```
 
-For Grok Build:
+Upgrade with the same command: `npx hippocamp@latest upgrade`. Installers default to `~/.lagoon` as the memory repo; pass `--global-root /absolute/path/to/lagoon` to use another clone.
 
-```bash
-npm install
-npm run install:grok
-```
-
-After publishing, the intended one-line install shape is:
-
-```bash
-npx hippocamp install-codex
-```
-
-Upgrade uses the same install path, so agents can refresh themselves without a separate state model:
-
-```bash
-npx hippocamp@latest upgrade-codex
-npx hippocamp@latest upgrade-claude
-npx hippocamp@latest upgrade-grok
-```
-
-From a source checkout:
-
-```bash
-git pull --ff-only
-npm install
-npm run upgrade:codex
-npm run upgrade:claude
-npm run upgrade:grok
-```
-
-Installers default to `~/.lagoon` as the memory repo. You can override it:
-
-```bash
-npm run install:codex -- --global-root /absolute/path/to/lagoon
-npm run install:claude -- --global-root /absolute/path/to/lagoon
-npm run install:grok -- --global-root /absolute/path/to/lagoon
-```
+Installed through npx, the agent starts the server with `npx -y --prefer-offline hippocamp@<version> mcp`, so it keeps working if npm cleans its cache. From a source checkout, `npm install` then `npm run install:claude` (or `install:codex`, `install:grok`) registers the checkout directly.
 
 The Codex installer refreshes `~/.codex/AGENTS.md`. The Claude installer refreshes `~/.claude/CLAUDE.md`. The Grok installer refreshes `~/.grok/rules/hippocamp.md` and registers the MCP server with `HIPPOCAMP_AGENT=grok`.
 Managed instruction blocks tell the agent to call `wake_up` at the start of new top-level coding tasks before repo exploration or edits.
 
 ## Lagoon Repo
 
-Hippocamp expects a local Git repo for memory:
+Hippocamp keeps memory in a local Git repo. The installer creates `~/.lagoon` if it is missing. To use memory you already have on another machine, clone it first:
 
 ```bash
 git clone git@github.com:YOUR_USER/lagoon.git ~/.lagoon
@@ -99,6 +126,12 @@ git clone git@github.com:YOUR_USER/lagoon.git ~/.lagoon
 
 The repo should usually be private. Pushing memory to a private remote keeps it available across machines and agent environments while still using normal Git access controls. Hippocamp does not need a GitHub token for local MCP mode; it uses your normal local Git credentials.
 
+Memory works locally without a remote. To back it up privately and share it across machines, create a private remote:
+
+```bash
+gh repo create lagoon --private --source ~/.lagoon --remote origin --push
+```
+
 If push auth is not configured yet, use your preferred GitHub setup. With GitHub CLI:
 
 ```bash
@@ -253,6 +286,7 @@ npm run mcp:smoke
 npm run dream
 npm run dream:railway
 npm run eval:lagoon
+npm run eval:qa
 npm run eval:retrieval
 npm run install:codex
 npm run install:claude
@@ -266,6 +300,8 @@ npm run upgrade:grok
 
 `npm run eval:lagoon` runs the same kind of check, read-only, against your real Lagoon. It reads cases from `<Lagoon root>/evals/retrieval-cases.json`: a project, a query, and the event ids that answer it. An empty list means no memory should be returned. Keep the cases in Lagoon, not in this repo, because they quote private work.
 
+`npm run eval:qa` measures answer accuracy on the same LongMemEval-S data with the official LongMemEval reader and grader prompts. It compares Hippocamp top 5 (`hippocamp`), top 10 (`hippocamp-k10`), and the labeled answer sessions (`oracle`). It calls an OpenAI-compatible endpoint from `MANIFEST_BASE_URL` and `MANIFEST_API_KEY`; set `QA_READER_MODEL` and `QA_GRADER_MODEL` to change models. Results are cached in `.context/qa-results.jsonl`, and `--dry-run` shows the jobs and token estimates without any API call.
+
 ## Releases
 
 Pull requests and pushes to `main` run syntax and MCP smoke checks in GitHub Actions.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "hippocamp",
-  "version": "0.2.2",
+  "version": "0.2.4",
   "private": false,
   "description": "Local Git-backed memory for AI agents.",
   "license": "MIT",
@@ -31,6 +31,7 @@
     "dream": "node scripts/hippocamp-dream.cjs",
     "dream:railway": "bash assets/railway/run-dream.sh",
     "eval:lagoon": "node scripts/eval-lagoon.cjs",
+    "eval:qa": "node scripts/eval-qa.cjs",
     "eval:retrieval": "node scripts/eval-retrieval.cjs",
     "install:claude": "node scripts/install-claude.cjs",
     "install:codex": "node scripts/install-codex.cjs",

package/scripts/eval-qa.cjs

@@ -0,0 +1,371 @@
+#!/usr/bin/env node
+
+const crypto = require("node:crypto");
+const fsSync = require("node:fs");
+const fs = require("node:fs/promises");
+const os = require("node:os");
+const path = require("node:path");
+const memory = require("./hippocamp-memory.cjs");
+
+const DEFAULT_DATA = ".context/longmemeval/longmemeval_s_cleaned.json";
+const DEFAULT_CACHE = ".context/qa-results.jsonl";
+const DEFAULT_SYSTEMS = "hippocamp,oracle";
+const SYSTEMS = ["hippocamp", "hippocamp-k10", "oracle"];
+const DATA_URL =
+  "https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json";
+
+// Prompts are verbatim from the official LongMemEval repo (xiaowu0162/LongMemEval):
+// src/generation/run_generation.py for the reader, src/evaluation/evaluate_qa.py for the grader.
+const READER_PROMPT =
+  "I will give you several history chats between you and a user. Please answer the question based on the relevant chat history. Answer the question step by step: first extract all the relevant information, and then reason over the information to get the answer.\n\n\nHistory Chats:\n\n{}\n\nCurrent Date: {}\nQuestion: {}\nAnswer (step by step):";
+
+const BASE_CHECK =
+  "I will give you a question, a correct answer, and a response from a model. Please answer yes if the response contains the correct answer. Otherwise, answer no. If the response is equivalent to the correct answer or contains all the intermediate steps to get the correct answer, you should also answer yes. If the response only contains a subset of the information required by the answer, answer no. ";
+const GRADER_PROMPTS = {
+  "single-session-user": `${BASE_CHECK}\n\nQuestion: {}\n\nCorrect Answer: {}\n\nModel Response: {}\n\nIs the model response correct? Answer yes or no only.`,
+  "single-session-assistant": `${BASE_CHECK}\n\nQuestion: {}\n\nCorrect Answer: {}\n\nModel Response: {}\n\nIs the model response correct? Answer yes or no only.`,
+  "multi-session": `${BASE_CHECK}\n\nQuestion: {}\n\nCorrect Answer: {}\n\nModel Response: {}\n\nIs the model response correct? Answer yes or no only.`,
+  "temporal-reasoning": `${BASE_CHECK}In addition, do not penalize off-by-one errors for the number of days. If the question asks for the number of days/weeks/months, etc., and the model makes off-by-one errors (e.g., predicting 19 days when the answer is 18), the model's response is still correct. \n\nQuestion: {}\n\nCorrect Answer: {}\n\nModel Response: {}\n\nIs the model response correct? Answer yes or no only.`,
+  "knowledge-update":
+    "I will give you a question, a correct answer, and a response from a model. Please answer yes if the response contains the correct answer. Otherwise, answer no. If the response contains some previous information along with an updated answer, the response should be considered as correct as long as the updated answer is the required answer.\n\nQuestion: {}\n\nCorrect Answer: {}\n\nModel Response: {}\n\nIs the model response correct? Answer yes or no only.",
+  "single-session-preference":
+    "I will give you a question, a rubric for desired personalized response, and a response from a model. Please answer yes if the response satisfies the desired response. Otherwise, answer no. The model does not need to reflect all the points in the rubric. The response is correct as long as it recalls and utilizes the user's personal information correctly.\n\nQuestion: {}\n\nRubric: {}\n\nModel Response: {}\n\nIs the model response correct? Answer yes or no only.",
+};
+
+function printHelp() {
+  console.log(`Hippocamp QA eval
+
+LongMemEval-S answer accuracy. A reader model answers each question from the
+retrieved history, and a grader model checks the answer, both with the official
+LongMemEval prompts. Calls go to an OpenAI-compatible chat completions endpoint.
+
+Systems:
+  hippocamp       searchMemory top 5 snippets
+  hippocamp-k10   searchMemory top 10 snippets
+  oracle          the labeled answer sessions, in full
+
+Usage:
+  npm run eval:qa -- [--data PATH] [--limit N] [--systems LIST] [--cache PATH] [--dry-run]
+
+Options:
+  --data PATH      LongMemEval-S JSON. Default: ${DEFAULT_DATA}
+  --limit N        Evaluate N questions spread evenly across the set. Default: all answerable questions
+  --systems LIST   Comma-separated systems. Default: ${DEFAULT_SYSTEMS}
+  --cache PATH     JSONL results cache; reruns only pay for missing calls. Default: ${DEFAULT_CACHE}
+  --concurrency N  Parallel reader and grader calls. Default: 6
+  --dry-run        Print jobs and prompt token estimates without any API call
+  --help           Show this help
+
+Environment:
+  MANIFEST_BASE_URL, MANIFEST_API_KEY   OpenAI-compatible endpoint and key
+  QA_READER_MODEL                       Default: anthropic/claude-sonnet-5
+  QA_GRADER_MODEL                       Default: openrouter/openai/gpt-4o
+
+Download the data once:
+  mkdir -p .context/longmemeval
+  curl -L -o ${DEFAULT_DATA} \\
+    ${DATA_URL}
+`);
+}
+
+function parsePositiveInteger(value, name) {
+  const number = Number(value);
+
+  if (!Number.isInteger(number) || number < 1) {
+    throw new Error(`${name} must be a positive integer.`);
+  }
+
+  return number;
+}
+
+function parseArgs(argv) {
+  const options = {
+    data: DEFAULT_DATA,
+    limit: null,
+    systems: DEFAULT_SYSTEMS.split(","),
+    cache: DEFAULT_CACHE,
+    concurrency: 6,
+    dryRun: false,
+    help: false,
+  };
+
+  for (let index = 0; index < argv.length; index += 1) {
+    const token = argv[index];
+
+    if (token === "--help" || token === "-h") {
+      options.help = true;
+    } else if (token === "--data") {
+      options.data = argv[++index];
+    } else if (token === "--limit") {
+      options.limit = parsePositiveInteger(argv[++index], "--limit");
+    } else if (token === "--systems") {
+      options.systems = String(argv[++index] || "").split(",").filter(Boolean);
+
+      for (const system of options.systems) {
+        if (!SYSTEMS.includes(system)) {
+          throw new Error(`Unknown system: ${system}. Use ${SYSTEMS.join(", ")}.`);
+        }
+      }
+    } else if (token === "--cache") {
+      options.cache = argv[++index];
+    } else if (token === "--concurrency") {
+      options.concurrency = parsePositiveInteger(argv[++index], "--concurrency");
+    } else if (token === "--dry-run") {
+      options.dryRun = true;
+    } else {
+      throw new Error(`Unknown argument: ${token}`);
+    }
+  }
+
+  return options;
+}
+
+function fill(template, ...values) {
+  return values.reduce((text, value) => text.replace("{}", () => String(value)), template);
+}
+
+function formatSession(sessionId, date, turns) {
+  const lines = [`# Session ${sessionId}`, "", `Date: ${date || "unknown"}`, ""];
+
+  for (const turn of turns || []) {
+    lines.push(`## ${turn.role === "assistant" ? "Assistant" : "User"}`, "", String(turn.content || "").trim(), "");
+  }
+
+  return `${lines.join("\n").trim()}\n`;
+}
+
+function historyString(chunks) {
+  return chunks
+    .sort((left, right) => left.date.localeCompare(right.date))
+    .map((chunk, index) => `\n### Session ${index + 1}:\nSession Date: ${chunk.date}\nSession Content:\n${chunk.content}\n`)
+    .join("");
+}
+
+async function hippocampChunks(item, projectRoot, maxResults) {
+  const pathToIndex = new Map();
+
+  for (let index = 0; index < item.haystack_sessions.length; index += 1) {
+    const relativePath = `sessions/${String(index).padStart(4, "0")}.md`;
+    const content = formatSession(item.haystack_session_ids[index], item.haystack_dates[index], item.haystack_sessions[index]);
+
+    await memory.writeMemoryFile({ scope: "project", path: relativePath, content, projectRoot, sync: false });
+    pathToIndex.set(relativePath, index);
+  }
+
+  const search = await memory.searchMemory({ query: item.question, scope: "project", projectRoot, maxResults });
+
+  return search.results.map((result) => ({
+    date: item.haystack_dates[pathToIndex.get(result.path)],
+    content: `\n${result.snippet}`,
+  }));
+}
+
+function oracleChunks(item) {
+  const answerIds = new Set(item.answer_session_ids.map(String));
+
+  return item.haystack_sessions
+    .map((turns, index) => ({ id: String(item.haystack_session_ids[index]), date: item.haystack_dates[index], turns }))
+    .filter((session) => answerIds.has(session.id))
+    .map((session) => ({
+      date: session.date,
+      content: `\n${JSON.stringify(session.turns.map(({ role, content }) => ({ role, content })))}`,
+    }));
+}
+
+// Models that rejected temperature (Claude 5 does); the official runs use 0 wherever it is accepted.
+const NO_TEMPERATURE = new Set();
+
+async function chat(endpoint, model, content, maxTokens) {
+  for (let attempt = 1; ; attempt += 1) {
+    try {
+      const body = { model, messages: [{ role: "user", content }], max_tokens: maxTokens };
+
+      if (!NO_TEMPERATURE.has(model)) {
+        body.temperature = 0;
+      }
+
+      const response = await fetch(endpoint, {
+        method: "POST",
+        headers: { Authorization: `Bearer ${process.env.MANIFEST_API_KEY}`, "Content-Type": "application/json" },
+        body: JSON.stringify(body),
+      });
+      const text = await response.text();
+
+      // Checks this request, not the set: parallel calls can all be refused before the first one marks the model.
+      if (response.status === 400 && /temperature/i.test(text) && "temperature" in body) {
+        NO_TEMPERATURE.add(model);
+        attempt -= 1;
+        continue;
+      }
+
+      if (!response.ok) {
+        throw Object.assign(new Error(`HTTP ${response.status}: ${text.slice(0, 300)}`), { status: response.status });
+      }
+
+      return JSON.parse(text).choices[0].message.content || "";
+    } catch (error) {
+      const retryable = !error.status || error.status === 429 || error.status >= 500;
+
+      if (!retryable || attempt >= 5) {
+        throw error;
+      }
+
+      await new Promise((resolve) => setTimeout(resolve, 2000 * attempt));
+    }
+  }
+}
+
+function percent(rows) {
+  return `${((rows.filter((row) => row.correct).length / Math.max(rows.length, 1)) * 100).toFixed(1)}%`;
+}
+
+async function run(options) {
+  if (!fsSync.existsSync(options.data)) {
+    throw new Error(`Missing ${options.data}. Run with --help for the download command.`);
+  }
+
+  const reader = process.env.QA_READER_MODEL || "anthropic/claude-sonnet-5";
+  const grader = process.env.QA_GRADER_MODEL || "openrouter/openai/gpt-4o";
+  const baseUrl = String(process.env.MANIFEST_BASE_URL || "").replace(/\/+$/, "");
+  const endpoint = `${baseUrl.endsWith("/v1") ? baseUrl : `${baseUrl}/v1`}/chat/completions`;
+  // The reader prompt's hash is in the key, so a change to the data or to retrieval never reuses a stale answer.
+  const promptHash = (prompt) => crypto.createHash("sha1").update(prompt).digest("hex").slice(0, 12);
+  const key = (system, id, hash) => `${system}|${reader}|${grader}|${id}|${hash}`;
+
+  const answerable = JSON.parse(await fs.readFile(options.data, "utf8")).filter(
+    (item) => !String(item.question_id).endsWith("_abs"),
+  );
+  const limit = Math.min(options.limit || answerable.length, answerable.length);
+  const items = Array.from({ length: limit }, (_, index) => answerable[Math.floor((index * answerable.length) / limit)]);
+  const done = new Map();
+
+  if (fsSync.existsSync(options.cache)) {
+    for (const line of (await fs.readFile(options.cache, "utf8")).split("\n").filter(Boolean)) {
+      const row = JSON.parse(line);
+      done.set(`${row.system}|${row.reader}|${row.grader}|${row.id}|${row.promptHash}`, row);
+    }
+  }
+
+  // Build every prompt first (retrieval is local and sequential), then call models concurrently.
+  const jobs = [];
+  const current = new Map();
+  const sandboxRoot = await fs.mkdtemp(path.join(os.tmpdir(), "hippocamp-eval-qa-"));
+  const previousGlobalRoot = process.env.HIPPOCAMP_GLOBAL_ROOT;
+
+  process.env.HIPPOCAMP_GLOBAL_ROOT = path.join(sandboxRoot, "lagoon");
+
+  try {
+    for (const [position, item] of items.entries()) {
+      const projectRoot = path.join(sandboxRoot, "questions", String(position));
+
+      for (const system of options.systems) {
+        const chunks =
+          system === "oracle" ? oracleChunks(item) : await hippocampChunks(item, projectRoot, system === "hippocamp-k10" ? 10 : 5);
+        const prompt = fill(READER_PROMPT, historyString(chunks), item.question_date, item.question);
+        const hash = promptHash(prompt);
+
+        current.set(`${system}|${item.question_id}`, key(system, item.question_id, hash));
+
+        if (!done.has(key(system, item.question_id, hash))) {
+          jobs.push({ system, item, prompt, hash });
+        }
+      }
+
+      await fs.rm(process.env.HIPPOCAMP_GLOBAL_ROOT, { recursive: true, force: true });
+    }
+  } finally {
+    if (previousGlobalRoot === undefined) {
+      delete process.env.HIPPOCAMP_GLOBAL_ROOT;
+    } else {
+      process.env.HIPPOCAMP_GLOBAL_ROOT = previousGlobalRoot;
+    }
+
+    await fs.rm(sandboxRoot, { recursive: true, force: true });
+  }
+
+  console.error(`${items.length} questions, ${jobs.length} new jobs; reader ${reader}, grader ${grader}`);
+
+  for (const system of options.systems) {
+    const chars = jobs.filter((job) => job.system === system).reduce((sum, job) => sum + job.prompt.length, 0);
+    console.error(`  ${system}: ${Math.round(chars / 4)} reader prompt tokens (estimate)`);
+  }
+
+  if (options.dryRun) {
+    return;
+  }
+
+  if (jobs.length && (!baseUrl || !process.env.MANIFEST_API_KEY)) {
+    throw new Error("Set MANIFEST_BASE_URL and MANIFEST_API_KEY.");
+  }
+
+  await fs.mkdir(path.dirname(path.resolve(options.cache)), { recursive: true });
+  let next = 0;
+  let finished = 0;
+  const worker = async () => {
+    while (next < jobs.length) {
+      const job = jobs[next++];
+      const response = await chat(endpoint, reader, job.prompt, 800);
+      const verdict = await chat(
+        endpoint,
+        grader,
+        fill(GRADER_PROMPTS[job.item.question_type], job.item.question, job.item.answer, response),
+        10,
+      );
+      const row = {
+        system: job.system,
+        reader,
+        grader,
+        id: job.item.question_id,
+        type: job.item.question_type,
+        // Same check as the official evaluate_qa.py: 'yes' in the lowercased verdict.
+        correct: /yes/i.test(verdict),
+        promptHash: job.hash,
+        contextChars: job.prompt.length,
+        response,
+      };
+
+      await fs.appendFile(options.cache, `${JSON.stringify(row)}\n`);
+      done.set(key(row.system, row.id, row.promptHash), row);
+
+      if (++finished % 20 === 0) {
+        console.error(`progress ${finished}/${jobs.length}`);
+      }
+    }
+  };
+
+  await Promise.all(Array.from({ length: options.concurrency }, worker));
+
+  console.log(`LongMemEval-S QA, ${items.length} answerable questions, reader ${reader}, grader ${grader}\n`);
+
+  for (const system of options.systems) {
+    const rows = items.map((item) => done.get(current.get(`${system}|${item.question_id}`))).filter(Boolean);
+    const contextChars = Math.round(rows.reduce((sum, row) => sum + row.contextChars, 0) / Math.max(rows.length, 1));
+    console.log(`${system.padEnd(14)} QA ${percent(rows)}  (${rows.length} graded, prompt ${contextChars} chars)`);
+  }
+
+  for (const system of options.systems) {
+    const rows = items.map((item) => done.get(current.get(`${system}|${item.question_id}`))).filter(Boolean);
+    console.log(`\n${system}`);
+
+    for (const type of [...new Set(rows.map((row) => row.type))].sort()) {
+      const byType = rows.filter((row) => row.type === type);
+      console.log(`  ${type.padEnd(26)} ${percent(byType).padStart(6)}  (${byType.length})`);
+    }
+  }
+}
+
+async function main() {
+  const options = parseArgs(process.argv.slice(2));
+
+  if (options.help) {
+    printHelp();
+    return;
+  }
+
+  await run(options);
+}
+
+main().catch((error) => {
+  console.error(error.message);
+  process.exit(1);
+});

package/scripts/hippocamp-mcp.cjs

@@ -162,7 +162,7 @@ async function main() {
     "search_memory",
     {
       description:
-        "Search task-specific memory. Event search uses cue indexes and returns only bounded coherent evidence; it does not scan unindexed event logs as a fallback. Event results carry their ISO timestamp as id; use it for when-questions.",
+        "Search task-specific memory. Returns no results when the question's distinctive words appear in no memory, unless the question names a date. Event search uses cue indexes and returns only bounded coherent evidence; it does not scan unindexed event logs as a fallback. Event results carry their ISO timestamp as id; use it for when-questions. A month name in the query favors events from that month. A YYYY-MM-DD date, today, yesterday, this/last week, or this/last month keeps only events from those dates (UTC); undated files still match. Default 5 results suits a single fact; ask for 10-20 when the question combines several past sessions (counts, totals, comparisons, timelines), and search again if results look incomplete.",
       inputSchema: {
         query: z.string(),
         scope: z.enum(["global", "project", "both"]).default("both"),