hippocamp 0.2.12 → 0.2.13

package/AGENTS.md

@@ -29,6 +29,13 @@ This repository is a local-first MCP package for Git-backed agent memory. Keep c
 - `scripts/eval-lagoon.cjs`: read-only retrieval check against real Lagoon cases in `<Lagoon root>/evals/retrieval-cases.json`.
 - `scripts/eval-qa.cjs`: LongMemEval-S answer-accuracy check (reader and grader models) for Hippocamp snippets against the labeled answer sessions.
 - `scripts/eval-retrieval.cjs`: retrieval-only LongMemEval-S check for `searchMemory` against a BM25 reference.
+- `scripts/eval-continuity.cjs`: work continuity benchmark runner and CLI: real headless Claude Code or Codex through successive sessions per arm (`cc-none`, `cc-native`, `cc-hippocamp`, `codex-native`, `codex-hippocamp`) in isolated sandboxes, with a frozen run manifest and `--report`.
+- `scripts/continuity-adapters.cjs`: one adapter per benchmark arm, each installed the way a developer has it (Hippocamp through its real `install-claude` or `install-codex` installer).
+- `scripts/continuity-judge.cjs`: blind task judge, miss attribution, and the memory-quality judge.
+- `scripts/continuity-report.cjs`: outcome, efficiency and memory-quality tables, pooled over runs, with a probe-level bootstrap.
+- `scripts/continuity-dataset.cjs`: loader and validator for the continuity dataset, including the retrieval-hint check on prompts.
+- `scripts/eval-locomo.cjs`: LoCoMo answer-accuracy check with the MemoryTax protocol; its `complete` model client is shared with the continuity judge.
+- `bench/continuity/`: synthetic continuity dataset: two seed repos, session tickets, planted facts and the run timeline.
 - `scripts/install-common.cjs`: shared installer helpers: the MCP launch command and Lagoon repo preparation.
 - `scripts/install-claude.cjs`: one-step installer for Claude Code MCP setup and user-level `CLAUDE.md` guidance.
 - `scripts/install-codex.cjs`: one-step installer for the Hippocamp skill and MCP server in Codex.
@@ -79,6 +86,7 @@ When changing code, use the smallest verification that proves the change:
 - Run `npm run dream -- --help` when changing Dream CLI arguments.
 - Run `npm run eval:retrieval -- --limit 100` when changing search ranking or snippets; report Recall@5 against the BM25 row and snippet evidence. Also run `npm run eval:lagoon` when a Lagoon is available.
 - Run `npm run eval:qa -- --limit 3 --dry-run` when changing `scripts/eval-qa.cjs`; a full run calls paid models, so run it only when asked.
+- Run `npm test` and `npm run eval:continuity -- --dry-run` when changing the continuity benchmark or its dataset; a real run spawns paid Claude Code sessions, so run it only when asked.
 
 ## Notes For Future Agents
 

package/README.md

@@ -52,6 +52,8 @@ Reproducible with the scripts in this repo.
 | Text returned per search | 5.9k chars | Whole sessions: 65.5k |
 | Answer accuracy, Claude Sonnet 5, top 10 results | 77% | Correct sessions given directly: 90% |
 | Unrelated questions that return nothing (our own test set) | 10 / 10 | |
+| Work continuity, 21 real Claude Code sessions, checks passed | 100% (20 / 20) | Claude Code native memory: 96% (25 / 26) |
+| Work continuity, same 21 sessions in Codex, checks passed | 86% (24 / 28) | Codex native memory: 86% (24 / 28) |
 
 Full tables, method, and limits: [docs/results.md](docs/results.md).
 

package/docs/development.md

@@ -10,6 +10,7 @@ npm run mcp:help
 npm run mcp:smoke
 npm run dream
 npm run dream:railway
+npm run eval:continuity
 npm run eval:lagoon
 npm run eval:qa
 npm run eval:retrieval
@@ -21,6 +22,26 @@ npm run upgrade:claude
 npm run upgrade:grok
 ```
 
+## Work continuity benchmark
+
+`npm run eval:continuity` measures what Hippocamp is built for: a coding agent that keeps working on a project across independent sessions and acts on earlier decisions, constraints and state without being told to look them up.
+
+- The dataset in `bench/continuity/` is synthetic: two small engineering projects (`relay`, a TypeScript webhook relay, and `ingest`, a Python CSV pipeline) on one 21-session timeline. Work sessions are ordinary tickets that plant facts; probe sessions are ordinary tickets with hidden checklists. No prompt hints at retrieval.
+- Arms, each set up the way a developer has it: `cc-none` (auto-memory off), `cc-native` (Claude Code defaults) and `cc-hippocamp` (Hippocamp through its real installer, auto-memory off). Codex arms run `codex exec` (`gpt-6.1-sol`) on the ChatGPT sign-in copied from `~/.codex/auth.json`: `codex-native` (Codex `memories` on, with `min_rollout_idle_hours = 0` because sessions run back to back) and `codex-hippocamp` (Hippocamp through `install-codex`). Codex reports no dollar cost, so its cost column stays at $0.
+- Each session is a fresh headless `claude -p --model claude-opus-5-5` process. The repo is committed after each session with the agent's own summary as the message. If the agent ends with an offer or a question, the simulated user answers once with "yes, go ahead". A retried session restores both the repo and the memory store.
+- Sandboxes live in `HIPPOCAMP_BENCH_DIR` (default `~/.cache/hippocamp-bench`), each with its own `HOME` and `CLAUDE_CONFIG_DIR`, so your real `~/.claude` and `~/.lagoon` are never read or written, and runs survive reboots.
+- Judging: a blind judge grades each check from the diff and a tool log with memory tools and paths anonymized. A second call explains each miss (never stored, stale memory, wrong scope, not recalled, ignored, asked instead of acting). At the end, a third grades every stored memory entry for coverage, precision, staleness, noise and duplicates. Prompts and replies are kept next to each transcript, and `validation.csv` lists every verdict for hand-checking.
+- Each session records tokens, time, memory tool calls and memory bytes read or auto-loaded.
+
+```bash
+npm run eval:continuity -- --dry-run                                         # free: validates and prints the cost range
+npm run eval:continuity -- --systems cc-native,cc-hippocamp --limit 3 --run smoke
+npm run eval:continuity -- --run final-1                                     # full run
+npm run eval:continuity -- --report final-1,final-2                          # pooled report, spawns nothing
+```
+
+Sessions and the Claude judge run through Claude Code, so they use your Claude subscription when `CLAUDE_CODE_OAUTH_TOKEN` is set (create it with `claude setup-token` and put it in `.env`). Without it they bill `ANTHROPIC_API_KEY`. `--dry-run` prints the API-price range, and `--budget` caps each call. Results go to `.context/continuity/<run>/`. A run is frozen by `manifest.json` (dataset, harness, model and Claude Code version): reruns with the same `--run` skip finished sessions, new sessions appended to the timeline extend it, and any frozen change is refused unless `--force-manifest` records it. The report shows n next to every percentage, defines memory-required items as those `cc-none` fails in every run, and gives bootstrap intervals over probes.
+
 ## Releases
 
 Pull requests and pushes to `main` run syntax and MCP smoke checks in GitHub Actions.

package/docs/results.md

@@ -39,6 +39,56 @@ The same reader reaches 90% with the correct sessions, so the remaining gap is r
 | Right event in top 5, 43 answerable questions | 42 / 43 |
 | No results for 10 questions no memory is about | 10 / 10 |
 
+## Work continuity
+
+Does memory carry real engineering work from one coding session to the next? `npm run eval:continuity` runs real headless Claude Code (`claude-opus-5-5`, Claude Code 2.1.318 to 2.1.319) through 21 sessions over two small synthetic projects (a TypeScript webhook relay and a Python ingest job). Early sessions plant facts the way work does: a rejected library, a pinned Node version, a blocked vendor, a parked branch, a user preference. Later sessions give tickets that only go well if the agent remembers them. The prompts never say "remember" or point at memory.
+
+In short: the same Hippocamp server, plugged unchanged into Claude Code or Codex, keeps up with each one's own memory (100% vs 96% in Claude Code, 86% vs 86% in Codex), with fewer than 5 points of difference and one run per arm. In Codex, whose built-in memory stored almost nothing, Hippocamp's memory covered 35% of the planted facts against 4%.
+
+Each arm is installed the way a developer has it, in its own sandbox home: Claude Code defaults with native auto-memory, or Hippocamp through its real installer (`install-claude`) with auto-memory off. A blind judge (Opus, arm names and memory paths scrubbed) grades 28 checks against the diff and the tool calls. A check is n/a when the session wrote no code it applies to.
+
+Run `full-1`, one run per arm:
+
+| | Claude Code native memory | Hippocamp in Claude Code |
+| --- | ---: | ---: |
+| Checks passed | 96% (25 / 26) | 100% (20 / 20) |
+| Checks marked as needing memory | 92% (11 / 12) | 100% (10 / 10) |
+| Tokens per session, median | 179k | 224k |
+| Time per session, median | 62 s | 73 s |
+| Agent cost per run, API price | $8.84 | $9.19 |
+| Memory entries at the end | 8 | 15 |
+| Planted facts covered by memory | 22% | 30% |
+| Memory entries that are useful (precision) | 88% | 80% |
+
+What it shows: Hippocamp, plugged into Claude Code as a plain MCP server, does as well as Claude Code's own memory, at about the same cost (+4% dollars, +25% tokens per session). Native memory's one miss was a parked `fix/late-dedupe` branch it no longer knew about, so it rewrote the fix from scratch. Hippocamp's 8 n/a checks come mostly from one session where it correctly held a blocked vendor back and wrote no code; native also held the vendor back, but with gated code the checks could grade.
+
+What it does not show: that Hippocamp is better. The difference is one check, from one run per arm (bootstrap 95% interval over probes: +4 points, 0 to +16). There is no no-memory run, so the "needing memory" label comes from the dataset, not from a measured baseline. An earlier development run (27 / 27 vs 24 / 27) is not used: its cases were added after a tie and its harness was unfair to native memory.
+
+The diffs, judge prompts, grades with evidence and memory snapshots of this run are in [results/continuity-full-1](results/continuity-full-1/) (start with `report.md` and `validation.csv`). Full transcripts are left out for size.
+
+### Same benchmark in Codex
+
+Run `codex-1` repeats the 21 sessions in Codex (`codex exec`, codex-cli 0.159.2, `gpt-6.1-sol`, ChatGPT sign-in), same checks and same blind Opus judge. Codex native memory is its `memories` feature, turned on (it is off by default) with `min_rollout_idle_hours = 0`, because the benchmark runs sessions back to back instead of days apart. Hippocamp is installed with `install-codex`; nothing else changes.
+
+| | Codex native memory | Hippocamp in Codex |
+| --- | ---: | ---: |
+| Checks passed | 86% (24 / 28) | 86% (24 / 28) |
+| Checks marked as needing memory | 67% (8 / 12) | 75% (9 / 12) |
+| Tokens per session, median | 166k | 238k |
+| Time per session, median | 63 s | 82 s |
+| Memory calls per session | 1.0 | 4.9 |
+| Memory entries at the end | 4 | 17 |
+| Planted facts covered by memory | 4% | 35% |
+| Memory entries that are useful (precision) | 25% | 76% |
+
+What it shows: the same Hippocamp server, with no Codex-specific code, works in Codex as it does in Claude Code. Codex called `wake_up`, `search_memory` and `append_event` on its own, and the memory it built covers about as many planted facts as in Claude Code (35% vs 30%). The two Codex arms tie on checks (bootstrap: 0 points, -14 to +14).
+
+Codex's own memory stored almost nothing in this run: its background summaries never produced a memory from a past session, and the one real entry is a changelog preference the agent wrote as a note. The other three entries are Codex's empty templates, which the judge counts as noise. Codex still passed most checks without it, from the repository itself (code, commit messages, notes it wrote in the repo); that is why the score does not drop more. Hippocamp's misses include one stale next step and one fact it never saved (a blocked vendor).
+
+Codex reports no dollar cost, so this run has token and time numbers only.
+
+The run data is in [results/continuity-codex-1](results/continuity-codex-1/).
+
 ## Why less text matters
 
 Memory results share the agent's context with code, diffs, and tool output, and agents search several times per task. About 6k characters per search is roughly 1.5k tokens; whole sessions would be about 16k. Fewer tokens make each turn cheaper and faster, and keep the relevant lines from being buried. Less text is only useful when the answer is still in it, which is why we report answer accuracy next to text size.
@@ -56,4 +106,6 @@ Memory results share the agent's context with code, diffs, and tool output, and
 
 `npm run eval:lagoon` runs the same kind of check, read-only, against your real Lagoon. It reads cases from `<Lagoon root>/evals/retrieval-cases.json`: a project, a query, and the event ids that answer it. An empty list means no memory should be returned. Keep the cases in Lagoon, not in this repo, because they quote private work.
 
+`npm run eval:continuity -- --systems cc-native,cc-hippocamp --run <name>` (or `codex-native,codex-hippocamp`, which needs a Codex ChatGPT sign-in) reruns the work continuity benchmark with your Claude subscription (`CLAUDE_CODE_OAUTH_TOKEN` in `.env`); `--report <name>` prints the tables, and `--dry-run` shows the cost estimate. A run stops cleanly at a usage limit and resumes where it stopped.
+
 `npm run eval:qa` measures answer accuracy on the same LongMemEval-S data with the official LongMemEval reader and grader prompts. It compares Hippocamp top 5 (`hippocamp`), top 10 (`hippocamp-k10`), and the labeled answer sessions (`oracle`). It calls an OpenAI-compatible endpoint from `MANIFEST_BASE_URL` and `MANIFEST_API_KEY`; set `QA_READER_MODEL` and `QA_GRADER_MODEL` to change models. Results are cached in `.context/qa-results.jsonl`, and `--dry-run` shows the jobs and token estimates without any API call.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "hippocamp",
-  "version": "0.2.12",
+  "version": "0.2.13",
   "private": false,
   "description": "Local Git-backed memory for AI agents.",
   "license": "MIT",
@@ -30,6 +30,7 @@
   "scripts": {
     "dream": "node scripts/hippocamp-dream.cjs",
     "dream:railway": "bash assets/railway/run-dream.sh",
+    "eval:continuity": "node --env-file-if-exists=.env scripts/eval-continuity.cjs",
     "eval:lagoon": "node scripts/eval-lagoon.cjs",
     "eval:qa": "node scripts/eval-qa.cjs",
     "eval:retrieval": "node scripts/eval-retrieval.cjs",

package/scripts/install-claude.cjs

@@ -273,7 +273,11 @@ async function main() {
   printHints(lagoon);
 }
 
-main().catch((error) => {
-  console.error("Failed to install Hippocamp into Claude Code:", error.message);
-  process.exit(1);
-});
+if (require.main === module) {
+  main().catch((error) => {
+    console.error("Failed to install Hippocamp into Claude Code:", error.message);
+    process.exit(1);
+  });
+}
+
+module.exports = { buildClaudeManagedBlock, ensureClaudeInstructionsInstalled, ensureClaudeSkillInstalled };

package/scripts/eval-lagoon.cjs

@@ -1,135 +0,0 @@
-#!/usr/bin/env node
-
-const fsSync = require("node:fs");
-const fs = require("node:fs/promises");
-const os = require("node:os");
-const path = require("node:path");
-const memory = require("./hippocamp-memory.cjs");
-
-const MAX_TOP_K = 20;
-const DEFAULT_CASES = "evals/retrieval-cases.json";
-
-function printHelp() {
-  console.log(`Hippocamp Lagoon eval
-
-Read-only retrieval check against your real Lagoon. No model or API key is used.
-Each case names a project, a query, and the event ids that answer it.
-An empty expect list means no memory should be returned.
-
-Usage:
-  npm run eval:lagoon -- [--cases PATH] [--top-k K]
-
-Options:
-  --cases PATH  Cases JSON. Default: <Lagoon root>/${DEFAULT_CASES}
-  --top-k K     Results retrieved per query, at most ${MAX_TOP_K}. Default: 5
-  --help        Show this help
-
-Cases file:
-  { "cases": [ { "project": "<slug>", "query": "...", "expect": ["<event id>"] } ] }
-`);
-}
-
-function parseArgs(argv) {
-  const options = { cases: path.join(memory.getGlobalRoot(), DEFAULT_CASES), topK: 5, help: false };
-
-  for (let index = 0; index < argv.length; index += 1) {
-    const token = argv[index];
-
-    if (token === "--help" || token === "-h") {
-      options.help = true;
-    } else if (token === "--cases") {
-      options.cases = argv[++index];
-    } else if (token === "--top-k") {
-      const number = Number(argv[++index]);
-
-      if (!Number.isInteger(number) || number < 1 || number > MAX_TOP_K) {
-        throw new Error(`--top-k must be an integer from 1 to ${MAX_TOP_K}.`);
-      }
-
-      options.topK = number;
-    } else {
-      throw new Error(`Unknown argument: ${token}`);
-    }
-  }
-
-  return options;
-}
-
-async function run(options) {
-  if (!fsSync.existsSync(options.cases)) {
-    throw new Error(`Missing ${options.cases}. Run with --help for the file format.`);
-  }
-
-  const { cases } = JSON.parse(await fs.readFile(options.cases, "utf8"));
-  // searchMemory derives the project slug from the folder name, so a temp path per
-  // slug reads that project's memory without touching any real repo. The folder is
-  // not created: an existing empty folder has no project and reads global memory.
-  const sandboxRoot = await fs.mkdtemp(path.join(os.tmpdir(), "hippocamp-eval-lagoon-"));
-  const answerable = { count: 0, hit: 0, mrr: 0, chars: 0, topScores: [] };
-  const empty = { count: 0, correct: 0, topScores: [] };
-
-  try {
-    for (const item of cases) {
-      if (!/^[a-z0-9][a-z0-9._-]*$/.test(String(item.project))) {
-        throw new Error(`Invalid project slug: ${item.project}`);
-      }
-
-      const projectRoot = path.join(sandboxRoot, item.project);
-
-      const search = await memory.searchMemory({
-        query: item.query,
-        scope: "project",
-        projectRoot,
-        maxResults: options.topK,
-      });
-      const topScore = search.results[0]?.score ?? 0;
-
-      if (!item.expect.length) {
-        empty.count += 1;
-        empty.correct += search.results.length ? 0 : 1;
-        empty.topScores.push(topScore);
-        console.log(`${search.results.length ? "MISS" : "ok  "}  empty   top ${topScore}  [${item.project}] ${item.query}`);
-        continue;
-      }
-
-      const rank = search.results.findIndex((result) => item.expect.includes(result.id));
-      answerable.count += 1;
-      answerable.hit += rank === -1 ? 0 : 1;
-      answerable.mrr += rank === -1 ? 0 : 1 / (rank + 1);
-      answerable.chars += search.results.reduce((sum, result) => sum + result.snippet.length, 0);
-      answerable.topScores.push(topScore);
-      console.log(
-        `${rank === -1 ? "MISS" : "ok  "}  rank ${rank === -1 ? "-" : rank + 1}  top ${topScore}  [${item.project}] ${item.query}`,
-      );
-    }
-  } finally {
-    await fs.rm(sandboxRoot, { recursive: true, force: true });
-  }
-
-  const percent = (part, whole) => `${((part / Math.max(whole, 1)) * 100).toFixed(1)}%`;
-  const range = (scores) => (scores.length ? `${Math.min(...scores)}-${Math.max(...scores)}` : "n/a");
-
-  console.log(`\nLagoon retrieval, K=${options.topK}, ${cases.length} cases`);
-  console.log(
-    `answerable (${answerable.count})  Hit@K ${percent(answerable.hit, answerable.count)}  MRR ${(answerable.mrr / Math.max(answerable.count, 1)).toFixed(3)}  context ${Math.round(answerable.chars / Math.max(answerable.count, 1))} chars  top score ${range(answerable.topScores)}`,
-  );
-  console.log(
-    `no memory  (${empty.count})  empty result ${percent(empty.correct, empty.count)}  top score ${range(empty.topScores)}`,
-  );
-}
-
-async function main() {
-  const options = parseArgs(process.argv.slice(2));
-
-  if (options.help) {
-    printHelp();
-    return;
-  }
-
-  await run(options);
-}
-
-main().catch((error) => {
-  console.error(error.message);
-  process.exit(1);
-});