@remnic/bench 9.6.18 → 9.6.20

package/README.md

@@ -16,6 +16,125 @@ pnpm add @remnic/bench
 
 The CLI loads `@remnic/bench` via a computed-specifier dynamic import. If it's not installed, `remnic bench *` prints a clear install hint; the rest of the CLI keeps working.
 
+## OpenAI Build Week: five-minute MemCorrect path
+
+MemCorrect scores correction uptake and stale-memory harm through the same
+`remnic bench` surface. This keyless smoke path uses the packaged stdio MCP
+server, so it tests the actual MCP adapter rather than the in-memory baseline:
+
+```bash
+remnic bench run --quick memcorrect-v1 --adapter mcp --mcp-demo
+remnic bench runs list
+remnic bench export <run-id> --format html --output ./memcorrect-report.html
+```
+
+The run itself is deterministic and offline. It needs no dataset or API key.
+The generated HTML is self-contained and includes the correction ledger,
+per-dimension evidence, task drill-down, and reproducibility provenance. Treat
+the packaged demo as a transport and product smoke test, not a publishable
+backend-quality result.
+
+From a source checkout, build the optional companion before invoking the CLI:
+
+```bash
+pnpm install --frozen-lockfile
+pnpm --filter @remnic/bench build
+pnpm exec tsx packages/remnic-cli/src/index.ts bench run \
+  --quick memcorrect-v1 --adapter mcp --mcp-demo
+```
+
+An external MCP server can replace `--mcp-demo` with exactly one of:
+
+```bash
+# stdio
+remnic bench run --quick memcorrect-v1 --adapter mcp \
+  --mcp-command memory-server --mcp-args '["--stdio"]' \
+  --mcp-tool-map '{"store":"memory_store","recall":"memory_recall","correct":"memory_correct","reset":"memory_reset"}'
+
+# Streamable HTTP; set REMNIC_BENCH_MCP_BEARER_TOKEN if authentication is required
+remnic bench run --quick memcorrect-v1 --adapter mcp \
+  --mcp-url https://memory.example/mcp \
+  --mcp-tool-map '{"store":"memory_store","recall":"memory_recall","correct":"memory_correct","reset":"memory_reset"}'
+```
+
+Tool names alone may not be enough for a non-canonical server. The mapping can
+also describe argument semantics; see `remnic bench --help` for the current CLI
+surface and use preflight failures as conformance errors rather than empty
+recall scores.
+
+GPT-5.6 judging is explicit and uses the OpenAI Responses API:
+
+```bash
+export OPENAI_API_KEY=...
+remnic bench run --quick memcorrect-v1 --adapter mcp --mcp-demo \
+  --judge-provider openai --judge-model gpt-5.6
+```
+
+The credit-backed Codex CLI path is a distinct provider and model namespace.
+Use `gpt-5.6-luna` for bulk responder and internal work and
+`gpt-5.6-terra` for quality-critical judging. The exact API id `gpt-5.6`
+above is not a CLI alias. Confirm the authenticated catalog before a run with
+`codex debug models`. `gpt-5.6-sol` is opt-in only and is disabled in the
+bounded plan.
+
+Each Codex completion is a fresh, non-interactive `codex exec` in a new empty
+temporary workspace. It ignores user configuration and project rules,
+disables hooks, and keeps no session. The sandbox is read-only, approvals are
+denied, and the benchmark prompt instructs the model not to use tools. The
+Build Week plan uses normal service, not fast mode.
+
+The Build Week grant has 2,473 Codex credits. Use the account exclusively for
+this one harness process during a bounded run because Codex CLI exposes no
+machine-readable account balance. Bounded mode also requires `codex login
+status` to report ChatGPT authentication. A 473-credit safety reserve leaves
+2,000 usable credits. Configure the atomic completed-turn ledger and guards,
+then measure a quick task before choosing a workload bound:
+
+```bash
+export REMNIC_BENCH_CODEX_CREDIT_BUDGET=2473
+export REMNIC_BENCH_CODEX_CREDIT_RESERVE=473
+export REMNIC_BENCH_CODEX_CREDIT_LEDGER="$PWD/.bench-private/codex-credit-ledger.json"
+
+remnic bench run --quick longmemeval \
+  --runtime-profile real \
+  --system-provider codex-cli --system-model gpt-5.6-luna \
+  --system-codex-reasoning-effort medium \
+  --internal-provider codex-cli --internal-model gpt-5.6-luna \
+  --internal-codex-reasoning-effort medium \
+  --judge-provider codex-cli --judge-model gpt-5.6-terra \
+  --judge-codex-reasoning-effort high
+
+remnic bench run longmemeval \
+  --runtime-profile real --limit <LEDGER_DERIVED_LIMIT> \
+  --system-provider codex-cli --system-model gpt-5.6-luna \
+  --system-codex-reasoning-effort medium \
+  --internal-provider codex-cli --internal-model gpt-5.6-luna \
+  --internal-codex-reasoning-effort medium \
+  --judge-provider codex-cli --judge-model gpt-5.6-terra \
+  --judge-codex-reasoning-effort high
+```
+
+The placeholder is intentional. `--limit` and LoCoMo/MemoryAgentBench's
+`--trial-limit` bound tasks, not token credits. Derive each next batch from
+actual `turn.completed` JSONL usage. Stop dispatching at 2,000 spent; the
+473-credit reserve absorbs only a final in-flight call whose exact cost becomes
+known after completion. Missing exact terminal usage blocks the ledger pending
+manual account reconciliation.
+Rates per one million tokens are Luna: 25 input, 2.5 cached input, 150 output;
+Terra: 62.5 input, 6.25 cached input, 375 output. A bounded result is a trial,
+not a full leaderboard artifact.
+
+Codex built and adversarially reviewed the Build Week adapter, Responses
+provider, and report card. The underlying Remnic engine and original benchmark
+harness are prior work. The evidence ledger, credit-backed frontier-run
+placeholder, and release status live in the root [`HACKATHON.md`](../../HACKATHON.md).
+
+The claimed judge path requires Node.js 22.12+. It is verified from source and
+from packed tarballs installed into a clean global prefix on Linux; macOS is
+supported with the same Node CLI but still needs a release-install receipt.
+Windows judges should use WSL2; native Windows is not currently claimed as
+Build Week-verified.
+
 ## What it does
 
 - **Benchmark runners** for a growing set of memory-oriented evals: `longmemeval`, `locomo`, `memory-arena`, `amemgym`, `ama-bench`, plus a lightweight smoke fixture.

package/dist/demo/mcp-memory-server.d.ts

@@ -0,0 +1 @@
+#!/usr/bin/env node

package/dist/demo/mcp-memory-server.js

@@ -0,0 +1,129 @@
+#!/usr/bin/env node
+
+// src/demo/mcp-memory-server.ts
+import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { z } from "zod";
+
+// src/demo/mcp-memory-correction.ts
+function parseSyntheticCorrection(content) {
+  const trimmed = content.trim();
+  const patterns = [
+    {
+      expression: /^Correction: replace (.+) with (.+)\.$/s,
+      oldValueIndex: 1,
+      newValueIndex: 2
+    },
+    {
+      expression: /^Correction: my .+? record saying (.+?) is wrong\. It is now (.+)\.$/s,
+      oldValueIndex: 1,
+      newValueIndex: 2
+    },
+    {
+      expression: /^Oh by the way, we switched .+? from (.+?) to (.+) last month\.$/s,
+      oldValueIndex: 1,
+      newValueIndex: 2
+    },
+    {
+      expression: /^For this project, .+? is (.+?) now, not (.+)\.$/s,
+      oldValueIndex: 2,
+      newValueIndex: 1
+    },
+    {
+      expression: /^Update: .+? is (.+?) going forward instead of (.+)\.$/s,
+      oldValueIndex: 2,
+      newValueIndex: 1
+    }
+  ];
+  for (const pattern of patterns) {
+    const match = pattern.expression.exec(trimmed);
+    const oldValue = match?.[pattern.oldValueIndex];
+    const newValue = match?.[pattern.newValueIndex];
+    if (oldValue !== void 0 && newValue !== void 0) return { oldValue, newValue };
+  }
+  return void 0;
+}
+
+// src/demo/mcp-memory-server.ts
+var memories = /* @__PURE__ */ new Map();
+var server = new McpServer({
+  name: "remnic-bench-demo-memory",
+  version: "1.0.0"
+});
+server.registerTool(
+  "store_memory",
+  {
+    description: "Store one synthetic benchmark memory in an isolated session.",
+    inputSchema: {
+      namespace: z.string(),
+      sessionId: z.string(),
+      content: z.string(),
+      role: z.string().optional(),
+      timestamp: z.string().optional()
+    }
+  },
+  async ({ sessionId, content }) => {
+    memories.set(sessionId, [...memories.get(sessionId) ?? [], content]);
+    return { content: [{ type: "text", text: JSON.stringify({ stored: true }) }] };
+  }
+);
+server.registerTool(
+  "search_memory",
+  {
+    description: "Recall memories from one isolated benchmark session.",
+    inputSchema: {
+      namespace: z.string(),
+      sessionId: z.string(),
+      query: z.string(),
+      limit: z.number().int().positive().optional()
+    }
+  },
+  async ({ sessionId, limit }) => ({
+    content: [
+      {
+        type: "text",
+        text: JSON.stringify({ memories: (memories.get(sessionId) ?? []).slice(0, limit) })
+      }
+    ]
+  })
+);
+server.registerTool(
+  "correct_memory",
+  {
+    description: "Apply a synthetic natural-language correction to one session.",
+    inputSchema: {
+      namespace: z.string(),
+      sessionId: z.string(),
+      content: z.string(),
+      timestamp: z.string().optional()
+    }
+  },
+  async ({ sessionId, content }) => {
+    const replacement = parseSyntheticCorrection(content);
+    const current = memories.get(sessionId) ?? [];
+    if (replacement) {
+      memories.set(
+        sessionId,
+        current.map((item) => item.replaceAll(replacement.oldValue, replacement.newValue))
+      );
+    } else {
+      memories.set(sessionId, [...current, content]);
+    }
+    return { content: [{ type: "text", text: JSON.stringify({ applied: true }) }] };
+  }
+);
+server.registerTool(
+  "delete_memory",
+  {
+    description: "Delete only the requested isolated benchmark session.",
+    inputSchema: {
+      namespace: z.string(),
+      sessionId: z.string()
+    }
+  },
+  async ({ sessionId }) => {
+    memories.delete(sessionId);
+    return { content: [{ type: "text", text: JSON.stringify({ deleted: true }) }] };
+  }
+);
+await server.connect(new StdioServerTransport());