canli-validation-mcp 0.13.0 → 0.14.0

package/README.md

@@ -207,6 +207,30 @@ On your own machine, `prices_file`, `series_file` and `returns_file` take a CSV
 of the numbers. Only numbers are read, plus an ISO date column when the file has one, so results can
 name the day a drawdown began.
 
+## The trial ledger: count every variant an agent tries
+
+An agent can try thousands of strategy variants in an hour, and the best of them looks skilled by
+luck alone. How much luck depends on how many were tried and how spread out their results were, so
+the honest verdict on a reported winner needs the whole search. The ledger keeps it.
+
+Turn it on with `CANLI_TOOLSETS=validate,lab,ledger` (or `all`), then:
+
+1. `ledger_record_trial` for every variant, as it is tried: its returns (or a returns file), or its
+   annualized Sharpe and sample size, plus a label and its parameters. The first call creates a
+   ledger and returns its name; reuse it. Each reply already counts the whole search so far.
+2. `ledger_summary` for the verdict: the best trial, the Sharpe that the best of that many
+   skill-less trials reaches by luck, the best trial's deflated Sharpe ratio after counting every
+   trial, and the probability of backtest overfitting (CSCV) when every trial has aligned returns.
+3. `ledger_export` to hand the search to someone else. Each trial is chained to the previous one
+   with SHA-256, so a removed, reordered or edited trial shows.
+
+Ledgers are JSON-lines files under `CANLI_LEDGER_DIR` (default `~/.canli/ledgers`); nothing leaves
+your machine. Twenty variants of pure noise, recorded this way, give a best annualized Sharpe of
+about 1.5, a luck-alone expectation of about 1.7 and a deflated Sharpe near 0.4: the ledger calls
+that winner what it is. The statistics are the same checked code as `validate_deflated_sharpe` and
+`validate_overfitting`; `test/ledger.test.mjs` checks both directions (noise is flagged, a real edge
+among noise keeps a deflated Sharpe above 0.95).
+
 ## Auditing a backtest in one call
 
 `audit_backtest` takes one strategy's return series and, ideally, every variant's returns
@@ -420,10 +444,11 @@ Every envelope this server returns carries these sentences, verbatim, from the A
   its costs, survivorship, or any lookahead in how the series was built.
 - A deflated Sharpe or overfitting probability above or below any threshold is not admission to
   anything and is not a forecast.
-- The receipt is content-hashed and reproducible from the open-source core it names. It is not
-  signed.
+- The receipt is content-hashed, reproducible from the open-source core it names, and signed with
+  Ed25519 by a key published at https://canlicapital.com/.well-known/canli-receipt-keys.json.
 - Quotas: 1000 validations per key per UTC day, 5 keys per client per UTC day, 1048576 bytes per
-  request, 20000 observations per series, 200 variants per matrix.
+  validation request, 1024 bytes per key revocation request, 20000 observations per series, 200
+  variants per matrix.
 
 Each tool's description also states one of these sentences, so an agent sees the boundary before
 it calls the tool, not only after.
@@ -443,7 +468,8 @@ A client sends the model the whole tool list on every turn, and it is most of ea
 a validation result is a few hundred tokens, the list of every tool several thousand. A client
 that needs one kind of tool can list only that kind, with `CANLI_TOOLSETS` (stdio) or
 `?toolsets=` (hosted endpoint). The default is every toolset except `company`, whose one tool
-canli-fundamentals-mcp does better and point in time; `all` adds it. `get_key` is listed only when
+canli-fundamentals-mcp does better and point in time, and `ledger`, which keeps files on your machine
+and is local only; `all` adds both. `get_key` is listed only when
 no key is configured.
 
 | toolset | tools |
@@ -453,20 +479,22 @@ no key is configured.
 | `company` | `company_financial_history` |
 | `status` | `service_status` |
 | `lab` | `backtest_strategy`, `summarize_series`, `stress_test`, `check_feasibility`, `check_leakage`, `placebo_test` |
+| `ledger` | `ledger_record_trial`, `ledger_summary`, `ledger_export` (stdio only) |
 
 Measured with `bench/tool_list_tokens.py` (tokenizer: tiktoken `o200k_base`; other models'
 tokenizers give different absolute counts), in the shape an OpenAI-style client sends the list:
 
 | CANLI_TOOLSETS | tools | tokens per turn | of all |
 |---|---|---|---|
-| (default) | 19 | 5,970 | 96% |
-| (default, with CANLI_KEY) | 18 | 5,834 | 93% |
-| `all` | 20 | 6,241 | 100% |
-| `validate` | 11 | 3,650 | 58% |
+| (default) | 19 | 5,978 | 85% |
+| (default, with CANLI_KEY) | 18 | 5,842 | 83% |
+| `all` | 23 | 7,019 | 100% |
+| `validate` | 11 | 3,658 | 52% |
 | `receipts` | 1 | 182 | 3% |
 | `company` | 1 | 273 | 4% |
 | `status` | 1 | 89 | 1% |
-| `lab` | 6 | 2,055 | 33% |
+| `lab` | 6 | 2,055 | 29% |
+| `ledger` | 3 | 772 | 11% |
 
 Providers cache a tool list that is identical from turn to turn and bill the cached part at a
 fraction of the price (`test/tool-list-stable.test.mjs` keeps each list byte-stable); a smaller list

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "canli-validation-mcp",
-  "version": "0.13.0",
+  "version": "0.14.0",
   "description": "Check whether a backtest is real: deflated Sharpe, CSCV probability of backtest overfitting, minimum track record and backtest length, haircut Sharpe and luck-equivalent trials, as an MCP server. Local mode, hosted endpoint, signed receipts.",
   "keywords": [
     "mcp",
@@ -38,7 +38,7 @@
     "test:package": "node test/package-smoke.mjs"
   },
   "dependencies": {
-    "@modelcontextprotocol/server": "2.1.0",
+    "@modelcontextprotocol/server": "2.3.1",
     "zod": "4.6.5"
   },
   "mcpName": "io.github.arhancanli/canli-validation-mcp",
@@ -52,7 +52,7 @@
     "url": "https://github.com/arhancanli/canlicapital/issues"
   },
   "devDependencies": {
-    "@modelcontextprotocol/client": "2.1.0",
+    "@modelcontextprotocol/client": "2.3.1",
     "fast-check": "4.10.2"
   }
 }

package/src/lab.mjs

@@ -242,6 +242,9 @@ export const EXAMPLE_ARGS = Object.freeze({
   check_feasibility: { broker: "alpaca", asset_class: "us_equity", account: "margin", capital_usd: 250000, orders_per_rebalance: 40, rebalances_per_year: 52, turnover_per_year: 8, adv_usd: 20000000, daily_volatility: 0.02, copies: 1, expected_gross_return: 0.12, spread_and_fees_bps: 3 },
   check_leakage: { action: "plan", observations: 2520, prefixes: 5 },
   placebo_test: { action: "plan", data_file: "prices.csv" },
+  ledger_record_trial: { ledger: "sma-search", label: "sma 20/100, 5 bps", sharpe_annualized: 1.1, observations: 1260, periods_per_year: 252, params: { fast: 20, slow: 100 } },
+  ledger_summary: { ledger: "sma-search" },
+  ledger_export: { ledger: "sma-search" },
 });
 
 const usesFile = (args) => Object.keys(args).some((k) => k.endsWith("_file"));

package/src/ledger-core.mjs

@@ -0,0 +1,138 @@
+// mcp/src/ledger-core.mjs
+//
+// The trial ledger: every strategy variant an agent tries in one research search, recorded as it is
+// tried, so the result it finally reports is judged against the whole search instead of alone. An
+// agent can run thousands of variants in an hour; the best of them looks skilled by luck alone, and
+// the size of that luck depends on how many were tried and how spread out their results were.
+//
+// Each trial is one line in a hash chain (sha256 over the previous hash and the trial's canonical
+// JSON), so an exported ledger shows any trial removed, reordered or edited after the fact. The
+// statistics reuse the site's checked cores: perPeriodMoments and calculateDsr (pinned to
+// standards/validation-api/vectors.json) and pboCscv. Pure functions only; storage is in ledger.mjs.
+import { createHash } from "node:crypto";
+
+import { calculateDsr, expectedMaxStandardNormal } from "./local/js/dsr-core.js";
+import { finiteNumbers, perPeriodMoments } from "./local/js/moments-core.js";
+import { pboCscv } from "./local/js/pbo-core.js";
+
+export const LEDGER_SCHEMA = "canli.trial-ledger.v1";
+export const GENESIS = "0".repeat(64);
+export const MAX_TRIALS = 10000;
+export const MAX_RETURNS = 20000;
+
+export const LEDGER_LIMITS_TEXT = Object.freeze([
+  "The ledger counts the trials recorded in it; a trial run but never recorded is invisible to it, so the correction is only as complete as the record.",
+  "Treating every recorded trial as independent overstates the search when trials are correlated variants of one idea, which makes the deflated Sharpe conservative; pass effective_trials when you have a better estimate.",
+  "A deflated Sharpe ratio is a probability that the best trial's true Sharpe is above zero after the search; it is not a forecast of returns and not a recommendation.",
+]);
+
+const sha256 = (text) => createHash("sha256").update(text).digest("hex");
+
+// Keys sorted at every level, so the same trial always hashes the same way.
+export function canonicalJson(value) {
+  if (Array.isArray(value)) return `[${value.map(canonicalJson).join(",")}]`;
+  if (value && typeof value === "object") return `{${Object.keys(value).sort().map((k) => `${JSON.stringify(k)}:${canonicalJson(value[k])}`).join(",")}}`;
+  return JSON.stringify(value);
+}
+
+// A trial as the agent describes it -> the record that is chained. Exactly one of returns, or a
+// Sharpe with its observation count. periodsPerYear annualizes; it is fixed for the whole ledger.
+export function buildTrial({ label, returns, sharpe_annualized, observations, params, periodsPerYear, n, prev, recordedAt }) {
+  if (typeof label !== "string" || !label.trim()) throw new RangeError("label must be a non-empty string naming the variant");
+  if (!(periodsPerYear > 0)) throw new RangeError("periods_per_year must be greater than zero");
+  const hasReturns = returns !== undefined && returns !== null;
+  const hasSharpe = sharpe_annualized !== undefined && sharpe_annualized !== null;
+  if (hasReturns === hasSharpe) throw new RangeError("give either returns or sharpe_annualized with observations, not both and not neither");
+  let trial;
+  if (hasReturns) {
+    const x = finiteNumbers(returns, "returns");
+    if (x.length > MAX_RETURNS) throw new RangeError(`returns holds ${x.length} observations; the limit is ${MAX_RETURNS}`);
+    const m = perPeriodMoments(x);
+    trial = { sharpe_annualized: m.sharpe_per_period * Math.sqrt(periodsPerYear), observations: m.observations, skew: m.skew, non_excess_kurtosis: m.non_excess_kurtosis, returns: x };
+  } else {
+    if (!Number.isFinite(sharpe_annualized)) throw new RangeError("sharpe_annualized must be a finite number");
+    if (!Number.isInteger(observations) || observations < 2) throw new RangeError("observations must be an integer of at least 2 when a Sharpe is given");
+    trial = { sharpe_annualized, observations, skew: null, non_excess_kurtosis: null, returns: null };
+  }
+  const body = { n, label: label.trim(), params: params ?? null, recorded_at: recordedAt, ...trial };
+  const hash = sha256(prev + canonicalJson(body));
+  return { ...body, prev, hash };
+}
+
+// Recomputes the chain. Returns { ok, head, trials } or { ok: false, at, reason }.
+export function verifyChain(trials) {
+  let prev = GENESIS;
+  for (let i = 0; i < trials.length; i += 1) {
+    const { prev: recordedPrev, hash, ...body } = trials[i];
+    if (body.n !== i + 1) return { ok: false, at: i + 1, reason: `trial ${i + 1} is numbered ${body.n}` };
+    if (recordedPrev !== prev) return { ok: false, at: i + 1, reason: `trial ${i + 1} does not follow trial ${i}` };
+    if (sha256(prev + canonicalJson(body)) !== hash) return { ok: false, at: i + 1, reason: `trial ${i + 1} was changed after it was recorded` };
+    prev = hash;
+  }
+  return { ok: true, head: prev, trials: trials.length };
+}
+
+const sampleSd = (xs) => {
+  if (xs.length < 2) return null;
+  const mean = xs.reduce((a, b) => a + b, 0) / xs.length;
+  return Math.sqrt(xs.reduce((a, b) => a + (b - mean) ** 2, 0) / (xs.length - 1));
+};
+
+// The largest even number of CSCV blocks (at most 16) that leaves at least two periods per block.
+const cscvSplits = (length) => {
+  for (let s = 16; s >= 4; s -= 2) if (length / s >= 2) return s;
+  return null;
+};
+
+const fmt = (x, d = 2) => (x === null || x === undefined ? "n/a" : Number(x).toFixed(d));
+
+// The verdict on the whole search so far.
+export function summarize({ trials, periodsPerYear, effectiveTrials }) {
+  const n = trials.length;
+  if (n === 0) return { trials: 0, plain_reading: "No trials are recorded in this ledger yet.", limits: LEDGER_LIMITS_TEXT };
+  const sharpes = trials.map((t) => t.sharpe_annualized);
+  const bestIndex = sharpes.reduce((bi, s, i) => (s > sharpes[bi] ? i : bi), 0);
+  const best = trials[bestIndex];
+  const spread = sampleSd(sharpes);
+  const counted = effectiveTrials ?? n;
+  const assumedShape = best.skew === null;
+  let deflated = null;
+  if (counted >= 2 && spread !== null) {
+    const d = calculateDsr({
+      observed_sharpe_annualized: best.sharpe_annualized, observations: best.observations, periods_per_year: periodsPerYear,
+      skew: assumedShape ? 0 : best.skew, non_excess_kurtosis: assumedShape ? 3 : best.non_excess_kurtosis,
+      effective_independent_trials: counted, cross_trial_sharpe_sd_annualized: spread,
+    });
+    deflated = { deflated_sharpe_ratio: d.deflated_sharpe_ratio, psr_against_zero: d.psr_against_zero, expected_max_sharpe_annualized: d.expected_max_sharpe_annualized, search_haircut_annualized: d.search_haircut_annualized };
+  }
+  let overfitting = null;
+  const withReturns = trials.filter((t) => Array.isArray(t.returns));
+  const lengths = new Set(withReturns.map((t) => t.returns.length));
+  if (withReturns.length === n && n >= 4 && lengths.size === 1) {
+    const length = [...lengths][0];
+    const splits = cscvSplits(length);
+    if (splits) {
+      const matrix = Array.from({ length }, (_, r) => withReturns.map((t) => t.returns[r]));
+      const p = pboCscv(matrix, { nSplits: splits });
+      overfitting = { probability_of_backtest_overfitting: p.pbo, n_splits: splits, combinations: p.n_combinations };
+    }
+  }
+  const luck = spread !== null && counted >= 2 ? spread * expectedMaxStandardNormal(counted) : null;
+  const sentences = [
+    `Across ${n} recorded trial${n === 1 ? "" : "s"}, the best is "${best.label}" with an annualized Sharpe of ${fmt(best.sharpe_annualized)}.`,
+  ];
+  if (luck !== null) sentences.push(`With this spread of results, the best of ${counted} skill-less trials would be expected to reach about ${fmt(luck)} by luck alone.`);
+  if (deflated) sentences.push(`Its deflated Sharpe ratio is ${fmt(deflated.deflated_sharpe_ratio)}: the probability that its true Sharpe is above zero once the search is counted.`);
+  else sentences.push("A deflated Sharpe needs at least two trials with different results.");
+  if (overfitting) sentences.push(`The probability of backtest overfitting (CSCV, ${overfitting.n_splits} blocks) is ${fmt(overfitting.probability_of_backtest_overfitting)}.`);
+  if (assumedShape) sentences.push("The best trial was recorded as a Sharpe without returns, so normal returns were assumed for it.");
+  return {
+    trials: n, effective_trials: counted,
+    best: { n: best.n, label: best.label, params: best.params, sharpe_annualized: best.sharpe_annualized, observations: best.observations },
+    cross_trial_sharpe_sd_annualized: spread,
+    expected_best_by_luck_annualized: luck,
+    deflated, overfitting,
+    plain_reading: sentences.join(" "),
+    limits: LEDGER_LIMITS_TEXT,
+  };
+}

package/src/ledger.mjs

@@ -0,0 +1,124 @@
+// mcp/src/ledger.mjs
+//
+// The trial-ledger tools (toolset "ledger", local stdio mode): ledger_record_trial, ledger_summary
+// and ledger_export. The rules and statistics are in ledger-core.mjs; this file stores each ledger as
+// one JSON-lines file (a header, then one chained trial per line) under CANLI_LEDGER_DIR, by default
+// ~/.canli/ledgers, so a search survives a restart of the agent or the server. Nothing leaves the
+// machine. A ledger is appended to, never rewritten.
+import { randomBytes } from "node:crypto";
+import { appendFileSync, existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
+import { homedir } from "node:os";
+import { join } from "node:path";
+import { z } from "zod";
+
+import { GENESIS, LEDGER_LIMITS_TEXT, LEDGER_SCHEMA, MAX_TRIALS, buildTrial, summarize, verifyChain } from "./ledger-core.mjs";
+import { readSeriesFile } from "./series-file.mjs";
+
+export const LEDGER_TOOLS = Object.freeze(["ledger_record_trial", "ledger_summary", "ledger_export"]);
+
+export const ledgerDir = (env = process.env) => env.CANLI_LEDGER_DIR || join(homedir(), ".canli", "ledgers");
+const ID = /^[a-z0-9][a-z0-9-]{2,63}$/;
+const text = (value) => ({ content: [{ type: "text", text: JSON.stringify(value) }], structuredContent: value });
+
+function pathFor(dir, id) {
+  if (!ID.test(id)) throw new RangeError("ledger must be 3 to 64 lowercase letters, digits or dashes, starting with a letter or digit");
+  return join(dir, `${id}.jsonl`);
+}
+
+export function readLedger(dir, id) {
+  const path = pathFor(dir, id);
+  if (!existsSync(path)) return null;
+  const lines = readFileSync(path, "utf8").split("\n").filter(Boolean).map((line) => JSON.parse(line));
+  const [header, ...trials] = lines;
+  if (header?.schema !== LEDGER_SCHEMA) throw new Error(`${path} is not a ${LEDGER_SCHEMA} file`);
+  const check = verifyChain(trials);
+  if (!check.ok) throw new Error(`ledger ${id} fails its own hash chain at trial ${check.at}: ${check.reason}`);
+  return { header, trials, head: check.head };
+}
+
+export function recordTrial(dir, input, now = () => new Date().toISOString()) {
+  const id = input.ledger ?? `search-${randomBytes(4).toString("hex")}`;
+  let ledger = readLedger(dir, id);
+  if (!ledger) {
+    if (!(input.periods_per_year > 0)) throw new RangeError("periods_per_year is required for a ledger's first trial (252 for daily stock returns, 365 for daily crypto, 12 for monthly)");
+    mkdirSync(dir, { recursive: true });
+    const header = { schema: LEDGER_SCHEMA, ledger: id, periods_per_year: input.periods_per_year, created_at: now() };
+    writeFileSync(pathFor(dir, id), `${JSON.stringify(header)}\n`, { flag: "wx" });
+    ledger = { header, trials: [], head: GENESIS };
+  } else if (input.periods_per_year !== undefined && input.periods_per_year !== ledger.header.periods_per_year) {
+    throw new RangeError(`ledger ${id} annualizes with periods_per_year ${ledger.header.periods_per_year}; record this trial in a new ledger`);
+  }
+  if (ledger.trials.length >= MAX_TRIALS) throw new RangeError(`ledger ${id} holds the maximum of ${MAX_TRIALS} trials`);
+  const returns = input.returns_file ? readSeriesFile(input.returns_file, input.column) : input.returns;
+  const trial = buildTrial({ label: input.label, returns, sharpe_annualized: input.sharpe_annualized, observations: input.observations, params: input.params,
+    periodsPerYear: ledger.header.periods_per_year, n: ledger.trials.length + 1, prev: ledger.head, recordedAt: now() });
+  appendFileSync(pathFor(dir, id), `${JSON.stringify(trial)}\n`);
+  const trials = [...ledger.trials, trial];
+  return { ledger: id, trial: trial.n, chain_head: trial.hash, ...summarize({ trials, periodsPerYear: ledger.header.periods_per_year }) };
+}
+
+export function ledgerSummary(dir, { ledger: id, effective_trials }) {
+  const ledger = readLedger(dir, id);
+  if (!ledger) throw new RangeError(`no ledger named ${id} in ${dir}`);
+  return { ledger: id, chain_head: ledger.head, ...summarize({ trials: ledger.trials, periodsPerYear: ledger.header.periods_per_year, effectiveTrials: effective_trials }) };
+}
+
+// The whole ledger, verifiable offline: recompute each hash from the previous one and the trial body.
+export function ledgerExport(dir, { ledger: id, include_returns = false }) {
+  const ledger = readLedger(dir, id);
+  if (!ledger) throw new RangeError(`no ledger named ${id} in ${dir}`);
+  return {
+    ...ledger.header, chain_head: ledger.head, trials: ledger.trials.length, file: pathFor(dir, id),
+    records: ledger.trials.map((t) => (include_returns ? t : { ...t, returns: t.returns ? `${t.returns.length} values (in the file)` : null })),
+    verify: "Each record's hash is sha256(previous hash + canonical JSON of the record without prev and hash), starting from 64 zeros; the file keeps every return value so the chain can be recomputed exactly.",
+    limits: LEDGER_LIMITS_TEXT,
+  };
+}
+
+const ledgerId = z.string().regex(ID).describe("The ledger's name. Omit on the first trial to create one; reuse the name it returns for every later trial of the same search.");
+export const recordTrialInput = z.object({
+  ledger: ledgerId.optional(),
+  label: z.string().min(1).max(200).describe("A short name for this variant, for example 'sma 20/100, 5 bps'."),
+  returns: z.array(z.number()).max(20000).optional().describe("The variant's period returns as decimals. Give returns, or returns_file, or sharpe_annualized with observations."),
+  returns_file: z.string().optional().describe("A CSV or JSON file of the variant's returns, read on this machine."),
+  column: z.string().optional().describe("Column of returns_file to read, when it has several."),
+  sharpe_annualized: z.number().optional().describe("The variant's annualized Sharpe ratio, when its returns are not available."),
+  observations: z.number().int().min(2).optional().describe("How many return periods the Sharpe was measured on."),
+  periods_per_year: z.number().positive().optional().describe("Required on a ledger's first trial: 252 daily stocks, 365 daily crypto, 52 weekly, 12 monthly."),
+  params: z.record(z.string(), z.any()).optional().describe("The variant's parameters, recorded as given."),
+});
+export const summaryInput = z.object({
+  ledger: ledgerId,
+  effective_trials: z.number().int().min(2).optional().describe("Use instead of the recorded count when many trials are near-copies of each other."),
+});
+export const exportInput = z.object({
+  ledger: ledgerId,
+  include_returns: z.boolean().optional().describe("Include every return value in the reply (large). The file always keeps them."),
+});
+const summaryShape = {
+  ledger: z.string(), chain_head: z.string().optional(), trial: z.number().optional(), trials: z.number(), effective_trials: z.number().optional(),
+  best: z.object({}).passthrough().optional(), cross_trial_sharpe_sd_annualized: z.number().nullable().optional(), expected_best_by_luck_annualized: z.number().nullable().optional(),
+  deflated: z.object({}).passthrough().nullable().optional(), overfitting: z.object({}).passthrough().nullable().optional(), plain_reading: z.string(), limits: z.array(z.string()),
+};
+export const summaryOutput = z.object(summaryShape).passthrough();
+export const exportOutput = z.object({ ledger: z.string(), chain_head: z.string(), trials: z.number(), records: z.array(z.object({}).passthrough()), verify: z.string(), limits: z.array(z.string()) }).passthrough();
+
+export const LEDGER_TOOL_DESCRIPTIONS = Object.freeze({
+  ledger_record_trial: `Record every strategy variant you try in one search, as you try it (its returns, or its Sharpe and sample size). Returns the trial count and the best trial's deflated Sharpe after counting all of them, so a lucky best of many is not reported as skill. Local; kept in a hash-chained file. ${LEDGER_LIMITS_TEXT[0]}`,
+  ledger_summary: `The verdict on a whole search: best trial, the Sharpe the best of that many skill-less trials reaches by luck, its deflated Sharpe, and the probability of backtest overfitting when trials share aligned returns. ${LEDGER_LIMITS_TEXT[2]}`,
+  ledger_export: "Export a ledger with its hash chain, so anyone can check that no trial was removed, reordered or changed after it was recorded.",
+});
+
+const ANNOTATIONS_WRITE = { readOnlyHint: false, destructiveHint: false, idempotentHint: false, openWorldHint: false };
+const ANNOTATIONS_READ = { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false };
+
+export function ledgerToolSpecs(session) {
+  const dir = session.ledgerDir ?? ledgerDir();
+  const wrap = (fn) => async (args) => text(fn(dir, args));
+  const spec = (name, title, input, output, handler, annotations) => [name, { title, annotations: { title, ...annotations }, description: LEDGER_TOOL_DESCRIPTIONS[name], inputSchema: input, outputSchema: output }, handler];
+  return [
+    spec("ledger_record_trial", "Record a trial in a search ledger", recordTrialInput, summaryOutput, wrap(recordTrial), ANNOTATIONS_WRITE),
+    spec("ledger_summary", "Judge a whole search", summaryInput, summaryOutput, wrap(ledgerSummary), ANNOTATIONS_READ),
+    spec("ledger_export", "Export a ledger with its hash chain", exportInput, exportOutput, wrap(ledgerExport), ANNOTATIONS_READ),
+  ];
+}

package/src/schemas.mjs

@@ -7,6 +7,7 @@
 // docs/superpowers/specs/2026-09-05-developer-key-validation-api-design.md section 4 for the
 // validator semantics these shapes mirror.
 import { z } from "zod";
+import { NULL_ZOO_V1 } from "./local/js/null-zoo-v1-sizes.js";
 
 // One description per input field, shared by every shape that uses the field, so a client reads
 // the same meaning, unit and example wherever the field appears. Agents choose arguments from
@@ -331,13 +332,26 @@ export const LIMITS_SENTENCES = Object.freeze({
 export const REGISTRY_LIMITS_CLAUSE = "is not admission to anything and is not a forecast.";
 export const REGISTRY_DESCRIPTION_MAX = 100;
 
+// The headline tests audit_backtest can run, by the names the shipped size table uses. The
+// description reads the chosen ones from that table, so it cannot name a test the audit does not run.
+export const HEADLINE_TEST_NAMES = Object.freeze({
+  reality_check: "White's Reality Check",
+  spa_consistent: "Hansen's SPA",
+  spa_upper: "Hansen's SPA (upper p-value)",
+  luck_trials_lo: "luck trials with Lo's correction",
+});
+const headlineName = (test) => {
+  if (!HEADLINE_TEST_NAMES[test]) throw new Error(`No description name for the headline test ${test}`);
+  return HEADLINE_TEST_NAMES[test];
+};
+
 // One tool description per tool, each stating (in one sentence lifted from the boundary language
 // above) what the tool's result cannot be used to claim, so an agent sees this before it ever
 // calls the tool, not only inside the returned envelope.
 export const TOOL_DESCRIPTIONS = Object.freeze({
   get_key: `Issue a free validation key for this session. Rarely needed: the first validation issues one itself unless CANLI_KEY or local mode is set, and the read tools need none. ${LIMITS_SENTENCES.quotas}`,
   validate_deflated_sharpe: `Deflated Sharpe ratio: the probability (0 to 1) that the selected strategy's Sharpe beats the best that luck gives across the variants tried, with the probabilistic Sharpe and that luck benchmark. Send the seven statistics or a return series. With every variant's returns use validate_overfitting; luck as a trial count, validate_luck_trials; a multiple-testing haircut, validate_haircut_sharpe. ${LIMITS_SENTENCES.notAdmission}`,
-  audit_backtest: `One-call audit of a strategy's returns. Headline: one test whose false-positive rate was measured on nine return shapes (Hansen's SPA with variants). Then deflated Sharpe, minimum track record and, with variants, overfitting and out-of-sample decay, each with its receipt, plus fix_next. Send every variant tried (variants_file) so trials are counted; point returns_file at a CSV instead of pasting. ${LIMITS_SENTENCES.notAdmission}`,
+  audit_backtest: `One-call audit of a strategy's returns. Headline: one test whose false-positive rate was measured on nine return shapes (${headlineName(NULL_ZOO_V1.with_variants.test)} with variants, ${headlineName(NULL_ZOO_V1.single_series.test)} without). Then deflated Sharpe, minimum track record and, with variants, overfitting and out-of-sample decay, each with its receipt, plus fix_next. Send every variant tried (variants_file) so trials are counted; point returns_file at a CSV instead of pasting. ${LIMITS_SENTENCES.notAdmission}`,
   validate_overfitting: `Probability of backtest overfitting (0 to 1) by CSCV: how often the in-sample best variant falls below the out-of-sample median. Needs every variant's returns (periods by variants); with summary statistics only, use validate_deflated_sharpe. ${LIMITS_SENTENCES.notAdmission}`,
   validate_reality_check: `Data-snooping tests on every variant a search tried: Hansen's SPA p-value that the best beat the benchmark only by luck, White's Reality Check, and the variants Romano-Wolf StepM finds better. Send all variants tried, not only the winners. ${LIMITS_SENTENCES.notAdmission}`,
   validate_paper_evidence: `Checks paper-evidence.v0 structure and disclosures. With CANLI_LOCAL=1, record_file reads an export bundle and journal_file verifies signatures, source hashes and recomputed claims. Without a journal, source facts and signatures are unchecked. Local files are never uploaded. ${LIMITS_SENTENCES.scope}`,

package/src/server.mjs

@@ -14,6 +14,7 @@ import { computeLocally } from "./local.mjs";
 import { validateLocalJournalEvidence } from "./journal-evidence.mjs";
 import { readMatrixFile, readSeriesFile } from "./series-file.mjs";
 import { LAB_TOOLS, labToolSpecs, registerCodeResources, registerLabPrompts } from "./lab.mjs";
+import { LEDGER_TOOLS, ledgerToolSpecs } from "./ledger.mjs";
 import { verifyReceipt } from "./local/js/receipt-statement.js";
 import { annualizeDecay, blockBootstrapT, fixNext, neweyWestT, nullZooFamily, returnShape, searchFromVariants, sharpeInterval } from "./local/js/audit-core.js";
 import { NULL_ZOO_V1 } from "./local/js/null-zoo-v1-sizes.js";
@@ -80,12 +81,15 @@ export const TOOLSETS = Object.freeze({
   company: Object.freeze(["company_financial_history"]),
   status: Object.freeze(["service_status"]),
   lab: Object.freeze([...LAB_TOOLS]),
+  ledger: Object.freeze([...LEDGER_TOOLS]),
 });
 
 // CANLI_TOOLSETS=validate,company (or ?toolsets= on the hosted endpoint): a comma-separated list of
 // TOOLSETS names, or "all". Empty or unsubstituted means the default; an unknown name is refused, so
 // a typo never silently leaves a client without the tools it asked for.
-export const DEFAULT_TOOLSETS = Object.freeze(Object.keys(TOOLSETS).filter((name) => name !== "company"));
+// The trial ledger is opt-in (CANLI_TOOLSETS=...,ledger or "all"): it keeps files on this machine, and
+// the default tool list, re-sent every turn, stays the size it was.
+export const DEFAULT_TOOLSETS = Object.freeze(Object.keys(TOOLSETS).filter((name) => name !== "company" && name !== "ledger"));
 export function configuredToolsets(value) {
   const v = configuredKey(value);
   if (!v) return [...DEFAULT_TOOLSETS];
@@ -339,8 +343,9 @@ async function runValidator(session, tool, path, body) {
 const sameJson = (a, b) => JSON.stringify(a) === JSON.stringify(b);
 
 // audit_backtest: one calibrated test first, then the checks behind it.
-//   headline  with every variant's returns, Hansen's SPA (validate_reality_check, with its receipt);
-//             with one series, the single-series test Null Zoo v1 chose. Either way the result quotes
+//   headline  with every variant's returns, the test Null Zoo v1 chose from validate_reality_check's
+//             result (White's Reality Check since 0.12.0), with its receipt; with one series, the
+//             single-series test it chose. Either way the result quotes
 //             that test's measured false-positive rate on returns shaped like these
 //             (local/js/null-zoo-v1-sizes.js) and the worst across the nine families.
 //   checks    the deflated Sharpe (an estimate), the minimum track record and, with variants, CSCV
@@ -781,6 +786,8 @@ export function registerTools(server, session) {
     (args) => toolCompanyFinancialHistory(session, args),
   );
   for (const [name, config, handler] of labToolSpecs(session)) register(name, config, handler);
+  // The ledger keeps state in files on this machine; the hosted endpoint is stateless, so it never lists it.
+  if (!session.hosted) for (const [name, config, handler] of ledgerToolSpecs(session)) register(name, config, handler);
   return catalog;
 }