canli-validation-mcp 0.9.1 → 0.10.0

package/README.md

@@ -7,8 +7,9 @@
 
 **Is your best backtest real, or just the luckiest of the variants you tried?** This MCP server
 lets Claude, Cursor or any MCP client answer that with the standard corrections: the deflated
-Sharpe ratio, the CSCV probability of backtest overfitting, the minimum track record length, the
-haircut Sharpe ratio, and luck-equivalent trials. Free and MIT-licensed.
+Sharpe ratio, the CSCV probability of backtest overfitting, data-snooping tests on every variant
+tried (Hansen's SPA, White's Reality Check, Romano-Wolf StepM), the minimum track record length,
+the haircut Sharpe ratio, and luck-equivalent trials. Free and MIT-licensed.
 
 ## Quick start
 
@@ -34,9 +35,10 @@ other stdio client (Cursor, VS Code) run the same `npx` command; see "Claude Des
 ## What it does
 
 An MCP (Model Context Protocol) server over canlicapital.com's free, keyed validation API. It
-gives a coding agent fourteen tools: issue a free key, run the eight validators (deflated Sharpe,
-CSCV overfitting, paper-evidence conformance, breadth ceiling, minimum track record length,
-minimum backtest length, haircut Sharpe ratio, luck-equivalent trials),
+gives a coding agent fifteen tools: issue a free key, run the nine validators (deflated Sharpe,
+CSCV overfitting, data-snooping tests (Hansen's SPA, White's Reality Check and Romano-Wolf StepM),
+paper-evidence conformance, breadth ceiling, minimum track record length, minimum backtest length,
+haircut Sharpe ratio, luck-equivalent trials),
 audit one backtest with three of them in a single call, fetch a stored receipt, verify a
 receipt's signature offline, read service status, and read a company's reported financial history
 from SEC filings. Every validation result carries, beside the number, the sentences that say what
@@ -80,6 +82,7 @@ repository for the full design.
 | `validate_backtest_length` | `POST /api/v1/validate/backtest-length` | yes |
 | `validate_haircut_sharpe` | `POST /api/v1/validate/haircut-sharpe` | yes |
 | `validate_luck_trials` | `POST /api/v1/validate/luck-trials` | yes |
+| `validate_reality_check` | `POST /api/v1/validate/reality-check` (a `matrix_file` is read on your machine and sent as numbers) | yes |
 | `audit_backtest` | the deflated Sharpe, track record and, with `variants`, overfitting routes, one validation each | yes |
 | `verify_receipt` | `GET /api/v1/receipts/{id}` when given an id; the checks run locally | no |
 | `get_receipt` | `GET /api/v1/receipts/{id}` | no |
@@ -97,6 +100,24 @@ repository for the full design.
 Sending fields from both shapes, or from neither, is rejected before any request leaves the
 process; see `src/schemas.mjs`.
 
+`validate_reality_check` takes the returns of every variant the search tried, one row per period
+and one column per variant, as `matrix` or as `matrix_file` (a CSV or JSON on your machine), and an
+optional `benchmark` series; with none, variants are tested against zero. It runs three tests on one
+seeded stationary bootstrap:
+
+- Hansen's SPA: the chance that the best variant's studentized excess return is this good if no
+  variant has an edge, with lower and upper bounds (`spa.p_value`, the consistent p-value, is the
+  headline, and `monte_carlo_se` its sampling error);
+- White's Reality Check: the same question without studentizing, so one volatile variant can
+  dominate it;
+- Romano and Wolf's StepM: which variants beat the benchmark, with the familywise error held at
+  `alpha`.
+
+A result reproduces exactly from its `seed`. On three fixed-seed cases the p-values agree with
+Python's `arch` 8.0 and with a numpy transcription of Hansen's formulas within Monte Carlo error, and
+the StepM sets match `arch`'s (`js/snooping-core.test.js`). Only the variants sent are counted: a
+search that tried more than it sends makes luck look smaller than it was.
+
 `company_financial_history` is different from the other tools: it reads the public company
 reference at canlicapital.com, not the validation API. Give it a CIK (1 to 10 digits) to list a
 company's available financial histories, or a CIK and a us-gaap concept such as `Revenues` to
@@ -317,7 +338,7 @@ On a breadth result this is about half the text. Set `CANLI_FULL_ENVELOPE=1` to
 ## Toolsets (tokens)
 
 A client sends the model the whole tool list on every turn, and it is most of each turn's prompt:
-a validation result is a few hundred tokens, the list of all fourteen tools several thousand. A
+a validation result is a few hundred tokens, the list of all fifteen tools several thousand. A
 client that needs one kind of tool can list only that kind, with `CANLI_TOOLSETS` (stdio) or
 `?toolsets=` (hosted endpoint). The default is every tool.
 
@@ -333,9 +354,9 @@ tokenizers give different absolute counts), in the shape an OpenAI-style client
 
 | CANLI_TOOLSETS | tools | tokens per turn | of all |
 |---|---|---|---|
-| `all` | 14 | 3,834 | 100% |
-| `validate` | 10 | 3,166 | 83% |
-| `receipts` | 2 | 312 | 8% |
+| `all` | 15 | 4,194 | 100% |
+| `validate` | 11 | 3,526 | 84% |
+| `receipts` | 2 | 312 | 7% |
 | `company` | 1 | 273 | 7% |
 | `status` | 1 | 89 | 2% |
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "canli-validation-mcp",
-  "version": "0.9.1",
+  "version": "0.10.0",
   "description": "Check whether a backtest is real: deflated Sharpe, CSCV probability of backtest overfitting, minimum track record and backtest length, haircut Sharpe and luck-equivalent trials, as an MCP server. Local mode, hosted endpoint, signed receipts.",
   "keywords": [
     "mcp",

package/src/local/js/snooping-core.js

@@ -0,0 +1,185 @@
+// =============================================================================
+// snooping-core.js
+// -----------------------------------------------------------------------------
+// Data-snooping tests for a search: did the best of K variants beat the benchmark by more than the
+// best of K would by luck? One stationary bootstrap (Politis and Romano 1994) feeds three tests:
+//
+//   White's Reality Check (2000)     the best mean excess return against the bootstrap distribution
+//                                    of the best recentred mean. Not studentized, so one volatile
+//                                    variant can dominate it.
+//   Hansen's SPA (2005)              the best t-statistic, each variant studentized by the
+//                                    Politis-Romano variance of its mean. Three recentrings give a
+//                                    lower bound, a consistent p-value and an upper bound: poor
+//                                    variants are not allowed to inflate the p-value.
+//   Romano and Wolf's StepM (2005)   step-down: which variants beat the benchmark, controlling the
+//                                    familywise error at alpha.
+//
+// d[t][k] is variant k's return minus the benchmark's in period t; higher is better. Every draw
+// comes from makeRandom(seed), so a result reproduces exactly from its inputs. The reference
+// implementation it is checked against (Python's arch 8.0, and a separate numpy transcription of
+// Hansen's formulas for the studentized SPA) draws with numpy, so agreement is within Monte Carlo
+// error, not to the digit; scripts/research/reality-check/ holds the comparison.
+// =============================================================================
+
+import { makeRandom } from "./selection-risk-core.js";
+
+/** Indices of one stationary-bootstrap resample of n periods with mean block length `block`. */
+export function stationaryIndices(n, block, next, out = new Int32Array(n)) {
+  const p = 1 / block;
+  let at = Math.floor(next() * n);
+  out[0] = at;
+  for (let t = 1; t < n; t += 1) {
+    at = next() < p ? Math.floor(next() * n) : at + 1 === n ? 0 : at + 1;
+    out[t] = at;
+  }
+  return out;
+}
+
+/** round(n^(1/3)), at least 1: the default mean block length. */
+export const defaultBlock = (n) => Math.max(1, Math.round(Math.cbrt(n)));
+
+// Weights below this are dropped from the variance sum. Each term they multiply is bounded by the
+// sample variance, so what is dropped is under 2e-14 * n of it (4e-10 at 20,000 periods).
+const KAPPA_FLOOR = 1e-14;
+
+/**
+ * n times the variance of each column's mean under the stationary bootstrap, in closed form
+ * (Politis and Romano 1994, as Hansen 2005 uses it): gamma_0 + 2 * sum_i kappa_i * gamma_i, with
+ * kappa_i = (1 - i/n)(1 - p)^i + (i/n)(1 - p)^(n - i). Lags whose weight is below KAPPA_FLOOR are
+ * skipped, which keeps the cost near n * k * 30 * block rather than n^2 * k.
+ */
+export function bootstrapVariance(d, n, k, block, means) {
+  const p = 1 / block;
+  const out = new Float64Array(k);
+  const cross = (lag, weight) => {
+    for (let t = 0; t + lag < n; t += 1) {
+      const a = t * k;
+      const b = (t + lag) * k;
+      for (let j = 0; j < k; j += 1) out[j] += weight * (d[a + j] - means[j]) * (d[b + j] - means[j]);
+    }
+  };
+  cross(0, 1 / n);
+  const q = 1 - p;
+  for (let i = 1; i < n; i += 1) {
+    const kappa = (1 - i / n) * q ** i + (i / n) * q ** (n - i);
+    if (kappa < KAPPA_FLOOR) {
+      // The first term has decayed; the wrap-around term (i/n)(1 - p)^(n - i) only reaches the
+      // floor again once n - i <= log(floor) / log(1 - p). Skip to there.
+      const tail = q > 0 ? Math.log(KAPPA_FLOOR) / Math.log(q) : 0;
+      const jump = Math.floor(n - tail);
+      if (jump > i + 1) i = Math.min(jump, n) - 1;
+      continue;
+    }
+    cross(i, (2 * kappa) / n);
+  }
+  return out;
+}
+
+/**
+ * All three tests on one bootstrap. `d` is a row-major Float64Array of n periods by k variants.
+ * Returns means, the variance estimates, the bootstrap maxima under each recentring and the
+ * StepM rejections; p-values are shares of `reps` draws strictly above the observed statistic.
+ */
+export function snoopingTests(d, n, k, { block = defaultBlock(n), reps = 1000, seed = 42, alpha = 0.05, studentize = true } = {}) {
+  if (!(n >= 2 && k >= 1)) throw new RangeError("need at least 2 periods and 1 variant");
+  if (!(block >= 1 && block <= n)) throw new RangeError(`block_length must be from 1 to the number of periods (${n})`);
+  if (!Number.isInteger(reps) || reps < 100) throw new RangeError("reps must be an integer of at least 100");
+  if (!(alpha > 0 && alpha < 1)) throw new RangeError("alpha must be between 0 and 1");
+  const means = new Float64Array(k);
+  for (let t = 0; t < n; t += 1) for (let j = 0; j < k; j += 1) means[j] += d[t * k + j];
+  for (let j = 0; j < k; j += 1) means[j] /= n;
+  const variance = bootstrapVariance(d, n, k, block, means);
+  const sd = new Float64Array(k);
+  for (let j = 0; j < k; j += 1) sd[j] = Math.sqrt(Math.max(variance[j], 0));
+  // A variant whose excess return never varies has no sampling distribution to studentize by.
+  const usable = Array.from(sd, (s) => s > 0);
+  if (!usable.some(Boolean)) throw new RangeError("every variant's excess return is constant; there is nothing to test");
+  const root = Math.sqrt(n);
+  const scale = (j) => (studentize ? root / sd[j] : 1);
+  // Hansen's recentrings: lower keeps poor variants at zero, consistent drops only those far below
+  // zero, upper recentres everything (White's null, the least favourable configuration).
+  const threshold = (j) => -Math.sqrt((variance[j] / n) * 2 * Math.log(Math.log(n)));
+  const centre = [
+    Float64Array.from(means, (m) => Math.max(m, 0)),
+    Float64Array.from(means, (m, j) => (m >= threshold(j) ? m : 0)),
+    Float64Array.from(means),
+  ];
+  let observed = Number.NEGATIVE_INFINITY;
+  let best = -1;
+  for (let j = 0; j < k; j += 1) {
+    if (!usable[j]) continue;
+    const s = means[j] * scale(j);
+    if (s > observed) { observed = s; best = j; }
+  }
+  let rcObserved = Number.NEGATIVE_INFINITY;
+  for (let j = 0; j < k; j += 1) if (usable[j]) rcObserved = Math.max(rcObserved, means[j]);
+
+  const next = makeRandom(seed);
+  const idx = new Int32Array(n);
+  const star = new Float64Array(k);
+  const boot = new Float64Array(reps * k);   // bootstrap mean of each variant, per draw
+  for (let r = 0; r < reps; r += 1) {
+    stationaryIndices(n, block, next, idx);
+    star.fill(0);
+    for (let t = 0; t < n; t += 1) {
+      const row = idx[t] * k;
+      for (let j = 0; j < k; j += 1) star[j] += d[row + j];
+    }
+    for (let j = 0; j < k; j += 1) boot[r * k + j] = star[j] / n;
+  }
+
+  const spaStat = Math.max(observed, 0);
+  const exceed = [0, 0, 0];
+  let rcExceed = 0;
+  for (let r = 0; r < reps; r += 1) {
+    const maxima = [Number.NEGATIVE_INFINITY, Number.NEGATIVE_INFINITY, Number.NEGATIVE_INFINITY];
+    let rcMax = Number.NEGATIVE_INFINITY;
+    for (let j = 0; j < k; j += 1) {
+      if (!usable[j]) continue;
+      const m = boot[r * k + j];
+      for (let c = 0; c < 3; c += 1) maxima[c] = Math.max(maxima[c], (m - centre[c][j]) * scale(j));
+      rcMax = Math.max(rcMax, m - means[j]);
+    }
+    for (let c = 0; c < 3; c += 1) if (Math.max(maxima[c], 0) > spaStat) exceed[c] += 1;
+    if (rcMax > rcObserved) rcExceed += 1;
+  }
+
+  // StepM: reject every variant above the (1 - alpha) quantile of the bootstrap maximum over those
+  // not yet rejected (consistent recentring), remove them, and repeat until nothing more goes.
+  const rejected = new Set();
+  const rounds = [];
+  for (;;) {
+    const live = [];
+    for (let j = 0; j < k; j += 1) if (usable[j] && !rejected.has(j)) live.push(j);
+    if (!live.length) break;
+    const maxima = new Float64Array(reps);
+    for (let r = 0; r < reps; r += 1) {
+      let m = Number.NEGATIVE_INFINITY;
+      for (const j of live) m = Math.max(m, (boot[r * k + j] - centre[1][j]) * scale(j));
+      maxima[r] = m;
+    }
+    maxima.sort();
+    const critical = quantile(maxima, 1 - alpha);
+    const now = live.filter((j) => means[j] * scale(j) > critical);
+    if (!now.length) break;
+    for (const j of now) rejected.add(j);
+    rounds.push({ critical, rejected: now });
+  }
+
+  return {
+    n, k, block, reps, seed, alpha, studentize,
+    means, variance, usable,
+    best, observed,
+    spa: { statistic: spaStat, p_lower: exceed[0] / reps, p_consistent: exceed[1] / reps, p_upper: exceed[2] / reps },
+    reality_check: { statistic: rcObserved, p_value: rcExceed / reps },
+    stepm: { superior: [...rejected].sort((a, b) => a - b), rounds },
+  };
+}
+
+// numpy's default percentile (linear interpolation between order statistics), on sorted values.
+function quantile(sorted, q) {
+  const h = (sorted.length - 1) * q;
+  const lo = Math.floor(h);
+  const hi = Math.min(lo + 1, sorted.length - 1);
+  return sorted[lo] + (h - lo) * (sorted[hi] - sorted[lo]);
+}

package/src/local/js/validate/reality-check.js

@@ -0,0 +1,89 @@
+// js/validate/reality-check.js
+// The pure computation behind POST /api/v1/validate/reality-check, shared by the API route and the
+// MCP package's local mode (mcp/src/local mirrors it byte for byte), so the two cannot disagree.
+//
+// Data-snooping tests on every variant a search tried: Hansen's SPA, White's Reality Check and
+// Romano and Wolf's StepM, all on one seeded stationary bootstrap (js/snooping-core.js).
+import { LIMITS } from "../../api/_lib/limits.js";
+import { defaultBlock, snoopingTests } from "../snooping-core.js";
+
+// Bootstrap work (periods x variants x draws) that fits the service's wall-time limit with room to
+// spare: about a second on one core.
+export const MAX_BOOTSTRAP_CELLS = 200_000_000;
+const DEFAULT_REPS = 2000;
+const MIN_REPS = 500;
+const MIN_OBSERVATIONS = 30;
+
+const fmt = (x, digits = 3) => (Number.isFinite(x) ? x.toFixed(digits) : String(x));
+
+export function compute(body) {
+  const { matrix, benchmark } = body;
+  if (!Array.isArray(matrix) || !matrix.length || !Array.isArray(matrix[0])) throw new RangeError("matrix must be an array of rows, each a row of variant returns for one period");
+  const n = matrix.length;
+  const k = matrix[0].length;
+  if (n > LIMITS.max_observations) throw new RangeError(`A matrix may hold at most ${LIMITS.max_observations} rows`);
+  if (k > LIMITS.max_variants) throw new RangeError(`A matrix may hold at most ${LIMITS.max_variants} variants`);
+  if (n < MIN_OBSERVATIONS) throw new RangeError(`A matrix needs at least ${MIN_OBSERVATIONS} rows (periods) for a bootstrap`);
+  if (k < 1) throw new RangeError("A matrix needs at least one variant");
+  let bench = null;
+  if (benchmark !== undefined && benchmark !== null) {
+    if (!Array.isArray(benchmark) || benchmark.length !== n) throw new RangeError(`benchmark must be one return per row of matrix (${n})`);
+    // JSON null or a string is refused, never read as a number: Number(null) is 0, a silent return.
+    const bad = benchmark.findIndex((x) => typeof x !== "number" || !Number.isFinite(x));
+    if (bad >= 0) throw new RangeError(`benchmark[${bad}] is not a finite number`);
+    bench = benchmark;
+  }
+  const d = new Float64Array(n * k);
+  for (let t = 0; t < n; t += 1) {
+    const row = matrix[t];
+    if (!Array.isArray(row) || row.length !== k) throw new RangeError(`matrix row ${t} has ${Array.isArray(row) ? row.length : "no"} values; every row needs ${k}, one per variant`);
+    for (let j = 0; j < k; j += 1) {
+      const x = row[j];
+      if (typeof x !== "number" || !Number.isFinite(x)) throw new RangeError(`matrix[${t}][${j}] is not a finite number; a variant with a missing period cannot be compared period by period`);
+      d[t * k + j] = x - (bench ? bench[t] : 0);
+    }
+  }
+  const block = body.block_length === undefined ? defaultBlock(n) : Number(body.block_length);
+  if (!Number.isInteger(block) || block < 1 || block > Math.floor(n / 2)) throw new RangeError(`block_length must be an integer from 1 to half the number of periods (${Math.floor(n / 2)})`);
+  const budget = Math.floor(MAX_BOOTSTRAP_CELLS / (n * k));
+  if (budget < MIN_REPS) throw new RangeError(`${n} periods x ${k} variants is too large for ${MIN_REPS} bootstrap draws within the service's limit (${MAX_BOOTSTRAP_CELLS} period-variant draws); send fewer periods (weekly instead of daily) or fewer variants`);
+  const reps = body.reps === undefined ? Math.min(DEFAULT_REPS, budget) : Number(body.reps);
+  if (!Number.isInteger(reps) || reps < MIN_REPS || reps > budget) throw new RangeError(`reps must be an integer from ${MIN_REPS} to ${budget} for a ${n} x ${k} matrix`);
+  const seed = body.seed === undefined ? 42 : Number(body.seed);
+  if (!Number.isInteger(seed)) throw new RangeError("seed must be an integer");
+  const alpha = body.alpha === undefined ? 0.05 : Number(body.alpha);
+  if (!(alpha > 0 && alpha <= 0.5)) throw new RangeError("alpha must be above 0 and at most 0.5");
+
+  const r = snoopingTests(d, n, k, { block, reps, seed, alpha });
+  const best = r.best;
+  const tStat = r.observed;
+  const p = r.spa.p_consistent;
+  const excluded = r.usable.map((ok, j) => (ok ? -1 : j)).filter((j) => j >= 0);
+  const superior = r.stepm.superior;
+  const against = bench ? "the benchmark" : "zero";
+  const verdict = p <= alpha
+    ? `The best variant beats ${against} by more than the best of ${k} would by luck (p ${fmt(p)} at alpha ${alpha}).`
+    : `The best variant does not beat ${against} by more than the best of ${k} could by luck (p ${fmt(p)} at alpha ${alpha}).`;
+  return {
+    observations: n,
+    variants: k,
+    benchmark: bench ? "series" : "zero",
+    block_length: block,
+    block_length_rule: body.block_length === undefined ? "round(n^(1/3)), the default" : "as given",
+    reps,
+    seed,
+    best_variant: { index: best, mean_excess_return: r.means[best], t_stat: tStat },
+    spa: {
+      statistic: r.spa.statistic,
+      p_value: p,
+      p_value_lower: r.spa.p_lower,
+      p_value_upper: r.spa.p_upper,
+      monte_carlo_se: Math.sqrt((p * (1 - p)) / reps),
+    },
+    reality_check: { statistic: r.reality_check.statistic, p_value: r.reality_check.p_value },
+    stepm: { alpha, superior_variants: superior, rounds: r.stepm.rounds.length },
+    ...(excluded.length ? { constant_variants_excluded: excluded } : {}),
+    verdict,
+    plain_reading: `Variant ${best} had the best studentized excess return over ${against} (t ${fmt(tStat, 2)}) of the ${k} variants passed. The chance that the best of ${k} variants with no edge looks at least this good is ${fmt(p)} by Hansen's SPA (consistent p-value; bounds ${fmt(r.spa.p_lower)} to ${fmt(r.spa.p_upper)}), and ${fmt(r.reality_check.p_value)} by White's Reality Check, which does not studentize. StepM at ${alpha} finds ${superior.length ? `${superior.length} variant${superior.length === 1 ? "" : "s"} that beat${superior.length === 1 ? "s" : ""} ${against}: ${superior.join(", ")}` : `no variant that beats ${against}`}. Only the variants passed are counted: a search that tried more than it passed makes luck look smaller than it was.`,
+  };
+}

package/src/local.mjs

@@ -10,6 +10,7 @@ import { compute as trackRecord } from "./local/js/validate/track-record.js";
 import { compute as backtestLength } from "./local/js/validate/backtest-length.js";
 import { compute as haircutSharpe } from "./local/js/validate/haircut-sharpe.js";
 import { compute as luckTrials } from "./local/js/validate/luck-trials.js";
+import { compute as realityCheck } from "./local/js/validate/reality-check.js";
 
 export const LOCAL_VALIDATORS = Object.freeze({
   validate_deflated_sharpe: { endpoint: "validate/deflated-sharpe", compute: deflatedSharpe },
@@ -20,6 +21,7 @@ export const LOCAL_VALIDATORS = Object.freeze({
   validate_backtest_length: { endpoint: "validate/backtest-length", compute: backtestLength },
   validate_haircut_sharpe: { endpoint: "validate/haircut-sharpe", compute: haircutSharpe },
   validate_luck_trials: { endpoint: "validate/luck-trials", compute: luckTrials },
+  validate_reality_check: { endpoint: "validate/reality-check", compute: realityCheck },
 });
 
 const NOTE = "Computed on this machine in local mode. Nothing was sent to canlicapital.com and no receipt was stored.";

package/src/schemas.mjs

@@ -24,6 +24,12 @@ export const FIELD_DESCRIPTIONS = Object.freeze({
   n_splits: "Even number of blocks, at least 2; default 16.",
   max_combinations: "Most splits evaluated, up to 2000 (default).",
   seed: "Sampling seed; default 42.",
+  rc_matrix: "Returns of every variant the search tried, one row per period, one column per variant.",
+  matrix_file: "Path to a CSV or JSON with one numeric column per variant, instead of matrix.",
+  rc_benchmark: "Benchmark return per period; default zero.",
+  block_length: "Mean bootstrap block in periods; default round(n^(1/3)).",
+  reps: "Bootstrap draws; default 2000.",
+  alpha: "Familywise error for StepM; default 0.05.",
   record: "A canli.paper-evidence.v0 record.",
   sleeve_sharpe: "Annualized Sharpe of one sleeve.",
   average_pairwise_correlation: "Average correlation between sleeves, -1 to 1.",
@@ -115,6 +121,24 @@ export const overfittingInput = z
   })
   .strict();
 
+// ---------------------------------------------------------------------------------------------
+// validate_reality_check: every variant's returns, inline or from a file on this machine.
+// ---------------------------------------------------------------------------------------------
+
+export const realityCheckToolShape = z
+  .object({
+    matrix: z.array(z.array(z.number()).max(200)).min(30).max(20000).optional().describe(d.rc_matrix),
+    matrix_file: z.string().min(1).max(4096).optional().describe(d.matrix_file),
+    benchmark: z.array(z.number()).min(30).max(20000).optional().describe(d.rc_benchmark),
+    block_length: z.number().int().min(1).max(10000).optional().describe(d.block_length),
+    reps: z.number().int().min(500).max(400000).optional().describe(d.reps),
+    seed: z.number().int().optional().describe(d.seed),
+    alpha: z.number().gt(0).max(0.5).optional().describe(d.alpha),
+  })
+  .strict();
+
+export const realityCheckInput = realityCheckToolShape.refine((v) => (v.matrix === undefined) !== (v.matrix_file === undefined), "Send exactly one of matrix or matrix_file");
+
 // ---------------------------------------------------------------------------------------------
 // validate_paper_evidence: the record shape is the open standard itself, so this schema only
 // checks that a record object was sent; canlicapital.com runs the real conformance check.
@@ -306,6 +330,7 @@ export const TOOL_DESCRIPTIONS = Object.freeze({
   validate_deflated_sharpe: `Deflated Sharpe ratio: the probability (0 to 1) that the selected strategy's Sharpe beats the best that luck gives across the variants tried, with the probabilistic Sharpe and that luck benchmark. Send the seven statistics or a return series. With every variant's returns use validate_overfitting; luck as a trial count, validate_luck_trials; a multiple-testing haircut, validate_haircut_sharpe. ${LIMITS_SENTENCES.notAdmission}`,
   audit_backtest: `One-call audit of a strategy's returns: deflated Sharpe, minimum track record and, with every variant's returns, the probability of backtest overfitting, each the matching validator's result with its own receipt. Point returns_file at the backtest's CSV or JSON instead of pasting long series. Prefer it to calling the validators one by one; one validation per check. ${LIMITS_SENTENCES.notAdmission}`,
   validate_overfitting: `Probability of backtest overfitting (0 to 1) by CSCV: how often the in-sample best variant falls below the out-of-sample median. Needs every variant's returns (periods by variants); with summary statistics only, use validate_deflated_sharpe. ${LIMITS_SENTENCES.notAdmission}`,
+  validate_reality_check: `Data-snooping tests on every variant a search tried: Hansen's SPA p-value that the best beat the benchmark only by luck, White's Reality Check, and the variants Romano-Wolf StepM finds better. Send all variants tried, not only the winners. ${LIMITS_SENTENCES.notAdmission}`,
   validate_paper_evidence: `Whether a paper or simulated performance record meets canli.paper-evidence.v0, with a JSON pointer per failure. Checks structure and required disclosures, not whether the returns are good. ${LIMITS_SENTENCES.scope}`,
   validate_backtest_length: `Minimum backtest length (years) before the best of N independent trials is not expected to reach a target Sharpe by luck; with backtest_years, the most trials those years allow. For planning a search; once it has a result, use validate_deflated_sharpe. ${LIMITS_SENTENCES.notAdmission}`,
   validate_luck_trials: `How many skill-less strategies a search would need for its best to reach this Sharpe by luck (Monte Carlo), and with a trial count, the chance it did. States luck as the best of N random tries; for the probability the Sharpe is real, use validate_deflated_sharpe. ${LIMITS_SENTENCES.notAdmission}`,

package/src/server.mjs

@@ -14,7 +14,7 @@ import { computeLocally } from "./local.mjs";
 import { readMatrixFile, readSeriesFile } from "./series-file.mjs";
 import { verifyReceipt } from "./local/js/receipt-statement.js";
 import { StdioServerTransport } from "@modelcontextprotocol/server/stdio";
-import { breadthInput, trackRecordInput, auditBacktestInput, verifyReceiptToolShape, backtestLengthInput, haircutSharpeInput, luckTrialsInput, auditBacktestToolShape, companyHistoryInput, companyHistoryToolShape, deflatedSharpeInput, deflatedSharpeToolShape, emptyInput, getKeyInput, getReceiptInput, overfittingInput, LIMITS_SENTENCES, paperEvidenceInput, TOOL_DESCRIPTIONS, validationOutput, auditOutput, keyOutput, receiptOutput, verifyReceiptOutput, statusOutput, companyHistoryOutput } from "./schemas.mjs";
+import { breadthInput, trackRecordInput, auditBacktestInput, verifyReceiptToolShape, backtestLengthInput, haircutSharpeInput, luckTrialsInput, auditBacktestToolShape, companyHistoryInput, companyHistoryToolShape, deflatedSharpeInput, deflatedSharpeToolShape, emptyInput, getKeyInput, getReceiptInput, overfittingInput, realityCheckInput, realityCheckToolShape, LIMITS_SENTENCES, paperEvidenceInput, TOOL_DESCRIPTIONS, validationOutput, auditOutput, keyOutput, receiptOutput, verifyReceiptOutput, statusOutput, companyHistoryOutput } from "./schemas.mjs";
 
 export const DEFAULT_BASE = "https://canlicapital.com";
 export const SERVER_NAME = "canlicapital-validation-mcp";
@@ -66,7 +66,7 @@ export function configuredLocal(value) {
 // is most of each turn's prompt (README, "Toolsets"; bench/tool_list_tokens.py measures it), so a
 // client that needs one kind of tool can load only that kind. Default: all.
 export const TOOLSETS = Object.freeze({
-  validate: Object.freeze(["get_key", "validate_deflated_sharpe", "validate_overfitting", "validate_paper_evidence", "validate_breadth", "validate_track_record", "validate_backtest_length", "validate_haircut_sharpe", "validate_luck_trials", "audit_backtest"]),
+  validate: Object.freeze(["get_key", "validate_deflated_sharpe", "validate_overfitting", "validate_paper_evidence", "validate_breadth", "validate_track_record", "validate_backtest_length", "validate_haircut_sharpe", "validate_luck_trials", "validate_reality_check", "audit_backtest"]),
   receipts: Object.freeze(["get_receipt", "verify_receipt"]),
   company: Object.freeze(["company_financial_history"]),
   status: Object.freeze(["service_status"]),
@@ -260,6 +260,20 @@ export async function toolValidateOverfitting(session, args) {
   return validationText(session, response);
 }
 
+// Every variant's returns, from the call or from a file on this machine (stdio only), tested for
+// data snooping in one call; the file is read here and only its numbers are sent.
+export async function toolValidateRealityCheck(session, args) {
+  const input = parseOrThrow(realityCheckInput, args, "validate_reality_check");
+  if (input.matrix_file && session.hosted) {
+    throw new Error("validate_reality_check: the hosted endpoint cannot read files on your machine; send matrix as numbers, or run the server locally with npx -y canli-validation-mcp.");
+  }
+  const { matrix_file: file, ...rest } = input;
+  const body = file ? { ...rest, matrix: readMatrixFile(file).matrix } : rest;
+  if (session.local) { const local = computeLocally("validate_reality_check", body); return validationText(session, local); }
+  const response = await validateRemote(session, "validate_reality_check", "/api/v1/validate/reality-check", body);
+  return validationText(session, response);
+}
+
 export async function toolValidatePaperEvidence(session, args) {
   const body = parseOrThrow(paperEvidenceInput, args, "validate_paper_evidence");
   if (session.local) { const local = computeLocally("validate_paper_evidence", body); return validationText(session, local); }
@@ -549,6 +563,11 @@ export function registerTools(server, session) {
     { title: "Validate overfitting (CSCV)", annotations: { title: "Validate overfitting (CSCV)", ...WRITES_RECEIPT }, description: TOOL_DESCRIPTIONS.validate_overfitting, inputSchema: overfittingInput, outputSchema: validationOutput },
     (args) => toolValidateOverfitting(session, args),
   );
+  register(
+    "validate_reality_check",
+    { title: "Data-snooping tests (SPA, Reality Check, StepM)", annotations: { title: "Data-snooping tests (SPA, Reality Check, StepM)", ...WRITES_RECEIPT }, description: TOOL_DESCRIPTIONS.validate_reality_check, inputSchema: realityCheckToolShape, outputSchema: validationOutput },
+    (args) => toolValidateRealityCheck(session, args),
+  );
   register(
     "validate_paper_evidence",
     { title: "Validate paper evidence", annotations: { title: "Validate paper evidence", ...WRITES_RECEIPT }, description: TOOL_DESCRIPTIONS.validate_paper_evidence, inputSchema: paperEvidenceInput, outputSchema: validationOutput },