canli-validation-mcp 0.13.1 → 0.14.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -207,6 +207,30 @@ On your own machine, `prices_file`, `series_file` and `returns_file` take a CSV
207
207
  of the numbers. Only numbers are read, plus an ISO date column when the file has one, so results can
208
208
  name the day a drawdown began.
209
209
 
210
+ ## The trial ledger: count every variant an agent tries
211
+
212
+ An agent can try thousands of strategy variants in an hour, and the best of them looks skilled by
213
+ luck alone. How much luck depends on how many were tried and how spread out their results were, so
214
+ the honest verdict on a reported winner needs the whole search. The ledger keeps it.
215
+
216
+ Turn it on with `CANLI_TOOLSETS=validate,lab,ledger` (or `all`), then:
217
+
218
+ 1. `ledger_record_trial` for every variant, as it is tried: its returns (or a returns file), or its
219
+ annualized Sharpe and sample size, plus a label and its parameters. The first call creates a
220
+ ledger and returns its name; reuse it. Each reply already counts the whole search so far.
221
+ 2. `ledger_summary` for the verdict: the best trial, the Sharpe that the best of that many
222
+ skill-less trials reaches by luck, the best trial's deflated Sharpe ratio after counting every
223
+ trial, and the probability of backtest overfitting (CSCV) when every trial has aligned returns.
224
+ 3. `ledger_export` to hand the search to someone else. Each trial is chained to the previous one
225
+ with SHA-256, so a removed, reordered or edited trial shows.
226
+
227
+ Ledgers are JSON-lines files under `CANLI_LEDGER_DIR` (default `~/.canli/ledgers`); nothing leaves
228
+ your machine. Twenty variants of pure noise, recorded this way, give a best annualized Sharpe of
229
+ about 1.5, a luck-alone expectation of about 1.7 and a deflated Sharpe near 0.4: the ledger calls
230
+ that winner what it is. The statistics are the same checked code as `validate_deflated_sharpe` and
231
+ `validate_overfitting`; `test/ledger.test.mjs` checks both directions (noise is flagged, a real edge
232
+ among noise keeps a deflated Sharpe above 0.95).
233
+
210
234
  ## Auditing a backtest in one call
211
235
 
212
236
  `audit_backtest` takes one strategy's return series and, ideally, every variant's returns
@@ -444,7 +468,8 @@ A client sends the model the whole tool list on every turn, and it is most of ea
444
468
  a validation result is a few hundred tokens, the list of every tool several thousand. A client
445
469
  that needs one kind of tool can list only that kind, with `CANLI_TOOLSETS` (stdio) or
446
470
  `?toolsets=` (hosted endpoint). The default is every toolset except `company`, whose one tool
447
- canli-fundamentals-mcp does better and point in time; `all` adds it. `get_key` is listed only when
471
+ canli-fundamentals-mcp does better and point in time, and `ledger`, which keeps files on your machine
472
+ and is local only; `all` adds both. `get_key` is listed only when
448
473
  no key is configured.
449
474
 
450
475
  | toolset | tools |
@@ -454,20 +479,22 @@ no key is configured.
454
479
  | `company` | `company_financial_history` |
455
480
  | `status` | `service_status` |
456
481
  | `lab` | `backtest_strategy`, `summarize_series`, `stress_test`, `check_feasibility`, `check_leakage`, `placebo_test` |
482
+ | `ledger` | `ledger_record_trial`, `ledger_summary`, `ledger_export` (stdio only) |
457
483
 
458
484
  Measured with `bench/tool_list_tokens.py` (tokenizer: tiktoken `o200k_base`; other models'
459
485
  tokenizers give different absolute counts), in the shape an OpenAI-style client sends the list:
460
486
 
461
487
  | CANLI_TOOLSETS | tools | tokens per turn | of all |
462
488
  |---|---|---|---|
463
- | (default) | 19 | 5,978 | 96% |
464
- | (default, with CANLI_KEY) | 18 | 5,842 | 93% |
465
- | `all` | 20 | 6,249 | 100% |
466
- | `validate` | 11 | 3,658 | 59% |
489
+ | (default) | 19 | 5,978 | 85% |
490
+ | (default, with CANLI_KEY) | 18 | 5,842 | 83% |
491
+ | `all` | 23 | 7,019 | 100% |
492
+ | `validate` | 11 | 3,658 | 52% |
467
493
  | `receipts` | 1 | 182 | 3% |
468
494
  | `company` | 1 | 273 | 4% |
469
495
  | `status` | 1 | 89 | 1% |
470
- | `lab` | 6 | 2,055 | 33% |
496
+ | `lab` | 6 | 2,055 | 29% |
497
+ | `ledger` | 3 | 772 | 11% |
471
498
 
472
499
  Providers cache a tool list that is identical from turn to turn and bill the cached part at a
473
500
  fraction of the price (`test/tool-list-stable.test.mjs` keeps each list byte-stable); a smaller list
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "canli-validation-mcp",
3
- "version": "0.13.1",
3
+ "version": "0.14.0",
4
4
  "description": "Check whether a backtest is real: deflated Sharpe, CSCV probability of backtest overfitting, minimum track record and backtest length, haircut Sharpe and luck-equivalent trials, as an MCP server. Local mode, hosted endpoint, signed receipts.",
5
5
  "keywords": [
6
6
  "mcp",
package/src/lab.mjs CHANGED
@@ -242,6 +242,9 @@ export const EXAMPLE_ARGS = Object.freeze({
242
242
  check_feasibility: { broker: "alpaca", asset_class: "us_equity", account: "margin", capital_usd: 250000, orders_per_rebalance: 40, rebalances_per_year: 52, turnover_per_year: 8, adv_usd: 20000000, daily_volatility: 0.02, copies: 1, expected_gross_return: 0.12, spread_and_fees_bps: 3 },
243
243
  check_leakage: { action: "plan", observations: 2520, prefixes: 5 },
244
244
  placebo_test: { action: "plan", data_file: "prices.csv" },
245
+ ledger_record_trial: { ledger: "sma-search", label: "sma 20/100, 5 bps", sharpe_annualized: 1.1, observations: 1260, periods_per_year: 252, params: { fast: 20, slow: 100 } },
246
+ ledger_summary: { ledger: "sma-search" },
247
+ ledger_export: { ledger: "sma-search" },
245
248
  });
246
249
 
247
250
  const usesFile = (args) => Object.keys(args).some((k) => k.endsWith("_file"));
@@ -0,0 +1,138 @@
1
+ // mcp/src/ledger-core.mjs
2
+ //
3
+ // The trial ledger: every strategy variant an agent tries in one research search, recorded as it is
4
+ // tried, so the result it finally reports is judged against the whole search instead of alone. An
5
+ // agent can run thousands of variants in an hour; the best of them looks skilled by luck alone, and
6
+ // the size of that luck depends on how many were tried and how spread out their results were.
7
+ //
8
+ // Each trial is one line in a hash chain (sha256 over the previous hash and the trial's canonical
9
+ // JSON), so an exported ledger shows any trial removed, reordered or edited after the fact. The
10
+ // statistics reuse the site's checked cores: perPeriodMoments and calculateDsr (pinned to
11
+ // standards/validation-api/vectors.json) and pboCscv. Pure functions only; storage is in ledger.mjs.
12
+ import { createHash } from "node:crypto";
13
+
14
+ import { calculateDsr, expectedMaxStandardNormal } from "./local/js/dsr-core.js";
15
+ import { finiteNumbers, perPeriodMoments } from "./local/js/moments-core.js";
16
+ import { pboCscv } from "./local/js/pbo-core.js";
17
+
18
+ export const LEDGER_SCHEMA = "canli.trial-ledger.v1";
19
+ export const GENESIS = "0".repeat(64);
20
+ export const MAX_TRIALS = 10000;
21
+ export const MAX_RETURNS = 20000;
22
+
23
+ export const LEDGER_LIMITS_TEXT = Object.freeze([
24
+ "The ledger counts the trials recorded in it; a trial run but never recorded is invisible to it, so the correction is only as complete as the record.",
25
+ "Treating every recorded trial as independent overstates the search when trials are correlated variants of one idea, which makes the deflated Sharpe conservative; pass effective_trials when you have a better estimate.",
26
+ "A deflated Sharpe ratio is a probability that the best trial's true Sharpe is above zero after the search; it is not a forecast of returns and not a recommendation.",
27
+ ]);
28
+
29
+ const sha256 = (text) => createHash("sha256").update(text).digest("hex");
30
+
31
+ // Keys sorted at every level, so the same trial always hashes the same way.
32
+ export function canonicalJson(value) {
33
+ if (Array.isArray(value)) return `[${value.map(canonicalJson).join(",")}]`;
34
+ if (value && typeof value === "object") return `{${Object.keys(value).sort().map((k) => `${JSON.stringify(k)}:${canonicalJson(value[k])}`).join(",")}}`;
35
+ return JSON.stringify(value);
36
+ }
37
+
38
+ // A trial as the agent describes it -> the record that is chained. Exactly one of returns, or a
39
+ // Sharpe with its observation count. periodsPerYear annualizes; it is fixed for the whole ledger.
40
+ export function buildTrial({ label, returns, sharpe_annualized, observations, params, periodsPerYear, n, prev, recordedAt }) {
41
+ if (typeof label !== "string" || !label.trim()) throw new RangeError("label must be a non-empty string naming the variant");
42
+ if (!(periodsPerYear > 0)) throw new RangeError("periods_per_year must be greater than zero");
43
+ const hasReturns = returns !== undefined && returns !== null;
44
+ const hasSharpe = sharpe_annualized !== undefined && sharpe_annualized !== null;
45
+ if (hasReturns === hasSharpe) throw new RangeError("give either returns or sharpe_annualized with observations, not both and not neither");
46
+ let trial;
47
+ if (hasReturns) {
48
+ const x = finiteNumbers(returns, "returns");
49
+ if (x.length > MAX_RETURNS) throw new RangeError(`returns holds ${x.length} observations; the limit is ${MAX_RETURNS}`);
50
+ const m = perPeriodMoments(x);
51
+ trial = { sharpe_annualized: m.sharpe_per_period * Math.sqrt(periodsPerYear), observations: m.observations, skew: m.skew, non_excess_kurtosis: m.non_excess_kurtosis, returns: x };
52
+ } else {
53
+ if (!Number.isFinite(sharpe_annualized)) throw new RangeError("sharpe_annualized must be a finite number");
54
+ if (!Number.isInteger(observations) || observations < 2) throw new RangeError("observations must be an integer of at least 2 when a Sharpe is given");
55
+ trial = { sharpe_annualized, observations, skew: null, non_excess_kurtosis: null, returns: null };
56
+ }
57
+ const body = { n, label: label.trim(), params: params ?? null, recorded_at: recordedAt, ...trial };
58
+ const hash = sha256(prev + canonicalJson(body));
59
+ return { ...body, prev, hash };
60
+ }
61
+
62
+ // Recomputes the chain. Returns { ok, head, trials } or { ok: false, at, reason }.
63
+ export function verifyChain(trials) {
64
+ let prev = GENESIS;
65
+ for (let i = 0; i < trials.length; i += 1) {
66
+ const { prev: recordedPrev, hash, ...body } = trials[i];
67
+ if (body.n !== i + 1) return { ok: false, at: i + 1, reason: `trial ${i + 1} is numbered ${body.n}` };
68
+ if (recordedPrev !== prev) return { ok: false, at: i + 1, reason: `trial ${i + 1} does not follow trial ${i}` };
69
+ if (sha256(prev + canonicalJson(body)) !== hash) return { ok: false, at: i + 1, reason: `trial ${i + 1} was changed after it was recorded` };
70
+ prev = hash;
71
+ }
72
+ return { ok: true, head: prev, trials: trials.length };
73
+ }
74
+
75
+ const sampleSd = (xs) => {
76
+ if (xs.length < 2) return null;
77
+ const mean = xs.reduce((a, b) => a + b, 0) / xs.length;
78
+ return Math.sqrt(xs.reduce((a, b) => a + (b - mean) ** 2, 0) / (xs.length - 1));
79
+ };
80
+
81
+ // The largest even number of CSCV blocks (at most 16) that leaves at least two periods per block.
82
+ const cscvSplits = (length) => {
83
+ for (let s = 16; s >= 4; s -= 2) if (length / s >= 2) return s;
84
+ return null;
85
+ };
86
+
87
+ const fmt = (x, d = 2) => (x === null || x === undefined ? "n/a" : Number(x).toFixed(d));
88
+
89
+ // The verdict on the whole search so far.
90
+ export function summarize({ trials, periodsPerYear, effectiveTrials }) {
91
+ const n = trials.length;
92
+ if (n === 0) return { trials: 0, plain_reading: "No trials are recorded in this ledger yet.", limits: LEDGER_LIMITS_TEXT };
93
+ const sharpes = trials.map((t) => t.sharpe_annualized);
94
+ const bestIndex = sharpes.reduce((bi, s, i) => (s > sharpes[bi] ? i : bi), 0);
95
+ const best = trials[bestIndex];
96
+ const spread = sampleSd(sharpes);
97
+ const counted = effectiveTrials ?? n;
98
+ const assumedShape = best.skew === null;
99
+ let deflated = null;
100
+ if (counted >= 2 && spread !== null) {
101
+ const d = calculateDsr({
102
+ observed_sharpe_annualized: best.sharpe_annualized, observations: best.observations, periods_per_year: periodsPerYear,
103
+ skew: assumedShape ? 0 : best.skew, non_excess_kurtosis: assumedShape ? 3 : best.non_excess_kurtosis,
104
+ effective_independent_trials: counted, cross_trial_sharpe_sd_annualized: spread,
105
+ });
106
+ deflated = { deflated_sharpe_ratio: d.deflated_sharpe_ratio, psr_against_zero: d.psr_against_zero, expected_max_sharpe_annualized: d.expected_max_sharpe_annualized, search_haircut_annualized: d.search_haircut_annualized };
107
+ }
108
+ let overfitting = null;
109
+ const withReturns = trials.filter((t) => Array.isArray(t.returns));
110
+ const lengths = new Set(withReturns.map((t) => t.returns.length));
111
+ if (withReturns.length === n && n >= 4 && lengths.size === 1) {
112
+ const length = [...lengths][0];
113
+ const splits = cscvSplits(length);
114
+ if (splits) {
115
+ const matrix = Array.from({ length }, (_, r) => withReturns.map((t) => t.returns[r]));
116
+ const p = pboCscv(matrix, { nSplits: splits });
117
+ overfitting = { probability_of_backtest_overfitting: p.pbo, n_splits: splits, combinations: p.n_combinations };
118
+ }
119
+ }
120
+ const luck = spread !== null && counted >= 2 ? spread * expectedMaxStandardNormal(counted) : null;
121
+ const sentences = [
122
+ `Across ${n} recorded trial${n === 1 ? "" : "s"}, the best is "${best.label}" with an annualized Sharpe of ${fmt(best.sharpe_annualized)}.`,
123
+ ];
124
+ if (luck !== null) sentences.push(`With this spread of results, the best of ${counted} skill-less trials would be expected to reach about ${fmt(luck)} by luck alone.`);
125
+ if (deflated) sentences.push(`Its deflated Sharpe ratio is ${fmt(deflated.deflated_sharpe_ratio)}: the probability that its true Sharpe is above zero once the search is counted.`);
126
+ else sentences.push("A deflated Sharpe needs at least two trials with different results.");
127
+ if (overfitting) sentences.push(`The probability of backtest overfitting (CSCV, ${overfitting.n_splits} blocks) is ${fmt(overfitting.probability_of_backtest_overfitting)}.`);
128
+ if (assumedShape) sentences.push("The best trial was recorded as a Sharpe without returns, so normal returns were assumed for it.");
129
+ return {
130
+ trials: n, effective_trials: counted,
131
+ best: { n: best.n, label: best.label, params: best.params, sharpe_annualized: best.sharpe_annualized, observations: best.observations },
132
+ cross_trial_sharpe_sd_annualized: spread,
133
+ expected_best_by_luck_annualized: luck,
134
+ deflated, overfitting,
135
+ plain_reading: sentences.join(" "),
136
+ limits: LEDGER_LIMITS_TEXT,
137
+ };
138
+ }
package/src/ledger.mjs ADDED
@@ -0,0 +1,124 @@
1
+ // mcp/src/ledger.mjs
2
+ //
3
+ // The trial-ledger tools (toolset "ledger", local stdio mode): ledger_record_trial, ledger_summary
4
+ // and ledger_export. The rules and statistics are in ledger-core.mjs; this file stores each ledger as
5
+ // one JSON-lines file (a header, then one chained trial per line) under CANLI_LEDGER_DIR, by default
6
+ // ~/.canli/ledgers, so a search survives a restart of the agent or the server. Nothing leaves the
7
+ // machine. A ledger is appended to, never rewritten.
8
+ import { randomBytes } from "node:crypto";
9
+ import { appendFileSync, existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
10
+ import { homedir } from "node:os";
11
+ import { join } from "node:path";
12
+ import { z } from "zod";
13
+
14
+ import { GENESIS, LEDGER_LIMITS_TEXT, LEDGER_SCHEMA, MAX_TRIALS, buildTrial, summarize, verifyChain } from "./ledger-core.mjs";
15
+ import { readSeriesFile } from "./series-file.mjs";
16
+
17
+ export const LEDGER_TOOLS = Object.freeze(["ledger_record_trial", "ledger_summary", "ledger_export"]);
18
+
19
+ export const ledgerDir = (env = process.env) => env.CANLI_LEDGER_DIR || join(homedir(), ".canli", "ledgers");
20
+ const ID = /^[a-z0-9][a-z0-9-]{2,63}$/;
21
+ const text = (value) => ({ content: [{ type: "text", text: JSON.stringify(value) }], structuredContent: value });
22
+
23
+ function pathFor(dir, id) {
24
+ if (!ID.test(id)) throw new RangeError("ledger must be 3 to 64 lowercase letters, digits or dashes, starting with a letter or digit");
25
+ return join(dir, `${id}.jsonl`);
26
+ }
27
+
28
+ export function readLedger(dir, id) {
29
+ const path = pathFor(dir, id);
30
+ if (!existsSync(path)) return null;
31
+ const lines = readFileSync(path, "utf8").split("\n").filter(Boolean).map((line) => JSON.parse(line));
32
+ const [header, ...trials] = lines;
33
+ if (header?.schema !== LEDGER_SCHEMA) throw new Error(`${path} is not a ${LEDGER_SCHEMA} file`);
34
+ const check = verifyChain(trials);
35
+ if (!check.ok) throw new Error(`ledger ${id} fails its own hash chain at trial ${check.at}: ${check.reason}`);
36
+ return { header, trials, head: check.head };
37
+ }
38
+
39
+ export function recordTrial(dir, input, now = () => new Date().toISOString()) {
40
+ const id = input.ledger ?? `search-${randomBytes(4).toString("hex")}`;
41
+ let ledger = readLedger(dir, id);
42
+ if (!ledger) {
43
+ if (!(input.periods_per_year > 0)) throw new RangeError("periods_per_year is required for a ledger's first trial (252 for daily stock returns, 365 for daily crypto, 12 for monthly)");
44
+ mkdirSync(dir, { recursive: true });
45
+ const header = { schema: LEDGER_SCHEMA, ledger: id, periods_per_year: input.periods_per_year, created_at: now() };
46
+ writeFileSync(pathFor(dir, id), `${JSON.stringify(header)}\n`, { flag: "wx" });
47
+ ledger = { header, trials: [], head: GENESIS };
48
+ } else if (input.periods_per_year !== undefined && input.periods_per_year !== ledger.header.periods_per_year) {
49
+ throw new RangeError(`ledger ${id} annualizes with periods_per_year ${ledger.header.periods_per_year}; record this trial in a new ledger`);
50
+ }
51
+ if (ledger.trials.length >= MAX_TRIALS) throw new RangeError(`ledger ${id} holds the maximum of ${MAX_TRIALS} trials`);
52
+ const returns = input.returns_file ? readSeriesFile(input.returns_file, input.column) : input.returns;
53
+ const trial = buildTrial({ label: input.label, returns, sharpe_annualized: input.sharpe_annualized, observations: input.observations, params: input.params,
54
+ periodsPerYear: ledger.header.periods_per_year, n: ledger.trials.length + 1, prev: ledger.head, recordedAt: now() });
55
+ appendFileSync(pathFor(dir, id), `${JSON.stringify(trial)}\n`);
56
+ const trials = [...ledger.trials, trial];
57
+ return { ledger: id, trial: trial.n, chain_head: trial.hash, ...summarize({ trials, periodsPerYear: ledger.header.periods_per_year }) };
58
+ }
59
+
60
+ export function ledgerSummary(dir, { ledger: id, effective_trials }) {
61
+ const ledger = readLedger(dir, id);
62
+ if (!ledger) throw new RangeError(`no ledger named ${id} in ${dir}`);
63
+ return { ledger: id, chain_head: ledger.head, ...summarize({ trials: ledger.trials, periodsPerYear: ledger.header.periods_per_year, effectiveTrials: effective_trials }) };
64
+ }
65
+
66
+ // The whole ledger, verifiable offline: recompute each hash from the previous one and the trial body.
67
+ export function ledgerExport(dir, { ledger: id, include_returns = false }) {
68
+ const ledger = readLedger(dir, id);
69
+ if (!ledger) throw new RangeError(`no ledger named ${id} in ${dir}`);
70
+ return {
71
+ ...ledger.header, chain_head: ledger.head, trials: ledger.trials.length, file: pathFor(dir, id),
72
+ records: ledger.trials.map((t) => (include_returns ? t : { ...t, returns: t.returns ? `${t.returns.length} values (in the file)` : null })),
73
+ verify: "Each record's hash is sha256(previous hash + canonical JSON of the record without prev and hash), starting from 64 zeros; the file keeps every return value so the chain can be recomputed exactly.",
74
+ limits: LEDGER_LIMITS_TEXT,
75
+ };
76
+ }
77
+
78
+ const ledgerId = z.string().regex(ID).describe("The ledger's name. Omit on the first trial to create one; reuse the name it returns for every later trial of the same search.");
79
+ export const recordTrialInput = z.object({
80
+ ledger: ledgerId.optional(),
81
+ label: z.string().min(1).max(200).describe("A short name for this variant, for example 'sma 20/100, 5 bps'."),
82
+ returns: z.array(z.number()).max(20000).optional().describe("The variant's period returns as decimals. Give returns, or returns_file, or sharpe_annualized with observations."),
83
+ returns_file: z.string().optional().describe("A CSV or JSON file of the variant's returns, read on this machine."),
84
+ column: z.string().optional().describe("Column of returns_file to read, when it has several."),
85
+ sharpe_annualized: z.number().optional().describe("The variant's annualized Sharpe ratio, when its returns are not available."),
86
+ observations: z.number().int().min(2).optional().describe("How many return periods the Sharpe was measured on."),
87
+ periods_per_year: z.number().positive().optional().describe("Required on a ledger's first trial: 252 daily stocks, 365 daily crypto, 52 weekly, 12 monthly."),
88
+ params: z.record(z.string(), z.any()).optional().describe("The variant's parameters, recorded as given."),
89
+ });
90
+ export const summaryInput = z.object({
91
+ ledger: ledgerId,
92
+ effective_trials: z.number().int().min(2).optional().describe("Use instead of the recorded count when many trials are near-copies of each other."),
93
+ });
94
+ export const exportInput = z.object({
95
+ ledger: ledgerId,
96
+ include_returns: z.boolean().optional().describe("Include every return value in the reply (large). The file always keeps them."),
97
+ });
98
+ const summaryShape = {
99
+ ledger: z.string(), chain_head: z.string().optional(), trial: z.number().optional(), trials: z.number(), effective_trials: z.number().optional(),
100
+ best: z.object({}).passthrough().optional(), cross_trial_sharpe_sd_annualized: z.number().nullable().optional(), expected_best_by_luck_annualized: z.number().nullable().optional(),
101
+ deflated: z.object({}).passthrough().nullable().optional(), overfitting: z.object({}).passthrough().nullable().optional(), plain_reading: z.string(), limits: z.array(z.string()),
102
+ };
103
+ export const summaryOutput = z.object(summaryShape).passthrough();
104
+ export const exportOutput = z.object({ ledger: z.string(), chain_head: z.string(), trials: z.number(), records: z.array(z.object({}).passthrough()), verify: z.string(), limits: z.array(z.string()) }).passthrough();
105
+
106
+ export const LEDGER_TOOL_DESCRIPTIONS = Object.freeze({
107
+ ledger_record_trial: `Record every strategy variant you try in one search, as you try it (its returns, or its Sharpe and sample size). Returns the trial count and the best trial's deflated Sharpe after counting all of them, so a lucky best of many is not reported as skill. Local; kept in a hash-chained file. ${LEDGER_LIMITS_TEXT[0]}`,
108
+ ledger_summary: `The verdict on a whole search: best trial, the Sharpe the best of that many skill-less trials reaches by luck, its deflated Sharpe, and the probability of backtest overfitting when trials share aligned returns. ${LEDGER_LIMITS_TEXT[2]}`,
109
+ ledger_export: "Export a ledger with its hash chain, so anyone can check that no trial was removed, reordered or changed after it was recorded.",
110
+ });
111
+
112
+ const ANNOTATIONS_WRITE = { readOnlyHint: false, destructiveHint: false, idempotentHint: false, openWorldHint: false };
113
+ const ANNOTATIONS_READ = { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false };
114
+
115
+ export function ledgerToolSpecs(session) {
116
+ const dir = session.ledgerDir ?? ledgerDir();
117
+ const wrap = (fn) => async (args) => text(fn(dir, args));
118
+ const spec = (name, title, input, output, handler, annotations) => [name, { title, annotations: { title, ...annotations }, description: LEDGER_TOOL_DESCRIPTIONS[name], inputSchema: input, outputSchema: output }, handler];
119
+ return [
120
+ spec("ledger_record_trial", "Record a trial in a search ledger", recordTrialInput, summaryOutput, wrap(recordTrial), ANNOTATIONS_WRITE),
121
+ spec("ledger_summary", "Judge a whole search", summaryInput, summaryOutput, wrap(ledgerSummary), ANNOTATIONS_READ),
122
+ spec("ledger_export", "Export a ledger with its hash chain", exportInput, exportOutput, wrap(ledgerExport), ANNOTATIONS_READ),
123
+ ];
124
+ }
package/src/server.mjs CHANGED
@@ -14,6 +14,7 @@ import { computeLocally } from "./local.mjs";
14
14
  import { validateLocalJournalEvidence } from "./journal-evidence.mjs";
15
15
  import { readMatrixFile, readSeriesFile } from "./series-file.mjs";
16
16
  import { LAB_TOOLS, labToolSpecs, registerCodeResources, registerLabPrompts } from "./lab.mjs";
17
+ import { LEDGER_TOOLS, ledgerToolSpecs } from "./ledger.mjs";
17
18
  import { verifyReceipt } from "./local/js/receipt-statement.js";
18
19
  import { annualizeDecay, blockBootstrapT, fixNext, neweyWestT, nullZooFamily, returnShape, searchFromVariants, sharpeInterval } from "./local/js/audit-core.js";
19
20
  import { NULL_ZOO_V1 } from "./local/js/null-zoo-v1-sizes.js";
@@ -80,12 +81,15 @@ export const TOOLSETS = Object.freeze({
80
81
  company: Object.freeze(["company_financial_history"]),
81
82
  status: Object.freeze(["service_status"]),
82
83
  lab: Object.freeze([...LAB_TOOLS]),
84
+ ledger: Object.freeze([...LEDGER_TOOLS]),
83
85
  });
84
86
 
85
87
  // CANLI_TOOLSETS=validate,company (or ?toolsets= on the hosted endpoint): a comma-separated list of
86
88
  // TOOLSETS names, or "all". Empty or unsubstituted means the default; an unknown name is refused, so
87
89
  // a typo never silently leaves a client without the tools it asked for.
88
- export const DEFAULT_TOOLSETS = Object.freeze(Object.keys(TOOLSETS).filter((name) => name !== "company"));
90
+ // The trial ledger is opt-in (CANLI_TOOLSETS=...,ledger or "all"): it keeps files on this machine, and
91
+ // the default tool list, re-sent every turn, stays the size it was.
92
+ export const DEFAULT_TOOLSETS = Object.freeze(Object.keys(TOOLSETS).filter((name) => name !== "company" && name !== "ledger"));
89
93
  export function configuredToolsets(value) {
90
94
  const v = configuredKey(value);
91
95
  if (!v) return [...DEFAULT_TOOLSETS];
@@ -782,6 +786,8 @@ export function registerTools(server, session) {
782
786
  (args) => toolCompanyFinancialHistory(session, args),
783
787
  );
784
788
  for (const [name, config, handler] of labToolSpecs(session)) register(name, config, handler);
789
+ // The ledger keeps state in files on this machine; the hosted endpoint is stateless, so it never lists it.
790
+ if (!session.hosted) for (const [name, config, handler] of ledgerToolSpecs(session)) register(name, config, handler);
785
791
  return catalog;
786
792
  }
787
793