canli-validation-mcp 0.11.0 → 0.12.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -83,7 +83,7 @@ receipt. The current npm/hosted release retains the existing record-only API.
83
83
 
84
84
  | Tool | Calls | Key required |
85
85
  |---|---|---|
86
- | `get_key` | `POST /api/v1/keys` | no |
86
+ | `get_key` | `POST /api/v1/keys` (listed only when no key is configured) | no |
87
87
  | `validate_deflated_sharpe` | `POST /api/v1/validate/deflated-sharpe` | yes |
88
88
  | `validate_overfitting` | `POST /api/v1/validate/overfitting` | yes |
89
89
  | `validate_paper_evidence` | `POST /api/v1/validate/paper-evidence` | yes |
@@ -93,9 +93,8 @@ receipt. The current npm/hosted release retains the existing record-only API.
93
93
  | `validate_haircut_sharpe` | `POST /api/v1/validate/haircut-sharpe` | yes |
94
94
  | `validate_luck_trials` | `POST /api/v1/validate/luck-trials` | yes |
95
95
  | `validate_reality_check` | `POST /api/v1/validate/reality-check` (a `matrix_file` is read on your machine and sent as numbers) | yes |
96
- | `audit_backtest` | the deflated Sharpe, track record and, with `variants`, overfitting routes, one validation each | yes |
97
- | `verify_receipt` | `GET /api/v1/receipts/{id}` when given an id; the checks run locally | no |
98
- | `get_receipt` | `GET /api/v1/receipts/{id}` | no |
96
+ | `audit_backtest` | the deflated Sharpe and track record routes, then the headline: with `variants`, the overfitting and data-snooping routes; without, luck trials. One validation each | yes |
97
+ | `verify_receipt` | `GET /api/v1/receipts/{id}` when given an id (`include_receipt` returns the stored receipt); the checks run locally | no |
99
98
  | `service_status` | `GET /api/v1/validate/status` | no |
100
99
  | `company_financial_history` | `GET /company-data/{cik}.json` (a ticker resolves through `GET /api/v1/company-tickers.json`) | no |
101
100
 
@@ -182,12 +181,27 @@ name the day a drawdown began.
182
181
 
183
182
  ## Auditing a backtest in one call
184
183
 
185
- `audit_backtest` takes one strategy's return series, the number of variants tried and their Sharpe
186
- dispersion, and optionally every variant's returns. It runs `validate_deflated_sharpe` on the
187
- series, then `validate_track_record` on the Sharpe, skew and kurtosis that check derived, then,
188
- with `variants`, `validate_overfitting`. Each check is exactly what its own tool returns, with its
189
- own receipt; the boundary sentences they share are stated once. A check that refuses (a Sharpe
190
- that cannot beat the benchmark has no minimum track record) is reported as that check's error.
184
+ `audit_backtest` takes one strategy's return series and, ideally, every variant's returns
185
+ (`variants` or `variants_file`, the dropped ones too); without them, the number of variants tried
186
+ and their Sharpe spread. The result opens with:
187
+
188
+ - `headline`: one test chosen because its false-positive rate was measured before anyone looked at
189
+ your numbers. Null Zoo v1 (`scripts/research/null-zoo/v1/` in the repository) drew hundreds of
190
+ thousands of simulated searches from nine return shapes with known ground truth; the bar and the
191
+ rule that picks the test were committed before the run. With variants the headline comes from
192
+ the data-snooping tests (`validate_reality_check`, with its receipt); without, from luck trials
193
+ with Lo's autocorrelation correction. Its sentence quotes the measured false-positive rate for
194
+ returns shaped like yours, and the worst across the nine shapes.
195
+ - `fix_next`: at most five next steps, each `{id, why, next}` with a stable id.
196
+ - `computed`: Lo's adjusted Sharpe, a stationary-bootstrap 95% interval for the Sharpe, the
197
+ haircut Sharpe, the minimum backtest length and, with variants, how many independent trials
198
+ they count as (Li and Ji) and the selected variant's out-of-sample decay.
199
+
200
+ Then the checks: `validate_deflated_sharpe` on the series (an estimate), `validate_track_record`
201
+ on the Sharpe, skew and kurtosis that check derived and, with variants, `validate_overfitting`.
202
+ Each is exactly what its own tool returns, with its own receipt; the deflated Sharpe runs with the
203
+ larger of the declared and counted trials and Sharpe spread, so a search cannot be made to look
204
+ smaller than it was. A check that refuses or cannot be reached is reported as that check's error.
191
205
  The audit adds no grade of its own. Through the API it uses one validation per check.
192
206
 
193
207
  When the server runs on your machine, `returns_file` and `variants_file` take the path of the
@@ -327,7 +341,7 @@ equals the hosted one; it names no receipt id because none was made.
327
341
  claude mcp add canli-local --env CANLI_LOCAL=1 -- npx -y canli-validation-mcp
328
342
  ```
329
343
 
330
- `get_receipt`, `service_status` and `company_financial_history` still read from canlicapital.com;
344
+ `verify_receipt` by id, `service_status` and `company_financial_history` still read from canlicapital.com;
331
345
  they send no series. In the Claude Desktop extension this is the "Private local mode" setting.
332
346
 
333
347
  ## Generic stdio client
@@ -392,20 +406,22 @@ A validation result is the answer (`data`), the sentences above except the quota
392
406
  receipt's id and URL; an error keeps its error. The rest of the API envelope (schema, endpoint,
393
407
  timestamps, claim and capital class, the human page, the source-file hashes and the quota line)
394
408
  describes the service rather than the answer, and an agent pays for every token of it on every
395
- call. It stays in the stored receipt, which `get_receipt` returns in full, and in `service_status`.
409
+ call. It stays in the stored receipt, which `verify_receipt` returns in full with `include_receipt`, and in `service_status`.
396
410
  On a breadth result this is about half the text. Set `CANLI_FULL_ENVELOPE=1` to receive every field.
397
411
 
398
412
  ## Toolsets (tokens)
399
413
 
400
414
  A client sends the model the whole tool list on every turn, and it is most of each turn's prompt:
401
- a validation result is a few hundred tokens, the list of all nineteen tools several thousand. A
402
- client that needs one kind of tool can list only that kind, with `CANLI_TOOLSETS` (stdio) or
403
- `?toolsets=` (hosted endpoint). The default is every tool.
415
+ a validation result is a few hundred tokens, the list of every tool several thousand. A client
416
+ that needs one kind of tool can list only that kind, with `CANLI_TOOLSETS` (stdio) or
417
+ `?toolsets=` (hosted endpoint). The default is every toolset except `company`, whose one tool
418
+ canli-fundamentals-mcp does better and point in time; `all` adds it. `get_key` is listed only when
419
+ no key is configured.
404
420
 
405
421
  | toolset | tools |
406
422
  |---|---|
407
423
  | `validate` | `get_key`, the eight validators, `audit_backtest` |
408
- | `receipts` | `get_receipt`, `verify_receipt` |
424
+ | `receipts` | `verify_receipt` |
409
425
  | `company` | `company_financial_history` |
410
426
  | `status` | `service_status` |
411
427
  | `lab` | `backtest_strategy`, `summarize_series`, `stress_test`, `check_feasibility` |
@@ -415,11 +431,13 @@ tokenizers give different absolute counts), in the shape an OpenAI-style client
415
431
 
416
432
  | CANLI_TOOLSETS | tools | tokens per turn | of all |
417
433
  |---|---|---|---|
418
- | `all` | 19 | 5,946 | 100% |
419
- | `validate` | 11 | 3,685 | 62% |
420
- | `receipts` | 2 | 312 | 5% |
434
+ | (default) | 17 | 5,581 | 95% |
435
+ | (default, with CANLI_KEY) | 16 | 5,445 | 93% |
436
+ | `all` | 18 | 5,852 | 100% |
437
+ | `validate` | 11 | 3,708 | 63% |
438
+ | `receipts` | 1 | 195 | 3% |
421
439
  | `company` | 1 | 273 | 5% |
422
- | `status` | 1 | 89 | 1% |
440
+ | `status` | 1 | 89 | 2% |
423
441
  | `lab` | 4 | 1,595 | 27% |
424
442
 
425
443
  Providers cache a tool list that is identical from turn to turn and bill the cached part at a
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "canli-validation-mcp",
3
- "version": "0.11.0",
3
+ "version": "0.12.0",
4
4
  "description": "Check whether a backtest is real: deflated Sharpe, CSCV probability of backtest overfitting, minimum track record and backtest length, haircut Sharpe and luck-equivalent trials, as an MCP server. Local mode, hosted endpoint, signed receipts.",
5
5
  "keywords": [
6
6
  "mcp",
package/src/lab.mjs CHANGED
@@ -114,7 +114,6 @@ export const EXAMPLE_ARGS = Object.freeze({
114
114
  validate_haircut_sharpe: { observed_sharpe_annualized: 1.5, periods_per_year: 252, observations: 1260, tests: 50 },
115
115
  validate_luck_trials: { observed_sharpe_annualized: 1.5, periods_per_year: 252, observations: 1260 },
116
116
  audit_backtest: { returns_file: "backtest_returns.csv", periods_per_year: 252, effective_independent_trials: 20, cross_trial_sharpe_sd_annualized: 0.5 },
117
- get_receipt: { id: "0123456789abcdef01234567" },
118
117
  verify_receipt: { id: "0123456789abcdef01234567" },
119
118
  service_status: {},
120
119
  company_financial_history: { ticker: "AAPL", concept: "Assets", limit: 8 },
@@ -0,0 +1,329 @@
1
+ // =============================================================================
2
+ // audit-core.js
3
+ // -----------------------------------------------------------------------------
4
+ // The parts of audit_backtest computed from the caller's own series rather than by a stored
5
+ // validator:
6
+ //
7
+ // returnShape lag-1 autocorrelation, volatility clustering, skew and tails of the
8
+ // returns, and Lo's (2002) autocorrelation-adjusted Sharpe ratio.
9
+ // sharpeInterval a stationary-bootstrap percentile interval for the annualized Sharpe.
10
+ // searchFromVariants what every variant's returns say about the search: how many were tried,
11
+ // how many independent ones that is (Nyholt 2004; Li and Ji 2005), how far
12
+ // their Sharpe ratios spread, and how correlated they are.
13
+ // nullZooFamily which Null Zoo family the shape resembles, by the rules fixed in
14
+ // config/research/null-zoo-v1-prereg.json before the calibration run.
15
+ // neweyWestT,
16
+ // blockBootstrapT the single-series tests the calibration scores (the same arithmetic as
17
+ // scripts/research/null-zoo/v1/validators.mjs; a test keeps them equal).
18
+ // oosDecay what the in-sample best did out of sample across CSCV splits.
19
+ // fixNext at most five concrete next steps, each naming the check that raised it.
20
+ //
21
+ // Nothing here decides whether a strategy is good. Each number is a description of the series and
22
+ // search exactly as sent.
23
+ // =============================================================================
24
+
25
+ import { autocorrelationFactor } from "./haircut-core.js";
26
+ import { perPeriodMoments } from "./moments-core.js";
27
+ import { makeRandom } from "./selection-risk-core.js";
28
+ import { defaultBlock, stationaryIndices } from "./snooping-core.js";
29
+
30
+ const mean = (xs) => { let s = 0; for (let i = 0; i < xs.length; i++) s += xs[i]; return s / xs.length; };
31
+
32
+ function lagOneCorrelation(xs) {
33
+ const m = mean(xs);
34
+ let c = 0, v = 0;
35
+ for (let i = 0; i < xs.length; i++) { const d = xs[i] - m; v += d * d; if (i > 0) c += d * (xs[i - 1] - m); }
36
+ return v > 0 ? c / v : 0;
37
+ }
38
+
39
+ /** The shape of one return series: what the Null Zoo families vary, plus Lo's adjusted Sharpe. */
40
+ export function returnShape(returns, periodsPerYear) {
41
+ const m = perPeriodMoments(returns);
42
+ const ppy = Number(periodsPerYear);
43
+ const xs = returns.map(Number);
44
+ const lag1 = lagOneCorrelation(xs);
45
+ const lag1Squares = lagOneCorrelation(xs.map((x) => x * x));
46
+ // Lo's factor needs rho strictly inside (-1, 1); a sample can sit on the edge only when it is tiny.
47
+ const rho = Math.max(-0.95, Math.min(0.95, lag1));
48
+ const sharpe = m.sharpe_per_period * Math.sqrt(ppy);
49
+ const factor = autocorrelationFactor(rho, ppy);
50
+ return {
51
+ observations: m.observations,
52
+ sharpe_annualized: sharpe,
53
+ skew: m.skew,
54
+ excess_kurtosis: m.non_excess_kurtosis - 3,
55
+ lag1_autocorrelation: lag1,
56
+ lag1_autocorrelation_of_squares: lag1Squares,
57
+ lo_factor: factor,
58
+ lo_adjusted_sharpe_annualized: sharpe * factor,
59
+ };
60
+ }
61
+
62
+ /**
63
+ * Percentile intervals for the annualized Sharpe from a stationary bootstrap (mean block n^(1/3)),
64
+ * and the share of resamples whose Sharpe is at or below zero.
65
+ */
66
+ export function sharpeInterval(returns, { periodsPerYear, reps = 2000, block, seed = 42, levels = [0.9, 0.95] } = {}) {
67
+ const xs = Float64Array.from(returns, Number);
68
+ const n = xs.length;
69
+ const b = block ?? defaultBlock(n);
70
+ const next = makeRandom(seed);
71
+ const idx = new Int32Array(n);
72
+ const root = Math.sqrt(Number(periodsPerYear));
73
+ const draws = new Float64Array(reps);
74
+ let atOrBelowZero = 0;
75
+ for (let r = 0; r < reps; r++) {
76
+ stationaryIndices(n, b, next, idx);
77
+ let s = 0, q = 0;
78
+ for (let i = 0; i < n; i++) { const v = xs[idx[i]]; s += v; q += v * v; }
79
+ const m = s / n;
80
+ const variance = (q - n * m * m) / (n - 1);
81
+ // A resample with no spread has no Sharpe ratio; it counts on the side its mean falls.
82
+ const sharpe = variance > 0 ? (m / Math.sqrt(variance)) * root : m > 0 ? Infinity : m < 0 ? -Infinity : 0;
83
+ draws[r] = sharpe;
84
+ if (sharpe <= 0) atOrBelowZero++;
85
+ }
86
+ draws.sort();
87
+ const at = (q) => {
88
+ const h = (reps - 1) * q;
89
+ const lo = Math.floor(h);
90
+ const hi = Math.min(lo + 1, reps - 1);
91
+ return draws[lo] + (h - lo) * (draws[hi] - draws[lo]);
92
+ };
93
+ return {
94
+ method: "stationary bootstrap, percentile",
95
+ reps, block: b, seed,
96
+ intervals: Object.fromEntries(levels.map((l) => [String(l), [at((1 - l) / 2), at(1 - (1 - l) / 2)]])),
97
+ share_at_or_below_zero: atOrBelowZero / reps,
98
+ };
99
+ }
100
+
101
+ // Eigenvalues of a symmetric matrix (cyclic Jacobi), for the effective number of trials.
102
+ function symmetricEigenvalues(a, k) {
103
+ const m = Float64Array.from(a);
104
+ for (let sweep = 0; sweep < 100; sweep++) {
105
+ let off = 0;
106
+ for (let i = 0; i < k; i++) for (let j = i + 1; j < k; j++) off += m[i * k + j] ** 2;
107
+ if (off < 1e-24) break;
108
+ for (let p = 0; p < k; p++) {
109
+ for (let q = p + 1; q < k; q++) {
110
+ const apq = m[p * k + q];
111
+ if (Math.abs(apq) < 1e-300) continue;
112
+ const theta = (m[q * k + q] - m[p * k + p]) / (2 * apq);
113
+ const t = Math.sign(theta || 1) / (Math.abs(theta) + Math.sqrt(theta * theta + 1));
114
+ const c = 1 / Math.sqrt(t * t + 1);
115
+ const s = t * c;
116
+ for (let r = 0; r < k; r++) {
117
+ const arp = m[r * k + p], arq = m[r * k + q];
118
+ m[r * k + p] = c * arp - s * arq;
119
+ m[r * k + q] = s * arp + c * arq;
120
+ }
121
+ for (let r = 0; r < k; r++) {
122
+ const apr = m[p * k + r], aqr = m[q * k + r];
123
+ m[p * k + r] = c * apr - s * aqr;
124
+ m[q * k + r] = s * apr + c * aqr;
125
+ }
126
+ }
127
+ }
128
+ }
129
+ return Array.from({ length: k }, (_, i) => m[i * k + i]);
130
+ }
131
+
132
+ /**
133
+ * What a search's variants say about it. `matrix[t][j]` is variant j's return in period t (the shape
134
+ * validate_overfitting takes). Constant variants are counted but left out of the correlations.
135
+ */
136
+ export function searchFromVariants(matrix, periodsPerYear) {
137
+ const periods = matrix.length;
138
+ const k = matrix[0].length;
139
+ const root = Math.sqrt(Number(periodsPerYear));
140
+ const means = new Float64Array(k);
141
+ for (let t = 0; t < periods; t++) for (let j = 0; j < k; j++) means[j] += Number(matrix[t][j]);
142
+ for (let j = 0; j < k; j++) means[j] /= periods;
143
+ const sds = new Float64Array(k);
144
+ for (let t = 0; t < periods; t++) for (let j = 0; j < k; j++) sds[j] += (Number(matrix[t][j]) - means[j]) ** 2;
145
+ for (let j = 0; j < k; j++) sds[j] = Math.sqrt(sds[j] / (periods - 1));
146
+ const live = [];
147
+ for (let j = 0; j < k; j++) if (sds[j] > 0) live.push(j);
148
+ const sharpes = live.map((j) => (means[j] / sds[j]) * root);
149
+ const sharpeMean = sharpes.reduce((a, b) => a + b, 0) / sharpes.length;
150
+ const sharpeSd = sharpes.length > 1 ? Math.sqrt(sharpes.reduce((a, b) => a + (b - sharpeMean) ** 2, 0) / (sharpes.length - 1)) : 0;
151
+ const L = live.length;
152
+ const corr = new Float64Array(L * L);
153
+ const pair = [];
154
+ for (let a = 0; a < L; a++) {
155
+ corr[a * L + a] = 1;
156
+ for (let b = a + 1; b < L; b++) {
157
+ const ja = live[a], jb = live[b];
158
+ let c = 0;
159
+ for (let t = 0; t < periods; t++) c += (Number(matrix[t][ja]) - means[ja]) * (Number(matrix[t][jb]) - means[jb]);
160
+ const r = c / ((periods - 1) * sds[ja] * sds[jb]);
161
+ corr[a * L + b] = corr[b * L + a] = r;
162
+ pair.push(r);
163
+ }
164
+ }
165
+ const pairMean = pair.length ? pair.reduce((a, b) => a + b, 0) / pair.length : 0;
166
+ const pairSd = pair.length > 1 ? Math.sqrt(pair.reduce((a, b) => a + (b - pairMean) ** 2, 0) / (pair.length - 1)) : 0;
167
+ let nyholt = L, liJi = L;
168
+ if (L > 1) {
169
+ const lambdas = symmetricEigenvalues(corr, L);
170
+ const lm = lambdas.reduce((a, b) => a + b, 0) / L;
171
+ const lv = lambdas.reduce((a, b) => a + (b - lm) ** 2, 0) / (L - 1);
172
+ nyholt = 1 + (L - 1) * (1 - lv / L);
173
+ liJi = lambdas.reduce((a, l) => { const x = Math.abs(l); return a + (x >= 1 ? 1 : 0) + (x - Math.floor(x)); }, 0);
174
+ }
175
+ let best = 0;
176
+ for (let i = 1; i < sharpes.length; i++) if (sharpes[i] > sharpes[best]) best = i;
177
+ return {
178
+ variants: k,
179
+ periods,
180
+ constant_variants: k - L,
181
+ best_variant: live.length ? live[best] : null,
182
+ best_sharpe_annualized: live.length ? sharpes[best] : null,
183
+ cross_trial_sharpe_sd_annualized: sharpeSd,
184
+ mean_pairwise_correlation: pairMean,
185
+ sd_pairwise_correlation: pairSd,
186
+ // Rounded up with a tolerance (a sum of eigenvalues equal to 12 can arrive as 12.000000000000014),
187
+ // never more than the variants that vary, and at least 2, the smallest count the DSR accepts.
188
+ effective_trials: { nyholt, li_ji: liJi, used: Math.max(2, Math.min(L, Math.ceil(liJi - 1e-9))), method: "Li and Ji (2005), rounded up, at most the variants sent, at least 2" },
189
+ };
190
+ }
191
+
192
+ /**
193
+ * The Null Zoo family whose measured size an audit quotes for this shape. The rules and their order
194
+ * are the ones pre-registered in config/research/null-zoo-v1-prereg.json; a test keeps them equal.
195
+ */
196
+ export function nullZooFamily(shape, search = null) {
197
+ if (search && search.mean_pairwise_correlation >= 0.3) return search.sd_pairwise_correlation >= 0.15 ? "block_cluster" : "correlated_trials";
198
+ if (Math.abs(shape.lag1_autocorrelation) >= 0.1) return "ar1";
199
+ if (shape.skew <= -0.5) return "skew_negative";
200
+ if (shape.skew >= 0.5) return "skew_positive";
201
+ if (shape.excess_kurtosis >= 2 && shape.lag1_autocorrelation_of_squares >= 0.1) return "garch";
202
+ if (shape.excess_kurtosis >= 2) return "student_t4";
203
+ if (shape.lag1_autocorrelation_of_squares >= 0.1) return "regimes";
204
+ return "iid_normal";
205
+ }
206
+
207
+ /** The t-statistic of a series' mean with a Newey-West (Bartlett, lag floor(4 (T/100)^(2/9))) variance. */
208
+ export function neweyWestT(xs) {
209
+ const n = xs.length;
210
+ const m = mean(xs);
211
+ const lag = Math.floor(4 * Math.pow(n / 100, 2 / 9));
212
+ let lrv = 0;
213
+ for (let i = 0; i < n; i++) { const d = xs[i] - m; lrv += d * d; }
214
+ for (let l = 1; l <= lag; l++) {
215
+ let g = 0;
216
+ for (let i = l; i < n; i++) g += (xs[i] - m) * (xs[i - l] - m);
217
+ lrv += 2 * (1 - l / (lag + 1)) * g;
218
+ }
219
+ lrv /= n;
220
+ return lrv > 0 ? m / Math.sqrt(lrv / n) : 0;
221
+ }
222
+
223
+ /** One-sided p-value of "the mean is above zero" from a stationary-bootstrap t (see the header). */
224
+ export function blockBootstrapT(xs, { reps, block = defaultBlock(xs.length), seed }) {
225
+ const n = xs.length;
226
+ let sum = 0, sq = 0;
227
+ for (let i = 0; i < n; i++) { sum += xs[i]; sq += xs[i] * xs[i]; }
228
+ const m0 = sum / n;
229
+ const sd = Math.sqrt(Math.max((sq - n * m0 * m0) / (n - 1), 0));
230
+ if (!(sd > 0)) return 1;
231
+ const observed = m0 / (sd / Math.sqrt(n));
232
+ const next = makeRandom(seed);
233
+ const idx = new Int32Array(n);
234
+ let hits = 0;
235
+ for (let b = 0; b < reps; b++) {
236
+ stationaryIndices(n, block, next, idx);
237
+ let s = 0, q = 0;
238
+ for (let i = 0; i < n; i++) { const v = xs[idx[i]]; s += v; q += v * v; }
239
+ const m = s / n;
240
+ const v = (q - n * m * m) / (n - 1);
241
+ if (!(v > 0)) { hits++; continue; }
242
+ if ((m - m0) / Math.sqrt(v / n) >= observed) hits++;
243
+ }
244
+ return (hits + 1) / (reps + 1);
245
+ }
246
+
247
+ const quantileSorted = (sorted, q) => {
248
+ const h = (sorted.length - 1) * q;
249
+ const lo = Math.floor(h);
250
+ const hi = Math.min(lo + 1, sorted.length - 1);
251
+ return sorted[lo] + (h - lo) * (sorted[hi] - sorted[lo]);
252
+ };
253
+
254
+ /**
255
+ * Out-of-sample decay from CSCV's in-sample-best pairs ([in-sample Sharpe, out-of-sample Sharpe],
256
+ * per period, as js/pbo-core.js returns them): the distribution of the selected variant's
257
+ * out-of-sample Sharpe, and the least-squares line of out-of-sample on in-sample. With
258
+ * periodsPerYear 1 the Sharpe ratios stay per period, as validate_overfitting reports them.
259
+ */
260
+ export function oosDecay(isOosPairs, periodsPerYear = 1) {
261
+ const root = Math.sqrt(Number(periodsPerYear));
262
+ const pairs = isOosPairs.map(([a, b]) => [a * root, b * root]);
263
+ const oos = pairs.map((p) => p[1]).sort((a, b) => a - b);
264
+ const n = pairs.length;
265
+ const mx = pairs.reduce((a, p) => a + p[0], 0) / n;
266
+ const my = pairs.reduce((a, p) => a + p[1], 0) / n;
267
+ let sxy = 0, sxx = 0, syy = 0;
268
+ for (const [x, y] of pairs) { sxy += (x - mx) * (y - my); sxx += (x - mx) ** 2; syy += (y - my) ** 2; }
269
+ const slope = sxx > 0 ? sxy / sxx : 0;
270
+ return {
271
+ splits: n,
272
+ unit: Number(periodsPerYear) === 1 ? "per period" : "annualized",
273
+ oos_sharpe: {
274
+ median: quantileSorted(oos, 0.5), p10: quantileSorted(oos, 0.1), p90: quantileSorted(oos, 0.9),
275
+ prob_below_zero: oos.filter((v) => v < 0).length / n,
276
+ },
277
+ degradation: { slope, intercept: my - slope * mx, r2: sxx > 0 && syy > 0 ? (sxy * sxy) / (sxx * syy) : 0 },
278
+ };
279
+ }
280
+
281
+ /** A per-period oosDecay result in annualized Sharpe units; the slope, r2 and shares do not change. */
282
+ export function annualizeDecay(decay, periodsPerYear) {
283
+ const root = Math.sqrt(Number(periodsPerYear));
284
+ const o = decay.oos_sharpe;
285
+ return {
286
+ ...decay,
287
+ unit: "annualized",
288
+ oos_sharpe: { median: o.median * root, p10: o.p10 * root, p90: o.p90 * root, prob_below_zero: o.prob_below_zero },
289
+ degradation: { ...decay.degradation, intercept: decay.degradation.intercept * root },
290
+ };
291
+ }
292
+
293
+ // The ids are stable: an agent can act on them, and a changelog can name them.
294
+ export const FIX_IDS = Object.freeze([
295
+ "count_every_variant", "declared_trials_below_counted", "not_distinguishable_from_luck", "interval_includes_zero",
296
+ "autocorrelated_returns", "track_record_too_short", "overfit_selection", "expected_live_sharpe_below_zero",
297
+ "stress_and_capacity_next",
298
+ ]);
299
+
300
+ /**
301
+ * At most five next steps, most decisive first. `context` carries what the audit found; each step
302
+ * is {id, why, next}: the finding with its numbers, and the change or call to make next.
303
+ */
304
+ export function fixNext(context) {
305
+ const steps = [];
306
+ const add = (id, why, next) => steps.push({ id, why, next });
307
+ const { headline, shape, interval, search, declaredTrials, trackRecord, overfitting, decay } = context;
308
+ const f2 = (x) => x.toFixed(2);
309
+ if (headline && headline.p > headline.level) {
310
+ add("not_distinguishable_from_luck", `p ${headline.p.toFixed(3)} > ${headline.level}.`, search ? "Pre-declare a smaller grid (backtest_strategy), then audit again." : "Try fewer variants or add out-of-sample data.");
311
+ }
312
+ if (search && declaredTrials !== undefined && declaredTrials < search.effective_trials.used) {
313
+ add("declared_trials_below_counted", `Declared ${declaredTrials} trials; the variants count as ${search.effective_trials.used}.`, "Already applied: the counted number was used.");
314
+ }
315
+ if (!search) add("count_every_variant", "Only declared trials counted.", "Send every variant's returns (variants_file).");
316
+ if (Math.abs(shape.lag1_autocorrelation) >= 0.1) {
317
+ add("autocorrelated_returns", `Lag-1 autocorrelation ${f2(shape.lag1_autocorrelation)}; Lo-adjusted Sharpe ${f2(shape.lo_adjusted_sharpe_annualized)}.`,
318
+ "Check for stale or smoothed prices (summarize_series); quote the adjusted Sharpe.");
319
+ }
320
+ if (interval && interval.intervals["0.95"][0] <= 0) {
321
+ add("interval_includes_zero", `95% Sharpe interval ${f2(interval.intervals["0.95"][0])} to ${f2(interval.intervals["0.95"][1])}.`, "Needs a longer history (validate_backtest_length).");
322
+ }
323
+ if (overfitting && overfitting.pbo >= 0.5) add("overfit_selection", `Overfitting probability ${f2(overfitting.pbo)}.`, "Shrink the grid or fix the selection rule in advance.");
324
+ if (decay && decay.oos_sharpe.median <= 0) add("expected_live_sharpe_below_zero", `Median out-of-sample Sharpe of the pick ${f2(decay.oos_sharpe.median)}.`, "Expect no live edge from this selection.");
325
+ if (trackRecord && trackRecord.short) add("track_record_too_short", trackRecord.finding, "Keep it running before scaling (validate_track_record).");
326
+ // Nothing to fix and luck rejected: the next hurdles are bad markets and a real broker.
327
+ if (!steps.length && headline && headline.p <= headline.level) add("stress_and_capacity_next", `p ${headline.p.toFixed(3)} <= ${headline.level}.`, "Run stress_test and check_feasibility before trading it.");
328
+ return steps.slice(0, 5);
329
+ }
@@ -0,0 +1,40 @@
1
+ // Generated by scripts/research/null-zoo/v1/evaluate.mjs from the Null Zoo v1 run. Do not edit.
2
+ // The measured false-positive rate (size) of audit_backtest's headline tests on each return family.
3
+ export const NULL_ZOO_V1 = Object.freeze({
4
+ "source": "https://github.com/arhancanli/canlicapital/blob/main/config/research/null-zoo-v1b-evaluation.json (10,000 searches per cell; bar pre-registered in config/research/null-zoo-v1b-prereg.json)",
5
+ "level": 0.05,
6
+ "with_variants": {
7
+ "test": "reality_check",
8
+ "size_by_family": {
9
+ "iid_normal": 0.0538,
10
+ "student_t4": 0.0494,
11
+ "skew_negative": 0.0535,
12
+ "skew_positive": 0.0521,
13
+ "garch": 0.0446,
14
+ "ar1": 0.0637,
15
+ "regimes": 0.0507,
16
+ "correlated_trials": 0.0553,
17
+ "block_cluster": 0.0565
18
+ },
19
+ "worst": "ar1",
20
+ "meets_every_bar": false,
21
+ "power_iid_skill_2": 0.5346
22
+ },
23
+ "single_series": {
24
+ "test": "luck_trials_lo",
25
+ "size_by_family": {
26
+ "iid_normal": 0.0551,
27
+ "student_t4": 0.0561,
28
+ "skew_negative": 0.0877,
29
+ "skew_positive": 0.0317,
30
+ "garch": 0.0487,
31
+ "ar1": 0.0549,
32
+ "regimes": 0.0551,
33
+ "correlated_trials": 0.0358,
34
+ "block_cluster": 0.0356
35
+ },
36
+ "worst": "skew_negative",
37
+ "meets_every_bar": false,
38
+ "power_iid_skill_2": 0.5276
39
+ }
40
+ });
@@ -3,6 +3,7 @@
3
3
  // MCP package's local mode (mcp/src/local mirrors it byte for byte), so the two cannot disagree.
4
4
  import { finiteNumbers } from "../moments-core.js";
5
5
  import { pboCscv } from "../pbo-core.js";
6
+ import { oosDecay } from "../audit-core.js";
6
7
  import { LIMITS } from "../../api/_lib/limits.js";
7
8
 
8
9
  const mean = (xs) => xs.reduce((a, b) => a + (Number.isFinite(b) ? b : 0), 0) / xs.length;
@@ -30,6 +31,8 @@ export function compute(body) {
30
31
  block_length: out.block_length,
31
32
  lambda_quantiles: { p05: q(0.05), p25: q(0.25), p50: q(0.5), p75: q(0.75), p95: q(0.95) },
32
33
  is_oos_summary: { mean_is_sharpe_of_selected: mean(out.is_oos_pairs.map((p) => p[0])), mean_oos_sharpe_of_selected: mean(out.is_oos_pairs.map((p) => p[1])), share_oos_below_is: degraded },
34
+ // What the in-sample best did out of sample across the splits, per period like is_oos_summary.
35
+ oos_decay: oosDecay(out.is_oos_pairs),
33
36
  plain_reading: `In ${(out.pbo * 100).toFixed(1)} percent of the ${out.n_combinations} in-sample and out-of-sample splits, the variant that looked best in sample ranked in the worse half out of sample. That share is the probability of backtest overfitting for this set of variants as submitted.`,
34
37
  };
35
38
  }
package/src/schemas.mjs CHANGED
@@ -40,7 +40,7 @@ export const FIELD_DESCRIPTIONS = Object.freeze({
40
40
  record_observations: "Record length so far, for its probabilistic Sharpe.",
41
41
  label: "Name for the key.",
42
42
  receipt_id: "Receipt id from a validation result.",
43
- receipt_object: "A receipt as get_receipt returns it, to verify without fetching.",
43
+ receipt_object: "A receipt object, to verify without fetching.",
44
44
  cik: "SEC CIK; send cik or ticker.",
45
45
  ticker: "Ticker such as AAPL; send ticker or cik.",
46
46
  concept: "us-gaap concept such as Assets; omit to list them.",
@@ -238,8 +238,8 @@ export const auditBacktestToolShape = z
238
238
  returns_file: z.string().min(1).max(4096).optional().describe(d.returns_file),
239
239
  returns_column: z.union([z.string().min(1).max(200), z.number().int().min(1)]).optional().describe(d.returns_column),
240
240
  periods_per_year: z.number().min(1).max(10000).describe(d.periods_per_year),
241
- effective_independent_trials: z.number().int().min(2).max(10000000).describe(d.effective_independent_trials),
242
- cross_trial_sharpe_sd_annualized: z.number().min(0).max(10).describe(d.cross_trial_sharpe_sd_annualized),
241
+ effective_independent_trials: z.number().int().min(2).max(10000000).optional().describe(`${d.effective_independent_trials} Optional with variants: counted from them, and the larger number is used.`),
242
+ cross_trial_sharpe_sd_annualized: z.number().min(0).max(10).optional().describe(`${d.cross_trial_sharpe_sd_annualized} Optional with variants: measured from them, and the larger is used.`),
243
243
  benchmark_sharpe_annualized: z.number().min(-10).max(10).optional().describe(d.benchmark_sharpe_annualized),
244
244
  confidence: z.number().gt(0).lt(1).optional().describe(d.confidence),
245
245
  variants: z.array(z.array(z.number()).max(200)).min(2).max(20000).optional().describe(d.variants),
@@ -253,7 +253,9 @@ export const auditBacktestToolShape = z
253
253
  export const auditBacktestInput = auditBacktestToolShape
254
254
  .refine((v) => (v.returns === undefined) !== (v.returns_file === undefined), "Send exactly one of returns or returns_file")
255
255
  .refine((v) => v.variants === undefined || v.variants_file === undefined, "Send variants or variants_file, not both")
256
- .refine((v) => v.returns_column === undefined || v.returns_file !== undefined, "returns_column applies only to returns_file");
256
+ .refine((v) => v.returns_column === undefined || v.returns_file !== undefined, "returns_column applies only to returns_file")
257
+ .refine((v) => v.variants !== undefined || v.variants_file !== undefined || (v.effective_independent_trials !== undefined && v.cross_trial_sharpe_sd_annualized !== undefined),
258
+ "Send variants or variants_file, or declare effective_independent_trials and cross_trial_sharpe_sd_annualized");
257
259
 
258
260
  // ---------------------------------------------------------------------------------------------
259
261
  // get_key / get_receipt
@@ -273,12 +275,13 @@ export const getReceiptInput = z
273
275
 
274
276
  export const emptyInput = z.object({}).strict();
275
277
 
276
- // verify_receipt: a receipt id to fetch, or a receipt already fetched (get_receipt's data), to check
278
+ // verify_receipt: a receipt id to fetch, or a receipt already fetched, to check
277
279
  // offline. Advertised flat; the handler enforces exactly one.
278
280
  export const verifyReceiptToolShape = z
279
281
  .object({
280
282
  id: z.string().regex(/^[0-9a-f]{24}$/, "A receipt id is 24 hex characters").optional().describe(d.receipt_id),
281
283
  receipt: z.record(z.string(), z.unknown()).optional().describe(d.receipt_object),
284
+ include_receipt: z.boolean().optional().describe("With id: also return the stored receipt."),
282
285
  })
283
286
  .strict();
284
287
 
@@ -334,7 +337,7 @@ export const REGISTRY_DESCRIPTION_MAX = 100;
334
337
  export const TOOL_DESCRIPTIONS = Object.freeze({
335
338
  get_key: `Issue a free validation key for this session. Rarely needed: the first validation issues one itself unless CANLI_KEY or local mode is set, and the read tools need none. ${LIMITS_SENTENCES.quotas}`,
336
339
  validate_deflated_sharpe: `Deflated Sharpe ratio: the probability (0 to 1) that the selected strategy's Sharpe beats the best that luck gives across the variants tried, with the probabilistic Sharpe and that luck benchmark. Send the seven statistics or a return series. With every variant's returns use validate_overfitting; luck as a trial count, validate_luck_trials; a multiple-testing haircut, validate_haircut_sharpe. ${LIMITS_SENTENCES.notAdmission}`,
337
- audit_backtest: `One-call audit of a strategy's returns: deflated Sharpe, minimum track record and, with every variant's returns, the probability of backtest overfitting, each the matching validator's result with its own receipt. Point returns_file at the backtest's CSV or JSON instead of pasting long series. Prefer it to calling the validators one by one; one validation per check. ${LIMITS_SENTENCES.notAdmission}`,
340
+ audit_backtest: `One-call audit of a strategy's returns. Headline: one test whose false-positive rate was measured on nine return shapes (Hansen's SPA with variants). Then deflated Sharpe, minimum track record and, with variants, overfitting and out-of-sample decay, each with its receipt, plus fix_next. Send every variant tried (variants_file) so trials are counted; point returns_file at a CSV instead of pasting. ${LIMITS_SENTENCES.notAdmission}`,
338
341
  validate_overfitting: `Probability of backtest overfitting (0 to 1) by CSCV: how often the in-sample best variant falls below the out-of-sample median. Needs every variant's returns (periods by variants); with summary statistics only, use validate_deflated_sharpe. ${LIMITS_SENTENCES.notAdmission}`,
339
342
  validate_reality_check: `Data-snooping tests on every variant a search tried: Hansen's SPA p-value that the best beat the benchmark only by luck, White's Reality Check, and the variants Romano-Wolf StepM finds better. Send all variants tried, not only the winners. ${LIMITS_SENTENCES.notAdmission}`,
340
343
  validate_paper_evidence: `Checks paper-evidence.v0 structure and disclosures. With CANLI_LOCAL=1, record_file reads an export bundle and journal_file verifies signatures, source hashes and recomputed claims. Without a journal, source facts and signatures are unchecked. Local files are never uploaded. ${LIMITS_SENTENCES.scope}`,
@@ -344,7 +347,6 @@ export const TOOL_DESCRIPTIONS = Object.freeze({
344
347
  validate_track_record: `Minimum track record length (observations and years) for an observed Sharpe to beat a benchmark at a confidence level; with observations, the record's probabilistic Sharpe so far. For live or paper records; to size a backtest for its trials, use validate_backtest_length. ${LIMITS_SENTENCES.notAdmission}`,
345
348
  validate_breadth: `Book Sharpe ceiling from adding sleeves of this quality and correlation, the Sharpe at a sleeve count, and the sleeves a target needs. For portfolio construction; it validates no single strategy. ${LIMITS_SENTENCES.scope}`,
346
349
  verify_receipt: `Verify a receipt offline: its Ed25519 signature against the bundled canlicapital.com key, its output hash and its id. Send an id to fetch it first, or the receipt itself. ${LIMITS_SENTENCES.unsigned}`,
347
- get_receipt: `Fetch a stored verdict by receipt id to re-read it. No key; verify_receipt checks it is genuine. ${LIMITS_SENTENCES.unsigned}`,
348
350
  service_status: `Whether the validation API is up, with its quotas; check after a timeout before resubmitting. No key. ${LIMITS_SENTENCES.scope}`,
349
351
  company_financial_history: `SEC-reported financial history for one company in the canlicapital.com reference, by cik or ticker: without a concept, the histories available; with one, observations newest first with accession, form, filed date and unit, plus the source's SHA-256. For point-in-time values use canli-fundamentals-mcp. ${COMPANY_REFERENCE_BOUNDARY}`,
350
352
  });
package/src/server.mjs CHANGED
@@ -15,6 +15,12 @@ import { validateLocalJournalEvidence } from "./journal-evidence.mjs";
15
15
  import { readMatrixFile, readSeriesFile } from "./series-file.mjs";
16
16
  import { LAB_TOOLS, labToolSpecs, registerCodeResources, registerLabPrompts } from "./lab.mjs";
17
17
  import { verifyReceipt } from "./local/js/receipt-statement.js";
18
+ import { annualizeDecay, blockBootstrapT, fixNext, neweyWestT, nullZooFamily, returnShape, searchFromVariants, sharpeInterval } from "./local/js/audit-core.js";
19
+ import { NULL_ZOO_V1 } from "./local/js/null-zoo-v1-sizes.js";
20
+ import { haircutSharpe } from "./local/js/haircut-core.js";
21
+ import { bestOfTrialsProbability } from "./local/js/luck-core.js";
22
+ import { minimumBacktestLength } from "./local/js/dsr-core.js";
23
+ import { studentTUpper } from "./local/js/student-t.js";
18
24
  import { StdioServerTransport } from "@modelcontextprotocol/server/stdio";
19
25
  import { breadthInput, trackRecordInput, auditBacktestInput, verifyReceiptToolShape, backtestLengthInput, haircutSharpeInput, luckTrialsInput, auditBacktestToolShape, companyHistoryInput, companyHistoryToolShape, deflatedSharpeInput, deflatedSharpeToolShape, emptyInput, getKeyInput, getReceiptInput, overfittingInput, realityCheckInput, realityCheckToolShape, LIMITS_SENTENCES, paperEvidenceInput, paperEvidenceToolShape, TOOL_DESCRIPTIONS, validationOutput, auditOutput, keyOutput, receiptOutput, verifyReceiptOutput, statusOutput, companyHistoryOutput } from "./schemas.mjs";
20
26
 
@@ -66,21 +72,24 @@ export function configuredLocal(value) {
66
72
 
67
73
  // Toolsets: which tools the server lists. The tool list is re-sent to the model on every turn and
68
74
  // is most of each turn's prompt (README, "Toolsets"; bench/tool_list_tokens.py measures it), so a
69
- // client that needs one kind of tool can load only that kind. Default: all.
75
+ // client that needs one kind of tool can load only that kind. Default: every toolset except
76
+ // company, whose one tool canli-fundamentals-mcp does better and point in time; "all" adds it.
70
77
  export const TOOLSETS = Object.freeze({
71
78
  validate: Object.freeze(["get_key", "validate_deflated_sharpe", "validate_overfitting", "validate_paper_evidence", "validate_breadth", "validate_track_record", "validate_backtest_length", "validate_haircut_sharpe", "validate_luck_trials", "validate_reality_check", "audit_backtest"]),
72
- receipts: Object.freeze(["get_receipt", "verify_receipt"]),
79
+ receipts: Object.freeze(["verify_receipt"]),
73
80
  company: Object.freeze(["company_financial_history"]),
74
81
  status: Object.freeze(["service_status"]),
75
82
  lab: Object.freeze([...LAB_TOOLS]),
76
83
  });
77
84
 
78
85
  // CANLI_TOOLSETS=validate,company (or ?toolsets= on the hosted endpoint): a comma-separated list of
79
- // TOOLSETS names, or "all". Empty or unsubstituted means all; an unknown name is refused, so a typo
80
- // never silently leaves a client without the tools it asked for.
86
+ // TOOLSETS names, or "all". Empty or unsubstituted means the default; an unknown name is refused, so
87
+ // a typo never silently leaves a client without the tools it asked for.
88
+ export const DEFAULT_TOOLSETS = Object.freeze(Object.keys(TOOLSETS).filter((name) => name !== "company"));
81
89
  export function configuredToolsets(value) {
82
90
  const v = configuredKey(value);
83
- if (!v || v.trim().toLowerCase() === "all") return Object.keys(TOOLSETS);
91
+ if (!v) return [...DEFAULT_TOOLSETS];
92
+ if (v.trim().toLowerCase() === "all") return Object.keys(TOOLSETS);
84
93
  const names = [...new Set(v.split(",").map((s) => s.trim().toLowerCase()).filter(Boolean))];
85
94
  const unknown = names.filter((n) => !Object.hasOwn(TOOLSETS, n));
86
95
  if (unknown.length || !names.length) throw new Error(`Unknown toolset ${unknown.join(", ") || "(none)"}; choose from ${Object.keys(TOOLSETS).join(", ")} or all`);
@@ -169,7 +178,7 @@ const asText = (envelope, failed = false) => ({
169
178
  // A validation result as the model reads it: the answer, the sentences saying what it does not
170
179
  // establish, and the receipt that holds the rest. Metadata (schema, endpoint, timestamps, claim and
171
180
  // capital class, human page), the source hashes and the quota sentence are about the service, not
172
- // the answer; they stay in the stored receipt (get_receipt) and service_status, and
181
+ // the answer; they stay in the stored receipt (verify_receipt with include_receipt) and service_status, and
173
182
  // CANLI_FULL_ENVELOPE=1 returns every field. On a result this cuts the text roughly in half.
174
183
  const QUOTA_SENTENCE = /^Quotas:/;
175
184
 
@@ -329,11 +338,58 @@ async function runValidator(session, tool, path, body) {
329
338
 
330
339
  const sameJson = (a, b) => JSON.stringify(a) === JSON.stringify(b);
331
340
 
332
- // audit_backtest: deflated Sharpe from the series, then the minimum track record length from the
333
- // Sharpe, skew and kurtosis that check derived, then (with variants) CSCV overfitting. Each check
334
- // keeps its envelope and receipt; the limits every envelope repeats are stated once at the top.
335
- // A refused later check (for example a Sharpe that does not exceed the benchmark) is reported as
336
- // that check's error, not as a failed audit; the audit fails only when the first check does.
341
+ // audit_backtest: one calibrated test first, then the checks behind it.
342
+ // headline with every variant's returns, Hansen's SPA (validate_reality_check, with its receipt);
343
+ // with one series, the single-series test Null Zoo v1 chose. Either way the result quotes
344
+ // that test's measured false-positive rate on returns shaped like these
345
+ // (local/js/null-zoo-v1-sizes.js) and the worst across the nine families.
346
+ // checks the deflated Sharpe (an estimate), the minimum track record and, with variants, CSCV
347
+ // overfitting with out-of-sample decay: each the validator's own result and receipt.
348
+ // computed in this process from the series sent: its shape, Lo's adjusted Sharpe, a bootstrap
349
+ // interval for the Sharpe, the haircut Sharpe, the minimum backtest length, and what the
350
+ // variants say about the search. Trials and their spread are the larger of declared and
351
+ // counted, so a search cannot be made to look smaller than it was.
352
+ // A refused later check is reported as that check's error, not as a failed audit; the audit fails
353
+ // only when the first check does.
354
+ export const AUDIT_SPA_REPS = 2000;
355
+
356
+ function singleSeriesHeadline(test, returns, trials) {
357
+ const xs = Float64Array.from(returns, Number);
358
+ if (test === "hac_t_sidak") return { p: bestOfTrialsProbability(studentTUpper(neweyWestT(xs), xs.length - 1), trials) };
359
+ if (test === "bootstrap_t_sidak") {
360
+ // Enough resamples that the smallest attainable p-value sits well inside the Sidak level for
361
+ // this many trials, up to a cap; beyond it the floor is stated.
362
+ const perTest = -Math.expm1(Math.log1p(-0.05) / trials);
363
+ const reps = Math.min(200000, Math.max(4999, Math.ceil(5 / perTest)));
364
+ const p = bestOfTrialsProbability(blockBootstrapT(xs, { reps, seed: 42 }), trials);
365
+ const floor = bestOfTrialsProbability(1 / (reps + 1), trials);
366
+ return { p, reps, ...(floor > 0.01 ? { resolution_floor: floor } : {}) };
367
+ }
368
+ if (test === "luck_trials_lo") return null; // run through validate_luck_trials below, for its receipt
369
+ throw new Error(`audit_backtest: no single-series headline test named ${test}`);
370
+ }
371
+
372
+ function headlineResult({ path, test, p, shape, search, receipt, extra, full }) {
373
+ const sizes = NULL_ZOO_V1[path];
374
+ const family = nullZooFamily(shape, search);
375
+ const size = sizes.size_by_family[family];
376
+ const worstSize = sizes.size_by_family[sizes.worst];
377
+ const pct = (x) => `${(x * 100).toFixed(1)}%`;
378
+ const rejects = p <= NULL_ZOO_V1.level;
379
+ return {
380
+ test,
381
+ p,
382
+ rejects_luck: rejects,
383
+ measured_size: { family, here: size, worst: worstSize, ...(full ? { worst_family: sizes.worst, source: NULL_ZOO_V1.source } : {}) },
384
+ sentence: `${rejects ? "Rejects" : "Cannot reject"} luck at ${pct(NULL_ZOO_V1.level)} (p ${p.toFixed(3)}); measured false-positive rate ${pct(size)} on ${family}-like returns, ${pct(worstSize)} worst.`,
385
+ ...(receipt ? { receipt } : {}),
386
+ ...(full ? { level: NULL_ZOO_V1.level, ...(extra ?? {}) } : {}),
387
+ };
388
+ }
389
+
390
+ const round = (x, digits = 4) => (Number.isFinite(x) ? Number(x.toPrecision(digits)) : x);
391
+ const roundAll = (o, digits = 4) => (typeof o === "number" ? round(o, digits) : Array.isArray(o) ? o.map((v) => roundAll(v, digits)) : o && typeof o === "object" ? Object.fromEntries(Object.entries(o).map(([k, v]) => [k, roundAll(v, digits)])) : o);
392
+
337
393
  export async function toolAuditBacktest(session, args) {
338
394
  const input = parseOrThrow(auditBacktestInput, args, "audit_backtest");
339
395
  if ((input.returns_file || input.variants_file) && session.hosted) {
@@ -343,9 +399,16 @@ export async function toolAuditBacktest(session, args) {
343
399
  const variantsRead = input.variants_file ? readMatrixFile(input.variants_file) : null;
344
400
  const returns = input.returns ?? returnsRead.values;
345
401
  const variants = input.variants ?? variantsRead?.matrix;
346
- const { periods_per_year, effective_independent_trials, cross_trial_sharpe_sd_annualized } = input;
402
+ const { periods_per_year } = input;
403
+
404
+ const shape = returnShape(returns, periods_per_year);
405
+ const search = variants ? searchFromVariants(variants, periods_per_year) : null;
406
+ const declaredTrials = input.effective_independent_trials;
407
+ const trials = Math.max(declaredTrials ?? 0, search?.effective_trials.used ?? 0);
408
+ const crossSd = Math.max(input.cross_trial_sharpe_sd_annualized ?? 0, search?.cross_trial_sharpe_sd_annualized ?? 0);
409
+
347
410
  const dsr = await runValidator(session, "validate_deflated_sharpe", "/api/v1/validate/deflated-sharpe", {
348
- returns, periods_per_year, effective_independent_trials, cross_trial_sharpe_sd_annualized,
411
+ returns, periods_per_year, effective_independent_trials: trials, cross_trial_sharpe_sd_annualized: crossSd,
349
412
  });
350
413
  if (dsr.failed) return validationText(session, { envelope: dsr.envelope, failed: true });
351
414
  const derived = dsr.envelope?.data?.derived_inputs ?? {};
@@ -365,20 +428,102 @@ export async function toolAuditBacktest(session, args) {
365
428
  ...(input.n_splits !== undefined ? { n_splits: input.n_splits } : {}),
366
429
  })
367
430
  : null;
368
- const shape = (e) => (session.fullEnvelope ? e : compactEnvelope(e));
369
- const envelopes = { deflated_sharpe: shape(dsr.envelope), track_record: shape(track.envelope), ...(overfit ? { overfitting: shape(overfit.envelope) } : {}) };
431
+
432
+ // The headline test.
433
+ let headline;
434
+ if (variants) {
435
+ // One stationary bootstrap gives the Reality Check and the three SPA p-values; the headline is
436
+ // whichever of them Null Zoo selected, and the result names it.
437
+ const test = NULL_ZOO_V1.with_variants.test;
438
+ // A later check that cannot run is that check's error, not a failed audit.
439
+ const rc = await runValidator(session, "validate_reality_check", "/api/v1/validate/reality-check", { matrix: variants, reps: AUDIT_SPA_REPS, seed: 42 })
440
+ .catch((e) => ({ failed: true, envelope: { error: { code: "unreachable", message: e.message } } }));
441
+ const data = rc.envelope?.data;
442
+ const p = { reality_check: data?.reality_check?.p_value, spa_consistent: data?.spa?.p_value, spa_upper: data?.spa?.p_value_upper }[test];
443
+ headline = rc.failed || typeof p !== "number"
444
+ ? { test, error: rc.envelope?.error?.message ?? "the data-snooping check returned no p-value" }
445
+ : headlineResult({ path: "with_variants", test, p, shape, search, receipt: compactEnvelope(rc.envelope).receipt ?? null, extra: { spa_consistent: data.spa?.p_value, reality_check: data.reality_check?.p_value, monte_carlo_se: Math.sqrt((p * (1 - p)) / AUDIT_SPA_REPS) }, full: session.fullEnvelope });
446
+ } else {
447
+ const test = NULL_ZOO_V1.single_series.test;
448
+ const local = singleSeriesHeadline(test, returns, trials);
449
+ if (local) {
450
+ headline = headlineResult({ path: "single_series", test, p: local.p, shape, search, extra: { trials, computed: "in this process", ...(local.reps ? { reps: local.reps } : {}) }, full: session.fullEnvelope });
451
+ if (local.resolution_floor) headline.resolution_floor = local.resolution_floor;
452
+ } else {
453
+ const luck = await runValidator(session, "validate_luck_trials", "/api/v1/validate/luck-trials", {
454
+ observed_sharpe_annualized: shape.sharpe_annualized, observations: shape.observations, periods_per_year, effective_independent_trials: trials, autocorrelation: Math.max(-0.95, Math.min(0.95, shape.lag1_autocorrelation)),
455
+ }).catch((e) => ({ failed: true, envelope: { error: { code: "unreachable", message: e.message } } }));
456
+ const p = luck.envelope?.data?.result?.best_of_trials_probability;
457
+ headline = luck.failed || typeof p !== "number"
458
+ ? { test, error: luck.envelope?.error?.message ?? "the luck-trials check returned no probability" }
459
+ : headlineResult({ path: "single_series", test, p, shape, search, receipt: compactEnvelope(luck.envelope).receipt ?? null, extra: { trials }, full: session.fullEnvelope });
460
+ }
461
+ }
462
+
463
+ const interval = sharpeInterval(returns, { periodsPerYear: periods_per_year });
464
+ const haircut = shape.sharpe_annualized > 0
465
+ ? haircutSharpe({ sharpeAnnualized: shape.sharpe_annualized, periodsPerYear: periods_per_year, observations: shape.observations, tests: trials, autocorrelation: Math.max(-0.95, Math.min(0.95, shape.lag1_autocorrelation)) })
466
+ : null;
467
+ const years = shape.observations / periods_per_year;
468
+ const minBtl = shape.sharpe_annualized > 0 ? minimumBacktestLength({ trials, targetSharpe: shape.sharpe_annualized }) : null;
469
+ const decayPerPeriod = overfit && !overfit.failed ? overfit.envelope?.data?.oos_decay : null;
470
+ const decay = decayPerPeriod ? annualizeDecay(decayPerPeriod, periods_per_year) : null;
471
+ const trackResult = track.failed ? null : track.envelope?.data?.result;
472
+ const fix = fixNext({
473
+ headline: headline.error ? null : { ...headline, level: NULL_ZOO_V1.level },
474
+ shape, interval, search, declaredTrials,
475
+ overfitting: overfit && !overfit.failed ? { pbo: overfit.envelope?.data?.pbo } : null,
476
+ decay,
477
+ trackRecord: trackResult?.record && !trackResult.record.long_enough
478
+ ? { short: true, finding: `The minimum track record for this Sharpe is ${trackResult.minimum_observations} periods; the series has ${trackResult.record.observations}.` }
479
+ : null,
480
+ });
481
+
482
+ // Compact: each check's plain reading is stated once, in readings, not again inside the check.
483
+ const dropReading = (e) => (e?.data && typeof e.data === "object" && "plain_reading" in e.data ? { ...e, data: Object.fromEntries(Object.entries(e.data).filter(([k]) => k !== "plain_reading")) } : e);
484
+ // Local mode: the same "computed locally" note on every check is stated once at the top.
485
+ const localNote = !session.fullEnvelope && dsr.envelope?.computed ? { computed: dsr.envelope.computed, note: dsr.envelope.note } : null;
486
+ const dropLocal = (e) => (localNote && e?.computed === localNote.computed && e?.note === localNote.note ? Object.fromEntries(Object.entries(e).filter(([k]) => k !== "computed" && k !== "note")) : e);
487
+ const shapeEnv = (e) => (session.fullEnvelope ? e : dropLocal(dropReading(compactEnvelope(e))));
488
+ const envelopes = { deflated_sharpe: shapeEnv(dsr.envelope), track_record: shapeEnv(track.envelope), ...(overfit ? { overfitting: shapeEnv(overfit.envelope) } : {}) };
370
489
  const limits = envelopes.deflated_sharpe?.limits;
371
490
  const shared = Object.values(envelopes).every((e) => sameJson(e?.limits, limits));
372
491
  const checks = Object.fromEntries(
373
492
  Object.entries(envelopes).map(([name, e]) => [name, shared ? Object.fromEntries(Object.entries(e).filter(([k]) => k !== "limits")) : e]),
374
493
  );
375
494
  const readings = Object.fromEntries(
376
- Object.entries(envelopes).map(([name, e]) => [name, e?.data?.plain_reading ?? e?.error?.message ?? null]),
495
+ Object.entries({ deflated_sharpe: dsr.envelope, track_record: track.envelope, ...(overfit ? { overfitting: overfit.envelope } : {}) })
496
+ .map(([name, e]) => [name, e?.data?.plain_reading ?? e?.error?.message ?? null]),
377
497
  );
378
498
  return asText({
379
- schema: "canli.audit.v1",
380
- note: "Each check is its validator's own result and receipt, side by side. The audit does not grade the strategy.",
499
+ schema: "canli.audit.v2",
500
+ headline: roundAll(headline),
501
+ fix_next: fix,
502
+ note: "One measured test, then each check's own result and receipt. No grade.",
503
+ ...(localNote ? { local: localNote } : {}),
381
504
  ...(shared ? { limits } : {}),
505
+ computed: {
506
+ // Exact: these are the inputs the deflated Sharpe check ran with, so they must reproduce it.
507
+ // Without variants they are the declared ones, which the check's derived inputs already show.
508
+ ...(search || session.fullEnvelope ? { trials: { used: trials, ...(declaredTrials !== undefined ? { declared: declaredTrials } : {}), ...(search ? { counted: search.effective_trials.used } : {}), cross_trial_sharpe_sd_annualized: crossSd } } : {}),
509
+ ...roundAll(session.fullEnvelope ? {
510
+ shape,
511
+ sharpe_interval: interval,
512
+ ...(haircut ? { haircut_sharpe_bonferroni: haircut.bonferroni.haircut_sharpe_annualized } : {}),
513
+ ...(minBtl ? { minimum_backtest_length_years: minBtl.years, years_available: years } : {}),
514
+ ...(search ? { search } : {}),
515
+ ...(decay ? { oos_decay: decay } : {}),
516
+ } : {
517
+ // Skew and kurtosis are in the deflated Sharpe check's derived inputs; not repeated here.
518
+ lag1_autocorrelation: shape.lag1_autocorrelation,
519
+ lo_adjusted_sharpe: shape.lo_adjusted_sharpe_annualized,
520
+ sharpe_95: interval.intervals["0.95"],
521
+ ...(haircut ? { haircut_sharpe: haircut.bonferroni.haircut_sharpe_annualized } : {}),
522
+ ...(minBtl ? { min_backtest_years: minBtl.years, years } : {}),
523
+ ...(search ? { variants: search.variants, mean_correlation: search.mean_pairwise_correlation, li_ji_trials: search.effective_trials.li_ji } : {}),
524
+ ...(decay ? { oos_sharpe_median: decay.oos_sharpe.median, oos_below_zero: decay.oos_sharpe.prob_below_zero, oos_on_is_slope: decay.degradation.slope } : {}),
525
+ }, 3),
526
+ },
382
527
  readings,
383
528
  checks,
384
529
  ...(input.returns_file || input.variants_file
@@ -387,7 +532,7 @@ export async function toolAuditBacktest(session, args) {
387
532
  ...(variantsRead ? { variants_file: input.variants_file, variants: variants[0].length, periods: variants.length, ...(variantsRead.skipped.length ? { skipped_variant_row_counter_columns: variantsRead.skipped } : {}) } : {}),
388
533
  } }
389
534
  : {}),
390
- ...(variants ? {} : { not_run: { overfitting: "Send variants or variants_file (every variant's returns) to add the overfitting check." } }),
535
+ ...(variants ? {} : { not_run: { overfitting: "Send variants for overfitting and SPA." } }),
391
536
  });
392
537
  }
393
538
 
@@ -418,7 +563,7 @@ export async function toolVerifyReceipt(session, args) {
418
563
  valid: false,
419
564
  checks: { well_formed: false, id_matches_content: false, signature_valid: false, key_published: false },
420
565
  key_id: data?.signature?.key_id ?? null,
421
- meaning: `The receipt is not well formed${missing.length ? `: it has no ${missing.join(", ")}` : ""}, so it cannot be checked. Send the whole receipt object as get_receipt or the validation result returned it, or send its id.`,
566
+ meaning: `The receipt is not well formed${missing.length ? `: it has no ${missing.join(", ")}` : ""}, so it cannot be checked. Send the whole receipt object as the validation result returned it, or send its id.`,
422
567
  });
423
568
  }
424
569
  return asText({
@@ -430,6 +575,8 @@ export async function toolVerifyReceipt(session, args) {
430
575
  meaning: result.valid
431
576
  ? "canlicapital.com signed this exact output for this exact input, computed by the source files whose hashes the receipt lists. It says nothing about how the input series was built."
432
577
  : "At least one check failed: do not treat this receipt as issued by canlicapital.com for this content.",
578
+ // With an id and include_receipt, the stored receipt itself: what get_receipt returned before 0.12.0.
579
+ ...(input.include_receipt && input.id !== undefined ? { receipt: data } : {}),
433
580
  });
434
581
  }
435
582
 
@@ -552,7 +699,11 @@ const WRITES_RECEIPT = { readOnlyHint: false, destructiveHint: false, openWorldH
552
699
  // Registers the enabled tools and returns what it registered, by name: { description, inputSchema,
553
700
  // outputSchema }, which the code-generation resources (canli://schemas/{tool}) serve.
554
701
  export function registerTools(server, session) {
555
- const enabled = new Set((session.toolsets ?? Object.keys(TOOLSETS)).flatMap((name) => TOOLSETS[name]));
702
+ const enabled = new Set((session.toolsets ?? DEFAULT_TOOLSETS).flatMap((name) => TOOLSETS[name]));
703
+ // get_key only helps a session with no key: with CANLI_KEY set, or a caller's or the shared key on
704
+ // the hosted endpoint, it would issue nothing, so it is not listed and costs no tokens.
705
+ const keyed = Boolean(session.envKey) || Boolean(session.hosted && session.hosted.keySource !== "none");
706
+ if (keyed) enabled.delete("get_key");
556
707
  const catalog = {};
557
708
  const register = (name, config, handler) => {
558
709
  if (!enabled.has(name)) return;
@@ -614,11 +765,6 @@ export function registerTools(server, session) {
614
765
  { title: "Audit a backtest", annotations: { title: "Audit a backtest", ...WRITES_RECEIPT }, description: TOOL_DESCRIPTIONS.audit_backtest, inputSchema: auditBacktestToolShape, outputSchema: auditOutput },
615
766
  (args) => toolAuditBacktest(session, args),
616
767
  );
617
- register(
618
- "get_receipt",
619
- { title: "Get a receipt", annotations: { title: "Get a receipt", ...READ_ONLY }, description: TOOL_DESCRIPTIONS.get_receipt, inputSchema: getReceiptInput, outputSchema: receiptOutput },
620
- (args) => toolGetReceipt(session, args),
621
- );
622
768
  register(
623
769
  "verify_receipt",
624
770
  { title: "Verify a receipt", annotations: { title: "Verify a receipt", ...READ_ONLY }, description: TOOL_DESCRIPTIONS.verify_receipt, inputSchema: verifyReceiptToolShape, outputSchema: verifyReceiptOutput },