overfitlab 0.1.0__tar.gz

overfitlab-0.1.0/.gitignore

@@ -0,0 +1,55 @@
+# Data: never commit raw or licensed data
+data/
+*.parquet
+*.feather
+*.h5
+*.itch
+*.pcap
+*.gz
+*.zst
+
+# Python
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+.ipynb_checkpoints/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+*.egg-info/
+dist/
+
+# C++ / Rust / CUDA builds
+build/
+cmake-build-*/
+target/
+*.o
+*.obj
+*.so
+*.dll
+*.pyd
+*.exe
+
+# Editors, OS, secrets
+.vscode/
+.idea/
+.DS_Store
+Thumbs.db
+.env
+
+# Local build spec (kept out of the public repo)
+SPEC.html
+
+# Run logs and tool output (added in M1)
+results/logs/
+results/sims/
+.hypothesis/
+.benchmarks/
+.coverage
+.coverage.*
+coverage.xml
+site/
+
+# The ingest package is code, not data; the data/ rule above would hide it.
+!study/src/overfitlab_study/data/

overfitlab-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 avmhdt
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

overfitlab-0.1.0/PKG-INFO

@@ -0,0 +1,178 @@
+Metadata-Version: 2.5
+Name: overfitlab
+Version: 0.1.0
+Summary: Statistics that separate a real trading signal from a lucky one: PSR, DSR, PBO, purged CV and data-snooping tests.
+Project-URL: Homepage, https://avmhdt.github.io/backtest-overfitting-lab/
+Project-URL: Repository, https://github.com/avmhdt/backtest-overfitting-lab
+Author: avmhdt
+License-Expression: MIT
+License-File: LICENSE
+Keywords: backtest,cross-validation,deflated-sharpe,multiple-testing,overfitting
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Financial and Insurance Industry
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Office/Business :: Financial :: Investment
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: numba>=0.61
+Requires-Dist: numpy>=2.0
+Requires-Dist: pyarrow>=17
+Requires-Dist: scipy>=1.13
+Requires-Dist: typer>=0.12
+Provides-Extra: polars
+Requires-Dist: polars>=1.10; extra == 'polars'
+Description-Content-Type: text/markdown
+
+# Backtest overfitting study
+
+> The statistics that tell a real trading signal from a lucky one, packaged as a library and turned on real research.
+
+`overfitlab` deflates a backtest for the search behind it. Run 1,000 strategies on pure noise and the best shows a median annual Sharpe ratio of 1.43. It looks publishable, and it's nothing. This project implements the tests that catch that, checks them against the papers and simulations, and applies them to public data and to the owner's own trial logs.
+
+The [ten-minute memo](https://avmhdt.github.io/backtest-overfitting-lab/memo/) gives the argument. The [docs site](https://avmhdt.github.io/backtest-overfitting-lab/) has the API, the theory and a [calculator](https://avmhdt.github.io/backtest-overfitting-lab/calculator/).
+
+## Quickstart
+
+```bash
+pip install overfitlab
+```
+
+```python
+import overfitlab as ol
+
+# A 2.5 annual Sharpe ratio over five years, the best of 100 trials.
+sr = 2.5 / 250**0.5
+result = ol.deflated_sharpe_ratio(
+    sr, 1250, n_trials=100, var_trials=0.5 / 250, skew=-3.0, kurtosis=10.0
+)
+print(round(result.dsr, 4))  # 0.9004
+```
+
+That Sharpe ratio misses the 0.95 bar once its 100 trials and fat left tail are counted. [Getting started](https://avmhdt.github.io/backtest-overfitting-lab/getting-started/) covers trial logs, cross-validation and the verdict gate.
+
+## The problem
+
+Researchers run many backtests and report the best. The best of many noisy trials has a high Sharpe ratio even when no trial has an edge, and a backtest report doesn't show how many trials stood behind it. The hypothesis here is that a small set of published corrections, applied with an honest trial count, separates luck from signal, and that each has failure modes worth measuring.
+
+## Data
+
+Raw and licensed data live in `data/`, which is git-ignored. Only download scripts, aggregates and small synthetic fixtures are committed.
+
+| Dataset | Cost | What it gives |
+|---|---|---|
+| Synthetic returns (GBM, block bootstrap) | Free | Ground truth for validating every test. |
+| [Stooq bulk daily data](https://stooq.com/db/h/) and Sharadar | Free, and the owner's bundle | Daily prices for 12 ETFs, 2006 to 2026. |
+| [Ken French data library](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html) and [AQR datasets](https://www.aqr.com/Insights/Datasets) | Free | 56 long factor histories for multiple-testing case studies. |
+
+AQR data: Century of Factor Premia, Time Series Momentum, Betting Against Beta and Quality Minus Junk, from [AQR Capital Management](https://www.aqr.com/Insights/Datasets), used with AQR's written permission. This repository publishes aggregate statistics and charts only, not AQR's files or series.
+| [Open Source Asset Pricing](https://www.openassetpricing.com) | Free | A real "many signals" universe of 212 predictors, release 2026.10. |
+
+## Method
+
+| Method | Question it answers |
+|---|---|
+| PSR and MinTRL | Is this Sharpe ratio real, given its length and return shape? |
+| DSR and MinBTL | Is it real, given how many trials produced it? |
+| PBO via CSCV | Does in-sample selection carry over out of sample? |
+| Reality Check, SPA, StepM | Does anything in this set beat the benchmark, and which? |
+| Harvey–Liu haircut | What is the Sharpe ratio worth after multiple testing? |
+| Purged k-fold and CPCV | How do I cross-validate overlapping labels without leaking? |
+
+A verdict gate requires the DSR, PBO, SPA, StepM and the haircut to pass together. Trial logs follow a frozen [contract](https://avmhdt.github.io/backtest-overfitting-lab/trial-log/), so N and V come from a record and not from memory.
+
+Each method is checked three ways: against the papers' printed examples, against closed-form power, and against independent implementations. [Validation](https://avmhdt.github.io/backtest-overfitting-lab/validation/) has the details.
+
+## Results
+
+| Experiment | Result |
+|---|---|
+| Best of 1,000 on noise, 100 seeds | Median best Sharpe 1.43. Naive PSR significant in 100 seeds, DSR in 0. Median PBO 0.50. |
+| 1,034 factory variants on noise | Median best Sharpe 1.13. DSR significant in 0 seeds. Median PBO 0.52. |
+| Leakage on noise, 20-day labels | Shuffled k-fold accuracy 0.703, purged 0.495. Gap 0.208. |
+| Factory on 12 ETFs, 1,034 trials | Sharpe 0.76 against a bar of 0.83. DSR 0.39, PBO 0.31, verdict FAIL. Holdout Sharpe 0.68. |
+| 56 factors from Ken French and AQR | 30 survive StepM. |
+| 212 OSAP predictors, full history | 173 at t > 1.96, 134 at t > 3, 149 after BHY, 96 after StepM. |
+| 208 OSAP predictors, after publication | 54 at t > 1.96, 24 at t > 3, 9 after BHY, 8 after StepM. |
+
+![The best of many noise strategies against its DSR](https://avmhdt.github.io/backtest-overfitting-lab/assets/figures/noise_best_vs_dsr.png)
+
+Detection takes a long record. With a true Sharpe ratio of 1 among 100 trials, the Reality Check, SPA and StepM need 20 years for 80% power, and the DSR doesn't reach it within the 20-year grid. See the [power curves](https://avmhdt.github.io/backtest-overfitting-lab/theory/power/).
+
+### The owner's own trials
+
+Only one of the six projects, `equity-factor-pipeline`, has a trial log so far.
+
+| Family | Own N | Sharpe | SR₀ | DSR | Verdict | DSR at N = 613 |
+|---|---:|---:|---:|---:|---|---:|
+| composite | 305 | 1.25 | 2.34 | 0.000 | FAIL | 0.000 |
+| investment | 32 | 1.00 | 0.70 | 0.952 | PASS | 0.429 |
+| low_risk | 32 | 0.80 | 0.45 | 0.957 | PASS | 0.734 |
+| momentum | 66 | 0.60 | 0.73 | 0.261 | FAIL | 0.036 |
+| quality | 44 | 0.98 | 0.63 | 0.951 | PASS | 0.678 |
+| replication | 86 | 0.81 | 0.60 | 0.840 | FAIL | 0.595 |
+| size | 8 | 0.09 | 0.05 | 0.588 | FAIL | 0.469 |
+| value | 40 | 0.68 | 0.51 | 0.802 | FAIL | 0.412 |
+
+Three families pass when each is judged alone. They were not found alone: the project ran 613 trials. At that count, no family reaches 0.95. `investment` fails outright, because the bar of 1.03 is above its Sharpe ratio of 1.00. Which count applies depends on whether the families were independent ideas or one search, so this table can't settle whether they're real. It does show that the verdict turns on that choice.
+
+## What failed
+
+- **DSR power (AC-W-02).** The DSR rejects less often than its reference: 0.036 against 0.168, 0.149 against 0.207, and 0.003 against 0.056 at the three cells that miss. V is estimated from the same trials as the planted winner, which raises its own bar. A version given the true V matches. This is kept as a finding, and no limit was changed.
+- **G-03c.** One of the 40 paper examples misses its tolerance by 5e-6. It's a documented exception.
+- **Studentised SPA check (AC-W-07).** Against `bsds.m`, 19 of 20 cases agree within 0.013, and StepM agrees in all 20. Case 2 has no positive mean, and `bsds.m` has no floor there, so it reports p = 1.
+- **Own-projects coverage (AC-H-02).** One of six projects has a trial log.
+- **Holm and BHY on the calculator (FR-34).** Deferred. The page gives the exact Bonferroni haircut.
+
+## Limitations
+
+- A trial log records what was run, not what was thought about, so N is a lower bound.
+- The tests here compare with zero. A long-only ETF strategy can beat zero on the equity premium alone, which is why the factory case passes SPA and StepM and still fails the DSR.
+- Fewer predictors survive after publication partly because those samples are shorter, so that drop isn't all decay.
+- PBO measures whether rankings persist, not whether money is made. An edge that died halfway through scores near 0. See [when PBO misleads](https://avmhdt.github.io/backtest-overfitting-lab/theory/pbo/).
+- Power curves assume independent strategies with one planted signal.
+
+## How to reproduce
+
+```bash
+uv sync
+uv run olab-study all
+```
+
+The full run takes about an hour and regenerates every table and figure from the data in `data/`. Without that data, the real-data steps are skipped with a message. `--quick` runs a reduced version in about two minutes. Each run adds rows to `results/experiment_log.csv`.
+
+## Requirements
+
+- [x] Purged k-fold with embargo and combinatorial purged CV (CPCV), handling overlapping multi-day labels. (FR-08, FR-09)
+- [x] Probabilistic and Deflated Sharpe Ratio (Bailey and López de Prado), adjusting for skew, kurtosis and the number and variance of trials. (FR-03, FR-05)
+- [x] Probability of Backtest Overfitting via CSCV (Bailey, Borwein, López de Prado and Zhu). (FR-10)
+- [x] Data-snooping tests: White's Reality Check, Hansen's SPA, and the Romano–Wolf stepdown. (FR-12, FR-13, FR-14)
+- [x] Minimum backtest length and a Sharpe ratio haircut calculator (Harvey and Liu). (FR-06, FR-07, FR-33)
+- [x] A strategy factory that generates thousands of parameter variants (crossovers, breakouts, mean reversion) to test against. (FR-15)
+- [x] A synthetic data generator with known ground truth: pure noise, and noise with a planted signal of chosen strength. (FR-16)
+
+## Success criteria
+
+### Correct and reproducible
+
+- [x] On pure noise, the best of 1,000 strategies shows a high naive Sharpe while the DSR is not significant and PBO is near or above 0.5. (AC-W-01)
+- [ ] On planted-signal data, each test detects the signal at roughly its theoretical power. (AC-W-02: met for PSR, RC, SPA and StepM, missed for the DSR)
+- [x] Implementations reproduce the numerical examples in the original papers to three decimals. (AC-W-03, with G-03c as a documented exception)
+- [x] A simulation shows the accuracy gap between standard k-fold and purged k-fold on overlapping labels. (AC-W-04)
+
+### Depth
+
+- [x] Power curves: how large a true Sharpe, and how long a backtest, each test needs before it detects a signal. (AC-H-01)
+- [ ] A table of naive vs deflated Sharpe for your own projects ([equity-factor-pipeline](https://github.com/avmhdt/equity-factor-pipeline), [llm-text-alpha](https://github.com/avmhdt/llm-text-alpha), [stat-arb-pairs](https://github.com/avmhdt/stat-arb-pairs)). (AC-H-02: one of six projects so far)
+- [ ] Packaged as a pip-installable library with docs and an interactive Sharpe-haircut page. (AC-H-03: docs and calculator are live, and the install from PyPI comes with the first release)
+- [x] A short paper-style writeup a PM could read in ten minutes. (AC-H-04)
+
+## Questions this project must answer
+
+- [Why does the DSR use the variance of trial Sharpes, not just the count?](https://avmhdt.github.io/backtest-overfitting-lab/theory/psr-dsr/)
+- [When is PBO misleading?](https://avmhdt.github.io/backtest-overfitting-lab/theory/pbo/)
+- [How do you count trials honestly when you iterated by hand?](https://avmhdt.github.io/backtest-overfitting-lab/theory/trial-counting/)

overfitlab-0.1.0/README.md

@@ -0,0 +1,148 @@
+# Backtest overfitting study
+
+> The statistics that tell a real trading signal from a lucky one, packaged as a library and turned on real research.
+
+`overfitlab` deflates a backtest for the search behind it. Run 1,000 strategies on pure noise and the best shows a median annual Sharpe ratio of 1.43. It looks publishable, and it's nothing. This project implements the tests that catch that, checks them against the papers and simulations, and applies them to public data and to the owner's own trial logs.
+
+The [ten-minute memo](https://avmhdt.github.io/backtest-overfitting-lab/memo/) gives the argument. The [docs site](https://avmhdt.github.io/backtest-overfitting-lab/) has the API, the theory and a [calculator](https://avmhdt.github.io/backtest-overfitting-lab/calculator/).
+
+## Quickstart
+
+```bash
+pip install overfitlab
+```
+
+```python
+import overfitlab as ol
+
+# A 2.5 annual Sharpe ratio over five years, the best of 100 trials.
+sr = 2.5 / 250**0.5
+result = ol.deflated_sharpe_ratio(
+    sr, 1250, n_trials=100, var_trials=0.5 / 250, skew=-3.0, kurtosis=10.0
+)
+print(round(result.dsr, 4))  # 0.9004
+```
+
+That Sharpe ratio misses the 0.95 bar once its 100 trials and fat left tail are counted. [Getting started](https://avmhdt.github.io/backtest-overfitting-lab/getting-started/) covers trial logs, cross-validation and the verdict gate.
+
+## The problem
+
+Researchers run many backtests and report the best. The best of many noisy trials has a high Sharpe ratio even when no trial has an edge, and a backtest report doesn't show how many trials stood behind it. The hypothesis here is that a small set of published corrections, applied with an honest trial count, separates luck from signal, and that each has failure modes worth measuring.
+
+## Data
+
+Raw and licensed data live in `data/`, which is git-ignored. Only download scripts, aggregates and small synthetic fixtures are committed.
+
+| Dataset | Cost | What it gives |
+|---|---|---|
+| Synthetic returns (GBM, block bootstrap) | Free | Ground truth for validating every test. |
+| [Stooq bulk daily data](https://stooq.com/db/h/) and Sharadar | Free, and the owner's bundle | Daily prices for 12 ETFs, 2006 to 2026. |
+| [Ken French data library](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html) and [AQR datasets](https://www.aqr.com/Insights/Datasets) | Free | 56 long factor histories for multiple-testing case studies. |
+
+AQR data: Century of Factor Premia, Time Series Momentum, Betting Against Beta and Quality Minus Junk, from [AQR Capital Management](https://www.aqr.com/Insights/Datasets), used with AQR's written permission. This repository publishes aggregate statistics and charts only, not AQR's files or series.
+| [Open Source Asset Pricing](https://www.openassetpricing.com) | Free | A real "many signals" universe of 212 predictors, release 2026.10. |
+
+## Method
+
+| Method | Question it answers |
+|---|---|
+| PSR and MinTRL | Is this Sharpe ratio real, given its length and return shape? |
+| DSR and MinBTL | Is it real, given how many trials produced it? |
+| PBO via CSCV | Does in-sample selection carry over out of sample? |
+| Reality Check, SPA, StepM | Does anything in this set beat the benchmark, and which? |
+| Harvey–Liu haircut | What is the Sharpe ratio worth after multiple testing? |
+| Purged k-fold and CPCV | How do I cross-validate overlapping labels without leaking? |
+
+A verdict gate requires the DSR, PBO, SPA, StepM and the haircut to pass together. Trial logs follow a frozen [contract](https://avmhdt.github.io/backtest-overfitting-lab/trial-log/), so N and V come from a record and not from memory.
+
+Each method is checked three ways: against the papers' printed examples, against closed-form power, and against independent implementations. [Validation](https://avmhdt.github.io/backtest-overfitting-lab/validation/) has the details.
+
+## Results
+
+| Experiment | Result |
+|---|---|
+| Best of 1,000 on noise, 100 seeds | Median best Sharpe 1.43. Naive PSR significant in 100 seeds, DSR in 0. Median PBO 0.50. |
+| 1,034 factory variants on noise | Median best Sharpe 1.13. DSR significant in 0 seeds. Median PBO 0.52. |
+| Leakage on noise, 20-day labels | Shuffled k-fold accuracy 0.703, purged 0.495. Gap 0.208. |
+| Factory on 12 ETFs, 1,034 trials | Sharpe 0.76 against a bar of 0.83. DSR 0.39, PBO 0.31, verdict FAIL. Holdout Sharpe 0.68. |
+| 56 factors from Ken French and AQR | 30 survive StepM. |
+| 212 OSAP predictors, full history | 173 at t > 1.96, 134 at t > 3, 149 after BHY, 96 after StepM. |
+| 208 OSAP predictors, after publication | 54 at t > 1.96, 24 at t > 3, 9 after BHY, 8 after StepM. |
+
+![The best of many noise strategies against its DSR](https://avmhdt.github.io/backtest-overfitting-lab/assets/figures/noise_best_vs_dsr.png)
+
+Detection takes a long record. With a true Sharpe ratio of 1 among 100 trials, the Reality Check, SPA and StepM need 20 years for 80% power, and the DSR doesn't reach it within the 20-year grid. See the [power curves](https://avmhdt.github.io/backtest-overfitting-lab/theory/power/).
+
+### The owner's own trials
+
+Only one of the six projects, `equity-factor-pipeline`, has a trial log so far.
+
+| Family | Own N | Sharpe | SR₀ | DSR | Verdict | DSR at N = 613 |
+|---|---:|---:|---:|---:|---|---:|
+| composite | 305 | 1.25 | 2.34 | 0.000 | FAIL | 0.000 |
+| investment | 32 | 1.00 | 0.70 | 0.952 | PASS | 0.429 |
+| low_risk | 32 | 0.80 | 0.45 | 0.957 | PASS | 0.734 |
+| momentum | 66 | 0.60 | 0.73 | 0.261 | FAIL | 0.036 |
+| quality | 44 | 0.98 | 0.63 | 0.951 | PASS | 0.678 |
+| replication | 86 | 0.81 | 0.60 | 0.840 | FAIL | 0.595 |
+| size | 8 | 0.09 | 0.05 | 0.588 | FAIL | 0.469 |
+| value | 40 | 0.68 | 0.51 | 0.802 | FAIL | 0.412 |
+
+Three families pass when each is judged alone. They were not found alone: the project ran 613 trials. At that count, no family reaches 0.95. `investment` fails outright, because the bar of 1.03 is above its Sharpe ratio of 1.00. Which count applies depends on whether the families were independent ideas or one search, so this table can't settle whether they're real. It does show that the verdict turns on that choice.
+
+## What failed
+
+- **DSR power (AC-W-02).** The DSR rejects less often than its reference: 0.036 against 0.168, 0.149 against 0.207, and 0.003 against 0.056 at the three cells that miss. V is estimated from the same trials as the planted winner, which raises its own bar. A version given the true V matches. This is kept as a finding, and no limit was changed.
+- **G-03c.** One of the 40 paper examples misses its tolerance by 5e-6. It's a documented exception.
+- **Studentised SPA check (AC-W-07).** Against `bsds.m`, 19 of 20 cases agree within 0.013, and StepM agrees in all 20. Case 2 has no positive mean, and `bsds.m` has no floor there, so it reports p = 1.
+- **Own-projects coverage (AC-H-02).** One of six projects has a trial log.
+- **Holm and BHY on the calculator (FR-34).** Deferred. The page gives the exact Bonferroni haircut.
+
+## Limitations
+
+- A trial log records what was run, not what was thought about, so N is a lower bound.
+- The tests here compare with zero. A long-only ETF strategy can beat zero on the equity premium alone, which is why the factory case passes SPA and StepM and still fails the DSR.
+- Fewer predictors survive after publication partly because those samples are shorter, so that drop isn't all decay.
+- PBO measures whether rankings persist, not whether money is made. An edge that died halfway through scores near 0. See [when PBO misleads](https://avmhdt.github.io/backtest-overfitting-lab/theory/pbo/).
+- Power curves assume independent strategies with one planted signal.
+
+## How to reproduce
+
+```bash
+uv sync
+uv run olab-study all
+```
+
+The full run takes about an hour and regenerates every table and figure from the data in `data/`. Without that data, the real-data steps are skipped with a message. `--quick` runs a reduced version in about two minutes. Each run adds rows to `results/experiment_log.csv`.
+
+## Requirements
+
+- [x] Purged k-fold with embargo and combinatorial purged CV (CPCV), handling overlapping multi-day labels. (FR-08, FR-09)
+- [x] Probabilistic and Deflated Sharpe Ratio (Bailey and López de Prado), adjusting for skew, kurtosis and the number and variance of trials. (FR-03, FR-05)
+- [x] Probability of Backtest Overfitting via CSCV (Bailey, Borwein, López de Prado and Zhu). (FR-10)
+- [x] Data-snooping tests: White's Reality Check, Hansen's SPA, and the Romano–Wolf stepdown. (FR-12, FR-13, FR-14)
+- [x] Minimum backtest length and a Sharpe ratio haircut calculator (Harvey and Liu). (FR-06, FR-07, FR-33)
+- [x] A strategy factory that generates thousands of parameter variants (crossovers, breakouts, mean reversion) to test against. (FR-15)
+- [x] A synthetic data generator with known ground truth: pure noise, and noise with a planted signal of chosen strength. (FR-16)
+
+## Success criteria
+
+### Correct and reproducible
+
+- [x] On pure noise, the best of 1,000 strategies shows a high naive Sharpe while the DSR is not significant and PBO is near or above 0.5. (AC-W-01)
+- [ ] On planted-signal data, each test detects the signal at roughly its theoretical power. (AC-W-02: met for PSR, RC, SPA and StepM, missed for the DSR)
+- [x] Implementations reproduce the numerical examples in the original papers to three decimals. (AC-W-03, with G-03c as a documented exception)
+- [x] A simulation shows the accuracy gap between standard k-fold and purged k-fold on overlapping labels. (AC-W-04)
+
+### Depth
+
+- [x] Power curves: how large a true Sharpe, and how long a backtest, each test needs before it detects a signal. (AC-H-01)
+- [ ] A table of naive vs deflated Sharpe for your own projects ([equity-factor-pipeline](https://github.com/avmhdt/equity-factor-pipeline), [llm-text-alpha](https://github.com/avmhdt/llm-text-alpha), [stat-arb-pairs](https://github.com/avmhdt/stat-arb-pairs)). (AC-H-02: one of six projects so far)
+- [ ] Packaged as a pip-installable library with docs and an interactive Sharpe-haircut page. (AC-H-03: docs and calculator are live, and the install from PyPI comes with the first release)
+- [x] A short paper-style writeup a PM could read in ten minutes. (AC-H-04)
+
+## Questions this project must answer
+
+- [Why does the DSR use the variance of trial Sharpes, not just the count?](https://avmhdt.github.io/backtest-overfitting-lab/theory/psr-dsr/)
+- [When is PBO misleading?](https://avmhdt.github.io/backtest-overfitting-lab/theory/pbo/)
+- [How do you count trials honestly when you iterated by hand?](https://avmhdt.github.io/backtest-overfitting-lab/theory/trial-counting/)

overfitlab-0.1.0/pyproject.toml

@@ -0,0 +1,130 @@
+[project]
+name = "overfitlab"
+version = "0.1.0"
+description = "Statistics that separate a real trading signal from a lucky one: PSR, DSR, PBO, purged CV and data-snooping tests."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.12"
+authors = [{ name = "avmhdt" }]
+keywords = ["backtest", "overfitting", "deflated-sharpe", "cross-validation", "multiple-testing"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Financial and Insurance Industry",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: Microsoft :: Windows",
+    "Operating System :: POSIX :: Linux",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Office/Business :: Financial :: Investment",
+    "Typing :: Typed",
+]
+dependencies = [
+    "numpy>=2.0",
+    "scipy>=1.13",
+    "numba>=0.61",
+    "pyarrow>=17",
+    "typer>=0.12",
+]
+
+[project.optional-dependencies]
+polars = ["polars>=1.10"]
+
+[project.scripts]
+overfitlab = "overfitlab.cli:app"
+
+[project.urls]
+Homepage = "https://avmhdt.github.io/backtest-overfitting-lab/"
+Repository = "https://github.com/avmhdt/backtest-overfitting-lab"
+
+[build-system]
+requires = ["hatchling>=1.25"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/overfitlab"]
+
+[tool.hatch.build.targets.sdist]
+include = ["src/overfitlab", "README.md", "LICENSE", "pyproject.toml"]
+
+[tool.uv.workspace]
+members = ["study"]
+
+[tool.uv.sources]
+overfitlab-study = { workspace = true }
+
+[dependency-groups]
+dev = [
+    "overfitlab-study",
+    "polars>=1.10",
+    "ruff>=0.7",
+    "mypy>=1.13",
+    "scipy-stubs>=1.13",
+    "pytest>=8.3",
+    "pytest-cov>=5",
+    "pytest-benchmark>=4",
+    "hypothesis>=6.112",
+    "pre-commit>=4.0",
+    "mkdocs-material>=9.5",
+    "mkdocstrings[python]>=0.26",
+    "types-pyyaml>=6.0.12.20261030",
+    "types-openpyxl>=3.1.5.20261017",
+]
+
+[tool.ruff]
+line-length = 100
+target-version = "py312"
+src = ["src", "study/src"]
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "B", "UP", "NPY", "PLE", "PLW", "PLC", "D"]
+ignore = ["D105", "D107"]
+
+[tool.ruff.lint.flake8-bugbear]
+extend-immutable-calls = ["overfitlab.verdict.VerdictConfig"]
+
+[tool.ruff.lint.pydocstyle]
+convention = "numpy"
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = ["D"]
+"study/**" = ["D"]
+"scripts/**" = ["D"]
+# Typer needs its option metadata in argument defaults.
+"**/cli.py" = ["B008"]
+# PyArrow is imported on use, so that import overfitlab stays light (NFR-11).
+"src/overfitlab/triallog.py" = ["PLC0415"]
+
+[tool.ruff.format]
+quote-style = "double"
+line-ending = "lf"
+
+[tool.mypy]
+python_version = "3.12"
+strict = true
+files = ["src/overfitlab", "study/src"]
+mypy_path = "src:study/src"
+explicit_package_bases = true
+
+[[tool.mypy.overrides]]
+module = ["numba.*", "arch.*", "sklearn.*", "pyarrow.*", "plotly.*"]
+ignore_missing_imports = true
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "--strict-markers --import-mode=importlib"
+markers = [
+    "golden: numbers transcribed from the original papers",
+    "xcheck: cross-checks against the independent arch package",
+    "slow: quick versions of the study simulations",
+    "data: needs real raw data files (local only)",
+    "bench: performance benchmarks (local only)",
+]
+
+[tool.coverage.run]
+source_pkgs = ["overfitlab", "overfitlab_study"]
+branch = false
+
+[tool.coverage.report]
+exclude_also = ["if TYPE_CHECKING:", "raise NotImplementedError"]

overfitlab-0.1.0/src/overfitlab/__init__.py

@@ -0,0 +1,119 @@
+"""overfitlab: statistics that separate a real trading signal from a lucky one."""
+
+from overfitlab._normal import EULER_GAMMA
+from overfitlab.bootstrap import optimal_block_length, stationary_bootstrap_indices
+from overfitlab.cv import CombinatorialPurgedCV, PurgedKFold, assemble_paths
+from overfitlab.deflated import (
+    deflated_sharpe_ratio,
+    dsr_from_returns,
+    expected_max_sharpe,
+    min_backtest_length,
+)
+from overfitlab.factory import StrategySpec, run_strategies, strategy_grid
+from overfitlab.haircut import haircut_sharpe, sharpe_to_tstat
+from overfitlab.pbo import PBOResult, pbo_cscv
+from overfitlab.results import DSRResult, HaircutResult, ReturnMoments
+from overfitlab.sharpe import (
+    annualize_sharpe,
+    min_track_record_length,
+    probabilistic_sharpe_ratio,
+    psr_from_returns,
+    return_moments,
+    sharpe_ratio,
+    sharpe_std,
+)
+from overfitlab.snooping import (
+    SnoopResult,
+    SPAResult,
+    StepMResult,
+    reality_check,
+    romano_wolf_stepdown,
+    spa_test,
+)
+from overfitlab.synth import (
+    SynthResult,
+    block_bootstrap_returns,
+    gaussian_noise,
+    gbm_prices,
+    planted_signal_matrix,
+    planted_signal_prices,
+    student_t_noise,
+)
+from overfitlab.triallog import (
+    TRIAL_LOG_COLUMNS,
+    TRIAL_LOG_VERSION,
+    TrialLog,
+    TrialLogError,
+    TrialLogger,
+    TrialLogIssue,
+    TrialLogReport,
+    TrialSharpes,
+    add_manual_trial,
+    canonical_params_json,
+    evaluate_trial_log,
+    read_trial_log,
+    validate_trial_log,
+)
+from overfitlab.verdict import VerdictConfig, VerdictReport, evaluate
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "EULER_GAMMA",
+    "CombinatorialPurgedCV",
+    "DSRResult",
+    "HaircutResult",
+    "PBOResult",
+    "PurgedKFold",
+    "ReturnMoments",
+    "SPAResult",
+    "SnoopResult",
+    "StepMResult",
+    "StrategySpec",
+    "SynthResult",
+    "TRIAL_LOG_COLUMNS",
+    "TRIAL_LOG_VERSION",
+    "TrialLog",
+    "TrialLogError",
+    "TrialLogIssue",
+    "TrialLogReport",
+    "TrialLogger",
+    "TrialSharpes",
+    "VerdictConfig",
+    "VerdictReport",
+    "__version__",
+    "add_manual_trial",
+    "annualize_sharpe",
+    "assemble_paths",
+    "block_bootstrap_returns",
+    "canonical_params_json",
+    "deflated_sharpe_ratio",
+    "dsr_from_returns",
+    "evaluate",
+    "evaluate_trial_log",
+    "expected_max_sharpe",
+    "gaussian_noise",
+    "gbm_prices",
+    "haircut_sharpe",
+    "min_backtest_length",
+    "min_track_record_length",
+    "optimal_block_length",
+    "pbo_cscv",
+    "planted_signal_matrix",
+    "planted_signal_prices",
+    "probabilistic_sharpe_ratio",
+    "psr_from_returns",
+    "read_trial_log",
+    "reality_check",
+    "return_moments",
+    "romano_wolf_stepdown",
+    "run_strategies",
+    "sharpe_ratio",
+    "sharpe_std",
+    "sharpe_to_tstat",
+    "spa_test",
+    "stationary_bootstrap_indices",
+    "strategy_grid",
+    "student_t_noise",
+    "validate_trial_log",
+]