quant-research-dataset 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (53) hide show
  1. quant_research_dataset-0.1.0/.gitignore +24 -0
  2. quant_research_dataset-0.1.0/CHANGELOG.md +23 -0
  3. quant_research_dataset-0.1.0/LICENSE +24 -0
  4. quant_research_dataset-0.1.0/PKG-INFO +254 -0
  5. quant_research_dataset-0.1.0/README.md +214 -0
  6. quant_research_dataset-0.1.0/hf/README.md +447 -0
  7. quant_research_dataset-0.1.0/pyproject.toml +83 -0
  8. quant_research_dataset-0.1.0/src/quant_research_dataset/__init__.py +67 -0
  9. quant_research_dataset-0.1.0/src/quant_research_dataset/_version.py +1 -0
  10. quant_research_dataset-0.1.0/src/quant_research_dataset/adapters/__init__.py +12 -0
  11. quant_research_dataset-0.1.0/src/quant_research_dataset/adapters/base.py +63 -0
  12. quant_research_dataset-0.1.0/src/quant_research_dataset/adapters/us_fundamentals.py +60 -0
  13. quant_research_dataset-0.1.0/src/quant_research_dataset/adapters/us_options.py +35 -0
  14. quant_research_dataset-0.1.0/src/quant_research_dataset/adapters/us_stocks.py +48 -0
  15. quant_research_dataset-0.1.0/src/quant_research_dataset/analytics.py +73 -0
  16. quant_research_dataset-0.1.0/src/quant_research_dataset/build.py +316 -0
  17. quant_research_dataset-0.1.0/src/quant_research_dataset/catalog.py +176 -0
  18. quant_research_dataset-0.1.0/src/quant_research_dataset/cli.py +169 -0
  19. quant_research_dataset-0.1.0/src/quant_research_dataset/config.py +107 -0
  20. quant_research_dataset-0.1.0/src/quant_research_dataset/errors.py +26 -0
  21. quant_research_dataset-0.1.0/src/quant_research_dataset/links.py +152 -0
  22. quant_research_dataset-0.1.0/src/quant_research_dataset/manifest.py +202 -0
  23. quant_research_dataset-0.1.0/src/quant_research_dataset/markets/__init__.py +50 -0
  24. quant_research_dataset-0.1.0/src/quant_research_dataset/markets/cn_hk.py +73 -0
  25. quant_research_dataset-0.1.0/src/quant_research_dataset/markets/us.py +236 -0
  26. quant_research_dataset-0.1.0/src/quant_research_dataset/publish.py +147 -0
  27. quant_research_dataset-0.1.0/src/quant_research_dataset/py.typed +0 -0
  28. quant_research_dataset-0.1.0/src/quant_research_dataset/query.py +349 -0
  29. quant_research_dataset-0.1.0/src/quant_research_dataset/registry/cn_hk/equity/ah_pairs.yaml +21 -0
  30. quant_research_dataset-0.1.0/src/quant_research_dataset/registry/cn_hk/equity/daily_bars.yaml +30 -0
  31. quant_research_dataset-0.1.0/src/quant_research_dataset/registry/cn_hk/fx/daily.yaml +25 -0
  32. quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/equity/stock_daily.yaml +64 -0
  33. quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/etf/bars_1min.yaml +27 -0
  34. quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/etf/bars_5min.yaml +32 -0
  35. quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/etf/daily_bars.yaml +31 -0
  36. quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/fundamentals/annual.yaml +90 -0
  37. quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/fundamentals/quarterly.yaml +93 -0
  38. quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/index/spx_daily.yaml +25 -0
  39. quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/options/spx_option_prices.yaml +54 -0
  40. quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/options/spx_vol_surface.yaml +29 -0
  41. quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/rates/zero_curve.yaml +23 -0
  42. quant_research_dataset-0.1.0/src/quant_research_dataset/registry.py +335 -0
  43. quant_research_dataset-0.1.0/src/quant_research_dataset/sql.py +54 -0
  44. quant_research_dataset-0.1.0/src/quant_research_dataset/storage.py +139 -0
  45. quant_research_dataset-0.1.0/tests/__init__.py +0 -0
  46. quant_research_dataset-0.1.0/tests/conftest.py +51 -0
  47. quant_research_dataset-0.1.0/tests/fakes.py +264 -0
  48. quant_research_dataset-0.1.0/tests/test_analytics.py +54 -0
  49. quant_research_dataset-0.1.0/tests/test_build.py +188 -0
  50. quant_research_dataset-0.1.0/tests/test_load.py +149 -0
  51. quant_research_dataset-0.1.0/tests/test_markets.py +144 -0
  52. quant_research_dataset-0.1.0/tests/test_registry.py +118 -0
  53. quant_research_dataset-0.1.0/tests/test_storage_publish.py +134 -0
@@ -0,0 +1,24 @@
1
+ # Python
2
+ __pycache__/
3
+ *.py[cod]
4
+ *.egg-info/
5
+ .venv/
6
+ build/
7
+ dist/
8
+ .pytest_cache/
9
+ .mypy_cache/
10
+ .ruff_cache/
11
+ .coverage
12
+ htmlcov/
13
+ .ipynb_checkpoints/
14
+ site/
15
+
16
+ # Data never goes into git: raw inputs, prepared partitions, caches
17
+ source_data_copy/
18
+ prepared/
19
+ .qrd_cache/
20
+ *.parquet
21
+ *.csv.gz
22
+
23
+ # Secrets
24
+ .env
@@ -0,0 +1,23 @@
1
+ # Changelog
2
+
3
+ ## 0.1.0 (unreleased)
4
+
5
+ First release.
6
+
7
+ * Generic API: `load`, `list_datasets`, `describe`, `download`, `files_for`.
8
+ * YAML dataset registry with typed schemas, units and descriptions; generated
9
+ data dictionary and Hugging Face dataset card.
10
+ * Storage backends: Hugging Face Hub (gated or private repos) and local folders.
11
+ * Build pipeline from raw vendor files to partitioned Parquet with a
12
+ checksummed manifest; `qrd verify` and `qrd publish`.
13
+ * 13 datasets, named by content (source kept as metadata): US stock daily,
14
+ US fundamentals quarterly/annual, SPX option prices and
15
+ vol surface, SPX index, USD zero curve, US ETF bars (1d, 5m, 1m), China A/H
16
+ daily bars, A/H pairs, CNY/HKD FX.
17
+ * Stock-company links are applied at build time: stock rows carry `gvkey` and
18
+ fundamentals carry `permno` (primary share class).
19
+ * US helpers: common-stock screen, point-in-time fundamentals,
20
+ `merge_fundamentals`, option chains with spot, rates and moneyness.
21
+ * China/HK helpers: A/H premium.
22
+ * Research helpers: quantile portfolio sorts, performance summary.
23
+ * SQL access through DuckDB.
@@ -0,0 +1,24 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Hardy-Z
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
22
+
23
+ This license covers the source code only. The datasets the code loads are
24
+ licensed separately by their vendors; see each dataset's license field.
@@ -0,0 +1,254 @@
1
+ Metadata-Version: 2.5
2
+ Name: quant-research-dataset
3
+ Version: 0.1.0
4
+ Summary: One API for research-grade market data across markets: equities, options, ETFs, fundamentals and rates, served from a versioned Hugging Face dataset.
5
+ Project-URL: Homepage, https://github.com/Hardy-Z/quant-research-dataset
6
+ Project-URL: Issues, https://github.com/Hardy-Z/quant-research-dataset/issues
7
+ Author: Hardy-Z
8
+ License-Expression: MIT
9
+ License-File: LICENSE
10
+ Keywords: backtesting,compustat,crsp,dataset,finance,hugging-face,optionmetrics,quant
11
+ Classifier: Development Status :: 3 - Alpha
12
+ Classifier: Intended Audience :: Financial and Insurance Industry
13
+ Classifier: Intended Audience :: Science/Research
14
+ Classifier: Operating System :: OS Independent
15
+ Classifier: Programming Language :: Python :: 3
16
+ Classifier: Programming Language :: Python :: 3.10
17
+ Classifier: Programming Language :: Python :: 3.11
18
+ Classifier: Programming Language :: Python :: 3.12
19
+ Classifier: Programming Language :: Python :: 3.13
20
+ Classifier: Topic :: Office/Business :: Financial :: Investment
21
+ Classifier: Typing :: Typed
22
+ Requires-Python: >=3.10
23
+ Requires-Dist: huggingface-hub>=0.23
24
+ Requires-Dist: pandas>=2.0
25
+ Requires-Dist: pyarrow>=14
26
+ Requires-Dist: pyyaml>=6
27
+ Provides-Extra: dev
28
+ Requires-Dist: duckdb>=1.0; extra == 'dev'
29
+ Requires-Dist: mypy>=1.10; extra == 'dev'
30
+ Requires-Dist: pandas-stubs; extra == 'dev'
31
+ Requires-Dist: pytest-cov>=5; extra == 'dev'
32
+ Requires-Dist: pytest>=8; extra == 'dev'
33
+ Requires-Dist: ruff>=0.6; extra == 'dev'
34
+ Requires-Dist: types-pyyaml; extra == 'dev'
35
+ Provides-Extra: docs
36
+ Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
37
+ Provides-Extra: duckdb
38
+ Requires-Dist: duckdb>=1.0; extra == 'duckdb'
39
+ Description-Content-Type: text/markdown
40
+
41
+ # quant-research-dataset
42
+
43
+ [![CI](https://github.com/Hardy-Z/quant-research-dataset/actions/workflows/ci.yml/badge.svg)](https://github.com/Hardy-Z/quant-research-dataset/actions/workflows/ci.yml)
44
+ [![PyPI](https://img.shields.io/pypi/v/quant-research-dataset)](https://pypi.org/project/quant-research-dataset/)
45
+ [![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20dataset-Hardy0102%2Fquant--research--dataset-yellow)](https://huggingface.co/datasets/Hardy0102/quant-research-dataset)
46
+ ![Python](https://img.shields.io/badge/python-3.10%E2%80%933.13-blue)
47
+ ![License](https://img.shields.io/badge/code-MIT-green)
48
+
49
+ **Code:** [GitHub](https://github.com/Hardy-Z/quant-research-dataset) ·
50
+ **Package:** [PyPI](https://pypi.org/project/quant-research-dataset/) ·
51
+ **Data:** [Hugging Face](https://huggingface.co/datasets/Hardy0102/quant-research-dataset) (gated) ·
52
+ **Docs:** [data dictionary](docs/datasets.md)
53
+
54
+ **One Python API for research-grade market data across markets.** Equities,
55
+ fundamentals, index options, rates and ETFs live in a versioned, partitioned
56
+ Parquet dataset on the Hugging Face Hub. The package downloads only the
57
+ partitions a query touches, pushes filters down into the Parquet scan and
58
+ returns clean, documented `pandas` DataFrames.
59
+
60
+ ```python
61
+ import quant_research_dataset as qrd
62
+
63
+ # Every US common stock's daily return in 2024 (one ~90 MB partition, cached after the first call)
64
+ stocks = qrd.load(
65
+ "us/equity/stock_daily",
66
+ start="2024-01-01",
67
+ end="2024-12-31",
68
+ columns=["date", "permno", "gvkey", "ret", "mktcap"],
69
+ common_only=True,
70
+ )
71
+
72
+ # Point-in-time fundamentals, joined without look-ahead bias
73
+ fund = qrd.get_fundamentals("quarterly", columns=["gvkey", "datadate", "atq", "ceqq"])
74
+ panel = qrd.merge_fundamentals(stocks, fund)
75
+
76
+ # SPX options for a day, with mid price, days to expiry, spot and a matched zero rate
77
+ chain = qrd.get_spx_options("2024-03-15", with_market=True)
78
+ ```
79
+
80
+ ## Why
81
+
82
+ Academic-quality data (CRSP, Compustat, OptionMetrics) is powerful but awkward:
83
+ wide vendor tables, cryptic codes, strikes stored ×1000, link tables with date
84
+ ranges, fiscal dates that leak the future. Every project re-implements the same
85
+ cleaning. This library does it once, in tested code, behind a small API:
86
+
87
+ | Problem | What the library does |
88
+ |---|---|
89
+ | Multi-GB files, slow to load | Year/month partitions + a manifest, so `load(start=..., end=...)` fetches only what it needs |
90
+ | Different column names per vendor | Every dataset has a typed schema, snake_case names, units and descriptions in one YAML file |
91
+ | Look-ahead bias in fundamentals | `point_in_time=True` adds `available_date` (report date, or a conservative lag) |
92
+ | Joining stocks to company fundamentals | Links are applied at build time: stock rows carry `gvkey`, fundamentals carry `permno`; `merge_fundamentals()` joins point in time |
93
+ | "Common stocks only" screens | `common_only=True` applies the CIZ-flag equivalent of `shrcd in (10, 11)` |
94
+ | Option chains need context | `mid`, `spread`, `dte`, `spot`, `moneyness` and an interpolated zero rate |
95
+ | US-only tools | Market-neutral design: `market/asset_class/dataset` ids, per-market adapters |
96
+
97
+ ## Datasets
98
+
99
+ | Id | Content | Source | Coverage |
100
+ |---|---|---|---|
101
+ | `us/equity/stock_daily` | Daily prices, returns, volume and market cap of all US-listed stocks, incl. delisted, with linked `gvkey` | CRSP | 2015–2025, 23.1M rows |
102
+ | `us/fundamentals/quarterly` | Quarterly financial statements, 60+ items, with linked `permno` | Compustat | 2000–2026, 1.05M rows |
103
+ | `us/fundamentals/annual` | Annual financial statements, 60+ items, with linked `permno` | Compustat | 2000–2026, 255K rows |
104
+ | `us/options/spx_option_prices` | S&P 500 index option quotes, implied vol and greeks | OptionMetrics | 2015–2025, 34.3M rows |
105
+ | `us/options/spx_vol_surface` | Standardized S&P 500 implied volatility surface | OptionMetrics | 2015–2025 |
106
+ | `us/index/spx_daily` | S&P 500 index daily OHLC | OptionMetrics | 2015–2025 |
107
+ | `us/rates/zero_curve` | USD zero-coupon yield curve | OptionMetrics | 2015–2025 |
108
+ | `us/etf/daily_bars`, `bars_5min`, `bars_1min` | SPY, QQQ, IWM, TLT, IEF, GLD bars | Alpaca (local only) | 2016–2026 |
109
+ | `cn_hk/equity/daily_bars`, `ah_pairs` | China A/H dual-listed shares | Yahoo Finance (local only) | 2014–2026 |
110
+ | `cn_hk/fx/daily` | CNY/HKD, USD/HKD exchange rates | Yahoo Finance (local only) | 2014–2026 |
111
+
112
+ Datasets marked *local only* are not on the Hub because of their vendor terms; build them
113
+ from your own copy with `qrd build`. Full column-level documentation: [docs/datasets.md](docs/datasets.md), or
114
+ `qrd.describe("us_stock_daily")` in Python.
115
+
116
+ ## Install and get access
117
+
118
+ ```bash
119
+ pip install quant-research-dataset # add [duckdb] for SQL
120
+ hf auth login # once; the dataset is gated
121
+ ```
122
+
123
+ 1. Request access on the [dataset page](https://huggingface.co/datasets/Hardy0102/quant-research-dataset)
124
+ and confirm you may use the underlying vendor data (see [License](#license)).
125
+ Requests are approved by hand.
126
+ 2. Log in with a Hugging Face token (read access is enough).
127
+ 3. `import quant_research_dataset as qrd` and load: files download on first use
128
+ and are cached.
129
+
130
+ Already have the files locally? Skip the Hub:
131
+
132
+ ```python
133
+ qrd.configure(data_dir="D:/qrd-data") # or set QRD_DATA_DIR
134
+ ```
135
+
136
+ ## Usage
137
+
138
+ ### Discover
139
+
140
+ ```python
141
+ qrd.list_datasets(market="us", asset_class="options")
142
+ info = qrd.describe("spx_options") # metadata, coverage, load options, column table
143
+ info["columns"]
144
+ ```
145
+
146
+ ### Load anything with one function
147
+
148
+ ```python
149
+ qrd.load(dataset, ids=None, start=None, end=None, columns=None, filters=None, **options)
150
+ ```
151
+
152
+ * `ids`: values of the dataset's id column (PERMNOs, gvkeys, symbols...)
153
+ * `start`, `end`: inclusive date range; partitions outside it are never downloaded
154
+ * `columns`: only these columns are read from disk
155
+ * `filters`: `{"ticker": "AAPL"}`, `{"cp_flag": ["C"]}`, `{"ret": (">", 0.1)}`
156
+ * `**options`: dataset-specific, e.g. `common_only=True` (CRSP), `point_in_time=True` (Compustat)
157
+
158
+ ### Shortcuts
159
+
160
+ ```python
161
+ qrd.get_prices(market="us", start="2024-01-01", common_only=True) # CRSP
162
+ qrd.get_prices(["600036.SS", "3968.HK"], market="cn_hk") # A/H shares
163
+ qrd.us.get_stock_daily(tickers="NVDA", start="2023-01-01")
164
+ qrd.get_vol_surface("2020-01-01", "2020-12-31", days=30, delta=50, cp_flag="C")
165
+ qrd.get_etf_bars("SPY", start="2024-01-02", end="2024-01-02", freq="1min")
166
+ qrd.cn_hk.ah_premium(start="2024-01-01") # A/H premium per pair
167
+ qrd.zero_rate(["2024-03-15"], [45]) # 45-day zero rate
168
+ ```
169
+
170
+ ### SQL
171
+
172
+ ```python
173
+ con = qrd.duckdb_connect(["us_stock_daily"], start="2024-01-01")
174
+ con.sql("select permno, exp(sum(ln(1 + ret))) - 1 as ret_2024 from us_stock_daily group by 1").df()
175
+ ```
176
+
177
+ ### Research helpers
178
+
179
+ ```python
180
+ port = qrd.quantile_portfolios(panel, signal="mom_12_1", ret="fwd_ret", n=10, weight="mktcap")
181
+ qrd.performance_summary(port["long_short"], periods_per_year=12)
182
+ ```
183
+
184
+ ## Example: 12-1 momentum on US stocks
185
+
186
+ [examples/momentum.ipynb](examples/momentum.ipynb) runs a classic momentum
187
+ study end to end in a few cells: common stocks above the NYSE 20th size
188
+ percentile, 12-1 month signal, value-weighted deciles, 2016–2025.
189
+
190
+ ![Momentum deciles](docs/img/momentum.png)
191
+
192
+ | 2016–2025, monthly | Ann. return | Ann. vol | Sharpe | Max drawdown |
193
+ |---|---|---|---|---|
194
+ | Winners (D10) | 20.9% | 23.1% | 0.94 | −29.8% |
195
+ | Losers (D1) | 8.9% | 30.1% | 0.43 | −55.4% |
196
+ | Winners − losers | 5.5% | 26.1% | 0.33 | −38.9% |
197
+ | Market (CRSP VW) | 13.4% | 15.4% | 0.90 | −24.7% |
198
+
199
+ [examples/quickstart.ipynb](examples/quickstart.ipynb) tours the rest: point-in-time
200
+ fundamentals, an SPX volatility smile, the 30-day ATM vol series, the A/H
201
+ premium and SQL.
202
+
203
+ ## Design
204
+
205
+ ```
206
+ ┌──────────────── registry/<market>/<asset>/<name>.yaml ───────────────┐
207
+ │ id, columns (type/unit/description), keys, partitioning, build rules │
208
+ └───────────────┬────────────────────────────────────┬─────────────────┘
209
+ raw vendor files │ qrd build │ qrd.load()
210
+ (WRDS parquet/CSV, ...) ───┴──► partitioned Parquet + manifest ─┴──► Backend ──► filter push-down ──► adapter ──► DataFrame
211
+ (SHA-256, rows, date ranges) (Hugging Face Hub or local dir) (CRSP screen, PIT dates, option fields)
212
+ ```
213
+
214
+ * **Registry** ([`registry/`](src/quant_research_dataset/registry)): one YAML file per
215
+ dataset is the single source of truth. The data dictionary and the Hub dataset
216
+ card are generated from it, so docs never drift from code.
217
+ * **Manifest**: lists each Parquet file with rows, bytes, SHA-256 and date range.
218
+ The client reads it first and fetches only overlapping partitions.
219
+ * **Backends** ([`storage.py`](src/quant_research_dataset/storage.py)): Hugging Face
220
+ Hub (cached, parallel downloads, token-aware) or a local directory.
221
+ * **Adapters** ([`adapters/`](src/quant_research_dataset/adapters)): market-specific
222
+ logic (row filters pushed into the scan, derived columns) kept out of the
223
+ generic loader.
224
+ * **Markets** ([`markets/`](src/quant_research_dataset/markets)): convenience
225
+ functions per market (`qrd.us`, `qrd.cn_hk`).
226
+
227
+ ### Adding a dataset
228
+
229
+ 1. Put the raw file(s) in the source folder.
230
+ 2. Add `registry/<market>/<asset>/<name>.yaml` (see
231
+ [docs/adding-a-dataset.md](docs/adding-a-dataset.md)).
232
+ 3. `qrd build --source <raw> --out <prepared> --dataset <id>` and `qrd publish`.
233
+
234
+ No Python changes are needed unless the data needs special load-time logic.
235
+ Private datasets can live outside the package: point `QRD_REGISTRY_PATH` at a
236
+ folder of YAML files.
237
+
238
+ ## Development
239
+
240
+ ```bash
241
+ git clone https://github.com/Hardy-Z/quant-research-dataset && cd quant-research-dataset
242
+ pip install -e ".[dev]"
243
+ pytest # 68 tests on synthetic vendor-shaped data, no network
244
+ ruff check . && ruff format --check . && mypy
245
+ ```
246
+
247
+ Maintainer workflow (build, verify, publish) is in [docs/maintainers.md](docs/maintainers.md).
248
+
249
+ ## License
250
+
251
+ The **code** is MIT licensed. The **data** is not: each dataset keeps its
252
+ vendor's terms (shown in `describe()` and the data dictionary). CRSP,
253
+ Compustat and OptionMetrics data are provided for non-commercial academic
254
+ research to approved users only; do not redistribute the files.
@@ -0,0 +1,214 @@
1
+ # quant-research-dataset
2
+
3
+ [![CI](https://github.com/Hardy-Z/quant-research-dataset/actions/workflows/ci.yml/badge.svg)](https://github.com/Hardy-Z/quant-research-dataset/actions/workflows/ci.yml)
4
+ [![PyPI](https://img.shields.io/pypi/v/quant-research-dataset)](https://pypi.org/project/quant-research-dataset/)
5
+ [![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20dataset-Hardy0102%2Fquant--research--dataset-yellow)](https://huggingface.co/datasets/Hardy0102/quant-research-dataset)
6
+ ![Python](https://img.shields.io/badge/python-3.10%E2%80%933.13-blue)
7
+ ![License](https://img.shields.io/badge/code-MIT-green)
8
+
9
+ **Code:** [GitHub](https://github.com/Hardy-Z/quant-research-dataset) ·
10
+ **Package:** [PyPI](https://pypi.org/project/quant-research-dataset/) ·
11
+ **Data:** [Hugging Face](https://huggingface.co/datasets/Hardy0102/quant-research-dataset) (gated) ·
12
+ **Docs:** [data dictionary](docs/datasets.md)
13
+
14
+ **One Python API for research-grade market data across markets.** Equities,
15
+ fundamentals, index options, rates and ETFs live in a versioned, partitioned
16
+ Parquet dataset on the Hugging Face Hub. The package downloads only the
17
+ partitions a query touches, pushes filters down into the Parquet scan and
18
+ returns clean, documented `pandas` DataFrames.
19
+
20
+ ```python
21
+ import quant_research_dataset as qrd
22
+
23
+ # Every US common stock's daily return in 2024 (one ~90 MB partition, cached after the first call)
24
+ stocks = qrd.load(
25
+ "us/equity/stock_daily",
26
+ start="2024-01-01",
27
+ end="2024-12-31",
28
+ columns=["date", "permno", "gvkey", "ret", "mktcap"],
29
+ common_only=True,
30
+ )
31
+
32
+ # Point-in-time fundamentals, joined without look-ahead bias
33
+ fund = qrd.get_fundamentals("quarterly", columns=["gvkey", "datadate", "atq", "ceqq"])
34
+ panel = qrd.merge_fundamentals(stocks, fund)
35
+
36
+ # SPX options for a day, with mid price, days to expiry, spot and a matched zero rate
37
+ chain = qrd.get_spx_options("2024-03-15", with_market=True)
38
+ ```
39
+
40
+ ## Why
41
+
42
+ Academic-quality data (CRSP, Compustat, OptionMetrics) is powerful but awkward:
43
+ wide vendor tables, cryptic codes, strikes stored ×1000, link tables with date
44
+ ranges, fiscal dates that leak the future. Every project re-implements the same
45
+ cleaning. This library does it once, in tested code, behind a small API:
46
+
47
+ | Problem | What the library does |
48
+ |---|---|
49
+ | Multi-GB files, slow to load | Year/month partitions + a manifest, so `load(start=..., end=...)` fetches only what it needs |
50
+ | Different column names per vendor | Every dataset has a typed schema, snake_case names, units and descriptions in one YAML file |
51
+ | Look-ahead bias in fundamentals | `point_in_time=True` adds `available_date` (report date, or a conservative lag) |
52
+ | Joining stocks to company fundamentals | Links are applied at build time: stock rows carry `gvkey`, fundamentals carry `permno`; `merge_fundamentals()` joins point in time |
53
+ | "Common stocks only" screens | `common_only=True` applies the CIZ-flag equivalent of `shrcd in (10, 11)` |
54
+ | Option chains need context | `mid`, `spread`, `dte`, `spot`, `moneyness` and an interpolated zero rate |
55
+ | US-only tools | Market-neutral design: `market/asset_class/dataset` ids, per-market adapters |
56
+
57
+ ## Datasets
58
+
59
+ | Id | Content | Source | Coverage |
60
+ |---|---|---|---|
61
+ | `us/equity/stock_daily` | Daily prices, returns, volume and market cap of all US-listed stocks, incl. delisted, with linked `gvkey` | CRSP | 2015–2025, 23.1M rows |
62
+ | `us/fundamentals/quarterly` | Quarterly financial statements, 60+ items, with linked `permno` | Compustat | 2000–2026, 1.05M rows |
63
+ | `us/fundamentals/annual` | Annual financial statements, 60+ items, with linked `permno` | Compustat | 2000–2026, 255K rows |
64
+ | `us/options/spx_option_prices` | S&P 500 index option quotes, implied vol and greeks | OptionMetrics | 2015–2025, 34.3M rows |
65
+ | `us/options/spx_vol_surface` | Standardized S&P 500 implied volatility surface | OptionMetrics | 2015–2025 |
66
+ | `us/index/spx_daily` | S&P 500 index daily OHLC | OptionMetrics | 2015–2025 |
67
+ | `us/rates/zero_curve` | USD zero-coupon yield curve | OptionMetrics | 2015–2025 |
68
+ | `us/etf/daily_bars`, `bars_5min`, `bars_1min` | SPY, QQQ, IWM, TLT, IEF, GLD bars | Alpaca (local only) | 2016–2026 |
69
+ | `cn_hk/equity/daily_bars`, `ah_pairs` | China A/H dual-listed shares | Yahoo Finance (local only) | 2014–2026 |
70
+ | `cn_hk/fx/daily` | CNY/HKD, USD/HKD exchange rates | Yahoo Finance (local only) | 2014–2026 |
71
+
72
+ Datasets marked *local only* are not on the Hub because of their vendor terms; build them
73
+ from your own copy with `qrd build`. Full column-level documentation: [docs/datasets.md](docs/datasets.md), or
74
+ `qrd.describe("us_stock_daily")` in Python.
75
+
76
+ ## Install and get access
77
+
78
+ ```bash
79
+ pip install quant-research-dataset # add [duckdb] for SQL
80
+ hf auth login # once; the dataset is gated
81
+ ```
82
+
83
+ 1. Request access on the [dataset page](https://huggingface.co/datasets/Hardy0102/quant-research-dataset)
84
+ and confirm you may use the underlying vendor data (see [License](#license)).
85
+ Requests are approved by hand.
86
+ 2. Log in with a Hugging Face token (read access is enough).
87
+ 3. `import quant_research_dataset as qrd` and load: files download on first use
88
+ and are cached.
89
+
90
+ Already have the files locally? Skip the Hub:
91
+
92
+ ```python
93
+ qrd.configure(data_dir="D:/qrd-data") # or set QRD_DATA_DIR
94
+ ```
95
+
96
+ ## Usage
97
+
98
+ ### Discover
99
+
100
+ ```python
101
+ qrd.list_datasets(market="us", asset_class="options")
102
+ info = qrd.describe("spx_options") # metadata, coverage, load options, column table
103
+ info["columns"]
104
+ ```
105
+
106
+ ### Load anything with one function
107
+
108
+ ```python
109
+ qrd.load(dataset, ids=None, start=None, end=None, columns=None, filters=None, **options)
110
+ ```
111
+
112
+ * `ids`: values of the dataset's id column (PERMNOs, gvkeys, symbols...)
113
+ * `start`, `end`: inclusive date range; partitions outside it are never downloaded
114
+ * `columns`: only these columns are read from disk
115
+ * `filters`: `{"ticker": "AAPL"}`, `{"cp_flag": ["C"]}`, `{"ret": (">", 0.1)}`
116
+ * `**options`: dataset-specific, e.g. `common_only=True` (CRSP), `point_in_time=True` (Compustat)
117
+
118
+ ### Shortcuts
119
+
120
+ ```python
121
+ qrd.get_prices(market="us", start="2024-01-01", common_only=True) # CRSP
122
+ qrd.get_prices(["600036.SS", "3968.HK"], market="cn_hk") # A/H shares
123
+ qrd.us.get_stock_daily(tickers="NVDA", start="2023-01-01")
124
+ qrd.get_vol_surface("2020-01-01", "2020-12-31", days=30, delta=50, cp_flag="C")
125
+ qrd.get_etf_bars("SPY", start="2024-01-02", end="2024-01-02", freq="1min")
126
+ qrd.cn_hk.ah_premium(start="2024-01-01") # A/H premium per pair
127
+ qrd.zero_rate(["2024-03-15"], [45]) # 45-day zero rate
128
+ ```
129
+
130
+ ### SQL
131
+
132
+ ```python
133
+ con = qrd.duckdb_connect(["us_stock_daily"], start="2024-01-01")
134
+ con.sql("select permno, exp(sum(ln(1 + ret))) - 1 as ret_2024 from us_stock_daily group by 1").df()
135
+ ```
136
+
137
+ ### Research helpers
138
+
139
+ ```python
140
+ port = qrd.quantile_portfolios(panel, signal="mom_12_1", ret="fwd_ret", n=10, weight="mktcap")
141
+ qrd.performance_summary(port["long_short"], periods_per_year=12)
142
+ ```
143
+
144
+ ## Example: 12-1 momentum on US stocks
145
+
146
+ [examples/momentum.ipynb](examples/momentum.ipynb) runs a classic momentum
147
+ study end to end in a few cells: common stocks above the NYSE 20th size
148
+ percentile, 12-1 month signal, value-weighted deciles, 2016–2025.
149
+
150
+ ![Momentum deciles](docs/img/momentum.png)
151
+
152
+ | 2016–2025, monthly | Ann. return | Ann. vol | Sharpe | Max drawdown |
153
+ |---|---|---|---|---|
154
+ | Winners (D10) | 20.9% | 23.1% | 0.94 | −29.8% |
155
+ | Losers (D1) | 8.9% | 30.1% | 0.43 | −55.4% |
156
+ | Winners − losers | 5.5% | 26.1% | 0.33 | −38.9% |
157
+ | Market (CRSP VW) | 13.4% | 15.4% | 0.90 | −24.7% |
158
+
159
+ [examples/quickstart.ipynb](examples/quickstart.ipynb) tours the rest: point-in-time
160
+ fundamentals, an SPX volatility smile, the 30-day ATM vol series, the A/H
161
+ premium and SQL.
162
+
163
+ ## Design
164
+
165
+ ```
166
+ ┌──────────────── registry/<market>/<asset>/<name>.yaml ───────────────┐
167
+ │ id, columns (type/unit/description), keys, partitioning, build rules │
168
+ └───────────────┬────────────────────────────────────┬─────────────────┘
169
+ raw vendor files │ qrd build │ qrd.load()
170
+ (WRDS parquet/CSV, ...) ───┴──► partitioned Parquet + manifest ─┴──► Backend ──► filter push-down ──► adapter ──► DataFrame
171
+ (SHA-256, rows, date ranges) (Hugging Face Hub or local dir) (CRSP screen, PIT dates, option fields)
172
+ ```
173
+
174
+ * **Registry** ([`registry/`](src/quant_research_dataset/registry)): one YAML file per
175
+ dataset is the single source of truth. The data dictionary and the Hub dataset
176
+ card are generated from it, so docs never drift from code.
177
+ * **Manifest**: lists each Parquet file with rows, bytes, SHA-256 and date range.
178
+ The client reads it first and fetches only overlapping partitions.
179
+ * **Backends** ([`storage.py`](src/quant_research_dataset/storage.py)): Hugging Face
180
+ Hub (cached, parallel downloads, token-aware) or a local directory.
181
+ * **Adapters** ([`adapters/`](src/quant_research_dataset/adapters)): market-specific
182
+ logic (row filters pushed into the scan, derived columns) kept out of the
183
+ generic loader.
184
+ * **Markets** ([`markets/`](src/quant_research_dataset/markets)): convenience
185
+ functions per market (`qrd.us`, `qrd.cn_hk`).
186
+
187
+ ### Adding a dataset
188
+
189
+ 1. Put the raw file(s) in the source folder.
190
+ 2. Add `registry/<market>/<asset>/<name>.yaml` (see
191
+ [docs/adding-a-dataset.md](docs/adding-a-dataset.md)).
192
+ 3. `qrd build --source <raw> --out <prepared> --dataset <id>` and `qrd publish`.
193
+
194
+ No Python changes are needed unless the data needs special load-time logic.
195
+ Private datasets can live outside the package: point `QRD_REGISTRY_PATH` at a
196
+ folder of YAML files.
197
+
198
+ ## Development
199
+
200
+ ```bash
201
+ git clone https://github.com/Hardy-Z/quant-research-dataset && cd quant-research-dataset
202
+ pip install -e ".[dev]"
203
+ pytest # 68 tests on synthetic vendor-shaped data, no network
204
+ ruff check . && ruff format --check . && mypy
205
+ ```
206
+
207
+ Maintainer workflow (build, verify, publish) is in [docs/maintainers.md](docs/maintainers.md).
208
+
209
+ ## License
210
+
211
+ The **code** is MIT licensed. The **data** is not: each dataset keeps its
212
+ vendor's terms (shown in `describe()` and the data dictionary). CRSP,
213
+ Compustat and OptionMetrics data are provided for non-commercial academic
214
+ research to approved users only; do not redistribute the files.