quant-research-dataset 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- quant_research_dataset-0.1.0/.gitignore +24 -0
- quant_research_dataset-0.1.0/CHANGELOG.md +23 -0
- quant_research_dataset-0.1.0/LICENSE +24 -0
- quant_research_dataset-0.1.0/PKG-INFO +254 -0
- quant_research_dataset-0.1.0/README.md +214 -0
- quant_research_dataset-0.1.0/hf/README.md +447 -0
- quant_research_dataset-0.1.0/pyproject.toml +83 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/__init__.py +67 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/_version.py +1 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/adapters/__init__.py +12 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/adapters/base.py +63 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/adapters/us_fundamentals.py +60 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/adapters/us_options.py +35 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/adapters/us_stocks.py +48 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/analytics.py +73 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/build.py +316 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/catalog.py +176 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/cli.py +169 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/config.py +107 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/errors.py +26 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/links.py +152 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/manifest.py +202 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/markets/__init__.py +50 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/markets/cn_hk.py +73 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/markets/us.py +236 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/publish.py +147 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/py.typed +0 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/query.py +349 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/registry/cn_hk/equity/ah_pairs.yaml +21 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/registry/cn_hk/equity/daily_bars.yaml +30 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/registry/cn_hk/fx/daily.yaml +25 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/equity/stock_daily.yaml +64 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/etf/bars_1min.yaml +27 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/etf/bars_5min.yaml +32 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/etf/daily_bars.yaml +31 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/fundamentals/annual.yaml +90 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/fundamentals/quarterly.yaml +93 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/index/spx_daily.yaml +25 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/options/spx_option_prices.yaml +54 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/options/spx_vol_surface.yaml +29 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/registry/us/rates/zero_curve.yaml +23 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/registry.py +335 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/sql.py +54 -0
- quant_research_dataset-0.1.0/src/quant_research_dataset/storage.py +139 -0
- quant_research_dataset-0.1.0/tests/__init__.py +0 -0
- quant_research_dataset-0.1.0/tests/conftest.py +51 -0
- quant_research_dataset-0.1.0/tests/fakes.py +264 -0
- quant_research_dataset-0.1.0/tests/test_analytics.py +54 -0
- quant_research_dataset-0.1.0/tests/test_build.py +188 -0
- quant_research_dataset-0.1.0/tests/test_load.py +149 -0
- quant_research_dataset-0.1.0/tests/test_markets.py +144 -0
- quant_research_dataset-0.1.0/tests/test_registry.py +118 -0
- quant_research_dataset-0.1.0/tests/test_storage_publish.py +134 -0
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Python
|
|
2
|
+
__pycache__/
|
|
3
|
+
*.py[cod]
|
|
4
|
+
*.egg-info/
|
|
5
|
+
.venv/
|
|
6
|
+
build/
|
|
7
|
+
dist/
|
|
8
|
+
.pytest_cache/
|
|
9
|
+
.mypy_cache/
|
|
10
|
+
.ruff_cache/
|
|
11
|
+
.coverage
|
|
12
|
+
htmlcov/
|
|
13
|
+
.ipynb_checkpoints/
|
|
14
|
+
site/
|
|
15
|
+
|
|
16
|
+
# Data never goes into git: raw inputs, prepared partitions, caches
|
|
17
|
+
source_data_copy/
|
|
18
|
+
prepared/
|
|
19
|
+
.qrd_cache/
|
|
20
|
+
*.parquet
|
|
21
|
+
*.csv.gz
|
|
22
|
+
|
|
23
|
+
# Secrets
|
|
24
|
+
.env
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.1.0 (unreleased)
|
|
4
|
+
|
|
5
|
+
First release.
|
|
6
|
+
|
|
7
|
+
* Generic API: `load`, `list_datasets`, `describe`, `download`, `files_for`.
|
|
8
|
+
* YAML dataset registry with typed schemas, units and descriptions; generated
|
|
9
|
+
data dictionary and Hugging Face dataset card.
|
|
10
|
+
* Storage backends: Hugging Face Hub (gated or private repos) and local folders.
|
|
11
|
+
* Build pipeline from raw vendor files to partitioned Parquet with a
|
|
12
|
+
checksummed manifest; `qrd verify` and `qrd publish`.
|
|
13
|
+
* 13 datasets, named by content (source kept as metadata): US stock daily,
|
|
14
|
+
US fundamentals quarterly/annual, SPX option prices and
|
|
15
|
+
vol surface, SPX index, USD zero curve, US ETF bars (1d, 5m, 1m), China A/H
|
|
16
|
+
daily bars, A/H pairs, CNY/HKD FX.
|
|
17
|
+
* Stock-company links are applied at build time: stock rows carry `gvkey` and
|
|
18
|
+
fundamentals carry `permno` (primary share class).
|
|
19
|
+
* US helpers: common-stock screen, point-in-time fundamentals,
|
|
20
|
+
`merge_fundamentals`, option chains with spot, rates and moneyness.
|
|
21
|
+
* China/HK helpers: A/H premium.
|
|
22
|
+
* Research helpers: quantile portfolio sorts, performance summary.
|
|
23
|
+
* SQL access through DuckDB.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Hardy-Z
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
22
|
+
|
|
23
|
+
This license covers the source code only. The datasets the code loads are
|
|
24
|
+
licensed separately by their vendors; see each dataset's license field.
|
|
@@ -0,0 +1,254 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: quant-research-dataset
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: One API for research-grade market data across markets: equities, options, ETFs, fundamentals and rates, served from a versioned Hugging Face dataset.
|
|
5
|
+
Project-URL: Homepage, https://github.com/Hardy-Z/quant-research-dataset
|
|
6
|
+
Project-URL: Issues, https://github.com/Hardy-Z/quant-research-dataset/issues
|
|
7
|
+
Author: Hardy-Z
|
|
8
|
+
License-Expression: MIT
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Keywords: backtesting,compustat,crsp,dataset,finance,hugging-face,optionmetrics,quant
|
|
11
|
+
Classifier: Development Status :: 3 - Alpha
|
|
12
|
+
Classifier: Intended Audience :: Financial and Insurance Industry
|
|
13
|
+
Classifier: Intended Audience :: Science/Research
|
|
14
|
+
Classifier: Operating System :: OS Independent
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Topic :: Office/Business :: Financial :: Investment
|
|
21
|
+
Classifier: Typing :: Typed
|
|
22
|
+
Requires-Python: >=3.10
|
|
23
|
+
Requires-Dist: huggingface-hub>=0.23
|
|
24
|
+
Requires-Dist: pandas>=2.0
|
|
25
|
+
Requires-Dist: pyarrow>=14
|
|
26
|
+
Requires-Dist: pyyaml>=6
|
|
27
|
+
Provides-Extra: dev
|
|
28
|
+
Requires-Dist: duckdb>=1.0; extra == 'dev'
|
|
29
|
+
Requires-Dist: mypy>=1.10; extra == 'dev'
|
|
30
|
+
Requires-Dist: pandas-stubs; extra == 'dev'
|
|
31
|
+
Requires-Dist: pytest-cov>=5; extra == 'dev'
|
|
32
|
+
Requires-Dist: pytest>=8; extra == 'dev'
|
|
33
|
+
Requires-Dist: ruff>=0.6; extra == 'dev'
|
|
34
|
+
Requires-Dist: types-pyyaml; extra == 'dev'
|
|
35
|
+
Provides-Extra: docs
|
|
36
|
+
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
|
|
37
|
+
Provides-Extra: duckdb
|
|
38
|
+
Requires-Dist: duckdb>=1.0; extra == 'duckdb'
|
|
39
|
+
Description-Content-Type: text/markdown
|
|
40
|
+
|
|
41
|
+
# quant-research-dataset
|
|
42
|
+
|
|
43
|
+
[](https://github.com/Hardy-Z/quant-research-dataset/actions/workflows/ci.yml)
|
|
44
|
+
[](https://pypi.org/project/quant-research-dataset/)
|
|
45
|
+
[](https://huggingface.co/datasets/Hardy0102/quant-research-dataset)
|
|
46
|
+

|
|
47
|
+

|
|
48
|
+
|
|
49
|
+
**Code:** [GitHub](https://github.com/Hardy-Z/quant-research-dataset) ·
|
|
50
|
+
**Package:** [PyPI](https://pypi.org/project/quant-research-dataset/) ·
|
|
51
|
+
**Data:** [Hugging Face](https://huggingface.co/datasets/Hardy0102/quant-research-dataset) (gated) ·
|
|
52
|
+
**Docs:** [data dictionary](docs/datasets.md)
|
|
53
|
+
|
|
54
|
+
**One Python API for research-grade market data across markets.** Equities,
|
|
55
|
+
fundamentals, index options, rates and ETFs live in a versioned, partitioned
|
|
56
|
+
Parquet dataset on the Hugging Face Hub. The package downloads only the
|
|
57
|
+
partitions a query touches, pushes filters down into the Parquet scan and
|
|
58
|
+
returns clean, documented `pandas` DataFrames.
|
|
59
|
+
|
|
60
|
+
```python
|
|
61
|
+
import quant_research_dataset as qrd
|
|
62
|
+
|
|
63
|
+
# Every US common stock's daily return in 2024 (one ~90 MB partition, cached after the first call)
|
|
64
|
+
stocks = qrd.load(
|
|
65
|
+
"us/equity/stock_daily",
|
|
66
|
+
start="2024-01-01",
|
|
67
|
+
end="2024-12-31",
|
|
68
|
+
columns=["date", "permno", "gvkey", "ret", "mktcap"],
|
|
69
|
+
common_only=True,
|
|
70
|
+
)
|
|
71
|
+
|
|
72
|
+
# Point-in-time fundamentals, joined without look-ahead bias
|
|
73
|
+
fund = qrd.get_fundamentals("quarterly", columns=["gvkey", "datadate", "atq", "ceqq"])
|
|
74
|
+
panel = qrd.merge_fundamentals(stocks, fund)
|
|
75
|
+
|
|
76
|
+
# SPX options for a day, with mid price, days to expiry, spot and a matched zero rate
|
|
77
|
+
chain = qrd.get_spx_options("2024-03-15", with_market=True)
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
## Why
|
|
81
|
+
|
|
82
|
+
Academic-quality data (CRSP, Compustat, OptionMetrics) is powerful but awkward:
|
|
83
|
+
wide vendor tables, cryptic codes, strikes stored ×1000, link tables with date
|
|
84
|
+
ranges, fiscal dates that leak the future. Every project re-implements the same
|
|
85
|
+
cleaning. This library does it once, in tested code, behind a small API:
|
|
86
|
+
|
|
87
|
+
| Problem | What the library does |
|
|
88
|
+
|---|---|
|
|
89
|
+
| Multi-GB files, slow to load | Year/month partitions + a manifest, so `load(start=..., end=...)` fetches only what it needs |
|
|
90
|
+
| Different column names per vendor | Every dataset has a typed schema, snake_case names, units and descriptions in one YAML file |
|
|
91
|
+
| Look-ahead bias in fundamentals | `point_in_time=True` adds `available_date` (report date, or a conservative lag) |
|
|
92
|
+
| Joining stocks to company fundamentals | Links are applied at build time: stock rows carry `gvkey`, fundamentals carry `permno`; `merge_fundamentals()` joins point in time |
|
|
93
|
+
| "Common stocks only" screens | `common_only=True` applies the CIZ-flag equivalent of `shrcd in (10, 11)` |
|
|
94
|
+
| Option chains need context | `mid`, `spread`, `dte`, `spot`, `moneyness` and an interpolated zero rate |
|
|
95
|
+
| US-only tools | Market-neutral design: `market/asset_class/dataset` ids, per-market adapters |
|
|
96
|
+
|
|
97
|
+
## Datasets
|
|
98
|
+
|
|
99
|
+
| Id | Content | Source | Coverage |
|
|
100
|
+
|---|---|---|---|
|
|
101
|
+
| `us/equity/stock_daily` | Daily prices, returns, volume and market cap of all US-listed stocks, incl. delisted, with linked `gvkey` | CRSP | 2015–2025, 23.1M rows |
|
|
102
|
+
| `us/fundamentals/quarterly` | Quarterly financial statements, 60+ items, with linked `permno` | Compustat | 2000–2026, 1.05M rows |
|
|
103
|
+
| `us/fundamentals/annual` | Annual financial statements, 60+ items, with linked `permno` | Compustat | 2000–2026, 255K rows |
|
|
104
|
+
| `us/options/spx_option_prices` | S&P 500 index option quotes, implied vol and greeks | OptionMetrics | 2015–2025, 34.3M rows |
|
|
105
|
+
| `us/options/spx_vol_surface` | Standardized S&P 500 implied volatility surface | OptionMetrics | 2015–2025 |
|
|
106
|
+
| `us/index/spx_daily` | S&P 500 index daily OHLC | OptionMetrics | 2015–2025 |
|
|
107
|
+
| `us/rates/zero_curve` | USD zero-coupon yield curve | OptionMetrics | 2015–2025 |
|
|
108
|
+
| `us/etf/daily_bars`, `bars_5min`, `bars_1min` | SPY, QQQ, IWM, TLT, IEF, GLD bars | Alpaca (local only) | 2016–2026 |
|
|
109
|
+
| `cn_hk/equity/daily_bars`, `ah_pairs` | China A/H dual-listed shares | Yahoo Finance (local only) | 2014–2026 |
|
|
110
|
+
| `cn_hk/fx/daily` | CNY/HKD, USD/HKD exchange rates | Yahoo Finance (local only) | 2014–2026 |
|
|
111
|
+
|
|
112
|
+
Datasets marked *local only* are not on the Hub because of their vendor terms; build them
|
|
113
|
+
from your own copy with `qrd build`. Full column-level documentation: [docs/datasets.md](docs/datasets.md), or
|
|
114
|
+
`qrd.describe("us_stock_daily")` in Python.
|
|
115
|
+
|
|
116
|
+
## Install and get access
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
pip install quant-research-dataset # add [duckdb] for SQL
|
|
120
|
+
hf auth login # once; the dataset is gated
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
1. Request access on the [dataset page](https://huggingface.co/datasets/Hardy0102/quant-research-dataset)
|
|
124
|
+
and confirm you may use the underlying vendor data (see [License](#license)).
|
|
125
|
+
Requests are approved by hand.
|
|
126
|
+
2. Log in with a Hugging Face token (read access is enough).
|
|
127
|
+
3. `import quant_research_dataset as qrd` and load: files download on first use
|
|
128
|
+
and are cached.
|
|
129
|
+
|
|
130
|
+
Already have the files locally? Skip the Hub:
|
|
131
|
+
|
|
132
|
+
```python
|
|
133
|
+
qrd.configure(data_dir="D:/qrd-data") # or set QRD_DATA_DIR
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
## Usage
|
|
137
|
+
|
|
138
|
+
### Discover
|
|
139
|
+
|
|
140
|
+
```python
|
|
141
|
+
qrd.list_datasets(market="us", asset_class="options")
|
|
142
|
+
info = qrd.describe("spx_options") # metadata, coverage, load options, column table
|
|
143
|
+
info["columns"]
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
### Load anything with one function
|
|
147
|
+
|
|
148
|
+
```python
|
|
149
|
+
qrd.load(dataset, ids=None, start=None, end=None, columns=None, filters=None, **options)
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
* `ids`: values of the dataset's id column (PERMNOs, gvkeys, symbols...)
|
|
153
|
+
* `start`, `end`: inclusive date range; partitions outside it are never downloaded
|
|
154
|
+
* `columns`: only these columns are read from disk
|
|
155
|
+
* `filters`: `{"ticker": "AAPL"}`, `{"cp_flag": ["C"]}`, `{"ret": (">", 0.1)}`
|
|
156
|
+
* `**options`: dataset-specific, e.g. `common_only=True` (CRSP), `point_in_time=True` (Compustat)
|
|
157
|
+
|
|
158
|
+
### Shortcuts
|
|
159
|
+
|
|
160
|
+
```python
|
|
161
|
+
qrd.get_prices(market="us", start="2024-01-01", common_only=True) # CRSP
|
|
162
|
+
qrd.get_prices(["600036.SS", "3968.HK"], market="cn_hk") # A/H shares
|
|
163
|
+
qrd.us.get_stock_daily(tickers="NVDA", start="2023-01-01")
|
|
164
|
+
qrd.get_vol_surface("2020-01-01", "2020-12-31", days=30, delta=50, cp_flag="C")
|
|
165
|
+
qrd.get_etf_bars("SPY", start="2024-01-02", end="2024-01-02", freq="1min")
|
|
166
|
+
qrd.cn_hk.ah_premium(start="2024-01-01") # A/H premium per pair
|
|
167
|
+
qrd.zero_rate(["2024-03-15"], [45]) # 45-day zero rate
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
### SQL
|
|
171
|
+
|
|
172
|
+
```python
|
|
173
|
+
con = qrd.duckdb_connect(["us_stock_daily"], start="2024-01-01")
|
|
174
|
+
con.sql("select permno, exp(sum(ln(1 + ret))) - 1 as ret_2024 from us_stock_daily group by 1").df()
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
### Research helpers
|
|
178
|
+
|
|
179
|
+
```python
|
|
180
|
+
port = qrd.quantile_portfolios(panel, signal="mom_12_1", ret="fwd_ret", n=10, weight="mktcap")
|
|
181
|
+
qrd.performance_summary(port["long_short"], periods_per_year=12)
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
## Example: 12-1 momentum on US stocks
|
|
185
|
+
|
|
186
|
+
[examples/momentum.ipynb](examples/momentum.ipynb) runs a classic momentum
|
|
187
|
+
study end to end in a few cells: common stocks above the NYSE 20th size
|
|
188
|
+
percentile, 12-1 month signal, value-weighted deciles, 2016–2025.
|
|
189
|
+
|
|
190
|
+

|
|
191
|
+
|
|
192
|
+
| 2016–2025, monthly | Ann. return | Ann. vol | Sharpe | Max drawdown |
|
|
193
|
+
|---|---|---|---|---|
|
|
194
|
+
| Winners (D10) | 20.9% | 23.1% | 0.94 | −29.8% |
|
|
195
|
+
| Losers (D1) | 8.9% | 30.1% | 0.43 | −55.4% |
|
|
196
|
+
| Winners − losers | 5.5% | 26.1% | 0.33 | −38.9% |
|
|
197
|
+
| Market (CRSP VW) | 13.4% | 15.4% | 0.90 | −24.7% |
|
|
198
|
+
|
|
199
|
+
[examples/quickstart.ipynb](examples/quickstart.ipynb) tours the rest: point-in-time
|
|
200
|
+
fundamentals, an SPX volatility smile, the 30-day ATM vol series, the A/H
|
|
201
|
+
premium and SQL.
|
|
202
|
+
|
|
203
|
+
## Design
|
|
204
|
+
|
|
205
|
+
```
|
|
206
|
+
┌──────────────── registry/<market>/<asset>/<name>.yaml ───────────────┐
|
|
207
|
+
│ id, columns (type/unit/description), keys, partitioning, build rules │
|
|
208
|
+
└───────────────┬────────────────────────────────────┬─────────────────┘
|
|
209
|
+
raw vendor files │ qrd build │ qrd.load()
|
|
210
|
+
(WRDS parquet/CSV, ...) ───┴──► partitioned Parquet + manifest ─┴──► Backend ──► filter push-down ──► adapter ──► DataFrame
|
|
211
|
+
(SHA-256, rows, date ranges) (Hugging Face Hub or local dir) (CRSP screen, PIT dates, option fields)
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
* **Registry** ([`registry/`](src/quant_research_dataset/registry)): one YAML file per
|
|
215
|
+
dataset is the single source of truth. The data dictionary and the Hub dataset
|
|
216
|
+
card are generated from it, so docs never drift from code.
|
|
217
|
+
* **Manifest**: lists each Parquet file with rows, bytes, SHA-256 and date range.
|
|
218
|
+
The client reads it first and fetches only overlapping partitions.
|
|
219
|
+
* **Backends** ([`storage.py`](src/quant_research_dataset/storage.py)): Hugging Face
|
|
220
|
+
Hub (cached, parallel downloads, token-aware) or a local directory.
|
|
221
|
+
* **Adapters** ([`adapters/`](src/quant_research_dataset/adapters)): market-specific
|
|
222
|
+
logic (row filters pushed into the scan, derived columns) kept out of the
|
|
223
|
+
generic loader.
|
|
224
|
+
* **Markets** ([`markets/`](src/quant_research_dataset/markets)): convenience
|
|
225
|
+
functions per market (`qrd.us`, `qrd.cn_hk`).
|
|
226
|
+
|
|
227
|
+
### Adding a dataset
|
|
228
|
+
|
|
229
|
+
1. Put the raw file(s) in the source folder.
|
|
230
|
+
2. Add `registry/<market>/<asset>/<name>.yaml` (see
|
|
231
|
+
[docs/adding-a-dataset.md](docs/adding-a-dataset.md)).
|
|
232
|
+
3. `qrd build --source <raw> --out <prepared> --dataset <id>` and `qrd publish`.
|
|
233
|
+
|
|
234
|
+
No Python changes are needed unless the data needs special load-time logic.
|
|
235
|
+
Private datasets can live outside the package: point `QRD_REGISTRY_PATH` at a
|
|
236
|
+
folder of YAML files.
|
|
237
|
+
|
|
238
|
+
## Development
|
|
239
|
+
|
|
240
|
+
```bash
|
|
241
|
+
git clone https://github.com/Hardy-Z/quant-research-dataset && cd quant-research-dataset
|
|
242
|
+
pip install -e ".[dev]"
|
|
243
|
+
pytest # 68 tests on synthetic vendor-shaped data, no network
|
|
244
|
+
ruff check . && ruff format --check . && mypy
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
Maintainer workflow (build, verify, publish) is in [docs/maintainers.md](docs/maintainers.md).
|
|
248
|
+
|
|
249
|
+
## License
|
|
250
|
+
|
|
251
|
+
The **code** is MIT licensed. The **data** is not: each dataset keeps its
|
|
252
|
+
vendor's terms (shown in `describe()` and the data dictionary). CRSP,
|
|
253
|
+
Compustat and OptionMetrics data are provided for non-commercial academic
|
|
254
|
+
research to approved users only; do not redistribute the files.
|
|
@@ -0,0 +1,214 @@
|
|
|
1
|
+
# quant-research-dataset
|
|
2
|
+
|
|
3
|
+
[](https://github.com/Hardy-Z/quant-research-dataset/actions/workflows/ci.yml)
|
|
4
|
+
[](https://pypi.org/project/quant-research-dataset/)
|
|
5
|
+
[](https://huggingface.co/datasets/Hardy0102/quant-research-dataset)
|
|
6
|
+

|
|
7
|
+

|
|
8
|
+
|
|
9
|
+
**Code:** [GitHub](https://github.com/Hardy-Z/quant-research-dataset) ·
|
|
10
|
+
**Package:** [PyPI](https://pypi.org/project/quant-research-dataset/) ·
|
|
11
|
+
**Data:** [Hugging Face](https://huggingface.co/datasets/Hardy0102/quant-research-dataset) (gated) ·
|
|
12
|
+
**Docs:** [data dictionary](docs/datasets.md)
|
|
13
|
+
|
|
14
|
+
**One Python API for research-grade market data across markets.** Equities,
|
|
15
|
+
fundamentals, index options, rates and ETFs live in a versioned, partitioned
|
|
16
|
+
Parquet dataset on the Hugging Face Hub. The package downloads only the
|
|
17
|
+
partitions a query touches, pushes filters down into the Parquet scan and
|
|
18
|
+
returns clean, documented `pandas` DataFrames.
|
|
19
|
+
|
|
20
|
+
```python
|
|
21
|
+
import quant_research_dataset as qrd
|
|
22
|
+
|
|
23
|
+
# Every US common stock's daily return in 2024 (one ~90 MB partition, cached after the first call)
|
|
24
|
+
stocks = qrd.load(
|
|
25
|
+
"us/equity/stock_daily",
|
|
26
|
+
start="2024-01-01",
|
|
27
|
+
end="2024-12-31",
|
|
28
|
+
columns=["date", "permno", "gvkey", "ret", "mktcap"],
|
|
29
|
+
common_only=True,
|
|
30
|
+
)
|
|
31
|
+
|
|
32
|
+
# Point-in-time fundamentals, joined without look-ahead bias
|
|
33
|
+
fund = qrd.get_fundamentals("quarterly", columns=["gvkey", "datadate", "atq", "ceqq"])
|
|
34
|
+
panel = qrd.merge_fundamentals(stocks, fund)
|
|
35
|
+
|
|
36
|
+
# SPX options for a day, with mid price, days to expiry, spot and a matched zero rate
|
|
37
|
+
chain = qrd.get_spx_options("2024-03-15", with_market=True)
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
## Why
|
|
41
|
+
|
|
42
|
+
Academic-quality data (CRSP, Compustat, OptionMetrics) is powerful but awkward:
|
|
43
|
+
wide vendor tables, cryptic codes, strikes stored ×1000, link tables with date
|
|
44
|
+
ranges, fiscal dates that leak the future. Every project re-implements the same
|
|
45
|
+
cleaning. This library does it once, in tested code, behind a small API:
|
|
46
|
+
|
|
47
|
+
| Problem | What the library does |
|
|
48
|
+
|---|---|
|
|
49
|
+
| Multi-GB files, slow to load | Year/month partitions + a manifest, so `load(start=..., end=...)` fetches only what it needs |
|
|
50
|
+
| Different column names per vendor | Every dataset has a typed schema, snake_case names, units and descriptions in one YAML file |
|
|
51
|
+
| Look-ahead bias in fundamentals | `point_in_time=True` adds `available_date` (report date, or a conservative lag) |
|
|
52
|
+
| Joining stocks to company fundamentals | Links are applied at build time: stock rows carry `gvkey`, fundamentals carry `permno`; `merge_fundamentals()` joins point in time |
|
|
53
|
+
| "Common stocks only" screens | `common_only=True` applies the CIZ-flag equivalent of `shrcd in (10, 11)` |
|
|
54
|
+
| Option chains need context | `mid`, `spread`, `dte`, `spot`, `moneyness` and an interpolated zero rate |
|
|
55
|
+
| US-only tools | Market-neutral design: `market/asset_class/dataset` ids, per-market adapters |
|
|
56
|
+
|
|
57
|
+
## Datasets
|
|
58
|
+
|
|
59
|
+
| Id | Content | Source | Coverage |
|
|
60
|
+
|---|---|---|---|
|
|
61
|
+
| `us/equity/stock_daily` | Daily prices, returns, volume and market cap of all US-listed stocks, incl. delisted, with linked `gvkey` | CRSP | 2015–2025, 23.1M rows |
|
|
62
|
+
| `us/fundamentals/quarterly` | Quarterly financial statements, 60+ items, with linked `permno` | Compustat | 2000–2026, 1.05M rows |
|
|
63
|
+
| `us/fundamentals/annual` | Annual financial statements, 60+ items, with linked `permno` | Compustat | 2000–2026, 255K rows |
|
|
64
|
+
| `us/options/spx_option_prices` | S&P 500 index option quotes, implied vol and greeks | OptionMetrics | 2015–2025, 34.3M rows |
|
|
65
|
+
| `us/options/spx_vol_surface` | Standardized S&P 500 implied volatility surface | OptionMetrics | 2015–2025 |
|
|
66
|
+
| `us/index/spx_daily` | S&P 500 index daily OHLC | OptionMetrics | 2015–2025 |
|
|
67
|
+
| `us/rates/zero_curve` | USD zero-coupon yield curve | OptionMetrics | 2015–2025 |
|
|
68
|
+
| `us/etf/daily_bars`, `bars_5min`, `bars_1min` | SPY, QQQ, IWM, TLT, IEF, GLD bars | Alpaca (local only) | 2016–2026 |
|
|
69
|
+
| `cn_hk/equity/daily_bars`, `ah_pairs` | China A/H dual-listed shares | Yahoo Finance (local only) | 2014–2026 |
|
|
70
|
+
| `cn_hk/fx/daily` | CNY/HKD, USD/HKD exchange rates | Yahoo Finance (local only) | 2014–2026 |
|
|
71
|
+
|
|
72
|
+
Datasets marked *local only* are not on the Hub because of their vendor terms; build them
|
|
73
|
+
from your own copy with `qrd build`. Full column-level documentation: [docs/datasets.md](docs/datasets.md), or
|
|
74
|
+
`qrd.describe("us_stock_daily")` in Python.
|
|
75
|
+
|
|
76
|
+
## Install and get access
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
pip install quant-research-dataset # add [duckdb] for SQL
|
|
80
|
+
hf auth login # once; the dataset is gated
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
1. Request access on the [dataset page](https://huggingface.co/datasets/Hardy0102/quant-research-dataset)
|
|
84
|
+
and confirm you may use the underlying vendor data (see [License](#license)).
|
|
85
|
+
Requests are approved by hand.
|
|
86
|
+
2. Log in with a Hugging Face token (read access is enough).
|
|
87
|
+
3. `import quant_research_dataset as qrd` and load: files download on first use
|
|
88
|
+
and are cached.
|
|
89
|
+
|
|
90
|
+
Already have the files locally? Skip the Hub:
|
|
91
|
+
|
|
92
|
+
```python
|
|
93
|
+
qrd.configure(data_dir="D:/qrd-data") # or set QRD_DATA_DIR
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
## Usage
|
|
97
|
+
|
|
98
|
+
### Discover
|
|
99
|
+
|
|
100
|
+
```python
|
|
101
|
+
qrd.list_datasets(market="us", asset_class="options")
|
|
102
|
+
info = qrd.describe("spx_options") # metadata, coverage, load options, column table
|
|
103
|
+
info["columns"]
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
### Load anything with one function
|
|
107
|
+
|
|
108
|
+
```python
|
|
109
|
+
qrd.load(dataset, ids=None, start=None, end=None, columns=None, filters=None, **options)
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
* `ids`: values of the dataset's id column (PERMNOs, gvkeys, symbols...)
|
|
113
|
+
* `start`, `end`: inclusive date range; partitions outside it are never downloaded
|
|
114
|
+
* `columns`: only these columns are read from disk
|
|
115
|
+
* `filters`: `{"ticker": "AAPL"}`, `{"cp_flag": ["C"]}`, `{"ret": (">", 0.1)}`
|
|
116
|
+
* `**options`: dataset-specific, e.g. `common_only=True` (CRSP), `point_in_time=True` (Compustat)
|
|
117
|
+
|
|
118
|
+
### Shortcuts
|
|
119
|
+
|
|
120
|
+
```python
|
|
121
|
+
qrd.get_prices(market="us", start="2024-01-01", common_only=True) # CRSP
|
|
122
|
+
qrd.get_prices(["600036.SS", "3968.HK"], market="cn_hk") # A/H shares
|
|
123
|
+
qrd.us.get_stock_daily(tickers="NVDA", start="2023-01-01")
|
|
124
|
+
qrd.get_vol_surface("2020-01-01", "2020-12-31", days=30, delta=50, cp_flag="C")
|
|
125
|
+
qrd.get_etf_bars("SPY", start="2024-01-02", end="2024-01-02", freq="1min")
|
|
126
|
+
qrd.cn_hk.ah_premium(start="2024-01-01") # A/H premium per pair
|
|
127
|
+
qrd.zero_rate(["2024-03-15"], [45]) # 45-day zero rate
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
### SQL
|
|
131
|
+
|
|
132
|
+
```python
|
|
133
|
+
con = qrd.duckdb_connect(["us_stock_daily"], start="2024-01-01")
|
|
134
|
+
con.sql("select permno, exp(sum(ln(1 + ret))) - 1 as ret_2024 from us_stock_daily group by 1").df()
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
### Research helpers
|
|
138
|
+
|
|
139
|
+
```python
|
|
140
|
+
port = qrd.quantile_portfolios(panel, signal="mom_12_1", ret="fwd_ret", n=10, weight="mktcap")
|
|
141
|
+
qrd.performance_summary(port["long_short"], periods_per_year=12)
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
## Example: 12-1 momentum on US stocks
|
|
145
|
+
|
|
146
|
+
[examples/momentum.ipynb](examples/momentum.ipynb) runs a classic momentum
|
|
147
|
+
study end to end in a few cells: common stocks above the NYSE 20th size
|
|
148
|
+
percentile, 12-1 month signal, value-weighted deciles, 2016–2025.
|
|
149
|
+
|
|
150
|
+

|
|
151
|
+
|
|
152
|
+
| 2016–2025, monthly | Ann. return | Ann. vol | Sharpe | Max drawdown |
|
|
153
|
+
|---|---|---|---|---|
|
|
154
|
+
| Winners (D10) | 20.9% | 23.1% | 0.94 | −29.8% |
|
|
155
|
+
| Losers (D1) | 8.9% | 30.1% | 0.43 | −55.4% |
|
|
156
|
+
| Winners − losers | 5.5% | 26.1% | 0.33 | −38.9% |
|
|
157
|
+
| Market (CRSP VW) | 13.4% | 15.4% | 0.90 | −24.7% |
|
|
158
|
+
|
|
159
|
+
[examples/quickstart.ipynb](examples/quickstart.ipynb) tours the rest: point-in-time
|
|
160
|
+
fundamentals, an SPX volatility smile, the 30-day ATM vol series, the A/H
|
|
161
|
+
premium and SQL.
|
|
162
|
+
|
|
163
|
+
## Design
|
|
164
|
+
|
|
165
|
+
```
|
|
166
|
+
┌──────────────── registry/<market>/<asset>/<name>.yaml ───────────────┐
|
|
167
|
+
│ id, columns (type/unit/description), keys, partitioning, build rules │
|
|
168
|
+
└───────────────┬────────────────────────────────────┬─────────────────┘
|
|
169
|
+
raw vendor files │ qrd build │ qrd.load()
|
|
170
|
+
(WRDS parquet/CSV, ...) ───┴──► partitioned Parquet + manifest ─┴──► Backend ──► filter push-down ──► adapter ──► DataFrame
|
|
171
|
+
(SHA-256, rows, date ranges) (Hugging Face Hub or local dir) (CRSP screen, PIT dates, option fields)
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
* **Registry** ([`registry/`](src/quant_research_dataset/registry)): one YAML file per
|
|
175
|
+
dataset is the single source of truth. The data dictionary and the Hub dataset
|
|
176
|
+
card are generated from it, so docs never drift from code.
|
|
177
|
+
* **Manifest**: lists each Parquet file with rows, bytes, SHA-256 and date range.
|
|
178
|
+
The client reads it first and fetches only overlapping partitions.
|
|
179
|
+
* **Backends** ([`storage.py`](src/quant_research_dataset/storage.py)): Hugging Face
|
|
180
|
+
Hub (cached, parallel downloads, token-aware) or a local directory.
|
|
181
|
+
* **Adapters** ([`adapters/`](src/quant_research_dataset/adapters)): market-specific
|
|
182
|
+
logic (row filters pushed into the scan, derived columns) kept out of the
|
|
183
|
+
generic loader.
|
|
184
|
+
* **Markets** ([`markets/`](src/quant_research_dataset/markets)): convenience
|
|
185
|
+
functions per market (`qrd.us`, `qrd.cn_hk`).
|
|
186
|
+
|
|
187
|
+
### Adding a dataset
|
|
188
|
+
|
|
189
|
+
1. Put the raw file(s) in the source folder.
|
|
190
|
+
2. Add `registry/<market>/<asset>/<name>.yaml` (see
|
|
191
|
+
[docs/adding-a-dataset.md](docs/adding-a-dataset.md)).
|
|
192
|
+
3. `qrd build --source <raw> --out <prepared> --dataset <id>` and `qrd publish`.
|
|
193
|
+
|
|
194
|
+
No Python changes are needed unless the data needs special load-time logic.
|
|
195
|
+
Private datasets can live outside the package: point `QRD_REGISTRY_PATH` at a
|
|
196
|
+
folder of YAML files.
|
|
197
|
+
|
|
198
|
+
## Development
|
|
199
|
+
|
|
200
|
+
```bash
|
|
201
|
+
git clone https://github.com/Hardy-Z/quant-research-dataset && cd quant-research-dataset
|
|
202
|
+
pip install -e ".[dev]"
|
|
203
|
+
pytest # 68 tests on synthetic vendor-shaped data, no network
|
|
204
|
+
ruff check . && ruff format --check . && mypy
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
Maintainer workflow (build, verify, publish) is in [docs/maintainers.md](docs/maintainers.md).
|
|
208
|
+
|
|
209
|
+
## License
|
|
210
|
+
|
|
211
|
+
The **code** is MIT licensed. The **data** is not: each dataset keeps its
|
|
212
|
+
vendor's terms (shown in `describe()` and the data dictionary). CRSP,
|
|
213
|
+
Compustat and OptionMetrics data are provided for non-commercial academic
|
|
214
|
+
research to approved users only; do not redistribute the files.
|