wmshare-qlib 0.5.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- wmshare_qlib-0.5.0/LICENSE +28 -0
- wmshare_qlib-0.5.0/PKG-INFO +306 -0
- wmshare_qlib-0.5.0/README.md +268 -0
- wmshare_qlib-0.5.0/pyproject.toml +83 -0
- wmshare_qlib-0.5.0/setup.cfg +4 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib/__init__.py +44 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib/cli.py +1134 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib/config.py +284 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib/cross_section.py +102 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib/dump.py +243 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib/estimate.py +113 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib/fetch.py +1478 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib/format.py +118 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib/normalize.py +353 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib/panel.py +193 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib/py.typed +0 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib/qlib_io.py +256 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib/reconcile.py +104 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib/resume.py +311 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib.egg-info/PKG-INFO +306 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib.egg-info/SOURCES.txt +34 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib.egg-info/dependency_links.txt +1 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib.egg-info/entry_points.txt +2 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib.egg-info/requires.txt +11 -0
- wmshare_qlib-0.5.0/src/wmshare_qlib.egg-info/top_level.txt +1 -0
- wmshare_qlib-0.5.0/tests/test_daily_basic.py +221 -0
- wmshare_qlib-0.5.0/tests/test_incremental.py +485 -0
- wmshare_qlib-0.5.0/tests/test_live.py +88 -0
- wmshare_qlib-0.5.0/tests/test_normalize.py +202 -0
- wmshare_qlib-0.5.0/tests/test_p1_ops.py +461 -0
- wmshare_qlib-0.5.0/tests/test_pipeline.py +371 -0
- wmshare_qlib-0.5.0/tests/test_resume.py +323 -0
- wmshare_qlib-0.5.0/tests/test_snapshot_cross_section.py +384 -0
- wmshare_qlib-0.5.0/tests/test_universe.py +268 -0
- wmshare_qlib-0.5.0/tests/test_valuation_window.py +50 -0
- wmshare_qlib-0.5.0/tests/test_vwap.py +101 -0
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
BSD 3-Clause License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026, open-winmale contributors
|
|
4
|
+
|
|
5
|
+
Redistribution and use in source and binary forms, with or without
|
|
6
|
+
modification, are permitted provided that the following conditions are met:
|
|
7
|
+
|
|
8
|
+
1. Redistributions of source code must retain the above copyright notice, this
|
|
9
|
+
list of conditions and the following disclaimer.
|
|
10
|
+
|
|
11
|
+
2. Redistributions in binary form must reproduce the above copyright notice,
|
|
12
|
+
this list of conditions and the following disclaimer in the documentation
|
|
13
|
+
and/or other materials provided with the distribution.
|
|
14
|
+
|
|
15
|
+
3. Neither the name of the copyright holder nor the names of its
|
|
16
|
+
contributors may be used to endorse or promote products derived from
|
|
17
|
+
this software without specific prior written permission.
|
|
18
|
+
|
|
19
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
|
20
|
+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
21
|
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
|
22
|
+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
|
23
|
+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
24
|
+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
|
25
|
+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
|
26
|
+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
|
27
|
+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
28
|
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
@@ -0,0 +1,306 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: wmshare-qlib
|
|
3
|
+
Version: 0.5.0
|
|
4
|
+
Summary: Qlib-native data layer powered by wmshare: A-share history panels, score CSVs and instrument universes in Microsoft Qlib format
|
|
5
|
+
Author: open-winmale contributors
|
|
6
|
+
Maintainer: open-winmale contributors
|
|
7
|
+
License-Expression: BSD-3-Clause
|
|
8
|
+
Project-URL: Homepage, https://github.com/open-winmale/wmshare-qlib
|
|
9
|
+
Project-URL: Repository, https://github.com/open-winmale/wmshare-qlib
|
|
10
|
+
Project-URL: Documentation, https://github.com/open-winmale/wmshare-qlib/blob/main/docs/getting-started.md
|
|
11
|
+
Project-URL: Issues, https://github.com/open-winmale/wmshare-qlib/issues
|
|
12
|
+
Project-URL: Changelog, https://github.com/open-winmale/wmshare-qlib/blob/main/CHANGELOG.md
|
|
13
|
+
Keywords: qlib,quantitative-finance,backtesting,a-share,china-stock-market,factor-research,alpha158,market-data
|
|
14
|
+
Classifier: Development Status :: 4 - Beta
|
|
15
|
+
Classifier: Intended Audience :: Financial and Insurance Industry
|
|
16
|
+
Classifier: Intended Audience :: Science/Research
|
|
17
|
+
Classifier: Natural Language :: English
|
|
18
|
+
Classifier: Operating System :: OS Independent
|
|
19
|
+
Classifier: Programming Language :: Python :: 3
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
22
|
+
Classifier: Topic :: Office/Business :: Financial :: Investment
|
|
23
|
+
Classifier: Topic :: Scientific/Engineering :: Information Analysis
|
|
24
|
+
Classifier: Typing :: Typed
|
|
25
|
+
Requires-Python: >=3.11
|
|
26
|
+
Description-Content-Type: text/markdown
|
|
27
|
+
License-File: LICENSE
|
|
28
|
+
Requires-Dist: wmshare<3,>=2.5.0
|
|
29
|
+
Requires-Dist: pandas>=2.0
|
|
30
|
+
Requires-Dist: pyarrow>=14
|
|
31
|
+
Requires-Dist: numpy>=1.24
|
|
32
|
+
Provides-Extra: qlib
|
|
33
|
+
Requires-Dist: pyqlib>=0.9; extra == "qlib"
|
|
34
|
+
Requires-Dist: numpy<2; extra == "qlib"
|
|
35
|
+
Provides-Extra: test
|
|
36
|
+
Requires-Dist: pytest>=7; extra == "test"
|
|
37
|
+
Dynamic: license-file
|
|
38
|
+
|
|
39
|
+
# wmshare-qlib
|
|
40
|
+
|
|
41
|
+
[](https://github.com/open-winmale/wmshare-qlib/actions/workflows/ci.yml)
|
|
42
|
+
[](https://pypi.org/project/wmshare-qlib/)
|
|
43
|
+
[](https://pypi.org/project/wmshare-qlib/)
|
|
44
|
+
[](LICENSE)
|
|
45
|
+
|
|
46
|
+
**wmshare-qlib turns [wmshare](https://github.com/open-winmale/wmshare) A-share market
|
|
47
|
+
data into Microsoft Qlib-native format** — backtestable `.bin` panels,
|
|
48
|
+
`ScoreFileModel` score CSVs, and instrument universe files — so you can run Qlib's
|
|
49
|
+
research pipeline without writing the format bridge yourself.
|
|
50
|
+
|
|
51
|
+
The adapter is a translation layer, not a data source: it never computes factors and
|
|
52
|
+
never fabricates history. What upstream serves with dates becomes a `.bin` series;
|
|
53
|
+
what upstream only serves as a *current* value is emitted as a dated score snapshot,
|
|
54
|
+
never pasted across history.
|
|
55
|
+
|
|
56
|
+
## Install
|
|
57
|
+
|
|
58
|
+
```bash
|
|
59
|
+
pip install wmshare-qlib # core
|
|
60
|
+
pip install 'wmshare-qlib[qlib]' # + pyqlib, for .bin read-back verification
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Requires **Python 3.11+**. From source:
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
git clone https://github.com/open-winmale/wmshare-qlib.git
|
|
67
|
+
cd wmshare-qlib && pip install -e '.[qlib]'
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
## Quick start A — zero cost, zero credentials (start here)
|
|
71
|
+
|
|
72
|
+
The adapter defaults to a **demo snapshot** (`wmshare.demo_client()`): a fixed
|
|
73
|
+
offline dataset, no network, no EC. Use it to confirm your install, then switch to
|
|
74
|
+
live.
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
# 1) History panel -> Qlib .bin
|
|
78
|
+
wmshare-qlib dump-history --symbols 600519 000858 --n 60 --out ./out/qlib_data
|
|
79
|
+
|
|
80
|
+
# 2) Read one value back with Qlib's own reader
|
|
81
|
+
# Without pyqlib this prints UNAVAILABLE (ok=None) — never a false PASS.
|
|
82
|
+
wmshare-qlib verify --qlib-dir ./out/qlib_data \
|
|
83
|
+
--check SH600519:close:2026-11-16:1453.9
|
|
84
|
+
|
|
85
|
+
# 3) Today's cross section -> score CSV (ScoreFileModel input)
|
|
86
|
+
wmshare-qlib dump-cross-section --as-of 2026-11-16 --out ./out/score
|
|
87
|
+
|
|
88
|
+
# 4) Append one daily PIT snapshot
|
|
89
|
+
wmshare-qlib accumulate --as-of 2026-11-16 --out ./out/cs_accum
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
> **The demo snapshot has fixed contents.** `2026-11-16` and the `1453.9` anchor are
|
|
93
|
+
> values from that snapshot, not market facts; they change when the wmshare SDK
|
|
94
|
+
> re-captures it. Re-derive the date with `tail -1 out/qlib_data/calendars/day.txt`.
|
|
95
|
+
> The demo covers 10 symbols, 120 kline bars and 30 moneyflow bars, stores only `pe`
|
|
96
|
+
> (not `pb/ps/pr`), and is single-threaded — outside those bounds it fails loudly on
|
|
97
|
+
> purpose. See [docs/getting-started.md](docs/getting-started.md) and
|
|
98
|
+
> [docs/faq.md](docs/faq.md).
|
|
99
|
+
|
|
100
|
+
## Quick start B — live (real data, real EC)
|
|
101
|
+
|
|
102
|
+
Live mode needs credentials from <https://open.winmale.com> (我的 → 开放平台 →
|
|
103
|
+
我的应用 → 查看应用信息 — copy both `client_id` and `api-key`):
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
export WMSHARE_CLIENT_ID=your_client_id
|
|
107
|
+
export WMSHARE_CLIENT_SECRET=your_api_key
|
|
108
|
+
# or write them to ~/.wmshare/credentials.env (the SDK reads that file)
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Then:
|
|
112
|
+
|
|
113
|
+
```bash
|
|
114
|
+
# One-time bulk: builds the fragment store under hfq_factor
|
|
115
|
+
wmshare-qlib dump-history --live --symbols 600519 000858 \
|
|
116
|
+
--n 60 --max-ec 20000 --out ./out/qlib_data
|
|
117
|
+
|
|
118
|
+
# Daily incremental after that (fetches only NEW bars, then re-dumps)
|
|
119
|
+
wmshare-qlib update-history --live --max-ec 20000 --out ./out/qlib_data
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
Two rules worth knowing before you type them:
|
|
123
|
+
|
|
124
|
+
- **`--max-ec` is required** on billed paths. Without it the command exits before
|
|
125
|
+
spending anything. It is a budget ceiling, not a hint.
|
|
126
|
+
- **A demo store cannot be grown incrementally.** Demo builds under `qfq_const`;
|
|
127
|
+
appends need an immutable basis, so `update-history` refuses and tells you to
|
|
128
|
+
rebuild with `dump-history --live`. See [docs/faq.md](docs/faq.md#hfq-vs-qfq--which-one-and-why).
|
|
129
|
+
|
|
130
|
+
`--dry-run` exists on `dump-history`, `update-history`, `dump-cross-section` and
|
|
131
|
+
`accumulate`:
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
wmshare-qlib dump-history --live --symbols 600519 --n 60 --dry-run --max-ec 20000
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
## What it produces
|
|
138
|
+
|
|
139
|
+
```
|
|
140
|
+
out/qlib_data/
|
|
141
|
+
├── calendars/day.txt # ascending trading dates
|
|
142
|
+
├── instruments/all.txt # INST\tstart\tend (write-universe adds named ones)
|
|
143
|
+
├── features/sh600519/*.day.bin # Qlib float32 format, first element = start index
|
|
144
|
+
│ # (includes $vwap, derived as amount/volume)
|
|
145
|
+
├── _resume/ # fragment store: manifest + per-symbol parquet
|
|
146
|
+
└── fetch_meta.json # params, EC metering, warnings
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
`format.py` is the single source of truth for the on-disk contract and is
|
|
150
|
+
Qlib-free (pure numpy). Only `qlib_io.py` imports Qlib, and only to read back — in a
|
|
151
|
+
child process, so importing this package never reconfigures your Qlib session.
|
|
152
|
+
|
|
153
|
+
## The three-layer data model
|
|
154
|
+
|
|
155
|
+
Only the first layer is backtestable today:
|
|
156
|
+
|
|
157
|
+
1. **Real history** — kline (OHLCV+amount), moneyflow, `pe/pb/ps/pr` (10y
|
|
158
|
+
window, `config.VALUATION_WINDOW`), plus the materialized `valuation.daily_basic` family
|
|
159
|
+
(`pe_ttm/total_mv/circ_mv/turnover_rate/dv_ttm`) via `--daily-basic`. Dumped
|
|
160
|
+
straight into Qlib `.bin`. `$vwap` is derived at dump time (`amount/volume`;
|
|
161
|
+
the stored volume is already factor-adjusted, so it is consistent with the
|
|
162
|
+
adjusted prices) — Alpha158 can consume it unmodified.
|
|
163
|
+
2. **Today only** — wmshare's ~2,750 precomputed metric codes expose a *current*
|
|
164
|
+
value with **no history-series endpoint**. Emitted as `score/<date>.csv` for
|
|
165
|
+
online scoring, marked `history=false`. Never pasted across past dates.
|
|
166
|
+
3. **Self-built PIT series** — `accumulate` appends one real snapshot per day; the
|
|
167
|
+
series is available from its first captured day and grows one real day at a
|
|
168
|
+
time, with `--min-days` gating promotion to a backtestable panel.
|
|
169
|
+
|
|
170
|
+
Full detail, including why layer 2 cannot be fixed client-side:
|
|
171
|
+
[docs/data-model.md](docs/data-model.md).
|
|
172
|
+
|
|
173
|
+
## Commands
|
|
174
|
+
|
|
175
|
+
| Command | Purpose | Key flags |
|
|
176
|
+
|---|---|---|
|
|
177
|
+
| `dump-history` | fetch history → Qlib `.bin` | `--live --symbols --n/--years --workers --max-ec --resume --adjust --store --daily-basic` |
|
|
178
|
+
| `update-history` | incremental daily tails → reconcile → re-dump (live + hfq only) | `--max-ec` (required) `--tail-overlap --valuation-days` |
|
|
179
|
+
| `dump-cross-section` | today's score CSV | `--via screener/snapshot --idx --max-ec` |
|
|
180
|
+
| `accumulate` | append one daily PIT snapshot (prints availability + promotion status) | same dispatch + `--min-days` |
|
|
181
|
+
| `write-universe` | `instruments/<name>.txt` from index membership or a codes file; `--index` prefers PIT membership episodes when the SDK exposes `index.members_history` (one episode per membership run; dropped members keep their closed episode — survivorship-safe) | `--index/--codes-file --exclude-st --exclude-suspended` |
|
|
182
|
+
| `verify` | read `.bin` back with Qlib's own reader | `--qlib-dir --check inst:field:date:value` |
|
|
183
|
+
| `reconcile-daily-basic` | Gate B: accumulated PIT vs official daily_basic history (100% or fail) | `--accumulated --upstream --tol`, or `--from-upstream --store --max-ec` |
|
|
184
|
+
|
|
185
|
+
`--dry-run` is available on four of them: `dump-history`, `update-history`,
|
|
186
|
+
`dump-cross-section`, `accumulate`.
|
|
187
|
+
|
|
188
|
+
## EC cost model
|
|
189
|
+
|
|
190
|
+
**EC is wmshare's billing unit.** The platform default application has a weekly
|
|
191
|
+
quota refilled every Monday; a self-created additional application holds a one-time
|
|
192
|
+
quota that is not refilled. Demo mode spends nothing.
|
|
193
|
+
|
|
194
|
+
Batching saves **requests**, not **EC** — wmshare bills by symbol and compute. The
|
|
195
|
+
anchors below are our measured values (`config.EC_*`) and the single source of truth
|
|
196
|
+
for estimates:
|
|
197
|
+
|
|
198
|
+
| Call | Formula / anchor |
|
|
199
|
+
|---|---|
|
|
200
|
+
| `kline.bars` | `156 + 67 × served_bars` EC (the live `hfq_factor` path pulls adj + raw, so it doubles) |
|
|
201
|
+
| `moneyflow.bars` | `156 + 16 × n` EC |
|
|
202
|
+
| `valuation.hist_stats` | 712 EC per metric (3y-window fit; the default is now 10y, which bills more) |
|
|
203
|
+
| `kline.snapshot` batch | ≈66–100 EC/票 |
|
|
204
|
+
| `kline.batch_live` | ≈45 EC/票 (percent/ex-div semantics unverified — not used) |
|
|
205
|
+
| `screener.query` | full market × 4 light columns = **204,430 EC**; small probes do NOT predict full-universe cost |
|
|
206
|
+
| `valuation.daily_basic` | bills by rows scanned (not cached): 1-day floor ≈1,707 EC, full history ≈18,317 EC, incremental ≈312 EC/票/日; `update-history --daily-basic` reads are **windowed** from the resume pointer |
|
|
207
|
+
| `margin.query` / `margin.latest` | ≈124 EC/行 / ≈182 EC for the one-row read |
|
|
208
|
+
| range queries (`begin/end`) | **force-pay the anchor window, not the returned rows** — a 49-row window measured 33,688 EC (= 156 + 67×500). Use `--n` for increments |
|
|
209
|
+
|
|
210
|
+
Every fetch result carries `metering` (`requests`, `ec_cost`, `note`) written to
|
|
211
|
+
`fetch_meta.json` / `*.meta.json`. Estimates are estimates; the paid
|
|
212
|
+
`metering.ec_cost` is the only real quote, and `--max-ec` stays the binding ceiling.
|
|
213
|
+
|
|
214
|
+
Sustainable cadence: full-market screener weekly/monthly (~200k EC/query); the daily
|
|
215
|
+
cross-section path is `--via snapshot` over an explicit ~300-name universe
|
|
216
|
+
(~17–30k EC/day). Full-market daily is not viable on any current endpoint.
|
|
217
|
+
|
|
218
|
+
## Data-correctness gates
|
|
219
|
+
|
|
220
|
+
- **Direction** — every series is asserted ascending (2.x contract); a silently
|
|
221
|
+
descending series would flip every rolling computation.
|
|
222
|
+
- **Unit invariants** — amounts in CNY, volumes in shares, ratios dimensionless;
|
|
223
|
+
absurd magnitudes raise rather than flow into a backtest.
|
|
224
|
+
- **Duplicate/conflict** — duplicated dates with identical values are deduplicated
|
|
225
|
+
with a warning; **conflicting** duplicates raise.
|
|
226
|
+
- **Reconciliation gate** — incremental tails must agree with stored history within
|
|
227
|
+
float tolerance. An upstream revision fails loud; history is never silently
|
|
228
|
+
rewritten.
|
|
229
|
+
- **Gate B** — accumulated PIT values are cross-checked against official
|
|
230
|
+
`daily_basic` history, 100% agreement or fail.
|
|
231
|
+
|
|
232
|
+
## Qlib compatibility & low-intrusion design
|
|
233
|
+
|
|
234
|
+
The dumper targets Qlib's own on-disk layout, verified against
|
|
235
|
+
`microsoft/qlib`'s data collector conventions:
|
|
236
|
+
|
|
237
|
+
- `factor = hfq/raw` per date, with adjusted prices stored (live default);
|
|
238
|
+
upstream volume is not adjusted, so it is divided by factor to keep
|
|
239
|
+
`close × volume == amount` invariant across ex-dividend dates.
|
|
240
|
+
- `qlib.init()` runs only in a **child process** during read-back; the parent
|
|
241
|
+
process's `qlib.config.C` is asserted untouched in the test suite.
|
|
242
|
+
- `pyqlib` is an optional extra; the dumper and its self-check need only numpy.
|
|
243
|
+
|
|
244
|
+
## Module map
|
|
245
|
+
|
|
246
|
+
- `config.py` — fields, units, metric-code selection, EC anchors (single source of truth)
|
|
247
|
+
- `normalize.py` — anti-pitfall guards (direction, code mapping, unit, cold cache); `merge_adjusted_klines` (hfq+raw → factor)
|
|
248
|
+
- `fetch.py` — demo/live acquisition + metering + EC-ceiling executor + bounded transient retries
|
|
249
|
+
- `panel.py` — integrity assertions (incl. the factor gate) + wide transform
|
|
250
|
+
- `dump.py` / `format.py` — Qlib-compatible `.bin` dumper and its on-disk contract (Qlib-free)
|
|
251
|
+
- `qlib_io.py` — **the only module that imports Qlib**; child-process read-back + score reader
|
|
252
|
+
- `resume.py` — fragment store (bulk + incremental tails) + manifest with convention guard and reconciliation gate
|
|
253
|
+
- `cross_section.py` — score CSV + PIT accumulation/promotion
|
|
254
|
+
- `estimate.py` — pre-flight EC estimator over the measured anchors (`--dry-run`)
|
|
255
|
+
- `reconcile.py` — Gate B comparison
|
|
256
|
+
- `cli.py` — the seven commands
|
|
257
|
+
|
|
258
|
+
## Tests
|
|
259
|
+
|
|
260
|
+
```bash
|
|
261
|
+
# Tier 1 (mandatory in CI, offline, no credentials, no EC)
|
|
262
|
+
pytest tests -q -m "not live"
|
|
263
|
+
|
|
264
|
+
# Tier 2 (optional) — adds the child-process Qlib read-back cross-check
|
|
265
|
+
pip install 'wmshare-qlib[qlib]'
|
|
266
|
+
pytest tests -q -m qlib
|
|
267
|
+
```
|
|
268
|
+
|
|
269
|
+
Offline tests use `wmshare.demo_client()`, so they need no credentials. Tier 1
|
|
270
|
+
verifies the dumped `.bin` with pure numpy (`format.read_bin_selfcheck`); Tier 2
|
|
271
|
+
additionally reads it back through Qlib itself. Live tests are opt-in and gated by
|
|
272
|
+
`WMSHARE_QLIB_LIVE=1` — see [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
273
|
+
|
|
274
|
+
## What we deliberately do NOT do
|
|
275
|
+
|
|
276
|
+
- Fabricate history from today's snapshot (red line — see layer 2 above).
|
|
277
|
+
- Recompute upstream precomputed metrics client-side and present them as wmshare values.
|
|
278
|
+
- Per-symbol `indicator.get` loops (EC explosion + 429).
|
|
279
|
+
- Run a live metric-code `screener.query` without an explicit EC budget.
|
|
280
|
+
- Call `qlib.init()` in the caller's process (global side effect).
|
|
281
|
+
- Touch the wmshare main repo (separate repo, no version-bump churn).
|
|
282
|
+
|
|
283
|
+
## See also
|
|
284
|
+
|
|
285
|
+
- [Getting started](docs/getting-started.md) — install to first `.bin`, demo then live.
|
|
286
|
+
- [Data model](docs/data-model.md) — the three availability tiers and the EC model.
|
|
287
|
+
- [FAQ](docs/faq.md) — demo boundaries, hfq vs qfq, error reference.
|
|
288
|
+
- [Contributing](CONTRIBUTING.md) — tests, lint, and the red lines.
|
|
289
|
+
- [Changelog](CHANGELOG.md)
|
|
290
|
+
- [wmshare](https://github.com/open-winmale/wmshare) — the data SDK this adapter translates.
|
|
291
|
+
- [wmshare-vnpy](https://github.com/open-winmale/wmshare-vnpy) — the same history panel
|
|
292
|
+
injected into vnpy.alpha.
|
|
293
|
+
- [Examples](examples/) — `dump_to_qlib.py`, `online_score.py`,
|
|
294
|
+
`alpha158_lgbm.py`, `benchmark/`.
|
|
295
|
+
- Design history and forward plans (engineering records, not user docs):
|
|
296
|
+
[docs/internal/](docs/internal/).
|
|
297
|
+
|
|
298
|
+
## Acknowledgements
|
|
299
|
+
|
|
300
|
+
Built on [wmshare](https://github.com/open-winmale/wmshare) (BSD-3-Clause).
|
|
301
|
+
Not affiliated with or endorsed by Microsoft; Qlib/pyqlib is used unmodified as an
|
|
302
|
+
optional verification dependency only.
|
|
303
|
+
|
|
304
|
+
## License
|
|
305
|
+
|
|
306
|
+
BSD-3-Clause — see [LICENSE](LICENSE).
|
|
@@ -0,0 +1,268 @@
|
|
|
1
|
+
# wmshare-qlib
|
|
2
|
+
|
|
3
|
+
[](https://github.com/open-winmale/wmshare-qlib/actions/workflows/ci.yml)
|
|
4
|
+
[](https://pypi.org/project/wmshare-qlib/)
|
|
5
|
+
[](https://pypi.org/project/wmshare-qlib/)
|
|
6
|
+
[](LICENSE)
|
|
7
|
+
|
|
8
|
+
**wmshare-qlib turns [wmshare](https://github.com/open-winmale/wmshare) A-share market
|
|
9
|
+
data into Microsoft Qlib-native format** — backtestable `.bin` panels,
|
|
10
|
+
`ScoreFileModel` score CSVs, and instrument universe files — so you can run Qlib's
|
|
11
|
+
research pipeline without writing the format bridge yourself.
|
|
12
|
+
|
|
13
|
+
The adapter is a translation layer, not a data source: it never computes factors and
|
|
14
|
+
never fabricates history. What upstream serves with dates becomes a `.bin` series;
|
|
15
|
+
what upstream only serves as a *current* value is emitted as a dated score snapshot,
|
|
16
|
+
never pasted across history.
|
|
17
|
+
|
|
18
|
+
## Install
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
pip install wmshare-qlib # core
|
|
22
|
+
pip install 'wmshare-qlib[qlib]' # + pyqlib, for .bin read-back verification
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
Requires **Python 3.11+**. From source:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
git clone https://github.com/open-winmale/wmshare-qlib.git
|
|
29
|
+
cd wmshare-qlib && pip install -e '.[qlib]'
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
## Quick start A — zero cost, zero credentials (start here)
|
|
33
|
+
|
|
34
|
+
The adapter defaults to a **demo snapshot** (`wmshare.demo_client()`): a fixed
|
|
35
|
+
offline dataset, no network, no EC. Use it to confirm your install, then switch to
|
|
36
|
+
live.
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
# 1) History panel -> Qlib .bin
|
|
40
|
+
wmshare-qlib dump-history --symbols 600519 000858 --n 60 --out ./out/qlib_data
|
|
41
|
+
|
|
42
|
+
# 2) Read one value back with Qlib's own reader
|
|
43
|
+
# Without pyqlib this prints UNAVAILABLE (ok=None) — never a false PASS.
|
|
44
|
+
wmshare-qlib verify --qlib-dir ./out/qlib_data \
|
|
45
|
+
--check SH600519:close:2026-11-16:1453.9
|
|
46
|
+
|
|
47
|
+
# 3) Today's cross section -> score CSV (ScoreFileModel input)
|
|
48
|
+
wmshare-qlib dump-cross-section --as-of 2026-11-16 --out ./out/score
|
|
49
|
+
|
|
50
|
+
# 4) Append one daily PIT snapshot
|
|
51
|
+
wmshare-qlib accumulate --as-of 2026-11-16 --out ./out/cs_accum
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
> **The demo snapshot has fixed contents.** `2026-11-16` and the `1453.9` anchor are
|
|
55
|
+
> values from that snapshot, not market facts; they change when the wmshare SDK
|
|
56
|
+
> re-captures it. Re-derive the date with `tail -1 out/qlib_data/calendars/day.txt`.
|
|
57
|
+
> The demo covers 10 symbols, 120 kline bars and 30 moneyflow bars, stores only `pe`
|
|
58
|
+
> (not `pb/ps/pr`), and is single-threaded — outside those bounds it fails loudly on
|
|
59
|
+
> purpose. See [docs/getting-started.md](docs/getting-started.md) and
|
|
60
|
+
> [docs/faq.md](docs/faq.md).
|
|
61
|
+
|
|
62
|
+
## Quick start B — live (real data, real EC)
|
|
63
|
+
|
|
64
|
+
Live mode needs credentials from <https://open.winmale.com> (我的 → 开放平台 →
|
|
65
|
+
我的应用 → 查看应用信息 — copy both `client_id` and `api-key`):
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
export WMSHARE_CLIENT_ID=your_client_id
|
|
69
|
+
export WMSHARE_CLIENT_SECRET=your_api_key
|
|
70
|
+
# or write them to ~/.wmshare/credentials.env (the SDK reads that file)
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
Then:
|
|
74
|
+
|
|
75
|
+
```bash
|
|
76
|
+
# One-time bulk: builds the fragment store under hfq_factor
|
|
77
|
+
wmshare-qlib dump-history --live --symbols 600519 000858 \
|
|
78
|
+
--n 60 --max-ec 20000 --out ./out/qlib_data
|
|
79
|
+
|
|
80
|
+
# Daily incremental after that (fetches only NEW bars, then re-dumps)
|
|
81
|
+
wmshare-qlib update-history --live --max-ec 20000 --out ./out/qlib_data
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Two rules worth knowing before you type them:
|
|
85
|
+
|
|
86
|
+
- **`--max-ec` is required** on billed paths. Without it the command exits before
|
|
87
|
+
spending anything. It is a budget ceiling, not a hint.
|
|
88
|
+
- **A demo store cannot be grown incrementally.** Demo builds under `qfq_const`;
|
|
89
|
+
appends need an immutable basis, so `update-history` refuses and tells you to
|
|
90
|
+
rebuild with `dump-history --live`. See [docs/faq.md](docs/faq.md#hfq-vs-qfq--which-one-and-why).
|
|
91
|
+
|
|
92
|
+
`--dry-run` exists on `dump-history`, `update-history`, `dump-cross-section` and
|
|
93
|
+
`accumulate`:
|
|
94
|
+
|
|
95
|
+
```bash
|
|
96
|
+
wmshare-qlib dump-history --live --symbols 600519 --n 60 --dry-run --max-ec 20000
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
## What it produces
|
|
100
|
+
|
|
101
|
+
```
|
|
102
|
+
out/qlib_data/
|
|
103
|
+
├── calendars/day.txt # ascending trading dates
|
|
104
|
+
├── instruments/all.txt # INST\tstart\tend (write-universe adds named ones)
|
|
105
|
+
├── features/sh600519/*.day.bin # Qlib float32 format, first element = start index
|
|
106
|
+
│ # (includes $vwap, derived as amount/volume)
|
|
107
|
+
├── _resume/ # fragment store: manifest + per-symbol parquet
|
|
108
|
+
└── fetch_meta.json # params, EC metering, warnings
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
`format.py` is the single source of truth for the on-disk contract and is
|
|
112
|
+
Qlib-free (pure numpy). Only `qlib_io.py` imports Qlib, and only to read back — in a
|
|
113
|
+
child process, so importing this package never reconfigures your Qlib session.
|
|
114
|
+
|
|
115
|
+
## The three-layer data model
|
|
116
|
+
|
|
117
|
+
Only the first layer is backtestable today:
|
|
118
|
+
|
|
119
|
+
1. **Real history** — kline (OHLCV+amount), moneyflow, `pe/pb/ps/pr` (10y
|
|
120
|
+
window, `config.VALUATION_WINDOW`), plus the materialized `valuation.daily_basic` family
|
|
121
|
+
(`pe_ttm/total_mv/circ_mv/turnover_rate/dv_ttm`) via `--daily-basic`. Dumped
|
|
122
|
+
straight into Qlib `.bin`. `$vwap` is derived at dump time (`amount/volume`;
|
|
123
|
+
the stored volume is already factor-adjusted, so it is consistent with the
|
|
124
|
+
adjusted prices) — Alpha158 can consume it unmodified.
|
|
125
|
+
2. **Today only** — wmshare's ~2,750 precomputed metric codes expose a *current*
|
|
126
|
+
value with **no history-series endpoint**. Emitted as `score/<date>.csv` for
|
|
127
|
+
online scoring, marked `history=false`. Never pasted across past dates.
|
|
128
|
+
3. **Self-built PIT series** — `accumulate` appends one real snapshot per day; the
|
|
129
|
+
series is available from its first captured day and grows one real day at a
|
|
130
|
+
time, with `--min-days` gating promotion to a backtestable panel.
|
|
131
|
+
|
|
132
|
+
Full detail, including why layer 2 cannot be fixed client-side:
|
|
133
|
+
[docs/data-model.md](docs/data-model.md).
|
|
134
|
+
|
|
135
|
+
## Commands
|
|
136
|
+
|
|
137
|
+
| Command | Purpose | Key flags |
|
|
138
|
+
|---|---|---|
|
|
139
|
+
| `dump-history` | fetch history → Qlib `.bin` | `--live --symbols --n/--years --workers --max-ec --resume --adjust --store --daily-basic` |
|
|
140
|
+
| `update-history` | incremental daily tails → reconcile → re-dump (live + hfq only) | `--max-ec` (required) `--tail-overlap --valuation-days` |
|
|
141
|
+
| `dump-cross-section` | today's score CSV | `--via screener/snapshot --idx --max-ec` |
|
|
142
|
+
| `accumulate` | append one daily PIT snapshot (prints availability + promotion status) | same dispatch + `--min-days` |
|
|
143
|
+
| `write-universe` | `instruments/<name>.txt` from index membership or a codes file; `--index` prefers PIT membership episodes when the SDK exposes `index.members_history` (one episode per membership run; dropped members keep their closed episode — survivorship-safe) | `--index/--codes-file --exclude-st --exclude-suspended` |
|
|
144
|
+
| `verify` | read `.bin` back with Qlib's own reader | `--qlib-dir --check inst:field:date:value` |
|
|
145
|
+
| `reconcile-daily-basic` | Gate B: accumulated PIT vs official daily_basic history (100% or fail) | `--accumulated --upstream --tol`, or `--from-upstream --store --max-ec` |
|
|
146
|
+
|
|
147
|
+
`--dry-run` is available on four of them: `dump-history`, `update-history`,
|
|
148
|
+
`dump-cross-section`, `accumulate`.
|
|
149
|
+
|
|
150
|
+
## EC cost model
|
|
151
|
+
|
|
152
|
+
**EC is wmshare's billing unit.** The platform default application has a weekly
|
|
153
|
+
quota refilled every Monday; a self-created additional application holds a one-time
|
|
154
|
+
quota that is not refilled. Demo mode spends nothing.
|
|
155
|
+
|
|
156
|
+
Batching saves **requests**, not **EC** — wmshare bills by symbol and compute. The
|
|
157
|
+
anchors below are our measured values (`config.EC_*`) and the single source of truth
|
|
158
|
+
for estimates:
|
|
159
|
+
|
|
160
|
+
| Call | Formula / anchor |
|
|
161
|
+
|---|---|
|
|
162
|
+
| `kline.bars` | `156 + 67 × served_bars` EC (the live `hfq_factor` path pulls adj + raw, so it doubles) |
|
|
163
|
+
| `moneyflow.bars` | `156 + 16 × n` EC |
|
|
164
|
+
| `valuation.hist_stats` | 712 EC per metric (3y-window fit; the default is now 10y, which bills more) |
|
|
165
|
+
| `kline.snapshot` batch | ≈66–100 EC/票 |
|
|
166
|
+
| `kline.batch_live` | ≈45 EC/票 (percent/ex-div semantics unverified — not used) |
|
|
167
|
+
| `screener.query` | full market × 4 light columns = **204,430 EC**; small probes do NOT predict full-universe cost |
|
|
168
|
+
| `valuation.daily_basic` | bills by rows scanned (not cached): 1-day floor ≈1,707 EC, full history ≈18,317 EC, incremental ≈312 EC/票/日; `update-history --daily-basic` reads are **windowed** from the resume pointer |
|
|
169
|
+
| `margin.query` / `margin.latest` | ≈124 EC/行 / ≈182 EC for the one-row read |
|
|
170
|
+
| range queries (`begin/end`) | **force-pay the anchor window, not the returned rows** — a 49-row window measured 33,688 EC (= 156 + 67×500). Use `--n` for increments |
|
|
171
|
+
|
|
172
|
+
Every fetch result carries `metering` (`requests`, `ec_cost`, `note`) written to
|
|
173
|
+
`fetch_meta.json` / `*.meta.json`. Estimates are estimates; the paid
|
|
174
|
+
`metering.ec_cost` is the only real quote, and `--max-ec` stays the binding ceiling.
|
|
175
|
+
|
|
176
|
+
Sustainable cadence: full-market screener weekly/monthly (~200k EC/query); the daily
|
|
177
|
+
cross-section path is `--via snapshot` over an explicit ~300-name universe
|
|
178
|
+
(~17–30k EC/day). Full-market daily is not viable on any current endpoint.
|
|
179
|
+
|
|
180
|
+
## Data-correctness gates
|
|
181
|
+
|
|
182
|
+
- **Direction** — every series is asserted ascending (2.x contract); a silently
|
|
183
|
+
descending series would flip every rolling computation.
|
|
184
|
+
- **Unit invariants** — amounts in CNY, volumes in shares, ratios dimensionless;
|
|
185
|
+
absurd magnitudes raise rather than flow into a backtest.
|
|
186
|
+
- **Duplicate/conflict** — duplicated dates with identical values are deduplicated
|
|
187
|
+
with a warning; **conflicting** duplicates raise.
|
|
188
|
+
- **Reconciliation gate** — incremental tails must agree with stored history within
|
|
189
|
+
float tolerance. An upstream revision fails loud; history is never silently
|
|
190
|
+
rewritten.
|
|
191
|
+
- **Gate B** — accumulated PIT values are cross-checked against official
|
|
192
|
+
`daily_basic` history, 100% agreement or fail.
|
|
193
|
+
|
|
194
|
+
## Qlib compatibility & low-intrusion design
|
|
195
|
+
|
|
196
|
+
The dumper targets Qlib's own on-disk layout, verified against
|
|
197
|
+
`microsoft/qlib`'s data collector conventions:
|
|
198
|
+
|
|
199
|
+
- `factor = hfq/raw` per date, with adjusted prices stored (live default);
|
|
200
|
+
upstream volume is not adjusted, so it is divided by factor to keep
|
|
201
|
+
`close × volume == amount` invariant across ex-dividend dates.
|
|
202
|
+
- `qlib.init()` runs only in a **child process** during read-back; the parent
|
|
203
|
+
process's `qlib.config.C` is asserted untouched in the test suite.
|
|
204
|
+
- `pyqlib` is an optional extra; the dumper and its self-check need only numpy.
|
|
205
|
+
|
|
206
|
+
## Module map
|
|
207
|
+
|
|
208
|
+
- `config.py` — fields, units, metric-code selection, EC anchors (single source of truth)
|
|
209
|
+
- `normalize.py` — anti-pitfall guards (direction, code mapping, unit, cold cache); `merge_adjusted_klines` (hfq+raw → factor)
|
|
210
|
+
- `fetch.py` — demo/live acquisition + metering + EC-ceiling executor + bounded transient retries
|
|
211
|
+
- `panel.py` — integrity assertions (incl. the factor gate) + wide transform
|
|
212
|
+
- `dump.py` / `format.py` — Qlib-compatible `.bin` dumper and its on-disk contract (Qlib-free)
|
|
213
|
+
- `qlib_io.py` — **the only module that imports Qlib**; child-process read-back + score reader
|
|
214
|
+
- `resume.py` — fragment store (bulk + incremental tails) + manifest with convention guard and reconciliation gate
|
|
215
|
+
- `cross_section.py` — score CSV + PIT accumulation/promotion
|
|
216
|
+
- `estimate.py` — pre-flight EC estimator over the measured anchors (`--dry-run`)
|
|
217
|
+
- `reconcile.py` — Gate B comparison
|
|
218
|
+
- `cli.py` — the seven commands
|
|
219
|
+
|
|
220
|
+
## Tests
|
|
221
|
+
|
|
222
|
+
```bash
|
|
223
|
+
# Tier 1 (mandatory in CI, offline, no credentials, no EC)
|
|
224
|
+
pytest tests -q -m "not live"
|
|
225
|
+
|
|
226
|
+
# Tier 2 (optional) — adds the child-process Qlib read-back cross-check
|
|
227
|
+
pip install 'wmshare-qlib[qlib]'
|
|
228
|
+
pytest tests -q -m qlib
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
Offline tests use `wmshare.demo_client()`, so they need no credentials. Tier 1
|
|
232
|
+
verifies the dumped `.bin` with pure numpy (`format.read_bin_selfcheck`); Tier 2
|
|
233
|
+
additionally reads it back through Qlib itself. Live tests are opt-in and gated by
|
|
234
|
+
`WMSHARE_QLIB_LIVE=1` — see [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
235
|
+
|
|
236
|
+
## What we deliberately do NOT do
|
|
237
|
+
|
|
238
|
+
- Fabricate history from today's snapshot (red line — see layer 2 above).
|
|
239
|
+
- Recompute upstream precomputed metrics client-side and present them as wmshare values.
|
|
240
|
+
- Per-symbol `indicator.get` loops (EC explosion + 429).
|
|
241
|
+
- Run a live metric-code `screener.query` without an explicit EC budget.
|
|
242
|
+
- Call `qlib.init()` in the caller's process (global side effect).
|
|
243
|
+
- Touch the wmshare main repo (separate repo, no version-bump churn).
|
|
244
|
+
|
|
245
|
+
## See also
|
|
246
|
+
|
|
247
|
+
- [Getting started](docs/getting-started.md) — install to first `.bin`, demo then live.
|
|
248
|
+
- [Data model](docs/data-model.md) — the three availability tiers and the EC model.
|
|
249
|
+
- [FAQ](docs/faq.md) — demo boundaries, hfq vs qfq, error reference.
|
|
250
|
+
- [Contributing](CONTRIBUTING.md) — tests, lint, and the red lines.
|
|
251
|
+
- [Changelog](CHANGELOG.md)
|
|
252
|
+
- [wmshare](https://github.com/open-winmale/wmshare) — the data SDK this adapter translates.
|
|
253
|
+
- [wmshare-vnpy](https://github.com/open-winmale/wmshare-vnpy) — the same history panel
|
|
254
|
+
injected into vnpy.alpha.
|
|
255
|
+
- [Examples](examples/) — `dump_to_qlib.py`, `online_score.py`,
|
|
256
|
+
`alpha158_lgbm.py`, `benchmark/`.
|
|
257
|
+
- Design history and forward plans (engineering records, not user docs):
|
|
258
|
+
[docs/internal/](docs/internal/).
|
|
259
|
+
|
|
260
|
+
## Acknowledgements
|
|
261
|
+
|
|
262
|
+
Built on [wmshare](https://github.com/open-winmale/wmshare) (BSD-3-Clause).
|
|
263
|
+
Not affiliated with or endorsed by Microsoft; Qlib/pyqlib is used unmodified as an
|
|
264
|
+
optional verification dependency only.
|
|
265
|
+
|
|
266
|
+
## License
|
|
267
|
+
|
|
268
|
+
BSD-3-Clause — see [LICENSE](LICENSE).
|