wmshare-qlib 0.5.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (36) hide show
  1. wmshare_qlib-0.5.0/LICENSE +28 -0
  2. wmshare_qlib-0.5.0/PKG-INFO +306 -0
  3. wmshare_qlib-0.5.0/README.md +268 -0
  4. wmshare_qlib-0.5.0/pyproject.toml +83 -0
  5. wmshare_qlib-0.5.0/setup.cfg +4 -0
  6. wmshare_qlib-0.5.0/src/wmshare_qlib/__init__.py +44 -0
  7. wmshare_qlib-0.5.0/src/wmshare_qlib/cli.py +1134 -0
  8. wmshare_qlib-0.5.0/src/wmshare_qlib/config.py +284 -0
  9. wmshare_qlib-0.5.0/src/wmshare_qlib/cross_section.py +102 -0
  10. wmshare_qlib-0.5.0/src/wmshare_qlib/dump.py +243 -0
  11. wmshare_qlib-0.5.0/src/wmshare_qlib/estimate.py +113 -0
  12. wmshare_qlib-0.5.0/src/wmshare_qlib/fetch.py +1478 -0
  13. wmshare_qlib-0.5.0/src/wmshare_qlib/format.py +118 -0
  14. wmshare_qlib-0.5.0/src/wmshare_qlib/normalize.py +353 -0
  15. wmshare_qlib-0.5.0/src/wmshare_qlib/panel.py +193 -0
  16. wmshare_qlib-0.5.0/src/wmshare_qlib/py.typed +0 -0
  17. wmshare_qlib-0.5.0/src/wmshare_qlib/qlib_io.py +256 -0
  18. wmshare_qlib-0.5.0/src/wmshare_qlib/reconcile.py +104 -0
  19. wmshare_qlib-0.5.0/src/wmshare_qlib/resume.py +311 -0
  20. wmshare_qlib-0.5.0/src/wmshare_qlib.egg-info/PKG-INFO +306 -0
  21. wmshare_qlib-0.5.0/src/wmshare_qlib.egg-info/SOURCES.txt +34 -0
  22. wmshare_qlib-0.5.0/src/wmshare_qlib.egg-info/dependency_links.txt +1 -0
  23. wmshare_qlib-0.5.0/src/wmshare_qlib.egg-info/entry_points.txt +2 -0
  24. wmshare_qlib-0.5.0/src/wmshare_qlib.egg-info/requires.txt +11 -0
  25. wmshare_qlib-0.5.0/src/wmshare_qlib.egg-info/top_level.txt +1 -0
  26. wmshare_qlib-0.5.0/tests/test_daily_basic.py +221 -0
  27. wmshare_qlib-0.5.0/tests/test_incremental.py +485 -0
  28. wmshare_qlib-0.5.0/tests/test_live.py +88 -0
  29. wmshare_qlib-0.5.0/tests/test_normalize.py +202 -0
  30. wmshare_qlib-0.5.0/tests/test_p1_ops.py +461 -0
  31. wmshare_qlib-0.5.0/tests/test_pipeline.py +371 -0
  32. wmshare_qlib-0.5.0/tests/test_resume.py +323 -0
  33. wmshare_qlib-0.5.0/tests/test_snapshot_cross_section.py +384 -0
  34. wmshare_qlib-0.5.0/tests/test_universe.py +268 -0
  35. wmshare_qlib-0.5.0/tests/test_valuation_window.py +50 -0
  36. wmshare_qlib-0.5.0/tests/test_vwap.py +101 -0
@@ -0,0 +1,28 @@
1
+ BSD 3-Clause License
2
+
3
+ Copyright (c) 2026, open-winmale contributors
4
+
5
+ Redistribution and use in source and binary forms, with or without
6
+ modification, are permitted provided that the following conditions are met:
7
+
8
+ 1. Redistributions of source code must retain the above copyright notice, this
9
+ list of conditions and the following disclaimer.
10
+
11
+ 2. Redistributions in binary form must reproduce the above copyright notice,
12
+ this list of conditions and the following disclaimer in the documentation
13
+ and/or other materials provided with the distribution.
14
+
15
+ 3. Neither the name of the copyright holder nor the names of its
16
+ contributors may be used to endorse or promote products derived from
17
+ this software without specific prior written permission.
18
+
19
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
20
+ AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
21
+ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
22
+ DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
23
+ FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
24
+ DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
25
+ SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
26
+ CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
27
+ OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
28
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
@@ -0,0 +1,306 @@
1
+ Metadata-Version: 2.4
2
+ Name: wmshare-qlib
3
+ Version: 0.5.0
4
+ Summary: Qlib-native data layer powered by wmshare: A-share history panels, score CSVs and instrument universes in Microsoft Qlib format
5
+ Author: open-winmale contributors
6
+ Maintainer: open-winmale contributors
7
+ License-Expression: BSD-3-Clause
8
+ Project-URL: Homepage, https://github.com/open-winmale/wmshare-qlib
9
+ Project-URL: Repository, https://github.com/open-winmale/wmshare-qlib
10
+ Project-URL: Documentation, https://github.com/open-winmale/wmshare-qlib/blob/main/docs/getting-started.md
11
+ Project-URL: Issues, https://github.com/open-winmale/wmshare-qlib/issues
12
+ Project-URL: Changelog, https://github.com/open-winmale/wmshare-qlib/blob/main/CHANGELOG.md
13
+ Keywords: qlib,quantitative-finance,backtesting,a-share,china-stock-market,factor-research,alpha158,market-data
14
+ Classifier: Development Status :: 4 - Beta
15
+ Classifier: Intended Audience :: Financial and Insurance Industry
16
+ Classifier: Intended Audience :: Science/Research
17
+ Classifier: Natural Language :: English
18
+ Classifier: Operating System :: OS Independent
19
+ Classifier: Programming Language :: Python :: 3
20
+ Classifier: Programming Language :: Python :: 3.11
21
+ Classifier: Programming Language :: Python :: 3.12
22
+ Classifier: Topic :: Office/Business :: Financial :: Investment
23
+ Classifier: Topic :: Scientific/Engineering :: Information Analysis
24
+ Classifier: Typing :: Typed
25
+ Requires-Python: >=3.11
26
+ Description-Content-Type: text/markdown
27
+ License-File: LICENSE
28
+ Requires-Dist: wmshare<3,>=2.5.0
29
+ Requires-Dist: pandas>=2.0
30
+ Requires-Dist: pyarrow>=14
31
+ Requires-Dist: numpy>=1.24
32
+ Provides-Extra: qlib
33
+ Requires-Dist: pyqlib>=0.9; extra == "qlib"
34
+ Requires-Dist: numpy<2; extra == "qlib"
35
+ Provides-Extra: test
36
+ Requires-Dist: pytest>=7; extra == "test"
37
+ Dynamic: license-file
38
+
39
+ # wmshare-qlib
40
+
41
+ [![CI](https://github.com/open-winmale/wmshare-qlib/actions/workflows/ci.yml/badge.svg)](https://github.com/open-winmale/wmshare-qlib/actions/workflows/ci.yml)
42
+ [![PyPI](https://img.shields.io/pypi/v/wmshare-qlib.svg)](https://pypi.org/project/wmshare-qlib/)
43
+ [![Python](https://img.shields.io/pypi/pyversions/wmshare-qlib.svg)](https://pypi.org/project/wmshare-qlib/)
44
+ [![License](https://img.shields.io/badge/license-BSD--3--Clause-blue.svg)](LICENSE)
45
+
46
+ **wmshare-qlib turns [wmshare](https://github.com/open-winmale/wmshare) A-share market
47
+ data into Microsoft Qlib-native format** — backtestable `.bin` panels,
48
+ `ScoreFileModel` score CSVs, and instrument universe files — so you can run Qlib's
49
+ research pipeline without writing the format bridge yourself.
50
+
51
+ The adapter is a translation layer, not a data source: it never computes factors and
52
+ never fabricates history. What upstream serves with dates becomes a `.bin` series;
53
+ what upstream only serves as a *current* value is emitted as a dated score snapshot,
54
+ never pasted across history.
55
+
56
+ ## Install
57
+
58
+ ```bash
59
+ pip install wmshare-qlib # core
60
+ pip install 'wmshare-qlib[qlib]' # + pyqlib, for .bin read-back verification
61
+ ```
62
+
63
+ Requires **Python 3.11+**. From source:
64
+
65
+ ```bash
66
+ git clone https://github.com/open-winmale/wmshare-qlib.git
67
+ cd wmshare-qlib && pip install -e '.[qlib]'
68
+ ```
69
+
70
+ ## Quick start A — zero cost, zero credentials (start here)
71
+
72
+ The adapter defaults to a **demo snapshot** (`wmshare.demo_client()`): a fixed
73
+ offline dataset, no network, no EC. Use it to confirm your install, then switch to
74
+ live.
75
+
76
+ ```bash
77
+ # 1) History panel -> Qlib .bin
78
+ wmshare-qlib dump-history --symbols 600519 000858 --n 60 --out ./out/qlib_data
79
+
80
+ # 2) Read one value back with Qlib's own reader
81
+ # Without pyqlib this prints UNAVAILABLE (ok=None) — never a false PASS.
82
+ wmshare-qlib verify --qlib-dir ./out/qlib_data \
83
+ --check SH600519:close:2026-11-16:1453.9
84
+
85
+ # 3) Today's cross section -> score CSV (ScoreFileModel input)
86
+ wmshare-qlib dump-cross-section --as-of 2026-11-16 --out ./out/score
87
+
88
+ # 4) Append one daily PIT snapshot
89
+ wmshare-qlib accumulate --as-of 2026-11-16 --out ./out/cs_accum
90
+ ```
91
+
92
+ > **The demo snapshot has fixed contents.** `2026-11-16` and the `1453.9` anchor are
93
+ > values from that snapshot, not market facts; they change when the wmshare SDK
94
+ > re-captures it. Re-derive the date with `tail -1 out/qlib_data/calendars/day.txt`.
95
+ > The demo covers 10 symbols, 120 kline bars and 30 moneyflow bars, stores only `pe`
96
+ > (not `pb/ps/pr`), and is single-threaded — outside those bounds it fails loudly on
97
+ > purpose. See [docs/getting-started.md](docs/getting-started.md) and
98
+ > [docs/faq.md](docs/faq.md).
99
+
100
+ ## Quick start B — live (real data, real EC)
101
+
102
+ Live mode needs credentials from <https://open.winmale.com> (我的 → 开放平台 →
103
+ 我的应用 → 查看应用信息 — copy both `client_id` and `api-key`):
104
+
105
+ ```bash
106
+ export WMSHARE_CLIENT_ID=your_client_id
107
+ export WMSHARE_CLIENT_SECRET=your_api_key
108
+ # or write them to ~/.wmshare/credentials.env (the SDK reads that file)
109
+ ```
110
+
111
+ Then:
112
+
113
+ ```bash
114
+ # One-time bulk: builds the fragment store under hfq_factor
115
+ wmshare-qlib dump-history --live --symbols 600519 000858 \
116
+ --n 60 --max-ec 20000 --out ./out/qlib_data
117
+
118
+ # Daily incremental after that (fetches only NEW bars, then re-dumps)
119
+ wmshare-qlib update-history --live --max-ec 20000 --out ./out/qlib_data
120
+ ```
121
+
122
+ Two rules worth knowing before you type them:
123
+
124
+ - **`--max-ec` is required** on billed paths. Without it the command exits before
125
+ spending anything. It is a budget ceiling, not a hint.
126
+ - **A demo store cannot be grown incrementally.** Demo builds under `qfq_const`;
127
+ appends need an immutable basis, so `update-history` refuses and tells you to
128
+ rebuild with `dump-history --live`. See [docs/faq.md](docs/faq.md#hfq-vs-qfq--which-one-and-why).
129
+
130
+ `--dry-run` exists on `dump-history`, `update-history`, `dump-cross-section` and
131
+ `accumulate`:
132
+
133
+ ```bash
134
+ wmshare-qlib dump-history --live --symbols 600519 --n 60 --dry-run --max-ec 20000
135
+ ```
136
+
137
+ ## What it produces
138
+
139
+ ```
140
+ out/qlib_data/
141
+ ├── calendars/day.txt # ascending trading dates
142
+ ├── instruments/all.txt # INST\tstart\tend (write-universe adds named ones)
143
+ ├── features/sh600519/*.day.bin # Qlib float32 format, first element = start index
144
+ │ # (includes $vwap, derived as amount/volume)
145
+ ├── _resume/ # fragment store: manifest + per-symbol parquet
146
+ └── fetch_meta.json # params, EC metering, warnings
147
+ ```
148
+
149
+ `format.py` is the single source of truth for the on-disk contract and is
150
+ Qlib-free (pure numpy). Only `qlib_io.py` imports Qlib, and only to read back — in a
151
+ child process, so importing this package never reconfigures your Qlib session.
152
+
153
+ ## The three-layer data model
154
+
155
+ Only the first layer is backtestable today:
156
+
157
+ 1. **Real history** — kline (OHLCV+amount), moneyflow, `pe/pb/ps/pr` (10y
158
+ window, `config.VALUATION_WINDOW`), plus the materialized `valuation.daily_basic` family
159
+ (`pe_ttm/total_mv/circ_mv/turnover_rate/dv_ttm`) via `--daily-basic`. Dumped
160
+ straight into Qlib `.bin`. `$vwap` is derived at dump time (`amount/volume`;
161
+ the stored volume is already factor-adjusted, so it is consistent with the
162
+ adjusted prices) — Alpha158 can consume it unmodified.
163
+ 2. **Today only** — wmshare's ~2,750 precomputed metric codes expose a *current*
164
+ value with **no history-series endpoint**. Emitted as `score/<date>.csv` for
165
+ online scoring, marked `history=false`. Never pasted across past dates.
166
+ 3. **Self-built PIT series** — `accumulate` appends one real snapshot per day; the
167
+ series is available from its first captured day and grows one real day at a
168
+ time, with `--min-days` gating promotion to a backtestable panel.
169
+
170
+ Full detail, including why layer 2 cannot be fixed client-side:
171
+ [docs/data-model.md](docs/data-model.md).
172
+
173
+ ## Commands
174
+
175
+ | Command | Purpose | Key flags |
176
+ |---|---|---|
177
+ | `dump-history` | fetch history → Qlib `.bin` | `--live --symbols --n/--years --workers --max-ec --resume --adjust --store --daily-basic` |
178
+ | `update-history` | incremental daily tails → reconcile → re-dump (live + hfq only) | `--max-ec` (required) `--tail-overlap --valuation-days` |
179
+ | `dump-cross-section` | today's score CSV | `--via screener/snapshot --idx --max-ec` |
180
+ | `accumulate` | append one daily PIT snapshot (prints availability + promotion status) | same dispatch + `--min-days` |
181
+ | `write-universe` | `instruments/<name>.txt` from index membership or a codes file; `--index` prefers PIT membership episodes when the SDK exposes `index.members_history` (one episode per membership run; dropped members keep their closed episode — survivorship-safe) | `--index/--codes-file --exclude-st --exclude-suspended` |
182
+ | `verify` | read `.bin` back with Qlib's own reader | `--qlib-dir --check inst:field:date:value` |
183
+ | `reconcile-daily-basic` | Gate B: accumulated PIT vs official daily_basic history (100% or fail) | `--accumulated --upstream --tol`, or `--from-upstream --store --max-ec` |
184
+
185
+ `--dry-run` is available on four of them: `dump-history`, `update-history`,
186
+ `dump-cross-section`, `accumulate`.
187
+
188
+ ## EC cost model
189
+
190
+ **EC is wmshare's billing unit.** The platform default application has a weekly
191
+ quota refilled every Monday; a self-created additional application holds a one-time
192
+ quota that is not refilled. Demo mode spends nothing.
193
+
194
+ Batching saves **requests**, not **EC** — wmshare bills by symbol and compute. The
195
+ anchors below are our measured values (`config.EC_*`) and the single source of truth
196
+ for estimates:
197
+
198
+ | Call | Formula / anchor |
199
+ |---|---|
200
+ | `kline.bars` | `156 + 67 × served_bars` EC (the live `hfq_factor` path pulls adj + raw, so it doubles) |
201
+ | `moneyflow.bars` | `156 + 16 × n` EC |
202
+ | `valuation.hist_stats` | 712 EC per metric (3y-window fit; the default is now 10y, which bills more) |
203
+ | `kline.snapshot` batch | ≈66–100 EC/票 |
204
+ | `kline.batch_live` | ≈45 EC/票 (percent/ex-div semantics unverified — not used) |
205
+ | `screener.query` | full market × 4 light columns = **204,430 EC**; small probes do NOT predict full-universe cost |
206
+ | `valuation.daily_basic` | bills by rows scanned (not cached): 1-day floor ≈1,707 EC, full history ≈18,317 EC, incremental ≈312 EC/票/日; `update-history --daily-basic` reads are **windowed** from the resume pointer |
207
+ | `margin.query` / `margin.latest` | ≈124 EC/行 / ≈182 EC for the one-row read |
208
+ | range queries (`begin/end`) | **force-pay the anchor window, not the returned rows** — a 49-row window measured 33,688 EC (= 156 + 67×500). Use `--n` for increments |
209
+
210
+ Every fetch result carries `metering` (`requests`, `ec_cost`, `note`) written to
211
+ `fetch_meta.json` / `*.meta.json`. Estimates are estimates; the paid
212
+ `metering.ec_cost` is the only real quote, and `--max-ec` stays the binding ceiling.
213
+
214
+ Sustainable cadence: full-market screener weekly/monthly (~200k EC/query); the daily
215
+ cross-section path is `--via snapshot` over an explicit ~300-name universe
216
+ (~17–30k EC/day). Full-market daily is not viable on any current endpoint.
217
+
218
+ ## Data-correctness gates
219
+
220
+ - **Direction** — every series is asserted ascending (2.x contract); a silently
221
+ descending series would flip every rolling computation.
222
+ - **Unit invariants** — amounts in CNY, volumes in shares, ratios dimensionless;
223
+ absurd magnitudes raise rather than flow into a backtest.
224
+ - **Duplicate/conflict** — duplicated dates with identical values are deduplicated
225
+ with a warning; **conflicting** duplicates raise.
226
+ - **Reconciliation gate** — incremental tails must agree with stored history within
227
+ float tolerance. An upstream revision fails loud; history is never silently
228
+ rewritten.
229
+ - **Gate B** — accumulated PIT values are cross-checked against official
230
+ `daily_basic` history, 100% agreement or fail.
231
+
232
+ ## Qlib compatibility & low-intrusion design
233
+
234
+ The dumper targets Qlib's own on-disk layout, verified against
235
+ `microsoft/qlib`'s data collector conventions:
236
+
237
+ - `factor = hfq/raw` per date, with adjusted prices stored (live default);
238
+ upstream volume is not adjusted, so it is divided by factor to keep
239
+ `close × volume == amount` invariant across ex-dividend dates.
240
+ - `qlib.init()` runs only in a **child process** during read-back; the parent
241
+ process's `qlib.config.C` is asserted untouched in the test suite.
242
+ - `pyqlib` is an optional extra; the dumper and its self-check need only numpy.
243
+
244
+ ## Module map
245
+
246
+ - `config.py` — fields, units, metric-code selection, EC anchors (single source of truth)
247
+ - `normalize.py` — anti-pitfall guards (direction, code mapping, unit, cold cache); `merge_adjusted_klines` (hfq+raw → factor)
248
+ - `fetch.py` — demo/live acquisition + metering + EC-ceiling executor + bounded transient retries
249
+ - `panel.py` — integrity assertions (incl. the factor gate) + wide transform
250
+ - `dump.py` / `format.py` — Qlib-compatible `.bin` dumper and its on-disk contract (Qlib-free)
251
+ - `qlib_io.py` — **the only module that imports Qlib**; child-process read-back + score reader
252
+ - `resume.py` — fragment store (bulk + incremental tails) + manifest with convention guard and reconciliation gate
253
+ - `cross_section.py` — score CSV + PIT accumulation/promotion
254
+ - `estimate.py` — pre-flight EC estimator over the measured anchors (`--dry-run`)
255
+ - `reconcile.py` — Gate B comparison
256
+ - `cli.py` — the seven commands
257
+
258
+ ## Tests
259
+
260
+ ```bash
261
+ # Tier 1 (mandatory in CI, offline, no credentials, no EC)
262
+ pytest tests -q -m "not live"
263
+
264
+ # Tier 2 (optional) — adds the child-process Qlib read-back cross-check
265
+ pip install 'wmshare-qlib[qlib]'
266
+ pytest tests -q -m qlib
267
+ ```
268
+
269
+ Offline tests use `wmshare.demo_client()`, so they need no credentials. Tier 1
270
+ verifies the dumped `.bin` with pure numpy (`format.read_bin_selfcheck`); Tier 2
271
+ additionally reads it back through Qlib itself. Live tests are opt-in and gated by
272
+ `WMSHARE_QLIB_LIVE=1` — see [CONTRIBUTING.md](CONTRIBUTING.md).
273
+
274
+ ## What we deliberately do NOT do
275
+
276
+ - Fabricate history from today's snapshot (red line — see layer 2 above).
277
+ - Recompute upstream precomputed metrics client-side and present them as wmshare values.
278
+ - Per-symbol `indicator.get` loops (EC explosion + 429).
279
+ - Run a live metric-code `screener.query` without an explicit EC budget.
280
+ - Call `qlib.init()` in the caller's process (global side effect).
281
+ - Touch the wmshare main repo (separate repo, no version-bump churn).
282
+
283
+ ## See also
284
+
285
+ - [Getting started](docs/getting-started.md) — install to first `.bin`, demo then live.
286
+ - [Data model](docs/data-model.md) — the three availability tiers and the EC model.
287
+ - [FAQ](docs/faq.md) — demo boundaries, hfq vs qfq, error reference.
288
+ - [Contributing](CONTRIBUTING.md) — tests, lint, and the red lines.
289
+ - [Changelog](CHANGELOG.md)
290
+ - [wmshare](https://github.com/open-winmale/wmshare) — the data SDK this adapter translates.
291
+ - [wmshare-vnpy](https://github.com/open-winmale/wmshare-vnpy) — the same history panel
292
+ injected into vnpy.alpha.
293
+ - [Examples](examples/) — `dump_to_qlib.py`, `online_score.py`,
294
+ `alpha158_lgbm.py`, `benchmark/`.
295
+ - Design history and forward plans (engineering records, not user docs):
296
+ [docs/internal/](docs/internal/).
297
+
298
+ ## Acknowledgements
299
+
300
+ Built on [wmshare](https://github.com/open-winmale/wmshare) (BSD-3-Clause).
301
+ Not affiliated with or endorsed by Microsoft; Qlib/pyqlib is used unmodified as an
302
+ optional verification dependency only.
303
+
304
+ ## License
305
+
306
+ BSD-3-Clause — see [LICENSE](LICENSE).
@@ -0,0 +1,268 @@
1
+ # wmshare-qlib
2
+
3
+ [![CI](https://github.com/open-winmale/wmshare-qlib/actions/workflows/ci.yml/badge.svg)](https://github.com/open-winmale/wmshare-qlib/actions/workflows/ci.yml)
4
+ [![PyPI](https://img.shields.io/pypi/v/wmshare-qlib.svg)](https://pypi.org/project/wmshare-qlib/)
5
+ [![Python](https://img.shields.io/pypi/pyversions/wmshare-qlib.svg)](https://pypi.org/project/wmshare-qlib/)
6
+ [![License](https://img.shields.io/badge/license-BSD--3--Clause-blue.svg)](LICENSE)
7
+
8
+ **wmshare-qlib turns [wmshare](https://github.com/open-winmale/wmshare) A-share market
9
+ data into Microsoft Qlib-native format** — backtestable `.bin` panels,
10
+ `ScoreFileModel` score CSVs, and instrument universe files — so you can run Qlib's
11
+ research pipeline without writing the format bridge yourself.
12
+
13
+ The adapter is a translation layer, not a data source: it never computes factors and
14
+ never fabricates history. What upstream serves with dates becomes a `.bin` series;
15
+ what upstream only serves as a *current* value is emitted as a dated score snapshot,
16
+ never pasted across history.
17
+
18
+ ## Install
19
+
20
+ ```bash
21
+ pip install wmshare-qlib # core
22
+ pip install 'wmshare-qlib[qlib]' # + pyqlib, for .bin read-back verification
23
+ ```
24
+
25
+ Requires **Python 3.11+**. From source:
26
+
27
+ ```bash
28
+ git clone https://github.com/open-winmale/wmshare-qlib.git
29
+ cd wmshare-qlib && pip install -e '.[qlib]'
30
+ ```
31
+
32
+ ## Quick start A — zero cost, zero credentials (start here)
33
+
34
+ The adapter defaults to a **demo snapshot** (`wmshare.demo_client()`): a fixed
35
+ offline dataset, no network, no EC. Use it to confirm your install, then switch to
36
+ live.
37
+
38
+ ```bash
39
+ # 1) History panel -> Qlib .bin
40
+ wmshare-qlib dump-history --symbols 600519 000858 --n 60 --out ./out/qlib_data
41
+
42
+ # 2) Read one value back with Qlib's own reader
43
+ # Without pyqlib this prints UNAVAILABLE (ok=None) — never a false PASS.
44
+ wmshare-qlib verify --qlib-dir ./out/qlib_data \
45
+ --check SH600519:close:2026-11-16:1453.9
46
+
47
+ # 3) Today's cross section -> score CSV (ScoreFileModel input)
48
+ wmshare-qlib dump-cross-section --as-of 2026-11-16 --out ./out/score
49
+
50
+ # 4) Append one daily PIT snapshot
51
+ wmshare-qlib accumulate --as-of 2026-11-16 --out ./out/cs_accum
52
+ ```
53
+
54
+ > **The demo snapshot has fixed contents.** `2026-11-16` and the `1453.9` anchor are
55
+ > values from that snapshot, not market facts; they change when the wmshare SDK
56
+ > re-captures it. Re-derive the date with `tail -1 out/qlib_data/calendars/day.txt`.
57
+ > The demo covers 10 symbols, 120 kline bars and 30 moneyflow bars, stores only `pe`
58
+ > (not `pb/ps/pr`), and is single-threaded — outside those bounds it fails loudly on
59
+ > purpose. See [docs/getting-started.md](docs/getting-started.md) and
60
+ > [docs/faq.md](docs/faq.md).
61
+
62
+ ## Quick start B — live (real data, real EC)
63
+
64
+ Live mode needs credentials from <https://open.winmale.com> (我的 → 开放平台 →
65
+ 我的应用 → 查看应用信息 — copy both `client_id` and `api-key`):
66
+
67
+ ```bash
68
+ export WMSHARE_CLIENT_ID=your_client_id
69
+ export WMSHARE_CLIENT_SECRET=your_api_key
70
+ # or write them to ~/.wmshare/credentials.env (the SDK reads that file)
71
+ ```
72
+
73
+ Then:
74
+
75
+ ```bash
76
+ # One-time bulk: builds the fragment store under hfq_factor
77
+ wmshare-qlib dump-history --live --symbols 600519 000858 \
78
+ --n 60 --max-ec 20000 --out ./out/qlib_data
79
+
80
+ # Daily incremental after that (fetches only NEW bars, then re-dumps)
81
+ wmshare-qlib update-history --live --max-ec 20000 --out ./out/qlib_data
82
+ ```
83
+
84
+ Two rules worth knowing before you type them:
85
+
86
+ - **`--max-ec` is required** on billed paths. Without it the command exits before
87
+ spending anything. It is a budget ceiling, not a hint.
88
+ - **A demo store cannot be grown incrementally.** Demo builds under `qfq_const`;
89
+ appends need an immutable basis, so `update-history` refuses and tells you to
90
+ rebuild with `dump-history --live`. See [docs/faq.md](docs/faq.md#hfq-vs-qfq--which-one-and-why).
91
+
92
+ `--dry-run` exists on `dump-history`, `update-history`, `dump-cross-section` and
93
+ `accumulate`:
94
+
95
+ ```bash
96
+ wmshare-qlib dump-history --live --symbols 600519 --n 60 --dry-run --max-ec 20000
97
+ ```
98
+
99
+ ## What it produces
100
+
101
+ ```
102
+ out/qlib_data/
103
+ ├── calendars/day.txt # ascending trading dates
104
+ ├── instruments/all.txt # INST\tstart\tend (write-universe adds named ones)
105
+ ├── features/sh600519/*.day.bin # Qlib float32 format, first element = start index
106
+ │ # (includes $vwap, derived as amount/volume)
107
+ ├── _resume/ # fragment store: manifest + per-symbol parquet
108
+ └── fetch_meta.json # params, EC metering, warnings
109
+ ```
110
+
111
+ `format.py` is the single source of truth for the on-disk contract and is
112
+ Qlib-free (pure numpy). Only `qlib_io.py` imports Qlib, and only to read back — in a
113
+ child process, so importing this package never reconfigures your Qlib session.
114
+
115
+ ## The three-layer data model
116
+
117
+ Only the first layer is backtestable today:
118
+
119
+ 1. **Real history** — kline (OHLCV+amount), moneyflow, `pe/pb/ps/pr` (10y
120
+ window, `config.VALUATION_WINDOW`), plus the materialized `valuation.daily_basic` family
121
+ (`pe_ttm/total_mv/circ_mv/turnover_rate/dv_ttm`) via `--daily-basic`. Dumped
122
+ straight into Qlib `.bin`. `$vwap` is derived at dump time (`amount/volume`;
123
+ the stored volume is already factor-adjusted, so it is consistent with the
124
+ adjusted prices) — Alpha158 can consume it unmodified.
125
+ 2. **Today only** — wmshare's ~2,750 precomputed metric codes expose a *current*
126
+ value with **no history-series endpoint**. Emitted as `score/<date>.csv` for
127
+ online scoring, marked `history=false`. Never pasted across past dates.
128
+ 3. **Self-built PIT series** — `accumulate` appends one real snapshot per day; the
129
+ series is available from its first captured day and grows one real day at a
130
+ time, with `--min-days` gating promotion to a backtestable panel.
131
+
132
+ Full detail, including why layer 2 cannot be fixed client-side:
133
+ [docs/data-model.md](docs/data-model.md).
134
+
135
+ ## Commands
136
+
137
+ | Command | Purpose | Key flags |
138
+ |---|---|---|
139
+ | `dump-history` | fetch history → Qlib `.bin` | `--live --symbols --n/--years --workers --max-ec --resume --adjust --store --daily-basic` |
140
+ | `update-history` | incremental daily tails → reconcile → re-dump (live + hfq only) | `--max-ec` (required) `--tail-overlap --valuation-days` |
141
+ | `dump-cross-section` | today's score CSV | `--via screener/snapshot --idx --max-ec` |
142
+ | `accumulate` | append one daily PIT snapshot (prints availability + promotion status) | same dispatch + `--min-days` |
143
+ | `write-universe` | `instruments/<name>.txt` from index membership or a codes file; `--index` prefers PIT membership episodes when the SDK exposes `index.members_history` (one episode per membership run; dropped members keep their closed episode — survivorship-safe) | `--index/--codes-file --exclude-st --exclude-suspended` |
144
+ | `verify` | read `.bin` back with Qlib's own reader | `--qlib-dir --check inst:field:date:value` |
145
+ | `reconcile-daily-basic` | Gate B: accumulated PIT vs official daily_basic history (100% or fail) | `--accumulated --upstream --tol`, or `--from-upstream --store --max-ec` |
146
+
147
+ `--dry-run` is available on four of them: `dump-history`, `update-history`,
148
+ `dump-cross-section`, `accumulate`.
149
+
150
+ ## EC cost model
151
+
152
+ **EC is wmshare's billing unit.** The platform default application has a weekly
153
+ quota refilled every Monday; a self-created additional application holds a one-time
154
+ quota that is not refilled. Demo mode spends nothing.
155
+
156
+ Batching saves **requests**, not **EC** — wmshare bills by symbol and compute. The
157
+ anchors below are our measured values (`config.EC_*`) and the single source of truth
158
+ for estimates:
159
+
160
+ | Call | Formula / anchor |
161
+ |---|---|
162
+ | `kline.bars` | `156 + 67 × served_bars` EC (the live `hfq_factor` path pulls adj + raw, so it doubles) |
163
+ | `moneyflow.bars` | `156 + 16 × n` EC |
164
+ | `valuation.hist_stats` | 712 EC per metric (3y-window fit; the default is now 10y, which bills more) |
165
+ | `kline.snapshot` batch | ≈66–100 EC/票 |
166
+ | `kline.batch_live` | ≈45 EC/票 (percent/ex-div semantics unverified — not used) |
167
+ | `screener.query` | full market × 4 light columns = **204,430 EC**; small probes do NOT predict full-universe cost |
168
+ | `valuation.daily_basic` | bills by rows scanned (not cached): 1-day floor ≈1,707 EC, full history ≈18,317 EC, incremental ≈312 EC/票/日; `update-history --daily-basic` reads are **windowed** from the resume pointer |
169
+ | `margin.query` / `margin.latest` | ≈124 EC/行 / ≈182 EC for the one-row read |
170
+ | range queries (`begin/end`) | **force-pay the anchor window, not the returned rows** — a 49-row window measured 33,688 EC (= 156 + 67×500). Use `--n` for increments |
171
+
172
+ Every fetch result carries `metering` (`requests`, `ec_cost`, `note`) written to
173
+ `fetch_meta.json` / `*.meta.json`. Estimates are estimates; the paid
174
+ `metering.ec_cost` is the only real quote, and `--max-ec` stays the binding ceiling.
175
+
176
+ Sustainable cadence: full-market screener weekly/monthly (~200k EC/query); the daily
177
+ cross-section path is `--via snapshot` over an explicit ~300-name universe
178
+ (~17–30k EC/day). Full-market daily is not viable on any current endpoint.
179
+
180
+ ## Data-correctness gates
181
+
182
+ - **Direction** — every series is asserted ascending (2.x contract); a silently
183
+ descending series would flip every rolling computation.
184
+ - **Unit invariants** — amounts in CNY, volumes in shares, ratios dimensionless;
185
+ absurd magnitudes raise rather than flow into a backtest.
186
+ - **Duplicate/conflict** — duplicated dates with identical values are deduplicated
187
+ with a warning; **conflicting** duplicates raise.
188
+ - **Reconciliation gate** — incremental tails must agree with stored history within
189
+ float tolerance. An upstream revision fails loud; history is never silently
190
+ rewritten.
191
+ - **Gate B** — accumulated PIT values are cross-checked against official
192
+ `daily_basic` history, 100% agreement or fail.
193
+
194
+ ## Qlib compatibility & low-intrusion design
195
+
196
+ The dumper targets Qlib's own on-disk layout, verified against
197
+ `microsoft/qlib`'s data collector conventions:
198
+
199
+ - `factor = hfq/raw` per date, with adjusted prices stored (live default);
200
+ upstream volume is not adjusted, so it is divided by factor to keep
201
+ `close × volume == amount` invariant across ex-dividend dates.
202
+ - `qlib.init()` runs only in a **child process** during read-back; the parent
203
+ process's `qlib.config.C` is asserted untouched in the test suite.
204
+ - `pyqlib` is an optional extra; the dumper and its self-check need only numpy.
205
+
206
+ ## Module map
207
+
208
+ - `config.py` — fields, units, metric-code selection, EC anchors (single source of truth)
209
+ - `normalize.py` — anti-pitfall guards (direction, code mapping, unit, cold cache); `merge_adjusted_klines` (hfq+raw → factor)
210
+ - `fetch.py` — demo/live acquisition + metering + EC-ceiling executor + bounded transient retries
211
+ - `panel.py` — integrity assertions (incl. the factor gate) + wide transform
212
+ - `dump.py` / `format.py` — Qlib-compatible `.bin` dumper and its on-disk contract (Qlib-free)
213
+ - `qlib_io.py` — **the only module that imports Qlib**; child-process read-back + score reader
214
+ - `resume.py` — fragment store (bulk + incremental tails) + manifest with convention guard and reconciliation gate
215
+ - `cross_section.py` — score CSV + PIT accumulation/promotion
216
+ - `estimate.py` — pre-flight EC estimator over the measured anchors (`--dry-run`)
217
+ - `reconcile.py` — Gate B comparison
218
+ - `cli.py` — the seven commands
219
+
220
+ ## Tests
221
+
222
+ ```bash
223
+ # Tier 1 (mandatory in CI, offline, no credentials, no EC)
224
+ pytest tests -q -m "not live"
225
+
226
+ # Tier 2 (optional) — adds the child-process Qlib read-back cross-check
227
+ pip install 'wmshare-qlib[qlib]'
228
+ pytest tests -q -m qlib
229
+ ```
230
+
231
+ Offline tests use `wmshare.demo_client()`, so they need no credentials. Tier 1
232
+ verifies the dumped `.bin` with pure numpy (`format.read_bin_selfcheck`); Tier 2
233
+ additionally reads it back through Qlib itself. Live tests are opt-in and gated by
234
+ `WMSHARE_QLIB_LIVE=1` — see [CONTRIBUTING.md](CONTRIBUTING.md).
235
+
236
+ ## What we deliberately do NOT do
237
+
238
+ - Fabricate history from today's snapshot (red line — see layer 2 above).
239
+ - Recompute upstream precomputed metrics client-side and present them as wmshare values.
240
+ - Per-symbol `indicator.get` loops (EC explosion + 429).
241
+ - Run a live metric-code `screener.query` without an explicit EC budget.
242
+ - Call `qlib.init()` in the caller's process (global side effect).
243
+ - Touch the wmshare main repo (separate repo, no version-bump churn).
244
+
245
+ ## See also
246
+
247
+ - [Getting started](docs/getting-started.md) — install to first `.bin`, demo then live.
248
+ - [Data model](docs/data-model.md) — the three availability tiers and the EC model.
249
+ - [FAQ](docs/faq.md) — demo boundaries, hfq vs qfq, error reference.
250
+ - [Contributing](CONTRIBUTING.md) — tests, lint, and the red lines.
251
+ - [Changelog](CHANGELOG.md)
252
+ - [wmshare](https://github.com/open-winmale/wmshare) — the data SDK this adapter translates.
253
+ - [wmshare-vnpy](https://github.com/open-winmale/wmshare-vnpy) — the same history panel
254
+ injected into vnpy.alpha.
255
+ - [Examples](examples/) — `dump_to_qlib.py`, `online_score.py`,
256
+ `alpha158_lgbm.py`, `benchmark/`.
257
+ - Design history and forward plans (engineering records, not user docs):
258
+ [docs/internal/](docs/internal/).
259
+
260
+ ## Acknowledgements
261
+
262
+ Built on [wmshare](https://github.com/open-winmale/wmshare) (BSD-3-Clause).
263
+ Not affiliated with or endorsed by Microsoft; Qlib/pyqlib is used unmodified as an
264
+ optional verification dependency only.
265
+
266
+ ## License
267
+
268
+ BSD-3-Clause — see [LICENSE](LICENSE).