quantdb-sdk 0.2.6__tar.gz → 0.2.9__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/CHANGELOG.md +24 -0
- {quantdb_sdk-0.2.6/quantdb_sdk.egg-info → quantdb_sdk-0.2.9}/PKG-INFO +13 -8
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/README.md +60 -55
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/pyproject.toml +3 -3
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/quantdb_sdk/__init__.py +1 -1
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/quantdb_sdk/_utils.py +25 -1
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/quantdb_sdk/async_client.py +100 -20
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/quantdb_sdk/client.py +89 -30
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9/quantdb_sdk.egg-info}/PKG-INFO +13 -8
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/quantdb_sdk.egg-info/SOURCES.txt +2 -1
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/tests/test_async_client.py +17 -0
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/tests/test_client.py +29 -1
- quantdb_sdk-0.2.9/tests/test_update_pipeline.py +42 -0
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/LICENSE +0 -0
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/MANIFEST.in +0 -0
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/quantdb_sdk/__main__.py +0 -0
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/quantdb_sdk/errors.py +0 -0
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/quantdb_sdk/py.typed +0 -0
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/quantdb_sdk.egg-info/dependency_links.txt +0 -0
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/quantdb_sdk.egg-info/entry_points.txt +0 -0
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/quantdb_sdk.egg-info/requires.txt +0 -0
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/quantdb_sdk.egg-info/top_level.txt +0 -0
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/setup.cfg +0 -0
- {quantdb_sdk-0.2.6 → quantdb_sdk-0.2.9}/tests/test_technical_indicators.py +0 -0
|
@@ -3,6 +3,19 @@
|
|
|
3
3
|
所有 notable 变更都会记录在此文件。格式基于 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/),
|
|
4
4
|
版本号遵循 [Semantic Versioning](https://semver.org/lang/zh-CN/)。
|
|
5
5
|
|
|
6
|
+
## [0.2.8] - 2026-07-30
|
|
7
|
+
|
|
8
|
+
### Fixed
|
|
9
|
+
- **CDN 直连下载 302 相对 URL 拼接**:服务端 CDN 直连模式(`QUANTDB_DOWNLOAD_REDIRECT`)的 Stage 1 302 返回相对 URL(`/api/v1/data/cdn-bridge?token=xxx`),同步/异步客户端此前直接将此相对路径作为请求 URL,导致 `requests`/`httpx` 无法解析(`MissingSchema`/`UnsupportedProtocol`),CDN 直连下载完全不可用。现在当 `Location` 以 `/` 开头时自动拼接 `api_host` 构造完整 URL。
|
|
10
|
+
- **CDN 直连请求无重试**:cdn-bridge 和 CDN 签名 URL 请求此前使用裸 `requests.get()`/`httpx.stream()` 调用,无重试、无超时容错。一次 connect timeout 或网络抖动即导致整个 `sync_dataset` 中断(且预扣流量不退)。现在两跳均带 2 次重试(指数退避 1s/2s),捕获 `ConnectionError`/`Timeout`/`ReadError` 自动重试。
|
|
11
|
+
- **302 重定向链路手动跟随**:此前只跟随第一跳 302,服务端两跳链路(`/download` → `/cdn-bridge?token=xxx` → CDN 签名 URL)的第二跳被忽略。现在统一用 `_follow_redirect_with_retry` 循环跟随最多 3 跳,ETag 逐跳透传。
|
|
12
|
+
|
|
13
|
+
## [0.2.7] - 2026-07-29
|
|
14
|
+
|
|
15
|
+
### Fixed
|
|
16
|
+
- **V1 回退对纯 V2 数据集失效**:`query_kline` / `a_query_kline` 无日期范围时不再默认走 V1。`margin_trading`、`features_daily`、`l1_factors`、`l2_factors` 等 11 个纯 V2 数据集在 COS 上已无 V1 逐股票文件,V1 回退会 404。现在无范围时也走 V2 manifest 路径(manifest 列出的所有分区全量下载),仅在有日期范围时才做日历完整性校验。显式 `layout="v1"` 仍可直接走逐股票文件。
|
|
17
|
+
- **异步客户端时区处理不一致**:`AsyncQuantDBClient._normalise_kline` 缺少 `tz_convert` 逻辑,与同步客户端行为不一致(相同输入可能产生不同的 `trade_date` 字符串),已对齐。
|
|
18
|
+
|
|
6
19
|
## [0.2.6] - 2026-07-27
|
|
7
20
|
|
|
8
21
|
### Fixed
|
|
@@ -10,6 +23,17 @@
|
|
|
10
23
|
- **凭证保护**:同步客户端此前跟随 302 时会把 `X-API-Key` 透传给 CDN 域;现两个客户端统一改为「手动处理 302 + 无鉴权头裸请求直连 CDN」,凭证只发给 QuantDB 网关。
|
|
11
24
|
- 302 响应中的 COS ETag 回填至 CDN 响应,保证进程内 ETag 缓存与 If-None-Match 304 逻辑不受影响。
|
|
12
25
|
|
|
26
|
+
### Security (后端)
|
|
27
|
+
- **一次性下载令牌**:302 直连模式改为两跳(`/download` → `/cdn-bridge?token=xxx` → CDN),令牌一次性消费,防止签名 URL 被重复使用或分享给他人免费下载。
|
|
28
|
+
- **移除 `?direct=1` 请求级覆盖**:用户无法再通过请求参数强制走 302 模式,仅全局开关 `QUANTDB_DOWNLOAD_REDIRECT` 可控制。
|
|
29
|
+
- 令牌有效期 90s,远小于 CDN 签名有效期 300s,窗口极小。
|
|
30
|
+
|
|
31
|
+
### Data Architecture (COS)
|
|
32
|
+
- **V2 数据集全面清理 V1 残留**:COS 上 11 个 V2 数据集已删除全部 V1 逐股票文件(~1.8 万对象),实现纯 V2 存储。ListObjects 分页效率提升 3-5 倍。
|
|
33
|
+
- **L1 日频因子 V2 全量上线**:`l1_factors` 2565 个交易日分区(2016-01-04 ~ 至今),支持按日范围高效查询。
|
|
34
|
+
- **L2 高频因子 V2 上线**:`l2_factors` 34 个交易日分区(2026-01-05 ~ 2026-02-27)。
|
|
35
|
+
- **融资融券纯 V2**:`margin_trading` 已从 V1 迁移至纯 V2 按日分区(2563 个分区)。
|
|
36
|
+
|
|
13
37
|
## [0.2.5] - 2026-07-26
|
|
14
38
|
|
|
15
39
|
### Security
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: quantdb-sdk
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.9
|
|
4
4
|
Summary: QuantDB 量化数据平台官方 Python SDK
|
|
5
5
|
Author: QuantDB Team
|
|
6
6
|
License: MIT
|
|
7
|
-
Project-URL: Homepage, https://
|
|
8
|
-
Project-URL: Documentation, https://
|
|
7
|
+
Project-URL: Homepage, https://quantdb.quantmind.cloud
|
|
8
|
+
Project-URL: Documentation, https://quantdb.quantmind.cloud/docs/sdk.html
|
|
9
9
|
Project-URL: Repository, https://github.com/quantdb/quantdb
|
|
10
10
|
Project-URL: Changelog, https://github.com/quantdb/quantdb/blob/main/CHANGELOG.md
|
|
11
11
|
Keywords: quant,finance,data,a-share,parquet
|
|
@@ -90,12 +90,17 @@ client = QuantDBClient(username="admin", password="admin123")
|
|
|
90
90
|
|
|
91
91
|
## V1 / V2 数据布局
|
|
92
92
|
|
|
93
|
-
|
|
94
|
-
`layout="auto" | "v1" | "v2"
|
|
95
|
-
|
|
93
|
+
QuantDB 数据采用两种物理布局:V1(按股票的全历史文件 `{Symbol}.parquet`)和 V2(按交易日的全市场分区 `dt=YYYYMMDD/data.parquet`)。
|
|
94
|
+
所有下载相关接口均可传入 `layout="auto" | "v1" | "v2"`。
|
|
95
|
+
|
|
96
|
+
**V2 数据集**(COS 纯 V2,零 V1 残留):daily_unadjusted / daily_forward / daily_backward / index_daily / valuation / technical_indicators / market_sentiment / features_daily / l1_factors / l2_factors / margin_trading
|
|
97
|
+
|
|
98
|
+
**V1 数据集**(纯 V1,无 V2 分区):min1_kline / min5_kline / tick_data / 财务七表 / 基础板块
|
|
99
|
+
|
|
100
|
+
默认 `auto` 始终优先 V2:有日期范围时聚合 V2 多日分区,若覆盖不完整则回退 V1(仅对仍保留 V1 文件的数据集有效);无日期范围时走 V2 全量(manifest 列出的所有分区),跳过完整性校验。
|
|
96
101
|
|
|
97
102
|
```python
|
|
98
|
-
#
|
|
103
|
+
# 始终优先 V2 按日分区;有范围且覆盖不完整时自动回退 V1
|
|
99
104
|
df = client.query_kline("600519.SH", start_date="2026-07-01", end_date="2026-07-24")
|
|
100
105
|
|
|
101
106
|
# 强制指定物理布局;layout="v2" 缺日时会明确报错
|
|
@@ -142,4 +147,4 @@ with ThreadPoolExecutor(max_workers=8) as pool:
|
|
|
142
147
|
|
|
143
148
|
## 文档
|
|
144
149
|
|
|
145
|
-
完整文档请访问:https://
|
|
150
|
+
完整文档请访问:https://quantdb.quantmind.cloud/docs/sdk.html
|
|
@@ -43,66 +43,71 @@ print(df.head())
|
|
|
43
43
|
client = QuantDBClient(username="admin", password="admin123")
|
|
44
44
|
```
|
|
45
45
|
|
|
46
|
-
## 核心功能
|
|
46
|
+
## 核心功能
|
|
47
47
|
|
|
48
48
|
- **数据查询**:K 线、Tick 通过下载 Parquet 切片后客户端解析(消耗流量);股票列表、交易日历、元数据走网关 JSON(不计流量)。
|
|
49
49
|
- **数据下载**:Parquet 文件下载或直读 DataFrame,计入订阅流量。
|
|
50
50
|
- **本地分析**:基于 DuckDB 对本地 Parquet 执行 SQL。
|
|
51
51
|
- **账户管理**:查询用户信息、用量、API Key、订阅与订单。
|
|
52
|
-
- **异步客户端**:基于 httpx,适用于 asyncio 量化框架。
|
|
53
|
-
|
|
54
|
-
## V1 / V2 数据布局
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
`layout="auto" | "v1" | "v2"
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
#
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
52
|
+
- **异步客户端**:基于 httpx,适用于 asyncio 量化框架。
|
|
53
|
+
|
|
54
|
+
## V1 / V2 数据布局
|
|
55
|
+
|
|
56
|
+
QuantDB 数据采用两种物理布局:V1(按股票的全历史文件 `{Symbol}.parquet`)和 V2(按交易日的全市场分区 `dt=YYYYMMDD/data.parquet`)。
|
|
57
|
+
所有下载相关接口均可传入 `layout="auto" | "v1" | "v2"`。
|
|
58
|
+
|
|
59
|
+
**V2 数据集**(COS 纯 V2,零 V1 残留):daily_unadjusted / daily_forward / daily_backward / index_daily / valuation / technical_indicators / market_sentiment / features_daily / l1_factors / l2_factors / margin_trading
|
|
60
|
+
|
|
61
|
+
**V1 数据集**(纯 V1,无 V2 分区):min1_kline / min5_kline / tick_data / 财务七表 / 基础板块
|
|
62
|
+
|
|
63
|
+
默认 `auto` 始终优先 V2:有日期范围时聚合 V2 多日分区,若覆盖不完整则回退 V1(仅对仍保留 V1 文件的数据集有效);无日期范围时走 V2 全量(manifest 列出的所有分区),跳过完整性校验。
|
|
64
|
+
|
|
65
|
+
```python
|
|
66
|
+
# 始终优先 V2 按日分区;有范围且覆盖不完整时自动回退 V1
|
|
67
|
+
df = client.query_kline("600519.SH", start_date="2026-07-01", end_date="2026-07-24")
|
|
68
|
+
|
|
69
|
+
# 强制指定物理布局;layout="v2" 缺日时会明确报错
|
|
70
|
+
latest = client.download_file("1", "daily_forward", trade_date="2026-07-24", layout="v2")
|
|
71
|
+
history = client.download_file("1", "daily_forward", symbol="600519.SH", layout="v1")
|
|
72
|
+
|
|
73
|
+
# 以发布清单为 cursor 做原子化增量同步(含 V2 patch)
|
|
74
|
+
result = client.sync_dataset("daily_forward", save_dir="D:/quantdb-data")
|
|
75
|
+
|
|
76
|
+
# 财务、ETF/可转债等尚无 V2 release 的数据集自动按 V1 Manifest 增量同步,
|
|
77
|
+
# 使用 ETag + size 校验;也可从命令行执行:
|
|
78
|
+
# quantdb sync qdb_xxx balance --save-dir D:/quantdb-data
|
|
79
|
+
financial = client.sync_dataset("balance", save_dir="D:/quantdb-data")
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
### 加速批量下载:建议从 8 个工作线程开始
|
|
83
|
+
|
|
84
|
+
`sync_dataset()` 单次调用会按发布顺序串行下载,以保证本地 SQLite 同步状态和 release cursor 一致。
|
|
85
|
+
当需要下载多个独立标的或文件时,可由调用方并行调度;建议先使用 **8 个工作线程**,再结合网络带宽、磁盘写入能力和账户流量配额调整。
|
|
86
|
+
|
|
87
|
+
```python
|
|
88
|
+
from concurrent.futures import ThreadPoolExecutor
|
|
89
|
+
from quantdb_sdk import QuantDBClient
|
|
90
|
+
|
|
91
|
+
API_KEY = "qdb_xxx..."
|
|
92
|
+
symbols = ["600519.SH", "000001.SZ", "600036.SH"]
|
|
93
|
+
|
|
94
|
+
def download_one(symbol: str) -> str:
|
|
95
|
+
# 每个 worker 使用独立客户端,避免跨线程共享 HTTP Session。
|
|
96
|
+
with_client = QuantDBClient(api_key=API_KEY)
|
|
97
|
+
return with_client.download_file(
|
|
98
|
+
"1", "daily_forward", symbol=symbol, save_dir="D:/quantdb-data"
|
|
99
|
+
)
|
|
100
|
+
|
|
101
|
+
with ThreadPoolExecutor(max_workers=8) as pool:
|
|
102
|
+
files = list(pool.map(download_one, symbols))
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
不要对相同 `save_dir` 并行调用多个 `sync_dataset()`:它们会共同写入 `quantdb_sync.sqlite`,可能产生锁竞争。批量同步本身仍建议一次一个数据集执行。
|
|
106
|
+
|
|
107
|
+
## 流量说明
|
|
108
|
+
|
|
109
|
+
免费注册用户获赠 100 MB 一次性体验流量;订阅用户每月含 30 GB 下载流量,超出部分按 ¥1/GB 从账户余额扣减。余额不足时下载会被拦截。
|
|
105
110
|
|
|
106
111
|
## 文档
|
|
107
112
|
|
|
108
|
-
完整文档请访问:https://
|
|
113
|
+
完整文档请访问:https://quantdb.quantmind.cloud/docs/sdk.html
|
|
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "quantdb-sdk"
|
|
7
|
-
version = "0.2.
|
|
7
|
+
version = "0.2.9"
|
|
8
8
|
description = "QuantDB 量化数据平台官方 Python SDK"
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
license = {text = "MIT"}
|
|
@@ -53,8 +53,8 @@ dev = [
|
|
|
53
53
|
]
|
|
54
54
|
|
|
55
55
|
[project.urls]
|
|
56
|
-
Homepage = "https://
|
|
57
|
-
Documentation = "https://
|
|
56
|
+
Homepage = "https://quantdb.quantmind.cloud"
|
|
57
|
+
Documentation = "https://quantdb.quantmind.cloud/docs/sdk.html"
|
|
58
58
|
Repository = "https://github.com/quantdb/quantdb"
|
|
59
59
|
Changelog = "https://github.com/quantdb/quantdb/blob/main/CHANGELOG.md"
|
|
60
60
|
|
|
@@ -5,6 +5,7 @@
|
|
|
5
5
|
|
|
6
6
|
import os
|
|
7
7
|
import re
|
|
8
|
+
from datetime import date
|
|
8
9
|
from typing import Dict, Optional
|
|
9
10
|
from urllib.parse import unquote, urlparse
|
|
10
11
|
|
|
@@ -20,7 +21,7 @@ SYNC_DATASET_CATEGORIES: Dict[str, str] = {
|
|
|
20
21
|
"holder_num": "3", "pershare_index": "3", "dividend_factors": "3",
|
|
21
22
|
"etf_pcf": "4", "convertible_bond": "4",
|
|
22
23
|
"valuation": "5", "technical_indicators": "5", "market_sentiment": "5",
|
|
23
|
-
"features_daily": "6", "l1_l2_factors": "6", "l2_factors": "6",
|
|
24
|
+
"features_daily": "6", "l1_factors": "6", "l1_l2_factors": "6", "l2_factors": "6",
|
|
24
25
|
}
|
|
25
26
|
|
|
26
27
|
|
|
@@ -97,6 +98,29 @@ def safe_join(root: str, relative_path: str) -> str:
|
|
|
97
98
|
return target
|
|
98
99
|
|
|
99
100
|
|
|
101
|
+
def normalise_tick_trade_date(trade_date: str) -> str:
|
|
102
|
+
"""规范化 Tick 交易日为 ``YYYY-MM-DD``,拒绝含糊或非法输入。"""
|
|
103
|
+
value = str(trade_date).strip()
|
|
104
|
+
if re.fullmatch(r"\d{8}", value):
|
|
105
|
+
value = f"{value[:4]}-{value[4:6]}-{value[6:]}"
|
|
106
|
+
if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", value):
|
|
107
|
+
raise ValueError("trade_date 必须为 YYYY-MM-DD 或 YYYYMMDD")
|
|
108
|
+
try:
|
|
109
|
+
date.fromisoformat(value)
|
|
110
|
+
except ValueError as exc:
|
|
111
|
+
raise ValueError("trade_date 不是有效交易日日期") from exc
|
|
112
|
+
return value
|
|
113
|
+
|
|
114
|
+
|
|
115
|
+
def tick_shard_relative_path(symbol: str, trade_date: str) -> str:
|
|
116
|
+
"""返回 Tick 平铺 shard 的规范相对路径。"""
|
|
117
|
+
clean_symbol = str(symbol).strip().upper()
|
|
118
|
+
if not re.fullmatch(r"[A-Z0-9]+\.[A-Z0-9]+", clean_symbol):
|
|
119
|
+
raise ValueError("Tick symbol 必须为类似 600519.SH 的代码")
|
|
120
|
+
yyyymmdd = normalise_tick_trade_date(trade_date).replace("-", "")
|
|
121
|
+
return f"1_kline_data/tick_data/{clean_symbol.replace('.', '_')}_{yyyymmdd}.parquet"
|
|
122
|
+
|
|
123
|
+
|
|
100
124
|
def validate_api_host(api_host: str) -> str:
|
|
101
125
|
"""Require HTTPS except for local development/test loopback endpoints."""
|
|
102
126
|
host = api_host.rstrip("/")
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
"""QuantDB 异步 Python SDK(基于 httpx)。"""
|
|
2
2
|
|
|
3
|
+
import asyncio
|
|
3
4
|
import hashlib
|
|
4
5
|
import io
|
|
5
6
|
import os
|
|
@@ -13,7 +14,7 @@ import pandas as pd
|
|
|
13
14
|
|
|
14
15
|
from ._utils import (
|
|
15
16
|
SYNC_DATASET_CATEGORIES, bytes_to_gb, check_download_size, default_download_dir,
|
|
16
|
-
max_download_bytes, parse_filename_from_content_disposition, safe_filename, safe_join,
|
|
17
|
+
max_download_bytes, normalise_tick_trade_date, parse_filename_from_content_disposition, safe_filename, safe_join,
|
|
17
18
|
validate_api_host,
|
|
18
19
|
)
|
|
19
20
|
from .errors import (
|
|
@@ -27,7 +28,7 @@ from .errors import (
|
|
|
27
28
|
)
|
|
28
29
|
|
|
29
30
|
# 维护提示:每次发版必须与 pyproject.toml 版本号同步
|
|
30
|
-
_USER_AGENT = "QuantDB-Python-SDK/0.2.
|
|
31
|
+
_USER_AGENT = "QuantDB-Python-SDK/0.2.8"
|
|
31
32
|
|
|
32
33
|
|
|
33
34
|
class AsyncQuantDBClient:
|
|
@@ -81,6 +82,10 @@ class AsyncQuantDBClient:
|
|
|
81
82
|
服务端开启 CDN 直连后,/api/v1/data/download 预扣流量成功时返回
|
|
82
83
|
302 -> CDN 签名 URL。httpx 默认不跟随重定向;这里用不带鉴权头的
|
|
83
84
|
裸客户端直连 Location,鉴权头不会泄露给 CDN 域。
|
|
85
|
+
|
|
86
|
+
重定向链路(CDN 直连模式):
|
|
87
|
+
/download 302→ /cdn-bridge?token=xxx 302→ https://cdn.xxx/key?sign=...
|
|
88
|
+
两跳均手动跟随,每跳带重试(connect timeout 网络抖动容错)。
|
|
84
89
|
"""
|
|
85
90
|
async with self.client.stream(
|
|
86
91
|
"GET", f"{self.api_host}/api/v1/data/download", params=params, headers=headers
|
|
@@ -92,14 +97,68 @@ class AsyncQuantDBClient:
|
|
|
92
97
|
etag = resp.headers.get("ETag", "")
|
|
93
98
|
if not location:
|
|
94
99
|
raise ServerError("下载重定向缺少 Location 头")
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
100
|
+
# 服务端 302 可能返回相对 URL(如 /api/v1/data/cdn-bridge?token=xxx),
|
|
101
|
+
# 需拼接 api_host 构造完整 URL,否则 httpx 无法解析。
|
|
102
|
+
if location.startswith("/"):
|
|
103
|
+
location = f"{self.api_host}{location}"
|
|
104
|
+
# 手动跟随重定向链路(最多 3 跳),每跳带重试
|
|
105
|
+
cdn_resp = await self._follow_redirect_with_retry(location, etag, max_hops=3)
|
|
106
|
+
async with cdn_resp:
|
|
101
107
|
yield cdn_resp
|
|
102
108
|
|
|
109
|
+
async def _follow_redirect_with_retry(
|
|
110
|
+
self, url: str, etag: str, max_hops: int = 3, retries: int = 2,
|
|
111
|
+
) -> httpx.Response:
|
|
112
|
+
"""手动跟随 302 重定向链路,每跳带重试。
|
|
113
|
+
|
|
114
|
+
cdn-bridge 和 CDN 直连请求均通过此方法,connect timeout 网络抖动
|
|
115
|
+
时自动重试,避免一次性 token(90s 有效期)场景下单次抖动导致整个
|
|
116
|
+
sync 中断。
|
|
117
|
+
|
|
118
|
+
返回已进入 stream 模式的 httpx.Response,调用方需用 async with 管理。
|
|
119
|
+
"""
|
|
120
|
+
import asyncio
|
|
121
|
+
current_etag = etag
|
|
122
|
+
for _ in range(max_hops):
|
|
123
|
+
last_exc: Exception | None = None
|
|
124
|
+
cdn_resp: httpx.Response | None = None
|
|
125
|
+
for attempt in range(retries + 1):
|
|
126
|
+
try:
|
|
127
|
+
# httpx stream() 是 async context manager,需 __aenter__ 进入
|
|
128
|
+
ctx = self._bare_client.stream("GET", url)
|
|
129
|
+
cdn_resp = await ctx.__aenter__()
|
|
130
|
+
# 触发实际连接(访问 status_code 即触发 head 读取)
|
|
131
|
+
_ = cdn_resp.status_code
|
|
132
|
+
break
|
|
133
|
+
except (httpx.ConnectError, httpx.ConnectTimeout, httpx.ReadError, httpx.ReadTimeout) as exc:
|
|
134
|
+
if cdn_resp is not None:
|
|
135
|
+
await cdn_resp.aclose()
|
|
136
|
+
cdn_resp = None
|
|
137
|
+
last_exc = exc
|
|
138
|
+
if attempt < retries:
|
|
139
|
+
await asyncio.sleep(1.0 * (attempt + 1))
|
|
140
|
+
else:
|
|
141
|
+
raise ServerError(f"CDN 直连下载失败(重试 {retries} 次后仍网络错误):{last_exc}")
|
|
142
|
+
if cdn_resp.status_code not in (200, 301, 302, 303, 307, 308):
|
|
143
|
+
await cdn_resp.aclose()
|
|
144
|
+
raise ServerError(f"CDN 直连下载失败:HTTP {cdn_resp.status_code}")
|
|
145
|
+
if cdn_resp.status_code == 200:
|
|
146
|
+
# ETag 回填:网关 302 携带的 COS ETag → CDN 响应缺失时补上
|
|
147
|
+
if current_etag and not cdn_resp.headers.get("ETag"):
|
|
148
|
+
cdn_resp.headers["ETag"] = current_etag
|
|
149
|
+
return cdn_resp
|
|
150
|
+
# 继续跟随 302
|
|
151
|
+
next_url = cdn_resp.headers.get("Location", "")
|
|
152
|
+
next_etag = cdn_resp.headers.get("ETag", current_etag)
|
|
153
|
+
await cdn_resp.aclose()
|
|
154
|
+
if not next_url:
|
|
155
|
+
raise ServerError("重定向缺少 Location 头")
|
|
156
|
+
if next_url.startswith("/"):
|
|
157
|
+
next_url = f"{self.api_host}{next_url}"
|
|
158
|
+
url = next_url
|
|
159
|
+
current_etag = next_etag
|
|
160
|
+
raise ServerError("重定向链路过深(超过 3 跳)")
|
|
161
|
+
|
|
103
162
|
@staticmethod
|
|
104
163
|
def _validate_layout(layout: str) -> Literal["auto", "v1", "v2"]:
|
|
105
164
|
if layout not in {"auto", "v1", "v2"}:
|
|
@@ -114,6 +173,8 @@ class AsyncQuantDBClient:
|
|
|
114
173
|
def _normalise_kline(df: pd.DataFrame, start_date: Optional[str], end_date: Optional[str], fields: str, limit: Optional[int]) -> pd.DataFrame:
|
|
115
174
|
if "time" in df.columns:
|
|
116
175
|
dt = pd.to_datetime(df["time"], errors="coerce")
|
|
176
|
+
if getattr(dt.dt, "tz", None) is not None:
|
|
177
|
+
dt = dt.dt.tz_convert(None)
|
|
117
178
|
elif "trade_date" in df.columns:
|
|
118
179
|
dt = pd.to_datetime(df["trade_date"], errors="coerce")
|
|
119
180
|
else:
|
|
@@ -278,22 +339,34 @@ class AsyncQuantDBClient:
|
|
|
278
339
|
limit: Optional[int] = None,
|
|
279
340
|
layout: Literal["auto", "v1", "v2"] = "auto",
|
|
280
341
|
) -> pd.DataFrame:
|
|
281
|
-
"""查询 K 线数据(下载 COS parquet 切片后客户端解析,消耗下载流量,异步)。
|
|
342
|
+
"""查询 K 线数据(下载 COS parquet 切片后客户端解析,消耗下载流量,异步)。
|
|
343
|
+
|
|
344
|
+
``auto`` 始终优先 V2 全市场日分区;仅当提供了日期范围且 V2 覆盖不完整时,
|
|
345
|
+
整次回退 V1 股票历史文件。未提供日期范围时走 V2 全量(manifest 列出的
|
|
346
|
+
所有分区),跳过日历完整性校验。显式 ``v1`` 直接走逐股票文件;显式
|
|
347
|
+
``v2`` 不会静默回退。
|
|
348
|
+
"""
|
|
282
349
|
layout = self._validate_layout(layout)
|
|
283
350
|
sub_category = f"daily_{adj_type}"
|
|
284
|
-
|
|
351
|
+
has_range = bool(start_date or end_date)
|
|
352
|
+
if layout == "v1":
|
|
285
353
|
return self._normalise_kline(await self.a_load_as_df("1", sub_category, symbol, layout="v1"), start_date, end_date, fields, limit)
|
|
286
354
|
files = await self.a_query_manifest("1", sub_category, layout="v2")
|
|
287
355
|
selected = [f for f in files if (not start_date or f.get("trade_date", "") >= start_date) and (not end_date or f.get("trade_date", "") <= end_date)]
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
if
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
if
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
356
|
+
# 仅在有日期范围时校验完整性;无范围时 manifest 列出什么就下什么。
|
|
357
|
+
complete = bool(selected)
|
|
358
|
+
if has_range:
|
|
359
|
+
calendar = await self.a_query_calendar(start_date, end_date)
|
|
360
|
+
expected = set()
|
|
361
|
+
if not calendar.empty:
|
|
362
|
+
date_col = next((c for c in ("trade_date", "date", "cal_date") if c in calendar.columns), None)
|
|
363
|
+
open_col = next((c for c in ("is_open", "is_trading_day", "open") if c in calendar.columns), None)
|
|
364
|
+
if date_col:
|
|
365
|
+
rows = calendar if not open_col else calendar[calendar[open_col].astype(str).isin(["1", "True", "true"])]
|
|
366
|
+
expected = set(pd.to_datetime(rows[date_col], errors="coerce").dropna().dt.strftime("%Y-%m-%d"))
|
|
367
|
+
found = {f.get("trade_date") for f in selected}
|
|
368
|
+
complete = bool(selected) and (not expected or expected.issubset(found))
|
|
369
|
+
if not complete:
|
|
297
370
|
if layout == "auto":
|
|
298
371
|
return self._normalise_kline(await self.a_load_as_df("1", sub_category, symbol, layout="v1"), start_date, end_date, fields, limit)
|
|
299
372
|
raise NotFoundError("V2 日切片在请求日期范围内覆盖不完整;请改用 layout='auto' 或 'v1'")
|
|
@@ -313,6 +386,7 @@ class AsyncQuantDBClient:
|
|
|
313
386
|
layout: Literal["auto", "v1", "v2"] = "auto",
|
|
314
387
|
) -> pd.DataFrame:
|
|
315
388
|
"""查询 Tick 分笔数据(下载 COS parquet 切片后客户端解析,消耗下载流量,异步)。"""
|
|
389
|
+
trade_date = normalise_tick_trade_date(trade_date)
|
|
316
390
|
df = await self.a_load_as_df("1", "tick_data", symbol, trade_date=trade_date, layout=layout)
|
|
317
391
|
ts_col = "ts" if "ts" in df.columns else ("time" if "time" in df.columns else None)
|
|
318
392
|
if ts_col and (start_ts or end_ts):
|
|
@@ -624,10 +698,16 @@ class AsyncQuantDBClient:
|
|
|
624
698
|
expected_size = obj.get("size")
|
|
625
699
|
old = state.execute("SELECT etag,size,path FROM objects WHERE key=?", (key,)).fetchone()
|
|
626
700
|
if old and old[0] == obj.get("etag") and os.path.exists(old[2]) and (expected_size is None or os.path.getsize(old[2]) == expected_size): continue
|
|
701
|
+
download_params = {"category_id": SYNC_DATASET_CATEGORIES[dataset], "sub_category": dataset, "layout": "v1", "symbol": obj.get("symbol", "")}
|
|
702
|
+
if dataset == "tick_data":
|
|
703
|
+
trade_date = obj.get("trade_date")
|
|
704
|
+
if not trade_date:
|
|
705
|
+
raise ServerError("Tick manifest 缺少 trade_date")
|
|
706
|
+
download_params["trade_date"] = normalise_tick_trade_date(trade_date)
|
|
627
707
|
os.makedirs(os.path.dirname(target), exist_ok=True)
|
|
628
708
|
tmp, written_size = target + ".part", 0
|
|
629
709
|
try:
|
|
630
|
-
async with self._download_stream(
|
|
710
|
+
async with self._download_stream(download_params) as resp:
|
|
631
711
|
if resp.status_code != 200:
|
|
632
712
|
body = await resp.aread(); self._check_response(httpx.Response(resp.status_code, content=body)); raise QuantDBError("下载失败")
|
|
633
713
|
with open(tmp, "wb") as fh:
|
|
@@ -6,6 +6,7 @@ import os
|
|
|
6
6
|
import re
|
|
7
7
|
import sqlite3
|
|
8
8
|
import glob
|
|
9
|
+
import time
|
|
9
10
|
from typing import Any, Dict, List, Optional, Literal
|
|
10
11
|
|
|
11
12
|
import pandas as pd
|
|
@@ -15,7 +16,7 @@ from urllib3.util.retry import Retry
|
|
|
15
16
|
|
|
16
17
|
from ._utils import (
|
|
17
18
|
SYNC_DATASET_CATEGORIES, bytes_to_gb, check_download_size, default_download_dir,
|
|
18
|
-
max_download_bytes, parse_filename_from_content_disposition, safe_filename, safe_join,
|
|
19
|
+
max_download_bytes, normalise_tick_trade_date, parse_filename_from_content_disposition, safe_filename, safe_join,
|
|
19
20
|
validate_api_host,
|
|
20
21
|
)
|
|
21
22
|
from .errors import (
|
|
@@ -29,7 +30,7 @@ from .errors import (
|
|
|
29
30
|
)
|
|
30
31
|
|
|
31
32
|
# 维护提示:每次发版必须与 pyproject.toml 版本号同步
|
|
32
|
-
_USER_AGENT = "QuantDB-Python-SDK/0.2.
|
|
33
|
+
_USER_AGENT = "QuantDB-Python-SDK/0.2.8"
|
|
33
34
|
|
|
34
35
|
|
|
35
36
|
class QuantDBClient:
|
|
@@ -124,6 +125,10 @@ class QuantDBClient:
|
|
|
124
125
|
服务端开启 CDN 直连后,/api/v1/data/download 预扣流量成功时返回
|
|
125
126
|
302 -> CDN 签名 URL。requests 自动跟随会把自定义 X-API-Key 头
|
|
126
127
|
透传给 CDN 域,故禁用自动跟随,改用不带鉴权头的裸请求直连。
|
|
128
|
+
|
|
129
|
+
重定向链路(CDN 直连模式):
|
|
130
|
+
/download 302→ /cdn-bridge?token=xxx 302→ https://cdn.xxx/key?sign=...
|
|
131
|
+
两跳均手动跟随,每跳带重试(connect timeout 网络抖动容错)。
|
|
127
132
|
"""
|
|
128
133
|
resp = self._request(
|
|
129
134
|
"GET", "/api/v1/data/download", params=params, stream=True,
|
|
@@ -136,20 +141,61 @@ class QuantDBClient:
|
|
|
136
141
|
resp.close()
|
|
137
142
|
if not location:
|
|
138
143
|
raise ServerError("下载重定向缺少 Location 头")
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
)
|
|
145
|
-
if cdn_resp.status_code != 200:
|
|
146
|
-
cdn_resp.close()
|
|
147
|
-
raise ServerError(f"CDN 直连下载失败:HTTP {cdn_resp.status_code}")
|
|
148
|
-
# 网关 302 响应携带 COS ETag;若 CDN 响应缺失则回填,保证客户端缓存逻辑一致。
|
|
149
|
-
if etag and not cdn_resp.headers.get("ETag"):
|
|
150
|
-
cdn_resp.headers["ETag"] = etag
|
|
144
|
+
# 服务端 302 可能返回相对 URL(如 /api/v1/data/cdn-bridge?token=xxx),
|
|
145
|
+
# 需拼接 api_host 构造完整 URL,否则 requests.get 无法解析。
|
|
146
|
+
if location.startswith("/"):
|
|
147
|
+
location = f"{self.api_host}{location}"
|
|
148
|
+
# 手动跟随重定向链路(最多 3 跳),每跳带重试
|
|
149
|
+
cdn_resp = self._follow_redirect_with_retry(location, etag, max_hops=3)
|
|
151
150
|
return cdn_resp
|
|
152
151
|
|
|
152
|
+
def _follow_redirect_with_retry(
|
|
153
|
+
self, url: str, etag: str, max_hops: int = 3, retries: int = 2,
|
|
154
|
+
) -> requests.Response:
|
|
155
|
+
"""手动跟随 302 重定向链路,每跳带重试。
|
|
156
|
+
|
|
157
|
+
cdn-bridge 和 CDN 直连请求均通过此方法,connect timeout 网络抖动
|
|
158
|
+
时自动重试,避免一次性 token(90s 有效期)场景下单次抖动导致整个
|
|
159
|
+
sync 中断。
|
|
160
|
+
"""
|
|
161
|
+
current_etag = etag
|
|
162
|
+
for _ in range(max_hops):
|
|
163
|
+
last_exc = None
|
|
164
|
+
for attempt in range(retries + 1):
|
|
165
|
+
try:
|
|
166
|
+
cdn_resp = requests.get(
|
|
167
|
+
url,
|
|
168
|
+
stream=True,
|
|
169
|
+
timeout=self.timeout,
|
|
170
|
+
headers={"User-Agent": _USER_AGENT},
|
|
171
|
+
)
|
|
172
|
+
break
|
|
173
|
+
except (requests.ConnectionError, requests.Timeout) as exc:
|
|
174
|
+
last_exc = exc
|
|
175
|
+
if attempt < retries:
|
|
176
|
+
time.sleep(1.0 * (attempt + 1))
|
|
177
|
+
else:
|
|
178
|
+
raise ServerError(f"CDN 直连下载失败(重试 {retries} 次后仍网络错误):{last_exc}")
|
|
179
|
+
if cdn_resp.status_code not in (200, 301, 302, 303, 307, 308):
|
|
180
|
+
cdn_resp.close()
|
|
181
|
+
raise ServerError(f"CDN 直连下载失败:HTTP {cdn_resp.status_code}")
|
|
182
|
+
if cdn_resp.status_code == 200:
|
|
183
|
+
# ETag 回填:网关 302 携带的 COS ETag → CDN 响应缺失时补上
|
|
184
|
+
if current_etag and not cdn_resp.headers.get("ETag"):
|
|
185
|
+
cdn_resp.headers["ETag"] = current_etag
|
|
186
|
+
return cdn_resp
|
|
187
|
+
# 继续跟随 302
|
|
188
|
+
next_url = cdn_resp.headers.get("Location", "")
|
|
189
|
+
next_etag = cdn_resp.headers.get("ETag", current_etag)
|
|
190
|
+
cdn_resp.close()
|
|
191
|
+
if not next_url:
|
|
192
|
+
raise ServerError("重定向缺少 Location 头")
|
|
193
|
+
if next_url.startswith("/"):
|
|
194
|
+
next_url = f"{self.api_host}{next_url}"
|
|
195
|
+
url = next_url
|
|
196
|
+
current_etag = next_etag
|
|
197
|
+
raise ServerError("重定向链路过深(超过 3 跳)")
|
|
198
|
+
|
|
153
199
|
def _post(self, path: str, json: Optional[dict] = None) -> dict:
|
|
154
200
|
resp = self._request("POST", path, json=json)
|
|
155
201
|
return self._check_response(resp)
|
|
@@ -337,29 +383,33 @@ class QuantDBClient:
|
|
|
337
383
|
) -> pd.DataFrame:
|
|
338
384
|
"""查询 K 线数据(下载 COS parquet 切片后客户端解析,消耗下载流量)。
|
|
339
385
|
|
|
340
|
-
``auto``
|
|
341
|
-
整次回退 V1
|
|
342
|
-
|
|
386
|
+
``auto`` 始终优先 V2 全市场日分区;仅当提供了日期范围且 V2 覆盖不完整时,
|
|
387
|
+
整次回退 V1 股票历史文件。未提供日期范围时走 V2 全量(manifest 列出的
|
|
388
|
+
所有分区),跳过日历完整性校验。显式 ``v1`` 直接走逐股票文件;显式
|
|
389
|
+
``v2`` 不会静默回退。
|
|
343
390
|
"""
|
|
344
391
|
layout = self._validate_layout(layout)
|
|
345
392
|
sub_category = f"daily_{adj_type}"
|
|
346
393
|
has_range = bool(start_date or end_date)
|
|
347
|
-
if layout == "v1"
|
|
394
|
+
if layout == "v1":
|
|
348
395
|
return self._normalise_kline(self.load_as_df("1", sub_category, symbol, layout="v1"), start_date, end_date, fields, limit)
|
|
349
396
|
|
|
350
397
|
files = self.query_manifest("1", sub_category, layout="v2")
|
|
351
398
|
selected = [f for f in files if (not start_date or f.get("trade_date", "") >= start_date) and (not end_date or f.get("trade_date", "") <= end_date)]
|
|
352
|
-
# 日历是 V2
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
if
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
if
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
399
|
+
# 日历是 V2 完整性的权威。仅在有日期范围时校验完整性;无范围时 manifest
|
|
400
|
+
# 列出什么就下什么,不要求日历覆盖。auto 在覆盖不完整时回退 V1;v2 明确报错。
|
|
401
|
+
complete = bool(selected)
|
|
402
|
+
if has_range:
|
|
403
|
+
calendar = self.query_calendar(start_date, end_date)
|
|
404
|
+
expected = set()
|
|
405
|
+
if not calendar.empty:
|
|
406
|
+
date_col = next((c for c in ("trade_date", "date", "cal_date") if c in calendar.columns), None)
|
|
407
|
+
open_col = next((c for c in ("is_open", "is_trading_day", "open") if c in calendar.columns), None)
|
|
408
|
+
if date_col:
|
|
409
|
+
rows = calendar if not open_col else calendar[calendar[open_col].astype(str).isin(["1", "True", "true"])]
|
|
410
|
+
expected = set(pd.to_datetime(rows[date_col], errors="coerce").dropna().dt.strftime("%Y-%m-%d"))
|
|
411
|
+
found = {f.get("trade_date") for f in selected}
|
|
412
|
+
complete = bool(selected) and (not expected or expected.issubset(found))
|
|
363
413
|
if not complete:
|
|
364
414
|
if layout == "auto":
|
|
365
415
|
return self._normalise_kline(self.load_as_df("1", sub_category, symbol, layout="v1"), start_date, end_date, fields, limit)
|
|
@@ -385,6 +435,7 @@ class QuantDBClient:
|
|
|
385
435
|
下载 trade_date 当日该 symbol 的 tick parquet,按 start_ts/end_ts 过滤时间、按 fields 选列。
|
|
386
436
|
start_ts/end_ts 可传完整时间戳或 "HH:MM:SS"(自动补 trade_date 日期)。
|
|
387
437
|
"""
|
|
438
|
+
trade_date = normalise_tick_trade_date(trade_date)
|
|
388
439
|
df = self.load_as_df("1", "tick_data", symbol, trade_date=trade_date, layout=layout)
|
|
389
440
|
# 时间过滤
|
|
390
441
|
ts_col = "ts" if "ts" in df.columns else ("time" if "time" in df.columns else None)
|
|
@@ -732,7 +783,15 @@ class QuantDBClient:
|
|
|
732
783
|
old = state.execute("SELECT etag, size, path FROM objects WHERE key=?", (key,)).fetchone()
|
|
733
784
|
if old and old[0] == obj.get("etag") and os.path.exists(old[2]) and (expected_size is None or os.path.getsize(old[2]) == expected_size):
|
|
734
785
|
continue
|
|
735
|
-
|
|
786
|
+
download_params = {"category_id": SYNC_DATASET_CATEGORIES[dataset], "sub_category": dataset, "layout": "v1", "symbol": obj.get("symbol", "")}
|
|
787
|
+
# Tick manifest 的每个对象都是独立交易日 shard。不能省略日期,
|
|
788
|
+
# 否则后端会回退下载该股票的最新 shard,造成历史同步错位。
|
|
789
|
+
if dataset == "tick_data":
|
|
790
|
+
trade_date = obj.get("trade_date")
|
|
791
|
+
if not trade_date:
|
|
792
|
+
raise ServerError("Tick manifest 缺少 trade_date")
|
|
793
|
+
download_params["trade_date"] = normalise_tick_trade_date(trade_date)
|
|
794
|
+
resp = self._download_stream(download_params)
|
|
736
795
|
if resp.status_code != 200: self._check_response(resp)
|
|
737
796
|
os.makedirs(os.path.dirname(target), exist_ok=True)
|
|
738
797
|
tmp = target + ".part"
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: quantdb-sdk
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.9
|
|
4
4
|
Summary: QuantDB 量化数据平台官方 Python SDK
|
|
5
5
|
Author: QuantDB Team
|
|
6
6
|
License: MIT
|
|
7
|
-
Project-URL: Homepage, https://
|
|
8
|
-
Project-URL: Documentation, https://
|
|
7
|
+
Project-URL: Homepage, https://quantdb.quantmind.cloud
|
|
8
|
+
Project-URL: Documentation, https://quantdb.quantmind.cloud/docs/sdk.html
|
|
9
9
|
Project-URL: Repository, https://github.com/quantdb/quantdb
|
|
10
10
|
Project-URL: Changelog, https://github.com/quantdb/quantdb/blob/main/CHANGELOG.md
|
|
11
11
|
Keywords: quant,finance,data,a-share,parquet
|
|
@@ -90,12 +90,17 @@ client = QuantDBClient(username="admin", password="admin123")
|
|
|
90
90
|
|
|
91
91
|
## V1 / V2 数据布局
|
|
92
92
|
|
|
93
|
-
|
|
94
|
-
`layout="auto" | "v1" | "v2"
|
|
95
|
-
|
|
93
|
+
QuantDB 数据采用两种物理布局:V1(按股票的全历史文件 `{Symbol}.parquet`)和 V2(按交易日的全市场分区 `dt=YYYYMMDD/data.parquet`)。
|
|
94
|
+
所有下载相关接口均可传入 `layout="auto" | "v1" | "v2"`。
|
|
95
|
+
|
|
96
|
+
**V2 数据集**(COS 纯 V2,零 V1 残留):daily_unadjusted / daily_forward / daily_backward / index_daily / valuation / technical_indicators / market_sentiment / features_daily / l1_factors / l2_factors / margin_trading
|
|
97
|
+
|
|
98
|
+
**V1 数据集**(纯 V1,无 V2 分区):min1_kline / min5_kline / tick_data / 财务七表 / 基础板块
|
|
99
|
+
|
|
100
|
+
默认 `auto` 始终优先 V2:有日期范围时聚合 V2 多日分区,若覆盖不完整则回退 V1(仅对仍保留 V1 文件的数据集有效);无日期范围时走 V2 全量(manifest 列出的所有分区),跳过完整性校验。
|
|
96
101
|
|
|
97
102
|
```python
|
|
98
|
-
#
|
|
103
|
+
# 始终优先 V2 按日分区;有范围且覆盖不完整时自动回退 V1
|
|
99
104
|
df = client.query_kline("600519.SH", start_date="2026-07-01", end_date="2026-07-24")
|
|
100
105
|
|
|
101
106
|
# 强制指定物理布局;layout="v2" 缺日时会明确报错
|
|
@@ -142,4 +147,4 @@ with ThreadPoolExecutor(max_workers=8) as pool:
|
|
|
142
147
|
|
|
143
148
|
## 文档
|
|
144
149
|
|
|
145
|
-
完整文档请访问:https://
|
|
150
|
+
完整文档请访问:https://quantdb.quantmind.cloud/docs/sdk.html
|
|
@@ -83,6 +83,23 @@ async def test_a_sync_financial_v1_validates_manifest_size(tmp_path):
|
|
|
83
83
|
assert (tmp_path / "3_financial_data" / "balance" / "600000.SH.parquet").read_bytes() == payload
|
|
84
84
|
|
|
85
85
|
|
|
86
|
+
@pytest.mark.asyncio
|
|
87
|
+
@respx.mock
|
|
88
|
+
async def test_a_sync_tick_uses_manifest_trade_date(tmp_path):
|
|
89
|
+
payload = b"tick-parquet"
|
|
90
|
+
respx.get(f"{API_HOST}/api/v1/data/releases").mock(return_value=httpx.Response(200, json={"releases": []}))
|
|
91
|
+
respx.get(f"{API_HOST}/api/v1/data/download/manifest").mock(return_value=httpx.Response(200, json={"files": [{
|
|
92
|
+
"key": "1_kline_data/tick_data/600519_SH_20260720.parquet",
|
|
93
|
+
"relative_path": "1_kline_data/tick_data/600519_SH_20260720.parquet",
|
|
94
|
+
"symbol": "600519.SH", "trade_date": "2026-07-20", "etag": "tick-etag", "size": len(payload),
|
|
95
|
+
}]}))
|
|
96
|
+
route = respx.get(f"{API_HOST}/api/v1/data/download").mock(return_value=httpx.Response(200, content=payload))
|
|
97
|
+
async with AsyncQuantDBClient(api_host=API_HOST, api_key="test-key") as client:
|
|
98
|
+
result = await client.a_sync_dataset("tick_data", str(tmp_path))
|
|
99
|
+
assert result["downloaded"] == ["1_kline_data/tick_data/600519_SH_20260720.parquet"]
|
|
100
|
+
assert parse_qs(urlparse(str(route.calls[0].request.url)).query)["trade_date"] == ["2026-07-20"]
|
|
101
|
+
|
|
102
|
+
|
|
86
103
|
@pytest.mark.asyncio
|
|
87
104
|
@respx.mock
|
|
88
105
|
async def test_a_auth_error_raises_auth_error():
|
|
@@ -9,7 +9,7 @@ import responses
|
|
|
9
9
|
|
|
10
10
|
from quantdb_sdk import AuthError, InsufficientTrafficError, QuantDBClient
|
|
11
11
|
from quantdb_sdk.errors import ServerError
|
|
12
|
-
from quantdb_sdk._utils import safe_join
|
|
12
|
+
from quantdb_sdk._utils import normalise_tick_trade_date, safe_join, tick_shard_relative_path
|
|
13
13
|
|
|
14
14
|
|
|
15
15
|
API_HOST = "http://localhost:5000"
|
|
@@ -55,6 +55,15 @@ def test_safe_join_rejects_path_traversal(tmp_path):
|
|
|
55
55
|
safe_join(str(tmp_path), "../../outside.parquet")
|
|
56
56
|
|
|
57
57
|
|
|
58
|
+
def test_tick_shard_path_is_flat_and_date_is_normalised():
|
|
59
|
+
assert normalise_tick_trade_date("20260720") == "2026-07-20"
|
|
60
|
+
assert tick_shard_relative_path("600519.SH", "2026-07-20") == (
|
|
61
|
+
"1_kline_data/tick_data/600519_SH_20260720.parquet"
|
|
62
|
+
)
|
|
63
|
+
with pytest.raises(ValueError):
|
|
64
|
+
normalise_tick_trade_date("2026-99-20")
|
|
65
|
+
|
|
66
|
+
|
|
58
67
|
@responses.activate
|
|
59
68
|
def test_download_file_uses_safe_fallback_for_malicious_filename(tmp_path):
|
|
60
69
|
responses.get(
|
|
@@ -130,6 +139,25 @@ def test_sync_financial_v1_validates_manifest_size(tmp_path):
|
|
|
130
139
|
assert (tmp_path / "3_financial_data" / "balance" / "600000.SH.parquet").read_bytes() == payload
|
|
131
140
|
|
|
132
141
|
|
|
142
|
+
@responses.activate
|
|
143
|
+
def test_sync_tick_uses_manifest_trade_date(tmp_path):
|
|
144
|
+
payload = b"tick-parquet"
|
|
145
|
+
responses.get(f"{API_HOST}/api/v1/data/releases", json={"releases": []}, status=200)
|
|
146
|
+
responses.get(
|
|
147
|
+
f"{API_HOST}/api/v1/data/download/manifest",
|
|
148
|
+
json={"files": [{
|
|
149
|
+
"key": "1_kline_data/tick_data/600519_SH_20260720.parquet",
|
|
150
|
+
"relative_path": "1_kline_data/tick_data/600519_SH_20260720.parquet",
|
|
151
|
+
"symbol": "600519.SH", "trade_date": "2026-07-20", "etag": "tick-etag", "size": len(payload),
|
|
152
|
+
}]}, status=200,
|
|
153
|
+
)
|
|
154
|
+
responses.get(f"{API_HOST}/api/v1/data/download", body=payload, status=200)
|
|
155
|
+
result = QuantDBClient(api_host=API_HOST, api_key="test-key").sync_dataset("tick_data", str(tmp_path))
|
|
156
|
+
assert result["downloaded"] == ["1_kline_data/tick_data/600519_SH_20260720.parquet"]
|
|
157
|
+
params = parse_qs(urlparse(responses.calls[-1].request.url).query)
|
|
158
|
+
assert params["trade_date"] == ["2026-07-20"]
|
|
159
|
+
|
|
160
|
+
|
|
133
161
|
@responses.activate
|
|
134
162
|
def test_sync_v1_size_failure_does_not_keep_partial_file(tmp_path):
|
|
135
163
|
responses.get(f"{API_HOST}/api/v1/data/releases", json={"releases": []}, status=200)
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
import importlib.util
|
|
2
|
+
import json
|
|
3
|
+
import sys
|
|
4
|
+
from pathlib import Path
|
|
5
|
+
|
|
6
|
+
|
|
7
|
+
MODULE = Path(__file__).resolve().parents[1] / "pipeline" / "update_pipeline.py"
|
|
8
|
+
SPEC = importlib.util.spec_from_file_location("update_pipeline", MODULE)
|
|
9
|
+
pipeline = importlib.util.module_from_spec(SPEC)
|
|
10
|
+
assert SPEC and SPEC.loader
|
|
11
|
+
sys.modules[SPEC.name] = pipeline
|
|
12
|
+
SPEC.loader.exec_module(pipeline)
|
|
13
|
+
|
|
14
|
+
|
|
15
|
+
def test_manual_tasks_are_not_automatic():
|
|
16
|
+
names = {task.name for task in pipeline.AUTO_TASKS}
|
|
17
|
+
assert "tick" not in names
|
|
18
|
+
assert "l2" not in names
|
|
19
|
+
assert pipeline.MANUAL_TASKS == {"tick", "l2"}
|
|
20
|
+
|
|
21
|
+
|
|
22
|
+
def test_dry_run_writes_machine_readable_manifest(tmp_path, monkeypatch):
|
|
23
|
+
monkeypatch.setattr(pipeline, "DATA_ROOT", tmp_path)
|
|
24
|
+
monkeypatch.setattr(pipeline, "RUNS_DIR", tmp_path / "_meta" / "update_runs")
|
|
25
|
+
args = type("Args", (), {"tasks": "market", "resume": None, "dry_run": True, "no_publish": True})()
|
|
26
|
+
assert pipeline.run(args) == 0
|
|
27
|
+
manifests = list((tmp_path / "_meta" / "update_runs").glob("*.json"))
|
|
28
|
+
assert len(manifests) == 1
|
|
29
|
+
data = json.loads(manifests[0].read_text(encoding="utf-8"))
|
|
30
|
+
assert data["status"] == "dry_run"
|
|
31
|
+
assert data["tasks"][0]["name"] == "market"
|
|
32
|
+
assert data["tasks"][0]["strategy"] == "always_full"
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
def test_dry_run_keeps_dependencies_plannable(tmp_path, monkeypatch):
|
|
36
|
+
monkeypatch.setattr(pipeline, "DATA_ROOT", tmp_path)
|
|
37
|
+
monkeypatch.setattr(pipeline, "RUNS_DIR", tmp_path / "_meta" / "update_runs")
|
|
38
|
+
args = type("Args", (), {"tasks": None, "resume": None, "dry_run": True, "no_publish": True})()
|
|
39
|
+
assert pipeline.run(args) == 0
|
|
40
|
+
manifest = next((tmp_path / "_meta" / "update_runs").glob("*.json"))
|
|
41
|
+
statuses = {task["name"]: task["status"] for task in json.loads(manifest.read_text(encoding="utf-8"))["tasks"]}
|
|
42
|
+
assert "skipped_dependency" not in statuses.values()
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|