tickdb-ts 0.0.0-stage → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +165 -2
- package/dist/config.d.ts +12 -0
- package/dist/config.js +24 -0
- package/dist/daemon.d.ts +11 -0
- package/dist/daemon.js +32 -0
- package/dist/diff.d.ts +8 -0
- package/dist/diff.js +16 -0
- package/dist/http.d.ts +32 -0
- package/dist/http.js +95 -0
- package/dist/index.d.ts +27 -0
- package/dist/index.js +109 -0
- package/dist/manifest.d.ts +3 -0
- package/dist/manifest.js +40 -0
- package/dist/status.d.ts +8 -0
- package/dist/status.js +16 -0
- package/dist/store.d.ts +54 -0
- package/dist/store.js +269 -0
- package/dist/sync.d.ts +9 -0
- package/dist/sync.js +122 -0
- package/dist/types.d.ts +58 -0
- package/dist/types.js +1 -0
- package/dist/util.d.ts +10 -0
- package/dist/util.js +32 -0
- package/package.json +27 -4
package/README.md
CHANGED
|
@@ -1,3 +1,166 @@
|
|
|
1
|
-
#
|
|
1
|
+
# tickdb-ts
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
offline-first A 股 K 线客户端(TypeScript / Node)。服务端(`growinpublic.ai/tickdb`)从通达信拉行情写 parquet,本客户端在本地维护 DuckDB,增量同步、离线可查——与 Rust CLI / Python 绑定同款体验。
|
|
4
|
+
|
|
5
|
+
## 安装
|
|
6
|
+
|
|
7
|
+
要求 Node ≥ 18(开发机验证:Node 22)。
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
# 本仓库内开发
|
|
11
|
+
cd tickdb/clients/tickdb-ts
|
|
12
|
+
npm install
|
|
13
|
+
npm run build # 产出 dist/
|
|
14
|
+
|
|
15
|
+
# 在你自己的项目里使用
|
|
16
|
+
npm install /path/to/growinpublic/tickdb/clients/tickdb-ts
|
|
17
|
+
# 或打 tarball: npm pack → npm install tickdb-ts-0.1.0.tgz
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## Token
|
|
21
|
+
|
|
22
|
+
`tik_` 开头的 opaque 字符串,`read` scope 即可。获取方式:
|
|
23
|
+
|
|
24
|
+
- 主站 `/console`(admin 在页面上签发/吊销 tickdb token)
|
|
25
|
+
- 或找管理员执行 `tickdb token-issue --label <名字> --scopes read`
|
|
26
|
+
|
|
27
|
+
⚠️ token 明文存在 `<dataDir>/config.json`,注意目录权限;泄露了让管理员吊销重发。
|
|
28
|
+
|
|
29
|
+
## 五分钟上手
|
|
30
|
+
|
|
31
|
+
```ts
|
|
32
|
+
import tickdb from 'tickdb-ts';
|
|
33
|
+
|
|
34
|
+
await tickdb.init({ token: 'tik_...' }); // 默认 dataDir ~/.tickdb;autoSync 默认 true(300s daemon + 立即一轮)
|
|
35
|
+
await tickdb.sync(); // 显式同步一轮(首次全量 ~1GB,耐心等)
|
|
36
|
+
|
|
37
|
+
const rows = await tickdb.query(`
|
|
38
|
+
SELECT ts, close, volume FROM kline
|
|
39
|
+
WHERE symbol = 'SH600000' AND cycle = 'D'
|
|
40
|
+
ORDER BY ts DESC LIMIT 5`);
|
|
41
|
+
console.log(rows);
|
|
42
|
+
// [{ ts: '2026-10-22T15:00:00.000Z', close: 12.34, volume: 5678900 }, ...]
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
不想自动同步:`init({ token, autoSync: false })`,之后自己调 `await tickdb.sync()`。
|
|
46
|
+
|
|
47
|
+
## API 参考
|
|
48
|
+
|
|
49
|
+
| 方法 | 说明 |
|
|
50
|
+
|---|---|
|
|
51
|
+
| `init(opts?)` | 初始化。`opts`: `dataDir`(默认 `~/.tickdb`)、`token`、`serverUrl`(默认生产地址)、`autoSync`(默认 `true`)、`syncIntervalSecs`(默认 300)。幂等——重复调用直接 return。autoSync 时起后台 daemon 并立即后台同步一轮 |
|
|
52
|
+
| `sync()` | 显式同步一轮,`await` 完成返回 `SyncReport`;**错误上抛**。并发调用第二个直接抛 `sync already running` |
|
|
53
|
+
| `syncStart()` | 后台立即一轮(fire-and-forget),错误只进 `syncStatus().lastError`,进程不 crash |
|
|
54
|
+
| `syncDaemonStart(intervalSecs?)` | 起/重启后台 daemon(默认 300s,强制 ≥ 60s) |
|
|
55
|
+
| `syncDaemonStop()` | 停 daemon |
|
|
56
|
+
| `syncStatus()` | 返回 `SyncStatus` 快照(见下),反映**最近完成的一轮** |
|
|
57
|
+
| `query(sql)` | 对本地 DuckDB 跑任意 SQL,返回 `Record<string, unknown>[]`;TIMESTAMP → ISO 字符串、DATE → `YYYY-MM-DD`、数值 → number |
|
|
58
|
+
| `close()` | 停 daemon + 关 DuckDB(进程退出前可选调用) |
|
|
59
|
+
|
|
60
|
+
```ts
|
|
61
|
+
interface SyncReport {
|
|
62
|
+
notModified: boolean; // true = ETag 304 整轮 no-op
|
|
63
|
+
filesTotal: number; filesFetched: number; filesUnchanged: number;
|
|
64
|
+
rowsUpserted: number; staleDeleted: number;
|
|
65
|
+
mirrorFetched: number; mirrorFallbacks: number; // v1.2:镜像命中 / 回落 VPS 计数
|
|
66
|
+
errors: string[]; // 单文件失败不中断整轮
|
|
67
|
+
}
|
|
68
|
+
interface SyncStatus {
|
|
69
|
+
running: boolean; startedAt: number; finishedAt: number; // unix 秒
|
|
70
|
+
filesTotal: number; filesFetched: number; filesUnchanged: number;
|
|
71
|
+
rowsUpserted: number; staleDeleted: number; errorsCount: number;
|
|
72
|
+
daemonIntervalSecs: number; daemonRunning: boolean;
|
|
73
|
+
lastError: string | null;
|
|
74
|
+
}
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
## 本地数据
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
~/.tickdb/ (或你传的 dataDir)
|
|
81
|
+
├── config.json # token/serverUrl/interval
|
|
82
|
+
├── tickdb.duckdb # 本地 DuckDB(数据 + local_manifest + sync_meta)
|
|
83
|
+
└── .cache/parquet/ # 下载暂存,upsert 成功后即删,正常为空
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
**kline 表**(主键 `symbol, cycle, ts`):
|
|
87
|
+
|
|
88
|
+
| 列 | 类型 | 说明 |
|
|
89
|
+
|---|---|---|
|
|
90
|
+
| ts | TIMESTAMP | bar 时间(A 股收盘 15:00 的日线 ts 就是 15:00:00,见 FAQ) |
|
|
91
|
+
| symbol | VARCHAR | `SH600000` / `SZ000001` 风格 |
|
|
92
|
+
| cycle | VARCHAR | `1m / 5m / 15m / 30m / 60m / D` |
|
|
93
|
+
| open/high/low/close | DOUBLE | 价格 |
|
|
94
|
+
| volume | BIGINT | 成交量(股) |
|
|
95
|
+
| amount | DOUBLE | 成交额 |
|
|
96
|
+
|
|
97
|
+
另有 `local_manifest`(已同步文件清单)、`sync_meta`(manifest ETag),以及与 Rust 客户端对齐的 extra 表和 `kline_qfq` / `kline_hfq` 复权视图——**v1 同步协议只覆盖 kline**,extra 表(含 `adjustment_factors`)没有数据,复权视图的 factor 恒为 1(等价不复权),后续版本再开。
|
|
98
|
+
|
|
99
|
+
常用 SQL:
|
|
100
|
+
|
|
101
|
+
```sql
|
|
102
|
+
-- 个股近 30 个交易日日线
|
|
103
|
+
SELECT ts, open, high, low, close, volume FROM kline
|
|
104
|
+
WHERE symbol = 'SH600000' AND cycle = 'D'
|
|
105
|
+
AND ts >= NOW()::TIMESTAMP - INTERVAL '30 DAY'
|
|
106
|
+
ORDER BY ts;
|
|
107
|
+
|
|
108
|
+
-- 全市场今日(最新一天)涨幅榜前 10
|
|
109
|
+
SELECT symbol, close / lag_close - 1 AS pct
|
|
110
|
+
FROM (
|
|
111
|
+
SELECT symbol, close, LAG(close) OVER (PARTITION BY symbol ORDER BY ts) AS lag_close,
|
|
112
|
+
ROW_NUMBER() OVER (PARTITION BY symbol ORDER BY ts DESC) AS rn
|
|
113
|
+
FROM kline WHERE cycle = 'D'
|
|
114
|
+
) WHERE rn = 1 AND lag_close IS NOT NULL
|
|
115
|
+
ORDER BY pct DESC LIMIT 10;
|
|
116
|
+
|
|
117
|
+
-- 同步状态
|
|
118
|
+
SELECT COUNT(*) AS files, MAX(synced_at) AS last_sync FROM local_manifest;
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
## 同步机制(简版)
|
|
122
|
+
|
|
123
|
+
1. `GET /api/v1/manifest?version=v1.2` 带 `If-None-Match`,未变收 304 整轮 no-op
|
|
124
|
+
2. 与本地 `local_manifest` 按 `path + sha256` 求 diff
|
|
125
|
+
3. 服务端消失的文件(stale)优先按该文件自己的 `ts_min..ts_max` 边界删本地行(月文件消失只删它那个月);边界缺失回退整年删除
|
|
126
|
+
4. 串行下载变化的 parquet:流式写 `.part` → size + sha256 双校验 → 原子 rename(3 次重试线性退避)。frozen 文件先试镜像(manifest v1.2 下发的公开 URL,无认证),失败/污染回落 VPS 不计错误
|
|
127
|
+
5. `INSERT OR REPLACE` 进 kline 表——**union 语义**:同主键的修正 bar 覆盖旧值;新文件里消失的行保留本地
|
|
128
|
+
6. 只有整轮零错误才写回 ETag(保证 304 ⇒ 本地确实与服务端一致)
|
|
129
|
+
|
|
130
|
+
## FAQ
|
|
131
|
+
|
|
132
|
+
**Q: ts 为什么日线是 15:00:00?**
|
|
133
|
+
A 股收盘 15:00 的墙钟时间。服务端 parquet 的时间戳是 `TIMESTAMPTZ`,本客户端打开 DuckDB 后第一句就 `SET timezone='UTC'` 再转 TIMESTAMP——和 Rust/Python 客户端同一套时区 pin 逻辑,保证去重主键一致、bar 不翻倍。查询时拿到的 ISO 字符串(带 Z)就是当日墙钟时间。
|
|
134
|
+
|
|
135
|
+
**Q: 报「tickdb 数据库被另一进程占用」?**
|
|
136
|
+
DuckDB 是单写者。同一份 `tickdb.duckdb` 被别的进程开着(python/streamlit/另一个 Node 进程)就会锁冲突。停掉对方进程,或者给本客户端换个 `dataDir`。
|
|
137
|
+
|
|
138
|
+
**Q: securities / dividend / financial_basic 这些表怎么没数据?**
|
|
139
|
+
服务端 manifest(v1/v1.1/v1.2)只覆盖 kline(K 线)。Python/Rust 客户端同样如此。要查财报/分红请走服务端 HTTP API(`POST /tickdb/api/v1/query`)或等后续版本扩展同步协议。
|
|
140
|
+
|
|
141
|
+
**Q: 断网能用吗?**
|
|
142
|
+
能。`query()` 全部走本地 DuckDB。`sync()` 会报错(显式调用错误上抛);后台 daemon 的错误只记在 `syncStatus().lastError`,恢复后下一轮自动追上。
|
|
143
|
+
|
|
144
|
+
**Q: 首次同步要多久 / 占多大空间?**
|
|
145
|
+
当前生产数据 ~60+ 个 parquet / ~1GB / 9000 万+ bar,串行下载,跨国链路大约 10-40 分钟。之后每轮只重拉 sha 变化的文件(通常只有当年/当月的几个)。
|
|
146
|
+
|
|
147
|
+
**Q: 和 Rust CLI / Python 绑定能用同一个 dataDir 吗?**
|
|
148
|
+
不建议。三者 schema 一致,但 DuckDB 单写者 + 版本差异,先后打开同一 db 文件容易锁冲突。给 TS 客户端单独一个 `dataDir` 最省心。
|
|
149
|
+
|
|
150
|
+
## 开发
|
|
151
|
+
|
|
152
|
+
```bash
|
|
153
|
+
npm test # node:test + tsx,43 个测试(含 fixture 服务器 + 镜像回落集成测试)
|
|
154
|
+
npm run typecheck # tsc --noEmit
|
|
155
|
+
npm run build # tsc -> dist/
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
协议实现与 Rust 客户端(`crates/tickdb-client`)逐条对照:
|
|
159
|
+
|
|
160
|
+
| 本包 | Rust 对照 |
|
|
161
|
+
|---|---|
|
|
162
|
+
| `src/http.ts` | `http_client.rs`(ETag/304、流式下载校验) |
|
|
163
|
+
| `src/sync.ts` | `manifest_sync.rs`(syncOnce 全流程) |
|
|
164
|
+
| `src/diff.ts` | `manifest_sync.rs::compute_diff` |
|
|
165
|
+
| `src/store.ts` | `duckdb_store.rs`(schema/upsert/时区 pin) |
|
|
166
|
+
| `src/daemon.ts` + `src/status.ts` | `tickdb-python/src/lib.rs` 的 daemon/atomics |
|
package/dist/config.d.ts
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
import type { InitOptions } from './types.js';
|
|
2
|
+
export declare const DEFAULT_SERVER_URL = "https://growinpublic.ai/tickdb";
|
|
3
|
+
export interface ResolvedConfig {
|
|
4
|
+
dataDir: string;
|
|
5
|
+
token: string;
|
|
6
|
+
serverUrl: string;
|
|
7
|
+
syncIntervalSecs: number;
|
|
8
|
+
downloadConcurrency: number;
|
|
9
|
+
}
|
|
10
|
+
export declare function configPath(dataDir: string): string;
|
|
11
|
+
export declare function loadMergedConfig(dataDir: string, opts: InitOptions): ResolvedConfig;
|
|
12
|
+
export declare function saveConfig(cfg: ResolvedConfig): void;
|
package/dist/config.js
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
import { readFileSync, writeFileSync } from 'node:fs';
|
|
2
|
+
import { join } from 'node:path';
|
|
3
|
+
export const DEFAULT_SERVER_URL = 'https://growinpublic.ai/tickdb';
|
|
4
|
+
export function configPath(dataDir) {
|
|
5
|
+
return join(dataDir, 'config.json');
|
|
6
|
+
}
|
|
7
|
+
export function loadMergedConfig(dataDir, opts) {
|
|
8
|
+
let saved = {};
|
|
9
|
+
try {
|
|
10
|
+
saved = JSON.parse(readFileSync(configPath(dataDir), 'utf8'));
|
|
11
|
+
}
|
|
12
|
+
catch { /* absent or corrupt: start fresh */ }
|
|
13
|
+
const token = opts.token && opts.token.length > 0 ? opts.token : (saved.token ?? '');
|
|
14
|
+
const serverUrl = opts.serverUrl && opts.serverUrl.length > 0 ? opts.serverUrl : (saved.serverUrl ?? DEFAULT_SERVER_URL);
|
|
15
|
+
const syncIntervalSecs = opts.syncIntervalSecs ?? saved.syncIntervalSecs ?? 300;
|
|
16
|
+
const downloadConcurrency = Math.min(6, Math.max(1, opts.downloadConcurrency ?? saved.downloadConcurrency ?? 4));
|
|
17
|
+
return { dataDir, token, serverUrl, syncIntervalSecs, downloadConcurrency };
|
|
18
|
+
}
|
|
19
|
+
export function saveConfig(cfg) {
|
|
20
|
+
writeFileSync(configPath(cfg.dataDir), JSON.stringify({
|
|
21
|
+
token: cfg.token, serverUrl: cfg.serverUrl, syncIntervalSecs: cfg.syncIntervalSecs,
|
|
22
|
+
downloadConcurrency: cfg.downloadConcurrency,
|
|
23
|
+
}, null, 2));
|
|
24
|
+
}
|
package/dist/daemon.d.ts
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
import type { SyncStateHolder } from './status.js';
|
|
2
|
+
export interface DaemonHandle {
|
|
3
|
+
tickNow(): Promise<void>;
|
|
4
|
+
stop(): void;
|
|
5
|
+
}
|
|
6
|
+
/** setInterval(unref) + generation 计数;tickNow 供测试与「立即触发」。 */
|
|
7
|
+
export declare function createDaemon(opts: {
|
|
8
|
+
onTick: () => Promise<void>;
|
|
9
|
+
intervalSecs: number;
|
|
10
|
+
state: SyncStateHolder;
|
|
11
|
+
}): DaemonHandle;
|
package/dist/daemon.js
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
let generation = 0;
|
|
2
|
+
/** setInterval(unref) + generation 计数;tickNow 供测试与「立即触发」。 */
|
|
3
|
+
export function createDaemon(opts) {
|
|
4
|
+
generation += 1;
|
|
5
|
+
const myGen = generation;
|
|
6
|
+
const intervalSecs = Math.max(60, opts.intervalSecs);
|
|
7
|
+
opts.state.update({ daemonIntervalSecs: intervalSecs, daemonRunning: true });
|
|
8
|
+
const runTick = async () => {
|
|
9
|
+
if (myGen !== generation)
|
|
10
|
+
return;
|
|
11
|
+
if (opts.state.get().running)
|
|
12
|
+
return;
|
|
13
|
+
try {
|
|
14
|
+
await opts.onTick();
|
|
15
|
+
}
|
|
16
|
+
catch {
|
|
17
|
+
// onTick 自己负责把错误写进 state.lastError;daemon 永不 crash 进程
|
|
18
|
+
}
|
|
19
|
+
};
|
|
20
|
+
const timer = setInterval(() => void runTick(), intervalSecs * 1000);
|
|
21
|
+
timer.unref();
|
|
22
|
+
return {
|
|
23
|
+
tickNow: runTick,
|
|
24
|
+
stop() {
|
|
25
|
+
if (myGen !== generation)
|
|
26
|
+
return;
|
|
27
|
+
generation += 1;
|
|
28
|
+
clearInterval(timer);
|
|
29
|
+
opts.state.update({ daemonRunning: false, daemonIntervalSecs: 0 });
|
|
30
|
+
},
|
|
31
|
+
};
|
|
32
|
+
}
|
package/dist/diff.d.ts
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
import type { FileEntry } from './types.js';
|
|
2
|
+
export interface ManifestDiff {
|
|
3
|
+
toFetch: FileEntry[];
|
|
4
|
+
stale: string[];
|
|
5
|
+
unchanged: number;
|
|
6
|
+
}
|
|
7
|
+
/** 纯函数:只按 path + sha256 判定(与 Rust 客户端一致)。local: path -> sha256。 */
|
|
8
|
+
export declare function computeDiff(serverFiles: FileEntry[], local: Map<string, string>): ManifestDiff;
|
package/dist/diff.js
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
/** 纯函数:只按 path + sha256 判定(与 Rust 客户端一致)。local: path -> sha256。 */
|
|
2
|
+
export function computeDiff(serverFiles, local) {
|
|
3
|
+
const serverPaths = new Set(serverFiles.map((f) => f.path));
|
|
4
|
+
const toFetch = [];
|
|
5
|
+
let unchanged = 0;
|
|
6
|
+
for (const f of serverFiles) {
|
|
7
|
+
if (local.get(f.path) === f.sha256)
|
|
8
|
+
unchanged++;
|
|
9
|
+
else
|
|
10
|
+
toFetch.push(f);
|
|
11
|
+
}
|
|
12
|
+
const stale = [...local.keys()].filter((p) => !serverPaths.has(p));
|
|
13
|
+
toFetch.sort((a, b) => (a.path < b.path ? -1 : a.path > b.path ? 1 : 0));
|
|
14
|
+
stale.sort();
|
|
15
|
+
return { toFetch, stale, unchanged };
|
|
16
|
+
}
|
package/dist/http.d.ts
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
import type { Manifest } from './types.js';
|
|
2
|
+
export type ManifestResult = {
|
|
3
|
+
kind: 'not_modified';
|
|
4
|
+
} | {
|
|
5
|
+
kind: 'fresh';
|
|
6
|
+
manifest: Manifest;
|
|
7
|
+
etag?: string;
|
|
8
|
+
};
|
|
9
|
+
/** 下载总超时:保底 10 分钟,按 64 KiB/s 地板速率随文件大小上浮(慢链路上大文件不会被误杀)。 */
|
|
10
|
+
export declare function downloadTimeoutMs(size: number): number;
|
|
11
|
+
export declare class TickdbHttp {
|
|
12
|
+
private readonly serverUrl;
|
|
13
|
+
private readonly token;
|
|
14
|
+
constructor(serverUrl: string, token: string);
|
|
15
|
+
private headers;
|
|
16
|
+
fetchManifest(ifNoneMatch?: string): Promise<ManifestResult>;
|
|
17
|
+
/** 单文件:流式写 .part -> size+sha256 校验 -> 原子 rename;失败重试 attempts 次(线性退避)。调用方负责串行。 */
|
|
18
|
+
downloadToFile(label: string, urlPath: string, dest: string, expect: {
|
|
19
|
+
size: number;
|
|
20
|
+
sha256: string;
|
|
21
|
+
}, opts?: {
|
|
22
|
+
attempts?: number;
|
|
23
|
+
backoffMs?: number;
|
|
24
|
+
}): Promise<void>;
|
|
25
|
+
/** 镜像下载:无认证绝对 URL,单次尝试(失败由调用方回落 VPS,VPS 的重试循环就是重试)。sha/size 校验与原子 rename 复用同一内核。 */
|
|
26
|
+
downloadMirrorToFile(url: string, dest: string, expect: {
|
|
27
|
+
size: number;
|
|
28
|
+
sha256: string;
|
|
29
|
+
}): Promise<void>;
|
|
30
|
+
private tryDownload;
|
|
31
|
+
private tryDownloadUrl;
|
|
32
|
+
}
|
package/dist/http.js
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
import { createHash } from 'node:crypto';
|
|
2
|
+
import { createWriteStream, renameSync, rmSync } from 'node:fs';
|
|
3
|
+
import { Readable, Transform } from 'node:stream';
|
|
4
|
+
import { pipeline } from 'node:stream/promises';
|
|
5
|
+
import { parseManifest } from './manifest.js';
|
|
6
|
+
import { sleep } from './util.js';
|
|
7
|
+
/** 下载总超时:保底 10 分钟,按 64 KiB/s 地板速率随文件大小上浮(慢链路上大文件不会被误杀)。 */
|
|
8
|
+
export function downloadTimeoutMs(size) {
|
|
9
|
+
return Math.max(600_000, Math.ceil(size / 65_536) * 1000);
|
|
10
|
+
}
|
|
11
|
+
export class TickdbHttp {
|
|
12
|
+
serverUrl;
|
|
13
|
+
token;
|
|
14
|
+
constructor(serverUrl, token) {
|
|
15
|
+
this.serverUrl = serverUrl;
|
|
16
|
+
this.token = token;
|
|
17
|
+
}
|
|
18
|
+
headers() {
|
|
19
|
+
return { Authorization: `Bearer ${this.token}` };
|
|
20
|
+
}
|
|
21
|
+
async fetchManifest(ifNoneMatch) {
|
|
22
|
+
const headers = this.headers();
|
|
23
|
+
if (ifNoneMatch)
|
|
24
|
+
headers['If-None-Match'] = ifNoneMatch;
|
|
25
|
+
const res = await fetch(`${this.serverUrl}/api/v1/manifest?version=v1.3`, {
|
|
26
|
+
headers,
|
|
27
|
+
signal: AbortSignal.timeout(60_000),
|
|
28
|
+
});
|
|
29
|
+
if (res.status === 304)
|
|
30
|
+
return { kind: 'not_modified' };
|
|
31
|
+
if (!res.ok)
|
|
32
|
+
throw new Error(`manifest HTTP ${res.status}: ${(await res.text()).slice(0, 200)}`);
|
|
33
|
+
const etag = res.headers.get('etag') ?? undefined;
|
|
34
|
+
return { kind: 'fresh', manifest: parseManifest(await res.json()), etag };
|
|
35
|
+
}
|
|
36
|
+
/** 单文件:流式写 .part -> size+sha256 校验 -> 原子 rename;失败重试 attempts 次(线性退避)。调用方负责串行。 */
|
|
37
|
+
async downloadToFile(label, urlPath, dest, expect, opts = {}) {
|
|
38
|
+
const attempts = opts.attempts ?? 3;
|
|
39
|
+
const backoffMs = opts.backoffMs ?? 250;
|
|
40
|
+
let lastErr = null;
|
|
41
|
+
for (let attempt = 1; attempt <= attempts; attempt++) {
|
|
42
|
+
try {
|
|
43
|
+
await this.tryDownload(urlPath, dest, expect);
|
|
44
|
+
return;
|
|
45
|
+
}
|
|
46
|
+
catch (e) {
|
|
47
|
+
lastErr = e;
|
|
48
|
+
if (attempt < attempts)
|
|
49
|
+
await sleep(backoffMs * attempt);
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
throw new Error(`download failed after ${attempts} attempts (${label}): ${errText(lastErr)}`);
|
|
53
|
+
}
|
|
54
|
+
/** 镜像下载:无认证绝对 URL,单次尝试(失败由调用方回落 VPS,VPS 的重试循环就是重试)。sha/size 校验与原子 rename 复用同一内核。 */
|
|
55
|
+
async downloadMirrorToFile(url, dest, expect) {
|
|
56
|
+
await this.tryDownloadUrl(url, dest, expect, {});
|
|
57
|
+
}
|
|
58
|
+
async tryDownload(urlPath, dest, expect) {
|
|
59
|
+
await this.tryDownloadUrl(`${this.serverUrl}/api/v1/data/${urlPath}`, dest, expect, this.headers());
|
|
60
|
+
}
|
|
61
|
+
async tryDownloadUrl(url, dest, expect, headers) {
|
|
62
|
+
const res = await fetch(url, {
|
|
63
|
+
headers,
|
|
64
|
+
signal: AbortSignal.timeout(downloadTimeoutMs(expect.size)),
|
|
65
|
+
});
|
|
66
|
+
if (!res.ok || !res.body)
|
|
67
|
+
throw new Error(`data HTTP ${res.status} (${url})`);
|
|
68
|
+
const part = `${dest}.part`;
|
|
69
|
+
const hash = createHash('sha256');
|
|
70
|
+
let bytes = 0;
|
|
71
|
+
const counter = new Transform({
|
|
72
|
+
transform(chunk, _enc, cb) {
|
|
73
|
+
bytes += chunk.length;
|
|
74
|
+
hash.update(chunk);
|
|
75
|
+
cb(null, chunk);
|
|
76
|
+
},
|
|
77
|
+
});
|
|
78
|
+
try {
|
|
79
|
+
await pipeline(Readable.fromWeb(res.body), counter, createWriteStream(part));
|
|
80
|
+
if (bytes !== expect.size)
|
|
81
|
+
throw new Error(`size mismatch: got ${bytes}, want ${expect.size}`);
|
|
82
|
+
const sha = hash.digest('hex');
|
|
83
|
+
if (sha !== expect.sha256)
|
|
84
|
+
throw new Error(`sha256 mismatch: got ${sha}, want ${expect.sha256}`);
|
|
85
|
+
renameSync(part, dest);
|
|
86
|
+
}
|
|
87
|
+
catch (e) {
|
|
88
|
+
rmSync(part, { force: true });
|
|
89
|
+
throw e;
|
|
90
|
+
}
|
|
91
|
+
}
|
|
92
|
+
}
|
|
93
|
+
function errText(e) {
|
|
94
|
+
return e instanceof Error ? e.message : String(e);
|
|
95
|
+
}
|
package/dist/index.d.ts
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
import type { InitOptions, SyncReport, SyncStatus } from './types.js';
|
|
2
|
+
export declare function init(opts?: InitOptions): Promise<void>;
|
|
3
|
+
/** 显式同步:await 完成,错误上抛。 */
|
|
4
|
+
export declare function sync(): Promise<SyncReport>;
|
|
5
|
+
/** 后台立即一轮:错误只进 syncStatus().lastError,进程不 crash。 */
|
|
6
|
+
export declare function syncStart(): void;
|
|
7
|
+
export declare function syncDaemonStart(intervalSecs?: number): void;
|
|
8
|
+
export declare function syncDaemonStop(): void;
|
|
9
|
+
export declare function syncStatus(): SyncStatus;
|
|
10
|
+
export declare function query(sql: string): Promise<Record<string, unknown>[]>;
|
|
11
|
+
export declare function close(): Promise<void>;
|
|
12
|
+
/** 测试专用:重置单例。勿在应用代码使用。 */
|
|
13
|
+
export declare function _resetForTest(): void;
|
|
14
|
+
/** 测试专用:手动触发 daemon tick。 */
|
|
15
|
+
export declare function _daemonTickForTest(): Promise<void>;
|
|
16
|
+
declare const tickdb: {
|
|
17
|
+
init: typeof init;
|
|
18
|
+
sync: typeof sync;
|
|
19
|
+
syncStart: typeof syncStart;
|
|
20
|
+
syncDaemonStart: typeof syncDaemonStart;
|
|
21
|
+
syncDaemonStop: typeof syncDaemonStop;
|
|
22
|
+
syncStatus: typeof syncStatus;
|
|
23
|
+
query: typeof query;
|
|
24
|
+
close: typeof close;
|
|
25
|
+
_daemonTickForTest: typeof _daemonTickForTest;
|
|
26
|
+
};
|
|
27
|
+
export default tickdb;
|
package/dist/index.js
ADDED
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
import { mkdirSync } from 'node:fs';
|
|
2
|
+
import { join } from 'node:path';
|
|
3
|
+
import { DuckdbStore } from './store.js';
|
|
4
|
+
import { TickdbHttp } from './http.js';
|
|
5
|
+
import { syncOnce } from './sync.js';
|
|
6
|
+
import { SyncStateHolder } from './status.js';
|
|
7
|
+
import { createDaemon } from './daemon.js';
|
|
8
|
+
import { loadMergedConfig, saveConfig } from './config.js';
|
|
9
|
+
import { expandHome, nowSec } from './util.js';
|
|
10
|
+
let lib = null;
|
|
11
|
+
export async function init(opts = {}) {
|
|
12
|
+
if (lib)
|
|
13
|
+
return; // 幂等(热重载场景安全)
|
|
14
|
+
const dataDir = expandHome(opts.dataDir ?? '~/.tickdb');
|
|
15
|
+
const cacheDir = join(dataDir, '.cache', 'parquet');
|
|
16
|
+
mkdirSync(cacheDir, { recursive: true });
|
|
17
|
+
const cfg = loadMergedConfig(dataDir, opts);
|
|
18
|
+
saveConfig(cfg);
|
|
19
|
+
const store = await DuckdbStore.open(join(dataDir, 'tickdb.duckdb'));
|
|
20
|
+
store.setParquetRoot(cacheDir); // v2 查询前置刷新用;upsert 模式零影响
|
|
21
|
+
const state = new SyncStateHolder();
|
|
22
|
+
const http = new TickdbHttp(cfg.serverUrl, cfg.token);
|
|
23
|
+
lib = { dataDir, cacheDir, store, http, state, daemon: null, downloadConcurrency: cfg.downloadConcurrency };
|
|
24
|
+
if ((opts.autoSync ?? true) && cfg.token) {
|
|
25
|
+
lib.daemon = createDaemon({
|
|
26
|
+
onTick: () => backgroundSync(),
|
|
27
|
+
intervalSecs: opts.syncIntervalSecs ?? cfg.syncIntervalSecs,
|
|
28
|
+
state,
|
|
29
|
+
});
|
|
30
|
+
void backgroundSync(); // init 后立即来一轮(后台,错误不抛)
|
|
31
|
+
}
|
|
32
|
+
}
|
|
33
|
+
function requireLib() {
|
|
34
|
+
if (!lib)
|
|
35
|
+
throw new Error('tickdb not initialized: call init() first');
|
|
36
|
+
return lib;
|
|
37
|
+
}
|
|
38
|
+
/** 带状态跟踪的一轮同步。 */
|
|
39
|
+
async function runRound() {
|
|
40
|
+
const l = requireLib();
|
|
41
|
+
if (l.state.get().running)
|
|
42
|
+
throw new Error('sync already running');
|
|
43
|
+
l.state.update({ running: true, startedAt: nowSec() });
|
|
44
|
+
try {
|
|
45
|
+
const report = await syncOnce(l.store, l.http, l.cacheDir, { downloadConcurrency: l.downloadConcurrency });
|
|
46
|
+
l.state.update({
|
|
47
|
+
running: false, finishedAt: nowSec(),
|
|
48
|
+
filesTotal: report.filesTotal, filesFetched: report.filesFetched,
|
|
49
|
+
filesUnchanged: report.filesUnchanged, rowsUpserted: report.rowsUpserted,
|
|
50
|
+
staleDeleted: report.staleDeleted, errorsCount: report.errors.length,
|
|
51
|
+
lastError: report.errors.length > 0 ? report.errors[0] : null,
|
|
52
|
+
});
|
|
53
|
+
return report;
|
|
54
|
+
}
|
|
55
|
+
catch (e) {
|
|
56
|
+
l.state.update({ running: false, finishedAt: nowSec(), lastError: e instanceof Error ? e.message : String(e) });
|
|
57
|
+
throw e;
|
|
58
|
+
}
|
|
59
|
+
}
|
|
60
|
+
/** 显式同步:await 完成,错误上抛。 */
|
|
61
|
+
export async function sync() {
|
|
62
|
+
return runRound();
|
|
63
|
+
}
|
|
64
|
+
/** 后台立即一轮:错误只进 syncStatus().lastError,进程不 crash。 */
|
|
65
|
+
export function syncStart() {
|
|
66
|
+
void backgroundSync();
|
|
67
|
+
}
|
|
68
|
+
async function backgroundSync() {
|
|
69
|
+
try {
|
|
70
|
+
await runRound();
|
|
71
|
+
}
|
|
72
|
+
catch (e) {
|
|
73
|
+
// close() 后 lib 可能为 null:只记不再抛,进程永不 crash
|
|
74
|
+
lib?.state.setError(e instanceof Error ? e.message : String(e));
|
|
75
|
+
}
|
|
76
|
+
}
|
|
77
|
+
export function syncDaemonStart(intervalSecs = 300) {
|
|
78
|
+
const l = requireLib();
|
|
79
|
+
l.daemon?.stop();
|
|
80
|
+
l.daemon = createDaemon({ onTick: () => backgroundSync(), intervalSecs, state: l.state });
|
|
81
|
+
}
|
|
82
|
+
export function syncDaemonStop() {
|
|
83
|
+
const l = requireLib();
|
|
84
|
+
l.daemon?.stop();
|
|
85
|
+
l.daemon = null;
|
|
86
|
+
}
|
|
87
|
+
export function syncStatus() {
|
|
88
|
+
return requireLib().state.get();
|
|
89
|
+
}
|
|
90
|
+
export async function query(sql) {
|
|
91
|
+
return requireLib().store.query(sql);
|
|
92
|
+
}
|
|
93
|
+
export async function close() {
|
|
94
|
+
if (!lib)
|
|
95
|
+
return;
|
|
96
|
+
lib.daemon?.stop();
|
|
97
|
+
await lib.store.close();
|
|
98
|
+
lib = null;
|
|
99
|
+
}
|
|
100
|
+
/** 测试专用:重置单例。勿在应用代码使用。 */
|
|
101
|
+
export function _resetForTest() {
|
|
102
|
+
lib = null;
|
|
103
|
+
}
|
|
104
|
+
/** 测试专用:手动触发 daemon tick。 */
|
|
105
|
+
export async function _daemonTickForTest() {
|
|
106
|
+
await requireLib().daemon?.tickNow();
|
|
107
|
+
}
|
|
108
|
+
const tickdb = { init, sync, syncStart, syncDaemonStart, syncDaemonStop, syncStatus, query, close, _daemonTickForTest };
|
|
109
|
+
export default tickdb;
|
package/dist/manifest.js
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
/** 把线上 JSON(snake_case)解析成 Manifest,字段不合法直接抛错。 */
|
|
2
|
+
export function parseManifest(raw) {
|
|
3
|
+
if (typeof raw !== 'object' || raw === null)
|
|
4
|
+
throw new Error('manifest: not an object');
|
|
5
|
+
const r = raw;
|
|
6
|
+
if (r.version !== 'v1' && r.version !== 'v1.1' && r.version !== 'v1.2' && r.version !== 'v1.3') {
|
|
7
|
+
throw new Error(`unsupported manifest version: ${String(r.version)}`);
|
|
8
|
+
}
|
|
9
|
+
if (!Array.isArray(r.files))
|
|
10
|
+
throw new Error('manifest: files is not an array');
|
|
11
|
+
const files = r.files.map((f, i) => {
|
|
12
|
+
if (typeof f !== 'object' || f === null)
|
|
13
|
+
throw new Error(`manifest files[${i}]: not an object`);
|
|
14
|
+
const e = f;
|
|
15
|
+
for (const k of ['path', 'sha256', 'symbol', 'cycle', 'ts_min', 'ts_max']) {
|
|
16
|
+
if (typeof e[k] !== 'string')
|
|
17
|
+
throw new Error(`manifest files[${i}].${k}: expected string`);
|
|
18
|
+
}
|
|
19
|
+
for (const k of ['size', 'row_count']) {
|
|
20
|
+
if (typeof e[k] !== 'number')
|
|
21
|
+
throw new Error(`manifest files[${i}].${k}: expected number`);
|
|
22
|
+
}
|
|
23
|
+
return {
|
|
24
|
+
path: e.path, sha256: e.sha256, size: e.size,
|
|
25
|
+
row_count: e.row_count, symbol: e.symbol, cycle: e.cycle,
|
|
26
|
+
ts_min: e.ts_min, ts_max: e.ts_max,
|
|
27
|
+
year: typeof e.year === 'number' ? e.year : undefined,
|
|
28
|
+
symbols_count: typeof e.symbols_count === 'number' ? e.symbols_count : undefined,
|
|
29
|
+
frozen: e.frozen === true ? true : undefined,
|
|
30
|
+
};
|
|
31
|
+
});
|
|
32
|
+
return {
|
|
33
|
+
version: r.version,
|
|
34
|
+
schema: typeof r.schema === 'string' ? r.schema : 'symbol-partitioned',
|
|
35
|
+
generatedAt: typeof r.generated_at === 'string' ? r.generated_at : '',
|
|
36
|
+
files,
|
|
37
|
+
mirror_base_url: typeof r.mirror_base_url === 'string' ? r.mirror_base_url : undefined,
|
|
38
|
+
disjoint: r.disjoint === true ? true : undefined,
|
|
39
|
+
};
|
|
40
|
+
}
|
package/dist/status.d.ts
ADDED
package/dist/status.js
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
function fresh() {
|
|
2
|
+
return {
|
|
3
|
+
running: false, startedAt: 0, finishedAt: 0,
|
|
4
|
+
filesTotal: 0, filesFetched: 0, filesUnchanged: 0,
|
|
5
|
+
rowsUpserted: 0, staleDeleted: 0, errorsCount: 0,
|
|
6
|
+
daemonIntervalSecs: 0, daemonRunning: false,
|
|
7
|
+
lastError: null,
|
|
8
|
+
};
|
|
9
|
+
}
|
|
10
|
+
export class SyncStateHolder {
|
|
11
|
+
s = fresh();
|
|
12
|
+
get() { return { ...this.s }; }
|
|
13
|
+
update(patch) { this.s = { ...this.s, ...patch }; }
|
|
14
|
+
/** 覆盖 lastError;传 null 清除。 */
|
|
15
|
+
setError(msg) { this.s = { ...this.s, lastError: msg }; }
|
|
16
|
+
}
|
package/dist/store.d.ts
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
import type { FileEntry } from './types.js';
|
|
2
|
+
/** local_manifest 全字段行(stale per-file ts 边界删除用)。 */
|
|
3
|
+
export interface LocalManifestRow {
|
|
4
|
+
path: string;
|
|
5
|
+
sha256: string;
|
|
6
|
+
cycle: string;
|
|
7
|
+
year: number | null;
|
|
8
|
+
ts_min: string | null;
|
|
9
|
+
ts_max: string | null;
|
|
10
|
+
}
|
|
11
|
+
/** sync_meta key:kline 存储模式('duckdb-upsert' | 'parquet-direct')。 */
|
|
12
|
+
export declare const STORE_MODE_KEY = "store_mode";
|
|
13
|
+
export declare class DuckdbStore {
|
|
14
|
+
private instance;
|
|
15
|
+
private conn;
|
|
16
|
+
/** v2 parquet-direct:kline view 挂载根(<dataDir>/.cache/parquet)。 */
|
|
17
|
+
private parquetRoot;
|
|
18
|
+
private constructor();
|
|
19
|
+
static open(path: string): Promise<DuckdbStore>;
|
|
20
|
+
private c;
|
|
21
|
+
close(): Promise<void>;
|
|
22
|
+
run(sql: string, params?: unknown[]): Promise<void>;
|
|
23
|
+
query(sql: string): Promise<Record<string, unknown>[]>;
|
|
24
|
+
/** v2 parquet-direct 查询前置:glob 重展开(view 注册时展开 glob,新文件才
|
|
25
|
+
* 可见——与服务端 refresh_views 同因)。upsert 模式零开销直通。
|
|
26
|
+
* sync_meta 读取走原始连接(getSyncMeta → query → 这里,避免自递归)。 */
|
|
27
|
+
private maybeRefreshKlineSurface;
|
|
28
|
+
setParquetRoot(root: string): void;
|
|
29
|
+
/** v2 parquet-direct:kline 以 view 挂载 parquet 目录(hive_partitioning 从
|
|
30
|
+
* cycle={c} 目录名注入 cycle 列——parquet 本身无 cycle 列;与服务端
|
|
31
|
+
* query_engine 同款)。CREATE OR REPLACE 保证幂等。空目录(sync 前置调用)
|
|
32
|
+
* 建同 schema 空 view,read_parquet 空 glob 会 IO Error。 */
|
|
33
|
+
refreshKlineView(cacheRoot: string): Promise<void>;
|
|
34
|
+
/** 把 kline 表面(surface)对齐到当前模式:direct → read_parquet VIEW
|
|
35
|
+
* (v1 遗留 TABLE 先 DROP);upsert → TABLE(v2 遗留 VIEW 先 DROP 重建,
|
|
36
|
+
* 服务端回滚场景)。依赖视图 qfq/hfq 一并重建。幂等。 */
|
|
37
|
+
ensureKlineSurface(mode: string, cacheRoot: string): Promise<void>;
|
|
38
|
+
/** sync_meta 读取走原始连接(不经 query() 的 surface 刷新)。 */
|
|
39
|
+
getSyncMeta(key: string): Promise<string | null>;
|
|
40
|
+
setSyncMeta(key: string, value: string): Promise<void>;
|
|
41
|
+
clearLocalManifest(): Promise<void>;
|
|
42
|
+
countParquetRows(parquetPath: string): Promise<number>;
|
|
43
|
+
upsertKlineFromFile(parquetPath: string, cycle: string): Promise<void>;
|
|
44
|
+
deleteKlineYear(cycle: string, year: number): Promise<void>;
|
|
45
|
+
/** 按文件 ts 边界删除(含端点,口径同 deleteKlineYear)。月切片 ts 互不重叠,不会误伤相邻月。 */
|
|
46
|
+
deleteKlineRange(cycle: string, tsMin: string, tsMax: string): Promise<void>;
|
|
47
|
+
listLocalManifest(): Promise<Map<string, string>>;
|
|
48
|
+
/** 全字段 local_manifest(stale per-file ts 边界删除用;NULL 边界 → null,调用方回退整年)。 */
|
|
49
|
+
listLocalManifestFull(): Promise<Map<string, LocalManifestRow>>;
|
|
50
|
+
upsertLocalManifest(f: FileEntry): Promise<void>;
|
|
51
|
+
deleteLocalManifest(paths: string[]): Promise<void>;
|
|
52
|
+
getEtag(): Promise<string | null>;
|
|
53
|
+
setEtag(etag: string): Promise<void>;
|
|
54
|
+
}
|
package/dist/store.js
ADDED
|
@@ -0,0 +1,269 @@
|
|
|
1
|
+
import { DuckDBInstance } from '@duckdb/node-api';
|
|
2
|
+
import { yearBounds } from './util.js';
|
|
3
|
+
/** sync_meta key:kline 存储模式('duckdb-upsert' | 'parquet-direct')。 */
|
|
4
|
+
export const STORE_MODE_KEY = 'store_mode';
|
|
5
|
+
/** 与 Rust 客户端 KLINE_SCHEMA 逐条对齐(duckdb_store.rs)。 */
|
|
6
|
+
const KLINE_TABLE_DDL = `CREATE TABLE IF NOT EXISTS kline (
|
|
7
|
+
ts TIMESTAMP, symbol VARCHAR, cycle VARCHAR,
|
|
8
|
+
open DOUBLE, high DOUBLE, low DOUBLE, close DOUBLE,
|
|
9
|
+
volume BIGINT, amount DOUBLE,
|
|
10
|
+
PRIMARY KEY (symbol, cycle, ts))`;
|
|
11
|
+
const KLINE_QFQ_VIEW_DDL = `CREATE OR REPLACE VIEW kline_qfq AS
|
|
12
|
+
SELECT k.ts, k.symbol, k.cycle,
|
|
13
|
+
k.open * COALESCE(f.factor, 1.0) AS open, k.high * COALESCE(f.factor, 1.0) AS high,
|
|
14
|
+
k.low * COALESCE(f.factor, 1.0) AS low, k.close * COALESCE(f.factor, 1.0) AS close,
|
|
15
|
+
k.volume, k.amount
|
|
16
|
+
FROM kline k
|
|
17
|
+
ASOF LEFT JOIN adjustment_factors f
|
|
18
|
+
ON f.symbol = k.symbol AND f.adjust_type = 'qfq' AND k.ts >= f.effective_from`;
|
|
19
|
+
const KLINE_HFQ_VIEW_DDL = `CREATE OR REPLACE VIEW kline_hfq AS
|
|
20
|
+
SELECT k.ts, k.symbol, k.cycle,
|
|
21
|
+
k.open * COALESCE(f.factor, 1.0) AS open, k.high * COALESCE(f.factor, 1.0) AS high,
|
|
22
|
+
k.low * COALESCE(f.factor, 1.0) AS low, k.close * COALESCE(f.factor, 1.0) AS close,
|
|
23
|
+
k.volume, k.amount
|
|
24
|
+
FROM kline k
|
|
25
|
+
ASOF LEFT JOIN adjustment_factors f
|
|
26
|
+
ON f.symbol = k.symbol AND f.adjust_type = 'hfq' AND k.ts >= f.effective_from`;
|
|
27
|
+
const SCHEMA_STATEMENTS = [
|
|
28
|
+
KLINE_TABLE_DDL,
|
|
29
|
+
`CREATE TABLE IF NOT EXISTS subscriptions (symbol VARCHAR, cycle VARCHAR, added_at TIMESTAMP, PRIMARY KEY (symbol, cycle))`,
|
|
30
|
+
`CREATE TABLE IF NOT EXISTS sync_state (symbol VARCHAR, cycle VARCHAR, last_ts TIMESTAMP, last_sync_at TIMESTAMP, last_rows BIGINT, PRIMARY KEY (symbol, cycle))`,
|
|
31
|
+
`CREATE TABLE IF NOT EXISTS sync_state_extra (data_type VARCHAR, last_sync_at TIMESTAMP, last_rows BIGINT, notes VARCHAR, PRIMARY KEY (data_type))`,
|
|
32
|
+
`CREATE TABLE IF NOT EXISTS securities (symbol VARCHAR PRIMARY KEY, code VARCHAR, name VARCHAR, market VARCHAR, fetched_at TIMESTAMP)`,
|
|
33
|
+
`CREATE TABLE IF NOT EXISTS quotes_snapshot (
|
|
34
|
+
ts TIMESTAMP, symbol VARCHAR, price DOUBLE, pre_close DOUBLE, open DOUBLE, high DOUBLE, low DOUBLE,
|
|
35
|
+
volume DOUBLE, amount DOUBLE,
|
|
36
|
+
bid1 DOUBLE, bid2 DOUBLE, bid3 DOUBLE, bid4 DOUBLE, bid5 DOUBLE,
|
|
37
|
+
ask1 DOUBLE, ask2 DOUBLE, ask3 DOUBLE, ask4 DOUBLE, ask5 DOUBLE,
|
|
38
|
+
bid_vol1 DOUBLE, bid_vol2 DOUBLE, bid_vol3 DOUBLE, bid_vol4 DOUBLE, bid_vol5 DOUBLE,
|
|
39
|
+
ask_vol1 DOUBLE, ask_vol2 DOUBLE, ask_vol3 DOUBLE, ask_vol4 DOUBLE, ask_vol5 DOUBLE,
|
|
40
|
+
PRIMARY KEY (symbol, ts))`,
|
|
41
|
+
`CREATE TABLE IF NOT EXISTS dividend (
|
|
42
|
+
symbol VARCHAR, date DATE, category SMALLINT, songzhuan DOUBLE, peigu DOUBLE, peigujia DOUBLE,
|
|
43
|
+
fenhong DOUBLE, liutong DOUBLE, zongguben DOUBLE,
|
|
44
|
+
PRIMARY KEY (symbol, date, category))`,
|
|
45
|
+
`CREATE TABLE IF NOT EXISTS sector_members (
|
|
46
|
+
block_type VARCHAR, block_name VARCHAR, symbol VARCHAR, name VARCHAR, market VARCHAR, asof_date DATE,
|
|
47
|
+
PRIMARY KEY (block_type, block_name, symbol, asof_date))`,
|
|
48
|
+
`CREATE TABLE IF NOT EXISTS financial_basic (
|
|
49
|
+
symbol VARCHAR, fetched_at TIMESTAMP, eps DOUBLE, bvps DOUBLE, cfps DOUBLE,
|
|
50
|
+
capital_reserve DOUBLE, retained_profit DOUBLE, operating_income DOUBLE, net_profit DOUBLE,
|
|
51
|
+
gross_profit DOUBLE, total_assets DOUBLE, net_assets DOUBLE, debt_ratio DOUBLE,
|
|
52
|
+
roe DOUBLE, gross_margin DOUBLE, net_margin DOUBLE,
|
|
53
|
+
PRIMARY KEY (symbol, fetched_at))`,
|
|
54
|
+
`CREATE TABLE IF NOT EXISTS adjustment_factors (
|
|
55
|
+
symbol VARCHAR, effective_from TIMESTAMP, factor DOUBLE, source_date DATE,
|
|
56
|
+
source_category SMALLINT, adjust_type VARCHAR,
|
|
57
|
+
PRIMARY KEY (symbol, adjust_type, effective_from))`,
|
|
58
|
+
KLINE_QFQ_VIEW_DDL,
|
|
59
|
+
KLINE_HFQ_VIEW_DDL,
|
|
60
|
+
`CREATE TABLE IF NOT EXISTS local_manifest (
|
|
61
|
+
path VARCHAR PRIMARY KEY, sha256 VARCHAR, size BIGINT, row_count BIGINT,
|
|
62
|
+
cycle VARCHAR, year INTEGER, ts_min TIMESTAMP, ts_max TIMESTAMP, synced_at TIMESTAMP)`,
|
|
63
|
+
`CREATE TABLE IF NOT EXISTS sync_meta (key VARCHAR PRIMARY KEY, value VARCHAR)`,
|
|
64
|
+
];
|
|
65
|
+
export class DuckdbStore {
|
|
66
|
+
instance = null;
|
|
67
|
+
conn = null;
|
|
68
|
+
/** v2 parquet-direct:kline view 挂载根(<dataDir>/.cache/parquet)。 */
|
|
69
|
+
parquetRoot = null;
|
|
70
|
+
constructor() { }
|
|
71
|
+
static async open(path) {
|
|
72
|
+
const s = new DuckdbStore();
|
|
73
|
+
try {
|
|
74
|
+
s.instance = await DuckDBInstance.create(path);
|
|
75
|
+
s.conn = await s.instance.connect();
|
|
76
|
+
}
|
|
77
|
+
catch (e) {
|
|
78
|
+
const msg = e instanceof Error ? e.message : String(e);
|
|
79
|
+
if (msg.includes('Conflicting lock')) {
|
|
80
|
+
throw new Error(`tickdb 数据库被另一进程占用(如 python/streamlit 或另一个 Node 进程):${path}。停止对方进程后重试。`);
|
|
81
|
+
}
|
|
82
|
+
throw e;
|
|
83
|
+
}
|
|
84
|
+
// 时区 pin:DuckDB TIMESTAMPTZ->TIMESTAMP 按 session tz 剥离,必须 UTC(与 Rust 客户端一致,否则 bar 翻倍)
|
|
85
|
+
await s.conn.run(`SET timezone='UTC'`);
|
|
86
|
+
for (const ddl of SCHEMA_STATEMENTS)
|
|
87
|
+
await s.conn.run(ddl);
|
|
88
|
+
return s;
|
|
89
|
+
}
|
|
90
|
+
c() {
|
|
91
|
+
if (!this.conn)
|
|
92
|
+
throw new Error('store closed');
|
|
93
|
+
return this.conn;
|
|
94
|
+
}
|
|
95
|
+
async close() {
|
|
96
|
+
try {
|
|
97
|
+
this.conn?.closeSync();
|
|
98
|
+
}
|
|
99
|
+
catch { /* already closed */ }
|
|
100
|
+
this.conn = null;
|
|
101
|
+
try {
|
|
102
|
+
this.instance?.closeSync();
|
|
103
|
+
}
|
|
104
|
+
catch { /* already closed */ }
|
|
105
|
+
this.instance = null;
|
|
106
|
+
}
|
|
107
|
+
async run(sql, params) {
|
|
108
|
+
await this.c().run(sql, params);
|
|
109
|
+
}
|
|
110
|
+
async query(sql) {
|
|
111
|
+
await this.maybeRefreshKlineSurface();
|
|
112
|
+
const r = await this.c().run(sql);
|
|
113
|
+
const cols = r.columnNames();
|
|
114
|
+
const rows = await r.getRows();
|
|
115
|
+
return rows.map((row) => Object.fromEntries(cols.map((c, i) => [c, mapValue(row[i])])));
|
|
116
|
+
}
|
|
117
|
+
/** v2 parquet-direct 查询前置:glob 重展开(view 注册时展开 glob,新文件才
|
|
118
|
+
* 可见——与服务端 refresh_views 同因)。upsert 模式零开销直通。
|
|
119
|
+
* sync_meta 读取走原始连接(getSyncMeta → query → 这里,避免自递归)。 */
|
|
120
|
+
async maybeRefreshKlineSurface() {
|
|
121
|
+
if (!this.parquetRoot)
|
|
122
|
+
return;
|
|
123
|
+
const r = await this.c().run(`SELECT value FROM sync_meta WHERE key = '${escapeSql(STORE_MODE_KEY)}'`);
|
|
124
|
+
const rows = await r.getRows();
|
|
125
|
+
if (rows[0]?.[0] === 'parquet-direct') {
|
|
126
|
+
await this.refreshKlineView(this.parquetRoot);
|
|
127
|
+
}
|
|
128
|
+
}
|
|
129
|
+
setParquetRoot(root) {
|
|
130
|
+
this.parquetRoot = root;
|
|
131
|
+
}
|
|
132
|
+
/** v2 parquet-direct:kline 以 view 挂载 parquet 目录(hive_partitioning 从
|
|
133
|
+
* cycle={c} 目录名注入 cycle 列——parquet 本身无 cycle 列;与服务端
|
|
134
|
+
* query_engine 同款)。CREATE OR REPLACE 保证幂等。空目录(sync 前置调用)
|
|
135
|
+
* 建同 schema 空 view,read_parquet 空 glob 会 IO Error。 */
|
|
136
|
+
async refreshKlineView(cacheRoot) {
|
|
137
|
+
const glob = `${cacheRoot.replaceAll("'", '')}/kline-compacted/cycle=*/*.parquet`;
|
|
138
|
+
const r = await this.c().run(`SELECT COUNT(*) AS n FROM glob('${glob}')`);
|
|
139
|
+
const n = Number((await r.getRows())[0][0]);
|
|
140
|
+
if (n > 0) {
|
|
141
|
+
await this.c().run(`CREATE OR REPLACE VIEW kline AS
|
|
142
|
+
SELECT * REPLACE (CAST(ts AS TIMESTAMP) AS ts)
|
|
143
|
+
FROM read_parquet('${glob}', hive_partitioning=1)`);
|
|
144
|
+
}
|
|
145
|
+
else {
|
|
146
|
+
// 列序与 read_parquet+hive 一致(分区列 cycle 追加在末尾)
|
|
147
|
+
await this.c().run(`CREATE OR REPLACE VIEW kline AS
|
|
148
|
+
SELECT CAST(NULL AS TIMESTAMP) AS ts, CAST(NULL AS VARCHAR) AS symbol,
|
|
149
|
+
CAST(NULL AS DOUBLE) AS open, CAST(NULL AS DOUBLE) AS high,
|
|
150
|
+
CAST(NULL AS DOUBLE) AS low, CAST(NULL AS DOUBLE) AS close,
|
|
151
|
+
CAST(NULL AS BIGINT) AS volume, CAST(NULL AS DOUBLE) AS amount,
|
|
152
|
+
CAST(NULL AS VARCHAR) AS cycle
|
|
153
|
+
WHERE 1=0`);
|
|
154
|
+
}
|
|
155
|
+
await this.c().run(KLINE_QFQ_VIEW_DDL);
|
|
156
|
+
await this.c().run(KLINE_HFQ_VIEW_DDL);
|
|
157
|
+
}
|
|
158
|
+
/** 把 kline 表面(surface)对齐到当前模式:direct → read_parquet VIEW
|
|
159
|
+
* (v1 遗留 TABLE 先 DROP);upsert → TABLE(v2 遗留 VIEW 先 DROP 重建,
|
|
160
|
+
* 服务端回滚场景)。依赖视图 qfq/hfq 一并重建。幂等。 */
|
|
161
|
+
async ensureKlineSurface(mode, cacheRoot) {
|
|
162
|
+
// 原始连接查询(绕过 query() 的 surface 刷新,避免自激)
|
|
163
|
+
const r = await this.c().run(`SELECT table_type FROM information_schema.tables
|
|
164
|
+
WHERE table_schema='main' AND table_name='kline'`);
|
|
165
|
+
const rows = await r.getRows();
|
|
166
|
+
const ty = rows[0]?.[0];
|
|
167
|
+
if (mode === 'parquet-direct') {
|
|
168
|
+
if (ty === 'BASE TABLE') {
|
|
169
|
+
await this.run('DROP VIEW IF EXISTS kline_qfq; DROP VIEW IF EXISTS kline_hfq; DROP TABLE kline;');
|
|
170
|
+
}
|
|
171
|
+
await this.refreshKlineView(cacheRoot);
|
|
172
|
+
}
|
|
173
|
+
else if (ty === 'VIEW') {
|
|
174
|
+
await this.run('DROP VIEW IF EXISTS kline_qfq; DROP VIEW IF EXISTS kline_hfq; DROP VIEW kline;');
|
|
175
|
+
await this.run(KLINE_TABLE_DDL);
|
|
176
|
+
await this.run(KLINE_QFQ_VIEW_DDL);
|
|
177
|
+
await this.run(KLINE_HFQ_VIEW_DDL);
|
|
178
|
+
}
|
|
179
|
+
}
|
|
180
|
+
/** sync_meta 读取走原始连接(不经 query() 的 surface 刷新)。 */
|
|
181
|
+
async getSyncMeta(key) {
|
|
182
|
+
const r = await this.c().run(`SELECT value FROM sync_meta WHERE key = '${escapeSql(key)}'`);
|
|
183
|
+
const rows = await r.getRows();
|
|
184
|
+
return rows.length > 0 ? rows[0][0] : null;
|
|
185
|
+
}
|
|
186
|
+
async setSyncMeta(key, value) {
|
|
187
|
+
await this.run(`INSERT OR REPLACE INTO sync_meta (key, value) VALUES (?, ?)`, [key, value]);
|
|
188
|
+
}
|
|
189
|
+
async clearLocalManifest() {
|
|
190
|
+
await this.run('DELETE FROM local_manifest');
|
|
191
|
+
}
|
|
192
|
+
async countParquetRows(parquetPath) {
|
|
193
|
+
const r = await this.c().run(`SELECT COUNT(*) AS n FROM read_parquet('${escapeSql(parquetPath)}')`);
|
|
194
|
+
return Number((await r.getRows())[0][0]);
|
|
195
|
+
}
|
|
196
|
+
async upsertKlineFromFile(parquetPath, cycle) {
|
|
197
|
+
await this.run(`INSERT OR REPLACE INTO kline (ts, symbol, cycle, open, high, low, close, volume, amount)
|
|
198
|
+
SELECT ts, symbol, '${escapeSql(cycle)}', open, high, low, close, volume, amount
|
|
199
|
+
FROM read_parquet('${escapeSql(parquetPath)}')`);
|
|
200
|
+
}
|
|
201
|
+
async deleteKlineYear(cycle, year) {
|
|
202
|
+
const [lo, hi] = yearBounds(year);
|
|
203
|
+
await this.run(`DELETE FROM kline WHERE cycle = ? AND ts >= CAST(? AS TIMESTAMP) AND ts <= CAST(? AS TIMESTAMP)`, [cycle, lo, hi]);
|
|
204
|
+
}
|
|
205
|
+
/** 按文件 ts 边界删除(含端点,口径同 deleteKlineYear)。月切片 ts 互不重叠,不会误伤相邻月。 */
|
|
206
|
+
async deleteKlineRange(cycle, tsMin, tsMax) {
|
|
207
|
+
await this.run(`DELETE FROM kline WHERE cycle = ? AND ts >= CAST(? AS TIMESTAMP) AND ts <= CAST(? AS TIMESTAMP)`, [cycle, tsMin, tsMax]);
|
|
208
|
+
}
|
|
209
|
+
async listLocalManifest() {
|
|
210
|
+
const rows = await this.query('SELECT path, sha256 FROM local_manifest');
|
|
211
|
+
return new Map(rows.map((r) => [r.path, r.sha256]));
|
|
212
|
+
}
|
|
213
|
+
/** 全字段 local_manifest(stale per-file ts 边界删除用;NULL 边界 → null,调用方回退整年)。 */
|
|
214
|
+
async listLocalManifestFull() {
|
|
215
|
+
const rows = await this.query('SELECT path, sha256, cycle, year, ts_min, ts_max FROM local_manifest');
|
|
216
|
+
return new Map(rows.map((r) => [r.path, {
|
|
217
|
+
path: r.path,
|
|
218
|
+
sha256: r.sha256,
|
|
219
|
+
cycle: r.cycle,
|
|
220
|
+
year: r.year === null || r.year === undefined ? null : Number(r.year),
|
|
221
|
+
ts_min: r.ts_min === null || r.ts_min === undefined ? null : sqlTs(String(r.ts_min)),
|
|
222
|
+
ts_max: r.ts_max === null || r.ts_max === undefined ? null : sqlTs(String(r.ts_max)),
|
|
223
|
+
}]));
|
|
224
|
+
}
|
|
225
|
+
async upsertLocalManifest(f) {
|
|
226
|
+
await this.run(`INSERT OR REPLACE INTO local_manifest
|
|
227
|
+
(path, sha256, size, row_count, cycle, year, ts_min, ts_max, synced_at)
|
|
228
|
+
VALUES (?, ?, ?, ?, ?, ?, CAST(? AS TIMESTAMP), CAST(? AS TIMESTAMP), now())`, [f.path, f.sha256, f.size, f.row_count, f.cycle, f.year ?? null, f.ts_min, f.ts_max]);
|
|
229
|
+
}
|
|
230
|
+
async deleteLocalManifest(paths) {
|
|
231
|
+
for (let i = 0; i < paths.length; i += 500) {
|
|
232
|
+
const chunk = paths.slice(i, i + 500);
|
|
233
|
+
await this.run(`DELETE FROM local_manifest WHERE path IN (${chunk.map(() => '?').join(',')})`, chunk);
|
|
234
|
+
}
|
|
235
|
+
}
|
|
236
|
+
async getEtag() {
|
|
237
|
+
const rows = await this.query(`SELECT value FROM sync_meta WHERE key = 'manifest_etag'`);
|
|
238
|
+
return rows.length > 0 ? rows[0].value : null;
|
|
239
|
+
}
|
|
240
|
+
async setEtag(etag) {
|
|
241
|
+
await this.run(`INSERT OR REPLACE INTO sync_meta (key, value) VALUES ('manifest_etag', ?)`, [etag]);
|
|
242
|
+
}
|
|
243
|
+
}
|
|
244
|
+
function escapeSql(s) {
|
|
245
|
+
return s.replaceAll("'", "''");
|
|
246
|
+
}
|
|
247
|
+
/** mapValue 产出的 ISO('2026-02-05T15:00:00.000Z')→ SQL timestamp 字面量('2026-02-05 15:00:00')。 */
|
|
248
|
+
function sqlTs(iso) {
|
|
249
|
+
return iso.replace('T', ' ').replace(/(\.\d+)?Z$/, '');
|
|
250
|
+
}
|
|
251
|
+
/** DuckDB node-api 值 -> JSON 友好值(对齐 Python cell_to_json 行为)。 */
|
|
252
|
+
function mapValue(v) {
|
|
253
|
+
if (v === null || typeof v !== 'object') {
|
|
254
|
+
if (typeof v === 'bigint')
|
|
255
|
+
return Number(v);
|
|
256
|
+
return v;
|
|
257
|
+
}
|
|
258
|
+
const name = v.constructor?.name;
|
|
259
|
+
if (name === 'DuckDBTimestampValue' || name === 'DuckDBTimestampTZValue') {
|
|
260
|
+
return new Date(String(v).replace(' ', 'T') + 'Z').toISOString();
|
|
261
|
+
}
|
|
262
|
+
if (name === 'DuckDBDecimalValue')
|
|
263
|
+
return v.toDouble();
|
|
264
|
+
if (name === 'DuckDBDateValue' || name === 'DuckDBTimeValue' || name === 'DuckDBIntervalValue'
|
|
265
|
+
|| name === 'DuckDBUUIDValue' || name === 'DuckDBBigIntValue' || name === 'DuckDBHugeIntValue') {
|
|
266
|
+
return String(v);
|
|
267
|
+
}
|
|
268
|
+
return v;
|
|
269
|
+
}
|
package/dist/sync.d.ts
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
import type { DuckdbStore } from './store.js';
|
|
2
|
+
import type { TickdbHttp } from './http.js';
|
|
3
|
+
import type { SyncReport } from './types.js';
|
|
4
|
+
/** 一轮完整同步:manifest(v1.3 disjoint → 模式判定/切换) -> diff -> stale 删除
|
|
5
|
+
* (direct 删文件 / upsert per-file ts 边界删行)-> runPool 并发 fetch+apply
|
|
6
|
+
* (mirror-first 全量文件)-> 零错误才写 etag。 */
|
|
7
|
+
export declare function syncOnce(store: DuckdbStore, http: TickdbHttp, cacheDir: string, opts?: {
|
|
8
|
+
downloadConcurrency?: number;
|
|
9
|
+
}): Promise<SyncReport>;
|
package/dist/sync.js
ADDED
|
@@ -0,0 +1,122 @@
|
|
|
1
|
+
import { join } from 'node:path';
|
|
2
|
+
import { mkdirSync, rmSync } from 'node:fs';
|
|
3
|
+
import { STORE_MODE_KEY } from './store.js';
|
|
4
|
+
import { computeDiff } from './diff.js';
|
|
5
|
+
import { parseCycleYear, runPool } from './util.js';
|
|
6
|
+
/** v2 嵌套布局(镜像服务端路径,hive cycle 注入依赖 cycle={c} 目录层级)。 */
|
|
7
|
+
const cachePath = (cacheDir, p) => join(cacheDir, p);
|
|
8
|
+
/** 一轮完整同步:manifest(v1.3 disjoint → 模式判定/切换) -> diff -> stale 删除
|
|
9
|
+
* (direct 删文件 / upsert per-file ts 边界删行)-> runPool 并发 fetch+apply
|
|
10
|
+
* (mirror-first 全量文件)-> 零错误才写 etag。 */
|
|
11
|
+
export async function syncOnce(store, http, cacheDir, opts = {}) {
|
|
12
|
+
const report = {
|
|
13
|
+
notModified: false, filesTotal: 0, filesFetched: 0, filesUnchanged: 0,
|
|
14
|
+
rowsUpserted: 0, staleDeleted: 0, mirrorFetched: 0, mirrorFallbacks: 0, errors: [],
|
|
15
|
+
};
|
|
16
|
+
const etag = await store.getEtag();
|
|
17
|
+
const res = await http.fetchManifest(etag ?? undefined);
|
|
18
|
+
if (res.kind === 'not_modified') {
|
|
19
|
+
report.notModified = true;
|
|
20
|
+
report.filesTotal = (await store.listLocalManifest()).size;
|
|
21
|
+
report.filesUnchanged = report.filesTotal;
|
|
22
|
+
return report;
|
|
23
|
+
}
|
|
24
|
+
const { manifest, etag: newEtag } = res;
|
|
25
|
+
report.filesTotal = manifest.files.length;
|
|
26
|
+
// ① 模式判定 + 切换(v1.3 disjoint → parquet-direct:文件即行集不 upsert)。
|
|
27
|
+
const mode = manifest.disjoint === true ? 'parquet-direct' : 'duckdb-upsert';
|
|
28
|
+
const direct = mode === 'parquet-direct';
|
|
29
|
+
const prevMode = await store.getSyncMeta(STORE_MODE_KEY);
|
|
30
|
+
if (prevMode !== mode) {
|
|
31
|
+
// 布局全清重来:v1→v2 旧 flat 文件/表行全废;v2→v1 回退按 manifest 重拉重灌。
|
|
32
|
+
// 例外:本地无 store_mode(v1 客户端升级)且服务端仍 upsert —— 数据布局
|
|
33
|
+
// 未变,保留 local_manifest/缓存,避免一次无意义的全量重下。
|
|
34
|
+
const wipe = !(prevMode === null && !direct);
|
|
35
|
+
if (wipe) {
|
|
36
|
+
rmSync(cacheDir, { recursive: true, force: true });
|
|
37
|
+
mkdirSync(cacheDir, { recursive: true });
|
|
38
|
+
await store.clearLocalManifest();
|
|
39
|
+
}
|
|
40
|
+
await store.setSyncMeta(STORE_MODE_KEY, mode);
|
|
41
|
+
}
|
|
42
|
+
store.setParquetRoot(cacheDir);
|
|
43
|
+
await store.ensureKlineSurface(mode, cacheDir);
|
|
44
|
+
const diff = computeDiff(manifest.files, await store.listLocalManifest());
|
|
45
|
+
report.filesUnchanged = diff.unchanged;
|
|
46
|
+
// ② stale 先删(best-effort:单条失败记 error 不中断)。
|
|
47
|
+
// direct → 删本地文件(行随文件消失);upsert → 优先按该文件自己的 ts 边界删
|
|
48
|
+
// (月文件消失只删它那个月;边界缺失回退整年删除)。
|
|
49
|
+
const localFull = direct ? new Map() : await store.listLocalManifestFull();
|
|
50
|
+
const staleOk = [];
|
|
51
|
+
for (const p of diff.stale) {
|
|
52
|
+
try {
|
|
53
|
+
if (direct) {
|
|
54
|
+
rmSync(cachePath(cacheDir, p), { force: true });
|
|
55
|
+
}
|
|
56
|
+
else {
|
|
57
|
+
const { cycle, year } = parseCycleYear(p);
|
|
58
|
+
const row = localFull.get(p);
|
|
59
|
+
if (row?.ts_min && row?.ts_max) {
|
|
60
|
+
await store.deleteKlineRange(cycle, row.ts_min, row.ts_max);
|
|
61
|
+
}
|
|
62
|
+
else {
|
|
63
|
+
await store.deleteKlineYear(cycle, year);
|
|
64
|
+
}
|
|
65
|
+
}
|
|
66
|
+
staleOk.push(p);
|
|
67
|
+
}
|
|
68
|
+
catch (e) {
|
|
69
|
+
report.errors.push(`stale ${p}: ${e instanceof Error ? e.message : String(e)}`);
|
|
70
|
+
}
|
|
71
|
+
}
|
|
72
|
+
if (staleOk.length > 0) {
|
|
73
|
+
await store.deleteLocalManifest(staleOk);
|
|
74
|
+
report.staleDeleted = staleOk.length;
|
|
75
|
+
}
|
|
76
|
+
// ③ 并发 fetch + apply(单文件失败不中断整轮)。mirror-first:所有文件
|
|
77
|
+
// (v1.2 的 frozen gate 移除——上传面已扩到全部文件,sha 校验兜底);
|
|
78
|
+
// 镜像失败/非 200/sha 不符 → 回落 VPS(带重试),不计 error。
|
|
79
|
+
const mirrorBase = manifest.mirror_base_url?.replace(/\/$/, '');
|
|
80
|
+
await runPool(diff.toFetch, opts.downloadConcurrency ?? 4, async (f) => {
|
|
81
|
+
try {
|
|
82
|
+
const dest = cachePath(cacheDir, f.path);
|
|
83
|
+
mkdirSync(join(dest, '..'), { recursive: true });
|
|
84
|
+
let viaMirror = false;
|
|
85
|
+
if (mirrorBase) {
|
|
86
|
+
const mUrl = `${mirrorBase}/${f.path}`;
|
|
87
|
+
try {
|
|
88
|
+
await http.downloadMirrorToFile(mUrl, dest, { size: f.size, sha256: f.sha256 });
|
|
89
|
+
viaMirror = true;
|
|
90
|
+
report.mirrorFetched += 1;
|
|
91
|
+
}
|
|
92
|
+
catch {
|
|
93
|
+
report.mirrorFallbacks += 1; // 回落 VPS,镜像失败不算 error
|
|
94
|
+
}
|
|
95
|
+
}
|
|
96
|
+
if (!viaMirror) {
|
|
97
|
+
await http.downloadToFile(f.path, f.path, dest, { size: f.size, sha256: f.sha256 });
|
|
98
|
+
}
|
|
99
|
+
if (direct) {
|
|
100
|
+
// v2:文件即行集——不 upsert、不删文件
|
|
101
|
+
await store.upsertLocalManifest(f);
|
|
102
|
+
report.rowsUpserted += 0;
|
|
103
|
+
}
|
|
104
|
+
else {
|
|
105
|
+
const rows = await store.countParquetRows(dest);
|
|
106
|
+
await store.upsertKlineFromFile(dest, f.cycle);
|
|
107
|
+
await store.upsertLocalManifest(f);
|
|
108
|
+
rmSync(dest, { force: true });
|
|
109
|
+
report.rowsUpserted += rows;
|
|
110
|
+
}
|
|
111
|
+
report.filesFetched += 1;
|
|
112
|
+
}
|
|
113
|
+
catch (e) {
|
|
114
|
+
report.errors.push(`${f.path}: ${e instanceof Error ? e.message : String(e)}`);
|
|
115
|
+
}
|
|
116
|
+
});
|
|
117
|
+
// ④ 仅零错误轮写 etag(304 => 本地确实与该 manifest 一致)
|
|
118
|
+
if (report.errors.length === 0 && newEtag) {
|
|
119
|
+
await store.setEtag(newEtag);
|
|
120
|
+
}
|
|
121
|
+
return report;
|
|
122
|
+
}
|
package/dist/types.d.ts
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
/** manifest FileEntry 字段与线上 JSON 完全一致(snake_case)。 */
|
|
2
|
+
export interface FileEntry {
|
|
3
|
+
path: string;
|
|
4
|
+
sha256: string;
|
|
5
|
+
size: number;
|
|
6
|
+
row_count: number;
|
|
7
|
+
symbol: string;
|
|
8
|
+
cycle: string;
|
|
9
|
+
ts_min: string;
|
|
10
|
+
ts_max: string;
|
|
11
|
+
year?: number;
|
|
12
|
+
symbols_count?: number;
|
|
13
|
+
/** v1.2:文件已冻结上云(内容永不再变,可从镜像下载)。 */
|
|
14
|
+
frozen?: boolean;
|
|
15
|
+
}
|
|
16
|
+
export interface Manifest {
|
|
17
|
+
version: string;
|
|
18
|
+
schema: string;
|
|
19
|
+
generatedAt: string;
|
|
20
|
+
files: FileEntry[];
|
|
21
|
+
/** v1.2:对象存储公开基底 URL(未启用 mirror 时缺省)。 */
|
|
22
|
+
mirror_base_url?: string;
|
|
23
|
+
/** v1.3:服务端 compacted 文件 (symbol, ts) 互斥(重分区完成)。 */
|
|
24
|
+
disjoint?: boolean;
|
|
25
|
+
}
|
|
26
|
+
export interface InitOptions {
|
|
27
|
+
dataDir?: string;
|
|
28
|
+
token?: string;
|
|
29
|
+
serverUrl?: string;
|
|
30
|
+
autoSync?: boolean;
|
|
31
|
+
syncIntervalSecs?: number;
|
|
32
|
+
downloadConcurrency?: number;
|
|
33
|
+
}
|
|
34
|
+
export interface SyncReport {
|
|
35
|
+
notModified: boolean;
|
|
36
|
+
filesTotal: number;
|
|
37
|
+
filesFetched: number;
|
|
38
|
+
filesUnchanged: number;
|
|
39
|
+
rowsUpserted: number;
|
|
40
|
+
staleDeleted: number;
|
|
41
|
+
mirrorFetched: number;
|
|
42
|
+
mirrorFallbacks: number;
|
|
43
|
+
errors: string[];
|
|
44
|
+
}
|
|
45
|
+
export interface SyncStatus {
|
|
46
|
+
running: boolean;
|
|
47
|
+
startedAt: number;
|
|
48
|
+
finishedAt: number;
|
|
49
|
+
filesTotal: number;
|
|
50
|
+
filesFetched: number;
|
|
51
|
+
filesUnchanged: number;
|
|
52
|
+
rowsUpserted: number;
|
|
53
|
+
staleDeleted: number;
|
|
54
|
+
errorsCount: number;
|
|
55
|
+
daemonIntervalSecs: number;
|
|
56
|
+
daemonRunning: boolean;
|
|
57
|
+
lastError: string | null;
|
|
58
|
+
}
|
package/dist/types.js
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
export {};
|
package/dist/util.d.ts
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
/** 固定并发工作池:items 逐个领取,任何 fn 抛错由 fn 自身捕获(调用方负责 per-item try/catch)。 */
|
|
2
|
+
export declare function runPool<T>(items: readonly T[], limit: number, fn: (item: T) => Promise<void>): Promise<void>;
|
|
3
|
+
export declare function parseCycleYear(path: string): {
|
|
4
|
+
cycle: string;
|
|
5
|
+
year: number;
|
|
6
|
+
};
|
|
7
|
+
export declare function yearBounds(year: number): [string, string];
|
|
8
|
+
export declare function expandHome(p: string): string;
|
|
9
|
+
export declare const nowSec: () => number;
|
|
10
|
+
export declare const sleep: (ms: number) => Promise<void>;
|
package/dist/util.js
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
import { homedir } from 'node:os';
|
|
2
|
+
import { join } from 'node:path';
|
|
3
|
+
/** 固定并发工作池:items 逐个领取,任何 fn 抛错由 fn 自身捕获(调用方负责 per-item try/catch)。 */
|
|
4
|
+
export async function runPool(items, limit, fn) {
|
|
5
|
+
let i = 0;
|
|
6
|
+
const n = Math.max(1, Math.min(limit, items.length));
|
|
7
|
+
const workers = Array.from({ length: n }, async () => {
|
|
8
|
+
while (i < items.length) {
|
|
9
|
+
const idx = i++;
|
|
10
|
+
await fn(items[idx]);
|
|
11
|
+
}
|
|
12
|
+
});
|
|
13
|
+
await Promise.all(workers);
|
|
14
|
+
}
|
|
15
|
+
export function parseCycleYear(path) {
|
|
16
|
+
const m = /cycle=([^/]+)\/year=(\d{4})(?:-month=\d{2})?\.parquet$/.exec(path);
|
|
17
|
+
if (!m)
|
|
18
|
+
throw new Error(`cannot parse cycle/year from manifest path: ${path}`);
|
|
19
|
+
return { cycle: m[1], year: Number(m[2]) };
|
|
20
|
+
}
|
|
21
|
+
export function yearBounds(year) {
|
|
22
|
+
return [`${year}-01-01 00:00:00`, `${year}-12-31 23:59:59`];
|
|
23
|
+
}
|
|
24
|
+
export function expandHome(p) {
|
|
25
|
+
if (p === '~')
|
|
26
|
+
return homedir();
|
|
27
|
+
if (p.startsWith('~/'))
|
|
28
|
+
return join(homedir(), p.slice(2));
|
|
29
|
+
return p;
|
|
30
|
+
}
|
|
31
|
+
export const nowSec = () => Math.floor(Date.now() / 1000);
|
|
32
|
+
export const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
|
package/package.json
CHANGED
|
@@ -1,6 +1,29 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "tickdb-ts",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"
|
|
5
|
-
"
|
|
6
|
-
|
|
3
|
+
"version": "0.2.0",
|
|
4
|
+
"description": "offline-first TypeScript client for tickdb (A-share market data): manifest sync + local DuckDB",
|
|
5
|
+
"type": "module",
|
|
6
|
+
"main": "dist/index.js",
|
|
7
|
+
"types": "dist/index.d.ts",
|
|
8
|
+
"exports": { ".": { "import": "./dist/index.js", "types": "./dist/index.d.ts" } },
|
|
9
|
+
"files": ["dist", "README.md"],
|
|
10
|
+
"engines": { "node": ">=18" },
|
|
11
|
+
"scripts": {
|
|
12
|
+
"build": "tsc",
|
|
13
|
+
"typecheck": "tsc --noEmit",
|
|
14
|
+
"test": "tsx --test test/*.test.ts",
|
|
15
|
+
"prepublishOnly": "npm run build && npm test"
|
|
16
|
+
},
|
|
17
|
+
"repository": {
|
|
18
|
+
"type": "git",
|
|
19
|
+
"url": "git+https://github.com/chuanlei-coding/growinpublic.git",
|
|
20
|
+
"directory": "tickdb/clients/tickdb-ts"
|
|
21
|
+
},
|
|
22
|
+
"license": "MIT",
|
|
23
|
+
"dependencies": { "@duckdb/node-api": "^2.0.0-r.1" },
|
|
24
|
+
"devDependencies": {
|
|
25
|
+
"@types/node": "^22",
|
|
26
|
+
"tsx": "^4",
|
|
27
|
+
"typescript": "^5"
|
|
28
|
+
}
|
|
29
|
+
}
|