tickdb-ts 0.0.0-stage → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +165 -2
- package/dist/config.d.ts +12 -0
- package/dist/config.js +24 -0
- package/dist/daemon.d.ts +11 -0
- package/dist/daemon.js +32 -0
- package/dist/diff.d.ts +8 -0
- package/dist/diff.js +16 -0
- package/dist/http.d.ts +32 -0
- package/dist/http.js +98 -0
- package/dist/index.d.ts +27 -0
- package/dist/index.js +109 -0
- package/dist/manifest.d.ts +3 -0
- package/dist/manifest.js +42 -0
- package/dist/status.d.ts +8 -0
- package/dist/status.js +16 -0
- package/dist/store.d.ts +100 -0
- package/dist/store.js +401 -0
- package/dist/sync.d.ts +10 -0
- package/dist/sync.js +142 -0
- package/dist/types.d.ts +61 -0
- package/dist/types.js +1 -0
- package/dist/util.d.ts +14 -0
- package/dist/util.js +38 -0
- package/package.json +27 -4
package/README.md
CHANGED
|
@@ -1,3 +1,166 @@
|
|
|
1
|
-
#
|
|
1
|
+
# tickdb-ts
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
offline-first A 股 K 线客户端(TypeScript / Node)。服务端(`growinpublic.ai/tickdb`)从通达信拉行情写 parquet,本客户端在本地维护 DuckDB,增量同步、离线可查——与 Rust CLI / Python 绑定同款体验。
|
|
4
|
+
|
|
5
|
+
## 安装
|
|
6
|
+
|
|
7
|
+
要求 Node ≥ 18(开发机验证:Node 22)。
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
# 本仓库内开发
|
|
11
|
+
cd tickdb/clients/tickdb-ts
|
|
12
|
+
npm install
|
|
13
|
+
npm run build # 产出 dist/
|
|
14
|
+
|
|
15
|
+
# 在你自己的项目里使用
|
|
16
|
+
npm install /path/to/growinpublic/tickdb/clients/tickdb-ts
|
|
17
|
+
# 或打 tarball: npm pack → npm install tickdb-ts-0.1.0.tgz
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## Token
|
|
21
|
+
|
|
22
|
+
`tik_` 开头的 opaque 字符串,`read` scope 即可。获取方式:
|
|
23
|
+
|
|
24
|
+
- 主站 `/console`(admin 在页面上签发/吊销 tickdb token)
|
|
25
|
+
- 或找管理员执行 `tickdb token-issue --label <名字> --scopes read`
|
|
26
|
+
|
|
27
|
+
⚠️ token 明文存在 `<dataDir>/config.json`,注意目录权限;泄露了让管理员吊销重发。
|
|
28
|
+
|
|
29
|
+
## 五分钟上手
|
|
30
|
+
|
|
31
|
+
```ts
|
|
32
|
+
import tickdb from 'tickdb-ts';
|
|
33
|
+
|
|
34
|
+
await tickdb.init({ token: 'tik_...' }); // 默认 dataDir ~/.tickdb;autoSync 默认 true(300s daemon + 立即一轮)
|
|
35
|
+
await tickdb.sync(); // 显式同步一轮(首次全量 ~1GB,耐心等)
|
|
36
|
+
|
|
37
|
+
const rows = await tickdb.query(`
|
|
38
|
+
SELECT ts, close, volume FROM kline
|
|
39
|
+
WHERE symbol = 'SH600000' AND cycle = 'D'
|
|
40
|
+
ORDER BY ts DESC LIMIT 5`);
|
|
41
|
+
console.log(rows);
|
|
42
|
+
// [{ ts: '2026-10-22T15:00:00.000Z', close: 12.34, volume: 5678900 }, ...]
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
不想自动同步:`init({ token, autoSync: false })`,之后自己调 `await tickdb.sync()`。
|
|
46
|
+
|
|
47
|
+
## API 参考
|
|
48
|
+
|
|
49
|
+
| 方法 | 说明 |
|
|
50
|
+
|---|---|
|
|
51
|
+
| `init(opts?)` | 初始化。`opts`: `dataDir`(默认 `~/.tickdb`)、`token`、`serverUrl`(默认生产地址)、`autoSync`(默认 `true`)、`syncIntervalSecs`(默认 300)。幂等——重复调用直接 return。autoSync 时起后台 daemon 并立即后台同步一轮 |
|
|
52
|
+
| `sync()` | 显式同步一轮,`await` 完成返回 `SyncReport`;**错误上抛**。并发调用第二个直接抛 `sync already running` |
|
|
53
|
+
| `syncStart()` | 后台立即一轮(fire-and-forget),错误只进 `syncStatus().lastError`,进程不 crash |
|
|
54
|
+
| `syncDaemonStart(intervalSecs?)` | 起/重启后台 daemon(默认 300s,强制 ≥ 60s) |
|
|
55
|
+
| `syncDaemonStop()` | 停 daemon |
|
|
56
|
+
| `syncStatus()` | 返回 `SyncStatus` 快照(见下),反映**最近完成的一轮** |
|
|
57
|
+
| `query(sql)` | 对本地 DuckDB 跑任意 SQL,返回 `Record<string, unknown>[]`;TIMESTAMP → ISO 字符串、DATE → `YYYY-MM-DD`、数值 → number |
|
|
58
|
+
| `close()` | 停 daemon + 关 DuckDB(进程退出前可选调用) |
|
|
59
|
+
|
|
60
|
+
```ts
|
|
61
|
+
interface SyncReport {
|
|
62
|
+
notModified: boolean; // true = ETag 304 整轮 no-op
|
|
63
|
+
filesTotal: number; filesFetched: number; filesUnchanged: number;
|
|
64
|
+
rowsUpserted: number; staleDeleted: number;
|
|
65
|
+
mirrorFetched: number; mirrorFallbacks: number; // v1.2:镜像命中 / 回落 VPS 计数
|
|
66
|
+
errors: string[]; // 单文件失败不中断整轮
|
|
67
|
+
}
|
|
68
|
+
interface SyncStatus {
|
|
69
|
+
running: boolean; startedAt: number; finishedAt: number; // unix 秒
|
|
70
|
+
filesTotal: number; filesFetched: number; filesUnchanged: number;
|
|
71
|
+
rowsUpserted: number; staleDeleted: number; errorsCount: number;
|
|
72
|
+
daemonIntervalSecs: number; daemonRunning: boolean;
|
|
73
|
+
lastError: string | null;
|
|
74
|
+
}
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
## 本地数据
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
~/.tickdb/ (或你传的 dataDir)
|
|
81
|
+
├── config.json # token/serverUrl/interval
|
|
82
|
+
├── tickdb.duckdb # 本地 DuckDB(数据 + local_manifest + sync_meta)
|
|
83
|
+
└── .cache/parquet/ # 下载暂存,upsert 成功后即删,正常为空
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
**kline 表**(主键 `symbol, cycle, ts`):
|
|
87
|
+
|
|
88
|
+
| 列 | 类型 | 说明 |
|
|
89
|
+
|---|---|---|
|
|
90
|
+
| ts | TIMESTAMP | bar 时间(A 股收盘 15:00 的日线 ts 就是 15:00:00,见 FAQ) |
|
|
91
|
+
| symbol | VARCHAR | `SH600000` / `SZ000001` 风格 |
|
|
92
|
+
| cycle | VARCHAR | `1m / 5m / 15m / 30m / 60m / D` |
|
|
93
|
+
| open/high/low/close | DOUBLE | 价格 |
|
|
94
|
+
| volume | BIGINT | 成交量(股) |
|
|
95
|
+
| amount | DOUBLE | 成交额 |
|
|
96
|
+
|
|
97
|
+
另有 `local_manifest`(已同步文件清单)、`sync_meta`(manifest ETag),以及与 Rust 客户端对齐的 extra 表和 `kline_qfq` / `kline_hfq` 复权视图——**v1 同步协议只覆盖 kline**,extra 表(含 `adjustment_factors`)没有数据,复权视图的 factor 恒为 1(等价不复权),后续版本再开。
|
|
98
|
+
|
|
99
|
+
常用 SQL:
|
|
100
|
+
|
|
101
|
+
```sql
|
|
102
|
+
-- 个股近 30 个交易日日线
|
|
103
|
+
SELECT ts, open, high, low, close, volume FROM kline
|
|
104
|
+
WHERE symbol = 'SH600000' AND cycle = 'D'
|
|
105
|
+
AND ts >= NOW()::TIMESTAMP - INTERVAL '30 DAY'
|
|
106
|
+
ORDER BY ts;
|
|
107
|
+
|
|
108
|
+
-- 全市场今日(最新一天)涨幅榜前 10
|
|
109
|
+
SELECT symbol, close / lag_close - 1 AS pct
|
|
110
|
+
FROM (
|
|
111
|
+
SELECT symbol, close, LAG(close) OVER (PARTITION BY symbol ORDER BY ts) AS lag_close,
|
|
112
|
+
ROW_NUMBER() OVER (PARTITION BY symbol ORDER BY ts DESC) AS rn
|
|
113
|
+
FROM kline WHERE cycle = 'D'
|
|
114
|
+
) WHERE rn = 1 AND lag_close IS NOT NULL
|
|
115
|
+
ORDER BY pct DESC LIMIT 10;
|
|
116
|
+
|
|
117
|
+
-- 同步状态
|
|
118
|
+
SELECT COUNT(*) AS files, MAX(synced_at) AS last_sync FROM local_manifest;
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
## 同步机制(简版)
|
|
122
|
+
|
|
123
|
+
1. `GET /api/v1/manifest?version=v1.2` 带 `If-None-Match`,未变收 304 整轮 no-op
|
|
124
|
+
2. 与本地 `local_manifest` 按 `path + sha256` 求 diff
|
|
125
|
+
3. 服务端消失的文件(stale)优先按该文件自己的 `ts_min..ts_max` 边界删本地行(月文件消失只删它那个月);边界缺失回退整年删除
|
|
126
|
+
4. 串行下载变化的 parquet:流式写 `.part` → size + sha256 双校验 → 原子 rename(3 次重试线性退避)。frozen 文件先试镜像(manifest v1.2 下发的公开 URL,无认证),失败/污染回落 VPS 不计错误
|
|
127
|
+
5. `INSERT OR REPLACE` 进 kline 表——**union 语义**:同主键的修正 bar 覆盖旧值;新文件里消失的行保留本地
|
|
128
|
+
6. 只有整轮零错误才写回 ETag(保证 304 ⇒ 本地确实与服务端一致)
|
|
129
|
+
|
|
130
|
+
## FAQ
|
|
131
|
+
|
|
132
|
+
**Q: ts 为什么日线是 15:00:00?**
|
|
133
|
+
A 股收盘 15:00 的墙钟时间。服务端 parquet 的时间戳是 `TIMESTAMPTZ`,本客户端打开 DuckDB 后第一句就 `SET timezone='UTC'` 再转 TIMESTAMP——和 Rust/Python 客户端同一套时区 pin 逻辑,保证去重主键一致、bar 不翻倍。查询时拿到的 ISO 字符串(带 Z)就是当日墙钟时间。
|
|
134
|
+
|
|
135
|
+
**Q: 报「tickdb 数据库被另一进程占用」?**
|
|
136
|
+
DuckDB 是单写者。同一份 `tickdb.duckdb` 被别的进程开着(python/streamlit/另一个 Node 进程)就会锁冲突。停掉对方进程,或者给本客户端换个 `dataDir`。
|
|
137
|
+
|
|
138
|
+
**Q: securities / dividend / financial_basic 这些表怎么没数据?**
|
|
139
|
+
服务端 manifest(v1/v1.1/v1.2)只覆盖 kline(K 线)。Python/Rust 客户端同样如此。要查财报/分红请走服务端 HTTP API(`POST /tickdb/api/v1/query`)或等后续版本扩展同步协议。
|
|
140
|
+
|
|
141
|
+
**Q: 断网能用吗?**
|
|
142
|
+
能。`query()` 全部走本地 DuckDB。`sync()` 会报错(显式调用错误上抛);后台 daemon 的错误只记在 `syncStatus().lastError`,恢复后下一轮自动追上。
|
|
143
|
+
|
|
144
|
+
**Q: 首次同步要多久 / 占多大空间?**
|
|
145
|
+
当前生产数据 ~60+ 个 parquet / ~1GB / 9000 万+ bar,串行下载,跨国链路大约 10-40 分钟。之后每轮只重拉 sha 变化的文件(通常只有当年/当月的几个)。
|
|
146
|
+
|
|
147
|
+
**Q: 和 Rust CLI / Python 绑定能用同一个 dataDir 吗?**
|
|
148
|
+
不建议。三者 schema 一致,但 DuckDB 单写者 + 版本差异,先后打开同一 db 文件容易锁冲突。给 TS 客户端单独一个 `dataDir` 最省心。
|
|
149
|
+
|
|
150
|
+
## 开发
|
|
151
|
+
|
|
152
|
+
```bash
|
|
153
|
+
npm test # node:test + tsx,43 个测试(含 fixture 服务器 + 镜像回落集成测试)
|
|
154
|
+
npm run typecheck # tsc --noEmit
|
|
155
|
+
npm run build # tsc -> dist/
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
协议实现与 Rust 客户端(`crates/tickdb-client`)逐条对照:
|
|
159
|
+
|
|
160
|
+
| 本包 | Rust 对照 |
|
|
161
|
+
|---|---|
|
|
162
|
+
| `src/http.ts` | `http_client.rs`(ETag/304、流式下载校验) |
|
|
163
|
+
| `src/sync.ts` | `manifest_sync.rs`(syncOnce 全流程) |
|
|
164
|
+
| `src/diff.ts` | `manifest_sync.rs::compute_diff` |
|
|
165
|
+
| `src/store.ts` | `duckdb_store.rs`(schema/upsert/时区 pin) |
|
|
166
|
+
| `src/daemon.ts` + `src/status.ts` | `tickdb-python/src/lib.rs` 的 daemon/atomics |
|
package/dist/config.d.ts
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
import type { InitOptions } from './types.js';
|
|
2
|
+
export declare const DEFAULT_SERVER_URL = "https://growinpublic.ai/tickdb";
|
|
3
|
+
export interface ResolvedConfig {
|
|
4
|
+
dataDir: string;
|
|
5
|
+
token: string;
|
|
6
|
+
serverUrl: string;
|
|
7
|
+
syncIntervalSecs: number;
|
|
8
|
+
downloadConcurrency: number;
|
|
9
|
+
}
|
|
10
|
+
export declare function configPath(dataDir: string): string;
|
|
11
|
+
export declare function loadMergedConfig(dataDir: string, opts: InitOptions): ResolvedConfig;
|
|
12
|
+
export declare function saveConfig(cfg: ResolvedConfig): void;
|
package/dist/config.js
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
import { readFileSync, writeFileSync } from 'node:fs';
|
|
2
|
+
import { join } from 'node:path';
|
|
3
|
+
export const DEFAULT_SERVER_URL = 'https://growinpublic.ai/tickdb';
|
|
4
|
+
export function configPath(dataDir) {
|
|
5
|
+
return join(dataDir, 'config.json');
|
|
6
|
+
}
|
|
7
|
+
export function loadMergedConfig(dataDir, opts) {
|
|
8
|
+
let saved = {};
|
|
9
|
+
try {
|
|
10
|
+
saved = JSON.parse(readFileSync(configPath(dataDir), 'utf8'));
|
|
11
|
+
}
|
|
12
|
+
catch { /* absent or corrupt: start fresh */ }
|
|
13
|
+
const token = opts.token && opts.token.length > 0 ? opts.token : (saved.token ?? '');
|
|
14
|
+
const serverUrl = opts.serverUrl && opts.serverUrl.length > 0 ? opts.serverUrl : (saved.serverUrl ?? DEFAULT_SERVER_URL);
|
|
15
|
+
const syncIntervalSecs = opts.syncIntervalSecs ?? saved.syncIntervalSecs ?? 300;
|
|
16
|
+
const downloadConcurrency = Math.min(6, Math.max(1, opts.downloadConcurrency ?? saved.downloadConcurrency ?? 4));
|
|
17
|
+
return { dataDir, token, serverUrl, syncIntervalSecs, downloadConcurrency };
|
|
18
|
+
}
|
|
19
|
+
export function saveConfig(cfg) {
|
|
20
|
+
writeFileSync(configPath(cfg.dataDir), JSON.stringify({
|
|
21
|
+
token: cfg.token, serverUrl: cfg.serverUrl, syncIntervalSecs: cfg.syncIntervalSecs,
|
|
22
|
+
downloadConcurrency: cfg.downloadConcurrency,
|
|
23
|
+
}, null, 2));
|
|
24
|
+
}
|
package/dist/daemon.d.ts
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
import type { SyncStateHolder } from './status.js';
|
|
2
|
+
export interface DaemonHandle {
|
|
3
|
+
tickNow(): Promise<void>;
|
|
4
|
+
stop(): void;
|
|
5
|
+
}
|
|
6
|
+
/** setInterval(unref) + generation 计数;tickNow 供测试与「立即触发」。 */
|
|
7
|
+
export declare function createDaemon(opts: {
|
|
8
|
+
onTick: () => Promise<void>;
|
|
9
|
+
intervalSecs: number;
|
|
10
|
+
state: SyncStateHolder;
|
|
11
|
+
}): DaemonHandle;
|
package/dist/daemon.js
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
let generation = 0;
|
|
2
|
+
/** setInterval(unref) + generation 计数;tickNow 供测试与「立即触发」。 */
|
|
3
|
+
export function createDaemon(opts) {
|
|
4
|
+
generation += 1;
|
|
5
|
+
const myGen = generation;
|
|
6
|
+
const intervalSecs = Math.max(60, opts.intervalSecs);
|
|
7
|
+
opts.state.update({ daemonIntervalSecs: intervalSecs, daemonRunning: true });
|
|
8
|
+
const runTick = async () => {
|
|
9
|
+
if (myGen !== generation)
|
|
10
|
+
return;
|
|
11
|
+
if (opts.state.get().running)
|
|
12
|
+
return;
|
|
13
|
+
try {
|
|
14
|
+
await opts.onTick();
|
|
15
|
+
}
|
|
16
|
+
catch {
|
|
17
|
+
// onTick 自己负责把错误写进 state.lastError;daemon 永不 crash 进程
|
|
18
|
+
}
|
|
19
|
+
};
|
|
20
|
+
const timer = setInterval(() => void runTick(), intervalSecs * 1000);
|
|
21
|
+
timer.unref();
|
|
22
|
+
return {
|
|
23
|
+
tickNow: runTick,
|
|
24
|
+
stop() {
|
|
25
|
+
if (myGen !== generation)
|
|
26
|
+
return;
|
|
27
|
+
generation += 1;
|
|
28
|
+
clearInterval(timer);
|
|
29
|
+
opts.state.update({ daemonRunning: false, daemonIntervalSecs: 0 });
|
|
30
|
+
},
|
|
31
|
+
};
|
|
32
|
+
}
|
package/dist/diff.d.ts
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
import type { FileEntry } from './types.js';
|
|
2
|
+
export interface ManifestDiff {
|
|
3
|
+
toFetch: FileEntry[];
|
|
4
|
+
stale: string[];
|
|
5
|
+
unchanged: number;
|
|
6
|
+
}
|
|
7
|
+
/** 纯函数:只按 path + sha256 判定(与 Rust 客户端一致)。local: path -> sha256。 */
|
|
8
|
+
export declare function computeDiff(serverFiles: FileEntry[], local: Map<string, string>): ManifestDiff;
|
package/dist/diff.js
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
/** 纯函数:只按 path + sha256 判定(与 Rust 客户端一致)。local: path -> sha256。 */
|
|
2
|
+
export function computeDiff(serverFiles, local) {
|
|
3
|
+
const serverPaths = new Set(serverFiles.map((f) => f.path));
|
|
4
|
+
const toFetch = [];
|
|
5
|
+
let unchanged = 0;
|
|
6
|
+
for (const f of serverFiles) {
|
|
7
|
+
if (local.get(f.path) === f.sha256)
|
|
8
|
+
unchanged++;
|
|
9
|
+
else
|
|
10
|
+
toFetch.push(f);
|
|
11
|
+
}
|
|
12
|
+
const stale = [...local.keys()].filter((p) => !serverPaths.has(p));
|
|
13
|
+
toFetch.sort((a, b) => (a.path < b.path ? -1 : a.path > b.path ? 1 : 0));
|
|
14
|
+
stale.sort();
|
|
15
|
+
return { toFetch, stale, unchanged };
|
|
16
|
+
}
|
package/dist/http.d.ts
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
import type { Manifest } from './types.js';
|
|
2
|
+
export type ManifestResult = {
|
|
3
|
+
kind: 'not_modified';
|
|
4
|
+
} | {
|
|
5
|
+
kind: 'fresh';
|
|
6
|
+
manifest: Manifest;
|
|
7
|
+
etag?: string;
|
|
8
|
+
};
|
|
9
|
+
/** 下载总超时:保底 10 分钟,按 64 KiB/s 地板速率随文件大小上浮(慢链路上大文件不会被误杀)。 */
|
|
10
|
+
export declare function downloadTimeoutMs(size: number): number;
|
|
11
|
+
export declare class TickdbHttp {
|
|
12
|
+
private readonly serverUrl;
|
|
13
|
+
private readonly token;
|
|
14
|
+
constructor(serverUrl: string, token: string);
|
|
15
|
+
private headers;
|
|
16
|
+
fetchManifest(ifNoneMatch?: string): Promise<ManifestResult>;
|
|
17
|
+
/** 单文件:流式写 .part -> size+sha256 校验 -> 原子 rename;失败重试 attempts 次(线性退避)。调用方负责串行。 */
|
|
18
|
+
downloadToFile(label: string, urlPath: string, dest: string, expect: {
|
|
19
|
+
size: number;
|
|
20
|
+
sha256: string;
|
|
21
|
+
}, opts?: {
|
|
22
|
+
attempts?: number;
|
|
23
|
+
backoffMs?: number;
|
|
24
|
+
}): Promise<void>;
|
|
25
|
+
/** 镜像下载:无认证绝对 URL,单次尝试(失败由调用方回落 VPS,VPS 的重试循环就是重试)。sha/size 校验与原子 rename 复用同一内核。 */
|
|
26
|
+
downloadMirrorToFile(url: string, dest: string, expect: {
|
|
27
|
+
size: number;
|
|
28
|
+
sha256: string;
|
|
29
|
+
}): Promise<void>;
|
|
30
|
+
private tryDownload;
|
|
31
|
+
private tryDownloadUrl;
|
|
32
|
+
}
|
package/dist/http.js
ADDED
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
import { createHash } from 'node:crypto';
|
|
2
|
+
import { createWriteStream, renameSync, rmSync } from 'node:fs';
|
|
3
|
+
import { Readable, Transform } from 'node:stream';
|
|
4
|
+
import { pipeline } from 'node:stream/promises';
|
|
5
|
+
import { parseManifest } from './manifest.js';
|
|
6
|
+
import { sleep } from './util.js';
|
|
7
|
+
/** 下载总超时:保底 10 分钟,按 64 KiB/s 地板速率随文件大小上浮(慢链路上大文件不会被误杀)。 */
|
|
8
|
+
export function downloadTimeoutMs(size) {
|
|
9
|
+
return Math.max(600_000, Math.ceil(size / 65_536) * 1000);
|
|
10
|
+
}
|
|
11
|
+
export class TickdbHttp {
|
|
12
|
+
serverUrl;
|
|
13
|
+
token;
|
|
14
|
+
constructor(serverUrl, token) {
|
|
15
|
+
this.serverUrl = serverUrl;
|
|
16
|
+
this.token = token;
|
|
17
|
+
}
|
|
18
|
+
headers() {
|
|
19
|
+
return { Authorization: `Bearer ${this.token}` };
|
|
20
|
+
}
|
|
21
|
+
async fetchManifest(ifNoneMatch) {
|
|
22
|
+
const headers = this.headers();
|
|
23
|
+
if (ifNoneMatch)
|
|
24
|
+
headers['If-None-Match'] = ifNoneMatch;
|
|
25
|
+
// v1.4 = v1.3 + extra/<name>.parquet 聚合数据集条目(disjoint 门保留)。
|
|
26
|
+
// pre-v1.4 服务端对未知 version 走默认分支回 v1.1 body——parseManifest
|
|
27
|
+
// 全版本兼容;syncOnce 的版本门保证旧 body 不翻转 store_mode。
|
|
28
|
+
const res = await fetch(`${this.serverUrl}/api/v1/manifest?version=v1.4`, {
|
|
29
|
+
headers,
|
|
30
|
+
signal: AbortSignal.timeout(60_000),
|
|
31
|
+
});
|
|
32
|
+
if (res.status === 304)
|
|
33
|
+
return { kind: 'not_modified' };
|
|
34
|
+
if (!res.ok)
|
|
35
|
+
throw new Error(`manifest HTTP ${res.status}: ${(await res.text()).slice(0, 200)}`);
|
|
36
|
+
const etag = res.headers.get('etag') ?? undefined;
|
|
37
|
+
return { kind: 'fresh', manifest: parseManifest(await res.json()), etag };
|
|
38
|
+
}
|
|
39
|
+
/** 单文件:流式写 .part -> size+sha256 校验 -> 原子 rename;失败重试 attempts 次(线性退避)。调用方负责串行。 */
|
|
40
|
+
async downloadToFile(label, urlPath, dest, expect, opts = {}) {
|
|
41
|
+
const attempts = opts.attempts ?? 3;
|
|
42
|
+
const backoffMs = opts.backoffMs ?? 250;
|
|
43
|
+
let lastErr = null;
|
|
44
|
+
for (let attempt = 1; attempt <= attempts; attempt++) {
|
|
45
|
+
try {
|
|
46
|
+
await this.tryDownload(urlPath, dest, expect);
|
|
47
|
+
return;
|
|
48
|
+
}
|
|
49
|
+
catch (e) {
|
|
50
|
+
lastErr = e;
|
|
51
|
+
if (attempt < attempts)
|
|
52
|
+
await sleep(backoffMs * attempt);
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
throw new Error(`download failed after ${attempts} attempts (${label}): ${errText(lastErr)}`);
|
|
56
|
+
}
|
|
57
|
+
/** 镜像下载:无认证绝对 URL,单次尝试(失败由调用方回落 VPS,VPS 的重试循环就是重试)。sha/size 校验与原子 rename 复用同一内核。 */
|
|
58
|
+
async downloadMirrorToFile(url, dest, expect) {
|
|
59
|
+
await this.tryDownloadUrl(url, dest, expect, {});
|
|
60
|
+
}
|
|
61
|
+
async tryDownload(urlPath, dest, expect) {
|
|
62
|
+
await this.tryDownloadUrl(`${this.serverUrl}/api/v1/data/${urlPath}`, dest, expect, this.headers());
|
|
63
|
+
}
|
|
64
|
+
async tryDownloadUrl(url, dest, expect, headers) {
|
|
65
|
+
const res = await fetch(url, {
|
|
66
|
+
headers,
|
|
67
|
+
signal: AbortSignal.timeout(downloadTimeoutMs(expect.size)),
|
|
68
|
+
});
|
|
69
|
+
if (!res.ok || !res.body)
|
|
70
|
+
throw new Error(`data HTTP ${res.status} (${url})`);
|
|
71
|
+
const part = `${dest}.part`;
|
|
72
|
+
const hash = createHash('sha256');
|
|
73
|
+
let bytes = 0;
|
|
74
|
+
const counter = new Transform({
|
|
75
|
+
transform(chunk, _enc, cb) {
|
|
76
|
+
bytes += chunk.length;
|
|
77
|
+
hash.update(chunk);
|
|
78
|
+
cb(null, chunk);
|
|
79
|
+
},
|
|
80
|
+
});
|
|
81
|
+
try {
|
|
82
|
+
await pipeline(Readable.fromWeb(res.body), counter, createWriteStream(part));
|
|
83
|
+
if (bytes !== expect.size)
|
|
84
|
+
throw new Error(`size mismatch: got ${bytes}, want ${expect.size}`);
|
|
85
|
+
const sha = hash.digest('hex');
|
|
86
|
+
if (sha !== expect.sha256)
|
|
87
|
+
throw new Error(`sha256 mismatch: got ${sha}, want ${expect.sha256}`);
|
|
88
|
+
renameSync(part, dest);
|
|
89
|
+
}
|
|
90
|
+
catch (e) {
|
|
91
|
+
rmSync(part, { force: true });
|
|
92
|
+
throw e;
|
|
93
|
+
}
|
|
94
|
+
}
|
|
95
|
+
}
|
|
96
|
+
function errText(e) {
|
|
97
|
+
return e instanceof Error ? e.message : String(e);
|
|
98
|
+
}
|
package/dist/index.d.ts
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
import type { InitOptions, SyncReport, SyncStatus } from './types.js';
|
|
2
|
+
export declare function init(opts?: InitOptions): Promise<void>;
|
|
3
|
+
/** 显式同步:await 完成,错误上抛。 */
|
|
4
|
+
export declare function sync(): Promise<SyncReport>;
|
|
5
|
+
/** 后台立即一轮:错误只进 syncStatus().lastError,进程不 crash。 */
|
|
6
|
+
export declare function syncStart(): void;
|
|
7
|
+
export declare function syncDaemonStart(intervalSecs?: number): void;
|
|
8
|
+
export declare function syncDaemonStop(): void;
|
|
9
|
+
export declare function syncStatus(): SyncStatus;
|
|
10
|
+
export declare function query(sql: string): Promise<Record<string, unknown>[]>;
|
|
11
|
+
export declare function close(): Promise<void>;
|
|
12
|
+
/** 测试专用:重置单例。勿在应用代码使用。 */
|
|
13
|
+
export declare function _resetForTest(): void;
|
|
14
|
+
/** 测试专用:手动触发 daemon tick。 */
|
|
15
|
+
export declare function _daemonTickForTest(): Promise<void>;
|
|
16
|
+
declare const tickdb: {
|
|
17
|
+
init: typeof init;
|
|
18
|
+
sync: typeof sync;
|
|
19
|
+
syncStart: typeof syncStart;
|
|
20
|
+
syncDaemonStart: typeof syncDaemonStart;
|
|
21
|
+
syncDaemonStop: typeof syncDaemonStop;
|
|
22
|
+
syncStatus: typeof syncStatus;
|
|
23
|
+
query: typeof query;
|
|
24
|
+
close: typeof close;
|
|
25
|
+
_daemonTickForTest: typeof _daemonTickForTest;
|
|
26
|
+
};
|
|
27
|
+
export default tickdb;
|
package/dist/index.js
ADDED
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
import { mkdirSync } from 'node:fs';
|
|
2
|
+
import { join } from 'node:path';
|
|
3
|
+
import { DuckdbStore } from './store.js';
|
|
4
|
+
import { TickdbHttp } from './http.js';
|
|
5
|
+
import { syncOnce } from './sync.js';
|
|
6
|
+
import { SyncStateHolder } from './status.js';
|
|
7
|
+
import { createDaemon } from './daemon.js';
|
|
8
|
+
import { loadMergedConfig, saveConfig } from './config.js';
|
|
9
|
+
import { expandHome, nowSec } from './util.js';
|
|
10
|
+
let lib = null;
|
|
11
|
+
export async function init(opts = {}) {
|
|
12
|
+
if (lib)
|
|
13
|
+
return; // 幂等(热重载场景安全)
|
|
14
|
+
const dataDir = expandHome(opts.dataDir ?? '~/.tickdb');
|
|
15
|
+
const cacheDir = join(dataDir, '.cache', 'parquet');
|
|
16
|
+
mkdirSync(cacheDir, { recursive: true });
|
|
17
|
+
const cfg = loadMergedConfig(dataDir, opts);
|
|
18
|
+
saveConfig(cfg);
|
|
19
|
+
const store = await DuckdbStore.open(join(dataDir, 'tickdb.duckdb'));
|
|
20
|
+
store.setParquetRoot(cacheDir); // v2 查询前置刷新用;upsert 模式零影响
|
|
21
|
+
const state = new SyncStateHolder();
|
|
22
|
+
const http = new TickdbHttp(cfg.serverUrl, cfg.token);
|
|
23
|
+
lib = { dataDir, cacheDir, store, http, state, daemon: null, downloadConcurrency: cfg.downloadConcurrency };
|
|
24
|
+
if ((opts.autoSync ?? true) && cfg.token) {
|
|
25
|
+
lib.daemon = createDaemon({
|
|
26
|
+
onTick: () => backgroundSync(),
|
|
27
|
+
intervalSecs: opts.syncIntervalSecs ?? cfg.syncIntervalSecs,
|
|
28
|
+
state,
|
|
29
|
+
});
|
|
30
|
+
void backgroundSync(); // init 后立即来一轮(后台,错误不抛)
|
|
31
|
+
}
|
|
32
|
+
}
|
|
33
|
+
function requireLib() {
|
|
34
|
+
if (!lib)
|
|
35
|
+
throw new Error('tickdb not initialized: call init() first');
|
|
36
|
+
return lib;
|
|
37
|
+
}
|
|
38
|
+
/** 带状态跟踪的一轮同步。 */
|
|
39
|
+
async function runRound() {
|
|
40
|
+
const l = requireLib();
|
|
41
|
+
if (l.state.get().running)
|
|
42
|
+
throw new Error('sync already running');
|
|
43
|
+
l.state.update({ running: true, startedAt: nowSec() });
|
|
44
|
+
try {
|
|
45
|
+
const report = await syncOnce(l.store, l.http, l.cacheDir, { downloadConcurrency: l.downloadConcurrency });
|
|
46
|
+
l.state.update({
|
|
47
|
+
running: false, finishedAt: nowSec(),
|
|
48
|
+
filesTotal: report.filesTotal, filesFetched: report.filesFetched,
|
|
49
|
+
filesUnchanged: report.filesUnchanged, rowsUpserted: report.rowsUpserted,
|
|
50
|
+
staleDeleted: report.staleDeleted, errorsCount: report.errors.length,
|
|
51
|
+
lastError: report.errors.length > 0 ? report.errors[0] : null,
|
|
52
|
+
});
|
|
53
|
+
return report;
|
|
54
|
+
}
|
|
55
|
+
catch (e) {
|
|
56
|
+
l.state.update({ running: false, finishedAt: nowSec(), lastError: e instanceof Error ? e.message : String(e) });
|
|
57
|
+
throw e;
|
|
58
|
+
}
|
|
59
|
+
}
|
|
60
|
+
/** 显式同步:await 完成,错误上抛。 */
|
|
61
|
+
export async function sync() {
|
|
62
|
+
return runRound();
|
|
63
|
+
}
|
|
64
|
+
/** 后台立即一轮:错误只进 syncStatus().lastError,进程不 crash。 */
|
|
65
|
+
export function syncStart() {
|
|
66
|
+
void backgroundSync();
|
|
67
|
+
}
|
|
68
|
+
async function backgroundSync() {
|
|
69
|
+
try {
|
|
70
|
+
await runRound();
|
|
71
|
+
}
|
|
72
|
+
catch (e) {
|
|
73
|
+
// close() 后 lib 可能为 null:只记不再抛,进程永不 crash
|
|
74
|
+
lib?.state.setError(e instanceof Error ? e.message : String(e));
|
|
75
|
+
}
|
|
76
|
+
}
|
|
77
|
+
export function syncDaemonStart(intervalSecs = 300) {
|
|
78
|
+
const l = requireLib();
|
|
79
|
+
l.daemon?.stop();
|
|
80
|
+
l.daemon = createDaemon({ onTick: () => backgroundSync(), intervalSecs, state: l.state });
|
|
81
|
+
}
|
|
82
|
+
export function syncDaemonStop() {
|
|
83
|
+
const l = requireLib();
|
|
84
|
+
l.daemon?.stop();
|
|
85
|
+
l.daemon = null;
|
|
86
|
+
}
|
|
87
|
+
export function syncStatus() {
|
|
88
|
+
return requireLib().state.get();
|
|
89
|
+
}
|
|
90
|
+
export async function query(sql) {
|
|
91
|
+
return requireLib().store.query(sql);
|
|
92
|
+
}
|
|
93
|
+
export async function close() {
|
|
94
|
+
if (!lib)
|
|
95
|
+
return;
|
|
96
|
+
lib.daemon?.stop();
|
|
97
|
+
await lib.store.close();
|
|
98
|
+
lib = null;
|
|
99
|
+
}
|
|
100
|
+
/** 测试专用:重置单例。勿在应用代码使用。 */
|
|
101
|
+
export function _resetForTest() {
|
|
102
|
+
lib = null;
|
|
103
|
+
}
|
|
104
|
+
/** 测试专用:手动触发 daemon tick。 */
|
|
105
|
+
export async function _daemonTickForTest() {
|
|
106
|
+
await requireLib().daemon?.tickNow();
|
|
107
|
+
}
|
|
108
|
+
const tickdb = { init, sync, syncStart, syncDaemonStart, syncDaemonStop, syncStatus, query, close, _daemonTickForTest };
|
|
109
|
+
export default tickdb;
|
package/dist/manifest.js
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
const SUPPORTED_VERSIONS = ['v1', 'v1.1', 'v1.2', 'v1.3', 'v1.4'];
|
|
2
|
+
/** 把线上 JSON(snake_case)解析成 Manifest,字段不合法直接抛错。 */
|
|
3
|
+
export function parseManifest(raw) {
|
|
4
|
+
if (typeof raw !== 'object' || raw === null)
|
|
5
|
+
throw new Error('manifest: not an object');
|
|
6
|
+
const r = raw;
|
|
7
|
+
const version = SUPPORTED_VERSIONS.find((v) => v === r.version);
|
|
8
|
+
if (!version) {
|
|
9
|
+
throw new Error(`unsupported manifest version: ${String(r.version)}`);
|
|
10
|
+
}
|
|
11
|
+
if (!Array.isArray(r.files))
|
|
12
|
+
throw new Error('manifest: files is not an array');
|
|
13
|
+
const files = r.files.map((f, i) => {
|
|
14
|
+
if (typeof f !== 'object' || f === null)
|
|
15
|
+
throw new Error(`manifest files[${i}]: not an object`);
|
|
16
|
+
const e = f;
|
|
17
|
+
for (const k of ['path', 'sha256', 'symbol', 'cycle', 'ts_min', 'ts_max']) {
|
|
18
|
+
if (typeof e[k] !== 'string')
|
|
19
|
+
throw new Error(`manifest files[${i}].${k}: expected string`);
|
|
20
|
+
}
|
|
21
|
+
for (const k of ['size', 'row_count']) {
|
|
22
|
+
if (typeof e[k] !== 'number')
|
|
23
|
+
throw new Error(`manifest files[${i}].${k}: expected number`);
|
|
24
|
+
}
|
|
25
|
+
return {
|
|
26
|
+
path: e.path, sha256: e.sha256, size: e.size,
|
|
27
|
+
row_count: e.row_count, symbol: e.symbol, cycle: e.cycle,
|
|
28
|
+
ts_min: e.ts_min, ts_max: e.ts_max,
|
|
29
|
+
year: typeof e.year === 'number' ? e.year : undefined,
|
|
30
|
+
symbols_count: typeof e.symbols_count === 'number' ? e.symbols_count : undefined,
|
|
31
|
+
frozen: e.frozen === true ? true : undefined,
|
|
32
|
+
};
|
|
33
|
+
});
|
|
34
|
+
return {
|
|
35
|
+
version,
|
|
36
|
+
schema: typeof r.schema === 'string' ? r.schema : 'symbol-partitioned',
|
|
37
|
+
generatedAt: typeof r.generated_at === 'string' ? r.generated_at : '',
|
|
38
|
+
files,
|
|
39
|
+
mirror_base_url: typeof r.mirror_base_url === 'string' ? r.mirror_base_url : undefined,
|
|
40
|
+
disjoint: r.disjoint === true ? true : undefined,
|
|
41
|
+
};
|
|
42
|
+
}
|
package/dist/status.d.ts
ADDED
package/dist/status.js
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
function fresh() {
|
|
2
|
+
return {
|
|
3
|
+
running: false, startedAt: 0, finishedAt: 0,
|
|
4
|
+
filesTotal: 0, filesFetched: 0, filesUnchanged: 0,
|
|
5
|
+
rowsUpserted: 0, staleDeleted: 0, errorsCount: 0,
|
|
6
|
+
daemonIntervalSecs: 0, daemonRunning: false,
|
|
7
|
+
lastError: null,
|
|
8
|
+
};
|
|
9
|
+
}
|
|
10
|
+
export class SyncStateHolder {
|
|
11
|
+
s = fresh();
|
|
12
|
+
get() { return { ...this.s }; }
|
|
13
|
+
update(patch) { this.s = { ...this.s, ...patch }; }
|
|
14
|
+
/** 覆盖 lastError;传 null 清除。 */
|
|
15
|
+
setError(msg) { this.s = { ...this.s, lastError: msg }; }
|
|
16
|
+
}
|