dsh-cost-meter 1.9.56 → 1.9.58
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +8 -8
- package/README.zh-CN.md +8 -8
- package/docs/update-notes-1.9.57.md +17 -0
- package/docs/update-notes-1.9.58.md +17 -0
- package/lib/backfill.js +48 -14
- package/lib/billion-context-history.js +18 -15
- package/lib/billion-context-inventory.js +18 -8
- package/lib/billion-context-repair.js +11 -7
- package/lib/billion-context-storage.js +11 -5
- package/lib/client.js +1 -1
- package/lib/index.js +36 -8
- package/lib/journal-storage.js +10 -4
- package/lib/native-search-history.js +5 -4
- package/lib/session-storage-compat.js +27 -4
- package/package.json +3 -1
package/README.md
CHANGED
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
**Session cost tracking plugin for the DeepSeek Harness web GUI (bilingual UI)**
|
|
8
8
|
|
|
9
|
-
[](https://github.com/Han-1413141/dsh-cost-meter)
|
|
10
10
|
[](https://www.npmjs.com/package/dsh-cost-meter)
|
|
11
11
|
[](https://dshfind.com/zh/plugins/Han-1413141/dsh-cost-meter?ref=badge)
|
|
12
12
|
[](LICENSE)
|
|
@@ -16,11 +16,11 @@
|
|
|
16
16
|
|
|
17
17
|
</div>
|
|
18
18
|
|
|
19
|
-
**v1.9.
|
|
19
|
+
**v1.9.58** fixes the session-cost entry disappearing when conversation usage arrives before plugin settings. The entry stays visible while loading and shows the amount once settings are ready.
|
|
20
20
|
|
|
21
21
|
**Host floor: DSH 0.2.0** · Adaptation target: **0.2.2-alpha.3**. Expand the compatibility notes below for exact declarations, host records and troubleshooting.
|
|
22
22
|
|
|
23
|
-
Release notes: [v1.9.
|
|
23
|
+
Release notes: [v1.9.58](docs/update-notes-1.9.58.md#english). Fully restart DSH after updating.
|
|
24
24
|
|
|
25
25
|

|
|
26
26
|
|
|
@@ -352,22 +352,22 @@ On Node.js 20, use `npm install -g pnpm@10` instead. See [pnpm installation and
|
|
|
352
352
|
dsh plugin --profile web add dsh-cost-meter
|
|
353
353
|
```
|
|
354
354
|
|
|
355
|
-
**PowerShell one-click script** (copy the whole line, paste, press Enter; pnpm is provisioned automatically, git is auto-detected — no clone needed; the install chain is **pinned to the release tag `v1.9.
|
|
355
|
+
**PowerShell one-click script** (copy the whole line, paste, press Enter; pnpm is provisioned automatically, git is auto-detected — no clone needed; the install chain is **pinned to the release tag `v1.9.58`** — review the script before running):
|
|
356
356
|
|
|
357
357
|
```powershell
|
|
358
|
-
irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.9.
|
|
358
|
+
irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.9.58/install.ps1 | iex
|
|
359
359
|
```
|
|
360
360
|
|
|
361
361
|
**Or a plain command line** (the machine must already have pnpm and git; also pinned to the tag):
|
|
362
362
|
|
|
363
363
|
```sh
|
|
364
|
-
dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.9.
|
|
364
|
+
dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.9.58
|
|
365
365
|
```
|
|
366
366
|
|
|
367
367
|
Without git, use the GitHub tag archive:
|
|
368
368
|
|
|
369
369
|
```sh
|
|
370
|
-
dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.9.
|
|
370
|
+
dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.9.58.tar.gz
|
|
371
371
|
```
|
|
372
372
|
|
|
373
373
|
After installing, **restart** `dsh web` (plugin rows, the Typert manifest and the client bundle are all scanned at startup):
|
|
@@ -432,7 +432,7 @@ Workarounds:
|
|
|
432
432
|
|
|
433
433
|
```yaml
|
|
434
434
|
minimumReleaseAgeExclude:
|
|
435
|
-
- dsh-cost-meter@1.9.
|
|
435
|
+
- dsh-cost-meter@1.9.58
|
|
436
436
|
```
|
|
437
437
|
|
|
438
438
|
3. For a one-off CLI operation, forward the pnpm flag: `dsh plugin --profile desktop remove dsh-cost-meter --config.minimum-release-age=0` (or add `--trust-lockfile` when only lockfile verification fails). Both flags were verified against pnpm 11.28.5 in a temporary profile.
|
package/README.zh-CN.md
CHANGED
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
**DeepSeek Harness 会话费用统计插件(界面中英双语)**
|
|
8
8
|
|
|
9
|
-
[](https://github.com/Han-1413141/dsh-cost-meter)
|
|
10
10
|
[](https://www.npmjs.com/package/dsh-cost-meter)
|
|
11
11
|
[](https://dshfind.com/zh/plugins/Han-1413141/dsh-cost-meter?ref=badge)
|
|
12
12
|
[](LICENSE)
|
|
@@ -16,11 +16,11 @@
|
|
|
16
16
|
|
|
17
17
|
</div>
|
|
18
18
|
|
|
19
|
-
**v1.9.
|
|
19
|
+
**v1.9.58** 修复输入框下方会话费用偶尔消失:会话用量先于插件设置到达时保留加载提示,设置加载后正常显示金额。
|
|
20
20
|
|
|
21
21
|
**宿主要求:DSH 0.2.0 起支持** · 适配目标 **0.2.2-alpha.3**。完整声明、历史记录与安装排障见下方展开项。
|
|
22
22
|
|
|
23
|
-
更新说明:[v1.9.
|
|
23
|
+
更新说明:[v1.9.58](docs/update-notes-1.9.58.md#中文)。更新后完全重启 DSH。
|
|
24
24
|
|
|
25
25
|

|
|
26
26
|
|
|
@@ -352,22 +352,22 @@ Node.js 20 请改用 `npm install -g pnpm@10`。版本要求见 [pnpm 官方安
|
|
|
352
352
|
dsh plugin --profile web add dsh-cost-meter
|
|
353
353
|
```
|
|
354
354
|
|
|
355
|
-
**PowerShell 一键脚本**(复制整行粘贴回车;自动补齐 pnpm、自动探测 git,无需克隆仓库;安装链**固定到发布 tag `v1.9.
|
|
355
|
+
**PowerShell 一键脚本**(复制整行粘贴回车;自动补齐 pnpm、自动探测 git,无需克隆仓库;安装链**固定到发布 tag `v1.9.58`**,建议先下载审阅再运行):
|
|
356
356
|
|
|
357
357
|
```powershell
|
|
358
|
-
irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.9.
|
|
358
|
+
irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.9.58/install.ps1 | iex
|
|
359
359
|
```
|
|
360
360
|
|
|
361
361
|
**或直接命令行**(机器上需已有 pnpm 与 git;同样固定到 tag):
|
|
362
362
|
|
|
363
363
|
```sh
|
|
364
|
-
dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.9.
|
|
364
|
+
dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.9.58
|
|
365
365
|
```
|
|
366
366
|
|
|
367
367
|
没有 git 时可用 GitHub tag 打包直链:
|
|
368
368
|
|
|
369
369
|
```sh
|
|
370
|
-
dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.9.
|
|
370
|
+
dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.9.58.tar.gz
|
|
371
371
|
```
|
|
372
372
|
|
|
373
373
|
安装后**重启** `dsh web`(插件行、Typert 清单与客户端 bundle 均在启动时扫描):
|
|
@@ -432,7 +432,7 @@ DSH **0.2.1-rc.1 / 0.2.1-rc.2** 宿主自带 pnpm **11.28.5**,默认启用内
|
|
|
432
432
|
|
|
433
433
|
```yaml
|
|
434
434
|
minimumReleaseAgeExclude:
|
|
435
|
-
- dsh-cost-meter@1.9.
|
|
435
|
+
- dsh-cost-meter@1.9.58
|
|
436
436
|
```
|
|
437
437
|
|
|
438
438
|
3. 仅本次操作用 CLI 透传 pnpm 参数:`dsh plugin --profile desktop remove dsh-cost-meter --config.minimum-release-age=0`;若只是 lockfile 校验阶段失败,可改用 `--trust-lockfile`。这两个参数已在本机 pnpm 11.28.5 临时 Profile 上实测可用。
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
# v1.9.57 — 桌面更新退出超时修复
|
|
2
|
+
|
|
3
|
+
## 中文
|
|
4
|
+
|
|
5
|
+
- 修复历史回填已经开始后,Desktop Host 无法在退出期限内结束、导致客户端更新失败的问题。此前只清理启动计时器,已经开始的读取仍继续运行;现在退出时取消扫描,等待文件关闭,再保存账本。
|
|
6
|
+
- 取消信号覆盖 JSONL、zstd 压缩会话、SQLite 会话分页,以及搜索和 Billion Context 历史。取消后不继续读取其他会话,也不把取消当成文件损坏后继续尝试其他后端。
|
|
7
|
+
- 被中断的导入和 fork 修复保留重试状态,下次启动继续处理;正常读取和计费方式保持不变。
|
|
8
|
+
|
|
9
|
+
将费用插件更新到 **1.9.57** 后,完全退出并重新启动 DSH,再执行客户端更新。若曾安装本地临时补丁,正式版本已包含该修复,无需重复修改文件。
|
|
10
|
+
|
|
11
|
+
## English
|
|
12
|
+
|
|
13
|
+
- Fix Desktop update failures caused by startup history reads keeping the Host alive beyond its shutdown deadline. Shutdown now cancels active scans, waits for their readers to close, and then saves the ledger.
|
|
14
|
+
- Propagate cancellation through JSONL, zstd, SQLite pagination, search history and Billion Context history. Cancelled reads stop instead of falling back to another backend or continuing with the next session.
|
|
15
|
+
- Keep interrupted imports and fork repairs eligible for retry on the next startup, without changing normal billing behavior.
|
|
16
|
+
|
|
17
|
+
Update the cost plugin to **1.9.57**, fully quit and restart DSH, then retry the Desktop update. This release includes the local temporary fix; manual file patches are no longer needed.
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
# v1.9.58 — 会话费用入口消失修复
|
|
2
|
+
|
|
3
|
+
## 中文
|
|
4
|
+
|
|
5
|
+
- 修复恢复已有会话或重新加载插件时,输入框下方会话金额偶尔消失的问题。会话用量或账本可能先于插件设置到达,旧版在设置尚未就绪时计算费用,触发渲染异常,使宿主停止显示费用入口。
|
|
6
|
+
- 现在设置未就绪时保留加载或错误提示,设置加载后正常显示已记录的金额。标题栏的费用入口也包含这一修复。
|
|
7
|
+
- 保留手动隐藏、显示位置、API/Plan 金额拆分和币种设置;切换会话时不会显示上一会话金额。
|
|
8
|
+
|
|
9
|
+
将费用插件更新到 **1.9.58** 后,完全退出并重新启动 DSH,即可重新加载费用入口,无需修改显示设置。
|
|
10
|
+
|
|
11
|
+
## English
|
|
12
|
+
|
|
13
|
+
- Fix the session-cost entry occasionally disappearing when restoring a conversation or reloading the plugin. Usage or ledger data could arrive before configuration, causing a render exception that made the host stop displaying the entry.
|
|
14
|
+
- Keep the loading or error message visible until configuration is available, then display the recorded amount. The header cost entry receives the same fix.
|
|
15
|
+
- Preserve explicit visibility preferences, display position, API/Plan separation and currency settings. Switching conversations does not display the previous conversation's amount.
|
|
16
|
+
|
|
17
|
+
Update the plugin to **1.9.58**, fully quit and restart DSH. No display-setting changes are required.
|
package/lib/backfill.js
CHANGED
|
@@ -156,10 +156,11 @@ export function scanZstdFrames(buffer) {
|
|
|
156
156
|
* 单文件累计解压字节数超过 maxDecompressed(默认 4GB,防解压炸弹)抛
|
|
157
157
|
* RangeError,调用方按单文件损坏处理。
|
|
158
158
|
* @param path - session.jsonl.zstd 或 session.jsonl 路径。
|
|
159
|
-
* @param options - { maxDecompressed?: number, yieldEvery?: number }。
|
|
159
|
+
* @param options - { maxDecompressed?: number, yieldEvery?: number, signal?: AbortSignal }。
|
|
160
160
|
* @yields {object} 逐行解析出的记录(空行/打包行/坏行已过滤)。
|
|
161
161
|
*/
|
|
162
162
|
export async function* iterateSessionRecords(path, options = {}) {
|
|
163
|
+
options.signal?.throwIfAborted()
|
|
163
164
|
const maxDecompressed = Number(options.maxDecompressed) > 0 ? Number(options.maxDecompressed) : MAX_DECOMPRESSED_PER_FILE
|
|
164
165
|
const yieldEvery = Number(options.yieldEvery) > 0 ? Number(options.yieldEvery) : YIELD_EVERY_FRAMES
|
|
165
166
|
const maxFrameBytes = Number(options.maxFrameBytes) > 0 ? Number(options.maxFrameBytes) : MAX_FRAME_BYTES
|
|
@@ -209,7 +210,9 @@ export async function* iterateSessionRecords(path, options = {}) {
|
|
|
209
210
|
// Only invalid frame structure stops the scan; an incomplete frame keeps reading.
|
|
210
211
|
let stopped = false
|
|
211
212
|
for (;;) {
|
|
213
|
+
options.signal?.throwIfAborted()
|
|
212
214
|
const chunk = await handle.read(Buffer.allocUnsafe(READ_CHUNK_BYTES), 0, READ_CHUNK_BYTES, null)
|
|
215
|
+
options.signal?.throwIfAborted()
|
|
213
216
|
if (chunk.bytesRead === 0) break
|
|
214
217
|
const data = tail.length > 0 ? Buffer.concat([tail, chunk.buffer.subarray(0, chunk.bytesRead)]) : chunk.buffer.subarray(0, chunk.bytesRead)
|
|
215
218
|
tail = Buffer.alloc(0)
|
|
@@ -217,7 +220,7 @@ export async function* iterateSessionRecords(path, options = {}) {
|
|
|
217
220
|
if (!isZstd) {
|
|
218
221
|
consumeBytes(data, records)
|
|
219
222
|
if (records.length >= YIELD_EVERY_EVENTS) {
|
|
220
|
-
yield* drain(records)
|
|
223
|
+
yield* drain(records, options.signal)
|
|
221
224
|
await new Promise(resolve => setImmediate(resolve))
|
|
222
225
|
}
|
|
223
226
|
continue
|
|
@@ -226,6 +229,7 @@ export async function* iterateSessionRecords(path, options = {}) {
|
|
|
226
229
|
// 解压→逐行→释放引用;剩余不完整帧(块边界截断)留在 tail 等下一块。
|
|
227
230
|
let offset = 0
|
|
228
231
|
while (offset < data.length) {
|
|
232
|
+
options.signal?.throwIfAborted()
|
|
229
233
|
const frame = scanZstdFrame(data, offset)
|
|
230
234
|
if (frame.status !== 'complete') {
|
|
231
235
|
if (frame.status === 'invalid') stopped = true
|
|
@@ -251,12 +255,12 @@ export async function* iterateSessionRecords(path, options = {}) {
|
|
|
251
255
|
framesSinceYield += 1
|
|
252
256
|
offset = frame.end
|
|
253
257
|
if (framesSinceYield >= yieldEvery || records.length >= YIELD_EVERY_EVENTS) {
|
|
254
|
-
yield* drain(records); framesSinceYield = 0
|
|
258
|
+
yield* drain(records, options.signal); framesSinceYield = 0
|
|
255
259
|
await new Promise(resolve => setImmediate(resolve))
|
|
256
260
|
}
|
|
257
261
|
}
|
|
258
262
|
if (framesSinceYield >= yieldEvery || records.length >= YIELD_EVERY_EVENTS) {
|
|
259
|
-
yield* drain(records)
|
|
263
|
+
yield* drain(records, options.signal)
|
|
260
264
|
framesSinceYield = 0
|
|
261
265
|
await new Promise(resolve => setImmediate(resolve))
|
|
262
266
|
}
|
|
@@ -270,15 +274,18 @@ export async function* iterateSessionRecords(path, options = {}) {
|
|
|
270
274
|
records.push(rec)
|
|
271
275
|
}
|
|
272
276
|
}
|
|
273
|
-
yield* drain(records)
|
|
277
|
+
yield* drain(records, options.signal)
|
|
274
278
|
} finally {
|
|
275
279
|
await handle.close()
|
|
276
280
|
}
|
|
277
281
|
}
|
|
278
282
|
|
|
279
283
|
/** 把 records 数组逐条 yield 出去并清空(内部助手)。 */
|
|
280
|
-
function* drain(records) {
|
|
281
|
-
for (const rec of records)
|
|
284
|
+
function* drain(records, signal) {
|
|
285
|
+
for (const rec of records) {
|
|
286
|
+
signal?.throwIfAborted()
|
|
287
|
+
yield rec
|
|
288
|
+
}
|
|
282
289
|
records.length = 0
|
|
283
290
|
}
|
|
284
291
|
|
|
@@ -313,12 +320,20 @@ export function listSessionLogs(root, onlySessionIds = null) {
|
|
|
313
320
|
* `listSessions(onlySessionIds)` 枚举 + `readRecords(id)` 读取(JSONL 文件或 SQLite
|
|
314
321
|
* 由适配器自动选择);为空时保持原有 `listSessionLogs` 文件枚举路径不变。
|
|
315
322
|
*/
|
|
323
|
+
function checkBackfillActive(ledger, sessionSource) {
|
|
324
|
+
sessionSource?.signal?.throwIfAborted()
|
|
325
|
+
if (ledger.closed) throw new DOMException('Ledger closed', 'AbortError')
|
|
326
|
+
}
|
|
327
|
+
|
|
316
328
|
async function* sessionBillingRecords(ledger, sessionsRoot, onlySessionIds = null, sessionSource = null) {
|
|
329
|
+
checkBackfillActive(ledger, sessionSource)
|
|
330
|
+
const signal = sessionSource?.signal
|
|
317
331
|
const seen = new Set()
|
|
318
332
|
const hasSource = sessionSource !== null && sessionSource !== undefined
|
|
319
333
|
const sessions = hasSource ? await sessionSource.listSessions(onlySessionIds) : null
|
|
320
334
|
const paths = hasSource ? null : listSessionLogs(sessionsRoot, onlySessionIds)
|
|
321
335
|
for (const entry of sessions ?? paths) {
|
|
336
|
+
checkBackfillActive(ledger, sessionSource)
|
|
322
337
|
let records
|
|
323
338
|
let id
|
|
324
339
|
try {
|
|
@@ -330,32 +345,46 @@ async function* sessionBillingRecords(ledger, sessionsRoot, onlySessionIds = nul
|
|
|
330
345
|
records = await readSessionRecordsAsync(entry)
|
|
331
346
|
id = records.find(record => record.type === 'session')?.id
|
|
332
347
|
}
|
|
333
|
-
} catch {
|
|
348
|
+
} catch (error) {
|
|
349
|
+
checkBackfillActive(ledger, sessionSource)
|
|
350
|
+
yield null
|
|
351
|
+
continue
|
|
352
|
+
}
|
|
353
|
+
checkBackfillActive(ledger, sessionSource)
|
|
334
354
|
if (!Array.isArray(records)) { yield null; continue }
|
|
335
355
|
if (typeof id === 'string') {
|
|
336
356
|
seen.add(id)
|
|
337
|
-
try { for (const record of await nativeSearchRecordsFor(ledger.path, id)) records.push(record) } catch {
|
|
338
|
-
try { records.push(...await billionRecordsFor(ledger.path, id)) } catch {
|
|
357
|
+
try { for (const record of await nativeSearchRecordsFor(ledger.path, id, signal)) records.push(record) } catch { checkBackfillActive(ledger, sessionSource) }
|
|
358
|
+
try { records.push(...await billionRecordsFor(ledger.path, id, signal)) } catch { checkBackfillActive(ledger, sessionSource) }
|
|
339
359
|
}
|
|
360
|
+
checkBackfillActive(ledger, sessionSource)
|
|
340
361
|
yield records
|
|
341
362
|
}
|
|
342
363
|
// A deleted/missing host log must not discard the plugin's surviving search history.
|
|
343
364
|
for (const path of listNativeSearchHistory(ledger.path, seen, onlySessionIds)) {
|
|
365
|
+
checkBackfillActive(ledger, sessionSource)
|
|
344
366
|
let history
|
|
345
|
-
try { history = await readNativeSearchHistory(path) } catch { continue }
|
|
367
|
+
try { history = await readNativeSearchHistory(path, undefined, signal) } catch { checkBackfillActive(ledger, sessionSource); continue }
|
|
368
|
+
checkBackfillActive(ledger, sessionSource)
|
|
346
369
|
const id = history.sessionId
|
|
347
370
|
if (!id || seen.has(id) || (onlySessionIds !== null && !onlySessionIds.has(id))) continue
|
|
348
371
|
seen.add(id)
|
|
349
|
-
|
|
372
|
+
const extra = await billionRecordsFor(ledger.path, id, signal)
|
|
373
|
+
checkBackfillActive(ledger, sessionSource)
|
|
374
|
+
yield [{ type: 'session', id, createdAt: 0 }, ...history.records, ...extra]
|
|
350
375
|
}
|
|
351
|
-
for (const file of listBillionFiles(ledger.path)) for await (const { row } of iterateBillionCalls(file)) {
|
|
376
|
+
for (const file of listBillionFiles(ledger.path)) for await (const { row } of iterateBillionCalls(file, 0, signal)) {
|
|
377
|
+
checkBackfillActive(ledger, sessionSource)
|
|
352
378
|
const id = row.sessionId
|
|
353
379
|
if (!seen.has(id) && (onlySessionIds === null || onlySessionIds.has(id))) {
|
|
354
380
|
seen.add(id)
|
|
355
|
-
|
|
381
|
+
const extra = await billionRecordsFor(ledger.path, id, signal)
|
|
382
|
+
checkBackfillActive(ledger, sessionSource)
|
|
383
|
+
yield [{ type: 'session', id, createdAt: 0 }, ...extra]
|
|
356
384
|
}
|
|
357
385
|
break
|
|
358
386
|
}
|
|
387
|
+
checkBackfillActive(ledger, sessionSource)
|
|
359
388
|
}
|
|
360
389
|
|
|
361
390
|
/**
|
|
@@ -1210,6 +1239,7 @@ function subtractBucketInto(target, source) {
|
|
|
1210
1239
|
* @returns { sessions, days, scanned } 受影响的 fork 会话数、扣除的日期数与扫描文件数。
|
|
1211
1240
|
*/
|
|
1212
1241
|
export async function repairForkSeed(ledger, sessionsRoot, sessionSource = null) {
|
|
1242
|
+
checkBackfillActive(ledger, sessionSource)
|
|
1213
1243
|
const result = { sessions: 0, days: 0, scanned: 0 }
|
|
1214
1244
|
const hasSource = sessionSource !== null && sessionSource !== undefined
|
|
1215
1245
|
// 统一记录源(T3):RDB 下从数据库枚举会话;未传源时保持原 JSONL 文件枚举路径不变。
|
|
@@ -1220,12 +1250,16 @@ export async function repairForkSeed(ledger, sessionsRoot, sessionSource = null)
|
|
|
1220
1250
|
// 全量同步扫描会长时间占住事件循环:每 8 份让出一次(与回填/导入同策略);
|
|
1221
1251
|
// 单文件内部亦为流式读取(v1.7.7,issue #88)。
|
|
1222
1252
|
if ((result.scanned += 1) % 8 === 0) await new Promise(resolve => setImmediate(resolve))
|
|
1253
|
+
checkBackfillActive(ledger, sessionSource)
|
|
1223
1254
|
let records
|
|
1224
1255
|
try {
|
|
1225
1256
|
records = hasSource ? await sessionSource.readRecords(entry) : await readSessionRecordsAsync(entry)
|
|
1226
1257
|
} catch {
|
|
1258
|
+
checkBackfillActive(ledger, sessionSource)
|
|
1227
1259
|
continue // 单文件损坏(含解压预算超限)不阻断整体清洗。
|
|
1228
1260
|
}
|
|
1261
|
+
checkBackfillActive(ledger, sessionSource)
|
|
1262
|
+
if (!Array.isArray(records)) continue
|
|
1229
1263
|
// 只处理 fork 会话(header 带 parentSession);普通会话日志无种子段。
|
|
1230
1264
|
const header = records.find(r => r?.type === 'session')
|
|
1231
1265
|
if (header === undefined || header === null || typeof header.parentSession !== 'string' || header.parentSession.length === 0) continue
|
|
@@ -59,9 +59,9 @@ export function readBillionStreamFile(file) {
|
|
|
59
59
|
&& billionSessionKey(stream.sessionId) === file.key && file.path.endsWith(stream.streamId + '.json') ? stream : null
|
|
60
60
|
} catch { return null }
|
|
61
61
|
}
|
|
62
|
-
export async function readBillionStreamFileAsync(file) {
|
|
62
|
+
export async function readBillionStreamFileAsync(file, signal) {
|
|
63
63
|
try {
|
|
64
|
-
const stream = await readSmallBillionJson(file.path)
|
|
64
|
+
const stream = await readSmallBillionJson(file.path, signal)
|
|
65
65
|
return stream?.protocol === 1 && uuid(stream.streamId) && validSessionId(stream.sessionId)
|
|
66
66
|
&& billionSessionKey(stream.sessionId) === file.key && file.path.endsWith(stream.streamId + '.json') ? stream : null
|
|
67
67
|
} catch (error) {
|
|
@@ -94,11 +94,12 @@ export function billionSourcePaths(ledgerPath, sessionId) {
|
|
|
94
94
|
...directories(streamRoot).filter(row => row.isFile() && row.name.endsWith('.json') && uuid(row.name.slice(0, -5))).map(row => join(streamRoot, row.name))]
|
|
95
95
|
}
|
|
96
96
|
/** Bounded UTF-8 rows; offsets only advance through complete newline-terminated writes. */
|
|
97
|
-
export async function* iterateBillionCalls(file, start = 0) {
|
|
97
|
+
export async function* iterateBillionCalls(file, start = 0, signal) {
|
|
98
98
|
let pending = Buffer.alloc(0), dropping = false, offset = start
|
|
99
|
-
for await (const chunk of journalChunks(file.path, start)) {
|
|
99
|
+
for await (const chunk of journalChunks(file.path, start, undefined, signal)) {
|
|
100
100
|
const bytes = Buffer.concat([pending, chunk]); let from = 0, end
|
|
101
101
|
while ((end = bytes.indexOf(10, from)) >= 0) {
|
|
102
|
+
signal?.throwIfAborted()
|
|
102
103
|
const line = bytes.subarray(from, end); offset += end - from + 1; from = end + 1
|
|
103
104
|
if (dropping) { dropping = false; continue }
|
|
104
105
|
let row
|
|
@@ -123,11 +124,11 @@ export function readBillionStream(ledgerPath, row) {
|
|
|
123
124
|
return stream.protocol === 1 && stream.streamId === row.streamId && stream.sessionId === row.sessionId ? stream : null
|
|
124
125
|
} catch { return null }
|
|
125
126
|
}
|
|
126
|
-
export async function readBillionStreamAsync(ledgerPath, row, archives = new Map()) {
|
|
127
|
+
export async function readBillionStreamAsync(ledgerPath, row, archives = new Map(), signal) {
|
|
127
128
|
const file = { path: billionStreamPath(billionRoot(ledgerPath), row.sessionId, row.streamId), key: billionSessionKey(row.sessionId) }
|
|
128
|
-
const stream = await readBillionStreamFileAsync(file)
|
|
129
|
+
const stream = await readBillionStreamFileAsync(file, signal)
|
|
129
130
|
if (stream) return stream
|
|
130
|
-
if (!archives.has(row.sessionId)) archives.set(row.sessionId, await archivedBillionStreams(ledgerPath, row.sessionId))
|
|
131
|
+
if (!archives.has(row.sessionId)) archives.set(row.sessionId, await archivedBillionStreams(ledgerPath, row.sessionId, signal))
|
|
131
132
|
return archives.get(row.sessionId).get(row.streamId) ?? null
|
|
132
133
|
}
|
|
133
134
|
const coverageCounts = () => ({ known: 0, unknown: 0, matched: false })
|
|
@@ -159,10 +160,11 @@ function coverageResult(stream, { known, unknown, matched }) {
|
|
|
159
160
|
return checkedBillionCoverage(stream, { version: 1, mode, known, unknown })
|
|
160
161
|
}
|
|
161
162
|
/** A persistence receipt says nothing about whether model usage was observed. */
|
|
162
|
-
export async function billionCoverage(ledgerPath, stream, offsets) {
|
|
163
|
+
export async function billionCoverage(ledgerPath, stream, offsets, signal) {
|
|
164
|
+
signal?.throwIfAborted()
|
|
163
165
|
if (stream.coverage?.version === 1) return checkedBillionCoverage(stream, stream.coverage)
|
|
164
166
|
const counts = coverageCounts()
|
|
165
|
-
for (const file of listBillionFiles(ledgerPath, stream.sessionId)) for await (const { row } of iterateBillionCalls(file, offsets?.get(file.path)?.offset ?? 0)) {
|
|
167
|
+
for (const file of listBillionFiles(ledgerPath, stream.sessionId)) for await (const { row } of iterateBillionCalls(file, offsets?.get(file.path)?.offset ?? 0, signal)) {
|
|
166
168
|
if (row.streamId === stream.streamId) observeCoverage(counts, stream, row)
|
|
167
169
|
}
|
|
168
170
|
return coverageResult(stream, counts)
|
|
@@ -188,21 +190,22 @@ export function billionHostRecord(stream, coverage = stream.coverage ?? { mode:
|
|
|
188
190
|
startedAtMs: stream.startedAtMs, requestId: stream.streamId, kind: 'model', turn: stream.turn ?? null, step: stream.step ?? null,
|
|
189
191
|
metering: { source: 'billion-context', requestId: stream.streamId, attempt: 1, complete: false, status: 'complete', upstream: 'host-usage' } } }
|
|
190
192
|
}
|
|
191
|
-
export async function billionRecordsFor(ledgerPath, sessionId) {
|
|
193
|
+
export async function billionRecordsFor(ledgerPath, sessionId, signal) {
|
|
194
|
+
signal?.throwIfAborted()
|
|
192
195
|
if (!ledgerPath) return []
|
|
193
196
|
const records = [], streams = new Map(), counts = new Map(), seen = new Set(), archives = new Map()
|
|
194
197
|
const key = billionSessionKey(sessionId), dir = join(billionRoot(ledgerPath), 'streams', key)
|
|
195
|
-
for (const entry of await safeEntries(dir)) {
|
|
198
|
+
for (const entry of await safeEntries(dir, signal)) {
|
|
196
199
|
if (!entry.isFile() || !entry.name.endsWith('.json') || !uuid(entry.name.slice(0, -5))) continue
|
|
197
|
-
const stream = await readBillionStreamFileAsync({ path: join(dir, entry.name), key })
|
|
200
|
+
const stream = await readBillionStreamFileAsync({ path: join(dir, entry.name), key }, signal)
|
|
198
201
|
if (stream?.finished) streams.set(stream.streamId, stream)
|
|
199
202
|
}
|
|
200
203
|
// Originals are removed only after archive publication. Read archives after
|
|
201
204
|
// originals so a concurrent compaction cannot create a gap in history.
|
|
202
|
-
for (const [id, stream] of await archivedBillionStreams(ledgerPath, sessionId)) if (!streams.has(id)) streams.set(id, stream)
|
|
203
|
-
for (const file of listBillionFiles(ledgerPath, sessionId)) for await (const { row } of iterateBillionCalls(file)) {
|
|
205
|
+
for (const [id, stream] of await archivedBillionStreams(ledgerPath, sessionId, signal)) if (!streams.has(id)) streams.set(id, stream)
|
|
206
|
+
for (const file of listBillionFiles(ledgerPath, sessionId)) for await (const { row } of iterateBillionCalls(file, 0, signal)) {
|
|
204
207
|
if (seen.has(row.callId)) continue
|
|
205
|
-
const stream = streams.get(row.streamId) ?? await readBillionStreamAsync(ledgerPath, row, archives)
|
|
208
|
+
const stream = streams.get(row.streamId) ?? await readBillionStreamAsync(ledgerPath, row, archives, signal)
|
|
206
209
|
if (!stream?.finished || !['full', 'partial'].includes(stream.policies?.[row.groupId])) continue
|
|
207
210
|
seen.add(row.callId); streams.set(stream.streamId, stream)
|
|
208
211
|
if (seen.size > 200000) throw Error('Proxy billing detail limit exceeded')
|
|
@@ -7,31 +7,40 @@ const id = '[a-f\\d]{8}(?:-[a-f\\d]{4}){3}-[a-f\\d]{12}'
|
|
|
7
7
|
const streamName = new RegExp('^' + id + '\\.json$', 'i')
|
|
8
8
|
const callName = new RegExp('^(' + id + ')(?:\\.sealed)?\\.jsonl(?:\\.gz)?$', 'i')
|
|
9
9
|
const archiveName = new RegExp('^' + id + '\\.jsonl\\.gz$', 'i')
|
|
10
|
-
export async function safeEntries(path) {
|
|
10
|
+
export async function safeEntries(path, signal) {
|
|
11
|
+
signal?.throwIfAborted()
|
|
11
12
|
try {
|
|
12
13
|
const info = await lstat(path)
|
|
14
|
+
signal?.throwIfAborted()
|
|
13
15
|
if (!info.isDirectory() || info.isSymbolicLink()) return []
|
|
14
|
-
|
|
16
|
+
const entries = await readdir(path, { withFileTypes: true })
|
|
17
|
+
signal?.throwIfAborted()
|
|
18
|
+
return entries
|
|
15
19
|
} catch (error) { if (error.code === 'ENOENT') return []; throw error }
|
|
16
20
|
}
|
|
17
|
-
export async function readSmallBillionJson(path) {
|
|
21
|
+
export async function readSmallBillionJson(path, signal) {
|
|
22
|
+
signal?.throwIfAborted()
|
|
18
23
|
const info = await lstat(path)
|
|
19
24
|
if (!info.isFile() || info.isSymbolicLink() || info.size > 8192) throw Object.assign(Error('Invalid proxy billing metadata'), { code: 'INVALID_BILLION_METADATA' })
|
|
20
|
-
|
|
25
|
+
const text = await readFile(path, { encoding: 'utf8', signal })
|
|
26
|
+
signal?.throwIfAborted()
|
|
27
|
+
return JSON.parse(text)
|
|
21
28
|
}
|
|
22
29
|
export function createBillionInventory(ledgerPath) {
|
|
23
30
|
const root = ledgerPath + '.billion-context'
|
|
24
31
|
let cache = new Map()
|
|
25
32
|
return {
|
|
26
|
-
async poll() {
|
|
33
|
+
async poll(signal) {
|
|
34
|
+
signal?.throwIfAborted()
|
|
27
35
|
const streams = [], calls = [], archives = [], seen = new Set(), next = new Map(cache)
|
|
28
36
|
let changed = false
|
|
29
37
|
// Inspect directories, not every completed stream, on an idle tick. Calls
|
|
30
38
|
// remain candidates because append-only writes do not change directory mtime.
|
|
31
|
-
if (!(await safeEntries(root)).length) { cache.clear(); return { streams, calls, archives, retained: new Set() } }
|
|
39
|
+
if (!(await safeEntries(root, signal)).length) { cache.clear(); return { streams, calls, archives, retained: new Set() } }
|
|
32
40
|
for (const kind of ['streams', 'calls', 'stream-archives']) {
|
|
33
41
|
const parent = join(root, kind)
|
|
34
|
-
for (const entry of await safeEntries(parent)) {
|
|
42
|
+
for (const entry of await safeEntries(parent, signal)) {
|
|
43
|
+
signal?.throwIfAborted()
|
|
35
44
|
if (!entry.isDirectory() || !sessionKey.test(entry.name)) continue
|
|
36
45
|
const directory = join(parent, entry.name)
|
|
37
46
|
let info
|
|
@@ -42,7 +51,7 @@ export function createBillionInventory(ledgerPath) {
|
|
|
42
51
|
let files = previous?.files
|
|
43
52
|
if (previous?.revision !== revision) {
|
|
44
53
|
changed = true
|
|
45
|
-
files = (await safeEntries(directory)).flatMap(file => {
|
|
54
|
+
files = (await safeEntries(directory, signal)).flatMap(file => {
|
|
46
55
|
if (!file.isFile()) return []
|
|
47
56
|
const match = kind === 'calls' ? callName.exec(file.name) : (kind === 'streams' ? streamName : archiveName).exec(file.name)
|
|
48
57
|
return match ? [{ path: join(directory, file.name), key: entry.name, ...(kind === 'calls' ? { source: match[1] } : {}) }] : []
|
|
@@ -55,6 +64,7 @@ export function createBillionInventory(ledgerPath) {
|
|
|
55
64
|
}
|
|
56
65
|
for (const directory of next.keys()) if (!seen.has(directory)) { next.delete(directory); changed = true }
|
|
57
66
|
// A failed poll must not mark undispatched candidates as already scanned.
|
|
67
|
+
signal?.throwIfAborted()
|
|
58
68
|
cache = next
|
|
59
69
|
const retained = changed ? new Set([...cache.values()].flatMap(item => item.files.map(file => file.path))) : null
|
|
60
70
|
return { streams, calls, archives, retained }
|
|
@@ -18,27 +18,29 @@ const contains = (row, delta) => row && fields.every(key => finite(amount(row, k
|
|
|
18
18
|
&& amount(row, key) + (['input', 'output', 'cacheRead', 'cacheWrite', 'reasoning', 'calls'].includes(key) ? 0 : 1e-8) >= delta[key])
|
|
19
19
|
const subtract = (row, delta) => { for (const key of fields) row[key] = Math.max(0, amount(row, key) - delta[key]) }
|
|
20
20
|
|
|
21
|
-
async function affectedStreams(path) {
|
|
22
|
-
const inventory = await createBillionInventory(path).poll(), streams = new Map()
|
|
21
|
+
async function affectedStreams(path, signal) {
|
|
22
|
+
const inventory = await createBillionInventory(path).poll(signal), streams = new Map()
|
|
23
23
|
for (const file of inventory.streams) {
|
|
24
|
-
const stream = await readBillionStreamFileAsync(file)
|
|
24
|
+
const stream = await readBillionStreamFileAsync(file, signal)
|
|
25
25
|
if (stream?.finished) streams.set(stream.streamId, stream)
|
|
26
26
|
}
|
|
27
27
|
// An archive can coexist with an active original during compaction. The active
|
|
28
28
|
// original wins; for duplicate archives the newest published record wins.
|
|
29
29
|
const active = new Set(streams.keys()), archives = []
|
|
30
30
|
for (const file of inventory.archives) {
|
|
31
|
+
signal?.throwIfAborted()
|
|
31
32
|
try { archives.push({ ...file, at: (await lstat(file.path)).mtimeMs }) }
|
|
32
33
|
catch (error) { if (error.code !== 'ENOENT') throw error }
|
|
33
34
|
}
|
|
34
35
|
archives.sort((a, b) => a.at - b.at || a.path.localeCompare(b.path))
|
|
35
|
-
for (const file of archives) for (const stream of await readBillionStreamArchive(file)) {
|
|
36
|
+
for (const file of archives) for (const stream of await readBillionStreamArchive(file, signal)) {
|
|
36
37
|
if (!active.has(stream.streamId)) streams.set(stream.streamId, stream)
|
|
37
38
|
if (streams.size > 200000) throw Error('Proxy billing repair stream limit exceeded')
|
|
38
39
|
}
|
|
39
40
|
const affected = []
|
|
40
41
|
for (const stream of streams.values()) {
|
|
41
|
-
|
|
42
|
+
signal?.throwIfAborted()
|
|
43
|
+
const coverage = checkedBillionCoverage(stream, stream.coverage ?? await billionCoverage(path, stream, undefined, signal))
|
|
42
44
|
if (unverifiedBillionHost(coverage)) affected.push(stream)
|
|
43
45
|
}
|
|
44
46
|
return affected
|
|
@@ -49,10 +51,12 @@ async function affectedStreams(path) {
|
|
|
49
51
|
* admitted. Sequence 2 retires that sequence-1 charge in the same transaction as
|
|
50
52
|
* its subtraction. No global residual deletion or current-price recalculation.
|
|
51
53
|
*/
|
|
52
|
-
export async function repairBillionHostLedger(ledger) {
|
|
54
|
+
export async function repairBillionHostLedger(ledger, signal) {
|
|
55
|
+
signal?.throwIfAborted()
|
|
53
56
|
const stats = { repaired: 0, skipped: 0, input: 0, cost: 0, backup: '' }
|
|
54
57
|
if (!ledger.path || ledger.closed) return stats
|
|
55
|
-
const streams = await affectedStreams(ledger.path)
|
|
58
|
+
const streams = await affectedStreams(ledger.path, signal)
|
|
59
|
+
signal?.throwIfAborted()
|
|
56
60
|
if (!streams.length || ledger.closed) return stats
|
|
57
61
|
// Drain our worker before acquiring its lock; keep other hosts' new calls by
|
|
58
62
|
// reading the latest disk snapshot inside that lock, then adopt the result.
|
|
@@ -42,10 +42,14 @@ export function validArchivedStream(stream, key) {
|
|
|
42
42
|
return stream?.protocol === 1 && uuid.test(stream.streamId ?? '') && typeof stream.sessionId === 'string'
|
|
43
43
|
&& stream.sessionId.length > 0 && stream.sessionId.length <= 512 && keyOf(stream.sessionId) === key && stream.finished === true
|
|
44
44
|
}
|
|
45
|
-
export async function readBillionStreamArchive(file) {
|
|
45
|
+
export async function readBillionStreamArchive(file, signal) {
|
|
46
|
+
signal?.throwIfAborted()
|
|
46
47
|
const info = await fileInfo(file.path)
|
|
47
48
|
if (!info || info.size > MAX_ARCHIVE) throw Error('Invalid archived stream metadata')
|
|
48
|
-
const
|
|
49
|
+
const compressed = await readFile(file.path, { signal })
|
|
50
|
+
signal?.throwIfAborted()
|
|
51
|
+
const bytes = await decompress(compressed, { maxOutputLength: MAX_ARCHIVE })
|
|
52
|
+
signal?.throwIfAborted()
|
|
49
53
|
const rows = []
|
|
50
54
|
for (const line of bytes.toString('utf8').split('\n')) {
|
|
51
55
|
if (!line) continue
|
|
@@ -56,18 +60,20 @@ export async function readBillionStreamArchive(file) {
|
|
|
56
60
|
}
|
|
57
61
|
return rows
|
|
58
62
|
}
|
|
59
|
-
export async function archivedBillionStreams(ledgerPath, sessionId) {
|
|
63
|
+
export async function archivedBillionStreams(ledgerPath, sessionId, signal) {
|
|
64
|
+
signal?.throwIfAborted()
|
|
60
65
|
const key = keyOf(sessionId), root = resolve(ledgerPath + '.billion-context'), dir = join(root, 'stream-archives', key), streams = new Map()
|
|
61
66
|
try { await confined(root, join(dir, 'record')) } catch (error) { if (error.code === 'ENOENT') return streams; throw error }
|
|
62
67
|
const files = []
|
|
63
|
-
for (const entry of await safeEntries(dir)) {
|
|
68
|
+
for (const entry of await safeEntries(dir, signal)) {
|
|
69
|
+
signal?.throwIfAborted()
|
|
64
70
|
if (!entry.isFile() || !entry.name.endsWith('.jsonl.gz') || !uuid.test(entry.name.slice(0, -9))) continue
|
|
65
71
|
const path = join(dir, entry.name), info = await fileInfo(path)
|
|
66
72
|
if (info) files.push({ path, key, at: info.mtimeMs })
|
|
67
73
|
}
|
|
68
74
|
files.sort((a, b) => a.at - b.at || a.path.localeCompare(b.path))
|
|
69
75
|
for (const file of files) {
|
|
70
|
-
for (const row of await readBillionStreamArchive(file)) streams.set(row.streamId, row)
|
|
76
|
+
for (const row of await readBillionStreamArchive(file, signal)) streams.set(row.streamId, row)
|
|
71
77
|
if (streams.size > 200000) throw Error('Proxy billing detail limit exceeded')
|
|
72
78
|
}
|
|
73
79
|
return streams
|