dsh-cost-meter 1.7.6 → 1.7.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +21 -7
- package/README.md +21 -7
- package/lib/backfill.js +173 -55
- package/lib/client.js +2 -2
- package/lib/gateway-quota-adapters.js +945 -0
- package/lib/gateway-quotas.js +577 -0
- package/lib/index.js +184 -8
- package/lib/net.js +3 -2
- package/lib/store.js +225 -2
- package/lib/typert.host.js +70 -0
- package/package.json +1 -1
package/README.en.md
CHANGED
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
Per-conversation cost · daily totals · OpenCode Go subscription quota display · budget with usage percentage · official account balance · custom provider balance · balance progress bar · history · peak/off-peak pricing hours display (peak hours UTC 01:00–04:00, 06:00–10:00; from Aug 23, 2026 weekends are billed at off-peak prices all day, shown as “Weekend — all off-peak”) · pre-switch popup & system-notification alerts for peak/off-peak changes (position / lead time / alert type configurable) · one-click price sync from the official docs · Codex-style token usage heat grid · multi-vendor model pricing (built-in 90+ model price catalog with auto-matching) · mainstream Coding Plan quota queries & display (Anthropic / Z.ai / MiniMax / Kimi / OpenRouter / SiliconFlow / CommandCode / SCNet) plan/API dual-track billing (subscription quota vs pay-as-you-go money separated, per-1% & full-window token/equivalent-cost estimates with daily/weekly/monthly curves) · · quota strip above the input box (budget / Go / coding-plan usage in one row, toggleable)
|
|
8
8
|
|
|
9
|
-
[](https://github.com/Han-1413141/dsh-cost-meter)
|
|
10
10
|
[](https://www.npmjs.com/package/dsh-cost-meter)
|
|
11
11
|
[](LICENSE)
|
|
12
12
|
[](https://github.com/deepseek-ai/deepseek-harness)
|
|
@@ -87,13 +87,27 @@ For NewApi `GET /api/usage/token` (response `{ "code": 200, "data": { "total_gra
|
|
|
87
87
|
- Unlimited-quota tokens (`unlimited_quota: true`) have no `total_available`, so `remaining` cannot be extracted and the query reports “remaining is missing or not numeric” — use a limited-quota token or a middle-layer endpoint that converts the units;
|
|
88
88
|
- Entry point: Settings → Cost (Quota tab) → “Custom provider balance” → expand config; or write `config.customBalance` in `storages/cost-meter/ledger.json`.
|
|
89
89
|
|
|
90
|
-
### Credentials & security (v1.7.
|
|
90
|
+
### Credentials & security (v1.7.8)
|
|
91
91
|
|
|
92
92
|
- **Variable naming**: `{{VAR_NAME}}` follows the `<ROUTE>_API_KEY` convention — `<ROUTE>` is the Provider ID from the DSH Models page (Settings → Models), uppercased with non-alphanumeric characters replaced by underscores, e.g. `openai`→`{{OPENAI_API_KEY}}`, `anthropic`→`{{ANTHROPIC_API_KEY}}`, `abc23-d`→`{{ABC23_D_API_KEY}}`. Sharing a name with the Models page means the balance query and model calls **share the same key** (both resolve from the DSH credential store). This note is also shown above the “Headers (JSON)” input in Settings.
|
|
93
93
|
- **Credential input fields**: after expanding an entry, the “Credential input” section renders one write-only field per `{{VAR}}` placeholder found in the headers — the key goes straight into the DSH credential store (never written to disk, never echoed back, never stored in `ledger.json`); no need to hand-edit environment variables or credential files.
|
|
94
|
-
- **Automatic plaintext migration**: older versions let a literal `Bearer sk-…` in the headers leak into `ledger.json` in plaintext. Since v1.7.
|
|
94
|
+
- **Automatic plaintext migration**: older versions let a literal `Bearer sk-…` in the headers leak into `ledger.json` in plaintext. Since v1.7.8 the plugin imports such keys into the DSH credential store at startup and replaces the header value with a `{{CUSTOM_BALANCE_KEY_…}}` placeholder (derived from the entry's host + header name, stable across restarts) — nothing breaks. From now on `ledger.json` and the config shipped to the browser **never contain plaintext keys**: suspected secret headers (Authorization / X-Api-Key / Bearer / sk- prefixes / long opaque strings) are blanked, while placeholders and ordinary headers pass through.
|
|
95
95
|
- **Credential allowlist `allowedHosts`**: when headers carry credentials (placeholders or plaintext), the outbound host must be on this list or the request is refused — protection against leaked keys when importing someone else's config. Without a list, requests proceed with a one-time logged warning. The entry panel provides an “Allowed hosts” input (comma-separated).
|
|
96
96
|
|
|
97
|
+
## CLIProxyAPI Gateway Quotas and WorkBuddy Credits (Issue #87)
|
|
98
|
+
|
|
99
|
+
Connects to a local or LAN-deployed [CLIProxyAPI](https://github.com/router-for-me/CLIProxyAPI) proxy gateway to observe quotas across upstream providers and WorkBuddy plugin credits in one place:
|
|
100
|
+
|
|
101
|
+
- **Native Multi-Provider Discovery & Quotas**: Auto-discovers active credentials via the CPA Management API and queries official native endpoints for Antigravity, Claude, Codex, Kimi, and xAI (Grok). Normalized into standard 5h / 7d / weekly / monthly usage percentage windows with ISO reset timestamps.
|
|
102
|
+
- **WorkBuddy Plugin Credits**: Queries WorkBuddy's read-only management route (`/v0/management/plugins/workbuddy/credits`) to display total, used, and remaining credit balances alongside package lifecycle end-dates, preserving separate account cards.
|
|
103
|
+
- **Zero Metering Side-Effects Guarantee**: Completely read-only — strictly avoids mutating actions such as Codex reset-credit consume, xAI token-consuming chat completion probes, and any WorkBuddy POST routes.
|
|
104
|
+
- **Strict Credential Isolation & Transport Security**:
|
|
105
|
+
- Management Keys are stored strictly write-only in the DSH credential store (`CLIPROXYAPI_MANAGEMENT_KEY_<SOURCE_ID>_<HASH>`), never exposed in config, state, or browser views.
|
|
106
|
+
- Management calls are strictly restricted to 3 fixed paths (`/v0/management/auth-files`, `/v0/management/api-call`, `/v0/management/plugins/workbuddy/credits`). Arbitrary upstream URLs are forbidden.
|
|
107
|
+
- Outbound host whitelist (`allowedHosts`): Non-loopback origins require exact host matches, and plain HTTP requires an explicit `allowInsecureHttp` opt-in.
|
|
108
|
+
- HTTP redirects are strictly forbidden (`redirect: 'manual'`; any 3xx response is rejected) to guard against credential leakage or request hijacking.
|
|
109
|
+
- Privacy and metadata sanitization: Strips raw tokens, cookies, and identity claims from auth-files discovery; emails are masked in public state (`s***@domain`).
|
|
110
|
+
|
|
97
111
|
## Bilingual UI
|
|
98
112
|
|
|
99
113
|
The plugin UI (session badge, sidebar balance row & budget box, and the entire Settings page) supports **Simplified Chinese** and **English**:
|
|
@@ -238,22 +252,22 @@ Real captures from an actual DSH sidebar of the period strip and collapsed verti
|
|
|
238
252
|
dsh plugin --profile web add dsh-cost-meter
|
|
239
253
|
```
|
|
240
254
|
|
|
241
|
-
**PowerShell one-click script** (copy the whole line, paste, press Enter; pnpm is provisioned automatically, git is auto-detected — no clone needed; the install chain is **pinned to the release tag `v1.7.
|
|
255
|
+
**PowerShell one-click script** (copy the whole line, paste, press Enter; pnpm is provisioned automatically, git is auto-detected — no clone needed; the install chain is **pinned to the release tag `v1.7.8`** — review the script before running):
|
|
242
256
|
|
|
243
257
|
```powershell
|
|
244
|
-
irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.7.
|
|
258
|
+
irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.7.8/install.ps1 | iex
|
|
245
259
|
```
|
|
246
260
|
|
|
247
261
|
**Or a plain command line** (the machine must already have pnpm and git; also pinned to the tag):
|
|
248
262
|
|
|
249
263
|
```sh
|
|
250
|
-
dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.7.
|
|
264
|
+
dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.7.8
|
|
251
265
|
```
|
|
252
266
|
|
|
253
267
|
Without git, use the GitHub tag archive:
|
|
254
268
|
|
|
255
269
|
```sh
|
|
256
|
-
dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.7.
|
|
270
|
+
dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.7.8.tar.gz
|
|
257
271
|
```
|
|
258
272
|
|
|
259
273
|
After installing, **restart** `dsh web` (plugin rows, the Typert manifest and the client bundle are all scanned at startup):
|
package/README.md
CHANGED
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
本会话费用 · 当日费用 · OpenCode Go 订阅额度显示 · 预算与已用百分比 · 官方账户余额 · 自定义 Provider 余额查询(可配任意 HTTP 端点) · 余额三段进度条 · 历史记录 · 峰谷计价时段显示(UTC 01:00–04:00、06:00–10:00 为峰时段;2026-08-23 起周末全天按谷价,显示「周末时段——全谷价」) · 峰/谷切换前弹窗与系统通知提醒(位置/提前量/提醒类型可配) · 官方价格一键同步 · 类 Codex Token 用量热图 · 多厂商多模型价格计费(内置 90+ 模型价格目录与自动匹配) · 主流 Coding Plan 额度查询与显示(Anthropic / Z.ai / MiniMax / Kimi / OpenRouter / SiliconFlow / CommandCode / SCNet / 火山方舟 九家,含 Volcano Ark AK/SK 签名) · Plan/API 双轨计费(订阅额度与按量金额分离统计,每 1% 额度与满窗的 token/等值金额估算及日/周/月曲线) · 输入框上方额度横条(预算/Go/Coding Plan 用量一条横排显示,可开关)
|
|
8
8
|
|
|
9
|
-
[](https://github.com/Han-1413141/dsh-cost-meter)
|
|
10
10
|
[](https://www.npmjs.com/package/dsh-cost-meter)
|
|
11
11
|
[](LICENSE)
|
|
12
12
|
[](https://github.com/deepseek-ai/deepseek-harness)
|
|
@@ -88,13 +88,27 @@
|
|
|
88
88
|
- 无限额度 token(`unlimited_quota: true`)没有 `total_available`,无法提取 `remaining`,查询会报「remaining is missing or not numeric」——请改用有限额度 token,或在中间层端点换算;
|
|
89
89
|
- 配置入口:设置 → 费用(额度标签)→「自定义 Provider 余额」展开配置;或直接改 `storages/cost-meter/ledger.json` 的 `config.customBalance`。
|
|
90
90
|
|
|
91
|
-
### 凭据与安全(v1.7.
|
|
91
|
+
### 凭据与安全(v1.7.8)
|
|
92
92
|
|
|
93
93
|
- **变量名命名规则**:`{{VAR_NAME}}` 的参考格式为 `<ROUTE>_API_KEY`——`<ROUTE>` 对应 DSH 模型配置页(设置 → 模型)里的 Provider ID,把 ID 大写、非字母数字字符替换为下划线,例如 `openai`→`{{OPENAI_API_KEY}}`、`anthropic`→`{{ANTHROPIC_API_KEY}}`、`abc23-d`→`{{ABC23_D_API_KEY}}`。与模型页共用同一变量名,自定义余额查询与模型调用即**共用同一把密钥**(都从 DSH 凭据库解析)。该说明也展示在设置页「请求头 (JSON)」输入框上方。
|
|
94
94
|
- **凭据输入框**:展开条目配置后,「凭据输入」区会为请求头里出现的每个 `{{VAR}}` 占位符显示一行 write-only 输入框,密钥直接存入 DSH 凭据库(不落盘、不回显、不经 `ledger.json`),无需再手改环境变量或凭据文件。
|
|
95
|
-
- **明文密钥自动迁移**:旧版本把 `Bearer sk-…` 明文写在请求头里时会明文落盘;v1.7.
|
|
95
|
+
- **明文密钥自动迁移**:旧版本把 `Bearer sk-…` 明文写在请求头里时会明文落盘;v1.7.8 起插件在启动时自动把这类明文导入 DSH 凭据库,并把头值替换为 `{{CUSTOM_BALANCE_KEY_…}}` 占位符(名称由条目 host + 头名派生,跨重启稳定),功能不受影响。此后 `ledger.json` 与下发给浏览器的配置**永不包含明文密钥**——疑似密钥头(Authorization / X-Api-Key / Bearer / sk- 前缀 / 长不透明串)一律置空,占位符与普通头照常保留。
|
|
96
96
|
- **凭据白名单 `allowedHosts`**:请求头携带密钥(占位符或明文)时,出站主机必须命中该白名单,否则直接拒绝——用于防止「导入他人配置」导致密钥外带。未配置白名单时放行并在日志警告一次。设置页条目面板内有「凭据白名单主机」输入框(逗号分隔)。
|
|
97
97
|
|
|
98
|
+
## CLIProxyAPI 网关额度与 WorkBuddy 积分 (Issue #87)
|
|
99
|
+
|
|
100
|
+
支持对接本地或局域网部署的 [CLIProxyAPI](https://github.com/router-for-me/CLIProxyAPI) 代理网关,集中观测代理的多个 Provider 账号额度与 WorkBuddy 插件积分:
|
|
101
|
+
|
|
102
|
+
- **多 Provider 账号原生适配**:自动通过 CPA Management API 发现账号,接入 Antigravity、Claude、Codex、Kimi、xAI (Grok) 官方原生配额/用量接口。统一归一化为 5h / 7d / weekly / monthly 等用量百分比窗口与 ISO 重置时刻。
|
|
103
|
+
- **WorkBuddy 插件积分**:只读接入 WorkBuddy 插件 `/v0/management/plugins/workbuddy/credits` 路由,展示账户总积分、已用/剩余与套餐周期,保留独立账号卡片。
|
|
104
|
+
- **零计费副作用保证**:严禁并杜绝任何产生计费消耗的操作——不调用 Codex 的 reset-credit consume 端点,不向 xAI 发送消耗 token 的 chat probe,不调用 WorkBuddy 的任何变更性 POST 路由。
|
|
105
|
+
- **严格凭据隔离与安全门禁**:
|
|
106
|
+
- Management Key 通过只写 (write-only) 形式托管于 DSH 凭据库 (`CLIPROXYAPI_MANAGEMENT_KEY_<SOURCE_ID>_<HASH>`),永不存入配置文件、账本或浏览器 state。
|
|
107
|
+
- 请求路径严格限制为三条固定管理路径 (`/v0/management/auth-files`、`/v0/management/api-call`、`/v0/management/plugins/workbuddy/credits`),禁止任意 URL 拼接。
|
|
108
|
+
- 严格出站白名单 (`allowedHosts`):非 loopback 来源必须精确命中白名单,非 loopback HTTP 必须显式开启 `allowInsecureHttp`。
|
|
109
|
+
- 强制禁止 HTTP 重定向 (`redirect: 'manual'`, 3xx 状态码坚决拒绝),防止网关重定向劫持或凭据外带。
|
|
110
|
+
- 账号隐私与元数据过滤:auth-files 响应仅提取必需字段,剔除 raw tokens / cookies / ID claims,展示层邮箱自动脱敏掩码 (`s***@domain`)。
|
|
111
|
+
|
|
98
112
|
## 双语界面
|
|
99
113
|
|
|
100
114
|
插件界面(会话徽章、侧边栏余额与预算图框、设置页全部文案)支持**简体中文**与**English**:
|
|
@@ -240,22 +254,22 @@
|
|
|
240
254
|
dsh plugin --profile web add dsh-cost-meter
|
|
241
255
|
```
|
|
242
256
|
|
|
243
|
-
**PowerShell 一键脚本**(复制整行粘贴回车;自动补齐 pnpm、自动探测 git,无需克隆仓库;安装链**固定到发布 tag `v1.7.
|
|
257
|
+
**PowerShell 一键脚本**(复制整行粘贴回车;自动补齐 pnpm、自动探测 git,无需克隆仓库;安装链**固定到发布 tag `v1.7.8`**,建议先下载审阅再运行):
|
|
244
258
|
|
|
245
259
|
```powershell
|
|
246
|
-
irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.7.
|
|
260
|
+
irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.7.8/install.ps1 | iex
|
|
247
261
|
```
|
|
248
262
|
|
|
249
263
|
**或直接命令行**(机器上需已有 pnpm 与 git;同样固定到 tag):
|
|
250
264
|
|
|
251
265
|
```sh
|
|
252
|
-
dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.7.
|
|
266
|
+
dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.7.8
|
|
253
267
|
```
|
|
254
268
|
|
|
255
269
|
没有 git 时可用 GitHub tag 打包直链:
|
|
256
270
|
|
|
257
271
|
```sh
|
|
258
|
-
dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.7.
|
|
272
|
+
dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.7.8.tar.gz
|
|
259
273
|
```
|
|
260
274
|
|
|
261
275
|
安装后**重启** `dsh web`(插件行、Typert 清单与客户端 bundle 均在启动时扫描):
|
package/lib/backfill.js
CHANGED
|
@@ -10,7 +10,8 @@
|
|
|
10
10
|
* 避免与实时计费重复计数。会话日志是宿主的只读数据,本模块从不写入。
|
|
11
11
|
*/
|
|
12
12
|
|
|
13
|
-
import { readdirSync,
|
|
13
|
+
import { readdirSync, statSync } from 'node:fs'
|
|
14
|
+
import { open } from 'node:fs/promises'
|
|
14
15
|
import { join } from 'node:path'
|
|
15
16
|
import * as zlib from 'node:zlib'
|
|
16
17
|
import { costOf, providerPriceEntryFor, usdFromCost, isWrapperProviderId, wrapperUpstreamProvider } from './pricing.js'
|
|
@@ -22,6 +23,61 @@ const ZSTD_MAGIC = 4247762216
|
|
|
22
23
|
/** 打包行(文本/推理/工具调用增量游程)不含 header 与 usage,回放时跳过。 */
|
|
23
24
|
const PACKED_ROW_TYPES = new Set(['text-chunks', 'reasoning-chunks', 'tool-call-chunks'])
|
|
24
25
|
|
|
26
|
+
// ── 内存与事件循环守卫(v1.7.7,issue #88) ───────────────────────────────
|
|
27
|
+
//
|
|
28
|
+
// 用户实测:一份 168MB(压缩后)的 session.jsonl.zstd 在启动回填时把 Electron
|
|
29
|
+
// 主进程堆耗尽(STATUS_BREAKPOINT 崩溃),且同步解压全程占住事件循环(agent
|
|
30
|
+
// 界面整体卡死)。旧缓解(逐帧解压代替全文拼接)只压住了解压峰值,仍有两个
|
|
31
|
+
// 洞:records 数组物化全部解压后事件(堆)与巨长打包行的 JSON.parse(瞬时峰值)。
|
|
32
|
+
// 现三层防御:
|
|
33
|
+
// · 打包行探针:超 4096 字节的行先看头部 512 字符,命中打包行模式直接跳过,
|
|
34
|
+
// 不再付出 JSON.parse 的整行解析峰值(探针只做加速,短行/未命中仍走解析+过滤,
|
|
35
|
+
// 行为不变);
|
|
36
|
+
// · 流式迭代器 iterateSessionRecords:分块读文件 + 增量扫描帧边界,逐帧解压
|
|
37
|
+
// 逐行产出记录,任一时刻只持有单帧解压结果;配合定期 setImmediate 让出,
|
|
38
|
+
// 回填期间宿主 UI 不再被卡死;
|
|
39
|
+
// · 解压预算:单文件累计解压字节数超上限(默认 4GB,防解压炸弹)抛错,
|
|
40
|
+
// 调用方按「单文件损坏」跳过该文件,绝不把主进程拖崩。
|
|
41
|
+
|
|
42
|
+
/** 打包行探针:巨长行头部出现 "type":"text-chunks" 等即判打包行,免整行解析。 */
|
|
43
|
+
const PACKED_ROW_PROBE = /"type"\s*:\s*"(?:text|reasoning|tool-call)-chunks"/
|
|
44
|
+
const PACKED_ROW_PROBE_HEAD = 512
|
|
45
|
+
const PACKED_ROW_PROBE_MIN_LINE = 4096
|
|
46
|
+
/** 单文件解压累计上限(防解压炸弹;正常会话日志解压后数百 MB~2GB)。 */
|
|
47
|
+
const MAX_DECOMPRESSED_PER_FILE = 4 * 1024 * 1024 * 1024
|
|
48
|
+
/** 流式读文件的分块大小。 */
|
|
49
|
+
const READ_CHUNK_BYTES = 8 * 1024 * 1024
|
|
50
|
+
/** 每 N 帧解压/每 N 条事件让出一次事件循环(保持宿主 UI 响应)。 */
|
|
51
|
+
const YIELD_EVERY_FRAMES = 64
|
|
52
|
+
const YIELD_EVERY_EVENTS = 4096
|
|
53
|
+
|
|
54
|
+
/** 巨长行探针判定(仅加速路径,未命中仍走 JSON.parse + 类型过滤)。 */
|
|
55
|
+
function lineLooksPacked(line) {
|
|
56
|
+
if (line.length <= PACKED_ROW_PROBE_MIN_LINE) return false
|
|
57
|
+
return PACKED_ROW_PROBE.test(line.slice(0, PACKED_ROW_PROBE_HEAD))
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
/**
|
|
61
|
+
* 解析单行日志为事件对象;空行、打包行与坏行返回 null。
|
|
62
|
+
* 短行走「解析后按类型丢弃」;巨长行先探针,命中打包行直接跳过(免解析峰值)。
|
|
63
|
+
* @param {string} line
|
|
64
|
+
* @returns {object | null}
|
|
65
|
+
*/
|
|
66
|
+
export function parseRecordLine(line) {
|
|
67
|
+
if (line.length === 0) return null
|
|
68
|
+
if (lineLooksPacked(line)) return null
|
|
69
|
+
try {
|
|
70
|
+
const rec = JSON.parse(line)
|
|
71
|
+
// 打包行(text/reasoning/tool-call 增量游程)在解析入口就丢弃:它们是
|
|
72
|
+
// 大日志的体积主体,回放阶段反正会跳过,提前过滤才能兑现 OOM 缓解。
|
|
73
|
+
if (rec !== null && typeof rec === 'object' && !PACKED_ROW_TYPES.has(rec.type)) return rec
|
|
74
|
+
return null
|
|
75
|
+
} catch {
|
|
76
|
+
// 坏行跳过:回放是尽力而为,不让单行损坏阻断整个会话。
|
|
77
|
+
return null
|
|
78
|
+
}
|
|
79
|
+
}
|
|
80
|
+
|
|
25
81
|
/**
|
|
26
82
|
* 结构化扫描拼接的 Zstandard frame 边界(不解压块内容),与宿主
|
|
27
83
|
* dsh-session-persistence-jsonl 的容器格式一致:每个追加批次一个独立
|
|
@@ -73,57 +129,117 @@ export function scanZstdFrames(buffer) {
|
|
|
73
129
|
}
|
|
74
130
|
|
|
75
131
|
/**
|
|
76
|
-
*
|
|
132
|
+
* 流式逐行读取一份会话日志(异步生成器,v1.7.7 issue #88 的核心原语)。
|
|
133
|
+
*
|
|
134
|
+
* zstd 容器按帧边界增量扫描:分块读入原始字节(READ_CHUNK_BYTES),已确认完整的
|
|
135
|
+
* 帧立即解压并逐行产出、随后释放引用——任一时刻只持有「单帧解压结果 + 尾部
|
|
136
|
+
* 跨块/跨帧的行缓冲 + 已积累的记录数组」,不再物化全文。巨长打包行经探针
|
|
137
|
+
* 跳过(免 JSON.parse 峰值)。每 YIELD_EVERY_FRAMES 帧让出一次事件循环。
|
|
77
138
|
*
|
|
78
|
-
*
|
|
79
|
-
*
|
|
80
|
-
* Buffer.concat 拼成全文再 split——峰值内存可达数 GB,4GB 堆限制下
|
|
81
|
-
* 会 OOM(实测一份 50MB / 11.5 万帧日志即 ~3GB)。现改为逐帧解压、
|
|
82
|
-
* 逐帧按行切片解析:任一时刻只保留单帧解压结果(通常 ≤ 数 MB),行缓冲
|
|
83
|
-
* 跨帧拼接兜底半行(追加批次以换行结尾,实际不会出现)。
|
|
139
|
+
* 单文件累计解压字节数超过 maxDecompressed(默认 4GB,防解压炸弹)抛
|
|
140
|
+
* RangeError,调用方按单文件损坏处理。
|
|
84
141
|
* @param path - session.jsonl.zstd 或 session.jsonl 路径。
|
|
85
|
-
* @
|
|
142
|
+
* @param options - { maxDecompressed?: number, yieldEvery?: number }。
|
|
143
|
+
* @yields {object} 逐行解析出的记录(空行/打包行/坏行已过滤)。
|
|
86
144
|
*/
|
|
87
|
-
export function
|
|
88
|
-
const
|
|
89
|
-
const
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
145
|
+
export async function* iterateSessionRecords(path, options = {}) {
|
|
146
|
+
const maxDecompressed = Number(options.maxDecompressed) > 0 ? Number(options.maxDecompressed) : MAX_DECOMPRESSED_PER_FILE
|
|
147
|
+
const yieldEvery = Number(options.yieldEvery) > 0 ? Number(options.yieldEvery) : YIELD_EVERY_FRAMES
|
|
148
|
+
const isZstd = path.endsWith('.zstd')
|
|
149
|
+
if (isZstd && typeof zlib.zstdDecompressSync !== 'function') return
|
|
150
|
+
const handle = await open(path, 'r')
|
|
151
|
+
try {
|
|
152
|
+
let pending = '' // 跨块/跨帧行缓冲:上一段末尾未换行的残行拼接进下一段首行。
|
|
153
|
+
let tail = Buffer.alloc(0) // 压缩字节尾部缓冲:块边界落在帧中间时拼接后继续扫。
|
|
154
|
+
let decompressedTotal = 0
|
|
155
|
+
let framesSinceYield = 0
|
|
156
|
+
const consumeText = (text, out) => {
|
|
157
|
+
if (text.length === 0) return
|
|
158
|
+
const lines = text.split('\n')
|
|
159
|
+
lines[0] = pending + lines[0]
|
|
160
|
+
pending = lines[lines.length - 1]
|
|
161
|
+
for (let i = 0; i < lines.length - 1; i++) {
|
|
162
|
+
const rec = parseRecordLine(lines[i])
|
|
163
|
+
if (rec !== null) out.push(rec)
|
|
106
164
|
}
|
|
107
165
|
}
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
166
|
+
const records = []
|
|
167
|
+
// 结构损坏标志:一旦出现「一个完整分块里连一个可解帧都没有」,按旧版
|
|
168
|
+
// 全量扫描的语义(在首个非法结构处停止)忽略文件后续内容——否则坏字节
|
|
169
|
+
// 会在 tail 里无限拼接,且行为与旧版不一致。
|
|
170
|
+
let stopped = false
|
|
171
|
+
for (;;) {
|
|
172
|
+
const chunk = await handle.read(Buffer.allocUnsafe(READ_CHUNK_BYTES), 0, READ_CHUNK_BYTES, null)
|
|
173
|
+
if (chunk.bytesRead === 0) break
|
|
174
|
+
const data = tail.length > 0 ? Buffer.concat([tail, chunk.buffer.subarray(0, chunk.bytesRead)]) : chunk.buffer.subarray(0, chunk.bytesRead)
|
|
175
|
+
tail = Buffer.alloc(0)
|
|
176
|
+
if (stopped) continue
|
|
177
|
+
if (!isZstd) {
|
|
178
|
+
consumeText(data.toString('utf8'), records)
|
|
179
|
+
if (records.length >= YIELD_EVERY_EVENTS) {
|
|
180
|
+
yield* drain(records)
|
|
181
|
+
await new Promise(resolve => setImmediate(resolve))
|
|
182
|
+
}
|
|
183
|
+
continue
|
|
184
|
+
}
|
|
185
|
+
// 增量消费完整帧:从 data 头部起逐帧取边界(帧区间相对 subarray 起点),
|
|
186
|
+
// 解压→逐行→释放引用;剩余不完整帧(块边界截断)留在 tail 等下一块。
|
|
187
|
+
let offset = 0
|
|
188
|
+
while (offset < data.length) {
|
|
189
|
+
const frames = scanZstdFrames(data.subarray(offset))
|
|
190
|
+
if (frames.length === 0) {
|
|
191
|
+
// 剩余字节连不出完整帧:未满一个分块 = 尾部截断,留 tail 等下一块;
|
|
192
|
+
// 已满一个分块仍无帧 = 结构损坏(与旧版全量扫描停在非法结构同语义),
|
|
193
|
+
// 标记停止,忽略文件后续。
|
|
194
|
+
if (data.length - offset >= READ_CHUNK_BYTES) stopped = true
|
|
195
|
+
else tail = data.subarray(offset)
|
|
196
|
+
break
|
|
197
|
+
}
|
|
198
|
+
for (const f of frames) {
|
|
199
|
+
decompressedTotal += f.end - f.start
|
|
200
|
+
if (decompressedTotal > maxDecompressed) {
|
|
201
|
+
throw new RangeError(`session log decompress budget exceeded (${decompressedTotal} > ${maxDecompressed} bytes): ${path}`)
|
|
202
|
+
}
|
|
203
|
+
const plain = zlib.zstdDecompressSync(data.subarray(offset + f.start, offset + f.end))
|
|
204
|
+
consumeText(plain.toString('utf8'), records)
|
|
205
|
+
framesSinceYield += 1
|
|
206
|
+
}
|
|
207
|
+
offset += frames[frames.length - 1].end
|
|
208
|
+
}
|
|
209
|
+
if (framesSinceYield >= yieldEvery || records.length >= YIELD_EVERY_EVENTS) {
|
|
210
|
+
yield* drain(records)
|
|
211
|
+
framesSinceYield = 0
|
|
212
|
+
await new Promise(resolve => setImmediate(resolve))
|
|
213
|
+
}
|
|
214
|
+
}
|
|
215
|
+
// 文件读完:tail 里剩的残帧(崩溃截断)按既有语义忽略;行缓冲兜底末行。
|
|
216
|
+
if (pending.length > 0) {
|
|
217
|
+
const rec = parseRecordLine(pending)
|
|
218
|
+
if (rec !== null) records.push(rec)
|
|
125
219
|
}
|
|
220
|
+
yield* drain(records)
|
|
221
|
+
} finally {
|
|
222
|
+
await handle.close()
|
|
126
223
|
}
|
|
224
|
+
}
|
|
225
|
+
|
|
226
|
+
/** 把 records 数组逐条 yield 出去并清空(内部助手)。 */
|
|
227
|
+
function* drain(records) {
|
|
228
|
+
for (const rec of records) yield rec
|
|
229
|
+
records.length = 0
|
|
230
|
+
}
|
|
231
|
+
|
|
232
|
+
/**
|
|
233
|
+
* 读取一份会话日志的全部事件行(zstd 逐 frame 解压;明文直接按行)。
|
|
234
|
+
* 同步兼容包装:逐行收集成数组。新的回填路径应改用流式 iterateSessionRecords
|
|
235
|
+
* (issue #88:records 数组本身在大日志下仍是堆压力,收集形态只留给需要数组
|
|
236
|
+
* 随机访问的调用方)。
|
|
237
|
+
* @param path - session.jsonl.zstd 或 session.jsonl 路径。
|
|
238
|
+
* @returns 逐行 JSON.parse 后的记录数组(坏行跳过)。
|
|
239
|
+
*/
|
|
240
|
+
export async function readSessionRecordsAsync(path, options = {}) {
|
|
241
|
+
const records = []
|
|
242
|
+
for await (const rec of iterateSessionRecords(path, options)) records.push(rec)
|
|
127
243
|
return records
|
|
128
244
|
}
|
|
129
245
|
|
|
@@ -448,14 +564,15 @@ export async function backfillLegacyLedger(ledger, sessionsRoot) {
|
|
|
448
564
|
}
|
|
449
565
|
}
|
|
450
566
|
for (const path of listSessionLogs(sessionsRoot, onlySessionIds)) {
|
|
451
|
-
// 会话日志多时逐份解压会长时间占住事件循环:每 8 份让出一次,不卡宿主 UI
|
|
567
|
+
// 会话日志多时逐份解压会长时间占住事件循环:每 8 份让出一次,不卡宿主 UI;
|
|
568
|
+
// 单文件内部亦为流式读取(v1.7.7,issue #88:解压期间周期性让出,大日志不再卡死宿主)。
|
|
452
569
|
if ((scannedCount += 1) % 8 === 0) await new Promise(resolve => setImmediate(resolve))
|
|
453
570
|
result.scanned += 1
|
|
454
571
|
let replayed
|
|
455
572
|
try {
|
|
456
|
-
replayed = replaySessionRecords(
|
|
573
|
+
replayed = replaySessionRecords(await readSessionRecordsAsync(path), ledger.config, needDates.size > 0 ? needDates : new Set(['-']))
|
|
457
574
|
} catch {
|
|
458
|
-
continue //
|
|
575
|
+
continue // 单文件损坏(含解压预算超限)不阻断整体回填。
|
|
459
576
|
}
|
|
460
577
|
if (replayed.sessionId.length === 0) continue
|
|
461
578
|
if (replayed.title.length > 0 && !titles.has(replayed.sessionId)) titles.set(replayed.sessionId, replayed.title)
|
|
@@ -692,9 +809,9 @@ export async function importLegacyHistory(ledger, sessionsRoot) {
|
|
|
692
809
|
result.scanned += 1
|
|
693
810
|
let replayed
|
|
694
811
|
try {
|
|
695
|
-
replayed = replaySessionRecords(
|
|
812
|
+
replayed = replaySessionRecords(await readSessionRecordsAsync(path), ledger.config, null)
|
|
696
813
|
} catch {
|
|
697
|
-
continue //
|
|
814
|
+
continue // 单文件损坏(含解压预算超限)不阻断整体导入。
|
|
698
815
|
}
|
|
699
816
|
if (replayed.sessionId.length === 0) continue
|
|
700
817
|
const existing = bySession.get(replayed.sessionId)
|
|
@@ -785,9 +902,9 @@ export async function recomputeLedgerPricingBasis(ledger, sessionsRoot) {
|
|
|
785
902
|
if ((result.scanned += 1) % 8 === 0) await new Promise(resolve => setImmediate(resolve))
|
|
786
903
|
let replayed
|
|
787
904
|
try {
|
|
788
|
-
replayed = replaySessionRecords(
|
|
905
|
+
replayed = replaySessionRecords(await readSessionRecordsAsync(path), ledger.config, null)
|
|
789
906
|
} catch {
|
|
790
|
-
continue //
|
|
907
|
+
continue // 单文件损坏(含解压预算超限)不阻断整体重算。
|
|
791
908
|
}
|
|
792
909
|
if (replayed.sessionId.length === 0) continue
|
|
793
910
|
const existing = bySession.get(replayed.sessionId)
|
|
@@ -874,13 +991,14 @@ function subtractBucketInto(target, source) {
|
|
|
874
991
|
export async function repairForkSeed(ledger, sessionsRoot) {
|
|
875
992
|
const result = { sessions: 0, days: 0, scanned: 0 }
|
|
876
993
|
for (const path of listSessionLogs(sessionsRoot)) {
|
|
877
|
-
// 全量同步扫描会长时间占住事件循环:每 8 份让出一次(与回填/导入同策略)
|
|
994
|
+
// 全量同步扫描会长时间占住事件循环:每 8 份让出一次(与回填/导入同策略);
|
|
995
|
+
// 单文件内部亦为流式读取(v1.7.7,issue #88)。
|
|
878
996
|
if ((result.scanned += 1) % 8 === 0) await new Promise(resolve => setImmediate(resolve))
|
|
879
997
|
let records
|
|
880
998
|
try {
|
|
881
|
-
records =
|
|
999
|
+
records = await readSessionRecordsAsync(path)
|
|
882
1000
|
} catch {
|
|
883
|
-
continue //
|
|
1001
|
+
continue // 单文件损坏(含解压预算超限)不阻断整体清洗。
|
|
884
1002
|
}
|
|
885
1003
|
// 只处理 fork 会话(header 带 parentSession);普通会话日志无种子段。
|
|
886
1004
|
const header = records.find(r => r?.type === 'session')
|