dsh-cost-meter 1.7.6 → 1.7.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.en.md CHANGED
@@ -6,7 +6,7 @@
6
6
 
7
7
  Per-conversation cost · daily totals · OpenCode Go subscription quota display · budget with usage percentage · official account balance · custom provider balance · balance progress bar · history · peak/off-peak pricing hours display (peak hours UTC 01:00–04:00, 06:00–10:00; from Aug 23, 2026 weekends are billed at off-peak prices all day, shown as “Weekend — all off-peak”) · pre-switch popup & system-notification alerts for peak/off-peak changes (position / lead time / alert type configurable) · one-click price sync from the official docs · Codex-style token usage heat grid · multi-vendor model pricing (built-in 90+ model price catalog with auto-matching) · mainstream Coding Plan quota queries & display (Anthropic / Z.ai / MiniMax / Kimi / OpenRouter / SiliconFlow / CommandCode / SCNet) plan/API dual-track billing (subscription quota vs pay-as-you-go money separated, per-1% & full-window token/equivalent-cost estimates with daily/weekly/monthly curves) · · quota strip above the input box (budget / Go / coding-plan usage in one row, toggleable)
8
8
 
9
- [![version](https://img.shields.io/badge/version-1.7.6-4176E6)](https://github.com/Han-1413141/dsh-cost-meter)
9
+ [![version](https://img.shields.io/badge/version-1.7.8-4176E6)](https://github.com/Han-1413141/dsh-cost-meter)
10
10
  [![npm](https://img.shields.io/npm/v/dsh-cost-meter?label=npm)](https://www.npmjs.com/package/dsh-cost-meter)
11
11
  [![license](https://img.shields.io/badge/license-MIT-green)](LICENSE)
12
12
  [![dsh](https://img.shields.io/badge/DeepSeek%20Harness-dsh--plugin-4176E6)](https://github.com/deepseek-ai/deepseek-harness)
@@ -87,13 +87,27 @@ For NewApi `GET /api/usage/token` (response `{ "code": 200, "data": { "total_gra
87
87
  - Unlimited-quota tokens (`unlimited_quota: true`) have no `total_available`, so `remaining` cannot be extracted and the query reports “remaining is missing or not numeric” — use a limited-quota token or a middle-layer endpoint that converts the units;
88
88
  - Entry point: Settings → Cost (Quota tab) → “Custom provider balance” → expand config; or write `config.customBalance` in `storages/cost-meter/ledger.json`.
89
89
 
90
- ### Credentials & security (v1.7.6)
90
+ ### Credentials & security (v1.7.8)
91
91
 
92
92
  - **Variable naming**: `{{VAR_NAME}}` follows the `<ROUTE>_API_KEY` convention — `<ROUTE>` is the Provider ID from the DSH Models page (Settings → Models), uppercased with non-alphanumeric characters replaced by underscores, e.g. `openai`→`{{OPENAI_API_KEY}}`, `anthropic`→`{{ANTHROPIC_API_KEY}}`, `abc23-d`→`{{ABC23_D_API_KEY}}`. Sharing a name with the Models page means the balance query and model calls **share the same key** (both resolve from the DSH credential store). This note is also shown above the “Headers (JSON)” input in Settings.
93
93
  - **Credential input fields**: after expanding an entry, the “Credential input” section renders one write-only field per `{{VAR}}` placeholder found in the headers — the key goes straight into the DSH credential store (never written to disk, never echoed back, never stored in `ledger.json`); no need to hand-edit environment variables or credential files.
94
- - **Automatic plaintext migration**: older versions let a literal `Bearer sk-…` in the headers leak into `ledger.json` in plaintext. Since v1.7.6 the plugin imports such keys into the DSH credential store at startup and replaces the header value with a `{{CUSTOM_BALANCE_KEY_…}}` placeholder (derived from the entry's host + header name, stable across restarts) — nothing breaks. From now on `ledger.json` and the config shipped to the browser **never contain plaintext keys**: suspected secret headers (Authorization / X-Api-Key / Bearer / sk- prefixes / long opaque strings) are blanked, while placeholders and ordinary headers pass through.
94
+ - **Automatic plaintext migration**: older versions let a literal `Bearer sk-…` in the headers leak into `ledger.json` in plaintext. Since v1.7.8 the plugin imports such keys into the DSH credential store at startup and replaces the header value with a `{{CUSTOM_BALANCE_KEY_…}}` placeholder (derived from the entry's host + header name, stable across restarts) — nothing breaks. From now on `ledger.json` and the config shipped to the browser **never contain plaintext keys**: suspected secret headers (Authorization / X-Api-Key / Bearer / sk- prefixes / long opaque strings) are blanked, while placeholders and ordinary headers pass through.
95
95
  - **Credential allowlist `allowedHosts`**: when headers carry credentials (placeholders or plaintext), the outbound host must be on this list or the request is refused — protection against leaked keys when importing someone else's config. Without a list, requests proceed with a one-time logged warning. The entry panel provides an “Allowed hosts” input (comma-separated).
96
96
 
97
+ ## CLIProxyAPI Gateway Quotas and WorkBuddy Credits (Issue #87)
98
+
99
+ Connects to a local or LAN-deployed [CLIProxyAPI](https://github.com/router-for-me/CLIProxyAPI) proxy gateway to observe quotas across upstream providers and WorkBuddy plugin credits in one place:
100
+
101
+ - **Native Multi-Provider Discovery & Quotas**: Auto-discovers active credentials via the CPA Management API and queries official native endpoints for Antigravity, Claude, Codex, Kimi, and xAI (Grok). Normalized into standard 5h / 7d / weekly / monthly usage percentage windows with ISO reset timestamps.
102
+ - **WorkBuddy Plugin Credits**: Queries WorkBuddy's read-only management route (`/v0/management/plugins/workbuddy/credits`) to display total, used, and remaining credit balances alongside package lifecycle end-dates, preserving separate account cards.
103
+ - **Zero Metering Side-Effects Guarantee**: Completely read-only — strictly avoids mutating actions such as Codex reset-credit consume, xAI token-consuming chat completion probes, and any WorkBuddy POST routes.
104
+ - **Strict Credential Isolation & Transport Security**:
105
+ - Management Keys are stored strictly write-only in the DSH credential store (`CLIPROXYAPI_MANAGEMENT_KEY_<SOURCE_ID>_<HASH>`), never exposed in config, state, or browser views.
106
+ - Management calls are strictly restricted to 3 fixed paths (`/v0/management/auth-files`, `/v0/management/api-call`, `/v0/management/plugins/workbuddy/credits`). Arbitrary upstream URLs are forbidden.
107
+ - Outbound host whitelist (`allowedHosts`): Non-loopback origins require exact host matches, and plain HTTP requires an explicit `allowInsecureHttp` opt-in.
108
+ - HTTP redirects are strictly forbidden (`redirect: 'manual'`; any 3xx response is rejected) to guard against credential leakage or request hijacking.
109
+ - Privacy and metadata sanitization: Strips raw tokens, cookies, and identity claims from auth-files discovery; emails are masked in public state (`s***@domain`).
110
+
97
111
  ## Bilingual UI
98
112
 
99
113
  The plugin UI (session badge, sidebar balance row & budget box, and the entire Settings page) supports **Simplified Chinese** and **English**:
@@ -238,22 +252,22 @@ Real captures from an actual DSH sidebar of the period strip and collapsed verti
238
252
  dsh plugin --profile web add dsh-cost-meter
239
253
  ```
240
254
 
241
- **PowerShell one-click script** (copy the whole line, paste, press Enter; pnpm is provisioned automatically, git is auto-detected — no clone needed; the install chain is **pinned to the release tag `v1.7.6`** — review the script before running):
255
+ **PowerShell one-click script** (copy the whole line, paste, press Enter; pnpm is provisioned automatically, git is auto-detected — no clone needed; the install chain is **pinned to the release tag `v1.7.8`** — review the script before running):
242
256
 
243
257
  ```powershell
244
- irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.7.6/install.ps1 | iex
258
+ irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.7.8/install.ps1 | iex
245
259
  ```
246
260
 
247
261
  **Or a plain command line** (the machine must already have pnpm and git; also pinned to the tag):
248
262
 
249
263
  ```sh
250
- dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.7.6
264
+ dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.7.8
251
265
  ```
252
266
 
253
267
  Without git, use the GitHub tag archive:
254
268
 
255
269
  ```sh
256
- dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.7.6.tar.gz
270
+ dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.7.8.tar.gz
257
271
  ```
258
272
 
259
273
  After installing, **restart** `dsh web` (plugin rows, the Typert manifest and the client bundle are all scanned at startup):
package/README.md CHANGED
@@ -6,7 +6,7 @@
6
6
 
7
7
  本会话费用 · 当日费用 · OpenCode Go 订阅额度显示 · 预算与已用百分比 · 官方账户余额 · 自定义 Provider 余额查询(可配任意 HTTP 端点) · 余额三段进度条 · 历史记录 · 峰谷计价时段显示(UTC 01:00–04:00、06:00–10:00 为峰时段;2026-08-23 起周末全天按谷价,显示「周末时段——全谷价」) · 峰/谷切换前弹窗与系统通知提醒(位置/提前量/提醒类型可配) · 官方价格一键同步 · 类 Codex Token 用量热图 · 多厂商多模型价格计费(内置 90+ 模型价格目录与自动匹配) · 主流 Coding Plan 额度查询与显示(Anthropic / Z.ai / MiniMax / Kimi / OpenRouter / SiliconFlow / CommandCode / SCNet / 火山方舟 九家,含 Volcano Ark AK/SK 签名) · Plan/API 双轨计费(订阅额度与按量金额分离统计,每 1% 额度与满窗的 token/等值金额估算及日/周/月曲线) · 输入框上方额度横条(预算/Go/Coding Plan 用量一条横排显示,可开关)
8
8
 
9
- [![version](https://img.shields.io/badge/version-1.7.6-4176E6)](https://github.com/Han-1413141/dsh-cost-meter)
9
+ [![version](https://img.shields.io/badge/version-1.7.8-4176E6)](https://github.com/Han-1413141/dsh-cost-meter)
10
10
  [![npm](https://img.shields.io/npm/v/dsh-cost-meter?label=npm)](https://www.npmjs.com/package/dsh-cost-meter)
11
11
  [![license](https://img.shields.io/badge/license-MIT-green)](LICENSE)
12
12
  [![dsh](https://img.shields.io/badge/DeepSeek%20Harness-dsh--plugin-4176E6)](https://github.com/deepseek-ai/deepseek-harness)
@@ -88,13 +88,27 @@
88
88
  - 无限额度 token(`unlimited_quota: true`)没有 `total_available`,无法提取 `remaining`,查询会报「remaining is missing or not numeric」——请改用有限额度 token,或在中间层端点换算;
89
89
  - 配置入口:设置 → 费用(额度标签)→「自定义 Provider 余额」展开配置;或直接改 `storages/cost-meter/ledger.json` 的 `config.customBalance`。
90
90
 
91
- ### 凭据与安全(v1.7.6)
91
+ ### 凭据与安全(v1.7.8)
92
92
 
93
93
  - **变量名命名规则**:`{{VAR_NAME}}` 的参考格式为 `<ROUTE>_API_KEY`——`<ROUTE>` 对应 DSH 模型配置页(设置 → 模型)里的 Provider ID,把 ID 大写、非字母数字字符替换为下划线,例如 `openai`→`{{OPENAI_API_KEY}}`、`anthropic`→`{{ANTHROPIC_API_KEY}}`、`abc23-d`→`{{ABC23_D_API_KEY}}`。与模型页共用同一变量名,自定义余额查询与模型调用即**共用同一把密钥**(都从 DSH 凭据库解析)。该说明也展示在设置页「请求头 (JSON)」输入框上方。
94
94
  - **凭据输入框**:展开条目配置后,「凭据输入」区会为请求头里出现的每个 `{{VAR}}` 占位符显示一行 write-only 输入框,密钥直接存入 DSH 凭据库(不落盘、不回显、不经 `ledger.json`),无需再手改环境变量或凭据文件。
95
- - **明文密钥自动迁移**:旧版本把 `Bearer sk-…` 明文写在请求头里时会明文落盘;v1.7.6 起插件在启动时自动把这类明文导入 DSH 凭据库,并把头值替换为 `{{CUSTOM_BALANCE_KEY_…}}` 占位符(名称由条目 host + 头名派生,跨重启稳定),功能不受影响。此后 `ledger.json` 与下发给浏览器的配置**永不包含明文密钥**——疑似密钥头(Authorization / X-Api-Key / Bearer / sk- 前缀 / 长不透明串)一律置空,占位符与普通头照常保留。
95
+ - **明文密钥自动迁移**:旧版本把 `Bearer sk-…` 明文写在请求头里时会明文落盘;v1.7.8 起插件在启动时自动把这类明文导入 DSH 凭据库,并把头值替换为 `{{CUSTOM_BALANCE_KEY_…}}` 占位符(名称由条目 host + 头名派生,跨重启稳定),功能不受影响。此后 `ledger.json` 与下发给浏览器的配置**永不包含明文密钥**——疑似密钥头(Authorization / X-Api-Key / Bearer / sk- 前缀 / 长不透明串)一律置空,占位符与普通头照常保留。
96
96
  - **凭据白名单 `allowedHosts`**:请求头携带密钥(占位符或明文)时,出站主机必须命中该白名单,否则直接拒绝——用于防止「导入他人配置」导致密钥外带。未配置白名单时放行并在日志警告一次。设置页条目面板内有「凭据白名单主机」输入框(逗号分隔)。
97
97
 
98
+ ## CLIProxyAPI 网关额度与 WorkBuddy 积分 (Issue #87)
99
+
100
+ 支持对接本地或局域网部署的 [CLIProxyAPI](https://github.com/router-for-me/CLIProxyAPI) 代理网关,集中观测代理的多个 Provider 账号额度与 WorkBuddy 插件积分:
101
+
102
+ - **多 Provider 账号原生适配**:自动通过 CPA Management API 发现账号,接入 Antigravity、Claude、Codex、Kimi、xAI (Grok) 官方原生配额/用量接口。统一归一化为 5h / 7d / weekly / monthly 等用量百分比窗口与 ISO 重置时刻。
103
+ - **WorkBuddy 插件积分**:只读接入 WorkBuddy 插件 `/v0/management/plugins/workbuddy/credits` 路由,展示账户总积分、已用/剩余与套餐周期,保留独立账号卡片。
104
+ - **零计费副作用保证**:严禁并杜绝任何产生计费消耗的操作——不调用 Codex 的 reset-credit consume 端点,不向 xAI 发送消耗 token 的 chat probe,不调用 WorkBuddy 的任何变更性 POST 路由。
105
+ - **严格凭据隔离与安全门禁**:
106
+ - Management Key 通过只写 (write-only) 形式托管于 DSH 凭据库 (`CLIPROXYAPI_MANAGEMENT_KEY_<SOURCE_ID>_<HASH>`),永不存入配置文件、账本或浏览器 state。
107
+ - 请求路径严格限制为三条固定管理路径 (`/v0/management/auth-files`、`/v0/management/api-call`、`/v0/management/plugins/workbuddy/credits`),禁止任意 URL 拼接。
108
+ - 严格出站白名单 (`allowedHosts`):非 loopback 来源必须精确命中白名单,非 loopback HTTP 必须显式开启 `allowInsecureHttp`。
109
+ - 强制禁止 HTTP 重定向 (`redirect: 'manual'`, 3xx 状态码坚决拒绝),防止网关重定向劫持或凭据外带。
110
+ - 账号隐私与元数据过滤:auth-files 响应仅提取必需字段,剔除 raw tokens / cookies / ID claims,展示层邮箱自动脱敏掩码 (`s***@domain`)。
111
+
98
112
  ## 双语界面
99
113
 
100
114
  插件界面(会话徽章、侧边栏余额与预算图框、设置页全部文案)支持**简体中文**与**English**:
@@ -240,22 +254,22 @@
240
254
  dsh plugin --profile web add dsh-cost-meter
241
255
  ```
242
256
 
243
- **PowerShell 一键脚本**(复制整行粘贴回车;自动补齐 pnpm、自动探测 git,无需克隆仓库;安装链**固定到发布 tag `v1.7.6`**,建议先下载审阅再运行):
257
+ **PowerShell 一键脚本**(复制整行粘贴回车;自动补齐 pnpm、自动探测 git,无需克隆仓库;安装链**固定到发布 tag `v1.7.8`**,建议先下载审阅再运行):
244
258
 
245
259
  ```powershell
246
- irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.7.6/install.ps1 | iex
260
+ irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.7.8/install.ps1 | iex
247
261
  ```
248
262
 
249
263
  **或直接命令行**(机器上需已有 pnpm 与 git;同样固定到 tag):
250
264
 
251
265
  ```sh
252
- dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.7.6
266
+ dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.7.8
253
267
  ```
254
268
 
255
269
  没有 git 时可用 GitHub tag 打包直链:
256
270
 
257
271
  ```sh
258
- dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.7.6.tar.gz
272
+ dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.7.8.tar.gz
259
273
  ```
260
274
 
261
275
  安装后**重启** `dsh web`(插件行、Typert 清单与客户端 bundle 均在启动时扫描):
package/lib/backfill.js CHANGED
@@ -10,7 +10,8 @@
10
10
  * 避免与实时计费重复计数。会话日志是宿主的只读数据,本模块从不写入。
11
11
  */
12
12
 
13
- import { readdirSync, readFileSync, statSync } from 'node:fs'
13
+ import { readdirSync, statSync } from 'node:fs'
14
+ import { open } from 'node:fs/promises'
14
15
  import { join } from 'node:path'
15
16
  import * as zlib from 'node:zlib'
16
17
  import { costOf, providerPriceEntryFor, usdFromCost, isWrapperProviderId, wrapperUpstreamProvider } from './pricing.js'
@@ -22,6 +23,61 @@ const ZSTD_MAGIC = 4247762216
22
23
  /** 打包行(文本/推理/工具调用增量游程)不含 header 与 usage,回放时跳过。 */
23
24
  const PACKED_ROW_TYPES = new Set(['text-chunks', 'reasoning-chunks', 'tool-call-chunks'])
24
25
 
26
+ // ── 内存与事件循环守卫(v1.7.7,issue #88) ───────────────────────────────
27
+ //
28
+ // 用户实测:一份 168MB(压缩后)的 session.jsonl.zstd 在启动回填时把 Electron
29
+ // 主进程堆耗尽(STATUS_BREAKPOINT 崩溃),且同步解压全程占住事件循环(agent
30
+ // 界面整体卡死)。旧缓解(逐帧解压代替全文拼接)只压住了解压峰值,仍有两个
31
+ // 洞:records 数组物化全部解压后事件(堆)与巨长打包行的 JSON.parse(瞬时峰值)。
32
+ // 现三层防御:
33
+ // · 打包行探针:超 4096 字节的行先看头部 512 字符,命中打包行模式直接跳过,
34
+ // 不再付出 JSON.parse 的整行解析峰值(探针只做加速,短行/未命中仍走解析+过滤,
35
+ // 行为不变);
36
+ // · 流式迭代器 iterateSessionRecords:分块读文件 + 增量扫描帧边界,逐帧解压
37
+ // 逐行产出记录,任一时刻只持有单帧解压结果;配合定期 setImmediate 让出,
38
+ // 回填期间宿主 UI 不再被卡死;
39
+ // · 解压预算:单文件累计解压字节数超上限(默认 4GB,防解压炸弹)抛错,
40
+ // 调用方按「单文件损坏」跳过该文件,绝不把主进程拖崩。
41
+
42
+ /** 打包行探针:巨长行头部出现 "type":"text-chunks" 等即判打包行,免整行解析。 */
43
+ const PACKED_ROW_PROBE = /"type"\s*:\s*"(?:text|reasoning|tool-call)-chunks"/
44
+ const PACKED_ROW_PROBE_HEAD = 512
45
+ const PACKED_ROW_PROBE_MIN_LINE = 4096
46
+ /** 单文件解压累计上限(防解压炸弹;正常会话日志解压后数百 MB~2GB)。 */
47
+ const MAX_DECOMPRESSED_PER_FILE = 4 * 1024 * 1024 * 1024
48
+ /** 流式读文件的分块大小。 */
49
+ const READ_CHUNK_BYTES = 8 * 1024 * 1024
50
+ /** 每 N 帧解压/每 N 条事件让出一次事件循环(保持宿主 UI 响应)。 */
51
+ const YIELD_EVERY_FRAMES = 64
52
+ const YIELD_EVERY_EVENTS = 4096
53
+
54
+ /** 巨长行探针判定(仅加速路径,未命中仍走 JSON.parse + 类型过滤)。 */
55
+ function lineLooksPacked(line) {
56
+ if (line.length <= PACKED_ROW_PROBE_MIN_LINE) return false
57
+ return PACKED_ROW_PROBE.test(line.slice(0, PACKED_ROW_PROBE_HEAD))
58
+ }
59
+
60
+ /**
61
+ * 解析单行日志为事件对象;空行、打包行与坏行返回 null。
62
+ * 短行走「解析后按类型丢弃」;巨长行先探针,命中打包行直接跳过(免解析峰值)。
63
+ * @param {string} line
64
+ * @returns {object | null}
65
+ */
66
+ export function parseRecordLine(line) {
67
+ if (line.length === 0) return null
68
+ if (lineLooksPacked(line)) return null
69
+ try {
70
+ const rec = JSON.parse(line)
71
+ // 打包行(text/reasoning/tool-call 增量游程)在解析入口就丢弃:它们是
72
+ // 大日志的体积主体,回放阶段反正会跳过,提前过滤才能兑现 OOM 缓解。
73
+ if (rec !== null && typeof rec === 'object' && !PACKED_ROW_TYPES.has(rec.type)) return rec
74
+ return null
75
+ } catch {
76
+ // 坏行跳过:回放是尽力而为,不让单行损坏阻断整个会话。
77
+ return null
78
+ }
79
+ }
80
+
25
81
  /**
26
82
  * 结构化扫描拼接的 Zstandard frame 边界(不解压块内容),与宿主
27
83
  * dsh-session-persistence-jsonl 的容器格式一致:每个追加批次一个独立
@@ -73,57 +129,117 @@ export function scanZstdFrames(buffer) {
73
129
  }
74
130
 
75
131
  /**
76
- * 读取一份会话日志的全部事件行(zstd frame 解压;明文直接按行)。
132
+ * 流式逐行读取一份会话日志(异步生成器,v1.7.7 issue #88 的核心原语)。
133
+ *
134
+ * zstd 容器按帧边界增量扫描:分块读入原始字节(READ_CHUNK_BYTES),已确认完整的
135
+ * 帧立即解压并逐行产出、随后释放引用——任一时刻只持有「单帧解压结果 + 尾部
136
+ * 跨块/跨帧的行缓冲 + 已积累的记录数组」,不再物化全文。巨长打包行经探针
137
+ * 跳过(免 JSON.parse 峰值)。每 YIELD_EVERY_FRAMES 帧让出一次事件循环。
77
138
  *
78
- * 内存约束:宿主会话日志可含数万帧(每次追加批次一个独立 zstd frame,
79
- * 长期会话可达 10 万+ 帧)且解压后数百 MB。旧实现把所有帧一次性解压、
80
- * Buffer.concat 拼成全文再 split——峰值内存可达数 GB,4GB 堆限制下
81
- * 会 OOM(实测一份 50MB / 11.5 万帧日志即 ~3GB)。现改为逐帧解压、
82
- * 逐帧按行切片解析:任一时刻只保留单帧解压结果(通常 ≤ 数 MB),行缓冲
83
- * 跨帧拼接兜底半行(追加批次以换行结尾,实际不会出现)。
139
+ * 单文件累计解压字节数超过 maxDecompressed(默认 4GB,防解压炸弹)抛
140
+ * RangeError,调用方按单文件损坏处理。
84
141
  * @param path - session.jsonl.zstd 或 session.jsonl 路径。
85
- * @returns 逐行 JSON.parse 后的记录数组(坏行跳过)
142
+ * @param options - { maxDecompressed?: number, yieldEvery?: number }
143
+ * @yields {object} 逐行解析出的记录(空行/打包行/坏行已过滤)。
86
144
  */
87
- export function readSessionRecords(path) {
88
- const buffer = readFileSync(path)
89
- const records = []
90
- let pending = '' // 跨帧行缓冲:上一帧末尾未换行的残行拼接进下一帧首行。
91
- const consumeText = (text) => {
92
- if (text.length === 0) return
93
- const lines = text.split('\n')
94
- lines[0] = pending + lines[0]
95
- pending = lines[lines.length - 1]
96
- for (let i = 0; i < lines.length - 1; i++) {
97
- const line = lines[i]
98
- if (line.length === 0) continue
99
- try {
100
- const rec = JSON.parse(line)
101
- // 打包行(text/reasoning/tool-call 增量游程)在解析入口就丢弃:它们是
102
- // 大日志的体积主体,回放阶段反正会跳过,提前过滤才能兑现 OOM 缓解。
103
- if (rec !== null && typeof rec === 'object' && !PACKED_ROW_TYPES.has(rec.type)) records.push(rec)
104
- } catch {
105
- // 坏行跳过:回放是尽力而为,不让单行损坏阻断整个会话。
145
+ export async function* iterateSessionRecords(path, options = {}) {
146
+ const maxDecompressed = Number(options.maxDecompressed) > 0 ? Number(options.maxDecompressed) : MAX_DECOMPRESSED_PER_FILE
147
+ const yieldEvery = Number(options.yieldEvery) > 0 ? Number(options.yieldEvery) : YIELD_EVERY_FRAMES
148
+ const isZstd = path.endsWith('.zstd')
149
+ if (isZstd && typeof zlib.zstdDecompressSync !== 'function') return
150
+ const handle = await open(path, 'r')
151
+ try {
152
+ let pending = '' // 跨块/跨帧行缓冲:上一段末尾未换行的残行拼接进下一段首行。
153
+ let tail = Buffer.alloc(0) // 压缩字节尾部缓冲:块边界落在帧中间时拼接后继续扫。
154
+ let decompressedTotal = 0
155
+ let framesSinceYield = 0
156
+ const consumeText = (text, out) => {
157
+ if (text.length === 0) return
158
+ const lines = text.split('\n')
159
+ lines[0] = pending + lines[0]
160
+ pending = lines[lines.length - 1]
161
+ for (let i = 0; i < lines.length - 1; i++) {
162
+ const rec = parseRecordLine(lines[i])
163
+ if (rec !== null) out.push(rec)
106
164
  }
107
165
  }
108
- }
109
- if (path.endsWith('.zstd')) {
110
- if (typeof zlib.zstdDecompressSync !== 'function') return []
111
- const frames = scanZstdFrames(buffer)
112
- for (const f of frames) {
113
- consumeText(zlib.zstdDecompressSync(buffer.subarray(f.start, f.end)).toString('utf8'))
114
- }
115
- } else {
116
- consumeText(buffer.toString('utf8'))
117
- }
118
- if (pending.length > 0) {
119
- try {
120
- const rec = JSON.parse(pending)
121
- // 与上方 consumeText 同规则:打包行不入记录(文件末尾无换行的残行兜底)
122
- if (rec !== null && typeof rec === 'object' && !PACKED_ROW_TYPES.has(rec.type)) records.push(rec)
123
- } catch {
124
- // 文件末尾无换行的残行:同上,尽力而为。
166
+ const records = []
167
+ // 结构损坏标志:一旦出现「一个完整分块里连一个可解帧都没有」,按旧版
168
+ // 全量扫描的语义(在首个非法结构处停止)忽略文件后续内容——否则坏字节
169
+ // 会在 tail 里无限拼接,且行为与旧版不一致。
170
+ let stopped = false
171
+ for (;;) {
172
+ const chunk = await handle.read(Buffer.allocUnsafe(READ_CHUNK_BYTES), 0, READ_CHUNK_BYTES, null)
173
+ if (chunk.bytesRead === 0) break
174
+ const data = tail.length > 0 ? Buffer.concat([tail, chunk.buffer.subarray(0, chunk.bytesRead)]) : chunk.buffer.subarray(0, chunk.bytesRead)
175
+ tail = Buffer.alloc(0)
176
+ if (stopped) continue
177
+ if (!isZstd) {
178
+ consumeText(data.toString('utf8'), records)
179
+ if (records.length >= YIELD_EVERY_EVENTS) {
180
+ yield* drain(records)
181
+ await new Promise(resolve => setImmediate(resolve))
182
+ }
183
+ continue
184
+ }
185
+ // 增量消费完整帧:从 data 头部起逐帧取边界(帧区间相对 subarray 起点),
186
+ // 解压→逐行→释放引用;剩余不完整帧(块边界截断)留在 tail 等下一块。
187
+ let offset = 0
188
+ while (offset < data.length) {
189
+ const frames = scanZstdFrames(data.subarray(offset))
190
+ if (frames.length === 0) {
191
+ // 剩余字节连不出完整帧:未满一个分块 = 尾部截断,留 tail 等下一块;
192
+ // 已满一个分块仍无帧 = 结构损坏(与旧版全量扫描停在非法结构同语义),
193
+ // 标记停止,忽略文件后续。
194
+ if (data.length - offset >= READ_CHUNK_BYTES) stopped = true
195
+ else tail = data.subarray(offset)
196
+ break
197
+ }
198
+ for (const f of frames) {
199
+ decompressedTotal += f.end - f.start
200
+ if (decompressedTotal > maxDecompressed) {
201
+ throw new RangeError(`session log decompress budget exceeded (${decompressedTotal} > ${maxDecompressed} bytes): ${path}`)
202
+ }
203
+ const plain = zlib.zstdDecompressSync(data.subarray(offset + f.start, offset + f.end))
204
+ consumeText(plain.toString('utf8'), records)
205
+ framesSinceYield += 1
206
+ }
207
+ offset += frames[frames.length - 1].end
208
+ }
209
+ if (framesSinceYield >= yieldEvery || records.length >= YIELD_EVERY_EVENTS) {
210
+ yield* drain(records)
211
+ framesSinceYield = 0
212
+ await new Promise(resolve => setImmediate(resolve))
213
+ }
214
+ }
215
+ // 文件读完:tail 里剩的残帧(崩溃截断)按既有语义忽略;行缓冲兜底末行。
216
+ if (pending.length > 0) {
217
+ const rec = parseRecordLine(pending)
218
+ if (rec !== null) records.push(rec)
125
219
  }
220
+ yield* drain(records)
221
+ } finally {
222
+ await handle.close()
126
223
  }
224
+ }
225
+
226
+ /** 把 records 数组逐条 yield 出去并清空(内部助手)。 */
227
+ function* drain(records) {
228
+ for (const rec of records) yield rec
229
+ records.length = 0
230
+ }
231
+
232
+ /**
233
+ * 读取一份会话日志的全部事件行(zstd 逐 frame 解压;明文直接按行)。
234
+ * 同步兼容包装:逐行收集成数组。新的回填路径应改用流式 iterateSessionRecords
235
+ * (issue #88:records 数组本身在大日志下仍是堆压力,收集形态只留给需要数组
236
+ * 随机访问的调用方)。
237
+ * @param path - session.jsonl.zstd 或 session.jsonl 路径。
238
+ * @returns 逐行 JSON.parse 后的记录数组(坏行跳过)。
239
+ */
240
+ export async function readSessionRecordsAsync(path, options = {}) {
241
+ const records = []
242
+ for await (const rec of iterateSessionRecords(path, options)) records.push(rec)
127
243
  return records
128
244
  }
129
245
 
@@ -448,14 +564,15 @@ export async function backfillLegacyLedger(ledger, sessionsRoot) {
448
564
  }
449
565
  }
450
566
  for (const path of listSessionLogs(sessionsRoot, onlySessionIds)) {
451
- // 会话日志多时逐份解压会长时间占住事件循环:每 8 份让出一次,不卡宿主 UI
567
+ // 会话日志多时逐份解压会长时间占住事件循环:每 8 份让出一次,不卡宿主 UI;
568
+ // 单文件内部亦为流式读取(v1.7.7,issue #88:解压期间周期性让出,大日志不再卡死宿主)。
452
569
  if ((scannedCount += 1) % 8 === 0) await new Promise(resolve => setImmediate(resolve))
453
570
  result.scanned += 1
454
571
  let replayed
455
572
  try {
456
- replayed = replaySessionRecords(readSessionRecords(path), ledger.config, needDates.size > 0 ? needDates : new Set(['-']))
573
+ replayed = replaySessionRecords(await readSessionRecordsAsync(path), ledger.config, needDates.size > 0 ? needDates : new Set(['-']))
457
574
  } catch {
458
- continue // 单文件损坏不阻断整体回填。
575
+ continue // 单文件损坏(含解压预算超限)不阻断整体回填。
459
576
  }
460
577
  if (replayed.sessionId.length === 0) continue
461
578
  if (replayed.title.length > 0 && !titles.has(replayed.sessionId)) titles.set(replayed.sessionId, replayed.title)
@@ -692,9 +809,9 @@ export async function importLegacyHistory(ledger, sessionsRoot) {
692
809
  result.scanned += 1
693
810
  let replayed
694
811
  try {
695
- replayed = replaySessionRecords(readSessionRecords(path), ledger.config, null)
812
+ replayed = replaySessionRecords(await readSessionRecordsAsync(path), ledger.config, null)
696
813
  } catch {
697
- continue // 单文件损坏不阻断整体导入。
814
+ continue // 单文件损坏(含解压预算超限)不阻断整体导入。
698
815
  }
699
816
  if (replayed.sessionId.length === 0) continue
700
817
  const existing = bySession.get(replayed.sessionId)
@@ -785,9 +902,9 @@ export async function recomputeLedgerPricingBasis(ledger, sessionsRoot) {
785
902
  if ((result.scanned += 1) % 8 === 0) await new Promise(resolve => setImmediate(resolve))
786
903
  let replayed
787
904
  try {
788
- replayed = replaySessionRecords(readSessionRecords(path), ledger.config, null)
905
+ replayed = replaySessionRecords(await readSessionRecordsAsync(path), ledger.config, null)
789
906
  } catch {
790
- continue // 单文件损坏不阻断整体重算。
907
+ continue // 单文件损坏(含解压预算超限)不阻断整体重算。
791
908
  }
792
909
  if (replayed.sessionId.length === 0) continue
793
910
  const existing = bySession.get(replayed.sessionId)
@@ -874,13 +991,14 @@ function subtractBucketInto(target, source) {
874
991
  export async function repairForkSeed(ledger, sessionsRoot) {
875
992
  const result = { sessions: 0, days: 0, scanned: 0 }
876
993
  for (const path of listSessionLogs(sessionsRoot)) {
877
- // 全量同步扫描会长时间占住事件循环:每 8 份让出一次(与回填/导入同策略)
994
+ // 全量同步扫描会长时间占住事件循环:每 8 份让出一次(与回填/导入同策略);
995
+ // 单文件内部亦为流式读取(v1.7.7,issue #88)。
878
996
  if ((result.scanned += 1) % 8 === 0) await new Promise(resolve => setImmediate(resolve))
879
997
  let records
880
998
  try {
881
- records = readSessionRecords(path)
999
+ records = await readSessionRecordsAsync(path)
882
1000
  } catch {
883
- continue // 单文件损坏不阻断整体清洗。
1001
+ continue // 单文件损坏(含解压预算超限)不阻断整体清洗。
884
1002
  }
885
1003
  // 只处理 fork 会话(header 带 parentSession);普通会话日志无种子段。
886
1004
  const header = records.find(r => r?.type === 'session')