dsh-cost-meter 1.9.33 → 1.9.34

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -6,7 +6,7 @@
6
6
 
7
7
  **Session cost tracking plugin for the DeepSeek Harness web GUI (bilingual UI)**
8
8
 
9
- [![version](https://img.shields.io/badge/version-1.9.33-4176E6)](https://github.com/Han-1413141/dsh-cost-meter)
9
+ [![version](https://img.shields.io/badge/version-1.9.34-4176E6)](https://github.com/Han-1413141/dsh-cost-meter)
10
10
  [![npm](https://img.shields.io/npm/v/dsh-cost-meter?label=npm)](https://www.npmjs.com/package/dsh-cost-meter)
11
11
  [![dshfind](https://dshfind.com/api/badge/Han-1413141/dsh-cost-meter?metric=downloads&lang=en)](https://dshfind.com/zh/plugins/Han-1413141/dsh-cost-meter?ref=badge)
12
12
  [![license](https://img.shields.io/badge/license-MIT-green)](LICENSE)
@@ -16,11 +16,11 @@
16
16
 
17
17
  </div>
18
18
 
19
- **v1.9.33** adds full settings to the plugin details page and fixes concurrent saves, CNY conversion and frozen currency totals. The future read-only settings page is prepared but remains disabled.
19
+ **v1.9.34** fixes billion-context idle scanning stalls and adds storage statistics, preview/confirm cleanup, 7/30-day retention and a CLI. Completed streams are compressed while billing history is retained. [Storage guide](docs/billing-storage.md).
20
20
 
21
21
  **Host floor: DSH 0.2.0** · Adaptation target: **0.2.1-rc.2** (npm `next`). Expand the compatibility notes below for exact declarations, host records and troubleshooting.
22
22
 
23
- Release notes: [v1.9.33](docs/update-notes-1.9.33.md#english). Fully restart DSH after updating.
23
+ Release notes: [v1.9.34](docs/update-notes-1.9.34.md#english). Fully restart DSH after updating.
24
24
 
25
25
  ![Promo art](docs/promo.en.png)
26
26
 
@@ -352,22 +352,22 @@ On Node.js 20, use `npm install -g pnpm@10` instead. See [pnpm installation and
352
352
  dsh plugin --profile web add dsh-cost-meter
353
353
  ```
354
354
 
355
- **PowerShell one-click script** (copy the whole line, paste, press Enter; pnpm is provisioned automatically, git is auto-detected — no clone needed; the install chain is **pinned to the release tag `v1.9.33`** — review the script before running):
355
+ **PowerShell one-click script** (copy the whole line, paste, press Enter; pnpm is provisioned automatically, git is auto-detected — no clone needed; the install chain is **pinned to the release tag `v1.9.34`** — review the script before running):
356
356
 
357
357
  ```powershell
358
- irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.9.33/install.ps1 | iex
358
+ irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.9.34/install.ps1 | iex
359
359
  ```
360
360
 
361
361
  **Or a plain command line** (the machine must already have pnpm and git; also pinned to the tag):
362
362
 
363
363
  ```sh
364
- dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.9.33
364
+ dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.9.34
365
365
  ```
366
366
 
367
367
  Without git, use the GitHub tag archive:
368
368
 
369
369
  ```sh
370
- dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.9.33.tar.gz
370
+ dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.9.34.tar.gz
371
371
  ```
372
372
 
373
373
  After installing, **restart** `dsh web` (plugin rows, the Typert manifest and the client bundle are all scanned at startup):
@@ -432,7 +432,7 @@ Workarounds:
432
432
 
433
433
  ```yaml
434
434
  minimumReleaseAgeExclude:
435
- - dsh-cost-meter@1.9.33
435
+ - dsh-cost-meter@1.9.34
436
436
  ```
437
437
 
438
438
  3. For a one-off CLI operation, forward the pnpm flag: `dsh plugin --profile desktop remove dsh-cost-meter --config.minimum-release-age=0` (or add `--trust-lockfile` when only lockfile verification fails). Both flags were verified against pnpm 11.28.5 in a temporary profile.
package/README.zh-CN.md CHANGED
@@ -6,7 +6,7 @@
6
6
 
7
7
  **DeepSeek Harness 会话费用统计插件(界面中英双语)**
8
8
 
9
- [![version](https://img.shields.io/badge/version-1.9.33-4176E6)](https://github.com/Han-1413141/dsh-cost-meter)
9
+ [![version](https://img.shields.io/badge/version-1.9.34-4176E6)](https://github.com/Han-1413141/dsh-cost-meter)
10
10
  [![npm](https://img.shields.io/npm/v/dsh-cost-meter?label=npm)](https://www.npmjs.com/package/dsh-cost-meter)
11
11
  [![dshfind](https://dshfind.com/api/badge/Han-1413141/dsh-cost-meter?metric=downloads&lang=zh)](https://dshfind.com/zh/plugins/Han-1413141/dsh-cost-meter?ref=badge)
12
12
  [![license](https://img.shields.io/badge/license-MIT-green)](LICENSE)
@@ -16,11 +16,11 @@
16
16
 
17
17
  </div>
18
18
 
19
- **v1.9.33** 插件详情页新增完整费用设置,修复并发保存、人民币折算和历史币种金额同步。实时费用与额度展示页已备好,暂不启用。
19
+ **v1.9.34** 修复 billion-context 空闲扫描卡顿,新增计费历史存储统计、清理预览与确认、7/30 天保留和 CLI;压缩旧流并保留费用明细。[存储管理说明](docs/billing-storage.md)。
20
20
 
21
21
  **宿主要求:DSH 0.2.0 起支持** · 适配目标 **0.2.1-rc.2**(npm `next`)。完整声明、历史记录与安装排障见下方展开项。
22
22
 
23
- 更新说明:[v1.9.33](docs/update-notes-1.9.33.md#中文)。更新后完全重启 DSH。
23
+ 更新说明:[v1.9.34](docs/update-notes-1.9.34.md#中文)。更新后完全重启 DSH。
24
24
 
25
25
  ![宣传图](docs/promo.png)
26
26
 
@@ -352,22 +352,22 @@ Node.js 20 请改用 `npm install -g pnpm@10`。版本要求见 [pnpm 官方安
352
352
  dsh plugin --profile web add dsh-cost-meter
353
353
  ```
354
354
 
355
- **PowerShell 一键脚本**(复制整行粘贴回车;自动补齐 pnpm、自动探测 git,无需克隆仓库;安装链**固定到发布 tag `v1.9.33`**,建议先下载审阅再运行):
355
+ **PowerShell 一键脚本**(复制整行粘贴回车;自动补齐 pnpm、自动探测 git,无需克隆仓库;安装链**固定到发布 tag `v1.9.34`**,建议先下载审阅再运行):
356
356
 
357
357
  ```powershell
358
- irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.9.33/install.ps1 | iex
358
+ irm https://raw.githubusercontent.com/Han-1413141/dsh-cost-meter/v1.9.34/install.ps1 | iex
359
359
  ```
360
360
 
361
361
  **或直接命令行**(机器上需已有 pnpm 与 git;同样固定到 tag):
362
362
 
363
363
  ```sh
364
- dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.9.33
364
+ dsh plugin --profile web add github:Han-1413141/dsh-cost-meter#v1.9.34
365
365
  ```
366
366
 
367
367
  没有 git 时可用 GitHub tag 打包直链:
368
368
 
369
369
  ```sh
370
- dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.9.33.tar.gz
370
+ dsh plugin --profile web add https://github.com/Han-1413141/dsh-cost-meter/archive/refs/tags/v1.9.34.tar.gz
371
371
  ```
372
372
 
373
373
  安装后**重启** `dsh web`(插件行、Typert 清单与客户端 bundle 均在启动时扫描):
@@ -432,7 +432,7 @@ DSH **0.2.1-rc.1 / 0.2.1-rc.2** 宿主自带 pnpm **11.28.5**,默认启用内
432
432
 
433
433
  ```yaml
434
434
  minimumReleaseAgeExclude:
435
- - dsh-cost-meter@1.9.33
435
+ - dsh-cost-meter@1.9.34
436
436
  ```
437
437
 
438
438
  3. 仅本次操作用 CLI 透传 pnpm 参数:`dsh plugin --profile desktop remove dsh-cost-meter --config.minimum-release-age=0`;若只是 lockfile 校验阶段失败,可改用 `--trust-lockfile`。这两个参数已在本机 pnpm 11.28.5 临时 Profile 上实测可用。
@@ -0,0 +1,44 @@
1
+ # 计费历史存储 / Billing history storage
2
+
3
+ ## 中文
4
+
5
+ 从 1.9.34 起,费用设置的「用量 → 计费历史存储」显示 billion-context 辅助文件的数量、大小,以及最早、最新修改时间。左下角设置和插件详情页共用此入口。
6
+
7
+ 默认保留最近 **7 天**的原始辅助文件;可以选择 **30 天**,或「全部保留」关闭自动清理。后台每小时检查一次,每次最多处理 10,000 个符合条件的文件。正在使用、尚未完成、内容无效或缺少完成依据的记录会保留。
8
+
9
+ - `streams`:将超过保留时间、已完成且已有计费归属的记录合并为 gzip 归档。每批最多 256 条,写入并同步归档后才移除未被并发修改的原始文件。
10
+ - `groups`:删除超过保留时间且已完成、没有待处理请求的预留记录与状态回执。
11
+ - `stream-archives`:保留压缩后的计费流元数据,供历史查询、重启和重放使用。
12
+ - 账本、`calls` 调用记录、费用金额、调用次数和历史明细保留。
13
+
14
+ 手动操作先点击「预览清理」,检查待归档条数、回执文件数、涉及会话数和预计释放空间,再点击「确认清理」。取消预览不会执行清理。后台或其他客户端可能在预览后处理文件,因此最终数量以执行结果为准。
15
+
16
+ ### CLI
17
+
18
+ 插件安装提供 `dsh-cost-meter` 命令;也可在安装目录使用 `node lib/billing-storage-cli.js`。默认只预览,只有 `--apply` 执行清理。
19
+
20
+ ```sh
21
+ dsh-cost-meter clean --streams --older-than 7d --dry-run
22
+ dsh-cost-meter clean --older-than 30d --dry-run
23
+ dsh-cost-meter clean --older-than 30d --apply
24
+ dsh-cost-meter clean --groups --older-than 7d --apply
25
+ dsh-cost-meter clean --ledger /path/to/ledger.json --older-than 7d --dry-run
26
+ ```
27
+
28
+ 不指定 `--streams` / `--groups` 时同时处理两者。`--older-than all` 表示全部保留,不清理任何记录。CLI 参数只作用于本次命令,不修改设置中的自动保留时间。默认账本为 `$DSH_HOME/storages/cost-meter/ledger.json`,未设置 `DSH_HOME` 时使用用户目录下的 `.dsh`;使用其他账本时显式传入 `--ledger`。
29
+
30
+ ### 归档与备份
31
+
32
+ 备份时将账本和同级的 `<ledger>.billion-context` 整个目录一起保存。压缩归档由 1.9.34 及后续版本读取;降级到更早版本会缺少已归档流的历史明细。需要降级时使用清理前的完整备份。尚未归档且计划降级时,可先选「全部保留」关闭自动清理。
33
+
34
+ 空闲计费扫描异步检查会话目录的修改时间。目录未变化时跳过已完成的 stream 文件;正在写入的调用日志和未完成流仍会继续跟进。存储统计仅在打开此面板、刷新或执行清理时加载,不加入每 3 秒的界面状态刷新。
35
+
36
+ ## English
37
+
38
+ Open **Cost settings → Usage → Billing history storage** from either Settings or the plugin details page. The panel shows file counts, bytes, and oldest/newest modification times for streams, request receipts and compressed archives.
39
+
40
+ Original auxiliary files are retained for **7 days** by default. Choose **30 days** or **Keep all** to disable cleanup. Hourly maintenance processes up to 10,000 eligible files per run. Active, unfinished, invalid or unconfirmed records are retained. Completed streams with resolved billing coverage are compressed in batches of up to 256 records; the archive is written and synced before unchanged originals are removed. Expired, completed request receipts are removed. Ledger totals, calls, historical details and replay metadata remain available.
41
+
42
+ Manual cleanup requires **Preview cleanup → Confirm cleanup**. Canceling performs no cleanup. The CLI examples above use the same operation: preview is the default, `--apply` executes it, and `--streams` / `--groups` select categories (both by default). `--older-than all` retains everything. CLI options do not change the automatic-retention setting. Use `--ledger` for a custom ledger; the default is `$DSH_HOME/storages/cost-meter/ledger.json`, with `~/.dsh` as the fallback home.
43
+
44
+ Back up both the ledger and its adjacent `.billion-context` directory. Versions older than 1.9.34 cannot read these new stream archives; restore a complete pre-cleanup backup when downgrading. Idle billing scans use asynchronous directory revisions and skip completed streams in unchanged directories. The storage panel is loaded on demand.
@@ -42,4 +42,4 @@ This patch adds the complete cost settings to plugin details, sharing the compon
42
42
  - Use the CNY conversion rate for CNY-priced external calls and legacy pricing repair, regardless of the display currency.
43
43
  - Update frozen CNY/EUR amounts alongside repaired USD costs, retaining recorded FX rates and unrelated amounts. Reconcile complete model currency totals even when USD totals already match; missing legacy currency fields do not imply zero.
44
44
 
45
- No new full-ledger repricing task is introduced. Tests cover currencies, idempotence, partial legacy rows, concurrent/failed saves, slot lifecycle and responsive read-only rendering with synthetic data. Update to **1.9.31** and restart DSH.
45
+ No new full-ledger repricing task is introduced. Tests cover currencies, idempotence, partial legacy rows, concurrent/failed saves, slot lifecycle and responsive read-only rendering with synthetic data. Update to **1.9.31** and restart DSH.
@@ -0,0 +1,35 @@
1
+ # v1.9.34 更新说明
2
+
3
+ ## 中文
4
+
5
+ 修复 billion-context 长时间运行后,空闲计费扫描周期性阻塞主线程的问题。原先每轮扫描都会同步查询每个 stream 文件的状态;现在异步检查会话目录变化,只重新读取变化目录中的计费流,同时继续跟进未完成流和追加写入的调用日志。
6
+
7
+ 新增「费用设置 → 用量 → 计费历史存储」,显示文件数量、大小、最早与最新修改时间,支持清理预览、确认和取消。原始辅助文件默认保留 7 天,也可改为 30 天或全部保留;自动维护每小时检查一次。
8
+
9
+ 旧的已完成流先合并压缩,再移除未被修改的原文件;过期且完成的请求回执可删除。保留账本、调用记录、费用、次数和历史明细,并覆盖清理中断、并发修改、重新导入和重复执行。新增等价 CLI,默认预览,使用 `--apply` 才执行。[操作与备份说明](billing-storage.md)。
10
+
11
+ ### 性能复现
12
+
13
+ 本地合成数据包含 **10,744 个 stream 文件、38 个会话目录**。为模拟 issue 中文件系统过滤驱动造成的开销,对旧实现的每次同步文件状态查询增加 **0.25 毫秒**,触发其 30 秒强制重扫。
14
+
15
+ | 指标 | 1.9.33 | 1.9.34 |
16
+ | --- | ---: | ---: |
17
+ | 空闲扫描耗时 | 2,918.61 ms | 3.21 ms |
18
+ | 扫描期间最大事件循环间隔 | 2,920.07 ms | 15.32 ms |
19
+ | 已完成 stream 文件同步状态查询 | 10,744 次 | 0 次 |
20
+
21
+ 这是受控复现数据,不是报告者电脑的实测。回归测试同时确认:原子替换旧文件后仍可发现新用量,未变化目录不逐文件查询,损坏元数据不会阻塞其他流。浏览器验证使用合成数据,覆盖预览、确认、取消、保留时间保存、错误反馈、中英文和 360/560/960 像素宽度。
22
+
23
+ 本地完整验证通过 314 组检查;另通过真实 DSH Registry/Gateway 的接口调用、参数校验和保留时间回显检查。前端主包为 259,018 字节,存储面板独立按需加载。
24
+
25
+ 感谢 **[@ZiYuan258](https://github.com/ZiYuan258)** 在 [#397](https://github.com/Han-1413141/dsh-cost-meter/issues/397) 提供问题报告、文件规模与主线程耗时证据、同步扫描定位,以及异步处理、清理入口和保留时间建议。
26
+
27
+ ## English
28
+
29
+ Fix periodic main-thread stalls during idle billion-context billing scans. Replace per-file synchronous scanning with asynchronous session-directory revisions, while continuing to follow unfinished streams and append-only call journals.
30
+
31
+ Add on-demand billing storage statistics and a preview/confirm cleanup flow. Retain original auxiliary files for 7 days by default, with 30-day and keep-all options. Hourly maintenance durably archives completed stream metadata before removing unchanged originals and removes expired completed request receipts. Preserve ledger totals, call counts, historical details and replay data. The new CLI previews by default and requires `--apply` to execute; see the [storage guide](billing-storage.md).
32
+
33
+ A controlled fixture with 10,744 streams in 38 sessions and a simulated 0.25 ms synchronous-stat cost reduced idle scan time from 2,918.61 ms to 3.21 ms. This is a synthetic benchmark, not a measurement on the reporter's machine. Regression checks cover replay, interruption, concurrent writes, repeated cleanup and invalid metadata; browser checks use synthetic data.
34
+
35
+ Credit **@ZiYuan258** for the report, reproduction evidence, synchronous-scan diagnosis and cleanup/retention proposals in [#397](https://github.com/Han-1413141/dsh-cost-meter/issues/397).
@@ -0,0 +1,25 @@
1
+ #!/usr/bin/env node
2
+ import { parseArgs } from 'node:util'
3
+ import { resolve, join } from 'node:path'
4
+ import { homedir } from 'node:os'
5
+ import { manageBillionStorage } from './billion-context-storage.js'
6
+
7
+ try {
8
+ const { values, positionals } = parseArgs({ allowPositionals: true, options: {
9
+ ledger: { type: 'string' }, 'older-than': { type: 'string', default: '7d' },
10
+ streams: { type: 'boolean' }, groups: { type: 'boolean' }, 'dry-run': { type: 'boolean' },
11
+ apply: { type: 'boolean', default: false }, help: { type: 'boolean', short: 'h' },
12
+ } })
13
+ if (values.help || !positionals.length) {
14
+ console.log('dsh-cost-meter clean [--ledger <ledger.json>] [--streams] [--groups] [--older-than 7d|30d|all] [--dry-run | --apply]\n默认预览;--apply 才执行。合并压缩已完成的旧计费流,清理过期请求回执,保留账本、调用记录和历史费用明细。\nPreview by default. --apply compacts completed streams and removes expired receipts; ledger, calls and historical cost details are preserved.')
15
+ } else {
16
+ if (positionals.length !== 1 || positionals[0] !== 'clean') throw Error('Expected command: clean')
17
+ const retentionDays = { '7d': 7, '30d': 30, all: 0 }[values['older-than']]
18
+ if (retentionDays === undefined) throw Error('--older-than must be 7d, 30d or all')
19
+ if (values.apply && values['dry-run']) throw Error('--dry-run and --apply cannot be combined')
20
+ const both = !values.streams && !values.groups
21
+ const ledger = values.ledger ? resolve(values.ledger) : join(resolve(process.env.DSH_HOME || join(homedir(), '.dsh')), 'storages', 'cost-meter', 'ledger.json')
22
+ console.log(JSON.stringify(await manageBillionStorage(ledger, { retentionDays, apply: values.apply,
23
+ streams: both || values.streams === true, groups: both || values.groups === true }), null, 2))
24
+ }
25
+ } catch (error) { console.error(error.message); process.exitCode = 1 }
@@ -1,13 +1,15 @@
1
1
  /** DSH side of the local billion-context metering bridge. No inference or HTTP polling. */
2
2
  import { AsyncLocalStorage } from 'node:async_hooks'
3
3
  import { randomUUID } from 'node:crypto'
4
- import { statSync } from 'node:fs'
4
+ import { lstat } from 'node:fs/promises'
5
5
  import { dirname, join } from 'node:path'
6
6
  import { billionVersionSupported } from './billion-context-compatibility.js'
7
7
  import { installBillionAuto } from './billion-context-auto.js'
8
- import { journalInfo, archiveUsageJournals, journalChangeRevision, notifyJournalChange } from './journal-storage.js'
9
- import { billionRoot, billionSessionKey, billionStreamPath, writeBillionJson, readBillionJson, listBillionFiles, iterateBillionCalls, readBillionStream,
10
- listBillionStreamFiles, readBillionStreamFile, billionProvider, usageOnly, usageBuckets, usageSignature, validUsage, validSessionId, billionCoverage, billionCoverageIndex, billionHostMeter } from './billion-context-history.js'
8
+ import { journalInfoAsync, archiveUsageJournals, notifyJournalChange } from './journal-storage.js'
9
+ import { createBillionInventory } from './billion-context-inventory.js'
10
+ import { maintainBillionStorage, readBillionStreamArchive } from './billion-context-storage.js'
11
+ import { billionRoot, billionSessionKey, billionStreamPath, writeBillionJson, readBillionJson, listBillionFiles, iterateBillionCalls, readBillionStreamAsync,
12
+ readBillionStreamFileAsync, billionProvider, usageOnly, usageBuckets, usageSignature, validUsage, validSessionId, billionCoverage, billionCoverageIndex, billionHostMeter } from './billion-context-history.js'
11
13
 
12
14
  const bridgeKey = Symbol.for('dsh-cost-meter.billion-context.v1'), bridges = new WeakMap()
13
15
  const plainId = value => typeof value === 'string' ? value.slice(0, 200) : ''
@@ -26,8 +28,7 @@ function stepIdentity(ctx, id) {
26
28
  export function installBillionContextBilling(ctx, ledger) {
27
29
  if (!ledger.path) return null
28
30
  const root = billionRoot(ledger.path), scope = new AsyncLocalStorage(), streams = new Map(), groups = new Map(), files = new Map()
29
- const dirtyStreams = new Map()
30
- let inventoryAt = 0, inventoryRevision = '', callFiles = []
31
+ const dirtyStreams = new Map(), retryStreams = new Map(), inventory = createBillionInventory(ledger.path)
31
32
  let stopped = false, pending = null, rerun = false, peerAt = 0, peerPending, releaseAuto
32
33
  const status = { protocol: 1, requests: 0, unreported: 0, partial: 0, uncovered: 0, connected: false, failed: false,
33
34
  detected: false, version: '', compatibility: 'unknown', noticeId: '', unresolved: 0 }
@@ -91,9 +92,7 @@ export function installBillionContextBilling(ctx, ledger) {
91
92
  })
92
93
  dirtyStreams.set(stream.streamId, { path: billionStreamPath(root, stream.sessionId, stream.streamId), key: billionSessionKey(stream.sessionId), sessionId: stream.sessionId })
93
94
  if (stream.finishedAt) {
94
- const previous = journalChangeRevision(root)
95
95
  notifyJournalChange(root)
96
- if (previous === inventoryRevision) inventoryRevision = journalChangeRevision(root)
97
96
  }
98
97
  }
99
98
  // The proxy reads this fixed private file. No filesystem path is sent upstream.
@@ -107,27 +106,33 @@ export function installBillionContextBilling(ctx, ledger) {
107
106
  rerun = false
108
107
  ledger.refresh()
109
108
  let work = 0
110
- const legacyCoverage = new Map()
109
+ const legacyCoverage = new Map(), archives = new Map()
111
110
  const dirty = [...dirtyStreams.values()]; dirtyStreams.clear()
112
- const revision = journalChangeRevision(root)
113
- const fullScan = Date.now() - inventoryAt >= 30000 || revision !== inventoryRevision
114
- const streamFiles = fullScan ? listBillionStreamFiles(ledger.path) : dirty
115
- if (fullScan) {
116
- callFiles = listBillionFiles(ledger.path); inventoryAt = Date.now(); inventoryRevision = revision
117
- const retainedPaths = new Set([...streamFiles, ...callFiles].map(file => file.path))
118
- for (const path of files.keys()) if (!retainedPaths.has(path)) files.delete(path)
111
+ const found = await inventory.poll(), callFiles = found.calls
112
+ if (stopped) return
113
+ const streamFiles = [...new Map([...dirty, ...retryStreams.values(), ...found.streams].map(file => [file.path, file])).values()]
114
+ if (found.retained) {
115
+ for (const path of files.keys()) if (!found.retained.has(path)) files.delete(path)
116
+ for (const path of retryStreams.keys()) if (!found.retained.has(path)) retryStreams.delete(path)
119
117
  }
120
- else if (dirty.length) {
121
- const updated = dirty.flatMap(file => listBillionFiles(ledger.path, file.sessionId))
122
- callFiles = [...new Map([...callFiles, ...updated].map(file => [file.path, file])).values()]
118
+ for (const file of found.archives) {
119
+ if (files.has(file.path)) continue
120
+ for (const stream of await readBillionStreamArchive(file)) { applyCoverage(stream, stream.coverage); importHost(stream) }
121
+ files.set(file.path, { archived: true })
123
122
  }
124
123
  // A completed stream can legitimately contain zero proxy call rows.
125
124
  // Discover its once-per-stream fallback independently of the call journal.
126
125
  for (const file of streamFiles) {
127
- const info = statSync(file.path), cached = files.get(file.path)
126
+ if (stopped) break
127
+ let info
128
+ try { info = await lstat(file.path) } catch (error) { if (error.code === 'ENOENT') { retryStreams.delete(file.path); continue }; throw error }
129
+ if (!info.isFile() || info.isSymbolicLink()) continue
130
+ const cached = files.get(file.path)
128
131
  if (cached?.size === info.size && cached?.mtime === info.mtimeMs) continue
129
- const stream = readBillionStreamFile(file)
130
- if (!stream?.finished) continue
132
+ const stream = await readBillionStreamFileAsync(file)
133
+ if (!stream) { retryStreams.delete(file.path); continue }
134
+ if (!stream?.finished) { retryStreams.set(file.path, file); continue }
135
+ retryStreams.delete(file.path)
131
136
  if (!stream.coverage) {
132
137
  if (!legacyCoverage.has(stream.sessionId)) legacyCoverage.set(stream.sessionId, await billionCoverageIndex(ledger.path, stream.sessionId))
133
138
  applyCoverage(stream, legacyCoverage.get(stream.sessionId).get(stream.streamId) ?? await billionCoverage(ledger.path, stream))
@@ -137,15 +142,17 @@ export function installBillionContextBilling(ctx, ledger) {
137
142
  files.set(file.path, { size: info.size, mtime: info.mtimeMs })
138
143
  }
139
144
  for (const file of callFiles) {
145
+ if (stopped) break
140
146
  let info
141
- try { info = journalInfo(file.path) } catch (error) { if (error.code === 'ENOENT') { inventoryAt = 0; rerun = true; continue }; throw error }
147
+ try { info = await journalInfoAsync(file.path) } catch (error) { if (error.code === 'ENOENT') { rerun = true; continue }; throw error }
142
148
  const cached = files.get(file.path)
143
149
  if (cached?.size === info.size && cached?.mtime === info.mtimeMs) continue
144
150
  let offset = cached?.offset <= info.size ? cached.offset : 0, blocked = false
145
151
  const prepared = new Map()
146
152
  for await (const item of iterateBillionCalls(file, offset)) {
153
+ if (stopped) break
147
154
  const row = item.row
148
- let stream = prepared.get(row.streamId) ?? readBillionStream(ledger.path, row)
155
+ let stream = prepared.get(row.streamId) ?? await readBillionStreamAsync(ledger.path, row, archives)
149
156
  if (!stream || !stream.finished) { blocked = true; break }
150
157
  if (!prepared.has(row.streamId)) {
151
158
  const originalCoverage = stream.coverage
@@ -262,7 +269,10 @@ export function installBillionContextBilling(ctx, ledger) {
262
269
  }
263
270
  globalThis[bridgeKey] = api; bridges.set(ledger, api)
264
271
  releaseAuto = installBillionAuto()
265
- const timer = setInterval(() => { void ingest(); void refreshPeer(); void archiveUsageJournals(ledger.path) }, 3000); timer.unref?.()
272
+ const timer = setInterval(() => {
273
+ void ingest(); void refreshPeer(); void archiveUsageJournals(ledger.path)
274
+ void maintainBillionStorage(ledger.path, ledger.config?.billingStorageRetentionDays ?? 7)
275
+ }, 3000); timer.unref?.()
266
276
  ctx.effect(() => () => { clearInterval(timer); api.dispose(); bridges.delete(ledger) }, 'cost-meter: billion-context billing')
267
277
  ctx.on('session/event', (session, event) => {
268
278
  if (!Number.isSafeInteger(event.seq) || event.seq < 0) return
@@ -3,6 +3,8 @@ import { createHash } from 'node:crypto'
3
3
  import { mkdirSync, lstatSync, readFileSync, writeFileSync, renameSync, readdirSync } from 'node:fs'
4
4
  import { join } from 'node:path'
5
5
  import { journalChunks } from './journal-storage.js'
6
+ import { readSmallBillionJson, safeEntries } from './billion-context-inventory.js'
7
+ import { archivedBillionStreams } from './billion-context-storage.js'
6
8
 
7
9
  export const BILLION_USAGE_EVENT = 'cost-meter/billion-context-usage'
8
10
  export const BILLION_REPLACEMENT_EVENT = 'cost-meter/billion-context-replacement'
@@ -57,6 +59,13 @@ export function readBillionStreamFile(file) {
57
59
  && billionSessionKey(stream.sessionId) === file.key && file.path.endsWith(stream.streamId + '.json') ? stream : null
58
60
  } catch { return null }
59
61
  }
62
+ export async function readBillionStreamFileAsync(file) {
63
+ try {
64
+ const stream = await readSmallBillionJson(file.path)
65
+ return stream.protocol === 1 && uuid(stream.streamId) && validSessionId(stream.sessionId)
66
+ && billionSessionKey(stream.sessionId) === file.key && file.path.endsWith(stream.streamId + '.json') ? stream : null
67
+ } catch { return null }
68
+ }
60
69
  export function listBillionFiles(ledgerPath, sessionId) {
61
70
  if (!ledgerPath) return []
62
71
  const root = join(billionRoot(ledgerPath), 'calls')
@@ -76,7 +85,9 @@ export function listBillionFiles(ledgerPath, sessionId) {
76
85
  export function billionSourcePaths(ledgerPath, sessionId) {
77
86
  if (!ledgerPath) return []
78
87
  const streamRoot = join(billionRoot(ledgerPath), 'streams', billionSessionKey(sessionId))
88
+ const archiveRoot = join(billionRoot(ledgerPath), 'stream-archives', billionSessionKey(sessionId))
79
89
  return [...listBillionFiles(ledgerPath, sessionId).map(row => row.path),
90
+ ...directories(archiveRoot).filter(row => row.isFile() && row.name.endsWith('.jsonl.gz') && uuid(row.name.slice(0, -9))).map(row => join(archiveRoot, row.name)),
80
91
  ...directories(streamRoot).filter(row => row.isFile() && row.name.endsWith('.json') && uuid(row.name.slice(0, -5))).map(row => join(streamRoot, row.name))]
81
92
  }
82
93
  /** Bounded UTF-8 rows; offsets only advance through complete newline-terminated writes. */
@@ -109,6 +120,13 @@ export function readBillionStream(ledgerPath, row) {
109
120
  return stream.protocol === 1 && stream.streamId === row.streamId && stream.sessionId === row.sessionId ? stream : null
110
121
  } catch { return null }
111
122
  }
123
+ export async function readBillionStreamAsync(ledgerPath, row, archives = new Map()) {
124
+ const file = { path: billionStreamPath(billionRoot(ledgerPath), row.sessionId, row.streamId), key: billionSessionKey(row.sessionId) }
125
+ const stream = await readBillionStreamFileAsync(file)
126
+ if (stream) return stream
127
+ if (!archives.has(row.sessionId)) archives.set(row.sessionId, await archivedBillionStreams(ledgerPath, row.sessionId))
128
+ return archives.get(row.sessionId).get(row.streamId) ?? null
129
+ }
112
130
  const coverageCounts = () => ({ known: 0, unknown: 0, matched: false })
113
131
  function observeCoverage(counts, stream, row) {
114
132
  if (!['full', 'partial'].includes(stream.policies?.[row.groupId]) || row.kind !== 'model') return
@@ -135,11 +153,11 @@ export async function billionCoverage(ledgerPath, stream, offsets) {
135
153
  }
136
154
  /** Legacy streams share one session scan instead of rescanning for every call. */
137
155
  export async function billionCoverageIndex(ledgerPath, sessionId) {
138
- const entries = new Map()
156
+ const entries = new Map(), archives = new Map()
139
157
  for (const file of listBillionFiles(ledgerPath, sessionId)) for await (const { row } of iterateBillionCalls(file)) {
140
158
  if (!entries.has(row.streamId)) {
141
159
  if (entries.size >= 64000) throw Error('Proxy coverage stream limit exceeded')
142
- entries.set(row.streamId, { stream: readBillionStream(ledgerPath, row), counts: coverageCounts() })
160
+ entries.set(row.streamId, { stream: await readBillionStreamAsync(ledgerPath, row, archives), counts: coverageCounts() })
143
161
  }
144
162
  const { stream, counts } = entries.get(row.streamId)
145
163
  if (stream?.finished && !stream.coverage) observeCoverage(counts, stream, row)
@@ -154,14 +172,20 @@ export function billionHostRecord(stream) {
154
172
  metering: { source: 'billion-context', requestId: stream.streamId, attempt: 1, complete: false, status: 'complete', upstream: 'host-usage' } } }
155
173
  }
156
174
  export async function billionRecordsFor(ledgerPath, sessionId) {
157
- const records = [], streams = new Map(), counts = new Map(), seen = new Set()
158
- for (const file of listBillionStreamFiles(ledgerPath, sessionId)) {
159
- const stream = readBillionStreamFile(file)
175
+ if (!ledgerPath) return []
176
+ const records = [], streams = new Map(), counts = new Map(), seen = new Set(), archives = new Map()
177
+ const key = billionSessionKey(sessionId), dir = join(billionRoot(ledgerPath), 'streams', key)
178
+ for (const entry of await safeEntries(dir)) {
179
+ if (!entry.isFile() || !entry.name.endsWith('.json') || !uuid(entry.name.slice(0, -5))) continue
180
+ const stream = await readBillionStreamFileAsync({ path: join(dir, entry.name), key })
160
181
  if (stream?.finished) streams.set(stream.streamId, stream)
161
182
  }
183
+ // Originals are removed only after archive publication. Read archives after
184
+ // originals so a concurrent compaction cannot create a gap in history.
185
+ for (const [id, stream] of await archivedBillionStreams(ledgerPath, sessionId)) if (!streams.has(id)) streams.set(id, stream)
162
186
  for (const file of listBillionFiles(ledgerPath, sessionId)) for await (const { row } of iterateBillionCalls(file)) {
163
187
  if (seen.has(row.callId)) continue
164
- const stream = readBillionStream(ledgerPath, row)
188
+ const stream = streams.get(row.streamId) ?? await readBillionStreamAsync(ledgerPath, row, archives)
165
189
  if (!stream?.finished || !['full', 'partial'].includes(stream.policies?.[row.groupId])) continue
166
190
  seen.add(row.callId); streams.set(stream.streamId, stream)
167
191
  if (seen.size > 200000) throw Error('Proxy billing detail limit exceeded')
@@ -0,0 +1,60 @@
1
+ /** Incremental asynchronous inventory. Atomic metadata replacement changes its parent directory. */
2
+ import { lstat, readdir, readFile } from 'node:fs/promises'
3
+ import { join } from 'node:path'
4
+
5
+ const sessionKey = /^[a-f\d]{64}$/
6
+ const id = '[a-f\\d]{8}(?:-[a-f\\d]{4}){3}-[a-f\\d]{12}'
7
+ const streamName = new RegExp('^' + id + '\\.json$', 'i')
8
+ const callName = new RegExp('^(' + id + ')(?:\\.sealed)?\\.jsonl(?:\\.gz)?$', 'i')
9
+ const archiveName = new RegExp('^' + id + '\\.jsonl\\.gz$', 'i')
10
+ export async function safeEntries(path) {
11
+ try {
12
+ const info = await lstat(path)
13
+ if (!info.isDirectory() || info.isSymbolicLink()) return []
14
+ return await readdir(path, { withFileTypes: true })
15
+ } catch (error) { if (error.code === 'ENOENT') return []; throw error }
16
+ }
17
+ export async function readSmallBillionJson(path) {
18
+ const info = await lstat(path)
19
+ if (!info.isFile() || info.isSymbolicLink() || info.size > 8192) throw Error('Invalid proxy billing metadata')
20
+ return JSON.parse(await readFile(path, 'utf8'))
21
+ }
22
+ export function createBillionInventory(ledgerPath) {
23
+ const root = ledgerPath + '.billion-context', cache = new Map()
24
+ return {
25
+ async poll() {
26
+ const streams = [], calls = [], archives = [], seen = new Set()
27
+ let changed = false
28
+ // Inspect directories, not every completed stream, on an idle tick. Calls
29
+ // remain candidates because append-only writes do not change directory mtime.
30
+ if (!(await safeEntries(root)).length) { cache.clear(); return { streams, calls, archives, retained: new Set() } }
31
+ for (const kind of ['streams', 'calls', 'stream-archives']) {
32
+ const parent = join(root, kind)
33
+ for (const entry of await safeEntries(parent)) {
34
+ if (!entry.isDirectory() || !sessionKey.test(entry.name)) continue
35
+ const directory = join(parent, entry.name)
36
+ let info
37
+ try { info = await lstat(directory, { bigint: true }) } catch (error) { if (error.code === 'ENOENT') continue; throw error }
38
+ if (!info.isDirectory() || info.isSymbolicLink()) continue
39
+ seen.add(directory)
40
+ const revision = `${info.ino}:${info.mtimeNs}:${info.ctimeNs}`, previous = cache.get(directory)
41
+ let files = previous?.files
42
+ if (previous?.revision !== revision) {
43
+ changed = true
44
+ files = (await safeEntries(directory)).flatMap(file => {
45
+ if (!file.isFile()) return []
46
+ const match = kind === 'calls' ? callName.exec(file.name) : (kind === 'streams' ? streamName : archiveName).exec(file.name)
47
+ return match ? [{ path: join(directory, file.name), key: entry.name, ...(kind === 'calls' ? { source: match[1] } : {}) }] : []
48
+ })
49
+ cache.set(directory, { revision, files })
50
+ }
51
+ if (kind === 'calls') for (const file of files) calls.push(file)
52
+ else if (previous?.revision !== revision) for (const file of files) (kind === 'streams' ? streams : archives).push(file)
53
+ }
54
+ }
55
+ for (const directory of cache.keys()) if (!seen.has(directory)) { cache.delete(directory); changed = true }
56
+ const retained = changed ? new Set([...cache.values()].flatMap(item => item.files.map(file => file.path))) : null
57
+ return { streams, calls, archives, retained }
58
+ },
59
+ }
60
+ }