dsh-cost-meter 1.9.37 → 1.9.39

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -42,4 +42,4 @@ This patch adds the complete cost settings to plugin details, sharing the compon
42
42
  - Use the CNY conversion rate for CNY-priced external calls and legacy pricing repair, regardless of the display currency.
43
43
  - Update frozen CNY/EUR amounts alongside repaired USD costs, retaining recorded FX rates and unrelated amounts. Reconcile complete model currency totals even when USD totals already match; missing legacy currency fields do not imply zero.
44
44
 
45
- No new full-ledger repricing task is introduced. Tests cover currencies, idempotence, partial legacy rows, concurrent/failed saves, slot lifecycle and responsive read-only rendering with synthetic data. Update to **1.9.31** and restart DSH.
45
+ No new full-ledger repricing task is introduced. Tests cover currencies, idempotence, partial legacy rows, concurrent/failed saves, slot lifecycle and responsive read-only rendering with synthetic data. Update to **1.9.31** and restart DSH.
@@ -0,0 +1,31 @@
1
+ # v1.9.38:dsh-xdpool 的 WorkBuddy 积分适配
2
+
3
+ ## 中文
4
+
5
+ 在“费用 → 额度”开启并展开“WorkBuddy 积分”,即可读取同一 DSH 中的 dsh-xdpool 账号积分,无需复制登录凭据。已验证 dsh-xdpool 2.4.6 的发布包接口。
6
+
7
+ “积分来源”支持自动识别、WorkBuddy 连接插件和 dsh-xdpool。自动识别优先读取 dsh-connect-workbuddy 的账号列表,其次读取 XD Pool;旧连接插件的状态接口与明文登录文件兼容方式继续保留。多个插件同时使用时,选择 dsh-xdpool 并刷新即可指定来源。
8
+
9
+ “统计账号”继续支持当前账号、全部账号和勾选账号。XD Pool 的当前账号按其状态接口返回的 activeAccountId 确定,国内和国际分别读取。账号 ID 区分地区和插件来源,已有的连接插件账号筛选不会误选同 ID 的池账号。
10
+
11
+ 积分读取失败、冷却期间未返回积分或已移除的勾选账号显示未知值,完整汇总显示“—”;已读取的积分可在账号列表查看。观测减少量沿用现有统计规则:充值或赠送不算减少,过期可能算减少,积分不折算为货币账单。
12
+
13
+ xdpool 和 xdpool-global 两条 WorkBuddy 路由按现有 WorkBuddy 的 API / Plan 设置归类,并尊重手动覆盖。升级时仅修正账本的 API / Plan 归属,保留总金额、Token、调用和历史汇率;其他聚合、扫码及组合路由不据此归入 WorkBuddy。
14
+
15
+ 兼容验证执行了发布包中的实际状态路由与序列化函数,并检查国内/国际账号、来源选择、失败及冷却状态、双端 codec、保存后重开和窄屏设置界面。积分查询使用同一宿主的 GET 状态接口,不执行切号、签到、自动化或模型测试。
16
+
17
+ 需求建议来自 [@qiyeren 的 issue #404](https://github.com/Han-1413141/dsh-cost-meter/issues/404);接口来源为 [@XDTrees 维护的 dsh-xdpool](https://github.com/XDTrees/dsh-xdpool)。
18
+
19
+ ## English
20
+
21
+ Enable **WorkBuddy credits** under cost settings → quotas to read dsh-xdpool accounts from the same DSH host. No credential copying is required. Compatibility was checked against the published dsh-xdpool 2.4.6 status route and serializers.
22
+
23
+ **Credit source** supports Automatic, WorkBuddy connectors and dsh-xdpool. Automatic uses dsh-connect-workbuddy account lists first, then XD Pool. Select dsh-xdpool and refresh when multiple connectors are installed. Existing legacy connector/file compatibility remains available.
24
+
25
+ Current, all and selected accounts are supported. Current uses the pool's activeAccountId per region. Account IDs include region and source so selections and observed credit decreases do not mix different plugins. Missing, failed or unavailable cooling balances remain unknown, and incomplete totals display a dash.
26
+
27
+ Only xdpool and xdpool-global belong to WorkBuddy billing. Existing API / Plan settings and explicit overrides apply. Historical classification repairs preserve recorded totals, tokens, calls and FX. Other aggregate, OAuth and combo routes are not treated as WorkBuddy credits.
28
+
29
+ Tests cover the published status implementation, source selection, partial data, codecs, persistence and narrow layouts. Integration only reads local GET status endpoints; it does not switch accounts, claim rewards, run automation or probe models.
30
+
31
+ Requested by [@qiyeren in issue #404](https://github.com/Han-1413141/dsh-cost-meter/issues/404), using the public interface maintained by [@XDTrees](https://github.com/XDTrees/dsh-xdpool).
@@ -0,0 +1,33 @@
1
+ # v1.9.39:计费读取恢复、请求取消与导出资源清理
2
+
3
+ ## 中文
4
+
5
+ 本次检查覆盖费用计算、账本保存、代理计费记录及归档、额度查询、配置和凭据变更、统计缓存与费用导出。
6
+
7
+ - Zod 从 4.6.5 升级至 4.6.6,修复 [GHSA-mrhx-c7xc-m3w5](https://github.com/advisories/GHSA-mrhx-c7xc-m3w5)。超长受限数组会直接拒绝,避免为每个元素累积错误对象;回归样本的错误对象从 2001 个降为 1 个。公告将分析贡献署名为 [@MohamedEslam04](https://github.com/MohamedEslam04)。
8
+ - 代理计费目录扫描中途失败时,下次继续发现未交付的记录;单个文件或归档暂时无法读取时保留重试,其他正常记录继续导入。读取恢复后自动补齐,沿用账本收据避免重复收费。格式无效的文件仍会跳过,空闲目录保持增量扫描。
9
+ - 压缩已封存的调用日志时,先同步压缩文件,再发布归档和移除原文件;支持目录同步的平台也会同步目录。同步失败保留原始记录。
10
+ - 本进程活跃计费流读取最近一次成功保存的元数据快照,避免异步读取占用文件期间,Windows 上的同步替换失败。历史和其他进程记录继续从磁盘读取。
11
+ - 网关停用、来源删除、配置变化、密钥更换或清除、插件卸载时取消旧请求;取消传递到账号发现、响应体读取、重试等待和后续账号查询。重定向被拒绝时释放响应体。
12
+ - WorkBuddy 连接插件和 XD Pool 的账号积分查询在读取响应体时也响应取消,释放不再使用的请求。
13
+ - 导出任务在最后一次访问后 10 分钟主动清理。单个对话的大量调用分批处理,取消可及时生效;收集过程限制数据量,CSV 按行检查输出大小,超过限制时提示缩小日期范围或选择单个对话。
14
+ - 某个 Token 分项为无穷大等非法值时,仅忽略该分项,其他有效 Token 继续按对应价格计费。
15
+
16
+ 新增测试模拟扫描中断、文件和归档读取失败、归档同步失败、响应体取消、重试取消、导出过期、大量调用取消及非法 Token,并检查配置和凭据变化后的请求状态。既有费用、历史价格、归档恢复、界面和兼容性检查继续纳入发布验证。
17
+
18
+ 更新后完全重启 DSH,使插件服务端加载新版本。
19
+
20
+ ## English
21
+
22
+ This maintenance release covers billing calculation, ledger persistence, proxy journals and archives, quota requests, configuration and credential changes, statistics caches, and exports.
23
+
24
+ - Upgrade Zod from 4.6.5 to 4.6.6 for [GHSA-mrhx-c7xc-m3w5](https://github.com/advisories/GHSA-mrhx-c7xc-m3w5). A constrained oversized-array regression now produces one issue instead of 2,001. The advisory credits [@MohamedEslam04](https://github.com/MohamedEslam04) for analysis.
25
+ - Retry incomplete inventory scans and transient stream/archive read failures without blocking healthy records. Existing ledger receipts prevent duplicate charges; unchanged directories retain incremental scanning.
26
+ - Sync compressed sealed journals before publishing archives and removing originals. Sync the parent directory on supported platforms. Preserve original records if syncing fails.
27
+ - Read successfully persisted in-memory metadata for streams owned by this process, avoiding Windows replacement conflicts with asynchronous reads. Historical and external streams still read from disk.
28
+ - Cancel gateway requests when a source is disabled, removed or changed, when its key changes or is cleared, and when the plugin unloads. Propagate cancellation through discovery, response bodies, retry waits and queued accounts; release refused redirect responses.
29
+ - Cancel WorkBuddy connector and XD Pool response-body reads when their requests become obsolete.
30
+ - Release inactive export jobs after ten minutes without requiring another API request. Yield during large single-conversation exports, bound collected data, and check CSV size while encoding rows.
31
+ - Ignore invalid Token buckets individually so valid usage retains its calculated cost.
32
+
33
+ Regression coverage includes injected filesystem failures, cancellation, idle export expiry, large-export limits, malformed Token values and gateway lifecycle changes. Fully restart DSH after updating.
@@ -17,7 +17,16 @@ const cell = value => {
17
17
  return '"' + text.replaceAll('"', '""') + '"'
18
18
  }
19
19
  export function billingExportCsv(headers, rows) {
20
- return '\uFEFF' + [headers, ...rows].map(row => row.map(cell).join(',')).join('\r\n') + '\r\n'
20
+ const chunks = ['\uFEFF']; let bytes = 3
21
+ const append = row => {
22
+ const text = row.map(cell).join(',') + '\r\n'
23
+ bytes += Buffer.byteLength(text)
24
+ if (bytes > MAX_BYTES) throw new Error('EXPORT_LIMIT')
25
+ chunks.push(text)
26
+ }
27
+ append(headers)
28
+ for (const row of rows) append(row)
29
+ return chunks.join('')
21
30
  }
22
31
 
23
32
  export function encodeBillingExport(data, format) {
@@ -33,12 +42,15 @@ export function encodeBillingExport(data, format) {
33
42
  const headers = [...common, text('实际会话 ID', 'Actual session ID'), text('时间 (UTC)', 'Time (UTC)'), text('类型', 'Kind'), text('轮次', 'Turn'), text('步骤', 'Step'), text('渠道', 'Provider'), text('模型', 'Model'),
34
43
  'Input tokens', 'Cache read tokens', 'Cache write tokens', 'Output tokens', 'Reasoning tokens', text('费用', 'Cost'), text('API 费用', 'API cost'), text('Plan 等值', 'Plan equivalent'), text('已定价', 'Priced'),
35
44
  text('会话账本总额(参考)', 'Conversation ledger total (reference)'), text('会话明细状态', 'Conversation detail status')]
36
- return billingExportCsv(headers, data.sessions.flatMap(row => row.calls.map(call => [row.id, row.title, data.currency, call.sessionId, new Date(call.atMs).toISOString(), call.kind, call.turn, call.step, call.provider, call.model,
37
- call.input, call.cacheRead, call.cacheWrite, call.output, call.reasoning, call.cost, call.apiCost, Math.max(0, call.cost - call.apiCost), call.priced, row.recorded.cost, row.detailStatus])))
45
+ function* records() {
46
+ for (const row of data.sessions) for (const call of row.calls) yield [row.id, row.title, data.currency, call.sessionId, new Date(call.atMs).toISOString(), call.kind, call.turn, call.step, call.provider, call.model,
47
+ call.input, call.cacheRead, call.cacheWrite, call.output, call.reasoning, call.cost, call.apiCost, Math.max(0, call.cost - call.apiCost), call.priced, row.recorded.cost, row.detailStatus]
48
+ }
49
+ return billingExportCsv(headers, records())
38
50
  }
39
51
 
40
52
  /** Internal detail snapshots never cross RPC without pagination. Only whitelisted billing fields leave here. */
41
- export async function collectBillingExport(ledger, ctx, raw, { active = () => true, progress = () => {}, detail = getSessionBillingSnapshot, now = Date.now } = {}) {
53
+ export async function collectBillingExport(ledger, ctx, raw, { active = () => true, progress = () => {}, detail = getSessionBillingSnapshot, now = Date.now, maxBytes = MAX_BYTES } = {}) {
42
54
  const cutoff = now(), query = ledgerStatisticsQuery(ledger, { ...raw, basis: 'total' })
43
55
  // Analytics transports USD-normalized display values; files contain real units
44
56
  // of their declared currency, including the saved historical FX amounts.
@@ -49,21 +61,33 @@ export async function collectBillingExport(ledger, ctx, raw, { active = () => tr
49
61
  const rows = query.sessionId ? [{ id: query.sessionId, title: stats.sessions.find(row => row.id === query.sessionId)?.title || query.sessionId, ...stats.totals }] : stats.sessions
50
62
  const data = { schema: 'dsh-cost-meter/billing-export/v1', exportedAt: new Date(cutoff).toISOString(), currency: raw.displayCurrency || ledger.config.currency || 'USD', locale: raw.locale === 'en' ? 'en' : 'zh',
51
63
  from: query.from, to: query.to, provider: query.provider, model: query.model, includeSubagents: ledger.config.includeSubagentCost === true, totals: buckets(stats.totals, factor), unassignedCost: stats.unassignedCost * factor, sessions: [] }
52
- let count = 0
64
+ let count = 0, examined = 0, bytes = 0
65
+ // Bound collected data before building the final string/Buffer. Allow for JSON
66
+ // indentation as well as contents; CSV has a second, exact output-size check.
67
+ const reserve = value => {
68
+ const text = JSON.stringify(value, null, 2)
69
+ bytes += Buffer.byteLength(text) + 8 * text.split('\n').length + 2
70
+ if (bytes > maxBytes) throw new Error('EXPORT_LIMIT')
71
+ }
72
+ reserve(data)
53
73
  for (const row of rows) {
54
74
  if (!active()) throw new Error('cancelled')
55
75
  const session = { id: row.id, title: row.title || row.id, recorded: buckets(row, factor), detailStatus: 'complete', calls: [] }
76
+ reserve(session)
56
77
  if (raw.format !== 'csv-summary') {
57
78
  try {
58
79
  const full = await detail(ledger, ctx, { ...query, sessionId: row.id, background: false })
59
80
  if (!active()) throw new Error('cancelled')
60
81
  for (const call of full.calls) {
82
+ if (++examined % 128 === 0) { await setImmediate(); if (!active()) throw new Error('cancelled') }
61
83
  if (call.atMs > cutoff) continue
62
84
  const tokens = { input: 0, cacheRead: 0, cacheWrite: 0, output: 0, reasoning: 0 }
63
85
  for (const part of call.rows) if (Object.hasOwn(tokens, part.bucket)) tokens[part.bucket] += numeric(part.tokens)
64
- session.calls.push({ sessionId: call.sessionId || row.id, atMs: call.atMs, kind: call.kind, turn: call.turn, step: call.step, provider: call.provider, model: call.model,
86
+ const record = { sessionId: call.sessionId || row.id, atMs: call.atMs, kind: call.kind, turn: call.turn, step: call.step, provider: call.provider, model: call.model,
65
87
  ...tokens, cost: numeric(call.cost) * factor, apiCost: numeric(call.apiCost) * factor, plan: call.plan === true, priced: call.priced === true,
66
- rates: call.rows.map(part => ({ bucket: part.bucket, tokens: numeric(part.tokens), requests: numeric(part.requests), rate: numeric(part.rate) * factor, cost: numeric(part.cost) * factor, priced: part.priced === true })) })
88
+ rates: call.rows.map(part => ({ bucket: part.bucket, tokens: numeric(part.tokens), requests: numeric(part.requests), rate: numeric(part.rate) * factor, cost: numeric(part.cost) * factor, priced: part.priced === true })) }
89
+ reserve(record)
90
+ session.calls.push(record)
67
91
  if (++count > 200000) throw new Error('EXPORT_LIMIT')
68
92
  }
69
93
  if (session.calls.length !== row.calls || !same(session.calls.reduce((sum, call) => sum + call.cost, 0), session.recorded.cost)) session.detailStatus = 'ledger-detail-difference'
@@ -76,6 +100,7 @@ export async function collectBillingExport(ledger, ctx, raw, { active = () => tr
76
100
  progress(data.sessions.length, rows.length)
77
101
  await setImmediate()
78
102
  }
103
+ if (!active()) throw new Error('cancelled')
79
104
  if (!query.sessionId) {
80
105
  const assigned = Object.fromEntries(STAT_FIELDS.map(key => [key, data.sessions.reduce((sum, row) => sum + row.recorded[key], 0)]))
81
106
  const remaining = Object.fromEntries(STAT_FIELDS.map(key => [key, Math.max(0, data.totals[key] - assigned[key])]))
@@ -84,8 +109,15 @@ export async function collectBillingExport(ledger, ctx, raw, { active = () => tr
84
109
  return data
85
110
  }
86
111
 
87
- export function createBillingExporter(ledger, ctx, { now = Date.now, collect = collectBillingExport } = {}) {
112
+ export function createBillingExporter(ledger, ctx, { now = Date.now, collect = collectBillingExport, ttlMs = TTL } = {}) {
88
113
  const jobs = new Map(); let closed = false
114
+ const expire = job => { job.cancelled = true; job.bytes = null; clearTimeout(job.timer); jobs.delete(job.id) }
115
+ const touch = (job, time) => {
116
+ job.touched = time
117
+ clearTimeout(job.timer)
118
+ job.timer = setTimeout(() => expire(job), ttlMs)
119
+ job.timer.unref?.()
120
+ }
89
121
  const response = (job, offset = 0) => {
90
122
  const bytes = job.bytes ?? Buffer.alloc(0), nextOffset = Math.min(bytes.length, offset + CHUNK_BYTES)
91
123
  return { id: job.id, status: job.status, filename: job.filename, mime: job.mime, totalBytes: bytes.length, offset, nextOffset, data: bytes.subarray(offset, nextOffset).toString('base64'),
@@ -94,15 +126,15 @@ export function createBillingExporter(ledger, ctx, { now = Date.now, collect = c
94
126
  return {
95
127
  get(raw = {}) {
96
128
  const time = now()
97
- for (const [id, job] of jobs) if (time - job.touched > TTL) { job.cancelled = true; jobs.delete(id) }
129
+ for (const job of jobs.values()) if (time - job.touched >= ttlMs) expire(job)
98
130
  if (closed) throw new Error('Export service is closed')
99
131
  if (raw.id) {
100
132
  const job = jobs.get(raw.id)
101
133
  if (!job) throw new Error('Export expired; start again')
102
- job.touched = time
103
- if (raw.cancel === true) { job.cancelled = true; jobs.delete(raw.id); return response({ ...job, status: 'cancelled', bytes: null }) }
134
+ if (raw.cancel === true) { expire(job); return response({ ...job, status: 'cancelled' }) }
104
135
  const offset = raw.offset ?? 0
105
136
  if (!Number.isSafeInteger(offset) || offset < 0 || offset > (job.bytes?.length ?? 0)) throw new Error('Invalid export offset')
137
+ touch(job, time)
106
138
  return response(job, offset)
107
139
  }
108
140
  if (jobs.size >= 2) throw new Error('Two exports are already active; close one before starting another')
@@ -117,7 +149,11 @@ export function createBillingExporter(ledger, ctx, { now = Date.now, collect = c
117
149
  const job = { id, touched: time, status: 'preparing', cancelled: false, completedSessions: 0, totalSessions: 0, rows: 0, message: '',
118
150
  filename: `dsh-cost-${suffix}-${new Date(time).toISOString().replace(/[:.]/g, '-')}.${raw.format === 'json' ? 'json' : 'csv'}`, mime: raw.format === 'json' ? 'application/json;charset=utf-8' : 'text/csv;charset=utf-8' }
119
151
  jobs.set(id, job)
120
- const active = () => !closed && !job.cancelled && jobs.get(id) === job
152
+ touch(job, time)
153
+ const active = () => {
154
+ if (!job.cancelled && now() - job.touched >= ttlMs) expire(job)
155
+ return !closed && !job.cancelled && jobs.get(id) === job
156
+ }
121
157
  Promise.resolve().then(async () => {
122
158
  const data = await collect(frozen, ctx, request, { now: () => time, active, progress: (completed, total) => { job.completedSessions = completed; job.totalSessions = total } })
123
159
  if (!active()) return
@@ -134,6 +170,6 @@ export function createBillingExporter(ledger, ctx, { now = Date.now, collect = c
134
170
  })
135
171
  return response(job)
136
172
  },
137
- dispose() { closed = true; for (const job of jobs.values()) job.cancelled = true; jobs.clear() },
173
+ dispose() { closed = true; for (const job of jobs.values()) expire(job) },
138
174
  }
139
175
  }
@@ -2,7 +2,7 @@
2
2
  import { AsyncLocalStorage } from 'node:async_hooks'
3
3
  import { randomUUID } from 'node:crypto'
4
4
  import { lstat } from 'node:fs/promises'
5
- import { dirname, join } from 'node:path'
5
+ import { basename, dirname, join } from 'node:path'
6
6
  import { billionVersionSupported } from './billion-context-compatibility.js'
7
7
  import { installBillionAuto } from './billion-context-auto.js'
8
8
  import { journalInfoAsync, archiveUsageJournals, notifyJournalChange } from './journal-storage.js'
@@ -28,7 +28,7 @@ function stepIdentity(ctx, id) {
28
28
  export function installBillionContextBilling(ctx, ledger) {
29
29
  if (!ledger.path) return null
30
30
  const root = billionRoot(ledger.path), scope = new AsyncLocalStorage(), streams = new Map(), groups = new Map(), files = new Map()
31
- const dirtyStreams = new Map(), retryStreams = new Map(), inventory = createBillionInventory(ledger.path)
31
+ const dirtyStreams = new Map(), retryStreams = new Map(), retryArchives = new Map(), inventory = createBillionInventory(ledger.path)
32
32
  let stopped = false, pending = null, rerun = false, peerAt = 0, peerPending, releaseAuto
33
33
  const status = { protocol: 1, requests: 0, unreported: 0, partial: 0, uncovered: 0, connected: false, failed: false,
34
34
  detected: false, version: '', compatibility: 'unknown', noticeId: '', unresolved: 0 }
@@ -82,7 +82,7 @@ export function installBillionContextBilling(ctx, ledger) {
82
82
  }
83
83
  const save = stream => {
84
84
  if (!stream.groups.length) return
85
- writeBillionJson(billionStreamPath(root, stream.sessionId, stream.streamId), {
85
+ const metadata = structuredClone({
86
86
  protocol: 1, streamId: stream.streamId, sessionId: stream.sessionId, model: stream.model, provider: stream.provider,
87
87
  turn: stream.turn, step: stream.step, startedAtMs: stream.startedAtMs, covered: stream.covered ?? stream.lastAck === true,
88
88
  finished: !!stream.finishedAt, policies: stream.policies,
@@ -90,6 +90,8 @@ export function installBillionContextBilling(ctx, ledger) {
90
90
  ...(stream.reported ? { reported: stream.reported } : {}), sourceSeqs: stream.sourceSeqs,
91
91
  ...(stream.coverage ? { coverage: stream.coverage, hostSuppressed: stream.hostSuppressed === true } : {}),
92
92
  })
93
+ writeBillionJson(billionStreamPath(root, stream.sessionId, stream.streamId), metadata)
94
+ stream.persisted = metadata
93
95
  dirtyStreams.set(stream.streamId, { path: billionStreamPath(root, stream.sessionId, stream.streamId), key: billionSessionKey(stream.sessionId), sessionId: stream.sessionId })
94
96
  if (stream.finishedAt) {
95
97
  notifyJournalChange(root)
@@ -105,79 +107,101 @@ export function installBillionContextBilling(ctx, ledger) {
105
107
  do {
106
108
  rerun = false
107
109
  ledger.refresh()
108
- let work = 0
110
+ let work = 0, failed = false
109
111
  const legacyCoverage = new Map(), archives = new Map()
110
- const dirty = [...dirtyStreams.values()]; dirtyStreams.clear()
111
112
  const found = await inventory.poll(), callFiles = found.calls
112
113
  if (stopped) return
114
+ const dirty = [...dirtyStreams.values()]; dirtyStreams.clear()
113
115
  const streamFiles = [...new Map([...dirty, ...retryStreams.values(), ...found.streams].map(file => [file.path, file])).values()]
114
116
  if (found.retained) {
115
117
  for (const path of files.keys()) if (!found.retained.has(path)) files.delete(path)
116
118
  for (const path of retryStreams.keys()) if (!found.retained.has(path)) retryStreams.delete(path)
119
+ for (const path of retryArchives.keys()) if (!found.retained.has(path)) retryArchives.delete(path)
117
120
  }
118
- for (const file of found.archives) {
119
- if (files.has(file.path)) continue
120
- for (const stream of await readBillionStreamArchive(file)) { applyCoverage(stream, stream.coverage); importHost(stream) }
121
- files.set(file.path, { archived: true })
121
+ for (const file of found.archives) retryArchives.set(file.path, file)
122
+ for (const file of retryArchives.values()) {
123
+ if (stopped) return
124
+ try {
125
+ if (!files.has(file.path)) {
126
+ for (const stream of await readBillionStreamArchive(file)) { applyCoverage(stream, stream.coverage); importHost(stream) }
127
+ files.set(file.path, { archived: true })
128
+ }
129
+ retryArchives.delete(file.path)
130
+ } catch (error) { if (error.code === 'ENOENT') retryArchives.delete(file.path); else failed = true }
122
131
  }
123
132
  // A completed stream can legitimately contain zero proxy call rows.
124
133
  // Discover its once-per-stream fallback independently of the call journal.
125
134
  for (const file of streamFiles) {
126
135
  if (stopped) break
127
- let info
128
- try { info = await lstat(file.path) } catch (error) { if (error.code === 'ENOENT') { retryStreams.delete(file.path); continue }; throw error }
129
- if (!info.isFile() || info.isSymbolicLink()) continue
130
- const cached = files.get(file.path)
131
- if (cached?.size === info.size && cached?.mtime === info.mtimeMs) continue
132
- const stream = await readBillionStreamFileAsync(file)
133
- if (!stream) { retryStreams.delete(file.path); continue }
134
- if (!stream?.finished) { retryStreams.set(file.path, file); continue }
135
- retryStreams.delete(file.path)
136
- if (!stream.coverage) {
137
- if (!legacyCoverage.has(stream.sessionId)) legacyCoverage.set(stream.sessionId, await billionCoverageIndex(ledger.path, stream.sessionId))
138
- applyCoverage(stream, legacyCoverage.get(stream.sessionId).get(stream.streamId) ?? await billionCoverage(ledger.path, stream))
139
- writeBillionJson(file.path, stream)
140
- } else applyCoverage(stream, stream.coverage)
141
- importHost(stream)
142
- files.set(file.path, { size: info.size, mtime: info.mtimeMs })
136
+ retryStreams.set(file.path, file)
137
+ try {
138
+ const cached = files.get(file.path)
139
+ const owned = streams.get(basename(file.path, '.json'))
140
+ let stream, info
141
+ if (owned) {
142
+ // Reuse only the successfully saved snapshot. Reading our active
143
+ // file asynchronously can hold it open across a Windows rename.
144
+ stream = owned.persisted
145
+ if (!stream?.finished) continue
146
+ if (cached?.snapshot === stream) { retryStreams.delete(file.path); continue }
147
+ } else {
148
+ try { info = await lstat(file.path) } catch (error) { if (error.code === 'ENOENT') { retryStreams.delete(file.path); continue }; throw error }
149
+ if (!info.isFile() || info.isSymbolicLink()) { retryStreams.delete(file.path); continue }
150
+ if (cached?.size === info.size && cached?.mtime === info.mtimeMs) { retryStreams.delete(file.path); continue }
151
+ stream = await readBillionStreamFileAsync(file)
152
+ }
153
+ if (!stream) { retryStreams.delete(file.path); continue }
154
+ if (!stream?.finished) { retryStreams.set(file.path, file); continue }
155
+ if (!stream.coverage) {
156
+ if (!legacyCoverage.has(stream.sessionId)) legacyCoverage.set(stream.sessionId, await billionCoverageIndex(ledger.path, stream.sessionId))
157
+ applyCoverage(stream, legacyCoverage.get(stream.sessionId).get(stream.streamId) ?? await billionCoverage(ledger.path, stream))
158
+ writeBillionJson(file.path, stream)
159
+ } else applyCoverage(stream, stream.coverage)
160
+ importHost(stream)
161
+ files.set(file.path, owned ? { snapshot: stream } : { size: info.size, mtime: info.mtimeMs })
162
+ retryStreams.delete(file.path)
163
+ } catch { failed = true }
143
164
  }
144
165
  for (const file of callFiles) {
145
166
  if (stopped) break
146
- let info
147
- try { info = await journalInfoAsync(file.path) } catch (error) { if (error.code === 'ENOENT') { rerun = true; continue }; throw error }
148
- const cached = files.get(file.path)
149
- if (cached?.size === info.size && cached?.mtime === info.mtimeMs) continue
150
- let offset = cached?.offset <= info.size ? cached.offset : 0, blocked = false
151
- const prepared = new Map()
152
- for await (const item of iterateBillionCalls(file, offset)) {
153
- if (stopped) break
154
- const row = item.row
155
- let stream = prepared.get(row.streamId) ?? await readBillionStreamAsync(ledger.path, row, archives)
156
- if (!stream || !stream.finished) { blocked = true; break }
157
- if (!prepared.has(row.streamId)) {
158
- const originalCoverage = stream.coverage
159
- if (!originalCoverage && !legacyCoverage.has(stream.sessionId)) legacyCoverage.set(stream.sessionId, await billionCoverageIndex(ledger.path, stream.sessionId))
160
- applyCoverage(stream, originalCoverage ?? legacyCoverage.get(stream.sessionId).get(stream.streamId))
161
- if (!originalCoverage) writeBillionJson(billionStreamPath(root, stream.sessionId, stream.streamId), stream)
162
- importHost(stream)
163
- prepared.set(row.streamId, stream)
164
- }
165
- offset = item.offset
166
- status.requests++
167
- if (!row.usage) status.unreported++
168
- else if (!row.complete) status.partial++
169
- const meter = { source: row.source + ':' + file.key, seq: row.seq }
170
- const reset = ledger.historyReset?.at ?? 0
171
- if ((ledger.usageCursors?.[meter.source] ?? 0) < meter.seq) {
172
- if (!row.usage || row.startedAtMs <= reset || stream.policies?.[row.groupId] === 'fallback') ledger.advanceUsageCursor(meter)
173
- else ledger.account(usageBuckets(row.usage), row.model, row.sessionId, row.startedAtMs, billionProvider(row, stream), meter)
167
+ try {
168
+ let info
169
+ try { info = await journalInfoAsync(file.path) } catch (error) { if (error.code === 'ENOENT') { rerun = true; continue }; throw error }
170
+ const cached = files.get(file.path)
171
+ if (cached?.size === info.size && cached?.mtime === info.mtimeMs) continue
172
+ let offset = cached?.offset <= info.size ? cached.offset : 0, blocked = false
173
+ const prepared = new Map()
174
+ for await (const item of iterateBillionCalls(file, offset)) {
175
+ if (stopped) break
176
+ const row = item.row
177
+ const owned = streams.get(row.streamId)
178
+ let stream = prepared.get(row.streamId) ?? (owned ? owned.persisted : await readBillionStreamAsync(ledger.path, row, archives))
179
+ if (!stream || !stream.finished) { blocked = true; break }
180
+ if (!prepared.has(row.streamId)) {
181
+ const originalCoverage = stream.coverage
182
+ if (!originalCoverage && !legacyCoverage.has(stream.sessionId)) legacyCoverage.set(stream.sessionId, await billionCoverageIndex(ledger.path, stream.sessionId))
183
+ applyCoverage(stream, originalCoverage ?? legacyCoverage.get(stream.sessionId).get(stream.streamId))
184
+ if (!originalCoverage) writeBillionJson(billionStreamPath(root, stream.sessionId, stream.streamId), stream)
185
+ importHost(stream)
186
+ prepared.set(row.streamId, stream)
187
+ }
188
+ offset = item.offset
189
+ status.requests++
190
+ if (!row.usage) status.unreported++
191
+ else if (!row.complete) status.partial++
192
+ const meter = { source: row.source + ':' + file.key, seq: row.seq }
193
+ const reset = ledger.historyReset?.at ?? 0
194
+ if ((ledger.usageCursors?.[meter.source] ?? 0) < meter.seq) {
195
+ if (!row.usage || row.startedAtMs <= reset || stream.policies?.[row.groupId] === 'fallback') ledger.advanceUsageCursor(meter)
196
+ else ledger.account(usageBuckets(row.usage), row.model, row.sessionId, row.startedAtMs, billionProvider(row, stream), meter)
197
+ }
198
+ if (++work >= 5000) break
174
199
  }
175
- if (++work >= 5000) break
176
- }
177
- files.set(file.path, { offset, size: blocked || offset < info.size && work >= 5000 ? -1 : info.size, mtime: info.mtimeMs })
178
- if (work >= 5000) break
200
+ files.set(file.path, { offset, size: blocked || offset < info.size && work >= 5000 ? -1 : info.size, mtime: info.mtimeMs })
201
+ if (work >= 5000) break
202
+ } catch { failed = true }
179
203
  }
180
- status.failed = false
204
+ status.failed = failed
181
205
  } while (rerun && !stopped)
182
206
  })().catch(() => { status.failed = true }).finally(() => { pending = null })
183
207
  return pending
@@ -62,9 +62,12 @@ export function readBillionStreamFile(file) {
62
62
  export async function readBillionStreamFileAsync(file) {
63
63
  try {
64
64
  const stream = await readSmallBillionJson(file.path)
65
- return stream.protocol === 1 && uuid(stream.streamId) && validSessionId(stream.sessionId)
65
+ return stream?.protocol === 1 && uuid(stream.streamId) && validSessionId(stream.sessionId)
66
66
  && billionSessionKey(stream.sessionId) === file.key && file.path.endsWith(stream.streamId + '.json') ? stream : null
67
- } catch { return null }
67
+ } catch (error) {
68
+ if (error.code === 'ENOENT' || error.code === 'INVALID_BILLION_METADATA' || error instanceof SyntaxError) return null
69
+ throw error
70
+ }
68
71
  }
69
72
  export function listBillionFiles(ledgerPath, sessionId) {
70
73
  if (!ledgerPath) return []
@@ -16,14 +16,15 @@ export async function safeEntries(path) {
16
16
  }
17
17
  export async function readSmallBillionJson(path) {
18
18
  const info = await lstat(path)
19
- if (!info.isFile() || info.isSymbolicLink() || info.size > 8192) throw Error('Invalid proxy billing metadata')
19
+ if (!info.isFile() || info.isSymbolicLink() || info.size > 8192) throw Object.assign(Error('Invalid proxy billing metadata'), { code: 'INVALID_BILLION_METADATA' })
20
20
  return JSON.parse(await readFile(path, 'utf8'))
21
21
  }
22
22
  export function createBillionInventory(ledgerPath) {
23
- const root = ledgerPath + '.billion-context', cache = new Map()
23
+ const root = ledgerPath + '.billion-context'
24
+ let cache = new Map()
24
25
  return {
25
26
  async poll() {
26
- const streams = [], calls = [], archives = [], seen = new Set()
27
+ const streams = [], calls = [], archives = [], seen = new Set(), next = new Map(cache)
27
28
  let changed = false
28
29
  // Inspect directories, not every completed stream, on an idle tick. Calls
29
30
  // remain candidates because append-only writes do not change directory mtime.
@@ -46,13 +47,15 @@ export function createBillionInventory(ledgerPath) {
46
47
  const match = kind === 'calls' ? callName.exec(file.name) : (kind === 'streams' ? streamName : archiveName).exec(file.name)
47
48
  return match ? [{ path: join(directory, file.name), key: entry.name, ...(kind === 'calls' ? { source: match[1] } : {}) }] : []
48
49
  })
49
- cache.set(directory, { revision, files })
50
+ next.set(directory, { revision, files })
50
51
  }
51
52
  if (kind === 'calls') for (const file of files) calls.push(file)
52
53
  else if (previous?.revision !== revision) for (const file of files) (kind === 'streams' ? streams : archives).push(file)
53
54
  }
54
55
  }
55
- for (const directory of cache.keys()) if (!seen.has(directory)) { cache.delete(directory); changed = true }
56
+ for (const directory of next.keys()) if (!seen.has(directory)) { next.delete(directory); changed = true }
57
+ // A failed poll must not mark undispatched candidates as already scanned.
58
+ cache = next
56
59
  const retained = changed ? new Set([...cache.values()].flatMap(item => item.files.map(file => file.path))) : null
57
60
  return { streams, calls, archives, retained }
58
61
  },