dsh-cost-meter 1.9.37 → 1.9.39
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +588 -586
- package/README.zh-CN.md +592 -590
- package/docs/update-notes-1.9.31.md +1 -1
- package/docs/update-notes-1.9.38.md +31 -0
- package/docs/update-notes-1.9.39.md +33 -0
- package/lib/billing-export.js +49 -13
- package/lib/billion-context-billing.js +82 -58
- package/lib/billion-context-history.js +5 -2
- package/lib/billion-context-inventory.js +8 -5
- package/lib/client.js +1 -1
- package/lib/client.plan-settings.js +1 -1
- package/lib/coding-plans.js +1 -1
- package/lib/external-usage.js +119 -119
- package/lib/gateway-quotas.js +33 -14
- package/lib/index.js +22 -13
- package/lib/journal-storage.js +7 -1
- package/lib/ledger-totals.js +57 -57
- package/lib/ledger-writer-worker.js +1 -1
- package/lib/plan-billing.js +1 -1
- package/lib/pricing.js +6 -5
- package/lib/store.js +2 -0
- package/lib/typert.host.js +1 -0
- package/lib/workbuddy-accounts.js +32 -12
- package/package.json +5 -3
|
@@ -42,4 +42,4 @@ This patch adds the complete cost settings to plugin details, sharing the compon
|
|
|
42
42
|
- Use the CNY conversion rate for CNY-priced external calls and legacy pricing repair, regardless of the display currency.
|
|
43
43
|
- Update frozen CNY/EUR amounts alongside repaired USD costs, retaining recorded FX rates and unrelated amounts. Reconcile complete model currency totals even when USD totals already match; missing legacy currency fields do not imply zero.
|
|
44
44
|
|
|
45
|
-
No new full-ledger repricing task is introduced. Tests cover currencies, idempotence, partial legacy rows, concurrent/failed saves, slot lifecycle and responsive read-only rendering with synthetic data. Update to **1.9.31** and restart DSH.
|
|
45
|
+
No new full-ledger repricing task is introduced. Tests cover currencies, idempotence, partial legacy rows, concurrent/failed saves, slot lifecycle and responsive read-only rendering with synthetic data. Update to **1.9.31** and restart DSH.
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# v1.9.38:dsh-xdpool 的 WorkBuddy 积分适配
|
|
2
|
+
|
|
3
|
+
## 中文
|
|
4
|
+
|
|
5
|
+
在“费用 → 额度”开启并展开“WorkBuddy 积分”,即可读取同一 DSH 中的 dsh-xdpool 账号积分,无需复制登录凭据。已验证 dsh-xdpool 2.4.6 的发布包接口。
|
|
6
|
+
|
|
7
|
+
“积分来源”支持自动识别、WorkBuddy 连接插件和 dsh-xdpool。自动识别优先读取 dsh-connect-workbuddy 的账号列表,其次读取 XD Pool;旧连接插件的状态接口与明文登录文件兼容方式继续保留。多个插件同时使用时,选择 dsh-xdpool 并刷新即可指定来源。
|
|
8
|
+
|
|
9
|
+
“统计账号”继续支持当前账号、全部账号和勾选账号。XD Pool 的当前账号按其状态接口返回的 activeAccountId 确定,国内和国际分别读取。账号 ID 区分地区和插件来源,已有的连接插件账号筛选不会误选同 ID 的池账号。
|
|
10
|
+
|
|
11
|
+
积分读取失败、冷却期间未返回积分或已移除的勾选账号显示未知值,完整汇总显示“—”;已读取的积分可在账号列表查看。观测减少量沿用现有统计规则:充值或赠送不算减少,过期可能算减少,积分不折算为货币账单。
|
|
12
|
+
|
|
13
|
+
xdpool 和 xdpool-global 两条 WorkBuddy 路由按现有 WorkBuddy 的 API / Plan 设置归类,并尊重手动覆盖。升级时仅修正账本的 API / Plan 归属,保留总金额、Token、调用和历史汇率;其他聚合、扫码及组合路由不据此归入 WorkBuddy。
|
|
14
|
+
|
|
15
|
+
兼容验证执行了发布包中的实际状态路由与序列化函数,并检查国内/国际账号、来源选择、失败及冷却状态、双端 codec、保存后重开和窄屏设置界面。积分查询使用同一宿主的 GET 状态接口,不执行切号、签到、自动化或模型测试。
|
|
16
|
+
|
|
17
|
+
需求建议来自 [@qiyeren 的 issue #404](https://github.com/Han-1413141/dsh-cost-meter/issues/404);接口来源为 [@XDTrees 维护的 dsh-xdpool](https://github.com/XDTrees/dsh-xdpool)。
|
|
18
|
+
|
|
19
|
+
## English
|
|
20
|
+
|
|
21
|
+
Enable **WorkBuddy credits** under cost settings → quotas to read dsh-xdpool accounts from the same DSH host. No credential copying is required. Compatibility was checked against the published dsh-xdpool 2.4.6 status route and serializers.
|
|
22
|
+
|
|
23
|
+
**Credit source** supports Automatic, WorkBuddy connectors and dsh-xdpool. Automatic uses dsh-connect-workbuddy account lists first, then XD Pool. Select dsh-xdpool and refresh when multiple connectors are installed. Existing legacy connector/file compatibility remains available.
|
|
24
|
+
|
|
25
|
+
Current, all and selected accounts are supported. Current uses the pool's activeAccountId per region. Account IDs include region and source so selections and observed credit decreases do not mix different plugins. Missing, failed or unavailable cooling balances remain unknown, and incomplete totals display a dash.
|
|
26
|
+
|
|
27
|
+
Only xdpool and xdpool-global belong to WorkBuddy billing. Existing API / Plan settings and explicit overrides apply. Historical classification repairs preserve recorded totals, tokens, calls and FX. Other aggregate, OAuth and combo routes are not treated as WorkBuddy credits.
|
|
28
|
+
|
|
29
|
+
Tests cover the published status implementation, source selection, partial data, codecs, persistence and narrow layouts. Integration only reads local GET status endpoints; it does not switch accounts, claim rewards, run automation or probe models.
|
|
30
|
+
|
|
31
|
+
Requested by [@qiyeren in issue #404](https://github.com/Han-1413141/dsh-cost-meter/issues/404), using the public interface maintained by [@XDTrees](https://github.com/XDTrees/dsh-xdpool).
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# v1.9.39:计费读取恢复、请求取消与导出资源清理
|
|
2
|
+
|
|
3
|
+
## 中文
|
|
4
|
+
|
|
5
|
+
本次检查覆盖费用计算、账本保存、代理计费记录及归档、额度查询、配置和凭据变更、统计缓存与费用导出。
|
|
6
|
+
|
|
7
|
+
- Zod 从 4.6.5 升级至 4.6.6,修复 [GHSA-mrhx-c7xc-m3w5](https://github.com/advisories/GHSA-mrhx-c7xc-m3w5)。超长受限数组会直接拒绝,避免为每个元素累积错误对象;回归样本的错误对象从 2001 个降为 1 个。公告将分析贡献署名为 [@MohamedEslam04](https://github.com/MohamedEslam04)。
|
|
8
|
+
- 代理计费目录扫描中途失败时,下次继续发现未交付的记录;单个文件或归档暂时无法读取时保留重试,其他正常记录继续导入。读取恢复后自动补齐,沿用账本收据避免重复收费。格式无效的文件仍会跳过,空闲目录保持增量扫描。
|
|
9
|
+
- 压缩已封存的调用日志时,先同步压缩文件,再发布归档和移除原文件;支持目录同步的平台也会同步目录。同步失败保留原始记录。
|
|
10
|
+
- 本进程活跃计费流读取最近一次成功保存的元数据快照,避免异步读取占用文件期间,Windows 上的同步替换失败。历史和其他进程记录继续从磁盘读取。
|
|
11
|
+
- 网关停用、来源删除、配置变化、密钥更换或清除、插件卸载时取消旧请求;取消传递到账号发现、响应体读取、重试等待和后续账号查询。重定向被拒绝时释放响应体。
|
|
12
|
+
- WorkBuddy 连接插件和 XD Pool 的账号积分查询在读取响应体时也响应取消,释放不再使用的请求。
|
|
13
|
+
- 导出任务在最后一次访问后 10 分钟主动清理。单个对话的大量调用分批处理,取消可及时生效;收集过程限制数据量,CSV 按行检查输出大小,超过限制时提示缩小日期范围或选择单个对话。
|
|
14
|
+
- 某个 Token 分项为无穷大等非法值时,仅忽略该分项,其他有效 Token 继续按对应价格计费。
|
|
15
|
+
|
|
16
|
+
新增测试模拟扫描中断、文件和归档读取失败、归档同步失败、响应体取消、重试取消、导出过期、大量调用取消及非法 Token,并检查配置和凭据变化后的请求状态。既有费用、历史价格、归档恢复、界面和兼容性检查继续纳入发布验证。
|
|
17
|
+
|
|
18
|
+
更新后完全重启 DSH,使插件服务端加载新版本。
|
|
19
|
+
|
|
20
|
+
## English
|
|
21
|
+
|
|
22
|
+
This maintenance release covers billing calculation, ledger persistence, proxy journals and archives, quota requests, configuration and credential changes, statistics caches, and exports.
|
|
23
|
+
|
|
24
|
+
- Upgrade Zod from 4.6.5 to 4.6.6 for [GHSA-mrhx-c7xc-m3w5](https://github.com/advisories/GHSA-mrhx-c7xc-m3w5). A constrained oversized-array regression now produces one issue instead of 2,001. The advisory credits [@MohamedEslam04](https://github.com/MohamedEslam04) for analysis.
|
|
25
|
+
- Retry incomplete inventory scans and transient stream/archive read failures without blocking healthy records. Existing ledger receipts prevent duplicate charges; unchanged directories retain incremental scanning.
|
|
26
|
+
- Sync compressed sealed journals before publishing archives and removing originals. Sync the parent directory on supported platforms. Preserve original records if syncing fails.
|
|
27
|
+
- Read successfully persisted in-memory metadata for streams owned by this process, avoiding Windows replacement conflicts with asynchronous reads. Historical and external streams still read from disk.
|
|
28
|
+
- Cancel gateway requests when a source is disabled, removed or changed, when its key changes or is cleared, and when the plugin unloads. Propagate cancellation through discovery, response bodies, retry waits and queued accounts; release refused redirect responses.
|
|
29
|
+
- Cancel WorkBuddy connector and XD Pool response-body reads when their requests become obsolete.
|
|
30
|
+
- Release inactive export jobs after ten minutes without requiring another API request. Yield during large single-conversation exports, bound collected data, and check CSV size while encoding rows.
|
|
31
|
+
- Ignore invalid Token buckets individually so valid usage retains its calculated cost.
|
|
32
|
+
|
|
33
|
+
Regression coverage includes injected filesystem failures, cancellation, idle export expiry, large-export limits, malformed Token values and gateway lifecycle changes. Fully restart DSH after updating.
|
package/lib/billing-export.js
CHANGED
|
@@ -17,7 +17,16 @@ const cell = value => {
|
|
|
17
17
|
return '"' + text.replaceAll('"', '""') + '"'
|
|
18
18
|
}
|
|
19
19
|
export function billingExportCsv(headers, rows) {
|
|
20
|
-
|
|
20
|
+
const chunks = ['\uFEFF']; let bytes = 3
|
|
21
|
+
const append = row => {
|
|
22
|
+
const text = row.map(cell).join(',') + '\r\n'
|
|
23
|
+
bytes += Buffer.byteLength(text)
|
|
24
|
+
if (bytes > MAX_BYTES) throw new Error('EXPORT_LIMIT')
|
|
25
|
+
chunks.push(text)
|
|
26
|
+
}
|
|
27
|
+
append(headers)
|
|
28
|
+
for (const row of rows) append(row)
|
|
29
|
+
return chunks.join('')
|
|
21
30
|
}
|
|
22
31
|
|
|
23
32
|
export function encodeBillingExport(data, format) {
|
|
@@ -33,12 +42,15 @@ export function encodeBillingExport(data, format) {
|
|
|
33
42
|
const headers = [...common, text('实际会话 ID', 'Actual session ID'), text('时间 (UTC)', 'Time (UTC)'), text('类型', 'Kind'), text('轮次', 'Turn'), text('步骤', 'Step'), text('渠道', 'Provider'), text('模型', 'Model'),
|
|
34
43
|
'Input tokens', 'Cache read tokens', 'Cache write tokens', 'Output tokens', 'Reasoning tokens', text('费用', 'Cost'), text('API 费用', 'API cost'), text('Plan 等值', 'Plan equivalent'), text('已定价', 'Priced'),
|
|
35
44
|
text('会话账本总额(参考)', 'Conversation ledger total (reference)'), text('会话明细状态', 'Conversation detail status')]
|
|
36
|
-
|
|
37
|
-
|
|
45
|
+
function* records() {
|
|
46
|
+
for (const row of data.sessions) for (const call of row.calls) yield [row.id, row.title, data.currency, call.sessionId, new Date(call.atMs).toISOString(), call.kind, call.turn, call.step, call.provider, call.model,
|
|
47
|
+
call.input, call.cacheRead, call.cacheWrite, call.output, call.reasoning, call.cost, call.apiCost, Math.max(0, call.cost - call.apiCost), call.priced, row.recorded.cost, row.detailStatus]
|
|
48
|
+
}
|
|
49
|
+
return billingExportCsv(headers, records())
|
|
38
50
|
}
|
|
39
51
|
|
|
40
52
|
/** Internal detail snapshots never cross RPC without pagination. Only whitelisted billing fields leave here. */
|
|
41
|
-
export async function collectBillingExport(ledger, ctx, raw, { active = () => true, progress = () => {}, detail = getSessionBillingSnapshot, now = Date.now } = {}) {
|
|
53
|
+
export async function collectBillingExport(ledger, ctx, raw, { active = () => true, progress = () => {}, detail = getSessionBillingSnapshot, now = Date.now, maxBytes = MAX_BYTES } = {}) {
|
|
42
54
|
const cutoff = now(), query = ledgerStatisticsQuery(ledger, { ...raw, basis: 'total' })
|
|
43
55
|
// Analytics transports USD-normalized display values; files contain real units
|
|
44
56
|
// of their declared currency, including the saved historical FX amounts.
|
|
@@ -49,21 +61,33 @@ export async function collectBillingExport(ledger, ctx, raw, { active = () => tr
|
|
|
49
61
|
const rows = query.sessionId ? [{ id: query.sessionId, title: stats.sessions.find(row => row.id === query.sessionId)?.title || query.sessionId, ...stats.totals }] : stats.sessions
|
|
50
62
|
const data = { schema: 'dsh-cost-meter/billing-export/v1', exportedAt: new Date(cutoff).toISOString(), currency: raw.displayCurrency || ledger.config.currency || 'USD', locale: raw.locale === 'en' ? 'en' : 'zh',
|
|
51
63
|
from: query.from, to: query.to, provider: query.provider, model: query.model, includeSubagents: ledger.config.includeSubagentCost === true, totals: buckets(stats.totals, factor), unassignedCost: stats.unassignedCost * factor, sessions: [] }
|
|
52
|
-
let count = 0
|
|
64
|
+
let count = 0, examined = 0, bytes = 0
|
|
65
|
+
// Bound collected data before building the final string/Buffer. Allow for JSON
|
|
66
|
+
// indentation as well as contents; CSV has a second, exact output-size check.
|
|
67
|
+
const reserve = value => {
|
|
68
|
+
const text = JSON.stringify(value, null, 2)
|
|
69
|
+
bytes += Buffer.byteLength(text) + 8 * text.split('\n').length + 2
|
|
70
|
+
if (bytes > maxBytes) throw new Error('EXPORT_LIMIT')
|
|
71
|
+
}
|
|
72
|
+
reserve(data)
|
|
53
73
|
for (const row of rows) {
|
|
54
74
|
if (!active()) throw new Error('cancelled')
|
|
55
75
|
const session = { id: row.id, title: row.title || row.id, recorded: buckets(row, factor), detailStatus: 'complete', calls: [] }
|
|
76
|
+
reserve(session)
|
|
56
77
|
if (raw.format !== 'csv-summary') {
|
|
57
78
|
try {
|
|
58
79
|
const full = await detail(ledger, ctx, { ...query, sessionId: row.id, background: false })
|
|
59
80
|
if (!active()) throw new Error('cancelled')
|
|
60
81
|
for (const call of full.calls) {
|
|
82
|
+
if (++examined % 128 === 0) { await setImmediate(); if (!active()) throw new Error('cancelled') }
|
|
61
83
|
if (call.atMs > cutoff) continue
|
|
62
84
|
const tokens = { input: 0, cacheRead: 0, cacheWrite: 0, output: 0, reasoning: 0 }
|
|
63
85
|
for (const part of call.rows) if (Object.hasOwn(tokens, part.bucket)) tokens[part.bucket] += numeric(part.tokens)
|
|
64
|
-
|
|
86
|
+
const record = { sessionId: call.sessionId || row.id, atMs: call.atMs, kind: call.kind, turn: call.turn, step: call.step, provider: call.provider, model: call.model,
|
|
65
87
|
...tokens, cost: numeric(call.cost) * factor, apiCost: numeric(call.apiCost) * factor, plan: call.plan === true, priced: call.priced === true,
|
|
66
|
-
rates: call.rows.map(part => ({ bucket: part.bucket, tokens: numeric(part.tokens), requests: numeric(part.requests), rate: numeric(part.rate) * factor, cost: numeric(part.cost) * factor, priced: part.priced === true })) }
|
|
88
|
+
rates: call.rows.map(part => ({ bucket: part.bucket, tokens: numeric(part.tokens), requests: numeric(part.requests), rate: numeric(part.rate) * factor, cost: numeric(part.cost) * factor, priced: part.priced === true })) }
|
|
89
|
+
reserve(record)
|
|
90
|
+
session.calls.push(record)
|
|
67
91
|
if (++count > 200000) throw new Error('EXPORT_LIMIT')
|
|
68
92
|
}
|
|
69
93
|
if (session.calls.length !== row.calls || !same(session.calls.reduce((sum, call) => sum + call.cost, 0), session.recorded.cost)) session.detailStatus = 'ledger-detail-difference'
|
|
@@ -76,6 +100,7 @@ export async function collectBillingExport(ledger, ctx, raw, { active = () => tr
|
|
|
76
100
|
progress(data.sessions.length, rows.length)
|
|
77
101
|
await setImmediate()
|
|
78
102
|
}
|
|
103
|
+
if (!active()) throw new Error('cancelled')
|
|
79
104
|
if (!query.sessionId) {
|
|
80
105
|
const assigned = Object.fromEntries(STAT_FIELDS.map(key => [key, data.sessions.reduce((sum, row) => sum + row.recorded[key], 0)]))
|
|
81
106
|
const remaining = Object.fromEntries(STAT_FIELDS.map(key => [key, Math.max(0, data.totals[key] - assigned[key])]))
|
|
@@ -84,8 +109,15 @@ export async function collectBillingExport(ledger, ctx, raw, { active = () => tr
|
|
|
84
109
|
return data
|
|
85
110
|
}
|
|
86
111
|
|
|
87
|
-
export function createBillingExporter(ledger, ctx, { now = Date.now, collect = collectBillingExport } = {}) {
|
|
112
|
+
export function createBillingExporter(ledger, ctx, { now = Date.now, collect = collectBillingExport, ttlMs = TTL } = {}) {
|
|
88
113
|
const jobs = new Map(); let closed = false
|
|
114
|
+
const expire = job => { job.cancelled = true; job.bytes = null; clearTimeout(job.timer); jobs.delete(job.id) }
|
|
115
|
+
const touch = (job, time) => {
|
|
116
|
+
job.touched = time
|
|
117
|
+
clearTimeout(job.timer)
|
|
118
|
+
job.timer = setTimeout(() => expire(job), ttlMs)
|
|
119
|
+
job.timer.unref?.()
|
|
120
|
+
}
|
|
89
121
|
const response = (job, offset = 0) => {
|
|
90
122
|
const bytes = job.bytes ?? Buffer.alloc(0), nextOffset = Math.min(bytes.length, offset + CHUNK_BYTES)
|
|
91
123
|
return { id: job.id, status: job.status, filename: job.filename, mime: job.mime, totalBytes: bytes.length, offset, nextOffset, data: bytes.subarray(offset, nextOffset).toString('base64'),
|
|
@@ -94,15 +126,15 @@ export function createBillingExporter(ledger, ctx, { now = Date.now, collect = c
|
|
|
94
126
|
return {
|
|
95
127
|
get(raw = {}) {
|
|
96
128
|
const time = now()
|
|
97
|
-
for (const
|
|
129
|
+
for (const job of jobs.values()) if (time - job.touched >= ttlMs) expire(job)
|
|
98
130
|
if (closed) throw new Error('Export service is closed')
|
|
99
131
|
if (raw.id) {
|
|
100
132
|
const job = jobs.get(raw.id)
|
|
101
133
|
if (!job) throw new Error('Export expired; start again')
|
|
102
|
-
|
|
103
|
-
if (raw.cancel === true) { job.cancelled = true; jobs.delete(raw.id); return response({ ...job, status: 'cancelled', bytes: null }) }
|
|
134
|
+
if (raw.cancel === true) { expire(job); return response({ ...job, status: 'cancelled' }) }
|
|
104
135
|
const offset = raw.offset ?? 0
|
|
105
136
|
if (!Number.isSafeInteger(offset) || offset < 0 || offset > (job.bytes?.length ?? 0)) throw new Error('Invalid export offset')
|
|
137
|
+
touch(job, time)
|
|
106
138
|
return response(job, offset)
|
|
107
139
|
}
|
|
108
140
|
if (jobs.size >= 2) throw new Error('Two exports are already active; close one before starting another')
|
|
@@ -117,7 +149,11 @@ export function createBillingExporter(ledger, ctx, { now = Date.now, collect = c
|
|
|
117
149
|
const job = { id, touched: time, status: 'preparing', cancelled: false, completedSessions: 0, totalSessions: 0, rows: 0, message: '',
|
|
118
150
|
filename: `dsh-cost-${suffix}-${new Date(time).toISOString().replace(/[:.]/g, '-')}.${raw.format === 'json' ? 'json' : 'csv'}`, mime: raw.format === 'json' ? 'application/json;charset=utf-8' : 'text/csv;charset=utf-8' }
|
|
119
151
|
jobs.set(id, job)
|
|
120
|
-
|
|
152
|
+
touch(job, time)
|
|
153
|
+
const active = () => {
|
|
154
|
+
if (!job.cancelled && now() - job.touched >= ttlMs) expire(job)
|
|
155
|
+
return !closed && !job.cancelled && jobs.get(id) === job
|
|
156
|
+
}
|
|
121
157
|
Promise.resolve().then(async () => {
|
|
122
158
|
const data = await collect(frozen, ctx, request, { now: () => time, active, progress: (completed, total) => { job.completedSessions = completed; job.totalSessions = total } })
|
|
123
159
|
if (!active()) return
|
|
@@ -134,6 +170,6 @@ export function createBillingExporter(ledger, ctx, { now = Date.now, collect = c
|
|
|
134
170
|
})
|
|
135
171
|
return response(job)
|
|
136
172
|
},
|
|
137
|
-
dispose() { closed = true; for (const job of jobs.values()) job
|
|
173
|
+
dispose() { closed = true; for (const job of jobs.values()) expire(job) },
|
|
138
174
|
}
|
|
139
175
|
}
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
import { AsyncLocalStorage } from 'node:async_hooks'
|
|
3
3
|
import { randomUUID } from 'node:crypto'
|
|
4
4
|
import { lstat } from 'node:fs/promises'
|
|
5
|
-
import { dirname, join } from 'node:path'
|
|
5
|
+
import { basename, dirname, join } from 'node:path'
|
|
6
6
|
import { billionVersionSupported } from './billion-context-compatibility.js'
|
|
7
7
|
import { installBillionAuto } from './billion-context-auto.js'
|
|
8
8
|
import { journalInfoAsync, archiveUsageJournals, notifyJournalChange } from './journal-storage.js'
|
|
@@ -28,7 +28,7 @@ function stepIdentity(ctx, id) {
|
|
|
28
28
|
export function installBillionContextBilling(ctx, ledger) {
|
|
29
29
|
if (!ledger.path) return null
|
|
30
30
|
const root = billionRoot(ledger.path), scope = new AsyncLocalStorage(), streams = new Map(), groups = new Map(), files = new Map()
|
|
31
|
-
const dirtyStreams = new Map(), retryStreams = new Map(), inventory = createBillionInventory(ledger.path)
|
|
31
|
+
const dirtyStreams = new Map(), retryStreams = new Map(), retryArchives = new Map(), inventory = createBillionInventory(ledger.path)
|
|
32
32
|
let stopped = false, pending = null, rerun = false, peerAt = 0, peerPending, releaseAuto
|
|
33
33
|
const status = { protocol: 1, requests: 0, unreported: 0, partial: 0, uncovered: 0, connected: false, failed: false,
|
|
34
34
|
detected: false, version: '', compatibility: 'unknown', noticeId: '', unresolved: 0 }
|
|
@@ -82,7 +82,7 @@ export function installBillionContextBilling(ctx, ledger) {
|
|
|
82
82
|
}
|
|
83
83
|
const save = stream => {
|
|
84
84
|
if (!stream.groups.length) return
|
|
85
|
-
|
|
85
|
+
const metadata = structuredClone({
|
|
86
86
|
protocol: 1, streamId: stream.streamId, sessionId: stream.sessionId, model: stream.model, provider: stream.provider,
|
|
87
87
|
turn: stream.turn, step: stream.step, startedAtMs: stream.startedAtMs, covered: stream.covered ?? stream.lastAck === true,
|
|
88
88
|
finished: !!stream.finishedAt, policies: stream.policies,
|
|
@@ -90,6 +90,8 @@ export function installBillionContextBilling(ctx, ledger) {
|
|
|
90
90
|
...(stream.reported ? { reported: stream.reported } : {}), sourceSeqs: stream.sourceSeqs,
|
|
91
91
|
...(stream.coverage ? { coverage: stream.coverage, hostSuppressed: stream.hostSuppressed === true } : {}),
|
|
92
92
|
})
|
|
93
|
+
writeBillionJson(billionStreamPath(root, stream.sessionId, stream.streamId), metadata)
|
|
94
|
+
stream.persisted = metadata
|
|
93
95
|
dirtyStreams.set(stream.streamId, { path: billionStreamPath(root, stream.sessionId, stream.streamId), key: billionSessionKey(stream.sessionId), sessionId: stream.sessionId })
|
|
94
96
|
if (stream.finishedAt) {
|
|
95
97
|
notifyJournalChange(root)
|
|
@@ -105,79 +107,101 @@ export function installBillionContextBilling(ctx, ledger) {
|
|
|
105
107
|
do {
|
|
106
108
|
rerun = false
|
|
107
109
|
ledger.refresh()
|
|
108
|
-
let work = 0
|
|
110
|
+
let work = 0, failed = false
|
|
109
111
|
const legacyCoverage = new Map(), archives = new Map()
|
|
110
|
-
const dirty = [...dirtyStreams.values()]; dirtyStreams.clear()
|
|
111
112
|
const found = await inventory.poll(), callFiles = found.calls
|
|
112
113
|
if (stopped) return
|
|
114
|
+
const dirty = [...dirtyStreams.values()]; dirtyStreams.clear()
|
|
113
115
|
const streamFiles = [...new Map([...dirty, ...retryStreams.values(), ...found.streams].map(file => [file.path, file])).values()]
|
|
114
116
|
if (found.retained) {
|
|
115
117
|
for (const path of files.keys()) if (!found.retained.has(path)) files.delete(path)
|
|
116
118
|
for (const path of retryStreams.keys()) if (!found.retained.has(path)) retryStreams.delete(path)
|
|
119
|
+
for (const path of retryArchives.keys()) if (!found.retained.has(path)) retryArchives.delete(path)
|
|
117
120
|
}
|
|
118
|
-
for (const file of found.archives)
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
121
|
+
for (const file of found.archives) retryArchives.set(file.path, file)
|
|
122
|
+
for (const file of retryArchives.values()) {
|
|
123
|
+
if (stopped) return
|
|
124
|
+
try {
|
|
125
|
+
if (!files.has(file.path)) {
|
|
126
|
+
for (const stream of await readBillionStreamArchive(file)) { applyCoverage(stream, stream.coverage); importHost(stream) }
|
|
127
|
+
files.set(file.path, { archived: true })
|
|
128
|
+
}
|
|
129
|
+
retryArchives.delete(file.path)
|
|
130
|
+
} catch (error) { if (error.code === 'ENOENT') retryArchives.delete(file.path); else failed = true }
|
|
122
131
|
}
|
|
123
132
|
// A completed stream can legitimately contain zero proxy call rows.
|
|
124
133
|
// Discover its once-per-stream fallback independently of the call journal.
|
|
125
134
|
for (const file of streamFiles) {
|
|
126
135
|
if (stopped) break
|
|
127
|
-
|
|
128
|
-
try {
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
136
|
+
retryStreams.set(file.path, file)
|
|
137
|
+
try {
|
|
138
|
+
const cached = files.get(file.path)
|
|
139
|
+
const owned = streams.get(basename(file.path, '.json'))
|
|
140
|
+
let stream, info
|
|
141
|
+
if (owned) {
|
|
142
|
+
// Reuse only the successfully saved snapshot. Reading our active
|
|
143
|
+
// file asynchronously can hold it open across a Windows rename.
|
|
144
|
+
stream = owned.persisted
|
|
145
|
+
if (!stream?.finished) continue
|
|
146
|
+
if (cached?.snapshot === stream) { retryStreams.delete(file.path); continue }
|
|
147
|
+
} else {
|
|
148
|
+
try { info = await lstat(file.path) } catch (error) { if (error.code === 'ENOENT') { retryStreams.delete(file.path); continue }; throw error }
|
|
149
|
+
if (!info.isFile() || info.isSymbolicLink()) { retryStreams.delete(file.path); continue }
|
|
150
|
+
if (cached?.size === info.size && cached?.mtime === info.mtimeMs) { retryStreams.delete(file.path); continue }
|
|
151
|
+
stream = await readBillionStreamFileAsync(file)
|
|
152
|
+
}
|
|
153
|
+
if (!stream) { retryStreams.delete(file.path); continue }
|
|
154
|
+
if (!stream?.finished) { retryStreams.set(file.path, file); continue }
|
|
155
|
+
if (!stream.coverage) {
|
|
156
|
+
if (!legacyCoverage.has(stream.sessionId)) legacyCoverage.set(stream.sessionId, await billionCoverageIndex(ledger.path, stream.sessionId))
|
|
157
|
+
applyCoverage(stream, legacyCoverage.get(stream.sessionId).get(stream.streamId) ?? await billionCoverage(ledger.path, stream))
|
|
158
|
+
writeBillionJson(file.path, stream)
|
|
159
|
+
} else applyCoverage(stream, stream.coverage)
|
|
160
|
+
importHost(stream)
|
|
161
|
+
files.set(file.path, owned ? { snapshot: stream } : { size: info.size, mtime: info.mtimeMs })
|
|
162
|
+
retryStreams.delete(file.path)
|
|
163
|
+
} catch { failed = true }
|
|
143
164
|
}
|
|
144
165
|
for (const file of callFiles) {
|
|
145
166
|
if (stopped) break
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
if (!
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
167
|
+
try {
|
|
168
|
+
let info
|
|
169
|
+
try { info = await journalInfoAsync(file.path) } catch (error) { if (error.code === 'ENOENT') { rerun = true; continue }; throw error }
|
|
170
|
+
const cached = files.get(file.path)
|
|
171
|
+
if (cached?.size === info.size && cached?.mtime === info.mtimeMs) continue
|
|
172
|
+
let offset = cached?.offset <= info.size ? cached.offset : 0, blocked = false
|
|
173
|
+
const prepared = new Map()
|
|
174
|
+
for await (const item of iterateBillionCalls(file, offset)) {
|
|
175
|
+
if (stopped) break
|
|
176
|
+
const row = item.row
|
|
177
|
+
const owned = streams.get(row.streamId)
|
|
178
|
+
let stream = prepared.get(row.streamId) ?? (owned ? owned.persisted : await readBillionStreamAsync(ledger.path, row, archives))
|
|
179
|
+
if (!stream || !stream.finished) { blocked = true; break }
|
|
180
|
+
if (!prepared.has(row.streamId)) {
|
|
181
|
+
const originalCoverage = stream.coverage
|
|
182
|
+
if (!originalCoverage && !legacyCoverage.has(stream.sessionId)) legacyCoverage.set(stream.sessionId, await billionCoverageIndex(ledger.path, stream.sessionId))
|
|
183
|
+
applyCoverage(stream, originalCoverage ?? legacyCoverage.get(stream.sessionId).get(stream.streamId))
|
|
184
|
+
if (!originalCoverage) writeBillionJson(billionStreamPath(root, stream.sessionId, stream.streamId), stream)
|
|
185
|
+
importHost(stream)
|
|
186
|
+
prepared.set(row.streamId, stream)
|
|
187
|
+
}
|
|
188
|
+
offset = item.offset
|
|
189
|
+
status.requests++
|
|
190
|
+
if (!row.usage) status.unreported++
|
|
191
|
+
else if (!row.complete) status.partial++
|
|
192
|
+
const meter = { source: row.source + ':' + file.key, seq: row.seq }
|
|
193
|
+
const reset = ledger.historyReset?.at ?? 0
|
|
194
|
+
if ((ledger.usageCursors?.[meter.source] ?? 0) < meter.seq) {
|
|
195
|
+
if (!row.usage || row.startedAtMs <= reset || stream.policies?.[row.groupId] === 'fallback') ledger.advanceUsageCursor(meter)
|
|
196
|
+
else ledger.account(usageBuckets(row.usage), row.model, row.sessionId, row.startedAtMs, billionProvider(row, stream), meter)
|
|
197
|
+
}
|
|
198
|
+
if (++work >= 5000) break
|
|
174
199
|
}
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
if (work >= 5000) break
|
|
200
|
+
files.set(file.path, { offset, size: blocked || offset < info.size && work >= 5000 ? -1 : info.size, mtime: info.mtimeMs })
|
|
201
|
+
if (work >= 5000) break
|
|
202
|
+
} catch { failed = true }
|
|
179
203
|
}
|
|
180
|
-
status.failed =
|
|
204
|
+
status.failed = failed
|
|
181
205
|
} while (rerun && !stopped)
|
|
182
206
|
})().catch(() => { status.failed = true }).finally(() => { pending = null })
|
|
183
207
|
return pending
|
|
@@ -62,9 +62,12 @@ export function readBillionStreamFile(file) {
|
|
|
62
62
|
export async function readBillionStreamFileAsync(file) {
|
|
63
63
|
try {
|
|
64
64
|
const stream = await readSmallBillionJson(file.path)
|
|
65
|
-
return stream
|
|
65
|
+
return stream?.protocol === 1 && uuid(stream.streamId) && validSessionId(stream.sessionId)
|
|
66
66
|
&& billionSessionKey(stream.sessionId) === file.key && file.path.endsWith(stream.streamId + '.json') ? stream : null
|
|
67
|
-
} catch {
|
|
67
|
+
} catch (error) {
|
|
68
|
+
if (error.code === 'ENOENT' || error.code === 'INVALID_BILLION_METADATA' || error instanceof SyntaxError) return null
|
|
69
|
+
throw error
|
|
70
|
+
}
|
|
68
71
|
}
|
|
69
72
|
export function listBillionFiles(ledgerPath, sessionId) {
|
|
70
73
|
if (!ledgerPath) return []
|
|
@@ -16,14 +16,15 @@ export async function safeEntries(path) {
|
|
|
16
16
|
}
|
|
17
17
|
export async function readSmallBillionJson(path) {
|
|
18
18
|
const info = await lstat(path)
|
|
19
|
-
if (!info.isFile() || info.isSymbolicLink() || info.size > 8192) throw Error('Invalid proxy billing metadata')
|
|
19
|
+
if (!info.isFile() || info.isSymbolicLink() || info.size > 8192) throw Object.assign(Error('Invalid proxy billing metadata'), { code: 'INVALID_BILLION_METADATA' })
|
|
20
20
|
return JSON.parse(await readFile(path, 'utf8'))
|
|
21
21
|
}
|
|
22
22
|
export function createBillionInventory(ledgerPath) {
|
|
23
|
-
const root = ledgerPath + '.billion-context'
|
|
23
|
+
const root = ledgerPath + '.billion-context'
|
|
24
|
+
let cache = new Map()
|
|
24
25
|
return {
|
|
25
26
|
async poll() {
|
|
26
|
-
const streams = [], calls = [], archives = [], seen = new Set()
|
|
27
|
+
const streams = [], calls = [], archives = [], seen = new Set(), next = new Map(cache)
|
|
27
28
|
let changed = false
|
|
28
29
|
// Inspect directories, not every completed stream, on an idle tick. Calls
|
|
29
30
|
// remain candidates because append-only writes do not change directory mtime.
|
|
@@ -46,13 +47,15 @@ export function createBillionInventory(ledgerPath) {
|
|
|
46
47
|
const match = kind === 'calls' ? callName.exec(file.name) : (kind === 'streams' ? streamName : archiveName).exec(file.name)
|
|
47
48
|
return match ? [{ path: join(directory, file.name), key: entry.name, ...(kind === 'calls' ? { source: match[1] } : {}) }] : []
|
|
48
49
|
})
|
|
49
|
-
|
|
50
|
+
next.set(directory, { revision, files })
|
|
50
51
|
}
|
|
51
52
|
if (kind === 'calls') for (const file of files) calls.push(file)
|
|
52
53
|
else if (previous?.revision !== revision) for (const file of files) (kind === 'streams' ? streams : archives).push(file)
|
|
53
54
|
}
|
|
54
55
|
}
|
|
55
|
-
for (const directory of
|
|
56
|
+
for (const directory of next.keys()) if (!seen.has(directory)) { next.delete(directory); changed = true }
|
|
57
|
+
// A failed poll must not mark undispatched candidates as already scanned.
|
|
58
|
+
cache = next
|
|
56
59
|
const retained = changed ? new Set([...cache.values()].flatMap(item => item.files.map(file => file.path))) : null
|
|
57
60
|
return { streams, calls, archives, retained }
|
|
58
61
|
},
|