tokenusage-cli 0.4.1 → 0.4.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -3,6 +3,40 @@
3
3
  本包(`tokenusage-cli`)的用户可见变更记录。发版时在顶部新增一节,
4
4
  `## <版本号>` 行会被 release workflow 取作 GitHub Release 正文。
5
5
 
6
+ ## 0.4.3
7
+
8
+ 2026-11-01。两处只有真机首跑才暴露的问题:调度摘要依旧看不出拒绝原因(修的过程中又追出
9
+ 一层 Windows 控制台代码页乱码),`reconcile` 把「还没上传」的桶报成漂移。
10
+
11
+ - **调度降级原因改取子进程输出**:`execFileSync` 的 `error.message` 以命令回显开头
12
+ (`Command failed: schtasks /Create /XML C:\Users\...`),0.4.2 打印第一行时正好把
13
+ 真正的 `ERROR: Access is denied.` 挤出 200 字符窗口。现在优先读 `stderr` / `stdout`,
14
+ 再退回 message,统一折成一行(Windows / macOS / Linux 的注册、启停、卸载错误文案
15
+ 共用这一处)。
16
+ - **Windows 控制台代码页不再乱码**:只取 stderr 后又暴露一层——中文 Windows 的
17
+ `schtasks` 按控制台代码页(936/GBK)写字节,Node 按 UTF-8 解出来是
18
+ `����: �ܾ����ʡ�`。现按 `chcp.com` 报告的活动代码页(932/936/949/950)回退
19
+ `TextDecoder` 重解,解不通才回到原 UTF-8 文本;65001 与单字节页原样直出。
20
+ zh-CN 非提升 shell 下现在的摘要就是可读的 `错误: 拒绝访问。`。
21
+ - **`reconcile` 区分 pending 与 drift**:`queue.state.json` 上传偏移之后的本地桶记入
22
+ `pending`,不计入漂移、不触发退出码 1——真机 `init` + 首跑曾报 5 个「漂移」,实际只是
23
+ 队列尾部尚未上传,一次 `sync` 后全部匹配。`--json` 新增 `upload_offset`、
24
+ `counts.pending` 与 `pending[]`;真实漂移仍照常报出并退出 1。
25
+ - 单测:stderr 优先 / 多行折叠 / 超长截断、GBK 字节解出可读文本与解不通时回退、
26
+ pending 分类、缺 `queue.state.json` 时全部按 pending、以及已上传行的真漂移仍失败
27
+ (`test/scheduled-task.test.js`、`test/reconcile.test.js`)。
28
+
29
+ ## 0.4.2
30
+
31
+ 2026-11-01。`init` 的调度摘要不再只写 `degraded: xml_rejected`:现在把被拒的第一行原因
32
+ (如 `ERROR: Access is denied.`,超过 100 字符截断)一并打印,用户能据此判断是「需要
33
+ 管理员权限」还是「任务定义本身有问题」。
34
+
35
+ - Windows:非提升 shell 下 `schtasks /Create /XML` 必然 `Access is denied`,降级路径照旧
36
+ 每小时采集(隐藏窗口);要启用 catch-up / retry / 无执行时限,需以管理员身份重跑
37
+ `tokenusage init`。
38
+ - 单测覆盖:原因可见、超长命令输出截断(`test/scheduled-task.test.js`)。
39
+
6
40
  ## 0.4.1
7
41
 
8
42
  2026-11-01。修复 `tokenusage reconcile` 在真实队列上的误报:队列行写的是
package/README.md CHANGED
@@ -8,7 +8,8 @@
8
8
 
9
9
  ```bash
10
10
  # 1) 初始化:扫描根配置、各工具用量 hook、并注册每小时自动采集的计划任务
11
- # (Windows 经 VBS 隐藏窗口运行,不弹黑框;--no-task 可跳过任务注册)
11
+ # (Windows 经 VBS 隐藏窗口运行,不弹黑框;--no-task 可跳过任务注册。增强任务定义
12
+ # 需要管理员权限,非提升 shell 下会自动降级为普通每小时触发并提示 degraded)
12
13
  tokenusage init
13
14
 
14
15
  # 2) 配对:输出一个 user_code,浏览器打开提示的链接批准
@@ -34,7 +35,7 @@ tokenusage sync --drain # 首次/积压补传(更大的批处理预算)
34
35
  | `tokenusage device-login` | 设备流配对:输出 `user_code`,在站点 `/device` 页批准后本机拿到设备令牌 |
35
36
  | `tokenusage sync` | 解析本地 AI 工具日志 → 上报小时桶 + 会话元数据 |
36
37
  | `tokenusage upload` | 只刷本地队列,不扫描来源(`--drain` / `--force` 抬批次上限、`--json` 输出结构化结果) |
37
- | `tokenusage reconcile` | 只读对账:本地 `queue.jsonl` 的 last-wins 基线 vs 云端该设备的桶明细,按 `(hour_start, source, model)` 逐键比对,有漂移退出码 1 |
38
+ | `tokenusage reconcile` | 只读对账:本地 `queue.jsonl` 的 last-wins 基线 vs 云端该设备的桶明细,按 `(hour_start, source, model)` 逐键比对,有漂移退出码 1;尚未上传的桶(`queue.state.json` 偏移之后)记为 pending,不算漂移 |
38
39
  | `tokenusage sync --watch` | 常驻模式,默认每 15 分钟增量一轮(`--interval <ms>` 可调,≥60000) |
39
40
  | `tokenusage sync --no-sessions` | 只上报小时桶,跳过会话元数据 |
40
41
  | `tokenusage sync --drain` | 补传模式:更大的批处理预算,适合首次接入/长期积压 |
@@ -73,6 +74,9 @@ tokenusage reconcile --json # 结构化漂移明细,供脚本/看
73
74
  - 漂移分三类:`missing_on_server`(本地有、云端没有,通常是待补传)、`server_only`
74
75
  (云端有、本地已无,队列被裁剪或换过机器身份)、`mismatch`(同键不同值,附逐字段
75
76
  `local → cloud` 差异)。任一存在即以退出码 `1` 结束,便于 cron / CI 判定。
77
+ - `queue.state.json` 偏移之后的本地桶记为 **pending**(尚未上传,云端只可能持有更早的值),
78
+ 单独列出、不计入漂移、不触发退出码 `1`;`tokenusage sync` 传完后重新对账即可比对这些键。
79
+ `--json` 里对应 `upload_offset`、`counts.pending` 与 `pending[]`。
76
80
 
77
81
  ## `sync --watch`:常驻自动上报
78
82
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "tokenusage-cli",
3
- "version": "0.4.1",
3
+ "version": "0.4.3",
4
4
  "description": "Self-hosted AI coding token usage collector for the Token Usage dashboard.",
5
5
  "keywords": [
6
6
  "token-usage",
@@ -14,10 +14,15 @@
14
14
  // queue.state.json (the upload offset) is not touched. Drift is reported per
15
15
  // key and exits 1 so a script can gate on it:
16
16
  //
17
- // missing_on_server local bucket the cloud never stored (pending/blocked upload)
17
+ // missing_on_server local bucket the cloud never stored (blocked upload)
18
18
  // server_only cloud row without a local counterpart (pruned queue, or
19
19
  // a row written under another machine identity)
20
- // mismatch same key, different values (partial upload / legacy repair)
20
+ // mismatch same key, different values (legacy repair / stale row)
21
+ //
22
+ // Keys whose last-wins row sits past the upload offset are reported as
23
+ // `pending` instead: they have never been sent, so the cloud holding the older
24
+ // value is expected, not drift (a fresh `init` + `reconcile` used to exit 1 on
25
+ // exactly those keys until a `sync` flushed them).
21
26
  //
22
27
  // Known residual: ingest dedupes a REPLAYED duplicate inside one request by
23
28
  // taking the larger total_tokens, so a queue that carried the same key twice
@@ -164,7 +169,7 @@ function bucketKey(hourMs, source, model) {
164
169
  // Local baseline: every usage row in queue.jsonl, normalized/sanitized exactly
165
170
  // like the upload path, keyed and filtered on the normalized hour. Control
166
171
  // records (kind-bearing account session states / watermarks) are not buckets.
167
- async function readLocalBaseline({ queuePath, fromMs, toMs }) {
172
+ async function collectBuckets({ queuePath, fromMs, toMs, start = 0 }) {
168
173
  let handle;
169
174
  try {
170
175
  handle = await fs.open(queuePath, "r");
@@ -175,7 +180,8 @@ async function readLocalBaseline({ queuePath, fromMs, toMs }) {
175
180
  const buckets = new Map();
176
181
  try {
177
182
  const before = await handle.stat();
178
- for await (const record of iterateQueueLines(handle, 0, before.size)) {
183
+ const from = Math.min(Math.max(Number(start) || 0, 0), before.size);
184
+ for await (const record of iterateQueueLines(handle, from, before.size)) {
179
185
  // A producer may be mid-write; an incomplete tail is not yet queued data.
180
186
  if (!record.complete) break;
181
187
  const line = record.bytes.toString("utf8");
@@ -208,6 +214,19 @@ async function readLocalBaseline({ queuePath, fromMs, toMs }) {
208
214
  return buckets;
209
215
  }
210
216
 
217
+ function readLocalBaseline({ queuePath, fromMs, toMs }) {
218
+ return collectBuckets({ queuePath, fromMs, toMs });
219
+ }
220
+
221
+ // The queue suffix past queue.state.json's upload offset. Every key found here
222
+ // has its last-wins row not uploaded yet, so comparing it against the cloud is
223
+ // premature: the cloud legitimately holds the older value. Reporting it as
224
+ // drift made the first `init` + `reconcile` on a machine look broken (5 "drifted"
225
+ // keys that a single `sync` fixed), so these keys are listed as pending.
226
+ function readPendingBuckets({ queuePath, offset = 0, fromMs, toMs }) {
227
+ return collectBuckets({ queuePath, fromMs, toMs, start: offset });
228
+ }
229
+
211
230
  async function fetchPage({ url, deviceToken, anonKey, timeoutMs, fetchImpl }) {
212
231
  const headers = { Accept: "application/json", Authorization: `Bearer ${deviceToken}` };
213
232
  if (anonKey) headers.apikey = anonKey;
@@ -317,15 +336,24 @@ function describeRow(row) {
317
336
  return out;
318
337
  }
319
338
 
320
- function compareBuckets({ local, server }) {
339
+ function compareBuckets({ local, server, pending = new Set() }) {
321
340
  const missingOnServer = [];
322
341
  const mismatches = [];
323
342
  const serverOnly = [];
343
+ const pendingUpload = [];
324
344
 
325
345
  for (const [key, localRow] of local) {
326
346
  const hourStart = new Date(Date.parse(localRow.hour_start)).toISOString();
327
347
  const base = { hour_start: hourStart, source: localRow.source, model: localRow.model };
328
348
  const serverRow = server.get(key);
349
+ if (pending.has(key)) {
350
+ pendingUpload.push({
351
+ ...base,
352
+ local: describeRow(localRow),
353
+ server: serverRow ? describeRow(serverRow) : null,
354
+ });
355
+ continue;
356
+ }
329
357
  if (!serverRow) {
330
358
  missingOnServer.push({ ...base, local: describeRow(localRow), server: null });
331
359
  continue;
@@ -366,10 +394,12 @@ function compareBuckets({ local, server }) {
366
394
  missing_on_server: sortEntries(missingOnServer),
367
395
  server_only: sortEntries(serverOnly),
368
396
  mismatch: sortEntries(mismatches),
397
+ pending: sortEntries(pendingUpload),
369
398
  counts: {
370
399
  missing_on_server: missingOnServer.length,
371
400
  server_only: serverOnly.length,
372
401
  mismatch: mismatches.length,
402
+ pending: pendingUpload.length,
373
403
  },
374
404
  drifted: missingOnServer.length + serverOnly.length + mismatches.length,
375
405
  };
@@ -388,17 +418,31 @@ function describeEntry(entry, isMismatch = false) {
388
418
  }
389
419
 
390
420
  function renderText(summary) {
421
+ const pendingCount = summary.counts?.pending ?? 0;
391
422
  const lines = [
392
423
  `Reconcile window (UTC, hour_start in [from, to)):`,
393
424
  ` from ${summary.from}`,
394
425
  ` to ${summary.to}`,
395
426
  summary.clamped ? " (from clamped to the server's 366-day ceiling)" : null,
396
427
  `Local baseline: ${summary.local_buckets.toLocaleString("en-US")} bucket(s) from queue.jsonl (last-wins)`,
428
+ pendingCount > 0
429
+ ? ` ${pendingCount.toLocaleString("en-US")} of them are past the upload offset (queue.state.json) and not compared yet`
430
+ : null,
397
431
  `Cloud device: ${summary.device_id || "unknown"} — ${summary.cloud_buckets.toLocaleString("en-US")} row(s) in ${summary.pages} page(s)`,
398
432
  ].filter(Boolean);
399
433
 
400
434
  if (summary.drifted === 0) {
401
- lines.push("Result: OK — every local bucket matches the cloud row.");
435
+ lines.push(
436
+ pendingCount > 0
437
+ ? `Result: OK — every uploaded bucket matches the cloud row (${pendingCount.toLocaleString("en-US")} still pending upload).`
438
+ : "Result: OK — every local bucket matches the cloud row.",
439
+ );
440
+ if (pendingCount > 0) {
441
+ lines.push(
442
+ "Pending buckets are local rows `sync` has not uploaded yet, so the cloud cannot",
443
+ "hold them; run `tokenusage sync` and re-run this command to compare them.",
444
+ );
445
+ }
402
446
  return `${lines.join("\n")}\n`;
403
447
  }
404
448
 
@@ -422,6 +466,11 @@ function renderText(summary) {
422
466
  lines.push(` … and ${entries.length - TEXT_DRIFT_LIMIT} more (use --json for the full list)`);
423
467
  }
424
468
  }
469
+ if (pendingCount > 0) {
470
+ lines.push(
471
+ `Note: ${pendingCount.toLocaleString("en-US")} local key(s) are still pending upload and were left out of the drift count.`,
472
+ );
473
+ }
425
474
  lines.push(
426
475
  "Hint: `tokenusage upload` pushes pending local buckets; `server_only` keys mean the",
427
476
  "cloud holds rows this queue no longer contains. This command never writes.",
@@ -453,6 +502,16 @@ async function cmdReconcile(argv = [], options = {}) {
453
502
  }
454
503
 
455
504
  const local = await readLocalBaseline({ queuePath, fromMs: window.fromMs, toMs: window.toMs });
505
+ // Rows past queue.state.json's offset have not been uploaded: their keys are
506
+ // pending (not drift), because the cloud can only hold the previous value.
507
+ const queueState = (await readJson(path.join(trackerDir, "queue.state.json"))) || {};
508
+ const uploadOffset = Number.isFinite(Number(queueState.offset)) ? Math.max(0, Number(queueState.offset)) : 0;
509
+ const pending = await readPendingBuckets({
510
+ queuePath,
511
+ offset: uploadOffset,
512
+ fromMs: window.fromMs,
513
+ toMs: window.toMs,
514
+ });
456
515
  const { buckets: server, deviceId, pages } = await fetchServerBuckets({
457
516
  baseUrl: runtime.baseUrl,
458
517
  anonKey: runtime.anonKey,
@@ -463,7 +522,7 @@ async function cmdReconcile(argv = [], options = {}) {
463
522
  timeoutMs: runtime.httpTimeoutMs,
464
523
  fetchImpl,
465
524
  });
466
- const compared = compareBuckets({ local, server });
525
+ const compared = compareBuckets({ local, server, pending: new Set(pending.keys()) });
467
526
 
468
527
  const summary = {
469
528
  status: compared.drifted === 0 ? "ok" : "drift",
@@ -474,6 +533,7 @@ async function cmdReconcile(argv = [], options = {}) {
474
533
  local_buckets: local.size,
475
534
  cloud_buckets: server.size,
476
535
  pages,
536
+ upload_offset: uploadOffset,
477
537
  counts: compared.counts,
478
538
  drifted: compared.drifted,
479
539
  drift: {
@@ -481,6 +541,7 @@ async function cmdReconcile(argv = [], options = {}) {
481
541
  server_only: compared.server_only,
482
542
  mismatch: compared.mismatch,
483
543
  },
544
+ pending: compared.pending,
484
545
  };
485
546
 
486
547
  // Output seam: tests render without hijacking process.stdout (which would
@@ -493,4 +554,12 @@ async function cmdReconcile(argv = [], options = {}) {
493
554
  return summary;
494
555
  }
495
556
 
496
- module.exports = { cmdReconcile, parseArgs, resolveWindow, readLocalBaseline, compareBuckets, RECONCILE_SLUG };
557
+ module.exports = {
558
+ cmdReconcile,
559
+ parseArgs,
560
+ resolveWindow,
561
+ readLocalBaseline,
562
+ readPendingBuckets,
563
+ compareBuckets,
564
+ RECONCILE_SLUG,
565
+ };
@@ -375,8 +375,79 @@ function run(command, args) {
375
375
  execFileSync(command, args, { stdio: "pipe", windowsHide: true });
376
376
  }
377
377
 
378
- function errorText(error) {
379
- return String(error?.message || error).slice(0, 200);
378
+ // `stdio: "pipe"` keeps the child's own output on `error.stderr` / `stdout`,
379
+ // while `error.message` starts with a command echo ("Command failed: schtasks
380
+ // /Create /XML C:\Users\me\..."). Reading the message first buried the real
381
+ // reason — the non-elevated Windows rejection "ERROR: Access is denied."
382
+ // lands on the *second* line and was clipped away. Child output wins; the
383
+ // command echo is the last resort.
384
+ //
385
+ // Windows console tools also write their message in the console codepage, not
386
+ // UTF-8: a zh-CN "schtasks" prints "错误: 拒绝访问。" as GBK bytes, which decode
387
+ // to replacement characters and reached the user as mojibake. decodeStream()
388
+ // retries with the active codepage before giving up on the original text.
389
+ const CONSOLE_CODEPAGE_DECODERS = {
390
+ 932: "shift_jis",
391
+ 936: "gbk",
392
+ 949: "euc-kr",
393
+ 950: "big5",
394
+ };
395
+
396
+ // UTF-8 itself (65001) and the legacy single-byte pages need no decoder here.
397
+ function consoleEncoding(codepage) {
398
+ return CONSOLE_CODEPAGE_DECODERS[codepage] || null;
399
+ }
400
+
401
+ let cachedConsoleDecoder;
402
+
403
+ function consoleDecoder() {
404
+ if (cachedConsoleDecoder !== undefined) return cachedConsoleDecoder;
405
+ cachedConsoleDecoder = null;
406
+ if (process.platform !== "win32") return null;
407
+ try {
408
+ // "Active code page: 936" — digits survive any localisation.
409
+ const output = execFileSync("chcp.com", [], { stdio: "pipe", windowsHide: true })
410
+ .toString("latin1");
411
+ const codepage = Number((output.match(/\d{3,5}/) || [])[0]);
412
+ const encoding = consoleEncoding(codepage);
413
+ if (encoding) cachedConsoleDecoder = new TextDecoder(encoding);
414
+ } catch (_error) {
415
+ cachedConsoleDecoder = null;
416
+ }
417
+ return cachedConsoleDecoder;
418
+ }
419
+
420
+ function decodeStream(value, fallbackDecoder) {
421
+ if (value == null) return "";
422
+ if (typeof value === "string") return value;
423
+ const bytes = Buffer.isBuffer(value) ? value : Buffer.from(value);
424
+ const utf8 = bytes.toString("utf8");
425
+ if (!utf8.includes("\uFFFD")) return utf8;
426
+ const decoder = fallbackDecoder || consoleDecoder();
427
+ if (!decoder) return utf8;
428
+ const decoded = decoder.decode(bytes);
429
+ return decoded.includes("\uFFFD") ? utf8 : decoded;
430
+ }
431
+
432
+ function firstLine(value) {
433
+ return (
434
+ String(value ?? "")
435
+ .split(/\r?\n/)
436
+ .map((part) => part.trim())
437
+ .find(Boolean) ?? ""
438
+ );
439
+ }
440
+
441
+ function clipLine(line, max) {
442
+ return line.length > max ? `${line.slice(0, max - 1)}…` : line;
443
+ }
444
+
445
+ function errorText(error, max = 200) {
446
+ for (const stream of [error?.stderr, error?.stdout]) {
447
+ const line = firstLine(decodeStream(stream));
448
+ if (line) return clipLine(line, max);
449
+ }
450
+ return clipLine(firstLine(decodeStream(error?.message)), max);
380
451
  }
381
452
 
382
453
  /**
@@ -637,12 +708,23 @@ function deleteAutoSyncSchedule({ home = os.homedir(), binDir, platform = proces
637
708
  }
638
709
 
639
710
  // One place for the init summary copy so Windows/macOS/Linux cannot drift.
711
+ //
712
+ // Degraded registrations carry the raw rejection in `detail`. On a normal
713
+ // (non-elevated) Windows install `schtasks /Create /XML` always fails with
714
+ // "Access is denied", so showing that first line lets the user tell "re-run as
715
+ // administrator" apart from a genuinely malformed definition. Clip it to one
716
+ // short line so the summary table stays readable.
717
+ function scheduleDetailLine(value, max = 100) {
718
+ return clipLine(firstLine(value), max);
719
+ }
720
+
640
721
  function formatScheduleRegistration(outcome) {
641
722
  if (!outcome) return "not registered";
642
723
  if (outcome.registered) {
643
724
  const when = "hourly, hidden window";
644
725
  if (outcome.degraded) {
645
- return `${outcome.label}: ${when} (degraded: ${outcome.reason}; no catch-up/retry settings)`;
726
+ const why = scheduleDetailLine(outcome.detail);
727
+ return `${outcome.label}: ${when} (degraded: ${outcome.reason}${why ? ` — ${why}` : ""}; no catch-up/retry settings)`;
646
728
  }
647
729
  return `${outcome.label}: ${when} (pause/resume toggles it)`;
648
730
  }
@@ -709,4 +791,7 @@ module.exports = {
709
791
  formatScheduleRegistration,
710
792
  formatScheduleToggle,
711
793
  formatScheduleDelete,
794
+ consoleEncoding,
795
+ decodeStream,
796
+ errorText,
712
797
  };