@vimoxshah/tokenflow 1.1.0 → 1.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +84 -0
- package/README.md +50 -1
- package/bin/tokenflow.js +222 -9
- package/package.json +2 -1
- package/src/analytics/anomalies.js +17 -0
- package/src/commands/diagnostics.js +14 -1
- package/src/core/live-status.js +14 -3
- package/src/core/store.js +51 -0
- package/src/core/sync.js +9 -1
- package/src/core/team.js +162 -0
- package/src/core/watch-agent.js +226 -0
- package/src/core/watch-lock.js +160 -0
- package/src/core/watch.js +29 -40
- package/src/providers/hermes/index.js +41 -4
- package/src/server/server.js +29 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to TokenFlow are recorded here. Versions follow
|
|
4
|
+
[semantic versioning](https://semver.org/spec/v2.0.0.html).
|
|
5
|
+
|
|
6
|
+
## 1.1.1 — 2026-09-03
|
|
7
|
+
|
|
8
|
+
A reliability release. Five defects had combined to leave the app paused, the
|
|
9
|
+
dashboard unreachable, the stop button inert, one day's token total inflated by
|
|
10
|
+
37 billion — and the alert that would have caught it ranked out of sight.
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
|
|
14
|
+
- **The watcher refused to start after a reboot.** The lock file held a bare
|
|
15
|
+
PID, and PID numbers restart and get reused at every boot, so a lock that
|
|
16
|
+
outlived a restart kept naming a live process — just not ours. One left at
|
|
17
|
+
pid 810 was inherited by `mobilerepaird`; `kill(810, 0)` went on succeeding
|
|
18
|
+
and every `tokenflow watch` refused to start, from the launch agent and the
|
|
19
|
+
menu bar's play button alike. The data went stale behind a phantom. The lock
|
|
20
|
+
now records the boot its PID was issued by, so a pidfile from an earlier boot
|
|
21
|
+
is stale by construction. A watcher that fails to start now reports why
|
|
22
|
+
instead of failing silently.
|
|
23
|
+
- **The Dashboard button opened a closed port.** It opened
|
|
24
|
+
`http://127.0.0.1:<port>` whether or not anything was serving. It now starts
|
|
25
|
+
the server when none is running, shows progress while the data bundle builds,
|
|
26
|
+
and reports a failure. `tokenflow dashboard` also binds the port from
|
|
27
|
+
`config.yaml` rather than a hardcoded default, and a second invocation opens
|
|
28
|
+
the window instead of failing on a busy port.
|
|
29
|
+
- **The stop button never stopped anything.** A `KeepAlive: true` launch agent
|
|
30
|
+
restarts the job after *any* exit, including the clean one a deliberate stop
|
|
31
|
+
produces — measured at about two seconds. The supported agent uses
|
|
32
|
+
`KeepAlive: { SuccessfulExit: false }`: a crash comes back, a stop stays
|
|
33
|
+
stopped.
|
|
34
|
+
- **The Hermes adapter invented usage that never happened.**
|
|
35
|
+
`session_model_usage` is keyed on six columns and the adapter's bookkeeping
|
|
36
|
+
key used four, omitting `billing_base_url` and `billing_mode`. Two real rows
|
|
37
|
+
differing only in billing mode shared one entry, each computed its delta
|
|
38
|
+
against the other's totals, and every refresh cycle re-emitted the difference
|
|
39
|
+
under a fresh id. Five colliding sessions turned one day into 39.9B tokens —
|
|
40
|
+
a figure that grew with the number of refresh cycles rather than with usage.
|
|
41
|
+
The key is now the table's whole primary key, and the tail is a high-water
|
|
42
|
+
mark, so a total that comes back lower can never manufacture usage.
|
|
43
|
+
- **The alert that caught it was buried.** That corruption was detected the day
|
|
44
|
+
it began: 240× the 60-day median, a modified z-score of 170.9, severity high.
|
|
45
|
+
Severity saturates at "high" around z=6 and the list then sorted by date, so
|
|
46
|
+
the outlier ranked third behind two request spikes of z=6.5 and z=11.2 from
|
|
47
|
+
later in the week — and the menu bar shows the top two. Anomalies of a
|
|
48
|
+
different order now outrank recency; ordinary alerts still read newest-first.
|
|
49
|
+
|
|
50
|
+
### Added
|
|
51
|
+
|
|
52
|
+
- `tokenflow watch --install-agent` — keeps the watcher running across reboots,
|
|
53
|
+
supervised by launchd. `--uninstall-agent` removes it. Installing replaces any
|
|
54
|
+
other agent that runs a watcher, because two of them fight over the same lock
|
|
55
|
+
for ever. `tokenflow watch --status` now reports the agent's state and warns
|
|
56
|
+
about conflicts.
|
|
57
|
+
- `tokenflow reset --source <id> --yes` — forget one source and re-read it from
|
|
58
|
+
scratch on the next refresh, leaving every other source untouched. This is the
|
|
59
|
+
repair path for a store holding data from an adapter that has since been
|
|
60
|
+
fixed.
|
|
61
|
+
|
|
62
|
+
### Changed
|
|
63
|
+
|
|
64
|
+
- `tokenflow setup` now installs the watcher agent on macOS, so live data works
|
|
65
|
+
without hand-rolling a LaunchAgent. `--no-agent` opts out, and the install is
|
|
66
|
+
announced rather than silent. On other platforms it prints the systemd/cron
|
|
67
|
+
equivalent instead.
|
|
68
|
+
- A dead watcher no longer leaves its identity in the status file, so a paused
|
|
69
|
+
TokenFlow cannot report itself as live.
|
|
70
|
+
|
|
71
|
+
### Repairing an affected store
|
|
72
|
+
|
|
73
|
+
A store that ingested the inflated Hermes records keeps them until it is told
|
|
74
|
+
to re-read the source:
|
|
75
|
+
|
|
76
|
+
```sh
|
|
77
|
+
tokenflow reset --source hermes --yes
|
|
78
|
+
tokenflow refresh
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
The Hermes database still holds the truth, so nothing is lost. On the corpus
|
|
82
|
+
this was found on, the re-read reconciled exactly — 1,568 source rows to 1,568
|
|
83
|
+
records, every token field matching — and the affected day fell from 39.9B to
|
|
84
|
+
2.35B, with usage returning to the days it actually happened on.
|
package/README.md
CHANGED
|
@@ -25,6 +25,16 @@
|
|
|
25
25
|
<p align="center"><em>The native macOS menu bar popover — live cost, tokens, per-provider,
|
|
26
26
|
per-source and per-model usage, capacity meters and forecast, in light and dark.</em></p>
|
|
27
27
|
|
|
28
|
+
<p align="center">
|
|
29
|
+
🌐 <a href="https://vimoxshah.github.io/tokenflow/"><strong>Landing page</strong></a>
|
|
30
|
+
·
|
|
31
|
+
🖥️ <a href="https://vimoxshah.github.io/tokenflow/demo/"><strong>Live demo</strong></a> <em>(synthetic data)</em>
|
|
32
|
+
·
|
|
33
|
+
📦 <a href="https://www.npmjs.com/package/@vimoxshah/tokenflow"><code>@vimoxshah/tokenflow</code> on npm</a>
|
|
34
|
+
·
|
|
35
|
+
🍺 <code>brew install --cask tokenflow</code> (via <a href="https://github.com/vimoxshah/tokenflow">this tap</a>)
|
|
36
|
+
</p>
|
|
37
|
+
|
|
28
38
|
**How deep it goes**
|
|
29
39
|
|
|
30
40
|
| Layer | What's inside |
|
|
@@ -38,9 +48,44 @@ per-source and per-model usage, capacity meters and forecast, in light and dark.
|
|
|
38
48
|
|
|
39
49
|
Zero runtime dependencies. Nothing leaves your machine. No API keys, no accounts, no telemetry.
|
|
40
50
|
|
|
51
|
+
## Uninstall
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
# macOS app
|
|
55
|
+
brew uninstall --cask tokenflow # if installed via the tap
|
|
56
|
+
# otherwise: quit TokenFlow, drag /Applications/TokenFlow.app to Trash
|
|
57
|
+
|
|
58
|
+
# CLI / npm
|
|
59
|
+
npm uninstall -g @vimoxshah/tokenflow # if installed globally
|
|
60
|
+
|
|
61
|
+
# data (everything local: config, records, digests, state)
|
|
62
|
+
rm -rf ~/.tokenflow
|
|
63
|
+
|
|
64
|
+
# optional extras you may have installed
|
|
65
|
+
launchctl unload ~/Library/LaunchAgents/app.tokenflow.digest.plist 2>/dev/null
|
|
66
|
+
rm -f ~/Library/LaunchAgents/app.tokenflow.digest.plist
|
|
67
|
+
launchctl unload ~/Library/LaunchAgents/app.tokenflow.bar.plist 2>/dev/null
|
|
68
|
+
rm -f ~/Library/LaunchAgents/app.tokenflow.bar.plist
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Nothing is stored anywhere else. Deleting `~/.tokenflow` and the app removes every trace.
|
|
72
|
+
|
|
41
73
|
### Install
|
|
42
74
|
|
|
43
|
-
**macOS
|
|
75
|
+
**Homebrew (macOS)** — the one-liner:
|
|
76
|
+
|
|
77
|
+
```bash
|
|
78
|
+
brew tap vimoxshah/tap https://github.com/vimoxshah/tokenflow
|
|
79
|
+
brew install --cask tokenflow
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
**npm (macOS, Linux, Windows)** — CLI + dashboard, no app bundle:
|
|
83
|
+
|
|
84
|
+
```bash
|
|
85
|
+
npx @vimoxshah/tokenflow@latest setup
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
**macOS app (manual)** — download `TokenFlow-*.dmg` from the
|
|
44
89
|
[**latest release**](https://github.com/vimoxshah/tokenflow/releases/latest) (each release also
|
|
45
90
|
carries `tokenflow-dashboard-demo.html`, an offline demo dashboard that opens in any browser).
|
|
46
91
|
Open the DMG, drag **TokenFlow.app** to Applications, launch from Launchpad.
|
|
@@ -156,6 +201,7 @@ node bin/tokenflow.js models-compare # cost/usage efficiency per model — your
|
|
|
156
201
|
node bin/tokenflow.js budget --set 200 # monthly cap + forecast alerts (fires once per state/month)
|
|
157
202
|
node bin/tokenflow.js schedule --install --at "Monday 09:00" # weekly digest via launchd
|
|
158
203
|
node bin/tokenflow.js diagnostics # local observability — nothing transmitted
|
|
204
|
+
node bin/tokenflow.js team # per-developer team view (needs sync + opt-in names)
|
|
159
205
|
node bin/tokenflow.js up # refresh → rebuild offline HTML → serve + open
|
|
160
206
|
|
|
161
207
|
npm link # optional: global `tokenflow` command
|
|
@@ -237,6 +283,9 @@ discard the rest. No telemetry. Two features can touch the network, both strictl
|
|
|
237
283
|
- **Multi-machine sync** (`sync:` in config) — exchanges daily totals (date, tokens, requests,
|
|
238
284
|
est. cost) with a folder you own (iCloud/Dropbox/Syncthing). Never prompts, code, credentials.
|
|
239
285
|
Default is OFF; nothing leaves this machine until you set `sync.enabled: true`.
|
|
286
|
+
- **Team view** (`tokenflow team`) — per-developer usage from that same shared folder. A name
|
|
287
|
+
appears next to a machine only if its owner set `sync.developerName` themselves; machines
|
|
288
|
+
without it stay anonymous and are excluded from per-person rows.
|
|
240
289
|
- **Prompt analytics** (`promptAnalytics:` in config) — OFF by default; even when enabled, only
|
|
241
290
|
one-way prompt hashes and keyword categories are stored. Raw text requires a separate opt-in.
|
|
242
291
|
- **Map location** (`map.showMyLocation: true`) — one cached IP geolocation of this machine to
|
package/bin/tokenflow.js
CHANGED
|
@@ -29,7 +29,7 @@ import {
|
|
|
29
29
|
buildLiveStatus, currentStatus, readLiveStatus, withComputedFreshness, barLine,
|
|
30
30
|
} from '../src/core/live-status.js';
|
|
31
31
|
import {
|
|
32
|
-
startWatch, runCycle, stopWatch, watchIsRunning, releaseWatchLock,
|
|
32
|
+
startWatch, runCycle, stopWatch, watchIsRunning, releaseWatchLock, readLock,
|
|
33
33
|
} from '../src/core/watch.js';
|
|
34
34
|
import { renderXbar, installSwiftBarPlugin } from '../src/export/menubar.js';
|
|
35
35
|
|
|
@@ -85,6 +85,7 @@ async function main() {
|
|
|
85
85
|
case 'sync': return cmdSync();
|
|
86
86
|
case 'models-compare': return cmdModelsCompare();
|
|
87
87
|
case 'diagnostics': return cmdDiagnostics();
|
|
88
|
+
case 'team': return cmdTeam();
|
|
88
89
|
default:
|
|
89
90
|
console.error(`${C.red}Unknown command "${cmd}".${C.r}\n`);
|
|
90
91
|
return help(1);
|
|
@@ -125,10 +126,38 @@ async function cmdSetup() {
|
|
|
125
126
|
|
|
126
127
|
console.log(`\n${C.g}✓${C.r} wrote ${file}`);
|
|
127
128
|
console.log(` enabled providers: ${detected.length ? detected.join(', ') : '(none)'}`);
|
|
128
|
-
console.log(` timezone: ${cfg.timezone}
|
|
129
|
+
console.log(` timezone: ${cfg.timezone}`);
|
|
130
|
+
await setupWatchAgent();
|
|
129
131
|
console.log(`Next: ${C.c}tokenflow refresh${C.r} then ${C.c}tokenflow dashboard${C.r}\n`);
|
|
130
132
|
}
|
|
131
133
|
|
|
134
|
+
/**
|
|
135
|
+
* Set up the watcher to run by default, because "live" is the point.
|
|
136
|
+
*
|
|
137
|
+
* Installing a login agent is a real side effect, so it is announced and
|
|
138
|
+
* `--no-agent` opts out. On anything other than macOS it prints the hint and
|
|
139
|
+
* does nothing.
|
|
140
|
+
*/
|
|
141
|
+
async function setupWatchAgent() {
|
|
142
|
+
if (flags.agent === false || flags['no-agent'] === true) {
|
|
143
|
+
console.log(` ${C.dim}watcher agent: skipped (--no-agent). Install later: tokenflow watch --install-agent${C.r}\n`);
|
|
144
|
+
return;
|
|
145
|
+
}
|
|
146
|
+
const agent = await import('../src/core/watch-agent.js');
|
|
147
|
+
if (!agent.supported()) {
|
|
148
|
+
console.log(` ${C.dim}watcher: run 'tokenflow watch' from a systemd --user unit, or 'watch --once' from cron${C.r}\n`);
|
|
149
|
+
return;
|
|
150
|
+
}
|
|
151
|
+
try {
|
|
152
|
+
const r = agent.install();
|
|
153
|
+
for (const f of r.removed) console.log(` ${C.y}!${C.r} removed a conflicting watcher agent: ${f.label}`);
|
|
154
|
+
console.log(`${C.g}✓${C.r} the watcher now starts at login ${C.dim}(tokenflow watch --uninstall-agent removes it)${C.r}\n`);
|
|
155
|
+
} catch (err) {
|
|
156
|
+
console.log(` ${C.y}! could not install the watcher agent: ${err.message}${C.r}`);
|
|
157
|
+
console.log(` ${C.dim}run it by hand with 'tokenflow watch', or retry with 'tokenflow watch --install-agent'${C.r}\n`);
|
|
158
|
+
}
|
|
159
|
+
}
|
|
160
|
+
|
|
132
161
|
// ================================================================ providers ==
|
|
133
162
|
|
|
134
163
|
async function cmdProviders() {
|
|
@@ -281,16 +310,43 @@ async function cmdStatus() {
|
|
|
281
310
|
// ================================================================ dashboard ==
|
|
282
311
|
|
|
283
312
|
async function cmdDashboard() {
|
|
284
|
-
const
|
|
313
|
+
const cfg = loadConfig();
|
|
314
|
+
// The configured port is the one every other surface opens — the menu bar
|
|
315
|
+
// reads it straight out of config.yaml — so the server must bind THAT port
|
|
316
|
+
// rather than a hardcoded default the rest of the app has never heard of.
|
|
317
|
+
const port = Number(flags.port) || Number(cfg.ui?.port) || 7799;
|
|
285
318
|
const host = flags.host || '127.0.0.1';
|
|
319
|
+
const wantOpen = flags.open !== false && flags['no-open'] !== true;
|
|
320
|
+
|
|
321
|
+
// Already serving? Then this invocation is a second Dashboard click, not a
|
|
322
|
+
// failure. Open the window and leave the running server alone.
|
|
323
|
+
const { pingServer } = await import('../src/server/server.js');
|
|
324
|
+
const alive = await pingServer({ host, port });
|
|
325
|
+
if (alive) {
|
|
326
|
+
const url = `http://${host}:${port}`;
|
|
327
|
+
console.log(`\n ${C.b}Tokenflow${C.r} ${C.dim}already serving${C.r}`);
|
|
328
|
+
console.log(` ${C.c}${url}${C.r}`);
|
|
329
|
+
console.log(` ${C.dim}${int(alive.records)} records · started elsewhere · loopback only${C.r}\n`);
|
|
330
|
+
if (wantOpen) tryOpen(url);
|
|
331
|
+
return;
|
|
332
|
+
}
|
|
333
|
+
|
|
286
334
|
const b = buildBundle();
|
|
287
|
-
|
|
335
|
+
let s;
|
|
336
|
+
try {
|
|
337
|
+
s = await startServer({ port, host, token: flags.token === false ? false : undefined });
|
|
338
|
+
} catch (err) {
|
|
339
|
+
if (err.code !== 'EADDRINUSE') throw err;
|
|
340
|
+
throw Object.assign(new Error(`port ${port} is busy, and whatever holds it is not TokenFlow`), {
|
|
341
|
+
hint: `Free it, or pick another port: tokenflow dashboard --port ${port + 1}`,
|
|
342
|
+
});
|
|
343
|
+
}
|
|
288
344
|
console.log(`\n ${C.b}Tokenflow${C.r}`);
|
|
289
345
|
console.log(` ${C.c}${s.url}${C.r}`);
|
|
290
346
|
console.log(` ${C.dim}${int(b.health.records)} records · ${b.health.coverage.from ? `${shortDate(b.health.coverage.from)} → ${shortDate(b.health.coverage.to)}` : 'no data'} · loopback only, nothing leaves this machine${C.r}`);
|
|
291
347
|
if (!b.health.records) console.log(` ${C.y}No data yet — click ↻ Refresh in the dashboard, or run 'tokenflow refresh'.${C.r}`);
|
|
292
348
|
console.log(` ${C.dim}Ctrl+C to stop${C.r}\n`);
|
|
293
|
-
if (
|
|
349
|
+
if (wantOpen) tryOpen(s.url);
|
|
294
350
|
await new Promise(() => {});
|
|
295
351
|
}
|
|
296
352
|
|
|
@@ -611,6 +667,17 @@ function safeReaddir(d) {
|
|
|
611
667
|
// ===================================================================== demo ==
|
|
612
668
|
|
|
613
669
|
async function cmdDemo() {
|
|
670
|
+
// Synthetic data must never mix into a real store: `demo` used to overwrite
|
|
671
|
+
// ~/.tokenflow/config.yaml with providers:['mock'] and ingest demo records
|
|
672
|
+
// into whatever home was active. An explicit $TOKENFLOW_HOME wins (scripted
|
|
673
|
+
// setups that want exactly that); otherwise a throwaway home under tmpdir()
|
|
674
|
+
// is created for this run, leaving the default store untouched.
|
|
675
|
+
if (!process.env.TOKENFLOW_HOME) {
|
|
676
|
+
const sandbox = fs.mkdtempSync(path.join(os.tmpdir(), 'tokenflow-demo-'));
|
|
677
|
+
process.env.TOKENFLOW_HOME = sandbox;
|
|
678
|
+
console.log(`\n ${C.dim}sandboxed demo store: ${sandbox}`);
|
|
679
|
+
console.log(` reopen later with: TOKENFLOW_HOME=${sandbox} tokenflow dashboard${C.r}`);
|
|
680
|
+
}
|
|
614
681
|
process.env.TOKENFLOW_DEMO = '1';
|
|
615
682
|
const cfg = loadConfig();
|
|
616
683
|
cfg.providers = ['mock'];
|
|
@@ -757,8 +824,10 @@ async function cmdRestore() {
|
|
|
757
824
|
}
|
|
758
825
|
|
|
759
826
|
async function cmdReset() {
|
|
827
|
+
const only = typeof flags.source === 'string' ? flags.source : null;
|
|
828
|
+
if (only) return resetOneSource(only);
|
|
760
829
|
if (!flags.yes) {
|
|
761
|
-
throw Object.assign(new Error('this deletes all ingested data'), { hint: `re-run with --yes to confirm. Config and pricing are kept. Data home: ${paths().root}` });
|
|
830
|
+
throw Object.assign(new Error('this deletes all ingested data'), { hint: `re-run with --yes to confirm. Config and pricing are kept. Data home: ${paths().root}\n To re-ingest a single source instead: tokenflow reset --source <id> --yes` });
|
|
762
831
|
}
|
|
763
832
|
const p = paths();
|
|
764
833
|
fs.rmSync(p.data, { recursive: true, force: true });
|
|
@@ -766,6 +835,48 @@ async function cmdReset() {
|
|
|
766
835
|
console.log(`${C.g}✓${C.r} cleared ${p.data} (config and pricing kept)`);
|
|
767
836
|
}
|
|
768
837
|
|
|
838
|
+
/**
|
|
839
|
+
* Forget one source and let the next refresh re-read it from scratch.
|
|
840
|
+
*
|
|
841
|
+
* This is the repair path for an adapter bug: the source's own logs or
|
|
842
|
+
* database still hold the truth, so dropping what was ingested from it and
|
|
843
|
+
* clearing its cursor re-derives that source correctly while every other
|
|
844
|
+
* source's records stay exactly as they are.
|
|
845
|
+
*/
|
|
846
|
+
async function resetOneSource(id) {
|
|
847
|
+
const store = new Store();
|
|
848
|
+
const known = Object.keys(store.state.sources || {});
|
|
849
|
+
const held = store.state.sources?.[id]?.records ?? null;
|
|
850
|
+
if (!known.includes(id)) {
|
|
851
|
+
throw Object.assign(new Error(`no ingested source called '${id}'`), {
|
|
852
|
+
hint: known.length ? `the store holds: ${known.sort().join(', ')}` : 'the store is empty — nothing to reset',
|
|
853
|
+
});
|
|
854
|
+
}
|
|
855
|
+
if (!flags.yes) {
|
|
856
|
+
throw Object.assign(new Error(`this drops every record ingested from '${id}'${held ? ` (${int(held)} at last count)` : ''}`), {
|
|
857
|
+
hint: `The next refresh re-reads ${id} from its own source, so nothing is lost that the source still knows. Re-run with --yes to confirm.`,
|
|
858
|
+
});
|
|
859
|
+
}
|
|
860
|
+
const { dropSourceRecords } = await import('../src/core/store.js');
|
|
861
|
+
const { rebuildAggregates } = await import('../src/core/ingest.js');
|
|
862
|
+
|
|
863
|
+
const res = dropSourceRecords(store, id);
|
|
864
|
+
delete store.state.sources[id];
|
|
865
|
+
const rb = rebuildAggregates(store);
|
|
866
|
+
store.state.counters.records = rb.records;
|
|
867
|
+
for (const sid of Object.keys(store.state.sources)) store.state.sources[sid].records = rb.bySource[sid] || 0;
|
|
868
|
+
store.saveCube();
|
|
869
|
+
store.saveSessions();
|
|
870
|
+
store.saveActivity();
|
|
871
|
+
store.saveState();
|
|
872
|
+
|
|
873
|
+
console.log(` ${C.g}✓${C.r} dropped ${int(res.dropped)} record(s) from '${id}' across ${res.shards} shard(s)`);
|
|
874
|
+
console.log(` ${C.g}✓${C.r} cleared its cursor — the next refresh re-reads it from the beginning`);
|
|
875
|
+
console.log(` ${C.g}✓${C.r} rebuilt aggregates from the remaining ${int(rb.records)} record(s)`);
|
|
876
|
+
for (const [sid, n] of Object.entries(rb.bySource).sort((a, b) => b[1] - a[1])) console.log(` ${sid.padEnd(12)} ${int(n).padStart(9)}`);
|
|
877
|
+
console.log(`\n ${C.dim}next: tokenflow refresh${C.r}\n`);
|
|
878
|
+
}
|
|
879
|
+
|
|
769
880
|
// ==================================================================== live ==
|
|
770
881
|
|
|
771
882
|
/** Fast path for live commands: watch snapshot when fresh, else compute now. */
|
|
@@ -783,7 +894,36 @@ async function liveStatus() {
|
|
|
783
894
|
* --status is a watcher running? how fresh is it?
|
|
784
895
|
* --stop stop a running watcher
|
|
785
896
|
*/
|
|
897
|
+
/**
|
|
898
|
+
* `tokenflow watch --install-agent` / `--uninstall-agent`.
|
|
899
|
+
*
|
|
900
|
+
* Live data needs a resident watcher. Without an agent one only lasts as long
|
|
901
|
+
* as the session that started it, so a reboot leaves stale numbers behind a
|
|
902
|
+
* paused menu bar.
|
|
903
|
+
*/
|
|
904
|
+
async function watchAgentCommand() {
|
|
905
|
+
const agent = await import('../src/core/watch-agent.js');
|
|
906
|
+
if (flags['uninstall-agent']) {
|
|
907
|
+
const r = agent.uninstall();
|
|
908
|
+
console.log(r.removed
|
|
909
|
+
? ` ${C.g}✓${C.r} removed the watcher agent ${C.dim}(${r.plist})${C.r}\n ${C.dim}the watcher no longer starts at login; 'tokenflow watch' still runs by hand.${C.r}\n`
|
|
910
|
+
: ` ${C.dim}○ no watcher agent was installed${C.r}\n`);
|
|
911
|
+
return;
|
|
912
|
+
}
|
|
913
|
+
const r = agent.install();
|
|
914
|
+
for (const f of r.removed) {
|
|
915
|
+
console.log(` ${C.y}!${C.r} removed a conflicting agent: ${C.b}${f.label}${C.r}`);
|
|
916
|
+
console.log(` ${C.dim}${f.file} — two agents running a watcher fight over the same lock${C.r}`);
|
|
917
|
+
}
|
|
918
|
+
console.log(` ${C.g}✓${C.r} installed the watcher agent ${C.dim}(${r.plist})${C.r}`);
|
|
919
|
+
console.log(` ${C.dim}starts at login · restarts on a crash · a deliberate stop stays stopped${C.r}`);
|
|
920
|
+
console.log(` ${C.dim}log: ${paths().root}/watch.log${C.r}`);
|
|
921
|
+
if (!r.started) console.log(` ${C.y}!${C.r} launchctl would not start it — check Console.app, or run 'tokenflow watch' by hand`);
|
|
922
|
+
console.log(`\n ${C.dim}remove it: tokenflow watch --uninstall-agent${C.r}\n`);
|
|
923
|
+
}
|
|
924
|
+
|
|
786
925
|
async function cmdWatch() {
|
|
926
|
+
if (flags['install-agent'] || flags['uninstall-agent']) return watchAgentCommand();
|
|
787
927
|
if (flags.stop) {
|
|
788
928
|
const r = stopWatch();
|
|
789
929
|
console.log(r.stopped ? `${C.g}✓${C.r} stopped watcher ${C.dim}(pid ${r.pid})${C.r}` : `${C.dim}○ ${r.reason}${C.r}`);
|
|
@@ -794,9 +934,17 @@ async function cmdWatch() {
|
|
|
794
934
|
const st = readLiveStatus();
|
|
795
935
|
console.log(` watcher ${running ? `${C.g}running${C.r}` : `${C.dim}not running${C.r}`}`);
|
|
796
936
|
// Identity lines only describe a live process — a dead watcher's leftovers
|
|
797
|
-
// are history, not status.
|
|
798
|
-
|
|
799
|
-
|
|
937
|
+
// are history, not status. The PID comes from the LOCK, which is the only
|
|
938
|
+
// thing that knows who holds it; the cycle count comes from the status
|
|
939
|
+
// file, and only when that file was written by this same watcher. After a
|
|
940
|
+
// restart the two disagree for one cycle, and reporting the dead PID then
|
|
941
|
+
// would be exactly the kind of small lie that hid a stale watcher before.
|
|
942
|
+
const lock = running ? readLock() : null;
|
|
943
|
+
if (lock) {
|
|
944
|
+
const own = st?.watcher?.pid === lock.pid ? st.watcher : null;
|
|
945
|
+
const every = own?.intervalSeconds ?? st?.watcher?.intervalSeconds ?? '?';
|
|
946
|
+
const cycles = own ? `${int(own.cycles)} cycle(s)` : 'first cycle pending';
|
|
947
|
+
console.log(` pid ${lock.pid} · every ${every}s · ${cycles}`);
|
|
800
948
|
}
|
|
801
949
|
const lastErr = st?.lastError;
|
|
802
950
|
if (lastErr) console.log(` ${C.y}last error${C.r} ${relativeTime(lastErr.at)}: ${lastErr.message}`);
|
|
@@ -805,6 +953,22 @@ async function cmdWatch() {
|
|
|
805
953
|
console.log(` data ${fresh.stale ? `${C.y}stale${C.r}` : `${C.g}fresh${C.r}`} ${st.freshness.lastRefresh ? `· updated ${relativeTime(st.freshness.lastRefresh)}` : '(never refreshed)'}`);
|
|
806
954
|
console.log(` status ${paths().status}`);
|
|
807
955
|
}
|
|
956
|
+
const agent = await import('../src/core/watch-agent.js');
|
|
957
|
+
if (agent.supported()) {
|
|
958
|
+
const a = agent.status();
|
|
959
|
+
const state = !a.installed ? `${C.dim}not installed${C.r}`
|
|
960
|
+
: a.loaded ? `${C.g}installed${C.r} ${C.dim}· starts at login, restarts on crash${C.r}`
|
|
961
|
+
: `${C.y}installed but not loaded${C.r}`;
|
|
962
|
+
console.log(` agent ${state}`);
|
|
963
|
+
if (!a.installed) console.log(` ${C.dim}install it: tokenflow watch --install-agent${C.r}`);
|
|
964
|
+
// Two agents running a watcher means the loser exits and respawns for
|
|
965
|
+
// ever; say so rather than letting it churn quietly in the log.
|
|
966
|
+
for (const f of a.foreign) {
|
|
967
|
+
console.log(` ${C.y}!${C.r} another agent also runs a watcher: ${f.label}`);
|
|
968
|
+
console.log(` ${C.dim}${f.file}${C.r}`);
|
|
969
|
+
console.log(` ${C.dim}'tokenflow watch --install-agent' replaces it with the supported one.${C.r}`);
|
|
970
|
+
}
|
|
971
|
+
}
|
|
808
972
|
if (!running && !flags.json) {
|
|
809
973
|
console.log(`\n ${C.dim}start one: tokenflow watch${C.r}`);
|
|
810
974
|
}
|
|
@@ -1146,6 +1310,33 @@ async function cmdDiagnostics() {
|
|
|
1146
1310
|
}
|
|
1147
1311
|
}
|
|
1148
1312
|
|
|
1313
|
+
/** `tokenflow team` — per-developer usage from the shared sync folder (P4-B). */
|
|
1314
|
+
async function cmdTeam() {
|
|
1315
|
+
const cfg = loadConfig();
|
|
1316
|
+
if (!cfg.sync?.enabled || !cfg.sync?.dir) {
|
|
1317
|
+
console.error(`${C.red}Team view reads the shared sync folder.${C.r}
|
|
1318
|
+
Enable multi-machine sync first — every team member points at the SAME folder:
|
|
1319
|
+
|
|
1320
|
+
sync:
|
|
1321
|
+
enabled: true
|
|
1322
|
+
dir: /path/to/shared/TokenFlow # same folder for everyone
|
|
1323
|
+
machineName: MacBook Pro # this machine's label
|
|
1324
|
+
developerName: Your Name # ← opt-in per person; omit to stay anonymous
|
|
1325
|
+
|
|
1326
|
+
Then run \`tokenflow sync\` on each machine and \`tokenflow team\` here.`);
|
|
1327
|
+
return;
|
|
1328
|
+
}
|
|
1329
|
+
const { aggregate, renderText } = await import('../src/core/team.js');
|
|
1330
|
+
const dir = cfg.sync.dir.replace(/^~(?=$|\/)/, os.homedir());
|
|
1331
|
+
const t = aggregate(dir, {
|
|
1332
|
+
from: typeof flags.from === 'string' ? flags.from : null,
|
|
1333
|
+
to: typeof flags.to === 'string' ? flags.to : null,
|
|
1334
|
+
includeAnonymous: !!flags['include-anonymous'],
|
|
1335
|
+
});
|
|
1336
|
+
if (flags.json) { console.log(JSON.stringify(t, null, 2)); return; }
|
|
1337
|
+
console.log(renderText(t));
|
|
1338
|
+
}
|
|
1339
|
+
|
|
1149
1340
|
async function cmdMenubar() {
|
|
1150
1341
|
const mode = String(flags.mode || loadConfig().ui?.menubarMode || 'auto');
|
|
1151
1342
|
|
|
@@ -1244,6 +1435,9 @@ function help(code = 0) {
|
|
|
1244
1435
|
tokenflow watch --once one cycle and exit (cron-friendly)
|
|
1245
1436
|
tokenflow watch --status is a watcher running? how fresh is the data?
|
|
1246
1437
|
tokenflow watch --stop stop a running watcher
|
|
1438
|
+
tokenflow watch --install-agent keep the watcher running across reboots
|
|
1439
|
+
(installed by default at setup on macOS;
|
|
1440
|
+
--uninstall-agent removes it)
|
|
1247
1441
|
tokenflow usage today / week / month tokens & cost (--json)
|
|
1248
1442
|
tokenflow cost estimated vs measured spend, projections
|
|
1249
1443
|
tokenflow capacity configured limits: %, burn, reset countdowns
|
|
@@ -1263,6 +1457,21 @@ function help(code = 0) {
|
|
|
1263
1457
|
tokenflow export --csv --all every normalized record
|
|
1264
1458
|
tokenflow export --html one self-contained offline dashboard file
|
|
1265
1459
|
|
|
1460
|
+
${C.b}Intelligence${C.r}
|
|
1461
|
+
tokenflow models-compare cost/request, tokens/request, cache-hit% per model
|
|
1462
|
+
(--from/--to <date> to pick the window)
|
|
1463
|
+
tokenflow budget --set 200 monthly cap → projected-overrun alerts (dedup'd)
|
|
1464
|
+
tokenflow budget current state: safe / approaching / over
|
|
1465
|
+
tokenflow digest --deliver build "Your AI Week" and send via configured channels
|
|
1466
|
+
tokenflow schedule --install --at "Monday 09:00" weekly digest via launchd
|
|
1467
|
+
tokenflow schedule --status is the digest schedule installed?
|
|
1468
|
+
tokenflow team per-developer usage from the shared sync folder
|
|
1469
|
+
tokenflow diagnostics version, providers, store freshness, feature states
|
|
1470
|
+
|
|
1471
|
+
${C.b}Sync (optional, off by default)${C.r}
|
|
1472
|
+
tokenflow sync push this machine's daily rollups + show merged view
|
|
1473
|
+
tokenflow sync --off disable sync entirely
|
|
1474
|
+
|
|
1266
1475
|
${C.b}Configure${C.r}
|
|
1267
1476
|
tokenflow pricing show which models have a price, and from where
|
|
1268
1477
|
tokenflow pricing --sources provenance of every built-in rate + tier multipliers
|
|
@@ -1276,6 +1485,10 @@ function help(code = 0) {
|
|
|
1276
1485
|
tokenflow validate re-validate every stored record
|
|
1277
1486
|
tokenflow compact drop superseded records after a rewrite
|
|
1278
1487
|
tokenflow reset --yes delete ingested data (keeps config)
|
|
1488
|
+
tokenflow reset --source <id> --yes
|
|
1489
|
+
forget ONE source and re-read it from
|
|
1490
|
+
scratch on the next refresh (repairs a
|
|
1491
|
+
source whose adapter was fixed)
|
|
1279
1492
|
|
|
1280
1493
|
${C.b}Flags${C.r} --json --quiet --provider <id> --from/--to <date> --port <n> --no-open --debug
|
|
1281
1494
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@vimoxshah/tokenflow",
|
|
3
|
-
"version": "1.1.
|
|
3
|
+
"version": "1.1.1",
|
|
4
4
|
"description": "Local-first, provider-agnostic analytics for the AI tokens you actually spend. Zero dependencies, nothing leaves your machine.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -23,6 +23,7 @@
|
|
|
23
23
|
"examples",
|
|
24
24
|
"scripts",
|
|
25
25
|
"README.md",
|
|
26
|
+
"CHANGELOG.md",
|
|
26
27
|
"LICENSE",
|
|
27
28
|
"SECURITY.md",
|
|
28
29
|
"CONTRIBUTING.md",
|
|
@@ -16,6 +16,18 @@
|
|
|
16
16
|
const MAD_SCALE = 1.4826;
|
|
17
17
|
const SPIKE_Z = 3.5; // Iglewicz–Hoaglin threshold for a modified z-score
|
|
18
18
|
const HIGH_Z = 6; // well past that: call it high severity
|
|
19
|
+
/**
|
|
20
|
+
* Far past "high": magnitude at this scale outranks recency in the list.
|
|
21
|
+
*
|
|
22
|
+
* Severity saturates at `high` around z=6, and the ordering below then falls
|
|
23
|
+
* back to date, newest first. That buried a real one: an adapter bug inflated
|
|
24
|
+
* one day to 240× its 60-day median, scoring z=171, and it ranked THIRD behind
|
|
25
|
+
* two request spikes of z=6.5 and z=11.2 from later in the week. The menu bar
|
|
26
|
+
* shows the top two alerts, so for a week the loudest signal the product had
|
|
27
|
+
* was the one thing it did not show. A z of 171 and a z of 6 are not the same
|
|
28
|
+
* news, whichever happened more recently.
|
|
29
|
+
*/
|
|
30
|
+
const EXTREME_Z = 25;
|
|
19
31
|
const BASELINE_WINDOW = 60;
|
|
20
32
|
const MIN_BASELINE = 10;
|
|
21
33
|
|
|
@@ -167,9 +179,14 @@ export function detectAnomalies(daily, opt = {}) {
|
|
|
167
179
|
}
|
|
168
180
|
|
|
169
181
|
const cap = opt.limit ?? 12;
|
|
182
|
+
// severity, then extremes by magnitude, then recency. Ordinary alerts keep
|
|
183
|
+
// reading as a feed; an outlier of a different order never gets buried in it.
|
|
184
|
+
const extreme = (a) => ((a.z ?? 0) >= EXTREME_Z ? 0 : 1);
|
|
170
185
|
return out
|
|
171
186
|
.sort((a, b) =>
|
|
172
187
|
SEV_ORDER[a.severity] - SEV_ORDER[b.severity]
|
|
188
|
+
|| extreme(a) - extreme(b)
|
|
189
|
+
|| (extreme(a) === 0 ? (b.z ?? 0) - (a.z ?? 0) : 0)
|
|
173
190
|
|| (a.date < b.date ? 1 : a.date > b.date ? -1 : 0)
|
|
174
191
|
|| (b.z ?? 0) - (a.z ?? 0))
|
|
175
192
|
.slice(0, cap);
|
|
@@ -10,6 +10,19 @@ import os from 'node:os';
|
|
|
10
10
|
import path from 'node:path';
|
|
11
11
|
import { loadConfig, paths } from '../core/config.js';
|
|
12
12
|
|
|
13
|
+
/**
|
|
14
|
+
* The shipped package.json is the only honest source for the version. A
|
|
15
|
+
* hardcoded fallback goes stale the moment anyone forgets it at release time,
|
|
16
|
+
* and then every diagnostics report quietly names the wrong release.
|
|
17
|
+
*/
|
|
18
|
+
function packageVersion() {
|
|
19
|
+
try {
|
|
20
|
+
return JSON.parse(fs.readFileSync(new URL('../../package.json', import.meta.url), 'utf8')).version || 'unknown';
|
|
21
|
+
} catch {
|
|
22
|
+
return 'unknown';
|
|
23
|
+
}
|
|
24
|
+
}
|
|
25
|
+
|
|
13
26
|
/**
|
|
14
27
|
* @param {{includePaths?: boolean}} opt
|
|
15
28
|
* @returns {object} diagnostics snapshot (plain JSON-able)
|
|
@@ -26,7 +39,7 @@ export function collect(opt = {}) {
|
|
|
26
39
|
};
|
|
27
40
|
|
|
28
41
|
return {
|
|
29
|
-
version: process.env.npm_package_version ||
|
|
42
|
+
version: process.env.npm_package_version || packageVersion(),
|
|
30
43
|
node: process.version,
|
|
31
44
|
platform: `${os.platform()} ${os.arch()} ${os.release()}`,
|
|
32
45
|
timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
|
package/src/core/live-status.js
CHANGED
|
@@ -22,6 +22,7 @@ import { loadConfig, paths, ensureDirs } from './config.js';
|
|
|
22
22
|
import { readJson } from './store.js';
|
|
23
23
|
import { compact, usd, countdown } from './units.js';
|
|
24
24
|
import { detectMilestones } from '../analytics/milestones.js';
|
|
25
|
+
import { lockIsLive, readLock } from './watch-lock.js';
|
|
25
26
|
|
|
26
27
|
// Formatting adapters over the shared units.js formatters (which the browser
|
|
27
28
|
// bundle also uses): null means "nothing to show", never "—", never 0.
|
|
@@ -324,8 +325,10 @@ export function withComputedFreshness(status, nowMs = Date.now()) {
|
|
|
324
325
|
* The cache window derives from the watcher's own cadence (interval + slack),
|
|
325
326
|
* because a snapshot written 90 seconds into a 120-second cycle is exactly as
|
|
326
327
|
* current as the product promised — not stale. When a fallback compute does
|
|
327
|
-
* happen, daemon identity (pid, cycles
|
|
328
|
-
*
|
|
328
|
+
* happen, daemon identity (pid, cycles) is carried over from the cached file
|
|
329
|
+
* so a slow poll never makes the UI claim no watcher is running — but only
|
|
330
|
+
* while the watcher lock is actually live, so a dead daemon's identity does
|
|
331
|
+
* not linger either.
|
|
329
332
|
*/
|
|
330
333
|
export function currentStatus(opt = {}) {
|
|
331
334
|
const cached = readLiveStatus();
|
|
@@ -337,7 +340,15 @@ export function currentStatus(opt = {}) {
|
|
|
337
340
|
if (age <= maxAgeMs) return { status: withComputedFreshness(cached), fromWatch: true };
|
|
338
341
|
}
|
|
339
342
|
const fresh = buildLiveStatus({ config: cfg });
|
|
340
|
-
|
|
343
|
+
// Carry daemon identity over ONLY while THAT daemon is really there. A
|
|
344
|
+
// watcher block outlives the process that wrote it, and repeating it after
|
|
345
|
+
// the watcher died is how a paused TokenFlow came to look live. Matching the
|
|
346
|
+
// pid against the lock holder also stops a restarted watcher from being
|
|
347
|
+
// described with its predecessor's pid and cycle count.
|
|
348
|
+
const lock = readLock();
|
|
349
|
+
if (cached?.watcher && lockIsLive(lock) && cached.watcher.pid === lock.pid) {
|
|
350
|
+
fresh.watcher = cached.watcher;
|
|
351
|
+
}
|
|
341
352
|
if (!fresh.lastCycle && cached?.lastCycle) fresh.lastCycle = cached.lastCycle;
|
|
342
353
|
if (!fresh.lastError && cached?.lastError) fresh.lastError = cached.lastError;
|
|
343
354
|
return { status: fresh, fromWatch: false };
|