@vimoxshah/tokenflow 1.1.2 → 1.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (86) hide show
  1. package/CHANGELOG.md +180 -0
  2. package/Dockerfile.team +20 -0
  3. package/README.md +30 -11
  4. package/bin/tokenflow.js +147 -12
  5. package/design/tokens.yaml +330 -0
  6. package/docs/architecture.md +5 -4
  7. package/docs/cli.md +204 -0
  8. package/docs/configuration.md +117 -2
  9. package/docs/design-system.md +187 -0
  10. package/docs/exports-and-budgets.md +85 -0
  11. package/docs/guard-codex.md +132 -0
  12. package/docs/ledger.md +144 -0
  13. package/docs/live-mode.md +40 -0
  14. package/docs/media/overview-aurora-dark.png +0 -0
  15. package/docs/media/receipts-aurora-dark.png +0 -0
  16. package/docs/providers-otel.md +179 -0
  17. package/docs/providers.md +54 -1
  18. package/docs/receipt-schema.md +74 -0
  19. package/docs/roadmap.md +182 -0
  20. package/docs/team-server.md +170 -0
  21. package/docs/ui-views.md +322 -0
  22. package/package.json +7 -2
  23. package/schemas/receipt.v0.json +160 -0
  24. package/scripts/build-menubar-app.sh +3 -1
  25. package/scripts/design-build.js +475 -0
  26. package/src/analytics/anatomy.js +467 -0
  27. package/src/analytics/branch-compare.js +159 -0
  28. package/src/analytics/cache-health.js +141 -0
  29. package/src/analytics/live-view.js +266 -0
  30. package/src/analytics/receipt-schema.js +214 -0
  31. package/src/analytics/receipt.js +709 -0
  32. package/src/analytics/rhythm.js +184 -0
  33. package/src/analytics/whatif.js +263 -0
  34. package/src/commands/budget-scopes.js +133 -0
  35. package/src/commands/doctor-checks.js +400 -0
  36. package/src/commands/guard.js +531 -0
  37. package/src/commands/hooks.js +238 -0
  38. package/src/commands/pricing-diff.js +316 -0
  39. package/src/commands/receipt.js +226 -0
  40. package/src/commands/team-serve.js +407 -0
  41. package/src/commands/week.js +86 -0
  42. package/src/core/annotations.js +97 -0
  43. package/src/core/budget.js +33 -0
  44. package/src/core/bundle.js +45 -2
  45. package/src/core/ingest.js +33 -0
  46. package/src/core/live-status.js +227 -2
  47. package/src/core/policy.js +103 -0
  48. package/src/core/receipt-note.js +123 -0
  49. package/src/core/repo.js +64 -0
  50. package/src/core/sync.js +163 -26
  51. package/src/core/team.js +0 -0
  52. package/src/export/html-snapshot.js +28 -1
  53. package/src/export/menubar.js +21 -0
  54. package/src/export/receipt-card.js +210 -0
  55. package/src/export/week-card.js +185 -0
  56. package/src/providers/mock/index.js +383 -52
  57. package/src/providers/openai/index.js +31 -1
  58. package/src/providers/otel/index.js +656 -0
  59. package/src/server/routes/annotations.js +42 -0
  60. package/src/server/routes/cache-health.js +95 -0
  61. package/src/server/routes/index.js +54 -0
  62. package/src/server/routes/session.js +157 -0
  63. package/src/server/server.js +47 -1
  64. package/src/ui/app.js +541 -308
  65. package/src/ui/charts.js +95 -0
  66. package/src/ui/first-run.js +144 -0
  67. package/src/ui/index.html +4 -1
  68. package/src/ui/palette.js +335 -0
  69. package/src/ui/styles/anatomy.css +117 -0
  70. package/src/ui/styles/annotations.css +40 -0
  71. package/src/ui/styles/branches.css +99 -0
  72. package/src/ui/styles/cache.css +6 -0
  73. package/src/ui/styles/first-run.css +31 -0
  74. package/src/ui/styles/live.css +100 -0
  75. package/src/ui/styles/palette.css +85 -0
  76. package/src/ui/styles/rhythm.css +8 -0
  77. package/src/ui/styles/whatif.css +55 -0
  78. package/src/ui/styles.css +303 -196
  79. package/src/ui/views/anatomy.js +567 -0
  80. package/src/ui/views/annotations.js +121 -0
  81. package/src/ui/views/branches.js +304 -0
  82. package/src/ui/views/cache.js +232 -0
  83. package/src/ui/views/index.js +85 -0
  84. package/src/ui/views/live.js +683 -0
  85. package/src/ui/views/rhythm.js +206 -0
  86. package/src/ui/views/whatif.js +196 -0
package/docs/ledger.md ADDED
@@ -0,0 +1,144 @@
1
+ # The Ledger — team cost per branch and per merged PR
2
+
3
+ The Ledger joins everyone's local branch-cost receipts (`src/analytics/receipt.js`)
4
+ across every machine that syncs, so a team lead can see "what did this branch
5
+ cost" and "what did merged PRs typically cost" without anyone reading anyone
6
+ else's session transcripts. It rides the same file sync as the rest of
7
+ `tokenflow sync` / `tokenflow team` — see [`configuration.md`](configuration.md)
8
+ for the `sync:` block and [`team-server.md`](team-server.md) if you push to a
9
+ self-hosted server instead of a shared folder.
10
+
11
+ Two files travel per machine, both written by `sync.push()`
12
+ (`src/core/sync.js`):
13
+
14
+ | file | shape | conflict rule |
15
+ | --- | --- | --- |
16
+ | `<machineId>.jsonl` | one line per day, coarse tokens/requests/cost | append-only, last-write-wins per line |
17
+ | `<machineId>.receipts.json` | one entry per (repo, branch) this machine has session history for | whole-state, last-write-wins |
18
+
19
+ The second file is the Ledger's input. It is written on every push unless you
20
+ set `sync.receipts: false`.
21
+
22
+ ## What leaves the machine
23
+
24
+ Per branch, exactly these fields — nothing else:
25
+
26
+ | Field | Meaning |
27
+ | --- | --- |
28
+ | `repo` | The repository's name — **basename only**, never a path. |
29
+ | `branch` | The branch name. |
30
+ | `costUsd` | Estimated spend on this branch (`null` if nothing was priced — never `0`). |
31
+ | `turns` | LLM requests attributed to this branch. |
32
+ | `sessions` | Distinct sessions that touched it. |
33
+ | `subagentTurns` | Of `turns`, how many ran as a subagent. |
34
+ | `contextShare` | Share of `costUsd` that re-sent prior context vs. fresh work. |
35
+ | `first` / `last` | ISO timestamps bounding the branch's activity. |
36
+ | `longLived` | True for `main` / `staging` / `develop`-style names — this is a receipt for a period of work, not one change. |
37
+ | `pr` | `null`, or `{ number, mergedAt }` for the PR this branch shipped in. |
38
+
39
+ The whole file also carries `schema` (currently `1`), `machineId`,
40
+ `generatedAt`, and — **only if you set `sync.machineName` yourself** —
41
+ `machineName`. Unlike the daily jsonl (which always falls back to the
42
+ hostname), the ledger stays anonymous by default: it joins branch and PR
43
+ identity, which is more identifying than a coarse daily total, so machine
44
+ labelling here is opt-in on top of opt-in.
45
+
46
+ Never included, at any level: file paths, commit hashes, PR titles, diffs or
47
+ line counts, prompts, or any model/code text. The data comes entirely from
48
+ `buildReceiptsForStore()` (`src/core/bundle.js`) — this module does not
49
+ re-derive cost or open a session transcript itself.
50
+
51
+ **Known gap:** `buildReceiptsForStore()` has no PR source wired in yet (PR
52
+ matching today only happens through `tokenflow receipt --gh`, a separate CLI
53
+ path). That means `pr` is `null` on every real push right now. The field
54
+ exists so a future cached PR list needs no format change — `perMergedPr` and
55
+ the `mergedPrCount` in `byMonth` will be empty until that lands.
56
+
57
+ ## Sending it to a server instead of a folder
58
+
59
+ ```yaml
60
+ sync:
61
+ enabled: true
62
+ to: https://team-box.example.com # or your self-hosted `tokenflow team serve`
63
+ token: <shared-secret> # or set TOKENFLOW_SYNC_TOKEN in the env
64
+ ```
65
+
66
+ With `sync.to` set, `push()` POSTs the exact same two files' text to
67
+ `<to>/api/rollup` as `{ machineId, files: { "<machineId>.jsonl": "…", "<machineId>.receipts.json": "…" } }`
68
+ with an `Authorization: Bearer <token>` header, instead of writing them into
69
+ `sync.dir`. A non-2xx response is reported as an error with the status line
70
+ only (`sync push to <to> failed: 401 Unauthorized`) — the response body is
71
+ never printed, since a server's error page is not this tool's to surface.
72
+ See [`team-server.md`](team-server.md) for the receiving end's contract
73
+ (path, auth model, file-name allowlist, size cap).
74
+
75
+ `push()` stays synchronous when writing to a folder (matches every existing
76
+ caller). The moment a destination server is configured it necessarily makes
77
+ a network call, so that branch returns a Promise — `await push(...)` works
78
+ either way.
79
+
80
+ ## The join: `team.aggregate(dir)`
81
+
82
+ `tokenflow team` (`src/core/team.js`) reads every sibling `*.receipts.json`
83
+ alongside the `*.jsonl` files it already merges for per-developer rows, and
84
+ adds a `receipts` section to its result:
85
+
86
+ - **`receipts.totals`** — `{ cost, branches, repos }` across every merged
87
+ (repo, branch).
88
+ - **`receipts.byRepo`** — `[{ repo, cost, branches, sessions }]`, sorted by
89
+ cost descending.
90
+ - **`receipts.byMonth`** — `[{ month, cost, mergedPrCount }]`, bucketed by the
91
+ month of each branch's `last` activity, sorted ascending. `mergedPrCount`
92
+ counts branches in that month whose `pr.mergedAt` is known.
93
+ - **`receipts.perMergedPr`** — `{ median, p90, sampleSize }` cost across
94
+ branches with a known `pr.mergedAt`. `p90` follows the same convention as
95
+ `sessionStats()`'s `p90SessionCost` in `receipt.js`: sort branch costs
96
+ descending, take the value 10% of the way down (10% of branches cost more
97
+ than this).
98
+ - **`receipts.concentration`** — `{ top10Share, top5 }`: the share of total
99
+ receipt cost held by the most expensive 10% of branches (at least one
100
+ branch), and the five most expensive branches as `{ repo, branch, cost }`.
101
+ - **`receipts.longLivedShare`** — `{ cost, share }`: how much of total
102
+ receipt cost sits on `longLived` branches — work that a PR never bounded,
103
+ labelled that way because it is a different kind of spend to reason about
104
+ than a feature branch that shipped and closed.
105
+
106
+ Merging across machines, per (repo, branch): **sum** `costUsd`/`turns`/
107
+ `sessions`; take the **earliest** `first` and **latest** `last`; keep
108
+ whichever machine's view of the PR **knows a `mergedAt`** (a machine that
109
+ hasn't re-synced since the merge only sees an open PR, or none — the merged
110
+ view wins once any machine has it). `longLived` is true if any machine
111
+ reports it true (the branch-name pattern is deterministic, so this only
112
+ matters for split-second sync races).
113
+
114
+ Receipts are **not** filtered by the `from`/`to` window `aggregate()` accepts
115
+ for the daily rollup: a receipt is a lifetime-of-the-branch ledger entry, not
116
+ a per-day bucket, so there is no single day to test it against. If a team
117
+ needs a windowed view of the Ledger later, that is a follow-on, not this
118
+ join.
119
+
120
+ `renderText(aggregate(dir))` prints a "Receipts (branch × PR cost ledger)"
121
+ section using all of the above; with no `*.receipts.json` files synced yet it
122
+ prints one line saying so, rather than an empty table.
123
+
124
+ ## Example
125
+
126
+ ```
127
+ Receipts (branch × PR cost ledger)
128
+ Total receipt cost $43.00 across 3 branch(es) in 2 repo(s)
129
+
130
+ By repo
131
+ demo-repo $23.00 2 branch(es) 4 session(s)
132
+ other-repo $20.00 1 branch(es) 3 session(s)
133
+
134
+ Cost per merged PR median $19.00 · p90 $20.00 (n=2)
135
+ Concentration top 10% of branches hold 46.5% of receipt cost
136
+ other-repo/feature-y $20.00
137
+ demo-repo/feature-x $18.00
138
+ demo-repo/main $5.00
139
+ Long-lived branches 11.6% of receipt cost ($5.00) sits on long-lived branches — work not bounded by a PR
140
+
141
+ By month
142
+ 2026-07 $20.00 1 merged PR(s)
143
+ 2026-08 $23.00 1 merged PR(s)
144
+ ```
package/docs/live-mode.md CHANGED
@@ -109,6 +109,29 @@ ratio, z-score — so you can check it rather than trust it. Notifications fire
109
109
  only for **same-day high-severity** anomalies; history replaying as alerts on
110
110
  a first run would be noise, so it doesn't happen.
111
111
 
112
+ ## Live status: sessions, receipts, guard, sparklines
113
+
114
+ Alongside the totals above, `data/status.json` carries four sections built from a second,
115
+ record-level scan of the store — the cube the rest of this file describes is pre-aggregated and
116
+ carries no session id, branch, or per-turn guard verdict. All four anchor on `lastRefresh` (the
117
+ store's last completed refresh), never the real clock, so a reader is only ever as current as the
118
+ last refresh cycle said.
119
+
120
+ | Section | What it carries |
121
+ |---|---|
122
+ | `liveSessions` | Up to 8 sessions with activity in the last 10 minutes of `lastRefresh`: source, model, project/repository/branch, turns, subagent turns, running cost, context tokens and share, and that session's own guard verdict. |
123
+ | `receiptsToday` | The 3 highest-cost (repo, branch) pairs with spend dated `lastRefresh`'s day, plus the total across all of them. |
124
+ | `guard` | The declared thresholds (from `~/.tokenflow/config.yaml`, overridden per repository by `.tokenflow/policy.yaml`) and the last guard verdict, with its `source`: `cache` when a session's own guard cache holds one, `derived` when it is inferred from the worst `liveSessions` entry instead (today the cache stores no verdict, so this is `derived` on every run). |
125
+ | `sparklines` | 24 hourly token/cost buckets per source, covering the 24 hours up to `lastRefresh`. |
126
+
127
+ ### The Live tab
128
+
129
+ The dashboard's **Live** tab (a registered view, `src/ui/views/live.js`) renders these same four
130
+ sections, polling `/api/live` every 30 seconds while the tab is open. Nothing on it animates
131
+ toward a number it was not given: every value is the last thing `data/status.json` said, and it
132
+ changes only when that file changes. In an offline snapshot the tab shows the numbers captured at
133
+ export time and hides the controls (like "Manage limits") that need a live server.
134
+
112
135
  ## Menu bar — TokenFlow.app (native, macOS)
113
136
 
114
137
  TokenFlow ships its own menu bar application: a ~370 KB native binary compiled
@@ -137,11 +160,19 @@ tokenflow menubar --app --login-item
137
160
 
138
161
  - header: live/watcher badge + data freshness ("live · data updated just now")
139
162
  - Today / Week / Month rows — tokens, requests, estimated cost
163
+ - **Live, as of `<time>`** — up to 3 sessions active in the last 10 minutes, each with a
164
+ context gauge, its own guard-coloured dot, and running cost
165
+ - **Today's receipts** — the day's top 3 (repo, branch) pairs by spend, and the total
166
+ - **Guard** — the declared caps, the last verdict, and two buttons: **Raise cost cap** and
167
+ **Clear caps**, both of which write through `tokenflow guard --set` (never by editing
168
+ `config.yaml` directly) and then run one refresh cycle so the popover reflects the change
140
169
  - Today by provider — inline share bars (▰▱), tokens and cost per provider
141
170
  - Today by source — the app that wrote the log (claude-code, opencode,
142
171
  hermes, git…). Provider attribution names the model's vendor, so Hermes
143
172
  traffic appears under each model's vendor there; this section shows it as
144
173
  "hermes"
174
+ - **Last 24 hours by source** — a small sparkline per source, tokens and cost, drawn only
175
+ when its data is present so an older status file shows exactly what it always showed
145
176
  - Top models today — token and cost leaders per model
146
177
  - Capacity — a real meter per declared limit with %, exhaustion ETA and reset
147
178
  countdown; "first projected hit" callout when one will cross before reset
@@ -149,12 +180,21 @@ tokenflow menubar --app --login-item
149
180
  - Alerts — high-severity anomalies with their arithmetic
150
181
  - Actions — `Refresh now` (⌘R, runs a watch cycle), `Open Dashboard`,
151
182
  `Start/Stop watcher`, `Quit`
183
+ - **Density** — Comfortable or Compact, a persisted preference that tightens row spacing
184
+ and type size for a laptop screen
185
+ - **Shortcut** — a global keyboard shortcut that toggles the popover from any app,
186
+ persisted and re-registered immediately on change
152
187
  - Appearance button (◐) — cycles system → light → dark; persisted across
153
188
  launches in `defaults` under `appearanceOverride`
154
189
 
155
190
  Dark/light follows the appearance override (◐ button, persisted) or the system
156
191
  appearance when set to follow; all figures use monospaced digits.
157
192
 
193
+ **Transient guard alert.** When a live session's guard level rises to warn or block since the
194
+ last load, a small card slides in from the status item to announce it — never on the first load,
195
+ and never on a recovery back down, so it only ever interrupts for something new. One card at a
196
+ time; clicking it opens the popover on the section that explains why.
197
+
158
198
  ## Menu bar — other platforms (SwiftBar/xbar text protocol)
159
199
 
160
200
  For Linux bars or if you prefer SwiftBar on macOS:
Binary file
@@ -0,0 +1,179 @@
1
+ # `otel` — OpenTelemetry (GenAI)
2
+
3
+ A machine sweep of common local coding-agent tools found none that write per-request token
4
+ counts to disk in a proprietary format for several of them — Gemini CLI among them. Several of
5
+ them CAN write [OpenTelemetry](https://opentelemetry.io/) data to a local file, though, so this
6
+ adapter reads that instead: point a tool's OTLP file exporter, or Gemini CLI's own file
7
+ telemetry, at a file this adapter can see, and it becomes a TokenFlow source with zero
8
+ vendor-specific code on the tool's side.
9
+
10
+ | | |
11
+ |---|---|
12
+ | Reads | `~/.gemini/telemetry.log` (best-effort default — see below) and `~/.tokenflow/otel/*.{jsonl,ndjson,json,log}` |
13
+ | Measurement | `primary` |
14
+ | Reports | `input`, `output`, `cache_read`, `cache_write` (standards-based path only), `reasoning`, session/conversation id, request id (span id), duration, model, provider |
15
+ | Cannot know | interface/surface (no OTel GenAI attribute carries it — every record classifies as `Unknown` unless a resource attribute happens to match a known IDE/CLI name) |
16
+
17
+ ## What actually lands on disk — two shapes, not one
18
+
19
+ This adapter reads two genuinely different file shapes, and it matters which one a given tool
20
+ produces:
21
+
22
+ **A. Standards-based OTLP JSON.** A compliant OTLP file exporter (for example the [OpenTelemetry
23
+ Collector's `file` exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib))
24
+ writes the real wire protocol: `resourceSpans[].scopeSpans[].spans[]` or
25
+ `resourceLogs[].scopeLogs[].logRecords[]`, with attributes as
26
+ `[{key, value: {stringValue|intValue|doubleValue|boolValue}}]`. Usage lives under
27
+ `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` / `gen_ai.usage.cache_read.input_tokens`
28
+ / `gen_ai.usage.cache_write.input_tokens` / `gen_ai.usage.reasoning.output_tokens`, per the
29
+ [GenAI spans semantic conventions](https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/main/docs/gen-ai/gen-ai-spans.md).
30
+ This is the shape any *other* instrumented tool should target if you want it to show up here.
31
+
32
+ **B. Gemini CLI's own file telemetry.** Verified against Gemini CLI's source
33
+ (`packages/core/src/telemetry/file-exporters.ts`), Gemini CLI does **not** write the OTLP wire
34
+ format to its outfile. Its `FileLogExporter`/`FileSpanExporter` pretty-print
35
+ (`JSON.stringify(record, null, 2)`) the OpenTelemetry JS SDK's own internal `ReadableLogRecord`/
36
+ `ReadableSpan` objects, one JSON value per flush — so a "record" in that file can span many text
37
+ lines, and the file is not, strictly, JSON Lines. This adapter's `ingestFile` scans for balanced
38
+ top-level JSON values by brace/bracket depth (not by newline), so both shapes are read correctly
39
+ from the same file, including a file that is a single JSON document with no line breaks at all.
40
+
41
+ ## Enabling Gemini CLI's file telemetry
42
+
43
+ Quoting the setting names from Gemini CLI's own docs
44
+ (`docs/cli/telemetry.md`, `GEMINI_TELEMETRY_*` env vars):
45
+
46
+ ```json
47
+ {
48
+ "telemetry": {
49
+ "enabled": true,
50
+ "target": "local",
51
+ "outfile": ".gemini/telemetry.log"
52
+ }
53
+ }
54
+ ```
55
+
56
+ in `.gemini/settings.json` (or the equivalent `GEMINI_TELEMETRY_ENABLED=true`,
57
+ `GEMINI_TELEMETRY_TARGET=local`, `GEMINI_TELEMETRY_OUTFILE=...` environment variables). The docs'
58
+ own worked example uses `.gemini/telemetry.log`, a path **relative to the CLI's working
59
+ directory** — the setting has no fixed default (`-` in the settings table until you set it), so
60
+ there is no single home-level location TokenFlow can rely on. This adapter's best-effort default
61
+ looks for `~/.gemini/telemetry.log`, but the reliable setup is either:
62
+
63
+ - an **absolute** `outfile` path pointed at the drop folder below, or
64
+ - `sources.otel.paths` in your TokenFlow config, pointed at wherever your project's
65
+ `outfile` actually is.
66
+
67
+ ```yaml
68
+ sources:
69
+ otel:
70
+ paths: ["~/code/my-project/.gemini/telemetry.log"]
71
+ ```
72
+
73
+ Gemini CLI logs two records per API call to the same file when telemetry is on: the
74
+ `gemini_cli.api_response` event (all the token fields) and a semantic twin,
75
+ `gen_ai.client.inference.operation.details`, that repeats the same usage under the
76
+ standards-based attribute names. This adapter recognizes the pairing and counts the call once,
77
+ from `gemini_cli.api_response` only.
78
+
79
+ ## Pointing another tool's OTLP file exporter here
80
+
81
+ Any tool whose OpenTelemetry SDK setup lets you choose a file-based (or a `file://`) exporter for
82
+ traces or logs can use the same drop folder:
83
+
84
+ ```
85
+ ~/.tokenflow/otel/<anything>.jsonl
86
+ ```
87
+
88
+ Anything ending in `.jsonl`, `.ndjson`, `.json`, or `.log` under that folder is picked up. Point
89
+ the OTel Collector's `file` exporter, or an OTLP/HTTP exporter aimed at a local static file
90
+ server that happens to append to disk, at a file under that folder, and it becomes a TokenFlow
91
+ source with no code change here — that is the point of reading the standards-based shape.
92
+
93
+ ## Token semantics
94
+
95
+ **Standards-based (`gen_ai.usage.*`).** Per the GenAI semantic conventions' own notes,
96
+ `gen_ai.usage.input_tokens` is documented to *include* both `cache_read.input_tokens` and
97
+ `cache_write.input_tokens` (the same inclusive convention OpenAI/Codex uses) — this adapter
98
+ subtracts both to get the schema's exclusive `input_tokens`. `reasoning.output_tokens` is
99
+ documented as already included in `output_tokens`, so it is copied through unmodified as a
100
+ subset.
101
+
102
+ **Gemini's own event.** Verified against
103
+ [the Generate Content API reference](https://ai.google.dev/api/generate-content):
104
+ `promptTokenCount` "is still the total effective prompt size ... includes the number of tokens
105
+ in the cached content" (inclusive of the cache, like the standards-based path), and
106
+ `totalTokenCount` is "prompt + thoughts + response candidates" — three **separate**, additive
107
+ terms. That second fact matters: it means the candidate/output count does NOT already include
108
+ thinking tokens, so to keep `reasoning_tokens` a subset of `output_tokens` (a schema invariant),
109
+ this adapter reports `output_tokens = output_token_count + thoughts_token_count` and
110
+ `reasoning_tokens = thoughts_token_count`.
111
+
112
+ **Left unverified, on purpose.** Whether `toolUsePromptTokenCount` is already folded into
113
+ `promptTokenCount`, or is additive and simply omitted from the documented `totalTokenCount`
114
+ formula, is not stated anywhere in the sources checked for this adapter. `tool_token_count` is
115
+ therefore recorded only in `metadata` (for a manual audit against `total_token_count`), never
116
+ added into `input_tokens` or `output_tokens`.
117
+
118
+ ## Privacy
119
+
120
+ OTel GenAI attributes, and Gemini CLI's own events, can carry full prompt/response text
121
+ (`gen_ai.input.messages`, `gen_ai.output.messages`, `response_text`, a human-readable log `body`
122
+ describing the call). This adapter never reads those keys, and never spreads a source record's
123
+ attributes into `metadata` — every metadata field is written one at a time, from an explicit
124
+ allow-list (`model`/`provider`/`operation`/`status`/`session_id`/`conversation_id`, matching this
125
+ product's data-collection rule for OTel sources specifically). Fields present on the source but
126
+ NOT retained include `auth_type`, `finish_reasons`, `installation.id`, and `user.email` — those
127
+ are on the wire, but not on the allow-list, so they are dropped even though nothing forces you to
128
+ turn on `telemetry.logPrompts: false` upstream.
129
+
130
+ ## `docs/providers.md` integration
131
+
132
+ Row for the provider table (`docs/providers.md` is not edited by this change — see the
133
+ CHANGELOG/report for the exact snippet a maintainer can paste in):
134
+
135
+ ````markdown
136
+ ## `otel` — OpenTelemetry (GenAI)
137
+
138
+ | | |
139
+ |---|---|
140
+ | Reads | `~/.gemini/telemetry.log` (best-effort) and `~/.tokenflow/otel/*.{jsonl,ndjson,json,log}` |
141
+ | Measurement | `primary` |
142
+ | Reports | `input`, `output`, `cache_read`, `cache_write`, `reasoning`, session/conversation id, request id, duration, model, provider |
143
+ | Cannot know | interface/surface (no OTel GenAI attribute carries it) |
144
+
145
+ Standards-based: point any tool's OTLP file/collector exporter at the drop folder. Gemini-CLI
146
+ specific: enable `telemetry.enabled`/`target: "local"`/`outfile` in `.gemini/settings.json`; see
147
+ [docs/providers-otel.md](providers-otel.md) for the field mapping, the double-count-avoidance
148
+ rule for Gemini's paired events, and the privacy allow-list.
149
+
150
+ ### Configuration
151
+
152
+ ```yaml
153
+ sources:
154
+ otel:
155
+ paths: ["~/code/my-project/.gemini/telemetry.log", "~/.tokenflow/otel"]
156
+ ```
157
+
158
+ ---
159
+ ````
160
+
161
+ ## Tools with no known on-disk usage log (needs a sample)
162
+
163
+ These tools were not verified for this adapter — either they have no documented local usage
164
+ export, or checking would have required an account/install this task didn't have. If you run one
165
+ of these and can point it at an OTLP file exporter (or find where it already writes per-request
166
+ token counts), a fixture line from a **synthetic** run is what turns this from a guess into a
167
+ supported source.
168
+
169
+ | Tool | Status |
170
+ |---|---|
171
+ | Aider | No documented on-disk per-request usage log found; Aider prints session totals to the terminal. Needs a sample. |
172
+ | Ollama | No documented usage-log file; token counts are only in the API response body, not persisted. Needs a sample. |
173
+ | LM Studio | No documented on-disk usage log found. Needs a sample. |
174
+ | Zed | Connects to Gemini CLI via ACP (surface tag `zed` per Gemini CLI's telemetry docs) but Zed itself was not checked for its own OTel/usage export. Needs a sample. |
175
+ | Windsurf / Codeium | No documented on-disk usage log found. Needs a sample. |
176
+ | JetBrains AI Assistant | Gemini CLI's ACP integration reports a `jetbrains` surface tag, but the JetBrains plugin's own usage/telemetry export was not checked. Needs a sample. |
177
+ | Continue | No documented on-disk usage log found. Needs a sample. |
178
+ | Copilot CLI | No documented on-disk usage log found. Needs a sample. |
179
+ | Antigravity | No public documentation found for this task; needs a sample or a pointer to its docs. |
package/docs/providers.md CHANGED
@@ -57,7 +57,7 @@ sources:
57
57
  |---|---|
58
58
  | Reads | `~/.codex/sessions/**/*.jsonl` and `~/.codex/archived_sessions/**` (`$CODEX_HOME` respected) |
59
59
  | Measurement | `primary` |
60
- | Reports | fresh `input`, `cache_read`, `cache_write`, `output`, `reasoning`, session, thread, turn id, project (from `cwd`), interface (from `source` / `originator`), gateway, reasoning effort, service tier, context window, time-to-first-token, subagent role |
60
+ | Reports | fresh `input`, `cache_read`, `cache_write`, `output`, `reasoning`, session, thread, turn id, project (from `cwd`), git branch and repository (from `session_meta.git`, CLI 0.149+), interface (from `source` / `originator`), gateway, reasoning effort, service tier, context window, time-to-first-token, subagent role |
61
61
  | Cannot know | cost; `cache_write` on older CLI builds that never emitted the field (recorded as `null`, not `0`) |
62
62
 
63
63
  ### Token semantics
@@ -95,6 +95,15 @@ reconstruction is auditable rather than a hidden fudge.
95
95
  `model_provider: "headroom"` (or any non-vendor value) is recorded as `gateway`, not as the
96
96
  vendor. The vendor comes from the model name.
97
97
 
98
+ ### Repository identity from `session_meta.git`
99
+
100
+ Recent CLIs (cli_version 0.149+ observed) include a `git` block on `session_meta`:
101
+ `{ commit_hash, branch, repository_url }`. `branch` becomes `git_branch`; `repository_url` is
102
+ reduced to its basename with any `.git` suffix stripped and becomes `repository`, falling back to
103
+ the cwd basename (same as `project`) when the block is absent — the pre-existing behaviour. Only
104
+ the derived name is kept: the commit hash and the URL itself are never stored, and no message
105
+ content is read.
106
+
98
107
  ---
99
108
 
100
109
  ## `cline` — Cline CLI
@@ -224,6 +233,50 @@ estimate instead of a single unverifiable number.
224
233
 
225
234
  ---
226
235
 
236
+ ## `otel` — OpenTelemetry (GenAI)
237
+
238
+ | | |
239
+ |---|---|
240
+ | Reads | `~/.gemini/telemetry.log` (best-effort default) and `~/.tokenflow/otel/*.{jsonl,ndjson,json,log}` |
241
+ | Measurement | `primary` |
242
+ | Reports | `input`, `output`, `cache_read`, `cache_write` (standards-based path only), `reasoning`, session/conversation id, request id (span id), duration, model, provider |
243
+ | Cannot know | interface/surface (no OTel GenAI attribute carries it) |
244
+
245
+ Standards-based: point any tool's OTLP file/collector exporter at the drop folder
246
+ (`~/.tokenflow/otel/`) and it becomes a source with no code change here. Gemini-CLI specific:
247
+ enable `telemetry.enabled` / `target: "local"` / `outfile` in `.gemini/settings.json`; this
248
+ adapter recognizes and de-duplicates Gemini's paired usage events (`gemini_cli.api_response` and
249
+ its standards-based twin) so a call is counted once.
250
+
251
+ ```yaml
252
+ sources:
253
+ otel:
254
+ paths: ["~/code/my-project/.gemini/telemetry.log", "~/.tokenflow/otel"]
255
+ ```
256
+
257
+ See [providers-otel.md](providers-otel.md) for the field mapping, the two on-disk shapes this
258
+ adapter reads, and the privacy allow-list.
259
+
260
+ ### Tools with no known on-disk usage log (needs a sample)
261
+
262
+ These were not verified for this adapter, either because they have no documented local usage
263
+ export or because checking would have needed an account this project does not have. A fixture
264
+ line from a **synthetic** run turns any of these from a guess into a supported source:
265
+
266
+ | Tool | Status |
267
+ |---|---|
268
+ | Aider | No documented on-disk per-request usage log found; prints session totals to the terminal. |
269
+ | Ollama | No documented usage-log file; token counts are only in the API response body. |
270
+ | LM Studio | No documented on-disk usage log found. |
271
+ | Zed | Connects to Gemini CLI via ACP (surface tag `zed`); Zed's own OTel/usage export not checked. |
272
+ | Windsurf / Codeium | No documented on-disk usage log found. |
273
+ | JetBrains AI Assistant | Gemini CLI's ACP integration reports a `jetbrains` surface tag; the plugin's own export not checked. |
274
+ | Continue | No documented on-disk usage log found. |
275
+ | Copilot CLI | No documented on-disk usage log found. |
276
+ | Antigravity | No public documentation found; needs a sample or a pointer to its docs. |
277
+
278
+ ---
279
+
227
280
  ## `git` — activity correlation
228
281
 
229
282
  | | |
@@ -0,0 +1,74 @@
1
+ # Receipt schema v0
2
+
3
+ A branch receipt that travels with the code — no server. `tokenflow hooks install`
4
+ attaches one to the local commit sha on every `git push`, as a git note under
5
+ `refs/notes/tokenflow`. The [GitHub Action](../action/README.md) reads that note back
6
+ on a pull request and posts (or updates) it as a comment.
7
+
8
+ The machine-readable shape is `schemas/receipt.v0.json` (JSON Schema, draft 2020-12).
9
+ `src/analytics/receipt-schema.js` provides:
10
+
11
+ - `toReceiptV0(branchReceipt, meta)` — maps one `buildReceipts()` branch entry into
12
+ this shape.
13
+ - `validateReceiptV0(obj)` — a small hand-written validator (no dependencies),
14
+ returns `{ ok, errors[] }`.
15
+ - `renderReceiptV0Markdown(receipt)` — the PR-comment table, with a leading
16
+ `<!-- tokenflow-receipt -->` marker so CI can find and update its own comment.
17
+
18
+ ## The estimated-vs-measured caveat
19
+
20
+ **Every dollar figure in this receipt is an estimate**, computed locally from token
21
+ counts against a local price table (`~/.tokenflow/pricing.json` plus TokenFlow's
22
+ built-in table) — never measured billing pulled from a vendor, and never a network
23
+ call. Two things follow from that:
24
+
25
+ - A model with no configured price contributes turns (counted in `turns`,
26
+ `subagentTurns`, `sessions`) but not dollars — `costUsd` reflects only the priced
27
+ turns, and `coverage` says what share of turns that was. A branch with **zero**
28
+ priced turns reports `costUsd: null`, never `0`.
29
+ - `contextShare` splits `costUsd` into what paid to re-send prior context (cache
30
+ reads + writes) versus fresh input/output. A high share is not waste — it is the
31
+ price of a long session — but it is where the lever is if you want a cheaper one.
32
+
33
+ `notes` carries these caveats (and others — a long-lived branch, work that predates
34
+ a matched pull request, earlier merged PRs on the same branch) as plain sentences,
35
+ so a reader of the JSON alone — not just the markdown table — sees them.
36
+
37
+ ## Fields
38
+
39
+ | Field | Type | Nullable | Meaning |
40
+ |---|---|---|---|
41
+ | `schemaVersion` | integer (`0`) | no | Which version of this document the receipt follows. |
42
+ | `generatedAt` | string (date-time) | no | When this receipt was computed, in UTC ISO-8601. |
43
+ | `toolVersion` | string | no | The TokenFlow package version that computed it. |
44
+ | `repo` | string | no | The repository's name — the basename of its **main checkout**, resolved through git worktrees (never a full path, never a worktree leaf's own directory name). |
45
+ | `branch` | string | no | The branch this receipt attributes spend to. |
46
+ | `headSha` | string | no | The commit sha this receipt is attached to (7-40 lowercase hex characters). |
47
+ | `window` | object or `null` | yes | `{ first, last }` — the first and last timestamp of a priced turn on this branch. `null` when there were none. |
48
+ | `window.first` / `window.last` | string (date-time) or `null` | yes | Endpoints of the window. |
49
+ | `costUsd` | number | yes | Estimated spend attributed to this branch, in US dollars. `null` — never `0` — when nothing on this branch was priced. |
50
+ | `contextShare` | number (0–1) | yes | Share of `costUsd` that re-sent prior context rather than doing fresh work. |
51
+ | `turns` | integer | no | Turns (LLM requests) attributed to this branch, priced or not. |
52
+ | `sessions` | integer | no | Distinct sessions that touched this branch. |
53
+ | `subagentTurns` | integer | no | Of `turns`, how many ran as a subagent. |
54
+ | `models` | array | no | `{ model, costUsd, share }` per model family, most expensive first. |
55
+ | `coverage` | number (0–1) | yes | Share of `turns` priced (i.e. counted in `costUsd`). |
56
+ | `largestPromptTokens` | integer | yes | The largest single prompt (input + cache read + cache write) seen on this branch. |
57
+ | `changedLines` | integer | yes | Additions + deletions of the matched pull request. `null` with no matched PR. |
58
+ | `pr` | object or `null` | yes | `{ number, mergedAt }` of the pull request this receipt's headline is scoped to. |
59
+ | `longLived` | boolean | no | True for branch names like `main` / `staging` / `develop` — this receipt covers a period of work, not one change. |
60
+ | `costPer100Lines` | number | yes | `costUsd` scaled to 100 changed lines. `null` with no matched PR, no changed lines, or no priced turns. |
61
+ | `notes` | array of strings | no | Honesty caveats — see above. |
62
+
63
+ ## Design notes
64
+
65
+ - `repo` is the repository's **name only** — never a filesystem path — because the
66
+ receipt is meant to be read on a different machine (a CI runner, a teammate's
67
+ laptop) than the one that produced it.
68
+ - A receipt with no local sessions for a branch is not produced at all
69
+ (`buildBranchReceipt()` in `src/core/receipt-note.js` returns `null`); there is
70
+ nothing to attach as a note.
71
+ - `pr` and `changedLines` are `null` for every receipt written by the pre-push hook,
72
+ because attaching a note happens at push time, before any PR exists to look up —
73
+ they are populated only when a receipt is built through `tokenflow receipt --gh`
74
+ (or an equivalent supplied `--prs` list) and re-serialized through `toReceiptV0()`.