@ponbac/token-ledger 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Pontus Backman
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md ADDED
@@ -0,0 +1,418 @@
1
+ # Token Ledger
2
+
3
+ Local token consumption and estimated API cost, grouped by project across coding agents.
4
+
5
+ An Effect v4 core reads provider histories and produces structured reports. An Effect
6
+ CLI handles discovery, configuration, and terminal/CSV/JSON output. You do not need
7
+ T3 Code, provider API keys, or a company dashboard to use it.
8
+
9
+ **Dollar figures are API-equivalent estimates, not subscription expenses.** Reports
10
+ keep input, cache-read, cache-write, and output tokens separate. Unknown model or
11
+ cache prices stay unknown; the known subtotal remains available.
12
+
13
+ ## Quick start
14
+
15
+ Requires **Node.js 24 or newer**. Run without cloning or building:
16
+
17
+ ```sh
18
+ npx @ponbac/token-ledger@latest report --provider copilot
19
+ pnpx @ponbac/token-ledger@latest report --provider copilot
20
+ bunx @ponbac/token-ledger@latest report --provider copilot
21
+ ```
22
+
23
+ These are equivalent alternatives; choose the package manager you already use.
24
+ The npm package contains a bundled CLI with no runtime dependencies or install
25
+ scripts. `bunx` uses the Node shebang, so Node 24+ is still required.
26
+
27
+ The default window is the current calendar month through today, in UTC. Read every
28
+ supported source by omitting `--provider`. Configure Copilot collection before
29
+ starting sessions as described below; installing the CLI does not enable telemetry.
30
+
31
+ ```sh
32
+ npx @ponbac/token-ledger@latest report --since 2026-09-01 --until 2026-09-30 --format csv > usage.csv
33
+ npx @ponbac/token-ledger@latest report --provider codex --json > report.json
34
+ npx @ponbac/token-ledger@latest config-example > token-ledger.json
35
+ ```
36
+
37
+ `--json` is a shortcut for `--format json` and takes precedence over `--format`.
38
+ JSON stdout contains only the structured report, including coverage and pricing
39
+ metadata; diagnostics go to stderr. Unknown costs are `null`, not zero.
40
+ Terminal reports sort projects by descending API estimate, with color when supported.
41
+ Set `NO_COLOR=1` for plain output. Run `--help` for options.
42
+
43
+ ### Run from source
44
+
45
+ Install [Bun](https://bun.sh), then:
46
+
47
+ ```sh
48
+ git clone https://github.com/ponbac/token-ledger.git
49
+ cd token-ledger
50
+ bun install --frozen-lockfile
51
+ bun run dev report --provider codex
52
+ ```
53
+
54
+ Examples below use `bun run dev`; replace that prefix with
55
+ `npx @ponbac/token-ledger@latest`, `pnpx @ponbac/token-ledger@latest`, or `bunx @ponbac/token-ledger@latest`
56
+ when using the published package.
57
+
58
+ Build a standalone Node executable and the npm package directory:
59
+
60
+ ```sh
61
+ bun run build
62
+ node packages/cli/dist/main.js report --provider codex
63
+ ```
64
+
65
+ ### Windows
66
+
67
+ The packaged CLI is tested in CI on Windows, macOS, and Linux using Node.js 24.
68
+ Those checks cover package installation and reports from synthetic histories; live
69
+ provider discovery has been verified on Linux. To run from source on Windows with
70
+ Git and Bun installed, use PowerShell:
71
+
72
+ ```powershell
73
+ git clone https://github.com/ponbac/token-ledger.git
74
+ cd token-ledger
75
+ bun install --frozen-lockfile
76
+ bun run dev report
77
+ ```
78
+
79
+ To build with Bun and run with Node.js:
80
+
81
+ ```powershell
82
+ bun run build
83
+ node packages/cli/dist/main.js report
84
+ ```
85
+
86
+ ## Collection coverage
87
+
88
+ | Provider | Default input | Coverage |
89
+ | -------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------- |
90
+ | Codex | `~/.codex/sessions` and `archived_sessions` | Rollout token events, model switches, session and working-directory metadata |
91
+ | Claude Code | `~/.claude/projects` | Assistant usage records; repeated message blocks counted once |
92
+ | Grok Build | `~/.grok/sessions/**/updates.jsonl` | Saved completed turns and their model breakdowns |
93
+ | Copilot CLI / VS Code Chat | `~/.copilot/otel` or `COPILOT_OTEL_FILE_EXPORTER_PATH` | JSONL request spans, VS Code inference events, and SQLite span exports |
94
+
95
+ `CODEX_HOME`, `CLAUDE_CONFIG_DIR`, `GROK_HOME`, and `COPILOT_HOME` override the usual
96
+ homes. Configuration supports additional accounts, machines' exported histories,
97
+ and arbitrary source directories. Histories are opened read-only, including work
98
+ performed outside T3 Code.
99
+
100
+ Copilot CLI requires enabling its own file export **before** a session starts:
101
+
102
+ ```sh
103
+ mkdir -p ~/.copilot/otel
104
+ export COPILOT_OTEL_FILE_EXPORTER_PATH="$HOME/.copilot/otel/usage.jsonl"
105
+ copilot
106
+ ```
107
+
108
+ On Windows, use PowerShell:
109
+
110
+ ```powershell
111
+ New-Item -ItemType Directory -Force "$HOME\.copilot\otel"
112
+ $env:COPILOT_OTEL_FILE_EXPORTER_PATH = "$HOME\.copilot\otel\usage.jsonl"
113
+ copilot
114
+ ```
115
+
116
+ Setting the file path automatically enables telemetry export to that local file;
117
+ no separate toggle or telemetry server is needed. Prompt and response content
118
+ capture is off by default.
119
+
120
+ The commands above set the variable for the current shell. To persist it for future
121
+ Windows terminals, also run:
122
+
123
+ ```powershell
124
+ [Environment]::SetEnvironmentVariable(
125
+ "COPILOT_OTEL_FILE_EXPORTER_PATH",
126
+ "$HOME\.copilot\otel\usage.jsonl",
127
+ "User"
128
+ )
129
+ ```
130
+
131
+ On Bash or Zsh, add the `export` line to your shell startup file to persist it.
132
+
133
+ Then run `bun run dev report --provider copilot`. This does not backfill earlier
134
+ sessions. Only inference requests are counted, so parent summaries and metric exports
135
+ cannot add the same usage twice. Both documented dotted and older underscored
136
+ cache-token attributes are accepted. See the [Copilot CLI reference](https://docs.github.com/en/copilot/reference/copilot-cli-reference/cli-command-reference#opentelemetry-monitoring).
137
+
138
+ ### VS Code Copilot Chat
139
+
140
+ In your active VS Code profile's user settings, enable local file export:
141
+
142
+ ```json
143
+ {
144
+ "github.copilot.chat.otel.enabled": true,
145
+ "github.copilot.chat.otel.exporterType": "file",
146
+ "github.copilot.chat.otel.outfile": "/absolute/path/to/.copilot/otel/vscode-chat.jsonl",
147
+ "github.copilot.chat.otel.captureContent": false
148
+ }
149
+ ```
150
+
151
+ Replace the output path with an absolute path on your machine, create its parent
152
+ directory, reload VS Code, and start a new chat. On Windows, use a path such as
153
+ `C:/Users/you/.copilot/otel/vscode-chat.jsonl`. See the
154
+ [VS Code monitoring documentation](https://code.visualstudio.com/docs/agents/guides/monitoring-agents).
155
+
156
+ Read that export, or combine CLI and Chat files from the same directory:
157
+
158
+ ```sh
159
+ bun run dev report --provider copilot --source ~/.copilot/otel/vscode-chat.jsonl
160
+ bun run dev report --provider copilot --source ~/.copilot/otel
161
+ ```
162
+
163
+ Both use the `copilot` provider. Default discovery reads `~/.copilot/otel`, unless
164
+ `COPILOT_OTEL_FILE_EXPORTER_PATH` selects a specific file; `--source` overrides it.
165
+ VS Code inference events require a response ID, timestamp, and input/output token
166
+ counts. Missing or invalid fields are reported as skipped usage. Response IDs
167
+ deduplicate copied events and matching request spans. Agent turn summaries,
168
+ notifications, and metrics are ignored. Background inference calls, such as title
169
+ generation, are included when exported as inference events.
170
+
171
+ ### Automatic project attribution
172
+
173
+ One shared export directory can contain usage from multiple projects. Token Ledger
174
+ reads Copilot CLI spans and the corrected VS Code JSONL span format, follows parent
175
+ spans to `github.copilot.git.repository` (or `copilot_chat.repo.remote_url`), and
176
+ matches inference events by trace/span context or response ID. Parent spans can
177
+ arrive later or in another file within the same source directory. Matching request
178
+ spans supply cache-token details that inference logs may omit; the request is
179
+ counted once. Agent summaries are never counted as additional requests.
180
+
181
+ This uses the repository remote, not a local directory. Configure repository
182
+ mappings below to choose project names. Requests without usable repository linkage,
183
+ including some background calls and folders without a remote, remain `Unassigned`.
184
+ Multi-root workspaces follow the repository Copilot attaches to each agent span;
185
+ tokens are not split between folders. A fixed `--project` overrides attribution for
186
+ an export belonging to one project.
187
+
188
+ ### Workaround for VS Code's file exporter bug
189
+
190
+ VS Code 1.139.1 can write spans as empty `{}` lines, losing the metadata needed for
191
+ attribution while inference logs still contain token counts. Microsoft's
192
+ [fix](https://github.com/microsoft/vscode/pull/337373) is marked released in Insiders
193
+ and targets 1.140 in the [tracking issue](https://github.com/microsoft/vscode/issues/319993).
194
+ Token Ledger supports the corrected format, but cannot reconstruct spans already
195
+ lost from an export.
196
+
197
+ On affected builds, enable the **built-in SQLite span exporter** alongside the file
198
+ settings above:
199
+
200
+ ```json
201
+ {
202
+ "github.copilot.chat.otel.dbSpanExporter.enabled": true,
203
+ "github.copilot.chat.otel.captureContent": false
204
+ }
205
+ ```
206
+
207
+ Reload the VS Code window, use Copilot Chat, then run **Chat: Export Agent Traces DB**
208
+ from the command palette. Save it as `~/.copilot/otel/agent-traces.db` alongside the
209
+ JSONL files, and report the whole directory:
210
+
211
+ ```sh
212
+ bun run dev report --provider copilot --source ~/.copilot/otel
213
+ ```
214
+
215
+ Token Ledger opens `.db` files read-only and selects only accounting attributes.
216
+ The exported database is a snapshot; export again to include subsequent sessions.
217
+ For ongoing collection you can instead read VS Code's live `agent-traces.db` with
218
+ `--source`, or symlink it into the shared export directory on systems that support
219
+ symlinks. For example, with the usual Linux stable storage path:
220
+
221
+ ```sh
222
+ ln -s "$HOME/.config/Code/User/globalStorage/github.copilot-chat/agent-traces.db" \
223
+ "$HOME/.copilot/otel/vscode-spans.db"
224
+ ```
225
+
226
+ Its location is under the active VS Code profile's extension global
227
+ storage (`github.copilot-chat`); custom profiles, remote hosts, and Insiders use
228
+ different storage roots. Resolve a symlink to the live database rather than copying
229
+ an open database without its WAL; the built-in export command flushes and checkpoints
230
+ before copying.
231
+
232
+ Keep related JSONL and database files in **one source directory** to correlate and
233
+ deduplicate them before project attribution. Separately configured sources use
234
+ first-source-wins deduplication. Export paths and custom OTel resource attributes
235
+ are application-scoped VS Code settings; a fixed project value is unsuitable for
236
+ simultaneous usage across projects.
237
+
238
+ Unknown model prices remain unknown. Autocomplete, other IDEs, and Copilot
239
+ session-state history import are outside this release. Copilot imports are covered
240
+ by synthetic fixtures and live CLI, VS Code JSONL, and SQLite export checks; database coverage
241
+ starts when its exporter is enabled.
242
+
243
+ ## Project attribution
244
+
245
+ Create an optional configuration:
246
+
247
+ ```sh
248
+ bun run dev config-example > token-ledger.json
249
+ ```
250
+
251
+ The default filename is `./token-ledger.json`; select another with `--config`.
252
+
253
+ ```json
254
+ {
255
+ "projects": [
256
+ {
257
+ "project": "Client A",
258
+ "paths": ["~/work/client-a"],
259
+ "repositories": [
260
+ "https://github.com/example/client-a-app.git",
261
+ "git@github.com:example/client-a-api.git"
262
+ ]
263
+ }
264
+ ]
265
+ }
266
+ ```
267
+
268
+ Attribution order:
269
+
270
+ 1. `--project` or the source's fixed `project`.
271
+ 2. The most specific matching directory prefix.
272
+ 3. A matching repository mapping.
273
+ 4. The Git remote, Git root, or recorded working directory.
274
+ 5. `Unassigned` when there is no usable metadata.
275
+
276
+ SSH and HTTPS remotes share a canonical identity. Local worktrees resolve through
277
+ their common Git directory. Relative config paths resolve from the config file;
278
+ `~/` resolves from the current developer's home. Directory mappings respect path
279
+ boundaries, so `/work/app` never matches `/work/application`.
280
+
281
+ Grok logs currently lack project metadata in the usage parser. Copilot CLI can put
282
+ repository metadata on a parent agent span rather than the counted request span;
283
+ Token Ledger associates the two. If an export has no repository metadata, use a
284
+ fixed source project:
285
+
286
+ ```json
287
+ {
288
+ "sources": [
289
+ { "provider": "codex", "path": "~/.codex/sessions" },
290
+ { "provider": "copilot", "path": "~/exports/client-a.jsonl", "project": "Client A" }
291
+ ]
292
+ }
293
+ ```
294
+
295
+ An explicit `sources` array replaces default discovery. Source order matters for
296
+ overlapping files: the first configured source owns their project attribution.
297
+
298
+ ## Prices and exports
299
+
300
+ The CLI fetches the public LiteLLM model-rate table and caches the decoded snapshot
301
+ for 24 hours under `$XDG_CACHE_HOME/token-ledger` (default `~/.cache/token-ledger`).
302
+ Only this public pricing request uses the network; histories are processed locally.
303
+ `--offline` uses cached prices and config overrides. A failed refresh falls back to
304
+ cached prices, with the snapshot timestamp retained in JSON.
305
+
306
+ Override exact model IDs in USD per million tokens:
307
+
308
+ ```json
309
+ {
310
+ "prices": {
311
+ "my-model": { "input": 2, "output": 10, "cacheRead": 0.2, "cacheWrite": null }
312
+ }
313
+ }
314
+ ```
315
+
316
+ `null` means unknown, while `0` explicitly means free. Rates are base-tier estimates
317
+ applied to the whole selected history; historical price changes, service tiers,
318
+ tool charges, taxes, negotiated discounts, and subscription allocations are not
319
+ modeled. Provider-reported credit or dollar figures do not override this estimate.
320
+
321
+ JSON is a versioned report with pricing provenance, applied unit rates, coverage diagnostics, and rows
322
+ grouped by project/day/provider/model. CSV exports those rows with empty estimated
323
+ cost cells when any usage is unpriced. Neither format includes prompts or responses;
324
+ JSON coverage does contain configured source paths. CSV escapes spreadsheet formula
325
+ prefixes in identifiers. Coverage diagnostics also go to stderr, leaving stdout
326
+ usable for piping.
327
+
328
+ Exit codes: `0` for a generated report, `1` for invalid input or a fatal error, and
329
+ `2` with `--strict` when a source is missing/partial/failed or tokens are unpriced.
330
+ Ordinary mode still produces a partial report when some providers are unavailable.
331
+ Keep developer identity alongside exported reports when combining them across a team.
332
+
333
+ ## Accuracy and limitations
334
+
335
+ - Reports reflect saved, readable local history, not an authoritative provider invoice.
336
+ - JSONL files are streamed, while Copilot buffers accounting records and span links
337
+ per source for attribution; SQLite imports read the selected accounting rows. Each
338
+ run rescans its sources, and memory grows with observed usage records.
339
+ - Copied files, overlapping source paths, and stable provider record IDs are deduplicated.
340
+ Missing stable IDs are called out where the provider cannot support that guarantee.
341
+ - Codex fork histories use T3 Code's timing heuristic to exclude the leading copied
342
+ burst. The report explicitly marks affected coverage as partial.
343
+ - A truncated or malformed JSONL line is skipped and counted. No prompts are printed
344
+ in diagnostics. An active session can change during a scan; this is not a transactional snapshot.
345
+ - Coverage diagnostics describe all scanned files, including events outside the selected
346
+ date window. Codex updates with no usage payload are ignored, not counted as lost usage.
347
+ - Default project inference uses the current local Git metadata when the transcript
348
+ lacks a remote. Explicit mappings make attribution more stable across machines.
349
+ - A session that works across projects still needs an explicit allocation policy;
350
+ tokens cannot identify which client benefited from an individual generated line.
351
+
352
+ ## Publishing
353
+
354
+ The workspace packages stay private. `bun run build` creates the publishable
355
+ package in `dist/npm`, copying only the bundled executable, manifest, README,
356
+ project license, and bundled dependency license notices. The CLI version comes
357
+ from `packages/cli/package.json`.
358
+
359
+ For manual publication:
360
+
361
+ 1. Log in with `npm login` using an account with publish access to `@ponbac/token-ledger`.
362
+ 2. Run `bun run check`. The CLI test packs the package, installs it offline outside
363
+ the repository with lifecycle scripts disabled, and exercises the installed CLI.
364
+ 3. Run `bun run pack` to create `dist/ponbac-token-ledger-0.1.0.tgz` (filename follows version).
365
+ 4. Inspect `npm publish ./dist/npm --dry-run`.
366
+ 5. Publish with `npm publish ./dist/npm --access public`; complete npm's authentication
367
+ prompt if requested.
368
+
369
+ For subsequent releases, update the CLI manifest version, commit it, and rerun the
370
+ checks. The manual **Publish npm package** GitHub Actions workflow also runs checks
371
+ before publishing from `main`. Configure an
372
+ [npm trusted publisher](https://docs.npmjs.com/trusted-publishers/) for GitHub owner
373
+ `ponbac`, repository `token-ledger`, workflow `publish.yml`, with direct publishing
374
+ allowed, before using it. This uses GitHub OIDC rather than a stored npm token.
375
+ An already published version cannot be reused.
376
+
377
+ ## Development
378
+
379
+ ```sh
380
+ bun run check
381
+ bun run fmt
382
+ bun run lint:fix
383
+ ```
384
+
385
+ - `packages/core`: Effect schemas, provider adapters, project attribution, pricing,
386
+ and the `Ledger` module. `Ledger.layer` receives filesystem/path dependencies;
387
+ `Ledger.report` returns data and typed failures, with no terminal rendering.
388
+ - `packages/cli`: Effect CLI and process wiring. A later UI can run the same core
389
+ through a local backend.
390
+ - Effect and platform packages are pinned to `4.0.0-rc.117`.
391
+ - Oxlint `1.80.0` with type-aware checks, Oxfmt `0.66.0`, and all 18 generic plus
392
+ five Effect anti-slop policies gate CI, including test code, without exemptions.
393
+ TypeScript owns duplicate declaration checking for schema/type pairs.
394
+ - Vendored anti-slop rules are pinned to upstream `c44ef22`; `main` was verified at
395
+ that revision on 2026-09-22. Provenance, local patches, and licenses live beside
396
+ the rules. The separate experimental Effect TSGo bridge is not installed.
397
+
398
+ ### Source references
399
+
400
+ `.reference/effect` and `.reference/t3code` are real squashed Git subtrees following
401
+ the [Effect team's recommendation](https://effect.website/blog/the-one-weird-git-trick-that-makes-coding-agents-more-effect-ive).
402
+ A normal clone includes them. They are read-only reference material, excluded from
403
+ application checks and editor indexing, and marked vendored for GitHub statistics.
404
+
405
+ ```sh
406
+ # Start with a clean working tree. Change the version deliberately when upgrading.
407
+ bun run reference:update effect 'effect@4.0.0-rc.117'
408
+ bun run reference:update t3code main
409
+ ```
410
+
411
+ Update Effect dependencies together with their reference. Git subtree commit
412
+ messages record the imported revisions. The initial T3 reference is upstream
413
+ `aff9318bf46beaf05cc7155b428d3f0b8711efd2`.
414
+
415
+ ## License
416
+
417
+ MIT. See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) and the licenses retained
418
+ inside each vendored reference and rule set.