tledger 0.2.1 → 0.3.0

package/README.md

@@ -1,171 +1,189 @@
 # Token Ledger
 
-Token Ledger is a small, local-only terminal dashboard for Codex usage. It
-shows ranked project bars, model mix, usage type, cache coverage, and reset
-cycle context without sending your data anywhere.
+Token Ledger is a local-only Codex usage dashboard with two outputs:
 
-## Install
+- a generated PNG report for trends, model composition, and weekly-meter
+  context;
+- a terminal dashboard for a fast project and token summary.
+
+This README describes the local checkout. It does not assume a hosted report
+or a published npm package.
 
-Requires Node.js 22.13 or newer.
+## Install
 
-Once published, install the CLI with:
+Requires Node.js 22.13 or newer. From this checkout:
 
 ```bash
-npm install -g tledger
+npm install
+npm install -g .
 ```
 
-For a one-time run without a permanent global install:
+The second command installs `tledger` on your PATH. From the checkout, you can
+use `npx tledger` instead.
+
+## Basics
+
+| What you need | Command |
+| --- | --- |
+| Quick guide | `tledger` |
+| Last 24 hours | `tledger 1d` |
+| Last 7 calendar days | `tledger week` |
+| Rolling 30 days | `tledger 30d` |
+| 7-day PNG report | `tledger report 7d` |
+| Cache-only PNG report | `tledger report 7d --cache-rate` |
+
+The main options are:
+
+- `--static` prints once instead of opening the terminal dashboard.
+- `--refresh` rebuilds the local usage cache.
+- `--image-output <file>` chooses where to save a PNG.
+- `--no-open` writes a PNG without opening it.
+- `--help-all` shows the complete command and option reference.
+
+`week` covers seven local calendar days ending on the selected day. `1d` is a
+rolling 24-hour view ending when the command starts. In a TTY, the project
+dashboard is interactive. `j`/`k` select a project; `q`, `Q`, `Esc`, or
+`Ctrl-C` exits. Enter does not inspect a project, and `d` / `w` / `m` do not
+change the range; choose the range in the command.
+
+## Cache and input controls
+
+`tledger report [Nd|Nw] --cache-rate` writes a separate, purpose-built cache
+report to `token-ledger-cache-report-<period>.png`. Its primary view is the
+input-token-weighted cache rate (`cached input / measured input`), shown
+across the selected period with input-volume context. A smaller model breakout
+shows which models account for that input and their individual cache rates.
+The cache report intentionally omits the general report's project, quota-meter,
+runway, and output-token sections. Cached input is clamped to input per event,
+and the report states how much token volume had a usable component breakdown.
+Use `--image-output`, `--image-width`, `--date`, `--tz`, and `--no-open` the
+same way as on the standard report.
+
+The default privacy-reduced snapshot is
+`~/.token-ledger/token-ledger-snapshot-v2.json.gz`. It is gzip-compressed,
+written atomically with mode `0600`, targets 12 MiB, and has a hard 16 MiB
+on-disk limit. Its expanded JSON representation also targets 48 MiB and has a
+64 MiB safety limit, so the old 93 MiB raw-cache behavior cannot recur on the
+default production path. Reads check the compressed size before loading and
+bound gzip expansion to the same 64 MiB JSON limit. The collector de-duplicates
+through a private temporary SQLite spool, then keeps exact recent calls while
+rolling older history into minute, hour, and day buckets. If a dense history
+approaches the target, it increases the bucket resolution automatically while
+preserving additive token, model, project, cache, tool-call, and thread totals.
+When a compacted bucket crosses a requested range or chart boundary, Token
+Ledger allocates its additive values proportionally across the overlap and
+marks the terminal result as estimated; exact recent calls remain exact. The
+temporary spool is removed when collection completes or exits with a handled
+error.
+
+If even the coarsest bounded representation exceeds the hard limit, Token
+Ledger preserves the previous cache and asks you to reduce the source with
+`--no-archived` or the collector's `--since` option. It never replaces the
+production cache with an oversized or partial file. On a normal default-path
+run, a snapshot whose mtime is in the past and less than one hour old skips the
+source walk. An exact-hour or future mtime is not fresh; an older snapshot is
+checked against local source mtimes before it is reused or rebuilt.
 
 ```bash
-npx tledger week
+# Force a rebuild from CODEX_HOME or ~/.codex
+tledger week --refresh
+
+# Read the existing default snapshot without a source-freshness check
+tledger week --no-refresh
+
+# Read an explicit snapshot without automatic freshness checks
+tledger week --input /path/to/token-ledger-snapshot-v2.json.gz
 ```
 
-Until the package is published, install it from this repository:
+`--refresh` rebuilds the default snapshot and cannot be combined with
+`--input` in this checkout. `--no-archived` excludes `archived_sessions` when a
+refresh occurs. The collector can also be run directly:
 
 ```bash
-git clone <repository-url>
-cd token-ledger
-npm install -g .
+node lib/token-ledger-importer.mjs --output /path/to/token-ledger-snapshot-v2.json.gz
 ```
 
-The package uses a small runtime image encoder for PNG report output, Node's
-built-in SQLite support, and reads Codex data directly from the local machine.
-
-## Run
+Explicit `.json` snapshots remain readable for fixtures and deliberate exports,
+but `.json.gz` is the bounded production cache format. After an upgrade, the
+new cache does not read or delete schema-v1 cache files; remove an old generated
+cache separately once the v2 cache is proven.
 
-The shortest useful command is:
+## Report versus CLI
 
-```bash
-tledger week
-```
+| | Generated report | Terminal dashboard |
+| --- | --- | --- |
+| Command | `tledger report 7d` or `tledger trend 7d --image` | `tledger week`, `tledger 1d`, or `tledger day <date>` |
+| Output | One PNG with a stat quad, weekly-meter pace/runway, daily token columns, an aligned cache-rate strip, top projects, and per-model cache rates | Interactive or static project rows with totals, shares, thread counts, model mix, usage type, cache split, and reset-cycle context |
+| Range | A selected local-calendar-day trend window, such as 7d or 2w | A calendar day/week or a rolling 24-hour/`Nd`/`Nw` window |
+| Estimate surface | Meter-derived burn and runway are called out; cache coverage comes from measured token breakdowns | The sidebar can show derived `View burn`; project detail includes rate-card credit shares when available |
 
-It defaults to today's seven-day window, the computer's local timezone, the
-top 10 projects, and the local Codex data directory. The default terminal view
-is interactive; press `q` or `esc` to exit.
+The report is the broader visual summary. The CLI does not open or generate a
+report unless you request the `report`, `trend --image`, or `--image` form.
 
-Other common views:
+## Examples
 
-```bash
-tledger 1d
-tledger 1d --static
-tledger day 2026-08-05
-tledger week --top 5
-tledger week --static
-tledger trend 7d --static
-tledger trend 7d --image --image-output artifacts/token-ledger-trend-7d.png
-tledger report 7d
-```
+### Seven-day report
 
-`tledger 1d` shows the `TOKENS BY PROJECT` breakdown for the rolling 24 hours
-ending when the command starts. It is different from `tledger day today`, which
-covers the current calendar day from local midnight.
-
-`tledger report [7d|14d|30d]` is the one-step report output: it writes the
-dashboard PNG (identical to `trend --image`) to
-`token-ledger-report-<period>.png` in the current directory. It accepts the
-same flags as the trend view (`--drain`, `--date`, `--tz`, `--image-output`,
-`--image-width`) and prints progress while rendering and encoding the image.
-
-The terminal trend view is a compact approximation of the image view. For the
-full chart grammar, use `--image`: it writes a PNG with two aligned panels
-sharing one time axis — the observed weekly meter as a line on its own 0–100%
-panel on top, and calendar-day columns of local token volume stacked by model
-below, each annotated with the observed drop ("−18.1%" means the meter fell
-18.1 points across that column's days). The drain numbers come straight from
-the meter — no pricing model involved — so token volume and actual limit
-consumption can be compared per day at a glance.
-
-Turns run in fast mode (service tier "priority") are drawn as a darker shade
-at the top of their model's segment, with each model's fast share in the
-legend; fast-mode turns are weighted 1.5× in the credit estimate because they
-debit the plan limit at a higher rate.
-
-Pass `--drain` to flip the columns into limit-drain units instead: each column
-becomes the weekly limit percentage the meter dropped, stacked by model using
-rate-card credit weights (an estimate — the official card is the best
-available proxy for per-model debit, but subscription limits are not billed
-per token), on the same percent scale as the meter line.
-
-Meter windows are keyed by their server-reported reset timestamp, so a fresh
-window that starts days early (a provider-initiated limit restart) is drawn as
-its own cycle and labeled `restart`, while a true weekly expiry is labeled
-`reset`.
-Stale readings from sessions still reporting a superseded window are dropped
-instead of being fused into the line as phantom drain. Drops observed after a
-sparse meter gap (over 36 hours) are spread across the covered days and marked
-with `≈`. When a range has no usable meter drain, columns fall back to raw
-local token counts and the chart says so.
-
-`--image` defaults to `token-ledger-trend-7d.png` in the current directory.
-Use `--image-output <file.png>` to choose the path and `--image-width <px>` to
-choose a width from 900 to 2400 pixels.
-
-Use `trend 14d` for daily columns across two weeks. At 30 days, the terminal
-uses readable multi-day bins: three-day bins at ordinary widths and two-day
-bins on wider terminals. The image view uses the same readable multi-day
-binning at 30 days. The image also reports the separate local rate-card credit
-estimate when token breakdowns are available; that estimate is an absolute
-credit count and never rescales the observed-drain columns.
-
-The CLI automatically checks the local Codex source files before rendering. If
-they are newer than the local cache, it rebuilds the privacy-reduced snapshot.
-The first refresh may scan historical rollout files; later runs use the cache
-for one hour before checking source freshness again. Use `--refresh` when you
-need to force an immediate rebuild.
-
-The `1d` project dashboard shows a compact snapshot-age line such as
-`SNAPSHOT · fresh · 12m old`. `fresh` means the snapshot is within the
-one-hour cache window, `stale` means it is older, and `age unknown` means the
-snapshot has no usable capture-time metadata. The indicator does not print a
-local path or trigger another source scan.
-
-Useful overrides:
+![Seven-day Token Ledger report example](docs/token-ledger-report-7-day.png)
 
-```bash
-# Use another timezone instead of the computer's local timezone
-tledger week --tz America/New_York
+### Non-report CLI output
 
-# Force a complete local refresh
-tledger week --refresh
+![Anonymized Token Ledger terminal output](docs/token-ledger-cli-week.png)
 
-# Skip the freshness check and use the existing cache
-tledger week --no-refresh
+The terminal capture was made from a local privacy-reduced snapshot after
+replacing project and thread labels with neutral names. It contains no prompts,
+responses, secrets, home paths, or project names.
 
-# Read a specific privacy-reduced snapshot
-tledger week --input /path/to/token-ledger-snapshot.json
-```
+## What is observed
 
-`--static` prints once for pipes, logs, or terminals without interactive input.
-`--plain` or `NO_COLOR=1` disables ANSI color. `--youplot` is an optional
-legacy renderer and is not required for the default dashboard.
+The collector reads local rollout JSONL under `sessions/` and, unless disabled,
+`archived_sessions/`, plus `session_index.jsonl` and read-only `state_5.sqlite`
+metadata. It retains positive `last_token_usage` model-call events and their
+timestamps, turn IDs, model/source attribution, and token categories.
 
-## Local data and privacy
+Observed totals are the sum of globally de-duplicated retained events. Events
+with turn IDs use the turn plus cumulative/last-usage/context signatures;
+legacy events without turn IDs use a high-specificity usage signature and are
+marked as heuristic. State database token counters are kept as non-additive
+reference values because forks and subagents inherit cumulative history.
 
-The CLI reads from `CODEX_HOME` when set, otherwise `~/.codex`. It uses local
-Codex rollout JSONL files, the session index, and local state metadata. It
-writes its privacy-reduced cache to:
+Input, cached-input, output, and reasoning are stored as separate categories:
+cached input is a subset of input, and reasoning is a subset of output. Any
+composition that presents them together subtracts those subsets to avoid
+double counting. Models are attributed from turn context/settings and local
+thread metadata, then normalized for display.
 
-```text
-~/.token-ledger/token-ledger-snapshot.json
-```
+The weekly meter uses local rate-limit observations for the account-wide
+weekly window. Reset timestamps identify windows; stale readings and separate
+named pools are not stitched into that meter. Remaining percentage,
+observation time, and the selected window are local observations; the reset
+type (weekly expiry versus restart) is derived by comparing the prior window's
+reset timestamp with the first reading of the new window. None of this is an
+official account-wide quota or billing record.
 
-The collector does not export message bodies, reasoning text, tool arguments or
-results, credentials, file contents, or full local paths. Display titles may
-contain user-written text. CLI errors and empty-state source labels show only a
-safe filename label, not an absolute input or source path. When a PNG or report
-is written, the explicit output path is reported so you can find the file. The
-CLI makes no network requests.
+The exported snapshot contains token metadata, model/use-type labels, project
+labels, and display titles. It omits message bodies, reasoning text, tool
+arguments/results, credential fields, and full local paths. Display titles and
+project labels are user-written or local metadata and should be reviewed before
+sharing. Normal successful dashboard output is privacy-reduced, but diagnostics
+or explicit PNG writes may echo configured snapshot, Codex, or output path
+labels.
 
-## Keyboard controls
+## What is estimated
 
-In the interactive dashboard:
+Credit values use the hardcoded rate card dated **2026-08-17**. Cached input is
+priced separately, priority/fast-mode events use the 1.5× multiplier, and
+events without a detailed breakdown or known model rate are unrated. These are
+rate-card estimates, not provider billing totals.
 
-- `↑` / `↓` or `j` / `k` moves between projects.
-- `q`, `Q`, `Esc`, or `Ctrl-C` exits.
-- Enter does not inspect a project, and `d` / `w` / `m` do not change the
-  range; choose the desired range in the command instead.
+When meter observations are available, report bars in `--drain` mode represent
+observed meter drops; model attribution within a drop uses rate-card weights
+when possible and token weights as a fallback. Long observation gaps are
+spread across local calendar days as estimates. Tokens-per-meter-point,
+model-split burn, runway, and CLI `View burn` are derived estimates. None is
+official billing, quota, or account-completeness truth.
 
-## Verify from source
+## Verify
 
 ```bash
 npm test
@@ -173,7 +191,3 @@ npm run lint
 npm run verify:release
 npm pack --dry-run --json
 ```
-
-`npm run verify:release` packs the allowlisted artifact, installs that tarball
-in a clean temporary directory with no network or Codex data access, and runs
-the installed `tledger --help` and synthetic `tledger 1d --static` smoke checks.