tledger 0.3.0 → 0.4.0

package/README.md

@@ -1,193 +1,321 @@
 # Token Ledger
 
-Token Ledger is a local-only Codex usage dashboard with two outputs:
-
-- a generated PNG report for trends, model composition, and weekly-meter
-  context;
-- a terminal dashboard for a fast project and token summary.
-
-This README describes the local checkout. It does not assume a hosted report
-or a published npm package.
+Token Ledger is a small, local-only terminal dashboard for Codex usage. It
+shows ranked project bars, model mix, usage type, cache coverage, and reset
+cycle context without sending your data anywhere.
 
 ## Install
 
-Requires Node.js 22.13 or newer. From this checkout:
+Requires Node.js 22.13 or newer.
+
+Install the published CLI with:
 
 ```bash
-npm install
-npm install -g .
+npm install -g tledger
 ```
 
-The second command installs `tledger` on your PATH. From the checkout, you can
-use `npx tledger` instead.
-
-## Basics
-
-| What you need | Command |
-| --- | --- |
-| Quick guide | `tledger` |
-| Last 24 hours | `tledger 1d` |
-| Last 7 calendar days | `tledger week` |
-| Rolling 30 days | `tledger 30d` |
-| 7-day PNG report | `tledger report 7d` |
-| Cache-only PNG report | `tledger report 7d --cache-rate` |
-
-The main options are:
-
-- `--static` prints once instead of opening the terminal dashboard.
-- `--refresh` rebuilds the local usage cache.
-- `--image-output <file>` chooses where to save a PNG.
-- `--no-open` writes a PNG without opening it.
-- `--help-all` shows the complete command and option reference.
-
-`week` covers seven local calendar days ending on the selected day. `1d` is a
-rolling 24-hour view ending when the command starts. In a TTY, the project
-dashboard is interactive. `j`/`k` select a project; `q`, `Q`, `Esc`, or
-`Ctrl-C` exits. Enter does not inspect a project, and `d` / `w` / `m` do not
-change the range; choose the range in the command.
-
-## Cache and input controls
-
-`tledger report [Nd|Nw] --cache-rate` writes a separate, purpose-built cache
-report to `token-ledger-cache-report-<period>.png`. Its primary view is the
-input-token-weighted cache rate (`cached input / measured input`), shown
-across the selected period with input-volume context. A smaller model breakout
-shows which models account for that input and their individual cache rates.
-The cache report intentionally omits the general report's project, quota-meter,
-runway, and output-token sections. Cached input is clamped to input per event,
-and the report states how much token volume had a usable component breakdown.
-Use `--image-output`, `--image-width`, `--date`, `--tz`, and `--no-open` the
-same way as on the standard report.
-
-The default privacy-reduced snapshot is
-`~/.token-ledger/token-ledger-snapshot-v2.json.gz`. It is gzip-compressed,
-written atomically with mode `0600`, targets 12 MiB, and has a hard 16 MiB
-on-disk limit. Its expanded JSON representation also targets 48 MiB and has a
-64 MiB safety limit, so the old 93 MiB raw-cache behavior cannot recur on the
-default production path. Reads check the compressed size before loading and
-bound gzip expansion to the same 64 MiB JSON limit. The collector de-duplicates
-through a private temporary SQLite spool, then keeps exact recent calls while
-rolling older history into minute, hour, and day buckets. If a dense history
-approaches the target, it increases the bucket resolution automatically while
-preserving additive token, model, project, cache, tool-call, and thread totals.
-When a compacted bucket crosses a requested range or chart boundary, Token
-Ledger allocates its additive values proportionally across the overlap and
-marks the terminal result as estimated; exact recent calls remain exact. The
-temporary spool is removed when collection completes or exits with a handled
-error.
-
-If even the coarsest bounded representation exceeds the hard limit, Token
-Ledger preserves the previous cache and asks you to reduce the source with
-`--no-archived` or the collector's `--since` option. It never replaces the
-production cache with an oversized or partial file. On a normal default-path
-run, a snapshot whose mtime is in the past and less than one hour old skips the
-source walk. An exact-hour or future mtime is not fresh; an older snapshot is
-checked against local source mtimes before it is reused or rebuilt.
+For a one-time run without a permanent global install:
 
 ```bash
-# Force a rebuild from CODEX_HOME or ~/.codex
-tledger week --refresh
-
-# Read the existing default snapshot without a source-freshness check
-tledger week --no-refresh
-
-# Read an explicit snapshot without automatic freshness checks
-tledger week --input /path/to/token-ledger-snapshot-v2.json.gz
+npx tledger week
 ```
 
-`--refresh` rebuilds the default snapshot and cannot be combined with
-`--input` in this checkout. `--no-archived` excludes `archived_sessions` when a
-refresh occurs. The collector can also be run directly:
+To install the development version from this repository:
 
 ```bash
-node lib/token-ledger-importer.mjs --output /path/to/token-ledger-snapshot-v2.json.gz
+git clone https://github.com/jskoiz/token-ledger.git
+cd token-ledger
+npm install -g .
 ```
 
-Explicit `.json` snapshots remain readable for fixtures and deliberate exports,
-but `.json.gz` is the bounded production cache format. After an upgrade, the
-new cache does not read or delete schema-v1 cache files; remove an old generated
-cache separately once the v2 cache is proven.
+The package uses a small runtime image encoder for PNG report output, Node's
+built-in SQLite support, and reads Codex data directly from the local machine.
 
-## Report versus CLI
+## Run
 
-| | Generated report | Terminal dashboard |
-| --- | --- | --- |
-| Command | `tledger report 7d` or `tledger trend 7d --image` | `tledger week`, `tledger 1d`, or `tledger day <date>` |
-| Output | One PNG with a stat quad, weekly-meter pace/runway, daily token columns, an aligned cache-rate strip, top projects, and per-model cache rates | Interactive or static project rows with totals, shares, thread counts, model mix, usage type, cache split, and reset-cycle context |
-| Range | A selected local-calendar-day trend window, such as 7d or 2w | A calendar day/week or a rolling 24-hour/`Nd`/`Nw` window |
-| Estimate surface | Meter-derived burn and runway are called out; cache coverage comes from measured token breakdowns | The sidebar can show derived `View burn`; project detail includes rate-card credit shares when available |
+The shortest useful command is:
 
-The report is the broader visual summary. The CLI does not open or generate a
-report unless you request the `report`, `trend --image`, or `--image` form.
-
-## Examples
+```bash
+tledger week
+```
 
-### Seven-day report
+It defaults to today's seven-day window, the computer's local timezone, the
+top 10 projects, and the local Codex data directory. The default terminal view
+is interactive; press `q` or `esc` to exit.
 
-![Seven-day Token Ledger report example](docs/token-ledger-report-7-day.png)
+Other common views:
 
-### Non-report CLI output
+```bash
+tledger 1d
+tledger 1d --static
+tledger 2d
+tledger 1w
+tledger 3w --static
+tledger day 2026-08-05
+tledger week --top 5
+tledger week --static
+tledger trend 7d --static
+tledger trend 7d --image --image-output artifacts/token-ledger-trend-7d.png
+tledger report 7d
+```
 
-![Anonymized Token Ledger terminal output](docs/token-ledger-cli-week.png)
+Bare duration aliases such as `tledger 1d`, `tledger 2d`, `tledger 3d`,
+`tledger 1w`, `tledger 2w`, and `tledger 3w` show the `TOKENS BY PROJECT`
+breakdown for a rolling window ending when the command starts. Day and week
+aliases accept any positive `Nd` or `Nw` value up to 3,650 days. `tledger 1d`
+is the existing rolling 24-hour view; `tledger 1w` is a rolling seven-day
+window. This is different from `tledger day today`, which covers the current
+calendar day from local midnight, and `tledger week`, which covers seven local
+calendar days.
+
+`tledger report [Nd|Nw]` is the one-step report output: it writes the
+dashboard PNG (identical to `trend --image`) to
+`token-ledger-report-<period>.png` in the current directory. It accepts the
+same flags as the trend view (`--drain`, `--date`, `--tz`, `--image-output`,
+`--image-width`) and prints progress while rendering and encoding the image.
+
+`tledger report 1d` shows the selected local calendar day in hourly columns,
+including hourly cache efficiency. The current hour is marked partial; future
+hours are not treated as observed zero usage. Use `--date yesterday` for a
+completed day. This differs from the rolling 24-hour `tledger 1d` terminal view.
+
+Use `tledger report 1d --private` to replace project names with `Project 1`,
+`Project 2`, and so on, ranked by usage within that report. Counts and usage
+remain visible; the local snapshot is unchanged. The default filename ends
+in `-private.png` so the named report is preserved.
+
+The terminal trend view is a compact approximation of the image view. For the
+full chart grammar, use `--image`: it writes a single shareable report card as
+a PNG. The report leads with a total-usage KPI (with a delta against the prior
+equivalent period), input-weighted cache efficiency, the fast-mode share of
+actual tokens, and the active-project count, beside the latest observed
+weekly-limit state. Below that sit a model-mix strip, calendar-day columns of
+local token volume stacked by model, daily cache efficiency with input
+volumes, top projects, and a per-model cache table.
+
+![Token Ledger 7-day report](docs/token-ledger-report-7-day.png)
+
+The weekly-limit line is drawn from sampled OpenAI observations, never
+continuous telemetry: solid runs mark spans confirmed by repeated equal
+readings, dashed runs bridge unobserved gaps, and the line never extends past
+the latest reading. When the report is generated partway
+through the final day, that column is marked `PARTIAL` with the actual cutoff
+time, and the prior-period delta compares an equally long partial window.
+Values allocated from compacted history or placed using an estimated event
+time are marked with `≈`. Splitting usage between meter readings does not by
+itself make the daily token total approximate. Reports built from an explicit
+or stale snapshot say `Snapshot generated …` (with a `STALE SNAPSHOT` badge on
+fallback).
+Detailed source and integrity diagnostics remain in the generated snapshot's
+coverage metadata.
+
+When run from a terminal, the finished PNG opens in the default image viewer
+automatically so the report lands on screen instead of in a file browser.
+Pass `--no-open` to skip that; piped or scripted runs never open a window.
+
+Turns run in fast mode (service tiers "priority" and "fast") are drawn with a
+diagonal hatch inside their model's segment — fast mode is a property of usage,
+not a separate model, so the hatch never adds bar height and stays legible in
+grayscale.
+
+The fast-mode KPI counts only usage explicitly recorded as `priority` or
+`fast`. Missing or unrecognized service tiers are shown separately as unknown;
+they are included in total usage but cannot be classified as fast or normal.
+Input and cache metrics use only records with the necessary component counts.
+Total-only records still contribute their reported tokens to total usage.
+
+Pass `--drain` to flip the columns into limit-drain units instead: each column
+becomes the weekly limit percentage the meter dropped, stacked by model using
+rate-card credit weights (an estimate — the official card is the best
+available proxy for per-model debit, but subscription limits are not billed
+per token), on the same percent scale as the meter line.
+
+Meter windows are keyed by their server-reported reset timestamp, so a fresh
+window that starts days early (a provider-initiated limit restart) is drawn as
+its own cycle and labeled `restart`, while a true weekly expiry is labeled
+`reset`.
+Stale readings from sessions still reporting a superseded window are dropped
+instead of being fused into the line as phantom drain. Drops observed after a
+sparse meter gap (over 36 hours) are spread across the covered days and marked
+with `≈`. When a range has no usable meter drain, columns fall back to raw
+local token counts and the chart says so.
+
+`--image` defaults to `token-ledger-trend-7d.png` in the current directory.
+Use `--image-output <file.png>` to choose the path and `--image-width <px>` to
+choose a width from 900 to 2400 pixels.
+
+Use `trend 14d` for daily columns across two weeks. At 30 days, the terminal
+uses readable multi-day bins: three-day bins at ordinary widths and two-day
+bins on wider terminals. The image view keeps daily columns while they stay
+legible and falls back to the same readable multi-day binning for longer
+windows. Meter-drain weighting uses the rate card bundled with this release;
+subscription limits are not billed per token.
+
+Models without a bundled price remain explicitly unrated in cost estimates.
+Astra is recognized in usage reports but is unrated in this release's
+purchased-credit and API-dollar estimates.
+
+The CLI checks the local Codex source manifest on every automatic load. If it
+differs from the cached watermark, the CLI rebuilds the privacy-reduced
+snapshot; otherwise it reuses the cache immediately. The first refresh may
+scan historical rollout files. The one-hour threshold affects only the
+displayed freshness label, not source validation. Use `--refresh` to force a
+rebuild even when the manifest is unchanged.
+
+The `1d` project dashboard shows a compact snapshot-age line such as
+`SNAPSHOT · fresh · 12m old`. `fresh` means the snapshot is within the
+one-hour cache window, `stale` means it is older, and `age unknown` means the
+snapshot has no usable capture-time metadata. The indicator does not print a
+local path or trigger another source scan. Terminal reports also show a
+separate provenance status: `VERIFIED CURRENT`, `STALE FALLBACK`,
+`UNCHECKED CACHE`, or `EXPLICIT SNAPSHOT`. The standard PNG report communicates
+freshness through its generation timestamp and stale/partial labels. Snapshot
+age describes capture time; it never implies that the local source was checked
+successfully.
+
+Useful overrides:
 
-The terminal capture was made from a local privacy-reduced snapshot after
-replacing project and thread labels with neutral names. It contains no prompts,
-responses, secrets, home paths, or project names.
+```bash
+# Use another timezone instead of the computer's local timezone
+tledger week --tz America/New_York
 
-## What is observed
+# Force a complete local refresh
+tledger week --refresh
 
-The collector reads local rollout JSONL under `sessions/` and, unless disabled,
-`archived_sessions/`, plus `session_index.jsonl` and read-only `state_5.sqlite`
-metadata. It retains positive `last_token_usage` model-call events and their
-timestamps, turn IDs, model/source attribution, and token categories.
+# Skip the freshness check and use the existing cache
+tledger week --no-refresh
 
-Observed totals are the sum of globally de-duplicated retained events. Events
-with turn IDs use the turn plus cumulative/last-usage/context signatures;
-legacy events without turn IDs use a high-specificity usage signature and are
-marked as heuristic. State database token counters are kept as non-additive
-reference values because forks and subagents inherit cumulative history.
+# Read a specific privacy-reduced snapshot
+tledger week --input /path/to/token-ledger-snapshot.json
+```
 
-Input, cached-input, output, and reasoning are stored as separate categories:
-cached input is a subset of input, and reasoning is a subset of output. Any
-composition that presents them together subtracts those subsets to avoid
-double counting. Models are attributed from turn context/settings and local
-thread metadata, then normalized for display.
+`--static` prints once for pipes, logs, or terminals without interactive input.
+`--plain` or `NO_COLOR=1` disables ANSI color. `--youplot` is an optional
+legacy renderer and is not required for the default dashboard.
 
-The weekly meter uses local rate-limit observations for the account-wide
-weekly window. Reset timestamps identify windows; stale readings and separate
-named pools are not stitched into that meter. Remaining percentage,
-observation time, and the selected window are local observations; the reset
-type (weekly expiry versus restart) is derived by comparing the prior window's
-reset timestamp with the first reading of the new window. None of this is an
-official account-wide quota or billing record.
+## Local data and privacy
 
-The exported snapshot contains token metadata, model/use-type labels, project
-labels, and display titles. It omits message bodies, reasoning text, tool
-arguments/results, credential fields, and full local paths. Display titles and
-project labels are user-written or local metadata and should be reviewed before
-sharing. Normal successful dashboard output is privacy-reduced, but diagnostics
-or explicit PNG writes may echo configured snapshot, Codex, or output path
-labels.
+The CLI reads from `CODEX_HOME` when set, otherwise `~/.codex`. It uses local
+Codex rollout JSONL files, the session index, and local state metadata. It
+writes a replaceable, privacy-reduced report snapshot to:
 
-## What is estimated
+```text
+~/.token-ledger/token-ledger-snapshot-v3.json.gz
+```
 
-Credit values use the hardcoded rate card dated **2026-08-17**. Cached input is
-priced separately, priority/fast-mode events use the 1.5× multiplier, and
-events without a detailed breakdown or known model rate are unrated. These are
-rate-card estimates, not provider billing totals.
+It also maintains the durable local ledger separately at:
 
-When meter observations are available, report bars in `--drain` mode represent
-observed meter drops; model attribution within a drop uses rate-card weights
-when possible and token weights as a fallback. Long observation gaps are
-spread across local calendar days as estimates. Tokens-per-meter-point,
-model-split burn, runway, and CLI `View burn` are derived estimates. None is
-official billing, quota, or account-completeness truth.
+```text
+~/.token-ledger/token-ledger-ledger.sqlite
+```
 
-## Verify
+The ledger is the append-and-deduplicate source of truth for committed token
+events, quota samples, source state, and useful thread metadata. The snapshot
+is a generated export and can be replaced or rebuilt without deleting ledger
+history. Refreshes scan both `sessions` and `archived_sessions`; a source that
+is removed is recorded as missing or tombstoned, and its committed observations
+remain available. A file replacement or truncation is recorded as a mutable
+source change and does not re-add earlier observations.
+
+When a token event is timestamped before its own recorded turn began, refresh
+can use the turn's time as an estimated placement and retain the original
+timestamp as provenance. The same repair applies to previously stored events.
+This changes their date attribution, not their token counts; it cannot recover
+the exact event time from a malformed source record.
+
+Exact observations are retained for 3,650 days. Older observations are
+compacted into UTC daily buckets with additive totals and source membership;
+the supported report window is never silently compacted away. Existing v3
+snapshots are migrated once, when readable, into explicitly marked
+`migrated_compacted` rows. Those rows preserve totals and ranges but do not
+invent exact event or turn identities, and remain marked as estimated in
+coverage. A missing legacy snapshot is also recorded as checked so a later
+refresh cannot unexpectedly migrate a different file into the same ledger.
+An existing snapshot with malformed usage/thread history, or one that is
+unreadable, oversized, non-regular, or non-v3, instead stops with
+`ERR_DURABLE_LEDGER_LEGACY_SNAPSHOT` before the ledger
+revision advances. Its bytes and the one-shot migration opportunity are
+preserved so the snapshot can be privately backed up, repaired or replaced,
+and retried.
+
+Legacy quota samples are imported only when the snapshot explicitly carries
+the current quota-identity contract. Older or markerless snapshots may have
+keyed unnamed limits by a display label or an anonymous placeholder, so their
+quota samples are skipped while safe usage and thread history still migrate.
+Current-contract quota claims that fail identity or measurement validation are
+also skipped independently instead of blocking otherwise valid usage history.
+The generated snapshot reports this separately as `legacyQuotaStatus` and
+`legacyQuotaRowsSkipped`.
+
+Compacted rows are retained for 7,300 days (20 years) before retirement.
+Source, quota, tool, and state-only thread metadata are pruned only after they
+are outside the applicable retention horizon; the supported report window is
+never silently dropped.
+Compacted usage buckets retain their deduplicated source-association scope, so
+`--no-archived` can exclude archived-only history without including it through
+an aggregate that also contains active usage.
+
+Legacy snapshot history is imported only when its collection scope and hashed
+Codex-home identity are both provable. If either check fails, exact rollout
+collection continues without that legacy history. The reason is recorded as
+`coverage.legacySnapshotStatus` in the generated snapshot.
+
+Codex quota records do not contain a ChatGPT account identifier. Token Ledger
+therefore treats one `CODEX_HOME` and its durable ledger as one account boundary
+and keys pools only by the canonical provider limit id; omitted or blank ids
+mean the default `codex` pool. If different ChatGPT accounts share a
+`CODEX_HOME`, their equal provider ids cannot be separated. Use distinct Codex
+homes under separate OS user profiles when account isolation matters.
+
+Raw provider limit ids are normalized and then hash-derived into stored pool
+keys; the raw ids are not written to the ledger or generated snapshot.
+Sanitized optional display labels are not identity, but may remain in the
+ledger, generated snapshot, terminal report, and image report.
+
+The application state directory is private (`0700`), and its ledger, lock, and
+SQLite transient files are private (`0600`). The durable database location is
+fixed at `~/.token-ledger/token-ledger-ledger.sqlite`; custom snapshot outputs
+never relocate it. An explicit `--input` reads only that deliberate snapshot
+input and does not use the durable ledger as a hidden data source.
+
+The collector does not export message bodies, reasoning text, tool arguments or
+results, credentials, file contents, or full local paths. Display titles may
+contain user-written text. CLI errors and empty-state source labels show only a
+safe filename label, not an absolute input or source path. When a PNG or report
+is written, the explicit output path is reported so you can find the file. The
+CLI makes no network requests.
+
+For schema health signals, crash-safety guidance, cache/ledger
+coherence, and the repeatable scaling benchmark, see
+[Durable ledger operations](docs/durable-ledger-operations.md).
+
+## Keyboard controls
+
+In the interactive dashboard:
+
+- `↑` / `↓` or `j` / `k` moves between projects.
+- `q`, `Q`, `Esc`, or `Ctrl-C` exits.
+- Enter does not inspect a project, and `d` / `w` / `m` do not
+  change the range; choose the desired range in the command instead.
+
+## Verify from source
 
 ```bash
-npm test
-npm run lint
-npm run verify:release
-npm pack --dry-run --json
+npm ci
+npm run test:fast  # quick contract feedback while editing
+npm run check      # all regular tests plus lint, before committing
+npm run prepublishOnly  # full suite, stress, lint, and installed-package checks
 ```
+
+`npm run verify:release` packs the allowlisted artifact, installs that tarball
+in a clean temporary directory with no network or Codex data access, and runs
+the installed help commands, a synthetic `tledger 1d --static` dashboard, and
+both standard and cache-rate PNG report smoke checks.
+
+New top-level test files are included automatically. `npm test` retains the full
+regular suite; `npm run test:integration` selects the CLI, importer subprocess,
+and durable-ledger suites omitted by `test:fast`. `npm run test:all` runs regular
+and stress tests together in one test-runner invocation. PR and main CI run all
+regular tests; stress tests also run for release tags and before publication.
+Superseded PR runs are cancelled automatically. Run the full gate once for the
+final source state, rather than repeating its constituent commands.