llm-cost-attribution 0.1.1 → 0.3.0

package/README.md

@@ -1,150 +1,197 @@
 # llm-cost-attribution
 
-Per-issue token, turn, and quota analytics for [Claude Code](https://docs.anthropic.com/en/docs/claude-code) and [Codex CLI](https://github.com/openai/codex) sessions. Reads the CLIs' own session JSONLs — **no telemetry pipeline, no database, no API keys**.
+Per-issue cost analytics for [Claude Code](https://docs.anthropic.com/en/docs/claude-code) and [Codex CLI](https://github.com/openai/codex) sessions — how many **tokens** an issue burned, how many **turns** it took (one agent request → response is a turn), and how much of your Codex/Claude plan's rate-limit **quota** it ate. It reads the CLIs' own session logs (JSONL = one JSON record per line) — **no telemetry pipeline, no database, no API keys**.
 
 ```bash
 npx llm-cost-attribution EPAC-1940
 ```
 
 ```
-════════════════════════════════════════════════════════════════════════
-LLM COST  —  EPAC-1940
-════════════════════════════════════════════════════════════════════════
-Sessions found:       5
-Total turns:          414
-Total tokens:         61,357,012
-
-────────────────────────────────────────────────────────────────────────
-CODEX  (4 sessions)
-────────────────────────────────────────────────────────────────────────
-  Models:             gpt-5-codex
-  Turns:              340
-  Tokens:
-    input uncached         1,517,206
-    cache read            51,024,768
-    output (visible)          44,683
-    output (reasoning)        18,649
-    grand total           52,605,306
-  Quota  (plan_type=pro, 345 samples):
-    5h window  58% → 64% used  (peak 64%)
-    7d window  56% → 57% used  (peak 57%)
+LLM COST — EPAC-1940
+Sessions: 5   Turns: 414   Tokens: 61,357,012
+
+CODEX  (4 sessions)   Models: gpt-5-codex   Turns: 340
+  input uncached      1,517,206
+  cache read         51,024,768
+  output (visible)       44,683
+  output (reasoning)     18,649
+  grand total        52,605,306
+  Quota (pro, 345 samples):  5h 58%→64% (peak 64%)   7d 56%→57% (peak 57%)
 ```
 
-## Designed for Symphony workflows
+Reading that block: **cache read** is tokens the provider served from its prompt cache (cheap, and usually most of the total); **output (reasoning)** is the model's hidden thinking tokens, billed separately from the **visible** answer; **Quota** is how much of your Codex plan's two rolling rate-limit windows — a 5-hour and a 7-day one — these sessions used.
 
-[OpenAI Symphony's specification](https://github.com/openai/symphony/blob/main/SPEC.md) requires that each issue gets its own filesystem workspace, and that the coding agent's `cwd` equals that workspace path:
+Requires Node 20+. Zero runtime dependencies.
 
-- **§4.1.4 Workspace** — "Filesystem workspace assigned to one issue identifier."
-- **Workspace path formula** — `<workspace.root>/<sanitized_issue_identifier>`.
-- **Invariant 1** — "Run the coding agent only in the per-issue workspace path... validate: `cwd == workspace_path`."
+## How it works
 
-Because of those requirements, the working directory of every Claude Code or Codex CLI session that Symphony (or any Symphony-spec-conformant orchestrator) launches always carries the issue identifier as its last path component. The CLI agents in turn record that `cwd` in every session JSONL they create. So the issue identifier is already in the transcript — no custom telemetry pipeline needed to join.
+Both CLIs persist every run as JSONL — Claude Code in `~/.claude/projects/<encoded-cwd>/<sessionId>.jsonl` (`<encoded-cwd>` is just the run's working directory with `/` and `.` rewritten to `-`), Codex in `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl` — and each file records, per turn, the provider-reported token counts (the same numbers your account is billed against) plus, for Codex, its rate-limit usage. This package walks both directories, keeps the sessions whose **working directory** matches the issue ID you ask for, and adds them up.
 
-This package's default `--cwd-pattern` matches the two most common `workspace.root` configurations:
+How does a session get matched to an issue? By its **working directory** (`cwd`). Under [Symphony](https://github.com/openai/symphony/blob/main/SPEC.md)'s spec — Symphony being an orchestrator that runs coding agents one issue at a time — each agent runs in a directory dedicated to its issue (`<workspace.root>/<ISSUE-ID>`), so the issue ID is already baked into every transcript's path; no custom pipeline needed. The default `--cwd-pattern` (the regex that pulls the issue ID out of that path) matches both the spec default (`<tmp>/symphony_workspaces/<ID>`) and the common in-repo layout (`<repo>/.symphony/workspaces/<ID>`). For any other layout, pass your own regex with one capture group around the ID:
 
-1. The Symphony spec default: `<system-temp>/symphony_workspaces/<ISSUE-ID>` (e.g. `/tmp/symphony_workspaces/EPAC-1940`).
-2. A common in-repo override: `<repo>/.symphony/workspaces/<ISSUE-ID>` (used by Autopilot and the Riddim factory's Symphony config).
+```bash
+llm-cost FOO-12 --cwd-pattern '-([A-Z]+-\d+)$'   # ../repo-worktrees/<ID>
+llm-cost 1234   --cwd-pattern '/issues/(\d+)$'    # ~/issues/<id>/
+```
 
-For any other `workspace.root` setting, pass `--cwd-pattern '<regex>'` with one capture group for the issue identifier — see "[The convention](#the-convention)" below.
+If your workflow doesn't give each issue its own directory, this package can't disambiguate sessions — see "What it doesn't do."
 
-## How it works
-
-Both CLIs persist every session they run as JSONL:
+## Install
 
-- **Claude Code** writes `~/.claude/projects/<encoded-cwd>/<sessionId>.jsonl` for every interactive and non-interactive run (encoded-cwd is the absolute working directory with `/` and `.` replaced by `-`).
-- **Codex CLI** writes `~/.codex/sessions/YYYY/MM/DD/rollout-<timestamp>-<id>.jsonl` for every run, with the working directory recorded in the first `session_meta` event.
+```bash
+npx llm-cost-attribution EPAC-1940     # one-shot
+npm install -g llm-cost-attribution    # then: llm-cost EPAC-1940
+```
 
-Each file carries provider-reported token usage per turn — the same numbers your Anthropic / OpenAI account is billed against:
+## CLI
 
-| Provider | Tokens captured |
-|---|---|
-| Claude | `input_tokens`, `cache_read_input_tokens`, `cache_creation.{ephemeral_5m,1h}_input_tokens`, `output_tokens` |
-| Codex | `input_tokens`, `cached_input_tokens`, `output_tokens`, `reasoning_output_tokens` (deltaed from cumulative) |
-| Codex (additionally) | `rate_limits.{primary,secondary}.used_percent` per turn |
+```
+llm-cost <ISSUE-ID> [options]
+llm-cost <ISSUE-ID> --from-usage <usage.jsonl-or-dir>
+llm-cost list
+llm-cost backfill --out <usage.jsonl-path>
+llm-cost calibrate <usage.jsonl-or-dir> [--seed N] [--holdout F]
+llm-cost --help
 
-This package walks both directories, filters sessions whose working directory matches an issue identifier you ask for, and aggregates.
+Options:
+  --cwd-pattern <regex>  JS regex matching the cwd; one capture group = issue ID.
+  --claude-dir <path>    Override ~/.claude/projects.
+  --codex-dir <path>     Override ~/.codex/sessions.
+  --from-usage <path>    Read a baked usage.jsonl file/dir instead of transcripts.
+  --out <path>           (backfill) Destination usage.jsonl. Appended.
+  --seed <int>           (calibrate) Held-out split seed. Default 1.
+  --holdout <0..1>       (calibrate) Fraction held out per cell. Default 0.2.
+  --quantile <0..1>      (calibrate) Band to test. Default 0.8.
+  --threshold <0..1>     (calibrate) Flag coverage drift beyond this. Default 0.1.
+  --json                 Emit JSON instead of a table.
+  --no-pricing           Suppress the dollar block.
+```
 
-## The convention
+## Delete transcripts, keep cost history
 
-You map sessions to issues via the **working directory at session start**. By default this package matches the Symphony-spec convention:
+Transcripts are large (MBs per session, GBs across a factory) and mostly conversation content the cost tool doesn't need. `backfill` bakes every transcript into a small append-only JSONL (~1 KB/turn, no prompt/response content); queries then read that file, and the transcripts are safe to delete:
 
-```
-<repo>/.symphony/workspaces/<ISSUE-ID>
+```bash
+llm-cost backfill --out ~/llm-cost-history.jsonl
+llm-cost EPAC-1940 --from-usage ~/llm-cost-history.jsonl
+rm -rf ~/.claude/projects ~/.codex/sessions   # once numbers verified
 ```
 
-A regex extracts `<ISSUE-ID>`. If your workflow uses a different layout, pass `--cwd-pattern '<regex>'` with one capture group:
+| | Before | After |
+|---|---:|---:|
+| Disk | 5.0 GB | 125 MB (40× smaller) |
+| Query time | ~3 min | ~0.3 s |
 
-```bash
-# Your workflow uses ../repo-worktrees/<ID>
-llm-cost FOO-12 --cwd-pattern '-([A-Z]+-\d+)$'
+The bake is lossless for everything the analysis uses (quota windows, Claude cache tiers, Codex reasoning/visible split, totals, models, timestamps, workspace provenance). The format follows the [Symphony Cost Telemetry Extension spec](https://github.com/RiddimSoftware/groove/blob/main/specs/symphony-cost-telemetry-extension/SPEC.md), so a conformant orchestrator can emit `usage.jsonl` directly and skip the bake — optional interop, not required.
 
-# Your workflow uses ~/issues/<id>/
-llm-cost 1234 --cwd-pattern '/issues/(\d+)$'
+## Is the forecast trustworthy? (`calibrate`)
+
+A **P80** is the 80th-percentile cost — the number 80% of comparable issues come in at or below. Claiming "P80 = 12K tokens" is only honest if, on issues the forecaster never saw, the real cost actually lands under 12K about 80% of the time; otherwise it's a horoscope. `calibrate` checks exactly that against a local `usage.jsonl` whose records are **estimate-tagged** (each one carries the issue's size estimate). It sorts the records into **cells** — groups of past issues sharing the same `{ size, model }` — holds out a reproducible slice of each cell (`--seed` makes the split repeatable), forecasts from what's left, and measures how often the held-out actuals really fell at or below the predicted P80. Any cell whose hit-rate drifts from 80% by more than `--threshold` is flagged ⚠. On a small dataset the coverage figures are themselves noisy — a cell with only a few held-out issues can read 0% or 100% by luck — so treat per-cell flags as directional until cells are well-populated.
+
+```bash
+llm-cost calibrate ~/backfill.out --seed 1 --holdout 0.2
 ```
 
-If your workflow doesn't give each issue its own working directory (e.g. you switch branches in a single checkout), this package can't disambiguate sessions for you — see "[What it doesn't (and can't) do](#what-it-doesnt-and-cant-do)" below.
+Read-only and local — the input is never written back or committed (point it at a gitignored file). Committed tests use only synthetic fixtures (`test/forecast-recovers-known-dist.test.mjs`).
 
-## Install
+## What drives your cost? (`cost-drivers`)
 
-```bash
-# One-shot via npx
-npx llm-cost-attribution EPAC-1940
+`cost-drivers` runs an end-to-end correlation analysis: it reads your LLM cost records, reads diff statistics from a local git repo, joins them by issue key, and prints Spearman rank correlation, linear Pearson, log-log Pearson, and a decile table. The goal is to understand which attributes of an issue predict how much it costs — using your own data, not anyone else's benchmarks.
+
+**Minimal inputs:** a local git repo whose commit subjects include issue keys, and transcripts (or a `usage.jsonl`) for the same issues.
 
-# Install globally
-npm install -g llm-cost-attribution
-llm-cost EPAC-1940
+```bash
+llm-cost cost-drivers --repo ~/code/my-project
+llm-cost cost-drivers --repo ~/code/my-project --metric turns
+llm-cost cost-drivers --repo ~/code/my-project --from-usage ~/llm-cost-history.jsonl
 ```
 
-Requires Node 20+. Zero runtime dependencies.
+Example readout (synthetic numbers — for illustration only):
 
-## CLI
+```
+════════════════════════════════════════════════════════════════════════
+COST DRIVERS  —  diff churn vs tokens
+════════════════════════════════════════════════════════════════════════
+Join strategy:  issue
+Source:         ~/code/my-project
+n = 42 pairs    unjoined: 3 usage, 5 diffs    unmatched commits: 11
+
+Correlations:
+  Spearman           0.34
+  Pearson(linear)    0.21
+  Pearson(log-log)   0.40
 
+Decile table:
+Decile   Feature range                n   Median cost
+────────────────────────────────────────────────────────────────────────
+1        14 – 87                      4        58.3K
+2        91 – 210                     4        72.1K
+3        215 – 380                    4        91.4K
+4        384 – 510                    4       103.2K
+5        512 – 740                    5       128.7K
+6        744 – 1.1K                   4       145.3K
+7        1.1K – 1.6K                  4       189.6K
+8        1.6K – 2.4K                  4       224.1K
+9        2.5K – 4.1K                  5       301.8K
+10       4.2K – 9.3K                  4       512.4K
 ```
-llm-cost <ISSUE-ID> [options]
-llm-cost <ISSUE-ID> --from-usage <usage.jsonl-or-dir>
-llm-cost list
-llm-cost backfill --out <usage.jsonl-path>
-llm-cost --help
 
-Options:
-  --cwd-pattern <regex>   JS regex matching the cwd; one capture group is the issue ID.
-                          Default matches both `<system-temp>/symphony_workspaces/<ID>`
-                          and `<repo>/.symphony/workspaces/<ID>` (raw or Claude-encoded).
-  --claude-dir <path>     Override ~/.claude/projects.
-  --codex-dir <path>      Override ~/.codex/sessions.
-  --from-usage <path>     Read from a usage.jsonl file or directory of `usage*.jsonl`
-                          files instead of the CLI transcripts. See "Delete transcripts,
-                          keep cost history" below.
-  --out <path>            (backfill only) Destination usage.jsonl path. Appended.
-  --json                  Emit JSON instead of a table.
-  -h, --help              Print help.
+Reading that block: **Feature range** is diff churn (additions + deletions) in lines; **Median cost** is the median token count for issues in that churn decile. The three correlation coefficients tell the same story from different angles — see "Reading the output" below.
+
+### Join model
+
+`cost-drivers` needs to know which cost record belongs to which diff. The `--join-by` flag selects the strategy:
+
+| Strategy | How it joins | When to use |
+|---|---|---|
+| `issue` (default) | Extracts issue keys (e.g. `ABC-123`) from commit subjects and from each cost record's `issueIdentifier` / workspace path | Works out of the box with Symphony's per-issue worktree convention and squash-merge commit messages |
+| `worktree` | Joins on the cost record's workspace path vs. the diff record's key | Useful when your diff records carry workspace paths instead of issue keys |
+| `time` | Attributes each cost record to the next commit within `--window` (e.g. `30m`, `2h`, `1d`) | Label-free fallback when commit subjects don't contain keys; inherently approximate |
+
+```bash
+# explicit strategies
+llm-cost cost-drivers --repo ~/code/my-project --join-by issue    # default
+llm-cost cost-drivers --repo ~/code/my-project --join-by worktree
+llm-cost cost-drivers --repo ~/code/my-project --join-by time --window 2h
+
+# override the key-extraction regex if your project uses a different format
+llm-cost cost-drivers --repo ~/code/my-project --key-pattern 'TICKET-\d+'
 ```
 
-## Delete transcripts, keep cost history (optional)
+The `keyOfUsage`, `keyOfDiff`, and `join` overrides are available via the library API (`joinCostWithFeature`) for cases the CLI flags don't cover — for example joining on a custom field, or implementing a fully custom reconciliation.
+
+#### Escape hatch: join externally with `dump-* → correlate`
 
-Transcripts are large — a few MB per session, growing to gigabytes across an active factory — and most of the bytes are conversation content the cost tool doesn't need. So `llm-cost` can **bake** every transcript into a small append-only JSONL file (~1 KB per turn, no prompt or response content), then read cost queries from that file instead. After the bake, transcripts are safe to delete.
+If none of the built-in strategies fit, emit the two streams and join them yourself:
 
 ```bash
-# Bake every transcript on this machine into one file.
-llm-cost backfill --out ~/llm-cost-history.jsonl
+# 1. dump the cost stream
+llm-cost dump-usage > usage.jsonl
 
-# Cost queries now run against the much smaller file:
-llm-cost EPAC-1940 --from-usage ~/llm-cost-history.jsonl
+# 2. dump the diff stream
+llm-cost dump-diffs --repo ~/code/my-project > diffs.jsonl
 
-# Once you've verified the numbers match, transcripts are safe to delete:
-rm -rf ~/.claude/projects ~/.codex/sessions
+# 3. join them however you like, then feed back a { feature, cost } CSV
+llm-cost correlate --pairs my-pairs.csv   # CSV: feature,cost[,key]
 ```
 
-Real-world numbers from a working factory:
+`correlate --pairs` accepts `.csv` (header `feature,cost`) or `.json` (array of `{feature, cost}` objects) and produces the same readout as `cost-drivers`.
 
-| | Before backfill | After backfill |
-|---|---:|---:|
-| Disk footprint | 5.0 GB | 125 MB (40× smaller) |
-| `llm-cost EPAC-1940` query time | ~3 min (full Codex scan) | ~0.3 s |
+### Reading the output
+
+**Three correlation views, not one.** LLM cost is heavy-tailed — a handful of expensive issues can dominate a linear average. `cost-drivers` therefore reports:
+
+- **Spearman** (rank correlation): captures monotonic relationships without being skewed by outliers. If big issues generally cost more than small ones, Spearman will pick that up even when the raw values vary wildly.
+- **Pearson (linear)**: the standard linear correlation on raw values. On heavy-tailed data it can read near zero even when Spearman is meaningful; it is sensitive to a few extreme issues.
+- **Pearson (log-log)**: Pearson on log₁₀-transformed values, the right view when both axes span orders of magnitude. If cost and diff size both grow geometrically, this is the coefficient that captures it.
 
-The backfill is lossless for everything the cost analysis cares about — including the Codex per-window quota readout, the Claude cache-tier split (5m vs 1h), and the Codex reasoning-vs-visible output split. Token grand totals, turn counts, models, timestamps, and workspace-path provenance are preserved exactly. The bake file can also be checked into a private repo, shipped to a billing host, or queried from CI without access to the machine that produced the agent sessions.
+A large gap between Spearman and linear Pearson is a signal that the relationship is real but nonlinear or that a few outliers are suppressing the linear view — not that the relationship is absent.
 
-This whole flow is a built-in feature of the package — you don't need to know anything about the file format to use it. As a side benefit: the format follows the [Symphony Coding-Agent Cost Telemetry Extension spec](https://github.com/RiddimSoftware/groove/blob/main/specs/symphony-cost-telemetry-extension/SPEC.md), so any other tool that conforms can read or write the same file (e.g. a Symphony-spec-conformant orchestrator can emit `usage.jsonl` directly during runs, skipping the bake step entirely). That interop is purely optional; the package works exactly the same whether you care about the spec or not.
+**Always check `n`.** With a small sample (say n < 20) the coefficients are unreliable and the decile table will have very few rows per bucket. Treat the output as directional until you have more history.
+
+**Diff size is output, not effort.** A feature that happens to touch many files will show high churn whether or not it was the most complex work. Churn is the most readily available proxy; other features (issue estimate, turn count) may or may not track cost better on your workload.
+
+**Local-git limits.** `readGitDiffs` only sees commits already in your local checkout — run `git fetch` or `git pull` first if you want remote-only commits. For the default `issue` strategy, commits must also carry issue keys in their subjects (the default pattern matches `ABC-123`-style keys; override with `--key-pattern`).
 
 ## Library
 
@@ -156,48 +203,38 @@ import {
   listKnownIssues,
 } from 'llm-cost-attribution';
 
-// Read from transcripts directly:
-const rollup = await computeIssueCost('EPAC-1940');
-console.log(rollup.combinedTokens);
-console.log(rollup.providerTotals.codex.quotaSamples);
-
-// Or read from a backfilled usage.jsonl:
+const rollup  = await computeIssueCost('EPAC-1940');
 const rollup2 = await computeIssueCostFromUsage('EPAC-1940', '~/llm-cost-history.jsonl');
-
-// Backfill programmatically:
-const result = await backfillUsageFromTranscripts({
-  outFile: '/tmp/usage.jsonl',
-  onProgress: ({ phase, processed, total }) => console.log(`${phase}: ${processed}/${total}`),
-});
-console.log(`Wrote ${result.recordsWritten} records`);
+const result  = await backfillUsageFromTranscripts({ outFile: '/tmp/usage.jsonl' });
 ```
 
-Pass `{ cwdPattern, claudeProjectsDir, codexSessionsDir }` to override defaults on any of the above.
+Pass `{ cwdPattern, claudeProjectsDir, codexSessionsDir }` to override defaults.
 
-## What it doesn't (and can't) do
+### Diff-size feature records
 
-- **Story-point estimate axis.** Estimates live in your issue tracker (Linear / Jira / GitHub Projects), not in the CLI transcripts. To get cost-vs-estimate rollups you'd need to join issue-tracker data — out of scope for this package.
-- **Attempt counts.** The CLI doesn't record "this was attempt #N of M"; if you ran `claude` 5 times on the same issue, this package sees 5 sessions but can't tell you which one shipped.
-- **PR-merge state, CI status, reviewer verdicts.** These come from GitHub, not from the CLIs — and the Symphony spec explicitly out-of-scopes them (§2.2 Non-Goals, §11.5): ticket mutations and PR outcomes are delegated to the coding agent's tooling, not recorded by the orchestrator. This package stops at the same boundary: "what's in the CLI transcript."
-- **Anything in the Claude Desktop app, claude.ai, ChatGPT, or direct API SDK calls.** Only Claude Code CLI and Codex CLI sessions are stored in the directories this package reads.
+`readGitDiffs(repoPath, { revRange, keyPattern })` reads local `git log --numstat`
+output and yields one aggregated record per issue key found in commit subjects:
 
-## Pricing
+```js
+for await (const diff of readGitDiffs('/path/to/repo')) {
+  console.log(diff.key, diff.additions + diff.deletions, diff.changedFiles);
+}
+```
 
-`llm-cost` shows API-equivalent dollar cost per bucket alongside the raw token counts, using a built-in rate table sourced from [anthropic.com/pricing](https://www.anthropic.com/pricing) and [platform.openai.com/docs/pricing](https://platform.openai.com/docs/pricing):
+It is local-first: no GitHub token, network, or API calls. The tradeoff is that it
+sees only history already present in the checkout, and commits must carry issue
+keys in their subjects, as with squash-merge subjects like `[ABC-12]: add widget`.
 
-```
-API-equivalent pricing (gpt-5.5 @ rates verified 2026-05-22):
-    input uncached        $7.59    (1.5M × $5.00/1M)
-    cache read           $25.51    (51.0M × $0.500/1M)
-    output (visible)      $1.34    (44.7K × $30.00/1M)
-    output (reasoning)    $0.56    (18.6K × $30.00/1M)
-    ───────────────────────────────────────────
-    total API cost       $35.00    [hypothetical — your Codex Pro plan covers this]
-```
+## What it doesn't (and can't) do
 
-**This is a counterfactual, not your actual spend.** If you're on a subscription plan (Claude Max, Codex Pro, etc.), the dollar number represents what the same token volume would have cost on pay-as-you-go API — useful for comparison, but the marginal cost of running it on your actual plan is captured by the Codex quota readout above (`5h primary 58% → 64% used`), not by the dollar total.
+- **Story-point estimates** — live in your tracker, not the transcripts (see the sibling `llm-cost-estimation`).
+- **Attempt counts** — the CLI doesn't record "attempt #N"; 5 runs look like 5 sessions with no winner marked.
+- **PR / CI / reviewer state** — comes from GitHub, not the CLIs; out of scope (matches Symphony §2.2/§11.5).
+- **Claude Desktop, claude.ai, ChatGPT, raw API SDK** — only Claude Code CLI and Codex CLI sessions are read.
+
+## Pricing
 
-The CLI warns when the bundled rate table is more than 90 days old. Pass `--no-pricing` to suppress the block entirely.
+`llm-cost` shows API-equivalent dollar cost per bucket from a built-in rate table ([Anthropic](https://www.anthropic.com/pricing), [OpenAI](https://platform.openai.com/docs/pricing)). **This is a counterfactual, not your actual spend:** on a subscription plan (Claude Max, Codex Pro) it's what the same tokens would cost pay-as-you-go — your real marginal cost is the quota readout, not the dollar total. The CLI warns when the table is >90 days old; `--no-pricing` suppresses the block.
 
 ## License