@vimoxshah/tokenflow 1.1.0

package/docs/media/architecture-hero.svg

@@ -0,0 +1,86 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 880 300" width="100%" role="img" aria-label="TokenFlow architecture: local AI tool logs flow through adapters into a token store, then to dashboard, CLI, and menu bar — nothing leaves the machine.">
+  <defs>
+    <linearGradient id="tfh-accent" x1="0" y1="0" x2="1" y2="0">
+      <stop offset="0" stop-color="#3987e5"/><stop offset="1" stop-color="#8b5cf6"/>
+    </linearGradient>
+    <linearGradient id="tfh-card" x1="0" y1="0" x2="0" y2="1">
+      <stop offset="0" stop-color="#1c2333"/><stop offset="1" stop-color="#141926"/>
+    </linearGradient>
+  </defs>
+  <style>
+    .tfh-title{font:600 13px -apple-system,'Segoe UI',sans-serif;fill:#eef1f7}
+    .tfh-sub{font:400 10px -apple-system,'Segoe UI',sans-serif;fill:#98a1b5}
+    .tfh-chip{font:500 10px ui-monospace,Menlo,monospace;fill:#c7cede}
+    .tfh-num{font:700 20px -apple-system,'Segoe UI',sans-serif;fill:url(#tfh-accent)}
+    .tfh-cap{font:400 9.5px -apple-system,'Segoe UI',sans-serif;fill:#98a1b5}
+    .card{fill:url(#tfh-card);stroke:#2a3348;rx:12}
+    .arrow{stroke:url(#tfh-accent);stroke-width:1.6;fill:none;stroke-dasharray:4 5;opacity:.85}
+  </style>
+
+  <!-- sources -->
+  <g>
+    <rect class="card" x="16" y="34" width="150" height="232" rx="12"/>
+    <text class="tfh-title" x="30" y="56">Your tools' logs</text>
+    <text class="tfh-chip" x="30" y="84">~/.claude/*.jsonl</text>
+    <text class="tfh-chip" x="30" y="106">~/.codex/sessions</text>
+    <text class="tfh-chip" x="30" y="128">opencode.db</text>
+    <text class="tfh-chip" x="30" y="150">~/.hermes/state.db</text>
+    <text class="tfh-chip" x="30" y="172">~/.cline · ~/.cursor</text>
+    <text class="tfh-chip" x="30" y="194">CSV · JSONL · SQLite</text>
+    <text class="tfh-cap" x="30" y="230">read-only, incremental,</text>
+    <text class="tfh-cap" x="30" y="244">byte-offset resumable</text>
+  </g>
+
+  <!-- engine -->
+  <g>
+    <rect class="card" x="220" y="34" width="190" height="232" rx="12"/>
+    <text class="tfh-title" x="234" y="56">TokenFlow engine</text>
+    <text class="tfh-cap" x="234" y="74">zero dependencies · Node 22+</text>
+    <text class="tfh-chip" x="234" y="100">adapters → normalize</text>
+    <text class="tfh-chip" x="234" y="122">cache ≠ input accounting</text>
+    <text class="tfh-chip" x="234" y="144">stream dedup (max-of-run)</text>
+    <text class="tfh-chip" x="234" y="166">versioned price table</text>
+    <text class="tfh-chip" x="234" y="188">pre-aggregated cube</text>
+    <text class="tfh-cap" x="234" y="222">null ≠ 0 · measured ≠ estimated</text>
+    <text class="tfh-cap" x="234" y="236">145 tests enforce it</text>
+  </g>
+
+  <!-- surfaces -->
+  <g>
+    <rect class="card" x="464" y="34" width="180" height="70" rx="12"/>
+    <text class="tfh-title" x="478" y="58">Web dashboard</text>
+    <text class="tfh-cap" x="478" y="76">12 views · loopback only · themes</text>
+
+    <rect class="card" x="464" y="118" width="180" height="70" rx="12"/>
+    <text class="tfh-title" x="478" y="142">Native menu bar app</text>
+    <text class="tfh-cap" x="478" y="160">cost, limits, forecast, source &amp;</text>
+    <text class="tfh-cap" x="478" y="173">model breakdown, theme toggle</text>
+
+    <rect class="card" x="464" y="202" width="180" height="64" rx="12"/>
+    <text class="tfh-title" x="478" y="226">CLI + exports</text>
+    <text class="tfh-cap" x="478" y="244">refresh · status · watch · CSV/HTML</text>
+  </g>
+
+  <!-- stats column -->
+  <g>
+    <rect class="card" x="688" y="34" width="176" height="232" rx="12"/>
+    <text class="tfh-num" x="702" y="72">15+</text>
+    <text class="tfh-cap" x="702" y="88">providers &amp; gateways</text>
+    <text class="tfh-num" x="702" y="124">35+</text>
+    <text class="tfh-cap" x="702" y="140">models auto-classified</text>
+    <text class="tfh-num" x="702" y="176">12</text>
+    <text class="tfh-cap" x="702" y="192">analytics views</text>
+    <text class="tfh-num" x="702" y="228">0</text>
+    <text class="tfh-cap" x="702" y="244">runtime dependencies</text>
+  </g>
+
+  <!-- arrows -->
+  <path class="arrow" d="M170 150 H 214"/>
+  <path class="arrow" d="M414 120 C 440 110, 442 80, 458 70"/>
+  <path class="arrow" d="M414 150 H 458"/>
+  <path class="arrow" d="M414 180 C 440 190, 442 220, 458 230"/>
+
+  <!-- privacy band -->
+  <rect x="16" y="278" width="848" height="16" rx="4" fill="#10141d"/>
+  <text class="tfh-cap" x="440" y="290" text-anchor="middle">local-first — logs are read in place, normalized on this machine, and never uploaded anywhere</text>
+</svg>

package/docs/providers.md

@@ -0,0 +1,309 @@
+# Providers
+
+Every adapter answers three questions honestly: **where it looks**, **what it can report**, and
+**what it cannot know**. The third one matters most — a source that doesn't log token counts must
+say so rather than emit zeros.
+
+Enable and disable adapters with:
+
+```bash
+tokenflow providers
+tokenflow provider add <id>
+tokenflow provider remove <id>
+```
+
+---
+
+## `anthropic` — Claude Code / Claude Agent SDK
+
+| | |
+|---|---|
+| Reads | `<claude-home>/projects/<slug>/<sessionId>.jsonl` |
+| Homes | `$CLAUDE_CONFIG_DIR`, `~/.claude`, `~/.config/claude`, and any `~/.claude-*` sibling containing `projects/` (people running several accounts keep them side by side) |
+| Measurement | `primary` |
+| Reports | `input`, `output`, `cache_read`, `cache_write`, `cache_refresh` (1h ephemeral), `reasoning` (thinking), session, project (from `cwd`), git branch, interface (from `entrypoint`), sidechain/subagent flag, CLI version, service tier |
+| Cannot know | cost (no rate in the transcript) |
+
+### The two traps it handles
+
+**Streaming snapshots.** A single assistant message is written to the transcript several times as
+it streams. Every line carries the same `requestId` + `message.id` with a *growing*
+`output_tokens` and constant prompt-side counts. Measured on a real corpus: 2,493 of 5,060 usage
+rows were re-reports. Summing them inflates output by 3–5x. The adapter groups by
+`(requestId, message.id)` and keeps the **per-field maximum**.
+
+**Synthetic messages.** Locally generated assistant entries have `requestId: null` and an
+all-zero usage block — they are not API calls. Counting them would add fake zero-token requests
+and skew "requests" and "tokens per request". They are skipped and counted separately as
+`synthetic`.
+
+A group can straddle the end of a live file. Rather than defer it (which would silently drop the
+last record of every settled file, forever), it is emitted at EOF and its totals are remembered;
+if the file grows and the same group reappears with larger counts, only the **delta** is emitted.
+
+### Configuration
+
+```yaml
+sources:
+  anthropic:
+    paths: ["~/.claude", "~/.claude-work"]   # skip auto-discovery
+```
+
+---
+
+## `openai` — Codex CLI / Codex IDE / Codex Desktop
+
+| | |
+|---|---|
+| Reads | `~/.codex/sessions/**/*.jsonl` and `~/.codex/archived_sessions/**` (`$CODEX_HOME` respected) |
+| Measurement | `primary` |
+| Reports | fresh `input`, `cache_read`, `cache_write`, `output`, `reasoning`, session, thread, turn id, project (from `cwd`), interface (from `source` / `originator`), gateway, reasoning effort, service tier, context window, time-to-first-token, subagent role |
+| Cannot know | cost; `cache_write` on older CLI builds that never emitted the field (recorded as `null`, not `0`) |
+
+### Token semantics
+
+Codex uses OpenAI's convention where `input_tokens` **includes** `cached_input_tokens`. The
+adapter converts:
+
+```
+input_tokens      = input_tokens − cached_input_tokens     (fresh only)
+cache_read_tokens = cached_input_tokens
+```
+
+### One record per turn, and why the obvious reading is wrong
+
+Codex emits a `token_count` event repeatedly while a turn runs, and `last_token_usage` is
+**re-reported as the turn's context grows** — 27k, then 29k, then 30k, all describing the same
+growing conversation. `total_token_usage` is a running **sum** of those re-reports, so it is not a
+usable cumulative counter.
+
+On a real corpus that error is enormous: one subagent session claimed **7.5 B tokens in
+2.5 minutes** (49,739 `token_count` events for 266 actual turns), and a single day totalled
+**82.8 B** instead of **1.8 B** — a **45x** inflation.
+
+The adapter instead splits each turn's `last_token_usage` series into monotonically
+non-decreasing **runs** (a drop means a context compaction or a new call) and sums each run's
+maximum. This reduces to the identity for simple sessions with one event per turn, and was
+cross-checked against an independent gateway billing log which agreed with the reconstruction
+and not with the naive sum.
+
+Each record carries `metadata.token_count_events` and `metadata.usage_segments` so the
+reconstruction is auditable rather than a hidden fudge.
+
+### Gateways
+
+`model_provider: "headroom"` (or any non-vendor value) is recorded as `gateway`, not as the
+vendor. The vendor comes from the model name.
+
+---
+
+## `cline` — Cline CLI
+
+| | |
+|---|---|
+| Reads | `~/.cline/data/sessions/<id>/<id>.json` (+ `.messages.json` for a message count) |
+| Measurement | `activity` |
+| Reports | session, model (so the vendor is derived — e.g. `deepseek/deepseek-v4-flash` → DeepSeek), provider, duration, status, exit code, project, git branch, team, agent message count |
+| **Cannot know** | **any token count — Cline's session files contain none** |
+
+Every token field is `null`. The dashboard counts these sessions in activity metrics and excludes
+them from token totals, rather than adding a session's worth of zeros. `metadata.tokens_reported`
+is `false` so this is visible in the Data Explorer.
+
+---
+
+## `cursor` — AI code activity
+
+| | |
+|---|---|
+| Reads | `~/.cursor/ai-tracking/ai-code-tracking.db` (read-only, via `node:sqlite`) |
+| Measurement | `activity` |
+| Reports | AI-authored code hashes (model, file, extension, conversation, attribution source: `composer` / `tab` / `human`), and per-commit AI vs human line attribution (`linesAdded`, `composerLinesAdded`, `humanLinesAdded`, AI percentage) |
+| Cannot know | token counts (Cursor does not store them locally) |
+
+This is the productivity-correlation source: it measures **work output**, which is the thing token
+counts are so often wrongly assumed to prove. `model: "default"` and `NULL` are Cursor's own
+placeholders and are recorded as unknown rather than guessed.
+
+The database is snapshotted to a temp directory before reading (with its `-wal`/`-shm` sidecars)
+so a live editor is never disturbed and a hot transaction never yields a stale page.
+
+---
+
+## `opencode` — OpenCode
+
+| | |
+|---|---|
+| Reads | `$XDG_DATA_HOME/opencode/opencode.db` (`~/.local/share/opencode/opencode.db`), via `node:sqlite` |
+| Measurement | `primary` |
+| Reports | per-request tokens: fresh input, cache read, cache write, output, reasoning; model, provider/gateway, session, project, agent mode, duration |
+| Cannot know | git branch (not stored per message); a surface field — interface stays `Unknown` |
+
+One record per assistant message. Token semantics were verified against a live corpus: `input`
+is **exclusive** of cache reads (on long sessions `cache.read` exceeds `input`, which is
+impossible under OpenAI's inclusive convention), and the five reported fields are disjoint
+addends of the source's own total.
+
+Two things the adapter handles that would otherwise corrupt the numbers:
+
+- **Rows are updated in place.** An assistant row is inserted when the request starts and its
+  totals are revised as the response finalises — on a real corpus every token-bearing row had
+  `time_updated != time_created`. A naive cursor would freeze the first partial snapshot. The
+  adapter re-reads a recent window ordered by `time_updated` and emits only the *delta* for a
+  message whose totals grew — the same tails mechanism the Anthropic adapter uses for streamed
+  snapshots.
+- **Reasoning is sometimes additive.** Most providers report reasoning as a subset of output,
+  but some report it on top. Where `reasoning > output` the only reading that satisfies the
+  schema's subset invariant without losing tokens is `output += reasoning`.
+
+A non-vendor `providerID` (e.g. `opencode`'s own gateway) is recorded as `gateway`; vendor
+slugs route direct. Zero-token assistant rows are aborted requests, not free API calls, and are
+skipped.
+
+```yaml
+sources:
+  opencode:
+    db: "~/.local/share/opencode/opencode.db"   # override auto-discovery
+```
+
+---
+
+## `hermes` — Hermes agent
+
+| | |
+|---|---|
+| Reads | `~/.hermes/state.db` (`$HERMES_HOME` honoured), via `node:sqlite` |
+| Measurement | `primary` |
+| Reports | per-session-per-model tokens: input, cache read, cache write, output, reasoning; billing provider (as gateway), measured cost when recorded, session source (cli/cron/whatsapp/telegram), cwd, git branch/repo, task, call count, duration |
+| Cannot know | per-request granularity — usage is aggregated per session × model; within-group timing |
+
+Hermes records its own LLM traffic in `session_model_usage` (one row per session × model ×
+billing provider × task) joined to `sessions`. There is no per-request log — `messages.token_count`
+is unpopulated — so the finest honest granularity is one record per group, timestamped at the
+group's first API call.
+
+Token semantics: `input_tokens` is already **exclusive** of cache reads (verified: cache reads
+exceed input many-fold on real sessions). The columns are `NOT NULL DEFAULT 0`, so an
+unreported split and a true zero are indistinguishable in this source; zeros are passed through
+as zeros rather than invented into nulls.
+
+Three things worth knowing:
+
+- **Rows grow.** A session's usage rows are upserted as it makes more calls. Like the opencode
+  adapter, this one re-reads a recent window ordered by `last_seen` and emits only deltas; the
+  base record sits at `first_seen`, a delta at the `last_seen` that revealed it.
+- **Gateway vs vendor.** `billing_provider` ("nous", "openrouter", "openai-codex") is routing,
+  so it becomes `gateway` — unless it IS the vendor ("anthropic" billing for a claude model is
+  a direct call). Unmapped models stay provider `unknown` with their gateway preserved;
+  `"default"` is treated as the placeholder it is.
+- **Measured cost wins.** When hermes recorded `actual_cost_usd > 0` that is passed through as
+  a measured cost and the price-table estimate stands down. Its `estimated_cost_usd` is ignored:
+  two competing estimates would make the Cost page unauditable.
+
+```yaml
+sources:
+  hermes:
+    db: "~/.hermes/state.db"   # override auto-discovery
+```
+
+---
+
+## `headroom` — local LLM gateway (overlay)
+
+| | |
+|---|---|
+| Reads | `~/.headroom/savings_events.jsonl` |
+| Measurement | `overlay` — **excluded from token totals by default** |
+| Reports | **measured** `cost_usd` per request, post-compression prompt tokens actually sent, the compression delta (`before` → `after`), model, client |
+| Cannot know | output tokens, cache split (the savings log has neither) |
+
+The proxy sees the same requests the Codex adapter already recorded. Adding both would double
+count every routed token — hence `overlay`. What only the gateway knows is what the request
+actually **cost**, which gives the Cost page an independent cross-check against the price-table
+estimate instead of a single unverifiable number.
+
+---
+
+## `git` — activity correlation
+
+| | |
+|---|---|
+| Reads | `git log --numstat` per repository |
+| Measurement | `activity` |
+| Reports | commits, files changed, insertions, deletions, author, branch, message |
+| Cannot know | anything about tokens |
+
+Repositories come from, in order: `sources.git.repos`, `sources.git.scanRoots` (searched for
+`.git`, depth 3), and `autoFromUsage: true` — the working directories already observed in ingested
+usage records, which means zero configuration for the common case where you code where you prompt.
+
+```yaml
+sources:
+  git:
+    scanRoots: ["~/code"]
+    author: "you@example.com"    # optional
+    since: "2026-01-01"          # optional
+    autoFromUsage: true
+```
+
+---
+
+## `generic` — CSV / TSV / JSON / JSONL / SQLite
+
+The escape hatch that keeps this project useful for providers nobody has written an adapter for.
+
+```bash
+tokenflow import ~/Downloads/openrouter.csv          # infers a mapping and previews it
+tokenflow import usage.jsonl --field timestamp=ts --field input_tokens=prompt_tokens
+tokenflow import app.db --format sqlite --table usage --field timestamp=created_at
+```
+
+The mapping is saved to `$TOKENFLOW_HOME/mappings/<name>.json` and reused automatically:
+
+```json
+{
+  "name": "openrouter-export",
+  "format": "csv",
+  "files": ["~/Downloads/openrouter-*.csv"],
+  "timestampFormat": "iso",
+  "defaults": { "client": "openrouter", "interface": "API" },
+  "fields": {
+    "timestamp": "created_at",
+    "model": "model",
+    "input_tokens": "prompt_tokens",
+    "output_tokens": "completion_tokens",
+    "cache_read_tokens": "cached_tokens",
+    "estimated_cost": "cost_usd",
+    "session_id": "generation_id"
+  }
+}
+```
+
+Unmapped token fields stay `null`. A row whose timestamp cannot be parsed is rejected rather than
+defaulted to "now". `timestampFormat` accepts `iso`, `epoch_ms`, `epoch_s`.
+
+---
+
+## `mock` — demo data
+
+Deterministic synthetic usage so a new contributor can run `npm run demo` and see a realistic
+dashboard without connecting anything. Only activates when explicitly asked for
+(`TOKENFLOW_DEMO=1` or `providers: [mock]`), so it can never contaminate a real dataset by
+accident. Every record carries `metadata.demo = true` and `machine: "demo-machine"`, and the
+dashboard shows a persistent banner whenever any of it is in scope.
+
+---
+
+## What is deliberately not here
+
+- **Vendor billing APIs.** They would need credentials and a network client. This project has
+  neither by design. If you can export a CSV from a billing console, `tokenflow import` will take it.
+- **Prompt or completion text.** Adapters read counts and metadata and discard content. There is
+  nowhere in the schema to put a prompt.
+- **Anything that writes to a source.** Adapters open source files read-only, and SQLite sources
+  are read from a temp copy.
+
+## Adding your own
+
+See [creating-provider.md](creating-provider.md). Drop a file in
+`$TOKENFLOW_HOME/providers/<id>.js` and it is loaded on the next run — no core changes.

package/docs/skill.md

@@ -0,0 +1,64 @@
+# Agent skill
+
+`skills/tokenflow/` is a self-contained skill that teaches an AI coding agent to install,
+configure, extend and debug this project on an unfamiliar machine. The point is that you should be
+able to say:
+
+> "Install and configure the Tokenflow for my environment."
+
+…and get a working dashboard without explaining anything about the internals.
+
+## What the agent can do with it
+
+- probe the machine for supported AI tools and report what it found
+- write a correct `config.yaml`, including non-standard source paths
+- run a resumable ingest that fits inside short-lived shells
+- validate the result and explain the data-health numbers
+- add a new provider adapter for an unsupported tool
+- build a generic import mapping from an arbitrary CSV/JSON/SQLite export
+- configure pricing, and explain why cost is blank until it is
+- diagnose inflated or missing numbers using the documented failure modes
+- explain any number on the dashboard, including what it deliberately does not claim
+
+## Layout
+
+```
+skills/tokenflow/
+  SKILL.md                    the instructions the agent reads
+  providers/
+    detection-matrix.md       where each tool stores usage, per OS
+    adapter-template.js       a commented starting point
+  schemas/
+    normalized-record.json    JSON Schema for the unified record
+    config.schema.json        JSON Schema for config.yaml
+  examples/
+    config.yaml               a fully worked configuration
+    generic-mapping.json      a saved import mapping
+    session-transcript.md     the awkward real-world shapes, annotated
+```
+
+## Using it
+
+**Claude Code / Cowork:** copy the folder into `.claude/skills/` (project) or `~/.claude/skills/`
+(global) and ask for it by name. **Any other agent:** point it at `SKILL.md` — it is plain
+Markdown with no tool-specific assumptions.
+
+```
+> Install and configure the Tokenflow for my environment.
+> Add an adapter for <tool>; its logs are at <path> and look like <sample>.
+> My Aug 15 total looks 40x too high. Diagnose it.
+> Explain what "cache hit rate 94.8%" on my dashboard actually measures.
+```
+
+## Design notes
+
+The skill front-loads the **invariants**, not the API surface, because those are what an agent
+gets wrong: missing is not zero, cache is not input, a gateway is not a vendor, interface is never
+inferred from the model, streaming logs re-report usage, and tokens are not productivity. Each one
+is stated with the concrete failure it prevents — including the measured 45x inflation that
+motivated the Codex reconstruction — because an agent that knows *why* a rule exists applies it to
+cases the rule didn't enumerate.
+
+It also tells the agent to verify rather than assert: run `npm run validate`, cross-check a
+suspicious day against an independent source, and write the "re-reading the same bytes emits
+nothing new" test for any new adapter.