@vimoxshah/tokenflow 1.1.0

package/docs/configuration.md

@@ -0,0 +1,281 @@
+# Configuration reference
+
+Everything Tokenflow knows about your setup lives in one file:
+
+```
+~/.tokenflow/config.yaml
+```
+
+`tokenflow setup` writes it for you by detecting what is on the machine, so most people never
+edit it. This page is the complete reference for when you do. `tokenflow config show` prints the
+*effective* config (defaults merged with your file), and `tokenflow config path` prints the data
+home.
+
+Nothing here is ever transmitted anywhere. There are no keys, tokens, or accounts to configure —
+if a field looks like it wants a credential, you are reading the wrong project.
+
+---
+
+## The whole file, annotated
+
+```yaml
+version: 1
+
+# Timezone used to bucket a timestamp into a day and an hour. Set it explicitly
+# if you travel, or if you want the numbers to match a colleague's. null = this
+# machine's current zone at ingest time.
+timezone: Asia/Kolkata          # null | any IANA zone
+
+# Stamped onto every record. Useful later if several people's exports are ever
+# merged; ignored entirely if you are the only user.
+identity:
+  user: null                    # null = the OS username
+  machine: null                 # null = the hostname
+  team: null
+
+# Which adapters are enabled. Empty means "every adapter that detects".
+# Manage with: tokenflow provider add|remove <id>
+providers:
+  - anthropic
+  - openai
+  - opencode
+  - hermes
+  - cline
+  - cursor
+  - headroom
+  - git
+
+# Per-adapter options. Anything omitted falls back to that adapter's default
+# locations, so this section is usually short.
+sources:
+  anthropic:
+    # Claude Code / Agent SDK. Several homes are normal: one per client, or a
+    # separate profile per company.
+    paths: ["~/.claude", "~/.claude-work"]
+  openai:
+    paths: ["~/.codex"]         # Codex CLI / IDE / Desktop
+  opencode:
+    db: "~/.local/share/opencode/opencode.db"   # $XDG_DATA_HOME-aware
+  hermes:
+    db: "~/.hermes/state.db"    # $HERMES_HOME honoured
+  cline:
+    path: "~/.cline/data/sessions"
+  cursor:
+    db: "~/.cursor/ai-tracking/ai-code-tracking.db"
+  headroom:
+    path: "~/.headroom"         # gateway overlay: measured cost, tokens excluded
+  git:
+    # Correlate usage with shipped work. Both are optional.
+    scanRoots: ["~/code", "~/work"]   # directories to scan for repositories
+    repos: []                          # or list them explicitly
+    autoFromUsage: true                # also use working dirs seen in usage records
+
+store:
+  keepRaw: true                 # keep request-level records (Data Explorer + full export)
+  rawRetentionDays: null        # null = keep forever; a number prunes older shards
+
+analytics:
+  # A gateway/proxy log describes traffic a client adapter already counted.
+  # Including it double-counts tokens; excluding it still surfaces its measured
+  # cost. Leave false unless you know you want the other behaviour.
+  includeOverlaySources: false
+  # Only used for sources that do not carry a session id of their own.
+  minSessionGapMinutes: 30
+
+# Your own model classification rules, evaluated before the built-ins. Use this
+# when a gateway renames models, or for an in-house model the built-ins can't know.
+modelMappings:
+  - match: "^acme-"             # regex against the model name
+    provider: acme
+    label: Acme
+    family: Acme v2
+
+# Force an interface for a client the adapter can't classify from its own signals.
+# Values: CLI | Desktop | IDE | Web | API | Unknown
+interfaceOverrides:
+  my-wrapper-script: CLI
+
+ui:
+  skin: aurora                  # aurora | terminal | editorial
+  mode: dark                    # dark | light
+  port: 7799                    # the dashboard binds 127.0.0.1 on this port
+  defaultRange: all             # all | 7d | 30d | 90d | mtd
+  defaultFrom: null             # e.g. "2026-03-14" — a floor for the default view only
+```
+
+---
+
+## The fields that actually change behaviour
+
+### `timezone`
+
+A day boundary is a decision, not a fact. Tokens are bucketed into `date` and `hour` **at ingest
+time** using this zone, and the value is stored on the record — so changing it later requires
+`tokenflow refresh --full` to take effect on history. Set it if you want stable days across
+travel.
+
+### `providers`
+
+An empty list means "auto": every adapter that detects gets used. Naming them explicitly is
+faster (no detection work for tools you don't have) and predictable. `tokenflow providers` shows
+what is detected, what is enabled, and *why* something is not available.
+
+### `sources.<id>.paths`
+
+Adapters look in conventional locations. Point them elsewhere when your tool is installed
+somewhere unusual, or when you keep several profiles:
+
+```yaml
+sources:
+  anthropic:
+    paths: ["~/.claude", "~/.claude-clientA", "/Volumes/work/.claude"]
+```
+
+A path that does not exist is skipped silently — listing a machine's worth of possibilities is
+fine and costs nothing.
+
+### `analytics.includeOverlaySources`
+
+Set this to `true` only if you *want* gateway traffic counted twice — for example when the
+gateway is your only record of a tool that writes no logs of its own. The default (`false`) keeps
+tokens honest and still reports the gateway's measured cost separately.
+
+### `store.rawRetentionDays`
+
+Aggregates (the cube, sessions, activity) are tiny and kept forever. The request-level shards are
+what grow. Set a number of days if you want them pruned; the aggregates already computed are not
+affected, so your history stays in the charts even after the raw rows are gone. `keepRaw: false`
+skips writing them entirely — the Data Explorer and the full CSV export then have nothing to
+show, so prefer a retention window over turning it off.
+
+### `ui.skin` / `ui.mode`
+
+The default look. A choice made in the dashboard header is remembered in the browser and wins
+over this. Series colours belong to the **mode**, not the skin, and were validated for
+colour-blind separation against every skin's chart surface — which is why changing the look can
+never change what a colour means.
+
+### `limits` — declared capacity caps
+
+TokenFlow never invents vendor quota data. A limit exists because you wrote it here, and the
+engine evaluates it against measured consumption (Live tab, `tokenflow capacity`, menu bar):
+
+```yaml
+limits:
+  - id: anthropic-monthly     # required, unique
+    provider: anthropic       # optional cube filters — provider | model | project
+    scope: month              # day | week | month  (your local calendar)
+    metric: tokens            # tokens | input | output | requests | cost
+    cap: 120000000            # tokens, or dollars when metric is cost
+    warnAt: 0.8               # optional warn threshold (default 0.8)
+```
+
+From each limit the engine derives: consumption in the window, % used, remaining,
+today's hourly pace, a trailing-7-day daily pace, projected exhaustion ETA and
+the exact reset instant in your timezone. Invalid definitions are reported via
+`tokenflow capacity --json` (`invalid`) and by the dashboard — never silently
+ignored. See [live-mode.md](live-mode.md) for semantics.
+
+### `watch` — the background refresher
+
+```yaml
+watch:
+  intervalSeconds: 120      # between incremental refresh passes
+  notifications: false      # OS notifications on limit crossings / new anomalies
+  staleAfterSeconds: 600    # when live surfaces should call the data stale
+```
+
+The watcher only runs while you started it (`tokenflow watch`); nothing is
+installed or auto-started. `notifications: true` is what `--notify` overrides
+per run.
+
+### `ui.menubarMode`
+
+Default display mode for `tokenflow status --bar` and the menu-bar plugin:
+`auto` (most urgent signal), `tokens`, `cost`, or `limit`.
+
+### `map.showMyLocation`
+
+Off by default — and it is the only geography feature that touches the network.
+When set to `true`, the dashboard's Global activity map resolves THIS machine's
+public IP once via an HTTPS lookup (ipapi.co), caches the derived place
+(city/region/country/coordinates) in `$TOKENFLOW_HOME/data/geo-cache.json`,
+and re-resolves at most monthly. The IP itself is never stored, never logged,
+never displayed. Leave it unset (or `false`) and TokenFlow performs no lookup
+at all: the map then shows only vendor-published provider regions. There is
+no per-request IP capture anywhere in the product.
+
+---
+
+## Pricing is separate
+
+Rates live in `~/.tokenflow/pricing.json`, not in `config.yaml`, because they are data rather
+than configuration — and because the built-in table ships with provenance:
+
+```bash
+tokenflow pricing                 # what is priced, from which source
+tokenflow pricing --sources       # every rate's origin, fetch date and confidence
+tokenflow pricing --set "my-model=3,15,0.3,3.75"   # input,output[,cacheRead[,cacheWrite]] per 1M
+```
+
+Your overrides always beat the built-in table. Anything with no rate stays visibly unpriced —
+never a plausible-looking `$0`.
+
+---
+
+## Environment variables
+
+| Variable | Effect |
+|---|---|
+| `TOKENFLOW_HOME` | move the whole data home (config, pricing, data, cache) |
+| `TOKENFLOW_DEMO=1` | enable the synthetic demo adapter |
+| `AI_USAGE_HOME` | legacy alias for `TOKENFLOW_HOME`, still honoured |
+
+Running several isolated datasets is just several homes:
+
+```bash
+TOKENFLOW_HOME=~/.tokenflow-work tokenflow refresh
+TOKENFLOW_HOME=~/.tokenflow-work tokenflow dashboard --port 7801
+```
+
+If `~/.tokenflow` does not exist but `~/.ai-usage-dashboard` does, the older directory is used —
+an install from before the rename keeps its ingested history instead of silently starting over.
+
+---
+
+## Moving a setup to another machine
+
+```bash
+# old machine
+tokenflow config export backup.json     # config + pricing + mappings, no usage data
+tokenflow export --csv --all            # the usage data, if you want it too
+
+# new machine
+tokenflow config import backup.json
+tokenflow restore tokenflow-usage-2026-08-20.csv --yes
+```
+
+The CSV carries counts and identifiers only — prompts, conversations and source code never leave
+the machine that produced them. See [cli.md § restore](cli.md#restore-filecsv).
+
+---
+
+## Where everything lives
+
+```
+~/.tokenflow/
+  config.yaml        this file
+  pricing.json       your rate overrides
+  preferences.json   dashboard state (filters, tab, theme)
+  mappings/          saved generic-import field mappings
+  providers/         your own adapters (*.js) — loaded automatically
+  data/
+    records/         YYYY-MM.jsonl, request-level facts
+    cube.json        the pre-aggregated table the dashboard loads
+    sessions.json    one row per session
+    activity.json    daily work-activity rollup
+    state.json       per-file ingest offsets — this is what makes refresh incremental
+  cache/
+```
+
+Delete the directory and Tokenflow knows nothing. Nothing lives anywhere else.

package/docs/creating-provider.md

@@ -0,0 +1,262 @@
+# Creating a provider
+
+An adapter parses; the engine does everything else. That split is why a new adapter is ~80 lines
+and why the analytics layer never sees a vendor-specific shape.
+
+The engine handles: model → vendor classification, interface classification, timezone resolution,
+cost estimation, id assignment, dedup bookkeeping, shard writes, and the cube / session /
+activity rollups. **Do not do any of that in an adapter.**
+
+## The contract
+
+```ts
+interface UsageProvider {
+  id: string;                       // lowercase slug, unique
+  name: string;                     // human name shown in `tokenflow providers`
+  description?: string;
+  measurement?: 'primary' | 'overlay' | 'activity';   // default 'primary'
+  requires?: string[];              // human prerequisites, shown on failure
+
+  /** Can this adapter run here? Reads nothing but existence/metadata. */
+  detect(ctx): Promise<{ available: boolean; detail?: string; paths?: string[] }>;
+
+  // ---- EITHER a file-based source: implement both -------------------------
+  discover?(ctx): Promise<Array<{ key: string; path: string; stat: fs.Stats }>>;
+  ingestFile?(ref, ctx, emit): Promise<{ offset: number; records?: number; malformed?: number }>;
+
+  // ---- OR a fetch-based source (SQLite, API, generator) -------------------
+  fetchUsage?(ctx, emit, sourceState): Promise<{ cursor?: any; records?: number; notes?: string[] }>;
+
+  /** Optional: exposed for tests and dry-run importers. */
+  normalize?(raw: unknown, opts?: unknown): object;
+  getMetadata?(): object;           // provided by createProvider()
+}
+```
+
+`createProvider()` validates this up front, so a broken adapter fails at load with a readable
+message instead of halfway through a multi-gigabyte ingest.
+
+## Minimal file-based adapter
+
+```js
+import fs from 'node:fs';
+import path from 'node:path';
+import os from 'node:os';
+import { createProvider, readLines, walk, MEASUREMENT } from 'tokenflow/sdk';
+
+const root = (ctx) => ctx?.config?.sources?.['my-provider']?.path
+  || path.join(ctx?.home || os.homedir(), '.my-tool', 'logs');
+
+export default createProvider({
+  id: 'my-provider',
+  name: 'My AI Provider',
+  description: 'Per-request usage from My Tool.',
+  measurement: MEASUREMENT.PRIMARY,
+  requires: ['~/.my-tool/logs/*.jsonl'],
+
+  async detect(ctx) {
+    const dir = root(ctx);
+    if (!fs.existsSync(dir)) return { available: false, detail: `no ${dir}` };
+    return { available: true, detail: dir, paths: [dir] };
+  },
+
+  async discover(ctx) {
+    return walk(root(ctx), (name) => name.endsWith('.jsonl')).map((p) => ({
+      key: path.relative(root(ctx), p),   // stable identity for incremental state
+      path: p,
+      stat: fs.statSync(p),
+    }));
+  },
+
+  async ingestFile(ref, ctx, emit) {
+    let records = 0;
+    let malformed = 0;
+    const res = readLines(ref.path, (line) => {
+      let o;
+      try { o = JSON.parse(line); } catch { malformed++; return; }
+      if (!o.usage) return;
+      emit({
+        timestamp: o.created_at,
+        model: o.model,
+        input_tokens: o.usage.prompt_tokens,
+        cache_read_tokens: o.usage.cached_tokens ?? null,
+        cache_write_tokens: null,
+        output_tokens: o.usage.completion_tokens,
+        session_id: o.conversation_id,
+        request_id: o.id,
+        project: o.cwd ? path.basename(o.cwd) : null,
+        interfaceSignals: [o.client, o.surface],
+      });
+      records++;
+    }, {
+      start: ref.start,           // ALWAYS resume from here
+      must: ['"usage"'],          // cheap prefilter: skip lines that cannot match
+    });
+    return { offset: res.offset, records, malformed };
+  },
+});
+```
+
+## The five rules
+
+### 1. Resume from `ref.start` and return `offset`
+
+```js
+const res = readLines(ref.path, handler, { start: ref.start });
+return { offset: res.offset };
+```
+
+This is what makes refresh incremental and exact. `readLines` returns the offset of the last
+**complete** line, so a partially-written trailing line is never consumed. Ignoring `ref.start`
+means every refresh re-emits the whole file.
+
+### 2. Unreported is `null`, not `0`
+
+```js
+cache_write_tokens: o.usage.cache_write ?? null,   // right
+cache_write_tokens: o.usage.cache_write || 0,      // WRONG — the linter fails the build
+```
+
+If the source *does* report zero, pass `0`. The difference is the entire point of the schema.
+
+### 3. Convert to the exclusive convention
+
+`input_tokens` must be **fresh** prompt tokens. If your vendor's `input_tokens` includes cached
+tokens (OpenAI's does), subtract:
+
+```js
+const fresh = input === null ? null : cached === null ? input : Math.max(0, input - cached);
+```
+
+`cache_refresh_tokens` must be a subset of `cache_write_tokens`, and `reasoning_tokens` a subset
+of `output_tokens`. `validateUsage` rejects records that break this.
+
+### 4. Never infer the interface from the model
+
+Pass `interfaceSignals: [...]` — an ordered, strongest-first list of **surface** fields
+(`entrypoint`, `originator`, `source`, an IDE marker, a client name). The engine classifies them.
+With no signal the interface is `Unknown`, which is the correct answer.
+
+Pass `interface: 'CLI'` directly only when your source *is* the surface (the git adapter does).
+
+### 5. Watch for re-reported usage
+
+If your source streams progress updates that re-report a growing total, summing them will inflate
+everything. Check whether a series of usage events for one logical call is monotonically growing
+(→ take the maximum) or genuinely disjoint (→ sum). Both real adapters in this repo needed the
+former; one was inflating by 45x before it was fixed. Record the event count in `metadata` so the
+reconstruction is auditable.
+
+## Fetch-based adapters
+
+For SQLite, APIs, or generators, implement `fetchUsage` and keep a cursor:
+
+```js
+async fetchUsage(ctx, emit, sourceState) {
+  const cursor = sourceState?.cursor || { lastId: 0 };
+  const db = openReadOnly(dbPath(ctx));            // snapshots + never writes
+  try {
+    for (const row of db.prepare('SELECT * FROM usage WHERE id > ? ORDER BY id').all(cursor.lastId)) {
+      emit({ timestamp: new Date(row.ts).toISOString(), model: row.model, /* … */ });
+      cursor.lastId = row.id;
+    }
+  } finally {
+    db.close();
+  }
+  return { cursor, notes: [] };
+}
+```
+
+The cursor is persisted by the engine, so the next run only reads new rows.
+
+## Per-file adapter state
+
+`ref.state` is a plain object persisted per source file across refreshes. Use it for header
+context that a mid-file resume would otherwise lose — Codex keeps the session metadata there,
+because `session_meta` is on line 1 and a resumed read starts at line 40,000.
+
+```js
+async ingestFile(ref, ctx, emit) {
+  const s = ref.state;                 // survives between refreshes
+  if (d.type === 'session_meta') { s.sessionId = d.payload.session_id; return; }
+  // …
+}
+```
+
+## Testing your adapter
+
+Put a small, realistic fixture in `test/fixtures/` — include the awkward cases: a re-reported
+usage series, a row with a missing field, a synthetic/zero row.
+
+```js
+import my from '../src/providers/my-provider/index.js';
+import { ingestFixtureAsync } from './helpers.js';
+import { validateUsage } from '../src/core/validate.js';
+
+test('my-provider: token semantics', async () => {
+  const { records, result } = await ingestFixtureAsync(my, 'my-provider.jsonl');
+  assert.equal(records.length, 2);
+  assert.equal(records[0].input_tokens, 6000, 'fresh input excludes the cached portion');
+  assert.equal(records[0].cache_write_tokens, null, 'unreported stays null');
+  for (const r of records) assert.ok(validateUsage(r).ok);
+});
+
+test('my-provider: re-reading the same bytes emits nothing new', async () => {
+  const first = await ingestFixtureAsync(my, 'my-provider.jsonl');
+  const again = await ingestFixtureAsync(my, 'my-provider.jsonl', {
+    start: first.result.offset, state: first.state,
+  });
+  assert.equal(again.records.length, 0);
+});
+```
+
+That second test is the one that catches the expensive class of bug.
+
+## Installing it
+
+**For yourself** — drop it in `$TOKENFLOW_HOME/providers/my-provider.js`. Loaded on the next run;
+a user adapter shadows a built-in with the same id.
+
+**For everyone** — add `src/providers/my-provider/index.js`, a fixture, and tests, then:
+
+```bash
+npm test && npm run lint && npm run validate
+```
+
+Please document, in the adapter's header comment, what each vendor field means and which
+convention it follows. That header is the most valuable part of an adapter.
+
+## SDK surface
+
+```js
+import {
+  createProvider, registerProvider, listProviders,
+  normalizeUsage, validateUsage, validateProvider,
+  createRecord, computeTotal, dateParts, hashId,
+  MEASUREMENT, INTERFACE, interfaceClass,
+  classifyModel, classifyInterface,
+  readLines, readJsonLines, walk,
+  openReadOnly, tables, columns, sqliteAvailable,
+  registerAnalytics,
+} from 'tokenflow/sdk';
+```
+
+## Adding analytics instead
+
+If you want a new *metric* rather than a new source, register an analytics plugin — no core
+changes either:
+
+```js
+import { registerAnalytics } from 'tokenflow/sdk';
+
+registerAnalytics({
+  id: 'carbon',
+  section: 'sustainability',
+  title: 'Estimated energy',
+  compute({ rows, ix, totals, sessions, range }) {
+    return { kwh: totals.total * 0.0000003, basis: 'estimated', note: '...' };
+  },
+});
+```
+
+The result appears at `view.plugins.carbon`. See [architecture.md](architecture.md#plugin-points).