@vimoxshah/tokenflow 1.1.0

package/skills/tokenflow/examples/session-transcript.md

@@ -0,0 +1,191 @@
+# Real log shapes, annotated
+
+The awkward parts of real transcripts, with the trap each one sets. Reading this before writing an
+adapter is worth more than reading the schema.
+
+---
+
+## 1. Claude Code — one message written three times
+
+```jsonc
+// line 1
+{"type":"assistant","requestId":"req_011Ce…","timestamp":"2026-08-18T10:14:45.100Z",
+ "sessionId":"ec553ff4…","cwd":"/Users/you/projects/billing","gitBranch":"main",
+ "entrypoint":"cli","version":"2.1.234","isSidechain":false,
+ "message":{"id":"msg_011Ce…","model":"claude-haiku-4-5-20251001",
+   "usage":{"input_tokens":10,"output_tokens":4,
+            "cache_creation_input_tokens":38477,"cache_read_input_tokens":21339,
+            "cache_creation":{"ephemeral_1h_input_tokens":38477,"ephemeral_5m_input_tokens":0},
+            "output_tokens_details":{"thinking_tokens":2}}}}
+// line 2 — SAME requestId + message.id, output has grown
+{… "usage":{"input_tokens":10,"output_tokens":131,"cache_creation_input_tokens":38477,
+            "cache_read_input_tokens":21339, …}}
+// line 3 — SAME again, final
+{… "usage":{"input_tokens":10,"output_tokens":254,"cache_creation_input_tokens":38477,
+            "cache_read_input_tokens":21339,"output_tokens_details":{"thinking_tokens":123}}}
+```
+
+**Trap:** summing gives `output = 389`. **Correct:** `output = 254` — the maximum. Prompt-side
+counts are constant across the snapshots, which is the tell.
+
+Measured on a real corpus: **2,493 of 5,060** usage rows were re-reports. All duplicates were
+within a single file, so a per-file group buffer is enough — no global index required.
+
+**Mapping:**
+```
+input_tokens                          -> input_tokens        (already fresh-only)
+cache_read_input_tokens               -> cache_read_tokens
+cache_creation_input_tokens           -> cache_write_tokens
+cache_creation.ephemeral_1h_…         -> cache_refresh_tokens  (SUBSET of cache_write)
+output_tokens_details.thinking_tokens -> reasoning_tokens      (SUBSET of output)
+entrypoint                            -> interface signal
+cwd                                   -> project
+```
+
+---
+
+## 2. Claude Code — a synthetic entry that is not an API call
+
+```jsonc
+{"type":"assistant","requestId":null,"uuid":"6e605aee…","sessionId":"eb4cf63e…",
+ "message":{"id":"c0dfe116…",
+   "usage":{"input_tokens":0,"output_tokens":0,"cache_creation_input_tokens":0,
+            "cache_read_input_tokens":0}}}
+```
+
+**Trap:** it looks like a request. It isn't — no `requestId` and every count is zero. Counting it
+adds fake zero-token requests and drags "tokens per request" down. **Rule:** skip when
+`requestId` is null AND every usage field is zero. On the sampled corpus this was ~60% of lines
+containing `"usage"`.
+
+---
+
+## 3. Codex — `input_tokens` includes cached tokens
+
+```jsonc
+{"timestamp":"2026-08-14T15:32:14.124Z","type":"event_msg","payload":{"type":"token_count",
+ "info":{"total_token_usage":{"input_tokens":25243,"cached_input_tokens":4864,
+                              "cache_write_input_tokens":0,"output_tokens":140,
+                              "reasoning_output_tokens":82,"total_tokens":25383},
+         "last_token_usage":{"input_tokens":25243,"cached_input_tokens":4864,
+                             "output_tokens":140,"reasoning_output_tokens":82,
+                             "total_tokens":25383}}}}
+```
+
+**Trap:** copying `input_tokens` straight through counts 4,864 tokens twice — once as fresh input,
+once as cache read. **Correct:** `fresh = 25243 − 4864 = 20379`.
+
+Note also `cache_write_input_tokens` is **absent** on older sessions. Absent → `null`, not `0`:
+"the CLI didn't report it" and "no cache was written" are different facts.
+
+---
+
+## 4. Codex — the expensive one: re-reported turn usage
+
+One turn (`turn_id: 375dc371`), 1,508 `token_count` events, `last_token_usage.input_tokens`:
+
+```
+27531, 29361, 29533, 29900, … 170071, 170257      ← the SAME context, growing
+```
+
+and `total_token_usage` climbing to **199,900,281** for that single turn.
+
+```
+sum of all last_token_usage           = 201,152,484   ← what a naive adapter reports
+max of the monotonic run              =     244,880   ← the turn's actual context
+```
+
+Whole-file view: 49,739 `token_count` events for **266** `task_started` turns, and 89,000 events
+sharing two wall-clock seconds — this file is a bulk replay, not 49,739 API calls.
+
+**Correct reconstruction:** per turn, split the `last_token_usage` series into monotonically
+non-decreasing **runs** (a drop = a context compaction or a new call — this file had 318
+`context_compacted` events) and sum each run's maximum.
+
+```
+turn with a compaction:  30000, 5000, 9000  ->  30000 + 9000 = 39000   (2 segments)
+simple turn:             20000, 24000, 30000 ->  30000                 (1 segment)
+single-event turn:       100                 ->  100                   (identity)
+```
+
+Effect on one real day: **82.8 B → 1.8 B**, a 45x correction. Validated against an independent
+gateway billing log (~2.1 B input lifetime for the same model), which agreed with the corrected
+figure and not the naive one.
+
+Every emitted record carries `metadata.token_count_events` and `metadata.usage_segments` so the
+reconstruction is auditable.
+
+---
+
+## 5. Codex — a gateway is not a vendor
+
+```jsonc
+{"type":"session_meta","payload":{
+  "session_id":"019fb754…","id":"01a00510…","cwd":"/Users/you/projects/app",
+  "originator":"codex_work_desktop","cli_version":"0.147.0-alpha.6.6",
+  "source":{"subagent":{"thread_spawn":{"depth":1,"agent_role":"luna_worker"}}},
+  "thread_source":"subagent","model_provider":"headroom"}}
+```
+
+```
+model_provider: "headroom"  -> gateway: "headroom"     (a local proxy)
+model: "gpt-5.6-luna"       -> provider: "openai"      (the vendor, from the MODEL)
+originator: codex_work_desktop -> interface: Desktop App
+thread_source: "subagent"   -> category: "subagent"
+```
+
+**Trap:** recording `provider: "headroom"` invents a vendor that doesn't exist and hides all the
+OpenAI usage. Note `source` is sometimes a string (`"cli"`, `"vscode"`, `"exec"`) and sometimes a
+nested object — flatten it before using it as an interface signal.
+
+---
+
+## 6. Cline — a source with no token counts at all
+
+```jsonc
+{"version":1,"session_id":"1785561978480_iuptv","source":"cli",
+ "started_at":"2026-08-01T05:27:23.189Z","ended_at":"2026-08-01T05:27:53.359Z",
+ "status":"completed","provider":"cline","model":"deepseek/deepseek-v4-flash",
+ "cwd":"/Users/you/projects/pipeline","metadata":{"git":{"branch":"main"}}}
+```
+
+**Two traps.** `provider: "cline"` names the **client**, not the vendor — the vendor comes from
+`deepseek/deepseek-v4-flash`. And there is **no usage block anywhere**: every token field must be
+`null` with `measurement: "activity"`. Emitting zeros would add 22 free sessions to the totals and
+pull every average down.
+
+---
+
+## 7. Cursor — activity, not tokens
+
+```sql
+CREATE TABLE ai_code_hashes (hash TEXT PRIMARY KEY, source TEXT, fileExtension TEXT,
+  fileName TEXT, requestId TEXT, conversationId TEXT, timestamp INTEGER,
+  createdAt INTEGER, model TEXT);
+CREATE TABLE scored_commits (commitHash TEXT, branchName TEXT, scoredAt INTEGER,
+  linesAdded INT, composerLinesAdded INT, humanLinesAdded INT, v2AiPercentage TEXT, …);
+```
+
+No token columns exist. This is the **work-output** signal for correlation — the thing token counts
+are so often wrongly assumed to prove. `model: "default"` and `NULL` are Cursor's own placeholders
+and must be recorded as unknown, not guessed. Most `scored_commits` columns are nullable and often
+null: `null` ≠ 0.
+
+---
+
+## 8. A gateway savings log — measured cost, partial tokens
+
+```jsonc
+{"v":1,"ts":"2026-08-13T10:05:06.449119+00:00","before":92783,"after":83161,
+ "saved":9622,"cost_usd":0.028866,"model":"gpt-5.6-sol","client":"codex","source":"proxy"}
+```
+
+```
+after     -> input_tokens        (post-compression: what was actually sent)
+cost_usd  -> measured_cost       (cost_basis: "measured" — a real bill, not an estimate)
+output, cache split              -> null: the savings log has neither
+measurement                      -> "overlay": the Codex adapter already counted this traffic
+```
+
+**Trap:** treating this as primary usage double counts every routed request. Its unique value is
+that `cost_usd` is *measured*, which is what makes the Cost page's estimate checkable.

package/skills/tokenflow/providers/adapter-template.js

@@ -0,0 +1,135 @@
+/**
+ * <PROVIDER NAME> — <what this tool is>
+ *
+ * Source: <exact path pattern>
+ *
+ * ## Token semantics  ← FILL THIS IN. It is the most valuable part of the file.
+ *
+ *   <vendor field>  ->  input_tokens          (fresh only? or inclusive of cache?)
+ *   <vendor field>  ->  cache_read_tokens
+ *   <vendor field>  ->  cache_write_tokens    (null if this version never emits it)
+ *   <vendor field>  ->  output_tokens
+ *   <vendor field>  ->  reasoning_tokens      (a SUBSET of output)
+ *
+ * ## Known quirks
+ *
+ *   - <does it re-report usage as a call streams? then take the max of each
+ *      monotonic run, never the sum>
+ *   - <are there synthetic/local entries with no real API call? skip them>
+ *   - <which fields does it simply not have? those stay null, never 0>
+ */
+import fs from 'node:fs';
+import path from 'node:path';
+import os from 'node:os';
+import { createProvider, readLines, walk, MEASUREMENT } from 'tokenflow/sdk';
+
+const ID = 'my-provider';
+
+function root(ctx) {
+  const configured = ctx?.config?.sources?.[ID]?.path;
+  if (configured) {
+    return configured.startsWith('~') ? path.join(os.homedir(), configured.slice(1)) : configured;
+  }
+  return path.join(ctx?.home || os.homedir(), '.my-tool', 'logs');
+}
+
+export default createProvider({
+  id: ID,
+  name: 'My AI Provider',
+  description: 'Per-request token usage from My Tool.',
+  // primary  = authoritative per-request usage      (counted in totals)
+  // overlay  = a gateway's view of the same traffic (excluded from totals by default)
+  // activity = no token accounting at all           (activity + correlation only)
+  measurement: MEASUREMENT.PRIMARY,
+  requires: ['~/.my-tool/logs/*.jsonl'],
+
+  /** Existence check only. Read no data, write nothing. */
+  async detect(ctx) {
+    const dir = root(ctx);
+    if (!fs.existsSync(dir)) return { available: false, detail: `no ${dir}` };
+    return { available: true, detail: dir, paths: [dir] };
+  },
+
+  /** Enumerate source files. `key` must be stable — it identifies incremental state. */
+  async discover(ctx) {
+    const dir = root(ctx);
+    return walk(dir, (name) => name.endsWith('.jsonl')).flatMap((p) => {
+      let stat;
+      try { stat = fs.statSync(p); } catch { return []; }   // vanished mid-scan
+      if (!stat.size) return [];
+      return [{ key: path.relative(dir, p), path: p, stat }];
+    });
+  },
+
+  async ingestFile(ref, ctx, emit) {
+    // `ref.state` is a plain object persisted per file across refreshes. Use it
+    // for header context a mid-file resume would otherwise lose.
+    const s = ref.state;
+    let records = 0;
+    let malformed = 0;
+
+    const res = readLines(
+      ref.path,
+      (line) => {
+        let o;
+        try { o = JSON.parse(line); } catch { malformed++; return; }
+
+        // Header/metadata lines: stash what later records need.
+        if (o.type === 'session_start') { s.sessionId = o.session_id; s.cwd = o.cwd; return; }
+        if (!o.usage) return;
+
+        // Skip entries that are not real API calls (no id AND all-zero usage).
+        // Counting them adds fake zero-token requests.
+        // if (!o.id && !o.usage.prompt_tokens && !o.usage.completion_tokens) return;
+
+        // If this vendor's input INCLUDES cached tokens, subtract here.
+        const inTotal = num(o.usage.prompt_tokens);
+        const cached = num(o.usage.cached_tokens);
+        const fresh = inTotal === null ? null : cached === null ? inTotal : Math.max(0, inTotal - cached);
+
+        emit({
+          timestamp: o.created_at,          // ISO or epoch; the engine normalises
+          model: o.model,                   // the vendor is derived FROM THIS
+          input_tokens: fresh,
+          cache_read_tokens: cached,
+          cache_write_tokens: num(o.usage.cache_write),   // null when unreported
+          output_tokens: num(o.usage.completion_tokens),
+          reasoning_tokens: num(o.usage.reasoning_tokens),
+          cache_refresh_tokens: null,
+          session_id: s.sessionId || ref.key,
+          request_id: o.id || null,
+          project: s.cwd ? path.basename(s.cwd) : null,
+          repository: s.cwd ? path.basename(s.cwd) : null,
+          git_branch: o.branch || null,
+          category: 'main',
+          client: 'my-tool',
+          application: 'My Tool',
+          // ORDERED, strongest first. Surface fields only — never the model.
+          interfaceSignals: [o.surface, o.client, o.entrypoint],
+          // Measured cost from the source, if it has one. Omit otherwise and the
+          // engine estimates from the price table (or leaves it null).
+          // measured_cost: o.cost_usd,
+          metadata: {
+            cwd: s.cwd || null,
+            version: o.version || null,
+            // Record anything that makes your reconstruction auditable:
+            // usage_events: n, usage_segments: k,
+          },
+        });
+        records++;
+      },
+      {
+        start: ref.start,          // ALWAYS. This is what makes refresh incremental.
+        must: ['"usage"'],         // cheap prefilter — skip lines that cannot match
+      },
+    );
+
+    // Return the offset of the last COMPLETE line so the next refresh resumes here.
+    return { offset: res.offset, records, malformed };
+  },
+});
+
+/** null, never 0, for an absent value. */
+function num(v) {
+  return v === undefined || v === null ? null : Number(v);
+}

package/skills/tokenflow/providers/detection-matrix.md

@@ -0,0 +1,142 @@
+# Detection matrix
+
+Where each tool keeps local usage data, and what can be recovered from it. Probe rather than
+assume — several of these move between versions, and users relocate them.
+
+## Claude Code / Claude Agent SDK → adapter `anthropic`
+
+| OS | Location |
+|---|---|
+| macOS / Linux | `~/.claude/projects/<slugified-cwd>/<sessionId>.jsonl` |
+| alt homes | `$CLAUDE_CONFIG_DIR` (may be `:`-separated), `~/.config/claude`, any `~/.claude-*` containing `projects/` |
+| Windows | `%USERPROFILE%\.claude\projects\...` |
+
+Multiple homes are common (one per account: `~/.claude-work`, `~/.claude-personal`). The adapter
+auto-discovers any `~/.claude*` with a `projects/` directory; add anything else to
+`sources.anthropic.paths`.
+
+**Recoverable:** input, output, cache read, cache write, 1h-ephemeral cache refresh, thinking
+tokens, model, session id, request id, `cwd` → project, git branch, `entrypoint` → interface,
+CLI version, service tier, sidechain (subagent) flag.
+
+**Not recoverable:** cost.
+
+**Retention:** Claude Code prunes transcripts (commonly ~30 days by default). Data already deleted
+cannot be recovered from anywhere.
+
+```bash
+ls -d ~/.claude ~/.claude-* ~/.config/claude 2>/dev/null
+find ~/.claude*/projects -name '*.jsonl' 2>/dev/null | wc -l
+grep -m1 '"usage"' "$(find ~/.claude*/projects -name '*.jsonl' | head -1)" | head -c 600
+```
+
+## Codex CLI / IDE / Desktop → adapter `openai`
+
+| OS | Location |
+|---|---|
+| macOS / Linux | `~/.codex/sessions/YYYY/MM/DD/rollout-<ts>-<id>.jsonl`, plus `~/.codex/archived_sessions/` |
+| alt home | `$CODEX_HOME` |
+
+**Recoverable:** fresh input (after subtracting cached), cached input, cache write (newer builds
+only), output, reasoning, session/thread/turn ids, `cwd` → project, `source`/`originator` →
+interface, `model_provider` → gateway, reasoning effort, service tier, context window,
+time-to-first-token, subagent role.
+
+**Not recoverable:** cost.
+
+**Two traps** — see invariants 3 and 4 in `SKILL.md`. `input_tokens` **includes**
+`cached_input_tokens`, and `token_count` events re-report a turn's usage as it grows while
+`total_token_usage` is a running sum of those re-reports (unusable as a cumulative counter).
+
+```bash
+find ~/.codex/sessions -name '*.jsonl' | wc -l
+f=$(find ~/.codex/sessions -name '*.jsonl' | tail -1)
+grep -c '"token_count"' "$f"; grep -c '"task_started"' "$f"   # ratio >> 1 means re-reporting
+```
+
+## Cline CLI → adapter `cline`
+
+`~/.cline/data/sessions/<id>/<id>.json` (+ `<id>.messages.json`).
+
+**Recoverable:** session id, `provider`, `model` (so the vendor is derived — `deepseek/...` →
+DeepSeek), start/end, status, exit code, `cwd`, git metadata, team, message counts.
+
+**NOT recoverable: any token count.** All token fields must be `null` and the record
+`measurement: activity`.
+
+```bash
+ls ~/.cline/data/sessions | wc -l
+cat ~/.cline/data/sessions/*/*[!s].json | head -40
+```
+
+## Cursor → adapter `cursor`
+
+`~/.cursor/ai-tracking/ai-code-tracking.db` (SQLite; needs `node:sqlite`, i.e. Node ≥ 22.5).
+
+Tables that matter: `ai_code_hashes` (AI-authored code events: model, file, extension,
+conversation, `source` ∈ composer/tab/human) and `scored_commits` (per-commit AI vs human line
+attribution).
+
+**Recoverable:** AI edit activity, per-commit attribution, model names as Cursor labels them
+(`claude-4.5-sonnet`, `composer-1`, and the placeholder `default`/`NULL` → unknown).
+
+**Not recoverable:** token counts — Cursor does not store them locally.
+
+Read from a temp snapshot (with `-wal`/`-shm`) so a live editor is never disturbed.
+
+```bash
+ls -la ~/.cursor/ai-tracking/
+node -e "const{DatabaseSync}=require('node:sqlite');const d=new DatabaseSync(process.argv[1]);console.log(d.prepare(\"select name from sqlite_master where type='table'\").all())" ~/.cursor/ai-tracking/ai-code-tracking.db
+```
+
+## Local gateways / proxies → adapter `headroom` (or a new overlay adapter)
+
+`~/.headroom/savings_events.jsonl`, plus `proxy_savings.json` and a SQLite store.
+
+**Recoverable:** measured `cost_usd` per request, post-compression prompt tokens actually sent,
+the compression delta, model, client.
+
+**Not recoverable:** output tokens, cache split.
+
+Always `measurement: overlay` — the proxy is a second view of traffic a client adapter already
+recorded. Its unique value is the *measured* cost.
+
+Any router of this shape (LiteLLM, OpenRouter's local proxy, a custom one) fits the same pattern:
+one overlay adapter, measured cost, excluded from token totals.
+
+## Git → adapter `git`
+
+Any repository. Needs `git` on `PATH`.
+
+**Recoverable:** commits, files changed, insertions, deletions, author, branch, message.
+
+Repos come from `sources.git.repos`, `sources.git.scanRoots` (depth 3), or `autoFromUsage: true`
+(the working directories already seen in ingested usage records — zero config when you code where
+you prompt).
+
+## Other tools worth probing
+
+| Tool | Look at | Typically has token counts? |
+|---|---|---|
+| GitHub Copilot CLI | `~/.copilot/logs/` | no |
+| Continue | `~/.continue/dev_data/*.jsonl` | sometimes (`tokensGenerated.jsonl`) |
+| opencode / crush / goose | `~/.opencode`, `~/.crush`, `~/.config/goose` | varies by version |
+| Aider | `.aider.chat.history.md`, `.aider.llm.history` | partial, text-formatted |
+| Zed | `~/.local/share/zed/` | no |
+| Ollama | `~/.ollama/logs/server.log` | prompt/eval counts in the log |
+| OpenAI / Anthropic consoles | CSV export | yes → use `tokenflow import` |
+| LiteLLM | its own DB/logs | yes → an overlay adapter |
+
+For anything with an export, prefer `tokenflow import` over an adapter — a saved mapping is less
+code to maintain. Write an adapter when the tool writes a log continuously.
+
+## Quick machine sweep
+
+```bash
+for d in .claude .claude-* .codex .cline .cursor .headroom .continue .copilot .opencode .crush .aider .ollama; do
+  [ -e "$HOME/$d" ] && printf '%-22s %s\n' "$d" "$(du -sh "$HOME/$d" 2>/dev/null | cut -f1)"
+done
+```
+
+Size is a useful signal: a directory in the hundreds of MB is almost certainly full of
+transcripts worth ingesting.

package/skills/tokenflow/schemas/config.schema.json

@@ -0,0 +1,107 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://tokenflow/schemas/config.schema.json",
+  "title": "tokenflow config.yaml",
+  "type": "object",
+  "additionalProperties": false,
+  "properties": {
+    "version": { "type": "integer", "const": 1 },
+    "timezone": { "type": ["string", "null"], "description": "IANA zone used to resolve date/hour/dow at ingest. null = this machine's zone." },
+    "identity": {
+      "type": "object",
+      "additionalProperties": false,
+      "description": "Stamped onto records so a future team deployment can aggregate across machines.",
+      "properties": {
+        "user": { "type": ["string", "null"] },
+        "machine": { "type": ["string", "null"] },
+        "team": { "type": ["string", "null"] }
+      }
+    },
+    "providers": {
+      "type": "array",
+      "items": { "type": "string" },
+      "description": "Enabled adapter ids. Empty means 'all detected'."
+    },
+    "sources": {
+      "type": "object",
+      "description": "Per-adapter options. Unknown keys are passed to the adapter untouched.",
+      "additionalProperties": true,
+      "properties": {
+        "anthropic": {
+          "type": "object",
+          "properties": { "paths": { "type": "array", "items": { "type": "string" }, "description": "Claude Code homes. Overrides auto-discovery." } }
+        },
+        "openai": {
+          "type": "object",
+          "properties": { "paths": { "type": "array", "items": { "type": "string" }, "description": "Codex homes." } }
+        },
+        "cline": { "type": "object", "properties": { "path": { "type": "string" } } },
+        "cursor": { "type": "object", "properties": { "db": { "type": "string" } } },
+        "headroom": { "type": "object", "properties": { "path": { "type": "string" } } },
+        "git": {
+          "type": "object",
+          "additionalProperties": false,
+          "properties": {
+            "repos": { "type": "array", "items": { "type": "string" } },
+            "scanRoots": { "type": "array", "items": { "type": "string" }, "description": "Searched for .git, depth 3." },
+            "autoFromUsage": { "type": "boolean", "default": true, "description": "Also use working directories seen in ingested usage records." },
+            "author": { "type": "string" },
+            "since": { "type": "string" }
+          }
+        },
+        "generic": {
+          "type": "object",
+          "properties": { "imports": { "type": "array", "items": { "type": "object" }, "description": "Inline mappings; see mappings/*.json." } }
+        },
+        "mock": {
+          "type": "object",
+          "properties": { "days": { "type": "integer" }, "seed": { "type": "integer" } }
+        }
+      }
+    },
+    "store": {
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+        "keepRaw": { "type": "boolean", "default": true, "description": "Keep request-level shards. false still builds the cube; only the Data Explorer and full CSV export need them." },
+        "rawRetentionDays": { "type": ["integer", "null"], "default": null }
+      }
+    },
+    "analytics": {
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+        "includeOverlaySources": { "type": "boolean", "default": false, "description": "Include gateway/proxy records in token totals. Off by default: it double counts." },
+        "minSessionGapMinutes": { "type": "integer", "default": 30, "description": "Used only for sources with no session id." }
+      }
+    },
+    "modelMappings": {
+      "type": "array",
+      "description": "User rules, PREPENDED to the built-in model->vendor ruleset.",
+      "items": {
+        "type": "object",
+        "required": ["match", "provider"],
+        "properties": {
+          "match": { "type": "string", "description": "Case-insensitive regex against the raw model string." },
+          "provider": { "type": "string" },
+          "label": { "type": "string" },
+          "family": { "type": "string" }
+        }
+      }
+    },
+    "interfaceOverrides": {
+      "type": "object",
+      "description": "client id -> interface. Use when you know a client's surface and the source doesn't say.",
+      "additionalProperties": { "type": "string", "enum": ["CLI", "IDE", "Desktop App", "Web", "API", "SDK", "Extension", "Unknown"] }
+    },
+    "ui": {
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+        "theme": { "type": "string", "enum": ["dark", "light"], "default": "dark" },
+        "defaultRange": { "type": "string", "enum": ["all", "7d", "30d", "90d", "mtd", "lastmonth", "today", "yesterday"], "default": "all" },
+        "defaultFrom": { "type": ["string", "null"], "description": "Floor for the DEFAULT VIEW only (YYYY-MM-DD). Does not filter the store or --all exports." }
+      }
+    }
+  }
+}

package/skills/tokenflow/schemas/normalized-record.json

@@ -0,0 +1,63 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://tokenflow/schemas/normalized-record.json",
+  "title": "NormalizedUsageRecord",
+  "description": "The unified record every provider adapter emits. MISSING-VALUE CONTRACT: a token field is number|null. null means 'this source does not report it'; 0 means 'it reported zero'. Analytics never coerce null to 0.",
+  "type": "object",
+  "required": ["id", "timestamp", "date", "hour", "dow", "provider", "model", "interface", "measurement", "source"],
+  "additionalProperties": false,
+  "properties": {
+    "id": { "type": "string", "description": "Stable dedup key: hash(source, session, request||timestamp, model, seq)." },
+    "timestamp": { "type": "string", "format": "date-time", "description": "ISO-8601, UTC." },
+    "date": { "type": "string", "pattern": "^\\d{4}-\\d{2}-\\d{2}$", "description": "Resolved in the capture timezone at ingest, then stored." },
+    "hour": { "type": "integer", "minimum": 0, "maximum": 23, "description": "Capture-timezone hour." },
+    "dow": { "type": "integer", "minimum": 0, "maximum": 6, "description": "0=Mon .. 6=Sun, capture timezone." },
+    "tz_offset": { "type": ["integer", "null"], "description": "Capture timezone offset in minutes." },
+
+    "provider": { "type": "string", "description": "Canonical vendor slug derived from the model name. 'unknown' when it cannot be determined — never a guess." },
+    "provider_label": { "type": "string" },
+    "gateway": { "type": ["string", "null"], "description": "Routing layer (proxy/router) that served the request. NOT the vendor." },
+    "model": { "type": "string", "description": "Raw model identifier from the source, preserved verbatim." },
+    "model_family": { "type": "string", "description": "Human tier, e.g. 'Claude Opus 5', 'GPT 5.6 (sol)'." },
+
+    "client": { "type": "string", "description": "Tool that made the call: claude-code | codex | cline | cursor | ..." },
+    "application": { "type": "string" },
+    "interface": {
+      "type": "string",
+      "enum": ["CLI", "IDE", "Desktop App", "Web", "API", "SDK", "Extension", "Unknown"],
+      "description": "Derived ONLY from an explicit surface field. Never inferred from the model or provider. 'Unknown' when there was no signal."
+    },
+
+    "input_tokens": { "type": ["number", "null"], "minimum": 0, "description": "FRESH prompt tokens, EXCLUDING cache read and cache write. Adapters for OpenAI-convention sources must subtract cached_input_tokens." },
+    "cache_read_tokens": { "type": ["number", "null"], "minimum": 0, "description": "Prompt tokens served from a prompt cache." },
+    "cache_write_tokens": { "type": ["number", "null"], "minimum": 0, "description": "Prompt tokens written into a prompt cache." },
+    "output_tokens": { "type": ["number", "null"], "minimum": 0, "description": "Generated tokens." },
+    "cache_refresh_tokens": { "type": ["number", "null"], "minimum": 0, "description": "SUBSET of cache_write_tokens (long-TTL / refreshed writes). Never added into a total again." },
+    "reasoning_tokens": { "type": ["number", "null"], "minimum": 0, "description": "SUBSET of output_tokens (thinking / reasoning). Never added into a total again." },
+    "total_tokens": { "type": ["number", "null"], "minimum": 0, "description": "DERIVED: input + cache_read + cache_write + output. Never trusted from the source." },
+    "total_is_partial": { "type": "boolean", "description": "True when at least one billable field was not available." },
+
+    "session_id": { "type": ["string", "null"] },
+    "conversation_id": { "type": ["string", "null"] },
+    "request_id": { "type": ["string", "null"] },
+
+    "project": { "type": ["string", "null"], "description": "Usually the basename of the working directory." },
+    "repository": { "type": ["string", "null"] },
+    "git_branch": { "type": ["string", "null"] },
+    "category": { "type": ["string", "null"], "description": "main | subagent | commit | ai-edit:* | ..." },
+
+    "estimated_cost": { "type": ["number", "null"], "description": "null when the model has no configured price. NEVER 0 as a stand-in for unknown." },
+    "cost_basis": { "type": ["string", "null"], "enum": ["measured", "estimated", null], "description": "'measured' = the source actually billed it; 'estimated' = computed from the price table." },
+
+    "source": { "type": "string", "description": "Adapter id." },
+    "measurement": {
+      "type": "string",
+      "enum": ["primary", "overlay", "activity"],
+      "description": "primary = authoritative per-request usage, counted in totals. overlay = a gateway's view of traffic already counted elsewhere, excluded from totals by default. activity = no token accounting at all."
+    },
+    "user": { "type": ["string", "null"] },
+    "machine": { "type": ["string", "null"] },
+    "duration_ms": { "type": ["number", "null"] },
+    "metadata": { "type": "object", "description": "Source-specific. Never interpreted by analytics. Put audit trails here (event counts, reconstruction segments, cwd, versions)." }
+  }
+}