context-doctor 0.18.0 → 0.19.0

package/README.md

@@ -90,14 +90,14 @@ Practical upshot: a developer who only wants cheaper, faster API calls never tou
 | Command | What it does |
 |---|---|
 | `context-doctor install` / `uninstall` | Wire (or remove) everything: MCP for Claude Desktop/Code/Cursor/Codex, the Agent Skill, the every-prompt hook |
-| `context-doctor instructions [--copy]` | The ~90-token standing rules for claude.ai / ChatGPT preferences, for web and phones where no server runs |
+| `context-doctor instructions [--copy]` | The ~180-token standing rules (~120 on GPT) for claude.ai / ChatGPT preferences, for web and phones where no server runs |
 | `context-doctor analyze <file>` | Profile a conversation: token breakdown, findings, cost + latency estimates. `--fail-over-budget` exits 1 on a breach, for CI |
 | `context-doctor optimize <file>` | Apply the safe fixes; add `--strategy trim-tool-calls` for big inline file writes, `--strategy prune-history` for consented lossy compaction |
 | `context-doctor session [file]` | Profile a Claude Code session: live context, findings, **measured tokens and prompt-cache economics**, **where the wall clock went** per tool, and **what its subagents cost** (their own windows, your bill; never in the parent's profile). Also reads ChatGPT data exports (`conversations.json`) |
 | `context-doctor init [preset]` | Write a `.contextdoctorrc` from a preset (`chat`, `agent`, `batch`) — a budget you can adopt in one command and tune later |
 | `context-doctor experiment --task "…"` | Run one task twice from the same commit, in a fresh session and forked from an `--existing` one, same model and tools; compare bill, cache split, wall clock, and whether `--check` passed. The only command here that spends money, so it caps spend per arm and refuses a dirty tree |
 | `context-doctor diff <before> <after>` | Compare two profiles: what moved by category, which findings were resolved or introduced, and what it saves in money and latency |
-| `context-doctor accuracy` | How much of what you are billed for is visible in your transcript — the fixed harness baseline and the per-turn injected content neither you nor the profiler can see |
+| `context-doctor accuracy` | How much of what you are billed for is visible in your transcript (the fixed harness baseline, per-turn injected content), plus a tokenizer check: real chars/token per model from the API's own counts, next to the ratio the estimator uses |
 | `context-doctor cursor [--list]` | Profile a chat from Cursor's local history (both storage formats) |
 | `context-doctor report` | Machine-wide impact report (proxy savings persist across restarts): exact proxy savings, hook activity, recoverable waste in recent sessions |
 | `context-doctor proxy` | Always-on local proxy that optimizes every Anthropic/OpenAI API request in flight (`/stats` for cumulative savings) |
@@ -203,13 +203,13 @@ Your API key still rides in the request headers, as before. The token protects t
 | **Claude Code** | Yes: hook on every prompt, status line on every refresh | Past ~80k tokens the model receives hygiene guidance naming the largest waste; compaction is offered. Measured: 115 automatic checks, 48 warnings, across 32 sessions on one machine |
 | **Cursor** | **Yes**, since 0.15: Cursor loads Claude Code's hook config (`~/.claude/settings.json`) and runs the same hook on every agent prompt, passing its own transcript. Output is accepted through Cursor's Claude-compat layer | Same guidance as Claude Code, inside Cursor's agent, for everyone who ran `install`. Before 0.15 the hook fired but could not read Cursor's transcript format, so it said nothing |
 | **API traffic through the proxy** | Yes: every request rewritten in flight | Fewer tokens, guaranteed, model not consulted |
-| **Claude Desktop** | The standing instruction in every chat (we confirmed in the app bundle that Desktop's `LocalMcpServerManager` reads it), a one-click `context_checkup` prompt, and since 0.17 a `profile_context` the model can actually afford to call from chat | Until 0.17 the tool wanted the whole conversation as its argument, so calling it from chat meant re-typing 50k tokens; nobody did, and Desktop's log showed zero calls in a month. Now the model passes a ~100-token **sketch** (turn count, the large or repeated blocks) and gets a sized estimate, findings and the fix to apply. Still a nudge, not a hook: Desktop chat has no hook API and no transcript on disk |
+| **Claude Desktop** | The standing instruction in every chat (we confirmed in the app bundle that Desktop's `LocalMcpServerManager` reads it), a one-click `context_checkup` prompt, and since 0.17 a `profile_context` the model can actually afford to call from chat | Until 0.17 the tool wanted the whole conversation as its argument, so calling it from chat meant re-typing 50k tokens; nobody did, and Desktop's log showed zero calls in a month. Now the model passes a ~120-token **sketch** (turn count, the large or repeated blocks) and gets a sized estimate, findings and the fix to apply. Still a nudge, not a hook: Desktop chat has no hook API and no transcript on disk |
 | **Codex (OpenAI): ChatGPT.app's Codex tab, the Codex IDE extension, the `codex` CLI** | **Yes**, since 0.16: `install` writes the hook to `~/.codex/hooks.json`, the MCP server to `~/.codex/config.toml`, and the skill to `~/.codex/skills/`. Codex uses Claude Code's hook contract almost verbatim and passes its own rollout transcript, which carries the API's real usage figures | Same guidance as Claude Code, from measured tokens. One extra step, Codex's rule not ours: a new hook runs only after you trust it once (type `/hooks` in Codex). `session` and `session --list` read Codex rollouts too |
 | **ChatGPT chat UI** | No | No MCP, no hooks, no data path in the chat product itself. Use Codex, or a developer-mode connector at a URL you host |
 
 So "every chat inherently better" is true for Claude Code, Cursor, Codex and the proxy; for Claude Desktop it is "the rules ride in every chat and the checkup is one cheap tool call away"; and not a claim we make for the ChatGPT chat UI.
 
-**Where there is no hook and no MCP at all** (claude.ai on the web, the Claude and ChatGPT phone apps, plain ChatGPT): the app's per-account preferences are read on every turn, which is the closest those surfaces have to a hook. `context-doctor instructions --copy` puts the ~90-token rules on your clipboard and tells you where to paste them (claude.ai Settings > Profile; ChatGPT Settings > Personalization > Custom instructions).
+**Where there is no hook and no MCP at all** (claude.ai on the web, the Claude and ChatGPT phone apps, plain ChatGPT): the app's per-account preferences are read on every turn, which is the closest those surfaces have to a hook. `context-doctor instructions --copy` puts the ~180-token rules on your clipboard and tells you where to paste them (claude.ai Settings > Profile; ChatGPT Settings > Personalization > Custom instructions).
 
 **Do you need to configure anything by hand? Usually no:**
 
@@ -238,7 +238,7 @@ For any other MCP client, the server entry is:
 
 1. Run `npx context-doctor install` (writes the config above for you) and restart Claude Desktop. Or open the `.mcpb` from the latest release: same server, no npm, installs as an Extension.
 2. From then on, **every conversation carries context-doctor's standing instructions**. The MCP server hands them to Desktop on connect and Desktop puts them in front of Claude: summarize big pastes instead of re-quoting them, refer to earlier content by name, never inline base64, and past ~30 turns or on any question about tokens, cost, speed or limits, call `profile_context` before answering.
-3. That call is cheap on purpose. Claude cannot export a Desktop chat, so it passes a **sketch**: how many turns, which blocks are large, repeated, stale or images, with one size hint each (~100 tokens). The server sizes it (±30%, and it says so), prices the per-turn re-read (on a subscription that is what spends your usage limit), and returns ranked findings with the action for each: "summarize *the nginx config* into the points still needed", "refer to *test output* by name", "offer a 300-token handoff summary for a fresh chat". The reply ends with an instruction to apply the top one, not just suggest it.
+3. That call is cheap on purpose. Claude cannot export a Desktop chat, so it passes a **sketch**: how many turns, which blocks are large, repeated, stale or images, with one size hint each (~120 tokens). The server sizes it (usually within ±20%, measured; see "Why token counts are ~"), prices the per-turn re-read (on a subscription that is what spends your usage limit), and returns ranked findings with the action for each: "summarize *the nginx config* into the points still needed", "refer to *test output* by name", "offer a 300-token handoff summary for a fresh chat". The reply ends with an instruction to apply the top one, not just suggest it.
 4. One click instead of asking: the `context_checkup` prompt in the **+** menu sends that request for you.
 5. Say *"optimize it"* on an exported conversation and Claude applies the safe fixes; if you agree to pruning old history, **Claude itself writes the replacement summary** (that's the no-API-key summarization).
 6. For the same rules on your phone and on claude.ai, where no MCP server runs: `context-doctor instructions --copy`, then paste into Settings > Profile > personal preferences.
@@ -359,9 +359,9 @@ The verdict line is the point: cheaper only counts if it also passed. Because th
 
 ## Exact counts, and what they teach the estimator
 
-The default token count is a chars-per-token heuristic so everything runs with no key and no tokenizer. Its error is content-dependent, and there is no honest way to fix that from transcripts alone (the billed number includes content the transcript never sees). `analyze --exact` fetches a true count for the exact bytes just estimated (Anthropic's count-tokens API with `ANTHROPIC_API_KEY`; tiktoken for GPT if installed) and prints the drift.
+The default token count is a chars-per-token heuristic so everything runs with no key and no tokenizer, with ratios per provider (see "Why token counts are ~" below). `analyze --exact` fetches a true count for the exact bytes just estimated (Anthropic's count-tokens API with `ANTHROPIC_API_KEY`; tiktoken for GPT if installed) and prints the drift.
 
-Since 0.13.9 it also **remembers the comparison**, per model family, on this machine, and later estimates for that family are scaled by it. Nothing about this is silent: the profile header says `estimates calibrated +12% from 3 exact count(s) you ran on this machine`. No exact count ever run means no calibration and unchanged numbers; out-of-range samples are ignored; `CONTEXT_DOCTOR_NO_CALIBRATION=1` returns to the raw heuristic.
+Since 0.13.9 it also **remembers the comparison**, per model family, on this machine, and later estimates for that family are scaled by it. Nothing about this is silent: the profile header says `estimates calibrated +12% from 3 exact count(s) you ran on this machine`. No exact count ever run means no calibration and unchanged numbers; out-of-range samples are ignored; `CONTEXT_DOCTOR_NO_CALIBRATION=1` returns to the raw heuristic. Samples are tied to the heuristic they were taken against: after 0.19 changed Claude's ratios, older samples are ignored and learning restarts, rather than stacking an old correction on a fixed estimator.
 
 ## What it detects
 
@@ -466,7 +466,7 @@ A tool that promises speed must be near-free. Measured overhead per touchpoint:
 | Touchpoint | When it runs | Overhead |
 |---|---|---|
 | Every-prompt hook (Claude Code) | Every prompt | **~80ms** (Node startup; logic ~1ms). Lean sessions exit on a single `stat()` — the transcript is never read. Full profiling (~200ms on a 4MB session) happens only when the transcript has grown ~40% since last checked |
-| MCP server | Spawned once per app session | Tools run only when called; standing instructions cost **~110 tokens per conversation** — deliberately terse |
+| MCP server | Spawned once per app session | Tools run only when called; standing instructions cost **~250 tokens per conversation on Claude, ~170 on GPT** — deliberately terse |
 | Proxy | Per API request | ~1–3ms of CPU (parse → optimize → re-serialize) against typical model latencies of hundreds of ms; responses stream through chunk-by-chunk, never buffered |
 | Skill | Loads only when relevant | ~1k tokens while active; its always-present description is ~60 tokens |
 | Profiling a session | On demand, and on hook growth events | ~160ms for an 8.5MB / 1,855-message transcript (near-duplicate pairs that cannot clear the similarity bar are skipped without comparison) |
@@ -476,18 +476,25 @@ Net effect is strongly negative overhead: the tokens these touchpoints save on e
 
 ## Why token counts are "~" (and where they are exact)
 
-Counting exactly needs each provider's tokenizer, so the default is a calibrated chars-per-token heuristic (denser for code and JSON). It is good enough to rank what is heavy and to measure the effect of a fix, and it keeps the tool offline and zero-config.
+Counting exactly needs each provider's tokenizer, so the default is a chars-per-token heuristic, with ratios per provider and denser ones for code and JSON. It keeps the tool offline and zero-config.
 
-Two ways to get real numbers instead:
+| Model | Prose | Code / tool output | Source |
+|---|---|---|---|
+| Claude (Opus 4.7 to 5.x, Fable 5.x, Sonnet 5) | 2.75 chars/token | 2.4 | Measured from the API's own counts, below |
+| GPT, Gemini, unknown | 4.0 | 3.2 | Usual figures for o200k-class tokenizers; not re-measured here |
 
-- **`analyze --exact`** uses the Anthropic count-tokens API for Claude models (set `ANTHROPIC_API_KEY`; opt-in network call, key never stored) or tiktoken for GPT models (install it alongside), and reports how far the heuristic drifted.
-- **Sessions report measured tokens automatically.** Claude Code transcripts record what the API actually charged, so `session`, the hook and the reports use that figure when it is present — no key, no estimate.
+Which row applies: the model you pass, else the request's own `model` field, else the request's shape (Anthropic's `system` field or `tool_use` blocks mean Claude). Cursor transcripts record no model and use Anthropic-style blocks, so Cursor sessions are counted at Claude density; for a GPT model in Cursor that reads about 40% high.
+
+**How the Claude figures were measured, with no key.** Claude Code transcripts record what the API billed, and two things in them are exact. A reply with no thinking block is billed as exactly its `output_tokens`, and all of it is visible text: 504 replies gave a median of 2.75 chars/token (p10 2.4, p90 3.0). Between two consecutive API calls the prompt grows by exactly what was appended; when that is one large block, its size is the growth minus the previous reply: 474 blocks of code and tool output gave 2.4 (p10 2.1, p90 2.8). The ratios this tool used until 0.19 (4.0 / 3.2 for everything) **undercounted current Claude models by about 40%**: hook warnings came late, savings and costs read low, and the proxy stayed silent on cacheable prefixes between 1,024 and ~1,670 tokens. `context-doctor accuracy` re-runs both measurements on your own sessions and prints them per model beside the ratio in use, so the next tokenizer change shows up as a number, not a surprise. On this machine every model lands within ±9%.
 
-One honest caveat worth knowing: a transcript stores the conversation, **not** the harness's system prompt, tool schemas or skills. Measured against the API's own numbers here, a message-only estimate undercounts the true context by roughly 60%. That is why sessions prefer the reported figure, and why the message breakdown is labelled as covering messages only.
+Two ways to get real numbers instead of estimates:
 
+- **`analyze --exact`** uses the Anthropic count-tokens API for Claude models (set `ANTHROPIC_API_KEY`; opt-in network call, key never stored) or tiktoken for GPT models (install it alongside), and reports how far the heuristic drifted.
+- **Sessions report measured tokens automatically.** Claude Code transcripts record what the API actually charged, so `session`, the hook and the reports use that figure when it is present — no key, no estimate.
 
+One honest caveat worth knowing: a transcript stores the conversation, **not** the harness's system prompt, tool schemas or skills (about 54k tokens before the first turn in Claude Code here), nor the reminders it injects each turn. With the corrected ratios the transcript accounts for a median 56% of each turn's billed growth; before 0.19 this read 39%, and about a third of that "invisible" gap was the estimator. That is why sessions prefer the reported figure, and why the message breakdown is labelled as covering messages only.
 
-Exact counts require each provider's private tokenizer. `context-doctor` uses a calibrated chars-per-token heuristic (denser for code/JSON) that lands within ~10% — plenty accurate for finding what's heavy and measuring savings, and it keeps the tool fully offline with zero configuration.
+**The chat-app sketch** (Claude Desktop, see above) is coarser by design, because the model describes the chat instead of sending it. Its sizes are measured, not assumed: a plain exchange is 2,060 chars (median of 1,283), a code line 42 chars (719 source files), a log line 56 (2,095 tool outputs), a word 6.3, all converted with the model's own ratio. Measured error: the total for a chat of 30+ exchanges from its turn count alone is within -20% to +9% (p10 to p90); a code block sized by lines is within about ±25%, by chars about ±15%. Logs vary from 38 to 100 chars a line, so the tool asks for their size in chars.
 
 ## Roadmap
 

package/dist/accuracy.d.ts

@@ -11,10 +11,12 @@
  * measures the distance between them on your own sessions, so "why is my bill
  * bigger than the profile?" has an answer with evidence behind it.
  *
- * WHAT THIS IS NOT: a tokenizer benchmark. It cannot be — the content behind
- * the gap is unavailable to us, so the gap cannot be attributed to estimator
- * drift. To measure the estimator itself, use `analyze --exact`, which counts
- * the same bytes with the provider's own tokenizer.
+ * The coverage figure uses the model-aware estimator. Estimator drift itself is
+ * measured separately (tokenizer-measure.ts) on content the transcript DOES
+ * hold: replies whose output_tokens are exact, and single large blocks whose
+ * size is the exact prompt growth. Until 0.19 this command attributed the whole
+ * gap to invisible content; about a third of it was the estimator undercounting
+ * Claude's tokenizer.
  */
 export interface AccuracyReport {
     sessionsScanned: number;

package/dist/accuracy.js

@@ -11,10 +11,12 @@
  * measures the distance between them on your own sessions, so "why is my bill
  * bigger than the profile?" has an answer with evidence behind it.
  *
- * WHAT THIS IS NOT: a tokenizer benchmark. It cannot be — the content behind
- * the gap is unavailable to us, so the gap cannot be attributed to estimator
- * drift. To measure the estimator itself, use `analyze --exact`, which counts
- * the same bytes with the provider's own tokenizer.
+ * The coverage figure uses the model-aware estimator. Estimator drift itself is
+ * measured separately (tokenizer-measure.ts) on content the transcript DOES
+ * hold: replies whose output_tokens are exact, and single large blocks whose
+ * size is the exact prompt growth. Until 0.19 this command attributed the whole
+ * gap to invisible content; about a third of it was the estimator undercounting
+ * Claude's tokenizer.
  */
 import { estimateTokens, formatTokens, MESSAGE_OVERHEAD_TOKENS } from "./tokens.js";
 import { parseConversation } from "./parse.js";
@@ -57,7 +59,7 @@ export function measureAccuracy(limit = 20, paths) {
         const normalized = parseConversation(parsed.conversationJson).messages;
         const estimateAt = (i) => {
             const m = normalized[i];
-            return m ? estimateTokens(m.text) + MESSAGE_OVERHEAD_TOKENS : 0;
+            return m ? estimateTokens(m.text, parsed.model) + MESSAGE_OVERHEAD_TOKENS : 0;
         };
         let baseline = usage[0].input;
         for (let i = 0; i < usage[0].index; i++)
@@ -118,7 +120,7 @@ export function renderAccuracy(report) {
     lines.push("still saves real money — it just starts from a higher floor than the profile");
     lines.push("implies.");
     lines.push("");
-    lines.push("This does NOT measure tokenizer drift: the missing content is not available");
-    lines.push("to compare against. For that, use `analyze --exact` (provider tokenizer).");
+    lines.push("The tokenizer check below measures the estimator itself, on content the");
+    lines.push("transcript does hold; `analyze --exact` checks it against the provider's tokenizer.");
     return lines.join("\n");
 }

package/dist/calibration.d.ts

@@ -14,6 +14,14 @@
  * them. No exact count ever run means no calibration, and the numbers are
  * exactly what they were before.
  */
+/**
+ * Bumped whenever the uncalibrated heuristic changes. A factor learned against
+ * an older heuristic would correct for an error that no longer exists (0.19
+ * moved Claude from 4.0 to 2.75 chars/token; an old 1.4x factor on top of that
+ * would overcount by 1.4x), so records from another version are ignored and
+ * restarted rather than blended.
+ */
+export declare const HEURISTIC_VERSION = 2;
 export interface Calibration {
     /** Multiply heuristic estimates by this. 1 means uncalibrated. */
     factor: number;

package/dist/calibration.js

@@ -17,6 +17,14 @@
 import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
 import { dirname, join } from "node:path";
 import { statePath } from "./ledger.js";
+/**
+ * Bumped whenever the uncalibrated heuristic changes. A factor learned against
+ * an older heuristic would correct for an error that no longer exists (0.19
+ * moved Claude from 4.0 to 2.75 chars/token; an old 1.4x factor on top of that
+ * would overcount by 1.4x), so records from another version are ignored and
+ * restarted rather than blended.
+ */
+export const HEURISTIC_VERSION = 2;
 /** Anything outside this is a bad sample, not a calibration. */
 const MIN_FACTOR = 0.5;
 const MAX_FACTOR = 2.0;
@@ -52,8 +60,14 @@ export function recordCalibration(model, exactTokens, heuristicTokens) {
     try {
         const all = readAll();
         const key = modelFamily(model);
-        const rec = all[key] ?? { exactSum: 0, heuristicSum: 0, samples: 0 };
-        all[key] = { exactSum: rec.exactSum + exactTokens, heuristicSum: rec.heuristicSum + heuristicTokens, samples: rec.samples + 1 };
+        const prev = all[key];
+        const rec = prev && prev.v === HEURISTIC_VERSION ? prev : { exactSum: 0, heuristicSum: 0, samples: 0 };
+        all[key] = {
+            exactSum: rec.exactSum + exactTokens,
+            heuristicSum: rec.heuristicSum + heuristicTokens,
+            samples: rec.samples + 1,
+            v: HEURISTIC_VERSION,
+        };
         mkdirSync(dirname(calibrationPath()), { recursive: true });
         writeFileSync(calibrationPath(), JSON.stringify(all, null, 2));
     }
@@ -66,7 +80,7 @@ export function calibrationFor(model) {
     if (process.env.CONTEXT_DOCTOR_NO_CALIBRATION)
         return { factor: 1, samples: 0 };
     const rec = readAll()[modelFamily(model)];
-    if (!rec || rec.samples < 1 || rec.heuristicSum <= 0)
+    if (!rec || rec.v !== HEURISTIC_VERSION || rec.samples < 1 || rec.heuristicSum <= 0)
         return { factor: 1, samples: 0 };
     const factor = rec.exactSum / rec.heuristicSum;
     if (!Number.isFinite(factor) || factor < MIN_FACTOR || factor > MAX_FACTOR)

package/dist/cli.js

@@ -20,6 +20,7 @@ import { runInstall, runUninstall } from "./install.js";
 import { listSessions, parseSessionFile } from "./session.js";
 import { runHook } from "./hook.js";
 import { buildImpactReport } from "./impact.js";
+import { measureTokenizer, renderTokenizer } from "./tokenizer-measure.js";
 import { renderPreferences, copyToClipboard, CHAT_PREFERENCES } from "./preferences.js";
 import { recordLedger } from "./ledger.js";
 import { runDoctor } from "./doctor.js";
@@ -100,7 +101,7 @@ Options:
                           Default: dedupe, trim-tool-results, strip-base64 (lossless-ish set)
   --keep-recent <n>       (optimize) Messages at the tail to leave untouched (default 6)
   --max-tool-tokens <n>   (optimize) Token budget for trimmed tool results (default 300)
-  --limit <n>             (accuracy) Sessions to sample (default 20)
+  --limit <n>             (accuracy, session --list) Sessions to sample or list (default 20)
   --check <cmd>           (experiment) Command whose exit code is the pass/fail for each arm
   --existing <id>         (experiment) Session id to fork the second arm from (never mutated)
   --budget <usd>          (experiment) Spend cap per arm (default 1)
@@ -327,7 +328,8 @@ function main() {
     }
     if (args.command === "accuracy") {
         const report = measureAccuracy(args.limit ?? 20);
-        console.log(args.json ? JSON.stringify(report, null, 2) : renderAccuracy(report));
+        const tokenizer = measureTokenizer(Math.max(args.limit ?? 20, 60));
+        console.log(args.json ? JSON.stringify({ ...report, tokenizer }, null, 2) : `${renderAccuracy(report)}\n\n${renderTokenizer(tokenizer)}`);
         return;
     }
     if (args.command === "report") {
@@ -375,9 +377,9 @@ function main() {
     }
     if (args.command === "session") {
         if (args.list) {
-            const sessions = listSessions();
+            const sessions = listSessions(args.limit ?? 20);
             if (sessions.length === 0) {
-                console.log("No Claude Code sessions found under ~/.claude/projects.");
+                console.log("No sessions found under ~/.claude/projects or ~/.codex/sessions.");
                 return;
             }
             for (const s of sessions) {

package/dist/hook.js

@@ -17,7 +17,7 @@ import { recordLedger, statePath } from "./ledger.js";
 import { parseConversation } from "./parse.js";
 import { profileConversation } from "./profile.js";
 import { parseSessionFile } from "./session.js";
-import { formatTokens } from "./tokens.js";
+import { formatTokens, CHARS_PER_TOKEN } from "./tokens.js";
 import { formatUsd } from "./pricing.js";
 import { checkBudget, loadConfig } from "./config.js";
 /** Default nudge threshold; a project budget or env var can lower/raise it. */
@@ -37,13 +37,16 @@ function warnThreshold(budgetMaxTokens) {
 /** Re-nudge only after the context grows another 40% — one reminder, not a nag. */
 const REGROWTH_FACTOR = 1.4;
 /**
- * Fast-path gate: text tokens are at least ~4 bytes each and the transcript
+ * Fast-path gate: no tokenizer we model packs more than one token into fewer
+ * bytes than its densest ratio (Claude on code, 2.4), and the transcript
  * carries JSON overhead on top, so a file smaller than this cannot possibly
  * hold that many tokens of context. Lean sessions cost one stat() call — the
- * transcript is never even read.
+ * transcript is never even read. (This used 4 bytes per token until 0.19,
+ * which let Claude sessions past the threshold take the fast path.)
  */
+const DENSEST_CHARS_PER_TOKEN = Math.min(...Object.values(CHARS_PER_TOKEN).map((r) => Math.min(r.prose, r.code)));
 function minBytesForWarn(threshold) {
-    return threshold * 4;
+    return Math.floor(threshold * DENSEST_CHARS_PER_TOKEN);
 }
 /**
  * State lives in one small file per session, not one shared map.

package/dist/index.d.ts

@@ -20,3 +20,4 @@ export type { Provider } from "./tokens.js";
 export { renderStatusLine, tailUsage } from "./statusline.js";
 export * from "./sketch.js";
 export * from "./preferences.js";
+export * from "./tokenizer-measure.js";

package/dist/index.js

@@ -15,3 +15,4 @@ export { estimateTokens, contextWindowFor, providerFor, formatTokens } from "./t
 export { renderStatusLine, tailUsage } from "./statusline.js";
 export * from "./sketch.js";
 export * from "./preferences.js";
+export * from "./tokenizer-measure.js";

package/dist/mcp.js

@@ -29,7 +29,7 @@ import { runSketch } from "./sketch.js";
  * "standing context-hygiene instructions in every chat" — no hook needed.
  */
 // Kept deliberately terse: these ride in EVERY conversation's context, and a
-// context-saving tool must not itself be context overhead (~130 tokens).
+// context-saving tool must not itself be context overhead (~250 Claude tokens).
 //
 // Written as conditions and actions, not offers. On Claude Desktop this string
 // is the only channel that reaches the model without the user asking — MCP has
@@ -40,7 +40,7 @@ import { runSketch } from "./sketch.js";
 const SERVER_INSTRUCTIONS = `Context hygiene rules (always on):
 1. Summarize any paste or tool result over ~2k tokens into the points you will use, then work from the summary; never carry it verbatim.
 2. Reference earlier content by name; never re-quote it. Never inline base64.
-3. When the conversation passes ~30 turns, or holds 3+ large pastes, or the user asks about tokens, cost, speed or limits: call profile_context BEFORE answering and act on its top finding. In a chat app pass a \`sketch\` (turn count + the large/repeated blocks, ~100 tokens), not the conversation. Do not estimate token counts yourself.
+3. When the conversation passes ~30 turns, or holds 3+ large pastes, or the user asks about tokens, cost, speed or limits: call profile_context BEFORE answering and act on its top finding. In a chat app pass a \`sketch\` (turn count + the large/repeated blocks, ~120 tokens), not the conversation. Do not estimate token counts yourself.
 4. If optimize_context returns a pruned-turns digest, you write the ≤150-token replacement summary.`;
 const STRATEGY_IDS = ["dedupe", "trim-tool-results", "trim-tool-calls", "strip-base64", "prune-history"];
 /**
@@ -49,8 +49,8 @@ const STRATEGY_IDS = ["dedupe", "trim-tool-results", "trim-tool-calls", "strip-b
  * recommended pattern.
  */
 function createServer() {
-    const server = new McpServer({ name: "context-doctor", version: "0.18.0" }, { instructions: SERVER_INSTRUCTIONS });
-    server.tool("profile_context", "Profile an LLM conversation or prompt: token breakdown, largest blocks, and actionable findings about wasted context (duplicates, oversized pastes or tool results, base64 blobs, long history). Two inputs, pass ONE: `conversation` (full OpenAI/Anthropic JSON or raw text, for agents, files and proxies) or `sketch` (for chat apps such as Claude Desktop or ChatGPT where you cannot export the conversation: the turn count plus the few blocks that matter, ~100 tokens to write). Call it whenever the user asks about token usage, context size, cost, speed or limits, and on your own once the conversation passes ~30 turns or holds 3+ large pastes. Act on the top finding in your reply.", {
+    const server = new McpServer({ name: "context-doctor", version: "0.19.0" }, { instructions: SERVER_INSTRUCTIONS });
+    server.tool("profile_context", "Profile an LLM conversation or prompt: token breakdown, largest blocks, and actionable findings about wasted context (duplicates, oversized pastes or tool results, base64 blobs, long history). Two inputs, pass ONE: `conversation` (full OpenAI/Anthropic JSON or raw text, for agents, files and proxies) or `sketch` (for chat apps such as Claude Desktop or ChatGPT where you cannot export the conversation: the turn count plus the few blocks that matter, ~120 tokens to write). Call it whenever the user asks about token usage, context size, cost, speed or limits, and on your own once the conversation passes ~30 turns or holds 3+ large pastes. Act on the top finding in your reply.", {
         conversation: z.string().optional().describe("Conversation JSON (OpenAI or Anthropic format, or bare message array) or raw prompt text. Omit in chat apps and pass `sketch`."),
         sketch: z.object({
             turns: z.number().int().nonnegative().describe("User+assistant exchanges so far"),
@@ -66,7 +66,7 @@ function createServer() {
                 repeated: z.number().int().positive().optional().describe("Times this same content appears (2+ = duplicate)"),
                 stale: z.boolean().optional().describe("Already acted on; nothing in it is still needed"),
             })).describe("Only the blocks over ~500 tokens, repeated, or images. Plain turns need not be listed."),
-        }).optional().describe("Coarse description of the conversation for chat apps. Give one size hint per block (lines, words, chars or tokens)."),
+        }).optional().describe("Coarse description of the conversation for chat apps. One size hint per block: lines for code, words for prose, chars (or tokens) for logs and tool output, which vary most per line."),
         model: z.string().optional().describe("Target model name for context-window math, e.g. claude-sonnet-5 or gpt-4o"),
     }, async ({ conversation, sketch, model }) => {
         if (sketch) {

package/dist/optimize.d.ts

@@ -9,6 +9,11 @@
 export type StrategyId = "dedupe" | "trim-tool-results" | "trim-tool-calls" | "prune-history" | "strip-base64";
 export interface OptimizeOptions {
     strategies?: StrategyId[];
+    /**
+     * Model the conversation is for; picks the tokenizer ratios for savings and
+     * trim budgets. Defaults to the conversation's own `model` field.
+     */
+    model?: string;
     /** Tool results older than this many messages from the end get trimmed. */
     keepRecent?: number;
     /** Max tokens a trimmed tool result — or tool-call argument set — keeps. */

package/dist/optimize.js

@@ -7,7 +7,7 @@
  * so the output is a drop-in replacement for the input conversation.
  */
 import { createHash } from "node:crypto";
-import { estimateTokens } from "./tokens.js";
+import { charsPerTokenFor, estimateTokens } from "./tokens.js";
 import { hasBase64Blob, stripBase64Blobs } from "./blob.js";
 const TRIM_BOUNDARY_STEP = 10;
 const DEFAULTS = {
@@ -63,8 +63,10 @@ function editedLaterWithoutRead(messages, writeIndex, path) {
     }
     return false;
 }
-function trimCallArguments(input, maxTokens) {
-    const budgetChars = maxTokens * 4;
+function trimCallArguments(input, maxTokens, model) {
+    // Prose ratio: argument strings are mostly file content and commands, and
+    // the prose ratio keeps a little more than the budget rather than less.
+    const budgetChars = Math.round(maxTokens * charsPerTokenFor("", model));
     const out = {};
     for (const [key, value] of Object.entries(input)) {
         if (typeof value === "string" && value.length > budgetChars) {
@@ -158,8 +160,8 @@ function replaceText(content, newText) {
         out.push({ type: "text", text: newText });
     return out;
 }
-function truncateToTokens(text, maxTokens) {
-    const approxChars = maxTokens * 4;
+function truncateToTokens(text, maxTokens, model) {
+    const approxChars = Math.round(maxTokens * charsPerTokenFor(text, model));
     if (text.length <= approxChars)
         return text;
     const head = text.slice(0, approxChars);
@@ -229,7 +231,15 @@ export function optimizeConversation(input, options = {}) {
         if (!messages[i] || typeof messages[i] !== "object")
             messages.splice(i, 1);
     }
-    const tokensBefore = messages.reduce((s, m) => s + estimateTokens(textOf(m.content)), 0);
+    // The request's own model, else Anthropic's shape (a top-level system, or
+    // tool_use/tool_result blocks) says whose tokenizer counts it.
+    const looksAnthropic = (!Array.isArray(data) && data?.system != null) ||
+        messages.some((m) => Array.isArray(m?.content) && m.content.some((b) => b?.type === "tool_use" || b?.type === "tool_result"));
+    const model = options.model ??
+        (!Array.isArray(data) && typeof data?.model === "string" ? data.model : undefined) ??
+        (looksAnthropic ? "claude" : undefined);
+    const tok = (text) => estimateTokens(text, model);
+    const tokensBefore = messages.reduce((s, m) => s + tok(textOf(m.content)), 0);
     const applied = [];
     // -- strip-base64: replace inline blobs with a placeholder --------------------
     if (opts.strategies.includes("strip-base64")) {
@@ -237,9 +247,9 @@ export function optimizeConversation(input, options = {}) {
             const text = textOf(m.content);
             if (!hasBase64Blob(text))
                 return;
-            const before = estimateTokens(text);
+            const before = tok(text);
             const cleaned = stripBase64Blobs(text);
-            const saved = before - estimateTokens(cleaned);
+            const saved = before - tok(cleaned);
             if (saved > 50) {
                 m.content = replaceText(m.content, cleaned);
                 applied.push({ strategy: "strip-base64", messageIndex: i, tokensSaved: saved, note: "Removed inline base64 data" });
@@ -272,7 +282,7 @@ export function optimizeConversation(input, options = {}) {
                 seen.set(h, i);
                 return;
             }
-            const saved = estimateTokens(text);
+            const saved = tok(text);
             m.content = replaceText(m.content, `[context-doctor: identical to message #${first} — content removed]`);
             applied.push({ strategy: "dedupe", messageIndex: i, tokensSaved: saved, note: `Duplicate of message #${first}` });
         });
@@ -284,20 +294,20 @@ export function optimizeConversation(input, options = {}) {
             if (i >= cutoff || !isToolResultMessage(m))
                 return;
             const text = textOf(m.content);
-            const before = estimateTokens(text);
+            const before = tok(text);
             if (before <= opts.maxToolResultTokens)
                 return;
-            const trimmed = truncateToTokens(text, opts.maxToolResultTokens);
+            const trimmed = truncateToTokens(text, opts.maxToolResultTokens, model);
             // The truncation notice has a length of its own, so a result only just
             // over the budget can come back LARGER than it went in. Measured on a
             // real session: 2,941 tokens "optimized" to 2,947.
-            if (estimateTokens(trimmed) >= before)
+            if (tok(trimmed) >= before)
                 return;
             m.content = replaceText(m.content, trimmed);
             applied.push({
                 strategy: "trim-tool-results",
                 messageIndex: i,
-                tokensSaved: before - estimateTokens(trimmed),
+                tokensSaved: before - tok(trimmed),
                 note: "Stale tool result truncated",
             });
         });
@@ -322,12 +332,12 @@ export function optimizeConversation(input, options = {}) {
                     // those alone; the 63% never touched again are still pure gain.
                     if (editedLaterWithoutRead(messages, i, writtenPath(b)))
                         continue;
-                    const before = estimateTokens(JSON.stringify(b.input));
+                    const before = tok(JSON.stringify(b.input));
                     if (before <= opts.maxToolResultTokens)
                         continue;
-                    const trimmedInput = trimCallArguments(b.input, opts.maxToolResultTokens);
+                    const trimmedInput = trimCallArguments(b.input, opts.maxToolResultTokens, model);
                     // Same trap as tool results: the marker can outweigh what it replaces.
-                    const after = estimateTokens(JSON.stringify(trimmedInput));
+                    const after = tok(JSON.stringify(trimmedInput));
                     if (after >= before)
                         continue;
                     b.input = trimmedInput;
@@ -339,11 +349,11 @@ export function optimizeConversation(input, options = {}) {
                 const args = tc?.function?.arguments;
                 if (typeof args !== "string")
                     continue;
-                const before = estimateTokens(args);
+                const before = tok(args);
                 if (before <= opts.maxToolResultTokens)
                     continue;
-                tc.function.arguments = truncateToTokens(args, opts.maxToolResultTokens);
-                saved += before - estimateTokens(tc.function.arguments);
+                tc.function.arguments = truncateToTokens(args, opts.maxToolResultTokens, model);
+                saved += before - tok(tc.function.arguments);
             }
             if (saved > 0) {
                 applied.push({
@@ -375,7 +385,7 @@ export function optimizeConversation(input, options = {}) {
             // 1,011-message session, which pruned to 7 messages with one orphan.
             dropOrphanedToolResults(messages.slice(keepFrom));
             const pruned = messages.slice(0, keepFrom);
-            const prunedTokens = pruned.reduce((s, m) => s + estimateTokens(textOf(m.content)), 0);
+            const prunedTokens = pruned.reduce((s, m) => s + tok(textOf(m.content)), 0);
             // Digest: first ~200 chars of each pruned turn — enough for a host LLM to
             // write a faithful summary, small enough not to defeat the pruning.
             prunedDigest = pruned
@@ -390,11 +400,11 @@ export function optimizeConversation(input, options = {}) {
             applied.push({
                 strategy: "prune-history",
                 messageIndex: 0,
-                tokensSaved: prunedTokens - estimateTokens(stub.content),
+                tokensSaved: prunedTokens - tok(stub.content),
                 note: `Pruned ${pruned.length} old messages`,
             });
         }
     }
-    const tokensAfter = messages.reduce((s, m) => s + estimateTokens(textOf(m.content)), 0);
+    const tokensAfter = messages.reduce((s, m) => s + tok(textOf(m.content)), 0);
     return { conversation: data, tokensBefore, tokensAfter, applied, prunedDigest };
 }

package/dist/parse.d.ts

@@ -34,6 +34,8 @@ export interface NormalizedConversation {
     messages: NormalizedMessage[];
     /** Format detected, for reporting. */
     sourceFormat: "openai" | "anthropic" | "array" | "text";
+    /** The request's own `model` field, when it has one. Picks the tokenizer ratios. */
+    model?: string;
     /**
      * Set when the input could not be read as a conversation. Silently profiling
      * a broken file as one big "user message" produces a confident, wrong report

package/dist/parse.js

@@ -146,5 +146,6 @@ export function parseConversation(input) {
         : messages.length === 0
             ? "This JSON has no `messages` array (and no `system`) — it does not look like a conversation. Expected {\"messages\":[{\"role\":…,\"content\":…}]}."
             : undefined;
-    return { sourceFormat: isAnthropic ? "anthropic" : "openai", parseWarning, messages };
+    const model = typeof obj.model === "string" && obj.model ? obj.model : undefined;
+    return { sourceFormat: isAnthropic ? "anthropic" : "openai", parseWarning, messages, model };
 }

package/dist/preferences.d.ts

@@ -4,7 +4,7 @@
  * "custom instructions" are read on every turn of every chat, which is the
  * closest thing those surfaces have to a UserPromptSubmit hook.
  *
- * Kept short (~90 tokens): it rides in every conversation.
+ * Kept short (~180 Claude tokens, ~120 GPT): it rides in every conversation.
  */
 export declare const CHAT_PREFERENCES: string;
 /** Where to paste, per app. Paths are the UI labels, not URLs, so they survive redesigns. */

package/dist/preferences.js

@@ -4,7 +4,7 @@
  * "custom instructions" are read on every turn of every chat, which is the
  * closest thing those surfaces have to a UserPromptSubmit hook.
  *
- * Kept short (~90 tokens): it rides in every conversation.
+ * Kept short (~180 Claude tokens, ~120 GPT): it rides in every conversation.
  */
 import { spawnSync } from "node:child_process";
 export const CHAT_PREFERENCES = [

package/dist/profile.js

@@ -146,11 +146,18 @@ function filesReadBy(toolName, toolCallText) {
     return [...paths];
 }
 export function profileConversation(conv, model) {
+    // An explicit model wins; otherwise the request's own model field decides
+    // which tokenizer ratios apply.
+    model = model ?? conv.model;
+    // With no model at all, Anthropic's request format still says whose
+    // tokenizer counts it. Only the ratios use this; pricing and the window
+    // stay unknown rather than guessed.
+    const tokenizerModel = model ?? (conv.sourceFormat === "anthropic" ? "claude" : undefined);
     // Learned from the user's own exact counts, if they ever fetched any.
-    const calibration = calibrationFor(model);
+    const calibration = calibrationFor(tokenizerModel);
     const perMessage = conv.messages.map((m) => ({
         msg: m,
-        tokens: Math.round(estimateTokens(m.text) * calibration.factor) + MESSAGE_OVERHEAD_TOKENS,
+        tokens: Math.round(estimateTokens(m.text, tokenizerModel) * calibration.factor) + MESSAGE_OVERHEAD_TOKENS,
     }));
     const totalTokens = perMessage.reduce((sum, p) => sum + p.tokens, 0);
     const categories = {

package/dist/proxy.js

@@ -14,7 +14,7 @@
 import http from "node:http";
 import { timingSafeEqual } from "node:crypto";
 import { optimizeConversation } from "./optimize.js";
-import { formatTokens } from "./tokens.js";
+import { formatTokens, CHARS_PER_TOKEN, providerFor } from "./tokens.js";
 import { formatUsd, inputCostUsd, pricingFor } from "./pricing.js";
 import { recordLedger } from "./ledger.js";
 /** Connection-level headers that must not be forwarded. */
@@ -155,7 +155,9 @@ export function startProxy(opts = {}) {
                         if (url.startsWith("/v1/messages") && requestModel) {
                             const stablePrefix = JSON.stringify(parsedBody.tools ?? null) + JSON.stringify(parsedBody.system ?? null);
                             const hasBreakpoint = body.includes("cache_control");
-                            if (stablePrefix.length > 4000 && !hasBreakpoint) {
+                            const stablePrefixTokens = Math.round(stablePrefix.length / CHARS_PER_TOKEN[providerFor(requestModel)].code);
+                            // Anthropic will not cache a prefix under ~1024 tokens; below that the advice is useless.
+                            if (stablePrefixTokens >= 1024 && !hasBreakpoint) {
                                 // Say WHERE, not just that. A breakpoint caches everything up
                                 // to and including the block it sits on, so it belongs on the
                                 // LAST stable block: the final tool definition if there are
@@ -163,7 +165,7 @@ export function startProxy(opts = {}) {
                                 const where = Array.isArray(parsedBody.tools) && parsedBody.tools.length > 0
                                     ? `the last entry in "tools" (tools come before system in the cached prefix)`
                                     : `the last block of "system"`;
-                                advise(`~${Math.round(stablePrefix.length / 4)}+ tokens of stable system/tools on ${requestModel} without cache_control. ` +
+                                advise(`~${stablePrefixTokens}+ tokens of stable system/tools on ${requestModel} without cache_control. ` +
                                     `Add {"cache_control":{"type":"ephemeral"}} to ${where}; everything before it then bills at ~10% on every call`);
                             }
                             const fp = fnv1a(stablePrefix);
@@ -188,7 +190,7 @@ export function startProxy(opts = {}) {
                                     stable++;
                                 if (stable >= 2) {
                                     const stableChars = msgs.slice(0, stable).reduce((n, m) => n + JSON.stringify(m).length, 0);
-                                    const stableTokens = Math.round(stableChars / 4);
+                                    const stableTokens = Math.round(stableChars / CHARS_PER_TOKEN[providerFor(requestModel)].code);
                                     // Anthropic will not cache a prefix under ~1024 tokens (2048 on Haiku).
                                     if (stableTokens >= 1024) {
                                         advise(`messages #0-#${stable - 1} (~${stableTokens} tokens) were identical to the previous ${requestModel} request and carry no cache_control. ` +

package/dist/sketch.d.ts

@@ -6,8 +6,10 @@
  * JSON means re-typing 50k+ tokens as a tool argument. No model does that, and
  * it would double the context it is meant to measure. A sketch is ~100 output
  * tokens: turn count plus the handful of blocks that matter (pastes, tool
- * results, images, repeats). The estimate is coarse (±30%) and says so, but it
- * turns "call profile_context" from an impossible instruction into a cheap one.
+ * results, images, repeats). Measured error: -20% to +9% on the conversation
+ * total from turn count alone, ±25% on a code block sized by lines, ±15% on
+ * one sized by chars. Coarse, stated, and enough to find what to drop; it turns
+ * "call profile_context" from an impossible instruction into a cheap one.
  */
 export type SketchKind = "paste" | "code" | "tool_result" | "image" | "base64" | "text";
 export interface SketchBlock {
@@ -52,7 +54,9 @@ export interface SketchProfile {
     perTurnUsd?: number;
     perTurnCachedUsd?: number;
 }
-export declare function blockTokens(b: SketchBlock): number;
+export declare function blockTokens(b: SketchBlock, model?: string): number;
+/** Tokens for one plain exchange under this model's tokenizer. */
+export declare function exchangeTokens(model?: string): number;
 export declare function profileSketch(sketch: ConversationSketch): SketchProfile;
 export declare function renderSketchProfile(p: SketchProfile): string;
 /** Profile a sketch, log it to the ledger like a hook check, and render. */

package/dist/sketch.js

@@ -6,51 +6,72 @@
  * JSON means re-typing 50k+ tokens as a tool argument. No model does that, and
  * it would double the context it is meant to measure. A sketch is ~100 output
  * tokens: turn count plus the handful of blocks that matter (pastes, tool
- * results, images, repeats). The estimate is coarse (±30%) and says so, but it
- * turns "call profile_context" from an impossible instruction into a cheap one.
+ * results, images, repeats). Measured error: -20% to +9% on the conversation
+ * total from turn count alone, ±25% on a code block sized by lines, ±15% on
+ * one sized by chars. Coarse, stated, and enough to find what to drop; it turns
+ * "call profile_context" from an impossible instruction into a cheap one.
  */
-import { contextWindowFor } from "./tokens.js";
-import { formatTokens } from "./tokens.js";
+import { CHARS_PER_TOKEN, contextWindowFor, formatTokens, providerFor } from "./tokens.js";
 import { formatUsd, inputCostUsd, pricingFor } from "./pricing.js";
 import { recordLedger } from "./ledger.js";
-// A plain chat turn without attachments: a short user message and a normal
-// assistant reply. Measured across Claude Code transcripts the median user turn
-// is ~120 tokens and the median assistant turn ~450; chat apps run similar.
-const BASELINE_TOKENS_PER_TURN = 570;
-// Tokens per unit when the model reports size in lines/words/chars. Code and
-// tool output are denser per line than prose; words are ~1.35 tokens each.
-const TOKENS_PER_LINE = {
-    code: 12, tool_result: 12, paste: 14, text: 14, base64: 40, image: 0,
+// Every size here is in CHARACTERS, measured on real data, and converted to
+// tokens with the model's own ratio (tokens.ts), so a Claude chat and a GPT
+// chat of the same text get different, correct counts.
+//
+// A plain exchange: the user's message plus the assistant's reply. Median over
+// 1,283 exchanges in 59 Claude Code sessions (p25 1,177, p75 3,008). Sizing a
+// 30+ exchange chat from its turn count alone landed within -20% to +9% of the
+// real visible total (p10 to p90, 12 sessions): individual turns vary a lot,
+// long chats average it out.
+const CHARS_PER_EXCHANGE = 2060;
+// Mean chars per non-empty line. Code: median over 719 source files (a line
+// count for code is within about ±25%). Logs and tool output: median over
+// 2,095 tool results, but they range 38 to 100 chars a line, so the schema
+// asks for chars or tokens on those, and lines are the fallback.
+const CHARS_PER_LINE = {
+    code: 42, tool_result: 56, paste: 56, text: 80, base64: 76, image: 0,
 };
-const TOKENS_PER_WORD = 1.35;
-const CHARS_PER_TOKEN = 4;
-// When a block carries no size at all. Images are billed at a near-fixed rate.
-const DEFAULT_TOKENS = {
-    image: 1500, paste: 800, code: 800, tool_result: 800, base64: 4000, text: 300,
+// Prose, including the space after each word (6.3 measured on assistant text).
+const CHARS_PER_WORD = 6.3;
+// When a block carries no size at all.
+const DEFAULT_CHARS = {
+    paste: 2400, code: 2400, tool_result: 2400, text: 900, base64: 12000, image: 0,
 };
+// Images are billed by pixels, not bytes: ~1,600 tokens for a full-size
+// screenshot on Claude, fewer on smaller images and on GPT. One figure is enough
+// for a sketch.
+const IMAGE_TOKENS = 1500;
 const LARGE_BLOCK_TOKENS = 2000;
 const LONG_HISTORY_TURNS = 30;
 const HANDOFF_SUMMARY_TOKENS = 300;
 const RECENT_TURNS_KEPT = 6;
-export function blockTokens(b) {
+/** Code, tool output and base64 tokenize at the code ratio; pastes and text at the prose ratio. */
+function ratioFor(kind, model) {
+    const r = CHARS_PER_TOKEN[providerFor(model)];
+    return kind === "code" || kind === "tool_result" || kind === "base64" ? r.code : r.prose;
+}
+export function blockTokens(b, model) {
     if (b.kind === "image")
-        return b.approx_tokens ?? DEFAULT_TOKENS.image;
+        return b.approx_tokens && b.approx_tokens > 0 ? Math.round(b.approx_tokens) : IMAGE_TOKENS;
     if (b.approx_tokens && b.approx_tokens > 0)
         return Math.round(b.approx_tokens);
-    if (b.approx_lines && b.approx_lines > 0)
-        return Math.round(b.approx_lines * TOKENS_PER_LINE[b.kind]);
-    if (b.approx_words && b.approx_words > 0)
-        return Math.round(b.approx_words * TOKENS_PER_WORD);
-    if (b.approx_chars && b.approx_chars > 0)
-        return Math.round(b.approx_chars / CHARS_PER_TOKEN);
-    return DEFAULT_TOKENS[b.kind];
+    const chars = b.approx_chars && b.approx_chars > 0 ? b.approx_chars
+        : b.approx_lines && b.approx_lines > 0 ? b.approx_lines * CHARS_PER_LINE[b.kind]
+            : b.approx_words && b.approx_words > 0 ? b.approx_words * CHARS_PER_WORD
+                : DEFAULT_CHARS[b.kind];
+    return Math.round(chars / ratioFor(b.kind, model));
+}
+/** Tokens for one plain exchange under this model's tokenizer. */
+export function exchangeTokens(model) {
+    return Math.round(CHARS_PER_EXCHANGE / CHARS_PER_TOKEN[providerFor(model)].prose);
 }
 export function profileSketch(sketch) {
     const turns = Math.max(0, Math.floor(sketch.turns || 0));
     const blocks = Array.isArray(sketch.blocks) ? sketch.blocks : [];
-    const baselineTokens = turns * BASELINE_TOKENS_PER_TURN;
+    const perExchange = exchangeTokens(sketch.model);
+    const baselineTokens = turns * perExchange;
     // A repeated block costs its size every time it appears.
-    const sized = blocks.map((b) => ({ block: b, tokens: blockTokens(b), copies: Math.max(1, Math.floor(b.repeated ?? 1)) }));
+    const sized = blocks.map((b) => ({ block: b, tokens: blockTokens(b, sketch.model), copies: Math.max(1, Math.floor(b.repeated ?? 1)) }));
     const blockTotal = sized.reduce((n, s) => n + s.tokens * s.copies, 0);
     const totalTokens = baselineTokens + blockTotal;
     const findings = [];
@@ -101,7 +122,7 @@ export function profileSketch(sketch) {
         });
     }
     if (turns >= LONG_HISTORY_TURNS) {
-        const afterHandoff = RECENT_TURNS_KEPT * BASELINE_TOKENS_PER_TURN + HANDOFF_SUMMARY_TOKENS;
+        const afterHandoff = RECENT_TURNS_KEPT * perExchange + HANDOFF_SUMMARY_TOKENS;
         findings.push({
             id: "long_history",
             severity: totalTokens > 100_000 ? "high" : "warn",
@@ -140,7 +161,7 @@ export function renderSketchProfile(p) {
     const lines = [];
     const window = p.usagePct !== undefined ? ` (~${p.usagePct}% of ${formatTokens(p.contextWindow)})` : "";
     lines.push(`Context estimate from sketch: ~${formatTokens(p.totalTokens)} tokens${window}, ${p.turns} turns.`);
-    lines.push(`  ${formatTokens(p.baselineTokens)} plain conversation + ${formatTokens(p.blockTokens)} in pastes, tool output and images. Estimate, ±30%.`);
+    lines.push(`  ${formatTokens(p.baselineTokens)} plain conversation + ${formatTokens(p.blockTokens)} in pastes, tool output and images. Estimate, usually within ±20%.`);
     if (p.perTurnUsd !== undefined) {
         lines.push(`  Re-read on every turn: ${formatUsd(p.perTurnUsd)} at list price, ${formatUsd(p.perTurnCachedUsd)} when cached. On a subscription this is what spends the usage limit.`);
     }

package/dist/tokenizer-measure.d.ts

@@ -0,0 +1,42 @@
+/**
+ * Measure a model's real chars-per-token from Claude Code transcripts, with no
+ * API key and no tokenizer: the API's own counts are already in the file.
+ *
+ * Two independent measurements, so one can check the other:
+ *
+ *  - PROSE. An assistant reply with no thinking block is billed as exactly
+ *    `output_tokens`, and all of it is visible text in the transcript. Visible
+ *    chars / output_tokens is the tokenizer's ratio on the model's own prose.
+ *
+ *  - BLOCKS (code, tool output, pastes). Between two consecutive API calls in
+ *    one session the prompt is the old prompt plus what was appended: the
+ *    previous reply (all of `output_tokens`, thinking included, since a tool
+ *    loop re-sends it) and the new user-side content. When that content is a
+ *    single large block, (growth − previous output_tokens) is its exact size.
+ *
+ * The injected reminders the harness adds make the block figure slightly
+ * pessimistic (a few dozen tokens on blocks of thousands), which is why only
+ * blocks over 6k chars count.
+ */
+export interface RatioStats {
+    samples: number;
+    median: number;
+    p10: number;
+    p90: number;
+}
+export interface ModelRatios {
+    model: string;
+    prose?: RatioStats;
+    blocks?: RatioStats;
+    /** What the estimator uses for this model: prose, code. */
+    assumed: {
+        prose: number;
+        code: number;
+    };
+}
+export interface TokenizerReport {
+    sessionsScanned: number;
+    models: ModelRatios[];
+}
+export declare function measureTokenizer(limit?: number, paths?: string[]): TokenizerReport;
+export declare function renderTokenizer(report: TokenizerReport): string;

package/dist/tokenizer-measure.js

@@ -0,0 +1,172 @@
+/**
+ * Measure a model's real chars-per-token from Claude Code transcripts, with no
+ * API key and no tokenizer: the API's own counts are already in the file.
+ *
+ * Two independent measurements, so one can check the other:
+ *
+ *  - PROSE. An assistant reply with no thinking block is billed as exactly
+ *    `output_tokens`, and all of it is visible text in the transcript. Visible
+ *    chars / output_tokens is the tokenizer's ratio on the model's own prose.
+ *
+ *  - BLOCKS (code, tool output, pastes). Between two consecutive API calls in
+ *    one session the prompt is the old prompt plus what was appended: the
+ *    previous reply (all of `output_tokens`, thinking included, since a tool
+ *    loop re-sends it) and the new user-side content. When that content is a
+ *    single large block, (growth − previous output_tokens) is its exact size.
+ *
+ * The injected reminders the harness adds make the block figure slightly
+ * pessimistic (a few dozen tokens on blocks of thousands), which is why only
+ * blocks over 6k chars count.
+ */
+import { forEachLine, listSessions } from "./session.js";
+import { CHARS_PER_TOKEN, providerFor } from "./tokens.js";
+const MIN_PROSE_CHARS = 1500;
+const MIN_BLOCK_CHARS = 6000;
+function stats(values) {
+    if (values.length === 0)
+        return undefined;
+    const v = [...values].sort((a, b) => a - b);
+    const at = (q) => v[Math.min(v.length - 1, Math.floor(q * (v.length - 1)))];
+    const mid = Math.floor(v.length / 2);
+    const median = v.length % 2 ? v[mid] : (v[mid - 1] + v[mid]) / 2;
+    return { samples: v.length, median, p10: at(0.1), p90: at(0.9) };
+}
+function promptTotal(u) {
+    const n = (k) => (typeof u[k] === "number" ? u[k] : 0);
+    return n("input_tokens") + n("cache_read_input_tokens") + n("cache_creation_input_tokens");
+}
+function userText(content) {
+    if (typeof content === "string")
+        return content;
+    if (!Array.isArray(content))
+        return "";
+    let out = "";
+    for (const p of content) {
+        if (p?.type === "text" && typeof p.text === "string")
+            out += p.text;
+        else if (p?.type === "tool_result") {
+            const c = p.content;
+            if (typeof c === "string")
+                out += c;
+            else if (Array.isArray(c))
+                for (const x of c)
+                    if (typeof x?.text === "string")
+                        out += x.text;
+        }
+    }
+    return out;
+}
+/** Scan one transcript into per-request records. Never throws on bad lines. */
+function readRequests(path) {
+    const reqs = [];
+    let cur;
+    forEachLine(path, (line) => {
+        let e;
+        try {
+            e = JSON.parse(line);
+        }
+        catch {
+            return;
+        }
+        if (!e || e.isSidechain)
+            return;
+        if (e.type === "assistant" && e.message && typeof e.message === "object") {
+            const m = e.message;
+            const u = (m.usage ?? {});
+            if (!cur || cur.id !== m.id) {
+                cur = { id: String(m.id), model: m.model, prompt: promptTotal(u), output: 0, types: new Set(), textChars: 0, after: [] };
+                reqs.push(cur);
+            }
+            if (typeof u.output_tokens === "number")
+                cur.output = Math.max(cur.output, u.output_tokens);
+            for (const p of Array.isArray(m.content) ? m.content : []) {
+                cur.types.add(String(p?.type));
+                if (p?.type === "text" && typeof p.text === "string")
+                    cur.textChars += p.text.length;
+            }
+        }
+        else if (e.type === "user" && cur) {
+            const meta = Boolean(e.isMeta || e.isCompactSummary);
+            const text = userText(e.message?.content);
+            cur.after.push({ chars: text.length, meta });
+        }
+    });
+    return reqs;
+}
+export function measureTokenizer(limit = 60, paths) {
+    const prose = new Map();
+    const blocks = new Map();
+    const push = (m, k, v) => (m.get(k) ?? m.set(k, []).get(k)).push(v);
+    let scanned = 0;
+    const targets = paths ?? listSessions(limit).map((s) => s.path);
+    for (const path of targets) {
+        let reqs;
+        try {
+            reqs = readRequests(path);
+        }
+        catch {
+            continue;
+        }
+        if (reqs.length === 0)
+            continue;
+        scanned++;
+        for (const r of reqs) {
+            if (!r.model || r.model.startsWith("<"))
+                continue;
+            if (r.types.size === 1 && r.types.has("text") && r.textChars >= MIN_PROSE_CHARS && r.output > 0) {
+                push(prose, r.model, r.textChars / r.output);
+            }
+        }
+        for (let i = 0; i + 1 < reqs.length; i++) {
+            const a = reqs[i], b = reqs[i + 1];
+            if (!a.model || a.model !== b.model)
+                continue;
+            const real = a.after.filter((x) => !x.meta);
+            if (a.after.some((x) => x.meta) || real.length !== 1 || real[0].chars < MIN_BLOCK_CHARS)
+                continue;
+            const exact = b.prompt - a.prompt - a.output;
+            if (exact <= 0)
+                continue; // a compaction or cache reset, not growth
+            push(blocks, a.model, real[0].chars / exact);
+        }
+    }
+    const models = [...new Set([...prose.keys(), ...blocks.keys()])]
+        .map((model) => ({
+        model,
+        prose: stats(prose.get(model) ?? []),
+        blocks: stats(blocks.get(model) ?? []),
+        assumed: CHARS_PER_TOKEN[providerFor(model)],
+    }))
+        .sort((x, y) => ((y.prose?.samples ?? 0) + (y.blocks?.samples ?? 0)) - ((x.prose?.samples ?? 0) + (x.blocks?.samples ?? 0)));
+    return { sessionsScanned: scanned, models };
+}
+export function renderTokenizer(report) {
+    const lines = [];
+    lines.push("Tokenizer check — chars per token, measured from the API's own counts");
+    lines.push("─".repeat(56));
+    if (report.models.length === 0) {
+        lines.push("No usable samples (needs Claude Code sessions with usage recorded).");
+        return lines.join("\n");
+    }
+    const f = (n) => n.toFixed(2);
+    const err = (assumed, real) => {
+        // Estimated tokens / real tokens − 1, from chars/token on each side.
+        const pct = Math.round((real / assumed - 1) * 100);
+        return pct === 0 ? "exact" : `estimates ${pct > 0 ? "+" : ""}${pct}%`;
+    };
+    const row = (label, st, unit, assumed) => `  ${label.padEnd(7)} ${f(st.median)}  ${`(p10 ${f(st.p10)}, p90 ${f(st.p90)}, ${st.samples} ${unit})`.padEnd(34)} ` +
+        `estimator ${f(assumed)} → ${err(assumed, st.median)}`;
+    for (const m of report.models) {
+        lines.push(m.model);
+        if (m.prose) {
+            lines.push(row("prose", m.prose, "replies", m.assumed.prose));
+        }
+        if (m.blocks) {
+            lines.push(row("blocks", m.blocks, "blocks", m.assumed.code));
+        }
+    }
+    lines.push("");
+    lines.push("\"estimates +x%\" means the estimator reports x% more tokens than the API bills for");
+    lines.push("the same text (negative: fewer). Blocks are mostly code and tool output.");
+    return lines.join("\n");
+}

package/dist/tokens.d.ts

@@ -11,7 +11,37 @@
 export type Provider = "anthropic" | "openai" | "google" | "generic";
 export declare function contextWindowFor(model?: string): number | undefined;
 export declare function providerFor(model?: string): Provider;
-export declare function estimateTokens(text: string): number;
+/**
+ * Characters per token, by provider and by content type.
+ *
+ * Anthropic: measured 2026-09 against the API's own counts in 59 Claude Code
+ * sessions (Opus 4.7 to 5, Fable 5.x), two independent ways that agree.
+ * Prose: 504 assistant replies with no thinking block, visible text divided by
+ * the exact output_tokens: median 2.75 (p10 2.4, p90 3.0). Code and tool
+ * output: 474 single appended blocks over 6k chars, sized by the exact growth
+ * of the billed prompt between consecutive calls: median 2.4 (p10 2.1, p90 2.8).
+ * The ratios this tool used before (4.0 / 3.2) undercounted current Claude
+ * models by about 1.45x on prose and 1.33x on code.
+ *
+ * OpenAI, Google and unknown models keep 4.0 / 3.2, the usual figures for
+ * o200k-class tokenizers on English and code. They are not re-measured here:
+ * Codex rollouts truncate tool output before the model sees it, so the same
+ * delta method does not isolate a block. `analyze --exact` calibrates any
+ * provider from its own tokenizer on your machine.
+ */
+export declare const CHARS_PER_TOKEN: Record<Provider, {
+    prose: number;
+    code: number;
+}>;
+/** True when text is dense with code/JSON symbols and tokenizes more finely. */
+export declare function isCodeLike(text: string): boolean;
+/** Chars per token to use for this text under this model's tokenizer. */
+export declare function charsPerTokenFor(text: string, model?: string): number;
+/**
+ * Estimated tokens for `text`. Pass the model when you know it: Claude's
+ * tokenizer produces ~40% more tokens than the provider-neutral default.
+ */
+export declare function estimateTokens(text: string, model?: string): number;
 /** Per-message structural overhead (role markers, delimiters) is roughly constant. */
 export declare const MESSAGE_OVERHEAD_TOKENS = 4;
 export declare function formatTokens(n: number): string;

package/dist/tokens.js

@@ -49,17 +49,51 @@ function symbolDensity(text) {
     const symbols = text.match(/[{}[\]()<>;:=_\/\\|"'`#$%&*+^~-]/g);
     return (symbols?.length ?? 0) / text.length;
 }
-export function estimateTokens(text) {
+/**
+ * Characters per token, by provider and by content type.
+ *
+ * Anthropic: measured 2026-09 against the API's own counts in 59 Claude Code
+ * sessions (Opus 4.7 to 5, Fable 5.x), two independent ways that agree.
+ * Prose: 504 assistant replies with no thinking block, visible text divided by
+ * the exact output_tokens: median 2.75 (p10 2.4, p90 3.0). Code and tool
+ * output: 474 single appended blocks over 6k chars, sized by the exact growth
+ * of the billed prompt between consecutive calls: median 2.4 (p10 2.1, p90 2.8).
+ * The ratios this tool used before (4.0 / 3.2) undercounted current Claude
+ * models by about 1.45x on prose and 1.33x on code.
+ *
+ * OpenAI, Google and unknown models keep 4.0 / 3.2, the usual figures for
+ * o200k-class tokenizers on English and code. They are not re-measured here:
+ * Codex rollouts truncate tool output before the model sees it, so the same
+ * delta method does not isolate a block. `analyze --exact` calibrates any
+ * provider from its own tokenizer on your machine.
+ */
+export const CHARS_PER_TOKEN = {
+    anthropic: { prose: 2.75, code: 2.4 },
+    openai: { prose: 4.0, code: 3.2 },
+    google: { prose: 4.0, code: 3.2 },
+    generic: { prose: 4.0, code: 3.2 },
+};
+/** True when text is dense with code/JSON symbols and tokenizes more finely. */
+export function isCodeLike(text) {
+    return symbolDensity(text) > 0.08;
+}
+/** Chars per token to use for this text under this model's tokenizer. */
+export function charsPerTokenFor(text, model) {
+    const ratios = CHARS_PER_TOKEN[providerFor(model)];
+    return isCodeLike(text) ? ratios.code : ratios.prose;
+}
+/**
+ * Estimated tokens for `text`. Pass the model when you know it: Claude's
+ * tokenizer produces ~40% more tokens than the provider-neutral default.
+ */
+export function estimateTokens(text, model) {
     // Public API: callers outside this package pass whatever they have, and a
     // TypeError from a token estimator is never the useful answer.
     if (typeof text !== "string")
         text = String(text ?? "");
     if (!text)
         return 0;
-    // Denser tokenization for code/JSON-like content, lighter for plain prose.
-    const density = symbolDensity(text);
-    const charsPerToken = density > 0.08 ? 3.2 : 4.0;
-    return Math.ceil(text.length / charsPerToken);
+    return Math.ceil(text.length / charsPerTokenFor(text, model));
 }
 /** Per-message structural overhead (role markers, delimiters) is roughly constant. */
 export const MESSAGE_OVERHEAD_TOKENS = 4;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "context-doctor",
-  "version": "0.18.0",
+  "version": "0.19.0",
   "description": "Profile and optimize LLM context windows. See what's eating your tokens and fix it — works with Claude Code, Claude Desktop, Cursor, Codex (OpenAI), and any MCP-capable AI app.",
   "keywords": [
     "claude",