context-doctor 0.17.0 → 0.19.0

package/README.md

@@ -90,14 +90,14 @@ Practical upshot: a developer who only wants cheaper, faster API calls never tou
 | Command | What it does |
 |---|---|
 | `context-doctor install` / `uninstall` | Wire (or remove) everything: MCP for Claude Desktop/Code/Cursor/Codex, the Agent Skill, the every-prompt hook |
-| `context-doctor instructions [--copy]` | The ~90-token standing rules for claude.ai / ChatGPT preferences, for web and phones where no server runs |
+| `context-doctor instructions [--copy]` | The ~180-token standing rules (~120 on GPT) for claude.ai / ChatGPT preferences, for web and phones where no server runs |
 | `context-doctor analyze <file>` | Profile a conversation: token breakdown, findings, cost + latency estimates. `--fail-over-budget` exits 1 on a breach, for CI |
 | `context-doctor optimize <file>` | Apply the safe fixes; add `--strategy trim-tool-calls` for big inline file writes, `--strategy prune-history` for consented lossy compaction |
 | `context-doctor session [file]` | Profile a Claude Code session: live context, findings, **measured tokens and prompt-cache economics**, **where the wall clock went** per tool, and **what its subagents cost** (their own windows, your bill; never in the parent's profile). Also reads ChatGPT data exports (`conversations.json`) |
 | `context-doctor init [preset]` | Write a `.contextdoctorrc` from a preset (`chat`, `agent`, `batch`) — a budget you can adopt in one command and tune later |
 | `context-doctor experiment --task "…"` | Run one task twice from the same commit, in a fresh session and forked from an `--existing` one, same model and tools; compare bill, cache split, wall clock, and whether `--check` passed. The only command here that spends money, so it caps spend per arm and refuses a dirty tree |
 | `context-doctor diff <before> <after>` | Compare two profiles: what moved by category, which findings were resolved or introduced, and what it saves in money and latency |
-| `context-doctor accuracy` | How much of what you are billed for is visible in your transcript — the fixed harness baseline and the per-turn injected content neither you nor the profiler can see |
+| `context-doctor accuracy` | How much of what you are billed for is visible in your transcript (the fixed harness baseline, per-turn injected content), plus a tokenizer check: real chars/token per model from the API's own counts, next to the ratio the estimator uses |
 | `context-doctor cursor [--list]` | Profile a chat from Cursor's local history (both storage formats) |
 | `context-doctor report` | Machine-wide impact report (proxy savings persist across restarts): exact proxy savings, hook activity, recoverable waste in recent sessions |
 | `context-doctor proxy` | Always-on local proxy that optimizes every Anthropic/OpenAI API request in flight (`/stats` for cumulative savings) |
@@ -179,7 +179,20 @@ The proxy dedupes repeated content, trims stale tool results, and strips base64
 
 Because prompt caching matches byte-identical prefixes, deterministic strategies are chosen so repeated requests stay stable — but if you rely on aggressive cache prefixes, start with `--strategy strip-base64 --strategy dedupe` and add more as you verify.
 
-> **Note on desktop chat apps:** Claude Desktop and the ChatGPT app talk to their own backends — no tool can sit in that path. For those, use the MCP integration below and add a line to your custom instructions like: *"When a conversation gets long or includes large pasted content, proactively use context-doctor's profile_context tool and tell me what to trim."* The model will then invoke it on its own.
+### Putting the proxy on a public URL (Cursor with your own OpenAI key, remote apps)
+
+Some apps let you set a base URL but call it from *their* servers, not your machine. Cursor is one: with your own OpenAI key, "Override OpenAI Base URL" is sent to Cursor's backend inside the model configuration, and Cursor's servers make the request (only the key-verification ping is client-side; we checked the app bundle, 3.18.25). So `127.0.0.1` cannot work there; the proxy has to be reachable from the internet, and an open relay on the internet is a bad idea. Hence the token:
+
+```bash
+npx context-doctor proxy --token "$(openssl rand -hex 16)"   # or CONTEXT_DOCTOR_PROXY_TOKEN=...
+ngrok http 8787                                               # or any HTTPS tunnel / reverse proxy
+```
+
+With `--token`, every path except `/health` must start with `/t/<token>/`; anything else gets 401 before any upstream call, and the comparison is constant time. Then in Cursor: Settings > Models > OpenAI API Key > Override OpenAI Base URL = `https://<your-host>/t/<token>/v1`. Every agent request Cursor makes with your key now passes through the proxy: deduped, stale tool results trimmed, base64 stripped, real usage counted in `/t/<token>/stats`. This is the one Cursor path that is model-independent and needs no hook. It applies only to BYO-key traffic; Cursor's own subscription models never leave Cursor's servers.
+
+Your API key still rides in the request headers, as before. The token protects the relay, not the key; keep the tunnel HTTPS.
+
+> **Note on desktop chat apps:** Claude Desktop and the ChatGPT app talk to their own backends — no tool can sit in that path. For those, use the MCP integration below (Claude Desktop gets the standing rules and a cheap `profile_context` sketch call) and `context-doctor instructions --copy` for the per-account preferences.
 
 ## Use with the Claude & ChatGPT apps
 
@@ -190,13 +203,13 @@ Because prompt caching matches byte-identical prefixes, deterministic strategies
 | **Claude Code** | Yes: hook on every prompt, status line on every refresh | Past ~80k tokens the model receives hygiene guidance naming the largest waste; compaction is offered. Measured: 115 automatic checks, 48 warnings, across 32 sessions on one machine |
 | **Cursor** | **Yes**, since 0.15: Cursor loads Claude Code's hook config (`~/.claude/settings.json`) and runs the same hook on every agent prompt, passing its own transcript. Output is accepted through Cursor's Claude-compat layer | Same guidance as Claude Code, inside Cursor's agent, for everyone who ran `install`. Before 0.15 the hook fired but could not read Cursor's transcript format, so it said nothing |
 | **API traffic through the proxy** | Yes: every request rewritten in flight | Fewer tokens, guaranteed, model not consulted |
-| **Claude Desktop** | The standing instruction in every chat (we confirmed in the app bundle that Desktop's `LocalMcpServerManager` reads it), a one-click `context_checkup` prompt, and since 0.17 a `profile_context` the model can actually afford to call from chat | Until 0.17 the tool wanted the whole conversation as its argument, so calling it from chat meant re-typing 50k tokens; nobody did, and Desktop's log showed zero calls in a month. Now the model passes a ~100-token **sketch** (turn count, the large or repeated blocks) and gets a sized estimate, findings and the fix to apply. Still a nudge, not a hook: Desktop chat has no hook API and no transcript on disk |
+| **Claude Desktop** | The standing instruction in every chat (we confirmed in the app bundle that Desktop's `LocalMcpServerManager` reads it), a one-click `context_checkup` prompt, and since 0.17 a `profile_context` the model can actually afford to call from chat | Until 0.17 the tool wanted the whole conversation as its argument, so calling it from chat meant re-typing 50k tokens; nobody did, and Desktop's log showed zero calls in a month. Now the model passes a ~120-token **sketch** (turn count, the large or repeated blocks) and gets a sized estimate, findings and the fix to apply. Still a nudge, not a hook: Desktop chat has no hook API and no transcript on disk |
 | **Codex (OpenAI): ChatGPT.app's Codex tab, the Codex IDE extension, the `codex` CLI** | **Yes**, since 0.16: `install` writes the hook to `~/.codex/hooks.json`, the MCP server to `~/.codex/config.toml`, and the skill to `~/.codex/skills/`. Codex uses Claude Code's hook contract almost verbatim and passes its own rollout transcript, which carries the API's real usage figures | Same guidance as Claude Code, from measured tokens. One extra step, Codex's rule not ours: a new hook runs only after you trust it once (type `/hooks` in Codex). `session` and `session --list` read Codex rollouts too |
 | **ChatGPT chat UI** | No | No MCP, no hooks, no data path in the chat product itself. Use Codex, or a developer-mode connector at a URL you host |
 
 So "every chat inherently better" is true for Claude Code, Cursor, Codex and the proxy; for Claude Desktop it is "the rules ride in every chat and the checkup is one cheap tool call away"; and not a claim we make for the ChatGPT chat UI.
 
-**Where there is no hook and no MCP at all** (claude.ai on the web, the Claude and ChatGPT phone apps, plain ChatGPT): the app's per-account preferences are read on every turn, which is the closest those surfaces have to a hook. `context-doctor instructions --copy` puts the ~90-token rules on your clipboard and tells you where to paste them (claude.ai Settings > Profile; ChatGPT Settings > Personalization > Custom instructions).
+**Where there is no hook and no MCP at all** (claude.ai on the web, the Claude and ChatGPT phone apps, plain ChatGPT): the app's per-account preferences are read on every turn, which is the closest those surfaces have to a hook. `context-doctor instructions --copy` puts the ~180-token rules on your clipboard and tells you where to paste them (claude.ai Settings > Profile; ChatGPT Settings > Personalization > Custom instructions).
 
 **Do you need to configure anything by hand? Usually no:**
 
@@ -225,7 +238,7 @@ For any other MCP client, the server entry is:
 
 1. Run `npx context-doctor install` (writes the config above for you) and restart Claude Desktop. Or open the `.mcpb` from the latest release: same server, no npm, installs as an Extension.
 2. From then on, **every conversation carries context-doctor's standing instructions**. The MCP server hands them to Desktop on connect and Desktop puts them in front of Claude: summarize big pastes instead of re-quoting them, refer to earlier content by name, never inline base64, and past ~30 turns or on any question about tokens, cost, speed or limits, call `profile_context` before answering.
-3. That call is cheap on purpose. Claude cannot export a Desktop chat, so it passes a **sketch**: how many turns, which blocks are large, repeated, stale or images, with one size hint each (~100 tokens). The server sizes it (±30%, and it says so), prices the per-turn re-read (on a subscription that is what spends your usage limit), and returns ranked findings with the action for each: "summarize *the nginx config* into the points still needed", "refer to *test output* by name", "offer a 300-token handoff summary for a fresh chat". The reply ends with an instruction to apply the top one, not just suggest it.
+3. That call is cheap on purpose. Claude cannot export a Desktop chat, so it passes a **sketch**: how many turns, which blocks are large, repeated, stale or images, with one size hint each (~120 tokens). The server sizes it (usually within ±20%, measured; see "Why token counts are ~"), prices the per-turn re-read (on a subscription that is what spends your usage limit), and returns ranked findings with the action for each: "summarize *the nginx config* into the points still needed", "refer to *test output* by name", "offer a 300-token handoff summary for a fresh chat". The reply ends with an instruction to apply the top one, not just suggest it.
 4. One click instead of asking: the `context_checkup` prompt in the **+** menu sends that request for you.
 5. Say *"optimize it"* on an exported conversation and Claude applies the safe fixes; if you agree to pruning old history, **Claude itself writes the replacement summary** (that's the no-API-key summarization).
 6. For the same rules on your phone and on claude.ai, where no MCP server runs: `context-doctor instructions --copy`, then paste into Settings > Profile > personal preferences.
@@ -346,9 +359,9 @@ The verdict line is the point: cheaper only counts if it also passed. Because th
 
 ## Exact counts, and what they teach the estimator
 
-The default token count is a chars-per-token heuristic so everything runs with no key and no tokenizer. Its error is content-dependent, and there is no honest way to fix that from transcripts alone (the billed number includes content the transcript never sees). `analyze --exact` fetches a true count for the exact bytes just estimated (Anthropic's count-tokens API with `ANTHROPIC_API_KEY`; tiktoken for GPT if installed) and prints the drift.
+The default token count is a chars-per-token heuristic so everything runs with no key and no tokenizer, with ratios per provider (see "Why token counts are ~" below). `analyze --exact` fetches a true count for the exact bytes just estimated (Anthropic's count-tokens API with `ANTHROPIC_API_KEY`; tiktoken for GPT if installed) and prints the drift.
 
-Since 0.13.9 it also **remembers the comparison**, per model family, on this machine, and later estimates for that family are scaled by it. Nothing about this is silent: the profile header says `estimates calibrated +12% from 3 exact count(s) you ran on this machine`. No exact count ever run means no calibration and unchanged numbers; out-of-range samples are ignored; `CONTEXT_DOCTOR_NO_CALIBRATION=1` returns to the raw heuristic.
+Since 0.13.9 it also **remembers the comparison**, per model family, on this machine, and later estimates for that family are scaled by it. Nothing about this is silent: the profile header says `estimates calibrated +12% from 3 exact count(s) you ran on this machine`. No exact count ever run means no calibration and unchanged numbers; out-of-range samples are ignored; `CONTEXT_DOCTOR_NO_CALIBRATION=1` returns to the raw heuristic. Samples are tied to the heuristic they were taken against: after 0.19 changed Claude's ratios, older samples are ignored and learning restarts, rather than stacking an old correction on a fixed estimator.
 
 ## What it detects
 
@@ -453,7 +466,7 @@ A tool that promises speed must be near-free. Measured overhead per touchpoint:
 | Touchpoint | When it runs | Overhead |
 |---|---|---|
 | Every-prompt hook (Claude Code) | Every prompt | **~80ms** (Node startup; logic ~1ms). Lean sessions exit on a single `stat()` — the transcript is never read. Full profiling (~200ms on a 4MB session) happens only when the transcript has grown ~40% since last checked |
-| MCP server | Spawned once per app session | Tools run only when called; standing instructions cost **~110 tokens per conversation** — deliberately terse |
+| MCP server | Spawned once per app session | Tools run only when called; standing instructions cost **~250 tokens per conversation on Claude, ~170 on GPT** — deliberately terse |
 | Proxy | Per API request | ~1–3ms of CPU (parse → optimize → re-serialize) against typical model latencies of hundreds of ms; responses stream through chunk-by-chunk, never buffered |
 | Skill | Loads only when relevant | ~1k tokens while active; its always-present description is ~60 tokens |
 | Profiling a session | On demand, and on hook growth events | ~160ms for an 8.5MB / 1,855-message transcript (near-duplicate pairs that cannot clear the similarity bar are skipped without comparison) |
@@ -463,18 +476,25 @@ Net effect is strongly negative overhead: the tokens these touchpoints save on e
 
 ## Why token counts are "~" (and where they are exact)
 
-Counting exactly needs each provider's tokenizer, so the default is a calibrated chars-per-token heuristic (denser for code and JSON). It is good enough to rank what is heavy and to measure the effect of a fix, and it keeps the tool offline and zero-config.
+Counting exactly needs each provider's tokenizer, so the default is a chars-per-token heuristic, with ratios per provider and denser ones for code and JSON. It keeps the tool offline and zero-config.
 
-Two ways to get real numbers instead:
+| Model | Prose | Code / tool output | Source |
+|---|---|---|---|
+| Claude (Opus 4.7 to 5.x, Fable 5.x, Sonnet 5) | 2.75 chars/token | 2.4 | Measured from the API's own counts, below |
+| GPT, Gemini, unknown | 4.0 | 3.2 | Usual figures for o200k-class tokenizers; not re-measured here |
 
-- **`analyze --exact`** uses the Anthropic count-tokens API for Claude models (set `ANTHROPIC_API_KEY`; opt-in network call, key never stored) or tiktoken for GPT models (install it alongside), and reports how far the heuristic drifted.
-- **Sessions report measured tokens automatically.** Claude Code transcripts record what the API actually charged, so `session`, the hook and the reports use that figure when it is present — no key, no estimate.
+Which row applies: the model you pass, else the request's own `model` field, else the request's shape (Anthropic's `system` field or `tool_use` blocks mean Claude). Cursor transcripts record no model and use Anthropic-style blocks, so Cursor sessions are counted at Claude density; for a GPT model in Cursor that reads about 40% high.
 
-One honest caveat worth knowing: a transcript stores the conversation, **not** the harness's system prompt, tool schemas or skills. Measured against the API's own numbers here, a message-only estimate undercounts the true context by roughly 60%. That is why sessions prefer the reported figure, and why the message breakdown is labelled as covering messages only.
+**How the Claude figures were measured, with no key.** Claude Code transcripts record what the API billed, and two things in them are exact. A reply with no thinking block is billed as exactly its `output_tokens`, and all of it is visible text: 504 replies gave a median of 2.75 chars/token (p10 2.4, p90 3.0). Between two consecutive API calls the prompt grows by exactly what was appended; when that is one large block, its size is the growth minus the previous reply: 474 blocks of code and tool output gave 2.4 (p10 2.1, p90 2.8). The ratios this tool used until 0.19 (4.0 / 3.2 for everything) **undercounted current Claude models by about 40%**: hook warnings came late, savings and costs read low, and the proxy stayed silent on cacheable prefixes between 1,024 and ~1,670 tokens. `context-doctor accuracy` re-runs both measurements on your own sessions and prints them per model beside the ratio in use, so the next tokenizer change shows up as a number, not a surprise. On this machine every model lands within ±9%.
 
+Two ways to get real numbers instead of estimates:
+
+- **`analyze --exact`** uses the Anthropic count-tokens API for Claude models (set `ANTHROPIC_API_KEY`; opt-in network call, key never stored) or tiktoken for GPT models (install it alongside), and reports how far the heuristic drifted.
+- **Sessions report measured tokens automatically.** Claude Code transcripts record what the API actually charged, so `session`, the hook and the reports use that figure when it is present — no key, no estimate.
 
+One honest caveat worth knowing: a transcript stores the conversation, **not** the harness's system prompt, tool schemas or skills (about 54k tokens before the first turn in Claude Code here), nor the reminders it injects each turn. With the corrected ratios the transcript accounts for a median 56% of each turn's billed growth; before 0.19 this read 39%, and about a third of that "invisible" gap was the estimator. That is why sessions prefer the reported figure, and why the message breakdown is labelled as covering messages only.
 
-Exact counts require each provider's private tokenizer. `context-doctor` uses a calibrated chars-per-token heuristic (denser for code/JSON) that lands within ~10% — plenty accurate for finding what's heavy and measuring savings, and it keeps the tool fully offline with zero configuration.
+**The chat-app sketch** (Claude Desktop, see above) is coarser by design, because the model describes the chat instead of sending it. Its sizes are measured, not assumed: a plain exchange is 2,060 chars (median of 1,283), a code line 42 chars (719 source files), a log line 56 (2,095 tool outputs), a word 6.3, all converted with the model's own ratio. Measured error: the total for a chat of 30+ exchanges from its turn count alone is within -20% to +9% (p10 to p90); a code block sized by lines is within about ±25%, by chars about ±15%. Logs vary from 38 to 100 chars a line, so the tool asks for their size in chars.
 
 ## Roadmap
 

package/dist/accuracy.d.ts

@@ -11,10 +11,12 @@
  * measures the distance between them on your own sessions, so "why is my bill
  * bigger than the profile?" has an answer with evidence behind it.
  *
- * WHAT THIS IS NOT: a tokenizer benchmark. It cannot be — the content behind
- * the gap is unavailable to us, so the gap cannot be attributed to estimator
- * drift. To measure the estimator itself, use `analyze --exact`, which counts
- * the same bytes with the provider's own tokenizer.
+ * The coverage figure uses the model-aware estimator. Estimator drift itself is
+ * measured separately (tokenizer-measure.ts) on content the transcript DOES
+ * hold: replies whose output_tokens are exact, and single large blocks whose
+ * size is the exact prompt growth. Until 0.19 this command attributed the whole
+ * gap to invisible content; about a third of it was the estimator undercounting
+ * Claude's tokenizer.
  */
 export interface AccuracyReport {
     sessionsScanned: number;

package/dist/accuracy.js

@@ -11,10 +11,12 @@
  * measures the distance between them on your own sessions, so "why is my bill
  * bigger than the profile?" has an answer with evidence behind it.
  *
- * WHAT THIS IS NOT: a tokenizer benchmark. It cannot be — the content behind
- * the gap is unavailable to us, so the gap cannot be attributed to estimator
- * drift. To measure the estimator itself, use `analyze --exact`, which counts
- * the same bytes with the provider's own tokenizer.
+ * The coverage figure uses the model-aware estimator. Estimator drift itself is
+ * measured separately (tokenizer-measure.ts) on content the transcript DOES
+ * hold: replies whose output_tokens are exact, and single large blocks whose
+ * size is the exact prompt growth. Until 0.19 this command attributed the whole
+ * gap to invisible content; about a third of it was the estimator undercounting
+ * Claude's tokenizer.
  */
 import { estimateTokens, formatTokens, MESSAGE_OVERHEAD_TOKENS } from "./tokens.js";
 import { parseConversation } from "./parse.js";
@@ -57,7 +59,7 @@ export function measureAccuracy(limit = 20, paths) {
         const normalized = parseConversation(parsed.conversationJson).messages;
         const estimateAt = (i) => {
             const m = normalized[i];
-            return m ? estimateTokens(m.text) + MESSAGE_OVERHEAD_TOKENS : 0;
+            return m ? estimateTokens(m.text, parsed.model) + MESSAGE_OVERHEAD_TOKENS : 0;
         };
         let baseline = usage[0].input;
         for (let i = 0; i < usage[0].index; i++)
@@ -118,7 +120,7 @@ export function renderAccuracy(report) {
     lines.push("still saves real money — it just starts from a higher floor than the profile");
     lines.push("implies.");
     lines.push("");
-    lines.push("This does NOT measure tokenizer drift: the missing content is not available");
-    lines.push("to compare against. For that, use `analyze --exact` (provider tokenizer).");
+    lines.push("The tokenizer check below measures the estimator itself, on content the");
+    lines.push("transcript does hold; `analyze --exact` checks it against the provider's tokenizer.");
     return lines.join("\n");
 }

package/dist/calibration.d.ts

@@ -14,6 +14,14 @@
  * them. No exact count ever run means no calibration, and the numbers are
  * exactly what they were before.
  */
+/**
+ * Bumped whenever the uncalibrated heuristic changes. A factor learned against
+ * an older heuristic would correct for an error that no longer exists (0.19
+ * moved Claude from 4.0 to 2.75 chars/token; an old 1.4x factor on top of that
+ * would overcount by 1.4x), so records from another version are ignored and
+ * restarted rather than blended.
+ */
+export declare const HEURISTIC_VERSION = 2;
 export interface Calibration {
     /** Multiply heuristic estimates by this. 1 means uncalibrated. */
     factor: number;

package/dist/calibration.js

@@ -17,6 +17,14 @@
 import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
 import { dirname, join } from "node:path";
 import { statePath } from "./ledger.js";
+/**
+ * Bumped whenever the uncalibrated heuristic changes. A factor learned against
+ * an older heuristic would correct for an error that no longer exists (0.19
+ * moved Claude from 4.0 to 2.75 chars/token; an old 1.4x factor on top of that
+ * would overcount by 1.4x), so records from another version are ignored and
+ * restarted rather than blended.
+ */
+export const HEURISTIC_VERSION = 2;
 /** Anything outside this is a bad sample, not a calibration. */
 const MIN_FACTOR = 0.5;
 const MAX_FACTOR = 2.0;
@@ -52,8 +60,14 @@ export function recordCalibration(model, exactTokens, heuristicTokens) {
     try {
         const all = readAll();
         const key = modelFamily(model);
-        const rec = all[key] ?? { exactSum: 0, heuristicSum: 0, samples: 0 };
-        all[key] = { exactSum: rec.exactSum + exactTokens, heuristicSum: rec.heuristicSum + heuristicTokens, samples: rec.samples + 1 };
+        const prev = all[key];
+        const rec = prev && prev.v === HEURISTIC_VERSION ? prev : { exactSum: 0, heuristicSum: 0, samples: 0 };
+        all[key] = {
+            exactSum: rec.exactSum + exactTokens,
+            heuristicSum: rec.heuristicSum + heuristicTokens,
+            samples: rec.samples + 1,
+            v: HEURISTIC_VERSION,
+        };
         mkdirSync(dirname(calibrationPath()), { recursive: true });
         writeFileSync(calibrationPath(), JSON.stringify(all, null, 2));
     }
@@ -66,7 +80,7 @@ export function calibrationFor(model) {
     if (process.env.CONTEXT_DOCTOR_NO_CALIBRATION)
         return { factor: 1, samples: 0 };
     const rec = readAll()[modelFamily(model)];
-    if (!rec || rec.samples < 1 || rec.heuristicSum <= 0)
+    if (!rec || rec.v !== HEURISTIC_VERSION || rec.samples < 1 || rec.heuristicSum <= 0)
         return { factor: 1, samples: 0 };
     const factor = rec.exactSum / rec.heuristicSum;
     if (!Number.isFinite(factor) || factor < MIN_FACTOR || factor > MAX_FACTOR)

package/dist/cli.js

@@ -20,6 +20,7 @@ import { runInstall, runUninstall } from "./install.js";
 import { listSessions, parseSessionFile } from "./session.js";
 import { runHook } from "./hook.js";
 import { buildImpactReport } from "./impact.js";
+import { measureTokenizer, renderTokenizer } from "./tokenizer-measure.js";
 import { renderPreferences, copyToClipboard, CHAT_PREFERENCES } from "./preferences.js";
 import { recordLedger } from "./ledger.js";
 import { runDoctor } from "./doctor.js";
@@ -100,7 +101,7 @@ Options:
                           Default: dedupe, trim-tool-results, strip-base64 (lossless-ish set)
   --keep-recent <n>       (optimize) Messages at the tail to leave untouched (default 6)
   --max-tool-tokens <n>   (optimize) Token budget for trimmed tool results (default 300)
-  --limit <n>             (accuracy) Sessions to sample (default 20)
+  --limit <n>             (accuracy, session --list) Sessions to sample or list (default 20)
   --check <cmd>           (experiment) Command whose exit code is the pass/fail for each arm
   --existing <id>         (experiment) Session id to fork the second arm from (never mutated)
   --budget <usd>          (experiment) Spend cap per arm (default 1)
@@ -110,6 +111,9 @@ Options:
                           overwrites a statusLine you already have)
   --port <n>              (proxy) Port to listen on (default 8787)
   --host <addr>           (proxy) Bind address (default 127.0.0.1; use 0.0.0.0 to expose)
+  --token <secret>        (proxy) Require /t/<secret>/ in every request path; needed before
+                          putting the proxy on a public URL (Cursor BYO-key, tunnels).
+                          Also read from CONTEXT_DOCTOR_PROXY_TOKEN
   --config <file>         (proxy) Per-route overrides: {"routes":[{"modelPrefix":"gpt","strategies":[...],
                           "keepRecent":n,"maxToolResultTokens":n}]} — first prefix match wins
   --upstream-anthropic <url>  (proxy) Override Anthropic upstream (testing)
@@ -199,6 +203,9 @@ function parseArgs(argv) {
             case "--host":
                 args.host = argv[++i];
                 break;
+            case "--token":
+                args.token = argv[++i];
+                break;
             case "--config":
                 args.config = argv[++i];
                 break;
@@ -321,7 +328,8 @@ function main() {
     }
     if (args.command === "accuracy") {
         const report = measureAccuracy(args.limit ?? 20);
-        console.log(args.json ? JSON.stringify(report, null, 2) : renderAccuracy(report));
+        const tokenizer = measureTokenizer(Math.max(args.limit ?? 20, 60));
+        console.log(args.json ? JSON.stringify({ ...report, tokenizer }, null, 2) : `${renderAccuracy(report)}\n\n${renderTokenizer(tokenizer)}`);
         return;
     }
     if (args.command === "report") {
@@ -369,9 +377,9 @@ function main() {
     }
     if (args.command === "session") {
         if (args.list) {
-            const sessions = listSessions();
+            const sessions = listSessions(args.limit ?? 20);
             if (sessions.length === 0) {
-                console.log("No Claude Code sessions found under ~/.claude/projects.");
+                console.log("No sessions found under ~/.claude/projects or ~/.codex/sessions.");
                 return;
             }
             for (const s of sessions) {
@@ -461,6 +469,7 @@ function main() {
             routes: routes,
             port: args.port,
             host: args.host,
+            token: args.token ?? process.env.CONTEXT_DOCTOR_PROXY_TOKEN,
             anthropicUpstream: args.upstreamAnthropic,
             openaiUpstream: args.upstreamOpenai,
             strategies: args.strategies.length > 0 ? args.strategies : loadedRc.config.strategies,

package/dist/hook.js

@@ -17,7 +17,7 @@ import { recordLedger, statePath } from "./ledger.js";
 import { parseConversation } from "./parse.js";
 import { profileConversation } from "./profile.js";
 import { parseSessionFile } from "./session.js";
-import { formatTokens } from "./tokens.js";
+import { formatTokens, CHARS_PER_TOKEN } from "./tokens.js";
 import { formatUsd } from "./pricing.js";
 import { checkBudget, loadConfig } from "./config.js";
 /** Default nudge threshold; a project budget or env var can lower/raise it. */
@@ -37,13 +37,16 @@ function warnThreshold(budgetMaxTokens) {
 /** Re-nudge only after the context grows another 40% — one reminder, not a nag. */
 const REGROWTH_FACTOR = 1.4;
 /**
- * Fast-path gate: text tokens are at least ~4 bytes each and the transcript
+ * Fast-path gate: no tokenizer we model packs more than one token into fewer
+ * bytes than its densest ratio (Claude on code, 2.4), and the transcript
  * carries JSON overhead on top, so a file smaller than this cannot possibly
  * hold that many tokens of context. Lean sessions cost one stat() call — the
- * transcript is never even read.
+ * transcript is never even read. (This used 4 bytes per token until 0.19,
+ * which let Claude sessions past the threshold take the fast path.)
  */
+const DENSEST_CHARS_PER_TOKEN = Math.min(...Object.values(CHARS_PER_TOKEN).map((r) => Math.min(r.prose, r.code)));
 function minBytesForWarn(threshold) {
-    return threshold * 4;
+    return Math.floor(threshold * DENSEST_CHARS_PER_TOKEN);
 }
 /**
  * State lives in one small file per session, not one shared map.

package/dist/index.d.ts

@@ -20,3 +20,4 @@ export type { Provider } from "./tokens.js";
 export { renderStatusLine, tailUsage } from "./statusline.js";
 export * from "./sketch.js";
 export * from "./preferences.js";
+export * from "./tokenizer-measure.js";

package/dist/index.js

@@ -15,3 +15,4 @@ export { estimateTokens, contextWindowFor, providerFor, formatTokens } from "./t
 export { renderStatusLine, tailUsage } from "./statusline.js";
 export * from "./sketch.js";
 export * from "./preferences.js";
+export * from "./tokenizer-measure.js";

package/dist/mcp.js

@@ -29,7 +29,7 @@ import { runSketch } from "./sketch.js";
  * "standing context-hygiene instructions in every chat" — no hook needed.
  */
 // Kept deliberately terse: these ride in EVERY conversation's context, and a
-// context-saving tool must not itself be context overhead (~130 tokens).
+// context-saving tool must not itself be context overhead (~250 Claude tokens).
 //
 // Written as conditions and actions, not offers. On Claude Desktop this string
 // is the only channel that reaches the model without the user asking — MCP has
@@ -40,7 +40,7 @@ import { runSketch } from "./sketch.js";
 const SERVER_INSTRUCTIONS = `Context hygiene rules (always on):
 1. Summarize any paste or tool result over ~2k tokens into the points you will use, then work from the summary; never carry it verbatim.
 2. Reference earlier content by name; never re-quote it. Never inline base64.
-3. When the conversation passes ~30 turns, or holds 3+ large pastes, or the user asks about tokens, cost, speed or limits: call profile_context BEFORE answering and act on its top finding. In a chat app pass a \`sketch\` (turn count + the large/repeated blocks, ~100 tokens), not the conversation. Do not estimate token counts yourself.
+3. When the conversation passes ~30 turns, or holds 3+ large pastes, or the user asks about tokens, cost, speed or limits: call profile_context BEFORE answering and act on its top finding. In a chat app pass a \`sketch\` (turn count + the large/repeated blocks, ~120 tokens), not the conversation. Do not estimate token counts yourself.
 4. If optimize_context returns a pruned-turns digest, you write the ≤150-token replacement summary.`;
 const STRATEGY_IDS = ["dedupe", "trim-tool-results", "trim-tool-calls", "strip-base64", "prune-history"];
 /**
@@ -49,8 +49,8 @@ const STRATEGY_IDS = ["dedupe", "trim-tool-results", "trim-tool-calls", "strip-b
  * recommended pattern.
  */
 function createServer() {
-    const server = new McpServer({ name: "context-doctor", version: "0.17.0" }, { instructions: SERVER_INSTRUCTIONS });
-    server.tool("profile_context", "Profile an LLM conversation or prompt: token breakdown, largest blocks, and actionable findings about wasted context (duplicates, oversized pastes or tool results, base64 blobs, long history). Two inputs, pass ONE: `conversation` (full OpenAI/Anthropic JSON or raw text, for agents, files and proxies) or `sketch` (for chat apps such as Claude Desktop or ChatGPT where you cannot export the conversation: the turn count plus the few blocks that matter, ~100 tokens to write). Call it whenever the user asks about token usage, context size, cost, speed or limits, and on your own once the conversation passes ~30 turns or holds 3+ large pastes. Act on the top finding in your reply.", {
+    const server = new McpServer({ name: "context-doctor", version: "0.19.0" }, { instructions: SERVER_INSTRUCTIONS });
+    server.tool("profile_context", "Profile an LLM conversation or prompt: token breakdown, largest blocks, and actionable findings about wasted context (duplicates, oversized pastes or tool results, base64 blobs, long history). Two inputs, pass ONE: `conversation` (full OpenAI/Anthropic JSON or raw text, for agents, files and proxies) or `sketch` (for chat apps such as Claude Desktop or ChatGPT where you cannot export the conversation: the turn count plus the few blocks that matter, ~120 tokens to write). Call it whenever the user asks about token usage, context size, cost, speed or limits, and on your own once the conversation passes ~30 turns or holds 3+ large pastes. Act on the top finding in your reply.", {
         conversation: z.string().optional().describe("Conversation JSON (OpenAI or Anthropic format, or bare message array) or raw prompt text. Omit in chat apps and pass `sketch`."),
         sketch: z.object({
             turns: z.number().int().nonnegative().describe("User+assistant exchanges so far"),
@@ -66,7 +66,7 @@ function createServer() {
                 repeated: z.number().int().positive().optional().describe("Times this same content appears (2+ = duplicate)"),
                 stale: z.boolean().optional().describe("Already acted on; nothing in it is still needed"),
             })).describe("Only the blocks over ~500 tokens, repeated, or images. Plain turns need not be listed."),
-        }).optional().describe("Coarse description of the conversation for chat apps. Give one size hint per block (lines, words, chars or tokens)."),
+        }).optional().describe("Coarse description of the conversation for chat apps. One size hint per block: lines for code, words for prose, chars (or tokens) for logs and tool output, which vary most per line."),
         model: z.string().optional().describe("Target model name for context-window math, e.g. claude-sonnet-5 or gpt-4o"),
     }, async ({ conversation, sketch, model }) => {
         if (sketch) {

package/dist/optimize.d.ts

@@ -9,6 +9,11 @@
 export type StrategyId = "dedupe" | "trim-tool-results" | "trim-tool-calls" | "prune-history" | "strip-base64";
 export interface OptimizeOptions {
     strategies?: StrategyId[];
+    /**
+     * Model the conversation is for; picks the tokenizer ratios for savings and
+     * trim budgets. Defaults to the conversation's own `model` field.
+     */
+    model?: string;
     /** Tool results older than this many messages from the end get trimmed. */
     keepRecent?: number;
     /** Max tokens a trimmed tool result — or tool-call argument set — keeps. */

package/dist/optimize.js

@@ -7,7 +7,7 @@
  * so the output is a drop-in replacement for the input conversation.
  */
 import { createHash } from "node:crypto";
-import { estimateTokens } from "./tokens.js";
+import { charsPerTokenFor, estimateTokens } from "./tokens.js";
 import { hasBase64Blob, stripBase64Blobs } from "./blob.js";
 const TRIM_BOUNDARY_STEP = 10;
 const DEFAULTS = {
@@ -63,8 +63,10 @@ function editedLaterWithoutRead(messages, writeIndex, path) {
     }
     return false;
 }
-function trimCallArguments(input, maxTokens) {
-    const budgetChars = maxTokens * 4;
+function trimCallArguments(input, maxTokens, model) {
+    // Prose ratio: argument strings are mostly file content and commands, and
+    // the prose ratio keeps a little more than the budget rather than less.
+    const budgetChars = Math.round(maxTokens * charsPerTokenFor("", model));
     const out = {};
     for (const [key, value] of Object.entries(input)) {
         if (typeof value === "string" && value.length > budgetChars) {
@@ -158,8 +160,8 @@ function replaceText(content, newText) {
         out.push({ type: "text", text: newText });
     return out;
 }
-function truncateToTokens(text, maxTokens) {
-    const approxChars = maxTokens * 4;
+function truncateToTokens(text, maxTokens, model) {
+    const approxChars = Math.round(maxTokens * charsPerTokenFor(text, model));
     if (text.length <= approxChars)
         return text;
     const head = text.slice(0, approxChars);
@@ -229,7 +231,15 @@ export function optimizeConversation(input, options = {}) {
         if (!messages[i] || typeof messages[i] !== "object")
             messages.splice(i, 1);
     }
-    const tokensBefore = messages.reduce((s, m) => s + estimateTokens(textOf(m.content)), 0);
+    // The request's own model, else Anthropic's shape (a top-level system, or
+    // tool_use/tool_result blocks) says whose tokenizer counts it.
+    const looksAnthropic = (!Array.isArray(data) && data?.system != null) ||
+        messages.some((m) => Array.isArray(m?.content) && m.content.some((b) => b?.type === "tool_use" || b?.type === "tool_result"));
+    const model = options.model ??
+        (!Array.isArray(data) && typeof data?.model === "string" ? data.model : undefined) ??
+        (looksAnthropic ? "claude" : undefined);
+    const tok = (text) => estimateTokens(text, model);
+    const tokensBefore = messages.reduce((s, m) => s + tok(textOf(m.content)), 0);
     const applied = [];
     // -- strip-base64: replace inline blobs with a placeholder --------------------
     if (opts.strategies.includes("strip-base64")) {
@@ -237,9 +247,9 @@ export function optimizeConversation(input, options = {}) {
             const text = textOf(m.content);
             if (!hasBase64Blob(text))
                 return;
-            const before = estimateTokens(text);
+            const before = tok(text);
             const cleaned = stripBase64Blobs(text);
-            const saved = before - estimateTokens(cleaned);
+            const saved = before - tok(cleaned);
             if (saved > 50) {
                 m.content = replaceText(m.content, cleaned);
                 applied.push({ strategy: "strip-base64", messageIndex: i, tokensSaved: saved, note: "Removed inline base64 data" });
@@ -272,7 +282,7 @@ export function optimizeConversation(input, options = {}) {
                 seen.set(h, i);
                 return;
             }
-            const saved = estimateTokens(text);
+            const saved = tok(text);
             m.content = replaceText(m.content, `[context-doctor: identical to message #${first} — content removed]`);
             applied.push({ strategy: "dedupe", messageIndex: i, tokensSaved: saved, note: `Duplicate of message #${first}` });
         });
@@ -284,20 +294,20 @@ export function optimizeConversation(input, options = {}) {
             if (i >= cutoff || !isToolResultMessage(m))
                 return;
             const text = textOf(m.content);
-            const before = estimateTokens(text);
+            const before = tok(text);
             if (before <= opts.maxToolResultTokens)
                 return;
-            const trimmed = truncateToTokens(text, opts.maxToolResultTokens);
+            const trimmed = truncateToTokens(text, opts.maxToolResultTokens, model);
             // The truncation notice has a length of its own, so a result only just
             // over the budget can come back LARGER than it went in. Measured on a
             // real session: 2,941 tokens "optimized" to 2,947.
-            if (estimateTokens(trimmed) >= before)
+            if (tok(trimmed) >= before)
                 return;
             m.content = replaceText(m.content, trimmed);
             applied.push({
                 strategy: "trim-tool-results",
                 messageIndex: i,
-                tokensSaved: before - estimateTokens(trimmed),
+                tokensSaved: before - tok(trimmed),
                 note: "Stale tool result truncated",
             });
         });
@@ -322,12 +332,12 @@ export function optimizeConversation(input, options = {}) {
                     // those alone; the 63% never touched again are still pure gain.
                     if (editedLaterWithoutRead(messages, i, writtenPath(b)))
                         continue;
-                    const before = estimateTokens(JSON.stringify(b.input));
+                    const before = tok(JSON.stringify(b.input));
                     if (before <= opts.maxToolResultTokens)
                         continue;
-                    const trimmedInput = trimCallArguments(b.input, opts.maxToolResultTokens);
+                    const trimmedInput = trimCallArguments(b.input, opts.maxToolResultTokens, model);
                     // Same trap as tool results: the marker can outweigh what it replaces.
-                    const after = estimateTokens(JSON.stringify(trimmedInput));
+                    const after = tok(JSON.stringify(trimmedInput));
                     if (after >= before)
                         continue;
                     b.input = trimmedInput;
@@ -339,11 +349,11 @@ export function optimizeConversation(input, options = {}) {
                 const args = tc?.function?.arguments;
                 if (typeof args !== "string")
                     continue;
-                const before = estimateTokens(args);
+                const before = tok(args);
                 if (before <= opts.maxToolResultTokens)
                     continue;
-                tc.function.arguments = truncateToTokens(args, opts.maxToolResultTokens);
-                saved += before - estimateTokens(tc.function.arguments);
+                tc.function.arguments = truncateToTokens(args, opts.maxToolResultTokens, model);
+                saved += before - tok(tc.function.arguments);
             }
             if (saved > 0) {
                 applied.push({
@@ -375,7 +385,7 @@ export function optimizeConversation(input, options = {}) {
             // 1,011-message session, which pruned to 7 messages with one orphan.
             dropOrphanedToolResults(messages.slice(keepFrom));
             const pruned = messages.slice(0, keepFrom);
-            const prunedTokens = pruned.reduce((s, m) => s + estimateTokens(textOf(m.content)), 0);
+            const prunedTokens = pruned.reduce((s, m) => s + tok(textOf(m.content)), 0);
             // Digest: first ~200 chars of each pruned turn — enough for a host LLM to
             // write a faithful summary, small enough not to defeat the pruning.
             prunedDigest = pruned
@@ -390,11 +400,11 @@ export function optimizeConversation(input, options = {}) {
             applied.push({
                 strategy: "prune-history",
                 messageIndex: 0,
-                tokensSaved: prunedTokens - estimateTokens(stub.content),
+                tokensSaved: prunedTokens - tok(stub.content),
                 note: `Pruned ${pruned.length} old messages`,
             });
         }
     }
-    const tokensAfter = messages.reduce((s, m) => s + estimateTokens(textOf(m.content)), 0);
+    const tokensAfter = messages.reduce((s, m) => s + tok(textOf(m.content)), 0);
     return { conversation: data, tokensBefore, tokensAfter, applied, prunedDigest };
 }

package/dist/parse.d.ts

@@ -34,6 +34,8 @@ export interface NormalizedConversation {
     messages: NormalizedMessage[];
     /** Format detected, for reporting. */
     sourceFormat: "openai" | "anthropic" | "array" | "text";
+    /** The request's own `model` field, when it has one. Picks the tokenizer ratios. */
+    model?: string;
     /**
      * Set when the input could not be read as a conversation. Silently profiling
      * a broken file as one big "user message" produces a confident, wrong report

package/dist/parse.js

@@ -146,5 +146,6 @@ export function parseConversation(input) {
         : messages.length === 0
             ? "This JSON has no `messages` array (and no `system`) — it does not look like a conversation. Expected {\"messages\":[{\"role\":…,\"content\":…}]}."
             : undefined;
-    return { sourceFormat: isAnthropic ? "anthropic" : "openai", parseWarning, messages };
+    const model = typeof obj.model === "string" && obj.model ? obj.model : undefined;
+    return { sourceFormat: isAnthropic ? "anthropic" : "openai", parseWarning, messages, model };
 }

package/dist/preferences.d.ts

@@ -4,7 +4,7 @@
  * "custom instructions" are read on every turn of every chat, which is the
  * closest thing those surfaces have to a UserPromptSubmit hook.
  *
- * Kept short (~90 tokens): it rides in every conversation.
+ * Kept short (~180 Claude tokens, ~120 GPT): it rides in every conversation.
  */
 export declare const CHAT_PREFERENCES: string;
 /** Where to paste, per app. Paths are the UI labels, not URLs, so they survive redesigns. */

package/dist/preferences.js

@@ -4,7 +4,7 @@
  * "custom instructions" are read on every turn of every chat, which is the
  * closest thing those surfaces have to a UserPromptSubmit hook.
  *
- * Kept short (~90 tokens): it rides in every conversation.
+ * Kept short (~180 Claude tokens, ~120 GPT): it rides in every conversation.
  */
 import { spawnSync } from "node:child_process";
 export const CHAT_PREFERENCES = [