@kunal_dutt/token-lens 0.0.0-stage → 0.2.16

package/GATEWAY.md

@@ -0,0 +1,491 @@
+# Token Lens gateway
+
+A self-hosted gateway that cuts the tokens your AI features, workflows and
+agents spend, on any model: OpenAI, Anthropic, Azure OpenAI, or a model you
+run yourself behind an OpenAI-compatible server (vLLM, TGI, SGLang, Ollama,
+LiteLLM, an in-house serving stack).
+
+It speaks the OpenAI and Anthropic APIs, so nothing in your app changes except
+its **base URL**. It runs inside your own network; prompts never leave it except
+to the upstream you configure, and it stores no prompt text (logs and reports
+hold counts only).
+
+```
+your app / agent / workflow  ──►  Token Lens gateway  ──►  OpenAI · Anthropic · Azure · your own model
+   (any language, any SDK)        optimize · measure · report
+```
+
+**Success criterion: fewer tokens, same work.** The gateway never changes the
+model a call asks for and never suggests a smaller one. It cuts what the agent
+sends (repeated prompts, history, tool definitions and results, retries), shows
+the saving in tokens and money before and after, and backs it with output
+checks: sampled calls are re-run as originally written and the answers compared.
+
+## Run it
+
+```bash
+docker build -t token-lens-gateway token-lens
+docker run -p 8080:8080 \
+  -e OPENAI_API_KEY=sk-... -e ANTHROPIC_API_KEY=sk-ant-... \
+  -e TOKEN_LENS_CLIENT_KEYS="payments-team:tl-key-1,support-agent:tl-key-2" \
+  token-lens-gateway
+```
+
+or without Docker (Node 20+): `npm install && npm run build:gateway && node dist-gateway/gateway.mjs`.
+The bundle is one file with no runtime dependencies.
+
+For several upstreams, routing and your own models, mount a config file and set
+`TOKEN_LENS_CONFIG` (see [deploy/gateway.example.json](deploy/gateway.example.json)
+and [deploy/docker-compose.yml](deploy/docker-compose.yml)).
+
+## Point your app at it
+
+| Client | Change |
+| --- | --- |
+| OpenAI SDK (Python, Node, Java, Go, .NET…) | `base_url="http://token-lens:8080/v1"`, `api_key=<your Token Lens client key>` |
+| Anthropic SDK | `base_url="http://token-lens:8080"`, `api_key=<client key>` |
+| LangChain / LangGraph | `ChatOpenAI(base_url=…/v1)` or `ChatAnthropic(base_url=…)` |
+| LlamaIndex, CrewAI, AutoGen, Semantic Kernel, Vercel AI SDK, Spring AI | their OpenAI or Anthropic provider's base URL setting |
+| OpenAI Agents SDK | `AsyncOpenAI(base_url=…/v1)` as the client (Responses API is supported) |
+| Anything else | `OPENAI_BASE_URL` / `ANTHROPIC_BASE_URL` environment variables, or raw HTTP |
+
+```python
+from openai import OpenAI
+client = OpenAI(base_url="http://token-lens:8080/v1", api_key="tl-key-1")
+client.chat.completions.create(model="acme-llm-70b", messages=[...])   # routed to your own model
+```
+
+Endpoints optimized: `/v1/chat/completions`, `/v1/responses`, `/v1/messages`,
+streaming included. Every other `/v1/...` path (models, embeddings, files,
+batches, count_tokens) is passed through unchanged.
+
+### One app, any model: calls between the two APIs
+
+An app written with the Anthropic SDK can call a model behind an
+OpenAI-compatible API (an in-house model on vLLM, Azure, any OpenAI-style
+server), and an app written with the OpenAI SDK can call Claude. Route the model
+to the upstream in the config (`"routes": [{ "model": "acme-*", "upstream": "inhouse" }]`)
+and keep calling the endpoint your SDK uses.
+
+- **What is translated:** system prompts, text, images, tool definitions, tool
+  calls and results, `tool_choice`, stop sequences, temperature and token limits,
+  in both directions. Answers, usage (including cached tokens) and errors come
+  back in your SDK's shape, and streams are translated event by event.
+- **What stays the same:** the model. Every Token Lens fix and output check works
+  the same on translated calls. The gateway's prompt caching becomes Anthropic
+  cache breakpoints when an OpenAI-style call goes to Claude.
+- **Keys:** if callers bring their own key, it is sent the way the other API
+  expects (`Authorization: Bearer` becomes `x-api-key`, and the reverse).
+
+## Your own model
+
+A company model isn't in anyone's built-in list, so register it: its context
+window, tokenizer and price. Matching is by model id, so `acme-llm` also covers
+`acme-llm-70b-instruct`.
+
+```json
+"models": [
+  { "id": "acme-llm", "match": ["acme-llm", "/^acme-.*-instruct$/"], "contextWindow": 32768,
+    "tokenizer": "heuristic", "charsPerToken": 3.6,
+    "pricing": { "inputPerMTok": 0.2, "outputPerMTok": 0.6 } }
+],
+"upstreams": { "inhouse": { "type": "openai", "baseUrl": "http://llm.internal:8000/v1" } },
+"routes": [{ "model": "acme-*", "upstream": "inhouse" }]
+```
+
+- `tokenizer`: `"o200k"` if the model uses OpenAI's o200k tokenizer, otherwise
+  `"heuristic"`. Heuristic counts **calibrate themselves**: each time the
+  upstream reports usage, the gateway compares it with its estimate and adjusts
+  that model's factor (see `calibration` in `/token-lens/stats`).
+- The same entry can override a built-in model, for example to add exact pricing
+  for `gpt-4o-mini`.
+- Models can also be added at runtime: `POST /token-lens/models` with the admin
+  key (`TOKEN_LENS_ADMIN_KEY`). Runtime entries live in memory; put permanent
+  ones in the config file.
+
+## What it does to each request
+
+The gateway sorts every request's input into parts: **system** prompt,
+**tools** definitions, **history**, **tool outputs**, **context** (material
+pasted or retrieved into the newest message) and the **prompt** itself. It
+counts each part in the model's own tokens, tracks the parts across the calls
+of a run, and applies the fix that fits the part that is growing.
+
+Changes that keep the meaning are applied automatically:
+
+| Fix | Part | What it does |
+| --- | --- | --- |
+| `prompt` | user messages | The Token Lens optimizer, adapted to the model and task. Code, quotes, URLs, XML-tagged and pasted material stay verbatim. Every user message is rewritten the same way on every call, so the model sees one consistent history and prompt caches keep hitting. |
+| `minify-json` | tool outputs | Pretty-printed JSON re-serialized without whitespace. Same data. |
+| `collapse-repeats` | tool outputs | Runs of identical lines become one line plus `[previous line repeated N more times]`. |
+| `dedupe-tool-outputs` | tool outputs | When the agent gets the same result twice (same tool, same input), the result is sent once; later copies point back to the first, which stays in place so caching still works. |
+| `dedupe-passages` | context, history, tool outputs | A paragraph of 200+ characters repeated within a request (the same retrieved chunk, a policy pasted twice) is sent once. Later copies point back to the first. |
+| `dedupe-images` | images | An image sent again later in the conversation (a repeated screenshot or upload) is sent once. Later copies become a one-line pointer to the first. |
+| `prompt-caching` | system, tools, history | Once a run re-sends a prefix of 1,024+ tokens: Anthropic cache breakpoints after the tools, the system prompt and the conversation; OpenAI's `prompt_cache_key` per run. Cached input costs about a tenth. Not added to one-off calls, where a cache write would cost more. |
+| retry replay | whole call | An identical, non-streamed, temperature-0 request within 2 minutes gets the earlier answer, at no token cost. Sampled requests (temperature unset or > 0) are always sent. |
+| embeddings cache | `/v1/embeddings` | A text this team already embedded with the same model and settings is answered from memory; only new texts go upstream. See [Embeddings](#embeddings-unchanged-documents-once). |
+
+Changes that could change the answer wait for an opt-in, either for everyone
+in the config's `optIn` or per request with `x-token-lens-allow`:
+
+| Opt-in | What it does |
+| --- | --- |
+| `elide-tool-outputs` | Tool outputs older than the newest 3 (and over 300 tokens) are replaced with a one-line stub. |
+| `lazy-tools` | Loads tool definitions only when the agent needs them (the idea behind Atlassian's mcp-compressor). See [Large tool lists](#large-tool-lists-lazy-tools). |
+| `trim-tool-fields` | In tool outputs older than the newest 3, JSON fields the agent never referred to afterwards (no value and not the field's name appears in any later message or tool call) are left out and listed in `_omitted_by_token_lens`. Small fields stay. |
+| `compact-json` | In JSON tool outputs, a list of 5+ records with the same keys is written once as `{"_columns": [...], "_rows": [[...], ...]}` instead of repeating every key in every record. Nothing is dropped. |
+| `park-tool-outputs` | Older tool outputs over 1,500 tokens stay outside the conversation: their first lines and an id remain, plus a `token_lens_read_output` tool the model calls to search them or read a slice. The gateway answers those look-ups itself and re-sends, so the app only gets the model's real answer; after 3 look-ups the request goes again with the outputs in full. Non-streamed OpenAI Chat and Anthropic requests. |
+| `flex` | The provider's cheaper, slower processing on the same model: OpenAI's `service_tier: "flex"`, about half price. Offered on api.openai.com upstreams (set `"flex": true` on others that support it). If flex is busy or not offered for the model, the same request goes again at the normal tier. Doesn't change the request's content. |
+| `warm-cache` | When several calls with the same 1,024+ token prefix arrive at once, the first goes alone and the rest wait for it (up to 15 seconds, `optIn.warmCache.maxWaitMs`), so they read its cache instead of each paying for the prefix. Trades a little speed; doesn't change requests. |
+| run budget | A cap per run on calls and/or spend (`runBudget` in the config, or `x-token-lens-max-calls` / `x-token-lens-max-cost-usd`). Calls past the cap get an error instead of being sent, which stops a looping agent. |
+
+Per-request controls (headers): `x-token-lens-mode: optimize | observe | off`,
+`x-token-lens-level: light | balanced | aggressive`, `x-token-lens-allow`,
+`x-token-lens-run` (names the run; otherwise calls are grouped by app, model,
+system prompt and first user message), `x-token-lens-app` (when no client keys
+are configured).
+
+Every response carries `x-token-lens-run`, `-profile`, `-input-tokens`,
+`-saved-tokens`, `-actions`, `-tips` and `-growing` (the part that grew most in
+this run). `x-token-lens-price-tier: near 200000` or `above 200000` warns that a
+call is close to or past the size where its model bills the whole request at a
+higher rate (see [Price tiers](#price-tiers)).
+
+## Savings report: tokens and money, before and after
+
+`GET /token-lens/report` is a page anyone can open in a browser (with
+`?key=<client key>` when keys are set): tokens and cost before and after, money
+saved, and the share of checked calls whose output was unchanged, broken down by
+team, agent, model and run. `GET /token-lens/report.json` has the same data.
+
+- **Before** is what the same calls would have cost without Token Lens: the
+  tokens it removed added back, cached input priced as normal input, and
+  replayed retries paid in full. **After** is what the provider billed, with
+  cached reads and cache writes at their own rates.
+- Built-in model families carry public API list prices. For your own models, and
+  to use your negotiated rates, give the model `pricing` in the config. Calls on
+  models without a price are counted in tokens only, and the report says how many.
+- Name the agent or workflow with the `x-token-lens-agent` header to get a row
+  per agent; the team is the client key's name.
+
+## Output checks: evidence the work is unchanged
+
+For a sample of calls the gateway changed, it also sends the original request
+to the same model in the background and compares the two answers. Streamed
+calls are checked too: the streamed answer is put back together and compared
+with an unstreamed original.
+
+- A check passes when the tool calls are identical (same tools, same arguments)
+  and the text says the same thing (word and word-pair overlap of at least
+  `minSimilarity`, 0.8 by default). A changed number counts as a different answer.
+- `verify.sampleRate` (default 5%; `TOKEN_LENS_VERIFY_RATE`) sets how many
+  calls are checked; calls using an opt-in fix are checked at least 25% of the
+  time. Checks cost a call each, so after the first 5 per team and agent they
+  stop whenever check tokens would exceed 20% of the tokens saved
+  (`maxShareOfSavings`). Their cost is shown separately and the report's net
+  saving subtracts it.
+- `x-token-lens-verify: 1` checks a specific call. Results appear per run
+  (`outputCheck`, `outputChecks`), in the report and in `/metrics`.
+- Even unchanged requests don't always get word-for-word identical answers when
+  sampling is on, so treat the rate as a lower bound on how often the work is the
+  same; temperature-0 calls give the cleanest signal.
+
+### The rule: no fix may change an answer
+
+- **A changed answer switches the fix off.** When a check finds a different
+  answer, the gateway asks the model the original request once more. If the
+  model's own two answers agree, the fixes that were in play are switched off for
+  that team and agent (the log says which). If the model's own answers differ just
+  as much, the difference isn't the fixes' and they stay on (the check shows
+  `modelVaries: true`). Switched-off fixes are listed at `GET /token-lens/guard`,
+  in `/token-lens/stats` and in the report, and stay off until an admin calls
+  `POST /token-lens/guard/reset` (body `{}` for all, or `{"app": "...", "agent": "..."}`).
+- **Opt-in fixes prove themselves first.** `elide-tool-outputs` and `lazy-tools`
+  can change an answer by design, so for each team and agent they start on
+  probation: the original request goes up alongside the optimized one, and the app
+  gets the optimized answer only when the two match. If they differ, the app gets
+  the original's answer (`x-token-lens-answer: original`) and the fix is switched
+  off as above. After `verify.probation` matches in a row (5 by default) the fix
+  runs on its own, still checked at least 25% of the time. During probation the
+  extra call is counted as check cost, and opt-in fixes are left out of streamed
+  and batched calls, whose answers can't be compared before the app gets them.
+- Fixes that never change the request's content (prompt caching, batching on the
+  same model, replaying an identical retry) aren't switched off. Images are never
+  shrunk: providers already resize them to their own limit before counting tokens.
+
+## Optimizing a running agent's whole run
+
+Send the agent's model calls through the gateway (base URL) and give each run a
+name with `x-token-lens-run` and the agent a name with `x-token-lens-agent`.
+Every step of the run is then optimized as above, and the run is judged as a
+whole, not call by call:
+
+- **Total cost of the run**: `/token-lens/runs/<id>` adds up every step,
+  replayed retries (free) and output checks (shown separately): `costBeforeUsd`,
+  `costUsd`, `savedUsd`, `checkCostUsd`, `netSavedUsd`, and each step's own cost.
+- **What the agent does across steps**: repeated tool calls (same tool, same
+  input), loops (the same call several times in a row), retries, output that
+  costs more than input, and context that keeps growing.
+- **What happens about it**: the gateway fixes what it can without changing the
+  answer (identical tool results sent once, caching, retry replay, compact tool
+  output). What only the agent's code can change, such as skipping a repeated
+  step, caching tool results or adding a step limit, is named in the run's
+  findings with the tokens it costs. A run budget is there for runaway loops when
+  you turn it on.
+
+### Test it: the same agent with and without Token Lens
+
+`scripts/agent-benchmark.mjs` runs a multi-step customer-care agent (4 tools,
+about 10 tool calls, refunds two delayed orders) through the gateway twice:
+once observed only, with requests sent as written, and once optimized. It
+compares the actions taken, the final answer, tokens and the run's cost,
+including the output checks.
+
+```bash
+npm install && npm run build:gateway
+npm run benchmark:agent                      # stand-in model, no key, runs locally and in CI
+npm run benchmark:agent -- --api anthropic   # the same stand-in behind the Anthropic API (simulates prompt caching)
+npm run benchmark:agent -- --extra-tools 30 --allow lazy-tools   # 30 more tools the agent doesn't need, with lazy-tools
+ANTHROPIC_API_KEY=... npm run benchmark:agent -- --provider anthropic --model claude-opus-5-5
+OPENAI_API_KEY=... npm run benchmark:agent -- --provider openai --model <your-model-id>
+```
+
+With a real provider `--model` is required. Pass the exact model your agent
+runs (for example `claude-opus-5-5` or `claude-sonnet-5-5`, or the OpenAI model
+id shown in your account), so the result is about that model, not a smaller one.
+
+With a real model it also runs the baseline a second time as a control, so the
+model's own run-to-run variation is visible next to any difference Token Lens
+makes. `--out result.json` saves everything. Checks run on every optimized step
+here (capped by `maxShareOfSavings`) to give the most evidence, so on a short run
+their cost can exceed the saving; in production about 5% of calls are checked.
+
+### Work that can wait: half price (`x-token-lens-batch`)
+
+Overnight reports, evaluations and bulk summaries don't need an answer in
+seconds. Send them with `x-token-lens-batch: true`:
+
+- **What happens:** the gateway applies its usual fixes, then queues the call
+  for the provider's batch option on the same model. That is OpenAI's Batch API
+  or Anthropic's Message Batches. Both bill about half the normal price and
+  answer within 24 hours.
+- **What the app gets back:** `202` and a job id straight away. Fetch the answer
+  later from `GET /token-lens/batches/<id>`. It returns `queued`, `submitted`,
+  `completed` with the provider's normal response body, or `failed` with the
+  reason. Each result is handed over once and kept for a day.
+- **Reports:** batched calls show at the batch price next to what they would have
+  cost, and carry a `batch` action.
+- **When it's refused:** streamed requests, and upstreams without a batch option.
+  In-house servers get `"batch": true` on their upstream if they speak OpenAI's
+  Batch API. A refused request returns an error instead of being sent at full
+  price.
+- **Timing:** queued calls go out every `batch.flushSeconds` (60) and results are
+  checked every `batch.pollSeconds` (60). `batch.discount` (0.5) is the price
+  factor used in reports.
+
+### Embeddings: unchanged documents once
+
+Search and retrieval pipelines often re-embed every document on each sync,
+changed or not. On `/v1/embeddings` the gateway keeps each vector it gets, keyed
+by a hash of the input with the team, upstream, model, `dimensions` and
+`encoding_format`, and sends only new inputs upstream. The response is the
+provider's own vectors in the request's order, with
+`x-token-lens-cached-embeddings` and `-saved-tokens` headers.
+
+- Texts aren't stored, only their hash and vector, in this gateway's memory up to
+  `embeddingsCacheMB` (256), least recently used out first.
+- Teams don't share entries. Turn it off with `"safe": { "cacheEmbeddings": false }`.
+
+### Price tiers
+
+Some models bill the whole request at a higher rate once its input passes a
+size: Gemini Pro and Claude Sonnet's 1M-token window above 200K input tokens,
+for example. Built-in models carry their list price for this, and `priceTiers`
+in the config adds or overrides models by id. The gateway:
+
+- prices calls past the size at the higher rate in every report, and prices the
+  "without Token Lens" cost at that rate when the original request was past it;
+- adds `x-token-lens-price-tier` to calls within 10% of it or past it;
+- reports a `price-tier` run finding with what to drop first.
+
+Nothing is cut automatically, since that would change the answer.
+
+### Images
+
+The gateway reads each image's size and estimates its tokens with the
+provider's published rules: about w×h/750 for Anthropic, and 512 px tiles for
+OpenAI. Run reports show those tokens, and a repeated image is sent once
+(`dedupe-images`).
+
+Hosted providers shrink large images before counting them. Anthropic fits them
+to 1,568 px; OpenAI fits them to 2,048 px with a 768 px short side. So sending a
+smaller image than that saves upload time, not tokens, and the gateway doesn't
+claim a saving for it. The changes that do save tokens are in the run report's
+"images" finding: keep a description instead of an old screenshot, crop to the
+part that matters, or use low detail where it is enough. Low detail can change
+answers, so check it with output checks on.
+
+### Large tool lists (`lazy-tools`)
+
+Agents connected to large MCP servers often send dozens of tool definitions on
+every call and use only a few. With `x-token-lens-allow: lazy-tools`, or
+`"optIn": { "lazyTools": { "minTools": 8 } }` in the config:
+
+- **What goes up:** tools the run hasn't used yet are sent as a name and a
+  one-line summary, plus a small `token_lens_load_tools` tool.
+- **Loading on demand:** when the model calls the loader, or calls a summarized
+  tool directly, the gateway adds those tools' full definitions and sends the same
+  request again. The app only ever gets the final answer. Any tool call in it was
+  made with that tool's full definition, and the loader never reaches the app.
+- **Later calls:** tools a run has loaded or used stay in full on its later calls,
+  so each tool is loaded at most once per run.
+- **When it applies:** non-streamed requests with at least 8 plain function tools.
+  A tool forced with `tool_choice` is always sent in full.
+- **Cost accounting:** the extra loading calls are counted in the run's cost.
+  The saving shown is net of them, and can be negative when a run uses most of
+  its tools.
+
+It can change which tool the model picks, so it is opt-in, and output checks
+sample it more often. In the stand-in benchmark with 34 tools, the run went from
+94,741 to 48,452 tokens (49% fewer). The agent took the same actions and gave an
+identical answer, and 5 of 5 output checks matched.
+
+### What else the run report spots
+
+These need a change in the agent, so the gateway shows them with the fix and
+its saving, and changes nothing itself:
+
+- **`cache-breaker`:** the system prompt or tool list starts the same as the
+  previous call's and then changes: a timestamp, a date, an id, a number, or
+  tools in a different order. The provider's cache misses from that point, so
+  those tokens are billed in full instead of at about a tenth. The report names
+  the kind of change, never the text, and gives the change to make (move it to the
+  newest message, or sort tools once).
+- **`json-retries`:** the model's JSON reply didn't parse and the agent asked
+  again, each time a whole extra step. The fix is the provider's strict
+  structured output on the same model, written for the API in use (`json_schema`
+  response format on OpenAI, a forced tool on Anthropic).
+- **`shared-reads`:** files or documents this run read that other runs of the
+  team (other sub-agents) read too, matched by a hash of the path. The fix loads
+  them once and passes them on, or puts them at the top of each agent's prompt so
+  the provider's cache serves them.
+- **`price-tier`:** see [Price tiers](#price-tiers).
+
+### Several gateway copies, and restarts: shared storage
+
+By default reports live in memory and start empty after a restart. To share
+them across copies behind a load balancer, and keep them across restarts, point
+the gateway at your own Redis or Valkey:
+
+```bash
+TOKEN_LENS_REDIS_URL=rediss://:password@redis.internal:6380/0 npm run gateway
+```
+
+or `"storage": { "redisUrl": "env:REDIS_URL" }` in the config. A single gateway
+can use a file instead (`"storage": { "file": "/var/lib/token-lens/reports.json" }`
+or `TOKEN_LENS_STORAGE_FILE`).
+
+- **Totals:** every copy adds its own changes to one set of counters every
+  `storage.flushSeconds` (10), so the report, CSV and stats show all copies.
+- **Run reports:** kept `storage.runDays` (7) days. Any copy can show any run.
+- **Switched-off fixes:** shared, so a fix that changed an answer on one copy is
+  off on all of them, and an admin reset turns it back on everywhere.
+- **What is stored:** only what reports already show: totals, run summaries,
+  findings and per-call counts, with tool inputs hashed. Never prompts or
+  answers. Keys start with `storage.keyPrefix` (`token-lens:`).
+
+### Savings export for finance
+
+`GET /token-lens/report.csv` downloads tokens and money before and after, output
+check cost and net saving, one row per team, agent and model.
+`/token-lens/report.csv?by=run` gives one row per recent run. The HTML report
+links both.
+
+## Where the tokens go: run reports
+
+`GET /token-lens/runs/<run id>` explains a run: its calls, the parts of each, and
+findings ranked by how much they would save. Each finding says what the gateway
+already does about it, what a run like this one would save in tokens and
+dollars, and the exact changes that cut it, each with where it goes (gateway
+config, request header, agent code, system prompt, model server) and what to
+paste. None of them switch to a different or smaller model. For example:
+
+```
+- The agent calls the same tool with the same input more than once - saves ~3,377 tokens ($0.0068) per run
+    agent code: Cache tool results for the run, so get_order with the same input runs once
+      const cache = new Map(); const run = (name, args) => { ... };
+    system prompt: Tell the agent results don't change within a run
+      Tool results stay valid for this whole task. Do not call a tool again with the same input; use the earlier result.
+- Tool outputs take a large share of every call
+    request header: Replace older tool outputs with a stub (opt-in; output checks run on a sample)
+      x-token-lens-allow: elide-tool-outputs
+    agent code: Return only the fields the agent uses from each tool
+      return JSON.stringify(pick(result, ["id", "status", "total"]));
+```
+
+The HTML report (`/token-lens/report`) shows the same under "What to change
+next" for each recent run.
+
+Findings cover repeated prefixes, growing history, tool outputs, large
+retrieved context, retries, repeated tool calls, loops, output-heavy runs, hidden
+reasoning that outweighs the answer, and runs nearing the context limit.
+
+- `GET /token-lens/runs` lists recent runs (`?app=` to filter).
+- `GET /token-lens/stats` has the report's totals plus tokenizer calibration.
+- `GET /metrics` exposes the same totals to Prometheus, labelled by team, agent
+  and model.
+- `POST /token-lens/analyze` with `{"prompt", "model"}` returns the Token Lens
+  analysis of one prompt without calling a model.
+
+## Security
+
+- Set client keys (`TOKEN_LENS_CLIENT_KEYS="name:key,…"` or `clientKeys` in the
+  config). Each key's name becomes the `app` in logs, stats and metrics. Without
+  keys, anyone who can reach the port can use the upstream keys; the gateway
+  warns about this at startup.
+- Upstream keys are read from the environment (`"env:NAME"` in the config) and
+  never logged. When an upstream has no key, the caller's own key is passed
+  through.
+- Logs are one JSON line per call with counts, model, status and the fixes
+  applied; never prompt or response text. Run reports hold counts only. Tool
+  inputs, which can hold customer data, show only as a short hash (for example
+  `get_order(input #3f2a9c1e)`) unless you set `"reportToolArgs": true`.
+- `/token-lens/*` and `/metrics` need a client key (when keys are set); model
+  registration needs the admin key.
+
+## Privacy: nothing leaves your systems
+
+The gateway runs entirely on your own servers and needs no account, licence
+server or Token Lens service.
+
+- **Outbound requests:** it makes requests only to the upstreams you configure.
+  There is no telemetry, no update check and no call home.
+- **Reports and metrics:** stats, run reports, the HTML report and `/metrics` are
+  kept in memory on your gateway (or in your own Redis or file, if you turn on
+  shared storage), and only your own clients can read them.
+- **What providers see:** the provider receives the same request your app sent,
+  minus the `x-token-lens-*` headers. OpenAI's prompt cache key is a hash, so app
+  and run names never reach it.
+- **Output checks:** these re-send a sample of requests to the same upstream your
+  app already uses. Set `"verify": { "sampleRate": 0 }` to turn them off.
+- **Embeddings and file reads:** cached embeddings are kept as a hash of the text
+  and its vector, in memory, per team. Files several agents read are matched by a
+  hash of the path; reports show only how many and which agents.
+- **Test coverage:** a test checks all of this on every change. It records every
+  outbound request and asserts that customer data never appears in reports or
+  logs.
+
+## Limits
+
+- Without shared storage, run reports and stats are kept in memory per gateway
+  copy (the most recent 2,000 runs). With it, totals are combined exactly; a run
+  served by two copies at once is reported by whichever wrote last, so route a run
+  to one copy (by the `x-token-lens-run` header) where that matters. `/metrics`
+  is always per copy.
+- Translation between the two APIs covers Anthropic Messages and OpenAI Chat
+  Completions. The OpenAI Responses API reaches only OpenAI-compatible upstreams.
+- Counts for models without a known tokenizer are estimates until the upstream
+  has reported usage a few times.
+- Switched-off fixes and probation counts are kept in memory per gateway
+  instance, so a restart puts opt-in fixes back on probation.
+- A streamed answer longer than 16 MB isn't checked.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kunal Dutt
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/PRIVACY.md

@@ -0,0 +1,36 @@
+# Token Lens privacy policy
+
+_Last updated: 13 October 2026_
+
+Token Lens collects no data. Not from individuals, not from companies, not from
+anyone.
+
+- **No accounts, analytics, telemetry, crash reports, ads or trackers.** Token
+  Lens never "phones home". There is no Token Lens server that receives your
+  usage, and the publisher cannot see who uses it or how.
+- **Your prompts and chats never leave your device.** Every count, rewrite and
+  tip is computed on your own computer, browser or servers.
+- **Nothing is sold or shared,** because nothing is collected.
+
+## Each way of running Token Lens
+
+| Where it runs | What it reads | What it keeps | What it sends |
+| --- | --- | --- | --- |
+| **Browser extension** (Chrome, Edge, Firefox) | The prompt box, model picker and visible messages on the AI chat sites listed in its permissions, to count tokens. | Your settings, prices, budget and savings totals, in your browser's own extension storage. Uninstalling deletes them. | Nothing. Its only network request is on claude.ai: at most once a minute it reads your own session and weekly usage from claude.ai, signed in as you, so plan tips use real numbers. The result is not stored and goes nowhere else. |
+| **MCP server on your computer** (`npx @kunal_dutt/token-lens`, Claude Desktop extension) | Only the text an AI app passes to its tools. | Nothing. | Nothing by default. Only if you set your own `ANTHROPIC_API_KEY` or `GEMINI_API_KEY` for it does it send the text to that provider's token-counting endpoint, with your key, for an exact count. |
+| **Your own connector** (`npx @kunal_dutt/token-lens --http`) | Only the text sent to its tools. | Nothing. | Nothing beyond the reply. |
+| **Public hosted connector** (`/api/mcp`) | Only the text an AI app passes to its tools. | Nothing. Each request is handled in memory and discarded; requests are not logged or stored. The hosting provider keeps standard request metadata (time, IP address, URL) for security, as any website does. | Nothing beyond the reply. Prefer running your own copy if even that metadata matters. |
+| **Developer library** (`@kunal_dutt/token-lens/sdk` on npm, `token-lens` on PyPI, inside your app) | Your app's requests to OpenAI or Anthropic, in your own process. | Savings counters in your app's memory. | Nothing of its own. Your requests still go only to the provider your code already calls. |
+| **Gateway** (Docker image or `npx @kunal_dutt/token-lens gateway`, on your servers) | Your apps' model requests, which you route through it. | Savings counters in memory on your server. | Only to the model endpoints *you* configure (your OpenAI, Anthropic, Azure or in-house model). Nothing goes to Token Lens or anyone else. |
+
+## Stores and registries
+
+Chrome Web Store, Microsoft Edge Add-ons, Firefox Add-ons, npm, PyPI, GitHub
+and the MCP Registry may record installs or downloads under their own privacy policies.
+Token Lens receives none of that.
+
+## Changes and contact
+
+If this ever changed, the policy would be updated here first and the change
+would ship as a new version, never silently. Questions: contact
+the publisher, Kunal Dutt, through the support link on any store listing.

package/README.md

@@ -1,3 +1,81 @@
-# Temporary Holding Version
+# Token Lens
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Count and cut the tokens your prompts, chats and agents use, adapted to the
+model and the task, without switching your model or changing the answer. Every
+saving is shown in tokens and dollars, priced for the model you use.
+
+**Collects no data.** No accounts, analytics or telemetry. It makes no network
+requests of its own, except an exact token count from Anthropic or Google when
+you set your own `ANTHROPIC_API_KEY` or `GEMINI_API_KEY` for it. See [PRIVACY.md](PRIVACY.md).
+
+## Add it to your AI app
+
+It runs on your own computer with Node 20 or newer. `@latest` means
+each start picks up the newest version, so there's nothing to update.
+
+| App | How |
+| --- | --- |
+| **Claude Code** | `claude mcp add token-lens -- npx -y @kunal_dutt/token-lens@latest` |
+| **Claude Desktop** | Add the JSON below to `claude_desktop_config.json` (keeps itself up to date), or for a one-click install without Node, download [token-lens.mcpb](https://unpkg.com/@kunal_dutt/token-lens@latest/token-lens.mcpb) and open it. |
+| **Cursor, VS Code, Windsurf, Zed, any MCP app** | Add a stdio server with the JSON below. |
+| **ChatGPT, claude.ai and mobile apps** | Need a connector URL: use the public one from the Token Lens listing, or run your own (below). |
+
+```json
+{
+  "mcpServers": {
+    "token-lens": { "command": "npx", "args": ["-y", "@kunal_dutt/token-lens@latest"] }
+  }
+}
+```
+
+Then ask, for example: "optimize this prompt with Token Lens", "how full is this
+chat?" or "optimize this agent's system prompt and tools".
+
+Tools: `optimize_prompt`, `optimize_agent`, `count_tokens`,
+`check_conversation`, `list_model_profiles`.
+
+## Run your own connector
+
+```bash
+npx -y @kunal_dutt/token-lens@latest --http 8787     # serves http://localhost:8787/mcp
+```
+
+Put it behind HTTPS (any reverse proxy or tunnel) and add `https://<host>/mcp`
+as a custom connector in ChatGPT or claude.ai. It stores nothing.
+
+## Cost check for agent builders
+
+Every pull request that changes an agent's prompts or tool definitions gets a
+comment with the token and dollar change per call (and per month), priced for
+the model the agent runs on, plus what could be trimmed with the same output.
+
+- **GitHub:** copy [agent-cost-check.yml](https://unpkg.com/@kunal_dutt/token-lens@latest/agent-cost-check.yml)
+  to `.github/workflows/`, then set `--model` and your agent file paths.
+- **Any other CI or locally:**
+  `npx -y @kunal_dutt/token-lens@latest check-agent --model claude-opus-5-5 --base origin/main 'agents/**'`
+  prints the same report as Markdown (`--out file.md` to save it,
+  `--fail-above 500` to fail when a change adds more than 500 tokens per call).
+
+Agent files are `.json` specs (`system`, `tools`, `examples`, `messages`,
+`prompt`; OpenAI- and Anthropic-style tool lists work as-is) or any text file,
+read as a system prompt. It runs inside your CI and sends nothing anywhere.
+
+## Gateway for companies
+
+A self-hosted proxy that speaks the OpenAI and Anthropic APIs and optimizes
+every call your apps, workflows and agents make, with any model, including your
+own. It talks only to the model endpoints you configure. See
+[GATEWAY.md](GATEWAY.md).
+
+```bash
+docker run -p 8080:8080 -e OPENAI_API_KEY ghcr.io/duttkunal101-lab/token-lens-gateway:latest
+# or
+OPENAI_API_KEY=... npx -y @kunal_dutt/token-lens@latest gateway
+```
+
+## Browser extension
+
+For ChatGPT, Claude, Gemini, DeepSeek, Le Chat and Grok on the web: search
+"Token Lens" in the Chrome Web Store, Microsoft Edge Add-ons or Firefox Add-ons.
+
+MIT licensed.