ai-token-estimator 1.1.0 → 1.2.0

package/README.md

@@ -1,12 +1,28 @@
 # ai-token-estimator
 
-Estimate token counts and costs for LLM API calls based on character count and model-specific ratios.
-
-> **Important:** This is a rough estimation tool for budgeting purposes, not a precise tokenizer. Actual token counts may vary by ±20% depending on:
-> - Content type (code vs prose)
-> - Language (CJK languages use more tokens)
-> - API message framing overhead
-> - Special characters and formatting
+[![npm](https://img.shields.io/npm/v/ai-token-estimator.svg)](https://www.npmjs.com/package/ai-token-estimator)
+[![CI](https://github.com/BitsAndBytesAI/ai-token-estimator/actions/workflows/ci.yml/badge.svg)](https://github.com/BitsAndBytesAI/ai-token-estimator/actions/workflows/ci.yml)
+[![license](https://img.shields.io/npm/l/ai-token-estimator.svg)](https://github.com/BitsAndBytesAI/ai-token-estimator/blob/main/LICENSE)
+
+The best way to estimate **tokens + input cost** for LLM calls — with **exact OpenAI tokenization** (tiktoken-compatible BPE) and optional **official provider token counting** for Claude/Gemini.
+
+> Accuracy depends on the tokenizer mode you choose:
+> - **Exact** for OpenAI models when you use `openai_exact` / `encode()` / `decode()`.
+> - **Exact** for Claude/Gemini when you use `estimateAsync()` with their official count-tokens endpoints.
+> - **Heuristic** fallback is available for speed and resilience.
+
+## Features
+
+- **Exact OpenAI tokenization** (tiktoken-compatible BPE): `encode()` / `decode()` / `openai_exact`
+- **Official provider token counting** (async):
+  - Anthropic `POST /v1/messages/count_tokens` (`anthropic_count_tokens`)
+  - Gemini `models/:countTokens` (`gemini_count_tokens`)
+- **Fast local fallback** options:
+  - Heuristic (`heuristic`, default)
+  - Local Gemma SentencePiece approximation (`gemma_sentencepiece`)
+  - Automatic fallback to heuristic on provider failures (`fallbackToHeuristicOnError`)
+- **Cost estimation** using a weekly auto-updated pricing/model list (GitHub Actions)
+- TypeScript-first, ships ESM + CJS
 
 ## Installation
 
@@ -17,7 +33,7 @@ npm install ai-token-estimator
 ## Usage
 
 ```typescript
-import { estimate, getAvailableModels } from 'ai-token-estimator';
+import { countTokens, estimate, getAvailableModels } from 'ai-token-estimator';
 
 // Basic usage
 const result = estimate({
@@ -37,6 +53,10 @@ console.log(result);
 // List available models
 console.log(getAvailableModels());
 // ['gpt-5.2', 'gpt-4o', 'claude-opus-4.5', 'gemini-3-pro', ...]
+
+// Exact tokens for OpenAI, heuristic for others
+console.log(countTokens({ text: 'Hello, world!', model: 'gpt-5.1' }));
+// { tokens: 4, exact: true, encoding: 'o200k_base' }
 ```
 
 ## Exact OpenAI tokenization (BPE)
@@ -62,6 +82,97 @@ console.log(roundTrip); // "Hello, world!"
 Supported encodings:
 `r50k_base`, `p50k_base`, `p50k_edit`, `cl100k_base`, `o200k_base`, `o200k_harmony`
 
+## Using the exact tokenizer with `estimate()`
+
+`estimate()` is heuristic by default (fast). If you want to use exact OpenAI token counting:
+
+```ts
+import { estimate } from 'ai-token-estimator';
+
+const result = estimate({
+  text: 'Hello, world!',
+  model: 'gpt-5.1',
+  tokenizer: 'openai_exact',
+});
+
+console.log(result.tokenizerMode); // "openai_exact"
+console.log(result.encodingUsed);  // "o200k_base"
+```
+
+Or use `tokenizer: 'auto'` to use exact counting for OpenAI models and heuristic for everything else.
+
+## Provider token counting (Claude / Gemini)
+
+If you want **more accurate token counts** for Anthropic or Gemini models, you can call their official token counting endpoints
+via `estimateAsync()`. This requires API keys, and therefore should be used **server-side** (never in the browser).
+
+If you want these modes to **fail open** (fallback to heuristic estimation) when the provider API is throttled/unavailable or the API key is invalid,
+set `fallbackToHeuristicOnError: true`.
+
+### Anthropic: `POST /v1/messages/count_tokens`
+
+- Env var: `ANTHROPIC_API_KEY`
+
+```ts
+import { estimateAsync } from 'ai-token-estimator';
+
+const out = await estimateAsync({
+  text: 'Hello, Claude',
+  model: 'claude-sonnet-4-5',
+  tokenizer: 'anthropic_count_tokens',
+  fallbackToHeuristicOnError: true,
+  anthropic: {
+    // apiKey: '...' // optional; otherwise uses process.env.ANTHROPIC_API_KEY
+    system: 'You are a helpful assistant',
+  },
+});
+
+console.log(out.estimatedTokens);
+```
+
+### Gemini: `models/:countTokens` (Google AI Studio)
+
+- Env var: `GEMINI_API_KEY`
+
+```ts
+import { estimateAsync } from 'ai-token-estimator';
+
+const out = await estimateAsync({
+  text: 'The quick brown fox jumps over the lazy dog.',
+  model: 'gemini-2.0-flash',
+  tokenizer: 'gemini_count_tokens',
+  fallbackToHeuristicOnError: true,
+  gemini: {
+    // apiKey: '...' // optional; otherwise uses process.env.GEMINI_API_KEY
+  },
+});
+
+console.log(out.estimatedTokens);
+```
+
+### Local Gemini option: Gemma SentencePiece (approximation)
+
+If you want a **local** tokenizer option for Gemini-like models, you can use a SentencePiece tokenizer model (e.g. Gemma's
+`tokenizer.model`) via `sentencepiece-js`.
+
+```ts
+import { estimateAsync } from 'ai-token-estimator';
+
+const out = await estimateAsync({
+  text: 'Hello!',
+  model: 'gemini-2.0-flash',
+  tokenizer: 'gemma_sentencepiece',
+  gemma: {
+    modelPath: '/path/to/tokenizer.model',
+  },
+});
+
+console.log(out.estimatedTokens);
+```
+
+Note:
+- This is **not** an official Gemini tokenizer; treat it as an approximation unless you have verified equivalence for your models.
+
 ## API Reference
 
 ### `estimate(input: EstimateInput): EstimateOutput`
@@ -79,6 +190,9 @@ interface EstimateInput {
 }
 ```
 
+Note:
+- Provider-backed modes (`anthropic_count_tokens`, `gemini_count_tokens`, `gemma_sentencepiece`) are only supported in `estimateAsync()`.
+
 **Returns:**
 
 ```typescript
@@ -93,6 +207,20 @@ interface EstimateOutput {
 }
 ```
 
+### `estimateAsync(input: EstimateAsyncInput): Promise<EstimateOutput>`
+
+Async estimator that supports provider token counting modes:
+- `anthropic_count_tokens` (Anthropic token count endpoint)
+- `gemini_count_tokens` (Gemini token count endpoint)
+- `gemma_sentencepiece` (local SentencePiece, requires `sentencepiece-js` and a model file)
+
+API keys should be provided via env vars (`ANTHROPIC_API_KEY`, `GEMINI_API_KEY`) or passed explicitly in the config objects.
+
+If you pass `fallbackToHeuristicOnError: true`, provider-backed modes will fall back to heuristic estimation on:
+- invalid/expired API key (401/403)
+- rate limiting (429)
+- provider errors (5xx) or network issues
+
 ### `countTokens(input: TokenCountInput): TokenCountOutput`
 
 Counts tokens for a given model:

package/dist/index.cjs

@@ -1,7 +1,9 @@
 "use strict";
+var __create = Object.create;
 var __defProp = Object.defineProperty;
 var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
 var __getOwnPropNames = Object.getOwnPropertyNames;
+var __getProtoOf = Object.getPrototypeOf;
 var __hasOwnProp = Object.prototype.hasOwnProperty;
 var __export = (target, all) => {
   for (var name in all)
@@ -15,6 +17,14 @@ var __copyProps = (to, from, except, desc) => {
   }
   return to;
 };
+var __toESM = (mod, isNodeMode, target) => (target = mod != null ? __create(__getProtoOf(mod)) : {}, __copyProps(
+  // If the importer is in node compatibility mode or this is not an ESM
+  // file that has been converted to a CommonJS file using a Babel-
+  // compatible transform (i.e. "__esModule" has not been set), then set
+  // "default" to the CommonJS "module.exports" for node compatibility.
+  isNodeMode || !mod || !mod.__esModule ? __defProp(target, "default", { value: mod, enumerable: true }) : target,
+  mod
+));
 var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: true }), mod);
 
 // src/index.ts
@@ -22,10 +32,14 @@ var index_exports = {};
 __export(index_exports, {
   DEFAULT_MODELS: () => DEFAULT_MODELS,
   LAST_UPDATED: () => LAST_UPDATED,
+  countAnthropicInputTokens: () => countAnthropicInputTokens,
+  countGeminiTokens: () => countGeminiTokens,
+  countGemmaSentencePieceTokens: () => countGemmaSentencePieceTokens,
   countTokens: () => countTokens,
   decode: () => decode,
   encode: () => encode,
   estimate: () => estimate,
+  estimateAsync: () => estimateAsync,
   getAvailableModels: () => getAvailableModels,
   getModelConfig: () => getModelConfig
 });
@@ -394,14 +408,21 @@ var models = {
 Object.values(models).forEach((config) => Object.freeze(config));
 var DEFAULT_MODELS = Object.freeze(models);
 function getModelConfig(model) {
-  const config = DEFAULT_MODELS[model];
-  if (!config) {
+  const direct = DEFAULT_MODELS[model];
+  if (direct) return direct;
+  const normalized = (() => {
+    if (!model.startsWith("claude-")) return model;
+    const withoutDate = model.replace(/-\d{8}$/, "");
+    return withoutDate.replace(/-(\d+)-(\d+)$/, (_m, major, minor) => `-${major}.${minor}`);
+  })();
+  const aliased = DEFAULT_MODELS[normalized];
+  if (!aliased) {
     const available = Object.keys(DEFAULT_MODELS).join(", ");
     throw new Error(
       `Unknown model: "${model}". Available models: ${available}`
     );
   }
-  return config;
+  return aliased;
 }
 function getAvailableModels() {
   return Object.keys(DEFAULT_MODELS);
@@ -491,13 +512,17 @@ function countCodePoints(text) {
 function estimate(input) {
   const { text, model, rounding = "ceil", tokenizer = "heuristic" } = input;
   const config = getModelConfig(model);
+  const tokenizerStr = tokenizer;
+  if (tokenizerStr === "anthropic_count_tokens" || tokenizerStr === "gemini_count_tokens" || tokenizerStr === "gemma_sentencepiece") {
+    throw new Error(`Tokenizer mode "${tokenizerStr}" requires async execution. Use estimateAsync(...) instead.`);
+  }
   const characterCount = countCodePoints(text);
-  const isNonOpenAIModel2 = model.startsWith("claude-") || model.startsWith("gemini-");
+  const isNonOpenAIModel3 = model.startsWith("claude-") || model.startsWith("gemini-");
   let estimatedTokens;
   let tokenizerModeUsed = "heuristic";
   let encodingUsed;
   const shouldTryExact = tokenizer === "openai_exact" || tokenizer === "auto";
-  if (shouldTryExact && !isNonOpenAIModel2) {
+  if (shouldTryExact && !isNonOpenAIModel3) {
     try {
       estimatedTokens = encode(text, { model, allowSpecial: "none" }).length;
       tokenizerModeUsed = "openai_exact";
@@ -507,7 +532,7 @@ function estimate(input) {
         throw error;
       }
     }
-  } else if (tokenizer === "openai_exact" && isNonOpenAIModel2) {
+  } else if (tokenizer === "openai_exact" && isNonOpenAIModel3) {
     throw new Error(
       `Tokenizer mode "openai_exact" requested for non-OpenAI model: "${model}"`
     );
@@ -539,13 +564,283 @@ function estimate(input) {
   };
 }
 
-// src/token-counter.ts
+// src/providers/anthropic.ts
+function getFetch(fetchImpl) {
+  const f = fetchImpl ?? globalThis.fetch;
+  if (!f) {
+    throw new Error("globalThis.fetch is not available; pass fetch in AnthropicCountTokensParams");
+  }
+  return f;
+}
+function withStatus(message, status) {
+  const err = new Error(message);
+  err.status = status;
+  return err;
+}
+function getApiKey(explicit) {
+  const key = explicit ?? (typeof process !== "undefined" ? process.env.ANTHROPIC_API_KEY : void 0);
+  if (!key) throw withStatus("Anthropic API key missing (set ANTHROPIC_API_KEY or pass apiKey)", 401);
+  return key;
+}
+function asRecord(value) {
+  if (!value || typeof value !== "object" || Array.isArray(value)) return null;
+  return value;
+}
+async function countAnthropicInputTokens(params) {
+  const fetchImpl = getFetch(params.fetch);
+  const apiKey = getApiKey(params.apiKey);
+  const baseUrl = (params.baseUrl ?? "https://api.anthropic.com").replace(/\/+$/, "");
+  const version = params.version ?? "2023-06-01";
+  const messages = params.messages ?? (typeof params.text === "string" ? [{ role: "user", content: params.text }] : null);
+  if (!messages) {
+    throw new Error("Anthropic token counting requires either `messages` or `text`");
+  }
+  const body = {
+    model: params.model,
+    messages
+  };
+  if (typeof params.system === "string" && params.system.trim()) {
+    body.system = params.system;
+  }
+  const response = await fetchImpl(`${baseUrl}/v1/messages/count_tokens`, {
+    method: "POST",
+    headers: {
+      "content-type": "application/json",
+      "x-api-key": apiKey,
+      "anthropic-version": version
+    },
+    body: JSON.stringify(body)
+  });
+  const text = await response.text();
+  let data = null;
+  try {
+    data = text ? JSON.parse(text) : null;
+  } catch {
+  }
+  const dataObj = asRecord(data);
+  if (!response.ok) {
+    const errorObj = asRecord(dataObj?.error);
+    const msg = typeof errorObj?.message === "string" ? errorObj.message : typeof dataObj?.message === "string" ? dataObj.message : `HTTP ${response.status}`;
+    throw withStatus(`Anthropic count_tokens failed: ${msg}`, response.status);
+  }
+  const inputTokens = dataObj?.input_tokens;
+  if (typeof inputTokens !== "number" || !Number.isFinite(inputTokens) || inputTokens < 0) {
+    throw new Error("Anthropic count_tokens returned invalid input_tokens");
+  }
+  return inputTokens;
+}
+
+// src/providers/gemini.ts
+function getFetch2(fetchImpl) {
+  const f = fetchImpl ?? globalThis.fetch;
+  if (!f) {
+    throw new Error("globalThis.fetch is not available; pass fetch in GeminiCountTokensParams");
+  }
+  return f;
+}
+function withStatus2(message, status) {
+  const err = new Error(message);
+  err.status = status;
+  return err;
+}
+function getApiKey2(explicit) {
+  const key = explicit ?? (typeof process !== "undefined" ? process.env.GEMINI_API_KEY : void 0);
+  if (!key) throw withStatus2("Gemini API key missing (set GEMINI_API_KEY or pass apiKey)", 401);
+  return key;
+}
+function toContents(text) {
+  return [{ role: "user", parts: [{ text }] }];
+}
+function asRecord2(value) {
+  if (!value || typeof value !== "object" || Array.isArray(value)) return null;
+  return value;
+}
+async function countGeminiTokens(params) {
+  const fetchImpl = getFetch2(params.fetch);
+  const apiKey = getApiKey2(params.apiKey);
+  const baseUrl = (params.baseUrl ?? "https://generativelanguage.googleapis.com").replace(/\/+$/, "");
+  const contents = params.contents ?? (typeof params.text === "string" ? toContents(params.text) : null);
+  if (!contents) {
+    throw new Error("Gemini token counting requires either `contents` or `text`");
+  }
+  const url = `${baseUrl}/v1beta/models/${encodeURIComponent(params.model)}:countTokens?key=${encodeURIComponent(apiKey)}`;
+  const response = await fetchImpl(url, {
+    method: "POST",
+    headers: { "content-type": "application/json" },
+    body: JSON.stringify({ contents })
+  });
+  const text = await response.text();
+  let data = null;
+  try {
+    data = text ? JSON.parse(text) : null;
+  } catch {
+  }
+  const dataObj = asRecord2(data);
+  if (!response.ok) {
+    const errorObj = asRecord2(dataObj?.error);
+    const msg = typeof errorObj?.message === "string" ? errorObj.message : typeof dataObj?.message === "string" ? dataObj.message : `HTTP ${response.status}`;
+    throw withStatus2(`Gemini countTokens failed: ${msg}`, response.status);
+  }
+  const totalTokens = dataObj?.totalTokens ?? dataObj?.total_tokens ?? dataObj?.total_tokens_count;
+  if (typeof totalTokens !== "number" || !Number.isFinite(totalTokens) || totalTokens < 0) {
+    throw new Error("Gemini countTokens returned invalid totalTokens");
+  }
+  return totalTokens;
+}
+
+// src/providers/gemma-sentencepiece.ts
+async function loadSentencePiece() {
+  try {
+    const mod = await import("sentencepiece-js");
+    if (mod.SentencePieceProcessor || mod.cleanText) return mod;
+    if (mod.default && typeof mod.default === "object" && mod.default.SentencePieceProcessor) {
+      return mod.default;
+    }
+    return mod;
+  } catch {
+    throw new Error(
+      "Local Gemma SentencePiece tokenization requires the optional dependency `sentencepiece-js`. Install it and try again."
+    );
+  }
+}
+async function countGemmaSentencePieceTokens(params) {
+  const sp = await loadSentencePiece();
+  const defaults = (sp.default && typeof sp.default === "object" ? sp.default : null) ?? {};
+  const SentencePieceProcessor = sp.SentencePieceProcessor ?? defaults.SentencePieceProcessor;
+  const cleanText = sp.cleanText ?? defaults.cleanText;
+  if (!SentencePieceProcessor || typeof SentencePieceProcessor !== "function") {
+    throw new Error("sentencepiece-js did not export SentencePieceProcessor as expected");
+  }
+  const processor = new SentencePieceProcessor();
+  const loaded = processor.load(params.modelPath);
+  if (loaded instanceof Promise) await loaded;
+  const cleaned = typeof cleanText === "function" ? cleanText(params.text) : params.text;
+  const ids = processor.encodeIds(cleaned);
+  if (!Array.isArray(ids)) {
+    throw new Error("sentencepiece-js returned invalid ids from encodeIds");
+  }
+  return ids.length;
+}
+
+// src/estimator-async.ts
+function countCodePoints2(text) {
+  let count = 0;
+  for (const _char of text) count++;
+  return count;
+}
 function isNonOpenAIModel(model) {
   return model.startsWith("claude-") || model.startsWith("gemini-");
 }
+function shouldFallbackToHeuristic(err) {
+  if (!err) return true;
+  const maybe = err;
+  const statusRaw = maybe.status;
+  const status = typeof statusRaw === "number" && Number.isFinite(statusRaw) ? statusRaw : null;
+  if (!status) return true;
+  if (status === 401 || status === 403 || status === 429) return true;
+  if (status >= 500 && status <= 599) return true;
+  return false;
+}
+async function estimateAsync(input) {
+  const { text, model, rounding = "ceil", tokenizer = "heuristic" } = input;
+  const config = getModelConfig(model);
+  const characterCount = countCodePoints2(text);
+  let estimatedTokens;
+  let tokenizerModeUsed = "heuristic";
+  let encodingUsed;
+  if (tokenizer === "anthropic_count_tokens") {
+    try {
+      estimatedTokens = await countAnthropicInputTokens({
+        model,
+        text,
+        system: input.anthropic?.system,
+        apiKey: input.anthropic?.apiKey,
+        baseUrl: input.anthropic?.baseUrl,
+        version: input.anthropic?.version,
+        fetch: input.fetch
+      });
+      tokenizerModeUsed = "anthropic_count_tokens";
+    } catch (error) {
+      if (input.fallbackToHeuristicOnError && shouldFallbackToHeuristic(error)) {
+        estimatedTokens = void 0;
+        tokenizerModeUsed = "heuristic";
+      } else {
+        throw error;
+      }
+    }
+  } else if (tokenizer === "gemini_count_tokens") {
+    try {
+      estimatedTokens = await countGeminiTokens({
+        model,
+        text,
+        apiKey: input.gemini?.apiKey,
+        baseUrl: input.gemini?.baseUrl,
+        fetch: input.fetch
+      });
+      tokenizerModeUsed = "gemini_count_tokens";
+    } catch (error) {
+      if (input.fallbackToHeuristicOnError && shouldFallbackToHeuristic(error)) {
+        estimatedTokens = void 0;
+        tokenizerModeUsed = "heuristic";
+      } else {
+        throw error;
+      }
+    }
+  } else if (tokenizer === "gemma_sentencepiece") {
+    const modelPath = input.gemma?.modelPath;
+    if (!modelPath) {
+      throw new Error("gemma_sentencepiece tokenizer requires gemma.modelPath (path to tokenizer.model)");
+    }
+    estimatedTokens = await countGemmaSentencePieceTokens({ modelPath, text });
+    tokenizerModeUsed = "gemma_sentencepiece";
+  } else {
+    const shouldTryExact = tokenizer === "openai_exact" || tokenizer === "auto";
+    if (shouldTryExact && !isNonOpenAIModel(model)) {
+      try {
+        estimatedTokens = encode(text, { model, allowSpecial: "none" }).length;
+        tokenizerModeUsed = "openai_exact";
+        encodingUsed = getOpenAIEncoding({ model });
+      } catch (error) {
+        if (tokenizer === "openai_exact") throw error;
+      }
+    } else if (tokenizer === "openai_exact" && isNonOpenAIModel(model)) {
+      throw new Error(`Tokenizer mode "openai_exact" requested for non-OpenAI model: "${model}"`);
+    }
+  }
+  if (estimatedTokens === void 0) {
+    const rawTokens = characterCount / config.charsPerToken;
+    switch (rounding) {
+      case "floor":
+        estimatedTokens = Math.floor(rawTokens);
+        break;
+      case "round":
+        estimatedTokens = Math.round(rawTokens);
+        break;
+      case "ceil":
+      default:
+        estimatedTokens = Math.ceil(rawTokens);
+    }
+    tokenizerModeUsed = "heuristic";
+  }
+  const estimatedInputCost = estimatedTokens * config.inputCostPerMillion / 1e6;
+  return {
+    model,
+    characterCount,
+    estimatedTokens,
+    estimatedInputCost,
+    charsPerToken: config.charsPerToken,
+    tokenizerMode: tokenizerModeUsed,
+    encodingUsed
+  };
+}
+
+// src/token-counter.ts
+function isNonOpenAIModel2(model) {
+  return model.startsWith("claude-") || model.startsWith("gemini-");
+}
 function countTokens(input) {
   const { text, model } = input;
-  if (isNonOpenAIModel(model)) {
+  if (isNonOpenAIModel2(model)) {
     return {
       tokens: estimate({ text, model }).estimatedTokens,
       exact: false
@@ -568,10 +863,14 @@ function countTokens(input) {
 0 && (module.exports = {
   DEFAULT_MODELS,
   LAST_UPDATED,
+  countAnthropicInputTokens,
+  countGeminiTokens,
+  countGemmaSentencePieceTokens,
   countTokens,
   decode,
   encode,
   estimate,
+  estimateAsync,
   getAvailableModels,
   getModelConfig
 });

package/dist/index.d.cts

@@ -8,6 +8,13 @@ interface ModelConfig {
     inputCostPerMillion: number;
 }
 type TokenizerMode = 'heuristic' | 'openai_exact' | 'auto';
+/**
+ * Tokenizer modes supported by `estimateAsync(...)`.
+ *
+ * This is intentionally separate from `TokenizerMode` to avoid breaking
+ * TypeScript users who exhaustively switch on the legacy `TokenizerMode` union.
+ */
+type TokenizerModeAsync = TokenizerMode | 'anthropic_count_tokens' | 'gemini_count_tokens' | 'gemma_sentencepiece';
 /**
  * Input parameters for the estimate function.
  */
@@ -26,6 +33,53 @@ interface EstimateInput {
      */
     tokenizer?: TokenizerMode;
 }
+interface EstimateAsyncInput extends Omit<EstimateInput, 'tokenizer'> {
+    /**
+     * Token counting strategy for async estimation.
+     * Includes provider-backed modes that require network access or local model files.
+     */
+    tokenizer?: TokenizerModeAsync;
+    /**
+     * Optional fetch implementation (useful for tests, edge runtimes, or custom fetch).
+     * Defaults to globalThis.fetch.
+     */
+    fetch?: typeof fetch;
+    /**
+     * If true, provider-backed tokenizer modes will fall back to heuristic token estimation
+     * when the provider API is throttled/unavailable or the API key is invalid.
+     *
+     * This never stores API keys; it only affects error handling.
+     *
+     * Default: false (throw on provider errors)
+     */
+    fallbackToHeuristicOnError?: boolean;
+    /**
+     * Configuration for Anthropic token counting.
+     * Only used when tokenizer === 'anthropic_count_tokens'.
+     */
+    anthropic?: {
+        apiKey?: string;
+        baseUrl?: string;
+        version?: string;
+        system?: string;
+    };
+    /**
+     * Configuration for Gemini token counting (Google AI Studio / Generative Language API).
+     * Only used when tokenizer === 'gemini_count_tokens'.
+     */
+    gemini?: {
+        apiKey?: string;
+        baseUrl?: string;
+    };
+    /**
+     * Configuration for local Gemma SentencePiece tokenization.
+     * Only used when tokenizer === 'gemma_sentencepiece'.
+     */
+    gemma?: {
+        /** Filesystem path to a SentencePiece model file (e.g. Gemma tokenizer.model). */
+        modelPath?: string;
+    };
+}
 /**
  * Output from the estimate function.
  */
@@ -41,7 +95,7 @@ interface EstimateOutput {
     /** The chars-per-token ratio used */
     charsPerToken: number;
     /** Which tokenizer strategy was used */
-    tokenizerMode?: TokenizerMode;
+    tokenizerMode?: TokenizerModeAsync;
     /** OpenAI encoding used when tokenizerMode is `openai_exact` */
     encodingUsed?: string;
 }
@@ -65,6 +119,8 @@ interface EstimateOutput {
  */
 declare function estimate(input: EstimateInput): EstimateOutput;
 
+declare function estimateAsync(input: EstimateAsyncInput): Promise<EstimateOutput>;
+
 /**
  * Default model configurations.
  *
@@ -141,4 +197,47 @@ interface TokenCountOutput {
  */
 declare function countTokens(input: TokenCountInput): TokenCountOutput;
 
-export { DEFAULT_MODELS, type EncodeOptions, type EstimateInput, type EstimateOutput, LAST_UPDATED, type ModelConfig, type OpenAIEncoding, type SpecialTokenHandling, type TokenCountInput, type TokenCountOutput, type TokenizerMode, countTokens, decode, encode, estimate, getAvailableModels, getModelConfig };
+interface AnthropicCountTokensParams {
+    /** Claude model id, e.g. `claude-sonnet-4-5` */
+    model: string;
+    /** Anthropic API key. If omitted, uses process.env.ANTHROPIC_API_KEY */
+    apiKey?: string;
+    /** Text-only helper; converted into a single user message. */
+    text?: string;
+    /** Optional system prompt. */
+    system?: string;
+    /** Full messages payload (wins over `text` when provided). */
+    messages?: unknown;
+    /** Override API base URL (default: https://api.anthropic.com) */
+    baseUrl?: string;
+    /** Override Anthropic version header (default: 2023-06-01) */
+    version?: string;
+    /** Optional fetch implementation. Defaults to globalThis.fetch. */
+    fetch?: typeof fetch;
+}
+declare function countAnthropicInputTokens(params: AnthropicCountTokensParams): Promise<number>;
+
+interface GeminiCountTokensParams {
+    /** Gemini model id, e.g. `gemini-2.0-flash` */
+    model: string;
+    /** Gemini API key. If omitted, uses process.env.GEMINI_API_KEY */
+    apiKey?: string;
+    /** Text-only helper; converted into a basic `contents` payload. */
+    text?: string;
+    /** Full `contents` payload (wins over `text` when provided). */
+    contents?: unknown;
+    /** Override API base URL (default: https://generativelanguage.googleapis.com) */
+    baseUrl?: string;
+    /** Optional fetch implementation. Defaults to globalThis.fetch. */
+    fetch?: typeof fetch;
+}
+declare function countGeminiTokens(params: GeminiCountTokensParams): Promise<number>;
+
+interface GemmaSentencePieceCountTokensParams {
+    /** Filesystem path to a SentencePiece model file (e.g. Gemma `tokenizer.model`). */
+    modelPath: string;
+    text: string;
+}
+declare function countGemmaSentencePieceTokens(params: GemmaSentencePieceCountTokensParams): Promise<number>;
+
+export { type AnthropicCountTokensParams, DEFAULT_MODELS, type EncodeOptions, type EstimateAsyncInput, type EstimateInput, type EstimateOutput, type GeminiCountTokensParams, type GemmaSentencePieceCountTokensParams, LAST_UPDATED, type ModelConfig, type OpenAIEncoding, type SpecialTokenHandling, type TokenCountInput, type TokenCountOutput, type TokenizerMode, type TokenizerModeAsync, countAnthropicInputTokens, countGeminiTokens, countGemmaSentencePieceTokens, countTokens, decode, encode, estimate, estimateAsync, getAvailableModels, getModelConfig };

package/dist/index.d.ts

@@ -8,6 +8,13 @@ interface ModelConfig {
     inputCostPerMillion: number;
 }
 type TokenizerMode = 'heuristic' | 'openai_exact' | 'auto';
+/**
+ * Tokenizer modes supported by `estimateAsync(...)`.
+ *
+ * This is intentionally separate from `TokenizerMode` to avoid breaking
+ * TypeScript users who exhaustively switch on the legacy `TokenizerMode` union.
+ */
+type TokenizerModeAsync = TokenizerMode | 'anthropic_count_tokens' | 'gemini_count_tokens' | 'gemma_sentencepiece';
 /**
  * Input parameters for the estimate function.
  */
@@ -26,6 +33,53 @@ interface EstimateInput {
      */
     tokenizer?: TokenizerMode;
 }
+interface EstimateAsyncInput extends Omit<EstimateInput, 'tokenizer'> {
+    /**
+     * Token counting strategy for async estimation.
+     * Includes provider-backed modes that require network access or local model files.
+     */
+    tokenizer?: TokenizerModeAsync;
+    /**
+     * Optional fetch implementation (useful for tests, edge runtimes, or custom fetch).
+     * Defaults to globalThis.fetch.
+     */
+    fetch?: typeof fetch;
+    /**
+     * If true, provider-backed tokenizer modes will fall back to heuristic token estimation
+     * when the provider API is throttled/unavailable or the API key is invalid.
+     *
+     * This never stores API keys; it only affects error handling.
+     *
+     * Default: false (throw on provider errors)
+     */
+    fallbackToHeuristicOnError?: boolean;
+    /**
+     * Configuration for Anthropic token counting.
+     * Only used when tokenizer === 'anthropic_count_tokens'.
+     */
+    anthropic?: {
+        apiKey?: string;
+        baseUrl?: string;
+        version?: string;
+        system?: string;
+    };
+    /**
+     * Configuration for Gemini token counting (Google AI Studio / Generative Language API).
+     * Only used when tokenizer === 'gemini_count_tokens'.
+     */
+    gemini?: {
+        apiKey?: string;
+        baseUrl?: string;
+    };
+    /**
+     * Configuration for local Gemma SentencePiece tokenization.
+     * Only used when tokenizer === 'gemma_sentencepiece'.
+     */
+    gemma?: {
+        /** Filesystem path to a SentencePiece model file (e.g. Gemma tokenizer.model). */
+        modelPath?: string;
+    };
+}
 /**
  * Output from the estimate function.
  */
@@ -41,7 +95,7 @@ interface EstimateOutput {
     /** The chars-per-token ratio used */
     charsPerToken: number;
     /** Which tokenizer strategy was used */
-    tokenizerMode?: TokenizerMode;
+    tokenizerMode?: TokenizerModeAsync;
     /** OpenAI encoding used when tokenizerMode is `openai_exact` */
     encodingUsed?: string;
 }
@@ -65,6 +119,8 @@ interface EstimateOutput {
  */
 declare function estimate(input: EstimateInput): EstimateOutput;
 
+declare function estimateAsync(input: EstimateAsyncInput): Promise<EstimateOutput>;
+
 /**
  * Default model configurations.
  *
@@ -141,4 +197,47 @@ interface TokenCountOutput {
  */
 declare function countTokens(input: TokenCountInput): TokenCountOutput;
 
-export { DEFAULT_MODELS, type EncodeOptions, type EstimateInput, type EstimateOutput, LAST_UPDATED, type ModelConfig, type OpenAIEncoding, type SpecialTokenHandling, type TokenCountInput, type TokenCountOutput, type TokenizerMode, countTokens, decode, encode, estimate, getAvailableModels, getModelConfig };
+interface AnthropicCountTokensParams {
+    /** Claude model id, e.g. `claude-sonnet-4-5` */
+    model: string;
+    /** Anthropic API key. If omitted, uses process.env.ANTHROPIC_API_KEY */
+    apiKey?: string;
+    /** Text-only helper; converted into a single user message. */
+    text?: string;
+    /** Optional system prompt. */
+    system?: string;
+    /** Full messages payload (wins over `text` when provided). */
+    messages?: unknown;
+    /** Override API base URL (default: https://api.anthropic.com) */
+    baseUrl?: string;
+    /** Override Anthropic version header (default: 2023-06-01) */
+    version?: string;
+    /** Optional fetch implementation. Defaults to globalThis.fetch. */
+    fetch?: typeof fetch;
+}
+declare function countAnthropicInputTokens(params: AnthropicCountTokensParams): Promise<number>;
+
+interface GeminiCountTokensParams {
+    /** Gemini model id, e.g. `gemini-2.0-flash` */
+    model: string;
+    /** Gemini API key. If omitted, uses process.env.GEMINI_API_KEY */
+    apiKey?: string;
+    /** Text-only helper; converted into a basic `contents` payload. */
+    text?: string;
+    /** Full `contents` payload (wins over `text` when provided). */
+    contents?: unknown;
+    /** Override API base URL (default: https://generativelanguage.googleapis.com) */
+    baseUrl?: string;
+    /** Optional fetch implementation. Defaults to globalThis.fetch. */
+    fetch?: typeof fetch;
+}
+declare function countGeminiTokens(params: GeminiCountTokensParams): Promise<number>;
+
+interface GemmaSentencePieceCountTokensParams {
+    /** Filesystem path to a SentencePiece model file (e.g. Gemma `tokenizer.model`). */
+    modelPath: string;
+    text: string;
+}
+declare function countGemmaSentencePieceTokens(params: GemmaSentencePieceCountTokensParams): Promise<number>;
+
+export { type AnthropicCountTokensParams, DEFAULT_MODELS, type EncodeOptions, type EstimateAsyncInput, type EstimateInput, type EstimateOutput, type GeminiCountTokensParams, type GemmaSentencePieceCountTokensParams, LAST_UPDATED, type ModelConfig, type OpenAIEncoding, type SpecialTokenHandling, type TokenCountInput, type TokenCountOutput, type TokenizerMode, type TokenizerModeAsync, countAnthropicInputTokens, countGeminiTokens, countGemmaSentencePieceTokens, countTokens, decode, encode, estimate, estimateAsync, getAvailableModels, getModelConfig };

package/dist/index.js

@@ -361,14 +361,21 @@ var models = {
 Object.values(models).forEach((config) => Object.freeze(config));
 var DEFAULT_MODELS = Object.freeze(models);
 function getModelConfig(model) {
-  const config = DEFAULT_MODELS[model];
-  if (!config) {
+  const direct = DEFAULT_MODELS[model];
+  if (direct) return direct;
+  const normalized = (() => {
+    if (!model.startsWith("claude-")) return model;
+    const withoutDate = model.replace(/-\d{8}$/, "");
+    return withoutDate.replace(/-(\d+)-(\d+)$/, (_m, major, minor) => `-${major}.${minor}`);
+  })();
+  const aliased = DEFAULT_MODELS[normalized];
+  if (!aliased) {
     const available = Object.keys(DEFAULT_MODELS).join(", ");
     throw new Error(
       `Unknown model: "${model}". Available models: ${available}`
     );
   }
-  return config;
+  return aliased;
 }
 function getAvailableModels() {
   return Object.keys(DEFAULT_MODELS);
@@ -457,13 +464,17 @@ function countCodePoints(text) {
 function estimate(input) {
   const { text, model, rounding = "ceil", tokenizer = "heuristic" } = input;
   const config = getModelConfig(model);
+  const tokenizerStr = tokenizer;
+  if (tokenizerStr === "anthropic_count_tokens" || tokenizerStr === "gemini_count_tokens" || tokenizerStr === "gemma_sentencepiece") {
+    throw new Error(`Tokenizer mode "${tokenizerStr}" requires async execution. Use estimateAsync(...) instead.`);
+  }
   const characterCount = countCodePoints(text);
-  const isNonOpenAIModel2 = model.startsWith("claude-") || model.startsWith("gemini-");
+  const isNonOpenAIModel3 = model.startsWith("claude-") || model.startsWith("gemini-");
   let estimatedTokens;
   let tokenizerModeUsed = "heuristic";
   let encodingUsed;
   const shouldTryExact = tokenizer === "openai_exact" || tokenizer === "auto";
-  if (shouldTryExact && !isNonOpenAIModel2) {
+  if (shouldTryExact && !isNonOpenAIModel3) {
     try {
       estimatedTokens = encode(text, { model, allowSpecial: "none" }).length;
       tokenizerModeUsed = "openai_exact";
@@ -473,7 +484,7 @@ function estimate(input) {
         throw error;
       }
     }
-  } else if (tokenizer === "openai_exact" && isNonOpenAIModel2) {
+  } else if (tokenizer === "openai_exact" && isNonOpenAIModel3) {
     throw new Error(
       `Tokenizer mode "openai_exact" requested for non-OpenAI model: "${model}"`
     );
@@ -505,13 +516,283 @@ function estimate(input) {
   };
 }
 
-// src/token-counter.ts
+// src/providers/anthropic.ts
+function getFetch(fetchImpl) {
+  const f = fetchImpl ?? globalThis.fetch;
+  if (!f) {
+    throw new Error("globalThis.fetch is not available; pass fetch in AnthropicCountTokensParams");
+  }
+  return f;
+}
+function withStatus(message, status) {
+  const err = new Error(message);
+  err.status = status;
+  return err;
+}
+function getApiKey(explicit) {
+  const key = explicit ?? (typeof process !== "undefined" ? process.env.ANTHROPIC_API_KEY : void 0);
+  if (!key) throw withStatus("Anthropic API key missing (set ANTHROPIC_API_KEY or pass apiKey)", 401);
+  return key;
+}
+function asRecord(value) {
+  if (!value || typeof value !== "object" || Array.isArray(value)) return null;
+  return value;
+}
+async function countAnthropicInputTokens(params) {
+  const fetchImpl = getFetch(params.fetch);
+  const apiKey = getApiKey(params.apiKey);
+  const baseUrl = (params.baseUrl ?? "https://api.anthropic.com").replace(/\/+$/, "");
+  const version = params.version ?? "2023-06-01";
+  const messages = params.messages ?? (typeof params.text === "string" ? [{ role: "user", content: params.text }] : null);
+  if (!messages) {
+    throw new Error("Anthropic token counting requires either `messages` or `text`");
+  }
+  const body = {
+    model: params.model,
+    messages
+  };
+  if (typeof params.system === "string" && params.system.trim()) {
+    body.system = params.system;
+  }
+  const response = await fetchImpl(`${baseUrl}/v1/messages/count_tokens`, {
+    method: "POST",
+    headers: {
+      "content-type": "application/json",
+      "x-api-key": apiKey,
+      "anthropic-version": version
+    },
+    body: JSON.stringify(body)
+  });
+  const text = await response.text();
+  let data = null;
+  try {
+    data = text ? JSON.parse(text) : null;
+  } catch {
+  }
+  const dataObj = asRecord(data);
+  if (!response.ok) {
+    const errorObj = asRecord(dataObj?.error);
+    const msg = typeof errorObj?.message === "string" ? errorObj.message : typeof dataObj?.message === "string" ? dataObj.message : `HTTP ${response.status}`;
+    throw withStatus(`Anthropic count_tokens failed: ${msg}`, response.status);
+  }
+  const inputTokens = dataObj?.input_tokens;
+  if (typeof inputTokens !== "number" || !Number.isFinite(inputTokens) || inputTokens < 0) {
+    throw new Error("Anthropic count_tokens returned invalid input_tokens");
+  }
+  return inputTokens;
+}
+
+// src/providers/gemini.ts
+function getFetch2(fetchImpl) {
+  const f = fetchImpl ?? globalThis.fetch;
+  if (!f) {
+    throw new Error("globalThis.fetch is not available; pass fetch in GeminiCountTokensParams");
+  }
+  return f;
+}
+function withStatus2(message, status) {
+  const err = new Error(message);
+  err.status = status;
+  return err;
+}
+function getApiKey2(explicit) {
+  const key = explicit ?? (typeof process !== "undefined" ? process.env.GEMINI_API_KEY : void 0);
+  if (!key) throw withStatus2("Gemini API key missing (set GEMINI_API_KEY or pass apiKey)", 401);
+  return key;
+}
+function toContents(text) {
+  return [{ role: "user", parts: [{ text }] }];
+}
+function asRecord2(value) {
+  if (!value || typeof value !== "object" || Array.isArray(value)) return null;
+  return value;
+}
+async function countGeminiTokens(params) {
+  const fetchImpl = getFetch2(params.fetch);
+  const apiKey = getApiKey2(params.apiKey);
+  const baseUrl = (params.baseUrl ?? "https://generativelanguage.googleapis.com").replace(/\/+$/, "");
+  const contents = params.contents ?? (typeof params.text === "string" ? toContents(params.text) : null);
+  if (!contents) {
+    throw new Error("Gemini token counting requires either `contents` or `text`");
+  }
+  const url = `${baseUrl}/v1beta/models/${encodeURIComponent(params.model)}:countTokens?key=${encodeURIComponent(apiKey)}`;
+  const response = await fetchImpl(url, {
+    method: "POST",
+    headers: { "content-type": "application/json" },
+    body: JSON.stringify({ contents })
+  });
+  const text = await response.text();
+  let data = null;
+  try {
+    data = text ? JSON.parse(text) : null;
+  } catch {
+  }
+  const dataObj = asRecord2(data);
+  if (!response.ok) {
+    const errorObj = asRecord2(dataObj?.error);
+    const msg = typeof errorObj?.message === "string" ? errorObj.message : typeof dataObj?.message === "string" ? dataObj.message : `HTTP ${response.status}`;
+    throw withStatus2(`Gemini countTokens failed: ${msg}`, response.status);
+  }
+  const totalTokens = dataObj?.totalTokens ?? dataObj?.total_tokens ?? dataObj?.total_tokens_count;
+  if (typeof totalTokens !== "number" || !Number.isFinite(totalTokens) || totalTokens < 0) {
+    throw new Error("Gemini countTokens returned invalid totalTokens");
+  }
+  return totalTokens;
+}
+
+// src/providers/gemma-sentencepiece.ts
+async function loadSentencePiece() {
+  try {
+    const mod = await import("sentencepiece-js");
+    if (mod.SentencePieceProcessor || mod.cleanText) return mod;
+    if (mod.default && typeof mod.default === "object" && mod.default.SentencePieceProcessor) {
+      return mod.default;
+    }
+    return mod;
+  } catch {
+    throw new Error(
+      "Local Gemma SentencePiece tokenization requires the optional dependency `sentencepiece-js`. Install it and try again."
+    );
+  }
+}
+async function countGemmaSentencePieceTokens(params) {
+  const sp = await loadSentencePiece();
+  const defaults = (sp.default && typeof sp.default === "object" ? sp.default : null) ?? {};
+  const SentencePieceProcessor = sp.SentencePieceProcessor ?? defaults.SentencePieceProcessor;
+  const cleanText = sp.cleanText ?? defaults.cleanText;
+  if (!SentencePieceProcessor || typeof SentencePieceProcessor !== "function") {
+    throw new Error("sentencepiece-js did not export SentencePieceProcessor as expected");
+  }
+  const processor = new SentencePieceProcessor();
+  const loaded = processor.load(params.modelPath);
+  if (loaded instanceof Promise) await loaded;
+  const cleaned = typeof cleanText === "function" ? cleanText(params.text) : params.text;
+  const ids = processor.encodeIds(cleaned);
+  if (!Array.isArray(ids)) {
+    throw new Error("sentencepiece-js returned invalid ids from encodeIds");
+  }
+  return ids.length;
+}
+
+// src/estimator-async.ts
+function countCodePoints2(text) {
+  let count = 0;
+  for (const _char of text) count++;
+  return count;
+}
 function isNonOpenAIModel(model) {
   return model.startsWith("claude-") || model.startsWith("gemini-");
 }
+function shouldFallbackToHeuristic(err) {
+  if (!err) return true;
+  const maybe = err;
+  const statusRaw = maybe.status;
+  const status = typeof statusRaw === "number" && Number.isFinite(statusRaw) ? statusRaw : null;
+  if (!status) return true;
+  if (status === 401 || status === 403 || status === 429) return true;
+  if (status >= 500 && status <= 599) return true;
+  return false;
+}
+async function estimateAsync(input) {
+  const { text, model, rounding = "ceil", tokenizer = "heuristic" } = input;
+  const config = getModelConfig(model);
+  const characterCount = countCodePoints2(text);
+  let estimatedTokens;
+  let tokenizerModeUsed = "heuristic";
+  let encodingUsed;
+  if (tokenizer === "anthropic_count_tokens") {
+    try {
+      estimatedTokens = await countAnthropicInputTokens({
+        model,
+        text,
+        system: input.anthropic?.system,
+        apiKey: input.anthropic?.apiKey,
+        baseUrl: input.anthropic?.baseUrl,
+        version: input.anthropic?.version,
+        fetch: input.fetch
+      });
+      tokenizerModeUsed = "anthropic_count_tokens";
+    } catch (error) {
+      if (input.fallbackToHeuristicOnError && shouldFallbackToHeuristic(error)) {
+        estimatedTokens = void 0;
+        tokenizerModeUsed = "heuristic";
+      } else {
+        throw error;
+      }
+    }
+  } else if (tokenizer === "gemini_count_tokens") {
+    try {
+      estimatedTokens = await countGeminiTokens({
+        model,
+        text,
+        apiKey: input.gemini?.apiKey,
+        baseUrl: input.gemini?.baseUrl,
+        fetch: input.fetch
+      });
+      tokenizerModeUsed = "gemini_count_tokens";
+    } catch (error) {
+      if (input.fallbackToHeuristicOnError && shouldFallbackToHeuristic(error)) {
+        estimatedTokens = void 0;
+        tokenizerModeUsed = "heuristic";
+      } else {
+        throw error;
+      }
+    }
+  } else if (tokenizer === "gemma_sentencepiece") {
+    const modelPath = input.gemma?.modelPath;
+    if (!modelPath) {
+      throw new Error("gemma_sentencepiece tokenizer requires gemma.modelPath (path to tokenizer.model)");
+    }
+    estimatedTokens = await countGemmaSentencePieceTokens({ modelPath, text });
+    tokenizerModeUsed = "gemma_sentencepiece";
+  } else {
+    const shouldTryExact = tokenizer === "openai_exact" || tokenizer === "auto";
+    if (shouldTryExact && !isNonOpenAIModel(model)) {
+      try {
+        estimatedTokens = encode(text, { model, allowSpecial: "none" }).length;
+        tokenizerModeUsed = "openai_exact";
+        encodingUsed = getOpenAIEncoding({ model });
+      } catch (error) {
+        if (tokenizer === "openai_exact") throw error;
+      }
+    } else if (tokenizer === "openai_exact" && isNonOpenAIModel(model)) {
+      throw new Error(`Tokenizer mode "openai_exact" requested for non-OpenAI model: "${model}"`);
+    }
+  }
+  if (estimatedTokens === void 0) {
+    const rawTokens = characterCount / config.charsPerToken;
+    switch (rounding) {
+      case "floor":
+        estimatedTokens = Math.floor(rawTokens);
+        break;
+      case "round":
+        estimatedTokens = Math.round(rawTokens);
+        break;
+      case "ceil":
+      default:
+        estimatedTokens = Math.ceil(rawTokens);
+    }
+    tokenizerModeUsed = "heuristic";
+  }
+  const estimatedInputCost = estimatedTokens * config.inputCostPerMillion / 1e6;
+  return {
+    model,
+    characterCount,
+    estimatedTokens,
+    estimatedInputCost,
+    charsPerToken: config.charsPerToken,
+    tokenizerMode: tokenizerModeUsed,
+    encodingUsed
+  };
+}
+
+// src/token-counter.ts
+function isNonOpenAIModel2(model) {
+  return model.startsWith("claude-") || model.startsWith("gemini-");
+}
 function countTokens(input) {
   const { text, model } = input;
-  if (isNonOpenAIModel(model)) {
+  if (isNonOpenAIModel2(model)) {
     return {
       tokens: estimate({ text, model }).estimatedTokens,
       exact: false
@@ -533,10 +814,14 @@ function countTokens(input) {
 export {
   DEFAULT_MODELS,
   LAST_UPDATED,
+  countAnthropicInputTokens,
+  countGeminiTokens,
+  countGemmaSentencePieceTokens,
   countTokens,
   decode,
   encode,
   estimate,
+  estimateAsync,
   getAvailableModels,
   getModelConfig
 };

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "ai-token-estimator",
-  "version": "1.1.0",
-  "description": "Estimate token counts and costs for LLM API calls",
+  "version": "1.2.0",
+  "description": "Estimate and count tokens (incl. exact OpenAI BPE) and input costs for LLM API calls",
   "type": "module",
   "main": "./dist/index.cjs",
   "module": "./dist/index.js",
@@ -18,13 +18,17 @@
       }
     }
   },
+  "publishConfig": {
+    "access": "public"
+  },
   "files": [
     "dist",
     "LICENSE",
     "README.md"
   ],
   "dependencies": {
-    "gpt-tokenizer": "^3.4.0"
+    "gpt-tokenizer": "^3.4.0",
+    "sentencepiece-js": "^1.1.0"
   },
   "scripts": {
     "build": "tsup src/index.ts --format cjs,esm --dts",
@@ -37,8 +41,14 @@
   },
   "keywords": [
     "llm",
+    "tokenizer",
+    "token-count",
+    "token-counter",
     "tokens",
     "estimator",
+    "cost-estimator",
+    "tiktoken",
+    "bpe",
     "openai",
     "anthropic",
     "claude",