bpe-lite 0.4.3 → 0.5.2

package/README.md

@@ -30,7 +30,6 @@ import { countTokens, encode, decode, isWithinTokenLimit, openai, openaiO200k, a
 ```
 
 ```js
-
 // Count tokens
 countTokens('Your text here', 'openai-o200k'); // → number (GPT-4o, o1, o3, o4, GPT-4.1, GPT-5)
 countTokens('Your text here', 'openai');       // → number (GPT-4, GPT-3.5)
@@ -53,44 +52,92 @@ tok.count('Hello, world!');   // → 4
 
 ## Providers
 
-| Provider | Vocab | Tokens | Models | Accuracy |
-|----------|-------|--------|--------|----------|
-| `openai-o200k` | o200k_base | 199,998 | GPT-4o, o1, o3, o4, GPT-4.1, GPT-5 | Exact — vocab sourced directly from OpenAI's CDN |
-| `openai` | cl100k_base | 100,256 | GPT-4, GPT-3.5 | Exact — vocab sourced directly from OpenAI's CDN |
-| `anthropic` | cl100k approximation | 100,256 | Claude 3+ | ~95% — Anthropic has not released the Claude 3+ tokenizer |
-| `gemini` | Gemma 3 | 262,144 | Gemini | Exact — Gemini uses the same tokenizer as Gemma 3 open-weights |
+| Provider | Vocab | Models | Accuracy |
+|----------|-------|--------|----------|
+| `openai-o200k` | o200k_base (200k) | GPT-4o, o1, o3, o4, GPT-4.1, GPT-5 | Exact — vocab sourced from OpenAI |
+| `openai` | cl100k_base (100k) | GPT-4, GPT-3.5 | Exact — vocab sourced from OpenAI |
+| `anthropic` | Xenova/claude-tokenizer (65k BPE) | Claude | See accuracy section below |
+| `gemini` | Gemma 3 SPM (262k) | Gemini | See accuracy section below |
 
 Vocab files are bundled in the package — no network required at runtime or install time.
 
+## Accuracy — Anthropic
+
+Anthropic has not released the Claude tokenizer. bpe-lite uses [`Xenova/claude-tokenizer`](https://huggingface.co/Xenova/claude-tokenizer), a community reverse-engineering of the ~65k BPE vocabulary, with hand-tuned byte-level corrections applied on top.
+
+### Benchmark methodology
+
+Tested against `claude-haiku-4-5-20251001` via the Anthropic `count_tokens` API on a 120-sample stratified corpus across 12 categories. The corpus deliberately over-represents difficult content (Arabic, symbols, emoji, numbers) to expose systematic failures — overall numbers are lower than you would see on typical prose-only workloads by design.
+
+### Overall results (120 samples, 114 eligible)
+
+| Metric | bpe-lite | ai-tokenizer |
+|--------|----------|--------------|
+| Within 5% | 46.5% | 18.4% |
+| Within 10% | 62.3% ±8.8% CI | 37.7% ±8.8% CI |
+| Mean abs error | 9.4% | 16.0% |
+| Median abs error | 5.7% | 13.6% |
+| Max abs error | 42.9% | 82.6% |
+
+### Per-category breakdown
+
+| Category | Within 10% | Mean error | Notes |
+|----------|-----------|------------|-------|
+| `code-js` | 100% | 4.2% | |
+| `english-prose` | 90% | 5.5% | |
+| `code-python` | 90% | 4.8% | |
+| `structured` | 90% | 3.6% | JSON, HTML, XML, Markdown, SQL |
+| `numbers` | 80% | 7.3% | |
+| `hex-binary` | 80% | 5.3% | |
+| `urls` | 80% | 3.6% | |
+| `cjk` | 40% | 8.8% | |
+| `short` | 30% | 6.8% | |
+| `emoji` | 20% | 17.7% | ZWJ sequences, flags, skin tones |
+| `symbols` | 10% | 17.6% | Cross-byte merges unreplicable |
+| `arabic` | 0% | 26.1% | Structural vocabulary gap — unfixable |
+
+For prose, code, structured data, and URLs — the dominant content types in real-world prompts — bpe-lite is within 10% on 80–100% of samples. Arabic and symbol-cluster-heavy content cannot be accurately estimated without the actual Claude tokenizer.
+
+### Why bpe-lite outperforms ai-tokenizer on Claude
+
+ai-tokenizer's `claude` encoding uses `\p{N}+` (greedy, unlimited digit chunks). Current Claude models use `\p{N}{1,3}` (1–3 digits). This causes 20–43% errors on anything involving numbers — including code, hex, and data. bpe-lite uses the correct pattern.
+
+ai-tokenizer also does not have a Gemini encoding: all Gemini models are mapped to OpenAI's `o200k_base` vocabulary with a fudge multiplier. This is wrong by construction — see below.
+
+## Accuracy — Gemini
+
+bpe-lite implements the full Gemma 3 SentencePiece BPE algorithm using the actual Gemini vocabulary. On a 25-sample test against the Gemini API, bpe-lite scored 100% exact (no failures found; 25 samples is a limited basis — treat this as a lower bound, not a guarantee across all content types).
+
+ai-tokenizer does not implement Gemini natively. Inspecting their bundled source (`dist/index.js`), every Gemini model is defined as `"encoding": "o200k_base"` with a `"contentMultiplier": 1.08` fudge factor — it runs the OpenAI vocabulary through a multiplier rather than using Gemini's actual tokenizer. bpe-lite uses the actual Gemma 3 vocabulary and SentencePiece algorithm.
+
 ## Performance
 
-Benchmarked on Node v24 (win32/x64). Benchmark command: `node --expose-gc scripts/bench.js`.
+Benchmarked on Node v24 (win32/x64). Run `node --expose-gc scripts/bench.js` locally for numbers on your hardware.
+
+**Large text (~500 KB) — ops/s**
 
-**OpenAI cl100k — large text (~54 KB)**
+| impl | cl100k | Anthropic | Gemini | note |
+|------|-------:|----------:|-------:|------|
+| bpe-lite | 291 | 289 | 998 | |
+| ai-tokenizer | 291 | 291 | 215 | Gemini column uses o200k — wrong algorithm |
+| js-tiktoken | 30 | — | — | WASM overhead |
 
-| impl | ops/s | tokens/s | MB/s |
-|------|------:|---------:|-----:|
-| bpe-lite | **257** | **3.15M** | **13.6** |
-| ai-tokenizer | 201 | 2.46M | 10.7 |
-| js-tiktoken | 23 | 282k | 1.2 |
+bpe-lite matches ai-tokenizer throughput for OpenAI and Anthropic large text. On Gemini, bpe-lite's SPM engine is ~4.6x faster — the ai-tokenizer column there is not a valid comparison since it uses a different algorithm.
 
-**Anthropic — large text (~54 KB)**
+**Small text cold vs warm**
 
-| impl | ops/s | tokens/s | MB/s |
-|------|------:|---------:|-----:|
-| bpe-lite | **257** | 3.15M | **13.6** |
-| ai-tokenizer | 253 | **4.62M** | 13.4 |
+bpe-lite maintains a per-instance chunk cache. For repeated text patterns (e.g. the same prompt template encoded thousands of times), cache hits eliminate BPE work entirely:
 
-**Gemini — large text (8 KB)**
+| scenario | bpe-lite | ai-tokenizer |
+|----------|----------|--------------|
+| Small text, cold (novel input) | ~352k ops/s | ~453k ops/s |
+| Small text, warm (repeated input) | ~1.45M ops/s | ~453k ops/s |
 
-| impl | ops/s | tokens/s | MB/s | note |
-|------|------:|---------:|-----:|------|
-| bpe-lite | **3,800** | **6.23M** | **29.7** | actual Gemma3 SPM |
-| ai-tokenizer | 1,220 | 2.01M | 9.6 | o200k BPE — different algorithm, different results |
+For diverse, non-repeating inputs, ai-tokenizer is ~29% faster on very short strings. For any repeated-text workload, bpe-lite is ~3x faster.
 
-ai-tokenizer does not implement Gemini tokenization. The row above uses their o200k encoding on the same input string; it produces different token ids and counts than the Gemini tokenizer, so it is not a real comparison.
+**Initialization**
 
-Numbers vary by machine — run the bench script locally for results on your hardware.
+bpe-lite lazy-loads the gzipped vocab on first encode — one-time cost of ~235ms per provider per process. Negligible for any persistent process. Relevant only for cold serverless invocations that encode once and exit.
 
 ## API
 
@@ -108,7 +155,7 @@ Decodes an array of token ids back to a string.
 
 ### `isWithinTokenLimit(text, limit, provider?)`
 
-Returns the token count if `text` is within `limit` tokens, or `false` if exceeded. More efficient than `encode()` for long texts — the tiktoken engine short-circuits as soon as the limit is crossed.
+Returns the token count if `text` is within `limit` tokens, or `false` if exceeded. More efficient than `encode()` for long texts — short-circuits as soon as the limit is crossed.
 
 ### Tokenizer instances
 
@@ -118,11 +165,6 @@ Returns the token count if `text` is within `limit` tokens, or `false` if exceed
 
 `tiktoken` is accurate for OpenAI but requires Rust/WASM native bindings, which can break in Docker containers, edge runtimes, and serverless environments. `bpe-lite` is pure JavaScript — it runs anywhere Node 18+ runs, with no native compilation step.
 
-## Caveats
-
-- **Anthropic**: Anthropic has not released the Claude 3+ tokenizer. The cl100k approximation is accurate to ~95% for most text.
-- **Node version**: Requires Node 18+ for Unicode property escapes (`\p{L}`, `\p{N}`) in the pre-tokenization regex.
-
 ## License
 
 MIT

package/index.d.ts

@@ -1,42 +1,42 @@
-export type Provider = 'openai' | 'openai-o200k' | 'anthropic' | 'gemini';
-
-export interface Tokenizer {
-  encode(text: string): number[];
-  decode(ids: number[]): string;
-  count(text: string): number;
-  countUpTo(text: string, limit: number): number;
-}
-
-/**
- * Count the number of tokens in text for the given provider.
- */
-export function countTokens(text: string, provider?: Provider): number;
-
-/**
- * Encode text to token ids.
- */
-export function encode(text: string, provider?: Provider): number[];
-
-/**
- * Decode token ids back to text.
- */
-export function decode(ids: number[], provider?: Provider): string;
-
-/**
- * Check if text is within a token limit.
- * Returns the token count if within the limit, or false if exceeded.
- * More efficient than encode() for long texts since it short-circuits.
- */
-export function isWithinTokenLimit(text: string, limit: number, provider?: Provider): number | false;
-
-/** Tokenizer instance for OpenAI cl100k_base (GPT-4, GPT-3.5). */
-export function openai(): Tokenizer;
-
-/** Tokenizer instance for OpenAI o200k_base (GPT-4o, o1, o3, o4, GPT-4.1, GPT-5). */
-export function openaiO200k(): Tokenizer;
-
-/** Tokenizer instance for Anthropic (cl100k approximation, ~95% accurate). */
-export function anthropic(): Tokenizer;
-
-/** Tokenizer instance for Gemini (Gemma 3 vocab, exact). */
-export function gemini(): Tokenizer;
+export type Provider = 'openai' | 'openai-o200k' | 'anthropic' | 'gemini';
+
+export interface Tokenizer {
+  encode(text: string): number[];
+  decode(ids: number[]): string;
+  count(text: string): number;
+  countUpTo(text: string, limit: number): number;
+}
+
+/**
+ * Count the number of tokens in text for the given provider.
+ */
+export function countTokens(text: string, provider?: Provider): number;
+
+/**
+ * Encode text to token ids.
+ */
+export function encode(text: string, provider?: Provider): number[];
+
+/**
+ * Decode token ids back to text.
+ */
+export function decode(ids: number[], provider?: Provider): string;
+
+/**
+ * Check if text is within a token limit.
+ * Returns the token count if within the limit, or false if exceeded.
+ * More efficient than encode() for long texts since it short-circuits.
+ */
+export function isWithinTokenLimit(text: string, limit: number, provider?: Provider): number | false;
+
+/** Tokenizer instance for OpenAI cl100k_base (GPT-4, GPT-3.5). */
+export function openai(): Tokenizer;
+
+/** Tokenizer instance for OpenAI o200k_base (GPT-4o, o1, o3, o4, GPT-4.1, GPT-5). */
+export function openaiO200k(): Tokenizer;
+
+/** Tokenizer instance for Anthropic (cl100k approximation, ~95% accurate). */
+export function anthropic(): Tokenizer;
+
+/** Tokenizer instance for Gemini (Gemma 3 vocab, exact). */
+export function gemini(): Tokenizer;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "bpe-lite",
-  "version": "0.4.3",
+  "version": "0.5.2",
   "description": "Offline BPE tokenizer for OpenAI, Anthropic, and Gemini — zero dependencies",
   "main": "src/index.js",
   "types": "index.d.ts",
@@ -36,6 +36,7 @@
   "license": "MIT",
   "devDependencies": {
     "ai-tokenizer": "^1.0.6",
+    "gpt-tokenizer": "^3.4.0",
     "js-tiktoken": "^1.0.21"
   },
   "repository": {

package/src/bpe.js

@@ -6,6 +6,10 @@
  * Lower rank = higher priority in merges.
  */
 
+// Fast check: does a pre-tokenized chunk contain any letter or digit?
+// Non-word chunks (pure symbols, punctuation, spaces) are eligible for batching.
+const WORD_CHAR_RE = /[\p{L}\p{N}]/u;
+
 // Byte → single-byte "binary string" lookup (pre-built at module load)
 const BYTE_STRS = (() => {
   const out = new Array(256);
@@ -80,7 +84,8 @@ class MinHeap {
   _siftUp(i) {
     while (i > 0) {
       const p = (i - 1) >> 1;
-      if (this.ranks[p] <= this.ranks[i]) break;
+      if (this.ranks[p] < this.ranks[i]) break;
+      if (this.ranks[p] === this.ranks[i] && this.left[p] <= this.left[i]) break;
       this._swap(i, p);
       i = p;
     }
@@ -93,8 +98,12 @@ class MinHeap {
       if (l >= n) break;
       const r = l + 1;
       let m = l;
-      if (r < n && this.ranks[r] < this.ranks[l]) m = r;
-      if (this.ranks[i] <= this.ranks[m]) break;
+      if (r < n) {
+        if (this.ranks[r] < this.ranks[l] ||
+            (this.ranks[r] === this.ranks[l] && this.left[r] < this.left[l])) m = r;
+      }
+      if (this.ranks[i] < this.ranks[m]) break;
+      if (this.ranks[i] === this.ranks[m] && this.left[i] <= this.left[m]) break;
       this._swap(i, m);
       i = m;
     }
@@ -136,7 +145,7 @@ function pretokenize(text, compiled) {
 }
 
 function buildPreparedTiktoken(vocabData) {
-  const { vocab, specialTokens = {}, pattern } = vocabData;
+  const { vocab, specialTokens = {}, pattern, normalize } = vocabData;
 
   const vocabBin = new Map();
   let maxId = -1;
@@ -164,6 +173,10 @@ function buildPreparedTiktoken(vocabData) {
     idToBytes,
     specials,
     patternCompiled: compilePretokenizer(pattern),
+    normalize: normalize || null,
+    // symbolBatch: merge consecutive non-word regex chunks before BPE so that
+    // cross-character byte merges can fire (matches Claude's no-regex byte BPE).
+    symbolBatch: !!vocabData.symbolBatch,
     // opt A — per-instance chunk cache: chunk string → ids[]
     cache: new Map(),
     // opt B — per-instance grow-only scratch (reused across chunks)
@@ -302,6 +315,7 @@ function splitOnSpecials(text, specials) {
 
 function encodeTiktokenPrepared(text, prepared) {
   if (!text) return [];
+  if (prepared.normalize) text = text.normalize(prepared.normalize);
 
   const ids = [];
   const { vocabBin, scratch, cache, patternCompiled, specials } = prepared;
@@ -316,14 +330,41 @@ function encodeTiktokenPrepared(text, prepared) {
       const re = patternCompiled.re;
       re.lastIndex = 0;
       let m;
-      while ((m = re.exec(t)) !== null) {
-        const chunk = m[0];
-        let chunkIds = cache.get(chunk);
-        if (chunkIds === undefined) {
-          chunkIds = bpeChunk(writeChunk(chunk), vocabBin, scratch);
-          cache.set(chunk, chunkIds);
+      if (prepared.symbolBatch) {
+        // Merge consecutive non-word chunks (symbols, punctuation, spaces) into one
+        // BPE input so cross-character byte merges can fire — matches Claude's behavior.
+        let symBuf = null;
+        while ((m = re.exec(t)) !== null) {
+          const chunk = m[0];
+          if (!WORD_CHAR_RE.test(chunk)) {
+            symBuf = symBuf === null ? chunk : symBuf + chunk;
+          } else {
+            if (symBuf !== null) {
+              let symIds = cache.get(symBuf);
+              if (symIds === undefined) { symIds = bpeChunk(writeChunk(symBuf), vocabBin, scratch); cache.set(symBuf, symIds); }
+              for (let i = 0; i < symIds.length; i++) ids.push(symIds[i]);
+              symBuf = null;
+            }
+            let chunkIds = cache.get(chunk);
+            if (chunkIds === undefined) { chunkIds = bpeChunk(writeChunk(chunk), vocabBin, scratch); cache.set(chunk, chunkIds); }
+            for (let i = 0; i < chunkIds.length; i++) ids.push(chunkIds[i]);
+          }
+        }
+        if (symBuf !== null) {
+          let symIds = cache.get(symBuf);
+          if (symIds === undefined) { symIds = bpeChunk(writeChunk(symBuf), vocabBin, scratch); cache.set(symBuf, symIds); }
+          for (let i = 0; i < symIds.length; i++) ids.push(symIds[i]);
+        }
+      } else {
+        while ((m = re.exec(t)) !== null) {
+          const chunk = m[0];
+          let chunkIds = cache.get(chunk);
+          if (chunkIds === undefined) {
+            chunkIds = bpeChunk(writeChunk(chunk), vocabBin, scratch);
+            cache.set(chunk, chunkIds);
+          }
+          for (let i = 0; i < chunkIds.length; i++) ids.push(chunkIds[i]);
         }
-        for (let i = 0; i < chunkIds.length; i++) ids.push(chunkIds[i]);
       }
     } else {
       const chunks = patternCompiled.type === 'none' ? [t] : (t.match(/\S+|\s+/g) || [t]);
@@ -355,6 +396,7 @@ function decodeTiktokenPrepared(ids, prepared) {
 
 function countTiktokenPrepared(text, prepared) {
   if (!text) return 0;
+  if (prepared.normalize) text = text.normalize(prepared.normalize);
 
   const { vocabBin, scratch, cache, patternCompiled, specials } = prepared;
   const pieces = splitOnSpecials(text, specials);
@@ -368,14 +410,39 @@ function countTiktokenPrepared(text, prepared) {
       const re = patternCompiled.re;
       re.lastIndex = 0;
       let m;
-      while ((m = re.exec(t)) !== null) {
-        const chunk = m[0];
-        let chunkIds = cache.get(chunk);
-        if (chunkIds === undefined) {
-          chunkIds = bpeChunk(writeChunk(chunk), vocabBin, scratch);
-          cache.set(chunk, chunkIds);
+      if (prepared.symbolBatch) {
+        let symBuf = null;
+        while ((m = re.exec(t)) !== null) {
+          const chunk = m[0];
+          if (!WORD_CHAR_RE.test(chunk)) {
+            symBuf = symBuf === null ? chunk : symBuf + chunk;
+          } else {
+            if (symBuf !== null) {
+              let symIds = cache.get(symBuf);
+              if (symIds === undefined) { symIds = bpeChunk(writeChunk(symBuf), vocabBin, scratch); cache.set(symBuf, symIds); }
+              count += symIds.length;
+              symBuf = null;
+            }
+            let chunkIds = cache.get(chunk);
+            if (chunkIds === undefined) { chunkIds = bpeChunk(writeChunk(chunk), vocabBin, scratch); cache.set(chunk, chunkIds); }
+            count += chunkIds.length;
+          }
+        }
+        if (symBuf !== null) {
+          let symIds = cache.get(symBuf);
+          if (symIds === undefined) { symIds = bpeChunk(writeChunk(symBuf), vocabBin, scratch); cache.set(symBuf, symIds); }
+          count += symIds.length;
+        }
+      } else {
+        while ((m = re.exec(t)) !== null) {
+          const chunk = m[0];
+          let chunkIds = cache.get(chunk);
+          if (chunkIds === undefined) {
+            chunkIds = bpeChunk(writeChunk(chunk), vocabBin, scratch);
+            cache.set(chunk, chunkIds);
+          }
+          count += chunkIds.length;
         }
-        count += chunkIds.length;
       }
     } else {
       const chunks = patternCompiled.type === 'none' ? [t] : (t.match(/\S+|\s+/g) || [t]);
@@ -396,6 +463,7 @@ function countTiktokenPrepared(text, prepared) {
 
 function countTiktokenUpToPrepared(text, prepared, limit) {
   if (!text) return 0;
+  if (prepared.normalize) text = text.normalize(prepared.normalize);
 
   const { vocabBin, scratch, cache, patternCompiled, specials } = prepared;
   const pieces = splitOnSpecials(text, specials);

package/src/index.js

@@ -1,90 +1,90 @@
-'use strict';
-
-const path = require('path');
-const fs = require('fs');
-const zlib = require('zlib');
-const { Tokenizer } = require('./tokenizer');
-
-const VOCABS_DIR = path.join(__dirname, '..', 'vocabs');
-
-// Lazy-loaded tokenizer instances (created once per provider per process)
-const _cache = {};
-
-function loadTokenizer(provider) {
-  if (_cache[provider]) return _cache[provider];
-
-  const gzPath   = path.join(VOCABS_DIR, `${provider}.json.gz`);
-  const jsonPath  = path.join(VOCABS_DIR, `${provider}.json`);
-
-  let data;
-  if (fs.existsSync(gzPath)) {
-    data = JSON.parse(zlib.gunzipSync(fs.readFileSync(gzPath)).toString('utf8'));
-  } else if (fs.existsSync(jsonPath)) {
-    data = JSON.parse(fs.readFileSync(jsonPath, 'utf8'));
-  } else {
-    throw new Error(
-      `Vocab file not found for provider "${provider}".\n` +
-      'Run "node scripts/build-vocabs.js" to build vocab files.'
-    );
-  }
-
-  _cache[provider] = new Tokenizer(data);
-  return _cache[provider];
-}
-
-/**
- * Count tokens in text for a given provider.
- * @param {string} text
- * @param {'openai'|'anthropic'|'gemini'} provider
- * @returns {number}
- */
-function countTokens(text, provider = 'openai') {
-  return loadTokenizer(provider).count(text);
-}
-
-/**
- * Encode text to token ids.
- * @param {string} text
- * @param {'openai'|'anthropic'|'gemini'} provider
- * @returns {number[]}
- */
-function encode(text, provider = 'openai') {
-  return loadTokenizer(provider).encode(text);
-}
-
-/**
- * Decode token ids back to text.
- * @param {number[]} ids
- * @param {'openai'|'openai-o200k'|'anthropic'|'gemini'} provider
- * @returns {string}
- */
-function decode(ids, provider = 'openai') {
-  return loadTokenizer(provider).decode(ids);
-}
-
-/**
- * Check if text is within a token limit without necessarily encoding the whole string.
- * Returns false if the limit is exceeded, otherwise returns the token count.
- * @param {string} text
- * @param {number} limit
- * @param {'openai'|'openai-o200k'|'anthropic'|'gemini'} provider
- * @returns {number|false}
- */
-function isWithinTokenLimit(text, limit, provider = 'openai') {
-  const count = loadTokenizer(provider).countUpTo(text, limit);
-  return count <= limit ? count : false;
-}
-
-/** Get a Tokenizer instance for OpenAI (cl100k_base — GPT-4, GPT-3.5). */
-function openai() { return loadTokenizer('openai'); }
-
-/** Get a Tokenizer instance for OpenAI modern models (o200k_base — GPT-4o, o1, o3, o4, GPT-4.1, GPT-5). */
-function openaiO200k() { return loadTokenizer('openai-o200k'); }
-
-/** Get a Tokenizer instance for Anthropic (cl100k approximation). */
-function anthropic() { return loadTokenizer('anthropic'); }
-
-/** Get a Tokenizer instance for Gemini (Gemma3 vocab). */
-function gemini() { return loadTokenizer('gemini'); }
-
-module.exports = { countTokens, encode, decode, isWithinTokenLimit, openai, openaiO200k, anthropic, gemini };
+'use strict';
+
+const path = require('path');
+const fs = require('fs');
+const zlib = require('zlib');
+const { Tokenizer } = require('./tokenizer');
+
+const VOCABS_DIR = path.join(__dirname, '..', 'vocabs');
+
+// Lazy-loaded tokenizer instances (created once per provider per process)
+const _cache = {};
+
+function loadTokenizer(provider) {
+  if (_cache[provider]) return _cache[provider];
+
+  const gzPath   = path.join(VOCABS_DIR, `${provider}.json.gz`);
+  const jsonPath  = path.join(VOCABS_DIR, `${provider}.json`);
+
+  let data;
+  if (fs.existsSync(gzPath)) {
+    data = JSON.parse(zlib.gunzipSync(fs.readFileSync(gzPath)).toString('utf8'));
+  } else if (fs.existsSync(jsonPath)) {
+    data = JSON.parse(fs.readFileSync(jsonPath, 'utf8'));
+  } else {
+    throw new Error(
+      `Vocab file not found for provider "${provider}".\n` +
+      'Run "node scripts/build-vocabs.js" to build vocab files.'
+    );
+  }
+
+  _cache[provider] = new Tokenizer(data);
+  return _cache[provider];
+}
+
+/**
+ * Count tokens in text for a given provider.
+ * @param {string} text
+ * @param {'openai'|'anthropic'|'gemini'} provider
+ * @returns {number}
+ */
+function countTokens(text, provider = 'openai') {
+  return loadTokenizer(provider).count(text);
+}
+
+/**
+ * Encode text to token ids.
+ * @param {string} text
+ * @param {'openai'|'anthropic'|'gemini'} provider
+ * @returns {number[]}
+ */
+function encode(text, provider = 'openai') {
+  return loadTokenizer(provider).encode(text);
+}
+
+/**
+ * Decode token ids back to text.
+ * @param {number[]} ids
+ * @param {'openai'|'openai-o200k'|'anthropic'|'gemini'} provider
+ * @returns {string}
+ */
+function decode(ids, provider = 'openai') {
+  return loadTokenizer(provider).decode(ids);
+}
+
+/**
+ * Check if text is within a token limit without necessarily encoding the whole string.
+ * Returns false if the limit is exceeded, otherwise returns the token count.
+ * @param {string} text
+ * @param {number} limit
+ * @param {'openai'|'openai-o200k'|'anthropic'|'gemini'} provider
+ * @returns {number|false}
+ */
+function isWithinTokenLimit(text, limit, provider = 'openai') {
+  const count = loadTokenizer(provider).countUpTo(text, limit);
+  return count <= limit ? count : false;
+}
+
+/** Get a Tokenizer instance for OpenAI (cl100k_base — GPT-4, GPT-3.5). */
+function openai() { return loadTokenizer('openai'); }
+
+/** Get a Tokenizer instance for OpenAI modern models (o200k_base — GPT-4o, o1, o3, o4, GPT-4.1, GPT-5). */
+function openaiO200k() { return loadTokenizer('openai-o200k'); }
+
+/** Get a Tokenizer instance for Anthropic (cl100k approximation). */
+function anthropic() { return loadTokenizer('anthropic'); }
+
+/** Get a Tokenizer instance for Gemini (Gemma3 vocab). */
+function gemini() { return loadTokenizer('gemini'); }
+
+module.exports = { countTokens, encode, decode, isWithinTokenLimit, openai, openaiO200k, anthropic, gemini };

package/src/index.mjs

@@ -1,24 +1,24 @@
-import { createRequire } from 'module';
-
-const require = createRequire(import.meta.url);
-const {
-  countTokens,
-  encode,
-  decode,
-  isWithinTokenLimit,
-  openai,
-  openaiO200k,
-  anthropic,
-  gemini,
-} = require('./index.js');
-
-export {
-  countTokens,
-  encode,
-  decode,
-  isWithinTokenLimit,
-  openai,
-  openaiO200k,
-  anthropic,
-  gemini,
-};
+import { createRequire } from 'module';
+
+const require = createRequire(import.meta.url);
+const {
+  countTokens,
+  encode,
+  decode,
+  isWithinTokenLimit,
+  openai,
+  openaiO200k,
+  anthropic,
+  gemini,
+} = require('./index.js');
+
+export {
+  countTokens,
+  encode,
+  decode,
+  isWithinTokenLimit,
+  openai,
+  openaiO200k,
+  anthropic,
+  gemini,
+};

package/src/spm.js

@@ -23,12 +23,31 @@ function buildPreparedSPM(vocabData) {
     idToStr.set(id, str);
   }
 
+  // Seed tokens: multi-char vocab entries with no producing merge.
+  // In the original SentencePiece model these were user-defined symbols
+  // (HTML tags, special tokens, etc.) that the encoder recognizes atomically
+  // before BPE. We handle the common '<' case with a greedy longest-match
+  // lookup keyed by '<' — the only first-char with multiple seed tokens.
+  const producible = new Set();
+  for (const m of merges) {
+    const sp = m.indexOf(' ');
+    if (sp !== -1) producible.add(m.slice(0, sp) + m.slice(sp + 1));
+  }
+  const seedsByAngleBracket = [];
+  for (const [str, id] of Object.entries(vocab)) {
+    if (str.length > 1 && str[0] === '<' && !producible.has(str)) {
+      seedsByAngleBracket.push({ str, id, chars: [...str] });
+    }
+  }
+  seedsByAngleBracket.sort((a, b) => b.chars.length - a.chars.length);
+
   return {
     vocab,
     merges,
     mergeRank,
     idToStr,
-    // opt A — segment-level cache: each ▁-prefixed word segment → ids[]
+    seedsByAngleBracket,
+    // opt A — segment-level cache: each word segment → ids[]
     // Generalises across different inputs (same words reused across texts).
     // Note: 1 of 514,906 Gemma merges crosses a ▁ boundary ("> ▁</"),
     // making this negligibly imprecise for that HTML pattern.
@@ -66,24 +85,44 @@ function encodeSPM(text, vocabData) {
  * Returns true if every segment was a cache hit; false on first miss.
  * Isolated from encodeSegment so V8 can keep this function optimised
  * even when encodeSPMPrepared is called with wildly different text lengths.
+ *
+ * Segmentation: each run of ▁ chars plus the following non-▁ word is one segment.
+ * Leading ▁s are included so BPE can naturally merge ▁▁→138, ▁▁▁→139, etc.
  */
 function _scanFromCache(normalized, cache, result) {
-  let segStart = 0;
-  for (let i = 1; i <= normalized.length; i++) {
-    if (i === normalized.length || normalized[i] === SPACE_CHAR) {
-      const seg = normalized.slice(segStart, i);
-      const segIds = cache.get(seg);
-      if (segIds === undefined) return false; // cache miss → caller handles cold path
-      for (let j = 0; j < segIds.length; j++) result.push(segIds[j]);
-      segStart = i;
+  let i = 0;
+  while (i < normalized.length) {
+    // Count leading ▁ chars
+    const segStart = i;
+    while (i < normalized.length && normalized[i] === SPACE_CHAR) i++;
+    const spaceCount = i - segStart;
+
+    if (i === normalized.length) {
+      // Trailing spaces only
+      if (spaceCount > 0) {
+        const seg = normalized.slice(segStart, i);
+        const segIds = cache.get(seg);
+        if (segIds === undefined) return false;
+        for (let j = 0; j < segIds.length; j++) result.push(segIds[j]);
+      }
+      break;
     }
+
+    // Collect word chars (non-▁)
+    while (i < normalized.length && normalized[i] !== SPACE_CHAR) i++;
+
+    // Emit all leading ▁s + word as one segment; BPE merges ▁▁, ▁▁▁, etc.
+    const seg = normalized.slice(segStart, i);
+    const segIds = cache.get(seg);
+    if (segIds === undefined) return false;
+    for (let j = 0; j < segIds.length; j++) result.push(segIds[j]);
   }
   return true;
 }
 
 // Cold-path helper — kept separate so it is never inlined into the hot loop.
-function _encodeAndCache(seg, vocab, mergeRank, scratch, cache) {
-  const ids = encodeSegment(seg, vocab, mergeRank, scratch);
+function _encodeAndCache(seg, vocab, mergeRank, scratch, cache, seeds) {
+  const ids = encodeSegment(seg, vocab, mergeRank, scratch, seeds);
   cache.set(seg, ids);
   return ids;
 }
@@ -91,10 +130,10 @@ function _encodeAndCache(seg, vocab, mergeRank, scratch, cache) {
 function encodeSPMPrepared(text, prepared) {
   if (!text) return [];
 
-  const { vocab, mergeRank, scratch, cache } = prepared;
+  const { vocab, mergeRank, scratch, cache, seedsByAngleBracket } = prepared;
 
-  // Normalize: replace spaces with ▁, prepend ▁
-  const normalized = SPACE_CHAR + text.replace(/ /g, SPACE_CHAR);
+  // Normalize: replace spaces with ▁ (Gemma3: no ▁ prepend for first char)
+  const normalized = text.replace(/ /g, SPACE_CHAR);
 
   // Fast path: serve every segment from the segment cache.
   // After the first call, this path handles all subsequent calls for common text.
@@ -104,20 +143,34 @@ function encodeSPMPrepared(text, prepared) {
   // Cold path: at least one segment is missing — encode everything from scratch.
   // (Simpler to re-scan than to continue from the miss point.)
   result.length = 0;
-  let segStart = 0;
-  for (let i = 1; i <= normalized.length; i++) {
-    if (i === normalized.length || normalized[i] === SPACE_CHAR) {
-      const seg = normalized.slice(segStart, i);
-      const segIds = cache.get(seg) ?? _encodeAndCache(seg, vocab, mergeRank, scratch, cache);
-      for (let j = 0; j < segIds.length; j++) result.push(segIds[j]);
-      segStart = i;
+  let i = 0;
+  while (i < normalized.length) {
+    const segStart = i;
+    while (i < normalized.length && normalized[i] === SPACE_CHAR) i++;
+    const spaceCount = i - segStart;
+
+    if (i === normalized.length) {
+      if (spaceCount > 0) {
+        const seg = normalized.slice(segStart, i);
+        const segIds = cache.get(seg) ?? _encodeAndCache(seg, vocab, mergeRank, scratch, cache, seedsByAngleBracket);
+        for (let j = 0; j < segIds.length; j++) result.push(segIds[j]);
+      }
+      break;
     }
+
+    while (i < normalized.length && normalized[i] !== SPACE_CHAR) i++;
+
+    const seg = normalized.slice(segStart, i);
+    const segIds = cache.get(seg) ?? _encodeAndCache(seg, vocab, mergeRank, scratch, cache, seedsByAngleBracket);
+    for (let j = 0; j < segIds.length; j++) result.push(segIds[j]);
   }
   return result;
 }
 
-// Encode a single ▁-prefixed segment using MinHeap BPE.
-function encodeSegment(seg, vocab, mergeRank, scratch) {
+// Encode a single segment using MinHeap BPE.
+// seeds: sorted array of {str, id, chars[]} for '<'-prefixed vocab entries that have
+// no producing merge (SentencePiece user-defined symbols); matched greedily before BPE.
+function encodeSegment(seg, vocab, mergeRank, scratch, seeds) {
   const chars = [...seg];
   const n = chars.length;
 
@@ -125,32 +178,59 @@ function encodeSegment(seg, vocab, mergeRank, scratch) {
   const { str, ids, prev, next, ver, alive, heap } = scratch;
   heap.reset();
 
-  for (let i = 0; i < n; i++) {
-    const c = chars[i];
-    prev[i]  = i - 1;
-    next[i]  = i + 1;
-    ver[i]   = 0;
-    alive[i] = 1;
+  // Initialize nodes: one node per Unicode char, except seed tokens which collapse
+  // multiple chars into a single node (longest match at each '<' position).
+  let nNodes = 0;
+  let pos = 0;
+  while (pos < n) {
+    const node = nNodes++;
+    prev[node] = node - 1;
+    next[node] = node + 1;
+    ver[node]  = 0;
+    alive[node] = 1;
+
+    let matched = false;
+    if (seeds && chars[pos] === '<') {
+      for (const seed of seeds) {
+        const sl = seed.chars.length;
+        if (pos + sl > n) continue;
+        let ok = true;
+        for (let k = 0; k < sl; k++) {
+          if (chars[pos + k] !== seed.chars[k]) { ok = false; break; }
+        }
+        if (ok) {
+          str[node] = seed.str;
+          ids[node] = seed.id;
+          pos += sl;
+          matched = true;
+          break;
+        }
+      }
+    }
 
-    if (vocab[c] !== undefined) {
-      str[i] = c;
-      ids[i] = vocab[c];
-    } else {
-      const codePoint = c.codePointAt(0);
-      const hex = codePoint.toString(16).toUpperCase().padStart(2, '0');
-      const byteKey = `<0x${hex}>`;
-      if (vocab[byteKey] !== undefined) {
-        str[i] = byteKey;
-        ids[i] = vocab[byteKey];
+    if (!matched) {
+      const c = chars[pos];
+      if (vocab[c] !== undefined) {
+        str[node] = c;
+        ids[node] = vocab[c];
       } else {
-        str[i] = c;
-        ids[i] = vocab['<unk>'] ?? 0;
+        const codePoint = c.codePointAt(0);
+        const hex = codePoint.toString(16).toUpperCase().padStart(2, '0');
+        const byteKey = `<0x${hex}>`;
+        if (vocab[byteKey] !== undefined) {
+          str[node] = byteKey;
+          ids[node] = vocab[byteKey];
+        } else {
+          str[node] = c;
+          ids[node] = vocab['<unk>'] ?? 0;
+        }
       }
+      pos++;
     }
   }
-  next[n - 1] = -1;
+  next[nNodes - 1] = -1;
 
-  for (let i = 0; i < n - 1; i++) {
+  for (let i = 0; i < nNodes - 1; i++) {
     const rank = mergeRank.get(`${str[i]} ${str[i + 1]}`);
     if (rank !== undefined) heap.push(rank, i, i + 1, ver[i], ver[i + 1]);
   }

package/src/tokenizer.js

@@ -1,59 +1,59 @@
-'use strict';
-
-const {
-  buildPreparedTiktoken,
-  encodeTiktokenPrepared,
-  decodeTiktokenPrepared,
-  countTiktokenPrepared,
-  countTiktokenUpToPrepared,
-} = require('./bpe');
-const { buildPreparedSPM, encodeSPMPrepared, decodeSPMPrepared } = require('./spm');
-
-class Tokenizer {
-  constructor(vocabData) {
-    this._data = vocabData;
-    this._engine = vocabData.engine;
-    this._preparedTiktoken = null;
-    this._preparedSPM = null;
-
-    if (this._engine !== 'tiktoken' && this._engine !== 'spm') {
-      throw new Error(`Unknown tokenizer engine: ${this._engine}`);
-    }
-
-    if (this._engine === 'tiktoken') {
-      this._preparedTiktoken = buildPreparedTiktoken(vocabData);
-    } else {
-      this._preparedSPM = buildPreparedSPM(vocabData);
-    }
-  }
-
-  encode(text) {
-    if (this._engine === 'tiktoken') return encodeTiktokenPrepared(text, this._preparedTiktoken);
-    return encodeSPMPrepared(text, this._preparedSPM);
-  }
-
-  decode(ids) {
-    if (this._engine === 'tiktoken') return decodeTiktokenPrepared(ids, this._preparedTiktoken);
-    return decodeSPMPrepared(ids, this._preparedSPM);
-  }
-
-  count(text) {
-    if (this._engine === 'tiktoken') return countTiktokenPrepared(text, this._preparedTiktoken);
-    return this.encode(text).length;
-  }
-
-  /**
-   * Count tokens, stopping as soon as the count exceeds limit.
-   * More efficient than encode() for token limit checks on long text.
-   * @param {string} text
-   * @param {number} limit
-   * @returns {number}
-   */
-  countUpTo(text, limit) {
-    if (this._engine === 'tiktoken') return countTiktokenUpToPrepared(text, this._preparedTiktoken, limit);
-    // SPM encodes the whole text as one unit — no clean early exit, just encode and count
-    return this.encode(text).length;
-  }
-}
-
-module.exports = { Tokenizer };
+'use strict';
+
+const {
+  buildPreparedTiktoken,
+  encodeTiktokenPrepared,
+  decodeTiktokenPrepared,
+  countTiktokenPrepared,
+  countTiktokenUpToPrepared,
+} = require('./bpe');
+const { buildPreparedSPM, encodeSPMPrepared, decodeSPMPrepared } = require('./spm');
+
+class Tokenizer {
+  constructor(vocabData) {
+    this._data = vocabData;
+    this._engine = vocabData.engine;
+    this._preparedTiktoken = null;
+    this._preparedSPM = null;
+
+    if (this._engine !== 'tiktoken' && this._engine !== 'spm') {
+      throw new Error(`Unknown tokenizer engine: ${this._engine}`);
+    }
+
+    if (this._engine === 'tiktoken') {
+      this._preparedTiktoken = buildPreparedTiktoken(vocabData);
+    } else {
+      this._preparedSPM = buildPreparedSPM(vocabData);
+    }
+  }
+
+  encode(text) {
+    if (this._engine === 'tiktoken') return encodeTiktokenPrepared(text, this._preparedTiktoken);
+    return encodeSPMPrepared(text, this._preparedSPM);
+  }
+
+  decode(ids) {
+    if (this._engine === 'tiktoken') return decodeTiktokenPrepared(ids, this._preparedTiktoken);
+    return decodeSPMPrepared(ids, this._preparedSPM);
+  }
+
+  count(text) {
+    if (this._engine === 'tiktoken') return countTiktokenPrepared(text, this._preparedTiktoken);
+    return this.encode(text).length;
+  }
+
+  /**
+   * Count tokens, stopping as soon as the count exceeds limit.
+   * More efficient than encode() for token limit checks on long text.
+   * @param {string} text
+   * @param {number} limit
+   * @returns {number}
+   */
+  countUpTo(text, limit) {
+    if (this._engine === 'tiktoken') return countTiktokenUpToPrepared(text, this._preparedTiktoken, limit);
+    // SPM encodes the whole text as one unit — no clean early exit, just encode and count
+    return this.encode(text).length;
+  }
+}
+
+module.exports = { Tokenizer };