bpe-lite 0.5.1 → 0.5.2

package/README.md

@@ -30,7 +30,6 @@ import { countTokens, encode, decode, isWithinTokenLimit, openai, openaiO200k, a
 ```
 
 ```js
-
 // Count tokens
 countTokens('Your text here', 'openai-o200k'); // → number (GPT-4o, o1, o3, o4, GPT-4.1, GPT-5)
 countTokens('Your text here', 'openai');       // → number (GPT-4, GPT-3.5)
@@ -53,44 +52,92 @@ tok.count('Hello, world!');   // → 4
 
 ## Providers
 
-| Provider | Vocab | Tokens | Models | Accuracy |
-|----------|-------|--------|--------|----------|
-| `openai-o200k` | o200k_base | 199,998 | GPT-4o, o1, o3, o4, GPT-4.1, GPT-5 | Exact — vocab sourced directly from OpenAI's CDN |
-| `openai` | cl100k_base | 100,256 | GPT-4, GPT-3.5 | Exact — vocab sourced directly from OpenAI's CDN |
-| `anthropic` | Xenova/claude-tokenizer | 65,000 | Claude | ~80% within 10%, mean error 7.0% — Anthropic has not released the tokenizer. Vocab sourced from Xenova's reverse-engineered 65k BPE (HuggingFace) with NFKC normalization, symbol/emoji byte-level corrections, whitespace sequence injections, and probe-based merge adjustments. Typical prose and code: well within 10%. Outliers: repeated chars (39%), Arabic (20%), symbols (16%), currency (14%). Tested against Claude 4 API across 25 diverse samples. |
-| `gemini` | Gemma 3 | 262,144 | Gemini | ~99.7% — 92% exact, mean error 0.34% across 25 diverse samples vs Gemini 2.0 Flash API |
+| Provider | Vocab | Models | Accuracy |
+|----------|-------|--------|----------|
+| `openai-o200k` | o200k_base (200k) | GPT-4o, o1, o3, o4, GPT-4.1, GPT-5 | Exact — vocab sourced from OpenAI |
+| `openai` | cl100k_base (100k) | GPT-4, GPT-3.5 | Exact — vocab sourced from OpenAI |
+| `anthropic` | Xenova/claude-tokenizer (65k BPE) | Claude | See accuracy section below |
+| `gemini` | Gemma 3 SPM (262k) | Gemini | See accuracy section below |
 
 Vocab files are bundled in the package — no network required at runtime or install time.
 
+## Accuracy — Anthropic
+
+Anthropic has not released the Claude tokenizer. bpe-lite uses [`Xenova/claude-tokenizer`](https://huggingface.co/Xenova/claude-tokenizer), a community reverse-engineering of the ~65k BPE vocabulary, with hand-tuned byte-level corrections applied on top.
+
+### Benchmark methodology
+
+Tested against `claude-haiku-4-5-20251001` via the Anthropic `count_tokens` API on a 120-sample stratified corpus across 12 categories. The corpus deliberately over-represents difficult content (Arabic, symbols, emoji, numbers) to expose systematic failures — overall numbers are lower than you would see on typical prose-only workloads by design.
+
+### Overall results (120 samples, 114 eligible)
+
+| Metric | bpe-lite | ai-tokenizer |
+|--------|----------|--------------|
+| Within 5% | 46.5% | 18.4% |
+| Within 10% | 62.3% ±8.8% CI | 37.7% ±8.8% CI |
+| Mean abs error | 9.4% | 16.0% |
+| Median abs error | 5.7% | 13.6% |
+| Max abs error | 42.9% | 82.6% |
+
+### Per-category breakdown
+
+| Category | Within 10% | Mean error | Notes |
+|----------|-----------|------------|-------|
+| `code-js` | 100% | 4.2% | |
+| `english-prose` | 90% | 5.5% | |
+| `code-python` | 90% | 4.8% | |
+| `structured` | 90% | 3.6% | JSON, HTML, XML, Markdown, SQL |
+| `numbers` | 80% | 7.3% | |
+| `hex-binary` | 80% | 5.3% | |
+| `urls` | 80% | 3.6% | |
+| `cjk` | 40% | 8.8% | |
+| `short` | 30% | 6.8% | |
+| `emoji` | 20% | 17.7% | ZWJ sequences, flags, skin tones |
+| `symbols` | 10% | 17.6% | Cross-byte merges unreplicable |
+| `arabic` | 0% | 26.1% | Structural vocabulary gap — unfixable |
+
+For prose, code, structured data, and URLs — the dominant content types in real-world prompts — bpe-lite is within 10% on 80–100% of samples. Arabic and symbol-cluster-heavy content cannot be accurately estimated without the actual Claude tokenizer.
+
+### Why bpe-lite outperforms ai-tokenizer on Claude
+
+ai-tokenizer's `claude` encoding uses `\p{N}+` (greedy, unlimited digit chunks). Current Claude models use `\p{N}{1,3}` (1–3 digits). This causes 20–43% errors on anything involving numbers — including code, hex, and data. bpe-lite uses the correct pattern.
+
+ai-tokenizer also does not have a Gemini encoding: all Gemini models are mapped to OpenAI's `o200k_base` vocabulary with a fudge multiplier. This is wrong by construction — see below.
+
+## Accuracy — Gemini
+
+bpe-lite implements the full Gemma 3 SentencePiece BPE algorithm using the actual Gemini vocabulary. On a 25-sample test against the Gemini API, bpe-lite scored 100% exact (no failures found; 25 samples is a limited basis — treat this as a lower bound, not a guarantee across all content types).
+
+ai-tokenizer does not implement Gemini natively. Inspecting their bundled source (`dist/index.js`), every Gemini model is defined as `"encoding": "o200k_base"` with a `"contentMultiplier": 1.08` fudge factor — it runs the OpenAI vocabulary through a multiplier rather than using Gemini's actual tokenizer. bpe-lite uses the actual Gemma 3 vocabulary and SentencePiece algorithm.
+
 ## Performance
 
-Benchmarked on Node v24 (win32/x64). Benchmark command: `node --expose-gc scripts/bench.js`.
+Benchmarked on Node v24 (win32/x64). Run `node --expose-gc scripts/bench.js` locally for numbers on your hardware.
+
+**Large text (~500 KB) — ops/s**
 
-**OpenAI cl100k — large text (~54 KB)**
+| impl | cl100k | Anthropic | Gemini | note |
+|------|-------:|----------:|-------:|------|
+| bpe-lite | 291 | 289 | 998 | |
+| ai-tokenizer | 291 | 291 | 215 | Gemini column uses o200k — wrong algorithm |
+| js-tiktoken | 30 | — | — | WASM overhead |
 
-| impl | ops/s | tokens/s | MB/s |
-|------|------:|---------:|-----:|
-| bpe-lite | **291** | **3.56M** | **15.4** |
-| ai-tokenizer | 291 | 3.56M | 15.4 |
-| js-tiktoken | 30 | 370k | 1.6 |
+bpe-lite matches ai-tokenizer throughput for OpenAI and Anthropic large text. On Gemini, bpe-lite's SPM engine is ~4.6x faster — the ai-tokenizer column there is not a valid comparison since it uses a different algorithm.
 
-**Anthropic — large text (~54 KB)**
+**Small text cold vs warm**
 
-| impl | ops/s | tokens/s | MB/s |
-|------|------:|---------:|-----:|
-| bpe-lite | **289** | **5.34M** | **15.3** |
-| ai-tokenizer | 291 | 5.31M | 15.4 |
+bpe-lite maintains a per-instance chunk cache. For repeated text patterns (e.g. the same prompt template encoded thousands of times), cache hits eliminate BPE work entirely:
 
-**Gemini — large text (~54 KB)**
+| scenario | bpe-lite | ai-tokenizer |
+|----------|----------|--------------|
+| Small text, cold (novel input) | ~352k ops/s | ~453k ops/s |
+| Small text, warm (repeated input) | ~1.45M ops/s | ~453k ops/s |
 
-| impl | ops/s | tokens/s | MB/s | note |
-|------|------:|---------:|-----:|------|
-| bpe-lite | **998** | **11.1M** | **52.9** | actual Gemma3 SPM |
-| ai-tokenizer | 215 | 2.40M | 11.4 | o200k BPE — different algorithm, different results |
+For diverse, non-repeating inputs, ai-tokenizer is ~29% faster on very short strings. For any repeated-text workload, bpe-lite is ~3x faster.
 
-ai-tokenizer does not implement Gemini tokenization. The row above uses their o200k encoding on the same input string; it produces different token ids and counts than the Gemini tokenizer, so it is not a real comparison.
+**Initialization**
 
-Numbers vary by machine — run the bench script locally for results on your hardware.
+bpe-lite lazy-loads the gzipped vocab on first encode — one-time cost of ~235ms per provider per process. Negligible for any persistent process. Relevant only for cold serverless invocations that encode once and exit.
 
 ## API
 
@@ -108,7 +155,7 @@ Decodes an array of token ids back to a string.
 
 ### `isWithinTokenLimit(text, limit, provider?)`
 
-Returns the token count if `text` is within `limit` tokens, or `false` if exceeded. More efficient than `encode()` for long texts — the tiktoken engine short-circuits as soon as the limit is crossed.
+Returns the token count if `text` is within `limit` tokens, or `false` if exceeded. More efficient than `encode()` for long texts — short-circuits as soon as the limit is crossed.
 
 ### Tokenizer instances
 
@@ -118,11 +165,6 @@ Returns the token count if `text` is within `limit` tokens, or `false` if exceed
 
 `tiktoken` is accurate for OpenAI but requires Rust/WASM native bindings, which can break in Docker containers, edge runtimes, and serverless environments. `bpe-lite` is pure JavaScript — it runs anywhere Node 18+ runs, with no native compilation step.
 
-## Caveats
-
-- **Anthropic**: Anthropic has not released the Claude tokenizer. The vocab is sourced from [Xenova/claude-tokenizer](https://huggingface.co/Xenova/claude-tokenizer), a community reverse-engineering of the ~65k BPE vocab. NFKC normalization and probe-based merge corrections are applied. Accuracy varies by text type — common prose and code are usually within 10%, but Arabic, Japanese, repeated characters, and some symbol/emoji combinations diverge significantly.
-- **Node version**: Requires Node 18+ for Unicode property escapes (`\p{L}`, `\p{N}`) in the pre-tokenization regex.
-
 ## License
 
 MIT

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "bpe-lite",
-  "version": "0.5.1",
+  "version": "0.5.2",
   "description": "Offline BPE tokenizer for OpenAI, Anthropic, and Gemini — zero dependencies",
   "main": "src/index.js",
   "types": "index.d.ts",

package/src/bpe.js

@@ -6,6 +6,10 @@
  * Lower rank = higher priority in merges.
  */
 
+// Fast check: does a pre-tokenized chunk contain any letter or digit?
+// Non-word chunks (pure symbols, punctuation, spaces) are eligible for batching.
+const WORD_CHAR_RE = /[\p{L}\p{N}]/u;
+
 // Byte → single-byte "binary string" lookup (pre-built at module load)
 const BYTE_STRS = (() => {
   const out = new Array(256);
@@ -170,6 +174,9 @@ function buildPreparedTiktoken(vocabData) {
     specials,
     patternCompiled: compilePretokenizer(pattern),
     normalize: normalize || null,
+    // symbolBatch: merge consecutive non-word regex chunks before BPE so that
+    // cross-character byte merges can fire (matches Claude's no-regex byte BPE).
+    symbolBatch: !!vocabData.symbolBatch,
     // opt A — per-instance chunk cache: chunk string → ids[]
     cache: new Map(),
     // opt B — per-instance grow-only scratch (reused across chunks)
@@ -323,14 +330,41 @@ function encodeTiktokenPrepared(text, prepared) {
       const re = patternCompiled.re;
       re.lastIndex = 0;
       let m;
-      while ((m = re.exec(t)) !== null) {
-        const chunk = m[0];
-        let chunkIds = cache.get(chunk);
-        if (chunkIds === undefined) {
-          chunkIds = bpeChunk(writeChunk(chunk), vocabBin, scratch);
-          cache.set(chunk, chunkIds);
+      if (prepared.symbolBatch) {
+        // Merge consecutive non-word chunks (symbols, punctuation, spaces) into one
+        // BPE input so cross-character byte merges can fire — matches Claude's behavior.
+        let symBuf = null;
+        while ((m = re.exec(t)) !== null) {
+          const chunk = m[0];
+          if (!WORD_CHAR_RE.test(chunk)) {
+            symBuf = symBuf === null ? chunk : symBuf + chunk;
+          } else {
+            if (symBuf !== null) {
+              let symIds = cache.get(symBuf);
+              if (symIds === undefined) { symIds = bpeChunk(writeChunk(symBuf), vocabBin, scratch); cache.set(symBuf, symIds); }
+              for (let i = 0; i < symIds.length; i++) ids.push(symIds[i]);
+              symBuf = null;
+            }
+            let chunkIds = cache.get(chunk);
+            if (chunkIds === undefined) { chunkIds = bpeChunk(writeChunk(chunk), vocabBin, scratch); cache.set(chunk, chunkIds); }
+            for (let i = 0; i < chunkIds.length; i++) ids.push(chunkIds[i]);
+          }
+        }
+        if (symBuf !== null) {
+          let symIds = cache.get(symBuf);
+          if (symIds === undefined) { symIds = bpeChunk(writeChunk(symBuf), vocabBin, scratch); cache.set(symBuf, symIds); }
+          for (let i = 0; i < symIds.length; i++) ids.push(symIds[i]);
+        }
+      } else {
+        while ((m = re.exec(t)) !== null) {
+          const chunk = m[0];
+          let chunkIds = cache.get(chunk);
+          if (chunkIds === undefined) {
+            chunkIds = bpeChunk(writeChunk(chunk), vocabBin, scratch);
+            cache.set(chunk, chunkIds);
+          }
+          for (let i = 0; i < chunkIds.length; i++) ids.push(chunkIds[i]);
         }
-        for (let i = 0; i < chunkIds.length; i++) ids.push(chunkIds[i]);
       }
     } else {
       const chunks = patternCompiled.type === 'none' ? [t] : (t.match(/\S+|\s+/g) || [t]);
@@ -376,14 +410,39 @@ function countTiktokenPrepared(text, prepared) {
       const re = patternCompiled.re;
       re.lastIndex = 0;
       let m;
-      while ((m = re.exec(t)) !== null) {
-        const chunk = m[0];
-        let chunkIds = cache.get(chunk);
-        if (chunkIds === undefined) {
-          chunkIds = bpeChunk(writeChunk(chunk), vocabBin, scratch);
-          cache.set(chunk, chunkIds);
+      if (prepared.symbolBatch) {
+        let symBuf = null;
+        while ((m = re.exec(t)) !== null) {
+          const chunk = m[0];
+          if (!WORD_CHAR_RE.test(chunk)) {
+            symBuf = symBuf === null ? chunk : symBuf + chunk;
+          } else {
+            if (symBuf !== null) {
+              let symIds = cache.get(symBuf);
+              if (symIds === undefined) { symIds = bpeChunk(writeChunk(symBuf), vocabBin, scratch); cache.set(symBuf, symIds); }
+              count += symIds.length;
+              symBuf = null;
+            }
+            let chunkIds = cache.get(chunk);
+            if (chunkIds === undefined) { chunkIds = bpeChunk(writeChunk(chunk), vocabBin, scratch); cache.set(chunk, chunkIds); }
+            count += chunkIds.length;
+          }
+        }
+        if (symBuf !== null) {
+          let symIds = cache.get(symBuf);
+          if (symIds === undefined) { symIds = bpeChunk(writeChunk(symBuf), vocabBin, scratch); cache.set(symBuf, symIds); }
+          count += symIds.length;
+        }
+      } else {
+        while ((m = re.exec(t)) !== null) {
+          const chunk = m[0];
+          let chunkIds = cache.get(chunk);
+          if (chunkIds === undefined) {
+            chunkIds = bpeChunk(writeChunk(chunk), vocabBin, scratch);
+            cache.set(chunk, chunkIds);
+          }
+          count += chunkIds.length;
         }
-        count += chunkIds.length;
       }
     } else {
       const chunks = patternCompiled.type === 'none' ? [t] : (t.match(/\S+|\s+/g) || [t]);

package/src/spm.js

@@ -86,10 +86,8 @@ function encodeSPM(text, vocabData) {
  * Isolated from encodeSegment so V8 can keep this function optimised
  * even when encodeSPMPrepared is called with wildly different text lengths.
  *
- * Segmentation: for a run of N consecutive ▁ chars before a word:
- *   N=1 → one segment [▁word]
- *   N>1 → two segments [▁×(N-1), ▁word]
- * This matches how Gemma BPE merges multi-space tokens (▁▁=138, ▁▁▁=139, …).
+ * Segmentation: each run of ▁ chars plus the following non-▁ word is one segment.
+ * Leading ▁s are included so BPE can naturally merge ▁▁→138, ▁▁▁→139, etc.
  */
 function _scanFromCache(normalized, cache, result) {
   let i = 0;
@@ -113,20 +111,11 @@ function _scanFromCache(normalized, cache, result) {
     // Collect word chars (non-▁)
     while (i < normalized.length && normalized[i] !== SPACE_CHAR) i++;
 
-    // If N>1 spaces, emit (N-1) ▁s as standalone segment first
-    if (spaceCount > 1) {
-      const spaceSeg = normalized.slice(segStart, segStart + spaceCount - 1);
-      const spaceIds = cache.get(spaceSeg);
-      if (spaceIds === undefined) return false;
-      for (let j = 0; j < spaceIds.length; j++) result.push(spaceIds[j]);
-    }
-
-    // Emit ▁ + word as one segment
-    const wordStart = spaceCount > 0 ? segStart + spaceCount - 1 : segStart;
-    const wordSeg = normalized.slice(wordStart, i);
-    const wordIds = cache.get(wordSeg);
-    if (wordIds === undefined) return false;
-    for (let j = 0; j < wordIds.length; j++) result.push(wordIds[j]);
+    // Emit all leading ▁s + word as one segment; BPE merges ▁▁, ▁▁▁, etc.
+    const seg = normalized.slice(segStart, i);
+    const segIds = cache.get(seg);
+    if (segIds === undefined) return false;
+    for (let j = 0; j < segIds.length; j++) result.push(segIds[j]);
   }
   return true;
 }
@@ -163,7 +152,7 @@ function encodeSPMPrepared(text, prepared) {
     if (i === normalized.length) {
       if (spaceCount > 0) {
         const seg = normalized.slice(segStart, i);
-        const segIds = cache.get(seg) ?? _encodeAndCache(seg, vocab, mergeRank, scratch, cache);
+        const segIds = cache.get(seg) ?? _encodeAndCache(seg, vocab, mergeRank, scratch, cache, seedsByAngleBracket);
         for (let j = 0; j < segIds.length; j++) result.push(segIds[j]);
       }
       break;
@@ -171,16 +160,9 @@ function encodeSPMPrepared(text, prepared) {
 
     while (i < normalized.length && normalized[i] !== SPACE_CHAR) i++;
 
-    if (spaceCount > 1) {
-      const spaceSeg = normalized.slice(segStart, segStart + spaceCount - 1);
-      const spaceIds = cache.get(spaceSeg) ?? _encodeAndCache(spaceSeg, vocab, mergeRank, scratch, cache);
-      for (let j = 0; j < spaceIds.length; j++) result.push(spaceIds[j]);
-    }
-
-    const wordStart = spaceCount > 0 ? segStart + spaceCount - 1 : segStart;
-    const wordSeg = normalized.slice(wordStart, i);
-    const wordIds = cache.get(wordSeg) ?? _encodeAndCache(wordSeg, vocab, mergeRank, scratch, cache);
-    for (let j = 0; j < wordIds.length; j++) result.push(wordIds[j]);
+    const seg = normalized.slice(segStart, i);
+    const segIds = cache.get(seg) ?? _encodeAndCache(seg, vocab, mergeRank, scratch, cache, seedsByAngleBracket);
+    for (let j = 0; j < segIds.length; j++) result.push(segIds[j]);
   }
   return result;
 }