bpe-lite 0.3.1 → 0.4.1

package/README.md

@@ -62,6 +62,36 @@ tok.count('Hello, world!');   // → 4
 
 Vocab files are bundled in the package — no network required at runtime or install time.
 
+## Performance
+
+Benchmarked on Node v24, win32/x64, against [js-tiktoken](https://github.com/dqbd/tiktoken) and [ai-tokenizer](https://github.com/nicepkg/ai-tokenizer) (`node --expose-gc scripts/bench.js`).
+
+**OpenAI cl100k — large text (~54 KB)**
+
+| impl | ops/s | tokens/s | MB/s |
+|------|------:|---------:|-----:|
+| bpe-lite | **257** | **3.15M** | **13.6** |
+| ai-tokenizer | 201 | 2.46M | 10.7 |
+| js-tiktoken | 23 | 282k | 1.2 |
+
+**Anthropic — large text (~54 KB)**
+
+| impl | ops/s | tokens/s | MB/s |
+|------|------:|---------:|-----:|
+| bpe-lite | **257** | 3.15M | **13.6** |
+| ai-tokenizer | 253 | **4.62M** | 13.4 |
+
+**Gemini — large text (8 KB)**
+
+| impl | ops/s | tokens/s | MB/s | note |
+|------|------:|---------:|-----:|------|
+| bpe-lite | **3,800** | **6.23M** | **29.7** | actual Gemma3 SPM |
+| ai-tokenizer | 1,220 | 2.01M | 9.6 | o200k BPE — different algorithm, different results |
+
+ai-tokenizer does not implement Gemini tokenization. The row above uses their o200k encoding on the same input string; it produces different token ids and counts than the Gemini tokenizer, so it is not a real comparison.
+
+Numbers vary by machine — run the bench script locally for results on your hardware.
+
 ## API
 
 ### `countTokens(text, provider?)`
@@ -86,12 +116,11 @@ Returns the token count if `text` is within `limit` tokens, or `false` if exceed
 
 ## Why not tiktoken?
 
-`tiktoken` is accurate for OpenAI but requires Rust/WASM native bindings, which can break in Docker containers, edge runtimes, and serverless environments. `bpe-lite` is pure JavaScript — it runs anywhere Node 18+ runs.
+`tiktoken` is accurate for OpenAI but requires Rust/WASM native bindings, which can break in Docker containers, edge runtimes, and serverless environments. `bpe-lite` is pure JavaScript — it runs anywhere Node 18+ runs, with no native compilation step.
 
 ## Caveats
 
 - **Anthropic**: Anthropic has not released the Claude 3+ tokenizer. The cl100k approximation is accurate to ~95% for most text.
-- **Speed**: Pure JS is slower than tiktoken's native implementation. For token counting (not bulk processing) the difference is negligible.
 - **Node version**: Requires Node 18+ for Unicode property escapes (`\p{L}`, `\p{N}`) in the pre-tokenization regex.
 
 ## License

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "bpe-lite",
-  "version": "0.3.1",
+  "version": "0.4.1",
   "description": "Offline BPE tokenizer for OpenAI, Anthropic, and Gemini — zero dependencies",
   "main": "src/index.js",
   "types": "index.d.ts",
@@ -20,8 +20,22 @@
   ],
   "scripts": {
     "build": "node scripts/build-vocabs.js",
+    "bench": "node --expose-gc scripts/bench.js",
+    "compat": "node scripts/compat.js",
     "test": "node tests/run-tests.js"
   },
-  "keywords": ["tokenizer", "bpe", "openai", "anthropic", "gemini", "tokens", "llm"],
-  "license": "MIT"
+  "keywords": [
+    "tokenizer",
+    "bpe",
+    "openai",
+    "anthropic",
+    "gemini",
+    "tokens",
+    "llm"
+  ],
+  "license": "MIT",
+  "devDependencies": {
+    "ai-tokenizer": "^1.0.6",
+    "js-tiktoken": "^1.0.21"
+  }
 }