gigatoken 0.1.1 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d36a860419519d55bf6d7f2fe8c30aa9c4bc25befe98ce810313758be8bf5380
-  data.tar.gz: 0fda91b895be4ef02de36cb16832628203bba09de6c53955997b69611e796bce
+  metadata.gz: eee37e720c1b0fba0305398b731c3bd909af1a6ebc1ae7393e924c66fd629377
+  data.tar.gz: 966c52f87dcff1b0cda6218c45b0cbc5362a02655cb0c570034b4d91249a2518
 SHA512:
-  metadata.gz: 65d3987a0943a5552a8e8b8493de03df9d37ac376cfc9dfc661a6c25a38cc03cd511ed4141b98580e4bd2cff64086802ad25705f9e841fb09cae779bafce0a71
-  data.tar.gz: 34eace98922cb0ef2cf2a4fbecc103131bdfaa26dad459254a1d458efedc28154f3b103dd3200e106f94988d2028aa66370e680c6acee284148f707a0212288b
+  metadata.gz: 75eda453e48567dda4b39765031ed1dab4e5d12715507f0515e6742de2dd0ce67a9c4b5e4c1581664bce956b01cf8978cc9198ee3878253bd26287ba0d558aa2
+  data.tar.gz: 3a4e18c3e8603ae524cfa48752177277297c94343623571ad32bb87434eacfa3456264769f9b3daf5421ad5d615de47502607db17e6ded024e8c9b35a542b9c1

data/Cargo.lock

@@ -735,7 +735,7 @@ dependencies = [
 
 [[package]]
 name = "gigatoken"
-version = "0.9.0"
+version = "0.10.0"
 dependencies = [
  "aho-corasick",
  "arrow-array",
@@ -771,7 +771,7 @@ dependencies = [
 
 [[package]]
 name = "gigatoken-rb"
-version = "0.1.1"
+version = "0.2.1"
 dependencies = [
  "gigatoken",
  "magnus",

data/Cargo.toml

@@ -3,7 +3,7 @@ members = [".", "ext/gigatoken"]
 
 [package]
 name = "gigatoken"
-version = "0.9.0"
+version = "0.10.0"
 edition = "2024"
 license = "MIT"
 

data/README.md

@@ -48,17 +48,50 @@ tok.vocab_size                          # => 50257
 tok.special_tokens                      # => {"<|endoftext|>" => 50256}
 ```
 
-`load` takes a `tokenizer.json` path, a directory holding one, a HuggingFace Hub repo id, or a `.tiktoken` mergeable-ranks file, and dispatches on shape. Hub downloads run over socketry's `async-http` — no Python anywhere. Know what you have? Skip the dispatch:
+`load` takes a `tokenizer.json` path, a directory holding one, a packaged tiktoken encoding name (`r50k_base`, `cl100k_base`, `o200k_base`), a HuggingFace Hub repo id, or a `.tiktoken` mergeable-ranks file, and dispatches on shape. Hub downloads run over socketry's `async-http` — no Python anywhere. Know what you have? Skip the dispatch:
 
 ```ruby
 Gigatoken::Tokenizer.from_file("tokenizer.json")
 Gigatoken::Tokenizer.from_hub("openai-community/gpt2", revision: "main")
-Gigatoken::Tokenizer.from_tiktoken("vocab.tiktoken")
+Gigatoken::Tokenizer.from_tiktoken("cl100k_base.tiktoken", pretokenizer: "gpt4", special_tokens: {"<|endoftext|>" => 100257})
 Gigatoken::Tokenizer.from_json(File.binread("tokenizer.json"))
 ```
 
+A `.tiktoken` file holds mergeable ranks only — its pretokenization scheme and special tokens live in the code that defines the encoding, not the file — so `pretokenizer:` is a required keyword (one of `Gigatoken::Native.pretokenizer_names`: `gpt2`/`r50k`, `gpt4`/`cl100k`, `qwen2`, `qwen35`, `olmo3`, `deepseek_v3`, `o200k`, `nemotron`, `kimi`) and `special_tokens:` defaults to none. Nothing is guessed: an unknown scheme raises `Gigatoken::Error` naming the valid ones, and `Tokenizer.load` on a `.tiktoken` path with no `pretokenizer:` raises rather than silently picking one.
+
 SentencePiece-BPE models (Llama, Gemma, Mistral — any `tokenizer.json` with `byte_fallback: true`) load through the same entry points and pick the right backend automatically. One difference: the SentencePiece core decodes text, so it validates input and raises `Gigatoken::Error` on invalid UTF-8 instead of guessing.
 
+### Packaged tiktoken encodings
+
+`r50k_base`, `cl100k_base`, `o200k_base`, and `o200k_harmony` are vendored directly — mergeable ranks, pretokenizer scheme, and special-token table all shipped inside the gem (`lib/gigatoken/encodings/`; see `PROVENANCE.md` there for exact source URLs and hashes) — so all four resolve by name through both entry points entirely offline: no network access, no writable cache directory. `o200k_harmony` vendors no new file at all: it reuses `o200k_base.tiktoken`'s ranks and the `o200k` scheme verbatim, differing only in its special-token table (10 named control tokens — `<|start|>`, `<|message|>`, `<|end|>`, `<|return|>`, and so on — plus 1081 reserved slots; see `PROVENANCE.md` for the exact table). It's also the one packaged encoding not checked against `tiktoken_ruby`: that gem's 0.0.17 harmony table drops `<|endofprompt|>` where `openai/tiktoken` 0.9.0 keeps it at id 200018, so the oracle is the outlier here — `spec/gigatoken/differential_spec.rb` proves harmony instead by reduction to `o200k_base` plus a pinned special-token table.
+
+```ruby
+Gigatoken::Tokenizer.from_encoding("cl100k_base")
+Gigatoken::Tokenizer.load("cl100k_base")          # same result — packaged names are
+                                                  # checked before the Hub-repo-id shape
+```
+
+`p50k_base` and `p50k_edit` are deliberately not packaged: both load the same non-dense ranks (id 50256 is left free for `<|endoftext|>`), and the rank loader rejects non-dense ranks. Both entry points raise `Gigatoken::Error` explaining that, rather than `load` falling through to the Hub for a name that happens to look like a legacy repo id.
+
+`encode` on a packaged tokenizer honours its special-token table: text containing `<|endoftext|>` (or any other literal special-token string) is tokenized as that special token, not as ordinary text. That matches [`tiktoken`](https://github.com/openai/tiktoken)'s `encode_with_special_tokens`, not its plain `encode`, which treats the same literal as ordinary text — a difference worth knowing if you're tokenizing untrusted input. To get tiktoken's non-honouring default instead, build a tokenizer from the same rank file with an empty special-token table:
+
+```ruby
+entry = Gigatoken::Encodings["cl100k_base"]
+Gigatoken::Tokenizer.from_tiktoken(entry[:rank_file], pretokenizer: entry[:pretokenizer], special_tokens: {})
+```
+
+### Encode-cache budget
+
+Each tokenizer's pretoken cache is capped process-globally (512 MiB per worker by default) so long-lived processes — a Rails worker, say — don't grow it unbounded; a full cache wipes back toward its seed level and refills, which costs a bit of re-computation but never changes encode output. Tune it before building tokenizers you want the new budget to apply to:
+
+```ruby
+Gigatoken.max_cache_bytes            # => 536870912 (512 MiB)
+Gigatoken.max_cache_bytes = 64 << 20 # only tokenizers built after this see the new budget
+Gigatoken.max_cache_bytes = nil      # unbounded
+
+tok.cache_entries                    # => cached pretoken/unit count right now
+```
+
 ### Tokenize files without leaving Rust
 
 `encode_files` reads and tokenizes files entirely on the native side — document contents never materialize as Ruby objects. `.gz` and `.zst` decompress transparently.
@@ -97,6 +130,14 @@ gigatoken validate openai-community/gpt2 owt_train.txt --doc-separator "<|endoft
 
 `bench` reports MB/s and Mtok/s (`--packed` for the fused packed path, `--no-parallel` for the serial core). `validate` confirms native split-and-encode agrees with a Ruby-side split through `encode_batch`.
 
+TOKENIZER also takes a bare `.tiktoken` file, which is where `--pretokenizer` comes in: the file carries mergeable ranks only, so the split regex has to come from the caller, same as `from_tiktoken` above. `--pretokenizer` takes one of the scheme names listed above for `pretokenizer:`:
+
+```bash
+gigatoken bench lib/gigatoken/encodings/cl100k_base.tiktoken README.md --pretokenizer gpt4
+```
+
+Leave it off against a `.tiktoken` TOKENIZER and both commands raise `Gigatoken::Error` naming the valid schemes instead of crashing; for every other TOKENIZER shape (`tokenizer.json`, a packaged name, a Hub repo id) `--pretokenizer` is accepted but ignored.
+
 ## Development
 
 ```bash

data/ext/gigatoken/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "gigatoken-rb"
-version = "0.1.1"
+version = "0.2.1"
 edition = "2021"
 
 [lib]

data/ext/gigatoken/src/cache.rs

@@ -0,0 +1,37 @@
+//! The process-global cache-budget knob, read once per tokenizer
+//! construction. Mirrors the pyo3 mechanism in the core crate's
+//! `src/bindings/cache.rs`, which is `pub(crate)` in a different crate and
+//! so cannot be called directly from here — the encode-cache semantics
+//! themselves live in `gigatoken_rs::Tokenizer::set_max_cache_bytes` and
+//! `gigatoken_rs::SentencePieceBPE::set_max_cache_bytes`.
+
+use gigatoken_rs::{SentencePieceBPE, Tokenizer};
+use std::sync::Mutex;
+
+/// The budget applied to tokenizers constructed after the last
+/// `set_max_cache_bytes` call; `None` = unbounded.
+static MAX_CACHE_BYTES: Mutex<Option<usize>> = Mutex::new(Some(Tokenizer::DEFAULT_MAX_CACHE_BYTES));
+
+/// Apply the global setting to a freshly constructed tokenizer (which
+/// already carries the built-in default, hence the `!=` skip).
+pub(crate) fn apply_max_cache_bytes(mut tokenizer: Tokenizer) -> Tokenizer {
+    let configured = *MAX_CACHE_BYTES.lock().unwrap();
+    if configured != tokenizer.max_cache_bytes() {
+        tokenizer.set_max_cache_bytes(configured);
+    }
+    tokenizer
+}
+
+/// SentencePiece analog of [`apply_max_cache_bytes`].
+pub(crate) fn apply_max_cache_bytes_sp(mut model: SentencePieceBPE) -> SentencePieceBPE {
+    model.set_max_cache_bytes(*MAX_CACHE_BYTES.lock().unwrap());
+    model
+}
+
+pub(crate) fn set_max_cache_bytes(max_bytes: Option<usize>) {
+    *MAX_CACHE_BYTES.lock().unwrap() = max_bytes;
+}
+
+pub(crate) fn get_max_cache_bytes() -> Option<usize> {
+    *MAX_CACHE_BYTES.lock().unwrap()
+}

data/ext/gigatoken/src/lib.rs

@@ -1,4 +1,5 @@
 use gigatoken_rs::load_tokenizer::hf::{self, HfTokenizer};
+use gigatoken_rs::pretokenize::PretokenizerType;
 use magnus::{Error, Module, RString, Ruby, Value, function};
 
 // XZM-WORKAROUND: macOS 26's xzm malloc zone SIGTRAPs on multi-GB Rust chunk
@@ -7,6 +8,7 @@ use magnus::{Error, Module, RString, Ruby, Value, function};
 #[global_allocator]
 static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
 
+mod cache;
 mod error;
 mod gvl;
 mod sentencepiece;
@@ -23,6 +25,26 @@ fn crate_version() -> String {
     env!("CARGO_PKG_VERSION").to_string()
 }
 
+/// The pretokenizer scheme names `BPETokenizer.from_tiktoken` and
+/// `Tokenizer.load` accept — the single source of truth `Gigatoken::Tokenizer.load`
+/// names in its "no scheme for a .tiktoken path" error, so that list can
+/// never drift from the one `PretokenizerType::from_name` actually accepts.
+fn pretokenizer_names() -> Vec<&'static str> {
+    PretokenizerType::NAMES.to_vec()
+}
+
+/// The process-global encode-cache budget in bytes per worker, applied to
+/// tokenizers constructed afterward; `None` removes the bound. Mirrors
+/// pyo3's `set_max_cache_bytes`/`get_max_cache_bytes` (`src/bindings/cache.rs`
+/// in the core crate) over the same public core API.
+fn set_max_cache_bytes(max_bytes: Option<usize>) {
+    cache::set_max_cache_bytes(max_bytes);
+}
+
+fn get_max_cache_bytes() -> Option<usize> {
+    cache::get_max_cache_bytes()
+}
+
 /// Load a tokenizer from in-memory HuggingFace `tokenizer.json` contents.
 /// Returns a `SentencePieceTokenizer` when the model uses `byte_fallback`, a
 /// `BPETokenizer` otherwise — the same split as pyo3's `load_hf_json` and
@@ -43,6 +65,9 @@ fn init(ruby: &Ruby) -> Result<(), Error> {
     let native = gigatoken.define_module("Native")?;
     native.define_module_function("crate_version", function!(crate_version, 0))?;
     native.define_module_function("load_hf_json", function!(load_hf_json, 1))?;
+    native.define_module_function("pretokenizer_names", function!(pretokenizer_names, 0))?;
+    native.define_module_function("set_max_cache_bytes", function!(set_max_cache_bytes, 1))?;
+    native.define_module_function("get_max_cache_bytes", function!(get_max_cache_bytes, 0))?;
     sources::init(ruby, native)?;
     tokenizer::init(ruby, native)?;
     sentencepiece::init(ruby, native)?;

data/ext/gigatoken/src/sentencepiece.rs

@@ -13,7 +13,7 @@
 //! `Gigatoken::Error` instead of ever calling `str::from_utf8_unchecked` on
 //! Ruby-supplied bytes.
 
-use std::cell::RefCell;
+use std::sync::Mutex;
 
 use gigatoken_rs::input::file_source::DocFormat;
 use gigatoken_rs::{EncodeState, SentencePieceBPE, sp_encode_docs_ragged, sp_encode_files_docs, sp_encode_files_docs_serial};
@@ -35,17 +35,24 @@ fn require_utf8<'a>(ruby: &Ruby, bytes: &'a [u8]) -> Result<&'a str, Error> {
 #[magnus::wrap(class = "Gigatoken::Native::SentencePieceTokenizer", free_immediately, size)]
 pub struct SentencePieceTokenizer {
     // `SentencePieceBPE`'s encode methods take `&self` (only `EncodeState`
-    // is mutated), so this `RefCell` is never `borrow_mut`'d — see the
-    // builder report's DISAGREEMENTS for why it's here anyway.
-    tokenizer: RefCell<SentencePieceBPE>,
-    state: RefCell<EncodeState>,
+    // is mutated), so the model needs no interior mutability at all — every
+    // path here reads it, including the ones that release the GVL.
+    tokenizer: SentencePieceBPE,
+    // The one mutable piece. A `Mutex` rather than a `RefCell` so the wrapped
+    // object is `Sync`: Ruby hands the same instance to every thread, and a
+    // `RefCell` shared that way is unsound (see `BPETokenizer`'s lock).
+    state: Mutex<EncodeState>,
 }
 
 impl SentencePieceTokenizer {
+    /// Wrap a loaded model, applying the global cache budget to it and to
+    /// the single-document encode state — mirrors pyo3's `with_model`.
     pub(crate) fn from_tokenizer(tokenizer: SentencePieceBPE) -> Self {
+        let tokenizer = crate::cache::apply_max_cache_bytes_sp(tokenizer);
+        let state = EncodeState::with_budget(tokenizer.max_cache_bytes());
         Self {
-            tokenizer: RefCell::new(tokenizer),
-            state: RefCell::new(EncodeState::new()),
+            tokenizer,
+            state: Mutex::new(state),
         }
     }
 
@@ -54,8 +61,8 @@ impl SentencePieceTokenizer {
         let bytes = unsafe { input.as_slice() };
         let text = require_utf8(ruby, bytes)?;
         let mut ids: Vec<u32> = Vec::new();
-        let mut state = rb_self.state.borrow_mut();
-        rb_self.tokenizer.borrow().encode_raw_cb(&mut state, text, &mut |tokens| {
+        let mut state = rb_self.state.lock().unwrap_or_else(|e| e.into_inner());
+        rb_self.tokenizer.encode_raw_cb(&mut state, text, &mut |tokens| {
             ids.extend(tokens.iter().map(|&t| u32::from(t)))
         });
         Ok(ids)
@@ -80,8 +87,7 @@ impl SentencePieceTokenizer {
             })
             .collect::<Result<_, _>>()?;
         let doc_refs: Vec<&str> = docs.iter().map(String::as_str).collect();
-        let tokenizer = rb_self.tokenizer.borrow();
-        let tokenizer: &SentencePieceBPE = &tokenizer;
+        let tokenizer: &SentencePieceBPE = &rb_self.tokenizer;
         Ok(without_gvl(|| sp_encode_docs_ragged(tokenizer, &doc_refs)))
     }
 
@@ -126,8 +132,7 @@ impl SentencePieceTokenizer {
             }
         }
 
-        let tokenizer = rb_self.tokenizer.borrow();
-        let tokenizer: &SentencePieceBPE = &tokenizer;
+        let tokenizer: &SentencePieceBPE = &rb_self.tokenizer;
         let encoded: std::io::Result<(Vec<u32>, Vec<i64>)> = without_gvl(|| {
             sources::encode_files_ragged(&source, parallel, |files, format| {
                 for &region in files {
@@ -162,16 +167,16 @@ impl SentencePieceTokenizer {
     fn decode(ruby: &Ruby, rb_self: &Self, tokens: RArray) -> Result<RString, Error> {
         let ids: Vec<u32> = tokens.to_vec()?;
         let ids: Vec<_> = ids.into_iter().map(Into::into).collect();
-        let bytes = rb_self.tokenizer.borrow().decode(&ids);
+        let bytes = rb_self.tokenizer.decode(&ids);
         Ok(binary_string(ruby, &bytes))
     }
 
     fn vocab_size(&self) -> usize {
-        self.tokenizer.borrow().vocab_size()
+        self.tokenizer.vocab_size()
     }
 
     fn vocab(ruby: &Ruby, rb_self: &Self) -> Result<RHash, Error> {
-        let tokenizer = rb_self.tokenizer.borrow();
+        let tokenizer = &rb_self.tokenizer;
         let hash = ruby.hash_new();
         for (id, bytes) in tokenizer.vocab_entries() {
             hash.aset(id, binary_string(ruby, bytes))?;
@@ -180,7 +185,7 @@ impl SentencePieceTokenizer {
     }
 
     fn merges(ruby: &Ruby, rb_self: &Self) -> Result<RArray, Error> {
-        let tokenizer = rb_self.tokenizer.borrow();
+        let tokenizer = &rb_self.tokenizer;
         let entries = tokenizer.merge_entries();
         let result = ruby.ary_new_capa(entries.len());
         for (a, b) in entries {
@@ -188,6 +193,12 @@ impl SentencePieceTokenizer {
         }
         Ok(result)
     }
+
+    /// Cached unit entries on the single-document `encode` path's state
+    /// (batch encoders are per-call); see `BPETokenizer::cache_entries`.
+    fn cache_entries(&self) -> usize {
+        self.state.lock().unwrap_or_else(|e| e.into_inner()).cache_size()
+    }
 }
 
 pub fn init(ruby: &Ruby, native: RModule) -> Result<(), Error> {
@@ -201,5 +212,6 @@ pub fn init(ruby: &Ruby, native: RModule) -> Result<(), Error> {
     class.define_method("vocab_size", method!(SentencePieceTokenizer::vocab_size, 0))?;
     class.define_method("vocab", method!(SentencePieceTokenizer::vocab, 0))?;
     class.define_method("merges", method!(SentencePieceTokenizer::merges, 0))?;
+    class.define_method("cache_entries", method!(SentencePieceTokenizer::cache_entries, 0))?;
     Ok(())
 }

data/ext/gigatoken/src/tokenizer.rs

@@ -3,12 +3,13 @@
 //! the core crate's `src/lib.rs` (the `python` feature), minus the
 //! numpy/awkward-array machinery that has no Ruby analog.
 
-use std::cell::RefCell;
+use std::sync::RwLock;
 use std::collections::{HashMap, HashSet};
 use std::os::raw::c_long;
 
 use gigatoken_rs::load_tokenizer::hf::HfTokenizer;
 use gigatoken_rs::load_tokenizer::{hf, tiktoken};
+use gigatoken_rs::pretokenize::PretokenizerType;
 use gigatoken_rs::{
     GatherBuf, GatherOutcome, Tokenizer, WorkerPool, encode_docs_into, encode_docs_ragged,
     encode_files_docs, encode_files_docs_serial,
@@ -352,14 +353,14 @@ fn marshal_inputs(inputs: RArray) -> Result<InputDocs, Error> {
 
 #[magnus::wrap(class = "Gigatoken::Native::BPETokenizer", free_immediately, size)]
 pub struct BPETokenizer {
-    tokenizer: RefCell<Tokenizer>,
+    tokenizer: RwLock<Tokenizer>,
     workers: WorkerPool,
 }
 
 impl BPETokenizer {
     pub(crate) fn from_tokenizer(tokenizer: Tokenizer) -> Self {
         Self {
-            tokenizer: RefCell::new(tokenizer),
+            tokenizer: RwLock::new(crate::cache::apply_max_cache_bytes(tokenizer)),
             workers: WorkerPool::new(),
         }
     }
@@ -379,21 +380,108 @@ impl BPETokenizer {
         }
     }
 
-    fn from_tiktoken(ruby: &Ruby, path: String) -> Result<Self, Error> {
-        match tiktoken::load_tiktoken(&path) {
+    /// Load from a .tiktoken rank file with the named pretokenizer scheme
+    /// and a {content => id} mapping of special tokens. The file carries
+    /// neither, so both are the caller's to supply — see
+    /// `Gigatoken::Tokenizer.from_tiktoken`, which knows them for the
+    /// encodings OpenAI publishes.
+    fn from_tiktoken(
+        ruby: &Ruby,
+        path: String,
+        pretokenizer: String,
+        special_tokens: HashMap<String, u32>,
+    ) -> Result<Self, Error> {
+        let scheme = PretokenizerType::from_name(&pretokenizer).ok_or_else(|| {
+            raise(
+                ruby,
+                format!(
+                    "unknown pretokenizer scheme {pretokenizer:?}; expected one of {}",
+                    PretokenizerType::NAMES.join(", ")
+                ),
+            )
+        })?;
+        let special_tokens: Vec<(String, u32)> = special_tokens.into_iter().collect();
+        match tiktoken::load_tiktoken(&path, scheme, special_tokens) {
             Ok(tokenizer) => Ok(Self::from_tokenizer(tokenizer)),
             Err(e) => Err(raise(ruby, e.to_string())),
         }
     }
 
+    /// Shared access to the tokenizer, for everything that only reads it —
+    /// the batch paths, `decode`, `vocab`, `merges`, the size accessors.
+    ///
+    /// Readers never exclude each other, which is what makes the long holds
+    /// safe: `encode_batch`/`encode_files` keep this across a GVL release
+    /// (they run the core pool over `&Tokenizer`), and any other Ruby thread
+    /// reading meanwhile just proceeds. The only exclusive holder is
+    /// [`Self::encode`], which is short.
+    fn read_tokenizer(&self) -> std::sync::RwLockReadGuard<'_, Tokenizer> {
+        self.tokenizer.read().unwrap_or_else(|e| e.into_inner())
+    }
+
+    /// Encode one string, mutating the tokenizer's pretoken cache — the only
+    /// exclusive use of the lock.
+    ///
+    /// Uncontended (every single-threaded caller, and the common case under
+    /// threads) this takes the fast path: grab the write guard, encode against
+    /// the Ruby string's own bytes, never release the GVL. Identical cost to
+    /// the pre-lock version plus one uncontended atomic.
+    ///
+    /// Contended, the writer is waiting on a reader that will hold the lock
+    /// for as long as a batch encode takes. Blocking there while holding the
+    /// GVL would stall every other Ruby thread in the VM, so instead the input
+    /// is copied and the whole wait-and-encode moves inside `without_gvl`.
+    /// The copy is what makes that sound: no Ruby `VALUE` and no `RString`
+    /// buffer may outlive the release (see `marshal_inputs`), and the guard is
+    /// taken and dropped inside the closure, so it never crosses OS threads
+    /// even when the scheduler offloads it (see `gvl`).
     fn encode(&self, input: RString) -> Vec<u32> {
-        // SAFETY: read-only, for the duration of this synchronous call.
-        let bytes = unsafe { input.as_slice() };
-        let mut out = Vec::new();
-        self.tokenizer
-            .borrow_mut()
-            .encode_with_added_tokens_flat(bytes, &mut out);
-        out
+        if let Ok(mut tokenizer) = self.tokenizer.try_write() {
+            // SAFETY: read-only, for the duration of this synchronous call,
+            // with no GVL release in between.
+            let bytes = unsafe { input.as_slice() };
+            let mut out = Vec::new();
+            tokenizer.encode_with_added_tokens_flat(bytes, &mut out);
+            return out;
+        }
+
+        self.encode_contended(input)
+    }
+
+    /// The contended half of [`Self::encode`], outlined and `#[cold]`.
+    ///
+    /// Keeping this out of `encode`'s body is a measured requirement, not
+    /// tidiness: the workspace builds with `lto = "fat"`, so the core encode
+    /// routine inlines into `encode`, and inlining is sensitive to the caller's
+    /// size. Written inline, this second path measured slower on single
+    /// encodes — no lock overhead, just a flipped inlining decision. Outlined,
+    /// `encode`'s hot body is the original three lines behind a `try_write`.
+    ///
+    /// Before you re-inline this "to simplify": rerun the evidence rather than
+    /// trusting a number. `ruby -Ilib bench/encode_ab.rb` with the attributes
+    /// stripped and again with them restored, and read
+    /// `docs/rb/benchmarks.md` first — no size resolves this on the hardware
+    /// measured so far. The instrument is honest (an interleaved same-build
+    /// run never calls a size faster or slower, at any size) and has power to
+    /// catch a couple-percent effect reliably, but the attributes' real
+    /// effect is small enough that even the tightest floor (medium,
+    /// well under 3%) swallows it more often than not. Removing the
+    /// attributes measures slower at medium and large, in the direction the
+    /// outlining was added to prevent, but neither delta clears the noise
+    /// floor. Keep it outlined on that direction and on the original
+    /// inline-regression measurement, not on a pinned-down magnitude.
+    #[cold]
+    #[inline(never)]
+    fn encode_contended(&self, input: RString) -> Vec<u32> {
+        // SAFETY: copied before any GVL release, so nothing Ruby-owned is
+        // captured by the closure below.
+        let owned = unsafe { input.as_slice() }.to_vec();
+        without_gvl(move || {
+            let mut tokenizer = self.tokenizer.write().unwrap_or_else(|e| e.into_inner());
+            let mut out = Vec::new();
+            tokenizer.encode_with_added_tokens_flat(&owned, &mut out);
+            out
+        })
     }
 
     /// Encode a batch on the core worker pool, with the GVL released for the
@@ -404,7 +492,7 @@ impl BPETokenizer {
     fn encode_batch_ragged(rb_self: &Self, inputs: RArray) -> Result<(Vec<u32>, Vec<i64>), Error> {
         let marshaled = marshal_inputs(inputs)?;
         let doc_slices = marshaled.as_slices();
-        let tokenizer = rb_self.tokenizer.borrow();
+        let tokenizer = rb_self.read_tokenizer();
         let tokenizer: &Tokenizer = &tokenizer;
         let workers = &rb_self.workers;
         Ok(without_gvl(|| encode_docs_ragged(workers, tokenizer, &doc_slices)))
@@ -453,7 +541,7 @@ impl BPETokenizer {
         // the duration of the gather below (see the allocation above).
         let dest = unsafe { GatherBuf::new(ptr, total_bytes) };
 
-        let tokenizer = rb_self.tokenizer.borrow();
+        let tokenizer = rb_self.read_tokenizer();
         let tokenizer: &Tokenizer = &tokenizer;
         let workers = &rb_self.workers;
         match without_gvl(|| encode_docs_into(workers, tokenizer, &doc_slices, dest)) {
@@ -494,7 +582,7 @@ impl BPETokenizer {
         };
 
         let source = sources::resolve(ruby, source)?;
-        let tokenizer = rb_self.tokenizer.borrow();
+        let tokenizer = rb_self.read_tokenizer();
         let tokenizer: &Tokenizer = &tokenizer;
         let workers = &rb_self.workers;
         let encoded: std::io::Result<(Vec<u32>, Vec<i64>)> = without_gvl(|| {
@@ -526,16 +614,16 @@ impl BPETokenizer {
     fn decode(ruby: &Ruby, rb_self: &Self, tokens: RArray) -> Result<RString, Error> {
         let ids: Vec<u32> = tokens.to_vec()?;
         let ids: Vec<_> = ids.into_iter().map(Into::into).collect();
-        let bytes: Vec<u8> = rb_self.tokenizer.borrow().decode(&ids).collect();
+        let bytes: Vec<u8> = rb_self.read_tokenizer().decode(&ids).collect();
         Ok(binary_string(ruby, &bytes))
     }
 
     fn vocab_size(&self) -> usize {
-        self.tokenizer.borrow().vocab_size()
+        self.read_tokenizer().vocab_size()
     }
 
     fn vocab(ruby: &Ruby, rb_self: &Self) -> Result<RHash, Error> {
-        let tokenizer = rb_self.tokenizer.borrow();
+        let tokenizer = rb_self.read_tokenizer();
         let hash = ruby.hash_new();
         for (id, bytes) in tokenizer.vocab_entries() {
             hash.aset(id, binary_string(ruby, bytes))?;
@@ -544,7 +632,7 @@ impl BPETokenizer {
     }
 
     fn merges(ruby: &Ruby, rb_self: &Self) -> Result<RArray, Error> {
-        let tokenizer = rb_self.tokenizer.borrow();
+        let tokenizer = rb_self.read_tokenizer();
         let entries = tokenizer.merge_entries();
         let result = ruby.ary_new_capa(entries.len());
         for (a, b) in entries {
@@ -552,12 +640,19 @@ impl BPETokenizer {
         }
         Ok(result)
     }
+
+    /// Cached pretoken entries on this tokenizer: grows as text is encoded,
+    /// drops back toward vocab-seed level when a budgeted cache wipes (see
+    /// `Gigatoken.max_cache_bytes`).
+    fn cache_entries(&self) -> usize {
+        self.read_tokenizer().cache_entries()
+    }
 }
 
 pub fn init(ruby: &Ruby, native: RModule) -> Result<(), Error> {
     let class: RClass = native.define_class("BPETokenizer", ruby.class_object())?;
     class.define_singleton_method("from_hf_json", function!(BPETokenizer::from_hf_json, 1))?;
-    class.define_singleton_method("from_tiktoken", function!(BPETokenizer::from_tiktoken, 1))?;
+    class.define_singleton_method("from_tiktoken", function!(BPETokenizer::from_tiktoken, 3))?;
     class.define_method("encode", method!(BPETokenizer::encode, 1))?;
     class.define_method("encode_batch", method!(BPETokenizer::encode_batch, 1))?;
     class.define_method("encode_batch_packed", method!(BPETokenizer::encode_batch_packed, 1))?;
@@ -567,5 +662,6 @@ pub fn init(ruby: &Ruby, native: RModule) -> Result<(), Error> {
     class.define_method("vocab_size", method!(BPETokenizer::vocab_size, 0))?;
     class.define_method("vocab", method!(BPETokenizer::vocab, 0))?;
     class.define_method("merges", method!(BPETokenizer::merges, 0))?;
+    class.define_method("cache_entries", method!(BPETokenizer::cache_entries, 0))?;
     Ok(())
 }

data/lib/gigatoken/cli/bench.rb

@@ -10,19 +10,20 @@ module Gigatoken
     class Bench < Dry::CLI::Command
       desc "Measure the time to encode FILES with TOKENIZER"
 
-      argument :tokenizer, required: true, desc: "tokenizer.json path or directory, HuggingFace repo id, or .tiktoken file"
+      argument :tokenizer, required: true, desc: "tokenizer.json path or directory, packaged encoding name (e.g. \"cl100k_base\"), HuggingFace repo id, or .tiktoken file (requires --pretokenizer)"
       argument :files, type: :array, required: true, desc: "UTF-8 text files to encode"
 
       option :doc_separator, desc: 'document separator to split the files on, e.g. "<|endoftext|>"; whole files are single documents otherwise'
       option :limit_bytes, default: "none", desc: "cap the bytes benchmarked, e.g. 100MB; 'none' for everything (parallel mode only — ignored with --no-parallel)"
       option :parallel, type: :boolean, default: true, desc: "encode on the worker pool instead of the fused serial core path"
       option :packed, type: :boolean, default: false, desc: "time the fused native file path with a packed IO::Buffer result instead of per-document Ruby arrays (ignores --limit-bytes)"
+      option :pretokenizer, desc: "pretokenizer scheme, required when TOKENIZER is a .tiktoken file (one of #{Native.pretokenizer_names.join(", ")}); ignored otherwise"
 
-      def call(tokenizer:, files:, doc_separator: nil, limit_bytes: "none", parallel: true, packed: false, **)
+      def call(tokenizer:, files:, doc_separator: nil, limit_bytes: "none", parallel: true, packed: false, pretokenizer: nil, **)
         limit = Support.parse_size(limit_bytes)
         out.puts "#{label("cpu")}: #{Support.cpu_info}"
 
-        gt_tokenizer = Support.load_tokenizer(tokenizer)
+        gt_tokenizer = Support.load_tokenizer(tokenizer, pretokenizer: pretokenizer)
 
         start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
         if packed

data/lib/gigatoken/cli/support.rb

@@ -14,10 +14,13 @@ module Gigatoken
       private_constant :SIZE_PATTERN
 
       class << self
-        # Load TOKENIZER: a tokenizer.json path/directory, a HuggingFace
-        # repo id, or a .tiktoken file — see Gigatoken::Tokenizer.load.
-        def load_tokenizer(spec)
-          Gigatoken::Tokenizer.load(spec)
+        # Load TOKENIZER: a tokenizer.json path/directory, a packaged
+        # tiktoken encoding name, a HuggingFace repo id, or a .tiktoken file
+        # — see Gigatoken::Tokenizer.load. `pretokenizer:` is forwarded
+        # as-is; it's required for a bare .tiktoken path (which carries no
+        # scheme of its own) and ignored for the other shapes.
+        def load_tokenizer(spec, pretokenizer: nil)
+          Gigatoken::Tokenizer.load(spec, pretokenizer: pretokenizer)
         end
 
         # Parse a decimal byte size like "100MB", "2.5GB", or "1000000";

data/lib/gigatoken/cli/validate.rb

@@ -13,13 +13,14 @@ module Gigatoken
     class Validate < Dry::CLI::Command
       desc "Check that encode_files agrees with a Ruby-side split plus encode_batch on FILES"
 
-      argument :tokenizer, required: true, desc: "tokenizer.json path or directory, HuggingFace repo id, or .tiktoken file"
+      argument :tokenizer, required: true, desc: "tokenizer.json path or directory, packaged encoding name (e.g. \"cl100k_base\"), HuggingFace repo id, or .tiktoken file (requires --pretokenizer)"
       argument :files, type: :array, required: true, desc: "UTF-8 text files to encode"
 
       option :doc_separator, desc: 'document separator to split the files on, e.g. "<|endoftext|>"; whole files are single documents otherwise'
+      option :pretokenizer, desc: "pretokenizer scheme, required when TOKENIZER is a .tiktoken file (one of #{Native.pretokenizer_names.join(", ")}); ignored otherwise"
 
-      def call(tokenizer:, files:, doc_separator: nil, **)
-        gt_tokenizer = Support.load_tokenizer(tokenizer)
+      def call(tokenizer:, files:, doc_separator: nil, pretokenizer: nil, **)
+        gt_tokenizer = Support.load_tokenizer(tokenizer, pretokenizer: pretokenizer)
 
         via_files = gt_tokenizer.encode_files(Support.text_file_source(files, doc_separator))
         via_batch = gt_tokenizer.encode_batch(Support.split_docs(files, doc_separator))