gigatoken 0.1.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d36a860419519d55bf6d7f2fe8c30aa9c4bc25befe98ce810313758be8bf5380
-  data.tar.gz: 0fda91b895be4ef02de36cb16832628203bba09de6c53955997b69611e796bce
+  metadata.gz: 071b3abb52387367f392e88e50d92c7413f5f6c70dae24accf6d378ad6e78fd8
+  data.tar.gz: 26da76745e877b6ee30821ab5fcae50aefdb6ecddcd660e9b8c7b48b4d94bd06
 SHA512:
-  metadata.gz: 65d3987a0943a5552a8e8b8493de03df9d37ac376cfc9dfc661a6c25a38cc03cd511ed4141b98580e4bd2cff64086802ad25705f9e841fb09cae779bafce0a71
-  data.tar.gz: 34eace98922cb0ef2cf2a4fbecc103131bdfaa26dad459254a1d458efedc28154f3b103dd3200e106f94988d2028aa66370e680c6acee284148f707a0212288b
+  metadata.gz: 402d1e74262ffd0d4ff8d6f6162c944747b75c567249f788fdf46d2c5e09cd001f1cd66d037156848023fb5aad62087c73def5c4d3134d2aba7a898575ce2631
+  data.tar.gz: c43c48249a2c7bd0d8146e20d830975dae6e3acec93b1bc42e125fdd72050da38d02d0a10068b6ad208c85bd1790a54629b3b2394c1b12762feddce834909995

data/Cargo.lock

@@ -735,7 +735,7 @@ dependencies = [
 
 [[package]]
 name = "gigatoken"
-version = "0.9.0"
+version = "0.10.0"
 dependencies = [
  "aho-corasick",
  "arrow-array",
@@ -771,7 +771,7 @@ dependencies = [
 
 [[package]]
 name = "gigatoken-rb"
-version = "0.1.1"
+version = "0.2.0"
 dependencies = [
  "gigatoken",
  "magnus",

data/Cargo.toml

@@ -3,7 +3,7 @@ members = [".", "ext/gigatoken"]
 
 [package]
 name = "gigatoken"
-version = "0.9.0"
+version = "0.10.0"
 edition = "2024"
 license = "MIT"
 

data/README.md

@@ -48,17 +48,50 @@ tok.vocab_size                          # => 50257
 tok.special_tokens                      # => {"<|endoftext|>" => 50256}
 ```
 
-`load` takes a `tokenizer.json` path, a directory holding one, a HuggingFace Hub repo id, or a `.tiktoken` mergeable-ranks file, and dispatches on shape. Hub downloads run over socketry's `async-http` — no Python anywhere. Know what you have? Skip the dispatch:
+`load` takes a `tokenizer.json` path, a directory holding one, a packaged tiktoken encoding name (`r50k_base`, `cl100k_base`, `o200k_base`), a HuggingFace Hub repo id, or a `.tiktoken` mergeable-ranks file, and dispatches on shape. Hub downloads run over socketry's `async-http` — no Python anywhere. Know what you have? Skip the dispatch:
 
 ```ruby
 Gigatoken::Tokenizer.from_file("tokenizer.json")
 Gigatoken::Tokenizer.from_hub("openai-community/gpt2", revision: "main")
-Gigatoken::Tokenizer.from_tiktoken("vocab.tiktoken")
+Gigatoken::Tokenizer.from_tiktoken("cl100k_base.tiktoken", pretokenizer: "gpt4", special_tokens: {"<|endoftext|>" => 100257})
 Gigatoken::Tokenizer.from_json(File.binread("tokenizer.json"))
 ```
 
+A `.tiktoken` file holds mergeable ranks only — its pretokenization scheme and special tokens live in the code that defines the encoding, not the file — so `pretokenizer:` is a required keyword (one of `Gigatoken::Native.pretokenizer_names`: `gpt2`/`r50k`, `gpt4`/`cl100k`, `qwen2`, `qwen35`, `olmo3`, `deepseek_v3`, `o200k`, `nemotron`, `kimi`) and `special_tokens:` defaults to none. Nothing is guessed: an unknown scheme raises `Gigatoken::Error` naming the valid ones, and `Tokenizer.load` on a `.tiktoken` path with no `pretokenizer:` raises rather than silently picking one.
+
 SentencePiece-BPE models (Llama, Gemma, Mistral — any `tokenizer.json` with `byte_fallback: true`) load through the same entry points and pick the right backend automatically. One difference: the SentencePiece core decodes text, so it validates input and raises `Gigatoken::Error` on invalid UTF-8 instead of guessing.
 
+### Packaged tiktoken encodings
+
+`r50k_base`, `cl100k_base`, `o200k_base`, and `o200k_harmony` are vendored directly — mergeable ranks, pretokenizer scheme, and special-token table all shipped inside the gem (`lib/gigatoken/encodings/`; see `PROVENANCE.md` there for exact source URLs and hashes) — so all four resolve by name through both entry points entirely offline: no network access, no writable cache directory. `o200k_harmony` vendors no new file at all: it reuses `o200k_base.tiktoken`'s ranks and the `o200k` scheme verbatim, differing only in its special-token table (10 named control tokens — `<|start|>`, `<|message|>`, `<|end|>`, `<|return|>`, and so on — plus 1081 reserved slots; see `PROVENANCE.md` for the exact table). It's also the one packaged encoding not checked against `tiktoken_ruby`: that gem's 0.0.17 harmony table drops `<|endofprompt|>` where `openai/tiktoken` 0.9.0 keeps it at id 200018, so the oracle is the outlier here — `spec/gigatoken/differential_spec.rb` proves harmony instead by reduction to `o200k_base` plus a pinned special-token table.
+
+```ruby
+Gigatoken::Tokenizer.from_encoding("cl100k_base")
+Gigatoken::Tokenizer.load("cl100k_base")          # same result — packaged names are
+                                                  # checked before the Hub-repo-id shape
+```
+
+`p50k_base` and `p50k_edit` are deliberately not packaged: both load the same non-dense ranks (id 50256 is left free for `<|endoftext|>`), and the rank loader rejects non-dense ranks. Both entry points raise `Gigatoken::Error` explaining that, rather than `load` falling through to the Hub for a name that happens to look like a legacy repo id.
+
+`encode` on a packaged tokenizer honours its special-token table: text containing `<|endoftext|>` (or any other literal special-token string) is tokenized as that special token, not as ordinary text. That matches [`tiktoken`](https://github.com/openai/tiktoken)'s `encode_with_special_tokens`, not its plain `encode`, which treats the same literal as ordinary text — a difference worth knowing if you're tokenizing untrusted input. To get tiktoken's non-honouring default instead, build a tokenizer from the same rank file with an empty special-token table:
+
+```ruby
+entry = Gigatoken::Encodings["cl100k_base"]
+Gigatoken::Tokenizer.from_tiktoken(entry[:rank_file], pretokenizer: entry[:pretokenizer], special_tokens: {})
+```
+
+### Encode-cache budget
+
+Each tokenizer's pretoken cache is capped process-globally (512 MiB per worker by default) so long-lived processes — a Rails worker, say — don't grow it unbounded; a full cache wipes back toward its seed level and refills, which costs a bit of re-computation but never changes encode output. Tune it before building tokenizers you want the new budget to apply to:
+
+```ruby
+Gigatoken.max_cache_bytes            # => 536870912 (512 MiB)
+Gigatoken.max_cache_bytes = 64 << 20 # only tokenizers built after this see the new budget
+Gigatoken.max_cache_bytes = nil      # unbounded
+
+tok.cache_entries                    # => cached pretoken/unit count right now
+```
+
 ### Tokenize files without leaving Rust
 
 `encode_files` reads and tokenizes files entirely on the native side — document contents never materialize as Ruby objects. `.gz` and `.zst` decompress transparently.
@@ -97,6 +130,14 @@ gigatoken validate openai-community/gpt2 owt_train.txt --doc-separator "<|endoft
 
 `bench` reports MB/s and Mtok/s (`--packed` for the fused packed path, `--no-parallel` for the serial core). `validate` confirms native split-and-encode agrees with a Ruby-side split through `encode_batch`.
 
+TOKENIZER also takes a bare `.tiktoken` file, which is where `--pretokenizer` comes in: the file carries mergeable ranks only, so the split regex has to come from the caller, same as `from_tiktoken` above. `--pretokenizer` takes one of the scheme names listed above for `pretokenizer:`:
+
+```bash
+gigatoken bench lib/gigatoken/encodings/cl100k_base.tiktoken README.md --pretokenizer gpt4
+```
+
+Leave it off against a `.tiktoken` TOKENIZER and both commands raise `Gigatoken::Error` naming the valid schemes instead of crashing; for every other TOKENIZER shape (`tokenizer.json`, a packaged name, a Hub repo id) `--pretokenizer` is accepted but ignored.
+
 ## Development
 
 ```bash

data/ext/gigatoken/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "gigatoken-rb"
-version = "0.1.1"
+version = "0.2.0"
 edition = "2021"
 
 [lib]

data/ext/gigatoken/src/cache.rs

@@ -0,0 +1,37 @@
+//! The process-global cache-budget knob, read once per tokenizer
+//! construction. Mirrors the pyo3 mechanism in the core crate's
+//! `src/bindings/cache.rs`, which is `pub(crate)` in a different crate and
+//! so cannot be called directly from here — the encode-cache semantics
+//! themselves live in `gigatoken_rs::Tokenizer::set_max_cache_bytes` and
+//! `gigatoken_rs::SentencePieceBPE::set_max_cache_bytes`.
+
+use gigatoken_rs::{SentencePieceBPE, Tokenizer};
+use std::sync::Mutex;
+
+/// The budget applied to tokenizers constructed after the last
+/// `set_max_cache_bytes` call; `None` = unbounded.
+static MAX_CACHE_BYTES: Mutex<Option<usize>> = Mutex::new(Some(Tokenizer::DEFAULT_MAX_CACHE_BYTES));
+
+/// Apply the global setting to a freshly constructed tokenizer (which
+/// already carries the built-in default, hence the `!=` skip).
+pub(crate) fn apply_max_cache_bytes(mut tokenizer: Tokenizer) -> Tokenizer {
+    let configured = *MAX_CACHE_BYTES.lock().unwrap();
+    if configured != tokenizer.max_cache_bytes() {
+        tokenizer.set_max_cache_bytes(configured);
+    }
+    tokenizer
+}
+
+/// SentencePiece analog of [`apply_max_cache_bytes`].
+pub(crate) fn apply_max_cache_bytes_sp(mut model: SentencePieceBPE) -> SentencePieceBPE {
+    model.set_max_cache_bytes(*MAX_CACHE_BYTES.lock().unwrap());
+    model
+}
+
+pub(crate) fn set_max_cache_bytes(max_bytes: Option<usize>) {
+    *MAX_CACHE_BYTES.lock().unwrap() = max_bytes;
+}
+
+pub(crate) fn get_max_cache_bytes() -> Option<usize> {
+    *MAX_CACHE_BYTES.lock().unwrap()
+}

data/ext/gigatoken/src/lib.rs

@@ -1,4 +1,5 @@
 use gigatoken_rs::load_tokenizer::hf::{self, HfTokenizer};
+use gigatoken_rs::pretokenize::PretokenizerType;
 use magnus::{Error, Module, RString, Ruby, Value, function};
 
 // XZM-WORKAROUND: macOS 26's xzm malloc zone SIGTRAPs on multi-GB Rust chunk
@@ -7,6 +8,7 @@ use magnus::{Error, Module, RString, Ruby, Value, function};
 #[global_allocator]
 static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
 
+mod cache;
 mod error;
 mod gvl;
 mod sentencepiece;
@@ -23,6 +25,26 @@ fn crate_version() -> String {
     env!("CARGO_PKG_VERSION").to_string()
 }
 
+/// The pretokenizer scheme names `BPETokenizer.from_tiktoken` and
+/// `Tokenizer.load` accept — the single source of truth `Gigatoken::Tokenizer.load`
+/// names in its "no scheme for a .tiktoken path" error, so that list can
+/// never drift from the one `PretokenizerType::from_name` actually accepts.
+fn pretokenizer_names() -> Vec<&'static str> {
+    PretokenizerType::NAMES.to_vec()
+}
+
+/// The process-global encode-cache budget in bytes per worker, applied to
+/// tokenizers constructed afterward; `None` removes the bound. Mirrors
+/// pyo3's `set_max_cache_bytes`/`get_max_cache_bytes` (`src/bindings/cache.rs`
+/// in the core crate) over the same public core API.
+fn set_max_cache_bytes(max_bytes: Option<usize>) {
+    cache::set_max_cache_bytes(max_bytes);
+}
+
+fn get_max_cache_bytes() -> Option<usize> {
+    cache::get_max_cache_bytes()
+}
+
 /// Load a tokenizer from in-memory HuggingFace `tokenizer.json` contents.
 /// Returns a `SentencePieceTokenizer` when the model uses `byte_fallback`, a
 /// `BPETokenizer` otherwise — the same split as pyo3's `load_hf_json` and
@@ -43,6 +65,9 @@ fn init(ruby: &Ruby) -> Result<(), Error> {
     let native = gigatoken.define_module("Native")?;
     native.define_module_function("crate_version", function!(crate_version, 0))?;
     native.define_module_function("load_hf_json", function!(load_hf_json, 1))?;
+    native.define_module_function("pretokenizer_names", function!(pretokenizer_names, 0))?;
+    native.define_module_function("set_max_cache_bytes", function!(set_max_cache_bytes, 1))?;
+    native.define_module_function("get_max_cache_bytes", function!(get_max_cache_bytes, 0))?;
     sources::init(ruby, native)?;
     tokenizer::init(ruby, native)?;
     sentencepiece::init(ruby, native)?;

data/ext/gigatoken/src/sentencepiece.rs

@@ -42,10 +42,14 @@ pub struct SentencePieceTokenizer {
 }
 
 impl SentencePieceTokenizer {
+    /// Wrap a loaded model, applying the global cache budget to it and to
+    /// the single-document encode state — mirrors pyo3's `with_model`.
     pub(crate) fn from_tokenizer(tokenizer: SentencePieceBPE) -> Self {
+        let tokenizer = crate::cache::apply_max_cache_bytes_sp(tokenizer);
+        let state = EncodeState::with_budget(tokenizer.max_cache_bytes());
         Self {
             tokenizer: RefCell::new(tokenizer),
-            state: RefCell::new(EncodeState::new()),
+            state: RefCell::new(state),
         }
     }
 
@@ -188,6 +192,12 @@ impl SentencePieceTokenizer {
         }
         Ok(result)
     }
+
+    /// Cached unit entries on the single-document `encode` path's state
+    /// (batch encoders are per-call); see `BPETokenizer::cache_entries`.
+    fn cache_entries(&self) -> usize {
+        self.state.borrow().cache_size()
+    }
 }
 
 pub fn init(ruby: &Ruby, native: RModule) -> Result<(), Error> {
@@ -201,5 +211,6 @@ pub fn init(ruby: &Ruby, native: RModule) -> Result<(), Error> {
     class.define_method("vocab_size", method!(SentencePieceTokenizer::vocab_size, 0))?;
     class.define_method("vocab", method!(SentencePieceTokenizer::vocab, 0))?;
     class.define_method("merges", method!(SentencePieceTokenizer::merges, 0))?;
+    class.define_method("cache_entries", method!(SentencePieceTokenizer::cache_entries, 0))?;
     Ok(())
 }

data/ext/gigatoken/src/tokenizer.rs

@@ -9,6 +9,7 @@ use std::os::raw::c_long;
 
 use gigatoken_rs::load_tokenizer::hf::HfTokenizer;
 use gigatoken_rs::load_tokenizer::{hf, tiktoken};
+use gigatoken_rs::pretokenize::PretokenizerType;
 use gigatoken_rs::{
     GatherBuf, GatherOutcome, Tokenizer, WorkerPool, encode_docs_into, encode_docs_ragged,
     encode_files_docs, encode_files_docs_serial,
@@ -359,7 +360,7 @@ pub struct BPETokenizer {
 impl BPETokenizer {
     pub(crate) fn from_tokenizer(tokenizer: Tokenizer) -> Self {
         Self {
-            tokenizer: RefCell::new(tokenizer),
+            tokenizer: RefCell::new(crate::cache::apply_max_cache_bytes(tokenizer)),
             workers: WorkerPool::new(),
         }
     }
@@ -379,8 +380,28 @@ impl BPETokenizer {
         }
     }
 
-    fn from_tiktoken(ruby: &Ruby, path: String) -> Result<Self, Error> {
-        match tiktoken::load_tiktoken(&path) {
+    /// Load from a .tiktoken rank file with the named pretokenizer scheme
+    /// and a {content => id} mapping of special tokens. The file carries
+    /// neither, so both are the caller's to supply — see
+    /// `Gigatoken::Tokenizer.from_tiktoken`, which knows them for the
+    /// encodings OpenAI publishes.
+    fn from_tiktoken(
+        ruby: &Ruby,
+        path: String,
+        pretokenizer: String,
+        special_tokens: HashMap<String, u32>,
+    ) -> Result<Self, Error> {
+        let scheme = PretokenizerType::from_name(&pretokenizer).ok_or_else(|| {
+            raise(
+                ruby,
+                format!(
+                    "unknown pretokenizer scheme {pretokenizer:?}; expected one of {}",
+                    PretokenizerType::NAMES.join(", ")
+                ),
+            )
+        })?;
+        let special_tokens: Vec<(String, u32)> = special_tokens.into_iter().collect();
+        match tiktoken::load_tiktoken(&path, scheme, special_tokens) {
             Ok(tokenizer) => Ok(Self::from_tokenizer(tokenizer)),
             Err(e) => Err(raise(ruby, e.to_string())),
         }
@@ -552,12 +573,19 @@ impl BPETokenizer {
         }
         Ok(result)
     }
+
+    /// Cached pretoken entries on this tokenizer: grows as text is encoded,
+    /// drops back toward vocab-seed level when a budgeted cache wipes (see
+    /// `Gigatoken.max_cache_bytes`).
+    fn cache_entries(&self) -> usize {
+        self.tokenizer.borrow().cache_entries()
+    }
 }
 
 pub fn init(ruby: &Ruby, native: RModule) -> Result<(), Error> {
     let class: RClass = native.define_class("BPETokenizer", ruby.class_object())?;
     class.define_singleton_method("from_hf_json", function!(BPETokenizer::from_hf_json, 1))?;
-    class.define_singleton_method("from_tiktoken", function!(BPETokenizer::from_tiktoken, 1))?;
+    class.define_singleton_method("from_tiktoken", function!(BPETokenizer::from_tiktoken, 3))?;
     class.define_method("encode", method!(BPETokenizer::encode, 1))?;
     class.define_method("encode_batch", method!(BPETokenizer::encode_batch, 1))?;
     class.define_method("encode_batch_packed", method!(BPETokenizer::encode_batch_packed, 1))?;
@@ -567,5 +595,6 @@ pub fn init(ruby: &Ruby, native: RModule) -> Result<(), Error> {
     class.define_method("vocab_size", method!(BPETokenizer::vocab_size, 0))?;
     class.define_method("vocab", method!(BPETokenizer::vocab, 0))?;
     class.define_method("merges", method!(BPETokenizer::merges, 0))?;
+    class.define_method("cache_entries", method!(BPETokenizer::cache_entries, 0))?;
     Ok(())
 }

data/lib/gigatoken/cli/bench.rb

@@ -10,19 +10,20 @@ module Gigatoken
     class Bench < Dry::CLI::Command
       desc "Measure the time to encode FILES with TOKENIZER"
 
-      argument :tokenizer, required: true, desc: "tokenizer.json path or directory, HuggingFace repo id, or .tiktoken file"
+      argument :tokenizer, required: true, desc: "tokenizer.json path or directory, packaged encoding name (e.g. \"cl100k_base\"), HuggingFace repo id, or .tiktoken file (requires --pretokenizer)"
       argument :files, type: :array, required: true, desc: "UTF-8 text files to encode"
 
       option :doc_separator, desc: 'document separator to split the files on, e.g. "<|endoftext|>"; whole files are single documents otherwise'
       option :limit_bytes, default: "none", desc: "cap the bytes benchmarked, e.g. 100MB; 'none' for everything (parallel mode only — ignored with --no-parallel)"
       option :parallel, type: :boolean, default: true, desc: "encode on the worker pool instead of the fused serial core path"
       option :packed, type: :boolean, default: false, desc: "time the fused native file path with a packed IO::Buffer result instead of per-document Ruby arrays (ignores --limit-bytes)"
+      option :pretokenizer, desc: "pretokenizer scheme, required when TOKENIZER is a .tiktoken file (one of #{Native.pretokenizer_names.join(", ")}); ignored otherwise"
 
-      def call(tokenizer:, files:, doc_separator: nil, limit_bytes: "none", parallel: true, packed: false, **)
+      def call(tokenizer:, files:, doc_separator: nil, limit_bytes: "none", parallel: true, packed: false, pretokenizer: nil, **)
         limit = Support.parse_size(limit_bytes)
         out.puts "#{label("cpu")}: #{Support.cpu_info}"
 
-        gt_tokenizer = Support.load_tokenizer(tokenizer)
+        gt_tokenizer = Support.load_tokenizer(tokenizer, pretokenizer: pretokenizer)
 
         start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
         if packed

data/lib/gigatoken/cli/support.rb

@@ -14,10 +14,13 @@ module Gigatoken
       private_constant :SIZE_PATTERN
 
       class << self
-        # Load TOKENIZER: a tokenizer.json path/directory, a HuggingFace
-        # repo id, or a .tiktoken file — see Gigatoken::Tokenizer.load.
-        def load_tokenizer(spec)
-          Gigatoken::Tokenizer.load(spec)
+        # Load TOKENIZER: a tokenizer.json path/directory, a packaged
+        # tiktoken encoding name, a HuggingFace repo id, or a .tiktoken file
+        # — see Gigatoken::Tokenizer.load. `pretokenizer:` is forwarded
+        # as-is; it's required for a bare .tiktoken path (which carries no
+        # scheme of its own) and ignored for the other shapes.
+        def load_tokenizer(spec, pretokenizer: nil)
+          Gigatoken::Tokenizer.load(spec, pretokenizer: pretokenizer)
         end
 
         # Parse a decimal byte size like "100MB", "2.5GB", or "1000000";

data/lib/gigatoken/cli/validate.rb

@@ -13,13 +13,14 @@ module Gigatoken
     class Validate < Dry::CLI::Command
       desc "Check that encode_files agrees with a Ruby-side split plus encode_batch on FILES"
 
-      argument :tokenizer, required: true, desc: "tokenizer.json path or directory, HuggingFace repo id, or .tiktoken file"
+      argument :tokenizer, required: true, desc: "tokenizer.json path or directory, packaged encoding name (e.g. \"cl100k_base\"), HuggingFace repo id, or .tiktoken file (requires --pretokenizer)"
       argument :files, type: :array, required: true, desc: "UTF-8 text files to encode"
 
       option :doc_separator, desc: 'document separator to split the files on, e.g. "<|endoftext|>"; whole files are single documents otherwise'
+      option :pretokenizer, desc: "pretokenizer scheme, required when TOKENIZER is a .tiktoken file (one of #{Native.pretokenizer_names.join(", ")}); ignored otherwise"
 
-      def call(tokenizer:, files:, doc_separator: nil, **)
-        gt_tokenizer = Support.load_tokenizer(tokenizer)
+      def call(tokenizer:, files:, doc_separator: nil, pretokenizer: nil, **)
+        gt_tokenizer = Support.load_tokenizer(tokenizer, pretokenizer: pretokenizer)
 
         via_files = gt_tokenizer.encode_files(Support.text_file_source(files, doc_separator))
         via_batch = gt_tokenizer.encode_batch(Support.split_docs(files, doc_separator))

data/lib/gigatoken/encodings/PROVENANCE.md

@@ -0,0 +1,110 @@
+# Vendored tiktoken encodings — provenance
+
+The `.tiktoken` files in this directory are OpenAI's published BPE mergeable-rank
+tables, vendored verbatim so `gigatoken` can resolve these encodings by name with
+no network access and no writable cache directory.
+
+Each file is a plain text table: one `base64(token_bytes) rank` pair per line. It
+carries **mergeable ranks only** — the pretokenizer split regex and the special
+tokens belong to the encoding's *definition*, not to the file, and live in code
+(see the table below).
+
+They live under `lib/` rather than a top-level `data/` directory because this
+repo's `.gitignore` ignores `/data/` ("downloaded test data"); these are shipped
+gem payload, not test fixtures.
+
+## Files
+
+| File | Bytes | sha256 |
+|------|------:|--------|
+| `r50k_base.tiktoken`   |   835,554 | `306cd27f03c1a714eca7108e03d66b7dc042abe8c258b44c199a7ed9838dd930` |
+| `cl100k_base.tiktoken` | 1,681,126 | `223921b76ee99bde995b7ff738513eef100fb51d18c93597a113bcffe865b2a7` |
+| `o200k_base.tiktoken`  | 3,613,922 | `446a9538cb6c348e3516120d7c08b09f57c36495e2acfffe59a5bf8b0cfb1a2d` |
+
+## Source
+
+Retrieved **2026-08-10** over HTTPS from OpenAI's public encodings endpoint:
+
+```
+https://openaipublic.blob.core.windows.net/encodings/r50k_base.tiktoken
+https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken
+https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken
+```
+
+These are the same URLs `openai/tiktoken` itself fetches from, in
+[`tiktoken_ext/openai_public.py`](https://github.com/openai/tiktoken/blob/main/tiktoken_ext/openai_public.py).
+
+## Authenticity
+
+Verified three independent ways at retrieval time:
+
+1. **Transport** — HTTPS directly from `openaipublic.blob.core.windows.net`, the
+   origin OpenAI publishes and `tiktoken` itself downloads from.
+2. **Publisher checksum** — each measured sha256 above matches the `expected_hash`
+   OpenAI publishes for that file in `openai_public.py`, fetched separately from
+   `github.com/openai/tiktoken`. Two independent channels agree on the bytes.
+3. **Behavioral, against a third-party implementation** — loaded through
+   `gigatoken` with the pretokenizer and special tokens below, every token id
+   matched [`tiktoken_ruby`](https://github.com/IAPark/tiktoken_ruby) 0.0.17
+   (which embeds its own copy of these ranks) across a corpus covering ASCII,
+   CJK, ZWJ emoji sequences, combining accents, whitespace runs, source code,
+   and URLs. Resulting `vocab_size`: 50257 / 100277 / 200019.
+
+## Encoding definitions
+
+Transcribed from `openai_public.py` (each encoding's `pat_str` and
+`special_tokens`), cross-checked against upstream gigatoken's own port in
+`gigatoken/_load/tiktoken.py`. The scheme names are `PretokenizerType::NAMES`
+values (`src/pretokenize/options.rs`).
+
+| Encoding | Pretokenizer scheme | Special tokens |
+|---|---|---|
+| `r50k_base`   | `gpt2`  | `<\|endoftext\|>`=50256 |
+| `cl100k_base` | `gpt4`  | `<\|endoftext\|>`=100257, `<\|fim_prefix\|>`=100258, `<\|fim_middle\|>`=100259, `<\|fim_suffix\|>`=100260, `<\|endofprompt\|>`=100276 |
+| `o200k_base`  | `o200k` | `<\|endoftext\|>`=199999, `<\|endofprompt\|>`=200018 |
+
+`p50k_base` is deliberately absent: its ranks are not dense (50256 is left free
+for `<|endoftext|>`), and the rank loader rejects non-dense ranks with
+`"ranks must be dense"`. `p50k_edit` loads the same `p50k_base.tiktoken` ranks
+and is absent for the identical reason.
+
+## `o200k_harmony`
+
+Packaged with **no new vendored file**: it reuses `o200k_base.tiktoken`'s
+mergeable ranks and the `o200k` pretokenizer scheme verbatim. Confirmed
+against `openai/tiktoken` 0.9.0 that `o200k_harmony()` in `openai_public.py`
+calls `mergeable_ranks` with the same rank file and uses the same `pat_str` as
+`o200k_base()` — only the special-token table differs.
+
+That table (1091 entries: 10 named control tokens, 1081
+`<|reserved_N|>` slots) is transcribed, not derived, from `o200k_harmony()`:
+base specials `<|endoftext|>` 199999 and `<|endofprompt|>` 200018, then
+`<|startoftext|>` 199998, `<|endoftext|>` 199999, `<|reserved_200000|>`
+200000, `<|reserved_200001|>` 200001, `<|return|>` 200002, `<|constrain|>`
+200003, `<|reserved_200004|>` 200004, `<|channel|>` 200005, `<|start|>`
+200006, `<|end|>` 200007, `<|message|>` 200008, `<|reserved_200009|>` 200009,
+`<|reserved_200010|>` 200010, `<|reserved_200011|>` 200011, `<|call|>`
+200012, then `<|reserved_N|>` for `N` in `200013..201087`. The reserved range
+is **not** contiguous from 200000 — the named control tokens sit inside
+200000..200012, leaving reserved slots only at `{200000, 200001, 200004,
+200009, 200010, 200011} ∪ [200013, 201087]`. A table built as "200000..201087
+minus the named ids" invents `<|reserved_200002|>` and drops
+`<|reserved_200018|>`; the transcription above avoids both.
+
+**Not verified against `tiktoken_ruby`** (unlike the three encodings above):
+`tiktoken_ruby` 0.0.17's own `o200k_harmony` table drops `<|endofprompt|>` —
+it encodes the literal as six ordinary-text tokens rather than `[200018]` —
+while treating `<|reserved_200018|>` as the sole literal at that id.
+`openai/tiktoken` 0.9.0 keeps both `<|endofprompt|>` and
+`<|reserved_200018|>`, at the same id, as Python dict construction preserves
+both keys. `openai/tiktoken` is authoritative here; `tiktoken_ruby` is the
+outlier. See `spec/gigatoken/differential_spec.rb` for the measurement and
+the reduction-plus-pinning proof used in place of a `tiktoken_ruby`
+comparison.
+
+## Licence
+
+The `tiktoken` project and its published encoding files are MIT licensed,
+Copyright (c) 2022 OpenAI, Shantanu Jain. See
+<https://github.com/openai/tiktoken/blob/main/LICENSE>. The files are vendored
+here unmodified.