gigatoken 0.1.1-aarch64-linux → 0.2.0-aarch64-linux

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fec464409d36594817dc67fac65e7ccdc9baac8eceeb8d80eae62d7985fa58e1
-  data.tar.gz: 86de8bbbbd4dd0cc4eb2e2b3ce9d6dba44abc12deba382c22554d27e8893bacb
+  metadata.gz: dc1e4fea36f3bb957cff72a6187587e56aeabb4251c93dd554946e3a6423103c
+  data.tar.gz: 31fd83175bb68f431114913cbccf36b47b3b176396f2294befc9689e1f53a254
 SHA512:
-  metadata.gz: 2662b1945d1a79d1adac6c8be8bc772b358eff14df211014b2890e0f266054f916928c405bb3ddca12ba6834a3cb7bac404a504d72572dd0fae750b8b9c02b6b
-  data.tar.gz: f17356a59acb9c12aee785a4d77d8b465d615eab2dbdeb297d93e4b015d8203bccdf155d44136075776dc9df80d982cef90b79a3f183848f7ed7cefd22d0e551
+  metadata.gz: b5ecfa8b1794d409df4f6a23fc5207815e662e4af5f049412c3187f0a9b442c39de1ea7d91ce294b29799aea693d58abd8c81508bb37be3b50d53d2d9cc1a68f
+  data.tar.gz: 3ffbf1a20d577fd0bb7a4fba0397e356f4981486e3e1c0eb0ce11a4669f1ee69fbf853e9e844d6235391c114e8227d1360b684aaae8f3e1f6e1c057818b8fbef

data/README.md

@@ -48,17 +48,50 @@ tok.vocab_size                          # => 50257
 tok.special_tokens                      # => {"<|endoftext|>" => 50256}
 ```
 
-`load` takes a `tokenizer.json` path, a directory holding one, a HuggingFace Hub repo id, or a `.tiktoken` mergeable-ranks file, and dispatches on shape. Hub downloads run over socketry's `async-http` — no Python anywhere. Know what you have? Skip the dispatch:
+`load` takes a `tokenizer.json` path, a directory holding one, a packaged tiktoken encoding name (`r50k_base`, `cl100k_base`, `o200k_base`), a HuggingFace Hub repo id, or a `.tiktoken` mergeable-ranks file, and dispatches on shape. Hub downloads run over socketry's `async-http` — no Python anywhere. Know what you have? Skip the dispatch:
 
 ```ruby
 Gigatoken::Tokenizer.from_file("tokenizer.json")
 Gigatoken::Tokenizer.from_hub("openai-community/gpt2", revision: "main")
-Gigatoken::Tokenizer.from_tiktoken("vocab.tiktoken")
+Gigatoken::Tokenizer.from_tiktoken("cl100k_base.tiktoken", pretokenizer: "gpt4", special_tokens: {"<|endoftext|>" => 100257})
 Gigatoken::Tokenizer.from_json(File.binread("tokenizer.json"))
 ```
 
+A `.tiktoken` file holds mergeable ranks only — its pretokenization scheme and special tokens live in the code that defines the encoding, not the file — so `pretokenizer:` is a required keyword (one of `Gigatoken::Native.pretokenizer_names`: `gpt2`/`r50k`, `gpt4`/`cl100k`, `qwen2`, `qwen35`, `olmo3`, `deepseek_v3`, `o200k`, `nemotron`, `kimi`) and `special_tokens:` defaults to none. Nothing is guessed: an unknown scheme raises `Gigatoken::Error` naming the valid ones, and `Tokenizer.load` on a `.tiktoken` path with no `pretokenizer:` raises rather than silently picking one.
+
 SentencePiece-BPE models (Llama, Gemma, Mistral — any `tokenizer.json` with `byte_fallback: true`) load through the same entry points and pick the right backend automatically. One difference: the SentencePiece core decodes text, so it validates input and raises `Gigatoken::Error` on invalid UTF-8 instead of guessing.
 
+### Packaged tiktoken encodings
+
+`r50k_base`, `cl100k_base`, `o200k_base`, and `o200k_harmony` are vendored directly — mergeable ranks, pretokenizer scheme, and special-token table all shipped inside the gem (`lib/gigatoken/encodings/`; see `PROVENANCE.md` there for exact source URLs and hashes) — so all four resolve by name through both entry points entirely offline: no network access, no writable cache directory. `o200k_harmony` vendors no new file at all: it reuses `o200k_base.tiktoken`'s ranks and the `o200k` scheme verbatim, differing only in its special-token table (10 named control tokens — `<|start|>`, `<|message|>`, `<|end|>`, `<|return|>`, and so on — plus 1081 reserved slots; see `PROVENANCE.md` for the exact table). It's also the one packaged encoding not checked against `tiktoken_ruby`: that gem's 0.0.17 harmony table drops `<|endofprompt|>` where `openai/tiktoken` 0.9.0 keeps it at id 200018, so the oracle is the outlier here — `spec/gigatoken/differential_spec.rb` proves harmony instead by reduction to `o200k_base` plus a pinned special-token table.
+
+```ruby
+Gigatoken::Tokenizer.from_encoding("cl100k_base")
+Gigatoken::Tokenizer.load("cl100k_base")          # same result — packaged names are
+                                                  # checked before the Hub-repo-id shape
+```
+
+`p50k_base` and `p50k_edit` are deliberately not packaged: both load the same non-dense ranks (id 50256 is left free for `<|endoftext|>`), and the rank loader rejects non-dense ranks. Both entry points raise `Gigatoken::Error` explaining that, rather than `load` falling through to the Hub for a name that happens to look like a legacy repo id.
+
+`encode` on a packaged tokenizer honours its special-token table: text containing `<|endoftext|>` (or any other literal special-token string) is tokenized as that special token, not as ordinary text. That matches [`tiktoken`](https://github.com/openai/tiktoken)'s `encode_with_special_tokens`, not its plain `encode`, which treats the same literal as ordinary text — a difference worth knowing if you're tokenizing untrusted input. To get tiktoken's non-honouring default instead, build a tokenizer from the same rank file with an empty special-token table:
+
+```ruby
+entry = Gigatoken::Encodings["cl100k_base"]
+Gigatoken::Tokenizer.from_tiktoken(entry[:rank_file], pretokenizer: entry[:pretokenizer], special_tokens: {})
+```
+
+### Encode-cache budget
+
+Each tokenizer's pretoken cache is capped process-globally (512 MiB per worker by default) so long-lived processes — a Rails worker, say — don't grow it unbounded; a full cache wipes back toward its seed level and refills, which costs a bit of re-computation but never changes encode output. Tune it before building tokenizers you want the new budget to apply to:
+
+```ruby
+Gigatoken.max_cache_bytes            # => 536870912 (512 MiB)
+Gigatoken.max_cache_bytes = 64 << 20 # only tokenizers built after this see the new budget
+Gigatoken.max_cache_bytes = nil      # unbounded
+
+tok.cache_entries                    # => cached pretoken/unit count right now
+```
+
 ### Tokenize files without leaving Rust
 
 `encode_files` reads and tokenizes files entirely on the native side — document contents never materialize as Ruby objects. `.gz` and `.zst` decompress transparently.
@@ -97,6 +130,14 @@ gigatoken validate openai-community/gpt2 owt_train.txt --doc-separator "<|endoft
 
 `bench` reports MB/s and Mtok/s (`--packed` for the fused packed path, `--no-parallel` for the serial core). `validate` confirms native split-and-encode agrees with a Ruby-side split through `encode_batch`.
 
+TOKENIZER also takes a bare `.tiktoken` file, which is where `--pretokenizer` comes in: the file carries mergeable ranks only, so the split regex has to come from the caller, same as `from_tiktoken` above. `--pretokenizer` takes one of the scheme names listed above for `pretokenizer:`:
+
+```bash
+gigatoken bench lib/gigatoken/encodings/cl100k_base.tiktoken README.md --pretokenizer gpt4
+```
+
+Leave it off against a `.tiktoken` TOKENIZER and both commands raise `Gigatoken::Error` naming the valid schemes instead of crashing; for every other TOKENIZER shape (`tokenizer.json`, a packaged name, a Hub repo id) `--pretokenizer` is accepted but ignored.
+
 ## Development
 
 ```bash

data/lib/gigatoken/cli/bench.rb

@@ -10,19 +10,20 @@ module Gigatoken
     class Bench < Dry::CLI::Command
       desc "Measure the time to encode FILES with TOKENIZER"
 
-      argument :tokenizer, required: true, desc: "tokenizer.json path or directory, HuggingFace repo id, or .tiktoken file"
+      argument :tokenizer, required: true, desc: "tokenizer.json path or directory, packaged encoding name (e.g. \"cl100k_base\"), HuggingFace repo id, or .tiktoken file (requires --pretokenizer)"
       argument :files, type: :array, required: true, desc: "UTF-8 text files to encode"
 
       option :doc_separator, desc: 'document separator to split the files on, e.g. "<|endoftext|>"; whole files are single documents otherwise'
       option :limit_bytes, default: "none", desc: "cap the bytes benchmarked, e.g. 100MB; 'none' for everything (parallel mode only — ignored with --no-parallel)"
       option :parallel, type: :boolean, default: true, desc: "encode on the worker pool instead of the fused serial core path"
       option :packed, type: :boolean, default: false, desc: "time the fused native file path with a packed IO::Buffer result instead of per-document Ruby arrays (ignores --limit-bytes)"
+      option :pretokenizer, desc: "pretokenizer scheme, required when TOKENIZER is a .tiktoken file (one of #{Native.pretokenizer_names.join(", ")}); ignored otherwise"
 
-      def call(tokenizer:, files:, doc_separator: nil, limit_bytes: "none", parallel: true, packed: false, **)
+      def call(tokenizer:, files:, doc_separator: nil, limit_bytes: "none", parallel: true, packed: false, pretokenizer: nil, **)
         limit = Support.parse_size(limit_bytes)
         out.puts "#{label("cpu")}: #{Support.cpu_info}"
 
-        gt_tokenizer = Support.load_tokenizer(tokenizer)
+        gt_tokenizer = Support.load_tokenizer(tokenizer, pretokenizer: pretokenizer)
 
         start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
         if packed

data/lib/gigatoken/cli/support.rb

@@ -14,10 +14,13 @@ module Gigatoken
       private_constant :SIZE_PATTERN
 
       class << self
-        # Load TOKENIZER: a tokenizer.json path/directory, a HuggingFace
-        # repo id, or a .tiktoken file — see Gigatoken::Tokenizer.load.
-        def load_tokenizer(spec)
-          Gigatoken::Tokenizer.load(spec)
+        # Load TOKENIZER: a tokenizer.json path/directory, a packaged
+        # tiktoken encoding name, a HuggingFace repo id, or a .tiktoken file
+        # — see Gigatoken::Tokenizer.load. `pretokenizer:` is forwarded
+        # as-is; it's required for a bare .tiktoken path (which carries no
+        # scheme of its own) and ignored for the other shapes.
+        def load_tokenizer(spec, pretokenizer: nil)
+          Gigatoken::Tokenizer.load(spec, pretokenizer: pretokenizer)
         end
 
         # Parse a decimal byte size like "100MB", "2.5GB", or "1000000";

data/lib/gigatoken/cli/validate.rb

@@ -13,13 +13,14 @@ module Gigatoken
     class Validate < Dry::CLI::Command
       desc "Check that encode_files agrees with a Ruby-side split plus encode_batch on FILES"
 
-      argument :tokenizer, required: true, desc: "tokenizer.json path or directory, HuggingFace repo id, or .tiktoken file"
+      argument :tokenizer, required: true, desc: "tokenizer.json path or directory, packaged encoding name (e.g. \"cl100k_base\"), HuggingFace repo id, or .tiktoken file (requires --pretokenizer)"
       argument :files, type: :array, required: true, desc: "UTF-8 text files to encode"
 
       option :doc_separator, desc: 'document separator to split the files on, e.g. "<|endoftext|>"; whole files are single documents otherwise'
+      option :pretokenizer, desc: "pretokenizer scheme, required when TOKENIZER is a .tiktoken file (one of #{Native.pretokenizer_names.join(", ")}); ignored otherwise"
 
-      def call(tokenizer:, files:, doc_separator: nil, **)
-        gt_tokenizer = Support.load_tokenizer(tokenizer)
+      def call(tokenizer:, files:, doc_separator: nil, pretokenizer: nil, **)
+        gt_tokenizer = Support.load_tokenizer(tokenizer, pretokenizer: pretokenizer)
 
         via_files = gt_tokenizer.encode_files(Support.text_file_source(files, doc_separator))
         via_batch = gt_tokenizer.encode_batch(Support.split_docs(files, doc_separator))

data/lib/gigatoken/encodings/PROVENANCE.md

@@ -0,0 +1,110 @@
+# Vendored tiktoken encodings — provenance
+
+The `.tiktoken` files in this directory are OpenAI's published BPE mergeable-rank
+tables, vendored verbatim so `gigatoken` can resolve these encodings by name with
+no network access and no writable cache directory.
+
+Each file is a plain text table: one `base64(token_bytes) rank` pair per line. It
+carries **mergeable ranks only** — the pretokenizer split regex and the special
+tokens belong to the encoding's *definition*, not to the file, and live in code
+(see the table below).
+
+They live under `lib/` rather than a top-level `data/` directory because this
+repo's `.gitignore` ignores `/data/` ("downloaded test data"); these are shipped
+gem payload, not test fixtures.
+
+## Files
+
+| File | Bytes | sha256 |
+|------|------:|--------|
+| `r50k_base.tiktoken`   |   835,554 | `306cd27f03c1a714eca7108e03d66b7dc042abe8c258b44c199a7ed9838dd930` |
+| `cl100k_base.tiktoken` | 1,681,126 | `223921b76ee99bde995b7ff738513eef100fb51d18c93597a113bcffe865b2a7` |
+| `o200k_base.tiktoken`  | 3,613,922 | `446a9538cb6c348e3516120d7c08b09f57c36495e2acfffe59a5bf8b0cfb1a2d` |
+
+## Source
+
+Retrieved **2026-08-10** over HTTPS from OpenAI's public encodings endpoint:
+
+```
+https://openaipublic.blob.core.windows.net/encodings/r50k_base.tiktoken
+https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken
+https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken
+```
+
+These are the same URLs `openai/tiktoken` itself fetches from, in
+[`tiktoken_ext/openai_public.py`](https://github.com/openai/tiktoken/blob/main/tiktoken_ext/openai_public.py).
+
+## Authenticity
+
+Verified three independent ways at retrieval time:
+
+1. **Transport** — HTTPS directly from `openaipublic.blob.core.windows.net`, the
+   origin OpenAI publishes and `tiktoken` itself downloads from.
+2. **Publisher checksum** — each measured sha256 above matches the `expected_hash`
+   OpenAI publishes for that file in `openai_public.py`, fetched separately from
+   `github.com/openai/tiktoken`. Two independent channels agree on the bytes.
+3. **Behavioral, against a third-party implementation** — loaded through
+   `gigatoken` with the pretokenizer and special tokens below, every token id
+   matched [`tiktoken_ruby`](https://github.com/IAPark/tiktoken_ruby) 0.0.17
+   (which embeds its own copy of these ranks) across a corpus covering ASCII,
+   CJK, ZWJ emoji sequences, combining accents, whitespace runs, source code,
+   and URLs. Resulting `vocab_size`: 50257 / 100277 / 200019.
+
+## Encoding definitions
+
+Transcribed from `openai_public.py` (each encoding's `pat_str` and
+`special_tokens`), cross-checked against upstream gigatoken's own port in
+`gigatoken/_load/tiktoken.py`. The scheme names are `PretokenizerType::NAMES`
+values (`src/pretokenize/options.rs`).
+
+| Encoding | Pretokenizer scheme | Special tokens |
+|---|---|---|
+| `r50k_base`   | `gpt2`  | `<\|endoftext\|>`=50256 |
+| `cl100k_base` | `gpt4`  | `<\|endoftext\|>`=100257, `<\|fim_prefix\|>`=100258, `<\|fim_middle\|>`=100259, `<\|fim_suffix\|>`=100260, `<\|endofprompt\|>`=100276 |
+| `o200k_base`  | `o200k` | `<\|endoftext\|>`=199999, `<\|endofprompt\|>`=200018 |
+
+`p50k_base` is deliberately absent: its ranks are not dense (50256 is left free
+for `<|endoftext|>`), and the rank loader rejects non-dense ranks with
+`"ranks must be dense"`. `p50k_edit` loads the same `p50k_base.tiktoken` ranks
+and is absent for the identical reason.
+
+## `o200k_harmony`
+
+Packaged with **no new vendored file**: it reuses `o200k_base.tiktoken`'s
+mergeable ranks and the `o200k` pretokenizer scheme verbatim. Confirmed
+against `openai/tiktoken` 0.9.0 that `o200k_harmony()` in `openai_public.py`
+calls `mergeable_ranks` with the same rank file and uses the same `pat_str` as
+`o200k_base()` — only the special-token table differs.
+
+That table (1091 entries: 10 named control tokens, 1081
+`<|reserved_N|>` slots) is transcribed, not derived, from `o200k_harmony()`:
+base specials `<|endoftext|>` 199999 and `<|endofprompt|>` 200018, then
+`<|startoftext|>` 199998, `<|endoftext|>` 199999, `<|reserved_200000|>`
+200000, `<|reserved_200001|>` 200001, `<|return|>` 200002, `<|constrain|>`
+200003, `<|reserved_200004|>` 200004, `<|channel|>` 200005, `<|start|>`
+200006, `<|end|>` 200007, `<|message|>` 200008, `<|reserved_200009|>` 200009,
+`<|reserved_200010|>` 200010, `<|reserved_200011|>` 200011, `<|call|>`
+200012, then `<|reserved_N|>` for `N` in `200013..201087`. The reserved range
+is **not** contiguous from 200000 — the named control tokens sit inside
+200000..200012, leaving reserved slots only at `{200000, 200001, 200004,
+200009, 200010, 200011} ∪ [200013, 201087]`. A table built as "200000..201087
+minus the named ids" invents `<|reserved_200002|>` and drops
+`<|reserved_200018|>`; the transcription above avoids both.
+
+**Not verified against `tiktoken_ruby`** (unlike the three encodings above):
+`tiktoken_ruby` 0.0.17's own `o200k_harmony` table drops `<|endofprompt|>` —
+it encodes the literal as six ordinary-text tokens rather than `[200018]` —
+while treating `<|reserved_200018|>` as the sole literal at that id.
+`openai/tiktoken` 0.9.0 keeps both `<|endofprompt|>` and
+`<|reserved_200018|>`, at the same id, as Python dict construction preserves
+both keys. `openai/tiktoken` is authoritative here; `tiktoken_ruby` is the
+outlier. See `spec/gigatoken/differential_spec.rb` for the measurement and
+the reduction-plus-pinning proof used in place of a `tiktoken_ruby`
+comparison.
+
+## Licence
+
+The `tiktoken` project and its published encoding files are MIT licensed,
+Copyright (c) 2022 OpenAI, Shantanu Jain. See
+<https://github.com/openai/tiktoken/blob/main/LICENSE>. The files are vendored
+here unmodified.