gigatoken 0.3.0-aarch64-linux → 0.5.0-aarch64-linux

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 281a4ef78df3d94470e35cb463085c9da8e9bb3549c552bf6ab66f3bb4df47b1
-  data.tar.gz: f3d1531a68d00ab75abf45121b6a2df3ea113a7da39946b4e480ad7e4adea338
+  metadata.gz: 7a3b5aeb985dcdb597d00404f93cfab092b74261cfbde1f63aa3dc436d3c3465
+  data.tar.gz: a25819637d27224914a3a93bbf031c587427c23abad2614b5262f439e807791b
 SHA512:
-  metadata.gz: fb9419f43f11adb7f0e181e5606b008f911439f486b332acc2ddbfd06e5e798721184ec6b37122e6dadc732b3543dc5666e3c53d38ad62a234948ba0c9abdc8e
-  data.tar.gz: f867d2440b5fda9a43fb67f9bd7b8159b933ee073eac1042e8176ccbbeef1ff69eefde75b6215dd8db1c909ece1dd7b6893a0d68b9c1217f2de459325024e180
+  metadata.gz: 907435605fdd23721af74c44d1ba47507cef21b6ae661e6784977e10cdfc0bd8c9a9b65b2e338655c93bc88af43ba9fa0ee8a2521969fdabe03524ccdb62952a
+  data.tar.gz: 2b38c06006819f1aa313cb1bf2bfdb5f7187ff6b959323b0f7a3d5bbcf0215e61bc496fc098afe9d0d6508e90342298306db63c18eaccd1ecaf3c7aae61d3375

data/README.md

@@ -57,13 +57,13 @@ Gigatoken::Tokenizer.from_tiktoken("cl100k_base.tiktoken", pretokenizer: "gpt4",
 Gigatoken::Tokenizer.from_json(File.binread("tokenizer.json"))
 ```
 
-A `.tiktoken` file holds mergeable ranks only — its pretokenization scheme and special tokens live in the code that defines the encoding, not the file — so `pretokenizer:` is a required keyword (one of `Gigatoken::Native.pretokenizer_names`: `gpt2`/`r50k`, `gpt4`/`cl100k`, `qwen2`, `qwen35`, `olmo3`, `deepseek_v3`, `o200k`, `nemotron`, `kimi`) and `special_tokens:` defaults to none. Nothing is guessed: an unknown scheme raises `Gigatoken::Error` naming the valid ones, and `Tokenizer.load` on a `.tiktoken` path with no `pretokenizer:` raises rather than silently picking one.
+A `.tiktoken` file holds mergeable ranks only — its pretokenization scheme and special tokens live in the code that defines the encoding, not the file — so `pretokenizer:` is a required keyword (one of `Gigatoken::Native.pretokenizer_names`: `gpt2`/`r50k`, `gpt4`/`cl100k`, `qwen2`, `qwen35`, `olmo3`, `deepseek_v3`, `o200k`, `nemotron`, `kimi`) and `special_tokens:` defaults to none. Nothing is guessed: an unknown scheme raises `Gigatoken::ModelError` naming the valid ones, and `Tokenizer.load` on a `.tiktoken` path with no `pretokenizer:` raises rather than silently picking one.
 
-SentencePiece-BPE models (Llama, Gemma, Mistral — any `tokenizer.json` with `byte_fallback: true`) load through the same entry points and pick the right backend automatically. One difference: the SentencePiece core decodes text, so it validates input and raises `Gigatoken::Error` on invalid UTF-8 instead of guessing.
+SentencePiece-BPE models (Llama, Gemma, Mistral — any `tokenizer.json` with `byte_fallback: true`) load through the same entry points and pick the right backend automatically. One difference: the SentencePiece core decodes text, so it validates input and raises `Gigatoken::InputError` on invalid UTF-8 instead of guessing.
 
-### Packaged tiktoken encodings
+### Packaged encodings
 
-`r50k_base`, `cl100k_base`, `o200k_base`, and `o200k_harmony` are vendored directly — mergeable ranks, pretokenizer scheme, and special-token table all shipped inside the gem (`lib/gigatoken/encodings/`; see `PROVENANCE.md` there for exact source URLs and hashes) — so all four resolve by name through both entry points entirely offline: no network access, no writable cache directory. `o200k_harmony` vendors no new file at all: it reuses `o200k_base.tiktoken`'s ranks and the `o200k` scheme verbatim, differing only in its special-token table (10 named control tokens — `<|start|>`, `<|message|>`, `<|end|>`, `<|return|>`, and so on — plus 1081 reserved slots; see `PROVENANCE.md` for the exact table). It's also the one packaged encoding not checked against `tiktoken_ruby`: that gem's 0.0.17 harmony table drops `<|endofprompt|>` where `openai/tiktoken` 0.9.0 keeps it at id 200018, so the oracle is the outlier here — `spec/gigatoken/differential_spec.rb` proves harmony instead by reduction to `o200k_base` plus a pinned special-token table.
+`r50k_base`, `cl100k_base`, `o200k_base`, and `o200k_harmony` are vendored directly — mergeable ranks, pretokenizer scheme, and special-token table all shipped inside the gem (`lib/gigatoken/encodings/`; see `PROVENANCE.md` there for exact source URLs and hashes) — as are `qwen35`, `qwen38` and `muse_spark`, below, so all seven resolve by name through both entry points entirely offline: no network access, no writable cache directory. `o200k_harmony` vendors no new file at all: it reuses `o200k_base.tiktoken`'s ranks and the `o200k` scheme verbatim, differing only in its special-token table (10 named control tokens — `<|start|>`, `<|message|>`, `<|end|>`, `<|return|>`, and so on — plus 1081 reserved slots; see `PROVENANCE.md` for the exact table). It's also the one packaged encoding not checked against `tiktoken_ruby`: that gem's 0.0.17 harmony table drops `<|endofprompt|>` where `openai/tiktoken` 0.9.0 keeps it at id 200018, so the oracle is the outlier here — `spec/gigatoken/differential_spec.rb` proves harmony instead by reduction to `o200k_base` plus a pinned special-token table.
 
 ```ruby
 Gigatoken::Tokenizer.from_encoding("cl100k_base")
@@ -71,7 +71,9 @@ Gigatoken::Tokenizer.load("cl100k_base")          # same result — packaged nam
                                                   # checked before the Hub-repo-id shape
 ```
 
-`p50k_base` and `p50k_edit` are deliberately not packaged: both load the same non-dense ranks (id 50256 is left free for `<|endoftext|>`), and the rank loader rejects non-dense ranks. Both entry points raise `Gigatoken::Error` explaining that, rather than `load` falling through to the Hub for a name that happens to look like a legacy repo id.
+`qwen35`, `qwen38` and `muse_spark` are whole HuggingFace `tokenizer.json` files, gzipped (`*.json.gz`, pinned to a repo revision in `PROVENANCE.md`) and loaded through `from_json`, so their special tokens come from the file. `qwen35` is the Qwen 3.5 and 3.6 tokenizer; `qwen38` is that file plus seven audio/TTS special tokens (`<|audio_start|>`, `<|audio_end|>`, `<tts_pad>`, `<tts_text_bos>`, `<tts_text_eod>`, `<tts_text_bos_single>`, `<|audio_pad|>`), so ordinary text encodes the same under both; `muse_spark` is `meta-models/Muse-Glimmer-30B`'s tokenizer, because Meta publishes no standalone Muse Spark one. Their output matches HuggingFace `tokenizers` with `add_special_tokens: false`: gigatoken applies no post-processor, so `encode` never prepends `<|begin_of_text|>` for `muse_spark`. Unlike the tiktoken entries, a registry entry for one of these is just `{json_file:}`, and each load reads, gunzips and parses its file (about 0.4 s).
+
+`p50k_base` and `p50k_edit` are deliberately not packaged: both load the same non-dense ranks (id 50256 is left free for `<|endoftext|>`), and the rank loader rejects non-dense ranks. Both entry points raise `Gigatoken::ModelError` explaining that, rather than `load` falling through to the Hub for a name that happens to look like a legacy repo id.
 
 `encode` on a packaged tokenizer honours its special-token table: text containing `<|endoftext|>` (or any other literal special-token string) is tokenized as that special token, not as ordinary text. That matches [`tiktoken`](https://github.com/openai/tiktoken)'s `encode_with_special_tokens`, not its plain `encode`, which treats the same literal as ordinary text — a difference worth knowing if you're tokenizing untrusted input. To get tiktoken's non-honouring default instead, build a tokenizer from the same rank file with an empty special-token table:
 
@@ -117,6 +119,12 @@ packed.token_count  # => total tokens
 packed[3]           # => document 3's ids as an Array, on demand
 ```
 
+### Errors
+
+Everything the library raises is a `Gigatoken::Error`, never a raw Rust panic — and, under it, one of three: `Gigatoken::ModelError` when a tokenizer can't be loaded (bad or hostile JSON, a missing file or directory, an unknown or unpackable encoding name, a malformed `.tiktoken`), `Gigatoken::InputError` when a document or an id can't be taken (a String that won't transcode, invalid UTF-8 on the SentencePiece path, an id outside the vocabulary in `decode`), and `Gigatoken::HubError` for everything `Gigatoken::Hub` raises (HTTP status, transport, timeout, the repo-id and header checks). `rescue Gigatoken::Error` catches all three.
+
+`encode` and `encode_batch` honour the String's encoding tag: UTF-8, US-ASCII and binary go through byte-wise, and a real non-UTF-8 encoding (ISO-8859-1, UTF-16LE, …) is transcoded first, so it gives the same ids as the same text read as UTF-8. See [the reference](docs/reference/tokenizer.md#input-encodings).
+
 ### Async
 
 `encode_batch` and `encode_files` release the GVL for the whole encode; the parallelism runs on the engine's rayon pool, not Ruby threads. Under `Async`, give the fiber scheduler a worker pool (`ASYNC_SCHEDULER_WORKER_POOL=true`) and the calling fiber yields to the reactor too. Design notes: [docs/how-to/run-under-async.md](docs/how-to/run-under-async.md).

data/lib/gigatoken/cli/bench.rb

@@ -20,30 +20,36 @@ module Gigatoken
       option :pretokenizer, desc: "pretokenizer scheme, required when TOKENIZER is a .tiktoken file (one of #{Native.pretokenizer_names.join(", ")}); ignored otherwise"
 
       def call(tokenizer:, files:, doc_separator: nil, limit_bytes: "none", parallel: true, packed: false, pretokenizer: nil, **)
+        Support.check_usage!(files, doc_separator)
         limit = Support.parse_size(limit_bytes)
         out.puts "#{label("cpu")}: #{Support.cpu_info}"
 
         gt_tokenizer = Support.load_tokenizer(tokenizer, pretokenizer: pretokenizer)
 
+        # Only the batch path materializes the documents; the native paths
+        # leave this nil and count their bytes off the clock below.
+        docs = nil
         start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
         if packed
           encoded = gt_tokenizer.encode_files(Support.text_file_source(files, doc_separator), parallel: parallel, packed: true)
-          n_bytes = files.sum { |file| File.size(file) }
           n_tokens = encoded.token_count
         elsif parallel
           docs = Support.subset_docs(Support.split_docs(files, doc_separator), limit)
           encoded = gt_tokenizer.encode_batch(docs)
-          n_bytes = docs.sum(&:bytesize)
           n_tokens = encoded.sum(&:length)
         else
           encoded = gt_tokenizer.encode_files(Support.text_file_source(files, doc_separator), parallel: false)
-          n_bytes = files.sum { |file| File.size(file) }
           n_tokens = encoded.sum(&:length)
         end
         seconds = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start
 
+        # The throughput is over the bytes encoded, which for a compressed
+        # file are its decompressed bytes. Counting those means reading the
+        # file, so the native paths count off the clock; the batch path
+        # already holds the documents it encoded.
+        n_bytes = docs ? docs.sum(&:bytesize) : Support.input_bytesize(files)
         out.puts report("gigatoken", seconds, n_bytes, n_tokens)
-      rescue Gigatoken::Error => e
+      rescue Gigatoken::Error, SystemCallError => e
         err.puts "error: #{e.message}"
         exit(1)
       end

data/lib/gigatoken/cli/support.rb

@@ -4,16 +4,29 @@ require "etc"
 
 module Gigatoken
   module CLI
-    # Helpers shared by the bench and validate commands: tokenizer loading,
-    # byte-size parsing, document splitting, and CPU identification.
+    # Helpers shared by the bench and validate commands: argument checking,
+    # tokenizer loading, byte-size parsing, document splitting, and CPU
+    # identification.
     module Support
       SIZE_UNITS = {"" => 1, "k" => 10**3, "m" => 10**6, "g" => 10**9, "t" => 10**12}.freeze
       private_constant :SIZE_UNITS
 
-      SIZE_PATTERN = /\A\s*(\d+(?:\.\d+)?)\s*([kmgt]?)i?b?\s*\z/i
+      # "KiB" and friends are binary, as everywhere else that spells the i.
+      BINARY_SIZE_UNITS = {"" => 1, "k" => 2**10, "m" => 2**20, "g" => 2**30, "t" => 2**40}.freeze
+      private_constant :BINARY_SIZE_UNITS
+
+      SIZE_PATTERN = /\A\s*(\d+(?:\.\d+)?)\s*([kmgt]?)(i?)b?\s*\z/i
       private_constant :SIZE_PATTERN
 
       class << self
+        # The argument shapes dry-cli itself lets through: FILES declared
+        # `required: true` still arrives empty, and an empty separator would
+        # split every byte into its own document.
+        def check_usage!(files, separator)
+          raise Gigatoken::Error, "FILES is required: name at least one file to encode" if files.empty?
+          raise Gigatoken::Error, "--doc-separator cannot be empty" if separator == ""
+        end
+
         # Load TOKENIZER: a tokenizer.json path/directory, a packaged
         # tiktoken encoding name, a HuggingFace repo id, or a .tiktoken file
         # — see Gigatoken::Tokenizer.load. `pretokenizer:` is forwarded
@@ -23,15 +36,17 @@ module Gigatoken
           Gigatoken::Tokenizer.load(spec, pretokenizer: pretokenizer)
         end
 
-        # Parse a decimal byte size like "100MB", "2.5GB", or "1000000";
-        # "none"/"unlimited" means no limit.
+        # Parse a byte size like "100MB", "2.5GB", "64KiB" or "1000000" —
+        # decimal units unless the prefix spells the i, which makes it
+        # binary; "none"/"unlimited" means no limit.
         def parse_size(text)
           return nil if ["none", "unlimited"].include?(text.strip.downcase)
 
           match = SIZE_PATTERN.match(text)
           raise Gigatoken::Error, "cannot parse size #{text.inspect}; expected something like 100MB (or 'none')" unless match
 
-          (match[1].to_f * SIZE_UNITS.fetch(match[2].downcase)).to_i
+          units = match[3].empty? ? SIZE_UNITS : BINARY_SIZE_UNITS
+          (match[1].to_f * units.fetch(match[2].downcase)).to_i
         end
 
         # A Native::TextFileSource for FILES, split on `separator` when
@@ -42,15 +57,23 @@ module Gigatoken
 
         # Whole files as raw bytes, one document per file, or (with a
         # separator) the separator-split pieces of each file in order, empty
-        # documents skipped.
+        # documents skipped. Compressed files are decompressed first, as the
+        # native file sources do, so both paths split the same bytes.
         def split_docs(files, separator)
-          raws = files.map { |file| File.binread(file.to_s) }
+          raws = files.map { |file| read_decompressed(file.to_s) }
           return raws if separator.nil?
 
           sep = separator.b
           raws.flat_map { |raw| raw.split(sep).reject(&:empty?) }
         end
 
+        # The bytes the tokenizer actually sees for FILES — a compressed
+        # file's decompressed size, not its size on disk — so throughput is
+        # reported over the input that was tokenized.
+        def input_bytesize(files)
+          files.sum { |file| read_decompressed(file.to_s).bytesize }
+        end
+
         # The prefix of `docs` totalling at most `limit_bytes`, byte-
         # truncating the final document to fill the budget. Unlike a
         # text-comparison tool, gigatoken encodes raw bytes and does not
@@ -90,6 +113,15 @@ module Gigatoken
 
         private
 
+        # One file's bytes, decompressed. `Native.read_input` is the core's
+        # own decoder — the one the native file sources load through — so
+        # `.gz`, `.zst`/`.zstd` and plain files are detected and read here
+        # exactly as they are on the other side of `validate`, and an
+        # unreadable file raises Gigatoken::Error naming it.
+        def read_decompressed(path)
+          Gigatoken::Native.read_input(path)
+        end
+
         def darwin_cpu_info
           name = sysctl("machdep.cpu.brand_string")
           [name, sysctl_int("hw.physicalcpu"), sysctl_int("hw.packages")]

data/lib/gigatoken/cli/validate.rb

@@ -20,6 +20,7 @@ module Gigatoken
       option :pretokenizer, desc: "pretokenizer scheme, required when TOKENIZER is a .tiktoken file (one of #{Native.pretokenizer_names.join(", ")}); ignored otherwise"
 
       def call(tokenizer:, files:, doc_separator: nil, pretokenizer: nil, **)
+        Support.check_usage!(files, doc_separator)
         gt_tokenizer = Support.load_tokenizer(tokenizer, pretokenizer: pretokenizer)
 
         via_files = gt_tokenizer.encode_files(Support.text_file_source(files, doc_separator))
@@ -38,7 +39,7 @@ module Gigatoken
         end
 
         out.puts "validation OK: #{via_files.length} documents match"
-      rescue Gigatoken::Error => e
+      rescue Gigatoken::Error, SystemCallError => e
         err.puts "error: #{e.message}"
         exit(1)
       end

data/lib/gigatoken/encodings/PROVENANCE.md

@@ -1,4 +1,4 @@
-# Vendored tiktoken encodings — provenance
+# Vendored encodings — provenance
 
 The `.tiktoken` files in this directory are OpenAI's published BPE mergeable-rank
 tables, vendored verbatim so `gigatoken` can resolve these encodings by name with
@@ -9,6 +9,10 @@ carries **mergeable ranks only** — the pretokenizer split regex and the specia
 tokens belong to the encoding's *definition*, not to the file, and live in code
 (see the table below).
 
+The three `.json.gz` files are HuggingFace `tokenizer.json` files, not tiktoken
+tables; they are covered in [their own section](#huggingface-tokenizerjson-files)
+at the end.
+
 They live under `lib/` rather than a top-level `data/` directory because this
 repo's `.gitignore` ignores `/data/` ("downloaded test data"); these are shipped
 gem payload, not test fixtures.
@@ -108,3 +112,111 @@ The `tiktoken` project and its published encoding files are MIT licensed,
 Copyright (c) 2022 OpenAI, Shantanu Jain. See
 <https://github.com/openai/tiktoken/blob/main/LICENSE>. The files are vendored
 here unmodified.
+
+## HuggingFace `tokenizer.json` files
+
+`qwen35.json.gz`, `qwen38.json.gz` and `muse_spark.json.gz` are HuggingFace
+`tokenizer.json` files, vendored byte-for-byte and gzipped. Unlike a
+`.tiktoken` table, a `tokenizer.json` carries the encoding's whole definition
+— vocabulary, merges, normalizer, pretokenizer, added tokens — so the registry
+records only the file, and `Tokenizer.from_encoding` loads it with
+`Tokenizer.from_json`, reading the special tokens from the file's
+`added_tokens` (those marked `"special": true`) exactly as `from_json` does.
+
+### Files
+
+| File | Decompressed bytes | sha256 (decompressed) |
+|------|-------------------:|-----------------------|
+| `qwen35.json.gz`     | 12,807,982 | `5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42` |
+| `qwen38.json.gz`     | 12,809,320 | `0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3` |
+| `muse_spark.json.gz` | 28,129,897 | `c9dbee66967b58f31a7c27f723c3760da3526ccd0427578e8905b0abb0031c4d` |
+
+### Source
+
+Retrieved **2026-10-07** over HTTPS from the Hub's `resolve` endpoint, each at
+a pinned commit:
+
+| Encoding | Repo | Revision | URL |
+|---|---|---|---|
+| `qwen35`     | `Qwen/Qwen3.5-27B`             | `fc05daec18b0a78c049392ed2e771dde82bdf654` | <https://huggingface.co/Qwen/Qwen3.5-27B/resolve/fc05daec18b0a78c049392ed2e771dde82bdf654/tokenizer.json> |
+| `qwen38`     | `Qwen/Qwen3.8-27B`             | `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0` | <https://huggingface.co/Qwen/Qwen3.8-27B/resolve/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/tokenizer.json> |
+| `muse_spark` | `meta-models/Muse-Glimmer-30B` | `a4e59da52a7bc87ae7251dd5545c0dd437c44b68` | <https://huggingface.co/meta-models/Muse-Glimmer-30B/resolve/a4e59da52a7bc87ae7251dd5545c0dd437c44b68/tokenizer.json> |
+
+**`qwen35` is shared across releases.** The Hub's tree API
+(`https://huggingface.co/api/models/<repo>?blobs=true`) reports the identical
+LFS sha256 `5f9e4d49…` for `tokenizer.json` in `Qwen/Qwen3.5-{0.8B, 2B, 4B, 9B,
+27B, 35B-A3B, 122B-A10B, 397B-A17B}` and `Qwen/Qwen3.6-{27B, 35B-A3B}`
+(queried 2026-10-07), so one file serves Qwen 3.5 and 3.6.
+`Qwen/Qwen3.6-Plus` is gated and was not checked.
+
+**`qwen38` ships its own file**, identical (`0997f410…`) across
+`Qwen/Qwen3.8-{27B, Flash-Next, 2.4T-A95B}` and their `-FP8` twins. It differs
+from `qwen35` only in seven more `"special": true` added tokens, ids
+248070–248076: `<|audio_start|>`, `<|audio_end|>`, `<tts_pad>`,
+`<tts_text_bos>`, `<tts_text_eod>`, `<tts_text_bos_single>`, `<|audio_pad|>`.
+The vocabulary, merges, normalizer, pretokenizer, decoder and post-processor
+are identical, so ordinary text encodes the same under both; only those seven
+literals, `vocab_size` and `special_tokens` differ. It is vendored as a second
+file rather than derived from the first because the engine reads added tokens
+from the JSON, and a JSON-backed tokenizer has no way to add specials after
+the fact. `Qwen3.8-27B` is the repo cited; `Flash-Next` and `2.4T-A95B` carry an
+"other" licence on the *model*, but their tokenizer file is byte-identical to
+the Apache-2.0 repo's.
+
+**`muse_spark` is the Muse-Glimmer-30B file.** Meta publishes no standalone
+Muse Spark tokenizer (a Hub search over `meta-models` finds none, and Meta's
+model-API docs at <https://dev.meta.ai/docs/muse-glimmer> point at none). The
+Muse-Glimmer-30B model card states "Tokenizer: 200,000 BPE tokens + 2,048
+special tokens" (vocabulary 202,048), trained from Muse Spark's outputs, and
+this exact repo and revision is the one the reference application pins. As of
+2026-10-07 that revision is also `main`.
+
+### Authenticity
+
+Verified three independent ways at retrieval time:
+
+1. **Transport** — HTTPS directly from `huggingface.co`, at the pinned
+   commit above.
+2. **Publisher checksum** — each file's sha256 above matches the LFS sha256 the
+   Hub's tree API reports for `tokenizer.json` at that revision, and so does
+   its size. Two independent channels agree on the bytes.
+3. **Behavioral, against the reference implementation** — loaded through
+   `Tokenizer.from_encoding`, every token id matched HuggingFace
+   [`tokenizers`](https://rubygems.org/gems/tokenizers) 0.7.0 loading the same
+   decompressed file with `add_special_tokens: false`, across this repo's own
+   source and docs (`spec/gigatoken/differential_spec.rb`). Resulting
+   `vocab_size`: 248070 / 248077 / 202048; `special_tokens.size`: 14 / 21 / 2048.
+
+### Compression
+
+Each file was compressed with `gzip -9 -n` (no name or timestamp in the gzip
+header, so the bytes are reproducible), and `gunzip -c` of it reproduces the
+published file byte-for-byte — the sha256s above are of the decompressed
+bytes. Compressed sizes: 3,514,616 (`qwen35`), 3,514,699 (`qwen38`), 4,475,666
+(`muse_spark`).
+
+### What gigatoken does not take from the file
+
+- **The post-processor.** gigatoken applies none. `muse_spark`'s is a
+  `TemplateProcessing` that prepends `<|begin_of_text|>`; `Tokenizer#encode`
+  does not, matching `tokenizers`' `add_special_tokens: false`. The Qwen files'
+  is a `ByteLevel` post-processor, a no-op on ids.
+- **Non-special added tokens in `special_tokens`.** Qwen's 12 non-special
+  added tokens (`<tool_call>`, `<think>`, the `<|fim_*|>` tokens,
+  `<|repo_name|>`, `<|file_sep|>`, `<tool_response>` and so on) are matched
+  atomically by the engine, like the special ones, but only `"special": true`
+  tokens appear in `Tokenizer#special_tokens` — 14 for `qwen35`, 21 for
+  `qwen38`.
+- **`tokenizer_config.json`.** Not vendored; gigatoken reads only
+  `tokenizer.json`.
+
+### Licence
+
+All three repos are Apache-2.0, each with its own `LICENSE` at the pinned
+revision:
+
+- <https://huggingface.co/Qwen/Qwen3.5-27B/blob/fc05daec18b0a78c049392ed2e771dde82bdf654/LICENSE>
+- <https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/LICENSE>
+- <https://huggingface.co/meta-models/Muse-Glimmer-30B/blob/a4e59da52a7bc87ae7251dd5545c0dd437c44b68/LICENSE>
+
+The files are vendored here unmodified (gzipped).

data/lib/gigatoken/encodings.rb

@@ -1,10 +1,11 @@
 # frozen_string_literal: true
 
 module Gigatoken
-  # The tiktoken encodings gigatoken vendors ranks for, and the pieces a
-  # .tiktoken file doesn't carry: its pretokenizer scheme and special-token
-  # table (see Tokenizer.from_tiktoken and lib/gigatoken/encodings/
-  # PROVENANCE.md, the source of truth this is transcribed from).
+  # The encodings gigatoken vendors a file for. For the tiktoken ones, the
+  # pieces a .tiktoken file doesn't carry: its pretokenizer scheme and
+  # special-token table (see Tokenizer.from_tiktoken and lib/gigatoken/
+  # encodings/PROVENANCE.md, the source of truth this is transcribed from).
+  # The HuggingFace ones are whole tokenizer.json files, which carry theirs.
   module Encodings
     DATA_DIR = File.expand_path("encodings", __dir__)
     private_constant :DATA_DIR
@@ -40,6 +41,10 @@ module Gigatoken
     HARMONY_RESERVED_TAIL = (200013..201087).to_h { |id| ["<|reserved_#{id}|>", id] }.freeze
     private_constant :HARMONY_RESERVED_TAIL
 
+    # Deep-frozen: entries, their `special_tokens` tables and the `rank_file` /
+    # `json_file` paths. `Tokenizer#special_tokens` hands the registry's own
+    # Hash back to callers, so anything less lets one caller's poke rewrite
+    # what every later `from_encoding` in the process loads.
     REGISTRY = {
       "r50k_base" => {
         rank_file: File.join(DATA_DIR, "r50k_base.tiktoken"),
@@ -66,8 +71,22 @@ module Gigatoken
         rank_file: File.join(DATA_DIR, "o200k_base.tiktoken"),
         pretokenizer: "o200k",
         special_tokens: HARMONY_HEAD_TOKENS.merge(HARMONY_RESERVED_TAIL).freeze
-      }
-    }.freeze
+      },
+      # The three below are gzipped HuggingFace tokenizer.json files, vendored
+      # verbatim (see PROVENANCE.md for repos, revisions and hashes). Nothing
+      # else is registered: the pretokenizer and the special tokens are the
+      # file's own, read at load as Tokenizer.from_json reads them.
+      #
+      # Qwen 3.5 and 3.6 share this one file.
+      "qwen35" => {json_file: File.join(DATA_DIR, "qwen35.json.gz")},
+      # The qwen35 file plus seven audio/TTS special tokens — vendored
+      # separately because a JSON-backed tokenizer takes its added tokens
+      # from the file.
+      "qwen38" => {json_file: File.join(DATA_DIR, "qwen38.json.gz")},
+      # meta-models/Muse-Glimmer-30B's file: Meta publishes no standalone
+      # Muse Spark tokenizer.
+      "muse_spark" => {json_file: File.join(DATA_DIR, "muse_spark.json.gz")}
+    }.each_value { |encoding| encoding.each_value(&:freeze).freeze }.freeze
     private_constant :REGISTRY
 
     # The packaged encoding names — the single source error messages naming
@@ -87,16 +106,17 @@ module Gigatoken
     private_constant :UNPACKABLE_REASONS
 
     class << self
-      # The {rank_file:, pretokenizer:, special_tokens:} registered for a
-      # packaged encoding name, or nil.
+      # The {rank_file:, pretokenizer:, special_tokens:} (tiktoken) or
+      # {json_file:} (HuggingFace) registered for a packaged encoding name, or
+      # nil. Names are Strings or Symbols, as Tokenizer.load accepts both.
       def [](name)
-        REGISTRY[name]
+        REGISTRY[name.to_s]
       end
 
       # Why `name` can't be packaged, or nil when there's no reason on
       # record (it's either packaged, or simply not one gigatoken knows of).
       def unpackable_reason(name)
-        UNPACKABLE_REASONS[name]
+        UNPACKABLE_REASONS[name.to_s]
       end
     end
   end

data/lib/gigatoken/hub.rb

@@ -2,6 +2,7 @@
 
 require "async"
 require "async/http"
+require "async/http/proxy"
 require "fileutils"
 require "pathname"
 
@@ -32,6 +33,39 @@ module Gigatoken
     MAX_REDIRECTS = 10
     private_constant :MAX_REDIRECTS
 
+    # Connect/read timeout in seconds, and the bound on the request phase —
+    # huggingface_hub's HF_HUB_ETAG_TIMEOUT / HF_HUB_DOWNLOAD_TIMEOUT
+    # default.
+    DEFAULT_TIMEOUT = 10
+    private_constant :DEFAULT_TIMEOUT
+
+    # A "." or ".." path segment: the traversal that must never reach a URL
+    # or the cache, wherever it comes from.
+    DOT_SEGMENT = /\A\.\.?\z/
+    private_constant :DOT_SEGMENT
+
+    # Everything outside RFC 3986's unreserved set, which is what
+    # huggingface_hub's `quote` percent-encodes.
+    RESERVED = /[^A-Za-z0-9\-._~]/
+    private_constant :RESERVED
+
+    # How a request, or the body read that follows it, fails underneath
+    # async-http: a connect/read timeout, a refused or reset connection, a
+    # proxy refusing the CONNECT tunnel, DNS resolution, a body cut short, a
+    # malformed endpoint, TLS. All of them reach the caller as
+    # Gigatoken::HubError.
+    TRANSPORT_ERRORS = [
+      Async::TimeoutError,
+      Async::HTTP::Proxy::ConnectFailure,
+      IOError,
+      SocketError,
+      SystemCallError,
+      URI::InvalidURIError,
+      Protocol::HTTP::Error,
+      OpenSSL::SSL::SSLError
+    ].freeze
+    private_constant :TRANSPORT_ERRORS
+
     class << self
       # Whether `name` is shaped like a HuggingFace Hub repo id: `org/name`,
       # or a bare legacy repo name like `gpt2`. At most one slash, and not
@@ -105,15 +139,44 @@ module Gigatoken
         env("HF_ENDPOINT") || DEFAULT_ENDPOINT
       end
 
+      # Whether `value` is usable as a URL path and a cache path component:
+      # not empty, not absolute, no NUL byte, no "." or ".." segment.
+      # huggingface_hub rejects the same traversals in validate_repo_id.
+      def safe_component?(value)
+        !value.empty? && !value.include?("\0") && !value.start_with?("/") &&
+          value.split("/", -1).none? { |segment| segment.match?(DOT_SEGMENT) }
+      end
+
+      # The proxy URL the environment names for a `scheme` request to
+      # `hostname`, or nil: http_proxy/https_proxy, lowercase spelling
+      # first, suppressed by a matching no_proxy entry — what requests does,
+      # and so what huggingface_hub inherits.
+      def proxy_url(scheme, hostname)
+        url = env("#{scheme}_proxy") || env("#{scheme.upcase}_PROXY")
+        url unless url.nil? || no_proxy?(hostname)
+      end
+
       private
 
+      # no_proxy is a comma-separated list of host suffixes, or "*" for
+      # everything.
+      def no_proxy?(hostname)
+        host = hostname.downcase
+        (env("no_proxy") || env("NO_PROXY")).to_s.split(",").any? do |entry|
+          entry = entry.strip.downcase.delete_prefix(".")
+          next true if entry == "*"
+
+          !entry.empty? && (host == entry || host.end_with?(".#{entry}"))
+        end
+      end
+
       def env(key)
         value = ENV[key]
         value unless value.nil? || value.empty?
       end
 
       def word_part?(part, first_alnum:)
-        return false if part.nil? || part.empty?
+        return false if part.nil? || part.empty? || part.match?(DOT_SEGMENT)
 
         first_ok = first_alnum ? part[0].match?(/[A-Za-z0-9]/) : word_char?(part[0])
         first_ok && part[1..].chars.all? { |c| word_char?(c) }
@@ -133,14 +196,26 @@ module Gigatoken
     #   for pointing at a local server in tests (dependency injection, not a
     #   mock); defaults to Hub.default_endpoint (HF_ENDPOINT, then
     #   huggingface.co).
-    def initialize(endpoint: self.class.default_endpoint)
+    # @parameter timeout [Numeric] connect/read timeout in seconds, applied
+    #   to every request and to the request phase as a whole.
+    def initialize(endpoint: self.class.default_endpoint, timeout: DEFAULT_TIMEOUT)
       @endpoint = endpoint.chomp("/")
-      @internet = Async::HTTP::Internet.new
+      @timeout = timeout
     end
 
     # Path of `filename` from Hub repo `repo_id` at `revision`, served from
-    # the standard HF cache, downloading into it first when absent.
+    # the standard HF cache, downloading into it first when absent. The
+    # three caller-supplied components are checked before any request or
+    # filesystem access: one of them carrying `..` would otherwise read and
+    # overwrite files outside the cache.
     def hub_file(repo_id, filename = "tokenizer.json", revision: "main")
+      {"repo id" => repo_id, "filename" => filename, "revision" => revision}.each do |what, value|
+        next if self.class.safe_component?(value)
+
+        raise HubError, "#{what} #{value.inspect}: must not be empty or absolute, " \
+          "or contain a NUL byte or a \".\" or \"..\" path segment"
+      end
+
       self.class.cached_file(repo_id, filename, revision) ||
         Sync { fetch(repo_id, filename, revision) }
     end
@@ -152,31 +227,127 @@ module Gigatoken
     # recording the branch ref so later lookups (ours and
     # huggingface_hub's) resolve it.
     def fetch(repo_id, filename, revision)
-      url = "#{@endpoint}/#{repo_id}/resolve/#{revision}/#{filename}"
+      clients = []
+      url = resolve_url(repo_id, filename, revision)
       token = self.class.hf_token
-      response = @internet.get(url, auth_headers(token))
+      headers = auth_headers(token)
+      response = get(url, headers, clients)
       # Unlisted headers parse as a Header::Generic (an Array of values);
       # x-repo-commit is always a single value, so flatten it to a String.
       commit = response.headers["x-repo-commit"]&.to_s
 
-      # Redirects are followed by hand: resolve/ URLs answer with the
-      # x-repo-commit header and a redirect to a CDN for LFS files, and the
-      # Authorization header must not travel to the other host.
+      # Redirects are followed by hand: a resolve/ URL answers an LFS file
+      # with a redirect to a CDN, a renamed repo with one to its new name,
+      # and both headers below need a rule of their own across the hop.
       hops = 0
       while (300...400).cover?(response.status)
-        ensure_ok!(url, response.status, !!token)
-        hops += 1
-        raise Error, "#{url}: too many redirects" if hops > MAX_REDIRECTS
-
-        location = response.headers["location"] ||
-          raise(Error, "#{url}: redirect with no Location header")
+        location = response.headers["location"]
         response.close
-        url = absolutize(location, url)
-        response = @internet.get(url, {"user-agent" => "gigatoken"})
+        raise HubError, "#{url}: redirect with no Location header" unless location
+        raise HubError, "#{url}: too many redirects" if (hops += 1) > MAX_REDIRECTS
+
+        target = absolutize(location, url)
+        # A renamed repo answers with a same-origin redirect that still needs
+        # the token; the LFS CDN elsewhere authenticates by signed URL and
+        # must never see it — huggingface_hub drops the header on the same
+        # rule.
+        headers = auth_headers(nil) unless same_origin?(target, url)
+        url = target
+        response = get(url, headers, clients)
+        # An LFS file carries x-repo-commit on the resolve/ hop; a renamed
+        # repo carries it only on the hop that finally answers 200.
+        commit ||= response.headers["x-repo-commit"]&.to_s
       end
-      ensure_ok!(url, response.status, !!token)
+      ensure_ok!(url, response, !!token)
+      ensure_commit!(url, commit)
+
+      write_to_cache(repo_id, filename, revision, commit, response)
+    rescue *TRANSPORT_ERRORS => e
+      raise HubError, "#{url}: #{e.message} (#{e.class})"
+    ensure
+      close_all(response, clients)
+    end
 
-      write_to_cache(repo_id, filename, revision, commit || revision, response)
+    # Release everything the fetch holds, in the one place every exit passes
+    # through. Order matters: an unread body keeps its connection checked
+    # out, and Async::HTTP::Client#close waits for its pool to drain, so the
+    # response goes first and the clients innermost-first — a tunnel client's
+    # connection is held by the proxy client opened under it. Bounded, and
+    # deaf to the ways a broken connection fails to close: a connection some
+    # failure left mid-stream would otherwise stall the drain, and with it
+    # the Sync this all runs inside, for good. A leaked socket is worth less
+    # than the caller's result or exception.
+    def close_all(response, clients)
+      Async::Task.current.with_timeout(@timeout) do
+        response&.close
+        clients.reverse_each(&:close)
+      end
+    rescue *TRANSPORT_ERRORS
+      nil
+    end
+
+    # `endpoint/repo/resolve/revision/filename`, percent-encoded the way
+    # huggingface_hub's `quote` does it: `revision` whole (`safe=""`), so
+    # `refs/pr/1` travels as `refs%2Fpr%2F1`, while the repo id and the
+    # filename keep their slashes.
+    def resolve_url(repo_id, filename, revision)
+      "#{@endpoint}/#{escape_path(repo_id)}/resolve/#{escape(revision)}/#{escape_path(filename)}"
+    end
+
+    def escape_path(value)
+      value.split("/", -1).map { |segment| escape(segment) }.join("/")
+    end
+
+    def escape(value)
+      value.gsub(RESERVED) { |char| char.bytes.map { |byte| format("%%%02X", byte) }.join }
+    end
+
+    # GET `url`, through the proxy the environment names for it when there
+    # is one: an http request travels to the proxy with the absolute URI in
+    # its request line, an https one through a CONNECT tunnel — how requests,
+    # and so huggingface_hub, routes them. The clients stay open (the body
+    # is still to be streamed); #fetch closes them.
+    def get(url, headers, clients)
+      endpoint = endpoint_for(url)
+      proxy = self.class.proxy_url(endpoint.scheme, endpoint.hostname)
+      proxy &&= endpoint_for(proxy)
+
+      client =
+        if proxy.nil?
+          open_client(endpoint, clients)
+        elsif endpoint.secure?
+          # https tunnels through the proxy with CONNECT, then speaks TLS to
+          # the origin as if it had connected to it directly.
+          open_client(open_client(proxy, clients).proxied_endpoint(endpoint), clients)
+        else
+          open_client(proxy, clients)
+        end
+      # An http proxy is addressed with the absolute URI in the request line;
+      # a direct or tunnelled request carries just the path.
+      target = (proxy && !endpoint.secure?) ? url : endpoint.path
+
+      request = Protocol::HTTP::Request["GET", target, headers, scheme: endpoint.scheme, authority: endpoint.authority]
+      # A CONNECT tunnel's socket carries no timeout of its own, so the
+      # request phase is bounded here whichever way it was routed; the body
+      # then streams under the endpoint's own connect/read timeout.
+      Async::Task.current.with_timeout(@timeout) { client.call(request) }
+    end
+
+    # A malformed URL comes back as URI::InvalidURIError, one that cannot be
+    # routed (no scheme or host) as ArgumentError; both mean the same thing
+    # to the caller, and neither is worth a backtrace.
+    def endpoint_for(url)
+      Async::HTTP::Endpoint.parse(url, timeout: @timeout)
+    rescue ArgumentError => e
+      raise HubError, "#{url}: #{e.message} (#{e.class})"
+    end
+
+    # A client for one hop, remembered in `clients` so #fetch can close it
+    # once the body is written. retries: 1 — a client built for this hop has
+    # no stale pooled connection for a retry to rescue, and async-http's
+    # default of 3 would pay @timeout three times over.
+    def open_client(endpoint, clients)
+      Async::HTTP::Client.new(endpoint, retries: 1).tap { |client| clients << client }
     end
 
     def auth_headers(token)
@@ -199,51 +370,75 @@ module Gigatoken
       rescue
         FileUtils.rm_f(tmp)
         raise
-      ensure
-        response.close
       end
       File.rename(tmp, target)
 
-      if !self.class.commit_hash?(revision) && revision != commit
-        refs_dir = repo_dir.join("refs")
-        FileUtils.mkdir_p(refs_dir)
-        File.write(refs_dir.join(revision), commit)
+      # A nested revision like refs/pr/1 is a nested ref file, so its parent
+      # has to exist — huggingface_hub mkdir -p's the same path.
+      if revision != commit
+        ref_path = repo_dir.join("refs", revision)
+        FileUtils.mkdir_p(ref_path.dirname)
+        File.write(ref_path, commit)
       end
 
       target
     end
 
-    def ensure_ok!(url, status, had_token)
+    # Raise on a non-success status. The unread body is left to #close_all,
+    # which every exit from #fetch passes through: it keeps the HTTP/1.x
+    # connection — and the Sync around it — alive until it is closed, so the
+    # exception would otherwise never reach the caller.
+    def ensure_ok!(url, response, had_token)
+      status = response.status
       return if (200...400).cover?(status)
 
       case status
       when 404
-        raise Error, "#{url}: HTTP 404 — no such repo with that file, and no such local file either"
+        raise HubError, "#{url}: HTTP 404 — no such repo with that file, and no such local file either"
       when 401, 403
         token_note = had_token ? "the request used the discovered token" : "no token was found"
-        raise Error,
+        raise HubError,
           "#{url}: HTTP #{status} — the repo may be private or gated (#{token_note}; set HF_TOKEN or run " \
           "`hf auth login`, and accept the repo's terms on huggingface.co if it is gated)"
       else
-        raise Error, "#{url}: HTTP #{status}"
+        raise HubError, "#{url}: HTTP #{status}"
       end
     end
 
+    # The snapshot directory is named by a server-controlled header, so only
+    # a real commit hash may become one: anything else would write the body
+    # wherever the header points. A missing header also means the endpoint
+    # isn't a Hub, whose snapshot would never be found again.
+    def ensure_commit!(url, commit)
+      return if self.class.commit_hash?(commit.to_s)
+
+      raise HubError, "#{url}: response is missing a usable x-repo-commit header (#{commit.inspect}) — it does not " \
+        "seem to be served by a HuggingFace Hub endpoint; if HF_ENDPOINT is set, check that it points to a " \
+        "Hub-compatible endpoint, and otherwise check your firewall and proxy settings"
+    end
+
     # A redirect Location resolved against the request URL: absolute URLs
     # pass through, host-relative (`/x/y`) and path-relative ones join the
     # base.
     def absolutize(location, base)
       return location if location.include?("://")
 
-      origin_end = base.index("://") ? base.index("://") + 3 : 0
-      origin_end = base.index("/", origin_end) || base.length
+      base_origin = origin(base)
+      return "#{base_origin}#{location}" if location.start_with?("/")
 
-      if location.start_with?("/")
-        "#{base[0...origin_end]}#{location}"
-      else
-        dir_end = base.rindex("/") || base.length
-        "#{base[0...[dir_end, origin_end].max]}/#{location}"
-      end
+      dir_end = base.rindex("/") || base.length
+      "#{base[0...[dir_end, base_origin.length].max]}/#{location}"
+    end
+
+    # Scheme and authority of a URL — everything before its path. Two URLs
+    # sharing one may pass the Authorization header between them.
+    def origin(url)
+      scheme_end = url.index("://")
+      url[0...(url.index("/", scheme_end ? scheme_end + 3 : 0) || url.length)]
+    end
+
+    def same_origin?(url, other)
+      origin(url).casecmp?(origin(other))
     end
   end
 end

data/lib/gigatoken/packed_result.rb

@@ -29,8 +29,12 @@ module Gigatoken
     end
 
     # Array of token ids for document `i`, materialized on demand; negative
-    # indices count from the end and out-of-range ones give nil, like Array.
-    def [](i)
+    # indices count from the end, out-of-range ones give nil, and a
+    # non-Integer index is a TypeError, all like Array.
+    def [](index)
+      i = Integer.try_convert(index)
+      raise TypeError, "no implicit conversion of #{index.class} into Integer" if i.nil?
+
       i += size if i.negative?
       return unless i >= 0 && i < size
 

data/lib/gigatoken/tokenizer.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require "json"
+require "zlib"
 
 module Gigatoken
   # A tokenizer: encode, batch encode, decode, and vocabulary introspection
@@ -10,18 +11,35 @@ module Gigatoken
     FILE_SOURCE_CLASSES = [Native::TextFileSource, Native::JsonlFileSource, Native::ParquetFileSource].freeze
     private_constant :FILE_SOURCE_CLASSES
 
+    # Encodings whose bytes reach the native call untouched. UTF-8 is checked
+    # by identity first — it is the hot path, see #encode — and these are the
+    # rest: US-ASCII already is UTF-8 bytes, and ASCII-8BIT is deliberately
+    # raw bytes. Anything else — ISO-8859-1, Windows-1252, UTF-16LE — is a
+    # real encoding whose bytes are not the text's UTF-8 bytes, so encoding it
+    # raw would give different ids than the same text read as UTF-8. (Invalid
+    # bytes in a UTF-8-tagged String stay raw: a documented difference from
+    # tiktoken, which rejects them.)
+    BYTEWISE_ENCODINGS = [Encoding::US_ASCII, Encoding::BINARY].freeze
+    private_constant :BYTEWISE_ENCODINGS
+
     # Load from in-memory tokenizer.json contents (String or bytes). Backed
     # by a BPETokenizer or a SentencePieceTokenizer, per the model's
-    # byte_fallback flag.
+    # byte_fallback flag. The document is parsed twice, deliberately: the Ruby
+    # parse runs first because JSON.parse's depth limit is what keeps hostile
+    # nesting away from the native parser (see parse_json), and it is also
+    # where the special-token table is read from.
     def self.from_json(data)
-      native = Native.load_hf_json(data)
-      new(native, special_tokens: special_tokens_from_json(data))
+      json = String.try_convert(data) or raise TypeError, "no implicit conversion of #{data.class} into String"
+      parsed = parse_json(json)
+      new(Native.load_hf_json(json), special_tokens: special_tokens_from(parsed))
     end
 
     # Load from a tokenizer.json path, or a directory containing one.
     def self.from_file(path)
-      path = File.join(path, "tokenizer.json") if File.directory?(path)
-      from_json(File.binread(path))
+      file = File.directory?(path) ? File.join(path, "tokenizer.json") : path.to_s
+      raise ModelError, "#{file.inspect}: no such file" unless File.file?(file)
+
+      from_json(File.binread(file))
     end
 
     # Load from a .tiktoken mergeable-ranks file. The file carries neither a
@@ -35,16 +53,18 @@ module Gigatoken
       new(native, special_tokens: special_tokens)
     end
 
-    # Load one of the tiktoken encodings gigatoken vendors ranks for, by
-    # name — see Gigatoken::Encodings::NAMES — entirely from the vendored
-    # files: no network, no writable cache.
+    # Load one of the encodings gigatoken vendors a file for, by name — see
+    # Gigatoken::Encodings::NAMES — entirely from the vendored files: no
+    # network, no writable cache. A tiktoken entry loads its ranks; a
+    # tokenizer.json entry is gunzipped and loaded by from_json.
     def self.from_encoding(name)
       encoding = Encodings[name]
+      return from_json(Zlib.gunzip(File.binread(encoding[:json_file]))) if encoding&.key?(:json_file)
       return from_tiktoken(encoding[:rank_file], pretokenizer: encoding[:pretokenizer], special_tokens: encoding[:special_tokens]) if encoding
 
       reason = Encodings.unpackable_reason(name)
       detail = reason ? " — #{reason}" : ""
-      raise Error, "#{name.inspect}: not a packaged encoding#{detail} (packaged encodings: #{Encodings::NAMES.join(", ")})"
+      raise ModelError, "#{name.inspect}: not a packaged encoding#{detail} (packaged encodings: #{Encodings::NAMES.join(", ")})"
     end
 
     # Load tokenizer.json from HuggingFace Hub repo `repo_id` at `revision`
@@ -71,7 +91,7 @@ module Gigatoken
       source = source.to_s
       if source.end_with?(".tiktoken")
         unless pretokenizer
-          raise Error, "#{source.inspect}: a .tiktoken file carries no pretokenizer scheme of its own — " \
+          raise ModelError, "#{source.inspect}: a .tiktoken file carries no pretokenizer scheme of its own — " \
             "pass pretokenizer: (one of #{Native.pretokenizer_names.join(", ")})"
         end
         return from_tiktoken(source, pretokenizer: pretokenizer, special_tokens: special_tokens)
@@ -80,25 +100,44 @@ module Gigatoken
       return from_encoding(source) if Encodings::NAMES.include?(source) || Encodings.unpackable_reason(source)
       return from_hub(source, revision: revision, hub: hub) if Hub.looks_like_repo_id?(source)
 
-      raise Error, "#{source.inspect}: no such file or directory, not a .tiktoken path, and doesn't look like a HuggingFace Hub repo id"
+      raise ModelError, "#{source.inspect}: no such file or directory, not a .tiktoken path, and doesn't look like a HuggingFace Hub repo id"
     end
 
     # `data` is UTF-8 JSON whatever its encoding tag says (File.binread tags
     # binary; a US-ASCII default_external tags that), so retag rather than let
     # JSON.parse transcode from the tag; a UTF-8-tagged String needs no copy.
-    def self.special_tokens_from_json(data)
+    # JSON.parse's default max_nesting of 100 is the guard: the native parser
+    # is recursive with no depth limit, and deep enough nesting overflows its
+    # stack rather than raising.
+    def self.parse_json(data)
       data = data.dup.force_encoding(Encoding::UTF_8) unless data.encoding == Encoding::UTF_8
-      added = JSON.parse(data)["added_tokens"] || []
+      JSON.parse(data)
+    rescue JSON::ParserError => e
+      raise ModelError, "failed to parse tokenizer JSON: #{e.message}"
+    end
+    private_class_method :parse_json
+
+    def self.special_tokens_from(parsed)
+      added = parsed["added_tokens"] || []
       added.each_with_object({}) { |t, h| h[t["content"]] = t["id"] if t["special"] }
     end
-    private_class_method :special_tokens_from_json
+    private_class_method :special_tokens_from
 
+    # The table is frozen so `#special_tokens` can neither alias the
+    # deep-frozen registry's Hash nor a `from_tiktoken` caller's; an
+    # already-frozen one is safe to share as-is.
     def initialize(native, special_tokens: {})
       @native = native
-      @special_tokens = special_tokens
+      @special_tokens = special_tokens.frozen? ? special_tokens : special_tokens.dup.freeze
     end
 
+    # A String already tagged UTF-8 — every ordinary document — reaches the
+    # native call over one inline identity check. A helper frame here is
+    # measurable against the encode itself at document sizes this small, so
+    # the check is spelled out rather than delegated to #transcode?; anything
+    # else takes the slow path in #utf8.
     def encode(text)
+      text = utf8(text) unless text.is_a?(String) && text.encoding == Encoding::UTF_8
       @native.encode(text)
     end
 
@@ -107,6 +146,7 @@ module Gigatoken
     # token ids plus per-document lengths), avoiding the per-token Ruby
     # array materialization the ragged shape costs.
     def encode_batch(texts, packed: false)
+      texts = utf8_batch(texts)
       if packed
         PackedResult.new(*@native.encode_batch_packed(texts))
       else
@@ -154,5 +194,34 @@ module Gigatoken
     end
 
     attr_reader :special_tokens
+
+    private
+
+    # UTF-8 first and by identity, so the batch walk below costs the same one
+    # comparison per document that #encode costs; the list is only consulted
+    # for the encodings that aren't it.
+    def transcode?(text)
+      text.is_a?(String) && text.encoding != Encoding::UTF_8 && !BYTEWISE_ENCODINGS.include?(text.encoding)
+    end
+
+    # A transcode that can't be done — a dummy encoding like UTF-7, bytes the
+    # tag doesn't allow, a character the target can't hold — is a fact about
+    # the document, not about the tokenizer, and reaches the caller as one.
+    def utf8(text)
+      transcode?(text) ? text.encode(Encoding::UTF_8) : text
+    rescue EncodingError => e
+      raise InputError, e.message
+    end
+
+    # The same rule per document, without copying the Array when — as in
+    # every UTF-8 batch — there is nothing to transcode. Elements are only
+    # type-checked, never coerced: a `to_str` object is left for the native
+    # side to convert under its own snapshot of the input.
+    def utf8_batch(texts)
+      array = Array.try_convert(texts)
+      return texts if array.nil? || array.none? { |text| transcode?(text) }
+
+      array.map { |text| utf8(text) }
+    end
   end
 end

data/lib/gigatoken/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Gigatoken
-  VERSION = "0.3.0"
+  VERSION = "0.5.0"
 end

data/lib/gigatoken.rb

@@ -4,9 +4,25 @@ require_relative "gigatoken/version"
 
 module Gigatoken
   # Raised for tokenizer load and encode failures surfaced from the native
-  # extension — never a raw Rust panic across the Ruby boundary.
+  # extension — never a raw Rust panic across the Ruby boundary. The base of
+  # the three below: rescue this to catch everything gigatoken raises.
+  # Anything narrower than the three has no class of its own — CLI usage
+  # errors and the odd leftover are this one directly.
   class Error < StandardError; end
 
+  # Everything Gigatoken::Hub raises: HTTP status, transport, timeout, repo-id
+  # / revision / filename / x-repo-commit validation.
+  class HubError < Error; end
+
+  # A document the tokenizer cannot take: an untranscodable or invalid-byte
+  # String, invalid UTF-8 on the SentencePiece path, an id outside the
+  # vocabulary in #decode.
+  class InputError < Error; end
+
+  # A tokenizer that cannot be loaded: bad or hostile JSON, a missing file or
+  # directory, an unknown or unpackable encoding name, a malformed .tiktoken.
+  class ModelError < Error; end
+
   class << self
     # The process-global encode-cache budget in bytes per worker (a parallel
     # batch encode may use up to workers x budget), applied to tokenizers of

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: gigatoken
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.5.0
 platform: aarch64-linux
 authors:
 - Eric Jacobs
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2026-09-19 00:00:00.000000000 Z
+date: 2026-10-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: async
@@ -75,7 +75,10 @@ files:
 - lib/gigatoken/encodings.rb
 - lib/gigatoken/encodings/PROVENANCE.md
 - lib/gigatoken/encodings/cl100k_base.tiktoken
+- lib/gigatoken/encodings/muse_spark.json.gz
 - lib/gigatoken/encodings/o200k_base.tiktoken
+- lib/gigatoken/encodings/qwen35.json.gz
+- lib/gigatoken/encodings/qwen38.json.gz
 - lib/gigatoken/encodings/r50k_base.tiktoken
 - lib/gigatoken/hub.rb
 - lib/gigatoken/packed_result.rb